September 1, 2015 - Issue 63
Hey folks,
Below I discuss the future of the newsletter and a little on how to have systems that are 100X (or more) lower cost than your competitors...
I discovered Ansible 2 years ago when writing the book "Taste Test: Puppet, Chef, SaltStack, Ansible" (3rd Edition was released today) and I was really blown away with Ansible's simplicity and ease-of-use. I could see it was a huge game-changer for systems - what used to take me weeks in Puppet or Chef took me hours in Ansible. Since then I've been advocating for Ansible and have helped many clients transition to it.
While I love Ansible, I'm finding that my focus is broader than that single tool and I often miss featuring other impressive systems tools and concepts. Also, my main newsletter has been left dormant for a year even though it is much larger and grows faster than this one. Rather than neglect them while I focus too narrowly on creating this newsletter, I'll be combining both newsletters into one called "Conquer Complexity".
What will I be covering in the new newsletter? Well, let me tell you the story of 2 companies and their systems. Both are former clients of mine, and demonstrate the polarized extremes of systems and their costs. (Company names are anonymized and the cost of engineering hours is included)
Intrepid had $115 million per year in revenue (and very profitable). For comparison, that's roughly the same as Twilio or Shopify. How many system administrators did they have? Zero (a developer spent about 1/4 of his time managing them). How many servers did they have? Four (1 web, 1 db, and a hot backup for each). How much did their systems cost? About $2500/month (~$30K/year). That's roughly $260 of systems cost per $1,000,000 of revenue.
Titanic had $5 million per year in revenue, $5 million in VC funding, and were not yet profitable. How many system administrators did they have? Three. How many servers did they have? Twenty (purchased the hardware and rented part of a colo to host them). How much did their systems cost? About $60,000/month (~$720K/year). That's roughly $144,000 of systems cost per $1,000,000 of revenue.
So, Titanic was spending about 550 times what Intrepid was in revenue-relative terms.
Let that sink in. They weren't spending 2X more, or 10X more, or even 100X more. They were spending 550X more. That would seem crazy extreme for anything else (accounting costs, plumbing costs, etc), but this is not only common in our industry, but commonly ignored.
Can you guess which company no longer exists? And can you guess which company now has over $300 million in annual revenue and a great growth rate? Yes, you are correct - it's so obvious I don't even have to say it.
Can you guess which company had super elite experts and followed "best practices" suggested by the top software vendors and consultants? This might be counterintuitive to some, but it was Titanic.
Can you guess which company had only 4 engineers total (including me) and none of them were systems experts at the time, so were cautious, skeptical, and just set up the absolute simplest thing possible? Yep, Intrepid.
I frequently get emails like this: "Hey Matt, would you be interested in joining our startup as our DevOps guy? We want to have awesome systems! We were hoping you could help us set up auto-scaling, multi-region redundancy, Docker, Hadoop, Cassandra, and you know, the other basic essentials. Thanks!"
Can you see the problem with this? They need zero of those things. Especially as a new startup. Most web companies can scale to over a $100 million in annual revenue and still not need any of those things.
The culture of systems advice has been deeply corrupted somehow. Engineers want to build complex Rube Goldberg machines that are insanely complex in order to demonstrate how "awesome" an engineer they are. That may impress other like-minded engineers, but that isn't going to impress your customers. Your customers care 0% about how cool your systems are.
Engineers that care more about their own awesomeness than that of their customer's are destined to fail at business.
Every hour wasted on unnecessary system complexity is an hour that doesn't go into building a great product for your customer.
The one upside of so many companies having terrible systems is that you can have a big advantage in out-competing your competition. You'll often be spending 100X less on your systems than they are. Assuming everything else is equal, you can run circles around them while they're busy trying to cram Docker into their systems so they can brag on their blog about how it took only 5 engineers 3 months to do.
The future of the newsletter will be focused on how to create extremely simple, low-cost, scalable systems. We'll also cover how to salvage and rescue complex legacy systems.
Now just a couple of housekeeping matters...
If you're not interested in this, that's totally ok, simply unsubscribe.
If you want to get Ansible updates, I'll still include some in this newsletter, but you'll probably want to join the "Ansible Announcements" Google Group.
Thanks!
Matt
P.S. Sometimes when I discuss this topic, engineers recognize their own overly complex and expensive systems and feel shame. Shame is counter-productive and not the goal here. Please don't feel bad! You're in good company! I've made nearly every mistake I'm talking about here at one time or another. The point is making progress from wherever your systems are right now and not worrying about the past.