portrait picture

TIMO ZIMMERMANN

balancing software engineering & infosec

AWS, CloudFlare and my ego

Last Saturday I migrated my blog to a self written app. No more Wordpress with its editor (which surprisingly still finds new ways to get worse with every release). No more CloudFlare as CDN (still have to migrate DNS). Just a Django app running on a virtual server behind a reverse proxy. That is it. Not really a setup most people would expect in 2021. This is not about the Django app or why I did not go with a static site generator, but why I ditched CloudFlare and our unhealthy reliance on a few infrastructure providers.

My first weblog - for my younger readers: that is what a blog was called when I was young - was a Perl script running on a shared hosting provider which actually offered Perl and Python beside PHP for CGI scripts. Whenever I published something new I added a key / value pair to the content hash. Till today I am convinced that this was peak blogging. So simple, yet powerful. Zero administration overhead. And it was fun to use. This might actually be the most important feature of all. While this might not work well with the amount of content I got today, something similar might still be possible and if I ever feel like it I will most likely explore this idea again.

One thing I do not know is what the uptime was. Because I did not care. If the site was down, it was down. Once my provider solved whatever problem they had it would be back up. Once or twice I pushed an update which broke something that was not obvious when doing a first smoke test. Thankfully someone emailed me that there was a problem and I fixed it. "Whoops, sorry - fixed. Thanks for reporting". Life was easy.

The more I transitioned from writing desktop applications to the web the more I learned about keeping websites and complex web apps available. At some point service level agreements needed to be met. "T3h cloud" with all its - true and false - promises happened. More aggressive SLAs showed up as reliance on web applications increased.

This is not supposed to sound like I am against service level agreements, there is a good reason they exist and they make a lot of sense in a business context. Spending a lot of time writing web applications and keeping them online, fast and reliable the mindset slowly became my standard modus operandi. I want the things I build to be as fast as possible. As fault tolerant as possible. Always available to meet SLAs. It is my job as a professional engineer and I take my job very serious.

When my blog got a bit more traction and needed a revamp I was stuck in my modus operandi. It needs to be fast. It needs to be always available. It needs to be all of the things which are relevant when working on a commercial project. How could I actually not meet all of the criteria, I am a professional, so this is the minimum I would expect of myself. Anything but an "A" in pingdoms speed test is unacceptable.

Except some tweaking of the HTML and CSS and benchmarking less conventional practices, like inlining all CSS on each page, there was not a lot to gain as I have always been a fan of minimalistic designs. So how do you keep the site always online and fast without spending serious money on the problem? The easy answer was generating a static site and fronting it with a CDN. CloudFront is a little bit of a frustrating experience, so after a few years I ended up with CloudFlare. The site was fast. And available. And got an "A".

Till it was not, because CloudFlare broke something. It felt like half of the Internet was down. The same that happens if an AWS availability zone or service is down. Or GitHub. We have a few central services which can cause problems for a lot of people if they are not available. And during this time it feels like those services power half of the Internet.

Beside familiarity with the tools, why are so many people using those services even for their private sites? Well, it is sometimes or mostly free, it is easy and if something happens nearly everyone is effected, so no individual would be blamed for their site being down.

Except, who would blame anyone anyway? Sites are sometimes down. Loosing a client, loosing a potential lead or having your reputation ruined because your site is down for a few minutes or an hour or two is unlikely. Especially if we are talking about a few thousand - which I think is a lot for a personal or small business / consulting site - visitors a day, none of them might actually notice.

Still, while availability surely does not need to be in the high 99.99999, hosting a website on a RaspberryPi with an LTE modem might not be the best solution for your business website - even if this setup seems to work shockingly well for some people. Same goes for not updating Wordpress plugins. Or not keeping your server OS updated. I do not think a bit of downtime is actually a big issue, but when a site is more down than up or selling viagra half of the time instead of IT consulting services it certainly would be a problem.

Compared to two decades ago hosting a website has become easy and inexpensive. If you have a static website you can get shared hosting with more than decent uptime and zero maintenance for 2€ per month. A domain included. 2€ is a lot of money for many people which is why I am thankful we got all the free alternatives. Virtual servers with enough resources to withstand the hug of death if an article makes it to the frontpage of HackerNews start around 3€ per month and give you lots of flexibility but also require a bit more work. Finding the sweet spot for your hosting needs is not that hard if you are willing to spend five minutes looking for offers from providers which are not AWS, Google or Microsoft.

Let me throw in an aside here. While I mostly talk about private or small business blogs and websites here, many hosters also have amazing offers for full production grade setups being able to power web apps at scale. The performance and resources of dedicated hardware you get get at Hetzner or OVH for example is astonishing if you are used to AWS. If you do not require fully managed services like RDS, a few dedicated boxes, or prefer to self manage your services anyway I would suggest checking out providers other than the big three. Many of them stepped up their game with floating IPs, firewalls and private networks making it straight forward to setup a production grade system for a lot less money than you would spend for a cloud offering. They are stable and reliable if a data center is not just burning down.

At some point we started to centralise online presences around a few providers. Providers which sometimes take the opportunity to lock accounts, delete content, enforce random rules they make up on the fly or simply inject cookies tracking your visitors - all which is their good right. They are a private business and you entered a contract with them. You play by their rules, if you like it or not.

Gone were the days of websites on Geocities, strange Perl scripts and PHP showing include errors. Personal websites became a portfolio. A CV. A way to get new business and employment opportunities. Personal was replaced with professional. I can only speak for myself, but for me it was mostly because of my ego. I am a professional. My site has to be fast. It has to be available. How could anyone take me serious if my own site is down for more than three seconds a year? How could I not use the same infrastructure that hosts so many large and important sites? How could I not optimise the single, last bit I deliver over the wire? For me, my ego was the only real reason.

Thinking about it makes me a bit sad. I tremendously enjoyed the days of Geocities, strange Perl scripts and include errors. And I partially gave up on it for the wrong reasons. Hosting a website was not more complex than getting a Route53, S3 and CloudFront setup to work properly. Self hosting today with a cheap VPS provider or shared hosting is not really more work than getting the former three setup properly. Availability is not a lot worse, if at all.

I love projects like neocities bringing back the spirit of small, personal websites. You get tons of features for free, great uptime and you support a small team building open source software while running a business. Not some faceless enterprise which tries to rip off OSS projects and grossly overcharges on things like traffic. Becoming a supporter for $5 / month is reasonably priced for all the features you get. Neocities essentially checks all the boxes for a great way to host your website.

This post is not meant to be some call to action to immediately rework your whole online presence for the holy crusade against big infrastructure providers. I am not suggestion that relying on those providers is bad. I had a few conversations about this more recently and thought about writing down my thoughts. I made a personal decision after thinking about the past, the present and what I want the future to look like. It feels like the right move to me to get at least one website off of the providers whose market dominance I do not appreciate.