Fire Them. Fire Them All.


Between 2005 and 2008, I worked as a principal engineer for Amazon where I had technical oversight responsibilities for a significant chunk of the Amazon.com retail website. Amazon is one of the most operationally competent companies on the planet but such competence doesn’t happen by accident.

The level of operational availability that Amazon achieves on its website is a consequence of intentional planning and foresight and it comes at a cost. To maintain availability in the face of unexpected events, substantial excess capacity is continuously maintained. At the time I was there, our operational doctrine required us to provision 150% of our expected peak load and to spread that total capacity across three separate geographies. This allowed for the possibility of losing an entire geography without losing the ability to still serve 100% of peak website requests. At one point while I was there, we were using fully 10% of our entire available capacity merely to probe the system for availability problems so that we would discover them before our customers did. A customer-visible problem caused by an engineer could be a career-ending event at Amazon during those years.

Part of my job was to serve on the “availability team”, which consisted of senior technical personnel who were tasked with doing an unforgiving postmortem analysis of every single event that affected the availability of the website. This “after-action” analysis was, to a certain extent, ruthless, and we did not play kissy-face games regarding human mistakes. Typically, if someone brought the website down due to having failed to follow operational best practice, he would be summarily fired. And the pre-firing interrogation could be…painful.

Which brings me to Texas.

As I write these words, I have been without power for 15 hours overnight and the temperature is 19 degrees outside. We were without power also the night before and the temperature was 1 degree in the morning. We’re doing well enough because we have a gas fireplace and we have rearranged the living room furniture – squeezed up close to the hearth we can generally maintain a situation where we’re not completely miserable. But others are not so fortunate and some are, quite literally, dying.

Something is seriously and dangerously amiss when websites are maintained with more competence and care than the competence being applied to the maintenance of the power grid which keeps actual human beings alive. There are preening politicians and bureaucrats in Austin who put plaques on their walls celebrating their “green” sensibilities, having supported the expansion of wind power and deepened Texans’ dependence on it. These politicians and bureaucrats should be made to stop licking the boots of the Greens and, instead, be frog-marched to the high plains of northwest Texas, and be made to lick the frozen turbines sitting motionless there, at least until they thaw enough to start turning again.

Texas has grown its dependence on wind energy to the point that 23% of Texas power is generated by wind turbines dotting the prairie landscape. But at the first unfortunate weather event, half of the wind generating capacity folded like a cheap suit, sending paroxysms of dysfunction throughout the power grid, leaving millions of Texans without power when they needed it most.

But that is not even the worst of it. The collapse of the power grid in the face of a mere 10-20% loss of generating capacity suggests we have been running the grid, something that human lives depend upon, with catastrophically thin operational margins. (We have apparently been running our healthcare system the same way, given the panic that ensued at the outset of the pandemic when we were besieged by a continuous alarmist drumbeat to “flatten the curve” lest we overwhelm ‘the system’.)

What the winter of 2021 has taught us is that the politicians and bureaucrats of Texas are less competent than the lowliest software engineer at amazon.com.

It appears 2021 is going to continue what started in 2020, when the “experts” and politicians were exposed for the pompous, fraudulent grifters that they are.

Fire them. Fire them all.

Written by

Keith Lowery

Follower of Christ. Husband. Father. Grandfather. Maker. Consumer of Data. Reader of Books. Writer of Code. Against the Darkness. For the Light.


Tags
PoliticsTechnologyCovid