Caveat promptor – Surfing Complexity
In the wake of a major incident, you’ll occasionally hear a leader admonish the engineering organization that we need to be more careful in the future in order to prevent such incidents from …| Surfing Complexity
In the wake of a major incident, you’ll occasionally hear a leader admonish the engineering organization that we need to be more careful in the future in order to prevent such incidents from …| Surfing Complexity
I was recently a guest on the This is Fine! podcast, hosted by Colette Alexander and Clint Byrum. Here’s a video clip from the episode.| Surfing Complexity
Here’s a thought experiment that John Allspaw related to me, in paraphrased form (John tells me that he will eventually capture this in a blog post of his own, at which time I’ll put a …| Surfing Complexity
2 posts published by Lorin Hochstein during October 2025| Surfing Complexity
Here’s another blog post on gathering some common threads from reading recent posts. Today’s topic is about the unassuming nature of talented software engineers. The first thread was a …| Surfing Complexity
According to the interim report on the elementary school shooting in Uvalde, there were 376(!) law enforcement officers that responded. How do we make sense of the fact it took such a long to neutr…| Surfing Complexity
(With apologies to the screenwriters of Forrest Gump) I’m going to use this post to pull together some related threads from different sources I’ve been reading lately. Rationalization as discarding information The first thread is from The Control Revolution by the late American historian and sociologist James Beniger, which was published back in the 1980s: … Continue reading A statistic is as a statistic does→| Surfing Complexity
Recent U.S. headlines have been dominated by school shootings. The bulk of the stories have been about the assassination of Charlie Kirk on the campus of Utah Valley University and the correspondin…| Surfing Complexity
3 posts published by Lorin Hochstein during April 2025| Surfing Complexity
4 posts published by Lorin Hochstein during June 2025| Surfing Complexity
2 posts published by Lorin Hochstein during September 2025| Surfing Complexity
A progressive rollout refers to the act of rolling out some new functionality gradually rather than all at once. This means that, when you initially deploy it, the change only impacts a fraction of your users. The idea behind a progressive rollout is to reduce the risk of a deployment by reducing the blast radius: … Continue reading The hidden trade-offs of fine-grained progressive rollouts→| Surfing Complexity
The Axiom of Experience: the future will be like the past, because, in the past, the future was like the past. – Gerald M. Weinberg, An Introduction to General Systems Thinking Last Friday, the San…| Surfing Complexity
You’ve undoubtedly heard of the psychological concept called flow state. This is the feeling you get when you’re in the zone, where you’re doing some sort of task, and you’re just really into it, and you’re focused, and it’s challenging but not frustratingly so. It’s a great feeling. You might experience this with a work … Continue reading My favorite developer productivity research method that nobody uses→| Surfing Complexity
There are software technologies that work really well in-the-small, but they don’t scale up well. The challenge here is that the problem size grows incrementally, and migrating off of them re…| Surfing Complexity
3 posts published by Lorin Hochstein during August 2025| Surfing Complexity
Accountability is a mechanism that achieves better outcomes by aligning incentives, in particular, negative ones. Specifically: if you do a bad thing, or fail to do a good thing, under your sphere …| Surfing Complexity
Amazon’s recent announcement of their spec-driven AI tool, Kiro, inspired me to write a blog post on a completely unrelated topic: formal specifications. In particular, I wanted to write abou…| Surfing Complexity
Here are a few anecdotes about safety from the past few years. In 2020, the world was struck by the COVID-19 pandemic. The U.S. response was… not great. Earlier in 2019, before the pandemic s…| Surfing Complexity
8 posts published by Lorin Hochstein during May 2025| Surfing Complexity
On Thursday (2025-06-12), Google Cloud Platform (GCP) had an incident that impacted dozens of their services, in all of their regions. They’ve already released an incident report (go read it!…| Surfing Complexity
Simplicity is prerequisite for reliability. — Edsger W. Dijkstra Think about a system whose reliability had significantly improved over some period of time. The first example that comes to my mind …| Surfing Complexity
One of the most famous physics experiments in modern history is the double-split experiment, originally performed by the English physicist Thomas Young back in 1801. You probably learned about this…| Surfing Complexity
I don’t know anything about your organization, dear reader, but I’m willing to bet that the amount of time and attention your organization spends on post-incident work is a function of …| Surfing Complexity
One of the criticisms leveled at resilience engineering is that the insights that the field generates aren’t actionable: “OK, let’s say you’re right, that complex systems ar…| Surfing Complexity
Safety researchers love using metaphors as a framework to describe how accidents happen, which they call accident models. One of the earliest models, dating back to 1931, is Herbert W. HeinrichR…| Surfing Complexity
If you’re a regular reader of this blog, you’ll have noticed that I tend to write about two topics in particular: Resilience engineering Formal methods I haven’t found many people…| Surfing Complexity
Laura Nolan of Slack recently published an excellent write-up of their Jan. 4, 2021 outage on Slack’s engineering blog. One of the things that struck me about this writeup is the contributing facto…| Surfing Complexity
(Some of my co-workers call this Lorin’s Law) Even highly reliable systems go down occasionally. After having read over the details of several incidents, I’ve started to notice a patter…| Surfing Complexity
FizzBee is a new formal specification language, originally announced back in May of last year. FizzBee’s author, Jayaprabhakar (JP) Kadarkarai, reached out to me recently and asked me what I …| Surfing Complexity
FAA data shows 30 near-misses at Reagan Airport – NPR, Jan 30, 2025 The amount of attention an incident gets is proportional to the severity of the incident: the greater the impact to the organizat…| Surfing Complexity
The sorry state of dashboards It’s true: the dashboards we use today for doing operational diagnostic work are … let’s say suboptimal. Charity Majors is one of the founders of Hon…| Surfing Complexity
Cloudflare consistently generates the highest quality public incident writeups of any tech company. Their latest is no exception: Cloudflare incident on November 14, 2024, resulting in lost logs. I…| Surfing Complexity
A play in one act Dramatis personae EM, an engineering manager TL, the tech lead for the team X, an engineering manager from a different team Scene 1: A meeting room in an office. The walls are ado…| Surfing Complexity
Justine Tunney recently wrote a blog post titled The Fastest Mutexes where she describes how she implemented mutexes in Cosmopolitan Libc. The post discusses how her implementation uses futexes by …| Surfing Complexity
Back in August, Murat Derimbas published a blog post about the paper by Herlihy and Wing that first introduced the concept of linearizability. When we move from sequential programs to concurrent on…| Surfing Complexity
Here’s a brief excerpt from a talk by David Woods on what he calls the component substitution fallacy (emphasis mine): Everybody is continuing to commit the component substitution fallacy. No…| Surfing Complexity
Cliff L. Biffle blogged a great write-up of a debugging odyssey at Oxide with the title Who killed the network switch? Here’s the bit that jumped out at me: At the time that code was written…| Surfing Complexity
Photo by Matthew Lancaster We know that not all of the services in our system are critical. For example, some of our internal services provide support functions (e.g., observability, analytics), wh…| Surfing Complexity
