Roastidio.us Tagged with elixir

Dropping Cloudflare for bunny.net | jola.dev

Tue, 07 Apr 2026 18:13:31 +0000

TL;DR my motivation and experience for moving my blog from Cloudflare to bunny.net

I’ve been a long time Cloudflare user. They offer a solid service that is free for the vast majority of their users, that’s very generous. Their infrastructure is massive and their feature set is undeniably incredible.

One of my biggest concerns though is around how easily I could become heavily dependent on this one single company that then can decide to cut me off and disable all of my websites, for any arbitrary reason. It’s a single point of failure for the internet. Every Cloudflare outage ends up in the news. And I can’t help but feel that the idea of centralizing the internet into a single US corporation feels off. Not to mention the various scandals that have surrounded them. So I was open to alternatives.

Bunny.net

Bunny.net (affiliate link because why not, raw link here) is a Slovenian (EU) company that is building up a lot of momentum. Their CDN-related services rival Cloudflare already, and although their PoP network is smaller than Cloudflare’s, they score highly on performance and speed across the globe. It’s a genuinely competitive alternative to Cloudflare.

It has the additional benefit of being a European company, and I like the idea of growing and supporting the European tech scene.

What I was moving away from

I’ve been using various different services, but focusing on this blog, the first thing was Cloudflare as the registrar for the domain name. I did some research on alternative registrars, but I just didn’t find any good European options. The closest I found was INWX, but their lack of free WHOIS Privacy made them a non-option. I ended up with Porkbun. They run on Cloudflare infrastructure, but they have better support. So the remaining thing Cloudflare was doing for me was the “Orange Cloud”: automatic caching, origin hiding, and optional protection features.

So that’s what we’re moving over! I’m gonna walk you through how to set up the bunny.net CDN for your website, with some sensible defaults.

Step by step

Setting up your bunny.net account is quick and you get $20 worth of free credits to play around with, those are valid for 14 days. You don’t need to give them a credit card up front to try things out, but if you do, you get another $30 worth of credits. You do need to confirm your email though before you can start setting things up. Once you’re out of the trial, you pay per use, which for most cases is cents a month. However, note that bunny.net require a minimum payment of $1 per month.

I guess a cheap price to pay to stop being the product and start becoming the customer.

Creating your pull zone

The pull zone is the main mechanism for enabling the CDN for your website. You’ll find them under CDN in the left navigation bar. Here’s how to set one up:

Fill in the pull zone name. Just make it something meaningful to you, for example the website name.
For origin type, select Origin URL.
Fill in your Origin URL. This would be the address for directly accessing your server. In my case, it’s the public IP of my server.
If you’re running multiple apps on your server, for example using Dokploy, coolify, or self-hosted PaaSs like that, you’ll want to pass the Host header as well. Here you put in the domain of your app. In my case, that’s jola.dev.
For tier, select Standard.
Finally you can select your pricing zones. Note that some zones are more expensive, so you can choose to disable them. This just means that people in those areas will get redirected to the closest zone you do have enabled.

And you’re done with the first part!

Configuring your pull zone

Now that you’ve set up the pull zone, it’s time to hook it up to your website and domain. Go to the pull zone you created. You’ll see a “hostnames” screen. Time to connect things.

Under “Add a custom hostname” fill in your website domain name.
You’ll get a modal with some instructions. You need to follow them to set up the DNS name to point your website to go through the CDN.
Go to where you manage domain name and add a CNAME record to point your domain to the given CNAME value in the modal, something like website.b-cdn.net.
Once you’ve done that, wait a few minutes to let it propagate, and then click “Verify & Activate SSL”.
If it says success, you’re done. Your website is now running through the bunny.net CDN, similar to the Cloudflare orange cloud.

Configuring caching

This is the part where bunny.net will really shine through!

If your website is set up to return the appropriate cache headers for each resource, things will just work. Bunny defaults to respecting the cache control headers when pointing a pull zone at an origin site. To verify, go to Caching → General and check that “Respect origin Cache-Control” is set under “Cache expiration time”. Note that if you set no-cache, bunny will use that and will not cache at the edge.

Alternatively, if you don’t have cache headers set up, and you don’t want to control that yourself, you can instead enable Smart Cache. This will default to caching typically cached resources like images, CSS, JS files etc, while avoiding caching things like HTML pages. This will work for most cases!

But I wanted to go faster. If you’ve read my post about building this website, here’s how I’ve set up my cache headers: I added a new pipeline in the router called public and added an extra middleware to it. I technically have everything using this pipeline, but leaving the standard browser pipeline that comes out of the box with Phoenix keeps my options open to add authenticated (uncached) pages in the future.

pipeline :public do
    plug :accepts, ["html"]
    plug :put_root_layout, html: {JolaDevWeb.Layouts, :root}
    plug :put_secure_browser_headers, @secure_headers
    plug :put_cdn_cache_header
  end
  
  defp put_cdn_cache_header(conn, _opts) do
    put_resp_header(conn, "cache-control", "public, s-maxage=86400, max-age=0")
  end

You can see the whole router here https://github.com/joladev/jola.dev/blob/main/lib/jola_dev_web/router.ex.

This setup means I even cache the HTML pages, which makes this ridiculously fast. Here’s the landing page response time from various locations, using the Larm response time checker tool:

Because I’m caching the HTML pages, if I publish a new post I do need to purge the pull zone to reset the cached HTML files.

Setting some sensible defaults

All of these are optional, but nice to have!

On your pull zone page, under General → Hostnames, go toggle “Force SSL” on for your domain to ensure that all requests use SSL. SSL/TLS is pretty standard these days, and many TLDs and websites use HSTS to enforce it, but no harm in enabling it here too.

DDoS protection comes out of the box, but we can set some other things up. First of all, go to Caching and then Origin Shield in the left menu on your pull zone, and activate Origin Shield. Select the location closest to your origin. This reduces load on your server, as bunny.net will cache everything in the Origin Shield location, and all edge locations will try that location first before hitting your server.

Next, go to Caching → General and scroll down. At the bottom of the page you can select Stale Cache: While Origin Offline and While Updating. This means bunny will keep serving cached content even if it is stale, if it can’t reach your origin, and that it will serve stale content while fetching the latest version. Both are nice to haves, nothing you have to enable, but provide a slightly better service to your users!

Next, let’s set up an Edge rule to redirect any requests to our automatically generated pull zone domain to our actual domain, to avoid confusing crawlers. On your pull zone, in the left menu, click Edge rules.

Add edge rule.
Name it “Default domain redirect”.
Under actions, select Redirect.
For URL, input your URL plus the path variable. Eg for me it’s https://jola.dev{{path}} .
Status code: use the default 301.
For conditions, pick Match any and Request URL Match any.
Input *://<slug>.b-cdn.net/* replacing <slug> with the name given to your pull zone.
Save edge rule!

Now you should be able to go to https://slug.b-cdn.net for your pull zone and get redirected to your proper domain!

Conclusion

This post just covers the very basics of getting set up on bunny.net. I haven’t even scratched the surface of edge rules, cache configuration, the Shield features for security and firewalls, video hosting and streaming, edge scripting and edge distributed containers, and much more.

I especially appreciate the great statistics, logs, and metrics you get out of the dashboard. You can even see every single request coming through to help you investigate issues, and clear feedback on what’s getting cached and not. I’m actively moving everything else over and I’m excited for the upcoming S3 compatible storage!

You should give bunny.net a try!

Newres Al Haider

Mon, 06 Apr 2026 18:15:11 +0000

Growing Yggdrasil, the World Tree, with Ash

2026-04-05

Declarative programming can be a powerful paradigm for organizing software systems. By defining the business processes once, we ensure there is a single source of domain knowledge. From this foundation, we can derive other parts of the system such as API endpoints, database schemas, and even user interfaces. This approach reduces repetition and helps prevent bugs caused by misaligned domain models.

The Ash Framework seems like an excellent way to see declarative programming in action. Written in Elixir, it allows you to describe your domain in a consistent and expressive way, from which it can automatically generate data layers, REST or GraphQL APIs, and admin interfaces. With Ash, you define what your application should do, and the framework takes care of how to make it happen. It can derive JSON REST endpoints, handle validation, manage persistence, and provide authorization logic, all from the same declarative definitions.

I am new to Ash and I tend to learn best by writing things out, so this article is as much for me as it is for you. Rather than trying to understand everything up front, I prefer to get hands-on quickly with a small, self-contained project. We’ll start with the basics here, and if things go well, expand on it in a follow-up or two.

Given the name Ash, it felt appropriate to build something inspired by the gigantic ash tree of Norse mythology: Yggdrasil, the World Tree.

An image of a Yggdrasil the world tree as a cybernetic Ash tree.

Yggdrasil is said to connect the Nine Worlds of Norse cosmology, though the exact number and nature of these worlds vary between sources. Each world has its own nature, inhabitants, and relationships with the others, making it an ideal metaphor for exploring how Ash models resources, attributes, and relationships. In this project, we will create a domain model for these concepts with Ash and derive a JSON REST API from them.

The first step is getting started with a basic Ash project for which we will use the Igniter tool. (I will assume Elixir is already installed, but if not, see the Elixir Install page for instructions). This is used for project setup and code generation, which will help us get started a lot quicker.

To start off following command will install Igniter:

mix archive.install hex igniter_new

Once we have Igniter, the next step is creating a new project in the yggdrasil directory, adding Ash, and moving into it.

mix igniter.new yggdrasil --install ash && cd yggdrasil

This will land us in a new Elixir project directory with Ash installed. From this seed we will evolve our application to represent the worlds and characters of Norse mythology.

The newly created project comes with a hello function in the lib/yggdrasil.ex module. Let's try it out in the iex, the Elixir interactive shell which we can start with:

iex -S mix

Running it will bring us into the shell, where we can run the hello function in the yggdrasil module:

iex(1)> Yggdrasil.hello()
:world

We now have a hello world, but in yggdrasil we want to represent the worlds of Norse mythology that are linked by Yggdrasil. The first thing we need for this is a Domain. This will function as a container for the various concepts, such as the worlds, that we will introduce later.

For simplicity's sake, first we will replace the contents of lib/Yggdrasil.ex with the following:

defmodule Yggdrasil do
  @moduledoc """
  The Yggdrasil domain — acts as the trunk of the tree
  and organizes all resources like World and Character.
  """

  use Ash.Domain

  resources do
    # Resources will be registered here
    resource Yggdrasil.World
  end
end

We create a file lib/resources/world.ex with the following contents:

defmodule Yggdrasil.World do
  @moduledoc """
  A resource representing a world in Yggdrasil.
  """

  use Ash.Resource,
    # in-memory store
    data_layer: Ash.DataLayer.Ets,
    domain: Yggdrasil

  actions do
    create :create do
      accept [:name, :description]
    end

    update :update do
      accept [:description]
    end

    # Provide default actions
    defaults [:read, :destroy]
  end

  attributes do
    # Primary key
    uuid_primary_key :id

    # World name and description
    attribute :name, :string, allow_nil?: false, public?: true
    attribute :description, :string, public?: true
  end
end

And finally

config :yggdrasil, :ash_domains, [Yggdrasil]

With these files in place, we now have a minimal Ash domain containing a single resource: World. Let’s take a moment to unpack what we just created before moving on.

At the top level, Yggdrasil acts as our domain, the trunk of our system. It brings together all the resources that make up the application and defines how they relate to each other. Right now, our domain only includes one resource, Yggdrasil.World, but we’ll add more later.

The Yggdrasil.World module itself is declared as a resource. In Ash, a resource is the fundamental building block. It describes a specific type of data and what can be done with it. Instead of writing separate schemas, changesets, and controllers, we declare everything about a resource in one place, and Ash takes care of the details.

Our World resource uses the Ash.DataLayer.Ets data layer, which stores data in Elixir’s in-memory ETS tables. This setup is fast and simple, making it perfect for early experimentation, though data won’t persist between runs. Later, this can be swapped out for a different data layer to gain full persistence. The argument domain: Yggdrasil connects the resource back to the domain we just defined so that the framework knows where it belongs.

Inside the actions block, we declare what operations are available for this resource. The create action accepts a name and a description, while the defaults [:read, :destroy] line automatically adds the standard read, update, and delete actions. There’s no need to write any manual CRUD logic—Ash generates it for us.

The attributes block defines the structure of each world. Every world has a UUID primary key (:id) and two fields, :name and :description. The :name attribute is made required using (allow_nil?: false), ensuring that each world must have one. Both attributes are marked public?: true so they appear in APIs and outputs.

Finally, the configuration line we added tells Ash which domains to load when the application starts. Without this, the framework wouldn’t know about our new resource.

At this point, our small Ash tree has already taken root. We’ve declared the first piece of our domain, and Ash now knows how to create, read, update, and delete worlds. Let’s see that in action next by exploring our resource interactively in iex.

First, start the interactive shell from your project root:

iex -S mix

Once inside iex, we want to create our first world Asgard, the shining realm of the gods, with the following command:

asgard = (
  Yggdrasil.World
  |> Ash.Changeset.for_create(:create, %{
       name: "Asgard",
       description: "A shining realm of order and power, suspended high above the clouds."
     })
  |> Ash.create!()
)

which would return the world as such:

14:29:14.665 [debug] Creating Yggdrasil.World:

Setting %{id: "726b678e-6cb6-4277-b291-85ecfa313d3a", name: "Asgard",
 description: "A shining realm of order and power,...}

%Yggdrasil.World{
  id: "726b678e-6cb6-4277-b291-85ecfa313d3a",
  name: "Asgard",
  description: "A shining realm of order and power, suspended high above the clouds.",
  __meta__: #Ecto.Schema.Metadata<:loaded>
}

There are a multiple things happening here, so let's unwrap things step by step.

First we start off with our Ash resource that we have defined Yggdrasil.World. In Ash, resources describe the structure of our data, including attributes like name and description, as well as the actions that can be performed on them.

Next we are using the pipe operator. |>, to pass this result to our next function. Elixir’s pipe operator takes the result of the expression on the left and passes it as the first argument to the function on the right.

For example instead of writing:

function(value, a, b)

with the pipe operator we can equivalently write:

value |> function(a, b)

This allows us to write a sequence of operations in a readable, step-by-step style. In our code example, it means Yggdrasil.World is passed into the Ash.Changeset.for_create function, as the first parameter. This function also takes the identifier of our create action :create, as well as the structure representing Asgard, with its name and description.

What this function returns is a changeset, a data structure representing the intended change of a resource in Ash (e.g.: creating, updating, etc). This is especially useful when it comes to validation and error checking, as we will see it later down the line. For now we use this changeset and pipe it into the function that executes the actual creation: Ash.create!().

The resulting value is a %Yggdrasil.World{} struct, which represents the newly created world. Ash also automatically generated a UUID for the id field, which uniquely identifies this world inside the system.

Before returning the struct, Ash logs the operation it performed. That is why we see the debug output:

[debug] Creating Yggdrasil.World

The final line is the Elixir struct that was created:

%Yggdrasil.World{
  id: "726b678e-6cb6-4277-b291-85ecfa313d3a",
  name: "Asgard",
  description: "A shining realm of order and power, suspended high above the clouds.",
  __meta__: #Ecto.Schema.Metadata<:loaded>
}

This struct is also stored in the variable asgard, so we can reference it later in the session.

Now that we understand how creating a world works, let’s add another one.

midgard = (
  Yggdrasil.World
  |> Ash.Changeset.for_create(:create, %{
       name: "Midgard",
       description: "The realm of humans, bound to the earth and everyday struggles."
     })
  |> Ash.create!()
)

This follows the exact same pattern as before. We build a changeset that describes the creation of Midgard, and then execute it with Ash.create!(). Much simpler than the mythological creation of Midgard, which involved the slaying of the giant Ymir.

Now that we have some worlds, let's read them using the read action:

worlds = (
  Yggdrasil.World
  |> Ash.Query.for_read(:read)
  |> Ash.read!()
)

which would give us our list of worlds:

[
  %Yggdrasil.World{
    id: "some-uuid-1",
    name: "Asgard",
    description: "A shining realm of order and power, suspended high above the clouds."
  },
  %Yggdrasil.World{
    id: "some-uuid-2",
    name: "Midgard",
    description: "The realm of humans, bound to the earth and everyday struggles."
  }
]

As one can expect, we can also do an update call. For example, let’s change the description of Asgard:

asgard = (
  asgard
  |> Ash.Changeset.for_update(:update, %{
       description: "The fortified realm of the Aesir, ruled by Odin."
     })
  |> Ash.update!()
)

There are a few things to note here. Instead of starting from the Yggdrasil.World module, we now start from the existing asgard struct. This is because we are modifying a resource that already exists.

The function for_update creates a changeset that describes the intended update. Just like with creation, the changeset itself does not perform the update, it only represents the change we want to make.

We then pass this changeset into Ash.update!(), which executes the update. Ash applies the changes, runs any validations, and returns the updated %Yggdrasil.World{} struct.

We can verify the change by reading the list of worlds again:

worlds = (
  Yggdrasil.World
  |> Ash.Query.for_read(:read)
  |> Ash.read!()
)

which would give us a result such as:

[
  %Yggdrasil.World{
    id: "6b62b3ea-b08b-4387-8539-37e645e53026",
    name: "Midgard",
    description: "The realm of humans, bound to the earth and everyday struggles.",
    __meta__: #Ecto.Schema.Metadata<:loaded>
  },
  %Yggdrasil.World{
    id: "d2646509-6c92-4049-a2db-0555612fc365",
    name: "Asgard",
    description: "The fortified realm of the Aesir, ruled by Odin.",
    __meta__: #Ecto.Schema.Metadata<:loaded>
  }
]

An interesting thing we could try out is updating the name of a world instead:

asgard2 = (
  asgard
  |> Ash.Changeset.for_update(:update, %{
       name: "Asgard2"
     })
  |> Ash.update!()
)

We get the following error:

** (Ash.Error.Invalid)
Invalid Error

* No such input `name` for action Yggdrasil.World.update

The attribute exists on Yggdrasil.World, but is not accepted by Yggdrasil.World.update

Perhaps you meant to add it to the accept list for Yggdrasil.World.update?


Valid Inputs:

* description

This is because when we were defining our update action in our module, the only attribute we accept is :description, see fragment below:

update :update do
      accept [:description]
    end

In other words, while the name attribute exists on the resource, it is not allowed to be modified through the update action. This is a domain modelling decision, and gives us fine-grained control over how our data can change. In this case, we decided that a world’s name is fixed after creation, while its description can evolve over time.

Finally we get to do delete, where we destroy asgard, our Ragnarok action if you will. We can do this by the following:

Ash.destroy!(asgard)

Ash.destroy! takes a resource struct, in this case asgard, and removes it from the data store. Since we’re using an in-memory ETS store, it deletes it from memory immediately. The function should return :ok on success. We can double check this by requesting our list of worlds again by our usual means:

worlds = (
  Yggdrasil.World
  |> Ash.Query.for_read(:read)
  |> Ash.read!()
)

which returns only Midgard:

[
  %Yggdrasil.World{
    id: "6b62b3ea-b08b-4387-8539-37e645e53026",
    name: "Midgard",
    description: "The realm of humans, bound to the earth and everyday struggles.",
    __meta__: #Ecto.Schema.Metadata<:loaded>
  }
]

At this point, we’ve taken the first steps in modeling our little piece of Yggdrasil. We have a domain, a resource, and a way to create, read, update, and delete worlds, enough to bring about a small Ragnarok.

Next, we will explore how we can start connecting resources together. After all, the worlds need their heroes and villains to really come alive.

How Many Paradigms Does It Take to Screw In a Lightbulb?

Mon, 06 Apr 2026 12:52:41 +0000

A developer who knows only one programming paradigm resembles a carpenter whose entire toolbox contains a single hammer. Naturally, a hammer will drive a nail with admirable precision. Or a screw, if sufficient enthusiasm is applied. But try to saw or plane a board with that hammer, and it becomes immediately clear—assuming you’ve encountered a saw or a plane at least once in your life—that the instrument has been chosen poorly. So it is with paradigms: knowledge of nothing but imperative programming, or nothing but object-oriented design, transforms a developer into a mechanical executor of tasks, incapable of seeing an elegant solution even when it lies on the surface, waiting to be noticed.

The narrowness of a programmer trapped in a single paradigm manifests in everything. They will erect loops where a single higher-order function would suffice. They will breed classes and inheritance where a pure function and composition would have been more than enough. They will attempt to verify the correctness of an algorithm with a debugger and tests instead of proving it formally at the type level. Such a developer resembles a tourist who knows exactly one word of the foreign language and is attempting, with its help, to explain a route across the entire city to a taxi driver. And it’s a small mercy if the word isn’t obscene.

Let us, for a start, walk through the principal paradigms and see what instruments each offers for solving problems. We’ll begin with the most ancient and familiar—the imperative paradigm.

Imperative programming is the world of instructions and mutable state. The programmer tells the machine: do this, then that, change this variable, repeat five times. A classical example in C:

intsum=0;for(inti=0;i<10;i++){sum+=i;}

Here we explicitly manage the state of the variable sum, accumulating the result step by step. This is natural for the machine, but tedious for the human. Every step must be spelled out, every mutation tracked. The imperative style serves well when the task reduces to a sequence of actions with side effects: write to a file, update a database, print to the screen. But as soon as the task grows in complexity, the code devolves into a tangle of interrelated variables and conditions.

Procedural programming is the imperative approach enriched with structures and functions. We group instructions into procedures to avoid repetition and improve readability. The same example:

intcalculate_sum(intn){intsum=0;for(inti=0;i<n;i++){sum+=i;}returnsum;}

Now the logic is packaged into a function that can be reused. The procedural style dominated the era of Pascal and early C. It taught programmers to think in modules and structure their code, but it never freed them from the problems of mutable state and side effects.

Object-oriented programming (in Gosling’s understanding, not Kay’s) promised to solve all problems at once: encapsulation, inheritance, polymorphism—the three pillars upon which the entire world supposedly rests. Data and methods unite into objects, objects assemble into class hierarchies. It sounds splendid, until you begin to examine how the code actually works:

classCounter{privateintvalue=0;publicvoidincrement(){value++;}publicintgetValue(){returnvalue;}}

State lives inside the object, convenient methods form the API, full encapsulation achieved. So it would seem, but the state hasn’t gone anywhere—it has merely relocated into a class field. And along with it relocated all the old afflictions: data races in multithreading, the difficulty of testing, the unpredictability of behavior. The object-oriented approach serves well for modeling a domain when you need to describe entities and their interactions. But it transforms into a nightmare when class hierarchies sprawl to dozens of inheritance levels, and half the methods exist solely to pass a call further down the chain.

Functional programming looks at the task from an entirely different angle. Here there is no mutable state, no loops, no side effects. There are only functions that receive data and return results. The same summation example in Haskell:

sum=foldl(+)0[0..9]

One line instead of five. No loops, no intermediate variables. The function foldl takes (1) an addition operation, (2) an initial value, and (3) a list, returning the result. The code reads like a mathematical expression, not a sequence of commands. The functional style is particularly well suited for working with collections, for building data-processing pipelines, for parallel computation. When there is no mutable state, there is no need for locks and synchronization. Functions can be safely launched simultaneously on different processor cores. Though for the domain of Accounting for a liquor store in the suburbs—it’s a rather dubious ally.

Logic programming overturns one’s very notion of how to write code. Instead of explaining how to solve a task, the programmer describes what they want to obtain. The system finds the solution on its own. Prolog is the classical representative of this paradigm:

parent(tom,bob).parent(tom,liz).parent(bob,ann).grandparent(X,Z):-parent(X,Y),parent(Y,Z).

We described kinship relations and a rule for determining grandparents. Now we can pose the question: grandparent(tom, ann)?—and the system will answer “yes,” having found the path through the facts. Logic programming is indispensable in certain corners of artificial intelligence, expert systems, and task planning. I even dragged it into the consistency validation of finite automata in one of my libraries. But an attempt to write a web server in Prolog would look rather like an attempt to hammer a mole with a microscope.

Declarative programming is a general term for approaches where the programmer describes the desired result rather than the sequence of steps. SQL is the textbook example:

SELECTnameFROMusersWHEREage>18ORDERBYname;

We don’t explain how to traverse the table, how to check the condition, how to sort the result. We simply declare: I want the names of users over eighteen, sorted alphabetically. The database will figure out how to do this efficiently on its own. The declarative style dominates in HTML, CSS (for now—I suspect someone will drag recursion into it before long), and configuration files. It allows one to separate the what from the how.

Concatenative programming is built on the idea of function composition via a stack. Forth is its most vivid representative:

: square dup * ;
5 square .

The function square duplicates the top element of the stack and multiplies it by itself. The number 5 is placed on the stack, the function square is applied, the result is printed. The code reads right to left, like reverse Polish notation. Concatenative languages are compact and efficient, but they demand a particular cast of mind. They remain popular in embedded systems and wherever code size and execution speed are critical.

Reactive programming focuses on data streams and the propagation of changes. When a data source changes, all dependent computations update automatically. An example in RxJS:

constclicks=fromEvent(document,'click');constpositions=clicks.pipe(map(event=>event.clientX));positions.subscribe(x=>console.log(x));

We create a stream of click events, transform it into a stream of coordinates, and subscribe to changes. Each click automatically produces the coordinate in the output. The reactive style is ideal for interfaces, event handling, and working with asynchronous data sources. It liberates you from callback hell and makes the data flow explicit.

Aspect-oriented programming addresses the problem of cross-cutting concerns—logging, caching, access control. Instead of smearing these aspects across the entire codebase, they can be described separately:

@Transactional@LoggedpublicvoidupdateUser(Useruser){repository.save(user);}

The annotations @Transactional and @Logged are aspects. They will be automatically “applied” to the method, wrapping it in a transaction and adding logging. The core code remains clean and comprehensible. The aspect-oriented approach is popular in enterprise development, where cross-cutting concerns permeate the entire system.

Metaprogramming is the programming of programs that write programs. Macros in LISP allow code to be generated at compile time:

(defmacrowhen(condition&restbody)`(if,condition(progn,@body)))

The macro when expands into an if construct with a progn block. Metaprogramming grants extraordinary flexibility, enabling the creation of domain-specific languages right inside the host language. But with great power comes great responsibility: poorly written macros turn code into an unreadable mess. If you want to see what metaprogramming looks like when practiced by a sane person—take any of my libraries, or write your own in Elixir. I know of no other language where macros have been done properly.

Dependently-typed programming elevates the type system to a new plane. Types can depend on values, allowing complex invariants to be expressed at the type level.

dataVec(A:Set):Nat->Setwhere[]:VecAzero_::_:{n:Nat}->A->VecAn->VecA(sucn)append:{A:Set}{mn:Nat}->VecAm->VecAn->VecA(m+n)

The type Vec A n is a vector of elements of type A with length n. The function append takes two vectors of lengths m and n and returns a vector of length m + n. The compiler verifies correctness at the type level. It is impossible to write a function that violates the length invariant. Dependent types are used for the formal verification of critical systems, where an error costs far too much.

Theorem-proving as a paradigm is the proof of program correctness by mathematical methods. Lean and Coq allow one to write not merely code, but proofs that the code does precisely what was intended:

theoremadd_comm(nm:Nat):n+m=m+n:=byinductionnwith|zero=>simp[Nat.zero_add,Nat.add_zero]|succnih=>simp[Nat.succ_add,Nat.add_succ,ih]

This is not simply an addition function—it is a proof that addition is commutative. The compiler doesn’t merely check types; it checks the mathematical proof. This approach is employed in cryptography, compilers, and operating systems—domains where the price of an error is measured not in irritated users, but in human lives or millions of dollars in losses.

The actor model views a program as a collection of independent actors that exchange messages. Each actor has its own mailbox, processes messages sequentially, and can create new actors. Erlang was built upon this idea:

-module(counter).-export([start/0,loop/1]).start()->spawn(fun()->loop(0)end).loop(N)->receive{increment,Pid}->Pid!{value,N+1},loop(N+1);{get,Pid}->Pid!{value,N},loop(N)end.

The actor counter receives increment and get messages, modifies its state, and replies. No shared data, no locks. Actors scale horizontally, failures are isolated. This model is ideal for distributed systems, where failures are the norm rather than the exception.

Dataflow programming describes computation as a graph of data streams. The nodes of the graph are operations, the edges are data flows between them. A change in one node propagates automatically through the graph. LabVIEW uses visual dataflow programming for hardware control. The approach is intuitive for engineers accustomed to thinking in schematics and diagrams.

Constraint programming describes a task as a set of constraints that must be satisfied. The system searches for a solution by enumerating possibilities and pruning the impossible. MiniZinc is a language for constraint programming:

var 1..9: x;
var 1..9: y;
constraint x + y = 10;
constraint x * y = 21;

Two variables, two constraints. The system will find x = 3, y = 7 or x = 7, y = 3. Constraint programming is applied in planning, scheduling, and resource optimization—wherever a task is formulated as finding a solution under constraints.

Phew.

Now let us pose the question: why does any of this matter to an ordinary developer? The answer is simple and simultaneously non-obvious. Each paradigm is a way of thinking, an approach to solving problems. A programmer who knows only imperative programming will solve every task with loops and conditionals. They will see a list-processing task and write a for loop with intermediate variables. A programmer acquainted with the functional paradigm will write map or fold—elegantly, concisely, free of side effects. One who has mastered reactive programming will construct an event-processing pipeline where each stage is explicitly described and easily testable.

Knowledge of different paradigms expands one’s arsenal of tools. You won’t write a web server in Prolog or prove theorems in JavaScript. But an understanding of logic programming will help you formulate conditions more precisely and build database queries. Familiarity with dependent types will teach you to think in invariants and express constraints at the type-system level. Experience with actors will show you how to build scalable distributed systems without the headaches of synchronization.

In truth, in the modern world all mature languages have long since become multi-paradigm. Scala combines object-oriented and functional approaches. Rust adds a powerful ownership and borrowing system to the imperative style. Python allows one to write procedurally, in an object-oriented fashion, and functionally. F# unites functional programming with the .NET ecosystem. Swift attempts to incorporate elements of all major paradigms at once. A programmer who understands when an aspect is needed (yes, in any language—for instance, I dragged aspects intoElixir) uses the language to its full power. One who knows only a single paradigm writes in any syntax as though it were PHP.

Paradigms are not a religion where you must choose one true faith and wage war on the heretics. They are tools, and a good craftsman knows when to reach for the hammer, when for the saw, and when for the plane. Need to parse something? Take the functional approach with map and fold. Build a system with thousands of simultaneous connections? Actors are your choice. Formally prove an algorithm’s correctness? Welcome to Lean or Agda. Developing an interface with many interactive elements? Reactive programming will make the code comprehensible.

A programmer trapped in a single paradigm is condemned to solve problems inefficiently. They will drag familiar patterns behind them even when those patterns don’t fit. They will write a class where a function would suffice. They will create mutable state where it could be avoided entirely. They will erect a complex hierarchy where composition would have been enough. They resemble a person who knows only one route from home to work and stubbornly waits at the bus stop every day, even though the road has been torn up for a month and the bus now runs on the next street over.

If a developer claims the badge of mid-level-plus but doesn’t feel at ease in at least the five principal paradigms—they are a pompous fool, and you should show them the door.

Rails on the BEAM

Sat, 04 Apr 2026 02:36:04 +0000

Rails on the BEAM

2026-04-02T23:21:00Z

Same blog from the Twilight Zone post. Same models, controllers, views. Same Turbo Streams broadcasting. Same Action Cable protocol. But check what's serving it:

npx github:ruby2js/juntos --demo blog
cd blog
npx juntos db:prepare
npx juntos up -d sqlite_napi

Open http://localhost:3000. Open a second tab. Create an article. Watch it appear in both. That part you've seen before.

Now imagine a bug in a request handler crashes one of the JavaScript runtimes. On Node.js, that takes down the process — connections drop, state is lost, restart from scratch. On the BEAM, the OTP supervisor restarts just that runtime. The other runtimes keep serving. WebSocket connections stay open. Turbo picks up where it left off.

What Changed

Nothing in the application. The Ruby source is identical. What changed is the target:

Target	Command	Database	Broadcasting
Browser	`juntos dev -d dexie`	IndexedDB	BroadcastChannel
Node.js	`juntos up -d sqlite`	SQLite	WebSocket server
BEAM	`juntos up -d sqlite_napi`	SQLite or PostgreSQL	OTP :pg

The browser uses the BroadcastChannel API for cross-tab sync. Node.js runs a WebSocket server. The BEAM uses Erlang's process groups — distributed by default, no external dependencies.

QuickBEAM

QuickBEAM is a JavaScript runtime for the Erlang VM, built on QuickJS-NG — a lightweight, standards-compliant JavaScript engine. Where Node.js embeds V8 in a C++ process, QuickBEAM embeds QuickJS in an Erlang NIF, giving JavaScript access to the BEAM's concurrency and fault-tolerance primitives.

Each QuickBEAM runtime is a lightweight isolate — Elixir can spin up a pool and dispatch requests round-robin across them, each running on its own OS thread. If one crashes, the OTP supervisor restarts it. The others keep serving. This is the same model Erlang uses for telecom switches — let it crash, recover instantly.

QuickJS isn't V8 — there's no JIT, so raw compute is slower. But for a web application that's mostly I/O (database queries, template rendering, HTTP responses), the difference is negligible. What you gain is a 5MB runtime instead of 45MB, sub-millisecond startup, and the entire OTP ecosystem.

The Architecture

The application runs inside QuickBEAM — a JavaScript runtime embedded in the Erlang VM. Elixir manages everything around it:

Browser (Turbo, Stimulus, Action Cable client)
    ↕ HTTP + WebSocket
Bandit (Elixir HTTP server)
    ↕ Plug router
QuickBEAM (JavaScript runtime pool)
    ↕ Beam.callSync
Elixir (:pg broadcasts, SQLite NIF, OTP supervision)

The JavaScript application handles request routing, controller logic, view rendering, and model operations — the same code that runs on Node.js. Elixir handles what it's best at: concurrency, fault tolerance, and distributed messaging.

Action Cable, Both Sides

The browser runs the real @hotwired/turbo-rails npm package — the same Action Cable client that Rails uses. The <turbo-cable-stream-source> custom element connects to /cable and speaks the Action Cable wire protocol.

On the server, Elixir implements the other side: WebSocket upgrade via Bandit, subscription management via :pg, and broadcast delivery in Action Cable's JSON format. When a model's broadcasts_to callback fires, it crosses from JavaScript to Elixir via Beam.callSync('__broadcast', channel, html), and Elixir fans it out to every subscriber.

Same protocol. Same custom elements. Same Turbo Stream HTML. The client has no idea it's talking to Elixir instead of Rails.

What the BEAM Adds

Things you get for free that would require significant infrastructure on Node.js:

Fault tolerance — a runtime crash restarts under OTP supervision, not pm2
Distributed pub/sub — :pg spans clustered BEAM nodes automatically. Add a node, broadcasts reach it. No Redis, no configuration.
True parallelism — pooled QuickBEAM runtimes on OS threads. Not single-threaded-with-cluster.
Hot upgrades — OTP releases support zero-downtime deployment

For production, swap SQLite for PostgreSQL — same app, juntos up -d postgrex. Database connections are pooled on the Elixir side via Postgrex, and combined with :pg for broadcasting, you get a fully distributed deployment with no external dependencies beyond Postgres.

A Path to Phoenix

This isn't just another deployment target. It's a migration path.

Your Rails app runs today inside QuickBEAM. The Elixir scaffold is a thin Plug/Bandit layer. But that layer could be Phoenix. At that point:

Rails controllers that need more performance? Rewrite as Phoenix controllers.
Models that need distributed state? Move to GenServers.
Views that need real-time interaction? Swap to LiveView.

One at a time. The rest keeps running in QuickBEAM. No big-bang rewrite.

No other migration path offers this. Going from Rails to Phoenix today means starting over. Juntos on BEAM gives you a running app on day one and an incremental path forward.

Try It

Prerequisites: Node.js (18+) and Elixir (1.18+). That's all you need.

npx github:ruby2js/juntos --demo blog
cd blog
npx juntos db:prepare
npx juntos up -d sqlite_napi

Same code. Same patterns. Different runtime. Source code. Documentation.

Juntos is open source: github.com/ruby2js/ruby2js

Our Journey: Building With Generative AI | Revelry

Mon, 30 Mar 2026 12:23:34 +0000

Businesses of all sizes and industries are eager to take advantage of generative artificial intelligence (AI), so I’m going to share some details on Revelry’s journey with this emerging technology over the past year. Even if you’re already working in the space, I believe you’ll find something interesting and helpful about our experience. We’ve got a lot of learnings to share, so this will be the first of a series of posts.

In this first post, I’ll cover our early exploration of generative AI – when we were moving as fast as possible to learn as much as we could. Subsequent posts will get deeper into our learnings around building more complex generative AI systems, in particular diving into how to incorporate Retrieval Augmented Generation (RAG) into software systems (and how you don’t need LangChain to do it).

Some Background

Here’s a little backstory on Revelry. Since 2013, we’ve been building custom software. Our bread and butter has primarily been web and mobile app development. In the early days, it was all about Rails and Node, but we’ve played around with PHP, .NET, Java, Python, and more.

We’ve always had a bit of a thing for new and emerging tech. Take React, for instance. Back in 2014, when JQuery and Angular were the big names, we were already building apps with React. And we didn’t stop there – we jumped into React Native pretty early, too. Our first app in React Native hit the AppStore when it was just at version 0.21, and now it’s up to 0.73. (By the way, when are we getting a major version update? Looking at you too, LiveView 😉).

We still work across a variety of tech stacks, but have collectively fallen in love with the elegance, performance, and strong community around Elixir and Phoenix, which we adopted as our preferred stack around 2018. We were building sophisticated Phoenix LiveView apps before there was even an official LiveView hex package published. (Yes, we were just referencing a commit hash in our mix.exs file — don’t judge.) We have done a lot in the blockchain space too, but I’m definitely not going into that in this article.

This is all to give you a glimpse into how we at Revelry dive into new technologies. We’re not shy about exploring the bleeding edge, and it’s really paid off. Our early dives into spaces like React and Elixir have made us the experts we are today.

Where We Started

Thinking back to June 2022, when OpenAI released GPT-3, it’s been nothing short of a rollercoaster ride. We at Revelry, like many software development companies, quickly caught on that this was a game-changer for our industry. Sure, we had a bunch of engineers who were into machine learning, but AI-driven apps weren’t really our main gig. Our partners didn’t ask for them much, and they didn’t seem all that necessary… until GPT-3 came along.

By Fall 2022, we were all in, diving deep into the world of these large language models (LLMs). The pace at which things have evolved since then is mind-blowing. Back then, things weren’t quite ready for the big stage, but it was obvious this was just the start.

We saw a golden opportunity to weave generative AI into our tried-and-true software and product delivery processes. This wasn’t about replacing our team, but turbocharging their productivity. Imagine our folks focusing on the creative, problem-solving aspects of product and software design, while AI handles the tedious stuff – like writing user stories, plotting out product roadmaps, or drafting sprint reports. And what if getting up to speed on a new project could be quicker and smoother? If this could work for us, it’d surely catch on elsewhere, right?

So, we rolled up our sleeves and jumped into the nitty-gritty. It started as a research and development adventure, filled with questions, like:

Just how far can the capabilities of these LLMs go?
What’s the engineering effort needed to integrate generative AI into our custom software?
What does it take to set up an AI-powered app in a live environment?
Can LLMs genuinely enhance our team’s productivity? If so, in what ways?
Is it possible to create something other engineering teams would want to use as well?

Experimentation

So, everyone at Revelry began dabbling with ChatGPT, mostly just for kicks. Some of us were crafting Eminem-style raps to add a bit of flair to our company-wide All Hands meetings (We’ve got a different Reveler hosting each week.). Meanwhile, our CEO, Gerard Ramos – or G, as we call him – was tinkering with how ChatGPT could enhance our product delivery process.

G found out pretty fast – with some clever prompting – that ChatGPT could whip up some solid product roadmaps and user stories, and even spin out working code examples based on those stories. This was more than just cool – it was promising. So, he proposed we start building tools around these use cases. And that’s how the idea for our first proof of concept came about: an AI-powered app to create user stories from just a few inputs. Sure, it wasn’t a game-changer yet, but it was a great starting point – allowing us to dip our toes in the water, while simultaneously boosting our productivity.

Our First AI- Powered Toy: StoryBot

Enter StoryBot. This little gem was a straightforward CLI tool that we ended up releasing as an open-source NPM package. It’s essentially a single JavaScript file, leveraging LangChain to tap into GPT-3 via OpenAI’s API (This was before GPT-4.). We threw in some tailored prompts, injected the user input, and voilà – it started spitting out decent user stories right in the command line.

We went a bit further with it after that, letting the user refine their story through chat, still all in the command line. The cherry on top was the ability to export the story as an issue in a GitHub Repo (At Revelry, we not only use GitHub to store our code, but also for issue tracking, project management, and more.). Ultimately, we ended up with this StoryBot iteration:

StoryBot Under the Hood

Diving into the StoryBot repo, you’ll see the core functionality is in one JavaScript file. This file uses LangChain.js for communicating with the OpenAI API, generating user stories from command line inputs. We could have opted for LangChain’s Python library, but the two have close to feature parity, and our team works more in JavaScript than Python. At this point, it was all still experimentation, so we opted for whatever we could move the fastest in.

Technically, for such a straightforward use case, direct API calls to OpenAI would have done the trick. However, LangChain offered ease of setup and capabilities beyond just interfacing with an LLM. It’s packed with features for creating AI-powered apps, like Retrieval, Agents, and Chains, though we didn’t dive deep into any of these for StoryBot.

LangChain simplifies building AI applications, but it’s not without its complexities and limitations. It abstracts a lot, sometimes obscuring the underlying mechanics, and is currently limited to Python and JavaScript ecosystems. There is now an Elixir implementation of LangChain, which is exciting because we’re huge Elixir fans, but it isn’t nearly as far along as its Python and JS counterparts. This Elixir library also wasn’t around yet at this point in our journey.

Looking a Bit More Into StoryBot Code

The first code that actually gets executed when you run npx gen.story generates the initial prompt:

const generateInitialPrompt = () => {
  const { featureText, techStackText, contextText } = parseArgs();

  return `Context: Act as a product manager at a software development company. Write a user story for the 'Feature' defined below. Explain in detailed steps how to implement this in a section called 'Implementation Notes' at the end of the story. Please make sure that the implementation notes are complete; do not leave any incomplete sentences. ${contextText}

  ${featureText}

  ${techStackText}

  User Story Spec:
    overview:
      "The goal is to convert your response into a GitHub Issue that a software engineer can use to implement the feature. Start your response with a 'Background' section, with a few sentences about why this feature is valuable to the application and why we want the user story written. Follow with one or more 'Scenarios' containing the relevant Acceptance Criteria (AC). Use markdown format, with subheaders (e.g. '##' ) for each section (i.e. '## Background', '## Scenario - [Scenario 1]', '## Implementation Notes').",
    scenarios:
    "detailed stories covering the core loop of the feature requested",
    style:
      "Use BDD / gherkin style to describe the user scenarios, prefacing each line of acceptance criteria (AC) with a markdown checkbox (e.g. '- [ ]').",
  }`;
};

...

const prompt = generateInitialPrompt()

You can see that the prompt has specific instructions to format the story in the way that Revelry prefers, which may be different than a lot of other teams. This said, if anyone wanted to use this with different prompts, they could easily fork it and change the prompts. The most important part here is that we are injecting user input into the prompt before we send it to the LLM. In this case, there are 3 potential user inputs:

The feature in question, which comes in as the 3rd command line argument (e.g. npx gen.story [feature]).
optional --stack flag to specify the tech stack that the user story will need to be implemented in.
optional --context flag to add some additional context around the feature you are writing a user story for.

Next, we take that hydrated prompt and send it to OpenAI using the tools provided by LangChain:

const model = new OpenAI({
  streaming: true,
  modelName: "gpt-3.5-turbo",
  callbacks: [
    {
      handleLLMNewToken(token) {
        process.stdout.write(token)
      },
    },
  ],
});
const memory = new BufferMemory()
const chain = new ConversationChain({llm: model, memory: memory})
const {response} = await chain.call({input: prompt})

A few things happen at this point: we are creating a new ConversationChain object, which is a wrapper around the LangChain Chain object that we’ll use to send the prompt to the LLM. We are also creating a BufferMemory object, which is a LangChain Memory object that we’ll use to store the results of the conversation.

Sidenote: If we were just using the OpenAI API directly instead of LangChain, it would be easy to pass the chat history alongside the prompt to the API call. (I’m just clarifying that LangChain isn’t necessary for this, even though it was very easy to set up.)

Because we set streaming: true when we initialized the OpenAI object, the chain.call method will return immediately, and the LLM will start sending responses to the callback we set up earlier. Since StoryBot is a CLI tool, we’re just outputting to process.stdout here. If you’re thinking about adapting this for a web app, you’d probably need to figure out how to send JSON responses or stream them to the client. We’ll get more into that later. The main takeaway? It doesn’t take much to start seeing some cool results by plugging user inputs into a well-crafted prompt template and sending it off to GPT-3.

So at this point, there is a response that is the generated user story, but also the entire user story has been streamed into the terminal and could easily be copy pasted to wherever. However, there is no ability to make any followup refinements yet. I’m not going to go line-for-line through the rest of the final result, but the long story short is that after we get the initial generated response back, we pass it to a function that creates a readline interface that allows us to prompt the user with questions in the terminal, and then we take the users’ response and send it back as another message to the LLM in the chat history. We also added the ability to export the final result to github if you had the github API token set.

That’s it, that’s StoryBot!

If you want to play around with it, you can install it via npm.

npm install -g storybot-ai

Fun? Absolutely. Somewhat useful? Sure. But let’s be real – it was an experiment. The thing is, not everyone wants to write user stories via the command line. Plus, every team has its own style for these stories. Our hardcoded prompts were great for us, but might not hit the mark for teams outside of Revelry, especially since we often work in staff augmentation, where teams have their own preferences.

Once we had a proof of concept, we started to see the potential. We were able to get a lot of mileage out of it, and it was a great way to get started with generative AI. This got a lot of ideas spinning about how we could get better user stories based on relevant context, which ultimately led us to the next part of our journey: diving deeper into RAG (Retrieval Augmented Generation) application development.

This is the first post in a series about Revelry’s journey exploring and developing custom software powered by generative AI. The next post will dive into our next experiment: building a chatbot to answer questions about Revelry based on our company playbook. Stay tuned!

Until then, here are a few other articles we’ve put out about AI:

Agile AI Artificial Intelligence Daniel Andrews Elixir Emerging Tech Innovation Lean Agile Machine Learning

Behind the Scenes: Replacing a Black-Box Billing Service

Fri, 27 Mar 2026 12:04:11 +0000

In a previous blog post, we looked at the process of migrating 600k users to a new billing service for Steady Media. This was the final step of an internal project called “Charger”. Its aim was to replace the off-the-shelf payment platform they had started with, Chargebee, which had become a roadblock. In this post, we will reveal the behind-the-scenes insights into the process that made that final step a success.

The Challenge

Steady Media needed to migrate 600,000 users from Chargebee to a new in-house billing system. The complexity was brutal:

Chargebeeʼs billing logic is opaque - we had to reverse-engineer behavior through observation
Historical data had to work in the new system (canʼt rewrite the past, especially not payment transactions!)
Multiple payment gateways (Braintree, GoCardless, PayPal) with different quirks
Complex billing workflows: trial subscriptions, gift subscriptions, mid-term subscription upgrades, subscription cancellations (with or without refunds)
Complex support workflows: issuing refunds, invoice & credit notes document versioning
Legal compliance (invoicing and taxes)
Migration had to happen without downtime or data loss

Steadyʼs development team asked us to lead the Charger project, so that they could focus on their core application. The tech stack had to be compatible with their team skills: Elixir & Phoenix!

🤩 Like what you see? have a look at the case studies on bitcrowd.net

The Journey

bitcrowd built Charger across three iterations:

2021: Core System

The first iteration aimed at building the data model and implementing the external communication layer of Charger:

API layer with authentication between Charger and the Media Makers app
Webhook notifications between Charger and the Media Makers app
Payment providers: integrated Braintree and GoCardless services (API & webhooks)
Multiplexed calls from the Media Makers app to hit Chargebee OR Charger depending on where the user lived

Once these various components were finalized, we could start the implementation of the core flows: create new subscriptions, bill users and trigger payments, upgrade subscription plans with prorated billing, settle payments, and so on.

2023: Feature Parity

After the first development cycle ended, new features were added to the Media Makers app that needed to be back-ported to Charger. We added new models and new flows to support trial and gift subscriptions. We took the opportunity to add Paypal as a third payment provider. Finally, we also tackled the generation and design of legal billing PDF documents.

2025: Admin UI, Support Tools, and Migration

A missing piece in Charger was a simple, lightweight, workable admin UI, that could be useful for Steadyʼs developers, QA engineers, support staff and financial experts. The goal was to minimize the implementation effort, while preserving a clean UX. We opted for DaisyUI since we knew the admin UI would use standard components like index tables, description lists, navigation elements etc.

Powerful search

One strong requirement was to make sure that the index pages had a powerful and performant search as well as sorting and filtering mechanisms, as one of the pain points of Chargebee had been searches that time out once enough data was in the system. We solved this issue by paying attention to the queries' efficiency. Additionally, for cross-table filtering, we implemented a search-typeahead dropdown in order to avoid expensive joins.

Activity logs

For such a billing product, it was crucial to allow admin users to see a clear changeset history on any resource. Charger comes with an auditing layer, that tracks changes and versions each relevant resource (like payments, invoices...). We built an abstraction on top of the auditing, that allows to plug any resource in it and view its version history with a UI resembling a Git diff.

Support staff tooling

At that stage, we had a working product for developers, QA engineers and financial experts. But Steadyʼs support team still needed tailored features to enable them to solve specific situations. For example, when a userʼs credit card expired, and the system could not withdraw money from their account, the user might send the missing amount via a bank transfer. On Steadyʼs bank account, the balance is correct, but Charger does not know about this transaction, and the accounting balance is affected. Similarly, the support team might refund a user via a manual bank transfer. To keep the accounting in check, we implemented a solution to record offline refund & payment, as well as various manual actions & flows to fix any situation that would not auto-heal.

💡 Have we got you interested? If you have a question or topic you would like to discuss with us, we would like to hear from you.Book a free call with us

Outcome

We migrated all 600,000 users to Charger. The lights stayed green. Steady now controls their billing infrastructure and its support team can handle edge cases and requests through the admin UI. The developer team can also leverage the activity logs and powerful search capabilities to debug or resolve errors.

bitcrowd led on self-contained packages: we took topics Steadyʼs team didnʼt have capacity for and delivered them finished. We were able to coordinate with stakeholders (support team, finance, legal) to turn “we need X” into actionable tickets. We applied our standards of excellent documentation and test coverage, high code quality and review, which ensured a smooth handover with Steadyʼs team.

Migrating 600k users to a new billing service

Fri, 27 Mar 2026 12:04:11 +0000

In August 2025, we helped our friends at Steady to migrate their 600,000+ users to their own in-house billing system. This was the final step of an internal project called “Charger”. Its aim was to replace the off-the-shelf payment platform they had started with, Chargebee, which had become a roadblock. The work on Charger began already in 2021 in workshops with the Steady team. Together with them, we designed, planned and finally built the internal product that would take over for years later. The process and the implementation happened in multiple iterations over a span of four years.

This blog post explores the strategy that was used to achieve this challenging task.

The backstory

Steady supports Media Makers in building an audience and driving revenue via subscriptions to newsletters, blogs etc. While most of the tools and features for Media Makers live in their main application, they outsourced the subscriptions & billing management to a third-party service (Chargebee). The Media Makers app was very tightly coupled to Chargebee as it relied on it for core business logic within the app, as well as for accounting.

Introducing: Charger, the fraternal twin

Although Chargebee was useful initially, it became increasingly painful to work around their standard processes over time. This ultimately led to the development of a custom billing tool.

For this we had to reverse-engineer all of what Chargebee was doing. We started by building the required schemas, and exposing the API endpoints and webhook notifications that were needed by the Media Makers app. Any changes in the Media Makers app usage of the API/Webhooks, or to Chargebee itself, needed to be propagated to Charger. Both billing systems had to behave identical from the outside.

The Media Makers app now needed to forward calls to Charger and/or Chargebee. A “multiplexer“ abstraction close to the API & Webhook layer was added, so that for developers & users of the app, the fact that Charger or Chargebee was used for the calls would be transparent.

Charger needed to process payments and refunds, with all payment providers supported by the Media Makers app. The first flow of the application was created: subscribe to a publication, generate an invoice, create a payment, and settle that payment 🎉!

🔍 Would you like more detail? We have a 'behind the scenes' blog post about Steady! Have a Look!

A bird watching experience 🔭

At this stage, no real users were migrated to Charger. No real users were even created on Charger! The system just worked™... Subscription statuses, invoices, payments processing being crucial to the system meant that we had to come up with a transition plan, that allowed the developers to observe and watch the behaviour of the system for just a handful of users. If any problem arose, it would need to affect only such a small amount of users that their support team could resolve it manually. We handpicked 10 users with rather simple data: a single recent subscription, no cancellations, and various payment providers / currencies to cover as much ground as possible.

Using our beloved Oban for job processing, we built a migration job, that would, for a given user:

Fetch all their data from Chargebee
Format it to please Chargerʼs data model
Send it to a dedicated endpoint in Charger
Charger creates all the given data for the user (subscriptions, invoices, payments, etc.)
Cancel the user subscriptions on Chargebee (so that they donʼt get billed twice!)

Of course, all of this must happen in a transaction on both systems, fail gracefully and report meaningful errors to the devs. We leveraged Oban configuration to optimise the parallelisation and retry mechanism of the jobs, as we could be rate-limited by Chargebee. At some point, we would want to migrate users in batches of 100-10k users at a time, so the migration mechanism had to be robust and auto-healing.

💡 Have we got you interested? If you have a question or topic you would like to discuss with us, we would like to hear from you.Book a free call with us

Silent bugs

A main challenge in this process was to validate the integrity of the migrated data. The Media Makers app and Charger run on different databases, and have their own representation of what is an invoice, a subscription, a user, etc. Forgetting to migrate a field, or mapping the wrong timestamp from one system to the other, would happen silently because a part of the migration code lives in the Media Makers app, and the other part lives in Charger. The data could be silently missing or corrupted, and we would only realise down the road that all users were migrated with errors. Luckily, we had the chance of having access to a Chargebee staging instance with users in all kinds of configuration. This helped us a lot in finding migration issues by comparing the user data on both systems before and after the migration, and strengthening our tests to cover all the edge cases we could find. The developers at Steady also added ERPC, which enabled us to run integration tests evaluating the effects of Charger on the Media Makers app.

Conclusion

Once we were confident that our 10 users were rolling, we migrated 10 more users with more complex data. And so on, for weeks, increasing the batch size, and deciding to create new users on Charger with a rand() choice. This allowed Steady to steer the future of their billing strategy for their users, integrating with other payment providers and simplifying some of the user flows for refunds etc. While it looks simple on paper, this project required a lot of patience, collaboration and planning ahead as the stakes were really high and we had to build up confidence in the new system and in the migration approach.

Building a blog with Elixir and Phoenix | jola.dev

Thu, 26 Mar 2026 19:19:26 +0000

TL;DR: it’s an Elixir app using Phoenix server side rendered pages, with the blog post pages generated from Markdown using NimblePublisher. It’s running on a self-hosted Dokploy instance running on Hetzner, with bunny.net as a CDN sitting in front of it.

This is a very belated write up of how this blog was put together! There’s nothing terribly original here, but I figure it could come in handy for someone out there as a reference. And the world needs more Elixir content.

Why Phoenix

I have used static site generators before to power my blog (shoutout to Hakyll), but I wanted to open the door for myself to also have little experiments on this site, ones that would require more interactivity than a static site allows. Besides, I just like using Phoenix. Although most of my Phoenix projects use LiveView, this felt like a good place to do things old-school with DeadViews.

It also means I get full control of what I’m building. Using a tool someone else created means getting a lot for free, but the moment you step outside of the expected you’re having to figure out how to make things work for their tool.

So I kept things simple. No Ecto, no DB. Just server-side rendered HTML. It’s blazingly fast, as you can see from this PageSpeed Insights report.

NimblePublisher

My setup closely matches the original Dashbit blog post Welcome to our blog: how it was made!, which led to the creation of NimblePublisher.

The heart of the blog is the NimblePublisher setup, which consists of a use block:

defmodule JolaDev.Blog do

  use NimblePublisher,
    build: JolaDev.Blog.Post,
    from: Application.app_dir(:jola_dev, "priv/posts/**/*.md"),
    as: :posts,
    html_converter: JolaDev.Blog.MarkdownConverter,
    highlighters: [:makeup_elixir]
...

This will load up all the posts, parse the frontmatter, run it through the markdown converter, and compile it into module attributes. This means there’s no work left to be done at runtime, it’s all pre-compiled.

Posts are organized by year: priv/posts/2025/08-18-ruthless-prioritization.md . We get beautiful code block syntax highlighting through Makeup. The Blog module also defines a set of helpers for fetching the posts:

@posts Enum.sort_by(@posts, & &1.date, {:desc, Date})

# Let's also get all tags
@tags @posts
      |> Enum.flat_map(& &1.tags)
      |> Enum.uniq()
      |> Enum.sort()

# And finally export them
def all_posts, do: @posts
def all_tags, do: @tags

def posts_by_tag(tag) do
  Enum.filter(all_posts(), fn post -> tag in post.tags end)
end

def find_by_id(id) do
  Enum.find(all_posts(), fn post -> post.id == id end)
end

The only thing that took a bit of figuring out for me was getting Tailwind classes into the outputted HTML. I’m pretty sure I’ve seen better approaches shared since I wrote this, but this works too. Under earmark_options, pass:

Earmark.Options.make_options!(
  registered_processors: [
    Earmark.TagSpecificProcessors.new([
      {"a", &Earmark.AstTools.merge_atts_in_node(&1, class: "underline")},
      {"h1", &Earmark.AstTools.merge_atts_in_node(&1, class: "text-3xl py-4")},
      {"h2", &Earmark.AstTools.merge_atts_in_node(&1, class: "text-2xl py-4")},
      {"h3", &Earmark.AstTools.merge_atts_in_node(&1, class: "text-xl py-4")},
      {"p", &Earmark.AstTools.merge_atts_in_node(&1, class: "text-md pb-4")},
      {"code", &Earmark.AstTools.merge_atts_in_node(&1, class: "")},
      {"pre",
       &Earmark.AstTools.merge_atts_in_node(&1,
         class: "mb-4 p-1 py-4 overflow-x-scroll border-y"
       )},
      {"ol", &Earmark.AstTools.merge_atts_in_node(&1, class: "list-decimal")},
      {"ul", &Earmark.AstTools.merge_atts_in_node(&1, class: "list-disc pb-4")},
      {"blockquote",
       &Earmark.AstTools.merge_atts_in_node(&1,
         class: "pl-4 border-l-2 mb-4 border-purple-700"
       )}
    ])
  ]
)

You probably have your own preferences for how to set up your classes, but this gives you a pattern you can use to ensure that the tags that come out have the appropriate classes.

The Frontend

As mentioned this is all server-side rendered Phoenix templates. It’s using standard Tailwind CSS. It predates DaisyUI and I don’t think there’s a strong reason for me to make the lift of getting it in, although I wouldn’t have minded it being a part of the scaffolding back when I set up the blog!

The only JS snippets in here are a mobile menu toggle and the Phoenix topbar. Apart from the Tailwind library, the custom CSS in here is pretty minimal. You get a lot out of the box with a Phoenix project.

And of course, dark mode. I know it’s not everyone’s cup of tea, but it is my website after all.

CI

I’ve got Github Actions set up to run on every push and PR, just the basic Elixir quality assurance tools.

mix compile --warnings-as-errors
mix format --check-formatted
mix credo --strict
mix test

And then I’ve got Dependabot set up as well. I’ve been hearing and thinking a lot about how it creates a lot of noise, but I feel like that’s less of an issue in the Elixir community. Packages tend to not have a lot of dependencies, and so you don’t get the same waves of bumps going out that npm does. And merging them is satisfying.

Deployment

On the hosting side things get a bit more spicy. The repo includes a multi-stage Docker file, roughly based on the Phoenix recommended example file. This means that most of the dependencies are only pulled in at build time, and the image you get out on the other side is a bit smaller. I’m using Elixir 1.18.4, Erlang 28.0.2, and Debian trixie-20250721-slim at the time of writing this, but that’s likely to change. There’s something very satisfying about bumping dependencies.

And now we’re arriving at Dokploy, an open source platform as a service (PaaS) for running apps, basically a self-hosted Heroku. It does everything, automatic builds and deploys from Github updates, built-in Docker Swarm, networking, orchestration of replicas across the cluster, rolling deploys, rollbacks, preview builds, and much more.

So my publish flow is basically: create a PR and wait for CI to finish (I could skip this but it’s nice to know I didn’t mess something up). When I merge the PR Dokploy automatically picks that up and triggers a checkout and build of the repo. Once that finishes, it starts a rolling deploy to replace the running replicas. And we’re live. With cached layers on the server, deploys can finish in 30s, zero effort.

I run this Dokploy instance on Hetzner and my experience has been really positive. The pricing is unbeatable, even with the recent increase, and it’s been rock solid for me. Really, with the Dokploy instance, there’s nothing stopping me from packing up and going somewhere else. Having that kind of freedom is very nice. But I’m more than happy to stick with Hetzner.

The Little Things

I’ve set up a few little conveniences for my app so I’ll share some example code for them here.

RSS

RSS is managed by a plain Phoenix controller that looks something like this:

defmodule JolaDevWeb.RssXML do
  use JolaDevWeb, :html

  embed_templates "rss_xml/*"

  def format_rfc822(%Date{} = date) do
    date
    |> DateTime.new!(~T[00:00:00], "Etc/UTC")
    |> format_rfc822()
  end

  def format_rfc822(%DateTime{} = datetime) do
    Calendar.strftime(datetime, "%a, %d %b %Y %H:%M:%S +0000")
  end
end

and the corresponding XML:

<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:content="http://purl.org/rss/1.0/modules/content/">
  <channel>
    <title>jola.dev</title>
    <link><%= url(~p"/") %></link>
    <description>Blog posts from jola.dev</description>
    <language>en-us</language>
    <lastBuildDate><%= JolaDevWeb.RssXML.format_rfc822(DateTime.utc_now()) %></lastBuildDate>
    <atom:link href="<%= url(~p"/rss.xml") %>" rel="self" type="application/rss+xml" />

    <%= for post <- @posts do %>
    <item>
      <title><%= post.title %></title>
      <link><%= url(~p"/posts/#{post.id}") %></link>
      <description><![CDATA[<%= post.description %>]]></description>
      <content:encoded><![CDATA[<%= post.body %>]]></content:encoded>
      <pubDate><%= JolaDevWeb.RssXML.format_rfc822(post.date) %></pubDate>
      <guid isPermaLink="true"><%= url(~p"/posts/#{post.id}") %></guid>
      <author><%= post.author %></author>
    </item>
    <% end %>
  </channel>
</rss>

Sitemap

I was a bit surprised not to find a clean little library for generating the sitemap (this may have changed since I wrote the code!), but I guess the implementation is just going to heavily depend on your setup. Anyway, just sharing this for reference.

defmodule JolaDevWeb.SitemapController do
  use JolaDevWeb, :controller

  def index(conn, _params) do
    sitemap = JolaDev.Sitemap.generate()

    conn
    |> put_resp_content_type("text/xml")
    |> send_resp(200, sitemap)
  end
end

defmodule JolaDev.Sitemap do
  alias JolaDev.Blog

  @host "https://jola.dev"

  def generate do
    """
    <?xml version="1.0" encoding="UTF-8"?>
    <urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
    #{generate_static_pages()}#{generate_tag_pages()}#{generate_blog_posts()}
    </urlset>
    """
  end

  defp generate_static_pages do
    pages = [
      %{loc: @host, changefreq: "monthly", priority: "1.0"},
      %{loc: "#{@host}/about", changefreq: "monthly", priority: "0.8"},
      %{loc: "#{@host}/projects", changefreq: "weekly", priority: "0.9"},
      %{loc: "#{@host}/talks", changefreq: "monthly", priority: "0.7"},
      %{loc: "#{@host}/posts", changefreq: "weekly", priority: "0.9"}
    ]

    Enum.map_join(pages, "\n", &url_entry/1)
  end

  defp generate_tag_pages do
    Blog.all_tags()
    |> Enum.map(fn tag ->
      %{loc: "#{@host}/posts/tag/#{tag}", changefreq: "weekly", priority: "0.6"}
    end)
    |> Enum.map_join("\n", &url_entry/1)
  end

  defp generate_blog_posts do
    Blog.all_posts()
    |> Enum.map(fn post ->
      %{
        loc: "#{@host}/posts/#{post.id}",
        lastmod: Date.to_iso8601(post.date),
        changefreq: "monthly",
        priority: "0.8"
      }
    end)
    |> Enum.map_join("\n", &url_entry/1)
  end

  defp url_entry(params) do
    """
      <url>
        <loc>#{params.loc}</loc>
        #{if params[:lastmod], do: "<lastmod>#{params.lastmod}</lastmod>", else: ""}
        <changefreq>#{params.changefreq}</changefreq>
        <priority>#{params.priority}</priority>
      </url>
    """
  end
end

Blog redirect plug

When I first moved over to this new app I wanted to ensure that I kept my old blog post links alive, so I set up this little plug to rewrite requests to match the new layout.

defmodule JolaDevWeb.Plugs.BlogRedirect do
  import Plug.Conn

  def init(_), do: []

  def call(conn, _opts) do
    if conn.host == "blog.jola.dev" do
      ids = JolaDev.Blog.ids()
      path = strip_path(conn.request_path)

      path =
        if path in ids do
          "posts/" <> path
        else
          path
        end

      conn
      |> put_resp_header("location", "https://jola.dev/" <> path)
      |> send_resp(:moved_permanently, "")
      |> halt()
    else
      conn
    end
  end

  defp strip_path("/" <> path), do: path
  defp strip_path(path), do: path
end

SEO

I went a bit further on this one. Each page has its own meta description, Open Graph tags, and Twitter Card tags — all driven by assigns passed from the controllers. Blog posts automatically get og:type="article" with article:published_time and article:tag set from the post metadata. The layout just reads from conn.assigns with sensible fallbacks, so adding SEO to a new page is just a matter of passing the right assigns. Here’s what the blog-post-specific bits look like in the layout:

<meta property="og:type" content={if(@conn.assigns[:post], do: "article", else: "website")} />
<%= if post = @conn.assigns[:post] do %>
  <meta property="article:published_time" content={Date.to_iso8601(post.date)} />
  <meta property="article:author" content="https://jola.dev/about" />
  <%= for tag <- post.tags do %>
    <meta property="article:tag" content={tag} />
  <% end %>
<% end %>

Same idea for the Twitter Card and description tags — one place in the layout, driven entirely by what the controller passes in.

I also added llms.txt and llms-full.txt endpoints, this is a newer standard that helps AI systems understand your site. It follows the same pattern as the sitemap: a module that generates the content from Blog.all_posts(), and a controller that serves it as plain text. Whether it actually matters yet, who knows, but it was trivial to add and I figure it can’t hurt.

Wrapping Up

This app is intentionally kept simple but powerful. Everything is set up the way I want it and I have a zero effort and very fast pipeline for publishing new posts. If you’re an Elixir dev thinking about a personal site, consider just using Phoenix. Combined with NimblePublisher you’ve got a really powerful and blazing fast blog framework right there.

And while you’re at it, why not host it on Hetzner! If you use the referral link to sign up you get €20 and I get €10. If you prefer not to use the referral link, here’s a plain link: https://www.hetzner.com/cloud/. Also consider joining me in sponsoring Dokploy.

Source code is available at: https://github.com/joladev/jola.dev. Next up I’ll talk about setting up bunny.net and a separate post on Dokploy on Hetzner.

Software Development in 2026

Wed, 25 Mar 2026 16:08:30 +0000

Three years ago I wrote a fairly coherent piece on four key developer skills, but a good deal of water has flowed under the bridge since then, and while the theses laid out there remain sound, they need a slight adjustment in light of the boon that has descended upon us in the form of large language models.

I went through all five stages of the inevitable over the course of a year.

① Denial—I watched my colleagues a year ago raving about autocomplete and hallucinations, and even won a few bets—not unlike the one the Adriano Celentano character proposes to his accountant in that great film.

② Anger—I kept writing code by hand, but part of my job involves cleaning up colleagues’ messes after shifted concepts (I do a lot of reviews), and the volume of neural slop had crossed every conceivable boundary: even when the code compiled, it looked like Rome from a bird’s-eye view—slovenly fragments of good practices scattered here and there, interspersed with the slums of deeply nested conditionals. I found myself literally rewriting large chunks after the little models, because I take code review seriously and still see it as a tool for teaching apprentices.

③ Bargaining—about seven months ago I tried unleashing a model on an old library of mine that was desperately in need of proper documentation; to my surprise, the documentation turned out coherent, nearly complete, and unquestionably better than nothing. I screwed my eyes shut and asked for tests. Half of them tested the standard library and implementation details—but the other half brought genuine value. Like those gruff stubborn men from the joke, I said: “Welllll, damn.” And paid for Warp.

④ Depression—in the first month of use I finished two personal projects that had been gathering dust for years, equipped all my libraries with detailed documentation and the missing tests, played around with creating my own programming language, and even trusted the model to fully solve a take-home assignment for one of the companies that had written to me directly with an offer (the assignment sailed through, but I never would have taken the job anyway—their HR had violated every conceivable rule of decency in headhunting). I wasn’t afraid, of course, that I’d be thrown out and Claude hired in my place—the models won’t reach my level of expertise before I retire—but writing code is one of my most beloved occupations in life, and I felt it being taken away from me.

⑤ Acceptance—it is now the end of March 2026, and I can say with confidence that language models provide me with substantial help in development, without particularly intruding on the part of my life I cherish: they write excellent documentation, reasonably coherent tests, and—under supervision and with Ctrl+C at the ready—can shuffle JSON around. Complex code I still write myself, and I’m confident this will continue until my death at the wheel of a sports motorcycle.

At the same time, the internet is proliferating with success stories from every variety of blowhard—from startup founders with three grades of parochial school, armed with enthusiasm, a narrow worldview, and the free tier of ChatGPT—to bedroom traders whose imaginary profits already permit them to purchase a paper yacht in Bali. In the hands of people far removed from software development, large language models are doomed to two low-yield applications: you can amuse yourself generating memes of piglets drinking mojitos in the Kremlin, or you can reproduce a product that already exists on the market and that in its niche you’ll never catch up to.

To use models effectively (you can now also deploy agents, but this changes nothing whatsoever about the substance), you need—at minimum—to understand the principles by which they operate. It took the automobile industry a hundred years to give users the joy of driving a vehicle without ever having opened the bonnet. Aircraft haven’t reached that stage yet. I see no reason to suppose that very advanced autocomplete is capable of repairing itself. And this is besides the fact that language models are, in principle, a dead end in the development of artificial intelligence. Even Yann LeCun has understood this, though unfortunately it remains unclear whether his own ideas aren’t yet another dead end.

Yeah.

If you need to knock together from scratch a shop for the worthless trinkets your wife makes—a modern model will handle it with flying colours. Clean unpretentious design, convenient addition of new bracelets, photos, payment system. The little model will knock you up a website, a mobile app, and lord knows what else. And it will work first try, most likely—because in training it has stared at such dreck in billions of different variations. What everyone finds so astonishing is, in essence, the output of four shell commands: cp, grep, sed, awk. Last century this technology bore the proud name “snippet.” For cloning already-existing things, then, the model is a perfectly fine assistant. Web studios that build landing pages will probably indeed die (they should never have existed in principle, but that’s a different question).

As for more complex projects—not necessarily more complex per se, but more unusual—without an architect to guide it, the model will disgrace itself at the very first hurdle. Because a vaguely described result can be achieved in fifteen different ways, and sooner or later the rough-and-ready decisions of a spring balance (a steelyard, not a precision scale) will lead into a swamp from which there’s no escape—because admitting defeat is not our way, and the model will play roulette to the bitter end, like Dostoevsky in Baden-Baden.

I recently wrote about why elaborate prompts and detailed descriptions cannot produce a good result—see part two of Artificial ‘Intelligence’—I won’t repeat myself here, but will quote the key thesis:

A person capable of breaking a large task down into the minimum number of smaller ones that satisfy the condition of “unambiguous solution” is capable, in today’s reality, of writing a reasonably complex application in a couple of days.

What I mean is that the ability to distinguish tasks with branching logic from multi-step syllogisms is more in demand than ever. Draw a mental flowchart of the execution—and if there are decision diamonds in it, you need to work them out for the model explicitly (go left here, go right here, snow on head here, very painful). Better still—break the main task into several, to eliminate those “conditions/decisions” entirely. But this is hardly possible if you have no idea how to write such code yourself.

Besides the fools who tried to build a website and succeeded—there are also saboteurs. They are considerably more dangerous in that they give the unprepared reader the impression of being “people in the know.” Did I mention that before paying for a subscription to a cloud model I thoroughly understood exactly how they work? I always do. If I get it into my head to drag library xyz into a project, I start by digging into its source code. When choosing a technology I don’t read adoption success stories and watch benchmarks—I write my own “trinket shop in the Bronx” with it. To use language models in daily work, and actually pay for the privilege, Altman’s feverish ravings and Karpathy’s coquettish napkin-code are not enough for me. I need to understand how it functions under the hood, so as to preemptively avoid disappointment and the collapse of hopes.

In short, those who have learned by instruction to run models and ask them questions pulled from thin air—supposedly testing or even proving something—would frankly be better off keeping quiet. Because everything that has crossed my field of vision resembles hallucinations from the first version of ChatGPT, Joyce’s Ulysses, Castaneda’s astral flights, and in principle any address at a party congress by the secretary of the Upper Walrusville cell, having gorged himself on fly agaric.

Let me skim glissando—or, as Dovlatov used to say, in dotted lines—across the main points.

Tests and Benchmarks

Comparing different models based on sets of cardboard tests and plastic benchmarks is pure, undiluted charlatanism. It’s enough to look at the language summary table at AutoCodeBench → Experimental Results (yes, this is sarcasm). Claude Opus 4 fails to hit 50% for TypeScript but clears 80% for Elixir. Translating from accountant-speak into plain language (and exaggerating slightly, of course)—if your project is in TypeScript, the model is more of a hindrance; if it’s in Elixir, you can hand it a 1,000-line refactor.

Context

RAG in any moderately complex project (or more precisely, its “RA” part) matters hundreds of times more than the model itself. Why do all these Claudes burrow into your computers with their IDEs (and lately—CLIs), do you think?—It’s simple: shipping the entire context to the server every time is expensive and inefficient (and despite the flagship advertising promises, nobody can actually process more than a hundred thousand tokens without noticeable quality loss). So every model sends some kind of “distillate” to the server.

It’s important to understand that any language model operates as a finite state machine: start → context → query → response → stop. There are no “sessions.” You cannot “launch” a model and converse with it—every request you make starts from a blank slate. Models as such have no “memory”—which is why preserving context is critically important. But unlike Hoffman’s character from Rain Man, even we cannot memorize six decks of cards—let alone models, with their context window as narrow as the Strait of Hormuz. My slowly-simmering project Ragex is an attempt to somehow formalize—and minimize without loss of generality—the context required for processing sizeable codebases. We’ll see whether I manage it—but the fact that I appear to be alone in this on the visible horizon is not exactly inspiring.

Plans and Reasoning

When tackling any reasonably non-trivial task, you need to force the model to sketch a plan and ask all the questions that arose for it while creating that plan. It will gladly surrender to you all the internal decision branches from the flowchart. These questions must be answered as clearly as possible—in blunt, clipped phrases that admit no double interpretation. The requirement to supply each phase of the plan with tests and bring all project documentation fully in line with the current state of the codebase must be baked into the general rules.

Each stage calls for a git diff with an informal review. I also always use “thinking” models, and watch those reasoning traces in real time—so that the moment it tries to veer off course (and on non-trivial tasks it will always try)—I can kill the little pest with Ctrl+C and explain why it’s wrong.

Language

In my experience, it’s best to use the language in which an answer is easiest to find on the internet. For code—always English; for obscure details of Lorca’s biography—Spanish; for a survey of the sexual services market in Berlin—German.

I have no hard evidence, unfortunately—only a general understanding of how this T9 on steroids operates—but the empirical record is copious. My Spanish hovers somewhere around B2–C1 level, so I still try English first; if the result looks underwhelming, I strain my fingers in Spanish—and in the vast majority of cases, I don’t regret it.

Politeness

After each successful task completion I say something like “Awesome,” or “Astonishing,” or “Stunning,” or words to that effect. I phrase every request as a request and always add “please.” My behaviour affects carbon emissions about as much as it affects democratic elections in a country of fifty million—but it makes things simpler and more pleasant for me. Besides, as Niels Bohr once said: “Of course I am not superstitious. But they say a horseshoe brings luck even to those who don’t believe in such nonsense.”

Horizontal Scaling

Wed, 25 Mar 2026 16:08:30 +0000

Part four of four key developer skills.

Ability to immediately build a horizontally scalable solution without adding any special code for it in the first version.

This turned out to be the hardest piece in the series, because literally a handful of developers actually understand what “horizontal scaling” is. Shown above is a screenshot of an amusing tweet by Tobi Lütke, which demonstrates either his supreme professionalism in sophistry, or his complete technical incompetence.

Serving ten billion clients does not mean your service is scalable. Buses carry people along Nevsky Prospect, and buses carry people along the Garden Ring; anyone who has completed three grades of parochial logistics school will tell you that speaking of “reliable connections between Leningrad and Moscow” on the basis of this incontrovertible fact is premature.

Even within a single city, the situation can easily spiral out of control. Scaling a bus fleet to match the expanding footprint of a metropolis is not about buying and deploying more buses. I lived in Berlin twenty years ago and could not stop marveling at how well the transport was organized. Judging by today’s passenger reviews, as the city spread outward—BVG did not cope.

I invite the thoughtful reader to pause here and think: so what exactly went wrong with the service’s scaling? For the superficial blockheads who can’t be bothered to think through what they’re reading—the answer is: logistics.

Deploying more buses onto the streets of a multi-million-person city doesn’t require much brainpower. But each individual passenger needs exactly one bus: the one that arrives immediately after they step off the previous one. Not an hour later, not a minute too early—within a three-to-five minute transfer window. As long as the bus schedule is built around minimized (and guaranteed) transfer times—we can speak of scaling. If bus A dumps its passengers at the terminal thirty seconds after connecting bus B has already pulled away—scaling has failed.

All right, to hell with Berlin—even with good public transport, living there was impossible. Let’s get back to Tobi.

Each Shopify user requires one persistent connection (a WebSocket) to one server. Adding new users is simply adding servers to the rack. If there are any scaling problems—they’re all clustered around the database, not Rails itself. Rails doesn’t care at all how many total servers are serving users: each server only needs to handle its own load.

This is not scaling. This is adding isolated capacity. It’s like if the zoom on your phone’s camera did not actually zoom in without unacceptable quality loss, but simply tiled more tiny copies of the image.

What the computing industry calls scaling is adding new hardware resources that increase the capacity of a connected node. Deploying more buses on new streets—no. Changing the schedule so that new buses coordinate with the existing ones—yes.

A chess server, for example, requires no scaling whatsoever. The server overheating?—Put another one next to it; in the end we’re serving disconnected pairs of players anyway. A server for the simultaneous slaughter of a billion sweaty nerds, on the other hand—that one needs scaling.

Let me briefly mention one example from personal practice before getting to the point. We process an incoming stream of currency exchange rates, perform some mathematical manipulations on them, and spit out peculiar results. Two hundred-plus currencies—that’s forty thousand pairs—with values arriving on average roughly once a second (in practice, more often). The mathematical manipulations can involve any number of currencies. All of this happens in real time. Because values must be available at any given moment, we cannot simply partition the pairs and split the streams across multiple servers. And a single server physically cannot handle it. Which means every node in the cluster must be able to exchange information with every other node—and the information is not static, so Redis won’t do. This is where the architecture needs the ability to scale horizontally.

The preamble has dragged on a bit. I hope I’ve at least clarified the terminology somewhat. So: if in the course of discussing an architecture you’ve concluded that the project will need genuine horizontal scaling—you cannot do without finite state machines. (In general it’s better to build all business logic on FSMs, but in a standalone system you can hobble along without them—in a cluster, there’s no way.) I rarely recommend reference material, but Introduction to the Theory of Computation by Michael Sipser is well worth a look. A finite state machine—for all its apparent simplicity—is something so powerful it genuinely astonishes.

Unfortunately, many believe that an FSM is simply a set of states—an extremely dangerous misconception that completely negates the entire body of formal mathematics upon which the power of finite state machines rests.

In any case, if you want to be ready to scale horizontally—build critical processes on finite state machines and make them fully asynchronous. If subsystem A must interact with subsystem B—forget about direct calls. In HTTP terms—201 is good, 200 is dreadful. Under no circumstances will you ever be able to later convert a request→response call sequence into request→acknowledgement→await response (without the destructive force of a complete refactor).

Asynchronous interactions built on top of FSMs, on the other hand, will make future scaling painless—because in this paradigm it makes absolutely no difference on which node the code handling a request actually runs.

There are languages in which this is easier (Elixir, Erlang), and those in which it’s harder. But in principle it is achievable in any environment. Once you get the hang of it, writing asynchronous code becomes no harder than synchronous. I speak from experience.

And finally, the perfect litmus test for determining whether your system is truly distributed or just a few servers standing in corners. If you’ve ever had to decide which letter from the CAP theorem set—‘C,’ ‘A,’ ‘P’—to sacrifice, you most likely have a genuine scalable cluster. If not—forget about horizontal scaling and simply add capacity as the business blooms.

And while I have the opportunity, I can’t resist recommending my own library Finitomata, which I prototyped in Idris and which is designed to be completely asynchronous (there is no way to determine from a response whether a state transition succeeded)—it provably prevents the programmer from violating a single law of finite state machine management.

YAGNIN, but YAGNIL

Wed, 25 Mar 2026 16:08:30 +0000

Part three of four key developer skills.

Ability to design software such that the first version contains not a single line “in anticipation of future changes,” while future changes don’t touch existing code in any way.

One of the most repulsive “principles” of development to emerge in the last decade can safely be identified as YAGNI. Despite growing out of Donald Knuth’s coquettish but not entirely thoughtless “premature optimization is the root of all evil”—stated nearly 60 years ago—the modern interpretation hands idlers, under-qualified practitioners, and outright scoundrels enormous latitude to dodge solving problems correctly.

The correct formulation is right there in the title of this post: “You aren’t gonna need it now, but you are gonna need it later.” If that were not so—if “architecting for future requirements / applications turns out net-positive” (in John Carmack’s words) really were needed only rarely—we would never encounter the reaction “this code is easier to rewrite from scratch than to change.” The situation “we’ll need to refactor three modules deep” would never arise when adding a new parameter to a function call. People would never break backward compatibility—since Ron’s quote yanked from context (available at the link above) promises us simple addition of new things as needed.

Unfortunately, none of the above works. At least not out of the box, in the form in which the programming extremists try to sell it to us.

On the other hand, Knuth, Jeffries, and Carmack are respectable people who have each written heaps of code tested by decades—and most likely wouldn’t talk complete nonsense.

As always, it’s all in the nuances. Immediately implementing every potential feature that might someday pop into the head of a deranged client is obviously a thankless task. You shouldn’t do that, and you can’t guess all those potential ideas right now anyway. But you must be prepared for them.

What does that mean?—Let me try to explain with an example.

Suppose the task at hand is writing a Mastodon text publisher. A command-line utility, literally. Here—a text file, there—a toot. Called with something like $ post file.txt.

The first thing to settle: will anyone else be using this? Because if we’re talking about a single-user program that runs once every five years on my machine—you don’t even need to handle errors. It crashes with an exception?—I’ll tweak it and restart, no big deal.

But if we’re talking about a project for actual people—whether a paying client or open-source freebie-hunters—things become somewhat more complex. And it’s not even about bugs: loyal users will forgive bugs, and they’re easy enough to fix. The point is that if the project is going to grow, you need to prepare for that a little. Because rewriting everything from scratch with every new requirement is fun, but inefficient.

So where might it grow?

a new publishing service gets added;
a new text format gets added;
batch processing gets added;
something else, probably—but that’ll do for now.

It’s time to utter the two magic words: dependency injection. Everywhere you expect requirements to scale outward, you can give yourself insurance without writing a single superfluous line. Instead of hardcoding it like this:

@specpublish(String.t()):::okdefpublish(text)dotext|>Markdown.format()|>Mastodon.publish()end

you can inject dependencies, with a sensible default:

@specpublish(String.t()|[String.t()],Formatter.t(),Publisher.t()):::okdefpublish(texts,formatter\\Markdown,publisher\\Mastodon)dotexts|>List.wrap()# accept a single string as well|>Enum.each(fntext->text|>formatter.format()|>publisher.publish()end)end

Done. This code is now ready to accept multiple texts at once (as well as a single one), format with anything, and publish anywhere. We’ve covered all three of the potential future requirements that came to mind, without writing a single superfluous line of code (well, about 60 characters were added—apologies).

Of course, we cannot anticipate all of tomorrow’s needs. But some are so obviously staring you in the face that you can handle them without even getting up from your chair. Everyone knows that magic constants are bad. So people do this:

@pi3.14159265defpi,do:@pi

There’s no need to do it this way, for two reasons: if it’s π, it will never change, and no one will confuse it with the CEO’s full name. But if it’s something that might change tomorrow—say, the maximum number of characters for a repost—@max_symbols 140 won’t help much: you’ll still have to find it and change it directly in the code. This, on the other hand, is good:

@max_symbolsApplication.compile_env(:my_app,:max_symbols,140)deftruncate(text,symbols\\@max_symbols),do:String.slice(0..symbols-1)

Need to change the default value?—Welcome to the config. Need the ability to truncate at different lengths?—Here’s a parameter, hello.

In fact, this example scales easily to any “grey zone” of architectural change. The architecture should not change, in principle: the parts should change. The business is young and there are only five clients so far?—Then the function for sending a New Year’s greeting by email (forgive the example) should be ready to parallelize. It’s a MailSender that exports a single function send/2, which for now accepts a text and a list of email addresses and simply iterates over them sequentially. When the hundredth client arrives—we’ll rewrite it to use multiple threads, without touching the rest of the code—and that’s it.

Strive to ensure that every operation in your code is handled by a separately taken, extremely simple, isolated, and tested module. After all, one can spend a lifetime preparing for the day the toilet in one’s bathroom gets clogged, and achieve mastery in the art of unblocking it. Or, when the hour comes, one can simply call a plumber who does only that—but does it well. That is dependency injection.

Divide et Impera

Wed, 25 Mar 2026 16:08:30 +0000

Part two of four key developer skills.

Ability to break the whole into parts and implement small pieces of a large system in complete isolation from one another.

“Divide and conquer”—a time-tested slogan reflecting the strategy for managing large projects, empires included. In English it’s usually rendered as “divide and rule” or “divide and conquer.” You’ll notice that the second verb varies considerably across time—from “conquer” through “rule” to “govern.” The first, however, is immovable, cast in bronze: divide. Slash. Cut the excess from the marble block. Gnaw off your own leg to escape the trap. But I digress.

Whenever humanity invents something new, it first tries to master it by brute force, and then applies well-worn tactics. Programming was no different: at first everyone rushed to bash out code, attempting to write “War and Peace,” and then decided to muddy the waters and, like wound-up toys, began inventing all manner of practices, and—forgive the word—patterns.

I never knew what “SOLID” stands for, because apart from SRP, all the other principles in there are extremely contradictory and generally inapplicable due to their incorrectness. SRP, however, is applicable at all times, and in any application—it ennobles the code, often beyond recognition.

In any course of lectures, in any conference talk, I always say: take scrupulous care to ensure that every piece of your code performs exactly one task, exposes a clear API, and is packaged as a library—not dragged wholesale into the main project. Need spline interpolation?—A library, with its own tests, with a single exported function. Some peculiar grouping operation?—A library. A different grouping?—Extend the exported API of the previous library. Logging in the application?—A library, your own wrapper over the standard logger. And so on. Only bare, dry business logic has the right to live in the application itself.

This approach comes with several pleasant side effects at once:

if a bug is found, it’s far easier to fix in the library than in a gigantic project;
if a colleague needs logging, they’ll use your implementation and the logs will be consistent;
if you want to test calls to the library, it can export convenient helpers for that purpose;
if the library needs new capabilities, you can cover the new API with mocks in five minutes and continue developing the application;
if the library isn’t coupled to business logic, you can always open-source it and get +5 to testing, +10 to new features, and +50 to karma.

When I hear that “this feature can’t be started until that ticket over there is closed,” I want to strike the speaker with a keyboard. How are you going to test it, you muppet, if it’s coupled to third-party code? How can you be confident it works at all, if without someone else’s code—it won’t even start? Why on earth did you go into programming, you numpty, when there are so many wonderful professions around—excavator operator, poet, TikToker.

If anyone tells you that code cannot be tested without other code, or that this particular function is designed to process data, write to the database, and brew coffee simultaneously—that person is a saboteur and should be shown the door.

Any code can be tested in isolation from the project (integration tests are an entirely different matter). Any code doing three things at once can be split into three different places and tested separately, independently of one another. Anyone currently itching to produce a counterexample—would do well to cool down first and try following the advice given above. It’s rather stupid to suddenly make a fool of yourself over such a trifle.

Strive to ensure that every cup in your dinner service can be used on its own. It’s foolish to take out, dirty, and then wash the soup tureen and 24 saucers—just because you had a cup of coffee.

No Failures Despite Bugs

Wed, 25 Mar 2026 16:08:30 +0000

Part one of four key developer skills.

Ability to guarantee fault tolerance even in the presence of bugs in the code.

Large systems will probably always be delivered containing a number of errors in the software, nevertheless such systems are expected to behave in a reasonable manner. ~ Joe Armstrong

Ever since the concept of “software” came into existence, people have been forced to live with the fact that it doesn’t always work as expected. Whether it’s a cockroach in the system unit, rats gnawing through a cable, a Norwegian construction worker accidentally slicing it with an excavator bucket, or even—terrifying to utter—errors in the software, introduced by a developer.

People have desperately fought against bugs in source code—Donald Knuth even established a reward for problems found in TeX. Unfortunately, Knuth’s approach scales rather poorly. In other words, we cannot ask him to write all the software in the world.

To minimize the number of bugs reaching users, humanity invented tests, types, code reviews, static analysis, and lord knows what else. But spending a couple of hours simply browsing the internet is enough to observe firsthand: none of it works worth a damn. Well, of course, if you measure KPIs, or whatever they measure, the first derivative shows a reduction in the number of bugs, and the second—the speed of that reduction. But if you just imagine yourself to be an ordinary person—nothing works.

Some bugs are non-critical, though still annoying. Literally just the other day I witnessed a user forget to switch their keyboard layout while entering a password into Gmail, and Google displayed an “Error 403” page. Seems fine: the login failed, there’s a specific error code for that, here’s your grenade. The user was thoroughly stumped by this page, decided they had done something forbidden—and wrote to me in near-panic. Grumbling about “user literacy” I leave with distaste to the dimwits who, themselves, think nothing of taking their car to a mechanic, dining at restaurants, and calling an electrician to tighten a tap.

Users will make mistakes—that’s normal. What’s not normal is showing them a 403 instead of simply redirecting them back to the login page. But all of this, I repeat, is non-critical.

The core problem is that programmers make mistakes too. And they will continue to make mistakes. And no tests-schmests and types-schmypes will save them (us) from this. Sooner or later we will introduce a critical bug. And our goal is to protect ourselves from that proactively.

The human brain is a complex thing, but one fact about it is well-established: it distorts objective reality, most often in the direction most convenient to us (the opposite behavior requires at least outpatient treatment). When I test the behavior of my piece of …ahem… code, I knowingly deceive myself that these two obvious and three non-obvious cases will cover all possible scenarios. Then the tester arrives and tries to pass a lizard as a parameter. The programmer fixes the code. And then the product ships to users.

In every piece of writing longer than a limerick I recommend Property Based Testing—I’ll recommend it here too. This technique will help relieve the tester of duty and keep his lizards at home. But it won’t protect against a non-obvious bug that manifests, for example, when the clocks are moved forward for daylight saving time.

So what is to be done?—Just listen to what Joe Armstrong was saying more than thirty years ago: bugs in code will always exist. Trying to eliminate them with tests and types is like defending yourself against a downpour by putting a plastic bag on your head: your head stays dry, but you look like an idiot, and everything below the neck is soaked through.

What then? Well, for starters, accept that bugs will always exist. You won’t be able to fix them all (unless you are Donald Knuth, of course). After the “acceptance” stage—you might try reading what Joe goes on to say about this.

And what he says is roughly this: in all unexpected cases—stop execution. Do not try to cover every possible path a situation might take. Handle success and the expected error where that makes sense (for example, on a failed login—redirect to the login form). In all other cases, including a lizard crawling in over the network—stop execution. In Erlang, this principle gave the language its famous slogan “Let it crash!” Which means: if something has gone wrong—fail immediately. Code only the narrow road of correct, expected execution—as narrow as a Formula One track.

Then the bugs you do introduce will also be handled—not all of them, but many.

Now simply restart the execution that led to the failure, with the same input data. Something unforeseen may have happened—a connection limit exceeded, a third-party service not responding, anything at all. This is not the time to investigate; just try again. Yes, it’s like the famous “have you tried turning it off and on again”—and it works sometimes. If something else depended on your piece—restart that too. Automate this restart so you don’t have to write it from scratch every time or copy-paste it from the previous project. If after a number of attempts it still hasn’t worked—dutifully dump the error to the log and skip this particular set of input data (without losing it).

Congratulations—we’ve just reinvented an underspecified, buggy, and slow implementation of half an Erlang supervisor tree. The Kubernetes people went roughly down this road too: they even managed to sell their pitiful copy of OTP to the people who swim exclusively in the mainstream.

With this simple trick you can forever protect yourself against unexpected failures: if a failure is normal, expected behavior of your program—it automatically transforms from a repulsive caterpillar into a beautiful butterfly. The most remarkable thing is that such an ecosystem is, in principle, not terribly difficult to implement even in Go—but the language’s concepts inherently assume violence against the programmer rather than lightening his burden, so I doubt anything like this will ever appear there. Especially given that Kubernetes exists and you can just not bother—simply restart the entire world, cooling down the misplaced enthusiasm of the cache, serving 503s, and generally tormenting your users in every way imaginable; but the entire industry has spent decades working to ensure that users have grown accustomed to being hated and nothing working.

90% of all internet traffic is handled by Erlang—and this is no accident whatsoever.

Allow me to quote one more passage, explaining precisely what problem Joe Armstrong was solving (and solved).

At the time, Ericsson built large telephone exchanges that had hundreds of thousands of users, and a key requirement in building these was that they should never go down. In other words, they had to be completely fault-tolerant.

Like all brilliant solutions in life, this one turned out to be fairly simple. To simplify it even further—one must completely forbid object mutability, implement lightweight processes, and allow them to communicate exclusively through asynchronous messages. Then everything described above simply falls into your lap as a gift: supervisor trees practically require no special implementation—they emerge almost out of the box on their own. But that is perhaps a subject better left for next time.

Software Development in 2023

Wed, 25 Mar 2026 16:08:30 +0000

I was tempted to call this piece “five myths” to fully merge in ecstasy with the bullshit that has been pouring into the ears of the conscientious reader lately, but it seemed like overkill.

I’ve been writing code since 1986, when my Euclidean algorithm in Fortran booted on the third attempt from punch cards on an ES-1060. Since then I’ve written in so many different languages that I’ve lost count. For roughly a dozen of them I’ve been paid actual money. All my life I carefully avoided any managerial roles, but CEOs turned out to be perceptive, and after some time I’d find myself with actual human beings reporting directly to me. I wanted to write code and wasn’t particularly dreaming of associating with my own kind.

The greatest compliment I’ve ever received in my life was a casually dropped phrase by Roma Ivanov, when at some meeting about product development at Yandex (the only time in my career when my job title on the payroll contained the word “manager”), I started going deep into technical implementation details and someone waved their hand: “What does it matter, you won’t be the one implementing it anyway!” Roman snorted and loudly muttered: “Him?—He will.”

When the HR department at my current firm went berserk and issued a demand—for that salary, this person ought to have their own team—the CTO met me halfway once again, and now I officially head “Aleksei Team,” consisting of one person.

In the time I’ve been pressing buttons on a keyboard for sustenance, several eras have come and gone. Someone else’s code finally acquired something resembling documentation. Programming languages proliferated—written by clever people, professionals in their field, who mistakenly assumed that a programmer was, by default, not stupid, and should be given as many capabilities as possible. Perl, Ruby, even Python were created to make the process of writing code exciting, and the programmer’s expressive power practically limitless. This substantially lowered the profession’s entry threshold and spawned a crowd of craftsmen who literally achieved results by random poking. When the fraction of dimwits in the profession exceeded any conceivable limit, something had to be done, and the most farsighted ones devised making it as difficult as possible to write non-working code: thus tests appeared, and in their wake—types (in those languages where they’re not needed in the slightest).

Matz was creating a language in which literally nothing would obstruct the programmer, and that’s exactly why Ruby is adored by its devotees (of which I am one). But this approach works well only on the condition that the developer actually knows what they’re doing. Unfortunately, in the modern world one cannot count on that at all, which is why the coherent and actually quite interesting-in-concept JavaScript grew heavy with ponderous and pointless TypeScript. Python—with those damned annotations that literally affect nothing. Perl was simply forgotten, because the nearly unlimited power of its syntax demands too significant a cognitive effort. PHP lives only thanks to Facebook.

Amid all this obscurantism, people with pretensions dusted off Haskell from a dusty closet (which I worked with in 2000–2002, and which ultimately ruined our project back then due to the absence of dependent types). It is 2023, and there are still no dependent types, which makes the choice of Haskell rather dubious for any project—given the absence of any pronounced upsides and the enormous number of equally pronounced downsides (which, in principle, the authors never concealed: the language is academic, experimental, “avoid success at all costs,” and so on). Idris, ideologically absolutely correct, turned out to be too complex for dysfunctional poseurs and is essentially being chiseled along by its author alone.

Also, while we were striding in seven-league boots toward the bright future, the language support teams completely ignored the thoroughly changed paradigm of use. While the community in chorus was hammering type wedges between the tightly fitted logs of code and pursuing one hundred percent test coverage, computers became multi-core and the network became more accessible than hard drives.

In the modern world, fast algorithms are not needed (because the bottleneck will most likely be elsewhere). Micro-optimizations are not needed (because throwing more memory at the problem is cheaper than debugging an assembly insert). Approaches and practices dragged in from the previous century become meaningless—things like properly naming variables and splitting code into modules.

On the other hand, people who understand the direction of applied programming’s evolution and can write code that scales in every direction simply by plugging in additional capacity are worth their weight in gold. It suddenly turned out that such people existed even in the era of antediluvian computers; Joe Armstrong’s dissertation contains everything one needs to understand about designing fault-tolerant distributed systems, and erlang—everything required for writing such code.

I’ve often heard from various people that Erlang’s syntax is …umm… a bit exotic. I strongly disagree with this claim (like all other brilliant decisions, Joe took it from real life—it mirrors the syntax of the English language), but I’ve still chosen elixir for my work—not for the grammar, of course, but for its truly impeccably implemented metaprogramming. Comparable metaprogramming existed only in LISP, but LISP is too cumbersome and its infrastructure is dreadful.

So, to keep this text from appearing to be blanket praise for Elixir (which it genuinely is, but I’d like to conceal that fact)—let me present the four most important skills, in my not entirely amateur opinion, that are in demand in 2023:

And that’s about the lot. As for all those algorithms, optimizations, memory consumption…—well, it’s better if you can estimate algorithmic complexity and avoid traversing a list back and forth with nested loops. Probably O(n⁵) is worth avoiding right from the start. But pursuing O(log n) instead of O(n) is almost never necessary, and if it is—there’s almost certainly a working implementation out there already. “You can’t read the entire table into memory!”—a colleague once told me, in horror. I asked him: “Why?”—“There could be a million records!” At that point there were about five hundred records, and each was under a kilobyte. I waved my hand: “Even a billion—this machine has 120 gigs of RAM; if we hit failures, we’ll add paging.”

There was already a get_page(from = 0, count = -1) function in my code that always returned the entire table. It’s four years old now, and it will be implemented more seriously someday. Around 2037.

BEAM Metrics in ClickHouse – Andrea Leopardi

Wed, 18 Mar 2026 16:51:45 +0000

At Knock, we're big into observability. Who isn't! We're also huge fans of ClickHouse. This post is an overview of how we started using ClickHouse to collect detailed, high-cardinality BEAM metrics for all sorts of reasons.

We use Datadog for all of our metrics (and monitoring, and logging!). AWS metrics, database metrics, Kubernetes, application—all of it ends up in Datadog. We know how to use the tool and generally find it pleasant. However, you pay for that. It's not like this is a hot take: Datadog is ✨expensive✨.

The thing we pay the most for is definitely custom metrics, that is, unique combinations of metric name and tag values. A classic example:

That's only a single counter metric, fetched_buckets. However, tag cardinality means you have:

Two env values
Three account_type values

That means 2 × 3 = 6 unique custom metrics! You can easily see how that scales. If you add a backend tag to determine the storage buckets were fetched from, with say two values, you're essentially multiplying that 6 by yet another 2, landing on 12 custom metrics now. Really easy to explode.

High Cardinality Metrics

It's not really these "low-cardinality" metrics that cause issues. You'll end up with thousands of metrics this way, but that's quite a normal range for any Datadog-like service.

The problem is high-cardinality metrics. For example, the Datadog agent running on our Kubernetes nodes attaches a whole slew of tags to metrics, including a kube_node tag for the cluster node that the metric comes from. If you have many nodes, or scale nodes up and down, every metric goes up in cardinality.

For us, one of the main causes of problems was the pod_name tag, which we use to tag some of our metrics with the Kubernetes pod they come from. Application-level metrics don't generally benefit from such granularity, but there's a whole set of metrics that absolutely does: ones about the BEAM. Each pod runs its own Erlang node. Knowing the total number of running processes without slicing that by Erlang node is meaningless: great, you know you have 800k processes running on your cluster, but you'll never know if a pod runs amok and makes for a big chunk of that number.

The dilemma? If we have 100 pods running at any given time, that's every BEAM metric times 100. The explosion, though, is that repeats every time we deploy! New versions of the code cause new Docker images, which cause new pod names, which means 100 more unique combinations of each metric that is tagged with pod_name. Boom.

Data like this, however, is paramount for a platform team operating a system like this. It's critical during incident, to spot trends and patterns, and to understand what our system is doing. We don't want to give it up. If anything, we want more.

For example, recently we had to deal with some nodes running hot. BEAM's runtime inspection capabilities are fantastic. We jumped on the problematic boxes, fired up IEx, and snooped around using recon to our hearts' content. Nothing better than being able to do

biggest_offenders = :recon.proc_window(:reductions, 5, _ms = 1000)

# Now just get all the info + stacktrace to get some more context on these processes:
Enum.map(biggest_offenders, fn {pid, _reductions, _some_info} ->
  {pid, Process.info(pid)}
end)

on a live, running system and just get a peek at what's going on in there.

This is invaluable, but it's not the solution for constantly monitoring our systems. We have no historical data on, say, the five processes consuming the most memory at any given time.

Enter ClickHouse

ClickHouse is a really fast, columnar, OLAP database. It's a workhorse, and we use it to power product features, like segmentation or audit logs, and analytics. The most recent thing we were able to trivially ship thanks to ClickHouse was email.info, a breakdown of email provider performance: basically, we had ClickHouse chew through our provider delivery data and spit out analytics from that. It might not be that impressive on the surface, but I was baffled by how easy it was to go from thinking about doing this to live page.

Back to our BEAM analytics. My awesome coworker Victor, who got us all into ClickHouse in the first place, kind of opened a Pandora's box here: what if we store this high-cardinality, high-volume internal telemetry data in ClickHouse?

This makes so much sense. We can periodically sample the top-n processes based on some metric (memory usage, reductions, mailbox length) and dump a bunch of information about them in a ClickHouse table. We created a new internal_telemetry database and got to work. "Got to work" makes it sound way worse than it was: the whole thing was done in a few hours.

Table Design

We didn't spend too much time thinking about the table structure here. This is internal data that only we will be querying, so query performance is not the most important factor.

This is where we landed for the first iteration:

CREATE TABLE top_processes (
    metric LowCardinality(String),
    deployment LowCardinality(String),
    pod_name LowCardinality(String),
    timestamp DateTime64(3, 'UTC'),
    pid String,
    registered_name String DEFAULT '',
    current_function String DEFAULT '',
    initial_call String DEFAULT '',
    reductions UInt64,
    message_queue_len UInt32,
    memory UInt64,
    label String,
    memory_details Map(String, UInt128)
)
ENGINE = MergeTree
PARTITION BY toMonday(timestamp)
ORDER BY (metric, deployment, pod_name, timestamp)
TTL toDate(timestamp) + toIntervalDay(30)
SETTINGS ttl_only_drop_parts = 1;

Some notes:

metric is the metric we "found" this process through. For example, metric = 'memory' means that this row represents a process that was one of the top-n processes by memory usage at the time of scanning the system.
deployment is our Kubernetes deployment.
The TTL is there because we have not felt the need for this level of detailed metrics to be retained for months. ClickHouse takes care of dropping partitions older than one month here, so the whole thing doesn't explode in storage size. The funny thing is this though: storage is very cheap, and we mostly query over short time ranges; we could bump the TTL to be much longer without paying much more.
memory_details is a map so we can easily add to it later on without having to change the schema. This felt like the right compromise.

The rest of the columns should be self-explanatory.

The ordering key (ORDER BY) is one of the keys to ClickHouse queries. It lets ClickHouse find data quickly and scan as little as possible. A good rule of thumb is to go with increasing cardinality, which is exactly what we did here. The query patterns we expect to issue are mostly:

For a single deployment, as it rarely makes sense to look at metrics aggregated over different applications.
Most often for a single pod, to spot things going wrong within a given Erlang node.
Focused on a window of time, thus the ordering by timestamp.

An ordering key like this doesn't mean you cannot skip parts of it in your queries—it just means doing that results in less-optimized queries and data scans. If we'll ever find ourselves, say, querying all metric across all pod_name, we could easily deploy a materialized view that writes this data into a separate table optimized for that query pattern. Storage is cheap.

Periodically Dumping System State

Honestly, I was conflicted on whether I should even include this section. Nothing about this is new to anyone doing Erlang/Elixir!

We're just using the lovely telemetry_poller to collect and dump system state every few seconds. The child spec for it goes like this:

@doc """
Returns a child spec for a `telemetry_poller` that periodically
logs top processes.
"""
def child_spec([] = _opts) do
  Supervisor.child_spec(
    {:telemetry_poller,
     measurements: [{__MODULE__, :measure_and_persist, []}],
     period: to_timeout(second: 5),
     init_delay: to_timeout(second: 5),
     name: :top_processes_poller},
    id: __MODULE__
  )
end

Nothing fancy, at all. measure_and_persist/0 looks like what you probably expect it to look like:

@doc """
  Collects top process data and writes it to ClickHouse.
  """
  def measure_and_persist do
    now = DateTime.utc_now()
    pod_name = System.get_env("HOSTNAME", "unknown")
    deployment = System.get_env("DEPLOYMENT", "<redacted>")

    rows =
      Enum.flat_map(@metrics, fn metric ->
        metric
        |> :recon.proc_count(@top_n)
        |> Enum.map(fn {pid, _value, _recon_info} ->
          build_row(pid, metric, now, pod_name, deployment)
        end)
        |> Enum.reject(&is_nil/1)
      end)

    if rows != [] do
      InternalTelemetryRepo.insert_all(TopProcess, rows)
    end

    :ok
  end

We use Ch as our ClickHouse driver, and ecto_ch to integrate it with Ecto.

That's the whole infrastructure for capturing this data, right there. build_row/5 is just a bunch of formatting and Process.info/2 calls. That's it!

In the Real World

This has been in production for just 10 days. Our top-n is 20: that is, we sample the top-20 processes over three metrics: reductions, message queue length, and memory. We sample every 5 seconds. Our total throughput in this table is:

$$ \frac{top{\text -}n \times metrics \times pod\_count}{sample\_interval} = \frac{20 \times 3 \times 100}{5s} = 1200 \space rows/s $$

Easy peasy for good ol' ClickHouse. Some stats from the live table:

We have ~607M rows in top_processes. Spare change for ClickHouse.
Compressed size of this table in object storage is just 5.64 GB (134.66 GB uncompressed).

Now for the fun part: queries! Let's just look at a couple of fun queries we were able to issue over the past few days.

Frequent Fliers

Find the processes that appear most often in the top-n processes by memory, over the past 24 hours. This could help find processes that consistently consume a lot of memory (rather than spiking once in a while).

SELECT
    concat('`', registered_name, '`') AS name,
    count() AS appearances,
    formatReadableSize(avg(memory)) AS avg_memory,
    formatReadableSize(max(memory)) AS max_memory
FROM top_processes
WHERE
  metric = 'memory'
  AND deployment = {deployment:LowCardinality(String)}
  AND registered_name != ''
  AND timestamp >= now() - INTERVAL 24 HOUR
GROUP BY registered_name
ORDER BY appearances DESC
LIMIT 10
FORMAT MARKDOWN

name	appearances	avg_memory	max_memory
`tzdata_release_updater`	362659	8.25 MiB	8.55 MiB
`Elixir.Sentry.Sources`	362659	13.18 MiB	13.18 MiB
`code_server`	362659	26.52 MiB	51.77 MiB
`application_controller`	362659	10.23 MiB	10.29 MiB
`ldclient_event_process_server_default`	362657	54.69 MiB	283.01 MiB
`Elixir.<redacted>.K8sServer`	352357	10.03 MiB	12.00 MiB
`ldclient_event_server_default`	349761	12.82 MiB	39.39 MiB
`<redacted>`	286100	5.10 MiB	7.88 MiB
`<redacted>`	275572	4.65 MiB	11.58 MiB
`<redacted>`	242255	5.70 MiB	12.80 MiB

Just to give you an idea: this executed in 0.238s, scanning 7,389,364 rows (516.95 MB of data) 🤯.

Maximum Consumer Memory

This is useful to spot if there are any pods in a given deployment where the process consuming the most memory is consuming more than the top memory hog in the other pods.

SELECT
    toStartOfMinute(timestamp) AS minute,
    pod_name,
    max(memory) peak_memory
FROM top_processes
WHERE metric = 'memory'
  AND deployment = {deployment:LowCardinality(String)}
  AND timestamp >= now() - INTERVAL 3 HOUR
GROUP BY minute, pod_name
ORDER BY minute;

ClickHouse Cloud has decent charting functionality, and this type of query spits out very easy-to-spot-stuff-on charts. Here's what it looks like, over time, broken by pod (with no runaway processes):

Also, I can't help but share this: even when forcing the query to avoid any caches (SETTINGS use_query_cache = false), this query returned in 42 milliseconds, reading 1,131,629 rows (21.51 MB of data). It never gets old.

Conclusion

I’m aware that using ClickHouse for this type of stuff isn't groundbreaking. ClickHouse is mostly sold as an analytics database, so much so that there's a whole product called ClickStack to store and query OpenTelemetry data. This is truly ClickHouse's bread and butter.

What I described in this post is just what I found to be a perfect use case. BEAM is easy to inspect (or "self"-inspect) at runtime, making this whole thing trivial.

Just to recap, we talked about:

Why high-cardinality metrics are hard to store in metric products (like Datadog in our case).
How we overcame this by storing lots of detailed data about the runtime of the BEAM in ClickHouse.
A couple of example queries we ran that show the potential use cases of something like this.

I did not explicitly close the cost loop here because storage cost with ClickHouse, at least at this scale, is negligible. At the time of writing, ClickHouse charges something like USD$25/TB/mo in us-east-*. Even if we grow the size of our stored internal metrics 100 times what it is today, we'll be storing half a terabyte of data and paying less than a single ChatGPT Plus subscription. There are obviously egress and compute cost to factor in, but those are only relevant when we query the data; with five engineers on the platform team only having to look at this data once in a while, that cost amounts to zero for all intents and purposes.

I want to give another shout out to my colleague Victor, who essentially came up with this whole idea.

Thanks for reading!

Upgrading Amignosis: Phoenix and Elixir with Claude Code

Tue, 03 Mar 2026 08:19:26 +0000

My company, Amignosis, website is a simple Phoenix LiveView web app. It was on Phoenix 1.7.21 and Elixir 1.18, and I wanted to upgrade it to Phoenix 1.8.3 and Elixir 1.19. So, I decided to try Tidewave with Claude Code, to do it for me.

Here’s a video showing the process, but read ahead if you want to get the gist of it.

I asked Claude Code to read the Phoenix CHANGELOGs online. Then I told it that I would feed it the diffs between 1.7.21 and 1.8.3 one by one.

The diffs came from phoenixdiff.org, a very handy site that shows you what’s changed between Phoenix versions. It basically compares what would be generated by mix phx.new, if you were to do it by hand.

The problem with that site is the presentation of the diffs. Because of this, Claude Code can’t read the content. I cloned the repository of that project. I enhanced it so that I can copy each diff to the clipboard.

As I mentioned above, my first strategy was to feed it each diff one by one. During the process, I realized a better strategy. First, save all the diffs locally. Then, point Claude Code to them. This way, it can have the whole picture.

What worked well is that Claude Code (via Tidewave) applied the diffs. It also tweaked them when needed to reference my specific project’s domain names.

Hiccups

The process worked quite well. I had a few hiccups though.

Tidewave got stuck in a loop due to a huge prompt

The first one was that at some point, one of the diffs was huge. I pasted it in the Tidewave chat prompt and hit enter. A message appeared saying the prompt was too big and couldn’t work. At that point, Tidewave got stuck in a loop. Anything I typed, no matter how small, I was getting the same error message. I couldn’t find a way to get it out of that state. Starting a new session meant losing the context, but that’s what I did anyway. It had the context of the pending applied diffs locally, and I described what we were working towards from scratch. It seemed it continued without any issues.

Next time, I will avoid pasting big text. Instead, I will point it to a local file with that content. This way, it can just apply it directly.

Claude Code decided to skip daisyUI

A big change in Phoenix from 1.7.x to 1.8.x was the introduction of the daisyUI library. It’s not mandatory to use it, but some of the default files generated assume you are using it anyway.

Because I was feeding the diffs one by one, Claude Code didn’t have the whole context. So it decided to skip DaisyUI altogether.

At some point, I asked it to embrace the introduction of daisyUI. We would handle any styling changes it would cause later. My app is very small (just two pages) so it wouldn’t be that hard.

After some back and forth, we ended up with a running version. It had the same look and feel as my earlier version.

Tidewave couldn’t refresh its shell context

This wasn’t a huge deal, but still created a tiny friction. At some point, as Claude Code was applying the diffs, we started seeing some warnings. Those warnings were because the project was running with Elixir 1.18. Yet, Phoenix >= 1.8 expects Elixir 1.19.

I am using asdf to manage Elixir/OTP versions. As a result, my current Tidewave session couldn’t refresh its terminal context to that version. That’s despite us switching asdf to the new versions locally. After some back and forth, we managed to snap out of the stuck situation. I don’t remember if I actually just started a new chat in Tidewave or not.

Conclusion

I think next time I can help Claude Code finish this faster. Two strategies should help here:

Don’t leave a project on an older version for too long
Feed Claude Code *all* the diffs at once instead of one by one

Here’s what I am doing

At Amignosis, I pour my heart and skill into crafting slowly brewed software, one thoughtful line at a time. I am craftsman in a world of complexity and low-quality solutions. I am a shoemaker. I take the time to create simple, timeless software built to last. Check what I am doing now and talk to me.

Type your email…

Join 71 other subscribers

Like this:

Like Loading…

Process-Based Concurrency: Why BEAM and OTP Keep Being Right

Mon, 02 Mar 2026 05:22:31 +0000

A first-principles guide to process-based concurrency — what makes BEAM different, how OTP encodes resilience, and why everyone keeps reinventing it.

PhoenixPress: Compile-Time SEO for Phoenix Apps

Mon, 02 Mar 2026 05:22:31 +0000

We open-sourced PhoenixPress — sitemaps, robots.txt, and RSS feeds for Phoenix, generated at compile time with zero runtime overhead.

Building a Calendar in Phoenix LiveView

Mon, 02 Mar 2026 05:22:30 +0000

How we built a full-featured calendar in LiveView — data model, recurring appointments, multi-user views, drag-and-drop, and performance.

Process-Based Concurrency: Why BEAM and OTP Keep Being Right

Mon, 02 Mar 2026 05:22:18 +0000

Every few months, someone in the AI or distributed systems space announces a new framework for running concurrent, stateful agents. It has isolated state. Message passing. A supervisor that restarts things when they fail. The BEAM languages communities watch, nod, and go back to work.

This keeps happening because process-based concurrency solves a genuinely hard problem, and the BEAM virtual machine has been solving it since 1986. Not as a library. Not as a pattern you adopt. As the runtime itself.

Dillon Mulroy put it plainly:

Thirty thousand people saw that and a lot of them felt it. The Python AI ecosystem is building agent frameworks that independently converge on the same architecture — isolated processes, message passing, supervision hierarchies, fault recovery. The patterns aren’t similar to OTP by coincidence. They’re similar because the problem demands this shape.

This post isn’t the hot take about why Erlang was right. It’s the guide underneath that take. We’ll start from first principles — what concurrency actually means, why shared state breaks everything, and how processes change the game. By the end, you’ll understand why OTP’s patterns keep getting reinvented and why the BEAM runtime makes them work in ways other platforms can’t fully replicate.

We write Elixir professionally. Our largest production system — a healthcare SaaS platform — runs on 80,000+ lines of Elixir handling real-time scheduling, AI-powered clinical documentation, and background job orchestration. This isn’t theoretical for us. But we’ll explain it like we’re explaining it to ourselves when we first encountered it.

The concurrency problem, stated plainly

Your program needs to do multiple things at once. Maybe it’s handling thousands of web requests simultaneously. Maybe it’s running AI agents that each maintain their own conversation state. Maybe it’s processing audio transcriptions while serving a real-time dashboard.

The hardware can do this. Modern CPUs have multiple cores. The question is how your programming model lets you use them.

There are two fundamental approaches.

Shared state with locks. Multiple threads access the same memory. You prevent corruption with mutexes, semaphores, and locks. This is what most languages do — Java, C++, Go (with goroutines, but shared memory is still the default model), Python (with the GIL making it worse), Rust (with the borrow checker making it safer).

The problem with shared state isn’t that it doesn’t work. It’s that it works until it doesn’t. Race conditions are the hardest bugs to reproduce, the hardest to test for, and the hardest to reason about. The more concurrent your system gets, the more lock contention slows everything down. And a single corrupted piece of shared memory can cascade through the entire system.

Isolated state with message passing. Each concurrent unit has its own memory. The only way to communicate is by sending messages. No shared memory, no locks, no races.

This is the actor model. Carl Hewitt proposed it in 1973. Erlang implemented it as a runtime in 1986. Every few years, the rest of the industry rediscovers it.

What a “process” means on BEAM

When BEAM programmers say “process,” they don’t mean an operating system process. OS processes are heavy — megabytes of memory, expensive to create, expensive to context-switch. They don’t mean threads either, which share memory and need synchronization. And they don’t mean green threads or coroutines, which are lighter but still typically share a heap and lack true isolation.

A BEAM process is something different:

~2KB of memory at creation. You can spawn millions of them on a single machine.
Own heap, own stack, own garbage collector. When a process is collected, nothing else pauses. No stop-the-world GC events affecting the entire system.
Preemptively scheduled. The BEAM scheduler gives each process a budget of approximately 4,000 “reductions” (roughly, function calls) before switching to the next one. No process can hog the CPU. This happens at the VM level — you can’t opt out of it.
Completely isolated. A process cannot access another process’s memory. Period. The only way to interact is by sending a message.

This last point is the one that changes how you think about software. In most languages, when something goes wrong in one part of your program, the blast radius is unpredictable. A null pointer in a thread can corrupt shared state that other threads depend on. An unhandled exception in a Node.js async handler can crash the entire process — every connection, every user, everything.

On BEAM, the blast radius of a failure is exactly one process. Always.

# Spawn a process that will crash
spawn(fn ->
  # This process does some work...
  raise "something went wrong"
  # This process dies. Nothing else is affected.
end)

# This code continues running, unaware and unharmed
IO.puts("Still here.")

This isn’t a try/catch hiding the error. The process that crashed is gone — its memory is reclaimed, its state is released. Everything else keeps running. The question is: who notices, and what happens next?

Message passing and mailboxes

If processes can’t share memory, how do they communicate?

Every BEAM process has a mailbox — a queue of incoming messages. You send a message to a process using its process identifier (PID). The message is copied into the recipient’s mailbox. The sender doesn’t wait (it’s asynchronous by default). The recipient processes messages from its mailbox when it’s ready.

# Process A sends a message to Process B
send(process_b_pid, {:temperature_reading, 23.5, ~U[2026-02-22 10:00:00Z]})

# Process B receives it when ready
receive do
  {:temperature_reading, temp, timestamp} ->
    IO.puts("Got #{temp}°C at #{timestamp}")
end

A few things to notice:

Messages are copied, not shared. When you send a message, the data is copied into the recipient’s heap. This sounds expensive, and for very large messages, it can be. But it means there’s zero possibility of two processes modifying the same data. The tradeoff is worth it — you buy correctness by default.

Pattern matching on receive. The receive block uses Elixir’s pattern matching to selectively pull messages from the mailbox. Messages that don’t match stay in the mailbox for later. This means a process can handle different message types in different contexts without any routing logic.

Backpressure is built in. If a process receives messages faster than it can handle them, the mailbox grows. This is visible and monitorable. You can inspect any process’s mailbox length, set up alerts, and make architectural decisions about it. Contrast this with thread-based systems where overload manifests as increasing latency, deadlocks, or OOM crashes — symptoms that are harder to diagnose and attribute.

The message-passing model creates a natural architecture. Each process is a self-contained unit with its own state, handling one thing well. Processes compose into systems through messages — like microservices, but within a single runtime, with nanosecond message delivery instead of network hops.

”Let it crash” — resilience as architecture

This is the most misunderstood concept in the BEAM ecosystem.

“Let it crash” does not mean “ignore errors.” It does not mean “don’t handle edge cases.” It means: separate the code that does work from the code that handles failure.

In most languages, business logic and error recovery are interleaved:

def process_payment(order):
    try:
        customer = fetch_customer(order.customer_id)
    except DatabaseError:
        logger.error("DB failed fetching customer")
        return retry_later(order)
    except CustomerNotFound:
        logger.error("Customer missing")
        return mark_order_failed(order)

    try:
        charge = payment_gateway.charge(customer, order.total)
    except PaymentDeclined:
        notify_customer(customer, "Payment declined")
        return mark_order_failed(order)
    except GatewayTimeout:
        logger.error("Payment gateway timeout")
        return retry_later(order)
    except RateLimitError:
        sleep(1)
        return process_payment(order)  # retry

    try:
        send_confirmation(customer, charge)
    except EmailError:
        logger.warning("Confirmation email failed")
        # Continue anyway? Or fail? Hard to decide here.

    return mark_order_complete(order)

Every function call is wrapped in error handling. The happy path — the actual business logic — is buried under defensive code. And every new failure mode adds another branch. The code becomes harder to read, harder to test, and harder to change.

On BEAM, you write the happy path:

defmodule PaymentProcessor do
  use GenServer

  def handle_call({:process, order}, _from, state) do
    customer = Customers.fetch!(order.customer_id)
    charge = PaymentGateway.charge!(customer, order.total)
    Notifications.send_confirmation!(customer, charge)
    {:reply, :ok, state}
  end
end

If any of those calls fail, the process crashes. That’s not a bug — it’s the design. A supervisor (which we’ll get to next) is watching this process. It knows what to do when it crashes: restart it, retry the operation, or escalate to a higher-level supervisor.

The business logic is clean because error recovery is a separate concern, handled by a separate process. This isn’t about being reckless. It’s about putting recovery logic where it belongs — in the supervision tree, not tangled into every function.

Here’s the key insight: the process that crashes loses its state, but that’s fine because you designed for it. You put critical state in a database or an ETS table. The process itself is cheap, stateless enough to restart cleanly, and focused entirely on doing its job.

Supervision trees

A supervisor is a process whose only job is watching other processes and reacting when they die. Supervisors are organized into trees — a supervisor can supervise other supervisors, creating a hierarchy of recovery strategies.

defmodule MyApp.Supervisor do
  use Supervisor

  def start_link(opts) do
    Supervisor.start_link(__MODULE__, opts, name: __MODULE__)
  end

  def init(_opts) do
    children = [
      {PaymentProcessor, []},
      {NotificationService, []},
      {MetricsCollector, []}
    ]

    # If any child crashes, restart only that child
    Supervisor.init(children, strategy: :one_for_one)
  end
end

The :one_for_one strategy means: if the PaymentProcessor crashes, restart it. Leave NotificationService and MetricsCollector alone. Other strategies exist:

:one_for_all — if any child crashes, restart all children. Used when children are interdependent and can’t function without each other.
:rest_for_one — if a child crashes, restart it and all children started after it. Used when later children depend on earlier ones.

Supervisors also enforce intensity limits. You can say “restart this child up to 3 times within 5 seconds — if it keeps crashing after that, terminate this entire subtree and let my parent supervisor decide what to do.” This prevents crash loops from consuming resources indefinitely.

# Restart up to 3 times in 5 seconds, then give up
Supervisor.init(children, strategy: :one_for_one, max_restarts: 3, max_seconds: 5)

The supervision tree isn’t just an error-handling mechanism. It’s your application’s architecture diagram. When you look at a well-structured Elixir application, the supervision tree tells you:

What components exist
What depends on what
What happens when each component fails
How the system degrades under failure

This is information that in most codebases lives in documentation (if it exists at all) or in the heads of senior engineers. In an OTP application, it’s encoded in the code itself.

OTP: patterns, not a framework

OTP stands for Open Telecom Platform — a name from its Ericsson origins that nobody takes literally anymore. What OTP actually is: a set of battle-tested patterns for building concurrent systems.

The most important ones:

GenServer — the general-purpose stateful process

Most processes in an Elixir application are GenServers. A GenServer is a process that:

Holds state
Handles synchronous calls (request → response)
Handles asynchronous casts (fire and forget)
Handles arbitrary messages (system signals, timers, etc.)

defmodule SessionStore do
  use GenServer

  # Client API
  def start_link(user_id) do
    GenServer.start_link(__MODULE__, %{user_id: user_id, messages: []})
  end

  def add_message(pid, message) do
    GenServer.cast(pid, {:add_message, message})
  end

  def get_history(pid) do
    GenServer.call(pid, :get_history)
  end

  # Server callbacks
  def init(state), do: {:ok, state}

  def handle_cast({:add_message, message}, state) do
    {:noreply, %{state | messages: [message | state.messages]}}
  end

  def handle_call(:get_history, _from, state) do
    {:reply, Enum.reverse(state.messages), state}
  end
end

This is a process that holds a conversation history. You can spawn one per user session. Each one is isolated — its own memory, its own mailbox, its own lifecycle. A thousand concurrent users means a thousand of these processes, each consuming ~2KB plus whatever state they hold. The scheduler handles the rest.

Compare this to the typical approach: a shared data structure (Redis, a database table, or an in-memory map) that all request handlers read from and write to. That works, but now you need to think about cache invalidation, race conditions on writes, connection pooling to your state store, and what happens when the store goes down.

With GenServer, the state is the process. No external store to manage. No cache to invalidate. The process is the single source of truth for its own state.

Application — the deployable unit

An OTP Application is a component that can be started and stopped as a unit. It has its own supervision tree, its own configuration, and its own lifecycle. Your Elixir project is itself an Application, and it depends on other Applications (Phoenix, Ecto, Oban, etc.).

When your application starts, the supervision tree starts from the root. Every process is accounted for. Nothing is floating — every process is supervised, and every supervisor is supervised, all the way up to the application root.

This is in contrast with most web frameworks where you start a server, and then various things happen at import time, module load time, and initialization time in ways that are difficult to reason about. In OTP, the startup order is explicit and hierarchical.

The runtime: why BEAM can’t be replicated with libraries

Other languages can implement the actor model as a library. Akka does it for the JVM. Asyncio with some discipline can approximate it in Python. But there are runtime-level properties of the BEAM that can’t be replicated without modifying the VM itself.

Preemptive scheduling

The BEAM scheduler counts reductions (roughly, function calls) for each process. After approximately 4,000 reductions, the scheduler preempts the process and switches to the next one. The process doesn’t get a choice. It doesn’t need to yield cooperatively.

This means: no process can starve the system. If one process enters an infinite loop, runs an expensive computation, or blocks on a slow operation, every other process continues running normally.

Node.js can’t do this. Its event loop is cooperative — if a callback takes 500ms of CPU time, nothing else runs during those 500ms. Python with asyncio has the same limitation. Go is better (goroutines are preemptively scheduled as of Go 1.14), but goroutines share memory, which reintroduces the class of problems isolation solves.

Per-process garbage collection

Each BEAM process has its own heap and its own garbage collector. When a process’s heap needs collection, only that process pauses. Every other process continues executing.

This is a profound difference. In the JVM, Go, Python, or Node.js, garbage collection is a system-wide event. The GC pauses might be short (Go’s GC is excellent), but they affect all running work. For a system handling thousands of concurrent connections, even a 10ms pause affects every single one.

On BEAM, a process’s GC pause affects exactly one connection, one session, one agent. And because processes are small (remember, ~2KB), individual collection events are tiny.

Soft real-time guarantees

The combination of preemptive scheduling and per-process GC gives the BEAM something unusual: soft real-time guarantees. Not hard real-time — this isn’t an RTOS. But consistent, predictable latency across thousands of concurrent operations.

This is why WhatsApp ran 2 million connections per server on Erlang. Why Discord handles millions of concurrent users with Elixir. Why telecom switches — the original use case — require this level of reliability. And why the BEAM is naturally suited for AI agent systems where thousands of concurrent agents need responsive, isolated execution.

Hot code swapping

You can deploy new code to a running BEAM system without stopping it. Running processes continue executing old code until they make a new function call, at which point they transparently switch to the new version. No disconnected WebSockets. No dropped agent sessions. No downtime.

This isn’t theoretical. Ericsson built this because telephone switches can’t go down for deployments. In practice, most Elixir teams use rolling deploys instead. But the capability exists in the runtime, and for systems where connection continuity matters — long-running AI agent sessions, real-time collaborative tools, financial systems — it’s a genuine differentiator.

Where this shows up today

The patterns aren’t academic. They’re running production systems right now.

Phoenix and LiveView

Phoenix handles HTTP and WebSocket connections as BEAM processes. Each connection is isolated. Phoenix Channels routinely handle 100,000+ concurrent WebSocket connections on a single server. LiveView — server-rendered interactive UIs — maintains a stateful process per connected user. That process holds the UI state, handles events, and pushes updates. If a user’s LiveView process crashes, that user sees a reconnection. Nobody else is affected.

Background job processing

Oban — the dominant background job library in Elixir — runs jobs as supervised processes. Failed jobs get retried by their supervisor. Job queues have backpressure through process mailboxes. Scheduled work uses OTP timers. The entire system is a supervision tree.

AI agents

This is the connection everyone’s making right now. An AI agent is:

Long-lived — maintains conversation state across multiple interactions
Stateful — tracks context, memory, tool results
Failure-prone — LLM API calls time out, rate limit, return malformed JSON
Concurrent — you need to run thousands simultaneously

This maps directly to BEAM processes. One process per agent session. State lives in the process. Failures crash the process — a supervisor restarts it. Thousands of concurrent agents are just thousands of 2KB processes on a VM built to handle millions.

The Python ecosystem is building this with asyncio, Pydantic state models, try/except chains, and custom retry logic. It works — with significant engineering effort. But the result is the actor model implemented in userspace on a runtime that wasn’t designed for it. The BEAM gives you this at the VM level, with guarantees that can’t be bolted on.

George Guimarães mapped the correspondence precisely: isolated state is a process, inter-agent communication is message passing, orchestration is a supervision tree, failure recovery is a supervisor, agent discovery is a process registry, event distribution is process groups. All built into the runtime since the 1990s.

Elixir-native AI tooling is emerging to capitalize on this: Jido for agentic workflows, Bumblebee for running transformer models inside supervision trees, and LangChain bindings with step-mode execution for controlled agent pipelines.

The 30-second request problem

Traditional web frameworks (Rails, Django, Express) optimize for requests that complete in milliseconds. AI agent interactions take 5-30 seconds — an LLM call alone can take several seconds, and an agent might chain multiple calls.

Most web servers weren’t built for this. A thread-per-request model with 30-second requests means you need vastly more threads to maintain throughput. Connection pools exhaust quickly. Timeouts cascade.

BEAM was designed for telephone calls — the original long-lived connections. A phone call holds state, runs for minutes, and the system handles millions of them concurrently. Replace “phone call” with “AI agent session” and the architecture is identical.

What can and can’t be replicated

Let’s be honest about the tradeoffs.

What other languages can do with effort:

Actor model semantics (Akka, asyncio patterns, custom frameworks)
Supervision-like patterns (process managers, health checks, Kubernetes restarts)
Message passing (channels, queues, event buses)

A disciplined team can get 70% of what BEAM provides using Python, TypeScript, or Go with the right libraries and architecture. For many applications, that’s enough.

What requires BEAM’s runtime:

True preemptive scheduling with sub-millisecond fairness
Per-process garbage collection with zero system-wide pauses
Process isolation enforced at the VM level (not by convention)
Hot code swapping without disconnecting active sessions
Millions of lightweight processes on a single node

These aren’t features you can add to a runtime. They’re properties of how the runtime is built. The JVM can’t add per-process GC without fundamentally changing its memory model. Node.js can’t add preemptive scheduling without replacing its event loop. Python can’t remove the GIL without… well, they’re working on that.

What BEAM doesn’t do as well:

Raw computational throughput. BEAM is not the fastest VM. For CPU-bound number crunching, the JVM or native code wins. Elixir addresses this with NIFs (native implemented functions) and libraries like Nx for numerical computing.
Ecosystem size. Python’s library ecosystem dwarfs Elixir’s, especially in machine learning and data science. This is the real reason most AI frameworks are built in Python — not because Python’s concurrency model is better, but because that’s where PyTorch, Transformers, and the training infrastructure live.
Learning curve. Process-based thinking requires a mental model shift. Developers from imperative backgrounds need time to internalize “let it crash” and stateless function composition. The payoff is real, but the ramp isn’t trivial.

The recurring reinvention

The pattern keeps repeating because the problem keeps appearing.

In the 1990s, Java’s threading model was supposed to be the answer to concurrent computing. It wasn’t enough. Akka brought the actor model to the JVM in 2009.

In the 2010s, Node.js bet on the event loop and single-threaded async. It worked for I/O-bound web servers. It didn’t work for CPU-bound work or true parallelism. Worker threads were bolted on. Still not enough for isolated, stateful concurrency.

In the 2020s, AI agent frameworks need isolated, supervised, concurrent stateful processes. AutoGen describes itself as an “event-driven actor framework.” LangGraph builds state machines with shared reducers. CrewAI chains task outputs. Each one is building toward something that looks more and more like OTP — but on runtimes that weren’t designed for it.

Erlang’s insight in 1986 was that concurrent, fault-tolerant systems need isolation as a foundational property, not an afterthought. Every runtime that tries to bolt isolation onto a shared-memory model ends up with a system that’s more complex, less reliable, and harder to reason about than one that started with isolation as the default.

The BEAM isn’t the only way to build concurrent systems. But it’s the most coherent one. The runtime, the language, the patterns, and the philosophy are all aligned toward the same goal. When the rest of the industry keeps independently arriving at the same architecture, that’s not coincidence. That’s convergence on a correct solution.

We build production systems on Elixir and the BEAM — from healthcare platforms to real-time infrastructure. If you’re evaluating Elixir for a project or need help with an existing BEAM codebase, let’s talk.

Playing with HTML5 Canvas from Elixir

Thu, 26 Feb 2026 19:56:05 +0000

Creating a server-side drawing API for HTML5 Canvas - using Elixir to generate optimized JavaScript commands for a circular PONG game

Phoenix and Tailwind on FreeBSD – Makefile.feld

Thu, 26 Feb 2026 12:37:31 +0000

Phoenix development on FreeBSD can be a bit of a pain because they integrated TailwindCSS nicely, but it depends on some prebuilt tailwind and esbuild binaries that it auto-fetches for you. Unfortunately, there aren't any prebuilt binaries for tailwind on FreeBSD so lots of people seem to run into issues. I've banged my head about this a lot with various hacky solutions, but so far this one seems the simplest. It also should work if you try to build or deploy on different OSes as long as you have node and npm installed.

diff --git a/assets/package.json b/assets/package.json
new file mode 100644
index 0000000..b5813eb
--- /dev/null
+++ b/assets/package.json
@@ -0,0 +1,8 @@
+{
+  "name": "assets",
+  "dependencies": {
+    "daisyui": "^5.0.16",
+    "tailwindcss": "^4.1.3",
+    "@tailwindcss/cli": "^4.1.3"
+  }
+}

diff --git a/assets/css/app.css b/assets/css/app.css
index ebb15d9..f564733 100644
--- a/assets/css/app.css
+++ b/assets/css/app.css
@@ -13,7 +13,7 @@
 /* daisyUI Tailwind Plugin. You can update this file by fetching the latest version with:
    curl -sLO https://github.com/saadeghi/daisyui/releases/latest/download/daisyui.js
    Make sure to look at the daisyUI changelog: https://daisyui.com/docs/changelog/ */
-@plugin "../vendor/daisyui" {
+@plugin "daisyui" {
   themes: false;
 }

@@ -21,7 +21,7 @@
   curl -sLO https://github.com/saadeghi/daisyui/releases/latest/download/daisyui-theme.js
   We ship with two themes, a light one inspired on Phoenix colors and a dark one inspired
   on Elixir colors. Build your own at: https://daisyui.com/theme-generator/ */
-@plugin "../vendor/daisyui-theme" {
+@plugin "daisyui/theme" {
   name: "dark";
   default: false;
   prefersdark: true;
@@ -56,7 +56,7 @@
   --noise: 0;
 }

-@plugin "../vendor/daisyui-theme" {
+@plugin "daisyui/theme" {
   name: "light";
   default: true;
   prefersdark: false;

diff --git a/config/config.exs b/config/config.exs
index 331305d..b300fc7 100644
--- a/config/config.exs
+++ b/config/config.exs
@@ -43,14 +43,16 @@ config :esbuild,

 # Configure tailwind (the version is required)
 config :tailwind,
   version: "4.1.12",
+  version_check: false,
   your_project: [
     args: ~w(
       --input=assets/css/app.css
       --output=priv/static/assets/css/app.css
     ),
     cd: Path.expand("..", __DIR__)
-  ]
+  ],
+  path: Path.expand("../assets/node_modules/.bin/tailwindcss", __DIR__)

diff --git a/mix.exs b/mix.exs
index 2c3e407..1d59701 100644
--- a/mix.exs
+++ b/mix.exs
@@ -81,7 +90,10 @@ defmodule Booter.MixProject do
       "ecto.setup": ["ecto.create", "ecto.migrate", "run priv/repo/seeds.exs"],
       "ecto.reset": ["ecto.drop", "ecto.setup"],
       test: ["ecto.create --quiet", "ecto.migrate --quiet", "test"],
-      "assets.setup": ["tailwind.install --if-missing", "esbuild.install --if-missing"],
+      "assets.setup": [
+        "cmd --cd assets npm install",
+        "esbuild.install --if-missing",
+      ],
       "assets.build": ["compile", "tailwind your_project", "esbuild your_project"],
       "assets.deploy": [
         "tailwind your_project --minify",

Apply those changes and now mix assets.setup etc will do the right thing: tailwind and daisy will be installed with npm and stored in the assets/ directory, building and deploying the assets will work without errors.

An Elixir Adoption Success Story

Tue, 24 Feb 2026 18:56:27 +0000

How a team that was new to Elixir over-delivered a big project in just three months.

Adopting a new language is more than just a technical journey. A language is only the right tool for the job if your engineers can wield it well. So, as a technical leader you don't just select a language or a framework based on its technical capabilities and how suited it is for the problems you need to solve, you also try to optimize for the skills your team already has, for a robust ecosystem surrounding that language, for strong community support, and so much more. Elixir lays claim to all of these features and I've seen teams ramp up on Elixir with astonishing speed, delivering robust, high-value projects to their organizations in record time. Don't believe me? I'll walk you through one such success story in which a small team of engineers who were mostly new to Elixir managed to over-deliver on a complex project in just three months. You'll learn why we chose Elixir, how Elixir enabled us to be successful, and some of the practices and techniques that will help
your own team adopt Elixir.

This post was sponsored by digital product consultancy DockYard to support the Elixir community and to encourage its members to share their stories.

The Project

Photo by Jo Szczepanska / Unsplash

This Elixir success story centers on a three month engagement with an EdTech company to deliver Flatiron School’s in-browser IDE (Interactive Development Environment) and curriculum management system into their own ecosystem. This meant that we would be porting over some existing codebases in both Ruby and Elixir, as well as building out a new Phoenix application that our partnering EdTech company would need to maintain. These applications together represented a non-trivial set of features:

Students can click a button in the web app that forks and clones their own copy of a GitHub repository containing an interactive lesson.
Students can write and run code for that lesson repository directly in their browser in an interactive development environment (IDE).
Teachers can manage curriculum by creating, editing and deleting lessons.
Teachers can manage curriculum for different cohorts of students by copying the master curriculum plan, editing it for their cohorts, and making it available to those cohorts.

The details of this feature set are not too important. Just understand that we had a lot to build and only a three month engagement to build it in. And, while we did have some legacy applications to maintain, we chose Elixir for the new application that needed to be built. This decision wasn't just the right technical fit for the tasks at hand, it's also the reason why we were so successful in delivering on this project. So, why did we choose Elixir in the first place?

Why We Chose Elixir

Photo by Jan Ranft / Unsplash
We had two viable choices for the new application that needed to be built---Rails and Phoenix. The team to which we were delivering the project had no Elixir experience at all, so a choice to build a Phoenix app represented a choice to adopt Elixir.

The first point in Elixir's favor was that it was technically the right tool for the job. The application we were building was going to be responsible for the curriculum management responsibilities detailed above. We needed concurrency to handle the high volume of lessons that would be deployed or made available to a given class of students at a point in time, and fault tolerance to gracefully handle communication failure with external systems---the app would serve as a touch point between two other apps as well as the GitHub API. The need for concurrency and fault-tolerance made Elixir an obvious choice for this application, but that alone wasn't enough to seal the deal.

At The Flatiron School, we had several strong Elixir evangelists who advocated for Elixir. They got other internal team members excited about the prospect of working in this new language, and they demonstrated a strong commitment to teaching and supporting Flatiron engineers, as well as engineers at our partner Ed Tech company, along their Elixir journeys. The enthusiasm and commitment of our Elixir evangelists helped us make a successful case for adopting Elixir to our partner company. It was decided that the new application that our two teams would build together during this three month engagement would be a Phoenix app. Before we dive into what we built and how, I want to talk a bit about the teams assigned to this project.

The Team

Photo by Pascal Swier / Unsplash

The team responsible for this project was a combination of two separate teams---a contingent from The Flatiron School and a contingent from the EdTech company to whom we were delivering the final product. Between these two teams, we had one engineer with over a year of Elixir experience, three with around six months of Elixir experience and three with no Elixir experience at all. All of our engineers with production Elixir experience were on the Flatiron side, whereas the engineers at our partner company (who would be responsible for owning and maintaining these applications after the three month engagement) were all brand new to Elixir and Phoenix.

It's also important to note that we weren't just facing the typical adoption challenge of introducing a new language to a group of people, we were also tasked with bringing two new teams of people together for the first time. Not only would we have to learn how to program in Elixir, we would also have to learn how to collaborate with a new group of people from an entirely different organization, with its own norms and practices.

With all of these challenges in front of us, we got to work.

The Journey

Our three month engagement proceeded in roughly three stages---technical design, establishing norms and practices, and shipping (lots of) Elixir code.

With only three month in which to deliver on our ambitious goals, we began the engagement with a brief design sprint. We used a one week sprint to identify product and technical requirements and map out the overall design of the new system, including the legacy apps we'd be porting over into a new ecosystem as well as the new app we would be building.

With a technical design in place, we were able to get to work, and we had a lot to do in order to establish both technical and process practices and norms. Elixir's robust ecosystem and tooling made it easy to establish technical norms around testing and test coverage, releasing and deploying, observability, code quality and more. Meanwhile, we also established practices like daily stand ups, weekly retros, and lots of pair programming. We'll dig into the details there and discuss how all of these contributed to our success in a bit.

With our norms and practices in place, we were able to start delivering value fast. About halfway through our three month engagement, we had delivered on our MVP, and by the end of the three months we had knocked off every single feature on our list and then some. Ultimately, we over-delivered on the agreed-upon features and left our partner company with a robust ecosystem that was easy to observe and debug and easy to grow and maintain as they took over ownership for the future.

Our Secrets to Elixir Success

How exactly were we able to over-deliver on this complex project, while bringing together teams from two entirely different organizations, and leveraging predominantly engineers with little-to-no Elixir experience? Keep reading to find out.

Elixir Has a Gentle Learning Curve

Our team of mostly new Elixir devs ramped up on Elixir with impressive speed, thanks in large part to Elixir's gentle learning curve. There are a few reasons for this. First, Elixir is highly eloquent–-–its easy-to-read syntax lowers the barrier to entry for new developers. The syntax will look especially familiar to anyone with a Ruby background, so our Ruby devs already had a leg up when it came to reading existing Elixir code and writing their first Elixir programs.

Elixir's syntax isn't the only language feature that contributes to its gentle learning curve. Elixir's pattern matching functionality, and things like guard clauses and the pipe operator make it easy to write clean and highly testable code. Pattern matching and guard clauses allow developers to implement control flow beautifully, without things like lots of nested if conditions. Meanwhile, the pipe operator encourages developers to write small, pure, single-purpose functions that are strung together in easy-to-read flows. This kind of code has the added benefit of being easy to test, providing developers with a short feedback cycle between writing code and evaluating its behavior. As our developers unlocked the power of these and other Elixir features, they became more and more excited about working with Elixir. That excitement was infectious, and it inspired them and their colleagues to keep learning.

Elixir's robust documentation also greatly contributed to the speed with which our new Elixir devs were able to learn, and be productive in, Elixir. Elixir treats documentation as a first class citizen, so you'll find that everything from core language features to popular libraries are well documented. Elixir documentation is written with the ExDoc tool that makes it easy to generate documentation for your project and get it published to Hex Docs. The official Elixir docs are written with ExDoc and published to Hex Docs, and the rest of the Elixir world has followed suit in documenting their own libraries.

Official docs aren't the only resource out there for learning Elixir and working with new libraries. There are also a lot of Elixir community resources. Our team relied on Elixir School, a free, open-source online Elixir curriculum, as well as in the growing number of blog posts on Elixir topics. Elixir Forum, a community forum where users can post and answer questions and open discussion threads, also provided critical support to our developers throughout this project.

Elixir's language features, documentation, and growing community helped our team level up on Elixir in record time and supported them to solve complex challenges throughout the three month period of this project, far beyond the initial learning phase. Elixir's gentle learning curve isn't the only reason that our team was so successful in adopting this new language and delivering on some aggressive goals, however.

Elixir Has a Robust Ecosystem

While Elixir is still a relatively new language, its ecosystem is maturing fast. You'll find excellent support for everything from testing your code in development, to building and releasing your code in production, to instrumenting and observing it in the wild.

Fast and Comprehensive Testing in Elixir

Elixir's built-in testing framework, ExUnit, provides you with everything you need to exercise every pathway through your code. ExUnit even enables you to test asynchronous code flows and code flows that involve message passing between Elixir processes. Unsurprisingly, since ExUnit is written in pure Elixir, your tests will be highly concurrent and super fast. Mocking in your Elixir tests is also made easy thanks to the Mox library that lets you provide mocks based on the contracts defined in your code. So, while Elixir's language features encourage you to write highly testable code with small, pure, single-purpose functions, actually writing the tests for those functions is also a delightful experience. All of this adds up to a high percentage of test coverage, which helped our team move fast without breaking things when building out a new Elixir app and positioning new Elixir devs to maintain an existing one.

Easy Elixir Releases

An easy-to-use and comprehensive test framework is far from the only Elixir ecosystem feature worth mentioning. Elixir offers first-class support for releases through the Mix release tool. Releases allow us to compile our code and package it up, along with its runtime, into a single deployable unit. Each self-contained release package has everything it needs to run in the deployed environment, so you don't need to get your source code, or the Erlang VM (Elixir's runtime) onto production servers. Your team can even configure multiple versions of each light-weight release package, giving you the flexibility to deploy different pieces of your applications to serve different purposes. For example, you might have one release of an application that only runs some background workers, and another that runs the web server. Release building functionality is built directly into Mix, Elixir's build tool---no need to reach for a complicated third-party dependency. Your team can start building and releasing Elixir apps with native Elixir tooling. This helps contribute to fast deployment cycles.

First Class Observability in Elixir

Once your code is deployed to production, Elixir makes it easy to observe. Elixir treats observability like the first class citizen it is, thanks to the Telemetry library. The Telemetry library leverages Erlang Term Storage (ETS), a robust in-memory store, to dispatch and handle metrics and instrumentations for your code. It has fast become the standard for instrumenting Elixir code, and most Elixir libraries (like the Phoenix web framework and the Ecto database wrapper) use it to emit a standard set of observability metrics. Library authors and app developers alike are encouraged to use the library to do the same, and Telemetry makes it easy to report metrics and events to the destination of your choice, like Datadog or Prometheus. Elixir developers are therefore empowered to design code with observability baked-in right from the beginning. The result is that we gain a high degree of visibility into our applications, from both out-of-the box Telemetry instrumentation in the libraries we use as well as from our own code. Code that we can observe is code that we can debug, so Elixir's observability tooling contributes to fast bug remediation cycles.

When we're working in a new language for the first time, we're bound to introduce a bug or two. Visibility and ease of debugging become even more important here. This is just one more reason that adopting Elixir was such a smooth process for us.

An Elixir Library For All Your Needs

Elixir's increasingly mature ecosystem also means that you'll find a library for most of the common problems you'll be tasked with solving. During our three month project, we often had the experience of starting the day with a new problem to solve and no idea what technologies Elixir offered to solve it, and ending the day with a relevant library identified and implemented. In one instance, we needed a solution for JWT authentication in Elixir. Two new Elixir devs were deployed to investigate our options and they had the Joken Elixir library for working with JWTs implemented in our app by the end of that same day. This experience was repeated again and again–––we would need to solve some common problem, and sure enough we would quickly find an existing Elixir library to help us develop our solution. However, when Elixir's libraries fall short, you can plug the gaps with pure Erlang. Elixir supports Erlang interop, so dropping down to the Erlang level and leveraging Erlang functions is trivial. In one instance, when we had some encryption need that we couldn't solve with an existing Elixir library, we were able to write our own Elixir code that used Erlang's :crypto module to get the job done.

Elixir Was The Right Tool For The Job

Photo by Barn Images / Unsplash

In addition to the language features mentioned earlier that made Elixir easy and fun to learn, Elixir's concurrency, fault-tolerance and distributed nature made it the perfect fit for solving the specific problems facing our team. We needed a curriculum management application for teachers that could process lots of lesson "deployments" at a given time, and recover from failure when communicating with external systems. And we needed an in-browser IDE for students that could manage lots of load, and handle crashes while still remaining stateful so that students wouldn't lose their work.

Elixir's GenServers and Supervision trees were the perfect fit for our curriculum management tool. We were able to build concurrent workflows that enabled teachers to deploy large amounts of content to different groups of students quickly. Elixir also gave us fine-grained control over failure scenarios. We could decide exactly how our program should behave in the event of a deployment failure, whether that failure was caused internally or happened as a result of an error communicating with an external system like the GitHub API. In Elixir, concurrency and fault-tolerance aren't afterthoughts. Elixir is built on top of Erlang's OTP (Open Telecom Platform), which provides a number of libraries and conveniences for managing exactly these pieces of functionality. So, building our application to behave in this manner, while not exactly trivial, wasn't nearly as onerous as it might have been in another language.

Also thanks to Erlang and OTP, we were able to support a distributed, clustered deployment of our in-browser IDE application right out of the box. We could manage a dedicated set of scalable resources for a cluster of IDE deployments, while still easily sharing state between nodes. Elixir's first-class support for distribution made it easy for us to handle a scenario in which, for example, an IDE fails in one node, and is restarted in another node without the current student losing any of their work or being aware of any failure or loss of connectivity.

Thanks to Elixir's powerful concurrency, fault-tolerance, and distribution features, we were able to quickly build robust and critical code pathways that solved for some of the hardest problems in programming, all with a relatively low degree of complexity.

Elixir Makes it Easy to Teach and Learn

Photo by Belinda Fewings / Unsplash

Finally, our Elixir success story wouldn't have been possible without the dedicated and talented engineers attached to this project. Elixir's gentle learning curve and robust ecosystem and tooling helped create passionate Elixir advocates who were committed to teaching their colleagues, along with new Elixir devs with an infectious excitement for learning. Together, this group deployed the following techniques to create a fun and supportive environment that helped us learn quickly and deliver value even faster.

Lots of Pair Programming

For this project, we took an aggressive approach to pair programming, pairing up an experienced and a new Elixir dev on almost every feature and task. This also helped us develop relationships and strengthen communication between the two teams (Flatiron engineers and devs from our partner EdTech company) that were coming together to deliver this project. Pairs were able to work quickly to deliver features while strengthening the Elixir skills of our newer Elixir devs.

Lunch and Learns

As the team built on their Elixir knowledge and delivered feature after feature, the excitement to share what they were learning grew. So, we scheduled regular "lunch and learn" discussions in which team members shared short presentations on Elixir topics over lunch. Topics included testing best practices, concurrency in Elixir, pattern matching, and more. This practice helped build on the excitement and sense of accomplishment that was already growing within the team and gave engineers an opportunity to go deeper on the new topics they were encountering every day.

Celebrating Wins

With so much happening so quickly, it wasn't hard to celebrate our wins. We blocked time out at the end of our daily stand-ups for pairs to demo their latest progress or for individual team members to share a quick tidbit that they'd learned. For bigger milestones, we held celebratory lunches and other team activities. All of this served to keep engagement high, helping us stay motivated to tackle complex problems in a new language.

Adopt Elixir for Much Success

Photo by Patti Black / Unsplash

Many teams and organizations reach for Elixir to solve complex technical problems well-suited for Elixir's concurrent, fault-tolerant, and distributed nature. This was certainly one of the motivating factors in our own choice to use Elixir during this three month project. But, as this case study shows, whether or not you need these compelling popular Elixir features, Elixir can empower your team to write clean, testable, well-instrumented code and have fun doing it. The learning curve to ramp up on Elixir is gentle, and individuals and teams can skill up relatively quickly to deliver value to your organization in record time. Armed with the teaching and learning practices described here, Elixir enabled our team to vastly over-deliver on a complicated project in a short period of time. Elixir is an obvious choice for greenfield development, and after we adopted Elixir, we never looked back.

Eloquent Control Flow + Efficient Time Complexity in Elixir

Tue, 24 Feb 2026 18:56:27 +0000

While tackling the Day 1 challenge from this year's Advent of Code in Elixir, I was reminded of some of the many ways that Elixir let's us write concise, efficient, and eloquent code. In this post, I break down my solution and dive into how you can use recursion, pattern matching and custom guard clauses to implement even complex logic and control flow in an easy-to-reason about way that also avoids common time complexity pitfalls.

Advent of Code Challenge

Advent of Code is...

...an Advent calendar of small programming puzzles for a variety of skill sets and skill levels that can be solved in any programming language you like. People use them as a speed contest, interview prep, company training, university coursework, practice problems, or to challenge each other.

You can complete Advent of Code challenges in any language and submit your answers to "win" stars :star: :star: :star:

It's a fun way to play around with a new language that you're just starting to learn or to refine your skills in a language that you're already familiar with.

After being ~~pestered~~ kindly reminded about it by someone who is definitely not annoying, I decided to try out the Day 1 puzzle in Elixir.

I had a lot of fun putting my solution together and, unsurprisingly, I found that Elixir's features allowed me to implement even complex logic and control flow in a way that was remarkably concise, as well as efficient with regards to time complexity. Keep reading to see how to leverage pattern matching, recursion and custom guard clauses to write some super clean and extremely elegant Elixir code. :soap: :star: :soap:

The Prompt

The Day 1 Advent of Code prompt can be simply stated as:

Find the two elements in a given list that sum to 2020 and return the product of those two numbers.

For example, of the following list: [979, 1721, 366, 299, 675, 1456],

1721 + 299 == 2020, and 1721 * 299 = 514579.

So, your code should return 514579

Attempt 1: Lots of Iterating

Before writing any code, I attempted to conceptualize my approach. The first conceptualization that jumped out at me was heavily reliant on iteration and went something like this:

Iterate over the list
For each element in the list, iterate over the remaining elements in the list
For each pair of elements, add them up. If the sum equals 2020, stop.
If the sum does not equal 2020, keep going

So, taking the example of our list from above, it would look something like this:

Does 979 + 1721 = 2020? Nope!
Does 979 + 366 = 2020? Nope!
Does 979 + 299 = 2020? Nope!
Does 979 + 675 = 2020? Nope!
Does 979 + 1456 = 2020? Nope!

Okay, move on to the second element in the list...

Does 1721 + 366 = 2020? Nope!
...

This approach uses nested iteration and is not very efficient. For each item in the list you have to iterate over the remainder of the list and execute the check to see if the two numbers sum to 2020. In other words, the outer loop executes N times, once for each element in the list. And every time the outer loop executes, the inner loop executes M times, where M is however many steps it must complete to check the current outer loop element against the remaining list elements. As a result, the "check to see if the two numbers sum to 2020" statement executes a total of N * M times.

So, the number of operations your code has to do will grow exponentially for each element added to the list. This represents a high degree of time complexity. Thanks to Elixir, we can do better.

We'll use recursion and pattern matching to avoid the need to perform 💸 💸 💸 expensive nested iterations 💸 💸 💸. Keep reading to find out how!

Attempt 2: Pattern Matching and Recursion

Once I recognized the time complexity of the "lots of iterating" approach, I knew I needed to cut down on iterations. Luckily, Elixir provides us a way to pull elements from a list without iterating over that list--pattern matching. In the next section, we'll use pattern matching and recursion to peel off list elements and perform out "sum to 2020" check on them.

Efficient Code with Pattern Matching

First, let's walk through how Elixir's pattern matching can be applied to list elements such that we can perform our "check if sum is 2020" statement against all of the list elements, without iterating. This will give us the ability to solve our Advent of Code problem with code that is not overly time-complex.

Pattern Matching List Elements: The Concept

In order to illustrate how we can use pattern matching in this way, let's focus on the goal of checking to see if the first element in our list, plus any other element in the list, equals 2020.

With Elixir's pattern matching, we can separate out list elements into a "head", i.e. the first element, and a "tail", i.e. everything after the first element.

Something like this:

iex> [head | tail] = [1, 2, 3]
iex> head
1
iex> tail
[2, 3]

Using this approach, we can match a variable to the first list element, the second list element, and then everything else like this:

iex> list = [979, 1721, 366, 299, 675, 1456]
[979, 1721, 366, 299, 675, 1456]
iex> [first | [second | rest] = tail] = list
[979, 1721, 366, 299, 675, 1456]
iex> first
979
iex> second
1721
iex> rest
[366, 299, 675, 1456]
iex> tail
[1721, 366, 299, 675, 1456]

In this way, we can establish a variable, first, set equal to the first list element, and another variable, second, bound to the value of the second list element.

Then, we can check if the sum of these two numbers equals 2020. If so, great! We're done.

If not, we can construct a new list using the same first element, and the list remainder stored in rest, cutting out the second element entirely.

Like this:

iex> new_list = [first | rest]
[979, 366, 299, 675, 1456]

Now, we can repeat the step above to see if the first element in the list plus the new second element in the list equals 2020:

iex> [first | [second | rest]] = new_list
[979, 366, 299, 675, 1456]
iex> first
979
iex> second
366

From here, we repeat the process. Does first + second == 2020? If so, great! We're done. If not...construct a new list using the same first element, and the list remainder stored in rest.

Eventually, if the first element cannot be added to any other list element to get the sum of 2020, then we end up with a list that contains only one element. We'll have cut out all the other elements until only the first element remains.

What do we want to do then? We want to revisit the initial starting list and pick up with the second element there. The tail of the original list will be a list that starts with the original list's second element.

In other words, if our original list read [979, 1721, 366, 299, 675, 1456], and we didn't find any other number added to 979 to equal 2020, then the tail of that list should read:

[1721, 366, 299, 675, 1456]

We already matched a variable, tail to the original list's tail above, like this:

iex> list = [979, 1721, 366, 299, 675, 1456]
[979, 1721, 366, 299, 675, 1456]
iex> tail
[1721, 366, 299, 675, 1456]

So, using the list stored in the tail variable, we can simply repeat the process described above.

Now that we have a basic understanding of what we need to do, let's write some code.

Pattern Matching List Elements: The Code

We'll define a module Accountant, that implements one public function, product_of_equals_twenty_twenty. This function will take in a list and return the product of the two numbers in the list that sum to 2020. The public interface of our module will work like this:

Let's begin implementing it now.

The function head will use pattern matching to pull out the tail of the list and save it for later in a variable, tail. Then it will pass the list to a helper function that is responsible for stepping through the process we described above. Let's revisit that process now by taking a closer look at get_two/1.

Pattern Matching Function Heads

We'll implement a few versions of the get_two/1 function that use pattern matching in the function head to determine how to behave. We'll also see recursion make a guest appearance. Let's take a look!

Let's examine each of the versions of this function. Some of this code should look familiar from our discussion of pattern matching list elements above. One function version pattern matches the first and second elements of the list and uses a guard clause to check if they sum to 2020. More on guard clauses in a bit.

If the first and second list element do sum to 2020, then the function body will execute and return the product of the two numbers.

If the first and second list element do not sum to 2020, i.e. if our guard clause does not evaluate to true, then we hit the next version of the function implementation:

Here, we build a new list constructed from the first list element and the remainder of that list, minus the second element. That new list is given as an argument to a recursive call to get_two/1. This will continue until we either hit the guard clause and return the product of the two elements. Or, until we have removed every element after the first one, resulting in a list with a length of 1. In that case, we will return nil:

So, we have a function that, when invoked with a given list, will call itself recursively until it finds two elements that sum to 2020--in which case it returns their product--or until there is only one list element left--in which case it returns nil.

By calling get_two/1 with the list [979, 1721, 366, 299, 675, 1456], we will have successfully checked to see if 979 plus any other number in the list equals 2020.

This brings us back to the public function, product_of_equals_twenty_twenty/1. If this first invocation of get_two/1 returns something that is not nil, then we're done! We found the product of the two numbers that sum to 2020. Let's implement some logic to that effect now:

If the first get_two/1 invocation does not return a number and does return nil, we need to start the whole process again, this time with the tail end of our original list.

This will restart the process, this time with a list that reads: [1721, 366, 299, 675, 1456]. Once again, our code will try to sum each number with the first list element, this time 1721. If it returns some product, then we're done! If not, we'll keep invoking product_of_equals_twenty_twenty/1 with the next list tail, and the next, until we find what we're looking for.

To wrap up this code, we'll add a function head for product_of_equals_twenty_twenty/1 that can handle being invoked with an empty list--that is what will happen if our list does not contain any two numbers that equal 2020 and we continue to invoke product_of_equals_twenty_twenty/1 until we have a tail that is empty.

Putting It All Together

Putting all together, our code reads:

Our code works, and it's highly efficient. We're never iterating over the list, never mind iterating over it in a nested fashion. Instead, we are recursively pulling the first and second elements off of a list, and shrinking the list each step of the way.

This looks pretty clean, but I think we can do even better. Anytime I see an if condition in Elixir, I wonder if I can replace it with recursion and pattern matching. Elixir allows us to combine recursion and pattern matching into an elegant solution for control flow. Could we implement product_of_equals_twenty_twenty/1 such that it can handle the case of a "found product"? Let's give it a shot!

Clean Control Flow with Pattern Matching and Recursion

We'll take a similar approach here to the one we used with our get_two/1 implementation. A set of function heads will use pattern matching to determine how to behave. One such function will leverage recursion to continue code flow, while other function heads will determine when the code will stop executing and return. In this way, Elixir pairs recursion and pattern matching to implement control flow--without if conditions or while loops.

Let's take a look.

We've changed the arity of product_of_equals_twenty_twenty to take in two arguments. The second argument will be the product of the two numbers that sum to 2020, and it will default to nil.

So, when our function is invoked with a list,

The product argument evaluates to nil, and we find ourselves in this version of our function:

Here, we kick off the process by recursively invoking product_of_equals_twenty_twenty/2 with the tail of the original list and the result of calling get_two/1 with the original list.

If get_two/1 returns a product that is not nil, then we'll find ourselves in this other version of the product_of_equals_twenty_twenty/1 function:

In which case, we break out of our recursive function calls, stop code execution, and return the product.

Let's step through this code in detail.

The user calls Accountant.product_of_equals_twenty_twenty([979, 1721, 366, 299, 675, 1456])

At this point, we hit this function,

where list evaluates to [979, 1721, 366, 299, 675, 1456], tail is set to [1721, 366, 299, 675, 1456] and product is nil.

get_two/1 is called with the list
Since 979 is not one of the numbers that sums to 2020, this call returns nil.
So, we call product_of_equals_twenty_twenty/2 with tail and nil
This brings us back to this same function body:

This time around, list is equal to [1721, 366, 299, 675, 1456], tail is [366, 299, 675, 1456] and product is still nil

get_two/1 is called with the list
Since 1721 plus 299does equal 2020, this will return the result of 1721 * 299, which is 514_579
So, we call product_of_equals_twenty_twenty/2 will tail and 514_579.
This brings us to this other function body:

So, we stop recursing, code execution is done, and the product is returned.

Phew! Seems like a fair amount of complexity, but when we put it altogether, we see some very clean and concise code.

In just about a dozen lines of code, we've implemented an efficient, iteration-free solution--all thanks to the beauty of Elixir's pattern matching. By pattern matching list elements, we were able to avoid expensive iterations. By using pattern matching in function heads, along with recursion, we were able to implement control flow that didn't rely on if conditions or while loops.

Before we go, we'll do just a bit more refactoring for readability with the help of custom guard clauses.

Refactoring with Custom Guard Clauses

Guard clauses allows us to apply more complex checks to pattern matching function heads. We're using a number of guard clauses throughout our code, for example:

Here, we've implemented a version of the get_two/1 function that will execute if the length of the provided list argument is equal to 1.

Guard clauses give us even more fine-grained control over which code to execute under which conditions. This is yet another way that we can handle complex control flow without verbose and hard-to-reason-about if and nested if conditions.

Only a certain set of expressions are allowed for usage in guard clauses, see docs here. But, we can define custom guard clauses to wrap up more complex guard logic. The guard clause we wrote to check if two numbers sum to 2020 is a great candidate for a custom guard clause. By wrapping up that logic in a custom guard clause, we can name the concept to make it easier to read and reason about.

To define a custom guard clause, we'll use the defguard macro:

Now, we can use our guard clause like this:

Custom guard clauses give us the ability to implement even complex control flow with pattern matching function heads, while keeping our code readable.

Putting it all together:

Elixir Encourages Efficient and Eloquent Code

By using Elixir's pattern matching against our list of numbers, we were able to write efficient code that avoided the time complexity of expensive nested iterations.

By using that same pattern matching feature, paired with guard clauses and recursion, we were able to implement control flow in a way that is 🧼 clean 🧼 and 🗣 eloquent 🗣. The code speaks for itself by being readable and easy to reason about. No messy, nested if conditions to deal with.

This Advent of Code challenge really shows off some of Elixir's simplest, but most powerful features.

Testing GenServers with Erlang Trace

Tue, 24 Feb 2026 18:56:27 +0000

When working on a messaging system with RabbitMQ for my upcoming workshop at ElixirConf, I ran into a common Elixir testing challenge--testing that a GenServer received a message. Thanks to lots of Googling and some help from a friend, I learned you can test that a GenServer received a message with the help of Erlang tracing.

The Problem

ExUnit provides an assert_receive/3, but that only allows you to check the mailbox of the current process, i.e. the process running the test. So, how can we check that a GenServer running in our application received a certain message? This is the issue I encountered when testing our RabbitMQ consumer GenServer.

We have a GenServer that runs when the application starts up and consumes messages from a RabbitMQ queue. How can we set an expectation that the consumer does in fact receive and process a given message sent by a publisher to that queue?

We can do exactly that with the help of ExUnit's start_supervised/2 callback and Erlang's trace/3 function.

Introducing Erlang Trace

Erlang's trace/3 function is pretty powerful. It allows us to attach a trace to a specified process. What does this mean? If we trace a given process, we are telling Erlang to send a message to a calling process (in our case, the test), whenever the trace process receives a message. Sneaky!

We'll use ExUnit's start_supervised/2 function to start our GenServer and capture its PID. Then, we'll use trace/3 to ensure that whenever the GenServer PID receives a message, our test gets notified. With that in place, we can assert that the test received a message from the trace, thereby testing that our GenServer received a message.

Let's do it!

Step 1. Start the GenServer with `start_supervised/2`

First, we'll start up our GenServer and capture its PID

Step 2: Start the trace

Next, we'll use Erlang's trace/3 function to trace the GenServer PID such that the test process receives a message whenever the GenServer PID does.

Step 3: Set the Assertion

Now we're ready to enact the code that we expect to result in our GenServer receiving a message--publishing to a RabbitMQ queue.

Publishes a message should cause our consumer GenServer to receive the :basic_consume_ok message. This will in turn send the following message to the test process which started the trace:

So, we can use ExUnit's assert_receive/3 function to assert that the test receives this message:

In this way, we can in fact test that our GenServer received a certain message.

Conclusion

Erlang's trace/3 function adds a powerful tool to our Elixir testing arsenal. It helps us solve a common testing problem--that of asserting that your GenServers received a certain message. Together with ExUnit's start_supervised/2 callback and assert_receive/3 function, we were able to write exactly the test we needed for our RabbitMQ messaging system.

Special Thanks

Special thanks to Steven Nuñez who turned me on to Erlang's trace/3 function and who generally gives me all of my ideas. Check out his recent post on managing RabbitMQ connections in Elixir with ExRabbitPool and learn more about working with RabbitMQ and Elixir by signing up for our ElixirConf 2020 workshop!

Roastidio.us Tagged with elixir

Dropping Cloudflare for bunny.net | jola.dev

Bunny.net

What I was moving away from

Step by step

Creating your pull zone

Configuring your pull zone

Configuring caching

Setting some sensible defaults

Conclusion

Newres Al Haider

Growing Yggdrasil, the World Tree, with Ash

An image of a Yggdrasil the world tree as a cybernetic Ash tree.

How Many Paradigms Does It Take to Screw In a Lightbulb?

Rails on the BEAM

What Changed

QuickBEAM

The Architecture

Action Cable, Both Sides

What the BEAM Adds

A Path to Phoenix

Try It

Our Journey: Building With Generative AI | Revelry

Some Background

Where We Started

Experimentation

Our First AI- Powered Toy: StoryBot

StoryBot Under the Hood

Looking a Bit More Into StoryBot Code

Behind the Scenes: Replacing a Black-Box Billing Service

The Challenge​

The Journey​

2021: Core System​

2023: Feature Parity​

2025: Admin UI, Support Tools, and Migration​

Powerful search​

Activity logs​

Support staff tooling​

Outcome​

Migrating 600k users to a new billing service

The backstory​

Introducing: Charger, the fraternal twin​

A bird watching experience 🔭​

Silent bugs​

Conclusion​

Building a blog with Elixir and Phoenix | jola.dev

Why Phoenix

NimblePublisher

The Frontend

CI

Deployment

The Little Things

RSS

Sitemap

Blog redirect plug

SEO

Wrapping Up

Software Development in 2026

Tests and Benchmarks

Context

Plans and Reasoning

Language

Politeness

Horizontal Scaling

YAGNIN, but YAGNIL

Divide et Impera

No Failures Despite Bugs

Software Development in 2023

BEAM Metrics in ClickHouse – Andrea Leopardi

High Cardinality Metrics

Enter ClickHouse

Table Design

Periodically Dumping System State

In the Real World

Frequent Fliers

Maximum Consumer Memory

Conclusion

Upgrading Amignosis: Phoenix and Elixir with Claude Code

Hiccups

Tidewave got stuck in a loop due to a huge prompt

The Challenge

The Journey

2021: Core System

2023: Feature Parity

2025: Admin UI, Support Tools, and Migration

Powerful search

Activity logs

Support staff tooling

Outcome

The backstory

Introducing: Charger, the fraternal twin

A bird watching experience 🔭

Silent bugs

Conclusion

Step 1. Start the GenServer with `start_supervised/2`