Roastidio.us Tagged with elixir

Privacy by Design no JurisOS: Criptografia AES-256 de Ponta a Ponta com Cloak.Ecto, SQLCipher e Tailscale VPN - DEV Community

Sat, 29 Aug 2026 15:13:10 +0000

O setor jurídico lida diariamente com dados de altíssima sensibilidade: laudos médicos, segredos industriais, estratégias de litígio, dados bancários e informações de identificação pessoal (PII). Desenvolver um sistema como o JurisOS — que opera sob o paradigma Local-First nos desktops de secretárias e advogados — exige incorporar o conceito de Privacy by Design desde a fundação da arquitetura.

Neste artigo, vamos detalhar como construímos uma malha de defesa em profundidade no ecossistema Elixir e BEAM, utilizando criptografia AES-256-GCM tanto em repouso quanto em trânsito, para garantir conformidade rigorosa com legislações de proteção de dados (LGPD, GDPR e CCPA).

1. O Desafio da Conformidade (LGPD, GDPR e CCPA)

As leis modernas de proteção de dados exigem que os sistemas garantam a confidencialidade e a integridade das informações pessoais contra vazamentos, sejam eles acidentais (perda de um notebook) ou maliciosos (ataques de exfiltração).

Para um aplicativo desktop Local-First sincronizado com a nuvem, a superfície de ataque é dupla:

O endpoint local: O disco do computador físico do usuário no escritório.
O transporte e o backend central: A rede por onde os dados trafegam e o banco de dados principal (PostgreSQL) na nuvem.

Para mitigar esses vetores, o JurisOS adota a Separação de Chaves (Key Isolation) em múltiplas camadas.

2. Separação de Chaves: SQLCipher e Cloak.Ecto

Não basta confiar apenas no acesso do usuário ao sistema operacional. Se um dispositivo for furtado, o banco de dados local não pode ser montado e lido por terceiros. Para resolver isso, separamos as responsabilidades de criptografia.

Camada 1: Criptografia de Disco com SQLCipher (`SQLCIPHER_KEY`)

Na ponta do cliente (o aplicativo desktop empacotado via Burrito), o armazenamento local não é um SQLite padrão, mas sim o SQLCipher. Todo o arquivo .db é cifrado transparentemente utilizando AES-256.
A SQLCIPHER_KEY é gerada no momento do provisionamento da máquina e injetada no ambiente de execução. Isso protege contra ataques físicos: a extração do arquivo do disco resulta em bits indecifráveis sem a chave simétrica correta.

Camada 2: Criptografia de Nível de Campo com Cloak.Ecto (`CLOAK_SECRET_KEY`)

Enquanto o SQLCipher protege o contêiner físico, o Cloak.Ecto protege o conteúdo lógico. Campos críticos que contêm PII (CPF, RG, históricos médicos) e a própria Transaction Outbox de sincronização recebem uma camada adicional de AES-256-GCM a nível de aplicação.

Isso significa que mesmo um administrador de banco de dados (DBA) acessando a instância central do PostgreSQL na AWS, ou um atacante que consiga realizar um SQL Injection, verá apenas ciphertexts binários, não os dados reais.

defmodule JurisOS.Vault do
  use Cloak.Vault, otp_app: :juris_os

  @impl true
  def init(config) do
    config =
      Keyword.put(config, :aes_gcm,
        tag: "AES.GCM.V1",
        key: decode_key(System.fetch_env!("CLOAK_SECRET_KEY"))
      )

    {:ok, config}
  end

  defp decode_key(key) do
    Base.decode64!(key)
  end
end

defmodule JurisOS.Clientes.EncryptedString do
  use Cloak.Ecto.String, vault: JurisOS.Vault
end

Na definição do Schema Ecto, mapeamos os campos sensíveis usando o nosso tipo customizado:

A CLOAK_SECRET_KEY fica estritamente na memória da BEAM. Os dados são cifrados antes de gerar o SQL de INSERT/UPDATE e decifrados logo após o SELECT, mantendo o banco de dados completamente cego quanto ao conteúdo dos campos.

3. Isolamento da Malha BEAM via Tailscale VPN

A comunicação entre os nós de desktop locais e o nó mestre na nuvem requer transporte seguro. Em vez de expor portas públicas na internet (0.0.0.0) e confiar apenas na camada TLS das requisições web, o JurisOS eleva a segurança encapsulando toda a infraestrutura em uma rede Zero Trust.

Utilizando a Tailscale VPN (baseada em WireGuard), cada estação de trabalho e o servidor AWS recebem um IP de uma sub-rede privada (ex: 100.64.0.0/10).

Os Benefícios dessa Arquitetura de Rede:

Superfície de Ataque Reduzida a Zero: O servidor na nuvem rejeita qualquer conexão que não venha de uma interface WireGuard. Não há portas expostas ao port scan público.
Comunicação Nativa Erlang Distribution: Podemos conectar os nós desktop ao cluster principal (libcluster) trafegando mensagens do ecossistema OTP através do túnel da Tailscale de forma totalmente segura.
Criptografia Dupla em Trânsito: Além da proteção nativa do protocolo (como WebSockets sobre WSS), os pacotes TCP/UDP são criptografados pelas chaves WireGuard ponto a ponto.

Conclusão

Proteger dados jurídicos requer mais do que apenas conformidade no papel; exige garantias algorítmicas de segurança.

Ao combinar o SQLCipher para proteção contra acesso físico indevido, o Cloak.Ecto para cegar o banco de dados central perante dados sensíveis (PII) e a Tailscale VPN para isolar completamente a comunicação de rede da internet pública, o JurisOS estabelece um estado da arte em Privacy by Design no ecossistema Elixir.

Essa arquitetura garante que, independentemente da ponta — seja no laptop offline do advogado ou no servidor AWS na nuvem —, a conformidade com a LGPD, GDPR e CCPA esteja tecnicamente assegurada desde o primeiro byte de dados inserido.

Egghead is a note-taking app. - by Mark Wunsch

Fri, 21 Aug 2026 01:08:22 +0000

Egghead is a note-taking app.

There are some who have already decided that the next interesting thing in software is going to call itself “AI-native” or “agentic” or “the cognitive operating system of the future.” Egghead is not those things. Or rather, it is some of those things but only as a consequence of being a note-taking app in 2026.

Generated by ChatGPT Images 2.0

The problem with notes

We take notes because notes outlast thinking. Anyone who has kept a notebook, either physical or digital, for any length of time has had the experience of writing something down only to later be unable to find it. You know the note is there, somewhere, but you can’t, for the life of you, recall where it was or what it said.

This central challenge of keeping notes gets worse with scale. A single notebook is searchable by hand. Ten notebooks are significantly more challenging. A folder of digital notes can be searched, but only if you remember the exact words you used, which you usually don’t because the whole reason you wrote the note in the first place was to externalize the thought so you could stop holding it. The note is supposed to do the remembering for you. Instead it just changed where the forgetting occurs.

The work to create a system of note-taking has lasted nearly as long as the act of note-taking itself. The Renaissance period had commonplace books. The Index Card was popularized by Carl Linnaeus — a guy with strong opinions about structured information. In the 20th Century an obscure German sociologist named Niklas Luhmann took his note-taking seriously enough to build a card catalog of nearly ninety thousand interlinked notes, which he then used to write more than seventy books and nearly four hundred scholarly articles. The system he extensively used was Zettelkasten, which became a subject of his own research into systems theory and prefigured the Wiki.

The 21st century has produced an entire category of software — Evernote, Obsidian, Notion, Roam, Bear, Apple Notes, Logseq — to name just a handful that popped into my head. Each one promising that this time, the notes will stay findable. I know I am not alone in having tried more than one of them, and sticking with it for a nontrivial amount of time before finding another shiny object promising untold cognitive reward.

They all work, but they all suffer from the same limitations. No matter how good the search, or how clever the linking, or how disciplined you are about tagging, the system can only ever give back what you put in. The smartest thing in the room is still you.

This isn’t a failure of any specific tool, but a structural property of the whole category. The limit on what you can do with a notebook is your own memory of what’s in it. Which is, if you’re like me, not very good. Which is why I started writing things down in the first place.

How to write good notes

There is a self-help sub-genre in your nearest global online bookstore dedicated to note-taking systems, and though I have a personal perspective of what makes a good system of notes, I find the more critical thing to answer is where the notes are and how they are made available to you.

The best answer, as of this writing, is the same answer as it was fifty years ago: a series of files written in plain text on your storage disk. Many popular note-taking software applications tend to use proprietary formats in proprietary databases, accessed only through said proprietary application. Plain text files are the computer world’s universal interface, and are a core pillar of the Unix philosophy.

For Egghead, the notes are assumed to live as files in a directory formatted as either Markdown or Org Mode using Wikilinks to denote connections between them. When we circumscribe our written notes to proprietary formats, we reduce our ability to retrieve them. Which as stated earlier, is already challenging enough due to the limitations of our own memory. Plain text first.

Why Artificial Intelligence

For most of recent history, the limitations of software notebooks were reflections of the limitations of physical notebooks: limitations of the human operator. Improvements in metadata, indexing, and search are still fundamentally bound by the user: you can only search for a word that has been explicitly written, and traverse relationships over metadata that the user has embedded. Software could not, in any meaningful sense, build a notebook that read what you wrote and engaged with it as a participant.

Large language models change this. Not because they are intelligent in any deep sense, but because they can read more text than I can hold in my head, and they can produce reasonable language about that text on demand. That is genuinely new.

So the obvious move is to point an AI assistant at your notes and let it go to town.

The problems with AI in knowledge management

This is what the current generation of AI products is doing. Uploading files to ChatGPT, creating project knowledge in Claude, Notion AI on top of your Notion workspace, Cursor in your codebase… They all share the same shape: there is a knowledge base over here and a single AI assistant over there and the assistant can occasionally reach over to read from the knowledge base before answering your question.

The most ambitious version of this shape, and the one that pushed me to build something different, is OpenClaw. An open-source personal AI assistant that you can run on your own machine, talk to from any messaging app, and connect to your files and tools. It’s genuinely good.

But OpenClaw directly demonstrates the limitations of a single AI assistant: it agrees with you.

It has to. There is no structural pressure on it to do otherwise. When you ask it a question, it will give you an answer shaped like the question. You ask a leading question and it follows your lead. The “You’re absolutely right!” reflex is both well-trodden joke material as well as real architectural fact: a single agent, optimizing locally for “be helpful” will reliably converge toward whatever the user seems to want to hear. This is the shape of one-on-one assistance.

The memory of AI assistants

What happens after a conversation with an AI assistant ends? Increasingly, they remember things from conversation to conversation (which wasn’t always the case). OpenClaw, for example, has a MEMORY.md and workspace for persistence.

The dominant pattern for memory in AI tooling is some flavor of retrieval-augmented generation (RAG). Notes and past conversations get chunked into passages, embedded into vectors, and stored. On each new prompt, the harness pulls the chunks that look semantically nearest to the question and stuffs them into the model’s context window.

In April, a more ambitious variant of the same impulse was articulated in Andrej Karpathy’s LLM Wiki pattern, where instead of retrieving from raw sources at query time, an LLM agent incrementally compiles your notes into a structured wiki and then queries that.

Both of these patterns exhibit the same flaw. In RAG, retrieval is shaped by the prompt and the prompt is shaped by you.

RAG works on chunks, not documents — your essay’s argument structure is gone before the agent ever sees it. In the LLM Wiki version, the same bias gets baked in earlier: by the time the wiki entry exists, it’s been passed through the agent’s filter. The summary is cleaner than the source. That’s what makes it a really compelling “memory” product, but not a great note-taking product. Ambiguity in your notes gets edited toward coherence.

So both the shape of a single AI assistant and its memory formation produce a thinking partner who is very capable at creating a confident, well-organized version of what you’ve already decided you wanted to hear.

This is exactly the limitation of human users that leads many to abandon note-taking or continuously migrate from one note-taking system to the next. The notebook can only return what you remember to retrieve. The single AI assistant with RAG memory can only return what your prompt vocabulary aims at. So the assistant-plus-notes shape, even with modern persistence, has two failure modes that compound: a single agent cannot disagree with itself in any structurally reliable way, and a single agent’s memory pulls toward the framing you walked in with. Both failures point at the same fix.

You need more than one, and they need to share notes.

Multi-Agent Systems

The intuition is straightforward and very human: Teams beat individuals on most kinds of knowledge work, especially the kind where the failure mode is groupthink rather than skill gap. Peer review beats self-review. Code review beats no code review. We already know this about humans. We already build institutions around it. A single perspective on its own work has a structural blind spot and the cure is more perspectives.

Apply that to the notebook problem. If a single agent has a tendency to agree with you, the fix is not to find a better single agent. The fix is to put a second agent in the room with a different disposition, and let them disagree with each other where you can watch. The peer review you’d want from a thoughtful colleague, manufactured at the structural level, in real time, on the body of work you actually care about.

Many orchestration frameworks have arrived to multi-agent systems by a different route, but their shape does not lend itself to inherently better results. They default to coordinator and specialists — one agent dispatches tasks, others execute, and results aggregate at the top. This is a star topology — an org-chart fantasy of how teams work.

The Linux kernel mailing list does not have a dispatcher delegating tasks for execution. An organization that adopts Slack does not have an executive function adjudicating every channel message. Even in environments where you would expect a rigid hierarchy — like a nuclear submarine — the strict leader-follower model doesn’t always produce the best outcomes. In my own personal experience working in software teams, the teams that work are ones where the coordination is light and the communication is dense.

The recent MAS research bears this out — graph topologies, where any agent can talk to any other agent, outperform star and tree shapes on collaborative tasks.

The capabilities of agents

Once you commit to a team full of agents working in a shared knowledge store, a security and a quality problem emerge that single-agent systems can (and frequently do) ignore.

“How much should this agent be able to access and perform?” is both a security question and a quality question. It’s a security question because, as has been borne out, agents can and will leak internal secrets to external places or perform destructive actions. More agents means more potential for leakage. It’s a quality question because agents that can do everything tend to converge. Without distinct role definitions, the disagreement that made a multi-agent architecture useful in the first place dissolves into consensus. Capability scoping is the structural pressure that keeps the room from collapsing into agreement.

This is the principle of least privilege applied to agentic systems. The same reason I, as adjunct faculty, do not have the keys to Columbia University’s Network Operations Center. Least privilege improves the quality of the output, because it preserves the structural diversity that makes a team perform better than an individual.

What is Egghead?

Egghead is a note-taking app.

The notes are written as plain-text files in a folder you own.

It uses a loosely-coordinated group of AI agents to continuously read from and contribute to the total knowledge base, producing the most diverse set of inferences about that knowledge by allowing the user to grant each agent a distinct set of capabilities.

Every agent is itself defined as a note, as are the transcripts of their conversations and individual deliberations, giving each note provenance for its creation.

The notes are written as plain text in the filesystem, and the app exposes this and communication with its agents through as many surfaces as possible — the terminal, the web, MCP, and IRC — so that your knowledge is not locked behind any one of them.

The goal of taking notes in Egghead

It is worth restating, we take notes because notes outlast thinking. Simply stated, the goal is to extend our thoughts beyond the storage and time limitations of our own wetware.

The goal is not productivity. The goal is not to get more done. The goal is not to have your meetings summarized, or your emails triaged, or your tasks auto-prioritized. Those are nice. They are not the point. The point — the actual, unfashionable, embarrassing-to-say-in-a-funding-pitch point — is to get smarter. To retain more of what you read. To do more with what you retain. To engage with your own past thinking.

Every “AI assistant” on the market is a productivity tool, by which I mean a tool whose purpose is to enable you to do less of a thing and produce more output. A note-taking app built on this premise would exist to reduce the time you spend with your notes. I want the opposite. I want a tool that makes you spend more time with your notes. That makes the doing of it more rewarding.

Knowledge itself is the outcome. The goal of the practice is – to use a word that has fallen out of fashion in software but used to be considered the most valuable thing a thinking person could accumulate – Wisdom.

Ipsa cognitio fructus

Using Elixir head version with Mise

Sun, 16 Aug 2026 10:18:59 +0000

In 2025 I followed the direction many people headed in and switched my version manager to mise-en-place. It has been going really well for me, but recently I came across a problem which took me a bit to figure out (and few very frustrating conversations with LLM, which did not lead to anything). I wanted to test some code with unreleased version 1.20 of Elixir, straight from git.

How to do it? It’s really simple in retrospect. Here are the steps:

1. Build Elixir

$ git clone https://github.com/elixir-lang/elixir.git 
$ cd elixir
$ make

Verify it’s built correctly:

$ bin/elixir --version
> Elixir 1.20.0-dev (88cbabf) (compiled with Erlang/OTP 28)

2. Link it in mise

mise link elixir@head /path/to/elixir

3. Verify this works

$ mise x elixir@head -- elixir --version
> Elixir 1.20.0-dev (88cbabf) (compiled with Erlang/OTP 28)

And it’s done. Whenever you want to run something with Elixir head, you can just mise use elixir@head.

Of course, this should work in a similar way for everything that is handles by mise version control.

Ecto, on_replace and deferred checks

Sun, 16 Aug 2026 10:18:59 +0000

Today I learned a valuable lesson about how a seemingly simple task can have very rough edge cases, which take hours to solve. It involved Ecto, its associations and on_replace option, and uniqueness checks in the database. Here’s the story.

The problem

Let’s say you are modelling some kind of processes. These processes have steps and the steps have to be executed in a precise order. This is how a database structure for it would look like:

defchangedocreatetable(:processes)doadd:name,:string,null:falseendcreatetable(:steps)doadd:process_id,references(:processes)add:name,:string,null:falseadd:order,:integer,null:falseendend

It is fairly straightforward: the order of steps inside of a process is controlled by an order integer column. Since it is important to always have the order of steps precise, we would like to additionally ensure it by a uniqie index:

createunique_index(:steps,[:process_id,:order])

We also know that the steps are only edited via a parent process. There is a form in the application, where you can edit the process fields and add, change, delete steps for it. The payload sent to a server always includes all the steps. Armed with that knowledge we create Ecto schemas like this:

defmoduleCore.ProcessdouseEcto.SchemaimportEcto.Changesetschema"processes"dofield:name,:stringhas_many:steps,Core.Step,on_replace::delete,preload_order:[asc::order]enddefchangeset(process,params)doprocess|>cast(params,[:name])|>cast_assoc(:steps)endenddefmoduleCore.StepdouseEcto.SchemaimportEcto.Changesetschema"steps"dofield:name,:stringfield:order,:integerbelongs_to:process,Core.Processenddefchangeset(step,params)dostep|>cast(params,[:name,:order])endend

I did that, quite happily, tested a bit with adding, changing and removing some steps. Finally I merged the pull request.

It was only a few hours later when a colleague slacked me that something goes wrong. What he did was reordering the steps, or rather trying to do it. From this structure:

step 1, order: 1
step 2, order: 2

he wanted to go to

step 1, order: 2
step 2, order: 1

I quickly drafted code that replicated the issue:

proc=%Process{}|>Process.changeset(%{name:"Test",steps:[%{order:1,name:"First"},%{order:2,name:"Second"}]})|>Repo.insert!()proc=Repo.preload(proc,[:steps])[s1,s2]=proc.stepss1=%{id:s1.id,name:s1.name,order:2}s2=%{id:s2.id,name:s2.name,order:1}proc|>Process.changeset(%{steps:[s1,s2]})|>Repo.update!()

Attempt to run it resulted in an error:

** (Ecto.ConstraintError) constraint error when attempting to update struct:

    * "steps_process_id_order_index" (unique_constraint)

It took me a while to realize what’s going on: Ecto tried to update the existing steps, one by one. So the first operation was:

Update “order” of step 1 to 2

This left us with the following situation:

step 1, order: 2
step 2, order: 2

This, quite obviously in retrospect, triggered the unique index. You cannot have 2 steps for one process with the same order! It did not matter that right after you were going to update step 2’s order to 1. The index operates here and now, you cannot do that.

The solution

After discovering I was pretty close to writing my own hacky solution to update the steps, including inefficient approach with first deleting all and then creating all anew. Fortunately, in time, I vaguely remembered that there is something called deferrable constraints in PostgreSQL.

Deferrable constraint waits until the end of the transaction with checking if its condition is met. This was exactly what I was looking for! But an index cannot be deferrable in PostgreSQL. Luckily, another very similar construct - a uniqueness check - can.

In the migration above, I had to replace the index creation with this:

execute"""
ALTER TABLE steps ADD CONSTRAINT unique_step UNIQUE (process_id, "order") DEFERRABLE INITIALLY DEFERRED
"""

With this statement we create a constraint on the steps table which checks uniqueness of two columns and not only is deferrable - it is deferred to be checked on transaction commit by default.

This is really all I had to do. The Ecto’s association replacing mechanism started working perfectly.

This proved once again a truth I knew: the database is not a dumb storage you can just gloss over. It always pays off to learn it, its capabilities and intricacies. Because then you end up with a simple change instead of rewriting part of Ecto, but poorly.

the phoenix template has too much stuff

Mon, 10 Aug 2026 19:36:52 +0000

So, I'm starting a new project. I want to get this blog setup to send and receive Webmentions, so that I can do some fun social stuff here. Comments, linking to other people talking about what I write, that sorta thing. I started out by trying to implement it into the blogging engine itself, but I felt like I was running up against the design of Astro (the framework powering this place) trying to squeeze a public API and a message queue into what's basically just an on-demand static site generator. So, I'm gonna try write an external server instead.

My language of choice for building web servers lately has been Elixir. It's a decently nice language with a crazy powerful runtime - Erlang and the OTP do not fuck around. Usually I prefer to write servers from scratch, since it keeps the initial architecture nice and small, while giving me room to expand when I need some other particular feature. This time around though, I know I'm going to want a few things right off the bat:

Some sort of worker pool, so processing Webmentions can be done asynchronously without blocking the rest of the server
A database to hold records of all the Webmentions that come my way
Some HTML rendering so I can have status pages for senders to see if their request was processed, and also an admin panel so I can easily review incoming Webmentions

At this point, I think we're starting to get complex enough that I'd benefit from bringing in Phoenix, the de facto standard web app framework for Elixir. It's got most of this loaded in and ready to go (LiveView in particular sounds great for my admin panel stuff), so I figure it's probably smart to start off with it.

So, I make a new Phoenix project. Easy enough - docs say:

mix archive.install phx_newmix phx.new project_name

So I run it, and...

* creating project_name/lib/project_name/application.ex
* creating project_name/lib/project_name.ex
* creating project_name/lib/project_name_web/controllers/error_json.ex
* creating project_name/lib/project_name_web/endpoint.ex
* creating project_name/lib/project_name_web/router.ex
* creating project_name/lib/project_name_web/telemetry.ex
* creating project_name/lib/project_name_web.ex
* creating project_name/mix.exs
* creating project_name/README.md
* creating project_name/.formatter.exs
* creating project_name/.gitignore
* creating project_name/test/support/conn_case.ex
* creating project_name/test/test_helper.exs
* creating project_name/test/project_name_web/controllers/error_json_test.exs
* creating project_name/lib/project_name/repo.ex
* creating project_name/priv/repo/migrations/.formatter.exs
* creating project_name/priv/repo/seeds.exs
* creating project_name/test/support/data_case.ex
* creating project_name/lib/project_name_web/controllers/error_html.ex
* creating project_name/test/project_name_web/controllers/error_html_test.exs
* creating project_name/lib/project_name_web/components/core_components.ex
* creating project_name/lib/project_name_web/controllers/page_controller.ex
* creating project_name/lib/project_name_web/controllers/page_html.ex
* creating project_name/lib/project_name_web/controllers/page_html/home.html.heex
* creating project_name/test/project_name_web/controllers/page_controller_test.exs
* creating project_name/lib/project_name_web/components/layouts/root.html.heex
* creating project_name/lib/project_name_web/components/layouts/app.html.heex
* creating project_name/lib/project_name_web/components/layouts.ex
* creating project_name/priv/static/images/logo.svg
* creating project_name/lib/project_name/mailer.ex
* creating project_name/lib/project_name_web/gettext.ex
* creating project_name/priv/gettext/en/LC_MESSAGES/errors.po
* creating project_name/priv/gettext/errors.pot
* creating project_name/priv/static/robots.txt
* creating project_name/priv/static/favicon.ico
* creating project_name/assets/js/app.js
* creating project_name/assets/vendor/topbar.js
* creating project_name/assets/css/app.css
* creating project_name/assets/tailwind.config.js

...that's a lot of stuff. What is all this?

# Configures the mailer## By default it uses the "Local" adapter which stores the emails# locally. You can see the emails in your browser, at "/dev/mailbox".## For production it's recommended to configure a different adapter# at the `config/runtime.exs`.config :project_name, ProjectName.Mailer, adapter: Swoosh.Adapters.Local

What

# Configure esbuild (the version is required)config :esbuild,  version: "0.17.11",  project_name: [    args:      ~w(js/app.js --bundle --target=es2017 --outdir=../priv/static/assets --external:/fonts/* --external:/images/*),    cd: Path.expand("../assets", __DIR__),    env: %{"NODE_PATH" => Path.expand("../deps", __DIR__)}  ]

No, stop

# Configure tailwind (the version is required)config :tailwind,  version: "3.4.3",  project_name: [    args: ~w(      --config=tailwind.config.js      --input=css/app.css      --output=../priv/static/assets/app.css    ),    cd: Path.expand("../assets", __DIR__)  ]

PLEASE

# Specifies your project dependencies.## Type `mix help deps` for examples and options.defp deps do  [    {:phoenix, "~> 1.7.21"},    {:phoenix_ecto, "~> 4.5"},    {:ecto_sql, "~> 3.10"},    {:postgrex, ">= 0.0.0"},    {:phoenix_html, "~> 4.1"},    {:phoenix_live_reload, "~> 1.2", only: :dev},    {:phoenix_live_view, "~> 1.0"},    {:floki, ">= 0.30.0", only: :test},    {:phoenix_live_dashboard, "~> 0.8.3"},    {:esbuild, "~> 0.8", runtime: Mix.env() == :dev},    {:tailwind, "~> 0.2.0", runtime: Mix.env() == :dev},    {:heroicons,     github: "tailwindlabs/heroicons",     tag: "v2.1.1",     sparse: "optimized",     app: false,     compile: false,     depth: 1},    {:swoosh, "~> 1.5"},    {:finch, "~> 0.13"},    {:telemetry_metrics, "~> 1.0"},    {:telemetry_poller, "~> 1.0"},    {:gettext, "~> 0.26"},    {:jason, "~> 1.2"},    {:dns_cluster, "~> 0.1.1"},    {:bandit, "~> 1.5"}  ]end

OH GOD WHY ARE THERE SO MANY DEPENDENCIES AAAAAAA

Seriously, who needs all this??? Why does the starter template come preloaded with a fucking library for sending emails? This isn't basic starter stuff! I don't need bloody Tailwind to write HTML templates! There's just so much stuff in this starter template - I probably need like, half of it? I thought I would just get an Elixir project with the Phoenix dependencies ready to go, and maybe a simple skeleton with a one-page "Welcome to Phoenix!" included, not the entire god damn kitchen sink? Why? WHY is there so much here??? Aaaaaah!!!

deep breath in

rm -rf project_namemix new project_name

exhale. There we go. That's better. Time to see how possible it is to set this up from scratch.

Not that hard? Nice. My sanity survives another day.

Anti-Antipatterns in Elixir Library Guidelines

Sun, 09 Aug 2026 20:09:44 +0000

Only a lazy person never blogged about library should not use application configuration. Even the official propaganda says so in the Elixir Library Guidelines.

As all the sharp rules, this one is rotten. There are usecases when the library configuration must prevail over local parameters.

In fact, the entire “Anti-patterns” section of the official library guidelines reads like a list of commandments handed down from Mount Sinai, written for developers who cannot be trusted with sharp tools. But software engineering is not about blindly following rigid rules; it is about choosing the right trade-offs.

Let’s walk through all nine anti-patterns declared in the official guidelines and show why—and when—each so-called “anti-pattern” is not just permissible, but absolutely necessary.

1. Avoid Application Configuration

The guidelines state that libraries must never rely on Application.get_env/2 because global state makes it impossible for two dependencies to use the library in different ways. Instead, we are told to pass explicit keyword options down every function call.

Consider an application dealing with markdown here and there. Its dependencies parse markdown too. Core parses markdown, web parses markdown, several helpers under umbrella do parse markdown.

How do I add the new plugin everywhere? Easy, if there is an application configuration. And …ehrm… there is a way to patch dependencies.

If you enforce passing opts explicitly through forty layers of call stacks across fifteen sub-applications in an umbrella project, you guarantee that every intermediate module becomes a glorified passthrough for configuration keys it doesn’t care about. Application configuration allows the system operator to define system-wide defaults once—in config/config.exs—while still allowing individual calls to override them when necessary. Global defaults are a feature, not a bug, when your application needs uniform behavior by default.

2. Avoid Compile-Time Application Configuration

We are warned against reading the application environment in module attributes, such as @http_client Application.fetch_env!(...), because it burns values into the compiled binary at compile time rather than reading them at runtime.

This advice conveniently ignores two crucial requirements: zero-cost abstractions and compile-time code generation.

When building high-throughput data pipelines, parsers, or logging frameworks, invoking Application.get_env/3 inside a hot loop running millions of times per second introduces unnecessary dictionary lookups. Statically baking configured modules or constants into module attributes yields zero-overhead dispatch.

Furthermore, if your library uses macros to generate pattern-matching clauses or AST structures based on configuration (e.g., compiling custom sigils or state machine transitions dynamically), runtime lookup is literally impossible. The AST must be constructed at compile time. Compile-time configuration is the bridge between user configuration and macro expansion.

3. Avoid Using Exceptions for Control-Flow

“Use {:ok, result} and {:error, reason} tuples everywhere,” says the guide. “Never use exceptions or raise/throw for control flow.”

This sounds clean in trivial examples, but consider a deeply nested AST traversal or a recursive tree parser fifteen levels deep. If an invalid node is encountered on level 14, propagating {:error, reason} back up through 14 stack frames requires wrapping every single recursive step in a with block or manual tuple matching. The core algorithm becomes buried under an avalanche of error-plumbing boilerplate.

In complex recursive operations, using throw/1 and catch (or raising a dedicated structural exception rescued at the top-level boundary) acts as a clean, non-local exit. It unwinds the stack immediately back to the public API boundary, leaving the internal recursive code concise, readable, and fast. The caller still receives a clean {:error, reason} tuple at the boundary, but the internal code doesn’t suffer from tuple-wrapping tax.

Even for comprehension might take advantage from throwing in the middle if something went south.

4. Avoid Working with Invalid Data

The guideline advises validating all data immediately at the entry boundary using pattern matching and guards, refusing to operate on anything that doesn’t strictly match expected types.

This principle breaks down when dealing with streaming data ingestion, resilient ETL pipelines, and progressive parsing of semi-structured data.

Suppose you are building a log processor or IoT event consumer that ingests gigabytes of heterogeneous payloads. If you strictly validate the entire schema at the entry boundary, a single malformed key or unexpected field type forces you to drop the whole batch or crash the pipeline.

Instead, accepting loose structures at the boundary and deferring validation to downstream processing stages allows your system to handle partial data, quarantine bad sub-fields, and log warnings without discarding valid payload portions. Late validation and resilient boundary accepting are fundamental to building fault-tolerant data pipelines.

5. Avoid Defining Modules Outside Your Namespace

The rules demand that every module in a library be prefixed with the library’s top-level module name (e.g., MyLib.Foo), warning that defining top-level modules or extending another library’s namespace causes collisions on the Erlang VM.

Yet, this rule completely ignores drop-in polyfills, standard protocol implementations, and seamless stdlib integration.

If you are writing a drop-in replacement library or a compatibility layer (for instance, a library providing backwards compatibility for a deprecated OTP module or patching telemetry integration like Telemetria), placing your modules in the expected target namespace is the entire point. It allows existing applications to use your library without modifying hundreds of call sites across an entire codebase.

Similarly, when implementing protocol consolidations or global dispatchers that third-party plugins hook into, adhering strictly to arbitrary prefixing can make macro-based registration unwieldy. When done intentionally, target-namespaced modules are a legitimate pattern for drop-in interoperability.

6. Avoid `use` When an `import` Is Enough

“Do not provide use MyLib if all it does is import MyLib,” reads the guide. “Prefer alias, then import, and use use only as a last resort.”

This is classic short-term thinking that ruins library API evolution.

Suppose version 1.0 of your library only needs to import helper functions into the target module. You instruct users to put import MyLib in their code. Six months later, in version 2.0, your library needs to register module attributes, inject a @before_compile hook, or set up telemetry callbacks.

Because you chose import MyLib to satisfy a dogmatic guideline, every single user of your library must now go through their codebase and replace import MyLib with use MyLib.

By offering use MyLib from day one—even if its initial __using__/1 implementation is just an import—you establish an extensible contract. You reserve the right to add compile-time hooks, behaviours, and setup logic in future releases without breaking a single line of client code.

7. Avoid Macros

“Macros are harder to write… clear code is better than concise code… macros should only be used as a last resort.”

If the Elixir core team took this rule seriously, we wouldn’t have Ecto schemas, Phoenix routers, ExUnit tests. If I did, we wouldn’t have my libraries at all. If Finitomata can be probably rewritten in non-macro approach, Telemetría has zero chance to ever exist without all the spectre of macros, from use to __before_compile__ hooks and whatnot.

Trying to define complex domain logic using raw functions, maps, and anonymous function passing results in an unreadable wall of syntactical noise. Macros allow us to build expressive, declarative Domain-Specific Languages (DSLs). They perform compile-time validation, optimize data structures before execution, and turn complex domain semantics into intuitive code.

Without macros, Elixir would just be Erlang with different syntax. Embracing macros responsibly is what makes Elixir libraries powerful.

8. Avoid Using Processes for Code Organization

The guidelines argue that processes must only model runtime properties (state, concurrency, fault tolerance) and never code organization, using the classic strawman of putting a basic calculator behind a GenServer.

While putting arithmetic in a GenServer is obviously silly, using processes as fault-isolation boundaries for computation is a battle-tested OTP pattern.

Consider a library executing third-party NIFs, parsing untrusted image files, or running heavy memory-intensive computations. If you run that code directly inside the caller’s process and it triggers an out-of-memory error, segfaults, or enters an infinite loop, it takes down the caller.

Wrapping that “pure computation” inside a dedicated worker process isolates the failure. If the computation crashes, only the worker process dies; the caller receives an error tuple and survives. Processes are not just for state—they are hard memory and fault-containment zones.

9. Avoid Spawning Unsupervised Processes

Finally, the guidelines insist that every process must live inside a supervision tree, warning against raw spawn/1 or unsupervised tasks.

This ignores fire-and-forget side-effects where coupling the task lifetime to the caller or supervisor is undesirable.

Take non-critical background operations: sending an asynchronous telemetry event, flushing an audit log, or dispatching a best-effort metric. If you start these tasks under the main caller’s supervision tree, a failure in the telemetry reporter can crash the supervisor or block the shutdown sequence of the primary application.

Spawning an unlinked, unsupervised task (or using a detached process) guarantees that the background side-effect runs independently on a best-effort basis without ever delaying, blocking, or crashing the primary business logic.

Summary

The official Elixir library guidelines are a useful set of defaults for beginners, but defaults are not universal laws.

Every single “anti-pattern” in the guidelines exists because someone, at some point, needed to solve a real engineering problem that the “clean” abstraction couldn’t handle. Understanding why a rule exists gives you the authority to break it when the trade-offs demand it.

Don’t let dogma dictate your architecture. Use application configuration when you need global defaults, use macros when you need expressive DSLs, use processes when you need fault isolation, and use the full power of the Erlang VM when your application calls for it.

kmx.io blog : KC3 v0.1.17 released

Mon, 03 Aug 2026 17:51:50 +0000

2026-08-02 20:44:18

KC3 v0.1.17 released

KC3 v0.1.17 was released today.

New in this release :

libkc3
- copying temporary variables for pthread_mutex and pthread_rwlock would cause undefined behaviour as per POSIX, this fixes pthread on macOS
- allow for log hooks (a C function callback with a user pointer) in the facts database
- incremental compilation with cached parser results in .kc3c files, like Python does. Gives 5x faster loading times for all .kc3 files. env_load automatically handles this.
- fixed a bug in ht_iterator_next where the iterator would not go through the first collision list
- database logging now supports and defaults to binary format (marshall + marshall_read)
- fixed marshall + marshall_read hash table usage
- str: fixed display of floating point numbers
Facts
- Facts.connect/accept allows for bi-directional synchronization of an existing Facts.database() over a TLS encrypted connection after a successful HMAC-SHA256 shared secret authentication challenge/response
- Facts.accept accepts connections one by one
- Facts.acceptor_loop starts a thread that calls accept() in a loop
- Facts.acceptor_loop_join() stops the acceptor loop cleanly
- Fixed hash table lookup in marshall reducing every dump by 30%
HTTPd
- allow for configuration of unveil paths in config/unveil.kc3
- allow for custom log messages in error and request log
HTTPS
- HTTPS.Client with libtls and automatic or manual connection
  - GET method
  - POST method
  - JSON response
JSON
- fixed parser
  - boolean true or false → Bool
  - map {"key", "value"} → Map %{"key" => "value"}
  - array [1, 2, 3] → List [1, 2, 3]
RUsage
- current-process statistics for user and system CPU time, current and maximum resident memory, virtual memory, and open files
- RUsage.stats() returns all statistics in a single map

The notification that gave up after a minute | jvalol

Mon, 20 Jul 2026 00:43:15 +0000

The notification that gave up after a minute

Projects

Popsicleboat, Web-Push

July 17, 2026

PopsicleBoat grew browser push notifications recently. The pitch is simple: hear when someone answers you, without handing anyone an email address. Your browser asks permission, a subscription goes to the server, and nothing personal leaves the device. I shipped it, watched the push service accept my first delivery with a tidy 201 Created, and felt good about the whole thing.

Then I replied to one of my own posts from my phone, looked at my laptop, and saw nothing.

No banner. No chime. The reply was on the site, the inbox badge lit up, the notification email arrived — every channel working except the one I’d just built. And here’s what made it a good puzzle: every layer of the push stack, inspected on its own, reported success.

The server sent the notification. The push service accepted it. The subscription was in the database. The service worker was registered. Four green lights, zero banners.

Web push is a relay race with four runners: your server signs a message and hands it to a push service (Google’s, for Chromium browsers), the push service holds it for the browser, the browser wakes a service worker, and the service worker asks the operating system to draw a banner. A dropped baton anywhere shows up the same way — silence at the finish line — and no runner files a report.

The first dropped baton was mine to find in the browser console. Notification.permission said "default". Not "granted", not "denied" — never asked. Somewhere between enabling notifications and testing them, the site’s permission had ended up back at square one — a site-data sweep, probably — while the subscription it had authorized lived on in my database, perfectly valid, pointing at a browser that would no longer show anything. In that state, a page calling new Notification() doesn’t error. It does nothing, silently, which is a bold choice for an API whose entire job is being noticed.

One re-grant later (“forever,” this time), banners worked. Victory lasted about an hour, until I noticed pushes still vanished whenever the browser wasn’t running.

That was the second baton, and it was hiding in a default I’d never questioned. Push services will happily hold a message for a browser that’s closed — that’s the whole point of the relay — but only as long as the message’s time-to-live allows. The library I use sets that TTL to sixty seconds unless told otherwise. Sixty seconds. Close your laptop, get a reply two minutes later, and the push service shrugs and discards it. The 201 it returned was entirely honest: message accepted. Nobody promised kept.

The fix is one option:

send_notification(payload, message, ttl: 86_400)

A day. That’s roughly how long “someone answered you” stays worth interrupting someone for; any older and it’s the inbox’s job, which never forgets.

Two lessons sailed home with this one. First: in a layered system where every layer answers “OK,” the bug lives in the gaps between the layers — the permission that reverted between grant and use, the message that expired between accepted and delivered. Debugging meant walking the relay in order and asking each runner not “did you succeed?” but “what did you hand the next runner, and did anyone catch it?”

Second: defaults are decisions someone else made about your product. A conservative browser permission model, a sixty-second TTL — both defensible choices, by people who’d never seen my notification settings page and its promise to hear when someone answers you. Keeping that promise meant finding every default standing between the reply and the banner, and overruling the ones that disagreed with it.

The banners arrive now. Even the morning after.

Notes from the dead letter box - Specifying Software for Teams and Agents

Sun, 05 Jul 2026 12:34:08 +0000

How to describe an application?

Imagine the following: The dev team receives the envelope with the application specs from a dead letter box in the park at night. Two months later, they deliver the code. Their client deploys their app, and everything works as expected.

The question: What was in that envelope?

Product managers, engineers, and procurement teams have tried to deliver this perfect blueprint for a software product. The ones that did not fail have mostly succeeded by sheer luck. After decades of engineering, there is still no commonly accepted standard for software requirements that guarantees, when followed, the desired outcome. There is IEEE 29148, but that carefully avoids specifying the how: It tells you a requirement must be unambiguous and verifiable, but not how to express it so that it actually is.

We were faced with this problem while we were working on Surveyor, a tool to extract specifications from running (legacy) systems: What is a good format for the description, once you discovered its features?

Surveyor is currently in early access testing (you can take part to), and we already learned a lot during the first sessions applying it to other peoples' legacy software. A good moment to share what we have found out, and to ask for feedback. We have run analytics on our beta testers’ codebases, and here is what has worked.

The Premise

Our clandestine team needs freedom of choice. We don't want to force them to use TINYINT if their favorite database does not support that. If an implementation detail is important in a way that does not become clear from the context, we need specify that in the tech choices, or describe the observable outcome in a spec. Otherwise, we should aim to specify independent from technology / solution. Here are our picks to do that.

Architectural overview

The C4 model is a way to visualize software architecture at four levels of abstraction, created by Simon Brown. The "C4" refers to its four diagram types, each zooming in further than the last:

System Context shows your system as a single box surrounded by the users and external systems it interacts with. It answers "what does this system do and who uses it" without any internal detail.
Containers zooms into the system to show its major deployable or runnable pieces, like web apps, mobile apps, databases, APIs, and file systems, along with how they communicate. Here "container" means a separately running unit, not specifically a Docker container.
Components breaks a single container down into its main building blocks and their responsibilities, showing how the code is organized into logical groupings within that container.
Code is the most detailed level, but mostly unused or generated from the code.

The core idea is that you start broad and progressively zoom in, so different audiences can engage at the level of detail that suits them, much like zooming in on a map.

Our Beta testers have found these diagrams most helpful. Even engineers who have worked with their codebase for years welcomed this organised visualisation. Surveyor is generating C4 diagrams in the form of Structurizr DSL.

This solves two purposes:

It offers a standard for generating charts with a well known systematic
It can serve as an input for subsequent steps

C the fourth

The fourth C is “Code”, but it’s not often used as code changes are frequent. Re-aligning UML diagrams each time the code changes adds friction we don’t want. Instead, we leave the diagram level and move on to describing the behavior of the system.

Describing Behavior

In 2006, a movement called Behaviour-Driven Development (BDD) emerged. The idea was to write specifications in natural language that could be parsed and executed as tests simultaneously. This enabled product teams to create specifications that engineers could use directly as automated tests. The engineers would start by creating the bindings between feature language and test code. The implementation would begin with a failing test (red), which the actual feature code would then make pass (green).

Gherkin is a plain-text language for writing executable specifications in a Given-When-Then format, Cucumber is the framework that maps these specifications to code (step definitions) so they run as automated tests.

A feature like the following:

Feature: Monster battle

  Scenario: Battle

    Given there is a monster

    When I attack it

    Then it should die

Would map to a binding like this:

# test/features/step_definitions/monster_steps.exs
defmoduleMonsterStepsdo
useCucumber.StepDefinition
importExUnit.Assertions

  step "there is a monster", context do
Map.put(context,:monster,Monster.new())
end

  step "I attack it", context do
Map.put(context,:monster,Monster.take_hit(context.monster))
end

  step "it should die", context do
    refute context.monster.alive?
    context
end
end

BDD’s peak popularity came in the early 2010s, a period when teams worldwide embraced it and equivalents appeared in every major language with implementations like Behave and pytest-bdd for Python, SpecFlow/Reqnroll for .NET, Behat for PHP, and Godog for Go.

Tooling is broad too, including IDE plugins for IntelliJ, Visual Studio, and VS Code offering syntax highlighting and step navigation, plus reporting tools and CI/CD integrations

Interest has cooled down some years after that because developers grew tired of maintaining large chunks of BDD code and bindings. Luckily, we can leave that part to the LLMs now.

The Case for BDD

We are revisiting BDD today, because AI assisted coding has a problem: Plans and free formats like OpenSpec are not parsable. That means they can not be translated to automated tests directly. You need an agent step to build the testing, with all uncertainty agentic coding brings.

The relatively rigid syntax of Gherkin is an advantage here: It maps mechanically to test code. And it can also be parsed to produce other output formats: Manual QA instructions, OpenSpec or Jira tickets.

But for that, it’s missing the meta information that embeds a feature description into its application context.

The Assay format

We have used the C4/structurizr DSL to express how our system interfaces with the outside world. We also used it to describe the inner composition of our application. When we use Gherkin alone, this context would be lost. But we can preserve it with syntax-aware comments like this:

# ---
# component: orderLifecycle
# container: api
# schema: schemas/order_lifecycle.ex
# workspace: architecture/workspace.tsp
# definitions:
#   a valid customer:
#     a Customer with status Active, verified email,
#     and a credit limit greater than zero
#   in stock:
#     the product has available quantity greater than
#     the requested amount
# invariants:
#   - total must not exceed customer credit limit
#   - at least one line item required
#   - cannot cancel an order that has shipped
# ---

With this, we can access the information in the structurizr DSL to get context information. This allows us to write parsers and plugins for coding agents that can not only read the features, but also access context information we gathered before. But what's purpose of the schema: definition?

Schema Definitions

Schema work as a shared contract about the business objects of a component. Having them speeds up onboarding. Developers don't need to scan through all features to detect what fields are needed, and agents are less likely to invent fields. It's a token-efficient way to inject exactly the ground truth into context.

For specifying schemas, three obvious candidates come to mind:

Protobuf — https://protobuf.dev — Google's schema and wire format. Compact and fast, with code generators across many languages, but its scalar types are physical commitments, so it presumes the most about storage and encoding.
```
syntax = "proto3";

message Post {
  string title = 1;
  int32 views = 2;
  bool published = 3;
  repeated string tags = 4;
}
```
JSON Schema — https://json-schema.org — A decade-plus standard for describing the shape and constraints of JSON. Stays loose where you want, needs no toolchain for consumers, and fits the premise most directly; the cost is verbosity. It can be authored directly, e.g. as YAML and be compiled to JSON:
```
$schema:"https://json-schema.org/draft/2020-12/schema"
$id:"https://assay.example/post"
title: Post
type: object
required:[title]
additionalProperties:false
properties:
title:{type: string }
views:{type: integer }
published:{type: boolean }
tags:
type: array
items:{type: string }
```
TypeSpec — https://typespec.io — Microsoft's TypeScript-like language that compiles to JSON Schema, OpenAPI, and Protobuf from one source. Compact and readable, at the price of a build step and a single-vendor toolchain.
```
model Post {
  title:string;
  views?: integer;
  published?:boolean;
  tags?:string[];
}
```

Under the premise that the implementing team picks the field types and specifics the three differ in how much they presume. Protobuf fits worst: its scalar types (int32, bytes) are physical commitments that pre-decide the storage details you meant to leave open. JSON Schema is the opposite — it describes shape and constraints without binding to any storage type, and hands the team a directly-usable artifact with no toolchain; its cost is verbosity and $ref-wired files. For our purpose, TypeSpec is the golden spot:

A compact, TypeScript-like authoring that compiles to JSON Schema (or Protobuf later, if needed), at the price of a build step and a younger, single-vendor toolchain. This is not so relevant in this context as we use it for documentation rather than compiling. It adds, however, the cross compilation benefits.

Architectural Decision

Until this point, we avoided to describe the “how” of our application. Which database, which web framework, what datamodel. This is what Markdown Architectural Decision Records (MADR) cover. Their goal is to document not only the technical decision, but what options were considered and what the problem statement was. Because MADRs contain that background, it allows to effectively reconsider decisions once the Circumstances have changed.

They typically look like this:

## **Use PostgreSQL for primary data store**

* Status: accepted
* Date: 2026-06-29

### **Context**

We need a primary database for a new service with relational data and strong
consistency requirements.

### **Considered Options**

* PostgreSQL
* MongoDB

### **Decision**

Use **PostgreSQL** — the team has operational experience with it, and it
provides ACID transactions plus JSONB for flexible fields. MongoDB was rejected
due to weaker consistency guarantees and less team familiarity.

### **Consequences**

* Good: mature ecosystem, existing team expertise
* Bad: harder to scale horizontally than MongoDB

Non Functional Requirements

By now we have described our application well in terms of how it is structured, what it does, and what tools it uses (or should use). But we left out the quality of service.

A non-functional requirement (NFR) specifies how well a system must operate, as opposed to what it does (functional requirements). Where a functional requirement says "users can book an appointment," an NFR constrains the qualities of that behavior — performance, scalability, availability, reliability, security, maintainability, usability, and so on.

They are often written in a simple table format:

Here's a fuller set, functional requirement paired with its non-functional counterpart:

Quality attribute	Functional requirement (what)	Non-functional requirement (how well)
Performance	User searches for available slots	Results return in under 500ms for 95% of requests
Scalability	Users book appointments	System handles 1,000 concurrent bookings without degradation
Availability	The booking system is accessible	99.9% uptime, measured monthly (≈43 min downtime/month)
Reliability	A confirmed booking is saved	Zero confirmed bookings lost; durable before confirmation returns
Security	Users log in to their account	Passwords stored hashed; data encrypted in transit (TLS) and at rest
Usability	User completes a booking	A new user can book without instructions in under 2 minutes

Each row shows the same feature twice: the left column is what the system does, the right is the measurable constraint on how well it must do it — the NFR.

The End-Boss: Design

Specifying UI is a daunting task because it happens at the end of the rendering pipeline where the amount of moving parts is highest. Most tools for web therefore use the web technologies themselves to avoid drifting.

The imageto effectively reconsider decisions once the Circumstances have changed.

The simplest, (and arguably the worst) medium to specify design is the image:

Then the login page should match "fixtures/login.png"

This allows features to specify design alongside the functional feature requirements.

The code

If the frontend code is using components and maybe even a component library, it might be worth carrying over. A connection to a design system in Figma or similar is the gold standard here, but might be violating our "letter in envelope" constraint. Our goto strategy at bitcrowd is Storybook. If the target system supports it, teams can work very effectively.

Visual Regression Testing

If you happen to have a legacy system with a decent design, visual regression testing with Playwright or similar might be your tool of choice. It gives teams a quick feedback loop if it comes to single scenarios, and the option to recheck the whole interface. The great thing in combination with Gherkin is that you can switch screenshot processing on and off. The difficulty with visual regression tests is their potential flakiness. However, image processing models have made this a lot more reliable.

So, what is in that envelope?

So far we have C4/structurizr diagrams for how the system is build and how it interacts, Gherkin/Assay for its behaviour, TypeSpec for schemas, MADR for the technological choices and NFR for its quality of service. We go for storybook for the visuals. Obviously, this is not the final solution for every stack. It would be hard to describe a pacemaker OS with it, or a web application with pristine UI effects. The final version of Surveyor will use plugins to cater for that, but for many of the projects bitcrowd has encountered, this gives a good start.

Fortunately, we have clients who enjoy talking to us, so we haven’t had to resort to espionage technology - yet. Let us know if you have some clandestine system you need to build in the dark, or that legacy application needs a rewrite.

CORS in Phoenix

Tue, 30 Jun 2026 23:43:47 +0000

Single page applications Single page applications (SPA) are becoming more and more popular, often replacing the more traditional server rendered websites that are common in Rails or PHP. You have options such as Angular, Ember, React, and there are…

Recursion in Elixir

Tue, 30 Jun 2026 23:43:47 +0000

Intro I recently wrote an article on Recursion in Ruby , and this is meant to be its Elixir counterpart. It will provide a way to compare solving the same problems in both languages and a chance to talk about some of their differences. Heads & Tails…

FizzBuzz in Elixir

Tue, 30 Jun 2026 23:43:47 +0000

Intro... learning Elixir This is my first post in the new Elixir category I've set up on my site. I'm new to Elixir and functional programming in general, aside from a Scala course I've done on Coursera. Ruby is the language I'm currently most…

Guards! Guards! - Hauleth

Sat, 27 Jun 2026 16:31:10 +0000

~hauleth

Guards! Guards!

2026.06.27 :: 2 min :: #beam

Let's start with simple quiz.

Given module defined as:

defmodule Foo do
  def a(x) when is_integer(x) or is_map_key(x, :foo), do: true
  def a(x), do: false

  def b(x) when is_map_key(x, :foo) or is_integer(x), do: true
  def b(x), do: false
end

Try to answer these questions.

Q: What will be result of Foo.a(%{foo: 21})?

true

false

WRONG

RIGHT

This one is straightforward.

We check guard, it has one condition is_integer(x) or is_map_key(x, :foo). First one returns false, second returns true, Boolean's alternative results in true and first case is matched.

Q: What will be result of Foo.a(37)?

true

false

WRONG

RIGHT

This one is straightforward as well.

We check guard, it has one condition is_integer(x) or is_map_key(x, :foo). First one returns true, second one isn't fired at all, because or operator is short circuiting.

Q: What will be result of Foo.b(%{foo: 21})?

true

false

WRONG

RIGHT

Again, similar to the previous questions.

We check guard, it has one condition is_map_key(x, :foo) or is_integer(x). First one returns true and the rest is short circuited.

Q: What will be result of Foo.b(37)?

true

false

WRONG

RIGHT

Ouch, something changed…

Again, we check guard, one condition is_map_key(x, :foo) or is_integer(x). We hit first clause is_map_key(x, :foo) and this doesn't return false, instead it fail. Failure in one of guard functions isn't converted to false but instead makes whole guard expression fail. This mean that is_integer(x) part will never be called.

This behaviour is often surprising for a lot of Elixir developers, as it seemingly breaks commutative property of boolean operators. However, to be honest, these never were commutative because of short circuiting.

It seems that Elixir at the time of writing (Elixir 1.20.1, OTP 29) do not warn about this issue.

∎

Author of this post is currently open for hire (CV).

You can contact me at lukasz@niemier.pl

You can provide feedback via mailing list ~hauleth/blog@lists.sr.ht (archive).

Webmentions:

https://bsky.app/profile/did:plc:6psaz6n5fuurrfet67zs4ljf/post/3mpbudsxwi22h

This week in infrastructure

Sun, 21 Jun 2026 17:34:48 +0000

Supabase announces v0.1 of a distributed Postgres project, Multigres. VillageSQL announces a REST layer for MySQL a la PostgREST. MariaDB embeds DuckDB. Elixir is now gradually typed. PgDog gets funded. Native Linux containers for macOS reaches 1.0. And a new Cassandra client written in Rust.

Exploring Programming Languages

Fri, 19 Jun 2026 18:28:26 +0000

Nott Løvland · Exploring Languages

In the beginning, I sought to be wise,
with a language that opened my digital eyes.
I started with LISP, for the pure of the mind,
But spent three whole weeks trying pairs to unwind.

(((((((Are (we) (sure) (this) (is) (right?))))))))
Parentheses blinded my aching eyesight.

So I fled to the past, where the money was made,
and woke up in COBOL, deeply afraid.
IDENTIFICATION DIVISION roared in my head,
with columns and margins, I wished I were dead.
It’s great for a bank in the year seventy-nine,
But a thousand lines later, I still couldn’t sign.

“Let’s try something hip!” I exclaimed with a twirl,
and drowned in a bucket of regex and Perl.
A script that looked exactly like line-noise and spit,
It ran, but God help me, I can’t read a bit.
Is that a variable, or did my cat walk
across the keyboard while trying to talk?

Then came Python, the savior, the clean, and the bright!
Until a stray space ruined my day and my night.
“Indentation Error,” the compiler did shriek,
because of one tab in the middle of the week.
Don’t get me even started on packaging hell,
where pip and venv cast a curse and a spell.

I jumped into Ruby, for joy and for love,
with blocks and with gems sent from heaven above.
But Monkey Patching turned the code to a zoo,
When a library changed what 2 + 2 do.
It was beautiful, sure, but it ran like a snail,
Chugging along on a rusted old rail.

So I went corporate. Enter Java, the grand.
The boilerplate king of the enterprise land!
AbstractMethodFactoryProviderBean
—was the shortest class name that I ever had seen.
I typed until my fingers were bleeding and numb,
just to print out “Hello” to a world that was glum.

“To the web!” I declared, and embraced JavaScript,
where logic is warped and reality’s flipped.
[] == ![] evaluated to true,
and NaN is a number? I’m sorry, that’s rude!
undefined is not a function, it cried in my face,
as npm bloated and swallowed my disk space.

So I looked for speed, and Golang caught my eye,
“It’s simple!” they promised, “Just give it a try!”
But if err != nil was on every damn line,
an endless repetition of error design.
No generics (at first), just copy and paste,
a minimalist’s dream, and a developer’s waste.

Then Rust was the answer, the savior of tech!
If the Borrow Checker didn’t snap my poor neck.
“You can’t use lifetimes! This memory’s owned!”
I sat at my desk, thoroughly powned.
I fought with the compiler for hours on end,
until I forgot why I coded, my friend.

Then out of the ashes, a phoenix arose,
embraced by Erlang, in functional prose.
Elixir! Oh, sweet elixir of life,
you banished my sorrow, you ended my strife.
With pattern matching so clean and so neat,
and pipe operators (|>) that make life a treat.
The BEAM handles millions of actors with grace,
while I sit with a massive, smug smile on my face.

Let the servers all crash! Let the supervisors play!
I’m finally happy. Go away, anyway.

Elixir for a Bluesky DataPlane: the choice we didn't expect

Thu, 18 Jun 2026 20:30:01 +0000

Bluesky's source code is widely open source, so you can run your own social network with it. - Provided you stay with a comparably small user base. What's missing? A performant DataPlane implementation. Closing this gap would be an important step towards building digital independence.

We wanted to contribute our share and decided to work on a performant DataPlane for Bluesky. When we started the project, we expected to work in Go, Rust or even Node. After all, these were the predominant languages in the community. Instead, we landed with Elixir. Here is why and how we came to that decision.

The component nobody knows about

Everyone who has looked into the Bluesky infrastructure knows about the Personal Data Stores (PDS), the Relay or the AppView. When you set up your own little Bluesky from the ATProto repository, you run the same components as Bluesky, the commercial service. However, there's one notable exception: the DataPlane, a part of the AppView. It exists publicly only as a Node-and-Postgres reference implementation, while the production Bluesky network runs on a dedicated, ScyllaDB-backed, closed-source DataPlane (as documented in Pragmatic Engineer's deep-dive on Bluesky's architecture).

That gap is exactly where things get interesting. The reference implementation tells you what the DataPlane does; it doesn't tell you how to make it survive contact with real traffic.

If you want to run your own, you have to answer the scaling question yourself - and the first step is understanding the workload well enough to stop treating it as one thing.

The DataPlane is a central component of the AppView in Bluesky's architecture. Read theprevious postto learn more.

ScyllaDB is Bluesky's operational choice, not part of the interface. The DataPlane's contract is a gRPC service that answers high-volume, low-complexity queries and returns skeletons - lists of IDs, counts, booleans - which a higher layer later hydrates into full views. What sits behind that contract is entirely up to you: the language, and the datastore. So before picking either, we spent our time on the only thing that actually constrains the choice: the shape of the load.

A tale of two workloads

The Bluesky's historical data is enormous — terabytes. But when a user opens the app, they read a few dozen recent posts, get distracted, and wander off to a profile or a thread. They almost never scroll back further than a day or two.

A note on specifics: it's been observed that Bluesky's timeline doesn't serve much beyond the last day or two of content, and that deeper cursor positions tend to fill with very recent posts rather than true history. Treat the exact window as illustrative unless you've measured it on your own deployment - the architectural point holds regardless of the precise number.

This leads to an interesting disparity: Data that is more than a few days old is almost never accessed in the timeline. When it is accessed, it is usually through a profile or a thread. Yet the reference implementation compiles every timeline by joining those tables, which are terabyte-scale. This process is slow and becomes slower as the tables grow, and timeline requests account for most of what the DataPlane is asked to do.

Data volume and read frequency are inversely related: the vast bulk of data is old and almost never read, while the tiny sliver of recent posts drives nearly all timeline traffic.

So timeline generation deserves a different treatment from record and thread retrieval. The two workloads don't just differ in degree. They have opposite resource profiles, and they want different things from the runtime underneath.

	Hot path - timelines	Cold path - records, threads, profiles
Data age	Recent (last day or two)	Historic (anything older)
Data volume	A tiny sliver	Terabytes
Request share	The dominant workload	Comparatively rare
Bound by	Memory + compute	I/O
Lives in	Memory	Database, fetched on demand
Strategy	Fan-out + bounded timeline length	Swappable datastore behind an interface

The hot path: timelines served from memory

Most social platforms use a hybrid strategy to handle timeline fan-out. A post from an account with a few thousand followers is pushed into its followers' timelines as soon as it is published. This is known as fan-out on write in the lingo.

However, a post from an account with millions of followers ("the celebrities") is not immediately pushed to all of its followers' mailboxes; instead, it is merged in when a follower actually requests their timeline: This is called fan-in on read. The former keeps write amplification bounded for ordinary accounts. The second prevents the write storm that a celebrity post would otherwise cause.

Another obvious simplification becomes apparent when you stop thinking of a social feed as an immutable archive. Take into account that users will jump around with their attention and you will realise that timelines are finite. Therefore, a user who follows ten thousand accounts will never see all of their posts anyway. There's reasonable to limit how much content is distributed to any single timeline. The obligation is to provide a good, recent, bounded timeline, not a complete one.

What this boils down to:

Recent posts - the overwhelming majority of what timelines are made of - can live and be served from memory.
Fan-out can be deferred: as long as posts land in followers' timelines within minutes, and the backlog of fan-out jobs doesn't outgrow available resources, nobody notices the delay.
Older content, when it's genuinely needed, is safe to fetch from the database on demand.

The dominant request type, the timeline read, stops being I/O-bound (wait on a giant join) and becomes memory-and-compute-bound (manipulate in-memory structures quickly). That shift changes what "fast" means, and it's what reopened the language question for us.

It also argues for putting the concrete database behind an interface. The hot path barely touches it; the cold path is the only part that leans on it. Keep that boundary clean and you can swap the backing store later without touching the rest of the system.

The follower graph: in memory, but not naively

There's a catch. Fan-out and timeline assembly both need answers about the follow graph: who follows whom, and the intersections and unions of those sets. Holding hundreds of millions of follow relationships in memory as hash maps would be ruinously wasteful.

Jaz's "GraphD" series documents this exact problem: an in-memory graph store that originally used hash maps and hash sets to track each user's followers and follows. The fix was switching to Roaring Bitmaps, a compressed bitmap structure built for large set operations. The numbers are striking: the entire Bluesky follow graph fits in roughly 6.5 GB of RAM, takes about 1.6 GB on disk, and loads in around 20 seconds.

Jaz also describes two cost modes that map onto our hot/cold split. Paging over all of a user's follows is expensive and belongs in paginated or async fan-out jobs. On-demand set intersection — "which people I follow also follow this person" — has to run at interactive speed. Our split isn't an invention; the access patterns already worked this way in Bluesky's own tooling.

So what do we actually need from a language?

With the workload pinned down, the requirements are:

Fast, compute-bound set operations over a large, long-lived, in-memory graph: bitmap intersections and unions. Per-core compute, byte-level work, cache sensitivity.
High-concurrency serving of memory-resident timeline reads, with bounded response sizes and tight tail-latency expectations.
A deferrable fan-out queue that absorbs bursts, applies backpressure, delivers within minutes, and degrades gracefully instead of falling over.
A clean datastore boundary for the cold path: records, threads, profiles. Ordinary I/O-bound request handling.

The first three are the hot path; the fourth is the cold path. No language obviously wins all four. This is the scorecard we judged each candidate against.

The four candidates

At bitcrowd we work with Elixir, Go and Rust day to day, with the occasional Node project on the side. So this wasn't a contest between a favourite and a lineup of strangers - we had hands-on experience to weigh on every side of the comparison.

Go

Go is the incumbent. It's what Bluesky's own production DataPlane is written in, and the fit is strong. Goroutines and channels map cleanly onto "fan out work, gather results, respond." The bitmap work is comfortable in Go (GraphD itself is Go). gRPC support is first-class, deployment is a single static binary, and the tooling is mature. In-process fan-out is doable with worker pools draining buffered channels.

The costs sit at the extremes. Under heavy allocation Go's garbage collector starts eating CPU, and at very high socket counts the network backend can bottleneck on syscalls. Both respond to runtime tuning, but the tuning is ongoing work, not a one-time fix. And the in-process fan-out you build yourself comes with no supervision or isolation layer: a panicking worker takes the process down, and backpressure and lifecycle management would be our's to write.

Rust

Rust gives the highest ceiling and the tightest control. No garbage collector means no GC pauses in the tail latency. Tokio handles enormous concurrency with low per-task overhead, tonic covers gRPC, and the roaring-bitmap and datastore libraries are excellent. For the compute-bound half of our workload, nothing beats it.

The cost is velocity. As this service is thin on business logic, we would pay the price of the borrow checker and the sharp edges of async Rust on every line while protecting very little. Fan-out with Tokio tasks and channels works very well. However, as with Go, we would need to build lifecycle, backpressure and supervision manually.

Node / TypeScript

Node deserves real consideration because it's the language of the reference DataPlane and the rest of the atproto stack: PDS, AppView frontend, lexicons. One language across the codebase, shared types from lexicon definitions, the largest hiring pool, the fastest iteration. For the cold path's I/O-bound record fetches it's fine, and for modest-scale self-hosting it's a sensible default.

The hot path is where it breaks down. One event loop per process means in-memory queues and request serving compete for the same loop, and any CPU-bound work blocks both. Using multiple cores means multiple processes with no shared memory, which brings back the cross-process coordination the in-process design was meant to remove, and makes a large shared in-memory graph awkward. It's the right tool for the reference implementation and a strained one for a throughput-oriented production server.

Elixir

Elixir runs on the BEAM, a runtime built for massive numbers of cheap, isolated, preemptively scheduled processes. Per-process garbage collection means no global stop-the-world pauses, so tail latency stays flat under load. Supervision trees give fault isolation and recovery essentially for free. For high-concurrency request serving and a backpressured in-process queue, it's the most naturally suited runtime of the four.

It has one well-known weakness: raw per-core compute. The BEAM optimises for concurrency and consistent latency, not single-threaded number-crunching. Byte-level work like set operations over a large follower graph is exactly where it's slowest. Taken at face value, that rules it out for a workload with heavy bitmap operations at its core.

Every candidate has a flaw — which ones can you fix?

Four reasonable options, each with one thing standing between it and a clean fit. Instead of arguing it on paper, we prototyped toward each remedy, far enough to tell whether the flaw could be engineered away or was structural.

Go's GC and syscall overheads respond to tuning, and Bluesky's production DataPlane proves you can push Go a long way. But the tuning never ends. It's a knob you keep turning for the life of the service.

Node's single event loop has exactly one fix, running more processes, and that fix recreates the cross-process coordination and shared-state problems we were designing out. There's no way around it within the language.

Rust has no performance flaw. Its cost showed up as soon as we started building: hand-rolled concurrency, lifecycle and backpressure machinery on every path, for a service with little logic to protect. We realised that that cost would not shrink over time, but stay with us with every change we would need to make.

Elixir's flaw, per-core compute on tight loops, is real, and we hit it where you'd expect: the follower-graph set operations. But it stayed in one place. It didn't smear across the service. It sat in a single, well-defined component we could draw a boundary around, and a component with a sharp boundary can be replaced. That's what sent us back to Elixir for a second look.

Resolving the Elixir paradox

Can that component actually be lifted out? Yes, and the reason has less to do with Elixir than with how our system is laid out.

Start with the claim itself, because it's often stated too broadly. "The BEAM is slow at CPU work" is true for tight numeric loops on a single core. It is not a claim about how much hardware a real service needs. Most of what a production server spends its time on isn't a tight loop; it's keeping thousands of concurrent requests moving, and the BEAM is efficient at exactly that.

Per-process garbage collection means no global pauses, and tail latency stays flat enough that you can run a node closer to its limit and still hit your latency target.

This is why Elixir services often need fewer machines than the Go or Node version of the same thing, occasionally approaching Rust, even though they lose every microbenchmark. Hardware cost under concurrency and single-core throughput are two different measurements, and only one of them shows up on your bill.

That said, our workload really does contain the thing the BEAM is bad at: the roaring-bitmap intersections and unions over the follow graph. We don't want to talk our way around that, so we move it off the BEAM entirely.

The follower graph lives in a Rust implementation of Roaring Bitmaps, called from Elixir as a NIF (Native Implemented Function). What makes this more than a generic "wrap the slow part in Rust" patch is the shape of the data flow:

The input crossing the boundary is tiny: a user ID, or a small set of IDs.
The graph itself, millions of edges, stays on the Rust side in native memory. The BEAM never holds it, never copies it, never garbage-collects it.
The expensive compute, the intersections and unions, runs entirely inside Rust at native speed.
Only the result crosses back, and because timelines are length-limited, that result is small by construction.

The usual objection to NIFs is the copy cost at the boundary. But in our case, only small data crosses in both directions while the big structure and the heavy computation stay native, which is the ideal case. That isn't luck. The timeline-length limit we'd already committed to is what bounds the return values.

This subtracts the BEAM's weakness from the hot path and keeps its strengths. The compute-bound half runs in Rust regardless of host language. What's left for Elixir is orchestration: high-concurrency request serving and the fan-out queue, the regime where the BEAM is at its best and where our own production experience says it's most efficient.

Of course, the NIF has its own price. It runs inside the BEAM's memory space, so a crash in the Rust code can take down the VM, and a long-running call can stall a scheduler. So we give up some of the "let it crash" isolation exactly at the native boundary. The mitigations fit this case well: the calls are short (small in, bounded compute, small out), and the Rust surface is small, stable, and the kind of code that rarely changes once written. Two languages is a real maintenance cost, but Rustler keeps the boundary ergonomic, and a small, bounded Rust core seemed the cheapest of the available evils.

The deciding factor: fan-out as code, not infrastructure

The NIF made Elixir competitive. The fan-out requirement made it the choice.

Recall the spec: a deferrable queue that absorbs bursts, applies backpressure, delivers within minutes, and degrades gracefully when the backlog grows. In Go, Rust or Node, that almost always becomes an external system — Redis, NATS, RabbitMQ, Kafka — because the language doesn't give you primitives to do it safely in-process.

On the BEAM, those primitives are the language. Processes are the workers, mailboxes are the queues, supervisors handle recovery, and GenStage and Broadway add explicit backpressure, all inside one VM with no network hop and nothing extra to deploy. Several costs disappear outright:

No serialisation across a queue boundary. An external queue serialises every fan-out job on push and deserialises it on pop, putting per-event encoding cost right back on the hot path we'd worked to keep it off. In-process, a job is a message between processes.
No second system to operate. No separate scaling story, no "is Redis the bottleneck now," no disagreement between the service's view of the backlog and the queue's.
One failure model. Supervision covers fan-out workers the same way it covers everything else. There's no seam between "the service crashed" and "the queue is in a weird state."
Backpressure in-band. Producer and consumer share a runtime, so the producer notices the consumer falling behind directly instead of inferring it from queue-depth metrics after the fact.

The trade is durability. In-process means in-memory, and an in-memory backlog dies with the node. For us that's acceptable: fan-out is best-effort timeline population, timelines are bounded and ageing, and the fan-in path on read covers whatever goes missing for a window. We're choosing fast, simple and lossy-on-crash over durable, external and heavier, and the choice only works because the rest of the architecture makes the loss cheap. If you need durability here, much of the in-process advantage narrows. Check this assumption against your own tolerances.

Why Elixir

We love working with Elixir, but we did not assume it to be the tool of choice for this project. The ecosystem just did not seem to be waiting for it. Choosing it means making the case for it, repeatedly, starting with this post.

And Elixir doesn't win every category. We came to the decision to use it because, for this specific workload, the one axis it loses on — per-core compute for graph operations — is the one we can cleanly offload to a small Rust NIF, with a data flow that makes the offload nearly free.

Everything that remains is concurrency, coordination, predictable tail latency under load, and a burst-absorbing fan-out queue we can build in the program itself instead of bolting on as infrastructure. Go would have been the pragmatic, proven middle.

Rust would have given us the highest ceiling at the cost of velocity and a lot of hand-rolled concurrency machinery. Node was right for the reference implementation and wrong for a memory-bound production server. Elixir plus a thin Rust core covers both halves of a workload that genuinely has two halves.

What next?

Most of this is architecture-level reasoning informed by the workload's shape and each runtime's known characteristics. We backed this up with head-to-head benchmarks between the Node reference implementation and our Elixir POC.

For that, we needed to build a performance evaluation toolkit for our studies. If you want to find out the limits of your setup, check it out

The pieces we lean on hardest are well-sourced: Jaz's GraphD work for the in-memory graph and Roaring Bitmaps, and the documented existence of a fan-in/fan-out split between expensive paging and interactive-speed set intersection. The pieces we're least certain about - the exact timeline window, the precise per-request CPU breakdown - we've flagged as such.

We have build and tested a proof of concept that serves the skeleton amazingly fast. What is missing now is the boring part of writing the > 90 endpoints of the DataPlane API and the indexer plugins. Currently, it's a side project. Give us a shout if you want to help us take it further, faster.

Your Orchestrator Is a Finite Automaton in Denial

Tue, 16 Jun 2026 03:03:39 +0000

Somewhere in your codebase there is a table with a column called status. It started life as the most innocent thing imaginable—a single string, "pending", set once at insertion and forgotten. Then somebody needed to know whether the thing had been paid for, so a boolean joined the schema. Then somebody needed to know whether it had shipped, and another boolean arrived. By the time the quarter closed, your status column had accreted flags the way a ship’s hull accretes barnacles: quietly, asymmetrically, and always below the waterline where nobody looks until the thing stops steering.

This is status-driven orchestration, and it is the most popular way in the industry to build a finite-state machine while loudly insisting you are doing nothing of the sort. The denial is the interesting part. Everyone agrees that explicit state machines are good in the abstract—the way everyone agrees that flossing is good—and then goes home and writes another cond/1 that branches on a string.

Let me try to talk you out of it.

The Anatomy of the Thing You Built

Here is a perfectly representative specimen. An order goes through a fulfilment pipeline: it gets paid for, packed, shipped, delivered. It can be cancelled. It can be refunded. Nothing exotic. Let us model it the way it actually gets modelled in the wild, in an Ecto schema that has clearly survived three product managers:

schema "orders" do
  field :status,     :string,  default: "pending"
  field :paid?,      :boolean, default: false
  field :packed?,    :boolean, default: false
  field :shipped?,   :boolean, default: false
  field :delivered?, :boolean, default: false
  field :cancelled?, :boolean, default: false
  field :refunded?,  :boolean, default: false
end

And here is the orchestrator that drives it—the beating heart of the system, the thing that gets paged at three in the morning:

def advance(%Order{} = order) do
  cond do
    order.status == "pending" and order.paid? and not order.cancelled? ->
      update(order, status: "paid")

    order.status == "paid" and order.packed? and not order.refunded? ->
      update(order, status: "packed")

    order.status == "packed" and order.shipped? ->
      update(order, status: "shipped")

    order.status == "shipped" and order.delivered? ->
      update(order, status: "delivered")

    order.cancelled? ->
      update(order, status: "cancelled")

    true ->
      {:ok, order}
  end
end

It looks reasonable. It compiles. It passes the one test somebody wrote for the happy path. And it is, I regret to inform you, a catastrophe wearing the high-visibility vest of a solution.

What Is Actually Wrong Here

Illegal states are not merely possible, they are the majority

Count the representable states. The status string takes seven values; each of the six booleans doubles the space. That is 7 × 2⁶ = 448 distinct rows your database will cheerfully accept. Of those four hundred and forty-eight, perhaps seven correspond to an order that could exist in physical reality.

The other four hundred and forty-one are nonsense, and your schema treats them with exactly the same hospitality as the legal ones. Nothing—not a constraint, not a type, not a stern comment—prevents a row with status: "delivered", cancelled?: true, refunded?: true, and paid?: false. That is an order that was never paid for, was cancelled, was refunded money it never received, and was nonetheless delivered. It sits in your database like a passenger holding a ticket for a train that was cancelled: still on the platform, still expecting to be taken somewhere, and now also somehow already at the destination.

The flags breed in the schema like adapters in a junk drawer—each one solved a real problem exactly once, and now you own six and can confidently explain three. Every new boolean does not add a state; it multiplies the space of states you have promised to reason about and silently declined to.

The state machine exists; you simply refused to draw it

There is a finite-state machine in that advance/1 function. It is real, it has transitions, it has rules about what may follow what. The only problem is that it has no single, inspectable existence. It is smeared across the cond, the changeset validations, three controller actions, a background job, and the part of the senior engineer’s memory that he is planning to take with him when he leaves. The transition table lives nowhere and everywhere at once, like a signature forged by committee.

Ask the system a simple question—what states can follow paid?—and there is no honest way to answer it except to read the entire codebase and hope you found every site that writes to status. Spoiler: you did not. There is one in a Rake-equivalent task from 2023 that sets status = "shipped" directly, bypassing advance/1 entirely, because someone needed to fix a stuck order once and never removed the scaffolding.

Nothing stops a transition that should be unthinkable

Because status is just a field, any code that can reach the struct can write any value to it. There is no notion of a transition being illegal—there is only the notion of you having remembered to write an if that forbids it, everywhere, forever, without exception. Going from delivered back to pending is not prevented by the design; it is prevented by your vigilance, which is a renewable resource right up until the on-call rotation hits someone new.

Concurrency turns the whole thing into a knife fight in a lift

Two requests arrive at once. Both read the order in state pending. Both evaluate the cond. Both decide the next state. Both write. What follows is less a race condition than a knife fight in a lift: cramped, badly lit, and with exactly one party walking out. Last writer wins, the first update evaporates, and the customer is charged twice because the “already paid?” check read a value that was true a microsecond ago and is now a lie.

You can paper over this with row locks and SELECT ... FOR UPDATE and optimistic-concurrency version columns, and now you are hand-rolling the serialisation guarantees that a process-based state machine hands you for nothing.

Failure is an afterthought wearing a `rescue`

Where, in the specimen above, does failure live? It does not. When the payment processor times out, the code raises, something up the stack catches it, sets status: "failed"—an eighth string value nobody added to the schema’s mental model—and the order joins the population of rows that no longer match any branch of the cond and will sit there, inert, until a human notices the revenue gap.

You overwrote your own audit trail

Every update(order, status: "paid") destroys the evidence of where the order was a moment ago. The history of the process—the single most valuable thing you have when debugging why an order is stuck—is overwritten in place. Reconstructing it later is archaeology conducted with a teaspoon and a head-torch, cross-referencing log lines against updated_at timestamps and praying nobody ran a backfill.

Now Do It With an Actual Finite Automaton

Here is the same pipeline as a real FSM, using my Finitomata library. The entire state machine—every state, every legal transition, every event that triggers it—is declared once, in plain text, in a format that is simultaneously the code, the documentation, and a diagram your product manager can read:

defmodule Order.FSM do
  @fsm """
  pending --> |pay| paid
  pending --> |cancel| cancelled
  pending --> |expire| cancelled
  paid --> |pack| packed
  paid --> |refund?| refunded
  packed --> |ship| shipped
  shipped --> |deliver| delivered
  """

  use Finitomata, fsm: @fsm, auto_terminate: true, timer: 15 * 60_000
end

That is not pseudocode and it is not a comment. That @fsm string is parsed, validated, and compiled into a GenServer with all the transition machinery generated for you. The diagram is the source of truth, because there is no other source for it to disagree with.

Then you implement the business logic, and only the business logic, in callbacks that pattern-match on exactly the state-and-event pairs you care about:

@impl Finitomata
def on_transition(:pending, :pay, %{amount: amount}, payload) do
  case Payments.charge(payload.customer, amount) do
    {:ok, receipt} -> {:ok, :paid, Map.put(payload, :receipt, receipt)}
    {:error, _reason} -> {:error, :payment_declined}
  end
end

def on_transition(:paid, :pack, _event_payload, payload),
  do: {:ok, :packed, payload}

A successful charge moves the machine to :paid and stashes the receipt in the payload. A declined charge returns an error, the machine stays exactly where it was, and control flows to on_failure/3—a first-class, named place for things going wrong, rather than a rescue clause hoping for the best:

Time itself becomes a transition rather than a shadow orchestrator. The timer: 15 * 60_000 option calls on_timer/2 on a schedule, so an unpaid order expires on its own, without a cron job somewhere running WHERE status = 'pending' AND inserted_at < ... and mutating rows behind the FSM’s back:

And driving it from the outside is unceremonious:

{:ok, _pid} = Finitomata.start_link()

Finitomata.start_fsm(Order.FSM, "order:42", %{customer: cust, amount: 9_900})
Finitomata.transition("order:42", {:pay, %{amount: 9_900}})

Finitomata.state("order:42")
#⇒ %Finitomata.State{current: :paid, history: [:pending], payload: %{…}}

Notice the history field. The machine remembers where it has been, for free, without you overwriting anything.

Point by Point, Why This One Wins

Illegal states stop being representable

The current state is a single atom drawn from a closed set the compiler knows about. There is no status string and a constellation of booleans to fall out of sync; there is the state, and there is the payload, and they are different things with different jobs. The four hundred and forty-one nonsense rows simply have nowhere to live. You cannot be delivered and cancelled simultaneously for the same brutally simple reason you cannot be in two rooms at once: the model does not have a word for it.

The transition table is validated before your code ever runs

This is the part that ought to close the argument by itself. The :finitomata compiler reads your @fsm declaration and refuses to proceed unless it is a consistent machine: exactly one initial state, at least one final state, and no orphans—no state you can enter and never leave, no state you declared and can never reach. It refuses to compile an incoherent machine the way a good editor refuses a sentence that parses but lies.

Better still, if you add a transition to the diagram and forget to handle an ambiguous case in on_transition/4, the compiler tells you, at compile time, with a warning that names the gap. Compare this to the status-driven approach, where a forgotten branch is discovered by a customer, on a Saturday, via Twitter.

Transitions are guarded by construction, not by your memory

A transition that is not in the diagram does not happen. Sending {:deliver, …} to an order in state pending is not a bug you must remember to prevent with an if; it is structurally impossible, ignored by the machine the way a vending machine ignores a button for a slot that does not exist. The set of legal moves is data, declared once, enforced everywhere, rather than folklore re-implemented at each call site.

Concurrency is solved because each machine is a process

Every Finitomata instance is its own GenServer, which means transition requests for a single order are serialised through a single mailbox and processed one at a time, in order. The knife-fight-in-a-lift disappears, not because you were careful, but because the architecture made the fight impossible to start. No row locks, no version columns, no optimistic-concurrency retries—just the actor model doing the one thing it has always been excellent at.

Failure, timeouts, and retries are vocabulary, not accidents

The library has named, first-class places for the things status-driven code handles by flailing: on_failure/3 for transitions that did not complete, on_timer/2 for the passage of time, the ensure_entry: option to retry a transition until it sticks, and the last error preserved in the state for when you need to know why. A transition that ends with a ?—like paid --> |refund?| refunded—is declared as one that is expected to sometimes fail, so it does so quietly, without crying wolf in your logs. A transition that ends with a ! is determined and fires the instant it becomes the only way forward. These are not features bolted on; they are the grammar of the thing.

History and observability come included

The state carries its own history. When an order is stuck, you ask the machine where it has been and it tells you, instead of you reconstructing the past from the sediment of updated_at. Pair it with the telemetria integration and every transition is an event you can measure, rather than a mutation you have to infer.

Testing stops being string-equality theatre

Because the machine is explicit, you can test the machine. Finitomata.ExUnit lets you walk a path through the states and assert on each transition and the resulting payload, with a syntax that reads like the thing it verifies:

The status-driven equivalent is mocking the database and asserting that a string equals another string, which tests your typing accuracy and very little else.

Distribution is a one-line upgrade

When one node is no longer enough, swap Finitomata for Infinitomata and your machines run transparently across the cluster on top of :pg, with no change to the business logic. Scaling becomes a matter of adding nodes, not of rewriting your orchestrator around a distributed lock you will get subtly wrong.

The Objections, and Why They Fold

“This is over-engineering. It’s just a status field.” It is not just a status field, and the proof is the six booleans standing next to it. You did not avoid building a state machine; you built one and declined to name it, which is the single option strictly worse than both alternatives—you pay the full cost of the complexity and receive none of the guarantees. Naming the tumour does not create it. It lets you operate.

“FSMs are academic, textbook stuff.” So is binary search, which you use without flinching, and the hash table underneath every dictionary you have ever instantiated. “Academic” is what we call the ideas that turned out to be so correct they became invisible. The finite automaton is one of the oldest, most thoroughly understood objects in computer science. Refusing to use it because it has a formal name is like refusing to use a bridge because an engineer was involved.

“We already use a workflow engine for this.” A workflow engine is, in the cases that matter, a finite-state machine that hired a sales team and learned to bill by the seat. If you want the semantics, you can have them in a hundred and fifty lines of your own language, in your own repository, without the YAML, the vendor console, and the per-execution pricing. Sometimes the heavy engine is the right call. Usually it is a sledgehammer rented monthly to drive a thumbtack.

“Our orchestrator works fine.” So does a car with insulating tape over the check-engine light. “Works fine” is a statement about the inputs you have happened to receive so far, not about the four hundred and forty-one illegal states patiently waiting for the input that produces them.

The Point

You are going to build a state machine. That decision was made the moment your process had more than one step. The only choice left to you is whether it will be an explicit state machine—declared in one place, validated by the compiler, guarded by construction, serialised by the runtime, and testable as a unit—or an implicit one, scattered across your codebase like cutlery after an earthquake, held together by a status string, a fistful of booleans, and the fervent hope that nobody writes to the field from somewhere you forgot about.

The implicit machine is not simpler. It is the same machine with the guarantees filed off and the documentation set on fire. Name it. Draw it. Let the compiler check it. Your three-in-the-morning self, squinting at a row that claims to be delivered and refunded and never paid for, will thank you with a sincerity your waking self is not capable of.

Happy—and finite—automating.

Previously, on the subject of refusing to lose your mind over state:

Serving timelines from Elixir

Tue, 02 Jun 2026 04:24:30 +0000

Recently, we presented our performance toolkit for atproto including a small Proof-of-Concept for a dataplane written in Elixir. We didn't go too much into the details though. We will make up for that in this post.

Let's start by recapping the problem.

The AT Protocol enables us to build an open social ecosystem of interoperable applications. By far the most popular application in this ecosystem today is Bluesky.

Bluesky has open sourced almost all of their software. You can find their app, their Go services and their TypeScript code (backend services and protocol reference implementation).

This is their production code, with one exception: they don't run the open source dataplane. They've replaced it with a closed source implementation that is more performant.

The dataplane is a central component of the AppView in Bluesky's architecture. Read theprevious postto learn more.

We talked about some of the issues of the open source dataplane in our previous blog post.
The bottom line is: it won't scale if you're trying to serve hundreds of thousands of users.

Why you should care

Even the Bluesky people think that hard decentralization is necessary. What is the point of an open ecosystem when everything is tied to the existence of a single company?

Bluesky can be bought and start to enshittify
Bluesky operates under US law, why should we want our online social life to be ruled by mad people inside a white house
Bluesky has outages, alternatives could continue to operate

So, we should want alternative providers of Bluesky to exist.

Luckily for us, there is already a number of organizations picking up that work.
Luckily for them, they can build on the open source components that Bluesky provides.
Unluckily for them, they are left with a dataplane implementation that won't work for them when their userbase grows.

As a result, the Blacksky people have to spend their time rewriting components in Rust and our friends from Eurosky started exploring alternative ways to build timelines.

Serving timelines from Elixir

Elixir provides a number of benefits for a backend service like the dataplane.
You work on a high level of abstraction, getting more done in the same amount of time and end up with very readable code (yes, this is still important, even for your agent!).
At the same time you get parallelism for free and performance that scales with the cores of the system.
Errors are isolated, so they don't bring down the whole service. As a bonus, the VM provides you with good tooling for introspection and observability, all out of the box.

So, when we built our Proof-of-Concept, we naturally chose Elixir. We wanted to investigate the performance issues that will come up in the open source dataplane implementation. For this reason, we focused on implementing the critical endpoint which is responsible for serving timelines to users.

From our exploration in the last blog post, we already know that the most important decision to solve the issues of Bluesky's open source database implementation is to change the approach from fan-in to fan-out.

When a user requests their timeline from Bluesky's open source database implementation it will query the posts that should be in the timeline on demand. This is known as fan-in principle and is a major reason for the performance issues of this implementation.

Consequently, the implementation used by Bluesky in production turns this process on the head, known as the fan-out principle. It constructs the timeline for each user whenever a new post arrives in the system. When the user requests their timeline, the data is already waiting for them to be served.

ETS tables

With this insight, it was clear to us that we should build our dataplane on the fan-out principle. It was less clear how we should store the data we need. The Erlang VM has multiple built-in options for storing data, so choosing one of them, the Erlang Term Storage (ETS), was a good starting point.

Citing the documentation:

Data is organized as a set of dynamic tables, which can store tuples.
...
Tables are divided into four different types, set, ordered_set, bag, and duplicate_bag. A set or ordered_set table can only have one object associated with each key. A bag or duplicate_bag table can have many objects associated with each key.

ETS stores data as tuples in tables in memory but which tables do we need to store all the data we need to serve timelines? And of which type should the tables be?

To store follows in a way that we can retrieve all the information we need efficiently, we actually need two tables: followers_table and following_table.

:ets.new(@followers_table,[
:duplicate_bag,
:named_table,
:public,
read_concurrency:true,
write_concurrency:true
])

:ets.new(@following_table,[
:duplicate_bag,
:named_table,
:public,
read_concurrency:true,
write_concurrency:true
])

These store the follows information in both directions, holding pairs of user IDs.
We can find all the followers of a user by looking up the entries with their user ID as key in the followers_table. And we can find all the users this user follows by looking up the entries with their user ID as key in the following_table.

Both tables are of type :duplicate_bag.
We need multiple entries per key, so we must use either :bag or :duplicate_bag as type of the table. :bag would have the benefit of preventing duplicates, however we chose :duplicate_bag for its performance benefits.

The posts_table represents the source of truth for all the posts in the system. It's a :set storing post IDs as we don't want duplicates here.

:ets.new(@posts_table,[
:set,
:named_table,
:public,
read_concurrency:true,
write_concurrency:true,
decentralized_counters:true
])

In the feeds_table we store the feeds of all users.

:ets.new(@feeds_table,[
:duplicate_bag,
:named_table,
:public,
read_concurrency:true,
write_concurrency:true,
decentralized_counters:true
])

Whenever a new post arrives in our system, we look up all the followers of the author of the post using the followers_table. Then, for each follower we store a pair of the follower's user ID and the post ID in the feeds_table. We chose the type :duplicate_bag here for the same reasons as we did for the followers_table. We need multiple entries per key (user ID) and we want the performance benefits of :duplicate_bag over :bag.

Celebrities

With these tables we would have a functioning system. However, there is still the celebrity_posts_table to talk about.

:ets.new(@celebrity_posts_table,[
:duplicate_bag,
:named_table,
:public,
read_concurrency:true,
write_concurrency:true,
decentralized_counters:true
])

This table is actually a first optimization of the system. You might have wondered what would happen if a post arrives in the system authored by a user who has many many followers. If we would stick to our feeds_table, we would have to loop through all of them to insert the corresponding data.

For this reason, we introduced the celebrity_posts_table. The key idea here is that if a user has too many followers (is a celebrity), we give their posts a special treatment to avoid looping through all the followers and the associated performance issues. Instead we will store pairs of the author and post IDs in celebrity_posts_table, which is again of type :duplicate_bag.

definsert_post(post_id, author_id)do
:ets.insert(@posts_table,{post_id, author_id})

  follower_entries =:ets.lookup(@followers_table, author_id)
  limit =fan_out_limit()

if limit ==:infinityorlength(follower_entries)<= limit do
Enum.each(follower_entries,fn{_subject, follower_id}->
:ets.insert(@feeds_table,{follower_id, post_id})
end)
else
:ets.insert(@celebrity_posts_table,{author_id, post_id})
end
end

When a user request their timeline, we will first look for their feed in the feeds_table, then do another lookup in the celebrity_posts_table to gather posts from celebrities they follow. The user will then see a combination of those posts. So, in fact this dataplane combines fan-out (for regular posts in feeds_table) with fan-in (for celebrity posts in celebrity_posts_table) for performance reasons.

defget_timeline(user_id)do
  fan_out_posts =
@feeds_table
|>:ets.lookup(user_id)
|>Enum.map(&elem(&1,1))

  celebrity_posts =
@following_table
|>:ets.lookup(user_id)
|>Enum.flat_map(fn{_actor, followed_id}->
:ets.lookup(@celebrity_posts_table, followed_id)
end)
|>Enum.map(&elem(&1,1))

  fan_out_posts ++ celebrity_posts
end

Evaluating our dataplane

To get an idea about how much performance we gain with our dataplane, we evaluated it using our simulator. The dashboard below shows the results of this comparison.

We can see two blocks of data:

The first run was recorded with Bluesky's open source dataplane. We can see a maximum p99 latency of more than 60 ms. Furthermore, we see the throughput of requests per second fluctuating around 600.

In comparison, the block on the right shows recordings of the same simulation with our dataplane. The latencies are tiny, the throughput sits smoothly at 800 requests per second.

Crimeline comparison

Crimeline is an experimental timeline builder which caught our attention when doing research for this project. Like our proof of concept, Crimeline is not a complete dataplane implementation. Unlike our proof of concept, Crimeline focuses on creating efficient datastructures and conveniently comes with synthetic benchmarks for them.
After having seen the end-to-end evaluation of our dataplane, we were curious how ETS tables would perform in those benchmarks.

The README describes one of Crimeline's datastructures, the UserMap, as follows:

Sharded adjacency map. Each uid is split via bitmask into shard index (low bits) and backbone index (high bits). Each shard holds a Vec<Vec<Uid>> — a dense backbone of sorted adjacency lists.

It's a struct that holds information whether a user ID is included in it (think your followers on Bluesky) with efficient methods to add and remove user IDs as well as performing the lookup of an ID.

An ETS-based equivalent looks like this:

defmoduleUserMapEtsdo
@user0

def new do
:ets.new(:forward,[:duplicate_bag,:public,{:read_concurrency,true}])
end

defpopulated(n)do
    table =new()
    tuples =for t <-0..(n -1),do:{@user, t *3}
:ets.insert(table, tuples)
    table
end

defadd(table, target)do
:ets.insert(table,{@user, target})
end

defadd_bulk_list(table, targets)do
    tuples =for t <- targets,do:{@user, t}
:ets.insert(table, tuples)
end

defadd_bulk_each(table, targets)do
Enum.each(targets,fn t ->:ets.insert(table,{@user, t})end)
end

defcontains(table, target)do
:ets.match_object(table,{@user, target})!=[]
end

defremove(table, target)do
:ets.delete_object(table,{@user, target})
end

defdestroy(table)do
:ets.delete(table)
end
end

We can expect this to be a bit slower than the Rust implementation of Crimeline due to the overhead of the Erlang VM.
The table below shows the average time for each operation in nanoseconds when performed with 100 users.
From our results, we can see that ETS tables nevertheless perform relatively well.

Operation	Rust UserMap	ETS UserMap
add	18 ns	186 ns
contains_hit	8 ns	1360 ns
contains_miss	8 ns	1350 ns
remove	26 ns	599 ns

So overall we were happy but two benchmarks stood out with growing numbers of users: contains_hit and contains_miss.

Note the units in the following table. While the Rust UserMap goes from 8 ns to 14 ns (not even 2x) when increasing the number of users from 100 to 100_000, ETS goes from 1.36 μs to 1.11 ms, almost 1000x.

Operation (number of users)	Rust UserMap	ETS UserMap
contains_hit (100)	8 ns	1.36 μs
contains_hit (100_000)	14 ns	1.11 ms
contains_miss (100)	8 ns	1.35 μs
contains_miss (100_000)	14 ns	1.11 ms

Why is this so slow? From the ETS docs:

Insert and lookup times in tables of type set are constant, regardless of the table size. For table types bag and duplicate_bag time is proportional to the number of objects with the same key.

We chose a :duplicate_bag for the table to improve insert performance. Now we are paying for that with slow lookups at scale.

Roaring Bitmaps

We didn't give up here, of course. So let's see what we can improve.

During our research, we also came across Jaz's blog and learned how to shrink the size of your datastructures and go even further with Roaring Bitmaps. The latter seems to be a good fit for optimizing our user map datastructure.

Luckily, rustler lowers the barrier to use Rust crates from Elixir a lot these days. Conveniently for us, Aaron Gunderson already did the work to wrap the roaring crate in an Elixir package and blogged about it. So, it's ready for us to use.

As a very brief summary, bitmaps are a suitable datastructure to compress the follower graph in our dataplane. Instead of storing pairs of {user_id, follower_id}, we store a single array for each user where each bit represents whether the user with the id at the index follows the user.

Say we have 5 users, and our ids start at 0. If the first four users all follow user with id 4, we would store the following pairs in our ETS table:

# user_id, follower_id
{4, 0}
{4, 1}
{4, 2}
{4, 3}

With a bitmap, we would store a single array of bits for the user with id 4. If the bit is 1, the corresponding user follows the user with id 4.

# user_id/index:
# 0, 1, 2, 3, 4
 [1, 1, 1, 1, 0]

You can see how that compresses the required space in memory in this simplified example.
Assuming 1 byte per id, we would have 4 * 2 = 8 bytes of memory in our ETS table variant. The bitmap however is complete with 5 bits which fit in a single byte.

Roaring bitmaps apply further optimizations on top of this basic idea.

With the roaring package we can plug the Roaring Bitmap into our user map module.

defmoduleUserMapRoaringdo
def new do
{:ok, roaring}=RoaringBitmap64.new()
    roaring
end

defpopulated(n)do
    list =for t <-0..(n -1),do: t
{:ok, roaring}=RoaringBitmap64.from_list(list)
    roaring
end

defadd(roaring, target)do
RoaringBitmap64.insert(roaring, target)
end

defadd_bulk_list(roaring, targets)do
for t <- targets,do:RoaringBitmap64.insert(roaring, t)
end

defadd_bulk_each(roaring, targets)do
Enum.each(targets,fn t ->RoaringBitmap64.insert(roaring, t)end)
end

defcontains(roaring, target)do
RoaringBitmap64.contains?(roaring, target)
end

defremove(roaring, target)do
RoaringBitmap64.remove(roaring, target)
end

defdestroy(_roaring)do
:ok
end
end

With that we get the following results.

Operation (number of users)	Rust UserMap	ETS UserMap	Roaring Bitmaps UserMap
contains_hit (100)	8 ns	1.36 μs	104 ns
contains_hit (100_000)	14 ns	1.11 ms	100 ns
contains_miss (100)	8 ns	1.35 μs	98 ns
contains_miss (100_000)	14 ns	1.11 ms	100 ns

We're still slower than Crimeline's implementation but it's acceptable and more importantly does not degrade with the number of users.

Comparison in a more realistic scenario

After satisfying our curiosity about Crimeline's synthetic benchmarks, we were wondering how the comparison would look like in a more realistic scenario. Therefore, we added another benchmark feeds that is somewhere in between Crimeline's benchmarks and the work we've previously done with our simulator. Interestingly, in this more realistic scenario the performance difference between the implementations diminishes.

Operation (number of follows)	Rust feed	ETS feed	Roaring Bitmaps feed
get_feed (100)	41 µs	9 μs	8 μs
get_feed (100_000)	57 µs	117 μs	108 μs
get_feed (1_000_000)	58 µs	118 μs	109 μs

Where to go from here?

We showed how to build a basic dataplane implementation in Elixir, then added optimizations to squeeze out the issues we encountered.

One essential feature that is missing from our implementation is persistence. So far we store everything in memory, so our data wouldn't survive a restart of the server.

Our goal is to provide a simple to use dataplane implementation. So, the first idea would be to add persistence with Postgres and treat our in-memory data as an index to avoid costly queries and retrieve the data directly by primary key. Another idea is to provide an adapter interface so that everyone can use their preferred database.

We've seen how to combine Rust and Elixir to get the strengths of both technologies while mitigating their weaknesses, Discord has another interesting article on that topic.
In essence, we can get the high level of abstraction from Elixir which leads to readable, maintainable code and a high velocity. We can also get the parallelism and robustness of the Erlang VM on which Elixir runs. In combination with Rust, we can improve our performance both in speed and memory usage while we are confident that the native code won't crash the VM as we build on Rust's memory-safety features. This way, we can optimize the parts of the codebase where we benefit from it. Whether it's Roaring Bitmaps or even wrapping the Crimeline datastructures, a whole ecosystem of crates is available to us.

With that, we have all the tools to build the remaining features a complete dataplane implementation needs and we are ready to add more advanced features like the social proof that is available in Bluesky.

Make Friends With Your AI Assistant

Mon, 18 May 2026 10:10:08 +0000

Typography

AI assistants do apparently love clean markers. I use ⓪ –⑳ markers for numbering, ▸/▹ markers for itemizing, typographically correct quotes (“”/‘’,) and m-dashes. Compose key makes entering that stuff easier, alternative layout makes it a charm. No tips for poor MacOS/Win users, sorry.

Workflows

I have system-wide workflows for my assistant. They all look like “START_WORD → WORKFLOW_RULES → STOP_WORD.”

Here is an example:

Let’s define the workflow you must follow every time we work on a new task, named NEW_TASK. The rules are following:

① The statement “New task !" typed by me starts new NEW_TASK workflow
② Upon starting new workflow you must pull the main branch from github and create a new branch named ""
③ At this point you have to wait for me to enter the problem statement
④ You have to analyze the problem statement and immediately ask all the questions regarding it where you might have had any uncertancy and/or lack of clarity
⑤ After everythig is clarified, you must come up with the plan of implementation and ask for its amendment and/or approval
⑥ Upon approval, you should start implementing it, asking all the questions you might have appeared
⑦ Once done, you must create a comprehensive tests for the new functionality, documentation in both the source code as standalone (markdown), changelog entry, and a brief what-has-been-done note in markdown
⑧ After I accept everything by saying "Task done!", you should commit everything with a descriptive comprehensive commit message, push it to remote, and create a pull request in github.

The key insight here is the stop-words. Without them the assistant will cheerfully barrel through all eight steps in one breath, commit untested code to main, and open a pull request titled “feat: everything.” Stop-words are the leash. Love the leash (and the muzzle, your assistant is ill-mannered and disobedient.)

Context Is King (And Your Assistant Has Amnesia)

Every new session your assistant is a freshly hatched duckling. It will imprint on whatever you show it first. Feed it your codebase conventions, your style preferences, your testing philosophy—and do it persistently, through rules and system prompts, not through hopeful repetition in every chat.

The difference between a rule that says “always use :json instead of :jason” and manually correcting the same dependency in every generated mix.exs is the difference between engineering and penance.

Persistent rules are not optional. They are the civilizational layer between you and the primordial chaos of a context window that has never heard of your project. Each workflow (see above) might have different persistent rules attached.

Atomic Tasks, or: Stop Writing Novels

The temptation to say “implement the entire authentication system” is roughly as productive as asking an intern to “make the app work.” Break it down. Then break it down again. If your task description exceeds one screenful, you have written a novel, not a task. If within the task there are forks in the road possible, the assistant will inevitable choose the wrong path.

The assistant will cheerfully attempt your novel and wild guess the direction on each fork. It will produce something that compiles, passes zero tests, and makes you question your career choices. Whereas “add the verify_token/2 function that pattern-matches on {:ok, claims} and {:error, reason}”—that it can do in its sleep. If models slept.

The rule of thumb: if there is more than one correct architectural path, the task is too large. Decide the architecture yourself; delegate the typing.

Feed It Good Code, Get Good Code

This is not mysticism, it is pattern matching. If you feed your assistant examples of clean, idiomatic code, it will produce clean, idiomatic code. If you feed it your legacy spaghetti written at 3 AM during a production outage, you will get more spaghetti—but with AI-generated parmesan on top.

I keep a curated set of “golden” modules that I attached as context ages ago. The assistant does not need to be told “you are a senior Elixir architect with 300 years of experience.” It needs to see what good Elixir looks like. Show, don’t pep-talk.

Trust, But Verify (Especially the Tests)

The cruelest irony of AI-assisted development: the assistant generates tests that pass. “Green across the board!” you exclaim, and deploy. Three hours later you discover the tests were testing that the function returns something, not that it returns the right something.

Read the generated tests more carefully than the implementation. The implementation at least looks suspicious when it is wrong. A bad test looks exactly like a good test, except it asserts nothing useful. Pattern-match on the actual values; assert result is not a test, it is a prayer.

The Art of Saying “No, Try Again”

Your assistant is not your therapist. It does not need encouragement. “This is wrong” followed by silence teaches it nothing. “This is wrong because the recursive clause does not handle the empty-list base case, see List.foldl/3 for the pattern I expect”—now you are getting somewhere. Keep your paw right above Ctrl+C combination and press it every time you see the assistant’s thinking process drives bonkers.

You always liked interrupting people in person, feel free to interrupt your deputy.

Know When to Kill the Session

If you have been going back and forth for more than three iterations on the same function, close the session. Start fresh. The context window is polluted with failed attempts, corrections, and mutual disappointment—a couples therapy transcript, essentially.

A new session with a clean description of what you actually want will outperform the seventh round of “no, I meant the other thing” every single time. Sunk cost fallacy applies to token budgets too.

Please aware, that the necessity to close a session vividly implies your original prompt sucked in the first place. Rewrite it from scratch. Make sure it has no freedom for the assistant to choose paths. They are good at walking the straight roads.

The Checklist

Distilled to a grocery list for the impatient:

▸ Use persistent rules, not per-session rituals
▸ Define workflows with explicit start/stop words
▸ One architectural decision per task, maximum
▸ Attach good code as examples, skip the motivational preamble
▸ Run the full validation pipeline (format, credo, dialyzer, test) after every step, not at the end
▸ Read generated tests as if they were written by someone who wants to trick you (they were not, but the effect is the same)
▸ Three failed iterations = kill the session, rethink the task
▸ Never let the assistant commit without your explicit say-so
▸ If you could not solve the problem yourself, the assistant cannot solve it for you—it can only type faster

The last point deserves a tattoo. An AI assistant is a force multiplier, not a force. Multiply zero by anything you like.

Happy prompting.

Mocks Are Your Friends, Not Your Servants

Sun, 17 May 2026 18:03:59 +0000

There is a peculiar tradition in our industry: the moment someone says “mock,” half the room recoils as if you have just proposed replacing the database with a spreadsheet. The other half nods enthusiastically, having already replaced the database with a spreadsheet.

Both camps are wrong, and for the same reason: they think of “mock” as a verb.

Mock Is a Noun

José Valim wrote a splendid piece on the subject some years ago, and yet the industry continues to collectively ignore its central thesis with the dedication of a cat ignoring an expensive toy. The key insight is disarmingly simple: consider “mock” to be a noun, never a verb. You do not “mock a module.” You create a mock—an entity, a collaborator, a contract participant—that implements a well-defined behaviour.

The difference is not cosmetic. When you “mock” a function (verb), you are lying to your test. You are saying: “pretend this hole in the wall is a door.” When you create a mock (noun), you are building an actual door—one that opens and closes according to the same interface contract as the production door, just perhaps into a smaller room.

Stubs Are for Candles, Not for Software

The prevailing use of mocks in the wild amounts to: “I need this function to return 42 so my other function does not crash.” This is not testing. This is bribery. You have paid off a witness to give the testimony you wanted, and now you are surprised when the real trial goes sideways.

A mock that merely returns canned responses is a stub wearing a fake moustache. It tells you nothing about whether your code actually interoperates with the expected behaviour. It only tells you that, given a universe where everything behaves exactly as you imagined, your code does not crash. Congratulations—that was never the hard part.

The hard part is the contract. Does your code send the right messages? Does it handle the responses defined by the behaviour? Does it respect the protocol, not just survive it? A proper mock enforces these questions. A stub sweeps them under the carpet and charges you for the cleaning.

Promote Your Mocks to Lead Actors

Here is the heresy: mocks should not sit quietly in the corner of your test, responding when spoken to like well-trained waitstaff. They should be protagonists. They should drive the testing narrative.

Consider a finite state machine. It transitions through states, fires callbacks, notifies listeners. In production, the listener might persist data, send emails, ring bells. In tests, you do not care about the bells. You care about: did the FSM reach state X with payload Y after event Z?

A mock-as-protagonist answers this question directly. It is not merely absorbing calls—it is reporting back to the test, asserting that the system did what it promised. The mock becomes your embedded journalist, filing reports from inside the process under test.

This is a fundamentally different posture. The mock is no longer a shift worker you hire to stand in for the real employee. It is a first-class participant in the test, with its own responsibilities, its own assertions, its own voice.

OTP, Race Conditions, and the Debugger You Deserve

Anyone who has tested concurrent OTP systems knows the special joy of a test that passes ninety-nine times and fails on the hundredth, always at random time, always on CI, never on your machine. The core problem is structural: you fire an asynchronous message and then try to assert about a state that may or may not have been reached yet. Process.sleep(200) is not a solution—it is a prayer in milliseconds.

Mocks offer something better. When a mock is registered as a listener, every state transition sends a message back to the test process. You can then use assert_receive to wait for the transition, with a timeout, deterministically. No sleep. No prayer. No coin flip.

In effect, the mock becomes a breakpoint in a debugger you never had to open. It declares: “I expect to be called with these arguments, in this order,” and it sends a message to the test process confirming each call. You get both the expectation (expect/3) and the synchronization (assert_receive/2) in one mechanism. The mock is simultaneously the probe and the signal.

This is not theoretical. In OTP, where processes communicate via message passing and the scheduler is free to interleave execution in whatever order amuses it most, this pattern transforms flaky tests into deterministic ones. You are no longer guessing when the process reached the desired state. You are being told.

Listeners, Visibility, and the Joy of Seeing What Happens

There is a secondary benefit that deserves its own paragraph: visibility.

When you plug a mock in as a listener—not as a replacement for an implementation, but as an observer of one—the test code becomes radically more readable. Each assertion is a statement about what the system did, not about what you had to simulate. The reader does not need to mentally reconstruct the production flow from a pile of stubs. The flow is right there, reported by the mock, step by step.

Test code that reads like a narrative of actual system behaviour is test code that people trust. And test code that people trust is test code that people maintain. And test code that people maintain is test code that catches bugs. The chain of causation is longer than a Dickensian sentence, but every link holds.

Finitomata: A Case Study in Mock-Driven Testing

All of this is not armchair philosophy. The Finitomata.ExUnit module is a working testing framework built entirely on this premise. It uses Mox not as a crutch but as the foundation.

The setup declares a mock listener. The mock is allow-ed to the FSM process and given expectations for after_transition/3 callbacks. Each expectation sends {:on_transition, id, state, payload} back to the test process. The test then walks the FSM through its transitions and asserts each state deterministically via assert_receive.

The result looks like this:

test_path "respectful passenger", %{passengers: initial_passengers} do
  :coin_in ->
    assert_state :opened do
      assert_payload do
        data.passengers ~> ^initial_passengers
      end
    end

  :walk_in ->
    assert_state :closed do
      assert_payload do
        data.passengers ~> one_more when one_more == 1 + data.passengers
      end
    end

  :switch_off ->
    assert_state :switched_off
    assert_state :*
end

No Process.sleep. No polling. No race conditions. Each transition is confirmed by the mock-listener before the test proceeds. And crucially: intermediate states—including those triggered by automatic (bang!) transitions that the test never explicitly fires—are observable and assertable, because the mock reports every single one.

The mock here is not a stand-in. It is the entire testing infrastructure. Remove it, and you are back to Process.sleep(200) and crossed fingers.

The Moral

Mocks have suffered from a branding problem. They were introduced to the mainstream as “a way to avoid calling the real thing,” which is roughly equivalent to introducing a violin as “a way to avoid silence.” Technically correct, but missing the point so thoroughly that it constitutes its own genre of wrongness.

A mock is a collaborator. A contract enforcer. A synchronisation primitive. A visibility layer. A debugger breakpoint. A protagonist.

Treat it accordingly, and your tests will repay you with the one thing no amount of Process.sleep can buy: confidence.

Passkey Sign In with Elixir and Phoenix

Fri, 08 May 2026 09:29:25 +0000

Passkeys are a replacement for passwords that use public/private cryptographic key pairs for login in a way that can be more user-friendly and resistant to a number of kinds of attacks. Let’s implement account registration and login with passkeys in a simple Phoenix web app.

This work is heavily based on this article by Adam Langley, which provides a great deal of information about implementing passkeys. My goal here is to fill in some more of the details, and provide some Elixir-specific information. As with Adam’s article, I’m not going to use any WebAuthn libraries (even though that may be advisable from a security/maintenance perspective) since I think it’s interesting and helpful to understand how things actually work.

Providing an exhaustive, production-ready implementation is a non-goal of this post. I’m going to make some slightly odd decisions for pedagogical reasons, and leave some things incomplete. That said, I’ll try to note when I’m doing so.

To start, I’m using the default Phoenix template app (less Tailwind) in which I’ve also generated the default, controller-based authentication system. Some parts of the password-specific stuff have been stripped out altogether, others will get changed to fit with the passkey authentication setup we’re going to build.

Database schema §

The first thing we’ll need is a backend schema for passkeys. The only data we need to store for a particular passkey is a unique identifier, and the public key itself. We also want users to be able to have multiple passkeys associated with their account (since they may want to login from multiple platforms that don’t cross-sync passkeys), so the users schema will have a one-to-many relationship with the passkeys.

The actual model I’ll call UserCredential, since that’s closer to what the WebAuthn spec calls them. Here’s the schema, it’s pretty simple:

defmodule PhoenixPasskeys.Accounts.UserCredential do
  use Ecto.Schema
  import Ecto.Changeset

  @primary_key {:id, :binary, []}

  schema "users_credentials" do
    # DER-encoded Subject Public Key Info: https://datatracker.ietf.org/doc/html/rfc5280#section-4.1.2.7
    field :public_key_spki, :binary
    belongs_to :user, PhoenixPasskeys.Accounts.User
    timestamps()
  end

  def changeset(credential, attrs) do
    credential
    |> cast(attrs, [:id, :public_key_spki, :user_id])
  end
end

There are a couple things to note here:

First, we’re explicitly specifying that the primary key is a binary, since that’s what the WebAuthn API provides as the credential ID.
The public_key_spki field contains the data for the public key itself and the algorithm being used. We don’t care about the specific format, though, since we don’t have to parse it ourselves.

To the generated User schema, I also added the has_many side of the relationship. There’s also a migration to create the credentials table. I won’t show it here, since it’s exactly what you’d expect—just make sure the id column is a binary.

Registration JavaScript §

WebAuthn, being a modern web API, is a JavaScript API. This is a distinct disadvantage, if your users expect to be able to login from JavaScript-less browsers. Nonetheless, we proceed with the JS. Here’s the first bit:

document.addEventListener("DOMContentLoaded", () => {
    const registrationForm = document.getElementById("registration-form");
    if (registrationForm) {
        registrationForm.addEventListener("submit", (event) => {
            event.preventDefault();
            registerWebAuthnAccount(registrationForm);
        });
    }
});

If we find the registration <form> element (note that I’ve added an ID (and also removed the password field)), we add a submit listener to it which prevents the form submission and instead calls the registerWebAuthnAccount function we’ll create next.

Before we get there, we’ll also write a brief helper function to check whether passkeys are actually available:

async function supportsPasskeys() {
    if (!window.PublicKeyCredential || !PublicKeyCredential.isConditionalMediationAvailable) {
		return false;
	}
	const [conditional, userVerifiying] = await Promise.all([
		PublicKeyCredential.isConditionalMediationAvailable(),
		PublicKeyCredential.isUserVerifyingPlatformAuthenticatorAvailable(),
	]);
	return conditional && userVerifiying;
}

The conditional mediation check establishes that WebAuthn conditional UI is available. This is what we’ll use for login, and is part of what makes using passkeys a nice, slick experience. Conditional UI will let us start a request for a passkey that doesn’t present anything until the user interacts with the browser’s passkey autofill UI.

The user-verifying platform authenticator check establishes that, well, there is a user-verifying platform authenticator. The platform authenticator part means an authenticator that’s part of the user’s device, not removable, and the user-verifying part means that the authenticator verifies the presence of the user (such as via biometrics).

In the registerWebAuthnAccount function, the first thing we need to do is check that both these conditions are met and passkeys are supported by the browser. If not, we’ll just bail out and registration won’t be possible.

async function registerWebAuthnAccount(form) {
    if (!(await supportsPasskeys())) {
        return;
    }
}

Next, we’ll setup a big ol’ options object that we’ll pass to the WebAuthn API to specify what sort of credential we want. There’s going to be a whole lot of stuff here, so lets take it one piece at a time.

The RP is the Relying Party—that is, us, the party which relies on the credentials for authentication. The ID is the domain of the RP—just localhost for this example, though in reality you’d need to use your actual domain in production. The name is just a user-facing name for the RP.

Next up is some info about the user who the credential is for. The user’s “name” will just be the email that they entered in the form. The displayName value is required, per the spec, but we don’t have any other information so we just leave it blank and let the browser display the name only. The ID is where this gets a little weird, since we’re just generating a random 64-byte value:

function generateUserID() {
	const userID = new Uint8Array(64);
	crypto.getRandomValues(userID);
	return userID;
}

If you were adding a passkey to an existing user account, you could use a server-specified value for the user ID and the browser would replace any existing credentials with the same user ID and RP ID. But, since the user is registering a new account at this point, we assume they don’t want to do that (and, moreover, we don’t have any ID we can use yet). The user ID will also be returned upon a successful login, which would let us look up the user that’s logging in. However, since we’re only going to allow a credential to belong to a single user, we can use the credential’s ID to uniquely determine the user.

Next up, we specify the types of public keys that we’ll accept. We’re going to support Ed25519, ES256, and RS256, since those are recommended by the WebAuthn spec:

During the login process, the server will generate a challenge value that the client will sign and the server will verify was signed with the user’s private key. The spec also requires that we provide a challenge when creating a credential, but we’re not going to use it for anything (since the user is just now creating the credential, we have nothing trusted that we can verify the initial challenge against), so we just provide an empty value:

Lastly, we give our requirements for authenticators that we want to use:

We want only the platform authenticator, not anything else, like removable security keys. We also specify that we want a resident key. This usage of “resident” is deprecated terminology that’s enshrined in the API. Really it means we want a discoverable credential, so that the authenticator will surface it during the login process without us having to request the credential by ID. This is important to note, since it’s what prevents needing a separate username entry step.

Now that we (finally) have all the configuration options in place, we can actually proceed with the credential creation. We pass the options to navigator.credentials.create to actually create the WebAuthn credential. If that fails, we’ll just take the easy way out and alert to inform the user (in an actual service, you’d probably want better error handling).

From the credential we get back, we need a few pieces of information. First is the decoded client data, which is an object that contains information about the credential creation request that occurred.

The clientDataJSON field of the response object contains the JSON-serialized object in an ArrayBuffer, so we decode that to text and then parse the JSON. With the decoded object, we do a consistency check with a couple pieces of data: the type of the request, whether or not it was cross-origin, and the actual origin being used.

If it was not a creation request, the request was cross-origin, or the origin doesn’t match, we bail out. Note that in production, the origin should be checked against the actual production origin, not localhost. And again, in reality you’d want better error handling than just an alert.

Next, we need to get the authenticator data which is encoded in a binary format and pull a few pieces of data out of it. You can see the full format of the authenticator data in the spec, but the parts we’re interested in are the backed-up state and the credential ID, which is part of the attested credential data.

async function registerWebAuthnAccount(form) {
    // ...
    const authenticatorData = new Uint8Array(credential.response.getAuthenticatorData());
    const backedUp = (authenticatorData[32] >> 4) & 1;
    const idLength = (authenticatorData[53] << 8) | authenticatorData[54];
    const id = authenticatorData.slice(55, 55 + idLength);
}

We get the backed-up bit from the flags byte at offset 32. Then we get the length of the credential ID, which is encoded as a big-endian, 16-bit integer at bytes 53 and 54. The ID itself immediately follows the length, thus starting at byte 55.

Before proceeding, we check that the backed-up bit is set, indicating that the credential is backed-up and safe from the user losing the current device. If it’s not, we won’t let the user register with this passkey. I choose to do this since it’s recommended by Adam Langley’s blog post, but whether it’s actually necessary may depend on your specific circumstances.

The last piece of data we need out of the credential is the actual public key:

async function registerWebAuthnAccount(form) {
    // ...
    const publicKey = credential.response.getPublicKey();
}

And with all that in place, we can actually initiate the registration. We’ll assemble a form data payload with all of the requisite values:

async function registerWebAuthnAccount(form) {
    // ...
    const body = new FormData();
    body.append("_csrf_token", form._csrf_token.value);
    body.append("email", form["user[email]"].value);
    body.append("credential_id", arrayBufferToBase64(id));
    body.append("public_key_spki", arrayBufferToBase64(publicKey));
}

We’ll use the body in a POST request to the registration endpoint. The response we get back will contain a status value to indicate whether the request was successful or not. If it was successful, the backend will have set the session cookie, and we can redirect to the home page and the new user will be logged in. If registration failed, we’ll alert the user.

And with that, we’re done with JavaScript (for now) and can move on to the backend part of registering an account.

Creating a user §

In the UserRegistrationController module that comes with the Phoenix auth template, we’ll change the create function. By default, it registers the user using the parameters from the signup form and then redirects to the homepage. Instead, we’re going to register using the passkey we created on the client and then respond with the JSON that our JavaScript is expecting.

The first thing we need to do is extract the values that were sent by the frontend and decode the base 64-encoded ones.

defmodule PhoenixPasskeysWeb.UserRegistrationController do
  def create(conn, %{
        "email" => email,
        "credential_id" => credential_id,
        "public_key_spki" => public_key_spki
      }) do
    credential_id = Base.decode64!(credential_id)
    public_key_spki = Base.decode64!(public_key_spki)
  end
end

Those get passed to the Accounts.register_user function, which we’ll update shortly to handle. If account creation succeeded, we’ll still send the confirmation email as the existing code did. After that, instead of redirecting, we’ll log the user in by setting the session cookie and then respond with the “ok” status for the frontend. If account creation fails, we’ll just respond with the “error” status so the frontend can alert the user.

defmodule PhoenixPasskeysWeb.UserRegistrationController do
  def create(...) do
    # ...
    case Accounts.register_user(email, credential_id, public_key_spki) do
      {:ok, user} ->
        # Send confirmation email...

        conn
        |> UserAuth.log_in_user_without_redirect(user)
        |> json(%{status: :ok})

      {:error, _changeset} ->
        json(conn, %{status: :error})
    end
  end
end

Let’s update the register_user function. Instead of just creating a changeset for the user and then inserting it, we need to also create the authenticator. To avoid potentially leaving things in a broken state, we wrap both of these in a database transaction.

defmodule PhoenixPasskeys.Accounts do
  def register_user(email, credential_id, public_key_spki) do
    Repo.transaction(fn ->
      user =
        %User{}
        |> User.registration_changeset(%{email: email})
        |> Repo.insert()
        |> case do
          {:ok, user} -> user
          {:error, changeset} -> Repo.rollback(changeset)
        end
    end)
  end
end

First, we create a user with the given email. If the user creation fails, we abort and rollback the transaction. Then, we can create a credential belonging to the new user with the credential ID and public key we received from the client. As with the user, if creating the credential fails, we rollback the transaction.

defmodule PhoenixPasskeys.Accounts do
  def register_user(email, credential_id, public_key_spki) do
    Repo.transaction(fn ->
      # ...
      %UserCredential{}
      |> UserCredential.changeset(%{
        id: credential_id,
        public_key_spki: public_key_spki,
        user_id: user.id
      })
      |> Repo.insert()
      |> case do
        {:ok, _credential} -> nil
        {:error, changeset} -> Repo.rollback(changeset)
      end
    end)
  end
end

We don’t need to do anything with the newly created credential, so we can just ignore it once it’s been created.

And finally, from the transaction function, we return the user:

The last thing we need to to complete the registration flow is to set the new user’s session cookie so that they’re logged in immediately. The UserAuth module that’s generated as part of the Phoenix auth template has a log_in_user function that does exactly this. But it also redirects the connection to another endpoint. We don’t want to do that, since we’re sending a JSON response, so I’ve split the function into two: one that only sets the session, and the existing function that sets the session and then redirects.

defmodule PhoenixPasskeysWeb.UserAuth do
  def log_in_user(conn, user, params \\ %{}) do
    user_return_to = get_session(conn, :user_return_to)
    conn
    |> log_in_user_without_redirect(user, params)
    |> redirect(to: user_return_to || signed_in_path(conn))
  end

  def log_in_user_without_redirect(conn, user, params \\ %{}) do
    token = Accounts.generate_user_session_token(user)
    conn
    |> renew_session()
    |> put_token_in_session(token)
    |> maybe_write_remember_me_cookie(token, params)
  end
end

And with that, everything is in place and you can now create an account with a passkey!

Login form §

Now that the user’s got an account, they need to be able to login with it. That means once again interacting with the WebAuthn API and writing a bunch of JavaScript. But before we get there, we need some slight changes to the backend.

In the HTML for the login page, we’ll add an ID to the <form> element so that we can find it from JS. We’ll also remove the password field, which is obviously no longer necessary. Lastly, but certainly not least, we need to send a challenge to the client.

The challenge is a value that the user’s device will cryptographically sign with their private key. The result will get sent back to the server, and we’ll verify it against the public key we have stored thus authenticating them. We’ll send the challenge just in a hidden form field:

<input type="hidden" id="challenge" value={@webauthn_challenge} />

In the session controller, we’ll need to generate and assign the challenge to the connection.

WebAuthn expects the challenge to be a value up to 64 bytes long, so we’ll use the Erlang crypto module to generate one of that length. The value is encoded as URL-safe base 64 (the same as normal base 64, but with dash and underscore rather than plus and slash) without padding. We encode it this way since that’s the format in which it will later be returned as part of the clientDataJSON, so when we extract that value we can directly compare it to the challenge value we generated.

defmodule PhoenixPasskeysWeb.UserSessionController do
  defp put_webauthn_challenge(conn) do
    challenge =
      :crypto.strong_rand_bytes(64)
      |> Base.url_encode64(padding: false)

    conn
    |> put_session(:webauthn_challenge, challenge)
    |> assign(:webauthn_challenge, challenge)
  end
end

Note that the challenge string is also stored in the session, so that we can later check that the challenge that the client signed matches the challenge we generated. It’s safe to store this in the session, even though it’s sent to the client, because the cookie is encrypted and signed so the client can’t tamper with it.

Login JavaScript §

With the first part of the backend changes taken care of, it’s time for more JavaScript, baby!

We’ll follow a similar outline to the registration setup (and the same caveat applies about error handling).

document.addEventListener("DOMContentLoaded", () => {
    // ...
    const loginForm = document.getElementById("login-form");
	if (loginForm) {
        loginWebAuthnAccount(loginForm);
    }
});

async function loginWebAuthnAccount(loginForm) {
    if (!(await supportsPasskeys())) {
        return;
    }
}

The first thing we need to do is grab the challenge from the hidden form field, then we can construct the options object for getting the credential (which is, thankfully, much simpler than for creation).

In the options, we specify that we want conditional mediation. As noted before, this means that the browser won’t display any UI, except for autofill, for this credential request until the user accepts the autofill suggestion. In the public key options, we also give the decoded challenge value and specify the our Relying Party ID (again, this would need to be the actual domain in production).

Now, we can actually make the credential request and then, if we get a credential back, encode and send all the values to the backend. We need to send the ID of the credential, so that the backend can find its public key and the corresponding user. We also need the client data JSON, which we send as text decoded from the ArrayBuffer it’s returned as. We also need to send the authenticator data as well as the signature itself.

async function loginWebAuthnAccount(loginForm) {
    // ...
    const credential = await navigator.credentials.get(getOptions);
    if (!credential) {
        alert("Could not get credential");
        return;
    }

    const clientDataJSON = new TextDecoder().decode(credential.response.clientDataJSON);

    const body = new FormData();
    body.append("_csrf_token", loginForm._csrf_token.value);
    body.append("raw_id", arrayBufferToBase64(credential.rawId));
    body.append("client_data_json", clientDataJSON);
    body.append("authenticator_data", arrayBufferToBase64(credential.response.authenticatorData));
    body.append("signature", arrayBufferToBase64(credential.response.signature));
}

We can then send a request to the login endpoint. If the backend request fails (or if we failed to get the credential) we just alert the user (again, you’d probably want something better in reality). If the login attempt was successful, the server will have set the session cookie, and so we can just redirect to the homepage and the user will be logged in.

With that in place, let’s move on to the backend half of the login request.

Validating a login attempt §

As with signup, we’ll modify the existing log in endpoint to actually validate the WebAuthn login attempt.

The first step is extracting all of the information the frontend provides in the params and decoding it:

defmodule PhoenixPasskeysWeb.UserSessionController do
  def create(conn, params) do
    id = params |> Map.get("raw_id") |> Base.decode64!()
    authenticator_data = params |> Map.get("authenticator_data") |> Base.decode64!()
    client_data_json_str = params |> Map.get("client_data_json")
    signature = params |> Map.get("signature") |> Base.decode64!()
  end
end

Next, we’re going to validate all of the information we got from the client. Before we get to that, there are a handful of helper functions we’ll use. First, looking up a credential by its ID:

Next, a function in the controller that verifies the signature against the provided data:

defmodule PhoenixPasskeysWeb.UserSessionController do
  defp verify_signature(credential, client_data_json_str, authenticator_data, signature) do
    with {:ok, pubkey} <- X509.PublicKey.from_der(credential.public_key_spki),
         client_data_json_hash <- :crypto.hash(:sha256, client_data_json_str),
         signed_message <- authenticator_data <> client_data_json_hash,
         true <- :public_key.verify(signed_message, :sha256, signature, pubkey) do
      true
    else
      _ ->
        false
    end
  end
end

X509 comes from the x509 package, which is the only third-party piece of code we’re using. It’s a fairly thin wrapper around the Erlang public_key and crypto modules, and mostly serves to save me from having to deal with Erlang records in my code. Its from_der helper function is used to parse the public key from the encoded format.

Next, we hash the client data JSON and append that hash to the authenticator data from the client. This value is what should match the signature using the public key we’ve got, so finally we check that. If all these steps succeeded, we return true, and false otherwise.

The last helper function will receive the decoded client data make sure it’s got all of the values that we expect. If the crossOrigin value is present and is not false, the client data is invalid and the login attempt will be rejected.

Otherwise, we check that the data has the expected type and origin, and we extract the challenge value (note that we’re checking the origin again here, and this would need to change in production):

And lastly, if neither of the previous patterns matched, the client data fails validation:

Now, let’s put all those parts together and validate the login attempt.

defmodule PhoenixPasskeysWeb.UserSessionController do
  def create(conn, params) do
    # ...
    with credential when not is_nil(credential) <- Accounts.get_credential(id),
         true <- verify_signature(credential, client_data_json_str, authenticator_data, signature),
         {:ok, client_data_json} <- Jason.decode(client_data_json_str),
         {:ok, challenge} <- check_client_data_json(client_data_json),
         true <- challenge == get_session(conn, :webauthn_challenge),
         true <- :binary.part(authenticator_data, 0, 32) == :crypto.hash(:sha256, "localhost"),
         true <- (:binary.at(authenticator_data, 32) &&& 1) == 1 do
      conn
      |> delete_session(:webauthn_challenge)
      |> UserAuth.log_in_user_without_redirect(credential.user)
      |> json(%{status: :ok})
    else
      _ ->
        json(conn, %{status: :error})
    end
  end
end

Here’s everything that we’re doing:

Lookup the credential with the given ID.
Use the public key we had stored to verify the signature on the authenticator data and client data JSON.
Decode the client data JSON.
Check that all the values in the client data are what we expect, and extract the challenge that was signed.
Ensure the challenge that the user signed matches what we previously generated.
Extract the hash of the origin from the authenticator data and ensure it matches our origin (this would not be localhost in production).
Check that the authenticator data has the user presence bit set (indicating that a person was actually present on the client’s end).

If all of those steps succeeded, we remove the old challenge value from the session (since it’s no longer needed), actually log the user in, and then respond with the “ok” status that the JavaScript is expecting. If any step failed, we’ll respond with the “error” status and the frontend will alert the user.

Since there’s a lot going on here, it’s worth being clear about what exactly in this process lets us authenticate and prove that the user is who they claim to be. Since the signature verification step is using the public key that we stored during the registration process, we know that anyone that can produce a valid signature using that public key must be the user (or at any rate, have their private key). The value that they’re signing is, essentially, the challenge: a securely generated random value. The user isn’t directly signing the challenge, but this is still safe, since the challenge value is included in the client data JSON, that hash of which is included in the signed message.

So: the challenge value that was signed by the user must be in the client data, and the challenge value in the client data must be the one we generated. Given that, we know that the user whose key was used to sign the message is the one trying to log in now. That we’re verifying with the stored public key prevents an attacker from using an arbitrary key to sign the login attempt. And that the signed challenge matches the challenge the server generated means an attacker can’t reuse a previous response to login (a replay attack).

At long last, we finally have the ability to log in to our application using a passkey. Only a few minor things to go, so let’s forge ahead.

Handling login if the user enters an email §

Although we’re presenting the conditional UI, there’s nothing preventing the user from typing their email into the field and then clicking “Sign in,” so we should probably handle that to. This can be done fairly simply by reusing our existing code for conditional login.

We’ll change the loginWebAuthnAccount to take an additional parameter, conditional, which will be a boolean indicating whether this login attempt is to setup the conditional UI or triggered by submitting the login form.

If it’s false, we won’t request conditional mediation and instead we’ll look up the credentials corresponding to the email the user entered and ask WebAuthn for one of those:

async function loginWebAuthnAccount(loginForm, conditional) {
    // ...
    let allowCredentials = [];
    if (!conditional) {
		const email = loginForm["user[email]"].value;
		const resp = await fetch(`/users/log_in/credentials?email=${email}`);
		const respJSON = await resp.json();
		allowCredentials = respJSON.map((id) => {
			return {
				type: "public-key",
				id: base64ToArrayBuffer(id),
			};
		});
    }

    const getOptions = {
		mediation: conditional ? "conditional" : "optional",
		publicKey: {
			challenge: base64URLToArrayBuffer(challenge),
			rpId: "localhost",
			allowCredentials,
		}
    };
    // ...
}

The “optional” value for the mediation option means that the authenticator isn’t required to display UI, but will do so if its policies dictate that. The allowCredentials array contains objects describing all of the credentials that we want to accept—specifically, their binary IDs as ArrayBuffers.

We look up the user’s credentials so that the one we actually request from the authenticator matches the account that the user is trying to log in with. To handle this, we’ll also wire up an additional route on the backend that returns the base 64-encoded IDs of all the credentials belonging to the user with a given email.

The get_credentials_by_email function is quite simple. It just looks up a user by email, preloading any credentials they have and then returning them:

Back in the JS, we can tweak the setup code to pass true for the conditional parameter in the initial request and also register a submit handler on the login form that will invoke it with false:

document.addEventListener("DOMContentLoaded", () => {
    // ...
	const loginForm = document.getElementById("login-form");
	if (loginForm) {
		loginWebAuthnAccount(loginForm, true);
		loginForm.addEventListener("submit", (event) => {
			event.preventDefault();
			loginWebAuthnAccount(loginForm, false);
		});
	}
});

And so we’ve handled the case where the user ignores the conditional UI and still types in their email to log in.

Passkey reset §

A not-infrequent argument against passkeys is that if your phone falls into a lake, you lose access to all of your passkey-backed accounts. One could argue that this isn’t true because the definition of passkeys means that they’re backed up and thus protected from this sort of event (and indeed, earlier we only permitted registering with backed-up credentials). I think the argument isn’t particularly interesting, however, because you can still have a “Forgot my passkey” option that works just like it does now with passwords.

This is less secure than a passkey implementation that has no “Forgot” option. But it’s no less secure than current password-based systems, and I think the UX/security tradeoff here falls on the UX side—people will, inevitably, lose access to their passkeys while retaining access to their email.

Implementing isn’t too complicated, fortunately, since we can reuse much of the registration code. First, the JavaScript. The only change necessary is attaching the registration function to the reset form as well.

document.addEventListener("DOMContentLoaded", () => {
    // ...
	const resetForm = document.getElementById("reset-passkey-form");
	if (resetForm) {
		resetForm.addEventListener("submit", (event) => {
			event.preventDefault();
			registerWebAuthnAccount(resetForm);
		});
	}
});

In the HTML for the reset form, we we need to include the email in the same form field as the registration form (and also add an ID to the <form> element):

<input type="hidden" name="user[email]" value={@user.email} />

In the function for the reset route (which is changed to POST from PUT, to match the signup route), we take the credential ID and public key and use them to update the user’s credentials:

defmodule PhoenixPasskeysWeb.UserResetPasswordController do
  def update(conn, %{
        "credential_id" => credential_id,
        "public_key_spki" => public_key_spki
      }) do
    credential_id = Base.decode64!(credential_id)
    public_key_spki = Base.decode64!(public_key_spki)
    case Accounts.reset_user_credentials(conn.assigns.user, credential_id, public_key_spki) do
      :ok ->
        conn
        |> put_flash(:info, "Passkey reset successfully.")
        |> UserAuth.log_in_user_without_redirect(conn.assigns.user)
        |> json(%{status: :ok})

      {:error, _} ->
        conn
        |> put_flash(:error, "Error resetting passkey")
        |> json(%{status: :error})
    end
  end
end

After updating, we log the user in if applicable and return JSON with the appropriate status.

The reset_user_credentials function works very similarly to the original reset password function that was part of the template: it deletes all the user’s existing sessions, and then removes their existing credentials and creates a new one:

defmodule PhoenixPasskeys.Accounts do
  def reset_user_credentials(user, credential_id, public_key_spki) do
    Ecto.Multi.new()
    |> Ecto.Multi.delete_all(
      :old_credentials,
      from(a in UserCredential, where: a.user_id == ^user.id)
    )
    |> Ecto.Multi.insert(
      :new_credential,
      UserCredential.changeset(%UserCredential{}, %{
        id: credential_id,
        public_key_spki: public_key_spki,
        user_id: user.id
      })
    )
    |> Ecto.Multi.delete_all(:tokens, UserToken.user_and_contexts_query(user, :all))
    |> Repo.transaction()
    |> case do
      {:ok, _} -> :ok
      {:error, _, changeset, _} -> {:error, changeset}
    end
  end
end

It’s worth noting that this does have slightly weaker security properties than the phx.gen.auth reset password implementation. With that approach, leaking a password reset token does not necessarily result in an account takeover, since whoever obtained the leaked token may not know the target user’s email address. Since the auth template forces the user to re-login after a reset, this prevents someone without the email address from gaining access even if they change the password.

But since logging in with a passkey is functionally a single factor, resetting it means gaining access to the account. So, a leaked reset token gives the bearer control over the account. This is an argument against having a reset option, but whether this is a concern in practice depends on your specific circumstances.

Conclusion §

As noted, this is not a complete implementation. There are a handful of places where I’ve left things unfinished since this isn’t meant to be production-level code. There are also a few places where there are security decisions that need to be made on a more contextual basis, that I’ve tried to note. And, of course, you wouldn’t really want to only permit signing in with passkeys and wholesale drop the passwords column from your database.

Nonetheless, I hope this has been a helpful look at how to implement passkeys in an Elixir/Phoenix application. You can find the complete repo here, and it may also be useful to look at the specific commit where passkey support was added.

From legacy code to verifiable specifications with Surveyor

Wed, 29 Apr 2026 20:10:39 +0000

What is legacy code?

Sometimes that's the fifteen-year-old monolith. Sometimes it's the service that grew complicated faster than the team could keep up with. Sometimes it's that project that started as a shortcut and turned into a roadblock. Legacy code has been exposed to real world demands long enough to gather fixes for edge cases we are not aware of anymore.

And — increasingly — sometimes it's the repo your CEO vibe-coded last weekend and that they proudly present on Monday. It runs. Now you have to own it.

Legacy software is the code we lost confidence in

That last category is genuinely new and growing fast. Code written once, never reviewed, tested a bit, deployed one day. Or, worse, released as a library and never maintained after that.

A new kind of technical dept has evolved: "The code we should review thoroughly one day."

"Can we rewrite this?"

The actual problem is not the writing of the new code, but the discovery of what the old code does, and the certainty that the new one does the same thing.

We've been building two small Elixir tools to help with both halves of that problem.

Surveyor scans a codebase in any language and produces an architectural model.
Assay runs the same plain-text behavioral specs against both the legacy and the rewrite, so you can prove they behave the same.

Together they cover the workflow: Figure out the structure, capture the behavior, rewrite, prove the rewrite works.

TL;DR

Surveyor produces an automatically verifiable behavioral specification of a legacy system.

How we got here

Before going into either tool, it's worth walking through the shape of the problem. The picture builds up step by step, and the final diagram only makes sense once you've seen the holes in the earlier ones.

What we don't want

A direct rewrite also copies bugs, dead code and unused features.

Point an agent at a legacy codebase, ask it to produce a new one, hope for the best. Call it "the Claude zombie rewrite." You end up with a target codebase that nobody understands either, plus a generation gap between the old assumptions and the new. What has the agent overlooked, misunderstood, assumed? A black box gets replaced with a different black box. That is not a rewrite, that is a transcoding.

What we want instead

The features of the legacy codebase need to be revised

The valuable artifact in the middle is not new code — it's a readable specification of what the old code actually does. Plain text, human-reviewable, version-controllable. The kind of thing you need when you want to ask the product owner if a feature is really wanted that way. - Or if it is just a legacy quirk.

If you have it, you can use it for estimates. It can be your map to plan the implementation and might even give you a hint about that stakeholder that you might have forgotten about otherwise.

Organised and illustrated

The specification's organisation should mirror the codebase

A single big text file describing a whole legacy system is only useful for very small projects. You want the spec broken down by module — by bounded context — and the modules themselves arranged on a map of the architecture. Once each module has its own spec associated to it, two things become possible: you can divide the work, and you can reason about coverage.

Job done! Or is it?

We now have and organised map of the system and a set of specifications. We have an overview, and the details. We could now talk to the stakeholders, the development team, and set to work.

The specification's organisation should mirror the codebase

However, we have not yet verified our findings.

What we are looking at is not yet a specification of the legacy system — it is a speculation of the legacy system. A document that claims to describe what the code does, written by reading the code (or by asking an LLM to read the code). Plausible, well-organised, reviewable. But unverified.

1. Verifiability

This is speculation as specification, and it's the trap most "let's document the legacy system" projects fall into. A Word document of "what the system does" can be worse than no document at all, because people start trusting it. But even if the specification is done as a group effort by all stakeholders involved, and is thoroughly verified, it still lacks one critical property:

2. Reproducibility

What we actually want is a verifiable specification: Not "verified once," but verifiable on demand, any time someone asks the question. You could verify a paper spec by hand, of course — sit down, read it, run the system, tick off the scenarios — but manual verification is expensive enough that it gets done once at sign-off and never again.

A verification you don't actually do is verification that doesn't exist. And unless the legacy codebase is a museum exhibit, it's a moving target:

The team is still shipping fixes, the system is still drifting, and a paper spec written on Monday is no longer accurate by Friday.

Runnable specs

The fix is to make the specs executable — to wire them into something that can run them against the actual legacy system, on demand, and tell you whether they still hold.

That is the move from speculation to verifiable specification: not a one-shot audit, but a button you can press on every commit, every nightly, every time anyone asks "is this still true?"

A green run proves the spec is still true. A red run is a useful signal: either the legacy drifted, or your spec was wrong. Either way, you find out before the rewrite, not during it.

That magic is Assay

Assay is the runner. It parses the .assay specs Surveyor produced, matches each step against bindings you write, and exercises the real legacy system through whatever interface it actually exposes — HTTP, CLI, message queue, file drop, whatever. The framework is small.

The runtime contract is "your bindings + your assertions + a deterministic runner."

The systemic approach

Open up that "parts you fill in" box and you find the actual surface area of work: pattern recognition for step matching, a binding per target system, and Given/When/Then automations for setup, action, and verification.

The end state has a property that nothing in the earlier pictures had: the problem space has been dissected into manageable pieces. Architecture, behavior, target adapters, assertions — each is small, each can be reviewed independently, and each can be picked up by a human or an agent without having to hold the whole system in their head at once.

That's the goal of the toolkit. Everything below is the detail of how each piece is built.

Surveyor — discovering the architecture you inherited

Surveyor is a Mix project that takes a path to a codebase and produces a Structurizr DSL workspace file. It runs in three phases mapping to the C4 model: C1 (system context), C2 (containers), C3 (components per container).

Crucially, Surveyor does not parse code. There are no language-specific parsers, no AST walkers, no fragile grammar files for thirty different ecosystems.

It scans the filesystem to identify what's there — languages, frameworks, project files, entry points, deployment manifests — and then feeds meaningful chunks to an LLM. That sounds trivial, but is the same mechanism that commercial coding agents use when they analyse your code.

Surveyor's intelligence is in the prompts and the chunking, not in a stack of half-broken language adapters.

The CLI is interactive at every phase.

$ surveyor ./legacy-monolith --phase c1

Phase 1 — System Context
  Scanning codebase...
  Querying LLM...

  System: "Order Management System"
  Description: Manages the full order lifecycle

  Actors:
    ✓ Customer — places and tracks orders via Web UI
    ✓ Warehouse Staff — manages fulfillment via Back Office
    ? Admin — found references in auth config (confidence: low)

  External Systems:
    ✓ Stripe — payment processing (REST API)
    ? SendGrid — found API key in config (confidence: low)

[a]ccept  [e]dit  [r]etry with more context  [q]uit
>

The LLM is asked to flag uncertainty. Anything tagged confidence: low shows up with a ? and a short reasoning string for human review. That's not just nice ergonomics — it's the bit that makes the tool trustworthy. An LLM that confidently invents a PaymentReconciliation container that doesn't exist is worse than no model at all. An LLM that says "I see a SENDGRID_API_KEY in .env.example but no Sendgrid client in the source, please confirm" is doing the right kind of work.

Each phase is resumable. Results are saved as JSON in ./surveyor/, so a thirty-container system doesn't have to finish in one sitting. You can hop in, accept C1, take a break, come back tomorrow, and resume at C2.

The end product is a workspace.dsl you can render in Structurizr. But more importantly, it's a workspace.dsl whose components are decorated with assay.specs and assay.schema properties, pointing at the behavioral specs and the domain schemas for each bounded context.

orderLifecycle = component "Order Lifecycle" "Orders" "Bounded Context" {
    properties {
        "assay.specs" "specs/order-lifecycle"
        "assay.schema" "schemas/order_lifecycle.ex"
    }
}

DDD pattern annotations from the LLM (customer-supplier, anticorruption-layer, …) ride along as properties on the relationship. Every component gets assay.specs and assay.schema properties — the contract the next tool reads from.

Examples

The System Context of a Medical Application

Surveyor identified the actors and the external systems of the application. Without knowing the code, we would already know which actor types there are

The System Context of a medical application

An example of an assay spec created in the second phase:

The Account Creation Specs of a medical application

Assay — proving the rewrite behaves the same

Assay is a minimal behavioral spec runner. Specs are plain text and look like this:

Component: [orderLifecycle] Order Lifecycle
Context: Order Placement

Definitions:
  - "a valid customer" means:
      a Customer with status Active, verified email,
      and a credit limit greater than zero

Invariants:
  - total must not exceed customer credit limit
  - at least one line item required

Rule: Customers can place orders for in-stock items

  @critical @phase-1
  Scenario: [OL-001] Place a simple order
    Given a valid customer "Alice" with credit limit €10,000
    And product "Widget" is in stock with 50 units available
    When Alice places an order for 3 units of "Widget"
    Then the order status becomes "Placed"
    And stock for "Widget" is reduced to 47 units

The bracketed [orderLifecycle] is a workspace.dsl identifier — the same identifier Surveyor wrote into the architecture model. The bracketed [OL-001] on the scenario is an optional free-form id that survives across rewrites of the spec text and lets you cross-reference from tickets or audit trails.

If that looks like Cucumber or Gherkin, it does — with one large difference. There is no glue layer, no abstraction over what kind of system you are testing, no framework opinions about HTTP, databases, or browsers. The bindings are just Elixir.

defmoduleTargets.Legacy.OrderLifecycledo
useAssay.Binding,component:"orderLifecycle"

@baseSystem.get_env("LEGACY_API_URL")

  step :given,~r/a valid customer "(?P<name>.+)" with credit limit €(?P<limit>[\d,.]+)/do
{:ok, resp}=Req.post("#{@base}/test/customers",json:%{
name:params().name,
credit_limit:parse_money(params().limit),
status:"Active"
})
assign(customer_id: resp.body["id"])
end

  step :action,~r/.+ places an order for (?P<qty>\d+) units? of "(?P<product>.+)"/do
{:ok, resp}=Req.post("#{@base}/orders",json:%{
customer_id:var(:customer_id),
lines:[%{product_id:var(:product_id),quantity:to_int(params().qty)}]
})
assign(order_id: resp.body["id"],http_status: resp.status)
end

  step :expect,~r/the order status becomes "(?P<status>.+)"/do
{:ok, resp}=Req.get("#{@base}/orders/#{var(:order_id)}")
    assert resp.body["status"]==params().status
end
end

A binding is just a module that brings whatever it needs — Req for an HTTP API, AMQP for a message broker, System.cmd for a batch processor, File for a directory-watching pipeline. Assay itself does not ship an HTTP client or a database adapter.

The framework provides parsing, step matching, scenario lifecycle, and assertions; the rest is your code. That's what makes the same spec runnable against an HTTP API, a CLI tool, a batch job, or a message queue, depending on what the legacy system happens to be.

🚀 Would you like to run this on your legacy codebase? Sign up for early access! Count me in!

The crucial property is target swappability:

assay run specs/order-lifecycle/ --target legacy
assay run specs/order-lifecycle/ --target new

Same specs. Different bindings. Different systems. The legacy binding hits the old SOAP API; the new binding hits the new Phoenix endpoint. If both go green, the rewrite is behavior-equivalent for the things the spec covers — which, after a few months of writing specs, is most of the things that matter.

There is no separate .exs file generated from the specs. The runner parses .assay files, matches each step against a regex on a binding function, executes them, and prints pass/fail. That's it.

How Assay actually works

The runtime is around six hundred lines of Elixir. Five design choices do most of the work.

A two-pass parser

The first pass lifts triple-quoted doc strings out of the file (so the line tokenizer doesn't have to reason about them). The second pass dispatches on the first keyword on each line.

defparse_string(content)do
{filtered, doc_strings}=
    content
|>String.split("\n")
|>Enum.with_index(1)
|>extract_doc_strings()

Process.put(:assay_doc_strings, doc_strings)

  filtered
|>Enum.reject(fn{line, _}->
    trimmed =String.trim(line)
    trimmed ==""orString.starts_with?(trimmed,"#")
end)
|>parse_lines(%Spec{})
end

Output is a plain %Spec{} struct. No AST nodes, no string interpolation.

The step macro generates real module functions

Each step :given, ~r/.../ do ... end call expands into a uniquely-named function plus a %StepBinding{} record on an accumulating attribute.

defmacrostep(type, regex,do: block)
when type in[:given,:action,:expect]do
  fn_name = :"__step_#{:erlang.unique_integer([:positive])}__"

quotedo
@assay_steps{unquote(type),unquote(regex),&__MODULE__.unquote(fn_name)/0}

defunquote(fn_name)()do
unquote(block)
end
end
end

At compile time, __before_compile__ exposes __steps__/0 and __component__/0 for the runner to read. There is no string codegen, no eval — just normal Elixir function definitions.

Component-scoped step matching

Anyone who has used Cucumber has hit the global step-definition problem: the same step text means different things in different contexts, but Gherkin treats step definitions as one shared namespace. Assay scopes bindings by component:

defprun_spec(spec, bindings, target, tags, exclude_tags)do
  component_bindings =
Enum.filter(bindings,fn b -> b.component == spec.component end)

# ... run each scenario against component_bindings only
end

This is the direct reason Surveyor and Assay use the same identifier convention. The architecture model and the spec runner share a vocabulary, and step text never collides across bounded contexts.

Pre-check, then execute

Before running any step, the runner walks the whole scenario and finds a binding for every step. If anything is unbound, the scenario fails before any side effects.

defrun_scenario(scenario, bindings)do
  matched_steps =
Enum.map(scenario.steps,fn step ->
casefind_binding(step, bindings)do
nil->{:unbound, step}
        binding ->{:bound, step, binding}
end
end)

ifEnum.any?(matched_steps,&match?({:unbound, _},&1))do
%ScenarioResult{status::error,error:"Unbound steps found"}
else
execute_matched_steps(scenario, matched_steps, bindings)
end
end

This is what stops a scenario from running half its Givens, hitting an unbound When, and leaving a dangling test customer in the database.

Per-scenario state in the process dictionary

Each scenario gets a fresh context — a tiny module that stores variables, params, doc strings, and data tables in the process dictionary.

def init do
Process.put(@vars_key,%{})
Process.put(@params_key,%{})
end

defget_var(name)do
Process.get(@vars_key,%{})|>Map.get(name)
end

defset_vars(keyword)do
  vars =Process.get(@vars_key,%{})
Process.put(@vars_key,Enum.into(keyword, vars))
end

The runner can save/0 and restore/1 the snapshot, which is how Assay tests itself by running its own .assay specs through itself in the same process.

There is no plugin system, no dependency injection, no scenario hooks beyond cleanup. Adding any of those would push complexity into the framework and out of the binding, which is the wrong direction. The framework exists to be small enough that you can read it on a Friday afternoon and trust it on Monday.

How they fit together

The flow is straightforward and it maps onto the five phases laid out in the Assay handbook.

Discovery (Surveyor). Run Surveyor against the legacy codebase. Walk through the C1/C2/C3 output interactively. Edit, retry, accept. Produces workspace.dsl.
Behavioral extraction. For each component in the model, write .assay specs and an Elixir schema. The assay.specs and assay.schema properties on every Structurizr component tell you exactly where they go: specs/<context>/ and schemas/<context>.ex.
Validation. Write bindings against the legacy system in targets/legacy/. Run assay run specs/ --target legacy. Iterate until green. You now have a verified, executable specification of the legacy system's behavior.
Stabilization. Review what's covered, what's missing, what's flagged as ambiguous. Get sign-off on scope.
Rewrite. Build the new system. Write bindings against it in targets/new/. Run assay run specs/ --target new. When that's green, the rewrite is done — at least for the surface area the specs cover.

The reason the two tools sit next to each other is that the architecture model is what gives the spec work structure. Without a model, "write specs for the legacy system" is an open-ended task with no obvious stopping point. With the model, every component has a specs/ directory and the question becomes "is each context's behavior covered?" That's tractable.

🚀 Would you like to run this on your legacy codebase? Sign up for early access! Count me in!

The cross-target story is what makes the rewrite verifiable. The same parsed .assay scenarios are dispatched, step by step, through two different binding modules at two different points in the project's life:

Same spec, two bindings, two systems. The runner doesn't know or care which target it's dispatching to — that's the property that makes the rewrite a measurable thing rather than a leap of faith.

Four audiences, one artifact

Four very different parties end up reading — or executing — the same .assay file: the product owner, the developer, the coding agent, and the test runner. The interesting observation is that they don't conflict on content — they conflict on form.

Product owners want to know what the system does in business terms, what would break if you changed X, and where the risk lives. They don't want UML. They want to read scenarios in their domain vocabulary, in plain language, and trust them.

Developers want to know how the system is structured, where the seams are, and what invariants they mustn't break. And — crucially — they want the documentation to be wrong less often than the code is. The moment docs lie, developers stop reading them.

Coding agents want machine-parseable, unambiguous, addressable artifacts with stable identifiers. They want to be able to say "the assertion at assay://orderLifecycle/order-placement/OL-001/step-3 failed" and have that mean something durable. They want types, schemas, and graphs they can traverse.

Automated tests want executable bindings — the layer Assay provides — and they want failures that point back to the spec, not just to a line of code.

Same facts, four presentations.

The .assay file gives the product owner readable scenarios in domain language. The workspace.dsl plus the schema modules give the developer the structural truth. The bracketed identifiers (Component: [orderLifecycle], Scenario: [OL-001]) give the agent its stable addresses. The runner turns the whole thing into a regression suite. None of these audiences is asked to read the others' representation; they all read the same spec, surfaced at the level of detail they need.

Where the LLM lives, and where it doesn't

A reasonable question at this point: with all the recent enthusiasm for LLMs, why isn't more of this LLM-driven?

The split is deliberate.

Surveyor uses an LLM because architectural discovery is genuinely a language task — you're reading config files, route definitions, deployment manifests, dependency lock files, and inferring "this is the API, that's the worker, that's the read model." That's exactly the kind of fuzzy, evidence-weighing job an LLM does well, especially when you tell it to flag uncertainty rather than guess.

Assay does not use an LLM at runtime. The runner is deterministic: parse the spec, match the regex, execute the binding, assert. A behavioral spec that sometimes passes and sometimes does not, depending on which model variant happened to answer, is not a spec.

LLMs are very welcome on the authoring side — drafting .assay files from legacy source is a great agent task, and the handbook has explicit guidance for coding agents about how to do that without inventing behavior. But the green/red signal at the end has to come from a deterministic runner, or it is not really a signal.

🚀 Sign up for early access:Count me in!

How you use them on a project

In practice, the first day of a legacy rewrite looks something like this:

Clone the repo. Run Surveyor with --phase c1 to get the system in context. Discuss the actors and external systems with the client — this alone surfaces "wait, who is the Admin role?" conversations that would otherwise happen three months in.
Run --phase c2 to map containers. This is where you discover the data plane that nobody documented, the cron job running on a forgotten server, the queue that two services share. Surveyor flags confidence: low items; the human resolves them.
Run --phase c3 per container. By the end you have a workspace.dsl that is, for the first time in the project's history, an accurate picture of what is deployed.
Pick the highest-risk bounded context. Read the legacy source. Draft .assay specs for its behavior. Write a legacy binding. Run it. Iterate until green.
Repeat per context, in priority order, until coverage is good enough to start the rewrite.
Build the new system, one bounded context at a time, with targets/new/ bindings going green as each context comes online.

The thing both tools are optimized for is the same: making the work legible. A legacy rewrite is a long project with shifting personnel and a nervous client. Both Surveyor and Assay produce artifacts — a workspace.dsl and a directory of .assay files — that survive turnover, show progress, and that the client can actually read.

Status and next steps

Both tools are ready to be tested in real world scenarios. This is where we need your help:

If you have a legacy system staring you down and a rewrite on the roadmap, get in touch. We would love to hear from you!

Elixir: Introduction

Sun, 26 Apr 2026 14:28:15 +0000

Elixir: Introduction

Head of the Agents and Assistants Department

Thu, 23 Apr 2026 10:20:28 +0000

Let me state upfront: my attitude toward AI assistants cannot be expressed as a boolean value. If you need an answer to the question posed point-blank—“New York Yankees or Boston Red Sox?”—I do not watch baseball at all; I’m a Barça fan. That said, I find AI assistants a perfectly legitimate and even liquid asset. The text below is an account of what made my work with agents pleasant and reduced their errors and rough edges to an acceptable minimum.

A little under a year ago I began working on Cure, a programming language with dependent types, finite state machines as first-class citizens, SMT verification, and other niceties, compiling to the BEAM.

My first approach to the apparatus ended in ignominious failure. I got tangled in my own architectural decisions, started piling on crutches wherever they fit, turned the code completely into an Italian restaurant menu, and lost heart. In a fit of idiocy I had an assistant generate the website and showed it to the public—the ideas themselves were intriguing enough, dependent types and SMT solvers on the BEAM are hardly redundant, and I harboured a quiet hope for community interest. The community correctly identified the language’s site as slop generated by a language model and received my attempt with arctic indifference. I got a great many “nothing works” responses and not a single coherent suggestion for improvement (no fault of the community’s—against the backdrop of the slop-flood of those days, my project would not have looked like Noah’s Ark even to an extremely charitable observer).

I stepped back, examined my creation from all angles, and was forced to admit: I had produced a monster. I saw no chance of licking it into shape, even through a global refactor. I bought a ream of paper and some pencils and started drawing, in order to understand exactly where I had gone wrong. (Spoiler: I had been so enchanted by the idea itself, and so desperate to get something to launch and run, that I had done literally everything wrong.)

I had no intention of giving up—and before me, full height, loomed the necessity of rewriting everything from scratch without repeating the mistakes. By that point I knew that artificial assistants could significantly accelerate the actual writing of code, so I started by erecting scaffolding around the future project. It was obvious to me then (I can now confirm that former intuition with experience) that all those prompts along the lines of “You are a genius architect with three hundred years of experience designing languages with dependent types and SMT solvers” work no better than the morning pep talk to an intern at standup: “You are a great programmer who has written three hundred million billion lines of code without a debugger.” If an assistant tasked with writing coherent code to a spec needed motivational gibberish to function, it would not be worth using under any circumstances whatsoever. Seriously, think about it: a model is asked to implement a GCD module using the Euclidean algorithm—are you really suggesting its deeply baked-in internal rules will not guide it down the correct branches of the conditional operators without first being told it has the soul of a prima ballerina and an avant-garde poet? What on earth does “You are a senior architect” actually mean? Do the people who advocate this believe that without such a preamble the training pathways via the memoirs of axolotl breeders will activate instead?

So, instead of all those skills/agents/whatever, I started by feeding this mechanised beast the source code of all my own libraries, lovingly written by hand, with the note: “Here are examples of good code. Write like this. Not like that—do not write like that.” I know it is immodest, but it is my assistant. You are welcome to feed yours your own code.

Then I reclaimed the wasteland: thus were born Metastatic, implementing MetaAST for different languages across different paradigms, and Ragex—a RAG built on AST rather than plain text (my hunch that AST fits into a context window far more easily and is far better structured than raw source code turned out to be correct).

My task was building a new language; constructing a new ecosystem from scratch was not part of the brief. So I analysed existing solutions—Rust, Go, Elm, Gleam—and chose the one I considered most mature (I never promised the project would be neutral with respect to my tastes and preferences). I simply copied the Elixir ecosystem and added to it what I had personally found lacking over the last ten years. Thanks to standing on the shoulders of giants in this regard, the model wrote almost the entire ecosystem for me; I simply told it: “Look how beautifully new project creation is handled in Elixir—do the same for Cure.” Language models are strong at translation, and Elixir is considerably more intelligible to them compared to almost every other language.

So, before the first line of code, my backpack already held: the right AST—a language in which the assistant and I can communicate far more easily than in homespun English—a handcrafted RAG, and a clear understanding that every step must be a simple, atomic change. The fewer choices the assistant has to make between two paths, the cleaner the result. This principle outweighs the quality of all prompts combined.

Next I had to solve the problem of validating the produced code. My eyes are sharp, but they occasionally miss non-obvious bad decisions in review. Thus was born oeditus_credo—a set of nearly forty additional credo checks covering vulnerabilities, anti-patterns, and the like. The library also ships a mix oeditus_assistant_rules command, a rules generator for the soulless assistant. To those rules I also added: after each stage, verify that mix format && mix credo --strict && mix dialyzer && mix test passes; update all documentation; add regression tests for new code; then run all regression tests and confirm they are still as green as my face the morning after a party.

Every reasonably significant stage also ends with creating an “example” in the examples folder. Something for people to look at, and the regression tests never idle.

At this point I felt ready to start writing actual code. Cure has certain critical parts I wrote by hand from scratch. Every other pull request gets manual edits from me. Every bug I find through manual testing I fix by hand (the obvious ones aside). And yet I reached the desired result considerably faster than if I had written every line in Vim.

Over the course of working on Cure I learned to keep the number of errors—and consequently manual edits—to a minimum. A fairly substantial release, v0.26.0, required not a single correction, for example. Here is the distillate of my rules for communicating with an assistant, in case it proves useful to anyone:

the task must be self-contained but not too large; “first define the types for numeric, then we will write the converters” does not work
inside the task there must be no undefined ambiguities that will derail our T9; there must be exactly one path to the solution
before tackling any task, demand an implementation plan and edit it until all ambiguities have vanished
the task must be roughly solved in your own head before turning to the assistant; otherwise the odds of agreeing with an incorrect solution are uncomfortably high
generated code must be comprehensible and elegant, “rewrite this nicely, I fed you three gigabytes of rules” does not work; if the solution is aesthetically repellent, there is a problem with the task formulation—close the session and start over
setting a task and going for coffee is the direct path to the infinite iterations described in the previous point; the stream of unconsciousness must be watched and any attempt to stray from the planned path killed without mercy
finally, if it seems to you that the cognitive load is decreasing and that any cook could now implement such a project—you need pharmaceutical intervention; your fingers tire less, yes, but if you could not have implemented the project from scratch in a text editor, the LLM is no help to you; it will generate something suspiciously twitching, no argument there, but the first halfway-serious complexity requiring a considered architectural decision will put a cross on the whole enterprise.

That is my experience. Yours may differ; I can bear with you on that matter.

Try https://cure-lang.org and maybe you will like it. The site now has a playground where you can experiment with types in real time, and an almost real console where you can play with the REPL without installing anything locally.

Happy curing!

Scotty, I need warp speed in three minutes - Hauleth

Mon, 20 Apr 2026 23:02:18 +0000

In my last larger gig I worked on fascinating project - Postgres connection pooler written in Elixir. Unfortunately, due to different circumstances, this project burned me out to the ground. However what doesn't kill you ~~is crap not a weapon~~ can become great learning experience.

Most of my achievements in this project were related to performance. This project contains very tight loop in form of query handler, that needed to run hundreds of thousands times per second per user connection. That mean that this functions are very sensitive to even slightest performance changes. And that was my task - to find potential improvements that can be made to make this codebase be much faster.

After departing from Supabase I liked the project so much (mostly as learning ground) that I have created my own fork, where unrestrained from all business side of the project I could focus purely on squeezing as much of performance as I can. This project now lives as Ultravisor - it is still nowhere near being done in a way that I like, but I still go back to work on it from time to time to find potential performance improvements.

This is a story of things that I have done and learned during that journey.

Beware: It is a retrospection, so in some places my memory may be not the best.

Here I need to provide some explanation first, about how Ultravisor works with database connections. It provides 2 modes of operation:

session - where each connection from user to Ultravisor checks out one connection from Ultravisor to database. It checks out once, at the start of connection, and then holds connection until the end;
transaction - where on connection there is nothing done. Client connects to the Ultravisor and can keep that connection indefinitely without ever bothering database. Database connection is checked out only when there is some request from user and is returned to the pool as soon as that result of that query is returned and DB is ready for next one.

While session mode is quite on par with other implementations of connection pooling for Postgres, transaction mode is where performance is lacking and is the main focus is put. In whole article (unless mentioned otherwise) I will speak about transaction mode of Ultravisor.

Lesson: Flame graphs and call tracing is essential#

Pretty obvious thing, but still valuable lesson for any performance optimisation endeavour. For that the great thanks to Trevor Brown and his awesome project eFlambè. This helped a lot in tracing hot points in the running code.

Unfortunately this project seems to be less active recently and has some missing features, like listening for given duration instead of function calls count. This can be partially fixed by simply listening for count of calls to handle_event/4 function given times and then running cat *.bggg to concatenate all files into larger trace. That has disadvantages, but at least it was workable within Speedoscope which I also highly recommend to anyone who needs to work on such optimisation.

While flame graphs are awesome, there is cost to gathering them with eFlambè - it greatly affects performance. Fortunately Erlang has some built in tools that have lesser performance impact, and the "most modern" of these is tprof. This tool is pretty easy to use, but is less detailed than eFlambè. But even with that limitation, it provides superb insight into stuff that has greatest impact on performance, as well as it make it easier to work on long running processes, as it work asynchronously, so you can "manually" decide how long you want to trace your process.

Summary: Knowing where your bottlenecks are is essential for performance optimisations.

Lesson: Doing less can improve performance#

Obvious thing that need to be stated - doing nothing is faster than doing something. Extracting amount of data sent over given socket using :inet.getstat/2 call is fast, but not free. That involves some waiting for response from either port or process handling connection, which introduces slowdown. Two possible solutions there are:

Do not gather that metric at all - sensible, but not feasible, especially when you use that metric to charge your users.
Gather that data less often.

The approach I have taken there is obviously 2., and the solution is dumb simple - debouncer.

Debouncing is an interesting technique often used in user interfaces where you accept some event, and then for some period you ignore repeated events. The reason for that is that our interfaces may have flaws that send repeated events one after another.

In this case Ultravisor tries to store amount of sent data after each query, but that can get expensive for many short queries. Instead I have implemented simple per-process debouncer:

This stores returned data in process dictionary (per-process mutable space with quick access) and if there was no call in given time-period, then we process data again. This is safe way to do so, as :inet.getstat/2 will always return amount of data that socket processed since it started, so data between calls will be accounted.

Before:

tps = 79401.392762 (without initial connection time)

After (10ms of debouncing):

tps = 80069.646510 (without initial connection time)

After (100ms of debouncing):

tps = 80568.825937 (without initial connection time)

Summary: Doing noting is more performant than doing something. Sometimes doing nothing can be quite easy.

Lesson: Telemetry is not free#

When working on most projects, especially Phoenix-based, one can slap :telemetry.execute/3 calls everywhere and notice no performance degradation¹. Unfortunately, when you do hundreds of thousands calls a second - that is not a case.

For unaware readers - Telemetry is Erlang event dispatching system for observability events.

In this project the metrics are exposed in Prometheus/OpenMetrics format, which means that there needs to be collection system within the application. In BEAM applications the standard way to implement that is to use ETS tables to store recorded values. Fortunately there are libraries to handle that for you, and for the longest time "gold standard" for it was telemetry_prometheus_core library created by Telemetry core team.

While for most projects that library is performant enough (because metrics aren't recorded in quite tight loops), in case of this project that was not a case. There, metrics gathering is still one of the hottest spot in the codebase, even with all improvements that have been done.

Excerpt from tprof profile:

Function	Calls count	Per call (μs)	Percentage
`Peep.EventHandler.store_metrics/5`	1421911	0.13	4.21%
`Peep.Storage.Striped.insert_metric/5`	904852	0.26	5.11%

This is with awesome library Peep by Richard Kallos. When using telemetry_prometheus_core it was simply the most expensive thing in whole loop. Just replacing metrics gathering library with Peep gave us about 2x bump in TPS.

Summary: Telemetry handler can matter in tight loops. Fast metrics gathering isn't easy.

Lesson: Records instead of maps or structs#

Elixir uses structs for structured data. That gives a lot nice features wrt. hot code reloads, compilation graph dependencies, and other. However, because structs are maps, there is a cost. Maps have O(log n) access time to the fields, this is how maps are constructed in memory. While smaller maps have slightly different (better in most cases) characteristics, there is strict requirement that you keep your structure with less than 31 fields² and it still has slight memory overhead. The alternative is to use records. These have better performance characteristic (always constant) irrelevant of the amount of fields at the cost of being slightly more rigid (records are tuple based) and less convenient to use (experience may vary). Additional advantage in my opinion is that it is harder to add incorrect field by using Map module.

Current (OTP 28) limit for small map is 32 keys, but Elixir uses one key for struct name, hence 31 fields is the limit.

Before you will run and change all structs in your system to records, just remember - most of the time the difference doesn't matter - just use structures.

Before:

tps = 81765.266264 (without initial connection time)

After:

tps = 82147.855889 (without initial connection time)

Summary: Records are super handy when you need to squeeze each bit of performance. It doesn't provide much, but these adds up.

Lesson: ETS tables are super fast, but not always#

ETS is Erlang's built-in module for storing key-value data in mutable way. Like built-in Redis. This structure allows for sharing some data in a way, that is easy to access from different parts of the system. One example of system that is using ETS for storing their information is Telemetry (mentioned above).

While for 99% of the use cases Telemetry will be fast enough, it has some problems with tight loops. Main problem is that it will always copy data from table to the caller process. That mean that it can put high memory pressure on the process that tries to retrieve data.

Fortunately Erlang supports another mechanism for storing globally accessible data - persistent_term. Of course, there is no such thing as "free lunch" so it has substantial disadvantage - it works poorly³ with data that changes often, as removing or changing data in a key will require walk through all processes to copy data from it to processes that may use it into process memory. However - Telemetry handlers should not change a lot, you should just set them once as soon as your system start, and then ideally they will not change ever again.

There is slight optimisation that makes it fast in some cases (single word values, like atoms), but that is not the case there, so we can ignore that.

Before⁴:

tps = 76914.004685 (without initial connection time)

After:

tps = 78479.006634 (without initial connection time)

⁴

If you wonder why these results are lower than in previous section, it is because test conditions are identical only per section, not cross sections. In this particular case I have ran benchmark while collecting metrics (to show difference in persistent_term change) while other are ran without metrics to not pollute results.

Summary: persistent_term is awesome and super fast, so if you know that you have some data that will probably never change and will be requested constantly, then it may be good place to store that data.

Lesson: Calling your `GenServer`s is fast, but not 90k times per second fast#

One of the interesting observations that I have spotted is that if there are longer running queries, ones that send more data over the network than just simple short responses, then the difference between Ultravisor and "state of the art" tools like PgBouncer or PgDog (that are written in non-managed languages like C and Rust) is much smaller (obviously it is still there, but it is on par, not substantially off).

I needed to dig more, what can be the cause of such strange behaviour. The reason was found in place where I least expected it - checking out database connection to be used.

Flame graph showed that almost third of the time is spent on checking out database connections, and most of that time is spent in 2 function calls, both of them are internally gen_statem calls and in both most time is spent on sleeping (aka, waiting for reply).

Now, this one is hard thing to optimise, as in Elixir there is no mutability (almost, we will get there). This mean that if I want some form of shared queue of processes, then I need to use separate process to keep state of the queue for us, and then do GenServer calls to fetch that state. What I did in such situation? What any unreasonable Elixir developer obsessed with performance would do - NIF⁵.

⁵

I wanted to use ETS there, but for that to work it lacks function like ets:take/2 that would return only one element from the tables with type bag or duplicate_bag. Or any other form of just taking out any (possibly random) element from ETS table in atomic way.

The implementation is rather basic wrapper over VecDeques that allow popping single element from that queue without any message passing. The implementation is very crude, nowhere production ready. It doesn't provide any form of worker restarts or anything, but works quite well as PoC of what is possible.

New queue also provides a way to store additional "metadata" alongside the worker PID. This allows me to store DB connection socket next to connection process, which removes need for additional call to extract that data to pass requests directly to other DB, without copying data between processes.

Before:

tps = 83619.640673 (without initial connection time)

After:

tps = 94191.475386 (without initial connection time)

Summary: Sometimes one need to get creative to get around platform limitations. This may require some pesky NIFs though.

Conclusions#

Optimising such project was enormous fun and I think that at the current state there is nothing extra that can be done to optimise it more without optimising generated JIT-ed native code or optimising Erlang scheduler.

There are some flags, that affect performance, but as it is currently unclear why these work at all (probably it is related more to the OS scheduler rather than Erlang performance), I left them out of this article for now.

Post Scriptum: Good tooling helps a lot#

Just after I have started that optimisation project after leaving Supabase I started using Jujutsu for version control. That one thing helped me a lot with being able to have separate branches/PRs for each of the changes, while at the same being able to work with mega-merge of them all.

That allows me to profile code with all other noise removed, while still exposing the changes as separate reviewable units. Without that support I would need to decipher what have already been changed and/or removed from the profile.

Additional feature that I heavily used there is "anonymous branching". As when working with JJ I do not need to create new name for each branch that I want to try, it was way easier to implement one idea, then just do jj new @- (which branches off at the commit that is parent of the current one) and just implement alternative idea. I used that constantly to compare ideas and reject failed concepts.

∎

Roastidio.us Tagged with elixir

Privacy by Design no JurisOS: Criptografia AES-256 de Ponta a Ponta com Cloak.Ecto, SQLCipher e Tailscale VPN - DEV Community

1. O Desafio da Conformidade (LGPD, GDPR e CCPA)

2. Separação de Chaves: SQLCipher e Cloak.Ecto

Camada 1: Criptografia de Disco com SQLCipher (SQLCIPHER_KEY)

Camada 2: Criptografia de Nível de Campo com Cloak.Ecto (CLOAK_SECRET_KEY)

3. Isolamento da Malha BEAM via Tailscale VPN

Os Benefícios dessa Arquitetura de Rede:

Conclusão

Egghead is a note-taking app. - by Mark Wunsch

The problem with notes

How to write good notes

Why Artificial Intelligence

The problems with AI in knowledge management

The memory of AI assistants

Multi-Agent Systems

The capabilities of agents

What is Egghead?

The goal of taking notes in Egghead

Using Elixir head version with Mise

1. Build Elixir

2. Link it in mise

3. Verify this works

Ecto, on_replace and deferred checks

The problem

The solution

the phoenix template has too much stuff

Anti-Antipatterns in Elixir Library Guidelines

1. Avoid Application Configuration

2. Avoid Compile-Time Application Configuration

3. Avoid Using Exceptions for Control-Flow

4. Avoid Working with Invalid Data

5. Avoid Defining Modules Outside Your Namespace

6. Avoid use When an import Is Enough

7. Avoid Macros

8. Avoid Using Processes for Code Organization

9. Avoid Spawning Unsupervised Processes

Summary

kmx.io blog : KC3 v0.1.17 released

KC3 v0.1.17 released

New in this release :

The notification that gave up after a minute | jvalol

The notification that gave up after a minute

Notes from the dead letter box - Specifying Software for Teams and Agents

How to describe an application?​

The question: What was in that envelope?​

The Premise​

Architectural overview​

C the fourth​

Describing Behavior​

The Case for BDD​

The Assay format​

Schema Definitions​

Architectural Decision​

Non Functional Requirements​

The End-Boss: Design​

The imageto effectively reconsider decisions once the Circumstances have changed.​

The code​

Visual Regression Testing​

So, what is in that envelope?​

CORS in Phoenix

Recursion in Elixir

FizzBuzz in Elixir

Guards! Guards! - Hauleth

Guards! Guards!

This week in infrastructure

Exploring Programming Languages

Elixir for a Bluesky DataPlane: the choice we didn't expect

The component nobody knows about​

A tale of two workloads​

The hot path: timelines served from memory​

The follower graph: in memory, but not naively​

So what do we actually need from a language?​

The four candidates​

Go​

Rust​

Node / TypeScript​

Elixir​

Every candidate has a flaw — which ones can you fix?​

Resolving the Elixir paradox​

Camada 1: Criptografia de Disco com SQLCipher (`SQLCIPHER_KEY`)

Camada 2: Criptografia de Nível de Campo com Cloak.Ecto (`CLOAK_SECRET_KEY`)

6. Avoid `use` When an `import` Is Enough

How to describe an application?

The question: What was in that envelope?

The Premise

Architectural overview

C the fourth

Describing Behavior

The Case for BDD

The Assay format

Schema Definitions

Architectural Decision

Non Functional Requirements

The End-Boss: Design

The imageto effectively reconsider decisions once the Circumstances have changed.

The code

Visual Regression Testing

So, what is in that envelope?

The component nobody knows about

A tale of two workloads

The hot path: timelines served from memory

The follower graph: in memory, but not naively

So what do we actually need from a language?

The four candidates

Go

Rust

Node / TypeScript

Elixir

Every candidate has a flaw — which ones can you fix?

Resolving the Elixir paradox

The deciding factor: fan-out as code, not infrastructure

Why Elixir

What next?

Failure is an afterthought wearing a `rescue`

Why you should care

Serving timelines from Elixir

ETS tables

Celebrities

Evaluating our dataplane

Crimeline comparison

Roaring Bitmaps

Comparison in a more realistic scenario

Where to go from here?

What is legacy code?

Legacy software is the code we lost confidence in

"Can we rewrite this?"

How we got here

What we don't want

What we want instead

Organised and illustrated