Roastidio.us Tagged with elixir

From legacy code to verifiable specifications with Surveyor

Wed, 29 Apr 2026 20:10:39 +0000

What is legacy code?

Sometimes that's the fifteen-year-old monolith. Sometimes it's the service that grew complicated faster than the team could keep up with. Sometimes it's that project that started as a shortcut and turned into a roadblock. Legacy code has been exposed to real world demands long enough to gather fixes for edge cases we are not aware of anymore.

And — increasingly — sometimes it's the repo your CEO vibe-coded last weekend and that they proudly present on Monday. It runs. Now you have to own it.

Legacy software is the code we lost confidence in

That last category is genuinely new and growing fast. Code written once, never reviewed, tested a bit, deployed one day. Or, worse, released as a library and never maintained after that.

A new kind of technical dept has evolved: "The code we should review thoroughly one day."

"Can we rewrite this?"

The actual problem is not the writing of the new code, but the discovery of what the old code does, and the certainty that the new one does the same thing.

We've been building two small Elixir tools to help with both halves of that problem.

Surveyor scans a codebase in any language and produces an architectural model.
Assay runs the same plain-text behavioral specs against both the legacy and the rewrite, so you can prove they behave the same.

Together they cover the workflow: Figure out the structure, capture the behavior, rewrite, prove the rewrite works.

TL;DR

Surveyor produces an automatically verifiable behavioral specification of a legacy system.

How we got here

Before going into either tool, it's worth walking through the shape of the problem. The picture builds up step by step, and the final diagram only makes sense once you've seen the holes in the earlier ones.

What we don't want

A direct rewrite also copies bugs, dead code and unused features.

Point an agent at a legacy codebase, ask it to produce a new one, hope for the best. Call it "the Claude zombie rewrite." You end up with a target codebase that nobody understands either, plus a generation gap between the old assumptions and the new. What has the agent overlooked, misunderstood, assumed? A black box gets replaced with a different black box. That is not a rewrite, that is a transcoding.

What we want instead

The features of the legacy codebase need to be revised

The valuable artifact in the middle is not new code — it's a readable specification of what the old code actually does. Plain text, human-reviewable, version-controllable. The kind of thing you need when you want to ask the product owner if a feature is really wanted that way. - Or if it is just a legacy quirk.

If you have it, you can use it for estimates. It can be your map to plan the implementation and might even give you a hint about that stakeholder that you might have forgotten about otherwise.

Organised and illustrated

The specification's organisation should mirror the codebase

A single big text file describing a whole legacy system is only useful for very small projects. You want the spec broken down by module — by bounded context — and the modules themselves arranged on a map of the architecture. Once each module has its own spec associated to it, two things become possible: you can divide the work, and you can reason about coverage.

Job done! Or is it?

We now have and organised map of the system and a set of specifications. We have an overview, and the details. We could now talk to the stakeholders, the development team, and set to work.

The specification's organisation should mirror the codebase

However, we have not yet verified our findings.

What we are looking at is not yet a specification of the legacy system — it is a speculation of the legacy system. A document that claims to describe what the code does, written by reading the code (or by asking an LLM to read the code). Plausible, well-organised, reviewable. But unverified.

1. Verifiability

This is speculation as specification, and it's the trap most "let's document the legacy system" projects fall into. A Word document of "what the system does" can be worse than no document at all, because people start trusting it. But even if the specification is done as a group effort by all stakeholders involved, and is thoroughly verified, it still lacks one critical property:

2. Reproducibility

What we actually want is a verifiable specification: Not "verified once," but verifiable on demand, any time someone asks the question. You could verify a paper spec by hand, of course — sit down, read it, run the system, tick off the scenarios — but manual verification is expensive enough that it gets done once at sign-off and never again.

A verification you don't actually do is verification that doesn't exist. And unless the legacy codebase is a museum exhibit, it's a moving target:

The team is still shipping fixes, the system is still drifting, and a paper spec written on Monday is no longer accurate by Friday.

Runnable specs

The fix is to make the specs executable — to wire them into something that can run them against the actual legacy system, on demand, and tell you whether they still hold.

That is the move from speculation to verifiable specification: not a one-shot audit, but a button you can press on every commit, every nightly, every time anyone asks "is this still true?"

A green run proves the spec is still true. A red run is a useful signal: either the legacy drifted, or your spec was wrong. Either way, you find out before the rewrite, not during it.

That magic is Assay

Assay is the runner. It parses the .assay specs Surveyor produced, matches each step against bindings you write, and exercises the real legacy system through whatever interface it actually exposes — HTTP, CLI, message queue, file drop, whatever. The framework is small.

The runtime contract is "your bindings + your assertions + a deterministic runner."

The systemic approach

Open up that "parts you fill in" box and you find the actual surface area of work: pattern recognition for step matching, a binding per target system, and Given/When/Then automations for setup, action, and verification.

The end state has a property that nothing in the earlier pictures had: the problem space has been dissected into manageable pieces. Architecture, behavior, target adapters, assertions — each is small, each can be reviewed independently, and each can be picked up by a human or an agent without having to hold the whole system in their head at once.

That's the goal of the toolkit. Everything below is the detail of how each piece is built.

Surveyor — discovering the architecture you inherited

Surveyor is a Mix project that takes a path to a codebase and produces a Structurizr DSL workspace file. It runs in three phases mapping to the C4 model: C1 (system context), C2 (containers), C3 (components per container).

Crucially, Surveyor does not parse code. There are no language-specific parsers, no AST walkers, no fragile grammar files for thirty different ecosystems.

It scans the filesystem to identify what's there — languages, frameworks, project files, entry points, deployment manifests — and then feeds meaningful chunks to an LLM. That sounds trivial, but is the same mechanism that commercial coding agents use when they analyse your code.

Surveyor's intelligence is in the prompts and the chunking, not in a stack of half-broken language adapters.

The CLI is interactive at every phase.

$ surveyor ./legacy-monolith --phase c1

Phase 1 — System Context
  Scanning codebase...
  Querying LLM...

  System: "Order Management System"
  Description: Manages the full order lifecycle

  Actors:
    ✓ Customer — places and tracks orders via Web UI
    ✓ Warehouse Staff — manages fulfillment via Back Office
    ? Admin — found references in auth config (confidence: low)

  External Systems:
    ✓ Stripe — payment processing (REST API)
    ? SendGrid — found API key in config (confidence: low)

[a]ccept  [e]dit  [r]etry with more context  [q]uit
>

The LLM is asked to flag uncertainty. Anything tagged confidence: low shows up with a ? and a short reasoning string for human review. That's not just nice ergonomics — it's the bit that makes the tool trustworthy. An LLM that confidently invents a PaymentReconciliation container that doesn't exist is worse than no model at all. An LLM that says "I see a SENDGRID_API_KEY in .env.example but no Sendgrid client in the source, please confirm" is doing the right kind of work.

Each phase is resumable. Results are saved as JSON in ./surveyor/, so a thirty-container system doesn't have to finish in one sitting. You can hop in, accept C1, take a break, come back tomorrow, and resume at C2.

The end product is a workspace.dsl you can render in Structurizr. But more importantly, it's a workspace.dsl whose components are decorated with assay.specs and assay.schema properties, pointing at the behavioral specs and the domain schemas for each bounded context.

orderLifecycle = component "Order Lifecycle" "Orders" "Bounded Context" {
    properties {
        "assay.specs" "specs/order-lifecycle"
        "assay.schema" "schemas/order_lifecycle.ex"
    }
}

DDD pattern annotations from the LLM (customer-supplier, anticorruption-layer, …) ride along as properties on the relationship. Every component gets assay.specs and assay.schema properties — the contract the next tool reads from.

Examples

The System Context of a Medical Application

Surveyor identified the actors and the external systems of the application. Without knowing the code, we would already know which actor types there are

The System Context of a medical application

An example of an assay spec created in the second phase:

The Account Creation Specs of a medical application

Assay — proving the rewrite behaves the same

Assay is a minimal behavioral spec runner. Specs are plain text and look like this:

Component: [orderLifecycle] Order Lifecycle
Context: Order Placement

Definitions:
  - "a valid customer" means:
      a Customer with status Active, verified email,
      and a credit limit greater than zero

Invariants:
  - total must not exceed customer credit limit
  - at least one line item required

Rule: Customers can place orders for in-stock items

  @critical @phase-1
  Scenario: [OL-001] Place a simple order
    Given a valid customer "Alice" with credit limit €10,000
    And product "Widget" is in stock with 50 units available
    When Alice places an order for 3 units of "Widget"
    Then the order status becomes "Placed"
    And stock for "Widget" is reduced to 47 units

The bracketed [orderLifecycle] is a workspace.dsl identifier — the same identifier Surveyor wrote into the architecture model. The bracketed [OL-001] on the scenario is an optional free-form id that survives across rewrites of the spec text and lets you cross-reference from tickets or audit trails.

If that looks like Cucumber or Gherkin, it does — with one large difference. There is no glue layer, no abstraction over what kind of system you are testing, no framework opinions about HTTP, databases, or browsers. The bindings are just Elixir.

defmoduleTargets.Legacy.OrderLifecycledo
useAssay.Binding,component:"orderLifecycle"

@baseSystem.get_env("LEGACY_API_URL")

  step :given,~r/a valid customer "(?P<name>.+)" with credit limit €(?P<limit>[\d,.]+)/do
{:ok, resp}=Req.post("#{@base}/test/customers",json:%{
name:params().name,
credit_limit:parse_money(params().limit),
status:"Active"
})
assign(customer_id: resp.body["id"])
end

  step :action,~r/.+ places an order for (?P<qty>\d+) units? of "(?P<product>.+)"/do
{:ok, resp}=Req.post("#{@base}/orders",json:%{
customer_id:var(:customer_id),
lines:[%{product_id:var(:product_id),quantity:to_int(params().qty)}]
})
assign(order_id: resp.body["id"],http_status: resp.status)
end

  step :expect,~r/the order status becomes "(?P<status>.+)"/do
{:ok, resp}=Req.get("#{@base}/orders/#{var(:order_id)}")
    assert resp.body["status"]==params().status
end
end

A binding is just a module that brings whatever it needs — Req for an HTTP API, AMQP for a message broker, System.cmd for a batch processor, File for a directory-watching pipeline. Assay itself does not ship an HTTP client or a database adapter.

The framework provides parsing, step matching, scenario lifecycle, and assertions; the rest is your code. That's what makes the same spec runnable against an HTTP API, a CLI tool, a batch job, or a message queue, depending on what the legacy system happens to be.

🚀 Would you like to run this on your legacy codebase? Sign up for early access! Count me in!

The crucial property is target swappability:

assay run specs/order-lifecycle/ --target legacy
assay run specs/order-lifecycle/ --target new

Same specs. Different bindings. Different systems. The legacy binding hits the old SOAP API; the new binding hits the new Phoenix endpoint. If both go green, the rewrite is behavior-equivalent for the things the spec covers — which, after a few months of writing specs, is most of the things that matter.

There is no separate .exs file generated from the specs. The runner parses .assay files, matches each step against a regex on a binding function, executes them, and prints pass/fail. That's it.

How Assay actually works

The runtime is around six hundred lines of Elixir. Five design choices do most of the work.

A two-pass parser

The first pass lifts triple-quoted doc strings out of the file (so the line tokenizer doesn't have to reason about them). The second pass dispatches on the first keyword on each line.

defparse_string(content)do
{filtered, doc_strings}=
    content
|>String.split("\n")
|>Enum.with_index(1)
|>extract_doc_strings()

Process.put(:assay_doc_strings, doc_strings)

  filtered
|>Enum.reject(fn{line, _}->
    trimmed =String.trim(line)
    trimmed ==""orString.starts_with?(trimmed,"#")
end)
|>parse_lines(%Spec{})
end

Output is a plain %Spec{} struct. No AST nodes, no string interpolation.

The step macro generates real module functions

Each step :given, ~r/.../ do ... end call expands into a uniquely-named function plus a %StepBinding{} record on an accumulating attribute.

defmacrostep(type, regex,do: block)
when type in[:given,:action,:expect]do
  fn_name = :"__step_#{:erlang.unique_integer([:positive])}__"

quotedo
@assay_steps{unquote(type),unquote(regex),&__MODULE__.unquote(fn_name)/0}

defunquote(fn_name)()do
unquote(block)
end
end
end

At compile time, __before_compile__ exposes __steps__/0 and __component__/0 for the runner to read. There is no string codegen, no eval — just normal Elixir function definitions.

Component-scoped step matching

Anyone who has used Cucumber has hit the global step-definition problem: the same step text means different things in different contexts, but Gherkin treats step definitions as one shared namespace. Assay scopes bindings by component:

defprun_spec(spec, bindings, target, tags, exclude_tags)do
  component_bindings =
Enum.filter(bindings,fn b -> b.component == spec.component end)

# ... run each scenario against component_bindings only
end

This is the direct reason Surveyor and Assay use the same identifier convention. The architecture model and the spec runner share a vocabulary, and step text never collides across bounded contexts.

Pre-check, then execute

Before running any step, the runner walks the whole scenario and finds a binding for every step. If anything is unbound, the scenario fails before any side effects.

defrun_scenario(scenario, bindings)do
  matched_steps =
Enum.map(scenario.steps,fn step ->
casefind_binding(step, bindings)do
nil->{:unbound, step}
        binding ->{:bound, step, binding}
end
end)

ifEnum.any?(matched_steps,&match?({:unbound, _},&1))do
%ScenarioResult{status::error,error:"Unbound steps found"}
else
execute_matched_steps(scenario, matched_steps, bindings)
end
end

This is what stops a scenario from running half its Givens, hitting an unbound When, and leaving a dangling test customer in the database.

Per-scenario state in the process dictionary

Each scenario gets a fresh context — a tiny module that stores variables, params, doc strings, and data tables in the process dictionary.

def init do
Process.put(@vars_key,%{})
Process.put(@params_key,%{})
end

defget_var(name)do
Process.get(@vars_key,%{})|>Map.get(name)
end

defset_vars(keyword)do
  vars =Process.get(@vars_key,%{})
Process.put(@vars_key,Enum.into(keyword, vars))
end

The runner can save/0 and restore/1 the snapshot, which is how Assay tests itself by running its own .assay specs through itself in the same process.

There is no plugin system, no dependency injection, no scenario hooks beyond cleanup. Adding any of those would push complexity into the framework and out of the binding, which is the wrong direction. The framework exists to be small enough that you can read it on a Friday afternoon and trust it on Monday.

How they fit together

The flow is straightforward and it maps onto the five phases laid out in the Assay handbook.

Discovery (Surveyor). Run Surveyor against the legacy codebase. Walk through the C1/C2/C3 output interactively. Edit, retry, accept. Produces workspace.dsl.
Behavioral extraction. For each component in the model, write .assay specs and an Elixir schema. The assay.specs and assay.schema properties on every Structurizr component tell you exactly where they go: specs/<context>/ and schemas/<context>.ex.
Validation. Write bindings against the legacy system in targets/legacy/. Run assay run specs/ --target legacy. Iterate until green. You now have a verified, executable specification of the legacy system's behavior.
Stabilization. Review what's covered, what's missing, what's flagged as ambiguous. Get sign-off on scope.
Rewrite. Build the new system. Write bindings against it in targets/new/. Run assay run specs/ --target new. When that's green, the rewrite is done — at least for the surface area the specs cover.

The reason the two tools sit next to each other is that the architecture model is what gives the spec work structure. Without a model, "write specs for the legacy system" is an open-ended task with no obvious stopping point. With the model, every component has a specs/ directory and the question becomes "is each context's behavior covered?" That's tractable.

🚀 Would you like to run this on your legacy codebase? Sign up for early access! Count me in!

The cross-target story is what makes the rewrite verifiable. The same parsed .assay scenarios are dispatched, step by step, through two different binding modules at two different points in the project's life:

Same spec, two bindings, two systems. The runner doesn't know or care which target it's dispatching to — that's the property that makes the rewrite a measurable thing rather than a leap of faith.

Four audiences, one artifact

Four very different parties end up reading — or executing — the same .assay file: the product owner, the developer, the coding agent, and the test runner. The interesting observation is that they don't conflict on content — they conflict on form.

Product owners want to know what the system does in business terms, what would break if you changed X, and where the risk lives. They don't want UML. They want to read scenarios in their domain vocabulary, in plain language, and trust them.

Developers want to know how the system is structured, where the seams are, and what invariants they mustn't break. And — crucially — they want the documentation to be wrong less often than the code is. The moment docs lie, developers stop reading them.

Coding agents want machine-parseable, unambiguous, addressable artifacts with stable identifiers. They want to be able to say "the assertion at assay://orderLifecycle/order-placement/OL-001/step-3 failed" and have that mean something durable. They want types, schemas, and graphs they can traverse.

Automated tests want executable bindings — the layer Assay provides — and they want failures that point back to the spec, not just to a line of code.

Same facts, four presentations.

The .assay file gives the product owner readable scenarios in domain language. The workspace.dsl plus the schema modules give the developer the structural truth. The bracketed identifiers (Component: [orderLifecycle], Scenario: [OL-001]) give the agent its stable addresses. The runner turns the whole thing into a regression suite. None of these audiences is asked to read the others' representation; they all read the same spec, surfaced at the level of detail they need.

Where the LLM lives, and where it doesn't

A reasonable question at this point: with all the recent enthusiasm for LLMs, why isn't more of this LLM-driven?

The split is deliberate.

Surveyor uses an LLM because architectural discovery is genuinely a language task — you're reading config files, route definitions, deployment manifests, dependency lock files, and inferring "this is the API, that's the worker, that's the read model." That's exactly the kind of fuzzy, evidence-weighing job an LLM does well, especially when you tell it to flag uncertainty rather than guess.

Assay does not use an LLM at runtime. The runner is deterministic: parse the spec, match the regex, execute the binding, assert. A behavioral spec that sometimes passes and sometimes does not, depending on which model variant happened to answer, is not a spec.

LLMs are very welcome on the authoring side — drafting .assay files from legacy source is a great agent task, and the handbook has explicit guidance for coding agents about how to do that without inventing behavior. But the green/red signal at the end has to come from a deterministic runner, or it is not really a signal.

🚀 Sign up for early access:Count me in!

How you use them on a project

In practice, the first day of a legacy rewrite looks something like this:

Clone the repo. Run Surveyor with --phase c1 to get the system in context. Discuss the actors and external systems with the client — this alone surfaces "wait, who is the Admin role?" conversations that would otherwise happen three months in.
Run --phase c2 to map containers. This is where you discover the data plane that nobody documented, the cron job running on a forgotten server, the queue that two services share. Surveyor flags confidence: low items; the human resolves them.
Run --phase c3 per container. By the end you have a workspace.dsl that is, for the first time in the project's history, an accurate picture of what is deployed.
Pick the highest-risk bounded context. Read the legacy source. Draft .assay specs for its behavior. Write a legacy binding. Run it. Iterate until green.
Repeat per context, in priority order, until coverage is good enough to start the rewrite.
Build the new system, one bounded context at a time, with targets/new/ bindings going green as each context comes online.

The thing both tools are optimized for is the same: making the work legible. A legacy rewrite is a long project with shifting personnel and a nervous client. Both Surveyor and Assay produce artifacts — a workspace.dsl and a directory of .assay files — that survive turnover, show progress, and that the client can actually read.

Status and next steps

Both tools are ready to be tested in real world scenarios. This is where we need your help:

If you have a legacy system staring you down and a rewrite on the roadmap, get in touch. We would love to hear from you!

Elixir: Introduction

Sun, 26 Apr 2026 14:28:15 +0000

Elixir: Introduction

Head of the Agents and Assistants Department

Thu, 23 Apr 2026 10:20:28 +0000

Let me state upfront: my attitude toward AI assistants cannot be expressed as a boolean value. If you need an answer to the question posed point-blank—“New York Yankees or Boston Red Sox?”—I do not watch baseball at all; I’m a Barça fan. That said, I find AI assistants a perfectly legitimate and even liquid asset. The text below is an account of what made my work with agents pleasant and reduced their errors and rough edges to an acceptable minimum.

A little under a year ago I began working on Cure, a programming language with dependent types, finite state machines as first-class citizens, SMT verification, and other niceties, compiling to the BEAM.

My first approach to the apparatus ended in ignominious failure. I got tangled in my own architectural decisions, started piling on crutches wherever they fit, turned the code completely into an Italian restaurant menu, and lost heart. In a fit of idiocy I had an assistant generate the website and showed it to the public—the ideas themselves were intriguing enough, dependent types and SMT solvers on the BEAM are hardly redundant, and I harboured a quiet hope for community interest. The community correctly identified the language’s site as slop generated by a language model and received my attempt with arctic indifference. I got a great many “nothing works” responses and not a single coherent suggestion for improvement (no fault of the community’s—against the backdrop of the slop-flood of those days, my project would not have looked like Noah’s Ark even to an extremely charitable observer).

I stepped back, examined my creation from all angles, and was forced to admit: I had produced a monster. I saw no chance of licking it into shape, even through a global refactor. I bought a ream of paper and some pencils and started drawing, in order to understand exactly where I had gone wrong. (Spoiler: I had been so enchanted by the idea itself, and so desperate to get something to launch and run, that I had done literally everything wrong.)

I had no intention of giving up—and before me, full height, loomed the necessity of rewriting everything from scratch without repeating the mistakes. By that point I knew that artificial assistants could significantly accelerate the actual writing of code, so I started by erecting scaffolding around the future project. It was obvious to me then (I can now confirm that former intuition with experience) that all those prompts along the lines of “You are a genius architect with three hundred years of experience designing languages with dependent types and SMT solvers” work no better than the morning pep talk to an intern at standup: “You are a great programmer who has written three hundred million billion lines of code without a debugger.” If an assistant tasked with writing coherent code to a spec needed motivational gibberish to function, it would not be worth using under any circumstances whatsoever. Seriously, think about it: a model is asked to implement a GCD module using the Euclidean algorithm—are you really suggesting its deeply baked-in internal rules will not guide it down the correct branches of the conditional operators without first being told it has the soul of a prima ballerina and an avant-garde poet? What on earth does “You are a senior architect” actually mean? Do the people who advocate this believe that without such a preamble the training pathways via the memoirs of axolotl breeders will activate instead?

So, instead of all those skills/agents/whatever, I started by feeding this mechanised beast the source code of all my own libraries, lovingly written by hand, with the note: “Here are examples of good code. Write like this. Not like that—do not write like that.” I know it is immodest, but it is my assistant. You are welcome to feed yours your own code.

Then I reclaimed the wasteland: thus were born Metastatic, implementing MetaAST for different languages across different paradigms, and Ragex—a RAG built on AST rather than plain text (my hunch that AST fits into a context window far more easily and is far better structured than raw source code turned out to be correct).

My task was building a new language; constructing a new ecosystem from scratch was not part of the brief. So I analysed existing solutions—Rust, Go, Elm, Gleam—and chose the one I considered most mature (I never promised the project would be neutral with respect to my tastes and preferences). I simply copied the Elixir ecosystem and added to it what I had personally found lacking over the last ten years. Thanks to standing on the shoulders of giants in this regard, the model wrote almost the entire ecosystem for me; I simply told it: “Look how beautifully new project creation is handled in Elixir—do the same for Cure.” Language models are strong at translation, and Elixir is considerably more intelligible to them compared to almost every other language.

So, before the first line of code, my backpack already held: the right AST—a language in which the assistant and I can communicate far more easily than in homespun English—a handcrafted RAG, and a clear understanding that every step must be a simple, atomic change. The fewer choices the assistant has to make between two paths, the cleaner the result. This principle outweighs the quality of all prompts combined.

Next I had to solve the problem of validating the produced code. My eyes are sharp, but they occasionally miss non-obvious bad decisions in review. Thus was born oeditus_credo—a set of nearly forty additional credo checks covering vulnerabilities, anti-patterns, and the like. The library also ships a mix oeditus_assistant_rules command, a rules generator for the soulless assistant. To those rules I also added: after each stage, verify that mix format && mix credo --strict && mix dialyzer && mix test passes; update all documentation; add regression tests for new code; then run all regression tests and confirm they are still as green as my face the morning after a party.

Every reasonably significant stage also ends with creating an “example” in the examples folder. Something for people to look at, and the regression tests never idle.

At this point I felt ready to start writing actual code. Cure has certain critical parts I wrote by hand from scratch. Every other pull request gets manual edits from me. Every bug I find through manual testing I fix by hand (the obvious ones aside). And yet I reached the desired result considerably faster than if I had written every line in Vim.

Over the course of working on Cure I learned to keep the number of errors—and consequently manual edits—to a minimum. A fairly substantial release, v0.26.0, required not a single correction, for example. Here is the distillate of my rules for communicating with an assistant, in case it proves useful to anyone:

the task must be self-contained but not too large; “first define the types for numeric, then we will write the converters” does not work
inside the task there must be no undefined ambiguities that will derail our T9; there must be exactly one path to the solution
before tackling any task, demand an implementation plan and edit it until all ambiguities have vanished
the task must be roughly solved in your own head before turning to the assistant; otherwise the odds of agreeing with an incorrect solution are uncomfortably high
generated code must be comprehensible and elegant, “rewrite this nicely, I fed you three gigabytes of rules” does not work; if the solution is aesthetically repellent, there is a problem with the task formulation—close the session and start over
setting a task and going for coffee is the direct path to the infinite iterations described in the previous point; the stream of unconsciousness must be watched and any attempt to stray from the planned path killed without mercy
finally, if it seems to you that the cognitive load is decreasing and that any cook could now implement such a project—you need pharmaceutical intervention; your fingers tire less, yes, but if you could not have implemented the project from scratch in a text editor, the LLM is no help to you; it will generate something suspiciously twitching, no argument there, but the first halfway-serious complexity requiring a considered architectural decision will put a cross on the whole enterprise.

That is my experience. Yours may differ; I can bear with you on that matter.

Try https://cure-lang.org and maybe you will like it. The site now has a playground where you can experiment with types in real time, and an almost real console where you can play with the REPL without installing anything locally.

Happy curing!

Scotty, I need warp speed in three minutes - Hauleth

Mon, 20 Apr 2026 23:02:18 +0000

In my last larger gig I worked on fascinating project - Postgres connection pooler written in Elixir. Unfortunately, due to different circumstances, this project burned me out to the ground. However what doesn't kill you ~~is crap not a weapon~~ can become great learning experience.

Most of my achievements in this project were related to performance. This project contains very tight loop in form of query handler, that needed to run hundreds of thousands times per second per user connection. That mean that this functions are very sensitive to even slightest performance changes. And that was my task - to find potential improvements that can be made to make this codebase be much faster.

After departing from Supabase I liked the project so much (mostly as learning ground) that I have created my own fork, where unrestrained from all business side of the project I could focus purely on squeezing as much of performance as I can. This project now lives as Ultravisor - it is still nowhere near being done in a way that I like, but I still go back to work on it from time to time to find potential performance improvements.

This is a story of things that I have done and learned during that journey.

Beware: It is a retrospection, so in some places my memory may be not the best.

Here I need to provide some explanation first, about how Ultravisor works with database connections. It provides 2 modes of operation:

session - where each connection from user to Ultravisor checks out one connection from Ultravisor to database. It checks out once, at the start of connection, and then holds connection until the end;
transaction - where on connection there is nothing done. Client connects to the Ultravisor and can keep that connection indefinitely without ever bothering database. Database connection is checked out only when there is some request from user and is returned to the pool as soon as that result of that query is returned and DB is ready for next one.

While session mode is quite on par with other implementations of connection pooling for Postgres, transaction mode is where performance is lacking and is the main focus is put. In whole article (unless mentioned otherwise) I will speak about transaction mode of Ultravisor.

Lesson: Flame graphs and call tracing is essential#

Pretty obvious thing, but still valuable lesson for any performance optimisation endeavour. For that the great thanks to Trevor Brown and his awesome project eFlambè. This helped a lot in tracing hot points in the running code.

Unfortunately this project seems to be less active recently and has some missing features, like listening for given duration instead of function calls count. This can be partially fixed by simply listening for count of calls to handle_event/4 function given times and then running cat *.bggg to concatenate all files into larger trace. That has disadvantages, but at least it was workable within Speedoscope which I also highly recommend to anyone who needs to work on such optimisation.

While flame graphs are awesome, there is cost to gathering them with eFlambè - it greatly affects performance. Fortunately Erlang has some built in tools that have lesser performance impact, and the "most modern" of these is tprof. This tool is pretty easy to use, but is less detailed than eFlambè. But even with that limitation, it provides superb insight into stuff that has greatest impact on performance, as well as it make it easier to work on long running processes, as it work asynchronously, so you can "manually" decide how long you want to trace your process.

Summary: Knowing where your bottlenecks are is essential for performance optimisations.

Lesson: Doing less can improve performance#

Obvious thing that need to be stated - doing nothing is faster than doing something. Extracting amount of data sent over given socket using :inet.getstat/2 call is fast, but not free. That involves some waiting for response from either port or process handling connection, which introduces slowdown. Two possible solutions there are:

Do not gather that metric at all - sensible, but not feasible, especially when you use that metric to charge your users.
Gather that data less often.

The approach I have taken there is obviously 2., and the solution is dumb simple - debouncer.

Debouncing is an interesting technique often used in user interfaces where you accept some event, and then for some period you ignore repeated events. The reason for that is that our interfaces may have flaws that send repeated events one after another.

In this case Ultravisor tries to store amount of sent data after each query, but that can get expensive for many short queries. Instead I have implemented simple per-process debouncer:

This stores returned data in process dictionary (per-process mutable space with quick access) and if there was no call in given time-period, then we process data again. This is safe way to do so, as :inet.getstat/2 will always return amount of data that socket processed since it started, so data between calls will be accounted.

Before:

tps = 79401.392762 (without initial connection time)

After (10ms of debouncing):

tps = 80069.646510 (without initial connection time)

After (100ms of debouncing):

tps = 80568.825937 (without initial connection time)

Summary: Doing noting is more performant than doing something. Sometimes doing nothing can be quite easy.

Lesson: Telemetry is not free#

When working on most projects, especially Phoenix-based, one can slap :telemetry.execute/3 calls everywhere and notice no performance degradation¹. Unfortunately, when you do hundreds of thousands calls a second - that is not a case.

For unaware readers - Telemetry is Erlang event dispatching system for observability events.

In this project the metrics are exposed in Prometheus/OpenMetrics format, which means that there needs to be collection system within the application. In BEAM applications the standard way to implement that is to use ETS tables to store recorded values. Fortunately there are libraries to handle that for you, and for the longest time "gold standard" for it was telemetry_prometheus_core library created by Telemetry core team.

While for most projects that library is performant enough (because metrics aren't recorded in quite tight loops), in case of this project that was not a case. There, metrics gathering is still one of the hottest spot in the codebase, even with all improvements that have been done.

Excerpt from tprof profile:

Function	Calls count	Per call (μs)	Percentage
`Peep.EventHandler.store_metrics/5`	1421911	0.13	4.21%
`Peep.Storage.Striped.insert_metric/5`	904852	0.26	5.11%

This is with awesome library Peep by Richard Kallos. When using telemetry_prometheus_core it was simply the most expensive thing in whole loop. Just replacing metrics gathering library with Peep gave us about 2x bump in TPS.

Summary: Telemetry handler can matter in tight loops. Fast metrics gathering isn't easy.

Lesson: Records instead of maps or structs#

Elixir uses structs for structured data. That gives a lot nice features wrt. hot code reloads, compilation graph dependencies, and other. However, because structs are maps, there is a cost. Maps have O(log n) access time to the fields, this is how maps are constructed in memory. While smaller maps have slightly different (better in most cases) characteristics, there is strict requirement that you keep your structure with less than 31 fields² and it still has slight memory overhead. The alternative is to use records. These have better performance characteristic (always constant) irrelevant of the amount of fields at the cost of being slightly more rigid (records are tuple based) and less convenient to use (experience may vary). Additional advantage in my opinion is that it is harder to add incorrect field by using Map module.

Current (OTP 28) limit for small map is 32 keys, but Elixir uses one key for struct name, hence 31 fields is the limit.

Before you will run and change all structs in your system to records, just remember - most of the time the difference doesn't matter - just use structures.

Before:

tps = 81765.266264 (without initial connection time)

After:

tps = 82147.855889 (without initial connection time)

Summary: Records are super handy when you need to squeeze each bit of performance. It doesn't provide much, but these adds up.

Lesson: ETS tables are super fast, but not always#

ETS is Erlang's built-in module for storing key-value data in mutable way. Like built-in Redis. This structure allows for sharing some data in a way, that is easy to access from different parts of the system. One example of system that is using ETS for storing their information is Telemetry (mentioned above).

While for 99% of the use cases Telemetry will be fast enough, it has some problems with tight loops. Main problem is that it will always copy data from table to the caller process. That mean that it can put high memory pressure on the process that tries to retrieve data.

Fortunately Erlang supports another mechanism for storing globally accessible data - persistent_term. Of course, there is no such thing as "free lunch" so it has substantial disadvantage - it works poorly³ with data that changes often, as removing or changing data in a key will require walk through all processes to copy data from it to processes that may use it into process memory. However - Telemetry handlers should not change a lot, you should just set them once as soon as your system start, and then ideally they will not change ever again.

There is slight optimisation that makes it fast in some cases (single word values, like atoms), but that is not the case there, so we can ignore that.

Before⁴:

tps = 76914.004685 (without initial connection time)

After:

tps = 78479.006634 (without initial connection time)

⁴

If you wonder why these results are lower than in previous section, it is because test conditions are identical only per section, not cross sections. In this particular case I have ran benchmark while collecting metrics (to show difference in persistent_term change) while other are ran without metrics to not pollute results.

Summary: persistent_term is awesome and super fast, so if you know that you have some data that will probably never change and will be requested constantly, then it may be good place to store that data.

Lesson: Calling your `GenServer`s is fast, but not 90k times per second fast#

One of the interesting observations that I have spotted is that if there are longer running queries, ones that send more data over the network than just simple short responses, then the difference between Ultravisor and "state of the art" tools like PgBouncer or PgDog (that are written in non-managed languages like C and Rust) is much smaller (obviously it is still there, but it is on par, not substantially off).

I needed to dig more, what can be the cause of such strange behaviour. The reason was found in place where I least expected it - checking out database connection to be used.

Flame graph showed that almost third of the time is spent on checking out database connections, and most of that time is spent in 2 function calls, both of them are internally gen_statem calls and in both most time is spent on sleeping (aka, waiting for reply).

Now, this one is hard thing to optimise, as in Elixir there is no mutability (almost, we will get there). This mean that if I want some form of shared queue of processes, then I need to use separate process to keep state of the queue for us, and then do GenServer calls to fetch that state. What I did in such situation? What any unreasonable Elixir developer obsessed with performance would do - NIF⁵.

⁵

I wanted to use ETS there, but for that to work it lacks function like ets:take/2 that would return only one element from the tables with type bag or duplicate_bag. Or any other form of just taking out any (possibly random) element from ETS table in atomic way.

The implementation is rather basic wrapper over VecDeques that allow popping single element from that queue without any message passing. The implementation is very crude, nowhere production ready. It doesn't provide any form of worker restarts or anything, but works quite well as PoC of what is possible.

New queue also provides a way to store additional "metadata" alongside the worker PID. This allows me to store DB connection socket next to connection process, which removes need for additional call to extract that data to pass requests directly to other DB, without copying data between processes.

Before:

tps = 83619.640673 (without initial connection time)

After:

tps = 94191.475386 (without initial connection time)

Summary: Sometimes one need to get creative to get around platform limitations. This may require some pesky NIFs though.

Conclusions#

Optimising such project was enormous fun and I think that at the current state there is nothing extra that can be done to optimise it more without optimising generated JIT-ed native code or optimising Erlang scheduler.

There are some flags, that affect performance, but as it is currently unclear why these work at all (probably it is related more to the OS scheduler rather than Erlang performance), I left them out of this article for now.

Post Scriptum: Good tooling helps a lot#

Just after I have started that optimisation project after leaving Supabase I started using Jujutsu for version control. That one thing helped me a lot with being able to have separate branches/PRs for each of the changes, while at the same being able to work with mega-merge of them all.

That allows me to profile code with all other noise removed, while still exposing the changes as separate reviewable units. Without that support I would need to decipher what have already been changed and/or removed from the profile.

Additional feature that I heavily used there is "anonymous branching". As when working with JJ I do not need to create new name for each branch that I want to try, it was way easier to implement one idea, then just do jj new @- (which branches off at the commit that is parent of the current one) and just implement alternative idea. I used that constantly to compare ideas and reject failed concepts.

∎

Resist Vendor Lock-In With Supabase

Sun, 19 Apr 2026 04:26:32 +0000

In one of his recent YouTube videos, Theo Browne highlights the potential pitfalls that comes with using a Platform-as-a-Service.

The video goes over recent work done to fuzz suspected Firebase-backed services looking for leaked credentials. As is shown in the video, this is an all-too common pattern in these platforms, where the "backend" is being directly manipulated from client-side SDKs, such that theres a high likelihood that credentials are carelessly exposed to the client. In addition to potential secret leakage, many of these platforms, including Firestore, do not have great default security settings. This has obvious MAJOR security concerns is a horrible practice for the industry to standardize on, but yet these platforms still remain extremely popular despite known security concerns due to their ease of use and how they enable rapid development.

In addition to these security concerns, Firebase has long drawn the ire of many developers due to its tight coupling with Google's proprietary ecosystem, and its cost structure which has lead to more than a few horror stories about unexpectedly high bills (although these issues can often be attributed to poor database practices that lead to exponentially ballooning costs). With much apprehension towards Firebase, many looked for alternatives, and found themselves needing to piece together a solution from many different parts. Here enters Supabase, which pitches itself as the "open-source Firebase alternative." Supabase offers similar solutions as Firebase, such as a realtime database, object storage, user authentication and management, edge functions, and much more. Supabase, however, is built entirely from open-source software (in fact, they stipulate that Supabase will only ever include software that comes with an MIT, Apache 2, or equivalent license).

When asked about his thoughts on Supabase, Theo rightly points out that these problems still exist when using their client SDKs, but he stresses that unlike Firebase, since Supabase is built entirely on open-source software (namely PostgreSQL) you not only can connect directly to the database, but it is actively encouraged by the CEO himself.

cc @chasers @filipecabaco @wenboxie

Alternatively: just connect to Postgres
¯\_(ツ)_/¯
— Paul Copplestone — e/postgres (@kiwicopple) April 19, 2024

Wow! What a breath of fresh air that is! And as it turns out, even when "just" using Supabase in this way, it is still great! You get an authentication table with Row-Level-Security and robust security practices out of the box, access to their wonderful front-end database management, and the ability to easily integrate their other features such as Edge Functions, Object Storage, all while maintaining tight control over your backend. It just makes sense! It offers the rapid development experience that is all too valuable, while not compromising on security.

Still, you might be wondering what this looks like from the developer perspective. Surely if you're writing directly to the database this must be more unwieldily than using those client SDKs right? Well, not necessarily!

Supabase still offer PostgREST out of the box, which exposes a REST API for database operations, but you have control over where you fire off those requests from. And since you have complete control over your database, you can even do a hybrid approach, which is what I prefer. Since Supabase takes care of the auth.users table (and even makes it read-only through the web frontend), you might want to just use the REST APIs they expose to handle authentication, but then handle everything else yourself.

💡

If you want to have a table NOT be exposed as an API through PostgREST, just add it to a different schema from the public schema. All tables under the public schema will be exposed.

Even though you might want Supabase to handle authentication, you still might want to extend features from that users.auth table. Since they discourage modifying that table directly, the suggested approach is to make a table in the public schema (which is the default schema – the auth schema is managed by Supabase) to manage any additional information related to the user. The documentation goes into much more detail about Managing User Data, but that is the gist of the situation. Well, if you want this new table (let's call it Profiles) to stay in sync with the auth.users table, it's best to have the Profiles table refer to the auth.users table, and add a Postgres trigger to run a function to create a new entry in the Profiles table whenever a new user is registered.

💡

It's also worth noting that in their recent General Availability Launch Week, Supabase also announced new efforts to increase security practices across projects, including a Postgres Linter and new Security Advisor and Performance Advisor dashboards to help you maintain good security posture.

GitHub - supabase/splinter: Supabase Postgres Linter

Supabase Postgres Linter. Contribute to supabase/splinter development by creating an account on GitHub.

GitHubsupabase

Supabase Security Advisor & Performance Advisor

We’re making it easier to build a secure and high-performing application.

Supabase

Let's walk through an example of what this might look like using my favorite backend language, Elixir (it's also a favorite of Supabase themselves). I'll be using Elixir's very nice ORM, Ecto.

One you make your new project (using mix new or perhaps more commonly in this situation, mix phx.new) you'll want to make sure you have Ecto as a dependency, and then you'll want to model the auth.users table as well as the new public.profiles table.

defmodule User do
  @moduledoc """
  This schema represents the default Supabase users table, which is under the 'auth' schema.

  Since we don't actually manage this schema, we will not make any migrations for it.

  This is mainly for convenience when unmarshalling data and working with users, so we
  can refer to the User struct rather than a generic map.

  Notice that we specify the primary key which will be referred to later

  Ecto schemas do not have to have 1-to-1 fields match the table in the database, so we can use whatever minimal fields we want to mirror in the profiles table (`id` at a minimum).
  """
  use Ecto.Schema
  import Ecto.Changeset

  # id is a UUID
  @primary_key {:id, :binary_id, autogenerate: false}
  schema "auth.users" do
    field :created_at, :naive_datetime_usec
    field :updated_at, :naive_datetime_usec
  end

end

defmodule Profile do
  @moduledoc """
  This schema holds extra information about users
  """
  use Ecto.Schema
  import Ecto.Changeset

  schema "profiles" do
    field :first_name, :string
    field :last_name, :string

    embeds_one :settings, Settings do
      field :default_portfolio, :string
      field :theme, Ecto.Enum, values: [dark: "Dark", light: "Light", system: "System"]
    end

    # This is the most important line and the one that is required to
    # properly link this table to `auth.users`
    # Make sure to set the type to :binary_id, which is what the Supabase 
    # auth uses
    belongs_to :user, User, type: :binary_id, references: :id, primary_key: true

    timestamps()
  end
end

Now we would create the necessary migrations, starting with creating the public.profiles table.

defmodule CreateProfiles do
  use Ecto.Migration

  def change do
    create table(:profiles, primary_key: false) do
      # notice the `prefix` since `public` is the default prefix
      # notice the specifying the type to match the Supabase defaults
      # make sure to set this as the primary key
      add :id, references(:users, on_delete: :delete_all, prefix: "auth", type: :uuid),
        primary_key: true

      # These fields should match what you have in your schema
      add :first_name, :string
      add :last_name, :string
      add :settings, :map

      # This represents the `inserted_at` and `updated_at` fields in the 
      # schema, and are required by default
      timestamps()
    end

    # You might also want to add indexes to improve performance and ensure data integrity
    create index(:profiles, [:id])
  end
end

Now we add a migration to add the trigger:

defmodule CreateProfilesTrigger do
  use Ecto.Migration

  def up do
    # Function to insert a new profile
    execute """
    CREATE OR REPLACE FUNCTION public.create_profile_for_new_user()
    RETURNS TRIGGER AS $$
    BEGIN
      INSERT INTO public.profiles (id, inserted_at, updated_at)
      VALUES (NEW.id, now(), now());
      RETURN NEW;
    END;
    $$ LANGUAGE plpgsql SECURITY DEFINER;
    """

    # Trigger to call the function after a user is inserted
    execute """
    CREATE TRIGGER trigger_create_profile_after_user_insert
    AFTER INSERT ON auth.users
    FOR EACH ROW
    EXECUTE FUNCTION public.create_profile_for_new_user();
    """
  end

  def down do
    execute "DROP TRIGGER IF EXISTS trigger_create_profile_after_user_insert ON auth.users;"
    execute "DROP FUNCTION IF EXISTS create_profile_for_new_user;"
  end
end

And that's it, you now have extended the ability to store information for the users while keeping the security provided by the protected auth.users table. But now how do you use it?

Well, let me show you how in fewer than 90 lines of code:

defmodule UserManagement do
  @req Req.new(
         base_url: Application.compile_env(:myapp, [:supabase, :base_url]),
         headers: [apikey: Application.compile_env(:myapp, [:supabase, :api_key])],
         url: "/auth/v1/:action"
       )

  def get_current_user(bearer_token) do
    Req.get!(
      @req,
      auth: {:bearer, bearer_token},
      path_params: [action: "user"]
    )
    |> Map.get(:body)
  end

  def signup_with_username_and_password(email, password) do
    Req.post!(
      @req,
      path_params: [action: "signup"],
      json: %{email: email, password: password}
    )
    |> Map.get(:body)
  end

  def login_with_email_and_password(email, password) do
    Req.post!(
      @req,
      path_params: [action: "token"],
      params: [grant_type: "password"],
      json: %{email: email, password: password}
    )
    |> Map.get(:body)
  end

  def send_password_recovery_email(email) do
    Req.post!(
      @req,
      path_params: [action: "recover"],
      json: %{email: email}
    )
    |> Map.get(:body)
  end

  def update_user(bearer_token, data \\ %{}) do
    {email, data} = Map.pop(data, "email")
    {password, data} = Map.pop(data, "password")

    body = %{
      "data" => data
    }

    body = if email, do: Map.put(body, "email", email), else: body
    body = if password, do: Map.put(body, "password", password), else: body

    Req.put!(
      @req,
      path_params: [action: "user"],
      auth: {:bearer, bearer_token},
      json: body
    )
    |> Map.get(:body)
  end

  def logout(bearer_token) do
    Req.post!(
      @req,
      auth: {:bearer, bearer_token},
      path_params: [action: "logout"]
    )
    |> Map.get(:body)
  end

  def send_email_invite(bearer_token, email) do
    Req.post!(
      @req,
      auth: {:bearer, bearer_token},
      json: %{email: email},
      path_params: [action: "invite"]
    )
    |> Map.get(:body)
  end
end

Pretty simple right? You could surely condense this more too if you so choose.

💡

The wonderful Req library takes care of a lot of the tedious parts of these requests, such as JSON-encoding and Bearer authentication. I highly recommend it over other HTTP clients for these reasons.

As long as you set your API key and Supabase instance URL into the application environment, this will have you ready to perform all of your user management tasks, and upon registration have the new user reflected in the auth.users table as well as the public.profiles table.

And lastly, make sure you connect to your database using the connection string (or you can specify each field if you'd like).

import Config

config :myapp, MyApp.Repo,
  url:
    "myconnectionstring",

This barely scratched the surface of what you can do with Supabase, but I hope it at least demonstrates how quick it is to get started with it and how flexible it is to have complete control over your stack.

Of course this example was in Elixir, but you could extend this to any other backend language and get the same benefits, and avoid falling victim to vendor lock-in!

Elevate Your Elixir With Sigils

Sun, 19 Apr 2026 04:26:32 +0000

Motivation

In a previous article of mine, I wrote about using NimbleOptions to add extremely powerful option handling to your Elixir applications. One of the custom validations I was using was a function in_range that would check if an option fell within a real-valued interval. This differs from Elixir's built-in Range in that it needed to be real-valued (rather than discrete integer steps). Additionally, mostly due to aesthetic and personal opinion, I wanted to be able to express the intervals using mathematical notation such as (0,1] to mean "allow any value greater than 0 and less than or equal to 1". I find Elixir to be such a beautiful language with a unique capacity for extensions that it felt wrong to use a function such as in_range or in_interval. Additionally, some implementations I've come across have somewhat unintuitive APIs, such as the following spec:

@spec in_range(float, float, float, bool, bool) :: bool
@doc """
  * `:value` - Value to test for inclusion
  * `:min` - Minimum value in range
  * `:max` - Maximum value in range
  * `:left` - Whether the left boundary is inclusive (true) or exclusive (false)
  * `:right` - Whether the right boundary is inclusive (true) or exclusive (false)
"""
def in_range(value, min \\ 0, max \\ 1, left \\ true, right \\ true)

There's nothing expressly wrong with this implementation, but with my use cases and as Elixir is being used more in the domain of Machine Learning which deals with these intervals quite often, I wanted a solution that felt a bit more integrated.

Solution

This led me to create a small 1-file library called Exterval which is available on Hex and can be installed with:

def deps do
[
  {:exterval, "~> 0.1.0"}
]
end

To make the interval feel more native to Elixir, I implemented it as a sigil that implements theEnumerable Protocol, which gives you several nice benefits:

Takes advantage of the member?/2 function which means we can use the in keyword to check for membership
Allows for checking of sub-interval membership (with some caveats)
Implements an optional step parameter that allows you to iterate/reduce over the interval
Implements a size function (remember, size refers to the ability to count the number of members without reducing over the whole structure, whereas lengths implies a need to reduce).
Allows for :infinity and :neg_infinity to be specified in the interval

This lets us write more succinct checks like:

iex> import Exterval
iex> ~i<[1, 10)//2>
[1, 10)//2
iex> ~i<[1, 10)//2> |> Enum.to_list()
[1.0, 3.0, 5.0, 7.0, 9.0]
iex> ~i<[1, 10)//2> |> Enum.sum()
25.0
iex> ~i<[-1, 3)//-0.5> |> Enum.to_list()
[2.5, 2.0, 1.5, 1.0, 0.5, 0.0, -0.5, -1.0]
iex> ~i<[1, 10]> |> Enum.count()
:infinity
iex> ~i<[1, 10)//2> |> Enum.count()
4
iex> ~i<[-2,-2]//1.0> |> Enum.count()
1
iex> ~i<[1,2]//0.5> |> Enum.count()
3
iex> ~i<[-2,-1]//0.75> |> Enum.count()
2
iex> 1 in ~i<[1, 10]>
true
iex> 1 in ~i<[1, 10)//2>
true
iex> 3 in ~i<(1, 10)//2>
true
# You can even do variable substitution using string interpolation syntax, since the sigil parameter is just a string
iex> min = 2
iex> 3 in ~i<(#{min + 1}, 10)//2>
false

Design Details

The decision to implement the interval as a sigil was not as straightforward as it might seem. As I mentioned before, Elixir is an extremely extensible language with superior support for meta-programming, so implementing this as a macro was my first instinct. I considered commandeering the opening brackets ( and [ to trigger the macro, or something similar with the comma , , but fortunately, I hit a brick wall with that effort. I say fortunately not only because it would have been a bad idea from a design perspective, but it certainly would have been a messier implementation and would have overly complicated it in addition to actually making the code less clear. I appreciate the usage of the sigil ~I because it makes it clear that the range that follows is not to be confused with the built-in Range.

💡

You can read more about Elixir sigils here and see their syntax reference here. Of note, you can use any of the allowed delimiter pairs that it lists to capture your sigil. I chose [ and ] so as to not conflict with the brackets used in the interval. You could also use something like ~i|[0,1)| if you prefer.

Once I decided on the usage of the Enumerable protocol, I knew I wanted to allow some way for an optional step size to be specified so that reduce could be used on the structure. Elixir sigils allow for parameters to be passed after the closing sigil, so initially, I considered passing in the step size as a parameter since zero or more ASCII letters and digits can be given as a modifier to the sigil, but this would prohibit having floats as step sizes. Another constraint to consider when using sigils is that string interpolation is only allowed within the sigil when using a lowercase sigil. Sigils start with ~ and are followed by one lowercase letter or by one or more uppercase letters, immediately followed by one of the allowed delimiter pairs. Within the context of our use case, we happen to be able to get by without having string interpolation since we use it within pre-defined parameters that are hardcoded, but the library becomes much more useful if we can have dynamically defined intervals, so this limits how the sigil is named.

Another major design decision was how to actually parse the sigil. I ultimately landed on the straightforward answer of just using a regex, but I had a decent back-and-forth with my friend Paulo from the Elixir-Nx core team regarding other options. He provided some nice proofs of concept using binary pattern matching as well as NimbleParsec, but I decided on a regex due to my familiarity, its ability to reduce the amount of code, and because I was not too concerned with performance concerns with what will typically be short patterns.

One of the last design details finalized was how to treat the step size and its effect on item membership. Paulo and I discussed whether it should support ranges where the min and max values did not necessarily have to be in the correct order (e.g. ~i<1,-1//0.5>) which would essentially imply that any iteration would start at 1 in this instance and would work towards -1 in steps of 0.5. This was discussed since it can be seen in some other implementations throughout other ecosystems. We decided that the most clear solution, as well as the solution that fit best within the spirit of the library, was to enforce that the first value specified be less than or equal to the second value, and any desire to iterate starting with the max value could be specified using a negative step size.

Implementation Details

Creation

An interval is stored as a struct with the following fields:

left - the left bracket, either [ or (.
right - the right bracket, either ] or ).
min - the lower bound of the interval. Can be :neg_infinity or any number.
max - the upper bound of the interval. Can be :infinity or any number.
step - the step size of the interval. If nil, the interval is continuous.

To define a sigil, you create a function with the name of the sigil prefixed by sigil_, so since I wish to use this sigil using ~i I define it as

def sigil_i(pattern, []) do
end

The second parameter are the options to the sigil I mentioned earlier. For now these are unused.

I parse the input to the sigil using the following regex:

^(?P<left>\[|\()\s*(?P<min>[-+]?(?:\d+|\d+\.\d+)(?:[eE][-+]?\d+)?|:neg_infinity)\s*,\s*(?P<max>[-+]?(?:\d+|\d+\.\d+)(?:[eE][-+]?\d+)?|:infinity)\s*(?P<right>]|\))(?:\/\/(?P<step>[-+]?(?:[1-9]+|\d+\.\d+)(?:[eE][-+]?\d+)?))?$

Using the named capture groups I perform some additional validation such as ensuring that the interval goes from the minimum value to the maximum.

Enumerable – Size / Count

The first function I need to implement for the protocol is the Enumerable.count/1 function. Logically, there are three conditions to account for. First are the instances where the size is either zero or infinity. Since Enumerable.count/1 must return a number on success, I choose to return {:error, Infinity} from Enumerable.count/1 when I wish to return :infinity. This would normally be used to return a module which can perform a reduction to compute the count, but if we just make a simple helper module

defmodule Infinity do
  @moduledoc false
  def reduce(%Exterval{}, _, _), do: {:halt, :infinity}
end

Now I can get my desired behavior. I implement these cases with the following:

def size(interval)
def size(%__MODULE__{step: nil}), do: {:error, Infinity}
def size(%__MODULE__{max: :neg_infinity}), do: 0
def size(%__MODULE__{min: :infinity}), do: 0

def size(%__MODULE__{min: min, max: max})
    when min in [:infinity, :neg_infinity] or max in [:infinity, :neg_infinity],
    do: {:error, Infinity}

Lastly I separate cases where the step size is negative and where its positive since the logic is different.

def size(%__MODULE__{left: left, right: right, min: min, max: max, step: step}) when step < 0 do
  case {left, right} do
    {"[", "]"} ->
      abs(trunc((max - min) / step)) + 1

    {"(", "]"} ->
      abs(trunc((max - (min - step)) / step)) + 1

    {"[", ")"} ->
      abs(trunc((max + step - min) / step)) + 1

    {"(", ")"} ->
      abs(trunc((max + step - (min - step)) / step)) + 1
  end
end

def size(%__MODULE__{left: left, right: right, min: min, max: max, step: step}) when step > 0 do
  case {left, right} do
    {"[", "]"} ->
      abs(trunc((max - min) / step)) + 1

    {"(", "]"} ->
      abs(trunc((max - (min + step)) / step)) + 1

    {"[", ")"} ->
      abs(trunc((max - step - min) / step)) + 1

    {"(", ")"} ->
      abs(trunc((max - step - (min + step)) / step)) + 1
  end
end

Enumerable – Reduce

The implementation for reduce is a great example of how Elixir's pattern matching in function headers can reduce visual complexity and even the implementation itself. First, we return :infinity if step is nil.

def reduce(%Exterval{step: nil}, acc, _fun) do
  {:done, acc}
end

Next, we again have different clauses depending on if the step is positive or negative, since that dictates which direction with respect to the interval the reduction occur.

def reduce(%Exterval{left: left, right: right, min: min, max: max, step: step}, acc, fun)
    when step > 0 do
  case left do
    "[" ->
      reduce(min, max, right, acc, fun, step)

    "(" ->
      reduce(min + step, max, right, acc, fun, step)
  end
end

def reduce(%Exterval{left: left, right: right, min: min, max: max, step: step}, acc, fun)
    when step < 0 do
  case right do
    "]" ->
      reduce(min, max, left, acc, fun, step)

    ")" ->
      reduce(min, max + step, left, acc, fun, step)
  end
end

Notice that these clauses to the reduce/3 implementation return a different reduce/6 function which is specific to our module.

Next we handle conditions where the reduction is halted or suspended:

efp reduce(_min, _max, _closing, {:halt, acc}, _fun, _step) do
  {:halted, acc}
end

defp reduce(min, max, closing, {:suspend, acc}, fun, step) do
  {:suspended, acc, &reduce(min, max, closing, &1, fun, step)}
end

Next we handle edge cases involving :infinity and :neg_infinity where we have no way to begin the reduction since we cannot move step increments away from either of these when they are our starting point:

defp reduce(:neg_infinity, _max, _closing, {:cont, acc}, _fun, step) when step > 0 do
  {:done, acc}
end

defp reduce(_min, :infinity, _closing, {:cont, acc}, _fun, step) when step < 0 do
  {:done, acc}
end

Interestingly, these are cases where the size of the intervals would be :infinity but we cannot reduce over them at all, as opposed to other infinitely sized intervals where we can begin iteration which will never end, such as ~i<[0,:infinity]//1> which would effectively be an infinite stream starting at 0 and incrementing by 1.

Next we add all of the main logic for the "typical" cases:

defp reduce(min, max, "]" = closing, {:cont, acc}, fun, step)
     when min <= max do
  reduce(min + step, max, closing, fun.(min, acc), fun, step)
end

defp reduce(min, max, ")" = closing, {:cont, acc}, fun, step)
     when min < max do
  reduce(min + step, max, closing, fun.(min, acc), fun, step)
end

defp reduce(min, max, "[" = closing, {:cont, acc}, fun, step)
     when min <= max do
  reduce(min, max + step, closing, fun.(max, acc), fun, step)
end

defp reduce(min, max, "(" = closing, {:cont, acc}, fun, step)
     when min < max do
  reduce(min, max + step, closing, fun.(max, acc), fun, step)
end

And lastly we add the final case where the condition that min < max (or min <= max depending on the brackets) is no longer met, which means the reduction is complete:

defp reduce(_, _, _, {:cont, acc}, _fun, _up) do
  {:done, acc}
end

Just like that the reduce/3 implementation is complete! As I mentioned before and notes in more detail, there are some opinions inherit to this implementation having to do with :infinity and :neg_infinity bounds as well as empty intervals, but I tried to keep the behavior consistent throughout.

Enumerable – Membership

Now on to the part that I was most interested in, which is interval membership. First, let's add support for checking membership between two intervals, which is essentially a check for one interval being a sub-interval of another.

Sub-interval must satisfy the following to be a subset:

The minimum value of the subset must belong to the superset.
The maximum value of the subset must belong to the superset.
The step size of the subset must be a multiple of the step size of the superset.

If the superset has no step size, then only the first two conditions must be satisfied.

if the superset has a step size, and the subset doesn't then membership is false.

def member?(%Exterval{step: nil} = outer, %Exterval{} = inner) do
  res = inner.max in outer && inner.min in outer
  {:ok, res}
end

def member?(%Exterval{}, %Exterval{step: nil}) do
  {:ok, false}
end

def member?(%Exterval{} = outer, %Exterval{} = inner) do
  res = inner.max in outer && inner.min in outer && :math.fmod(inner.step, outer.step) == 0
  {:ok, res}
end

Then that just leaves the main implementation for membership checks, which is basically just a case statement which changes the output depending on the brackets supplied. Additionally, if the interval contains a step then the value being checked must be a multiple of the step.

def member?(%Exterval{} = rang, value) when is_number(value) do
  res =
    if Exterval.size(rang) == 0 do
      {:ok, false}
    else
      case {rang.left, rang.min, rang.max, rang.right} do
        {_, :neg_infinity, :infinity, _} ->
          true

        {_, :neg_inf, max_val, "]"} ->
          value <= max_val

        {_, :neg_infinity, max_val, ")"} ->
          value < max_val

        {"[", min_val, :infinity, _} ->
          value >= min_val

        {"(", min_val, :infinity, _} ->
          value > min_val

        {"[", min_val, max_val, "]"} ->
          value >= min_val and value <= max_val

        {"(", min_val, max_val, "]"} ->
          value > min_val and value <= max_val

        {"[", min_val, max_val, ")"} ->
          value >= min_val and value < max_val

        {"(", min_val, max_val, ")"} ->
          value > min_val and value < max_val

        _ ->
          raise ArgumentError, "Invalid range specification"
      end
    end

  res =
    unless is_nil(rang.step) || rang.min == :neg_infinity || rang.max == :infinity do
      res && :math.fmod(value - rang.min, rang.step) == 0
    else
      res
    end

  {:ok, res}
end

Inspect

Lastly, to make the user experience a bit better, it's not too difficult to implement the Inpect Protocol to provide a cleaner output:

defimpl Inspect do
  import Inspect.Algebra
  import Kernel, except: [inspect: 2]

  def inspect(%Exterval{left: left, right: right, min: min, max: max, step: nil}, opts) do
    concat([string(left), to_doc(min, opts), ",", to_doc(max, opts), string(right)])
  end

  def inspect(%Exterval{left: left, right: right, min: min, max: max, step: step}, opts) do
    concat([
      string(left),
      to_doc(min, opts),
      ",",
      to_doc(max, opts),
      string(right),
      "//",
      to_doc(step, opts)
    ])
  end
end

Future Plans

Currently I am weighing the options between adding more functionality to the library or keeping it as thin as it currently is. The main additions could be more robust set operations on the intervals, but I currently do not have a need for it so it will probably not make it into the library in the near future.

For now, I hope this provided a detailed look at the process of identifying a problem, and subsequently designing and implementing the solution. I found this to be an elegant solution to the problem, but as I mentioned it was not a straight-line path. I would be interested to hear about any other solutions people have seen!

Sign up for The Stack Canary

If you enjoyed reading, let me know!

Email sent! Check your inbox to complete your signup.

No spam. Unsubscribe anytime.

From Python to Elixir Machine Learning

Sun, 19 Apr 2026 04:26:32 +0000

As Elixir's Machine Learning (ML) ecosystem grows, many Elixir enthusiasts who wish to adopt the new machine learning libraries in their projects are stuck at a crossroads of wanting to move away from their existing ML stack (typically Python) while not having a clear path of how to do so. I would like to take some time to talk to WHY I believe now is a good time to start porting over Machine Learning code into Elixir, and HOW I went about doing just this for two libraries I wrote: EXGBoost (from Python XGBoost) and Mockingjay (from Python Hummingbird).

Why is Python not Sufficient?

There's a common saying in programming languages that no language is perfect, but that different languages are suited for different jobs. Languages such as C, Rust, and now even Zig are known for their targeting systems development, while languages such as C++, C#, and Java are more commonly used for application development, and obviously there are the web languages such as JavaScript/TypeScript, PHP, Ruby (on Rails), and more. There are gradations to these rules of course, but more often than not there are good reasons that languages tend to exist within the confines of particular use cases.

Languages such as Elixir and Go tend to be used in large distributed systems because they place an emphasis on having great support for common concurrency patterns, which can come at the cost of supporting other domains. Go, for example, has barely (if any?) support for machine learning libraries, but it's also not trying to cater to that as a target domain. For a long time, the same could have been said about Elixir, but over the past two or so years, there has been a massive concerted push from the Elixir community to not only have support for machine learning, but to push the envelope with the maintaining state of the art libraries that are beginning to compete with the other dominant machine learning languages - namely Python.

Python has long been the gold standard in the realm of machine learning. The breadth of libraries and the low entry barrier makes Python a great language to work with, but it does create a bit of a bottleneck. Any application that wishes to integrate machine learning has historically had only a couple of options: have a Python component or reach into the underlying libraries that power much of the Python libraries directly. Despite all the good parts of Python I mentioned before, speed and support for concurrency are not on that list. Elixir-Nx is striving to give another option - an option that can take advantage of the native distributed support that Elixir and the BEAM VM have to offer. Nx's Nx.Serving construct is a drop-in solution for serving distributed machine-learning models.

How to Proceed

Sean Moriarity, the co-creator of Nx, creator of Axon, and author of Machine Learning in Elixir, has talked many times about how the initial creation of Nx and Axon involved hours upon hours of reading source code from reference implementations of libraries in Python and C++, namely the TensorFlow source code. While I was writing EXGBoost and Mockingjay, much of my time, especially towards the beginning, was spent referencing the Python and C++ implementations of the original libraries. This builds a great fundamental understanding of the libraries as well as taught me how to identify patterns in Python and C++ and identify the Elixir pattern that could express the same ideas. This skill is invaluable, and the better I got at it the faster I could write. Below is a summary and key takeaways from my process of porting Python / PyTorch to Elixir / Nx.

Workflow Overview

Before I get to the examples from the code bases, I would like to briefly explain the high-level cyclical workflow I established while working on this effort, and what I would recommend to anyone pursuing a similar endeavor.

Understand the Macro System

Much like how there's a common strategy to reading comprehension which involves reading through the entire document once to get a high-level understanding and then doing subsequent shorter reads to gain more in-depth understanding with the added context of the entire piece, you can consider doing the same when reading code. My first step was to follow the logical flow from the call of hummingbird.ml.convert to the final result. You can use tools such as function tracers and callgraph generators to accelerate this part of the process, or manually trace depending on the extent of the codebase. I felt in my case that it was manageable to trace myself.

Read the Documentation

Once you have a general understanding of the flow and process of the original system, you can start referring to the documentation for some additional context. In my case, this lead me to the academic paper Taming Model Serving Complexity, Performance and Cost: A Compilation to Tensor Computations Approach, which was the underlying ground work and basis for their implementation. I could write a whole other blog post about the process of transcribing algorithms and code from academic papers and pseudocode, but for now just know that these are some of the most important pieces you can refer to while re-implementing or porting over a piece of source code.

Read the Source Code in Detail

This is the point in which you want to disambiguate the higher-level ideas from the first step and really gain a fine, high-resolution understanding of what is happening. There might even be some points in which you need to deconflict the source code with its documentation and/or paper reference. In those cases, the source code almost always wins, and if not, then you likely have a bug report you can file. If you see things you don't fully understand, you don't necessarily need to address it here, but you should make note of it and keep it in mind while working in case new details help resolve it.

Implement the New Code

At this point, you should feel comfortable enough to start implementing the code. I found this to be a very iterative process, meaning I would think I had a grasp on something, then would start working on implementing it, then would realize I did not understand it as well as I had thought and would work my way back through the previous steps.

Example

💡

In case you would like to follow along going forward, the Python code I will be referencing is the Microsoft Hummingbird source code (specifically their implementation of Decision Tree Compilation), and the Elixir code is from the Mockingjay source code.

Class vs. Behaviour

As a result of the reading and comprehension I did of the Hummingbird code base, I realized fairly early on that my library was going to have some key differences. One of the main reasons for these differences was the fact that the Hummingbird code base was built as a retroactive library that needed to cater to existing APIs that existed throughout the Python ecosystem. They chose to only add support for converting decision trees according to the SKLearn API. I, conversely, chose to write Mockingjay in such a way that it would be incumbent upon the authors of decision tree libraries to implement a protocol to interface with Mockingjay's convert function. This difference meant that I could establish a Mockingjay.Tree data structure that I would use throughout my library, rather than having to reconstruct tree features from various other APIs as is done in Hummingbird.

Next, Hummingbird approaches its pipeline in a very-object oriented manner, as makes sense when using Python. Here' we are focusing on the implementation of the three decision tree conversion strategies: GEMM, Tree Traversal, and PErfect Tree Traversal. It implements the following base class for tree conversions as well as PyTorch networks.

💡

Since they're inheriting from torch.nn.model they must also implement the forward method.

class AbstracTreeImpl(PhysicalOperator):
    """
    Abstract class definig the basic structure for tree-base models.
    """

    def __init__(self, logical_operator, **kwargs):
        super().__init__(logical_operator, **kwargs)

    @abstractmethod
    def aggregation(self, x):
        """
        Method defining the aggregation operation to execute after the model is evaluated.

        Args:
            x: An input tensor

        Returns:
            The tensor result of the aggregation
        """
        pass

class AbstractPyTorchTreeImpl(AbstracTreeImpl, torch.nn.Module):
    """
    Abstract class definig the basic structure for tree-base models implemented in PyTorch.
    """

    def __init__(
        self, logical_operator, tree_parameters, n_features, classes, n_classes, decision_cond="<=", extra_config={}, **kwargs
    ):
        """
        Args:
            tree_parameters: The parameters defining the tree structure
            n_features: The number of features input to the model
            classes: The classes used for classification. None if implementing a regression model
            n_classes: The total number of used classes
            decision_cond: The condition of the decision nodes in the x <cond> threshold order. Default '<='. Values can be <=, <, >=, >
        """
        super(AbstractPyTorchTreeImpl, self).__init__(logical_operator, **kwargs)

They then proceed to inherit from these base classes and have different classes for each of the three decision tree strategies as well as their gradient-boosted counterparts, leaving them with three classes for each strategies (1 base class per strategy, 1 for ensemble implementations, and 1 for normal implementations) and nine total classes.

I chose to approach this using a behaviour

defmodule Mockingjay.Strategy do
  @moduledoc false
  @type t :: Nx.Container.t()

  @callback init(data :: any(), opts :: Keyword.t()) :: term()
  @callback forward(x :: Nx.Container.t(), term()) :: Nx.Tensor.t()
  ...
end

init will perform setup functionality depending on the strategy and return the parameters that will need to be passed to forward later on. This allows for a very simple top-level api. The whole top-level mockingjay.ex file can fit here:

def convert(data, opts \\ []) do
    {strategy, opts} = Keyword.pop(opts, :strategy, :auto)

    strategy =
      case strategy do
        :gemm ->
          Mockingjay.Strategies.GEMM

        :tree_traversal ->
          Mockingjay.Strategies.TreeTraversal

        :perfect_tree_traversal ->
          Mockingjay.Strategies.PerfectTreeTraversal

        :auto ->
          Mockingjay.Strategy.get_strategy(data, opts)

        _ ->
          raise ArgumentError,
                "strategy must be one of :gemm, :tree_traversal, :perfect_tree_traversal, or :auto"
      end

    {post_transform, opts} = Keyword.pop(opts, :post_transform, nil)
    state = strategy.init(data, opts)

    fn data ->
      result = strategy.forward(data, state)
      {_, n_trees, n_classes} = Nx.shape(result)

      result
      |> aggregate(n_trees, n_classes)
      |> post_transform(post_transform, n_classes)
    end
  end

As you can see, the use of a behaviour here allows a strategy-agnostic approach to generating a prediction pipeline. In the object-oriented implementation, each class implements init, forward, aggregate, and post_transform. We get the same result from a functional pipeline approach, where each step generates the needed information as input parameters for the next step. So, instead of storing intermediate results as object properties or values in an object's __dict__, we just pass them along in the pipeline. I would argue this creates a much simpler and easier to follow implementation (but I am also quite biased).

PyTorch to Nx

For these examples, we will be looking at porting the implementations of the forward function for the three conversion strategies from Python to Nx.

GEMM

Next, let's look at the forward function implementation for GEMM, one of the three conversion strategies. In Hummingbird, they implemented the forward step in the base class for each strategy. So given three GEMM classes with the signatures of GEMMTreeImpl(AbstractPyTorchTreeImpl), GEMMDecisionTreeImpl(GEMMTreeImpl), and GEMMGBDTImpl(GEMMTreeImpl), the forward function is defined in the GEMMTreeImpl class, since both ensemble and non-ensemble decision tree models share the same forward step.

def forward(self, x):
      x = x.t()
      x = self.decision_cond(torch.mm(self.weight_1, x), self.bias_1)
      x = x.view(self.n_trees, self.hidden_one_size, -1)
      x = x.float()

      x = torch.matmul(self.weight_2, x)

      x = x.view(self.n_trees * self.hidden_two_size, -1) == self.bias_2
      x = x.view(self.n_trees, self.hidden_two_size, -1)
      if self.tree_op_precision_dtype == "float32":
          x = x.float()
      else:
          x = x.double()

      x = torch.matmul(self.weight_3, x)
      x = x.view(self.n_trees, self.hidden_three_size, -1)

Now, here is the Nx implementation:

@impl true
  deftransform forward(x, {arg, opts}) do
    opts =
      Keyword.validate!(opts, [
        :condition,
        :n_trees,
        :n_classes,
        :max_decision_nodes,
        :max_leaf_nodes,
        :n_weak_learner_classes,
        :custom_forward
      ])

    _forward(x, arg, opts)
  end

  defnp _forward(x, arg, opts \\ []) do
    %{mat_A: mat_A, mat_B: mat_B, mat_C: mat_C, mat_D: mat_D, mat_E: mat_E} = arg

    condition = opts[:condition]
    n_trees = opts[:n_trees]
    n_classes = opts[:n_classes]
    max_decision_nodes = opts[:max_decision_nodes]
    max_leaf_nodes = opts[:max_leaf_nodes]
    n_weak_learner_classes = opts[:n_weak_learner_classes]

    mat_A
    |> Nx.dot([1], x, [1])
    |> condition.(mat_B)
    |> Nx.reshape({n_trees, max_decision_nodes, :auto})
    |> then(&Nx.dot(mat_C, [2], [0], &1, [1], [0]))
    |> Nx.reshape({n_trees * max_leaf_nodes, :auto})
    |> Nx.equal(mat_D)
    |> Nx.reshape({n_trees, max_leaf_nodes, :auto})
    |> then(&Nx.dot(mat_E, [2], [0], &1, [1], [0]))
    |> Nx.reshape({n_trees, n_weak_learner_classes, :auto})
    |> Nx.transpose()
    |> Nx.reshape({:auto, n_trees, n_classes})
  end

Do not be distracted by the length of this code snippet, as much of the lines are taken up by validating arguments. Let's look at a more stripped-down version without that:

@impl true
  deftransform forward(x, {arg, opts}) do
    _forward(x, arg, opts)
  end

  defnp _forward(x, arg, opts \\ []) do
    mat_A
    |> Nx.dot([1], x, [1])
    |> condition.(mat_B)
    |> Nx.reshape({n_trees, max_decision_nodes, :auto})
    |> then(&Nx.dot(mat_C, [2], [0], &1, [1], [0]))
    |> Nx.reshape({n_trees * max_leaf_nodes, :auto})
    |> Nx.equal(mat_D)
    |> Nx.reshape({n_trees, max_leaf_nodes, :auto})
    |> then(&Nx.dot(mat_E, [2], [0], &1, [1], [0]))
    |> Nx.reshape({n_trees, n_weak_learner_classes, :auto})
    |> Nx.transpose()
    |> Nx.reshape({:auto, n_trees, n_classes})
  end

Let's take a look at some obvious difference:

The Nx code does not have to transpose in the first step since Nx.dot/4 allows you to specify the contracting axes.
You can use Nx.dot/6 to get the same behavior as torch.matmul
- torch.matmul does a lot of wizardry with broadcasting to make this instance work
We use functions such as Nx.equal to fit into the pipeline rather than using the == oeprator (which would work outside of a pipeline)
torch.view is equivalent to Nx.reshape
Nx uses the :auto atom to where torch uses -1 to reference infering the sie of an axis

Outside of these differences, the code translates fairly easily. Let's take a look at a bit of a more complex instance.

Tree Traversal

Here is the Python implementation:

def _expand_indexes(self, batch_size):
        indexes = self.nodes_offset
        indexes = indexes.expand(batch_size, self.num_trees)
        return indexes.reshape(-1)

def forward(self, x):
        indexes = self.nodes_offset
        indexes = indexes.expand(batch_size, self.num_trees).reshape(-1)

        for _ in range(self.max_tree_depth):
            tree_nodes = indexes
            feature_nodes = torch.index_select(self.features, 0, tree_nodes).view(-1, self.num_trees)
            feature_values = torch.gather(x, 1, feature_nodes)

            thresholds = torch.index_select(self.thresholds, 0, indexes).view(-1, self.num_trees)
            lefts = torch.index_select(self.lefts, 0, indexes).view(-1, self.num_trees)
            rights = torch.index_select(self.rights, 0, indexes).view(-1, self.num_trees)

            indexes = torch.where(self.decision_cond(feature_values, thresholds), lefts, rights).long()
            indexes = indexes + self.nodes_offset
            indexes = indexes.view(-1)

        output = torch.index_select(self.values, 0, indexes).view(-1, self.num_trees, self.n_classes)

And here is the Nx implementation:

defn _forward(x, features, lefts, rights, thresholds, nodes_offset, values, opts \\ []) do
    max_tree_depth = opts[:max_tree_depth]
    num_trees = opts[:num_trees]
    n_classes = opts[:n_classes]
    condition = opts[:condition]
    unroll = opts[:unroll]

    batch_size = Nx.axis_size(x, 0)

    indices =
      nodes_offset
      |> Nx.broadcast({batch_size, num_trees})
      |> Nx.reshape({:auto})

    {indices, _} =
      while {tree_nodes = indices, {features, lefts, rights, thresholds, nodes_offset, x}},
            _ <- 1..max_tree_depth,
            unroll: unroll do
        feature_nodes = Nx.take(features, tree_nodes) |> Nx.reshape({:auto, num_trees})
        feature_values = Nx.take_along_axis(x, feature_nodes, axis: 1)
        local_thresholds = Nx.take(thresholds, tree_nodes) |> Nx.reshape({:auto, num_trees})
        local_lefts = Nx.take(lefts, tree_nodes) |> Nx.reshape({:auto, num_trees})
        local_rights = Nx.take(rights, tree_nodes) |> Nx.reshape({:auto, num_trees})

        result =
          Nx.select(
            condition.(feature_values, local_thresholds),
            local_lefts,
            local_rights
          )
          |> Nx.add(nodes_offset)
          |> Nx.reshape({:auto})

        {result, {features, lefts, rights, thresholds, nodes_offset, x}}
      end

    values
    |> Nx.take(indices)
    |> Nx.reshape({:auto, num_trees, n_classes})
  end

Here there are some much more striking differences, namely the use of Nx's while expression compared to a for loop in Python. We use while in this case since it can achieve the same purpose as the Python for loop and it is supported by Nx within a defn expression. Otherwise, we might have to perform some of the calculations within a deftransform, as we will see in the next example. Another obvious difference is that in the Nx implementation, we have to pass the required variables around throughout these operation, whereas Python can use stored class attributes.

Still, the conversion is quite straightforward. I hope you are beginning to see that this is not an impossible effort, and can be accomplished given you have a firm understanding of the source material.

Perfect Tree Traversal

Lastly, let's look at the last conversion strategy. Yet again, this conversion is even slightly more complex, but hopefully seeing this example will help you in your case:

def forward(self, x):
        prev_indices = (self.decision_cond(torch.index_select(x, 1, self.root_nodes), self.root_biases)).long()
        prev_indices = prev_indices + self.tree_indices
        prev_indices = prev_indices.view(-1)

        factor = 2
        for nodes, biases in zip(self.nodes, self.biases):
            gather_indices = torch.index_select(nodes, 0, prev_indices).view(-1, self.num_trees)
            features = torch.gather(x, 1, gather_indices).view(-1)
            prev_indices = (
                factor * prev_indices + self.decision_cond(features, torch.index_select(biases, 0, prev_indices)).long()
            )

        output = torch.index_select(self.leaf_nodes, 0, prev_indices).view(-1, self.num_trees, self.n_classes)

And the Elixir implementation:

defnp _forward(
          x,
          root_features,
          root_thresholds,
          features,
          thresholds,
          values,
          indices,
          opts \\ []
        ) do
    prev_indices =
      x
      |> Nx.take(root_features, axis: 1)
      |> opts[:condition].(root_thresholds)
      |> Nx.add(indices)
      |> Nx.reshape({:auto})
      |> forward_reduce_features(x, features, thresholds, opts)

    Nx.take(values, prev_indices)
    |> Nx.reshape({:auto, opts[:num_trees], opts[:n_classes]})
  end

  deftransformp forward_reduce_features(prev_indices, x, features, thresholds, opts \\ []) do
    Enum.zip_reduce(
      Tuple.to_list(features),
      Tuple.to_list(thresholds),
      prev_indices,
      fn nodes, biases, acc ->
        gather_indices = nodes |> Nx.take(acc) |> Nx.reshape({:auto, opts[:num_trees]})
        features = Nx.take_along_axis(x, gather_indices, axis: 1) |> Nx.reshape({:auto})

        acc
        |> Nx.multiply(@factor)
        |> Nx.add(opts[:condition].(features, Nx.take(biases, acc)))
      end
    )
  end

You can see that in this case, we have a function defined in a deftransform within our forward pipeline. Why is this so? Well, when writing definitions within defn you forfeit the use of the default Elixir kernel for the Nx.Kernel module. If you want full access to all of the normal Elixir modules, you need to use a deftransform. We needed to use Enum.zip_reduce in this instance (rather than Nx's while like before) since the features and thresholds lists are not of uniform shape. Their shape represents the length of a given depth of a binary tree, so they will be a nested list of lengths [1,2,4,8...]. This is an optimization as opposed to normal TreeTraversal, but required a bit of a different approach as opposed to the Python implementation which took advantage of torch.nn.ParameterList to build out the same lists. You might also notice the use of Tuple.to_list on lines 25 and 26. This was required since we needed features and thresholds to be stored in Nx.container's when passed into the deftransform, and Tuple implements the Nx.Container protocol, while lists do not. Even still, given that knowledge of the intricacies of defn and deftransform, the final ported solution is very similar to the reference solution.

Conclusion

In this post, I tried to accomplish several things at once, and perhaps that lead to a cluttered article, but I felt the need to address all of these points at once. I do not mean to suggest that Machine Learning has no place in Python or that Python will not continue to be the most dominant player in Machine Learning, but that I think some healthy competition is a good thing, and that perhaps Python does have some shortcomings that might give other languages valid reasons to coexist in the space.

Next, I wanted to address some specifics as to what Elixir has to offer to the machine learning space. I think it is uniquely positioned to be quite competitive considering the large community push to support more and more libraries, as well as the large application development community that can benefit from an in-house solution.

Lastly, I wanted to share some practical tips for those looking to move on from Python to Elixir, but feeling somewhat helpless in the process. I think that Sean Moriarity's book that I mentioned at the beginning of this article is an invaluable resource and great step in the education of machine learning for Elixir developers, but it can nonetheless feel daunting to seemingly throw out existing working solutions for new-fangled, perhaps not as well respected solutions. I hope I showed how anybody can approach this problem, and any existing Elixir developer can be a machine learning developer going forward. The ground work has been laid, and the tools are available. Thank you for reading (especially if you made it to the end)!

Cure, Four Releases Deep: From FSMs to Furniture

Sat, 18 Apr 2026 11:56:20 +0000

There is a particular kind of intellectual cowardice endemic to programming-language design, which consists of writing a feature that almost works, convincing oneself that it works enough, and then moving on to the next item on the roadmap before the cracks start to show. I have indulged in this pastime for years. Cure, for the first dozen or so releases, indulged in it with me. The last four tags—v0.16.0, v0.17.0 with its two patch siblings, v0.18.0, and v0.19.0 followed by v0.19.1—are my belated attempt to stop. Each of them picks one area of the language that had previously been garnished with a thin coat of varnish and strips it down to bare wood.

What follows is the tour. I will try to explain both what changed and why it had to. If the tone reads as exasperated in places, that is because I am mostly exasperated at the person who wrote the earlier versions, who was, as luck would have it, me.

v0.16.0: the turnstile that could not contain itself

For fifteen releases Cure had finite state machines as a language primitive, and for fifteen releases the canonical FSM example—the turnstile—looked like this: four lines of beautifully declarative transition graph in turnstile.cure, followed by a hundred and twenty lines of Elixir GenServer plumbing in a wrapper module that had nothing to do with turnstiles and everything to do with bridging the gap between a gen_statem process and the rest of the application. The FSM definition said what you meant. The wrapper said what the runtime demanded. You read one of them to understand the domain and the other to make the program run. I cannot in good conscience call that a first-class primitive. It was more of a first-class primitive’s slightly embarrassed cousin, the one who turns up at family dinners and talks about his crypto portfolio.

Finitomata, my Elixir library for finite automata that has been in production for years, got this right on the first attempt by insisting that the graph and the transition handler belong in the same module and behind the same abstraction. v0.16.0 borrows that insight wholesale, and then some. The turnstile now reads:

fsm Turnstile with Integer
  Locked   --coin-->  Unlocked
  Unlocked --push-->  Locked
  Unlocked --coin-->  Unlocked
  Locked   --push-->  Locked

  on_transition
    (:locked, :coin, _payload, data)   -> %[:ok, :unlocked, data + 1]
    (:unlocked, :push, _payload, data) -> %[:ok, :locked,   data]
    (:unlocked, :coin, _payload, data) -> %[:ok, :unlocked, data + 1]
    (_, _, _, data)                    -> %[:ok, :__same__, data]

The four transition lines on top are the graph you always wrote. The on_transition block underneath takes pattern-matching clauses of shape (current_state, event, event_payload, state_payload) and returns either %[:ok, next_state, new_payload] or %[:error, reason]. When the compiler sees an on_transition block it silently changes mode: instead of generating raw gen_statem Erlang abstract forms, it produces a GenServer-based Elixir module with an embedded transition table, pre-dispatch validation, compiled do_on_transition/4 clauses, and the optional lifecycle hooks on_enter, on_exit, on_failure, and on_timer. If you do not write on_transition, the FSM compiles the way it always did, through Erlang abstract forms to gen_statem. No flags, no configuration—the compiler decides based on whether you asked for the new thing.

Finitomata also contributed two conventions that I refused to live without once I tried them. A hard event, written with a trailing exclamation mark, must be the sole outgoing event of its source state, and fires automatically on arrival, so initialisation chains and guaranteed progressions stop requiring baby-sitting from the caller. A soft event, written with a trailing question mark, silently leaves the state alone if it cannot transition, instead of logging a warning and invoking on_failure. Health checks, optimistic polling, any kind of “try it, and if it does not apply this tick, never mind”—all become one-liners. The lexer grew a small counter that tracks whether it is currently inside an arrow --...-->, and only inside an arrow does it absorb the trailing ! or ? into the identifier. Everywhere else, ! stays reserved for effect annotations and ? for predicates and holes. The verifier enforces the hard-event rule; the compiler uses {:continue, ...} from the GenServer return tuple to fire hard events without yielding.

The turnstile’s wrapper is now fifty lines rather than one-hundred-and-twenty, and the fifty it retains are real application logic: counting how many people went through today. The twelve turnstile tests pass without modification, which is the only part of this release I will allow myself to take unironic pride in.

v0.17.0: toward Idris, at last

For seven versions Cure had been marketing itself as a dependently-typed language while behaving, in every practical respect, like a vaguely refinement-typed one. You could write Vector(T, n) in a signature. The checker would nod politely. It would not, however, verify that the length you promised was the length you produced, because the machinery to do so did not exist. v0.17.0 stops nodding.

Three type shapes that should have been there from the start arrive simultaneously. Sigma types pair a value with a type that depends on it; Sigma(n: Nat, Vector(T, n)) is “a natural number together with a vector of exactly that length”. Pi types let a function’s return type depend on its arguments, so the canonical example

fn append(xs: Vector(T, m), ys: Vector(T, n)) -> Vector(T, m + n)

now actually means what it reads as: at each call site the checker substitutes the concrete arguments into the return type, normalises with a tiny terminating reducer called Cure.Types.Reduce, and resolves the result. Closed type-level arithmetic never troubles the SMT solver—Vector(T, 2 + 3) is syntactically the same type as Vector(T, 5) before Z3 is even woken up. Equality types arrive with the single constructor refl(x) : Eq(T, x, x) and the single eliminator rewrite eq in expr, both erased at codegen to the atom :cure_refl. The standard library gains a Std.Equal module exposing refl, sym, trans, and cong, which is to say the usual suspects.

The second big arrival is implicit arguments with proper first-order unification. Write fn id({T}, x: T) -> T = x, call id(42), and at the call site an occurs-check-equipped unifier resolves T from the explicit argument. When resolution fails—and it does, more often than one would like—the pipeline emits a :unification_trace event carrying the argument, the position, and the substitution that killed it. The LSP renders the trace in hover; the CLI prints it in error output. No more staring at “cannot infer T” the way one stares at a Rorschach blot. Implicit parameters are erased at codegen, so they cost exactly nothing at runtime.

What ties the dependent-type stack together, in practical terms, is hole-driven development. Write

fn safe_head(xs: List(T)) -> T = ?body

and compile, and the compiler does not merely tell you “?body is missing”: it tells you ?body : T in scope: xs : List(T). Anonymous holes (??) get numbered ?_1, ?_2, in source order. Every encountered hole emits a :hole_goal event with the goal type and the local context, and the REPL’s new :holes meta-command lists everything recorded during the last evaluation. This is how Idris programmers write programs. It is now, finally, how Cure programmers can.

Totality joins the party, gently. Cure.Types.Totality classifies every function as :total, :partial, or :unknown, combining pattern-coverage (via Cure.Types.PatternChecker) with a structural-recursion check. By default totality is report-only; decorating a function with #[total] promotes the classification to a hard error if it fails:

#[total]
fn factorial(n: Int) -> Int
  | 0 -> 1
  | n -> n * factorial(n - 1)

Only direct structural recursion is caught in v0.17.0. Mutual recursion has to wait for v0.19.0.

Refinement types grow a backbone. Path-sensitive refinement flows along if and match guards, so inside

if x != 0 then 100 / x else 0

the then branch sees x : {x: Int | x != 0}, and the division is safe without an explicit refinement annotation. The new Std.Refine module ships drop-in refinements one does not wish to keep rewriting: NonZero, Positive, Negative, NonNegative, Percentage, Probability, plus predicate helpers. These are the kinds of tiny conveniences that, had they been present in v0.10.0, would have saved everyone a small amount of grief each day for a year.

The tooling catches up in the same release. Cure.REPL is a complete rewrite: multi-line input (terminated by a blank line or ;;), the meta-commands :t, :doc, :effects, :load, :reload, :use, :holes, :env, :reset, :fmt, :help, :quit, and command history persisted to ~/.cure_history. A watch mode, invoked as cure watch lib/ --action check, recompiles, type-checks, or tests on every save with a 200 ms debounce, and works without :file_system thanks to a small polling fallback. The LSP server, Cure.LSP.Server, acquires inlay hints, signature help, formatting via a round-trip-tested source-preserving printer, prepare-rename and rename, code lenses, semantic tokens, and workspace symbols—seven capabilities in one commit, which is the sort of thing that happens when one has been putting off the LSP for three releases in a row. Doctests arrive, too: any ## or ### docstring immediately above a function whose body contains cure> / => pairs is executed by cure test --doctests.

The patch releases tidied up. v0.17.1 stopped the stdlib preloader from polluting the code path, fixed two LSP crashes around inlay hints and semantic tokens, and retargeted the vicure grammar at v0.17.0 syntax. v0.17.2 re-enabled LSP formatting once the new source-preserving formatter proved it would not eat users’ comments for breakfast—a promise the old AST-based formatter had never quite been willing to make.

v0.18.0: pattern matching grows up

Here is a question I did not want to answer for a long time. If one writes

match value
  %{list: [h | t]} -> handle(h, t)
  _                -> default

and the subject value is a map whose list field is not present, what does the old compiler do? The correct answer is “fails the match and falls through to the wildcard”. The answer the old compiler produced is “succeeds, because the map pattern is miscompiled into Erlang’s construction form (K => V) rather than the match form (K := V), so it accepts any subject, and h and t are bound to whatever happens to be lying around, possibly nothing sensible”. There were tests pointing straight at this. What there was not, was a pattern compiler willing to recurse.

v0.18.0 replaces the pattern layer wholesale. The headline is that this now actually works:

match value
  %[_, %{list: [head | tail]}, _]          -> handle(head, tail)
  Person{name, address: Address{city}}     -> greet(name, city)
  [Ok(v) | _]                              -> v
  _                                        -> default

Every sub-pattern compiles as a real pattern. Nested map patterns use exact matching and actually require the key. Record patterns check the struct tag and each field. The cons pattern binds v through the Ok(...) constructor. The wildcard mops up the rest. There is no magic here; merely a module (Cure.Compiler.PatternCompiler) that does the one thing its name promises, and which every other codegen path—compile_multi_clause_function, compile_pattern_match, compile_assignment, compile_comprehension, compile_catch_and_finally—now routes through rather than reinventing pattern handling locally. The original sin, I discovered after far too long, was treating patterns and expressions uniformly because their AST nodes have the same shape. Patterns are not evaluated; they are matched against a subject with bindings as side effects. Nothing good comes of pretending otherwise.

Alongside the compiler rewrite, the type checker’s bind_pattern_vars/3 was rewritten to thread the scrutinee type through every pattern shape, so nested pattern variables now carry the tightest type the structure allows rather than the old :any. Tuple patterns zip element-wise. List and cons patterns bind head and tail at the correct element and list types respectively. Map patterns look up each key through the scrutinee’s schema when it is a known record, or through the map value type otherwise. Record patterns resolve every field against the schema registered at rec time, and unknown fields emit a warning under the new error code E021. The practical consequence is that match p { Person{age: a} when a > 17 -> ... } finally knows a : Int, so subsequent refinement fires against the right type.

Field punning arrives in both directions: Point{x, y} desugars to Point{x: x, y: y} in pattern position and in construction position. It is purely a parser change, and if you have ever found yourself writing Person{name: name, age: age, email: email, ...} seven fields deep you will be glad of it. Repeated occurrences of the same variable in one pattern now do what a reasonable person would expect: the first occurrence binds, later ones lower to equality guards. %[x, x] matches exactly the pairs whose slots are equal, which I should have supported five releases ago.

The pin operator (^name) lands behind --experimental-pin as the official escape hatch for “compare against this already-bound value”:

let target = get_tag()
match event.tag
  ^target -> :hit
  _       -> :miss

Internally it lowers to a fresh variable plus a V_fresh =:= V_target guard—the same transformation one used to write by hand. Zero runtime cost, considerably fewer characters on screen. It is promoted to default in v0.19.0.

Exhaustiveness checking grows a second pass. The old flat classifier (:wildcard | :empty_list | :cons | {:literal, ...} | {:constructor, ...} | {:tuple, n}) is still there, because it is fast and covers the common case; a new Maranget-style column walker, Cure.Types.PatternChecker.check_nested/2, now descends into tuple scrutinees whose element types are enumerable and emits concrete, source-shaped witnesses for missing patterns:

Warning: match expression has nested non-exhaustive cases (E025)
  missing: %[Error(_), _]

Five new error codes—E021 through E025—land in Cure.Compiler.Errors for unknown record fields in patterns, record field type mismatches, non-literal map-pattern keys, unbound pin variables, and non-exhaustive nested matches. cure explain E0xx works for every one.

There is a caveat worth naming explicitly. Map patterns that used to silently succeed against arbitrary subjects no longer do so. If your code relied on the old broken behaviour—for example, a map pattern whose key was not in the subject but which the compiler accepted anyway—you will now see either a runtime badmatch or a compile-time non-exhaustive warning. Fix the pattern. The compiler is right and you are wrong; I say this with the humility of the person who wrote the earlier compiler.

v0.19.0: the furniture arrives

After the pattern engine settled, a queue of previously-slated features was ready to land. v0.19.0 is called “Bring the Furniture” because every item on it is the kind of thing one should not have to talk about in a release announcement; one should merely discover, pleasantly, that it is there.

Propositions acquire a syntactic home. A proof container is a module-shaped block whose bindings must return Eq(...) or a refinement witness:

proof Std.Proof
  fn plus_zero(_n: Int)       -> Eq(Int, n, n)        = :cure_refl
  fn append_nil(_xs: List(T)) -> Eq(List(T), xs, xs)  = :cure_refl

The compiler enforces the shape under E026. Runtime values are plain :cure_refl atoms; the checker does the interesting work; the resulting BEAM module is ordinary-looking code you can load alongside any other. assert_type expr : T arrives as a zero-cost compile-time assertion: the checker verifies the type, the codegen strips the wrapper, and mismatches surface as E027. fn doubled(n: Int) -> Int = assert_type n * 2 : Int does what it looks like.

Records gain field defaults, which is the sort of feature whose absence one does not forgive oneself:

rec Person
  name: String = "Anonymous"
  age: Int = 0
  active: Bool = true

Omitted fields fall back to the declared defaults at construction time; any caller-supplied value always wins. Type mismatches between the default and the declared field type are caught as E028. In the same spirit, @derive(Show, Eq, Ord) is finally wired end-to-end. The Cure.Types.Derive module had been sitting on disk since v0.12.0 without a codegen path; v0.19.0 plumbs it through so that decorating

@derive(Show, Eq, Ord)
rec Point
  x: Int
  y: Int

synthesises plain show/1, eq/2, and compare/2 exports. The accompanying example, examples/derived_show.cure, constructs two Point values, asks eq(p, q), and returns one if they compare equal—which, naturally, they do.

Property-based testing shows up via two cooperating stdlib modules. Std.Gen ships tiny stateless generators (int_in, bool, list_int, list_of_int, one_of, constant) backed by :rand. Std.Test.forall(gen, property, runs) runs the property against samples and returns :ok or raises :property_failed:

mod Laws
  use Std.Gen
  use Std.Test

  fn test_plus_zero() -> Atom =
    forall(fn(_) -> int_in(-100, 100), fn(n) -> n + 0 == n, 100)

Shrinking, histograms, and stateful generators are future work; this is deliberately the minimum viable property tester, and it is enough to catch the kind of bug one always catches with a property tester in the first week.

Std.Iter is the matching minimal lazy iterator protocol. Constructors empty/0, from_list/1, and range/2; consumers fold/3, take/2, to_list/1. take/2 stops before materialising the tail, so unbounded ranges are safe as long as you only peek at a prefix:

use Std.Iter

fn sum_range(n: Int) -> Int =
  let it = range(1, n)
  let add = fn(x) -> fn(acc) -> acc + x
  fold(it, 0, add)

The package-registry groundwork lands as a version parser and a dependency resolver. Cure.Project.Version handles SemVer plus compound constraints (~>, >=, <=, <, >, ==, combined with and), and accepts MAJOR.MINOR as shorthand for MAJOR.MINOR.0. Cure.Project.Resolver.resolve/2 is a deterministic backtracking resolver over a local registry that picks the newest compatible version and surfaces conflicts as E030. A remote index service, signing, and Hex.pm cross-publishing are what v0.20.0 is for.

Totality catches up with multi-function cycles. Cure.Types.Totality.check_mutual/1 runs Tarjan’s SCC algorithm on the module call graph, then classifies each non-trivial strongly-connected component as either :ok (structural decrease proven on at least one path) or :suspect (E029). It is not a full termination checker—nothing running at compile time on the BEAM is going to be—but it is strictly more than nothing, and it catches the obvious cases.

One last syntactic nicety: multi-head cons patterns,

match xs
  [a, b, c | rest] -> a + b + c
  _                -> 0

desugared by the parser into right-associated cons cells. Works in pattern and construction position. It is the sort of feature that every functional language grows eventually; we might as well have grown ours here.

Five new error codes (E026 through E030) round out the error catalog for the release, all of them available via cure explain.

v0.19.1: Dialyzer and the small sins

The point release should never be the interesting one, and v0.19.1 honours that convention. Its job was to add Dialyzer to the CI matrix (.github/workflows/ci.yml), resolve the backlog of specs it turned up across Cure.Compiler, Cure.Compiler.Codegen, Cure.FSM.Compiler, Cure.Types.Env, and friends, and tidy a handful of things that were unclean but not broken: the mix cure.escript task got proper CLI integration, mix.exs lost a dead dependency, and the stdlib preloader started force-compiling in tests instead of hoping the beams from the last run would still be fresh. None of this is exciting. All of it needed to happen, because a language without Dialyzer in CI is a language whose maintainer is lying to themselves about how robust its internals are.

The small print, for anyone upgrading. The lexer rule for trailing ? on identifiers introduced in v0.17.0 means x?y now tokenises as x? followed by y; add whitespace or parentheses if you need the old behaviour. Map patterns that relied on the pre-v0.18.0 construction-form bug will now fail matches honestly. None versus None() inside a record pattern now emits a warning when you probably meant the nullary constructor. And Cure.lock lockfiles produced by v0.19.0 and v0.19.1 remain source-compatible with v0.17.0 and later, but a mix cure.compile_stdlib && mix cure.check after pulling is cheap insurance.

What all this adds up to

If you have read this far you may have noticed a pattern. Each of these releases took a subsystem that the earlier Cure treated as decorative and made it load-bearing. FSMs stopped being a transition graph with a separately-authored runtime and became a language primitive one can actually define in one place. The dependent-type core stopped nodding politely at Vector(T, m + n) and started checking it. The pattern engine stopped accepting whatever came past and started matching. The furniture release filled in the items one would notice the absence of but would not necessarily think to name: defaults, derive, property tests, a lazy iterator, a version parser, mutual-recursion totality. And the point release added the tool (Dialyzer) that would have prevented some of these embarrassments from landing in the first place.

There is still a long queue. Full bitstring pattern specifiers. Refinement narrowing through nested record and map patterns. The remote package-registry index service with its signing and Hex.pm cross-publishing. True dependent types in their full glory—fn len(l: Vector(_, n)) -> NonNegInt = n remains, for the moment, a thing I write in the v0.2x plan rather than in actual Cure. None of this will be done quickly. All of it will be done, if the last four releases are any indication, by ripping out the almost-works version and replacing it with one that does.

Clone it, build it, break it:

git clone https://github.com/am-kantox/cure-lang.git
cd cure
mix deps.get && mix test
mix escript.build
./cure version
./cure run examples/destructuring.cure
./cure run examples/derived_show.cure
./cure run examples/lazy_iter.cure

The repository is at github.com/am-kantox/cure-lang, the site is at cure-lang.org, and the furniture, as of v0.19.1, is indoors.

SAFE: Bringing Real Static Analysis to the BEAM - Erlang Solutions

Sat, 18 Apr 2026 03:56:15 +0000

In recent years, software security has become a hot topic due to regulatory pressure (NIS2, EU Cyber Resilience Act, etc.). Beyond regulations, software communities and open source maintainers have also put security into focus, because open source libraries are often part of commercial software supply chains. This has reached the Erlang/Elixir ecosystem as well, and that is a good thing.

In this post, the SAFE team takes us through SAFE, Erlang Solutions’ security analysis tool for the BEAM, covering what it does, how it works, and what makes it effective in practice.

The BEAM is secure by default (Up to a point)

The BEAM’s architecture eliminates entire classes of bugs for free. Isolated process memory means processes can’t manipulate each other’s state, they only communicate through messages. Immutable data structures rule out a whole category of aliasing bugs. These are real wins, and they come without any effort from the developer.

But “secure by default” only goes so far. Application-level vulnerabilities, e.g., XSS, SQL injection, CSRF, unsafe deserialization, atom exhaustion, are just as possible in Erlang and Elixir as anywhere else. That’s the gap SAFE exists to cover.

What is SAFE?

SAFE (Security Analysis for Erlang/Elixir) is Erlang Solutions’ static analysis tool for the BEAM. It analyses compiled BEAM files rather than source code, so it works consistently across Erlang, Elixir, and Phoenix, including mixed-language codebases. SAFE is free for open source projects (subject to approval) and commercially available for other use cases.

It is developed in collaboration with academic research on static analysis from Eötvös Loránd University , and are aligned with the security recommendations of the Erlang Ecosystem Foundation (EEF).

SAFE detects a broad range of vulnerabilities, including:

Cross-Site Scripting (XSS)
SQL injection
Command injection
Remote Code Execution
Denial of Service (e.g. atom exhaustion)
Unsafe serialisation
Cross-Site Request Forgery (CSRF)
Session hijacking, fixation, and information leakage
Content Security Policy (CSP) misconfigurations

Data-flow analysis: the core of what makes SAFE different

The central feature that sets SAFE apart is data-flow analysis. Most static analysis tools work by pattern matching, they look for known dangerous function calls and flag them. The problem is that not every call to a dangerous function is actually dangerous. Without understanding the possible values flowing through the code, a tool has no way to tell the difference, and the result is a high rate of false positives.

SAFE takes a different approach. Data-flow analysis tracks what values variables can hold at each point in the program. This information is then used to filter the initial list of vulnerability candidates, eliminating findings where the data can be proven safe, and surfacing only the ones that represent real risk.

The practical impact of this is significant. In our tests across 7 popular open source BEAM projects (~70,000 lines of code), SAFE produced a false positive rate of 7.78% and that number continues to improve as we refine our analysis. We also manually review findings during development to further sharpen the filtering.

How data-flow analysis eliminates false positives

Example 1 — guarded atom creation

A common pattern in Elixir is to convert a binary to an atom only after validating it against an allowlist:

     
    def safe_to_atom(binary, allowed) do    
      if Enum.member?(allowed, binary), do: String.to_atom(binary)    
    end

A pattern-matching tool sees String.to_atom/1 and flags it. SAFE’s data-flow analysis traces the possible values ofbinaryat the point of the call and determines that it is always a member of a finite, controlled list so it eliminates the finding entirely.

Example 2 — finite compile-time atom generation

Metaprogramming is common in Elixir. Consider this pattern where atoms are generated at compile time:

 @variants [:case_a, :case_b, :case_c]    
    #    
    # ...    
    #    
    for var <- @variants do    
      defp unquote(var)() do    
        env = Application.get_env(:my_app, :environment)    
        if env == "test" do    
          unquote(Macro.escape(Module.get_attribute(__MODULE__, :"test_#{var}")))    
        else    
          unquote(Macro.escape(Module.get_attribute(__MODULE__, var)))    
        end    
      end    
    end

The number of atoms created here is strictly bounded by the length of @variants, a compile-time constant. SAFE calculates this and correctly determines the atom count is finite hence no vulnerability. A tool without data-flow analysis cannot make this determination.

What SAFE catches in practice

Session management vulnerabilities

Session management vulnerabilities allow attackers to gain unauthorised access to user sessions, which can lead to data theft, unauthorised actions, and account takeover. SAFE detects session hijacking, session fixation, and session information leakage. Session hijacking occurs when an attacker gains access to cookie contents. To prevent this, thehttp_only and secure attributes should both be set to true when setting a cookie. Below is a vulnerable example:

 @spec set_cookie(Plug.Conn.t()) :: Plug.Conn.t()    
    def set_cookie(conn) do    
      Plug.Conn.put_resp_cookie(conn, "my_cookie", "true",    
        http_only: false,    
        max_age: @max_age # an integer    
      )    
    end

Session fixation is an attack where a malicious user plants a session ID for a victim to use, then hijacks their account after login. The Plug.Session API provides the configure_session/2 function for renewing the session ID. When this function is misconfigured by setting the renew option to false, session fixation can occur.

Session information leakage can be prevented by encrypting cookie contents. Encryption can be enabled by setting the encryption_saltin Plug.Session. For non-session cookies, encryption can be enabled via the encrypt option in Plug.Conn.put_resp_cookie/4.

Content Security Policy misconfigurations

When it comes to Content Security Policy (CSP), any policy is better than none. Using :put_secure_browser_headers without a custom policy won’t be enough on its own:

   plug :put_secure_browser_headers, %{    
      "content-security-policy": "[Your Policy]"    
    }

There are also dedicated plugs for this, such as PlugContentSecurityPolicy:

   plug PlugContentSecurityPolicy,

SAFE inspects the policy content itself and will flag an overly permissive default. You should define your own policy, keeping it as restrictive as possible to avoid unintentionally whitelisting too much. Beyond that, SAFE checks that all pipelines accepting HTML have CSP protection at all, a gap that is easy to miss during development.

Results from the field

The top three vulnerability types are XSS, DoS, and CSP. Notably, a large share of DoS findings trace back to unguarded String.to_atom/1 calls, a known footgun that is consistently flagged in Erlang/Elixir documentation, yet still appears frequently in practice.

SAFE found a total of 90 vulnerabilities, of which 7 were false positives after manual investigation, a false positive ratio of 7.78%, and an area of active improvement.

The projects tested are anonymized to avoid identification, since the disclosed vulnerabilities may still be present in production systems. While this limits full reproducibility, responsible disclosure takes priority. Contact us at safe@erlang-solutions.com to discuss the methodology in detail.

Closing notes

Security analysis on the BEAM is a genuinely hard problem. The same flexibility that makes Erlang and Elixir so expressive also makes it easy to introduce subtle vulnerabilities without realising it. SAFE is built specifically for this environment, grounded in academic research, and designed to give you signals you can act on rather than noise you have to filter.

If you maintain an open-source project, SAFE is free: reach out to us at safe@erlang-solutions.com and after a short approval process you’ll receive a licence at no cost. For commercial use or a third-party security review of your system, get in touch with the team.

The Long Road to Cure

Thu, 16 Apr 2026 09:19:20 +0000

Just under a year ago I set out to fulfil a long-standing ambition: building a language that compiles to BEAM and implements two capabilities essential to my daily work—dependent types resolved and verified at compile time, and verifiable finite state machines as a first-class language primitive.

I chose Erlang as the implementation language because it seemed to me that writing a compiler would be easier that way. I piled together a heap of attractive little baubles, many of which were added simply “because it is fun to have them.” Quite quickly I lost my way, and before long I lost control of the resulting spaghetti. I stopped understanding what was breaking and why whenever I added something that looked innocuous. I brought in an LLM, and the blasted thing informed me that the codebase had been authored by a drunk lumberjack with a primary-school education—someone who could not be trusted with anything more complex than bubble sort. Not in those exact words, but close enough.

That upset me. I lost my temper, demanded that the wretched language model knock together a presentation site in five minutes on the spot (without specifying any details whatsoever) and shipped it. The announcements, predictably, were received with polite indifference. People liked the ambition behind the idea (naturally!), but calling the implementation good was something not even my mother would have attempted.

I shelved the coding—and recoding—for several months and set about thinking about what I had done wrong.

First: over the last decade I had worked primarily with Elixir, and choosing Erlang had been a momentary, entirely unjustified whim. One can call :compile.forms/2 perfectly well from Elixir. Moreover, the ecosystem allowed me to abandon make-files and a rather crude build setup.

Second: on my first approach to the apparatus I was trying to hunt ducks, hares, and wild boar simultaneously. There was nothing systematic about that approach—I just kept adding and adding new things, propping up the collapsing frame with sticks as I went. Adding a new operator could break module parsing entirely. In the current version I followed the plan strictly: better slow and coherent than fast and obscure.

And most importantly: in the first version I was in such a hurry that I repeated the mistake of almost every existing language—the AST was present as an annexe, a shed around the back. The lexer and parser could hand control directly to the compiler. That turned out to be the critical error.

Having reconsidered all this, I decided to start the rebuilding from the foundations. That is how the metastatic library came into being. Put another way: I not only placed the AST at the centre of things but made it tower above every other entity in the system.

Metastatic is a library that provides a unified MetaAST (Meta-level Abstract Syntax Tree) intermediate representation for parsing, transforming, and analyzing code across multiple programming languages using a three-layer meta-model architecture. Build tools once, apply them everywhere. Create a universal meta-model for program syntax that enables cross-language code analysis, transformation, and tooling. Metastatic provides the foundation—the MetaAST meta-model and language adapters. Tools that leverage this foundation (mutation testing, purity analysis, complexity metrics) are built separately.

If you look closely, MetaAST bears a strong resemblance to Elixir’s own AST, because there is simply no better way to represent a tree than {node, meta, children}. In any case, I invested considerable effort in creating and debugging this new AST. It became the foundation of my second attempt at a “dependently-typed programming language for the BEAM virtual machine with first-class finite state machines and SMT-backed verification,” as it reads on the landing page.

After that I opened the casket marked “things from the previous century,” pulled out an actual pen and a notebook, and wrote out all the features from the first version that I had managed to implement, however imperfectly. I grouped them by importance, utility, and complexity. At this first stage I decided to forgo support for true dependent types—fn len(l: Vector(_, n)) -> NonNegInt = n—but a great deal of interesting things survived.

FSMs as first-class citizens are still in a fairly embryonic state, but I am in no hurry now, and I shall bring them to completion steadily and carefully.

Give https://cure-lang.org a try—perhaps you will like it.

Avoiding Platform Lock-In in Regulated Environments

Mon, 13 Apr 2026 05:37:46 +0000

Platform lock-in is often discussed as a commercial issue. Organisations adopt infrastructure that works well initially and later realise that moving away from those services becomes expensive or operationally disruptive.

For platforms that run continuously under heavy demand, the consequences appear somewhere else first. They appear in architecture.

Infrastructure choices influence how systems scale, how faults are contained, and how easily the platform can evolve as requirements change. In regulated environments those decisions often remain in place for years, which means architectural flexibility matters as much as technical capability.

When infrastructure becomes tightly coupled to a particular provider, systems may still perform well day to day. The real impact usually surfaces later when workloads grow, regulations change, or operational expectations increase. At that point platform lock-in risks begin to affect reliability as well as flexibility.

Why Platform Lock-in Matters in Regulated Environments

These architectural constraints become particularly visible in regulated industries where infrastructure decisions cannot be changed casually.

Financial services platforms must maintain traceable transactions and strict audit trails. Betting platforms process large volumes of activity during live sporting events. Streaming platforms deliver real-time content to global audiences who expect uninterrupted interaction.

Systems supporting these environments often remain active for long periods, which means infrastructure decisions made early in the system’s lifecycle can shape how the platform changes years later.

The Operational Impact of Vendor Lock-in

Many organisations already recognise the risks associated with vendor lock-in. The 2024 Flexera State of the Cloud Report found that 89% of organisations now operate multi-cloud strategies, with reducing infrastructure dependency and avoiding vendor concentration cited as key motivations.

The concern goes beyond procurement strategy. When platforms rely heavily on provider-specific services for messaging, orchestration, or event processing, those dependencies begin shaping how the system behaves under load.

In regulated environments that dependency can become a reliability concern. Infrastructure decisions that once simplified development may later restrict how systems scale, evolve, or respond to operational change.

Distributed Systems Architecture and Long-Running Platforms

The reason platform lock-in becomes particularly serious in regulated environments is tied to how many of these platforms operate: as long-running distributed systems.

Large-scale entertainment services rarely behave like short-lived workloads that restart frequently. Messaging layers, real-time interaction systems, and event pipelines maintain persistent connections while processing continuous streams of activity.

Why Long-Running Systems Behave Differently

Gaming platforms illustrate this clearly. Competitive environments host thousands of players interacting simultaneously, all of whom expect consistent state across the system. Betting platforms experience similar behaviour during major sporting events when users react instantly to changing odds. Streaming platforms see comparable spikes as audiences interact during live broadcasts.

These platforms rely on distributed systems architecture that must coordinate large numbers of connections and events while remaining continuously available.

Research published by ACM Queue examining large-scale distributed systems highlights how persistent connections and real-time workloads increase coordination pressure across system components, particularly during sudden spikes in concurrency.

When coordination layers rely heavily on platform-specific services, architectural dependency gradually builds. Over time the system begins to inherit those infrastructure constraints.

Reliability Requirements in High Reliability Systems

Systems operating under these conditions often prioritise stability over rapid iteration. Platforms designed as high reliability systems must remain available while managing constant traffic, evolving workloads, and unpredictable user behaviour.

Infrastructure decisions therefore have long-term consequences. When coordination, messaging, or state management rely on proprietary platform services, architectural flexibility narrows over time.

Why Gaming, Betting and Streaming Platforms Reveal Infrastructure Limits

Systems built as long-running distributed environments face their toughest tests during moments of concentrated demand. Entertainment platforms provide a clear example.

Large audiences often react simultaneously. A football match entering extra time can trigger thousands of betting transactions within seconds. A major esports tournament can bring large numbers of players online at once. Streaming platforms experience bursts of interaction as viewers respond together during live broadcasts.

Traffic Spikes and Scalable Distributed Systems

Systems supporting these environments must function as scalable distributed systems capable of handling sudden increases in activity without losing consistency or responsiveness.

Instead of steady growth, activity often arrives in waves. Large numbers of users connect, interact, and generate events within very short timeframes. The system must coordinate these interactions across multiple nodes while maintaining reliable communication between services.

Infrastructure that appears sufficient under normal conditions can struggle during these spikes if the surrounding architecture relies too heavily on provider-specific services.

Real-World Example: BET Software

These architectural pressures are particularly visible in betting platforms where activity surges during live sporting events.

BET Software operates large-scale betting technology platforms where thousands of users interact with markets simultaneously. During major sporting events systems must process rapid updates, recalculate market information, and distribute new data to users in real time.

Their distributed systems illustrate how reliability and responsiveness become essential in environments where activity concentrates around shared moments.

Architectures designed with flexibility across infrastructure layers tend to scale and recover more predictably than those tightly coupled to provider-specific services.

Architectural Patterns to Avoid Vendor Lock-in

Recognising the risks of vendor lock-in is useful only if it leads to better architectural decisions. Systems that remain adaptable across infrastructure layers often share several structural characteristics.

Decoupling Infrastructure Dependencies

Architectures designed to avoid vendor lock-in typically separate application logic from infrastructure services wherever possible. This allows teams to evolve system components independently without redesigning the entire system,

Designing Fault Tolerant Systems

Platforms that must operate continuously also benefit from architectures designed as fault tolerant systems, where failures can be contained locally rather than cascading across the entire platform.

Common patterns include:

Decoupled services that scale independently
Communication through open protocols rather than proprietary messaging layers
Distributed state management instead of provider-specific coordination services
Horizontal scaling across nodes
Infrastructure abstraction layers separating application logic from provider-specific implementations
These approaches help ensure that infrastructure choices support the system rather than define its limitations.

These patterns help ensure that infrastructure choices support the system rather than define its limitations.

Where Elixir Supports High Reliability Systems

Technology choices also influence how easily distributed systems can maintain reliability while remaining adaptable.

Languages built on the Erlang virtual machine, including Elixir, were designed for environments where systems must remain available while handling large numbers of concurrent processes. The runtime emphasises process isolation and supervision structures that allow failures to be contained locally rather than cascading across the system.

Building Fault Tolerant Systems for Long-Running Platforms

These characteristics make the platform particularly well suited for high reliability systems that must remain active while managing heavy concurrency.

The advantage lies in the runtime model rather than any single infrastructure provider. Systems built around resilient distributed behaviour are easier to evolve because they remain stable even as infrastructure decisions change around them.

Designing Systems That Reduce Platform Lock-in

Looking across these examples reveals a consistent pattern.

Platform lock-in becomes most visible in systems that must operate continuously while adapting to changing demand. Regulated environments amplify the challenge because infrastructure decisions often remain in place for years while platforms continue to evolve.

Gaming, betting, and streaming services make these limits easier to see. Sudden spikes in activity quickly expose architectural weaknesses, and systems designed with flexible infrastructure tend to scale and recover more predictably.

If you are building platforms where reliability and long-running distributed workloads matter, it may be worth assessing how your architecture handles platform lock-in. To explore these challenges further, get in touch with the Erlang Solutions team.

The post Avoiding Platform Lock-In in Regulated Environments appeared first on Erlang Solutions.

Reliability is a Product Decision

Mon, 13 Apr 2026 05:37:45 +0000

Reliability is often treated as something that can be improved once a system is live. When things break, the focus shifts to monitoring, incident response, and recovery, with the belief that resilience can be strengthened over time as scale reveals weaknesses.

In reality, most of it is set much earlier.

Long before a system faces sustained demand, its underlying design has already shaped how it will respond under pressure. Choices about service boundaries, data handling, deployment models, and fault management influence whether a problem stays contained or spreads.

The conversation is gradually moving from reliability to resilience because distributed systems rarely operate without failure. The more useful question is how a platform continues running when parts of it inevitably fail. The sections that follow explore how early architectural decisions shape that outcome, why their impact becomes more visible at scale, and what it means to build resilience from the beginning rather than react to it later.

Early Decisions Create Long-Term Behaviour

Large-scale failures rarely emerge without warning. What appears sudden at scale is often the predictable outcome of structural decisions made earlier, when different commercial pressures shaped priorities.

In the early stages of a product, the focus is understandably on delivering value quickly, reducing development friction, and validating the market. These are rational business decisions. However, architecture chosen primarily for speed can quietly define the operational ceiling of the system, setting limits that only become visible once demand increases.

Systems Behave as They Were Built to Behave

Outages are often described as “unexpected events,” but distributed systems typically respond to pressure in ways that reflect their design. How services communicate, how state is shared, where dependencies sit, and how failure is managed all influence whether disruption remains contained within a single component or spreads across the wider platform.

Research from Google’s Site Reliability Engineering work shows that around 70% of outages are caused by changes to a live system, such as configuration updates, deployments, or operational changes, rather than by hardware failures. Similarly, the Uptime Institute’s Annual Outage Analysis identifies configuration errors and dependency failures as leading causes of major disruption.

These findings are unsurprising. In distributed environments, dependencies increase and recovery paths become harder to trace, which means that architectural shortcuts that once seemed minor can have disproportionate impact under sustained load. Systems tend to fail along the structural lines already drawn into them, and those lines are shaped by early design decisions, even when those decisions were commercially sensible at the time.

Trade-offs That Compound Over Time

Architectural decisions are rarely made under ideal conditions. Early on, speed to market matters, simplicity reduces friction, and shipping is the priority. A tightly coupled service can help teams move faster, a single-region deployment keeps things straightforward, and limited observability may feel acceptable when traffic is still modest.

But overtime, these trade-offs compound.

Limited isolation between services makes it easier for problems in one area to affect others.
Shared infrastructure can create hidden dependencies that only become visible under heavy demand.
Concentrated regional deployments increase the impact of a local outage or cloud disruption.
Observability that felt sufficient at launch can fall short when trying to understand complex behaviour at scale.

At a smaller scale, these constraints can go largely unnoticed. As usage increases and demand becomes less predictable, they start to shape how the system responds under pressure. What once felt manageable begins to show its limits.

This is rarely about a lack of technical ability. It is simply what happens as complexity builds over time. Every system reflects the trade-offs made in its early stages, whether those choices were deliberate or just practical at the time.

When Architecture Becomes Business Exposure

As systems grow in scale and complexity, the way they are built starts to show up in practical ways. When services are tightly connected, recovery takes longer. When failures are not well contained, a problem in one area can disrupt others. Incidents become harder to resolve and more expensive to manage.

The cost of disruption is not abstract. ITIC’s 2023 Hourly Cost of Downtime Survey reports that more than 90% of mid-size and large enterprises estimate a single hour of downtime costs over $300,000, and roughly 41% place that figure between $1 million and $5 million per hour. At that level, even short-lived incidents carry material financial impact.

For organisations that rely on digital platforms to generate revenue, those numbers represent missed transactions, operational strain, and damage to customer trust. At that point, system design is no longer just an engineering decision. It becomes a business decision with measurable financial consequences.

When Failure Is Public

Some systems fail quietly, disrupting internal workflows or back-office processes with limited external visibility. Others operate in real time, where performance issues are experienced directly by customers, investors, and partners.

In sectors such as entertainment, demand is often synchronised and predictable. Premieres, sporting events, ticket releases, and major launches concentrate traffic into specific windows, placing simultaneous pressure on application layers, databases, and third-party services. These moments are not unusual spikes; they are built into the operating model. Platforms designed for large-scale engagement are expected to handle peak demand as part of normal business activity.

That expectation changes the stakes. When performance degrades in these environments, it is noticed immediately and often publicly. Frustration spreads quickly, confidence can shift in hours, and what might have been an operational issue becomes a visible business problem.

In this context, resilience shapes whether a high-demand event reinforces confidence in the platform or exposes its limits. When failure is experienced directly by users, it moves beyond internal metrics and becomes part of the customer experience itself.

Designing for Resilience

If failure is inevitable in distributed systems, then resilience has to be built in from the start. It cannot be something added later when the first serious incident forces the issue.

Resilient systems are structured so that problems stay contained. A fault in one component should not automatically take others down with it, and services should be able to keep operating even when parts of the system are degraded. External dependencies will fail. Traffic will spike. The design needs to account for that reality.

This way of thinking shifts the focus. Instead of trying to prevent every possible issue, teams concentrate on limiting the impact when something goes wrong. Speed still matters, but so does the ability to grow without introducing instability.

Technology choices can support that approach. Elixir programming language, running on the BEAM, was designed for environments where downtime had real consequences. Its structure reflects that:

Applications are made up of many small, independent processes rather than large, tightly connected components.
Failures are expected and handled locally.
Supervision and recovery are built into the runtime so the wider system keeps running.

No language guarantees reliability, but tools built around fault tolerance make it easier to create systems that continue operating under pressure.

To conclude

By the time serious issues appear at scale, most of the important decisions have already been made.

Failure is part of running distributed systems. What matters is whether problems stay contained and whether the platform keeps operating when something goes wrong.

Thinking about resilience early makes growth easier later. It helps protect revenue, maintain trust, and avoid the instability that forces costly redesigns.If you are building distributed platforms where reliability directly affects performance and reputation, now is the time to treat resilience as a core design decision. Get in touch to discuss how to build it into your architecture from the start.

The post Reliability is a Product Decision appeared first on Erlang Solutions.

The Always-On Economy: Fintech as Critical Infrastructure

Mon, 13 Apr 2026 05:37:45 +0000

We are living in an economy that rarely sleeps. Payments clear late at night. Payroll runs in the background. Businesses expect every digital touchpoint to work when they need it.

Most people do not think about this shift. They assume the systems behind it will hold.

That assumption carries weight.

Fintechs, including small and mid-sized ones, now sit inside the basic infrastructure of how money moves. Their uptime affects cash flow. Their stability affects trust. When something breaks, real businesses feel it immediately.

Expectations changed faster than most systems did.This follows on from our earlier piece, From Prototype to Production, which explored how early technical shortcuts surface as systems scale. Here, we look at what happens next, when those systems become part of the infrastructure businesses rely on every day.

When “tech downtime” becomes infrastructure failure

Outages no longer feel contained. A single failure can affect services that millions of people and businesses depend on.

Large providers experience this as much as small ones. When a shared platform falters, the impact spreads quickly. More than 60 percent of outages now come from third-party providers rather than internal systems, highlighting how tightly connected the ecosystem has become.

The true cost is trust

For fintechs, the impact is immediate because money is involved.

A payment delay blocks cash flow. A failed identity check stops onboarding. A stalled platform damages credibility.

For an SME, this can play out over the course of a single day. Payroll does not process in the morning. Supplier payments stall in the afternoon. Customer support queues fill up while teams wait for systems to recover. Even short interruptions create knock-on effects that last far longer than the outage itself.

And the numbers reflect that risk:

£25,000 is the average cost of downtime for SMEs.
40% of customers consider switching providers after a single outage.
Fintech revenue losses account for approximately US$37 million of downtime-related costs each year.

At this level of dependency, downtime stops being a technical issue. It becomes an infrastructure failure.

Fintechs have become infrastructure whether they intended to or not

Fintech services have moved beyond convenience. They now underpin everyday economic activity for businesses that depend on constant access to money, credit, and financial data.

This shift shows up in uptime expectations. Platforms that handle financial activity are measured against standards once reserved for mission-critical systems. Even brief disruption can have outsized consequences when services are expected to remain available throughout the day..

Customers do not adjust expectations based on company size or stage. If money flows through a service, users expect it to be available around the clock. When it is not, the failure feels systemic rather than technical. What breaks is not just functionality, but confidence.

That is the environment fintech leaders are operating in now.

Where resilience typically breaks down

Most fintech systems do not fail because the idea was weak. They fail because early decisions prioritised speed over durability.

Teams optimise for launch. They prove demand. They ship quickly. That works early on, but systems designed for experimentation often struggle once demand becomes constant rather than occasional.

Shortcuts that felt harmless early on start to surface under pressure. Shared components become bottlenecks. Manual processes turn into operational risk. Integrations that worked at low volume become fragile at scale. Recent analysis shows average API uptime has fallen year over year, adding more than 18 hours of downtime annually for systems dependent on third-party APIs.

Common pressure points include:

Shared components that act as single points of failure
Manual operational work that cannot keep up with growth
Third-party dependencies with limited visibility or control
Architecture built for bursts of usage instead of continuous demand

These are not accidental outcomes. They are the result of trade-offs made under pressure. Funding milestones, launch timelines, and growth targets shape architecture as much as technical skill does.

When outages happen, they rarely trace back to a single bug. They trace back to earlier choices about what mattered and what could wait.

From product thinking to infrastructure grade systems

Product thinking is about features and speed. Infrastructure thinking is about continuity.

Infrastructure-grade systems assume failure will happen. They are built to contain it, recover quickly, and keep the wider platform running. The goal is not perfection. The goal is staying available.

The goal is not perfection. The goal is staying available.

Continuous availability is now expected in financial services. Systems are updated and maintained without noticeable downtime because users do not tolerate interruptions when money is involved.

This approach does not slow teams down. It reduces risk. Deployments feel routine instead of stressful. Engineering effort shifts away from incident response and toward steady improvement.

Over time, this changes how organisations operate. Teams plan differently. Roadmaps become more realistic. Reliability becomes part of delivery rather than a separate concern.

Elixir and the always-on economy

Elixir programming language is designed for systems that are expected to stay available. It runs on the BEAM virtual machine, which was built in environments where downtime carried real consequences.

That background shows up in how Elixir applications are structured. Systems are composed of many small, isolated processes rather than large, tightly coupled components. When something fails, it fails locally. Recovery is expected. The wider system continues to operate.

Elixir in Fintech

This matters in fintech, where failure is inevitable and interruptions are costly. External services misbehave. Load changes without warning. Elixir applications are built to absorb those conditions and recover quickly without cascading outages.

Teams working in Elixir tend to spend less time managing fragile behaviour and more time improving core functionality. Systems evolve instead of being replaced. Reliability becomes part of the foundation rather than a promise teams struggle to maintain.

For fintechs operating in an always-on economy, that approach aligns with the expectations already placed on them.

Reliability as a competitive advantage

Reliable systems can completely change how a fintech operates day to day.

For growing fintechs, uptime supports trust and regulatory confidence. Customers stay because the platform behaves predictably. Growth becomes steadier because teams are not constantly reacting to incidents. Downtime costs make this real, especially for small businesses that lose revenue when systems are unavailable.

For larger providers, reliability reduces operational strain. Fewer incidents mean fewer emergency fixes and fewer difficult conversations with partners and regulators. Teams spend more time improving core services and less time managing fallout.

Reliability also shapes perception. Platforms that stay up become easier to trust with deeper integrations and higher volumes. Over time, that trust compounds and turns stability into a real advantage, even if it is rarely visible from the outside.

To conclude

The always-on economy creates real opportunity for fintechs, but it also raises expectations that many platforms were not originally built to meet.

The question is whether your system has the resilience to operate as infrastructure day after day. If you are a fintech and want to build with reliability in mind, get in touch.

Designing for resilience early makes it far easier to scale without introducing fragility later on.

The post The Always-On Economy: Fintech as Critical Infrastructure appeared first on Erlang Solutions.

Building a performance evaluation toolkit and a dataplane PoC for atproto

Mon, 13 Apr 2026 04:05:45 +0000

There is a new open social ecosystem emerging around atproto. Never heard of it? You should check it out, it's cool tech with an ethos.

The open system is a dream for builders: from day one you can tap into the social graph of more than 40 million Bluesky users. Building on this userbase, you can create everyones new favorite social network on atproto. It's no surprise that there is already a number of alternative communities emerging: Northsky, Eurosky, Blacksky, to name a few.

This is a blog post though, so you already know there is a problem.

The Bluesky dataplane

Bluesky runs almost entirely on open sourced components which enables alternatives to get started quickly.

There is one notable exception: the part of the stack that processes the event stream of the atproto firehose, the AppView, needs one component to digest and store data. This part is called dataplane, and it does the most heavy lifting in processing the event stream. However, the open source dataplane implementation is not very performant. Bluesky uses a closed source dataplane implementation in production for this reason.

The open source dataplane functions on fan-in principle:

events are streamed into the system (e.g. "new post")
the dataplane stores and indexes the events in ordinary Postgres tables
on user request, data is queried and presented to the user (fan-in)

This approach leads to the classic Twitter timeline problem, and in fact limits the user numbers that can safely be served to 100k users, often less.

Fan-out to ETS dataplane

The trick to solve this problem is to change the approach. Instead of querying the data on user request, we prepare data in a dedicated place for each user on write (fan-out). When the user makes a request, the data is already sitting there, just waiting to be served to the user.

This is the also the approach followed by the Bluesky closed source dataplane which builds on ScyllaDB. From all publicly available information, it seems to be pretty optimized to their hardware, and pretty expensive to run.

As you know, we are big fans of Elixir at bitcrowd, so we've built a Proof of Concept of a dataplane with ETS taking on the role of ScyllaDB. The idea here is that we can build something that runs on commodity hardware but scales way better than Bluesky's open source dataplane implementation.

A peek into performance

To evaluate the PoC, we measured reading and writing directly to the dataplane. We compared a Postgres backed version (representing Bluesky's open source implementation) with an ETS backed version.

We simulated write traffic and let a number of users make requests to load their timeline (get_timeline). In the dashboards below, you can see the number of simultaneously active users ("Active Sessions"), the write traffic ("Posts created"), the latencies of get_timeline requests, and the throughput of get_timeline requests.

Under heavy load, the Postgres backed fan-in implementation showed significantly larger latencies and lower throughput in comparison to the ETS backed fan-out implementation.

Dashboard showing data for Postgres backed fan-in implementation

Dashboard showing data for ETS backed fan-in implementation

Advanced simulations

We extended this PoC into a performance toolkit you can use to simulate traffic via atproto events and user requests.

As a basic principle, we want to be able to repeatedly simulate scenarios. Therefore, you provide the configuration for the scenario as file that can be repeatedly loaded into the system.

We provide three components:

Base data

You can quickly create base data, such as users, to prepare your application instance for the simulated scenario. It makes a difference for your app's performance whether you already have millions of users in your database or none.

Traffic simulation

Based on the simulation plan, we perform requests or emit events to create read and write traffic.

Measuring performance

Based on the simulated traffic, we can measure the performance users would experience. For instance, how long would it take for their timeline to load?

A Score for an Invisible Orchestra

Sun, 12 Apr 2026 18:34:09 +0000

Imagine a five-storey building with no lift, erected in the late fifties somewhere on the outskirts of Avtozavodskaya—or better still, in Kupchino. Every floor speaks its own language. Not figuratively but in the most literal sense: the ground floor communicates in Cyrillic, the second in Latin script, the third in ideograms, the fourth in cuneiform, and the fifth, in the manner of Wittgenstein, maintains a principled silence on the grounds that whereof one cannot speak, thereof one must be silent. The postman, delivering the correspondence, is obliged to carry five copies of one and the same letter, translated into each of these tongues, and to knock on the door every time, hoping the addressee has not moved to another floor.

That is precisely how the world of programming is arranged—if one looks at it from the wings rather than the stalls. Every language has its own internal representation of code. Python stores its AST the way a thrifty housewife stores dry goods: in tidy labelled containers—BinOp, FunctionDef, Name. Elixir, with characteristic self-assurance, uses triples {atom, metadata, children} and calls them quoted expressions, as though what we have before us is not a syntax tree but a collection of quotations from William Blake. Ruby stockpiles its S-expressions the way an antiquarian bookseller stockpiles yellowing volumes, and Erlang, faithful to tradition, converses in tuples and atoms intelligible only to Ericsson engineers and, by a curious coincidence, doctoral students at a handful of Swedish universities.

The problem is obvious to anyone who has ever attempted to build a code-analysis tool. Suppose you have written a superb cyclomatic-complexity analyser for Python. It is magnificent: it finds nested conditionals, counts branching points, draws control-flow graphs. Then a colleague comes along and asks, “Could you do one for Ruby?” And it transpires that all your work—all those tree walkers, all that pattern matching over Python’s AST—must be rewritten from scratch. From zero. For a different tree, with different nodes, different semantics, and different booby traps. And then a third colleague will turn up and request the same for Haskell.

Imagine a conductor forced to relearn musical notation every time a new instrument joins the orchestra. Violin—one system of writing. Cello—another. Oboe—a third, with reversed polarity, no less. The trumpet flatly refuses to acknowledge the existence of the staff and insists on a tablature of its own invention. Absurd, of course. In the real world every instrument reads the same score. Notes, rhythm, dynamics are universal. Only the technique of execution differs.

MetaAST is that score.

Before we turn to the details (and they deserve attention in the way a well-constructed detective plot deserves it), permit me a brief digression into theory. Fear not: no formulae, only an analogy. Though one formula will appear after all—but it is so elegant that failing to cite it would be a crime against aesthetics.

In the early two-thousands—when mobile telephones already existed but had not yet taken charge of our lives—the OMG consortium (Object Management Group, bearing no relation whatsoever to the divine or the exclamatory) released a standard called MOF: Meta-Object Facility. Its essence fits into four lines, yet it took the industry two decades to understand those four lines. MOF defines a four-level hierarchy of models:

M⁰ is running code. There it goes, spitting out results, crashing with errors, consuming memory. This is reality.

M¹ is a model of reality. For programs, it is the AST: the abstract syntax tree. Python’s BinOp(op=Add(), left=Name('x'), right=Num(5)) is M¹. Elixir’s {:+, [context: Elixir], [{:x, [], Elixir}, 5]} is also M¹. Every language describes its own code in its own M¹, the way every painter paints an apple in their own way.

M² is the model of models. The meta-model. It defines what a node of any AST can be. Not a concrete node of a concrete language, but the concept of a node. A binary operation is neither Python’s BinOp nor Elixir’s {:+, ...}. A binary operation is the idea that two operands are connected by an operator. UML lives at the M² level. And so does MetaAST.

M³ is the meta-meta-model. That which defines what meta-models themselves can be. The type system, the rules of composition. MOF lives here. In the context of MetaAST, this role is played by Elixir’s type system—@type and @spec.

The fundamental difference between MetaAST and LLVM IR, Java bytecode, or any other intermediate representation lies precisely here: all of those are models (M¹). They describe concrete code in a concrete format. MetaAST is a meta-model (M²). It describes what descriptions of code can be. The difference is roughly the same as between a dictionary and a language: a dictionary catalogues words, whereas a language defines the rules by which those words are possible in the first place.

Metastatic is an Elixir library that implements this idea in code. The name, as befits a respectable technical project, is charged with a double meaning: Met(a)-AST-atic, that is, “pertaining to the meta-level of AST.” The medical connotations are the house’s treat.

The architecture is three-layered, and this is not caprice but a consequence of theory:

M².1 — Core. Concepts present in every programming language on the planet. Literals, variables, binary operations, conditionals, function calls, assignments. Nothing exotic here. x + 5 in Python, Elixir, Ruby, Erlang, and Haskell is one and the same thing. An identical MetaAST representation. Literally:

{:binary_op, [category: :arithmetic, operator: :+],
  [{:variable, [], "x"}, {:literal, [subtype: :integer], 5}]}

Five languages. One tree. One analysis tool. A score legible to any instrument in the orchestra.

M².2 — Extended. Constructs that exist in most languages but not all. Loops, lambdas, collection operations, pattern matching, exception handling. Haskell knows nothing of imperative loops—so be it: it has recursion at the M².1 level. Ruby knows nothing of guards—no matter: the adapter’s metadata will preserve the context.

M².3 — Native. The emergency exit for constructs that resist generalisation. Rust’s lifetimes, Haskell’s type classes, Elixir’s metaprogramming. They are wrapped in {:language_specific, :rust, ...}—like fragile porcelain in bubble wrap—and travel through the system without losing their identity, yet without claiming universality either.

Every MetaAST node is a triple: type, metadata, children (or value). The format is deliberately borrowed from Elixir’s quoted expressions, because if you already know how to write macros in Elixir, you already know how to work with MetaAST. The difference is semantic: where Elixir uses :+ (the operator itself), MetaAST uses :binary_op (the concept of a binary operation) and tucks the operator into metadata. Where Elixir inlines literals, MetaAST wraps them in {:literal, [subtype: :integer], 42}, ensuring structural uniformity.

Language adapters form the bridge between M¹ and M². The Python adapter takes Python’s AST (obtained via subprocess) and abstracts it up to MetaAST. The Elixir adapter takes quoted expressions and does the same. The Ruby adapter—likewise. The inverse operation—reification—turns MetaAST back into the native AST of the target language. This pair of operations, abstraction and reification, constitutes what mathematicians call a Galois connection:

Adapter_L = (alpha_L, rho_L)

alpha_L: AS_L -> MetaAST x Metadata    (abstraction: M¹ -> M²)
rho_L:   MetaAST x Metadata -> AS_L    (reification: M² -> M¹)

What does this mean in practice? Here is a scenario. You have written a function-purity analyser. It operates on MetaAST: traverses the tree, looks for side effects (I/O, state mutation, calls to random-number generators), and renders its verdict. This analyser was written once. Once. And it works with Python, Elixir, Ruby, Erlang, and Haskell. Because it analyses not the Python or Elixir AST but the meta-level. You write:

{:ok, doc} = Metastatic.Adapter.abstract(Python, "print('hello')", :python)
{:ok, result} = Metastatic.Analysis.Purity.analyze(doc)
result.pure?    # => false — side effect: I/O
result.effects  # => [:io]

And precisely the same code, unchanged, works if you replace Python with Ruby and print('hello') with puts 'hello'. Because both calls are {:function_call, [name: "print"], [{:literal, [subtype: :string], "hello"}]} at the M² level.

Let us return to MOF and the four-level hierarchy. Why is any of this necessary? Why erect meta-models when one could simply write a converter from one AST to another?

The answer is simple and brutal, like the truth of life. A converter works pairwise. For five languages you need twenty converters (5 × 4). For ten—ninety. For twenty—three hundred and eighty. A meta-model works through a hub: each language connects once, via its own adapter. Five languages—five adapters. Ten—ten. Twenty—twenty. Linear growth instead of quadratic. A mathematician who had spent half his life as a reporter would appreciate the irony.

But it is not only a matter of combinatorics. A meta-model provides standardisation. Every tool written for MetaAST is guaranteed to work with any language for which an adapter exists. This is not “Python support” and “Ruby support” as separate features. It is one feature: MetaAST support. Everything else follows as a consequence.

OMG understood this in 2002 when it released MOF. The entire UML industry is built on the same principle: the meta-model defines what models can be, and concrete diagrams are merely instances of that meta-model. MDA (Model-Driven Architecture) took the idea further: transformations between models are defined at the meta-level and applied automatically to any instances.

Metastatic does the same, but not for class diagrams—for the syntax trees of programs. It is not an IR, not a compiler, not a transpiler. It is the foundation upon which all of the above can be built—once, and for every language at once.

There is an old story about a farmer, I’ve heard from a stranger on a train. A man spends his entire life breeding different eggplant varieties. In decades, when the farmer is already old and blind, his fields have produced an unprecedented harvest. He calls for each and every agronomist in the world, they come, and the most famous one says, “These aubergines are adorable!” The farmer dies of frustration. The man does not know what aubergines are but he’s certain the great eggplant nobody would have called ‘aubergine.’ He considers his life wasted.

The ASTs of different languages are the eggplants. MetaAST is the knowledge that they all grow on the same field. The difference between them is terminological, not semantic. Python’s BinOp(op=Add()), Elixir’s {:+, [], [...]}, and Ruby’s s(:send, ..., :+, ...) are different names for one and the same thing: {:binary_op, [category: :arithmetic, operator: :+], [left, right]}.

One could, of course, spend an entire life rewriting tools for every new language. One could relearn musical notation each time, bow before every new instrument in the orchestra, translate letters into five tongues for five floors of one and the same building. But why, when one can ascend a single level of abstraction—to the place where eggplants and aubergines are indistinguishable, and the score is one for all?

Dropping Cloudflare for bunny.net | jola.dev

Tue, 07 Apr 2026 18:13:31 +0000

TL;DR my motivation and experience for moving my blog from Cloudflare to bunny.net

I’ve been a long time Cloudflare user. They offer a solid service that is free for the vast majority of their users, that’s very generous. Their infrastructure is massive and their feature set is undeniably incredible.

One of my biggest concerns though is around how easily I could become heavily dependent on this one single company that then can decide to cut me off and disable all of my websites, for any arbitrary reason. It’s a single point of failure for the internet. Every Cloudflare outage ends up in the news. And I can’t help but feel that the idea of centralizing the internet into a single US corporation feels off. Not to mention the various scandals that have surrounded them. So I was open to alternatives.

Bunny.net

Bunny.net (affiliate link because why not, raw link here) is a Slovenian (EU) company that is building up a lot of momentum. Their CDN-related services rival Cloudflare already, and although their PoP network is smaller than Cloudflare’s, they score highly on performance and speed across the globe. It’s a genuinely competitive alternative to Cloudflare.

It has the additional benefit of being a European company, and I like the idea of growing and supporting the European tech scene.

What I was moving away from

I’ve been using various different services, but focusing on this blog, the first thing was Cloudflare as the registrar for the domain name. I did some research on alternative registrars, but I just didn’t find any good European options. The closest I found was INWX, but their lack of free WHOIS Privacy made them a non-option. I ended up with Porkbun. They run on Cloudflare infrastructure, but they have better support. So the remaining thing Cloudflare was doing for me was the “Orange Cloud”: automatic caching, origin hiding, and optional protection features.

So that’s what we’re moving over! I’m gonna walk you through how to set up the bunny.net CDN for your website, with some sensible defaults.

Step by step

Setting up your bunny.net account is quick and you get $20 worth of free credits to play around with, those are valid for 14 days. You don’t need to give them a credit card up front to try things out, but if you do, you get another $30 worth of credits. You do need to confirm your email though before you can start setting things up. Once you’re out of the trial, you pay per use, which for most cases is cents a month. However, note that bunny.net require a minimum payment of $1 per month.

I guess a cheap price to pay to stop being the product and start becoming the customer.

Creating your pull zone

The pull zone is the main mechanism for enabling the CDN for your website. You’ll find them under CDN in the left navigation bar. Here’s how to set one up:

Fill in the pull zone name. Just make it something meaningful to you, for example the website name.
For origin type, select Origin URL.
Fill in your Origin URL. This would be the address for directly accessing your server. In my case, it’s the public IP of my server.
If you’re running multiple apps on your server, for example using Dokploy, coolify, or self-hosted PaaSs like that, you’ll want to pass the Host header as well. Here you put in the domain of your app. In my case, that’s jola.dev.
For tier, select Standard.
Finally you can select your pricing zones. Note that some zones are more expensive, so you can choose to disable them. This just means that people in those areas will get redirected to the closest zone you do have enabled.

And you’re done with the first part!

Configuring your pull zone

Now that you’ve set up the pull zone, it’s time to hook it up to your website and domain. Go to the pull zone you created. You’ll see a “hostnames” screen. Time to connect things.

Under “Add a custom hostname” fill in your website domain name.
You’ll get a modal with some instructions. You need to follow them to set up the DNS name to point your website to go through the CDN.
Go to where you manage domain name and add a CNAME record to point your domain to the given CNAME value in the modal, something like website.b-cdn.net.
Once you’ve done that, wait a few minutes to let it propagate, and then click “Verify & Activate SSL”.
If it says success, you’re done. Your website is now running through the bunny.net CDN, similar to the Cloudflare orange cloud.

Configuring caching

This is the part where bunny.net will really shine through!

If your website is set up to return the appropriate cache headers for each resource, things will just work. Bunny defaults to respecting the cache control headers when pointing a pull zone at an origin site. To verify, go to Caching → General and check that “Respect origin Cache-Control” is set under “Cache expiration time”. Note that if you set no-cache, bunny will use that and will not cache at the edge.

Alternatively, if you don’t have cache headers set up, and you don’t want to control that yourself, you can instead enable Smart Cache. This will default to caching typically cached resources like images, CSS, JS files etc, while avoiding caching things like HTML pages. This will work for most cases!

But I wanted to go faster. If you’ve read my post about building this website, here’s how I’ve set up my cache headers: I added a new pipeline in the router called public and added an extra middleware to it. I technically have everything using this pipeline, but leaving the standard browser pipeline that comes out of the box with Phoenix keeps my options open to add authenticated (uncached) pages in the future.

pipeline :public do
    plug :accepts, ["html"]
    plug :put_root_layout, html: {JolaDevWeb.Layouts, :root}
    plug :put_secure_browser_headers, @secure_headers
    plug :put_cdn_cache_header
  end
  
  defp put_cdn_cache_header(conn, _opts) do
    put_resp_header(conn, "cache-control", "public, s-maxage=86400, max-age=0")
  end

You can see the whole router here https://github.com/joladev/jola.dev/blob/main/lib/jola_dev_web/router.ex.

This setup means I even cache the HTML pages, which makes this ridiculously fast. Here’s the landing page response time from various locations, using the Larm response time checker tool:

Because I’m caching the HTML pages, if I publish a new post I do need to purge the pull zone to reset the cached HTML files.

Setting some sensible defaults

All of these are optional, but nice to have!

On your pull zone page, under General → Hostnames, go toggle “Force SSL” on for your domain to ensure that all requests use SSL. SSL/TLS is pretty standard these days, and many TLDs and websites use HSTS to enforce it, but no harm in enabling it here too.

DDoS protection comes out of the box, but we can set some other things up. First of all, go to Caching and then Origin Shield in the left menu on your pull zone, and activate Origin Shield. Select the location closest to your origin. This reduces load on your server, as bunny.net will cache everything in the Origin Shield location, and all edge locations will try that location first before hitting your server.

Next, go to Caching → General and scroll down. At the bottom of the page you can select Stale Cache: While Origin Offline and While Updating. This means bunny will keep serving cached content even if it is stale, if it can’t reach your origin, and that it will serve stale content while fetching the latest version. Both are nice to haves, nothing you have to enable, but provide a slightly better service to your users!

Next, let’s set up an Edge rule to redirect any requests to our automatically generated pull zone domain to our actual domain, to avoid confusing crawlers. On your pull zone, in the left menu, click Edge rules.

Add edge rule.
Name it “Default domain redirect”.
Under actions, select Redirect.
For URL, input your URL plus the path variable. Eg for me it’s https://jola.dev{{path}} .
Status code: use the default 301.
For conditions, pick Match any and Request URL Match any.
Input *://<slug>.b-cdn.net/* replacing <slug> with the name given to your pull zone.
Save edge rule!

Now you should be able to go to https://slug.b-cdn.net for your pull zone and get redirected to your proper domain!

Conclusion

This post just covers the very basics of getting set up on bunny.net. I haven’t even scratched the surface of edge rules, cache configuration, the Shield features for security and firewalls, video hosting and streaming, edge scripting and edge distributed containers, and much more.

I especially appreciate the great statistics, logs, and metrics you get out of the dashboard. You can even see every single request coming through to help you investigate issues, and clear feedback on what’s getting cached and not. I’m actively moving everything else over and I’m excited for the upcoming S3 compatible storage!

You should give bunny.net a try!

Newres Al Haider

Mon, 06 Apr 2026 18:15:11 +0000

Growing Yggdrasil, the World Tree, with Ash

2026-04-05

Declarative programming can be a powerful paradigm for organizing software systems. By defining the business processes once, we ensure there is a single source of domain knowledge. From this foundation, we can derive other parts of the system such as API endpoints, database schemas, and even user interfaces. This approach reduces repetition and helps prevent bugs caused by misaligned domain models.

The Ash Framework seems like an excellent way to see declarative programming in action. Written in Elixir, it allows you to describe your domain in a consistent and expressive way, from which it can automatically generate data layers, REST or GraphQL APIs, and admin interfaces. With Ash, you define what your application should do, and the framework takes care of how to make it happen. It can derive JSON REST endpoints, handle validation, manage persistence, and provide authorization logic, all from the same declarative definitions.

I am new to Ash and I tend to learn best by writing things out, so this article is as much for me as it is for you. Rather than trying to understand everything up front, I prefer to get hands-on quickly with a small, self-contained project. We’ll start with the basics here, and if things go well, expand on it in a follow-up or two.

Given the name Ash, it felt appropriate to build something inspired by the gigantic ash tree of Norse mythology: Yggdrasil, the World Tree.

An image of a Yggdrasil the world tree as a cybernetic Ash tree.

Yggdrasil is said to connect the Nine Worlds of Norse cosmology, though the exact number and nature of these worlds vary between sources. Each world has its own nature, inhabitants, and relationships with the others, making it an ideal metaphor for exploring how Ash models resources, attributes, and relationships. In this project, we will create a domain model for these concepts with Ash and derive a JSON REST API from them.

The first step is getting started with a basic Ash project for which we will use the Igniter tool. (I will assume Elixir is already installed, but if not, see the Elixir Install page for instructions). This is used for project setup and code generation, which will help us get started a lot quicker.

To start off following command will install Igniter:

mix archive.install hex igniter_new

Once we have Igniter, the next step is creating a new project in the yggdrasil directory, adding Ash, and moving into it.

mix igniter.new yggdrasil --install ash && cd yggdrasil

This will land us in a new Elixir project directory with Ash installed. From this seed we will evolve our application to represent the worlds and characters of Norse mythology.

The newly created project comes with a hello function in the lib/yggdrasil.ex module. Let's try it out in the iex, the Elixir interactive shell which we can start with:

iex -S mix

Running it will bring us into the shell, where we can run the hello function in the yggdrasil module:

iex(1)> Yggdrasil.hello()
:world

We now have a hello world, but in yggdrasil we want to represent the worlds of Norse mythology that are linked by Yggdrasil. The first thing we need for this is a Domain. This will function as a container for the various concepts, such as the worlds, that we will introduce later.

For simplicity's sake, first we will replace the contents of lib/Yggdrasil.ex with the following:

defmodule Yggdrasil do
  @moduledoc """
  The Yggdrasil domain — acts as the trunk of the tree
  and organizes all resources like World and Character.
  """

  use Ash.Domain

  resources do
    # Resources will be registered here
    resource Yggdrasil.World
  end
end

We create a file lib/resources/world.ex with the following contents:

defmodule Yggdrasil.World do
  @moduledoc """
  A resource representing a world in Yggdrasil.
  """

  use Ash.Resource,
    # in-memory store
    data_layer: Ash.DataLayer.Ets,
    domain: Yggdrasil

  actions do
    create :create do
      accept [:name, :description]
    end

    update :update do
      accept [:description]
    end

    # Provide default actions
    defaults [:read, :destroy]
  end

  attributes do
    # Primary key
    uuid_primary_key :id

    # World name and description
    attribute :name, :string, allow_nil?: false, public?: true
    attribute :description, :string, public?: true
  end
end

And finally

config :yggdrasil, :ash_domains, [Yggdrasil]

With these files in place, we now have a minimal Ash domain containing a single resource: World. Let’s take a moment to unpack what we just created before moving on.

At the top level, Yggdrasil acts as our domain, the trunk of our system. It brings together all the resources that make up the application and defines how they relate to each other. Right now, our domain only includes one resource, Yggdrasil.World, but we’ll add more later.

The Yggdrasil.World module itself is declared as a resource. In Ash, a resource is the fundamental building block. It describes a specific type of data and what can be done with it. Instead of writing separate schemas, changesets, and controllers, we declare everything about a resource in one place, and Ash takes care of the details.

Our World resource uses the Ash.DataLayer.Ets data layer, which stores data in Elixir’s in-memory ETS tables. This setup is fast and simple, making it perfect for early experimentation, though data won’t persist between runs. Later, this can be swapped out for a different data layer to gain full persistence. The argument domain: Yggdrasil connects the resource back to the domain we just defined so that the framework knows where it belongs.

Inside the actions block, we declare what operations are available for this resource. The create action accepts a name and a description, while the defaults [:read, :destroy] line automatically adds the standard read, update, and delete actions. There’s no need to write any manual CRUD logic—Ash generates it for us.

The attributes block defines the structure of each world. Every world has a UUID primary key (:id) and two fields, :name and :description. The :name attribute is made required using (allow_nil?: false), ensuring that each world must have one. Both attributes are marked public?: true so they appear in APIs and outputs.

Finally, the configuration line we added tells Ash which domains to load when the application starts. Without this, the framework wouldn’t know about our new resource.

At this point, our small Ash tree has already taken root. We’ve declared the first piece of our domain, and Ash now knows how to create, read, update, and delete worlds. Let’s see that in action next by exploring our resource interactively in iex.

First, start the interactive shell from your project root:

iex -S mix

Once inside iex, we want to create our first world Asgard, the shining realm of the gods, with the following command:

asgard = (
  Yggdrasil.World
  |> Ash.Changeset.for_create(:create, %{
       name: "Asgard",
       description: "A shining realm of order and power, suspended high above the clouds."
     })
  |> Ash.create!()
)

which would return the world as such:

14:29:14.665 [debug] Creating Yggdrasil.World:

Setting %{id: "726b678e-6cb6-4277-b291-85ecfa313d3a", name: "Asgard",
 description: "A shining realm of order and power,...}

%Yggdrasil.World{
  id: "726b678e-6cb6-4277-b291-85ecfa313d3a",
  name: "Asgard",
  description: "A shining realm of order and power, suspended high above the clouds.",
  __meta__: #Ecto.Schema.Metadata<:loaded>
}

There are a multiple things happening here, so let's unwrap things step by step.

First we start off with our Ash resource that we have defined Yggdrasil.World. In Ash, resources describe the structure of our data, including attributes like name and description, as well as the actions that can be performed on them.

Next we are using the pipe operator. |>, to pass this result to our next function. Elixir’s pipe operator takes the result of the expression on the left and passes it as the first argument to the function on the right.

For example instead of writing:

function(value, a, b)

with the pipe operator we can equivalently write:

value |> function(a, b)

This allows us to write a sequence of operations in a readable, step-by-step style. In our code example, it means Yggdrasil.World is passed into the Ash.Changeset.for_create function, as the first parameter. This function also takes the identifier of our create action :create, as well as the structure representing Asgard, with its name and description.

What this function returns is a changeset, a data structure representing the intended change of a resource in Ash (e.g.: creating, updating, etc). This is especially useful when it comes to validation and error checking, as we will see it later down the line. For now we use this changeset and pipe it into the function that executes the actual creation: Ash.create!().

The resulting value is a %Yggdrasil.World{} struct, which represents the newly created world. Ash also automatically generated a UUID for the id field, which uniquely identifies this world inside the system.

Before returning the struct, Ash logs the operation it performed. That is why we see the debug output:

[debug] Creating Yggdrasil.World

The final line is the Elixir struct that was created:

%Yggdrasil.World{
  id: "726b678e-6cb6-4277-b291-85ecfa313d3a",
  name: "Asgard",
  description: "A shining realm of order and power, suspended high above the clouds.",
  __meta__: #Ecto.Schema.Metadata<:loaded>
}

This struct is also stored in the variable asgard, so we can reference it later in the session.

Now that we understand how creating a world works, let’s add another one.

midgard = (
  Yggdrasil.World
  |> Ash.Changeset.for_create(:create, %{
       name: "Midgard",
       description: "The realm of humans, bound to the earth and everyday struggles."
     })
  |> Ash.create!()
)

This follows the exact same pattern as before. We build a changeset that describes the creation of Midgard, and then execute it with Ash.create!(). Much simpler than the mythological creation of Midgard, which involved the slaying of the giant Ymir.

Now that we have some worlds, let's read them using the read action:

worlds = (
  Yggdrasil.World
  |> Ash.Query.for_read(:read)
  |> Ash.read!()
)

which would give us our list of worlds:

[
  %Yggdrasil.World{
    id: "some-uuid-1",
    name: "Asgard",
    description: "A shining realm of order and power, suspended high above the clouds."
  },
  %Yggdrasil.World{
    id: "some-uuid-2",
    name: "Midgard",
    description: "The realm of humans, bound to the earth and everyday struggles."
  }
]

As one can expect, we can also do an update call. For example, let’s change the description of Asgard:

asgard = (
  asgard
  |> Ash.Changeset.for_update(:update, %{
       description: "The fortified realm of the Aesir, ruled by Odin."
     })
  |> Ash.update!()
)

There are a few things to note here. Instead of starting from the Yggdrasil.World module, we now start from the existing asgard struct. This is because we are modifying a resource that already exists.

The function for_update creates a changeset that describes the intended update. Just like with creation, the changeset itself does not perform the update, it only represents the change we want to make.

We then pass this changeset into Ash.update!(), which executes the update. Ash applies the changes, runs any validations, and returns the updated %Yggdrasil.World{} struct.

We can verify the change by reading the list of worlds again:

worlds = (
  Yggdrasil.World
  |> Ash.Query.for_read(:read)
  |> Ash.read!()
)

which would give us a result such as:

[
  %Yggdrasil.World{
    id: "6b62b3ea-b08b-4387-8539-37e645e53026",
    name: "Midgard",
    description: "The realm of humans, bound to the earth and everyday struggles.",
    __meta__: #Ecto.Schema.Metadata<:loaded>
  },
  %Yggdrasil.World{
    id: "d2646509-6c92-4049-a2db-0555612fc365",
    name: "Asgard",
    description: "The fortified realm of the Aesir, ruled by Odin.",
    __meta__: #Ecto.Schema.Metadata<:loaded>
  }
]

An interesting thing we could try out is updating the name of a world instead:

asgard2 = (
  asgard
  |> Ash.Changeset.for_update(:update, %{
       name: "Asgard2"
     })
  |> Ash.update!()
)

We get the following error:

** (Ash.Error.Invalid)
Invalid Error

* No such input `name` for action Yggdrasil.World.update

The attribute exists on Yggdrasil.World, but is not accepted by Yggdrasil.World.update

Perhaps you meant to add it to the accept list for Yggdrasil.World.update?


Valid Inputs:

* description

This is because when we were defining our update action in our module, the only attribute we accept is :description, see fragment below:

update :update do
      accept [:description]
    end

In other words, while the name attribute exists on the resource, it is not allowed to be modified through the update action. This is a domain modelling decision, and gives us fine-grained control over how our data can change. In this case, we decided that a world’s name is fixed after creation, while its description can evolve over time.

Finally we get to do delete, where we destroy asgard, our Ragnarok action if you will. We can do this by the following:

Ash.destroy!(asgard)

Ash.destroy! takes a resource struct, in this case asgard, and removes it from the data store. Since we’re using an in-memory ETS store, it deletes it from memory immediately. The function should return :ok on success. We can double check this by requesting our list of worlds again by our usual means:

worlds = (
  Yggdrasil.World
  |> Ash.Query.for_read(:read)
  |> Ash.read!()
)

which returns only Midgard:

[
  %Yggdrasil.World{
    id: "6b62b3ea-b08b-4387-8539-37e645e53026",
    name: "Midgard",
    description: "The realm of humans, bound to the earth and everyday struggles.",
    __meta__: #Ecto.Schema.Metadata<:loaded>
  }
]

At this point, we’ve taken the first steps in modeling our little piece of Yggdrasil. We have a domain, a resource, and a way to create, read, update, and delete worlds, enough to bring about a small Ragnarok.

Next, we will explore how we can start connecting resources together. After all, the worlds need their heroes and villains to really come alive.

How Many Paradigms Does It Take to Screw In a Lightbulb?

Mon, 06 Apr 2026 12:52:41 +0000

A developer who knows only one programming paradigm resembles a carpenter whose entire toolbox contains a single hammer. Naturally, a hammer will drive a nail with admirable precision. Or a screw, if sufficient enthusiasm is applied. But try to saw or plane a board with that hammer, and it becomes immediately clear—assuming you’ve encountered a saw or a plane at least once in your life—that the instrument has been chosen poorly. So it is with paradigms: knowledge of nothing but imperative programming, or nothing but object-oriented design, transforms a developer into a mechanical executor of tasks, incapable of seeing an elegant solution even when it lies on the surface, waiting to be noticed.

The narrowness of a programmer trapped in a single paradigm manifests in everything. They will erect loops where a single higher-order function would suffice. They will breed classes and inheritance where a pure function and composition would have been more than enough. They will attempt to verify the correctness of an algorithm with a debugger and tests instead of proving it formally at the type level. Such a developer resembles a tourist who knows exactly one word of the foreign language and is attempting, with its help, to explain a route across the entire city to a taxi driver. And it’s a small mercy if the word isn’t obscene.

Let us, for a start, walk through the principal paradigms and see what instruments each offers for solving problems. We’ll begin with the most ancient and familiar—the imperative paradigm.

Imperative programming is the world of instructions and mutable state. The programmer tells the machine: do this, then that, change this variable, repeat five times. A classical example in C:

intsum=0;for(inti=0;i<10;i++){sum+=i;}

Here we explicitly manage the state of the variable sum, accumulating the result step by step. This is natural for the machine, but tedious for the human. Every step must be spelled out, every mutation tracked. The imperative style serves well when the task reduces to a sequence of actions with side effects: write to a file, update a database, print to the screen. But as soon as the task grows in complexity, the code devolves into a tangle of interrelated variables and conditions.

Procedural programming is the imperative approach enriched with structures and functions. We group instructions into procedures to avoid repetition and improve readability. The same example:

intcalculate_sum(intn){intsum=0;for(inti=0;i<n;i++){sum+=i;}returnsum;}

Now the logic is packaged into a function that can be reused. The procedural style dominated the era of Pascal and early C. It taught programmers to think in modules and structure their code, but it never freed them from the problems of mutable state and side effects.

Object-oriented programming (in Gosling’s understanding, not Kay’s) promised to solve all problems at once: encapsulation, inheritance, polymorphism—the three pillars upon which the entire world supposedly rests. Data and methods unite into objects, objects assemble into class hierarchies. It sounds splendid, until you begin to examine how the code actually works:

classCounter{privateintvalue=0;publicvoidincrement(){value++;}publicintgetValue(){returnvalue;}}

State lives inside the object, convenient methods form the API, full encapsulation achieved. So it would seem, but the state hasn’t gone anywhere—it has merely relocated into a class field. And along with it relocated all the old afflictions: data races in multithreading, the difficulty of testing, the unpredictability of behavior. The object-oriented approach serves well for modeling a domain when you need to describe entities and their interactions. But it transforms into a nightmare when class hierarchies sprawl to dozens of inheritance levels, and half the methods exist solely to pass a call further down the chain.

Functional programming looks at the task from an entirely different angle. Here there is no mutable state, no loops, no side effects. There are only functions that receive data and return results. The same summation example in Haskell:

sum=foldl(+)0[0..9]

One line instead of five. No loops, no intermediate variables. The function foldl takes (1) an addition operation, (2) an initial value, and (3) a list, returning the result. The code reads like a mathematical expression, not a sequence of commands. The functional style is particularly well suited for working with collections, for building data-processing pipelines, for parallel computation. When there is no mutable state, there is no need for locks and synchronization. Functions can be safely launched simultaneously on different processor cores. Though for the domain of Accounting for a liquor store in the suburbs—it’s a rather dubious ally.

Logic programming overturns one’s very notion of how to write code. Instead of explaining how to solve a task, the programmer describes what they want to obtain. The system finds the solution on its own. Prolog is the classical representative of this paradigm:

parent(tom,bob).parent(tom,liz).parent(bob,ann).grandparent(X,Z):-parent(X,Y),parent(Y,Z).

We described kinship relations and a rule for determining grandparents. Now we can pose the question: grandparent(tom, ann)?—and the system will answer “yes,” having found the path through the facts. Logic programming is indispensable in certain corners of artificial intelligence, expert systems, and task planning. I even dragged it into the consistency validation of finite automata in one of my libraries. But an attempt to write a web server in Prolog would look rather like an attempt to hammer a mole with a microscope.

Declarative programming is a general term for approaches where the programmer describes the desired result rather than the sequence of steps. SQL is the textbook example:

SELECTnameFROMusersWHEREage>18ORDERBYname;

We don’t explain how to traverse the table, how to check the condition, how to sort the result. We simply declare: I want the names of users over eighteen, sorted alphabetically. The database will figure out how to do this efficiently on its own. The declarative style dominates in HTML, CSS (for now—I suspect someone will drag recursion into it before long), and configuration files. It allows one to separate the what from the how.

Concatenative programming is built on the idea of function composition via a stack. Forth is its most vivid representative:

: square dup * ;
5 square .

The function square duplicates the top element of the stack and multiplies it by itself. The number 5 is placed on the stack, the function square is applied, the result is printed. The code reads right to left, like reverse Polish notation. Concatenative languages are compact and efficient, but they demand a particular cast of mind. They remain popular in embedded systems and wherever code size and execution speed are critical.

Reactive programming focuses on data streams and the propagation of changes. When a data source changes, all dependent computations update automatically. An example in RxJS:

constclicks=fromEvent(document,'click');constpositions=clicks.pipe(map(event=>event.clientX));positions.subscribe(x=>console.log(x));

We create a stream of click events, transform it into a stream of coordinates, and subscribe to changes. Each click automatically produces the coordinate in the output. The reactive style is ideal for interfaces, event handling, and working with asynchronous data sources. It liberates you from callback hell and makes the data flow explicit.

Aspect-oriented programming addresses the problem of cross-cutting concerns—logging, caching, access control. Instead of smearing these aspects across the entire codebase, they can be described separately:

@Transactional@LoggedpublicvoidupdateUser(Useruser){repository.save(user);}

The annotations @Transactional and @Logged are aspects. They will be automatically “applied” to the method, wrapping it in a transaction and adding logging. The core code remains clean and comprehensible. The aspect-oriented approach is popular in enterprise development, where cross-cutting concerns permeate the entire system.

Metaprogramming is the programming of programs that write programs. Macros in LISP allow code to be generated at compile time:

(defmacrowhen(condition&restbody)`(if,condition(progn,@body)))

The macro when expands into an if construct with a progn block. Metaprogramming grants extraordinary flexibility, enabling the creation of domain-specific languages right inside the host language. But with great power comes great responsibility: poorly written macros turn code into an unreadable mess. If you want to see what metaprogramming looks like when practiced by a sane person—take any of my libraries, or write your own in Elixir. I know of no other language where macros have been done properly.

Dependently-typed programming elevates the type system to a new plane. Types can depend on values, allowing complex invariants to be expressed at the type level.

dataVec(A:Set):Nat->Setwhere[]:VecAzero_::_:{n:Nat}->A->VecAn->VecA(sucn)append:{A:Set}{mn:Nat}->VecAm->VecAn->VecA(m+n)

The type Vec A n is a vector of elements of type A with length n. The function append takes two vectors of lengths m and n and returns a vector of length m + n. The compiler verifies correctness at the type level. It is impossible to write a function that violates the length invariant. Dependent types are used for the formal verification of critical systems, where an error costs far too much.

Theorem-proving as a paradigm is the proof of program correctness by mathematical methods. Lean and Coq allow one to write not merely code, but proofs that the code does precisely what was intended:

theoremadd_comm(nm:Nat):n+m=m+n:=byinductionnwith|zero=>simp[Nat.zero_add,Nat.add_zero]|succnih=>simp[Nat.succ_add,Nat.add_succ,ih]

This is not simply an addition function—it is a proof that addition is commutative. The compiler doesn’t merely check types; it checks the mathematical proof. This approach is employed in cryptography, compilers, and operating systems—domains where the price of an error is measured not in irritated users, but in human lives or millions of dollars in losses.

The actor model views a program as a collection of independent actors that exchange messages. Each actor has its own mailbox, processes messages sequentially, and can create new actors. Erlang was built upon this idea:

-module(counter).-export([start/0,loop/1]).start()->spawn(fun()->loop(0)end).loop(N)->receive{increment,Pid}->Pid!{value,N+1},loop(N+1);{get,Pid}->Pid!{value,N},loop(N)end.

The actor counter receives increment and get messages, modifies its state, and replies. No shared data, no locks. Actors scale horizontally, failures are isolated. This model is ideal for distributed systems, where failures are the norm rather than the exception.

Dataflow programming describes computation as a graph of data streams. The nodes of the graph are operations, the edges are data flows between them. A change in one node propagates automatically through the graph. LabVIEW uses visual dataflow programming for hardware control. The approach is intuitive for engineers accustomed to thinking in schematics and diagrams.

Constraint programming describes a task as a set of constraints that must be satisfied. The system searches for a solution by enumerating possibilities and pruning the impossible. MiniZinc is a language for constraint programming:

var 1..9: x;
var 1..9: y;
constraint x + y = 10;
constraint x * y = 21;

Two variables, two constraints. The system will find x = 3, y = 7 or x = 7, y = 3. Constraint programming is applied in planning, scheduling, and resource optimization—wherever a task is formulated as finding a solution under constraints.

Phew.

Now let us pose the question: why does any of this matter to an ordinary developer? The answer is simple and simultaneously non-obvious. Each paradigm is a way of thinking, an approach to solving problems. A programmer who knows only imperative programming will solve every task with loops and conditionals. They will see a list-processing task and write a for loop with intermediate variables. A programmer acquainted with the functional paradigm will write map or fold—elegantly, concisely, free of side effects. One who has mastered reactive programming will construct an event-processing pipeline where each stage is explicitly described and easily testable.

Knowledge of different paradigms expands one’s arsenal of tools. You won’t write a web server in Prolog or prove theorems in JavaScript. But an understanding of logic programming will help you formulate conditions more precisely and build database queries. Familiarity with dependent types will teach you to think in invariants and express constraints at the type-system level. Experience with actors will show you how to build scalable distributed systems without the headaches of synchronization.

In truth, in the modern world all mature languages have long since become multi-paradigm. Scala combines object-oriented and functional approaches. Rust adds a powerful ownership and borrowing system to the imperative style. Python allows one to write procedurally, in an object-oriented fashion, and functionally. F# unites functional programming with the .NET ecosystem. Swift attempts to incorporate elements of all major paradigms at once. A programmer who understands when an aspect is needed (yes, in any language—for instance, I dragged aspects intoElixir) uses the language to its full power. One who knows only a single paradigm writes in any syntax as though it were PHP.

Paradigms are not a religion where you must choose one true faith and wage war on the heretics. They are tools, and a good craftsman knows when to reach for the hammer, when for the saw, and when for the plane. Need to parse something? Take the functional approach with map and fold. Build a system with thousands of simultaneous connections? Actors are your choice. Formally prove an algorithm’s correctness? Welcome to Lean or Agda. Developing an interface with many interactive elements? Reactive programming will make the code comprehensible.

A programmer trapped in a single paradigm is condemned to solve problems inefficiently. They will drag familiar patterns behind them even when those patterns don’t fit. They will write a class where a function would suffice. They will create mutable state where it could be avoided entirely. They will erect a complex hierarchy where composition would have been enough. They resemble a person who knows only one route from home to work and stubbornly waits at the bus stop every day, even though the road has been torn up for a month and the bus now runs on the next street over.

If a developer claims the badge of mid-level-plus but doesn’t feel at ease in at least the five principal paradigms—they are a pompous fool, and you should show them the door.

Rails on the BEAM

Sat, 04 Apr 2026 02:36:04 +0000

Rails on the BEAM

2026-04-02T23:21:00Z

Same blog from the Twilight Zone post. Same models, controllers, views. Same Turbo Streams broadcasting. Same Action Cable protocol. But check what's serving it:

npx github:ruby2js/juntos --demo blog
cd blog
npx juntos db:prepare
npx juntos up -d sqlite_napi

Open http://localhost:3000. Open a second tab. Create an article. Watch it appear in both. That part you've seen before.

Now imagine a bug in a request handler crashes one of the JavaScript runtimes. On Node.js, that takes down the process — connections drop, state is lost, restart from scratch. On the BEAM, the OTP supervisor restarts just that runtime. The other runtimes keep serving. WebSocket connections stay open. Turbo picks up where it left off.

What Changed

Nothing in the application. The Ruby source is identical. What changed is the target:

Target	Command	Database	Broadcasting
Browser	`juntos dev -d dexie`	IndexedDB	BroadcastChannel
Node.js	`juntos up -d sqlite`	SQLite	WebSocket server
BEAM	`juntos up -d sqlite_napi`	SQLite or PostgreSQL	OTP :pg

The browser uses the BroadcastChannel API for cross-tab sync. Node.js runs a WebSocket server. The BEAM uses Erlang's process groups — distributed by default, no external dependencies.

QuickBEAM

QuickBEAM is a JavaScript runtime for the Erlang VM, built on QuickJS-NG — a lightweight, standards-compliant JavaScript engine. Where Node.js embeds V8 in a C++ process, QuickBEAM embeds QuickJS in an Erlang NIF, giving JavaScript access to the BEAM's concurrency and fault-tolerance primitives.

Each QuickBEAM runtime is a lightweight isolate — Elixir can spin up a pool and dispatch requests round-robin across them, each running on its own OS thread. If one crashes, the OTP supervisor restarts it. The others keep serving. This is the same model Erlang uses for telecom switches — let it crash, recover instantly.

QuickJS isn't V8 — there's no JIT, so raw compute is slower. But for a web application that's mostly I/O (database queries, template rendering, HTTP responses), the difference is negligible. What you gain is a 5MB runtime instead of 45MB, sub-millisecond startup, and the entire OTP ecosystem.

The Architecture

The application runs inside QuickBEAM — a JavaScript runtime embedded in the Erlang VM. Elixir manages everything around it:

Browser (Turbo, Stimulus, Action Cable client)
    ↕ HTTP + WebSocket
Bandit (Elixir HTTP server)
    ↕ Plug router
QuickBEAM (JavaScript runtime pool)
    ↕ Beam.callSync
Elixir (:pg broadcasts, SQLite NIF, OTP supervision)

The JavaScript application handles request routing, controller logic, view rendering, and model operations — the same code that runs on Node.js. Elixir handles what it's best at: concurrency, fault tolerance, and distributed messaging.

Action Cable, Both Sides

The browser runs the real @hotwired/turbo-rails npm package — the same Action Cable client that Rails uses. The <turbo-cable-stream-source> custom element connects to /cable and speaks the Action Cable wire protocol.

On the server, Elixir implements the other side: WebSocket upgrade via Bandit, subscription management via :pg, and broadcast delivery in Action Cable's JSON format. When a model's broadcasts_to callback fires, it crosses from JavaScript to Elixir via Beam.callSync('__broadcast', channel, html), and Elixir fans it out to every subscriber.

Same protocol. Same custom elements. Same Turbo Stream HTML. The client has no idea it's talking to Elixir instead of Rails.

What the BEAM Adds

Things you get for free that would require significant infrastructure on Node.js:

Fault tolerance — a runtime crash restarts under OTP supervision, not pm2
Distributed pub/sub — :pg spans clustered BEAM nodes automatically. Add a node, broadcasts reach it. No Redis, no configuration.
True parallelism — pooled QuickBEAM runtimes on OS threads. Not single-threaded-with-cluster.
Hot upgrades — OTP releases support zero-downtime deployment

For production, swap SQLite for PostgreSQL — same app, juntos up -d postgrex. Database connections are pooled on the Elixir side via Postgrex, and combined with :pg for broadcasting, you get a fully distributed deployment with no external dependencies beyond Postgres.

A Path to Phoenix

This isn't just another deployment target. It's a migration path.

Your Rails app runs today inside QuickBEAM. The Elixir scaffold is a thin Plug/Bandit layer. But that layer could be Phoenix. At that point:

Rails controllers that need more performance? Rewrite as Phoenix controllers.
Models that need distributed state? Move to GenServers.
Views that need real-time interaction? Swap to LiveView.

One at a time. The rest keeps running in QuickBEAM. No big-bang rewrite.

No other migration path offers this. Going from Rails to Phoenix today means starting over. Juntos on BEAM gives you a running app on day one and an incremental path forward.

Try It

Prerequisites: Node.js (18+) and Elixir (1.18+). That's all you need.

npx github:ruby2js/juntos --demo blog
cd blog
npx juntos db:prepare
npx juntos up -d sqlite_napi

Same code. Same patterns. Different runtime. Source code. Documentation.

Juntos is open source: github.com/ruby2js/ruby2js

Our Journey: Building With Generative AI | Revelry

Mon, 30 Mar 2026 12:23:34 +0000

Businesses of all sizes and industries are eager to take advantage of generative artificial intelligence (AI), so I’m going to share some details on Revelry’s journey with this emerging technology over the past year. Even if you’re already working in the space, I believe you’ll find something interesting and helpful about our experience. We’ve got a lot of learnings to share, so this will be the first of a series of posts.

In this first post, I’ll cover our early exploration of generative AI – when we were moving as fast as possible to learn as much as we could. Subsequent posts will get deeper into our learnings around building more complex generative AI systems, in particular diving into how to incorporate Retrieval Augmented Generation (RAG) into software systems (and how you don’t need LangChain to do it).

Some Background

Here’s a little backstory on Revelry. Since 2013, we’ve been building custom software. Our bread and butter has primarily been web and mobile app development. In the early days, it was all about Rails and Node, but we’ve played around with PHP, .NET, Java, Python, and more.

We’ve always had a bit of a thing for new and emerging tech. Take React, for instance. Back in 2014, when JQuery and Angular were the big names, we were already building apps with React. And we didn’t stop there – we jumped into React Native pretty early, too. Our first app in React Native hit the AppStore when it was just at version 0.21, and now it’s up to 0.73. (By the way, when are we getting a major version update? Looking at you too, LiveView 😉).

We still work across a variety of tech stacks, but have collectively fallen in love with the elegance, performance, and strong community around Elixir and Phoenix, which we adopted as our preferred stack around 2018. We were building sophisticated Phoenix LiveView apps before there was even an official LiveView hex package published. (Yes, we were just referencing a commit hash in our mix.exs file — don’t judge.) We have done a lot in the blockchain space too, but I’m definitely not going into that in this article.

This is all to give you a glimpse into how we at Revelry dive into new technologies. We’re not shy about exploring the bleeding edge, and it’s really paid off. Our early dives into spaces like React and Elixir have made us the experts we are today.

Where We Started

Thinking back to June 2022, when OpenAI released GPT-3, it’s been nothing short of a rollercoaster ride. We at Revelry, like many software development companies, quickly caught on that this was a game-changer for our industry. Sure, we had a bunch of engineers who were into machine learning, but AI-driven apps weren’t really our main gig. Our partners didn’t ask for them much, and they didn’t seem all that necessary… until GPT-3 came along.

By Fall 2022, we were all in, diving deep into the world of these large language models (LLMs). The pace at which things have evolved since then is mind-blowing. Back then, things weren’t quite ready for the big stage, but it was obvious this was just the start.

We saw a golden opportunity to weave generative AI into our tried-and-true software and product delivery processes. This wasn’t about replacing our team, but turbocharging their productivity. Imagine our folks focusing on the creative, problem-solving aspects of product and software design, while AI handles the tedious stuff – like writing user stories, plotting out product roadmaps, or drafting sprint reports. And what if getting up to speed on a new project could be quicker and smoother? If this could work for us, it’d surely catch on elsewhere, right?

So, we rolled up our sleeves and jumped into the nitty-gritty. It started as a research and development adventure, filled with questions, like:

Just how far can the capabilities of these LLMs go?
What’s the engineering effort needed to integrate generative AI into our custom software?
What does it take to set up an AI-powered app in a live environment?
Can LLMs genuinely enhance our team’s productivity? If so, in what ways?
Is it possible to create something other engineering teams would want to use as well?

Experimentation

So, everyone at Revelry began dabbling with ChatGPT, mostly just for kicks. Some of us were crafting Eminem-style raps to add a bit of flair to our company-wide All Hands meetings (We’ve got a different Reveler hosting each week.). Meanwhile, our CEO, Gerard Ramos – or G, as we call him – was tinkering with how ChatGPT could enhance our product delivery process.

G found out pretty fast – with some clever prompting – that ChatGPT could whip up some solid product roadmaps and user stories, and even spin out working code examples based on those stories. This was more than just cool – it was promising. So, he proposed we start building tools around these use cases. And that’s how the idea for our first proof of concept came about: an AI-powered app to create user stories from just a few inputs. Sure, it wasn’t a game-changer yet, but it was a great starting point – allowing us to dip our toes in the water, while simultaneously boosting our productivity.

Our First AI- Powered Toy: StoryBot

Enter StoryBot. This little gem was a straightforward CLI tool that we ended up releasing as an open-source NPM package. It’s essentially a single JavaScript file, leveraging LangChain to tap into GPT-3 via OpenAI’s API (This was before GPT-4.). We threw in some tailored prompts, injected the user input, and voilà – it started spitting out decent user stories right in the command line.

We went a bit further with it after that, letting the user refine their story through chat, still all in the command line. The cherry on top was the ability to export the story as an issue in a GitHub Repo (At Revelry, we not only use GitHub to store our code, but also for issue tracking, project management, and more.). Ultimately, we ended up with this StoryBot iteration:

StoryBot Under the Hood

Diving into the StoryBot repo, you’ll see the core functionality is in one JavaScript file. This file uses LangChain.js for communicating with the OpenAI API, generating user stories from command line inputs. We could have opted for LangChain’s Python library, but the two have close to feature parity, and our team works more in JavaScript than Python. At this point, it was all still experimentation, so we opted for whatever we could move the fastest in.

Technically, for such a straightforward use case, direct API calls to OpenAI would have done the trick. However, LangChain offered ease of setup and capabilities beyond just interfacing with an LLM. It’s packed with features for creating AI-powered apps, like Retrieval, Agents, and Chains, though we didn’t dive deep into any of these for StoryBot.

LangChain simplifies building AI applications, but it’s not without its complexities and limitations. It abstracts a lot, sometimes obscuring the underlying mechanics, and is currently limited to Python and JavaScript ecosystems. There is now an Elixir implementation of LangChain, which is exciting because we’re huge Elixir fans, but it isn’t nearly as far along as its Python and JS counterparts. This Elixir library also wasn’t around yet at this point in our journey.

Looking a Bit More Into StoryBot Code

The first code that actually gets executed when you run npx gen.story generates the initial prompt:

const generateInitialPrompt = () => {
  const { featureText, techStackText, contextText } = parseArgs();

  return `Context: Act as a product manager at a software development company. Write a user story for the 'Feature' defined below. Explain in detailed steps how to implement this in a section called 'Implementation Notes' at the end of the story. Please make sure that the implementation notes are complete; do not leave any incomplete sentences. ${contextText}

  ${featureText}

  ${techStackText}

  User Story Spec:
    overview:
      "The goal is to convert your response into a GitHub Issue that a software engineer can use to implement the feature. Start your response with a 'Background' section, with a few sentences about why this feature is valuable to the application and why we want the user story written. Follow with one or more 'Scenarios' containing the relevant Acceptance Criteria (AC). Use markdown format, with subheaders (e.g. '##' ) for each section (i.e. '## Background', '## Scenario - [Scenario 1]', '## Implementation Notes').",
    scenarios:
    "detailed stories covering the core loop of the feature requested",
    style:
      "Use BDD / gherkin style to describe the user scenarios, prefacing each line of acceptance criteria (AC) with a markdown checkbox (e.g. '- [ ]').",
  }`;
};

...

const prompt = generateInitialPrompt()

You can see that the prompt has specific instructions to format the story in the way that Revelry prefers, which may be different than a lot of other teams. This said, if anyone wanted to use this with different prompts, they could easily fork it and change the prompts. The most important part here is that we are injecting user input into the prompt before we send it to the LLM. In this case, there are 3 potential user inputs:

The feature in question, which comes in as the 3rd command line argument (e.g. npx gen.story [feature]).
optional --stack flag to specify the tech stack that the user story will need to be implemented in.
optional --context flag to add some additional context around the feature you are writing a user story for.

Next, we take that hydrated prompt and send it to OpenAI using the tools provided by LangChain:

const model = new OpenAI({
  streaming: true,
  modelName: "gpt-3.5-turbo",
  callbacks: [
    {
      handleLLMNewToken(token) {
        process.stdout.write(token)
      },
    },
  ],
});
const memory = new BufferMemory()
const chain = new ConversationChain({llm: model, memory: memory})
const {response} = await chain.call({input: prompt})

A few things happen at this point: we are creating a new ConversationChain object, which is a wrapper around the LangChain Chain object that we’ll use to send the prompt to the LLM. We are also creating a BufferMemory object, which is a LangChain Memory object that we’ll use to store the results of the conversation.

Sidenote: If we were just using the OpenAI API directly instead of LangChain, it would be easy to pass the chat history alongside the prompt to the API call. (I’m just clarifying that LangChain isn’t necessary for this, even though it was very easy to set up.)

Because we set streaming: true when we initialized the OpenAI object, the chain.call method will return immediately, and the LLM will start sending responses to the callback we set up earlier. Since StoryBot is a CLI tool, we’re just outputting to process.stdout here. If you’re thinking about adapting this for a web app, you’d probably need to figure out how to send JSON responses or stream them to the client. We’ll get more into that later. The main takeaway? It doesn’t take much to start seeing some cool results by plugging user inputs into a well-crafted prompt template and sending it off to GPT-3.

So at this point, there is a response that is the generated user story, but also the entire user story has been streamed into the terminal and could easily be copy pasted to wherever. However, there is no ability to make any followup refinements yet. I’m not going to go line-for-line through the rest of the final result, but the long story short is that after we get the initial generated response back, we pass it to a function that creates a readline interface that allows us to prompt the user with questions in the terminal, and then we take the users’ response and send it back as another message to the LLM in the chat history. We also added the ability to export the final result to github if you had the github API token set.

That’s it, that’s StoryBot!

If you want to play around with it, you can install it via npm.

npm install -g storybot-ai

Fun? Absolutely. Somewhat useful? Sure. But let’s be real – it was an experiment. The thing is, not everyone wants to write user stories via the command line. Plus, every team has its own style for these stories. Our hardcoded prompts were great for us, but might not hit the mark for teams outside of Revelry, especially since we often work in staff augmentation, where teams have their own preferences.

Once we had a proof of concept, we started to see the potential. We were able to get a lot of mileage out of it, and it was a great way to get started with generative AI. This got a lot of ideas spinning about how we could get better user stories based on relevant context, which ultimately led us to the next part of our journey: diving deeper into RAG (Retrieval Augmented Generation) application development.

This is the first post in a series about Revelry’s journey exploring and developing custom software powered by generative AI. The next post will dive into our next experiment: building a chatbot to answer questions about Revelry based on our company playbook. Stay tuned!

Until then, here are a few other articles we’ve put out about AI:

Behind the Scenes: Replacing a Black-Box Billing Service

Fri, 27 Mar 2026 12:04:11 +0000

In a previous blog post, we looked at the process of migrating 600k users to a new billing service for Steady Media. This was the final step of an internal project called “Charger”. Its aim was to replace the off-the-shelf payment platform they had started with, Chargebee, which had become a roadblock. In this post, we will reveal the behind-the-scenes insights into the process that made that final step a success.

The Challenge

Steady Media needed to migrate 600,000 users from Chargebee to a new in-house billing system. The complexity was brutal:

Chargebeeʼs billing logic is opaque - we had to reverse-engineer behavior through observation
Historical data had to work in the new system (canʼt rewrite the past, especially not payment transactions!)
Multiple payment gateways (Braintree, GoCardless, PayPal) with different quirks
Complex billing workflows: trial subscriptions, gift subscriptions, mid-term subscription upgrades, subscription cancellations (with or without refunds)
Complex support workflows: issuing refunds, invoice & credit notes document versioning
Legal compliance (invoicing and taxes)
Migration had to happen without downtime or data loss

Steadyʼs development team asked us to lead the Charger project, so that they could focus on their core application. The tech stack had to be compatible with their team skills: Elixir & Phoenix!

🤩 Like what you see? have a look at the case studies on bitcrowd.net

The Journey

bitcrowd built Charger across three iterations:

2021: Core System

The first iteration aimed at building the data model and implementing the external communication layer of Charger:

API layer with authentication between Charger and the Media Makers app
Webhook notifications between Charger and the Media Makers app
Payment providers: integrated Braintree and GoCardless services (API & webhooks)
Multiplexed calls from the Media Makers app to hit Chargebee OR Charger depending on where the user lived

Once these various components were finalized, we could start the implementation of the core flows: create new subscriptions, bill users and trigger payments, upgrade subscription plans with prorated billing, settle payments, and so on.

2023: Feature Parity

After the first development cycle ended, new features were added to the Media Makers app that needed to be back-ported to Charger. We added new models and new flows to support trial and gift subscriptions. We took the opportunity to add Paypal as a third payment provider. Finally, we also tackled the generation and design of legal billing PDF documents.

2025: Admin UI, Support Tools, and Migration

A missing piece in Charger was a simple, lightweight, workable admin UI, that could be useful for Steadyʼs developers, QA engineers, support staff and financial experts. The goal was to minimize the implementation effort, while preserving a clean UX. We opted for DaisyUI since we knew the admin UI would use standard components like index tables, description lists, navigation elements etc.

Powerful search

One strong requirement was to make sure that the index pages had a powerful and performant search as well as sorting and filtering mechanisms, as one of the pain points of Chargebee had been searches that time out once enough data was in the system. We solved this issue by paying attention to the queries' efficiency. Additionally, for cross-table filtering, we implemented a search-typeahead dropdown in order to avoid expensive joins.

Activity logs

For such a billing product, it was crucial to allow admin users to see a clear changeset history on any resource. Charger comes with an auditing layer, that tracks changes and versions each relevant resource (like payments, invoices...). We built an abstraction on top of the auditing, that allows to plug any resource in it and view its version history with a UI resembling a Git diff.

Support staff tooling

At that stage, we had a working product for developers, QA engineers and financial experts. But Steadyʼs support team still needed tailored features to enable them to solve specific situations. For example, when a userʼs credit card expired, and the system could not withdraw money from their account, the user might send the missing amount via a bank transfer. On Steadyʼs bank account, the balance is correct, but Charger does not know about this transaction, and the accounting balance is affected. Similarly, the support team might refund a user via a manual bank transfer. To keep the accounting in check, we implemented a solution to record offline refund & payment, as well as various manual actions & flows to fix any situation that would not auto-heal.

💡 Have we got you interested? If you have a question or topic you would like to discuss with us, we would like to hear from you.Book a free call with us

Outcome

We migrated all 600,000 users to Charger. The lights stayed green. Steady now controls their billing infrastructure and its support team can handle edge cases and requests through the admin UI. The developer team can also leverage the activity logs and powerful search capabilities to debug or resolve errors.

bitcrowd led on self-contained packages: we took topics Steadyʼs team didnʼt have capacity for and delivered them finished. We were able to coordinate with stakeholders (support team, finance, legal) to turn “we need X” into actionable tickets. We applied our standards of excellent documentation and test coverage, high code quality and review, which ensured a smooth handover with Steadyʼs team.

Migrating 600k users to a new billing service

Fri, 27 Mar 2026 12:04:11 +0000

In August 2025, we helped our friends at Steady to migrate their 600,000+ users to their own in-house billing system. This was the final step of an internal project called “Charger”. Its aim was to replace the off-the-shelf payment platform they had started with, Chargebee, which had become a roadblock. The work on Charger began already in 2021 in workshops with the Steady team. Together with them, we designed, planned and finally built the internal product that would take over for years later. The process and the implementation happened in multiple iterations over a span of four years.

This blog post explores the strategy that was used to achieve this challenging task.

The backstory

Steady supports Media Makers in building an audience and driving revenue via subscriptions to newsletters, blogs etc. While most of the tools and features for Media Makers live in their main application, they outsourced the subscriptions & billing management to a third-party service (Chargebee). The Media Makers app was very tightly coupled to Chargebee as it relied on it for core business logic within the app, as well as for accounting.

Introducing: Charger, the fraternal twin

Although Chargebee was useful initially, it became increasingly painful to work around their standard processes over time. This ultimately led to the development of a custom billing tool.

For this we had to reverse-engineer all of what Chargebee was doing. We started by building the required schemas, and exposing the API endpoints and webhook notifications that were needed by the Media Makers app. Any changes in the Media Makers app usage of the API/Webhooks, or to Chargebee itself, needed to be propagated to Charger. Both billing systems had to behave identical from the outside.

The Media Makers app now needed to forward calls to Charger and/or Chargebee. A “multiplexer“ abstraction close to the API & Webhook layer was added, so that for developers & users of the app, the fact that Charger or Chargebee was used for the calls would be transparent.

Charger needed to process payments and refunds, with all payment providers supported by the Media Makers app. The first flow of the application was created: subscribe to a publication, generate an invoice, create a payment, and settle that payment 🎉!

🔍 Would you like more detail? We have a 'behind the scenes' blog post about Steady! Have a Look!

A bird watching experience 🔭

At this stage, no real users were migrated to Charger. No real users were even created on Charger! The system just worked™... Subscription statuses, invoices, payments processing being crucial to the system meant that we had to come up with a transition plan, that allowed the developers to observe and watch the behaviour of the system for just a handful of users. If any problem arose, it would need to affect only such a small amount of users that their support team could resolve it manually. We handpicked 10 users with rather simple data: a single recent subscription, no cancellations, and various payment providers / currencies to cover as much ground as possible.

Using our beloved Oban for job processing, we built a migration job, that would, for a given user:

Fetch all their data from Chargebee
Format it to please Chargerʼs data model
Send it to a dedicated endpoint in Charger
Charger creates all the given data for the user (subscriptions, invoices, payments, etc.)
Cancel the user subscriptions on Chargebee (so that they donʼt get billed twice!)

Of course, all of this must happen in a transaction on both systems, fail gracefully and report meaningful errors to the devs. We leveraged Oban configuration to optimise the parallelisation and retry mechanism of the jobs, as we could be rate-limited by Chargebee. At some point, we would want to migrate users in batches of 100-10k users at a time, so the migration mechanism had to be robust and auto-healing.

💡 Have we got you interested? If you have a question or topic you would like to discuss with us, we would like to hear from you.Book a free call with us

Silent bugs

A main challenge in this process was to validate the integrity of the migrated data. The Media Makers app and Charger run on different databases, and have their own representation of what is an invoice, a subscription, a user, etc. Forgetting to migrate a field, or mapping the wrong timestamp from one system to the other, would happen silently because a part of the migration code lives in the Media Makers app, and the other part lives in Charger. The data could be silently missing or corrupted, and we would only realise down the road that all users were migrated with errors. Luckily, we had the chance of having access to a Chargebee staging instance with users in all kinds of configuration. This helped us a lot in finding migration issues by comparing the user data on both systems before and after the migration, and strengthening our tests to cover all the edge cases we could find. The developers at Steady also added ERPC, which enabled us to run integration tests evaluating the effects of Charger on the Media Makers app.

Conclusion

Once we were confident that our 10 users were rolling, we migrated 10 more users with more complex data. And so on, for weeks, increasing the batch size, and deciding to create new users on Charger with a rand() choice. This allowed Steady to steer the future of their billing strategy for their users, integrating with other payment providers and simplifying some of the user flows for refunds etc. While it looks simple on paper, this project required a lot of patience, collaboration and planning ahead as the stakes were really high and we had to build up confidence in the new system and in the migration approach.

Building a blog with Elixir and Phoenix | jola.dev

Thu, 26 Mar 2026 19:19:26 +0000

TL;DR: it’s an Elixir app using Phoenix server side rendered pages, with the blog post pages generated from Markdown using NimblePublisher. It’s running on a self-hosted Dokploy instance running on Hetzner, with bunny.net as a CDN sitting in front of it.

This is a very belated write up of how this blog was put together! There’s nothing terribly original here, but I figure it could come in handy for someone out there as a reference. And the world needs more Elixir content.

Why Phoenix

I have used static site generators before to power my blog (shoutout to Hakyll), but I wanted to open the door for myself to also have little experiments on this site, ones that would require more interactivity than a static site allows. Besides, I just like using Phoenix. Although most of my Phoenix projects use LiveView, this felt like a good place to do things old-school with DeadViews.

It also means I get full control of what I’m building. Using a tool someone else created means getting a lot for free, but the moment you step outside of the expected you’re having to figure out how to make things work for their tool.

So I kept things simple. No Ecto, no DB. Just server-side rendered HTML. It’s blazingly fast, as you can see from this PageSpeed Insights report.

NimblePublisher

My setup closely matches the original Dashbit blog post Welcome to our blog: how it was made!, which led to the creation of NimblePublisher.

The heart of the blog is the NimblePublisher setup, which consists of a use block:

defmodule JolaDev.Blog do

  use NimblePublisher,
    build: JolaDev.Blog.Post,
    from: Application.app_dir(:jola_dev, "priv/posts/**/*.md"),
    as: :posts,
    html_converter: JolaDev.Blog.MarkdownConverter,
    highlighters: [:makeup_elixir]
...

This will load up all the posts, parse the frontmatter, run it through the markdown converter, and compile it into module attributes. This means there’s no work left to be done at runtime, it’s all pre-compiled.

Posts are organized by year: priv/posts/2025/08-18-ruthless-prioritization.md . We get beautiful code block syntax highlighting through Makeup. The Blog module also defines a set of helpers for fetching the posts:

@posts Enum.sort_by(@posts, & &1.date, {:desc, Date})

# Let's also get all tags
@tags @posts
      |> Enum.flat_map(& &1.tags)
      |> Enum.uniq()
      |> Enum.sort()

# And finally export them
def all_posts, do: @posts
def all_tags, do: @tags

def posts_by_tag(tag) do
  Enum.filter(all_posts(), fn post -> tag in post.tags end)
end

def find_by_id(id) do
  Enum.find(all_posts(), fn post -> post.id == id end)
end

The only thing that took a bit of figuring out for me was getting Tailwind classes into the outputted HTML. I’m pretty sure I’ve seen better approaches shared since I wrote this, but this works too. Under earmark_options, pass:

Earmark.Options.make_options!(
  registered_processors: [
    Earmark.TagSpecificProcessors.new([
      {"a", &Earmark.AstTools.merge_atts_in_node(&1, class: "underline")},
      {"h1", &Earmark.AstTools.merge_atts_in_node(&1, class: "text-3xl py-4")},
      {"h2", &Earmark.AstTools.merge_atts_in_node(&1, class: "text-2xl py-4")},
      {"h3", &Earmark.AstTools.merge_atts_in_node(&1, class: "text-xl py-4")},
      {"p", &Earmark.AstTools.merge_atts_in_node(&1, class: "text-md pb-4")},
      {"code", &Earmark.AstTools.merge_atts_in_node(&1, class: "")},
      {"pre",
       &Earmark.AstTools.merge_atts_in_node(&1,
         class: "mb-4 p-1 py-4 overflow-x-scroll border-y"
       )},
      {"ol", &Earmark.AstTools.merge_atts_in_node(&1, class: "list-decimal")},
      {"ul", &Earmark.AstTools.merge_atts_in_node(&1, class: "list-disc pb-4")},
      {"blockquote",
       &Earmark.AstTools.merge_atts_in_node(&1,
         class: "pl-4 border-l-2 mb-4 border-purple-700"
       )}
    ])
  ]
)

You probably have your own preferences for how to set up your classes, but this gives you a pattern you can use to ensure that the tags that come out have the appropriate classes.

The Frontend

As mentioned this is all server-side rendered Phoenix templates. It’s using standard Tailwind CSS. It predates DaisyUI and I don’t think there’s a strong reason for me to make the lift of getting it in, although I wouldn’t have minded it being a part of the scaffolding back when I set up the blog!

The only JS snippets in here are a mobile menu toggle and the Phoenix topbar. Apart from the Tailwind library, the custom CSS in here is pretty minimal. You get a lot out of the box with a Phoenix project.

And of course, dark mode. I know it’s not everyone’s cup of tea, but it is my website after all.

CI

I’ve got Github Actions set up to run on every push and PR, just the basic Elixir quality assurance tools.

mix compile --warnings-as-errors
mix format --check-formatted
mix credo --strict
mix test

And then I’ve got Dependabot set up as well. I’ve been hearing and thinking a lot about how it creates a lot of noise, but I feel like that’s less of an issue in the Elixir community. Packages tend to not have a lot of dependencies, and so you don’t get the same waves of bumps going out that npm does. And merging them is satisfying.

Deployment

On the hosting side things get a bit more spicy. The repo includes a multi-stage Docker file, roughly based on the Phoenix recommended example file. This means that most of the dependencies are only pulled in at build time, and the image you get out on the other side is a bit smaller. I’m using Elixir 1.18.4, Erlang 28.0.2, and Debian trixie-20250721-slim at the time of writing this, but that’s likely to change. There’s something very satisfying about bumping dependencies.

And now we’re arriving at Dokploy, an open source platform as a service (PaaS) for running apps, basically a self-hosted Heroku. It does everything, automatic builds and deploys from Github updates, built-in Docker Swarm, networking, orchestration of replicas across the cluster, rolling deploys, rollbacks, preview builds, and much more.

So my publish flow is basically: create a PR and wait for CI to finish (I could skip this but it’s nice to know I didn’t mess something up). When I merge the PR Dokploy automatically picks that up and triggers a checkout and build of the repo. Once that finishes, it starts a rolling deploy to replace the running replicas. And we’re live. With cached layers on the server, deploys can finish in 30s, zero effort.

I run this Dokploy instance on Hetzner and my experience has been really positive. The pricing is unbeatable, even with the recent increase, and it’s been rock solid for me. Really, with the Dokploy instance, there’s nothing stopping me from packing up and going somewhere else. Having that kind of freedom is very nice. But I’m more than happy to stick with Hetzner.

The Little Things

I’ve set up a few little conveniences for my app so I’ll share some example code for them here.

RSS

RSS is managed by a plain Phoenix controller that looks something like this:

defmodule JolaDevWeb.RssXML do
  use JolaDevWeb, :html

  embed_templates "rss_xml/*"

  def format_rfc822(%Date{} = date) do
    date
    |> DateTime.new!(~T[00:00:00], "Etc/UTC")
    |> format_rfc822()
  end

  def format_rfc822(%DateTime{} = datetime) do
    Calendar.strftime(datetime, "%a, %d %b %Y %H:%M:%S +0000")
  end
end

and the corresponding XML:

<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:content="http://purl.org/rss/1.0/modules/content/">
  <channel>
    <title>jola.dev</title>
    <link><%= url(~p"/") %></link>
    <description>Blog posts from jola.dev</description>
    <language>en-us</language>
    <lastBuildDate><%= JolaDevWeb.RssXML.format_rfc822(DateTime.utc_now()) %></lastBuildDate>
    <atom:link href="<%= url(~p"/rss.xml") %>" rel="self" type="application/rss+xml" />

    <%= for post <- @posts do %>
    <item>
      <title><%= post.title %></title>
      <link><%= url(~p"/posts/#{post.id}") %></link>
      <description><![CDATA[<%= post.description %>]]></description>
      <content:encoded><![CDATA[<%= post.body %>]]></content:encoded>
      <pubDate><%= JolaDevWeb.RssXML.format_rfc822(post.date) %></pubDate>
      <guid isPermaLink="true"><%= url(~p"/posts/#{post.id}") %></guid>
      <author><%= post.author %></author>
    </item>
    <% end %>
  </channel>
</rss>

Sitemap

I was a bit surprised not to find a clean little library for generating the sitemap (this may have changed since I wrote the code!), but I guess the implementation is just going to heavily depend on your setup. Anyway, just sharing this for reference.

defmodule JolaDevWeb.SitemapController do
  use JolaDevWeb, :controller

  def index(conn, _params) do
    sitemap = JolaDev.Sitemap.generate()

    conn
    |> put_resp_content_type("text/xml")
    |> send_resp(200, sitemap)
  end
end

defmodule JolaDev.Sitemap do
  alias JolaDev.Blog

  @host "https://jola.dev"

  def generate do
    """
    <?xml version="1.0" encoding="UTF-8"?>
    <urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
    #{generate_static_pages()}#{generate_tag_pages()}#{generate_blog_posts()}
    </urlset>
    """
  end

  defp generate_static_pages do
    pages = [
      %{loc: @host, changefreq: "monthly", priority: "1.0"},
      %{loc: "#{@host}/about", changefreq: "monthly", priority: "0.8"},
      %{loc: "#{@host}/projects", changefreq: "weekly", priority: "0.9"},
      %{loc: "#{@host}/talks", changefreq: "monthly", priority: "0.7"},
      %{loc: "#{@host}/posts", changefreq: "weekly", priority: "0.9"}
    ]

    Enum.map_join(pages, "\n", &url_entry/1)
  end

  defp generate_tag_pages do
    Blog.all_tags()
    |> Enum.map(fn tag ->
      %{loc: "#{@host}/posts/tag/#{tag}", changefreq: "weekly", priority: "0.6"}
    end)
    |> Enum.map_join("\n", &url_entry/1)
  end

  defp generate_blog_posts do
    Blog.all_posts()
    |> Enum.map(fn post ->
      %{
        loc: "#{@host}/posts/#{post.id}",
        lastmod: Date.to_iso8601(post.date),
        changefreq: "monthly",
        priority: "0.8"
      }
    end)
    |> Enum.map_join("\n", &url_entry/1)
  end

  defp url_entry(params) do
    """
      <url>
        <loc>#{params.loc}</loc>
        #{if params[:lastmod], do: "<lastmod>#{params.lastmod}</lastmod>", else: ""}
        <changefreq>#{params.changefreq}</changefreq>
        <priority>#{params.priority}</priority>
      </url>
    """
  end
end

Blog redirect plug

When I first moved over to this new app I wanted to ensure that I kept my old blog post links alive, so I set up this little plug to rewrite requests to match the new layout.

defmodule JolaDevWeb.Plugs.BlogRedirect do
  import Plug.Conn

  def init(_), do: []

  def call(conn, _opts) do
    if conn.host == "blog.jola.dev" do
      ids = JolaDev.Blog.ids()
      path = strip_path(conn.request_path)

      path =
        if path in ids do
          "posts/" <> path
        else
          path
        end

      conn
      |> put_resp_header("location", "https://jola.dev/" <> path)
      |> send_resp(:moved_permanently, "")
      |> halt()
    else
      conn
    end
  end

  defp strip_path("/" <> path), do: path
  defp strip_path(path), do: path
end

SEO

I went a bit further on this one. Each page has its own meta description, Open Graph tags, and Twitter Card tags — all driven by assigns passed from the controllers. Blog posts automatically get og:type="article" with article:published_time and article:tag set from the post metadata. The layout just reads from conn.assigns with sensible fallbacks, so adding SEO to a new page is just a matter of passing the right assigns. Here’s what the blog-post-specific bits look like in the layout:

<meta property="og:type" content={if(@conn.assigns[:post], do: "article", else: "website")} />
<%= if post = @conn.assigns[:post] do %>
  <meta property="article:published_time" content={Date.to_iso8601(post.date)} />
  <meta property="article:author" content="https://jola.dev/about" />
  <%= for tag <- post.tags do %>
    <meta property="article:tag" content={tag} />
  <% end %>
<% end %>

Same idea for the Twitter Card and description tags — one place in the layout, driven entirely by what the controller passes in.

I also added llms.txt and llms-full.txt endpoints, this is a newer standard that helps AI systems understand your site. It follows the same pattern as the sitemap: a module that generates the content from Blog.all_posts(), and a controller that serves it as plain text. Whether it actually matters yet, who knows, but it was trivial to add and I figure it can’t hurt.

Wrapping Up

This app is intentionally kept simple but powerful. Everything is set up the way I want it and I have a zero effort and very fast pipeline for publishing new posts. If you’re an Elixir dev thinking about a personal site, consider just using Phoenix. Combined with NimblePublisher you’ve got a really powerful and blazing fast blog framework right there.

And while you’re at it, why not host it on Hetzner! If you use the referral link to sign up you get €20 and I get €10. If you prefer not to use the referral link, here’s a plain link: https://www.hetzner.com/cloud/. Also consider joining me in sponsoring Dokploy.

Source code is available at: https://github.com/joladev/jola.dev. Next up I’ll talk about setting up bunny.net and a separate post on Dokploy on Hetzner.

Software Development in 2026

Wed, 25 Mar 2026 16:08:30 +0000

Three years ago I wrote a fairly coherent piece on four key developer skills, but a good deal of water has flowed under the bridge since then, and while the theses laid out there remain sound, they need a slight adjustment in light of the boon that has descended upon us in the form of large language models.

I went through all five stages of the inevitable over the course of a year.

① Denial—I watched my colleagues a year ago raving about autocomplete and hallucinations, and even won a few bets—not unlike the one the Adriano Celentano character proposes to his accountant in that great film.

② Anger—I kept writing code by hand, but part of my job involves cleaning up colleagues’ messes after shifted concepts (I do a lot of reviews), and the volume of neural slop had crossed every conceivable boundary: even when the code compiled, it looked like Rome from a bird’s-eye view—slovenly fragments of good practices scattered here and there, interspersed with the slums of deeply nested conditionals. I found myself literally rewriting large chunks after the little models, because I take code review seriously and still see it as a tool for teaching apprentices.

③ Bargaining—about seven months ago I tried unleashing a model on an old library of mine that was desperately in need of proper documentation; to my surprise, the documentation turned out coherent, nearly complete, and unquestionably better than nothing. I screwed my eyes shut and asked for tests. Half of them tested the standard library and implementation details—but the other half brought genuine value. Like those gruff stubborn men from the joke, I said: “Welllll, damn.” And paid for Warp.

④ Depression—in the first month of use I finished two personal projects that had been gathering dust for years, equipped all my libraries with detailed documentation and the missing tests, played around with creating my own programming language, and even trusted the model to fully solve a take-home assignment for one of the companies that had written to me directly with an offer (the assignment sailed through, but I never would have taken the job anyway—their HR had violated every conceivable rule of decency in headhunting). I wasn’t afraid, of course, that I’d be thrown out and Claude hired in my place—the models won’t reach my level of expertise before I retire—but writing code is one of my most beloved occupations in life, and I felt it being taken away from me.

⑤ Acceptance—it is now the end of March 2026, and I can say with confidence that language models provide me with substantial help in development, without particularly intruding on the part of my life I cherish: they write excellent documentation, reasonably coherent tests, and—under supervision and with Ctrl+C at the ready—can shuffle JSON around. Complex code I still write myself, and I’m confident this will continue until my death at the wheel of a sports motorcycle.

At the same time, the internet is proliferating with success stories from every variety of blowhard—from startup founders with three grades of parochial school, armed with enthusiasm, a narrow worldview, and the free tier of ChatGPT—to bedroom traders whose imaginary profits already permit them to purchase a paper yacht in Bali. In the hands of people far removed from software development, large language models are doomed to two low-yield applications: you can amuse yourself generating memes of piglets drinking mojitos in the Kremlin, or you can reproduce a product that already exists on the market and that in its niche you’ll never catch up to.

To use models effectively (you can now also deploy agents, but this changes nothing whatsoever about the substance), you need—at minimum—to understand the principles by which they operate. It took the automobile industry a hundred years to give users the joy of driving a vehicle without ever having opened the bonnet. Aircraft haven’t reached that stage yet. I see no reason to suppose that very advanced autocomplete is capable of repairing itself. And this is besides the fact that language models are, in principle, a dead end in the development of artificial intelligence. Even Yann LeCun has understood this, though unfortunately it remains unclear whether his own ideas aren’t yet another dead end.

Yeah.

If you need to knock together from scratch a shop for the worthless trinkets your wife makes—a modern model will handle it with flying colours. Clean unpretentious design, convenient addition of new bracelets, photos, payment system. The little model will knock you up a website, a mobile app, and lord knows what else. And it will work first try, most likely—because in training it has stared at such dreck in billions of different variations. What everyone finds so astonishing is, in essence, the output of four shell commands: cp, grep, sed, awk. Last century this technology bore the proud name “snippet.” For cloning already-existing things, then, the model is a perfectly fine assistant. Web studios that build landing pages will probably indeed die (they should never have existed in principle, but that’s a different question).

As for more complex projects—not necessarily more complex per se, but more unusual—without an architect to guide it, the model will disgrace itself at the very first hurdle. Because a vaguely described result can be achieved in fifteen different ways, and sooner or later the rough-and-ready decisions of a spring balance (a steelyard, not a precision scale) will lead into a swamp from which there’s no escape—because admitting defeat is not our way, and the model will play roulette to the bitter end, like Dostoevsky in Baden-Baden.

I recently wrote about why elaborate prompts and detailed descriptions cannot produce a good result—see part two of Artificial ‘Intelligence’—I won’t repeat myself here, but will quote the key thesis:

A person capable of breaking a large task down into the minimum number of smaller ones that satisfy the condition of “unambiguous solution” is capable, in today’s reality, of writing a reasonably complex application in a couple of days.

What I mean is that the ability to distinguish tasks with branching logic from multi-step syllogisms is more in demand than ever. Draw a mental flowchart of the execution—and if there are decision diamonds in it, you need to work them out for the model explicitly (go left here, go right here, snow on head here, very painful). Better still—break the main task into several, to eliminate those “conditions/decisions” entirely. But this is hardly possible if you have no idea how to write such code yourself.

Besides the fools who tried to build a website and succeeded—there are also saboteurs. They are considerably more dangerous in that they give the unprepared reader the impression of being “people in the know.” Did I mention that before paying for a subscription to a cloud model I thoroughly understood exactly how they work? I always do. If I get it into my head to drag library xyz into a project, I start by digging into its source code. When choosing a technology I don’t read adoption success stories and watch benchmarks—I write my own “trinket shop in the Bronx” with it. To use language models in daily work, and actually pay for the privilege, Altman’s feverish ravings and Karpathy’s coquettish napkin-code are not enough for me. I need to understand how it functions under the hood, so as to preemptively avoid disappointment and the collapse of hopes.

In short, those who have learned by instruction to run models and ask them questions pulled from thin air—supposedly testing or even proving something—would frankly be better off keeping quiet. Because everything that has crossed my field of vision resembles hallucinations from the first version of ChatGPT, Joyce’s Ulysses, Castaneda’s astral flights, and in principle any address at a party congress by the secretary of the Upper Walrusville cell, having gorged himself on fly agaric.

Let me skim glissando—or, as Dovlatov used to say, in dotted lines—across the main points.

Tests and Benchmarks

Comparing different models based on sets of cardboard tests and plastic benchmarks is pure, undiluted charlatanism. It’s enough to look at the language summary table at AutoCodeBench → Experimental Results (yes, this is sarcasm). Claude Opus 4 fails to hit 50% for TypeScript but clears 80% for Elixir. Translating from accountant-speak into plain language (and exaggerating slightly, of course)—if your project is in TypeScript, the model is more of a hindrance; if it’s in Elixir, you can hand it a 1,000-line refactor.

Context

RAG in any moderately complex project (or more precisely, its “RA” part) matters hundreds of times more than the model itself. Why do all these Claudes burrow into your computers with their IDEs (and lately—CLIs), do you think?—It’s simple: shipping the entire context to the server every time is expensive and inefficient (and despite the flagship advertising promises, nobody can actually process more than a hundred thousand tokens without noticeable quality loss). So every model sends some kind of “distillate” to the server.

It’s important to understand that any language model operates as a finite state machine: start → context → query → response → stop. There are no “sessions.” You cannot “launch” a model and converse with it—every request you make starts from a blank slate. Models as such have no “memory”—which is why preserving context is critically important. But unlike Hoffman’s character from Rain Man, even we cannot memorize six decks of cards—let alone models, with their context window as narrow as the Strait of Hormuz. My slowly-simmering project Ragex is an attempt to somehow formalize—and minimize without loss of generality—the context required for processing sizeable codebases. We’ll see whether I manage it—but the fact that I appear to be alone in this on the visible horizon is not exactly inspiring.

Plans and Reasoning

When tackling any reasonably non-trivial task, you need to force the model to sketch a plan and ask all the questions that arose for it while creating that plan. It will gladly surrender to you all the internal decision branches from the flowchart. These questions must be answered as clearly as possible—in blunt, clipped phrases that admit no double interpretation. The requirement to supply each phase of the plan with tests and bring all project documentation fully in line with the current state of the codebase must be baked into the general rules.

Each stage calls for a git diff with an informal review. I also always use “thinking” models, and watch those reasoning traces in real time—so that the moment it tries to veer off course (and on non-trivial tasks it will always try)—I can kill the little pest with Ctrl+C and explain why it’s wrong.

Language

In my experience, it’s best to use the language in which an answer is easiest to find on the internet. For code—always English; for obscure details of Lorca’s biography—Spanish; for a survey of the sexual services market in Berlin—German.

I have no hard evidence, unfortunately—only a general understanding of how this T9 on steroids operates—but the empirical record is copious. My Spanish hovers somewhere around B2–C1 level, so I still try English first; if the result looks underwhelming, I strain my fingers in Spanish—and in the vast majority of cases, I don’t regret it.

Politeness

After each successful task completion I say something like “Awesome,” or “Astonishing,” or “Stunning,” or words to that effect. I phrase every request as a request and always add “please.” My behaviour affects carbon emissions about as much as it affects democratic elections in a country of fifty million—but it makes things simpler and more pleasant for me. Besides, as Niels Bohr once said: “Of course I am not superstitious. But they say a horseshoe brings luck even to those who don’t believe in such nonsense.”

Horizontal Scaling

Wed, 25 Mar 2026 16:08:30 +0000

Part four of four key developer skills.

Ability to immediately build a horizontally scalable solution without adding any special code for it in the first version.

This turned out to be the hardest piece in the series, because literally a handful of developers actually understand what “horizontal scaling” is. Shown above is a screenshot of an amusing tweet by Tobi Lütke, which demonstrates either his supreme professionalism in sophistry, or his complete technical incompetence.

Serving ten billion clients does not mean your service is scalable. Buses carry people along Nevsky Prospect, and buses carry people along the Garden Ring; anyone who has completed three grades of parochial logistics school will tell you that speaking of “reliable connections between Leningrad and Moscow” on the basis of this incontrovertible fact is premature.

Even within a single city, the situation can easily spiral out of control. Scaling a bus fleet to match the expanding footprint of a metropolis is not about buying and deploying more buses. I lived in Berlin twenty years ago and could not stop marveling at how well the transport was organized. Judging by today’s passenger reviews, as the city spread outward—BVG did not cope.

I invite the thoughtful reader to pause here and think: so what exactly went wrong with the service’s scaling? For the superficial blockheads who can’t be bothered to think through what they’re reading—the answer is: logistics.

Deploying more buses onto the streets of a multi-million-person city doesn’t require much brainpower. But each individual passenger needs exactly one bus: the one that arrives immediately after they step off the previous one. Not an hour later, not a minute too early—within a three-to-five minute transfer window. As long as the bus schedule is built around minimized (and guaranteed) transfer times—we can speak of scaling. If bus A dumps its passengers at the terminal thirty seconds after connecting bus B has already pulled away—scaling has failed.

All right, to hell with Berlin—even with good public transport, living there was impossible. Let’s get back to Tobi.

Each Shopify user requires one persistent connection (a WebSocket) to one server. Adding new users is simply adding servers to the rack. If there are any scaling problems—they’re all clustered around the database, not Rails itself. Rails doesn’t care at all how many total servers are serving users: each server only needs to handle its own load.

This is not scaling. This is adding isolated capacity. It’s like if the zoom on your phone’s camera did not actually zoom in without unacceptable quality loss, but simply tiled more tiny copies of the image.

What the computing industry calls scaling is adding new hardware resources that increase the capacity of a connected node. Deploying more buses on new streets—no. Changing the schedule so that new buses coordinate with the existing ones—yes.

A chess server, for example, requires no scaling whatsoever. The server overheating?—Put another one next to it; in the end we’re serving disconnected pairs of players anyway. A server for the simultaneous slaughter of a billion sweaty nerds, on the other hand—that one needs scaling.

Let me briefly mention one example from personal practice before getting to the point. We process an incoming stream of currency exchange rates, perform some mathematical manipulations on them, and spit out peculiar results. Two hundred-plus currencies—that’s forty thousand pairs—with values arriving on average roughly once a second (in practice, more often). The mathematical manipulations can involve any number of currencies. All of this happens in real time. Because values must be available at any given moment, we cannot simply partition the pairs and split the streams across multiple servers. And a single server physically cannot handle it. Which means every node in the cluster must be able to exchange information with every other node—and the information is not static, so Redis won’t do. This is where the architecture needs the ability to scale horizontally.

The preamble has dragged on a bit. I hope I’ve at least clarified the terminology somewhat. So: if in the course of discussing an architecture you’ve concluded that the project will need genuine horizontal scaling—you cannot do without finite state machines. (In general it’s better to build all business logic on FSMs, but in a standalone system you can hobble along without them—in a cluster, there’s no way.) I rarely recommend reference material, but Introduction to the Theory of Computation by Michael Sipser is well worth a look. A finite state machine—for all its apparent simplicity—is something so powerful it genuinely astonishes.

Unfortunately, many believe that an FSM is simply a set of states—an extremely dangerous misconception that completely negates the entire body of formal mathematics upon which the power of finite state machines rests.

In any case, if you want to be ready to scale horizontally—build critical processes on finite state machines and make them fully asynchronous. If subsystem A must interact with subsystem B—forget about direct calls. In HTTP terms—201 is good, 200 is dreadful. Under no circumstances will you ever be able to later convert a request→response call sequence into request→acknowledgement→await response (without the destructive force of a complete refactor).

Asynchronous interactions built on top of FSMs, on the other hand, will make future scaling painless—because in this paradigm it makes absolutely no difference on which node the code handling a request actually runs.

There are languages in which this is easier (Elixir, Erlang), and those in which it’s harder. But in principle it is achievable in any environment. Once you get the hang of it, writing asynchronous code becomes no harder than synchronous. I speak from experience.

And finally, the perfect litmus test for determining whether your system is truly distributed or just a few servers standing in corners. If you’ve ever had to decide which letter from the CAP theorem set—‘C,’ ‘A,’ ‘P’—to sacrifice, you most likely have a genuine scalable cluster. If not—forget about horizontal scaling and simply add capacity as the business blooms.

And while I have the opportunity, I can’t resist recommending my own library Finitomata, which I prototyped in Idris and which is designed to be completely asynchronous (there is no way to determine from a response whether a state transition succeeded)—it provably prevents the programmer from violating a single law of finite state machine management.

Roastidio.us Tagged with elixir

From legacy code to verifiable specifications with Surveyor

What is legacy code?​

Legacy software is the code we lost confidence in​

"Can we rewrite this?"​

How we got here​

What we don't want​

What we want instead​

Organised and illustrated​

Job done! Or is it?​

1. Verifiability​

2. Reproducibility​

Runnable specs​

That magic is Assay​

The systemic approach​

Surveyor — discovering the architecture you inherited​

Examples​

The System Context of a Medical Application​

An example of an assay spec created in the second phase:​

Assay — proving the rewrite behaves the same​

How Assay actually works​

A two-pass parser​

The step macro generates real module functions​

Component-scoped step matching​

Pre-check, then execute​

Per-scenario state in the process dictionary​

How they fit together​

Four audiences, one artifact​

Where the LLM lives, and where it doesn't​

How you use them on a project​

Status and next steps​

Elixir: Introduction

Head of the Agents and Assistants Department

Scotty, I need warp speed in three minutes - Hauleth

Lesson: Flame graphs and call tracing is essential#

Lesson: Doing less can improve performance#

Lesson: Telemetry is not free#

Lesson: Records instead of maps or structs#

Lesson: ETS tables are super fast, but not always#

Lesson: Calling your GenServers is fast, but not 90k times per second fast#

Conclusions#

Post Scriptum: Good tooling helps a lot#

Resist Vendor Lock-In With Supabase

Elevate Your Elixir With Sigils

Motivation

Solution

Design Details

Implementation Details

Creation

Enumerable – Size / Count

Enumerable – Reduce

Enumerable – Membership

Inspect

Future Plans

Sign up for The Stack Canary

From Python to Elixir Machine Learning

Why is Python not Sufficient?

How to Proceed

Workflow Overview

Understand the Macro System

Read the Documentation

Read the Source Code in Detail

Implement the New Code

Example

Class vs. Behaviour

PyTorch to Nx

GEMM

Tree Traversal

Perfect Tree Traversal

Conclusion

Cure, Four Releases Deep: From FSMs to Furniture

v0.16.0: the turnstile that could not contain itself

v0.17.0: toward Idris, at last

v0.18.0: pattern matching grows up

v0.19.0: the furniture arrives

v0.19.1: Dialyzer and the small sins

What all this adds up to

SAFE: Bringing Real Static Analysis to the BEAM - Erlang Solutions

The BEAM is secure by default (Up to a point)

What is SAFE?

What is legacy code?

Legacy software is the code we lost confidence in

"Can we rewrite this?"

How we got here

What we don't want

What we want instead

Organised and illustrated

Job done! Or is it?

1. Verifiability

2. Reproducibility

Runnable specs

That magic is Assay

The systemic approach

Surveyor — discovering the architecture you inherited

Examples

The System Context of a Medical Application

An example of an assay spec created in the second phase:

Assay — proving the rewrite behaves the same

How Assay actually works

A two-pass parser

The step macro generates real module functions

Component-scoped step matching

Pre-check, then execute

Per-scenario state in the process dictionary

How they fit together

Four audiences, one artifact

Where the LLM lives, and where it doesn't

How you use them on a project

Status and next steps

Lesson: Calling your `GenServer`s is fast, but not 90k times per second fast#

The Bluesky dataplane

Fan-out to ETS dataplane

A peek into performance

Advanced simulations

Base data

Traffic simulation

Measuring performance