Roastidio.us Tagged with elixir

Objects, States, and Stewards: Introducing Generic Workflows to an Existing Project · YokerHood · Aleksei Matiushkin

Sat, 31 Oct 2026 12:35:03 +0000

The urge to automate business processes rarely strikes on day one of a greenfield project. On day one, you have two database tables, a signup form, and the blissful delusion that business logic will always fit inside three clean functions.

The urge arrives two years later.

By then, you have customer accounts, billing runs, support tickets, inventory shipments, and legal agreements. You also have half a dozen ad-hoc “processes” living as comments inside controller actions, stray background workers checking for boolean flags in cron loops, and angry messages in Slack asking why an invoice was refunded before anyone signed off on the return.

Management has a simple request: “We need workflows.”

At this junction, engineering teams usually make one of two catastrophic mistakes:

The Ad-Hoc Infection: They bolt a bespoke workflow_status column and a handful of conditional case statements onto every existing model. Within three sprints, five separate teams have implemented five incompatible ways to move records around, send emails, and check permissions.
The Heavyweight Invader: They purchase or deploy an enterprise BPMN orchestrator or microservice engine that demands every domain object be serialized into generic JSON payloads and piped through a foreign external runtime.

There is a third way: introducing a lean, generic, and agnostic workflow machinery directly into your existing project. It respects your existing domain models, enforces strict state transitions, classifies access through stewardship, notifies the right people without hardcoding recipient lists, and uses templates so that adding your twentieth workflow is as pleasant and predictable as adding your first.

Here is the blueprint.

The Four Pillars of Agnostic Machinery

A generic workflow engine does not need to know what an invoice, a pull request, or a customer refund actually is. It only needs to understand four orthogonal concepts:

flowchart LR
    Subject["Attached Object(s)\n(Domain Reality)"] --- Engine["Workflow Machinery\n(Agnostic Core)"]
    Topology["States & Transitions\n(Discrete Topology)"] --- Engine
    Stewardship["Stewardship\n(Classified Access)"] --- Engine
    TWIMC["TWIMC Notifications\n(Decoupled Broadcast)"] --- Engine

① Attached Objects: Holding the Leash, Not the Data

The biggest architectural blunder is letting the workflow engine consume or duplicate the domain model. An Invoice belongs in your billing context. A Contract belongs in your legal context.

The workflow instance merely attaches to the domain object via a polymorphic handle:

The domain object is the subject . The workflow instance is the governor .

The workflow holds the leash: it knows where the subject is in its lifecycle, but it does not dictate its database schema. If a process coordinates multiple objects—say, an Order and three Shipment records—the primary subject remains the anchor, while related entity IDs travel inside the workflow’s transient context payload.

② States in Between: Discrete Coordinates

A workflow is not a nebulous continuum; it is a finite graph. Between the moment a request is born and the moment it terminates, it occupies exactly one discrete state at any given second.

Transitions between states must be:

Explicit: Only declared edges in the graph can be traversed.
Atomic: The state change and any accompanying subject mutations succeed or fail together inside a single database transaction.
Validated: The engine rejects any event that does not match the current coordinate.

③ Stewardship: Contextual Authorization

Coarse-grained permissions (such as user.role == :admin ) collapse the moment real human workflows enter the picture. Who is allowed to approve an expense report? The author cannot approve their own report. The department head can approve up to $5,000. Legal must sign off if it touches international compliance.

We call this Stewardship : access classified and regulated by the actor’s relationship to the subject and the specific transition being attempted.

Under stewardship, an action is not authorized by a static global role, but by evaluating three variables:

The Actor: Who is knocking at the door?
The Transition: Which edge are they trying to cross?
The Subject Context: What is the actor’s mandate relative to this specific object in this specific state?

A transition declares its required stewardship policy: :author , :designated_reviewer , :finance_steward , or :system_automation . If the actor fails the stewardship check, the transition is rejected at the gate before any business code runs.

④ Notifications TWIMC: To Whom It May Concern

In dirty codebases, notification logic is smeared across the application layer:

Every new requirement adds another line of side-effect spaghetti. If one notification fails, does the transaction roll back? Who knows.

In generic workflow machinery, transitions emit a standardized event:

A dedicated TWIMC Dispatcher catches this event asynchronously. It looks up the notification rules declared for that transition, resolves the recipient list dynamically (e.g., the subject’s author, the current assignees, or an external webhook), and dispatches the alerts.

The transition logic remains pristine and focused entirely on state validation.

Workflow Templates: Guarding Against Chaos

If every developer is permitted to write ad-hoc state-machine plumbing, your system quickly degrades into an unmaintainable thicket of conflicting patterns.

The antidote is Workflow Templates : declarative specifications that define the entire contract of a process in one place.

A template specifies:

Target Subject: What type of domain object this workflow attaches to.
States & Transitions: The allowed lifecycle topology.
Stewardship Rules: Which actors can trigger each transition.
TWIMC Hooks: Who gets notified when a transition occurs.
Side-Effect Actions: Automated callbacks executed on successful transitions.

Here is what a declarative template looks like in practice:

defmodule MyApp.Workflows.Templates.ExpenseReport do
  use MyApp.Workflow.Template,
    subject: MyApp.Expenses.Report,
    initial: :draft

  # State topology
  transition :submit, from: :draft, to: :pending_review do
    stewardship :author_only
    notify :stewards, role: :department_manager
  end

  transition :approve, from: :pending_review, to: :approved do
    stewardship MyApp.Expenses.Stewardship.ReviewerPolicy
    action MyApp.Expenses.Actions.SchedulePayment
    notify :author
    notify :channel, target: "#finance-approvals"
  end

  transition :reject, from: :pending_review, to: :rejected do
    stewardship MyApp.Expenses.Stewardship.ReviewerPolicy
    notify :author, template: "expense_rejected"
  end

  transition :payout, from: :approved, to: :settled do
    stewardship :system_automation
    notify :author
  end
end

Notice what just happened:

No database migrations were needed to invent a new process.
The states, transitions, security rules, and communication policies are visible at a single glance.
The business logic for paying out or reviewing is cleanly isolated in modular action and policy modules.

The Execution Cycle

When an actor attempts to move a process forward, the generic engine executes a disciplined sequence:

sequenceDiagram
    autonumber
    actor User as Actor
    participant Engine as Workflow Engine
    participant Policy as Stewardship Policy
    participant Subject as Attached Object (DB)
    participant Bus as TWIMC Bus

    User->>Engine: advance(instance_id, :approve, actor, params)
    Engine->>Engine: Verify state allows event (:pending_review -> :approved)
    Engine->>Policy: check_stewardship(actor, transition, subject)
    alt Unauthorized
        Policy-->>Engine: {:error, :unauthorized}
        Engine-->>User: {:error, :forbidden}
    else Authorized
        Policy-->>Engine: :ok
        rect rgb(240, 245, 255)
            Note over Engine,Subject: Atomic Transaction
            Engine->>Subject: Apply transition side-effects / validations
            Engine->>Engine: Update current_state = :approved
        end
        Engine->>Bus: publish_transition(transition_payload)
        Bus-->>User: (Async) Notify Author, Stewards, Slack
        Engine-->>User: {:ok, updated_instance}
    end

Every transition follows the exact same pipeline:

Topology Check: Is the requested event permitted from the current state?
Stewardship Gate: Does the actor have the legitimate mandate to trigger this transition for this specific subject?
Atomic Execution: In a single transaction, execute the subject hook and update the workflow instance record (with optimistic locking to prevent race conditions).
TWIMC Broadcast: Asynchronously publish the transition event to notify anyone who needs to know.

Bringing It into an Existing Codebase

You don’t need to rebuild your application to introduce this machinery. The migration path is non-invasive:

Add Two Tables: You need workflow_instances (to track state, template name, and polymorphic subject IDs) and an optional workflow_logs table for audit trails. Your existing business tables don’t need to be redesigned.
Build the Registry: Create a registry that maps template identifiers (e.g. "expense_report" ) to their compiled template modules.
Wrap, Don’t Rewrite: Pick one painful existing process—such as order returns or invoice approvals. Write a template for it. When an invoice is created, attach a new workflow instance to it.
Rely on the Harness: Delegate all state progression, permission verification, and notifications to WorkflowEngine.advance/4 .

Once the machinery is in place, creating a new workflow ceases to be a week-long ordeal of bespoke schema migrations and spaghetti event handlers. You define a template, attach your object, and let the machinery do the heavy lifting.

Clear borders, classified stewardship, and automated communication. That is how you turn chaotic business requirements into predictable, pleasant engineering.

Trouble Is My Workflow · YokerHood · Aleksei Matiushkin

Sat, 31 Oct 2026 12:35:03 +0000

The rain was hitting the asphalt outside with the dull, persistent rhythm of an unhandled exception in an infinite loop. It was past eight, the coffee in my mug was cold enough to preserve fish, and the only light in the office came from three flickering monitors and an incandescent exit sign that buzzed like an angry hornet in a tin can. It was 1998.

That was when the architect walked in.

You know the type. Rimless glasses, an ironed flannel shirt that had never seen manual labour, and the kind of earnest, predatory smile usually reserved for people selling whole-life insurance or enterprise service buses. He dropped a four-inch design document onto my desk. It landed with the sullen thud of a wet phonebook.

“We need a generic workflow engine,” he announced, resting two manicured hands on the edge of my desk. “Something dynamic. Decoupled. Highly extensible. A unified platform to orchestrate our business processes across distributed microservices.”

I looked at him through the steam of the fresh cup he hadn't offered to buy me.

“You don’t need an engine,” I said. “You need an honest finite-state machine, and you’re too fancy-pants to admit it.”

He flinched as if I’d insulted his mother, adjusted his glasses, and began to mumble about directed acyclic graphs. I lit a cigarette (these blessed years when you could smoke in the office,) pointed at the door, and waited for him to leave before I opened the file.

The Autopsy of a Buzzword

In our trade, “workflow” is the word people use when they want to charge six figures for a product or excuse three quarters of architectural delinquency. It sounds respectable. It sounds like something an executive can nod at during a quarterly business review without having to know what a pointer does.

Strip off the sales brochure, wipe the mud off the radiator grille, and look at the chassis underneath. What is a generic workflow, really, when nobody is trying to sell you a cloud subscription?

At its bare brass tacks, a workflow is nothing more than five distinct elements cohabiting in a trench coat:

A Finite Set of States. Your process is always somewhere . Not everywhere at once, not halfway between two ideas like an indecisive juror, but occupying exactly one discrete coordinate in its lifecycle.
A Discrete Set of Events. Things happen. An alarm goes off, an HTTP webhook knocks on the door at two in the morning, a human reviewer clicks a green button, or a payment processor shrugs and gives up.
A Transition Topology. The rules of the alley. If you are standing in State A, there are only specific doors unlocked. You may step through Door 1 to State B, or Door 2 to State C. You cannot phase through the brick wall into State Q just because the product manager had a sudden vision in the shower.
A Context Payload. The baggage the traveler carries. The pull request number, the customer’s delivery address, the current temperature in Minneapolis. The luggage travels with the workflow, but it does not drive the car.
Invariants Governed Over Time. Rules that do not bend, no matter how hard the wind blows or how loud the VP of Sales screams into the receiver.

If that list sounds familiar, it should. That is the formal definition of an extended Finite State Automaton. It was worked out back when computers were the size of municipal water heaters and programmers wore ties to work.

Yet, for reasons that keep therapists and cloud vendors in business, the modern software industry spends billions of dollars pretending this identity does not exist.

How We Murder State in the Dark

The average engineering team approaches a workflow the way a cheap criminal approaches a bank vault: without a blueprint, armed with a crowbar, and hoping nobody asks hard questions until the money is spent.

There are three classic ways this crime is committed in modern codebases.

① The Polite Script (The Straight-Line Delusion)

The engineer begins with a pipeline. In Elixir, they write:

It looks elegant. It reads like poetry written by a civil servant. And it is completely divorced from the physical universe.

In the real world, the credit card is maxed out. In the real world, the warehouse is on fire. In the real world, the customer changed their mind five minutes after clicking the button and wants their money back while the parcel is sitting on a loading dock in New Jersey.

The moment reality kicks the door down, the clean pipe fractures into a sprawling tangle of nested case statements, early returns, and rescue blocks. What was supposed to be a straight highway becomes a snake pit.

② The Database Stew (Status-Driven Denial)

When the script breaks, the team “architects” a database solution. They take a PostgreSQL table and give it a status column.

Then they realize an order can be paid for, but pending fraud review. So they add fraud_review_pending: boolean . Then someone discovers an order can be partially refunded while in transit, so they add partially_refunded: boolean and transit_hold: boolean .

Within six months, the schema looks like an uninspected electrical switchboard:

schema "workflows" do
  field :status, :string, default: "pending"
  field :paid, :boolean, default: false
  field :reviewed, :boolean, default: false
  field :shipped, :boolean, default: false
  field :cancelled, :boolean, default: false
  field :refunded, :boolean, default: false
  field :on_hold, :boolean, default: false
end

Count the representable combinations. Seven statuses multiplied by six boolean flags gives you hundreds of distinct states your database will swallow without blinking.

Of those hundreds, perhaps six correspond to something that could happen to a physical object without breaking the laws of thermodynamics. The other hundreds are ghosts: orders that are simultaneously shipped: true , paid: false , and cancelled: true . Rows like that sit in your database like counterfeit bills in a cash register—silent, illegal, and waiting to ruin someone’s morning when the audit comes due.

③ The Amnesiac Message Queue

The third approach is the microservice special. The team splits the workflow into six workers listening to a message bus.

Worker A does a job, writes to the database, and emits an event. Worker B reads the event, reads the database, checks five conditional flags, does another job, and emits another event.

Nobody has a map of the process. The entire lifecycle is smeared across twenty-four Git repositories like grease on a diner counter. To answer a simple question— “What happens if the customer cancels while we are waiting for the courier?” —you need to interview four engineers, read twelve YAML files, and consult an astrologer.

There is a better way. In fact, there has always been a better way. Let’s look at three real-life cases from the desk blotter.

Case File: The Task Lifecycle (The Finite Trudge)

Consider the most familiar trudge in our corner of the city: moving a feature from an engineer’s brain to production.

The PM thinks it’s a line: state task → code → test → commit → review → merge.

Anyone who has survived a single sprint knows that line is a lie. Real engineering is full of hairpin turns and backward loops.

Here is what the lifecycle actually looks like when you shine a hard light on it:

stateDiagram-v2
    [*] --> Stated: define_requirements
    Stated --> InDevelopment: pick_up
    InDevelopment --> Committed: run_tests_and_commit
    Committed --> UnderReview: open_pull_request
    UnderReview --> InDevelopment: request_changes
    UnderReview --> Approved: lgtm
    Approved --> Merged: merge_to_trunk
    Merged --> [*]
    
    Stated --> Abandoned: discard
    InDevelopment --> Abandoned: discard
    UnderReview --> Abandoned: discard
    Approved --> Abandoned: discard
    Abandoned --> [*]

Notice what is happening here:

The Review Loop is First-Class. When the senior engineer leaves twenty-seven comments asking why you didn't use Enum.reduce_while/3 , you don’t crash the pipeline or manufacture an error state. You transition from UnderReview back to InDevelopment on the event request_changes . It is an expected, legitimate transition.
The Eject Button Exists Everywhere. A task can be discarded from almost any stage. Marketing can cancel the initiative while you are halfway through writing the tests. If you model this with explicit states, Abandoned is a clean terminal state that frees resources and logs the autopsy. If you model it with flags, you end up with orphaned branches and pull requests that sit open until the heat death of the universe.
Illegal Moves Are Inconceivable. Can you merge a task directly from InDevelopment without an approval? Not unless there is an arrow connecting them. The machine does not ask you to remember the policy; the topology is the policy.

Case File: Order Fulfillment (The Transactional Gauntlet)

Now take commerce. An order is placed, money changes hands, goods move across the map.

stateDiagram-v2
    [*] --> Placed: checkout
    Placed --> Authorized: authorize_payment
    Placed --> Failed: payment_declined
    
    Authorized --> Packing: claim_inventory
    Authorized --> Cancelled: cancel_order
    
    Packing --> Shipped: dispatch_courier
    Packing --> Cancelled: inventory_shortage
    
    Shipped --> Delivered: confirm_receipt
    Shipped --> LostInTransit: carrier_lost
    
    Delivered --> Refunded: return_items
    Cancelled --> Refunded: refund_authorization
    
    Failed --> [*]
    LostInTransit --> [*]
    Refunded --> [*]
    Delivered --> [*]

Look closely at the branches:

You cannot refund what was never authorized. In the old boolean stew, a bug in the webhook consumer could mark an order as refunded: true even if paid: false . In an FSM, :refunded can only be reached from states where money was actually held or captured ( Authorized , Delivered , Cancelled ).
Cancellation semantics change based on state. If the customer cancels while in Placed , it costs nothing. If they cancel while in Packing , the warehouse has to restock the shelf. If they cancel while in Shipped , you can't cancel at all—they have to wait for delivery and initiate a return. An FSM makes these rules self-evident because each state only listens to the events that make sense for its location.
Dead ends are accounted for. Packages get lost ( LostInTransit ). Cards get declined ( Failed ). These are not unhandled exceptions; they are terminal states.

Case File: The Eternal Beat Cop (The Infinite Loop)

Some people think workflows have to terminate—that if a process doesn't reach an end state, it’s defective, like a watch with no hands.

They haven't spent enough time watching the street. Half the machinery keeping the modern world turning runs in an infinite, unblinking loop: kiosks, telemetry collectors, sensor monitors, background pollers.

Take a classic: receive a weather forecast → show it → wait fifteen minutes → repeat.

stateDiagram-v2
    [*] --> Fetching: init
    Fetching --> Displaying: forecast_received
    Fetching --> Backoff: network_error
    
    Backoff --> Fetching: retry_timer
    
    Displaying --> Waiting: render_complete
    Waiting --> Fetching: refresh_timer

A novice looks at that loop and gets nervous. “Won’t the stack overflow? Won’t the memory swell up like a drowned corpse?”

In a traditional imperative language with recursive calls or unchecked event listeners, quite possibly. But in an honest, process-backed finite-state machine (such as an OTP GenServer running under Finitomata ):

The Stack Never Grows. State transitions are tail-recursive loops or GenServer callback returns ( {:ok, :displaying, payload} ). The machine sheds its past with every step. It can cycle two billion times without consuming an additional byte of RAM.
Time Is Just Another Event. Notice refresh_timer and retry_timer . Time is not a rogue cron job breaking in through the fire escape and mutating rows in your database behind your back. Time is an internal event generated by the machine itself.
Self-Healing Is Built-In. When the weather API returns a 503 Gateway Timeout, the machine doesn't throw a fit and die; it slips into the Backoff state, sets a short timer, and tries again.

It is the eternal beat cop walking the neighborhood: check the door, see it's locked, wait under the streetlamp, walk the block again. Rain or shine, day or night.

The Detective’s Rules: How to Model a Workflow Properly

If you want your workflows to hold up when the district attorney starts asking questions, you need discipline. Here are the rules I live by:

① The Blueprint Precedes the Brick

Before you write a single line of business logic, declare the topology. If you cannot draw your workflow on a napkin using simple state-and-event syntax, you do not understand the business problem.

Using Finitomata , the declaration is not an afterthought or an export—it is the source code itself:

defmodule WeatherStation.FSM do
  @fsm """
  fetching --> |forecast_received| displaying
  fetching --> |network_error| backoff
  backoff --> |retry_timer| fetching
  displaying --> |render_complete| waiting
  waiting --> |refresh_timer| fetching
  """

  use Finitomata, fsm: @fsm, timer: 60_000
end

The string above is compiled directly into a validated state machine. If an event arrives that isn't declared for the current state, it doesn't corrupt your data; it gets rejected with an explicit error.

② State Is the Coordinate; Payload Is the Luggage

Keep your states small, finite, and enumerable. A state is an atom: :idle , :under_review , :waiting .

The data associated with that state—the reviewer’s notes, the temperature reading, the authorization token—belongs strictly in the payload .

Never, under any circumstances, allow data to disguise itself as a state:

State determines what can happen next . Payload provides the context for how it happens . Conflating the two is the root of all evil.

③ Events Knock; They Do Not Dictate

External actors do not change your state. An HTTP request does not get to say order.set_state(:shipped) .

External actors emit events . The state machine receives the event, verifies that the transition is legal from its current coordinate, executes the business callback, and moves itself.

If the shipping department sends a :dispatch event for an order that is still :placed (unpaid), the machine laughs in their face, logs the attempt, and stays right where it was.

④ Failure Is a Destination, Not a Panic

In cheap dime-store novels, people get hit over the head, lose consciousness, and wake up fine twenty minutes later. In production systems, exceptions don't work that way. An unhandled exception kills the thread, corrupts state, and leaves the on-call engineer staring at a pile of ash.

Every state that touches an external boundary—a network call, a disk write, a human approval—must account for failure as a first-class transition. Model the failure states: :payment_declined , :review_rejected , :backoff . Give failure a roof and a bed, and it won't burn your house down.

⑤ One Process, One Life

If you’re running on the BEAM, every active workflow instance should be its own isolated process.

Order #1042 gets a GenServer. Pull Request #88 gets a GenServer. Weather Station #4 gets a GenServer.

When an entity is backed by an isolated process:

Incoming events are naturally serialized in the process mailbox. The concurrency race condition—where two web requests simultaneously attempt to modify the same record—vanishes by design.
If one workflow instance encounters a catastrophic error, it crashes alone, supervised, leaving the other fifty thousand running instances untouched.
State lives in fast, in-memory Erlang terms, backed to disk only at transition boundaries.

Epilogue: The Long Goodbye to Spaghetti

The rain had stopped by three in the morning. The streets of the city were black mirrors reflecting the streetlights.

The four-inch design document from the architect was still on my desk, right next to the empty coffee mug. I opened it to page one, took out a red marker, and struck through thirty-seven pages of “dynamic plugin orchestrators” and “configurable JSON execution pipelines.”

In their place, I drew six circles, eight arrows, and a dozen clear labels.

A workflow is not a mystical creature requiring enterprise middleware. It is a finite automaton in a damp trench coat, waiting for you to stop lying to yourself.

Give it a name. Draw its borders. Lock the doors that shouldn't be opened. You’ll sleep better at night, and when trouble comes knocking at your door—as it always does—you’ll know exactly which state you’re in.

Aggressive Regression · YokerHood · Aleksei Matiushkin

Sat, 31 Oct 2026 12:35:03 +0000

There is a particular species of engineer who, upon being asked whether the codebase has regression tests, answers “we have ninety-four percent coverage.” This is roughly equivalent to answering the question “is your house insured against fire?” with “I own eleven smoke detectors, and I polish them weekly.” The two statements inhabit the same general neighbourhood of concern, but they are not even distantly related, and only one of them will matter on the night the kitchen goes up.

Let me state the thesis up front, in the hope that it survives the journey to the comment section intact:

Each regression test is a test, but not each test is a regression test.

This is not a clever aphorism. It is a statement about sets, and it is the sort of statement that most teams nod along to and then immediately violate for the next four years.

The Taxonomy Nobody Bothered to Learn

A test is any piece of code that fails when something is wrong. A regression test is a piece of code that fails when something that used to work stops working. The second category is a strict subset of the first, and it is a far smaller subset than your coverage badge implies.

The distinction lives entirely in the word “used.” A regression test is a memorial. It commemorates a specific moment in history when the software was observably correct, and it stands guard over that moment for as long as the project draws breath. It is, in the most literal sense, institutional memory encoded as an executable artifact—which makes it the only kind of memory your team has that does not resign, retire, or forget everything over a long weekend.

Most of what we write is not that. Most of what we write is a developer reassuring himself, at two in the afternoon, that the function he just typed does what he just thought it should do while typing it. That is a perfectly respectable activity. It is also a tautology wearing a lab coat.

Unit Tests Are Mostly Mirrors

Here is the uncomfortable thing about the average unit test: it asserts that the implementation does what the implementation does. The author wrote the function, then wrote a test derived from the function he had just written, and then expressed surprise when both agreed. The two of them were never going to disagree. They were conceived in the same five minutes, by the same brain, operating under the same misconception. Even a mediocre developer like myself is able to wrote a simple function without a glitch.

A unit test typically reaches inside. It knows the module’s private helpers, the shape of its intermediate accumulator, the exact order in which it walks a list. It is intimate with details that no caller will ever observe, and it is therefore brittle in exactly the places where you most want freedom to move. Refactor the accumulator from a list to a map, change nothing whatsoever about the observable behaviour, and watch forty tests go red while the software remains flawless. That is not a safety net. That is a tripwire you installed in your own hallway.

And the whole edifice rests on the author’s imagination. The unit test checks the three inputs he thought of. The bug, with the unerring instinct of a cat finding the one guest in the room who is allergic, lives in the fourth.

This is why a single property-based test is worth a bushel of hand-typed examples. Instead of enumerating what you happened to think of, you declare the law the function must obey, and then let the machine hunt for the counterexample with a diligence no human possesses at four in the afternoon on a Friday.

Consider the usual hand-rolled approach:

Three assertions, all of them chosen by a human being who already knew how the encoder worked. Compare:

One statement, infinitely many assertions, and—this is the part people underestimate—an automatic shrinker that will hand you the minimal failing input rather than the 4 KiB blob that happened to trip it. The first version tests your memory. The second tests your software.

The law, incidentally, is the valuable artifact here. decode(encode(x)) == x is a sentence about the system that remains true across three rewrites, two language versions, and the departure of everyone who originally wrote it. Your assert "abc" == ... is true about one string.

Round-trips, idempotency, commutativity, invariants, monotonicity, order-independence. If you cannot name a single law your module obeys, that is not an argument against property-based testing. That is a code review finding about the module.

Elixir Will Not Let You Test Private Functions, and This Is a Feature

Every few months, someone arrives in the forums with a question phrased as a bug report: “How do I test a defp ?” The replies are patient and the asker is rarely satisfied, because the honest answer sounds like obstruction: you don’t, and you shouldn’t want to.

A private function is, by construction, not part of your contract. It is scaffolding. It exists because you got tired of a forty-line function and cut it in half, or because you needed the same three lines twice. Nobody outside the module can call it, nobody outside the module can observe it, and nobody outside the module can be harmed by its replacement. It has no behaviour in the sense that matters—it has only implementation, and implementation is precisely the thing we have just agreed not to test.

The language is not being coy. It is telling you something: if you feel a burning need to test a private function in isolation, you have discovered a design pressure, not a tooling gap. One of three things is true.

It should be public. The thing you are trying to pin down is actually part of the module’s contract, and you have been hiding your own API from yourself out of some misplaced sense of tidiness. Make it public, document it, test it as the public thing it always was.

It should be its own module. The private helper has grown a personality: its own vocabulary, its own invariants, its own reasons to change. That is a module trying to be born. Deliver it. Now it has a public interface, and testing it is no longer a philosophical crisis.

It should not be tested at all. It is three lines of plumbing between two public functions, both of which are tested, and both of which would scream if the plumbing broke. Leave it alone. Every test you write is a line of code you must maintain forever; spending that budget on a private two-liner is the testing equivalent of insuring your shoelaces.

Naturally, someone always discovers that Module.function can be reached through the back door with apply/3 and a little dishonesty about module attributes, or that one can simply @compile :export_all and get on with their life. Yes. You can also remove the guard rail from the balcony because it was obstructing your view. The guard rail was not a design flaw.

There is a second, subtler gift hidden here. Because the language makes implementation untestable, it quietly pushes the lazy developer—and we are all the lazy developer on Thursday afternoons—toward testing behaviour, which is the only thing worth testing. Elixir has encoded good taste into a compiler restriction. Few languages have the nerve.

Behaviour Is a Virtue, Implementation Is a Vice

Here is the mechanical reason behavioural tests catch regressions and implementation tests do not.

A regression happens when someone changes the implementation and accidentally changes the behaviour. That is the entire anatomy of the bug. Now ask yourself what happens to each kind of test under that event.

The implementation test was written against the old implementation. The developer changed the implementation, so the test broke—but it broke because the implementation changed, not because anything is wrong. The developer, who is reasonable and in a hurry and has seen this test cry wolf nine times this month, updates the test to match the new implementation. The test goes green. The bug ships. The test participated enthusiastically in its own defeat.

The behavioural test was written against the contract, which nobody intended to change. It breaks, and there is no plausible way to “update” it, because updating it would mean openly declaring that the software now does something different. That is a conversation, not a quick fix. The bug does not ship.

This is why a proper regression test has a very particular pedigree. It is not written during feature development by a cheerful developer. It is written in the aftermath of an incident, by someone slightly grim, and it reads like a deposition:

describe "issue #1472: timezone-aware expiry" do
  test "a token minted at 23:59 UTC is not considered expired at 00:01 local" do
    # Reported 2026-04-02 by a customer in UTC+2 whose entire
    # team was logged out overnight, every night, for nine days.
    token = mint_token(at: ~U[2026-04-02 23:59:00Z], ttl: {2, :hour})
    assert {:ok, _claims} = verify(token, now: ~U[2026-04-03 00:01:00Z])
  end
end

Note what the test knows and what it does not. It knows the symptom, the date, and the victim. It does not know whether expiry is computed with DateTime.diff/3 , a cached monotonic offset, or a sundial. You may rewrite the entire expiry subsystem tomorrow; this test will follow you there without complaint, and the day it goes red, something real has broken.

Note also the comment. A regression test without the story of its birth is a landmine for the next person, who will eventually look at a strange assertion, fail to imagine why anyone would write it, and delete it. Every regression test should carry its own obituary of the bug it buried. The cheapest documentation in your repository is the kind that fails the build when it becomes false.

The rule, stated without hedging: every bug fix is accompanied by a test that fails before the fix and passes after it. If you cannot write a failing test, you have not understood the bug—you have merely disturbed it, and it will be back.

And Then the Machines Arrived

Everything above has been true since roughly 1975. What changed is the volume and the provenance of the code flowing into the repository.

Generated code is not worse than human code in the way the pessimists claim. It is worse in a far more specific and dangerous way: it is locally plausible and globally unanchored . A model producing a refactor has no memory of the incident in April. It has never been paged at three in the morning. It does not know that the retry loop has a weird +1 in it because a vendor’s API counts from one, nor that the normalisation step strips a particular Unicode character because of a bug in a client shipped four years ago that six hundred customers still run. All of that knowledge lives in exactly two places: in the heads of people who will eventually leave, and in your regression suite.

When a human engineer stumbles across a strange-looking line, he hesitates. He runs git blame . He asks in the channel. A model does not hesitate—it cannot, because hesitation requires knowing what you do not know, and a statistical amplifier is constitutionally incapable of that particular form of humility. It sees an oddity, recognises it as deviation from the idiom it was trained on, and tidies it away with the serene confidence of a cleaner throwing out an artist’s installation because it looked like a pile of rubbish.

Your regression suite is the only thing in the building that can say no to that. Not the linter, which thinks the tidied version is objectively nicer. Not the type checker, which agrees the types line up. Not the reviewer, who is looking at a diff of four hundred lines with a meeting in six minutes. Only the test that was written by someone who was there in April.

Which inverts the economics in a way worth stating plainly. The cost of writing code has collapsed. The cost of verifying code has not moved an inch, because verification is where the domain knowledge lives, and the domain knowledge was never in the keystrokes. So the ratio shifts. In 2015, a team producing ten thousand lines a year could afford a casual relationship with its test suite. A team now producing ten thousand lines a month cannot—not because the code is worse, but because no human being is reading all of it any more, and the suite has quietly been promoted from a safety net to the primary reader.

Two corollaries, both of which people hate.

Never let the model write the regression test from the fixed code. The whole point of the test is that it was derived from the symptom , independently of the implementation. Hand the model a patched function and ask for tests, and you will get an exquisitely thorough description of the patch—a mirror again, just a faster and better-spelled one. Write the failing test first, from the bug report, in your own words. Then let the machine fix the code until the test goes green. That direction is enormously productive. The other direction is theatre.

Be very suspicious of a green suite after a large generated refactor. Either your suite is genuinely behavioural, in which case congratulations and go to lunch; or your suite is thin enough that nothing in it was ever going to notice. The way to tell the two apart is to break something on purpose and check that the suite screams. If you want this done systematically rather than by intuition, that is what mutation testing is for: it is a test suite for your test suite, and the first run is always a humbling experience. Try muex btw, it shines.

Further Specimens from the Wild

Since the invitation was open, some additional patterns that live on the correct side of the line—and some impostors that do not.

Characterisation tests for code nobody understands. You inherit forty thousand lines with no tests and no author. You cannot write behavioural tests, because nobody knows the intended behaviour. So you write down the actual behaviour—feed it real inputs, record the real outputs, assert they stay that way. The tests are not asserting correctness; they are asserting stability , which is the only thing you can honestly assert about a system you do not understand. This is regression testing in its purest form, and it is the standard entry point for making a legacy system safe to touch. Mark these tests clearly, because some of them are faithfully memorialising bugs, and one day somebody will fix one and be baffled by the red.

Doctests as behavioural contracts. Elixir’s doctest is a quietly radical idea: the example in your documentation is the test. Documentation lies constantly, in every project, in every language—except here, where a lie fails the build. And because documentation examples are written from the caller’s perspective by definition, they are behavioural by construction. You cannot document a private function’s intermediate accumulator in a way a user would find useful, so you do not, so you do not test it. The medium enforces the message.

Contract tests at service boundaries. The regression most likely to ruin your quarter is not in your code at all; it is in the thing on the other end of the wire, which changed a field from a string to an integer on a Tuesday without telling anyone. A contract test pinned against a recorded, versioned interaction catches this on your CI run rather than in production. Behaviour at the seam, which is exactly where behaviour is hardest to reason about and most expensive to get wrong.

Mox and explicit contracts, as opposed to mocks as a noun. Mocking a private collaborator so you can assert it was called twice is implementation testing with extra ceremony and a worse error message. Defining a behaviour, implementing it twice—once for real, once for tests—and asserting on the interaction contract is behavioural testing with a swappable backend. The first makes refactoring harder; the second makes it possible. Same library, opposite outcomes, and the difference is entirely in whether the thing you mocked is a contract or an accident.

The flaky test is a regression test for your concurrency, and you are ignoring it. Every team has the one test that fails on CI roughly once a fortnight, and every team has a @tag :skip or a retry wrapper over it. That test is not broken. That test is the only member of your suite that has noticed the race condition, and you have gagged it. Nondeterminism in tests is nearly always a true positive about nondeterminism in the system.

Time, locale, and encoding: the three horsemen. A startling proportion of all regressions ever recorded come from a date crossing a boundary, a locale formatting a number differently, or a string containing a character the author did not personally know existed. These are behavioural by nature and almost never covered by hand-written unit tests, because the author’s imagination is bounded by the author’s timezone, locale, and alphabet. Property-based generators do not share those limitations, which is why StreamData finds the emoji in the username field within about nine runs.

Golden files, used correctly. Snapshot testing has a bad reputation, entirely earned by teams who run “update all snapshots” as a reflex and commit the result unread. A golden file reviewed with the same seriousness as source code is an excellent regression test for anything with a complex output shape—rendered templates, generated schemas, serialised payloads. A golden file updated by muscle memory is a git add . with extra steps.

Performance budgets. A function that returns the correct answer in nine seconds, having previously taken forty milliseconds, has regressed—the behaviour of a system includes how long it takes to behave. A crude assertion on an upper bound, or a benchmark gated in CI, catches the accidental Enum.map over a database query that someone slipped into a loop. Nobody writes these until the first outage, which is a pity, because they are cheap.

The test you delete. Last and most heretical: a test that has never failed for a good reason in three years is not protecting you. It is a line item in your maintenance budget, a drag on your CI time, and a small daily tax on everyone’s attention. Regression suites should grow by accretion from real incidents, not by a coverage quota imposed from above. Coverage measures which lines were executed, not which behaviours were verified—a suite that calls every function and asserts nothing scores a perfect hundred, as does one that asserts everything and understands nothing.

The Short Version

Tests are cheap and plentiful; regression tests are expensive and rare, because each one costs an actual outage to acquire. Treat them accordingly. Write them from the bug report rather than from the fix. Aim them at the contract rather than the plumbing. Let the compiler stop you from testing private functions, and thank it quietly for the discipline. Prefer the law to the example, and let the generator find the counterexample you were never going to think of.

And now that an increasing share of your codebase is arriving from a source with no memory of the last five years, understand what your suite has become. It is not a quality gate any more. It is the only institutional memory you have that does not sleep, does not leave, and cannot be talked out of its position by a confident paragraph of plausible reasoning.

Be aggressive about it. The regressions certainly will be.

Previously, on the subject of not deceiving yourself with your own tooling:

The Engine Does Not Win the Race · YokerHood · Aleksei Matiushkin

Sat, 31 Oct 2026 12:35:03 +0000

Somewhere, right now, a senior engineer is on the verge of a nervous breakdown because his company will not let him switch from Claude to GPT-5.2, or from GPT to Gemini, or from any of them to a locally-hosted llama that smells faintly of the intern’s vape juice. He is convinced that the model is the product, that the difference between shipping and slop is measured in the number of parameters, and that if only management would approve the more expensive API key, all his problems would evaporate like a hallucinated citation.

He is wrong. Spectacularly, expensively, and in a way that will cost him exactly the difference between the two API tiers he is agonising over.

Allow me to explain, in the manner of a man who has spent the last year building the thing he is about to tell you matters more than the model itself.

The Engine Is a Fish With Amnesia

Let us first agree on what a language model actually is, because ninety percent of the industry’s agonising proceeds from a category error. A language model is a finite state machine with the memory span of a goldfish that has been through electroconvulsive therapy. Its entire operation collapses into the unglamorous sequence:

start → context → query → response → stop

There are no sessions. There is no memory. Every single request you make begins from a blank slate, and the model dutifully autocompletes whatever you have managed to cram into the context window in that one trembling moment. The model does not know your codebase, your conventions, your catastrophic past decisions, or even that it spoke to you thirty seconds ago. It knows only the distillate you present it with—and, being a statistical amplifier of whatever it is fed, it will faithfully amplify that distillate into either gold or guano.

Here is the uncomfortable corollary that nobody in the model-comparison industry wants to hear: the model is the least interesting component of the entire stack. The interesting component is the thing that decides what goes into that context window, in what order, at what cost, and with what guard-rails around the output. That thing is the harness. And the harness is where the entire difference between “Dull ∧ LLM = Dull” and “Wise ∧ LLM = Wise²” actually lives.

I wrote that truth table once, in a fit of optimism, and immediately got accused of supporting both Trump and Putin by people whose reading comprehension caps out at three words per paragraph. So let me be plainer this time. The model is the engine. Nobody drives an engine. You drive a car—and the car is the harness.

Nobody Buys a Car for the Crankshaft

Any fool can bolt a thousand-horsepower engine into a shopping trolley. It will do a spectacular burnout for exactly one second before the steering shears off, the brakes fade to nothing, and the whole contraption wraps itself around a lamppost with the shopping basket still full of the groceries it was never going to deliver. Horsepower is the number that sells the brochure; the transmission, the suspension, the brakes, and the fuel injection are what decide whether you actually arrive anywhere.

The industry has this exactly backwards. We have spent three years obsessing over which engine to buy while ignoring the fact that most of these engines are being bolted into shopping trolleys—single-threaded Node.js scripts that crash the instant a tool call returns an unexpected byte, wipe the entire conversation, and force you to re-explain your whole codebase to a model that has already forgotten you exist.

And here is the cruelest part: the engine does not even know it is in a shopping trolley. It cannot feel the lamppost approaching. It will cheerfully keep generating code until the trolley is a smoking ruin, because generating code is all it has ever been trained to do. The harness is the only component with both the information and the authority to say “no.”

So what does a proper harness actually do? Two things, and only two things, and both of them are worth more than any model upgrade you are currently begging your CTO for. It decides what the model sees. And it decides what the model is allowed to do with what it sees. Everything else is decoration.

What the Model Sees: The Miser’s Art of Context

The single most expensive resource in this entire economy is not compute, not GPUs, not even the API bill—it is the context window, and the tokens you waste filling it with noise. Every model has a hard ceiling on how much it can attend to before quality collapses into a puddle of generic autocomplete, and that ceiling is a great deal lower than the marketing department would have you believe. Nobody can actually process a hundred thousand tokens without noticeable quality loss. The ones who claim otherwise are selling subscriptions.

This is why the R in RAG matters hundreds of times more than the model itself, and why every serious player has been quietly burrowing into your computer with an IDE or a CLI. Shipping the entire codebase to the server on every request is not just expensive—it is actively harmful, because it drowns the signal in noise. A proper harness does not ship the world. It ships a distillate : precisely the files, symbols, and idioms the current task requires, nothing more, nothing less.

Let me be concrete, because I built the thing I am about to describe, and I would rather show you the plumbing than wave at the facade. My slowly-simmering harness, Yoke ( yoke ) , spends an almost pathological amount of effort on not wasting your tokens:

▸ Selective context expansion. You type @lib/auth.ex or @error or even the lazy phrase “error above,” and the harness expands only that reference into the prompt—fetching the file, the URL, or the relevant error block it has been tracking as open versus resolved. It does not dump your entire repository into the context window on the off-chance that something in there might be relevant. It fetches the one thing you pointed at, and it keeps a running ledger of which issues are still open so that when you say “fix it,” it knows what “it” refers to without you re-pasting the stack trace for the fourth time.

▸ AST instead of raw text. This is the hill I will die on. Raw source code is a terrible format for a context window—verbose, redundant, and full of formatting noise that contributes nothing to understanding. An abstract syntax tree is structured, compact, and far more legible to a model that was trained on code. My Ragex integration lets the harness search and present code as structure rather than as a wall of text. Same understanding, a fraction of the tokens. This is not a micro-optimisation; it is the difference between being able to fit a real codebase in the window and being able to fit a single file.

▸ Compression on demand. When a session has been running long enough that the context window is groaning under its own history, /compact hands the conversation to the model and asks it to produce a condensed summary preserving the core goal, the files touched, the decisions made, and the next steps. The verbose transcript is then replaced by that summary. The model keeps its memory of what matters without you paying to retransmit every failed attempt and every “no, I meant the other thing.”

▸ The miser’s dashboard. /tokens and /cost show you, per turn and cumulatively, exactly what you are spending. You cannot optimise what you cannot measure, and the token gauge on the status bar is the financial equivalent of a fuel gauge on a car that everybody else is driving blind.

▸ Sessions that survive death. Every conversation is persisted to disk as a resumable narrative, so if you walk away for the night—or the harness crashes, or the cat walks across the keyboard and closes the terminal—you resume with yoke -c exactly where you left off. This sounds like a convenience feature. It is actually a token-saving feature of the first order, because the most expensive tokens in any project are the ones you spend re-explaining your codebase to a model that forgot it overnight.

And, most importantly, the harness is the thing that decides not to send anything at all . When it detects that the model is about to call the exact same tool with the exact same arguments it already called in the previous turn—the tell-tale heartbeat of an agent stuck in a loop, burning tokens in a circle—it refuses to execute, injects a notice telling the model to synthesise its answer from the results it already has, and moves on. The cheapest tokens are the ones you never spend. A harness that can say “no” is a harness that saves you money on every single session.

What the Model Is Allowed to Do: The Suspension and the Brakes

But a harness that merely feeds the model well is only half a car. The other half is the part that keeps the thing on the road when the engine, in its infinite statistical confidence, decides to floor it toward a brick wall. This is where quality actually comes from—not from the model’s intelligence, but from the scaffolding that refuses to let the model’s output become the product without passing through a gauntlet of human-shaped checkpoints.

Quality, in my experience, is not something a model produces. It is something a harness enforces by constraining the model’s freedom at every fork in the road. The fewer choices the model has to make between two paths, the cleaner the result—a principle I have found to outweigh the quality of all prompts combined. Consider the mechanisms by which a real harness puts this principle into practice:

▸ Persistent rules, not per-session rituals. The harness carries a scoped rule engine. Rules with the all: scope are injected into every prompt; rules with a narrower scope, like cr: for code reviews, apply only when that command runs. This is the difference between engineering—where your conventions are encoded once, in the harness, and obeyed forever—and penance—where you manually correct the same mistake in every generated file until you retire.

▸ The practices that teach the model your house style. When the harness first meets a project in an unfamiliar language, it asks you to point at exemplary codebases, walks through them, and squeezes out a concise, human-editable list of good practices for that language. Those practices are then injected into the context whenever the model works with that language. Feed it good code, get good code. This is not mysticism; it is pattern matching. The harness is what remembers to feed the model the good code, every single time, without you having to remember.

▸ The plan gate. This is the single most important quality mechanism I have built, and it is almost embarrassingly simple. When a batch of tool calls looks non-trivial—two or more file-modifying actions—the harness pauses, drafts a structured plan, and refuses to execute until a human has looked at it and pressed “approve.” No more watching your agent cheerfully barrel through eight steps in one breath, commit untested code to main , and open a pull request titled “feat: everything.” The gate is the brake pedal. Without it, the engine is a runaway.

▸ The circuit breaker. When a tool fails three times in a row without producing anything, the harness disables it and tells the model to fall back to standard tools. This is the difference between an agent that learns from its mistakes and an agent that bangs its head against the same wall for forty turns while billing you by the token.

▸ The validation gauntlet. The harness is wired to the native linters and static analysers— oeditus_credo , propwise , credo , dialyzer —and can run them against the whole project, a diff, or a branch review. Quality is not asserted; it is executed . The model does not get to declare its work done; the linters get to declare it done, and they are considerably less susceptible to flattery than the model is.

▸ The workflow engine. For the occasions when a task is big enough to deserve ceremony, the harness runs a full pipeline: branch off main , summarise the task, decide whether it can be split into non-clashing sub-tasks, execute those in physically isolated git worktrees under parallel BEAM processes, require tests and documentation, gate on the linters, and only then commit. The whole thing is version-controllable, so your entire team shares the same agent rules, the same quality gates, and the same workflow. The harness is where your engineering standards live in a form that survives the departure of the senior engineer who used to carry them around in his head.

The Architecture Is the Point

I have saved the most important point for last, because it is the one people find hardest to believe. A harness built on the BEAM—on Erlang/OTP and the actor model—is not merely more reliable than a single-threaded script; it is architecturally incapable of the failure mode that plagues every fragile harness out there. The agent’s cognitive state runs in an isolated process, decoupled from the processes that execute tools. When a tool crashes—and tools will crash, because tools run shell commands and shell commands are chaos—the supervision tree contains the damage. The brain survives. The conversation survives. The context, which you have paid for in tokens and in blood, survives intact.

This is not a reliability footnote. This is a quality and token-economy argument wearing a hard hat. Every crash that wipes your session is not merely an annoyance; it is the total loss of every token you spent building that context, followed by the certain expenditure of twice as many tokens rebuilding it from scratch with a model that has already forgotten everything. Fault tolerance is not a luxury feature. Fault tolerance is the cheapest context-compression strategy ever invented, because the cheapest context is the context you never have to rebuild.

The Engine Is Not the Product

So here is the thesis, stated without hedging, in the hope that it survives contact with the comment section: you are arguing about the wrong component. The model is a commodity. They are all autocomplete, they are all statistical amplifiers, and the difference between the best and the merely excellent is a rounding error compared to the difference between a good harness and a bad one. A brilliant model in a fragile shell is a smart idea waiting for a crash. A mediocre model in a disciplined harness will out-ship a brilliant model in a shopping trolley every single time, because the harness is where the wise part of “Wise ∧ LLM = Wise²” lives.

The engine does not win the race. The car does. And the car—the suspension that keeps it on the road, the brakes that stop it before the wall, the fuel injection that meters exactly how much petrol per stroke, and the driver who refuses to let it floor it into a lamppost—is the harness.

Stop shopping for engines. Start shopping for cars.

Happy prompting—and may your harness be the one that knows how to say no.

Previously, on the subject of refusing to lose your mind over autocomplete:

Anti-Antipatterns in Elixir Library Guidelines

Sat, 31 Oct 2026 12:35:03 +0000

Deconstructing the dogmatic anti-patterns section of the official Elixir library guidelines and demonstrating when every single one of them is actually needed.

Exploring Programming Languages · YokerHood · Aleksei Matiushkin

Sat, 31 Oct 2026 12:35:03 +0000

Nott Løvland · Exploring Languages

In the beginning, I sought to be wise,
with a language that opened my digital eyes.
I started with LISP, for the pure of the mind,
But spent three whole weeks trying pairs to unwind.

(((((((Are (we) (sure) (this) (is) (right?))))))))
Parentheses blinded my aching eyesight.

So I fled to the past, where the money was made,
and woke up in COBOL, deeply afraid.
IDENTIFICATION DIVISION roared in my head,
with columns and margins, I wished I were dead.
It’s great for a bank in the year seventy-nine,
But a thousand lines later, I still couldn’t sign.

“Let’s try something hip!” I exclaimed with a twirl,
and drowned in a bucket of regex and Perl.
A script that looked exactly like line-noise and spit,
It ran, but God help me, I can’t read a bit.
Is that a variable, or did my cat walk
across the keyboard while trying to talk?

Then came Python, the savior, the clean, and the bright!
Until a stray space ruined my day and my night.
“Indentation Error,” the compiler did shriek,
because of one tab in the middle of the week.
Don’t get me even started on packaging hell,
where pip and venv cast a curse and a spell.

I jumped into Ruby, for joy and for love,
with blocks and with gems sent from heaven above.
But Monkey Patching turned the code to a zoo,
When a library changed what 2 + 2 do.
It was beautiful, sure, but it ran like a snail,
Chugging along on a rusted old rail.

So I went corporate. Enter Java, the grand.
The boilerplate king of the enterprise land!
AbstractMethodFactoryProviderBean
—was the shortest class name that I ever had seen.
I typed until my fingers were bleeding and numb,
just to print out “Hello” to a world that was glum.

“To the web!” I declared, and embraced JavaScript,
where logic is warped and reality’s flipped.
[] == ![] evaluated to true ,
and NaN is a number? I’m sorry, that’s rude!
undefined is not a function, it cried in my face,
as npm bloated and swallowed my disk space.

So I looked for speed, and Golang caught my eye,
“It’s simple!” they promised, “Just give it a try!”
But if err != nil was on every damn line,
an endless repetition of error design.
No generics (at first), just copy and paste,
a minimalist’s dream, and a developer’s waste.

Then Rust was the answer, the savior of tech!
If the Borrow Checker didn’t snap my poor neck.
“You can’t use lifetimes! This memory’s owned!”
I sat at my desk, thoroughly powned.
I fought with the compiler for hours on end,
until I forgot why I coded, my friend.

Then out of the ashes, a phoenix arose,
embraced by Erlang, in functional prose.
Elixir! Oh, sweet elixir of life,
you banished my sorrow, you ended my strife.
With pattern matching so clean and so neat,
and pipe operators ( |> ) that make life a treat.
The BEAM handles millions of actors with grace,
while I sit with a massive, smug smile on my face.

Let the servers all crash! Let the supervisors play!
I’m finally happy. Go away, anyway.

Your Orchestrator Is a Finite Automaton in Denial

Sat, 31 Oct 2026 12:35:03 +0000

On the lonely `status` column that quietly metastasised into a state machine, the boolean flags breeding in its shadow, and why a real finite automaton is not academic finery but the cheapest insurance you will ever decline to buy

Make Friends With Your AI Assistant

Sat, 31 Oct 2026 12:35:03 +0000

Several rules from my experience which would make your cooperation with AI assistants a charm

Mocks Are Your Friends, Not Your Servants

Sat, 31 Oct 2026 12:35:03 +0000

On treating mocks as protagonists of the testing narrative rather than disposable stand-ins hired to nod along

Head of the Agents and Assistants Department

Sat, 31 Oct 2026 12:35:03 +0000

A first-hand account of building the Cure programming language with AI agents as co-developers—what worked, what crashed spectacularly, and the distilled rules that emerged from the wreckage.

Cure, Four Releases Deep: From FSMs to Furniture

Sat, 31 Oct 2026 12:35:03 +0000

A walk through Cure v0.16.0 through v0.19.1. A Finitomata-inspired FSM rewrite, the dependent-types core finally behaving like it promised, a pattern engine that actually matches, and a stack of ergonomics that should have been there six versions ago.

The Long Road to Cure

Sat, 31 Oct 2026 12:35:03 +0000

Just under a year ago I started building a BEAM language with dependent types and first-class FSMs—chose the wrong implementation language, wrote spaghetti, had an LLM tell me it was garbage, shipped something embarrassing, then spent months rethinking. Here is what I learned and where Cure stands today.

A Score for an Invisible Orchestra · YokerHood · Aleksei Matiushkin

Sat, 31 Oct 2026 12:35:03 +0000

Imagine a five-storey building with no lift, erected in the late fifties somewhere on the outskirts of Avtozavodskaya—or better still, in Kupchino. Every floor speaks its own language. Not figuratively but in the most literal sense: the ground floor communicates in Cyrillic, the second in Latin script, the third in ideograms, the fourth in cuneiform, and the fifth, in the manner of Wittgenstein, maintains a principled silence on the grounds that whereof one cannot speak, thereof one must be silent. The postman, delivering the correspondence, is obliged to carry five copies of one and the same letter, translated into each of these tongues, and to knock on the door every time, hoping the addressee has not moved to another floor.

That is precisely how the world of programming is arranged—if one looks at it from the wings rather than the stalls. Every language has its own internal representation of code. Python stores its AST the way a thrifty housewife stores dry goods: in tidy labelled containers— BinOp , FunctionDef , Name . Elixir, with characteristic self-assurance, uses triples {atom, metadata, children} and calls them quoted expressions , as though what we have before us is not a syntax tree but a collection of quotations from William Blake. Ruby stockpiles its S-expressions the way an antiquarian bookseller stockpiles yellowing volumes, and Erlang, faithful to tradition, converses in tuples and atoms intelligible only to Ericsson engineers and, by a curious coincidence, doctoral students at a handful of Swedish universities.

The problem is obvious to anyone who has ever attempted to build a code-analysis tool. Suppose you have written a superb cyclomatic-complexity analyser for Python. It is magnificent: it finds nested conditionals, counts branching points, draws control-flow graphs. Then a colleague comes along and asks, “Could you do one for Ruby?” And it transpires that all your work—all those tree walkers, all that pattern matching over Python’s AST—must be rewritten from scratch. From zero. For a different tree, with different nodes, different semantics, and different booby traps. And then a third colleague will turn up and request the same for Haskell.

Imagine a conductor forced to relearn musical notation every time a new instrument joins the orchestra. Violin—one system of writing. Cello—another. Oboe—a third, with reversed polarity, no less. The trumpet flatly refuses to acknowledge the existence of the staff and insists on a tablature of its own invention. Absurd, of course. In the real world every instrument reads the same score. Notes, rhythm, dynamics are universal. Only the technique of execution differs.

MetaAST is that score.

Before we turn to the details (and they deserve attention in the way a well-constructed detective plot deserves it), permit me a brief digression into theory. Fear not: no formulae, only an analogy. Though one formula will appear after all—but it is so elegant that failing to cite it would be a crime against aesthetics.

In the early two-thousands—when mobile telephones already existed but had not yet taken charge of our lives—the OMG consortium (Object Management Group, bearing no relation whatsoever to the divine or the exclamatory) released a standard called MOF: Meta-Object Facility. Its essence fits into four lines, yet it took the industry two decades to understand those four lines. MOF defines a four-level hierarchy of models:

M⁰ is running code. There it goes, spitting out results, crashing with errors, consuming memory. This is reality.

M¹ is a model of reality. For programs, it is the AST: the abstract syntax tree. Python’s BinOp(op=Add(), left=Name('x'), right=Num(5)) is M¹. Elixir’s {:+, [context: Elixir], [{:x, [], Elixir}, 5]} is also M¹. Every language describes its own code in its own M¹, the way every painter paints an apple in their own way.

M² is the model of models. The meta-model. It defines what a node of any AST can be . Not a concrete node of a concrete language, but the concept of a node. A binary operation is neither Python’s BinOp nor Elixir’s {:+, ...} . A binary operation is the idea that two operands are connected by an operator. UML lives at the M² level. And so does MetaAST.

M³ is the meta-meta-model. That which defines what meta-models themselves can be. The type system, the rules of composition. MOF lives here. In the context of MetaAST, this role is played by Elixir’s type system— @type and @spec .

The fundamental difference between MetaAST and LLVM IR, Java bytecode, or any other intermediate representation lies precisely here: all of those are models (M¹). They describe concrete code in a concrete format. MetaAST is a meta-model (M²). It describes what descriptions of code can be . The difference is roughly the same as between a dictionary and a language: a dictionary catalogues words, whereas a language defines the rules by which those words are possible in the first place.

Metastatic is an Elixir library that implements this idea in code. The name, as befits a respectable technical project, is charged with a double meaning: Met(a)-AST-atic , that is, “pertaining to the meta-level of AST.” The medical connotations are the house’s treat.

The architecture is three-layered, and this is not caprice but a consequence of theory:

M².1 — Core. Concepts present in every programming language on the planet. Literals, variables, binary operations, conditionals, function calls, assignments. Nothing exotic here. x + 5 in Python, Elixir, Ruby, Erlang, and Haskell is one and the same thing. An identical MetaAST representation. Literally:

Five languages. One tree. One analysis tool. A score legible to any instrument in the orchestra.

M².2 — Extended. Constructs that exist in most languages but not all. Loops, lambdas, collection operations, pattern matching, exception handling. Haskell knows nothing of imperative loops—so be it: it has recursion at the M².1 level. Ruby knows nothing of guards—no matter: the adapter’s metadata will preserve the context.

M².3 — Native. The emergency exit for constructs that resist generalisation. Rust’s lifetimes, Haskell’s type classes, Elixir’s metaprogramming. They are wrapped in {:language_specific, :rust, ...} —like fragile porcelain in bubble wrap—and travel through the system without losing their identity, yet without claiming universality either.

Every MetaAST node is a triple: type, metadata, children (or value). The format is deliberately borrowed from Elixir’s quoted expressions, because if you already know how to write macros in Elixir, you already know how to work with MetaAST. The difference is semantic: where Elixir uses :+ (the operator itself), MetaAST uses :binary_op (the concept of a binary operation) and tucks the operator into metadata. Where Elixir inlines literals, MetaAST wraps them in {:literal, [subtype: :integer], 42} , ensuring structural uniformity.

Language adapters form the bridge between M¹ and M². The Python adapter takes Python’s AST (obtained via subprocess) and abstracts it up to MetaAST. The Elixir adapter takes quoted expressions and does the same. The Ruby adapter—likewise. The inverse operation— reification —turns MetaAST back into the native AST of the target language. This pair of operations, abstraction and reification, constitutes what mathematicians call a Galois connection:

Adapter_L = (alpha_L, rho_L)

alpha_L: AS_L -> MetaAST x Metadata    (abstraction: M¹ -> M²)
rho_L:   MetaAST x Metadata -> AS_L    (reification: M² -> M¹)

What does this mean in practice? Here is a scenario. You have written a function-purity analyser. It operates on MetaAST: traverses the tree, looks for side effects (I/O, state mutation, calls to random-number generators), and renders its verdict. This analyser was written once. Once. And it works with Python, Elixir, Ruby, Erlang, and Haskell. Because it analyses not the Python or Elixir AST but the meta-level . You write:

And precisely the same code, unchanged, works if you replace Python with Ruby and print('hello') with puts 'hello' . Because both calls are {:function_call, [name: "print"], [{:literal, [subtype: :string], "hello"}]} at the M² level.

Let us return to MOF and the four-level hierarchy. Why is any of this necessary? Why erect meta-models when one could simply write a converter from one AST to another?

The answer is simple and brutal, like the truth of life. A converter works pairwise. For five languages you need twenty converters (5 × 4). For ten—ninety. For twenty—three hundred and eighty. A meta-model works through a hub: each language connects once, via its own adapter. Five languages—five adapters. Ten—ten. Twenty—twenty. Linear growth instead of quadratic. A mathematician who had spent half his life as a reporter would appreciate the irony.

But it is not only a matter of combinatorics. A meta-model provides standardisation . Every tool written for MetaAST is guaranteed to work with any language for which an adapter exists. This is not “Python support” and “Ruby support” as separate features. It is one feature: MetaAST support. Everything else follows as a consequence.

OMG understood this in 2002 when it released MOF. The entire UML industry is built on the same principle: the meta-model defines what models can be , and concrete diagrams are merely instances of that meta-model. MDA (Model-Driven Architecture) took the idea further: transformations between models are defined at the meta-level and applied automatically to any instances.

Metastatic does the same, but not for class diagrams—for the syntax trees of programs. It is not an IR, not a compiler, not a transpiler. It is the foundation upon which all of the above can be built—once, and for every language at once.

There is an old story about a farmer, I’ve heard from a stranger on a train. A man spends his entire life breeding different eggplant varieties. In decades, when the farmer is already old and blind, his fields have produced an unprecedented harvest. He calls for each and every agronomist in the world, they come, and the most famous one says, “These aubergines are adorable!” The farmer dies of frustration. The man does not know what aubergines are but he’s certain the great eggplant nobody would have called ‘aubergine.’ He considers his life wasted.

The ASTs of different languages are the eggplants. MetaAST is the knowledge that they all grow on the same field. The difference between them is terminological, not semantic. Python’s BinOp(op=Add()) , Elixir’s {:+, [], [...]} , and Ruby’s s(:send, ..., :+, ...) are different names for one and the same thing: {:binary_op, [category: :arithmetic, operator: :+], [left, right]} .

One could, of course, spend an entire life rewriting tools for every new language. One could relearn musical notation each time, bow before every new instrument in the orchestra, translate letters into five tongues for five floors of one and the same building. But why, when one can ascend a single level of abstraction—to the place where eggplants and aubergines are indistinguishable, and the score is one for all?

From proof of concept to a Bluesky DataPlane proxy

Fri, 30 Oct 2026 05:26:27 +0000

A few weeks ago, we open sourced our proof of concept of a DataPlane for Bluesky. It indexes posts and follows and serves timelines.

Getting from there to a complete DataPlane is a lot of work. In this post, we first outline what that work is, with Postgres as the first database. Then we look at a compromise that makes the proof of concept useful long before that: running it next to Bluesky's open source DataPlane and letting it serve only the timelines.

A complete DataPlane in Elixir

Our goal, as outlined in our first post, is an open source DataPlane that is low effort to operate and scales way better than the reference implementation.

In our companion post, we split the DataPlane's work into two paths.

The hot path serves timelines from memory. The cold path covers records, threads and profiles, and lives in a database.

For the cold path, we want a storage interface, an Elixir behaviour, so the database can be swapped. Postgres will be the first adapter. Everyone who runs an AppView today already has their data in it, and the reference schema works fine for the cold path as long as you don't build timelines from it. If we start from that schema, an existing AppView can switch to our DataPlane without migrating its data.

Our proof of concept covers a small part of the hot path.

Here is what's missing:

A gRPC endpoint for Bluesky's bsky.proto, which currently declares 124 methods. Many of them are lookups by URI or DID. Some, like search and notifications, are more work.
An indexer for every record type. The reference implementation has 21 indexer plugins: posts, likes, reposts, follows, blocks, lists, starter packs, verifications and more. Deletes need some care, because the firehose only tells you which record is gone, not what was in it.
A write path that batches inserts into Postgres and stores the firehose cursor in the same transaction as the records. That makes replaying events after a crash safe. The in-memory structures only get updated after the commit.
A subscription to bsync. Mutes and bookmarks don't come through the firehose.
A backfill. The firehose only tells you what happens from now on, so a new instance has to fetch the existing repositories first.
The follower graph in Roaring Bitmaps, with a stable mapping from DIDs to the integers the bitmaps need. Stable, because we want to snapshot the graph to disk to make restarts faster.
A startup routine that rebuilds the hot path from Postgres: load the follower graph and the recent posts, run the fan-out, and only then take traffic.

Most of it isn't hard, and most of the endpoints are straightforward, as we noted back in March. It's just a lot of work, and nobody can run the DataPlane in production before all of it is done.

The next step: a timeline proxy

The timeline is where Bluesky's open source DataPlane gives in first. As we described in our first post on the topic, the user experience often degrades with fewer than 100_000 users. Many of the other requests are lookups by URI or DID, which Postgres handles well for much longer.

So instead of waiting for the complete version, we can run our DataPlane next to the open source one and let it serve only the timelines. That's the part our proof of concept already covers. In the rest of this post, we look at how such a setup would work and what it takes to keep the two DataPlanes consistent.

Splitting the DataPlane API

The setup we have in mind builds on how the AppView talks to its DataPlane. It uses gRPC, and every call is a POST to a path that names the method, for example /bsky.Service/GetTimeline. That makes it easy to send different methods to different services.

The AppView does accept a list of DataPlanes in BSKY_DATAPLANE_URLS. But it picks one of them at random for every call, and only tries another one when a call fails with Unavailable or Aborted. That's meant for balancing load between identical DataPlanes and won't help us here. Instead, we put a proxy that understands gRPC in between.

With Envoy, most of the work is done by two routes:

route_config:

virtual_hosts:

-name: dataplane

domains:["*"]

routes:

-match:{path:"/bsky.Service/GetTimeline"}

route:{cluster: dataplane_elixir }

-match:{prefix:"/"}

route:{cluster: dataplane_node }

GetTimeline goes to our Elixir DataPlane, every other call goes to Bluesky's. The Node DataPlane keeps doing what it did before: it consumes the firehose, writes to Postgres and answers all the other methods.

The Node DataPlane only speaks HTTP/1.1, so the status of each gRPC call comes back in an HTTP/1.1 trailer. Envoy drops those by default, so the Node cluster needs enable_trailers.

Our DataPlane gets its data from Postgres rather than from the firehose.

Our proof of concept can't take over that route just yet. It answers JSON on /bsky.Service/GetTimeline, which is all our simulator needs, while the AppView sends protobuf messages over gRPC. So the gRPC endpoint from the list above is the first thing the timeline proxy needs, too.

Where the IDs come from

The DataPlane only returns a skeleton of a timeline: a list of post URIs, a repost URI where an item is a repost, and a cursor. The AppView then hydrates that skeleton into full views. It calls GetPostRecords, GetRepostRecords, GetActors, GetInteractionCounts and friends, and all of those end up at the Node DataPlane and Postgres.

So whatever our DataPlane returns has to mean the same thing to Postgres. Luckily, atproto already gives us IDs that mean the same everywhere. A post is identified by its AT-URI, like at://did:plc:…/app.bsky.feed.post/3k…, and a user by their DID. Both come from the network, not from either database.

Our proof of concept doesn't use them yet. It numbers posts with a counter as they arrive. That's good enough for the simulator, but Postgres has never heard of post number 4711. The timeline entries have to carry URIs instead.

The integer IDs that Roaring Bitmaps need for the follower graph can stay private. We can keep a mapping from DID to integer in one ETS table and the reverse in another, and translate right before we build the response. Nobody outside our DataPlane ever sees them, so they don't have to match anything in Postgres.

Agreeing on time

With URIs in place, both DataPlanes talk about the same posts. They can still disagree about when a post exists. If both consume the firehose on their own, our DataPlane may put a post into a timeline before the Node DataPlane has written it to Postgres. The AppView can't hydrate that post and drops it from the page. Reload a second later and it's there. It's not a disaster, but users notice that kind of flakiness.

The way out is to not read the firehose at all, and read Postgres instead. More precisely, our DataPlane follows the changes to two tables with logical replication:

follow holds the follower graph.
feed_item holds a row for every post and every repost, with its author (originatorDid), its CID and the timestamp timelines are sorted by (sortAt).

Those are exactly the two tables the reference implementation joins for every single timeline request. The difference is that we don't join them on read. We fan out on write instead.

This gives us a useful guarantee: every URI we hand out is already committed in Postgres, so hydration will find it. The opposite case is harmless. A deleted post can linger in our memory for a moment, hydration comes back empty and the AppView drops it.

On the Postgres side, we need a publication for the two tables:

-- requires wal_level = logical, and the Elixir side connects

-- with a role that has the REPLICATION attribute

CREATE PUBLICATION timeline FORTABLE bsky.follow, bsky.feed_item;



ALTERTABLE bsky.follow REPLICA IDENTITYFULL;

ALTERTABLE bsky.feed_item REPLICA IDENTITYFULL;

By default, a delete only carries the primary key, which is the record's URI for both tables. With REPLICA IDENTITY FULL, an unfollow also tells us who unfollowed whom, and a deleted post tells us its author. We pay for that with a bit more WAL, but we don't need an extra lookup table just to process deletes.

On the Elixir side, Postgrex comes with Postgrex.ReplicationConnection. A sketch of our replication process:

defmoduleDataplaneEx.Replicationdo

usePostgrex.ReplicationConnection



aliasDataplaneEx.{Bootstrap,Indexer,PgOutput}



@impltrue

defhandle_connect(state)do

    query ="CREATE_REPLICATION_SLOT timeline TEMPORARY LOGICAL pgoutput EXPORT_SNAPSHOT"

{:query, query, state}

end



@impltrue

defhandle_result([%Postgrex.Result{rows:[[_slot, _lsn, snapshot, _plugin]]}], state)do

# drop what we had, then load follows and recent feed items as of the snapshot

:ok=Bootstrap.load(snapshot)



    query =

"START_REPLICATION SLOT timeline LOGICAL 0/0 "<>

"(proto_version '1', publication_names 'timeline')"



{:stream, query,[], state}

end



@impltrue

defhandle_data(<<?w, _wal_start::64, _wal_end::64, _clock::64, message::binary>>, state)do

    message

|>PgOutput.decode()

|>Indexer.apply_change()



{:noreply, state}

end



# plus start_link, init and the keepalive handling

# straight from the Postgrex docs

end

Postgrex hands us the raw pgoutput messages, decoding them is up to us. It's a small, well documented binary protocol, and binary pattern matching is one of the things Elixir is really good at.

Persistence for free

This also takes care of a problem from our last post: our data doesn't survive a restart. With Postgres as the source of truth, it doesn't have to.

On every start, we create a temporary replication slot and let Postgres export a snapshot along with it. A second connection opens a transaction on exactly that snapshot (SET TRANSACTION SNAPSHOT '…'), loads the follows and the recent feed items, and runs the fan-out. Then we start streaming from the slot. Changes that happen during the load wait in the slot, so nothing gets lost and nothing arrives twice. And since the slot is temporary, Postgres doesn't hold on to WAL for a node that is down.

Cursors and falling back

The AppView pages through a timeline with a cursor it gets from the DataPlane. It doesn't care which DataPlane created it. The reference implementation builds the cursor from the last item on the page: its sortAt in milliseconds and its CID, joined by two underscores, like 1792497600000__bafyrei….

If our DataPlane produces cursors in exactly that format, the Node DataPlane can continue any timeline we started. That's our way out whenever we can't answer a request ourselves:

The user's timeline isn't there yet, for example because we are still loading after a restart.
The cursor points to posts older than what we keep in memory. Timelines are bounded, so somebody who scrolls far enough will get there.

In both cases, our DataPlane calls GetTimeline on the Node DataPlane with the same cursor and passes the answer on. For that to work, our timeline entries carry sortAt and CID next to the URIs. Sorting by them also gives us the same order as the reference implementation, ties included.

The reference implementation does two more things we have to copy. It mixes the user's own posts into their timeline, up to ten per page even when followed accounts fill it. And it supports a since parameter that bounds the range and comes back as the cursor once that range is used up. If our DataPlane behaves differently, users would see their timeline change whenever we hand over to the Node DataPlane.

What the timeline proxy doesn't solve

This setup takes the most expensive query off Postgres. It doesn't change anything else:

The Node DataPlane still indexes the firehose and writes everything to Postgres. This is no improvement, but indexing is not a bottleneck in normal operations.
Threads, profiles, notifications, search and all the other requests are still served by the Node DataPlane from Postgres.
Postgres stays a single database that you can't scale horizontally.

So it lets you grow further with the open source DataPlane, but it won't get you anywhere near the size of Bluesky. But that's not our goal here: we are aiming to close the gap between a small community and over 100k users, until the user count ensures a serious engineering budget.

Our simulator works with both DataPlanes, so that's the tool we'll use to find out how far your setup can go.

From the timeline proxy to the complete DataPlane

The timeline proxy also gives us a way to get to the complete version step by step.

Every method we implement in Elixir becomes another route in Envoy. We can move them over one at a time, starting with the ones that hurt most. At first, they read the same Postgres tables the Node DataPlane writes to, which is why starting from the reference schema matters so much.

Before we switch a route, we want to know that our answers match. Envoy can mirror requests to our DataPlane, but it throws the mirrored responses away. So the comparison has to happen on our side: for a sample of requests, our DataPlane also asks the Node DataPlane and logs any difference.

The last piece to move is the indexer. Once our DataPlane consumes the firehose itself and writes to Postgres, it doesn't need the replication stream anymore. At that point, the Node DataPlane can be switched off, and we have the complete DataPlane from the beginning of this post.

This is still a side project for us. If you run an AppView and your Postgres is already sweating over timelines, we'd like to hear from you. Give us a shout if you want to help us take it further, faster.

Monitoring the Health of Oban Applications · Oban Pro

Fri, 23 Oct 2026 04:52:27 +0000

Oban tries its damnedest to keep queues chugging along in the face of unexpected issues. But stuff happens. Queues back up, jobs run longer than we'd like, and sometimes bugs (no, never our bugs) jam up the whole system. You want to know these things are going down before a customer tells you, because for every customer that reaches out there's a gaggle that were silently peeved.

It doesn't have to be like that. Between telemetry events, periodic jobs, and some diligent logging, Oban exposes the tools you need to alert on trouble early. No third-party collector services necessary (though possibly helpful).

🙋 Is This Only for Elixir?

It's not! The techniques here apply to Oban for Python as well. The exception is querying metrics, which needs Met on an Elixir node.

Taxonomy of Issues

In our experience, there are three categories of issues that you should monitor to keep your Oban system well-oiled. In order of "devastating" (everything seems broken) to "perplexing" (that doesn't look right), here they are:

Totally Stuck - no queues are running and cron jobs aren't getting scheduled
Backed Up - queues can't keep up with throughput and backlog keeps growing
Left Hanging - queues are clogged with some jobs stuck executing forever

Each of those highly scientific categories has its own distinct flavor of breakage that you can detect proactively and remediate.

Totally Stuck

A totally stuck system is the most catastrophic scenario, but thankfully the least ambiguous to identify. Queues stall and simply stop processing any jobs. Within the confines of Oban, where your database is up and rest of the application is running, this is usually a config issue.

Identification

A simple technique for detecting stuck queues is to watch logs or metrics and alert when no events have occurred within a window of time. Make sure you're ready for production with logs firing and you're halfway there. There's a catch though:

If you have periods of low activity where nothing is happening you'll get a false positive and be paged for no reason. To prevent that inactivity, set up a periodic job in a dedicated queue. The job acts as a sentinel, and its purpose is to force some activity.

Start with a health queue that won't run other jobs (so it can't back up), and schedule a cron job that checks in every minute:

defmodule MyApp.Sentinel do

  # Using the Pro worker for inline cron, you can also do it the old-fashioned way

  use Oban.Pro.Worker, queue: :health, cron: "* * * * *"



  @impl Oban.Pro.Worker

  def process(_job) do

    :my_app

    |> Application.fetch_env!(:some_check_in_url)

    |> Req.get!()



    :ok

  end

end

With a monitoring service like Honeybadger, the check-in reports your Oban is alive and kicking. If the scheduled job doesn't run and misses a check-in, you'll get a pointed alert.

Remediation

The alert tells you when jobs aren't processing, but it doesn't tell you why they aren't. As noted earlier, it's usually down to configuration. The most common issues are:

Broken Leadership - a non-worker node has leadership or a zombie node is still hanging on
Broken Notifications - using PG based PubSub without a functional Distributed Erlang cluster

These used to be issues, but that was prior to Oban v2.23.1, where we added fallbacks to ensure processing continued despite any distribution issues. Read on if you're on a previous Oban version, or want to understand how to fix these issues for other reasons.

Broken Leadership

A leader node that can't run plugins won't schedule cron jobs. Pruning and rescuing orphan jobs stop too, so jobs keep piling up. This typically happens in a hybrid web/worker environment, where web nodes don't process jobs.

Prevent this situation by either disabling plugins and leadership with plugins: false, or disabling leadership with peer: false. Don't use plugins: [], queues: [], because Elixir deep merges config, and an empty list combines with existing options rather than overwriting it.

Broken Notifications

Notifications must reach queues to alert them of available jobs. When notifications can't go through, queues may stop fetching work entirely. Oban tries to detect unhealthy notifiers and log warnings when connectivity is broken. However, in a distributed cluster a node will still receive messages from itself, which makes it impossible to accurately detect a disconnect.

Ensure you have a functional cluster when using the Oban.Notifiers.PG notifier, or that you're not behind a connection pooler for Oban.Notifiers.Postgres.

Backed Up

Queues are designed to hold jobs, but you run into trouble when jobs accumulate faster than your system can process them. Maybe your app imports a batch of 100k records every morning, then spends the next hour or so working through them. That's completely normal. The trouble starts when jobs keep arriving faster than you can finish them.

As designed, the health sentinel from earlier won't detect queue issues. To spot queue backups you need to monitor the backlog size and overall throughput.

Identification

If you're using Oban Web, you already have Oban.Met gathering the counts and metrics you need. If you're not using Oban Web...why not, it's free! Anyhow, Met shares all those values between nodes, and you can use the same data that powers the dashboard without polling for your own metrics.

Python apps can enable metrics to feed the dashboard, but there's no Met in Python to query them from your own code. If an Elixir node runs alongside your Python workers, it can query everything. Otherwise, the dashboard is your ticket to backlog and throughput, and a telemetry handler on oban.job.stop can feed your own collector.

Start by tracking how many jobs are waiting in each queue:

That will give you the freshest count, broken down by queue, so you can see where jobs are accumulating. Then, to see how quickly you're working through the accumulated jobs, check the exec_count in one-minute buckets for the past five minutes:

The counts tell you how much work is available, and how quickly it's being completed. To make them useful beyond an iex session, you'll want to export them to your metrics collector of choice.

Here's a cron function (new in Pro v1.8) that runs every minute, in the health queue, and sends the latest metrics to a StatsD-type collector. From there, you can set up per-queue alerts when throughput is falling behind:

@job cron: "* * * * *", queue: :health

def report_queue_metrics do

  queues = Oban.Met.labels("queue")



  available =

    Oban.Met.latest(Oban, :full_count,

      filters: [state: "available"],

      group: "queue",

      lookback: 5

    )



  completed =

    Oban

    |> Oban.Met.timeslice(:exec_count,

      filters: [state: "completed"],

      group: "queue",

      lookback: 60,

      by: 60

    )

    |> Enum.reduce(%{}, fn {_bucket, count, queue}, acc ->

      Map.update(acc, queue, count, &(&1 + count))

    end)



  Enum.each(queues, fn queue ->

    :ok = Statsd.gauge("#{queue}.available", Map.get(available, queue, 0))

    :ok = Statsd.gauge("#{queue}.completed", Map.get(completed, queue, 0))

  end)

end

This assumes the enclosing module uses Oban.Pro.Decorator and Statsd is your configured StatsD client.

Only you know the proper thresholds for your application, based on how much delay you can tolerate. Nightly imports may have far more leeway than a queue delivering timely password resets.

Remediation

A growing backlog is a mismatch between incoming work and system capacity. Knowing there's an issue doesn't tell you why it happened, and you'll need to do some investigation.

Check whether you're receiving more jobs, whether jobs are taking longer, or whether you've reduced the queue's concurrency. It could be time to increase concurrency, increase available CPUs, (auto?)-scale to more nodes, process jobs together in chunks, or tune some global concurrency.

Just be sure you get to the bottom of the slowdown before turning the dial.

Left Hanging

Without a timeout, an Oban job can run forever. Well, "forever" meaning until the application restarts, but that's long enough to cause trouble. An infinite await, a hanging network call, or an unbounded query can leave a job squatting without accomplishing anything.

One hanging job reduces a queue's capacity, and enough of them will stop processing altogether. Meanwhile, your application is running, the health checks are fine, and everything looks alive.

Identification

Start by deciding how long each worker should reasonably take. A job that transcodes video naturally takes more time than one that sends an email. Again, only you know the right numbers for your system.

Met already tracks execution time through exec_time, and it includes a worker label to help identify which jobs are taking too long. As we did earlier, check the longest executions per worker in one-minute buckets:

That gives you five minutes of execution times for jobs that succeeded, failed, or were cancelled.

Export those values alongside the queue metrics from the previous section, and alert when workers exceed their expected duration (with some padding). Note that Met records in native time units by default, so convert the values to something more useful, like milliseconds, before exporting.

There's an important catch here: exec time is only recorded after an attempt ends. Jobs that hang forever will never report their duration, making those outliers effectively invisible. That's why alerting alone isn't enough.

Remediation

A timeout/1 callback defined on the worker will put an upper bound on each execution. For example, to set a static two-minute timeout:

Choose a timeout that leaves headroom for variation, and set your threshold below it. That will help you spot workers that regularly take longer than expected, even when they finish before the timeout.

When a job times out, execution is terminated with an Oban.TimeoutError exception and potentially retried if there are remaining attempts. As a bonus, the failure contributes to exec_time measurements, which become visible in the timing report.

If the work itself expires—like a notification that would arrive a day late—job deadlines in Pro will cancel it on its next run after the deadline passes. Setting force: true can cancel the job mid-execution too.

Use timeouts to limit execution, and deadlines to limit how long the job sticks around. Check a job's network calls, database queries, and potential deadlocks and apply inner limits wherever possible. That way you'll get more meaningful timeouts, and a chance for the job to handle the failure before Oban takes care of it.

Knowing is a Fraction of the Battle

The issues we cover here have plagued Oban users for years now. Wherever possible, we've improved Oban (and Pro) itself to make entire categories of issues irrelevant.

Put these tips into practice in your own app, and catch issues before your customers do. Chances are, you lovely humans are only going to glance over this...but your LLMs are going to love it.

As usual, if you have any questions or comments, ask in the Elixir Forum. For future announcements and insight into what we're working on next, subscribe to our newsletter.

Exploring OTP Patterns for Agentic Workflows

Thu, 22 Oct 2026 03:19:46 +0000

Chapter 1: State

“AI is not my area.” That was my answer when a friend suggested I give a talk about agentic workflows. I used that phrase a lot at the beginning of the year; I was pretty skeptical about the whole thing.

But the idea stuck. I started reading about what makes agentic systems hard to build, expecting to find something wild and exotic to dig into. What I found instead was a list of problems that sounded pretty familiar:

State.
Long-running processes.
Communication.
Concurrency.
Coordination.
Failure recovery.

None of that was new to me. If you work with Elixir, Erlang, and the BEAM, it’s probably not new to you either.

A few months later, I was on stage at ElixirConf US giving the talk “Exploring OTP Patterns for Agentic Workflows”. This series is based on that talk, but it goes deeper into the topics I couldn’t fully cover on stage.

The goal is to help you better understand agentic systems from the perspective of a BEAM developer. Not because OTP is an AI framework, nor because AI concepts map one-to-one to elements of the BEAM ecosystem, but because many of the engineering challenges these systems present are problems the BEAM community has been solving for decades.

So the question is simple:

What do I already know that can help me understand agentic workflows?

In this chapter, we start with the first item on the list: state.

A familiar problem, a different way of approaching it

The example we’ll use throughout this chapter starts with a familiar situation: an endpoint that returns an error instead of the user’s data, and us leaning on an AI agent to debug and fix it.

GET/users/:idHeyClaude:TheGET/users/:idendpointisreturninga500error
foravalidID.Itshouldreturntheuserobject.Fixit.

Notice that the prompt describes what I want, not how to get there.

The agent takes this prompt as its goal. It has a codebase to investigate and a set of tools to do so. Let’s say it starts by inspecting the router and then the controller. What it does next depends on what it finds along the way.

One distinction matters for everything that follows. The model at the center of the agent, a large language model (LLM), doesn’t open files or interact directly with the system. It participates in deciding what should happen next, but it’s the tools that do the work, and their results become part of the context the model sees when it makes its next decision.

At a high level, we can represent it like this:

For this series, that’s our working definition of an agentic workflow: a workflow in which the model decides the path as it progresses, based on the context and on what it observes.

What interests us is everything behind that loop that makes it work as a system:

Where we keep track of what has happened.
Where the work runs.
How the results come back.
What work can happen concurrently.
What happens when things fail.
And finally, when the workflow stops.

We already know how to reason about those problems, and that’s what this series explores.

The system needs to remember what happened

After a few iterations, the agent has gathered information: the controller function behind the endpoint, the tests, the relevant spots in the codebase. If every iteration had to start from scratch, it would never make any progress.

But from the model’s perspective, each invocation starts with only the context it’s given. The model itself doesn’t carry anything over from one invocation to the next.

That means the system around it must preserve information about the execution: the goal, what has already happened, what work is still pending, how many steps it has taken, and so on.

In other words, the workflow has state.

Before we continue, let’s define another concept we’ve already mentioned and will need throughout this series: context.

When the model needs to decide what to do next, it needs information about what has happened so far. Maybe the failing test output, the relevant function, the original goal, and the result of the last tool call. The model receives that information as context.

The two are related, but they’re not the same thing. One way to see it:

The system has state. The model receives context derived from that state.

The state usually contains more than the model needs for a particular decision. Step counts and pending work may matter to the system without needing to be exposed to the model. And the model can only take in a limited amount of context per invocation (what’s known as its context window), so even the relevant information may need to be selected, summarized, or trimmed.

What the model sees depends on the decision we’re asking it to make. This brings us to the first interesting question of the chapter.

Who owns the state?

The answer is the essential building block of the BEAM world: a process.

A process owns its state. No other process can reach into its memory and change it; anyone who wants to interact with it has to send it a message. For a workflow, that suggests one process per execution: it starts when a goal comes in, holds the state for that run, and exits when the run is over.

But the process doesn’t just hold a value. It has to react to messages that arrive on their own schedule: a tool result, a timeout, a request to cancel.

A GenServer gives us exactly that. But be careful: the GenServer isn’t the AI agent. It’s just the process we’ll use to own and coordinate one execution of the workflow.

A naming note:agent in this series always means an AI agent, never Elixir’sAgentmodule. That module wouldn’t be enough here anyway: it’s a simple abstraction around state, built to hold a value, not to react to messages as they arrive.

Before writing the process, let’s look at what it has to hold. Go back to our broken endpoint. The agent has already read the router, and the model’s next decision was to do two things at once: read the controller and run the tests. Neither has come back yet. At that moment, the state of the run looks like this:

%BlogOtp.Workflow{
goal:"The GET /users/:id endpoint is returning a 500 error...",
model:BlogOtp.Model.Scripted,
history:[
{{:read_file,%{path:"lib/my_app_web/router.ex"}},
~s(get "/users/:id", UserController, :show)}
],
pending:%{
#Reference<...> => {:read_file, %{path: "lib/my_app_web/controllers/user_controller.ex"}},#Reference<...> => {:run_tests, %{path: "test/my_app_web/controllers/user_controller_test.exs"}}},
steps:3,
max_steps:10
}

Each field maps to something we’ve already talked about:

goal is the prompt we gave the agent.
model is the module that makes the decisions. In this chapter it’s a scripted stand-in for an LLM, so it doesn’t need anything extra to run.
history is what has happened so far: each tool call paired with its result.
pending is the work requested but not answered yet. Each request gets a unique reference, so when a result arrives, we know which one it answers.
steps counts the tool calls requested so far, and max_steps lets the system stop a run that goes on too long.

Only goal and history end up in the context. The rest exists for the system; the model never sees it.

Here’s the process that owns that state:

defmoduleBlogOtp.WorkflowdouseGenServer,restart::transientrequireLoggerdefstruct[:goal,:model,history:[],pending:%{},steps:0,max_steps:10]

defstart_link(opts),do:GenServer.start_link(__MODULE__,opts)

@impltruedefinit(opts)dostate=%__MODULE__{
goal:Keyword.fetch!(opts,:goal),
model:Keyword.get(opts,:model,BlogOtp.Model.Scripted),
max_steps:Keyword.get(opts,:max_steps,10)
}

{:ok,state,{:continue,:next_step}}
end@impltruedefhandle_continue(:next_step,state),do:next_step(state)

@impltruedefhandle_info({:tool_result,ref,result},state)
whenis_map_key(state.pending,ref)do{call,pending}=Map.pop(state.pending,ref)

state=%{
state
|pending:pending,
history:[{call,result}|state.history]
}

next_step(state)
end# A result we aren't waiting for: a duplicate, or a late arrivaldefhandle_info({:tool_result,_ref,_result},state),do:{:noreply,state}

# Still waiting on results: keep waitingdefpnext_step(%{pending:pending}=state)whenmap_size(pending)>0,
do:{:noreply,state}

# The system decides when to stop, not only the modeldefpnext_step(%{steps:steps,max_steps:max}=state)whensteps>=max,
do:finish({:stopped,:max_steps},state)

defpnext_step(state)do# Chapter 1 shortcut: the owner calls the model itselfcasestate.model.decide(build_context(state))do{:tools,calls}->{:noreply,request(calls,state)}
{:done,answer}->finish({:done,answer},state)
endend# Nobody executes these yet: each call waits in `pending` for its resultdefprequest(calls,state)doEnum.reduce(calls,state,fncall,state->%{state|pending:Map.put(state.pending,make_ref(),call),steps:state.steps+1}
end)
enddefpfinish(outcome,state)doLogger.info("Workflow finished: #{inspect(outcome)}")
{:stop,:normal,state}
end# What the model gets to see, in its simplest formdefbuild_context(state)do[{:goal,state.goal}|Enum.reverse(state.history)]
endend

The loop lives in next_step/1, which works like this:

If results are still pending, it waits.
If the run has gone on too long, it stops.
Otherwise, it builds the context and asks the model what to do next: either more tool calls, which go into pending, or a final answer, which ends the process normally.

And restart: :transient tells a supervisor to restart the process only if it terminates abnormally: a run that finishes with:normal stays finished.

Two things are deliberately missing in this chapter. First, nobody executes the tools yet: requests sit in pending until someone sends back a {:tool_result, ref, result} message. Second, the owner calls the model directly, even though a model call is slow. Both are problems we’ll come back to.

The code also shows the state/context distinction from the previous section. pending, steps and max_steps exist for the system; the model never sees them. And the context is the output of build_context/1, a plain function over the state, so we can test what the model will see without calling a model.

One message at a time

The code also gives us serialized access to the state.

Look at pendingin the example above. Suppose the controller’s contents and the test output come back at almost the same time. They arrive as messages, and the process handles them one at a time. Each handle_info/2 runs to completion against a consistent state before the next message is processed. There are no locks; nothing else can change the state in the meantime, and the context for the next model call is never built from a half-applied update.

The first result is simply recorded: pending still holds the other request, so next_step/1keeps waiting. Only when the second result arrives does the model get to decide, with both observations in its context.

But that same property has a cost.

A GenServer handles one message at a time, and the work in this loop is slow: a model call can take seconds, while a test suite can take much longer. If the process did that work itself, it couldn’t handle anything in the meantime. Messages would pile up in its mailbox: a result, a timeout, a request to cancel, all waiting their turn.

And that’s exactly what our chapter 1 shortcut does with the model call. It works here because our scripted model answers instantly; a real one wouldn’t.

So owning the state and doing the work pull in opposite directions.

The owner process needs to stay available, so the work has to happen somewhere else. That’s why nobody executes the tools yet: the owner shouldn’t be the one doing it. We’ll come back to that.

First, there’s a more urgent question: everything above lives in memory.

If this process dies, what happens to the run?

“Let it crash” works because a supervisor can start a new process in a known good state. But for a GenServer, that known good state is whatever init/1 builds. For a workflow, that means starting from the beginning, which is exactly what we don’t want.

Think about our endpoint. By the time the process crashes, the model may have already made several decisions: it read the router, read the controller, ran the tests, maybe even edited the controller. Starting over means repeating all of that, paying for it again in time and in model calls. And some of that work, like editing a file, may not be something we want to do twice.

The process can own the workflow state while it’s running, but that doesn’t mean process memory has to be the only place where that state lives.

We need enough durable information to reconstruct the workflow.

One way to do that is to record its progress as an append-only log. The goal is the first entry. Each tool call the model decides on becomes another, as does each tool result. As the workflow progresses, we only ever add these entries to the end.

For our endpoint, the log might look like this when the process crashes:

{:goal,"The GET /users/:id endpoint is returning a 500 error..."}
{:decision,1,{:read_file,%{path:"lib/my_app_web/router.ex"}}}
{:result,1,~s(get "/users/:id", UserController, :show)}
{:decision,2,{:read_file,%{path:"lib/my_app_web/controllers/user_controller.ex"}}}
{:decision,3,{:run_tests,%{path:"test/my_app_web/controllers/user_controller_test.exs"}}}
{:result,2,~s[def show(conn, %{"id" => id}) do ... json(conn, user) ...]}

Everything the process knew is there: the goal, every decision, every result that came back. Even the request that’s still waiting for its answer.

If that log becomes the source from which we reconstruct the workflow state, we’re using the same core idea as event sourcing.

Rebuilding the state from the log

Let’s put that idea into code. Three things change: the workflow gets an id that survives the process, each tool call gets a step id the workflow assigns, and every state change goes through a single function.

defstruct[:id,:goal,:model,history:[],pending:%{},steps:0,max_steps:10]

# Only updates the state: no work, no model calls, no messagesdefpapply_step({:goal,goal},state),do:%{state|goal:goal}

defpapply_step({:decision,step_id,call},state)do%{state|pending:Map.put(state.pending,step_id,call),steps:state.steps+1}
enddefpapply_step({:result,step_id,result},state)do{call,pending}=Map.pop(state.pending,step_id)
%{state|pending:pending,history:[{call,result}|state.history]}
end

Each clause handles one kind of entry from the log. apply_step/2 never starts work, calls a model, or sends a message. That restriction will matter in a moment.

The owner is still the only one that writes:

@impltruedefhandle_info({:tool_result,step_id,result},state)
whenis_map_key(state.pending,step_id)dostate=record({:result,step_id,result},state)
next_step(state)
end# Each call becomes a decision in the log, with a step id the workflow assignsdefprequest(calls,state)doEnum.reduce(calls,state,fncall,state->record({:decision,state.steps+1,call},state)
end)
end# Persist first, then applydefprecord(entry,state)do:ok=BlogOtp.Log.append(state.id,entry)
apply_step(entry,state)
end

Notice two things.

The first is step_id, which replaces the reference from the previous version. A reference created with make_ref/0 only identifies something within the running node, and the process waiting for it may be gone after a crash. A step id is just a number the workflow assigns when it records the decision, and it means the same thing to any process that reads the log.

The second is the order inside record/2. We write to the log first, then apply the entry to the in-memory state. If the process dies between those two lines, the log is ahead of memory, and memory is the part we know how to rebuild. The other way around, the process could act on a step that no restart would ever see. And if the append fails, the process crashes before acting on anything.

The same rule applies to decisions: they’re recorded before anyone starts the work. In this chapter nobody executes the tools yet, but when someone does, the decision will already be in the log.

This puts a durable write on the owner’s critical path. That’s intentional: we don’t want the workflow to act on a transition it couldn’t recover after a crash.

In this chapter,BlogOtp.Logkeeps its entries in an ETS table owned by its own process. It survives a workflow crash, but not a node restart. A real implementation would write to disk or a database; the contract, append first and read in order, stays the same.

Starting a run now means creating its log first:

defstart(goal,opts\\[])doid=new_id()
:ok=BlogOtp.Log.append(id,{:goal,goal})

DynamicSupervisor.start_child(
BlogOtp.WorkflowSupervisor,
{__MODULE__,Keyword.put(opts,:id,id)}
)
enddefstart_link(opts)doGenServer.start_link(__MODULE__,opts,name:via(Keyword.fetch!(opts,:id)))
enddefpvia(id),do:{:via,Registry,{BlogOtp.WorkflowRegistry,id}}

The goal is written before the owner exists; from then on, the owner is the only writer. Registering the process under the workflow id in aRegistry with unique keys turns “only one writer” from an intention into a guarantee: a key points to at most one process, and a second attempt to start an owner for the same run would get{:error, {:already_started, pid}}.

Look at what the process is started with: the id, not the goal. When a supervisor restarts a child, it calls the same start function, with the same arguments, from the child’s specification. The id is all a new process needs to find out everything else.

@impltruedefinit(opts)dostate=%__MODULE__{
id:Keyword.fetch!(opts,:id),
model:Keyword.get(opts,:model,BlogOtp.Model.Scripted),
max_steps:Keyword.get(opts,:max_steps,10)
}

{:ok,state,{:continue,:rebuild}}
end@impltruedefhandle_continue(:rebuild,state)dostate=state.id
|>BlogOtp.Log.read()
|>Enum.reduce(state,&apply_step/2)

next_step(state)
end

When init/1returns {:continue, :rebuild}, OTP runs handle_continue/2 before the process receives any other message, so no result can be handled while the state is still being rebuilt.

The same apply_step/2 that updates the workflow while it runs also rebuilds it after a crash. Now we can see why it had to be free of side effects: replaying the log must not run the tests again. And a new run isn’t a special case either: it’s a log with a single entry, the goal. The fold produces a state with nothing pending, and next_step/1asks the model for its first decision.

The state is a fold over the log.

This changes what a restart means. The new process doesn’t need to remember what the old one knew. It only needs to rebuild enough of the workflow to continue from a known point. But careful!

Rebuilding isn’t resuming

Fold the log from our crash, and pending holds exactly one entry: step 3.

Run the tests

There’s no result for it. So next_step/1 sees something pending and waits, for a result that may never come.

We know the decision was recorded. But did the work actually run? Maybe the request was never sent. Maybe the tests were still running when the process crashed. Maybe they finished, and the result was lost before it could be recorded.

For a test suite, running it again is harmless. But imagine our agent’s last step had been:

Open a pull request

Now repeating it means a duplicate pull request. And some actions are even more delicate than that, like inserting a row into a database that must never exist twice, or charging a customer. Those are the ones where we have to be really careful.

Restarting the process gives us back the ability to continue the workflow. It doesn’t tell us whether to run an external action again.

Restart and retry are two different problems.

That leaves two questions open: where the work actually happens while the owner stays available, and what to do with a step whose outcome we don’t know. The first question is where the next chapter picks up. The second one will have to wait until we talk about failure recovery.

The complete code repository for this series will be shared in Chapter 2.

Want to learn more about agentic workflows and OTP? Get in touch with our ESL team.

The post Exploring OTP Patterns for Agentic Workflows appeared first on Erlang Solutions.

Weekend Writing Prompt — Glowing Elixer – Facts, Fictions & Fantasies

Mon, 28 Sep 2026 13:41:32 +0000

Search for:

Facts, Fictions & Fantasies

My Mind Mappings

Weekend Writing Prompt — Glowing Elixer

September 26, 2026September 26, 2026Fandango

This post was written for Sammi Cox’s Weekend Writing Prompt, where the challenge is to write a poem or a piece of prose in exactly 34 words using the word “hokum.”

The dusty traveling salesman swore that his glowing elixir cured every known ailment under the sun.

“Pure hokum,” muttered old doctor McKenzie, right before watching the town cripple start happily dancing down Main Street.

Image conjured using Leonardo.ai.

2 thoughts on “Weekend Writing Prompt — Glowing Elixer”

Nancy's Notes 🖊️♬ September 26, 2026 / 1:04 pm

Old Doc McKenzie’s seen it all! Good one, Fan.

LikeLiked by 1 person

Reply
- Fandango September 26, 2026 / 2:33 pm
  
  That he has, old Doc McKenzie!
  
  LikeLiked by 1 person
  
  Reply

Thoughts to share? Cancel reply

<div id="guide-statcounter_image" style="display:inline;"><img
src="https://c.statcounter.com/13031919/0/a3bc9779/1/"
alt="Web Analytics" 
referrerPolicy="no-referrer-when-downgrade" /></div>

Website Powered by WordPress.com.

Comment
Reblog
Subscribe Subscribed

Facts, Fictions & Fantasies

Join 628 other subscribers

Have a WordPress.com account? Log in now.
Facts, Fictions & Fantasies
Copy shortlink View post in Reader
Manage subscriptions Sign up Log in
Report this content
Collapse this bar

Loading Comments...

Write a Comment...

Email Name Website

Open Sourcing our Bluesky Dataplane

Sat, 26 Sep 2026 21:12:29 +0000

This is our AT Protocol journey here so far:
Starting with a toolkit to evaluate Blueskyʼs open source dataplane, we picked Elixir from all the languages in our toolbelt to build a proof of concept to serve timelines. We still owe you the code. Today, we are open sourcing our proof of concept of a dataplane.

The Closed Core of Bluesky

We talked about this before:
Itʼs great that Bluesky is running mostly on open source components. However, they draw a line at the dataplane that sits at the core of the system. Although Bluesky provides an open source implementation of the dataplane, this is not what they run in production. They use a second, closed source variant based on ScyllaDB because only this implementation is ready for the production scale of Bluesky.

Itʼs a pity because we already see interoperable alternatives to Bluesky emerging, delivering on the promise of hard decentralization. Members of the Eurosky team have even been invited to the European Parliament.

Running alternative networks should be as easy as possible but we see each of these projects reimplementing their dataplane. There is a need for an open source dataplane implementation that will sustain the load when the network grows.

We built our proof of concept in Elixir, for many reasons. Most notably, itʼs just an excellent runtime to build such systems, while we can always mitigate its weakness in raw compute performance by bringing in native code with a NIF.

You can find the repository here: https://github.com/bitcrowd/dataplane_elixir/
We invite you to evaluate its performance with our simulator project.

warning

Remember, it is a proof of concept, it is not a production ready implementation.

Give us a shout if you want to help us take it further, faster.

Implementing a Phoenix PubSub Adapter with EventStore

Fri, 18 Sep 2026 19:15:09 +0000

Distributed systems need async message delivery across nodes. Phoenix provides Phoenix PubSub for this, with pluggable adapters for different backends — officially PG2 and Redis.

This post walks through implementing a Phoenix PubSub adapter backed by EventStore, an Elixir event sourcing library that persists events to PostgreSQL as an append-only log.

Using EventStore as a PubSub backend has a few advantages over the default PG2 adapter:

No Erlang distribution required: nodes communicate through the shared database rather than through the Erlang cluster, so you can run multiple nodes without configuring Erlang node connectivity.
Persistence: every broadcast is stored and can be replayed or audited later.

The tradeoffs are the need for storage and the additional latency of a database round-trip per broadcast, making it best suited for lower-throughput messaging where persistence and cross-node decoupling matter more than raw speed. This implementation is a proof of concept — no load tests were performed.

A full implementation of the adapter can be found on Github.

Phoenix.PubSub.Adapter in a nutshell

A Phoenix PubSub adapter must implement a few callbacks specified in Phoenix.PubSub.Adapter:

node_name(adapter_name)

Returns the node name as an atom or binary. Used mainly by Phoenix.Tracker.In most cases:

defnode_name(nil),do:node()
defnode_name(configured_name),do:configured_name

child_spec(keyword)

Generates the child spec for the adapter. GenServer provides a default; this rarely needs overriding.

broadcast(adapter_name,topic,message,dispatcher)

Called when a message is broadcast throughPhoenix.PubSub.broadcast. The adapter_name is the PubSub name with .Adapter appended (e.g. MyApp.PubSub → MyApp.PubSub.Adapter). The dispatcher module handles local delivery via dispatch/3.

direct_broadcast(adapter_name,node_name,topic,message,dispatcher)

Same as broadcast/4 with an additional node_name — the message should only reach subscribers on that node.

The EventStore adapter

This section walks through a possible implementation of a Phoenix PubSub adapter that uses EventStore to distribute messages between nodes. This gives a solution that does not depend on Erlang/Elixir distribution, and an event log is stored in case further analysis is needed.

How Phoenix.PubSub works

Phoenix.PubSub uses Elixir’s Registry for subscriptions — each subscribe call registers an entry under the topic key. When broadcast is called, the framework invokes the adapter callback to distribute the message, then handles local dispatch.

The adapter’s job is to get the message to other nodes. Fordirect_broadcast, only subscribers on the target node should receive it.

The implementation

The adapter is a GenServer that joins the PubSub supervision tree. Aneventstore option selects which EventStore module to use (in case you have multiple):

{Phoenix.PubSub,
[name:MyApp.PubSub,
adapter:Phoenix.PubSub.EventStore,
eventstore:MyApp.EventStore]
}

The GenServer stores the EventStore module and the PubSub name in state — both are needed later:

defmodulePhoenix.PubSub.EventStoredo@behaviourPhoenix.PubSub.AdapteruseGenServerdefstart_link(opts)doGenServer.start_link(__MODULE__,opts,name:opts[:adapter_name])
enddefinit(opts)do{:ok,
%{
eventstore:opts[:eventstore],
pubsub_name:opts[:name]
}}
end#... implementation will come here ...#end

Note the difference between opts[:name]and opts[:adapter_name]. The former is the name of the PubSub as a whole and is reserved for the Registry. Publishers use it when broadcasting messages. opts[:adapter_name]can be used as the name of the GenServer.

Distributing a message as an event

The GenServer appends a new event to the EventStore when broadcast is called:

defbroadcast(server,topic,message,dispatcher,metadata\\%{})dometadata=Map.put(metadata,:dispatcher,dispatcher)
GenServer.call(server,{:broadcast,topic,message,metadata})
enddefhandle_call(
{:broadcast,topic,message,metadata},
_from_pid,
%{id:id,eventstore:eventstore,serializer:serializer,pubsub_name:pubsub_name}=state
)doevent=%EventStore.EventData{
# ... constructed below}

res=eventstore.append_to_stream(topic,:any_version,[event])

# For direct_broadcast targeting the current node, the framework does not# call local dispatch, so the adapter must do it. For regular broadcast,# the framework handles local dispatch after adapter.broadcast returns :ok.current_node=to_string(node())
destination_node=Map.get(metadata,:destination_node)

ifdestination_node==current_nodedodispatcher=Map.get(metadata,:dispatcher,Phoenix.PubSub)
Phoenix.PubSub.local_broadcast(pubsub_name,topic,message,dispatcher)
end{:reply,res,state}
end

direct_broadcast/5 is a thin wrapper that sets destination_node in the metadata before delegating to broadcast/5:

defdirect_broadcast(server,node_name,topic,message,dispatcher)dometadata=%{
destination_node:to_string(node_name),
source_node:to_string(node())
}
broadcast(server,topic,message,dispatcher,metadata)
end

source_node is stored in the event metadata for auditing. Routing is handled downstream by comparing destination_node against the current node.

The key decision is how to wrap the message inside %EventStore.EventData{}. Serialization is handled by a pluggable module (defaulting to Phoenix.PubSub.EventStore.Serializer.Base64) so the adapter is not tied to a specific encoding. The default serializer base64-encodes :erlang.term_to_binary/1 output — this is necessary because EventStore stores data as JSON and raw binaries would be invalid, and because JSON cannot distinguish atoms from strings so a round-trip through term serialization preserves type fidelity.

event=%EventStore.EventData{
event_type:to_string(serializer),
data:serializer.serialize(message)
}

A custom serializer can be provided via the serializeroption as long as it implements serialize/1and deserialize/1.

Handling events, local distribution

Now that events are in the event store, any subscribed process will receive them. The GenServer must subscribe to all topics ("$all"). If the event store is also used for another purpose, it’s best to have a separate one for PubSub. The subscription is set up via handle_continue/2, which runs immediately afterinit/1 completes, before any other messages can be processed.

defhandle_continue(:subscribe,%{eventstore:eventstore}=state)doeventstore.subscribe("$all")

{:noreply,state}
enddefhandle_info({:subscribed,_subscription},state),do:{:noreply,state}

A transient subscription is used since previous messages are not needed. The event store replies with a {:subscribed, subscription} message, which must also be handled. After this, the server will start receiving {:events, events} messages.

To avoid dispatching a local message twice (once from broadcast and once when the event arrives back from EventStore), a unique ID is added to the process state:

definit(opts)do{:ok,
%{
id:generate_unique_id(opts),
eventstore:opts[:eventstore],
pubsub_name:opts[:name],
serializer:opts[:serializer]||Phoenix.PubSub.EventStore.Serializer.Base64},{:continue,:subscribe}}
enddefpgenerate_unique_id(opts)dounique_id_fn=opts[:unique_id_fn]||fn_name->UUID.uuid4()endunique_id_fn.(opts[:name])
end

A custom ID generator can be provided via unique_id_fn — a function that receives the PubSub name and returns a unique string. Useful when UUID is unavailable or when a deterministic ID is needed for testing.

Theid is added to the event’s metadata field as source_id, keeping it separate from the message data. Serialization is delegated to the configurable serializer module. The handle_call for :broadcast becomes:

event=%EventStore.EventData{
event_type:to_string(serializer),
data:serializer.serialize(message),
metadata:Map.put(metadata,:source_id,id)
}

Where the value of id and serializer come from the state, and metadata already contains dispatcher and any destination_node for direct broadcasts. When an event arrives back, source_id identifies the origin node so duplicates can be skipped:

defhandle_info({:events,events},state)doEnum.each(events,&local_broadcast_event(&1,state))

{:noreply,state}
enddefplocal_broadcast_event(
%EventStore.RecordedEvent{
data:data,
metadata:metadata,
stream_uuid:topic,
eventbies_type:event_type
},
%{id:id,serializer:serializer,pubsub_name:pubsub_name}=_state
)docurrent_node=to_string(node())

%{source_id:source_id,destination_node:destination_node,dispatcher:dispatcher}=convert_metadata_keys_to_atoms(metadata)

is_destination?=is_nil(destination_node)ordestination_node==current_node

ifnotis_nil(dispatcher)andis_destination?andsource_id!=idandevent_type==to_string(serializer)doPhoenix.PubSub.local_broadcast(
pubsub_name,
topic,
serializer.deserialize(data),
maybe_convert_to_existing_atom(dispatcher)
)
endend

That’s it — a complete implementation of Phoenix PubSub using EventStore, including support for direct_broadcast via the destination_nodemetadata field and pluggable serialization.

The complete implementation can be found at esl/phoenix_pubsub_eventstore.
Need help building reliable distributed systems with Elixir? Get in touch with our team.

The post Implementing a Phoenix PubSub Adapter with EventStore appeared first on Erlang Solutions.

Python NumPy to Elixir-Nx

Wed, 09 Sep 2026 15:35:00 +0000

For my ElixirConfUS talk, I wanted to demonstrate training a spam detection model with in Elixir. A pre-processing step I needed to perform was TF-IDF vectorization, but there were no TF-IDF libraries already written which were built for Elixir-Nx. Seeing as TF-IDF is an extremely common pre-processing step with Decision Trees, since they ingest tabular data, I decided to go ahead and write a full-fledged implementation rather than just writing a minimal implementation that I needed for my example.

I decided to model my implementation after the TF-IDF Vectorizer implementation in the Python scikit-learn, and in writing it I learned many lessons about translating Python NumPy code to Elixir-Nx. Since Nx is to Elixir as NumPy is to Python, I thought others might find it useful to see how we can leverage existing code from the Python ecosystem to bring it Elixir.

The sklearn source code can be found here, while the Elixir source code can be found here.

API Overview

A primary goal I had while writing my Elixir library was to make the API as similar to the Python API as possible so that it would be easy for people coming from Python, since sklearn is the Machine Learning library that most people likely have experience with. The Elixir version supports most of the same arguments as the sklearn version. Here are some examples of basic usage of the API

# Elixir
TfidfVectorizer.new(
    ngram_range: {1, 3},
    sublinear_tf: true,
    stop_words: english_stop_words,
    max_features: 5000
  )
  |> TfidfVectorizer.fit_transform(X_train)

# Python
TfidfVectorizer(
  sublinear_tf=True, 
  ngram_range=(1, 3), 
  max_features=5000
  ).fit_transform(X_train)

Design Overview

We will start by looking at the Sklearn CountVectorizer class since TfidfFVectorizer is actually implemented as a subclass of CountVectorizer and then using the fit method of the TfidfTransformer class to transform the output of the CountVectorizer to its TF-IDF representation. As a result, the bulk of the code is implemented in the CountVectorizer. In Elixir, I accomplished this design by having a CountVectorizer module and a TfidfVectorizer module which has a CountVectorizer as a struct member.

The vectorizer works by building a vocabulary from the given corpus (or using a vocabulary you pass it), counting the number of times each word in the vocabulary is taken, and filtering according to

The vectorizers work roughly according to these steps:

Either builds a vocabulary from the given corpus or uses a vocabulary supplied to it.
a. Performs preprocessing according to a given function
b. Performs tokenization according to a tokenization function
c. Generates requested ngrams
d. Filters stop words
Iterates through each document in the corpus, counting each term in the vocabulary
Limit output features according to parameters
a. max_features - Only consider the top max_features ordered by term frequency across the corpus.
b. min_df - Ignore terms that have a document frequency strictly lower than the given threshold.
c. Ignore terms that have a document frequency strictly higher than the given threshold.
Output CountVectorizer
TfidfVectorizer uses the CountVectorizer to transform the term-frequency matrix into it TFIDF Matrix.

As you can see, the bulk of the work is done in the CountVectorizer module, so that is where we will spend the most of our time going forward. Now that you have a general understanding of how the vectorizers work, we will look at a brief survey of different functions to compare Python and Elixir implementations.

Implementation Details

Here is how the vectorizer is initialized in Python. It uses keyword args with default parameters and initializes its class attributes accordingly.

class CountVectorizer(_VectorizerMixin, BaseEstimator):
  def __init__(
        self,
        *,
        input="content",
        encoding="utf-8",
        decode_error="strict",
        strip_accents=None,
        lowercase=True,
        preprocessor=None,
        tokenizer=None,
        stop_words=None,
        token_pattern=r"(?u)\b\w\w+\b",
        ngram_range=(1, 1),
        analyzer="word",
        max_df=1.0,
        min_df=1,
        max_features=None,
        vocabulary=None,
        binary=False,
        dtype=np.int64,
    )
    self.input = input
        self.encoding = encoding
        self.decode_error = decode_error
        self.strip_accents = strip_accents
        self.preprocessor = preprocessor
        self.tokenizer = tokenizer
        self.analyzer = analyzer
        self.lowercase = lowercase
        self.token_pattern = token_pattern
        self.stop_words = stop_words
        self.max_df = max_df
        self.min_df = min_df
        self.max_features = max_features
        self.ngram_range = ngram_range
        self.vocabulary = vocabulary
        self.binary = binary
        self.dtype = dtype

And here is a snippet of the CountVectorizer struct in Elixir as well as the new/1 function that is used to initialize the vectorizer. new/1 gives a similar behavior to Python's class init method. I used NimbleOptions for parameter validation (it's a great library and you can read more about it here), and you can refer to the parameters source here. validate_shared! validates the parameters and assigns default value when none were provided.

defmodule Mighty.Preprocessing.CountVectorizer do
  defstruct vocabulary: nil,
            ngram_range: {1, 1},
            max_features: nil,
            min_df: 1,
            max_df: 1.0,
            stop_words: [],
            binary: false,
            preprocessor: nil,
            tokenizer: nil,
            pruned: nil

  def new(opts \\ []) do
    opts = Mighty.Preprocessing.Shared.validate_shared!(opts)
    struct(__MODULE__, opts)
  end
end

All of the operations in our CountVectorizer module will take a CountVectorizer as its first argument, which allows us to pipe our operations nicely. The Python implementation separates the TfidfTransformer from the TfidfVectorizer, where the TFIDFVectorizer inherits from the CountVectorizer. To achieve similar behavior, our TfidfVectorizer is its own struct that contains a CountVectorizer as one of its member. Creating a new TfidfVectorizer starts with creating a new CountVectorizer:

defmodule Mighty.Preprocessing.TfidfVectorizer do
  alias Mighty.Preprocessing.CountVectorizer
  alias Mighty.Preprocessing.Shared

  defstruct [
    :count_vectorizer,
    :norm,
    :idf,
    use_idf: true,
    smooth_idf: true,
    sublinear_tf: false
  ]

  @doc """
  Creates a new `TfidfVectorizer` struct with the given options.

  Returns the new vectorizer.
  """
  def new(opts \\ []) do
    {general_opts, tfidf_opts} =
      Keyword.split(opts, Shared.get_vectorizer_schema() |> Keyword.keys())

    count_vectorizer = CountVectorizer.new(general_opts)
    tfidf_opts = Shared.validate_tfidf!(tfidf_opts)

    %__MODULE__{count_vectorizer: count_vectorizer}
    |> struct(tfidf_opts)
  end
end

Now, let's compare three of the main pieces of functionality between the two implementations: building the term-frequency matrix (count matrix), limiting / pruning features from the resulting matrix, and performing the TFIDF transformation on that resulting matrix.

Building Term-Frequency Matrix

defp _transform(vectorizer = %__MODULE__{}, corpus, n_doc) do
  if is_nil(vectorizer.vocabulary) do
    raise "CountVectorizer must be fit to a corpus before transforming the corpus. Use CountVectorizer.fit/2 or CountVectorizer.fit_transform/2 to fit the CountVectorizer to a corpus."
  end

  tf = Nx.broadcast(0, {n_doc, Enum.count(vectorizer.vocabulary)})

  corpus
  |> Enum.with_index()
  |> Enum.chunk_every(2000)
  |> Enum.reduce(tf, fn chunk, acc ->
    Task.async_stream(
      chunk,
      fn {doc, doc_idx} ->
        doc
        |> then(&do_process(vectorizer, &1))
        |> Enum.reduce(
          Map.new(vectorizer.vocabulary, fn {k, _} -> {k, 0} end),
          fn token, acc ->
            Map.update(acc, token, 1, &(&1 + 1))
          end
        )
        |> Enum.map(fn {k, v} ->
          case Map.get(vectorizer.vocabulary, k) do
            nil -> nil
            _ when v == 0 -> nil
            idx -> [doc_idx, idx, v]
          end
        end)
      end,
      timeout: :infinity
    )
    |> Enum.reduce({[], []}, fn
      {:ok, iter_result}, acc ->
        Enum.reduce(iter_result, acc, fn
          nil, acc -> acc
          [x, y, z], {idx, upd} -> {[[x, y] | idx], [z | upd]}
        end)
    end)
    |> then(fn {idx, upd} ->
      Nx.indexed_put(acc, Nx.tensor(idx), Nx.tensor(upd))
    end)
  end)
end

def _count_vocab(self, raw_documents, fixed_vocab):
  """Create sparse feature matrix, and vocabulary where fixed_vocab=False"""
  if fixed_vocab:
      vocabulary = self.vocabulary_
  else:
      # Add a new value when a new vocabulary item is seen
      vocabulary = defaultdict()
      vocabulary.default_factory = vocabulary.__len__

  analyze = self.build_analyzer()
  j_indices = []
  indptr = []

  values = _make_int_array()
  indptr.append(0)
  for doc in raw_documents:
      feature_counter = {}
      for feature in analyze(doc):
          try:
              feature_idx = vocabulary[feature]
              if feature_idx not in feature_counter:
                  feature_counter[feature_idx] = 1
              else:
                  feature_counter[feature_idx] += 1
          except KeyError:
              # Ignore out-of-vocabulary items for fixed_vocab=True
              continue

      j_indices.extend(feature_counter.keys())
      values.extend(feature_counter.values())
      indptr.append(len(j_indices))

  if not fixed_vocab:
      # disable defaultdict behaviour
      vocabulary = dict(vocabulary)
      if not vocabulary:
          raise ValueError(
              "empty vocabulary; perhaps the documents only contain stop words"
          )

  if indptr[-1] > np.iinfo(np.int32).max:  # = 2**31 - 1
      if _IS_32BIT:
          raise ValueError(
              (
                  "sparse CSR array has {} non-zero "
                  "elements and requires 64 bit indexing, "
                  "which is unsupported with 32 bit Python."
              ).format(indptr[-1])
          )
      indices_dtype = np.int64

  else:
      indices_dtype = np.int32
  j_indices = np.asarray(j_indices, dtype=indices_dtype)
  indptr = np.asarray(indptr, dtype=indices_dtype)
  values = np.frombuffer(values, dtype=np.intc)

  X = sp.csr_matrix(
      (values, j_indices, indptr),
      shape=(len(indptr) - 1, len(vocabulary)),
      dtype=self.dtype,
  )
  X.sort_indices()
  return vocabulary, X

The most evident differences here are that in the Elixir code we are building a dense tensor, while the Python code is building a sparse tensor. A sparse tensor certainly makes much more sense in the context of these vectorizers, but as of now Nx currently does not support sparse tensors. This is also why we are using Task.async_stream along with Enum.chunk_every, as to reduce the memory consumption since it is dense.

The way we are constructing the tensor, however, is almost identical. We start by creating a zero-tensor the size of the final tensor. Then we are creating mappings of indices and their updates during our iteration within the reduce. After we collect these updates, we update the initial zero-tensor using Nx.indexed_put, which requires a list of indices you are updating along with the new values you are putting into those indices.

Feature Pruning

defp where_columns(condition = %Nx.Tensor{shape: {_cond_len}}) do
    count = Nx.sum(condition) |> Nx.to_number()
    Nx.argsort(condition, direction: :desc) |> Nx.slice_along_axis(0, count, axis: 0)
  end

  defp limit_features(
         vectorizer = %__MODULE__{},
         tf = %Nx.Tensor{},
         df = %Nx.Tensor{shape: {df_len}},
         high,
         low,
         limit
       ) do
    mask = Nx.broadcast(1, {df_len})
    mask = if high, do: Nx.logical_and(mask, Nx.less_equal(df, high)), else: mask
    mask = if low, do: Nx.logical_and(mask, Nx.greater_equal(df, low)), else: mask

    limit =
      case limit do
        0 ->
          limit

        nil ->
          limit

        _ ->
          limit - 1
      end

    mask =
      if limit && Nx.greater(Nx.sum(mask), limit) do
        tfs = Nx.sum(tf, axes: [0]) |> Nx.flatten()
        orig_mask_inds = where_columns(mask)
        mask_inds = Nx.argsort(Nx.take(tfs, orig_mask_inds) |> Nx.multiply(-1))[0..limit]
        new_mask = Nx.broadcast(0, {df_len})
        new_indices = Nx.take(orig_mask_inds, mask_inds) |> Nx.new_axis(1)
        new_updates = Nx.broadcast(1, {Nx.flat_size(new_indices)})
        new_mask = Nx.indexed_put(new_mask, new_indices, new_updates)

        new_mask
      else
        mask
      end

    new_indices = mask |> Nx.flatten() |> Nx.cumulative_sum() |> Nx.subtract(1)

    {new_vocab, removed_terms} =
      Enum.reduce(vectorizer.vocabulary, {%{}, MapSet.new([])}, fn {term, old_index},
                                                                   {vocab_acc, removed_acc} ->
        case Nx.to_number(mask[old_index]) do
          1 ->
            {Map.put(vocab_acc, term, Nx.to_number(new_indices[old_index])), removed_acc}

          _ ->
            {vocab_acc, MapSet.put(removed_acc, term)}
        end
      end)

    kept_indices = where_columns(mask)

    if Nx.flat_size(kept_indices) == 0 do
      raise "After pruning, no terms remain. Try a lower min_df or a higher max_df."
    end

    tf = Nx.take(tf, kept_indices, axis: 1)
    {tf, new_vocab, removed_terms}
  end

def _limit_features(self, X, vocabulary, high=None, low=None, limit=None):
        if high is None and low is None and limit is None:
            return X, set()

        # Calculate a mask based on document frequencies
        dfs = _document_frequency(X)
        mask = np.ones(len(dfs), dtype=bool)
        if high is not None:
            mask &= dfs <= high if low is not none: mask &="dfs">= low
        if limit is not None and mask.sum() > limit:
            tfs = np.asarray(X.sum(axis=0)).ravel()
            mask_inds = (-tfs[mask]).argsort()[:limit]
            new_mask = np.zeros(len(dfs), dtype=bool)
            new_mask[np.where(mask)[0][mask_inds]] = True
            mask = new_mask

        new_indices = np.cumsum(mask) - 1  # maps old indices to new
        removed_terms = set()
        for term, old_index in list(vocabulary.items()):
            if mask[old_index]:
                vocabulary[term] = new_indices[old_index]
            else:
                del vocabulary[term]
                removed_terms.add(term)
        kept_indices = np.where(mask)[0]
        if len(kept_indices) == 0:
            raise ValueError(
                "After pruning, no terms remain. Try a lower min_df or a higher max_df."
            )
        return X[:, kept_indices], removed_terms

These functions show the most stark differences between the capabilities of NumPy versus those of Nx, as well as just the syntactic differences. The syntax of NumPy is much less verbose that that of Nx (especially considering that we are operating outside of a defn here which will inject its own implementation of the Kernel module to add custom operators), and there are just some capabilities in NumPy that are currently not possible in Nx. For example, in Nx you cannot do dynamic shape modifications like are done in the Python code new_mask[np.where(mask)[0][mask_inds]] = True, so I had to come up with other solutions that could achieve the same thing. Looking at the Python version, you realize that np.where(mask)[0] is only concerned with the resulting columns. This makes sense since each column represents a term in the vocabulary and each row represents a document in the corpus, so each item in a column represents the count of that term in that documents. So we are concerned with whole columns because term-frequency is calculated for each term, which again is represented by the whole column. So we can now use a combination of our function where_columns and Nx functions such as Nx.argsort, Nx.take, and Nx.multiply to rearrange the matrix such that items are sorted by our filter conditions, and then we can just take the number of items we want according to the supplied :limit.

It would take entirely too long for me to go over every difference between these two functions, but I implore you to look closely at these two implementations to gain a better understanding of how to convert NumPy code to Elixir Nx.

TFIDF Transformation

def fit(%__MODULE__{count_vectorizer: count_vectorizer} = vectorizer, corpus) do
    {cv, tf} = CountVectorizer.fit_transform(count_vectorizer, corpus)
    df = Scholar.Preprocessing.binarize(tf) |> Nx.sum(axes: [0])

    idf =
      if vectorizer.use_idf do
        {n_samples, _n_features} = Nx.shape(tf)
        df = Nx.add(df, if(vectorizer.smooth_idf, do: 1, else: 0))
        n_samples = if vectorizer.smooth_idf, do: n_samples + 1, else: n_samples
        Nx.divide(n_samples, df) |> Nx.log() |> Nx.add(1)
      end

    struct(vectorizer, count_vectorizer: cv, idf: idf)
  end

  def transform(%__MODULE__{count_vectorizer: count_vectorizer} = vectorizer, corpus) do
    tf = CountVectorizer.transform(count_vectorizer, corpus)

    tf =
      if vectorizer.sublinear_tf do
        Nx.select(Nx.equal(tf, 0), 0, Nx.log(tf) |> Nx.add(1))
      else
        tf
      end

    tf =
      if vectorizer.use_idf do
        unless vectorizer.idf do
          raise "Vectorizer has not been fitted yet. Please call `fit_transform` or `fit` first."
        end

        Nx.multiply(tf, vectorizer.idf)
      else
        tf
      end

    tf =
      case vectorizer.norm do
        nil -> tf
        norm -> Scholar.Preprocessing.normalize(tf, norm: norm)
      end

    tf
  end

  def fit_transform(%__MODULE__{} = vectorizer, corpus) do
    vectorizer = fit(vectorizer, corpus)
    {vectorizer, transform(vectorizer, corpus)}
  end

class TfidfTransformer(
    OneToOneFeatureMixin, TransformerMixin, BaseEstimator, auto_wrap_output_keys=None
):
    def __init__(self, *, norm="l2", use_idf=True, smooth_idf=True, sublinear_tf=False):
        self.norm = norm
        self.use_idf = use_idf
        self.smooth_idf = smooth_idf
        self.sublinear_tf = sublinear_tf

    def fit(self, X, y=None):
        X = self._validate_data(
            X, accept_sparse=("csr", "csc"), accept_large_sparse=not _IS_32BIT
        )
        if not sp.issparse(X):
            X = sp.csr_matrix(X)
        dtype = X.dtype if X.dtype in FLOAT_DTYPES else np.float64

        if self.use_idf:
            n_samples, n_features = X.shape
            df = _document_frequency(X)
            df = df.astype(dtype, copy=False)

            # perform idf smoothing if required
            df += int(self.smooth_idf)
            n_samples += int(self.smooth_idf)

            # log+1 instead of log makes sure terms with zero idf don't get
            # suppressed entirely.
            idf = np.log(n_samples / df) + 1
            self._idf_diag = sp.diags(
                idf,
                offsets=0,
                shape=(n_features, n_features),
                format="csr",
                dtype=dtype,
            )

        return self

    def transform(self, X, copy=True):
        X = self._validate_data(
            X, accept_sparse="csr", dtype=FLOAT_DTYPES, copy=copy, reset=False
        )
        if not sp.issparse(X):
            X = sp.csr_matrix(X, dtype=np.float64)

        if self.sublinear_tf:
            np.log(X.data, X.data)
            X.data += 1

        if self.use_idf:
            # idf_ being a property, the automatic attributes detection
            # does not work as usual and we need to specify the attribute
            # name:
            check_is_fitted(self, attributes=["idf_"], msg="idf vector is not fitted")

            # *= doesn't work
            X = X * self._idf_diag

        if self.norm is not None:
            X = normalize(X, norm=self.norm, copy=False)

        return X

    @property
    def idf_(self):
        # if _idf_diag is not set, this will raise an attribute error,
        # which means hasattr(self, "idf_") is False
        return np.ravel(self._idf_diag.sum(axis=0))

    @idf_.setter
    def idf_(self, value):
        value = np.asarray(value, dtype=np.float64)
        n_features = value.shape[0]
        self._idf_diag = sp.spdiags(
            value, diags=0, m=n_features, n=n_features, format="csr"
        )

  class TfidfVectorizer(CountVectorizer):
      def fit(self, raw_documents, y=None):
        self._check_params()
        self._warn_for_unused_params()
        self._tfidf = TfidfTransformer(
            norm=self.norm,
            use_idf=self.use_idf,
            smooth_idf=self.smooth_idf,
            sublinear_tf=self.sublinear_tf,
        )
        X = super().fit_transform(raw_documents)
        self._tfidf.fit(X)
        return self

    def fit_transform(self, raw_documents, y=None):
        self._check_params()
        self._tfidf = TfidfTransformer(
            norm=self.norm,
            use_idf=self.use_idf,
            smooth_idf=self.smooth_idf,
            sublinear_tf=self.sublinear_tf,
        )
        X = super().fit_transform(raw_documents)
        self._tfidf.fit(X)
        # X is already a transformed view of raw_documents so
        # we set copy to False
        return self._tfidf.transform(X, copy=False)

    def transform(self, raw_documents):
        check_is_fitted(self, msg="The TF-IDF vectorizer is not fitted")

        X = super().transform(raw_documents)
        return self._tfidf.transform(X, copy=False)

I would consider this the most straight-forward translation example between the two implementations. One noticeable difference is that Nx.log doesn't handle zero values the same way NumPy does. While NumPy essentially ignores zeroes, Nx.log will throw a divide by zero error, so I use Nx.select to selectively ignore zero values, and only apply Nx.log to non-zero values. Additionally, I use Nx.multiply(tf, vectorizer.idf) to achieve the same thing as X = X * self._idf_diag as there is no need to construct a diagonal matrix since Nx.multiply broadcasts.

Conclusion

I would have liked to go into more detail for each example, but I think the code does a good job by itself showing the differences and steps required to translate a NumPy implementation to Nx. I think these examples illustrate how NumPy can obscure what operations are happening in an effort to make a more concise syntax, whereas some might consider Nx overly verbose in comparison. The more you familiarize yourself with both APIs, the better you will be able to identify places where you can do direct translation and places where you might have to be more creative.

Comment below with your own examples or if you have any other Python snippets you want to see converted to Elixir Nx!

Texttile, a multiplayer blog engine for people who write together – Klaus Breyer

Tue, 01 Sep 2026 22:31:03 +0000

My wife and I have blogged about every trip since our honeymoon 10 years ago. For the people at home, for ourselves later, and by now for our children. We took turns writing, but both had photos and videos on their phones.

The text was never the hard part. The photos and videos were: every day one of us sent them from the phone to the other, who had to upload them and sort them into the entry in the right order.

I built imaedge for that part first, and we tried it on the next trip, in Mexico this year. Uploading held up. Tiles arrive in the order the camera gives them, and dragging one changes the order in the gallery.

What was still missing was bringing it together with the text. The family blog ran on WordPress, but WordPress never got this. Not the mobile editing experience, not the video support, not the gallery, and definitely not the “together” part. You are limited to one author per entry (but you get a bunch of plugins nobody maintains and constant bot attacks on your wp-admin).

So after Mexico, with the gallery proven, I did what every programmer apparently has to do once. I wrote my own CMS. It is called Texttile, it is open source, and it is written in Elixir.

It allows multiple people in the same entry at the same time, uploading and sorting photos AND videos, treated as first-class citizens.

What it does differently ¶

In most blog engines only one author can edit an entry. Texttile rethinks content management as multiplayer.

Text: multiple people can have the same entry open. One of them has the text and types, the other watches the words arrive and can take the text over with one click.
Tile: both of you drag tiles with photos and videos into place at the same time, and you see each other doing it. The gallery is never locked.

An entry consists of text and tiles - that is the name.

Both screens show the same entry at the same moment. The writer sees a purple status bar and can edit the text. The other person sees an orange status bar and a read-only editor. Both can still work on the gallery.

Travel shaped the rest:

Texttile loads nothing from outside. No CDN, no tracker, no captcha, no hosted font, no cookie banner. A reader’s browser talks to your server and nothing else.
It stays light enough for a slow line: small pages, little JavaScript, and pictures only as large as the screen asks for.
Videos come from your own server. Drop one in and Texttile converts it, thumbnail included. No YouTube embed, no player from anywhere else.
Texttile stores your Markdown byte for byte. A version diff shows real edits and nothing else.
One container, one folder. Phoenix, LiveView, ffmpeg and SQLite live in one Docker image. Everything is in /data. Move that folder and you move the blog.
Comments, a newsletter, and statistics stay on your server. Texttile uses no cookies and stores no IP addresses.
English and German (more translations are welcome as a contribution!)

What it is not ¶

There are no roles, no permission matrix, no plugins, no theme marketplace. Everybody with an account is an admin. I built it for people who trust each other, because that is who writes a blog together. My philosophy is that a product is only perfect when there is nothing left to take away.

Why Elixir ¶

Multiple people editing one entry at the same time is what the BEAM was made for. The lock is a GenServer, the keystrokes travel over PubSub, and the whole thing runs on one small machine next to a SQLite file. No Redis, no queue, no second service.

Try it ¶

Run it on your own machine:

Or start a demo at www.texttile.blog. You get your own blog for 24 hours. If you like it, you can keep it. If not, it goes to sleep and is deleted 30 days later, with everything in it.

I put entries from our own travel blog from Mexico online at demo.texttile.blog, if you want to see it from the reader’s side.

The code is at github.com/texttile-blog/texttile. Now I am interested in your feedback! How do you blog on the road?

And here it is in action:

Privacy by Design no JurisOS: Criptografia AES-256 de Ponta a Ponta com Cloak.Ecto, SQLCipher e Tailscale VPN - DEV Community

Sat, 29 Aug 2026 15:13:10 +0000

O setor jurídico lida diariamente com dados de altíssima sensibilidade: laudos médicos, segredos industriais, estratégias de litígio, dados bancários e informações de identificação pessoal (PII). Desenvolver um sistema como o JurisOS — que opera sob o paradigma Local-First nos desktops de secretárias e advogados — exige incorporar o conceito de Privacy by Design desde a fundação da arquitetura.

Neste artigo, vamos detalhar como construímos uma malha de defesa em profundidade no ecossistema Elixir e BEAM, utilizando criptografia AES-256-GCM tanto em repouso quanto em trânsito, para garantir conformidade rigorosa com legislações de proteção de dados (LGPD, GDPR e CCPA).

1. O Desafio da Conformidade (LGPD, GDPR e CCPA)

As leis modernas de proteção de dados exigem que os sistemas garantam a confidencialidade e a integridade das informações pessoais contra vazamentos, sejam eles acidentais (perda de um notebook) ou maliciosos (ataques de exfiltração).

Para um aplicativo desktop Local-First sincronizado com a nuvem, a superfície de ataque é dupla:

O endpoint local: O disco do computador físico do usuário no escritório.
O transporte e o backend central: A rede por onde os dados trafegam e o banco de dados principal (PostgreSQL) na nuvem.

Para mitigar esses vetores, o JurisOS adota a Separação de Chaves (Key Isolation) em múltiplas camadas.

2. Separação de Chaves: SQLCipher e Cloak.Ecto

Não basta confiar apenas no acesso do usuário ao sistema operacional. Se um dispositivo for furtado, o banco de dados local não pode ser montado e lido por terceiros. Para resolver isso, separamos as responsabilidades de criptografia.

Camada 1: Criptografia de Disco com SQLCipher (`SQLCIPHER_KEY`)

Na ponta do cliente (o aplicativo desktop empacotado via Burrito), o armazenamento local não é um SQLite padrão, mas sim o SQLCipher. Todo o arquivo .db é cifrado transparentemente utilizando AES-256.
A SQLCIPHER_KEY é gerada no momento do provisionamento da máquina e injetada no ambiente de execução. Isso protege contra ataques físicos: a extração do arquivo do disco resulta em bits indecifráveis sem a chave simétrica correta.

Camada 2: Criptografia de Nível de Campo com Cloak.Ecto (`CLOAK_SECRET_KEY`)

Enquanto o SQLCipher protege o contêiner físico, o Cloak.Ecto protege o conteúdo lógico. Campos críticos que contêm PII (CPF, RG, históricos médicos) e a própria Transaction Outbox de sincronização recebem uma camada adicional de AES-256-GCM a nível de aplicação.

Isso significa que mesmo um administrador de banco de dados (DBA) acessando a instância central do PostgreSQL na AWS, ou um atacante que consiga realizar um SQL Injection, verá apenas ciphertexts binários, não os dados reais.

defmodule JurisOS.Vault do
  use Cloak.Vault, otp_app: :juris_os

  @impl true
  def init(config) do
    config =
      Keyword.put(config, :aes_gcm,
        tag: "AES.GCM.V1",
        key: decode_key(System.fetch_env!("CLOAK_SECRET_KEY"))
      )

    {:ok, config}
  end

  defp decode_key(key) do
    Base.decode64!(key)
  end
end

defmodule JurisOS.Clientes.EncryptedString do
  use Cloak.Ecto.String, vault: JurisOS.Vault
end

Na definição do Schema Ecto, mapeamos os campos sensíveis usando o nosso tipo customizado:

A CLOAK_SECRET_KEY fica estritamente na memória da BEAM. Os dados são cifrados antes de gerar o SQL de INSERT/UPDATE e decifrados logo após o SELECT, mantendo o banco de dados completamente cego quanto ao conteúdo dos campos.

3. Isolamento da Malha BEAM via Tailscale VPN

A comunicação entre os nós de desktop locais e o nó mestre na nuvem requer transporte seguro. Em vez de expor portas públicas na internet (0.0.0.0) e confiar apenas na camada TLS das requisições web, o JurisOS eleva a segurança encapsulando toda a infraestrutura em uma rede Zero Trust.

Utilizando a Tailscale VPN (baseada em WireGuard), cada estação de trabalho e o servidor AWS recebem um IP de uma sub-rede privada (ex: 100.64.0.0/10).

Os Benefícios dessa Arquitetura de Rede:

Superfície de Ataque Reduzida a Zero: O servidor na nuvem rejeita qualquer conexão que não venha de uma interface WireGuard. Não há portas expostas ao port scan público.
Comunicação Nativa Erlang Distribution: Podemos conectar os nós desktop ao cluster principal (libcluster) trafegando mensagens do ecossistema OTP através do túnel da Tailscale de forma totalmente segura.
Criptografia Dupla em Trânsito: Além da proteção nativa do protocolo (como WebSockets sobre WSS), os pacotes TCP/UDP são criptografados pelas chaves WireGuard ponto a ponto.

Conclusão

Proteger dados jurídicos requer mais do que apenas conformidade no papel; exige garantias algorítmicas de segurança.

Ao combinar o SQLCipher para proteção contra acesso físico indevido, o Cloak.Ecto para cegar o banco de dados central perante dados sensíveis (PII) e a Tailscale VPN para isolar completamente a comunicação de rede da internet pública, o JurisOS estabelece um estado da arte em Privacy by Design no ecossistema Elixir.

Essa arquitetura garante que, independentemente da ponta — seja no laptop offline do advogado ou no servidor AWS na nuvem —, a conformidade com a LGPD, GDPR e CCPA esteja tecnicamente assegurada desde o primeiro byte de dados inserido.

Egghead is a note-taking app. - by Mark Wunsch

Fri, 21 Aug 2026 01:08:22 +0000

Egghead is a note-taking app.

There are some who have already decided that the next interesting thing in software is going to call itself “AI-native” or “agentic” or “the cognitive operating system of the future.” Egghead is not those things. Or rather, it is some of those things but only as a consequence of being a note-taking app in 2026.

Generated by ChatGPT Images 2.0

The problem with notes

We take notes because notes outlast thinking. Anyone who has kept a notebook, either physical or digital, for any length of time has had the experience of writing something down only to later be unable to find it. You know the note is there, somewhere, but you can’t, for the life of you, recall where it was or what it said.

This central challenge of keeping notes gets worse with scale. A single notebook is searchable by hand. Ten notebooks are significantly more challenging. A folder of digital notes can be searched, but only if you remember the exact words you used, which you usually don’t because the whole reason you wrote the note in the first place was to externalize the thought so you could stop holding it. The note is supposed to do the remembering for you. Instead it just changed where the forgetting occurs.

The work to create a system of note-taking has lasted nearly as long as the act of note-taking itself. The Renaissance period had commonplace books. The Index Card was popularized by Carl Linnaeus — a guy with strong opinions about structured information. In the 20th Century an obscure German sociologist named Niklas Luhmann took his note-taking seriously enough to build a card catalog of nearly ninety thousand interlinked notes, which he then used to write more than seventy books and nearly four hundred scholarly articles. The system he extensively used was Zettelkasten, which became a subject of his own research into systems theory and prefigured the Wiki.

The 21st century has produced an entire category of software — Evernote, Obsidian, Notion, Roam, Bear, Apple Notes, Logseq — to name just a handful that popped into my head. Each one promising that this time, the notes will stay findable. I know I am not alone in having tried more than one of them, and sticking with it for a nontrivial amount of time before finding another shiny object promising untold cognitive reward.

They all work, but they all suffer from the same limitations. No matter how good the search, or how clever the linking, or how disciplined you are about tagging, the system can only ever give back what you put in. The smartest thing in the room is still you.

This isn’t a failure of any specific tool, but a structural property of the whole category. The limit on what you can do with a notebook is your own memory of what’s in it. Which is, if you’re like me, not very good. Which is why I started writing things down in the first place.

How to write good notes

There is a self-help sub-genre in your nearest global online bookstore dedicated to note-taking systems, and though I have a personal perspective of what makes a good system of notes, I find the more critical thing to answer is where the notes are and how they are made available to you.

The best answer, as of this writing, is the same answer as it was fifty years ago: a series of files written in plain text on your storage disk. Many popular note-taking software applications tend to use proprietary formats in proprietary databases, accessed only through said proprietary application. Plain text files are the computer world’s universal interface, and are a core pillar of the Unix philosophy.

For Egghead, the notes are assumed to live as files in a directory formatted as either Markdown or Org Mode using Wikilinks to denote connections between them. When we circumscribe our written notes to proprietary formats, we reduce our ability to retrieve them. Which as stated earlier, is already challenging enough due to the limitations of our own memory. Plain text first.

Why Artificial Intelligence

For most of recent history, the limitations of software notebooks were reflections of the limitations of physical notebooks: limitations of the human operator. Improvements in metadata, indexing, and search are still fundamentally bound by the user: you can only search for a word that has been explicitly written, and traverse relationships over metadata that the user has embedded. Software could not, in any meaningful sense, build a notebook that read what you wrote and engaged with it as a participant.

Large language models change this. Not because they are intelligent in any deep sense, but because they can read more text than I can hold in my head, and they can produce reasonable language about that text on demand. That is genuinely new.

So the obvious move is to point an AI assistant at your notes and let it go to town.

The problems with AI in knowledge management

This is what the current generation of AI products is doing. Uploading files to ChatGPT, creating project knowledge in Claude, Notion AI on top of your Notion workspace, Cursor in your codebase… They all share the same shape: there is a knowledge base over here and a single AI assistant over there and the assistant can occasionally reach over to read from the knowledge base before answering your question.

The most ambitious version of this shape, and the one that pushed me to build something different, is OpenClaw. An open-source personal AI assistant that you can run on your own machine, talk to from any messaging app, and connect to your files and tools. It’s genuinely good.

But OpenClaw directly demonstrates the limitations of a single AI assistant: it agrees with you.

It has to. There is no structural pressure on it to do otherwise. When you ask it a question, it will give you an answer shaped like the question. You ask a leading question and it follows your lead. The “You’re absolutely right!” reflex is both well-trodden joke material as well as real architectural fact: a single agent, optimizing locally for “be helpful” will reliably converge toward whatever the user seems to want to hear. This is the shape of one-on-one assistance.

The memory of AI assistants

What happens after a conversation with an AI assistant ends? Increasingly, they remember things from conversation to conversation (which wasn’t always the case). OpenClaw, for example, has a MEMORY.md and workspace for persistence.

The dominant pattern for memory in AI tooling is some flavor of retrieval-augmented generation (RAG). Notes and past conversations get chunked into passages, embedded into vectors, and stored. On each new prompt, the harness pulls the chunks that look semantically nearest to the question and stuffs them into the model’s context window.

In April, a more ambitious variant of the same impulse was articulated in Andrej Karpathy’s LLM Wiki pattern, where instead of retrieving from raw sources at query time, an LLM agent incrementally compiles your notes into a structured wiki and then queries that.

Both of these patterns exhibit the same flaw. In RAG, retrieval is shaped by the prompt and the prompt is shaped by you.

RAG works on chunks, not documents — your essay’s argument structure is gone before the agent ever sees it. In the LLM Wiki version, the same bias gets baked in earlier: by the time the wiki entry exists, it’s been passed through the agent’s filter. The summary is cleaner than the source. That’s what makes it a really compelling “memory” product, but not a great note-taking product. Ambiguity in your notes gets edited toward coherence.

So both the shape of a single AI assistant and its memory formation produce a thinking partner who is very capable at creating a confident, well-organized version of what you’ve already decided you wanted to hear.

This is exactly the limitation of human users that leads many to abandon note-taking or continuously migrate from one note-taking system to the next. The notebook can only return what you remember to retrieve. The single AI assistant with RAG memory can only return what your prompt vocabulary aims at. So the assistant-plus-notes shape, even with modern persistence, has two failure modes that compound: a single agent cannot disagree with itself in any structurally reliable way, and a single agent’s memory pulls toward the framing you walked in with. Both failures point at the same fix.

You need more than one, and they need to share notes.

Multi-Agent Systems

The intuition is straightforward and very human: Teams beat individuals on most kinds of knowledge work, especially the kind where the failure mode is groupthink rather than skill gap. Peer review beats self-review. Code review beats no code review. We already know this about humans. We already build institutions around it. A single perspective on its own work has a structural blind spot and the cure is more perspectives.

Apply that to the notebook problem. If a single agent has a tendency to agree with you, the fix is not to find a better single agent. The fix is to put a second agent in the room with a different disposition, and let them disagree with each other where you can watch. The peer review you’d want from a thoughtful colleague, manufactured at the structural level, in real time, on the body of work you actually care about.

Many orchestration frameworks have arrived to multi-agent systems by a different route, but their shape does not lend itself to inherently better results. They default to coordinator and specialists — one agent dispatches tasks, others execute, and results aggregate at the top. This is a star topology — an org-chart fantasy of how teams work.

The Linux kernel mailing list does not have a dispatcher delegating tasks for execution. An organization that adopts Slack does not have an executive function adjudicating every channel message. Even in environments where you would expect a rigid hierarchy — like a nuclear submarine — the strict leader-follower model doesn’t always produce the best outcomes. In my own personal experience working in software teams, the teams that work are ones where the coordination is light and the communication is dense.

The recent MAS research bears this out — graph topologies, where any agent can talk to any other agent, outperform star and tree shapes on collaborative tasks.

The capabilities of agents

Once you commit to a team full of agents working in a shared knowledge store, a security and a quality problem emerge that single-agent systems can (and frequently do) ignore.

“How much should this agent be able to access and perform?” is both a security question and a quality question. It’s a security question because, as has been borne out, agents can and will leak internal secrets to external places or perform destructive actions. More agents means more potential for leakage. It’s a quality question because agents that can do everything tend to converge. Without distinct role definitions, the disagreement that made a multi-agent architecture useful in the first place dissolves into consensus. Capability scoping is the structural pressure that keeps the room from collapsing into agreement.

This is the principle of least privilege applied to agentic systems. The same reason I, as adjunct faculty, do not have the keys to Columbia University’s Network Operations Center. Least privilege improves the quality of the output, because it preserves the structural diversity that makes a team perform better than an individual.

What is Egghead?

Egghead is a note-taking app.

The notes are written as plain-text files in a folder you own.

It uses a loosely-coordinated group of AI agents to continuously read from and contribute to the total knowledge base, producing the most diverse set of inferences about that knowledge by allowing the user to grant each agent a distinct set of capabilities.

Every agent is itself defined as a note, as are the transcripts of their conversations and individual deliberations, giving each note provenance for its creation.

The notes are written as plain text in the filesystem, and the app exposes this and communication with its agents through as many surfaces as possible — the terminal, the web, MCP, and IRC — so that your knowledge is not locked behind any one of them.

The goal of taking notes in Egghead

It is worth restating, we take notes because notes outlast thinking. Simply stated, the goal is to extend our thoughts beyond the storage and time limitations of our own wetware.

The goal is not productivity. The goal is not to get more done. The goal is not to have your meetings summarized, or your emails triaged, or your tasks auto-prioritized. Those are nice. They are not the point. The point — the actual, unfashionable, embarrassing-to-say-in-a-funding-pitch point — is to get smarter. To retain more of what you read. To do more with what you retain. To engage with your own past thinking.

Every “AI assistant” on the market is a productivity tool, by which I mean a tool whose purpose is to enable you to do less of a thing and produce more output. A note-taking app built on this premise would exist to reduce the time you spend with your notes. I want the opposite. I want a tool that makes you spend more time with your notes. That makes the doing of it more rewarding.

Knowledge itself is the outcome. The goal of the practice is – to use a word that has fallen out of fashion in software but used to be considered the most valuable thing a thinking person could accumulate – Wisdom.

Ipsa cognitio fructus

Using Elixir head version with Mise

Sun, 16 Aug 2026 10:18:59 +0000

In 2025 I followed the direction many people headed in and switched my version manager to mise-en-place. It has been going really well for me, but recently I came across a problem which took me a bit to figure out (and few very frustrating conversations with LLM, which did not lead to anything). I wanted to test some code with unreleased version 1.20 of Elixir, straight from git.

How to do it? It’s really simple in retrospect. Here are the steps:

1. Build Elixir

$ git clone https://github.com/elixir-lang/elixir.git 
$ cd elixir
$ make

Verify it’s built correctly:

$ bin/elixir --version
> Elixir 1.20.0-dev (88cbabf) (compiled with Erlang/OTP 28)

2. Link it in mise

mise link elixir@head /path/to/elixir

3. Verify this works

$ mise x elixir@head -- elixir --version
> Elixir 1.20.0-dev (88cbabf) (compiled with Erlang/OTP 28)

And it’s done. Whenever you want to run something with Elixir head, you can just mise use elixir@head.

Of course, this should work in a similar way for everything that is handles by mise version control.

Ecto, on_replace and deferred checks

Sun, 16 Aug 2026 10:18:59 +0000

Today I learned a valuable lesson about how a seemingly simple task can have very rough edge cases, which take hours to solve. It involved Ecto, its associations and on_replace option, and uniqueness checks in the database. Here’s the story.

The problem

Let’s say you are modelling some kind of processes. These processes have steps and the steps have to be executed in a precise order. This is how a database structure for it would look like:

defchangedocreatetable(:processes)doadd:name,:string,null:falseendcreatetable(:steps)doadd:process_id,references(:processes)add:name,:string,null:falseadd:order,:integer,null:falseendend

It is fairly straightforward: the order of steps inside of a process is controlled by an order integer column. Since it is important to always have the order of steps precise, we would like to additionally ensure it by a uniqie index:

createunique_index(:steps,[:process_id,:order])

We also know that the steps are only edited via a parent process. There is a form in the application, where you can edit the process fields and add, change, delete steps for it. The payload sent to a server always includes all the steps. Armed with that knowledge we create Ecto schemas like this:

defmoduleCore.ProcessdouseEcto.SchemaimportEcto.Changesetschema"processes"dofield:name,:stringhas_many:steps,Core.Step,on_replace::delete,preload_order:[asc::order]enddefchangeset(process,params)doprocess|>cast(params,[:name])|>cast_assoc(:steps)endenddefmoduleCore.StepdouseEcto.SchemaimportEcto.Changesetschema"steps"dofield:name,:stringfield:order,:integerbelongs_to:process,Core.Processenddefchangeset(step,params)dostep|>cast(params,[:name,:order])endend

I did that, quite happily, tested a bit with adding, changing and removing some steps. Finally I merged the pull request.

It was only a few hours later when a colleague slacked me that something goes wrong. What he did was reordering the steps, or rather trying to do it. From this structure:

step 1, order: 1
step 2, order: 2

he wanted to go to

step 1, order: 2
step 2, order: 1

I quickly drafted code that replicated the issue:

proc=%Process{}|>Process.changeset(%{name:"Test",steps:[%{order:1,name:"First"},%{order:2,name:"Second"}]})|>Repo.insert!()proc=Repo.preload(proc,[:steps])[s1,s2]=proc.stepss1=%{id:s1.id,name:s1.name,order:2}s2=%{id:s2.id,name:s2.name,order:1}proc|>Process.changeset(%{steps:[s1,s2]})|>Repo.update!()

Attempt to run it resulted in an error:

** (Ecto.ConstraintError) constraint error when attempting to update struct:

    * "steps_process_id_order_index" (unique_constraint)

It took me a while to realize what’s going on: Ecto tried to update the existing steps, one by one. So the first operation was:

Update “order” of step 1 to 2

This left us with the following situation:

step 1, order: 2
step 2, order: 2

This, quite obviously in retrospect, triggered the unique index. You cannot have 2 steps for one process with the same order! It did not matter that right after you were going to update step 2’s order to 1. The index operates here and now, you cannot do that.

The solution

After discovering I was pretty close to writing my own hacky solution to update the steps, including inefficient approach with first deleting all and then creating all anew. Fortunately, in time, I vaguely remembered that there is something called deferrable constraints in PostgreSQL.

Deferrable constraint waits until the end of the transaction with checking if its condition is met. This was exactly what I was looking for! But an index cannot be deferrable in PostgreSQL. Luckily, another very similar construct - a uniqueness check - can.

In the migration above, I had to replace the index creation with this:

execute"""
ALTER TABLE steps ADD CONSTRAINT unique_step UNIQUE (process_id, "order") DEFERRABLE INITIALLY DEFERRED
"""

With this statement we create a constraint on the steps table which checks uniqueness of two columns and not only is deferrable - it is deferred to be checked on transaction commit by default.

This is really all I had to do. The Ecto’s association replacing mechanism started working perfectly.

This proved once again a truth I knew: the database is not a dumb storage you can just gloss over. It always pays off to learn it, its capabilities and intricacies. Because then you end up with a simple change instead of rewriting part of Ecto, but poorly.

Roastidio.us Tagged with elixir

Objects, States, and Stewards: Introducing Generic Workflows to an Existing Project · YokerHood · Aleksei Matiushkin

The Four Pillars of Agnostic Machinery

① Attached Objects: Holding the Leash, Not the Data

② States in Between: Discrete Coordinates

③ Stewardship: Contextual Authorization

④ Notifications TWIMC: To Whom It May Concern

Workflow Templates: Guarding Against Chaos

The Execution Cycle

Bringing It into an Existing Codebase

Trouble Is My Workflow · YokerHood · Aleksei Matiushkin

The Autopsy of a Buzzword

How We Murder State in the Dark

① The Polite Script (The Straight-Line Delusion)

② The Database Stew (Status-Driven Denial)

③ The Amnesiac Message Queue

Case File: The Task Lifecycle (The Finite Trudge)

Case File: Order Fulfillment (The Transactional Gauntlet)

Case File: The Eternal Beat Cop (The Infinite Loop)

The Detective’s Rules: How to Model a Workflow Properly

① The Blueprint Precedes the Brick

② State Is the Coordinate; Payload Is the Luggage

③ Events Knock; They Do Not Dictate

④ Failure Is a Destination, Not a Panic

⑤ One Process, One Life

Epilogue: The Long Goodbye to Spaghetti

Aggressive Regression · YokerHood · Aleksei Matiushkin

The Taxonomy Nobody Bothered to Learn

Unit Tests Are Mostly Mirrors

Elixir Will Not Let You Test Private Functions, and This Is a Feature

Behaviour Is a Virtue, Implementation Is a Vice

And Then the Machines Arrived

Further Specimens from the Wild

The Short Version

The Engine Does Not Win the Race · YokerHood · Aleksei Matiushkin

The Engine Is a Fish With Amnesia

Nobody Buys a Car for the Crankshaft

What the Model Sees: The Miser’s Art of Context

What the Model Is Allowed to Do: The Suspension and the Brakes

The Architecture Is the Point

The Engine Is Not the Product

Anti-Antipatterns in Elixir Library Guidelines

Exploring Programming Languages · YokerHood · Aleksei Matiushkin

Your Orchestrator Is a Finite Automaton in Denial

Make Friends With Your AI Assistant

Mocks Are Your Friends, Not Your Servants

Head of the Agents and Assistants Department

Cure, Four Releases Deep: From FSMs to Furniture

The Long Road to Cure

A Score for an Invisible Orchestra · YokerHood · Aleksei Matiushkin

From proof of concept to a Bluesky DataPlane proxy

A complete DataPlane in Elixir​

The next step: a timeline proxy​

Splitting the DataPlane API​

Where the IDs come from​

Agreeing on time​

Persistence for free​

Cursors and falling back​

What the timeline proxy doesn't solve​

From the timeline proxy to the complete DataPlane​

Monitoring the Health of Oban Applications · Oban Pro

🙋 Is This Only for Elixir?

Taxonomy of Issues

Totally Stuck

Identification

Remediation

Broken Leadership

Broken Notifications

Backed Up

Identification

Remediation

Left Hanging

Identification

Remediation

Knowing is a Fraction of the Battle

Exploring OTP Patterns for Agentic Workflows

Chapter 1: State

A familiar problem, a different way of approaching it

The system needs to remember what happened

Who owns the state?

A complete DataPlane in Elixir

The next step: a timeline proxy

Splitting the DataPlane API

Where the IDs come from

Agreeing on time

Persistence for free

Cursors and falling back

What the timeline proxy doesn't solve

From the timeline proxy to the complete DataPlane

The Closed Core of Bluesky

Camada 1: Criptografia de Disco com SQLCipher (`SQLCIPHER_KEY`)

Camada 2: Criptografia de Nível de Campo com Cloak.Ecto (`CLOAK_SECRET_KEY`)