Roastidio.us Tagged with linux

Weird Things Are Afoot In The Honeypot

Tue, 06 Oct 2026 22:43:24 +0000

Here's something you don't see every day. The logs from my SSH honeypot show someone brute-forcing the password for root and then executing:

ls /data/data/com.android.providers.telephony/databases

This is a strange directory to look for because it's where Android devices store the SQLite databases for SMS messages and contacts. Why would an attacker except an SSH server on the internet to be an Android device? Are there IoT devices based on Android that run SSH servers and also store contacts? If someone knows, please tell me!

Debian Apt Repo for libipt

Tue, 06 Oct 2026 22:43:24 +0000

As part of my Ph.D. research, I play around with Intel Processor Trace a lot. As a result, I frequently use libipt; both as a library for my own software and for the reference programs it includes. ptdump and ptxed are my goto utilities for quickly checking and manipulating traces. They're super useful!

Sadly on Debian and Ubuntu, the default package repositories only have a package for the main library (no pre-compiled program binaries) that is woefully out of date (last update was in 2016). Having to repeatably compile xed and libipt from source quickly got annoying, so I've decided to publish my own repository. I've also made it public in hopes that others will find it useful.

The repository tracks the master branch on libipt and xed, so its packages should always contain the latest code. I've made adding it to apt super easy:

sh-c"$(wget-qO-https://super.gtisc.gatech.edu/libipt.sh)"

It currently has the following libraries:

libxed
libxed-dev
libipt (includes the sideband library)

And the following pre-compiled programs:

ptdump
pttc
ptxed

More information about these libraries and programs is available in their respective documentation. I hope to add more packages in the coming days.

For people interested in learning how to host their own repositories, I built this server using gocd, aptly, and apache.

How ASLR Helps Enable Exploits (CVE-2013-2028)

Tue, 06 Oct 2026 22:43:24 +0000

The other day I was playing around with CVE-2013-2028 along with my peer Hong Hu when we came across something odd: CVE-2013-2028 is only exploitable on 64-bit GNU/Linux when ASLR is enabled. After confirming this observation multiple times, we were left very surprised. How could ASLR possibly worsen the security of an application? Driven by curiosity, we decided to find the root cause of this result. Ultimately, we had to go all the way to the Linux kernel code to find our answer. What we found was a kernel quirk that can't really be called a bug from the kernel's perspective, but does go against the expectations of the user. So without further ado, allow me to share how ASLR can enable the exploitation of applications.

For those unfamiliar with CVE-2013-2028, all that needs to be known is it's an exploitable vulnerability in older versions of nginx stemming from a stack buffer overflow that can be triggered by specially crafted HTTP requests. The bug occurs because an integer provided to nginx by the user that is intended to be an unsigned value is accidentally casted temporarily into a signed value. If an attacker passes a sufficiently large value, the worker thread handling the request will copy too much data from its network socket into a fixed sized buffer causing the stack to get smashed. For the curious reader, a more in-depth analysis is available here and a repository for reproducing it is available here.

So why is this bug only exploitable when ASLR is turned on? We can find the user space answer with a simple strace. If we make a chunked HTTP request and claim the total size is going to be 0xaaaaaaaaaaaaaaaa, nginx's worker will make a recvfrom() system call for 0xaaaaaaaaaaaaaab0 bytes from the network socket. When ASLR is turned on, the Linux kernel will copy our request (which is not actually 0xaaaaaaaaaaaaaaaa bytes long) into the worker's buffer, smashing the stack. However, when ASLR is turned off, the kernel will return -EFAULT and the worker will safely report the error and close the session.

We could stop here, but Hong and I were not satisfied. Why is the kernel returning -EFAULT when ASLR is disabled but not when it is enabled? The space allocated for the stack is the same in both cases, so that can't be the problem. The only obvious difference is ASLR moves the stack's address range to randomize it. When ASLR is disabled, the stack's highest address is placed at the boundary between user and kernel space, which is 0x7fffffffffff in Linux kernels compiled for x86_64. However, 0xaaaaaaaaaaaaaab0 is such a large number it shouldn't matter where the stack is placed. It's not going to fit into the memory segment and it's going to cross the boundary. So what's really happening in the kernel when it handles a recvfrom() system call?

Taking a look at Linux's implementation of recvfrom(), we see the following code:

SYSCALL_DEFINE6(recvfrom,int,fd,void__user*,ubuf,size_t,size,unsignedint,flags,structsockaddr__user*,addr,int__user*,addr_len){structsocket*sock;structioveciov;structmsghdrmsg;structsockaddr_storageaddress;interr,err2;intfput_needed;err=import_single_range(READ,ubuf,size,&iov,&msg.msg_iter);if(unlikely(err))returnerr;sock=sockfd_lookup_light(fd,&err,&fput_needed);if(!sock)gotoout;msg.msg_control=NULL;msg.msg_controllen=0;/* Save some cycles and don't copy the address if not needed */msg.msg_name=addr?(structsockaddr*)&address:NULL;/* We assume all kernel code knows the size of sockaddr_storage */msg.msg_namelen=0;msg.msg_iocb=NULL;if(sock->file->f_flags&O_NONBLOCK)flags|=MSG_DONTWAIT;err=sock_recvmsg(sock,&msg,flags);if(err>=0&&addr!=NULL){err2=move_addr_to_user(&address,msg.msg_namelen,addr,addr_len);if(err2<0)err=err2;}fput_light(sock->file,fput_needed);out:returnerr;}

This code performs two relevant checks. The first occurs in:

err=import_single_range(READ,ubuf,size,&iov,&msg.msg_iter);

And the second occurs in:

err2=move_addr_to_user(&address,msg.msg_namelen,addr,addr_len);

However, we can rule out move_addr_to_user() because it's passed the number of bytes actually fetched from the socket, which is the same in our attack regardless of ASLR. This leaves import_single_range(), which is implemented as follows:

intimport_single_range(intrw,void__user*buf,size_tlen,structiovec*iov,structiov_iter*i){if(len>MAX_RW_COUNT)len=MAX_RW_COUNT;if(unlikely(!access_ok(!rw,buf,len)))return-EFAULT;iov->iov_base=buf;iov->iov_len=len;iov_iter_init(i,rw,iov,1,len);return0;}EXPORT_SYMBOL(import_single_range);

In this function, a sanity check is performed via access_ok() to make sure the number of bytes requested by the caller cannot cause a write that would cross into kernel space. But as we pointed out before, the value nginx's worker is passing here is 0xaaaaaaaaaaaaaab0, which should easily cross the boundary regardless of ASLR. The type size_t is defined as an unsigned 64-bit integer in our case, so access_ok() should be passed 0xaaaaaaaaaaaaaab0, right? Actually, if we look more closely, we can see the following lines enforce a limit on len:

if(len>MAX_RW_COUNT)len=MAX_RW_COUNT;

If we lookup MAX_RW_COUNT, we can see it equals (INT_MAX & PAGE_MASK), which turns out to be a 32-bit value. So in other words, even though recvfrom() allows 64-bit unsigned integer lengths on x86_64, import_single_range() truncates them into 32-bit unsigned integers! On a 64-bit processor, this truncation combined with ASLR's relocation of the stack allows our attack to pass the access_ok() check and smash nginx's stack.

Technically, this isn't a bug from the kernel's perspective because import_single_range() also calls iov_iter_init() with the truncated length. This means recvfrom() can only receive up to the truncated length worth of bytes from the socket and therefore passing the truncated value to access_ok() is safe.

That said, it's a really odd way of implementing this system call. From the caller's perspective, it's not made clear that even though it can pass a 64-bit length, only the lower 32-bits will be considered. Also recvfrom() treats the length as 64-bits all the way through its logic, so it's not immediately obvious that the length is being truncated by MAX_RW_COUNT. Additionally, as Hong and I discovered, there is a security consequence to this choice. Performing the access_ok() check on the truncated length allows network attacks that rely on integer overflow and underflow to succeed where they would otherwise more likely be blocked by the kernel due to a failed system call. We find this to be an interesting consequence since it results from seemingly unrelated design decisions. It is hard to recommend that the Linux kernel developers revise import_single_range() given that the real problem is a bug in nginx and not the Linux kernel itself, but we find this discovery fascinating regardless.

Intel PT Data at Rest: A Compression Experiment

Tue, 06 Oct 2026 22:43:24 +0000

Full Disclosure: I am a researcher in Georgia Tech's ISTC-ARSA, which is funded by Intel. Although I reference two publications that share Xinyang Ge and Weidong Cui as authors, I am neither associated with them nor Microsoft Research at the time of writing.

Intel Processor Trace (PT) is a powerful hardware feature for recording the behavior of CPUs. With it, developers and researchers can monitor the control-flow path taken by threads, hardware interrupts, and more, all with cycle-accurate timing. However, this rich stream of data comes at the cost of size. Depending on what PT is configured to trace, it can output hundreds of megabytes of data per second per core. PT does take steps to save bandwidth by only recording changes in control-flow, excluding redundant high-order bits in target addresses, and compressing returns leading to predictable locations. However, despite this compression, the volume of data is still massive.

As a consequence, much of the work published so far handles tracing in one of two ways. One option is to consume the trace as it is generated. This works as long as the consumer can keep up with the producer, which is the case in the control-flow integrity (CFI) system Griffin. The other common approach is to configure PT to write in a circular buffer. This option is suitable for crash dump analysis systems like Snorlax, which only need a fixed size window into a thread's past.

However, while some applications are feasible using the two previous methods, there are still situations were it is desirable to store the entire trace for postmortem analysis. If nothing else, it is useful for repeatable experiments. With this in mind, I performed a naive experiment last night to explore if more can be done to compress PT traces when the data is at rest. Based on the observations that the compression PT applies is highly localized (i.e. a target address verses the previously recorded target address and a return verses the previously recorded call) and that programs often execute repetitive loops, I hypothesized that even a general purpose compression algorithm should be able to compress traces with a good ratio.

Procedure

The overall idea for the experiment is very simple: gather some PT traces, compress them with a commonly used algorithm, and compare the sizes. For a subject I used the simple HTTP server that comes with Python 2.7 to host a copy of this blog. For each trial I had a crawler request pages from the server for a set duration. Once the time expired, I terminated the server and crawler and stopped the tracing. I then compressed the trace using the GNU/Linux utility gzip, which uses Lempel-Ziv coding. I also fed it through a disassembler that matches the PT packets to the binary's static code to produce a linear sequence of instructions. From this I counted the number of unique basic blocks executed during the trace to serve as a rough proxy for code coverage. To summarize the procedure:

Configure and enable PT tracing.
Start the Python HTTP server.
Start the crawler.
Wait for a specified duration.
Terminate the crawler and server.
Stop PT tracing.
Compress the resulting trace and count the number of unique basic blocks executed.

Results

Comparing the original size of the PT trace to the size after compression produces the above graph. Both plots best match linear regressions and are increasing over time. However, the size of the compressed traces increases at a slower rate than the uncompressed traces, meaning these two plots are diverging as time increases.

Another observation to note is the large volume of trace data produced during the server's startup. This explains why even the shortest trial produced a 1GB trace. For the same reason, counting the number of unique basic blocks turned out to not be useful. The number of new basic blocks executed while serving requests was small.

The next graph shows the relationship between the compressed and uncompressed sizes as a space savings percentage. The plot best fits a linear regression and shows the savings decreasing over time. This is likely due to the design of the underlying compression algorithm, which is intended for general use and does not take into consideration the unique characteristics of PT traces.

To summarize, this experiment shows that more can be done to compress PT traces for storage at rest.

Discussion

It is understandable that the compression used by PT would produce small space savings compared to general compression algorithms given the limitations of hardware memory and Intel's very strict performance overhead requirements. In practice, PT produces an overhead of less than 4% in the worst case, and less than 2% on average. These numbers are based on my own observations and the results published by other researchers. In short, PT has very few clock cycles and very little space available for performing compression.

Another factor that deserves consideration is compression's impact on processing time. For systems that consume PT traces on the fly, the largest source of performance overhead is not PT tracing itself but rather the time spent buffering and consuming it. In CFI, for example, the PT trace has to be matched with the executed code in order to reconstruct control-flow. This is why the authors of Griffin report a 11.9% overhead on the SPECint benchmark despite the 4% overhead of PT itself. Adding better space saving compression could increase this overhead further.

That said, for storing PT traces at rest, more can be done to better conserve space.

Intel Processor Trace, execvp, and ptrace

Tue, 06 Oct 2026 22:43:24 +0000

Lately, I've been playing around with Intel Processor Trace (PT); a x86 hardware feature that allows for complete tracing of process control flows. As part of my research, I've been developing my own Linux driver and user program to control PT.

Tracing can be configured using a handful of model specific registers (MSRs) in the Intel CPU. One useful configuration supported by PT is CR3 filtering. For those readers less familiar with x86 architecture, when a user process is executed, the CPU's CR3 register holds the physical address of the process's page table. Since every process has its own page table, each process will also have a CR3 value that is unique from every other currently scheduled process. By configuring PT to use a CR3 filter, tracing can be limited to a single process.

Early versions of my program could only trace already running processes. I would use the GNU debugger to start the target process and trap its first instruction and then I would manually feed its PID into my program as an argument. The Linux driver would then convert the PID into a CR3 by traversing the process's task structure (virt_to_phys(task_struct->mm_struct->pgd)) and use this address to configure PT (IA32_RTIT_CR3_MATCH). Needless to say, having to manually start and trap the target process got very tiring after repeated tracing.

To simplify tracing a process, I wanted my program to take as parameters the file path of an executable and its arguments and automatically start and trace the process. My first attempt roughly followed this pseudo code:

pid=fork();if(pid==0){// Child process// Wait for parent to signal that PT is readyexecvp(target_program,args);}else{// Parent processenable_cr3_filter(pid);enable_pt();// Signal child that PT is ready}

Easy enough, right? I compiled the program, ran my first trace and got... nothing.

execvp and CR3

So what went wrong? It turns out we can demonstrate the problem with a simple test. Consider this simple C program:

// test_1.c#include<stdio.h>#include<unistd.h>voidpid_to_cr3(){intm_pid=getpid();charpid_str[20];snprintf(pid_str,20,"%d",m_pid);FILE*chardev=fopen("/dev/pid_to_cr3","w");fputs(pid_str,chardev);fclose(chardev);}voidmain(){char*argv[]={"./test_2",NULL};intpid=fork();if(pid==0){// Child processpid_to_cr3();// printed to dmesgexecvp(argv[0],argv);}}

In this example, /dev/pid_to_cr3 is a simple Linux character device that processes can write a PID into and it will print the corresponding CR3 value into the kernel log:

// pid_to_cr3.cstaticunsignedlongpid_to_cr3(intpid){structtask_struct*task;structmm_struct*mm;void*cr3_virt;task=pid_task(find_vpid(pid),PIDTYPE_PID);if((uintptr_t)task<1)return0;mm=task->mm;// mm can be NULL in cases such as kthreads, in which case we want the active_mmif(mm==NULL)mm=task->active_mm;if(mm==NULL)return0;cr3_virt=(void*)mm->pgd;returnvirt_to_phys(cr3_virt);}

After test_1.c passes its PID to /dev/pid_to_cr3, it then uses execvp to overwrite its memory with a new program: test_2.c. This program simply passes its PID to /dev/pid_to_cr3 as well:

#include<stdio.h>#include<unistd.h>voidpid_to_cr3(){intm_pid=getpid();charpid_str[20];snprintf(pid_str,20,"%d",m_pid);FILE*chardev=fopen("/dev/pid_to_cr3","w");fputs(pid_str,chardev);fclose(chardev);}voidmain(){pid_to_cr3();// printed to dmesg}

If we compile these source files and execute test_1, we expect that the PID before and after executing execvp will be the same because execvp causes the kernel to overwrite the caller's own memory. But what happens to the CR3 value? As it turns out:

[ 1757.437572] PID 17319 = CR3 18759503872
[ 1757.438414] PID 17319 = CR3 18826612736

Rather than rewriting the existing caller's page table when execvp is called, the Linux kernel actually allocates and populates an entirely new page table! Since our original PT program was getting the CR3 before the execvp, our trace wasn't including the target program's execution.

ptrace

So how do we get the CR3 value afterexecvp is called by the child? We can't simply have the parent signal the child, like in the first attempt, because any code we give to the child process will be overwritten when execvp is called. The solution instead lies in an OS feature known as ptrace. Using ptrace, we can have the child process attach itself to the parent process for debugging. When execvp is completed, the OS will pause the child and signal the parent. The parent can catch this signal using waitpid(), do whatever it needs to do, and then resume the child. The code looks something like this:

pid=fork()if(pid==0){// Child processptrace(PTRACE_TRACEME,0,NULL,NULL);execvp(args[0],args);}else{// Parent processwaitpid(pid,NULL,0);// Wait for child to complete execvp()enable_cr3_filter(pid);enable_pt();ptrace(PTRACE_DETACH,pid,0,0);// Resume child}

Note that this code detaches the parent from the child once the child has been paused. This causes the child to resume normal execution. If we wanted to continue monitoring the child (for example, to detect fork() or clone()), we could do so.

Making the above modification allows the parent to capture the correct CR3 value and get a complete PT trace.

Getting the CR3 value for a PID in Linux

Tue, 06 Oct 2026 22:43:24 +0000

Writing low level code can be difficult due to the lack of examples on the internet. The answer is generally sitting somewhere in a 3,000 page manual where only the most dedicated programmers will find it.

Last week I had such an experience. Currently my research involves a lot of x86 specific programming and virtual machine introspection (VMI). To test one of the proof-of-concept hypervisors I'm working on, I needed a way to quickly convert Linux PID values into the corresponding value that gets loaded into the CR3 register when that process is executing on the CPU. For those who are unfamiliar with the x86 CPU architecture, I recommend reading this page on Linux x86 page table management. The short story is when a process is executed on an x86 CPU, the CR3 register is loaded with the physical address of that process's page global directory (PGD). This is necessary so the CPU can perform translations from virtual memory address to physical memory addresses. Since every process needs its own PGD, the value in the CR3 register will be unique for each scheduled process in the system. This is very convenient for VMI because it means we don't need to constantly scan the guest kernel's memory to keep track of which process is being executed. Instead, we can just monitor writes to the CR3 register.

However, just tracking changes to the CR3 register doesn't give us much insight into what the guest kernel is doing. This is commonly referred to as the semantic gap problem. In order to cross this gap, we need to map the PID values of the processes we're interested in to their corresponding CR3 values. The following Linux kernel module code snippet does just that:

#include<linux/module.h>#include<linux/kernel.h>#include<linux/sched.h>#include<linux/pid.h>#include<asm/io.h>unsignedlongpid_to_cr3(intpid){structtask_struct*task;structmm_struct*mm;void*cr3_virt;unsignedlongcr3_phys;task=pid_task(find_vpid(pid),PIDTYPE_PID);if(task==NULL)return0;// pid has no task_structmm=task->mm;// mm can be NULL in some rare cases (e.g. kthreads)// when this happens, we should check active_mmif(mm==NULL){mm=task->active_mm;}if(mm==NULL)return0;// this shouldn't happen, but just in casecr3_virt=(void*)mm->pgd;cr3_phys=virt_to_phys(cr3_virt);returncr3_phys;}

It should be noted that while the CR3 register is useful for tracking which process is being executed, it cannot track which thread is executing because threads share memory and therefore will have the same PGD and CR3 value. Keeping track of the scheduling of threads via introspection is a more complicated task and is a topic for another time.

For simplicity I implemented the conversion code as a Linux kernel module. If you're interested in how to do this conversion using pure introspection on an unmodified kernel, you should checkout libVMI's code.

How Orange Helps You Sleep At Night

Tue, 06 Oct 2026 22:43:24 +0000

Originally written for the Syracuse University College of Engineering blog.

Everyone at Syracuse University knows that orange is the very best college color, but who knew it could also help you sleep? Research conducted in recent years has shown that sleep problems are on the rise and one theory gaining momentum points to our electronics as the cause. Studies find that the abundance of blue light produced by our smartphones, tablets, and computer screens has a tangible impact on the chemistry in our bodies that regulates when to wake up and when to go to sleep. This isn’t a problem during the day when the sun naturally produces its own blue light, but staring at our own personal mini sun before bed can make falling asleep much more difficult. So what can we do about it? We could restrict ourselves from staring at screens an hour before bedtime, but the world is a busy place and our nighttime reading isn't always ink on paper. Instead, programmers are experimenting with software that reduces the level of blue light our screens produce after sunset. As the sun goes down, the screen shifts from a bluish glow to an orange tint and then back to blue with the following sunrise—promising a better night’s sleep for those of us that are unable (or unwilling) to give up our screens at night, This software is already publicly available for computers thanks to groups like f.lux, but availability on mobile devices is limited. Luckily, the big companies have taken notice and are taking action. In an upcoming version of iOS for iPhones and iPads, Apple plans to introduce Night Shift. Flip it on and it'll automatically determine when the sun sets and rises in your area and adjust the screen's color accordingly. Just another reason to GO ORANGE.

About The Author

Carter Yagemann ’15 is a master’s student studying computer science in Syracuse University’s College of Engineering and Computer Science. A research assistant in Professor Kevin Du‘s Android security lab, his interests include mobile security and security education. He explores problems such as how to ensure security and privacy in Android inter-component communication. Yagemann is a student member of ACM and IEEE and competes in cybersecurity competitions with the Information Security Club in Syracuse University’s School of Information Studies (iSchool).

The importance of boot partitions in Linux systems.

Tue, 06 Oct 2026 22:43:23 +0000

Over the weekend, the lab I work in experienced a power outage. After power was restored, one of our servers failed to boot. It ultimately became my responsibility to figure out if the server could be repaired and failure wasn't an option because the server was configured (with no backups) to run a bunch of services and hosted lots of data (with no backups) for many users. Typical sysop problem (lol), but our lab has no personnel for managing the systems; so there I was.

In the process of finding and fixing the issue, I learned a lot of specifics regarding how the Grub bootloader and Linux work during system boot, so I decided to document my experience for future reference by others. This documentation will be lengthy, so if you only care about avoiding this type of problem, skip ahead to the Remediation section.

Finding the Problem

The server in question was a Dell Poweredge T620 running Ubuntu. The server consisted of two Intel Xeon processors, about 128GB of RAM, and three 2TB hard drives connected to a RAID controller.

The problem occurred during system start-up. The BIOS would start Grub, and then Grub would produce the error error: attempt to read or write outside of disk hd0. After that, the Linux kernel would start, but shortly after would spit out a stack trace and crash.

My first response was to run a disk check on the hard drives to make sure there weren't any problems with their sectors, but this turned up nothing. The disks were operating normally. This meant that the problem was most likely related to Grub.

Diagnosing the Problem

Since the error message that was appearing was being generated by Grub, the next thing I did was try to manually start the Linux kernel. The easiest way to do this is to press 'c' when the Grub menu appears. This will start a Grub command shell through which operating systems can be manually booted.

Understanding Partitions

Before explaining the commands I used while in the Grub command shell, I'll summarize some basics regarding partitions here.

First, hard drives and their partitions can be accessed in Linux via the "/dev" directory. In most modern Linux systems, hard drives follow a naming convention which starts with "sd" followed by a letter designating the drive. "sda", "sdb", "sdc", etc. Partition names are prefixed with a hard drive name followed by a digit designation. The hard drive "sda", for example, might have the partitions "sda1", "sda2", and so forth in the "/dev" directory.

Grub also uses the notion of disks and partitions, but the naming syntax is a little different. The syntax takes the form of "hd#,#" where the first # represents the disk number and the second # represents the partition on that disk. So "hd0,1", "hd0,2", and so on.

Boot Sequence

While I'm covering general Linux background knowledge, I'll also mention a portion of the boot sequence for Linux because this will also be important for understanding the Grub commands. In most Linux systems which boot using the BIOS (as opposed to newer EFI or UEFI booting) the critical pieces are: BIOS, Grub, initrd, and vmlinux.

BIOS stands for Basic Input Output System and is the first thing the system executes upon receiving power. Once the BIOS is started, it'll perform some basic system checks and then load and execute the bootloader. There are many bootloaders out there, but Linux systems tend to use a particular one called Grub. Grub's job is to load the pieces into memory that are necessary to start the operating system. In the case of Linux, the two important pieces which Grub needs to write into memory are initrd and vmlinux. I won't discuss these files in great detail, but to explain them briefly, initrd is the initial RAM disk for the Linux operating system. What that means is that initrd contains a minimalistic copy of Linux comprising of only the kernel and essential programs. Grub will load it into memory and then execute it and it will start up the full Linux kernel contained in the vmlinux file. Once the full Linux kernel is booted, initrd will unload itself from memory.

Diagnosing Grub

Now that I've covered all the necessary background information, on to the Grub commands.

The first step is to find where the boot directory is on the system. First, list all the partitions on the system:

grub> ls

This will create a list of all the partitions on the system. The next step is to figure out which one contains Grub, initrd, and vmlinux (since we're trying to boot a Linux operating system). This can also be done with the list command:

grub> ls (hd0,1)
grub> ls (hd0,1)/boot

Once we've found the location of the boot files, set that partition as Grub's root directory:

grub> set root=(hd0,1)

I used "hd0,1" in the above commands, but you might find the boot files in a different partition. Either way, the boot files for a Linux system should always be either in the root directory of the partition (if that partition is a standalone boot partition) or in "/boot" (if that partition also holds other files).

After we've found the correct root partition, we next need to give Grub the names of the vmlinux and initrd files:

grub> /boot/vmlinux root=/dev/sda1
grub> initrd /boot/initrd

Note, the vmlinux command needs to be passed the parameter "root". This parameter should be the partition which holds the Linux operating system. This may or may not be the same partition holding the boot files.

Once all the files have been specified, all that's left is to boot the operating system:

grub> boot

At this point, you'll get a basic command shell for Linux. Of course, our lab's server was failing to boot, so this didn't happen. Instead, the error mentioned at the beginning appeared when I tried to execute the initrd command. Because of this, I now know that initrd is the problematic file. So how do we fix this?

Remediation

So what went wrong? Basically, the problem is with the partitions on the server's hard drives. For some reason, when Grub tries to load the initrd file into memory, it reaches the end of what it can read from the hard drive before reaching the end of the file. In our case, the problem is that our RAID controller makes the hard drives appear as a single 4TB drive. This is quite large and the initrd file could reside anywhere in those 4TBs. As it turns out, Grub could not address the memory location of our initrd file and therefore couldn't load it. So the power outage was not the direct cause of our problem. At some point, most likely during an Ubuntu update, the initrd file was modified and ended up in a location on the logical hard drive which Grub can't reach.

The solution to this problem is to keep the boot files to a small partition which resides at the start of the hard drive. For those readers who skipped straight to this section, that's all you need to know. When you install a Linux system, you really should make a dedicated 256MB partition for holding the boot files, despite the fact that most Linux installers do not require you to do this.

In my case, however, I couldn't just reinstall the operating system, so in the following paragraphs I'll describe how I migrated the boot files in my already existing Linux installation into a new dedicated boot partition.

Creating a Boot Partition in an Existing Linux Installation

I started by flashing a copy of Ubuntu to a flash drive and booting it. There are plenty of tutorials on the internet about how to boot Ubuntu from a flash drive, so I'll forgo those instructions here.

The first thing I did was use gparted to create a new partition which will serve as our dedicated boot partition. Since the hard drive is already formatted, doing this requires shrinking the main partition and shifting it 256MB over. This frees up space at the start of the hard drive which can then be formatted into the new dedicated boot partition. If you've never used gparted before, I recommend using the GUI version since it's pretty intuitive. The new partition can be formatted to "ext4" and needs to have the "boot" flag enabled. Completing the shift will take awhile, so you'll probably want to do this overnight.

Once the new partition has been created, we next need to copy the existing "/boot" directory's contents over to the new partition. This can be done by mounting the two partitions while still in the Ubuntu Live USB. In the following commands, sda2 is my main partition and sda4 is the new boot partition:

sudo-s
mkdir/mnt/sda2
mkdir/mnt/sda4
mount/dev/sda2/mnt/sda2
mount/dev/sda4/mnt/sda4
cp-R/mnt/sda2/boot/*/mnt/sda4/
rm-rf/mnt/sda2/boot/
mkdir/mnt/sda2/boot
umount/mnt/sda2
umount/mnt/sda4

Finally, Linux and Grub need to be reconfigured so they know that the "/boot" directory is now in a separate partition. This can be done manually by modifying Grub's "grub.cfg" file and Linux's "/etc/fstab" file, but for simplicity, you can use Boot Repair. If you choose to go the Boot Repair route, make sure to switch the GUI into advance mode and go through all the options. Specifically, you need to make sure the "boot partition" option is set to your new boot partition and the "main operating system"" option is set to your main partition. Also, make sure the "set boot flag" option is pointed at your new boot partition and you can save a lot of time by disabling the "check filesystem for errors" option. If you instead decide to go the manual route, you'll need to manually edit "grub.cfg" looking for every "hd" reference and changing it to point at the correct partitions and the fstab file will need an additional entry to mount your new partition at start-up to the "/boot" directory.

If you do everything correctly, this should fix your Linux system and prevent similar issues from arising in the future.

Conclusion

Even though modern Linux installers do not require you to create a separate partition for the boot files, I recommend doing it anyway, especially if you have large hard drives. Otherwise, you might run into the problem I did.

Installing psad on Raspberry Pi Running Arch Linux - Carter Yagemann

Tue, 06 Oct 2026 22:43:23 +0000

I've been fooling around with IDS and specifically psad and I thought it would be fun to try installing psad on my raspberry pi. Little did I know, installing psad on an ARM processor running Arch Linux with systemd is not a simple process. It took me great effort to get psad running correctly, so I thought I'd take the time to document my struggles in the hopes that this will be useful to someone else.

What is psad?

psad is an intrusion detection system (IDS) which works by monitoring logs generated by iptables (a network firewall common to most Linux distros). You can find more information on psad here.

Scope of this document

The focus of this document is on challenges I ran into while trying to get psad to install and run on a raspberry pi and my solutions. This document does not cover how to configure or use psad. It does cover things which I had to taken into consideration due to the raspberry pi CPU being an ARM processor and due to my OS being Arch Linux with systemd.

Contact

Many of my solutions are hacks and probably suboptimal hacks at that. If you see anything wrong with this guide or have better solutions to the problems I covered here, feel free to contact me at cmyagema@syr.edu.

Other Useful Resources

psad homepage
An installation guide that helped me (Edit: This blog no longer exists)
A guide on configuring psad

Installing psad for ARM from AUR

Since psad is not included in the main Arch Linux repositories, it has to be downloaded, compiled, and built from the AUR repository.

First, create a file (I will name it "list.txt") and write in it the following URLs:

https://aur.archlinux.org/packages/pe/perl-unix-syslog/perl-unix-syslog.tar.gz
https://aur.archlinux.org/packages/pe/perl-iptables-parse/perl-iptables-parse.tar.gz
https://aur.archlinux.org/packages/pe/perl-iptables-chainmgr/perl-iptables-chainmgr.tar.gz
https://aur.archlinux.org/packages/ps/psad/psad.tar.gz

These are the tarballs which we will need from AUR.

Next, run the following commands to untar the tarballs, build them, and install them:

cat list.txt | xargs wget
tar xzvf perl-iptables-parse.tar.gz
cd perl-iptables-parse
makepkg -Acs
sudo pacman -U perl-iptables-parse-1.1-2-any.pkg.tar.xz
cd ..
tar xzvf perl-unix-syslog.tar.gz
cd perl-unix-syslog
makepkg -Acs
sudo pacman -U perl-unix-syslog-1.1-4-any.pkg.tar.xz
cd ..
tar xzvf perl-iptables-chainmgr.tar.gz
cd perl-iptables-chainmgr
makepkg -Acs
sudo pacman -U perl-iptables-chainmgr-1.2-2-any.pkg.tar.xz
cd ..
tar xzvf psad.tar.gz
cd psad
makepkg -Acs
sudo pacman -U --force psad-2.2.3-1-armv6h.pkg.tar.xz

Now if you are lucky, unlike me, this should be all you have to do. However, I ran into many additional problems which is what I will focus on in the next section.

Configuration

As I mentioned earlier, I am not going to cover how to configure psad. There is, however, one configuration which I will mention because it's different from other systems. Namely, the location for syslog is in an usual location because of how systemd logs.

To fix this setting, list the contents of your /var/log/journal/ directory. You should see a directory containing a bunch of letters and numbers and inside that directory should be a file called system.journal. I found that this is the file which psad has to be pointed to.

Once you have identified this path, open /etc/psad/psad.conf and point IPT_SYSLOG_FILE to this file. In my case, this means:

If you want to try port scanning yourself or in general test your psad installation, be mindful that the raspberry pi has very limited computing resources so it might take awhile for your test to reflect in psad's status.

Troubleshooting

wget fails due to certificates

This one is easy, just replace the cat link | xargs wget with cat link | xargs wget --no-check-certificate.

makepkg fails and returns a build error

Try rebooting the raspberry pi. Sometimes not having enough memory can cause the build to fail.

psad is installed, but when I run `sudo psad -S` I get the message `pid file [...]/psadwatchd.pid does not exist`

If you're seeing this towards the top of the output for sudo psad -S:

[-] psad: pid file /var/run/psad/psadwatchd.pid does not exist for psadwatchd on HOSTNAME

Then you probably have the same problem I had.

This was the most painful of the problems I ran into and this was the problem which was big enough to convince me to write this document. If I hadn't ran into this issue (and the systemd logging issue), I wouldn't have bothered writing any of this. The problem in my case was "psadwatchd" wasn't starting for some reason when "psad" started. To confirm this as the source of the problem, run:

ps -A | grep "psad"

If you only see one process called "psad" and no "psadwatchd", then you're having the same problem as me.

The solution I came up for this is very much a hack, but it works decently. Basically, I got around this by making a separate service for psadwatchd.

First, create a new file: /etc/systemd/system/psadwatchd.service

In this file, write:

Next, confirm that you wrote this service file correctly by starting it in systemctl:

sudo systemctl start psadwatchd

If all went as it should, you should be able to execute the following two commands:

ps -A | grep "psad"
sudo psad -S

The first command should return both a psad process and a psadwatchd process. The second command should now show information on psadwatchd and no longer show an error about missing PID files.

Now that you've made a working psadwatchd service file, add this new service to systemd's startup list:

sudo systemctl enable psadwatchd

And that should be it (hopefully).

Last rites for Gentoo's Chromium package [LWN.net]

Tue, 06 Oct 2026 22:42:59 +0000

Firefox maintenance

Posted Oct 6, 2026 18:03 UTC (Tue) by sam_c (subscriber, #139836) [Link]

I was under the impression (I think wrongly?) that anonymous comments could be made with subscription links but it looks like not.

Posting juippis's comment on his behalf then:

Hey, sam asked me to fill in as Gentoo's current Firefox maintainer. Firefox is a lot of work, but I'd like to share few points why I believe it's easier than Chromium. Note that I have close to 0 experience packaging Chromium, I've just aided with some updates previously.

I could write a long detailed post about everything I do with each Firefox release, but since I'm a guest here, I'll try to keep it short.

TLDR as a whole is:

Doing the version bumps properly takes a lot of time - mostly because I test with multiple USE flag combinations, gcc/clang, and simply testing pgo itself takes like an hour to build once. It can be hard, because Gentoo carries few custom patches, e.g. to enable different system-* libraries to be used instead of bundled ones, and those regularly break / need patch rebasing. But upstream is rather welcoming, and bigger issues affecting the whole Linux ecology are usually fixed fast by upstream. Upstream also has a CI building using Ubuntu. We usually find toolchain / system-lib related issues amongst the first ones due to the nature of Gentoo being a rolling-release distro, with newer toolchain available. Some of the Firefox developers seem to use Gentoo, too. Therefore I'd say the relations between upstream and other distros is stronger with Firefox, especially since Firefox is often the default browser in other distributions.

Gentoo offers "two channels" - Firefox-157.0 with testing keywords, and Firefox-153.4.0esr with stable keywords. Thunderbird is identical; Thunderbird-157.0 and Thunderbird-153.4.0esr. Thunderbird releases come immediately after Firefox and sometimes requires TB-specific patches to be added. Firefox recently switched to 2-week release cadence, where it used to be 4 weeks, so the work is now doubled each month. It currently takes me ~20 hours every 2 weeks to do everything related to version bumps (both Firefoxes, both Thunderbirds, Spidermonkey, security bugs...) and I've even skipped a step recently, and am pushing ESR's straight-to-stable (after testing on stable system) to save some time. Most of the hours go into running build-tests, which can be done by leaving the pc to run on its own. There's I'd guess ~3-4 hours of "active" work (without bugs) whenever major releases happen. So there's one less active channel to monitor than what Chromium currently has, but I think handling Thunderbirds make it equal here.

Gentoo also offers different variety outside use flags: you have musl, libcxx, different architechtures (arm, x86, riscv) that all require their own quirks. Luckily the power users with these setups know to provide patches, which I then test on glibc and add them if it's ok. Any rust-related issues are somewhat hard to debug, and you can bet there's plenty. Chromium has similar challenges there, but I'd say the nature of Chromium's bundled toolchain vs. system toolchain makes it harder for everyone even with "normal" setup.

Overall Firefox takes a lot less time to build than Chromium which makes it more bearable. And as sam said, the build system changes affecting us between major releases are usually rather minimal. As the sole Firefox maintainer in Gentoo, I also do feel like there's a bit more community support for Firefox. Often times when a bug for Firefox in Gentoo is filed, I'm not even the first one to reply there. And through collaboration there's been cases where we discovered a bug in GCC directly, which was then fixed in GCC (by GCC developer sam) :)

Janet on x32: 32-bit Pointers, 64-bit Speed, 25% Less RAM | Alex Alejandre

Tue, 06 Oct 2026 22:42:59 +0000

See the Scripts and Results

This is on cachyos (arch, btw) with lib32-glibc lib32-gcc-libs. Adding this to a build script forces libraires like spork to build at 32 bit too!

For mem.janet:

Note, these use Fish and I don’t want to rebuild my site with other language support:

sudo pacman -S time # different from just time
command time -v ./bin/janet mem.janet # command skips the fish version
command time -v ./janet32/bin/janet mem.janet

2000000
       Command being timed: "./bin/janet mem.janet"
       User time (seconds): 0.24
       System time (seconds): 0.05
       Percent of CPU this job got: 100%
       Elapsed (wall clock) time (h:mm:ss or m:ss): 0:00.30
       Average shared text size (kbytes): 0
       Average unshared data size (kbytes): 0
       Average stack size (kbytes): 0
       Average total size (kbytes): 0
       Maximum resident set size (kbytes): 145196
       Average resident set size (kbytes): 0
       Major (requiring I/O) page faults: 0
       Minor (reclaiming a frame) page faults: 33513
       Voluntary context switches: 1
       Involuntary context switches: 9

2000000
       Command being timed: "./janet32/bin/janet mem.janet"
       User time (seconds): 0.47
       System time (seconds): 0.03
       Percent of CPU this job got: 99%
       Elapsed (wall clock) time (h:mm:ss or m:ss): 0:00.50
       Average shared text size (kbytes): 0
       Average unshared data size (kbytes): 0
       Average stack size (kbytes): 0
       Average total size (kbytes): 0
       Maximum resident set size (kbytes): 113716
       Average resident set size (kbytes): 0
       Major (requiring I/O) page faults: 0
       Minor (reclaiming a frame) page faults: 25686
       Voluntary context switches: 1
       Involuntary context switches: 21

I added a 0 in the test

command time -v ./bin/janet mem.janet
                  command time -v ./janet32/bin/janet mem.janet
20000000
        Command being timed: "./bin/janet mem.janet"
        User time (seconds): 2.83
        System time (seconds): 0.46
        Percent of CPU this job got: 99%
        Elapsed (wall clock) time (h:mm:ss or m:ss): 0:03.31
        Average shared text size (kbytes): 0
        Average unshared data size (kbytes): 0
        Average stack size (kbytes): 0
        Average total size (kbytes): 0
        Maximum resident set size (kbytes): 1410668
        Average resident set size (kbytes): 0
        Major (requiring I/O) page faults: 0
        Minor (reclaiming a frame) page faults: 316820
        Voluntary context switches: 1
        Involuntary context switches: 164
        Exit status: 0
20000000
        Command being timed: "./janet32/bin/janet mem.janet"
        User time (seconds): 4.64
        System time (seconds): 0.35
        Percent of CPU this job got: 99%
        Elapsed (wall clock) time (h:mm:ss or m:ss): 0:05.00
        Average shared text size (kbytes): 0
        Average unshared data size (kbytes): 0
        Average stack size (kbytes): 0
        Average total size (kbytes): 0
        Maximum resident set size (kbytes): 1098372
        Average resident set size (kbytes): 0
        Major (requiring I/O) page faults: 0
        Minor (reclaiming a frame) page faults: 238633
        Voluntary context switches: 1
        Involuntary context switches: 193

For e.g. declarative-dsls/tests.janet

98 passed
        Command being timed: "./bin/janet ./lib/janet/bin/judge declarative-dsls/tests.janet"
        User time (seconds): 0.42
        System time (seconds): 0.02
        Percent of CPU this job got: 100%
        Elapsed (wall clock) time (h:mm:ss or m:ss): 0:00.45
        Average shared text size (kbytes): 0
        Average unshared data size (kbytes): 0
        Average stack size (kbytes): 0
        Average total size (kbytes): 0
        Maximum resident set size (kbytes): 61900
declarative-dsls/tests.janet

98 passed
        Command being timed: "./bin/janet ./lib/janet/bin/judge declarative-dsls/tests.janet"
        User time (seconds): 0.70
        System time (seconds): 0.02
        Percent of CPU this job got: 99%
        Elapsed (wall clock) time (h:mm:ss or m:ss): 0:00.72
        Average shared text size (kbytes): 0
        Average unshared data size (kbytes): 0
        Average stack size (kbytes): 0
        Average total size (kbytes): 0
        Maximum resident set size (kbytes): 54108
        Average resident set size (kbytes): 0
        Major (requiring I/O) page faults: 0
        Minor (reclaiming a frame) page faults: 12735

The Hopefully Eventually Ultimate Desktop Linux Troubleshooting Guide: Recovering from Crashes and Freezes

Tue, 06 Oct 2026 16:54:48 +0000

Note: this post is part of #100DaysToOffload, a challenge to publish 100 posts in 365 days. These posts are generally shorter and less polished than our normal posts; expect typos and unfiltered thoughts! View more posts in this series.

I've been using Linux as my main/only operating system for my entire life. While I would like to brag that Linux is more stable than Mac, the simple fact is that it isn’t. This is sad, because it’s great in every other way, and the bugs are mostly fairly surface-level, yet give it a patchy reputation among normal users.

“Oh no! That Was The Backup…”, ¬Oil on ¬Canvas

Because I spend a lot of time in the terminal, and all my friends also run Linux, I often get called in to help when their systems have issues. Often, I fail to solve the problem, which is frustrating. So over the past few years I have taken some time to learn more about recovering from all the issues that need recovering from, and collecting my techniques. Here are some of the most useful tools and approaches I have found.

Over the years, I have used Knoppix, Ubuntu, Linux Mint, Manjaro, NetBSD, OpenBSD, and others. These tips are mostly for Ubuntu, Linux Mint, and other Debian-based distros. I have used Xfce, Compiz, CWM, Cinnamon, Sway, Mate, and Gnome. These are mostly about Gnome and Cinnamon.

Note: this article is a living document that I intend to add to as I learn more and have more time. If you have a specific problem, write in or drop a comment, and I will prioritize that section.

This post pairs well with the more high-level advice in Troubleshooting: A Skill That Never Goes Obsolete.

If a program has crashed

killall PROGRAM_NAME (usually, killall firefox)

Scary black screen

If all you see is text, there's a goodish chance that it’s a problem with the graphics drivers. Try re-installing them. If you're using the proprietary Nvidia drivers (which are a nightmare in my experience) try using the open Nouveau driver. Also try switching to a different Nvidia driver if one is available. Also, try fully purging and re-installing Nvidia drivers. If you're using the Nouveau driver, try switching to Nvidia drivers. If you're using Xorg, try switching to Wayland. If you're using Wayland, try switching to Xorg.

sudo apt purge *nvidia* # potentially dangerous
sudo ubuntu-drivers install # follow prompts

If you’re using a Macbook, good luck. In my experience, despite what everyone says, Macbooks have some of the worst Linux support. The four Macbooks I’ve encountered that were running Linux were all unmitigated problem cases, and graphics drivers were the weakest point. Once I spent seven hours getting a friend’s Macbook back to graphical functionality after an Ubuntu upgrade. I breathe a sigh of relief when Macbooks finally conk for good, and ardently recommend the owner upgrade to a ThinkPad if they want me to keep doing tech support.

Google the issue + the model of your laptop. Often you can edit /etc/default/grub and add something to the GRUB_CMDLINE_LINUX_DEFAULT=... line. Often that something is nomodeset. This is also often how I fix issues with screen brightness and backlight.

Once you get back to functionality, you can try to figure out why the specific configuration you had before broke, by looking in the logs. Or if you're happy with the new config, just forget about it and use what works.

If the mouse slows down and gets jittery, and the fans rev up, and the I/O light blinks fast or goes solid, and then the system fully freezes

This usually means that:

Somebody opened too many tabs in Firefox (likely something JavaScript heavy like Reddit, YouTube, or Facebook)
The computer ran out of RAM
It started swapping
Now it’s crashed

We can stop this at any point in the chain, but the farther up, the better.

Don’t open so many tabs. Or, use about:unloads to unload the heaviest tabs when you start to notice slow downs. If you catch it when the mouse starts to lag, immediately hit alt-f4 to quit the most resource intensive app that won't lose work (usually your browser), then wait. Do not press close over and over again: it will just make the computer mad. If you manage to get a heavy app closed before the system becomes unresponsive, it will likely recover, and you can take further steps.

You can also start the culprit app (usually a browser or media transcoding workload or LLM) with a systemd memory limit, which generally prevents it from causing this issue. (Or you can try to renice the app, but this doesn't seem to be as effective.)

Here’s how to run an app with a 4GB memory limit:

systemd-run --scope -p MemoryMax=4G --user COMMAND_NAME

(Change COMMAND_NAME to firefox, ffmpeg, llama-server, or whatever is taking the system down with it.)

There’s also ulimit, but I haven't used it as much.

The simplest fix: BUY MORE RAM. Years ago, going from 8gb to 12gb of RAM on my ThinkPad went from daily seizures of this type to annual seizures. Basically cured!

Bad ram (or badly-seated ram) can also be the culprit.

On servers with limited RAM, adding swap can save the day, but I find that when a desktop Linux system starts swapping, that’s pretty much the end of the good times and the beginning of the bad times.

An SSD might help, because swap speed will be closer to RAM speed. (An SSD is one of the most effective ways to make your computer go faster, other than adding more RAM.)

There are various settings related to swapping behaviour ("swappiness") and overcommiting memory that may help (see here), but I haven't actually used them on desktop Linux.

If you don't manage to get the app closed, try to get to a console. Try: opening a terminal (likely won't work), alt+f2 (also likely won't work, because it's part of the window manager which is crashed), and then dropping to a virtual console with Ctrl+Alt+f2. Press each key slowly, on after the other, holding down the previous keys: the response time is way slower than a normal keyboard. The exact f-key varies: generally Ctrl+Alt+f1 gets you to a login session screen, which you don't want, and the main session is on Ctrl+Alt+f5 or Ctrl+Alt+f7, so I find that f2 or f3 are most likely to get me to a virtual terminal.

Enter username and password. Now you should be able to run killall firefox (or ffmpeg, or ollama, or whisper-stream, or whatever it is). You can also inspect what's crashed with top or htop. htop is good stuff! (and also makes it easy to send stop signals with f9 and then a number; in this case, likely 9) If you can get to the virtual console, you can usually can rescue an out-of-ram freeze caused by a specific app.

But the most powerful technique I've found for this is enabling Magic System Request keys, and then using Ctrl+Alt+PrintScr+f at the first sign of trouble to activate the OOM_KILLER to terminate the most resource-intensive process (usually a firefox tab). It works wonders.

Check your current sysrq value with:

cat /proc/sys/kernel/sysrq

Edit the kernel.sysrq numerical value in the alphabetically last file listed by this command:

ls /etc/sysctl.d/ | xargs -I{} grep -l kernel.sysrq /etc/sysctl.d/"{}"

(After rebooting, check that the value stuck using the previous command.)

Then you can see what it quit with:

dmesg -T | egrep -i 'killed process'

... and hope it wasn’t anything important.

If it’s the desktop environment that has crashed

If you’re running Cinnamon (Linux Mint’s default desktop environment), try running this from a virtual console:

DISPLAY=:0 cinnamon --replace

If you’re using Gnome (on Xorg), you can try these, but I have never managed to restart a Gnome session without it quitting all open apps, and losing data:

pkill -3 gnome-shell
sudo pkill -9 ^gnome-shell

If you know how to, let me know!

If you’re unclear about the difference between a window manager, a desktop environment, a display server, and a distro(bution) are

I’m a bit fuzzy on this, too! Specifically, the where the line is between a window manager and a desktop environment. But the general stack goes, from low-level to high level:

Operating System Family (ie, Linux, BSD, Mac, Windows) → Distro (ie, Linux Mint, Ubuntu, NetBSD) → Display Server (X Window System, Wayland) → Desktop Environment (ie, Xfce, Gnome, Cinnamon, KDE) → Window Manager (Compiz, Openbox, Sway, i3) → Theme & Settings (these days I just use the default, but I used to regularly break my system trying to make it look fancy).

If your screen goes all colorful or develops lines and weird patterns

If you’re using an external monitor, likely the cable connector isn’t seated correctly, or the contacts are dirty.

If it’s a laptop, it’s most likely an issue with the graphics drivers, or the screen connector. I have had this happen a few times, usually related to GPU intensive workloads and suspend, but I haven’t found the root cause, because I encounter it so rarely. All I know is that restarting Cinnamon from a virtual console doesn’t help.

If it has fully crashed and rebooted on its own

Oof, this is serious. In my experience this is fairly rare. I/O issues on peripherals have caused this in the past (ie, excessive power draw from a dying hard drive). Check the logs for insight.

Suspend issues

These often have to do with the graphics drivers, and the specific hardware. Sometimes adding grub command line options helps. Often changing drivers helps.

If you want to know what a program is doing

sudo strace -p$(pidof firefox) -e trace=write -s1000 -fp 18386 2>&1 | grep -o '".\+[^"]"'

(replace firefox with the name of the program you want to know about)

If a program has a bug, this is useful for viewing the error message / stack trace, without quitting the program and re-running it from terminal.

If you want to know what is hogging all the resources

My go-to is the wonderful htop. top is installed by default. glances and btop can also be helpful.

For figuring out suspiciously heavy network or disk write activity, try iotop and nethogs. Also, powertop, if you want to tune battery life.

If you want to know whether a script actually quit or crashed and left pieces churning along in the background

Run htop and search with f3. I often find that a script called some process, then crashed, and left a heavy process still running. This mostly happens when you're developing (ie, for me, Bash/Python scripts that called Whisper C++ or Selenium), or using really bad quality software. Also, viruses.

If you're stuck in Vim

:q!

If the screen is black

Is the backlight on?

Try plugging in an external monitor. This will tell you whether it's your graphics drivers, or hardware.

If external monitor works, it could be your backlight (common), or your screen connector. (These are both pretty easy to fix if you have a repairable laptop like a thinkpad.) Or, it could be the LCD panel itself.

It also could be a hardware/driver issue that can be fixed with the grub command line options.

If the Wifi doesn't work

Try wired internet. If it doesn’t work either, the problem isn’t the Wifi drivers.

Make sure you can get internet from the network on a different device.

Check what the output of ifconfig looks like.

Try running sudo service NetworkManager restart (fixes it surprisingly often!)

If your WiFi hotspot has no internet in it

Maybe it’s your VPN, or firewall. But in my case, it was broadcasting on channel 13, which is only supported in Japan.

I was able to fix it with:

nmcli con modify "Hotspot" 802-11-wireless.band bg # band must be set. Options, in my case, were "a" and "bg". I do not know the difference, but "bg" worked.
nmcli con modify "Hotspot" 802-11-wireless.channel 6 # set the channel
nmcli con up "Hotspot" # bring it up

(Change "Hotspot" to your network name or UUID.)

If the printer won’t print

Try rebooting the printer.

Try a different USB port!

Try clearing the print queue and re-sending the job (very often works!)

Make sure you sent it to the right printer (there are often multiple entries for the same printer, only one of which works.)

Try running sudo service cups restart. This works quite often.

If your mouse or touchpad has stopped moving

Open terminal and run this command to reload the drivers:

sudo modprobe -r psmouse && sudo modprobe psmouse

This usually fixes it.

If it doesn’t: try switching to Synaptics drivers if you’re using libinput drivers, and to libinput drivers if you’re using Synaptics drivers.

If you're having trouble installing software

Try running sudo apt update to make sure you have up-to-date package cache. Look at the error message when you run sudo apt upgrade or sudo apt install PACKAGE_NAME_HERE. The error messages here are often more helpful than the ones you get from the GUI.

Remove old PPAs etc that you don't need any more.

Try running sudo apt install --fix-missing; this usually fixes it.

If an app installed with snap has mysteriously disappeared, it might be in the process of being updated.

If you get Error Splicing File errors

Your external media or internal HDD are likely full! This is a terrifying situation that happens all the time. Many things will cease to function, because they require tmp space, even things that you would expect to keep working.

Find out if your internal root disk is full with:

df -h /

Here are some commands for finding big files that you might be able to delete:

du -a 2>/dev/null | sort -n -r

du --max-depth=1 /path | sort -r -k1,1n

find /home -type f -exec du -s {} \; | sort -r -k1,1n | head

du -a -h /path | sort -h -r | head -n 10

At the time of writing, my 1TB internal drive is 100% full, with 7gb free. Wish me luck!

If the issue is with external media (thumbdrive, HDD)

If it doesn't show up in file manager, check in Gnome Disks; if it doesn't show up there, check in the logs (and also try it on a different computer, and a different USB port, and try a known-good flashdrive/HDD on the computer in the same USB port to isolate the problem).

Errors tend to show up in dmesg. Grep for the manufacturer name (ie ADATA), or for usb or device. Sometimes the actual device is rebranded, so it will say one thing, but will show up as a bigger manufacturer (often Toshiba).

This will help get the /dev/sdX, and then you can go from there.

If a SD card / flashdrive got corrupted

If it is critical, take it to a specialist. This will cost a bundle. Otherwise:

Create a disk image with ddrescue
Do all further operations on a copy of the disk image, not on the disk image, and definitely not on the device itself (read/write intensive operations might damage it more)
Use fsck to repair corrupted file systems. Often it works. (It sometimes leaves garbage fragments of deleted files.)
Use photorec to recover photos and videos that fsck can't recover
Use untrunc to piece back together corrupted videos (as long as you have a good video from the same camera)
^^^ these two tools can save rashers and rashers of bacon. Consider donating to the maintainers (and also, the bloggers who told you about them)

Free advice: never buy no-name-brand ultra-good-price storage from Ebay or Amazon. Never. When it comes to storage, cheap can be very, very expensive.

Check the logs

The logs are the troubleshooter’s best friend. On Linux, the good stuff is mostly in /var/log

There, you will likely find logs for all kinds of things. The most interesting ones:

dmesg (includes peripherals, startup stuff. #1 stop)

syslog (#2 stop)

kern.log (kernel stuff, obviously)

Xorg.log (X Window System)

Probably there will be multiple files for each log, because logs are rotated. So you'll have dmesg, dmesg.0, dmesg.1.gz, dmesg.2.gz, etc, etc.

Five ways to view a log:

cat filename (print whole thing to console. most basic)
open in a text editor vim filename. Useful for searching, reading at leisure, filtering, etc. Some text editors will choke on very large files (gedit, for example). Vim is pretty good at opening gigantic text files.
view top of bottom part with tail or head (ie, tail -n 10 filename to view last ten lines), or watch a file live with tail -f filename, then do the thing that might cause the error message, and see what gets output.
search in a file grep -i fail syslog or in the whole log directory (which tends to be slow) grep -i fail
journalctl is the fancy new thing that I don't really know how to use. It's supposedly great. Try it with journalctl -u cron or journalctl --since=-1hour. It can filter by process ID, executable name, date (including since-last-boot), urgency, and more
Bonus option: print it out on a thousand sheets of paper like a real hacker and then inspect it with a magnifying glass

The main things I do:

Just browse through all the logs and look for something that looks suspicious, or related to the problem
Tail the most-likely-to-be-relevant log with -f (follow) and then do the thing that triggers the bug and see what comes out of the log (very effective if the bug is reproducible, ie, an external drive won't mount, or a program crashes on startup)
Explicitly grep for error messages. These can take several forms. Sometimes it says Error, and sometimes it says Err: ... sometimes it says Failure to X and sometimes it says Failed to start Y. Other watchwords are "segfault", "panic", "fatal", "crash", (core) "dumped". You can find them all with:

grep -i -E "fail|fatal|err|panic|segfault|segmentation fault|dumped|core dumped|killed|warn|bad" syslog # grep case insensitive with extended regex for any of the following

Befriend the terminal

You won't get far troubleshooting Linux crashes without getting used to running commands.

Command history (up arrow) and completion (tab) and reverse history search (ctrl+r) and copying-and-pasting (ctrl+shift+c and ctrl+shift+v instead of ctrl+c and ctrl+v) are essential.

Also, normal keystrokes like Ctrl+Z and Ctrl+S do weird things in the terminal. Avoid.

Learn a console text editor

You will also want to get a decent grasp of a fully console-based text editor that is installed everywhere – Vi(m) or Nano. Nano is simpler, Vim is more powerful.

Just practice editing files with them for fun (write me an email in one!), and then you'll know how to open files, navigate around, edit, save, and quit, when it matters.

Also:

System files generally need to be opened with sudo (ie, sudo vim path/to/file.conf or sudoedit path/to/file.conf). If you don't, you might make all your changes and then not be able to save!

If you're in Vim when this happens, you can use :w !sudo tee %. (In Helix, it’s %:pipe-to sudo tee <Ctrl-r>%, but heed this warning about the importance of that first percentage sign to select all.)

General advice

Search the forums.

If necessary, post to the forums. Try to do your homework first. If people trying to help you are rude in their replies, try to ignore the tone and make use of the advice they give, be polite and grateful, and maybe do more homework next time.

The Arch Linux wiki is helpful even you don’t run Arch. Arch users have encountered nearly every bug under the sun. It tends to be pretty technical, though.

Reboot the machine.

Always backup first.

Try to understand commands before running them (but in practice sometimes you have to fly blind).

Unplug anything you don't need plugged in.

Try to figure out what changed that might have triggered the problem.

Beware "Partial Upgrades".

Do a fresh install from time to time.

Write down the exact time the problem occurred (so you can match it to log timestamps). Also, make sure to account for potential timezone differences: sometimes logs are in UTC.

Read my essay on troubleshooting.

Read the docs/manual for the software in question.

Keep trying.

Conclusion

If this article helped you, share it with a friend. If it didn’t help you, share it with an enemy. <3

If you know more than me, send me corrections and/or invective. If you know less than me, send me money and/or praise.

If you have an idea of how to fix the problem, try it. If you have no idea how to fix the problem, look it up. If the idea you had generated a new error message, you are either getting closer or farther away from solving the problem. Google the new error message.

If nothing works, keep trying. If everything works, switch to BSD.

EOF

Montray - a tray icon for systemd service health

Tue, 06 Oct 2026 16:33:05 +0000

Comments

Por qué hoy uso Fedora · El blog de Lázaro

Tue, 06 Oct 2026 11:58:11 +0000

Si llevas leyendo El Blog de Lázaro un tiempo, sabrás que llevo muchos años utilizando Linux como mi sistema principal. He vivido varias “eras” distintas delante de la pantalla, y todavía recuerdo el boom de Ubuntu a mediados de los 2000, cuando Canonical te enviaba los CD de instalación completamente gratis a casa.

Era otra época: pedías el disco, esperabas unas semanas y, cuando llegaba el sobre, era casi como recibir un regalo de Reyes.

Aquello hizo que mucha gente, incluida la que no tenía ni idea de lo que era Linux, se atreviera a probarlo.

Muerte de las pequeñas

Con los años he visto nacer proyectos muy ilusionantes y también he visto desaparecer muchos. Es casi inevitable en un ecosistema tan vivo como el de Linux. A lo largo del tiempo se han ido desapareciendo nombres que en su momento sonaban fuerte: Mandrake, luego Mandriva, Antergos, Devuan, CrunchBang y unas cuantas más que hoy sólo sobreviven en capturas de pantalla y en la memoria de quienes las usaron.

Durante mucho tiempo esto no me importaba demasiado. Las distros iban y venían, y yo iba saltando de una a otra con la misma alegría con la que cambias de tema de fondo de pantalla.

El espejismo de la popularidad

Si miras hoy el top de DistroWatch, verás un montón de distribuciones muy populares, muchas de ellas sin una empresa grande detrás: CachyOS, EndeavourOS, MX Linux, Manjaro, Garuda, Linux Mint, Arch Linux, todas muy queridas por la comunidad y con un empuje impresionante. A excepción de Debian, que sí tiene una historia y una estructura muy particular, la mayoría se apoyan sobre todo en el trabajo voluntario de desarrolladores y colaboradores.

Eso está genial mientras el proyecto está vivo y la comunidad tiene fuerzas. El problema es qué pasa dentro de cinco o diez años.

Las grandes que siempre resisten

Con el tiempo he ido viendo un patrón muy claro: las distribuciones que aguantan el paso de los años suelen ser las grandes. Ahí están SUSE, Fedora y, por supuesto, la madre de todas las distros, Debian, que lleva décadas sirviendo de base a medio ecosistema Linux.

Todas ellas tienen algo en común: detrás hay estructuras organizadas y empresas serias. Eso no las hace perfectas, pero sí les da una continuidad que empiezas a valorar mucho más que a la novedad.

La búsqueda de la estabilidad

Cuando era más joven, no me importaba ir saltando entre distros. Me encantaba ese momento de instalar desde cero, pelearme con algún driver, afinar la configuración y dejar todo “niquelado”. Si algo se rompía después de una actualización, me lo tomaba casi como un reto personal.

Hoy no. Hoy enciendo el ordenador y quiero que arranque, que mi entorno esté tal y como lo dejé ayer y que no tenga que perder la tarde solucionando problemas. Ahora prefiero dedicarla a trabajar, escribir, aprender o simplemente disfrutar.

Por qué elijo Fedora

En ese cambio de prioridades es donde entra Fedora. Lo que me da Fedora es precisamente esa combinación: un sistema moderno, con software relativamente reciente, pero con una base muy sólida, bien pensada y respaldada por una comunidad enorme y por Red Hat detrás. No necesito que sea la distro más top de DistroWatch; necesito que sea la que menos ruido hace en mi día a día.

Ya no me apetece andar reinstalando cada dos por tres ni persiguiendo pequeñas errores después de cada actualización importante. Con Fedora siento que puedo actualizar con bastante tranquilidad y seguir con mi vida. No es perfecta (ninguna lo es), pero se ha ganado mi confianza.

Lo que hace realmente grande a Linux

Y, a pesar de todo esto, hay algo que quiero dejar claro: que yo haya elegido Fedora no significa que sea “la distro correcta” ni que el resto no merezcan la pena. En realidad, lo que hace a Linux tan interesante y tan grande es precisamente su variedad de opciones. Hay distribuciones minimalistas, otras para principiantes, algunas casi experimentales, otras pensadas sólo para servidore y otas ultra ligeras para equipos antiguos.

No todos queremos lo mismo, ni trabajamos igual, ni tenemos las mismas prioridades. Hay quien disfruta probando una distro nueva cada mes y hay quien prefiere instalar una vez y olvidarse durante años. Hay quien valora la novedad por encima de todo y quien, como me pasa a mí ahora, valora por encima de todo la estabilidad y la tranquilidad.

Mi elección

Al final, al igual que nosotros cambiamos con el tiempo, también lo hacen nuestras necesidades y siempre habrá una distribución que encaje con la etapa en la que estamos. En mi caso, hoy esa pieza se llama Fedora. Mañana, ya veremos.

Espero que te haya gustado, pasa un buen día. 🐧

Forgot Your Root Password? Here’s the Easy Way (and the Hard Way) to Recover It

Tue, 06 Oct 2026 11:45:38 +0000

It happens. No judging here. You created a 32-character password chock full of special character goodness and you thought you’d made a note in your password manager but somehow, you didn’t. You didn’t setup sudo or doas, and now you need to do something as root and you can’t.

Oh no!

But not all is lost. There’s at least two ways to fix the issue.

Using Your Panel

The first thing to check is if the provider’s panel supports root password resets. It’s almost like hosting companies get so many requests to reset root passwords that the hosting panel providers built it in to save them headaches…hmmm.

I logged into Solus on RackNerd and here is what I see:

So let’s try that.

After clicking change, you get this message:

And then after clicking Yes:

(Don’t worry – I’ll change it before this article is published!)

After a little processing, it’s successful:

Now in my case, there was only one problem. As part of my VPS setup, my Ansible scripts put this in /etc/ssh/sshd_config:

PermitRootLogin without-password

So it actually was not possible for me to login as root. In this particular case, losing the root password didn’t actually matter because I could login with my ssh key, but you get the point. If I’d lost the ssh key, I’d have been hosed.

Using Rescue

Let’s look at the SolusVM rescue mode:

Clicking that gives a warning:

I clicked “Enable Rescue Mode” and SolusVM politely asked me to confirm:

Yes, please.

After clicking Yes:

I let it boot a bit, and then was able to login from my PC:

$ ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null root@<my IP>

Whoa…what’s up with all those -o flags? If you don’t use them, you’ll get the dramatic warning that “IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!”

The reason is that the IP is in my hosts file as associated with a certain host key, and when I boot in rescue mode, its’s a completely different OS image. So my ssh client gets a totally different host key and throws the flag. The -o flags I listed ignore that temporarily, without having to go in and modify the known_hosts file.

Now after logging in, it’s important to note that I am not logged into my VPS per se. I’m logged into a rescue image. For example:

rescue # cat /etc/hostname
rescue

If I was to run passwd root now, I’d be changing the rescue image password, which is pointless.

So we have a little work to do. First, find the actual root partition of your VPS:

rescue # mount | grep "on / "
/dev/vdb1 on / type ext3 (rw,relatime,discard,errors=remount-ro,data=ordered)
rescue # lsblk -f
NAME   FSTYPE LABEL UUID                                 MOUNTPOINT
sr0                                                      
vda                                                      
├─vda1 ext4         a09aaadc-93fd-4c32-af29-f007d7c429f6 
└─vda2 swap         ab346e4e-071f-491e-a512-2dcabaff20bd 
vdb                                                      
└─vdb1 ext3         0e8b889c-8efb-4093-9d42-817ae3ca1f90 /
rescue #

OK, so the rescue environment is /dev/vdb1. /dev/vda1 is my actual root. So let’s prep a chroot environment for that.

rescue # mount /dev/vda1 /mnt
rescue # mount --bind /dev  /mnt/dev
rescue # mount --bind /proc /mnt/proc
rescue # mount --bind /sys  /mnt/sys
rescue # mount --bind /run  /mnt/run

And then enter the chroot environment:

rescue # chroot /mnt /bin/bash

Now I am in my actual VPS. For example:

root@rescue:/# cat /etc/hostname
vpn-chi-1
root@rescue:/#

And now I can reset the root password:

root@rescue:/# passwd
New password: 
Retype new password: 
passwd: password updated successfully
root@rescue:/# exit
exit
rescue # reboot

The post Forgot Your Root Password? Here’s the Easy Way (and the Hard Way) to Recover It appeared first on LowEndBox.

Taming GCE Memory Tax: Disabling OS Login and OS Config on Small Instances // Tony Metzidis

Tue, 06 Oct 2026 11:34:46 +0000

Google Compute Engine’s e2-micro instances are a staple of the GCP Free Tier, offering 2 vCPUs and roughly 1 GB of RAM (969 MiB usable). They are ideal for lightweight bastions, cron runners, and small test beds. However, if you run memory-sensitive workloads—such as Go builds, package extractions, or continuous process forks—you may have experienced intermittent, baffling SSH connection timeouts and unresponsive terminals.

The root cause is rarely your application alone. By default, GCE provisions a suite of Google guest daemons that consume a disproportionate share of resident memory on small VMs, creating an environment primed for SSH exhaustion.

The Anatomy of the 15% Memory Tax

Running a bare Debian 12 Bookworm instance on an e2-micro without any user workload active reveals an astonishing baseline:

Process / Daemon	Command Name	Resident RAM (RSS)	% of 1 GB RAM
Google OS Config Agent	`google_osconfig`	~61.8 MB	~6.2%
Guest Agent Core Plugin	`core_plugin`	~27.7 MB	~2.8%
Guest Agent Manager	`google_guest_ag`	~27.2 MB	~2.7%
Guest Compat Manager	`google_guest_co`	~19.7 MB	~2.0%
Guest Telemetry Extension	`guest_telemetry`	~18.7 MB	~1.9%
Total Google Guest Footprint		~155.1 MB	~15.6%

On a 16 GB node, a 155 MB footprint is negligible noise (< 1%). On a 1 GB instance without swap space, 15.6% of your physical memory is permanently claimed before your application even boots.

Why OS Config Hoards 62 MB of RAM

Static analysis of the GoogleCloudPlatform/osconfig repository reveals why google_osconfig claims over 60 MB alone:

Embedded Vulnerability Scanners: Rather than calling out to an external CLI tool and exiting, osconfig embeds Google’s osv-scalibr scanner directly in-process. It parses package databases (dpkg, rpm, pip, gem) into memory trees.
Heavy Protobuf & gRPC Slices: Every inventory cycle instantiates fresh gRPC client transports and serializes thousands of package messages into contiguous memory chunks.
Go Runtime Retention: The Go garbage collector runs, but memory marked as free (madvise(MADV_DONTNEED)) is not eagerly returned to the kernel RSS table. It holds its high-water mark indefinitely.

The SSH Starvation Cascade: How OS Login Fails Under Load

When your application executes builds or unpacks files, kernel VFS slab caches (dentry, ext4_inode_cache) grow dynamically. When total physical RAM nears exhaustion, Linux enters severe memory pressure (surfacing in /proc/pressure/memory).

If OS Login (enable-oslogin=TRUE) is active, incoming SSH connections must complete the following chain:

sshd receives a TCP connection on port 22.
sshd executes AuthorizedKeysCommand (/usr/bin/google_authorized_keys).
The PAM stack loads /lib/security/pam_oslogin_login.so.
PAM performs IPC calls or local HTTP round-trips to the GCE metadata server (169.254.169.254) to verify IAM permissions and POSIX mappings.

Under memory stall conditions (PSI some avg10 > 2.0), fork() calls stall, metadata socket round-trips lag, and the SSH client hits its default ConnectTimeout (5–10s). The user sees:

ssh: connect to host 34.xx.xx.xx port 22: Connection timed out
# or
Connection reset by peer / Permission denied (publickey)

Conversely, with standard SSH authorized keys, sshd opens local /home/<user>/.ssh/authorized_keys directly with a single disk read, bypassing external IPC entirely.

Reclaiming Your RAM: Disabling OS Config & OS Login

If you don’t use GCP VM Manager (automated patch management dashboards in the Cloud Console) or IAM-based OS Login, disabling them recovers immediate memory and makes SSH bulletproof.

Option 1: Project-Level Metadata (Recommended)

You can enforce this across all instances in a project with two gcloud commands:

# 1. Disable OS Config and OS Login project-wide
gcloud compute project-info add-metadata \
    --project="my-project" \
    --metadata=enable-osconfig=FALSE,enable-oslogin=FALSE

# 2. Add your static SSH public key to project metadata
gcloud compute project-info add-metadata \
    --project="my-project" \
    --metadata="ssh-keys=tonymet:$(cat ~/.ssh/id_rsa.pub)"

Note: Disabling OS Login allows the standard google-guest-agent to automatically read the project-wide ssh-keys metadata and write /home/tonymet/.ssh/authorized_keys on every instance.

Option 2: Inside the Operating System

If you prefer to disable the services directly inside the VM:

# Stop and mask the OS Config service
sudo systemctl stop google-osconfig-agent
sudo systemctl disable --now google-osconfig-agent
sudo systemctl mask google-osconfig-agent

# Optional: Completely remove the package
sudo apt-get purge -y google-osconfig-agent

Telemetry Comparison: Before vs. After

Using our synthetic stress harness on an e2-micro running Debian 12 (pinned 250 MB heap anchor, 10,000 file/dentry cycles, and continuous process execution churn):

Metric	Agents Active (Control)	Agents Stopped (Treatment)	Improvement
Baseline Resident RAM	~325 MB used	~170 MB used	~155 MB recovered
Max Memory Stall (`some avg10`)	2.37	0.62	-73.8% pressure stall reduction
Max Full Memory Stall (`full avg10`)	1.74	0.38	-78.2% full stall reduction
SSH Probe Failure Rate	0 timeouts	0 timeouts	Sub-second latency

Key Takeaways

Know Your Guest Footprint: Google guest agents provide cloud-native conveniences, but their ~155 MB overhead is disproportionately taxing on free-tier 1 GB instances.
OS Config is Optional: Disabling it only removes centralized fleet patching and inventory dashboards from the Cloud Console; basic VM operations, networking, and standard updates remain 100% intact.
Simpler Auth is Resilient Auth: On constrained systems, local authorized keys eliminate IPC round-trips to the metadata server, ensuring reliable console access even when the system is under stress.

Installing CuBox-i firmware on Debian Linux (CuBox-i1, CuBox-i2, CuBox-i2eX, or a CuBox-i4Pro)

Tue, 06 Oct 2026 10:01:55 +0000

After installing Debian Linux on the CuBox-i1, CuBox-i2, CuBox-i2eX, or CuBox-i4Pro, additional firmware files are required for Wi-Fi, SDMA, and hardware-accelerated video decoding. Specifically, the VPU needs proprietary firmware, the Wi-Fi module relies on symlinking files from an official Debian package, and the SDMA engine depends on the non-free firmware package. For context, the following dmesg output from a CuBox-i4Pro demonstrates the missing firmware warnings:

coda 2040000.vpu: firmware: failed to load vpu_fw_imx6q.bin (-2)
coda 2040000.vpu: Direct firmware load for vpu_fw_imx6q.bin failed with error -2
coda 2040000.vpu: firmware: failed to load vpu/vpu_fw_imx6q.bin (-2)
coda 2040000.vpu: Direct firmware load for vpu/vpu_fw_imx6q.bin failed with error -2

coda 2040000.vpu: firmware: failed to load v4l-coda960-imx6q.bin (-2)
coda 2040000.vpu: Direct firmware load for v4l-coda960-imx6q.bin failed with error -2

brcmfmac mmc0:0001:1: firmware: failed to load brcm/brcmfmac4330-sdio.solidrun,cubox-i-q.bin (-2)
brcmfmac mmc0:0001:1: firmware: failed to load brcm/brcmfmac4330-sdio.solidrun,cubox-i-q.bin (-2)
brcmfmac mmc0:0001:1: firmware: failed to load brcm/brcmfmac4330-sdio.solidrun,cubox-i-q.bin (-2)
brcmfmac mmc0:0001:1: Direct firmware load for brcm/brcmfmac4330-sdio.solidrun,cubox-i-q.bin failed with error -2

This article explains how to install the missing firmware.

Firmware 1: The integrated Video Processing Unit (VPU)

The CuBox-i computers includes an integrated Video Processing Unit (VPU) for hardware-accelerated video encoding and decoding. The Debian non-free repositories do not package the proprietary Video Processing Unit firmware files. You can download the files directly from the Armbian firmware repository.

For the CuBox-i2eX or CuBox-i4Pro VPU

Run the following command as root to download the Video Processing Unit firmware for the CuBox-i2eX or CuBox-i4Pro:

wget -O /lib/firmware/vpu_fw_imx6q.bin https://github.com/armbian/firmware/raw/refs/heads/master/vpu/vpu_fw_imx6q.bin
wget -O /lib/firmware/v4l-coda960-imx6q.bin https://github.com/armbian/firmware/raw/refs/heads/master/v4l-coda960-imx6q.bin

For the CuBox-i1 or CuBox-i2 VPU

Run the following command as root to download the Video Processing Unit firmware for the CuBox-i1 or CuBox-i2:

wget -O /lib/firmware/v4l-coda960-imx6dl.bin https://github.com/armbian/firmware/raw/refs/heads/master/v4l-coda960-imx6dl.bin
wget -O /lib/firmware/vpu_fw_imx6d.bin https://github.com/armbian/firmware/raw/refs/heads/master/imx/vpu/vpu_fw_imx6d.bin

Firmware 2: SDMA

The mainline i.MX6 device tree specifies imx/sdma/sdma-imx6q.bin for the SDMA engine (Smart Direct Memory Access). If the external firmware is unavailable, the Linux driver can fall back to firmware embedded in the SoC ROM. The external firmware provides additional and updated SDMA scripts for peripherals that use SDMA.

To provide the required firmware files, install the firmware-misc-nonfree package:

apt-get install firmware-misc-nonfree

Once the package is installed, update the initramfs to ensure the binaries are included in the boot image:

update-initramfs -u -k all

Firmware 3: Broadcom Wi-Fi firmware

Step 1: Install the Broadcom Wi-Fi firmware package

Install the firmware-brcm80211 package:

apt-get install firmware-brcm80211

The firmware-brcm80211 package provides the generic firmware file. The Linux driver can additionally request a board-specific firmware and NVRAM file based on the device tree.

Step 2: Configure the missing Broadcom Wi-Fi firmware files

Run the following commands as root to symlink the installed firmware and fallback configuration to the board-specific filenames:

ln -s /lib/firmware/brcm/brcmfmac4330-sdio.bin /lib/firmware/brcm/brcmfmac4330-sdio.solidrun,cubox-i-q.bin
ln -s /lib/firmware/brcm/brcmfmac4329-sdio.bin /lib/firmware/brcm/brcmfmac4329-sdio.solidrun,cubox-i-q.bin
ln -s /lib/firmware/brcm/brcmfmac4330-sdio.bin /lib/firmware/brcm/brcmfmac4330-sdio.solidrun,cubox-i-dl.bin
ln -s /lib/firmware/brcm/brcmfmac4329-sdio.bin /lib/firmware/brcm/brcmfmac4329-sdio.solidrun,cubox-i-dl.bin

Step 3: Download the CuBox-i Broadcom NVRAM configuration

The Linux brcmfmac driver expects a board-specific NVRAM file. Download the configuration provided by SolidRun and install it under /lib/firmware/brcm/.

wget -O /lib/firmware/brcm/brcmfmac4330-sdio.solidrun,cubox-i-q.txt https://raw.githubusercontent.com/SolidRun/meta-solidrun-arm-imx6/jethro/recipes-bsp/broadcom-nvram-config/files/solidrun-imx6/brcmfmac4330-sdio.txt
wget -O /lib/firmware/brcm/brcmfmac4329-sdio.solidrun,cubox-i-q.txt https://raw.githubusercontent.com/SolidRun/meta-solidrun-arm-imx6/jethro/recipes-bsp/broadcom-nvram-config/files/solidrun-imx6/brcmfmac4329-sdio.txt
wget -O /lib/firmware/brcm/brcmfmac4330-sdio.solidrun,cubox-i-dl.txt https://raw.githubusercontent.com/SolidRun/meta-solidrun-arm-imx6/jethro/recipes-bsp/broadcom-nvram-config/files/solidrun-imx6/brcmfmac4330-sdio.txt
wget -O /lib/firmware/brcm/brcmfmac4329-sdio.solidrun,cubox-i-dl.txt https://raw.githubusercontent.com/SolidRun/meta-solidrun-arm-imx6/jethro/recipes-bsp/broadcom-nvram-config/files/solidrun-imx6/brcmfmac4329-sdio.txt

Step 4: The WiFi regulatory domain

The NVRAM file needs to have the correct WiFi regulatory domain set. This can be done by setting the value of the ccode parameter to the country code in which the device will be operating in. For example, to set the WiFi regulatory domain for the United States:

sed -i 's/^ccode=.*/ccode=US/' /lib/firmware/brcm/brcmfmac4330-sdio.solidrun,cubox-i-q.txt
sed -i 's/^ccode=.*/ccode=US/' /lib/firmware/brcm/brcmfmac4329-sdio.solidrun,cubox-i-q.txt
sed -i 's/^ccode=.*/ccode=US/' /lib/firmware/brcm/brcmfmac4330-sdio.solidrun,cubox-i-dl.txt
sed -i 's/^ccode=.*/ccode=US/' /lib/firmware/brcm/brcmfmac4329-sdio.solidrun,cubox-i-dl.txt

Note: Replace 'US' with your country code.

Reboot

Rebooting the system will allow the drivers to load the firmware files and clear the initialization failures.

BIOS settings for performance on older ThinkPads, such as T420, T420s, T520, W520, X220, T430...

Tue, 06 Oct 2026 10:01:55 +0000

Older Lenovo ThinkPad models such as the T420, T420s, T520, X220, W520, and T430 provide BIOS settings that control CPU power management, processor features, storage controllers, and boot behavior. Adjusting some of these settings can improve responsiveness or reduce boot time.

CPU and Power Delivery

CPU Power Management

Path: Config -> Power -> CPU Power Management
Setting: Disabled.

Disabling CPU Power Management turns off the BIOS feature that stops the processor clock during periods of inactivity. The benefit is that the operating system gains unrestricted control over processor state transitions, reducing latency.

The tradeoff is higher baseline power consumption when the system is idle. Ensure your operating system is configured with an appropriate CPU frequency scaling governor, otherwise the processor will constantly run at its maximum clock speed.

Intel SpeedStep Technology

Path: Config -> Power -> Intel SpeedStep Technology
Setting: Set Mode for AC and Mode for Battery to Maximum Performance.

Setting both AC and battery operation to Maximum Performance selects the highest-performance SpeedStep mode rather than the battery-optimized mode. (Note: This can increase CPU power consumption and battery usage.)

CPU Core Utilization

Path: Config -> CPU
Settings: Ensure both Core Multi-Processing and Intel Hyper-Threading Technology are Enabled.

These settings allow the operating system to use all available physical cores and logical processors.

Boot Time Reduction

Boot order

Path: Startup -> Boot
Setting: Move the primary OS drive (SSD/HDD) to the top of the boot priority list.

Placing the primary OS drive near the top of the boot order can reduce delays caused by firmware attempting earlier boot targets before reaching the operating system drive. The tradeoff is requiring manual intervention via the boot menu to boot from external media.

Boot mode

Path: Startup -> Boot Mode
Setting: Quick.

The benefit is a reduction in total system startup time, accelerating the transition from the Lenovo splash screen to the OS bootloader.

Storage Controller Optimization

Path: Config -> Serial ATA (SATA)
Setting: Change Controller Mode Option to AHCI.

AHCI is generally the appropriate SATA controller mode for modern Linux and Windows installations. (Note: Older operating systems must have AHCI drivers.)

Simple sudo settings to prevent unwanted Linux or UNIX privilege escalation: Per-command sudo authentication and path restriction

Tue, 06 Oct 2026 10:01:55 +0000

The default sudo configuration caches successful authentication using a timestamp, allowing commands to execute without a password prompt for a limited period. This can cause security risks, such as local privilege escalation if an attacker accesses an unlocked terminal session or if a malicious script automatically escalates its own privileges within the cached timeframe.

Additionally, restricting the execution path ensures that sudo only executes verified, system-installed packages.

Here is how to configure sudo to require per-command authentication and restrict the execution path for commands.

Edit the sudoers file

Start modifying the /etc/sudoers configuration file by using the following command as root:

visudo

Editing /etc/sudoers file using visudo locks the file and checks the syntax before saving changes.

Requiring password authentication for every sudo command

Set timestamp_timeout to zero in the /etc/sudoers file to enforce per-command authentication:

Defaults timestamp_timeout=0

The benefit is that an attacker accessing an unlocked terminal after a legitimate sudo command cannot reuse that authorization. For example, if a system administrator runs sudo apt update and temporarily steps away from their desk without locking the screen, a malicious insider cannot walk up to the active terminal and type sudo cat /etc/shadow to extract password hashes, as the system will immediately demand the password again.

Additionally, it prevents both background processes and user-executed programs from exploiting a cached session to automatically escalate their privileges:

In a legitimate but unintended context, a user might run a standard development command like npm install or make on a third-party project. If the project contains build hooks configured to modify system directories or install global binaries, an active sudo session allows the script to silently execute these elevated changes, completely bypassing the user's opportunity to review or approve the action.
In a malicious context, a script running silently in the background could monitor for an active sudo timestamp. Once the user legitimately authenticates, the script exploits the cached token, for example, to append an attacker's SSH key to /root/.ssh/authorized_keys without generating a visible password prompt.

The tradeoff is requiring repeated password entry when running several privileged commands interactively.

Restricting the execution path

The secure_path directive replaces the user's PATH with a fixed value during a sudo execution:

Defaults secure_path="/usr/sbin:/usr/bin:/sbin:/bin"

Update these directories to match your operating system and local administrative tools.

Validating the recommended configuration

If you choose to modify files in /etc/sudoers.d/ instead of using the visudo command, validate the configuration by running the following command as root to parse the sudo configuration files for syntax errors:

visudo -c

Conslusion

These sudo configurations provide explicit, deterministic control over privilege escalation. Forcing authentication for all elevated actions and locking the execution path to verified system directories eliminate the risk of cached credentials and manipulated environment variables. This ensures every command executed as root becomes an intentional, manually authorized event.

Installing Debian on a CuBox-i computer (CuBox-i1, CuBox-i2, CuBox-i2eX, or CuBox-i4Pro)

Tue, 06 Oct 2026 10:01:55 +0000

The CuBox-i is a series of compact, fanless computers, by SolidRun, that feature a cube-shaped enclosure.

Installing Debian Linux on CuBox-i series computers, such as the CuBox-i1, CuBox-i2, CuBox-i2eX, or a CuBox-i4Pro, requires specific bootloader preparation.

After spending four hours troubleshooting multiple failed installations, I found the reliable solution documented here. This article details a deployment strategy using the Debian hd-media offline installation method over a serial console connection, avoiding the network-based installer and HDMI issues. After booting the installed Debian system, the network and other hardware work as expected.

Requirements

Computers:

A Linux computer used to prepare the storage media and access the CuBox serial console.
A CuBox-i series computer (CuBox-i1, CuBox-i2, CuBox-i2eX, or a CuBox-i4Pro).

Accessories:

Micro-USB Data Cable: A standard Micro-USB to USB-A cable capable of data transfer (not a charge-only cable). This connects the host computer to the CuBox micro-USB service port to access the serial console, enabling terminal access during boot and installation.
USB Flash Drive: A drive used to store the Debian ISO for offline installation, such as "debian-12.x.x-armhf-netinst.iso".
MicroSD Card: Required to boot the Debian installer and will serve as the storage medium where Debian will be installed. A minimum of 4 GB is recommended.

Packages:

screen: Install the screen, to connect to "/dev/ttyUSB0", using: apt-get install screen
dosfstools: Install the mkfs.vfat command-line tool using apt-get install dosfstools

Known Debian installer issues on the CuBox-i computers

Before beginning, it is important to understand why the Debian network-based installation method can fail on this hardware:

Network installer issues: Some Debian installer versions have had problems detecting or configuring the CuBox-i Ethernet interface.
HDMI display issues: The Debian installer can fail to provide a usable HDMI display on some CuBox-i models. Using the serial console avoids depending on HDMI output during installation.

The solution is to perform an offline installation using the hd-media method while operating the system through a serial console over USB.

Preparing the MicroSD card

Requirement: A MicroSD card. This card is required to boot the Debian installer and will serve as the storage medium where Debian will be installed. A minimum of 4 GB is recommended.

The CuBox-i requires a device-specific U-Boot bootloader to initialize the hardware.

Download the following files:

firmware.MX6_Cubox-i.img.gz
http://deb.debian.org/debian/dists/stable/main/installer-armhf/current/images/netboot/SD-card-images/firmware.MX6_Cubox-i.img.gz
partition.img.gz
http://deb.debian.org/debian/dists/stable/main/installer-armhf/current/images/netboot/SD-card-images/partition.img.gz

Note: The links above download both files from the Debian stable (/stable/) version.

Then run the following commands as root to write the firmware and partition images directly to the SD card:

zcat firmware.MX6_Cubox-i.img.gz partition.img.gz > /dev/sdX

Warning: This command will erase all data on /dev/sdX. Verify that /dev/sdX is the correct SD card device before running it.

Note: Replace /dev/sdX with the exact target device identifier. (The SD card identifier will typically appear as /dev/sdX for USB adapters or /dev/mmcblkX for native readers.)

Finally, flush file system buffers to ensure all data is written to the device:

sync

This creates the Debian installer image from the device-specific U-Boot portion and the installer partition image.

Preparing the offline installation media (USB Flash Drive)

Requirements:

A USB flash drive. A drive used to store the Debian ISO for offline installation, such as "debian-12.x.x-armhf-netinst.iso".
A Linux computer used to prepare the storage media.

The hd-media installer can use a Debian ISO image stored on a separate USB drive.

Step 1: Download the netinst ISO image: https://cdimage.debian.org/debian-cd/current/armhf/iso-cd/

(For example, debian-x.x.x-armhf-netinst.iso, where x.x.x represents the current version number from the official image repository.)

Step 2: Copy the ISO as a normal file onto the USB filesystem. (Do not use block-level writing tools like dd for this step.) Run the following as root:

# Format the first partition of the USB drive to FAT32
mkfs.fat -F 32 /dev/sdX1

# Mount the newly formatted partition to a temporary directory
mkdir -p /mnt/debian-usb
mount /dev/sdX1 /mnt/debian-usb

# Copy the Debian netinst ISO file as a standard file into the FAT32 partition
cp debian-x.x.x-armhf-netinst.iso /mnt/debian-usb/

# Safely unmount the USB drive
umount /mnt/debian-usb

# Flush buffers to prevent data corruption on large file transfers
sync

The Debian installer can then locate and mount the ISO from the USB drive.

Note: The Debian hd-media installer kernel on the MicroSD card must be the same release version as the kernel inside the .iso file. Mixing daily installer builds with stable ISO images will result in a kernel mismatch error. As a rule of thumb, if a stable Debian (current) ISO is downloaded, the hd-media kernel should also be downloaded from the corresponding stable Debian release rather than from the daily installer builds.

Reboot and start installing Debian

With both storage media prepared:

Insert the prepared MicroSD card into the MicroSD card slot on the CuBox-i.
Insert the FAT32 USB flash drive containing the Debian .iso file into one of the on the CuBox-i USB ports.
Ensure the micro-USB data cable connects the CuBox-i service port to the host Linux machine.
Unplug and plug the power adapter into the CuBox-i.

Accessing the serial console

Requirement: A Micro-USB data cable. A standard Micro-USB to USB-A cable capable of data transfer. This connects the host computer to the CuBox micro-USB service port to access the serial console.

Early boot graphics can fail to provide usable HDMI output on some CuBox-i configurations. Instead, use the built-in serial interface via the micro-USB port to access the terminal output.

Run the following on the Linux host connected to the CuBox-i through a Micro-USB cable to access the CuBox serial interface:

screen /dev/ttyUSB0 115200

(To exit the screen session safely: press Ctrl+A, release, press uppercase K, press Y)

The serial console provides direct, text-based access to early kernel panic messages and hardware initialization logs. The interface is strictly command-line.

Note: Disconnecting the HDMI cable is recommended during setup. HDMI cables carry a 5V line that can backfeed into the CuBox power regulator, causing an "error enabling vbus supply" warning and halting the boot sequence.

The Debian Installer: Navigating the installation steps

Locale and keyboard: Select the preferred language, country, and keyboard layout.
Missing firmware prompt: If the installer reports missing firmware, select No. The installer can then continue with the offline installation and search local storage for the Debian .iso file.
Network setup: If prompted to configure the network, select the option to configure the network later or proceed without a network connection. (Some Debian installer versions have had problems detecting or configuring the CuBox-i Ethernet interface. The hd-media method avoids depending on network access during installation by providing the Debian ISO locally. The Debian installer can locate the ISO on a separate USB storage device and use it to obtain the installation files. This allows the installation to proceed without relying on the CuBox-i Ethernet interface during the installer stage. Network connectivity after installation depends on the installed kernel, drivers, firmware, and configuration.)
Partition disks: At the partitioning screen, choose "Guided - use entire disk" and select the MicroSD card. (The Debian installer runs from system memory after startup and that existing partitions on the installation card can be deleted or replaced.)
Software selection: Select standard system utilities.

When the installer reaches the "Installation complete" screen:

Remove the USB flash drive from the CuBox-i.
Select Continue to reboot the CuBox-i.

The CuBox-i will reboot and load the Debian operating system from the MicroSD card.

Installing the CuBox-i firmware

After installing Debian on the CuBox-i1, CuBox-i2, CuBox-i2eX, or a CuBox-i4Pro, additional firmware files are required for Wi-Fi and hardware-accelerated video decoding.

The VPU requires proprietary firmware to operate, and the Wi-Fi module requires symlinking files from an official Debian package. This article explains how to configure them.

Read the following article:
Installing CuBox i firmware on Debian Linux (CuBox-i1, CuBox-i2, CuBox-i2eX, or a CuBox-i4Pro)

Protecting SSH with Fail2ban on Linux systems, such as Debian, Ubuntu, Arch Linux, Linux Mint, Debian, Kali Linux, raspbian, Gentoo, etc.

Tue, 06 Oct 2026 10:01:55 +0000

Brute-force attacks against SSH frequently target internet facing Linux systems, such as Debian, Ubuntu, Arch Linux, Linux Mint, Kali Linux, Raspberry Pi OS (Raspbian), Gentoo, etc. Fail2ban resolves this by monitoring authentication logs and dynamically updating firewall rules to block IP addresses that show malicious behavior such as repeated failed login attempts.

This article covers configuring custom jail parameters and applying the rules to protect SSH.

When does Fail2ban actually improve SSH security?

The real value of Fail2ban depends on the authentication model. If a server allows password authentication, Fail2ban reduces exposure to automated brute-force attacks by limiting how many authentication attempts an attacker can make from a given IP address.

If a server enforces key-only authentication (highly recommended) and disables root logins, password guessing is no longer an effective attack method. Even so, public-key authentication does not prevent resource exhaustion. Botnets frequently scan exposed SSH ports, and even with password authentication disabled, the SSH server still has to accept incoming TCP connections and process the initial SSH protocol handshake before rejecting the attempt.

Fail2ban addresses this by dynamically updating firewall rules to block any IP address that exceeds a configured threshold of failed attempts. Blocking repeat offenders at the network layer reduces the CPU load of handling malicious SSH handshakes and keeps authentication logs clean. It also generates explicit ban events, providing an audit trail of abusive IP addresses without cluttering logs with background noise.

Installing packages

The initial step requires installing the fail2ban, the daemon responsible for parsing logs and managing bans.

On Debian-based systems, fail2ban can be installed with:

apt-get install fail2ban

For other distributions, install fail2ban using the package manager.

Configuring jail.local

The fail2ban package ships with /etc/fail2ban/jail.conf by default. Do not edit this file directly. Instead, create /etc/fail2ban/jail.local. Using jail.local ensures package upgrades do not overwrite your configurations. (Fail2ban reads .conf files first, followed by .local files, which override the default values.)

Create the /etc/fail2ban/jail.local file with the following content:

[DEFAULT]
# Uncomment to use journalctl to find SSH authentication attempts
# backend = systemd

# The duration a host is banned (supports seconds or abbreviations like 1h)
bantime = 1h

# Number of failures (e.g., failed SSH login attempts) before a host is banned
maxretry = 5

# A host is banned if it has generated 'maxretry' attempts during the last 'findtime'
findtime = 10m

# Never ban these ip addresses
ignoreip = 127.0.0.1/8 ::1

[sshd]
enabled = true
port = ssh
filter = sshd

To prevent accidental lockouts, add trusted IP addresses, CIDR ranges, or hostnames to ignoreip. Fail2ban skips rule enforcement for these addresses. Separate multiple entries with spaces (e.g., ignoreip = 127.0.0.1/8 ::1 192.168.1.0/24 203.0.113.40). Always include management workstations, VPN gateways, and jump hosts before enforcing bans.

Managing repeat offenders with the recidive jail

Configuring the recidive jail handles persistent attackers who return after their initial ban expires by monitoring the Fail2ban log itself and applying longer, stricter bans to IP addresses repeatedly banned by other jails.

Add the following to /etc/fail2ban/jail.local to extend the ban for persistent abusers:

[recidive]
enabled = true

# Monitor the fail2ban log file for repeated bans
backend = polling
logpath = /var/log/fail2ban.log

banaction = %(banaction_allports)s

# Ban repeat offenders for 1 week
bantime = 1w

# Look for repeat bans over the last 1 day
findtime = 1d

# Trigger after 5 previous bans
maxretry = 5

Applying the configuration

After modifying the /etc/fail2ban/jail.local file, restart the daemon to apply the new rules:

systemctl restart fail2ban

Verifying the status

This reports the status of the sshd jail, including the current number of banned IP addresses:

fail2ban-client status sshd

How to unblock an IP address?

To unblock an IP address managed by Fail2bans ssh jail, use:

fail2ban-client set sshd unbanip 8.8.8.8

Alternatively, you can use the following to remove an IP address from all active jails simultaneously:

fail2ban-client unban 8.8.8.8

Beyond Fail2ban

While Fail2ban reduces automated SSH brute-force attacks, it is reactive and insufficient as a standalone security strategy. Since Fail2ban reacts to events after they have been recorded by the monitored service or log source, the underlying attack surface remains exposed to initial probes and attackers that rotate IP addresses or stay below the configured thresholds.

Rather than treating Fail2ban as a complete solution, It is recommended to implement a layered defense architecture that includes:

Moving the actual SSH daemon off public-facing interfaces,
enforcing key-only authentication,
disabling root logins,
setting up a default-deny firewall,
implementing multi-factor authentication,
using a VPN,
hardening cryptographic algorithms,
deploying a decoy honeypot (e.g., Cowrie),
Limiting malicious attempts using OpenSSH's native features, such as PerSourcePenalties and MaxStartups to reduce attack surface,
and configuring sshd to listen on a non-standard port, such as 22000 instead of the default 22. (This is another method to reduce automated scanning. Many botnets are programmed to look exclusively for open port 22. While changing the port is security by obscurity and will not defeat a dedicated attacker, it drops the volume of opportunistic background traffic hitting your server.)

Fixing the throttling that slow down Thinkpad laptops, such as the Lenovo Thinkpad T420s (BD PROCHOT) | James Cherti

Tue, 06 Oct 2026 10:01:55 +0000

Author: James Cherti
Share: Hacker News X Reddit LinkedIn

Even after configuring Linux on a Lenovo ThinkPad, such as the Thinkpad T420s, the laptop can still experience unexplained CPU throttling that degrades performance.

A common cause is an externally asserted BD PROCHOT (Bi-Directional Processor Hot). While standard PROCHOT is initiated internally by the processor when it overheats, BD-PROCHOT allows other components, such as power supply sensors or battery circuitry, to send a hardware signal directly to the CPU, which triggers processor throttling regardless of actual CPU core temperatures.

What does BD-PROCHOT do to the Lenovo ThinkPad laptops?

The CPU clock speed is locked at a low frequency, which causes severe system lag during basic desktop tasks such as video playback or web browsing. Because BD-PROCHOT can be asserted by power management sensors rather than CPU temperature, the CPU remains throttled even when core temperatures are normal.

How to check if PROCHOT is enabled?

Install the msr-tools package
Load the msr Linux module using: modprobe msr
Run the following command:

echo -n "PROCHOT "; (( 0x$(rdmsr -p0 0x1fc) & 1 )) && echo "enabled" || echo "disabled"

What causes BD-PROCHOT and throttling on the Lenovo ThinkPad laptops?

Several hardware, power supply, and thermal conditions trigger BD-PROCHOT or general CPU throttling on the Lenovo ThinkPad laptops:

Using underpowered or aging power adapters or an unverified third-party charger.
Using a heavily degraded or failing battery, which can misreport electrical metrics to the embedded controller.
Failing hardware or bad sensors on the motherboard can incorrectly signal a thermal or power fault.
Old, dried-up thermal paste on the CPU.
Accumulated dust in the fan assembly.
Using an underpowered 65W (OUTPUT: 20V, 3.25 A) charger instead of the 90W power adapter (OUTPUT: 20V, 4.5A).

Solution: throttled

One solution is to use the the throttled daemon, which writes directly to CPU and chipset registers:

Download and install throttled from the 'throttled' repository

The primary configuration file is located at /etc/throttled.conf. It contains separate profiles for power limits, update intervals, and temperature targets.

Note: The thermald daemon can conflict with throttled by applying its own temperature policy.

Can a Twenty-Year-Old Computer Still Be Useful?

Mon, 05 Oct 2026 22:40:46 +0000

A computer from 2006 is old now. That sounds obvious until you stop and do the math. A machine from that period may have shipped with Windows XP, maybe Windows Vista if it was unlucky, or some version of Mac OS X that is now a museum piece by modern support standards. On the Linux side, it may have been running a distribution from an era when wireless drivers still made people mutter at their desks.

Yet a surprising number of those machines still turn on. The screen may be dim. The battery may be useless. The hard drive may sound like it is auditioning for mechanical failure. But the keyboard works, the ports are still there, and the processor can still add numbers just as honestly as it did when the machine was new.

The better question is not whether a twenty-year-old computer is useful in the abstract. The better question is useful for what.

If the job is the modern commercial web, the answer gets rough quickly. Current browsers are heavy. Many sites assume current TLS support, recent JavaScript engines, hardware acceleration, video codecs, plenty of memory, and a processor that can keep up with pages written like desktop applications. A 2006 machine can sometimes be coaxed into loading those pages, especially with enough RAM and a lightweight Linux installation, but it is not going to feel young.

Security is the other hard limit. Running an unsupported operating system on the open Internet is asking for trouble. That does not mean the machine must be thrown away. It means the role needs to be chosen carefully. A twenty-year-old computer used for online banking, password management, main email, or account administration is a bad idea. A twenty-year-old computer used for writing, local tools, retro software, terminal access, or a contained lab is a different matter.

There is still plenty of useful work that does not require the heaviest parts of today’s web. A machine from that period can be a writing station. It can run a text editor, keep notes, read plain documents, manage a local archive, play older media files, run period games, connect to BBSes, SSH into newer systems, or serve as a terminal for machines doing the harder work elsewhere. In some cases it can run a lightweight desktop well enough for email, feeds, and simple browsing, as long as expectations stay grounded.

Linux can help, but Linux is not a time machine. A lightweight distribution may give old hardware a cleaner second life. Replacing a spinning hard drive with an SSD can make a dramatic difference, if the machine supports it. Adding RAM helps, up to the limits of the hardware. Cleaning dust from the cooling system can matter more than people expect. Still, there comes a point where the browser, the kernel, the graphics stack, or basic electrical age sets the boundary.

The good news is that usefulness does not have to mean daily-driver status. We have a bad habit of measuring every computer by whether it can be the one machine for everything. That is a narrow standard. A twenty-year-old laptop may be a poor main computer and an excellent distraction-free writing box. An old desktop may be silly for video editing and perfectly fine as a serial console, ham radio station, test machine, file sorting station, or retro gaming rig.

Old hardware is also useful because it teaches limits. Modern computers hide a lot behind speed. Wasteful software can survive because the machine underneath is fast enough to absorb the damage. On older hardware, waste becomes visible. A bloated web page stutters. A careless background service matters. A simple native program feels good because it respects the machine. That kind of feedback is educational.

There is a repair lesson too. Twenty-year-old computers often come from the last period when many machines were still reasonably serviceable. That was never universal, but it was common enough to matter. Screws were common. Drives and memory were reachable. Batteries were more likely to be replaceable. Ports were generous. Documentation existed. Compared with sealed modern devices, some older computers feel less polished and more honest. You can open them without feeling like you are violating a product manager’s wishes.

That does not mean keeping every old machine alive makes sense. Some are power-hungry. Some are unreliable. Some use aging capacitors, dying displays, brittle plastics, or storage that should not be trusted. Sometimes the right answer is to salvage parts, wipe the drive, recycle the machine properly, and move on. Sentiment is not a maintenance plan.

But useful old computers deserve a fair assessment. Start with the hardware. Does it boot reliably? Can it take more RAM? Is the storage healthy? Does it overheat? Are replacement parts available? Then look at the job. Does that job require current security patches? Does it require modern web services? Does it require high performance, or only patience and a keyboard? The answer usually becomes clear once the job is specific.

For me, the most interesting uses are the ones that respect the age of the machine. Use it as a writing system. Put a small Linux or BSD setup on it if the hardware allows. Keep it off important accounts. Let it talk to newer machines over SSH. Use it for old software that belongs on it. Use it to learn how much computing can still happen without a stack of subscriptions and background services.

A twenty-year-old computer can still be useful. It probably should not be asked to ignore the last twenty years of software inflation. Give it a job that fits, keep the risks contained, and it may still earn its place on the desk.

My 5 favorite Linux distros for security – and how they protect your privacy

Mon, 05 Oct 2026 19:10:38 +0000

If security is a top priority, you should consider one of these five Linux distributions. They’re all free to install and use.

Using docker-compose with Podman rootless — Elouworld

Mon, 05 Oct 2026 16:33:07 +0000

Using docker-compose with Podman rootless

2026-10-05 09:22:33 UTC

Podman is a daemonless Docker alternative for Linux that can run without root access. However, it is less well known that Podman can expose a UNIX-domain socket compatible with the Docker API. This makes it work with most tools in the Docker ecosystem, such as docker-compose.

Podman uses a Linux feature called user namespaces. With this, the root user inside a container is mapped to your host user. Other UIDs and GIDs are mapped to the ranges defined in /etc/subuid and /etc/subgid, respectively.

This tutorial assumes that Podman and docker-compose are already installed, for example using your Linux distribution’s package manager.

Enable and start the Podman socket

To enable and start the Podman socket, run this command (as your user, not as root):

1

systemctl --user enable --now podman.socket

This command creates a UNIX-domain socket at ${XDG_RUNTIME_DIR}/podman/podman.sock. ${XDG_RUNTIME_DIR} is a private tmpfs automatically mounted for each user.

Note: The command requires a systemd session. If you are trying to run this command as another user, be aware that using sudo is not supported, because it doesn’t create a systemd session. You can use machinectl shell --uid=your-username (part of the systemd-container package on some Linux distributions) if you are part of the wheel group. Alternatively, log in as your user on a TTY or via SSH.

Expose it as the Docker host

Tools like docker-compose read the DOCKER_HOST environment variable. Set it to point to the Podman socket like this:

1

export DOCKER_HOST="unix://${XDG_RUNTIME_DIR}/podman/podman.sock"

Add this line to your shell configuration (e.g. ~/.zshrc for Zsh) to make it permanent.

Use docker-compose

You can now run docker-compose as usual:

1
2
3
4

docker-compose config
docker-compose up -d
docker-compose ps
docker-compose down --volumes

Depending on your Linux distribution, docker-compose may be available as docker compose instead of docker-compose, but it works the same way. You can add an alias in your shell:

1

alias docker-compose='docker compose'

Tips and tricks

Stop and disable rootful Docker

If you have Docker installed but are not ready to uninstall it, you can stop and disable its systemd service by running (as root):

1
2

systemctl disable --now docker.service docker.socket
rm -f /var/run/docker.sock

Note: These commands do not erase Docker data.

If you change your mind, run this to start it again (as root):

1

systemctl enable --now docker.service

Using `docker`

The podman command accepts the same arguments as docker, but you can also keep using the docker command if you prefer: it can read the DOCKER_HOST variable and talk to the Podman socket, just like docker-compose.

`podman unshare`

If you want to become root without starting a container, you can use the podman unshare command, which starts a new shell as root (in a user namespace, not real host root), much like sudo -i. You will then be able to manipulate files owned by container users (for example with chown or chmod).

`podman mount`

You can access the files of a running container with podman mount.

First, run podman unshare, then change directory to the path returned by podman mount container-name-or-id:

1
2

podman unshare
cd "$(podman mount container-name-or-id)"

You will then be able to run your usual TUI editor to edit files in the container.

Docker rootless

If you are not ready to switch to Podman, Docker also supports a rootless installation. See their documentation.

Once it is set up and started, you also need to set the DOCKER_HOST environment variable:

1

export DOCKER_HOST="unix://${XDG_RUNTIME_DIR}/docker.sock"

Unfortunately, rootless Docker has no equivalent of podman unshare and podman mount, although you can achieve similar things with unshare and nsenter.

User lingering

By default, Podman containers are stopped when the last systemd session of your user is closed.

To keep them running after you log out, enable user lingering for your user (as root):

1

loginctl enable-linger your-username

Roastidio.us Tagged with linux

Weird Things Are Afoot In The Honeypot

Debian Apt Repo for libipt

How ASLR Helps Enable Exploits (CVE-2013-2028)

Intel PT Data at Rest: A Compression Experiment

Procedure

Results

Discussion

Intel Processor Trace, execvp, and ptrace

execvp and CR3

ptrace

Getting the CR3 value for a PID in Linux

How Orange Helps You Sleep At Night

About The Author

The importance of boot partitions in Linux systems.

Finding the Problem

Diagnosing the Problem

Understanding Partitions

Boot Sequence

Diagnosing Grub

Remediation

Creating a Boot Partition in an Existing Linux Installation

Conclusion

Installing psad on Raspberry Pi Running Arch Linux - Carter Yagemann

What is psad?

Scope of this document

Contact

Other Useful Resources

Installing psad for ARM from AUR

Configuration

Troubleshooting

wget fails due to certificates

makepkg fails and returns a build error

psad is installed, but when I run sudo psad -S I get the message pid file [...]/psadwatchd.pid does not exist

Last rites for Gentoo's Chromium package [LWN.net]

Firefox maintenance

Janet on x32: 32-bit Pointers, 64-bit Speed, 25% Less RAM | Alex Alejandre

The Hopefully Eventually Ultimate Desktop Linux Troubleshooting Guide: Recovering from Crashes and Freezes

If a program has crashed

Scary black screen

If the mouse slows down and gets jittery, and the fans rev up, and the I/O light blinks fast or goes solid, and then the system fully freezes

If it’s the desktop environment that has crashed

If you’re unclear about the difference between a window manager, a desktop environment, a display server, and a distro(bution) are

If your screen goes all colorful or develops lines and weird patterns

If it has fully crashed and rebooted on its own

Suspend issues

If you want to know what a program is doing

If you want to know what is hogging all the resources

If you want to know whether a script actually quit or crashed and left pieces churning along in the background

If you're stuck in Vim

If the screen is black

If the Wifi doesn't work

If your WiFi hotspot has no internet in it

If the printer won’t print

If your mouse or touchpad has stopped moving

If you're having trouble installing software

If you get Error Splicing File errors

If the issue is with external media (thumbdrive, HDD)

If a SD card / flashdrive got corrupted

Check the logs

Befriend the terminal

Learn a console text editor

General advice

Conclusion

Montray - a tray icon for systemd service health

Por qué hoy uso Fedora · El blog de Lázaro

Muerte de las pequeñas

El espejismo de la popularidad

Las grandes que siempre resisten

La búsqueda de la estabilidad

Por qué elijo Fedora

Lo que hace realmente grande a Linux

Mi elección

Forgot Your Root Password? Here’s the Easy Way (and the Hard Way) to Recover It

Using Your Panel

Using Rescue

Taming GCE Memory Tax: Disabling OS Login and OS Config on Small Instances // Tony Metzidis

The Anatomy of the 15% Memory Tax

Why OS Config Hoards 62 MB of RAM

The SSH Starvation Cascade: How OS Login Fails Under Load

psad is installed, but when I run `sudo psad -S` I get the message `pid file [...]/psadwatchd.pid does not exist`