The Marginalia Search index has been partially rewritten to perform much better, using new data structures designed to make better use of modern hardware. This post will cover the new design, and will also touch upon some of the unexpected and unintuitive performance characteristics of NVMe SSDs when it comes to read sizes. The index is already fairly large, but can sometimes feel smaller than it is, and paradoxically, query performance is a big part of why.| www.marginalia.nu
In this post I will introduce a much more efficient method for accounting block I/O latencies with eBPF on Linux. In my stress test, the “new biolatency” accounting method had 59x lower CPU and probe latency overhead compared to the current biolatency approach. So I couldn’t help it and ended up putting 21 NVMe SSDs into one of my homelab servers. 8 of them are PCIe5 and the remaining 13 are PCIe4. - Linux, Oracle, SQL performance tuning and troubleshooting - consulting & training.| tanelpoder.com
One evening back in January I finally had enough of thermal issues within my homelab server. You know, every time the computer fans make more noise than I think they should, I can’t help but investigate! Also, the RTX4090 is so thick that it takes 3 PCIe slots worth of space on a typical motherboard - and you also need space for the airflow! I was using the same machine that achieved 11M IOPS & 66 GB/s IO on a Single ThreadRipper Workstation back in 2020. - Linux, Oracle, SQL performance tu...| tanelpoder.com
TL;DR: Some Linux distros and even different kernel versions within a distro have disabled IRQ time accounting. In such case, your monitoring tools will report zero time spent in hardware interrupt handlers (shown as %hi, %irq, hiq, etc). It’s easy to check how your kernel is behaving by looking at /proc/stat and you can still measure interrupt CPU usage using perf and a little creativity. Index Index Problem Explanation Kernel configuration options Measuring IRQ time with perf Different di...| tanelpoder.com
Hands-on technical analysis of a novel data platform for high-performance block I/O in the cloud, tested by Tanel Poder, a database consultant and a long-time computer performance geek. Index Background and motivation Architecture and internals of the Silk Platform I/O throughput for table scanning (for your big data warehouses) IOPS, I/O latency and write performance (OLTP and data loads) Scaling out I/O performance in the cloud (demo of live operation) Summary and architectural opportunitie...| tanelpoder.com
You may remember my previous article about high-performance I/O for cloud databases, where I tested out the Silk Platform and got a single Azure VM to do over 5 GB/s (gigaBytes/s) of I/O for my large database query workload. The magic was in using iSCSI over the switched cloud compute network to multiple “data VMs” at the virtualized storage layer for high I/O throughput and scalability. Well, the Silk folks are at it again and Chris Buckel (@flashdba) just sent me screenshots of similar ...| tanelpoder.com