scottstuff.net
Homepage and blog by Scott Laird| scottstuff.net
In my continuing quest for the perfect Zoom background video, I hiked around Paradise at Mt Rainier for a while last weekend with a camera. Mt Rainier| All Posts - scottstuff.net
I’ll admit: I’m terrible at managing SSH keys. Every time I build a new desktop I end up generating a new SSH key with ssh-keygen and then half-heartedly copy it to authorized_keys on machines that I ssh into. The machines that I use most often have huge authorized_keys files including keys from machines that died years ago, while systems that I rarely use don’t have any keys at all and I end up falling back to passwords until it annoys me enough to copy keys over. This has been especia...| All Posts - scottstuff.net
Direct routing is Arista’s name for deploying “virtual gateways” in an EVPN fabric. It lets multiple devices all act as the default router for a network. It’s sort of like you get with VRRP, but it’s active/active(/active/active…) instead of active/standby like with VRRP. You configure your devices to share a virtual router MAC address and then add a bit of coordination magic so that ARP and friends don’t go wild, and then you can make every L3 switch that’s part of a specific...| All Posts - scottstuff.net
I have a pair of Intel E810-XXVDA4T timing NICs at home that I’m using as NTP time sources and I wanted to write up how to configure these. The E810-XXCDA4T is a 4-port 25 GbE NIC that is optimized for use as a network time source. It includes a GNSS/GPS receiver, an extremely accurate oscillator, and a very low-latency NIC all in a single package. This is far and away the most accurate network time source that I’ve had my hands on, but it takes a little bit of tweaking to use it properly...| All Posts - scottstuff.net
Since I’m swimming in QSFP switch ports at home, I’m curious about what the lowest-power NICs are that use QSFP ports of any speed — QSFP+4x 10 Gb lanes, for 40 Gb Ethernet. , QSFP284x 25 Gb lanes. , QSFP564x 50 Gb lanes. , etc. I already determined that adding a ConnectX-5 to one of my servers adds 13-19W, depending on which optics are in use. I decided to do a bit of research to see if there were any lower-power options available for cases where I want more bandwidth than I can get fr...| All Posts - scottstuff.net
This is the 13th article that I’ve written lately on NTP and PTP timing with Linux. I set out to answer a couple questions for myself and ended up spending two months swimming in an ocean of nanosecond-scale measurements. When I started, I saw a lot of misinformation about NTP and PTP online. Things like: Conventional wisdom said that NTP was good for millisecond-scale timing accuracy. I expected that to be rather pessimistic, and expected to see low microsecond to high nanosecond-range syn...| All Posts - scottstuff.net
So, I’ve tested how well I can sync 2 systems with expensive timing NICs. I’ve tested NTP settings. I’ve tested a boatload of NICs. Now I’m down to the last part of the test: network switches. Specifically, how much delay do good network switches add to PTP? TL;DR: almost no delay at all. Setup This is the same basic setup as in previous tests. My test system is an older Supermicro 1018R-WR with an Intel E5-2683 v4 CPU. The primary NIC is an Intel E810-XXVDA4T in one x16 PCIe 3.0 slot...| All Posts - scottstuff.net
So, now that I have a good baseline for understanding NTP and PTP performance, it’s time to turn my attention to the box full of Network Interface Cards that that I have in front of me to see how good they are at time-sensitive networking. I’ve seen a wide range of NTP performance in the field for systems with different NICs, and I’d like to measure and see how well the various NICs I have on hand perform in a more controlled environment. So far I’ve successfully tested 14 different N...| All Posts - scottstuff.net
In part 1 yesterday I went through all of the work needed to measure NTP and PTP accuracy between two computers on my desk using an oscilloscope. I demonstrated that PTP was accurate to a mean of 4 ns with 2 ns of standard deviation. Under ideal circumstances NTP was only slightly worse at 8–10 ns with a SD of 12–20 ns, depending on the test setup. I measured these with extremely high NTP polling rates, thousands of times more frequent than Chrony’s defaults. I ran a few tests with slow...| All Posts - scottstuff.net
I’ve been working on measuring NTP (and PTP) timing accuracy in Linux for the past month or two, and I’m finally down to what I think is the very end. I’ve demonstrated that NTP (using Chrony) can sync to within ~500 ns on a non-trivial network. By all accounts, ptp4l is able to do better than that, at least when all of the switches and NICs involved have hardware support for PTP. At some point, though, it’s very difficult to trust network time syncing software’s estimates of their ...| All Posts - scottstuff.net
Minisforum has finally started shipping their MS-A2 mini PC. I don’t have one on hand yet, but Patrick at ServeTheHome has a review of the MS-A2 up. As a reminder, this is the 16-core system that I talked about in January when it was announced. They’ve changed the specs slightly since it was first announced; it originally came with a 16-core Zen 4 7945HX or possibly a large-cache 7945HX3D; they’ve now added a Zen 5 9955HX option. Patrick says that his MS-A2 with the 9955HX idles at arou...| All Posts - scottstuff.net
I mentioned recently that one of my Raspberry Pi CM5s wasn’t able to keep stable time for some reason. I’ve mostly fixed the problem, but as usual I’m left with a handful of new questions that might be more interesting than the original problem.| All Posts - scottstuff.net
I was trying to figure out why the Raspberry Pintp4 from my post on NTP accuracy, a Raspberry Pi CM5 on my desk’s time isn’t as accurate as the time coming from the PC that’s sitting next to it. I was looking at some statistics, and found something fascinating. On one level, the two systems are basically identical. They’re both running Ubuntu 24.04. They both have on-board GPS devices with PPSPulse Per Second — the GPS chip blips a signal exactly when each second starts. Or as exact...| All Posts - scottstuff.net
As mentioned late last year, I gave a talk at SREcon Americas 2025 on some of the work that we did at Figma on improving OpenSearch speed. Usenix has now made the recording of the talk available on Youtube: This was actually my first conference talk ever. It went better than I’d feared, but I finished 10 minutes faster than I had on any of my test runs. No, you didn’t accidentally set the video to play back at 1.5x speed; I actually presented it at 1.5x speed. Also, for the record, the au...| All Posts - scottstuff.net
If you look at the datasheet for any of ublox’s current GPS chips, you’ll see that they’re supposed to be able to keep tracking GPS satellites down to -167 dBm. That is a terrifyingly small amount of power, and probably the smallest number that I’ve ever actually seen in real life. To make sense of this, you need to know what a dBm is — it’s a decibel relative to 1 milliwatt. Decibels are logarithmic, so every time you add or subtract 10 dB the power gets multiplied or divided by ...| All Posts - scottstuff.net
One of the requirements for having accurate time is having a clock that actually runs at the right speed. Not surprisingly, none of the PCs that I’m monitoring at home actually have particularly accurate system clocks. They range from about 30 PPM fastMy file server, from Supermicro. to 60 PPM slow,Two Minsforum MS-01s and an Asus Epyc server. and even the best is still 1.5 PPM off.A cheap mini-PC, surprisingly. Measured clock frequency skew across servers, before correction by Chrony. To p...| All Posts - scottstuff.net
As part of my recent quest to get the best possible time syncing in Linux, I’ve done a lot of research into the best NICs for timing use under Linux. Somewhat surprisingly, there are a number of network interface cards that are designed for timing use, with either an onboard GNSS/GPS device or support for external pulse-per-second (PPS) inputs. Many modern NICs have a hardware clock on them, and NICs that can sync their clock with a high-accuracy PPS source should be able to maintain time w...| All Posts - scottstuff.net
Note: I’m working on a set of posts on the limits of NTP time syncing with Linux, but before I do that I wanted to write up an overview of the basic terms and concepts involved. This was originally part of a giant post on accuracy, and I’m going to break it out into its own post instead, and then post the other pieces over the next week. So, suppose you have a bunch of Linux systems on a network and you want to keep their clocks in sync. It’s a common enough problem, and there are lots ...| All Posts - scottstuff.net
I recently added an Intel E810-xxx network interface card to my desktop and discovered that I needed to use Intel’s latest driver to unlock some features that I needed. The driver in the Linux kernel source tree is too old to support the features and Intel hasn’t really tried to keep it up to date.In my specific case, I needed support for pulse-per-second (PPS) inputs and outputs for timing, but there are years of changes in the newer driver that haven’t been merged upstream. It’s eas...| All Posts - scottstuff.net
I ran into a weird problem last week while trying to measure timing accuracy, and I can’t see anyone else having a similar problem on the Internet. It feels like a kind of fundamental problem, not just a small bug, so I figured I’d write something up for posterity. There are two main protocols for syncing high-accuracy time to servers over the network today: NTP and PTP. NTP is kind of the old standby.Originally from 1985, although it’s been updated repeatedly over the years. Originally...| All Posts - scottstuff.net
How good is your clock, really? When I was young, if the clock on the wall was within 5 minutes of correct, that was usually good enough for most people. The only easy way to find the “right” time was to dial a phone number that would read out the time to you (“at the tone the time will be… eight fifty two and twenty seconds… beep.”). We’ve come a ways since then; most modern computers use NTP to synchronize their clocks over the Internet automatically, generally getting their t...| All Posts - scottstuff.net
I can’t count the number of times that I’ve started working with a new service and discovered that they had an alert configured to page based on the number of HTTP 404s served. Every single time I’ve found a 404s-too-high alert, I’ve deleted it entirely from monitoring. It’s too noisy of a signal; there’s just too much random traffic on the Internet probing for security vulnerabilities to make 404 monitoring even slightly useful. Most of the time, at best we end up with a graph of...| All Posts - scottstuff.net
I’ve finished moving my commenting system here from Commento to Comentario as previously planned. This should be complete now, but I’m sure that something is still broken. I’ve enabled Google, Facebook, and Github logins. You can also create an account manually, or anonymous should work. Mail me at scott@sigkill.org if you try to comment and run into problems.| All Posts - scottstuff.net
I’ve been using Commento for comments here for a few years, because it seemed like the easiest option for adding comments to Hugo at the time. Unfortunately, the project is all-but-dead; there haven’t been any changes in Git in 4 years. Now the (paid!) hosted Commento service is failing. I noticed that comments weren’t loading consistently a couple weeks ago, and today I can’t even log into their admin interface. Fortunately I downloaded all of the comments a few days ago, so there sh...| All Posts - scottstuff.net
I had an interesting hardware failure yesterday that I’ve never seen before, and there’s no way I would have been able to debug it if I hadn’t had an excessive amount of power monitoring in place. In the end, it turned out to be a partial PSU failure on a Juniper 715W PSU in a Juniper EX4300-48MP switch. The fun part is that the switch itself kept working and thought everything was fine. This was kind of a fun debugging story, so I figured I’d share it.| All Posts - scottstuff.net
A couple weeks ago I described how I was collecting client side HTTP metrics by using Clickhouse and Vector to record Network Event Log (“NEL”) data from browsers. NEL lets browsers (mostly Chrome and its derivatives today) tell web servers about problems and performance without requiring any client-side Javascript or anything else fancy or error-prone. So far, the data collected by this has been fascinating, and it’s pointed out several minor problems that I’ve been able to fix. Howe...| All Posts - scottstuff.net
A few weeks ago I read a social media post that talked about Star Trek; I can’t seem to find it right now, but it’s been stuck in my head since I read it. Star Trek was full of people doing art in their free time — painting, music, and so forth. And they were almost uniformly bad at it. Or, if not bad, at least not particularly good. The Star Trek universe seemed to have billions of people producing mediocre-at-best paintings with no commercial value, not because they wanted to be famou...| All Posts - scottstuff.net
There’s a pretty well-known problem with the Mastodon federated social network, where every time you post a link, each of your followers’ Mastodon servers will immediately download a copy of the page along with linked images. That’s fine when you don’t have a ton of followers, or the link goes to a fast server. But when popular folks post links to slow sites, they tend to go down.This is not a new problem in general, although the way that Mastodon does it behind the scenes is kind of ...| All Posts - scottstuff.net
Now that I’m storing my blog’s request logs in Clickhouse, I can do some fun things with the data without a ton of extra work. For instance, it’s not all that hard to draw a map that shows where in the world my traffic is coming from: Grafana can now render a heatmap of where my traffic comes from for me. To do this, you need to add a bunch of geolocation boilerplate to Clickhouse, but once that’s done mapping IPv4 or IPv6 addresses to points on a map is fast and easy.| All Posts - scottstuff.net
I made a few changes to this website this afternoon and I was worried that I’d accidentally published links to the staging version of the site instead of the real one. “If only there was an easy way to get errors back from web browsers without requiring some big, heavyweight Javascript monstrosity that causes at least as many problems as it solves,” I said to myself. Then I remembered that there actually is a standard way to get HTTP errors and status information fed back from stock web...| All Posts - scottstuff.net
So, you want to stream video on the web without dealing with Youtube or other video providers? Where do you even start? Welcome to DIY Streaming Video 101. I’m slowly working on building the infrastructure here for encoding and serving videos myself, and I’m trying to share what I’ve learned. I’m doing everything on Linux, mostly using FFMPEG for video conversion and serving the video myself, from a web server in my house. Obviously, there are many services that you can pay to do this...| All Posts - scottstuff.net
I’m trying to figure out how best to compress my videos for streaming over the web. I’m using FFMPEG for compression, and it has dozens of potentially-useful flags. Looking at random guides and forum posts online, there’s obviously a lot of cargo-culting of compression parameters, and it’s not clear at all to me what the best choices are. Since there’s no authoritative guide to picking options, and in any case it’d vary widely depending on the nature of the videos that you’re en...| All Posts - scottstuff.net
I’ve been working on infrastructure for streaming my own videos, and part of that work is deciding on which video codecs and settings I want to use. There are a bewildering number of options for ffmpeg and there doesn’t seem to be any consensus on what actually works best. I figured that the only way to really know would be to run a set of tests myself and see what the results looked like.| All Posts - scottstuff.net
It’s a well-known fact that the Internet is for cat pictures. There is extensive documentation of this in The Literature. However, with the rise of this whole new-fangled streaming video thing, a secondary purpose for the internet has arisen: streaming bird videos to cats. Of our 3 cats, 2 are essentially blind to TVs and monitors, but Sputnik is a huge fan. He’ll sit and watch the mouse pointer move if he has to, but streaming bird videos make his day. I would therefore like to announce ...| All Posts - scottstuff.net
I’ve been working on adding more video content here, which isn’t precisely easy given the way that I’ve built this site. The main site is all built using Hugo, which turns a giant pile of Markdown text into a static HTML website. Then I have a pile of scripts that publish the site using rsync, copying the HTML onto local SSD on the web servers for low latency and ease of serving. The source code (including images) for the website runs around 180 MB right now, and that balloons up to 400...| All Posts - scottstuff.net
As mentioned previously, I’ve been experimenting with video a lot more than still photography these days. My main focus is on producing backgrounds for desktop video conferencing, so no one needs to look at an artificially fuzzy version of my home office in Zoom. Last week, I spent a couple mornings at Deception Pass, which is a narrow gap in Puget Sound between two islandsWell, sort of 2 gaps between 3 islands, and one of the islands is only sort of an island, but whatever. with a pair of ...| All Posts - scottstuff.net
Yesterday, I mentioned that I’m now using Caddy running in a container on VyOS for HTTP load balancing this site. A few months ago, I did a bunch of benchmarking on Caddy performance, and was able to push about 20k QPS of HTTPS traffic through this site through HAProxy running on a slightly different iteration of my router. That was with HTTPS being decoded on the web servers, while the current config does HTTPS on the load balancer. So this isn’t really an apples-to-apples comparison, bu...| All Posts - scottstuff.net
I went down a bit of a rabbit hole this week, and I figured I’d share so hopefully no one else has to dive quite this deep. Modern web browsers generally prefer to fetch compressed content from web servers; they’ll pass an Accept-Encoding header as part of the HTTP request listing which compression types they support, and then the server will attempt to return content compressed in a form that the client supports. Most web servers have the ability to compress web content on the fly, but t...| All Posts - scottstuff.net
As mentioned before, this web site is mostly served servers in my house. My ISP provides me with a single static IPv4 address, so I need some way to get HTTP and HTTPS (and HTTP/3!) traffic from my router to the pair of web servers that I’m using today. I’m currently using VyOS on my router, and it provides two different ways to load-balance HTTP traffic: high-availability virtual-server, which is based on IPVS in the Linux kernel. load-balancing haproxy, which is handled by HAProxy runni...| All Posts - scottstuff.net
As mentioned before, I’ve been using the Caddy web server running on a couple machines to serve this site. I’ve been dumping Caddy’s access logs into Grafana’s Loki log system, but I haven’t been very happy with it for web logs. It’s kind of a pain to configure for small uses (a few GB of data on one server), and it’s slow for my use case. I’m sure I could optimize it one way or another, but even without the performance issues I’m still not very happy with it for logs analys...| All Posts - scottstuff.net
I run a number of servers that are attached to two different L3 network switches for redundancy. My usual pattern for this is to use L3 links to the switches, and then use OSPF to share a loopback IP from Linux into the switches’ dynamic routing instances. That way, either switch can go down (software upgrades, hardware faults, cleaning up network cables, whatever) and traffic keeps flowing, because I still have at least one path to the server. On some systems, this has worked flawlessly fo...| All Posts - scottstuff.net
Part 7 of a series on VyOS routing performance on a Minisforum MS-01 mini-PC. Maybe start with part 1 for context. Routing performance as routing table size changes So, after yesterday it was clear that firewall rules really slow down routing; 256 rules reduced our peak Mpps rate by around 36%, clearly indicating that Linux processes firewall rulesnftable entries, specifically. linearly. Doubling the number of rules makes rule handling take twice as long. But what about routes? Does Linux rea...| All Posts - scottstuff.net
Part 6 of a series on VyOS performance on a Minisforum MS-01 mini-pc. You might want to start with part 1 I’m almost done with VyOS performance testing for now, but I wanted to get a feel for how much firewall rules impact VyOS performance. This set of tests are from VyOS 1.5-rolling-202501060800 on an Minisforum MS-01 with an i5-12600H, the same hardware and software as the past few entries in the series. As with the previous set of tests, these are generated using TRex with its stl/bench....| All Posts - scottstuff.net
Cisco’s open-source TRex network load-testing tool is pretty much the standard for testing routers and other network equipment, but it’s not very well maintainedCase in point – their SSL cert is currently expired and it’s tricky to get working on modern machines. I’m far from an expert on this, but I just jumped through a bunch of hoops to get it working on Ubuntu 24.04 with a Mellanox NIC as part of my VyOS testing series, and I figured that I’d share my process.| All Posts - scottstuff.net
Apparently I just can’t end this series. It’d probably be easier if I didn’t keep learning more about performance and discovering that a lot of my earlier results we’re quite as accurate as I’d thought they were. Spectre/Meltdown and Router Performance VyOS’s page on VPP performance includes a few additional tuning settings that I hadn’t tried; most of those are specific to their hardware (NUMA tweaks, disabling hyperthreading, adjusting NIC RX/TX queues to fit the reduced numbe...| All Posts - scottstuff.net
After finishing part 3 of my series on routing with VyOS on a Minisforum MS-01 mini-PC, I wasn’t entirely happy with the benchmark results. Sure, they showed that you can send 40 Gbps of HTTP traffic through a small PC running Linux on a laptop CPU, but I had no data at all on small-packet performance and I didn’t know how it scaled past 40 Gbps. Conventional wisdom holds that Linux’s kernel isn’t all that great at large amounts of small packets, so I figured I’d pull out bigger gun...| All Posts - scottstuff.net
So, as soon as I start writing up how impressive I find the Minisforum MS-01 mini-PC, they just had to come along and announce a new member of the family, the MS-A2. This is very nearly the same as the MS-01, with the same network options2x 2.5 GbE, 2x 10 GbE, and a low-profile PCIe slot that will take an extra NIC. , but it swaps in an AMD 7945HX instead of the Intel i9-13800HThere’s an existing MS-A1 in the same form-factor, but with a desktop AMD CPU, no 10 GbE, and no PCIe slot. Nice fo...| All Posts - scottstuff.net
This is part 3 of a series on running a VyOS router on an Minisforum MS-01 PC. See part 1 (background) and part 2 (hardware) for additional details. To help compare my old Xeon E5-2683v4 router with the new MS-01-based model, I ran a bunch of controlled tests to measure throughput, latency, and power consumption, along with CPU load, CPU speeds, CPU temperatures, IPC rates, and any other metrics that seemed potentially easy. Since this is primarily a home router, I’m really most concerned w...| All Posts - scottstuff.net
In Part 1, I discussed why I’m moving my home router from VyOS on an old 1U Xeon E5-2683v4 to a Minisforum MS-01 mini-PC. I’m largely looking to save power (and money, noise, and heat) by switching to a more efficient platform. My main concerns are (a) how should I actually configure the MS-01 and (b) how will it perform? This article will cover configuration and migration, and Part 3 will cover performance.| All Posts - scottstuff.net
This is New Router Weekend at home, and I’m going to write up the whole process because I’ve learned a few things that others may find useful. There aren’t a whole lot of good benchmarks out there for what Linux routers can do, or how much power they draw while doing it. Hopefully this will help fill a few gaps. This is the first of at least 3 parts. Background For the past couple years, I’ve been using VyOS on a repurposed SuperMicro 1U serverSpec: Intel Xeon E5-2683 v4 (16 cores), 6...| All Posts - scottstuff.net
So, I ran into a fun problem when doing some more benchmarking for a future post: my router isn’t actually distributing traffic evenly across its two outbound interfaces. The router is a Linux system running VyOS, with a pair of 40 GbE links, one to each of a pair of Arista 7050QX L3 switches. These are L3 interfaces, each with its own /30, and I’m using OSPF and BGP over the top of them, all configured for ECMP. The problem is that it’s not balancing outbound HTTP response traffic acro...| All Posts - scottstuff.net
A few months ago I mentioned that I was serving this site from a pair of Minisforum MS-01 mini-PCs. The MS-01 is a smallish PC with an Intel laptop CPU, 2 Gigabit Ethernet interfaces, 2 10 GbE SFP+ interfaces, and a low-profile PCIe slot. I was able to get amazingly good performance–almost 60 Gbps– using a single MS-01s as a web server, but it required adding a 2-port Mellanox 100 GbE NIC to the PCIe slot. Unfortunately, the NIC drew more power than the rest of the machine combined and ne...| All Posts - scottstuff.net
According to The Verge, HDMI 2.2 was announced at CES today, doubling the peak bandwidth from 48 Gbps to 96 Gbps. The Verge’s author wasn’t very impressed by that in general, since HDMI 2.1 can already do 8k at 60 FPS and 4k at 120 FPS. In their view, HDMI 2.2’s extra bandwidth isn’t particularly useful. When using an 8k display for a TV, that’s probably reasonable, but the higher bandwidth is critical for 8k monitors. Sure, in an ideal world everyone would be using DisplayPort for ...| All Posts - scottstuff.net
Sometimes the sheer speed of modern networks still manages to leave me surprised. $ ping -4 router1 -A -c 1000 -q PING router1 (10.0.0.6) 56(84) bytes of data. --- router1 ping statistics --- 1000 packets transmitted, 1000 received, 0% packet loss, time 1998ms rtt min/avg/max/mdev = 0.024/0.055/0.150/0.022 ms, ipg/ewma 1.999/0.062 ms $ ping -6 router1 -A -c 1000 -q PING router1 (2600:a801:30:300::6) 56 data bytes --- router1 ping statistics --- 1000 packets transmitted, 1000 received, 0% pack...| All Posts - scottstuff.net
So, yesterday’s article on sidenotes in Hugo led me down yet another rabbithole, and I’m still not quite sure where it leads. I want to add notes to my blogLike this. and have them usually displayed in the right margin alongside the text. I’d like them to be usable on mobile devices, which don’t really have a right margin. I’d like them to be easy to write, without a lot of extra work or friction. And I’d like them to be supportable long-term, not needing a pile of bad hacks or a ...| All Posts - scottstuff.net
I’m going to be giving a talk at SREcon Americas 2025 in Santa Clara in March. I’m going to discuss how I approached a giant performance problem in a service that I knew almost nothing about; the slightly-less-technical version of the story is on Figma’s blog. This is less a talk on “how I made things go fast”, and more a talk on how you approach unknown problems. What do you know? What don’t you know? What can you measure? How can you perform experiments? Who can you reach out to...| All Posts - scottstuff.net
I spent a bit of time over the weekend trying to add sidenotes to this blog. This is a fairly common thing to want; Tufte makes great use of them, and there’s an entire package for creating Tufte-styled sites. There are no shortage of examples for adding sidenotes to copy from. The problem is that none of them actually look right with the rest of this theme.| All Posts - scottstuff.net
The term “homelab” has always annoyed me. Generally, it’s used to mean something like “having a pile of servers at home” with the implication that they’re mostly used for learning and experimentation. So, er, I certainly meet the “have a pile of servers” at home definition (see this post from last year, for example), and a certain amount of experimentation does in fact occur, but that’s not really what it’s all for. To me at least, it’s home production, not a home lab. A...| All Posts - scottstuff.net
Like everyone else in the known universe, I haven’t been doing a great job of home data backups. I used Crashplan for quite a while and was mostly happy with it, but they pivoted away from home backups as a product years ago. Since then, I’ve tried a number of different systems and none have really worked well enough. I really wanted to like Bacula for Linux backups, but it never really worked quite right for me. The deeper I looked at it the less happy I was with what I saw. It just had ...| All Posts - scottstuff.net
We’ve mostly standardized on nVidia Shields for streaming TV at home since they first came out almost a decade ago. They’re starting to get a little bit long in the tooth these days, but they still work fine, even when streaming 4k content, and they’re still getting software updates. We’ve just had one little problem that until today I didn’t even realize was a problem with the Shield; I thought it was a problem with my TV. My LG C1 TV would power off sometimes when switching HDMI i...| All Posts - scottstuff.net
I mentioned last year that I’m using VXLAN and EVPN in my home network, and since there isn’t nearly enough documentation out there on any of this with Linux, I figured I’d expand a bit.| All Posts - scottstuff.net
As I’ve mentioned before, I have a lot of networking equipment at home. I have a mix of Arista and Juniper switches for complicated things, a bunch of UniFi switches for simple things, a PC running VyOS for a router, a bunch of servers, plus the usual assortment of WiFi APs, random IOT devices, TVs, speakers, computers, and so forth. For the most part, this all Just Works. The hard part is keeping track of what is plugged in where and how it’s all supposed to be configured. There’s no w...| All Posts - scottstuff.net
Since I’m apparently writing more long-form content this month, I decided that it’s time to update Ye Olde Blog a bit. I’ve been using some variant of the hello-friend-ng theme since I switched to Hugo a few years ago. I was getting bored with it, and it wasn’t quite right in a few ways, plus I wanted a theme with more integrated features out of the box (like the ability to create charts on the fly). After doing the usual troll through the giant heap of Hugo themes, it looked like Lov...| All Posts - scottstuff.net
So, I have a problem with photography. No, I don’t mean the “I spend too much money on gear” problem – I can stop any time, honest. The problem is that I have too many pictures and there’s really no point to most of them. I love the creative side to photography, but once I’m done shooting, I’m left with a pile of pictures that no one will ever see, that I’ll have to pay to store for years, and I’ll feel guilty about if I don’t at least spend some trying to organize them. I...| All Posts - scottstuff.net
After doing a bunch of HTTP benchmarking, I’d like to revisit my router’s performance. When I last looked into network performance late last year, I was mostly concerned that my home router could route at least 10 Gbps to the Internet, which it can. At the time, I extrapolated that it was probably good for at least 25 Gbps if traffic is spread across multiple TCP flows. In light of recent events, that seems to be conservative. I’m routinely seeing 56 Gbps traversing my router during HTT...| All Posts - scottstuff.net
After getting 125k HTTP requests per second out of an Intel i5-12600H with 16 threads, I was kind of curious what my desktop could do on the same benchmark. My desktop has a 32-core (64-thread) AMD Threadripper PRO 5975WX, so it brings a lot more horsepower to the table.| All Posts - scottstuff.net
One of the things that happens when you spend decades (!?) dealing with complex application servers is that you kind of lose touch with what the state of the art is in fast web servers. The last time I remember really paying attention was in 2007 or so, when I remember someone going to great lengths to get 10,000 requests per second out of a single web server. I was impressed – that felt like a huge amount of traffic for a single machine to handle. I’m starting a new side project that wil...| All Posts - scottstuff.net
Like a lot of people these days, I spend a huge amount of time each week on video calls for work. During the height of the pandemic, I put a fair amount of effort into video hardware in my home office, with a nice camera feeding video into a Blackmagic Design ATEM video controller and then feeding Zoom or Google Meet from it over USB. Originally I was using still photos as backdrops, but about a year ago I started a recording subtle video backgrounds to use instead of stills. Streams flowing ...| All Posts - scottstuff.net
So, I have a bit of a problem. I keep over-engineering my home network. Like, a lot. I gave some background in 2022, just before I started another round of changes, so I figure it’s time to update things a bit and explain a few of my new favorite things.| All Posts - scottstuff.net
Long, long ago, I described “my dream monitor” here, in terms of a newly announced (in 2007) 52” 4K TV. My take at the time was that it was maybe a bit big (the corners would be hard to use) and maybe a bit low-DPI, but overall it’d be a great improvement over the 24” 1080P screen I had at the time, although connecting it to a computer would be a bit entertaining.| All Posts - scottstuff.net
As I mentioned a week and a half ago, I’ve shifted this blog from being self-hosted to living on Google Cloud, because I didn’t want to spend time maintaining a web-serving infrastructure at home. Obviously, there were many ways that I could have solved this problem. I decided on Google Cloud for a couple simple reasons. First, I work for them, but have very little experience as a customer (anti-disclaimer: I pay retail prices without a discount). Second, I already have an account and bil...| All Posts - scottstuff.net
There isn’t a lot of documentation on debugging SAS-specific storage problems under Linux, unfortunately. I’ve had a couple issue recently that were tied to flaky SAS cables and mostly had to debug them through swapping cables and logic to figure out which device was causing issues. Along the way I’ve learned a few helpful things. First, Linux actually has counters for tracking SAS-level problems. They’re just really hard to find, and very few tools expose them. The kernel would love ...| All Posts - scottstuff.net
As mentioned earlier, I’ve been playing with using a Juniper NFX250 as a backup home router for the past month or two. The NFX line (and the NFX250 specifically) is kind of a weird beast. It has most of the capabilities of an SRX, while adding the ability to run high-bandwidth virtual machines directly on the router. It looks like it’s supposed to sort of sit next to the SRX1500 or maybe SRX380 in Juniper’s lineup, while being much cheaper than either. The NFX250 lists for $4,700–$8,5...| All Posts - scottstuff.net
Let’s just agree on this up front: my home network is overly complicated. There’s a nice, boring LAN with WiFi APs and TVs and Playstations and so forth. There are family members trying to conduct business and school and social lives, and they mostly depend on things working. And then… let’s call it The Dark Side of the Network. Where things get complicated. Where servers live and talk OSPF to multiple L3 switches, so everything keeps working even when I reboot a switch or move cables...| All Posts - scottstuff.net
Another year, another lack of updates. I’ve been running this site on a computer in my garage, but keeping it up and running hasn’t been a priority to me, and were too many moving parts to actually keep it stable without work. So I’ve punted and moved the serving part of this to Google Cloud, and updated Hugo to be current along the way. I’ve had a number of entertaining network adventures over the past couple years, and they’re really too long for Twitter, so I’ll be adding them ...| All Posts - scottstuff.net
This is likely of no use to anyone but me, ever, but I’ll post it anyway. I’m trying to install Linux on a 10-year old Beckhoff CX5020 industrial computer, and everything works fine until about somewhere in the middle of the user-mode startup process when the screen goes blank and never comes back. I tried disabling X and that didn’t matter. Even booting in single-user mode didn’t help. It still went blank while booting. I could SSH in over the network to debug, but couldn’t get a w...| All Posts - scottstuff.net
(See part 1 for the beginning of the process) Yesterday, I set out to measure backlash in the Z axis of Beaver HDZero CNC. I concluded that it had about 15 microns (0.015mm, or roughly 0.0005”) of backlash in the Z axis. Today I’m going to measure the X and Y axes. They’re each a bit different: They use larger ballscrews than the Z axis (1610 vs 1605) with twice the pitch. That means that one turn of the stepper motor goes twice as far. A single step should be 12.5 microns, and the step...| All Posts - scottstuff.net
I wanted to measure how much backlash my CNC has. For today, I’m measuring the backlash in the Z axis, because it’s the easiest, and also because it came fully assembled and there wasn’t much of an opportunity for me to mis-assemble it and screw things up.| All Posts - scottstuff.net
Another week, another chunk of source code. This week, I’ve uploaded Camsync to Github. It uses last week’s Garmin Virb Library to talk to a Garmin Virb camera over WiFi and sync photo and video files.| All Posts - scottstuff.net
Today’s new project is a Go library for talking to Garmin Virb cameras over WiFi. It’s on Github as scottlaird/virb. I’ve implemented almost the entire API covered by Garmin’s Camera Network Services API, including listing, copying, and deleting files; shooting video and still images; and configuring the camera.| All Posts - scottstuff.net
A few weeks ago, I was testing drilling holes in aluminum on my CNC and broke a bit. I’m not going to go into what went wrong here to make the bit break; that’s not the important part right now. What matters is understanding just how much force can be involved in milling, even with a light-duty CNC router like mine. Here’s the video: (Or YouTube) One of the fascinating things (to me, at least) was how much the CNC’s gantry moved. If you watch, in addition to seeing the vise shift, you...| All Posts - scottstuff.net
My CNC build is nearing completion. There are still a lot of things left on my to-do list, but they look a lot more like upgrades and improvements and less like requirements. I’m almost done with the last big assembly project: milling an aluminum base plate for the CNC. I could just use MDF for mounting things to the CNC, but MDF isn’t nearly as solid as the rest of the machine. I decided to buy a 1/2” thick piece of aluminum plate that would cover the entire working surface of the CNC,...| All Posts - scottstuff.net
I’ve open sourced the code that drives my CNC tool changer and put it up on Github as scottlaird/cncatc. At the moment, it’s just an Arduino project (a single .ino file, basically C++) that probably isn’t useful to anyone but me, but I’m planning on cleaning it up a bit and extending it to cover other bits of the tool changing infrastructure. <!–Things->- more hthat I’d like to add soonish: A couple weeks of pending bug fixes, which somehow weren’t in the repo that was pushed to...| All Posts - scottstuff.net
One of the aspects of my CNC that I’m liking is the embedded Raspberry Pi that is powered directly off of the CNC’s 24VDC power supply. When the CNC is powered on, the Raspberry Pi comes on. When the CNC is powered off, the Raspberry Pi goes off. Except, of course, Linux systems don’t like to have the power yanked out from under them repeatedly, and SD cards tend to like it even less. Also, Raspberry Pis don’t like 24 VDC power. (From reading about people’s problems with micro USB p...| All Posts - scottstuff.net
One of the things that I did when I was testing my new CNC was throw random-ish move commands at it. I’d tell it to move a few inches to the left, and then a few inches to the right, then further to the left, and further to the right, and so forth. Eventually, I’d decide that it was working okay and then I’d either turn up the speed or move on to the next part of the assembly process. There are at least two problems with this, of course. First, that’s not even remotely random. Second,...| All Posts - scottstuff.net
With small CNCs, the easiest way to adjust height (and sometimes the other axes) is to use a probe block. Generally, they’re just aluminum blocks of a known size that the CNC controller sends 5 volts through. Then you connect a ground clip to your spindle bit, and as soon as it touches the probe block the controller sees that electricity is flowing and knows that the tool is touching the probe block. If you know that your block is 15mm thick and sitting exactly on the top of your material, ...| All Posts - scottstuff.net
I recently found myself wanting a GoPro-ish camera that I could power for extended periods of time using a cable and download images to my desktop over WiFi programmatically, without needing to pull a MicroSD card out or fire up a mobile app and drag MP4 files off via my phone one at a time. After looking at options on B&H’s website, I decided that the Garmin Virb Ultra 30, even though it’s getting kinda old, was probably the best choice for my needs. Garmin publishes a doc with a reasona...| All Posts - scottstuff.net
Once I decided that I was going to upgrade my CNC, the first upgrade that I commited to was replacing the DeWalt DWP611 that came with my Shapeoko. The DeWalt is really just a small trim router. It’s well made but it has a very limited RPM range–16,000 to 26,000 RPM. A lot of milling activities need lower RPMs than that. While researching possible options, I kind of fell in love with the idea of a tool-changing spindle. Expensive professional CNC machines generally have the ability to swa...| All Posts - scottstuff.net
After owning a Shapeoko 3 for several years but barely using it, I suddenly found myself swimming in milling projects. I’d finally spent enough time in Fusion 360 to be comfortable drawing up what I wanted to build, and my spate of Christmas projects had given me enough experience to finally be comfortable using the CNC. So, it was time to move on to bigger projects. Except… not quite yet. I managed to crack one of the plastic wheels on my Shapeoko almost immediately upon using it this ye...| All Posts - scottstuff.net
Continuing on in the “making gifts with computer-driven tools” series, next was a quick box for my mother’s birthday. I drew up a small CNCed jewelry box in Fusion 360. I milled it out of 2 small pieces of Sapele. I deliberately left milling marks inside; in this case I liked the texture it gave to things. Also, I was rushing to make a shipping cutoff. Inside The box closes with magnets; they’re hot-glued into place. Outside| All Posts - scottstuff.net
As mentioned before, I decided to hand-make as many Christmas gifts as possible this year. I’ve been collecting cool tools, but haven’t really done enough to use them or exercise my creativity. One of the things that I decided to make was a set of acrylic laser-etched topographical maps. I couldn’t find any easy examples online, but I know that I’ve seen similar things before–basically, you use a laser cutter to remove material from the back of a block of acrylic, leaving a 3D model...| All Posts - scottstuff.net
In a fit of optimism, I pre-ordered a Glowforge laser cutter in 2015, but it didn’t actually arrive until late 2017. I’d have been more excited about it shipping if I hadn’t been out of town for a family funeral when they delivered it. The end of 2017 was a giant mess, and I didn’t really get much use out of the Glowforge during most of 2018. For Christmas 2018, my wife and I decided that we really wanted to try to make presents for as many people as possible, and to encourage our kid...| All Posts - scottstuff.net
Back in the mists of time (oh, say, 2003-2010), I was a semi-prolific blogger. I had a reasonably public presence as an open source programmer and needed a way to announce new releases of software, but also to comment on interesting things that I learned or just to rant or vent. Over time, the amount of time that I spent on public non-work computer work shrank until there was really no point in updating a blog, and indeed not really even any time to keep it up to date and running. For a while...| All Posts - scottstuff.net
Something reminded me recently of project that I worked on around 10 years ago. I was a simple, clean solution to a problem that had remained unsolved for over a year. It was also one of the most horrible hacks that I’ve ever put into production. At the time, I was working for a startup that was suffering from explosive growth. We’d began rolling out new sales and engineering offices all over the US faster than we could build the infrastructure that we needed to manage the offices. We bui...| All Posts - scottstuff.net
So, lo and behold, it’s Saturday morning. It’s the day before Mother’s Day, and we’re going to be driving most of the day tomorrow, so it seemed like the perfect time to cook something a bit special for breakfast. After thinking about it a bit, I decided that what I really wanted was German Pancakes. That posed a bit of a problem. You see, German Pancakes want to be cooked in the oven. And we’re in the middle of remodeling. The oven goes about where the white bucket is sitting. Unti...| All Posts - scottstuff.net
These iPad reviews and App Store links are starting to get hazardous to my (financial) health. I mean, I really don’t need one. I have 2 Nexus One phones in my pocket, a Kindle DX, and I rarely go anywhere without a Macbook Pro. The biggest thing that I’d use a small tablet for would be to-do/calendar management, and I don’t think the iPad is particularly good at that–it’s not really possible to integrate information from multiple silos (Calendar, To-Do, Email, etc) given the iPhone...| All Posts - scottstuff.net
Back in 2004 or so, I desperately wanted a 8-10” tablet from Apple. Something that would let me browse the web and check my mail while I was away from my desk, without forcing me to lug my laptop around. Something that would let me play with new ideas over lunch. Something that could use to build cool stuff. Somewhere along the way, I decided that it was never going to happen. There just wasn’t enough of a market for it, and the form-factor requires UI that is drastically different from d...| All Posts - scottstuff.net
I’m currently filled with self-loathing, as I failed in my attempt to be the last person in North America without a Facebook account. It’s still entirely unclear why I’d want to share a single social network with my grandmother, my neices’ friends, and a bunch of current and former co-workers. I mean, really–what could I possibly say that’s appropriate and interesting for all of them? Bleh.| All Posts - scottstuff.net
