tl;dr - At the risk of continuing a flamewar, I’m going to suggest you never use merge commits. Yup, end of the year is flamewar time. The context of this particular (possibly unreasonable) suggestion comes with some context. I assume you do one of the folllowing in your hacking: Make branches (i.e. a fully private repo/one you have maintainer/committer access to) Fork repositories and make pull requests back to a main repo Merge strategies In general I consider the contrast here between tw...| VADOSWARE
tl;dr - pinentry (in particular pinentry-gnome3) stopped working, pinentry-emacs didn’t work well with daemon mode, just use pinentry-gtk or pinentry-qt5. This post outlines a shallow dive into a rabbit hole for a broken pinentry setup w/ Emacs – if your setup is already working, skip this. What is pinentry? When a program like Gnu Privacy Guard (AKA GnuPG, gpg) requires user input, but that entry has to be secure (and possibly access system password stores/keychains), how do they do it?| VADOSWARE
tl;dr - I wrote some Inter-Process Communication (“IPC”) experiments in Rust w/ slightly more abstraction and more realistic serialization (JSON) – Check out the code. UPDATE (10/10/2024) Thanks to feedback from Reddit, I've updated my references to "throughput" to "latency" which is the more accurate description of what roundtrips measures. Rust is kinda hot right now. OK it’s been hot for a while – and I’ve been enjoying using it, but that’s not we’re here to talk about today.| VADOSWARE
/ tl;dr - I switched from tmux to zellij, but returned to tmux primarily because of missing no-mouse on-screen text selection, and easy screen rotation To set the record straight – both tools are great. Zellij is awesome. I made up my mind to try it for many reasons: Zellij looks great Zellij has a great set of out-of-the-box features Zellij is extendable Zellij is written in Rust Obviously, tmux is also great – I consider it the defacto choice these days (though there are screen diehards...| VADOSWARE
TL;DR This post exists because I forgot how to start locally installed Postgres before a talk. I don’t normally have a section like this, but since I’ll probably be the one referencing this post again in the future: apt install postgres # or whatever for your distro initdb --no-locale -D <data-dir> pg_ctl -D <data-dir> -l <log-file> start on MacOS: brew install postgresql brew services start postgres psql postgres Context Every once in a while you forget how to use a tool and are humbled ...| VADOSWARE
tl;dr - Update your Cargo.toml and make sure doc_auto_cfg is turned on in lib.rs How do you properly show feature specific functionality (and mark them with the required feature) for library crates in 2024? First, you need to update your Cargo.toml: [package.metadata.docs.rs] all-features = true If you don’t want all features to be enabled (if you want some features and related code to be legitimately hidden from docs), you can use specific features:| VADOSWARE
+ tl;dr - I did the thing where authors disappear for year(s) due to $DAYJOB – I’ve been working with Rust and WebAssembly. I’m going to try to write a bit more! I’m not back but it has occurred to me that this blog hasn’t seen any content in a whole year (and a day). Sorry about that – I’ve been busy working at $DAYJOB, and have been doing a lot of Rust and a bunch of WebAssembly.| VADOSWARE
tl;dr - A combination of pre-baked very heavy docker images, sccache, and explicitly setting CARGO_HOME and CARGO_TARGET_DIR will do it. If you’re squeamish about large docker images in your build pipeline, close this tab. Today was the last day I put up with unnecessary rebuilds in my Rust CI pipelines. Sure, I probably shouldn’t have ported this particular project to Rust (Nimbus is powered by Typescript and Rust), but when someone gives you strong type safety, performance, and static b...| VADOSWARE
tl;dr - __dirname and package information are a bit harder to get from an ES module. In case you’re not using them yet, take a second to read up on Javascript (“ES” – EMCAScript) modules – they’re quite the large feature/platform change, big steps up (mostly) from CommonJS and RequireJS (AMD). One of the things that isn’t quite as nice as RequireJS in NodeJS-land was is how difficult it is to programmatically get package and version information!| VADOSWARE
tl;dr - AWS App Runner is awesome for deploying applications/services as containers, and it can finally be used with apex domains (ex. domain.tld) It looks like App Runner has finally added support for top level domains: AWS release notes for Route53 A while back I did a mini bake-off of simple AWS container deployment mechanisms for a customer. I explored some options that are container friendly: Elastic Container Service (ECS) AWS Elastic Kubernetes Service (EKS) App Runner Elastic Beanstal...| VADOSWARE
tl;dr - While guest writing a Supabase blog post on how identifiers work in Postgres, I was able to build pg_idkit which makes it easier to generate new UUIDs in Postgres UUIDs are better supported in Postgres than in a lot of other databases. Postgres has the uuid datatype and you can easily generate UUIDs as primary keys with SQL like this: CREATE TABLE example ( uuid uuid PRIMARY KEY GENERATED BY DEFAULT AS IDENTITY uuid_generate_v1mc(), col1 text NOT NULL, col2 integer NOT NULL, created_a...| VADOSWARE
tl;dr - This post was inspired by Carlos Fenollosa’s post (and the resulting discussion on HN). Self hosting email is the easiest it’s ever been – don’t let the email provider oligopoly (Gmail, Microsoft, Yahoo, etc) win. Discussion up until now: Deliverability for self-hosted emails has never been harder The top page on HN had an interesting post on it titled “After self-hosting my email for twenty-three years I have thrown in the towel.| VADOSWARE
tl;dr - Turns out Amazon SES has an IP that is currently blocked by Spamhaus Imagine my susprise while investigating some email delivery issues lately when I discovered that Amazon SES was using an IP that is currently blacklisted by Spamhaus! The IP in question is 54.240.27.189 and of course I took images to prove it happened: And there ya have it – I don’t know what automated system is going to ping someone (or another robot?| VADOSWARE
tl;dr - Large components of systems you build deserve an interface, even if there is only ever one implementation. If you want your code to be good, put a bird interface on it! I’ve gone back and forth, but for the last maybe 5 years my opinion has hardened. My stance is now that it is unquestionably right to always build an interface for large components of a given system, even if you only have one implementation early on.| VADOSWARE
tl;dr - I started a newsletter for ideas, and I love seeing subscribers with emails like someone+ideas@provider.com Last month I started a newsletter for unvalidated ideas. I know what you’re thinking – Execution is everything – aren’t ideas nearly worthless? Wouldn’t that make the newsletter worth zero or possibly less due to cost of reading? Well I don’t have the answers to any of those probing questions, but one of the most unexpected things I’ve run into is just how much I e...| VADOSWARE
tl;dr - If you haven’t heard of lstopo before, you’re welcome. While watching a recent ServeTheHome video on Youtube (I really should watch that channel more, they’ve been quality for an very long time), I noticed them mention a tool I’d never heard of before called lstopo. It was mentioned casually along wiht lscpu (which is very useful and well known) and it had an awesome graphic (kind of like the difference between htop and top) which was very useful.| VADOSWARE
+ + tl;dr - I installed Loki and Fluent Bit on my Kubernetes cluster for some light log retention, in the past I’ve used EFKK but this setup is lighter and easier for low maintenance projects. Assuming you have a Grafana instance handy, Fluent Bit + Loki is pretty great for a low effort log aggregation! It’s a relatively “new” stack compared to options like Graylog. Another great option in this space is Graylog!| VADOSWARE
tl;dr - CSI is awesome but it doesn’t help you with cross-StorageClass data migration. I wrote a small, painfully procedural script & docker container called pvcloney to perform this task repeatably for myself. The code is a great crash-course to @kubernetes/client-node Today’s yak shave is of the storage variety – useful only to people who use/evaluate different storage mechanisms on their clusters. I’ve gone hostPath -> Rook -> Longhorn -> OpenEBS ZFS LocalPV -> OpenEBS LVM -> Rook ...| VADOSWARE
tl;dr - cert-manager and moved from Bazel to Make (i.e. Makefile) based build in 1.8.0. I’m into Makefiles (+ kustomize) for builds and deploys so I’m taking an undeserved victory lap. A welcome surprise during a long round of updates Recently I’ve been upgrading some of the critical services in my cluster and like everyone else, cert-manager is one of the best parts of my cluster. The cert-manager maintainers and contributors have saved what I can only imagine is millions of person yea...| VADOSWARE
UPDATE (2022/08/02) Looks like Svelte has a new mechanism for dealing with runtime/static env -- check out the docs on $env/dynamic/private I'll probably be using this method in my projects for a bit, but will be migrating towards the built-in solution! tl;dr - I use ENV in Sveltekit by loading .env/.envrc formatted files (thanks direnv!) and making sure to do filtering only for PUBLIC_* in hooks.ts before passing to the frontend, rather than using the usual VITE_* replacement approach| VADOSWARE
tl;dr - We should call the market-validating pre-launch landing pages people make “prospecting pages”. Everyone says it, it sounds trite, but I had to figure it out for myself – 0-1 startups are many times market discovery vehicles. Smarter entrepreneurs enter existing markets and do something better than existing incumbents, but I think even that could be phrased as “market rediscovery”. Either way, I’ve learned that I’m not one of those smart entrepreneurs!| VADOSWARE
+ tl;dr - Host some Thunderbird autoconfig formatted XML, and text files at the right places (thanks to MTA-STS) so clients can have easy-peasy email setup. This was a fun chance to flex my Traefik installation, and kcup. You’re hosting your own email server right? Cool, me too. Good thing we ignored all the naysayers out there. In the past I’ve run postfix + dovecot setups and now I run many small maddy instances (for various projects), but I never looked into autoconfig.| VADOSWARE
tl;dr - AGPL aligns creators, maintainers, tinkerers, and users of software while leaving avenues for monetization. AGPL isn’t actually anti-hosting (IANAL) but is instead a forced improvement repatriation scheme. Nimbus Web Services will host AGPL software and contribute back – hyperscalers can too. Read the fundamentals of the AGPLv3 and Why the affero GPL. UPDATE (2022/04/27) After some discussion on HN I figured I'd highlight it here -- this post is meant to address software that can ...| VADOSWARE
+ tl;dr - This post explains how I built LoginWithHN.com (LWHN), an unofficial OAuth2+OpenID connect provider for HackerNews. LWHN is for builders who want to share things with the HN audience, and cuts down friction like any other OAuth2+OpenID connect provider. LWHN is now open to the public for new client app registration, so check it out if you’re interested. DISCLAIMER: I wrote this article with the knowledge that Ory would share it later on their blog/marketing channels.| VADOSWARE
tl;dr - Adding a rebuilt node with a different IP under an already-used node name did not sit well with Calico in my setup. I could see the problem (mismatched IP address information in Node resources), but I couldn’t fix it – ultimately I gave the new node a new never-before-used name (i.e.. worker-< n+1 > instead of worker-2). Recently during an expansion and some shifting of my k8s cluster, I found out (the hard way, of course) that if you add a new node that has the same name (let’s...| VADOSWARE
While growing my Kubernetes cluster and working on some Ansible scripts for my infrastructure I recently realized that I’ve been under-utilizing one of the staples of ansible code, tying node-specific information to nodes in the inventory file! It was harder than I thought to use so this a quick writeup on how to do it: Given an inventory file like this: all: children: some_machine_group_name: hosts: some.machine.you.have: # possibly 'xxx.xxx.xxx.xxx' os_partition_size_gb: 128 drive_paths: ...| VADOSWARE
tl;dr - Make sure the reported linux version of uname -a and pacman -Q linux versions match after updates (or just be more cognizant of full system upgrades, i.e. updating the linux package). Recently after restarting and performing an system update (pacman -Syu) post-restart, I tried to start doing some dev work and was greeted by a broken docker systemd service (root-full docker, so not the user-level docker service you’d use in rootless docker).| VADOSWARE
+ tl;dr - You can expose SSH over the same port HTTPS runs on (443), turns out you can run a combination of stunnel (in my particular case stunnel3) and sslh as sidecar containers that work together to some container that runs SSH (i.e. sshd). At the ingress layer Traefik makes this easy to pull off by providing the IngressRouteTCP CRD along with TLS passthrough. The infrastructure-as-code is up on GitLab.| VADOSWARE
+ UPDATE (03/23/2021) After trying to restart my cluster I ran into issues not having the --kubeconfig option. kube-router couldn't find 10.96.0.1 (the API server) and CoreDNS couldn't start because it couldn't find anything (since kube-router) was down. It's a chicken and egg and though this PR was supposed to solve it, it's certainly still a problem. Not sure if this is because I'm on older versions of kube-router (1.| VADOSWARE
)
+ tl;dr - I made a component library for Mithril called Maille – if you’re into Mithril, check it out! There comes a time in a young front end developer’s life when they feel the need to make yet another front end framework. In my case that time is now, and the frontend framework isn’t so much a framework as it is a component library for an existing component library called Mithril.| vadosware.io
)
+ tl;dr - The kata containers 3.0.2 documentation on custom kernels is great, but I ran into some issues, so I rehash exactly what I did below I recently spent some time exploring how to build custom kernels to work with kata, and it’s way easier than it seems, though the docs are a bit stale/don’t work as expected. Generally there are a few reasons you might want to do this, but usually it’s to enable kernel flags/features that weren’t enabled before.| vadosware.io
)
Recently I ran into some GPG errors with pacman that were quite frustrating to fix. The symptoms look something like this when trying to run a command like sudo pacman -Syu: error: GPGME error: No data error: GPGME error: No data error: GPGME error: No data :: Synchronising package databases... core 974.9 KiB 1134 KiB/s 00:01 [#########################################################] 100% extra 974.9 KiB 1547 KiB/s 00:01 [#########################################################] 100% commun...| vadosware.io
)
tl;dr - Some tips for writing better makefiles (using preambles, generating help text), and why you might want to use just instead. The makefile preamble This preamble is online @ davis-hansson.com SHELL := bash .ONESHELL: .SHELLFLAGS := -eu -o pipefail -c .DELETE_ON_ERROR: MAKEFLAGS += --warn-undefined-variables MAKEFLAGS += --no-builtin-rules ifeq ($(origin .RECIPEPREFIX), undefined) $(error This Make does not support .RECIPEPREFIX. Please use GNU Make 4.0 or later) endif .RECIPEPREFIX = > ...| vadosware.io
)
tl;dr - I hopped on the Software Sessions podcast to talk about yak shaving and some projects I’ve been working on. Take a Listen (it’s kinda long so consider the transcript as well) I was contacted a while ago by Jeremy who runs the Software Sessions podcast. I’d actually come across his Podcast in the past so I was super excited to get a chance to interview and talk about some of the projects I’ve done in the past.| vadosware.io
)
tl;dr - Service Catalog might ahve been a redundant concept after all… Was it just the hobby-horse of large integration-happy stakeholders? I’ve always been sort of confused by the concept of the Service Catalog – so much so I’ve posted about it before to get a feel for public consensus. For those who don’t know what service catalog was – here’s a great post from WeaveWorks on it People have put hard work into supporting this concept, chairing the SIG, and pushing it forward, bu...| vadosware.io
)
+ + tl;dr - Slonik is my go-to Postgres driver for recent projects (mostly SvelteKit-based) – it is amazingly lightweight, simple and powerful. I prefer it even though it lacks time-saving ORM methods (ex. Model.save()). UPDATE (2023/05/09) As much as I loved using Slonik due to how lightweight it is, it just isn't a reasonable choice when compared with something like TypeORM. With issues like this one in the ecosystem, Slonik seems to constantly introduce breaking changes without brining t...| vadosware.io
)
UPDATE (2022/06/30) After some feedback from HN I also have an audio sample if you want to hear it! (Turn the sound down a little! There's also a bunch of dead air after 15 seconds.) Your browser does not support the audio tag. tl;dr - Don’t chuck the Sony 1000XMs/AirPods just yet. Bluetooth headsets are more convenient, can have better mic quality, and none of the downsides. That said, throat mics might an improvement over a really bad, 20 year old standalone wired built-for-computers micr...| vadosware.io
)
tl;dr - Your program should support 465 and optionally 587 – implicit TLS via submissions over port 465 is where we’re moving to, according to RFC 8314 from 2018. UPDATE (2022/12/05) Thanks to some feedback from very helpful folks on r/sysadmin, it's clear that STARTTLS is even worse than I thought! After experiencing a bit of pain making my somewhat complicated email setup work properly with SNI-based forwarding, I realized that up until now I’ve also believed that STARTTLS on 587 was ...| vadosware.io
)
tl;dr - You should be using direnv to load ENV variables dynamically per-project. You don’t necessarily have to buy into the entirety of the 12 factor app manifesto, but you should almost definitely be using ENV-as-configuration by this point. Even if it’s just a CONFIG ENV var that points to a config.toml that your application reads, passing configuration via environment variables is the tried and true method of configuring applications these days (inside a docker container or out).| vadosware.io
)
tl;dr - I started a newsletter called Awesome F/OSS – I’m featuring one awesome Free and Open Source project every day for the next… forever(?). I’ve got an huge list of Free and Open Source (F/OSS) projects that I think are awesome. I get to recommend them to friends and help people pick solutions a few times a month, but lately it’s felt like only being able to recommend solutions to such a small audience is a waste.| vadosware.io
)
tl;dr - Crypto does have a few usecases, ETH2’s switch to Proof of Stake means it’s worth building on, and Blockchain represents a chance to reshape policy and regulation that is ossified. Slightly longer TL;DR I am keenly aware of how much everyone hates reading about crypto so I’m going to try and front load as much of my thesis as possible, as plainly as possible. You do not have to read this whole thing, you can literally just read this section and be done.| vadosware.io
)
tl;dr - Use this shortcut (shared on RoutineHub) to open .ics files (invite.ics/calendar.ics) and add them to your iOS calendar You know those attachment files that you get when someone sends you a meeting invite – invite.ics/calendar.ics? Well for some reason they don’t open right on iOS. iOS will let you review the details of the event, but won’t let you add it to your calendar. It’s infuriating. Generally there are only two known ways to get a calendar file to actually show the “...| vadosware.io
)
tl;dr - Remember FirefoxOS and the Firefox Phone(s)? I do, because I still have two (!) Fx0 handsets sitting in my closet. Remembering Firefox phone Every once in a while I like to spend some time clutching my fists and thinking about what the Firefox phone could have been. I usually feel like that after looking in my closet and running my hands across this thing: Taken on an iPhone SE2 😢| vadosware.io
)
tl;dr - stream-of-consciousness notes as I stumbled through upgrading a k0s cluster from single-node to HA control plane. You probably shouldn’t attempt to do this though, just make a new HA-from-day-one cluster and move your workloads over. Recently I went through the somewhat traumatic experience of upgrading a k0s Kubernetes cluster from a single master (I was running 1 controller+worker node with other worker-only nodes) to a group of HA controller-only nodes with the same set of workers.| vadosware.io
)
tl;dr - I wrote a small library for CountMin sketches in Haskell (you can find the code on GitLab, @ mrman/haskell-countmin). I then try to optimize it and fail, but throw in an example of doctest usage, CI with Gitlab, and how to host the generated Haddock on GitLab Pages. In some work and exploration earlier this year I found some time to read more about sketches – probablistic data structures usually used for summarizing data.| vadosware.io
)
tl;dr - I gave two short talks to different audiences on awesome things you can do with Postgres and how you can use containers with GitLab. You can find all of thses on my talk repo or read them below. I recently gave a couple talks, one at the Tokyo Tech Meetup and another at the Gitlab JP (this one is in japanese, so you may need to bone up on your 日本語) which I thought a wider audience might find interesting.| vadosware.io
)
tl;dr - I upgraded my small single-node cluster (6C/12T 16GB Ubuntu 18.10 Hetzner dedicated server) to Kubernetes 1.15 from 1.13. See the TLDR summary section section for the list of steps (basically, use kubeadm). I run a tiny single-node kubernetes cluster which hosts this site, client demos, experiments, and many of my own projects – I’ve written about it lots over the years, and today I thought I’d cover one of the mostly mundane (for my setup) parts of running Kubernetes – upgrad...| vadosware.io
)
tl;dr - I write my thoughts on why I don’t (as of now) quite fully trust the CNCF. I also posted this to hacker news to get some feedback. If you’ve got a tin foil hat close by this would be the time to put it on. I’ve vaguely distrusted the CNCF for a long time now. While I like the majority of what they’re doing (helping fund and manage open source projects), it’s never sat right to me… Companies don’t move out/donate large sums out of pure altruism, and the consistent and per...| vadosware.io
)
+ tl;dr - I had to use a slightly modified indentation setup for java-mode to get close to the IntelliJ Java indentation scheme while doing some Java development. I also had to run Maven (mvn) lots from the command line so I’ve included some tips on that. If you hate unnecessary rambling, jump to the elisp code Generally, working with the Java programming language is not my first choice, but sometimes when money is involved I absolutely ignore my first choices.| vadosware.io
)
+ tl;dr - I bought a System76 Oryx Pro to cut down on the number of computers I own. Initially the cooling system was broken but System76 support was fantastic; repaired it and sent it back post-haste. System76’s commitment to F/OSS, use of Rust and the ever-heroic work of Arch Linux contributors, along with my extensive history of making mistakes made it a relatively pain-free to install Arch with proper hardware support.| vadosware.io
)
+ tl;dr - I started up a local Zulip instance on my tiny k8s cluster for some friends and I to use – it was surprisingly challenging to do so this post contains the recipe (k8s resource configs). While everyone is getting really into Slack and other workplace-use chat programs, the group of really good open source options has been growing steadily. At this point, Slack is so well known and used that it’s sort of become a must-have for trendy startups and midsize companies, at least in my ...| vadosware.io
)
+ tl;dr - A while back I had to set up Python E2E tests on CircleCI. It took lots of experimentation so I thought I’d share While working with a previous client I had the distinct misfortune of not working with Gitlab’s CI system but instead with CircleCI’s. Alright, CircleCI isn’t that bad, but it is distinctly more complicated and less well documented than GitLab’s and as such is harder to use.| vadosware.io
)
- tl;dr - I took down my k8s cluster by letting it’s TLS certificates expire. Regenerating certificates, deleting /var/lib/kubelet/pki/kubelet-client-current, restarting the kubelet, recreating service accounts and restarting pods/services/deployments/daemonsets was what got me back to a working system without blowing everything away. Towards the end of 2019 I was visited by a small bit of failure adventure – resuscitating my tiny Kubernetes cluster after it’s TLS certificates had expir...| vadosware.io
)
tl;dr - I finally got around to releasing exceptionfree-readfile, a small haskell package for reading files without logging exceptions when running a binary built for profiling (+RTS -xc -RTS). Why write exceptionfree-readfile? I decided to create exceptionfree-readfile while trying to clean up the output of my +RTS -xc -RTS output (I also wrote about it on reddit). For those unfamiliar, when haskell programs are built with profiling enabled, the runtime system can be triggered with special o...| vadosware.io
)
+ tl;dr - I set up a mailing list (for this blog) with Mailtrain on my tiny k8s cluster. Along the way I created a small rust binary for converting POST-ed forms to mailtrain API calls and a Mithril component for mailing list signupg call-to-actions. UPDATE (06/02/2020) A reader named Damien pointed out that I didn't leave a decent example for how I was using kustomize so I wanted to add to the mrman/makeinfra-pattern repository to point out how.| vadosware.io
)
+ tl;dr - UDP support is coming to traefik soon, so I’m updating my cluster’s traefik to be ready to take advantage of it and all the other new features. Going from v1 -> v2.2 (the latest) requires some config changes so I detail them below. NGINX is one of the most venerated load balancers on the internet and when I first set up my tiny kubernetes cluster I used it.| vadosware.io
)
tl;dr - I share some code snippets to test endpoints on a servant-powered API I work on. The servant tutorial (and the code on the servant-client hackage page) is better but this code might be a tad more realistic. I’m a pretty verbal supporter of Haskell and when I write APIs in Haskell I exclusively use Servant – it’s a fantastic example of how well-crafted types and use of Haskell’s advanced type system can lead to code that is correct, beautiful, and performant (depends what you c...| vadosware.io
)
+ tl;dr - I wrote a tool called rbb (redis-bootleg-backup) in Rust for taking backups of redis instances when all you have is a client connection. It does redis backups the dumbest way possible – enumerating all the keys (via the KEYS command) and dumping them all (via the DUMP command). A little while ago I had a friend run into a rather interesting problem – he had access to a Redis cluster but did not control the hosting/instance and could not use the normal redis backup mechanisms (--...| vadosware.io
)
+ tl;dr - During E2E testing I spin up containers of dependencies to run tests against them – the setup I normally use recently stopped working so I fixed it. E2E tests are the most important tests for a project – it doesn’t matter if no endpoints malfunction due to invalid input if no users can checkout or perform crucial functions with good inputs. For past projects, I’ve used containerization (docker) containers in E2E tests in CI to test systems I’ve built as faithfully as possi...| vadosware.io
)
+ tl;dr - Overly complex use of postgres DOMAIN and composite TYPEs (with CASTing to jsonb) used to support the better-than-naive definition of a price as a currency + natural number. For actual production usage do yourself and your team a favor and use a currency type column + integer for amount (ex. cents in USD). For those who like hacking I also got TypeORM to read/work with the composite TYPE with even more dirty dirty hacks.| vadosware.io
)
+ tl;dr - I go through upgrading cert-manager (formerly known as kube-lego) from version 0.4.0 to 0.9.0 (due to deprecations of cert-manager 0.8.1 and lower) to 0.16.0. After upgrading to well known issues with the upgrade from v0.15 to v0.16 make me downgrade to v0.15. Background Let’s Encrypt is legitimately one of the best things to happen to the internet in the last decade. For those who like to build distributed systems or over-invest in building platforms to depoy only a handful of ap...| vadosware.io
)
+ + tl;dr - Add some functions that run docker (or any other container manager, like podman) during your E2E tests for effortless E2E tests including “real” versions of your dependencies. After developing a bunch of nice new features, testing is important – while I rarely find myself writing intricate mocks unless I’m in a Java or Ruby codebase, E2E tests are what I spend most of my time writing.| vadosware.io
)
+ tl;dr - Quick guide to getting rootless containers up and running on Arch Linux (also see the excellent Arch Wiki entry) Steps As usual, the Arch Wiki is a fantastic resource, and has basically everything you need, if not a little bit spread out. The relevant pages you’ll want to look at: https://wiki.archlinux.org/index.php/Podman https://wiki.archlinux.org/index.php/cgroups#Switching_to_cgroups_v2 https://wiki.archlinux.org/index.php/Kernel_parameters 1. Enable the kernel.unprivileged_u...| vadosware.io
)
tl;dr - Make sure /etc/subuid and /etc/subgid have good ranges, podman stores it’s auth config at ${XDG_RUNTIME_DIR}/containers/auth.json, and make sure you use :z on volumes you want to re-use with different containers (ex. postgres). After getting my podman setup working I ran into a few issues that were work writing up real quick just in case anyone else runs into them. ISSUE: Not enough subuids and/or subgids While building a relatively large image with many layers I ran out of subuid/s...| vadosware.io
)
tl;dr - nuxt-auth is a good plugin, but does not support a bread-and-butter feature like HttpOnly cookie based authentication (which is an OWASP Top 10 issue), so here’s how to do basic cookie authentication without nuxt-auth, with routing and backend considered. No clone-able repo, because I’d rather you thought hard about whether you want to use random auth code from a random blog on the internet. UPDATE (01/07/2021) Nuxt has since resolved the issue by introducting the cookie scheme in...| vadosware.io
)
+ tl;dr - Bluetooth died after a recent upgrade to Linux kernel 5.9.1 on Arch which people noticed nVidia didn’t support – I upgraded anyway, graphics were fine, but Bluetooth was not. So very recently a release of NVIDIA’s drivers was confirmed not to work with Linux kernel 5.9.x. It’s been discussed widely on the internet (by those affected) so I won’t go into it here: Phoronix Article Phoronix Forums A warning on NVIDIA forums Arch forums r/archlinux Hacker News Read those source...| vadosware.io
)
+ tl;dr - I upgraded traefik and added some resources (IngressRoute, Middleware) to get a better security score from Mozilla’s (HTTP) Observatory. The upgrade from 2.2.0-rc1 to 2.3.2 came with a few breaking changes so it was a bit involved (see Traefik v1 to v2 docs and also the general v2.x migration docs) I recently came across an insanely helpful and concise Written by Sam Texas from simplecto.com post while surfing r/Traefik subreddit.| vadosware.io
)
WARNING: This post is incomplete! This post was started actually way back in 2018, but I just lost interest writing it. I thought it was a shame to waste the procedural style and explanations in here, so I'm releasing it in it's incomplete-ness. Here in 2020 I think it's safe to say that I probably won't regain interest in this topic, especialy with Vue 3 reaching the mainstream. This guide was meant to explain how to get started with Vue 2, but starting from the perspective of how you would ...| vadosware.io
)
tl;dr - Screen brightness/light control with redshift/flux and eye break timing with safeeyes/breaktimer is good for your eyes, check it out. Deutsche Welle (DW) provides an excellent service that I can’t stop raving about to friends – they have a YouTube channel for documentaries that I love watching. I mostly use YouTube for watching conferences and getting at some of the insane wealth of information out there in the tech community, but I do enjoy watching some news from time to time am...| vadosware.io
)
+ + tl;dr - I over-engineered a script to run a small experiment to figure out what was the right cluster size for PM2 in various CPU + RAM configurations. Turns out having lots of PM2 instances is really good when there’s not much memory and only slightly bad when there is sufficient memory, so a blanket recommendation like 32 actually works out. The code is on Gitlab – you can skip to the results.| vadosware.io
)
- + tl;dr - Turns out the tar archives created by podman didn’t work properly for my kubernetes cluster, and podman doesn’t support linking. I went back to docker (which is containerd underneath), and am a happy camper again. I also discuss some options for hosting your own registries. After [writing about switch to podman]podman-post (and writing a follow up), I recently had to switch back to docker and enable rotoless containers there (the ever-useful Arch Wiki makes it easy).| vadosware.io
)
tl;dr - Wrap your robust ThingService/ThingStore in a web component and adhere to a minimal API so you can reuse them with greater ease. View the Demo or check out the code in the Gitlab Repo. The Idea in historical context I’ve been spending a lot of my time lately working on a product (yet to be released) for making SaaS companies and startups in general easier to launch (I’m talking like 0 to landing + client portal + charge a paying customer via Stripe and issue an API key in 5 minute...| vadosware.io
)
tl;dr - I published async-wait-for-promise to NPM which helps you wait for the result of a promise with a timeout. It’s very similar but slightly different from some other small NPM packages. You can find the code on GitLab. Well I guess we all joke about things like this until you’re part of it. I recently found that I wanted to wait for a specific condition in my testing code but couldn’t find code that would work exactly like I wanted to use it, so I wrote my own package – async-wa...| vadosware.io
)
+ ( || ) DISCLOSURE (03/16/2021) Sitting around after releasing this post I realized that I accepted becoming a "Traefik Ambassador" due to my previous post about Traefik (the one about adding HTTPS settings easily), and submitted the post to their listing when they asked. Such treatment can certainly bea source of bias so I want to make it clear that this is not a sponsored post in any way (I have not been compensated).| vadosware.io
)
/ UPDATE (03/30/2021) A bunch of readers have submitted suggestions and changes to both the Rust and the Go code so I've updated them and released new versions as appropriate! I added a section to the bottom of the post so check that out! -- the biggest changes were in the Go-related code. UPDATE (03/26/2021) A reader named Pavel (Pawel) helped out on the Go implementation by using []byte and io.| vadosware.io
)
tl;dr - I did some maintenance and upgrades to packages on my small k8s cluster – I added NodeLocal DNSCache (which prompted an upgrade to prometheus and node_exporter along the way). I also upgraded kube-router (v1.0.1 -> v1.2.0), jaeger (1.3.0 -> v1.22.0), cert-manager (v0.16.1 -> v1.2.0), traefik (v2.3.2 -> v2.4.8) and containerd (v.2.7 -> v1.4.4). Welcome to the first (and possibly last) installment of Upgrade Sunday – a series of posts where I share some notes on difficulties/process...| vadosware.io
)
+ / / tl;dr - I did another round of drive testing (originally I only tested OpenEBS and hostPath), this time with some rented Hetzner machines and Ansible-powered automation. The GitLab repository isn’t ready for mass consumption yet but I’ll update here (and this tl;dr) when it is, along with the results. UPDATE (04/09/2020) The GitLab repository is up! You can skip this entire article and just go there. NOTE: This a multi-part blog-post!| vadosware.io
)
tl;dr - If you clicked because of the title/description, you got jebaited (consider not falling for bait? I don’t know I fall for it too), but it’s not far off – this is a rant against using automation on F/OSS repositories to automatically close issues I’ll keep this short (originally I considered doing a tongue-in-cheek woke culture roleplay thing) because I just made a breakthrough in some other yak shaving (the Kubernetes storage provider tests round 2 series of posts).| vadosware.io
)
+ / / tl;dr - In order to test storage performance I set up a completely automated test bed for all the storage plugins, this article chronicles the installations of some of the plugins. It’s particularly long because I made lots of mistakes. Mostly useless sections are prefaced with a notice on why you can skip them, skim the ToC and click on anything you like. UPDATE (04/09/2020) The GitLab repository is up!| vadosware.io
)
+ / / tl;dr - I install even more providers, this time OpenEBS cStor, OpenEBS Jiva, OpenEBS LocalPV hostPath, and OpenEBS LocalPV ZFS, LINSTOR via kvaps/kube-linstor. I skipped OpenEBS LocalPV because it threw a panic linked to a supposed kernel issue complaining about old kernels. Ain’t nobody got time for that. The GitLab Repo is finally public though, so you could skip this entire article and go there. UPDATE (04/10/2021) I had some more issues with linstor than I thought -- I've updated...| vadosware.io
)
+ / / tl;dr - I explain the YAML and Makefile scripts that power the fio and pgbench (oltpbench) tests I’m going to run. UPDATE (04/10/2021) Turns out I was mistaken -- OpenEBS Mayastor doesn't support single-node disk-level failure domains. It's very well described on their website in the FAQ, but I somehow missed and/or forgot that, so the tests for Mayastor will only represent JBOD setup (no replication). On a different but related note, cStor supports cross disk replication (mirroring o...| vadosware.io
)
+ / / tl;dr - Finally, the results of the benchmarking. You can find the code on GitLab. There are some issues with the benchmarks but there was enough decent data to make a decision for me at least. As far as which storage plugins I’m going to run, I’m actually going to run both OpenEBS Mayastor and Ceph via Rook on LVM. I look forward to emails from users/corporations/devrel letting me know how I misused their products if I did – please file an issue on GitLab!| vadosware.io
)
+ tl;dr - Steps on how to set up SES with Pulumi (their brand new logo is the one above), an infrastructure-as-code (written in code) solution. Skip to the end for all the code, all at once UPDATE (08/16/2021) SES no longer requires separate domain identitify verification -- this is now integrated with the DKIM records you build for SES. You do not need the aws.ses.DomainIdentityVerification Pulumi object at all anymore -- the post has been updated to reflect that.| vadosware.io
)
+ tl;dr - In this post I go through a bunch of improvments and extras to the code from part 3, 2 years after this initial series was started. The improvements range from development aids to post-deployment “observability” (metrics, logging, tracing) and how I’ve handled them in the past. It’s pretty ridiculous to finish a blog series 2 years later, so I also released the code for a Job Board API I wrote (also the frontend that manages it) for those who want to see how a CRUD-y product...| vadosware.io
)
tl;dr - A few more Paxos papers have been produced since the original post, I do the usual writeup. Main benefit/focuses this time look to be centered around increasing throughput by taking load off of the leader node(s) (and assigning the extra work to existing or new nodes). Multi-part blog post alert This is (now) a multi-part blog-post! Part 1 - Paxosmon: Gotta consensus them all Part 2 - Paxosmon 2: The Journey Continues (this post) What’s new?| vadosware.io
)
tl;dr - Code for converting a primary index (for example a pre-created one) to a covering index so you can do Index-Only scans and get your two (or more) columns back faster than ever Context Recently while abusing Postgres to do things it’s not supposed to do (I may go into this particular hack at a later date), I ended up trying to convert an existing primary key index to a covering index to make sure a certain column was always easily accessible since most queries only returned this value.| vadosware.io
)
+ tl;dr - Automate your deployments without adding a new reconciliation loop (i.e. Flux or ArgoCD), make a limited-permission ServiceAccount, drop your credentials in a GitLab protected variable and set up some CI steps to build your containers (with CI-powered Docker in Docker) and update your deployments. Context DevOps has been growing increasingly important and complicated these days. There’s a wave of new approaches, enthusiasm, companies, and organizations trying to make things to ach...| vadosware.io
)
tl;dr - I built mrman/landing-gear on GitLab, a repository full of lit (formerly lit-element) components that are packaged versions of the awesome components on wickedblocks. Recently I came across some amazing F/OSS work done by WickedTemplates – WickedBlocks on Hacker News. WickedBlocks is a set of pre-made Tailwind CSS-powered components (“blocks”) that were open sourced by WickedTemplates. The blocks are really awesome, but one of the first things I thought was how much more awesome...| vadosware.io
)
tl;dr - Manual SSH, then automated SSH (ex. Ansible) and friends (Salt/Puppet/Chef), Cloud-init (AKA cram-it-in-userData), pre-built VMs (ex. Packer), infrastructure-as-code (CloudFormation, Terraform, Pulumi, etc) and finally containers and container orchestrators (ex. Kubernetes, Nomad) The trends in application deployment have been pretty identifiable over the years and I rarely see it discussed so I figured I’d take a stab. I gave a similar but different presentation in the past on the ...| vadosware.io
)
tl;dr - There are at least two ways to wait for Kubernetes resources you probably care about: kubectl wait for Pods, initContainers for everything else One somewhat rarely talked about issue in Kubernetes land is how exactly people wait for stuff to happen. “Stateful workloads” can get hand-waved to using StatefulSets, and most intricate large deployable things (databases, etc) have Operators that you can use. Sometimes you just want to make sure a Service is up before you start a pod tha...| vadosware.io
)
+ + tl;dr - You’re going to want to use a Hetzner-external resolver (1.1.1.1/8.8.8.8/9.9.9.9) if you’re going to attempt to resolve your own server’s address(es) from your own server. This is a variation of an issue that has already been discussed some on reddit, but it’s not about Hetzner DNS being down but more about recursive queries not working (and finding this via cert-manager running on a Kubernetes cluser).| vadosware.io
)
+ tl;dr - If you didn’t know how to set up a SystemD timer and service to backup your k0s cluster, you no longer have an excuse, scroll down for the code Recently while going through my workloads and making sure everything’s backed up to external storage (I’m using BackBlaze B2) and it’s been great so far). I came across the problem of how I should back up the cluster itself.| vadosware.io
)
tl;dr - Haskell’s worth checking out if you haven’t yet, you’ll find some features you may want to bring back to your beloved $LANGUAGE. Rah-Rah Haskell posts in 2021? You betcha. Note that all examples are written in simple Haskell where possible. No eta reductions for example, the code should be very easy to understand. Most of the code in here will sail past the compiler (you could use it for your own projects!| vadosware.io
)
+ + UPDATE (10/28/2021) I've gotten some great feedback from a post in r/kubernetes and another post in r/zfs. I've added a section with some thoughts tl;dr - I recently switched my baremetal cluster storage setup to OpenEBS ZFS LocalPV + Longhorn. Some issues with Longhorn not running on PVCs aside, the setup is flexible perf wise (ZFS LocalPV) and I at least have a low-complexity option for distributed/HA storage (Longhorn).| vadosware.io
)
tl;dr - I used a self-hosted Baserow instance as the backend for a landing page announcing my most ambitious endeavor yet (a cloud provider called NimbusWS). UPDATE (11/25/2021) Baserow 1.7 has just been released!. Lots of big features in this new release, like Lookups (which look like a way to do JOINs) -- and someting I pontificated in this very post, webhooks! Building landing pages is fun, but I always hesitate a little bit when it comes time to store the data that they collect (emails, p...| vadosware.io
)
tl;dr - A drive attached to a Hetzner dedicated server failed, my drive has both mdraid and ZFS set up so I restored thanks to Hetzner documentation and Joyent’s nice high level docs on ZFS drive replacement combined with the OpenBSD Handbook’s section on ZFS and OpenZFS (ZFS On Linux) Docs. Recently I’ve undergone the pretty traumatic expansion of my Kubernetes control plane from single controller (which was also acting as a node in the cluster and running workloads) to a 3 Hetzner Clo...| vadosware.io
)
- tl;dr - My cluster-wide prometheus scraping config was picking up port 25 as a good place to send GET /metrics requests, and broke my surprisingly fragile email setup (Haraka -> Maddy), simply by inundating it with invalid HTTP requests to a port meant for SMTP. My mailer setup makes use of two pretty good F/OSS tools that I rarely have problems with: Haraka Maddy They work great, and I don’t spend much time thinking about them – I often set up a new maddy instance with separate backups...| vadosware.io
)
+ tl;dr - Ceph (Bluestore) (via Rook) on top of ZFS (ZFS on Linux) (via OpenEBS ZFS LocalPV) on top of Kubernetes. It’s as wasteful as it sounds – 200TPS on pgbench compared to ~1700TPS with lightly tuned ZFS and stock Postgres. The setup is at least usable and can get up to 1000TPS (2 replica ceph block pool) with synchronous_commit=off and some other less risky optimizations. Check out the numbers at the end in context and since this post is very Kubernetes heavy, if you don’t run Kub...| vadosware.io
)
/ tl;dr - Line by line explanation of my ansible-powered ZFS install script for use on Hetzner’s dedicated hardware (Ubuntu 20.04 - “Focal”) – it’s not perfect/minimal, but it works for me. A while back I started using ZFS on all my bare metal dedicated hardware hosted at Hetzner to wrangle the attached HDDs and SSDs. There are lots of choices in the space (standard LVM, mdraid btrfs, etc), but I chose ZFS for it’s feaureset and ergonomics.| vadosware.io
)
+ UPDATE (2021/12/21) After lots of feedback on Reddit (thanks /u/BucketOfSpinningRust!) and doing some more experimenting and digging, I've updated this post with more information -- new/updated sections are marked "Update". tl;dr - Tips for running Postgres on (Open)ZFS-on-Linux (ZoL), which I’m nicknaming “PoZoL”, check out the TOC (or skip to the resource list at the bottom) and browse to whatever looks interesting. This post is a shameless ripoff/reformulation of everything I’ve ...| vadosware.io