I am Max, a strange loop interested in networking, distributed systems and type theory. Welcome to my little island on this astounding thing called The Internet. You might enjoy browsing through my list of resources I consider worth sharing, get some inspiration from the numerous summaries of past reading group sessions that I organize or simply scroll through my resume to see what I am up to lately. You can best reach me via e-mail: mail@max-inden.de.| Max Inden
General Books Gregg, B. (2013). Systems performance: enterprise and the cloud. Pearson Education. Beyer, B., Jones, C., Petoff, J., & Murphy, N. R. (2016). Site Reliability Engineering: How Google Runs Production Systems. " O’Reilly Media, Inc.". Ousterhout, John K. A philosophy of software design. Vol. 98. Palo Alto, CA, USA: Yaknyam Press, 2018. Bakhvalov, Denis. Performance Analysis and Tuning of Modern CPUs. D. Bakhvalov, 2020. Tanenbaum, A. S., & Van Steen, M. (2017). Distributed syste...| Max Inden
Experience Software Engineer at Mozilla June 2024 - now Rewrite Firefox’s QUIC UDP IO path. Move to Rust via quinn-udp. See bugzilla#1901295. Switch to multi-packet UDP IO using OS mechanisms like recvmmsg, generic receive offloading, across Linux, Windows, and Android. Enables support for ancillary IP metadata, such as Explicit Congestion Notification (ECN), improving congestion signals. See bugzilla#1902065. Refactor UDP QUIC receive path to not allocate up to encryption layer. Introduce ...| max-inden.de
I am Max, a strange loop …| max-inden.de
I presented my recent work on Firefox’s HTTP3/QUIC stack in the Network Devroom at FOSDEM 2025. QUIC is a new transport protocol on top of UDP, transporting a large portion of the Internet traffic today. UDP I/O performance is crucial for QUIC implementations, where e.g. system call overhead can significantly impact throughput at high network speeds. To improve QUIC throughput, Firefox is switching to a modern UDP IO stack in Rust, using mechanisms like recvmmsg, and GRO across Linux, Windo...| Posts on Max Inden
I don’t see myself making major contributions to (rust-) libp2p in the near future and thus I am stepping down as a maintainer. As announced before, I have left Protocol Labs in December 2023. After a 2 month re-orientation break, I have decided to move on entirely. My first commit was 5y ago, a small bug fix in our address handling. Since then lots happened. A couple of milestones I was involved in:| Posts on Max Inden
How you can connect everything (browsers & non-browsers), everywhere (public or private), all at once (using libp2p). Slides| Posts on Max Inden
I presented at IPFS Camp 2022 on mitigating Denial-of-Service attacks in peer-to-peer networks. I discussed resource management strategies such as enforcing backpressure and provided examples of coding pitfalls to avoid in Rust and Go. You can find the recording and slides of my talk below. Slides DOS Denial-of-service attack Hard in peer-to-peer as identities are cheap Relevant for any scarce resource, e.g. CPU, memory(, file descriptors) Do’s Bound EVERYTHING Once a bound is exceeded: Dro...| Posts on Max Inden
Dennis and I presented Hole punching in the wild, learnings from running libp2p hole punching in production, measured from vantage points across the globe in the network devroom at FOSDEM 2023. At FOSDEM 2022 I presented libp2p’s hole punching mechanism, overcoming NATs and firewalls with no dependencies on central infrastructure. One year has passed since. We rolled it out to live networks. We launched a large measurement campaign with many volunteers deploying vantage points in their home...| Posts on Max Inden
I presented an overview on Peer-to-peer Browser Connectivity options in the network devroom at FOSDEM 2023. Connecting from the browser to a public server with a valid TLS certificate is easy. But what if the server has a self-signed certificate? What if it isn’t public? What if it is another browser? This talk covers the intricacies of browser communication beyond the standard browser-to-server use-case. I will give an overview of the many protocols available and how they can be used in a ...| Posts on Max Inden
We’ll take a closer look at the recently added decentralized hole punching feature, allowing NAT and firewall traversal without the need for any central coordination servers (STUN and TURN). Recording| Posts on Max Inden
We will discuss the current state of the project, eyeball the various language implementations, take a look at the many live networks running on top of libp2p today and finally cover the project roadmap for the years to come. Recording| Posts on Max Inden
We have been working on hole punching support in libp2p for the last year. It finally landed in the Go and Rust libp2p implementation. Read all about it on the IPFS blog: https://blog.ipfs.io/2022-01-20-libp2p-hole-punching/| Posts on Max Inden
Gave a talk on hole punching in libp2p at FOSDEM 2022. Peer-to-peer hole punching without centralized infrastructure How libp2p can traverse NATs and firewalls without coordination through central STUN and TURN servers. libp2p is a universal, cross-platform, multi-language, modular peer-to-peer networking library powering multiple large-scale networks, for example IPFS. In the first part of the talk we’ll present the basic building blocks of the library and explain how they fit together. In...| Posts on Max Inden
Gave a talk on the state of libp2p at FOSDEM 2022. State of libp2p Status quo and future roadmap of the peer-to-peer networking library libp2p. libp2p is a universal, cross-platform, multi-language, modular peer-to-peer networking library powering multiple large-scale networks, for example IPFS, Ethereum 2, Filecoin or Polkadot. We will discuss the current state of the project, eyeball the various language implementations, take a look at the many live networks running on top of libp2p today a...| Posts on Max Inden
On Tuesday 20th of July I gave a talk introducing libp2p, a modular network stack. Recording Slides| Posts on Max Inden
In our previous session we looked into Mixnets, more specifically Loopix. With this session we stayed in the space, talking about a network adopting many concepts from Loopix, namely Nym. As a preparation, we read section 1 through 4 of the Nym whitepaper. Introduction Network-level surveillance The Nym Network Design Mixnet for Network-Level Privacy Diaz, Claudia, Harry Halpin, and Aggelos Kiayias. “The Nym Network.” (2021). https://nymtech.net/nym-whitepaper.pdf| Posts on Max Inden
34th session was on Mixnets as lots of people suggested in the past. As a preparation we read the Loopix paper. Piotrowska, Ania M., et al. “The loopix anonymity system.” 26th {USENIX} Security Symposium ({USENIX} Security 17). 2017. https://www.usenix.org/system/files/conference/usenixsecurity17/sec17-piotrowska.pdf The corresponding talk by Ania Piotrowska is very much worth watching as an additional resource. https://www.youtube.com/watch?v=R-yEqLX_UvI| Posts on Max Inden
In our third BGP session we focused on BGP security, more specifically we discussed how to perform hijack and interception attacks using BGP. The primary paper of the session was: Birge-Lee, Henry, et al. “Sico: Surgical interception attacks by manipulating bgp communities.” Proceedings of the 2019 ACM SIGSAC Conference on Computer and Communications Security. 2019. A good introduction to RPKI can be found here.| Posts on Max Inden
After our first BGP session, introducing the protocol based on the convergence problem of BGP, our second session on BGP covered route propagation and tuning. The primary paper was: Gray, Caitlin, et al. “BGP Beacons, Network Tomography, and Bayesian Computation to Locate Route Flap Damping.” Proceedings of the ACM Internet Measurement Conference. 2020.| Posts on Max Inden
Below is a summary of our efforts to optimize flow control in the Rust Yamux implementation. While not a novel approach, I still find the end result worth sharing thus my forum post. https://discuss.libp2p.io/t/optimizing-yamux-flow-control-sending-window-update-frames-early/843| Posts on Max Inden
We decided to turn our interest to BGP which we will devote 3 sessions to. In today’s session - the first one - we introduced BGP, looked at the convergence problem, as well as the solution suggested in the paper below. Gao, Lixin, and Jennifer Rexford. “Stable Internet routing without global coordination.” IEEE/ACM Transactions on networking 9.6 (2001): 681-692. To play around with BGP as well as general Internet routing:| Posts on Max Inden
I have created a new GPG key replacing my old GPG key. I did not loose access to the old one, nor was it, as far as I am able to tell, ever compromised. As I did with the old key, I will use the new key both for signing my open source work as well as securing my e-mail communication. You can find both the old and the new key on all major key servers. Along with the new key you find a signature of the old key signing the new key.| Posts on Max Inden
What better way to start a new year than with a paper discussing how to change time? In the 30th session we discussed a paper which I think has much up its sleeves - Attacking the Network Time Protocol. First off the paper gives us a good introduction to the inner working of the network time protocol. Next up it examines the broader ecosystem as well as why we need accurate time in the first place. Once we established enough background, the paper dives into how one can attack the protocol, st...| Posts on Max Inden
With the 28th session we jumped into the space of byzantine fault tolerant consensus protocols. We covered fault tolerant consensus with various Paxos variants in the past, but this session was the first one looking into how to solve the byzantine generals problem. Instead of using PBFT [1] as a first paper we went with Hotstuff [2] instead. The reasoning behind this choice was (a) Hotstuff presenting a somewhat easy up-to-date consensus algorithm and (b) that it provides a framework enabling...| Posts on Max Inden
I find it helpful to know the orders of magnitude by which certain computer operations differ. Certainly it is not worth the effort to pay attention to every digit or learn these by heart, especially since they differ (slightly) across systems, but having a basic understanding of what a tiny fraction of time a CPU cycle occupies compared to sending a TCP packet is incredibly helpful whenever reasoning about systems performance.| Posts on Max Inden
We have long been planning to cover the caching mechanisms in CPUs. As a shared knowledge base for the discussions in this session we chose the following two articles by Martin Thompson among other things known for his work on the LMAX Disruptor: CPU Cache Flushing Fallacy including a good overview over the different caches in modern Intel CPUs. Write Combining exemplifying the advanced mechanisms one can find in today’s CPUs and how one can make use of them.| Posts on Max Inden
In the session today we covered Madhavapeddi Shreedhar and George Varghese paper “Efficient fair queuing using deficit round-robin” [1]. While the session was not so much about the relatively simple algorithmic details of deficit-round-robin (still worth checking out) we talked about: Its benefits over basic FIFO queuing and thus its impact for congestion controlled traffic (tcp) compared to not congestion controlled traffic (udp). Its wide deployment still seen today. Its derivatives DRR...| Posts on Max Inden
After a bit of a break due to current pandemic we decided to carry on and continue our meetings as virtual calls. Ignoring the usual initial hiccups and the missing whiteboard the medium worked well for us. Topic and reading of this session was the ACM Queue article BBR: Congestion-Based Congestion Control [1], as well as the Dropbox article Evaluating BBRv2 on the Dropbox Edge Network [2]. We started off with a quick recap of the previous session covering why we need congestion control, how ...| Posts on Max Inden
I recently stumbled upon the idea of an Elimination Back-off Stack promising to be a parallel, linearizable and lock-free stack. In case you are not familiar with it, I would suggest either reading my previous post or the corresponding paper [1] itself. Being quite intrigued by the ideas of the above stack I wrote my own implementation in Rust with a little help from crossbeam.| Posts on Max Inden
Reading The Art of Multiprocessor Programming [1] I came across the Elimination Back-off Stack [2] datastructure introduced in 2004 by Danny Hendler, Nir Shavit, and Lena Yerushalmi. It promises to be a parallel lock-free stack. How can a stack allow parallel operations without going through a single serialization point, e.g. a Mutex or an Atomic? Let’s dive into it. A lock-free stack A lock-free stack, also often referred to as a Treiber stack [3] due to Kent Treiber, operates on top of a ...| Posts on Max Inden
In January 2017 I joined the company CoreOS as a test-engineer helping the monitoring team and the rkt container engine team write reliable software. Eventually I joined the CoreOS’ monitoring team full-time as a software engineer and ultimately was invited to be part of the upstream Prometheus team due to my contributions to the Alertmanager sub-project. Over the next 2 years and 4 month I worked a lot on Alertmanager, e.g. writing parts of its Elm-based UI and introducing API v2, a bit on...| Posts on Max Inden
At the end of the previous session one of us suggested to dive into congestion control algorithms. This has found a greater echo, thus the 23rd session covered congestion control algorithms in general and TCP’s Reno as well as TCP’s Tahoe in particular. This weeks reading was: Chapter 13 “TCP Reno and Congestion Management” from the comprehensive online book “An Introduction to Computer Networks” [1] from the Loyola University Chicago.| Posts on Max Inden
In the 22nd session we took a look at io_uring - a new Kernel interface for asynchronous I/O. Tyler, who is currently implementing an io_uring library in Rust [4] for his database sled [7] guided us through the concepts as well as a bunch of source code. Tyler started off introducing the status quo of I/O interfaces within the Linux Kernel like read, pread and preadv, jumped over to asynchronous I/O like aio and eventually helped us develop a sense of what the perfect asynchronous I/O interfa...| Posts on Max Inden
We started the new year with a session on epidemic / gossip protocols. To decide what to read I compiled the following list of papers that I either enjoyed reading in the past, or that were recommended to me. The Swim (Scalable failure detection and membership protocol) paper won the poll. Das, Abhinandan, Indranil Gupta, and Ashish Motivala. “Swim: Scalable weakly-consistent infection-style process group membership protocol.” Proceedings International Conference on Dependable Systems and...| Posts on Max Inden
Last Tuesday we meet again to discuss different attacks and possible countermeasures for distributed hash tables. More in particular we looked at Kademlia and its security extension S/Kademlia [1], possible eclipse attacks on the Ethereum network [2], a novel approach of hiding its own connection buckets as well as using an existing social graph as a network topology in the Whanau paper[3], security extensions to the Chord DHT [4], as well as a larger study of different security techniques fo...| Posts on Max Inden
I have been organizing a distributed systems paper reading group in Berlin for the last year. We meet every other week discussing a paper in the distributed systems space. This could be anything from Chandy–Lamport’s algorithm for global distributed snapshots [1] to things like conflict free replicated datatypes [2]. The event is open for anyone interested. I only ask people to come prepared. In the last meeting (19th) we covered distributed hash tables. They play a crucial role in e.g. d...| Posts on Max Inden
Within my work at Red Hat and Kubernetes SIG instrumentation I have been working on kube-state-metrics , a Prometheus exporter exposing the state of a Kubernetes cluster to a Prometheus monitoring system. In particular I have focused on performance optimizing metric rendering for both latency as well as resource usage. Below I want to describe our approach of metric driven performance tuning, using Prometheus to monitor kube-state-metrics on top of Kubernetes, which in itself enables Promethe...| Posts on Max Inden
Kube-state-metrics exposes Prometheus metrics of the state of a given Kubernetes cluster. The project uses the standard Prometheus client Golang library, which is not optimized for the very specific use case of kube-state-metrics. This talk covers different optimizations like metric caching and improved text marshaling dividing CPU usage by a factor of 6 and memory and response time by a factor of 3 through introducing an intelligent Prometheus metric cache in the code hot path and optimizing...| Posts on Max Inden
Monitoring plays a crucial role in a microservice architecture. Restricting the management and configuration of the monitoring stack to the operations team results in workflow bottlenecks. Instead one could provide a self-service monitoring platform, enabling each team to easily setup monitoring for their applications and customize it to their needs. This gives each team the ability to deeply introspect their application, benchmark new features and alert on failures on their own.| Posts on Max Inden
What If Component xxx Dies? Introducing Self-Healing Kubernetes Kubernetes promises healing your application on all kinds of failure scenarios, but why not self-heal Kubernetes itself? This talk introduces self-hosted Kubernetes (K8s inside itself) to autonomously recover from failure scenarios with the help of e.g. itself, systemd and checkpointing. We will ask and answer questions like “What happens when xxx dies”. The theory will be followed by a demo on a live cluster showcasing what ...| Posts on Max Inden
Kubernetes is a powerful system to build and operate a modern cloud-native infrastructure. Monitoring with Prometheus ensures that Kubernetes stays healthy. Prometheus is a stateful application, so operating it in a cloud native environment can be a challenging task. The Prometheus Operator makes running highly available Prometheus clusters, and even an entire end to end monitoring pipeline, easily manageable. Max will explain the functionality of the Prometheus Operator and describe a desira...| Posts on Max Inden
Distributing and deploying software inside (Docker-) containers for security, isolation and ease of use is the new big thing. But once you got all your services nicely wrapped - who takes care of all these containers? The open source project Kubernetes, originating from Google, helps you manage containerized applications, as the operating system of your datacenter, treating hundreds of machines as a single resource pool. This talk introduces the core concepts of Kubernetes, its benefits and i...| Posts on Max Inden
Alertmanager deduplicates, groups, and routes alerts from Prometheus to all kinds of paging services. With it comes a dated UI which does not live up to the expectations of the users, nor does it attract new contributors. From this talk, you will learn how we addressed these issues when building the new UI from scratch. We made it friendlier to users by removing unnecessary domain language noise. In addition we added new power features such as filtering and grouping. As a result, it is now mu...| Posts on Max Inden
Finished my very first Triathlon today here in Berlin. I survived the swimming way better than I thougt, biking wasn’t a big problem, but the 10 k at the end were more of a stumbling then a running.| Posts on Max Inden
Sunday five mile hike around Angels Island. Great to get out of the city from time to time.| Posts on Max Inden
My first 16 miles run. Nearly blacked out at the end. Amazing route!| Posts on Max Inden
After two delayed flights, one canceled flight and two lost bags I made it back home to Germany.| Posts on Max Inden
Made it back to San Francisco. Crashed on a good friends couch.| Posts on Max Inden
On my way back to San Francisco I made a quick stop in Las Vegas. A friend I met in Austin showed me around. Visited Downtwon, the strip and the Red Rocks.| Posts on Max Inden
I went to the Texas State Capitol, learned a bunch about Texas history gaining independence, joined a local hiking Meetup and looked at the graffiti park.| Posts on Max Inden
20:20 - Hi Max, wanna join Yoga Class tonight? 20:21 - Sure, never tried though. 20:30 - Class starts My first Yoga class. It was extremly exhausting. Definitely doing it again some time.| Posts on Max Inden
I got an AirBnB across the Colorado River fairly close to downtown. My hosts are great. Looking forward to the next days. Surrounded by great restaurants.| Posts on Max Inden
I am going to Austin Texas for the NodeJS Interactive conference. Am looking forward to some great talks both on NodeJS itself and microservice architectures. Flying over night - saving one night of accomodation.| Posts on Max Inden
Hiking to the Pierce Point. We have seen something that looked like a fox and plenty of Elks. It is a scenic view along the coast.| Posts on Max Inden
With two friends of mine I wen to Berkeley. We started with downtown Berkeley and then made our way up to the university campus. We hiked all the way up to Lawrence Hall of Science from where we had an amazing view over the entire bay.| Posts on Max Inden
A very fun friend of mine from Covo with whom I shared a lot of great discussions on coding and code quality in particular invited me over for Thanksgiving. I felt very welcome. We talked a lot about politics. It is super interesting to see an American point of view on it.| Posts on Max Inden
I booked a tour for Alcatraz. My first time on a boat in the Bay Area. The weather was horrible - setting the perfect mood for visiting a prison.| Posts on Max Inden
A cowoker / great friend of mine from Covo invited me over for a friendsgiving at his place. Like the day before, great food and a great party in a beautiful house in Potrero Hill.| Posts on Max Inden
The coworking space I currently work at, Covo, arranged a friendsgiving for lunch. Everybody contributed to the dinner. We had plenty of food and a great time. My first friendsgiving.| Posts on Max Inden
There was a conference at the Google campus on Sunday. I really enjoyed the talks. I was especially interested in the topics architecture and artificial intelligence.| Posts on Max Inden
I woke up reasonably early on Saturday to go to the Golden Gaten Bridge. The bus ride from Ingleside took me an hour. I started my hike in the Presidio park along the west coast. I found some lonely beaches. I felt nothing like being in a city. Afterwards I walked over the Golden Gate Bridge where two friends of mine, Simon and Marc, picked me up. We drove to Muir Beach, Stinson Beach, Tiboron and Sausalito.| Posts on Max Inden
As we are working from San Francisco for the rest of the month, we were looking for a coworking space. Well, … not just one coworking space, but the coworking space. We started our search on Wednesday with the coworking space, Thursday we looked at NextSpace, Wework and Bespoke In the evening we decided to go with Bespoken until we passed by two more coworking spaces (Covo and Impact Hub) on our way to a restaurant. Friday we visited the last two coworking spaces and finally settled on Covo...| Posts on Max Inden
Been to the MongoDB Launchpad event, a release event for their new version. Sadly it was more of a marketing event than a nerd event. But the location was really nice with a view on the Golden Gate Bridge (Fort Mason 2 Marina Blvd San Francisco, CA).| Posts on Max Inden
I moved to San Francisco to the Coliving Club. I will miss Palo Alto.| Posts on Max Inden
I went out with my roommates again - this time a pub crawl mostly in the city center.| Posts on Max Inden
We, my roommates and I, went to a crazy Halloween Party atPier 70 in San Francisco, an old industrial complex. The costumes of the other guests were amazing. They seemed to be inspired by the Burning Man festival.| Posts on Max Inden
I have been to a JS meetup in San Francisco in the Yelp HQ. It is an impressiv building from 1925 known as the Pacific Bell Building.| Posts on Max Inden
Worked in a rooftop apartment today with a view over entire Palo Alto and parts of the bay area. Sadly I can’t work from there every day.| Posts on Max Inden
My first day at the first Hacker House in Palo Alto. House cleaning is every Tuesday. I arrived on Monday. The people here are amazing. Many startup founders - smart people all over the place. The location, 635 Webster St., Palo Alto, is awesome, right in the center. This makes up for sleeping in a room with six other guys, living in an apartment with 21 guys with two bathrooms.| Posts on Max Inden
Hearst Castle is a huge Landmark. We did a tour with an amazing tour guide, it was like Mr. Hearst showed us around himself.| Posts on Max Inden
We did a very short walk through Los Angeles and I took a longer walk around Santa Monica and (https://en.wikipedia.org/wiki/Venice,_Los_Angeles)[Venice beach]. I liked the last two a lot. In the next two days I am heading back to the San Francisco bay area.| Posts on Max Inden
We started of at theBalboa Park, walked down to Little Italy, visited the USS Midway aircraft carrier museum ship and walked back to the car through down town. This was my first time on an aircraft carrier, these things are huge.| Posts on Max Inden
Signed up for audience seats at the sitcom Last Man Standing. We had to arrive at 3 for the show to start at 6. They hired a comedian just to cheer up the audience - a little strange. It is very interesting to see how small the entire set actually is. Most of the times the actors only need 2 tries for one scene - extremly professional.| Posts on Max Inden
We walked over the campus of the University, ate in the canteen and worked outside at desks with solar powered outlets.| Posts on Max Inden
First night in the Los Angeles area mostly for the mexican restaurant El Mirasol Cocina Mexicana across the street. There are many huge beautiful villas like little oasis in Palm Springs.| Posts on Max Inden
We did a 10 Miles hike in the Kings Canyon National Park up to the Mist Falls. This has been my favourite National Park so far because of the very untouched nature.| Posts on Max Inden
During the day we worked in the fancy Hana Haus, a coworking space by SAP. Afterwards we stayed for a Design Thinking Talk at the same place and finished the evening at a steake house close to the Apple offices.| Posts on Max Inden
We visited the Docker HQ for a meetup.| Posts on Max Inden
We did two hikes, including a cave and three scary tarantulas.| Posts on Max Inden
We went to the San Francisco Fleet Week. My first time to see SF downtown during daylight and the Golden Gate Bridge. Perfect weather in a very crowded town.| Posts on Max Inden
Exploring the beach area with all the surfers, working in a cafe and the hotel.| Posts on Max Inden
First thing to do in Silicon Valley- the computer history museum. It is an amazing museum with all the hardware I have ever dreamed of. Showed me how much I don’t know about computers, especially anything below logic gates.| Posts on Max Inden
Posts - Max Inden| max-inden.de
Motivation Around 20% of Firefox’s HTTP traffic today uses HTTP/3, which runs over QUIC, which in turn runs over UDP. This translates to substantial UDP I/O activity. Firefox uses NSPR for most of its network I/O. When it comes to UDP I/O, NSPR only offers a limited set of dated APIs, most relevant here PR_SendTo and PR_RecvFrom, wrappers around POSIX’s sendto and recvfrom. The N in NSPR stands for Netscape, giving you a hint of its age.| max-inden.de