Podless Kubernetes | howardjohn's blog
Sidecarless? Why not podless?| blog.howardjohn.info
In the past few years, CPUs have gotten really fast. Shockingly fast! Yet most people are stuck on previous generation mobile chips (whether by choice, or by their companies choice), at a huge detriment to their productivity. Meanwhile, AI coding subscriptions like Cursor are all the rage these days. I'll skip the debate on exactly how useful these tools are, and focus on the pricing. Cursor is $480/year for the team plan (the cheapest corporate plan), and other providers are around the same,...| howardjohn's blog
Posts - howardjohn's blog| howardjohn's blog
AI workloads introduce new requirements on networking infrastructure, but the same core requirements that service meshes solve not only remain, but are exacerbated. Your AI workloads still need a service mesh - they just need a better one| howardjohn's blog
Gloo Mesh’s ambient multi-cluster mode sets a new benchmark for scalability.| blog.howardjohn.info
Go 1.24 introduces new support for "Tools", which allows easy consumption of tools (which are written in Go) as a dependency for a project. This could be anything from golangci-lint to protoc-gen-go. In this post, I will cover usage and limitations. Basic usage Adding a tool to a project is nearly the same as a standard runtime dependency, with the additional -tool flag: $ goimports # I don't have goimports yet! zsh: command not found: goimports $ go get -tool golang.org/x/tools/cmd/goimports...| blog.howardjohn.info
Is it a per-node proxy? Is it a sidecar? The reality is somewhere in between| howardjohn's blog
Reimagine service mesh with Istio’s ambient mode—lightweight, efficient, and scalable. Gloo Mesh now extends this innovation to multi-cluster environments, delivering unmatched reliability, simplicity, and scale.| blog.howardjohn.info
A deep dive into the termination behavior of Istio ambient mode and how it impacts your applications.| Posts on howardjohn's blog
How Istio ambient mode greatly simplifies and improves egress gateway use cases.| howardjohn's blog
How Gloo mesh brings HTTP observability to your mesh, without compromises.| blog.howardjohn.info
How Istio ambient mode brings an enterprise-ready service mesh to Amazon ECS.| howardjohn's blog
External Blog Post.| howardjohn's blog
How to achieve an architecture similar to "Waypoint Proxies" without ambient mesh, or even Istio.| blog.howardjohn.info
How ambient achieves massive scale without toil.| howardjohn's blog
When we first started designing what eventually became Istio ambient mode, there were many directions we explored, both in terms of implementation, and what our goals were. What resonated most, though, was that we wanted to provide an incredibly easy onboarding story for a subset of functionality. This subset, ultimately, was getting Mutual TLS deployed for all service-to-service communication within a cluster. I talk a bit more about this here. Since then, I think we have delivered on this p...| howardjohn's blog
Over 2 years ago, I started working on some ideas to build better Kubernetes controllers. In this post, I wanted to give a bit of a retrospective on how things have gone since then. Over the years working on Istio and other projects, I observed a number of major issues with controllers: Most code was about error-prone event handling and state reconciliation, rather than business logic. Most tests, in turn, were about the same. This, in turn, made the code extremely complex, brittle, and often...| blog.howardjohn.info
Tools to create reproducible development environments are basically everywhere these days, from Development Containers to Nix wrappers to questionable Docker hacks. However, all of these (that I have found) have a common flaw that bothers me: they all require eagerly fetching the entire environment to get anything done. This kills the premise of these environments providing any easy on-ramp for users when the first step is to download GBs of binaries. Across projects I work on, we have probab...| blog.howardjohn.info
JetBrains IDEs (IntelliJ, GoLand, etc) have a nifty feature called Language Injection that lets you get full language features when a language is embedded within another. For example, a SQL query within a string within a Go file. A few of these come out of the box, but they are pretty limited -- I only had some XML ones prior to enabling the Databases plugin which added a few SQL ones. Fortunately, there is the ability to add custom ones. Unfortunately, this is expressed in a proprietary lang...| blog.howardjohn.info
An overview of options to deploy mTLS on Kubernetes| blog.howardjohn.info
Core problems with the API make it a challenging to use in a secure, scalable manner.| blog.howardjohn.info
Zero trust architectures should not treat nodes as highly privileged components.| blog.howardjohn.info
Istio's installation has a long, winding, complex history, leading to an interesting current state. In this post, I hope to explain some of the historical context of how we arrived to the current state, and where I think the project is going. This is all my personal perspective and memory of things that happened years ago, so there is likely some divergence from reality. The Past When I first started working on Istio in 2019, Istio 1.0 had just been released. The ecosystem was a pretty differ...| blog.howardjohn.info
go run bash-script@latest| blog.howardjohn.info
How we built a best-of-both-worlds experience with Istio ambient mode.| blog.howardjohn.info
Installing Istio... hard could it be? A simple istioctl install is all you need... right?| blog.howardjohn.info
Does Istio ambient introduce a SPOF? No.| blog.howardjohn.info
How I iterate on Grafana dashboards| blog.howardjohn.info
How Istio tests its networking proxy without Kubernetes, Docker, or root.| blog.howardjohn.info
tl;dr: it just works| blog.howardjohn.info
Like most other Kubernetes controllers in, Istio is written in Go and relies on the client-go library. While this provides an excellent low-level building block, usage in higher level code in Istio led to a variety of issues that led us to develop our own higher level, opinionated client for Istio. This post covers the issues we faced and how we incrementally solved them. Background knowledge At a high level, client-go provides a few layers for interactions with the API server:| blog.howardjohn.info
talks| blog.howardjohn.info
Rage bait for YAML templating haters| blog.howardjohn.info
Which features I recommend using, or not using, in Istio| blog.howardjohn.info
In Analyzing Go Build Times, I went over how to analyze and understand Go build times, and what factors impact build times. A close cousin to build times is build sizes. Large binaries can lead to a variety of issues such as: Generally, slower build times Increased costs of storage Increased costs and time to distribute Increased memory usage at runtime (more on this in another article, hopefully) So its generally nice to keep them small.| blog.howardjohn.info
Exploring an extreme service mesh architecture to maximize extensibility.| blog.howardjohn.info
The OSI model attempts to build a model for network communications, where increasingly high level layers are built upon lower layers. This is only slightly useful in practice, as the real world is not so simple. In service mesh, generally discussion is reduced to L4 and L7, or TCP and HTTP. This oversimplifies the problem, leading to some confusion. Thinking in terms of termination Simply saying "HTTP" is not really clear about what is going on.| blog.howardjohn.info
When looking at service mesh (or even general networking) architectures, the basic idea is to send network traffic through some component, which handles various functionality. This could be authentication, authorization, encryption, observability, reliability, networking, etc. There are a few different classes of components that can do this, though: Different types of proxy deployments Native application enhancement. The application itself is compiled in with functionality. This could be some...| blog.howardjohn.info
In the past, Istio has suffered from performance issues from OpenCensus, which was used for metrics reporting. At extremes, we saw up to 20% of CPU spent just on incrementing various metrics. This was mitigated to some extent by batching metrics updates, optimizing OpenCensus itself, and caching parts of our OpenCensus usage. At best, we got down to roughly 600ns and 3 allocations per metric update. As OpenCensus is now deprecated, I have been looking into migration to OpenTelemetry - and hop...| blog.howardjohn.info
How versioning works in Kubernetes, especially with CustomResourceDefinitions, is a common source of confusion. The documentation is pretty comprehensive but a bit complicated. This post aims to give a simple description of how versioning works and dispel some misunderstandings. There is only one primary version Consider a CRD with versions alpha and beta. A user can create and view either resource version. Intuitively, they must be distinct things -- they are not.| blog.howardjohn.info
It is a very common misconception that egress policies in Istio can be used for security purposes. This is not true. Despite repeatedly explaining this (and documenting it), I still often see people that do not believe it, and that they can just add one more check to lock things down. In this post, I will show a variety of ways to bypass any possible check, and prove that these policies cannot be used as secure policies.| blog.howardjohn.info
This post follows the journey Istio has taken over the years to optimize our docker image builds. While there is some useful tips to take out of this, this is not intended to be a guide on how a project should build images - the steps taken here go far past the needs of a typical project, optimizing exclusively for speed (and fun) regardless of the complexity to maintain. For background, over the years Istio has consisted of ~10-20 docker images (many are for tests only) made up of ~10-15 Go ...| blog.howardjohn.info
In-kernel networking solutions, such as WireGuard, are not always faster than user space.| blog.howardjohn.info
Exploring an (unfortunately, hypothetical) CI/CD system for end to end tests on Kubernetes.| blog.howardjohn.info
The common messaging around Istio Ambient Mesh is that is a "node proxy." For example, from The New Stack ... architecture that moves the proxy functionality from the pod-level to the node-level. While this is technically accurate, it is misleading and really missing the point and benefits of Ambient. A brief history of service mesh architectures This skips quite a bit of information, but is close enough. One of the earlier service meshes on the market was Linkerd 1 - not to be confused with ...| blog.howardjohn.info
Fully utilizing buildkit's potentional| blog.howardjohn.info
How and why to easily these fields| blog.howardjohn.info
As an open source maintainer, I am reviewing roughly 25 ideas per day - whether they are feature requests, design proposals, or pull requests. Inevitably, this leads to saying "No" quite a bit as well. Usually, this is in a softer for like "No, not right now", "No, not in its current form", or "No, unless someone else approves", but the outcome is the same: the change is not accepted, and the emotional impact on the reviewer and contributor is similar.| blog.howardjohn.info
Most people using Kubernetes extensive have already defined alias k=kubectl and are using tools like kubectx. As someone really lazy though, I have found/developed a few less common tools to help work with Kubernetes efficiently. kubectl apply from clipboard This relies on zsh, and uses the zle to define a custom command. function zle_apply { LBUFFER=" cat <| blog.howardjohn.info
A little bit of OpenTelemetry, a lot of shell hackery| blog.howardjohn.info
Across the ecosystem, a variety of software support policies can be found, where "support" can mean, bug fixes, security patches, and sometimes technical support. However, they can be roughly categorized into two types: Rolling release: only latest release supported. At an extreme, this is the HEAD git commit. LTS (Long Term Support): Support latest N versions. Sometimes specific versions have extended support. "Long" is relative: Kubernetes supports a version for around a year, while RHEL do...| blog.howardjohn.info
In many languages, on of the things I find myself doing is maping over a list to extract some field. For example, coverting a []Person to []Name. Most languages these days have ways to do this pretty easily: Kotlin: people.map { it.Name } JavaScript: people.map(p => p.Name) Rust: people.map(|p| p.Name) Scala: people.map(_.Name) With generics, Go finally can do this in a type safe manner: Map(people, func(t Person) string { return t.| blog.howardjohn.info
Kubernetes Operators are powerful tools when used right, pushing complex operations from human operators to code. This sounds great - and is in some cases - but often the tradeoffs operators expose are not taken into account, both by operator developers and users. This article goes over my take on when operators are useful and not, and what makes a good operator. A basic installation operator A common feature amongst almost all operators is the ability to deploy Kubernetes resources. For exam...| blog.howardjohn.info
When deploying a proxy outside of Kubernetes, users typically reach for solutions like nginx or Traefik; it's pretty rare to see Kubernetes-native load balancers, like Istio, used outside of the Kubernetes context. While it is not documented anywhere on istio.io, it is completely possible to set up an Istio Gateway outside of Kubernetes - not just running out of the cluster, but not connected to any Kubernetes api-server whatsoever. In this post, we will set up Istiod, and Istio ingress gatew...| blog.howardjohn.info
The following graph shows PR approvals for istio/istio since 2020. DevA 100 <-> DevB 50 indicates DevB approved 100 PRs from DevA, and DevA approved 50 PRs from DevB. The data was collected from our policy bot data and graphed using flourish: gcloud spanner databases execute-sql main --instance istio-policy-bot --project istio-testing --format=json --sql='select prr.Author as Approver, pr.Author as Author, count(*) from PullRequestReviews as prr join PullRequests as pr on pr.PullRequestNumber...| blog.howardjohn.info
Go is often praised for its fast build times. While they are pretty quick, they are slow enough that I spend a lot of time waiting for them, enough that it prompted me to go down the rabbit hole of thoroughly analyzing them. This post covers all aspects of what makes Go builds fast or slow. Throughout this blog, we will use Istio as an example of real-world codebase. For reference on its size:| blog.howardjohn.info
