Jack Lindamood's blog and other random thoughts| Jack's home on the web
Kube dns -> link-local -> tcpdump -> sudo| Jack's home on the web
Setup Link to heading Most cloud providers allow you to set commands to run when a host comes online. The general format for this is called cloud-init or user-data. There’s full documentation for cloud-init here. In my particular case, I had extra files I wanted to write to the host using pulumi. To make it all fit, I was gzip compressing the files then base64 encoding the result.| Jack's home on the web
I tried changing the root-dir for kubelet and other things started to break.| Jack's home on the web
Simple way to debug a node on kubernetes.| Jack's home on the web
Image from UnSplash Preface Link to heading This post describes how I’ve been conducting sprint planning on my current team. It deviates from standard agile in a few ways. I document how and why towards the bottom. I’ve found this format works well for us, but there is no process that works for everyone. I script out the sprint planning process to keep it focused and make sure we don’t forget to cover anything important.| Jack's home on the web
Context Link to heading We are using AWS Auora to manager our databases and I want to setup automatic rotation of database passwords. I start by storing them in AWS Secrets Manager. AWS actually releases publicly usable rotations for PostgreSQL, but I want to add the extra fields connstring and host_ro so clients can directly pull both the read only Auora DNS and a full PostgreSQL connection string. Idea Link to heading I decided to fork AWS’s example code and deploy my own Lambda.| Jack's home on the web
An unambiguous best practice for systems is to codify your infrastructure in a repeatable text format, rather than clicks in a UI. The common name for this is Infrastructure as Code. By far, the two most popular ways to do this, especially in AWS, are Terraform and CloudFormation. I used Terraform for about 3 years at a startup before working for Twitch(AKA Amazon Jr) where I used Terraform very heavily before the company pushed hard to switch to all things Amazon, including CloudFormation.| Jack's home on the web
Table driven tests are a popular model for testing in Go and a clear improvement to traditional testing of one function per case. However, there’s an alternative style that can often be clearer than table driven tests that relies on function closures. I’ll reference the example code on Dave Cheney’s post Prefer table driven tests for my examples. The original way Link to heading The code we want to test is a function that splits strings.| Jack's home on the web
Histograms are an important part of metrics and monitoring. Unfortunately, it is computationally expensive to accurately calculate a histogram or quantile value. A workaround is to use an approximation and a great one to use is t-digest. Unfortunately again, there’s no such thing as an accurate approximation. There are various t-digest implementations in Go, but they do things slightly differently and there’s no way to track how accurate their approximations are over time and no great way...| Jack's home on the web
A recent vulnerability in the go-redis library allowed correctly configured clients to return the results of other redis commands in the event of redis failures. While all code is subject to bugs, this one could have been prevented if proper care was taken when creating and using interfaces. Bug behavior Link to heading Most clients reuse TCP connections between RPC calls to eliminate the delay of TCP connection overhead. TCP connections have both an input and output stream.| Jack's home on the web
Problem Link to heading You have a constructor function that creates an object with a background goroutine. You want the object to garbage collect itself even if people don’t explicitly close it. This isn’t possible since since the background goroutine is always running and pointing to your created object. Solution Link to heading Wrap the returned object and use a finalizer on the returned object to shut down the background goroutine.| Jack's home on the web
Problem Link to heading You’ve written an object or class. It’s somewhat complex and you want to iterate ways to better abstract your code but are unsure how to start. There exists documentation about what a good abstraction looks like, but no concrete iterative steps. Being deep inside your own code can make you blind to obvious alternatives. Solution Link to heading Create a bipartite graph of instance methods and member variables.| Jack's home on the web
Problem Example Link to heading I commonly see bugs appending to slices in Go in a way that is not thread safe. A simple example is the unit tests below. This test has two goroutines append to the same slice. If you run this test with the -race flag, it works just fine. package main import ( "sync" "testing" ) func TestAppend(t *testing.T) { x := []string{"start"} wg := sync.| Jack's home on the web
I had an integration test that was very complex (most are, sadly) and took too long to start. I was able to diagnose why using new features of Go’s 1.10 trace tool. This post describes my workflow and what led me there. The example code I’m using for this post is at https://github.com/cep21/trace-110. My integration test was a bit more complex, but generally had startup that was slow due to a single goroutine out of many that did lots of different things.| Jack's home on the web
A short checklist of what I wish for in a good Go library, in no particular order. This is a companion to the effective go list, Go code review comments list, and Go proverbs list. In general, when given two reasonable ways to do something, defer to the option that does not violate these rules. Bend these rules only with a strong explanation why. Dependencies Link to heading Tagged library versions Link to heading Use git tags to version your library.| Jack's home on the web
What I wanted to do Link to heading Today’s pkg/errors Link to heading Package errors is a popular go error wrapping package. The StackTrace type represents the stack trace with an []Frame. Errors wrapped by package errors implicitly implement the stackTracer interface, which allows users to extract stack information. type StackTrace []Frame type Frame uintptr type stackTracer interface { StackTrace() errors.StackTrace } Simultaneously to this, there exist libraries that would like to extra...| Jack's home on the web
Problem Link to heading The go language spec has no way to simultaneously add or augment behavior to an interface while maintaining commonly accepted best programming practices around minimal interface design. While not unique to Go itself, a combination of Go’s feature set and evolved best practices have caused Go library authors to suffer disproportionately to other language developers. Abstract example Link to heading Given an interface I of a concrete type S, there is no way to add beha...| Jack's home on the web
I get woken up to an alert about read capacity on one of my DynamoDB tables. Here is the error being logged. ProvisionedThroughputExceededException: The level of configured provisioned throughput for the table was exceeded. Consider increasing your provisioning level with the UpdateTable API. When I load my CloudWatch graphs, and it appears I’m well below capacity. Cloudwatch graph of DynamoDB metrics. These graphs show consumed capacity (blue on the left) well below provisioned capacity (r...| Jack's home on the web
Go’s pprof package is frequently used for heap and CPU profiles, but a little used feature of the package is being able to create custom profiles. Custom profiles are useful for tracking down leaked resources and troublesome stack traces that are abusing a library’s API. Profile Link to heading From the godoc for pprof itself: A Profile is a collection of stack traces showing the call sequences that led to instances of a particular event, such as allocation.| Jack's home on the web
Not lacking nuance Link to heading I’ve mentioned a general guideline of accept interfaces, return structs in a previous post and multiple times on code reviews to coworkers, but often get the question “Why”. Especially since this isn’t a hard rule. The crux of the idea, and understanding when to bend it, is in the balance of avoiding preemptive abstractions while maintaining flexibility. The abstract gopher Preemptive abstractions make systems complex Link to heading All problems in ...| Jack's home on the web
I recently ran into a -race error message that stumped me for a while, so I’ll retell it here in phases to see if you can catch it. The code Link to heading The intent of the code was a service abstraction that ran a goroutine in the background to send runtime stats to Stastd. A small example of the code follows: type Stats struct { prev runtime.MemStats } func (p *Stats) stats() { ms := runtime.| Jack's home on the web
An interesting property of Go’s built-in data structures is that read operations work even when they are nil! Not only that, but read operations on nil built-in structures behave just as if they are non-nil but empty. This is different from other languages and allows some interesting properties of structures in Go. Here is a full playground link. Why nil matters for built-in structures Link to heading Go does not have a way to force constructors for structs.| Jack's home on the web
Package httptrace is a new addition in Go 1.7 that allows users to trace the steps of a HTTP request. This package also models an interesting design pattern around adding optional debugging to a library. However, it’s not the only way to accomplish this goal. I’ll briefly overview the package and provide alternative implementations for httptrace. httptrace Link to heading The basic idea of httptrace is that a laundry list of callbacks are optionally attached to a Context object that is ad...| Jack's home on the web
This post will talk about a new library in Go 1.7, the context library, and when or how to correctly use it. Required reading to start is the introductory post that talks a bit about the library and generally how it is used. You can read documentation for the context library on tip.golang.org. How to integrate Context into your API Link to heading The most important thing to remember when integrating Context into your API is that it is intended to be request scoped.| Jack's home on the web
This post describes what I feel are best practices when creating a client library for a service. The initial setup is that you’re writing a library with an API to talk to a RESTful service over HTTPS, and all the library needs to do is return JSON unmarshalled objects. I’ll use some examples from a client I wrote to talk to Smite’s API. Directly use the struct Link to heading While constructor functions are sometimes needed for Go libraries, it’s strongly preferred to support users di...| Jack's home on the web
This post will describe common approaches to organizing Go code into packages and what I feel is the ideal directory structure for Go repositories. Before starting with this post, it may be useful to also read https://blog.golang.org/package-names first for context. A setup with package proliferation Link to heading Let’s take an example application called gatekeeper. It may have a public client that other programs use to connect to it and may have parts around authentication and payment, u...| Jack's home on the web
In this post I will describe the Preemptive Interface pattern often used in code and why I think it is generally an incorrect pattern to follow in Go. What is a Preemptive Interface Link to heading Interfaces are a way to describe behavior and exist in most typed languages. Preemptive interfaces are when a developer codes to an interface before an actual need arrises. An example may look like the following.| Jack's home on the web
I’m a software engineer and gamer. Here I post mostly technical stuff, but also some gaming related things. Find my posts on the blog page. Games Link to heading I play a lot of strategy games. Here are some of my favorites: Factorio Oxygen Not Included Slay the Spire Dyson Sphere Program Work history Link to heading Facebook Link to heading At Facebook I co-owned and wrote most parts of their media serving infrastructure including the networking and storage stack required to store and serv...| Jack's home on the web
We'd love to hear from you| cep.dev
Assortment of technology startup infrastructure recommendations| cep.dev