In kr8s I have a GitHub Actions workflow which runs a script nightly on a cron job. The workflow grabs a list of actively supported Kubernetes versions from endoflife.date and then cross-references them with the available kind container images for running the tests in CI.| jacobtomlinson.dev
I write a lot of markdown. I use it on GitHub when creating issues/PRs, I use it in Obsidian when I take notes, I use it in Hugo when writing blog posts (like this one), I use it in Jupyter Notebooks when working with data and I use it in Sphinx with MyST when writing documentation.| jacobtomlinson.dev
If you want to point tools like kubectl to a config file other than ~/.kube/config you can set the environment variable KUBECONFIG. But did you know that KUBECONFIG behaves sort of like a path, and kubectl will load all the config files it finds?| jacobtomlinson.dev
If you’ve been around open source projects on GitHub you may have encountered a project with a stale bot.| Posts on Jacob Tomlinson
According to folks who enjoy outdoor activities there are three types of fun. I’ve been using this scale for a while to categorize my own enjoyment of things and wanted to share my version.| Posts on Jacob Tomlinson
Choosing a good profile picture will make collaborating with others easier, especially if you haven't met them yet. Here are some tips to help you pick a good one.| jacobtomlinson.dev
In PEP440 Python introduced Version Epochs as a mechanism to allow projects to change versioning scheme.| jacobtomlinson.dev
GitHub Releases is a feature where you can create a page associated with a git tag that contains a description of the changes in that tag along with build artifacts for users to download.| jacobtomlinson.dev
Managing Kubernetes resources with Python has never been easier thanks to the kr8s Kubernetes client for Python.| jacobtomlinson.dev
Databricks is a very popular data analytics platform used by data scientists, engineers, and businesses around the world.| jacobtomlinson.dev
Sometimes you want to write some code using Dask which can then be run against multiple different cluster backends.| jacobtomlinson.dev
Sometimes when writing code in a blog post I want to emphasize a couple of lines in particular.| jacobtomlinson.dev
I love using typer for creating CLI tools in Python. It makes creating complex trees of subcommands really straightforward.| jacobtomlinson.dev
I recently read Loving and hating the Streak by Cassidy Williams. The post was all about committing code on GitHub every single day to maintain a streak.| jacobtomlinson.dev
In Theo’s previous posts on storing high momentum data and its accompanying metadata we get some interesting insights into the future of cloud based data storage.| jacobtomlinson.dev
This article was originally written for the the Met Office workshop run at the Intro to Earth Information event on the 12th of March 2019.| jacobtomlinson.dev
Mozfest workshop facilitators meeting Figuring out the right format for a workshop can be tricky.| jacobtomlinson.dev
Sometimes I find that something goes wrong in a container and some data stored in a persistent volume gets corrupted.| jacobtomlinson.dev
Introduction For a while I’ve been searching for a decent light switch solution for my home automation setup.| jacobtomlinson.dev
I spent some time this week exploring whether it would be possible to run Dask and Distributed on a function as a service platform like AWS Lambda.| jacobtomlinson.dev
Originally published on the Met Office Informatics Lab blog on February 7th, 2018. We are excited to announce that the work we’ve been doing with distributed Dask clusters running on Kubernetes has been absorbed into an awesome new tool called Daskernetes through our work on the Pangeo project.| jacobtomlinson.dev
Originally published on the Met Office Informatics Lab blog on December 19th, 2017. ChatOps - Automation via chat This article is a companion to a workshop on using chat to automate ops workflows.| jacobtomlinson.dev
ZEIT is a great platform for deploying your opsdroid instance. Particularly because it is free for light use, which many opsdroid deployments will be.| jacobtomlinson.dev
Originally published on the Met Office Informatics Lab blog on July 21st, 2017. Introduction This article assumes a basic understanding of Amazon Web Services (AWS), Kubernetes, Docker and Dask.| jacobtomlinson.dev
Originally published on the Met Office Informatics Lab blog on May 8th, 2017. Recently we’ve been thinking a lot about monitoring.| jacobtomlinson.dev
Originally published on the Met Office Informatics Lab blog on April 20th, 2017. We just moved ~80TB of data to S3 (stay tuned to hear what we’re doing with it).| jacobtomlinson.dev
Did you know that the Dask scheduler has a --jupyter flag that will start a Jupyter server running within the Dask Dashboard?| jacobtomlinson.dev
When writing and speaking about linux container technologies I’m trying to be more intentional with the words I use, which means often avoiding the word docker.| jacobtomlinson.dev
Sometimes I want to oversubscribe the GPUs in my Kubernetes cluster. This is especially useful when I’m developing but could also be useful in light workloads where you have ample GPU memory and don’t mind the occasional failure.| jacobtomlinson.dev
This week I’ve been working on some custom Sphinx extensions for a documentation site. Sphinx is a pretty complex tool with a broad ecosystem so documentation tends to be spread across the upstream project, dependencies like docutils and popular extensions like MyST.| jacobtomlinson.dev
Over the last few years, many open-source Python projects that I work on have switched to CalVer.| jacobtomlinson.dev
In July I published a blog post on using Dask on KubeFlow with the Dask Kubernetes Operator.| jacobtomlinson.dev
In the machine learning and MLOps world, GPUs are widely used to speed up model training and inference, but what about the other stages of the workflow like ETL pipelines or hyperparameter optimization?| jacobtomlinson.dev
When using GPUs with Kubernetes it can be important to know which driver and CUDA versions are installed on the nodes.| jacobtomlinson.dev
Kubeflow is a popular Machine Learning and MLOps platform built on Kubernetes for designing and running Machine Learning pipelines for training models and providing inference services.| jacobtomlinson.dev
A big frustration for me when reviewing Pull Requests on GitHub is coming back to a PR you’ve already reviewed to check on recent changes and be greeted with “We went looking everywhere, but couldn’t find those commits”.| jacobtomlinson.dev
When someone opens a Pull Request (PR) on your GitHub project it can be helpful for a bot to comment on the PR.| jacobtomlinson.dev
Today is the deadline for the license changes to Docker Desktop for Mac and Windows. This means that if you are employed at a company with more than 250 employees or your company makes more than $10m you need to start paying a subscription to continue using Docker Desktop.| jacobtomlinson.dev
In a couple of days Docker will begin charging employees of companies with >250 employees to use Docker Desktop.| jacobtomlinson.dev
This week I’ve been playing around with Kubeflow as part of a larger effort to make it simpler to use Dask and RAPIDS in MLOps workflows.| jacobtomlinson.dev
This post has been superseded with this tutorial that no longer requires any code changes. Please read that instead.| jacobtomlinson.dev
Before I was a maintainer of open source software I was a user of open source software, and I sometimes behaved badly.| jacobtomlinson.dev
Having a brand can help give your open source project some legitimacy, and you don’t need to be a designer to see these benefits.| jacobtomlinson.dev
Both Dask and RAPIDS are Python libraries to scale your workflow and empower you to process more data and leverage more compute resources.| jacobtomlinson.dev
This week was the 2021 Dask Summit and one of the workshops that we ran covered many deployment options for Dask Distributed.| jacobtomlinson.dev
With our open source project published on GitHub we probably want to allow folks to contribute changes.| jacobtomlinson.dev
Once your open source Python project has users and a community you will likely want to communicate with them in an official capacity.| jacobtomlinson.dev
Now that our open source Python project exists and users can install it we will want to turn our attention to sustainability, reach and ongoing maintenance.| jacobtomlinson.dev
In part four of this series we discussed documenting our code as we went along by adding docstrings throughout out project.| jacobtomlinson.dev
Prometheus is a popular monitoring tool within the cloud community. It has out-of-the-box integration with popular platforms including Kubernetes, Open Stack, and the major cloud vendors, and integrates with dashboarding tools like Grafana.| jacobtomlinson.dev
In this post we will cover automatically packaging and releasing our project when a new git tag is pushed to GitHub.| jacobtomlinson.dev
Python objects are synchronous by default. When working with asyncio if we create an object the __init__ is a regular function and we cannot do any async work in here.| jacobtomlinson.dev
What is test driven development (TDD)? Test driven development is a style of development where you write your tests before you write your code.| jacobtomlinson.dev
In this post we will cover testing our code. Testing There are many many great resources out there for learning about testing software.| jacobtomlinson.dev
This post will cover documenting our code. Specifically adding documentation within the code itself. Docstrings Right now our code is undocumented, so if the user inspects our function they will only see the interface (the way you call it) but with no other context.| jacobtomlinson.dev
Update: This was a fun experiment and I recommend you check out the post for a fun read on setting up reverse shells.| jacobtomlinson.dev
Hacktoberfest is approaching once again. In previous years I have both participated and contributed to open source, and also tried to leverage the community in the open source projects I maintain by curating and labeling issues.| jacobtomlinson.dev
Originally published on the Dask blog on August 21st, 2020. For the last couple of months we’ve been running community tutorials every three weeks or so.| jacobtomlinson.dev
Originally published on the Dask blog on July 23rd, 2020. Dask enables you to build up a graph of the computation you want to perform and then executes it in parallel for you.| jacobtomlinson.dev
⚠️ This post is no longer valid. Running SSH on Binder has not been possible since late 2020.| jacobtomlinson.dev
In this post, we will cover a few project hygiene things that we may want to put into place to make our lives easier in the future.| jacobtomlinson.dev
When you build a website you want pages to load as quickly as possible for users.| jacobtomlinson.dev
Have you had a great idea for an open-source Python library that you think people will find useful, but you don’t know where to begin in creating and publishing it?| jacobtomlinson.dev
Note: This post is also available in Go flavour. GitHub Actions provide a way to automate your software development workflows on GitHub.| jacobtomlinson.dev
Note: This post is also available in Python flavour. GitHub Actions provide a way to automate your software development workflows on GitHub.| jacobtomlinson.dev
In my day to day work I generally access a variety of Jupyter installations. Sometimes these are short lived installations in conda environments on my laptop, sometimes they are running on a remote server, and sometimes I use a managed service like JupyterHub or Binder.| jacobtomlinson.dev
Originally published on the RAPIDS AI blog on August 13th, 2019. Background In my first week working at NVIDIA, I have been spending some time with my previous colleagues at the Met Office to explore how the two organizations can collaborate.| jacobtomlinson.dev
It has been nearly two years since I published a new blog post on this website.| jacobtomlinson.dev
Originally published on the Met Office Informatics Lab blog on May 8th, 2017. Jupyter notebooks are awesome.| jacobtomlinson.dev
I’ve been working on kr8s for a while now and one of my core goals is to build a Python library for Kubernetes that is the most simple, readable and produces the most maintainable code.| jacobtomlinson.dev
The Dask blog is a bit neglected these days. The website is an aging Jekyll blog and is well past it’s prime.| jacobtomlinson.dev
Since upgrading to a UniFi Dream Machine (UDM) Pro I’ve had a problem with some of my UniFi devices showing as offline.| jacobtomlinson.dev
This post will be updated with notes from the livestream throughout the day. Today I will be streaming some open source code refactoring.| jacobtomlinson.dev
For the last few months I’ve been tinkering with a new Kubernetes client library for Python called kr8s.| jacobtomlinson.dev
When writing tests the balance between avoiding indirection and DRY-ness should be much more weighted towards avoiding indirection than in the code it is testing.| jacobtomlinson.dev
As software engineers we should all be able to communicate the things we have built to others, but giving a formal demo of something you’ve been working on can be daunting.| jacobtomlinson.dev
The more we scale up our workloads the more we run into bugs that only appear at scale.| jacobtomlinson.dev
Sometimes when I give live demos with Kubernetes clusters I want to make sure that the container images I’m going to use are already pulled onto all of the nodes in my cluster.| jacobtomlinson.dev
Today I found myself needing a Go application’s main thread to stop and wait until the user wants it to exit with a ctrl+c keyboard interrupt.| jacobtomlinson.dev
Sometimes when working with a new Kubernetes cluster you will be given a config file to use when authenticating with the cluster.| jacobtomlinson.dev
Often when I’m developing or debugging in Python I end up creating throw away conda environments.| jacobtomlinson.dev
This week Computer Weekly have published an article interviewing me about how the Met Office is tackling the vast amount of data we are producing.| jacobtomlinson.dev
It is common practice for release notes to consist of a list of the Pull Requests which have been merged since the last release.| jacobtomlinson.dev
Introduction When using Hashicorp’s Vault you may want to have an authentication token which only has permissions to seal the vault.| jacobtomlinson.dev
My team won a Real IT Award for Innovation! Check out the full post here.| jacobtomlinson.dev
Introduction I recently acquired an Apple Airport Express wireless hotspot which wouldn’t power on. This was most likely down to a fault on the power supply board and so I decided to have a go at fixing it.| jacobtomlinson.dev
Introduction Currently InfluxData do not provide a 32bit FreeBSD build of Telegraf as part of their standard packages.| jacobtomlinson.dev
Introduction Building games for iOS is straight forward with Game Maker. You create the game as normal in Game Maker, but in order to build it you must have a Mac with Xcode installed.| jacobtomlinson.dev
Originally published on the Met Office Informatics Lab blog on June 24th, 2016. With some friends look out the window and each choose a weather symbol which represents what you see.| jacobtomlinson.dev
When you contribute code to collaborative projects, whether they are open source community projects or large internal projects inside organisations, the feeling of having your code running inside a large application can be very rewarding.| jacobtomlinson.dev
When working with Dask clusters you often need the remote worker environment to match you local environment.| jacobtomlinson.dev
In this post we will cover automatically running our tests when we push new code to GitHub, and when contributors raise Pull Requests against our project.| jacobtomlinson.dev
Many open source projects are taking steps to update terminology to be more inclusive. The largest of these changes has been renaming the “trunk” branch of git repositories from master to main.| jacobtomlinson.dev
A popular tool with streamers and YouTubers is Open Broadcaster Software®️ Studio or OBS for short.| jacobtomlinson.dev
In this post we will cover making our code available to people. This is the bit where we open the source!| jacobtomlinson.dev
Introduction It has been a few years since I published a list of technology and media I enjoyed this year, so here we go for 2019.| jacobtomlinson.dev
Applying for internships and entry-level positions can be tricky. Interviewers want to hear you talk about your experiences and things you’ve done that prove you’re a good fit for the job.| jacobtomlinson.dev
Helm has become a pervasive tool in the Kubernetes community for packaging, managing, upgrading and distributing applications.| jacobtomlinson.dev