I recently set out to run Stable Diffusion on GKE in Autopilot mode, building a container from scratch using the AUTOMATIC1111's webui. This is likely not how you'd host a stable diffusion service for production (which would make for a good topic of another blog post), but it's a fun way to try out| William Denniss
Per the NVIDIA docs, CUDA 12 applications require driver 525.60.04+. This driver is available as part of GKE 1.28. To upgrade an existing cluster to the latest version of 1.28: VERSION="1.28" REGION="us-central1" CLUSTER_NAME="autopilot-cluster-1" gcloud container clusters upgrade $CLUSTER_NAME| William Denniss
Image streaming is a really great way to speed up workload scaling on GKE. Take for example the deep learning image from Google. In my testing, the container is created in just 20s, instead of 3m50s. While there is slightly higher latency on reads while the image streams, the 3m30s head start is goi| William Denniss
Update: this information is now available in the official docs. If you want to know what version of your GPU drivers are active on GKE, here's a one-liner: kubectl logs -l k8s-app=nvidia-gpu-device-plugin -c "nvidia-gpu-device-plugin" --tail=-1 -n kube-system | grep Driver What th| William Denniss