If you read my previous blog post, you probably already know that I like my smart home open-source and very local, and that certainly includes any voice assistant I may have. If you watched the video demo, you have probably also found out that it’s… slow. Trust me, I did too. Prefix caching helps, but it feels like cheating. Sure, it’ll look amazing in a demo, but as soon as I start using my LLM for other things (which I do, quite often), that cache is going to get evicted and that firs...| John's Personal Website
When I started hosting my services, I quickly ran into a major problem. Everything was timing out, but it was somehow working just fine when I was not connected to my home network! So turns out, this was because my router does not support what’s called NAT Loopback (also called NAT Hairpinning). Like many things you’ll see in production, the 32-bit address space of IPv4 was meant to be for a prototype.| John's Personal Website