2025/08/28| boyter.org
By the time you are reading this, searchcode.com’s DNS should have been updated to point at a placeholder index.html file I created.| boyter.org
With the rise of AI LLM technology, I naturally wanted to learn more about using it. Both from a what can it do but also where can I use it in various tasks. Since I am also investing more time in searchcode.com these days I wanted to find a way I could use these tools to improve it.| boyter.org
Previously I had covered the worst program I had ever worked on (although that probably needs updating now) and the worst individual and the most incompetent individual I ever worked with.| boyter.org
Back in August 2021 during the covid madness I updated my rather aged desktop to have new CPU, GPU, PSU and fans. The goal being to make it quieter and get more life out of the system. I also kept it running Windows at the time, since I already have a stable Windows 10 install on it. However it is now 2025 and we are getting close to the end of life time for it in October. I never had any major issues with Windows up till 10. However Microsoft’s increasingly hostile approach to the hardware...| boyter.org
At GopherCon2023 I presented a talk about how I built a custom index for searchcode. In there somewhere I mentioned that having a visualization of how the search works is something I would like to add in. With some free time on a weekend rather than “wasting” time playing a game I decided to actually build it, and as is the current trend on the internet attempted to vide code it, at least for the UI portion anyway.| Ben E. C. Boyter
I was a speaker at the first Go meetup in Sydney 2025… the first speaker in fact. As is my habit I am posting my notes for everyone who attended to view them. I believe it was recorded and will post the link to that if I ever get the link. You can find the link to the talk based on the blog post searchcode.com’s SQLite database is probably 6 terabytes bigger than yours.| Ben E. C. Boyter
I have been working on a secret project within https://searchcode.com/ and ran into a blocker where the UI had become a road-block. I will be the first to admit that I do not like styling things with CSS very much. In fact working with HTML and CSS is probably my least favorite part of modern web development. I have never discussed any use of “AI” on this blog. Not for lack of using it, as I had been using ChatGPT and other tools since day one. I had always previously found that the model...| Ben E. C. Boyter
Happy new year! I am in the middle of converting searchcode.com over to SQLite, which is a far more interesting future post, and as a result had to rewrite some of the core service logic. The previously implemented code had grown over time so I decided to start fresh. Of course the catch of this is introducing regressions, which I did by stripping out some of the caching logic I had implemented.| Ben E. C. Boyter
As much as I like ZeroLog for Go logging Slog being added to the standard library fits in with my rule of avoiding 3rd party packages when they are not entirely required hence I have started moving anything I work on over to it. One thing I find especially annoying however is log messages inside tests. As such here is a quick code snippet which turns off all slog outputs, but only for tests. Stick it into any _test.go file to disable logging outputs for that package.| Ben E. C. Boyter
I was talking to someone recently about how searchcode.com works with regards to its bloom filter. The question came up about why it stores the index in memory and not on disk considering how fast disks are these days. Now the answer is firstly performance, but the second is that writing all the code to store the index on disk is a lot of work and I didn’t feel like working on it. However I was curious… since I have been using SQLite a lot recently, how about storing the bloom filter on d...| Ben E. C. Boyter
A work colleague (Dazza!) messaged me this morning about a new code counter with some pretty serious performance claims. Not only was it claiming to be accurate it was claiming to have a 38x speed improvement over tools such as tokei! Code counters are something I am interested in so I did some digging to find the repository https://github.com/mainak55512/stto Interesting it is written in Go. My first though about the times was that all it was measuring is the startup time of a binary in what...| Ben E. C. Boyter
Welcome. If you are reading this you either follow me on RSS, watch my blog or were directed here at the start of my ServerlessDaysANZ 2024 talk about “Abusing Go, AWS Lambda and bloom filters to make a true Australian serverless search engine”. Whatever the case may be, welcome! For the slides visit https://boyter.org/static/serverlessdaysanz2024/ then just use the space bar to advance, the arrow keys to move around or the ESC key to see everything and then enter to select a slide. Press...| Ben E. C. Boyter
I recently pushed out a new release of Sloc Cloc and Code (scc) with two main pieces of functionality. The first being accurate .gitignore support. The latter being the first new feature to hit the codebase in a long time in the form of a new metric you can access Unique Lines of Code or ULOC. A few years ago exwhyz (who appears to have dropped off the internet) posted a feature request to include this new metric into scc’s outputs.| Ben E. C. Boyter
I recently X’ed/Tweeted about how my searchcode.com since Nov 2022 using its custom index https://boyter.org/posts/how-i-built-my-own-index-for-searchcode/ had processed over 50 million queries with a rolling average time around 80ms to process them. That was a low-ball estimate though. I actually checked and its actually just over 100,000,000 queries. Which I find astonishing considering its running though a index I wrote. However that number excludes all of the attempted queries. As you m...| Ben E. C. Boyter
I have always been a big fan of SQLite and recently was using it with Go/Golang. However due to fact that goroutines are very convenient I quickly ran into the “database is locked” issue where multiple writes would try to update the database at any one time. Multiple reads worked without issue. I could have solved this by throwing a mutex lock around the database writes, but thats something I could easily mess up by missing the mutex and locking the database at the wrong time. I could als...| Ben E. C. Boyter
One of the things I see commonly on any mockup is a search box, usually with vaguely defined requirements as to how this is meant to actually work. However most people tend to assume its an instant update as you type with a debounce option of 100ms or something. The fastest way to implement this with a database is a horrible concatenated like query. However where you can store things in memory you can do a much better brute force search. A possible implementation in Go could look like the below.| Ben E. C. Boyter
Having had to install direnv multiple times over the last few days to get a team up and running im writing the method down because I keep having to look it up. Install based on the following instructions https://github.com/direnv/direnv/blob/master/docs/installation.md Then hook it up to your shell via https://github.com/direnv/direnv/blob/master/docs/hook.md Lastly to have it work with .env files create the following directory and file ~/.config/direnv/direnv.toml and edit the contents to co...| Ben E. C. Boyter
Sometime last year Justin Vincent of the techzing podcast suggests that I should add a license to Quote, I “strongly” suggest that you wrap each product into a paid version - where you sell a “license” for usage with priority support. New year means new energy and so I got around to do this. All the tools will remain free software, so there is no need to worry about a rug pull. However it does mean I am now offering proper email and if required face time support for every license sold...| Ben E. C. Boyter
In no particular order a list of Go libraries/packages I really like and some reasons why. zerolog I know that the changes to slog make it far closer to being what zero log is now, but I still prefer it for my own use. requests Anyone who works with python for a while learns about the requests library for making HTTP calls. This is similar but for Go. It has a few sharp edges when dealing with websites that don’t follow standards, but otherwise is amazing. Coupled with the next library in t...| Ben E. C. Boyter
So on occasion, I find myself wanting to run SQL against customer databases. Where this gets annoying is that because we are often in AWS using lambda I have no ability to connect to them. Occasionally im able to bully the friendly devops’ and security people to allow me access to a bastion host, which I can then connect to the SQL database. However this is often a painful process if allowed at all, and so with permission I sometimes add a special route into the application allowing me to r...| Ben E. C. Boyter
Welcome. If you are reading this you either follow me on RSS, watch my blog or were directed here at the start of my GopherConSyd 2023 talk about “Bloom Filters: Building a Cutting Edge Go Search Engine to Explore the World’s Source Code”. Whatever the case may be, welcome! For the slides visit https://boyter.org/static/gophercon-syd-presentation-2023/ then just use the space bar to advance, the arrow keys to move around or the ESC key to see everything and then enter to select a slide....| Ben E. C. Boyter
TL/DR; Play/Download a copy from itch.io I had always wanted to create games. In fact its one of the reasons I went down the development path. For various reasons I never invested enough time (probably too much time playing them) and went into writing code for businesses. In hindsight that was probably the right move from a financial perspective, since it seems most game developers tend to be fairly underpaid and the drama in the game development space makes high school teen drama look tame.| Ben E. C. Boyter
I was recently selected to present at DataEngBytes 2023 after submitting an idea for a talk based on my blog post Processing 40 TB of code from ~10 million projects with a dedicated server and Go for $100. I think it went well, other than not having the slides able to present for the first 10 minutes. After 5 mins or so I decided to just plough on since it was less of a visual talk anyway. I did have a few people come to me afterwards and thank me for it which was nice.| Ben E. C. Boyter
Since I maintain a few projects that deal with source code one of the things I needed badly was a way to parse and understand gits .gitignore and .ignore files in order to get as much accuracy as possible. I had previously tried using code I lifted from The Platinum Searcher albeit with some fixes to avoid crashes. Annoyingly however it never implemented glob’s correctly. I tried searching around for another implementation but none appeared to work as expected.| Ben E. C. Boyter
I released code spelunker a few days ago https://github.com/boyter/cs and literally one person asked for details on how it worked. Fitting in with my habit of putting any content I produce into my blog what follows is a built out version of it. So code spelunker started when I noticed someone using Visual Studio to search files recursively in a directory and my own use of ripgrep with fzf. It’s development has been on and off over the last 4 years, roughly broken down into the following cat...| Ben E. C. Boyter
Code spelunker (cs) or code search is a new command line tool I have been working on and off over the last few years. It allows you to search over code or text files in the current directory either on the console, via a TUI or HTTP server, using some boolean queries or regular expressions. I just recently pushed out a stable first version release and thought I would write a quick post about it.| Ben E. C. Boyter
I previously wrote about starting a Go project in 2018. A lot has changed since I wrote that and I had been wanting to write an updated version. What follows should be enough for anyone new to Go to get started and ideally start them being productive. Quicklinks Install / Setup Starting a Project Learning Go Searching Building / Installing Linting / Static Analysis / Security Scanning Profiling Unit Testing Integration Testing Community Multiple Main Entry Points OS Specific Code Docker Usefu...| Ben E. C. Boyter
Another progress update on https://searchcode.com where I was trying to reduce the time it takes to index source code. Since the this was a real world example of profiling code trying to reduce its cost I thought I would document how it went. The first step is to add code allowing the collection of a profile. I don’t do this often enough in searchcode to have a want to enable/disable this so I just comment/uncomment this when needed. Note that I have this set to stop profiling 30 seconds af...| Ben E. C. Boyter
TL;DR: If you don’t need to preserve order of the elements, sort the slice first and dedupe. I have been working on https://searchcode.com a fair bit recently, having been dealing with memory issues which were causing it to crash with out of memory exceptions. The cause of that was due to backtracking regular expressions in the syntax highlighter, which will be the subject of another post sometime. As a result of the above I have spent a fair amount of time looking at memory profiles. One t...| Ben E. C. Boyter
I have been doing interviews recently, and whenever someone mentions knowing a lot of Go and Java one of the questions I put to them is how to implement Go channels in Java. Since there isn’t anything online to suggest how to do this that I liked reading (that I could find) I thought I would write one, and do it in native Java without needing to import a library.| Ben E. C. Boyter
I have been investigating how ActivityPub is meant to work. What follows is how a boost/announce/retweet works in Mastodon and any other system that implements against ActivityPub. You can find the full collection on github https://github.com/boyter/activitypub and contribute your own details. ActivityPub -> Announce -> Post When a user clicks on bonk/boost/retweet the following sequence happens. The users intent is converted to an announce. This announce is sent to every follower they have a...| Ben E. C. Boyter
I didn’t really expect to have a post this late in the year but here we go. Lets go through some assumptions that were broken for me recently. Not just my own through perhaps some other peoples assumptions too. I have been working with ActivityPub for a bit. I want to run my own server, beyond installing some 3rd party. I want to build my own from scratch. However this rapidly exposed some problems with testing ActivityPub.| Ben E. C. Boyter
Some time ago I wrote about creating leet speak guid’s as a way of having fun while working with sitecore. https://boyter.org/2014/05/unique-guid/ I recently realised I wanted to have this around so I could create more interesting guids for one reason or another, but couldn’t find the script I was using back then. So I created it again, and its now on github as a gist for myself and anyone else to use https://gist.github.com/boyter/6f5b7c1b375e79449d01916f5828e1db feel free to use it as y...| Ben E. C. Boyter
Recently on my M1 Macbook I ran into the following issue when trying to compile some Go code with C dependencies. The “clang” command requires the command line developer tools. Being a Mac this also prompted me with a button I could click to install them. Naturally I clicked it, only to be prompted with the following.| Ben E. C. Boyter
Working for a large corporate company as a mid level developer is generally rather boring. Churn out some CRUD applications. Write ETL as businesses live on data (usually processed in excel). Meetings that are meant to resolve things but just result in more meetings. So in short, it is usually boring. However occasionally you get some sort of interesting problem, especially if you gain a reputation for actually solving them.| Ben E. C. Boyter
Thought I would write done some of the more annoying/interesting software bugs I have had to deal with recently. Both were dealing with an archive system where large amounts of video/audio/images are uploaded into, then searchable. It allows the editing of metadata for each type of content, as well sending to external systems. Problem #1 When a record is being edited a flag “This is being edited” is added to the record and removed on save. However randomly the flag would not be removed, a...| Ben E. C. Boyter
Recently have been doing interviews at work for Go developers. The filter for this is a simple review exercise. We present a small chunk of code and ask them to review it over 15 minutes pointing out any issues they see. The idea is to respect their and our time. It works pretty well and we can determine how much experience someone has by their ability to pick up the obvious vs subtle bugs.| Ben E. C. Boyter
If you succeed, you will fail. Something said at a client engagement I was at some time ago. I figure enough time has passed that it is worth sharing this story. One of the early engagements was to deliver for a very large company an iOS Application powered by a RESTful API. This was problematic due to what is common for every large organisation, a collection of older legacy backends. Some with their own API’s, and bizzare authentication processes. Part of the deliverable was also working w...| Ben E. C. Boyter
Some time ago I wrote about building an Australian search engine by abusing how AWS Lambdas work. You can view it at bonzamate.com.au, and read the post about it here. Recently I noticed that Bonzamate.com.au was in turn being abused by a series of bots repeatedly making queries against it. Now normally I don’t worry about bots. There is little you can do to stamp them out entirely, and its always nice to see someone trying to exploit the things you work on. However should the bots either r...| Ben E. C. Boyter
A living document where I will be putting information about search index implementations as I learn about them. So there are a few talked about ways to search across content or build an index that I am aware of. Lets discuss each in term with its advantages and disadvantages Brute Force Brute Force, note that this isn’t an index strategy per say, but worth discussing anyway. Assuming you can get the entire corpus you are search into RAM it is possible to brute force search. I’m including ...| Ben E. C. Boyter
I have been working with AWS API Gateway and Lambda using Go a lot recently. One of the more annoying things about API Gateway and Lambda is the inability to run things locally. As a result I tend to write a lot of unit tests to compensate. This works up till you start doing things like SQL queries as mocking away the database is a less than ideal situation if you are working with raw SQL, which I tend to do since I don’t like most ORM’s. These limitations of unit tests is why I prefer in...| Ben E. C. Boyter
Previously I had covered the worst program I had ever worked on (although that probably needs updating now) and the worst individual. I thought it was time branch out into the most incompetent person I ever worked with. So a bit of back history. This was at lendlease where we were working on a system that massaged all employee data into a single place allowing accurate counts of employees, and getting titles mapped to real ones and such. This was important because lendlease being an umbrella ...| Ben E. C. Boyter
I have been playing around with offline non cloud tool replacements for doing things like speech to text and the like. One that I found which looked promising was voice2json which is a command line tool for turning speech intent into text. It can do generic conversion too, but a review of that is not the subject of this post. Following the instructions from the website http://voice2json.org/install.html worked fine up till I tried to run it and got the following issue.| Ben E. C. Boyter
I remember hearing about Bogan Ipsum, which was an alternative to Lorem Ipsum for you know bogan’s back in 2011 through LifeHacker. However it since became defunct. I became aware of this fact from Darren at work given the following message on slack? Dude Boganipsum where? Clearly this needed to be rectified! A quick search found this https://www.npmjs.com/package/boganipsum implementation which seemed to work well enough, so I ported it to basic JavaScript (think 2001 style JavaScript) and...| Ben E. C. Boyter
Development Practices Follow a development practice of developer empowerment. Encourage change and experimentation with zero or low penalty for defects allowing them to be found early and resolved with additional tests or automation. I embrace the fact that bugs and problems will be found, but by making the penalty for them exceedingly low we can ensure that they are identified and resolved quickly with little to no production impact. All processes from CI/CD and development are built around ...| Ben E. C. Boyter
I’m in the middle of moving searchcode.com’s database to another server while upgrading it. This means I need to copy it somehow. It’s especially difficult in this case because the server its sitting on does not have enough disk space to do my usual mysqldump then copy the file across using SCP. I investigated splitting the database using --where clauses allowing me to get chunks of the database across, but that also means adding things like --skip-create-options and --skip-add-drop-tab...| Ben E. C. Boyter
UPDATE 2025-05-19 Somewhat un-suprisingly AWS has closed the loophole that I was exploiting for this https://aws.amazon.com/service-terms/#30._AWS_Lambda and as such compiling data into your lambda for “free” storage is no longer allowed. As such bonzamate.com.au in its current iteration has been shut down. Watch this space however! I have some ideas on how to bring it back, albeit in a different form. TL/DR I wrote an Australian search engine. You can view it at bonzamate.com.au. It’s ...| Ben E. C. Boyter
A little over a year ago I wrote about my experience getting the NBN and concluding that it was a massive disappointment. I ended with Anyway thats the end of it. A 7 years wait for internet that I guess might be the envy of people living in Libya. Obviously since this post I have something more to say. NBN changed their policy allowing you to get a TCP (technology upgrade) quote without having to pay. Of course I tried it. The dollar amount was not small but with the WFH situation continuing...| Ben E. C. Boyter
I used to be really into gaming and even more so building computers, reading CPU reviews, video card reviews, learning about water cooling, what is the best hard disk, psu and such. That was a rather long time ago now. In fact the last time I put any effort into this was the Athlon XP days where I think I had a 2700+ with a Radeon 9700, which was a pretty sick build at the time.| Ben E. C. Boyter
So had an interesting experience with Go the other day. This is part of building a bloom filter index which is going to replace the index in searchcode.com So I was trying to verify how long a search would take when the index was full. When trying it out on a machine to see what the performance would be I built a version to create some very large slices over 64,000,000 items in length to simulate the expected workload. This should take about 53 GB in memory with all the overhead and such adde...| Ben E. C. Boyter
Mostly here because I keep forgetting all the things I want to put into a Go project when I start it. Collected into a list of things I actually need to do, tools to install and how to get things working. This page is a work in progress and something I will add to as I remember things or discover something new. Static Analysis https://golangci-lint.run/usage/install/ Install that and run golangci-lint run ./... from the root when checking in.| Ben E. C. Boyter
Recently my workplace has been accepting some interns as part of contributing back to the industry in general, as well as promoting the brand and hopefully being able to grow and grab excellent candidates in the future when they graduate. One recently wanted some time to sit down with me to ask a few questions. In order to give him the best use of time I asked to have the questions written in advance so I could respond to them before chatting and then build out my answers verbally.| Ben E. C. Boyter
A bloom filter is one of those data structures you are probably already aware of, or have at least heard about. For those looking for a simple recap, they are a probabilistic data structure which can be used to determine if something is in a set or not, giving a slight chance of returning a false positive result for some checks but while using less space than a full hashmap.| Ben E. C. Boyter
At my current workplace we have been doing themed Friday stand-ups. The most recent was about your first PC love, or whatever technology you look back on fondly. What follows is my one. My first PC was I think a Commodore PC40-III. I’m a little fuzzy on the details and its hard to find it based on pictures. Was not my first taste of technology, but the one I look on back most fondly.| Ben E. C. Boyter
TL/DR: I wrote a fast literal string matching library in Go get it here https://github.com/boyter/go-string/ Recently I rebuilt searchcode in Go, as mentioned. While there was a variety of reasons for this one was that performance in the previous Python version was not great. Please note this was mostly due to my own shortcomings and not the language itself. However I have always used searchcode as a test bed for interesting problems, since it gets enough traffic to verify things at scale, an...| Ben E. C. Boyter
SI live in an area with a popular driving road. As such you see a lot of people flying around in fancy cars, motorbikes and bike riders. Between where I am and the start there is an awful bit of road that funnels everyone together. As such a lot of people jump the line to get past the bike riders because there isn’t enough space for them to have their own lane (I keep complaining to my local member about this).| Ben E. C. Boyter
I write this mostly so I can move on. The NBN has been a source of constant frustration and anger for me. While I will never forgive those who caused the current shambles I can perhaps redirect my anger elsewhere will keeping it in the back of my mind. So here is my brief time-line of NBN. I moved to area that was due to get NBN FTTP in ~2014. The fact that it was due to get such a connection influenced the decision. With the change in government that ended up going away. I was then promised ...| Ben E. C. Boyter
Unicode support in my experience seems to be one of those hand wavy things where most people respond to the question of “Do you support unicode?” with Yeah we support emojis, so yes we support unicode! 😃😭😈 Which is partially correct… It’s certainly a good start. However unicode is more than emojis. So lets take a step back. What is unicode? It’s actually easy to explain, here it is taken from the wiki page “A standard for representing the worlds text”. That means it inc...| Ben E. C. Boyter
So regular readers of this site might know that I run https://searchcode.com which I recently converted over to Go. While I had always run searchcode cheaply I never wanted to implement rate limits and instead operate as Bill and Ted by being “Excellent to each other”. Sadly the internet is not from the Bill and Ted universe and due to some level of abuse against the API I have been forced to implement a rate limiter.| Ben E. C. Boyter
So as of about 12 hours a new version of https://searchcode.com/ has been rolled out. It marks the biggest update to the site in over 4 years since I moved over to version 3 which you can read about https://boyter.org/2016/07/searchcode-com-architecture-migration-3-0/ The reason for it was that searchcode has always been something I kept running as a showpiece. It’s something that I can point to publicly and say “I did that”. The idea being to ensure I remain employable. Recently though...| Ben E. C. Boyter
I write and publish open source code. I quite often get bug fixes, patches, requests and questions regarding the things I have published. Most of the feedback is respectful and helpful. However occasionally there are other requests. This page is where I plan to redirect them as a final warning before I block and ignore them. The three F’s to open source development. Fix it, Fork it, Fuck off Fix it If you find a bug, notice missing, incorrect or misleading documentation, or want an enhancem...| Ben E. C. Boyter
For those that wish to follow along with the presentation use the below link, https://boyter.org/static/gophercon-syd-presentation/ You can press S to see the speaker notes as you go along.| Ben E. C. Boyter
Developer? Need to know how to use Elasticsearch? Want to know how to get working with it quickly? This is the book for you! Buy now using Leanpub --- In elasticsearch mappings define type and values in documents. You use them to specify that fields within your document should be treated as numbers, dates, geo-locations and whatever other types elasticsearch supports. You can also define the stemming algorithm used and other useful index fields.| Ben E. C. Boyter
Developer? Need to know how to use Elasticsearch? Want to know how to get working with it quickly? This is the book for you! Buy now using Leanpub --- It is useful to understand is how elasticsearch stores documents. Documents that are indexed need to go into an index and have a type. Indexes can contain one or more types. This may sound limiting but you can search over all indexes or all types within an index or just one type within an index if you require.| Ben E. C. Boyter
A post that is constantly a work in progress. ripgrep - Similar to grep, but a drop in replacement for large file scanning without unicode penalities. It’s also similar to Ag and Ack and optimised for searching code, ignoring .git directories and the like. https://github.com/burntsushi/ripgrep tmux - Terminal multiplexer. I wrote a 5 min guide to this. Use it to have resumable sessions after closing ssh sessions and to have split pane windows. bat - Replacement for cat but provides syntax h...| Ben E. C. Boyter
The command line tool I created Sloc Cloc and Code (scc) (which is now modified and maintained by many other excellent people) counts lines of code, comments and makes a complexity estimate for files inside a directory. The latter is something you need a good sample size to make good use of. The way it works is that it counts branch statements in code. However what does that actually mean? For example “This file has a complexity of 10” is not very useful without some context. To solve thi...| Ben E. C. Boyter
Today I presented at the Sydney Dev Warriors meet-up about a project I was working on from the start to end of 2018. In a nutshell it was a self serve digital archive for video, audio, images and metadata. Since I am in the habit of collecting everything I do into one place I chose to do the presentation using reveal.js which means I can also include it here. How AWS and Go Transformed a Public Broadcasters Digital Archive.| Ben E. C. Boyter
This is now part of a series of blog posts about scc Sloc Cloc and Code which has now been optimised to be the fastest code counter for almost every workload. Read more about it at the following links. Sloc Cloc and Code - What happened on the way to faster Cloc 2018-04-16 Sloc Cloc and Code Revisited - A focus on accuracy 2018-08-28 Sloc Cloc and Code Revisited - Optimizing an already fast Go application 2018-09-19 Sloc Cloc and Code a Performance Update 2019-01-09 Sloc Cloc and Code Badges ...| Ben E. C. Boyter
Let me start by stating that I think all chat-bots are rubbish and that a better solution is a decent search implementation. Type in some terms and get some useful results. Much better than typing to a machine. A while ago I was working at a client site where there was an internal discussion group about building a chat-bot. Many meetings had passed, discussing using neural networks, building flows, should we use IBM Watson. Lots of talk, but not much delivery.| Ben E. C. Boyter
About 12 years ago my old man was working as one of those hire a husband/jim’s maintenance deals. This was in the lower Blue Mountains. During my university break I didn’t have much on and would often come with him as something to do. We were attempting to repair a deck of some flea hole house near Glenbrook. The place would have been improved only by a match and petrol. The owner was around and mentioned that the place was to be rented out and real estate had just sent some people on the...| Ben E. C. Boyter
A long weekend just recently passed in NSW Australia. With a larger than normal amount of time on my hands I decided to try out the game Sea of Thieves. Ignoring the fact that the NBN rollout here has ruined my ADSL2+ which has gone from a 18-19 Mbps connection to 3 Mbps and that it took 24 hours to download, what follows is probably one of the more memorable gaming experiences I have ever had.| Ben E. C. Boyter
It started with this chain of blog posts, Using Java to Read Really, Really Large Files -> Processing Large Files in Java -> Processing Large Files – Java, Go and ‘hitting the wall’ I thought after reading them I would add to the chain. While I don’t like solving arbitrary company made “programming tests” when doing interviews https://boyter.org/2016/09/companies-recruiting/ whenever I see a post comparing Go to Java where Java is trounced in performance gets my attention. Unless ...| Ben E. C. Boyter
Recently my job/career path has had me looking “under the hood” at clients development practices. The idea being to see where their strengths and weaknesses are and then make actionable suggestions. Generally every client wants the same outcome. They want to know how to improve software quality and deliver features faster. When doing this I focus on the development side, but honestly a lot of this applies to DevOp’s as well. In most situations I realized very quickly that I would be ter...| Ben E. C. Boyter
A short time ago one of the more interesting blog posts (to me anyway) about cache eviction popped up on Hacker News which prompted me to post the following comment. Love reading this. It has always been one of those interesting things I kept in the back of my mind in my day to day. I was very excited when I actually got to implement it on a real world project.| Ben E. C. Boyter
Go as a language is fairly decent. However because questions about why I have issues with tends to come up often enough on the company slack programming channel (see what I did there?) I figured I would write them down and put it here so I can point people at a link when they ask what my complaints are. For the record I have been using Go heavily for the last year or so, writing command line applications, scc, lc and API’s. These include large scale API’s for clients to a syntax highlight...| Ben E. C. Boyter
tmux the terminal multiplexer is one of those brilliant tools I use all day. I am not a tmux expert, nor do I have any need to be. What follows is the bare minimum you need to know to feel productive in tmux, written for some work colleagues. The following assumes you have not configured or tweaked tmux in any way opting for the default settings. From the command line to start or connect to tmux| Ben E. C. Boyter
I had a great deal of difficulty searching for an answer to this I have included it here to hopefully save someone else the effort. How to make a S3 request to retrieve items that have transitioned into AWS Glacier from an S3 bucket using the Go AWS SDK API. packagemainimport ( "fmt""github.com/aws/aws-sdk-go/aws""github.com/aws/aws-sdk-go/aws/awserr""github.com/aws/aws-sdk-go/aws/session""github.com/aws/aws-sdk-go/service/s3") funcmain() { sess, err:=session.NewSession(&aws.Config{ Region: a...| Ben E. C. Boyter
Update 2019-03-13 This is now part of a series of blog posts about scc Sloc Cloc and Code which has now been optimised to be the fastest code counter for almost every workload. Read more about it at the following links. Sloc Cloc and Code - What happened on the way to faster Cloc 2018-04-16 Sloc Cloc and Code Revisited - A focus on accuracy 2018-08-28 Sloc Cloc and Code Revisited - Optimizing an already fast Go application 2018-09-19 Sloc Cloc and Code a Performance Update 2019-01-09 Sloc Clo...| Ben E. C. Boyter
I have recently started work again on searchcode server with the continued plan to upgrade searchcode to use the same code-base. One big issue with this which I recently tweeted about was that Java has a decent well documented fast library for just about everything, with the exception of a good syntax highlighter like Pygments http://pygments.org/ or Chroma https://github.com/alecthomas/chroma It is however possible to use Pygments in Java using Jython http://pygments.org/docs/java/ which was...| Ben E. C. Boyter
A while back I collected all of the fake Orly book covers I could find an posted them online https://boyter.org/2016/04/collection-orly-book-covers/ I also mentioned that I was seriously considering writing content for the “Expert Excuses for Not Writing Unit Tests” one. I think I had this idea at the time that it was something I could potentially sell or use to raise my profile. Since I am not a comedy writer there is a market of about 0 people in the universe who would pay for content l...| Ben E. C. Boyter
Types of Development #1 Rage driven development Where an existing application/process causes one to rage so hard you spend the next 6 weeks over-engineering a solution to solve a simple problem. Example Perl. --- Types of Development #2 Test driven development Where previous projects and death marches result in a violent reaction of over-correcting for past failures and everything must be tested dammit. Example includes any project where the goal is 100% test coverage.| Ben E. C. Boyter
Occasionally I try to branch out from working on https://searchcode.com/https://searchcodeserver.com/ or https://github.com/boyter/scc and find something that I find interesting to toy around with. Usually I go for something that will only take a few hours to play around with. I had recently re-read a book series I remember loving a while ago which is the Empire Trilogy of books by Feist and Wurts https://en.wikipedia.org/wiki/Empire_Trilogy One of the things that stuck out to me as I was rea...| Ben E. C. Boyter
Update 2019-03-13 This is now part of a series of blog posts about scc Sloc Cloc and Code which has now been optimised to be the fastest code counter for almost every workload. Read more about it at the following links. Sloc Cloc and Code - What happened on the way to faster Cloc 2018-04-16 Sloc Cloc and Code Revisited - A focus on accuracy 2018-08-28 Sloc Cloc and Code Revisited - Optimizing an already fast Go application 2018-09-19 Sloc Cloc and Code a Performance Update 2019-01-09 Sloc Clo...| Ben E. C. Boyter
Update 2019-03-13 This is now part of a series of blog posts about scc Sloc Cloc and Code which has now been optimised to be the fastest code counter for almost every workload. Read more about it at the following links. Sloc Cloc and Code - What happened on the way to faster Cloc 2018-04-16 Sloc Cloc and Code Revisited - A focus on accuracy 2018-08-28 Sloc Cloc and Code Revisited - Optimizing an already fast Go application 2018-09-19 Sloc Cloc and Code a Performance Update 2019-01-09 Sloc Clo...| Ben E. C. Boyter
I have been wanting to learn Rust for a while now. I did play with it some time back https://boyter.org/2017/09/working-rust/ for solving some of the project Euler problems and was reasonably impressed with how it turned out. However as I had no practical use for it at the time I ended up investing more time in Go. Go as it turns out is a pretty decent language, and somewhat akin to Python in terms of getting things done. There is usually one obvious way to solve any problem. I was even very ...| Ben E. C. Boyter
A work colleague (let’s call him Owen as that’s his name) asked me the other day “I dont understand the problem space scc et al solve. If you wanted to write a short post, i’d read and share the hell out of it. Basically, it seems like a heap of people can see the need for it, and I’m trying to understand it myself” Owen is one of the more switched on people I know. As such if he is asking whats the point of tools such as scc, tokei, sloccount, cloc, loc and gocloc then I suspect ...| Ben E. C. Boyter
I was looking to add multiple tags (the declaration after the struct field) to a struct in Go such that I could have it defined in GORM and in JSON. This was much harder to find via any search engine I tried than I would have guessed. The answer thankfully is simple once you know and was provided to me by Josh of https://boyter.org/2017/03/golang-solution-faster-equivalent-java-solution/ fame. The trick is to define multiple tags but with a space separating them. Easy enough and hopefully thi...| Ben E. C. Boyter
This post is fairly old. See the updated version Starting a Go project in 2023 Getting started with a Go project in 2018 is frankly a little more painful then getting anything else started IMHO. With that here is what I have been doing to get started. The first thing to do is download and install Go. I would suggest always installing from the Go website itself https://golang.org/ and following the instructions for your OS of choice.| Ben E. C. Boyter
Something I always forget and have to search for is how to use Python to serve a directory for local testing and the like. Including it below so I have a copy I can always easily look it up. python -m SimpleHTTPServer 8090 and in Python 3 python3 -m http.server| Ben E. C. Boyter
It started by wanting to write a code counter that would be faster than cloc Update 2019-03-13 This is now part of a series of blog posts about scc Sloc Cloc and Code which has now been optimised to be the fastest code counter for almost every workload. Read more about it at the following links. Sloc Cloc and Code - What happened on the way to faster Cloc 2018-04-16 Sloc Cloc and Code Revisited - A focus on accuracy 2018-08-28 Sloc Cloc and Code Revisited - Optimizing an already fast Go appli...| Ben E. C. Boyter
UPDATE The below is here for historical reasons, but since 2021 very out of date. See this post https://engineering.kablamo.com.au/posts/2021/quick-comparison-between-go-file-walk-implementations for an updated comparison. Whats the fastest way to get all the names of all files in a directory using Go? I had a feeling that the native walk might not be the fastest way to do it. A quick search showed that several projects claimed to be faster. Since the application I am currently working on nee...| Ben E. C. Boyter
A collection of my favorite posts to read and re-read about optimizing code to an extreme. Unlikely that I will ever need to go to the extremes that these very talented individuals go to but its nice to learn the techniques. In no particular order. Announcement of Ripgrep a tool for searching code. Brilliantly written with benchmarks and analysis to explain what is happening for each https://blog.burntsushi.net/ripgrep/ the HN comments about it are worth reading as well https://news.ycombinat...| Ben E. C. Boyter
A simple blog post here to introduce a new command line tool licensechecker (lc), which is similar in purpose to the library Licensee http://ben.balter.com/licensee/ which attempts to identify what software license(s) code is released under. lc itself is dual licensed under the MIT and Unlicense. Licensechecker (lc) was designed to be run either on your command line or using your CI tool of choice and produce either CSV, JSON, SPDX, Summary or CLI tabular or progress output with the option to...| Ben E. C. Boyter
Another one of those things I need to look up every now and then. Below is a snippet of how to encrypt and decrypt a string using Python and KMS in AWS. The interesting thing is that you don’t need to supply the KMS key alias in the decryption portion. So long as whatever role or key you are using can access the key it should work. For the encryption you can either supply the full ARN of the key or the alias so long as you prefix it with alias/| Ben E. C. Boyter
Plexus “A combination of interlaced parts; a network.” For a while I have been neglecting searchcode.com while working on searchcode server. This is of course not an ideal situation and as such I have started working on it again. The following is just a brief list of things I am considering, problems, issues etc… So back when I first started working on searchcode I wrote it using PHP. For searchcode next (the current version) I rewrote it using Python and Django. I had intented to conti...| Ben E. C. Boyter
So a few weeks back I received an interesting email that at first thought looked like a scam. In short I was being invited to a BBQ at the Consul General’s residence in Sydney to meet an ICT/start up delegation from the Netherlands. Let me straight up say I have no idea why I would receive something like this. I am not dutch, have no dutch ancestry (that I am aware of) and while searchcode.com somehow is listed on the top startups in Sydney somehow its not as though it has lots of press cov...| Ben E. C. Boyter
For a while I have been wanting to play with a new programming language. Professionally I work with Java, C#, Python and JavaScript and so I went looking for something complimentary to them. My first stop was was Go. I started by implementing a vector space search in it which you can see here https://github.com/boyter/golangvectorspace While I liked the syntax (I still cannot think in Go), the libraries and the performance I realized that Go is close to a hybrid of Java and Python for me. It...| Ben E. C. Boyter
A very brief update about the progress of searchcode server. Currently I am in the middle of reworking how the index is built and maintained. The idea being I want to add zero downtime index rebuilds which requires a blue/green index strategy. It is still very much in flux but the current design is to merge the indexer and searcher which should allow this to happen. I have been playing around with using an iPad as a production device these days and produced the following document.| Ben E. C. Boyter
Abstract TL/DR| boyter.org