While on a fourteen-hour international flight, I finally managed to come up with an architecture for Clew's web crawler that I'm happy with. Here's the run-down.| Building Clew
I believe I've reached a point in Clew's development where, armed with the knowledge I've acquired from months of crawling sites and using that data to search the index, it's time to wipe the index and start over.| Building Clew
The current way that result ranking works in Clew is very different from what I want.| Building Clew
In which we launch the insanity that is this development blog for Clew.| blog.clew.se
Now, more than ever, people are giving up on mainstream search engines and looking for alternatives. Most alternatives, though, have a simple problem: they still rely on the mainstream indexes.| blog.clew.se
The web is mind-bogglingly huge; let's look at how personal websites can thrive and interact despite that.| blog.clew.se