As some of the work planned for Marginalia Search this year has been progressing a bit faster than anticipated, there was time to implement an unplanned change. This post details the implementation of a system for detecting when servers are online, to avoid serving dead links and improve data quality, and for detecting when websites have significant changes including ownership transfers and parking. Table Of Contents Feature Rationale Data Representation Live Data Event Data Change Detection ...| www.marginalia.nu
You wake at 05:30 in the morning, feeling somewhat groggy. Instead of the alarm clock ringing like it normally does, a cheerful hologram appears: “Hi! I’m Kyle, your new alarm clock assistant!” You get dressed as Kyle explains all of the fantastic things he is capable of. You head over to the coffee machine. “Hey there! I’m Evan! Are you ready for AI in your coffee? But first - tell me about yourself!| www.marginalia.nu
The most recent change to the search engine is a system that profiles websites based on their rendered DOM. The goal is identifying advertisements, trackers, nuisance popovers, and similar elements. The search engine already tries to do this, but isn’t very good at it because it’s only looking at static code. It turns out to be somewhat difficult to determine what a website that has non-trivial javascript will look like based its source code alone, as this would require us to among other ...| www.marginalia.nu
The search engine has recently gained the ability to index the PDF file format. The change will deploy over a few months. Extracting text information from PDFs is a significantly bigger challenge than it might seem. The crux of the problem is that the file format isn’t a text format at all, but a graphical format. It doesn’t have text in the way you might think of it, but more of a mapping of glyphs to coordinates on “paper”.| www.marginalia.nu
Some time ago, I migrated the crawler off the okhttp library, to use Java’s builtin HTTP client. This seemed like a good idea at the time, but has led to a fair number of headaches. Java’s HttpClient has one damning flaw, and that that it doesn’t support socket timeouts. Its only supported timeout values are time to connect, and time until first byte of the response. This means the client can get stuck on a read call if a server stops responding, potentially for a very long time!| www.marginalia.nu