Since my search engine has expanded its scope to include blogs as well as primordial text documents, I’ve done some thinking about how to keep up with newer websites that actually grow and see updates. Otherwise, as the crawl goes on, it tends to find fewer and fewer interesting web pages, and as the interesting pages are inevitably crawled to exhaustion, accumulate an ever growing amount of junk. Re-visiting each page and looking for new links in previously visited pages is probably off th...
