A fairly frequent job of a sysadmin is to do some benchmarking of a website. The first requirement of this is that you have some URLs to give to the benchmarking tool. I have found that getting a list of all the site’s URLs for assets like HTML pages, images, CSS files etc makes the test a bit more representative. The following command will very quickly spider a website and generate a text file of all the URLs that it finds:
