Reply to this post | Go Back

View Post [edit]

Poster:	tycio	Date:	Oct 22, 2015 10:57pm
Forum:	web	Subject:	Entire website archival

http://www.davidgentle.com/ looks like an amazing successor to http://www.sandowplus.co.uk/ which has fallen into disrepair.

I would love to make sure that it is fully archived, so that every profile for every old-time strong man and the collections of public-domain manuals from the 20s and 30s are all chronicled for future use.

I have no idea how long the site will be up, but I know Archive.org is very reliable.

The thing is, going to every page to check it's archived is tedious and a drain on archive.org bandwidth.

Is there a way to just program a bot to scroll a site and verify it is mirrored on archive.org without actually downloading the data? Especially if the bot could also send a request to archive a page if there are no copies of it?

Is this frowned upon? It just seems really important but a lot of work.

All I can think is, if botting or entire-site-saving (scroll page, mirror all links, access those links, mirror all links, etc.) is frowned upon, is to browse the homepage archived only access it via archive.org until I go through everything?