By now, many of you have noticed our bot, , visiting your sites. We’re the Internet Archive (archive.org), and we’re building the Wayback Machine – a historical record of the web.
How to Control the Internet Archive’s Crawler ( ia_archiver ) – A 2005 Guide robots 2005 internet archive
User-agent: * Disallow: /