# All robots will spider the domain User-agent: * Disallow: /awstats/ # Disallow Wayback archiver # User-agent: ia_archiver # Disallow: / # Disallow directory // Disallow: /cgi-bin/ Disallow: /images/ Disallow: /signup/ Disallow: /mylogs/