# All robots will spider the domain User-agent: * Disallow: /pages/ Disallow: /stats/ Disallow: /private/ Disallow: /Scripts/ Disallow: /downloads/private/ Disallow: /loggedin.htm Disallow: /downloads.htm Disallow: /includes/