User-agent: * allow * Disallow: /cgi-bin/ #some of these will recurse Disallow: /proxy.pac #dont proxy thru here