Robots.txt

See information about robots.txt at Webmasters versus HTTrack.