Robots.txt is a file that can be used to control search engine crawlers and web robots. This file tells crawlers which parts of the websi...