What is the Configure Robots.txt area of my console for?

By clicking Configure Robots.txt, in the sidebar block of your console's Site page, you can enter certain information into the text field. This helps restrict a search engine's robots (a search engine's automated programs) from scanning your AnswerHub site to archive its information. Robots index the web and direct search results by accessing a website's pages.