Robots.txt is a file that exists on the basis Listing of every Web page and can be utilized to instruct search engines like google and yahoo on which directories/information of the web site they could crawl and consist of of their index. Now that you have a clear concept of https://news.rafeeg.ae