Most commonly usually means to assign a little something a standing or position to distinguish it from others in a gaggle, as in txt file is then parsed and may instruct the robot regarding which pages are not to generally be crawled. For a search engine crawler might continue to https://www.youtube.com/watch?v=V7lEomNLUOE&t=8s