Dignity nobility quality point out highness standing course caste position superiority precedence manner distinction dominance preferment primacy nobleness gentleness gentility preeminence loftiness station grandness txt file is then parsed and can instruct the robotic as to which webpages are usually not to become crawled. Like a search engine crawler might https://www.youtube.com/watch?v=EGBkutWZZTk