Material Approach: The cornerstones of the material technique are key phrases. They serve as a blueprint for crafting weblog entries, posts, products descriptions, and different kinds of material. txt file is then parsed and may instruct the robot concerning which pages aren't to be crawled. Like a online search engine https://www.youtube.com/watch?v=p3IfcO3MzMs