History
The invention of "robots.txt" is attributed to Martijn Koster, when working for WebCrawler in 1994. "robots.txt" was then popularized with the advent of AltaVista, and other popular search engines, in the following years.
Read more about this topic: Robots Exclusion Standard
Famous quotes containing the word history:
“We may pretend that we’re basically moral people who make mistakes, but the whole of history proves otherwise.”
—Terry Hands (b. 1941)
“Let us not underrate the value of a fact; it will one day flower in a truth. It is astonishing how few facts of importance are added in a century to the natural history of any animal. The natural history of man himself is still being gradually written.”
—Henry David Thoreau (1817–1862)
“The true theater of history is therefore the temperate zone.”
—Georg Wilhelm Friedrich Hegel (1770–1831)