History
The invention of "robots.txt" is attributed to Martijn Koster, when working for WebCrawler in 1994. "robots.txt" was then popularized with the advent of AltaVista, and other popular search engines, in the following years.
Read more about this topic: Robots Exclusion Standard
Famous quotes containing the word history:
“A man acquainted with history may, in some respect, be said to have lived from the beginning of the world, and to have been making continual additions to his stock of knowledge in every century.”
—David Hume (1711–1776)
“To history therefore I must refer for answer, in which it would be an unhappy passage indeed, which should shew by what fatal indulgence of subordinate views and passions, a contest for an atom had defeated well founded prospects of giving liberty to half the globe.”
—Thomas Jefferson (1743–1826)
“The true theater of history is therefore the temperate zone.”
—Georg Wilhelm Friedrich Hegel (1770–1831)