History
The invention of "robots.txt" is attributed to Martijn Koster, when working for WebCrawler in 1994. "robots.txt" was then popularized with the advent of AltaVista, and other popular search engines, in the following years.
Read more about this topic: Robots Exclusion Standard
Famous quotes containing the word history:
“The History of the world is not the theatre of happiness. Periods of happiness are blank pages in it, for they are periods of harmony—periods when the antithesis is in abeyance.”
—Georg Wilhelm Friedrich Hegel (1770–1831)
“History does nothing; it does not possess immense riches, it does not fight battles. It is men, real, living, who do all this.... It is not “history” which uses men as a means of achieving—as if it were an individual person—its own ends. History is nothing but the activity of men in pursuit of their ends.”
—Karl Marx (1818–1883)
“The history of men’s opposition to women’s emancipation is more interesting perhaps than the story of that emancipation itself.”
—Virginia Woolf (1882–1941)