History
The invention of "robots.txt" is attributed to Martijn Koster, when working for WebCrawler in 1994. "robots.txt" was then popularized with the advent of AltaVista, and other popular search engines, in the following years.
Read more about this topic: Robots Exclusion Standard
Famous quotes containing the word history:
“It is true that this man was nothing but an elemental force in motion, directed and rendered more effective by extreme cunning and by a relentless tactical clairvoyance .... Hitler was history in its purest form.”
—Albert Camus (1913–1960)
“A people without history
Is not redeemed from time, for history is a pattern
Of timeless moments.”
—T.S. (Thomas Stearns)
“American time has stretched around the world. It has become the dominant tempo of modern history, especially of the history of Europe.”
—Harold Rosenberg (1906–1978)