History
The invention of "robots.txt" is attributed to Martijn Koster, when working for WebCrawler in 1994. "robots.txt" was then popularized with the advent of AltaVista, and other popular search engines, in the following years.
Read more about this topic: Robots Exclusion Standard
Famous quotes containing the word history:
“If man is reduced to being nothing but a character in history, he has no other choice but to subside into the sound and fury of a completely irrational history or to endow history with the form of human reason.”
—Albert Camus (1913–1960)
“When the history of guilt is written, parents who refuse their children money will be right up there in the Top Ten.”
—Erma Brombeck (20th century)
“The basic idea which runs right through modern history and modern liberalism is that the public has got to be marginalized. The general public are viewed as no more than ignorant and meddlesome outsiders, a bewildered herd.”
—Noam Chomsky (b. 1928)