History
The invention of "robots.txt" is attributed to Martijn Koster, when working for WebCrawler in 1994. "robots.txt" was then popularized with the advent of AltaVista, and other popular search engines, in the following years.
Read more about this topic: Robots Exclusion Standard
Famous quotes containing the word history:
“The history of our era is the nauseating and repulsive history of the crucifixion of the procreative body for the glorification of the spirit.”
—D.H. (David Herbert)
“The principle that human nature, in its psychological aspects, is nothing more than a product of history and given social relations removes all barriers to coercion and manipulation by the powerful.”
—Noam Chomsky (b. 1928)
“It may be well to remember that the highest level of moral aspiration recorded in history was reached by a few ancient Jews—Micah, Isaiah, and the rest—who took no count whatever of what might not happen to them after death. It is not obvious to me why the same point should not by and by be reached by the Gentiles.”
—Thomas Henry Huxley (1825–95)