History
The invention of "robots.txt" is attributed to Martijn Koster, when working for WebCrawler in 1994. "robots.txt" was then popularized with the advent of AltaVista, and other popular search engines, in the following years.
Read more about this topic: Robots Exclusion Standard
Famous quotes containing the word history:
“As History stands, it is a sort of Chinese Play, without end and without lesson.”
—Henry Brooks Adams (1838–1918)
“Gossip is charming! History is merely gossip. But scandal is gossip made tedious by morality.”
—Oscar Wilde (1854–1900)
“Considered in its entirety, psychoanalysis won’t do. It’s an end product, moreover, like a dinosaur or a zeppelin; no better theory can ever be erected on its ruins, which will remain for ever one of the saddest and strangest of all landmarks in the history of twentieth-century thought.”
—Peter B. Medawar (1915–1987)