History
The invention of "robots.txt" is attributed to Martijn Koster, when working for WebCrawler in 1994. "robots.txt" was then popularized with the advent of AltaVista, and other popular search engines, in the following years.
Read more about this topic: Robots Exclusion Standard
Famous quotes containing the word history:
“Gossip is charming! History is merely gossip. But scandal is gossip made tedious by morality.”
—Oscar Wilde (1854–1900)
“Look through the whole history of countries professing the Romish religion, and you will uniformly find the leaven of this besetting and accursed principle of action—that the end will sanction any means.”
—Samuel Taylor Coleridge (1772–1834)
“When the history of this period is written, [William Jennings] Bryan will stand out as one of the most remarkable men of his generation and one of the biggest political men of our country.”
—William Howard Taft (1857–1930)