History
The invention of "robots.txt" is attributed to Martijn Koster, when working for WebCrawler in 1994. "robots.txt" was then popularized with the advent of AltaVista, and other popular search engines, in the following years.
Read more about this topic: Robots Exclusion Standard
Famous quotes containing the word history:
“History is not what you thought. It is what you can remember. All other history defeats itself.
In Beverly Hills ... they don’t throw their garbage away. They make it into television shows.
Idealism is the despot of thought, just as politics is the despot of will.”
—Mikhail Bakunin (1814–1876)
“There is a history in all men’s lives,
Figuring the natures of the times deceased,
The which observed, a man may prophesy,
With a near aim, of the main chance of things
As yet not come to life.”
—William Shakespeare (1564–1616)
“A country grows in history not only because of the heroism of its troops on the field of battle, it grows also when it turns to justice and to right for the conservation of its interests.”
—Aristide Briand (1862–1932)