History
The invention of "robots.txt" is attributed to Martijn Koster, when working for WebCrawler in 1994. "robots.txt" was then popularized with the advent of AltaVista, and other popular search engines, in the following years.
Read more about this topic: Robots Exclusion Standard
Famous quotes containing the word history:
“The history of persecution is a history of endeavors to cheat nature, to make water run up hill, to twist a rope of sand.”
—Ralph Waldo Emerson (1803–1882)
“American time has stretched around the world. It has become the dominant tempo of modern history, especially of the history of Europe.”
—Harold Rosenberg (1906–1978)
“I believe that history has shape, order, and meaning; that exceptional men, as much as economic forces, produce change; and that passé abstractions like beauty, nobility, and greatness have a shifting but continuing validity.”
—Camille Paglia (b. 1947)