History
The invention of "robots.txt" is attributed to Martijn Koster, when working for WebCrawler in 1994. "robots.txt" was then popularized with the advent of AltaVista, and other popular search engines, in the following years.
Read more about this topic: Robots Exclusion Standard
Famous quotes containing the word history:
“What is most interesting and valuable in it, however, is not the materials for the history of Pontiac, or Braddock, or the Northwest, which it furnishes; not the annals of the country, but the natural facts, or perennials, which are ever without date. When out of history the truth shall be extracted, it will have shed its dates like withered leaves.”
—Henry David Thoreau (1817–1862)
“To history therefore I must refer for answer, in which it would be an unhappy passage indeed, which should shew by what fatal indulgence of subordinate views and passions, a contest for an atom had defeated well founded prospects of giving liberty to half the globe.”
—Thomas Jefferson (1743–1826)
“Properly speaking, history is nothing but the crimes and misfortunes of the human race.”
—Pierre Bayle (1647–1706)