Robots Exclusion Standard - History

History

The invention of "robots.txt" is attributed to Martijn Koster, when working for WebCrawler in 1994. "robots.txt" was then popularized with the advent of AltaVista, and other popular search engines, in the following years.

Read more about this topic:  Robots Exclusion Standard

Famous quotes containing the word history:

    “Social history might be defined negatively as the history of a people with the politics left out.”
    —G.M. (George Macaulay)

    “Both place and time were changed, and I dwelt nearer to those parts of the universe and to those eras in history which had most attracted me.”
    —Henry David Thoreau (1817–1862)

    “It takes a great deal of history to produce a little literature.”
    —Henry James (1843–1916)