Robots Exclusion Standard - History

History

The invention of "robots.txt" is attributed to Martijn Koster, when working for WebCrawler in 1994. "robots.txt" was then popularized with the advent of AltaVista, and other popular search engines, in the following years.

Read more about this topic:  Robots Exclusion Standard

Famous quotes containing the word history:

    “We have need of history in its entirety, not to fall back into it, but to see if we can escape from it.”
    —José Ortega Y Gasset (1883–1955)

    “As I am, so shall I associate, and so shall I act; Caesar’s history will paint out Caesar.”
    —Ralph Waldo Emerson (1803–1882)

    “The myth of independence from the mother is abandoned in mid- life as women learn new routes around the mother—both the mother without and the mother within. A mid-life daughter may reengage with a mother or put new controls on care and set limits to love. But whatever she does, her child’s history is never finished.”
    —Terri Apter (20th century)