History
The invention of "robots.txt" is attributed to Martijn Koster, when working for WebCrawler in 1994. "robots.txt" was then popularized with the advent of AltaVista, and other popular search engines, in the following years.
Read more about this topic: Robots Exclusion Standard
Famous quotes containing the word history:
“We have need of history in its entirety, not to fall back into it, but to see if we can escape from it.”
—José Ortega Y Gasset (18831955)
“As I am, so shall I associate, and so shall I act; Caesars history will paint out Caesar.”
—Ralph Waldo Emerson (18031882)
“The myth of independence from the mother is abandoned in mid- life as women learn new routes around the motherboth the mother without and the mother within. A mid-life daughter may reengage with a mother or put new controls on care and set limits to love. But whatever she does, her childs history is never finished.”
—Terri Apter (20th century)