History
The invention of "robots.txt" is attributed to Martijn Koster, when working for WebCrawler in 1994. "robots.txt" was then popularized with the advent of AltaVista, and other popular search engines, in the following years.
Read more about this topic: Robots Exclusion Standard
Famous quotes containing the word history:
“It gives me the greatest pleasure to say, as I do from the bottom of my heart, that never in the history of the country, in any crisis and under any conditions, have our Jewish fellow citizens failed to live up to the highest standards of citizenship and patriotism.”
—William Howard Taft (1857–1930)
“What we call National-Socialism is the poisonous perversion of ideas which have a long history in German intellectual life.”
—Thomas Mann (1875–1955)
“In the history of the United States, there is no continuity at all. You can cut through it anywhere and nothing on this side of the cut has anything to do with anything on the other side.”
—Henry Brooks Adams (1838–1918)