History
The invention of "robots.txt" is attributed to Martijn Koster, when working for WebCrawler in 1994. "robots.txt" was then popularized with the advent of AltaVista, and other popular search engines, in the following years.
Read more about this topic: Robots Exclusion Standard
Famous quotes containing the word history:
“In the history of the human mind, these glowing and ruddy fables precede the noonday thoughts of men, as Aurora the sun’s rays. The matutine intellect of the poet, keeping in advance of the glare of philosophy, always dwells in this auroral atmosphere.”
—Henry David Thoreau (1817–1862)
“A great proportion of the inhabitants of the Cape are always thus abroad about their teaming on some ocean highway or other, and the history of one of their ordinary trips would cast the Argonautic expedition into the shade.”
—Henry David Thoreau (1817–1862)
“I believe my ardour for invention springs from his loins. I can’t say that the brassiere will ever take as great a place in history as the steamboat, but I did invent it.”
—Caresse Crosby (1892–1970)