History
The invention of "robots.txt" is attributed to Martijn Koster, when working for WebCrawler in 1994. "robots.txt" was then popularized with the advent of AltaVista, and other popular search engines, in the following years.
Read more about this topic: Robots Exclusion Standard
Famous quotes containing the word history:
“Let it suffice that in the light of these two facts, namely, that the mind is One, and that nature is its correlative, history is to be read and written.”
—Ralph Waldo Emerson (1803–1882)
“The principle office of history I take to be this: to prevent virtuous actions from being forgotten, and that evil words and deeds should fear an infamous reputation with posterity.”
—Tacitus (c. 55–117)
“Classes struggle, some classes triumph, others are eliminated. Such is history; such is the history of civilization for thousands of years.”
—Mao Zedong (1893–1976)