History
The invention of "robots.txt" is attributed to Martijn Koster, when working for WebCrawler in 1994. "robots.txt" was then popularized with the advent of AltaVista, and other popular search engines, in the following years.
Read more about this topic: Robots Exclusion Standard
Famous quotes containing the word history:
“The visual is sorely undervalued in modern scholarship. Art history has attained only a fraction of the conceptual sophistication of literary criticism.... Drunk with self-love, criticism has hugely overestimated the centrality of language to western culture. It has failed to see the electrifying sign language of images.”
—Camille Paglia (b. 1947)
“The one duty we owe to history is to rewrite it.”
—Oscar Wilde (1854–1900)
“To care for the quarrels of the past, to identify oneself passionately with a cause that became, politically speaking, a losing cause with the birth of the modern world, is to experience a kind of straining against reality, a rebellious nonconformity that, again, is rare in America, where children are instructed in the virtues of the system they live under, as though history had achieved a happy ending in American civics.”
—Mary McCarthy (1912–1989)