Robots Exclusion Standard
The Robot Exclusion Standard, also known as the Robots Exclusion Protocol or robots.txt protocol, is a convention to prevent cooperating web crawlers and other web robots from accessing all or part of a website which is otherwise publicly viewable. Robots are often used by search engines to categorize and archive web sites, or by webmasters to proofread source code. The standard is different from, but can be used in conjunction with, Sitemaps, a robot inclusion standard for websites.
Read more about Robots Exclusion Standard: History, About The Standard, Disadvantages, Alternatives, Examples
Famous quotes containing the words robots, exclusion and/or standard:
“The danger of the past was that men became slaves. The danger of the future is that men may become robots. True enough, robots do not rebel. But given man’s nature, robots cannot live and remain sane, they become “Golems,” they will destroy their world and themselves because they cannot stand any longer the boredom of a meaningless life.”
—Erich Fromm (1900–1980)
“We belong to the community. It is not the tailor alone who is the ninth part of a man; it is as much the preacher, and the merchant, and the farmer. Where is this division of labor to end? and what object does it finally serve? No doubt another may also think for me; but it is not therefore desirable that he should do so to the exclusion of my thinking for myself.”
—Henry David Thoreau (1817–1862)
“I find it interesting that the meanest life, the poorest existence, is attributed to God’s will, but as human beings become more affluent, as their living standard and style begin to ascend the material scale, God descends the scale of responsibility at a commensurate speed.”
—Maya Angelou (b. 1928)