Examples
This example tells all robots to visit all files because the wildcard *
specifies all robots:
This example tells all robots to stay out of a website:
User-agent: * Disallow: /The next is an example that tells all robots not to enter four directories of a website:
User-agent: * Disallow: /cgi-bin/ Disallow: /images/ Disallow: /tmp/ Disallow: /private/Example that tells a specific robot not to enter one specific directory:
User-agent: BadBot # replace 'BadBot' with the actual user-agent of the bot Disallow: /private/Example that tells all robots not to enter one specific file:
User-agent: * Disallow: /directory/file.htmlNote that all other files in the specified directory will be processed.
Example demonstrating how comments can be used:
# Comments appear after the "#" symbol at the start of a line, or after a directive User-agent: * # match all bots Disallow: / # keep them outIt is also possible to list multiple robots with their own rules. The actual robot string is defined by the crawler. A few sites, such as Google, support several user-agent strings that allow you to turn off a subset of their services by using specific user-agent strings.
Example demonstrating multiple user-agents:
User-agent: googlebot # all services Disallow: /private # disallow this directory User-agent: googlebot-news # only the news service Disallow: / # on everything User-agent: * # all robots Disallow: /something # on this folderRead more about this topic: Robots Exclusion Standard
Famous quotes containing the word examples:
“It is hardly to be believed how spiritual reflections when mixed with a little physics can hold peoples attention and give them a livelier idea of God than do the often ill-applied examples of his wrath.”
—G.C. (Georg Christoph)
“In the examples that I here bring in of what I have [read], heard, done or said, I have refrained from daring to alter even the smallest and most indifferent circumstances. My conscience falsifies not an iota; for my knowledge I cannot answer.”
—Michel de Montaigne (15331592)
“There are many examples of women that have excelled in learning, and even in war, but this is no reason we should bring em all up to Latin and Greek or else military discipline, instead of needle-work and housewifry.”
—Bernard Mandeville (16701733)