Robots Exclusion Standard - Examples

Examples

This example tells all robots to visit all files because the wildcard * specifies all robots:

User-agent: * Disallow:

This example tells all robots to stay out of a website:

User-agent: * Disallow: /

The next is an example that tells all robots not to enter four directories of a website:

User-agent: * Disallow: /cgi-bin/ Disallow: /images/ Disallow: /tmp/ Disallow: /private/

Example that tells a specific robot not to enter one specific directory:

User-agent: BadBot # replace 'BadBot' with the actual user-agent of the bot Disallow: /private/

Example that tells all robots not to enter one specific file:

User-agent: * Disallow: /directory/file.html

Note that all other files in the specified directory will be processed.

Example demonstrating how comments can be used:

# Comments appear after the "#" symbol at the start of a line, or after a directive User-agent: * # match all bots Disallow: / # keep them out

It is also possible to list multiple robots with their own rules. The actual robot string is defined by the crawler. A few sites, such as Google, support several user-agent strings that allow you to turn off a subset of their services by using specific user-agent strings.

Example demonstrating multiple user-agents:

User-agent: googlebot # all services Disallow: /private # disallow this directory User-agent: googlebot-news # only the news service Disallow: / # on everything User-agent: * # all robots Disallow: /something # on this folder

Read more about this topic:  Robots Exclusion Standard

Famous quotes containing the word examples:

    “It is hardly to be believed how spiritual reflections when mixed with a little physics can hold people’s attention and give them a livelier idea of God than do the often ill-applied examples of his wrath.”
    —G.C. (Georg Christoph)

    “In the examples that I here bring in of what I have [read], heard, done or said, I have refrained from daring to alter even the smallest and most indifferent circumstances. My conscience falsifies not an iota; for my knowledge I cannot answer.”
    —Michel de Montaigne (1533–1592)

    “Histories are more full of examples of the fidelity of dogs than of friends.”
    —Alexander Pope (1688–1744)