Robot exclusion protocol • Crawl configuration • 2026 edition
Basic Structure:
User-agent: *
Disallow: /private/
Allow: /public/
Crawl-delay: 10
Sitemap: https://example.com/sitemap.xml
Where:
Robots.txt follows the Robot Exclusion Protocol to control web crawler access.
Robots.txt is a text file that follows the Robot Exclusion Protocol to tell web robots which areas of your site they can access. It's placed in the root directory of your website and provides instructions to search engine crawlers about which pages or sections to exclude from indexing.
Essential robots.txt directives include:
Each directive serves a specific purpose in controlling crawler behavior.
Typical robots.txt configurations:
Must be placed at https://example.com/robots.txt (root directory).
Plain text file with UTF-8 encoding, maximum 500KB size.
Use wildcards: /path/* matches all subdirectories.
Where must the robots.txt file be located to be recognized by web crawlers?
The answer is B) At the root directory (https://example.com/robots.txt). The robots.txt file must be placed in the root directory of your website to be recognized by web crawlers. This is a fundamental requirement of the Robot Exclusion Protocol. Crawlers will specifically look for the file at https://yoursite.com/robots.txt, and placing it anywhere else will make it inaccessible to most search engine robots.
This question emphasizes a critical requirement of the robots.txt protocol. The location is standardized because search engine crawlers follow a predictable pattern when looking for the robots.txt file. They always check the root directory first, and if they don't find it there, they won't continue searching elsewhere. This standardization ensures consistency across the web and prevents confusion about where crawlers should look for crawling instructions.
Root Directory: The top-level directory of a website, accessible at the domain root
Robot Exclusion Protocol: Standard for controlling web robot behavior
Crawler Recognition: Ability of web crawlers to locate and parse robots.txt
• Must be in root directory: /robots.txt
• Accessible via: https://domain.com/robots.txt
• Follows standardized protocol
• Always place in root directory
• Test accessibility with browser
• Use search console to verify
• Placing robots.txt in subdirectories
• Using incorrect file permissions
• Not testing file accessibility
Which of the following robots.txt directives correctly blocks Googlebot from accessing the /admin/ directory while allowing access to the rest of the site?
The correct directive is:
User-agent: Googlebot
Disallow: /admin/
This specifically targets Google's crawler and blocks access to the /admin/ directory while leaving all other paths accessible. The blank line after the User-agent line is optional but recommended for readability.
This demonstrates the specificity feature of robots.txt. You can target individual crawlers (like Googlebot, Bingbot) with their own set of rules. The syntax requires a User-agent line followed by the directives that apply to that specific crawler. When you specify a particular bot, other bots will follow different rules or the default rules for "all crawlers" (User-agent: *). This allows for fine-grained control over different search engines' access.
User-agent: Specifies which crawler the rules apply to
Specific Targeting: Applying rules to individual crawlers
Default Rules: Rules that apply to all crawlers (*)
• Each User-agent section is independent
• Specific rules override general ones
• Case-sensitive user-agent names
• Use * for all crawlers
• Be specific with bot names
• Group related directives together
• Using incorrect user-agent names
• Mixing rules for different agents
• Not understanding scope of rules
An e-commerce website wants to block search engine crawlers from accessing the following sensitive paths: /admin/, /cart/, /checkout/, and /orders/. However, they want to allow access to the /api/public/ directory for legitimate API access. Write the robots.txt directives that accomplish this, and explain the order of precedence for Allow and Disallow directives.
The robots.txt directives would be:
User-agent: *
Disallow: /admin/
Disallow: /cart/
Disallow: /checkout/
Disallow: /orders/
Allow: /api/public/
Regarding precedence: In most implementations, Allow directives take precedence over Disallow directives when they conflict. If a path matches both an Allow and a Disallow rule, the Allow rule wins. However, the order can matter in some implementations, so it's generally safer to place more specific rules (like Allow) after general rules (like Disallow).
This demonstrates the practical application of robots.txt for e-commerce sites, which often have sensitive areas that shouldn't be indexed. The Allow directive is particularly useful for overriding more general Disallow rules. The precedence rule (Allow overriding Disallow) is important because it allows you to make exceptions to general blocking rules. This is especially valuable when you want to block a broad area but still allow access to specific parts of it.
Precedence: The order of importance when rules conflict
Override: One rule taking priority over another
Path Blocking: Preventing access to specific directories
• Allow overrides Disallow in conflicts
• More specific paths take precedence
• Order can matter in some implementations
• Place specific rules after general ones
• Use Allow to create exceptions
• Test rules with search console
• Not understanding Allow/Disallow precedence
• Blocking important content accidentally
• Not testing the configuration
A high-traffic news website publishes 100 articles per hour and has limited server resources. They want to control crawl rate to prevent server overload while ensuring fresh content gets indexed promptly. What robots.txt configuration would best balance these requirements, and how would the crawl-delay directive impact different search engines?
Optimal robots.txt configuration:
User-agent: *
Crawl-delay: 5
Disallow: /temp/
Disallow: /drafts/
The Crawl-delay directive specifies minimum time (in seconds) between successive requests from the same crawler. However, support varies by search engine:
For Google specifically, use Search Console to manage crawl rate rather than relying on Crawl-delay.
This example highlights the reality that different search engines implement robots.txt directives differently. While Crawl-delay is part of the standard, not all major engines respect it. This demonstrates why it's important to understand the specific behaviors of different search engines rather than assuming universal compliance with the protocol. For Google, which handles the majority of searches, alternative methods like Search Console are more effective for crawl rate management.
Crawl-delay: Minimum time between crawler requests
Server Load: Resource consumption from crawling activity
Implementation Variance: Different behavior across search engines• Support varies by search engine
• Google ignores Crawl-delay
• Use Search Console for Google
• Understand each engine's behavior
• Use Search Console for Google control
• Monitor server logs for crawl activity
• Assuming all engines respect all directives
• Relying solely on Crawl-delay for Google
• Not monitoring actual crawl behavior
Which of the following statements about robots.txt limitations is TRUE?
The answer is B) Malicious bots and scrapers may ignore robots.txt entirely. Robots.txt is a voluntary protocol that ethical search engines follow, but malicious bots, scrapers, and other unauthorized crawlers can and often do ignore these directives. For true security and privacy, additional measures like password protection, IP restrictions, or the noindex meta tag should be used instead of or in addition to robots.txt.
This question addresses a common misconception about robots.txt. It's important to understand that robots.txt is a guideline, not a security mechanism. It works on an honor system with reputable search engines, but provides no technical barrier to access. This is why sensitive information should never rely solely on robots.txt for protection. The protocol is designed to help well-behaved crawlers respect website owners' wishes, not to enforce access controls.
Honor System: Voluntary compliance without enforcement
Security Mechanism: Technical protection method
Access Control: Methods to restrict resource access
• Not a security tool
• Voluntary compliance
• Use additional protections for sensitive content
• Use noindex for private content
• Implement authentication for sensitive areas
• Combine with other security measures
• Using robots.txt as security measure
• Expecting complete privacy protection
• Not implementing additional safeguards
Q: Does the Crawl-delay directive work with all search engines in 2026?
A: No, Crawl-delay support varies significantly between search engines:
For Google, which handles the majority of searches, configure crawl rate through Google Search Console rather than relying on the robots.txt Crawl-delay directive. This is why it's important to understand that different search engines implement the robots.txt protocol differently.
Q: Should I use robots.txt or the noindex meta tag to prevent pages from being indexed?
A: Use different methods for different purposes:
Robots.txt:
Noindex Meta Tag:
For maximum control, use robots.txt to prevent crawling and noindex for pages that might be discovered through other means.