What is a robots.txt file?

A robots.txt file is a plain text file placed at the root of your website that tells search engine crawlers which pages or sections of your site they can or cannot access. It follows the Robots Exclusion Protocol and is one of the first files a crawler looks for when visiting your domain.

Where do I put robots.txt?

The robots.txt file must be placed at the root of your domain. For example, if your website is https://example.com, the file should be accessible at https://example.com/robots.txt. It will not work if placed in a subdirectory.

Can robots.txt block pages from Google?

Robots.txt can prevent Googlebot from crawling pages, but it does not guarantee those pages won't appear in search results. If other pages link to a disallowed page, Google may still index the URL (without content). To fully block a page from search results, use a noindex meta tag or X-Robots-Tag HTTP header instead.

Crawl-delay is a directive that tells bots to wait a specified number of seconds between successive requests to your server. This can help reduce server load if you have limited resources. Note that Google does not officially support crawl-delay; instead, you can set crawl rate in Google Search Console.

Should I block CSS and JS files?

No. Google recommends allowing access to CSS and JavaScript files so that Googlebot can render your pages properly. Blocking these resources can prevent Google from understanding your page layout, which may negatively impact your search rankings.

+91 93093 80958

All Tools

Free Tool

Robots.txt Generator

Create a properly formatted robots.txt file for your website in seconds. Configure crawl rules, add your sitemap, and download the ready-to-use file.

User Agent

Crawl Rules

Sitemap URL (optional)

Crawl Delay (seconds, optional)

Quick Presets

Live Preview

User-agent: *
Disallow: /admin/
Disallow: /api/

Upload this file to the root of your domain (e.g. https://yourdomain.com/robots.txt).

What is Robots.txt?

A robots.txt file is a standard text file that sits at the root of your website and communicates with search engine crawlers. It acts as a gatekeeper, telling bots like Googlebot and Bingbot which parts of your site they are allowed to crawl and which they should avoid. Every major search engine respects the Robots Exclusion Protocol, making this file a fundamental part of technical SEO.

While robots.txt does not directly impact rankings, it plays a critical role in how search engines discover and access your content. By properly configuring your robots.txt, you can prevent crawlers from wasting resources on irrelevant pages, protect sensitive directories, and help search engines focus on the content that matters most to your visibility.

Control Crawling

Tell search engines exactly which pages and directories to crawl or skip, giving you full control over how bots interact with your site.

Protect Private Pages

Keep admin panels, staging environments, and internal tools hidden from search engine indexes by blocking crawler access.

Optimize Crawl Budget

Ensure search engines spend their crawl budget on your most important pages instead of wasting it on low-value or duplicate content.

Common Robots.txt Rules

Disallow: /

Blocks all crawlers from the entire site

Allow: /

Allows all crawlers to access the entire site

Disallow: /admin/

Blocks access to the admin directory

Disallow: /api/

Prevents crawling of API endpoints

Sitemap: https://example.com/sitemap.xml

Points crawlers to your XML sitemap

Crawl-delay: 10

Asks bots to wait 10 seconds between requests

Frequently Asked Questions

Need Full Technical SEO?

Robots.txt is just one piece of the puzzle. Let our team handle your entire technical SEO strategy for maximum search visibility.