Where do I upload my robots.txt file?

Upload the file to the root directory of your website so it is accessible at https://yourdomain.com/robots.txt.

Why do I need to allow Mediapartners-Google?

Mediapartners-Google is Google's contextual advertising crawler. If this crawler is blocked by robots.txt, Google AdSense cannot evaluate your page content to display high-paying relevant ads or approve your site.

Should I allow or block AI search crawlers like GPTBot?

If you want your website to be cited and recommended in ChatGPT Search and Perplexity AI answers (Generative Engine Optimization), you should allow GPTBot and PerplexityBot.

Can robots.txt completely hide a page from Google?

No. Robots.txt prevents crawling, but if other websites link to that URL, Google may still index the bare link. To guarantee a page is never indexed, use a 'noindex' meta tag.

How do I check if my robots.txt is working properly?

You can visit https://yourdomain.com/robots.txt directly in any web browser or use the Google Search Console robots.txt inspection tool.

Robots.txt & Crawler Directives Generator

Build and validate robots.txt rules for Googlebot, Bingbot, AdSense, and Generative AI engines.

Quick Presets:

XML Sitemap URL

Crawl Delay (Seconds, optional)

Specialized Crawler Settings

Explicitly Allow Google AdSense Bot (Mediapartners-Google)Mandatory for Google AdSense site approval and contextual ads revenue.

Allow AI Search Bots for GEO (GPTBot, PerplexityBot, ClaudeBot)Enables ChatGPT Search and Perplexity to index your pages for AI answer citations.

Disallowed Paths (4)

/dashboard//admin//api//auth/verify

Live robots.txt OutputUTF-8 Plain Text

# robots.txt generated by InstantTool SEO Suite
# Compliant with Googlebot, Bingbot, AdSense & AI Search Crawlers

User-agent: *
Allow: /
Disallow: /dashboard/
Disallow: /admin/
Disallow: /api/
Disallow: /auth/verify

# Allow Google AdSense bot for ad verification
User-agent: Mediapartners-Google
Allow: /

# Allow AI search engines for Generative Engine Optimization (GEO)
User-agent: GPTBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: ClaudeBot
Allow: /

Sitemap: https://devigo.cloud/sitemap.xml

How to Install Your robots.txt File:

1. Download the generated file using the button above.

2. Upload it to the root directory of your web server (e.g. public/robots.txt or via FTP to /public_html/robots.txt).

3. Verify by visiting https://yourdomain.com/robots.txt in your browser.

Google AdSense Crawler Toggle (Mediapartners-Google)

AI Search Engine Bots Support (GPTBot, Perplexity, ClaudeBot)

Disallowed Paths & Directory Access Management

Instant File Download & Webmaster Installation Guide

Protect Sensitive Directories & Save Crawl Budget

Prevent search engines from indexing internal admin portals, checkout pages, or staging environments, ensuring crawlers focus exclusively on your high-value content.

Mandatory for Google AdSense Approval

AdSense uses the Mediapartners-Google crawler to review webpage content before approving ads. Our tool ensures this bot is explicitly allowed so your monetization approval is smooth.

Control Generative AI Crawlers (GEO Optimization)

Choose whether to allow next-generation search bots like GPTBot and PerplexityBot to index your site for AI answer citations, or block scrapers to protect content.

How to use Robots.txt & Crawler Directives Generator

1
Enter your site's full XML sitemap URL (e.g. https://yourdomain.com/sitemap.xml).
2
Add restricted directory paths you want to keep private (e.g. /admin/, /api/, /cart).
3
Toggle crawler permissions for Google AdSense and AI Answer Engine bots.
4
Optionally specify a crawl delay if your server has bandwidth constraints.
5
Click 'Download robots.txt' and upload the file to your website's root directory.

Everything you need in a Robots.txt & Crawler Directives Generator tool

Googlebot & Bingbot Rules

Standard directives for primary global search engines.

AdSense Verification

Dedicated Mediapartners-Google allowance rule.

AI Search Engine Toggles

Enable or restrict GPTBot, PerplexityBot, and ClaudeBot.

Path Customization

Add unlimited custom disallow paths with 1-click removal.

Crawl Delay Settings

Set throttles to prevent server overload.

Direct Download

Generates pure UTF-8 robots.txt ready for root domain deployment.

Robots.txt & Crawler Directives Generator FAQs

Robots.txt is a plain text file placed in your website's root folder that instructs search engine web crawlers which sections of your site they can or cannot visit.

Rate this tool

★★★★★

4.8 / 5

Buy Us a Coffee

Support InstantTool & Keep Tools Free

We build free multimedia, PDF, and SEO tools for everyone. A small contribution helps us cover server infrastructure and maintain high-speed processing.

Secure Razorpay•100% Voluntary•Community Supported

Select Contribution:₹100

₹

256-bit encrypted Razorpay checkout