Skip to main content

Robots.txt Generator

Generate a robots.txt file for your website to control search engine crawlers.

Written by: Ajay Kumar

Robots.txt Protocol Guide

Written by: Ajay Kumar
| Updated on June 04, 2026

How to Use

  1. Select the Target User-agent: Choose the crawler you want to write rules for. Select the asterisk wildcard `*` to apply the rules to all search engine bots.
  2. Choose the Directive Type: Select "Disallow" to block bots from accessing a path, or "Allow" to whitelist a folder within a disallowed directory.
  3. Specify the Path: Enter the relative directory path. For example, use `/admin/` or `/private-scripts/`.
  4. Include your Sitemap (Optional): Enter the absolute URL of your XML sitemap. This helps crawlers locate and index your pages quickly.
  5. Generate the Script: Click the "Generate robots.txt" button to compile your text.
  6. Deploy the File: Copy the text, paste it into a file named `robots.txt`, and upload it to the root folder of your domain (e.g. `example.com/robots.txt`).

Frequently Asked Question

Q: Is this robots.txt generator free to use?

A: Yes. The Robots.txt Generator is 100% free with no usage limits or premium upgrades. You can generate custom configurations for all your websites without needing to create an account.