What SEO tools does Texterfly offer?

Texterfly offers 11 free SEO and content tools: Robots.txt Generator, XML Sitemap Generator, Schema Markup Generator, Meta Tag Generator, Content Optimizer, Advanced Readability & Content Grader, Keyword Research Tool, Keyword Density Checker, SEO Analyzer, Alt Text Generator, and Privacy Policy Generator. All are free, browser-based, and require no signup.

What is the ideal meta title length for SEO?

Google displays meta titles up to approximately 600px wide, which corresponds to roughly 50–60 characters. Titles shorter than 50 characters may miss keyword opportunities; titles longer than 60 characters are truncated in search results with an ellipsis. The Meta Tag Generator includes a live character counter and pixel-width preview.

What is the ideal meta description length?

Google typically shows meta descriptions of 150–160 characters in search results (up to about 920px). Longer descriptions are truncated. Meta descriptions do not directly affect rankings, but they influence click-through rate (CTR). Write descriptions that include the primary keyword and a clear call to action.

What should a robots.txt file contain?

A valid robots.txt file uses User-agent directives to specify which crawlers the rules apply to, Disallow directives to block specific paths, Allow directives to override Disallow rules for subpaths, a Sitemap directive pointing to your sitemap.xml URL, and optionally a Crawl-delay to throttle bot crawl frequency. Example: "User-agent: * / Disallow: /admin/ / Sitemap: https://example.com/sitemap.xml".

What is schema markup and why does it matter for SEO?

Schema markup (structured data) is JSON-LD or Microdata code added to your HTML that helps search engines understand your content. It can unlock rich results in Google Search — such as star ratings, FAQ dropdowns, breadcrumbs, event dates, and How-To steps — which increase visibility and CTR without directly affecting rankings. The Schema Markup Generator produces ready-to-paste JSON-LD for 15+ schema types.

What is keyword density and what is the ideal percentage?

Keyword density is the percentage of times a keyword appears relative to the total word count. The formula is: (keyword occurrences ÷ total words) × 100. A density of 1–3% is generally considered optimal. Below 1% may signal that the topic is not well covered; above 3% risks keyword stuffing, which can trigger Google penalties. The Keyword Density Checker calculates this instantly.

Why is alt text important for SEO?

Alt text (alternative text) serves two SEO purposes: it describes images to search engine crawlers, helping images rank in Google Images; and it makes content accessible to screen readers used by visually impaired users. Good alt text is descriptive, includes the primary keyword naturally where relevant, and is typically 8–15 words. The Alt Text Generator analyses image content and suggests optimised alt text.

What is an XML sitemap and how does it help SEO?

An XML sitemap is a file (sitemap.xml) that lists all important URLs on your website with optional metadata (lastmod, changefreq, priority). It helps search engines discover pages they might miss through crawling — especially new pages, orphaned pages, and pages with few internal links. Submitting it to Google Search Console can significantly speed up indexing.

What is the Flesch-Kincaid readability score?

The Flesch-Kincaid Reading Ease score rates text on a 0–100 scale. Higher scores mean easier reading: 90–100 is very easy (5th grade); 60–70 is standard (8th–9th grade); 30–50 is difficult (college level); 0–30 is very difficult (professional). Most general web content should aim for 60–70. The Readability Grader calculates Flesch-Kincaid and four other readability formulas.

What schema markup types does the generator support?

The Schema Markup Generator supports: Article, BlogPosting, Product, Review, FAQPage, HowTo, Recipe, Event, LocalBusiness, Organization, Person, BreadcrumbList, WebSite (with SearchAction), VideoObject, and WebApplication. Each generates valid JSON-LD that can be tested in Google's Rich Results Test.

Are these SEO tools free?

Yes. All 11 SEO and content tools on Texterfly are 100% free with no registration, subscription, or payment required.

Is my content private when using these tools?

Yes. All processing runs entirely in your browser using JavaScript. Your content, URLs, and data are never sent to any server or stored anywhere.

What is a robots.txt generator?

A robots.txt generator is a free online tool that creates a correctly formatted robots.txt file by letting you configure rules through a visual interface. It eliminates formatting errors that could accidentally block Googlebot from your site.

Can I block AI bots like ChatGPT from my website using robots.txt?

Yes. OpenAI's GPTBot, Anthropic's ClaudeBot, Google-Extended, CCBot, and others all respect robots.txt. Add User-agent: GPTBot followed by Disallow: / to block them. Use the AI Crawlers section in this tool to select and block them all at once.

Does robots.txt stop my page from appearing in Google?

No. robots.txt prevents crawling but not indexing. If your page is linked to from external sites, Google may still index it. To remove a page from search results, use a noindex meta tag instead.

How do I create a robots.txt file for WordPress?

Click the WordPress preset button. It automatically configures the standard rules: blocking /wp-admin/ and /wp-includes/ while allowing /wp-admin/admin-ajax.php for AJAX functionality.

Where should I put my robots.txt file?

Place it in your website's root directory, accessible at https://yourdomain.com/robots.txt. It cannot be in a subdirectory. Each subdomain needs its own separate robots.txt file.

Is robots.txt case sensitive?

Yes. Directives must be capitalised correctly (User-agent, Allow, Disallow, Sitemap). Path values are also case-sensitive on Linux servers — /Admin is different from /admin.

llms.txt is an emerging standard file that tells AI language models how they may use your website's content — whether training is permitted and how to contact you for licensing. Place it at yourdomain.com/llms.txt alongside your robots.txt.

How long does it take for Google to see robots.txt changes?

Google caches robots.txt for up to 24 hours. Force an immediate re-read using the robots.txt tester in Google Search Console.

Should I disallow CSS and JavaScript files?

No. Google needs to render your CSS and JavaScript to understand your page layout. Blocking them causes ranking drops. Always leave CSS and JS files fully accessible to crawlers.

Back to SEO Content Tools

Robots.txt Generator

Generate, validate, and download a robots.txt file for any platform. Configure rules for 20+ bots including all major search engines and AI training crawlers. Includes a bonus llms.txt generator.

All Major BotsAI Bot BlockingLive Validation6 Presetsllms.txt GeneratorDownload File100% Free

Support Our Free Tools

If you find this calculator helpful, please consider supporting our work. Your contribution helps us build and maintain these free tools for everyone.

Buy me a coffee

What Is a robots.txt File?

A robots.txt file is a plain-text file placed at the root of your domain that tells web crawlers which pages or directories they can and cannot access. It is the first file any well-behaved bot reads when it visits your site. Getting it right is one of the most impactful technical SEO tasks you can do.

# Minimal valid robots.txt

User-agent: *

Allow: /

Disallow: /admin/

Sitemap: https://yourdomain.com/sitemap.xml

🔍

Control Crawl Budget

Tell Googlebot not to waste time crawling admin panels, staging pages, or search result pages — so it spends more time on your important content.

🤖

Block AI Training Bots

GPTBot, CCBot, Claude, Gemini — AI companies send crawlers to train their models on your content. Block them individually or all at once.

🗺️

Declare Your Sitemap

Adding a Sitemap: line helps all secondary crawlers discover your content structure without needing to submit to each search engine separately.

AI Bot Reference — User-Agent Strings

Use these exact strings in your robots.txt User-agent: directives to target specific AI crawlers.

User-agent	Company	Purpose	Block with
GPTBot	OpenAI	ChatGPT training data	Disallow: /
ChatGPT-User	OpenAI	ChatGPT browsing plugin	Disallow: /
CCBot	Common Crawl	AI training datasets	Disallow: /
anthropic-ai	Anthropic	Claude model training	Disallow: /
ClaudeBot	Anthropic	Claude crawling	Disallow: /
Google-Extended	Google	Bard/Gemini AI training	Disallow: /
PerplexityBot	Perplexity	AI search training	Disallow: /
Bytespider	ByteDance	TikTok AI training	Disallow: /
FacebookBot	Meta	Meta AI model training	Disallow: /
cohere-ai	Cohere	Cohere model training	Disallow: /

robots.txt Syntax Reference

Directive	Example	Meaning
User-agent: *	User-agent: *	Applies to all crawlers
User-agent: Googlebot	User-agent: Googlebot	Applies only to Googlebot
Disallow: /path/	Disallow: /admin/	Block this path and everything under it
Disallow: /	Disallow: /	Block the entire site
Disallow:	Disallow:	Empty = allow everything (same as Allow: /)
Allow: /path/	Allow: /public/	Explicitly allow (overrides a broader Disallow)
Disallow: /?	Disallow: /?filter=*	Wildcard — blocks URLs with any query param
Disallow: /*.pdf$	Disallow: /*.pdf$	$ = end of URL — blocks all .pdf files
Crawl-delay: N	Crawl-delay: 10	Wait N seconds between requests (not Google)
Sitemap: URL	Sitemap: https://x.com/sitemap.xml	Declare your XML sitemap location

How to Deploy Your robots.txt

1
Generate and download
Use the generator above to configure your rules, then click Download to get your robots.txt file.
2
Upload to your site root
Place the file in your website's root directory — the same folder that contains your index.html or homepage. It must be accessible at yourdomain.com/robots.txt (not /assets/robots.txt).
3
Verify it's live
Open your browser and navigate to https://yourdomain.com/robots.txt. You should see the plain text content of your file.
4
Notify Google
Go to Google Search Console → Settings → robots.txt and click Submit. This tells Google to re-read your file immediately instead of waiting for its ~24hr cache.

Frequently Asked Questions

What is a robots.txt generator?: A robots.txt generator is a free online tool that creates a correctly formatted robots.txt file by letting you configure rules through a visual interface rather than writing the syntax by hand. It eliminates formatting errors that could accidentally block Googlebot from your site.
Does robots.txt stop my page from appearing in Google?: No — robots.txt prevents crawling, not indexing. If your page is linked to from external sites, Google may still index it without crawling it. To guarantee removal from search results, use a noindex meta tag or X-Robots-Tag HTTP header instead.
Can I block AI bots like ChatGPT from my website?: Yes. OpenAI's GPTBot, Anthropic's ClaudeBot, Google-Extended, CCBot, and others all respect robots.txt. Use the AI Crawlers section above to select which ones to block with Disallow: /.
What is the difference between robots.txt and noindex?: robots.txt controls access (crawling) — it tells bots whether they can visit a URL. noindex controls indexing — it tells Google not to show the page in search results. A page can be crawled but not indexed (noindex), or blocked from crawling but still indexed if linked from elsewhere.
Where exactly should I put my robots.txt file?: It must be in your website's root directory, accessible at https://yourdomain.com/robots.txt. It cannot be in a subdirectory. Each subdomain needs its own robots.txt (e.g. blog.yourdomain.com/robots.txt).
Is robots.txt case sensitive?: Yes. Directives (User-agent, Allow, Disallow, Sitemap) must be capitalised exactly as shown. Path values are also case-sensitive on Linux servers — /Admin is different from /admin.
What is llms.txt and should I create one?: llms.txt is an emerging standard that declares to AI language models how they may use your content — whether training is permitted and who to contact for licensing. It is not yet universally enforced but is increasingly respected by major AI providers. Creating one now establishes your rights posture early.
How long does it take for Google to see robots.txt changes?: Google caches robots.txt for up to 24 hours. You can force an immediate re-read by using the Robots.txt tester in Google Search Console and clicking Submit.
Should I disallow CSS and JavaScript files?: No — never. Google needs to render your CSS and JavaScript to understand your page layout and content. Blocking these files was a common mistake years ago and is still seen causing ranking drops. Modern robots.txt best practice is to leave all CSS/JS files fully accessible.
What is Crawl-delay and does Google respect it?: Crawl-delay tells bots to wait N seconds between requests to reduce server load. Bing, Yandex, and other bots respect it. Google does not — to control Google's crawl rate, use the Crawl Rate settings in Google Search Console.

Explore All Tools

82 free tools — no signup required

All 82 tools are free · No signup · No ads