By Mehdi Hassanrobots.txt and the XML sitemap are the two smallest files on your site and two of the most consequential. One wrong line in robots.txt can hide the whole site from search. A messy sitemap can send Google to redirects and dead pages instead of the content you care about. Both take minutes to set up correctly.
This guide gives you the setup steps, the lines a startup's robots.txt actually needs, the AI crawler decision and the most common mistakes. It's part of our technical SEO guide for marketing websites. If you're new to either file, our plain-English definitions of robots.txt and XML sitemaps are a quick primer.
What do robots.txt and an XML sitemap do?
robots.txt is a plain-text file at the root of your domain that tells crawlers which URLs they may request. An XML sitemap is a list of the URLs you want search engines to find and index. One limits where crawlers go; the other suggests where they should go.
robots.txt | XML sitemap | |
|---|---|---|
Lives at | yourdomain.com/robots.txt | Usually yourdomain.com/sitemap.xml |
Purpose | Allow or disallow crawling of paths | List canonical URLs for discovery |
Controls indexing? | No. Blocked URLs can still be indexed from links | No. It's a hint, not a command |
Who reads it | Any well-behaved crawler | Search engines, some other crawlers |
Most common error | Disallow: / left from staging | Listing redirects, 404s or noindex pages |
The distinction that trips people up: robots.txt doesn't remove pages from Google. If a blocked URL has links pointing to it, Google can still index the bare URL without its content. To keep a page out of results, let it be crawled and add a noindex tag, or put it behind a login.
How should a startup set up robots txt and sitemap files?
Set up both files in six steps: generate the sitemap from your CMS or framework, write a short production robots.txt, reference the sitemap in it, protect staging separately, submit the sitemap in Search Console and verify both files by hand. Most platforms do the first step for you.
Generate the sitemap automatically
Webflow and Framer create a sitemap from published pages and CMS items. In Next.js, a sitemap file in the app directory generates one from your routes and CMS data. Avoid hand-maintained sitemaps; they go stale.
Exclude pages that shouldn't be indexed
Remove thank-you pages, legal duplicates, internal search and anything set to noindex from the sitemap. In Webflow and Framer this is a per-page setting.
Write a short robots.txt
Allow all crawlers by default, disallow only private or useless paths such as internal search results or API routes, and nothing else.
Add the sitemap line
Include a Sitemap line with the full absolute URL, for example Sitemap: https://yourdomain.com/sitemap.xml.
Protect staging
Put staging and preview domains behind a password or an X-Robots-Tag noindex header, so their robots.txt never matters.
Submit and verify
Submit the sitemap in Google Search Console and Bing Webmaster Tools, then open both files in a browser on the production domain and read them line by line.
What should a production robots.txt contain?
A production robots.txt for a marketing site should be short: a default group that allows crawling, a few disallow lines for private paths, any AI crawler rules you've decided on and the sitemap location. If yours runs much longer than that, something is probably being over-managed.
Line | What it does | Keep it? |
|---|---|---|
User-agent: * (all crawlers) | Starts the default group for every crawler | Yes |
Disallow: /search | Stops crawling of internal search result pages | If you have site search |
Disallow: /api/ | Keeps crawlers off API routes | On Next.js and custom builds |
Disallow: /thank-you | Saves crawling on form confirmation pages | Optional; noindex is better |
User-agent: GPTBot plus Disallow: / | Opts out of OpenAI model training | Only if you've decided to |
Sitemap: https://yourdomain.com/sitemap.xml | Points every crawler to the sitemap | Yes |
Two things don't belong there. Don't block CSS, JavaScript or image folders, because Google needs them to render pages; see our guide to JavaScript SEO for why rendering matters. And don't add noindex or crawl-delay lines: Google doesn't support noindex in robots.txt and ignores crawl-delay.
Hypothetical example: a startup robots.txt, line by line
For example, suppose a SaaS startup runs its marketing site on Next.js with the app on a separate subdomain.
Its production robots.txt has one default group for all crawlers that disallows /api/ and /search, and allows everything else. It has a second group for GPTBot and Google-Extended that disallows the whole site, because the founders decided not to contribute to model training. It deliberately has no rule for OAI-SearchBot, PerplexityBot or Claude-SearchBot, so those can still fetch pages for answers. The last line is the Sitemap URL.
The app subdomain has its own robots.txt that disallows everything, and the staging domain sits behind a password.
Should you allow or block AI crawlers in robots.txt?
Most startups should allow AI search crawlers, because they fetch pages to answer questions and cite sources, and blocking them removes you from those answers. Training crawlers are a separate choice: blocking them opts your content out of model training without affecting search visibility.
Crawler token | Operator | Main job | Typical startup choice |
|---|---|---|---|
Googlebot | Search indexing, including AI features in Search | Allow | |
Google-Extended | Controls use in Gemini models, not Search | Your call | |
OAI-SearchBot | OpenAI | ChatGPT search results | Allow |
GPTBot | OpenAI | Model training | Your call |
ClaudeBot | Anthropic | Model training | Your call |
PerplexityBot | Perplexity | Perplexity search index | Allow |
CCBot | Common Crawl | Open web dataset used for training | Your call |
Crawler names and roles change, so check each operator's current documentation before you write rules. Our guide to blocking or allowing AI crawlers covers the trade-offs in depth. You may also see advice to add an llms.txt file; it's optional, and 2026 studies including SE Ranking's analysis of 300,000 domains found no measurable citation lift. Our explainer on what llms.txt is and whether it helps has the details.
How do you keep staging sites out of Google?
Keep staging out of Google with password protection or a noindex header on every response, not robots.txt alone. A Disallow rule stops crawling but not indexing, and it's the setting most likely to be copied to production by mistake.
On Webflow, the staging subdomain can be excluded from indexing in site settings. Framer and Vercel offer password protection for preview deployments on paid plans; check your current plan. Our technical SEO checklist for new website launches includes this as a go-live check.
What belongs in an XML sitemap?
An XML sitemap should contain every canonical, indexable URL on the site that returns a 200 status, and nothing else. Every entry is a signal that the page matters, so redirects, 404s, noindex pages and parameter URLs make the whole file less trustworthy.
- Include: home, product, pricing, solution, comparison and blog pages, plus useful category hubs.
- Exclude: redirected URLs, 404s, noindex pages, canonicalized duplicates, UTM and filter URLs.
- Use lastmod honestly: update it when content changes, not on every build. Google uses lastmod when it's consistently accurate and ignores priority and changefreq.
- Split large sites with a sitemap index. Google caps how many URLs and how much data a single sitemap file can hold, a limit few marketing sites approach; check Google's sitemap documentation for the current figures.
- Match canonicals: every sitemap URL should be self-canonical, using the same protocol, host and trailing-slash style.
If pages are in the sitemap but still not indexed, the sitemap isn't the problem. Search Console's Pages report shows the reason, and our guide to pages not indexed by Google explains each status.
What are the most common robots.txt and sitemap mistakes?
The most common mistakes are blocking the whole site, using robots.txt to remove pages from Google, and sitemaps full of URLs that don't return 200. Each is quick to spot if you know where to look.
- Disallow: / on production, usually inherited from staging.
- Blocking a page you also set to noindex, so Google can't see the noindex tag and may keep the URL indexed.
- Blocking CSS or JavaScript, which stops Google rendering pages properly.
- Sitemaps listing redirects and 404s after a redesign, because the old sitemap was never regenerated. Pair a redesign with a redirect map.
- A robots.txt on the wrong host, such as www when the site serves from the bare domain.
- Forgetting subdomains: each subdomain needs its own robots.txt and sitemap.
