robots.txt is a plain text file at the root of your domain (yourdomain.com/robots.txt) that tells automated crawlers which parts of your site they're allowed to fetch. It's the first thing a well-behaved bot checks.
For AI visibility, it's a one-line decision with real consequences: each AI crawler has its own user-agent token (GPTBot, OAI-SearchBot, ClaudeBot, Google-Extended, PerplexityBot), and you can allow or block each one, per path. Block a training bot and future models learn less about you; block a live-answer fetcher and you forfeit citations in AI answers.
A common, costly mistake is blocking AI crawlers by accident. Inherited from an old rule or a security plugin's default, then wondering why the brand never appears. It's worth reading your own robots.txt the way a bot would. Note that robots.txt differs from an llms.txt, which guides rather than gates.