← Glossary

robots.txt

robots.txt is a small file at the root of your site that gives search-engine crawlers instructions about which areas to crawl or skip. A misconfigured robots.txt can accidentally hide your whole site from search.

It usually also points crawlers to your sitemap.