robots.txt verification

What it is: robots.txt is a small file at your site’s root that tells crawlers which pages they may visit. One wrong line — "Disallow: /" — silently blocks your entire site from being indexed. Verification means regularly fetching robots.txt and checking for blocks, reachability, and a sitemap reference.

Why it matters: a broken robots.txt is one of the most common and damaging SEO mistakes — your content can vanish from search with no error shown to you.

Goal: BurstWriter checks your robots.txt daily and warns you on the dashboard the moment it could be blocking search engines, so you can fix it fast. The same daily check also verifies your sitemap is actually reachable (not just referenced), so a missing or broken sitemap is flagged too.