Crawl Controls
One set is yours to configure: how CiteWorth inspects the site, how widely and how often. The other is already published to every crawler that arrives — and is where accidental exclusions tend to hide. Crawl Controls covers both.
No card required.
Two halves
These get grouped together because they are both about crawling, but they do opposite things. One is a control panel. The other is a reading of rules you already published.
You configure
The scope of what gets looked at, how often inspection runs, and how the agent behaves while it is doing so. This is your setting, and it constrains us rather than your site.
You inspect
The directives published to everything that arrives. These were set at some point, often by someone else, and frequently do something other than what was intended.
Where exclusions hide
Accidental exclusion is quiet. There is no error, nothing breaks, and the page simply stops being considered. These are the four places it usually happens.
A rule written for a staging environment, or a pattern broader than intended, quietly removing a section from consideration.
Coverage that has drifted from the site as it now exists — missing what matters, listing what no longer does.
Pages pointing elsewhere as authoritative, effectively withdrawing themselves from consideration.
Chains and loops accumulated across migrations, making a destination expensive or impossible to reach.
Visibility into what was tested
A finding only means something if you know what was looked at to produce it. Scope is visible alongside results, so the absence of a finding can be read correctly — either nothing was wrong, or that area was never in scope. Those are very different conclusions.
Inspection scope is a setting, not an entitlement claim. Plan details live on pricing.
Do this early
If pages are being excluded before anything else happens, every view built on that crawl inherits the gap — and the picture will look calmer than it should. Reviewing discovery hygiene first means the rest of the evidence is describing your actual site.
Discovery sits underneath the evidence, which is why problems here propagate everywhere.
Review your discovery rules and see what is genuinely reachable.
Inherited a site with unclear directives? Talk to us
