CiteWorthBot
CiteWorthBot helps site owners and authorised teams audit crawler accessibility, metadata readiness, indexability and other technical signals used by CiteWorth.
Crawler identity
Crawler name: CiteWorthBot
Version: 1.0
Canonical public information URL: https://citeworth.com/bot/
User-Agent format:
CiteWorthBot/1.0 (+https://citeworth.com/bot/; module=<module>)
The optional module value identifies the CiteWorth component performing the request.
Why CiteWorthBot visits websites
CiteWorthBot performs technical diagnostics and crawl-access verification for sites being analysed through CiteWorth. Scans are initiated as part of authorised CiteWorth workflows and are designed to collect evidence about how a site can be accessed and interpreted.
It may inspect signals such as:
- page accessibility
- HTTP response behaviour
- metadata
- indexability directives
- robots.txt
- sitemaps
- canonical information
- structured technical page signals
CiteWorthBot is not a search-engine crawler, and not every module performs every check.
Transparent crawler identification
CiteWorthBot identifies itself as CiteWorth and does not impersonate Googlebot, Bingbot or other third-party search-engine crawlers.
Customer-owned crawler requests may include the headers X-CiteWorth-Scan-Id and
X-CiteWorth-Component.
These identifiers are used to help correlate authorised CiteWorth scan requests and assist with technical troubleshooting. They are correlation and identification signals, not cryptographic proof of request origin.
Responsible crawling and rate limits
CiteWorth uses governed request pacing when accessing customer sites.
If a site signals that it is being rate limited, CiteWorth can reduce its request rate rather than continuing at the original pace.
HTTP 429 responses are treated as temporary rate-limiting signals and are not
automatically treated as evidence that a website has a technical defect.
Where applicable, CiteWorth respects valid Retry-After instructions returned by the
origin.
Allowing or blocking CiteWorthBot
Site owners remain in control of access to their websites. Security platforms, CDNs, WAF rules and origin configuration may allow, rate-limit or block CiteWorthBot.
If CiteWorth cannot reliably access a page, the platform may report that evidence could not be fully verified rather than treating inaccessible content as a confirmed website defect.
Questions or crawler access issues
If you have questions about CiteWorthBot or need help identifying CiteWorth traffic, contact:
FAQ
Is CiteWorthBot a search engine?
No. CiteWorthBot is operated by CiteWorth to perform authorised technical and visibility analysis.
Does CiteWorthBot impersonate Googlebot or Bingbot?
No. CiteWorthBot identifies itself using its own CiteWorthBot User-Agent.
What happens if my site returns HTTP 429?
CiteWorth treats HTTP 429 as a rate-limiting signal and can reduce its request rate. A 429 response is not automatically treated as a website defect.
How can I identify a CiteWorth request?
Look for the CiteWorthBot User-Agent. Authorised customer-site crawler traffic may also include
X-CiteWorth-Scan-Id and X-CiteWorth-Component correlation headers.