Probeo
probeo
Menu
Crawler

Probeo Bot

Probeo Bot is the crawler used by Probeo to observe how a website behaves as a system. It is designed to be predictable, limited in scope, and respectful of site infrastructure.

This page explains what Probeo Bot does, what it does not do, and how site owners can control its behavior.

What it does

What Probeo Bot does

Probeo Bot observes your site on every run and updates a shared view as things change.

  • Requests HTML documents only by default
  • Observes pages as they are delivered to real browsers
  • Builds an inventory of pages, templates, and shared systems
  • Makes very limited asset requests only when required to understand page structure
  • Operates in a read-only manner
  • Does not modify site content or configuration

Probeo Bot exists to observe behavior, not to interact with the site.

What it does not do

Limited and cautious by design

Probeo Bot is designed to avoid side effects on your site.

  • Submit forms
  • Execute transactions
  • Log in to user accounts
  • Trigger application workflows
  • Write data
  • Modify content
  • Execute destructive actions
  • Perform load testing or stress testing

Probeo Bot does not attempt to bypass authentication or access restricted areas.

Technical details

How it behaves

Request behavior

  • Requests are rate-limited and controlled
  • Unnecessary repeat requests are avoided
  • Large sites are processed in stages
  • Site stability is prioritized over crawl speed

Assets & scripts

  • Requests HTML documents
  • Does not load all page assets
  • Does not execute JavaScript beyond basic rendering
  • Tracking scripts and analytics requests are ignored

Identification

  • Dedicated User-Agent string
  • Optional request signatures (v1.1+)
  • Requests can be validated to confirm origin

robots.txt

  • Respects allow and disallow rules
  • Honors crawl-delay directives
  • Respects explicit path restrictions
  • Behavior can be configured per site

Crawl scope

  • Crawls only domains associated with a site
  • Does not discover unrelated domains
  • Does not follow links outside defined scope
  • Scope is defined before crawling begins

Security

  • No credential storage
  • No session reuse
  • No form submission
  • No write operations
Contact

Troubleshooting

If Probeo Bot appears to behave in unexpected ways, or if crawl behavior needs to be adjusted, contact the team.

[email protected]

Please include: the affected domain, timestamps of observed requests, and relevant request headers.