Probeo crawls the public surface.

what it does

Sitemaps and public pages.

No access required, no credentials, no integration.

It identifies as probeo, not as Googlebot. We do not disguise the crawler, because a crawler that lies about itself is not something you should let into your site.

if we are blocked

Most sites do not block legitimate crawlers.

If your bot protection is aggressive enough to block probeo, that is worth knowing independently of us. It means something is making a judgment about legitimate automated traffic, and it is worth checking what else it is catching.

If it does block probeo, allowing it through is the one piece of setup.

rate and load

Prioritised, not exhaustive.

Crawling is prioritised by change and page importance rather than re-crawling everything every pass. Wide, fast crawls are possible when useful, and are bounded by what your infrastructure can absorb rather than by what we can send.

what we take

Public HTML, headers, links.

Nothing behind a login. Nothing personal.

See what AI says about you How it works