SofyaBot

The web crawler for Sofya.

SofyaBot is the web crawler for Sofya. It builds the search index behind Sofya's API, which gives AI agents access to the web.

We are building an independent index rather than reselling another engine's results, which means crawling the web ourselves.

How to identify it

SofyaBot sends this User-Agent on every request:

SofyaBot/1.0 (+https://sofya.co/bot)

It currently crawls from:

159.195.72.86

If you need to allowlist us, use that address. We will update this page before adding others, so it is safe to treat this list as authoritative.

How it behaves

  • It reads robots.txt and obeys it. Rules for SofyaBot take precedence; otherwise we follow the rules for *.
  • It honours Crawl-delay. If your robots.txt sets one, we use it.
  • Without a Crawl-delay it makes at most one request per second to a site, with jitter, regardless of how large the site is.
  • It backs off on 429, 503 and 509. A rate-limit response slows the whole site down for us, not just that one URL, and we honour Retry-After.
  • It does not execute JavaScript. SofyaBot fetches HTML over plain HTTP; it does not run scripts, click, log in, or submit forms.
  • It fetches each URL once. SofyaBot is not a monitoring service and will not re-request the same page repeatedly.
  • It ignores pages behind a login or a paywall. We only read what an anonymous visitor can read.

How to control it

Block it entirely

User-agent: SofyaBot
Disallow: /

Block part of your site

User-agent: SofyaBot
Disallow: /private/
Disallow: /checkout/

Slow it down

One request every ten seconds:

User-agent: SofyaBot
Crawl-delay: 10

Changes to robots.txt are picked up within 24 hours.

If something is wrong

If SofyaBot is crawling too aggressively, requesting something it should not, or causing you any problem at all, email bot@sofya.co and we will fix it. A report from a site owner takes priority over anything else we are doing.

If you would like content removed from our index, email the same address with the URLs.