SofyaBot

The web crawler for Sofya.

SofyaBot is the web crawler for Sofya. It builds Sofya's own search index, served at search.sofya.co.

We are building an independent index rather than reselling another engine's results, which means crawling the web ourselves.

How to identify it

SofyaBot sends this User-Agent on every request:

SofyaBot/1.0 (+https://sofya.co/bot)

It currently crawls from:

159.195.72.86    crawl-1.sofya.co
159.195.250.248  crawl-2.sofya.co
159.195.254.79   crawl-3.sofya.co
206.42.109.58    crawl-4.sofya.co

If you need to allowlist us, use those addresses. Each address listed with a name has forward-confirmed reverse DNS: the name it resolves to resolves back to it.

The same list is published as JSON at /bot/ips.json, for firewalls and bot-verification services that read it automatically.

How it behaves

It reads robots.txt and obeys it. Rules for SofyaBot take precedence; otherwise we follow the rules for *.
It honours Crawl-delay. If your robots.txt sets one, we use it.

How to control it

Block it entirely

User-agent: SofyaBot
Disallow: /

Block part of your site

User-agent: SofyaBot
Disallow: /private/
Disallow: /checkout/

Slow it down

One request every ten seconds:

User-agent: SofyaBot
Crawl-delay: 10

If something is wrong

If SofyaBot is crawling too aggressively, requesting something it should not, or causing you any problem at all, email bot@sofya.co and we will fix it. A report from a site owner takes priority over anything else we are doing.

If you would like content removed from our index, email the same address with the URLs.