Aspan-User
Aspan-User is the agent our AI assistant uses to open a web page while answering a person's request.
What it does, and what it doesn't
- It opens a page only while answering a person's request. It is not a crawler: it doesn't follow links on its own or build an index.
- Pages it reads are not used to train AI models.
- It doesn't run JavaScript, use a headless browser, solve CAPTCHAs, or use proxies. If a site blocks it or needs a login, it tells the user and stops.
How to recognize it
Mozilla/5.0 (compatible; Aspan-User/1.0; +https://aspan.io/bot)
The robots.txt product token is Aspan-User.
robots.txt
Aspan-User follows the Robots Exclusion Protocol (RFC 9309). It reads your robots.txt before any other request, obeys the Aspan-User group if you have one (otherwise *), and treats your site as disallowed if robots.txt returns a server error or times out.
To block it from your whole site:
User-agent: Aspan-User Disallow: /
To block only part of your site:
User-agent: Aspan-User Disallow: /private/
What we keep
Fetched pages are held in memory only (up to 1 hour) so the model can read long documents. They are never written to disk and never linked to the person who asked. Our logs record only the site, the response status and the size.
Contact
Questions or problems with Aspan-User: bn@jas.vc
PDF reading uses MinerU (opendatalab).