LockleyBot
LockleyBot is the crawler operated by Lockley. It reads public business websites to understand what a company sells and who buys it, so that we can suggest relevant companies to our customers.
What it does
LockleyBot fetches a small number of pages from a company's public website — the homepage and up to eleven pages linked from it, such as about, services or contact. It reads published text only. It does not attempt to access anything behind a login, submit forms, or collect personal data from your pages.
How to identify it
User-Agent: LockleyBot/1.0 (+https://lockley.ai/bot) From: crawler@lockley.ai
Requests are signed with HTTP Message Signatures over Ed25519. Our public keys are published at /.well-known/http-message-signatures-directory, so you can verify that a request claiming to be LockleyBot is genuine. Anything using our name without a valid signature is not us.
How often
At most one request per second per site, regardless of how many pages we want. We send conditional requests (If-Modified-Since, If-None-Match) and accept gzip, so repeat visits usually cost you a 304 and nothing else. We honour Crawl-delay, back off on 429 and 503, and stop contacting a site entirely for 30 days after five consecutive failures.
How to stop it
Any one of these is enough, and we check all four on every visit:
# robots.txt
User-agent: LockleyBot
Disallow: /
# /.well-known/tdmrep.json — reserves text and data mining rights
[{ "location": "/", "tdm-reservation": 1 }]
# response header
tdm-reservation: 1
# or in the page head
<meta name="tdm-reservation" content="1">A reservation takes effect on the next visit and we record the date we saw it. If you would rather email, write to crawler@lockley.ai and we will add your domain to our blocklist and delete anything we already hold for it.
Where the data goes
Text we fetch is used to build a short structured description of the business. We do not republish your pages, sell your content, or train foundation models on it.