NewmanKinBot

Why it visits

Newman & Kin runs SeeVerbatim and Faroleta, which help businesses with their listings, reviews and websites. NewmanKinBot reads a business's public website to learn what the business does and how its site is doing: whether it works on a phone, which opening hours and contact details it publishes, and how recently it was updated.

We use what it finds to judge whether one of our services would help that business. If one would, we email the business about it: one email and at most one follow-up.

What it fetches

• Your robots.txt first. It reads it again at most once a day.
• The home page, then at most four pages the home page links to: about, contact, careers, and a security, trust or privacy page. Five pages per site at most.
• GET requests only, at least two seconds apart on the same site. A Crawl-delay in your robots.txt makes the gap longer, up to 30 seconds.
• One retry when a request fails.
• A page that is empty without JavaScript may be loaded once in a headless browser, which also loads the files that page needs. It sends the same user agent.

What it never does

• Fill in forms or log in.
• Request pages your site doesn't link to.
• Scan ports or probe for weaknesses.
• Crawl a site whose robots.txt answers with an error other than "not found". It treats the site as blocked and checks again later.

Block it

Add this to your robots.txt:

User-agent: NewmanKinBot
Disallow: /

To keep it out of part of your site, list those paths instead. With no group for NewmanKinBot, it follows your rules for User-agent: *. A change reaches us within 24 hours.

What we keep

The visible text of the pages it reads and the facts we draw from them, each with the address of the page it came from. That can include the names, job titles and email addresses a site publishes. We don't keep page code or images.

Questions or problems

Email admin@newmanandkin.com with your domain and roughly when you saw us. You can also ask us to delete what we hold about your business, or never to contact it.

We don't crawl from a fixed list of IP addresses yet, so identify us by the user agent.