WHAT IT IS
A crawler that helps people choose what to block.
NETjunky Solutions builds web content-filtering block lists, so that people can choose what to block on their own networks. To categorise a website, the bot reads the public home page of the site and works out what kind of site it is (for example adult, gambling or similar categories).
HOW TO RECOGNISE IT
Requests carry this User-Agent.
NerataBlocklistBot/1.0 (+https://netjunky.co.za/bot; abuse@netjunky.co.za)
Earlier runs used the spelling NetaraBlocklistBot/1.0. It is the same bot.
What it does
- Requests the public home page of a domain, trying HTTPS first and HTTP only if HTTPS fails, and follows redirects.
- Keeps the page title, the page description and a short text excerpt, only to decide the category.
- Classifies the result with rules and a locally run model, and a person reviews the uncertain results.
What it does not do
- It does not log in, submit forms or post anything.
- It does not scan ports or probe for vulnerabilities.
- It does not copy or republish your content.
Want your domain left out?
Email abuse@netjunky.co.za with the domain name and, if you can, the time of the request (with time zone). We will reply and exclude the domain.