What means that a domain is not accessible?
There are different reasons why a domain might be blocked to be scanned by Trusted Humans, in this article we will list the most common ones.
Blocked
Some profiles have violated Trusted Humans policies, or have explicitly requested us to remove them from the platform. These profiles cannot be accessed by our bot, and therefore cannot be scanned.
These platforms could be taken as “non-trustable” in terms of their content, as they have either tried to fake their results/checks, or they don’t want to be listed in this trustability platform.
Unfortunately, we cannot do anything about it. If you think that this might be an error, you can contact us and we will investigate it.
Bot information
Before you start with the profile claiming, or performing an scan, you need to make sure that the Trusted Humans bot has access to the domain that you’re trying to claim, otherwise you won’t be able to finish the claiming process.
Our bot IP might change from time to time, so you can scope it by its user agent. The bot User-Agent header is:
TrustedHumansBot/1.0 (+https://trustedhumans.ai/bot-info)We strongly advise to target the bot reference as TrustedHumansBot as the version, and URL might change in the future.
In order to include the exception to the bot firewall, you’ll need to add an exception in your provider. This will depend on which one you’re using.
Here you have some guides for the most common providers:
If you’re using one that is not listed here, or need help, feel free to contact us, and we will support you during the process.
Robots.txt
Trusted humans strictly follow robots.txt indications, meaning that if a page is blocked by the robots.txt file, we won’t fetch it.
Companies might use that to their advantage. If we find that they are trying to fake their results using the robots.txt file, we will block them from the platform.
We have a strict no-tolerance rule for false information. We base our content on truth, and therefore we won’t tolerate falseness.
It’s extremely important that the robots.txt file is not abused for that purpose.
Apart from that, feel free to include routes that shouldn’t be scanned, for example, because they show private or sensitive information.
If you are doubting if a page should be blocked or not, please contact us, and we will gladly help you solve that question.