By monitoring the activity of web crawlers, a company offering distributed denial-of-service (DDoS) mitigation solutions noticed that in 4% of the cases the crawlers were impostors with fake user-agents that allowed them to pass as Googlebots.Googlebots are crawlers that scour websites for information that can be indexed for the search engines. They are generally given limited access, since some parts of some websites contain sensitive information they should not have access to.Incapsula has p…
Comments are closed.
