In response to the findings, Reddy Doddipalli, technical counselor and product lead at Info-Tech Research Group, said, “with the rise of genAI models and embedded systems, the demand for large-scale data aggregation has led to a surge in crawler activity, with bots sifting through billions of web pages to feed ML algorithms.”
While there are many advantages of AI crawlers, he said, the bottom line for organizations, website owners, and users is to understand the negative implications, such as data privacy, security, ethics, intellectual property, infrastructure [and] bandwidth consumption.
Doddipalli says he recommends developing a framework and best practices to manage and mitigate crawler activity. “For example, many of these crawlers now mimic human behavior, bypassing traditional defenses and controls and requiring innovative detection techniques,” he noted, adding that thought is required to ensure AI bot behavior remains a constructive force, not a destructive element.
