正文 · AI 翻译
IT Home reported on November 3 that a creator in San Diego had hoped more people would visit her website. Instead of visitors, she got more than 550,000 visits from Meta crawlers, and she is now scrambling to keep the site running.
Children's picture-book author and illustrator Chelsea Paulus told Business Insider that the automated visitor exhausted her web hosting resources, forcing her to work with her provider to troubleshoot the problem.
“I've never encountered anything like this,” Paulus said. “Even though I know a bit about technology, this week has been exhausting.”
Paulus has run her business for 10 years and became a full-time creator last year. She previously worked in technology and web design, so she quickly noticed something was wrong when traffic surged abnormally.
“I don't think this was a deliberate malicious attack by Meta,” Paulus said. “But small business owners need to know about this kind of risk and have tools they can use to block crawlers.”
Struggling to keep the website online
On October 24, she received an email alert from hosting provider SiteGround saying that 80% of her allocated CPU resources had been used. Paulus's four websites share a hosting plan with the Bulgarian provider.
Paulus said that on Monday she checked her site's security dashboard and found several requests coming in every second.
“I opened WordFence, and the access logs kept refreshing on the screen every second,” Paulus said. “When I opened the details in WordFence, I could see the IP address. After looking it up, I found that the IP belonged to a Meta node in Dallas.”
The automated visitor identified itself as “Meta External Agent,” which staff at hosting provider SiteGround later confirmed.
According to Meta's developer page, Meta External Agent crawls content across the web for purposes including training foundation large language models and directly indexing content to improve products.
Paulus occasionally runs Meta ads and had seen Facebook crawlers visit before, but this was the first time she had encountered traffic on this scale.
“I changed the robots.txt file and configured the firewall, but those rules often take 24 hours to take effect, and I needed to stop the crawler immediately,” Paulus said. “So I contacted my hosting provider.”
“The provider gave me a piece of code to add to a hidden configuration file on the website,” she added. “Ordinary website builders generally don't touch files like that, but it was the only way to block the crawler at the time.”
But the fix worked only temporarily. By the middle of this week, another Meta crawler called “Meta Web Indexer” arrived and drove hosting resource usage to its limit.
According to IT Home, Meta's developer documentation defines Meta Web Indexer as a web crawler that browses webpages to improve the quality of search results Meta AI returns to users.
The issue remains unresolved
A Meta spokesperson said on Monday that website operators have the right to control crawler behavior and directed people to the Meta Web Crawlers developer documentation to learn what the various crawlers are used for.
“Like many companies, Meta uses web crawlers for a variety of purposes. Website administrators can use tools such as robots.txt to set how our crawlers interact with their sites,” the Meta spokesperson said. “We publicly provide robots.txt configuration guidance for our crawlers and respect the rules set by site owners.”
Screenshots of emails between Paulus and SiteGround show the provider saying its “server-side protection” was already limiting crawler requests.
As of Monday, Paulus's website remained unstable. She had to remove the website link from her Instagram profile to prevent visits from real people, on top of crawler traffic, from taking the site completely offline.
Paulus cannot afford to pay for a higher traffic allowance: crawlers are not customers and bring in no revenue. There is currently no law prohibiting AI crawlers from accessing publicly viewable websites.
She is also trying Cloudflare's human verification tool to block crawlers. The tool is currently free.
“Meta should put safeguards in place to prevent incidents like this from happening again,” Paulus said. “If unrestricted AI crawlers keep doing this, will providers like Cloudflare be able to keep offering protection for free?”
She also said: “If this kind of thing happens frequently, small businesses will ultimately have to spend more money, either to increase their resource allowances or to buy protection services.”