Lawmakers in the United States are increasingly taking aim at web crawlers with a recent focus on transparency. In June, the New York legislature passed the Stealth Crawler Prohibition Act, and in July, a similar bill, the Stealth Bot Prohibition Act, was introduced in the U.S. House of Representatives. Both bills seek to limit bots from masking their identity when scraping online content for particular purposes or under certain circumstances.
NY Stealth Crawler Prohibition Act (SB S9934)
The New York legislature passed the Stealth Crawler Prohibition Act on June 5, 2026, and the bill has been awaiting the Governor’s signature. SB S9934 amends the New York General Business law to prohibit using “stealth crawlers” that do not explicitly disclose their identity and purpose when scraping content on a specific subset of news websites.
The bill applies to operators deploying crawlers on covered news sources. An “operator” is defined to include both individuals and businesses (without an express exemption for non-profits), and a “covered news source” is the website or other source of any print, television, radio, network, cable, satellite, or digital publication or service that:
- Performs a public-information function comparable to traditional journalism organizations;
- Makes a substantial expenditure of labor, skill, and money to create, edit, produce, and distribute original content;
- Publishes new content or updates on at least a monthly basis with a process for error correction; and
- Has at least 1,000 monthly active viewers, listeners, users, or subscribers in New York.
A “crawler” is broadly defined to include any software that retrieves, scans, indexes, scrapes, or otherwise accesses a website, including online crawlers, spiders, bots, user agents, artificial intelligence (AI) agents, or equivalent tools. And a “stealth crawler” is a crawler that does not meet the new affirmative disclosure obligations under the law (discussed below).
Obligations and Prohibitions
The bill requires crawlers to disclose their identity and purpose at the time, or before accessing a covered news source. Specifically, crawlers must (1) identify themselves via a valid and accurate User-Agent string, stating the identity and version of the software product and the company behind it; and (2) disclose the specific nature and purpose of the crawler, including all uses the accessed content could be put to, in a format accessible to the journalism provider.
The bill prohibits any operator from deploying a “stealth crawler,” defined as a crawler that fails to comply with the disclosure requirements, in a manner that would damage, impair, or burden the operation of a covered news source, or otherwise cause a news source economic harm.
Enforcement
The New York Attorney General is granted exclusive enforcement authority and may bring an action to enjoin unlawful practices and seek civil penalties of up to $15,000 per day for each violation.
While there is no private right of action in the bill, it does allow Journalism Providers (defined as owners of covered news sources) to issue subpoenas to service providers (such as ISPs) to identify alleged violators without filing a lawsuit. While there is some inherent judicial oversight in obtaining the subpoena, the legislation does not appear to place a restriction on how such information may be used, leaving open the possibility that a news organization could leverage it in communicating with the scraper to send a cease and desist or to bring other claims against scrapers.
Takeaways
When it comes to compliance, New York’s Stealth Crawler Prohibition Act presents more questions than clear answers. The definition of “covered news source,” will likely require some analysis, including of what is considered a “public-information function comparable to traditional journalism organizations” in order to ascertain what organizations are covered. It is worth noting that covered news sources do not need to be headquartered or local to New York, rather they must have at least 1,000 monthly active viewers, listeners, users, or subscribers who are located in New York.
Separately, the bill provides limited technical implementation guidance for the crawler disclosure requirement. Given the level of detail that must be disclosed, it may be technically impossible to provide all necessary information via a “user-agent string” as required by the bill.
The enforcement mechanisms in the bill are also not clear cut. The stealth crawler prohibition provides that it is a violation “for any operator to deploy a stealth crawler in a manner that woulddamage, impair or burden the operation of a covered news source or otherwise cause… economic harm.” There are no provisions in the bill providing guidance as to how the attorney general will decide which uses of stealth crawlers may cause such harms. The attorney general can seek civil penalties of up to $15,000 per day for each violation. Yet it is unclear what constitutes a single violation—e.g., will that be based on a single discrete stealth crawl or the number of days that stealth crawling occurred. Lastly, the bill provides Journalism Providers with a noteworthy subpoena power— the right for a private party to obtain a subpoena to compel service providers to identify the alleged violators. This provision means that service providers will need to ready themselves to respond to legal process regarding “stealth crawlers” even before any lawsuit has been filed.
Should the bill be signed into law, it may have broad implications for scraping companies, news organizations, and service providers (such as ISPs, VPNs, and proxy providers).
Stealth Bot Prohibition Act (H.R. 9915)
New York’s Stealth Crawler Prohibition Act shares the same basic goal as the federal Stealth Bot Prohibition Act, but the federal bill is broader in its scope and prohibitions. Additionally, the federal bill is in a much earlier part of an approval process as it is pending before the House Energy and Commerce Committee.
H.R. 9915 would apply to stealth bots accessing any website, digital platform, or online service, whereas the New York law would apply only to crawlers accessing covered news sources. The federal bill also prohibits (i) using a stealth bot to access a website in a manner that is reasonably likely to damage, impair, or burden the technical or commercial operation of a website, and (ii) disguising a bot as a human user when used in connection with a generative AI model or service.
H.R. 9915 would be enforced by the Federal Trade Commission, which would have to grapple with the many questions that the bill introduces, such as what constitutes “damage” or “burden” to a website, and whether the bill implicitly requires an affirmative disclosure where use with generative AI to comply with the second prohibition.
H.R. 9915 also authorizes civil penalties up to $53,000 per violation, and state attorneys general to bring certain enforcement actions. In its current form, the bill does not create a private right of action.
We will continue to monitor these legislative developments as they evolve.