What ChatGPT Is and How It Was Misused
ChatGPT, created by the U.S.-based company OpenAI, is a powerful artificial intelligence system capable of writing, coding, analyzing data, and generating ideas in natural language. It is used worldwide by businesses, researchers, and governments for productivity and innovation. But a new OpenAI report has revealed how suspected Chinese government operatives exploited this same technology to plan surveillance tools and censorship systems.
According to OpenAI’s threat report, several China-linked users asked ChatGPT to help design proposals for monitoring citizens and tracking online speech. These activities violated OpenAI’s national security policy, leading the company to ban the accounts.
How the Chinese Were Using ChatGPT
OpenAI’s investigation showed that the operatives’ requests were not for hacking or advanced cyberattacks but for making state surveillance more efficient. The following activities were uncovered:
- Tracking Uyghurs and “High-Risk” People:
One user, likely tied to a Chinese government entity, asked ChatGPT to draft a proposal for a system to analyze travel data and police records of Uyghur Muslims and other individuals labeled “high-risk.” This aligns with Beijing’s long-standing repression of the Uyghur minority, which the U.S. government under President Trump labeled as genocide and crimes against humanity. - Scanning Social Media for Political and Religious Speech:
Another Chinese-speaking account requested help creating promotional materials for a tool that would monitor platforms such as X (formerly Twitter) and Facebook. The tool was designed to identify “extremist,” religious, and political content, tightening the state’s control over public expression. - Outlining Social Media “Listening” Tools:
Other users asked ChatGPT to describe systems for mass “social media listening”—a euphemism for state monitoring of online conversations. These would allow the Chinese Communist Party to detect dissent and identify people spreading unwanted narratives. - Assisting Phishing and Malware Operations:
Some banned accounts also used ChatGPT to support phishing and malware campaigns and to explore how China’s own AI model, DeepSeek, could automate such operations.
How ChatGPT Caught the Chinese
OpenAI’s monitoring systems detected these activities through behavioral tracking and language analysis. Its security team, led by researchers including Ben Nimmo and Michael Flossman, identified suspicious Chinese-language prompts that requested guidance on surveillance design. When the pattern matched state-linked cyber behavior, OpenAI banned the accounts and documented the findings in its report.
The report provided what OpenAI called “a rare snapshot into the broader world of authoritarian abuses of AI.” It emphasized that these users were not creating new cyberweapons but refining traditional surveillance techniques with AI’s help—making them faster, cheaper, and more scalable.
China’s Response
Liu Pengyu, spokesperson for China’s embassy in Washington, D.C., rejected OpenAI’s accusations, calling them “groundless attacks and slanders.” He insisted that China’s AI governance system “balances development and security” and includes ethical guidelines and data security laws. Chinese officials maintain that their system promotes innovation while preventing misuse—claims viewed skeptically by Western observers.
The Growing Danger of AI Surveillance
Experts warn that China’s use of generative AI for repression could strengthen its authoritarian model and export it abroad. Ben Nimmo of OpenAI explained that Beijing has long used surveillance to control its citizens, but now sees AI as a way to “get a little bit better” at it. By using AI to analyze language, automate propaganda, and detect political behavior, the Chinese government could dramatically expand its monitoring reach.
Michael Flossman added that state actors are using AI “to refine existing tradecraft,” not to invent new cyberattacks. In other words, the same tools that make businesses more efficient can make digital repression more precise.
OpenAI noted that China is not alone. Hackers and criminal networks in Russia, North Korea, and Southeast Asia have also used AI to refine malware, phishing, and scam operations. Yet, there is also a positive side: more users now employ ChatGPT to identify scams before falling victim, with OpenAI estimating that it helps stop fraud “three times more often than it is used for scams.”
Still, the episode underscores a larger truth about AI’s dual use. As the U.S. and China race to dominate artificial intelligence, the line between innovation and abuse is thinning. What began as a tool for learning and creativity has become a potential instrument of surveillance and control.
OpenAI’s swift action against the Chinese-linked accounts may have stopped one attempt, but it also signals how future battles in the AI era may unfold—not just over who leads in technology, but over who uses it, and for what purpose.
ACZ Editor: By the way, this means that ChatGPT is also monitoring your interactions with its AI engine.