In its efforts to curb AI-generated spam, TikTok has announced that it will launch tests for new ways to detect this kind of abuse targeted specifically at users who post misleading content on political, financial, and medical topics.
The company posted the news on its newsroom blog, saying that this move is part of its overall campaign to help users understand what constitutes AI-generated content on its platform. The tests will roll out soon, but the company hasn’t disclosed the exact date yet.
The time chosen for this initiative is particularly important because it marks the period during which the amount of AI-generated content on social media platforms has dramatically increased. This development makes it increasingly difficult for regular users to distinguish whether some content was created by humans or machines.
Why These Three Topics
In their press release, TikTok specifically pointed to political or current events coverage, financial advice, and medical content as areas that will be subject to the detection testing for AI-generated spam. This particular emphasis makes sense given the potentially harmful nature of misleading information posted in these fields.
This specific selection aligns with the results already observed by independent researchers. According to a Kapwing report, also covered by Search Engine Journal in June, health-related content occupies one of the leading positions in the number of AI-generated posts on TikTok, easily outranking many other categories where the creators feature on-camera. Videos with guidance on medical conditions tend to be easy to produce using AI algorithms and don’t require the participation of actual creators, which might explain the high number of low-quality, AI-generated posts in this category.
Financial advice poses similar problems – misleading information on investments, taxes, or personal finance can cause significant harm to the audience and AI tools allow for rapid production of huge volumes of generic-looking content in this field with little costs involved. Political content also comes with its risks, especially given the potential for misinformation campaigns and shaping of public opinion, especially during sensitive news periods.
How the Detection Testing Will Work
The description provided by TikTok indicated that the testing operates at the account level and doesn’t involve scanning for individual videos. This suggests that the detection process involves analyzing the posting patterns of an account as a whole, rather than analyzing specific posts.
It is also important to note that the update will continue existing efforts to combat spam. According to TikTok, the platform already removed more than 86 million spam accounts in the first quarter of this year alone, giving an indication of how widespread the spam problem currently is before the update was even released.
As the press release does not provide much information about how exactly TikTok defines a qualifying account and what happens to such an account once it is identified, the exact implementation of the testing is somewhat unclear. No information has been provided on whether the identified accounts will be completely removed from the platform, restricted in terms of distribution, or otherwise punished.
This aspect is worth following closely as the effectiveness of the update will largely depend on how strict the definition of a qualifying account ends up being. In case of an overly broad definition, a lot of legitimate creators might end up in trouble due to their frequent posting about health or financial issues, while too narrow a definition may cause the platform to overlook spam accounts.
How Other Platforms Have Responded
TikTok’s update is just one of the multiple moves made by social platforms trying to counter AI-generated content flood. For example, in 2021 YouTube announced updates to its monetization guidelines, aimed at dealing with inauthentic content. Shortly afterwards Meta released its own measures, targeted at removing repetitive and unoriginal content.
Google also addressed the issue. In June the company published a research paper about the system designed for identification and removal of the groups of accounts producing AI-generated spam.
All these moves clearly indicate that the problem of AI-generated spam has been recognized as one requiring continuous and active moderation by the platform management. As more and more content generation tools become available, platforms appear to race to develop their detection algorithms.
TikTok Joins the C2PA Steering Committee
Together with the spam detection update, TikTok has announced its participation in the steering committee of Coalition for Content Provenance and Authenticity (C2PA). The Coalition develops C2PA standard that allows tracking the origin and modification of a piece of digital content.
By joining the steering committee, TikTok has stated its goal of helping to drive adoption of the technology beyond the platform itself.
According to TikTok, the platform has labeled more than 3 billion videos as AI-generated content by means of a combination of Content Credentials, creator labels, and TikTok proprietary invisible watermarking technology.
This update reflects the recent trend of adoption of standardized content verification systems seen from other platforms. In May Google announced extension of SynthID verification to Search, while C2PA verification will also be implemented soon.
These moves indicate the industry recognition of the importance of widespread usage of content provenance systems on both social and search platforms. Participation in the steering committee, rather than adoption of the technology as a regular member, might indicate that TikTok is interested in shaping the future of C2PA standard.
Why This Matters for Creators and Marketers
This news update is of greatest importance to those who create content in three specific categories that were pointed out by TikTok. Health, financial and political subjects are among the hottest marketing niches for people seeking to address engaged audiences, which makes them of high interest for spam detection testing.
It means that if you create content on one of these topics, your account can receive a closer inspection even if all your posts meet the rules and are original and legitimate. This is due to the fact that the testing is carried out on the account level; therefore, the way you post your content can become a part of the assessment of your profile.
If you use TikTok for creating posts in politics, health care or financial advice, you have to watch how the testing develops further. It can be expected that if the testing process affects the account visibility or distribution, your campaigns will get affected based on the results of this process.
Also, those brands that work with TikTok creators from the above-listed areas of interest should analyze their current relations with those creators to make sure they produce original, properly cited content rather than relying on AI generation tools.
Looking Ahead
There are still some open questions. The company has not provided any information regarding the criteria that qualify an account as spam or what happens after an account is detected in such testing. Moreover, there is still no exact date when the process will start, apart from the vague term of “coming weeks.”
Regarding the content labeling aspect, it does not look like TikTok made any changes to its labeling system in connection with the news about the new committee.
Frequently Asked Questions
Q 1: What is TikTok testing to fight AI-generated spam?
TikTok is testing improvements to its detection systems aimed at accounts that post AI-generated spam, with a focus on politics, financial advice, and medical content.
Q 2: Why did TikTok choose these three categories?
TikTok said these categories are areas where misleading content could affect public trust or personal well-being, and outside research has found health content to be a major source of AI-generated material on the platform.
Q 3: When will TikTok’s new detection testing begin?
TikTok said testing will begin in the coming weeks, though the company has not provided an exact start date.
Q 4: Does TikTok’s detection work at the account level or the video level?
TikTok explained that this testing operates at the account level, meaning it looks at broader account behavior rather than scanning individual videos alone.
Q 5: How many fake accounts has TikTok already removed?
TikTok said it removed more than 86 million fake accounts in the first three months of this year, separate from this new detection testing.
Q 6: What is the C2PA and why did TikTok join its steering committee?
The C2PA is the Coalition for Content Provenance and Authenticity, which develops standards for tracking the origin of digital content. TikTok joined its steering committee to help drive wider adoption of the technology.
Q 7: How many videos has TikTok tagged as AI-generated?
TikTok said it has tagged more than 3 billion videos as AI-generated content using Content Credentials, creator labels, and invisible watermarking.
Q 8: Have other platforms taken similar steps against AI spam?
Yes. YouTube and Meta have both updated policies targeting inauthentic or repetitive content, and Google published a paper on detecting coordinated AI spam networks.
Q 9: Who should be most concerned about this new testing?
Creators and marketers producing political, financial, or medical content on TikTok should pay close attention, since these are the specific categories TikTok is targeting.
Q 10: Has TikTok said what happens to accounts flagged during testing?
No. TikTok has not detailed what actions will follow once an account is flagged, and this remains one of the open questions about the rollout.
