As Artificial Intelligence becomes more sophisticated, it’s opening new doors: from increased efficiency in diagnosis to personalized edtech. And yet, not all of its uses lead somewhere good. Among the most disturbing is the rise of AI-generated child sexual abuse material (CSAM) – a form of content that, while not based on real children, replicates exploitation in realistic ways. Along with the rest of the West, Australia has become one of the first countries in the world to confront the emerging threat head-on.

Rising Numbers, Global Failure

According to an analysis by the Internet Watch Foundation, AI-generated CSAM has risen 380% with 245 confirmed reports in 2024 compared with 51 in 2023. As stated by the ambassador of the Australia-wide organization Act for Kids, Jon Rouse: “This is a stark reminder of the pervasive trauma this offending can cause survivors and, overarchingly, how significant the global failure has been to address it.” 

Behind the “global failure” lies an intricate web of outdated legislation, ineffective moderation, and lack of political urgency in adaptation to the novel technology.

The Growing Threat

While many praise the benefits of the open-source nature of generative AI in improving models and their safety capabilities, with increased researcher access, the broad accessibility has increased the risk of malicious use. In particular, the technology used to generate explicit, hyper-realistic imagery through AI, especially deepfakes, has advanced rapidly. One example is CivitAI, a site created for AI image generation that allows users to browse thousands of models to generate pornographic content and synthetic child sexual abuse imagery, streamlining the nonconsensual AI porn economy (Maiberg, 2023a). A more recent example is a Stanford Observatory investigation that found hundreds of images of CSAM in an open dataset called LAION-5B, which is commonly used in image generation models, such as Stable Diffusion (Thiel, 2023).  

Stable Diffusion, just like many image generation models, relies on a form of content moderation, the failures of which are becoming increasingly obvious: blocking specific keywords in content generation. For one, they have enacted keyword-blocking for words related to the reproductive system to prevent the generation of pornographic material (Heikkilä, 2023). The form of moderation is ineffective when visual synonyms are used, e.g, “dark red liquid” for blood. 

Because AI-generated CSAM doesn’t involve real children in its production, it has occupied a grey area in many Western jurisdictions. Intricacies in statute language on sexually exploitative material criminalize “digitally reproduced visual material,” limiting the prosecutability of the “produced.”

Successes and Challenges

Australia has long established itself as a leader in digital safety. The eSafety Commissioner, an independent statutory organization, has long worked to combat online abuse through international collaboration, education, policy advising, and takedown powers. Meanwhile, the law enforcement agency Australian Centre to Counter Child Exploitation (ACCCE) coordinates intelligence gathering, engages in policy advising, and collaborates with international agencies like INTERPOL and Europol to combat cross-border child exploitation crimes, including AI-generated CSAM. 

In 2024, the Australian government took a significant step forward by introducing the Criminal Code Amendment (Deepfake Sexual Material) Act 2024 which introduced offenses for creation and non-consensual sharing of sexually explicit material online, including deepfakes. By addressing some legal gaps, the legislation empowered the Australian Federal Police (AFP) to prosecute harmful materials even when no real person was involved in their creation. Additionally, pre-existing child exploitation laws cover real and digitally altered child abuse materials. Still, it remains unclear how they apply to non-photographic images that don’t involve actual children or the tools used to create them.

Operation Cumberland – an Europol-led sting operation that, among 23 others, concluded in the arrest of two Australian individuals involved in the production of CSAM by generative AI models, further highlighted the difficulty of prosecuting the borderless crime. As the AI models used to generate CSAM were primarily open-source or hosted on international servers, effective prosecution required international cooperation.

Looking forward

Australia boasts high digital connectivity, plays an active role in internet safety partnerships like the Five Eyes Alliance with countries like Canada and New Zealand as intelligence partners, and a strong history of child protection laws. 

Contrary to the fragmented nature of law enforcement in the United States due to “state rule” and the difficulties of enforceability under Section 230(c)(1) of the First Amendment limiting stronger moderation of content-hosting platforms or the slow nature of EU-wide legislative action, Australia possesses the infrastructure to lead in child protection against the novel form of exploitation.

With the growing recognition of the dangers of image-based abuse and the aforementioned explicit failures of certain models in moderating user-generated content, the country must follow the lead of the UK in going beyond criminalizing the generation of CSAM, but also the possession and distribution of tools to generate such images.