What happens when AI chatbot companies become intermediaries between users and law enforcement? Follow the incentives, discover the outcomes.
Ottawa’s new Digital Safety Act effectively regulates both chatbot inputs and outputs: what may be said in a chatbot, and what a chatbot may say. But faced with unclear obligations and penalties of up to 3 percent of gross global revenue, AI companies will have every incentive to avoid non-compliance by increasingly monitoring, assessing, and disclosing to police Canadians’ chatbot conversations.
Bill C-34 is the first piece of federal legislation to establish legal obligations for companies operating chatbot services, such as OpenAI’s ChatGPT, xAI’s Grok, and Google’s Gemini. On the surface, the Bill accomplishes exactly what many expected of their federal government following the Tumbler Ridge school shooting: regulating how AI companies manage chatbot conversations involving fantasies of violence.
While the bill does not directly prohibit Canadians from saying particular things to chatbot services, it creates obligations and penalties for chatbot services that will shape what users can realistically discuss.
Section 51 will require regulated chatbot services to immediately interrupt chats where users express suicidal ideation, an intention to self-harm, or an intention to commit an act that could cause death or serious bodily harm to an individual. Accordingly, regulated services will be required to monitor everything to identify conversations that meet this statutory threshold. A system cannot reliably identify every prohibited conversation without scrutinizing a vast number of lawful ones. Eyes peeled, and every shadow assumes the shape of a threat. This will, in all likelihood, generate a chilling effect on legitimate exploration: chats about history, politics, literature, terrorism, extremist ideologies, crime, or mental health issues.
The problem ranges beyond self-censorship, however, to exposure to police review.
Section 51 requires chatbot services to refer conversations expressing intentions to commit suicide, harm to self, or harm to others to “crises intervention services that are appropriate to the situation.” No framework for understanding what service may be appropriate to any situation is provided. Presumably, a crisis intervention service could include not just mental health practitioners but also the RCMP.
Section 58(1) goes even further. Regulated chatbot companies will have to submit a safety plan to the Digital Safety Commission disclosing the service’s criteria and processes, if any exist, for escalating to the RCMP any chats where “there is a risk that an individual will commit an act that would cause death or serious bodily harm to another individual.”
While reasonable on the surface, chatbot companies will likely interpret this as encouragement to disclose to police information about sensitive, personal chats. This, in turn, may provide police with access to private communications they would otherwise require judicial authorization to obtain. The Supreme Court of Canada has recognized that individuals maintain a reasonable expectation of privacy in their electronic communications, even when those communications are stored or transmitted through third-party systems. But the combination of ambiguity and significant penalties for non-compliance may pressure chatbot companies to escalate “false positives” to police out of an abundance of caution.
User inputs are not the only target. The bill directly regulates chatbot outputs.
Sections 49, 50, and 53 will require regulated chatbot services to implement measures mitigating the risk of harming users. Harmful behaviour includes posing as a human being or as a medical, legal, or other licensed professional, manipulating users to form emotional attachments to the chatbot, encouraging self-harm, suicide, or acts that could cause serious death or bodily harm to others. Importantly, section 53(e) also prohibits “any other type of behaviour specified in the regulations.”
These regulations do not yet exist.
What a future regulator determines to be “any other type” of harmful chatbot output may have profound implications for Canadians’ access to information. Increasingly, Canadians use AI chatbots not merely for conversation but for research, education, and discovery. No longer an experimental or niche technology, AI chatbots and search tools have become a mainstream method for accessing information. The more Canadians rely on AI systems to locate information, conduct research, and explore ideas, the more consequential government authority over chatbot outputs becomes.
Section 53(e) is, therefore, a “trust the government” provision. It empowers a future Commission to regulate an increasingly important gateway to information without identifying any constraints on that Commission’s authority. Absent statutory clarity, chatbot services may preemptively implement more conservative generative models in order to avoid a charge of distributing harmful outputs. An increasingly significant sector for exploration and discovery will likely narrow as a result. Users will self-censor, and chatbot services will draw upon more conservative models.
Ambiguity creates pressure. Significant penalties create pressure. Bill C-34 contains both. Faced with severe penalties and unclear obligations, AI companies will have powerful incentives to retain more, monitor more, disclose more, and implement increasingly cautious generative models. Alternatively, they’ll have incentives to simply exit the Canadian market.
If this bill passes, the rational course of action for Canadians will be to act as though their AI conversations are being monitored, either by AI companies or by police. The rational course of action for AI companies will be to err on the side of caution at the expense of their users’ autonomy, freedom of expression, access to information, and privacy.
Luke Neilson is Vice President of the Justice Centre for Constitutional Freedoms.
