Replika: €5 Million fine for the U.S.-based company – the Italian DPA launches new investigation into AI training practices

The Italian Data Protection Authority ("Garante") has imposed a €5 million fine on the U.S. company Luka Inc., operator of the chatbot “Replika”, and has launched a separate investigation to assess the lawfulness of the personal data processing activities carried out by the generative artificial intelligence system underpinning the service.

What is Replika?

“Replika” is a chatbot offering both text-based and voice interfaces, enabling users to create a “virtual friend” capable of acting as a confidant, advisor, romantic partner, or mentor.

During its initial investigation, the Garante confirmed the existence of the violations previously identified in its February 2023 decision, which had ordered the suspension of the application. Specifically, as of 2 February 2023, the company had failed to identify a valid legal basis for the processing of personal data carried out through the “Replika” service and had provided data subjects with a privacy notice that was incomplete and not compliant with the requirements of the applicable data protection framework (i.e., Regulation (EU) 2016/679 – “GDPR”).

Furthermore, the Garante found that the company had not implemented any effective mechanisms for verifying users’ age – neither at the registration stage nor during use of the service – despite having previously stated that minors were explicitly excluded from the potential user base. However, technical assessments subsequently conducted revealed that the current age verification system implemented by the data controller remains inadequate in several respects.

In light of these infringements, in addition to imposing the administrative fine, the Garante has ordered the company to adopt corrective measures to bring the data processing activities into compliance with the GDPR.

As part of a new investigation, initiated through a formal request for information, the Garante has required Luka Inc. to provide detailed explanations regarding the processing of personal data throughout the entire lifecycle of the generative AI system powering the “Replika” service. In particular, Replika has been asked to provide information on: (i) the risk assessments conducted; (ii) the security measures implemented during the development and training phases of the language model; (iii) the categories and types of personal data processed; and (iv) whether any anonymization or pseudonymization techniques have been applied.

Conclusions
The Replika case once again highlights the challenges associated with the use of generative AI systems that process personal data on a large scale, particularly in the absence of robust legal and technical safeguards. The Garante’s decision reaffirms the importance of transparency, lawfulness of processing, and the protection of minors as core principles of the GDPR.

The newly launched investigation aims to clarify the training practices of the model and to verify whether the measures adopted by the company are capable of ensuring compliance with European data protection law.

This proceeding represents a significant milestone in assessing the effectiveness of safeguards applied to AI systems, whose increasing complexity and widespread adoption demand a careful balance between technological advancement and the protection of individuals’ fundamental rights.