Challenge
Ensuring the quality and reliability of a dataset intended for training a large language model (LLM), while addressing risks related to copyright, bias, and content toxicity.
Description
We conducted a pioneering evaluation of multiple data sources to support the creation of a training dataset for an LLM developed by Translated. The assessment examined the nature, origin, and potential risks associated with the data, identifying key issues related to copyright, fairness, and toxicity.
The findings informed decisions on which data to include, exclude, or modify, contributing to a more transparent, fair, and responsible training process.
