{"id":74919,"date":"2024-04-08T15:00:35","date_gmt":"2024-04-08T15:00:35","guid":{"rendered":"https:\/\/creativecommons.org\/?p=74919"},"modified":"2024-04-08T19:04:39","modified_gmt":"2024-04-08T19:04:39","slug":"exploring-a-books-data-commons-for-ai-training","status":"publish","type":"post","link":"https:\/\/creativecommons.org\/2024\/04\/08\/exploring-a-books-data-commons-for-ai-training\/","title":{"rendered":"Exploring a Books Data Commons for AI Training"},"content":{"rendered":"

\"A<\/span><\/p>\n

Our work on copyright has long focused on supporting libraries and archives in the service of their missions to preserve and ensure access to culture. Our 2022 copyright reform <\/span>agenda<\/span><\/a> centers those sorts of institutions (and more generally GLAMs) and the critical role they play in society. Among other things, that agenda calls attention to the ways in which copyright might impede libraries and archives who wish to make their collections available for research uses, including use for AI training in order to fulfill their public interest missions.<\/span><\/p>\n

That issue \u2013 AI training \u2013 has become ever more relevant. The concept of mass digitization of books, including to support text and data mining, of which AI training is a subset, is not new. But AI training is newly of the zeitgeist, and its transformative use makes questions about how we digitize, preserve, and make accessible knowledge and cultural heritage salient in a distinct way.<\/span><\/p>\n

In 2023, multiple news publications <\/span>reported<\/span><\/a> on the availability and use of a dataset of books called \u201cBooks3\u201d to train large language models (LLMs), a form of generative AI tool.\u00a0 The Books3 dataset contains text from over 170,000 books, which are a mix of in-copyright and out-of-copyright works. It is believed to have been originally sourced from a website that was not authorized to distribute all of the works therein. In lawsuits brought against OpenAI, Microsoft, Meta, and Bloomberg related to their LLMs, the use of Books3 as training data was specifically cited.\u00a0<\/span><\/p>\n

The Books3 controversy highlights a critical question at the heart of generative AI: what role do books play in training AI models, and how might digitized books be made widely accessible for the purposes of training AI for the public good? What dataset of books could be constructed and under what circumstances?\u00a0<\/span><\/p>\n

Earlier this year, we collaborated with <\/span>Open Future<\/span><\/a> and <\/span>Proteus Strategies<\/span><\/a> on a series of workshops to explore these questions and more. We brought together practitioners on the front lines of building next-generation AI models, as well as legal and policy scholars with expertise in the copyright and licensing challenges surrounding digitized books. Our goal was also to bridge the perspective of stewards of content repositories, like libraries, with that of AI developers. A \u201cbooks data commons\u201d needs to be both responsibly managed, and useful for developers of AI models. Today, we\u2019re releasing a <\/span>paper<\/span><\/a> based on those workshops and additional research.\u00a0<\/span><\/p>\n

While this paper does not prescribe a particular path forward, we do think it\u2019s important to move beyond the status quo. Today, large swaths of knowledge contained in books are effectively locked up and inaccessible to most everyone. Large companies have huge advantages when it comes to access to books for AI training (and access to data in general). At the same time, as the paper highlights, there are already relevant examples of nonprofit and library-led efforts to provide responsible, fair access to books for many more people, not just the privileged few. We hope this paper can support further research, collaboration and investment in this space.<\/span><\/p>\n

Read the full paper<\/a>.\u00a0<\/strong><\/p>\n","protected":false},"excerpt":{"rendered":"

What role do books play in training AI models, and how might digitized books be made widely accessible for the purposes of training AI? What dataset of books could be constructed and under what circumstances? A new paper investigates the concept of a responsibly designed, broadly accessible dataset of digitized books to be used in training AI models.<\/p>\n","protected":false},"author":10,"featured_media":65658,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_monsterinsights_skip_tracking":false,"_monsterinsights_sitenote_active":false,"_monsterinsights_sitenote_note":"","_monsterinsights_sitenote_category":0,"footnotes":""},"categories":[21],"tags":[143,254,417,624,840,894],"acf":[],"_links":{"self":[{"href":"https:\/\/creativecommons.org\/wp-json\/wp\/v2\/posts\/74919"}],"collection":[{"href":"https:\/\/creativecommons.org\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/creativecommons.org\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/creativecommons.org\/wp-json\/wp\/v2\/users\/10"}],"replies":[{"embeddable":true,"href":"https:\/\/creativecommons.org\/wp-json\/wp\/v2\/comments?post=74919"}],"version-history":[{"count":4,"href":"https:\/\/creativecommons.org\/wp-json\/wp\/v2\/posts\/74919\/revisions"}],"predecessor-version":[{"id":74924,"href":"https:\/\/creativecommons.org\/wp-json\/wp\/v2\/posts\/74919\/revisions\/74924"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/creativecommons.org\/wp-json\/wp\/v2\/media\/65658"}],"wp:attachment":[{"href":"https:\/\/creativecommons.org\/wp-json\/wp\/v2\/media?parent=74919"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/creativecommons.org\/wp-json\/wp\/v2\/categories?post=74919"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/creativecommons.org\/wp-json\/wp\/v2\/tags?post=74919"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}