AI bots are destroying Open Access

There's a war going on on the Internet. AI companies with billions to burn are hard at work destroying the websites of libraries, archives, non-profit organizations, and scholarly publishers, anyone who is working to make quality information universally available on the internet. And the technologists defending against this broad-based attack are doing everything they can to preserve their outlets while trying to remain true to the mission of providing the digital lifeblood of science and culture to the world.

Yes, many of these beloved institutions are under financial pressures in the current political environment, but politics swings back and forth. The AI armies are only growing more aggressive, more rapacious, more deceitful and ever more numerous.

I'm talking about the voracious hunger of AI companies for good data to train Large Language Models (LLMs). These are the trillion-parameter sets of statistical weights that power things like Claude, ChatGPT and hundreds of systems you've never heard of. Good training data has lots of text, lots of metadata, is reliable and unbiased. It's unsullied by Search Engine Optimization (SEO) practitioners. It doesn't constantly interrupt the narrative flow to try to get you to buy stuff. It's multilingual, subject specific, and written by experts. In other words, it's like a library.

At last week's Code4lib conference hosted by Princeton University Library, technologists from across the library world gathered to share information about library systems, how to make them better, how to manage them, and how to keep them running. The hot topic, the thing everyone wanted to talk about, was how to deal with bots from the dark side.

Bots on the internet are nothing new, but a sea change has occurred over the past year. For the past 25 years, anyone running a web server knew that the bulk of traffic was one sort of bot or another. There was googlebot, which was quite polite, and everyone learned to feed it - otherwise no one would ever find the delicious treats we were trying to give away. There were lots of search engine crawlers working to develop this or that service. You'd get "script kiddies" trying thousands of prepackaged exploits. A server secured and patched by a reasonably competent technologist would have no difficulty ignoring these.

The old style bots were rarely a problem. They respected robot exclusions and "nofollow" warnings. The warning helped bots avoid volatile resources and infinite parameter spaces. Even when they ignored exclusions they seemed to be careful about it. They declared their identity in "user-agent" headers. They limited the request rate and number of simultaneous requests to any particular server. Occasionally there would be a malicious bot like a card-tester or a registration spammer. You'd often have to block these based on IP address. It was part of the landscape, not the dominant feature.

The current generation of bots is mindless. They use as many connections as you have room for. If you add capacity, they just ramp up their requests. They use randomly generated user-agent strings. They come from large blocks of IP addresses. They get trapped in endless hallways. I observed one bot asking for 200,000 nofollow redirect links pointing at Onedrive, Google Drive and Dropbox. (which of course didn't work, but Onedrive decided to stop serving our Canadian human users). They use up server resources - one speaker at Code4lib described a bug where software they were running was using 32 bit integers for session identifiers, and it ran out!

The good guys are trying their best. They're sharing block lists and bot signatures. Many libraries are routinely blocking entire countries (nobody in china could possibly want books!) just to be able to serve a trickle of local requests. They are using commercial services such as Cloudflare to outsource their bot-blocking and captchas, without knowing for sure what these services are blocking, how they're doing it, or whether user privacy and accessibility is being flushed down the toilet. But nothing seems to offer anything but temporary relief. Not that there's anything bad about temporary relief, but we know the bots just intensify their attack on other content stores.

The view of MIT Press's Open-Access site from the Wayback Machine.

The surge of AI bots has hit Open Access sites particularly hard, as their mission conflicts with the need to block bots. Consider that Internet Archive can no longer save snapshots of one of the best open-access publishers, MIT Press because of cloudflare blocking. (see above) Who know how many books will be lost this way?  Or consider that the bots took down OAPEN, the worlds most important repository of Scholarly OA books, for a day or two. That's 34,000 books that AI "checked out" for two days. Or recent outages at Project Gutenberg, which serves 2 million dynamic pages and a half million downloads per day. That's hundreds of thousands of downloads blocked! The link checker at doab-check.ebookfoundation.org (a project I worked on for OAPEN) is now showing 1,534 books that are unreachable due to "too many requests". That's 1,534 books that AI has stolen from us! And it's getting worse.

Thousands of developer hours are being spent on defense against the dark bots and those hours are lost to us forever. We'll never see the wonderful projects and features they would have come up with in that time.

The thing that gets me REALLY mad is how unnecessary this carnage is. Project Gutenberg makes all its content available with one click on a file in its feeds directory. OAPEN makes all its books available via an API. There's no need to make a million requests to get this stuff!! Who (or what) is programming these idiot scraping bots? Have they never heard of a sitemap??? Are they summer interns using ChatGPT to write all their code? Who gave them infinite memory, CPUs and bandwidth to run these monstrosities? (Don't answer.)

We are headed for a world in which all good information is locked up behind secure registration barriers and paywalls, and it won't be to make money, it will be for survival. Captchas will only be solvable by advanced AIs and only the wealthy will be able to use internet libraries.

Or maybe we can find ways to destroy the bad bots from within. I'm thinking a billion rickrolls?

Notes:

1. I've found that I can no longer offer more than 2 facets of faceted search. Another problematic feature is "did you mean" links. AI bots try to follow every link you offer even if there are a billion different ones.
2. Two projects, iocaine and nepenthes are enabling the construction of "tarpits" for bots. These are automated infinite mazes that bots get stuck in, perhaps keeping the bots occupied and not bothering anyone else. I'm skeptical.
3. Here is an implementation of the Cloudflare Turnstyle service (supposedly free) that was mentioned favorably at the conference.
4. It's not just open access, it's also Open Source.
5. Cloudflare has announced an "AI honeypot". Should be interesting.
6. One way for Open Access site to encourage good bot behavior is to provide carrots to good robots. For this reason, it would be good to add Common Crawl to greenlists: https://commoncrawl.org/ccbot
7. Ian Mulvaney (BMJ) concurs. 

Why (Code4) Libraries Exist

"We have one more talk left and we'll see how Eric Hellman manages to do the closing talk, and if he deserves the slot or not" was how Ed Corrado introduced me yesterday at the Code4Lib 2011 Conference. Thanks, Ed. A little pressure never hurt anyone. I suppose I had it coming. Here's the abstract I wrote for the talk:

Libraries have historically delivered value to society by facilitating the sharing of books. The library "brand" is built around the building and exploitation of their collections. These collections have been acquired and owned. As ebook readers become the preferred consumption platform for books, libraries are beginning to come to terms with the fact that they don't own their digital collections, and can't share books as they'd like to. Yet libraries continue to be valuable in many ways. In this transitional period, only one thing can save libraries from irrelevance and dissipation: Code.

Talks were selected by vote of Code4Lib participants, which increased my motivation to do a good job.

You can view the talk yourself via the excellent streaming video archive from Indiana University. (My talk starts at 171:24) I was told that at times, there were as many as a thousand people viewing the live stream, and having a twitter and IRC backchannel allowed remote users to join in the snark: "not sure why people voted for this presentation" and the inexplicable: "This presentation is good, but I would have expected multiple Celtic fonts."

I gave a brief intro to the trends in ebook adoption, and went through an explanation of the economics of book lending that I wrote about last year. I urged everyone to break the wifi by downloading Eli Neiburger's "Libraries are screwed" talk. Then I suggested that Code4Lib participants were the people with the power, placement and passion to drive the changes that libraries will need to survive.

Last week, I sent an email to the Code4Lib list, asking for information about how many people were developing or maintaining software (broadly defined) in libraries, and what fraction of the library staff they represented. The results are displayed in this graph:

Notes on the graph: Red dots are from Code4Lib respondents; the blue dots are taken from a peer group survey made available to me by one respondent of the "technology staff" in a cohort of mid-sized US Universities. The determination was made by title alone, without knowledge of actual job responsibilities. My own survey of job titles in libraries suggests that many job title terms such as "technical services" encompass responsibilities ranging from zero to full time involvement in software, even when broadly defined. One Code4Lib respondent reported that he was the only developer in his library and had only just been hired; the peer group survey indicated 2 technical staff for the same library. To place the data on the same chart, I've arbitrarily divided the technical staffing for the peer-group survey by 3.

One thing to remark in this graph is that most developers in libraries are very dispersed. Although they're embedded in library operations, they're often the only person at the location that does any coding at all. Many of my respondents reported that only a fraction of their responsibilities involved coding. As a result they tend to seek a peer group outside their institution, for example, in the various Code4Lib forums.

One reason that such a group is nonetheless well placed to effect change in libraries is the changing nature of software development. Today, software development is more and more about the use and deployment of software modules and connecting to APIs. The presence at Code4Lib of several vendor representatives promoting their APIs: OCLC, SciVerse, SerialsSolutions, Mendeley is testament to the importance of putting powerful tools into the hand of people who can see what to do with them.

To do this, libraries need to recognize the importance of code. Developers in libraries need time and support to develop their skills and focus on developing new ways for libraries to deliver value. Although I suggest a focus on Spaces, People, and Communities, I don't really know the answer. I just know that a lot of experimentation is needed.

I worry about public libraries. They are the data points at the bottom of the graph. I fear that publics will not have the technical capacity to do anything differently from what they've always done, except less of it, because of budget cuts.

Beer4Lib

Notes: In my talk, I mention a talk from earlier in the morning about how libraries are using software to improve their spaces. The talk I was referring to was by Jason Casdan and Joyce Chapman at NCSU. I should also note that a recurring theme of the conference was that the one thing that people most needed from mobile apps was the opening hours of the library locations. That's worth thinking about.

Libraries Creating Connections

"Libraries should stop focusing on their collections and start focusing on their communities" is something I've said before. Today, I learned that I may need to revise that. Two presentations today at Code4Lib 2011 showed how a library might do both at the same time.

Nell Taylor and Margaret Heller talked about the Chicago Underground Library, a grass-roots special collection that focuses on a specific place, Chicago.

The Chicago Underground Library is an all-inclusive collection of Chicago-specific media, produced by and for the community. Through our unique indexing of contributors and our open venue, we provide a space for individuals, organizations and ideas to come together, and creative connections to emerge.

They'll take anything into their collection, even a book of poems bought for a dollar made by a local guy Taylor met on a bus. Their goal is to use the collection and "obsessive cataloging" to create connections between people and places.

I'm guessing that a hot topic at Code4Lib 2012 will be Search Engine Optimization (SEO), but Taylor and Heller were the first and only to mention it this year. They make the metadata on their item pages easy for search engines to consume, and that's what drives the connections to people. People google themselves and their places, and lots of things happen and have happened that never enter the pages of The Chicago Tribune.

The Underground Library has a lot in common with the Chinatown Storefront Library project that I wrote about last year. They're running on a shoestring, living in borrowed space, and relying on volunteer labor. With some luck, they'll get some funding and some space of their own, but they have no shortage of courage and audacity.

The most surprising talk today came in the "lightning session". Haruki Ono, a student at University of Tsukuba in Japan, told us about project Shizuku. Shizuku is named after the heroine of Hayao Miyazaki's movie Whisper of the Heart. Shizuku loves reading books, and goes to the library every day. She notices that one other user has signed the checkout slip for many of the books she loves. To make a long story short enough for this blog post, they end up getting married.

Code4Lib 2011 Conference Lightning Talks - Haruki Ono

View more presentations from milkyalab.

Project Shizuku aims to use library circulation data to connect users, not to other books they would like to read, but to other users. When I wrote about "Biblio-Social Objects" in December, somehow it didn't occur to me how much local libraries could leverage their non-virtuality to create relationships and build community.

You learn something new and meet someone new every day, especially at Code4Lib.

Toys and Tools vs. the Enterprise at Code4Lib

© CERN

In 1991, the world's top researchers into hypertext met in a hotel in San Antonio, Texas. One poster presented there was entitled "An Architecture for Wide Area Hypertext", by a guy from CERN. Nine years later, I attended the same meeting in the same hotel. Attendees who had also been at the earlier meeting told me that the uniform reaction to the poster had been what what I'd describe now as "meh". It was too simple, not enough expressive power. There was nothing new, nothing interesting. Who could possibly care about about the physicist's stupid little toy hypertext system.

That physicist is now a Knight Commander of the Order of the British Empire, and Sir Tim's little toy system is the today's World Wide Web. Here's a few of the enterprise-ready systems that the conference organizers of HT 1991 thought were more important than "the web":   

• Industrial Strength Hypermedia: Requirements for a Large Engineering Enterprise
• Implementing Hypertext Database Relationships through Aggregations and Exceptions
• Applications Navigator: Using Hypertext to Support Effective Scientific Information Exchange

You get the idea. If you make a list of the most important technologies for libraries and publishing today, your list will include a lot of things in addition to the web that started out as toys, and were derided as such for years after they became important building blocks. Linux, unix, mySQL, perl, ruby, apache web server, and so on. Even Google started out with Lego blocks as key components. The list of software technologies that began as enterprise-class applications is smaller and less loved.

This morning at the Code4Lib Conference in Bloomington, Indiana, Brad Wheeler gave a welcoming talk. He's a Vice President for Information Technology and Chief Information Officer at Indiana University and Chairman and Co-Founder of the Kuali Foundation. He emphasized how important it is for libraries to submit to "volitional interdependence for macro solutions". I think he was trying to say that libraries need to pool their software development efforts and stop focusing on their local needs and peculiarities. But there was one thing he said that I strongly disagreed with. He said that libraries should stop move beyond building toys.

Wheeler's remark was in the context of a story about his impression of a Digital Library Federation (DLF) meeting two years ago. He thought the projects being reported there were too small and too focused on individual libraries. DLF has fresh new leadership in the person of Rachel Frick, and a new structure as part of CLIR, but I'm not sure that Wheeler's criticism of "the old DLF" is justified.

Libraries need to build more toys, especially in this time of tectonic changes in the ways that users interact with the information of the world. By toys, I mean simple experiments that do interesting things, or tools that focus on solving specific problems. The first day of Code4Lib was replete with examples of toy projects. Karen Coombs from OCLC showed 10 different toys, the most interesting a mashup of geocoded library subject headings with google maps. Josh Bishoff from the University of Illinois showed how the mess of links on his library's homepage could be distilled down to a nifty mobile webapp, complete with local bus schedules. Demian Katz showed how VuFind, a tool that has already made a transition from toy to essential tool is being pushed to be even more flexible. Jay Luker, from ADS, did the same with Blacklight and Solr.

For me the highlight was Scott Hanrath's report on Anthologize, a WordPress plugin designed to turn blogs into ebooks. The initial work on Anthologize was done using a "one week one tool" process, and his account of how 12 strangers banded together to produce a working product in one week was truly inspiring. They even included 4 user interviews in developing their user experience, and achievement which proved to be very difficult to pull off, because of the tight, parallelized development schedule.

The non-toy approach to software development was also on display. Tim McGeary of Lehigh University reported on the Kuali OLE project, which has attracted $5 million in funding from Mellon Foundation and others. OLE has an impressive, state-of-the-art, three tiered modular, buzzword-compliant software architecture and specification set. But after a year of work involving multiple committees, coding has only started last week. (Update 2/9/2011: Kuali's funding for writing code has only been in place for 6 months) It all looks very good, but I'm yet to be convinced that the end result will turn out to be what the market needs.

Development of things like OLE is expensive. Georgia PINES spent about a million dollars in its successful developing Evergreen, perhaps the most recent analog to OLE. The advantage to developing toys is that failure is not expensive. By spending so much on OLE, Kuali is putting a lot of eggs in one basket, and if their extensive committee work has failed to correctly predict what the market requirements in 5 years will be, they won't get another shot at it. In contrast, developing toys lets you try a lot of things out. Most will die, but the few of them that manage to solve sticky problems will get picked up and will grow into essential infrastructure.