Booksmash's Lust-O-Meter Shows How Innovation Happens

When HarperCollins decided to sponsor a hacking competition called BookSmash, they probably expected the participants to be a rag-tag collection of smart students, hungry young startups, and underemployed misfit coders. It's very unlikely that they expected Nobel Prize winners or seasoned tech entrepreneurs to show up. But, as I pointed out in June, they had made some interesting and fun resources available as part of the competition: 196 full-text books from some popular authors. I'll let you in on a secret: despite what you may hear elsewhere, it's fun, more than anything else, that drives innovation.

The results of the competition were unveiled yesterday. Some of the teams I was already familiar with: I met the BookCities, Coverlist and LibraryAtlas teams at Publishing Hackathon. ReadUp, from the the great folks at ReadSocial, is a neat idea definitely worth checking out. But Text Textures was the submission that popped out at me. The Text Textures team is Mira and Frank Wilczek, a father-daughter team. Frank is a Nobel Prize winning physicist, Mira is a ethical-coding serial tech entrepreneur. (Lyric Semiconductor and Red Panda Security. A new project is BookGobble.)

Text Textures starts out by imagining how fun it would be if you could just skip to the "juicy parts" of a book. It turns out that with access to the full text of a book, a pretty simple combination of weighted word counts supplemented with pacing heuristics allows a text analysis engine to measure things like lustiness (hence the "Lust-O-Meter"), affection, violence and occult themes. By graphing each of these attributes versus page number, it's easy to see where the "juicy bits" of a book are. But that's not where the fun ends. You can density-plot one attribute versus another. And so we find out that "the lustiest scenes in For A Few Demons More appear to have almost no affection". You can plot compare multiple books, and use the measures to decide what sort of book to read next.

I asked Mira about the genesis of Text Textures. She responded:

I've always been neural-net-curious. So when I found myself with a nice nest egg and some free time, I took the opportunity to round out my education. My dad (Frank) has conveniently also been curious about neural nets -- although he was more intrigued by the analogy to human cognition -- so we decided to work through Hinton's Machine Learning lectures on Coursera together. We've been doing fun technical projects together for as long as I can remember. When I was seven, we built a foot-stomping robot using Lego MindStorms. When I was sixteen, we used genetic algorithms to solve N queens.

As we went through the Hinton course, we started to think about real-world problems it might be interesting to tackle using some of those mathematical tools. Eventually we started playing with tracking characters through Sherlock Holmes .... then finding the action scenes where those characters appear ... then looking at other ways to classify scenes ... and thus the underlying idea of Text Textures was born.

The Lust-O-Meter in Text Textures is a fun toy. Which is to say that I would like to be able to play with it myself. I would build a snark-o-meter.  I'm not sure if a "Skip to Good Bits" button is something people want in the reader applications, and even if they wanted it they might not admit it. But eBooks don't have inherent page numbers, so new ways to navigate ebooks would be really useful. It's rather a shame that today's prevailing ebook environment of walled-garden DRM-encumbered marketplaces is hostile to innovations such as Text Textures. Even libraries are prohibited from doing textual analysis of most of the ebooks they buy. And because lustiness data, for example, is not protectable by copyright, rightsholders such as HarperCollins typically deploy restrictive terms of use on anyone they allow to access the full text of their works. It's not enough to open up just a crack for a hacking competition.

Everyone should be able to have fun with their books.

Note: you can vote for Text Textures or any of the other BookSmash submissions until September 27 at 5:00pm EDT by going here.
Update: @skyberrys notes that the Illuminate entry also has roots in #pubhack. I note that it's yet another contribution to the book world by a physicist!

I Hired a Book

I have an article up at Library Journal about startups that have been getting hired for reading ecosystem jobs over the past three years. The startups that I profile are GoodReads, Wattpad, Readmill, SIPX and Zolabooks.

This view of hiring companies for jobs comes from Clayton Christensen's concept of Milkshake Marketing, explained here.

Christensen describes the case of a fast food restaurant that wanted to improve sales of its milkshakes, but really didn't understand the job that consumers were hiring it for. Customer observations revealed that almost half of the milkshakes sold were to early morning customers who had long boring commutes; the milkshake was being hired to relieve boredom and postpone hunger.

Libraries too need to think about the jobs their users are hiring them to do. Sometimes it's just to relieve boredom or escape bad weather.

silo schematics
by Jerry Yeti

Over Labor Day weekend, I hired a book for a very specific job. I went to an actual bookstore and bought a book to occupy myself during a cross country flight from LAX to EWR, with a connection I was sure to miss in DFW. I chose Hugh Howey's Wool. Given the book's publishing history, you might think it perverse that I bought the print version. Wool was, for a long while, an ebook-only self-published series; Howey did a precedent-setting deal with Simon & Schuster for the print rights only. But consider the job I was hiring it to do: why would I buy an ebook that I couldn't read during the long waits on the runways? (My son had dibs on the window seat.)

Having spent the outbound trip coding new features for unglue.it, I needed a mental break, and Wool enveloped me in a completely absorbing self-contained world that did the job I hired it for quite nicely. Wool tells the story of the inhabitants of an isolated post-apocalyptic silo, built and supplied with technology from the year 2012. "IT" is the villain.

There was one continual distraction for me. My engineer's brain couldn't stop calculating the dimensions of the silo. The narrative made the silo seem large- after all it's the whole world for its inhabitants. But of necessity, it has to be compact. But how compact? Towards the end of the fifth part, I get some measurements: the bottom 8 floors are flooded, a depth of "70 to 80 feet". So 10 feet per level. That's pretty cramped!

Wool seems so relevant to our current environment, especially today's revelations about the extent of the NSA's decryption effort. Books have a way of doing jobs other than what you hired them for, just like the best employees. Just like libraries. Think about it.

Update 9/9: I purchased the DRM-free ebook package for Shift, the second book in the series. Which so far reveals that the silo tech is supposed to be from 2050 and that the top level is 10 meters, not feet.