Booksmash's Lust-O-Meter Shows How Innovation Happens

When HarperCollins decided to sponsor a hacking competition called BookSmash, they probably expected the participants to be a rag-tag collection of smart students, hungry young startups, and underemployed misfit coders. It's very unlikely that they expected Nobel Prize winners or seasoned tech entrepreneurs to show up. But, as I pointed out in June, they had made some interesting and fun resources available as part of the competition: 196 full-text books from some popular authors. I'll let you in on a secret: despite what you may hear elsewhere, it's fun, more than anything else, that drives innovation.

The results of the competition were unveiled yesterday. Some of the teams I was already familiar with: I met the BookCities, Coverlist and LibraryAtlas teams at Publishing Hackathon. ReadUp, from the the great folks at ReadSocial, is a neat idea definitely worth checking out. But Text Textures was the submission that popped out at me. The Text Textures team is Mira and Frank Wilczek, a father-daughter team. Frank is a Nobel Prize winning physicist, Mira is a ethical-coding serial tech entrepreneur. (Lyric Semiconductor and Red Panda Security. A new project is BookGobble.)

Text Textures starts out by imagining how fun it would be if you could just skip to the "juicy parts" of a book. It turns out that with access to the full text of a book, a pretty simple combination of weighted word counts supplemented with pacing heuristics allows a text analysis engine to measure things like lustiness (hence the "Lust-O-Meter"), affection, violence and occult themes. By graphing each of these attributes versus page number, it's easy to see where the "juicy bits" of a book are. But that's not where the fun ends. You can density-plot one attribute versus another. And so we find out that "the lustiest scenes in For A Few Demons More appear to have almost no affection". You can plot compare multiple books, and use the measures to decide what sort of book to read next.

I asked Mira about the genesis of Text Textures. She responded:

I've always been neural-net-curious. So when I found myself with a nice nest egg and some free time, I took the opportunity to round out my education. My dad (Frank) has conveniently also been curious about neural nets -- although he was more intrigued by the analogy to human cognition -- so we decided to work through Hinton's Machine Learning lectures on Coursera together. We've been doing fun technical projects together for as long as I can remember. When I was seven, we built a foot-stomping robot using Lego MindStorms. When I was sixteen, we used genetic algorithms to solve N queens.

As we went through the Hinton course, we started to think about real-world problems it might be interesting to tackle using some of those mathematical tools. Eventually we started playing with tracking characters through Sherlock Holmes .... then finding the action scenes where those characters appear ... then looking at other ways to classify scenes ... and thus the underlying idea of Text Textures was born.

The Lust-O-Meter in Text Textures is a fun toy. Which is to say that I would like to be able to play with it myself. I would build a snark-o-meter.  I'm not sure if a "Skip to Good Bits" button is something people want in the reader applications, and even if they wanted it they might not admit it. But eBooks don't have inherent page numbers, so new ways to navigate ebooks would be really useful. It's rather a shame that today's prevailing ebook environment of walled-garden DRM-encumbered marketplaces is hostile to innovations such as Text Textures. Even libraries are prohibited from doing textual analysis of most of the ebooks they buy. And because lustiness data, for example, is not protectable by copyright, rightsholders such as HarperCollins typically deploy restrictive terms of use on anyone they allow to access the full text of their works. It's not enough to open up just a crack for a hacking competition.

Everyone should be able to have fun with their books.

Note: you can vote for Text Textures or any of the other BookSmash submissions until September 27 at 5:00pm EDT by going here.
Update: @skyberrys notes that the Illuminate entry also has roots in #pubhack. I note that it's yet another contribution to the book world by a physicist!