EPUB has a Steep Road Ahead (Notes from Open Book Hack 2014)

People I talk to about ebook technology belong to one of two camps.

1. EPUB3 is the future of ebooks. 
2. EPUB3 has lost the ebook format war because no one is supporting it. 

Here I'm helping Fortitude fix a bug.
(photo by Ray Schwartz)

So it was very enlightening to join a group of developers at New York Public Library last weekend at Open Book Hack 2014. Sponsored by NYPL Labs and the Readium Foundation, the event was convened to examine the challenges of the "Open Book" on the Web. Since I had written that another publishing hackathon "pretty much ignored ebooks", I felt compelled to attend, and write a report.

Open Book Hack was the first event I've been to where developers were actually working on EPUB, the format that is emerging as the underpinning of the digital book industry. This was both encouraging and discouraging. Encouraging because you could start to glimpse possibilities that EPUB will enable, and discouraging because the road ahead looks so steep.

Open Book Hack attracted a very stimulating mix of developers from around the world (who were in New York for Digital Book World) and local developers looking for interesting problems. Notably, there was a very impressive contingent of students from New York's Flatiron School. The projects are listed on a github wiki.

I was very eager to meet the folks from Berkeley's iSchool that have been working on epub.js, a javascript package that lets you read EPUBs from the web in your browser. One of them, Jake Hartnell, is also on the team building Hypothes.is, a tool designed to let everyone annotate the web (and one of the sponsors of the event). (The others were AJ and Fred.) Sooner or later you'll see these tools added to Unglue.it. But there's still a lot of work left to do.

For example, the project at Open Book Hack with the highest ratio of usefulness to impressive-sounding-ness was the project to add scrolling to epub.js (check it out, it works!). That's right, they added an option (mostly) to make chapter 2 come below chapter 1. You will soon be able to use ebooks on the web using an open-source 9th century reading interface.

There was one prize awarded. The winning project, "See and Read" allowed two people with two screens to interact via an ebook. Very nicely executed and well-deserved, but if I told you that I had invented something that allows two people to interact with a book, you would say "oh?" because you don't really need 100 billion transistors and two glowing screens to interact with a book.

Another project "Breadcrumbs and Beanstalks", extended Harvard's StackLife interface to enable ebook browsing similar to that found in a physical collection of books, with 2 dimensional browsing (the 2 axes being publication date and the general-to-specific axis of subject headings. It looked a bit clunky, but thar's gold in there somewhere.

Hugh MacGuire, Max Fenton, Jean Kaplansky, and Fendi M were doing something brave with linking in  PressBooks, but having already done too much linking in my day, I decided not to understand it. A team from Sophia University in Japan was doing something clever with textbooks and Readium. There was work on converting PDF to EPUB and a project to use Phonegap to make apps from EPUBs.

The project I joined up with focused on making a book club application around a shared EPUB reader. (It works, too!) We based it on epub.js and Hypothes.is. I wasn't very useful to the effort, because the three Flatiron-trained Ruby-on-Rails developers in our group were too awesome for my plodding-but-powerful python to compete with. So I helped with exploration and documentation of the Hypothes.is API, and finding bugs and deployment gotchas in epub.js. I now know how to configure CORS for buckets in S3. Yeah that was my weekend. I also fixed a bug by staring, in an intimidating way, over someone's shoulder. Ah, good times.

What became apparent to me in working with these tools was that the freshly trained developers got everything to work by un-EPUB-ing everything. The web platform just works, with the one exception being that centering text blocks in CSS just doesn't, unless you look away from the screen. The EPUB platform always throws something in your way, for reasons that even StackOverflow doesn't explain. Ruby Zips won't unzip. Cross Sites won't request. Java Scripts won't bind.

EPUB's competition isn't Amazon and KF8 fixed layout, it's the web and HTML5 and its huge gravitational pull. For 90% of ebooks, the benefits of EPUB over HTML are scant (because EPUB is based on HTML!) and the development barriers are significant. It's been years, and still EPUB authoring tools aren't mature or mass market. Deployment tools are barebones.

Don't get me wrong. I'm still betting big on EPUB, but dammit, Publishing Industry, for an $80 billion pillar of modern society, you're investing a nanoscopic amount on your basic infrastructure (i.e. EPUB), despite the herculean efforts of the people I met last weekend.

Notes:

1. I was really impressed with the Flatiron School students I worked with. If the rest of the students are anything like Edina, Tiff, and Dan (Ivan helped a bit, too), they are going to have a huge impact on the New York area economy. Maybe I should learn RoR.
2. Bill McCoy has done an amazing job bringing people together under the IDPF and Readium umbrellas. Imagine what he could do with financial support commensurate to his task. Perhaps he should take up bootlegging.
3. Jake wrote up his impressions, too.
4. As have Virginie Clayssen and Camille Pène, in French.
5. Would have posted sooner, but MILESTONE IN UNLUE.IT.

In 2013, eBook Sales Collapsed... in My Household.

2013 was not the year the ebook industry was expecting. We hoped that ebooks would continue their explosive year-on-year revenue growth, and that the replacement of print by digital would proceed apace. We suspected that the growth of ebook sales might moderate, because, as HarperCollins CEO Brian Murray told Publisher's Weekly, "Nothing grows by triple digits for too long." But just as CDs replaced vinyl and digital downloads replaced CDs, it seemed obvious that the the age of the printed book was nearing its end; the century of the ebook was dawning.

We got a few things right. Internationally, ebook sales growth was strong. Print continued its slow decline. Bookstores continued to close. But for some reason, ebook sales in the US stopped increasing. And even started declining!

There are many possible explanations for this turn of events. There are technicalities with the data collection, particularly with publishers such as Amazon's imprints that don't report their sales numbers. Young Adult sales dropped steeply, as there was no smash hit to follow on the huge success of Hunger Games. 50 Shades of Gray didn't turn out to be a lasting relationship. And there's been a downward trend on prices, particularly as publishers start to use dynamic pricing to stimulate sales. But it seems to me that something in the environment is changing, more than just a market maturation.

Amazon probably has enough data to understand what's happening, but they're notoriously opaque about reporting numbers. On the other hand, they're quite good about reporting to customers what they've bought. So I decided to analyze my own household's Amazon data. I had the impression that my family was spending less on ebooks, but I wasn't sure, because they still seem to spend all hours of the day reading. The results were kind of shocking.

The graph shows my household Kindle ebook purchases from 2009-2013. As you can see, 2013 marked a steep drop from the 2009-2011 peak years of about $1000 per year.

I don't buy Kindle ebooks myself (I buy ePub only, so I can hack on them) but other members of my household have bought quite a lot. The average price paid is about $7, and this has held quite steady. But in 2013, Kindle purchases stopped almost completely, and they were not replaced by purchases on other platforms.

Based on detailed "interviews" with the subject ebook purchasers, here are some non-factors in this collapse:

1. "Netflix-for-Books" services. Nobody but me has heard of them.
2. Kindle Owner's Lending Library. Despite an well-used Amazon Prime subscription, they haven't figured out how to use it for ebooks.
3. Our public library. Nobody but me has used it for ebooks.
4. Piracy. As if!

The two main reasons for this spending collapse turn out to be:

1. The Kindle acquired in early 2009 reached end-of-life due to a cheaply made power cord, and was replaced by an iPad. The lack of in-app purchase for the Kindle App has resulted in a significant impediment to Kindle purchases. The iBookStore has not attracted a single ebook purchase.
2. The iPad owner now spends the vast majority of her reading time in fan-fiction websites, mostly fanfiction.net and ArchiveOfOurOwn.org. Same for the iPad borrower, but a different mix of websites.

I find it worrying that the Justice Department pursued an big antitrust suit against Apple and 5 of the big 6 publishers, won, and despite making an issue of Apple's in-app purchase ban in iOS, it seems to have lost the argument with Judge Cote. We'll see how that turns out.

It's worth paying close attention to the fan fiction sites. After all, 2012's biggest revenue engine for the book industry, 50 Shades, was a repackaged fanfic. On an iPad with a decent internet connection, the fanfic sites work better than ePubs. They link and they script. Just try making a link from one ePub to another and you'll get my point.  They deliver content in smaller, more addictive chunks, and they integrate popular culture MUCH more effectively than books do, for reasons relating primarily to copyright. The authors are responsive and deeply connected to readers; they often ARE the readers!

There's a fanfic site to appeal to every reader; I highlighted Wattpad earlier this year. ArchiveOfOurOwn.org ("AO3"), a project of the Organization for Transformative Works, a non-profit, experienced the growth in 2013 that was missing from the ebook sector. The number of works hosted by AO3 doubled to just under a million works, covering almost 14,000 "fandoms". (A good example of a fandom is the "Dragonriders of Pern" fandom, which currently hosts 534 works). Fanfiction.net, an advertising supported site, hosts almost 2000 fandoms and over 1.3 million works, more than half of which are in the Harry Potter or Twilight fandoms. Game oriented discussion forums also engage in fanfiction. (Popular in my house is spacebattles.com)

My anecdata might be completely anomalous, although Amazon, a very data-driven company, seems to be aware of the same phenomena. They've been making the Kindle into a full featured tablet to go head-to-head with the iPad. They've also launched a fanfic site called Kindle Worlds, which has 15 worlds and 341 works.

Early stage venture capitalist Josh Kopelman says that many of the best opportunities for startups are not those in expanding markets. "We love investing in technologies and business models that are able to shrink existing markets. If your company can take $5 of revenue from a competitor for every $1 you earn – let's talk!",  he has written on his firm's website. Kopelman founded Half.com in the early days of the internet, a company which shrank the book market by getting people to resell the books they had just bought for a fraction of the price of a new book. Microsoft's Encarta shrank the Encyclopedia business from $1.2B to $600M before Wikipedia shrank the business by another 90%.

In 2014, I'm guessing it's the book publishing industry's time to shrink. A convergence of tech startups, tech monsters, and tech non profits seems to be ready for the assault. The fanfic sites, the Wattpads, the Project Gutenbergs and the Manybooks, the Readmills, the Leanpubs and the Smashwords (and I hope the Unglue.its); these are people building the foundations of a creative industry that will flourish even if the ebook sales collapse that I see around me spreads to your house as well.

Happy New Year!

A Scientist Needs More

I visited my high school French teacher at the end of October. She's retired and lives in Alsace. My French stays with may, mostly unused, but something else I learned from her has stayed with me- she showed me how to eat steamed artichokes. Whenever I eat and enjoy artichokes that way, my mouth remembers. So often, we give something to other people, and it it lives on, having an impact out of proportion to the original gift. A girl in my Calculus class introduced me to Bruce Springsteen and The Wild, the Innocent and the E Street Shuffle. Wherever you are, Sarah Strong, thank you, whether you remember or not!

On October 29, I lost a friend from graduate school, Kanji Yoh (陽完治), and I've been reflecting on so many things I learned from Kanji that have remained with me over the years. But what I learned has been hard for me to articulate, because it's not like artichokes or the Boss. OK, I blame Kanji for taking me to the Opera my first and only time, so I suppose I can thank him for helping me cross that off my list. But there's a lot more.

Kanji and I both started graduate school at Stanford the same year. For me graduate school was getting back on a familiar academic track towards being a scientist. For Kanji, graduate school was a step off the conventional track. He had been an engineer at Hitachi, in Japan, at the sort of job where you were expected to work for the company your entire working life. And it was reasonable to expect the company to take care of you in return. Kanji was a rare, remarkable, courageous creature, and my advisor recognized him as such. Kanji needed something more.

Valuing that "more" is what I learned from Kanji.

Kanji and I shared a tiny, sloped-ceiling office with 2 other students in the attic of Stanford's McCullough Building for four years. We were among the first batch students of a new Professor, "Coach" Jim Harris, and we were charged with building a new research lab and launch a new research program. Kanji would ask me interesting questions, and I'd startle him with odd conjectures and amazing facts.

Kanji worked on a counterintuitive idea, to engineer p-channel transistors from layers of Gallium Arsenide instead of the usual n-channel transistors. It was a sort of semiconductor jiujitsu, flipping the weak characteristics of a material into strengths, so as to balance two complementary transistors and achieve both speed and low power. It was an elegant idea; the job of a researcher is to see whether physical reality has the same beauty as the mathematical description.

My wife and I spent a memorable Thanksgiving with Kanji, his wife and some friends. We rented a condo at Lake Tahoe, went skiing, and did a real Thanksgiving turkey. Then we did some more skiing; Kanji was so eager to get out on the slopes early, and there was a foot of fresh snow.

After a post-doc at IBM Watson Labs, Kanji got a job as an Assistant Professor in Osaka. It was a difficult time for him, because the strict Japanese academic system doesn't allow much freedom for junior Faculty. I had a chance to visit him once. We walked around his neighborhood and talked about cicadas, which were making a huge noise. I told him that New Jersey cicadas, unlike Japanese cicadas, come in broods that appear in 13 or 17 year cycles. Always a prime number to fool cyclical predators. Kanji's face lit up with wonderment at the beauty of this fact. "Really?" he said.

Kanji persevered in Osaka and got a position at Hokkaido University in Sapporo. His work there covered quantum dots and wires, graphene transistors, and spin polarized tunneling. His recent work has been on "spintronics", trying to find ways to use and manipulate electron spins to do calculations with quantum mechanics. When I left semiconductor physics 15 years ago, these things were just starting to be thought about and the idea seemed so beautiful and futuristic and impossible.

Kanji played both the violin and flute and performed with three different ensembles. He was just as serious about his music as he was about his semiconductors. He became a fan of  Soprano Anna Netrebko, and would fly around the world to go to her performances. I was a bit shocked when he first told me about this, especially the part about the protocol for delivering roses to the star after a performance. But it's inspired me to take more seriously some of things I've long thought would be fun to do, but who has the time? Like visiting my high-school French teacher in Alsace and drinking Auxerrois with her friend the viticulteur.

I miss him. I'm sure Kanji's wife, a translator and prize-winning poet, and his two kids feel a great absence. They're not alone.

Physics and Testosterone, Part 5: Missing Nobelists

Tomorrow morning at 9AM CET, the Nobel Prize Lectures in Physics will be streamed live on the internet. François Englert will speak about The BEH Mechanism and its Scalar Boson. Peter Higg's talk is entitled Evading the Goldstone Theorem. Robert Brout, the B in BEH, will not share in the Prize. Not because he didn't make a prize-worthy contribution, but because he didn't live long enough. Brout died two and a half years ago, at the age of 82. Higgs, 84, gets the name recognition because he noticed that the "mechanism" proposed by Brout and Englert implied the existence of a particle. Englert, at 81 is the youngster of the trio. 3 other physicists Tom Kibble, Gerald Guralnik and C. R. Hagen, did pretty much the same thing, but got their paper out a bit later, so they get to be "almost Nobelists".

Ironically, Brout might have won the prize if he was a woman. Women live longer than men. But it's not as if women haven't missed out on the Nobel Prize in Physics because they were women.

Two women have been awarded Nobel Prizes in Physics, Marie Curie(1903) and Maria Goeppert-Mayer (1963). 192 men are physics Nobelists, Higgs and Englert make 194.

From what I've come to understand, there is at least one case where "being a woman" is a probable explanation for a physicist's exclusion from a Physics Nobel Prize. Chien-Shiung Wu most likely deserved a share of the Nobel Prize awarded to T. D. Lee and C. N. Yang. It's also hard to explain why another physicist, Lise Meitner, didn't share the 1944 Chemistry Prize awarded to Otto Hahn.

It's not as if the landscape for prospective prizes looks very different. Of the 42 eligible "Nobel class" physicists on Thomson-Reuters "citation laureate" list  only two are women.  (not that it's the best list or anything, but it's a list, and both Lene Hau and Vera Rubin are widely respected).

Reading through the list of 83 eminent 20th century women physicists who made their contributions before 1976, it's striking to me how many of these scientists made significant contributions despite NOT BEING ALLOWED to, or being accorded second class status of some sort.

Also I was surprised to find that many of the 83 are still eligible to win a Nobel (in addition to having made major contributions, they seem to be still around!):

This is my favorite discovery from reading
about the 83 eminent physicists. It's a photo of
Helen Megaw, a crystallographer whose
work on perovskite minerals I had encountered
in my past life as a physics researcher. It was
taken on the occasion of being awarded an
honorary degree a year or two before
her death in 2002 at age 95,
but as you can see by her expression,
she totally won life.

• Betsy Ancker-Johnson 86
• Margaret Burbidge 94
• Jocelyn Bell Burnell 70
• Nina Byers 83
• Yvonne Choquet-Bruhat 89
• Patricia Cladis 76
• Esther Conwell 91
• Cécile DeWitt-Morette 91
• Louise Dolan 63
• Mildred Dresselhaus 83
• Helen Thom Edwards 77
• Magda Galula Ericson 84
• Sandra Moore Faber 68
• Judy R. Franz 73?
• Mary Katharine Gaillard 74
• Gail Gulledge Hanson 66
• Evans Hayward 91?
• Caroline Stuart Littlejohn Herzenberg 81
• Shirley Ann Jackson 67
• Renata Kallosh 70
• Margaret Galland Kivelson 85
• Noemie Benczer Koller 80
• Juliet Lee-Franzini 80?
• Marcia Neugebauer 81
• Helen R. Quinn 70
• Vera Cooper Rubin 85
• Myriam P. Sarachik 80
• Johanna Levelt Sengers 80+
• Sau Lan Wu 70?

I was unable to find information about two others on the list, Christiane Bonnelle and Janine Connes.

There may be two or three future Nobelists on this list, or there may be zero. A few of the women on the list are eminent for their contributions to society, and wouldn't be considered for a Nobel. Sometimes it takes a long time for pioneering work to be recognized as such; sometimes today's hot topic seems inconsequential or worse, wrong, 10 years later. But imagine what the list would look like without our history of discouraging women who felt the calling of physics.

In the earlier parts of this series, I've written about the culture of physics, defined by men for the personality norms of men. It shouldn't be surprising that only a minority of women have been overcoming the barriers it imposes to reach the highest levels. Looking forward, physicists need to recognize that there are more ways to nurture physicists with the diverse talents needed today. 

My own view is that physics, broadly defined as figuring out how physical things work, is a fundamental human impulse that gets expressed in many ways. Once you start, you can't stop doing it, even if you've left the profession to do it in non-physics places. It still needs to be done, and the rewards of doing it transcends prizes that stigmatize 194 men and 2 women.

Physics and Testosterone Part 4: Bang

True (but embellished) story:

Toni, a newly hired physicist at Bell Labs, decided to set up a new lab to do the most sensitive electrical measurements ever. So the young researcher spent hundreds of thousands of dollars building a metal cage with insulators and coatings to electrically isolate the lab from the rest of the world. When it was ready for testing, Toni's friends were summoned. They got inside the cage and Toni cranked up a high voltage power supply. "We're floating 50 thousand volts above the rest of the room!" Toni said, excitedly. "Wouldn't it be cool if we could get to a hundred kilovolts?" Toni's face had that evil grin that made them laugh, nervously. At sixty kilovolts, there was hardly a blip of noise on the oscilloscope, but the smell of ozone in the room was unmistakable. Toni didn't stop there; at 65 kilovolts there was a loud bang, the circuit breakers flipped, and the room exuded a smell of molten plastic. The cage was now arc-welded to the back wall; the expensive insulators instantly useless.

"Whoops." said Toni.

Here's a quiz for you- Which of the following do you think happened to Toni?

A. Toni was fired for her stupidity and for wasting Bell Labs' money. She never worked in Physics again.

B. Toni was mortified and considered quitting immediately. But months of taunting by colleagues who would occasionally make "bang" noises at her was too much. She is now a very successful High School teacher in Northern New Jersey.

C. Toni suddenly developed an interest in a different type of experiment, which had surprising results and today Toni is often mentioned as a candidate for the Nobel Prize in Physics

If you guessed C, you win a prize. You probably also guessed that Tony was a white male, because what woman smart enough to get hired at Bell Labs would ever do something so utterly stupid???

I invite you to consider whether a woman would have been allowed to screw up the way Tony did. I'm told that Tony's friends would taunt him with loud "bangs" whenever he needed to be taken down a notch. Maybe that's what drove him to excel. Maybe he channelled his willingness to make mistakes into more productive experimentation. But I imagine that a woman with Tony's recklessness would have been perceived as a hopeless idiot the same way a 5'10" tall woman is perceived as tall, and a 6'3" woman is freakish. Her career would have been over. And that's a problem, because Toni's contribution would have been on the same level as Tony's contribution turned out to be.