OpenMoko: 10 Years After (Mickey’s Story)
For the 10th anniversary since the legendary OpenMoko announcement at the „Open Source in Mobile“ (7th of November 2006 in Amsterdam), I’ve been meaning to write an anthology or – as Paul Fertser suggested on #openmoko-cdevel – an obituary. I’ve been thinking about objectively describing the motivation, the momentum, how it all began and – sadly – ended. I did even plan to include interviews with Sean, Harald, Werner, and some of the other veterans. But as with oh so many projects of (too) wide scope this would probably never be completed.
As November 2016 passed without any progress, I decided to do something different instead. Something way more limited in scope, but something I can actually finish. My subjective view of the project, my participation, and what I think is left behind: My story, as OpenMoko employee #2. On top of that you will see a bunch of previously unreleased photos (bear with me, I'm not a good photographer and the camera sucked as well).
Prehistoric
I've always been a programmer. Well... not always, but for a quite some time. I got into computer science when my dad brought a Commodore PET from work home for some days. That was around 1980/1981, with me being 8 years old and massively impressed by the green text scrolling down on a black monitor, depending on what you typed on the keyboard. He showed me how to do small programs in BASIC. It was cool.
Unfortunately he had to bring it back after a few weeks, but I was already infected and begged for an own computer. In 1982 – a few months before his sudden and completely unexpected death – he bought me a Commodore 64, which opened up a whole world for me. I learned to program in BASIC, SYSed, POKEed and PEEKed my way through the hardware registers, and had way more fun than with any other toy I possessed.
Naturally, not all of that was programming. The Commodore 64 was an excellent gaming machine, in particular due to the massive amount of cough "free" games available. The first game I actually bought was The Hobbit), a text adventure with the occasional graphic here and there. Rendering an image back then took minutes, and you could watch the computer painting line by line, and sometimes pixel by pixel. But I was patient. I was young and time seemed basically unlimited.
Fast forward to 1985 – when the Commodore 64 suddenly seemed somewhat obsolete at the time the AMIGA was announced. This machine looked so much more powerful that it found its way into my dreams... until that one great day in 1986 when my mother surprised me with lots of white boxes, labelled Commodore AMIGA, in the hallway.
Apart from the natural amount of gaming (which made even more fun on the AMIGA), I learned Motorola 68K assembler and became part of the early demo scene – here's more about my AMIGA history, if you're interested.
Back in those days, it was completely normal that a demo shuts down the operating system (OS) and takes over the whole hardware. Usually you had to reboot after quitting the demo. I did some early experiments with working on my own OS. Alas, my knowledge back then was not sufficient enough to really make one, though I quite liked the idea of being able to hack every part of a system.
This was one thing I started missing when I migrated to a shock, horror DOS/WINDOWS machine in the early 90s. Found my way doing C++, MS Foundation Classes, Win32-API, and the lot, but it never felt the same as in the good 'ole days of the AMIGA. (A bit of that came back when a few years later I installed Linux a PC and learned a lot of different UNIX flavors as part of my computer science studies.)
After completing my diploma thesis, I was asked by a professor (Prof. Drobnik at the institute of telematics, whom I deeply admire and thank for mentoring me!) whether I had interest in pursuing an academic career. I felt my computer science knowledge wasn't complete enough for the "world outside" yet, hence I agreed to work on a Ph.D. in his department.
Since by then I had quite a bit of Linux-experience, one of my first tasks was to help a colleague flashing Linux on his COMPAQ IPAQ. His work involved routing algorithms and handover strategies, so they had a test WiFi network with a bunch of laptops and PDAs equiped with 802.11b PCMCIA cards. Since he ran into lots of problems with the locked down Windows Mobile on the IPAQ, he wanted to give Linux a try.
So I learned about Familiar Linux, got into GPE and Opie, became maintainer of Opie, helped with OpenZaurus (which was the an open source distribution for the soon-to-appear-on-the-stage SHARP ZAURUS), co-founded OpenEmbedded, and eventually received my Ph.D. for µMiddle, a component-based adaptive middleware for ad-hoc networks.
The birth of a project
Some months before my graduation though, it was mid 2006, I was working on OpenEZX, an alternative Linux distribution for Motorola's EZX series of Linux-phones. Smartphones had just began to render PDAs obsolete and were the new hot thing. I was adding EZX support to OpenEmbedded and Opie, when Linux-hacker Harald 'LaF0rge' Welte (who worked on OpenEZX kernel drivers) asked me one day whether I wanted to work on a new Linux-based Smartphone project with a completely open distribution right from the start – as opposed to OpenEZX which was based on a lot of reverse engineering due to the closed nature of that hardware platform (Motorola promised a proper EZX SDK for almost a decade, but never delievered...).
The mastermind behind the project was Sean Moss-Pultz, an american living in Taiwan, working as a product designer for First International Computer (FIC).
Naturally I was excited and started to work on it. I was supposed to be responsible for the Linux distribution build system aspect (e.g. OpenEmbedded integration) and some UI tasks, in particular to create something the chinese engineers could use to base their applications on. It was already decided that we should base the display subsystem on X11 and the UI on GTK+.
While I wasn't lucky with that decision, at this point of time I was not strong enough to question and discuss this. In hindsight I view this as my earliest (unfortunately not the last one though) mistake in the project.
While doing the first work on some GOBJECTs, Sean Moss-Pultz came over to Germany and we met for one week to design the basic human interface guidelines & streamlining his interface mockups. We decided that we want to distinguish two basic types of applications, so-called "finger apps" and "stylus apps". Here are some of the results of this phase. Note that these have been designed before the actual dimensions of the device (hence display) were finalized. Working on HIGs without being able to create actual paper prototypes (to check finger distances and gesture dimensions) must have been my 2nd mistake.
While (in my opinion) these mockups are looking beautiful – even by today's standards – from the viewpoint of a developer who needs to implement these, they're just crazy. Non-rectangular widgets, semi-transparency, shadows, gradients, etc. everywhere. Have fun doing this with GTK+ 2.6 in 2006. I'm not sure it's even possible today with version 4 – let alone the necessary hardware requirements for blending and compositing.
Alas, I tried my best to come up with an UI framework which came close to the renderings to give our Shanghai Team (which were supposed to create the actual applications) the necessary tools. I even created a bunch of demo applications. Here's a guitar toolkit application I programmed using an arm development board:
Speaking about development boards... as I've mentioned before, a lot of the early UI concepts and prototyping code was done while the final device specifics and capabilities (and the housing!) were still unknown. This lead to a series of expectations which were nowhere to be met by the actual hardware.
Back then my idea of the ideal software stack for Openmoko looked like that:
If you're curious about the code for the Openmoko Application Framework libraries, feel free to browse the Openmoko SVN.
Here is the first successful run of kdrive and matchbox on the 3rd development hardware revision of the Neo1973. The picture was taken on the 4th of November 2006 on my IKEA desk. Next to the PCB you are seeing the glimpse of a SHARP ZAURUS Stylus ;-)
The Announcement
In November 2006, the OpenMoko Core Team (which at that time consisted of Sean Moss-Pultz, Harald Welte, and me) flew to Amsterdam where Sean made the legendary announcement of the Neo1973 as "mystery guest speaker" on the "Open Source in Mobile" conference. More details about that (including the slides of the presentation) can be found at the linuxdevices article "Cheap, hackable Linux smartphone due soon".
After the announcement we received a lot of publicity. The mailing lists were flooding with many great ideas (many of those still waiting to be realized). That one small company had the balls to create something hackable from the start against the ongoing trend of locking down mobile embedded devices was very well received. Many interesting leads were made, Universities and labs contacted us, Hardware vendors approached us, etc.
I remember one great quote I picked up from Harald during a presentation that pretty much summed up our approach:
WARRANTY VOID WHEN NOT OPENED – Harald Welte
However, we had yet to deliver and when the first hardware prototypes approached me, I was devastated. Things didn't look good. The device was tiny, much slower than I had expected (the PXA270 in the MOTOROLA EZX series ran circles around our S3C2410), the resistive touchscreen needed too much pressure, and the display was framed by a massive bezel:
We were already struggling on many software layers (in particular to come anywhere near 80% of the mockups with GTK+), but the hardware constraints killed most of our early HIG ideas.
At this point, we brought the London-based OpenedHand (later being acquired by Intel) on board to work on the launcher and PIM applications. They already had the Matchbox window manager and a set of applications based around an embedded port of the Evolution Data Server running on arm and were quite experienced with GTK+. They came up with a massively reduced version of our mockups, but at least something that worked on the Neo1973.
To handle various low-level device aspects (buttons, powermanagement), I wrote neod.
Phase 0
One of the OpenMoko special features was the so-called phase 0, in which we sent out a dozen of Neo1973 pre-versions (for free, without any obligations) to well-known people with a history in open source. The idea was to get some early feedback, perhaps even some contributions, and have those people spread the news about it, hence act as some kind of multiplicator.
On 14th of February 2007, the OpenMoko.org website went live, with it all our source code and the tools necessary to build the current flash images. Also the mailing lists and IRC channels were populated. For most of its lifetime, OpenMoko was really run much more like an Open Source project (with all implications, good and bad) rather than like a proprietary corporate project.
On 25th of February 2007, we shipped the Phase 0 developer devices – and even though we didn't get as much feedback as we would have liked, we felt it was a good practice to do this.
The phase 0 developer devices shipped with Openmoko 2007.1, a pretty bare-bones Linux distribution based on an OpenEmbedded matchbox+kdrive image with the Openmoko GTK+ theme and some rudimentary applications.
Neo1973 is shipping
We originally planned to ship the Neo1973 in January 2007, but both hard- and software issues made us postpone shipping until July 2007. By then we decided that we would need a faster design to target the general audience. So the Neo1973 was repurposed as a developer's device and the Openmoko (by then someone decided that the capital 'M' had to be dropped) "Freerunner" was announced for early 2008. Apart from a faster display subsystem it was supposed to add WiFi (which was missing in the Neo1973), LED buttons, and removing the proprietary GPS chip for something that spoke NMEA. Unfortunately it had also been decided that there could not be a change in the casing, hence we needed to continue living with the bezel and the physical display size.
In June 2007 – shortly before the official launch of the Neo1973 – I had the opportunity to join Harald visiting the OpenMoko office in Taipeh, Taiwan. This is a picture from Computex 2007, where OpenMoko presented the Neo1973 with its slogan "Free Your Phone":
This is our room in the FIC building, where Harald and me worked on the Neo1973 phase 1 (general availability through the webshop) release code:
The Hacker's Lunchbox, an enhanced version of the Neo1973 release package with an additional battery, debug board, and some more goodies:
If for nothing else, for this first visit in Taipeh alone it was worth joining the project. I've never been further away from home, but still felt so much being welcome.
Back in Germany, work continued – with a bunch of hackathons and presentations on conferences. Here's the OpenMoko tent from Chaos Communication Camp 2007:
Although we were hired by OpenMoko to make sure the software we wrote was working great on their devices, we always tried to do our work in the most generic way as possible. We still had a sweet spot for OpenEZX and tried to make the OpenMoko software work there:
On the 15h of September 2007, all the produced Neo1973 were sold-out. Since we already announced the Freerunner for 2008, we had nothing to sell for almost a whole year. And this exactly in the phase where Google and Apple had an opportunity to sell lots of devices. This was another major fault in the project.
Another OpenMoko speciality was the release of the CAD data. On the 4th of March 2008, we released the full CAD data to allow for 3rd party cases.
freesmartphone.org
By the time the Neo1973 was in the hands of developers, I was getting more and more frustrated with the non-UI part of the system: gsmd was unstable, there was still only my prototype neod to handle lowlewel device specifics and I felt there could be much more experimentation if only there was a solid and uniform set of APIs available. I asked Sean whether I could switch my focus to that and he agreed, allowing me to work with my colleagues in Braunschweig Stefan Schmidt, Daniel Willmann, and Jan Lübbe as an independent unit.
By that time, dbus was spreading more and more and looked like a good choice for inter-process-communication, hence I decided to specify a set of dbus APIs that would allow people to come up with all kinds of cool applications using whatever language they wanted. Since I did not want to tie it to the Openmoko devices, I created the FSO project (in analogy to the freedesktop.org project which did great work in standardizing desktop APIs before).
If there was anything I learned out of the TCP/IP vs. ISO/OSI debate in the 1980s, then that APIs without a reference implementation are worthless and it is often more important to create things that work (de facto), than to have huge committees negotiate (de jure) standards. Which meant I had to also come up with code. In order to get things up and running quickly (and to allow hacking directly on the device), I chose one of my favorite languages, Python. The choice of using an interpretated (comparatively slow) language on an already slow device may sound odd, but back then I wanted to give people something to base on as fast as possible, hence development efficiency seemed way more important than runtime efficiency.
Within only a few months, we created a working set of APIs and a Python-based reference implementation that finally allowed us to a) make dozens of calls in a row without the modem hanging up, and b) an EFL-based slick proof-of-concept-and-testing UI to evaluate our APIs (eating our own dogfood was very important to us).
Here's the server code for an SMS echo service, which illustrates how simple it was to access the telephony parts:
For a while, I did regular status updates (e.g. Status Update 5) to keep the community informed about how the framework project went on.
Some of this work has been motivated by a very energetic group lead by Michael 'Emdete' Dietrich, which I dubbed the "Openmoko Underground" effort. They showed me that a) Python was indeed fast enough to handle things like AT, NMEA, and echoing characters to sysfs files, and also convinced me to give the Enlightenment Foundation Libraries a go.
Freerunner is shipping
By Summer 2008, we finally shipped the Freerunner. In the meantime, we were joined by Carsten 'Rasterman' Heitzler, who worked on optimizing Enlightenment (Foundation Libraries) for embedded platforms. Unfortunately the plans to make the Freerunner a much faster version of the Neo1973 didn't quite work out: The SMEDIA gfx chip which was supposed to be a display accelerator turned out to be an actual deccelerator. The chip could (under pressure) handle VGA, but was running much better with QVGA – hence its framebuffer interface was even slower than the Neo1973's.
And there were more hardware problems. As the frameworks and APIs increased in stability, they exposed more and more – partly long standing – issues with the hardware design, in particular the infamous bug #1024 (which yours truly originally found). This bug had the effect of the TI Calypso GSM chip losing network connection when in deep sleep mode. Although we later found a workaround, it massively damaged the reputation of the Openmoko devices, since it made the devices lose calls.
In September 2008, I visited Taipei again, joined by the FSO team plus Harald and Carsten, to present the current status, fix bugs, and teach the local engineers how to best use the framework APIs. For two weeks, we lived in an apartment rented by Openmoko and coded almost 24/7:
Here's a view of the Openmoko office plus the room where we experimented with different case color & material combinations:
Trials and Tribulations
2008 was one of the most intense years in the project's lifetime. With more Freerunner devices in the wild, a lot of people (finally) started building alternatives to the "official" Openmoko distribution. As always, this was good and bad. While I personally was satisfied that people tried out different ideas on all layers of the system – after all, this kind of freedom was one of the very reasons why I joined the project and created FSO in the first place – the multitude of possibilities scared many non-technical, but interested, buyers away. (The kind of people who don't want to actively work on open source projects, but want to support them. In the same way as chosing Firefox over IE, or OpenOffice over Word).
Here is a bunch of screenshots taken from a presentation I held at the FOSDEM 2010, where we had an Openmoko developer room:
Here's a photo (courtesy Josch) from the first Openmoko user meeting in Karlsruhe, which also shows a glimpse of the Openmoko devices' diversity:
Freedom of choice truly is a double-edged sword: Many people would have preferred one solid software stack rather than having 10 half-done ones, each of them lacking in different areas. In- and outside Openmoko, tensions arose whether to continue with the Enlightenment-based route, switch to a Qtopia-based system, revive the GTK+-based stack, or just stop doing any work above the kernel and move to Android.
Personally, I think we should have limited our software-contributions to a kernel, FSO, and a monolithic application that would have covered voice & messages and super stable day2day operation.
Here are some more screenshots of 3rd-party distributions (Thanks, Walter!).
One distribution I particulary liked was the one from SHR Project, a community of skilled hackers working closely together with the freesmartphone.org framework team. We did a FSOSHRUDCON (FSO+SHR Users and Developers Conference) in May 2009, in the LinuxHotel in Essen, Germany. Even though it was officially "past-Openmoko", we had an incredible time there. If anyone of you still has a group photo which shows all of us, please send it to me!
With a much smaller audience, we came back to LinuxHotel in 2011 for the 2nd – and unfortunately also the last – FSOSHR conference. By the way... this was some weeks after my daughter Lara Marie was born and I enjoyed sleeping a bit longer than usually :-)
Freerunner in Space
One of the most remarkable, nah... frickin' coolest things ever done with an Openmoko device was to mount a Freerunner inside a rocket and send it into space. In early 2009, a german space agency (DLR: Deutsches Zentrum für Luft- und Raumfahrt e.V.) carried out an experiment to measure how accelerometers (and other parts of consumer electronics) reacted to massive changes in velocity. One of the DLR directors, Prof. Dr. Felix Huber himself wrote the Freerunner-software for this experiment. He based it on an FSO image (and also submitted a bunch of patches to FSO and Zhone, but that's another story)! According to him, the Freerunner was the only smartphone where he trusted application software to have control over the GSM part, hence no other device could be used for this experiment. Images (C) DLR e.V.:
Afterwards, Prof. Huber invited me to the DLR in Oberpfaffenhofen and gave me a tour. It was a great experience which I'll never forget. If you want to read more about the MAPHEUS experiments, please also see this paper.
GTA03 – Our last Hope for Freedom
After it was clear that the Freerunner was not the silver bullet we had hoped for, plans to design a successor launched. This one – codenamed GTA03 (and later 3D7K... for a reason I don't want to disclose) – was supposed to be designed free of any existing legacy we inherited from FIC's stock.
GTA03 was supposed to contain a new S3C chip, a Cinterion modem, a capacitive (!) touchscreen, and a slick round design with a semitransparent cutout where you could see a part of the PCB (an elegant self-reference to the project's transparency & openness). This device would have made a significant change and put Openmoko into another league.
Here's a plastic prototype of the device:
Here's the PCB:
Here's me hacking FSO and OE to incorporate the GTA03-specifics using development boards:
Alas, due to a number of circumstances, the device was cancelled – although it was already 80% done.
Nails in the coffin
On the OpenExpo 2009 in Switzerland, Sean announced that Openmoko was quitting smartphone development. To me, at the very point, when framework-wise things finally started to look good. The 2nd reference implementation of the freesmartphone.org middleware had just begun and there were many promising side-projects using the FSO API.
Here's a screenshot of the freesmartphone.org website back then (thanks to the Wayback Machine):
For two more years, I continued to work on FSO in my spare time, trying to find an alternative hardware reference platform to run on, but nothing convincing showed up. All reverse-engineering-based efforts to replace other operating systems with our stack failed due to the short hardware life-time.
When I dropped out in June 2011 due to the birth of my daughter Lara-Marie, those projects more or less came to a full stop.
It's not easy to pinpoint exactly what went wrong with the project. I think I can mention a couple of nails in the coffin though:
- Transparency – if you design hardware in the open, every single bug (that can perhaps be worked around in software) is immediately being revealed and talked to death. This scares potential buyers.
- The financial crisis of 2008 – Openmoko's venture capital dried up when some of the investors had serious cash problems.
- The competition – With Apple and Google two major players came out very soon after our initial announcement. Apple's iPhone made it tough to compete hardware-wise, and Google's seemingly open Android dragged a lot of people who perceived it as being open enough out of our community.
- Not enough focus, not enough structure – As mentioned before, the basics (phone, messages, power management) were never stable enough to make the devices really work well as your main phone. I'm afraid we wanted too much too fast.
What’s left behind
During 2007 – 2011, I travelled a lot and presented on conferences, if I recall correctly I've been in Aalborg (Denmark), Bern (Switzerland), Berlin, Brussels (Belgium), Birmingham (UK), Chemnitz, St.Augustin, Munich, Paris (France), Porto de Galinhas (Brazil) [Summerville Beach Resort, best conference venue ever], Taipeh (Taiwan), Vienna (Austria), and Zürich (Switzerland). It was an incredible time and I rediscovered a spirit that I had first experienced 20 years ago during the early days of the C64 and AMIGA demo scene. I'm glad having met so many great people. I learned more about (prototype) hardware than I would have ever wanted.
Over the two years of operation, Openmoko sold ~13000 phones (3000 Neo1973, 10000 Freerunner). Openmoko Inc. rose from 4 people (Sean, Harald, me, Werner) to about 50 in their high-time.
Alas, this concludes the positive aspects though. I wished that we could have left a bigger footprint in history, but as things stand, the mobile soft- and hardware landscape in 2017 is way more closed than it was ten years ago. I really had hoped for the opposite. Many projects we started are now either obsolete or on hiatus, waiting (forever?) for an open hardware platform to run on. However, they are still there and could be revived, if there was enough interest.
Two notable active projects are Dr. Nikolaus Schaller's GTA04 – which is a completely new design fitting in the Freerunner's case and the Neo900 by ex-Openmoko engineer Jörg Reisenweber – attempting to do the same with the Nokia N900.
Both of these projects share the approach to build upon an existing case and retrofitting most of the innards with a newer PCB (think "second live"). While the Neo900 is still in conceptual phase (they just announced the next prototype PCB a couple of days ago), the GTA04 has already seen a number of shipping board revisions and there is still a small, die-hard, community following tireless Nikolaus' progress to keep the Openmoko spirit alive.
Basic support for the GTA04 has already been added to FSO and there are people actively working on kernel support as well as porting various userlands like QtMoko and Replicant to it. I really suggest browsing the archives of gta04-owner to understand the incredible amount of problems a small series of custom smartphone hardware brings – including component sourcing, CAD programs, fighting against Linux mainline (people which apparantly are not really interested in smartphone code), defending a relatively high price, etc. It's hell of a ride.
While I applaud all these efforts, something broke inside me when Openmoko shut down – and that's one of the reasons why I find it pretty hard to motivate myself working on FSO again. Another one is that Vala – the language of the 2nd reference implementation – had their own set of problems with parents and maintainers losing interest... although it just recently seemed to have found new love).
The Future?
By now, Android and iOS have conquered the mobile world. Blackberry is dead, HP killed WebOS and the Palm Pre, Windows Phone is fading into oblivion and even big players such as Ubuntu or Mozilla are having a hard time coming up with an alternative to platforms that contain millions of apps running on a wide variety of the finest hardware.
IF YOU'RE SERIOUS ABOUT SOFTWARE, YOU HAVE TO DESIGN YOUR OWN HARDWARE – Alan Kay
Right now my main occupation is writing software for Apple's platforms – and while it's nice to work on apps using a massive set of luxury frameworks and APIs, you're locked and sandboxed within the software layers Apple allows you. I'd love to be able to work on an open source Linux-based middleware again.
However, the sad truth is that it looks like there is no business case anymore for a truly open platform based on custom-designed hardware, since people refuse to spend extra money for tweakability, freedom, and security. Despite us living in times where privacy is massively endangered.
If anyone out there thinks different and plans a project, please holler and get me on board!
Acknowledgements
Thanks to you for reading that far! Thanks to Sean Moss-Pultz for his crazy and ambitioned idea. Thanks to Harald Welte for getting me on board. Thanks to Daniel, Jan, and Stefan for working with me on FSO. Thanks to the countless organizers and helpers on conferences where we presented our work. Thanks to Nikolaus and Walter for commenting on an early draft of this. And finally... thanks to all enthusiasts who used a Neo1973 and/or a Freerunner in the past, present, or future.