The Gecko monoculture

I remember a time, not so very long ago, when Gecko powered 4 or 5 non-Mozilla browsers, some of them on exotic platforms, as well as GPS devices, wysiwyg editors, geographic platforms, email clients, image editors, eBook readers, documentation browsers, the UX of virus scanners, etc, as well as a host of innovative and exotic add-ons. In these days, Gecko was considered, among other things, one of the best cross-platform development toolkits available.

The year is now 2016 and, if you look around, you’ll be hard-pressed to find Gecko used outside of Firefoxen (alright, and Thunderbird and Bluegriffon). Did Google or Apple or Microsoft do that? Not at all. I don’t know how many in the Mozilla community remember this, but this was part of a Mozilla strategy. In this post, I’d like to discuss this strategy, its rationale, and the lessons that we may draw from it.

Building a Gecko monoculture

For the first few years of Firefox, enthusiasm for the technology behind the browser was enormous. After years of implementing Gecko from scratch, Mozilla had a kick-ass cross-platform toolkit that covered almost everything from system interaction to network, cryptography, user interface, internationalization, even an add-on mechanism, a scripting language and a rendering engine. For simplicity, let’s call this toolkit XUL. Certainly, XUL had a number of drawbacks, but in many ways, this toolkit was years ahead of everything that other toolkits had to offer at the time. And many started to use XUL for things that had never been planned. Dozens of public projects and certainly countless more behind corporate doors. Attempts were made to extend XUL towards Python, .Net, Java and possibly more. These were the days of the “XUL Planet”. All of this was great – for one, that is how I joined the Mozilla community, embedding Gecko in exotic places and getting it to work nicely with exotic network protocols.

But this success was also hurting Mozilla’s mission in two ways. The first way was the obvious cost. The XUL platform had a huge API, in JavaScript, in C, in C++, in IDL, in declarative UI (XUL and XBL), not to mention its configuration files and exotic query language (hello, RDF, I admit that I don’t really miss you that much), and I’m certain I miss a few. Oh, that’s not including the already huge web-facing API that can never abandon backwards compatibility with any feature, of course. Since third-party developers could hit any point of this not-really-internal API, any change made to the code of Gecko had the potential of breaking applications in subtle and unexpected ways – applications that we often couldn’t test ourselves. This meant that any change needed to be weighed carefully as it could put third-party developers out of business. That’s hardly ideal when you attempt to move quickly. To make things worse, this API was never designed for such a scenario, many bits were extremely fragile and often put together in haste with the idea of taking them down once a better API was available. Unfortunately, in many cases, fixing or replacing components often proved impossible, for the sake of compatibility. And to make things even worse, the XUL platform was targeting an insane number of Operating Systems, including Solaris, RiscOS, OS/2, even the Amiga Workbench if I recall correctly. Any change had to be kept synchronized between all these platforms, or, once again, we could end up putting third-party developers out of business by accident.

So this couldn’t last forever.

Another way this success was hurting Mozilla is that XUL was not the web. Recall that Mozilla’s objectives were not to create yet another cross-platform toolkit, no matter how good, but to Take Back the Web from proprietary and secret protocols. When the WhatWG and HTML5 started rising, it became clear that the web was not only taken back, but that we were witnessing the dawn of a new era of applications, which could run on all operating systems, which were based on open protocols and at least at some level on open-source. The Web Applications were the future – an ideal future, by some criteria – and the future was there. In this context, non-standard, native cross-platform toolkits were a thing of the past, something that Mozilla was fighting, not something that Mozilla should be promoting. It made entire sense to stop putting resources in XUL and concentrate more tightly on the web.

So XUL as a cross-platform toolkit couldn’t last forever.

I’m not sure exactly who took the decision but at some point around 2009, Mozilla’s strategy changed. We started deprecating the use cases of Gecko that were not the Web Platform. This wasn’t a single decision or a single fell swoop, and this didn’t go in one single unambiguous direction, but this happened. We got rid of the impedimenta.

We reinvented Gecko as a Firefox monoculture.

Living in a monoculture

We have now been living in a Gecko monoculture long enough to be able to draw lessons from our choices. So let’s look at the costs and benefits.

API and platform cost

Now that third-party developers using Gecko and hitting every single internal API are gone, it is much easier to refactor. Some APIs are clearly internal and I can change them without referring to anyone. Some are still accessible by add-ons, and I need to look for add-ons that use them and get in touch with their developers, but this is still infinitely simpler than it used to be. Already, dozens of refactorings that were critically needed but that had been blocked at some point in the past by backwards internal compatibility have been made possible. Soon, Jetpack WebExtensions will become the sole entry point for writing most add-ons, and Gecko developers will finally be free to refactor their code at will as long as it doesn’t break public APIs, much like developers of every single other platform on the planet.

Similarly, dropping support for exotic platforms made it possible to drop plenty of legacy code that was hurting refactoring, and in many cases, made it possible to write richer APIs without being bound by the absolute need to implement everything on every single platform.

In other words, by the criteria of reducing costs and increasing agility, yes, the Gecko monoculture has been a clear success.

Web Applications

Our second objective was to promote web applications. And if we look around, these days, web applications are everywhere – except on mobile. Actually, that’s not entirely true. On mobile, a considerable number of applications are built using PhoneGap/Cordova. In other word, these are web applications, wrapped in native applications, with most of the benefits of both worlds. Indeed, one could argue that PhoneGap/Cordova applications are more or less applications which could have been developed with XUL, and are instead developed with a closer-to-standards approach. As a side-note, it is a little-known fact is that one of the many (discarded) designs of FirefoxOS was as a runtime somewhat comparable to PhoneGap/Cordova, and which would have replaced the XUL platform.

Despite the huge success of web applications and even the success of hybrid web/native applications, the brand new world in which everything would be a web application hasn’t arrived yet, and it is not sure that it ever will. The main reason is that mobile has taken over the world. Mobile applications need to integrate with a rather different ecosystem, with push notifications, working disconnected, transactions and microtransactions, etc. not to mention a host of new device-specific features that were not initially web-friendly. Despite the efforts of most browser vendors, browser still haven’t caught up this moving target. New mobile device have gained voice recognition and in the meantime, the WhatWG is still struggling to design a secure, cross-platform API for accessing local files.

In other words, by the criteria of pushing web applications, I would say that the Gecko monoculture has had a positive influence, but not quite enough to be called a success.

The Hackerbase

Now that we have seen the benefits of this Gecko monoculture, perhaps it is time to look at the costs.

By turning Gecko into a Firefox monoculture, we have lost dozens of products. We have almost entirely lost the innovations that were not on the roadmap of the WhatWG, as well as the innovators themselves. Some of them have turned to web applications, which is what we wanted, or hybrid applications, which is close enough to what we wanted. In the meantime, somewhere else in the world, the ease of embedding first WebKit and now Chromium (including Atom/Electron) have made it much easier to experiment and innovate with these platforms, and to do everything that has ever been possible with XUL, and more. Speaking only for myself, if I were to enter the field today with the same kind of technological needs I had 15 years ago, I would head towards Chromium without a second thought. I find it a bit sad that my present self is somehow working against my past self, while they could be working together.

By turning our back on our Hackerbase, we have lost many things. In the first place, we have lost smart people, who may have contributed ideas or code or just dynamism. In the second place, we have lost plenty of opportunities for our code and our APIs to be tested for safety, security, or just good design. That’s already pretty bad.

Just as importantly, we have lost opportunities to be part of important projects. Chris Lord has more to say on this topic, so I’ll invite you to read his post if you are interested.

Also, somewhere along the way, we have largely lost any good reason to provide clean and robust APIs, to separate concerns between our libraries. I would argue that the effects of this can be witnessed in our current codebase. Perhaps not in the web-facing APIs, that are still challenged by their (mis)usage in terms of convenience, safety and robustness, but in all our internal+addons APIs, many of which are sadly under-documented, under-tested, and designed to break in new and exciting ways whenever they are confronted with unexpected inputs. One could argue that the picture I am painting is too bleak, and that some of our fragile APIs are, in fact, due to backwards compatibility with add-ons or, at some point, third-party applications.

Regardless, by the criteria of our Hackerbase, I would count the Gecko monoculture as a bloody waste.

Bottom line

So the monoculture has succeeded at making us faster, has somewhat helped propagate Web Applications, and has hurt us by severing our hackerbase.

Before starting to write this blogpost, I felt that turning Gecko into a Firefox monoculture was a mistake. Now, I realize that this was probably a necessary phase. The Gecko from 2006 was impossible to fix, impossible to refactor, impossible to improve. The Firefox from 2006 would have needed a nearly-full reimplementation to support e10s or Rust-based code (ok, I’m excluding Rust-over-XPConnect, which would be a complete waste). Today’s Gecko is much fitter to fight against WebKit and Chromium. I believe that tomorrow’s Gecko – not Firefox, just Gecko – with full support for WebExtensions and progressive addition of new, experimental WebExtensions, would be a much better technological base for implementing, say, a cross-platform e-mail client, or an e-Book reader, or even a novel browser.

As all phases, though, this monoculture needs to end sooner or later, and I certainly hope that it ends soon, because we keep paying the cost of this transformation through our community.

Surviving the monoculture

An exit strategy from the Gecko monoculture

It is my belief that we now need to consider an exit strategy from the Gecko monoculture. No matter which strategy is picked, it will have a cost. But I believe that the potential benefits in terms of community and innovation will outweigh these costs.

First, we need to avoid repeating past mistakes. While WebExtensions may not cover all the use cases for which we need an extension API for Gecko, they promise a set of clean and high-level APIs, and this is a good base. We need to make sure that whatever we offer as part of WebExtensions or in addition to them remains a set high-level, well-insulated APIs, rather than the panic-inducing entanglement that is our set of internal APIs.

Second, we need to be able to extend our set of extension APIs in directions we not planned by any single governing body, including Mozilla. When WebExtensions were first announced, the developers in charge of the project introduced a uservoice survey to determine the features that the community expected. This was a good start, but this will not be sufficient in the long run. Around that time, Giorgio Maone drafted an API for developing and testing experimental WebExtensions features. This was also a good start, because experimenting is critical for innovation. Now, we need a bridge to progressively turn experimental extension APIs into core APIs. For this purpose, I believe that the best mechanism is a RFC forum and a RFC process for WebExtensions, inspired from the success of RFCs in the Rust (or Python) community.

Finally, we need a technological brick to get applications other than Firefox to run Gecko. We have experience doing this, from XULRunner to Prism. A few years ago, Mike De Boer introduced “Chromeless 2”, which was roughly in the Gecko world what Electron is nowadays in the Chromium world. Clearly, this project was misunderstood by the Mozilla community – I know that it was misunderstood by me, and that it took Electron to make me realize that Mike was on the right track. This project was stopped, but it could be resumed or rebooted. To make it easier for the community, using the same API as Electron, would be a possibility.

Keeping projects multicultural

Similarly, I believe that we need to consider strategies that will let us avoid similar monocultures in our other projects. This includes (and is not limited to) B2G OS (formerly known as Firefox OS), Rust, Servo and Connected Devices.

So far, Rust has proved very open to innovation. For one thing, Rust has its RFC process and it works very well. Additionally, while Rust was originally designed for Servo, it has already escaped this orbit and the temptation of a Servo monoculture. Rust is now used for cryptocurrencies, operating systems, web servers, connected devices… So far, so good.

Similarly, Servo has proved quite open, albeit in very different directions. For one thing, Servo is developed separately from any web browser that may embed it, whether Servo Shell or Browser.html. Also, Servo is itself based on dozens of libraries developed, tested and released individually, by community members. Similarly, many of the developments undertaken for Servo are released themselves as independent libraries that can independently be maintained or integrated in yet other projects… I have hopes that Servo, or at least large subsets, will eventually find its way into projects unrelated to Mozilla, possibly unrelated to web browsers. My only reservation is that I have not checked how much effort the Servo team has made into checking that the private APIs of Servo remain private. If this is the case, so far, so good.

The case of Firefox OS/B2G OS is quite different. B2G OS was designed from scratch as a Gecko application and was entirely dependent on Gecko and some non-standard extensions. Since the announcement that Firefox OS would be retired – and hopefully continue to live as B2G OS – it has been clear that B2G-specific Gecko support would be progressively phased out. The B2G OS community is currently actively reworking the OS to make sure that it can live in a much more standard environment. Similarly, the Marketplace, which was introduced largely to appease carriers, will disappear, leaving B2G OS to live as a web OS, as it was initially designed. While the existence of the project is at risk, I believe that these two changes, together, have the potential to also set it free from a Gecko + Marketplace + Telephone monoculture. If B2G is still alive in one or two years, it may have become a cross-platform, cross-rendering engine operating system designed for a set of devices that may be entirely different from the Firefox Phones. So, I’m somewhat optimistic.

As for Connected Devices, well, these projects are too young to be able to judge. It is our responsibility to make sure that we do not paint ourselves into monocultural corners.

edit Added a link to Chris Lord’s post on the topic of missed opportunities.

Advertisement