The Gecko monoculture

March 7, 2016 § 8 Comments

I remember a time, not so very long ago, when Gecko powered 4 or 5 non-Mozilla browsers, some of them on exotic platforms, as well as GPS devices, wysiwyg editors, geographic platforms, email clients, image editors, eBook readers, documentation browsers, the UX of virus scanners, etc, as well as a host of innovative and exotic add-ons. In these days, Gecko was considered, among other things, one of the best cross-platform development toolkits available.

The year is now 2016 and, if you look around, you’ll be hard-pressed to find Gecko used outside of Firefoxen (alright, and Thunderbird and Bluegriffon). Did Google or Apple or Microsoft do that? Not at all. I don’t know how many in the Mozilla community remember this, but this was part of a Mozilla strategy. In this post, I’d like to discuss this strategy, its rationale, and the lessons that we may draw from it.

Building a Gecko monoculture

For the first few years of Firefox, enthusiasm for the technology behind the browser was enormous. After years of implementing Gecko from scratch, Mozilla had a kick-ass cross-platform toolkit that covered almost everything from system interaction to network, cryptography, user interface, internationalization, even an add-on mechanism, a scripting language and a rendering engine. For simplicity, let’s call this toolkit XUL. Certainly, XUL had a number of drawbacks, but in many ways, this toolkit was years ahead of everything that other toolkits had to offer at the time. And many started to use XUL for things that had never been planned. Dozens of public projects and certainly countless more behind corporate doors. Attempts were made to extend XUL towards Python, .Net, Java and possibly more. These were the days of the “XUL Planet”. All of this was great – for one, that is how I joined the Mozilla community, embedding Gecko in exotic places and getting it to work nicely with exotic network protocols.

But this success was also hurting Mozilla’s mission in two ways. The first way was the obvious cost. The XUL platform had a huge API, in JavaScript, in C, in C++, in IDL, in declarative UI (XUL and XBL), not to mention its configuration files and exotic query language (hello, RDF, I admit that I don’t really miss you that much), and I’m certain I miss a few. Oh, that’s not including the already huge web-facing API that can never abandon backwards compatibility with any feature, of course. Since third-party developers could hit any point of this not-really-internal API, any change made to the code of Gecko had the potential of breaking applications in subtle and unexpected ways – applications that we often couldn’t test ourselves. This meant that any change needed to be weighed carefully as it could put third-party developers out of business. That’s hardly ideal when you attempt to move quickly. To make things worse, this API was never designed for such a scenario, many bits were extremely fragile and often put together in haste with the idea of taking them down once a better API was available. Unfortunately, in many cases, fixing or replacing components often proved impossible, for the sake of compatibility. And to make things even worse, the XUL platform was targeting an insane number of Operating Systems, including Solaris, RiscOS, OS/2, even the Amiga Workbench if I recall correctly. Any change had to be kept synchronized between all these platforms, or, once again, we could end up putting third-party developers out of business by accident.

So this couldn’t last forever.

Another way this success was hurting Mozilla is that XUL was not the web. Recall that Mozilla’s objectives were not to create yet another cross-platform toolkit, no matter how good, but to Take Back the Web from proprietary and secret protocols. When the WhatWG and HTML5 started rising, it became clear that the web was not only taken back, but that we were witnessing the dawn of a new era of applications, which could run on all operating systems, which were based on open protocols and at least at some level on open-source. The Web Applications were the future – an ideal future, by some criteria – and the future was there. In this context, non-standard, native cross-platform toolkits were a thing of the past, something that Mozilla was fighting, not something that Mozilla should be promoting. It made entire sense to stop putting resources in XUL and concentrate more tightly on the web.

So XUL as a cross-platform toolkit couldn’t last forever.

I’m not sure exactly who took the decision but at some point around 2009, Mozilla’s strategy changed. We started deprecating the use cases of Gecko that were not the Web Platform. This wasn’t a single decision or a single fell swoop, and this didn’t go in one single unambiguous direction, but this happened. We got rid of the impedimenta.

We reinvented Gecko as a Firefox monoculture.

Living in a monoculture

We have now been living in a Gecko monoculture long enough to be able to draw lessons from our choices. So let’s look at the costs and benefits.

API and platform cost

Now that third-party developers using Gecko and hitting every single internal API are gone, it is much easier to refactor. Some APIs are clearly internal and I can change them without referring to anyone. Some are still accessible by add-ons, and I need to look for add-ons that use them and get in touch with their developers, but this is still infinitely simpler than it used to be. Already, dozens of refactorings that were critically needed but that had been blocked at some point in the past by backwards internal compatibility have been made possible. Soon, Jetpack WebExtensions will become the sole entry point for writing most add-ons, and Gecko developers will finally be free to refactor their code at will as long as it doesn’t break public APIs, much like developers of every single other platform on the planet.

Similarly, dropping support for exotic platforms made it possible to drop plenty of legacy code that was hurting refactoring, and in many cases, made it possible to write richer APIs without being bound by the absolute need to implement everything on every single platform.

In other words, by the criteria of reducing costs and increasing agility, yes, the Gecko monoculture has been a clear success.

Web Applications

Our second objective was to promote web applications. And if we look around, these days, web applications are everywhere – except on mobile. Actually, that’s not entirely true. On mobile, a considerable number of applications are built using PhoneGap/Cordova. In other word, these are web applications, wrapped in native applications, with most of the benefits of both worlds. Indeed, one could argue that PhoneGap/Cordova applications are more or less applications which could have been developed with XUL, and are instead developed with a closer-to-standards approach. As a side-note, it is a little-known fact is that one of the many (discarded) designs of FirefoxOS was as a runtime somewhat comparable to PhoneGap/Cordova, and which would have replaced the XUL platform.

Despite the huge success of web applications and even the success of hybrid web/native applications, the brand new world in which everything would be a web application hasn’t arrived yet, and it is not sure that it ever will. The main reason is that mobile has taken over the world. Mobile applications need to integrate with a rather different ecosystem, with push notifications, working disconnected, transactions and microtransactions, etc. not to mention a host of new device-specific features that were not initially web-friendly. Despite the efforts of most browser vendors, browser still haven’t caught up this moving target. New mobile device have gained voice recognition and in the meantime, the WhatWG is still struggling to design a secure, cross-platform API for accessing local files.

In other words, by the criteria of pushing web applications, I would say that the Gecko monoculture has had a positive influence, but not quite enough to be called a success.

The Hackerbase

Now that we have seen the benefits of this Gecko monoculture, perhaps it is time to look at the costs.

By turning Gecko into a Firefox monoculture, we have lost dozens of products. We have almost entirely lost the innovations that were not on the roadmap of the WhatWG, as well as the innovators themselves. Some of them have turned to web applications, which is what we wanted, or hybrid applications, which is close enough to what we wanted. In the meantime, somewhere else in the world, the ease of embedding first WebKit and now Chromium (including Atom/Electron) have made it much easier to experiment and innovate with these platforms, and to do everything that has ever been possible with XUL, and more. Speaking only for myself, if I were to enter the field today with the same kind of technological needs I had 15 years ago, I would head towards Chromium without a second thought. I find it a bit sad that my present self is somehow working against my past self, while they could be working together.

By turning our back on our Hackerbase, we have lost many things. In the first place, we have lost smart people, who may have contributed ideas or code or just dynamism. In the second place, we have lost plenty of opportunities for our code and our APIs to be tested for safety, security, or just good design. That’s already pretty bad.

Just as importantly, we have lost opportunities to be part of important projects. Chris Lord has more to say on this topic, so I’ll invite you to read his post if you are interested.

Also, somewhere along the way, we have largely lost any good reason to provide clean and robust APIs, to separate concerns between our libraries. I would argue that the effects of this can be witnessed in our current codebase. Perhaps not in the web-facing APIs, that are still challenged by their (mis)usage in terms of convenience, safety and robustness, but in all our internal+addons APIs, many of which are sadly under-documented, under-tested, and designed to break in new and exciting ways whenever they are confronted with unexpected inputs. One could argue that the picture I am painting is too bleak, and that some of our fragile APIs are, in fact, due to backwards compatibility with add-ons or, at some point, third-party applications.

Regardless, by the criteria of our Hackerbase, I would count the Gecko monoculture as a bloody waste.

Bottom line

So the monoculture has succeeded at making us faster, has somewhat helped propagate Web Applications, and has hurt us by severing our hackerbase.

Before starting to write this blogpost, I felt that turning Gecko into a Firefox monoculture was a mistake. Now, I realize that this was probably a necessary phase. The Gecko from 2006 was impossible to fix, impossible to refactor, impossible to improve. The Firefox from 2006 would have needed a nearly-full reimplementation to support e10s or Rust-based code (ok, I’m excluding Rust-over-XPConnect, which would be a complete waste). Today’s Gecko is much fitter to fight against WebKit and Chromium. I believe that tomorrow’s Gecko – not Firefox, just Gecko – with full support for WebExtensions and progressive addition of new, experimental WebExtensions, would be a much better technological base for implementing, say, a cross-platform e-mail client, or an e-Book reader, or even a novel browser.

As all phases, though, this monoculture needs to end sooner or later, and I certainly hope that it ends soon, because we keep paying the cost of this transformation through our community.

Surviving the monoculture

An exit strategy from the Gecko monoculture

It is my belief that we now need to consider an exit strategy from the Gecko monoculture. No matter which strategy is picked, it will have a cost. But I believe that the potential benefits in terms of community and innovation will outweigh these costs.

First, we need to avoid repeating past mistakes. While WebExtensions may not cover all the use cases for which we need an extension API for Gecko, they promise a set of clean and high-level APIs, and this is a good base. We need to make sure that whatever we offer as part of WebExtensions or in addition to them remains a set high-level, well-insulated APIs, rather than the panic-inducing entanglement that is our set of internal APIs.

Second, we need to be able to extend our set of extension APIs in directions we not planned by any single governing body, including Mozilla. When WebExtensions were first announced, the developers in charge of the project introduced a uservoice survey to determine the features that the community expected. This was a good start, but this will not be sufficient in the long run. Around that time, Giorgio Maone drafted an API for developing and testing experimental WebExtensions features. This was also a good start, because experimenting is critical for innovation. Now, we need a bridge to progressively turn experimental extension APIs into core APIs. For this purpose, I believe that the best mechanism is a RFC forum and a RFC process for WebExtensions, inspired from the success of RFCs in the Rust (or Python) community.

Finally, we need a technological brick to get applications other than Firefox to run Gecko. We have experience doing this, from XULRunner to Prism. A few years ago, Mike De Boer introduced “Chromeless 2”, which was roughly in the Gecko world what Electron is nowadays in the Chromium world. Clearly, this project was misunderstood by the Mozilla community – I know that it was misunderstood by me, and that it took Electron to make me realize that Mike was on the right track. This project was stopped, but it could be resumed or rebooted. To make it easier for the community, using the same API as Electron, would be a possibility.

Keeping projects multicultural

Similarly, I believe that we need to consider strategies that will let us avoid similar monocultures in our other projects. This includes (and is not limited to) B2G OS (formerly known as Firefox OS), Rust, Servo and Connected Devices.

So far, Rust has proved very open to innovation. For one thing, Rust has its RFC process and it works very well. Additionally, while Rust was originally designed for Servo, it has already escaped this orbit and the temptation of a Servo monoculture. Rust is now used for cryptocurrencies, operating systems, web servers, connected devices… So far, so good.

Similarly, Servo has proved quite open, albeit in very different directions. For one thing, Servo is developed separately from any web browser that may embed it, whether Servo Shell or Browser.html. Also, Servo is itself based on dozens of libraries developed, tested and released individually, by community members. Similarly, many of the developments undertaken for Servo are released themselves as independent libraries that can independently be maintained or integrated in yet other projects… I have hopes that Servo, or at least large subsets, will eventually find its way into projects unrelated to Mozilla, possibly unrelated to web browsers. My only reservation is that I have not checked how much effort the Servo team has made into checking that the private APIs of Servo remain private. If this is the case, so far, so good.

The case of Firefox OS/B2G OS is quite different. B2G OS was designed from scratch as a Gecko application and was entirely dependent on Gecko and some non-standard extensions. Since the announcement that Firefox OS would be retired – and hopefully continue to live as B2G OS – it has been clear that B2G-specific Gecko support would be progressively phased out. The B2G OS community is currently actively reworking the OS to make sure that it can live in a much more standard environment. Similarly, the Marketplace, which was introduced largely to appease carriers, will disappear, leaving B2G OS to live as a web OS, as it was initially designed. While the existence of the project is at risk, I believe that these two changes, together, have the potential to also set it free from a Gecko + Marketplace + Telephone monoculture. If B2G is still alive in one or two years, it may have become a cross-platform, cross-rendering engine operating system designed for a set of devices that may be entirely different from the Firefox Phones. So, I’m somewhat optimistic.

As for Connected Devices, well, these projects are too young to be able to judge. It is our responsibility to make sure that we do not paint ourselves into monocultural corners.

edit Added a link to Chris Lord’s post on the topic of missed opportunities.

Tagged: , , , , , , , , , , , , , , , , , , , , ,

§ 8 Responses to The Gecko monoculture

  • Actually, the strategy change happened earlier than 2009. I’ve been to the Developer Days in Paris in 2007 (https://wiki.mozilla.org/DeveloperDays/ParisJune2007), that’s where I heard this for the first time. I don’t remember who did this presentation but the Mozilla guys were talking about deprecating XUL in favor of HTML and claiming that they would add all necessary functionality to HTML. Given the state of the HTML platform back then I wasn’t too keen on this – I tried to make very clear that I’m not looking forward to ending up in the world of hurt that is HTML. Well, it took almost ten years and HTML today is in a state I’d consider “usable” but still not as convenient as XUL was.

    • yoric says:

      That’s quite possible, I don’t remember the dates exactly. Plus it wasn’t done in a single step.

      • Sure. I think people realized pretty early on that they were supporting two systems (HTML and XUL) that were similar yet different in so many details that code sharing became problematic. As HTML evolved and Mozilla had to re-implement essentially the same mechanisms in a slightly different way, it became quite obvious that XUL wouldn’t be sustainable. It was only a matter of time.

        Well, almost all XUL features are available for HTML now, albeit often in a more verbose way. Web components will hopefully be done as well soon. The major pain point for extensions is styling: styling your content the same way as application content is close to impossible now. Chrome had extensions with inconsistent look&feel all along, Firefox is getting there as well (in fact, even its own HTML content is styled very inconsistently).

  • […] This post is some insightful further reading on roughly the same topic. […]

  • Well. Two examples: 1. XUL is perfect for UI because of the XUL box model while the Flex Box Model for html has complex bits because we don’t have spacers in html… 2. XBL, with all its bugs, has been superbly working for more than fifteen years, while WebComponents are a IMHO mess of epic magnitude

  • voracity says:

    Watching Mozilla shoot itself in the foot repeatedly over the last 18 something years has been a little bit frustrating. But anyway.

    XUL was never going to work long term because Mozilla expressly did not want to standardise any of it, and yet it clearly had enormous overlap with what almost all web-based developers were either doing or desperately wanted to be doing.

    Mozilla: We listen to web developers.
    Web developers: Fantastic! XUL/XBL are great, but we want to use web standards.
    Mozilla replied: Absolutely, we will never give you non-standard technologies like XUL/XBL.
    Web developers: Um, you may have misunderstood…

    It essentially took a huge web developer (i.e. Google) to change things for the better. (Of course, being a huge web developer, they’re doing quite well at making things difficult for small to mid-size web developers. The HTTPS-everywhere push has been so anti-IoT/maker, it’s infuriating that Mozilla lapped it up whole. Note: It doesn’t need to be like that.)

    Mozilla could improve things dramatically if it split itself into two parts: engines and products. In addition, anything that an engine provides and is available to a Mozilla product would also be available to anyone else, and in a developer-friendly way. If Chris Lord’s post is a sign of things to come, my recent pessimism about Mozilla’s prospects may come to an end soon.

    • Quite frankly, there was a significant effort to make XUL and XBL web standards. However, it soon became obvious that nobody would standardize them as they are – there would be very significant changes. So for Mozilla, the effort of switching to existing web standards and to standardized versions of XUL/XBL (e.g. XBL 2.0) would be roughly equal. Add to it that no other browser vendors showed much interest in this (predictably IMHO) and you get the idea why that effort didn’t go anywhere.

      So Mozilla instead invested into improving existing web standards, with much success IMHO. Note that HTTPS-everywhere doesn’t mean that you have to use HTTPS but it’s still a good idea – also for IoT. And Let’s Encrypt makes this significantly simpler.

      • voracity says:

        “Add to it that no other browser vendors showed much interest in this…” I assume you mean no vendor was interested specifically in XUL/XBL, and not that no vendor was interested in standardised web apps. Because the only vendor that I can remember being hostile to standardised web apps was Microsoft (who were nonetheless trying to push MS-XUL, otherwise known as XAML). Opera, Safari, Chrome — all on board.

        I think Firefox OS was largely done in the right spirit — aiming high with proposed standards-based capabilities and strict dogfooding, which would benefit web developers regardless of the success of the project itself. But the project suffered many of Mozilla’s other recurring problems: imitation over innovation, zero-risk security models that cripple nimble experimentation, a preference for in-house development rather than community development, a monolithic approach to the software and a lack of persistence and self-belief.

        But I don’t wish to be all negative. Things may not be great right now for Mozilla, but Servo and browser.html are very promising, both in terms of the products themselves and the approach to development. They won’t have much impact in the short term, but as 5 year prospects, they look very good. Let’s just hope Google doesn’t steal Mozilla’s ideas, and implement them quicker and more polished, as it has often done in the past. (If Google comes out with a parallelised browser in the next couple of years that it was working on in secret, I’ll be furious.)

        On security: Security everywhere is a great idea. HTTPS may or may not be. A centralised trust model is dangerous and at odds with nimble, small-scale experimentation, and I haven’t seen any proposals that deal with this (beyond the very limited suggestion of relaxed permissions and no HTTPS for localhost). AFAICT, HTTPS requires globally unique names. In automated contexts, this requires an internet connection — which should automatically raise eyebrows, because IoT is most safely done (perhaps ironically) with no internet connection.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

What’s this?

You are currently reading The Gecko monoculture at Il y a du thé renversé au bord de la table.

meta

%d bloggers like this: