• MSN For Firefox
• Bing For Firefox
• Twitter
• Tab Mix Plus
• IE Tab
• Undo Closed Tabs Button
• SessionPage
• Tab Scope
• Taboo
• Tree Style Tab
• New Tab JumpStart
• TabGroups Manager
• Browse Periodically
• Tab Utilities
• Tile Tabs
• IE Tab 2
• Vertical Tabs
• Master Password+
• Home Dash
• Extension Test
• Progress Bar on Tab
• Fox Splitter
• Dormancy
• Tabkit 2nd Edition
• UnloadTab
• Tabby2
• Tab Restore
• Side Tabs
• Home Dash Lite-Dial
• Fire IE
• Tab Utilities Lite CustomEdition
• Tabulous
• Bar Tab Lite X
• PageExpand
• IE Tab 2 (SeaMonkey)
• All Tabs Helper
• multiplaceHolder

All of these add-ons tap directly into undocumented, internal data structures that will disappear in the near future (expected: Firefox 25). If you are the author or maintainer of one of these add-ons, you should monitor carefully bug 874381 and its blockers for all technical details.

If you need technical help or if you need us to provide alternative public APIs, please get in touch as soon as possible.

Unfortunately, a large number of add-ons seem to rely upon these undocumented data structures. Some of their features might therefore stop working. If you are the author of one such add-on, you should monitor carefully bug 874381 and its blockers. If you realize that we are about to break your add-on, please inform us asap, so that we can work out a solution.

Our intrepid heroes, a splinter cell from Snappy, have set out on a quest to offer alternatives to all JavaScript-accessible APIs that blocks the main thread of Firefox & Mozilla Apps.

Recently completed

Various cleanups on Session Restore (Bug 863227, 862442, 861409)

Summary Currently, we regularly (~every 15 seconds) save the state of every single window, tab, iframe, form being displayed, so as to be able to restore the session quickly in case of crash, power failure, etc. As this can be done only on the main thread, just the jank of data collection is often noticeable (i.e. > 50 ms). We are in the process of refactoring Session Restore to make it both faster and more responsive. These bugs are steps towards optimizations.

Status Landed. More cleanups in progress.

Telemetry for Number of Threads (Bug 724368)

Summary As we make Gecko and add-ons more and more concurrent, we need to measure whether this concurrency can cause accidental internal Denial of Service. The objective of this bug is to measure.

Status Landed. You can follow progression in histograms BACKGROUNDFILESAVER_THREAD_COUNT and SIMPLEMEASURES_MAXIMALNUMBEROFCONCURRENTTHREADS.

Reduce number of fsync() in Firefox Health Report (Bug 830492)

Summary Firefox Health Report stores its data using mozStorage. The objective of this bug is to reduce the number of expensive disk synchronizations performed by FHR.

Status Landed.

Ongoing bugs

Out Of Process Thumbnailing (Bug 841495)

Summary Currently, we capture thumbnails of all pages, e.g. for display in about:newtab, in Panorama or in add-ons. This blocks the main thread temporarily. This bug is about using another process to capture thumbnails of pages currently being visited. This can be useful for both security/privacy reasons (i.e. to ensure that bank account numbers do not show up in thumbnails) and responsiveness (i.e. to ensure that we never block browsing).

Status In progress.

Cutting Session Restore data collection into chunks (Bug 838577)

Summary Currently, we regularly (~every 15 seconds) save the state of every single window, tab, iframe, form being displayed, so as to be able to restore the session quickly in case of crash, power failure, etc. As this can be done only on the main thread, just the jank of data collection is often noticeable (i.e. > 50 ms). This bug is about cutting data collection in smaller chunks, to remove that jank.

Status Working prototype.

Off Main Thread database storage (Bug 702559)

Summary We are in the process of moving as many uses of mozStorage out of the main thread. While mozStorage already supports doing much I/O off the main thread, there is no clear way for developers to enforce this. This bug is about providing a subset of mozStorage that performs all I/O off the main thread and that will serve as target for both ongoing refactorings and future uses of mozStorage, in particular by add-ons.

Status Working prototype.

Improvements transaction management by JavaScript API for Off Main Thread database storage (Bug 856925)

Summary Sqlite.jsm is our JavaScript library for convenient Off Main Thread database storage. This bug is about improving how implicit transactions are handled by the library, hence improving performance.

Status In progress.

Refactor how Places data is backed up (Bugs 852040, 852041, 852034, 852032, 855638, 865643, 846636, 846635, 860625, 854927, 855190)

Summary Places is the database containing bookmarks, history, etc. Historically, Places was implemented purely on the main thread, which is something we very much want to remove, as any main thread I/O can block the user interface for arbitrarily lengthy durations. This set of bugs is part of the larger effort to get rid of Places main thread I/O. The objective here is to isolate and cleanup Places backup, to later allow removing it entirely from the main thread.

Status Working prototype.

APIs for updating/reading Places Off Main Thread (Bugs 834539, 834545)

Summary These bugs are part of the effort to provide a clean API, usable by platform and add-ons, to access/modify Places information off the main thread.

Status In progress.

Move Form History to use Off Main Thread storage (Bug 566746)

Summary This bug is part of the larger effort to get rid of Places main thread I/O. The objective here is to move Form History I/O off the main thread.

Status In progress.

Make about:home use IndexedDB instead of LocalStorage (Bug 789348)

Summary Currently, page about:home uses localStorage to store some data. This is not good, as localStorage does blocking main thread I/O. This bug is about porting about:home to use indexedDB instead.

Status In progress.

Download Architecture Improvements (Bug 825588 and sub-bugs)

Summary Our Architecture for Downloads has grown organically for about 15 years. Part of it is executed on the main thread and synchronously. The objective of this meta-bug is to re-implement Downloads with a modern architecture, asynchronous, off the main thread, and accessible from JavaScript.

Status In progress.

Constant stack space Promise (Bug 810490)

Summary Much of our main thread asynchronous work uses Promises. The current implementation of Promise is very recursive and eats considerable amounts of stack. The objective here is to replace it with a new implementation of Promise that works in (almost) constant stack space.

Status Working partial prototype.

Reduce amount of I/O in session restore (Bug 833286)

Summary The algorithm used by Session Restore to back up its state is needlessly expensive. The objective of this bug is to replace it by an alternative implementation that requires much less I/O.

Status Working prototype.

Planning stage

Move session recording and activity monitoring to Gecko (Bug 841561)

Summary Firefox Health Report implements a sophisticated mechanism for determining whether a user is actively using the browser. This mechanism could be reimplemented in a more efficient and straightforward manner by moving it to Gecko itself. This is the objective of this bug.

Status Looking for developer.

Non-blocking Thumbnailing (Bug 744100)

Summary Currently, we capture thumbnails of all pages, e.g. for display in about:newtab, in Panorama or in add-ons. This blocks the main thread temporarily, as capturing a page requires repainting it into memory from the main thread. This bug is about removing completely the “repaint into memory” step and rather collaborate with the renderer to obtain a copy of the latest image rendered.

Status Design in progress.

Evaluate dropping “www.” from rev_host column (Bug 843357)

Summary The objective of this bug is to simplify parts of the Places database and its users by removing some data that appears unnecessary.

Status Evaluation in progress.

Optimize worker communication for large messages (Bug 852187)

Summary We sometimes need to communicate very large messages between the main thread and workers. For large enough messages, the communication itself ends up being very expensive for the main thread. This bug is about optimizing such communications.

Status Design in progress.

Project Snappy has been retired and replaced by several smaller projects, including Async & Responsive. The objective of this project is to improve the responsiveness of Firefox and the Mozilla Platform by converting key components to make them asynchronous and, wherever possible, to move them off the main thread.

The setting

Firefox and other Mozilla applications are great products, in particular in terms of performance. They are based on an extremely fast rendering engine, Gecko, and its companion JavaScript engine, which in addition to being the richest JS engine around, is also, these days, quite possibly the fastest. What is not so great, unfortunately, is that despite these great core performances, Mozilla applications have often been perceived as slow and sluggish.

Project Snappy was formed about 18 months ago to focus the effort by Mozilla developers to fight this perceived sluggishness. During this period, we have made tremendous progress, thanks to the commitment of everyone involved. Indeed, most of the long-term objectives of Snappy have been reached already. We have therefore decided to retire project Snappy, in favor of both a larger project Performance, and several sub-projects focusing on distinct aspects of Performance.

Let me introduce Asynchronous & Responsive [1], one of the sub-projects of Performance.

Project outline

Despite considerable progress, much of Firefox still behaves as a single-threaded application. Most services and components are initialized sequentially in the main thread, run in the main thread, are shutdown sequentially in the main thread. Also, most add-ons execute essentially in the main thread. As a consequence, any long-lived task can disrupt the user experience.

There are historical reasons for this, but in most cases, there is not deep blocker that would prevent us from rewriting services. Project Asynchronous & Responsive is now starting to support and focus the ongoing effort to get rid of main thread services and components, both in platform code and in add-on code, for the betterment of all Mozillakind.

This entails:

• identifying blockers that prevent platform and add-on developers from deploying their code on non-main threads (generally, worker threads);
• helping platform and add-on developers transition their code off-main thread;
• actually transitioning some of our services and components off the main thread.

Please note that we have no intention of working on the JavaScript VM, on DOM or Graphics. These teams already have dedicated developers working on moving things off the main thread.

Following our progress

As I am the tech lead of this project, you will find more information on this blog, under category Performance.

I will try and post updates every second week.

[1] If you have an idea of a nicer name that does not sound too much like “Snappy”, we are interested Marxist jokes about Workers might or might not be accepted.

However, all this craze is missing one feature that makes Firefox OS my choice of mobile operating system: I can write a playable prototype for a simple game, from scratch, in two hours.

Of course, this was a prototype, and completing the game took me a few more days of adding 8 bit graphics, optimizing, toying with the rules, adding difficulty levels, high scores, etc. But after just two hours, I could play the game on computer, tablet and cellphone, and decide where to proceed from here. This was both my first HTML5 game and my first mobile game, by the way. It is by no means an AAA game, but it is fun enough that I sometimes play it in the subway. By the way, did I mention that, once I was satisfied with this game, I could publish it in just a few seconds, simply by hosting it anywhere on the web?

Oh, and another feature: I wrote a quite usable comic book reader in the subway, while commuting from/to work. It took me a few days of commuting (three days, I seem to remember) to obtain a tool that works quite nicely. Due to screen size, I prefer using it on my Android tablet than on a cellphone, but that’s the wonders of HTML5 and Open Web Applications: I developed for one, and it worked for both. Did I mention that this was my first attempt at writing a web application that does file I/O or that uses the touch screen intelligently? I will try and finalize and release this application one of these days.

Now, other developers or users might not share this feeling, but this simplicity to start coding and publish and evolve a game or application is of tremendous importance to me. Because one day, I will have a child in age of playing video games. And for his birthday, I will have a chance to download a 5€ game from the Firefox Marketplace (or anywhere else), but more importantly, I will be able to build a game with his favorite characters as support cast and him as a hero. I hope he will love it. And I will not need to ask for permission.

If there is some application you want to develop, neither will you.

Well, without further ado, let us see how the student projects have progressed.

MSP #1: An Open Web App for editing rich text documents (see the application!)

Start date: October 6th, 2012.
Team: Abhishek Potnis, Amod Narvekar, Pratik Patil (Vidyavardhini’s College Of Engineering and Technology, Mumbai University, India).
Objective: Build an Open Web Application that can be used to edit simple web pages, using HTML contentEditable. This application must work offline and must be friendly to touch screen devices.
What works:

• Editing rich text documents.
• Saving documents to IndexedDB.
• Loading documents from IndexedDB.
• Removing documents from IndexedDB.
• Several drafts of the User Interface.
• The application can be installed as an Open Web Application.

What doesn’t work yet:

• The user interface is very much unfinished.
• Exporting to other applications using Web Activitities.
• Making it usable on a small screen.

Message to the team: Good work so far. I am looking forward to seeing the application once it is completed.

MSP #2: An Open Web Application e-Book reader (see the application!)

Start date: January 2013.
Team: Jonathan Barbin, Anthony Rata-Global (Université d’Évry, France).
Objective: Read e-Books in standard format epub from any Open Web Application-compliant device and browser.
What works:

• Opening a book.
• Displaying the table of contents.
• Displaying the contents of the book.

What doesn’t work yet:

• Decompressing the book should take place as a background task.
• User Interface needs to be finalized.
• Bookmarks.
• Opening a file on FirefoxOS.

Message to the team:

• Impressive work! I want to use it on my tablet.
• This project is very promising. You should communicate and attract contributors.

MSP #8: An Open Web App for reading RSS streams (see the application!)

Start date: October 30th, 2012
Team: [Guys, I’d like to have your names!] (Université d’Évry)
Objective: Read RSS streams on any open web app-enabled device.
Message to the team: I have no clue about the status of the project. It looks like stuff works, but developing in silence is not a good idea.

MSP #9: Run-run-run (see the code)

Start date: October 30th, 2012
Team: Simon Blum (Université d’Évry)
Objective: An open web application that can be used to record times and positions while running, biking, hiking, etc.
What works: No idea. I see code but I have received no word from the developers.
What doesn’t work: No idea.
Message to the team: Are you alive?

MSP #10: Map Your Friends (see the application!)

Start date: October 30th, 2012
Team: Bharath Thiruveedula, Yeshwanth Jagini, Abhiram ? (Gokaraju RangaRaju Engineering and Technology?)
Objective: A group of people wants to be able to track each other on a map in real-time. Useful for sport, games, but also rescue teams.
What works:

• Creating maps.
• Joining maps.
• Tracking the rest of the group on the map.

What doesn’t work:

• Not suited for small screens yet.
• Not installable as an Open Web Application yet.
• User interface needs some polish.
• Needs a security review.

Message to the team: Keep up the good work. As soon as the application is a little more polished, you should be able to submit it for a broader audience.

MSP #42: Bombermaaaan (see the code)

Start date: October 30th, 2012
Team: Angélo Lima, Vincent Texier, Robin Vasseur (Université d’Évry)
Objective: A great game, for the touchscreen web.
What works:

• The engine is quite playable.
• Simple AI for single-player game.
• Multi-player game using a server (I haven’t tested it yet).
• Works nicely on a cellphone.

What doesn’t work yet:

• Still some problems of performance.
• You can’t play the game from the webpage (trivial to fix).
• Not installable yet.

Messages to the team:

• That looks good. I want to see the final version!
• This project is very promising. You should communicate and attract contributors.

MSP #45: Tux of Math Command (see the application)

Start date: October 30th, 2012
Team: Yassir Barchi, Lucas Pinto, Sylvain Ribous, Virgil Triquet (Université d’Évry)
Objective: Port game Tux of Math Command to all web app-enabled devices, in particular touch screen devices.
What works:

• The game was quite playable last time I checked (not today, though, there seems to be a regression).

What doesn’t work yet:

• Levels.
• Performance issues.
• Lacks some polish.

Message to the team: Good work. You should soon be able to submit the game to the marketplace. Looking forward to the completed project!

MSP #48 A plotting calculator for FirefoxOS / the web (see the code)

Start date: January 2013
Team: Sylvie Ranga (Université d’Évry)
Objective: Using any FirefoxOS / Open Web Application-enabled cellphone as a plotting calculator.
Message to the team: I see some code. I have left issues on your tracker. I haven’t seen any reply. Not a good sign.

MSP #51: Reimplement Qix as a FirefoxOS-friendly Open Web Application (see the application!)

Start date: October 30th, 2012
Team: [Guys, I’d like to have your names!] (Université d’Évry)
Objective: A great game, for the touchscreen web.
What works:

• On the desktop, the game seems to work nicely.
• Installable.

What doesn’t work:

• Doesn’t seem to work with a touchscreen yet.
• Some performance issues.
• Doesn’t work without a connection.

Message to the team: The project looks good. However, your complete silence hurts you a lot. You haven’t been in touch, so you have received no code or feature review. Also, developing in silence hurts your chances of having a successful project, both in terms of code and in terms of users. So, please don’t be silent.

Mystery project: Where to sleep tonight? (see the code)

Start date: January 2013, I believe.
Team: Omar BAKRIM, Jean-Marc NZENZA, Souleymane KEITA (Université d’Évry)
Objective: I have no idea.
Message to the team: I hear that I am expected to mentor your project. I don’t remember ever having heard from this project before. If you want me as a mentor, you should at least come and discuss the project with me.

Qui a dit cela ? Personne, à ma connaissance. Webkit est une très bonne famille de moteurs de rendu. Suffisamment bonne pour qu’Apple et Google aient tous deux décidé de construire leurs navigateurs autour de Webkit. Mieux que cela : il y a quelques jours, Opera Software a annoncé l’abandon de Presto, le moteur de rendu qui faisait tourner Opera, et le passage à Webkit. L’un dans l’autre, les références en faveur de Webkit sont impressionnantes.

Mais il y a une chose à ne pas oublier : Webkit est une technologie âgée. La première version de Webkit a été publiée en 1998, sous le nom KTHML, soit à peu près en même temps que Gecko, le moteur de Mozilla, ou Trident, celui de Microsoft, et quatre ans avant Presto, celui d’Opera. À l’époque, les threads étaient trop lents pour être utilisés, les systèmes multi-cœurs relevaient presque de la science fiction, les cartes graphiques ne servaient qu’à jouer et les micro-processeurs n’arrêtaient pas de doubler de vitesse. En plus de cela, personne hors du monde académique ne prenait au sérieux la compilation JIT, les systèmes de types ou les langages dynamiques. Oh, et il fallait être idiot pour s’imaginer travailler sur des documents dans un navigateur, ou encore jouer à des jeux un peu complexes. Sans compter que la vie privée et le web ouvert étaient deux concepts auxquels seul Opera s’intéressait.

Nous voici quinze ans plus tard. Les utilisateurs attendent autre chose de leurs navigateurs. Sur un navigateur, on peut maintenant jouer à des jeux en 3d et utiliser des suites bureautiques. Les navigateurs doivent fonctionner sur des micro-processeurs à basse consommation. Les architectures multi-cœurs sont devenues bien moins onéreuses que les processeurs rapides et la puissance des cartes graphiques a augmenté de manière démesurée. De nos jours, la plupart des applications visent des plateformes compilées en JIT. Les langages C et C++ ont des concurrents sérieux. N’oublions pas non plus que votre navigateur est devenu la cible favorite de tous les espions du monde, de tous les mafieux et de leurs petits cousins, ou encore que Google, Facebook et les autres gagnent des milliards de dollars en déterminant exactement ce que vous êtes en train de faire devant votre navigateur. En quinze ans, Webkit, Gecko, Trident et Presto ont été améliorés, optimisés, patchés, verrouillés et réglés des milliers de fois, pour obtenir les plate-formes impressionnantes que nous utilisons maintenant tous les jours.

Il y a quelques jours, Opera a jeté l’éponge sur Presto et rejoint le mouvement Webkit. J’avoue avoir quelques doutes sur les conséquences stratégiques de ce mouvement pour Opera, mais ce n’est pas de cela que traite mon billet. Cette volte-face est bien entendu décevante. Décevante pour la communauté Mozilla car il semble bien que nous venons de perdre notre plus solide allié dans le combat pour la vie privée et le web ouvert. Décevante aussi car la raison avancée par Opera pour cet abandon est qu’il avait trop de développeurs qui développaient non pas des applications web mais des applications webkit.

C’est bien ce point qui m’inquiète, car il s’agit d’un réel danger pour l’avenir du web entier. À chaque fois qu’une application est développée à partir de standards ouverts, elle peut fonctionner sur tous les navigateurs standards – pas uniquement les navigateurs principaux, mais aussi des navigateurs moins connus ou des navigateurs qui n’existent pas encore. À l’inverse, par définition, une application webkit ne fonctionnera que sur les navigateurs webkit. Si vous êtes un développeur web, souvenez-vous de ces chiffres : cela signifie que votre application va rejeter environ 60% de ses utilisateurs potentiels. Et encore, ce chiffre suppose que 1/ tous les utilisateurs actuels d’Opera vont passer à webkit ; 2/ tous les navigateurs webkit sont compatibles bug-pour-bug et bizarrerie-pour-bizarrerie, ce qui est loin d’être le cas.

De fait, c’est même un peu pire. Si votre application webkit dépend des bugs ou bizarreries webkit, il y a de fortes chances que ces bugs et bizarreries finissent par être corrigées. Le cas échéant, d’ici un ou deux ans, votre base d’utilisateurs potentiels aura fondu, et ne resteront que les utilisateurs qui ne mettent jamais à jour leur navigateur. Soudainement, dépendre des fonctionnalités avancées et non-standard de Webkit est devenu un handicap.

Oh, et vous vous souvenez de cette non-citation à propos du boulet Webkit ? Un de ces jours, vous l’entendrez. Il n’est un secret pour personne que Mozilla travaille sur Servo, son moteur de rendu de la prochaine génération, et je serais fort surpris que Google et Microsoft ne soient pas en train de faire la même chose quelque part dans leurs laboratoires R&D. Servo est réécrit à partir de zéro. Pas avec l’objectif d’être compatible bug-pour-bug ou bizarrerie-pour-bizarrerie avec Gecko ou Webkit, mais purement à partir des spécifications du web ouvert. Sans aucun doute, les homologues de Google et de Microsoft suivent ou suivront la même stratégie, tout simplement parce que développer un moteur de rendu est une tâche d’une complexité impressionnante, et essayer d’émuler des bugs ne fait qu’ajouter un ordre de complexité.

Alors, si vous souhaitez maximiser votre base d’utilisateurs et vous assurer que votre application va continuer à utiliser dans quelques années, vous connaissez la technique :

Sauvez le web. Utilisez des standards ouverts.

]]> http://dutherenverseauborddelatable.wordpress.com/2013/02/23/qui-a-parle-de-webkit-nous-avons-parle-devolution/feed/ 3 yoric Please don't hurt the web, use open standards http://dutherenverseauborddelatable.wordpress.com/2013/02/20/its-not-about-webkit-silly-its-about-evolution/ http://dutherenverseauborddelatable.wordpress.com/2013/02/20/its-not-about-webkit-silly-its-about-evolution/#comments Wed, 20 Feb 2013 15:58:33 +0000 yoric http://dutherenverseauborddelatable.wordpress.com/?p=1229 ]]> « Webkit is a rust bucket. We can’t move away from it, because our users rely on its bugs as much as on its features, but it’s based on deprecated technologies, concepts that don’t scale anymore, and it just won’t match today’s needs or hardware. If we had any choice, we would dump the whole thing and restart from scratch. »

Who said that? Nobody, I guess. Webkit is a very nice family of engines. Nice enough that Apple and Google have been using it in major browsers. A few days ago, Opera has announced that they will be abandoning Presto, their own engine, and switch to Webkit. This is certainly an impressive portofolio.

But let’s face it: Webkit is ancient technology. Née KHTML, Webkit was first released in 1998, roughly the same date as Mozilla’s Gecko or Microsoft’s Trident, and four years before Opera’s own Presto. At the time, multi-threading was expensive, multi-core systems were a thing of the future, GPUs could only be used by games, tomorrow’s CPU was twice faster than yesterday’s, compiling C++ on various platforms involved plenty of guesswork and nobody outside of academic circles took JITs, type-safe programming or dynamic programming seriously. Oh, and nobody in their right state of mind could imagine using a browser for working on documents or for playing non-trivial games. Also, privacy and open web were two things that only Opera was even considering.

Fast-forward 15 years. Expectations about browsers have changed. Browsers now support first-person shooters and office suites. Browsers need to work on low-powered CPUs. Multi-core has become cheaper than faster CPUs, and GPUs are now insanely powerful. Most applications written nowadays target JIT-ed platforms. C and C++ have very serious competitors. And let’s not forget that every spy, every mobster and their nephew is attempting to steal control of your browser, and that Google, Facebook and others have made billions of dollars finding out exactly what you are doing online. During these 15 years, Webkit, Gecko, Trident and Presto have been upgraded, optimized, hacked, hardened and tweaked countless times into the powerhouses we use every day.

A few days ago, Opera threw the towel on Presto and joined the Webkit movement. While I have my doubts about the business consequences of this move, I am no business strategist, so I will not comment upon something that is doubtless beyond my ken. This change is, of course, disappointing. Disappointing for the Mozilla community because it seems that we have lost our strongest ally in the war for privacy and the open web. Disappointing also because the very reason given by Opera for this choice is that there were too many developers writing not Web Applications but Webkit Applications.

And see, that’s a major problem for the future of the whole web. Whenever a Web application is developed that targets open standards, it can work with any standards-compliant browser, not just one of the major browsers, but in fact any browser that has yet to be developed. On the other hand, by definition, developing a Webkit application means effectively that your application will work only on Webkit browsers. If you are a web developer, remember this statistics: this means that writing webkit applications ensures that you reject ~60% of your potential user base – and that’s assuming that all Opera users will migrate to Webkit. And that’s also assuming that all Webkit browsers are all bug-for-bug and quirk-for-quirk compatible, which is actually far from true.

Actually, it is a little worse than that. Because if your Webkit application depends upon Webkit quirks, bugs that end up fixed or non-standard features that end up abandoned, this means that, within one or two years, your potential user base will have evaporated to only the users who never upgrade their browser. All of a sudden, relying upon cutting-edge Webkit features has turned into a handicap.

Oh, and remember that non-quote about Webkit being a rust bucket? One day soon, you will hear it. It is no secret that Mozilla is working in Servo, its next-generation rendering engine, and I would be surprised if Google and Microsoft were not doing the same thing behind closed doors. Servo is written mostly from scratch. Not with the objective of being bug-to-bug and quirk-to-quirk compatible with Gecko or with Webkit, but purely from the specifications of the open web. Doubtless, Google’s and Microsoft’s hypothetical replacements for Webkit and Trident follow or will follow the same strategy, simply because writing a rendering engine is a task of considerable complexity, and attempting to reverse-engineer behavior from bugs increases complexity and likelihood of unexpectedbugs by an other of magnitude.

So, do you want to maximize your user base and ensure that your application will continue working a few years from now?

Please don’t hurt the web, use open standards

As you will see, for the moment, students seem to have a few difficulties with communication. So, here is a general message to all students working on these projects: please work in the open! Use your issue tracker, your git, communicate on IRC, blog about your work, etc. Working in silence is a good way to ensure that your project will also die in silence.

Running on time

MSP #10: Map Your Friends

Start date: October 30th, 2012
Team: [Guys, I’d like to have your names!]
Objective: A group of people wants to be able to track each other on a map in real-time. Useful for sport, games, but also rescue teams.
Message to the team: Currently on hold due to exams, but early results look quite encouraging. Sorry it took me so long to review your code. Keep up the good work, but please, please, separate cleanly your JavaScript from your PHP code.

Too early to tell

MSP #42: Bombermaaaan

Start date: October 30th, 2012
Team: [Guys, I’d like to have your names!] (Université d’Évry)
Objective: A great game, for the touchscreen web.
Message to the team: No current activity that I know of, but I see some issues on the tracker, so I will not classify this as “Worrying silence” just yet. Hey, team, what’s your status?

MSP #51: Reimplement Qix as a FirefoxOS-friendly Open Web Application

Start date: October 30th, 2012
Team: [Guys, I’d like to have your names!] (Université d’Évry)
Objective: A great game, for the touchscreen web.
Message to the team: No activity that I know of, the team recently asked questions regarding testing with FFOS, so I will not classify this as “Worrying silence” just yet. Hey, team, what’s your status?

Worrying silence

MSP #1: An Open Web App for editing rich text file

Start date: October 6th, 2012.
Team: Abhishek Potnis, Amod Narvekar, Pratik Patil (Vidyavardhini’s College Of Engineering and Technology, Mumbai University, India).
Objective: Build an Open Web Application that can be used to edit simple web pages, using HTML contentEditable. This application must work offline and must be friendly to touch screen devices.
Message to the team: For the moment, I have received a number of questions but I have not seen anything that suggests progress. Team, I am counting on you all to get back on tracks.

MSP #8: An Open Web App for reading RSS streams

Start date: October 30th, 2012
Team: [Guys, I’d like to have your names!] (Université d’Évry)
Objective: Read RSS streams on any open web app-enabled device.
Message to the team: Are you alive?

MSP #9: Run-run-run

Start date: October 30th, 2012
Team: [Guys, I’d like to have your names!] (Université d’Évry)
Objective: An open web application that can be used to record times and positions while running, biking, hiking, etc.
Message to the team: Are you alive?

MSP #45: Tux of Math Command

Start date: October 30th, 2012
Team: [Guys, I’d like to have your names!] (Université d’Évry)
Objective: Port game Tux of Math Command to all web app-enabled devices, in particular touch screen devices.
Message to the team: Are you alive?

MSP #50: A 3d demo in Rust

Start date: October 30th, 2012
Team: [Guys, I’d like to have your names!] (Université d’Évry)
Objective: Demonstrate how to unleash the power of Rust, in 3D.
Message to the team: Are you alive?

Today, I would like to tell you the tale of one of these Science-Fiction Bugs: The Thing That Killed Talos.

As they often do, this tale started innocuously, with a minor refactoring to our Session Store mechanism. Ten lines were changed in the code, to move an already asynchronous operation off the main thread. As this simple change did not alter any API and simplified the code a little, it quickly passed review. Also, as it passed all our unit tests, it was promptly landed.

And immediately reverted.

As it turns out, the patch killed Talos, our performance-testing system.

Step 1 Is that really a failure?

No amount of double-checking or updating to the latest version of our source code changed anything. This almost trivial patch, that passed all unit tests, that worked nicely when tested manually, that did not touch anything remotely related to Talos, killed Talos. Reproducibly. On all platforms.

In other words, there really was a bug, hidden somewhere in the code. The hunt for the Thing That Killed Talos was on.

At this stage, I had the following information:

• there was a bug, somewhere in the code;
• this bug was causing timeouts on all platforms, but not on the exact same tests for all platforms;
• this bug was somehow related to replacing one use of NetUtils.jsm by one essentially equivalent use of OS.File;
• the problem took place at some point during Talos shutdown [1].

Step 2 Killing me softly

To fix a bug, one needs to reproduce it. Reproducing it on our test servers was a sure thing, so I set out to reproduce it locally, figuring that it would only be a matter of a little setup and configuration effort. So, I installed Talos locally, I rebuilt Firefox with the exact same configuration, I configured Talos exactly as it is configured on the server, and I expected to end up with a sample that could be used for experimenting and debugging purposes.

Unfortunately, this is not what happened. Locally, all Talos tests passed. Again, no amount of double-checking build options or Talos options, no amount of moving backwards in the source code of Talos or Firefox, or of using the binaries produced by the test server, could change the fact that the tests succeeded. Reproducibly. On both MacOS X and Linux – I had not tested Windows.

Oh, great. This trivial issue was turning into a Science-Fiction Bug.

Additional information gathered:

• the bug was somehow related to the server on which Talos tests were executed or its configuration;
• that was going to be a long ride.

Step 3 Psychic

Since reproducing the bug locally proved impossible, the next step was to attempt and corner it remotely, by carefully experimenting with our code, and combining this with a healthy dose of psychic debugging. Unfortunately, this also meant waiting a few hours between the start of an experiment and its result. Luckily for me, I don’t sleep a lot these days, so I could keep up with a near-permanent stream of experiments.

During the course of the following week, I tinkered with the following parts of the code:

• introducing logging at every step of OS.File [result: everything was in order, all calls completed successfully];
• introducing logging in the use of OS.File by SessionStore [result: everything was in order];
• monitoring the shutdown of OS.File [result: everything was in order];
• monitoring the shutdown of threads [result: locally, everything was in order – remotely, it never finished];
• programmatically turning logging to its maximal level [result: locally, nothing interesting – remotely, some cycle collection shortly before the failure];
• determining whether the issue disappeared if we used the same temporary file that NetUtils used [result: it did not];
• determining whether the issue disappeared if all uses of OS.File were removed [result: it did];
• determining whether the issue disappeared if the number of calls to OS.File was artificially limited [result: it did not];
• determining whether the issue disappeared if OS.File stopped being used long before shutdown [result: it did not];
• determining whether the issue disappeared if OS.File cleaned up after itself manually, instead of counting on the runtime [result: it changed nothing];
• determining whether the issue disappeared if we replaced the call to OS.File with another call to OS.File that did nothing [result: it changed nothing];
• adding time information to all logs [result: remotely, the DOM worker subsystem shutdown was caught in an apparently endless loop repeated exactly every 5 seconds].

Additional information gathered:

• the DOM worker subsystem failed to shutdown;
• the issue involved a 5 second timeout in the shutdown of DOM workers;
• the offending operation in DOM workers was NS_ProcessNextEvent(), which could in turn cause a call to just about anything;
• the issue was not due to anything high-technology in OS.File, but most likely only to the fact that OS.File was spawning a thread;
• the issue might be related to cycle collection.

All the signs pointed to something preventing DOM workers from closing, but not from the code of OS.File. The hunt was still on.

Interlude – Fixing a DOM Worker shutdown bug

At some point during the hunt, I was informed of another OS.File-related bug that was also related to DOM workers shutting down improperly. By opposition to my Science-Fiction bug, however, this new bug was simple to reproduce and to fix [2].

Unfortunately, the fix did not help resolve my Science-Fiction bug. Drats.

Step 4 Working the slaves

At this stage, there did not seem to be anything more that could be learnt remotely. The only way forward was to request access to one of the machines used to run the Talos tests (the so-called “slaves”) and get Firefox through a debugger [3]. This involved a few technical stumbles along the way (in particular, Apple does seem to put a lot of efforts in not letting me install gdb on that Mac slave), but nothing that could not be fixed [4].

A quick test confirmed that the whole world had not gone crazy and that the Talos tests did fail on the slave, even when launched by me.

So, once armed with a Talos slave, a ssh connection, my trusty gdb, and a version of Talos slightly hacked into some compatibility with said gdb, I had the ability to obtain stack information:

• as expected, the immediate cause of the freeze was the DOM worker subsystem looping forever in its shutdown loop, waiting for all DOM workers to have terminated;
• the call to NS_ProcessNextEvent() did not seem to do anything interesting, just timing out;
• one of the DOM Workers (presumably the one spawned by OS.File) was effectively still active, despite the fact that had several minutes ago stopped executing any code.

Now, that is interesting. Upon further examination:

• this DOM Worker was also calling NS_ProcessNextEvent(), presumably waiting for instructions;
• the C stack of this DOM Worker is somewhat surprising, insofar as it involves no less than 5 depths of importScripts() [5].

Step 5 Shooting in the dark

What is that? Calling importScripts() is an operation that leaves a trace on the stack long after the end of the execution of said importation? This deserves at least a wild attempt at fixing the issue.

The implementation of OS.File was designed for modularity. Each file defines a small module (scoped within a function) which imports its dependencies through importScripts. Changing this to ensure that the main file imports all dependencies breaks this modularity and is not very satisfying but this is the matter of less than 20 lines of code.

And Talos confirms that it works.

Victory, I guess.

Step 6 Pitch Black

The bug is fixed, all rejoice, updated SessionStore can land.

But let’s push the matter a little further and try and determine what can have caused the issue in the first place. Unfortunately, I cannot draw decisive conclusions, but I can build a few educated guesses.

Firstly, now that we know that importScripts() is related to the issue, we can deduce a few things. Function importScripts() can be used with any URL, and can be called from any point in the source code, so it must be able to deal with slow sources. This function, which is synchronous in JavaScript is therefore implemented asynchronously in C++, as its source code confirms. This reminds me very much of synchronous XMLHttpRequest or of alert(), two features (some would say “misfeatures”) with a long history of bugs. A typical implementation of such features requires nested event loops to block the execution of JavaScript code without actually blocking the underlying thread [6]. I have not checked the corresponding parts of the implementation, but it would make sense to have the underlying thread continue to receive and enqueue messages from other threads while importScripts() is executed. This would explain nicely why the C++ stack shows 5 depths of importScripts() while the JavaScript stack is empty.

Also, since these nested event loops are introduced to deal with a slow sources, their presence suggests one factor that was missing to explain the difference between running on a Talos slave and running on a development machine: speed. A very slow input device might cause the nested event loop to run, while a fast one might avoid this. As the Talos slaves are used for benchmarking, one can theorize that the hardware is seldom or never upgraded, to keep results consistent, by opposition to unit testing slaves or developer machines.

If so, I might have accidentally hit a sophisticated race condition that might show up again one of these days on devices with slow file access, such as cellphones. If so, the issue might show up again with Firefox for Android or with Firefox OS and should be monitored.

Is this scenario credible? Ben Turner or Kyle Huey, if one of you reads these lines, you might be able to tell me whether I am wrong.

In the meantime, I will return to my code. Something new has come up. Somehow, by changing a few lines of JavaScript in our code, I have managed to trigger an Access Violation in Visual C++ 64 bits. Truly, this is life on the Frontier.

[1] My thanks to jmaher for decoding this information.

[2] My thanks to khuey for pinpointing the issue immediately and suggesting a fix.

[3] My thanks to espindola for suggesting this avenue.

[4] My thanks to the RelEng team for helping me out treading this unfamiliar ground.

[5] My thanks to bent for going “hey, that’s funny” when looking at these importScripts.

[6] Implementing the same feature in Rust or in a language with call/cc would, of course, have been different, but C++ is what we have.