Q2 2014 Report

July 1, 2014 § Leave a comment

Q2 2014 was a difficult quarter at Mozilla, with all the agitation around Brendan Eich, Australis, Media Extensions, etc. Still, I have the feeling that we managed to get a lot done despite the intense pressure. Here is a quick highlight of my main accomplishments for Q2 2014.

Session Restore

A considerable amount of my time was spent working on Session Restore. The main objective is to decrease the jank caused by Session Restore taking snapshots of the session and to decrease the time Session Restore takes to restore the state of Firefox. Much of the activity this quarter dealt with measuring performance, so as to best optimize it and improving safety.

Reworking Session Restore backups

With Firefox 33, the backups of Session Restore state have been completely redesigned. The new system should prove orders of magnitude safer, in addition to now being fully transparent.

Next steps We are still lacking measurements to confirm that this is as successful as the mathematics suggest. If you are interested, there is a mentored bug open.

Talos tests and Telemetry on Session Restore startup

Optimizing startup is difficult, and generally impossible if you do not know what to optimize. With Firefox 32 and 33, we have new benchmarks and real world measurements to help us determine immediately the influence of patches on Session Restore startup.

Next steps Using these benchmarks to experiment with possible optimizations. This is in progress.

Cleaning up Session Restore file

One of our objectives is to decrease the size of the Session Restore file, to reduce the amount of I/O (hence battery use and hardware wear and tear) and memory usage. As a first step, we have introduced a mechanism that progressively removes from the “Undo Close” feature tabs and windows that have been closed at least 2 weeks ago. Interestingly, Telemetry indicates that this clean-up has no effect on the size of the Session Restore file. Experiments run later during the quarter, using the Talos tests, also strongly suggest that the data that we could clean up and that we do not clean up yet have essentially no influence on startup duration.

Next steps I believe that this strategy will therefore not be pursued during the next quarters.

Preserving compatibility with Tor Browser

While refactoring Session Restore, we have hit a number of obstacles in the form of add-ons using private or semi-private APIs that we wished to remove. We have managed to work along with add-on authors and, as far as I know, we have not broken any add-on yet. In particular, we have maintained compatibility with the Tor Browser, which is a heavily customized distribution of Firefox targeted towards privacy.

Next steps Providing a clean API for add-ons. This will require discussing with add-on authors to find out what they need.

Async tooling

I am in charge of the Async Project, which is all about giving front-end and add-on developers tools to develop asynchronous code that does not jank. As usual, this involved plenty of activity in a number of different directions.

Auto-closing Sqlite.jsm databases (mentoring Michael Brennan)

Sqlite.jsm databases can now be closed automatically during garbage-collection. On user’s computers, this will increase safety, as failing to close a database causes shutdown-time assertion failures. However, to use resources effectively, pragmatism dictates that databaes should be closed manually, so failing to close a database in the Mozilla codebase will still cause test failures.

Reworking OS.File shutdown

On devices with little memory (typically Firefox Phones), one of the techniques used to save memory is to shutdown the OS.File worker as early as possible, re-launching it later if necessary. As it turns out, the task is more complicated than it seems, due to possibilities of race conditions. Unfortunately, this means that in some extreme cases, Firefox OS applications could lock down and fail to shutdown properly without being killed by the OS. This is now fixed. Somewhere along the way, this helped us to make the PromiseWorker used by OS.File more resilient to low-level errors.

Next steps Making the PromiseWorker usable by other modules than OS.File, including testing and add-ons.

OS.File for Android and Firefox OS

OS.File was initially designed for desktop devices. Now that it is used in a number of places on mobile devices, I have mercilessly hunted down all compatibility issues between OS.File and our two mobile platforms. Compatibility tests are now activated on all platforms and should avoid any regression.

AsyncShutdown Barrier mechanism

The shutdown process of Firefox has always been a dark and scary place, full of unspecified dependencies. As a result, any refactoring or addition a new dependency could break many things in new and interesting ways. I have introduced the AsyncShutdown Barrier mechanism that lets us specify clear, explicit and extensible dependencies, handles ordering of shutdowns, as well as error reporting if a dependency is unmet. This Barrier is now used by Sqlite.jsm, OS.File, Firefox Health Report, Session Restore, Page Thumbnails and fixes a number of major issues.

Next steps Porting AsyncShutdown Barrier to allow native components to register with it.

Fixing Firefox 30 shutdown freezes (with Tim Taubert)

Many users of Firefox 30 encountered issues that caused Firefox to freeze during shutdown. We found out that the issue was caused was triggered by Page Thumbnails and caused by a bug in ChromeWorkers, which did not handle an error case gracefully. I applied AsyncShutdown Barrier to ensure that Page Thumbnails always completed without triggering the error case, while Tim Taubert ensured that the Chrome Workers handled the error robustly.

Making Firefox Health Report shutdown more robust

While porting Firefox Health Report to AsyncShutdown, we encountered an elusive bug that manifested itself by causing rare shutdown crashes. After months of experimenting, instrumenting and attempting to fix the issue, we eventually traced it back to a more serious bug in shutdown, which apparently does not always send the proper notifications. Using the AsyncShutdown Barrier, we managed to work around the issue and make FHR’s shutdown both more robust and better instrumented in case of crash. This later helped us locate another issue that prevents a proper shutdown when some databases have been corrupted.

Next steps Fix the upstream shutdown bug, make our shutdown more resilient in case of database corruption.

Async testing

The other aspect of writing asynchronous code is making sure that developers can debug it. Now that we have hit a critical mass of developers writing async code, it was high time to help them work with it.

Rewriting Task stack traces to be meaningful

Now that we know how to handle uncaught errors, the main remaining weaknesses of Promise-based and Task-based code is that their stack traces lose much information. Since Firefox 33, Task-based stack traces are now transparently rewritten into something developer-redable. Somewhere along the way, I have also patched xpcshell and mochitests to ensure that they take advantage of this rewriting. Experience shows that this is very useful and that the runtime cost is negligible.

Next steps Evaluate the runtime cost of doing the same thing for Promise-based code.

Making xpcshell tests fail in case of uncaught promise error

Uncaught promise errors were treated by the test suites as warnings, TBPL did not report them, and they remained consequantly more often than not ignored (or even unseen) by the developers. I have reworked the xpcshell test harness to consider all uncaught promise errors as oranges and fixed all offenders.

Next steps Doing the same for mochitests. Code is ready, but a few offenders remain.

Community

Dealing with political feedback around the nomination and departure of Brendan Eich

Along with many others, I made my best to engage people who voiced their negative feedback either at the nomination or at the departure of Brendan Eich. Unfortunately, this took time and efforts, but I believe that staying in touch with our users is part of what makes the difference between Mozilla and other browser vendors.

Working with new contributors

I estimate that I have worked with ~30 potential new contributors during the quarter. Many have unfortunately decided to postpone or abandon their efforts towards contributing, but a few have stayed, to work either with me or with other teams. At the moment, I am following 5 promising contributors. In particular, I am quite happy to welcome Dexter (who is working on a very sophisticated patch to let code watch for file modifications) and Kushagra (who has landed several test suite bugs).

Next steps More of it!

Working with universities

A group of École Centrale de Lyon successfully completed an online tool to help grassroot projects find volunteers. It was nice mentoring them.

Zedge

I was invited to deliver a presentation on performance at Zedge, in Trondheim, Norway. That was fun :)

Next steps Publish the slides.

And now?

Let’s get started with Q3!

Firefox, the Browser that has your Back[up]

June 26, 2014 § 21 Comments

One of the most important features of Firefox, in my opinion, is Session Restore. This component is responsible for ensuring that, even in case of crash, or if you upgrade your browser or an add-on that requires restart, your browser can reopen immediately and in the state in which you left it. As far as I am concerned, this feature is a life-safer.

Unfortunately, there are a few situations in which the Session Restore file may be corrupted – typically, if the computer is rebooted before the write is complete, or if it loses power, or if the operating system crashes or the disk is disconnected, we may end up losing our precious Session Restore. While any of these circumstances happens quite seldom, it needs to be applied as part of the following formula:

seldom · .5 billion users = a lot

I am excited to announce that we have just landed a new and improved Session Restore component in Firefox 33 that protects your precious data better than ever.

How it works

Firefox needs Session Restore to handle the following situations:

  • restarting Firefox without data loss after a crash of either Firefox, the Operating System, a driver or the hardware, or after Firefox has been killed by the Operating System during shutdown;
  • restarting Firefox without data loss after Firefox has been restarted due to an add-on or an upgrade;
  • quitting Firefox and, later, restarting without data loss.

In order to handle all of this, Firefox needs to take a snapshot of the state of the browser whenever anything happens, whether the user browses, fills a form, scrolls, or an application sets a Session Cookie, Session Storage, etc. (this is actually capped to one save every 15 seconds, to avoid overloading the computer). In addition, Firefox performs a clean save during shutdown.

While at the level of the application, the write mechanism itself is simple and robust, a number of things beyond the control of the developer can prevent either the Operating System or the hard drive itself from completing this write consistently – a typical example being tripping on the power plug of a desktop computer during the write.

The new mechanism involves two parts:

  • keeping smart backups to maximize the chances that at least one copy will be readable;
  • making use of the available backups to transparently avoid or minimize data loss.

The implementation actually takes very few lines of code, the key being to know the risks against which we defend.

Keeping backups

During runtime, Firefox remembers which files are known to be valid backups and which files should be discarded. Whenever a user interaction or a script requires it, Firefox writes the contents of Session Restore to a file called sessionstore-backups/recovery.js. If it is known to be good, the previous version of sessionstore-backups/recovery.js is first moved to sessionstore-backups/recovery.bak. In most cases, both files are valid and recovery.js contains a state less than 15 seconds old, while recovery.bak contains a state less than 30 seconds old. Additionally, the writes on both files are separated by at least 15 seconds. In most circumstances, this is sufficient to ensure that, even of hard drive crash during a write to recover.js, at least recovery.bak has been entirely written to disk.

During shutdown, Firefox writes a clean startup file to sessionstore.js. In most cases, this file is valid and contains the exact state of Firefox at the time of shutdown (minus some privacy filters). During startup, if sessionstore.js is valid, Firefox moves it to sessiontore-backup/previous.js. Whenever this file exists, it is valid and contains the exact state of Firefox at the time of the latest clean shutdown/startup. Note that, in case of crash, the latest clean shutdown/startup might be older than the latest actual startup, but this backup is useful nevertheless.

Finally, on the first startup after an update, Firefox copies sessionstore.js, if it is available and valid, to sessionstore-backups/upgrade.js-[build id]. This mechanism is designed primarily for testers of Firefox Nightly, who keep on the very edge, upgrading Firefox every day to check for bugs. Testers, if we introduce a bug that affects Session Restore, this can save your life.

As a side-note, we never use the operating system’s flush call, as 1/ it does not provide the guarantees that most developers expect; 2/ on most operating systems, it causes catastrophic slowdowns.

Recovering

All in all, Session Restore may contain the following files:

  • sessionstore.js (contains the state of Firefox during the latest shutdown – this file is absent in case of crash);
  • sessionstore-backups/recovery.js (contains the state of Firefox ≤ 15 seconds before the latest shutdown or crash – the file is absent in case of clean shutdown, if privacy settings instruct us to wipe it during shutdown, and after the write to sessionstore.js has returned);
  • sessionstore-backups/recovery.bak (contains the state of Firefox ≤ 30 seconds before the latest shutdown or crash – the file is absent in case of clean shutdown, if privacy settings instruct us to wipe it during shutdown, and after the removal of sessionstore-backups/recovery.js has returned);
  • sessionstore-backups/previous.js (contains the state of Firefox during the previous successful shutdown);
  • sessionstore-backups/upgrade.js-[build id] (contains the state of Firefox after your latest upgrade).

All these files use the JSON format. While this format has drawbacks, it has two huge advantages in this setting:

  • it is quite human-readable, which makes it easy to recover manually in case of an extreme crash;
  • its syntax is quite rigid, which makes it easy to find out whether it was written incompletely.

As our main threat is a crash that prevents us from writing the file entirely, we take advantage of the latter quality to determine whether a file is valid. Based on this, we test each file in the order indicated above, until we find one that is valid. We then proceed to restore it.

If Firefox was shutdown cleanly:

  1. In most cases, sessionstore.js is valid;
  2. In most cases in which sessionstore.js is invalid, sessionstore-backups/recovery.js is still present and valid (the likelihood of it being present is obviously higher if privacy settings do not instruct Firefox to remove it during shutdown);
  3. In most cases in which sessionstore-backups/recovery.js is invalid, sessionstore-backups/recovery.bak is still present, with an even higher likelihood of being valid (the likelihood of it being present is obviously higher if privacy settings do not instruct Firefox to remove it during shutdown);
  4. In most cases in which the previous files are absent or invalid, sessionstore-backups/previous.js is still present, in which case it is always valid;
  5. In most cases in which the previous files are absent or invalid, sessionstore-backups/upgrade.js-[...] is still present, in which case it is always valid.

Similarly, if Firefox crashed or was killed:

  1. In most cases, sessionstore-backups/recovery.js is present and valid;
  2. In most cases in which sessionstore-backups/recovery.js is invalid, sessionstore-backups/recovery.bak is pressent, with an even higher likelihood of being valid;
  3. In most cases in which the previous files are absent or invalid, sessionstore-backups/previous.js is still present, in which case it is always valid;
  4. In most cases in which the previous files are absent or invalid, sessionstore-backups/upgrade.js-[...] is still present, in which case it is always valid.

Numbers crunching

Statistics collected on Firefox Nightly 32 suggest that, out of 11.95 millions of startups, 75,310 involved a corrupted sessionstore.js. That’s roughly a corrupted sessionstore.js every 158 startups, which is quite a lot. This may be influenced by the fact that users of Firefox Nightly live on pre-alpha, so are more likely to encounter crashes or Firefox bugs than regular users, and that some of them use add-ons that may modify sessionstore.js themselves.

With the new algorithm, assuming that the probability for each file to be corrupted is independent and is p = 1/158, the probability of losing more than 30 seconds of data after a crash goes down to p^3 ≅ 1 / 4,000,000. If we haven’t removed the recovery files, the probability of losing more than 30 seconds of data after a clean shutdown and restart goes down to p^4 ≅ 1 / 630,000,000. This still means that , statistically speaking, at every startup, there is one user of Firefox somewhere around the world who will lose more than 30 seconds of data, but this is much, better than the previous situation by several orders of magnitude.

It is my hope that this new mechanism will transparently make your life better. Have fun with Firefox!

Shutting down things asynchronously

February 14, 2014 § Leave a comment

This blog entry is part of the Making Firefox Feel As Fast As Its Benchmarks series. The fourth entry of the series was growing much too long for a single blog post, so I have decided to cut it into bite-size entries.

A long time ago, Firefox was completely synchronous. One operation started, then finished, and then we proceeded to the next operation. However, this model didn’t scale up to today’s needs in terms of performance and performance perception, so we set out to rewrite the code and make it asynchronous wherever it matters. These days, many things in Firefox are asynchronous. Many services get started concurrently during startup or afterwards. Most disk writes are entrusted to an IO thread that performs and finishes them in the background, without having to stop the rest of Firefox.

Needless to say, this raises all sorts of interesting issues. For instance: « how do I make sure that Firefox will not quit before it has finished writing my files? » In this blog entry, I will discuss this issue and, more generally, the AsyncShutdown mechanism, designed to implement shutdown dependencies for asynchronous services.

« Read the rest of this entry »

Alors comme ça, votre projet a besoin de contributeurs ?

June 21, 2013 § Leave a comment

Comment décevoir un contributeur

Il était une fois un projet de logiciel libre (ou, d’ailleurs, un projet associatif). Un jour, un anonyme se présenta et annonça qu’il souhaitait aider. C’était une bonne nouvelle, car le projet avait bien besoin de contributeurs supplémentaires. Malheureusement, au bout de quelques jours, l’anonyme disparût, car il n’arrivait pas à aider.

C’est une histoire assez triste. Elle vous est peut-être familière, soit dans la peau du contributeur existant, soit dans la peau de l’anonyme qui voulait contribuer. Cette histoire est malheureusement fréquente dans les projets qui cherchent à s’étendre. Voyons ce que nous pourrions faire pour changer la fin du conte.

Il était une fois un projet de logiciel libre (ou, d’ailleurs, un projet associatif). Un jour, un anonyme se présenta et annonça qu’il souhaitait aider. C’était une bonne nouvelle, car le projet avait bien besoin de contributeurs supplémentaires. Les contributeurs existants avaient justement préparé des documents pour guider des nouveaux venus et étaient prêts à répondre aux questions de l’anonyme. L’anonyme suivit le guide de contribution. Après avoir suivi ce guide, l’anonyme chercha à quoi il pouvait bien contribuer. Malheureusement, au bout de quelques jours ou quelques semaines, l’anonyme n’avait pas trouvé en quoi il pouvait aider et il disparût.

Malgré tous les efforts des contributeurs existants, l’histoire est toujours aussi triste. Alors que faire pour arriver à une fin heureuse ?

Il était une fois un projet de logiciel libre (ou, d’ailleurs, un projet associatif). Comme dans tous les projets de ce genre, il y avait énormément de choses à faire et pas assez de contributeurs pour tout accomplir. Les contributeurs avaient pris pour habitude de noter sur une liste de tâche facilement accessible tout ce qu’ils n’avaient pas encore eu le temps de mener à bien. Certaines de ces tâches étaient accessibles à des nouveaux venus. Pensant aux futurs contributeurs, les contributeurs existants s’assuraient donc que ces tâches accessibles étaient faciles à trouver et que n’importe quel nouveau venu pouvait facilement contacter la personne qui avait ajouté la tâche dans la liste, pour lui demander des conseils. De plus, les contributeurs existants avaient préparé des documents pour guider des nouveaux venus et étaient prêts à répondre aux questions de l’anonyme. L’anonyme suivit le guide de contribution, qui le mena à des tâches accessibles. Il trouva une tâche qui l’intéressait et un mentor pour l’aider à démarrer. Ils vécurent heureux et eurent beaucoup de contributions.

Le système des bugs mentorés

Chez Mozilla, nous utilisons depuis quelques années le système que je viens de décrire, avec un succès impressionnant. Tous les quelques jours, sur les projets que je suis, de nouveaux contributeurs se présentent, suivent les tutoriels, choisissent une tâche, se mettent immédiatement au travail – et finissent la plupart du temps par publier leurs contributions, et un peu plus tard par devenir eux-mêmes mentors sur d’autres tâches.

Marquer une tâche comme mentorée prend environ deux secondes.

  1. je viens d’ouvrir un bug sur Bugzilla et je réalise qu’un débutant pourrait certainement le traiter avec un peu d’aide ;
  2. j’ajoute dans le champ libre (« whiteboard ») du bug l’information [mentor=Yoric] – à partir de ce moment-là, les nouveaux venus peuvent trouver ce bug dans notre moteur de recherche de bugs mentorés ;
  3. j’en profite pour ajouter dans ce même champ libre l’information [lang=js][lang=c++] – à partir de ce moment-là, les nouveaux venus cherchant des bugs dans l’une des deux technologies “JavaScript” ou “C++” verront s’afficher ce bug ;
  4. c’est fini – un de ces jours, un contributeur me contactera peut-être pour demander s’il peut travailler sur ce bug.

Bien entendu, l’exemple utilise Bugzilla et des contributions techniques mais il est assez simple d’étendre le système à d’autres gestionnaires de tâches et à des tâches purement non techniques.

Pour un nouveau venu, commencer est aussi très simple :

  1. lire notre document d’introduction et suivre le lien vers le moteur de recherche de bugs mentorés ;
  2. choisir des centres d’intérêt et un bug ;
  3. contacter le mentor par mail ou par irc.

Certaines étapes peuvent encore être fluidifiées (le nom et l’adresse du mentor ne sont pas toujours évidents à trouver à l’écran, etc.) mais c’est en cours. Nous espérons que le système pourra, à terme, être généralisé à tous les projets de Mozilla, techniques ou non.

Du coup, si vous participez à un projet (Mozilla ou autre) qui n’emploie pas un tel système de bugs mentorés et qui cherche des contributeurs, je vous invite vivement à essayer.

Mozilla Student Projects update

February 27, 2013 § 3 Comments

It has been quite some time since the last update. Since then, many things have happened, both with the Student Projects and with the world of Mozilla. We have had the exciting FirefoxOS AppDays, many alpha, beta and near-final versions of FirefoxOS, and the MWC launch of FirefoxOS.

Well, without further ado, let us see how the student projects have progressed.

« Read the rest of this entry »

Asynchronous file I/O for the Mozilla Platform

October 3, 2012 § 18 Comments

The Mozilla platform has recently been extended with a new JavaScript library for asynchronous, efficient, file I/O. With this library, developers of Firefox, Firefox OS and add-ons can easily write code that behave nicely with respect to the process and the operating system. Please use it, report bugs and contribute.

Off-main thread file I/O

Almost one year ago, Mozilla started Project Snappy. The objective of Project Snappy is to improve, wherever possible, the responsiveness of Firefox, the Mozilla Platform, and now, Firefox OS, based on performance data collected from volunteer users. Thanks to this real-world performance data, we have been able to identify a number of bottlenecks at all levels of Firefox. As it turns out, one of the main bottlenecks is main thread file I/O, i.e. reading from a file or writing to a file from the thread that also runs most of the code of Firefox and its add-ons.

« Read the rest of this entry »

Appel à enseignants/intervenants

September 26, 2012 § 2 Comments

Pour Mozilla, l’année universitaire 2012-2013 est l’année de Firefox OS, l’année des Open Web Applications et l’année à laquelle la communauté Mozilla lance sa campagne de libération des téléphones et applications portables !

Nous avons besoin de vous pour enseigner les technologies des Open Web Applications aux communautés francophones.
L’objectif des cours est de former des étudiants ingénieurs ou équivalents aux technologies nécessaires pour le développement ouvert d’applications web ouvertes. Nous cherchons notamment des enseignements sur les sujets suivants :
  • JavaScript avancé (clôtures, événements, prototypes, itérateurs/générateurs, timeouts)

Déroulement des cours

Autant que possible, les cours seront donnés en français face à une classe de MIAGE de  l’Université d’Évry, où ils seront aussi enregistrés. Les cours seront mis à disposition de toute la communauté Mozilla francophone dans le cadre de Mozilla Education. Les documents correspondants seront eux-mêmes mis à  disposition sous licence libre sur github.
Si  vous ne pouvez pas vous rendre à Évry, mais si vous pouvez enregistrer des cours par vous-mêmes, n’hésitez pas à nous contacter. Nous mettrons vos cours à disposition du public francophone.
Si vous avez d’autres idées, n’hésitez pas à nous les proposer. En ce moment, nos capacités d’enregistrement sont limitées mais nous ferons notre possible pour enregistrer et mettre votre cours à disposition.

Pour participer

  • Si vous pouvez assurer ces cours (entièrement ou en partie), suivez les liens ci-dessus
  • Pour proposer d’autres cours que vous pouvez vous-même assurer, ajoutez la description de vos cours sur https://github.com/Yoric/Mozilla-Courses/issues (cliquez sur “New issue”). Précisez si vous avez des contraintes géographiques ou besoin de matériel spécifique.

Et après ?

Notre objectif est d’étendre cette initiative hors de la région parisienne et surtout hors de France. Nous espérons notamment pouvoir organiser des cours en Afrique Francophone. La suite au prochain épisode !

 

 

 

Where Am I?

You are currently browsing entries tagged with open-source at Il y a du thé renversé au bord de la table.

Follow

Get every new post delivered to your Inbox.

Join 31 other followers