How to be an evil start-up CEO

April 1, 2014 § 3 Comments

Being the CEO of a start-up is fun. Being evil and mischievous is fun. Completely destroying one’s life dream is fun. However, reaching expertise in all three requires considerable subtlety. Here are a few notes for the day I decide to become an evil mischievous start-up CEO.

  1. I will keep in mind that my main currencies are time and credibility, both inside and outside my startup. Therefore, I will make my best to maintain that credibility and save that time.
  2. For this reason, although I pay them, I will describe my employees as trusted colleagues. I will, however, treat them as incompetent children.
  3. Conversely, to ensure credibility, I will encourage my trusted colleagues to worship me.
  4. For some reason, my relationship with trusted colleagues tends to alter when trusted colleagues realize that I lie to them. Which is why I will use threats and dissimulation to ensure that they do not.
  5. Being worthy of worship, I am the sole holder of the truth. Consequently, everything I have just told my investors or prospects is true. Trusted colleagues who fail to base their reality upon my truth will be punished.
  6. I will have trusted advisors, be they COO, CTO, CSO, CFO, GPU, tech leads, mentors, janitors, nannies or anything else. Listening to them is important. However, I know better, so there is no need to take anything they say into account.
  7. Being a CEO is all about taking decisions quickly. For this reason, I will avoid smoking pot or drinking alcohol. I will remain on coke.
  8. One of the roles of my trusted advisors is to help me differentiate the real world from my imagination. Do they wonder aloud whether my Reality Adjustment Factor is misaligned? Well, that is the sign that I should put them on coke, too.
  9. I will realize that some of my trusted advisors might be polite. Therefore, if one of them asks “er… are you really, really sure?”, I will take this as a hint that they may be politely inquiring about my being high on LSD. Since I am actually high on coke, there is nothing to worry about.
  10. If I have to divide my start-up in teams, I can ensure that teams can work in complement of each other. Of course, I can also ensure that teams will be at each other’s throat, which is much more amusing, especially if I live in a country where pitbull fighting is illegal. If I do organize my start-up as a pitbull fighting ring, this will, of course, open the possibility of taking bets to determine which team goes down first.
  11. Nothing motivates trusted colleagues quite as much as calling their colleagues “stupid”, “lazy” or “incompetent”, except perhaps calling them “stupid”, “lazy” or “incompetent” within earshot of said colleagues.
  12. Also, nothing motivates trusted colleagues quite as much as non-existent deadlines for imaginary clients on fabulous contracts. My trusted colleagues will be permitted to thank me for such managerial prowesses.
  13. I will claim that Microsoft, or Google, or Mozilla, or Apple, or Amazon, or Facebook, are a bunch of incompetent morons. They merely got to #1 in their respective sectors while I intend to do nothing less than revolutionize the world!
  14. In the spirit of encouraging team work, I will occasionally let a trusted colleague put his/her name along with mine as a co-author/inventor/creator of the work they have done. This way, in case of problem, it will be easy to find someone to take the blame.
  15. I could organize my company so that decisions are taken at the appropriate level by my trusted colleagues. However, this is clearly inefficient, so all decisions, no matter how small, must go through me. Should this be tiresome, I reserve the right to turn off my cellphone while I sip a Piña Colada in the sun.
  16. I realize that some of my trusted colleagues will be better than me at something. With time, many might end up better than me at most things. I could save face by deciding to take this as a proof of my recruiting skills, but this is hardly as satisfying as belittling their achievements and skills and then firing them.
  17. If I need to get rid of one of my trusted colleagues, I will have three options. The first one is to offer conditions to that trusted colleague for leaving. The second one is to fire the trusted colleague. The last one is to mount a cabal inside my start-up to get that trusted colleague to leave in disgust. The cabal-oriented solution will prove much more diverting, as I will be able to watch the cascade of consequences, plots and counter-plots, the consequent loss of time, productivity and morale, and I will be able to find out just exactly how much a lawyer charges for defending my company in court.
  18. At some point, my dream project will near completion. By then, I will have dreamed up tons of new features, and it only makes sense to start piling them up until requirements dwarf what has been realized so far. My trusted colleagues will certainly complete these final few features in a matter of days. Additionally, by the time my trusted colleagues are done, I can certainly have dreamed up a few new features. Ain’t that great?
  19. Also, a project approaching completion signals that I can reassign everybody to another project. The project will certainly find a way to finish itself.
  20. It may happen that my project cannot have all of the following features: working, released, everything I dreamed it to be. The first two features are not really important, so I can remove either.
  21. If a project has failed, or if the company has pivoted, I will inform my trusted employees. Of course, I might decide to keep the news for after the end of an ongoing death-march-to-finish-the-project-in-emergency. Just imagine the look on their face.
  22. This is also true for my commercially-oriented trusted colleagues. Just imagine them telling to their prospects that everything they have promised until this date is false.
  23. As a CEO, I will be approached by countless people with ideas. With a little effort, this will give me the opportunity to pivot as often as twice a week. My trusted colleagues need the exercise.
  24. Being an avid Mac user, I can do as well as Steve Jobs. Even better, being a Facebook user, I can also do as well as me Mark Zuckerberg. Where else could you find a CEO as exceptional as me?
  25. I will occasionally take breaks, or even vacations. Whenever I do go on vacation, however, I will make sure to keep this a secret. Just think how funny it will be when they realize how much time they have wasted coming by every few minutes to check if I had arrived in the office.
  26. While this is my company, not all trusted colleagues may realize that its money is mine to use as I see fit and that I can invest it in my luxury vacations. Some of them might conclude that I am abusing everybody’s trust, work and livelihood. The simplest solution is to fire them, but I should check with a lawyer whether I could also sue them.
  27. While I may have to lie to potential clients and investors, I will refrain from lying to (alleged) mobsters and secret services. My health matters too much to me.
  28. Also, should I be faced with (alleged) mobsters and secret services, and should I determine that the people in front of me are morons and/or easy milk cows, I will refrain from making this realization obvious to said idiots.
  29. Some of my colleagues will insist that we should not release a product that we have not tested. If they refuse to use our product because they hate it, it means that the product is ready for release. If I refuse to use our project because I hate it, it means that we should restart from scratch, re-implement everything in two weeks, then fire my trusted colleagues.
  30. From time to time, I will feel like taking a scapegoat among my trusted colleagues, because this is a cheap way to vent out frustration at failed projects or meetings. After all, there is no way said trusted colleague could figure out that competition would pay more and grab the experience gained at my company.
  31. If I attempt to enter a market saturated of products that are free-to-use and just as good as mine – possibly even dominated by Google, Microsoft, Apple or Mozilla – my investors and trusted colleagues should not worry, as I have a secret weapon: I am the smartest person in the world.
  32. If my product requires that user give up on their existing data, about their existing code, about their existing infrastructure, I will not be too surprised when users rather decide to give up on my product. That is because users are stupid.
  33. If potential users start describing my sole product as vaporware, I can just retort that Microsoft got around with delivering vaporware for more than 10 years. Of course, Microsoft got to #1 before doing so, but we are almost there ourselves.
  34. Occasionally, I will make mistakes and choose wrong paths for my company. The most amusing manner of getting away with mistakes is to pretend that my trusted colleagues have disobeyed me and taken bad initiatives. Doing so is, of course, a mistake and places the company on a wrong path, which makes things recursively more amusing [1].
  35. My trusted colleagues will often be occupied with non-productive trivialities such as building my product, signing paychecks or hunting for clients. I will have to remind them regularly that I am the sole efficient worker in this company.
  36. As my Reality Distortion Field can tell you, there is no difference between “clients”, “contacts” and “people who have looked at the website”. For instance, if our website gets 30,000 hits per month (or per day), we surely have 30,000 paying users.
  37. Occasionally, my trusted colleagues will reach milestones, possibly even releasable/deployable versions of our product. These milestones/versions will probably not match my dream ideas. The simplest way to deal with such imperfections is to treat these unworthy versions with disgust, refuse to give them a name or number and retroactively add some set of features that must absolutely be completed before the milestone is hit.
  38. Occasionally, I will be unhappy with some of my trusted colleagues, but not sufficiently to fire them. I will assigning them to a punishment project, designed solely for maximal pain. That will teach them!
  39. If one of my projects succeeds, I will know – and remind everybody – that I am the sole reason for this success.
  40. If one of my ideas succeeds, I will attempt to remember if, by any chance, one of my trusted colleagues may have spent some time attempting to convince me that this was a good idea, before this was my idea. If so, I will make sure that he/she understand how the first idea was bad and mine was a stroke of genius. If this is not sufficient, I will fire him/her.
  41. My main argument for selling to potential clients will be that they have been idiots to not use my product/service. With my help, however, they can become intelligent enough to know that they should buy my wares. Am I not generous?
  42. Although I may not have heard of weird concepts such as revision control systems, bug trackers or customer relationship managers, if one of my trusted colleagues informs me that they cannot work without such exotic software, I will simply dismiss my trusted colleague as incompetent. After all, I could do the work without such niceties, so they cannot be really useful, can they?
  43. Once I know of weird concepts such as revision control systems, bug trackers or customer relationship managers, I should probably build one. After all, how hard can it be? Additionally, I am so much smarter than everybody who has built one before us. The world will be so grateful that they will beg to use our product, once we have gotten around to building it.
  44. Should I decide that we need to reinvent yet another revision control systems, bug tracker or customer relationship managers and to use it, I will make sure that the product is tested, at least marginally, before it hosts our critical data. Of course, if my trusted colleagues insist that the product does not work, there is always time to ignore their feedback. Otherwise, how could I watch the revision control system lose all the source code of the revision control system?
  45. Should investors ever decide to audit my company with the perspective of buying it, I will adjust my Reality Perception Filter to ensure that the esteemed idiots rake. This will, of course, not prevent me from complaining to my trusted colleagues that all this work was for naught and for so little money. I would not want them to waste their health by envying me too much.
  46. Of course, to ensure credibility, I will make sure that said trusted colleagues will not receive one cent from the sale. Their health really is that important to me.
  47. As my company will undoubtedly be #1 soon, I will make sure that my salary corresponds to that rank. Unfortunately, trusted colleagues will have to satisfy themselves with salaries slightly below the minimum wage, to ensure the survival of the company. Remember, we are just a start-up.
  48. I might I come from a commercial background, but I know technical matters better than my trusted technical colleagues. After all, I know Excel.
  49. I might I come from a technical background, but I know commercial matters better than my trusted commercial colleagues. After all, I know Excel.
  50. It is a well-known fact that trusted colleagues are wimps and that they burnout faster than matches. I will keep vigilant for such occurrences, not only because it is fun to watch one trusted colleague turn into an improductive bag of nerves that the mere mention of my name will send into fits of dementia, but also because with the proper balance of (mis)management, such burnouts can propagate faster than forest fires.

Thanks for the contributions of Team TGCM, Team Malsain, Team Dixous, Team HokutoNoOpa. No animals were hurt in the process, but I intend to remedy this in part 2, to be released soon.

[1] Or coinductively more amusing, if you want to be precise.

The Battle of Session Restore – Pilot

March 26, 2014 § 7 Comments

Plot Our heroes received their assignment. They had to go deep into the Perflines, in the long lost territory of Session Restore, and do whatever it took to get Session Restore back into Perfland. However, they quickly realized that they had been sent on a mission without return – and without a map. This is their tale.

Session Restore is a critical component of Firefox. This component records the current state of your browser to ensure that you can always resume browsing without losing the state of your browser, even if Firefox crashes, if your computer loses power, or if your browser is being upgraded. Unfortunately, we have had many reports of Session Restore slowing down Firefox. In February 2013, a two person Perf/Fx-team task force started working on the Performance of Session Restore. This task force eventually grew to four persons from Perf, Fx-team and e10s, along with half a dozen of punctual contributors.

To this day, the effort has lasted 13 months. In this series of blg entries, I intend to present our work, our results and, more importantly, the lessons we have learnt along the way, sometimes painfully.

Fixing yes, but fixing what?

We had reports of Session Restore blocking Firefox for several seconds every 15 seconds, which made Firefox essentially useless.

The job of Session Restore is to record everything possible of the state of the current browsing session. This means the list of windows, the list of tabs, the current address of each tab, but also the history of each tab, scroll position, anchors, DOM SessionStorage. session cookies, etc. Oh, and this goes recursively for both nested frames and history. All of this is saved to a JSON-formatted file called sessionstore.js, every 15 seconds of user activity. To this day, the largest reported sessionstore.js files is 150Mb, but Telemetry indicates that 95% of users used to have a file of less than 1Mb (numbers are lower these days, after we spent time eliminating unnecessary data from sessionstore.js).

We started the effort to fix Session Restore from only a few bug reports:


  • sometimes, users lost sessionstore.js data;
  • sometimes, data collection took ages.

Unfortunately, we had no data on:


  • the size of the file;
  • the actual duration of data collection;
  • how long it took to write data to the disk.

To complicate things further, Session Restore had been left without owner for several years. Irregular patching to support new features of the web and new configurations had progressively turned the code and data structures into a mess that nobody fully understood.

We had, however, a few hints:

  • Session Restore needs to collect lots of data;
  • Session Restore had been designed a long time ago, for users with few tabs, and when web pages stored very little information;
  • serializing and writing to JSON is inefficient;
  • in bad cases, saving could take several seconds;
  • the collection of data was purely monolithic;
  • reading and writing data was done entirely on the main thread, which was a very bad thing to do;
  • the client API caused full recollections at each request;
  • the data structure used by Session Restore had progressively become an undocumented mess.

While there were a number of obvious sources of inefficiency that we could fix without further data, and that we set out to fix immediately. In a few cases, however, we found out the hard way that optimizing without hard data is a time-consuming and useless exercise. Consequently, a considerable part of our work has been to use Telemetry to determine where we could best apply our optimization effort, and to confirm that this effort yielded results. In many cases, this meant adding coarse-grained probes, then progressively completing them with finer-grained probes, in parallel with actually writing optimizations.

To be continued…

In the next episode, our heroes will fight Main Thread File I/O… and the consequences of removing it.

Shutting down things asynchronously

February 14, 2014 § Leave a comment

This blog entry is part of the Making Firefox Feel As Fast As Its Benchmarks series. The fourth entry of the series was growing much too long for a single blog post, so I have decided to cut it into bite-size entries.

A long time ago, Firefox was completely synchronous. One operation started, then finished, and then we proceeded to the next operation. However, this model didn’t scale up to today’s needs in terms of performance and performance perception, so we set out to rewrite the code and make it asynchronous wherever it matters. These days, many things in Firefox are asynchronous. Many services get started concurrently during startup or afterwards. Most disk writes are entrusted to an IO thread that performs and finishes them in the background, without having to stop the rest of Firefox.

Needless to say, this raises all sorts of interesting issues. For instance: « how do I make sure that Firefox will not quit before it has finished writing my files? » In this blog entry, I will discuss this issue and, more generally, the AsyncShutdown mechanism, designed to implement shutdown dependencies for asynchronous services.

« Read the rest of this entry »

Is my data on the disk? Safety properties of OS.File.writeAtomic

February 5, 2014 § 1 Comment

If you have been writing front-end or add-on code recently, chances are that you have been using library OS.File and, in particular, OS.File.writeAtomic to write files. (Note: If you have been writing files without using OS.File.writeAtomic, chances are that you are doing something wrong that will cause Firefox to jank – please don’t.) As the name implies, OS.File.writeAtomic will make efforts to write your data atomically, so as to ensure its survivability in case of crash, power loss, etc.

However, you should not trust this function blindly, because it has its limitations. Let us take a look at exactly what the guarantees provided by writeAtomic.

Algorithm: just write

Snippet OS.File.writeAtomic(path, data)

What it does

  1. reduce the size of the file at path to 0;
  2. send data to the operating system kernel for writing;
  3. close the file.

Worst case scenarios

  1. if the process crashes between 1. and 2. (a few microseconds), the full content of path may be lost;
  2. if the operating system crashes or the computer loses power suddenly before the kernel flushes its buffers (which may happen at any point up to 30 seconds after 1.), the full content of path may be lost;
  3. if the operating system crashes or the computer loses power suddenly while the operating system kernel is flushing (which may happen at any point after 1., typically up to 30 seconds), and if your data is larger than one sector (typically 32kb), data may be written incompletely, resulting in a corrupted file at path.

Performance very good.

Algorithm: write and rename

Snippet OS.File.writeAtomic(path, data, { tmpPath: path + ".tmp" })

What it does

  1. create a new file at tmpPath;
  2. send data to the operating system kernel for writing to tmpPath;
  3. close the file;
  4. rename tmpPath on top of path.

Worst case scenarios

  1. if the process crashes at any moment, nothing is lost, but a file tmpPath may be left on the disk;
  2. if the operating system crashes or the computer loses power suddenly while the operating system kernel is flushing metadata (which may happen at any point after 1., typically up to 30 seconds), the full content of path may be lost;
  3. if the operating system crashes or the computer loses power suddenly while the operating system kernel is flushing (which may happen at any point after 1., typically up to 30 seconds), and if your data is larger than one sector (typically 32kb), data may be written incompletely, resulting in a corrupted file at path.

Performance almost as good as Just Write.

Side-note On the ext4fs file system, the kernel automatically adds a flush, which transparently transforms the safety properties of this operation into those of the algorithm detailed next.

Native equivalent In XPCOM/C++, the mostly-equivalent solution is the atomic-file-output-stream.

Algorithm: write, flush and rename

Use OS.File.writeAtomic(path, data, { tmpPath: path + ".tmp", flush: true })

What it does

  1. create a new file at tmpPath;
  2. send data to the operating system kernel for writing to tmpPath;
  3. close the file;
  4. flush the writing of data to tmpPath;
  5. rename tmpPath on top of path.

Worst case scenarios

  1. if the process crashes at any moment, nothing is lost, but a file tmpPath may be left on the disk;
  2. if the operating system crashes, nothing is lost, but a file tmpPath may be left on the disk;
  3. if the computer loses power suddenly while the hard drive is flushing its internal hardware buffers (which is very hard to predict), nothing is lost, but an incomplete file tmpPath may be left on the disk;.

Performance some operating systems (Windows) or file systems (ext3fs) cannot flush a single file and rather need to flush all the files on the device, which considerably slows down the full operating system. On some others (ext4fs) this operation is essentially free. On some versions of MacOS X, flushing actually doesn’t do anything.

Native equivalent In XPCOM/C++, the mostly-equivalent solution is the safe-file-output-stream.

Algorithm: write, backup, rename

(not landed yet)

Snippet OS.File.writeAtomic(path, data, { tmpPath: path + ".tmp", backupTo: path + ".backup"})

What it does

  1. create a new file at tmpPath;
  2. send data to the operating system kernel for writing to tmpPath;
  3. close the file;
  4. rename the file at path to backupTo;
  5. rename the file at tmpPath on top of path;

Worst case scenarios

  1. if the process crashes between 4. and 5, file path may be lost and backupTo should be used instead for recovery;
  2. if the operating system crashes or the computer loses power suddenly while the operating system kernel is flushing metadata (which may happen at any point after 1., typically up to 30 seconds), the file at path may be empty and backupTo should be used instead for recovery;
  3. if the operating system crashes or the computer loses power suddenly while the operating system kernel is flushing (which may happen at any point after 1., typically up to 30 seconds), and if your data is larger than one sector (typically 32kb), data may be written incompletely, resulting in a corrupted file at path and backupTo should be used instead for recovery;

Performance almost as good as Write and Rename.

Making Firefox Feel as Fast as its Benchmarks – part 3 – Going multi-threaded

October 29, 2013 § 11 Comments

As we saw in the previous posts, our browser behaves as follows

function browser() {
  while (true) {
    handleEvents();  // Let's make this faster

The key to making the browser smooth is to make handleEvents() do less. We have already discussed the ongoing work to make Firefox multi-process, their goals and their limitations. Another, mostly orthogonal, path, is to go multi-threaded.

Going multi-threaded

Going multi-threaded is all about splitting the event loop in several loops, executed concurrently, on several cores whenever applicable and possible:

function browser() {
  main() ||| worker() ||| worker() // Running concurrently

task main() { // Main thread (time-critical)
  while (true) {
    handleEvents(); // Some of your code here

task worker() {
  while (true) {
    handleEvents(); // Some of your code here

task worker() {
  while (true) {

The main thread remains time-critical and needs to loop 60 times per second, while other threads handle some of the workload of both handleEvents() and updateDisplay(). Now, this treatment is only useful if we can isolate operations that slow down the main loop measurably. As it turns out, there are many such operations lying around, including:

  • Network I/O;
  • Disk I/O;
  • Database I/O;
  • GPU I/O;
  • Treating large amounts of data.

It is easy to see why Network I/O could considerably slow down the main loop, if it were handled on the main thread – after all, some requests take several seconds to receive a reply, or never do, and if the main thread had to wait for the completion of these requests before it proceeded, this would cause multi-second gaps between two frames, which is simply not acceptable.

The cost of disk I/O, however, is often underestimated. Few developers realize that _any_ disk operation can take an unbounded amount of time – even closing a file or checking whether a file exists can, in some cases, take several seconds. This may seem counter-intuitive, as these operations do very little besides book-keeping, but one must not forget that they rely upon the device itself and that said device can unpredictably become very slow, typically because it is otherwise busy, or asleep – or even because that device is actually a network device. Database I/O is a special case of Disk I/O that we generally single out because its cost is often much higher than users suspect – recall that, in addition to saving, a database management system will typically need to maintain a journal and to flush the drive regularly, to protect data against both software or hardware failures, including sudden power loss. Consequently, unless the database has been heavily customized to lift the safety requirements in favor of performance, you should expect that every operation on your database will cause heavy disk I/O.

Finally, treating large amounts of data, or applying any other form of heavy algorithm, will of course take time.

None of these operations should take place on the main thread. Moving them off the main thread will largely contribute to getting rid of the jank caused by these operations.

Coding for multi-threading

In the Firefox web browser, threads are materialized as instances of nsIThread in C++ code and as instances of ChromeWorker in JavaScript code. For this discussion, I will concentrate on JavaScript code as refactoring C++ code is, well, complicated. Side-note: if you are new here, recall that Chrome Workers have nothing to with the Chrome Web Browser and everything to do with the Mozilla Chrome, i.e. the parts of Gecko and Firefox written in JavaScript.

Chrome Workers are an extension of Web Workers, and have the same semantics, plus a few additions. Instantiating a ChromeWorker requires a source file:

let worker = new ChromeWorker("resource://path/to/my_file.js");

We may send messages to and from a Chrome Worker

// In the parent

// In the worker

and, of course, receive messages

// In the parent
worker.addEventListener("message", function(msg) {
// A copy of the message appears in

// In the worker
self.addEventListener("message", function(msg) {
// A copy of the message appears in

In either case, the contents of the message gets copied between threads, with essentially the same semantics as JSON.stringify/JSON.parse. If necessary, binary data in messages (ArrayBuffer or the upcoming Typed Objects) can be transferred instead of being copied, which is faster.

As Web Workers, Chrome Workers are very good to perform computations. In addition, they have a number of low-level libraries to access system features. Such libraries can be loaded with the chrome worker module loader:

let MyModule = require("resource://...");

Further modules can be defined for consumption with the chrome worker module loader:

module.exports = {
  foo: // ...

Finally, they can call into C code using the js-ctypes foreign function interface:

let lib ="path/to/my_lib");
let fun = lib.declare("myFunction", ctypes.void); // void myFunction()
fun(); // Call into C

Combining the module loader and js-ctypes makes for a powerful combination that has been used to provide access to low-level libraries, including low-level file manipulation (module OS.File), phone communication (module RIL, shorthand for Radio Interface Layer), file (de)compression, etc.


Where multi-process is good at protecting a process against other processes, going multi-threaded works nicely to protect a process (a tab, the ui, etc.) against itself. Threads take up much less resources than processes and are also much faster to start and stop. However, they have very strict limitations.

The main limitation is that they do not have access to all the main thread APIs. Each API needs to be ported individually to chrome workers. Until recently, there was no manner to define or load modules. At the moment, there is no way to read or write a compressed file from a Chrome Worker, or to access a database from a Chrome Worker. In most cases, this is only a question of time and manpower, and we can hope to eventually bring almost all important APIs to Chrome Workers. However, some APIs cannot be ported at all, in particular any API that requires a DOM window, which is most (fortunately not all) DOM APIs.

Also, the paradigm behind Chrome Workers is purely asynchronous. This means that there is no way for a Chrome Worker to wait synchronously until some treatment has been completed by the main thread. This complicates code in a few cases but, in general, this is rarely a problem.

Also, the communication mechanism needs to be taken into account:  as copying long messages can block the main thread. In some cases, it may be necessary to perform aggressive optimization of messages to avoid such situations.

Refactoring for multi-threading

The first thing to take into consideration while refactoring for multi-process is whether this is the best strategy. Since most APIs and most customization possibilities live on the main thread, most features need to be produced and/or consumed by the main thread. This does not mean that going multi-threaded is not possible, only that your code will probably end up looking like an asynchronous API meant to be used mostly on the main thread but implemented off the main thread. This also means that your consumers must be architectured to accept an asynchronous API. We will cover making things asynchronous in another entry of this series.

Once we have decided to go multi-threaded, the next part is to determine what goes of the main thread. Generally, you want to move as much as you can off the main thread. The only limits are things that you simply cannot move off the main thread (e.g. access to the document), or if you realize that the data you need to copy (not transfer) across threads will slow down the main thread inacceptably. This, of course, is something that can be determined only by benchmarking.

Next, you will need to define a communication protocol between the main thread and the worker. Threads communicate by sending pure data (i.e. objects without methods, without DOM nodes, etc.) and binary data can be transfered for high-performance. Recall that communications are asynchronous, so if you want a thread to respond to another one, you will need to build into your protocol identification to match a reply to a request. This is not built-in, but quite easy to do. Handling errors requires a little finesse, as uncaught exceptions on the worker are transmitted to a onerror listener instead of the usual onmessage listener, and lose some information along the way.

In some (hopefully rare) cases, you will need to add new bindings to native code, so as to call C functions (only C, not C++) from JavaScript. For this purpose, take a look at the documentation of js-ctypes, our JavaScript FFI, and osfile_shared_allthreads.jsm, a set of lightweight extensions to js-ctypes that handle a number of platform-specific gotchas. As finding the correct libraries to link is sometimes tricky, you should take advantage of OS.Constants.Path, that already lists some of them. Don’t hesitate to file bugs if you realize that something important is missing. Also, in a few (hopefully almost non-existent) cases, you will need to expose additional C code to native code, typically to expose some C++-only features. For this purpose, take a look at an example.

Unsurprisingly, the next step is to write the JS code. The usual caveats apply, just don’t forget to use the module system. Worker code goes into its own file, typically with extension “.js”. It is generally a good idea to mention “worker” in the name of the file, e.g. “foo_worker.js”, and to deploy your code to "resource://.../worker/..." or "chrome://.../worker/..." to avoid ambiguities. To construct the worker, it is then sufficient to call new ChromeWorker("resource://path/to/your/file.js"). The worker code will be started lazily when the first message is sent.

For automated testing, you can for instance use mochitest-chrome or (once bug 930924 has landed) xpcshell-tests. In the latter, if you need to add new worker code for the sake of testing, you should install it with the chrome:// protocol. Also, for any testing, don’t forget to look at your system console, as worker errors are displayed on that console by default.

That’s it! In a future blog entry, I will write more about common patterns for writing or refactoring asynchronous code, which comes in very handy for code that uses your new API.


Refactoring Firefox as a set of asynchronous APIs backed by off main thread implementations is a considerable task. To make it happen, the best way is to contribute to coding, testing or documentation

Copying streams asynchronously

October 18, 2013 § Leave a comment

In the Mozilla Platform, I/O is largely about streams. Copying streams is a rather common activity, e.g. for the purpose of downloading files, decompressing archives, saving decoded images, etc. As usual, doing any I/O on the main thread is a very bad idea, so the recommended manner of copying streams is to use one of the asynchronous string copy APIs provided by the platform: NS_AsyncCopy (in C++) and NetUtil.asyncCopy (in JavaScript). I have recently audited both to ascertain whether they accidentally cause main thread I/O and here are the results of my investigations.

In C++

What NS_AsyncCopy does

NS_AsyncCopy is a well-designed (if a little complex) API. It copies the full contents of an input stream into an output stream, then closes both. NS_AsyncCopy can be called with both synchronous and asynchronous streams. By default, all operations take place off the main thread, which is exactly what is needed.

In particular, even when used with the dreaded Safe File Output Stream, NS_AsyncCopy will perform every piece of I/O out of the main thread.

The default setting of reading data by chunks of 4kb might not be appropriate to all data, as it may cause too much I/O, in particular if you are reading a small file. There is no obvious way for clients to detect the right setting without causing file I/O, so it might be a good idea to eventually extend NS_AsyncCopy to autodetect the “right” chunk size for simple cases.

Bottom line: NS_AsyncCopy is not perfect but it is quite good and it does not cause main thread I/O.


NS_AsyncCopy will, of course, not remove main thread I/O that takes place externally. If you open a stream from the main thread, this can cause main thread I/O. In particular, file streams should really be opened with flag DEFER_OPEN flag. Other streams, such as nsIJARInputStream do not support any form of deferred opening (bug 928329), and will cause main thread I/O when they are opened.

While NS_AsyncCopy does only off main thread I/O, using a Safe File Output Stream will cause a Flush. The Flush operation is very expensive for the whole system, even when executed off the main thread. For this reason, Safe File Output Stream is generally not the right choice of output stream (bug 928321).

Finally, if you only want to copy a file, prefer OS.File.copy (if you can call JS). This function is simpler, entirely off main thread, and supports OS-specific accelerations.

In JavaScript

What NetUtil.asyncCopy does

NetUtil.asyncCopy is a utility method that lets JS clients call NS_AsyncCopy. Theoretically, it should have the same behavior. However, some oddities make its performance lower.

As NS_AsyncCopy requires one of its streams to be buffered, NetUtil.asyncCopy calls nsIIOUtil::inputStreamIsBuffered and nsIIOUtil::outputStreamIsBuffered. These methods detect whether a stream is buffered by attempting to perform buffered I/O. Whenever they succeed, this causes main thread I/O (bug 928340).


Generally speaking, NetUtil.asyncCopy has the same limitations as NS_AsyncCopy. In particular, in any case in which you can replace NetUtil.asyncCopy with OS.File.copy, you should pick the latter, which is both simpler and faster.

Also, NetUtil.asyncCopy cannot read directly from a Zip file (bug 927366).

Finally, NetUtil.asyncCopy does not fit the “modern” way of writing asynchronous code on the Mozilla Platform (bug 922298).

Helping out

We need to fix a few bugs to improve the performance of asynchronous copy. If you wish to help, please do not hesitate to pick any of the bugs listed above and get in touch with me.

Trapping uncaught asynchronous errors

October 14, 2013 § 2 Comments

While the official specifications of DOM Promise is still being worked on, Mozilla has been using Promise internally for several years already. This API is available to the platform front-end and to add-ons. In the past few weeks, Promise.jsm (our implementation of Promise) and Task.jsm (our implementation of Beautiful Concurrency in JavaScript, built on top of Promise) have been updated with a few new features that should make everybody’s life much easier.

Reporting errors

The #1 issue developers encounter with the use of Promise and Task is error-handling. In non-Promise code, if a piece of code throws an error, by default, that error will eventually be reported by window.onerror or any of the other error-handling mechanisms.

function fail() {
  let x;
  return x.toString();

fail(); // Displays somewhere: "TypeError: x is undefined"

By opposition, with Promise and/or Task, if a piece of code throws an error or rejects, by default, this error will be completely ignored:

Task.spawn(function*() {
  fail(); // Error is ignored


Task.spawn(function*() {
  yield fail(); // Error is ignored


somePromise.then(function onSuccess() {
  fail(); // Error is ignored


somePromise.then(function onSuccess() {
  return fail(); // Error is ignored

Debugging the error requires careful instrumentation of the code, which is error-prone, time-consuming, often not compositional and generally ugly to maintain:

Task.spawn(function*() {
  try {
  } catch (ex) {
    throw ex;
    // The error report is incomplete, re-throwing loses stack information
    // and can cause double-reporting

The main reason we errors end up dropped silently is that it is difficult to find out whether an error is eventually caught by an error-handler – recall that, with Promise and Task, error handlers can be registered long after the error has been triggered.

Well, after long debates, we eventually found solutions to fix the issue :)

Simple case: Reporting programming errors

Our first heuristic is that programming errors are, well, programming errors, and that programmers are bound to be looking for them.


Task.spawn(function*() {
  fail(); // Error is not invisible anymore

will now cause the following error message

A coding exception was thrown and uncaught in a Task.

Full message: TypeError: x is undefined
Full stack: fail@Scratchpad/2:23

The message appears on stderr (if you have launched Firefox from the command-line) and in the system logs, so it won’t disrupt your daily routine, but if you are running tests or debugging your code, you should see it.

A similar error message will be printed out if the error is thrown from a raw Promise, without use of Task.

These error messages are limited to programming errors and appear only if the errors are thrown, not passed as rejections.

General case: Reporting uncaught errors

Now, we have just landed a more general support for displaying uncaught errors.

Uncaught thrown error

Task.spawn(function* () {
  throw new Error("BOOM!"); // This will be displayed

Uncaught rejection

Task.spawn(function* () {
  yield Promise.reject("BOOM!"); // This will also be displayed

Uncaught and clearly incorrect rejection

Task.spawn(function* () {
  // Oops, forgot to yield.
  // Nevermind, this will be displayed, too

These will be displayed in the browser console as follows:

A promise chain failed to handle a rejection: on Mon Oct 14 2013 16:50:15 GMT+0200 (CEST), Error: BOOM! at

These error messages appear for every uncaught error or rejection, once it is certain that the error/rejection cannot be caught anymore. If you are curious about the implementation, just know that it hooks into the garbage-collector to be informed once the error/promise cannot be caught anymore.

This should prove very helpful when debugging Promise- or Task-related errors. Have fun :)

Support for ES6 generators

You may have noticed that the above examples use function*() instead of function(). Be sure to thank Brandon Benvie who has recently updated Task.jsm to be compatible with ES6 generators :)

Making Firefox Feel as Fast as its Benchmarks – Part 2 – Towards multi-process

October 9, 2013 § 2 Comments

As we saw in the first post of this series, our browser behaves as follows:

function browser() {
  while (true) {
    handleEvents();  // Let's make this faster

As we discussed, the key to making the browser smooth is to make handleEvents() do less. One way of doing this is to go multi-process.

Going multi-process

Chrome is multi-process. Internet Explorer 4 was multi-process and so is Internet Explorer 8+ do (don’t ask me where IE 5, 6, 7 went). Well, Firefox OS is multi-process, too and Firefox for Android used to be multi-process until we canceled this approach due to Android-specific issues. For the moment, Firefox Desktop is only slightly multi-process, although we are heading further in this direction with project electrolysis (e10s, for short).

In a classical browser (i.e. not FirefoxOS, not the Firefox Web Runtime), going multi-process means running the user interface and system-style code in one process (the “parent process”) and running code specific to the web or to a single tab in another process (the “child process”). Whether all tabs share a process or each tab is a separate process, or even each iframe is a process, is an additional design choice that, I believe, is still open to discussion in Firefox. In FirefoxOS and in the Firefox Web Runtime (part of Firefox Desktop and Firefox for Android), going multi-process generally means one process per (web) application.

Since code is separated between processes, each handleEvents() has less to do and will therefore, at least in theory, execute faster. Additionally, this is better for security, insofar as a compromised web-specific process affords an attacker less control than compromising the full process. Finally, this gives the possibility to crash a child process if necessary, without having to crash the whole browser.

Coding for multi-process

In the Firefox web browser, the multi-process architecture is called e10s and looks as follows:

function parent() {
  while (true) {
    handleEvents();  // Some of your code here
    updateDisplay(); // Just the ui
function child() {
  while (true) {
    handleEvents();  // Some of your code here
    updateDisplay(); // Just some web

parent() ||| child()

The parent process and the child process are not totally independent. Very often, they need to communicate. For instance, when the user browses in a tab, the parent needs to change the history menu displayed by the parent process. Similarly, every few seconds, Firefox saves its state to provide quick recovery in case of crash – the parent asks each tab for its information and, once all replies have arrived, gathers them into one data structure and saves them all.

For this purpose, parent and the child can send messages to each other through the Message Manager. A Message Manager can let a child process communicate with a single parent process and can let a parent process communicate with one or more children processes:

// Sender-side
messageManager.sendAsyncMessage("MessageTopic", {data})

// Receiver-side
messageManager.addMesageListener("MessageTopic", this);
// ...
receiveMessage: function(message) {
  switch ( {
  case "MessageTopic":
    // do something with
    // ...

Additionally, code executed in the parent process can inject code in the child process using the Message Manager, as follows:

messageManager.loadFrameScript("resource://path/to/script.js", true);

Once injected, the code behaves as any (privileged) code in the child process.

As you may see, communications are purely asynchronous – we do not wish the Message Manager to stop a process and wait until another process is done with it tasks, as this would totally defeat the purpose of multi-processing. There is an exception, called the Cross Process Object Wrapper, which I am not going to cover, as this mechanism is meant to be used only during a transition phase.


It is tempting to see multi-process architecture as a silver bullet that semi-magically makes Firefox (or its competitors) fast and smooth. There are, however, quite clear limitations to the model.

Firstly, going multi-process has a cost. As demonstrated by Chrome, each process consumes lots of memory. Each process needs to load its libraries, its JavaScript VM, each script must be JIT-ed for each process, each process needs its communication channgel towards the GPU etc. Optimizing this is possible, as demonstrated by FirefoxOS (which runs nicely with 256 Mb) but is a challenge.

Similarly, starting a multi-process browser can be much slower than starting a single-process browser. Between the instant the user launches the browser and the instant it becomes actually usable, many things need to happen: launching the parent process, which in turn launches the children processes, setting up the communication channels, JIT compiling all the scripts that need compilation, etc. The same cost appears when shutting down the processes.

Also, using several processes brings about a risk of contention on resources. Two processes may need to access the disk cache at the same time, or the cookies, or the session storage, or the GPU or the audio device. All of this needs to be managed carefully and can, in certain cases, slow down considerably both processes.

Also, some APIs are synchronous by specifications. If, for some reason, a child process needs to access the DOM of another child process – as may happen in the case of iframes – both child processes need to become synchronous. During the operation, they both behave as a single process, with just extremely slower DOM operations.

And finally, going multi-process will of course not make a tab magically responsive if this tab itself is the source of the slowdown – in other words, multi-process it not very useful for games.

Refactoring for multi-process

Many APIs, both in Firefox itself and in add-ons, are not e10s-compliant yet. The task of refactoring Firefox APIs into something e10s-compliant is in progress and can be followed here. Let’s see what needs to be done to refactor an API for multi-process.

Firstly, this does not apply to all APIs. APIs that access web content for non-web content need to be converted to e10s-style – an example is Page Info, which needs to access web content (the list of links and images from that page) for the purpose of non-web content (the Page Info button and dialog). As multi-process communications is asynchronous, this means that such APIs must be asynchronous already or must be made asynchronous if they are not, and that code that calls these APIs needs to be made asynchronous if it is not asynchronous already, which in itself is already a major task. We will cover making things asynchronous in another entry of this series.

Once we have decided to make an asynchronous API e10s-compliant, the following step is to determine which part of the implementation needs to reside in a child process and which part in the parent process. Typically, anything that touches the web content must reside in the child process. As a rule of thumb, we generally consider that the parent process is more performance-critical than children-processes, so if you have code that could reside either in the child process or in the parent process, and if placing that code in the child process will not cause duplication of work or very expensive communications, it is a good idea to move it to the child process. This is, of course, a rule of thumb, and nothing replaces testing and benchmarking.

The next step is to define a communication protocol. Messages exchanged between the parent process and children processes all need a name. If are working on feature Foo, by conventions, the name of your messages should start with “Foo:”. Recall that message sending is asynchronous, so if you need a message to receive an answer, you will need two messages: one for sending the request (“Foo:GetState”) and one for replying once the operation is complete (“Foo:State”). Messages can carry arbitrary data in the form of a JavaScript structure (i.e. any object that can be converted to/from JSON without loss). If necessary, these structures may be used to attach unique identifiers to messages, so as to easily match a reply to its request – this feature is not built into the message manager but may easily be implemented on top of it. Also, do not forget to take into account communication timeouts – recall that a process may fail to reply because it has crashed or been killed for any reason.

The last step is to actually write the code. Code executed by the parent process typically goes into some .js file loaded from XUL (e.g. browser.js) or a .jsm module, as usual. Code executed by a child process goes into its own file, typically a .js, and must be injected into the child process during initialization by using window.messageManager.loadFrameScript (to inject in all children process) or browser.messageManager.loadFrameScript (to inject in a specific child process).

That’s it! In a future blog entry, I will write more about common patterns for writing or refactoring asynchronous code, which comes in very handy for code that uses your new API.

Contributing to e10s

The ongoing e10s refactoring of Firefox is a considerable task. To make it happen, the best way is to contribute to coding or testing.

What’s next?

In the next blog entry, I will demonstrate how to make front-end and add-on code multi-threaded.

Making Firefox Feel as Fast as its Benchmarks – Part 1 – Towards Async

October 8, 2013 § 17 Comments

Most of us have seen the numbers: according to benchmarks, Firefox is faster than Chrome. Sadly, that is sometimes hard to believe, as our eyes often indicate the opposite. The good news is that Firefox really is very, very fast. However, this is misleading: what our eyes notice is not speed (an indication of how much time an operation takes to complete) but smoothness (an indication of how often we have refreshed the screen in the meantime). And on that domain, on many pages, we are still lagging behind Chrome.

In this series of blog entries about the subject,  I plan to introduce the issues and discuss the solutions that are envisioned.

How browsers work

Let’s start by a short refresher about how browsers work. All current browser engines (except Servo) share the same overall behavior:

function browser() {
  while (true) {
    handleEvents(); // <-- Your code goes here

Function handleEvents() handles all the system-specific event handling (“oh, it looks like the mouse has moved”, “some time has passed”, “the http request is complete”) and dispatches this to all higher-level code designed to provide the user experience (“call onmousemove“, “add something to the history menu”, “capture thumbnail”, “call XMLHttpRequest callback”, …). Whether you are writing back-end code, front-end code, an add-on or even web-side JavaScript, unless you are part of the Layout team, chances are that your C++ or JavaScript code is executed somewhere in handleEvents() and your CSS is executed in updateDisplay().

To ensure perfect smoothness, updateDisplay() must be called at least 30 times per second, preferably 60. This means that the whole loop should be executed in roughly 15ms, which is pretty short. This is a problem that hits all browsers and we have a number of techniques to fit our code in this limit. As I am not a member of the Layout team, I will concentrate on the code that runs in handleEvents().

So, we need to make handleEvents() faster. Can we optimize it? Certainly we can. However, recall that benchmarks suggest that Firefox is already the fastest browser around, and this is not sufficient. Perhaps we can improve that speed by a few percents, but that is not the best course of action.

We need to go the other way: handleEvents() must do less things.

In the following parts of this series, I will explore several options: going multi-process, multi-thread, or interruptible. If I have time, I will also discuss Servo, Mozilla’s next-generation rendering engine.

Async & Responsive: What’s next?

October 1, 2013 § 2 Comments

For the past few months/years adding APIs to the Mozilla Platform to simplify everybody’s task of writing asynchronous or, even better, off-main thread code.

From the top of my head, we have added and gradually improved:

  • support for async programming with Promise, including Promise.jsm, Task.jsm, better error reporting …
  • support for off main thread I/O with OS.File, Sqlite.jsm, mozIStorageAsyncConnection, nsIBackgroundFileSaver, async transactions for places, …
  • support for making things safer with AsyncShutdown.jsm …
  • support for testing async code, with  add_task for xpcshell and mochitest-browser …
  • support for clear off main thread code with the chrome worker module loader, extensions to js-ctypes …

Do you have wishes for Q4 or beyond? [De]compressing files on chrome workers? Accessing sqlite from chrome workers? More tooling for Promise? New preference APIs?

Please drop us a line, either here or in the relevant Bugzilla component.

Additionally, I will attend the Mozilla Summit in Brussels where I will host an Open Session on Async development in the Mozilla platform. Don’t hesitate to come and have a chat.

Where Am I?

You are currently browsing the Firefox category at Il y a du thé renversé au bord de la table.


Get every new post delivered to your Inbox.

Join 29 other followers