Shutting down things asynchronously

This blog entry is part of the Making Firefox Feel As Fast As Its Benchmarks series. The fourth entry of the series was growing much too long for a single blog post, so I have decided to cut it into bite-size entries.

A long time ago, Firefox was completely synchronous. One operation started, then finished, and then we proceeded to the next operation. However, this model didn’t scale up to today’s needs in terms of performance and performance perception, so we set out to rewrite the code and make it asynchronous wherever it matters. These days, many things in Firefox are asynchronous. Many services get started concurrently during startup or afterwards. Most disk writes are entrusted to an IO thread that performs and finishes them in the background, without having to stop the rest of Firefox.

Needless to say, this raises all sorts of interesting issues. For instance: « how do I make sure that Firefox will not quit before it has finished writing my files? » In this blog entry, I will discuss this issue and, more generally, the AsyncShutdown mechanism, designed to implement shutdown dependencies for asynchronous services.

Running example

Consider a service that needs to:

1. write something to disk during startup;
2. write something to disk during runtime;
3. write something to disk during shutdown.

Since writes are delegated to a background thread, we need a way to ensure that any write is complete before Firefox shuts down – which might be before startup is complete, in a few cases. More precisely, we need a way to ensure that any write is complete before the background thread is terminated. Otherwise, we could lose any of the data of a., b., c., which would be quite annoying.

A. Maybe we can pause threads (warning: don’t)

A first solution would be to “simply” synchronize threads: during shutdown, stop the main thread using a Mutex until the IO thread has completed its work. While this would be possible, this technique has a few big drawbacks:

• it is heavy handed and prone to stopping many things that we don’t want to stop (e.g. progress bars, JavaScript garbage-collection and finalization, add-on code);
• while the main thread is frozen, the IO thread cannot interact with the main thread, which is often necessary if it needs additional information or complex network interactions;
• it introduces the possibility of deadlocks, which are never fun to deal with, especially when they are in the code of add-ons.

B. Maybe we can pause notifications (warning: ugly)

The second solution requires a little more knowledge of the workings of the Firefox codebase. Many steps during the execution (including shutdown) are represented by synchronous notifications. To prevent shutdown from proceeding, it is sufficient to block the execution of a notification, by having an observer that returns only once the IO is complete:

Services.obs.addObserver(function observe() {
  writeStuffThatShouldBeWrittenDuringShutdown();
  while (!everythingHasBeenWritten()) {
    Services.tm.processNextEvent();
  }
}, "some-stage-of-shutdown");

This snippet introduces an observer executed at some point during shutdown notification “some-stage-of-shutdown”. Once execution of the observer starts, it returns only once condition everythingHasBeenWritten() is satisfied. Until then, the process can handle system events, proceed with garbage-collection, refresh the user interface including progress bars, etc.

This mechanism is called “spinning the event loop.” It is used in a number of places throughout Firefox, not only during shutdown, and it works. It has, however, considerable drawbacks:

• spinning the event loop prevents the stack from being emptied, which prevents some data from being garbage-collected;
• spinning the event loop has sometimes “interesting” effects on the order of execution of events;
• it introduces the possibility of livelocks, as other observers of “some-stage-of-shutdown” are blocked until everythingHasBeenWritten() is satisfied, and livelocks are even worse than deadlocks.

C. Maybe we can introduce explicit dependencies

The third solution is the AsyncShutdown module. This module simply lets us register “blockers”, instructing shutdown to not proceed past some phase until some condition is complete.

AsyncShutdown.profileBeforeChange.addBlocker(
  // The name of the blocker
  "My Service: Need to write down all my data to disk before shutdown",

  // Code executed during the phase
  function trigger() {
    let promise = ...; // Some promise satisfied once the write is complete
    return promise;
  },

  // A status, used for debugging shutdown locks
  function status() {
    if (...) {
      return "I'm doing something";
    } else {
      return "I'm doing something else entirely";
    }
  }
);

In this snippet, the promise returned by function trigger blocks phase profileBeforeChange until this promise is resolved. Note that function trigger can return any promise – including one that may already be satisfied. This makes it very simple to handle both operations that were started during startup (case a. of our running example – the promise is generally satisfied already), during runtime (case b. – always store the promise when we write, return the latest) or during shutdown (case c. – actually start the write in trigger, return the promise).

AsyncShutdown has a few interesting properties:

• while the current implementation is based on spinning the event loop, all the blockers for each phase share the same event loop, which considerably reduces the unwanted side-effects in terms of garbage-collection, order of execution and debuggability;
• AsyncShutdown can detect (live/dead)locks and automatically crashes Firefox to ensure that the process doesn’t remain frozen during shutdown, consuming battery and preventing the user from stopping the operating system or restarting Firefox;
• additionally, AsyncShutdown produces crash reports which contain the information on the blockers participating in a (live/dead)lock and their status, to simplify bugfixing such locks.

Status

AsyncShutdown has been in use in both Firefox Desktop, Firefox Mobile and Firefox OS for a few months, although not all shutdown dependencies have been ported to AsyncShutdown. At the moment, this module is available only to JavaScript, although there are plans to eventually port it to C++ (if need AsyncShutdown for a C++ client, please get in touch).

Porting all existing spinning-the-event-loop clients during shutdown is a long task and will certainly take quite some time. However, if you are writing services that need to write data at any point and to be sure that data is fully written before shutdown, you should definitely use AsyncShutdown.