Designing the Firefox Performance Monitor (2): Monitoring Add-ons and Webpages

In part 1, we discussed the design of time measurement within the Firefox Performance Monitor. Despite the intuition, the Performance Monitor had neither the same set of objectives as the Gecko Profiler, nor the same set of constraints, and we ended up picking a design that was not a sampling profiler. In particular, instead of capturing performance data on stacks, the Monitor captures performance data on Groups, a notion that we have not discussed yet. In this part, we will focus on bridging the gap between our low-level instrumentation and actual add-ons and webpages, as may be seen by the user.

Detecting slow add-ons

When it is at its best, Firefox is fast. Really, really fast. When things start slowing down, though, using Firefox is much less fun. So, one of main objectives of the developers of Firefox is making sure that Firefox is and remains as smooth and responsive as humanly possible. There is, however, one thing that can slow down Firefox, and that remains out of the control of the developers: add-ons. Good add-ons are extraordinary, but small coding errors – or sometimes necessary hacks – can quickly drive the performance of Firefox into the ground.

So, how can an add-on developer (or add-on reviewer) find out whether her add-on is fast? Sadly, not much. Testing certainly helps, and the Profiler is invaluable to help pinpoint a slowdown once it has been noticed, but what about the performance of add-ons in everyday use? What about the experience of users?

To solve this issue, we decided to work on a set of tools to help add-on developers and reviewers find out the performance of their add-ons. Oh, and also to let users find out quickly if an add-on is slowing down their everyday experience.

about:performance

On recent Nightly builds of Firefox, you may now open about:performance to get an overview of the performance cost of add-ons and webpages :

The main resources we monitor are :

• jank, which measures how much the add-on impacts the responsiveness of Firefox. For 60fps performance, jank should always remain ≤ 4. If an add-on regularly causes jank to increase past 6, you should be worried.
• CPOW aka blocking cross-process communications, which measures how much the add-on is causing Firefox to freeze waiting for a process to respond. Anything above 0 is bad.

Note that the design of this page is far from stable. I realise it’s not very user-friendly at the moment, so don’t hesitate to file bugs to help us improve it. Also note that, when running with e10s, the page doesn’t display all the useful information. We are working on it.

add-on Telemetry

Add-on developers and reviewers can now find information on the performance of their add-ons on a dedicated dashboard.

These are real-world performance data, as extracted from user’s computers. The two histograms available for the time being are:

• MISBEHAVING_ADDONS_JANK_LEVEL, which measures the jank, as detailed above;
• MISBEHAVING_ADDONS_CPOW_TIME_MS, which measure the amount of time spent in CPOW, as detailed above.

If you are an add-on developer, you should monitor regularly the performance of your add-on on this page. If you notice suspicious values, you should try and find out what causes these performance issues. Don’t hesitate and reach out to us, we will try and help you.

Slow add-on Notification

Add-on developers and reviewers, as well as end-users, are now informed when an add-on causes either jank or CPOW performance issues:

Note that this feature is not ready to ride the trains, and we do not have a specific idea of when it will be made available for users of Aurora/DeveloperEdition. This is partly because the UX is not good enough yet, partly because the thresholds will certainly change, and partly because we want to give add-on developers time to fix any issue before the users see a dialog that suggest that an add-on should be uninstalled.

Performance Stats API

By the way, we have an API for accessing performance stats. Very imaginatively, it’s called PerformanceStats.jsm [link]. While this API will still change during the coming weeks you can start playing with it if you are interested. Some add-ons may be able to throttle their performance use based on this data. Also, I hope that, in time, someone will be able to write a version of about:performance much nicer than mine 🙂

Challenges and work ahead

For the moment, we are in the process of stabilizing the API, its implementation and its performance. In parallel, we are working on making the UX of about:performance more useful. Once both are done, we are going to proceed with adding more measurements, making the code more e10s-friendly and measuring the performance of webpages.

If you are an add-on developer and if you feel that your add-on is tagged as slow by error, or if you have great ideas on how to make this data useful, feel free to ping me, preferably on IRC. You can find me on irc.mozilla.org, channel #developers, where I am Yoric.

RFC: We deserve better than runtime warnings

Consider the following scenario:

1. Module A prints warnings when it’s used incorrectly;
2. Module B uses module A correctly;
3. Some future refactoring of module B starts using module A incorrectly, hence displaying the warnings;
4. Nobody realises for months, because we have too many warnings;
5. Eventually, something breaks.

How often has this happened to everyone of us?

This scenario has many variants (e.g. module A changed and nobody realized that module B is now in a situation it misuses module A), but they all boil down to the same thing: runtime warnings are designed to be lost, not fixed. To make things worse, many of our warnings are not actionable, simply because we have no way of knowing where they come from – I’m looking at you, Cu.reportError.

So how do we fix this?

We would certainly save considerable amounts of time if warnings caused immediate assertion failures, or alternatively test failures (i.e. fail, but only when running the unit tests). Unfortunately, we can do neither, as we have a number of tests that trigger the warnings either

• by design (e.g. to check that we can recover from such misuses of A, or because we still need a now-considered-incorrect use of an API to keep working until we have ported all the clients to the better API);
• or for historical reasons (e.g. the now incorrect use of A used to be correct, but we haven’t fixed all tests that depend on it yet).

However, I believe that causing test failures is still the solution. We just need a mechanism that supports a form of whitelisting to cope with the aforementioned cases.

Introducing RuntimeAssert

RuntimeAssert is an experiment at provoding a standard mechanism to replace warnings. I have a prototype implemented as part of bug 1080457. Feedback would be appreciated.

The key features are the following:

• when a test suite is running, a call to `RuntimeAssert` causes the test suite to fail;
• when a test suite is running, a call to `RuntimeAssert` contains at least the filename/line number of where it was triggered, preferably a stack wherever available;
• individual tests can whitelist families of calls to `RuntimeAssert` and mark them either as expected;
• individual tests can whitelist families of calls to `RuntimeAssert` and mark them as pending fix;
• when a test suite is not running, a call to `RuntimeAssert` does nothing costly (it may default to PR_LOG or Cu.reportError).

Possible API:

• in JS, we trigger a test failure by calling RuntimeAssert.fail(keyword, string or Error) from production code;
• in C++, we likewise trigger a test failure by calling MOZ_RUNTIME_ASSERT(keyword, string);
• in the testsuite, we may whitelist errors by calling Assert.whitelist.expected(keyword, regexp)  or Assert.whitelist.FIXME(keyword, regexp).

Examples:

//
// Module
//
let MyModule = {
  oldAPI: function(foo) {
    RuntimeAssert.fail(“Deprecation”, “Please use MyModule.newAPI instead of MyModule.oldAPI”);
    // ...
  },
  newAPI: function(foo) {
    // ...
  },
};

let MyModule2 = {
  api: function() {
    return somePromise().then(null, error => {
      RuntimeAssert.fail(“MyModule2.api”, error);
      // Rather than leaving this error uncaught, let’s make it actionable.
    });
  },

  api2: function(date) {
    if (typeof date == “number”) {
      RuntimeAssert.fail(“MyModule2.api2”, “Passing a number has been deprecated, please pass a Date”);
      date = new Date(date);
    }
    // ...
   }
}


//
// Whitelisting a RuntimeAssert in a test.
//

// This entire test is about MyModule.oldAPI, warnings are normal.
Assert.whitelist.expected(“Deprecation”, /Please use MyModule.newAPI/);

// We haven’t fixed all calls to MyModule2.api2, so they should still warn, but not cause an orange.
Assert.whitelist.FIXME(“MyModule2.api2”, /please pass a Date/);

Assert.whitelist.expected(“MyModule2.api”, /TypeError/, function() {
  // In this test, we will trigger a TypeError in MyModule2.api, that’s entirely expected.
  // Ignore such errors within the (async) scope of this function.
});

Applications

In the long-term, I believe that RuntimeAssert (or some other mechanism) should replace almost all our calls to Cu.reportError.

In the short-term, I plan to use this for reporting

• uncaught Promise rejections, which currently require a bit too much hacking for my tastes;
• errors in XPCOM.lazyModuleGetter & co;
• failures during AsyncShutdown;
• deprecation warnings as part of Deprecated.jsm.

The Future of Promise

If you are writing JavaScript in mozilla-central or in an add-on, or if you are writing WebIDL code, by now, you have probably made use of Promise. You may even have noticed that we now have several implementations of Promise in mozilla-central, and that things are moving fast, and sometimes breaking.

At the moment, we have two active implementations of Promise:

• Promise.jsm, as provided by the toolkit;

• DOM Promise, as provided by the DOM.

(as well as a little code using an older, long deprecated, implementation of Promise)

This is somewhat confusing, but the good news is that we are working hard at making it simpler and moving everything to DOM Promise.

General Overview

Many components of mozilla-central have been using Promise for several years, way before a standard was adopted, or even discussed. So we had to come up with our implementation(s) of Promise. These implementations were progressively folded into Promise.jsm, which is now used pervasively in mozilla-central and add-ons.

In parallel, Promise were specified, submitted for standardisation, implemented in Firefox, and finally standardised. This is the second implementation we call DOM Promise. This implementation is starting to be used in many places on the web.

Having two implementations of Promise with the same feature set doesn’t make sense. Fortunately, Promise.jsm was designed to match the API of Promise that we believed would be standardised, and was progressively refactored and extended to follow these developments, so both APIs are almost identical.

Our objective is to move entirely to DOM Promise. There are still a few things that need to happen before this is possible, but we are getting close. I hope that we can get there by the end of 2014.

Missing pieces

Debugging and testing

At the moment, Promise.jsm is much better than DOM Promise in two aspects:

• it is easier to inspect a promise from Promise.jsm for debugging purposes (not anymore, things have been moving fast while I was writing this blog entry);
  

• Promise.jsm integrates nicely in the test suite, to make sure that uncaught errors are reported and cause test failures.

In both topics, we are hard at work bringing DOM Promise to feature parity with Promise.jsm and then some (bug 989960, bug 1083361). Most of the patches are in the pipeline already.

API differences

• Promise.jsm offers an additional function Promise.defer, which didn’t make it to standardization.

This function may easily be written on top of DOM Promise, so this is not a hard blocker. We are going to add this function to a module `PromiseUtils.jsm`.

• Also, there is a slight bug in DOM Promise that gives it a slightly unexpected behavior in a few edge cases. This should not hit developers who use DOM Promise as expected, but this might surprise people who know the exact scheduling algorithm and expect it to be consistent between Promise.jsm and DOM Promise.
  

Oh, wait, that’s fixed already.

Wrapping it up

Once we have done all of this, we will be able to replace Promise.jsm with an empty shell that defers all implementations to DOM Promise. Eventually, we will deprecate and remove this module.

As a developer, what should I do?

For the moment, you should keep using Promise.jsm, because of the better testing/debugging support. However, please do not use Promise.defer. Rather, use PromiseUtils.defer, which is strictly equivalent but is not going away.

We will inform everyone once DOM Promise becomes the right choice for everything.

If your code doesn’t use Promise.defer, migrating to DOM Promise should be as simple as removing the line that imports Promise.jsm in your module.

Copying streams asynchronously

In the Mozilla Platform, I/O is largely about streams. Copying streams is a rather common activity, e.g. for the purpose of downloading files, decompressing archives, saving decoded images, etc. As usual, doing any I/O on the main thread is a very bad idea, so the recommended manner of copying streams is to use one of the asynchronous string copy APIs provided by the platform: NS_AsyncCopy (in C++) and NetUtil.asyncCopy (in JavaScript). I have recently audited both to ascertain whether they accidentally cause main thread I/O and here are the results of my investigations.

In C++

What NS_AsyncCopy does

NS_AsyncCopy is a well-designed (if a little complex) API. It copies the full contents of an input stream into an output stream, then closes both. NS_AsyncCopy can be called with both synchronous and asynchronous streams. By default, all operations take place off the main thread, which is exactly what is needed.

In particular, even when used with the dreaded Safe File Output Stream, NS_AsyncCopy will perform every piece of I/O out of the main thread.

The default setting of reading data by chunks of 4kb might not be appropriate to all data, as it may cause too much I/O, in particular if you are reading a small file. There is no obvious way for clients to detect the right setting without causing file I/O, so it might be a good idea to eventually extend NS_AsyncCopy to autodetect the “right” chunk size for simple cases.

Bottom line: NS_AsyncCopy is not perfect but it is quite good and it does not cause main thread I/O.

Limitations

NS_AsyncCopy will, of course, not remove main thread I/O that takes place externally. If you open a stream from the main thread, this can cause main thread I/O. In particular, file streams should really be opened with flag DEFER_OPEN flag. Other streams, such as nsIJARInputStream do not support any form of deferred opening (bug 928329), and will cause main thread I/O when they are opened.

While NS_AsyncCopy does only off main thread I/O, using a Safe File Output Stream will cause a Flush. The Flush operation is very expensive for the whole system, even when executed off the main thread. For this reason, Safe File Output Stream is generally not the right choice of output stream (bug 928321).

Finally, if you only want to copy a file, prefer OS.File.copy (if you can call JS). This function is simpler, entirely off main thread, and supports OS-specific accelerations.

In JavaScript

What NetUtil.asyncCopy does

NetUtil.asyncCopy is a utility method that lets JS clients call NS_AsyncCopy. Theoretically, it should have the same behavior. However, some oddities make its performance lower.

As NS_AsyncCopy requires one of its streams to be buffered, NetUtil.asyncCopy calls nsIIOUtil::inputStreamIsBuffered and nsIIOUtil::outputStreamIsBuffered. These methods detect whether a stream is buffered by attempting to perform buffered I/O. Whenever they succeed, this causes main thread I/O (bug 928340).

Limitations

Generally speaking, NetUtil.asyncCopy has the same limitations as NS_AsyncCopy. In particular, in any case in which you can replace NetUtil.asyncCopy with OS.File.copy, you should pick the latter, which is both simpler and faster.

Also, NetUtil.asyncCopy cannot read directly from a Zip file (bug 927366).

Finally, NetUtil.asyncCopy does not fit the “modern” way of writing asynchronous code on the Mozilla Platform (bug 922298).

Helping out

We need to fix a few bugs to improve the performance of asynchronous copy. If you wish to help, please do not hesitate to pick any of the bugs listed above and get in touch with me.

Asynchronous database connections in the Mozilla Platform

One of the core components of the Mozilla Platform is mozStorage, our low-level database, based on sqlite3. mozStorage is used just about everywhere in our codebase, to power indexedDB, localStorage, but also site permissions, cookies, XUL templates, the download manager (*), forms, bookmarks, the add-ons manager (*), Firefox Health Report, the search service (*), etc. – not to mention numerous add-ons.

(*) Some components are currently moving away from mozStorage for performance and footprint reasons as they do not need the safety guarantees provided by mozStorage.

A long time ago, mozStorage and its users were completely synchronous and main-thread based. Needless to say, this eventually proved to be a design that doesn’t scale very well. So, we set out on a quest to make mozStorage off main thread-friendly and to move all these uses off the main thread.

These days, whether you are developing add-ons or contributing to the Mozilla codebase, everything you need to access storage off the main thread are readily available to you. Let me introduce the two recommended flavors.

Note: This blog entry does not cover using database from *web applications* but from the *Mozilla Platform*. From web applications, you should use indexedDB.

Tales of Science-Fiction Bugs: The Thing That Killed Talos

Have you ever encountered one of these bugs? One in which every single line of your code is correct, in which every type-check passes, every single unit test succeeds, the specifications are fulfilled but somehow, for no reason that can be explained rationally, it just does not work? I call them Science-Fiction Bugs. I am sure that you have met some of them. For some reason, the Mozilla Performance Team seems to stumble upon such bugs rather often, perhaps because we spend so much time refactoring other team’s code long after the original authors have moved on to other features, and combining their code with undertested libraries and technologies. Truly, this is life on the Frontier.

Today, I would like to tell you the tale of one of these Science-Fiction Bugs: The Thing That Killed Talos.

Fighting the good fight for fun and credits, with Mozilla Education

Are you a student?

Do you want to fight the good fight, for the Future of the Web, and earn credits along the way?

Mozilla Education maintains a tracker of student project topics. Each project is followed by one (or more) mentor from the Mozilla Community.

Then what are you waiting for? Come and pick or join a project or get in touch to suggest new ideas!

Are you an educator?

The tracker is also open to you. Do not hesitate to pick projects for your students, send students to us or contact us with project ideas.

We offer/accept both Development-oriented, Research-oriented projects and not-CS-oriented-at-all projects.

Are you an open-source developer/community?

If things work out smoothly, we intend to progressively open this tracker to other (non-Mozilla) projects related to the future of the web. Stay tuned – or contact us!

Scoped resources for all

A small class hierarchy has been added to MFBT, the “Mozilla Framework Based on Templates” which contains some of the core classes of the Mozilla Platform. This hierarchy introduces general-purpose and specialized classes for scope-based resource management. When it applies, Scope-based resource management is both faster than reference-counting and closer to the semantics of the algorithm, so you should use it 🙂

The codebase of Mozilla is largely written in C++. While C++ does not offer any form of automatic memory management, the (sometimes scary) flexibility of the language has allowed numerous projects to customize the manner in which memory and other resources are managed, and Mozilla is no exception. Largely, the Mozilla C++ codebase uses reference-counting, to provide automatic memory management in most cases.

While reference-counting is quite imperfect, and while future versions of Mozilla might possibly use other forms of memory management, it is also a very useful tool for such a large codebase. However, in some cases, reference-counting is just too much. Indeed, in a number of simple cases, we prefer the simpler mechanism of scope-based resource management, that is both more predictable, faster and more resource-efficient – at the cost of not being able to scale up to the more complex cases for which reference-counting or even more powerful mechanisms become much more suited.

Scope-based resource management is designed to handle resources that should be cleaned-up as soon as you leave a given scope (typically, the function), regardless of how you leave it (by reaching the end, with a break, a return or an exception).

The following extract illustrates the use of scoped resource allocation:

// returns true in case of success, false in case of error
bool copy(const char* sourceName, const char* destName, size_t bufSize) {
   ScopedFD source(open(sourceName, O_RDONLY));
   if (source.get() == -1) return false;

   ScopedFD dest(open(destName, O_WRONLY|O_CREAT, 0600));
   if (dest.get() == -1) return false;
     // source is closed automatically

   ScopedDeleteArray buf(new char[bufSize]);
   if (buf.get() == NULL) return false;
     // source, dest are closed automatically

   while (true) {
     const int bytesRead = read(source.get(), buf.rwget(), bufSize);
     if (bytesRead == 0) break;
     if (bytesRead == -1) return false;
       // source, dest, buf are cleaned-up

     const int writePos = 0;
     while (writePos < bytesRead) {
       const int bytesWritten = write(dest.get(), buf.get(),
                                      bytesRead - writePos);
       if (bytesWritten == -1) return false ;
         // source, dest, buf are cleaned-up
       writePos += bytesWritten;
     }
   }

   return true;
      // source, dest, buf are cleaned-up
}

As you can see, the main point of these scope-based resource management classes is that they are cleaned up automatically both in case of success and in case of error. In some cases, we wish to clean up resources only in case of error, as follows:

// returns -1 in case of error, the destination file descriptor in case of success
int copy(const char* sourceName, const char* destName, size_t bufSize) {
   ScopedFD source(open(sourceName, O_RDONLY));
   if (source.get() == -1) return -1;

   ScopedFD dest(open(destName, O_WRONLY|O_CREAT, 0600));
   if (dest.get() == -1) return -1;
      // source is closed automatically

   ScopedDeleteArray buf(new char[bufSize]);
   if (buf.get() == NULL) return -1;
      // source, dest are closed automatically

   while (true) {
     const int bytesRead = read(source.get(), buf.rwget(), bufSize);
     if (bytesRead == 0) break;
     if (bytesRead == -1) return -1;
      // source, dest, buf are cleaned-up

     const int writePos = 0;
     while (writePos < bytesRead) {
       const int bytesWritten = write(dest.get(), buf.get(),
                                      bytesRead - writePos);
       if (bytesWritten == -1) return -1 ;
        // source, dest, buf are cleaned-up
       writePos += bytesWritten;
     }
   }

   return dest.forget();
   // source and buf are cleaned-up, not dest
}

While both examples could undoubtedly be implemented with reference-counting or without any form of automated resource management, this would either make the source code much more complex and harder to maintain (for purely manual resource management) or make the executable slower and less explicit in terms of ownership (for reference-counting). In other words, scoped-based resource management is the right choice for these algorithms.

Now, the Mozilla codebase has offered a few classes for scope-based resource management. Unfortunately, these classes were scattered throughout the code, some of them were specific to some compilers, and they were generally not designed to be reusable.

We have recently starting consolidating these classes into a simple and extensible hierarchy of classes. If you need them, you can find the root of this hierarchy, as well as the most commonly used classes, on mozilla-central, as part of the MFBT:

• ScopedFreePtr<T> is suited to deallocate C-style pointers allocated with malloc;
• ScopedDeletePtr<T> is suited to deallocate C++-style pointers allocated with new;
• ScopedDeleteArray<T> is suited to deallocate C++-style pointers allocated with new[];
• root class Scoped<Trait> and macro SCOPED_TEMPLATE are designed to make it extremely simple to define new classes to handle other cases.

For instance, class ScopedFD as used in the above examples to close Unix-style file descriptors, can be defined with the following few lines of code:

struct ScopedFDTrait
{
public:
  typedef int type;
  static type empty() { return -1; }
  static void release(type fd) {
    if (fd != -1) {
      close(fd);
    }
  }
};
SCOPED_TEMPLATE(ScopedFD, ScopedFDTrait);

So, well, if you need scoped-based resource management, you know where to find it!

I will blog shortly about the situation in JavaScript.

The OPA type system, part 1

edit Part 2 of this post was never written. I no longer work on Opa. For any question regarding Opa, please contact MLstate.

Since the initial announcement regarding OPA, we have received a number of questions regarding all the aspects of the language (including, suprisingly, a few demands for answers and documentation). Well, while we’re busy putting together documentation, benchmarks and FAQ, here’s a quick tour of one of the most fundamental pieces of the language: the type system.