Asynchronous database connections in the Mozilla Platform

One of the core components of the Mozilla Platform is mozStorage, our low-level database, based on sqlite3. mozStorage is used just about everywhere in our codebase, to power indexedDB, localStorage, but also site permissions, cookies, XUL templates, the download manager (*), forms, bookmarks, the add-ons manager (*), Firefox Health Report, the search service (*), etc. – not to mention numerous add-ons.

(*) Some components are currently moving away from mozStorage for performance and footprint reasons as they do not need the safety guarantees provided by mozStorage.

A long time ago, mozStorage and its users were completely synchronous and main-thread based. Needless to say, this eventually proved to be a design that doesn’t scale very well. So, we set out on a quest to make mozStorage off main thread-friendly and to move all these uses off the main thread.

These days, whether you are developing add-ons or contributing to the Mozilla codebase, everything you need to access storage off the main thread are readily available to you. Let me introduce the two recommended flavors.

Note: This blog entry does not cover using database from *web applications* but from the *Mozilla Platform*. From web applications, you should use indexedDB.

In JavaScript, Sqlite.jsm (Firefox 20+)

JavaScript users can access mozStorage directly but it is generally both much more convenient and much safer to use module Sqlite.jsm. This module has been introduce in Firefox 20 and upgraded in Firefox 25 with a number of performance/concurrency improvements, without change in client code.

Firstly, we will import Sqlite.jsm and, to ensure that our code is easy to read, Task.jsm:

Components.utils.import("resource://gre/modules/Sqlite.jsm");
Components.utils.import("resource://gre/modules/Task.jsm");

We may now open (and later, close) the database:

Task.spawn(function() {
  let db;
  try {
    // Open a database
    db = yield Sqlite.openConnection({ path: "myDatabase.sqlite" });
    // ...
    // Work with the database
    // ...
    //
  } catch (ex) {
    // Handle or report synchronous and asynchronous errors in any way you see fit
  } finally {
    // Don't forget to close the database or you will have bad surprises during shutdown
    if (db) {
      yield db.close();
    }
  }
});

Recall that we may use a synchronous syntax to perform asynchronous operations thanks to Task.spawn. Starting with Firefox 25, however, openConnection is executed off the main thread (by the time Firefox 25 graduates to beta, close should also be executed off the main thread, but this particular feature has not landed yet).

While the database is open, we may execute any SQL request, as follows:

// Work with the database.
// mozStorage uses the same dialect of SQL as sqlite3.

// Create a table
yield db.execute("CREATE TABLE table_name (column_name INTEGER)");

// Select content from another table
// Note parameter :domain. We give its value in a JavaScript object
yield db.execute("SELECT * FROM cookies_table WHERE domain = :domain", {
  domain: "example.com"
});

// If the statement is executed often, we will want to cache it
// for performance, as follows:
yield db.executeCached("SELECT * FROM cookies_table WHERE domain = :domain", {
  domain: "example.com"
});

All execute and executeCached operations are executed off the main thread.

Of course, Sqlite.jsm also supports transactions and schema management, but this is beyond the scope of this blog entry. For more details, see the full documentation of Sqlite.jsm.

In C++, mozIStorageAsyncConnection (Firefox 25+)

Firefox 25 introduces a new C++ API, mozIStorageAsyncConnection.

Opening a mozIStorageAsyncConnection differs slightly from opening a mozIStorageConnection.

First, let us import the mozStorage service:

nsCOMPtr<mozIStorageService> dbService =
  do_GetService(MOZ_STORAGE_SERVICE_CONTRACTID, &rv);
if (!NS_SUCCEEDED(rv)) {
  // FIXME: Handle error;
}

We may now open the database asynchronously:

// Initialize file argument
nsCOMPtr<nsIFile> dbFile;
rv = NS_GetSpecialDirectory(NS_APP_USER_PROFILE_50_DIR,
  getter_AddRefs(dbFile));
if (!NS_SUCCEEDED(rv)) {
  // FIXME: Handle error;
}
rv = dbFile->Append(NS_LITERAL_STRING("myDatabase.sqlite"));
if (!NS_SUCCEEDED(rv)) {
  // FIXME: Handle error;
}
// Open database asynchronously
nsCOMPtr<mozIStorageCompletionCallback> onOpened = new DBOpened();
nsCOMPtr<mozIStorageConnection> dbConn;
rv = dbService->AsyncOpenDatabase(dbFile, nullptr, getter_AddRefs(dbConn), onOpened);
if (!NS_SUCCEEDED(rv)) {
  // FIXME: Handle error;
}

As the database is opened asynchronously, we need to provide a callback to handle the
result:

class DBOpened: public mozIStorageCompletionCallback {
  NS_DECL_ISUPPORTS

  NS_IMETHOD Complete(nsresult aStatus, nsISupports* aDB)
  {
    if (!NS_SUCCEEDED(aStatus)) {
      // FIXME: Handle error
      return NS_OK;
    }

    nsCOMPtr<mozIStorageAsyncConnection> db = do_QueryInterface(aDB);
    MOZ_ASSERT(db, "Could not QI db");

    // ...
    // Work with the database
    // ...
    //
  }
};

As above, this database supports the dialect of SQL understood by sqlite3.

// Create the statement
nsCOMPtr<mozIStorageAsyncStatement> statement;
nsresult rv = db->CreateAsyncStatement("SELECT * FROM cookies_table WHERE domain = :domain",
  getter_AddRefs(statement));
if (!NS_SUCCEEDED(rv)) {
  // FIXME: handle error
}

// Now bind argument :domain
rv = statement->BindUTF8StringByName(NS_LITERAL_CSTRING("domain"),
  "example.com");
if (!NS_SUCCEEDED(rv)) {
  // FIXME: handle error
}

// Execute the statement
nsCOMPtr<mozIStorageStatementCallback> onComplete = new OnComplete();
rv = statement->ExecuteAsync(onComplete);
if (!NS_SUCCEEDED(rv)) {
  // FIXME: handle error
}

Before closing a database, you need to ensure that you hold no reference to its statements,
so as to ensure that the statements are properly finalized:

statement = nullptr;
rv = db->AsyncClose(nullptr); // Optionally, add a callback
if (!NS_SUCCEEDED(rv)) {
  // FIXME: handle error
}

Of course, mozIStorageAsyncConnection supports transactions, but that goes beyond the scope of this blog entry.

When should I use mozStorage?

If you develop code for the Mozilla Platform (either contributions to the codebase or add-ons), you need to be very careful about responsiveness. A very good way to break responsiveness is to use mozStorage on the main thread. If you need to read/write complex data and if you need a high level of safety, mozStorage is the right tool for the job and Sqlite.jsm and mozIStorageAsyncConnection are the right ways to use this tool. For simple uses of mozStorage, you may wish to use the higher-level indexedDB.

If you only need to read/write simple data and if you do not need extreme safety, you should rather consider using OS.File to read/write all your data as json.

Whatever you do, you should avoid using the preference system for anything other than preferences, and in particular for anything large or that changes often, as writing preferences is an expensive operation that is currently executed on the main thread. Moreover, adding large data to the preferences will slow down both the startup of Firefox (loading preferences become slower) and every single preferences write.