Asynchronous file I/O for the Mozilla Platform

The Mozilla platform has recently been extended with a new JavaScript library for asynchronous, efficient, file I/O. With this library, developers of Firefox, Firefox OS and add-ons can easily write code that behave nicely with respect to the process and the operating system. Please use it, report bugs and contribute.

Off-main thread file I/O

Almost one year ago, Mozilla started Project Snappy. The objective of Project Snappy is to improve, wherever possible, the responsiveness of Firefox, the Mozilla Platform, and now, Firefox OS, based on performance data collected from volunteer users. Thanks to this real-world performance data, we have been able to identify a number of bottlenecks at all levels of Firefox. As it turns out, one of the main bottlenecks is main thread file I/O, i.e. reading from a file or writing to a file from the thread that also runs most of the code of Firefox and its add-ons.

Getting file information with OS.File

OS.File keeps gaining new features.

Today, let me show you OS.File.stat and OS.File.prototype.stat, two data structures used to get information on a file, such as its size, its creation date or its nature.

How to

There are two ways to get information on a file.

The first technique is to simply call OS.File.stat with the path of the file you wish to open:

// File sessionstore.js in the user’s profile directory
let path = OS.Path.join(OS.Constants.Path.profileDir, "sessionstore.js");
let stat = OS.File.stat(path)

This returns a OS.File.Info object containing all the interesting information on the file.

if (stat.isDir) {
  dump("This is a directory\n");
} else if (stat.isSymLink) {
  dump("This is a symbolic link\n");
}
dump("The file contains " + stat.size + "bytes\n”);
dump("The file was created at " + stat.creationDate + "\n");
dump("The file was last accessed at " + stat.lastAccessDate + "\n");
dump("The file was last modified at " + stat.lastModificationDate + "\n");

Additionally, under Unix, some security information is available:

if ("unixOwner" in OS.File.Info.prototype) {
  dump("The file belongs to user " + stat.unixOwner +
    " in group " + stat.unixGroup +
    " and has mode " + stat.unixMode);
}

That’s it.

The second technique will let you get information on a file that is already opened:

let file = OS.File.open(path);
let stat = file.stat();

The result is exactly the same. Of course, file.stat() is faster if you have already opened the file, while OS.File.stat(path) if faster than opening the file, calling file.stat() then closing it.

Exercise

Let’s put OS.File.stat and OS.File.DirectoryIterator to good use for getting the list of all files in a directory, ordered by last modification date.

function sortedEntries(path) {
  // Get the list of all files in directory
  let iterator = new OS.File.DirectoryIterator(path);
  let entries;
  try {
    entries = [entry for (entry in iterator)];
  } finally {
    iterator.close();
  }

  // If we are under Windows, we have all information in entries already
  // We can make this happen without any further I/O
  if ("winLastModificationDate" in OS.File.DirectoryIterator.prototype) {
    return entries.sort(function compare(x,y) {
      return x.winLastModificationDate - y.winLastModificationDate;
    }
  } else {
    // On other systems, we have to call stat before we can order
    let sortable = [{entry: entry, stat: OS.File.stat(entry.path)} for (entry in entries)];
    // Array comprehension is cool
    let sorted = sortable.sort(function compare(x, y)) {
      return x.stat.lastModificationDate - y.stat.lastModificationDate;
    }
    return [x.entry for (x in sorted)];
  }
}

Note that OS.File.DirectoryIterator does not return special files “.” and “..”.

For bonus points, let’s do the same, but only for non-directory files in the directory:

function nonDirectoryEntries(path) {
  // Get the list of all files in directory
  let iterator = new OS.File.DirectoryIterator(path);
  try {
    for (let entry in iterator) {
      if (!entry.isDir) {
        // Generators are cool, too
        yield entry;
      }
    }
  } finally {
    iterator.close();
  }
}

function sortedEntries(path) {
  // Get the list of all non-directory files in directory
  let entries = nonDirectoryEntries(path);
  if ("winLastModificationDate" in OS.File.DirectoryIterator.prototype) {
    // ... as above
  } else {
    // ... as above
  }
}

We could of course remove directories after sorting, but removing it initially saves both computation time (we sort through a shorter array) and I/O (under non-Windows platforms, we only need to call stat on a smaller set of files).

Homework

As a Programming Language guy, I see an opportunity to develop this API into a nice Domain Specific Language that would let developers formulate queries and would let the engine generate OS-optimized functions to execute these queries.

For instance:

OS.File.Query.SelectFromDir().
  where({isDir: false}).
  sortedBy({lastModificationDate: true})
  // returns the above function, including the optimizations

 

OS.File.Query.SelectFromDir().
  where({path: /.*\.tmp^/, isSymLink:false}).
  sortedBy({creationDate: true})

I do not have plans to implement anything such at the moment, but this sounds like a nice student project. If you are interested, do not hesitate to drop me a line.

That’s all folks.

In the next entries of this blog, I expect to introduce, in no particular order:

• path manipulation with OS.File;
• reading/writing with encodings in OS.File;
• off-main-thread async I/O for the main thread;

benchmarks.

Fun with Windows paths.

I am currently attempting to implement a JavaScript library to handle file system paths in a portable manner.

Right now, I am having lots of fun with Windows paths and I wanted to share a few tidbits.

Under Windows, a path name can look like:

1. “\\?\drivename:” followed by backslash-separated components.
   Such paths can be either relative or absolute.
   In such paths, “.”, “..” and “/” are regular file names.
2. “\\.\drivename:” followed by backslash-separated components.
   Such paths can be either relative or absolute.
   In such paths, “.”, “..” and “/” are special names.
3. “\\?\UNC\servername” followed by backslash-separated components.
   Such paths can only be absolute.
   In such paths, “.”, “..” and “/” are regular file names.
4. “\\servername” followed by slash- or backslash- components.
   Such paths can only be absolute.
   In such paths, “.”, “..” and “/” are special names.
5. “drivename:” followed by slash- or backslash- components.
   Such paths can be either relative or absolute.
   In such paths, “.”, “..” and “/” are special names.
6. Just a series of slash- or backslash- components.
   Such paths can be either relative or absolute.
   In such paths, “.”, “..” and “/” are special names.

To simplify things further, depending on the version of Windows, a drive name can be:

• only one letter between A and Z;
• any sequence of letters between A an Z;
• something that looks like Volume{41AF5D4F-04CC-4D15-9389-734BD6F52A7E}.

Also

• if a path starts with “\\?\”, its length is limited to 32,767 chars;
• otherwise, its length is limited to 260 chars.

Also

• some names such as “LPT”, “COM”, etc. are reserved and cannot be used as file names;
• … unless your path starts with “\\”.

Also

• paths are case-insensitive;
• … except when they are case-sensitive because of the disk format;
• … except when they are case-sensitive because of something else.

Fortunately, the Windows APIs provides the following functions to simplify matters:

• PathCanonicalize (completely broken);
• GetFullPathName (broken);
• GetLongPathName (requires access permissions just to tell you if a path is well-formatted);
• UriCanonicalize (not sure what it does exactly, I haven’t tested it yet).

Of course, not all Windows API functions accept all schemes.

As you can imagine, I am having lots of fun.

Quick exercise given two paths A and B (either absolute or relative), how do you determine the path obtained by concatenating A and B?

If you are interested in following my progress, details are on bugzilla.

Introducing JavaScript native file management

Summary

The Mozilla Platform keeps improving: JavaScript native file management is an undergoing work to provide a high-performance JavaScript-friendly API to manipulate the file system.

The Mozilla Platform, JavaScript and Files

The Mozilla Platform is the application development framework behind Firefox, Thunderbird, Instantbird, Camino, Songbird and a number of other applications.

While the performance-critical components of the Mozilla Platform are developed in C/C++, an increasing number of components and add-ons are implemented in pure JavaScript. While JavaScript cannot hope to match the speed or robustness of C++ yet (edit: at least not on all aspects), the richness and dynamism of the language permit the creation of extremely flexible and developer-friendly APIs, as well as quick prototyping and concise implementation of complex algorithms without the fear of memory errors and with features such as higher-level programming, asynchronous programming and now clean and efficient multi-threading. If you combine this with the impressive speed-ups experienced by JavaScript in the recent years, it is easy to understand why the language has become a key element in the current effort to make the Mozilla Platform and its add-ons faster and more responsive at all levels.

Extrapol, première partie : de C aux effets

Après un billet dans la langue de Turing, voici une présentation d’Extrapol en version française. En quelques mots, le projet Extrapol (pour Extraction de Politiques de Sécurité) vise à combler un vide dans le jeu d’outils dont dispose l’administrateur pour maintenir un système dans un état sûr.

Le logiciel libre

Je commence ce billet depuis la Journée du Libre à Bourges, assis tranquillement dans l’amphithéâtre, en train d’écouter un exposé fort intéressant. Dans l’assistance, personne ne demande ce qu’est un Logiciel Libre ou à quoi cela peut bien servir. C’est que les orateurs, il faut l’avouer, prêchent à des convertis, certains d’entre eux acteurs du monde du Logiciel Libre.

Lundi dernier, par contre, lorsque j’ai mentionné les Logiciels Libres à mes étudiants, la classe s’est divisée en deux parties : ceux d’entre eux qui n’en avaient jamais entendu parler et ceux qui m’ont répondu sans hésiter que, oui, il s’agissait du “freeware”, c’est-à-dire du logiciel gratuit. J’espère que vous ne m’en voudrez pas si je les corrige par le biais de ce blog.

Un logiciel libre est caractérisé non pas par son coût mais par la liberté — c’est-à-dire par des droits accordés aux utilisateurs.