Beautiful Off Main Thread File I/O

October 18, 2012 § 7 Comments

Now that the main work on Off Main Thread File I/O for Firefox is complete, I have finally found some time to test-drive the combination of Task.js and OS.File. Let me tell you one thing: it rocks!

« Read the rest of this entry »

(re)introducing OS.File

June 27, 2012 § 6 Comments

OS.File is a new JavaScript library available to Firefox and Thunderbird developers and add-on developers. This library offers efficient, low-level, backgrounded, interaction with the file system, with a number of primitives to take advantage of the specific features of each platform. It is also a nice example of systems programming in JavaScript. Please use it, look at the code, and please report bugs and missing features.

(re)Introducing OS.File

A considerable aspect of our work, at Mozilla, is to ensure that the user experience is smooth and responsive. One of the main tools available to developers to permit such responsive code is multi-threading: any computation or interaction with the system that takes too long can (and should) be pushed into the background, and should interact asynchronously with the user interface.

Now, one of critical bottlenecks in any application is I/O: accessing the disk (or the network, or the database…) is typically orders of magnitude slower than any in-memory operation – plus it can sometimes disrupt the user experience of the complete system. This is true on desktop systems and this is even more true on smartphones and tablets.

What this means is that we need a nice library to perform I/O, and by nice, I mean:

  • I/O should be backgrounded;
  • the number of I/O operations should be carefully controlled.

This is what OS.File is all about: OS.File is a library available to developers (including add-on developers) on the Mozilla platforms
(Firefox, Thunderbird, Songbird, InstantBird, Boot-to-Gecko, etc.). This library is available (only) to JavaScript, and it offers
low-level access to the file system, available to background threads.

As its name implies, OS.File is a system library, not a web library, so web application developers will not have access to it.

A first usable version of OS.File has landed a few days ago and is now available on nightly build of Mozilla Platform applications. We are progressively working on adding features, and I would like to invite all developers who need to do I/O to try it, report any bugs and request any features they need.

Using OS.File

OS.File offers both a cross-platform API (module OS.File itself) and bindings to platform-specific functions (modules OS.Win.File and OS.Unix.File), as well as utilities for system programming (modules OS.Shared and OS.Constants). In this post, I will only discuss module OS.File itself.

By design, in this first delivery, module OS.File is quite minimalistic. Features will be added progressively (see next section). You can find the documentation of OS.File on MDN, as usual.

For the moment, module OS.File can be used only from a chrome worker (i.e. a privileged JavaScript background thread).

Renaming a file


OS.File.move("a.tmp", "b.tmp");

In case of error, this will raise an exception of type OS.File.Error.

Copying a file, handling errors, options


try {
  OS.File.copy("b.tmp", "c.tmp", {noOverwrite: true});
} catch(ex) {
  if (ex.becauseNoSuchFile) {
    // b.tmp does not exist
  } else if (ex.becauseFileExists) {
    // c.tmp exists and we do not want to overwrite it
  }
}

Open a file, read a prefix


let buffer = new ArrayBuffer(12); // Also works with a js-ctypes C pointer
let file
try {
  file = OS.File.open("myfile.tmp"); // No options: open for reading
  let bytes = file.read(buffer, 12);
  // Do something with these bytes
  // ...
} finally {
  if (file) {
    file.close();
  }
}

Open a file for writing


let file = OS.File.open("myfile.tmp", {create:true}); // Fail if the file already exists

Note that this operation will only require one I/O interaction with the operating system – this is much faster than first checking whether the file already exists, and then creating it if it does not.

Open a file with OS-specific options


let file = OS.File.open("myfile.tmp",
  {create:true},
  {unixMode: OS.Constants.libc.S_IRWXU | OS.Constants.libc.S_IRWXG }
);

Short FAQ

What’s good about OS.File?

  • Finally, file I/O for JavaScript workers.
  • An API much more JavaScript-friendly than what already existed in the Mozilla Platform.
  • Options and low-level functions to ensure that we perform minimal amount of actual I/O.

Wasn’t all that already possible?

The existing I/O libraries on the Mozilla Platform could not be used from background threads. Some functions could be backgrounded, but only very few of them.

JavaScript-friendly wrappers had been written around these libraries, but they only covered a few of the features of these libraries, in addition to which they could not be used from background threads either.

How is OS.File implemented?

OS.File is implemented in pure JavaScript, using the (very nice) js-ctypes library to perform calls to the OS APIs.

Why JavaScript and not C++?

Because we want the code to be easily accessible to the community.

Isn’t that slow?

Well, firstly, JavaScript has grown into a very fast language. These days, expecting without benchmarks that C++ is faster than JavaScript on hot code can cause surprises.

In addition, writing the library in C++ would have meant that we needed to cross language barriers quite often, which is bad for performance, due to:

  • complex memory management;
  • bad JIT-ability; and
  • need to convert all data structures, in particular strings.

We attempt to avoid this as much as possible.

For the moment, however, OS.File has not been benchmarked. We await real-world applications.

Work in progress

We are currently hard at work extending OS.File. The next few landings should add:

Features are driven by application requirements, so if you need some other feature, please do not hesitate to contact me on IRC or to file a bug on Bugzilla.

Fun with Windows paths.

June 19, 2012 § 3 Comments

I am currently attempting to implement a JavaScript library to handle file system paths in a portable manner.

Right now, I am having lots of fun with Windows paths and I wanted to share a few tidbits.

Under Windows, a path name can look like:

  1. “\\?\drivename:” followed by backslash-separated components.
    Such paths can be either relative or absolute.
    In such paths, “.”, “..” and “/” are regular file names.
  2. “\\.\drivename:” followed by backslash-separated components.
    Such paths can be either relative or absolute.
    In such paths, “.”, “..” and “/” are special names.
  3. “\\?\UNC\servername” followed by backslash-separated components.
    Such paths can only be absolute.
    In such paths, “.”, “..” and “/” are regular file names.
  4. “\\servername” followed by slash- or backslash- components.
    Such paths can only be absolute.
    In such paths, “.”, “..” and “/” are special names.
  5. “drivename:” followed by slash- or backslash- components.
    Such paths can be either relative or absolute.
    In such paths, “.”, “..” and “/” are special names.
  6. Just a series of slash- or backslash- components.
    Such paths can be either relative or absolute.
    In such paths, “.”, “..” and “/” are special names.

To simplify things further, depending on the version of Windows, a drive name can be:

  • only one letter between A and Z;
  • any sequence of letters between A an Z;
  • something that looks like Volume{41AF5D4F-04CC-4D15-9389-734BD6F52A7E}.

Also

  • if a path starts with “\\?\”, its length is limited to 32,767 chars;
  • otherwise, its length is limited to 260 chars.

Also

  • some names such as “LPT”, “COM”, etc. are reserved and cannot be used as file names;
  • … unless your path starts with “\\”.

Also

  • paths are case-insensitive;
  • … except when they are case-sensitive because of the disk format;
  • … except when they are case-sensitive because of something else.

Fortunately, the Windows APIs provides the following functions to simplify matters:

  • PathCanonicalize (completely broken);
  • GetFullPathName (broken);
  • GetLongPathName (requires access permissions just to tell you if a path is well-formatted);
  • UriCanonicalize (not sure what it does exactly, I haven’t tested it yet).

Of course, not all Windows API functions accept all schemes.

As you can imagine, I am having lots of fun.

Quick exercise given two paths A and B (either absolute or relative), how do you determine the path obtained by concatenating A and B?

If you are interested in following my progress, details are on bugzilla.

Where Am I?

You are currently browsing entries tagged with os at Il y a du thé renversé au bord de la table.

Follow

Get every new post delivered to your Inbox.

Join 25 other followers