(re)introducing OS.File

June 27, 2012 § 6 Comments

OS.File is a new JavaScript library available to Firefox and Thunderbird developers and add-on developers. This library offers efficient, low-level, backgrounded, interaction with the file system, with a number of primitives to take advantage of the specific features of each platform. It is also a nice example of systems programming in JavaScript. Please use it, look at the code, and please report bugs and missing features.

(re)Introducing OS.File

A considerable aspect of our work, at Mozilla, is to ensure that the user experience is smooth and responsive. One of the main tools available to developers to permit such responsive code is multi-threading: any computation or interaction with the system that takes too long can (and should) be pushed into the background, and should interact asynchronously with the user interface.

Now, one of critical bottlenecks in any application is I/O: accessing the disk (or the network, or the database…) is typically orders of magnitude slower than any in-memory operation – plus it can sometimes disrupt the user experience of the complete system. This is true on desktop systems and this is even more true on smartphones and tablets.

What this means is that we need a nice library to perform I/O, and by nice, I mean:

  • I/O should be backgrounded;
  • the number of I/O operations should be carefully controlled.

This is what OS.File is all about: OS.File is a library available to developers (including add-on developers) on the Mozilla platforms
(Firefox, Thunderbird, Songbird, InstantBird, Boot-to-Gecko, etc.). This library is available (only) to JavaScript, and it offers
low-level access to the file system, available to background threads.

As its name implies, OS.File is a system library, not a web library, so web application developers will not have access to it.

A first usable version of OS.File has landed a few days ago and is now available on nightly build of Mozilla Platform applications. We are progressively working on adding features, and I would like to invite all developers who need to do I/O to try it, report any bugs and request any features they need.

Using OS.File

OS.File offers both a cross-platform API (module OS.File itself) and bindings to platform-specific functions (modules OS.Win.File and OS.Unix.File), as well as utilities for system programming (modules OS.Shared and OS.Constants). In this post, I will only discuss module OS.File itself.

By design, in this first delivery, module OS.File is quite minimalistic. Features will be added progressively (see next section). You can find the documentation of OS.File on MDN, as usual.

For the moment, module OS.File can be used only from a chrome worker (i.e. a privileged JavaScript background thread).

Renaming a file


OS.File.move("a.tmp", "b.tmp");

In case of error, this will raise an exception of type OS.File.Error.

Copying a file, handling errors, options


try {
  OS.File.copy("b.tmp", "c.tmp", {noOverwrite: true});
} catch(ex) {
  if (ex.becauseNoSuchFile) {
    // b.tmp does not exist
  } else if (ex.becauseFileExists) {
    // c.tmp exists and we do not want to overwrite it
  }
}

Open a file, read a prefix


let buffer = new ArrayBuffer(12); // Also works with a js-ctypes C pointer
let file
try {
  file = OS.File.open("myfile.tmp"); // No options: open for reading
  let bytes = file.read(buffer, 12);
  // Do something with these bytes
  // ...
} finally {
  if (file) {
    file.close();
  }
}

Open a file for writing


let file = OS.File.open("myfile.tmp", {create:true}); // Fail if the file already exists

Note that this operation will only require one I/O interaction with the operating system – this is much faster than first checking whether the file already exists, and then creating it if it does not.

Open a file with OS-specific options


let file = OS.File.open("myfile.tmp",
  {create:true},
  {unixMode: OS.Constants.libc.S_IRWXU | OS.Constants.libc.S_IRWXG }
);

Short FAQ

What’s good about OS.File?

  • Finally, file I/O for JavaScript workers.
  • An API much more JavaScript-friendly than what already existed in the Mozilla Platform.
  • Options and low-level functions to ensure that we perform minimal amount of actual I/O.

Wasn’t all that already possible?

The existing I/O libraries on the Mozilla Platform could not be used from background threads. Some functions could be backgrounded, but only very few of them.

JavaScript-friendly wrappers had been written around these libraries, but they only covered a few of the features of these libraries, in addition to which they could not be used from background threads either.

How is OS.File implemented?

OS.File is implemented in pure JavaScript, using the (very nice) js-ctypes library to perform calls to the OS APIs.

Why JavaScript and not C++?

Because we want the code to be easily accessible to the community.

Isn’t that slow?

Well, firstly, JavaScript has grown into a very fast language. These days, expecting without benchmarks that C++ is faster than JavaScript on hot code can cause surprises.

In addition, writing the library in C++ would have meant that we needed to cross language barriers quite often, which is bad for performance, due to:

  • complex memory management;
  • bad JIT-ability; and
  • need to convert all data structures, in particular strings.

We attempt to avoid this as much as possible.

For the moment, however, OS.File has not been benchmarked. We await real-world applications.

Work in progress

We are currently hard at work extending OS.File. The next few landings should add:

Features are driven by application requirements, so if you need some other feature, please do not hesitate to contact me on IRC or to file a bug on Bugzilla.

About these ads

Tagged: , , , ,

§ 6 Responses to (re)introducing OS.File

  • skierpage says:

    All methods are synchronous, correct? Unlike I think W3C File Writer and File system API, http://www.w3.org/TR/file-system-api/ . I understand that Mozilla and other browser vendors aren’t fans of this Google-only html5 non-standard; but then why not adopt http://nodejs.org/api/fs.html API, which has asynchronous and synchronous forms, is more full-featured, and quite nice to use?

    • yoric says:

      Just for clarification: all methods are synchronous but usable only in background threads. In other words, from the point of view of the main thread, they are asynchronous. I am working at the moment on an asynchronous API for the main thread, which will defer all I/O to a background thread.

      While the Node.js API is nice, it is designed to map closely to a subset of an old version of Posix. This API is sufficient for server operations but not for a user-facing desktop application as ours. This shows in terms of features: for instance, the Node.js API cannot handle Windows security or sharing attributes, which are sometimes critical for us. This also shows in terms of performances: with Node.js, traversing a directory recursively requires two I/O operations per file under Unix (readdir + stat) and three under Windows (FindFirstFile or FindNextFile, CreateFile, GetFileInformationByHandle). Experience gathered during the development of Firefox shows that this is an unacceptable performances for tasks such as emptying huge directories such as the cache, in particular on very slow devices such as smartphones and tablets – by comparison, during this traversal OS.File requires one single I/O operation per file, both under any recent Unix and under Windows.

      So, in a nutshell: both APIs are nice, but they are targeted towards very different usage scenarios.

  • Does this have any support for hacking around file system idiosyncrasies? (I.e. Windows not allowing certain names (AUX, CON1-9)

    • yoric says:

      Not as such. We do not take measures to rename “AUX” to “_AUX” or whatever as this would be a never-ending pit of despair.

      What we do, however, is take advantage of non-portable file system / system API features. For instance, when traversing a directory under Windows, the developer can access a field winCreationDate that contains the creation date, at zero additional cost. Under Unix, if the developer the developer needs to call File.getInfo if she wants the same piece of information.

      • Is roundtrip Unicode file‐naming handled well?

      • yoric says:

        I am not sure I understand the question. Under Windows, we use the Unicode-based API (“W”-suffixed functions). Under Unix, at the moment, we use the default conversion between JavaScript Strings and C Strings, which is UTF-8. There is support for conversion to/from other encodings, but for the moment, the library does not take the initiative of applying them to paths. The implementation is high-level enough that this should be relatively straightforward to setup, if necessary (e.g. the implementation knows which C strings actually represent paths).

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

What’s this?

You are currently reading (re)introducing OS.File at Il y a du thé renversé au bord de la table.

meta

Follow

Get every new post delivered to your Inbox.

Join 25 other followers

%d bloggers like this: