October 18, 2013 § Leave a Comment
In the Mozilla Platform, I/O is largely about streams. Copying streams is a rather common activity, e.g. for the purpose of downloading files, decompressing archives, saving decoded images, etc. As usual, doing any I/O on the main thread is a very bad idea, so the recommended manner of copying streams is to use one of the asynchronous string copy APIs provided by the platform:
NS_AsyncCopy (in C++) and
NS_AsyncCopy is a well-designed (if a little complex) API. It copies the full contents of an input stream into an output stream, then closes both.
NS_AsyncCopy can be called with both synchronous and asynchronous streams. By default, all operations take place off the main thread, which is exactly what is needed.
In particular, even when used with the dreaded Safe File Output Stream,
NS_AsyncCopy will perform every piece of I/O out of the main thread.
The default setting of reading data by chunks of 4kb might not be appropriate to all data, as it may cause too much I/O, in particular if you are reading a small file. There is no obvious way for clients to detect the right setting without causing file I/O, so it might be a good idea to eventually extend
NS_AsyncCopy to autodetect the “right” chunk size for simple cases.
NS_AsyncCopy is not perfect but it is quite good and it does not cause main thread I/O.
NS_AsyncCopy will, of course, not remove main thread I/O that takes place externally. If you open a stream from the main thread, this can cause main thread I/O. In particular, file streams should really be opened with flag
DEFER_OPEN flag. Other streams, such as nsIJARInputStream do not support any form of deferred opening (bug 928329), and will cause main thread I/O when they are opened.
NS_AsyncCopy does only off main thread I/O, using a Safe File Output Stream will cause a Flush. The Flush operation is very expensive for the whole system, even when executed off the main thread. For this reason, Safe File Output Stream is generally not the right choice of output stream (bug 928321).
Finally, if you only want to copy a file, prefer
OS.File.copy (if you can call JS). This function is simpler, entirely off main thread, and supports OS-specific accelerations.
NetUtil.asyncCopy is a utility method that lets JS clients call
NS_AsyncCopy. Theoretically, it should have the same behavior. However, some oddities make its performance lower.
NS_AsyncCopy requires one of its streams to be buffered,
nsIIOUtil::outputStreamIsBuffered. These methods detect whether a stream is buffered by attempting to perform buffered I/O. Whenever they succeed, this causes main thread I/O (bug 928340).
NetUtil.asyncCopy has the same limitations as
NS_AsyncCopy. In particular, in any case in which you can replace
OS.File.copy, you should pick the latter, which is both simpler and faster.
NetUtil.asyncCopy cannot read directly from a Zip file (bug 927366).
NetUtil.asyncCopy does not fit the “modern” way of writing asynchronous code on the Mozilla Platform (bug 922298).
We need to fix a few bugs to improve the performance of asynchronous copy. If you wish to help, please do not hesitate to pick any of the bugs listed above and get in touch with me.
December 6, 2011 § 28 Comments
September 27, 2008 § Leave a Comment
Just a quick word for people who may be curious about the development of OCaml Batteries Included. Work is proceeding nicely and we’re getting close to a first official release. We’ve moved things around quite a lot recently, worked on the documentation and added a few nice features (read-only strings and arrays, uniform numeric modules with type-class-style dictionaries). We’re about to add Unicode support for inputs and outputs (based on Camomile) and an improved Scanf module and that should be it for a first release.
As a side-note, the Haskell community seems to be involved much in the same process as Batteries Included, with the Haskell Platform, aka Haskell Batteries Included. Both their schedule and their list of packages seem a little more precise than ours but the overall objective remains the same: take a great programming language used mostly by academics and turn it into a complete development platform able to compete with the best the industrial world is able to offer. The main difference, it seems, is that the Haskell Platform doesn’t have a glue layer designed to uniformize APIs. The other main difference, I’m afraid, is that the Haskell community seems much larger these days than the OCaml community — or perhaps just more active or more verbal. It is my hope that a larger and more convenient standard library will help draw (back?) both academics and developers to the OCaml world. A little more academic support wouldn’t hurt, of course.
Back to OCaml Batteries Included, I hope we’ll be able release by October 10th. At that point, we’ll need beta-testing and it will be time to decide of what should get into Batteries Included 0.2. I’m sure everyone has ideas and suggestions — it will soon be time to share them.
August 29, 2008 § Leave a Comment
After a few discussions on IRC, by e-mail and on forums, I have come to realize that both the purpose of Batteries Included and what the development of Batteries involved were quite unclear to most people — and that we should probably have started our work in quite a different manner. All these discussions have prompted a few changes and the release of a first pre-version of Batteries Included, which you may find on the OCamlForge project (you may also browse source code here and API documentation here).
This release represents what we should have produced in the first place; a simple and uniform presentation layer on top of existing libraries.