Copying streams asynchronously
October 18, 2013 § Leave a comment
In the Mozilla Platform, I/O is largely about streams. Copying streams is a rather common activity, e.g. for the purpose of downloading files, decompressing archives, saving decoded images, etc. As usual, doing any I/O on the main thread is a very bad idea, so the recommended manner of copying streams is to use one of the asynchronous string copy APIs provided by the platform:
NS_AsyncCopy (in C++) and
NS_AsyncCopy is a well-designed (if a little complex) API. It copies the full contents of an input stream into an output stream, then closes both.
NS_AsyncCopy can be called with both synchronous and asynchronous streams. By default, all operations take place off the main thread, which is exactly what is needed.
In particular, even when used with the dreaded Safe File Output Stream,
NS_AsyncCopy will perform every piece of I/O out of the main thread.
The default setting of reading data by chunks of 4kb might not be appropriate to all data, as it may cause too much I/O, in particular if you are reading a small file. There is no obvious way for clients to detect the right setting without causing file I/O, so it might be a good idea to eventually extend
NS_AsyncCopy to autodetect the “right” chunk size for simple cases.
NS_AsyncCopy is not perfect but it is quite good and it does not cause main thread I/O.
NS_AsyncCopy will, of course, not remove main thread I/O that takes place externally. If you open a stream from the main thread, this can cause main thread I/O. In particular, file streams should really be opened with flag
DEFER_OPEN flag. Other streams, such as nsIJARInputStream do not support any form of deferred opening (bug 928329), and will cause main thread I/O when they are opened.
NS_AsyncCopy does only off main thread I/O, using a Safe File Output Stream will cause a Flush. The Flush operation is very expensive for the whole system, even when executed off the main thread. For this reason, Safe File Output Stream is generally not the right choice of output stream (bug 928321).
Finally, if you only want to copy a file, prefer
OS.File.copy (if you can call JS). This function is simpler, entirely off main thread, and supports OS-specific accelerations.
NetUtil.asyncCopy is a utility method that lets JS clients call
NS_AsyncCopy. Theoretically, it should have the same behavior. However, some oddities make its performance lower.
NS_AsyncCopy requires one of its streams to be buffered,
nsIIOUtil::outputStreamIsBuffered. These methods detect whether a stream is buffered by attempting to perform buffered I/O. Whenever they succeed, this causes main thread I/O (bug 928340).
NetUtil.asyncCopy has the same limitations as
NS_AsyncCopy. In particular, in any case in which you can replace
OS.File.copy, you should pick the latter, which is both simpler and faster.
NetUtil.asyncCopy cannot read directly from a Zip file (bug 927366).
NetUtil.asyncCopy does not fit the “modern” way of writing asynchronous code on the Mozilla Platform (bug 922298).
We need to fix a few bugs to improve the performance of asynchronous copy. If you wish to help, please do not hesitate to pick any of the bugs listed above and get in touch with me.