Firefox, the Browser that has your Back[up]

June 26, 2014 § 44 Comments

One of the most important features of Firefox, in my opinion, is Session Restore. This component is responsible for ensuring that, even in case of crash, or if you upgrade your browser or an add-on that requires restart, your browser can reopen immediately and in the state in which you left it. As far as I am concerned, this feature is a life-safer.

Unfortunately, there are a few situations in which the Session Restore file may be corrupted – typically, if the computer is rebooted before the write is complete, or if it loses power, or if the operating system crashes or the disk is disconnected, we may end up losing our precious Session Restore. While any of these circumstances happens quite seldom, it needs to be applied as part of the following formula:

seldom · .5 billion users = a lot

I am excited to announce that we have just landed a new and improved Session Restore component in Firefox 33 that protects your precious data better than ever.

How it works

Firefox needs Session Restore to handle the following situations:

  • restarting Firefox without data loss after a crash of either Firefox, the Operating System, a driver or the hardware, or after Firefox has been killed by the Operating System during shutdown;
  • restarting Firefox without data loss after Firefox has been restarted due to an add-on or an upgrade;
  • quitting Firefox and, later, restarting without data loss.

In order to handle all of this, Firefox needs to take a snapshot of the state of the browser whenever anything happens, whether the user browses, fills a form, scrolls, or an application sets a Session Cookie, Session Storage, etc. (this is actually capped to one save every 15 seconds, to avoid overloading the computer). In addition, Firefox performs a clean save during shutdown.

While at the level of the application, the write mechanism itself is simple and robust, a number of things beyond the control of the developer can prevent either the Operating System or the hard drive itself from completing this write consistently – a typical example being tripping on the power plug of a desktop computer during the write.

The new mechanism involves two parts:

  • keeping smart backups to maximize the chances that at least one copy will be readable;
  • making use of the available backups to transparently avoid or minimize data loss.

The implementation actually takes very few lines of code, the key being to know the risks against which we defend.

Keeping backups

During runtime, Firefox remembers which files are known to be valid backups and which files should be discarded. Whenever a user interaction or a script requires it, Firefox writes the contents of Session Restore to a file called sessionstore-backups/recovery.js. If it is known to be good, the previous version of sessionstore-backups/recovery.js is first moved to sessionstore-backups/recovery.bak. In most cases, both files are valid and recovery.js contains a state less than 15 seconds old, while recovery.bak contains a state less than 30 seconds old. Additionally, the writes on both files are separated by at least 15 seconds. In most circumstances, this is sufficient to ensure that, even of hard drive crash during a write to recover.js, at least recovery.bak has been entirely written to disk.

During shutdown, Firefox writes a clean startup file to sessionstore.js. In most cases, this file is valid and contains the exact state of Firefox at the time of shutdown (minus some privacy filters). During startup, if sessionstore.js is valid, Firefox moves it to sessiontore-backup/previous.js. Whenever this file exists, it is valid and contains the exact state of Firefox at the time of the latest clean shutdown/startup. Note that, in case of crash, the latest clean shutdown/startup might be older than the latest actual startup, but this backup is useful nevertheless.

Finally, on the first startup after an update, Firefox copies sessionstore.js, if it is available and valid, to sessionstore-backups/upgrade.js-[build id]. This mechanism is designed primarily for testers of Firefox Nightly, who keep on the very edge, upgrading Firefox every day to check for bugs. Testers, if we introduce a bug that affects Session Restore, this can save your life.

As a side-note, we never use the operating system’s flush call, as 1/ it does not provide the guarantees that most developers expect; 2/ on most operating systems, it causes catastrophic slowdowns.

Recovering

All in all, Session Restore may contain the following files:

  • sessionstore.js (contains the state of Firefox during the latest shutdown – this file is absent in case of crash);
  • sessionstore-backups/recovery.js (contains the state of Firefox ≤ 15 seconds before the latest shutdown or crash – the file is absent in case of clean shutdown, if privacy settings instruct us to wipe it during shutdown, and after the write to sessionstore.js has returned);
  • sessionstore-backups/recovery.bak (contains the state of Firefox ≤ 30 seconds before the latest shutdown or crash – the file is absent in case of clean shutdown, if privacy settings instruct us to wipe it during shutdown, and after the removal of sessionstore-backups/recovery.js has returned);
  • sessionstore-backups/previous.js (contains the state of Firefox during the previous successful shutdown);
  • sessionstore-backups/upgrade.js-[build id] (contains the state of Firefox after your latest upgrade).

All these files use the JSON format. While this format has drawbacks, it has two huge advantages in this setting:

  • it is quite human-readable, which makes it easy to recover manually in case of an extreme crash;
  • its syntax is quite rigid, which makes it easy to find out whether it was written incompletely.

As our main threat is a crash that prevents us from writing the file entirely, we take advantage of the latter quality to determine whether a file is valid. Based on this, we test each file in the order indicated above, until we find one that is valid. We then proceed to restore it.

If Firefox was shutdown cleanly:

  1. In most cases, sessionstore.js is valid;
  2. In most cases in which sessionstore.js is invalid, sessionstore-backups/recovery.js is still present and valid (the likelihood of it being present is obviously higher if privacy settings do not instruct Firefox to remove it during shutdown);
  3. In most cases in which sessionstore-backups/recovery.js is invalid, sessionstore-backups/recovery.bak is still present, with an even higher likelihood of being valid (the likelihood of it being present is obviously higher if privacy settings do not instruct Firefox to remove it during shutdown);
  4. In most cases in which the previous files are absent or invalid, sessionstore-backups/previous.js is still present, in which case it is always valid;
  5. In most cases in which the previous files are absent or invalid, sessionstore-backups/upgrade.js-[…] is still present, in which case it is always valid.

Similarly, if Firefox crashed or was killed:

  1. In most cases, sessionstore-backups/recovery.js is present and valid;
  2. In most cases in which sessionstore-backups/recovery.js is invalid, sessionstore-backups/recovery.bak is pressent, with an even higher likelihood of being valid;
  3. In most cases in which the previous files are absent or invalid, sessionstore-backups/previous.js is still present, in which case it is always valid;
  4. In most cases in which the previous files are absent or invalid, sessionstore-backups/upgrade.js-[…] is still present, in which case it is always valid.

Numbers crunching

Statistics collected on Firefox Nightly 32 suggest that, out of 11.95 millions of startups, 75,310 involved a corrupted sessionstore.js. That’s roughly a corrupted sessionstore.js every 158 startups, which is quite a lot. This may be influenced by the fact that users of Firefox Nightly live on pre-alpha, so are more likely to encounter crashes or Firefox bugs than regular users, and that some of them use add-ons that may modify sessionstore.js themselves.

With the new algorithm, assuming that the probability for each file to be corrupted is independent and is p = 1/158, the probability of losing more than 30 seconds of data after a crash goes down to p^3 ≅ 1 / 4,000,000. If we haven’t removed the recovery files, the probability of losing more than 30 seconds of data after a clean shutdown and restart goes down to p^4 ≅ 1 / 630,000,000. This still means that , statistically speaking, at every startup, there is one user of Firefox somewhere around the world who will lose more than 30 seconds of data, but this is much, better than the previous situation by several orders of magnitude.

It is my hope that this new mechanism will transparently make your life better. Have fun with Firefox!

About these ads

Tagged: , , , , , , ,

§ 44 Responses to Firefox, the Browser that has your Back[up]

  • Nawfel says:

    Does this solve the problem of cache invalidation?

    • yoric says:

      If I understand correctly, you are referring to the ongoing debate whether Session Restore should favor caching tabs or reloading them. This update is entirely orthogonal to that debate.

      • Nawfel says:

        No, i am talking about offline mode not working after a firefox crash

      • yoric says:

        This hasn’t been addressed, no. Do you know if there is a bug open on the topic?

      • Nawfel says:

        I found this http://www.janbambas.cz/new-firefox-http-cache-backend-implementation/ which is a little outdated. But I don’t think it has found its way to the stable release yet

        This is what conserns me most:

        > The two main goals we’ve met are:
        > * Be resilient to crashes and process kills
        > …

        My problem is that after a Firefox crash, it loses all the cache and no web page will be accessible in offline mode.

        I haven’t tested if that happen with firefox 30. I will try now

      • Nawfel says:

        Ah! I found the option `browser.cache.use_new_backend` set to 0. Is it still an experimental feature ? Or would it be set to 1 if i did a fresh install.

    • Martyn says:

      A key concern. There is not one word about security in this article or comment on the ability for malicious / crooked folks to restore a user’s session.

      Additionally, how does Sync play into this? Firefox has been crashing a lot laterlly and who really knows what is beneath Firefox’s ever increasingly complex hood?

      • yoric says:

        I’m not sure what you are asking about security. If a malicious/crooked piece of software has access to your hard drive, it has access to your session data. However, this is nothing new, as it also has access to all the data files of Firefox (or any other browser you may be using), including your cache, your history, your cookies, etc.

        Sync and Session Restore have no interactions.

  • […] just announced that it has made improvements to Firefox's Session Restore feature that it plans to ship with […]

  • […] just announced that it has made improvements to Firefox’s Session Restore feature that it plans to ship with […]

  • sqramp says:

    Why sqlite with his atomic operations is not used?

    • yoric says:

      Initially, for historical reasons. Session Restore has been implemented with JSON since the first version. Changing now would 1/ require a very consequent refactoring; 2/ break a number of add-ons. Also, it is not clear to me how safe the data would be if we disallow flush() calls. Finally, backups are useful not only against data loss, but also against user errors, as a last defense.

      • derss says:

        Slight off topic: Yoric, do you know when you guys are going to implement “lazy unfolding” of sessionstore into the memory?

        Because as it is now if I have many tabs — even when “click-to-load” option is on — FireFox still allocated empty memory pages for each and every tab, which slows down start, as well as occupies HALF GIGABYTE or more memory even if you only see one tab and the rest are not loaded.

        This situation is obviously ridiculous and has to change. Can you help pro users, please?

      • yoric says:

        Yes, we are aware of the issue.
        The problem is that it requires a very deep refactoring of the implementation of `tabbrowser`, something which is both complicated, long, and hard to test. It would be great if we could have someone work on this fulltime for a few months, but I’m not sure when this is going to happen.

        The following bug was opened by a contributor who has ideas on how to get this done.

  • […] just announced that it has made improvements to Firefox’s Session Restore feature that it plans to ship with […]

  • […] just announced that it has made improvements to Firefox’s Session Restore feature that it plans to ship with […]

  • […] juin : Vient d’embarquer dans Firefox Nightly 33, un nouveau composant de restauration de session, par David Teller ; RÉSOLU RÉGLÉ bogue 883 […]

  • Nicolas B. says:

    This is a great step in the direction of reliability !

    Let’s have a closer look at the maths. :-)

    “In most cases in which sessionstore-backups/recovery.js is invalid, sessionstore-backups/recovery.bak is pressent, with an even higher likelihood of being valid”

    Why “an even higher likelihood” ?

    “This still means that, statistically speaking, at every startup, there is one user of Firefox somewhere around the world who will lose more than 30 seconds of data”

    What does this mean ? I understand this sentence as : “The probability of losing more than 30 seconds of data at startup is 1.” Which is wrong, hopefully !

    • yoric says:

      Why “an even higher likelihood” ?

      Well,

      1. rather than entirely writing the file, we simply rename recovery.js, and renaming has basically no chance of corrupting data, unless your file system is so damaged that you have much bigger concerns;
      2. the probability of recovery.js being corrupted is a decreasing function of the time since the write order was sent to the operating system, with “almost 0″ generally being reached somewhere between 10 seconds and 20 seconds unless the harddrive is badly broken, in which case you also have much bigger concerns – by the time we perform the rename, we know that at least 15 seconds have elapsed;
      3. as a side-note, the “almost 0″ probability is reached faster on a desktop, slower on a laptop running on low battery, and we plan to land shortly a patch that delays by 30 seconds instead of 15 the write if the laptop is running on battery, both to save battery life and to increase the chance of being in the “almost 0″ region;

      What does this mean ? I understand this sentence as : “The probability of losing more than 30 seconds of data at startup is 1.” Which is wrong, hopefully !

      I wanted to give the image that if every single user of Firefox on Earth decides to start Firefox at the same time, according to the probabilities above, only one of them might lose more than 30 seconds of data.

      • Nicolas B. says:

        Thank you for these answers.

        “the probability of recovery.js being corrupted is a decreasing function of the time since the write order was sent to the operating system”

        You must mean the probability *if the system is still alive*. I don’t think that a file being corrupted at t + 1 s would uncorrupt itself with time, for instance at t + 10 s.

        I understand this as : trying to copy, or rename, a file only when we are almost sure that it has been correctly written.

        The delaying to 30 s on battery is a nice move.

        An aside : the probabilities of not losing > 30 s of data discussed here are presuming that the mechanism of session store itself does not lose data. ;-)

        Keep up the good work !

      • yoric says:

        You must mean the probability *if the system is still alive*. I don’t think that a file being corrupted at t + 1 s would uncorrupt itself with time, for instance at t + 10 s.

        Not exactly, so let me rephrase: the probability of recovery.js ending up corrupted is a decreasing function of the duration between the instant the write order was sent to the operating system and the instant the computer freezes. If the computer doesn’t freeze at all within 10-20seconds after the write, there is basically no chance that the file may be corrupted (again, unless you have bigger problems).

        An aside : the probabilities of not losing > 30 s of data discussed here are presuming that the mechanism of session store itself does not lose data. ;-)

        Correct :)

  • Nicolas B. says:

    “Your comment is awaiting moderation.”

    Oh yes, some moderation, only tepid ideas. :-/ And so great the censorship !

  • flofl0 says:

    This look very good :)

    Does this mean using for example http://sessionmanager.mozdev.org/ , to protect your precious tabs is now obsolete ?

    Is Fx for android using the same scheme ?

    You mean that now, and even before Fx dump all the tabs, their state (I suppose including the position in the page, and text entered in the textarea) every 15 seconds ?
    That explain at least why unloaded tabs are still slowing your browser.
    But still session store is just a few Mo (for the thousands of tabs I have opened), so I don’t really understand the slow down (especially for opening an closing) do you have any hint on that ?

    Thanks again for enhancing a such cool function of Fx.

    • yoric says:

      Well, Session Manager lets you take snapshots, which is also useful for other reasons. We hope that this won’t be necessary to protect your tabs.

      Firefox for Android is using something entirely different, so this change doesn’t apply.

      And yes, Firefox dumps everything every 15 seconds, even unloaded tabs. I don’t know if this is what’s causing your slowdowns. Do they take place every 15 seconds?

      • Jerome says:

        Can you please answer if this new function makes the Session Manager add-on obsolete?

        I don’t care about losing what I wrote in a form, all I need is a function that backs up my currently open website tabs. If the new built-in function is reliable, I can drop Session Manager safely.

      • yoric says:

        I am not a user of Session Manager, but I seem to remember that Session Manager is a bit more flexible. However, with this refactoring, adding the flexibility of Session Manager to Session Restore would just be a few lines of code. If you know someone who is interested (perhaps the authors of Session Manager?), I’d be glad to help them putting Session Manager in Firefox.

        Also, I seem to remember that Session Manager needs to rely on some slow mechanisms, which are pretty much the only way to communicate with Session Restore at the moment, so that would also help the performance for users of Session Manager.

  • nimbosa says:

    Reblogged this on the Assault of Nimbosa and commented:
    this is an old but Timely article, as @Firefox 33 Beta 5 is being rolled out amazing! #SessionRestore #FB

  • Glad to hear about improvements. Being victim of ~500 lost tabs (4 lost sessions over 6months), looking forward to try scenarios I’ve had / know about on FF33.

  • Herb says:

    “As far as I am concerned, this feature is a life-safer.”

    That would be “life saver.”

    Anyway – my comment is, I cannot STAND session restore. Some fire fox is better than IE and Opera and I will not use Chrome due to it being Google’s.

    However sometimes FF starts to eat up more and more CPU to the point I need to shut it down and sometimes it hangs when I’m shutting down all my FF’s and I have to “end process” out of them – a LAST resort because I KNOW that when I finally shut them all down – when I restart FF – it will do the session restore and I’m back to square 1 with it reloading all the heavy websites that brought my system to a crawl in the FIRST place.

    Gotta find the settings that DISable session restore…

    • yoric says:

      That would be Preferences > General > “When Firefox starts, Show My Home Page”.

      I am also working on the hangs during shutdown, by the way.

  • Peter says:

    What’s the performance cost of this feature (15 seconds)?

    Somebody commented here: http://www.ghacks.net/2014/06/27/mozilla-launches-improved-session-restore-firefox-33/#comment-2250412 that:

    “You have to wonder at the complexity of what looks like a simple task. You’d think Ff would simply keep a trail of opened tabs (History?!). Instead they seem to fire a steady stream of access to places.sqlite.
    Recently i realised that the cpu was constantly working away while idle at 2-5% and traced it to a horrendous amount of disk activity. Changing the Browser.sessionstore.interval to 120000 (2min) seemed to calm things significantly. Everything has a price.”

    So, what do you think of this?

  • Peter says:

    My previous comment is still awaiting moderation after I posted it on 18th day, so here it is again without a website link which might trigger the pre-moderation.

    What’s the performance cost of this feature (15 seconds)?

    Somebody commented on ghacks website that:

    “You have to wonder at the complexity of what looks like a simple task. You’d think Ff would simply keep a trail of opened tabs (History?!). Instead they seem to fire a steady stream of access to places.sqlite.
    Recently i realised that the cpu was constantly working away while idle at 2-5% and traced it to a horrendous amount of disk activity. Changing the Browser.sessionstore.interval to 120000 (2min) seemed to calm things significantly. Everything has a price.”

    So, what do you think of this?

  • Christopher says:

    In the past, my Firefox crashed from time to time and then again sometimes the session was lost. It did not happen often, but when it did was annoying. Now with the new restore feature I could just recover fro such a crash and restore my session out of one of the backups, which was a big help!

    Thank you so much for this great feature!

  • Peter says:

    Sad.. just “Your comment is awaiting moderation” for my messages.. why? Did I touch a too sensitive issue?

  • Vinicius says:

    This change actually solved a small annoying behavior, where if you had only pinned tabs on your previous session, Firefox would restore them plus an unwanted new blank tab.

  • Hi,

    A slightly offtopic comment/doubt:

    Do we have a documentation for the contents of sessionstore.js ?

  • JohnE says:

    This is great to hear. I’d resorted to making my own backups of the sessionstore files for the occasion corruption since I keep hundreds of tabs open at a time… :)

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

What’s this?

You are currently reading Firefox, the Browser that has your Back[up] at Il y a du thé renversé au bord de la table.

meta

Follow

Get every new post delivered to your Inbox.

Join 33 other followers

%d bloggers like this: