June 3, 2015 § 17 Comments
Gerv’s recent post on the Jeeves Test got me thinking of the Firefox of my dreams. So I decided to write down a few ideas on how I would like to experience the web. Today: Beyond Bookmarks. Let me emphasize that the features described in this blog post do not exist.
« Look, here is an interesting website. I want to read that content (or watch that video, or play that game), just not immediately. » So, what am I going to do to remember that I wish to read it later:
- Bookmark it?
- Save it to disk?
- Pocket it?
- Remember that I saw it and find it in my history later?
- Remember that I saw it and find it in my Awesome Bar later?
- Hope that it shows up in the New Tab page?
- Open a tab?
- Install the Open Web App for that website?
- Open a tab and put that tab in a tab group?
Wow, that’s 9 ways of fulfilling the same task. Having so many ways of doing the same thing is not a very good sign, so let’s see if we can find a way to unify a few of these abstractions into something more generic and powerful.
Bookmarking is saving is reading later
What are the differences between Bookmarking and Saving?
- Bookmarking keeps a URL, while Saving keeps a snapshot.
- Bookmarks can be used only from within the browser, while Saved files can be used only from without.
Merging these two features is actually quite easy. Let’s introduce a new button, the Awesome Bookmarks which will serve as a replacement for both the Bookmark button and Save As.
- Clicking on the Awesome Bookmarks icon saves both the URL to the internal database and a snapshot to the Downloads directory (also accessible through the Downloads menu).
- Opening an Awesome Bookmark, whether from the browser or from the OS both lead the user to (by default) the live version of the page, or (if the computer is not connected) to the snapshot.
- Whenever visiting a page that has an Awesome Bookmark, the Awesome Bookmark icon changes color to offer the user the ability to switch between the live version or the snapshot.
- The same page can be Awesome Bookmarked several times, offering the ability to switch between several snapshots.
By switching to Awesome Bookmarks, we have merged Saving, Bookmarking and the Read it Later list of Pocket. Actually, since Firefox already offers Sync and Social Sharing, we have just merged all the features of Pocket.
So we have removed collapsed items from our list into one.
Bookmarks are history are tiles
What are the differences between Bookmarks and History?
- History is recorded automatically, while Bookmarks need to be recorded manually.
- History is eventually forgotten, while Bookmarks are not.
- Bookmarks can be put in folders, History cannot.
Let’s keep doing almost that, but without segregating the views. Let us introduce a new view, the Awesome Pages, which will serve as a replacement for both Bookmarks Menu and the History Menu.
This view shows a grid of thumbnails of visited pages, iOS/Android/Firefox OS style.
- first the pages visited most often during the past few hours (with the option of scrolling for all the pages visited during the past few hours);
- then the Awesome Bookmarks (because, after all, the user has decided to mark these pages)/Awesome Bookmarks folders (with the option of scrolling for more favourites);
- then, if the user has opted in for suggestions, a set of Awesome Suggested Tiles (with the option of scrolling for more suggestions);
- then the pages visited the most often today (with the option of scrolling for the other pages visited today);
- then the pages visited most often this week (with the option of scrolling for the other pages visited this week);
By default, clicking on an Awesome Bookmark (or history entry, or suggested page, etc.) for a page that is already opened switches to that page. Non-bookmarked pages can be turned into Awesome Bookmarks trivially, by starring them or putting them into folders.
An Awesome Bar at the top of this Awesome Pages lets users quickly search for pages and folders. This is the same Awesome Bar that is already at the top of tabs in today’s Firefox, just with the full-screen Awesome Pages replacing the current drop-down menu.
Oh, and by the way, this Awesome Pages is actually our new New Tab page.
By switching to the Awesome Pages, we have merged:
- the history menu;
- the bookmarks menu;
- the new tab page;
- the awesome bar.
Bookmarks are tabs are apps
What are the differences between Bookmarks and Tabs?
- Clicking on a bookmark opens the page by loading it, while clicking on a tab opens the page by switching to it.
That’s not much of a difference, is it?
So let’s make a few more changes to our UX:
- Awesome Bookmarks record the state of the page, in the style of Session Restore, so clicking on an Awesome Bookmark actually restores that page, whenever possible, instead of reloading it;
- The ribbon on top of the browser, which traditionally contains tabs, is actually a simplified display of the Awesome Pages, which shows, by default, the pages most often visited during the past few hours;
- Whether clicking on a ribbon item switches to a page or restores it is an implementation detail, which depends on whether the browser has decided that unloading a page was a good idea for memory/CPU/battery usage;
- Replace Panorama with the Awesome Page, without further change.
So, with a little imagination (and, I’ll admit, a little hand-waving), we have merged tabs and bookmarks. Interestingly, we have done that by moving to an Apps-like model, in which whether an application is loaded or not is for the OS to decide, rather than the user.
By the way, what are the differences between Tabs and Open Web Apps?
- Apps can be killed by the OS, while Tabs cannot.
- Apps are visible to the OS, while Tabs appear in the browser only.
Well, if we decide that Apps are just Bookmarks, since Bookmarks have been made visible to the OS in section 1., and since Bookmarks have just been merged with Tabs which have just been made killable by the browser, we have our Apps model.
We have just removed three more items from our list.
We are down to one higher-level abstraction (the Awesome Bookmark) and one view of it (the Awesome Page). Of course, if this is eventually released, we are certainly going to call both Persona.
This new Firefox is quite different from today’s Firefox. Actually, it looks much more like Firefox OS, which may be a good thing. While I realize that many of the details are handwavy (e.g. how do you open the same page twice simultaneously?), I believe that someone smarter than me can do great things with this preliminary exploration.
I would like to try that Firefox. Would you?
May 29, 2015 § 9 Comments
Gerv’s recent post on the Jeeves Test got me thinking of the Firefox of my dreams. So I decided to write down a few ideas on how I would like to experience the web. Today: Firefox Agents. Let me emphasise that the features described in this blog post do not exist.
Marcel uses Firefox every day, for quite a number of things.
- He uses Firefox for fun, for watching videos and playing online games. For this purpose, he has installed a few tools for finding and downloading videos. Also, one of his main search engines is YouTube. Suggested movies? Sure, as long as they are fun.
- He uses Firefox for social networks. He follows his friends, he searches on Facebook, or Twitter, or Google+. If anything looks fun, or useful, he’d like to be informed.
- He uses Firefox for managing his bank accounts, his taxes, his health insurance. For this purpose, he has paranoid security settings – to avoid phishing, he can only browse to a few whitelisted websites – and no add-ons. He may be interested in getting information from these few websites, and in security updates, but that’s about it. Also, since Firefox handles all his passwords, it must itself be protected by a password.
- He uses Firefox to read his Gmail account. And to read his other Gmail account. And he doesn’t want to leak privacy information by doing so on the same Firefox that he’s using for browsing.
- Oh, and he may also be using Firefox for browsing websites that are sensitive for any kind of reason, whether he’s hunting for gifts for his close family, dating online, chatting with hackers, discussing politics, helping NGOs in sensitive parts of the globe, visiting BitTorrent trackers, consulting a physician through some online service, or, well, anything else that requires privacy. He’d like to perform such browsing with additional anonymity guarantees. This also means locking Firefox with a password.
- Sometimes, his children or friends borrow his computer and use Firefox, too.
Of course, since Marcel brings his own device at (or from) work, that’s the same Firefox that he’s using for all of these tasks, and he’s probably even doing several of these tasks at the same time.
So, Marcel has a set of contradictory requirements, not to mention that each of his uses of Firefox needs to pass a distinct Jeeves Test. How do we keep him happy nevertheless?
Introducing Firefox Agents
In the rest of this post, I will be calling each of these uses of Firefox an Agent (if we ever implement this feature, it will, of course, be called Persona). Each Agent matches one way you use Firefox. While Firefox may be delivered with a predefined set of Agents, users can easily create new Agents. In the example, Marcel has his “Fun Agent”, his “Social Agent”, his “Work Agent”, etc.
Each Agent is unique and separate:
- Each Agent has its own icon on Marcel’s menu/desktop/tablet/phone and task list.
- Each Agent has its own visual identity, to make sure that work-related stuff doesn’t end up accidentally in the Fun Agent.
- Each Agent has its own set of preferences, bookmarks, remembered passwords, cookies, cache, and add-ons.
- Each website may be connected to a given Agent, so that links received through Gmail or through Thunderbird, for instance, automatically open with the right Agent.
As a consequence, any technology that can come bundled with Firefox to, for instance, provide search suggestions or any other kind of website suggestions is tied to an Agent. For instance, Marcel’s browsing a dating site, or shopping for shoes, or having religious activities will not be visible to any of his colleagues looking above his shoulder at his Work Agent, nor will it be tied to either of Marcel’s Gmail accounts. This greatly increases the chances of suggestion technologies passing the Jeeves Test.
Agents are also connected:
- A menu in each Agent, as well as a keyboard shortcut, lets users quickly open/switch to other Agents.
- When an Agent follows a link to a website that belongs to another Agent, the relevant Agent opens automatically.
- Bookmarks may be pushed, on demand, from one Agent to another one.
- Passwords may be pulled, on demand, from one Agent to another one.
How far are we from Agents?
Technologically speaking, Firefox Agents almost exist. Indeed, Firefox has supported Profiles forever, since way before Firefox 1.0. I generally have three instances of Firefox opened at the same time (four when I’m doing web development), and it works nicely.
With a few add-ons, you can get almost everything, although not entirely connected together:
- Profilist helps a lot with switching between profiles, and the dev version adds distinct icons;
- Firefox Themes implement distinct appearances;
- there are add-ons implementing whitelist browsing;
- there are add-ons implementing password-protected Firefox.
A few features are missing, but as you can see, the list is actually quite short:
- Pushing/pulling passwords and bookmarks between Agents (although that’s a subset of what Firefox Accounts can do).
- Attaching specific websites to specific Agents (although this doesn’t seem too difficult to implement).
- Connecting this all together.
I would like to browse with this Firefox. Would you?
January 13, 2015 § 2 Comments
(This text has initially been written for the French-speaking Mozilla Community. Most members of the community haven’t had a chance to review or sign it yet.)
I am Charlie. Some of us grew up with Cabu’s children cartoons or Wolinkski’s willies. Some of us laughed at Charb’s cover pages, others much less, and yet others had never even heard of Charlie Hebdo. Despite our differences, from the bottom of our heart, we are with those who defend Free Speech, the right to discuss, draw, make laugh or cringe.
I am a Cop. Some among us work directly with law enforcement, or ensure the online safety of individuals or organizations. Some of us make their voice heard when legal or executive powers around the world decide that security, convenience or economic interests matter more than the rights of users. All, we salute the police officers murdered or wounded these last few days as they attempted to save innocents.
I am Jew, or Muslim, or Anything else. Some among us are Jew, or Muslim, or Christian, or anything else, and, frankly, most of us don’t care who is what. All, we are horrified that, in the 21st century, extremists may still decide to murder innocents, solely because they might be Jew, and because they had decided to go the grocery store. All, we are appalled that, in the 21st century, extremists may still decide to attack innocents, just because they might be Muslems, through threats, physical violence or through their places of cult. All, we are shocked whenever opportunists praise murders or violence, or call for hatred or the ones or the others.
I am Collateral. Before we are the Mozilla Community, we are a community of individuals. Any one of us could have been at the front desk of this building, or on the path of that car, hostage or collateral kill of the assassins. Our minute of silence is for the anonymous ones, too.
I am Vigilant. Some of us believe that strong and immediate measures must be taken. Others prefer to wait until the emotion has passed before we can think of an appropriate response. All, we wait to see how the attacks of January 7th and January 9th 2015 will change our society. All, we remain vigilant, to make sure that, on top of the blood of the dead, our society does not choose to sacrifice Human Rights, Free Speech and Privacy, in the name of a securitarian ideology or opportunistic economical interests.
I am the French-speaking Mozilla Community.
Text edited by myself. List of signatures of the French version.
November 19, 2014 § Leave a comment
Oh, wait, that’s fixed already.
September 30, 2014 § 6 Comments
September is ending, and with it Q3 of 2014. It’s time for a brief report, so here is what happened during the summer.
After ~18 months working on Session Restore, I am progressively switching away from that topic. Most of the main performance issues that we set out to solve have been solved already, we have considerably improved safety, cleaned up lots of the code, and added plenty of measurements.
During this quarter, I have been working on various attempts to optimize both loading speed and saving speed. Unfortunately, both ongoing works were delayed by external factors and postponed to a yet undetermined date. I have also been hard at work on trying to pin down performance regressions (which turned out to be external to Session Restore) and safety bugs (which were eventually found and fixed by Tim Taubert).
In the next quarter, I plan to work on Session Restore only in a support role, for the purpose of reviewing and mentoring.
Also, a rant The work on Session Restore has relied heavily on collaboration between the Perf team and the FxTeam. Unfortunately, the resources were not always available to make this collaboration work. I imagine that the FxTeam is spread too thin onto too many tasks, with too many fires to fight. Regardless, the symptom I experienced is that during the course of this work, both low-priority, high-priority and safety-critical patches have been left to rot without reviews, despite my repeated requests, for 6, 8 or 10 weeks, much to the dismay of everyone involved. This means man·months of work thrown to /dev/null, along with quarterly objectives, morale, opportunities, contributors and good ideas.
I will try and blog about this, eventually. But please, in the future, everyone: remember that in the long run, the priority of getting reviews done (or explaining that you’re not going to) is a quite higher than the priority of writing code.
Many improvements to Async Tooling landed during Q3. We now have the PromiseWorker, which simplifies considerably the work of interacting between the main thread and workers, for both Firefox and add-on developers. I hear that the first add-on to make use of this new feature is currently being developed. New features, bugfixes and optimizations landed for OS.File. We have also landed the ability to watch for changes in a directory (under Windows only, for the time being).
Sadly, my work on interactions between Promise and the Test Suite is currently blocked until the DevTools team manages to get all the uncaught asynchronous errors under control. It’s hard work, and I can understand that it is not a high priority for them, so in Q4, I will try to find a way to land my work and activate it only for a subset of the mochitest suites.
I have recently joined the newly restarted effort to improve the performance of Places, the subsystem that handles our bookmarks, history, etc. For the moment, I am still getting warmed up, but I expect that most of my work during Q4 will be related to Places.
As it turns out, we had many crashes during asynchronous shutdown, a few of them safety-critical. At the time, we did not have the necessary tools to determine to prioritize our efforts or to find out whether our patches had effectively fixed bugs, so I built a dashboard to extract and display the relevant information on such crashes. This proved a wise investment, as we spent plenty of time fighting AsyncShutdown-related fires using this dashboard.
In addition to the “clean shutdown” mechanism provided by AsyncShutdown, we also now have the Shutdown Terminator. This is a watchdog subsystem, launched during shutdown, and it ensures that, no matter what, Firefox always eventually shuts down. I am waiting for data from our Crash Scene Investigators to tell us how often we need this watchdog in practice.
I lost track of how many code contributors I interacted with during the quarter, but that represents hundreds of e-mails, as well as countless hours on IRC and Bugzilla, and a few hours on ask.mozilla.org. This year’s mozEdu teaching is also looking good.
We also launched FirefoxOS in France, with big success. I found myself in a supermarket, presenting the ZTE Open C and the activities of Mozilla to the crowds, and this was a pleasing experience.
For Q4, expect more mozEdu, more mentoring, and more sleepless hours helping contributors debug their patches 🙂
July 17, 2014 § 4 Comments
Plot For the second time, our heroes prepared for battle. The startup of Firefox was too slow and Session Restore was one of the battle fields.
When Firefox starts, Session Restore is in charge of restoring the browser to its previous state, in case of a crash, a restart, or for the users who have configured Firefox to resume from its previous state. This entails numerous activities during startup:
- read sessionstore.js from disk, decode it and parse it (recall that the file is potentially several Mb large), handling errors;
- backup sessionstore.js in case of startup crash.
- create windows, tabs, frames;
- populate history, scroll position, forms, session cookies, session storage, etc.
It is common wisdom that Session Restore must have a large impact on Firefox startup. But before we could minimize this impact, we needed to measure it.
Benchmarking is not easy
When we first set foot on Session Restore territory, the contribution of that module to startup duration was uncharted. This was unsurprising, as this aspect of the Firefox performance effort was still quite young. To this day, we have not finished chartering startup or even Session Restore’s startup.
So how do we measure the impact of Session Restore on startup?
A first tool we use is Timeline Events, which let us determine how long it takes to reach a specific point of startup. Session Restore has had events `
sessionRestoreInitialized` and `
sessionRestored` for years. Unfortunately, these events did not tell us much about Session Restore itself.
The first serious attempt at measuring the impact of Session Restore on startup Performance was actually not due to the Performance team but rather to the metrics team. Indeed, data obtained through Firefox Health Report participants indicated that something wrong had happened.
d2` in the graph measures the duration between `
firstPaint` (which is the instant at which we start displaying content in our windows) and `
sessionRestored` (which is the instant at which we are satisfied that Session Restore has opened its first tab). While this measure is imperfect, the dip was worrying – indeed, it represented startups that lasted several seconds longer than usual.
Upon further investigation, we concluded that the performance regression was indeed due to Session Restore. While we had not planned to start optimizing the startup component of Session Restore, this battle was forced upon us. We had to recover from that regression and we had to start monitoring startup much better.
A second tool is Telemetry Histograms for measuring duration of individual operations, such as reading sessionstore.js or parsing it. We progressively added measures for most of the operations of Session Restore. While these measures are quite helpful, they are also unfortunately very unstable in real-world conditions, as they are affected both by scheduling (the operations are asynchronous), by the work load of the machine, by the actual contents of sessionstore.js, etc.
Difference in colors represent successive versions of Firefox. As we can see, this graph is quite noisy, certainly due to the factors mentioned above (the spikes don’t correspond to any meaningful change in Firefox or Session Restore). Also, we can see a considerable increase in the duration of the read operation. This was quite surprising for us, given that this increase corresponds to the introduction of a much faster, off the main thread, reading and decoding primitive. At the time, we were stymied by this change, which did not correspond to our experience. We have now concluded that by changing the asynchronous operation used to read the file, we have simply changed the scheduling, which makes the operation appear longer, while in practice it simply does not block the rest of the startup from taking place on another thread.
One major tool was missing for our arsenal: a stable benchmark, always executed on the same machine, with the same contents of sessionstore.js, and that would let us determine more exactly (almost daily, actually) the impact of our patches upon Session Restore:
This test, based on our Talos benchmark suite, has proved both to be very stable, and to react quickly to patches that affected its performance. It measures the duration between the instant at which we start initializing Session Restore (a new event `
sessionRestoreInit`) and the instant at which we start displaying the results (event `
With these measures at hand, we are now in a much better position to detect performance regressions (or improvements) to Session Restore startup, and to start actually working on optimizing it – we are now preparing to using this suite to experiment with “what if” situations to determine which levers would be most useful for such an optimization work.
Evolution of startup duration
Our first benchmark measures the time elapsed between start and stop of Session Restore if the user has requested all windows to be reopened automatically
As we can see, the performance on Linux 32 bits, Windows XP and Mac OS 10.6 is rather decreasing, while the performance on Linux 64 bits, Windows 7 and 8 and MacOS 10.8 is improving. Since the algorithm used by Session Restore upon startup is exactly the same for all platforms, and since “modern” platforms are speeding up while “old” platforms are slowing down, this suggests that the performance changes are not due to changes inside Session Restore. The origin of these changes is unclear. I suspect the influence of newer versions of the compilers or some of the external libraries we use, or perhaps new and improved (for some platforms) gfx.
Still, seeing the modern platforms speed up is good news. As of Firefox 31, any change we make that causes a slowdown of Session Restore will cause an immediate alert so that we can react immediately.
Our second benchmark measures the time elapsed if the user does not wish windows to be reopened automatically. We still need to read and parse sessionstore.js to find whether it is valid, so as to decide whether we can show the “Restore” button on about:home.
The influence of factors upon startup
With the help of our benchmarks, we were able to run “what if” scenarios to find out which of the data manipulated by Session Restore contributed to startup duration. We did this in a setting in which we restore windows:
and in a setting in which we do not:
Interestingly, increasing the size of sessionstore.js has apparently no influence on startup duration. Therefore, we do not need to optimize reading and parsing sessionstore.js. Similarly, optimizing history, cookies or form data would not gain us anything.
The single largest most expensive piece of data is the set of open windows – interestingly, this is the case even when we do not restore windows. More precisely, any optimization should target, by order of priority:
- the cost of opening/restoring windows;
- the cost of opening/restoring tabs;
- the cost of dealing with windows data, even when we do not restore them.
Now that we have information on which parts of Session Restore startup need to be optimized, the next step is to actually optimize them. Stay tuned!