Tales of Science-Fiction Bugs: The Thing That Killed Talos

Have you ever encountered one of these bugs? One in which every single line of your code is correct, in which every type-check passes, every single unit test succeeds, the specifications are fulfilled but somehow, for no reason that can be explained rationally, it just does not work? I call them Science-Fiction Bugs. I am sure that you have met some of them. For some reason, the Mozilla Performance Team seems to stumble upon such bugs rather often, perhaps because we spend so much time refactoring other team’s code long after the original authors have moved on to other features, and combining their code with undertested libraries and technologies. Truly, this is life on the Frontier.

Today, I would like to tell you the tale of one of these Science-Fiction Bugs: The Thing That Killed Talos.

Fighting the good fight for fun and credits, with Mozilla Education

Are you a student?

Do you want to fight the good fight, for the Future of the Web, and earn credits along the way?

Mozilla Education maintains a tracker of student project topics. Each project is followed by one (or more) mentor from the Mozilla Community.

Then what are you waiting for? Come and pick or join a project or get in touch to suggest new ideas!

Are you an educator?

The tracker is also open to you. Do not hesitate to pick projects for your students, send students to us or contact us with project ideas.

We offer/accept both Development-oriented, Research-oriented projects and not-CS-oriented-at-all projects.

Are you an open-source developer/community?

If things work out smoothly, we intend to progressively open this tracker to other (non-Mozilla) projects related to the future of the web. Stay tuned – or contact us!

Scoped resources for all

A small class hierarchy has been added to MFBT, the “Mozilla Framework Based on Templates” which contains some of the core classes of the Mozilla Platform. This hierarchy introduces general-purpose and specialized classes for scope-based resource management. When it applies, Scope-based resource management is both faster than reference-counting and closer to the semantics of the algorithm, so you should use it

The codebase of Mozilla is largely written in C++. While C++ does not offer any form of automatic memory management, the (sometimes scary) flexibility of the language has allowed numerous projects to customize the manner in which memory and other resources are managed, and Mozilla is no exception. Largely, the Mozilla C++ codebase uses reference-counting, to provide automatic memory management in most cases.

While reference-counting is quite imperfect, and while future versions of Mozilla might possibly use other forms of memory management, it is also a very useful tool for such a large codebase. However, in some cases, reference-counting is just too much. Indeed, in a number of simple cases, we prefer the simpler mechanism of scope-based resource management, that is both more predictable, faster and more resource-efficient – at the cost of not being able to scale up to the more complex cases for which reference-counting or even more powerful mechanisms become much more suited.

Scope-based resource management is designed to handle resources that should be cleaned-up as soon as you leave a given scope (typically, the function), regardless of how you leave it (by reaching the end, with a break, a return or an exception).

The following extract illustrates the use of scoped resource allocation:

// returns true in case of success, false in case of error
bool copy(const char* sourceName, const char* destName, size_t bufSize) {
   ScopedFD source(open(sourceName, O_RDONLY));
   if (source.get() == -1) return false;

   ScopedFD dest(open(destName, O_WRONLY|O_CREAT, 0600));
   if (dest.get() == -1) return false;
     // source is closed automatically

   ScopedDeleteArray buf(new char[bufSize]);
   if (buf.get() == NULL) return false;
     // source, dest are closed automatically

   while (true) {
     const int bytesRead = read(source.get(), buf.rwget(), bufSize);
     if (bytesRead == 0) break;
     if (bytesRead == -1) return false;
       // source, dest, buf are cleaned-up

     const int writePos = 0;
     while (writePos < bytesRead) {
       const int bytesWritten = write(dest.get(), buf.get(),
                                      bytesRead - writePos);
       if (bytesWritten == -1) return false ;
         // source, dest, buf are cleaned-up
       writePos += bytesWritten;
     }
   }

   return true;
      // source, dest, buf are cleaned-up
}

As you can see, the main point of these scope-based resource management classes is that they are cleaned up automatically both in case of success and in case of error. In some cases, we wish to clean up resources only in case of error, as follows:

// returns -1 in case of error, the destination file descriptor in case of success
int copy(const char* sourceName, const char* destName, size_t bufSize) {
   ScopedFD source(open(sourceName, O_RDONLY));
   if (source.get() == -1) return -1;

   ScopedFD dest(open(destName, O_WRONLY|O_CREAT, 0600));
   if (dest.get() == -1) return -1;
      // source is closed automatically

   ScopedDeleteArray buf(new char[bufSize]);
   if (buf.get() == NULL) return -1;
      // source, dest are closed automatically

   while (true) {
     const int bytesRead = read(source.get(), buf.rwget(), bufSize);
     if (bytesRead == 0) break;
     if (bytesRead == -1) return -1;
      // source, dest, buf are cleaned-up

     const int writePos = 0;
     while (writePos < bytesRead) {
       const int bytesWritten = write(dest.get(), buf.get(),
                                      bytesRead - writePos);
       if (bytesWritten == -1) return -1 ;
        // source, dest, buf are cleaned-up
       writePos += bytesWritten;
     }
   }

   return dest.forget();
   // source and buf are cleaned-up, not dest
}

While both examples could undoubtedly be implemented with reference-counting or without any form of automated resource management, this would either make the source code much more complex and harder to maintain (for purely manual resource management) or make the executable slower and less explicit in terms of ownership (for reference-counting). In other words, scoped-based resource management is the right choice for these algorithms.

Now, the Mozilla codebase has offered a few classes for scope-based resource management. Unfortunately, these classes were scattered throughout the code, some of them were specific to some compilers, and they were generally not designed to be reusable.

We have recently starting consolidating these classes into a simple and extensible hierarchy of classes. If you need them, you can find the root of this hierarchy, as well as the most commonly used classes, on mozilla-central, as part of the MFBT:

• ScopedFreePtr<T> is suited to deallocate C-style pointers allocated with malloc;
• ScopedDeletePtr<T> is suited to deallocate C++-style pointers allocated with new;
• ScopedDeleteArray<T> is suited to deallocate C++-style pointers allocated with new[];
• root class Scoped<Trait> and macro SCOPED_TEMPLATE are designed to make it extremely simple to define new classes to handle other cases.

For instance, class ScopedFD as used in the above examples to close Unix-style file descriptors, can be defined with the following few lines of code:

struct ScopedFDTrait
{
public:
  typedef int type;
  static type empty() { return -1; }
  static void release(type fd) {
    if (fd != -1) {
      close(fd);
    }
  }
};
SCOPED_TEMPLATE(ScopedFD, ScopedFDTrait);

So, well, if you need scoped-based resource management, you know where to find it!

I will blog shortly about the situation in JavaScript.

The OPA type system, part 1

edit Part 2 of this post was never written. I no longer work on Opa. For any question regarding Opa, please contact MLstate.

Since the initial announcement regarding OPA, we have received a number of questions regarding all the aspects of the language (including, suprisingly, a few demands for answers and documentation). Well, while we’re busy putting together documentation, benchmarks and FAQ, here’s a quick tour of one of the most fundamental pieces of the language: the type system.

Extrapol update

A quick work regarding the current status of Extrapol and its release.

Development of Extrapol progresses. With our current set of sample, Extrapol works flawlessly. We’re now adding features, improving error reporting and de-hard-wiring the model of the C standard library from the tool and moving it towards an external configuration file as well as progressively moving towards larger and more realistic samples. Development will come to an abrupt (and temporary) halt at the end of this week, though, due to personal matters (i.e. I’m getting married).

The release planned for next week, on the other hand, is canceled. As the research field of applied security is very competitive, and after careful discussion with the rest of my research team, we have decided to only release a version of Extrapol after the scientific content has been accepted for publication in a conference or journal. At the request of one of the institutes which founds this research, I will also refrain from posting detailed information on the theory and algorithms behind Extrapol, until these are cleared by the institute and accepted for publication. Without entering the details, Extrapol is expected to serve in critical infrastructures, which explains the need for clearance.

However, rest assured that there will be a release and it will be open-source (presumably licenced under a combination of MIT and LGPL). The only question is when — and this probably won’t happen before November.

Extrapol source code available (not)

A quick note to inform you that the repository for Extrapol is now public. The source code as available on the repository does not have a licence yet and will not compile as such, due to dependencies on libraries available somewhere else. Stay tuned for an actual release.

Update: Sorry, repository cut off by the administrator. I’ll inform you when the sources are back.

Note rapide pour vous informer que le code source d’Extrapol est maintenant disponible au public. Il ne s’agit pas encore d’une version officielle — en particulier, le code n’a pas encore de licence et il manque des bibliothèques (disponibles ailleurs). Plus de détails dès qu’une version officielle est disponible.

Additif: Désolé, je viens d’apprendre que le dépôt de source a été isolé par l’administrateur. Je vous tiendrai au courant dès que le code source est de nouveau public.

Extrapol, part 1: from C to Effects

Here comes the long-promised description of Extrapol, my main ongoing research project. In a few words, our objective with Extrapol is to fill a hole in the current suite of tools built to ensure the security of systems. While there’s an ample amount of stuff designed to analyse the behaviour of processes either during their execution (dynamic analysis) or after their completion (trace analysis), there is little work on applying static analysis to actual system security.

Security Extensions for Firefox, the final word (for this year) is :(

As I mentioned a few months ago, two master students of mine have been working for the best part of one year on improving the security of extensions in Firefox and Thunderbird. To sum up the current situation in Firefox, extensions have no protection mechanism from each other, nor is the core of Firefox protected in any way from extensions. The objective of this work was to design and implement a mechanism allowing system administrators to define fine-grained policies for accepting or rejecting interactions between extensions or between extensions and the core of Firefox.

MLS for Thunderbird, final word (for now)

As mentioned a few times, I have (had) two students working on a Thunderbird extension to support confidentiality and help avoid involuntary leaks of critical information by e-mail. Yesterday, these students officially turned this project in, which gave me and my colleagues the opportunity of reviewing the code and documentation.

So, what works ?

• The Thunderbird extension can detect that you’re trying to send e-mail to someone with a lower level of accreditation — although, for the moment, it gets confused easily. For this purpose, it may use either SELinux or a text database of recipients.
• The Thunderbird extension can warn you that you need to sanitize and decrease confidentiality of the information and can mark the outgoing e-mail as sanitized for a given level — the UI needs a bit polish, but that works.
• The Sendmail extension can detect that you’re trying to send e-mail to someone with a lower level of accreditation — although, for the moment, it gets confused just as easily and is subject to a number of security holes.
• The Sendmail extension can reject unsanitized e-mail going to unaccredited targets — logging needs a bit of polish, but that works.
• That’s it.

All in all, that’s about 150 lines of code in JavaScript, XUL, C and C++. Not quite ready for prime-time but a good 0.1 release. I expect either the students or I will upload it somewhere for public release in the near future.

Don’t fear the monads

If you are a developer keeping somewhat up-to-date with the latest and upcoming trends, chances are that you have heard of monads. Now, one of the traditions among developers who hear about monads is to figure them out then write a tutorial. I haven’t quite reached the second stage, but I can point you to a 1h on-line video lecture, targeted for C# and Java developers, and introducing both functional programming and monads.

Enjoy !