Bon, retour au boulot…

Interruption d’une dizaine de jours pour le blog. Je reviendrai la bague au doigt.

A quick work regarding the current status of Extrapol and its release.

Development of Extrapol progresses. With our current set of sample, Extrapol works flawlessly. We’re now adding features, improving error reporting and de-hard-wiring the model of the C standard library from the tool and moving it towards an external configuration file as well as progressively moving towards larger and more realistic samples. Development will come to an abrupt (and temporary) halt at the end of this week, though, due to personal matters (i.e. I’m getting married).

The release planned for next week, on the other hand, is canceled. As the research field of applied security is very competitive, and after careful discussion with the rest of my research team, we have decided to only release a version of Extrapol after the scientific content has been accepted for publication in a conference or journal. At the request of one of the institutes which founds this research, I will also refrain from posting detailed information on the theory and algorithms behind Extrapol, until these are cleared by the institute and accepted for publication. Without entering the details, Extrapol is expected to serve in critical infrastructures, which explains the need for clearance.

However, rest assured that there will be a release and it will be open-source (presumably licenced under a combination of MIT and LGPL). The only question is when — and this probably won’t happen before November.

Short version

Catch me if you can is a small library for OCaml 3.10. The latest release is version 0.2, which you may find here. This library improves management of errors in OCaml. It is released under the LGPL licence. It has been written by David Teller, Arnaud Spiwack, Till Varoquaux and Gabriel Scherer.

Long version

As all languages of the ML family — and most modern languages indeed — OCaml permits the management of exceptional situations using exceptions. This mechanism lets programmer register protected sections of code, as well as exception handlers to handle any exception which may be raised during the execution of a protected section. Whenever an exception is raised, the protected section of code is immediately stopped and the corresponding exception handler is executed instead. In addition, exceptions may convey some information regarding the nature of the exceptional circumstance.

In OCaml, the mechanism is fast, it’s convenient and it’s type-safe, much like the rest of the language (barring any type-unsafe interaction with C). However, a few things are missing. If we consider the rest of the language, exceptions are both heavyweight and clumsy: each exception must be declared before being used and there’s no way to introduce a polymorphic type parameter in the exception. In addition, languages such as Java offer to important features missing in OCaml: automatic case coverage and exception hierarchies. While a nice tool exists  to provide case coverage for exceptions in OCaml, this tool is complex and  unfortunately unmaintained.

Catch me if you can offers an alternative mechanism, comparable to ML exceptions, to handle errors. In comparison with OCaml’s native exception mechanism, this library adds:

• automatic inference of exceptions (i.e. no need to declare your exceptions, unless you want to)
• more flexible exceptions (i.e. exceptions may have polymorphic type parameters, constraints, etc.)
• hierarchies (i.e. an IOException is a sub-case of Exception and a super-case of NetworkException)
• case coverage (i.e. the compiler can tell you if you forgot a case or sometimes if you wrote useless ones)
• conditional success handlers (i.e. do something with the result in case of success)
• conditional success-and-failure handlers (i.e. “finally”).

To attain this, we replace the mechanism of exceptions by an error monad, we replace exception constructors with polymorphic variants and we introduce a dose of syntactic sugar.

Examples

Expression evaluator

Let’s write a simple expression evaluator for the following set of expressions:

type expr =
  | Value of float
  | Div     of expr * expr
  | Add    of expr * expr
  | Mult   of expr * expr
  | Subs  of expr * expr

These may be evaluated using the following function:

let rec eval = function
 | Value x    -> x
 | Add (x,y) -> eval x +. eval y
 | Mult(x,y)  -> eval x *. eval y
 | Div(x,y)   -> eval x /. eval y
 | Subs(x,y) -> eval x -. eval y

Of course, this function is bound to fail in case of division by zero. While this is expected, there is nothing in the source code — much less in the type of the function — to let us know which exception will be raised in case of division by zero.

An alternative would be to add manual error checking, as follows:

type ('a, 'b) result =
 | Ok of 'a
 | Error of 'b

let rec eval = function
 | Value x    -> OK x
 | Add (x,y) -> (match eval x with
                      | Error e -> Error e
                      | Ok x' ->  match eval y with
                               | Error e -> Error e
                               | Ok y'    -> Ok (x' +. y'))
 | Mult (x,y) -> (match eval x with
                      | Error e -> Error e
                      | Ok x' ->  match eval y with
                               | Error e -> Error e
                               | Ok y'    -> Ok (x' *. y'))
 | Div (x,y) -> (match eval x with
                      | Error e -> Error e
                      | Ok x' ->  match eval y with
                               | Error e -> Error e
                               | Ok y'    -> if y' = 0. then Error "Division by zero"
                                                else              Ok (x' /. y'))
 | Subs (x,y) -> (match eval x with
                      | Error e -> Error e
                      | Ok x' ->  match eval y with
                               | Error e -> Error e
                               | Ok y'    -> Ok (x' -. y'))
(*eval : expr -> (float, string) result*)

After this transformation, the type of the exception appears in the type of eval — here, we used strings, but anything else would have been fine. Of course, the downside is that this is unreadable. Well, what about the following ?

let rec eval = function
 | Value x    -> return x
 | Add (x,y) -> perform with module Error
                        x' <-- eval x;
                        y' <-- eval y;
                        return (x' +. y' 
 | Mult (x,y) -> perform with module Error
                        x' <-- eval x;
                        y' <-- eval y;
                        return (x' *. y' 
 | Div (x,y) -> perform with module Error
                        x' <-- eval x;
                        y' <-- eval y;
                        if y'=0. then throw "Division by zero"
                        else            return (x' /. y' 
 | Subs (x,y) -> perform with module Error
                        x' <-- eval x;
                        y' <-- eval y;
                        return (x' -. y' 
(*eval : expr -> (float, string) result*)

This extract uses [our customized version of] Pa_monad (included in the package), which brings syntactic support for monads. While this is more verbose than the original version, it’s also safer, insofar as we can guarantee that exceptions won’t remain uncaught.

Still too long? Then what about using the appropriate operators?

open Error.Operators
let rec eval = function
 | Value x    -> x
 | Add (x,y) -> eval x +. eval y
 | Mult(x,y) -> eval x *. eval y
 | Div(x,y)   -> eval x /. eval y
 | Subs(x,y) -> eval x -. eval y

Except for the module opening, that’s the same thing as our first listing. Just with the added safety.

Throwing, catching and hierarchies

By the way, the type of the result is

(*eval : expr -> (float, [> `Arithmetic of (unit, [> `Div_by_zero of (unit, _) exc ]) exc ]) result*)

That is, eval may either succeed and return a float or fail and return an arithmetic exception, which also turns out to be a division by zero. That’s classes of exceptions.

With our syntactic sugar, raising such an exception is done by

throw (exception Arithmetic (); Div_by_zero ())

Note that we could have put some content instead of (). Note that exceptions are typed as they appear in the code and don’t need to be declared (if you wonder, polymorphic variants are involved in this).

Of course, various kinds of exceptions may be combined, as in the following extract:

match ... with
| 1 -> throw (exception Arithmetic (); Div_by_zero ())
| 2 -> throw (exception Arithmetic (); Overflow "by gosh !&quot 
| 3 -> throw (exception IO file_descr)
| ...
(*
('a, [> `Arithmetic of (unit, [> `Div_by_zero of (unit, _) exc
                                     |   `Overflow of (string, _)   exc]) exc
     |   `IO of (int, [`> ]) exc ]) result*)
*)

While the type of the expression is difficult to read, catching is easy

attempt ... with
 | val s -> (*success*)
 | Arithmetic (); Div_by_zero () -> (*Division by zero*)
 | Arithmetic (); _                   -> (*Other arithmetic*)
 | IO _                                  -> (*Some IO stuff *)
 | finally _                             -> (*Don't forget to close the door*)

This extract introduces three keywords:

• attempt is our replacement for try
• val is used to pattern-match against the result of a successful evaluation
• finally is used to pattern-match against the final result, whether this result was obtained after a successful evaluation or after an exception was raised and handled.

Unbreaking tail-recursion

The following extract is wrong:

let line_count filename =
  let rec loop file count =
  try
    ignore (input_line file);
    loop file (count + 1)
  with
    End_of_file -> count
  in
    loop (open_file filename) 0

Don’t get me wrong, it will compile and run. The problem is that it’s not tail-recursive. In other words, it will be much slower and much more memory-consuming than if exceptions had been ignored. Why ? Because exception End_of_file may have been raised from the next call to loop, so the recursive call cannot be optimized into a non-recursive call. Of course, exceptions can’t be ignored in this extract, as they are required to determine when to stop reading the file. Now, a simple transformation would make the problem go away :

let line_count filename =
  let rec loop file count =
    let should_continue =
    try
      ignore (input_line file);
      true
    with End_of_file -> false
  in
    if should_continue then loop file (count + 1)
    else                    count
  in
    loop (open_file filename) 0

Well, the transformation is simple, but it’s annoying and hard to read. What’s even more annoying is that it’s quite common. With Catch me if you can, we would rather write the following:

let input_line2 x = Error.legacy input_line x

let line_count filename =
  let rec loop file count =
    attempt input_line2 file with
      | val _ -> loop file (count + 1)
      | _     -> count
  in
  loop (open_file filename) 0

In this extract, legacy is a simple manner of wrapping an existing, one-argument, function and convert it to our new exception style. It’s not quite as good as wrapping the function manually and giving it an actual exception, but it’s better than nothing.

All in all, the resulting function line_count is shorter, easier to read, takes less memory and is also faster than the original.

What about performance?

Now, that’s a complex question. Short answer: there’s an acceptable performance loss. False short answer: we wrote a paper on that subject.

A new version of the exception monad for OCaml is now available for download. It’s now richer, comes with extensive syntactic sugar and a brand new system of compile-time optimizations. More on this in the corresponding research paper — and whenever I find the time, on this blog.

A colleague recently pointed me towards Korset, a program developed by Ohad Ben-Cohen and Avishai Wool promising features comparable to Extrapol. While I must admit I’m slightly skeptical about the promise of “provable zero false alarm” — since the problem is undecidable, usually people tend to develop “provably complete” rather than “provably sound” analysis — it sounds like an interesting development.

Now, from what I understand, Korset will be presented to Blackhat in a few months, and the rules of the conference forbid the developers from giving away any detail. Until then, we have no way of comparing the unreleased Extrapol and the equally unreleased Korset.

Note: the tarball for the first prototype of Extrapol is waiting on my hard-drive for release clearance. I hope I’ll be able to release it next Tuesday or Wednseday. Stay tuned.

]]> http://dutherenverseauborddelatable.wordpress.com/2008/06/19/extrapol-and-korset/feed/ yoric http://dutherenverseauborddelatable.wordpress.com/2008/06/18/firefox-3-est-sorti/ http://dutherenverseauborddelatable.wordpress.com/2008/06/18/firefox-3-est-sorti/#comments Wed, 18 Jun 2008 14:23:48 +0000 yoric http://dutherenverseauborddelatable.wordpress.com/?p=98

Le titre de ce billet devrait se suffire. Pour consulter la liste des améliorations récentes ou télécharger cette nouvelle version, rendez-vous sur getfirefox.com.

A quick note to inform you that the repository for Extrapol is now public. The source code as available on the repository does not have a licence yet and will not compile as such, due to dependencies on libraries available somewhere else. Stay tuned for an actual release.

Update: Sorry, repository cut off by the administrator. I’ll inform you when the sources are back.

Note rapide pour vous informer que le code source d’Extrapol est maintenant disponible au public. Il ne s’agit pas encore d’une version officielle — en particulier, le code n’a pas encore de licence et il manque des bibliothèques (disponibles ailleurs). Plus de détails dès qu’une version officielle est disponible.

Additif: Désolé, je viens d’apprendre que le dépôt de source a été isolé par l’administrateur. Je vous tiendrai au courant dès que le code source est de nouveau public.

Sans commentaires :

Au passage, je pense que ceci fera une bonne entrée pour le jeu permanent “En entendrons-nous parler ?” Ça y est, j’ai fini mon tour. Aux médias.

Suite du jeu “en entendrons-nous parler ?”, commencé dans le billet précédent. Cette fois-ci, la question concerne les méchants employés de la SNCF, qui ne font rien qu’à prendre les usagers en otages, par opposition aux gentils camionneurs, qui eux organisent des opérations escargot.

Aujourd’hui, c’était le début des épreuves écrites du bac, le journal télévisé en a abondamment parlé. C’était aussi le jour d’une opération escargot que les sympathiques camionneurs ont décidé de ne commencer qu’après le début des épreuves. Ce point aussi a été abondamment mentionné par le journal télévisé. Et c’était aussi le jour d’une grève de la SNCF, qui n’a elle-même pas commencé avant 9h du matin, heure à laquelle les candidats étaient normalement tous face à leurs copies.

Donc, pour l’épreuve numéro deux de “En entendrons-nous parler ?”, je propose le sujet : entendrons-nous parler du fait que les employés de la SNCF n’a pas commencé elle non plus avant le début des écrits ?

Question subsidiaire : sachant que le journal télévisé a dit et répété que les camionneurs faisaient grève contre la hausse des prix du carburant (circonstance sur laquelle le gouvernement ne peut pas grand chose), le journal télévisé mentionnera-t-il la raison de la grève de la SNCF ? Pour information, il s’agissait de la défense des 35h, qui elle est liée à la politique du Parti au Pouvoir.