JavaScript is a dynamic language. However, by borrowing a few pages from static languages – and a few existing tools – we can considerable improve reliability and maintainability.

« Writing one million lines of code of JavaScript is simply impossible »

(source: speaker in a recent open-source conference)

JavaScript is a dynamic language – a very dynamic one, in which programs can rewrite themselves, objects may lose or gain methods through side-effects on themselves on on their prototypes, and, more generally, nothing is fixed.

And dynamic languages are fun. They make writing code simple and fast. They are vastly more suited to prototyping than static languages. Dynamism also makes it possible to write extremely powerful tools that can perform JIT translation from other syntaxes, add missing features to existing classes and functions and more generally fully customize the experience of the developer.

Unfortunately, such dynamism comes with severe drawbacks. Safety-minded developers will tell you that, because of this dynamism, they simply cannot trust any snippet, as this snippet may behave in a manner that does not match its source code. They will conclude that you cannot write safe, or even modular, applications in JavaScript.

Many engineering-minded developers will also tell you that they simply cannot work in JavaScript, and they will not have much difficulty finding examples of situations in which the use of a dynamic language in a complex project can, effectively, kill the project. If you do not believe them, consider a large codebase, and the (rather common) case of a large transversal refactoring, for instance to replace an obsolete API by a newer one. Do this in Java (or, even better, in a more modern mostly-static language such as OCaml, Haskell, F# or Scala), and you can use the compiler to automatically and immediately spot any place where the API has not been updated, and will spot a number of errors that you may have made with the refactoring. Even better, if the API was designed to be safe-by-design, the compiler will automatically spot even complex errors that you may have done during refactoring, including calling functions/methods in the wrong order, or ownership errors. Do the same in JavaScript and, while your code will be written faster, you should expect to be hunting bugs weeks or even months later.

I know that the Python community has considerably suffered from such problems during version transitions. I am less familiar with the world of PHP, but I believe this is no accident that Facebook is progressively arming itself with PHP static analysis tools. I also believe that this is no accident that Google is now introducing a typed language as a candidate replacement for JavaScript.

That is because today is the turn of JavaScript, or if not today, surely tomorrow. I have seen applications consisting in hundreds of thousands of lines of JavaScript. And if just maintaining these applications is not difficult enough, the rapid release cycles of both  Mozilla and Chrome, mean that external and internal APIs are now changing every six weeks. This means breakage. And, more precisely, this means that we need new tools to help us predict breakages and help developers (both add-on developers and browser contributors) react before these breakages hit their users.

So let’s do something about it. Let’s make our JavaScript a strongly, statically typed language!

Or let’s do something a little smarter.

JavaScript, with discipline

At this point, I would like to ask readers to please kindly stop preparing tar and feathers for me. I realize fully that JavaScript is a dynamic language and that turning it into a static language will certainly result in something quite disagreeable to use. Something that is verbose, has lost most of the power of JavaScript, and gained no safety guarantees.

Trust me on this, there is a way to obtain the best of both worlds, without sacrificing anything. Before discussing the manner in which we can attain this, let us first set objectives that we can hope to achieve with a type-disciplined JavaScript.

Finding errors

The main benefit of strong, static typing, is that it helps find errors.

• Even the simplest analyses can find all syntax errors, all unbound variables, all variables bound several times and consequently almost all scoping errors, which can already save considerable time for developers. Such an analysis requires no human intervention from the developer besides, of course, fixing any error that has been thus detected. As a bonus, in most cases, the analysis can suggest fixes.
• Similarly trivial forms of analysis can also detect suspicious calls to break or continue, weird uses of switch(), suspicious calls to private fields of objects, as well as suspicious occurrences of eval – in my book, eval is always suspicious.
• Slightly more sophisticated analyses can find most occurrences of functions or methods invoked with the wrong number of arguments. Again, this is without human intervention. With type annotations/documentation, we can move from most occurrences to all occurrences.
• This same analysis, when applied to public APIs, can provide developers with more informations regarding how their code can be (mis)used.
• At the same level of complexity, analysis can find most erroneous access to fields/methods, suspicious array traversals, suspicious calls to iterators/generators, etc. Again, with type annotations/documentation, we can move from most to all.
• Going a little further in complexity, analysis can find fragile uses of this, uncaught exceptions, etc.

Types as documentation

Public APIs must be documented. This is true in any language, no matter where it stands on the static/dynamic scale. In static languages, one may observe how documentation generation tools insert type information, either from annotations provided by the user (as in Java/JavaDoc) or from type information inferred by the compiler (as in OCaml/OCamlDoc). But look at the documentation of Python, Erlang or JavaScript libraries and you will find the exact same information, either clearly labelled or hidden somewhere in the prose: every single value/function/method comes with a form of type signature, whether formal or informal.

In other words, type information is a critical piece of documentation. If JavaScript developers provide explicit type annotations along with their public APIs, they have simply advanced the documentation, not wasted time. Even better, if such type can be automatically inferred from the source code, this piece of documentation can be automatically written by the type-checker.

Types as QA metric

While disciples of type-checking tend to consider typing as something boolean, the truth is more subtle: it quite possible that one piece of code does not pass type-checking while the rest of the code does. Indeed, with advanced type systems that do not support decidable type inference, this is only to be expected.

The direct consequence is that type-checking can be seen as a spectrum of quality. A code can be seen as failing if the static checking phrase can detect evident errors, typically unbound values or out-of-scope break, continue, etc. Otherwise, every attempt to type a value that results in a type error is a hint of poor QA practice that can be reported to the developer. This yields a percentage of values that can be typed – obtain 100% and get a QA stamp of approval for this specific metric.

Typed JavaScript, in practice

Most of the previous paragraphs are already possible in practice, with existing tools. Indeed, I have personally experienced using JavaScript static type checking as a bug-finding tool and a QA metric. On the first day, this technique has helped me find both plenty of dead code and 750+ errors, with only a dozen false positives.

For this purpose, I have used Google’s Closure Compiler. This tool detects errors, supports a simple vocabulary for documentation/annotations, fails only if very clear errors are detected (typically syntax errors) and provides as metric a percentage of well-typed code. It does not accept JavaScript 1.7 yet, unfortunately, but this can certainly be added.

I also know of existing academic work to provide static type-checking for JavaScript, although I am unsure as to the maturity of such works.

Finally, Mozilla is currently working on a different type inference mechanism for JavaScript. While this mechanism is not primarily aimed at finding errors, my personal intuition is that it may be possible to repurpose it.

What’s next?

I hope that I have convinced you of the interest of investigating manners of introducing static, strong type-checking to JavaScript. In a second part, I will detail how and where I believe that this can be done in Mozilla.

Two years ago, I accepted to join MLstate, to take lead of the R&D group, and turn Opa from a promising early-stage demo into a world-class technology. And I am happy to say that we succeeded. Certainly, there are still many things that we would like to improve in Opa, but looking back on those two years, I am proud of the work we have accomplished, of the number of topics upon which we have pushed forward the state of the art, and even of many of the mistakes we have made, because they have expanded our understanding so much.

Now, after two years at MLstate, I am leaving. Our work is accomplished and I do not feel that I can contribute in any meaningful way to what MLstate has now become, nor that today’s MLstate can keep me excited and interested any longer. In the past few days, Opa has been featured on Lambda the Ultimate, on Hacker News and on Slashdot. Small and large high-tech companies have tried and enjoyed the technology. What better time than this to set sail and say goodbye to these two exciting years of my life?

As of today, I am not the Head of Research & Development, Chief Scientific Officer or Technological Evangelist at MLstate anymore. I will keep a distant eye on Opa, but I will not design or supervise its future versions. Mathieu Baudet, our COO, is replacing me as the supervisor for the development of Opa, while Adam Koprowski is replacing me as Technological Evangelist. Mathieu is a very intelligent security researcher and I am sure that he will impose a new style to the Opa team, and Adam is a bright and enthusiastic researcher/developer, and certainly the best person at MLstate to carry on Opa advocacy.

I would like to thank my University for supporting this foray into the exciting world of start-ups. I would like to thank our CEO for recruiting such a talented team. I would also like to thank Mehdi Ben Soltane, our CFO/HR director, who managed to do his job with a nice and welcome pinch of humor, even in the toughest of times. And mostly, I would like to thank all the R&D team: Maxime Audouin, Mathieu Barbin, Vincent Benayoun, Anthonin Bonnefoy, Raja Boujbel, Quentin Bourgerie, Sébastien Briais, Valentin Gatien-Baron, Louis Gesbert, Nicolas Glondu, Hugo Heuzard, Adrien Jonquet, Mikolaj Konarski, Adam Koprowski, Laurent LeBrun, Sarah Maarek, Grégoire Makridis, François Pessaux, Guillem Rieu, Pascal Rigaux, Norman Scaife, Rudy Sicard, François-Régis Sinot, Cédric Soulas, Quickie Squeaky, Hugo Venturini, Frédéric Ye, and all our successive generations of interns[1]: you are the best team I have ever had the chance to join, it really was an honor and a pleasure working with you all and I hope that those among you who have chosen to remain in MLstate have as much fun working under Mathieu’s leadership as I had working with you all.

Time to set sail! My next missive should arrive from the next port.

[1] Sorry, I do not have the list of interns at hand. But do not worry, I enjoyed working with you, too

See also

• the Hacker News thread spawned by this LtU thread;
• the Slashdot thread spawned by this LtU thread.

Edit Added the Slashdot thread.

Edit Gasp, Slashdot is down. Hey, GeekNet, if you need a scalable programming language for the next version of Slashcode, just ping us

A few days ago, we announced the Opa platform, and I’m happy to announce that things are going very well. We have received numerous applications for the closed preview – we now have onboard people from Mozilla, Google and Twitter, to quote but a few, from many startups, and even from famous defense contractors – and I’d like to start this post by thanking all the applicants. It’s really great to have you guys & gals and your feedback. We are still accepting applications, by the way.

Speaking of feedback, we got plenty of it, too, on just about everything Opa, much of it on the syntax. This focus on syntax is only fair, as syntax is both the first thing a new developer sees of a language and something that they have to live with daily. And feedback on the syntax indicates rather clearly that our syntax, while being extremely concise, was perceived as too exotic by many developers.

Well, we aim to please, so we have spent some time with our testers working on possible syntax revisions, and we have converged on two possible syntaxes. In this post, I will walk you through syntax changes. Please keep in mind that we are very much interested in feedback, so do not hesitate to contact us, either by leaving comments on this blog, by IRC, or at feedback@opalang.org .

Important note: that we will continue supporting the previous syntax for some time and we will provide tools to automatically convert from the previous syntax to the revised syntax.

Let me walk you through syntax changes.

Edit

• Fixed typoes.
• Removed most comments from revised versions, they were redundant.

Hello, web

Original syntax

start() = <>Hello, web!</>
server = one_page_server("Hello", start)

or, equivalently,

server = one_page_server("Hello", -> <>Hello, web!</>)

This application involves the following operations:

• define some HTML content – note that this is actually a data structure, not inline HTML;
• put this content in a either a function called start (first version) or an anonymous function (second version);
• call function one_page_server to build a server;
• use this server as our main server.

Revised syntax, candidate 1

/**
 * The function defining our user interface.
 */
start() {
  <>Hello, web!</> //HTML-like content.
  //As the last value of the function, this is the result.
}

/**
 * Create and start a server delivering user interface [start]
 */
start_server(one_page_server("Hello", start))

Fork me on github

or, equivalently

/**
 * The function defining our user interface.
 */
start = -> <>Hello, web!</> //HTML-like content.
  //Using the syntax for anonymous functions

/**
 * Create and start a server delivering user interface [start]
 */
start_server(one_page_server("Hello", start))

Fork me on github

or, equivalently

start_server(one_page_server("Hello", -> <>Hello, web!</> ));

Fork me on github

Rationale of the redesign

• JS-style {} around function bodies indicate clearly where a function starts and where a function ends, which makes it easier for people who do not know the language to make sense of source code;
• an explicit call to function start_server makes it more discoverable and intelligible that you can define several servers in one application.

Not redesigned

• the HTML-like for user interface – feedback indicates that developers understand it immediately;
• anonymous functions can still be written with -> – this syntax is both lightweight and readable;
• the fact that the last value of a function is its result – now that we have the curly braces, it’s clear, and it fits much better with our programming paradigm than a return that would immediately stop the flow of the function and would not interact too nicely with our concurrency model;
• the syntax of comments – it works as it is.

Revised syntax, candidate 2

/**
 * The function defining our user interface.
 */
def start():
  <>Hello, web!</> //HTML-like content.
  //As the last value of the function, this is the result.

/**
 * Create and start a server delivering user interface [start]
 */
server:
   one_page_server("Hello", start)

Fork me on github

or, equivalently

server:
   one_page_server("Hello", def(): <>Hello, web!</>)

Fork me on github

Redesign and rationale

• Python-style meaningful indents force readable pagination;
• in the second version, Python-inspired anonymous “def” makes it easier to spot anonymous functions and their arguments – note that this is not quite Python “lambda“, as there is no semantic difference between what an anonymous function can do and what a named function can do ;
• Keyword server: is clearer than declaration server = .

Not redesigned

as above

Distributed key-value store

Original syntax

/**
 * Add a path called [/storage] to the schema of our
 * graph database.
 *
 * This path is used to store one value with type
 * [stringmap(option(string))]. A [stringmap] is a dictionary.
 * An [option(string)] is an optional [string],
 * i.e. a value that may either be a string or omitted.
 *
 * This path therefore stores an association from [string]
 * (the key) to either a [string] (the value) or nothing
 * (no value).
 */
db /storage: stringmap(option(string))

This extract adds a path to the database schema and provides the type of the value stored at this path. Note that Opa offers a graph database. Each path contains exactly one value. To store several values at one path, we actually store a container, which integrates nicely into the graph. Here, Opa will detect that what we are storing is essentially a table, and will automatically optimize storage to take advantage of this information.

/**
 * Handle requests.
 *
 * @param request The uri of the request. The URI is converted
 * to a key in [/storage], the method determines what should be
 * done, and in the case of [{post}] requests, the body is used
 * to set the value in the db
 *
 * @return If the request is rejected, [{method_not_allowed}].
 * If the request is a successful [{get}], a "text/plain"
 * resource with the value previously stored. If the request
 * is a [{get}] to an unknown key, a [{wrong_address}].
 * Otherwise, a [{success}].
 */
dispatch(request) =
(
  key = List.to_string(request.uri.path)
  match request.method with
   | {some = {get}}    ->
     match /storage[key] with
       | {none}        -> Resource.raw_status({wrong_address})
       | {some = value}-> Resource.raw_response(value,
               "text/plain", {success})
     end
   | {some = {post}}   ->
         do /storage[key] <- request.body
         Resource.raw_status({success})
   | {some = {delete}} ->
         do Db.remove(@/storage[key])
         Resource.raw_status({success})
   | _ -> Resource.raw_status({method_not_allowed})
  end
)

This extract  inspects the HTTP method of the request to decide what to do with the request – this is called “pattern-matching”. First case handles GET and performs further matching on the database to determine whether the key is already associated to a value.

/**
 * Main entry point: launching the server.
 */
server = Server.simple_request_dispatch(dispatch)

Finally, this extract launches the server.

Revised syntax, candidate 1

Fork me on github

db {
  option<string> /storage[string];
}

or, equivalently,

db option<string> /storage[string];

Redesigns and rationale

• The type of the value appears before the value – this is more understandable by developers used to C-style syntax.
• Syntactic sugar makes it clear that the path is indexed by strings – this syntax matches the syntax used to place requests or to update the value.
• Allowing braces around schema declaration is a good visual clue.
• We now use <> for generics syntax – again, this matches the syntax of C++, Java, C# and statically typed JS extensions.

Not redesigned

• Keyword db – we need a keyword to make it clear that we are talking about the database.

dispatch(request) {
  key = List.to_string(request.uri.path);
  match(request.method) {
    case {some: {get}}:
        match(/storage[key]) {
           case {none}:  Resource.raw_status({wrong_address});
           case {some: value}: Resource.raw_response(value,
              "text/plain", {success});
        }
    case {some: {post}}: {
         /storage[key] <- request.body;
         Resource.raw_status({success})
    }
    case {some: {delete}}: {
         Db.remove(@/storage[key]);
         Resource.raw_status({success});
    }
    case *: Resource.raw_status({method_not_allowed});
  }
}

Redesigns and rationale

• Pattern-matching syntax  becomes  match(…) { case case_1: …; case case_2: …; … } – this syntax resembles that of switch(), and is therefore more understandable by developers who are not accustomed to pattern-matching. Note that pattern-matching is both more powerful than switch and has a different branching mechanism, so reusing keywords switch and default would have mislead developers.
• Records now use : instead of =, as in JavaScript – now that we use curly braces, this is necessary to ensure that there is a visual difference between blocks and structures.

Not redesigned

• Operator <- for updating a database path – we want developers to be aware that this operation is very different from =, which serves to define new values.
• The syntax for paths – it’s simple, concise and it’s an immediate cue that we are dealing with a persistent value.

start_server(Server.simple_request_dispatch(dispatch))

No additional redesigns or rationales.

Revised syntax, candidate 2

Fork me on github

db:
  /storage[string] as option(string)

Redesigns and rationale

• Again, Python-style meaningful indents force readable pagination;
• syntactic sugar makes it clear that the path is indexed by strings – this syntax matches the syntax used to place requests or to update the value;
• keyword as (inspired by Boo) replaces : (which is used pervasively in Python syntax);
• python-style keyword db: is more visible than db.

Not redesigned

• We still use parentheses for generic types – no need to clutter the syntax with Java-like Foo<Bar>

def dispatch(request):
  key = List.to_string(request.uri.path)
  match request.method:
    case {some = {get}}:
       match /storage[key]:
          case {none}:        Resource.raw_status({wrong_address})
          case {some: value}: Resource.raw_response(value,
                "text/plain", {success});
    case {some = {post}}:
         /storage[key] <- request.body
         Resource.raw_status({success})
    case {some = {delete}}:
         Db.remove(@/storage[key])
         Resource.raw_status({success})
    case *:
         Resource.raw_status({method_not_allowed})

Redesigns and rationale

• Pattern-matching syntax  becomes  match: and case …: … .

Not redesigned

As above

server:
   Server.simple_request_dispatch(dispatch)

No further redesign.

Web chat

Original syntax

/**
 * {1 Network infrastructure}
 */

/**
 * The type of messages sent by a client to the chatroom
 */
type message = {author: string /**Arbitrary, untrusted, name*/
              ; text: string  /**Content entered by the user*/}

/**
 * A structure for routing and broadcasting values of type
 * [message].
 *
 * Clients can send values to be broadcasted or register
 * callbacks to be informed of the broadcast. Note that
 * this routing can work cross-client and cross-server.
 *
 * For distribution purposes, this network will be
 * registered to the network as "mushroom".
 */
room = Network.cloud("mushroom"): Network.network(message)

In this extract, we define a type and the distribution infrastructure to broadcast value changes between servers or between clients and servers. Needless to say, these two lines hide some very powerful core concepts of Opa.

/**
 * Update the user interface in reaction to reception
 * of a message.
 *
 * This function is meant to be registered with [room]
 * as a callback. Its sole role is to display the new message
 * in [#conversation].
 *
 * @param x The message received from the chatroom
 */
user_update(x) =
(
  line = <div>
     <div>{x.author}:</div>
     <div>{x.text}</div>
  </div>
  do Dom.transform([#conversation +<- line ])
  Dom.scroll_to_bottom(#conversation)
)

/**
 * Broadcast text to the [room].
 *
 * Read the contents of [#entry], clear these contents and send
 * the message to [room].
 *
 * @param author The name of the author. Will be included in the
 * message broadcasted.
 */
broadcast(author) =
  do Network.broadcast({author=author
     text=Dom.get_value(#entry)}, room)
  Dom.clear_value(#entry)

/**
 * Build the user interface for a client.
 *
 * Pick a random author name which will be used throughout
 * the chat.
 *
 * @return The user interface, ready to be sent by the server to
 * the client on connection.
 */
start() =
(
    author = Random.string(8)
    <div id=#conversation
     onready={_ -> Network.add_callback(user_update, room)}></div>
    <input id=#entry  onnewline={_ -> broadcast(author)}/>
    <div onclick={_ -> broadcast(author)}>Send!</div>
)

In this extract, we define the user interface and connect it to the aforementioned distribution mechanism. Again, we describe the user interface as a datastructure in a HTML-like syntax.

/**
 * Main entry point.
 *
 * Construct an application called "Chat" (users
 * will see the name in the title bar), embedding
 * statically the contents of directory "resources",
 * using the global stylesheet "resources/css.css"
 * and the user interface defined in [start].
 */
server = Server.one_page_bundle("Chat",
    [@static_resource_directory("resources")],
    ["resources/css.css"], start)

Finally, as usual, we define our main entry point, with our user interface and a bundle of resources.

Revised syntax, candidate 1

Fork me on github

type message = {author: string /**Arbitrary, untrusted, name*/
               ,text:   string} /**Content entered by the user*/

Network.network<message> room = Network.cloud("mushroom")

user_update(x) {
  line = <div>
     <div>{x.author}:</div>
     <div>{x.text}</div>
  </div>;
  Dom.transform([#conversation +<- line ]);//Note: If we want to change the syntax of actions, now is the right time
  Dom.scroll_to_bottom(#conversation)
}

broadcast(author){
  Network.broadcast({author=author,
    text:Dom.get_value(#entry)}, room);
  Dom.clear_value(#entry)
}

start() {
  author = Random.string(8);
  <div id=#conversation
    onready={ * -> Network.add_callback(user_update, room) }></div>
  <input id=#entry  onnewline={ * -> broadcast(author) }/>
  <div onclick={ * -> broadcast(author) }>Send!</div>
}

start_server(Server.one_page_bundle("Chat",
   [@static_resource_directory("resources")],
   ["resources/css.css"], start))

Revised syntax, candidate 2

Fork me on github

type message:
   author as string //Arbitrary, untrusted, name
   text   as string   //Content entered by the user

room = Network.cloud("mushroom") as Network.network(message)

Redesign and rationale

• We introduce a Python-ish/Boo-ish syntax for defining types.
• Again, we use as instead of : for type annotations.

def user_update(x):
  line = <div>
    <div>{x.author}:</div>
    <div>{x.text}</div>
  </div>
  Dom.transform([#conversation +<- line ])//Note: If we want to change the syntax of actions, now is the right time
  Dom.scroll_to_bottom(#conversation)

def broadcast(author):
   message = new:
      author: author
      text:   Dom.get_value(#entry)
   Network.broadcast(message, room)
   Dom.clear_value(#entry)

Redesign and rationale

• We introduce a new keyword new: to define immediate records – we find this both clearer than the Python syntax for defining objects as dictionaries, and more suited to both our paradigm and our automated analysis.

def start():
   author = Random.string(8)
   html:
    <div id=#conversation onready={def *: Network.add_callback(user_update, room)}></div>
    <input id=$entry onnewline={def *: broadcast(author)}/>
    <div onclick={def *: broadcast(author)}>Send!</div>

Redesign and rationale

• We introduce keyword html: to introduce a block of HTML-like notations. A similar keyword xml: will be used when producing XML documents with a XML-like notation.

server:
  Server.one_page_bundle("Chat",
    [@static_resource_directory("resources")],
    ["resources/css.css"], start)

No additional change here.

What now?

At this stage, we have not switched syntax yet and we have the following options:

• keep our current syntax;
• adopt revised syntax 1, or a variant thereof – so, start coding a conversion tool, the new syntax itself, and start updating the documentation;
• adopt revised syntax 2, or a variant thereof – so, start coding a conversion tool, the new syntax itself, and start updating the documentation;

Yes, I mention variants, because I am certain that many among you will have interesting ideas. So please feel free to express yourselves.

You can provide feedback:

• on this blog;
• by e-mail, at feedback@opalang.org;
• on IRC.

Remember, we are still accepting applications for the preview.

• OpaChat – simple real-time web chat (works)
• OpaStorage – simple distributed key/value store (works)
• opaCAS – single sign-on (in progress)
• Contre-Jour – thumbnail viewer (works)
• OpaTetris – I’m sure you can guess what it’s about – based on HTML5 canvas (works)

Know of any other open-source Opa app? Then let me know!

• Edit The video spawned a conversation on Reddit.
• Edit Interesting followup on Hacker News.
• Edit Reworked source code & comments for clarity. Thanks for the feedback.
• EditCome and chat with us on Freenode, channel #opalang .

If you are a true coder, sometimes, you meet a problem so irritating, or a solution so clumsy, that challenging it is a matter of engineering pride. I assume that many of the greatest technologies we have today were born from such challenges, from OpenGL to the web itself. The pain of pure LAMP-based web development begat Ruby on Rails, Django or Node.js, as well as the current NoSQL generation. Similarly, the pains of scalable large system development with raw tools begat Erlang, Map/Reduce or Project Voldemort.

Opa was born from the pains of developing scalable, secure web applications. Because, for all the merits of existing solutions, we just knew that we could do much, much better.

Unsurprisingly, getting there was quite a challenge. Between the initial idea and an actual platform lay blood, sweat and code, many experiments and failed prototypes, but finally, we got there. After years of development and real-scale testing, we are now getting ready to release the result.

The parents are proud to finally introduce Opa.

Different means to different ends

Opa is a new approach to scalable, secure web development.

The core idea behind Opa is that once you use the right paradigm, scalability, security and the web model just happen naturally.

To implement our idea, we had to provide developers with a programming language that was:

1. powerful enough to describe the complete behavior of the web application, including user interface, interactivity, concurrency, general-purpose computations and database manipulation;
2. clean enough to support automated security analysis;
3. high-level enough to support transparent distribution, optimization and injection of security checks;
4. understandable by any developer.

This is not a benign idea. Most approaches to web development, to security or scalability rely either on libraries, external tiers, or reflexivity. For all their merits, and even when applied to the best/most modular/most extensible programming languages available, these techniques are still heavily rooted on whichever paradigm is best handled by that language. Some of the results can be impressive – including your favorite framework, whichever it may be – but in the end, they are necessarily limited by the underlying tools. Unfortunately, we could not find any existing language – whether static, dynamic or hybrid – that could fit all criteria. So, we had to build our own.

This is also not an easy idea for us. We spent years designing, testing, fine-tuning our paradigm, as well as ensuring that the result was indeed usable by any developer.

Opa is a new programming language and its runtime environment

With Opa, write your complete application in just one language, and the compiler will transform it into a self-sufficient executable containing:

• server-side code;
• client-side code (cross-browser JavaScript and HTML, generated automatically from your source code);
• database code (compiled queries for our own NoSQL, scalable database);
• distribution code;
• all the glue to connect everything to everything else;
• security checks at boundaries;
• the HTTP server itself;
• the database engine itself;
• the distribution layers themselves.

Launch this executable locally, or ask it to deploy itself on any number of servers, and your web application is running. Do not deploy or configure a DBMS. Do not deploy or configure a web server. Do not deploy or configure a distributed file system. It just works.

Programming with Opa

Opa may be a new language, but it is quite understandable if you have notions of web development. Let me show you a few simple but complete applications. Should you wish to play with them, I have uploaded the source code of each application on github as AGPL.

Hello, web

First variant: 1 eloc

 server = one_page_server("Hello", -> <>Hello, web!</>)

Second variant: 2 eloc

server = Server.simple_dispatch(_ ->
  html("Hello", <>Hello, web!</>)
)

Build & launch:

$ opa hello_web.opa
$ ./hello_web.exe

That’s it. Your application is launched, you can connect with any (recent) browser.

A minimal (distributed, load-balanced) key-value store

Source code, in 17 eloc:

/**
 * Add a path called [/storage] to the schema of our graph database.
 *
 * This path is used to store one value with type
 * [stringmap(option(string))]. A [stringmap] is a dictionary.
 * An [option(string)] is an optional [string],
 * i.e. a value that may either be a string or omitted.
 *
 * This path therefore stores an association from [string]
 * (the key) to either a [string] (the value) or nothing
 * (no value).
 */
db /storage: stringmap(option(string))

/**
 * Handle requests.
 *
 * @param request The uri of the request. The URI is converted to
 * a key in [/storage], the method determines what should be done,
 * and in the case of [{post}] requests, the body is used to set
 * the value in the db
 *
 * @return If the request is rejected, [{method_not_allowed}].
 * If the request is a successful [{get}], a "text/plain" resource
 * with the value previously stored. If the request is a [{get}] to
 * an unknown key, a [{wrong_address}].
 * Otherwise, a [{success}].
 */
dispatch(request) =
(
  key = List.to_string(request.uri.path)
  match request.method with
   | {some = {get}}    ->
     match /storage[key] with
       | {none}        -> Resource.raw_status({wrong_address})
       | {some = value}-> Resource.raw_response(value,
               "text/plain", {success})
     end
   | {some = {post}}   ->
         do /storage[key] <- request.body
         Resource.raw_status({success})
   | {some = {delete}} ->
         do /storage[key]
         do Db.remove(@/storage[key])
         Resource.raw_status({success})
   | _ -> Resource.raw_status({method_not_allowed})
  end
)

/**
 * Main entry point: launching the server.
 */
server = Server.simple_request_dispatch(dispatch)

Build:

$ opa opa_storage.opa

Launch on one server

$ ./opa_storage.exe

Or auto-deploy and launch on several servers:

$ opa-cloud opa_storage.exe --host localhost --host me@host1 --host me@host2

Again, that’s it. Key/value pairs are replicated/distributed on the various nodes (default settings are generally ok, but replication factor can be configured if necessary), requests are load-balanced and it just works.

Just as importantly, note that we have not written any single line of code for ensuring security with respect to database injection. By construction, Opa ensures automatically that such injections cannot happen.

Real-time web chat

Source code, in 20 eloc:

/**
 * {1 Network infrastructure}
 */

/**
 * The type of messages sent by a client to the chatroom
 */
type message = {author: string /**Arbitrary, untrusted, name*/
              ; text: string  /**Content entered by the user*/}

/**
 * A structure for routing and broadcasting values of type
 * [message].
 *
 * Clients can send values to be broadcasted or register
 * callbacks to be informed of the broadcast. Note that
 * this routing can work cross-client and cross-server.
 *
 * For distribution purposes, this network will be
 * registered to the network as "mushroom".
 */
room = Network.cloud("mushroom"): Network.network(message)

/**
 * {1 User interface}
 */

/**
 * Update the user interface in reaction to reception of a message.
 *
 * This function is meant to be registered with [room] as a callback.
 * Its sole role is to display the new message in [#conversation].
 *
 * @param x The message received from the chatroom
 */
user_update(x) =
(
  line = <div>
     <div>{x.author}:</div>
     <div>{x.text}</div>
  </div>
  do Dom.transform([#conversation +<- line ])
  Dom.scroll_to_bottom(#conversation)
)

/**
 * Broadcast text to the [room].
 *
 * Read the contents of [#entry], clear these contents and send
 * the message to [room].
 *
 * @param author The name of the author. Will be included in the
 * message broadcasted.
 */
broadcast(author) =
  do Network.broadcast({author=author text=Dom.get_value(#entry)}, room)
  Dom.clear_value(#entry)

/**
 * Build the user interface for a client.
 *
 * Pick a random author name which will be used throughout the chat.
 *
 * @return The user interface, ready to be sent by the server to the client
 * on connection.
 */
start() =
(
    author = Random.string(8)
    <div id=#conversation
     onready={_ -> Network.add_callback(user_update, room)}></div>
   <input id=#entry  onnewline={_ -> broadcast(author)}/>
   <div class="button" onclick={_ -> broadcast(author)}>Send!</div>
)

/**
 * {1 Application}
 */

/**
 * Main entry point.
 *
 * Construct an application called "Chat" (users will see the name in the title bar),
 * embedding statically the contents of directory "resources", using the global
 * stylesheet "resources/css.css" and the user interface defined in [start].
 */
server = Server.one_page_bundle("Chat",
    [@static_resource_directory("resources")],
    ["resources/css.css"], start)

Fork me on github

Build and launch as above:

$ opa opa_chat.opa

$ opa-cloud opa_chat.exe --host localhost --host me@host1 --host me@host2

Users connecting to the launch server are load-balanced among servers. Users connecting to one server can chat transparently with users connected to other servers.

Just as importantly, note that we have not written any single line of code for ensuring security with respect to Cross-Site Scripting. Still, you can try and inject code in this application – and you will fail. Opa has transparently ensured that this cannot happen.

Our experience with Opa

We have used Opa to develop a number of web applications, including CMSes, online games, high-security communication tools or e-Commerce apps.

What can I tell you? In our experience, Opa is awesome It saves us considerable amounts of time and pain and it vastly extended the size of projects that we could undertake with small agile teams.

Firstly, Opa handles transparently all communications between the client and the server, and can generate JavaScript or server binary code from the same source, depending on what is required. This considerably simplifies prototyping and agile development, by letting us concentrate on getting things to work first, experimenting and showing to clients second, and freezing the design only much later. Countless times, this also made us very much more flexible with respect to design changes, by letting us instantaneously move (or reuse) server code on the client, or in the database, or vice-versa, without having to port from one language to another, or to reimplement communication protocols, or validation, or to redesign for asynchronicity. The added benefit of automated XSS protection also considerably improved our confidence in such agile code.

Secondly, Opa’s paradigm is a natural match for scalability concerns. It favors stateless services, makes sure that state can be easily marked as local (e.g. caches) or shared (e.g. accounts), and it also makes it quite easy to place local caches in front of anything shared. Most of our applications written on one server worked even better on several servers, out-of-the-box. To push scalability even further, marking data as local/shared/cached is extremely simple, which has always helped us experiment quickly, before deciding whether to push such optimizations into production.

On the security side, I’m not sure exactly how many men·months Opa saved us by guaranteeing that we were automatically safe against injections (including XSS and SQL/SQL-like), and I’m not quite sure how to measure it, but this definitely relieved us of plenty of work, stress and emergency calls.

How does this work?

We make it work

More seriously, last time we counted, including testing, around 100 man·years of R&D had been spent on Opa. We took advantage of that time to make Opa the best solution we could imagine. I’ll try and explain some of the key techniques progressively, in a series of blog entries.

Limitations

We are extremely proud of everything that is possible with Opa, but, as any product, Opa has limitations.

Firstly, while the Opa compiler and runtime can perform very aggressive optimizations on distribution and database requests for instance, for the moment, some of these optimizations cannot be performed automatically. In such cases, a developer needs to annotate the code here and there, to mark code chunks as safe for such optimizations. We have a number of plans to push forward the automation of these optimizations, but we haven’t had a chance to implement them yet.

Other limitations are related to our objectives. Opa is designed for security on the web. Consequently, a number of primitives that are just too dangerous are not accessible for Opa developers, so don’t expect to encounter innerHTML, eval(), document.print() or execvp(), for instance. These primitives are available as part of the platform, should you wish to work on extending the runtime, but not as part of the language/library.

Also, as we dedicated Opa to the web, do not look too hard for Gtk or DirectX bindings – nothing prevent such system bindings, but we have no plans on introducing these ones. Similarly, Opa is designed for scalability, so the language favors stateless programming, or when state is required, as in our web chat, states that can be shared between several instances of a server. So, while Opa will let you write an application with messy state, the design of the language will try and guide you on another way.

We also have a few other limitations, that may be considered anecdotical in this day and age. For instance, the client side of Opa applications that have a client (i.e. non-pure web services) requires JavaScript and will not work with IE6 or Lynx.

Show me the code!

Soon, but not quite yet.

We’re working full-time on the open-source release. If you are interested, I suggest you visit opalang.org to find some information and documentation or to request invitations to the closed beta. You can also follow our updates on Twitter or come and chat with us on IRC.

The welcome was great, with plenty of people interested in OPA — some of them actually looking enthusiastic. I was quite surprised to realize that a number of researchers, developers and consultants in the web security community are very much aware of the limitations of current-generation approaches to security, but just don’t have the resources to start working on a next-generation approach. Speaking of resources, we’re now getting close to being 7 years into the OPA project, a commitment that not many research groups or companies could make.

Interestingly, during his talk, Dave Wichers, the editor for the OWASP Top 10 Web Application Security Risks project, mentioned that the solution was certainly to switch language and paradigm, to something cleaner and easier to secure. This is, of course, exactly what we have been working on during all these years.

All the slides and videos of the conference should be uploaded soon on the official website. In the meantime, I have uploaded my slides. I’ll try and add some sound if I can work out some sound problems I’ve been encountering recently with my presentations.

Edit The presentation of OPA available on Dailymotion had sound issues. I’ve finally managed to fix them. Enjoy!

Si vous êtes informaticien de haut niveau, inspiré et ingénieux, si vous êtes doté d’une forte culture informatique et scientifique, d’une grande connaissance des langages fonctionnels et impératifs, de la compilation, des systèmes de types, contactez-nous. Le candidat idéal, docteur ou non, avec ou sans expérience industrielle, aura aussi des connaissances en distribution, parallélisme, bases de données, sera capable d’évoluer dans un environnement polyglotte et disposera de la finesse nécessaire pour construire des produits finis.

Les problèmes à résoudre sont difficiles. Pour relever le défi, contactez-nous à careers@mlstate.com .