Unbreaking Scalable Web Development, One Loc at a Time

The Opa platform was created to address the problem of developing secure, scalable web applications. Opa is a commercially supported open-source programming language designed for web, concurrency, distribution, scalability and security. We have entered closed beta and the code will be released soon on http://opalang.org, as an Owasp project .

• Edit The video spawned a conversation on Reddit.
• Edit Interesting followup on Hacker News.
• Edit Reworked source code & comments for clarity. Thanks for the feedback.
• EditCome and chat with us on Freenode, channel #opalang .

If you are a true coder, sometimes, you meet a problem so irritating, or a solution so clumsy, that challenging it is a matter of engineering pride. I assume that many of the greatest technologies we have today were born from such challenges, from OpenGL to the web itself. The pain of pure LAMP-based web development begat Ruby on Rails, Django or Node.js, as well as the current NoSQL generation. Similarly, the pains of scalable large system development with raw tools begat Erlang, Map/Reduce or Project Voldemort.

Opa was born from the pains of developing scalable, secure web applications. Because, for all the merits of existing solutions, we just knew that we could do much, much better.

Unsurprisingly, getting there was quite a challenge. Between the initial idea and an actual platform lay blood, sweat and code, many experiments and failed prototypes, but finally, we got there. After years of development and real-scale testing, we are now getting ready to release the result.

The parents are proud to finally introduce Opa.

Different means to different ends

Opa is a new approach to scalable, secure web development.

The core idea behind Opa is that once you use the right paradigm, scalability, security and the web model just happen naturally.

To implement our idea, we had to provide developers with a programming language that was:

1. powerful enough to describe the complete behavior of the web application, including user interface, interactivity, concurrency, general-purpose computations and database manipulation;
2. clean enough to support automated security analysis;
3. high-level enough to support transparent distribution, optimization and injection of security checks;
4. understandable by any developer.

This is not a benign idea. Most approaches to web development, to security or scalability rely either on libraries, external tiers, or reflexivity. For all their merits, and even when applied to the best/most modular/most extensible programming languages available, these techniques are still heavily rooted on whichever paradigm is best handled by that language. Some of the results can be impressive – including your favorite framework, whichever it may be – but in the end, they are necessarily limited by the underlying tools. Unfortunately, we could not find any existing language – whether static, dynamic or hybrid – that could fit all criteria. So, we had to build our own.

This is also not an easy idea for us. We spent years designing, testing, fine-tuning our paradigm, as well as ensuring that the result was indeed usable by any developer.

Opa is a new programming language and its runtime environment

With Opa, write your complete application in just one language, and the compiler will transform it into a self-sufficient executable containing:

• server-side code;
• client-side code (cross-browser JavaScript and HTML, generated automatically from your source code);
• database code (compiled queries for our own NoSQL, scalable database);
• distribution code;
• all the glue to connect everything to everything else;
• security checks at boundaries;
• the HTTP server itself;
• the database engine itself;
• the distribution layers themselves.

Launch this executable locally, or ask it to deploy itself on any number of servers, and your web application is running. Do not deploy or configure a DBMS. Do not deploy or configure a web server. Do not deploy or configure a distributed file system. It just works.

Programming with Opa

Opa may be a new language, but it is quite understandable if you have notions of web development. Let me show you a few simple but complete applications. Should you wish to play with them, I have uploaded the source code of each application on github as AGPL.

Hello, web

First variant: 1 eloc

 server = one_page_server("Hello", -> <>Hello, web!</>)

Fork me on github

Second variant: 2 eloc

server = Server.simple_dispatch(_ ->
  html("Hello", <>Hello, web!</>)
)

Fork me on github

Build & launch:

$ opa hello_web.opa
$ ./hello_web.exe

That’s it. Your application is launched, you can connect with any (recent) browser.

A minimal (distributed, load-balanced) key-value store

Source code, in 17 eloc:

/**
 * Add a path called [/storage] to the schema of our graph database.
 *
 * This path is used to store one value with type
 * [stringmap(option(string))]. A [stringmap] is a dictionary.
 * An [option(string)] is an optional [string],
 * i.e. a value that may either be a string or omitted.
 *
 * This path therefore stores an association from [string]
 * (the key) to either a [string] (the value) or nothing
 * (no value).
 */
db /storage: stringmap(option(string))

/**
 * Handle requests.
 *
 * @param request The uri of the request. The URI is converted to
 * a key in [/storage], the method determines what should be done,
 * and in the case of [{post}] requests, the body is used to set
 * the value in the db
 *
 * @return If the request is rejected, [{method_not_allowed}].
 * If the request is a successful [{get}], a "text/plain" resource
 * with the value previously stored. If the request is a [{get}] to
 * an unknown key, a [{wrong_address}].
 * Otherwise, a [{success}].
 */
dispatch(request) =
(
  key = List.to_string(request.uri.path)
  match request.method with
   | {some = {get}}    ->
     match /storage[key] with
       | {none}        -> Resource.raw_status({wrong_address})
       | {some = value}-> Resource.raw_response(value,
               "text/plain", {success})
     end
   | {some = {post}}   ->
         do /storage[key] <- request.body
         Resource.raw_status({success})
   | {some = {delete}} ->
         do /storage[key]
         do Db.remove(@/storage[key])
         Resource.raw_status({success})
   | _ -> Resource.raw_status({method_not_allowed})
  end
)

/**
 * Main entry point: launching the server.
 */
server = Server.simple_request_dispatch(dispatch)

Fork me on github

Build:

$ opa opa_storage.opa

Launch on one server

$ ./opa_storage.exe

Or auto-deploy and launch on several servers:

$ opa-cloud opa_storage.exe --host localhost --host me@host1 --host me@host2

Again, that’s it. Key/value pairs are replicated/distributed on the various nodes (default settings are generally ok, but replication factor can be configured if necessary), requests are load-balanced and it just works.

Just as importantly, note that we have not written any single line of code for ensuring security with respect to database injection. By construction, Opa ensures automatically that such injections cannot happen.

Real-time web chat

Source code, in 20 eloc:

/**
 * {1 Network infrastructure}
 */

/**
 * The type of messages sent by a client to the chatroom
 */
type message = {author: string /**Arbitrary, untrusted, name*/
              ; text: string  /**Content entered by the user*/}

/**
 * A structure for routing and broadcasting values of type
 * [message].
 *
 * Clients can send values to be broadcasted or register
 * callbacks to be informed of the broadcast. Note that
 * this routing can work cross-client and cross-server.
 *
 * For distribution purposes, this network will be
 * registered to the network as "mushroom".
 */
room = Network.cloud("mushroom"): Network.network(message)

/**
 * {1 User interface}
 */

/**
 * Update the user interface in reaction to reception of a message.
 *
 * This function is meant to be registered with [room] as a callback.
 * Its sole role is to display the new message in [#conversation].
 *
 * @param x The message received from the chatroom
 */
user_update(x) =
(
  line = <div>
     <div>{x.author}:</div>
     <div>{x.text}</div>
  </div>
  do Dom.transform([#conversation +<- line ])
  Dom.scroll_to_bottom(#conversation)
)

/**
 * Broadcast text to the [room].
 *
 * Read the contents of [#entry], clear these contents and send
 * the message to [room].
 *
 * @param author The name of the author. Will be included in the
 * message broadcasted.
 */
broadcast(author) =
  do Network.broadcast({author=author text=Dom.get_value(#entry)}, room)
  Dom.clear_value(#entry)

/**
 * Build the user interface for a client.
 *
 * Pick a random author name which will be used throughout the chat.
 *
 * @return The user interface, ready to be sent by the server to the client
 * on connection.
 */
start() =
(
    author = Random.string(8)
    <div id=#conversation
     onready={_ -> Network.add_callback(user_update, room)}></div>
   <input id=#entry  onnewline={_ -> broadcast(author)}/>
   <div class="button" onclick={_ -> broadcast(author)}>Send!</div>
)

/**
 * {1 Application}
 */

/**
 * Main entry point.
 *
 * Construct an application called "Chat" (users will see the name in the title bar),
 * embedding statically the contents of directory "resources", using the global
 * stylesheet "resources/css.css" and the user interface defined in [start].
 */
server = Server.one_page_bundle("Chat",
    [@static_resource_directory("resources")],
    ["resources/css.css"], start)

Fork me on github

Build and launch as above:

$ opa opa_chat.opa

$ opa-cloud opa_chat.exe --host localhost --host me@host1 --host me@host2

Users connecting to the launch server are load-balanced among servers. Users connecting to one server can chat transparently with users connected to other servers.

Just as importantly, note that we have not written any single line of code for ensuring security with respect to Cross-Site Scripting. Still, you can try and inject code in this application – and you will fail. Opa has transparently ensured that this cannot happen.

Our experience with Opa

We have used Opa to develop a number of web applications, including CMSes, online games, high-security communication tools or e-Commerce apps.

What can I tell you? In our experience, Opa is awesome 🙂 It saves us considerable amounts of time and pain and it vastly extended the size of projects that we could undertake with small agile teams.

Firstly, Opa handles transparently all communications between the client and the server, and can generate JavaScript or server binary code from the same source, depending on what is required. This considerably simplifies prototyping and agile development, by letting us concentrate on getting things to work first, experimenting and showing to clients second, and freezing the design only much later. Countless times, this also made us very much more flexible with respect to design changes, by letting us instantaneously move (or reuse) server code on the client, or in the database, or vice-versa, without having to port from one language to another, or to reimplement communication protocols, or validation, or to redesign for asynchronicity. The added benefit of automated XSS protection also considerably improved our confidence in such agile code.

Secondly, Opa’s paradigm is a natural match for scalability concerns. It favors stateless services, makes sure that state can be easily marked as local (e.g. caches) or shared (e.g. accounts), and it also makes it quite easy to place local caches in front of anything shared. Most of our applications written on one server worked even better on several servers, out-of-the-box. To push scalability even further, marking data as local/shared/cached is extremely simple, which has always helped us experiment quickly, before deciding whether to push such optimizations into production.

On the security side, I’m not sure exactly how many men·months Opa saved us by guaranteeing that we were automatically safe against injections (including XSS and SQL/SQL-like), and I’m not quite sure how to measure it, but this definitely relieved us of plenty of work, stress and emergency calls.

How does this work?

We make it work 🙂

More seriously, last time we counted, including testing, around 100 man·years of R&D had been spent on Opa. We took advantage of that time to make Opa the best solution we could imagine. I’ll try and explain some of the key techniques progressively, in a series of blog entries.

Limitations

We are extremely proud of everything that is possible with Opa, but, as any product, Opa has limitations.

Firstly, while the Opa compiler and runtime can perform very aggressive optimizations on distribution and database requests for instance, for the moment, some of these optimizations cannot be performed automatically. In such cases, a developer needs to annotate the code here and there, to mark code chunks as safe for such optimizations. We have a number of plans to push forward the automation of these optimizations, but we haven’t had a chance to implement them yet.

Other limitations are related to our objectives. Opa is designed for security on the web. Consequently, a number of primitives that are just too dangerous are not accessible for Opa developers, so don’t expect to encounter innerHTML, eval(), document.print() or execvp(), for instance. These primitives are available as part of the platform, should you wish to work on extending the runtime, but not as part of the language/library.

Also, as we dedicated Opa to the web, do not look too hard for Gtk or DirectX bindings – nothing prevent such system bindings, but we have no plans on introducing these ones. Similarly, Opa is designed for scalability, so the language favors stateless programming, or when state is required, as in our web chat, states that can be shared between several instances of a server. So, while Opa will let you write an application with messy state, the design of the language will try and guide you on another way.

We also have a few other limitations, that may be considered anecdotical in this day and age. For instance, the client side of Opa applications that have a client (i.e. non-pure web services) requires JavaScript and will not work with IE6 or Lynx.

Show me the code!

Soon, but not quite yet.

We’re working full-time on the open-source release. If you are interested, I suggest you visit opalang.org to find some information and documentation or to request invitations to the closed beta. You can also follow our updates on Twitter or come and chat with us on IRC.