Crowdsourcing the syntax

Feedback from Opa testers suggests that we can improve the syntax and make it easier for developers new to Opa to read and write code. We have spent some time both inside the Opa team and with the testers designing two possible revisions to the syntax. Feedback on both possible revisions, as well as alternative ideas, are welcome.

A few days ago, we announced the Opa platform, and I’m happy to announce that things are going very well. We have received numerous applications for the closed preview – we now have onboard people from Mozilla, Google and Twitter, to quote but a few, from many startups, and even from famous defense contractors – and I’d like to start this post by thanking all the applicants. It’s really great to have you guys & gals and your feedback. We are still accepting applications, by the way.

Speaking of feedback, we got plenty of it, too, on just about everything Opa, much of it on the syntax. This focus on syntax is only fair, as syntax is both the first thing a new developer sees of a language and something that they have to live with daily. And feedback on the syntax indicates rather clearly that our syntax, while being extremely concise, was perceived as too exotic by many developers.

Well, we aim to please, so we have spent some time with our testers working on possible syntax revisions, and we have converged on two possible syntaxes. In this post, I will walk you through syntax changes. Please keep in mind that we are very much interested in feedback, so do not hesitate to contact us, either by leaving comments on this blog, by IRC, or at feedback@opalang.org .

Important note: that we will continue supporting the previous syntax for some time and we will provide tools to automatically convert from the previous syntax to the revised syntax.

Let me walk you through syntax changes.

Edit

• Fixed typoes.
• Removed most comments from revised versions, they were redundant.

Hello, web

Original syntax

start() = <>Hello, web!</>
server = one_page_server("Hello", start)

or, equivalently,

server = one_page_server("Hello", -> <>Hello, web!</>)

This application involves the following operations:

• define some HTML content – note that this is actually a data structure, not inline HTML;
• put this content in a either a function called start (first version) or an anonymous function (second version);
• call function one_page_server to build a server;
• use this server as our main server.

Revised syntax, candidate 1

/**
 * The function defining our user interface.
 */
start() {
  <>Hello, web!</> //HTML-like content.
  //As the last value of the function, this is the result.
}

/**
 * Create and start a server delivering user interface [start]
 */
start_server(one_page_server("Hello", start))

Fork me on github

or, equivalently

/**
 * The function defining our user interface.
 */
start = -> <>Hello, web!</> //HTML-like content.
  //Using the syntax for anonymous functions

/**
 * Create and start a server delivering user interface [start]
 */
start_server(one_page_server("Hello", start))

Fork me on github

or, equivalently

start_server(one_page_server("Hello", -> <>Hello, web!</> ));

Fork me on github

Rationale of the redesign

• JS-style {} around function bodies indicate clearly where a function starts and where a function ends, which makes it easier for people who do not know the language to make sense of source code;
• an explicit call to function start_server makes it more discoverable and intelligible that you can define several servers in one application.

Not redesigned

• the HTML-like for user interface – feedback indicates that developers understand it immediately;
• anonymous functions can still be written with -> – this syntax is both lightweight and readable;
• the fact that the last value of a function is its result – now that we have the curly braces, it’s clear, and it fits much better with our programming paradigm than a return that would immediately stop the flow of the function and would not interact too nicely with our concurrency model;
• the syntax of comments – it works as it is.

Revised syntax, candidate 2

/**
 * The function defining our user interface.
 */
def start():
  <>Hello, web!</> //HTML-like content.
  //As the last value of the function, this is the result.

/**
 * Create and start a server delivering user interface [start]
 */
server:
   one_page_server("Hello", start)

Fork me on github

or, equivalently

server:
   one_page_server("Hello", def(): <>Hello, web!</>)

Fork me on github

Redesign and rationale

• Python-style meaningful indents force readable pagination;
• in the second version, Python-inspired anonymous “def” makes it easier to spot anonymous functions and their arguments – note that this is not quite Python “lambda“, as there is no semantic difference between what an anonymous function can do and what a named function can do ;
• Keyword server: is clearer than declaration server = .

Not redesigned

as above

Distributed key-value store

Original syntax

/**
 * Add a path called [/storage] to the schema of our
 * graph database.
 *
 * This path is used to store one value with type
 * [stringmap(option(string))]. A [stringmap] is a dictionary.
 * An [option(string)] is an optional [string],
 * i.e. a value that may either be a string or omitted.
 *
 * This path therefore stores an association from [string]
 * (the key) to either a [string] (the value) or nothing
 * (no value).
 */
db /storage: stringmap(option(string))

This extract adds a path to the database schema and provides the type of the value stored at this path. Note that Opa offers a graph database. Each path contains exactly one value. To store several values at one path, we actually store a container, which integrates nicely into the graph. Here, Opa will detect that what we are storing is essentially a table, and will automatically optimize storage to take advantage of this information.

/**
 * Handle requests.
 *
 * @param request The uri of the request. The URI is converted
 * to a key in [/storage], the method determines what should be
 * done, and in the case of [{post}] requests, the body is used
 * to set the value in the db
 *
 * @return If the request is rejected, [{method_not_allowed}].
 * If the request is a successful [{get}], a "text/plain"
 * resource with the value previously stored. If the request
 * is a [{get}] to an unknown key, a [{wrong_address}].
 * Otherwise, a [{success}].
 */
dispatch(request) =
(
  key = List.to_string(request.uri.path)
  match request.method with
   | {some = {get}}    ->
     match /storage[key] with
       | {none}        -> Resource.raw_status({wrong_address})
       | {some = value}-> Resource.raw_response(value,
               "text/plain", {success})
     end
   | {some = {post}}   ->
         do /storage[key] <- request.body
         Resource.raw_status({success})
   | {some = {delete}} ->
         do Db.remove(@/storage[key])
         Resource.raw_status({success})
   | _ -> Resource.raw_status({method_not_allowed})
  end
)

This extract  inspects the HTTP method of the request to decide what to do with the request – this is called “pattern-matching”. First case handles GET and performs further matching on the database to determine whether the key is already associated to a value.

/**
 * Main entry point: launching the server.
 */
server = Server.simple_request_dispatch(dispatch)

Finally, this extract launches the server.

Revised syntax, candidate 1

Fork me on github

db {
  option<string> /storage[string];
}

or, equivalently,

db option<string> /storage[string];

Redesigns and rationale

• The type of the value appears before the value – this is more understandable by developers used to C-style syntax.
• Syntactic sugar makes it clear that the path is indexed by strings – this syntax matches the syntax used to place requests or to update the value.
• Allowing braces around schema declaration is a good visual clue.
• We now use <> for generics syntax – again, this matches the syntax of C++, Java, C# and statically typed JS extensions.

Not redesigned

• Keyword db – we need a keyword to make it clear that we are talking about the database.

dispatch(request) {
  key = List.to_string(request.uri.path);
  match(request.method) {
    case {some: {get}}:
        match(/storage[key]) {
           case {none}:  Resource.raw_status({wrong_address});
           case {some: value}: Resource.raw_response(value,
              "text/plain", {success});
        }
    case {some: {post}}: {
         /storage[key] <- request.body;
         Resource.raw_status({success})
    }
    case {some: {delete}}: {
         Db.remove(@/storage[key]);
         Resource.raw_status({success});
    }
    case *: Resource.raw_status({method_not_allowed});
  }
}

Redesigns and rationale

• Pattern-matching syntax  becomes  match(…) { case case_1: …; case case_2: …; … } – this syntax resembles that of switch(), and is therefore more understandable by developers who are not accustomed to pattern-matching. Note that pattern-matching is both more powerful than switch and has a different branching mechanism, so reusing keywords switch and default would have mislead developers.
• Records now use : instead of =, as in JavaScript – now that we use curly braces, this is necessary to ensure that there is a visual difference between blocks and structures.

Not redesigned

• Operator <- for updating a database path – we want developers to be aware that this operation is very different from =, which serves to define new values.
• The syntax for paths – it’s simple, concise and it’s an immediate cue that we are dealing with a persistent value.

start_server(Server.simple_request_dispatch(dispatch))

No additional redesigns or rationales.

Revised syntax, candidate 2

Fork me on github

db:
  /storage[string] as option(string)

Redesigns and rationale

• Again, Python-style meaningful indents force readable pagination;
• syntactic sugar makes it clear that the path is indexed by strings – this syntax matches the syntax used to place requests or to update the value;
• keyword as (inspired by Boo) replaces : (which is used pervasively in Python syntax);
• python-style keyword db: is more visible than db.

Not redesigned

• We still use parentheses for generic types – no need to clutter the syntax with Java-like Foo<Bar>

def dispatch(request):
  key = List.to_string(request.uri.path)
  match request.method:
    case {some = {get}}:
       match /storage[key]:
          case {none}:        Resource.raw_status({wrong_address})
          case {some: value}: Resource.raw_response(value,
                "text/plain", {success});
    case {some = {post}}:
         /storage[key] <- request.body
         Resource.raw_status({success})
    case {some = {delete}}:
         Db.remove(@/storage[key])
         Resource.raw_status({success})
    case *:
         Resource.raw_status({method_not_allowed})

Redesigns and rationale

• Pattern-matching syntax  becomes  match: and case …: … .

Not redesigned

As above

server:
   Server.simple_request_dispatch(dispatch)

No further redesign.

Web chat

Original syntax

/**
 * {1 Network infrastructure}
 */

/**
 * The type of messages sent by a client to the chatroom
 */
type message = {author: string /**Arbitrary, untrusted, name*/
              ; text: string  /**Content entered by the user*/}

/**
 * A structure for routing and broadcasting values of type
 * [message].
 *
 * Clients can send values to be broadcasted or register
 * callbacks to be informed of the broadcast. Note that
 * this routing can work cross-client and cross-server.
 *
 * For distribution purposes, this network will be
 * registered to the network as "mushroom".
 */
room = Network.cloud("mushroom"): Network.network(message)

In this extract, we define a type and the distribution infrastructure to broadcast value changes between servers or between clients and servers. Needless to say, these two lines hide some very powerful core concepts of Opa.

/**
 * Update the user interface in reaction to reception
 * of a message.
 *
 * This function is meant to be registered with [room]
 * as a callback. Its sole role is to display the new message
 * in [#conversation].
 *
 * @param x The message received from the chatroom
 */
user_update(x) =
(
  line = <div>
     <div>{x.author}:</div>
     <div>{x.text}</div>
  </div>
  do Dom.transform([#conversation +<- line ])
  Dom.scroll_to_bottom(#conversation)
)

/**
 * Broadcast text to the [room].
 *
 * Read the contents of [#entry], clear these contents and send
 * the message to [room].
 *
 * @param author The name of the author. Will be included in the
 * message broadcasted.
 */
broadcast(author) =
  do Network.broadcast({author=author
     text=Dom.get_value(#entry)}, room)
  Dom.clear_value(#entry)

/**
 * Build the user interface for a client.
 *
 * Pick a random author name which will be used throughout
 * the chat.
 *
 * @return The user interface, ready to be sent by the server to
 * the client on connection.
 */
start() =
(
    author = Random.string(8)
    <div id=#conversation
     onready={_ -> Network.add_callback(user_update, room)}></div>
    <input id=#entry  onnewline={_ -> broadcast(author)}/>
    <div onclick={_ -> broadcast(author)}>Send!</div>
)

In this extract, we define the user interface and connect it to the aforementioned distribution mechanism. Again, we describe the user interface as a datastructure in a HTML-like syntax.

/**
 * Main entry point.
 *
 * Construct an application called "Chat" (users
 * will see the name in the title bar), embedding
 * statically the contents of directory "resources",
 * using the global stylesheet "resources/css.css"
 * and the user interface defined in [start].
 */
server = Server.one_page_bundle("Chat",
    [@static_resource_directory("resources")],
    ["resources/css.css"], start)

Finally, as usual, we define our main entry point, with our user interface and a bundle of resources.

Revised syntax, candidate 1

Fork me on github

type message = {author: string /**Arbitrary, untrusted, name*/
               ,text:   string} /**Content entered by the user*/

Network.network<message> room = Network.cloud("mushroom")

user_update(x) {
  line = <div>
     <div>{x.author}:</div>
     <div>{x.text}</div>
  </div>;
  Dom.transform([#conversation +<- line ]);//Note: If we want to change the syntax of actions, now is the right time
  Dom.scroll_to_bottom(#conversation)
}

broadcast(author){
  Network.broadcast({author=author,
    text:Dom.get_value(#entry)}, room);
  Dom.clear_value(#entry)
}

start() {
  author = Random.string(8);
  <div id=#conversation
    onready={ * -> Network.add_callback(user_update, room) }></div>
  <input id=#entry  onnewline={ * -> broadcast(author) }/>
  <div onclick={ * -> broadcast(author) }>Send!</div>
}

start_server(Server.one_page_bundle("Chat",
   [@static_resource_directory("resources")],
   ["resources/css.css"], start))

Revised syntax, candidate 2

Fork me on github

type message:
   author as string //Arbitrary, untrusted, name
   text   as string   //Content entered by the user

room = Network.cloud("mushroom") as Network.network(message)

Redesign and rationale

• We introduce a Python-ish/Boo-ish syntax for defining types.
• Again, we use as instead of : for type annotations.

def user_update(x):
  line = <div>
    <div>{x.author}:</div>
    <div>{x.text}</div>
  </div>
  Dom.transform([#conversation +<- line ])//Note: If we want to change the syntax of actions, now is the right time
  Dom.scroll_to_bottom(#conversation)

def broadcast(author):
   message = new:
      author: author
      text:   Dom.get_value(#entry)
   Network.broadcast(message, room)
   Dom.clear_value(#entry)

Redesign and rationale

• We introduce a new keyword new: to define immediate records – we find this both clearer than the Python syntax for defining objects as dictionaries, and more suited to both our paradigm and our automated analysis.

def start():
   author = Random.string(8)
   html:
    <div id=#conversation onready={def *: Network.add_callback(user_update, room)}></div>
    <input id=$entry onnewline={def *: broadcast(author)}/>
    <div onclick={def *: broadcast(author)}>Send!</div>

Redesign and rationale

• We introduce keyword html: to introduce a block of HTML-like notations. A similar keyword xml: will be used when producing XML documents with a XML-like notation.

server:
  Server.one_page_bundle("Chat",
    [@static_resource_directory("resources")],
    ["resources/css.css"], start)

No additional change here.

What now?

At this stage, we have not switched syntax yet and we have the following options:

• keep our current syntax;
• adopt revised syntax 1, or a variant thereof – so, start coding a conversion tool, the new syntax itself, and start updating the documentation;
• adopt revised syntax 2, or a variant thereof – so, start coding a conversion tool, the new syntax itself, and start updating the documentation;

Yes, I mention variants, because I am certain that many among you will have interesting ideas. So please feel free to express yourselves.

You can provide feedback:

• on this blog;
• by e-mail, at feedback@opalang.org;
• on IRC.

Remember, we are still accepting applications for the preview.