nLeyten

Ocsigen by example, Part 7: Co-services

Dario Teixeira — Wed, 01 Apr 2009 19:57:00 GMT

Today's example focuses on co-services. A co-service is basically a service that shares its URL with a "parent" service, but which may have different parameters. Co-services are often used together with sessions to personalise a given URL for one specific user.

Since co-services share the same URL, how does Eliom know which one is being invoked? That distinction is made via a special hidden parameter. However, it is nothing that you should be worried about, as it happens transparently to the developer.

The code below is a minimalistic illustration of the possible use for a co-service. I suggest that you use the "view source" functionality of your browser to observe the hidden parameter contained in the page.

(************************************************************************)
(* Coservice demonstration.                                             *)
(************************************************************************)

open XHTML.M


(************************************************************************)
(*  Declaration of the "foobar" service.  Note that we are not
    registering it yet, and therefore we do not provide the handler
    at this point.
*)

let foobar_service =
    Eliom_services.new_service
        ~path: [""]
        ~get_params: Eliom_parameters.unit
        ()


(************************************************************************)
(*  Declaration of the "foobar2" service, which is actually a
    coservice for "foobar".  Coservices share the same path as
    the main service, but may have different parameters.
*)

let foobar2_service = 
    Eliom_services.new_coservice
        ~fallback: foobar_service
        ~get_params: (Eliom_parameters.int "amount")
        ()


(************************************************************************)
(*  Handler for "foobar" service.  It displays the current value of
    the counter and provides links to reload itself or to invoke the
    coservice with different parameters.
*)

let counter = ref 0

let foobar_handler sp () () = 
    Lwt.return
        (html
        (head (title (pcdata "")) [])
        (body [p    [
                pcdata "counter is equal to ";
                pcdata (string_of_int !counter);
                br ();
                Eliom_predefmod.Xhtml.a foobar_service sp [pcdata "reload"] ();
                br ();
                Eliom_predefmod.Xhtml.a foobar2_service sp [pcdata "increment by 1"] 1;
                br ();
                Eliom_predefmod.Xhtml.a foobar2_service sp [pcdata "increment by 2"] 2;
                ]]))


(************************************************************************)
(*  Handler for the "foobar2" service (the coservice of "foobar").
    Note that this handler just increments the counter by the amount
    specified in the GET parameter and then invokes the handler of
    "foobar" to display the actual page.
*)

let foobar2_handler sp amount () =
    counter := !counter + amount;
    foobar_handler sp () ()


(************************************************************************)
(*  Registration of the services.
*)

let () =
    Eliom_predefmod.Xhtml.register foobar_service foobar_handler;
    Eliom_predefmod.Xhtml.register foobar2_service foobar2_handler

Ocsigen by example, Part 6: POST services

Dario Teixeira — Tue, 24 Mar 2009 16:57:00 GMT

In the previous instalment of this series, the example illustrated the creation of a form for a GET service. This time we'll see a similar example, but one whose service uses POST instead of GET.

GET and POST are just two of the verbs part of the HTTP specification (other verbs are HEAD, PUT, DELETE, etc). The main visible difference is that a GET request encodes the parameters into the URL, whereas POST does so in the body of the request. It is therefore common that web developers pick one or the other based solely on this distinction. This criterion is however fundamentally wrong. GET should be used only for services that do not cause side-effects, and which are typically idempotent. Using GET disregarding this rule may subject your users to possible cross-site request forgery attacks.

As illustrated by the code below, the first thing to notice about POST services is that they require the declaration of the GET fallback service. The reason is because POST services encode parameters in the body of the request, and therefore their parameters are not recorded if the user bookmarks the URL. This requirement may seem annoying at first, but it fits very well with the emphasis on correctness that permeates Ocsigen/Eliom (and the Ocaml ecosystem in general). Another thing to notice is that function Eliom_predefmod.Xhtml.post_form (the POST counterpart of Eliom_predefmod.Xhtml.get_form) also requires the GET parameters for the service.

When trying this example, I suggest you reload the "coucou" URL after its first invocation via the form. You will see that upon the second invocation the fallback page is displayed, because no POST parameters were specified. (Depending on the browser, simply hitting "reload" will resend the POST parameters; therefore, select the URL and press ENTER for a clean invocation).

(************************************************************************)
(* Services with POST parameters.                                       *)
(************************************************************************)

open XHTML.M
open Eliom_parameters


(************************************************************************)
(*  This service is just the fallback for the "coucou" service, and
    is never meant to be directly invoked.  Eliom forces one to declare
    these fallbacks because browsers bookmark only the GET parameters
    and not the POST ones.  There is therefore a chance the service is
    invoked without any POST parameters.
*)

let fallback_handler _ () () =
    Lwt.return
        (html
        (head (title (pcdata "Fallback")) [])
        (body [p [pcdata "You've invoked the fallback service for coucou"]]))

let fallback_service =
    Eliom_predefmod.Xhtml.register_new_service
        ~path: ["coucou"]
        ~get_params: Eliom_parameters.unit
        fallback_handler


(************************************************************************)
(*  Service "coucou" takes no GET parameters and one POST parameter.
    Note that during the service registration the service path is not
    indicated, since it will be the same as the fallback's.  The same
    goes for the GET parameters.
*)

let coucou_handler _ () foo =
    Lwt.return
        (html
        (head (title (pcdata "Coucou")) [])
        (body [p [pcdata ("The POST parameter is " ^ foo)]]))

let coucou_service =
    Eliom_predefmod.Xhtml.register_new_post_service
        ~fallback: fallback_service
        ~post_params: (Eliom_parameters.string "foo")
        coucou_handler


(************************************************************************)
(*  This function creates a form for the "coucou" service.  Note that
    proper XHTML forms should be divided into fieldsets, and that any
    input widget should have a matching label, as indicated via its ID.
*)

let coucou_form enter_foo =
    let enter_foo_label = "enter_foo"
    in  [
        fieldset
            [
            label ~a:[a_for enter_foo_label] [pcdata "String foo:"];
            Eliom_predefmod.Xhtml.string_input ~a:[a_id enter_foo_label] ~input_type:`Text ~name:enter_foo ();
            Eliom_predefmod.Xhtml.string_input ~input_type:`Submit ~value:"Click" ()
            ]
        ]



(************************************************************************)
(*  The "main" service creates a form for the "coucou" POST service.
*)

let main_handler sp () () =
    Lwt.return
        (html
        (head (title (pcdata "Main")) [])
        (body   [
            p [pcdata "Please fill in this form:"];
            Eliom_predefmod.Xhtml.post_form coucou_service sp coucou_form ()
            ]))

let main_service =
    Eliom_predefmod.Xhtml.register_new_service
        ~path: [""]
        ~get_params: Eliom_parameters.unit
        main_handler

Ocsigen by example, Part 5: GET forms

Dario Teixeira — Thu, 19 Mar 2009 19:33:00 GMT

We've previously seen that Eliom ensures that hyperlinks to services are type-safe, ie, you cannot for example provide an alphabetic string (or no parameters at all) to a service that expects an integer. This type-safety is also extended to the creation and validation of XHTML forms. You can therefore provide together with a service a function that creates forms for that service. Eliom — or rather the compiler — makes sure the types are in harmony. Moreover, bear in mind that before invoking a service with any parameters, Eliom first checks that the parameters have values as expected. It will therefore raise an error if the user provided us with an alphabetic string in a form field that required an integer or a float, for example (it is of course possible to provide a custom handler for this sort of error situations instead of relying on the default one — check the Eliom tutorial for details).

Another cool feature is the possibility of Eliom services accepting custom types, and not just the predefined integers, floats, strings, etc. Again, check the tutorial for details.

The code below demonstrates the use of GET forms. It defines a "coucou" service taking three GET parameters of different types, and defines also a function "coucou_form" to create the contents of a form for "coucou". In service "main", this function is used as a parameter to Eliom_predefmod.Xhtml.get_form to actually create the form.

(************************************************************************)
(* Demonstration of forms for services with GET parameters.             *)
(************************************************************************)

open XHTML.M
open Eliom_parameters


(************************************************************************)
(*  Service "coucou" takes three GET parameters and no POST parameters.
*)
let coucou_handler _ (i, (j, s)) () =
    Lwt.return
        (html
        (head (title (pcdata "Coucou")) [])
        (body [p
            [
            pcdata "You sent: ";
            strong [pcdata (string_of_int i)];
            pcdata ", ";
            strong [pcdata (string_of_float j)];
            pcdata " and ";
            strong [pcdata s]
            ]]))


let coucou_service =
    Eliom_predefmod.Xhtml.register_new_service
        ~path: ["coucou"]
        ~get_params: (Eliom_parameters.int "i" ** (Eliom_parameters.float "j" ** Eliom_parameters.string "s"))
        coucou_handler


(************************************************************************)
(*  This function is used for building forms for service "coucou".
    Note that its parameter is a nested pair defined in the same
    fashion as the GET parameters for "coucou".  Also note that
    proper XHTML forms should make use of fieldsets for logically
    dividing the form, and that field labels should be declared
    with the  element and associated to the form field they
    refer to via the ID attribute.  Declaring labels this way helps
    to improve usability (think of screen readers for blind users).
*)

let coucou_form (enter_i, (enter_j, enter_s)) =
    let enter_i_label = "enter_i"
    and enter_j_label = "enter_j"
    and enter_s_label = "enter_s"
    in  [
        fieldset
            [
            label ~a:[a_for enter_i_label] [pcdata "Int i:"];
            Eliom_predefmod.Xhtml.int_input ~a:[a_id enter_i_label] ~input_type:`Text ~name:enter_i ();
            label ~a:[a_for enter_j_label] [pcdata "Float j:"];
            Eliom_predefmod.Xhtml.float_input ~a:[a_id enter_j_label] ~input_type:`Text ~name:enter_j ();
            label ~a:[a_for enter_s_label] [pcdata "String s:"];
            Eliom_predefmod.Xhtml.string_input ~a:[a_id enter_s_label] ~input_type:`Text ~name:enter_s ();

            Eliom_predefmod.Xhtml.string_input ~input_type:`Submit ~value:"Click" ()
            ]
        ]


(************************************************************************)
(*  The "main" service simply displays the form that submits data
    to service "coucou".  Note that the actual form is created by
    the Eliom_predefmod.Xhtml.get_form function using the form
    creation function "coucou_form" previously defined.
*)

let main_handler sp () () =
    let form = Eliom_predefmod.Xhtml.get_form coucou_service sp coucou_form
    in Lwt.return
        (html
            (head (title (pcdata "Main")) [])
            (body [p [pcdata "Form for coucou:"]; form]))


let main_service = 
    Eliom_predefmod.Xhtml.register_new_service
        ~path: [""]
        ~get_params: Eliom_parameters.unit
        main_handler

Ocsigen by example, Part 4: suffix parameters

Dario Teixeira — Tue, 17 Mar 2009 17:52:00 GMT

In the examples we've seen so far, GET parameters were passed using the standard scheme (RFC 3986), ie, an ampersand delimited list of key-value assignments. As an alternative, Eliom offers also the possibility of encoding parameters as slash-delimited suffixes to the main URL. For many applications this produces friendlier and cleaner looking URLs.

As an example, consider a service "foobar" taking three integers as parameter: "year", "month", and "day". In the standard scheme, the service could be invoked with today's date by calling "foobar?year=2009&month=3&day=17". Using the suffix scheme, the same invocation becomes "foobar/2009/3/17". You trade the flexibility of specifying parameters by any order for the improved legibility.

The code sample below shows the definition of the service "foobar" using suffix parameters. Note that only the declaration of the service needs to change; any links towards this service created with Eliom_predefmod.Xhtml.a will automatically reflect the new convention (yet another reason to always use Eliom_predefmod.Xhtml.a!).

(************************************************************************)
(* Example for suffix parameters.                                       *)
(************************************************************************)

open XHTML.M
open Eliom_parameters


(************************************************************************)
(*  The "foobar" service takes three GET parameters, all integers.
    It illustrates the use of Eliom_parameters.suffix to construct
    friendlier looking URLs.
*)

let foobar_handler _ (year, (month, day)) () =
    let date = (string_of_int year) ^ "/" ^ (string_of_int month) ^ "/" ^ (string_of_int day)
    in Lwt.return
        (html
        (head (title (pcdata "Foobar")) [])
        (body [p [pcdata ("The date specified by the GET parameters is " ^ date)]]))


let foobar_service =
    Eliom_predefmod.Xhtml.register_new_service
        ~path: ["foobar"]
        ~get_params: (Eliom_parameters.suffix
                    (Eliom_parameters.int "year" **
                    (Eliom_parameters.int "month" **
                    Eliom_parameters.int "day")))
        foobar_handler


(************************************************************************)
(*  The "main" service just provides a link to the "foobar" service.
*)

let main_handler sp () () =
    Lwt.return
        (html
            (head (title (pcdata "Main")) [])
            (body [p [Eliom_predefmod.Xhtml.a foobar_service sp [pcdata "Click for foobar"] (2009, (3, 17))]]))

let main_service =
    Eliom_predefmod.Xhtml.register_new_service
        ~path: [""]
        ~get_params: Eliom_parameters.unit
        main_handler

Ocsigen by example, Part 3: delayed service registration

Dario Teixeira — Sun, 15 Mar 2009 18:07:00 GMT

In the example from the previous article, the various services are registered with Eliom upon declaration. Note that we used the function Eliom_predefmod.Xhtml.register_new_service for this purpose. It is however also possible to separate these two actions, by first just declaring a service, and registering it later. You may use function Eliom_services.new_service for the former action, and function Eliom_predefmod.Xhtml.register for the latter.

Why is the separation between the declaration and registration actions useful? The most obvious application lies in services that call each other. Without this feature, you would most likely would need to resort to a "let rec ... and ..." construct, which isn't always the most practical.

The code below depicts two services, "coucou" and "foobar", which call each other. Note that we begin by just declaring each service and that we only register them in the end.

(************************************************************************)
(* Example for delayed registration.                                    *)
(************************************************************************)

open XHTML.M
open Eliom_parameters


(************************************************************************)
(*  We begin by declaring services "coucou" and "foobar".  The first
    takes no parameters, while the second takes a float as a GET
    parameter.
*)

let coucou_service =
    Eliom_services.new_service
        ~path: ["coucou"]
        ~get_params: Eliom_parameters.unit
        ()

let foobar_service =
    Eliom_services.new_service
        ~path: ["foobar"]
        ~get_params: (Eliom_parameters.float "x")
        ()


(************************************************************************)
(*  We now build the handler functions for each service.  Note how
    each handler creates a link to the other service.
*)

let coucou_handler sp () () =
    Lwt.return
        (html
        (head (title (pcdata "Coucou")) [])
        (body [p
            [
            pcdata "Here's a ";
            Eliom_predefmod.Xhtml.a foobar_service sp [pcdata "link"] 3.1416;
            pcdata " to service foobar";
            ]]))


let foobar_handler sp x () =
    Lwt.return
        (html
        (head (title (pcdata "Foobar")) [])
        (body [p
            [
            pcdata ("The parameter X is " ^ (string_of_float x));
            br ();
            pcdata "Here's a ";
            Eliom_predefmod.Xhtml.a coucou_service sp [pcdata "link"] ();
            pcdata " to service coucou";
            ]]))



(************************************************************************)
(*  Finally we register the declared services and assign to each
    one its handler function.
*)

let () =
    Eliom_predefmod.Xhtml.register coucou_service coucou_handler;
    Eliom_predefmod.Xhtml.register foobar_service foobar_handler

Postscriptum: If you're on the bleeding edge, you may have noted that the ocsigen.conf file I presented in the first chapter of this series needs a tweak in order to work with latest development version of Ocsigen (soon to be version 1.2). The change is minimal: basically, the "hostname" attribute for the host configuration is now called "defaulthostname". Here's the revised file:

<ocsigen>
    <server>
        <port>8080</port>
        <charset>utf-8</charset>
        <mimefile>/etc/mime.types</mimefile>
        <debugmode/>
        <findlib path="/home/dario/.local/lib/ocsigen/METAS"/>
        <findlib path="."/>

        <extension findlib-package="ocsigen_ext.ocsipersist-sqlite"/>
        <extension findlib-package="ocsigen_ext.eliom"/>
        <extension findlib-package="ocsigen_ext.staticmod"/>

        <host defaulthostname="localhost">
            <site dir="">
                <eliom findlib-package="module"/>
                <static dir="./"/>
            </site>
        </host>
    </server>
</ocsigen>

Ocsigen by example, Part 2: hello world

Dario Teixeira — Fri, 13 Mar 2009 18:17:00 GMT

The second instalment of this series presents the first actual Eliom code sample. You can use the Makefile shown in the first part to compile it, and the META.module and ocsigen.conf files for starting the Ocsigen web server.

This example contains three different services: first, the hello-worldish service "coucou" (so named as a homage of sorts to the convention used in the Eliom tutorial); second, the "foobar" service, which illustrates the use of GET parameters; and third, the "main" service, which makes use of Eliom_predefmod.Xhtml.a to create type safe hyperlinks to the first two services. I suggest you fiddle with the types of the GET parameters — you'll see that Eliom makes good use of the type safety of the Ocaml language by making it impossible to create hyperlinks that pass the wrong type to a service.

(************************************************************************)
(* Simple "Hello World!" Eliom examples.                                *)
(************************************************************************)

open XHTML.M
open Eliom_parameters


(************************************************************************)
(*  The "coucou" service takes no arguments, and just displays a
    welcome message with the client's IP address.  The IP can be
    obtained via the "sp" (server parameters) parameter that is
    passed to every handler.
*)

let coucou_handler sp () () =
    let greeting = "Hello " ^ (Eliom_sessions.get_remote_ip sp) ^ "!"
    in Lwt.return
        (html
        (head (title (pcdata "Coucou")) [])
        (body [p [pcdata greeting]]))

let coucou_service =
    Eliom_predefmod.Xhtml.register_new_service
        ~path: ["coucou"]
        ~get_params: Eliom_parameters.unit
        coucou_handler


(************************************************************************)
(*  The "foobar" service takes two GET parameters, an integer "i"
    and a string "s".  Note that the handler is declared as an
    anonymous function instead of as a separate function like we
    did with "coucou".  Though sometimes useful, this practice can
    lead to confusing code; for this reason, this is the only time
    we'll use it in this tutorial series.
*)

let foobar_service =
    Eliom_predefmod.Xhtml.register_new_service
        ~path: ["foobar"]
        ~get_params: (Eliom_parameters.int "i" ** Eliom_parameters.string "s")
        (fun _ (i, s) () ->
            Lwt.return
                (html
                (head (title (pcdata "Foobar")) [])
                (body [p [pcdata ("The GET parameters are: " ^ (string_of_int i) ^ " and " ^ s)]])))


(************************************************************************)
(*  The "main" service just provides links to both the "coucou" and
    "foobar" services.  Try changing the types of the parameters to
    see that the compiler catches any typing inconstencies.
*)

let main_handler sp () () =
    Lwt.return
        (html
            (head (title (pcdata "Main")) [])
            (body   [
                p   [
                    pcdata "Here's a ";
                    Eliom_predefmod.Xhtml.a coucou_service sp [pcdata "link"] ();
                    pcdata " to the coucou service"
                    ];
                p   [
                    pcdata "And here's a ";
                    Eliom_predefmod.Xhtml.a foobar_service sp [pcdata "link"] (10, "hello");
                    pcdata " to the foobar service"
                    ]
                ]))

let main_service =
    Eliom_predefmod.Xhtml.register_new_service
        ~path: [""]
        ~get_params: Eliom_parameters.unit
        main_handler

Ocsigen by example, Part 1: basic setup

Dario Teixeira — Thu, 12 Mar 2009 17:58:00 GMT

This article is the first of a series I'll be doing on Ocsigen programming. Ocsigen is both a web server and the umbrella name for an Ocaml-based framework that brings the power and safety of functional programming to the world of web development. Strictly speaking, this tutorial is about Eliom, the actual programming framework part of Ocsigen. However, because the two are closely related, and because the name "Ocsigen" has much better recognition, I've used it as a title.

Note that the Ocsigen site already offers a very comprehensive tutorial on Eliom. This series of articles is not intended to compete with it. In fact, I won't be doing any tutoring at all and I'll assume that readers are familiar with the existing tutorial. The goal of this series is simply to present self-contained and documented code samples illustrating various aspects of Eliom programming. These are taken from my own self-study experiments with Eliom, and I'm only sharing them with the hope they might be useful to others.

Let's start with the development setup. All the examples in this series can be compiled with the same Makefile and invoked with the same ocsigen.conf configuration file for the server and META file with findlib information. For the sake of simplicity, you can put all of these files in the same directory and just run the Ocsigen web server from that directory in debug mode by invoking "ocsigen -v -c ocsigen.conf" for the byte-code version or "ocsigen.opt -v -c ocsigen.conf" for the native-code version (if you're running Ocaml >= 3.11). The Makefile is as follows:

NAME=module

all: $(NAME).cma $(NAME).cmxs

%.cma: %.cmo
	ocamlc -a -o $@ $+

%.cmxa: %.cmx
	ocamlopt -a -o $@ $+

%.cmxs: %.cmxa
	ocamlopt -shared -linkall -I `pwd` -o $@ $<

%.cmo: %.ml
	ocamlfind ocamlc -thread -package lwt,ocsigen -c $<

%.cmx: %.ml
	ocamlfind ocamlopt -thread -package lwt,ocsigen -c $<

clean:
	rm -f $(NAME).cm[ioax] $(NAME).cmx[as] $(NAME).[oa]

As for the ocsigen.conf file, it can be defined like the code below (just change the findlib path according to your installation). Note the we're using port 8080 so the server can be started by any user:

<ocsigen>
    <server>
        <port>8080</port>
        <charset>utf-8</charset>
        <mimefile>/etc/mime.types</mimefile>
        <debugmode/>
        <findlib path="/home/dario/.local/lib/ocsigen/METAS"/>
        <findlib path="."/>

        <extension findlib-package="ocsigen_ext.ocsipersist-sqlite"/>
        <extension findlib-package="ocsigen_ext.eliom"/>
        <extension findlib-package="ocsigen_ext.staticmod"/>

        <host hostname="dual">
            <site dir="">
                <eliom findlib-package="module"/>
                <static dir="./"/>
            </site>
        </host>
    </server>
</ocsigen>

Finally, note that because we rely on findlib for the installation of the module containing the actual code for the web site, we must provide a META file indicating the dependencies and the names of the compiled code units. As per the ocsigen.conf declaration, the name of this file must be META.module:

directory = "."
requires = ""
archive(plugin,byte) = "module.cma"
archive(plugin,native) = "module.cmxs"

And that's it as far as the development setup is concerned. The next article will present the first real example. Stay tuned!

Moglen on the origins of copyright and patents

Dario Teixeira — Fri, 30 Jan 2009 16:25:00 GMT

I mentioned Eben Moglen before on this blog. If you are still not familiar with any of his writings, I strongly suggest you listen to this episode of the Software Freedom Law Show podcast. It features a recording of one of Eben's recent talks on the origins of copyright and patents. It is well worth it, I promise.

A moment of Zen

Dario Teixeira — Mon, 22 Dec 2008 17:24:00 GMT

I recommend a moment of Zen before breakfast. Here's one for the holidays:

Consider the set S of all possible files of length n. A file compressor (such as Gzip, 7-Zip, and many others) is a programme that given any randomly chosen file from S, returns a new file whose size is on average greater than or equal to n.

Meditate on the statement above until you realise its truth. Then, grasshopper, you shall be enlightened.

Note: The "equal to" part is necessary because there is an entire family of compressors (the identity compressor being the most obvious one) that always return a file the same size as the one provided as input.

Introducing Camlhighlight

Dario Teixeira — Mon, 24 Nov 2008 15:41:00 GMT

Last time, I told you about Blahcaml, a library that wraps Blahtex, thus providing facilities for converting equations in TeX format into MathML. The news is that in the meantime I've completed the missing pieces in Blahcaml (namely proper error reporting), and released version 1.2. Functionality-wise, I consider the library "finished". Please let me know if you find any glaring omissions or bugs.

Another feature I want for Lambdoc is the ability to display syntax-highlighted source code. And so I spent some time investigating the available options for highlighting source code within Ocaml. Note that I wanted a generic highlighter, one that could handle most common programming and markup languages. Because there is no Ocaml library up to the task, I looked at the C/C++ options and even at the possibility of co-opting Vim for the task.

Using Vim is a seductive idea given the high-quality of Vim's syntax-highlighting, but unfortunately flounders on Vim's manifest unsuitability to be used inside scripts. Yes, you can invoke Vim from the command line with a sequence of commands, but I saw no way of camouflaging the fact that Vim's interface still pops up, even if for a split-second.

On the C/C++ side the options are GNU Source-highlight and Highlight. While both are command line applications, their core routines can easily be extracted and made into a library. Moreover, their licenses are friendly to this operation. The downside is that the quality of the highlight of either application is not as good as Vim's, at least as far as highlighting Ocaml is concerned.

I've chosen Highlight for the time being. The wrapper library is called Camlhighlight and is now at version 0.5. While there are still some features missing, it's already quite usable. The library has functions for generating an Ocsigen-friendly XHTML.M output, both in fancy and more plain style (it's possible to customise whether or not one wants line numbers and/or zebra stripes).

Blah, blah, blah

Dario Teixeira — Wed, 29 Oct 2008 16:59:00 GMT

Lambdoc supports mathematical expressions in both TeX and MathML format. The former was chosen because it is the de facto standard for inputting mathematics in a human-friendly manner (for slightly unconventional definitions of both "human" and "friendly"). The latter because several graphical tools for designing math expressions now have the capability of converting their own internal (often proprietary) formats into MathML.

Though browser support for MathML is still largely broken, that will hopefully change with the imminent release of the STIX fonts. So rather than doing a server-side rendering of all maths into images and serving those to the client (the solution that Wikipedia currently uses), I have chosen to take the plunge into the MathML world and serve that instead. It follows that I must somehow convert all those equations in TeX format into MathML.

Now, I was hoping this task would be a no-brainer. The tool that the Wikipedia itself currently uses is texvc, which is written — drumroll — in Ocaml. Unfortunately, texvc's forte is the conversion of TeX into images, having only very minimal support for converting TeX into MathML (it basically chokes on any expression more complex than "x = y"). I was therefore forced to look into alternatives.

I ended up settling on Blahtex. It is being developed with the intention of being integrated into the Mediawiki package that powers Wikipedia. Moreover, besides the commonplace rendering of TeX equations into images, Blahtex also aims at converting TeX into MathML, and is already very competent at that task. Unfortunately, and unline texvc, Blahtex is developed in C++. This is how I ended up spending some time learning about the ways of interfacing C/C++ code with Ocaml.

Ocaml has very good support for interfacing with C code (and indirectly also with C++, since inside a C++ programme you can declare functions to have C-style linking). You can invoke C functions from the Ocaml side, Ocaml functions from the C side, and mix and match as per your requirements. The authorative reference on this lore is the Ocaml manual itself, though I recommend that newcomers start with Florent Monnier's excellent tutorial. While not rocket science, the interfacing does require some attention, particularly on matters where Ocaml's garbage collector is involved. Moreover, when writing the C stubs we are constantly reminded that we have left the safety of Ocaml shores and that we now find ourselves in "Here Be Dragons" territory.

Another issue that need handling was Blahtex's lack of any sanity checks on its input. Since web applications are the most obvious application of a TeX to MathML converter, it worried me that a crafty user could pull a cross-site scripting attack via a math expression. To solve this issue I chose an approach that though heavy-handed is simple to implement and offers beneficial side-effects: the Ocaml side can ensure the sanity of the generated MathML by validating it against the official MathML2 DTD from the W3C. During this process I discovered that the very capable though complex PXP library is the only Ocaml XML parser that can handle the complexity of the MathML2 DTD.

The end result is Blahcaml, a library that offers basic bindings to Blahtex, but with added (optional) sanity checking. I have just released version 1.0, and though it still hasn't been through a lot of testing, it's performing nicely and can be considered stable. Try it out and let me know if you find any problems!

Nleyten@1500,1600: Lambdoc takes shape

Dario Teixeira — Tue, 23 Sep 2008 20:48:00 GMT

It's been a while since I've given an update on the developments at my Subversion repository. In the meantime the revision log has passed the 1500 and 1600 mark, and a substantial part of it has been moved to a public repository hosted at the OcamlForge.

Revision 1500 reads "Added Sexplib (de)serialization functions for Lambdoc documents" while 1600 reads "Added support for appendix sections". Both of these reflect where development has been taking place in the last few months: I've been working on Lambdoc, a library that adds support for semantic-rich documents within the Ocsigen framework. The centrepiece of this library is Lambtex, a markup language inspired by LaTeX, but with simpler, more consistent syntax and designed with web applications in mind. I'm still putting the finishing touches on Lambdoc, and I promise I'll make a much more comprehensive description of the library once it is ready for prime-time.

Postscriptum: Concerning the ongoing financial meltdown, a note to the friends who know me well: a) yes, I do feel a certain degree of schadenfreude and vindication: do you believe me now when I say that capitalism is inherently flawed and that it will destroy itself? b) don't worry, because my potential smugness is more than tempered by the realisation that we're all together on the same boat. Unfortunately, us long time critics of capitalism don't live in a bubble that protects us from its nefarious effects; c) look at the bright side: society now has a chance to finally move on unimpeded by the dogma of liberal capitalism and faith in the "invisible hand"; if nothing else, those Economist-following, Ayn Rand-worshipping crowds will be a lot more silent — that alone almost makes a financial collapse bearable! d) the crisis is far from over: I would pay close attention to what will be happening to the dollar; e) These are indeed interesting times — pass the popcorn!

Use Ocaml, save the world

Dario Teixeira — Fri, 23 May 2008 18:47:00 GMT

(Note: this post is written in a tongue-in-cheek manner; take it too seriously at your own peril).

Since this blog is syndicated in the Ocamlcore Planet, readers may be wondering about the implications for our favourite language of the coming energy crisis I just wrote about. And in general, just how well are different computer languages adapted to an energy-constrained economy?

There are two different factors to consider: a) how efficient is the language at converting CPU cycles into useful work, and b) how efficient is the language at converting programmer time into useful programmes.

The importance of the first factor should be self-evident. If not, note that computers consume an increasing percentage of our energy, and that data centres all over the world also require a significant amount of power. Moreover, there is a wide disparity in the amount of time different languages require to complete the same task. This difference boils down to different approaches to programme execution (compiled languages tend to be more efficient than interpreted ones) and the quality of the code generated by compilers. The Computer Language Shootout, though not without its flaws, provides some ballpark figures on the efficiency of different languages. As you can see, Ocaml fares very well on the existing benchmarks.

The second factor, though it may not be obvious at first sight, may indeed be even more relevant to this discussion. Programmer time requires energy not only to sustain the Homo sapiens specimen that does the programming, but also because programming is to a large extent done in front of an energy-hungry computer. Therefore, by making the programmer more productive, a computer language can result in significant energy savings.

Measuring language productivity is a difficult task. Nevertheless, I would like to posit that the availability of a strong static type checker such as Ocaml's — one that catches most common mistakes at compile-time — is a plus in this regard. Even more so if we consider runtime errors as a loss of productivity. Another good proxy for language productivity is indicated by the terseness of its programmes. Again, the Computer Language Shootout may be of assistance.

So, how well does Ocaml fare when these two factors are taken into account? Very well, I am pleased to say. In fact, using some reasonable parametrisation, the results of the Shootout show Ocaml as the top rated language. We can therefore conclude that Ocaml will also have a role to play in saving civilisation from an energy crisis.

Peak Oil revisited

Dario Teixeira — Fri, 23 May 2008 13:39:00 GMT

More than a year ago I posted here my ramblings on the possibility of world oil production being at, or close to, its peak. At the time, the topic of "Peak Oil" was still rather obscure, and there were still many advocating that the world's production capacity could continue to grow for decades to come. They were the ones saying that the price would soon drop down to $20 or $30 a barrel. Needless to say, since then events have precipitated, and just last week the price of oil jumped above the psychological barrier (for computer scientists, anyway!) of $127 a barrel (127 being of course the largest integer quantity that can be expressed with only 7 bits). It seems like a perfect time to revisit and update those predictions.

On the whole, I have nothing to detract from what I wrote 14 months ago. I may have been overly optimistic in some areas (more on that below) and events seem to be happening faster than what I expected, but I still expect the picture to remain more-or-less the same. While I may be now slightly less optimistic about our prospects for the coming couple of decades, overall I still think that the transition away from fossil fuels will occur without the catastrophic collapse of civilisation presaged by those with an inclination for doom.

Note, however, that these are far from being cornucopian predictions. If reality scares you, perhaps you should be visiting this site instead.

Is the peak real and happening now?

In other words, how can we be sure the rise in prices reflects a real constraint on the production side instead of just speculation and/or conscious throttling on the part of exporting nations in order to drive up revenue? Moreover, even if the constraints are real, couldn't this just be a temporary situation caused by a past underestimation of the current demand needs (bear in mind the huge rise in demand in the past few years by China and others), and that everything will be resolved as soon as production catches up?

Well, regardless of the actual numbers, we should reflect anyway on the utter and complete absurdity of having these crucial statistics about the wealth of the world economy be dependent on whims and state secrets of despotic regimes (Saudi Arabia, for one). This is a shortsightedness for which we could pay very dearly.

In any case, what leads me to believe we are now on the peak's "bumpy" plateau is the following:

Discoveries peaked in the 60s and despite all technological advancements, have been declining ever since. Recent discoveries (despite all the media attention they get) are pathetic in comparison to the fields discovered decades ago, and are typically made off-shore in very deep waters (read expensive, dangerous, and desperate locations). This fact alone shows that oil production is bound to peak sooner or later, though of course does not prove it is actually occurring now.
The number of countries past their peak keeps increasing. Three well-known recent examples are Mexico, Norway and the United Kingdom. The latter went in less than a decade from being a major exporter to a net importer of oil; as for Mexico, predictions are that in less than ten years it will cease exports altogether (bad news for the US, which imports a huge amount of oil from its southern neighbour).
The two big exporters, Saudi Arabia and Russia, seem unable to increase their production. Many argue they may also be close to peaking.
There are indeed a couple of oil megaprojects coming online in the next couple of years, but afterwards the situation seems dire. We seem to be now in what has been called a "Peak Lite" scenario. This means that even though the absolute peak may not yet have been reached, the oil industry is starting to show the symptoms of a post-peak situation. Namely, production is unable to keep up with demand, leading to a significant increase in prices.
Currently, world production sits at about 87 million barrels per day. If you take into account the new projects coming online, and the rates of depletion in existing fields, it is possible that this number may yet increase slightly within the next couple of years. However, I very much doubt it will ever reach 90 mbpd. Couple this with the ongoing increase in demand and the Export Land Model (more on that below), and I would bet against oil ever going below $100 a barrel (barring a major global recession, of course).

Aggravating circumstances

I didn't factor in the so-called "Export Land Model" (ELM), but in retrospect I should have. In short, because the internal demand for oil in the big exporting countries is increasing at a fast pace, the amount of oil available for export decreases at a much faster rate than what could be expected from depletion alone. This means that once past the peak, the decline in oil available for export may be much steeper than anticipated. In other words, the transition away from fossil fuels could feel more like shock therapy than a smooth ride.
Society is still by and large in denial. Moreover, most people (including politicians and journalists) are completely clueless about basic principles of thermodynamics and the orders of magnitude involved in energy discussions. To give you a concrete example that illustrates most people's complete detachment from the workings of the physical world, some weeks ago a leading Portuguese news magazine published a vision of society in 15 years time; they talked of "cars running on wind power", and showed a picture of a tiny wind turbine on top of a futuristic-looking car! It would almost be funny if the consequences of this kind of thinking were not so tragic.
Energy Return on Energy Invested (EROEI) is a problem that goes beyond oil. Not only do fossil fuels offer relatively cheap and very dense energy, but you get a huge return for each unit of energy you invest in extracting them. In the early days of the oil industry, when extraction was as simple as digging a hole in the backyard, EROEI was as high as 100:1. Today, because drilling at huge depths and off-shore is quite energy-intensive, the EROEI for oil has decreased to 20:1 or 15:1. And note that for renewables and nuclear, the EROEI is even much lower than that. (As a side note, take heed that scholars have pointed out that a decrease in EROEI was a contributing factor in the decline of the Roman empire. One of the most interesting interviews I've listened to recently addresses this issue: it features Thomas Homer-Dixon discussing his latest book, "The Upside of Down"; I very strongly recommend listening to it).

Consequences for industry, commerce, and tourism

I won't repeat what I wrote last year. I merely wish to emphasise a few points that I find most relevant:

Locality will become much more important. With transportation costs rising significantly, many current assumptions will no longer be valid. This will have impacts on industry (suddenly it won't make as much sense to produce cheap plastic toys in China), agriculture (such as importing out-of-season fruit half-way across the globe), and of course, tourism. I think you can find countless other examples where the assumption that transportation costs are near zero have shaped our economy.
The airline industry as we know it is seriously screwed. Note however that "screwed" is not the same as "doomed to extinction" like many are quick to predict. Fares are still ludicrously cheap by historical standards, and even if five years from now they are 3-4 times higher, there will still be a sizable amount of air travel going on. Nevertheless, air travel will take a serious hit from the effects of Peak Oil, and I reiterate that current plans for expanding and building new airports will look very foolish in less than ten years time.
In addition, the travel industry will undoubtedly adapt and many of its facets will look fairly different in 10 or 15 years from now. Part of the adaptations will be purely technological (as an example, consider that turboprop planes are more fuel efficient than jets), others will come from purely a business choice (the "no frills" concept will go away, because in a context of much higher fares due to fuel costs, the cost of frills will be proportionally tiny, thus giving an advantage to companies that do provide them), and others may stem from a rethinking of the very concept of travel. Consider for example the possibilities offered by the eventual return of the airship (as in the "Hindenburg", hopefully minus the fire and oh-the-humanity). Presently, the "getting there" part of air travelling consists of just the ordeal of having to sit in a confined space for hours on end. What if instead the holiday trip were to start the moment you stepped on a comfortable and spacious airship? It wouldn't matter so much if a trip took a couple of days instead of a few hours.
I doubt that economy will fare well during the first 5-10 years of the transition. The scenario I find most likely is a return of the stagflation that plagued the 1970s. However, note that the economy doing badly for an extended period of time is not the same as catastrophic economic collapse. Last time I checked, there were no zombie hordes roaming the streets during the 1970s (though they would explain the whole Disco phenomenon).

Winners and losers

In the grand scope of things we'll all be losers. Even if somehow an energy-rich country were able to isolate itself from the events on the rest of the world, it would still lose simply because the misfortune of any single country reflects the moral collective failure of all humanity.

With the above caveat in mind, some countries are without a doubt better prepared than others to face an energy crisis. Here's my take on some noteworthy examples:

I stick to my previous observation that China will suffer serious growing pains. I realise that the consensus among many is that this will be "the century of China", but I beg to differ. On the short term, the rise in oil prices will impact its status as the "world's factory" (note that its export-based economy is dependent on the assumption that transport costs are negligible). On the longer term, China faces a serious sustainability crisis. The challenges of the energy crisis will combine with the effects of climate change, soil erosion, and aquifer depletion. Bear in mind that this a country that needs to feed 1.3 billion people.
The picture for India is similar to that of China, though India's environmental problems are not yet quite as severe. On the other hand, there is the further complication that India's population time bomb is far from being under control.
Most of the developing world is in serious trouble. Note, however, that the energy crisis is only one small part of the problems faced: climate change, environmental degradation, and — the elephant in the room — overpopulation. Many countries are indeed just disasters waiting to happen (Haiti comes immediately to mind; it's just one hurricane away from catastrophe). Also, I wouldn't be surprised to see Somalia-style collapses of individual states.
Countries rich in resources and with fairly low population densities are likely to fare relatively well. Brazil and Canada come to mind. Note, however, that I would not include Australia in that list. Though rich in energy resources, Australia is bound to face problems stemming mainly from the degradation of its very fragile environment and the effects of climate change.
Europe as a whole should fare relatively well, though there are of course stark differences in the degree of preparedness of different countries. France, thanks to its reliance on nuclear power, extensive train network, and ongoing projects to build and expand tram and light rail lines is an example to follow. The United Kingdom, on the other hand, will be a victim of years of neglect in its public transit system (particularly railways), and its electric power production capacity. In short, problems stemming from what has been termed "The Anglo Disease".
As it currently stands, the United States is very badly prepared for an energy crisis. I would really, really, like to think that Americans will soon wake up and put their ingenuity to work. They would embark on a "Manhattan plan" for fuel alternatives, abandon the suburbia experiment, and redesign their cities in a public-transit and walk-friendly manner. A friend of mine likes to remind me of a quote by Churchill, who said that "You can always count on Americans to do the right thing — after they've tried everything else".
Unfortunately, recent history does not bode well for the type of reaction we can expect from the US. And note that I'm not just referring to the invasion of Iraq (is there still anyone who maintains the illusion that its purpose was not to take control of the sizable Iraqi reserves?). No, that is simply a reflection of the path Americans chose to take in the 1980 election. Back then, they were given the choice between a candidate aware of the future energy crisis and who advocated some near-term sacrifices to move their economy away from fossil fuels (that candidate was Jimmy Carter), and a candidate who instead told the Americans to screw conservation, that it was "morning in America", and that it felt much better to simply bury their heads in the sand and not to worry about the future. The latter candidate was Ronald Reagan, who has you might recall won by a landslide victory. (On a related note, it is disheartening to see Hillary Clinton — otherwise the most intellectually capable of all three major candidates — follow the short-sighted and populist gimmick proposed by McCain of dropping the gas tax for the holidays).

Note, however, that I wouldn't be so quick to dismiss the US entirely. It's a huge country, rich in resources (one can sometimes forget that even though it is well past its peak, the US is still the third largest producer of oil in the world), a major agricultural exporter, and with a population density that is sufficiently low to make the country sustainable in the long term.

In a sense, the biggest challenges the US faces are a crooked political system, and a blind faith on capitalism and the invisible hand of the markets. The biggest danger for the US (and the rest of the world) is that the energy crisis is successfully exploited by the usual ensemble of populist and extremist politicians. Despite this, I reckon that Americans will eventually do the right thing — they've been trying everything else up till now, after all.

(A brief note on populism: the energy crisis we're facing is in large part a consequence of past populist and short-sighted policies; it is somewhat ironic and definitely discouraging to note that the very same populism that got us into trouble in the first place is the one that typically profits the most from situations of crisis).

In the short term

Though there may be indeed a speculative component to the current price and therefore some ease up won't be surprising, I very much doubt oil ever again get below $100 (again, barring a major recession).

OPEC will have their annual meeting in September. If any, the increases in production announced will be small and insufficient to meet the rise in demand. A couple of months later, the International Energy Agency (the watchdog organisation over matters of energy) is scheduled to issue a report concerning oil reserves. I expect their current rosy predictions that production could grow all the way to 2030 to be revised. This is likely to be the moment that "Peak Oil" and "Energy Crisis" enter the mainstream media. Hopefully not to long afterwards, the true nature of the problem — that these are just aspects of a much broader "Sustainability Crisis" — will become the focus of society's concerns.

Oh, the humanity!

Dario Teixeira — Thu, 24 Apr 2008 23:55:00 GMT

Labelled and optional arguments, and polymorphic variants rank among Ocaml's most interesting features. They can lead, however, to some pretty funky error messages. Here's one example:

File "main.ml", line 160, characters 38-44:
Error: This expression has type
         (unit, unit,
          [ `Attached of
              [ `Internal of ([< `Coservice | `Service ] as 'a) * [ `Get ] ]
              Eliom_services.a_s ],
          [< Eliom_services.suff ] as 'b, 'c, unit, [ `Registrable ])
         Eliom_services.service ->
         (Eliom_sessions.server_params ->
          ('d, Litiom_blocks.out_t) Litiom_blocks.t ->
          Eliom_predefmod.Xhtml.page Lwt.t) ->
         (unit, unit,
          [ `Attached of
              [ `Internal of [< `Coservice | `Service ] * [ `Get ] ]
              Eliom_services.a_s ],
          [< Eliom_services.suff ] as 'e, 'f, unit, [ `Registrable ])
         Eliom_services.service ->
         (Eliom_sessions.server_params ->
          ('g, Litiom_blocks.out_t) Litiom_blocks.t ->
          Eliom_predefmod.Xhtml.page Lwt.t) ->
         carry:unit ->
         Eliom_sessions.server_params ->
         (unit, int,
          [> `Attached of
               [> `Internal of [> `Coservice ] * [> `Post ] ]
               Eliom_services.a_s ],
          'e, 'f, [ `One of int ] Eliom_parameters.param_name,
          [> `Registrable ])
         Eliom_services.service
       but is here used with type
         (unit, unit,
          [ `Attached of [ `Internal of 'a * [ `Get ] ] Eliom_services.a_s ],
          'b, 'c, unit, [ `Registrable ])
         Eliom_services.service ->
         (Eliom_sessions.server_params ->
          ('d, Litiom_blocks.out_t) Litiom_blocks.t ->
          Eliom_predefmod.Xhtml.page Lwt.t) ->
         carry:unit ->
         Eliom_sessions.server_params ->
         (unit, 'h, [< Eliom_services.post_service_kind ],
          [< Eliom_services.suff ], 'i,
          [< int Eliom_parameters.setoneopt ] Eliom_parameters.param_name,
          [< Eliom_services.registrable ])
         Eliom_services.service

Obvious, isn't?

Simple benchmarks on the Ocsigen server

Dario Teixeira — Mon, 21 Apr 2008 14:24:00 GMT

I have looked for a web framework shootout similar to the one that exists for computer languages. Sure, performance is hardly the deciding factor when choosing a web framework, and benchmarks don't say anything about features, robustness, and security. Note, however, that the same can be said about programming languages in general and that does not prevent people from using the shootout in pissing contests. I can, for example, easily place Ocaml on top of that list just by giving a larger weight to the gzip size of the programme (a choice not altogether arbitrary: bear in mind that gzip size is a good indicator of a language's expressiveness; the smaller, typically the more expressive a language is).

Another important caveat to consider is that the bottleneck for the typical web application lies on the database side, not on the webserver. In many cases, the webserver does little more than to fetch and to do minimal processing on the results provided by the database.

Anyway, though I can't make comparisons between these numbers and those from other frameworks using other languages (who in their right mind would want to use PHP?), it is nevertheless interesting to have an idea of how well Ocsigen applications perform. I am therefore sharing the results of some simple tests I ran on two different machines: an old Celeron running at 500 Mhz (a very slow machine by today's standards), and a modern Intel Pentium Dual E2160 running at 1.80 GHz. Note that architecture-wise, the former machine is an x86, while the latter is an AMD64.

All of the tests concern the generation of dynamic content using Ocsigen's Eliom. Though test 1 always produces the same result and could therefore easily be cached, that has not been done. The idea was to test the most demanding operation for a web application: the dynamic generation of pages (typically personalised for individual users). Also, there are two sets of results for each test; one running Ocsigen in bytecode, and another using native code. Note that because Ocsigen relies on dynlink, and Ocaml 3.10 does not support the dynamic linking of native code, I had to use the Ocaml CVS HEAD for the tests. Native dynlinking will arrive with Ocaml 3.11 (coming this summer?).

I used Siege to perform the actual benchmarking. Each test was performed with 10 concurrent client threads, and ran for 30 minutes. Siege was executed on the same machine where the Ocsigen server was located. The results are presented in terms of the number of transactions performed per second. The higher the better, of course.

Test 1: simple page

This service takes no parameters and simply outputs a page with a "bench1" header. This is the Eliom handler that creates the page:

let bench1_handler sp () () =
    Lwt.return
        (html
            (head (title (pcdata "bench1")) [])
            (body [h1 [pcdata "bench1"]]))

And here are the results obtained:

	Celeron	Pentium Dual
Bytecode	110.41	590.49
Native	197.32	1461.10

Test 2: arithmetic on integer parameters

This service takes two integers as a GET parameter, and outputs the result of some common arithmetic functions performed on those numbers:

let rec gcd a = function
    | 0 -> a
    | b -> gcd b (a mod b)


let lcm a b = (a * b) / (gcd a b)


let bench2_handler sp (a, b) () =
  Lwt.return
    (html
      (head (title (pcdata "bench2")) [])
        (body
          [
          h1 [pcdata "bench2:"];
          p [pcdata (Printf.sprintf "%d plus %d is %d" a b (a+b))];
          p [pcdata (Printf.sprintf "%d minus %d is %d" a b (a-b))];
          p [pcdata (Printf.sprintf "%d times %d is %d" a b (a*b))];
          p [pcdata (Printf.sprintf "%d divided by %d is %d" a b (a/b))];
          p [pcdata (Printf.sprintf "%d mod %d is %d" a b (a mod b))];
          p [pcdata (Printf.sprintf "The GCD of (%d,%d) is %d" a b (gcd a b))];
          p [pcdata (Printf.sprintf "The LCM of (%d,%d) is %d" a b (lcm a b))]
          ]))

Note that the client had to request a page with two GET parameters. These were generated randomly with the special shell variable $RANDOM, and stored in a urls.txt file read by siege. $RANDOM produces random integers between 0 and 32767 (yes, I know there was a chance that both "b" would be 0, causing an exception; I checked the urls.txt beforehand, making sure that didn't happen). Anyway, here are the results obtained:

	Celeron	Pentium Dual
Bytecode	85.76	444.06
Native	172.39	1215.19

Test 3: generate page with 100 pseudo-random paragraphs

The final test is a bit more demanding. It takes no parameters, and outputs a page consisting of 100 paragraphs containing both a deterministic and a random component:

let random_paragraph =
    let rng = Cryptokit.Random.device_rng "/dev/urandom"
    and to_hex = Cryptokit.Hexa.encode () in
    fun i ->
        let random_num = Cryptokit.Random.string rng 80 in
        let random_str = Cryptokit.transform_string to_hex random_num in
        p [pcdata (Printf.sprintf "This is paragraph %d: %s." i random_str)]


let bench3_handler sp () () =
    Lwt.return
        (html
            (head (title (pcdata "bench3")) [])
                (body
                    [
                    h1 [pcdata "bench3:"];
                    div (List.init 100 random_paragraph)
                    ]))

Results obtained:

	Celeron	Pentium Dual
Bytecode	8.01	40.66
Native	28.67	185.00

Conclusion

First, there is a clear advantage to using native code (no surprise there!). Ocaml 3.11 will therefore be very welcome for Ocsigen users. Moreover, note that the byte/native code difference is even more acute on the AMD64 architecture. There is at the moment still some discussion on whether or not the AMD64 port should use the "-dlcode" compiler option by default (this option is necessary for native dynlinking). Hopefully further tests will come to the conclusion that there is little or no performance impact of using it.

It would be interesting to investigate how other languages/frameworks fare on similar tests. Even more interesting will be to repeat these tests on a more real world scenario, with an actual application that accesses a database, etc. I intend to conduct these soon on Lambdium-light.

Nleyten@1400: Netplexing

Dario Teixeira — Thu, 17 Apr 2008 16:28:00 GMT

The Ocamlnet set of libraries ranks among the many gems available for Ocaml. The blurb on the project's page describes it this way:

Ocamlnet implements a number of Internet protocols (http client & server, cgi and cgi variants, SunRPC, FTP, POP, SMTP) and is a strong base for web and Internet programming. Many protocols can even be driven in an asychronous way, since Ocamlnet defines a framework for asynchronous implementations (equeue). There is also a generic server framework (netplex). There are a number of accompanying data structures like mail messages, URLs, buffers, channels, and also routines for character set conversions.

Netplex is of particular interest for me at the moment. It is basically a generic server framework, allowing one to build full-fledged servers with very little effort. If you've ever built a server from scratch, you will know that while building a basic server is relatively straightforward, things start to get complicated if you intend to move from a simple toy into a production-quality application. Pretty soon you'll start having to worry about keeping pools of slave workers, managing the workload, and synchronising the entire apparatus. Netplex takes care of all of that stuff for you. The end result is that you get an efficient, secure, and fully-featured server with just a few lines of code.

I am using Netplex to build a "parsing server" (aka the 'Parserver'). It is basically a little daemon that sits in the background providing a service that translates a document formatted in one of the supported markups (like Lambtex) into the native representation used by Lambdium. Placing this service in an external, independent daemon makes sense for a number of reasons:

it makes the system more scalable, since we can easily move the parsing server into its own machine(s) if necessary.
it makes the system more secure, since the parsing daemon can run as 'nobody' as shield the rest of the system from potential attacks based from the parsing routines.
Applications built using the Ocsigen framework should be built using the Lwt library for cooperative threads (Lwt stands for 'Lightweight Threads'). At the present no parser-generator includes support for Lwt. It does make sense therefore to place the parsing outside the main application.

As for SVN's 1400 revision log itself, it reads "Defined the protocol for communication with Parserver".

Ocsigen 1.0.0 released!

Dario Teixeira — Tue, 01 Apr 2008 16:46:00 GMT

Vincent Balat announced today on the Ocsigen mailing-list the release of version 1.0.0 of that of that web server + web programming framework. This is great news, as that magical "1.0.0" will hopefully attract more users to this exciting project. Moreover, I can assure you this is a rock solid release; in fact, it has been fairly mature since the 0.6 days, and the 0.99.x series has been in out and about for almost a year now. (In case you haven't noticed, this is the framework I have been using to develop Lambdium and Lambdium-light).

So, why should you care about Ocsigen? Well, it offers much more than a high-performance web server written in Ocaml (though that's also cool). Integrated with Ocsigen comes Eliom, a web programming framework based on the Ocaml language. It brings the advantages of a safe, functional, strongly-typed, and amazingly fast language to web development. Here are some of the highlights:

Static checking of XHTML:
Either using native Ocaml or Ocamlduce, the type system makes sure at compile time (no runtime penalties!) that the pages you generate are valid XHTML.
Web sites consistent by construction:
Again thanks to Ocaml's type system, there's just no way for a website to include broken links, type mismatching of form fields, wrong evaluation of page parameters, and similar brittleness. And again, all this is checked at compile-time.
One page, one function:
This not only helps to ensure the modularity and conciseness of web sites, but it is a more elegant solution to the consistency problem than the templating offered by other frameworks.
Continuation-based web programming:
This is one of those features that is a bit hard to explain, but whose advantages are obvious once you actually start using it. It does simplify tremendously the correct use of the "back" button in browsers or "what if" scenarios when users open multiple tabs from the same form.
Automatic handling of sessions:
Eliom takes care of most of the low-level stuff, automatically taking care of session management. Moreover, it offers you a wealth of possibilities, including the choice between volatile or persistent sessions (even if the web server is restarted!), public versus private sessions, etc.
Amazing speed without sacrificing expressiveness:
Ocaml is known to be a very fast language, comparable to C++ in terms of speed. Moreover, it is a very high-level language, with a degree of expressiveness comparable to Python or Ruby. Wouldn't you like to to have the best of both worlds? Now you can.

In short, Ocsigen brings web development from the infant age into maturity. So, if you are curious, or just simply tired of dealing with the slowness of Ruby on Rails or the epic brain damage that goes by the name of PHP, I suggest you pay a visit to Ocsigen's site and take a look at the tutorial. The only prerequisite is that you know the Ocaml language. But it's well worth it.

S-expressions for long-term storage of Ocaml values

Dario Teixeira — Mon, 10 Mar 2008 16:21:00 GMT

Call it marshalling, pickling, serialisation, or whatever else you wish; this operation — where a value (or "variable" in non-FP languages) is extracted from a programme at runtime in order to be stored in disk or transferred through the wire — is critical for many applications. Most programming languages therefore include in their standard libraries some means of performing it.

Ocaml ships with the venerable Marshal module. It is extremely simple to use: suppose we wished to convert into a string str a marshalled representation of a value manuscript of type Manuscript.t. This would suffice: let str = Marshal.to_string manuscript []. The inverse operation — taking a string with a marshalled representation and converting back into a programme value — is also dead easy: let manuscript : Manuscript.t = Marshal.from_string str 0.

The attent observer will have noticed the type annotation in the unmarshalling example above; in general, unmarshalling does require an explicit type annotation (which otherwise is rarely needed in Ocaml). The reason is that the type inference mechanism may not have enough information to determine what is the type of the marshalled representation (imagine, for example, that the routines performing the marshalling and unmarshalling reside in different programmes!). And here lies the Achilles' heel of marshalling in Ocaml: should the programmer make a mistake in specifying the type to be unmarshalled, the programme will most surely segfault. This problem also occurs if a version of the programme stores a marshalled value, which is then read back by a subsequent of the same programme where the value's type has been modified (even if ever so slightly).

Over the years, there appeared a number of extensions to the Ocaml language offering type-safe marshalling ( HashCaml may be the most well-known, but it's not the only one). And judging from comments by Xavier Leroy (the primary developer of the Ocaml language) at the Ocaml users meeting in Paris this last January, there's a good chance that type-safe marshalling will make it into the core language in the near future.

In Lambdium-light, stories and comments are stored in the database backend using the marshalled representation offered by the Marshal module. While this works and is extremely fast, it does have the problem of not being very resilient to future changes in the story and comment format. Therefore, I have been looking at alternatives to Marshal that provide some degree of backwards compatibility while not sacrificing too much speed.

I suspect that the XML-fanboys in the audience wouldn't even think twice, but personally I am far from finding XML a good solution for many of the problem domains where it is applied. For this reason, I asked about this problem in the Caml-list.

Anyway, I am currently leaning towards choosing a solution based on Sexplib, a library that converts Ocaml values to/from S-expressions. It comes with a syntax extension that given a type t automatically writes the sexp_of_t and t_of_sexp "marshalling" functions. Making the transition from Marshal to Sexplib is therefore very straightforward. Another advantage is that S-expressions are essentially just text and are therefore human-readable. Moreover, the format is very compact (a lot more than XML!) and fairly easy to parse. Speed-wise, while obviously not being as fast as Marshal, it is still reasonably fast, especially in native code.

Suppose I have a fairly large story of value Manuscript.t. On my machine, and using Ocaml byte code, marshalling and unmarshalling this value 100,000 times takes approximately 19.68 seconds. Using Sexplib, these operations take 1175 seconds, which is about 60 times slower. However, the times in native code are respectively 17.98 and 105.3 seconds — Sexplib is less than 6 times slower than Marshal. Given the other advantages of Sexplib, these are numbers I can live with.

If you are curious about how I got these numbers (and you should — never take anyone's word at face value when benchmarks are involved!), here follows the run-down of the small programme I used for testing.

run_marshal is a function that given a manuscript, does a marshalling followed by an unmarshalling using the Marshal module. run_sexplib does the same thing but using Sexplib. Note that the latter function actually first converts the Manuscript.t into its Sexp.t representation and then this latter value into a string (and vice-versa for the reverse operation):

let run_marshal manuscript () =
        let marshalled = Marshal.to_string manuscript [] in
        let unmarshalled : Manuscript.t = Marshal.from_string marshalled 0 in
        ignore (unmarshalled)


let run_sexplib manuscript () =
        let manuscript_sexp_old = Manuscript.sexp_of_t manuscript in
        let mach_str = Sexplib.Sexp.to_string_mach manuscript_sexp_old in
        let manuscript_sexp_new = Sexplib.Sexp.of_string mach_str in
        let manuscript_new = Manuscript.t_of_sexp manuscript_sexp_new in
        ignore (manuscript_new)

I also define a generic benchmarking function that loops a provided function 100,000 times. It uses Unix.gettimeofday to retrieve timing information:

let benchmark test =
        let start = Unix.gettimeofday () in
        for i = 1 to 100000 do
                test ()
        done;
        let finish = Unix.gettimeofday () in
        let duration = finish -. start in
        duration

Finally, the main programme simply creates a new manuscript (assume that function get_manuscript returns a new parsed manuscript) and calls the benchmark function with the marshal and sexplib routines:

let () =
        let manuscript = get_manuscript () in
        let duration_marshal = benchmark (run_marshal manuscript) in
        let duration_sexplib = benchmark (run_sexplib manuscript) in
        Printf.printf "Marshal: %f\n" duration_marshal;
        Printf.printf "Sexplib: %f\n" duration_sexplib

Nleyten@1300: the joy of parsing

Dario Teixeira — Tue, 26 Feb 2008 14:59:00 GMT

I know this is bound to brand me as an incurable language geek, but I find Abstract Syntax Trees a thing of extraordinary beauty. Yeap, I am still immersed in the creation of the languages and associated parsers for the markup of Lambdium stories and comments. And I'm finding that I'm really enjoying all the stuff in this trade.

I have now completed the definition and most of parsing tools for the full-fledged Lambtex parser. As it turned out, introducing support for labelling and references forced me to rethink some of the prior assumptions I had for the Lambtex language. It was a difficult birth of sorts, and the language definition did go through several revisions until it arrived in its present state. I am now however quite pleased with the result: Lambtex manages to be both simple and powerful, gracefully handling most complex cases without burdening beginners with an arcane syntax. I will write a small manual (to be part of Nleyten's help system) soon enough — you will then get a clearer idea of what I mean.

I also realise that I'm lucky enough to be able to be creating both the language and the tools to parse it at the same time. The two play an intricate dance together, and even minor changes to the syntax of a language can have a profound effect on the parsing tools. I am also thankful to be doing this in Ocaml: the language is just perfect for compiler writing!

As for the SVN changelog at revision 1300, it reads "Adapted tokenizer to new Lambtex scanner". The Lambtex parsing chain is composed of four separate modules (as typical with most compilers): at the lowest level there is a scanner that splits the input into recognisable language atoms; closely tied to it there is a tokenizer that converts those atoms into proper parser tokens; then we have the parser proper (I am using Menhir to generate the parser; it's similar to ocamlyacc, but better); and at last there is a postprocessor that verifies that the document is semantically correct (all references point to valid targets, for example) and produces its final version.