I have looked for a web framework shootout similar to the one that exists for computer languages. Sure, performance is hardly the deciding factor when choosing a web framework, and benchmarks don't say anything about features, robustness, and security. Note, however, that the same can be said about programming languages in general and that does not prevent people from using the shootout in pissing contests. I can, for example, easily place Ocaml on top of that list just by giving a larger weight to the gzip size of the programme (a choice not altogether arbitrary: bear in mind that gzip size is a good indicator of a language's expressiveness; the smaller, typically the more expressive a language is).

Another important caveat to consider is that the bottleneck for the typical web application lies on the database side, not on the webserver. In many cases, the webserver does little more than to fetch and to do minimal processing on the results provided by the database.

Anyway, though I can't make comparisons between these numbers and those from other frameworks using other languages (who in their right mind would want to use PHP?), it is nevertheless interesting to have an idea of how well Ocsigen applications perform. I am therefore sharing the results of some simple tests I ran on two different machines: an old Celeron running at 500 Mhz (a very slow machine by today's standards), and a modern Intel Pentium Dual E2160 running at 1.80 GHz. Note that architecture-wise, the former machine is an x86, while the latter is an AMD64.

All of the tests concern the generation of dynamic content using Ocsigen's Eliom. Though test 1 always produces the same result and could therefore easily be cached, that has not been done. The idea was to test the most demanding operation for a web application: the dynamic generation of pages (typically personalised for individual users). Also, there are two sets of results for each test; one running Ocsigen in bytecode, and another using native code. Note that because Ocsigen relies on dynlink, and Ocaml 3.10 does not support the dynamic linking of native code, I had to use the Ocaml CVS HEAD for the tests. Native dynlinking will arrive with Ocaml 3.11 (coming this summer?).

I used Siege to perform the actual benchmarking. Each test was performed with 10 concurrent client threads, and ran for 30 minutes. Siege was executed on the same machine where the Ocsigen server was located. The results are presented in terms of the number of transactions performed per second. The higher the better, of course.

Test 1: simple page

This service takes no parameters and simply outputs a page with a "bench1" header. This is the Eliom handler that creates the page:

let bench1_handler sp () () =
    Lwt.return
        (html
            (head (title (pcdata "bench1")) [])
            (body [h1 [pcdata "bench1"]]))

And here are the results obtained:

Test 2: arithmetic on integer parameters

This service takes two integers as a GET parameter, and outputs the result of some common arithmetic functions performed on those numbers:

let rec gcd a = function
    | 0 -> a
    | b -> gcd b (a mod b)


let lcm a b = (a * b) / (gcd a b)


let bench2_handler sp (a, b) () =
  Lwt.return
    (html
      (head (title (pcdata "bench2")) [])
        (body
          [
          h1 [pcdata "bench2:"];
          p [pcdata (Printf.sprintf "%d plus %d is %d" a b (a+b))];
          p [pcdata (Printf.sprintf "%d minus %d is %d" a b (a-b))];
          p [pcdata (Printf.sprintf "%d times %d is %d" a b (a*b))];
          p [pcdata (Printf.sprintf "%d divided by %d is %d" a b (a/b))];
          p [pcdata (Printf.sprintf "%d mod %d is %d" a b (a mod b))];
          p [pcdata (Printf.sprintf "The GCD of (%d,%d) is %d" a b (gcd a b))];
          p [pcdata (Printf.sprintf "The LCM of (%d,%d) is %d" a b (lcm a b))]
          ]))

Note that the client had to request a page with two GET parameters. These were generated randomly with the special shell variable $RANDOM, and stored in a urls.txt file read by siege. $RANDOM produces random integers between 0 and 32767 (yes, I know there was a chance that both "b" would be 0, causing an exception; I checked the urls.txt beforehand, making sure that didn't happen). Anyway, here are the results obtained:

Test 3: generate page with 100 pseudo-random paragraphs

The final test is a bit more demanding. It takes no parameters, and outputs a page consisting of 100 paragraphs containing both a deterministic and a random component:

let random_paragraph =
    let rng = Cryptokit.Random.device_rng "/dev/urandom"
    and to_hex = Cryptokit.Hexa.encode () in
    fun i ->
        let random_num = Cryptokit.Random.string rng 80 in
        let random_str = Cryptokit.transform_string to_hex random_num in
        p [pcdata (Printf.sprintf "This is paragraph %d: %s." i random_str)]


let bench3_handler sp () () =
    Lwt.return
        (html
            (head (title (pcdata "bench3")) [])
                (body
                    [
                    h1 [pcdata "bench3:"];
                    div (List.init 100 random_paragraph)
                    ]))

Results obtained:

Conclusion

First, there is a clear advantage to using native code (no surprise there!). Ocaml 3.11 will therefore be very welcome for Ocsigen users. Moreover, note that the byte/native code difference is even more acute on the AMD64 architecture. There is at the moment still some discussion on whether or not the AMD64 port should use the "-dlcode" compiler option by default (this option is necessary for native dynlinking). Hopefully further tests will come to the conclusion that there is little or no performance impact of using it.

It would be interesting to investigate how other languages/frameworks fare on similar tests. Even more interesting will be to repeat these tests on a more real world scenario, with an actual application that accesses a database, etc. I intend to conduct these soon on Lambdium-light.