Call it marshalling, pickling, serialisation, or whatever
else you wish; this operation — where a value
(or "variable" in non-FP languages) is extracted from
a programme at runtime in order to be stored in disk or
transferred through the wire — is critical for many
applications. Most programming languages therefore include
in their standard libraries some means of performing it.
Ocaml ships with the venerable Marshal module. It is
extremely simple to use: suppose we wished to convert
into a string str a marshalled representation of a
value manuscript of type Manuscript.t. This would
suffice: let str = Marshal.to_string manuscript
[]. The inverse operation — taking a string
with a marshalled representation and converting back
into a programme value — is also dead easy: let
manuscript : Manuscript.t = Marshal.from_string str 0.
The attent observer will have noticed the type annotation
in the unmarshalling example above; in general,
unmarshalling does require an explicit type annotation
(which otherwise is rarely needed in Ocaml). The reason
is that the type inference mechanism may not have enough
information to determine what is the type of the marshalled
representation (imagine, for example, that the routines
performing the marshalling and unmarshalling reside in
different programmes!). And here lies the Achilles'
heel of marshalling in Ocaml: should the programmer make
a mistake in specifying the type to be unmarshalled, the
programme will most surely segfault. This problem also
occurs if a version of the programme stores a marshalled
value, which is then read back by a subsequent of the
same programme where the value's type has been modified
(even if ever so slightly).
Over the years, there appeared a number of extensions
to the Ocaml language offering type-safe marshalling
(
HashCaml may be the most well-known, but it's not
the only one). And judging from comments by Xavier Leroy
(the primary developer of the Ocaml language) at the Ocaml
users meeting in Paris this last January, there's a
good chance that type-safe marshalling will make it into
the core language in the near future.
In Lambdium-light, stories and comments are stored in
the database backend using the marshalled representation
offered by the Marshal module. While this works and is
extremely fast, it does have the problem of not being
very resilient to future changes in the story and comment
format. Therefore, I have been looking at alternatives to
Marshal that provide some degree of backwards compatibility
while not sacrificing too much speed.
I suspect that the XML-fanboys in the audience
wouldn't even think twice, but personally I am far
from finding XML a good solution for many of the
problem domains where it is applied. For this reason, I
asked about this problem in the Caml-list.
Anyway, I am currently leaning
towards choosing a solution based on Sexplib,
a library that converts Ocaml values to/from S-expressions.
It comes with a syntax extension that given a type
t automatically writes the sexp_of_t and
t_of_sexp "marshalling" functions. Making the
transition from Marshal to Sexplib is therefore very
straightforward. Another advantage is that S-expressions
are essentially just text and are therefore human-readable.
Moreover, the format is very compact (a lot more than
XML!) and fairly easy to parse. Speed-wise, while
obviously not being as fast as Marshal, it is still
reasonably fast, especially in native code.
Suppose I have a fairly large story of value
Manuscript.t. On my machine, and using Ocaml byte
code, marshalling and unmarshalling this value 100,000
times takes approximately 19.68 seconds. Using Sexplib,
these operations take 1175 seconds, which is about 60
times slower. However, the times in native code are
respectively 17.98 and 105.3 seconds — Sexplib is
less than 6 times slower than Marshal. Given the other
advantages of Sexplib, these are numbers I can live with.
If you are curious about how I got these numbers (and you
should — never take anyone's word at face value when
benchmarks are involved!), here follows the run-down of
the small programme I used for testing.
run_marshal is a function that given a manuscript,
does a marshalling followed by an unmarshalling using
the Marshal module. run_sexplib does the same
thing but using Sexplib. Note that the latter function
actually first converts the Manuscript.t into its
Sexp.t representation and then this latter value
into a string (and vice-versa for the reverse operation):
let run_marshal manuscript () =
let marshalled = Marshal.to_string manuscript [] in
let unmarshalled : Manuscript.t = Marshal.from_string marshalled 0 in
ignore (unmarshalled)
let run_sexplib manuscript () =
let manuscript_sexp_old = Manuscript.sexp_of_t manuscript in
let mach_str = Sexplib.Sexp.to_string_mach manuscript_sexp_old in
let manuscript_sexp_new = Sexplib.Sexp.of_string mach_str in
let manuscript_new = Manuscript.t_of_sexp manuscript_sexp_new in
ignore (manuscript_new)
I also define a generic benchmarking function that
loops a provided function 100,000 times. It uses
Unix.gettimeofday to retrieve timing information:
let benchmark test =
let start = Unix.gettimeofday () in
for i = 1 to 100000 do
test ()
done;
let finish = Unix.gettimeofday () in
let duration = finish -. start in
duration
Finally, the main programme simply creates a new manuscript
(assume that function get_manuscript returns a
new parsed manuscript) and calls the benchmark function
with the marshal and sexplib routines:
let () =
let manuscript = get_manuscript () in
let duration_marshal = benchmark (run_marshal manuscript) in
let duration_sexplib = benchmark (run_sexplib manuscript) in
Printf.printf "Marshal: %f\n" duration_marshal;
Printf.printf "Sexplib: %f\n" duration_sexplib
]]>
