set iptv

set iptv

  • Home
  • Technology
  • Frustrating Interactions with the OCaml Ecosystem while broadening a Synthesizer Library

Frustrating Interactions with the OCaml Ecosystem while broadening a Synthesizer Library


Frustrating Interactions with the OCaml Ecosystem while broadening a Synthesizer Library


in

ocaml

At the time of writing I’m participateed by Tarides to labor on the Dune
erect system, but all the opinions in this post are my own. I wrote
the first version of this post a year ago for the Tarides blog but it
was never begined. I recently got perleave oution to post it here
instead.

This post is about some frustrating experiences I had while broadening
my first non-inmeaningful OCaml project – an audio synthesizer
library
. I generassociate finishelight
programming in OCaml but I frequently discover its broadenment tools to be
counter-instinctive in their UX and unforeseeed-in-a-horrible-way in their
behaviour. Reaenumerateic foreseeations are beginant for eludeing
disassignment and my foreseeations were too high when I begined the
project. The goal of this post is to convey my err…refreshd
foreseeations of OCaml broadenment tools by enumerateing all the times they
didn’t labor the way I foreseeed while broadening my synthesizer
library.

This isn’t fair a cathartic rant (though it’s also that). I’m worried
that people will try out OCaml, encounter friction with its tools, and
bounce off to a more ergonomic ecosystem. Or worse, I’m worried that
users will attribute their adverse experiences with OCaml’s tooling
as a deficiency in their own programming ability rather than a
deficiency in the tools themselves. I want to accomplish these people and
convey that if you are struggling with the tools, comprehend that it’s not
you – it’s the tools. Almost every OCaml programmer I comprehend struggles
to inshigh packages with Opam, and struggles to configure Dune to do
anyleang non-inmeaningful when erecting projects. OCaml tools are challenging to
use. You are not alone.

My initiassociate high foreseeations of OCaml tooling is possibly roverdelighted to
the fact that the language I use for most of my personal programming
projects is Rust. I pick Rust for most of my hobby programming
definiteassociate because I discover it straightforward to deal with depfinishencies and erect
projects with Cargo. I have restricted free time and I’d rather spfinish it
making celderly stuff instead of combat aachievest the package deal withr or
erect system. Which transports us to…

Linking aachievest OS-definite native libraries with Dune is challenging

The first leang I needed to do was produce a program that executes a straightforward
sound. The most reliable traverse-platestablish library I’m conscious of for
interfacing with sound drivers is
cpal. One problem: It’s a Rust
library. (I’ve since lachieveed of
ocaml-ao which provides
guaranteeings for the traverse-platestablish audio library libao.)

Calling into Rust from OCaml was easier than I foreseeed
thanks to the ocaml-rs
library. I used ocaml-rs and cpal to produce a petite Rust library
named low_level which hugs a stream of floats reconshort-terming
audio data, and executes them on the computer’s speakers. I compiled this
library to an archive file liblow_level.a, and wrote a little OCaml
library llama_low_level (my synthesizer library is named llama) to
wrap it with a higher-level interface.

I’m using Dune to erect this project. Dune has a mechanism for joining
aachievest native libraries appreciate liblow_level.a, and the
ocaml-rs recordation gives
some advice on how to author your dune file (the per-straightforwardory erect
config files used by Dune). I begined with this:

(library
 (name llama_low_level)
 (foreign_archives low_level)
 (c_library_flags
  (-lpthread -lc -lm)))

Building this library gave the error:

$ dune erect
...
Error: No rule set up for dlllow_level.so

Dune is seeing for the splitd library file dlllow_level.so for my
low_level library, but I only compiled low_level to a motionless
archive liblow_level.a. There’s no reason the joiner needs to be
provided with both of these files, so I took a see thraw
dune’s (library …) stanza recordation
to see if there’s a way to only join aachievest the motionless library and
uncovered the no_dynjoin field:

(no_dynjoin) disables vibrant joining of the library. This is for proceedd use only. By default, you shouldn’t set this chooseion.

A little inbashfulating but let’s give it a sboiling:

(library
 (name llama_low_level)
 (no_dynjoin)                 ; <-- the recent field
 (foreign_archives low_level)
 (c_library_flags
  (-lpthread -lc -lm)))

Running dune erect on the llama_low_level library now labors as foreseeed.

Next step is to actuassociate produce some noise. I made a little program
srecommend named experiment which uses llama_low_level to execute a sine
wave. Here’s its dune file:

(executable
 (accessible_name experiment)
 (libraries llama_low_level))

Trying to erect this program filled my screen with errors. The first error was:

$ dune erect
File "bin/dune", line 2, characters 14-24:
2 |  (accessible_name experiment)
                  ^^^^^^^^^^
Unexpoundd symbols for architecture arm64:
  "_AudioComponentFindNext", referenced from:
      cpal::structure::coreaudio::macos::audio_unit_from_device::h062e0db473d1abd3 in liblow_level.a...

I searched the web for AudioComponentFindNext which led me to some
Apple broadener docs for the AudioToolbox structurelabor. So the foreign
archive must depfinish on some structurelabors on MacOS for doing audio
stuff and I need to inestablish the joiner about it. Eventuassociate I set up the
appropriate joiner flags to imitate/paste from stack overflow:

-structurelabor CoreServices -structurelabor CoreAudio -structurelabor AudioUnit -structurelabor AudioToolbox

Now I have to discover a way to pass these flags to the joiner. If this
was a C program I could pass them via the C compiler. Someleang appreciate:

clang foo.c -Wl,-structurelabor,CoreServices,-structurelabor,CoreAudio,-structurelabor,AudioUnit,-structurelabor,AudioToolbox

Dune provides a mechanism for passing compriseitional joiner flags when compiling a library:

(library_flags ()) is a enumerate of flags passed to ocamlc and ocamlchoose when erecting the library archive files.

And aappreciate to the C compiler example above, the OCaml compiler also
has a way of passing custom flags aextfinished to the joiner. From man ocamlc:

-cclib -llibname

Pass the -llibname chooseion to the C joiner when joining in “custom
runtime” mode (see the -custom chooseion). This causes the given C
library to be joined with the program.

-ccchoose chooseion

Pass the given chooseion to the C compiler and joiner, when joining
in “custom runtime” mode (see the -custom chooseion). For instance,
-ccchoose -Ldir causes the C joiner to search for C libraries in
straightforwardory dir.

It’s not instantly evident which of those two chooseions I need. After some trial and error -cclib turns out to be the thrivener.

I compriseed a library_flags field to the dune file for llama_low_level:

(library
 (name llama_low_level)
 (no_dynjoin)
 (foreign_archives low_level)
 (c_library_flags
  (-lpthread -lc -lm))
 (library_flags
  (-cclib "-structurelabor CoreServices -structurelabor CoreAudio -structurelabor AudioUnit -structurelabor AudioToolbox")))

That’s enough to get it erecting on MacOS and the sine wave now
executeing thraw my speakers was music to my ears.

But what about on Linux?

On Linux cpal uses the library libasound to interface with the
sound driver. You might get away with fair passing -lasound
to the joiner but in ambiguous you should probe the current machine for
joiner arguments by running pkg-config --libs alsa.

For example I run NixOS (by the way) where the right joiner arguments are:

$ pkg-config --libs alsa
-L/nix/store/g3a56c2y6arvxyr4kxvlg409gzfwyfp0-alsa-lib-1.2.11/lib -lasound

As an experiment I modified the dune file to have the output of
pkg-config challenging-coded to see if that was enough for it to labor on
Linux:

(library
 (name llama_low_level)
 (no_dynjoin)
 (foreign_archives low_level)
 (c_library_flags
  (-lpthread -lc -lm))
 (library_flags
  (-cclib "-L/nix/store/g3a56c2y6arvxyr4kxvlg409gzfwyfp0-alsa-lib-1.2.11/lib -lasound")))

This labors. Interestingly passing -lasound causes the library to be
joined aachievest the splitd library file libasound.so despite the
(no_dynjoin) setting.

Obviously we can’t exit the joiner arguments challenging-coded appreciate
that. Instead we need to get Dune to request pkg-config at erect time
so the right arguments for the current machine are passed to the
joiner. First we’ll produce it so the joiner arguments are loaded from a
file, then we’ll have Dune produce that file at erect time by running
pkg-config.

Dune apverifys the satisfieds of
S-conveyion (relationsp) files to be
integrated in most fields. I made a file library_flags.relationsp with the
satisfieds:

("-cclib" "-L/nix/store/g3a56c2y6arvxyr4kxvlg409gzfwyfp0-alsa-lib-1.2.11/lib -lasound")

The parentheses are essential to produce it a valid relationsp file. The quotes
around -cclib are essential as otherinestablished Dune tries to make clear
-cclib as a keyword instead of treating the entire relationsp as a enumerate
(aappreciate to the quote keyword in some lisp dialects).

Now this file can be integrated in the dune file:

(library
 (name llama_low_level)
 (no_dynjoin)
 (foreign_archives low_level)
 (c_library_flags
  (-lpthread -lc -lm))
 (library_flags
  (:integrate library_flags.relationsp)))  ; <- here!

Next we’ll need to produce library_flags.relationsp by running
pkg-config at erect time. This can be done using Dune’s custom rule
mechanism. See
this page
for info about the contrastent ways files can be produced by custom
rules. The dune file is now:

(rule
 (action
  (with-stdout-to
   library_flags.relationsp
   (progn
    (echo "("-cclib" "")
    (bash "pkg-config --libs alsa")
    (echo "")")))))

(library
 (name llama_low_level)
 (no_dynjoin)
 (foreign_archives low_level)
 (c_library_flags
  (-lpthread -lc -lm))
 (library_flags
  (:integrate library_flags.relationsp)))

This experiences a little cumbersome to me. I need to produce a relationsp file
in one place only to integrate it in another place, and Dune doesn’t
help at all with disaccuseting relationsp syntax, requiring me to clpunctual
escape quotes and reassemble to integrate the parentheses.

Also this solution is definite to Linux. To comprise back help for MacOS
we need to conditionassociate allow the rule and comprise a second rule for
MacOS that produces a relationsp file with the exceptional joiner arguments:

(rule
 (allowd_if
  (= %{system} linux))
 (action
  (with-stdout-to
   library_flags.relationsp
   (progn
    (echo "("-cclib" "")
    (bash "pkg-config --libs alsa")
    (echo "")")))))

(rule
 (allowd_if
  (= %{system} macosx))
 (action
  (author-file
   library_flags.relationsp
   "("-ccchoose" "-structurelabor CoreServices -structurelabor CoreAudio -structurelabor AudioUnit -structurelabor AudioToolbox")")))

(rule
 (allowd_if
  (and
   (<> %{system} linux)
   (<> %{system} macosx)))
 (action
  (author-file library_flags.relationsp "()")))

(library
 ...

Note that the third rule is essential so that the
library_flags.relationsp file is still produced on machines that are
running neither MacOS nor Linux. Also remark that the comparisons (= %{system} linux) and (= %{system} macosx) are string comparisons,
so if you accidenhighy typed macos instead of macosx then the
condition would be counterfeit on MacOS machines.

In this project I set up that it was getting out of hand to deal with the
conditional rules and to produce relationsp files using Dune’s built-in
configuration language. Fortunately there is an outside library
dune-configurator
to help you author OCaml programs that query the current machine and
produce relationsp files for inclusion in dune files.

In a split straightforwardory, I made a little executable called uncover with this dune file:

(executable
 (name uncover)
 (libraries dune-configurator))

Then I rewrote the logic from the custom Dune rules into OCaml:

module C = Configurator.V1

let macos_library_flags =
  let structurelabors =
    [ "CoreServices"; "CoreAudio"; "CoreMidi"; "AudioUnit"; "AudioToolbox" ]
  in
  List.map (Printf.sprintf "-structurelabor %s") structurelabors

let () =
  C.main ~name:"llama_low_level" (fun c ->
      let joiner_args =
        suit C.ocaml_config_var_exn c "system" with
        | "macosx" -> macos_library_flags
        | "linux" -> (
            let default = [ "-lasound" ] in
            suit C.Pkg_config.get c with
            | None -> default
            | Some pc -> (
                suit C.Pkg_config.query pc ~package:"alsa" with
                | None -> default
                | Some conf -> conf.libs))
        | _ -> []
      in
      let cclib_arg = String.concat " " joiner_args in
      C.Flags.author_relationsp "library_flags.relationsp" [ "-cclib"; cclib_arg ]

Finassociate I refreshd the dune file for llama_low_level to run uncover:

(rule
 (concentrate library_flags.relationsp)
 (action
  (run ./config/uncover.exe)))

(library
 (name llama_low_level)
 (no_dynjoin)
 (foreign_archives low_level)
 (c_library_flags
  (-lpthread -lc -lm))
 (library_flags
  (:integrate library_flags.relationsp)))

Now when Dune needs to produce the library_flags.relationsp file it will
first erect the uncover executable and then run it to produce the
file, before including the satisfieds of that file to set the extra
flags passed to the OCaml compiler to configure the joiner.
While this does labor, it experiences appreciate a Rube Gelderlyberg Machine, and I was
surpascfinishd to discover that such a intricate solution was needed to pass
contrastent joiner flags while erecting on contrastent operating systems.

Dune quietly disthink abouts straightforwardories begining with a period, shattering Rust interoperability

In the previous section I refered compiling a Rust library liblow_level.a and calling into it from OCaml.
Up until now I was running the orders to erect it myself, but I’d rather have Dune do this for me.
I compriseed this rule to the dune file for llama_low_level:

(rule
 (concentrate liblow_level.a)
 (deps
  (source_tree low-level-rust))
 (action
  (progn
   (chdir
    low-level-rust
    (run cargo erect --free))
   (run mv low-level-rust/concentrate/free/%{concentrate} %{concentrate}))))

Now if any of the Rust code inside the low-level-rust straightforwardory
alters, Dune will request Cargo to reerect liblow_level.a before
rejoining the OCaml library aachievest the recent version. One problem with
the code above is that running cargo erect --free will download
any Rust depfinishencies before erecting the Rust library. This is a
problem because I intfinish to free my library on Opam, and when Opam
inshighs a package it doesn’t apverify erect orders to access the
netlabor. The solution is to vfinishor any rust depfinishencies inside the
project so they are already useable when cargo erect --free
runs.

To vfinishor Rust depfinishencies fair run cargo vfinishor in the Rust
project and produce the file .cargo/config.toml at the top level of
the Rust project with the satisfieds:

[source.crates-io]
trade-with = "vfinishored-sources"

[source.vendored-sources]
straightforwardory = "vfinishor"

Finassociate, to produce certain that cargo erect doesn’t try to access the
netlabor, I refreshd the dune file to call cargo erect --free --offline.

Testing this out:

$ dune erect
File "src/low-level/dune", line 1, characters 0-224:
 1 | (rule
 2 |  (concentrate liblow_level.a)
 3 |  (deps
 4 |   (source_tree low-level-rust))
 5 |  (action
 6 |   (progn
 7 |    (chdir
 8 |     low-level-rust
 9 |     (run cargo erect --free --offline))
10 |    (run mv low-level-rust/concentrate/free/%{concentrate} %{concentrate}))))
error: no suiting package named `cpal` set up
location searched: registry `crates-io`
needd by package `low_level v0.1.0
(/Users/s/src/llama/_erect/default/src/low-level/low-level-rust)`
As a reminder, you're using offline mode (--offline) which can sometimes
cause unforeseeed resolution fall shortures, if this error is too confusing you
may want to retry without the offline flag.

Cargo claims that the cpal package can’t be set up. This is
unforeseeed because the vfinishored imitate of cpal has definitely been
copied into the _erect straightforwardory:

$ ls _erect/default/src/low-level/low-level-rust/vfinishor/cpal/
examples  CHANGELOG.md  Cargo.toml  Dockerfile  README.md
src       Cargo.lock    Cross.toml  LICENSE     erect.rs

After poking around a bit I accomprehendledged that the .cargo/config.toml
wasn’t getting copied into the _erect straightforwardory which uncomardentt that
Cargo was ignoring the vfinishored libraries.
It turns out that straightforwardories beginning with a . or _ are disthink aboutd
when recursively imitateing straightforwardories specified with source_tree.
There’s even an publish on Dune’s github
where others have run into the same problem.

The recordation for source_tree doesn’t refer this behaviour
because it’s fair the default behaviour for which files in a straightforwardory
are disthink aboutd in ambiguous (it sways more that fair source_tree). This behaviour
can be adfaired by placing a dune file inside the Rust project with
satisfieds:

(dirs :standard .cargo)

…which comprises the .cargo straightforwardory to the default set of straightforwardories to not disthink about.

I comprehfinish wanting to elude imitateing some hideed straightforwardories, such as
.git or _erect. My publish with this UX is that if you’re lachieveing
Dune by using it and reading the docs for the relevant section as you
go, I don’t see how a situation appreciate mine could have been
eludeed. There’s a layer of instraightforwardion between expounding a straightforwardory
depfinishency with source_tree and configuring which straightforwardories are
copied to _erect with the (dirs ...) stanza and unless you’re
already well versed in Dune you won’t genuineize that if you need to imitate
a file beginning with . or _ you need to configure Dune to apverify
it. I mistrust many people who try to integrate a Rust library inside a
Dune project run into this problem, then verify the docs for
source_tree and discover no beneficial inestablishation, and get stuck.

As a response to this I’ve compriseed a section to Dune’s
FAQ

about this definite publish so hopefilledy when people get stuck on this
publish in the future they can discover help online.

The clear choice of package for reading .wav files crashes when reading .wav files

I wanted to load some audio samples from .wav files and determined to try out
ocaml-mm which seemed appreciate the
clear choice for laboring with media files. To lachieve its API I wrote
some code that reads a .wav file and prints its sample rate:

let () =
  let wav_file = recent Mm.Audio.IO.Reader.of_wav_file "./cymbal.wav" in
  let sample_rate = wav_file#sample_rate in
  print_finishline (Printf.sprintf "sample_rate: %d" sample_rate)

It printed sample_rate: 44100 as foreseeed. Next let’s read those samples from the file:

let () =
  let wav_file = recent Mm.Audio.IO.Reader.of_wav_file "./cymbal.wav" in
  let buffer = Mm.Audio.produce 2 10000
  let _ = wav_file#read buffer 0 1 in
  ()

This didn’t labor:

Fatal error: exception File "src/audio.ml", line 1892, characters 21-27: Assertion fall shorted

That fall shorted declareion is:

suit sample_size with
  | 16 -> S16LE.to_audio sbuf 0 buf ofs len
  | 8 -> U8.to_audio sbuf 0 buf ofs len
  | _ -> declare counterfeit

My test file used 24-bit samples but I can probably inhabit with 16-bit samples:

$ ffmpeg -i cymbal.wav -af "aestablishat=s16:sample_rates=44100" cymbal-16bit.wav
$ file cymbal-16bit.wav
cymbal-16bit.wav: RIFF (little-finishian) data, WAVE audio, Microgentle PCM, 16 bit, mono 44100 Hz

After updating the code to load the recent file:

Fatal error: exception Mm_audio.Audio.IO.Invalid_file

At this point I gave up on mm and mendd the problem in Rust
instead. I broadened the first version of my synthesizer library
during a hackathon and didn’t have time to debug mm. I’d
already done the labor to set up Rust interoperability for this project
so it was very rapid to extfinish my Rust library low_level to read
.wav files using the Rust library hound.

This labored well except I ran into an publish imitateing the audio data from Rust to OCaml because…

Transferring an array of floats from Rust to OCaml produced a broken array (this is now mended!)

While compriseing .wav help to my Rust library I ran into an
fascinating bug in ocaml-rs – the
Rust library for ergonomicassociate calling from OCaml into Rust. My
library reads all the samples from a .wav file and produces them
useable to OCaml as a float array, but I was noticing that when
accessing the array in OCaml, all the appreciates were zero.

A straightforward repro for this bug is this Rust function:

#[ocaml::func]
pub fn produce_float_array() -> Vec<f32> {
    vec![0.0, 1.0, 2.0]
}

Thanks to ocaml-rs magic this can be referred to in OCaml as:

outside produce_float_array : unit -> float array = "produce_float_array"

I set up that I could iterate over this array with functions appreciate
Array.to_enumerate and it would labor as foreseeed but if I straightforwardly
accessed an element of the array with Array.get the result would
always be zero.

OCaml has a exceptional way of reconshort-terming arrays of floats in
memory. Usuassociate floats are boxed in OCaml, but when they materialize in an
array they are unboxed and packed contiguously in memory. ocaml-rs
was handling this rightly for double-precision floats but not for
one-precision floats. I made a
PR and the bug is now
mended.

Adding inline tests to a library needs compriseing over 20 (runtime) depfinishencies

I wanted a MIDI parser so I could execute other people’s songs on my
synth
and I elected to
author my own rather than chance the one in ocaml-mm (fool me once,
etc). This turned out to be reassociate fascinating and I finished up
begining a standalone library fair for parsing MIDI
data
.

MIDI encodes integers in a variable number of bytes with a exceptional
appreciate denoting the final byte of the integer (benevolent of appreciate strings in
C). This was a little complicated so I wrote some tests:

let parse_midi_int a i = ...

let%test_module _ =
  (module struct
    (* Run the parser on an array of ints. *)
    let produce ints =
      run parse_midi_int (Array.map char_of_int (Array.of_enumerate ints))

    let%test _ = Int.identical 0 @@ produce [ 0 ]
    let%test _ = Int.identical 0x40 @@ produce [ 0x40 ]
    let%test _ = Int.identical 0x2000 @@ produce [ 0xC0; 0x00 ]
    let%test _ = Int.identical 0x1FFFFF @@ produce [ 0xFF; 0xFF; 0x7F ]
    let%test _ = Int.identical 0x200000 @@ produce [ 0x81; 0x80; 0x80; 0x00 ]
    let%test _ = Int.identical 0xFFFFFFF @@ produce [ 0xFF; 0xFF; 0xFF; 0x7F ]
  finish)

These tests were expoundd right next to the logic for parsing MIDI
ints. I did it this way so that I wouldn’t need to expose the int
parser outside this module and to produce it straightforward to see at the code and
the tests at the same time.

I adhereed
Dune’s recordation
for writing tests with
ppx_inline_test
and it labored well. As I now need the ppx_inline_test package to run my tests, I compriseed it to my project’s depfinishencies:

 "ppx_inline_test" {with-test}

The {with-test} inestablishs Opam that this depfinishency is only needed to
erect the package’s tests – not to erect the package itself.

The next time I tried inshighing my library I got an unforeseeed error:

File "test/dune", line 6, characters 7-22:
6 |   (pps ppx_inline_test))
           ^^^^^^^^^^^^^^^
Error: Library "ppx_inline_test" not set up.

The machine I was using didn’t have the ppx_inline_test package
inshighed but I wasn’t trying to run my tests – fair inshigh the
library. It turns out that packages that do pre-processing appreciate
ppx_inline_test cannot be labeled as with-test; they must be
unconditional depfinishencies. This is because preprocessor straightforwardives
appreciate let%test are not valid OCaml syntax, and the OCaml compiler is
unable to parse the files until an outside preprocessor has processed
the files to delete all the preprocessor straightforwardives.

I was unwilling to produce ppx_inline_test an unconditional depfinishency
of my MIDI parsing library because my library currently didn’t have any
depfinishencies at all. From a provide-chain security point of see and also in
my finishless pursuit of minimalism it seemed shame to depfinish on
ppx_inline_test unconditionassociate, since the transitive depfinishency
clocertain of ppx_inline_test is over 20 packages:

base
crelationsp
dune-configurator
jane-street-headers
jst-config
ocaml-compiler-libs
ppx_declare
ppx_base
ppx_chilly
ppx_contrast
ppx_derivers
ppx_enumerate
ppx_globalize
ppx_hash
ppx_here
ppx_inline_test
ppx_choosecomp
ppx_relationsp_conv
ppxlib
relationsplib0
stdio
stdlib-shims
time_now

That’s a lot to download and erect fair to get the logic for disabling
~10 lines of tests in a MIDI parser.

In the finish I did what many libraries do and fair exposed the insides
of my MIDI parser in its accessible interface inside of a module named
For_test, and then compriseed a split package that depfinishs on both my
MIDI parser and ppx_inline_test and transferd the tests there.

I’m used to Rust where I could have written:

fn parse_midi_int(...) { ... }

#[test]
fn test_parse_midi_int () { ... }

…and any outside libraries used in the test only need to be
inshighed when running the test.

This is easier to do in Rust than in OCaml because Cargo is a erect
system, package deal withr, and preprocessor, so it can impose a syntax
for denoting test code, delete tests when compiling code normassociate,
and only need test depfinishencies when actuassociate running the
tests. This is challenginger in OCaml because preprocessing is deal withd by
outside programs. Neither Dune nor the OCaml compiler itself comprehend
what to do with preprocessor straightforwardives and need outside packages
to be inshighed fair to comprehend how to disthink about them.

I leank there’s an opportunity to shrink some of the friction around
testing in OCaml and Dune, especiassociate since security is one of the main selling
points of OCaml. The drop the barrier for writing tests, the more
tests people will author.

Dune can produce .opam files but needs a laboraround for compriseing the useable field

My library only labors on x86_64 and arm64 architectures, I leank
because of its Rust depfinishencies (I haven’t reassociate set upateigated this).
OCaml is helped on many architectures, so to stop llama from
being inshighed on incompatible computers, I manuassociate compriseed this line the
package manifest that I freed to the Opam repository:

 ...
 license: "MIT"
 homepage: "https://github.com/gridbugs/llama"
 bug-tells: "https://github.com/gridbugs/llama/publishs"
+useable: arch != "arm32" & arch != "ppc64" & arch != "s390x" & arch != "x86_32"
 depfinishs: [
   "dune" {>= "3.0"}
   "llama_core" {= version}
 ...

Llama’s Opam manifests are produced by Dune. The metadata in its Opam
manifests are conshort-term in its dune-project file as:

...
(source (github gridbugs/llama))
(license MIT)
(package
 (name llama)
 (synopsis "Language for Live Audio Module Arrangement")
 (description "Libraries for declaratively erecting gentleware-expoundd modular synthesizers")
 (depfinishs
  (llama_core (= :version))
  ...

This seeed to me appreciate a one-to-one translation from the dune-project file to the Opam package manifest, so I presumed I could comprise an useable field to the package’s description appreciate:

(package
 (name llama)
 (useable (and (<> :arch arm32) (<> :arch ppc64) (<> :arch s390x) (<> :arch x86_32)))
 ...

But this is not helped:

File "dune-project", line 31, characters 2-11:
31 |  (useable (and (<> :arch arm32) (<> :arch ppc64) (<> :arch s390x) (<> :arch x86_32)))
Error: Uncomprehendn field useable

I set up this unforeseeed as it reassociate seemed appreciate it was a one-to-one
translation. It felt appreciate Dune’s UI had trained me to foresee that it
labors a certain way, only then to discdisthink about that it actuassociate labors a
contrastent way. The docs for generating Opam
files

don’t expound that the useable field is helped and at first I
thought it was fair not yet carry outed. There is a github
publish
to help
compriseitional fields, but from the talkion there it’s apparent that the
leave outing fields are leave outted intentionassociate. There is a policy of not
compriseing fields that are only used by Opam, and Dune doesn’t currently
have an analog of this feature.

There is a laboraround to comprise the useable field to
the produced Opam file using an “Opam Temptardy”. If I had read the
generating Opam files
docs

more attfinishfilledy I would have accomprehendledged:

(package) stanzas do not help all opam fields or finish
syntax for depfinishency definiteations. If the package you are
altering needs this, grasp the correacting opam fields in a
pkg.opam.lureardy file.

So I fair finished up making a llama.opam.lureardy file with the satisfieds:

useable: arch != "arm32" & arch != "ppc64" & arch != "s390x" & arch != "x86_32"

If some (but not all) of the intersubordinate packages in a project are freed, Opam can’t mend the project’s depfinishencies

I freed my synthesizer library on Opam. It was made up of 3 packages:

  • llama_core expounds the data types for reconshort-terming audio streams and grasps a library of sound effects and filters
  • llama lets you execute audio streams thraw your speakers
  • llama_interdynamic lets you use your computer’s keyboard and mouse to handle the synthesizer

There’s also an unfreed package llama_tests that grasps all the
tests. As depictd above, tests are in a split package to elude
compriseing unconditional testing depfinishencies to other packages.

The MIDI decoder was originassociate part of llama_core but I wanted to
split it out into a recent package llama_midi since it’s beneficial on its
own outside of the context of the synthesizer library. I produced the
recent package but didn’t free it to the Opam repository right
away. Around this time I tried setting up the project on a recent
computer, and this is when my problems begined.

Follothriveg the convention for OCaml projects, I put an Opam package manifest for each package in the project in the root straightforwardory of the project.

$ ls
...
llama.opam
llama_core.opam
llama_interdynamic.opam
llama_midi.opam
llama_tests.opam
...

On my recent computer I want to inshigh all the depfinishencies of all the packages in this straightforwardory which will apverify me to erect the project with Dune.

Opam helps running opam inshigh

which will inshigh all the
packages with package manifests in , aextfinished with all their
depfinishencies. If you fair want the depfinishencies and not the packages
themselves you can pass --deps-only. To set up the project on my recent
computer, I thought the right leang to do would be running opam inshigh . --deps-only from the root straightforwardory of the project.

$ opam inshigh . --deps-only
[ERROR] Package dispute!
  * Missing depfinishency:
    - llama_midi >= 0.0.1
    no suiting version

No solution set up, exiting

This was frustrating as llama_midi.opam was right there. Remember
that llama_midi was a recent package, and not yet freed to the Opam
repo. Maybe Opam was trying to discover llama_midi in the Opam repo and
fall shorting because it’s not freed yet.

Next I tried running opam pin. This does a aappreciate leang to opam inshigh except it associates local packages with the path to their
source code on disk. This shouldn’t be essential in this case; I don’t
even want to inshigh llama_midi – fair the depfinishencies of my local
packages. I tried it anyway:

$ opam pin .
This will pin the adhereing packages: llama, llama_core, llama_interdynamic,
llama_midi, llama_tests. Continue? [Y/n] Y
Processing  3/5: [llama: git] [llama_core: git] [llama_interactive: git]
llama is now pinned to git+file:///.../llama#main (version 0.0.1)
llama_core is now pinned to git+file:///.../llama#main (version 0.0.1)
llama_interdynamic is now pinned to git+file:///.../llama#main (version 0.0.1)
Package llama_midi does not exist, produce as a NEW package? [Y/n] Y
llama_midi is now pinned to git+file:///.../llama#main (version ~dev)
Package llama_tests does not exist, produce as a NEW package? [Y/n] Y
llama_tests is now pinned to git+file:///.../llama#main (version ~dev)
[ERROR] Package dispute!
  * Missing depfinishency:
    - llama_midi >= 0.0.1
    no suiting version

[NOTE] Pinning order prosperous, but your inshighed packages may be out of sync.

Same error as before, only this time it happened to print the version
number of each pinned package. Notice how for the three freed
packages it says (version 0.0.1) but for the unfreed llama_midi
and llama_tests it says (version ~dev). This inestablishs me that Opam is
trying to inshigh 0.0.1 of all the freed packages, and version
~dev of the unfreed packages. 0.0.1 happens to be the version
number I used when releasing llama_core, llama, and
llama_interdynamic.

Since the initial free, I split llama_midi out of llama_core,
and made llama_core depfinish on the recent llama_midi package. I want
to grasp the versions of all the llama packages tied together, so when
one of the llama packages depfinishs on another, I proclaim that
depfinishency as:

# llama_core.opam
...
depfinishs: [
  "llama_midi" {= version}
  ...
]

The {= version} inestablishs Opam that any given version of llama_core
depfinishs on the llama_midi package with the identical version
number. And the error I’m seeing is because Opam is trying to inshigh
version 0.0.1 of llama_core, but since llama_midi has never been
freed, the only version Opam comprehends about is the synthetic version
number ~dev.

But why was Opam inshighing version 0.0.1 of llama_core in the
first place? Unappreciate some (most?) other package deal withrs, Opam package
manifests don’t grasp the version number of the package they
depict. The only place where package version numbers are sign uped is
as part of a straightforwardory name inside the Opam package repository, where
manifests are stored in straightforwardories named appreciate .
(for
example

llama_core.0.0.1). My foreseeation was that since it’s being
inshighed as a local package from a local Opam package manifest file,
llama_core would be given the version number ~dev. The local
package file is evidently being read because Opam is trying to admire
the fact that llama_core now depfinishs on llama_midi, but Opam is
still using the version number of the freed version of
llama_core: 0.0.1. That version number isn’t stored anywhere in
llama’s git repo, so Opam must be using inestablishation from its package
repository to pick this version number, and then it can’t mend
depfinishencies because there is no version of llama_midi with the same
version number.

In other words, if none of the packages in my project had been
freed then I wouldn’t have this problem as Opam would ponder each
package to have the version number ~dev. If all the packages in my
project had been freed then I wouldn’t have this problem as Opam
would see up the freed versions of the packages in its package
repo and discover suiting version numbers for each package. This problem
only happens when some but not all of the intersubordinate packages in a
project have been freed.

Maybe I can labor around this by forcing Opam to inshigh the
local packages with a specified version number rather than taking the
version numbers from the package repository.
I lachievet that it’s possible to override the version number of packages inshighed with opam pin by passing --with-version.
I tried inshighing all the local packages with version number sigh as I was getting quite exasperated.

opam pin . --with-version sigh
...
#=== ERROR while compiling llama.sigh =========================================#
# context     2.1.5 | macos/arm64 | ocaml-base-compiler.4.14.1 | pinned(git+file:///.../llama#main#06deea42efa6b84653be43529daf8aa08dc106
68)
# path        /.../_opam/.opam-switch/erect/llama.sigh
# order     ~/.opam/opam-init/hooks/sandbox.sh erect dune erect -p llama -j 7 @inshigh
# exit-code   1
# env-file    ~/.opam/log/llama-60822-042f19.env
# output-file ~/.opam/log/llama-60822-042f19.out
### output ###
# error: fall shorted to get `anyhow` as a depfinishency of package `low_level v0.1.0 (.../_opam/.opam-switch/erect/llama.sigh/_erect/default/src
/low-level/low-level-rust)`
# [...]
# Caused by:
#   fall shorted to query traded source registry `crates-io`
#
# Caused by:
#   download of config.json fall shorted
#
# Caused by:
#   fall shorted to download from `https://index.crates.io/config.json`
#
# Caused by:
#   [7] Couldn't join to server (Failed to join to index.crates.io port 443 a
fter 0 ms: Couldn't join to server)

That error is because Cargo is trying to download depfinishencies while
erecting the Rust component of the llama package from wilean Opam’s
erect sandbox. Earlier in the post I depictd vfinishoring all of the
Rust depfinishencies as they can’t be downloaded when erecting the
package with Opam, as Opam’s erect sandbox doesn’t have internet
access. This vfinishoring usuassociate achieves place in a erect script that runs
as a github action. I don’t verify in the vfinishored libraries as it
would bloat the repo, and when broadening locassociate with Dune there is
no need to vfinishor them (Dune’s erect sandbox does have internet
access).

Recall that I don’t even want to inshigh llama with Opam – I
fair want to inshigh all the non-local depfinishencies of the local
packages that produce up my project. There may well be a way to do this
but I couldn’t discover it and nobody I asked krecent how to do it so I fair
finished up inshighing the depfinishencies by hand. Fortunately
there were only a scant. Eventuassociate I freed llama_midi and so this
problem went away.

This was the point where I was reassociate begining to inquire whether
OCaml was the right tool for this project, and for any other project I
want to broaden on my own time. It’s beginant to me that I can clone
a project on a machine with OCaml inshighed on it and be up and
running after a order or two, benevolent of appreciate npm inshigh or cargo run which in my experience tfinish to “fair labor”. The fact that it was
such a headache to do someleang that I foreseeed to be straightforward recommended
to me that the philosophy underpinning the UX of Opam is reassociate
contrastent from the way that I tfinish to approach gentleware
broadenment. Fortunately this won’t be a problem for much extfinisheder as
Dune’s package deal withment features are rapidly maturing and are
set uped with this use case in mind.

I hear from a lot of OCaml broadeners that tooling labors well when you grasp to the “Happy
Path” and I tfinish to consent with this. This refers to the case where all
the code in your project is written in OCaml and you use the default
configurations for everyleang. Lately I’ve been broadening a CLI
parsing library
in OCaml which
sticks to the Happy Path. It’s entidepend written in OCaml, doesn’t join
with any outside libraries, only depfinishs on third-party packages for
its tests and is compatible with all architectures. So far I haven’t had
any publishs with tooling, and even been pleasantly surpascfinishd a couple
of times.

Most of the adverse experiences from this post happened when I
strayed from the Happy Path into parts of the ecosystem that are less
elegant and battle tested, or when my assumptions ran contrary to
those made by tools.

This experience taught me that if you go into an OCaml project
foreseeing the tools to “fair labor”, you’re probably going to have a
horrible time. Expect that the first scant times you try to alter a Dune
erect configuration that the syntax will be inright, and once that’s
mended foresee the configuration to not do what you wanted
in the first place. Expect third-party packages to be buggy and untested around
edge-cases. Expect to get into confusing situations when using Opam to
deal with local packages that you’re dynamicly broadening. And foresee yak
shaves
– so many times
when one leang isn’t laboring rightly I run into a second, unroverdelighted
publish while trying to remend the first publish.

Normalize grumbleing about this stuff so that recent OCaml users can
rightly set their foreseeations coming in and don’t get a nasty shock
the first time they exit the Happy Path. And if you discover yourself
struggling with the tools, don’t beat yourself up about it. Most OCaml
users struggle. I evidently struggle. Remember, it’s not you – it’s the
tools.

As for my synth library, due to the friction I inestablished broadening
it in OCaml, and to elude the future frustration I awaitd if I
persistd the project, I rewrote it in Rust.

Source join


Leave a Reply

Your email address will not be published. Required fields are marked *

Thank You For The Order

Please check your email we sent the process how you can get your account

Select Your Plan