Fixing broken Haskell packages in Nixpkgs

It doesn’t matter who you are, it doesn’t matter when or where, sooner rather than later you’ll get a broken Haskell package in your Haskell project, what will you do when that happens? Fear not, this is what you do…

I won’t assume nothing, so let’s start with a default.nix template you can use to build your Cabal project using Nix:

{ nixpkgs ? import <nixpkgs> { }, ...}:

nixpkgs.haskellPackages.callCabal2nix "my-app" ./. { }

We’re doing only two things there; first, importing the <nixpkgs> available and using the callCabal2nix function that creates a Nix derivation from our Cabal project (so we don’t have to use the cabal2nix binary), please note that relying on <nixpkgs> is discouraged because it gets resolved from the $NIX_PATH and is therefore machine dependent, but let’s ignore that for simplicity, pinning is not the main focus of this post.

Now, if we try to build this (using nix-build) as part of a real Haskell project with a *.cabal file on it, the dependencies will get resolved to Nix derivations under nixpkgs.haskellPackages.* and you will need to be very lucky to not hit any errors. In my case beam-core is giving issues:

error: Package ‘beam-core-0.9.1.0’ in /nix/store/80rw6id4d16bx3nhghz8h5r9vkrc2vbz-source/pkgs/development/haskell-modules/hackage-packages.nix:39342 is marked as broken, refusing to evaluate.

a) To temporarily allow broken packages, you can use an environment variable
   for a single invocation of the nix tools.

     $ export NIXPKGS_ALLOW_BROKEN=1

b) For `nixos-rebuild` you can set
  { nixpkgs.config.allowBroken = true; }
in configuration.nix to override this.

c) For `nix-env`, `nix-build`, `nix-shell` or any other Nix command you can add
  { allowBroken = true; }
to ~/.config/nixpkgs/config.nix.

The Nixpkgs' Haskell infrastructure, has a “self defense” mechanism that flags faulty packages as broken, this is so we don’t have to wait for long build times just to get a build error at the end, however, to know what’s wrong we need to do the build anyways:

$ NIXPKGS_ALLOW_BROKEN=1 nix-build

If you wish to not use the NIXPKGS_ALLOW_BROKEN env var for whatever reason, remember you can pass the desired config while importing Nixpkgs in your default.nix, e.g. import <nixpkgs> { config = { allowBroken = true; }; } will achieve the same thing.

Near the bottom of the output you could see what’s wrong:

Setup: Encountered missing or private dependencies:
dlist >=0.7.1.2 && <0.9

builder for '/nix/store/v42aw8qs7b2klhchn49rc1dgas8ksj3z-beam-core-0.9.1.0.drv' failed with exit code 1

It says dlist (a dependency of beam-core) is not available, we can check by trying to build it:

$ nix-build -f "<nixpkgs>" -A haskellPackages.dlist
/nix/store/mndwim1ffv509w5mn5dpi9nzq111aai5-dlist-1.0

So we do have dlist but we can’t satisfy the version constraints imposed by beam-core, it wants >=0.7.1.2 && <0.9 but the version available in Nixpkgs is 1.0, no can do!

Overriding Haskell packages

At this point let me uncover the lies, we’re not actually going to fix the broken package, we’re just going to hack our way to victory. The Nixpkgs Haskell infrastructure has little functions that let us override attributes of the package being built, what we could try here, is to use the nixpkgs.haskell.lib.doJailbreak function so it gives back a modified package with unbound constraints, we can try it right from the terminal:

$ nix build --no-link '(let pkgs = import <nixpkgs> {}; in pkgs.haskell.lib.doJailbreak pkgs.haskellPackages.beam-core)'

Tada! it doesn’t throw the error any more! who would have thought beam-core builds just fine with dlist-1.0, so now we just need to “inject” this modified package into our dependencies, at first we could think of trying something like this:

{ nixpkgs ? import <nixpkgs> { }, ...}:

let
  beam-core = nixpkgs.haskell.lib.doJailbreak nixpkgs.haskellPackages.beam-core;
in
  nixpkgs.haskellPackages.callCabal2nix "my-app" ./. { inherit beam-core; }

Or this:

{ nixpkgs ? import <nixpkgs> { }, ...}:

let
  beam-core = nixpkgs.haskell.lib.doJailbreak nixpkgs.haskellPackages.beam-core;
in
  (nixpkgs.haskellPackages.callCabal2nix "my-app" ./. { }).override { inherit beam-core; }

But it doesn’t work like that, because we need to override the beam-core package from the nixpkgs.haskellPackages set, hopefully there’s an easy way to do this:

{ nixpkgs ? import <nixpkgs> { }, ...}:

let
  # Create a modified Haskell package set specifically tailored to the project
  hpkgs = nixpkgs.haskellPackages.override {
    overrides = hself: hsuper: {
      # Can add/override packages here
      beam-core = nixpkgs.haskell.lib.doJailbreak hsuper.beam-core;
    };
  };

in hpkgs.haskellPackages.callCabal2nix "my-app" ./. { }

By using override on nixpkgs.haskellPackages we can provide an overlay which overrides haskell packages from the existing Haskell package set, so in essence hpkgs refers to this modified version of nixpkgs.haskellPackages, you can call it myHaskellPackages or whatever, the important thing is that now beam-core refers to our overridden version. Also note how we use hpkgs to resolve the callCabal2nix function we’re calling to build our project, with all of this set, now when we build we amaze:

stopPostgres cannot be connected to
prettyPrintConfig seems to work
prettyPrintDB seems to work
can support backup and restore FAILED [1]

Failures:

  test/Main.hs:748:3:
  1) can support backup and restore
       uncaught exception: StartError
       StartPostgresFailed (ExitFailure 1)

  To rerun use: --match "/can support backup and restore/"

Randomized with seed 569491683

Finished in 18.1076 seconds
36 examples, 1 failure
Test suite test: FAIL
Test suite logged to: dist/test/tmp-postgres-1.34.1.0-test.log
0 of 1 test suites (0 of 1 test cases) passed.
builder for '/nix/store/l6i5bwqn2a6z560zl1j71n5czjnf30fg-tmp-postgres-1.34.1.0.drv' failed with exit code 1
cannot build derivation '/nix/store/vn79gqvz07i8cs9chrb5m77iwmmjcfd4-beam-postgres-0.5.1.0.drv': 1 dependencies couldn't be built

Errors, and more errors… but hey! at least is not beam-core any more!, this time the failure occurred while building beam-postgres, a dependency of it couldn’t be built, if you look carefully we can deduce what’s wrong:

Test suite test: FAIL
Test suite logged to: dist/test/tmp-postgres-1.34.1.0-test.log
0 of 1 test suites (0 of 1 test cases) passed.

Note that the package builds fine, it is the tests which are failing and thus failing the whole operation. Once more, the Nixpkgs Haskell infrastructure has us covered since it provides the nixpkgs.haskellPackages.dontCheck function that gives us a modified package in which tests won’t be run, let’s override beam-postgres then:

{ nixpkgs ? import <nixpkgs> { }, ...}:

let
  # Create a modified Haskell package set specifically tailored to the project
  hpkgs = nixpkgs.haskellPackages.override {
    overrides = hself: hsuper: {
      # Can add/override packages here
      beam-core = nixpkgs.haskell.lib.doJailbreak hsuper.beam-core;
      beam-postgres = nixpkgs.haskell.lib.dontCheck hsuper.beam-postgres;
    };
  };

in hpkgs.haskellPackages.callCabal2nix "my-app" ./. { }

And test it out:

$ nix-build
/nix/store/f2s0rl673ngd1d5kyzd010cw0qzixy6h-my-app-0.1.0.0

🥲 We’re done with the main topic of this post, but lets do a final improvement; if we’re overriding the Haskell package set and our project is a Haskell project, we can include our own project just as another package in the package set:

{ nixpkgs ? import <nixpkgs> { }, ...}:

let
  # Create a modified Haskell package set with our app in it
  hpkgs = nixpkgs.haskellPackages.override {
    overrides = hself: hsuper: {
      my-app = hself.callCabal2nix "my-app" ./. { };
      beam-core = nixpkgs.haskell.lib.doJailbreak hsuper.beam-core;
      beam-postgres = nixpkgs.haskell.lib.dontCheck hsuper.beam-postgres;
    };
  };

in hpkgs.my-app

Note how we use hself or hsuper instead of nixpkgs.haskellPackages while overriding the package set, the distinction is quite subtle, use hsuper to refer to packages you want to override and hself to refer to the set you’re overriding once the overrides have been applied, so hself in this context contains the overridden beam-core and beam-postgres packages.

Finally, just remember that instead of referring to haskellPackages we could specify the compiler for which the package set is for, so you can replace any mention of haskellPackages with haskell.packages.ghc8104 to have a package set that is specifically built for GHC 8.10.4.

Fixing the package in Nixpkgs

At this point we can move on with our lives, we’ve fixed it for ourselves, but what if we want to submit the fixed package so others could benefit of this? There’s an excelent video on this very topic, for easier reference, let me provide a resumed text version of it.

The first thing to do here, is to fork the Nixpkgs repository and clone your fork, switch to the haskell-updates branch, which is the very “edge” branch for Haskell packages:

$ git clone [email protected]:my-user/nixpkgs.git
$ cd nixpkgs
$ git checkout -b haskell-updates --track origin/haskell-updates

Within this branch we need to try to build the broken packages because it might be the case that someone else already fixed them:

$ nix-build --no-out-link -A haskellPackages.beam-core --arg config '{ allowBroken = true; }'
...
Setup: Encountered missing or private dependencies:
dlist >=0.7.1.2 && <0.9

In this case I confirm the package is still broken, so this makes me wonder, how do Haskellers using Nix, talk to a database in a type-safe way? Let’s not derail, and apply the same overlay in the pkgs/development/haskell-modules/configuration-common.nix file, look for your package, in my case I see there’s no override for beam-core so I add it in:

# Lots of packages ...

# Requires dlist <0.9 but it works fine with dlist-1.0
beam-core = doJailbreak super.beam-core;

# Lots of packages ...

And we also remove the package from the broken packages section in pkgs/development/haskell-modules/configuration-hackage2nix/broken.yaml, so that Nixpkgs allows to build the package without adding the allowBroken = true; config thing, thus removing the broken flag on the package.

One last test to check this works:

$ nix-build --no-out-link -A haskellPackages.beam-core --arg config '{ allowBroken = true; }'
...
/nix/store/cq4sbqbd9w65y2dr905wybk8vwrnnsrz-beam-core-0.9.1.0

Nice, now we can push that to our fork, and create a pull request 🏃‍♂️.

Links

Really all you need to know it’s already out there, sadly it is scattered all over the internet, these are some of the helpful links that made it click for me:

• How to create nix builds for your won private haskell packages.
• How to fix broken Haskell packages?