The situation: you’re trying to build something, but one of your configured substituters (a.k.a binary caches) is either offline, or having a moment of being very slow. Nix doesn’t automatically time out, and skip that cache. No, you just can’t build. You want to disable the problem cache so you can get on with your life. But since you use NixOS you need to run nixos-rebuild to update your substituter settings. A rebuild means hitting the problem cache…

When I’ve run into this problem I’ve thought, “I really need a way to selectively disable a cache in the nix build command.” Previously I’ve had a hard time searching for such an option. Today I found it! Here it is:

$ nix build --option substituters "https://cache.nixos.org/ https://nix-community.cachix.org/"

or

$ nixos-rebuild build --option substituters "https://cache.nixos.org/ https://nix-community.cachix.org/"

The flag --option overrides settings that are normally read from /etc/nix/nix.conf. The idea here is instead of specifying a cache to disable, you list all of the caches that you do want to use.

Unless you are running as a “trusted user” you can’t use this method to use substituters that aren’t already configured because that would be a security problem. That means that substituter URLs need to be exactly the same as they are specified in /etc/nix/nix.conf including query parameters like ?priority.

I run into the misbehaving cache problem in two situations:

  • From time to time I get an error from cachix. I think it might be something like the cache claims to have a store path, but then actually downloading it fails. I’m not sure. Anyway the cache error makes the whole build command fail.
  • Sometimes garnix, as helpful as it is for avoiding expensive rebuilds on my slow laptop, gets very slow serving large packages like slack and google-chrome. These are unfree so they aren’t cached on cache.nixos.org which usually takes precedence over garnix for unmodified nixpkgs packages. But since I build my nixos config on garnix the unfree packages do get cached there. I could wait all day for my nixos rebuild, or I could bypass the cache, download binaries from their original URLs, and be done in seconds.
  • balsoft@lemmy.mlEnglish
    2·
    3 days ago

    I prefer to do it the other way around: only have substituters = https://cache.nixos.org/ in /etc/nix/nix.conf, add all other binary caches to trusted-substituters there, and use them only when necessary (typically by adding them to nixConfig.extra-substituters in each flake.nix that needs them). This speeds up builds significantly, because otherwise Nix checks for each path in each substituter before building it locally, which can take literal minutes if you’re building a lot of paths.

  • Corbin@programming.devEnglish
    1·
    6 days ago

    Good notes. Another trick is to replace /etc/hosts (which is usually a symlink to /etc/static/hosts) with a custom file; for example, copy all of the hosts from /etc/static/hosts and then add new hostnames for the failing caches. This can turn an indefinite network timeout into a fairly quick connection-failed error.

    Personally I think it’s a design deficit in Nix that is compounded by the serial, one-at-a-time, timeout-based way of operating. A Nix implementation should have a sense of trading off disk, bandwidth, compute, and time; a substitution should only be preferred when it is likely to save at least one of those resources, and abandoned if it isn’t making progress.