NixOS on the OpenWRT one

Porting NixOS to embedded, or more correctly, non x86_64 devices, can be a difficult task. The difficulty of a port is mainly influenced by these 4 primary factors:

1. The Linux kernel

   What matters most here is that you can get the source code of a functional Linux kernel, how functional that kernel is and how mainline it is. The less functional and the less mainline, the more difficult a port will be and more things will not work. If something doesn't work due to the kernel, you'll have to resort to writing C code to fix it.

2. The uboot

   With uboot, generally, if you have the source code and it's not utterly ancient, you'll be fine. Rarely is it the case that uboot will give you trouble. It's easy to compile and generally just works.

3. The hardware itself

   Certain devices can be hard to open, locked down, not have a serial port easily accessible. You can't do much about these issues, once you've decided on a device to port.

4. Graphics or no graphics?

   If your port requires a functional GPU and display, you have to add Mesa to the list of things you'll need to worry about. If you don't need graphics, contratulations! Your project just got easier.

The OpenWRT One is a router with one 1 gigabit ethernet port and one 2.5 gigabit ethernet port. It also contains a WiFi chipset capable of AP mode and as such serves as a "WiFi router". It's released under the OpenWRT name, but manufactured by BananaPi. Its heart consists of a Mediatek MT7981B, which is a weaker cousin to the one found in the BananaPi-R4.

There two main sources of information I discovered while working on the OpenWRT One, the OpenWRT wiki and the BananaPi wiki. The first one is by the OpenWRT people and contains information specific to OpenWRT, but also some more general tid-bits. The BananaPi wiki is maintained by the vendor and, honestly not sure who, so it's more vendor brained. I referred to both when working on the NixOS port.

Firstly though, we have figure out what kernel we'll be using as a base, that can be done by digging through the OpenWRT source code, I recommend a healthy dose of ripgrep. The file we want to see is target/linux/generic/kernel-6.12 (we're operating with an older OpenWRT version, at the time of writing, the newest is kernel-6.18). Opening that file we see the exact patch revision of the kernel they're using.

LINUX_VERSION-6.12 = .74
LINUX_KERNEL_HASH-6.12.74 = 3b56eeb1dc9a437f189ca56b823be3769994f59a4ea0895b08ec0d20acaca13e

We fetch the Linux kernel tree at the exact same version that OpenWRT uses, while at it, we also fetch the OpenWRT source tree.

fetchurl {
  url = "https://cdn.kernel.org/pub/linux/kernel/v6.x/linux-6.12.74.tar.gz";
  hash = "sha256-pp/WQ1FCr9yopZLSj/qUFOSXo1Er0RtaJrScBna4CGs=";
}

fetchFromGitHub {
  owner = "openwrt";
  repo = "openwrt";
  rev = "v25.12.2";
  hash = "sha256-rygtfeSGwibRu6QJIHn/yKnACn8iTyU9jZcon3DVPuM=";
}

Now, while we do have the correct kernel tree, it's still missing a lot of patches that OpenWRT folks apply on top. We could do this easily in Nix by manually gathering the patches and pasting them into the patches argument of linuxManualConfig, however that's hard to maintain. Another way would be to utilize "import from derivation" and dynamically populate the patches argument, however IFD is slow.

As such, we can use a setup hook. A setup hook is a reusable piece of Bash code, that can be added to any nixpkgs derivation. It can register callbacks for each phase (unpack, patch, configure, build, …), which will get executed by the nixpkgs' stdenv. In those hooks, we can then dynamically enumerate and add the required patches, neat!

# this function is executed before the patch phase, we add it to `prePatchHooks` down below
_openwrtifyHook() {
  # quick helper, its helpful to see what patches we're adding
  _openwrtifyAddPatches() {
    for patch in $* ; do
      echo "  added $patch"
      patches="$patches $patch"
    done
  }

  echo "Adding patches from OpenWRT..."
  # the locations here were found manually just by searching for files ending with `.patch`
  _openwrtifyAddPatches $_openwrt_source/target/linux/generic/backport-$_linux_version_major_minor/*.patch
  _openwrtifyAddPatches $_openwrt_source/target/linux/generic/pending-$_linux_version_major_minor/*.patch
  _openwrtifyAddPatches $_openwrt_source/target/linux/generic/hack-$_linux_version_major_minor/*.patch
  _openwrtifyAddPatches $_openwrt_source/target/linux/$_linux_variant/patches-$_linux_version_major_minor/*.patch
  # just in case we need to specify any other patches and we want to apply them at this exact point in time
  _openwrtifyAddPatches "${_extra_kernel_patches[@]}"
  echo "Adding patches from OpenWRT done."

  echo "Copying files from OpenWRT..."
  # once again, what files to copy was found out manually
  cp --no-preserve=all --verbose --recursive $_openwrt_source/target/linux/generic/files/. .
  cp --no-preserve=all --verbose --recursive $_openwrt_source/target/linux/$_linux_variant/files/. .
  cp --no-preserve=all --verbose --recursive $_openwrt_source/target/linux/$_linux_variant/files-$_linux_version_major_minor/. .
  # without this line, we'll be missing the DTS, found by looking where OpenWRT stores device trees
  cp --no-preserve=all --verbose --recursive $_openwrt_source/target/linux/$_linux_variant/dts arch/arm64/boot/dts/$_linux_variant
  echo "Copying files from OpenWRT done"
}

_openwrtifyPostHook() {
  true
  # sed --in-place --expr 's/zynqmp_pm_efuse_access/zynqmp_efuse_access/g' drivers/nvmem/zynqmp_nvmem.c
}

prePatchHooks+=(_openwrtifyHook)
postPatchHooks+=(_openwrtifyPostHook)

You might have noticed, that we use variables like $_openwrt_source and $_linux_version_major_minor, but we never define them. Well, see the next code block.

{
  makeSetupHook,
  lib,
  runtimeShell,
  writeScript,
  ...
}: {
  openwrtSrc,
  variant,
  version,
  extraPatches ? [],
}:
makeSetupHook
{
  name = "openwrtify-linux";
}
(writeScript "openwrtify-linux-hook.sh" ''
  _openwrt_source="${openwrtSrc}"
  _linux_version_major_minor="${lib.versions.majorMinor version}"
  _linux_variant="${variant}"
  _extra_kernel_patches=(${lib.escapeShellArgs extraPatches})

  ${builtins.readFile ./openwrtify-linux-hook.sh}
'')

As you can see, we include the contents of openwrtify-linux-hook.sh into a larger script, which sets the correct variables. This is due to me prefering to working with shell script files, rather than shell script strings in a larger Nix file. Editors tend to not support the second case well.

Now, this section is really cargo-culted, the following configuration options were assembled from:

• what I found in my workstations /proc/config.gz (and it seemed reasonable)
• whatever I had to enable to get NixOS to not complain about missing modules.

Essentially, there were 3 different major hurdles to overcome:

1. NixOS insists on having some modules available, even if they're not strictly required, like atkkb or the modules for ext2 and ext4.
2. I wanted NFS available so I can mount the OpenWRT One's Nix store over NFS significantly speeding up iteration
3. getting the OpenWRT One to successfully enter stage-2 took a bit of convincing, I hade to enable RD_ZSTD and COMPAT_BINFMT_ELF to get it to boot

{
  # this segment is what was required to get NixOS to not complain about missing modules
  DEVTMPFS = "y";
  CGROUPS = "y";
  FHANDLE = "y";
  CRYPTO_USER_API_HASH = "y";
  DMIID = "y";
  AUTOFS_FS = "y";
  TMPFS_POSIX_ACL = "y";
  SECCOMP = "y";
  CRC32 = "m";
  EXT2_FS = "m";
  EXT4_FS = "m";
  F2FS_FS = "m";
  KEYBOARD_ATKBD = "m";
  BLK_DEV_LOOP = "m";

  # this I needed to boot the NixOS initrd and enter stage-2
  RD_ZSTD = "y";
  COMPAT_BINFMT_ELF = "y";
}

Finally, we can get to actually building a kernel with all the setup we've done.

We need to assemble a Kconfig file from the options we have available as a Nix attribute set, we can do that with some use of lib, prefixing each key, value pair with CONFIG_ then making a file.

lib.pipe (import ./kconfig.nix) [
  (lib.mapAttrsToList lib.nameValuePair)
  (lib.concatMapStringsSep "\n" ({
    name,
    value,
  }: "CONFIG_${name}=${value}"))
  (x: writeText "extra-defconfig" (x + "\n"))
]

Recall that we implemented a setup hook to openwrt-ify any given Linux kernel checkout, we'll use that now. Sadly we have to do the patching in a different derivation from the main kernel build derivation. This is due to the linuxManualConfig function actually outputting a group of derivations, rather than just one and there being no easy way to apply a setup hook to all of them. Trying to do a naive x.overrideAttrs (old: { nativeBuildInputs = (old.nativeBuildInputs or []) ++ [setupHook]; }) will cause the derivation which builds the Kconfig to not use the patched kernel tree, leading to weirdness.

Instead we make a super simple auxilliary derivation with just 3 phases, unpackPhase, patchPhase, and installPhase.

• unpackPhase - the nixpkgs stdenv will unpack $src into the derivation build directory (/build, $PWD)
• patchPhase - this is where our setup hook triggers, openwrt-ifying the kernel tree
• installPhase - we copy $PWD to $out, sadly we have to do this copy

stdenv.mkDerivation {
  name = "linux-openwrt-src";
  inherit version;
  src =
    fetchurl {
      url = "https://cdn.kernel.org/pub/linux/kernel/v6.x/linux-6.12.74.tar.gz";
      hash = "sha256-pp/WQ1FCr9yopZLSj/qUFOSXo1Er0RtaJrScBna4CGs=";
    };

  nativeBuildInputs = [
    (openwrtifyLinux {
      inherit openwrtSrc version;
      variant = "mediatek";
    })
  ];

  phases = [
    "unpackPhase"
    "patchPhase"
    "installPhase"
  ];

  installPhase = ''
    mkdir $out
    cp --recursive . $out
  '';
}

Now we need to actually merge the extra Kconfig we have built before, we cannot do this during evaluation as that would be IFD again, rather we make another helper derivation.

runCommand "openwrt-one-defconfig" {} ''
  # the ^# is important, otherwise efivarsfs wont be built, i dont know why
  cat ${openwrtSrc}/target/linux/mediatek/filogic/config-${lib.versions.majorMinor version} ${openwrtSrc}/target/linux/generic/config-${lib.versions.majorMinor version} ${extraConfig} \
    | grep -Ev 'CONFIG_MFD_AIROHA_AN8855|CONFIG_MFD_CORE|CONFIG_JFFS2_LZMA|^#' > $out
''

And to top it all off, we need to use linuxManualConfig, instead of buildLinux which doesn't allow us to specify the exact Kconfig we want to use. The NixOS default Kconfig is geared towards desktop usecases, it will either:

1. not build
2. not work at runtime
3. produce needlessly kernel images with drivers for hardware the OpenWRT One doesn't have

You may notice a importFromDerivation = true, sadly, that is required unless we want to vendor the Kconfig we're given by OpenWRT into our Nix code. Ideally there would be no IFD whatsoever, but one won't kill us.

Now, let me present the final file we've been slowly building up to.

{
  linuxManualConfig,
  fetchFromGitHub,
  fetchurl,
  lib,
  stdenv,
  runCommand,
  writeText,

  openwrtifyLinux,
  version ? "6.12.74"
}: let
  openwrtSrc =
    fetchFromGitHub {
      owner = "openwrt";
      repo = "openwrt";
      rev = "v25.12.2";
      hash = "sha256-rygtfeSGwibRu6QJIHn/yKnACn8iTyU9jZcon3DVPuM=";
    };

  extraConfig =
    lib.pipe (import ./kconfig.nix) [
      (lib.mapAttrsToList lib.nameValuePair)
      (lib.concatMapStringsSep "\n" ({
        name,
        value,
      }: "CONFIG_${name}=${value}"))
      (x: writeText "extra-defconfig" (x + "\n"))
    ];
in
  linuxManualConfig {
    src =
      stdenv.mkDerivation {
        name = "linux-openwrt-src";
        inherit version;
        src =
          fetchurl {
            url = "https://cdn.kernel.org/pub/linux/kernel/v6.x/linux-6.12.74.tar.gz";
            hash = "sha256-pp/WQ1FCr9yopZLSj/qUFOSXo1Er0RtaJrScBna4CGs=";
          };

        nativeBuildInputs = [
          (openwrtifyLinux {
            inherit openwrtSrc version;
            variant = "mediatek";
          })
        ];

        phases = [
          "unpackPhase"
          "patchPhase"
          "installPhase"
        ];

        installPhase = ''
          mkdir $out
          cp --recursive . $out
        '';
      };

    inherit version;
    modDirVersion = version;

    configfile =
      runCommand "openwrt-one-defconfig" {} ''
        # the ^# is important, otherwise efivarsfs wont be built, i dont know why
        cat ${openwrtSrc}/target/linux/mediatek/filogic/config-${lib.versions.majorMinor version} ${openwrtSrc}/target/linux/generic/config-${lib.versions.majorMinor version} ${extraConfig} \
          | grep -Ev 'CONFIG_MFD_AIROHA_AN8855|CONFIG_MFD_CORE|CONFIG_JFFS2_LZMA|^#' > $out
      '';

    allowImportFromDerivation = true;
  }

With that we have successfully built the kernel, we can move onto U-Boot.

nix-build -A kernel -Q

/nix/store/ipwgkhmgc9kncdmig3dhfic20dhy7ijq-linux-aarch64-unknown-linux-gnu-6.12.74

Generally assembling the correct U-Boot, should be less work than the kernel 🤞.

Disclaimer, the following is once again largely cargo-culted. It is an adaptation of the code I had for the BananaPi-R4 from a few years ago and I frankly do not remember how I arrived at the exact settings. But I think the source was mostly the BananaPi-R4 vendor wiki and then frank-w's wiki.

Before we dive in, some backstory. On arm64, unusually, uboot isn't the first bootloader that is booted. That would be the "ATF" or "Arm Trusted Firmware" for long. It is what is responsible for initialising the hardware, stuff like DRAM training and such. Only once that's complete, it hands over to U-Boot a mostly initialized board. U-Boot is merely responsible for finding and loading the next stage, which is Linux in our case.

This is similar to how on x86 the firmware on the motherboard, generally referred to as "BIOS", actually does all the hardware initialization and the next bootloader (limine, GRUB, systemd-boot, or even U-Boot) is handed a already initialized device.

So to begin, we first need to get the source code of ATF and of U-Boot, the exact revisions we need can be found in the files package/boot/arm-trusted-firmware-mediatek/Makefile and package/boot/uboot-mediatek/Makefile. We'll also need to OpenWRT-ify the U-Boot and we could also OpenWRT-ify the ATF, though I haven't had the need to do that for the OpenWRT One. The support for the newer Mediatek chips is quite good in the mtk-openwrt/arm-trusted-firmware repository.

fetchFromGitHub {
  owner = "mtk-openwrt";
  repo = "arm-trusted-firmware";
  rev = "78a0dfd927bb00ce973a1f8eb4079df0f755887a";
  hash = "sha256-m9ApkBVf0I11rNg68vxofGRJ+BcnlM6C+Zrn8TfMvbY=";
}

fetchFromGitHub {
  owner = "u-boot";
  repo = "u-boot";
  rev = "v2024.10";
  hash = "sha256-UPy7XM1NGjbEt+pQr4oQrzD7wWWEtYDOPWTD+CNYMHs=";
}