Things have been working great and managing these machines has never been this pleasant, except for one thing - the cooling fans which come with the PoE+ hats refuse to work out of the box with nixos.
These fans are enabled without any extra configuration under Raspberry Pi OS, so I know that they can work, it just leaves me to figure out how to make them work...
After doing a bit of research I found that due to an incompatibility with the RPi bootloader and how nixos is designed to be used, the device tree cannot be passed down from the bootloader to the kernel, and thus a dtb must be "pre-baked" before some of the optional hardware, like these fans, can be used. This means that all overlays must be applied at compile time before the final dtb is produced. Nixos provides
hardware.deviceTree.overlays to do just that.
hardware.deviceTree.overlay would be the way to go, and there's already examples of taking almost unchanged overlays from rpi/linux and applying them through this mechanism.
Unfortunately, some of the overlays that are available in raspberrypi/linux can not be successfully applied with the
fdtoverlay tool used here. It just fails with barely any output other than a not-so-helpful
FDT_NOT_FOUND error, and even running it in verbose mode with
fdtoverlay -v does not provide any additional hints.
Digging into it more, I found that these overlays actually require RPiFs own dtmerge tool for successfully application.
After a bit of fiddling, I found that exchanging the implementation of applyOverlay with one that uses
dtmerge instead of
fdtoverlay was enough to successfully produce a valid dtb, and with it, kick my rpi fans into motion.
I have created a pull-request in nixos-hardware using the previously mentioned findings as a solution.
I'm not sure if this will ever be merged, but even if it won't be, it should serve as an example to anyone who would like to get their PoE+ hat fans working under nixos.
UPDATE: It's been merged!