It is slow as hell and I don't think I need the benefits.
No need for overlays. Instead create own packages and call them via callPackage. That will keep them simpler and allows dependency injection. I can follow existing patterns from nixpkgs, etc.