Consider something like the aoostar R1 with Intel N100. Small and low power like a commercial consumer NAS but cheaper and you can chuck whatever OS you want.

Would you consider making the LLM/GPU monster server as a gaming desktop? Depends on how you plan to use it, you could have a beast gaming PC than can do LLM/stable diffusion stuff when not gaming. You can install loads of AI stuff on windows, arguably easier.