If you really want to stick to UFW, you can ignore me, but this looks like a situation where finding another firewall may be best. UFW is a front end for IPtables and is mostly meant for desktop or simple server app usage. I’d recommend Shorewall, which is also a front end for IPtables but implements a zone based firewall and allows for more complex setups to be handled easier than with UFW. You can put your podman containers into a zone and define all of the network access you need for that zone separate from the host system.
I would configure a Backblaze B2 bucket and copy your repos and configs there, should be dirt cheap compared to a VPS and very durable.