If you really want to stick to UFW, you can ignore me, but this looks like a situation where finding another firewall may be best. UFW is a front end for IPtables and is mostly meant for desktop or simple server app usage. I’d recommend Shorewall, which is also a front end for IPtables but implements a zone based firewall and allows for more complex setups to be handled easier than with UFW. You can put your podman containers into a zone and define all of the network access you need for that zone separate from the host system.
For so many Linux server packages I find the manual to be more of a reference than a guide, so not very useful if you’re just getting started and aren’t sure what to do, but Shorewall is an exception, its manual is wonderful and Tom the creator really goes into detail about how to fit it into many different setups.
https://shorewall.org/GettingStarted.html
You’ll probably want to follow the two interface guide, the two interfaces in your case are your public IP interface, and the virtual interface connected to the Podman network side. You’ll essentially treat shorewall as a firewall/router for your Podman containers which will act as your “LAN” in this case. The warning about not installing Shorewall on a remote system is not to be ignored, you’re generally fine to install the package, but do not start the shorewall service without first setting up some rules to allow SSH. The safest way is to log in via your VPS console instead of SSH to keep you from getting locked out. Most VPS providers have some sort of out-of-band connection utility like VNC or a simple console access you’ll want to use.