Hi Tomislav,
I looked at Transmission Remote GUI, and finally managed to configure it to accept reverse proxy host connection. Btw: the proxy tab in Transmission Remote GUI is not to be confused with reverse proxy, it is not related.
First thing, make sure you have the default nginx reverse proxy location, without any modifications, as it is in the guide. If you have done any changes, revert those back.
Here is a screenshot of how to setup connection in Transmission Remote GUI:
Obviously, enter the IP address of your RPi where transmission is located.
For port number you select 80 as it is the default port for nginx web server acting as a reverse proxy. You add the Transmission username and password, and see the other settings, adjust everything as on the screenshot.
This is if you are using a plain, http nginx configuration over port 80. If you use https, then most likely you need to use port 443 and not sure about the SSL option. I was not able to try https connection atm. If you have a DDNS for your nginx reverse proxy, then you can use the DDNS address instead of the IP and you should be able to access Transmission from outside of your local network too. In this case, I strongly recommend to use https to encrypt connection.
Let me know if this works for you!