Rewrite X-Forwarded-For header to remove port

Question

I am running a Crowd/Confluence server behind Azure App Gateway LB. When making request to my crowd server, Azure is forwarding IP:PORT for the X-Forwarded-For in the header. There is an issue in crowd/confluence handling of IP's that makes it think this is an IPv6. Atlassian is aware of the issue but has yet to implement a fix so I am left to correct it myself. I do not have the ability to modify Azure's config to turn off the port sending in the header so the other option that other users have done is to run Nginx reverse proxy and strip the port myself.

With Nginx installed and configured to reverse proxy, I am not able to figure out the proxy setting needed to strip that port. I am logging as such to see the IP:PORT in the header:

        log_format  main        '$remote_addr - $remote_user [$time_local] "$request" '
                                '$status $body_bytes_sent "$http_referer" '
                                '"$http_user_agent" "$http_x_forwarded_for"';

I have tried variations of the proxy_set X-Forwarded-for in the default conf and none seem to work or even make a change to the log output.

        location /crowd {
                proxy_pass http://localhost:8095/crowd;
                proxy_set_header X-Forwarded-For $remote_addr;
                #proxy_set_header X-Forwarded-For "";
                #proxy_set_header X-Forwarded-For "0.0.0.0" ;
        }

Is there a way to just override the XFF entirely to set it as the source IP or a different way to strip the port off of the end of the IP?