@xii-nyth well the name was missing the .scale, so tried changing to a different image instead of modifying the one in assets. Oddly neither of them worked, I guess that xml is unused now because I didn't see anything else change to the image I set.
Afterwards I went to change the shortcut icon back to the original and I forgot where the image was, so I selected the exe rather than an ico. When pinned to start it chose a smaller scale just as I was trying to do manually.
Turns out that this is treated differently than an ico so you can just change the shortcut "icon" as sgunhouse suggested. Also I found that for apps like firefox which already use an image file outside of the icon group bound to a program, that image overrides this. It however also works on discord for example.
0f9d89f9-f87e-486e-8dfd-939a34125416-image.png