http://blog.petrockblock.com/forums/topic/usb-snes-controller-d-pad-not-working-in-games/#post-97091
So I did as Floob said and removed all config files in /opt/retropie/emulators/retroarch/configs/*.cfg
then ran
sudo /home/pi/RetroPie-Setup/retropie_setup.sh
and did the configuration for my controller.
Now it works as it should be 