I found the issue. I actually followed this guide (https://github.com/RetroPie/RetroPie-Setup/wiki/Setting-up-the-XBox360-controller) but instead of installed the “Xboxdrv” via the command line, I used the RetroPie Setup script. This actually besides installed the Xbox driver, also updates update.rc to run the driver.
So long story short, I undid the actions I did as described in the guide above and my controllers were put on js0 and js1. I also blacklisted the xpad module to make sure no conflicts could come from that.