The file you edited is for retroarch emulation, but it looks like you are using the non-retroarch based emulator.
Try using the lr-mupen64plus
https://github.com/retropie/retropie-setup/wiki/Nintendo-64
Although to be honest the N64 is one of the trickier systems to get up and running as a lot of the games still arent that well supported.
Are you going to set it up with the arcade emulators?
RetroPie help guides --> https://goo.gl/Yfy8kj
Please read this before asking for help --> http://goo.gl/eLErnl