Check
this out. It describes how the n64's controller works. The analogue joystick consists of a pair of photo-interrupters.
A circuit to decode the output would not be trivial I expect. It would certainly involve much more than just wiring it up right.
If all you want is up/down, left/right, then you should consider a digital joystick.
It may also be easier to look for a joystick that consists of a couple of potentiometers to give position info, and they may be easier to interface to something.