Simple solution: low dropout voltage regulator, e.g. LM2940 (or any similar). Drawback: need at least 5.5V at the input, so operated from 4 batteries the output will quickly stop regulating.
Better solution: Use a buck-boost converter, e.g. LTC3534 (500mA) or LTC31112 (2.5A).
Drawback: not so simple.