OK - so I now turn my attention to 3.7v LiPoly batteries.
If I can get one to fit in the case, I'm presumably going to need a small circuit to knock it down to 2v and also allow charging when in the walkman.
Where do I start with this please?
Why do you need to knock it down to 2V? I don't think the motor will even turn at 2V. Yesteryear semiconductor don't operate at 2V.
You can temporary wire a 2V into the circuit board to see if it will operate.
As someone has said, 3 batteries are connected together in series to form at least 6V to operate this player.
As for the lipo, 2 in series will form over 7V, so may be a diode IN4001 in series will drop the volt to 6V region but I am sure it will work fine for your player.
So you should check and confirm the voltage the player is designed to operate, definitely not 2V.
Edit: If you ever dismantle any of those GPS/Car camera etc.. you will find these bare unprotected small capacity lipo in it. Because the VA is low, you don't exactly need to be overly concerned or worried about it.