That's really strange? I'm a qualified electrician (UK - mainly domestic) and I've never had an issue with the insulation resistance test mode. Like you say, if there's no internal short-circuit/leakage then I'm not sure what the problem could be. Are you using it at 500V for domestic 230V R1-R2 & R1-Rn tests? If so, as long as you're over 2MOhm then it's compliant in the UK. Some approved multi-function testers only measure up to 10MOhm anyway, 999MOhm is probably overkill in this case.
Another point (I'm sure you've checked this, not insulting you're inelegance in any way as I've done it myself) sometimes if neon's (among other things such as PIR's, etc) are present on circuits, they can have a fairly high impedance and can fool you into thinking the test equipment is faulty when it's not. Perhaps try a test on a bare piece of cable to make sure this is not the case... I'm sure you already have, but I was fooled years ago.
My advice, test it against another insulation resistance tester on the same circuit (not at the same time, obviously) and compare results to check the accuracy - if all is good, carry on and maybe look at getting it calibrated; however if it's not then perhaps look at buying a new multi-function tester. I'd personally recommend the Di-Log testers, they are far cheaper than Fluke or Megger brands but conform to the same standards and often have a lot more features in terms of value-for-money (like auto-RCD testing, combined PFC, PSC & Zs/Ze, etc....)
Hope it all works out well anyway!