Fully charged alkaline (non-rechargeable) batteries have a voltage of around 1.6-1.65v ... doesn't stay there for much time, but pretty much any device should be designed to handle that, so I think it would be safe to use 4x1.7v batteries.
If you don't want to risk it, keep in mind that a DMM would most likely have to be designed in a way that accepts regular rechargeable batteries, which have 1.2-1.35v depending on charge, so the meter should also work with 4.8 - 5.4v, therefore the DMM would probably work with just 3 x 1.7v ( total 5.1v) batteries installed and a simple wire connecting the poles of the last battery "compartment" for continuity.
I'm not sure it's really worth investing money in such batteries, I would think you can buy 2-3x the amount of regular alkaline batteries for the price of lithium ones, and replacing 4 batteries doesn't take a huge amount of time.
If you think about running dmm for long periods of times (for data logging for example), it would make more sense to make yourself an isolated power supply for your DMM, just buy a very small isolation transformer, add a bridge rectifier and a capacitor and a linear regulator and you have 5-6v (and you can just solder a couple of wires to the battery terminals. Perfectly safe and with small wires, unobtrusive, no damage to the meter etc.