My guess is that the efficiency of your converter is not good at the current you're drawing. I just finished converting my scope to lipo power using an MT3608 boost converter module, and it seems to work fine going from a single cell lipo to 8.2V. It's more expensive, but runs at 1.2 MHz, and from the datasheet it looks like you'd be at over 90% efficiency even at low current.
But you have several things working against you as far as battery life is concerned when using the boost converter versus powering directly from the battery:
Your circuit probably uses less current when run at 3.5V than at 5V.
Even with 100% efficiency, converter power in must at least equal power out. So if you need 25ma at 5V for your circuit, you'll need 36ma at 3.5V coming out of the battery. Powered directly from the battery, you might only need 20ma at 3.5V.
No converter is 100% efficient, and may require tweaking for the current you will be drawing.
Drawing more current out of the battery drops its instantaneous voltage a bit more, which makes it hit the protection low-voltage disconnect earlier. So you're actually using a lower percentage of the battery's capacity.
For all these reasons, it wouldn't be at all surprising if battery life was cut in half using the converter, even with a very efficient converter. But you're off by 6 to 1, and unless you just got a bad copy of the converter, I suspect it just isn't at its best at low current.
The current measurements should tell you a lot about what's going on.