That is the way mechanical drawings are meant to be.
The reason is datums and tolerances.
Indeed. All mechanical stuff has a certain tolerance and when you attempt to place all "measurements" in the drawing, then you get completely lost of where the tolerances are. So for 30+ years I add or subtract some measurements to extract the numbers I'm interested in. For the last 10 or so years I copy and paste numbers into a python console (I use python not for programming, only as a calculator).
For the rest, I use both FreeCAD and KiCad, and both can import a (pixel based) image and scale it. Then doing the calculations to get the "intermediate numbers" you need to design your (for example) footprint is easy. When half the footprint is drawn, it's often beneficial to "re scale" the image, to make a best fit with the partial drawing you have made. I still get all the "important" measurements from real calculations, but scaling the footprint and comparing the "best fit" with the drawing adds confidence that no big mistakes have been made (or it points out dubious areas).
I once had a gear with unknown parameters, I did not know whether it was metric, or based on bananas. So I put it in a flatbed scanner, imported the image in FreeCAD, drew some gear tooth in FreeCAD and this made it easy to average out the "fuzzy edges" of the scanned image, and get a good estimate of the real dimensions of the gear.
Reverse engineering from photographs is a separate topic. Especially reducing the distortions made by projecting a 3D object via a lens on a flat surface is an important step.