Ian has covered it well.
But the basic answer is that, in general, when it comes to decoupling capacitors we just pretty much pick the value out of the air.
Why? Well it's extremely difficult, if not impossible to accurately calculate sometimes, and it's not actually going to matter too much as whole range of value will do the same job. There are so many interacting variables like lead inductance, trace resistance/inductance, plane capacitance/resistance/inductance, and the list goes on. Some people try and model it, but most just throw in a value they know will roughly work based on experience.
So when you see a 100nF decoupling cap, there is no major reason for that value, apart from it being in a rough ballpack and a nice round figure. How about an oddball value like 2.2nF ? Well, maybe it's just because they have used that value elsewhere on the board, so it was convenient and they needed something smaller than a standard 100nF. You get the idea!
Often you just need "something" there, and the actual value doesn't really matter to a point.
Yes, we also "poke around until the weirdness/problem disappears"!
Say your prototype works but doesn't pass some EMI test (radiating too much noise). You might throw a few caps around and see how it changes, usually you'll nail it on the first few goes. Trying to calculate and model the whole system could take years and an entire PhD thesis! So you just throw a cap here or there (once again based on experience of the likely culprit) until you fix it. Then you measure the performance you are interested in to make sure it's all hunky-dory.
And yes, follow the manufacturers datasheet or app note recommendations is the usual way to go. Where do they get their value from?, see above! A lot of electronics is "warm and fuzzy", not the clinical calculated ideal stuff you learn in school.
It's not all like that of course, there are times when you need to accurately calculate values like this for some apps.
Dave.