Good point.
Buy 1 or 2 variety packs at the outset, just so you have a range of sizes available. Once you start using it, you will find you tend to use just a few of those sizes. When you need more, just get the sizes you've been using, otherwise you'll be building up a stock of sizes you'll rarely, if ever, use.
Edit: Aside from size, there are also the colours that you find you will want to use more frequently, so variety pack purchasing will force you into using colours you may prefer not to use in a given situation or to accumulate them.