Make sure RCC_CR_HSEBYP is not set -- you need the oscillator to be running, not in "external bypass" mode. Probe OSC_OUT with an oscilloscope and make sure you see something coming back out. Other than that, I don't know. 10K or 100K or 1M shouldn't make a difference -- I used that value because I was already using it elsewhere. The -3dB knee of this filter is orders of magnitude below the oscillation frequency, so it should work just fine. Using a bigger capacitor and/or resistor will push it even further down which is good.
I wouldn't leave out the resistor. I'm not sure why the appnote recommends that configuration, because the only thing determining the DC bias would be leakage currents. It might work on the bench but not in the field, or stop working from one production run to the next. It also would not produce a 50% duty cycle which can potentially cause problems.
If you can't get it to work using the STM32's oscillator then use an external inverter as shown on page 3 of the appnote. I recommend an unbuffered inverter without a schmitt trigger input; if you've only got 0.8Vpp that can be less than the hysteresis of some schmitt trigger inputs and in theory won't trigger at all. I've used 74LVC2GU04 for this in the past.
To validate output frequency, use the MCO pin. You can set it to output SYSCLK to directly measure the CPU clock frequency. Even though outputs are too slew-limited to output a clean square wave at STM32F4's max frequency, you'll still see enough of a wave to validate with an oscilloscope or frequency counter.