You have a lot of variables here. You did not post any base line numbers so it is hard to judge. After you read this, you can decide if you missed anything, and if what you missed is fundamental.
There is at least two lights we have discussed, (#L1) your ambient light hitting the photo diode and (#L2) your light source aiming at and hitting direct on the photo diode. A third one is actually in play: (#L3) reflection. Some of the light from your source that isn't hitting directly on the photo diode will hit directly on something else. They get reflected around and eventually some amount of this reflected hitting your photo diode.
The diode reading is a result of #L1 + #L2 + #L3. Your measurement strategy is to measure delta in #L2, but #L1 plus #L3 may overwhelm your delta in #L2. You may need to determine if that is the case. This is referenced in Case 2 described later. But there is some homework to do first.
First lets do some base line validation described below as Case 1.
Case 1 overview
Try rotating the blades manually and see the range of measurements between max and min light block. Ideally, you don't want your hands there during measurement if you can help it. Your hand would block some light and reflect some light thus complicate things. This is tricky - since most fans have magnet and they may cause the blades to stop at "preferred positions" instead of stopping at the position you want. So you may need something to hold the blade in max blocking position and min blocking position. May be hold it with a small piece of tape, or may be a small piece of paper to jam it at the position. Note the reading at max light blocking (VmaxBlock) and at min light blocking (VminBlock). Also note the difference as Vdelta.
Case 1 (#C1) Test if your diode can measure a difference (between when the blade is stopped (jamed/taped) at max and at min blocking:
If NO (ie: can't measure a difference), go to Case 2
Else stay here on Case 1
Your setup is working and measurable when static at both positions. Failure to plot a good graph when FAN is spinning with POWER ON means (#C1a) scope setting may merely be incorrect for the speed of the FAN, or (#C1b) perhaps your circuit is indeed too slow.
Now you can narrow it down, is it #C1a, #C1b, or both combined.
C1#a, scope trigger setting issues:
(#C1a1) The scope should set to expected voltage is between VmaxBlock and VminBlock.
(#C1a2) The scope should set timing to what is expected, but expected frequency depends on the RPM. We know the lower bound is 0 RPM. For PC fans, RPM wont be millions or 100K+, it is probably < 10K RPM and likely < 5K RPM. Each blade cause a cycle, so the upper bound is number-of-blades times RPM. For 6 blades fan at 10K RPM max, that would be 60KHz max. Likely it would be 5K RPM max and that would equal 30KHz max.
So, when successful, you should be seeing a plot from VmaxBlock to VminBlock with frequency at number-of-blades times RPM expected to be.
Note that when the blade is turning, the reflected light changes. You may want to proceed Case 1b's slow down test to alter the FAN speed a bit to see the RPM range it works.
For Case 1 #C1b, your fan is faster than your circuit, but before you accept that, you need to verify that before working on speeding up the circuit. You verify by slowing the fan by either lowering the voltage running the fan, or by putting some friction on the fan blade-assembly as a friction-brake. Whatever you use to friction-brake the fan should not be adding shadow or reflecting more light. A very small rod like the wood chopstick you get from Chinese restaurant may work. You figure that out.
If by slowing it down you can now measure, then you have validated FAN speed faster than your system is the problem. You go solve that and solving that equals success.
Case 2 (#C2) if your diode cannot measure a difference (between when the blade is stopped (jammed/taped) at max and at min blocking, then it is the light level between different lights. L#2 delta is indeed overwhelmed by L#1 and L#3. This is when you will be adjust light left and right to find the right mix so L#2delta is not overwhelmed by L#1 and L#3. Success-exit is not here. Once you succeed in solving this, you go back to Case 1 to get to success-exit.
For ambient light (L#1), it is easy to control, so L#1 is easy.
For reflected light (L#3), this is harder. You need to minimize light from your light source NOT hitting directly on the photo diode so there is less to reflect around.
Here are some (but not all) of the way to minimize the reflection:
- Focusable flash light or laser as you pointed out, but you need a laser with frequency that can drive the diode. Flashlight's light wave has a mix of frequency. Laser has light with a single frequency. So it is important to check that the laser light is indeed one that your diode can work with.
- Source blocking - send the light pass a light guide like a pipe. Put your flashlight inside a PVC pipe or the core of a roll of paper towel.
- Target blocking - cover your photo diode with only a small hole opening that is aligned to the source-blocking's exit hole.
Each time you make adjustments, you test it with Case 1's manual blade position change to see if you get it to register a change. When succeed in dealing with Case 2, you got back to Case 1. The Case 1's tests has the success-exit.