Start by redrawing your schematic in a proper way.
As a guide, put high voltages on top of the schematic, and low voltages on the bottom.
Then try to use signal flow from left to right (which you already did).
Also group components by function.
For example put VCC on top, then connect the pullup resistors to it.
If you do this, then you will discover you've drawn your schematic wrong.
The switch (pins 4 & 5) does not have a pullup to VCC, while I guess your intention was to do so.
Even if you did mean to connect it in this way, I still consider it an error, as your schematic does not convey it's intention, and leaves me guessing.
You may be thinking I'm nitpicking here, and you're right about that. This is a quite simple schematic. When schematics become more complicated though, this becomes an important factor in readability. In the long run you'd be better off developing a habit of drawing neat schematics.
Your scope traces also show RC constants, while the only capacitor (10nF) appears to be some kind of decoupling capacitor connected between GND and VCC.
I see no capacitors for the RC time constants.
Deriving meaningful data from your scope picture is difficult, because a lot of info is missing.
I do see that the duty cycle of the blue trace in the first screenshot is more assymetrical then in the second screenshot, but this can also be because of the way the knob is turned.
Trouble shooting such things is easier when some variables are eliminated. For example by rotating the knob with a motor at a (sort of) constant speed, or by a big heavy knob that can be used as a flywheel.
----------------
In the past I once connected such a rotary encoder wrongly. I switched one of the "signal wires" with the "GND" connection. The encoder worked fine in one direction, but in the other direction both signals switched at the same time on some of the flanks.
And where do you probe your signals?
I assume you do it on the right side of the resistors. If you do it on the left side, directly on the encoder, you may see debounce noise on your scope, but also see more of what is happening.