Thankyou for the photos, they help a lot

The motor uses a brushless DC motor (BLDC)
The motor you photographed looks like a
brushed DC motor, not a brushless. I can only see two wires going to it (red and black) and
this online listing for a WS-38ZYT64-R shows a brushed motor.
Brushed motors tend to create quite a bit of noise. You may want to try:
(1) Earthing the case of the motor. Connect a thick wire between the case and the green terminal on your power supply.
(2) Placing a small (1nF to 10nF?) film or ceramic capacitor across the two terminals on the motor, as close to the motor as possible.
The encoder signals are connected to the workstation for measurement counting.
If you have a look at the pins of your encoder circuit board you will notice that the Z, A and B signals are split into positive (+) and negative (-) versions. This likely means they are
differential signalling. Differential signalling gives you very good noise immunity if you use it correctly.
I suspect that the designers of this encoder knew that noise was a serious issue, so they provided differential signalling as a fix.
You will definitely want to:
(3) Decode the + and - signals with a differential receiver circuit. You can use off the shelf "differential receiver" ICs or make you own "differential amplifier" circuit with opamps.
Can an RC circuit be used to improve?
EDIT: Yes, you will want to implement this at the same time as the differential decoder. Slow opamps might (?) misbehave without RC filtering.
If you are not sure how to do this then first try some differential amplifier opamp circuits, then try adding RC filters to the + & - lines before they enter these circuits.