What FW version do you use?
I tried mine SDS1204X-E with latest V6.1.37R10, it works fine.
>>>FORM:DATA CUSTom,3
>>>C1:PAVA? FREQ
C1:PAVA FREQ,1.00E+03Hz
>>>FORM:DATA CUSTom,7
>>>C1:PAVA? FREQ
C1:PAVA FREQ,1.000000E+03Hz
Wait a second...
Okay, I got it. It's a bug

>>>FORM:DATA CUST,3
>>>C1:PAVA? FREQ
C1:PAVA FREQ,1.000000E+03Hz
>>>FORM:DATA CUST,5
>>>C1:PAVA? FREQ
C1:PAVA FREQ,1.000000E+03Hz
The short form of CUSTom doesn't work

>>>FORM:DATA CUSTom,6
>>>:FORMat:DATA?
FORM:DATA CUSTOM,6
>>>FORM:DATA CUST,3
>>>:FORMat:DATA?
FORM:DATA CUSTOM,6