I thank you guys who gave a pointer! You were right! It wasn't the oscilloscope, but it was the LED strips.
So, there are two shelves above my work bench, and my scope is sitting on the first shelf. Under each shelves, there is an LED strip powered by a single AC adapter. There's a Y splitter to power those strips and each strip has its own switch. If I turn on one of these strips, I get the noise. However, if I directly connect the strip under the second shelf to the power adapter and power it on, I don't get the noise. If I directly connect the strip under the first shelf and power it on, I get the noise. But, if I move my scope away from the shelf, the noise is gone. In conclusion, I think the cause is an electromagnetic interference from the first shelf strip since the scope is sitting on it.
Also, while testing, I do notice that when the scope is in AC coupling mode, there's still a DC offset about 4mV in each channel when the input is disconnected, and with strips turned off. It's not a sign of damage from the EMI, is it?