It turned out the service menu didn't work because I used a notebook with too modern graphics card (Geforce 1070) as a signal source. It had only HDMI output, so I used an active HDMI to VGA converter. When I connected the monitor to a VGA output on an old PC with Radeon HD79xx, the service menu worked at the first try. Curiously, it also worked when I used a HDMI output and the converter on the old PC. I guess it's not so surprising, there is not much incentive to support the obsolete VGA interface in new GPU drivers.