The CUI AMT332 encoder came in, so I replaced the hall sensor board on the motor with the encoder. Followed the procedure for aligning the encoder (sets the relation between U,V,W and the index pulse) and set it up for a 4-pole motor with 1024 ppr; default is 2048, I don't need such a high resolution and the A/B/I signals will have a lower frequency at speed.
I re-did the hall sensor calibration within the drive configuration software and set it up for incremental encoder feedback from the motor shaft. Once encoder feedback is enabled, the drive offers sinusoidal excitation (instead of trapezoidal), which is the equivalent of microstepping a stepper motor. The velocity feedback is far better than using the back-EMF method, the velocity loop tuning can be made much tighter. Motor motion is smooth down to 10 rpm, though the drive quickly bumps against its 5A continuous current limit when applying a load by grabbing the output shaft. The graphs of actual motor position are much smoother.
The motor also runs a lot more smoothly, with less audible noise. I enabled closed-loop position and attempted some tuning of the position loop. The results were surprisingly good, though the only load is the motor's own inertia, and the 40/60 mm stepped output pulley. The green curve below is the following error, the pink and blue are commanded and actual positions, respectively.
In summary, to get good control over BLDC motors at lower speeds, one needs encoder feedback and sinusoidal excitation. Large currents will be needed at lower speeds when using a motor with a low pole count.