Yeah, the limiting point will be the refresh cycles. As long as your controller can ensure that the right number of refresh cycles are executed within the specified time interval, there should be no problem.
Of course, since the required number of refresh cycles will be the same, the slower the clock, the more refresh cycles will affect overall data throughput. At some point, there won't be enough cycles left to issue any other command than auto refresh.