Are you programming flash or RAM? For flash that sounds about right. For RAM it should be about 10 secs I think.
During development you should be using RAM download as it's much much faster, and only use flash when you need it to come on at powerup.
Also, I don't know if it's still the case, but last time I checked, the programmer software in Diamond was a lot slower than Lattice's standalone ISPVM software.