How important is it that you wear the wrist strap when working on the mat? Will the mat just dissipate the static if you touch it periodically, or do you need to be in constant contact with either the mat or strap?
Any static-safe workstation in industry has a minimum requirement that a table mat and a wrist strap are both used and both grounded through a safety resistor.
You can build up a static charge in seconds just by wearing polyester clothing, so if you happen to touch a sensitive component after building up such a charge you could damage it.
Also, bear in mind that the surface of an ESD mat will typically have a resistance to ground in the hundreds of megohms, so discharge will be much slower than through a wrist strap.
You can work out how long it will take using the equations for capacitor discharge, since the human body has a capacitance of around 10 to 200 pF, depending on environmental conditions. For example, a static charge of 3 kV will take around a third of a second to discharge to below a volt, assuming a body capacitance of 200 pF and a mat with a 200 megohm resistance to ground.