If the power dissipation rating of the speaker is not exceeded, then I wouldn't expect anything bad to happen to the speaker. Of course the power being dissipated by the speaker will be equal to the DC power and AC power added together, so it will depend on the signal strength too.
What's the output impedance of the signal generator? I suspect it will be around 50Ohm, which will be enough to limit the current to a safe level.
Your audio amplifier has AC coupled inputs, which will block the DC bias from the amplifier, so I doubt it's a problem.