First, connect the op amp output back to the inverting input, as you suggested.
Then, make a voltage divider (with two equal resistors) between +5 and common to produce +2.5 V with a Thévenin-equivalent output resistance equal to half of the single resistor value. Connect that node to the non-inverting input.
If it be too low, it will pull excessive current from the power supply.
If it be too high, the input bias current through the resistance will increase the DC offset voltage at the amplifier input.
With a Goldilocks' choice of resistors, choose the input capacitor so that it passes the lowest AC frequency of interest into the voltage-divider resistance.
(The output resistance of your source should dominate the input noise, but if the capacitor is too small and the input noise current is too high, that will increase the amplifier noise. The safe choice is to make the capacitor's reactance at the lowest frequency of interest less than the source resistance, which is a higher capacitance than in the previous statement.)
Similarly, choose a suitable capacitor to couple the output (with a DC level close to +2.5 V) to your load, where the capacitor's reactance is low compared with the load resistance.
If your +5 V supply is noisy or has too much ripple, then you can connect a capacitor from the center of your voltage divider to ground to reduce the noise, and then connect a single resistor from that node to the non-inverting input.