I don't understand what you mean by "flipped in the time domain". So, I'm not sure if this is your problem, but my guess is: you have a
real signal \$x_0, x_1,\ldots, x_{n-1}\$. You compute its Fourier transform \$y_0, y_1,\ldots,y_{n-1}\$. There is a well-known fact that, for real signals, the spectrum verifies the "conjugation condition" \$y_k = \overline{y_{n-k}}\$.
There is a subtlety. The "conjugation condition" does not reverse the bins of the spectrum, since \$y_0\$ stays unreversed. The conjugation means: \$y_0, y_1, y_2,\ldots, y_{n-2}, y_{n-1}\quad = \quad \overline{y_0}, \overline{y_{n-1}}, \overline{y_{n-2}},\ldots,\overline{y_{2}},\overline{y_1}\$. This is due to the fact that from \$y_k = \overline{y_{n-k}}\$ follows: \$y_0 = \overline{y_n} = \overline{y_0}\$ (periodicity), \$y_1 = \overline{y_{n-1}}\$, \$y_2 = \overline{y_{n-2}}\$, etc.
So, if you completely reverse the spectrum: \$y_{n-1}, y_{n-2},\ldots,y_{0}\$, keeping in mind the "conugation condition", what you are really doing is the same as\$\overline{y_1}, \overline{y_2},\ldots,\overline{y_{n-1}},\overline{y_0}\$, that is,
when you reverse the spectrum, you are shifting the spectrum one position to the left and conjugating it.
There is another interesting fact about the DFT: the time delay relation. If you shift the spectrum one position to the left, you are multiplying the signal by \$e^{\frac{2\pi i}{n}k}\$, and viceversa (changing left to right and the sign in the exponential). This means that, if you conjugate your inverted spectrum, all that remains is a shifted spectrum, so if you compute its inverse Fourier transform, you get the original signal multiplied by \$e^{\frac{2\pi i}{n}k}\$. This is what you mean when you say that inverting the spectrum and then conjugating it, you recover the signal without flipping?
If you do not conjugate the spectrum, there is no immediate inverse Fourier transform I can think of now (it's past 4AM here, maybe with a more clear head

). The trouble is that, the inverse Fourier transform of a conjugate turns into a direct Fourier transform, \$\mathcal{F}^{-1}(\overline{y}) \ = \ \frac{1}{N}\overline{\mathcal{F}(y)}\$, so you wind up doing the
direct Fourier transform of the shifted spectrum, not its inverse.
Hope this helps.
-----------
Edit:Hmmm. Maybe this works.
The inverse DFT verifies: \$\mathcal{F}^{-1}(y_0, y_1,y_2,\ldots, y_n) \ = \ \frac{1}{N}\mathcal{F}(y_0,y_{n-1},y_{n-2},\ldots,y_{2},y_1)\$.
So, let's compute the inverse DFT of your inverted spectrum using all the previous results. Let's begin with the inverse DFT of the reversed spectrum:
\$\mathcal{F}^{-1}(y_{n-1}, y_{n-2},\ldots, y_2, y_1, y_0) \ = \ \mathcal{F}^{-1}(\overline{y_1}, \overline{y_2}, \ldots , \overline{y_{n-2}}, \overline{y_{n-1}}, \overline{y_0})\$
This is due to the "conjugation condition", as we saw. Now, using that the inverse transform of the conjugate is the conjugate of the direct transform, divided by N:
\$ \mathcal{F}^{-1}(\overline{y_1}, \overline{y_2}, \ldots , \overline{y_{n-2}}, \overline{y_{n-1}}, \overline{y_0}) \ = \
\frac{1}{N}\overline{\mathcal{F}(y_1, y_2, \ldots, y_{n-2}, y_{n-1}, y_0)}\$
And now, using the above relation to turn the direct into the inverse:
\$\frac{1}{N}\overline{\mathcal{F}(y_1, y_2, \ldots, y_{n-2}, y_{n-1}, y_0)} \ = \ \overline{\mathcal{F}^{-1}(y_1, y_0, y_{n-1}, y_{n-2}, \ldots, y_3, y_2)}\$
Hm, no. We are back to the beginning, with a conjugate and two shifts to the right. I was hoping this would give a shifted spectrum, but it gives an inverted shifted spectrum. Maybe this is the flipping you were mentioning.