It's very hard to know what's going on without seeing a schematic - can you provide one?
Along with the miller capacitance pulling the gate up, as discussed above, the other option is possible reversal of the drain-source pins, meaning the body diode will conduct - this would also cause the symptoms you're seeing.