A little bit more detail on what I wrote earlier.
1) I used the previous pin state + current pin state to detect rotation. The convention I followed is ENC_A..ENC_B. For example, a clock-wise rotation off of 00->10 (ENC_A channel goes high while ENC_B stays low) should yield +1, indicating clock-wise rotation. Like wise, a counter-clock-wise rotation off of 11->10 (ENC_B goes low while ENC_A stays high) should yield -1, indicating CCW rotation.
2) the four bits of encoder pin states, as indicated above, can be used to index the return value, through a look-up table.
So my routine would look like this:
//encoder states.
//coding convention: ENC_A previous..ENC_B previous..ENC_A..ENC_B
const int8_t enc_states={0, -1, 1, 0, 1, 0, 0, -1, -1, 0, 0, 1, 0, 1, -1, 0};
int8_t enc_read(void) {
static uint8_t enc_ab=0; //encoder stats, lowest 4 bits effective
enc_ab = (enc_ab << 2) | (IO_GET(ENC_PORT, ENC_A)?0x02:0) | (IO_GET(ENC_PORT, ENC_A)?0x01:0);
return enc_states[enc_ab & 0x0f]; //index through the lowest 4 bits
}
You can actually write it into a macro if you care about minimizing the overhead.
Once your isr is set-up, you can simply call enc_read() to return / update the encoder reads.
I have used this approach on motors speed sensors. It can easily respond up to 20Khz+ on a 1MIPS mcu so speed is never a problem.
The only thing I would caution you is hardware wiring. If you intend to use a capacitor to debounce, make sure that there is a serial resistor there. Or you run the risk of resetting the mcu. Here is a case where a low quality switch (with high on resistance) is actually helpful.