I'm jumping in here..
I took a quick look. That code looks reasonable so far. I don't know about the call to millis() in your ISR.
last_time = millis();
Does this work inside the ISR ? Just a warning to make make sure that it won't be a problem area for you.
Second, you would be better to get the table lookup first and do something only if it's non-zero. That way you don't call millis() when you don't need to, since it might be a slow operation:
(here's your own code with some of my suggested changes)
void encoder()
{
int8_t direction;
long this_time, diff;
enc_val = enc_val << 2;
enc_val = enc_val | ((PIND & 0b1100) >> 2);
direction = lookup_table[enc_val & 0b1111];
if (direction) {
this_time = millis(); // millis() only lasts 50 days and may overflow.
diff = this_time - last_time; // you will need to deal with overflow here
// TODO: set multiplier equal to 1, 2, or 5, for example, depending on time elapsed.
enc_count = enc_count + direction * multiplier; // this may go negative , is that OK?
last_time = this_time;
}
}
Thirdly, you really should low pass filter your A and B encoder inputs with at least an RC time constant of of around 20-30 ms. Bounces can cause 100's of singular microsecond changes that can last a total 10-50 ms long. Yes, a fast processor can probably handle the 100k per second (or more!) interrupt rate, but why do that? You really don't want that when most of them will be thrown away, and it will become troublesome when you want to do other things but are hindered by a high interrupt rate caused by switch bounce. This will manifest itself as issues in other areas of your code when you try to rotate the encoder.
A simple RC debouncer on the switches is all you need.
See below. The horizontal line on the simulation marks the approx. 2.0V CMOS threshold. Some circuits use a schmitt trigger buffer before going into the microcontroller input pin, but if you can set the micro to use level-triggered interrupts, not edge triggered, then you can just go straight in with this circuit. Many micro's have schmitt trigger inputs anyways.
(note: the diodes are not absolutely necessary -- they help the rise time, and keep the rise time and fall time about the same ... in this example it is about 20-30ms)