Stack overflow? You are using quite a lot of large variables; AVRs do not have too much memory.
Just some ideas to optimize the code:
Declare constant (non-changeable) arrays as const. The digipot_linear_values array will otherwise be copied to memory at start; instead it should be accessed from flash as needed. Flash is not modifiable (at least for C code), thus everiything non-const goes to ram.
The digipot_linear_values array could actually be specified as sparse array. For example, you store only every 2nd or fourth value; then perform linear approximation for values inbetween. Using power of two (2,4,8,...) as the step of the array makes calculations much faster. For longer arrays, the space savings may be quite large and the added code space taken by the calculation could be less.
Also, you should make static the functions and global variables that are not used outside that compilation object (outside that c file). This way the compiler may find better ways to optimize the code if it does not have to keep the standard interface (to be compatible in case maybe somebody calls from outside). Then you can make more smaller functions; for example move the lcd output of frequency and lcd ouput of digipot value to separate functions, that just take value as argument. it is much more readable to write update_digipot(value); update_lcd_digipot(value);. I like the code style that every function must fit fully on screen; if you need to scroll, it is too long.
freq_Hz_avg divide by three rounds always down. Maybe use 4 samples and add value of 2 before dividing by 4?
The main and TIMER0_OVF_vect are not synchronized properly? freq_Hz can be modified from ISR at the same time it is read from main (reading multi-byte data from ram is not atomic, some bytes may change during the read)? Also, you are not waiting until measurement is complete, so you may add the same value twice etc. The IRQ handler might fire for example after evaluating switch condition, so you'll use new result with old switch condition. One way would be to move the closing brace from line 170 to line 208 and the init_freq_measurement also to line 208. So you handle the result only if it is ready and then start another.
You do not have to stop the timers when taking result reading. Just read the TCNT value as the first thing in the IRQ handler and substract the last value (that you got the last IRQ handler run). Then store the TCNT value you read out to a variable for using it next time. Just disable the irqs while reading the volatile global variables to local variables to get consistent state, then re-enable.
Instead of 3-measurement average, you could also use a simple low-pass filter, like exponential moving average. For example, after every measurement, output=(1/4)*input+(3/4)*last_output. This can be optimized: sum = sum - output + input; output = sum/4; (store both sum and output vars). This could be also calculated in IRQ handler as it is fast enough. Then you have only one global variable shared between irq and main, the filtered output frequency itself. As it is multi-byte, still disable irqs while reading it out to a local var in main.
How precise the freq measurement has to be? If some kind of interrupt fires and wastes time at init_freq_measurement between lines 59/60, your timings and result would be off. If it is important, disable IRQs during those 2 lines. The same happens when something delays timer0 ovf irq. If you have spare pins, you could connect timer0 output compare output pin to timer1 input capture pin and handle the timer1 input capture IRQ. Then capturing is done on every cycle of timer0 by hardware and you just post-process the result. You get single-clock-cycle precision.
sprintf is large and slow. If it is the only use of it you may consider replacing it. Even divide-output-divide-output unrolled cycle would be smaller for uint16.
enc_count may go out of bounds (line216). C does not check array bounds. Check it yourself, do not allow the value to go outside of bounds. Also, enc_count may be uint8_t?
Hope this helps. Maybe the real problem is somewhere else, but these are the things that I happened to notice.