The Interrupt Latency is the time in between some event that should cause an interrupt occurs, and the time that the code used to service the interrupt actually starts running.
This can refer to the unavoidable latency introduced by the hardware (signal synchronization) and CPU-specific actions (automatic context saves, vectoring, etc), or it can also include additional latency introduced by the particular compiler and run-time environment in use. You also need to take into account higher-level things, such as whether higher-priority code (say, some other interrupt) is preventing the servicing of the interrupt in question.
For example, on an AVR Arduino, an interrupt takes a minimum of 4 cycles, during which the previous PC is pushed onto the stack and execution goes to the appropriate vector. The vector is normally a JMP to the ISR, so that's another 3 cycles. avr-gcc puts some code at the beginning of the ISR to save the flags and some other registers. If the ISR calls another function, it has to save more registers. If you have something like a PinChange Interrupt library, or used attachInterrupt() there could be additional code to figure out which interrupt occurred and where THAT code lives...
All of which might have been delayed by quite a few cycles if the AVR was already servicing the timer interrupt that maintains millis()
It's rather different on an ARM.
"Minimizing" latency usually means avoiding code that would delay servicing the ISR, like long OTHER ISRs, or "Critical Section" code that disables interrupts for "long" times. "Long" here means 50+ CPU cycles, I guess. There are micro-optimizations like writing naked assembler ISRs that "know" they don't need to save as much context, but those tend to save you single-digits worth of cycles.