Simon, thank you for bringing this to my attention. I have been using the tinyAVR 1-series for a while, but never with external ADC reference. I looked through Microchip's appnotes for ATtiny, but couldn't find any information beyond what is found in the datasheet (e.g. ATtiny1616 -> ADC operation -> "external reference").
"When using the external reference voltage V_REFA , configure ADCnREFSEL[0:2] in the corresponding VREF.CTRLn
register to the value that is closest, but above the applied reference voltage. For external references higher than
4.3V, use ADCnREFSEL[0:2] = 0x3."
Looking at this I can only suspect some structural diode.
The implication of this requirement is that other peripherals (e.g. DAC) are limited in their range of reference, when the ADC uses the external reference.
I don't have an isolated tinyAVR 1-series ready for measurement. If anybody had one, I would be curious as how currents (supply, reference input) and analog accuracy change, when this requirement is violated.