Why do the file formats keep changing?
File format includes generating KiCad version. Even if nothing changes each major version will have a "unique" format. This might change in the release if no actual changes to the format happen.
Edit: Although it looks like it does not include the version of the creating software, but rather a version of the file format as a date. So, current v6 format for PCB is "20211014" and for the schematics "20211123". So if version changes, there must be some changes in the data that will not be understood by older versions.
New versions import old formats without issues. Sure, this creates some inconvenience for users of older versions of the software, but intervals between major versions are not that short in this case, you have plenty of time to update.
And for v6 formats absolutely had to change, as there were massive improvements. And for the schematic format had changed entirely, making it more like PCB format.
You can make an argument for more flexible fields that can be dropped by software versions that don't understand them, but in practice it will inevitably mean that older versions will not read full data. Sure, they may open it somewhat, but it will not be full data, which rarely makes sense.