For how long can be held back a Linux kernel, yet upgrade to a new distro?

[RoGeorge]

The kernel must stay at v5.x, anything v6 does not match some specific driver I need.  For now, the distro is Debian 11 Bullseye, and that is a little older.  For example, the latest Python from Bullseye repo is 3.9, and some scripts require 3.10, etc. 

To avoid manually installing from sources each time, would be nice to just migrate to a newer distro that has new enough packages, for example by adding the new Debian 12 Bookworm into /etc/apt/sources.list.  For now, the must have kernel v5 and its corresponding linux-libc-dev are manually marked to hold back, from apt.  This is for an offline machine, so no security concerns by freezing the kernel version.

The question is, how good is the separation between the rest of a Linux distro and the kernel version?  For how long an old kernel can remain unchanged, without runtime conflicts with the rest of a distro?

[RoGeorge]

Oh well, did the mistake to try to build Python 12.3 from sources (on a Raspberri Pi 1B from 10 years ago, ARMv6 32-bit, single core 700MHz, 256MB RAM, Debian11 and SD card).

Already lost the count of how many hours have past since the build was started.  Proc is always 100%, and once in a while the screen scrolls up some text. 

[ejeffrey]

The kernel must stay at v5.x, anything v6 does not match some specific driver I need.  For now, the distro is Debian 11 Bullseye, and that is a little older.  For example, the latest Python from Bullseye repo is 3.9, and some scripts require 3.10, etc. 

In theory, start with a distribution version that ships with a 5.x kernel and pin the version of the kernel.  Ideally, anything that requires 6.X kernels will be prevented from updating and you should be fine.

For python, if you are requiring specific python versions, I consider moving away from using the distribution release to installing your own and using virtual environments.

The question is, how good is the separation between the rest of a Linux distro and the kernel version?  For how long an old kernel can remain unchanged, without runtime conflicts with the rest of a distro?

It's actually pretty well separated.  The linux kernel implements very strong backwards compatibility for userspace, and the utilities that directly manipulate the kernel data (like iptables) generally have pretty good kernel version support.  If you don't actually need features of the newer kernel it should work fine, but it just gets hard to manage dependencies.  What you can do if you want is to run a stripped down OS of an older version, then run a newer OS in a container (docker, podman, etc) and use that for everything.  Then if you run into a specific tool where the newer container version is incompatible with the old kernel you can run it in the host OS.

[magic]

I can boot a recent Arch installation with Linux v5.10-LTS.

That's with sys****d, which is IMO the part which would be most likely to depend on some new kernel API.

[RoGeorge]

That's good news, because compiling is unusable slow.  The Python install from sources that I've started this morning, took 10+ hours to compile but it works.  Except, now I have no Raspberry Pi specific Python modules, e.g. 'gpiozero', and would need to compile those, too. 

I'll just migrate the old Debian 11 Bullseye to the current Debian 12 Bookworm, then sudo apt full-upgrade with the kernel pinned down to the older v5.x.  Even if the migration breaks something, would still be faster to reinstall all the OS than to compile yet another package like Python.

[SiliconWizard]

Note that on the Pi you can use ArchlinuxARM, which is based on Arch and may be easier to set up the way you want.

[magic]

Unless the special snowflake driver is strictly required for boot, I think I would first install normally, then make sure everything works as expected, then downgrade the kernel and check if it still works.

Also, do you have sources to this driver? Any chance that the "incompatibility" is as simple as an explicit version check?

[RoGeorge]

The "driver" that can only work with v5 for now, is me. 

I am following a workshop of 30+ lessons, with examples of how to write LKM (Loadable Kernel Module) drivers for Raspberry Pi.  Doing that to learn more about the Linux kernel in general, and hardware drivers in particular.  The classes are for v5 kernel, and they do not apply any more for v6 kernel.

The GPIO access has changed in the last year 2-3 times.  With the v6 kernel, there is now a different interface, with different function names, different functionality of the callback functions, and different mechanisms of requesting and handling hardware pins (GPIO, I2C, SPI, etc.).  If a driver was written using gpio pins the old style, there are zero chances for the same code to work in kernel v6, without rewriting the driver.

Due to the history of GPIO interfaces in the kernel, there are two different ways to obtain and use GPIOs:
        - The descriptor-based interface is the preferred way to manipulate GPIOs, and is described by all the files in this directory excepted Legacy GPIO Interfaces.
        - The legacy integer-based interface which is considered deprecated (but still usable for compatibility reasons) is documented in Legacy GPIO Interfaces.
The remainder of this document applies to the new descriptor-based interface. Legacy GPIO Interfaces contains the same information applied to the legacy integer-based interface.

Quote from:  https://www.kernel.org/doc/html/latest/driver-api/gpio/intro.html

They say the old style is still usable, but I've tried for my RPi and drivers that worked with kernel v5 do not work in v6.  Some examples do not even compile, other examples compile but give runtime errors.

To make things even more confusing, while the old style of accessing gpio pins was already removed from v6, there is still in place a mocking interface with the old function names, but they do not work, the mocking old style function names are there only to emit compilation warnings. 

TL;DR would like to use kernel v5 during the workshop, to avoid any doubt when eventual errors appear.  Once I get a better understanding of how the kernel interacts with the rest of the hardware and software, I will rewrite the workshop examples for the new gpio style, as an exercise.  Or, at least that's the plan. 

[SiliconWizard]

I haven't followed really what exactly you need to do with the GPIOs and other constraints, but to me, adapting to the new GPIO control interface would be time much better invested. It's also much faster.

[magic]

Yeah, that's what it's like. Internal APIs are a moving target, documentation is not always great and external resources become stale after a few years.

They say the old style is still usable, but I've tried for my RPi and drivers that worked with kernel v5 do not work in v6.  Some examples do not even compile, other examples compile but give runtime errors.

To make things even more confusing, while the old style of accessing gpio pins was already removed from v6, there is still in place a mocking interface with the old function names, but they do not work, the mocking old style function names are there only to emit compilation warnings. 

Chances are the commit series which removed old functions forgot to update this particular part of the docs. Misinformation will stay there until somebody complains on the mailing lists.

[RoGeorge]

I haven't followed really what exactly you need to do with the GPIOs and other constraints, but to me, adapting to the new GPIO control interface would be time much better invested. It's also much faster.

Don't need to do anything in particular, it's a hobby thing.  Only learning for the first time, just out of curiosity, about the Linux kernel and how it gives access to gpio.  The way I am learning this, is by following a workshop.  The workshop code examples are for kernel v5, and the code from that workshop does not work in kernel v6 (I've tried).  Debian 11 was using kernel v5, Debian 12 is using v6.

I would like to have (as a nice to have) new programs from the RPi Bookworm repository (deb12, default kernel v6), yet at the same time to have kernel v5, mandatory for the driver workshop lessons.

Agree that I should just use the new gpio access style (gpiod), just that I do not know enough yet to adapt those lessons to the new gpio interface, or write my own drivers (don't know enough about the Linux kernel in general).  That workshop also teaches me other aspects about the kernel (as a side effect of teaching gpio drivers).  So I would just use kernel v5 to finish the workshop first.

[DiTBho]

I haven't followed really what exactly you need to do with the GPIOs and other constraints, but to me, adapting to the new GPIO control interface would be time much better invested. It's also much faster.

faster for sure, but... I don't know, it seems like a mess that breaks the compatibility with the scripts in bash
then they're all open source enthusiasts, but no one ever wants to fix anything
and I always find myself alone to fix stuff

so, to be honest, I don't like it for this practical reason, and even because if I have to use GPIO for "fast things"
(e.g. bigbanging? step-motor drivers? seriously?!?) well, I do first either to use an external MPU, or rethink what I want to do.

[DiTBho]

I can boot a recent Arch installation with Linux v5.10-LTS.

here: kernel v.3 && 2024 ARM64 rootfs
the reason is that many things don't work on kernels 5 and 6, and we are still "fixing" things.
however, if there are no particular needs, the kernels v{>3.3.*, 4.*, 5.*} are still fine (within limits)

pythong dev is one of those things that may be kernel-header-specific
and that damn thing breaks very easily, and typically wastes months of fixing

[RoGeorge]

Meanwhile the sudo apt full-upgrade has ended (while holding the kernel to 5.15.48+, with its corresponding linux-libc-dev and bootloader, too, held back with apt hold).  The OS is now Raspberry Pi Bookworm (Debian 12), with the former kernel v5.

It all seems to be working just fine, drivers examples compile without errors, and also they run as expected.

There is a new warning I see now, complaining the kernel was compiled with GCC10, while the current GCC in Bookworm is GCC12.

Code: [Select]

pi@rpi:~/workshop/code/10_spi_bmp280 $ make     
make -C /lib/modules/5.15.84+/build M=/home/pi/workshop/code/10_spi_bmp280 modules
make[1]: Entering directory '/usr/src/linux-headers-5.15.84+'
warning: the compiler differs from the one used to build the kernel
  The kernel was built by: gcc (Raspbian 10.2.1-6+rpi1) 10.2.1 20210110
  You are using:           gcc (Raspbian 12.2.0-14+rpi1) 12.2.0
  CC [M]  /home/pi/workshop/code/10_spi_bmp280/spi_bmp280.o
  MODPOST /home/pi/workshop/code/10_spi_bmp280/Module.symvers
  CC [M]  /home/pi/workshop/code/10_spi_bmp280/spi_bmp280.mod.o
  LD [M]  /home/pi/workshop/code/10_spi_bmp280/spi_bmp280.ko
make[1]: Leaving directory '/usr/src/linux-headers-5.15.84+'
pi@rpi:~/workshop/code/10_spi_bmp280 $
My guess is this warning happens because I've hold back the old linux-libc-dev.  I did that because I've read this package deals with very intimate kernel POSIX calls, and also because the kernel update tools upgrades all 3 components at once:  the kernel, the bootloader, and the linux-libc-dev.  (the upgrade tool is some Raspberry Pi specific tool meant to upgrade the kernel manually, called rpi-upgrade, it's a developers tool, not for end users)

What could be the side effects when using different GCC versions?  It should be something, or else the compiler wouldn't warn about that.  Is this GCC10 for kernel, but GCC12 when compiling drivers, something to worry about? 

[magic]

My guess is you get this warning because you are compiling a module with different GCC version than your kernel was built with
Not sure what libc has to do with anything. Downgrade the compiler if you don't want to see it.

edit
And it's the kernel Makefile checking this and printing the warning:

Code: [Select]

$ grep 'the compiler differs from the one used to build the kernel' -r *
Makefile:               echo >&2 "warning: the compiler differs from the one used to build the kernel"; \
Next obvious step:

Code: [Select]

$ git blame Makefile |grep 'the compiler differs from the one used to build the kernel'
6072b2c49d23e (Masahiro Yamada          2021-08-01 11:53:46 +0900 1793)                 echo >&2 "warning: the compiler differs from the one used to build the kernel"; \
$ git show 6072b2c49d23e
commit 6072b2c49d23eb69b6dca06ad095d3a9633b2b80
Author: Masahiro Yamada <masahiroy@kernel.org>
Date:   Sun Aug 1 11:53:46 2021 +0900

    kbuild: warn if a different compiler is used for external module builds
   
    It is always safe to use the same compiler for the kernel and external
    modules, but in reality, some distributions such as Fedora release a
    different version of GCC from the one used for building the kernel.
   
    There was a long discussion about mixing different compilers [1].
   
    I do not repeat it here, but at least, showing a heads up in that
    case is better than nothing.
   
    Linus suggested [2]:
      And a warning might be more palatable even if different compiler
      version work fine together. Just a heads up on "it looks like you
      might be mixing compiler versions" is a valid note, and isn't
      necessarily wrong. Even when they work well together, maybe you want
      to have people at least _aware_ of it.
   
    This commit shows a warning unless the compiler is exactly the same.
   
      warning: the compiler differs from the one used to build the kernel
        The kernel was built by: gcc (GCC) 11.1.1 20210531 (Red Hat 11.1.1-3)
        You are using:           gcc (GCC) 11.2.1 20210728 (Red Hat 11.2.1-1)
   
    Check the difference, and if it is OK with you, please proceed at your
    risk.
   
    To avoid the locale issue as in commit bcbcf50f5218 ("kbuild: fix
    ld-version.sh to not be affected by locale"), pass LC_ALL=C to
    "$(CC) --version".
   
    [1] https://lore.kernel.org/linux-hardening/efe6b039a544da8215d5e54aa7c4b6d1986fc2b0.1611607264.git.jpoimboe@redhat.com/
    [2] https://lore.kernel.org/lkml/CAHk-=wgjwhDy-y4mQh34L+2aF=n6BjzHdqAW2=8wri5x7O04pA@mail.gmail.com/