gcc v12 catastrophic internal compiler error, cc1 randomly fails

[DiTBho]

my temporary stages are ready for mips2-el-o32 and mips2-el-n32, but when Catalyst tried to move on "compiled" compilers it immediately stopped working and reported "no C compiler", which is always a sign of something catastrophic.

Inspecting what the frog was happening I manually mounted the stage (on a Qemu/MIPS32el vm) and found we still have this problem, already reported by others years ago.

Funny:
g++ always works
gcc randomly crashes with "bus error, internal compiler error"

no matter which flags you use, -o0, o1, o2, ... os, it randomly works and randomly crashes, say 50% 50% 

Also, when it works, on the same testing-source, it's
-5x slower than gcc v4.1.2!!!
- eats 3x more ram than gcc v4.1.2!!!

sign of regression?

I am going to move Catalyst to gcc 4.7.* (because I know it works well)

[SiliconWizard]

You should add a post to the ticket you mentioned, that will at least make it pop back up for developers. There's nothing we can do here.
I've never had such an issue using GCC 12 (x86_64, RISC-V and ARM), but no experience with the MIPS32 target.

If you can come up with a minimal example eliciting this behavior... problem is it "randomly crashes".
Also, you mentioned this happening on Qemu - does it happen only on Qemu or on real hardware as well?

[DiTBho]

Also on real hardware.
There is nothing to comment

Gcc hallo.c

Sometimes you get nothing but "Internal computer bug", sometimes you get a.out

[DiTBho]

Does anyone use mips2le/gcc v12 on as a say ... built by Openwrt or OE rootfs?

[brucehoult]

Also on real hardware.
There is nothing to comment

Gcc hallo.c

Sometimes you get nothing but "Internal computer bug", sometimes you get a.out

That's not very minimal. A HelloWorld tends to #include <stdio.h>, which is a huge amount of stuff to process.

Does it still fail if you do a really simple compile, such as...

Code: [Select]

int puts(const char *s);

int main(){
  puts("HelloWorld\n");
  return 0;
}

... which is functionally the same, but a much much simpler job for cc1 to process it.


Also try using C-reduce on the preprocessed source code:

https://embed.cs.utah.edu/creduce/

If cc1 is crashing randomly 50% of the time then running it 100 times in a row (or until it crashes) should be enough to find the problem. You can easily put a loop in the script you give to C-reduce.

[DiTBho]

OK, I'm going to go through that stage1 again this afternoon and try as you suggested.
I will also specifically test cc1 on pure C code without any #define

[DiTBho]

Code: [Select]

CC=gcc
CFLAGS=-O -Wall

all: obj/hallo1 obj/hallo2

obj/hallo1: obj/hallo1.o
        @echo -n "linking hallo1 ... "
        @mkdir -p obj
        @$(CC) -o obj/hallo1 obj/hallo1.o
        @echo "done"

obj/hallo2: obj/hallo2.o
        @echo -n "linking hallo2 ... "
        @mkdir -p obj
        @$(CC) -o obj/hallo2 obj/hallo2.o
        @echo "done"

obj/hallo1.o: hallo1.c hallo1.h
        @echo -n "compiling hallo1 ... "
        @mkdir -p obj
        @$(CC) $(CFLAGS) -o obj/hallo1.o -c hallo1.c
        @echo "done"

obj/hallo2.o: hallo2.c hallo2.h
        @echo -n "compiling hallo2 ... "
        @mkdir -p obj
        @$(CC) $(CFLAGS) -o obj/hallo2.o -c hallo2.c
        @echo "done"

clean:
        @echo -n "cleaning ... "
        @rm -f obj/*
        @echo "done"


Code: [Select]

int puts
(
   char *s
);

int main()
{
  puts("hAllo\n");

  return 0;
}


Code: [Select]

#include <stdio.h>

int main()
{
   printf("hAllo\n");

   return 0;
}


Code: [Select]

for cycle in {0..99}
    do
        echo "cycle$cycle"
        mem_total="`cat /proc/meminfo_total`"
        mem_avail="`cat /proc/meminfo_avail`"
        mem_ready="`cat /proc/meminfo_ready`"
        make clean
        echo "mem $mem_ready of $mem_avail of $mem_total"

        make obj/hallo2
    done


Code: [Select]

cycle0
cleaning ... done
mem     8436 of    44388 of    59152
compiling hallo1 ... done
linking hallo1 ... done
...
cycle99
cleaning ... done
mem     8188 of    44140 of    59152
compiling hallo1 ... done
linking hallo1 ... done


Code: [Select]

cycle0
cleaning ... done
mem     8228 of    44168 of    59152
compiling hallo2 ... during RTL pass: cse1
hallo2.c: In function ‘main’:
hallo2.c:8:1: internal compiler error: Bus error
    8 | }
      | ^
0x15fbeab internal_error(char const*, ...)
        ???:0
0x74c71b df_insn_rescan(rtx_insn*)
        ???:0
0x74e373 df_process_deferred_rescans()
        ???:0
0x736d87 df_finish_pass(bool)
        ???:0
Please submit a full bug report, with preprocessed source (by using -freport-bug).
Please include the complete backtrace with any bug report.
See <https://bugs.gentoo.org/> for instructions.
make: *** [Makefile:27: obj/hallo2.o] Error 1
..
cycle1
cleaning ... done
mem     8168 of    44024 of    59152
compiling hallo2 ... done
linking hallo2 ... done
..
cycle19
cleaning ... done
mem     8484 of    44340 of    59152
compiling hallo2 ... during RTL pass: reload
hallo2.c: In function ‘main’:
hallo2.c:8:1: internal compiler error: Bus error
    8 | }
      | ^
0x15fbeab internal_error(char const*, ...)
        ???:0
0x74ddaf df_update_entry_block_defs()
        ???:0
0x74e07b df_update_entry_exit_and_calls()
        ???:0
0x74e547 df_process_deferred_rescans()
        ???:0
0x736d87 df_finish_pass(bool)
        ???:0
Please submit a full bug report, with preprocessed source (by using -freport-bug).
Please include the complete backtrace with any bug report.
See <https://bugs.gentoo.org/> for instructions.
make: *** [Makefile:27: obj/hallo2.o] Error 1

do(make obj/hallo1): 100 of 100 all successful
do(make obj/hallo2): 73 of 100 successful

[DiTBho]

gcc version 12.2.1 2023-03-04, supported LTO compression algorithms: zlib

[brucehoult]

Cool. So do "gcc -E" on hallo2, then modify you test script to do exit 0 when a compile fails and exit 1 when all 100 succeed, and hit it with creduce.

[Nominal Animal]

gcc randomly crashes with "bus error, internal compiler error"

Bus error occurs when a virtual memory mapping cannot be backed (by memory, or by inode contents for a MAP_NORESERVE file-backed mapping).
In other words, that is the overcommitted system equivalent of "Out of memory error".

You can verify by temporarily adding e.g. a swap file of a couple of gigabytes, and disabling overcommit.

Also, before compiling anything, running
    sudo dash -c 'sync ; echo 3 >/proc/sys/vm/drop_caches ; sync'
will ensure maximum available RAM and swap for following operations (but also causes everything to be re-loaded from storage, so slows things down).

[brucehoult]

Running out of memory should not be a problem as the stats are showing more than 2/3 of RAM as available i.e. used for caches etc but can be dropped automatically. It's only an estimate, but still..

Randomness is troubling. It could be faulty RAM.

[DiTBho]

Randomness is troubling. It could be faulty RAM.

On a Qemu/MIPS virtual machine? Same behavior.
On the real hardware? The same kernel (6.0.2) with a 2008-rootfs, gcc-v4.1.2 is able to compile *everything* without a single error.

[DiTBho]

"Out of memory error".

The kernel shouldn't warn something on the console? I see no-warn
(console=/dev/ttyS0, I am monitoring with Minicom)

Code: [Select]

swapon /dev/sda2
Adding 131068k swap on /dev/s.  Priority:-xtents:1 acro131068k


Code: [Select]

function check_result()
{
     local test="$?"
     local ans

     if [ "$test" == "0" ]
        then
            ans="success"
            echo "$ans"
        else
            ans="failure"
            echo "$ans"
            exit 0
        fi
}


for cycle in {0..99}
    do
        echo "cycle$cycle"
        mem_total="`cat /proc/meminfo_total`"
        mem_avail="`cat /proc/meminfo_avail`"
        mem_ready="`cat /proc/meminfo_ready`"
        make clean
        echo "mem $mem_ready of $mem_avail of $mem_total"

        sync
        sync
        sync
        echo 3 > /proc/sys/vm/drop_caches
        sync
        sync
        sync

        make obj/hallo2
        check_result
    done


Code: [Select]

CC=gcc
CFLAGS=-std=c89 -O -Wall -freport-bug
MAKEOPTS=-j1

all: obj/hallo1 obj/hallo2

obj/hallo1: obj/hallo1.o
        @echo -n "linking hallo1 ... "
        @mkdir -p obj
        @$(CC) -o obj/hallo1 obj/hallo1.o
        @echo "done"

obj/hallo2: obj/hallo2.o
        @echo -n "linking hallo2 ... "
        @mkdir -p obj
        @$(CC) -o obj/hallo2 obj/hallo2.o
        @echo "done"

obj/hallo1.o: hallo1.c hallo1.h
        @echo -n "compiling hallo1 ... "
        @mkdir -p obj
        @$(CC) $(CFLAGS) -o obj/hallo1.o -c hallo1.c
        @echo "done"

obj/hallo2.o: hallo2.c hallo2.h
        @echo -n "compiling hallo2 ... "
        @mkdir -p obj
        @$(CC) $(CFLAGS) -o obj/hallo2.o -c hallo2.c
        @echo "done"

# Stop after the preprocessing stage
# do not run the compiler proper.
# The output is in the form of preprocessed source code
# which is sent to the standard output.
obj/hallo1.txt: hallo1.c hallo1.h
        @echo -n "expanding hallo1 ... "
        @mkdir -p obj
        @$(CC) $(CFLAGS) -o obj/hallo1.txt -E hallo1.c
        @echo "done"

obj/hallo2.txt: hallo2.c hallo2.h
        @echo -n "expanding hallo2 ... "
        @mkdir -p obj
        @$(CC) $(CFLAGS) -o obj/hallo2.txt -E hallo2.c
        @echo "done"

clean:
        @echo -n "cleaning ... "
        @rm -f obj/*
        @echo "done"


Code: [Select]

cycle22
cleaning ... done
mem    17784 of    45964 of    59152
compiling hallo2 ... during RTL pass: reload
hallo2.c: In function ‘main’:
hallo2.c:11:1: internal compiler error: Bus error
   11 | }
      | ^
0x15fbeab internal_error(char const*, ...)
        ???:0
0x74ddaf df_update_entry_block_defs()
        ???:0
0x74e07b df_update_entry_exit_and_calls()
        ???:0
0x74e547 df_process_deferred_rescans()
        ???:0
0x736d87 df_finish_pass(bool)
        ???:0
Please submit a full bug report, with preprocessed source.
Please include the complete backtrace with any bug report.
See <https://bugs.gentoo.org/> for instructions.
The bug is not reproducible, so it is likely a hardware or OS problem.
make: *** [Makefile:22: obj/hallo1.o] Error 1
success

 


the swap on sda2 is 128Mbyte
now even hallo1 randomly fails ...

[Nominal Animal]

"Out of memory error".

The kernel shouldn't warn something on the console?

No, it's not a kernel OOM (out of memory) situation, just a failure to immediately provide backing for a specific page in a specific process.  That's why I put "out of memory error" in quotes.

The kernel doesn't actually kill the process, it sends (the offending thread) a SIGBUS signal, with siginfo_t and context populated and si_addr member containing the address that faulted.  The default disposition is to terminate the process and dump core, but if an userspace signal handler can actually rectify the situation, all it needs to do is return, and the kernel will re-execute the offending machine instruction.  It is also possible to emulate the offending instruction, and modify the context to continue at the next instruction, but you need an async-signal safe instruction decoder to do that.

On MIPS, this happens in arch/mips/mm/fault.c:__do_page_fault(), in the do_sigbus: branch (and not in the out_of_memory: branch).  In most kernels there is a #if 0 printk() just before the force_sig_fault(SIGBUS,...) that you can uncomment to get more information about SIGBUS events; I recommend you do that if possible.

Even running gcc under gdb might provide useful info.  Personally, I am aware that newer GCC versions are quite horrible memory hogs, because typical current computers just tend to have insane amounts of RAM, compared to say fifteen years ago; so I'm not at all surprised by this.

[DiTBho]

(15 hours later ...)

Code: [Select]

# gcc-config -l 1
 [1] mipsel-unknown-linux-gnu-4.1.2
 [2] mipsel-unknown-linux-gnu-12 *

since it's a modern rootfs, to check glibc &c, I cross compiled gcc-v4.1.2 and selected it as the default c and c++ compiler. It seems working correct, and it's faster.

I wonder ... why the kernel does'n t just use the swap? 

[Nominal Animal]

I wonder ... why the kernel does'n t just use the swap? 

It is a transitory problem, so by the time gcc is killed, the kernel memory statistics no longer reflect the exact situation when the problem happened.
I do believe it does use swap, all of it.  The reason it is not deterministic is that each command invocation creates a set of processes (four or five or something like that), and their order and time-wise overlap varies.

The other possibility is that there are still some kernel-userspace-API/ABI issues left that cause GCC to occasionally keel over, but I don't think so at all.  Newer versions of GCC are horrible memory hogs.

To be sure, I would run
    strace -o log -ff gcc "$CFLAGS" -o obj/hallo2.o -c hallo2.c
and examine the trace, log, and in particular compare it to a trace obtained on x86-64; and then repeat that until it triggers the SIGBUS, to find out exactly where it happens.  This will generate a set of log.PID files.  Running
    strace -o log -e trace=%memory -ff gcc "$CFLAGS" -o obj/hallo2.o -c hallo2.c
limits the logs to memory allocations, which will tell you how much memory gcc and its subprocesses actually allocate.  (Before running either command, do remember to run rm -f log.* to remove old log files.  Since the process IDs will change, new files will be created each time.)

It would be very simple to implement a tiny library (loaded using LD_PRELOAD) interposing sigaction() and chaining SIGBUS so that a custom handler would be called before the set handler (GCC does set a SIGBUS handler).  Whenever delivered, the handler would dump the address and /proc/self/maps and/or /proc/self/smaps to standard error or a dedicated file, describing the exact state of the process memory mappings at the point of error, remove itself from the chain, then call the set handler.  The only tricky bit is that it has to be async-signal safe, too, so <unistd.h> I/O and a small (4k) static/global array for I/O.
Basically something akin to catchsegv, except for SIGBUS.  Let me know if you'd like one.

[SiliconWizard]

Not enough memory available is not surprising.
GCC 12 requires a gigantic amount of RAM while compiling and that would be one major difference compared to older versions.

It can take several GB of memory to compile a moderately long C++ source file.

For a small C program, I wouldn't expect something that bad, but certainly if you have less than say 512MB of RAM in total, I wouldn't consider using GCC 12 on it.

Your best bet instead of trying to work around it is to cross-compile, IMHO.

[DiTBho]

Your best bet instead of trying to work around it is to cross-compile, IMHO.

Catalyst doesn't cross compile, it compiles natively.

My builder can cross compile, in fact the stage1 compiled successfully and failed as soon as it was moved to Qemu/MIPS, where it should be re-compiled native, as well as all tests failed on the final real hardware, which has only 64 Mbytes of Physical RAM.

So, gcc-v12 is a bloody *big* problem 

[SiliconWizard]

I'm sure you can find a way to cross-compile successfully. I don't see why not.
And unfortunately yes forget about recent GCC versions and 64MB of RAM.
I wouldn't count on LLVM either.

But otherwise, can you not just use Qemu to compile, and allocate a shitload of RAM to the virtual machine?

[DiTBho]

I'm afraid it will also be a damn problem for my armv5tel-softfloat-linux-gnueabi GNU/LINUX PDA, which has only 64Mbytes of ram. Still untested, last night, after 96 hours, everything was successfully built by my builder, but I need to have a Gcc onboard on the PDA, and it can't be gcc-v12 because I am really afraid it has the same problem.

damn 

[DiTBho]

But otherwise, can you not just use Qemu to compile, and allocate a shitload of RAM to the virtual machine?

Yes, it can be done, but I also need an on-board C compiler.
edit:
scheduled gcc-v4.1.2, cooking new hybrid stages


p.s.

Code: [Select]

checking for armv5tel-softfloat-linux-gnueabi-gcc... armv5tel-softfloat-linux-gnueabi-gcc
checking for C compiler default output... configure: error: C compiler cannot create executables

confirmed: there is the same problem even on the armv5tel-softfloat-linux-gnueabi GNU/Linux PDA 

I'm afraid you are right, I have to forget gcc-v12

[DiTBho]

so, the hybrid modern rootfs with old gcc-v4.1.2. seems to somehow mostly work but ...

Code: [Select]

/usr/lib/gcc/mipsel-unknown-linux-gnu/4.1.2/../../../../mipsel-unknown-linux-gnu/bin/ld: dwarf_sig8_hash.c:(.text+0xadc): undefined reference to `__sync_fetch_and_sub_4'
/usr/lib/gcc/mipsel-unknown-linux-gnu/4.1.2/../../../../mipsel-unknown-linux-gnu/bin/ld: libdw_pic.a(libdw_alloc.os): in function `__libdw_alloc_tail':
libdw_alloc.c:(.text+0x40c): undefined reference to `__sync_fetch_and_add_4'
collect2: ld returned 1 exit status

forget any elfutils support in strace: no dice 

[Nominal Animal]

Code: [Select]

libdw_alloc.c:(.text+0x40c): undefined reference to `__sync_fetch_and_add_4'


These should be provided by libgcc (see libgcc/sync.c in the GCC source tree).

(GCC supports two different atomic built-ins.  One (4.1+) is Intel-style __sync builtins, the other (4.8+) is __atomic built-ins.  On architectures where these do not directly map to machine instructions, like on MIPS, function calls (using various locking techniques) are used instead.  MIPS is an LL/SC architecture, so all these functions can easily be implemented using a trivial loop around LL/SC.)

If your desired GCC version does not implement these for MIPS or any reason, they're very easy to backport; even easier to implement as a separate library (say, libgccsync) you can link against to fix such problems.  For example:

Code: [Select]

__sync_fetch_and_add_4:
    sync
loop:
    ll   v0,0(a0)
    addu at,v0,a1
    sc   at,0(a0)
    beqz at,loop
    nop
    sync
    jr  ra

However, GCC 4.1.2 on MIPS should (IIRC!) have all __sync_ builtins, up to 32 bits, definitely including __sync_fetch_and_add_4().  A trivial C program to check (mipsel-linux-gnu-gcc -O2 test.c -o test) is

Code: [Select]

int main(void) {
    volatile int a = 1;
    int b = __sync_fetch_and_add(&a, 2);
    return a * b;
}You do need 4.8 with libatomic (-latomic) for 64-bit atomics (__atomic_..._8()), though. 

[DiTBho]

Code: [Select]

# mipsel-unknown-linux-gnu-gcc-4.1.2 test_gcc_mips_sync_fetch_and_add_4.c -o test_gcc_mips_sync_fetch_and_add_4.c
/usr/lib/gcc/mipsel-unknown-linux-gnu/4.1.2/../../../../mipsel-unknown-linux-gnu/bin/ld: /tmp/ccEfV3mL.o: in function `main':
test_gcc_mips_sync_fetch_and_add_4.c:(.text+0x34): undefined reference to `__sync_fetch_and_add_4'
collect2: ld returned 1 exit status
(your c testing code)

also confirmed by Catalyst. Three confirmations, so there are no official patches, "sync_fetch_and_add_4" is not supported, needs to be added.

Since I need dev-util/strace, I relaxed its use_flags and it compiled successfully without dev-libs/elfutils

Code: [Select]

2023-03-22--08-44-20---2023-03-22--10-43-17 - [ dev-libs/elfutils ] - failure - root@dev2.40.0/4.1.2
/usr/lib/gcc/mipsel-unknown-linux-gnu/4.1.2/../../../../mipsel-unknown-linux-gnu/bin/ld: dwarf_sig8_hash.c:(.text+0xadc): undefined reference to `__sync_fetch_and_sub_4'
/usr/lib/gcc/mipsel-unknown-linux-gnu/4.1.2/../../../../mipsel-unknown-linux-gnu/bin/ld: libdw_pic.a(libdw_alloc.os): in function `__libdw_alloc_tail':
libdw_alloc.c:(.text+0x40c): undefined reference to `__sync_fetch_and_add_4'
collect2: ld returned 1 exit status
2023-03-22--19-45-37---2023-03-22--21-30-49 - [ dev-util/strace ] - success - root@dev2.40.0/4.1.2
***use_flags, dev-util/strace: (-aio -elfutils -perl -selinux -static -unwind)

better than nothing, at least now we have
- sys-devel/gcc v4.1.2: working, able to compile (works on { Qemu/MIPS, real hardware })
- dev-util/strace v6.1: minimal setup
- sys-libs/glibc v2.36-r7
- dev-lang/python v3.11.2_p1


changed the gcc system profile

Code: [Select]

-COMMON_FLAGS="-O2 -march=mips2 -mabi=32 -mplt -pipe"
+COMMON_FLAGS="-O2 -march=mips2 -mabi=32 -pipe"
-mplt is NOT supported by gcc v4.1.2

Now I can go on, and try to investigate about gcc v12.


We need to support gcc v4.1.2, I have the feeling there are other packages like dev-libs/elfutils 

[DiTBho]

Code: [Select]

/usr/lib/gcc/mipsel-unknown-linux-gnu/12/libgcc_s.so.1
/usr/lib/gcc/mipsel-unknown-linux-gnu/12/libgcc.a
sync_fetch_and_add is there