Migrating from Buildroot 2015.08.03 with GCC 4.9.3/Make 3.81/gblibc 2.20 to a Buildroot 2020.02 setup to support GCC > 8.x/Make 4.1/glibc 2.30.
Encountered an issue with GCC 8.3 compiler when building a Linux 3.14.17 ARM Cortex-A9 kernel. Using GCC 7.5, the file in question builds (other issues debugging). Also builds fine with the GCC 4.9.3 compiler.Details below - more can be provided.Seems to be an issue with intermediate GCC generation, causing ASM to generate incorrect code?? Has anyone encountered this?
Tried different Kernel build options, no changes.
Thanks for your expertise and wisdom.Stephen Beckwith
Host System: VMWare Fusion setup on MAC OSX Mojave (10.14.6) on a 2014 vintage 15” Macbook Pro with a Core i7, 16GB RAM, 512GB SSD.
VM gets 4 cores (HT enabled) and 8GB RAMSources located on an external USB-3 HDD, where the VM is booted from and it’s Filesystems is located.VM is Ubuntu 18.04.01 - updated in Mid-March
sbeckwit@ubuntu:~$ uname -aLinux ubuntu 5.3.0-42-generic #34~18.04.1-Ubuntu SMP Fri Feb 28 13:42:26 UTC 2020 x86_64 x86_64 x86_64 GNU/Linuxsbeckwit@ubuntu:~$ gcc --versiongcc (Ubuntu 7.4.0-1ubuntu1~18.04.1) 7.4.0Copyright (C) 2017 Free Software Foundation, Inc.This is free software; see the source for copying conditions. There is NOwarranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.sbeckwit@ubuntu:/lib32$ ldd --versionldd (Ubuntu GLIBC 2.27-3ubuntu1) 2.27Copyright (C) 2018 Free Software Foundation, Inc.This is free software; see the source for copying conditions. There is NOwarranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.Written by Roland McGrath and Ulrich Drepper.The setup:Our previous setup is a Buildroot 2015.08.03 setup.Using the original Buildroot setup - we use a Buildroot Built Compiler, version 4.9.3, glibc 2.20, Make 3.81. - This system builds our Embedded Linux setup, including kernel/uboot/busybox/utilites/applications.
- In this setup, everything builds just fine. - This system runs in a server environment on older RHEL 5.x based system.
The Linux Kernel we are building is a 3.14.17 kernel, with some 3rd party patches for specific SOC (EMULEX PIlot4 BMC, now owned by ASPEED - this is a dual-core ARM Cortex-A9 ARM SoC) - Build options are “stock” with the exception of the adding the -save-temps flag to get the .s output files.
New Setup:We are migrating to a newer Buildroot setup in order to mitigate recent CVE issues related to GLIBC and GCC. Therefore, we have put into place new buildroot setup:Buildroot 2020.02: - This supports building the following: GCC 7.5, GCC 8.3, GCC 9.2 Glibc 2.30 (the one we’ve chosen for now) Uses Make 4.1
Initially we built a GCC 9.2 Compiler and ran into the following error:
/tmp/ccxaGNai.s: Assembler messages:/tmp/ccxaGNai.s:2262: Error: .err encounteredarm-buildroot-linux-gnueabi-gcc.br_real: warning: /home/sbeckwit/sp_dev/bowie_dev/src/include: linker input file unused because linking not done CC security/keys/key.oscripts/Makefile.build:309: recipe for target 'kernel/fork.o' failed- similar messages for kernel/exit.o and fs/fat/dir.o
Built a GCC 8.3 Compiler and ran into the exact same errors.
Built a GCC 7.5 Compiler and this error no longer exists (though we have other issues)
It was determined that in order to comply with Corporate Security policy, we are required to have GCC 8.x or above.
Therefore: - I re-setup the GCC 7.5 and obtained the .i/.s output files (using the -save-temps switch in the Kernel Makefile) - I then switched to the GCC 8.3 and obtained the same files.
GCC 8.3 Information:
sbeckwit@ubuntu:~/sp_dev/sp_archives/toolchains/arm8.x-toolchain/bin$ ./arm-arm-linux-gnueabi-gcc-8.3.0 --versionarm-arm-linux-gnueabi-gcc-8.3.0.br_real (Buildroot 2020.02) 8.3.0Copyright (C) 2019 Free Software Foundation, Inc.This is free software; see the source for copying conditions. There is NOwarranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.Error:
fork.s: Assembler messages:fork.s:2277: Error: .err encounteredscripts/Makefile.build:308: recipe for target 'kernel/fork.o' failedThis is in the function: mm_release()
.LBB1932: .loc 8 110 35 is_stmt 0 view .LVU654 mov r2, sp bic r2, r2, #8128 bic r2, r2, #63.LBE1932:.LBE1933: .loc 1 784 278 view .LVU655 mov r0, r5.LVL102: .loc 1 784 303 is_stmt 1 view .LVU656 .loc 1 784 39 is_stmt 0 view .LVU657 ldr r1, [r2, #8].LVL103: .loc 1 784 326 view .LVU658 sub r1, r1, #1.LVL104: .loc 1 784 351 is_stmt 1 view .LVU659 .loc 1 784 379 view .LVU660 .loc 1 784 947 view .LVU661 .syntax divided@ 784 "kernel/fork.c" 1 .ifnc r0,r0 ; .err ; .endif **.ifnc r3,r2 ; .err ; .endif** .ifnc r1,r1 ; .err ; .endif bl __put_user_4@ 0 "" 2.LVL105:highlighted line is line 2277 from the error message
When I grep’d through the .s files generated for the 8.3 build, looking for the “.ifnc” in the current files. Yet, this file is the only file where this .ifnc has “different” registers: all other instances show the same register. Why is this one different.
From the fork.i file for this function:
# 747 "kernel/fork.c"void mm_release(struct task_struct *tsk, struct mm_struct *mm){ if (__builtin_expect(!!(tsk->robust_list), 0)) { exit_robust_list(tsk); tsk->robust_list = ((void *)0); } if (__builtin_expect(!!(!list_empty(&tsk->pi_state_list)), 0)) exit_pi_state_list(tsk); uprobe_free_utask(tsk); do { } while (0);# 777 "kernel/fork.c" if (tsk->clear_child_tid) { if (!(tsk->flags & 0x00000400) && (*(volatile int *)&(&mm->mm_users)->counter) > 1) { ({ might_fault(); ({ unsigned long __limit = current_thread_info()->addr_limit - 1; const typeof(*(tsk->clear_child_tid)) *__tmp_p = (tsk->clear_child_tid); register const typeof(*(tsk->clear_child_tid)) __r2 asm("r2") = (0); register const typeof(*(tsk->clear_child_tid)) *__p asm("r0") = __tmp_p; register unsigned long __l asm("r1") = __limit; register int __e asm("r0"); switch (sizeof(*(__p))) { case 1: __asm__ __volatile__ ( ".ifnc ""%0"",""r0"" ; .err ; .endif\n\t"".ifnc ""%2"",""r2"" ; .err ; .endif\n\t"".ifnc ""%3"",""r1"" ; .err ; .endif\n\t""bl __put_user_""1" : "=&r" (__e) : "0" (__p), "r" (__r2), "r" (__l) : "ip", "lr", "cc"); break; case 2: __asm__ __volatile__ ( ".ifnc ""%0"",""r0"" ; .err ; .endif\n\t"".ifnc ""%2"",""r2"" ; .err ; .endif\n\t"".ifnc ""%3"",""r1"" ; .err ; .endif\n\t""bl __put_user_""2" : "=&r" (__e) : "0" (__p), "r" (__r2), "r" (__l) : "ip", "lr", "cc"); break; case 4: __asm__ __volatile__ ( ".ifnc ""%0"",""r0"" ; .err ; .endif\n\t"".ifnc ""%2"",""r2"" ; .err ; .endif\n\t"".ifnc ""%3"",""r1"" ; .err ; .endif\n\t""bl __put_user_""4" : "=&r" (__e) : "0" (__p), "r" (__r2), "r" (__l) : "ip", "lr", "cc"); break; case 8: __asm__ __volatile__ ( ".ifnc ""%0"",""r0"" ; .err ; .endif\n\t"".ifnc ""%2"",""r2"" ; .err ; .endif\n\t"".ifnc ""%3"",""r1"" ; .err ; .endif\n\t""bl __put_user_""8" : "=&r" (__e) : "0" (__p), "r" (__r2), "r" (__l) : "ip", "lr", "cc"); break; default: __e = __put_user_bad(); break; } __e; }); }); sys_futex(tsk->clear_child_tid, 1, 1, ((void *)0), ((void *)0), 0); } tsk->clear_child_tid = ((void *)0); } if (tsk->vfork_done) complete_vfork_done(tsk);}ASSUME: - Given that the GCC 9.2.0 produces the same error, we are assuming that the same issue exists in this compiler as well.
GCC 7.5 Information:
sbeckwit@ubuntu:~/sp_dev/sp_archives/toolchains/arm7.5-toolchain/bin$ ./arm-buildroot-linux-gnueabi-gcc --versionarm-buildroot-linux-gnueabi-gcc.br_real (Buildroot 2020.02) 7.5.0Copyright (C) 2017 Free Software Foundation, Inc.This is free software; see the source for copying conditions. There is NOwarranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.using the same generated error for reference:
fork.s: Assembler messages:fork.s:2277: Error: .err encounteredscripts/Makefile.build:308: recipe for target 'kernel/fork.o' failedThis is in the function: mm_release()
.LBE1879: .loc 1 784 0 mov r2, r3.LVL102: mov r0, r5.LVL103: ldr r1, [r1, #8] sub r1, r1, #1.LVL104: .syntax divided@ 784 "kernel/fork.c" 1 .ifnc r0,r0 ; .err ; .endif **.ifnc r2,r2 ; .err ; .endif** .ifnc r1,r1 ; .err ; .endif bl __put_user_4@ 0 "" 2.LVL105: .arm .syntax unifiedhighlighted line correlates to the error above in the 8.3.0 compiler
From the fork.i file for this function:
# 747 "kernel/fork.c"void mm_release(struct task_struct *tsk, struct mm_struct *mm){ if (__builtin_expect(!!(tsk->robust_list), 0)) { exit_robust_list(tsk); tsk->robust_list = ((void *)0); } if (__builtin_expect(!!(!list_empty(&tsk->pi_state_list)), 0)) exit_pi_state_list(tsk); uprobe_free_utask(tsk); do { } while (0);# 777 "kernel/fork.c" if (tsk->clear_child_tid) { if (!(tsk->flags & 0x00000400) && (*(volatile int *)&(&mm->mm_users)->counter) > 1) { ({ might_fault(); ({ unsigned long __limit = current_thread_info()->addr_limit - 1; const typeof(*(tsk->clear_child_tid)) *__tmp_p = (tsk->clear_child_tid); register const typeof(*(tsk->clear_child_tid)) __r2 asm("r2") = (0); register const typeof(*(tsk->clear_child_tid)) *__p asm("r0") = __tmp_p; register unsigned long __l asm("r1") = __limit; register int __e asm("r0"); switch (sizeof(*(__p))) { case 1: __asm__ __volatile__ ( ".ifnc ""%0"",""r0"" ; .err ; .endif\n\t"".ifnc ""%2"",""r2"" ; .err ; .endif\n\t"".ifnc ""%3"",""r1"" ; .err ; .endif\n\t""bl __put_user_""1" : "=&r" (__e) : "0" (__p), "r" (__r2), "r" (__l) : "ip", "lr", "cc"); break; case 2: __asm__ __volatile__ ( ".ifnc ""%0"",""r0"" ; .err ; .endif\n\t"".ifnc ""%2"",""r2"" ; .err ; .endif\n\t"".ifnc ""%3"",""r1"" ; .err ; .endif\n\t""bl __put_user_""2" : "=&r" (__e) : "0" (__p), "r" (__r2), "r" (__l) : "ip", "lr", "cc"); break; case 4: __asm__ __volatile__ ( ".ifnc ""%0"",""r0"" ; .err ; .endif\n\t"".ifnc ""%2"",""r2"" ; .err ; .endif\n\t"".ifnc ""%3"",""r1"" ; .err ; .endif\n\t""bl __put_user_""4" : "=&r" (__e) : "0" (__p), "r" (__r2), "r" (__l) : "ip", "lr", "cc"); break; case 8: __asm__ __volatile__ ( ".ifnc ""%0"",""r0"" ; .err ; .endif\n\t"".ifnc ""%2"",""r2"" ; .err ; .endif\n\t"".ifnc ""%3"",""r1"" ; .err ; .endif\n\t""bl __put_user_""8" : "=&r" (__e) : "0" (__p), "r" (__r2), "r" (__l) : "ip", "lr", "cc"); break; default: __e = __put_user_bad(); break; } __e; }); }); sys_futex(tsk->clear_child_tid, 1, 1, ((void *)0), ((void *)0), 0); } tsk->clear_child_tid = ((void *)0); } if (tsk->vfork_done) complete_vfork_done(tsk);}The kernel code is the same. The Build options are the same.Difference is the compiler used, all other infrastructure utilities (i.e. Make 4.1) are the same.