How to prevent "relocation truncated to fit: R_AARCH64_PREL32 against `.text'" when creating an aarch64 baremetal program?

When playing with creating baremetal executables, I hit this error:

main.o:(.eh_frame+0x1c): relocation truncated to fit: R_AARCH64_PREL32 against `.text'
collect2: error: ld returned 1 exit status

I then managed to create a minimal reproduction example:

main.c

void _start(void) {}

notmain.S

.skip 32

link.ld

ENTRY(_start)
SECTIONS
{
  .text : {
    */bootloader.o(.text)
    *(.text)
    *(.rodata)
    *(.data)
    *(COMMON)
  }
  .bss : { *(.bss) }
  heap_low = .;
  . = . + 0x1000000;
  heap_top = .;
  . = . + 0x1000000;
  stack_top = .;
}

Compilation command:

aarch64-linux-gnu-gcc \
-save-temps \
-T link.ld \
-Wall \
-Werror \
-Wextra \
-Wl,--section-start=.text=0x80000000 \
-Xassembler -march=all \
-fno-pie \
-ggdb3 \
-no-pie \
-nostartfiles \
-nostdlib \
-static \
-o 'main.out' \
-pedantic \
notmain.S \
'main.c'

where aarch64-linux-gnu-gcc is GCC version 9.2.1 from Ubuntu 19.10's gcc-9-aarch64-linux-gnu package.

I later also tried on Ubuntu 18.04 GCC 7.5.0, and there it worked, so it would be good to understand what changed in between.

I know that it is not ideal to use a non-baremetal cross compiler for baremetal stuff, but can anyone point out if there is some command line option or code modification I could make to make the link work?

And if this is not possible with that toolchain, can someone clarify why? Which GCC build configuration option in particular makes this impossible?

I actually had an aarch64 crosstool-NG toolchain lying around described here and it works with that one, so it could actually be a problem with the toolchain.

I know that R_AARCH64_PREL32 is documented at: https://static.docs.arm.com/ihi0044/g/aaelf32.pdf and I have a general understanding of relocaiton: What does this GCC error "... relocation truncated to fit..." mean? but this one is a bit more than what I'd like to dig into right now.