Regression in rust-cross-canadian-aarch64


Peter Bergin
 

Hi,

I experience build failure in the compilation step of the recipe rust-cross-canadian-aarch64. I've used latest master from poky and just changed the MACHINE to qemuarm64. When I execute 'bitbake rust-cross-canadian-aarch64' it ends up in an error with object files in wrong format. I have been trying to bisect this issue but can not get a reproducible result. First thing I would like to get help with is to hear if someone else also experience this issue?

Here is a summary of what I see in log.do_compile:

<snip>
release/deps/std-b23e9faab40803e6.std.1f52b5e3-cgu.0.rcgu.o: Relocations in generic ELF (EM: 62)
/work/yocto/poky/build/tmp/work/x86_64-nativesdk-pokysdk-linux/rust-cross-canadian-aarch64/1.60.0-r0/recipe-sysroot-native/usr/bin/aarch64-poky-linux/../../libexec/aarch64-poky-linux/gcc/aarch64-poky-linux/12.1.0/ld: /work/yocto/poky/build/tmp/work/x86_64-nativesdk-pokysdk-linux/rust-cross-canadian-aarch64/1.60.0-r0/rustc-1.60.0-src/build/x86_64-unknown-linux-gnu/stage2-std/aarch64-poky-linux/release/deps/std-b23e9faab40803e6.std.1f52b5e3-cgu.0.rcgu.o: error adding symbols: file in wrong format
          collect2: error: ld returned 1 exit status
<snip>
RuntimeError: failed to run: /work/yocto/poky/build/tmp/work/x86_64-nativesdk-pokysdk-linux/rust-cross-canadian-aarch64/1.60.0-r0/rustc-1.60.0-src/build/bootstrap/debug/bootstrap -j 8 build --stage 2 --verbose
WARNING: /work/yocto/poky/build/tmp/work/x86_64-nativesdk-pokysdk-linux/rust-cross-canadian-aarch64/1.60.0-r0/temp/run.do_compile.1244376:177 exit 1 from 'python3 src/bootstrap/bootstrap.py -j 8 "$@" --verbose'
WARNING: Backtrace (BB generated script):
    #1: rust_runx, /work/yocto/poky/build/tmp/work/x86_64-nativesdk-pokysdk-linux/rust-cross-canadian-aarch64/1.60.0-r0/temp/run.do_compile.1244376, line 177
    #2: do_compile, /work/yocto/poky/build/tmp/work/x86_64-nativesdk-pokysdk-linux/rust-cross-canadian-aarch64/1.60.0-r0/temp/run.do_compile.1244376, line 160
    #3: main, /work/yocto/poky/build/tmp/work/x86_64-nativesdk-pokysdk-linux/rust-cross-canadian-aarch64/1.60.0-r0/temp/run.do_compile.1244376, line 200


Here is my build info:

Build Configuration:
BB_VERSION           = "2.0.1"
BUILD_SYS            = "x86_64-linux"
NATIVELSBSTRING      = "universal"
TARGET_SYS           = "aarch64-poky-linux"
MACHINE              = "qemuarm64"
DISTRO               = "poky"
DISTRO_VERSION       = "4.1+snapshot-cf7d8894545b83f55420fa33f7848e1bfc6754ff"
TUNE_FEATURES        = "aarch64 armv8a crc cortexa57"
TARGET_FPU           = ""
meta
meta-poky
meta-yocto-bsp       = "master:cf7d8894545b83f55420fa33f7848e1bfc6754ff"

/Peter


Peter Bergin
 

Hi again,

some progress on this issue.

On 2022-06-19 22:47, Peter Bergin wrote:
Hi,

I experience build failure in the compilation step of the recipe rust-cross-canadian-aarch64. I've used latest master from poky and just changed the MACHINE to qemuarm64. When I execute 'bitbake rust-cross-canadian-aarch64' it ends up in an error with object files in wrong format. I have been trying to bisect this issue but can not get a reproducible result. First thing I would like to get help with is to hear if someone else also experience this issue?

Here is a summary of what I see in log.do_compile:

<snip>
release/deps/std-b23e9faab40803e6.std.1f52b5e3-cgu.0.rcgu.o: Relocations in generic ELF (EM: 62)
/work/yocto/poky/build/tmp/work/x86_64-nativesdk-pokysdk-linux/rust-cross-canadian-aarch64/1.60.0-r0/recipe-sysroot-native/usr/bin/aarch64-poky-linux/../../libexec/aarch64-poky-linux/gcc/aarch64-poky-linux/12.1.0/ld: /work/yocto/poky/build/tmp/work/x86_64-nativesdk-pokysdk-linux/rust-cross-canadian-aarch64/1.60.0-r0/rustc-1.60.0-src/build/x86_64-unknown-linux-gnu/stage2-std/aarch64-poky-linux/release/deps/std-b23e9faab40803e6.std.1f52b5e3-cgu.0.rcgu.o: error adding symbols: file in wrong format
          collect2: error: ld returned 1 exit status
<snip>
RuntimeError: failed to run: /work/yocto/poky/build/tmp/work/x86_64-nativesdk-pokysdk-linux/rust-cross-canadian-aarch64/1.60.0-r0/rustc-1.60.0-src/build/bootstrap/debug/bootstrap -j 8 build --stage 2 --verbose
WARNING: /work/yocto/poky/build/tmp/work/x86_64-nativesdk-pokysdk-linux/rust-cross-canadian-aarch64/1.60.0-r0/temp/run.do_compile.1244376:177 exit 1 from 'python3 src/bootstrap/bootstrap.py -j 8 "$@" --verbose'
WARNING: Backtrace (BB generated script):
    #1: rust_runx, /work/yocto/poky/build/tmp/work/x86_64-nativesdk-pokysdk-linux/rust-cross-canadian-aarch64/1.60.0-r0/temp/run.do_compile.1244376, line 177
    #2: do_compile, /work/yocto/poky/build/tmp/work/x86_64-nativesdk-pokysdk-linux/rust-cross-canadian-aarch64/1.60.0-r0/temp/run.do_compile.1244376, line 160
    #3: main, /work/yocto/poky/build/tmp/work/x86_64-nativesdk-pokysdk-linux/rust-cross-canadian-aarch64/1.60.0-r0/temp/run.do_compile.1244376, line 200


Here is my build info:

Build Configuration:
BB_VERSION           = "2.0.1"
BUILD_SYS            = "x86_64-linux"
NATIVELSBSTRING      = "universal"
TARGET_SYS           = "aarch64-poky-linux"
MACHINE              = "qemuarm64"
DISTRO               = "poky"
DISTRO_VERSION       = "4.1+snapshot-cf7d8894545b83f55420fa33f7848e1bfc6754ff"
TUNE_FEATURES        = "aarch64 armv8a crc cortexa57"
TARGET_FPU           = ""
meta
meta-poky
meta-yocto-bsp       = "master:cf7d8894545b83f55420fa33f7848e1bfc6754ff"
The reason for this issue seems to be this commit:

    commit 781eaa955dce5deab47371c25dae72b36c011900
    Author: Richard Purdie <richard.purdie@...>
    Date:   Sat May 21 14:02:47 2022 +0100

        rust-common: Drop LLVM_TARGET and simplify

        This all seems over complicated for something which is basically always
        one of two values. This might even help cross-canadian work on something
        which isn't x86-64.

        (From OE-Core rev: bd36593ba3db758b3eacc974e48468a665967961)

        Signed-off-by: Richard Purdie <richard.purdie@...>


When building rust-cross-canadian-aarch64 the file 'tmp/work/x86_64-nativesdk-pokysdk-linux/rust-cross-canadian-aarch64/1.60.0-r0/targets/aarch64-poky-linux.json' will be populated with the information '"llvm-target": "x86_64-unknown-linux-gnu"'. This will lead to object files in wrong format during 'Building stage2 std artifacts (x86_64-unknown-linux-gnu -> aarch64-poky-linux)' in the do_compile step.

In 'meta/recipes-devtools/rust/rust-common.inc':

    # build tspec
    tspec = {}
    if bb.data.inherits_class('cross-canadian', d):
        tspec['llvm-target'] = d.getVar('RUST_HOST_SYS', arch_abi)
    else:
        tspec['llvm-target'] = d.getVar('RUST_TARGET_SYS', arch_abi)

So for some reason it seems intentional to set 'llvm-target' to HOST_SYS when 'cross-canadian'. The behavior has changed with this patch. Before 'llvm-target' was set to RUST_TARGET_SYS for all target archs other than x86_64 where it was set to RUST_HOST_SYS. I guess this change has to do with the commit message 'This might even help cross-canadian work on something which isn't x86-64.'? But unfortunately it seems to break the case building on x86_64 for aarch64. It would be good to get some help here to sort this out.

Thanks,
/Peter


Richard Purdie
 

On Tue, 2022-06-28 at 11:00 +0200, Peter Bergin wrote:
On 2022-06-19 22:47, Peter Bergin wrote:


The reason for this issue seems to be this commit:

    commit 781eaa955dce5deab47371c25dae72b36c011900
    Author: Richard Purdie <richard.purdie@...>
    Date:   Sat May 21 14:02:47 2022 +0100

        rust-common: Drop LLVM_TARGET and simplify

        This all seems over complicated for something which is
basically always
        one of two values. This might even help cross-canadian work on
something
        which isn't x86-64.

        (From OE-Core rev: bd36593ba3db758b3eacc974e48468a665967961)

        Signed-off-by: Richard Purdie <richard.purdie@...>


When building rust-cross-canadian-aarch64 the file
'tmp/work/x86_64-nativesdk-pokysdk-linux/rust-cross-canadian-aarch64/1.60.0-r0/targets/aarch64-poky-linux.json'
will be populated with the information '"llvm-target":
"x86_64-unknown-linux-gnu"'. This will lead to object files in wrong
format during 'Building stage2 std artifacts (x86_64-unknown-linux-gnu
-> aarch64-poky-linux)' in the do_compile step.

In 'meta/recipes-devtools/rust/rust-common.inc':

    # build tspec
    tspec = {}
    if bb.data.inherits_class('cross-canadian', d):
        tspec['llvm-target'] = d.getVar('RUST_HOST_SYS', arch_abi)
    else:
        tspec['llvm-target'] = d.getVar('RUST_TARGET_SYS', arch_abi)

So for some reason it seems intentional to set 'llvm-target' to HOST_SYS
when 'cross-canadian'. The behavior has changed with this patch. Before
'llvm-target' was set to RUST_TARGET_SYS for all target archs other than
x86_64 where it was set to RUST_HOST_SYS. I guess this change has to do
with the commit message 'This might even help cross-canadian work on
something which isn't x86-64.'? But unfortunately it seems to break the
case building on x86_64 for aarch64. It would be good to get some help
here to sort this out.
I'm wondering if we need something like:

diff --git a/meta/recipes-devtools/rust/rust-common.inc b/meta/recipes-devtools/rust/rust-common.inc
index 621cd4ad576..ef70c48d0f4 100644
--- a/meta/recipes-devtools/rust/rust-common.inc
+++ b/meta/recipes-devtools/rust/rust-common.inc
@@ -309,10 +309,7 @@ def rust_gen_target(d, thing, wd, features, cpu, arch, abi=""):

# build tspec
tspec = {}
- if bb.data.inherits_class('cross-canadian', d):
- tspec['llvm-target'] = d.getVar('RUST_HOST_SYS', arch_abi)
- else:
- tspec['llvm-target'] = d.getVar('RUST_TARGET_SYS', arch_abi)
+ tspec['llvm-target'] = d.getVar('RUST_TARGET_SYS', arch_abi)
tspec['data-layout'] = d.getVarFlag('DATA_LAYOUT', arch_abi)
tspec['max-atomic-width'] = int(d.getVarFlag('MAX_ATOMIC_WIDTH', arch_abi))
tspec['target-pointer-width'] = d.getVarFlag('TARGET_POINTER_WIDTH', arch_abi)


?

Cheers,

Richard


Peter Bergin
 

On 2022-06-28 12:09, Richard Purdie wrote:
On Tue, 2022-06-28 at 11:00 +0200, Peter Bergin wrote:
On 2022-06-19 22:47, Peter Bergin wrote:


The reason for this issue seems to be this commit:

    commit 781eaa955dce5deab47371c25dae72b36c011900
    Author: Richard Purdie <richard.purdie@...>
    Date:   Sat May 21 14:02:47 2022 +0100

        rust-common: Drop LLVM_TARGET and simplify

        This all seems over complicated for something which is
basically always
        one of two values. This might even help cross-canadian work on
something
        which isn't x86-64.

        (From OE-Core rev: bd36593ba3db758b3eacc974e48468a665967961)

        Signed-off-by: Richard Purdie <richard.purdie@...>


When building rust-cross-canadian-aarch64 the file
'tmp/work/x86_64-nativesdk-pokysdk-linux/rust-cross-canadian-aarch64/1.60.0-r0/targets/aarch64-poky-linux.json'
will be populated with the information '"llvm-target":
"x86_64-unknown-linux-gnu"'. This will lead to object files in wrong
format during 'Building stage2 std artifacts (x86_64-unknown-linux-gnu
-> aarch64-poky-linux)' in the do_compile step.

In 'meta/recipes-devtools/rust/rust-common.inc':

    # build tspec
    tspec = {}
    if bb.data.inherits_class('cross-canadian', d):
        tspec['llvm-target'] = d.getVar('RUST_HOST_SYS', arch_abi)
    else:
        tspec['llvm-target'] = d.getVar('RUST_TARGET_SYS', arch_abi)

So for some reason it seems intentional to set 'llvm-target' to HOST_SYS
when 'cross-canadian'. The behavior has changed with this patch. Before
'llvm-target' was set to RUST_TARGET_SYS for all target archs other than
x86_64 where it was set to RUST_HOST_SYS. I guess this change has to do
with the commit message 'This might even help cross-canadian work on
something which isn't x86-64.'? But unfortunately it seems to break the
case building on x86_64 for aarch64. It would be good to get some help
here to sort this out.
I'm wondering if we need something like:

diff --git a/meta/recipes-devtools/rust/rust-common.inc b/meta/recipes-devtools/rust/rust-common.inc
index 621cd4ad576..ef70c48d0f4 100644
--- a/meta/recipes-devtools/rust/rust-common.inc
+++ b/meta/recipes-devtools/rust/rust-common.inc
@@ -309,10 +309,7 @@ def rust_gen_target(d, thing, wd, features, cpu, arch, abi=""):
# build tspec
tspec = {}
- if bb.data.inherits_class('cross-canadian', d):
- tspec['llvm-target'] = d.getVar('RUST_HOST_SYS', arch_abi)
- else:
- tspec['llvm-target'] = d.getVar('RUST_TARGET_SYS', arch_abi)
+ tspec['llvm-target'] = d.getVar('RUST_TARGET_SYS', arch_abi)
tspec['data-layout'] = d.getVarFlag('DATA_LAYOUT', arch_abi)
tspec['max-atomic-width'] = int(d.getVarFlag('MAX_ATOMIC_WIDTH', arch_abi))
tspec['target-pointer-width'] = d.getVarFlag('TARGET_POINTER_WIDTH', arch_abi)


?
Thanks for suggestion! Yes, that will solve the problem for my setup. I was just wondering if you had any purpose with this if-else clause that should be kept or fixed?

I can send a patch for this to oe-core if nothing more is needed to be fixed.

Thanks,
/Peter


Richard Purdie
 

On Tue, 2022-06-28 at 11:00 +0200, Peter Bergin wrote:
When building rust-cross-canadian-aarch64 the file
'tmp/work/x86_64-nativesdk-pokysdk-linux/rust-cross-canadian-aarch64/1.60.0-r0/targets/aarch64-poky-linux.json'
will be populated with the information '"llvm-target":
"x86_64-unknown-linux-gnu"'. This will lead to object files in wrong
format during 'Building stage2 std artifacts (x86_64-unknown-linux-gnu
-> aarch64-poky-linux)' in the do_compile step.

In 'meta/recipes-devtools/rust/rust-common.inc':

    # build tspec
    tspec = {}
    if bb.data.inherits_class('cross-canadian', d):
        tspec['llvm-target'] = d.getVar('RUST_HOST_SYS', arch_abi)
    else:
        tspec['llvm-target'] = d.getVar('RUST_TARGET_SYS', arch_abi)

So for some reason it seems intentional to set 'llvm-target' to HOST_SYS
when 'cross-canadian'. The behavior has changed with this patch. Before
'llvm-target' was set to RUST_TARGET_SYS for all target archs other than
x86_64 where it was set to RUST_HOST_SYS. I guess this change has to do
with the commit message 'This might even help cross-canadian work on
something which isn't x86-64.'? But unfortunately it seems to break the
case building on x86_64 for aarch64. It would be good to get some help
here to sort this out.
I've spent an age staring at this code this morning. I don't think
cross-canadian has ever worked properly. It might happen that some
combinations build but there are also some that don't and clearly never
have.

SDKMACHINE = "aarch64", MACHINE = "qemuarm64" certainly doesn't work.
It appears to be missing a ninja-native dependency and when that is
added, it can't find rust-llvm.

Looking at the do_configure from rust.inc, it considers BUILD triplets
and TARGET triplets but not HOST ones, which probably dooms cross-
canadian to failure since there, BUILD != HOST != TARGET.

Someone is going to have to step up and sort out the canadian cross
rust pieces properly. Whilst my patch probably did regress one config,
I think there are much deeper issues in there.

Cheers,

Richard


Randy MacLeod
 

On 2022-06-28 07:38, Richard Purdie wrote:
On Tue, 2022-06-28 at 11:00 +0200, Peter Bergin wrote:
When building rust-cross-canadian-aarch64 the file
'tmp/work/x86_64-nativesdk-pokysdk-linux/rust-cross-canadian-aarch64/1.60.0-r0/targets/aarch64-poky-linux.json'
will be populated with the information '"llvm-target":
"x86_64-unknown-linux-gnu"'. This will lead to object files in wrong
format during 'Building stage2 std artifacts (x86_64-unknown-linux-gnu
-> aarch64-poky-linux)' in the do_compile step.

In 'meta/recipes-devtools/rust/rust-common.inc':

    # build tspec
    tspec = {}
    if bb.data.inherits_class('cross-canadian', d):
        tspec['llvm-target'] = d.getVar('RUST_HOST_SYS', arch_abi)
    else:
        tspec['llvm-target'] = d.getVar('RUST_TARGET_SYS', arch_abi)

So for some reason it seems intentional to set 'llvm-target' to HOST_SYS
when 'cross-canadian'. The behavior has changed with this patch. Before
'llvm-target' was set to RUST_TARGET_SYS for all target archs other than
x86_64 where it was set to RUST_HOST_SYS. I guess this change has to do
with the commit message 'This might even help cross-canadian work on
something which isn't x86-64.'? But unfortunately it seems to break the
case building on x86_64 for aarch64. It would be good to get some help
here to sort this out.
I've spent an age staring at this code this morning. I don't think
cross-canadian has ever worked properly. It might happen that some
combinations build but there are also some that don't and clearly never
have.

SDKMACHINE = "aarch64", MACHINE = "qemuarm64" certainly doesn't work.
It appears to be missing a ninja-native dependency and when that is
added, it can't find rust-llvm.

Looking at the do_configure from rust.inc, it considers BUILD triplets
and TARGET triplets but not HOST ones, which probably dooms cross-
canadian to failure since there, BUILD != HOST != TARGET.

Someone is going to have to step up and sort out the canadian cross
rust pieces properly. Whilst my patch probably did regress one config,
I think there are much deeper issues in there.
Sundeep and/or Naveen please take a look.

Peter,
Please open a Yocto bug and document the steps to reproduce the error.

../Randy



Cheers,

Richard




--
# Randy MacLeod
# Wind River Linux


Peter Bergin
 

On 2022-06-28 17:54, Randy MacLeod wrote:
On 2022-06-28 07:38, Richard Purdie wrote:
On Tue, 2022-06-28 at 11:00 +0200, Peter Bergin wrote:
When building rust-cross-canadian-aarch64 the file
'tmp/work/x86_64-nativesdk-pokysdk-linux/rust-cross-canadian-aarch64/1.60.0-r0/targets/aarch64-poky-linux.json'
will be populated with the information '"llvm-target":
"x86_64-unknown-linux-gnu"'. This will lead to object files in wrong
format during 'Building stage2 std artifacts (x86_64-unknown-linux-gnu
-> aarch64-poky-linux)' in the do_compile step.

In 'meta/recipes-devtools/rust/rust-common.inc':

      # build tspec
      tspec = {}
      if bb.data.inherits_class('cross-canadian', d):
          tspec['llvm-target'] = d.getVar('RUST_HOST_SYS', arch_abi)
      else:
          tspec['llvm-target'] = d.getVar('RUST_TARGET_SYS', arch_abi)

So for some reason it seems intentional to set 'llvm-target' to HOST_SYS
when 'cross-canadian'. The behavior has changed with this patch. Before
'llvm-target' was set to RUST_TARGET_SYS for all target archs other than
x86_64 where it was set to RUST_HOST_SYS. I guess this change has to do
with the commit message 'This might even help cross-canadian work on
something which isn't x86-64.'? But unfortunately it seems to break the
case building on x86_64 for aarch64. It would be good to get some help
here to sort this out.
I've spent an age staring at this code this morning. I don't think
cross-canadian has ever worked properly. It might happen that some
combinations build but there are also some that don't and clearly never
have.

SDKMACHINE = "aarch64", MACHINE = "qemuarm64" certainly doesn't work.
It appears to be missing a ninja-native dependency and when that is
added, it can't find rust-llvm.

Looking at the do_configure from rust.inc, it considers BUILD triplets
and TARGET triplets but not HOST ones, which probably dooms cross-
canadian to failure since there, BUILD != HOST != TARGET.

Someone is going to have to step up and sort out the canadian cross
rust pieces properly. Whilst my patch probably did regress one config,
I think there are much deeper issues in there.
Sundeep and/or Naveen please take a look.

Peter,
Please open a Yocto bug and document the steps to reproduce the error.
I've sent a patch to oe-core to fix the regression in my setup: https://lists.openembedded.org/g/openembedded-core/message/167368

Opened a bugzilla entry for rust cross-canadian: https://bugzilla.yoctoproject.org/show_bug.cgi?id=14849

Great if you could look in to this.

Thanks!
/Peter