Topics

Python3-native illegal instruction on poky dunfell


Daniela-Marinela Bistrean
 

Hello community,

I am trying to build core-image-minimal using poky only, on the dunfell branch. On certain machines the build fails unexpectedly with the following error:
Build Configuration:
BB_VERSION           = "1.46.0"
BUILD_SYS            = "x86_64-linux"
NATIVELSBSTRING      = "universal"
TARGET_SYS           = "x86_64-poky-linux"
MACHINE              = "qemux86-64"
DISTRO               = "poky"
DISTRO_VERSION       = "3.1.4"
TUNE_FEATURES        = "m64 core2"
TARGET_FPU           = ""
meta                
meta-poky            
meta-yocto-bsp       = "dunfell:cd3abf42dae2310ecaa8a97b2c08378a9f6e1282"

Initialising tasks: 100% |####################################################################################################################################################################| Time: 0:00:11
Sstate summary: Wanted 1025 Found 0 Missed 1025 Current 126 (0% match, 10% complete)
NOTE: Executing Tasks
ERROR: python3-setuptools-native-45.2.0-r0 do_compile: 'python3 setup.py build ' execution failed.
ERROR: python3-setuptools-native-45.2.0-r0 do_compile: Execution of '/home/builder/src/build_yocto-test/tmp/work/x86_64-linux/python3-setuptools-native/45.2.0-r0/temp/run.do_compile.141' failed with exit code 1:
Illegal instruction (core dumped)
WARNING: exit code 1 from a shell command.

ERROR: Logfile of failure stored in: /home/builder/src/build_yocto-test/tmp/work/x86_64-linux/python3-setuptools-native/45.2.0-r0/temp/log.do_compile.141
Log data follows:
| DEBUG: Executing shell function do_compile
| Illegal instruction (core dumped)
| ERROR: 'python3 setup.py build ' execution failed.
| WARNING: exit code 1 from a shell command.
| ERROR: Execution of '/home/builder/src/build_yocto-test/tmp/work/x86_64-linux/python3-setuptools-native/45.2.0-r0/temp/run.do_compile.141' failed with exit code 1:
| Illegal instruction (core dumped)
| WARNING: exit code 1 from a shell command.
|
ERROR: Task (virtual:native:/home/builder/src/poky/meta/recipes-devtools/python/python3-setuptools_45.2.0.bb:do_compile) failed with exit code '1'
NOTE: Tasks Summary: Attempted 576 tasks of which 572 didn't need to be rerun and 1 failed.

The do_compile task executes the following:
do_compile() {
    distutils3_do_compile
}

distutils3_do_compile() {
        cd /home/builder/src/build_sw.sys.icp21/tmp/work/x86_64-linux/python3-setuptools-native/45.2.0-r0/setuptools-45.2.0
        NO_FETCH_BUILD=1 \
        STAGING_INCDIR=/home/builder/src/build_sw.sys.icp21/tmp/work/x86_64-linux/python3-setuptools-native/45.2.0-r0/recipe-sysroot-native/usr/include \
        STAGING_LIBDIR=/home/builder/src/build_sw.sys.icp21/tmp/work/x86_64-linux/python3-setuptools-native/45.2.0-r0/recipe-sysroot-native/usr/lib \
        /home/builder/src/build_sw.sys.icp21/tmp/work/x86_64-linux/python3-setuptools-native/45.2.0-r0/recipe-sysroot-native/usr/bin/python3-native/python3 /home/builder/src/build_sw.sys.icp21/tmp/work/x86_64-linux/python3-setuptools-native/45.2.0-r0/setuptools-45.2.0/setup.py \
        build --build-base=/home/builder/src/build_sw.sys.icp21/tmp/work/x86_64-linux/python3-setuptools-native/45.2.0-r0/build  || \
        bbfatal_log "'python3 setup.py build ' execution failed."
}

On this particular machine the build fails even when it is tried out from within a container. The configuration of this machine is:
Distributor ID: Ubuntu
Description: Ubuntu 18.04.5 LTS
Release: 18.04
Codename: bionic

Linux vm 5.4.0-54-generic #60~18.04.1-Ubuntu SMP Fri Nov 6 17:25:16 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

4 x Intel(R) Core(TM) i5-8350U CPU @ 1.70GHz

The python3-native binary generated by Yocto works and I can run it on the host. I have identified that setup.py file of python3-setuptools-native recipe, part of do_compile fails with illegal instruction when trying to import setuptools. On other machines the build has no issues. The machine with issues is under a VPN and a proxy (company's machine).

Do you have any hint as why this might happen?

Thank you and looking forward to your feedback,
Daniela-Mărinela Bistrean


Ross Burton
 

Are you sharing sstate between multiple builders?

Ross

On Mon, 11 Jan 2021 at 08:58, Daniela-Marinela Bistrean
<daniela.marinela.bistrean@gmail.com> wrote:

Hello community,

I am trying to build core-image-minimal using poky only, on the dunfell branch. On certain machines the build fails unexpectedly with the following error:
Build Configuration:
BB_VERSION = "1.46.0"
BUILD_SYS = "x86_64-linux"
NATIVELSBSTRING = "universal"
TARGET_SYS = "x86_64-poky-linux"
MACHINE = "qemux86-64"
DISTRO = "poky"
DISTRO_VERSION = "3.1.4"
TUNE_FEATURES = "m64 core2"
TARGET_FPU = ""
meta
meta-poky
meta-yocto-bsp = "dunfell:cd3abf42dae2310ecaa8a97b2c08378a9f6e1282"

Initialising tasks: 100% |####################################################################################################################################################################| Time: 0:00:11
Sstate summary: Wanted 1025 Found 0 Missed 1025 Current 126 (0% match, 10% complete)
NOTE: Executing Tasks
ERROR: python3-setuptools-native-45.2.0-r0 do_compile: 'python3 setup.py build ' execution failed.
ERROR: python3-setuptools-native-45.2.0-r0 do_compile: Execution of '/home/builder/src/build_yocto-test/tmp/work/x86_64-linux/python3-setuptools-native/45.2.0-r0/temp/run.do_compile.141' failed with exit code 1:
Illegal instruction (core dumped)
WARNING: exit code 1 from a shell command.

ERROR: Logfile of failure stored in: /home/builder/src/build_yocto-test/tmp/work/x86_64-linux/python3-setuptools-native/45.2.0-r0/temp/log.do_compile.141
Log data follows:
| DEBUG: Executing shell function do_compile
| Illegal instruction (core dumped)
| ERROR: 'python3 setup.py build ' execution failed.
| WARNING: exit code 1 from a shell command.
| ERROR: Execution of '/home/builder/src/build_yocto-test/tmp/work/x86_64-linux/python3-setuptools-native/45.2.0-r0/temp/run.do_compile.141' failed with exit code 1:
| Illegal instruction (core dumped)
| WARNING: exit code 1 from a shell command.
|
ERROR: Task (virtual:native:/home/builder/src/poky/meta/recipes-devtools/python/python3-setuptools_45.2.0.bb:do_compile) failed with exit code '1'
NOTE: Tasks Summary: Attempted 576 tasks of which 572 didn't need to be rerun and 1 failed.

The do_compile task executes the following:
do_compile() {
distutils3_do_compile
}

distutils3_do_compile() {
cd /home/builder/src/build_sw.sys.icp21/tmp/work/x86_64-linux/python3-setuptools-native/45.2.0-r0/setuptools-45.2.0
NO_FETCH_BUILD=1 \
STAGING_INCDIR=/home/builder/src/build_sw.sys.icp21/tmp/work/x86_64-linux/python3-setuptools-native/45.2.0-r0/recipe-sysroot-native/usr/include \
STAGING_LIBDIR=/home/builder/src/build_sw.sys.icp21/tmp/work/x86_64-linux/python3-setuptools-native/45.2.0-r0/recipe-sysroot-native/usr/lib \
/home/builder/src/build_sw.sys.icp21/tmp/work/x86_64-linux/python3-setuptools-native/45.2.0-r0/recipe-sysroot-native/usr/bin/python3-native/python3 /home/builder/src/build_sw.sys.icp21/tmp/work/x86_64-linux/python3-setuptools-native/45.2.0-r0/setuptools-45.2.0/setup.py \
build --build-base=/home/builder/src/build_sw.sys.icp21/tmp/work/x86_64-linux/python3-setuptools-native/45.2.0-r0/build || \
bbfatal_log "'python3 setup.py build ' execution failed."
}

On this particular machine the build fails even when it is tried out from within a container. The configuration of this machine is:
Distributor ID: Ubuntu
Description: Ubuntu 18.04.5 LTS
Release: 18.04
Codename: bionic

Linux vm 5.4.0-54-generic #60~18.04.1-Ubuntu SMP Fri Nov 6 17:25:16 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

4 x Intel(R) Core(TM) i5-8350U CPU @ 1.70GHz

The python3-native binary generated by Yocto works and I can run it on the host. I have identified that setup.py file of python3-setuptools-native recipe, part of do_compile fails with illegal instruction when trying to import setuptools. On other machines the build has no issues. The machine with issues is under a VPN and a proxy (company's machine).

Do you have any hint as why this might happen?

Thank you and looking forward to your feedback,
Daniela-Mărinela Bistrean



Daniela-Marinela Bistrean
 

Thank you very much for your response!

I have only one builder and the sstate is not shared. Additionally I also tried building on the zeus branch and the build fails with the same error (illegal instruction) but this time at do_configure task. And do_configure calls the same python3-native with setup.py but with a different parameter.

Any help or hint is appreciated.

On Mon, 11 Jan 2021 at 14:43, Ross Burton <ross@...> wrote:
Are you sharing sstate between multiple builders?

Ross

On Mon, 11 Jan 2021 at 08:58, Daniela-Marinela Bistrean
<daniela.marinela.bistrean@...> wrote:
>
> Hello community,
>
> I am trying to build core-image-minimal using poky only, on the dunfell branch. On certain machines the build fails unexpectedly with the following error:
> Build Configuration:
> BB_VERSION           = "1.46.0"
> BUILD_SYS            = "x86_64-linux"
> NATIVELSBSTRING      = "universal"
> TARGET_SYS           = "x86_64-poky-linux"
> MACHINE              = "qemux86-64"
> DISTRO               = "poky"
> DISTRO_VERSION       = "3.1.4"
> TUNE_FEATURES        = "m64 core2"
> TARGET_FPU           = ""
> meta
> meta-poky
> meta-yocto-bsp       = "dunfell:cd3abf42dae2310ecaa8a97b2c08378a9f6e1282"
>
> Initialising tasks: 100% |####################################################################################################################################################################| Time: 0:00:11
> Sstate summary: Wanted 1025 Found 0 Missed 1025 Current 126 (0% match, 10% complete)
> NOTE: Executing Tasks
> ERROR: python3-setuptools-native-45.2.0-r0 do_compile: 'python3 setup.py build ' execution failed.
> ERROR: python3-setuptools-native-45.2.0-r0 do_compile: Execution of '/home/builder/src/build_yocto-test/tmp/work/x86_64-linux/python3-setuptools-native/45.2.0-r0/temp/run.do_compile.141' failed with exit code 1:
> Illegal instruction (core dumped)
> WARNING: exit code 1 from a shell command.
>
> ERROR: Logfile of failure stored in: /home/builder/src/build_yocto-test/tmp/work/x86_64-linux/python3-setuptools-native/45.2.0-r0/temp/log.do_compile.141
> Log data follows:
> | DEBUG: Executing shell function do_compile
> | Illegal instruction (core dumped)
> | ERROR: 'python3 setup.py build ' execution failed.
> | WARNING: exit code 1 from a shell command.
> | ERROR: Execution of '/home/builder/src/build_yocto-test/tmp/work/x86_64-linux/python3-setuptools-native/45.2.0-r0/temp/run.do_compile.141' failed with exit code 1:
> | Illegal instruction (core dumped)
> | WARNING: exit code 1 from a shell command.
> |
> ERROR: Task (virtual:native:/home/builder/src/poky/meta/recipes-devtools/python/python3-setuptools_45.2.0.bb:do_compile) failed with exit code '1'
> NOTE: Tasks Summary: Attempted 576 tasks of which 572 didn't need to be rerun and 1 failed.
>
> The do_compile task executes the following:
> do_compile() {
>     distutils3_do_compile
> }
>
> distutils3_do_compile() {
>         cd /home/builder/src/build_sw.sys.icp21/tmp/work/x86_64-linux/python3-setuptools-native/45.2.0-r0/setuptools-45.2.0
>         NO_FETCH_BUILD=1 \
>         STAGING_INCDIR=/home/builder/src/build_sw.sys.icp21/tmp/work/x86_64-linux/python3-setuptools-native/45.2.0-r0/recipe-sysroot-native/usr/include \
>         STAGING_LIBDIR=/home/builder/src/build_sw.sys.icp21/tmp/work/x86_64-linux/python3-setuptools-native/45.2.0-r0/recipe-sysroot-native/usr/lib \
>         /home/builder/src/build_sw.sys.icp21/tmp/work/x86_64-linux/python3-setuptools-native/45.2.0-r0/recipe-sysroot-native/usr/bin/python3-native/python3 /home/builder/src/build_sw.sys.icp21/tmp/work/x86_64-linux/python3-setuptools-native/45.2.0-r0/setuptools-45.2.0/setup.py \
>         build --build-base=/home/builder/src/build_sw.sys.icp21/tmp/work/x86_64-linux/python3-setuptools-native/45.2.0-r0/build  || \
>         bbfatal_log "'python3 setup.py build ' execution failed."
> }
>
> On this particular machine the build fails even when it is tried out from within a container. The configuration of this machine is:
> Distributor ID: Ubuntu
> Description: Ubuntu 18.04.5 LTS
> Release: 18.04
> Codename: bionic
>
> Linux vm 5.4.0-54-generic #60~18.04.1-Ubuntu SMP Fri Nov 6 17:25:16 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
>
> 4 x Intel(R) Core(TM) i5-8350U CPU @ 1.70GHz
>
> The python3-native binary generated by Yocto works and I can run it on the host. I have identified that setup.py file of python3-setuptools-native recipe, part of do_compile fails with illegal instruction when trying to import setuptools. On other machines the build has no issues. The machine with issues is under a VPN and a proxy (company's machine).
>
> Do you have any hint as why this might happen?
>
> Thank you and looking forward to your feedback,
> Daniela-Mărinela Bistrean
>
>
>


--
Cu deosebită consideraţie,
Daniela-Mărinela Bistrean


Daniela-Marinela Bistrean
 

Hello again,

I solved the mystery and I am describing it here in case anyone has the same problem.

Basically, what I omitted last time and seems to be critical to the problem is that I was running a virtual machine. The one bit causing the issue was that avx2 was disabled in my VM and python requires it. Enabling avx2 fixed the issue.

I managed to reproduce the issue with qemu using the following command line:
$ qemu-system-x86_64 -enable-kvm  -boot d -m 8G -cpu host,-avx2 -smp 4 -hda disk.bin

Hope it helps,
Daniela-Marinela Bistrean

On Mon, 11 Jan 2021 at 15:19, Daniela-Marinela Bistrean via lists.yoctoproject.org <daniela.marinela.bistrean=gmail.com@...> wrote:
Thank you very much for your response!

I have only one builder and the sstate is not shared. Additionally I also tried building on the zeus branch and the build fails with the same error (illegal instruction) but this time at do_configure task. And do_configure calls the same python3-native with setup.py but with a different parameter.

Any help or hint is appreciated.

On Mon, 11 Jan 2021 at 14:43, Ross Burton <ross@...> wrote:
Are you sharing sstate between multiple builders?

Ross

On Mon, 11 Jan 2021 at 08:58, Daniela-Marinela Bistrean
<daniela.marinela.bistrean@...> wrote:
>
> Hello community,
>
> I am trying to build core-image-minimal using poky only, on the dunfell branch. On certain machines the build fails unexpectedly with the following error:
> Build Configuration:
> BB_VERSION           = "1.46.0"
> BUILD_SYS            = "x86_64-linux"
> NATIVELSBSTRING      = "universal"
> TARGET_SYS           = "x86_64-poky-linux"
> MACHINE              = "qemux86-64"
> DISTRO               = "poky"
> DISTRO_VERSION       = "3.1.4"
> TUNE_FEATURES        = "m64 core2"
> TARGET_FPU           = ""
> meta
> meta-poky
> meta-yocto-bsp       = "dunfell:cd3abf42dae2310ecaa8a97b2c08378a9f6e1282"
>
> Initialising tasks: 100% |####################################################################################################################################################################| Time: 0:00:11
> Sstate summary: Wanted 1025 Found 0 Missed 1025 Current 126 (0% match, 10% complete)
> NOTE: Executing Tasks
> ERROR: python3-setuptools-native-45.2.0-r0 do_compile: 'python3 setup.py build ' execution failed.
> ERROR: python3-setuptools-native-45.2.0-r0 do_compile: Execution of '/home/builder/src/build_yocto-test/tmp/work/x86_64-linux/python3-setuptools-native/45.2.0-r0/temp/run.do_compile.141' failed with exit code 1:
> Illegal instruction (core dumped)
> WARNING: exit code 1 from a shell command.
>
> ERROR: Logfile of failure stored in: /home/builder/src/build_yocto-test/tmp/work/x86_64-linux/python3-setuptools-native/45.2.0-r0/temp/log.do_compile.141
> Log data follows:
> | DEBUG: Executing shell function do_compile
> | Illegal instruction (core dumped)
> | ERROR: 'python3 setup.py build ' execution failed.
> | WARNING: exit code 1 from a shell command.
> | ERROR: Execution of '/home/builder/src/build_yocto-test/tmp/work/x86_64-linux/python3-setuptools-native/45.2.0-r0/temp/run.do_compile.141' failed with exit code 1:
> | Illegal instruction (core dumped)
> | WARNING: exit code 1 from a shell command.
> |
> ERROR: Task (virtual:native:/home/builder/src/poky/meta/recipes-devtools/python/python3-setuptools_45.2.0.bb:do_compile) failed with exit code '1'
> NOTE: Tasks Summary: Attempted 576 tasks of which 572 didn't need to be rerun and 1 failed.
>
> The do_compile task executes the following:
> do_compile() {
>     distutils3_do_compile
> }
>
> distutils3_do_compile() {
>         cd /home/builder/src/build_sw.sys.icp21/tmp/work/x86_64-linux/python3-setuptools-native/45.2.0-r0/setuptools-45.2.0
>         NO_FETCH_BUILD=1 \
>         STAGING_INCDIR=/home/builder/src/build_sw.sys.icp21/tmp/work/x86_64-linux/python3-setuptools-native/45.2.0-r0/recipe-sysroot-native/usr/include \
>         STAGING_LIBDIR=/home/builder/src/build_sw.sys.icp21/tmp/work/x86_64-linux/python3-setuptools-native/45.2.0-r0/recipe-sysroot-native/usr/lib \
>         /home/builder/src/build_sw.sys.icp21/tmp/work/x86_64-linux/python3-setuptools-native/45.2.0-r0/recipe-sysroot-native/usr/bin/python3-native/python3 /home/builder/src/build_sw.sys.icp21/tmp/work/x86_64-linux/python3-setuptools-native/45.2.0-r0/setuptools-45.2.0/setup.py \
>         build --build-base=/home/builder/src/build_sw.sys.icp21/tmp/work/x86_64-linux/python3-setuptools-native/45.2.0-r0/build  || \
>         bbfatal_log "'python3 setup.py build ' execution failed."
> }
>
> On this particular machine the build fails even when it is tried out from within a container. The configuration of this machine is:
> Distributor ID: Ubuntu
> Description: Ubuntu 18.04.5 LTS
> Release: 18.04
> Codename: bionic
>
> Linux vm 5.4.0-54-generic #60~18.04.1-Ubuntu SMP Fri Nov 6 17:25:16 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
>
> 4 x Intel(R) Core(TM) i5-8350U CPU @ 1.70GHz
>
> The python3-native binary generated by Yocto works and I can run it on the host. I have identified that setup.py file of python3-setuptools-native recipe, part of do_compile fails with illegal instruction when trying to import setuptools. On other machines the build has no issues. The machine with issues is under a VPN and a proxy (company's machine).
>
> Do you have any hint as why this might happen?
>
> Thank you and looking forward to your feedback,
> Daniela-Mărinela Bistrean
>
>
>


--
Cu deosebită consideraţie,
Daniela-Mărinela Bistrean