Dumping state on this failure/lockup:

https://autobuilder.yoctoproject.org/typhoon/#/builders/97/builds/2753

Was locked up in eSDK testing of core-image-minimal in execution of
"devtool sdk-install meta-extsdk-toolchain".

python was sitting in pthread_cond_wait()

no other processes, no active fds that were interesting.

In the end I broke the lock in gdb which resulted in:

NOTE: Starting bitbake server...
Loading cache...done.
Loaded 0 entries from dependency cache.
Parsing recipes...done.
Parsing of 825 .bb files complete (0 cached, 825 parsed). 1459 targets, 53 skipped, 0 masked, 0 errors.
INFO: meta-extsdk-toolchain is already installed
Fatal Python error: drop_gil: PyCOND_WAIT(gil->switch_cond) failed
Python runtime state: finalizing (tstate=0xaaab08055150)
Current thread 0x0000ffffb6bae010 (most recent call first):
File "/home/pokybuild/yocto-worker/qemuarm64-armhost/build/build/tmp/work/qemuarm64-poky-linux/core-image-minimal/1.0-r0/testsdkext/buildtools/sysroots/aarch64-pokysdk-linux/usr/lib/python3.9/_weakrefset.py", line 39 in _remove
Aborted (core dumped)


i.e. it was sitting waiting on the python GIL.

Looks like a pthread bug.

pthreads being used were:

/home/pokybuild/yocto-worker/qemuarm64-armhost/build/build/tmp/work/qemuarm64-poky-linux/core-image-minimal/1.0-r0/testsdkext/buildtools/sysroots/aarch64-pokysdk-linux/lib/libpthread.so.0

i.e. *our* pthreads (and our python3.9).

Cheers,

Richard