|
Re: meta-aws warnings: SRCREV_FORMAT variable must be set when multiple SCMs are used
elberger@...
Hi Alexandre, I'm adding Nathan to see if there's anything he can see.
toggle quoted messageShow quoted text
I am not familiar with the warning at all, it looks new to me. Which build step has the commit hash for the repos we're checking out? That way I can compare apples-apples. -- rich
On 2/2/22 16:53, Alexandre Belloni wrote:
CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe. --
Richard Elberger Principal Technologist AWS IoT +1 646.927.6897 +1 203.942.8039 – mobile elberger@... – email
|
|
|
|
meta-aws warnings: SRCREV_FORMAT variable must be set when multiple SCMs are used
Alexandre Belloni
Hello,
It's been a few days that I'm struggling to understand what happened in this build. I've been looking at the fetcher changes, meta-aws changes but couldn't find anything relevant. https://autobuilder.yoctoproject.org/typhoon/#/builders/122/builds/713/steps/16/logs/warnings I would take any hint that you have ;) Regards, -- Alexandre Belloni, co-owner and COO, Bootlin Embedded Linux and Kernel engineering https://bootlin.com
|
|
|
|
SWAT Rotation schedule
Alexandre Belloni
Hello everyone,
Following the winter break, I would like to organize SWAT duty rotation. I've again prepared a schedule so it is easier for each one of you to plan for SWAT duty. Please check the table and let me know if you are not available for the selected week. SWAT duty will be from Friday to Thursday and the goal is to triage all the failures on swatbot before the weekly triage call happening at 2:30pm UTC. At the current failure rate, I would expect the task to take between a half a day and a day of work. ┌───────────────────────────────┬──────┬────────────┐ │ │ Week │ Start │ ├───────────────────────────────┼──────┼────────────┤ │ Anibal Limon │ 4 │ 28/01/2022 │ │ Saul Wold │ 5 │ 04/02/2022 │ │ Alejandro Hernandez Samaniego │ 6 │ 11/02/2022 │ │ Oleksiy Obitotskyy │ 7 │ 18/02/2022 │ │ Naveen Saini │ 8 │ 25/02/2022 │ │ Paul Eggleton │ 9 │ 04/03/2022 │ │ Christopher Larson │ 10 │ 11/03/2022 │ │ Jon Mason │ 11 │ 18/03/2022 │ │ Lee Chee Yang │ 12 │ 25/03/2022 │ │ Minjae Kim │ 13 │ 01/04/2022 │ │ Jaga │ 14 │ 08/04/2022 │ │ Leo Sandoval │ 15 │ 15/04/2022 │ │ Ross Burton │ 16 │ 22/04/2022 │ │ Valerii Chernous │ 17 │ 29/04/2022 │ └───────────────────────────────┴──────┴────────────┘ Anibal you would be the next one on the list, starting this Friday, can you confirm you are available? There are currently 2 failures to triage on swatbot, I'm going to take care of those. When filling in bugs or bug comments, please also add the worker name and whether it has an SSD. Currently, we have: ┌──────────────────┬─────────┐ │ alma8-ty-1 │ SSD │ │ alma8-ty-2 │ SSD │ │ debian11-ty-3 │ SSD │ │ fedora35-ty-1 │ SSD │ │ fedora35-ty-2 │ SSD │ │ opensuse153-ty-1 │ SSD │ │ stream8-ty-1 │ SSD │ │ ubuntu2004-ty-1 │ SSD │ │ ubuntu2110-ty-2 │ SSD │ │ centos7-ty-4 │ Non-SSD │ │ debian11-ty-1 │ Non-SSD │ │ fedora34-ty-1 │ Non-SSD │ │ perf-centos7 │ Non-SSD │ │ perf-ubuntu1604 │ Non-SSD │ │ tumbleweed-ty-3 │ Non-SSD │ │ ubuntu1604-ty-1 │ Non-SSD │ │ ubuntu1804-arm-1 │ Non-SSD │ │ ubuntu1804-ty-3 │ Non-SSD │ │ ubuntu2004-arm-1 │ Non-SSD │ │ centos8-ty-1 │ Non-SSD │ │ centos8-ty-2 │ Non-SSD │ │ debian10-ty-1 │ Non-SSD │ │ debian11-ty-2 │ Non-SSD │ │ debian9-ty-2 │ Non-SSD │ │ opensuse154-ty-1 │ Non-SSD │ └──────────────────┴─────────┘ Thanks! -- Alexandre Belloni, Bootlin Embedded Linux and Kernel engineering https://bootlin.com
|
|
|
|
Re: New warning in meta-aws
Alexandre Belloni
On 17/01/2022 23:05:18+0000, Elberger, Richard wrote:
Thank you - it is in our current sprint to fix, it was noted last week when the distutils warnings across openembedded really alarmed us.Sure, it was kind of hidden in all those meta-oe warnings so I wanted to make sure you knew about it. It’s a simple fix so I don’t expect an extended CR :D-- Alexandre Belloni, co-owner and COO, Bootlin Embedded Linux and Kernel engineering https://bootlin.com
|
|
|
|
Re: New warning in meta-aws
elberger@...
Thank you - it is in our current sprint to fix, it was noted last week when the distutils warnings across openembedded really alarmed us.
toggle quoted messageShow quoted text
It’s a simple fix so I don’t expect an extended CR :D
On Jan 17, 2022, at 17:44, Alexandre Belloni <alexandre.belloni@...> wrote:
|
|
|
|
New warning in meta-aws
Alexandre Belloni
Hello Richard,
There is a new warning in meta-aws: WARNING: /home/pokybuild/yocto-worker/meta-aws/build/meta-aws/recipes-iot/aws-iot-securetunneling-localproxy/aws-iot-securetunneling-localproxy_git.bb: URL: git://git@.../aws-samples/aws-iot-securetunneling-localproxy.git;protocol=ssh does not set any branch parameter. The future default branch used by tools and repositories is uncertain and we will therefore soon require this is set in all git urls. I believe this is the only affected recipe. -- Alexandre Belloni, co-owner and COO, Bootlin Embedded Linux and Kernel engineering https://bootlin.com
|
|
|
|
Re: SWAT Rotation schedule
Valerii Chernous -X (vchernou - GLOBALLOGIC INC at Cisco)
Good morning Alexandre,
Sorry, I forgot about this work
I am starting my work today
Where I can check rotation shedule in feature?
Also I will do this work first time, Oleksiy(oobitots@...) from my team provide to me information how this work should be done but can I write to you if I have some additional questions?
Best regards,
Valerii
From: Alexandre Belloni <alexandre.belloni@...>
Sent: Tuesday, December 14, 2021 12:38 AM To: Valerii Chernous -X (vchernou - GLOBALLOGIC INC at Cisco) <vchernou@...> Cc: Ross Burton <ross@...> Subject: SWAT Rotation schedule Hello Valerii,
A quick reminder that you are on SWAT duty this week. ┌──────┬────────────┐ │ Week │ Start │ ┌───────────────────────────────┼──────┼────────────┤ │ Alejandro Hernandez Samaniego │ 37 │ 17/09/2021 │ │ Oleksiy Obitotskyy │ 38 │ 24/09/2021 │ │ Naveen Saini │ 39 │ 01/10/2021 │ │ Thomas Perrot │ 40 │ 08/10/2021 │ │ Paul Eggleton │ 41 │ 15/10/2021 │ │ Christopher Larson │ 42 │ 22/10/2021 │ │ Jon Mason │ 43 │ 29/10/2021 │ │ Lee Chee Yang │ 44 │ 05/11/2021 │ │ Minjae Kim │ 45 │ 12/11/2021 │ │ Jaga │ 46 │ 19/11/2021 │ │ Leo Sandoval │ 47 │ 26/11/2021 │ │ Ross Burton │ 48 │ 03/12/2021 │ │ Valerii Chernous │ 49 │ 10/12/2021 │ │ Anibal Limon │ 50 │ 17/12/2021 │ │ Saul Wold │ 51 │ 24/12/2021 │ └───────────────────────────────┴──────┴────────────┘ SWAT duty will be from Friday to Thursday and the goal is to triage all the failures on swatbot before the weekly triage call happening at 2:30pm UTC. Thanks! -- Alexandre Belloni, Bootlin Embedded Linux and Kernel engineering https://bootlin.com
|
|
|
|
qemux86 shutdown timeout on ubuntu1804-ty-3 rcu stall logs
Richard Purdie
qemux86 didn't shutdown 120s on ubuntu1804-ty-3
https://autobuilder.yoctoproject.org/typhoon/#/builders/59/builds/4421 showed an rcu stall in the kernel logs cat pokybuild@ubuntu1804-ty-3:~/yocto-worker/qemux86/build/build-st-48375/tmp/work/qemux86-poky-linux/core-image-minimal/1.0-r0/testimage/qemu_boot_log.20211205013910 c[?7l[2J[0mSeaBIOS (version rel-1.14.0-0-g155821a1990b-prebuilt.qemu.org) iPXE (http://ipxe.org) 00:02.0 CA00 PCI2.10 PnP PMM+0FF86E80+0FEE6E80 CA00 Press Ctrl-B to configure iPXE (PCI 00:02.0)...^M Booting from ROM... c[?7l[2J[ 0.000000] Linux version 5.14.18-yocto-standard (oe-user@oe-host) (i686-poky-linux-gcc (GCC) 11.2.0, GNU ld (GNU Binutils) 2.37.20210721) #1 SMP PREEMPT Tue Nov 16 03:53:43 UTC 2021 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Enabled xstate features 0x3, context size is 576 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1504 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x000000000ffd7fff] usable [ 0.000000] BIOS-e820: [mem 0x000000000ffd8000-0x000000000fffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000b0000000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fed1c000-0x00000000fed1ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] Notice: NX (Execute Disable) protection cannot be enabled: non-PAE kernel! [ 0.000000] user-defined physical RAM map: [ 0.000000] user: [mem 0x0000000000000000-0x000000000009fbff] usable [....] [ 2.491515] scsi 2:0:0:0: CD-ROM QEMU QEMU DVD-ROM 2.5+ PQ: 0 ANSI: 5 [ 2.517660] sr 2:0:0:0: [sr0] scsi3-mmc drive: 4x/4x cd/rw xa/form2 tray [ 2.518434] cdrom: Uniform CD-ROM driver Revision: 3.20 [ 2.569165] input: QEMU QEMU USB Tablet as /devices/pci0000:00/0000:00:1d.7/usb1/1-1/1-1:1.0/0003:0627:0001.0001/input/input4 [ 2.571240] hid-generic 0003:0627:0001.0001: input: USB HID v0.01 Mouse [QEMU QEMU USB Tablet] on usb-0000:00:1d.7-1/input0 [ 2.638291] input: ImExPS/2 Generic Explorer Mouse as /devices/platform/i8042/serio1/input/input3 [ 2.714786] IP-Config: Complete: [ 2.715049] device=eth0, hwaddr=52:54:00:12:34:02, ipaddr=192.168.7.2, mask=255.255.255.0, gw=192.168.7.1 [ 2.715846] host=192.168.7.2, domain=, nis-domain=(none) [ 2.716053] bootserver=255.255.255.255, rootserver=255.255.255.255, rootpath= [ 2.725860] md: Waiting for all devices to be available before autodetect [ 2.726464] md: If you don't use raid, use raid=noautodetect [ 2.726757] md: Autodetecting RAID arrays. [ 2.726959] md: autorun ... [ 2.727195] md: ... autorun DONE. [ 2.842432] EXT4-fs (vda): mounted filesystem with ordered data mode. Opts: (null). Quota mode: disabled. [ 2.843771] VFS: Mounted root (ext4 filesystem) on device 253:0. [ 2.846695] devtmpfs: mounted [ 2.936371] Freeing unused kernel image (initmem) memory: 884K [ 2.937830] Write protecting kernel text and read-only data: 15120k [ 2.938432] Run /sbin/init as init process ^MINIT: version 3.00 booting Starting udev [ 4.824244] udevd[156]: starting version 3.2.10 [ 4.949028] udevd[157]: starting eudev-3.2.10 [ 6.976057] EXT4-fs (vda): re-mounted. Opts: (null). Quota mode: disabled. [ 32.954837] hrtimer: interrupt took 5666197 ns ^MINIT: Entering runlevel: 5 Configuring network interfaces... ip: RTNETLINK answers: File exists Starting syslogd/klogd: done Poky (Yocto Project Reference Distro) 3.4+snapshot-22ba018eb357be2575ae346f69c7591a0bc80dde qemux86 /dev/ttyS0 ^Mqemux86 login: ^MINIT: Switching to runlevel: 0 Stopping syslogd/klogd: stopped syslogd (pid 311) stopped klogd (pid 314) done Deconfiguring network interfaces... ifdown: interface lo not configured done. Sending all processes the TERM signal... Sending all processes the KILL signal... Unmounting remote filesystems... [ 141.107314] rcu: INFO: rcu_preempt self-detected stall on CPU [ 141.108515] rcu: 0-...0: (1 GPs behind) idle=76a/1/0x40000000 softirq=1893/1894 fqs=20 [ 141.109282] (t=66396 jiffies g=3577 q=2) [ 141.110253] NMI backtrace for cpu 0 [ 141.111179] CPU: 0 PID: 348 Comm: S38urandom Not tainted 5.14.18-yocto-standard #1 [ 141.111875] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS rel-1.14.0-0-g155821a1990b-prebuilt.qemu.org 04/01/2014 [ 141.113333] Call Trace: [ 141.114141] dump_stack_lvl+0x34/0x44 [ 141.114808] dump_stack+0xd/0x10 [ 141.115161] nmi_cpu_backtrace.cold+0x16/0x5b [ 141.115442] ? lapic_can_unplug_cpu+0x70/0x70 [ 141.115591] nmi_trigger_cpumask_backtrace+0x97/0xc0 [ 141.115751] arch_trigger_cpumask_backtrace+0x15/0x20 [ 141.115906] rcu_dump_cpu_stacks+0x9c/0xc8 [ 141.116059] rcu_sched_clock_irq.cold+0xa9/0x3cc [ 141.116306] ? cpuacct_account_field+0x59/0x70 [ 141.116459] ? account_user_time+0xa8/0xb0 [ 141.116598] ? account_process_tick+0xab/0x170 [ 141.116747] update_process_times+0x77/0xb0 [ 141.116915] tick_sched_timer+0xcb/0x100 [ 141.117055] __hrtimer_run_queues+0x14a/0x2c0 [ 141.117261] ? get_cpu_iowait_time_us+0x150/0x150 [ 141.117407] hrtimer_interrupt+0x113/0x2e0 [ 141.117539] ? syscall_exit_to_user_mode+0x1b/0x40 [ 141.117692] ? sysvec_call_function_single+0x30/0x30 [ 141.117846] __sysvec_apic_timer_interrupt+0x59/0x100 [ 141.117998] sysvec_apic_timer_interrupt+0x1a/0x30 [ 141.118169] handle_exception+0x133/0x133 [ 141.118462] EIP: 0x4ccf15 [ 141.118965] Code: 83 f2 01 09 d0 88 46 01 8b 45 a8 85 c0 0f 85 88 f1 ff ff c6 07 88 0f b6 1e 83 c7 01 80 fb 02 0f 85 ba f0 ff ff e9 a8 f0 ff ff <0f> b7 06 66 25 00 fb 66 0d 01 04 66 89 06 8b 46 04 85 c0 74 c9 eb [ 141.119615] EAX: 00000004 EBX: 00000000 ECX: 004ccf15 EDX: 00000004 [ 141.119833] ESI: bfefab50 EDI: 0052f3ac EBP: bfefab88 ESP: bfefab10 [ 141.120039] DS: 007b ES: 007b FS: 0000 GS: 0033 SS: 007b EFLAGS: 00000203 [ 141.120407] ? sysvec_call_function_single+0x30/0x30 Deactivating swap... Unmounting local filesystems...
|
|
|
|
Another hang on centos8-ty-1 in make with a zombie
Richard Purdie
https://autobuilder.yoctoproject.org/typhoon/#/builders/79/builds/2808
2060878 ? S 3:51 bitbake-server /home/pokybuild/yocto-worker/oe-selftest-centos/build/bitbake/bin/bitbake-server decafbad 3 5 /home/pokybuild/yocto-worker/oe-selftest-centos/build/build-st-351364/bitbake-cookerdaemon.log /home/pokybuild/yocto-worker/oe-selftest-centos/build/build-st-351364/bitbake.lock /home/pokybuild/yocto-worker/oe-selftest-centos/build/build-st-351364/bitbake.sock 0 None 0 2069952 ? Sl 1:09 python3 /home/pokybuild/yocto-worker/oe-selftest-centos/build/bitbake/bin/bitbake-worker decafbad 2085859 ? I 0:00 [kworker/49:1-mm_percpu_wq] 2092425 ? SNs 0:05 python3 /home/pokybuild/yocto-worker/oe-selftest-centos/build/bitbake/bin/bitbake-worker decafbad 2092748 ? SN 0:00 /bin/sh /home/pokybuild/yocto-worker/oe-selftest-centos/build/build-st-351364/tmp/work/qemux86_64-poky-linux/linux-yocto/5.14.15+git999-r0/temp/run.do_compile_kernelmodules.2092425 2092766 ? SN 0:00 make HOSTCC=gcc HOSTCFLAGS=-isystem/home/pokybuild/yocto-worker/oe-selftest-centos/build/build-st-351364/tmp/work/qemux86_64-poky-linux/linux-yocto/5.14.15+git999-r0/recipe-sysroot-native/usr/include -O2 -pipe HOSTLDFLAGS=-L/home/pokybuild/yocto-worker/oe-selftest-centos/build/build-st-351364/tmp/work/qemux86_64-poky-linux/linux-yocto/5.14.15+git999-r0/recipe-sysroot-native/usr/lib -L/home/pokybuild/yocto-worker/oe-selftest-centos/build/build-st-351364/tmp/work/qemux86_64-poky-linux/linux-yocto/5.14.15+git999-r0/recipe-sysroot-native/lib -Wl,--enable-new-dtags -Wl,-rpath-link,/home/pokybuild/yocto-worker/oe-selftest-centos/build/build-st-351364/tmp/work/qemux86_64-poky-linux/linux-yocto/5.14.15+git999-r0/recipe-sysroot-native/usr/lib -Wl,-rpath-link,/home/pokybuild/yocto-worker/oe-selftest-centos/build/build-st-351364/tmp/work/qemux86_64-poky-linux/linux-yocto/5.14.15+git999-r0/recipe-sysroot-native/lib -Wl,-rpath,/home/pokybuild/yocto-worker/oe-selftest-centos/build/build-st-351364/tmp/work/qemux86_64-poky-linux/linux-yocto/5.14.15+git999-r0/recipe-sysroot-native/usr/lib -Wl,-rpath,/home/pokybuild/yocto-worker/oe-selftest-centos/build/build-st-351364/tmp/work/qemux86_64-poky-linux/linux-yocto/5.14.15+git999-r0/recipe-sysroot-native/lib -Wl,-O1 -Wl,--allow-shlib-undefined -Wl,--dynamic-linker=/home/pokybuild/yocto-worker/oe-selftest-centos/build/build-st-351364/tmp/sysroots-uninative/x86_64-linux/lib/ld-linux-x86-64.so.2 HOSTCPP=gcc -E HOSTCXX=g++ HOSTCXXFLAGS=-isystem/home/pokybuild/yocto-worker/oe-selftest-centos/build/build-st-351364/tmp/work/qemux86_64-poky-linux/linux-yocto/5.14.15+git999-r0/recipe-sysroot-native/usr/include -O2 -pipe -C /home/pokybuild/yocto-worker/oe-selftest-centos/build/build-st-351364/tmp/work/qemux86_64-poky-linux/linux-yocto/5.14.15+git999-r0/linux-yocto-5.14.15+git999/ -j 16 -l 52 modules CC=x86_64-poky-linux-gcc -fuse-ld=bfd -fmacro-prefix-map=/home/pokybuild/yocto-worker/oe-selftest-centos/build/build-st-351364/tmp/work/qemux86_64-poky-linux/linux-yocto/5.14.15+git999-r0=/usr/src/debug/linux-yocto/5.14.15+git999-r0 -fdebug-prefix-map=/home/pokybuild/yocto-worker/oe-selftest-centos/build/build-st-351364/tmp/work/qemux86_64-poky-linux/linux-yocto/5.14.15+git999-r0=/usr/src/debug/linux-yocto/5.14.15+git999-r0 -fdebug-prefix-map=/home/pokybuild/yocto-worker/oe-selftest-centos/build/build-st-351364/tmp/work/qemux86_64-poky-linux/linux-yocto/5.14.15+git999-r0/recipe-sysroot= -fdebug-prefix-map=/home/pokybuild/yocto-worker/oe-selftest-centos/build/build-st-351364/tmp/work/qemux86_64-poky-linux/linux-yocto/5.14.15+git999-r0/recipe-sysroot-native= -fdebug-prefix-map=/home/pokybuild/yocto-worker/oe-selftest-centos/build/build-st-351364/tmp/work-shared/qemux86-64/kernel-source=/usr/src/kernel LD=x86_64-poky-linux-ld.bfd 2092768 ? SN 0:00 make -C /home/pokybuild/yocto-worker/oe-selftest-centos/build/build-st-351364/tmp/work/qemux86_64-poky-linux/linux-yocto/5.14.15+git999-r0/linux-yocto-5.14.15+git999 -f /tmp/devtoolqabmdf720w/Makefile modules 2095203 ? SN 0:00 make -f /tmp/devtoolqabmdf720w/scripts/Makefile.build obj=sound single-build= need-builtin=1 need-modorder=1 2095631 ? SN 0:00 make -f /tmp/devtoolqabmdf720w/scripts/Makefile.build obj=sound/i2c need-builtin= need-modorder=1 2095638 ? SN 0:00 make -f /tmp/devtoolqabmdf720w/scripts/Makefile.build obj=sound/i2c/other need-builtin= need-modorder=1 2095648 ? ZN 0:00 [sh] <defunct> Worker(2069952)─┬─linux-yocto:com(2092425)───run.do_compile_(2092748)───make(2092766)───make(2092768)───make(2095203)───make(2095631)───make(2095638)───sh(2095648) └─{Worker}(2070710) I did a: kill -17 2095638 which seems to have unblocked it. Cheers, Richard
|
|
|
|
Hung linux-yocto build in do_compile, centos8
Richard Purdie
We have another hung build:
https://autobuilder.yoctoproject.org/typhoon/#/builders/79/builds/2805 [pokybuild@centos8-ty-1 ~]$ pstree -p 3610382 Worker(3610382)─┬─linux-yocto:com(3686989)───run.do_compile.(3687229)───make(3687234)───make(3687235)───make(3700121)───make(3703528)───sh(3703566) └─{Worker}(3610616) 3610382 ? Sl 0:28 python3 /home/pokybuild/yocto-worker/oe-selftest-centos/build/bitbake/bin/bitbake-worker decafbad 3614645 ? S 0:00 /home/pokybuild/yocto-worker/oe-selftest-centos/build/build-st-2484929/tmp/sysroots-components/x86_64/pseudo-native/usr/bin/pseudo bitbake-worker decafbadbeef 3614650 ? Sl 0:27 python3 /home/pokybuild/yocto-worker/oe-selftest-centos/build/bitbake/bin/bitbake-worker decafbadbeef 3686989 ? SNs 0:02 python3 /home/pokybuild/yocto-worker/oe-selftest-centos/build/bitbake/bin/bitbake-worker decafbad 3687229 ? SN 0:00 /bin/sh /home/pokybuild/yocto-worker/oe-selftest-centos/build/build-st-2484929/tmp/work/qemux86_64-poky-linux/linux-yocto/5.14.15+gitAUTOINC+edb4dc09c5_f04b30fc15-r0/temp/run.do_compile.3686989 3687234 ? SN 0:00 make -j 16 -l 52 HOSTCC=gcc HOSTCFLAGS=-isystem/home/pokybuild/yocto-worker/oe-selftest-centos/build/build-st-2484929/tmp/work/qemux86_64-poky-linux/linux-yocto/5.14.15+gitAUTOINC+edb4dc09c5_f04b30fc15-r0/recipe-sysroot-native/usr/include -O2 -pipe HOSTLDFLAGS=-L/home/pokybuild/yocto-worker/oe-selftest-centos/build/build-st-2484929/tmp/work/qemux86_64-poky-linux/linux-yocto/5.14.15+gitAUTOINC+edb4dc09c5_f04b30fc15-r0/recipe-sysroot-native/usr/lib -L/home/pokybuild/yocto-worker/oe-selftest-centos/build/build-st-2484929/tmp/work/qemux86_64-poky-linux/linux-yocto/5.14.15+gitAUTOINC+edb4dc09c5_f04b30fc15-r0/recipe-sysroot-native/lib -Wl,--enable-new-dtags -Wl,-rpath-link,/home/pokybuild/yocto-worker/oe-selftest-centos/build/build-st-2484929/tmp/work/qemux86_64-poky-linux/linux-yocto/5.14.15+gitAUTOINC+edb4dc09c5_f04b30fc15-r0/recipe-sysroot-native/usr/lib -Wl,-rpath-link,/home/pokybuild/yocto-worker/oe-selftest-centos/build/build-st-2484929/tmp/work/qemux86_64-poky-linux/linux-yocto/5.14.15+gitAUTOINC+edb4dc09c5_f04b30fc15-r0/recipe-sysroot-native/lib -Wl,-rpath,/home/pokybuild/yocto-worker/oe-selftest-centos/build/build-st-2484929/tmp/work/qemux86_64-poky-linux/linux-yocto/5.14.15+gitAUTOINC+edb4dc09c5_f04b30fc15-r0/recipe-sysroot-native/usr/lib -Wl,-rpath,/home/pokybuild/yocto-worker/oe-selftest-centos/build/build-st-2484929/tmp/work/qemux86_64-poky-linux/linux-yocto/5.14.15+gitAUTOINC+edb4dc09c5_f04b30fc15-r0/recipe-sysroot-native/lib -Wl,-O1 -Wl,--allow-shlib-undefined -Wl,--dynamic-linker=/home/pokybuild/yocto-worker/oe-selftest-centos/build/build-st-2484929/tmp/sysroots-uninative/x86_64-linux/lib/ld-linux-x86-64.so.2 HOSTCPP=gcc -E HOSTCXX=g++ HOSTCXXFLAGS=-isystem/home/pokybuild/yocto-worker/oe-selftest-centos/build/build-st-2484929/tmp/work/qemux86_64-poky-linux/linux-yocto/5.14.15+gitAUTOINC+edb4dc09c5_f04b30fc15-r0/recipe-sysroot-native/usr/include -O2 -pipe bzImage CC=x86_64-poky-linux-gcc -fuse-ld=bfd -fmacro-prefix-map=/home/pokybuild/yocto-worker/oe-selftest-centos/build/build-st-2484929/tmp/work/qemux86_64-poky-linux/linux-yocto/5.14.15+gitAUTOINC+edb4dc09c5_f04b30fc15-r0=/usr/src/debug/linux-yocto/5.14.15+gitAUTOINC+edb4dc09c5_f04b30fc15-r0 -fdebug-prefix-map=/home/pokybuild/yocto-worker/oe-selftest-centos/build/build-st-2484929/tmp/work/qemux86_64-poky-linux/linux-yocto/5.14.15+gitAUTOINC+edb4dc09c5_f04b30fc15-r0=/usr/src/debug/linux-yocto/5.14.15+gitAUTOINC+edb4dc09c5_f04b30fc15-r0 -fdebug-prefix-map=/home/pokybuild/yocto-worker/oe-selftest-centos/build/build-st-2484929/tmp/work/qemux86_64-poky-linux/linux-yocto/5.14.15+gitAUTOINC+edb4dc09c5_f04b30fc15-r0/recipe-sysroot= -fdebug-prefix-map=/home/pokybuild/yocto-worker/oe-selftest-centos/build/build-st-2484929/tmp/work/qemux86_64-poky-linux/linux-yocto/5.14.15+gitAUTOINC+edb4dc09c5_f04b30fc15-r0/recipe-sysroot-native= -fdebug-prefix-map=/home/pokybuild/yocto-worker/oe-selftest-centos/build/build-st-2484929/tmp/work-shared/qemux86-64/kernel-source=/usr/src/kernel LD=x86_64-poky-linux-ld.bfd 3687235 ? SN 0:00 make -C /home/pokybuild/yocto-worker/oe-selftest-centos/build/build-st-2484929/tmp/work/qemux86_64-poky-linux/linux-yocto/5.14.15+gitAUTOINC+edb4dc09c5_f04b30fc15-r0/linux-qemux86_64-standard-build -f /home/pokybuild/yocto-worker/oe-selftest-centos/build/build-st-2484929/tmp/work-shared/qemux86-64/kernel-source/Makefile bzImage 3700121 ? SN 0:00 make -f /home/pokybuild/yocto-worker/oe-selftest-centos/build/build-st-2484929/tmp/work-shared/qemux86-64/kernel-source/scripts/Makefile.build obj=lib single-build= need-builtin=1 need-modorder=1 3703528 ? SN 0:00 make -f /home/pokybuild/yocto-worker/oe-selftest-centos/build/build-st-2484929/tmp/work-shared/qemux86-64/kernel-source/scripts/Makefile.build obj=lib/crypto need-builtin=1 need-modorder=1 3703566 ? ZN 0:00 [sh] <defunct> Once again we have a zombie subprocess with make waiting on nothing. This is on a linux-yocto build within an oe-selftest. It is once again on centos8 with make 4.2.1, similar to the perl build failure earlier in the week. I suspect if I send a SIGCHLD to make, it will continue as if nothing happened like the previous issue. Does anyone have any ideas what to do about this (or any better debugging we can do whilst it is hung)? Cheers, Richard
|
|
|
|
Re: Info for swat for hung build
Richard Purdie
On Sat, 2021-10-30 at 14:14 +0100, Richard Purdie via lists.yoctoproject.org
wrote: On Sat, 2021-10-30 at 14:04 +0100, Richard Purdie via lists.yoctoproject.orgSending an ECHLD (sig 17) to the process had it exit and start building again. Logs show it was stuck in wic.Wic.test_bootloader_config. That means it could be the above bug and something got messed up with the job counting? Or the ECHLD from the true process was somehow missed and there is a race in make somewhere but that seems less likely?... Cheers, Richard
|
|
|
|
Re: Info for swat for hung build
Richard Purdie
On Sat, 2021-10-30 at 14:04 +0100, Richard Purdie via lists.yoctoproject.org
wrote: I noticed:This was on centos8-ty-2. gdb -p 909354 GNU gdb (GDB) Red Hat Enterprise Linux 8.2-16.el8 Copyright (C) 2018 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-redhat-linux-gnu". Type "show configuration" for configuration details. For bug reporting instructions, please see: <http://www.gnu.org/software/gdb/bugs/>. Find the GDB manual and other documentation resources online at: <http://www.gnu.org/software/gdb/documentation/>. For help, type "help". Type "apropos word" to search for commands related to "word". Attaching to process 909354 Reading symbols from /usr/bin/make...Reading symbols from .gnu_debugdata for /usr/bin/make...(no debugging symbols found)...done. (no debugging symbols found)...done. Reading symbols from /lib64/libdl.so.2...(no debugging symbols found)...done. Reading symbols from /lib64/libc.so.6...(no debugging symbols found)...done. Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done. 0x00007f79a1bc146d in pselect () from /lib64/libc.so.6 Missing separate debuginfos, use: yum debuginfo-install make-4.2.1-11.el8.x86_64 (gdb) bt #0 0x00007f796a1bc146(gdb) btd in pselect () from /lib64/libc.so.6 #1 0x000055851c9f700d in jobserver_acquire () #2 0x000055851c9f3935 in new_job () #3 0x000055851c9ffc07 in update_file () #4 0x000055851ca00055 in check_dep () #5 0x000055851c9fefe6 in update_file () #6 0x000055851ca00055 in check_dep () #7 0x000055851c9fefe6 in update_file () #8 0x000055851ca00055 in check_dep () #9 0x000055851c9fefe6 in update_file () #10 0x000055851ca00055 in check_dep () #11 0x000055851c9fefe6 in update_file () #12 0x000055851ca00055 in check_dep () #13 0x000055851c9fefe6 in update_file () #14 0x000055851ca004cf in update_goal_chain () #15 0x000055851c9e3f56 in main () (gdb) Centos 8 has make 4.2.1 which made me wonder about: https://git.savannah.gnu.org/cgit/make.git/commit/src/posixos.c?id=d79fe162c009788888faaf0317253b6f0cac7092 ? Cheers, Richard
|
|
|
|
Info for swat for hung build
Richard Purdie
I noticed:
https://autobuilder.yoctoproject.org/typhoon/#/builders/79/builds/2779 has hung. Since this will be gone by the time anyone looks, inspection on the system suggests it is hung in perl do_compile building a core-image-minimal: $ pstree -p 3934003 run.do_compile.(3934003)───make(3934023)───make(4035804)───make(909354)───true(909946) 909354 ? SN 0:00 make -C ext/XS-Typemap/ all PERL_CORE=1 LIBPERL=libperl.so.5.34.0 LINKTYPE=dynamic 909946 ? ZN 0:00 [true] <defunct> 3933716 ? SNs 0:02 python3 /home/pokybuild/yocto-worker/oe-selftest-centos/build/bitbake/bin/bitbake-worker decafbad 3934003 ? SN 0:00 /bin/sh /home/pokybuild/yocto-worker/oe-selftest-centos/build/build-st-3687560/tmp/work/core2-64-poky-linux/perl/5.34.0-r0/temp/run.do_compile.3933716 3934023 ? SN 0:00 make -j 16 -l 52 4035804 ? SN 0:00 make perl nonxs_ext utilities extensions pods so the true exit code was never looked at by make? I think it was just starting to run wic tests (wic.Wic, not wic.Wic2). Cheers, Richard
|
|
|
|
Re: Swatbot
Larson, Chris
Yes, please, I’d appreciate that.
--
Christopher “kergoth” Larson
chris_larson@..., chris.larson@..., kergoth@...
Founder - BitBake, OpenEmbedded, OpenZaurus
Senior Software Engineer, Mentor Graphics, a Siemens Business
On Oct 22, 2021, 10:53 AM -0700, Alexandre Belloni <alexandre.belloni@...>, wrote:
Hello,
|
|
|
|
Re: Swatbot
Alexandre Belloni
Hello,
On 22/10/2021 17:37:34+0000, Larson, Chris wrote: I’m starting on triage for swat now, it’ll actually be my first timeI sent your account password on Tue, 8 Jun 2021 21:56:26 +0200, I can resend if you need. -- Alexandre Belloni, co-owner and COO, Bootlin Embedded Linux and Kernel engineering https://bootlin.com
|
|
|
|
Swatbot
Larson, Chris
I’m starting on triage for swat now, it’ll actually be my first time doing it, so bear with me. I’m awaiting a response to a password reset attempt on swatbot at the moment (though honestly not sure I ever had an account.. do I need to register? 😊.
-Chris
|
|
|
|
Re: Intermittent network issue on centos7-ty-4
Richard Purdie
On Tue, 2021-10-19 at 13:24 +0100, Richard Purdie via lists.yoctoproject.org
wrote: On Tue, 2021-10-19 at 12:43 +0100, Richard Purdie via lists.yoctoproject.orgNot sure if this means anything or not (run on centos7-ty-4 for around 10 mins): -bash-4.2$ while true; do ping typhoon.yocto.io -c 1 > /dev/null || date; sleep 0.1; done Tue 19 Oct 12:53:35 GMT 2021 Tue 19 Oct 12:54:32 GMT 2021 Tue 19 Oct 12:55:39 GMT 2021 Tue 19 Oct 12:57:01 GMT 2021 Tue 19 Oct 12:59:54 GMT 2021 Tue 19 Oct 13:00:46 GMT 2021 Tue 19 Oct 13:02:54 GMT 2021 Tue 19 Oct 13:03:11 GMT 2021 Cheers, Richard
|
|
|
|
Re: Intermittent network issue on centos7-ty-4
Richard Purdie
On Tue, 2021-10-19 at 12:43 +0100, Richard Purdie via lists.yoctoproject.org
wrote: Yesterday there was some talk in irc about a breif networking failure onChecking this again, I think the timestamps are not an hour different and I was misreading some tests I made. The do_install task finished at 17:07 and the do_package finished by 17:11 so we have a 4 minute window which does not match the last dhcp refresh before the error at 16:55. It was a nice idea :/. Cheers, Richard
|
|
|
|
Intermittent network issue on centos7-ty-4
Richard Purdie
Yesterday there was some talk in irc about a breif networking failure on
centos7-ty-4. The build was: https://autobuilder.yoctoproject.org/typhoon/#/builders/104/builds/3083 Showing these warnings: stdio: WARNING: gtk+3-3.24.30-r0 do_package: Error contacting Hash Equivalence Server typhoon.yocto.io:8686: [Errno -2] Name or service not known stdio: WARNING: gtk+3-3.24.30-r0 do_packagedata: Error contacting Hash Equivalence Server typhoon.yocto.io:8686: [Errno -2] Name or service not known The worry being why was the DNS suddenly not working. Looking at the log: /home/pokybuild/yocto-worker/genericx86-64-alt/build/build-renamed/tmp/work/core2-64-poky-linux/gtk+3/3.24.30-r0/temp/log.do_package.41319 The warning is near the end of that and the log timestamp is Oct 18 17:11 (i.e. when it was last written to). We didn't spot anything yesterday but I realised the systemd journalctl timestamps are an hour different that the timestamp I displayed from the filesystem thanks for daylight savings. Looking at journalctl an hour different, I see: Oct 18 16:00:03 centos7-ty-4.yocto.io systemd[1]: Removed slice User Slice of root. Oct 18 16:01:03 centos7-ty-4.yocto.io systemd[1]: Created slice User Slice of root. Oct 18 16:01:03 centos7-ty-4.yocto.io systemd[1]: Started Session 483 of user root. Oct 18 16:01:03 centos7-ty-4.yocto.io CROND[25699]: (root) CMD (run-parts /etc/cron.hourly) Oct 18 16:01:04 centos7-ty-4.yocto.io run-parts(/etc/cron.hourly)[25822]: starting 0anacron Oct 18 16:01:04 centos7-ty-4.yocto.io run-parts(/etc/cron.hourly)[25923]: finished 0anacron Oct 18 16:01:04 centos7-ty-4.yocto.io systemd[1]: Removed slice User Slice of root. Oct 18 16:05:17 centos7-ty-4.yocto.io dhclient[3905]: DHCPREQUEST on enp4s0f0 to 172.29.10.1 port 67 (xid=0x1579bf79) Oct 18 16:05:17 centos7-ty-4.yocto.io dhclient[3905]: DHCPACK from 172.29.10.1 (xid=0x1579bf79) Oct 18 16:05:18 centos7-ty-4.yocto.io NetworkManager[1436]: <info> [1634573118.2367] dhcp4 (enp4s0f0): address 172.29.10.2 Oct 18 16:05:18 centos7-ty-4.yocto.io NetworkManager[1436]: <info> [1634573118.2373] dhcp4 (enp4s0f0): plen 24 (255.255.255.0) Oct 18 16:05:18 centos7-ty-4.yocto.io NetworkManager[1436]: <info> [1634573118.2373] dhcp4 (enp4s0f0): gateway 172.29.10.1 Oct 18 16:05:18 centos7-ty-4.yocto.io NetworkManager[1436]: <info> [1634573118.2373] dhcp4 (enp4s0f0): lease time 7200 Oct 18 16:05:18 centos7-ty-4.yocto.io NetworkManager[1436]: <info> [1634573118.2373] dhcp4 (enp4s0f0): hostname 'yct-ab02' Oct 18 16:05:18 centos7-ty-4.yocto.io NetworkManager[1436]: <info> [1634573118.2373] dhcp4 (enp4s0f0): nameserver '172.29.10.1' Oct 18 16:05:18 centos7-ty-4.yocto.io NetworkManager[1436]: <info> [1634573118.2373] dhcp4 (enp4s0f0): domain name 'yocto.io' Oct 18 16:05:18 centos7-ty-4.yocto.io NetworkManager[1436]: <info> [1634573118.2373] dhcp (enp4s0f0): domain search 'pdx.yoctoproject.org.' Oct 18 16:05:18 centos7-ty-4.yocto.io NetworkManager[1436]: <info> [1634573118.2374] dhcp (enp4s0f0): domain search 'yocto.io.' Oct 18 16:05:18 centos7-ty-4.yocto.io NetworkManager[1436]: <info> [1634573118.2374] dhcp4 (enp4s0f0): state changed bound -> bound Oct 18 16:05:18 centos7-ty-4.yocto.io dhclient[3905]: bound to 172.29.10.2 -- renewal in 3024 seconds. Oct 18 16:05:18 centos7-ty-4.yocto.io dbus[1414]: [system] Activating via systemd: service name='org.freedesktop.nm_dispatcher' unit='dbus-org.freedesktop.nm-d Oct 18 16:05:18 centos7-ty-4.yocto.io systemd[1]: Starting Network Manager Script Dispatcher Service... Oct 18 16:05:19 centos7-ty-4.yocto.io dbus[1414]: [system] Successfully activated service 'org.freedesktop.nm_dispatcher' Oct 18 16:05:19 centos7-ty-4.yocto.io systemd[1]: Started Network Manager Script Dispatcher Service. Oct 18 16:05:19 centos7-ty-4.yocto.io nm-dispatcher[24137]: req:1 'dhcp4-change' [enp4s0f0]: new request (4 scripts) Oct 18 16:05:19 centos7-ty-4.yocto.io nm-dispatcher[24137]: req:1 'dhcp4-change' [enp4s0f0]: start running ordered scripts... Oct 18 16:10:05 centos7-ty-4.yocto.io systemd[1]: Created slice User Slice of root. Oct 18 16:10:05 centos7-ty-4.yocto.io systemd[1]: Started Session 484 of user root. Oct 18 16:10:05 centos7-ty-4.yocto.io CROND[26785]: (root) CMD (/usr/lib64/sa/sa1 1 1) Oct 18 16:10:08 centos7-ty-4.yocto.io systemd[1]: Removed slice User Slice of root. i.e. it renewed the DHCP lease about that time and ran network manager scripts. This looks a bit like a smoking gun to me. Not sure what to do about it but I think we may know where the networking glitches come from. Cheers, Richard
|
|
|
|
Re: SWAT Rotation schedule
Paul Eggleton
Yep, haven't got very far yet but I'm on it 😊
toggle quoted messageShow quoted text
Cheers Paul
-----Original Message-----
From: swat@... <swat@...> On Behalf Of Alexandre Belloni Sent: Tuesday, 19 October 2021 9:06 am To: Paul Eggleton <Paul.Eggleton@...> Subject: SWAT Rotation schedule [You don't often get email from alexandre.belloni@.... Learn why this is important at http://aka.ms/LearnAboutSenderIdentification.] Hello Paul, A quick reminder that you are on SWAT duty but I see you started triaging yesterday ;) Valerii, I've added you in the schedule following a discussion with Oleksiy: ┌──────┬────────────┐ │ Week │ Start │ ┌───────────────────────────────┼──────┼────────────┤ │ Alejandro Hernandez Samaniego │ 37 │ 17/09/2021 │ │ Oleksiy Obitotskyy │ 38 │ 24/09/2021 │ │ Naveen Saini │ 39 │ 01/10/2021 │ │ Thomas Perrot │ 40 │ 08/10/2021 │ │ Paul Eggleton │ 41 │ 15/10/2021 │ │ Christopher Larson │ 42 │ 22/10/2021 │ │ Jon Mason │ 43 │ 29/10/2021 │ │ Lee Chee Yang │ 44 │ 05/11/2021 │ │ Minjae Kim │ 45 │ 12/11/2021 │ │ Jaga │ 46 │ 19/11/2021 │ │ Leo Sandoval │ 47 │ 26/11/2021 │ │ Ross Burton │ 48 │ 03/12/2021 │ │ Valerii Chernous │ 49 │ 10/12/2021 │ │ Anibal Limon │ 50 │ 17/12/2021 │ │ Saul Wold │ 51 │ 24/12/2021 │ └───────────────────────────────┴──────┴────────────┘ Please check the table and let me know if you are not available for the selected week. SWAT duty will be from Friday to Thursday and the goal is to triage all the failures on swatbot before the weekly triage call happening at 2:30pm UTC. Thanks! -- Alexandre Belloni, Bootlin Embedded Linux and Kernel engineering https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fbootlin.com%2F&;data=04%7C01%7Cpaul.eggleton%40microsoft.com%7C6cfd8cfecda5417e04d608d99272aa38%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637701843481103286%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=NYz%2BobYcufHZWJ7LL6483IJhPW7Ck9%2FT0Ysj83HoGnE%3D&reserved=0
|
|
|