|
SWAT Rotation
Alexandre Belloni
Jon,
Once again, you are the next one on the list (https://wiki.yoctoproject.org/wiki/Yocto_Build_Failure_Swat_Team#Members) and SWAT duty will rotate from me to you at EOD 01/29/2021. Please reply to let me know whether you will be able to work on this task. Also, please update me if during the week, you can't spend enough time on the topic. I know Ross will walk you through the process but don't hesitate to contact me if you have any question. Thanks! -- Alexandre Belloni, Bootlin Embedded Linux and Kernel engineering https://bootlin.com
|
|
|
|
Re: SWAT Rotation
Alexandre Belloni
On 22/01/2021 06:57:13-0500, Jon Mason wrote:
Hello Alexandre,Indeed, I will take this rotation and you'll get the next one. -- Alexandre Belloni, Bootlin Embedded Linux and Kernel engineering https://bootlin.com
|
|
|
|
Re: SWAT Rotation
Jon Mason <jdmason@...>
Hello Alexandre,
I believe it was already discussed with you, but I'll take the next rotation. Thanks, Jon On Thu, Jan 21, 2021 at 5:38 PM Alexandre Belloni <alexandre.belloni@...> wrote:
|
|
|
|
Introducing "SwatBot"
Richard Purdie
Hi All,
I have some good news, the SWAT django app I've talked about is nearing completion and will soon be ready to use. We now have a live instance and are checking how it works with live data from the autobuilder. https://swatbot.yoctoproject.org/ It is a bit ugly and rough around the edges but it should be able to display a list of the things SWAT need to attend too and allow them to be marked as resolved, tracking the resolutions. This should mean that the process is easier for the person doing SWAT and for us to make sure nothing gets "lost" during handovers etc. Source code is available: http://git.yoctoproject.org/cgit.cgi/swatbot and the notifier plugin that connects to buildbot: http://git.yoctoproject.org/cgit.cgi/yocto-autobuilder2 I'm expecting that where it isn't doing what we need, we should be able to get it into shape in relatively short order. Alexandre and Bootlin are now coming up to speed and will be taking over leading the SWAT process and as part of that, helping get the app working for everyone. I think Alexandre will issue accounts to people on SWAT as we rotate through the list. Over the last week I've been worried some issues haven't been logged/handled so I made a note of the ones we need to check have been handled: meta-arm edk2 intermittent failure needs bug: https://autobuilder.yoctoproject.org/typhoon/#/builders/113/builds/629 update ltp bug: https://autobuilder.yoctoproject.org/typhoon/#/builders/96/builds/1403 need a bug for slow boot failure intermittent issue?: (was it slow, have we more logs, may be too late now?) https://autobuilder.yoctoproject.org/typhoon/#/builders/109/builds/1873 intermittent ptest warnings, need to updates to bugs so we can track frequency: https://autobuilder.yoctoproject.org/typhoon/#/builders/82/builds/1435 https://autobuilder.yoctoproject.org/typhoon/#/builders/81/builds/1716 https://autobuilder.yoctoproject.org/typhoon/#/builders/82/builds/1443 https://autobuilder.yoctoproject.org/typhoon/#/builders/81/builds/1723 https://autobuilder.yoctoproject.org/typhoon/#/builders/82/builds/1439 https://autobuilder.yoctoproject.org/typhoon/#/builders/81/builds/1719 bitbake timeout bug needs updating adding this? https://autobuilder.yoctoproject.org/typhoon/#/builders/79/builds/1739 should be a multiprocess bug somewhere, add or reopen? https://autobuilder.yoctoproject.org/typhoon/#/builders/86/builds/1729 qemuarm64 shutdown bug https://autobuilder.yoctoproject.org/typhoon/#/builders/42/builds/2943 ping test fail: https://autobuilder.yoctoproject.org/typhoon/#/builders/74/builds/2945 mips backtrace: https://autobuilder.yoctoproject.org/typhoon/#/builders/74/builds/2942 There were a ton more failures but I think I've responded/handled the others. I'll admit to focusing on getting the app done since I think it should improve this process so much and help everyone. Cheers, Richard
|
|
|
|
SWAT Rotation
Alexandre Belloni
Jon,
You are the next one on the list (https://wiki.yoctoproject.org/wiki/Yocto_Build_Failure_Swat_Team#Members) and SWAT will rotate from Lee Chee to you at EOD 01/22/2021. Please reply to let me know whether you will be able to work on this task. Also, please update me if during the week, you can't spedn enough time on the topic. Thanks! -- Alexandre Belloni, Bootlin Embedded Linux and Kernel engineering https://bootlin.com
|
|
|
|
Build and Integration help from Bootlin
Richard Purdie
Hi All,
I'm pleased to be able to announce that Alexandre Belloni and some of his colleagues from Bootlin are going to be helping us with the process for testing and integrating patches within the project. There are multiple reasons for this, not least that it helps not to have a single point of failure and that I could do with the help and a break! :) As such, they are are going to be leading the SWAT process and you'll therefore be seeing them giving initial responses on patches and helping with the autobuilder failures. I'd like to welcome them on board and hope everyone will support them as we figure out how the finer details of this will work. This is being made possible by support from the Yocto Project members which is very much appreciated, thanks! I will be continuing to do the final patch review and merge and am not going anywhere, this is some very much welcomed support to try and ensure we continue to scale and maintain our quality and testing. Cheers, Richard
|
|
|
|
Re: SWAT Rotation
Lee Chee Yang
Hi Ross
toggle quoted messageShow quoted text
sure, will reach out when I need help. In Malaysia, the Covid cases hike drastically this year and now we have to lock down again.
-----Original Message-----
|
|
|
|
Re: SWAT Rotation
Ross Burton <ross@...>
Hi Chee Yang!
toggle quoted messageShow quoted text
Feel free to email or grab me on IRC (rburton in #yocto) if you have any questions. Naveen did a great job Swatting so hopefully he'll be able to give you more direct assistance. Then again maybe you're all working remotely anyway, I've no idea how Covid is going over there. Here in the UK we're struggling to say the least, and the office is expected to be shut for six months at least still. Cheers, Ross
On Fri, 15 Jan 2021 at 03:17, Lee Chee Yang <chee.yang.lee@...> wrote:
|
|
|
|
Re: SWAT Rotation
Lee Chee Yang
Noted
From: sjolley.yp.pm@... <sjolley.yp.pm@...>
Sent: Friday, 15 January, 2021 11:08 AM To: Lee, Chee Yang <chee.yang.lee@...> Cc: swat@...; 'Christopher Larson' <kergoth@...> Subject: SWAT Rotation
Lee Chee,
SWAT will rotate from Christopher to you at EOD 01/15/2021. Please RSVP ASAP that you know you are on SWAT!
See: https://wiki.yoctoproject.org/wiki/Yocto_Build_Failure_Swat_Team#Members
Thanks,
Stephen K. Jolley Yocto Project Program Manager ( Cell: (208) 244-4460 * Email: sjolley.yp.pm@...
|
|
|
|
SWAT Rotation
Stephen Jolley <sjolley.yp.pm@...>
Lee Chee,
SWAT will rotate from Christopher to you at EOD 01/15/2021. Please RSVP ASAP that you know you are on SWAT!
See: https://wiki.yoctoproject.org/wiki/Yocto_Build_Failure_Swat_Team#Members
Thanks,
Stephen K. Jolley Yocto Project Program Manager ( Cell: (208) 244-4460 * Email: sjolley.yp.pm@...
|
|
|
|
Re: Current autobuilder status
Ross Burton <ross@...>
https://bugzilla.yoctoproject.org/show_bug.cgi?id=14183https://autobuilder.yoctoproject.org/typhoon/#/builders/76/builds/2876/steps/14/logs/stdioStill needs filing. https://bugzilla.yoctoproject.org/show_bug.cgi?id=14180https://autobuilder.yoctoproject.org/typhoon/#/builders/87/builds/1720/steps/15/logs/stdioStill needs filing. These have all been filed already.arm ptest warnings which will need to be added to bugs or bugs opened:Still needs filing. All filed already.x86 ptest warnings which will need to be added to bugs or bugs opened:Still needs filing. vulkan-samples-staticdev reproducibility issue:https://bugzilla.yoctoproject.org/show_bug.cgi?id=14182 Intermittent ssh failure:Nothing useful in the log beyond "cannot connect to host". qemuarm rcu backtrace and QA failure:https://bugzilla.yoctoproject.org/show_bug.cgi?id=14184 bitbake pickling failure:https://bugzilla.yoctoproject.org/show_bug.cgi?id=14181 Ross
|
|
|
|
Re: Current autobuilder status
Richard Purdie
To try and ensure we don't lose track of issues:
On Thu, 2021-01-07 at 11:07 +0000, Richard Purdie via lists.yoctoproject.org wrote: I'm not sure what is happening with SWAT but we're in danger of losingFix merged, bug closed so resolved. https://autobuilder.yoctoproject.org/typhoon/#/builders/76/builds/2876/steps/14/logs/stdioStill needs filing. https://autobuilder.yoctoproject.org/typhoon/#/builders/87/builds/1720/steps/15/logs/stdioStill needs filing. https://autobuilder.yoctoproject.org/typhoon/#/builders/87/builds/1716/steps/15/logs/stdioIssue resolved, patch sent upstream. https://autobuilder.yoctoproject.org/typhoon/#/builders/80/builds/1688/steps/14/logs/stdioIssue resolved, patch queued in master-next with the bash upgrade. arm ptest warnings which will need to be added to bugs or bugs opened:Still needs filing. x86 ptest warnings which will need to be added to bugs or bugs opened:Still needs filing. perl install failure in man pages, think open bug needs updating?Added to https://bugzilla.yoctoproject.org/show_bug.cgi?id=14096 so done. The issue with new qemu needing python 3.6+ is fixed by changes inqemu upgrade merged, issues resolved. Still need to upstream the mingw issue. New issues: vulkan-samples-staticdev reproducibility issue: https://autobuilder.yocto.io/pub/repro-fail/oe-reproducible-20210109-uud35nir/packages/reproducibleA/tmp/deploy/deb/core2-64/ https://autobuilder.yoctoproject.org/typhoon/#/builders/79/builds/1712 Intermittent ssh failure: https://autobuilder.yoctoproject.org/typhoon/#/builders/72/builds/2936 qemuarm rcu backtrace and QA failure: https://autobuilder.yoctoproject.org/typhoon/#/builders/53/builds/2921 bitbake pickling failure: https://autobuilder.yoctoproject.org/typhoon/#/builders/101/builds/1800 [have seen this very occasionally before, evidently previous fixes haven't solved everything, bugs were closed for it I think :(] Doing my best to try and keep on top of some of the issues! Cheers, Richard
|
|
|
|
SWAT Rotation
Stephen Jolley <sjolley.yp.pm@...>
Christopher,
SWAT will rotate from Armin to you at EOD 01/08/2021. Please RSVP ASAP that you know you are on SWAT!
See: https://wiki.yoctoproject.org/wiki/Yocto_Build_Failure_Swat_Team#Members
Thanks,
Stephen K. Jolley Yocto Project Program Manager ( Cell: (208) 244-4460 * Email: sjolley.yp.pm@...
|
|
|
|
Re: Current autobuilder status
On 1/7/21 3:07 AM, Richard Purdie wrote:
I'm not sure what is happening with SWAT but we're in danger of losingMy work schedule is such that I have no time to put into this activity. -armin
|
|
|
|
arm worker unwell
Richard Purdie
I've also noticed one of the arm workers appears unwell:
https://autobuilder.yoctoproject.org/typhoon/#/workers/39 Michael: Can you take a look when you're up tomorrow please? Thanks, Richard
|
|
|
|
Current autobuilder status
Richard Purdie
I'm not sure what is happening with SWAT but we're in danger of losing
track of issues which is sad after Naveen did a great job and had everything up to date. I'm aware of the following: https://autobuilder.yoctoproject.org/typhoon/#/builders/56/builds/1305 https://autobuilder.yoctoproject.org/typhoon/#/builders/56/builds/1306 both failed in initramfs-boot, already open bug, fix in master-next. https://autobuilder.yoctoproject.org/typhoon/#/builders/76/builds/2876/steps/14/logs/stdio https://autobuilder.yoctoproject.org/typhoon/#/builders/76/builds/2890/steps/14/logs/stdio failure running apt-get update. Second time we've seen it, was on debian10-ty-1 here, denian10-ty-3 last time. pseudo.log says: path mismatch [1 link]: ino 143525936 db '/home/pokybuild/yocto-worker/pkgman-non-rpm/build/build/tmp/work/qemux86-poky-linux/core-image-sato/1.0-r0/apt/lists/partial/_home_pokybuild_yocto-worker_pkgman-non-rpm_build_build_tmp_work_qemux86-poky-linux_core-image-sato_1.0-r0_oe-rootfs-repo_qemux86_._Release.CBFzPd' req '/home/pokybuild/yocto-worker/pkgman-non-rpm/build/build/tmp/work/qemux86-poky-linux/core-image-sato/1.0-r0/apt/lists/partial/_home_pokybuild_yocto-worker_pkgman-non-rpm_build_build_tmp_work_qemux86-poky-linux_core-image-sato_1.0-r0_oe-rootfs-repo_qemux86_._Release'. similar to the last time. path mismatch [1 link]: ino 27666766 db '/home/pokybuild/yocto-worker/pkgman-non-rpm/build/build/tmp/work/qemux86-poky-linux/core-image-sato/1.0-r0/apt/lists/partial/_home_pokybuild_yocto-worker_pkgman-non-rpm_build_build_tmp_work_qemux86-poky-linux_core-image-sato_1.0-r0_oe-rootfs-repo_all_._Release.RpRzHw' req '/home/pokybuild/yocto-worker/pkgman-non-rpm/build/build/tmp/work/qemux86-poky-linux/core-image-sato/1.0-r0/apt/lists/partial/_home_pokybuild_yocto-worker_pkgman-non-rpm_build_build_tmp_work_qemux86-poky-linux_core-image-sato_1.0-r0_oe-rootfs-repo_all_._Release' https://autobuilder.yoctoproject.org/typhoon/#/builders/87/builds/1720/steps/15/logs/stdio new issue where dnf is complaining checksums are incorrect https://autobuilder.yoctoproject.org/typhoon/#/builders/87/builds/1716/steps/15/logs/stdio ppp intermittent reproducibility issue https://autobuilder.yoctoproject.org/typhoon/#/builders/80/builds/1688/steps/14/logs/stdio bash reproducibility issue arm ptest warnings which will need to be added to bugs or bugs opened: https://autobuilder.yoctoproject.org/typhoon/#/builders/82/builds/1403 https://autobuilder.yoctoproject.org/typhoon/#/builders/82/builds/1401 https://autobuilder.yoctoproject.org/typhoon/#/builders/82/builds/1400 https://autobuilder.yoctoproject.org/typhoon/#/builders/82/builds/1399 x86 ptest warnings which will need to be added to bugs or bugs opened: https://autobuilder.yoctoproject.org/typhoon/#/builders/81/builds/1680 https://autobuilder.yoctoproject.org/typhoon/#/builders/81/builds/1676 perl install failure in man pages, think open bug needs updating? https://autobuilder.yoctoproject.org/typhoon/#/builders/47/builds/2909 The issue with new qemu needing python 3.6+ is fixed by changes in autobuilder-helper for debian9. I also resolved the gio postinst execution issue with new qemu. The mingw failure with new qemu is known Joshua is looking into that. Cheers, Richard
|
|
|
|
Re: SWAT Rotation
Naveen Saini
New bug:
toggle quoted messageShow quoted text
Tcl ptest failure: https://bugzilla.yoctoproject.org/show_bug.cgi?id=14177 Updated existing bug: Ltp test failure: https://bugzilla.yoctoproject.org/show_bug.cgi?id=13802 Strace qemux86-64 ptest failure (same as arm): https://bugzilla.yoctoproject.org/show_bug.cgi?id=14165 Regards, Naveen
-----Original Message-----
|
|
|
|
Re: SWAT Rotation
Richard Purdie
Hi Naveen,
Thanks for looking at these! On Tue, 2021-01-05 at 02:38 +0000, Saini, Naveen Kumar wrote: Hi Richard,Bug for this one sounds good, thanks! (2) test_ltp_groups test failure (Should I raise bug for this ?)I think there is an open ltp ssh connection issue so we probably need to mention this failure on that bug? (3) Reproducibility testcase failing due to missing pulseaudioThese ones have patches in master which fix this so this one needs no further action. (4) ptest failure on qemux86-64 ( Seems intermittent, alreadyIs this the same strace test failing as the arm failure? If not it should be a separate bug. Cheers, Richard
|
|
|
|
Re: SWAT Rotation
Naveen Saini
Hi Richard,
toggle quoted messageShow quoted text
I have located few more old build failures (master-next): (1) ptest failure on qemux86-64-ptest (Looks intermittent, I will raise Bug) AssertionError: Failed ptests: {'tcl': ['socket.test']} https://autobuilder.yoctoproject.org/typhoon/#/builders/81/builds/1654 (2) test_ltp_groups test failure (Should I raise bug for this ?) Reason: Looks like it's due to ssh connection failure RESULTS - ltp.LtpTest.test_ltp_groups: ERROR (10948.27s) SUMMARY: core-image-sato () - Ran 6 tests in 16385.717s core-image-sato - FAIL - Required tests failed (successes=4, skipped=1, failures=0, errors=1) https://autobuilder.yoctoproject.org/typhoon/#/builders/95/builds/1476 (3) Reproducibility testcase failing due to missing pulseaudio dependency (debian) (need to analysis, which patch causing this failure) pulseaudio was skipped: pulseaudio - pulseaudio-server: system username pulse does not have a static ID defined. Add pulse to one of these files: /home/pokybuild/yocto-worker/oe-selftest-debian/build/build-st-17416/meta-selftest/files/static-passwd ERROR: Required build target 'meta-world-pkgdata' has no buildable providers. Missing or unbuildable dependency chain was: ['meta-world-pkgdata', 'gstreamer1.0-plugins-good', 'mpg123', 'pulseaudio'] https://autobuilder.yoctoproject.org/typhoon/#/builders/80/builds/1660 Centos: Missing or unbuildable dependency chain was: ['meta-world-pkgdata', 'alsa-plugins', 'pulseaudio'] https://autobuilder.yoctoproject.org/typhoon/#/builders/79/builds/1678 Fedora: https://autobuilder.yoctoproject.org/typhoon/#/builders/86/builds/1665 Ubuntu: https://autobuilder.yoctoproject.org/typhoon/#/builders/87/builds/1690 (4) ptest failure on qemux86-64 ( Seems intermittent, already available for arm machine) AssertionError: Failed ptests: {'strace': ['qual_fault-syscall.test', 'qual_fault.test']} https://autobuilder.yoctoproject.org/typhoon/#/builders/83/builds/1698 Regards, Naveen
-----Original Message-----
|
|
|
|
Re: SWAT Rotation
Richard Purdie
On Thu, 2020-12-31 at 13:24 -0800, akuster wrote:
Thanks for the reminder.That sounds like me! :) Thanks Naveen, you've done well at tracking the issues that came up and its much appreciated. There have been quite a few more than expected, despite a fairly low number of builds! Happy New Year everyone! Cheers, Richard
|
|
|