Date   

Introducing "SwatBot"

Richard Purdie
 

Hi All,

I have some good news, the SWAT django app I've talked about is nearing
completion and will soon be ready to use. We now have a live instance
and are checking how it works with live data from the autobuilder.

https://swatbot.yoctoproject.org/

It is a bit ugly and rough around the edges but it should be able to
display a list of the things SWAT need to attend too and allow them to
be marked as resolved, tracking the resolutions. This should mean that
the process is easier for the person doing SWAT and for us to make sure
nothing gets "lost" during handovers etc.

Source code is available:
http://git.yoctoproject.org/cgit.cgi/swatbot
and the notifier plugin that connects to buildbot:
http://git.yoctoproject.org/cgit.cgi/yocto-autobuilder2

I'm expecting that where it isn't doing what we need, we should be able
to get it into shape in relatively short order.

Alexandre and Bootlin are now coming up to speed and will be taking
over leading the SWAT process and as part of that, helping get the app
working for everyone. I think Alexandre will issue accounts to people
on SWAT as we rotate through the list.

Over the last week I've been worried some issues haven't been
logged/handled so I made a note of the ones we need to check have been
handled:

meta-arm edk2 intermittent failure needs bug:
https://autobuilder.yoctoproject.org/typhoon/#/builders/113/builds/629

update ltp bug:
https://autobuilder.yoctoproject.org/typhoon/#/builders/96/builds/1403

need a bug for slow boot failure intermittent issue?:
(was it slow, have we more logs, may be too late now?)
https://autobuilder.yoctoproject.org/typhoon/#/builders/109/builds/1873


intermittent ptest warnings, need to updates to bugs so we can track
frequency:
https://autobuilder.yoctoproject.org/typhoon/#/builders/82/builds/1435
https://autobuilder.yoctoproject.org/typhoon/#/builders/81/builds/1716
https://autobuilder.yoctoproject.org/typhoon/#/builders/82/builds/1443
https://autobuilder.yoctoproject.org/typhoon/#/builders/81/builds/1723
https://autobuilder.yoctoproject.org/typhoon/#/builders/82/builds/1439
https://autobuilder.yoctoproject.org/typhoon/#/builders/81/builds/1719

bitbake timeout bug needs updating adding this?
https://autobuilder.yoctoproject.org/typhoon/#/builders/79/builds/1739

should be a multiprocess bug somewhere, add or reopen?
https://autobuilder.yoctoproject.org/typhoon/#/builders/86/builds/1729

qemuarm64 shutdown bug
https://autobuilder.yoctoproject.org/typhoon/#/builders/42/builds/2943

ping test fail:
https://autobuilder.yoctoproject.org/typhoon/#/builders/74/builds/2945

mips backtrace:
https://autobuilder.yoctoproject.org/typhoon/#/builders/74/builds/2942

There were a ton more failures but I think I've responded/handled the
others. I'll admit to focusing on getting the app done since I think it
should improve this process so much and help everyone.

Cheers,

Richard


SWAT Rotation

Alexandre Belloni
 

Jon,

You are the next one on the list
(https://wiki.yoctoproject.org/wiki/Yocto_Build_Failure_Swat_Team#Members)
and SWAT will rotate from Lee Chee to you at EOD 01/22/2021.

Please reply to let me know whether you will be able to work on this
task. Also, please update me if during the week, you can't spedn enough
time on the topic.

Thanks!

--
Alexandre Belloni, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com


Build and Integration help from Bootlin

Richard Purdie
 

Hi All,

I'm pleased to be able to announce that Alexandre Belloni and some of
his colleagues from Bootlin are going to be helping us with the process
for testing and integrating patches within the project.

There are multiple reasons for this, not least that it helps not to
have a single point of failure and that I could do with the help and a
break! :)

As such, they are are going to be leading the SWAT process and you'll
therefore be seeing them giving initial responses on patches and
helping with the autobuilder failures.

I'd like to welcome them on board and hope everyone will support them
as we figure out how the finer details of this will work.

This is being made possible by support from the Yocto Project members
which is very much appreciated, thanks!

I will be continuing to do the final patch review and merge and am not
going anywhere, this is some very much welcomed support to try and
ensure we continue to scale and maintain our quality and testing.

Cheers,

Richard


Re: SWAT Rotation

Lee Chee Yang
 

Hi Ross

sure, will reach out when I need help.

In Malaysia, the Covid cases hike drastically this year and now we have to lock down again.

-----Original Message-----
From: swat@... <swat@...> On Behalf Of
Ross Burton
Sent: Friday, 15 January, 2021 6:47 PM
To: swat@...
Subject: Re: [swat] SWAT Rotation

Hi Chee Yang!

Feel free to email or grab me on IRC (rburton in #yocto) if you have any questions.
Naveen did a great job Swatting so hopefully he'll be able to give you more direct
assistance. Then again maybe you're all working remotely anyway, I've no idea
how Covid is going over there.
Here in the UK we're struggling to say the least, and the office is expected to be
shut for six months at least still.

Cheers,
Ross

On Fri, 15 Jan 2021 at 03:17, Lee Chee Yang <chee.yang.lee@...> wrote:

Noted



From: sjolley.yp.pm@... <sjolley.yp.pm@...>
Sent: Friday, 15 January, 2021 11:08 AM
To: Lee, Chee Yang <chee.yang.lee@...>
Cc: swat@...; 'Christopher Larson'
<kergoth@...>
Subject: SWAT Rotation



Lee Chee,



SWAT will rotate from Christopher to you at EOD 01/15/2021. Please RSVP
ASAP that you know you are on SWAT!



See:
https://wiki.yoctoproject.org/wiki/Yocto_Build_Failure_Swat_Team#Membe
rs



Thanks,



Stephen K. Jolley

Yocto Project Program Manager

( Cell: (208) 244-4460

* Email: sjolley.yp.pm@...






Re: SWAT Rotation

Ross Burton <ross@...>
 

Hi Chee Yang!

Feel free to email or grab me on IRC (rburton in #yocto) if you have
any questions. Naveen did a great job Swatting so hopefully he'll be
able to give you more direct assistance. Then again maybe you're all
working remotely anyway, I've no idea how Covid is going over there.
Here in the UK we're struggling to say the least, and the office is
expected to be shut for six months at least still.

Cheers,
Ross

On Fri, 15 Jan 2021 at 03:17, Lee Chee Yang <chee.yang.lee@...> wrote:

Noted



From: sjolley.yp.pm@... <sjolley.yp.pm@...>
Sent: Friday, 15 January, 2021 11:08 AM
To: Lee, Chee Yang <chee.yang.lee@...>
Cc: swat@...; 'Christopher Larson' <kergoth@...>
Subject: SWAT Rotation



Lee Chee,



SWAT will rotate from Christopher to you at EOD 01/15/2021. Please RSVP ASAP that you know you are on SWAT!



See: https://wiki.yoctoproject.org/wiki/Yocto_Build_Failure_Swat_Team#Members



Thanks,



Stephen K. Jolley

Yocto Project Program Manager

( Cell: (208) 244-4460

* Email: sjolley.yp.pm@...




Re: SWAT Rotation

Lee Chee Yang
 

Noted

 

From: sjolley.yp.pm@... <sjolley.yp.pm@...>
Sent: Friday, 15 January, 2021 11:08 AM
To: Lee, Chee Yang <chee.yang.lee@...>
Cc: swat@...; 'Christopher Larson' <kergoth@...>
Subject: SWAT Rotation

 

Lee Chee,

 

SWAT will rotate from Christopher to you at EOD 01/15/2021.   Please RSVP ASAP that you know you are on SWAT!

 

See: https://wiki.yoctoproject.org/wiki/Yocto_Build_Failure_Swat_Team#Members

 

Thanks,

 

Stephen K. Jolley

Yocto Project Program Manager

(    Cell:                (208) 244-4460

* Email:              sjolley.yp.pm@...

 


SWAT Rotation

Stephen Jolley <sjolley.yp.pm@...>
 

Lee Chee,

 

SWAT will rotate from Christopher to you at EOD 01/15/2021.   Please RSVP ASAP that you know you are on SWAT!

 

See: https://wiki.yoctoproject.org/wiki/Yocto_Build_Failure_Swat_Team#Members

 

Thanks,

 

Stephen K. Jolley

Yocto Project Program Manager

(    Cell:                (208) 244-4460

* Email:              sjolley.yp.pm@...

 


Re: Current autobuilder status

Ross Burton <ross@...>
 

https://autobuilder.yoctoproject.org/typhoon/#/builders/76/builds/2876/steps/14/logs/stdio
https://autobuilder.yoctoproject.org/typhoon/#/builders/76/builds/2890/steps/14/logs/stdio
failure running apt-get update. Second time we've seen it, was on
debian10-ty-1 here, denian10-ty-3 last time. pseudo.log says:
path mismatch [1 link]: ino 143525936 db '/home/pokybuild/yocto-worker/pkgman-non-rpm/build/build/tmp/work/qemux86-poky-linux/core-image-sato/1.0-r0/apt/lists/partial/_home_pokybuild_yocto-worker_pkgman-non-rpm_build_build_tmp_work_qemux86-poky-linux_core-image-sato_1.0-r0_oe-rootfs-repo_qemux86_._Release.CBFzPd' req '/home/pokybuild/yocto-worker/pkgman-non-rpm/build/build/tmp/work/qemux86-poky-linux/core-image-sato/1.0-r0/apt/lists/partial/_home_pokybuild_yocto-worker_pkgman-non-rpm_build_build_tmp_work_qemux86-poky-linux_core-image-sato_1.0-r0_oe-rootfs-repo_qemux86_._Release'.
similar to the last time.
path mismatch [1 link]: ino 27666766 db '/home/pokybuild/yocto-worker/pkgman-non-rpm/build/build/tmp/work/qemux86-poky-linux/core-image-sato/1.0-r0/apt/lists/partial/_home_pokybuild_yocto-worker_pkgman-non-rpm_build_build_tmp_work_qemux86-poky-linux_core-image-sato_1.0-r0_oe-rootfs-repo_all_._Release.RpRzHw' req '/home/pokybuild/yocto-worker/pkgman-non-rpm/build/build/tmp/work/qemux86-poky-linux/core-image-sato/1.0-r0/apt/lists/partial/_home_pokybuild_yocto-worker_pkgman-non-rpm_build_build_tmp_work_qemux86-poky-linux_core-image-sato_1.0-r0_oe-rootfs-repo_all_._Release'
Still needs filing.
https://bugzilla.yoctoproject.org/show_bug.cgi?id=14183

https://autobuilder.yoctoproject.org/typhoon/#/builders/87/builds/1720/steps/15/logs/stdio
new issue where dnf is complaining checksums are incorrect
Still needs filing.
https://bugzilla.yoctoproject.org/show_bug.cgi?id=14180

arm ptest warnings which will need to be added to bugs or bugs opened:
https://autobuilder.yoctoproject.org/typhoon/#/builders/82/builds/1403
https://autobuilder.yoctoproject.org/typhoon/#/builders/82/builds/1401
https://autobuilder.yoctoproject.org/typhoon/#/builders/82/builds/1400
https://autobuilder.yoctoproject.org/typhoon/#/builders/82/builds/1399
Still needs filing.
These have all been filed already.

x86 ptest warnings which will need to be added to bugs or bugs opened:
https://autobuilder.yoctoproject.org/typhoon/#/builders/81/builds/1680
https://autobuilder.yoctoproject.org/typhoon/#/builders/81/builds/1676
Still needs filing.
All filed already.

vulkan-samples-staticdev reproducibility issue:
https://autobuilder.yocto.io/pub/repro-fail/oe-reproducible-20210109-uud35nir/packages/reproducibleA/tmp/deploy/deb/core2-64/
https://autobuilder.yoctoproject.org/typhoon/#/builders/79/builds/1712
https://bugzilla.yoctoproject.org/show_bug.cgi?id=14182

Intermittent ssh failure:
https://autobuilder.yoctoproject.org/typhoon/#/builders/72/builds/2936
Nothing useful in the log beyond "cannot connect to host".

qemuarm rcu backtrace and QA failure:
https://autobuilder.yoctoproject.org/typhoon/#/builders/53/builds/2921
https://bugzilla.yoctoproject.org/show_bug.cgi?id=14184

bitbake pickling failure:
https://autobuilder.yoctoproject.org/typhoon/#/builders/101/builds/1800
[have seen this very occasionally before, evidently previous fixes
haven't solved everything, bugs were closed for it I think :(]
https://bugzilla.yoctoproject.org/show_bug.cgi?id=14181

Ross


Re: Current autobuilder status

Richard Purdie
 

To try and ensure we don't lose track of issues:

On Thu, 2021-01-07 at 11:07 +0000, Richard Purdie via lists.yoctoproject.org wrote:
I'm not sure what is happening with SWAT but we're in danger of losing
track of issues which is sad after Naveen did a great job and had
everything up to date. I'm aware of the following:

https://autobuilder.yoctoproject.org/typhoon/#/builders/56/builds/1305
https://autobuilder.yoctoproject.org/typhoon/#/builders/56/builds/1306
both failed in initramfs-boot, already open bug, fix in master-next.
Fix merged, bug closed so resolved.

https://autobuilder.yoctoproject.org/typhoon/#/builders/76/builds/2876/steps/14/logs/stdio
https://autobuilder.yoctoproject.org/typhoon/#/builders/76/builds/2890/steps/14/logs/stdio
failure running apt-get update. Second time we've seen it, was on
debian10-ty-1 here, denian10-ty-3 last time. pseudo.log says:
path mismatch [1 link]: ino 143525936 db '/home/pokybuild/yocto-worker/pkgman-non-rpm/build/build/tmp/work/qemux86-poky-linux/core-image-sato/1.0-r0/apt/lists/partial/_home_pokybuild_yocto-worker_pkgman-non-rpm_build_build_tmp_work_qemux86-poky-linux_core-image-sato_1.0-r0_oe-rootfs-repo_qemux86_._Release.CBFzPd' req '/home/pokybuild/yocto-worker/pkgman-non-rpm/build/build/tmp/work/qemux86-poky-linux/core-image-sato/1.0-r0/apt/lists/partial/_home_pokybuild_yocto-worker_pkgman-non-rpm_build_build_tmp_work_qemux86-poky-linux_core-image-sato_1.0-r0_oe-rootfs-repo_qemux86_._Release'.
similar to the last time.
path mismatch [1 link]: ino 27666766 db '/home/pokybuild/yocto-worker/pkgman-non-rpm/build/build/tmp/work/qemux86-poky-linux/core-image-sato/1.0-r0/apt/lists/partial/_home_pokybuild_yocto-worker_pkgman-non-rpm_build_build_tmp_work_qemux86-poky-linux_core-image-sato_1.0-r0_oe-rootfs-repo_all_._Release.RpRzHw' req '/home/pokybuild/yocto-worker/pkgman-non-rpm/build/build/tmp/work/qemux86-poky-linux/core-image-sato/1.0-r0/apt/lists/partial/_home_pokybuild_yocto-worker_pkgman-non-rpm_build_build_tmp_work_qemux86-poky-linux_core-image-sato_1.0-r0_oe-rootfs-repo_all_._Release'
Still needs filing.

https://autobuilder.yoctoproject.org/typhoon/#/builders/87/builds/1720/steps/15/logs/stdio
new issue where dnf is complaining checksums are incorrect
Still needs filing.

https://autobuilder.yoctoproject.org/typhoon/#/builders/87/builds/1716/steps/15/logs/stdio
ppp intermittent reproducibility issue
Issue resolved, patch sent upstream.

https://autobuilder.yoctoproject.org/typhoon/#/builders/80/builds/1688/steps/14/logs/stdio
bash reproducibility issue
Issue resolved, patch queued in master-next with the bash upgrade.

arm ptest warnings which will need to be added to bugs or bugs opened:
https://autobuilder.yoctoproject.org/typhoon/#/builders/82/builds/1403
https://autobuilder.yoctoproject.org/typhoon/#/builders/82/builds/1401
https://autobuilder.yoctoproject.org/typhoon/#/builders/82/builds/1400
https://autobuilder.yoctoproject.org/typhoon/#/builders/82/builds/1399
Still needs filing.

x86 ptest warnings which will need to be added to bugs or bugs opened:
https://autobuilder.yoctoproject.org/typhoon/#/builders/81/builds/1680
https://autobuilder.yoctoproject.org/typhoon/#/builders/81/builds/1676
Still needs filing.

perl install failure in man pages, think open bug needs updating?
https://autobuilder.yoctoproject.org/typhoon/#/builders/47/builds/2909
Added to https://bugzilla.yoctoproject.org/show_bug.cgi?id=14096 so
done.

The issue with new qemu needing python 3.6+ is fixed by changes in
autobuilder-helper for debian9. I also resolved the gio postinst
execution issue with new qemu. The mingw failure with new qemu is known
Joshua is looking into that.
qemu upgrade merged, issues resolved. Still need to upstream the mingw
issue.

New issues:

vulkan-samples-staticdev reproducibility issue:
https://autobuilder.yocto.io/pub/repro-fail/oe-reproducible-20210109-uud35nir/packages/reproducibleA/tmp/deploy/deb/core2-64/
https://autobuilder.yoctoproject.org/typhoon/#/builders/79/builds/1712

Intermittent ssh failure:
https://autobuilder.yoctoproject.org/typhoon/#/builders/72/builds/2936

qemuarm rcu backtrace and QA failure:
https://autobuilder.yoctoproject.org/typhoon/#/builders/53/builds/2921

bitbake pickling failure:
https://autobuilder.yoctoproject.org/typhoon/#/builders/101/builds/1800
[have seen this very occasionally before, evidently previous fixes
haven't solved everything, bugs were closed for it I think :(]

Doing my best to try and keep on top of some of the issues!

Cheers,

Richard


SWAT Rotation

Stephen Jolley <sjolley.yp.pm@...>
 

Christopher,

 

SWAT will rotate from Armin to you at EOD 01/08/2021.   Please RSVP ASAP that you know you are on SWAT!

 

See: https://wiki.yoctoproject.org/wiki/Yocto_Build_Failure_Swat_Team#Members

 

Thanks,

 

Stephen K. Jolley

Yocto Project Program Manager

(    Cell:                (208) 244-4460

* Email:              sjolley.yp.pm@...

 


Re: Current autobuilder status

Armin Kuster
 

On 1/7/21 3:07 AM, Richard Purdie wrote:
I'm not sure what is happening with SWAT but we're in danger of losing
track of issues which is sad after Naveen did a great job and had
everything up to date. I'm aware of the following:
My work schedule is such that I have no time to put into this activity.

-armin

https://autobuilder.yoctoproject.org/typhoon/#/builders/56/builds/1305
https://autobuilder.yoctoproject.org/typhoon/#/builders/56/builds/1306
both failed in initramfs-boot, already open bug, fix in master-next.

https://autobuilder.yoctoproject.org/typhoon/#/builders/76/builds/2876/steps/14/logs/stdio
https://autobuilder.yoctoproject.org/typhoon/#/builders/76/builds/2890/steps/14/logs/stdio
failure running apt-get update. Second time we've seen it, was on
debian10-ty-1 here, denian10-ty-3 last time. pseudo.log says:
path mismatch [1 link]: ino 143525936 db '/home/pokybuild/yocto-worker/pkgman-non-rpm/build/build/tmp/work/qemux86-poky-linux/core-image-sato/1.0-r0/apt/lists/partial/_home_pokybuild_yocto-worker_pkgman-non-rpm_build_build_tmp_work_qemux86-poky-linux_core-image-sato_1.0-r0_oe-rootfs-repo_qemux86_._Release.CBFzPd' req '/home/pokybuild/yocto-worker/pkgman-non-rpm/build/build/tmp/work/qemux86-poky-linux/core-image-sato/1.0-r0/apt/lists/partial/_home_pokybuild_yocto-worker_pkgman-non-rpm_build_build_tmp_work_qemux86-poky-linux_core-image-sato_1.0-r0_oe-rootfs-repo_qemux86_._Release'.
similar to the last time.
path mismatch [1 link]: ino 27666766 db '/home/pokybuild/yocto-worker/pkgman-non-rpm/build/build/tmp/work/qemux86-poky-linux/core-image-sato/1.0-r0/apt/lists/partial/_home_pokybuild_yocto-worker_pkgman-non-rpm_build_build_tmp_work_qemux86-poky-linux_core-image-sato_1.0-r0_oe-rootfs-repo_all_._Release.RpRzHw' req '/home/pokybuild/yocto-worker/pkgman-non-rpm/build/build/tmp/work/qemux86-poky-linux/core-image-sato/1.0-r0/apt/lists/partial/_home_pokybuild_yocto-worker_pkgman-non-rpm_build_build_tmp_work_qemux86-poky-linux_core-image-sato_1.0-r0_oe-rootfs-repo_all_._Release'

https://autobuilder.yoctoproject.org/typhoon/#/builders/87/builds/1720/steps/15/logs/stdio
new issue where dnf is complaining checksums are incorrect

https://autobuilder.yoctoproject.org/typhoon/#/builders/87/builds/1716/steps/15/logs/stdio
ppp intermittent reproducibility issue

https://autobuilder.yoctoproject.org/typhoon/#/builders/80/builds/1688/steps/14/logs/stdio
bash reproducibility issue

arm ptest warnings which will need to be added to bugs or bugs opened:
https://autobuilder.yoctoproject.org/typhoon/#/builders/82/builds/1403
https://autobuilder.yoctoproject.org/typhoon/#/builders/82/builds/1401
https://autobuilder.yoctoproject.org/typhoon/#/builders/82/builds/1400
https://autobuilder.yoctoproject.org/typhoon/#/builders/82/builds/1399

x86 ptest warnings which will need to be added to bugs or bugs opened:
https://autobuilder.yoctoproject.org/typhoon/#/builders/81/builds/1680
https://autobuilder.yoctoproject.org/typhoon/#/builders/81/builds/1676

perl install failure in man pages, think open bug needs updating?
https://autobuilder.yoctoproject.org/typhoon/#/builders/47/builds/2909

The issue with new qemu needing python 3.6+ is fixed by changes in
autobuilder-helper for debian9. I also resolved the gio postinst
execution issue with new qemu. The mingw failure with new qemu is known
Joshua is looking into that.

Cheers,

Richard








arm worker unwell

Richard Purdie
 

I've also noticed one of the arm workers appears unwell:

https://autobuilder.yoctoproject.org/typhoon/#/workers/39

Michael: Can you take a look when you're up tomorrow please?

Thanks,

Richard


Current autobuilder status

Richard Purdie
 

I'm not sure what is happening with SWAT but we're in danger of losing
track of issues which is sad after Naveen did a great job and had
everything up to date. I'm aware of the following:

https://autobuilder.yoctoproject.org/typhoon/#/builders/56/builds/1305
https://autobuilder.yoctoproject.org/typhoon/#/builders/56/builds/1306
both failed in initramfs-boot, already open bug, fix in master-next.

https://autobuilder.yoctoproject.org/typhoon/#/builders/76/builds/2876/steps/14/logs/stdio
https://autobuilder.yoctoproject.org/typhoon/#/builders/76/builds/2890/steps/14/logs/stdio
failure running apt-get update. Second time we've seen it, was on
debian10-ty-1 here, denian10-ty-3 last time. pseudo.log says:
path mismatch [1 link]: ino 143525936 db '/home/pokybuild/yocto-worker/pkgman-non-rpm/build/build/tmp/work/qemux86-poky-linux/core-image-sato/1.0-r0/apt/lists/partial/_home_pokybuild_yocto-worker_pkgman-non-rpm_build_build_tmp_work_qemux86-poky-linux_core-image-sato_1.0-r0_oe-rootfs-repo_qemux86_._Release.CBFzPd' req '/home/pokybuild/yocto-worker/pkgman-non-rpm/build/build/tmp/work/qemux86-poky-linux/core-image-sato/1.0-r0/apt/lists/partial/_home_pokybuild_yocto-worker_pkgman-non-rpm_build_build_tmp_work_qemux86-poky-linux_core-image-sato_1.0-r0_oe-rootfs-repo_qemux86_._Release'.
similar to the last time.
path mismatch [1 link]: ino 27666766 db '/home/pokybuild/yocto-worker/pkgman-non-rpm/build/build/tmp/work/qemux86-poky-linux/core-image-sato/1.0-r0/apt/lists/partial/_home_pokybuild_yocto-worker_pkgman-non-rpm_build_build_tmp_work_qemux86-poky-linux_core-image-sato_1.0-r0_oe-rootfs-repo_all_._Release.RpRzHw' req '/home/pokybuild/yocto-worker/pkgman-non-rpm/build/build/tmp/work/qemux86-poky-linux/core-image-sato/1.0-r0/apt/lists/partial/_home_pokybuild_yocto-worker_pkgman-non-rpm_build_build_tmp_work_qemux86-poky-linux_core-image-sato_1.0-r0_oe-rootfs-repo_all_._Release'

https://autobuilder.yoctoproject.org/typhoon/#/builders/87/builds/1720/steps/15/logs/stdio
new issue where dnf is complaining checksums are incorrect

https://autobuilder.yoctoproject.org/typhoon/#/builders/87/builds/1716/steps/15/logs/stdio
ppp intermittent reproducibility issue

https://autobuilder.yoctoproject.org/typhoon/#/builders/80/builds/1688/steps/14/logs/stdio
bash reproducibility issue

arm ptest warnings which will need to be added to bugs or bugs opened:
https://autobuilder.yoctoproject.org/typhoon/#/builders/82/builds/1403
https://autobuilder.yoctoproject.org/typhoon/#/builders/82/builds/1401
https://autobuilder.yoctoproject.org/typhoon/#/builders/82/builds/1400
https://autobuilder.yoctoproject.org/typhoon/#/builders/82/builds/1399

x86 ptest warnings which will need to be added to bugs or bugs opened:
https://autobuilder.yoctoproject.org/typhoon/#/builders/81/builds/1680
https://autobuilder.yoctoproject.org/typhoon/#/builders/81/builds/1676

perl install failure in man pages, think open bug needs updating?
https://autobuilder.yoctoproject.org/typhoon/#/builders/47/builds/2909

The issue with new qemu needing python 3.6+ is fixed by changes in
autobuilder-helper for debian9. I also resolved the gio postinst
execution issue with new qemu. The mingw failure with new qemu is known
Joshua is looking into that.

Cheers,

Richard


Re: SWAT Rotation

Naveen Saini
 

New bug:
Tcl ptest failure: https://bugzilla.yoctoproject.org/show_bug.cgi?id=14177

Updated existing bug:
Ltp test failure: https://bugzilla.yoctoproject.org/show_bug.cgi?id=13802
Strace qemux86-64 ptest failure (same as arm): https://bugzilla.yoctoproject.org/show_bug.cgi?id=14165

Regards,
Naveen

-----Original Message-----
From: swat@... <swat@...> On Behalf
Of Richard Purdie
Sent: Tuesday, January 5, 2021 8:41 PM
To: Saini, Naveen Kumar <naveen.kumar.saini@...>;
swat@...; akuster@...
Subject: Re: [swat] SWAT Rotation

Hi Naveen,

Thanks for looking at these!

On Tue, 2021-01-05 at 02:38 +0000, Saini, Naveen Kumar wrote:
Hi Richard,

I have located few more old build failures (master-next):

(1) ptest failure on qemux86-64-ptest (Looks intermittent, I will
raise Bug)

AssertionError: Failed ptests:
{'tcl': ['socket.test']}
https://autobuilder.yoctoproject.org/typhoon/#/builders/81/builds/1654
Bug for this one sounds good, thanks!

(2) test_ltp_groups test failure (Should I raise bug for this ?)

Reason: Looks like it's due to ssh connection failure

RESULTS - ltp.LtpTest.test_ltp_groups: ERROR (10948.27s)
SUMMARY:
core-image-sato () - Ran 6 tests in 16385.717s core-image-sato - FAIL
- Required tests failed (successes=4, skipped=1, failures=0, errors=1)


https://autobuilder.yoctoproject.org/typhoon/#/builders/95/builds/1476
I think there is an open ltp ssh connection issue so we probably need to
mention this failure on that bug?


(3) Reproducibility testcase failing due to missing pulseaudio
dependency  (debian) (need to analysis, which patch causing this
failure)

pulseaudio was skipped: pulseaudio - pulseaudio-server: system
username pulse does not have a static ID defined. Add pulse to one of
these files: /home/pokybuild/yocto-worker/oe-selftest-
debian/build/build-st-17416/meta-selftest/files/static-passwd
ERROR: Required build target 'meta-world-pkgdata' has no buildable
providers.
Missing or unbuildable dependency chain was: ['meta-world-pkgdata',
'gstreamer1.0-plugins-good', 'mpg123', 'pulseaudio']


https://autobuilder.yoctoproject.org/typhoon/#/builders/80/builds/1660
Centos:
Missing or unbuildable dependency chain was: ['meta-world-pkgdata',
'alsa-plugins', 'pulseaudio']

https://autobuilder.yoctoproject.org/typhoon/#/builders/79/builds/1678
Fedora:

https://autobuilder.yoctoproject.org/typhoon/#/builders/86/builds/1665
Ubuntu:

https://autobuilder.yoctoproject.org/typhoon/#/builders/87/builds/1690
These ones have patches in master which fix this so this one needs no
further action.


(4) ptest failure on qemux86-64 ( Seems intermittent, already
available for arm machine)

AssertionError: Failed ptests:
{'strace': ['qual_fault-syscall.test', 'qual_fault.test']}

https://autobuilder.yoctoproject.org/typhoon/#/builders/83/builds/1698
Is this the same strace test failing as the arm failure? If not it should be a
separate bug.

Cheers,

Richard






Re: SWAT Rotation

Richard Purdie
 

Hi Naveen,

Thanks for looking at these!

On Tue, 2021-01-05 at 02:38 +0000, Saini, Naveen Kumar wrote:
Hi Richard,

I have located few more old build failures (master-next):

(1) ptest failure on qemux86-64-ptest (Looks intermittent, I will
raise Bug)

AssertionError: Failed ptests:
{'tcl': ['socket.test']}
https://autobuilder.yoctoproject.org/typhoon/#/builders/81/builds/1654
Bug for this one sounds good, thanks!

(2) test_ltp_groups test failure (Should I raise bug for this ?)

Reason: Looks like it's due to ssh connection failure

RESULTS - ltp.LtpTest.test_ltp_groups: ERROR (10948.27s)
SUMMARY:
core-image-sato () - Ran 6 tests in 16385.717s
core-image-sato - FAIL - Required tests failed (successes=4,
skipped=1, failures=0, errors=1)


https://autobuilder.yoctoproject.org/typhoon/#/builders/95/builds/1476
I think there is an open ltp ssh connection issue so we probably need
to mention this failure on that bug?


(3) Reproducibility testcase failing due to missing pulseaudio
dependency  (debian) (need to analysis, which patch causing this
failure)

pulseaudio was skipped: pulseaudio - pulseaudio-server: system
username pulse does not have a static ID defined. Add pulse to one of
these files: /home/pokybuild/yocto-worker/oe-selftest-
debian/build/build-st-17416/meta-selftest/files/static-passwd
ERROR: Required build target 'meta-world-pkgdata' has no buildable
providers.
Missing or unbuildable dependency chain was: ['meta-world-pkgdata',
'gstreamer1.0-plugins-good', 'mpg123', 'pulseaudio']


https://autobuilder.yoctoproject.org/typhoon/#/builders/80/builds/1660
Centos:
Missing or unbuildable dependency chain was: ['meta-world-pkgdata',
'alsa-plugins', 'pulseaudio']

https://autobuilder.yoctoproject.org/typhoon/#/builders/79/builds/1678
Fedora:

https://autobuilder.yoctoproject.org/typhoon/#/builders/86/builds/1665
Ubuntu:

https://autobuilder.yoctoproject.org/typhoon/#/builders/87/builds/1690
These ones have patches in master which fix this so this one needs no
further action.


(4) ptest failure on qemux86-64 ( Seems intermittent, already
available for arm machine)

AssertionError: Failed ptests:
{'strace': ['qual_fault-syscall.test', 'qual_fault.test']}

https://autobuilder.yoctoproject.org/typhoon/#/builders/83/builds/1698
Is this the same strace test failing as the arm failure? If not it
should be a separate bug.

Cheers,

Richard


Re: SWAT Rotation

Naveen Saini
 

Hi Richard,

I have located few more old build failures (master-next):

(1) ptest failure on qemux86-64-ptest (Looks intermittent, I will raise Bug)

AssertionError: Failed ptests:
{'tcl': ['socket.test']}
https://autobuilder.yoctoproject.org/typhoon/#/builders/81/builds/1654


(2) test_ltp_groups test failure (Should I raise bug for this ?)

Reason: Looks like it's due to ssh connection failure

RESULTS - ltp.LtpTest.test_ltp_groups: ERROR (10948.27s)
SUMMARY:
core-image-sato () - Ran 6 tests in 16385.717s
core-image-sato - FAIL - Required tests failed (successes=4, skipped=1, failures=0, errors=1)

https://autobuilder.yoctoproject.org/typhoon/#/builders/95/builds/1476



(3) Reproducibility testcase failing due to missing pulseaudio dependency (debian) (need to analysis, which patch causing this failure)

pulseaudio was skipped: pulseaudio - pulseaudio-server: system username pulse does not have a static ID defined. Add pulse to one of these files: /home/pokybuild/yocto-worker/oe-selftest-debian/build/build-st-17416/meta-selftest/files/static-passwd
ERROR: Required build target 'meta-world-pkgdata' has no buildable providers.
Missing or unbuildable dependency chain was: ['meta-world-pkgdata', 'gstreamer1.0-plugins-good', 'mpg123', 'pulseaudio']

https://autobuilder.yoctoproject.org/typhoon/#/builders/80/builds/1660
Centos:
Missing or unbuildable dependency chain was: ['meta-world-pkgdata', 'alsa-plugins', 'pulseaudio']
https://autobuilder.yoctoproject.org/typhoon/#/builders/79/builds/1678
Fedora:
https://autobuilder.yoctoproject.org/typhoon/#/builders/86/builds/1665
Ubuntu:
https://autobuilder.yoctoproject.org/typhoon/#/builders/87/builds/1690



(4) ptest failure on qemux86-64 ( Seems intermittent, already available for arm machine)

AssertionError: Failed ptests:
{'strace': ['qual_fault-syscall.test', 'qual_fault.test']}
https://autobuilder.yoctoproject.org/typhoon/#/builders/83/builds/1698

Regards,
Naveen

-----Original Message-----
From: swat@... <swat@...> On Behalf
Of Richard Purdie
Sent: Friday, January 1, 2021 7:47 AM
To: swat@...; sjolley.yp.pm@...;
akuster@...
Cc: Saini, Naveen Kumar <naveen.kumar.saini@...>
Subject: Re: [swat] SWAT Rotation

On Thu, 2020-12-31 at 13:24 -0800, akuster wrote:
Thanks for the reminder.

If anyone is building on 1/1/2021, they need their head examined.  I
will do my best over the Holiday weekend. See you next Yr
That sounds like me! :)

Thanks Naveen, you've done well at tracking the issues that came up and its
much appreciated. There have been quite a few more than expected,
despite a fairly low number of builds!

Happy New Year everyone!

Cheers,

Richard






Re: SWAT Rotation

Richard Purdie
 

On Thu, 2020-12-31 at 13:24 -0800, akuster wrote:
Thanks for the reminder.

If anyone is building on 1/1/2021, they need their head examined.  I
will do my best over the Holiday weekend. See you next Yr
That sounds like me! :)

Thanks Naveen, you've done well at tracking the issues that came up and
its much appreciated. There have been quite a few more than expected,
despite a fairly low number of builds!

Happy New Year everyone!

Cheers,

Richard


Re: SWAT Rotation

akuster <akuster808@...>
 

Stephen,

Thanks for the reminder.

If anyone is building on 1/1/2021, they need their head examined.  I
will do my best over the Holiday weekend. See you next Yr

- armin

On 12/31/20 12:01 PM, sjolley.yp.pm@... wrote:

Armin,

 

SWAT will rotate from Naveen to you at EOD 01/01/2021.   Please RSVP
ASAP that you know you are on SWAT!

 

See:
https://wiki.yoctoproject.org/wiki/Yocto_Build_Failure_Swat_Team#Members

 

Thanks,

 

*/Stephen K. Jolley/**//*

*Yocto Project Program Manager*

(    *Cell*:                (208) 244-4460

* *Email*:              _sjolley.yp.pm@...
<mailto:sjolley.yp.pm@...>_

 


SWAT Rotation

Stephen Jolley <sjolley.yp.pm@...>
 

Armin,

 

SWAT will rotate from Naveen to you at EOD 01/01/2021.   Please RSVP ASAP that you know you are on SWAT!

 

See: https://wiki.yoctoproject.org/wiki/Yocto_Build_Failure_Swat_Team#Members

 

Thanks,

 

Stephen K. Jolley

Yocto Project Program Manager

(    Cell:                (208) 244-4460

* Email:              sjolley.yp.pm@...

 


Re: Autobuilder pending failures

Naveen Saini
 

Hi Richard,

-----Original Message-----
From: swat@... <swat@...> On Behalf
Of Richard Purdie
Sent: Wednesday, December 23, 2020 8:44 PM
To: swat@...
Subject: [swat] Autobuilder pending failures

The autobuilder doesn't stop running over the holidays so I'm going to list the
failures I haven't been able to log/handle (and I'll probably reply to this to
add new ones if I spot them):

initramfs-boot reproducibility failure:
https://autobuilder.yoctoproject.org/typhoon/#/builders/56/builds/1291/st
eps/14/logs/stdio

[Naveen] Cause of failure: device node permission bits are not set (/dev/console).
From the recipe do_install():
mknod -m 622 ${D}/dev/console c 5 1

From diff-html:

reproducibleA/tmp/deploy/deb/all/initramfs-boot_1.0-r2_all.deb vs
reproducibleB/tmp/deploy/deb/all/initramfs-boot_1.0-r2_all.deb

3 crw--w--w-···0·root·········(0)·root·········(0)····5,··1·1970-01-01·00:00:00.000000·./dev/console
3 crw-------···0·root·········(0)·root·········(0)····5,··1·1970-01-01·00:00:00.000000·./dev/console

Raised Bug#14162 [https://bugzilla.yoctoproject.org/show_bug.cgi?id=14162]



qemuppc-alt ping timeout:
https://autobuilder.yoctoproject.org/typhoon/#/builders/107/builds/1207
[Naveen] Ping test is failing only with core-image-sato, but passing with core-image-sato-sdk and core-image-full-cmdline. Not sure why it is happening. This issue is observed with master-next. Should I raise bug ?


intermittent arm ptest failures:
https://autobuilder.yoctoproject.org/typhoon/#/builders/82/builds/1364
https://autobuilder.yoctoproject.org/typhoon/#/builders/82/builds/1359
(the exact ones change a bit each run but we really need open bugs for each
one so that someone can ultimately investigate and fix them)
[Naveen] Bug raised.
libevent: https://bugzilla.yoctoproject.org/show_bug.cgi?id=14163
libinput: https://bugzilla.yoctoproject.org/show_bug.cgi?id=14164
strace: https://bugzilla.yoctoproject.org/show_bug.cgi?id=14165


qemumips64 shutdown backtrace:
https://autobuilder.yoctoproject.org/typhoon/#/builders/74/builds/2836
[Naveen] Bug raised #14166 [ https://bugzilla.yoctoproject.org/show_bug.cgi?id=14166]


If anyone does have some time to file/update bugs or investigate any of the
above please feel free and reply to the email. If not, we'll get to things in
January unless I get bored!

master-next also have something toxic to oe-selftest in it which I've yet to
track down. I plan to keep -next running at a slower pace just to keep on top
of incoming patches as not everywhere is on holiday right now.

Cheers,

Richard