Date   

Autobuilder pending failures

Richard Purdie
 

The autobuilder doesn't stop running over the holidays so I'm going to
list the failures I haven't been able to log/handle (and I'll probably
reply to this to add new ones if I spot them):

initramfs-boot reproducibility failure:
https://autobuilder.yoctoproject.org/typhoon/#/builders/56/builds/1291/steps/14/logs/stdio

qemuppc-alt ping timeout:
https://autobuilder.yoctoproject.org/typhoon/#/builders/107/builds/1207

intermittent arm ptest failures:
https://autobuilder.yoctoproject.org/typhoon/#/builders/82/builds/1364
https://autobuilder.yoctoproject.org/typhoon/#/builders/82/builds/1359
(the exact ones change a bit each run but we really need open bugs for
each one so that someone can ultimately investigate and fix them)

qemumips64 shutdown backtrace:
https://autobuilder.yoctoproject.org/typhoon/#/builders/74/builds/2836

If anyone does have some time to file/update bugs or investigate any of
the above please feel free and reply to the email. If not, we'll get to
things in January unless I get bored!

master-next also have something toxic to oe-selftest in it which I've
yet to track down. I plan to keep -next running at a slower pace just
to keep on top of incoming patches as not everywhere is on holiday
right now.

Cheers,

Richard


Update to Swat process

Ross Burton <ross@...>
 

Hi,

I've updated the Swat process wiki page to hopefully make it clearer
and explain the process better:

https://wiki.yoctoproject.org/wiki/Yocto_Build_Failure_Swat_Team

Please do read this and let me know if there's anything that isn't clear!

Also, there are still people on this list who are not listed under
Members on that page. As the list on the wiki page is how we rotate
between people on call, can everyone please ensure that they're
present.

Thanks,
Ross


Swat is on a holiday break

Ross Burton <ross@...>
 

Hi,

Because it's the holidays for many of us, Swat is taking a two week
break. It will resume on Monday 4th January.

With any luck, the autobuilder will be stable over that two week period!

Ross


Re: Yocto SWAT team

Ross Burton <ross@...>
 

Hi Oleksiy,

On Tue, 15 Dec 2020 at 17:09, Oleksiy Obitotskyy via
lists.yoctoproject.org <oobitots=cisco.com@...>
wrote:
We have plans to participate in yocto swat team from Cisco (Oleksiy Obitotskyy oobitots@... & Valerii Chernous vchernou@...). Could you provide more information about process (in addition to wiki https://wiki.yoctoproject.org/wiki/Yocto_Build_Failure_Swat_Team)?
I wrote that wiki page but am very aware that explaining something to
yourself is not the same as to someone else, so I'll expand the page
after this to make it clearer.


Here is what we have regarding yocto swat:
- we could monitor build status with console or by mail list. It's ok we have mail list or console and we could get error logs and some meta information about build like branch/reason, etc.
Yes, currently my preferred method is with the yocto-builds mailing
list, as the failure mails now include the build notes (comments from
the person starting the build), and links to the error reporting
service, the build log, and so on.

- criteria what to do with failed build is quite unclear for now. master and master-next branches is ok - we have to file bugs and or report to mail lists (?).
You're the second to not understand this so this obviously needs to be
rewritten!

The release branches (gatesgarth, dunfell, etc) and master are
critical. Any failure should be in bugzilla, either a new bug or
identifying an existing bug.

The other branches the policy is a little more fluid. master-next
should be monitored by Swat with discussion with RP. I believe Steve
does build monitoring himself with stable/dunfell so those can be left
to him. Other branches, the build starter should leave a comment in
the build notes: if I've started a build to exercise a patchset then
I'll often just put "SWAT: ignore" to say that Swat should ignore any
errors as I'll handle them myself.

I'm looking into how to have this information encoded into the build
directly so we can make it clear in the build failure mail.

- after pre-triage we should write something to buildlog wiki - this step is unclear;
Yes. Every failure in the build log should have a comment from Swat
about what the problem was. A link to a bug, or a note that a mail was
sent. Just something so that the build owner can look at the log and
see what happened.

- we have no access to bug tracker and wiki account, but already have access to swat/yocto-builds/yocto-main mail list.
Requesting accounts on both of those should be simple enough. If
you're having problems (the wiki is definitely
request-and-wait-for-approval) then contact Michael Halstead.

It would be nice to have some workflow example like:
- we got bug during build;
- decided on some criterion that we have to:
- send report (to what mail lists?);
- file bug (with some template/tags);
- or just ignore it.
- other steps.
Good idea. I shall update the wiki page today.

If anything else is unclear feel free to ask.

Cheers,
Ross


Re: SWAT Rotation

Paul Eggleton <bluelightning@...>
 

Acknowledged.

On Friday, 18 December 2020 04:46:29 NZDT sjolley.yp.pm@... wrote:
Paul,



SWAT will rotate from Alejandro to you at EOD 12/18/2020. Please RSVP ASAP
that you know you are on SWAT!



See:
https://wiki.yoctoproject.org/wiki/Yocto_Build_Failure_Swat_Team#Members



Thanks,



Stephen K. Jolley

Yocto Project Program Manager

* Cell: (208) 244-4460

* Email: sjolley.yp.pm@...
<mailto:sjolley.yp.pm@...>


SWAT Rotation

Stephen Jolley <sjolley.yp.pm@...>
 

Paul,

 

SWAT will rotate from Alejandro to you at EOD 12/18/2020.   Please RSVP ASAP that you know you are on SWAT!

 

See: https://wiki.yoctoproject.org/wiki/Yocto_Build_Failure_Swat_Team#Members

 

Thanks,

 

Stephen K. Jolley

Yocto Project Program Manager

(    Cell:                (208) 244-4460

* Email:              sjolley.yp.pm@...

 


Yocto SWAT team

Oleksiy Obitotskyy
 

Hi,

We have plans to participate in yocto swat team from Cisco (Oleksiy Obitotskyy oobitots@... & Valerii Chernous vchernou@...). Could you provide more information about process (in addition to wiki https://wiki.yoctoproject.org/wiki/Yocto_Build_Failure_Swat_Team)?

Here is what we have regarding yocto swat:
- we could monitor build status with console or by mail list. It's ok we have mail list or console and we could get error logs and some meta information about build like branch/reason, etc.

- criteria what to do with failed build is quite unclear for now. master and master-next branches is ok - we have to file bugs and or report to mail lists (?).

- after pre-triage we should write something to buildlog wiki - this step is unclear;
- we have no access to bug tracker and wiki account, but already have access to swat/yocto-builds/yocto-main mail list.

It would be nice to have some workflow example like:
- we got bug during build;
- decided on some criterion that we have to:
    - send report (to what mail lists?);
    - file bug (with some template/tags);
    - or just ignore it.
- other steps.

Regards,

Oleksiy


Re: [EXTERNAL] Re: [swat] SWAT Rotation

Ross Burton <ross@...>
 

Yes, according to the list it's Alejandro next week.

Ross

On Fri, 11 Dec 2020 at 02:23, Alejandro Hernandez Samaniego via
lists.yoctoproject.org
<Alejandro.Hernandez=microsoft.com@...> wrote:


I believe its my turn next week guys

Alejandro
________________________________
From: swat@... <swat@...> on behalf of Anibal Limon via lists.yoctoproject.org <anibal.limon=linaro.org@...>
Sent: Thursday, December 10, 2020 2:37 PM
To: swat@... <swat@...>; Denys Dmytriyenko <denis@...>
Cc: Alejandro Hernandez <alhe@...>; Saul Wold <Saul.Wold@...>
Subject: [EXTERNAL] Re: [swat] SWAT Rotation



On Thu, 10 Dec 2020 at 15:35, Denys Dmytriyenko <denis@...> wrote:

Stephen,

FYI, I believe the name and the date below are incorrect.


Hi,

I took the place of Saul about 1 week ago.

Regards,
Anibal


--
Denys


On Thu, Dec 10, 2020 at 01:26:28PM -0800, Stephen Jolley wrote:
Saul,



SWAT will rotate from Leo to you at EOD 12/04/2020. Please RSVP ASAP that
you know you are on SWAT!



See:
https://wiki.yoctoproject.org/wiki/Yocto_Build_Failure_Swat_Team#Members



Thanks,



Stephen K. Jolley

Yocto Project Program Manager

* Cell: (208) 244-4460

* Email: sjolley.yp.pm@...
<mailto:sjolley.yp.pm@...>





Re: [EXTERNAL] Re: [swat] SWAT Rotation

Alejandro Hernandez Samaniego
 

I believe its my turn next week guys

Alejandro


From: swat@... <swat@...> on behalf of Anibal Limon via lists.yoctoproject.org <anibal.limon=linaro.org@...>
Sent: Thursday, December 10, 2020 2:37 PM
To: swat@... <swat@...>; Denys Dmytriyenko <denis@...>
Cc: Alejandro Hernandez <alhe@...>; Saul Wold <Saul.Wold@...>
Subject: [EXTERNAL] Re: [swat] SWAT Rotation
 


On Thu, 10 Dec 2020 at 15:35, Denys Dmytriyenko <denis@...> wrote:
Stephen,

FYI, I believe the name and the date below are incorrect.

Hi,

I took the place of Saul about 1 week ago.

Regards,
Anibal


--
Denys


On Thu, Dec 10, 2020 at 01:26:28PM -0800, Stephen Jolley wrote:
> Saul,
>

>
> SWAT will rotate from Leo to you at EOD 12/04/2020.   Please RSVP ASAP that
> you know you are on SWAT!
>

>
> See:
> https://wiki.yoctoproject.org/wiki/Yocto_Build_Failure_Swat_Team#Members
>

>
> Thanks,
>

>
> Stephen K. Jolley
>
> Yocto Project Program Manager
>
> *    Cell:                (208) 244-4460
>
> * Email:              sjolley.yp.pm@...
> <mailto:sjolley.yp.pm@...>






Re: SWAT Rotation

Anibal Limon
 



On Thu, 10 Dec 2020 at 15:35, Denys Dmytriyenko <denis@...> wrote:
Stephen,

FYI, I believe the name and the date below are incorrect.

Hi,

I took the place of Saul about 1 week ago.

Regards,
Anibal


--
Denys


On Thu, Dec 10, 2020 at 01:26:28PM -0800, Stephen Jolley wrote:
> Saul,
>

>
> SWAT will rotate from Leo to you at EOD 12/04/2020.   Please RSVP ASAP that
> you know you are on SWAT!
>

>
> See:
> https://wiki.yoctoproject.org/wiki/Yocto_Build_Failure_Swat_Team#Members
>

>
> Thanks,
>

>
> Stephen K. Jolley
>
> Yocto Project Program Manager
>
> *    Cell:                (208) 244-4460
>
> * Email:              sjolley.yp.pm@...
> <mailto:sjolley.yp.pm@...>






Re: SWAT Rotation

Denys Dmytriyenko
 

Stephen,

FYI, I believe the name and the date below are incorrect.

--
Denys

On Thu, Dec 10, 2020 at 01:26:28PM -0800, Stephen Jolley wrote:
Saul,



SWAT will rotate from Leo to you at EOD 12/04/2020. Please RSVP ASAP that
you know you are on SWAT!



See:
https://wiki.yoctoproject.org/wiki/Yocto_Build_Failure_Swat_Team#Members



Thanks,



Stephen K. Jolley

Yocto Project Program Manager

* Cell: (208) 244-4460

* Email: sjolley.yp.pm@...
<mailto:sjolley.yp.pm@...>


SWAT Rotation

Stephen Jolley <sjolley.yp.pm@...>
 

Saul,

 

SWAT will rotate from Leo to you at EOD 12/04/2020.   Please RSVP ASAP that you know you are on SWAT!

 

See: https://wiki.yoctoproject.org/wiki/Yocto_Build_Failure_Swat_Team#Members

 

Thanks,

 

Stephen K. Jolley

Yocto Project Program Manager

(    Cell:                (208) 244-4460

* Email:              sjolley.yp.pm@...

 


SWAT Rotation

Stephen Jolley <sjolley.yp.pm@...>
 

Saul,

 

SWAT will rotate from Leo to you at EOD 12/04/2020.   Please RSVP ASAP that you know you are on SWAT!

 

See: https://wiki.yoctoproject.org/wiki/Yocto_Build_Failure_Swat_Team#Members

 

Thanks,

 

Stephen K. Jolley

Yocto Project Program Manager

(    Cell:                (208) 244-4460

* Email:              sjolley.yp.pm@...

 


Swat process and update

Ross Burton <ross@...>
 

Hi all,

We've had two weeks of Swat being back in action now and I think it's
progressing well on the whole. I've made several edits to the wiki
page[1] to clarify the process, specifically:

* Calling out that the coverage is from the "end of Friday". This is
deliberately vague to avoid having to play games with timezones, but
basically the person on Swat duty is relieved at the end of day on
Friday. This means that the incoming person is responsible for any
failures that happened over the weekend. I'll be talking to Stephen
shortly so that he can handle the rotation, and we'll send the
notification of who is next on call on Thursday so that there is time
to arrange an alternative if required.

* Reiterating that everyone looking at the builds should be updating
the build log on failures.

Myself and Richard have been designing a replacement for the buildlog
but this hasn't progressed to actual code yet. If anyone is
interested in assisting writing a Django webapp then please let us
know.

Finally, can everyone please ensure that they've added themselves to
the wiki page[1]. This is a test that you are reading the list and
have a wiki account: both requirements to do Swat! :). If you know
anyone else who would be a good fit for the Swat team (just roughly 30
minutes a day, for a week, every few months) then get in touch.

Cheers,
Ross

[1] https://wiki.yoctoproject.org/wiki/Yocto_Build_Failure_Swat_Team


Re: [EXTERNAL] Re: [swat] Restarting SWAT

Richard Purdie
 

On Tue, 2020-11-10 at 14:28 +0000, Jon Mason wrote:
Feel free to add me on the end of the list. Also, it might be
beneficial to put the list on the wiki so that we can trade weeks if
there is a vacation, holiday, etc.
It *is* on the wiki:

https://wiki.yoctoproject.org/wiki/SWAT

so you can add yourself somewhere as appropriate!

Cheers,

Richard


Re: [EXTERNAL] Re: [swat] Restarting SWAT

Jon Mason
 

Outlook and top posting, hate me!

 

Feel free to add me on the end of the list.  Also, it might be beneficial to put the list on the wiki so that we can trade weeks if there is a vacation, holiday, etc.

 

Thanks,

Jon

 

From: swat@... <swat@...> On Behalf Of Naveen Saini via lists.yoctoproject.org
Sent: Monday, November 9, 2020 6:52 PM
To: swat@...
Cc: Stephen K Jolley (via Google Docs) <sjolley.yp.pm@...>
Subject: Re: [EXTERNAL] Re: [swat] Restarting SWAT

 

Hi all,

 

I am new to this process, but I can go after Alejandro.

 

Regards,

Naveen

 

From: swat@... <swat@...> On Behalf Of Alejandro Hernandez Samaniego via lists.yoctoproject.org
Sent: Tuesday, November 10, 2020 2:33 AM
To: swat@...
Cc: Stephen K Jolley (via Google Docs) <sjolley.yp.pm@...>
Subject: Re: [EXTERNAL] Re: [swat] Restarting SWAT

 

Hello everyone,

 

I'm glad we're setting this up again, just added myself to the Wiki header, I can go after Anibal for old times sake.

 

Alejandro


From: swat@... <swat@...> on behalf of Anibal Limon via lists.yoctoproject.org <anibal.limon=linaro.org@...>
Sent: Monday, November 9, 2020 8:50 AM
To: swat@... <swat@...>
Cc: Stephen K Jolley (via Google Docs) <sjolley.yp.pm@...>
Subject: [EXTERNAL] Re: [swat] Restarting SWAT

 

 

 

On Mon, 9 Nov 2020 at 09:30, Saul Wold <Saul.Wold@...> wrote:



On 11/9/20 7:15 AM, Leonardo Sandoval wrote:
>
>
> On Fri, 6 Nov 2020 at 12:27, Ross Burton <ross@...
> <mailto:ross@...>> wrote:
>
>     On Fri, 6 Nov 2020 at 17:18, Richard Purdie
>     <richard.purdie@...
>     <mailto:richard.purdie@...>> wrote:
>      > If anyone would like to volunteer to start this next week, that would
>      > be great although I appreciate its short notice.
>
>     As an "old hand" (less of the 'old', thanks, I prefer 'learned') I
>     volunteer to take the first week.  I know the process well, will
>     happily tweak it as needed, and will update the wiki where appropriate
>     to ensure that the process is well documented.  After that first
>     shakedown week we can start the rotation: any takers?
>
>
> Hi Ross, Richard and team!
> I have included my name in the members section. I can start anytime
> starting next week. Although I was part of the swat team some years ago
> (at Intel), I need to refresh many things but I can work on this during
> this week.
>
Same here, I will let Leo go after Ross to help shake things out for us
old-timers!

 

Hi all, I can go after Sau,

 

Cheers,

Anibal

 


> After three years, I am happy to come back to the YP!
>
+1!

Sau!

> lsg
>
>
>     Ross
>
>
>
>
>
>

--
Sau!



IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.


Re: [EXTERNAL] Re: [swat] Restarting SWAT

Naveen Saini
 

Hi all,

 

I am new to this process, but I can go after Alejandro.

 

Regards,

Naveen

 

From: swat@... <swat@...> On Behalf Of Alejandro Hernandez Samaniego via lists.yoctoproject.org
Sent: Tuesday, November 10, 2020 2:33 AM
To: swat@...
Cc: Stephen K Jolley (via Google Docs) <sjolley.yp.pm@...>
Subject: Re: [EXTERNAL] Re: [swat] Restarting SWAT

 

Hello everyone,

 

I'm glad we're setting this up again, just added myself to the Wiki header, I can go after Anibal for old times sake.

 

Alejandro


From: swat@... <swat@...> on behalf of Anibal Limon via lists.yoctoproject.org <anibal.limon=linaro.org@...>
Sent: Monday, November 9, 2020 8:50 AM
To: swat@... <swat@...>
Cc: Stephen K Jolley (via Google Docs) <sjolley.yp.pm@...>
Subject: [EXTERNAL] Re: [swat] Restarting SWAT

 

 

 

On Mon, 9 Nov 2020 at 09:30, Saul Wold <Saul.Wold@...> wrote:



On 11/9/20 7:15 AM, Leonardo Sandoval wrote:
>
>
> On Fri, 6 Nov 2020 at 12:27, Ross Burton <ross@...
> <mailto:ross@...>> wrote:
>
>     On Fri, 6 Nov 2020 at 17:18, Richard Purdie
>     <richard.purdie@...
>     <mailto:richard.purdie@...>> wrote:
>      > If anyone would like to volunteer to start this next week, that would
>      > be great although I appreciate its short notice.
>
>     As an "old hand" (less of the 'old', thanks, I prefer 'learned') I
>     volunteer to take the first week.  I know the process well, will
>     happily tweak it as needed, and will update the wiki where appropriate
>     to ensure that the process is well documented.  After that first
>     shakedown week we can start the rotation: any takers?
>
>
> Hi Ross, Richard and team!
> I have included my name in the members section. I can start anytime
> starting next week. Although I was part of the swat team some years ago
> (at Intel), I need to refresh many things but I can work on this during
> this week.
>
Same here, I will let Leo go after Ross to help shake things out for us
old-timers!

 

Hi all, I can go after Sau,

 

Cheers,

Anibal

 


> After three years, I am happy to come back to the YP!
>
+1!

Sau!

> lsg
>
>
>     Ross
>
>
>
>
>
>

--
Sau!





Re: [EXTERNAL] Re: [swat] Restarting SWAT

Alejandro Hernandez Samaniego
 

Hello everyone,

I'm glad we're setting this up again, just added myself to the Wiki header, I can go after Anibal for old times sake.

Alejandro


From: swat@... <swat@...> on behalf of Anibal Limon via lists.yoctoproject.org <anibal.limon=linaro.org@...>
Sent: Monday, November 9, 2020 8:50 AM
To: swat@... <swat@...>
Cc: Stephen K Jolley (via Google Docs) <sjolley.yp.pm@...>
Subject: [EXTERNAL] Re: [swat] Restarting SWAT
 


On Mon, 9 Nov 2020 at 09:30, Saul Wold <Saul.Wold@...> wrote:


On 11/9/20 7:15 AM, Leonardo Sandoval wrote:
>
>
> On Fri, 6 Nov 2020 at 12:27, Ross Burton <ross@...
> <mailto:ross@...>> wrote:
>
>     On Fri, 6 Nov 2020 at 17:18, Richard Purdie
>     <richard.purdie@...
>     <mailto:richard.purdie@...>> wrote:
>      > If anyone would like to volunteer to start this next week, that would
>      > be great although I appreciate its short notice.
>
>     As an "old hand" (less of the 'old', thanks, I prefer 'learned') I
>     volunteer to take the first week.  I know the process well, will
>     happily tweak it as needed, and will update the wiki where appropriate
>     to ensure that the process is well documented.  After that first
>     shakedown week we can start the rotation: any takers?
>
>
> Hi Ross, Richard and team!
> I have included my name in the members section. I can start anytime
> starting next week. Although I was part of the swat team some years ago
> (at Intel), I need to refresh many things but I can work on this during
> this week.
>
Same here, I will let Leo go after Ross to help shake things out for us
old-timers!

Hi all, I can go after Sau,

Cheers,
Anibal
 

> After three years, I am happy to come back to the YP!
>
+1!

Sau!

> lsg
>
>
>     Ross
>
>
>
>
>
>

--
Sau!






Re: Restarting SWAT

Anibal Limon
 



On Mon, 9 Nov 2020 at 09:30, Saul Wold <Saul.Wold@...> wrote:


On 11/9/20 7:15 AM, Leonardo Sandoval wrote:
>
>
> On Fri, 6 Nov 2020 at 12:27, Ross Burton <ross@...
> <mailto:ross@...>> wrote:
>
>     On Fri, 6 Nov 2020 at 17:18, Richard Purdie
>     <richard.purdie@...
>     <mailto:richard.purdie@...>> wrote:
>      > If anyone would like to volunteer to start this next week, that would
>      > be great although I appreciate its short notice.
>
>     As an "old hand" (less of the 'old', thanks, I prefer 'learned') I
>     volunteer to take the first week.  I know the process well, will
>     happily tweak it as needed, and will update the wiki where appropriate
>     to ensure that the process is well documented.  After that first
>     shakedown week we can start the rotation: any takers?
>
>
> Hi Ross, Richard and team!
> I have included my name in the members section. I can start anytime
> starting next week. Although I was part of the swat team some years ago
> (at Intel), I need to refresh many things but I can work on this during
> this week.
>
Same here, I will let Leo go after Ross to help shake things out for us
old-timers!

Hi all, I can go after Sau,

Cheers,
Anibal
 

> After three years, I am happy to come back to the YP!
>
+1!

Sau!

> lsg
>
>
>     Ross
>
>
>
>
>
>

--
Sau!






Re: Restarting SWAT

Saul Wold
 

On 11/9/20 7:15 AM, Leonardo Sandoval wrote:
On Fri, 6 Nov 2020 at 12:27, Ross Burton <ross@... <mailto:ross@...>> wrote:
On Fri, 6 Nov 2020 at 17:18, Richard Purdie
<richard.purdie@...
<mailto:richard.purdie@...>> wrote:
> If anyone would like to volunteer to start this next week, that would
> be great although I appreciate its short notice.
As an "old hand" (less of the 'old', thanks, I prefer 'learned') I
volunteer to take the first week.  I know the process well, will
happily tweak it as needed, and will update the wiki where appropriate
to ensure that the process is well documented.  After that first
shakedown week we can start the rotation: any takers?
Hi Ross, Richard and team!
I have included my name in the members section. I can start anytime starting next week. Although I was part of the swat team some years ago (at Intel), I need to refresh many things but I can work on this during this week.
Same here, I will let Leo go after Ross to help shake things out for us old-timers!

After three years, I am happy to come back to the YP!
+1!

Sau!

lsg
Ross
--
Sau!