Hi All!

Welcome, and thanks for agreeing to help out with the project's "SWAT"
process.

I appreciate we have some people who've never done this before which is
great, its also great to see familiar names on the list!

We've setup this mailing list so that we can communicate with the SWAT
members easily and ensure everyone sees information related to the
team/process.

The overall idea with this process is to spread the load of triaging
the changes/testing that the project processes through the autobuilder
amongst a team of people. This is critically important to the project
as the flow of patches is important but so is maintaining our quality,
we need to do both.

Triaging the autobuilder day in, day out can be a pain when done alone
but when spread amongst a team each taking a turn once every few months
for a week, it becomes much more manageable. It also means people can
have an opportunity to interact with the community, see what is going
on in the project and become more familiar with our submission and
importantly, our testing processes. We're also keen to try and reduce
the "bus factor" around me by having processes which don't need me to
operate.

As some know, we have run this process before. Our plan is not to do
exactly what we did previously but to try and adapt the process to what
makes most sense. Where we can have tools to help us, we aim to do so
and if needed we want to improve/develop them.

Our canonical definition is maintained on this wiki page:

https://wiki.yoctoproject.org/wiki/SWAT

The process to follow is listed there. We have recently updated
BuildLog so that it only sees failing perf builds rather than all the
successful ones to reduce noise. We've also made the page autorotate to
stop it growing to an unmanageable size. We will update the process
there as we improve and better document things (I have already made
edits).

We're keen to restart this process, hopefully as soon as next week. I
had hoped to send out this email sooner but its taken us a while to get
all the members email addresses from the member companies.

To start with I'm hoping some of the "old hands" who've done this
before might like to volunteer to go first? Armin and Ross have kindly
volunteered to help mentor people new to the process and to help
oversee the first runs through the process and help get things moving.

In due course we do plan to optionally extend the role of SWAT for
people who are interested and want to engage with the community, where
instead of just triaging the failures, they would also collect up,
queue and trigger tests for the patches too. For some this makes the
work easier as they have already seen the patches under tests so can
better visualise where failures have come from. This would only be for
people who want to try that and have the experience interacting with
the community and performing some basic review.

We do have a position for a "build and integration" engineer for which
there was an open RFQ. We're in the process of appointing some people
to work in that role and in due course, our intention is that they will
help with queuing patches where needed and ensuring people are
performing the SWAT role consistently.

I'd like to request people start adding their names under the Members
heading on the wiki page, we'll then start iterating through the list.
Our plan is to post onto the list as we update the process and to
evolve things as we gain experience of what works and what can be
improved.

If anyone would like to volunteer to start this next week, that would
be great although I appreciate its short notice.

We are actively looking to collect feedback and improve the process so
please do talk to Ross, Armin and myself as appropriate if there are
suggestions/feedback etc.

Thanks,

Richard

On Fri, 6 Nov 2020 at 17:18, Richard Purdie
<richard.purdie@...> wrote:

If anyone would like to volunteer to start this next week, that would
be great although I appreciate its short notice.

As an "old hand" (less of the 'old', thanks, I prefer 'learned') I
volunteer to take the first week. I know the process well, will
happily tweak it as needed, and will update the wiki where appropriate
to ensure that the process is well documented. After that first
shakedown week we can start the rotation: any takers?

Ross

On 11/9/20 7:15 AM, Leonardo Sandoval wrote:

On Fri, 6 Nov 2020 at 12:27, Ross Burton <ross@... <mailto:ross@...>> wrote:
On Fri, 6 Nov 2020 at 17:18, Richard Purdie
<richard.purdie@...
<mailto:richard.purdie@...>> wrote:
> If anyone would like to volunteer to start this next week, that would
> be great although I appreciate its short notice.
As an "old hand" (less of the 'old', thanks, I prefer 'learned') I
volunteer to take the first week.  I know the process well, will
happily tweak it as needed, and will update the wiki where appropriate
to ensure that the process is well documented.  After that first
shakedown week we can start the rotation: any takers?
Hi Ross, Richard and team!
I have included my name in the members section. I can start anytime starting next week. Although I was part of the swat team some years ago (at Intel), I need to refresh many things but I can work on this during this week.

Same here, I will let Leo go after Ross to help shake things out for us old-timers!

After three years, I am happy to come back to the YP!

+1!

Sau!

lsg
Ross

--
Sau!

On Tue, 2020-11-10 at 14:28 +0000, Jon Mason wrote:

Feel free to add me on the end of the list. Also, it might be
beneficial to put the list on the wiki so that we can trade weeks if
there is a vacation, holiday, etc.

It *is* on the wiki:

https://wiki.yoctoproject.org/wiki/SWAT

so you can add yourself somewhere as appropriate!

Cheers,

Richard

Hi all,

We've had two weeks of Swat being back in action now and I think it's
progressing well on the whole. I've made several edits to the wiki
page[1] to clarify the process, specifically:

* Calling out that the coverage is from the "end of Friday". This is
deliberately vague to avoid having to play games with timezones, but
basically the person on Swat duty is relieved at the end of day on
Friday. This means that the incoming person is responsible for any
failures that happened over the weekend. I'll be talking to Stephen
shortly so that he can handle the rotation, and we'll send the
notification of who is next on call on Thursday so that there is time
to arrange an alternative if required.

* Reiterating that everyone looking at the builds should be updating
the build log on failures.

Myself and Richard have been designing a replacement for the buildlog
but this hasn't progressed to actual code yet. If anyone is
interested in assisting writing a Django webapp then please let us
know.

Finally, can everyone please ensure that they've added themselves to
the wiki page[1]. This is a test that you are reading the list and
have a wiki account: both requirements to do Swat! :). If you know
anyone else who would be a good fit for the Swat team (just roughly 30
minutes a day, for a week, every few months) then get in touch.

Cheers,
Ross

[1] https://wiki.yoctoproject.org/wiki/Yocto_Build_Failure_Swat_Team

Stephen,

FYI, I believe the name and the date below are incorrect.

--
Denys

Yes, according to the list it's Alejandro next week.

Ross

On Fri, 11 Dec 2020 at 02:23, Alejandro Hernandez Samaniego via
lists.yoctoproject.org
<Alejandro.Hernandez=microsoft.com@...> wrote:

Hi,

We have plans to participate in yocto swat team from Cisco (Oleksiy Obitotskyy oobitots@... & Valerii Chernous vchernou@...). Could you provide more information about process (in addition to wiki https://wiki.yoctoproject.org/wiki/Yocto_Build_Failure_Swat_Team)?

Here is what we have regarding yocto swat:

- we could monitor build status with console or by mail list. It's ok we have mail list or console and we could get error logs and some meta information about build like branch/reason, etc.

- criteria what to do with failed build is quite unclear for now. master and master-next branches is ok - we have to file bugs and or report to mail lists (?).

- after pre-triage we should write something to buildlog wiki - this step is unclear;

- we have no access to bug tracker and wiki account, but already have access to swat/yocto-builds/yocto-main mail list.

It would be nice to have some workflow example like:

- we got bug during build;

- decided on some criterion that we have to:

    - send report (to what mail lists?);

    - file bug (with some template/tags);

    - or just ignore it.

- other steps.

Regards,

Oleksiy

Acknowledged.

Hi Oleksiy,

On Tue, 15 Dec 2020 at 17:09, Oleksiy Obitotskyy via
lists.yoctoproject.org <oobitots=cisco.com@...>
wrote:

We have plans to participate in yocto swat team from Cisco (Oleksiy Obitotskyy oobitots@... & Valerii Chernous vchernou@...). Could you provide more information about process (in addition to wiki https://wiki.yoctoproject.org/wiki/Yocto_Build_Failure_Swat_Team)?

I wrote that wiki page but am very aware that explaining something to
yourself is not the same as to someone else, so I'll expand the page
after this to make it clearer.

Here is what we have regarding yocto swat:
- we could monitor build status with console or by mail list. It's ok we have mail list or console and we could get error logs and some meta information about build like branch/reason, etc.

Yes, currently my preferred method is with the yocto-builds mailing
list, as the failure mails now include the build notes (comments from
the person starting the build), and links to the error reporting
service, the build log, and so on.

- criteria what to do with failed build is quite unclear for now. master and master-next branches is ok - we have to file bugs and or report to mail lists (?).

You're the second to not understand this so this obviously needs to be
rewritten!

The release branches (gatesgarth, dunfell, etc) and master are
critical. Any failure should be in bugzilla, either a new bug or
identifying an existing bug.

The other branches the policy is a little more fluid. master-next
should be monitored by Swat with discussion with RP. I believe Steve
does build monitoring himself with stable/dunfell so those can be left
to him. Other branches, the build starter should leave a comment in
the build notes: if I've started a build to exercise a patchset then
I'll often just put "SWAT: ignore" to say that Swat should ignore any
errors as I'll handle them myself.

I'm looking into how to have this information encoded into the build
directly so we can make it clear in the build failure mail.

- after pre-triage we should write something to buildlog wiki - this step is unclear;

Yes. Every failure in the build log should have a comment from Swat
about what the problem was. A link to a bug, or a note that a mail was
sent. Just something so that the build owner can look at the log and
see what happened.

- we have no access to bug tracker and wiki account, but already have access to swat/yocto-builds/yocto-main mail list.

Requesting accounts on both of those should be simple enough. If
you're having problems (the wiki is definitely
request-and-wait-for-approval) then contact Michael Halstead.

It would be nice to have some workflow example like:
- we got bug during build;
- decided on some criterion that we have to:
- send report (to what mail lists?);
- file bug (with some template/tags);
- or just ignore it.
- other steps.

Good idea. I shall update the wiki page today.

If anything else is unclear feel free to ask.

Cheers,
Ross