Date   

Further info on the fetch issues in meta-aws

Richard Purdie
 

Hi Richard,

I hate to be the bearer of bad news but there is a bit of an issue with meta-aws
and it may be more serious then we originally thought.

I've been trying to fix some fetcher issues and came up with a fix:

https://git.yoctoproject.org/poky/commit/?h=master-next&id=222dd76493ea18c4fde5d3d34eeda9257adf34a2

and some tests:

https://git.yoctoproject.org/poky/commit/?h=master-next&id=66ee0b94a88c6aaaddb3580f044a5cc5d35b7cc4

however the newly added error shows issues in meta-aws:

https://autobuilder.yoctoproject.org/typhoon/#/builders/122/builds/769/steps/12/logs/stdio

In the first commit linked above I explain why the lack of use of get_srcrev()
is a problem:

"""
Where a git url uses a tag instead of a full source revision, breakage
can currently occur in builds. Issues include:

* the revision being looked up in multiple tasks (fetch and unpack)
* the risk a different revision may be obtained in those tasks
* that some tasks may not be allowed to access the network
* that a revision may not be consistent throughout a given build
* rerunning a specific task may given inconsistent results
"""

and I suspect these are things you care about.

You don't have to set PV to include SRCPV, you could just call
bb.fetch2.get_srcrev(d) manually in those recipes instead.

This probably explains some of the previous strange behaviour for some of the
recipes.

Cheers,

Richard


Re: Failure on AB with meta-agl-core

Jan Simon Moeller
 

Scott is aware and on top of it.

js

Richard Purdie <richard.purdie@...> schrieb am Fr., 4. Feb. 2022, 18:51:

On Fri, 2022-02-04 at 09:49 -0800, Saul Wold wrote:
> Hi Richard,
>
> Reaching out due to a failure in meta-agl-core on the Autobuilder.
> Weston has been updated to 10.x, but the bbappend in meta-agl-core is
> still 9.x based.
>
> Do you have plans for updating?

I did already talk with Scott Murray and there are patches ready for this. I
think meta-aws was ok?

Cheers,

Richard







Re: Failure on AB with meta-agl-core

Richard Purdie
 

On Fri, 2022-02-04 at 09:49 -0800, Saul Wold wrote:
Hi Richard,

Reaching out due to a failure in meta-agl-core on the Autobuilder.
Weston has been updated to 10.x, but the bbappend in meta-agl-core is
still 9.x based.

Do you have plans for updating?
I did already talk with Scott Murray and there are patches ready for this. I
think meta-aws was ok?

Cheers,

Richard


Failure on AB with meta-agl-core

Saul Wold
 

Hi Richard,

Reaching out due to a failure in meta-agl-core on the Autobuilder. Weston has been updated to 10.x, but the bbappend in meta-agl-core is still 9.x based.

Do you have plans for updating?

Thanks
--
Sau!


Re: SWAT Rotation schedule

Saul Wold
 

I am on SWAT and will start reviewing the logs in my morning.

Sau!

On 1/26/22 16:09, Alexandre Belloni wrote:
Hello everyone,
Following the winter break, I would like to organize SWAT duty rotation.
I've again prepared a schedule so it is easier for each one of you to
plan for SWAT duty.
Please check the table and let me know if you are not available for the
selected week. SWAT duty will be from Friday to Thursday and the goal is
to triage all the failures on swatbot before the weekly triage call
happening at 2:30pm UTC.
At the current failure rate, I would expect the task to take between a
half a day and a day of work.
┌───────────────────────────────┬──────┬────────────┐
│ │ Week │ Start │
├───────────────────────────────┼──────┼────────────┤
│ Anibal Limon │ 4 │ 28/01/2022 │
│ Saul Wold │ 5 │ 04/02/2022 │
│ Alejandro Hernandez Samaniego │ 6 │ 11/02/2022 │
│ Oleksiy Obitotskyy │ 7 │ 18/02/2022 │
│ Naveen Saini │ 8 │ 25/02/2022 │
│ Paul Eggleton │ 9 │ 04/03/2022 │
│ Christopher Larson │ 10 │ 11/03/2022 │
│ Jon Mason │ 11 │ 18/03/2022 │
│ Lee Chee Yang │ 12 │ 25/03/2022 │
│ Minjae Kim │ 13 │ 01/04/2022 │
│ Jaga │ 14 │ 08/04/2022 │
│ Leo Sandoval │ 15 │ 15/04/2022 │
│ Ross Burton │ 16 │ 22/04/2022 │
│ Valerii Chernous │ 17 │ 29/04/2022 │
└───────────────────────────────┴──────┴────────────┘
Anibal you would be the next one on the list, starting this Friday,
can you confirm you are available?
There are currently 2 failures to triage on swatbot, I'm going to take
care of those.
When filling in bugs or bug comments, please also add the worker name
and whether it has an SSD. Currently, we have:
┌──────────────────┬─────────┐
│ alma8-ty-1 │ SSD │
│ alma8-ty-2 │ SSD │
│ debian11-ty-3 │ SSD │
│ fedora35-ty-1 │ SSD │
│ fedora35-ty-2 │ SSD │
│ opensuse153-ty-1 │ SSD │
│ stream8-ty-1 │ SSD │
│ ubuntu2004-ty-1 │ SSD │
│ ubuntu2110-ty-2 │ SSD │
│ centos7-ty-4 │ Non-SSD │
│ debian11-ty-1 │ Non-SSD │
│ fedora34-ty-1 │ Non-SSD │
│ perf-centos7 │ Non-SSD │
│ perf-ubuntu1604 │ Non-SSD │
│ tumbleweed-ty-3 │ Non-SSD │
│ ubuntu1604-ty-1 │ Non-SSD │
│ ubuntu1804-arm-1 │ Non-SSD │
│ ubuntu1804-ty-3 │ Non-SSD │
│ ubuntu2004-arm-1 │ Non-SSD │
│ centos8-ty-1 │ Non-SSD │
│ centos8-ty-2 │ Non-SSD │
│ debian10-ty-1 │ Non-SSD │
│ debian11-ty-2 │ Non-SSD │
│ debian9-ty-2 │ Non-SSD │
│ opensuse154-ty-1 │ Non-SSD │
└──────────────────┴─────────┘
Thanks!
--
Sau!


Re: meta-aws warnings: SRCREV_FORMAT variable must be set when multiple SCMs are used

elberger@...
 

This makes a lot of sense to me and it's clear now. It didn't come across this way in prior responses. Thank you for taking the time to put this together.

We can't remove aws-c-common from the sources since ... it's a build quirk ... virtually all repos require it to be cloned in the source tree context.

Fixing this is going to take quite some time - will need to get the work scheduled. Hopefully will get it done over the next week.

-----Original Message-----
From: Jan-Simon Moeller <jsmoeller@...>
Sent: Thursday, February 3, 2022 9:41 AM
To: Elberger, Richard <elberger@...>
Subject: FW: [EXTERNAL] [swat] meta-aws warnings: SRCREV_FORMAT variable must be set when multiple SCMs are used

CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe.



Forgot to add all CC back ...
Best regards,
Jan-Simon

------
Jan-Simon Möller
AGL Release Manager
The Linux Foundation

Visit us at:
www.automotivegradelinux.org
lists.automotivelinux.org
www.linuxfoundation.org


---------- Forwarded message ---------
From: Jan-Simon Moeller <jsmoeller@...>
Date: Thu, Feb 3, 2022 at 3:40 PM
Subject: Re: [swat] meta-aws warnings: SRCREV_FORMAT variable must be set when multiple SCMs are used
To: <swat@...>
Cc: Denys Dmytriyenko <denis@...>, Glimsdale, Nathan <glimsdal@...>


So the main error is:
aws-c-auth-0.6.1-r0 do_packagedata_setscene: ExpansionError('SRCPV', '${@bb.fetch2.get_srcrev(d)}', FetchError('The SRCREV_FORMAT variable must be set when multiple SCMs are used.\nThe SCMs are:\ngit://github.com/awslabs/aws-c-common.git;protocol=https;branch=main;tag=v0.6.8;destsuffix=/home/pokybuild/yocto-worker/meta-aws/build/build/tmp/work/cortexa57-poky-linux/aws-c-auth/0.6.1-r0/git/aws-c-common;name=common\ngit://github.com/awslabs/aws-c-auth.git;protocol=https;branch=main;tag=v0.6.1;destsuffix=/home/pokybuild/yocto-worker/meta-aws/build/build/tmp/work/cortexa57-poky-linux/aws-c-auth/0.6.1-r0/git/aws-c-auth;name=auth',
None))


Let's try to dissect:

The SRCREV_FORMAT variable must be set when multiple SCMs are used.
The SCMs are:
git://github.com/awslabs/aws-c-common.git;protocol=https;branch=main;tag=v0.6.8;destsuffix=/home/pokybuild/yocto-worker/meta-aws/build/build/tmp/work/cortexa57-poky-linux/aws-c-auth/0.6.1-r0/git/aws-c-common;name=common
git://github.com/awslabs/aws-c-auth.git;protocol=https;branch=main;tag=v0.6.1;destsuffix=/home/pokybuild/yocto-worker/meta-aws/build/build/tmp/work/cortexa57-poky-linux/aws-c-auth/0.6.1-r0/git/aws-c-auth;name=auth


I read this like:
- we're using two git:// url's in the SRC_URI
- bitbake needs one single hash value for SRCREV for the recipe to be 'uniq'
- we cannot decide what to use as SRCREV based on this SRC_URI (with multiple git:// entries)

I found in meta-arm:
SRC_URI = "git://git.trustedfirmware.org/TF-A/trusted-firmware-a.git;protocol=https;name=tfa"
# Use TF-A for version
SRCREV_FORMAT = "tfa"
# ^^^^^^^^^^^^^^^^^^^^^^^^ !!!!!!!! tfa = name from above # TF-A v2.1 SRCREV_tfa = "e1286bdb968ee74fc52f96cf303a4218e1ae2950"
#
# mbed TLS source
# Those are used in trusted-firmware-a.inc if TFA_MBEDTLS is set to 1 SRC_URI_MBEDTLS = "git://github.com/ARMmbed/mbedtls.git;name=mbedtls;protocol=https;destsuffix=git/mbedtls;branch=mbedtls-2.16"
SRCREV_mbedtls = "d81c11b8ab61fd5b2da8133aa73c5fe33a0633eb"


So to apply this to meta-aws, you'd have to pick one which should be the canonical SRCREV of the two for this recipe.
E.g. (untested)

aws-c-auth_0.6.1.bb :
[...]
TAG ?= "v${PV}"
TAG_COMMON ?= "v0.6.8"
SRC_URI = "git://github.com/awslabs/aws-c-common.git;protocol=https;branch=${BRANCH};tag=${TAG_COMMON};destsuffix=${S}/aws-c-common;name=common
\
git://github.com/awslabs/aws-c-auth.git;protocol=https;branch=${BRANCH};tag=${TAG};destsuffix=${S}/aws-c-auth;name=auth
\
"
S= "${WORKDIR}/git"
BRANCH ?= "main"
# _____________________
SRCREV_FORMAT = "auth"
# ^^^^^^^^^^^^^^^^^^^^



But the main question here is: why do you need aws-c-common in SRC_URI ? . aws-c-common has its own recipe, so if it installs the headers properly it can be in DEPENDS and will be pulled automagically.

The optimization wrt tag and network / offline that Denis mentioned is important imho as well ! It is really better to use a SRCREV. But yes, it might affect workflow and means bumping SRCREV's.

TLDR:
a) If possible make aws-c-common a DEPENDS in the auth recipe
b) if above does not work, try SRCREV_FORMAT = "auth" (other recipes likewise)
c) tag should be optimized to a SRCREV to avoid the enforced network fetch


HTH.
Best,
Jan-Simon

------
Jan-Simon Möller
AGL Release Manager
The Linux Foundation

Visit us at:
www.automotivegradelinux.org
lists.automotivelinux.org
www.linuxfoundation.org

Best regards,
Jan-Simon

------
Jan-Simon Möller
AGL Release Manager
The Linux Foundation

Visit us at:
www.automotivegradelinux.org
lists.automotivelinux.org
www.linuxfoundation.org


On Thu, Feb 3, 2022 at 3:12 PM Alexandre Belloni via lists.yoctoproject.org <alexandre.belloni=bootlin.com@...> wrote:

On 03/02/2022 14:02:54+0000, elberger via lists.yoctoproject.org wrote:
The choice to use tags is a deliberate one.

For all AWS repos that meta-aws builds recipes we use release tagging to align with a specific release. While tags can move (aka the old school CVS floating label practice which is very frowned upon except for noting "stable") they do not for these repos. Of course, there is the exception when a release is about to be made and a late breaking change must be added to the release. Then the tag shifts to meet that demand. This happens very rarely and is not a contributor to the general concern of repo maintainers implementing the floating tag/label practice and the recipe using that tag.

Further, I don't think such a check in yocto-check-layer is appropriate. Rather, if a recipe author knows that a specific CM practice is in place for particular repos then have the opportunity to tell the various stages "don't check, I know what I'm doing". If YP/OE really is against using the tag mechanism then I suggest we deprecate it. I have found it very nice for maintainership when we know exactly how the development team handles the CM.
The main drawback is that then you are not able to build with
BB_NO_NETWORK = 1 and so there is no way to ensure you have locally
all the code you need to reproduce the build in case Amazon gets out
of business one day ;)

-----Original Message-----
From: Denys Dmytriyenko <denis@...>
Sent: Wednesday, February 2, 2022 6:17 PM
To: swat@...
Cc: Glimsdale, Nathan <glimsdal@...>; Elberger, Richard
<elberger@...>
Subject: RE: [EXTERNAL] [swat] meta-aws warnings: SRCREV_FORMAT
variable must be set when multiple SCMs are used

CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe.



On Wed, Feb 02, 2022 at 11:04:57PM +0000, Richard Purdie wrote:
On Wed, 2022-02-02 at 23:54 +0100, Alexandre Belloni via
lists.yoctoproject.org
wrote:
On 02/02/2022 22:24:26+0000, Glimsdale, Nathan wrote:
I see this intermittently when building "aws-c-iot" directly.
3 of
5 attempts so far. The network restriction is also affected
some of the dependencies consistently.
Ah, interesting. Will you be tracking that on your side or
should I open a bug on bugzilla.yp.org?
I did just have a look at the aws-c-iot recipes:

https://git.yoctoproject.org/meta-aws/tree/recipes-sdk/aws-c-iot/a
ws-c
-iot_0.0.2.bb
https://git.yoctoproject.org/meta-aws/tree/recipes-sdk/aws-c-iot/a
ws-c
-iot_0.0.7.bb

and I notice they use git urls with tags.

This does not do what people expect since bitbake resolves the tag
each time over the network since people can change the tags (and do!).
I suspect that is why it is this recipe which is failing.
Ouch, no SRCREVs?

I'm guessing we need to extend yocto-check-layer with another test?
:)

--
Denys




--
Alexandre Belloni, co-owner and COO, Bootlin Embedded Linux and Kernel
engineering https://bootlin.com





Re: meta-aws warnings: SRCREV_FORMAT variable must be set when multiple SCMs are used

Richard Purdie
 

On Thu, 2022-02-03 at 14:02 +0000, elberger via lists.yoctoproject.org wrote:
The choice to use tags is a deliberate one.
It may be a deliberate choice but I suspect there are some issues you aren't
considering, specifically:

a) it breaks anyone doing offline builds
b) it means source archives don't function correct
c) it means your code isn't reproducible

For all AWS repos that meta-aws builds recipes we use release tagging to align
with a specific release. While tags can move (aka the old school CVS floating
label practice which is very frowned upon except for noting "stable") they do
not for these repos. Of course, there is the exception when a release is about
to be made and a late breaking change must be added to the release. Then the
tag shifts to meet that demand. This happens very rarely and is not a
contributor to the general concern of repo maintainers implementing the
floating tag/label practice and the recipe using that tag.
You are basically making my point here for me. Our metadata is meant to describe
a specific build exactly, at least "out the box" without changing things. By
using tags and allowing them to shift even if it is "very rarely", it isn't
explicit and therefore isn't reproducible.

Further, I don't think such a check in yocto-check-layer is appropriate. 
I'm going to disagree, let me explain why.

Rather, if a recipe author knows that a specific CM practice is in place for
particular repos then have the opportunity to tell the various stages "don't
check, I know what I'm doing".
I have no issue with anyone using tags if they want to. Most people use them not
realising there are serious consequences though (as listed above). I therefore
think this is exactly the kind of thing yocto-check-layer should be flagging up.
We don't want to "ban" them outright but we do want our best practises to say
not to use them.

If YP/OE really is against using the tag mechanism then I suggest we
deprecate it. I have found it very nice for maintainership when we know
exactly how the development team handles the CM.
I agree it is "nice" for maintainership but it has several implications which at
the very least need to be thought through. Are you happy that meta-aws isn't
reproducible, buildable offline and can't work against source archives?

Cheers,

Richard


Re: meta-aws warnings: SRCREV_FORMAT variable must be set when multiple SCMs are used

Jan Simon Moeller
 

So the main error is:
aws-c-auth-0.6.1-r0 do_packagedata_setscene: ExpansionError('SRCPV',
'${@bb.fetch2.get_srcrev(d)}', FetchError('The SRCREV_FORMAT variable
must be set when multiple SCMs are used.\nThe SCMs
are:\ngit://github.com/awslabs/aws-c-common.git;protocol=https;branch=main;tag=v0.6.8;destsuffix=/home/pokybuild/yocto-worker/meta-aws/build/build/tmp/work/cortexa57-poky-linux/aws-c-auth/0.6.1-r0/git/aws-c-common;name=common\ngit://github.com/awslabs/aws-c-auth.git;protocol=https;branch=main;tag=v0.6.1;destsuffix=/home/pokybuild/yocto-worker/meta-aws/build/build/tmp/work/cortexa57-poky-linux/aws-c-auth/0.6.1-r0/git/aws-c-auth;name=auth',
None))


Let's try to dissect:

The SRCREV_FORMAT variable must be set when multiple SCMs are used.
The SCMs are:
git://github.com/awslabs/aws-c-common.git;protocol=https;branch=main;tag=v0.6.8;destsuffix=/home/pokybuild/yocto-worker/meta-aws/build/build/tmp/work/cortexa57-poky-linux/aws-c-auth/0.6.1-r0/git/aws-c-common;name=common
git://github.com/awslabs/aws-c-auth.git;protocol=https;branch=main;tag=v0.6.1;destsuffix=/home/pokybuild/yocto-worker/meta-aws/build/build/tmp/work/cortexa57-poky-linux/aws-c-auth/0.6.1-r0/git/aws-c-auth;name=auth


I read this like:
- we're using two git:// url's in the SRC_URI
- bitbake needs one single hash value for SRCREV for the recipe to be 'uniq'
- we cannot decide what to use as SRCREV based on this SRC_URI (with
multiple git:// entries)

I found in meta-arm:
SRC_URI = "git://git.trustedfirmware.org/TF-A/trusted-firmware-a.git;protocol=https;name=tfa"
# Use TF-A for version
SRCREV_FORMAT = "tfa"
# ^^^^^^^^^^^^^^^^^^^^^^^^ !!!!!!!! tfa = name from above
# TF-A v2.1
SRCREV_tfa = "e1286bdb968ee74fc52f96cf303a4218e1ae2950"
#
# mbed TLS source
# Those are used in trusted-firmware-a.inc if TFA_MBEDTLS is set to 1
SRC_URI_MBEDTLS =
"git://github.com/ARMmbed/mbedtls.git;name=mbedtls;protocol=https;destsuffix=git/mbedtls;branch=mbedtls-2.16"
SRCREV_mbedtls =
"d81c11b8ab61fd5b2da8133aa73c5fe33a0633eb"


So to apply this to meta-aws, you'd have to pick one which should be
the canonical SRCREV of the two for this recipe.
E.g. (untested)

aws-c-auth_0.6.1.bb :
[...]
TAG ?= "v${PV}"
TAG_COMMON ?= "v0.6.8"
SRC_URI = "git://github.com/awslabs/aws-c-common.git;protocol=https;branch=${BRANCH};tag=${TAG_COMMON};destsuffix=${S}/aws-c-common;name=common
\
git://github.com/awslabs/aws-c-auth.git;protocol=https;branch=${BRANCH};tag=${TAG};destsuffix=${S}/aws-c-auth;name=auth
\
"
S= "${WORKDIR}/git"
BRANCH ?= "main"
# _____________________
SRCREV_FORMAT = "auth"
# ^^^^^^^^^^^^^^^^^^^^



But the main question here is: why do you need aws-c-common in SRC_URI
? . aws-c-common has its own recipe, so if it installs the headers
properly it can be in DEPENDS and will be pulled automagically.

The optimization wrt tag and network / offline that Denis mentioned is
important imho as well ! It is really better to use a SRCREV. But yes,
it might affect workflow and means bumping SRCREV's.

TLDR:
a) If possible make aws-c-common a DEPENDS in the auth recipe
b) if above does not work, try SRCREV_FORMAT = "auth" (other recipes likewise)
c) tag should be optimized to a SRCREV to avoid the enforced network fetch


HTH.
Best,
Jan-Simon

------
Jan-Simon Möller
AGL Release Manager
The Linux Foundation

Visit us at:
www.automotivegradelinux.org
lists.automotivelinux.org
www.linuxfoundation.org

Best regards,
Jan-Simon

------
Jan-Simon Möller
AGL Release Manager
The Linux Foundation

Visit us at:
www.automotivegradelinux.org
lists.automotivelinux.org
www.linuxfoundation.org


On Thu, Feb 3, 2022 at 3:12 PM Alexandre Belloni via
lists.yoctoproject.org
<alexandre.belloni=bootlin.com@...> wrote:


On 03/02/2022 14:02:54+0000, elberger via lists.yoctoproject.org wrote:
The choice to use tags is a deliberate one.

For all AWS repos that meta-aws builds recipes we use release tagging to align with a specific release. While tags can move (aka the old school CVS floating label practice which is very frowned upon except for noting "stable") they do not for these repos. Of course, there is the exception when a release is about to be made and a late breaking change must be added to the release. Then the tag shifts to meet that demand. This happens very rarely and is not a contributor to the general concern of repo maintainers implementing the floating tag/label practice and the recipe using that tag.

Further, I don't think such a check in yocto-check-layer is appropriate. Rather, if a recipe author knows that a specific CM practice is in place for particular repos then have the opportunity to tell the various stages "don't check, I know what I'm doing". If YP/OE really is against using the tag mechanism then I suggest we deprecate it. I have found it very nice for maintainership when we know exactly how the development team handles the CM.
The main drawback is that then you are not able to build with
BB_NO_NETWORK = 1 and so there is no way to ensure you have locally all
the code you need to reproduce the build in case Amazon gets out of
business one day ;)

-----Original Message-----
From: Denys Dmytriyenko <denis@...>
Sent: Wednesday, February 2, 2022 6:17 PM
To: swat@...
Cc: Glimsdale, Nathan <glimsdal@...>; Elberger, Richard <elberger@...>
Subject: RE: [EXTERNAL] [swat] meta-aws warnings: SRCREV_FORMAT variable must be set when multiple SCMs are used

CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe.



On Wed, Feb 02, 2022 at 11:04:57PM +0000, Richard Purdie wrote:
On Wed, 2022-02-02 at 23:54 +0100, Alexandre Belloni via
lists.yoctoproject.org
wrote:
On 02/02/2022 22:24:26+0000, Glimsdale, Nathan wrote:
I see this intermittently when building "aws-c-iot" directly. 3 of
5 attempts so far. The network restriction is also affected some
of the dependencies consistently.
Ah, interesting. Will you be tracking that on your side or should I
open a bug on bugzilla.yp.org?
I did just have a look at the aws-c-iot recipes:

https://git.yoctoproject.org/meta-aws/tree/recipes-sdk/aws-c-iot/aws-c
-iot_0.0.2.bb
https://git.yoctoproject.org/meta-aws/tree/recipes-sdk/aws-c-iot/aws-c
-iot_0.0.7.bb

and I notice they use git urls with tags.

This does not do what people expect since bitbake resolves the tag
each time over the network since people can change the tags (and do!).
I suspect that is why it is this recipe which is failing.
Ouch, no SRCREVs?

I'm guessing we need to extend yocto-check-layer with another test? :)

--
Denys




--
Alexandre Belloni, co-owner and COO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com





Re: meta-aws warnings: SRCREV_FORMAT variable must be set when multiple SCMs are used

Alexandre Belloni
 

On 03/02/2022 14:02:54+0000, elberger via lists.yoctoproject.org wrote:
The choice to use tags is a deliberate one.

For all AWS repos that meta-aws builds recipes we use release tagging to align with a specific release. While tags can move (aka the old school CVS floating label practice which is very frowned upon except for noting "stable") they do not for these repos. Of course, there is the exception when a release is about to be made and a late breaking change must be added to the release. Then the tag shifts to meet that demand. This happens very rarely and is not a contributor to the general concern of repo maintainers implementing the floating tag/label practice and the recipe using that tag.

Further, I don't think such a check in yocto-check-layer is appropriate. Rather, if a recipe author knows that a specific CM practice is in place for particular repos then have the opportunity to tell the various stages "don't check, I know what I'm doing". If YP/OE really is against using the tag mechanism then I suggest we deprecate it. I have found it very nice for maintainership when we know exactly how the development team handles the CM.
The main drawback is that then you are not able to build with
BB_NO_NETWORK = 1 and so there is no way to ensure you have locally all
the code you need to reproduce the build in case Amazon gets out of
business one day ;)

-----Original Message-----
From: Denys Dmytriyenko <denis@...>
Sent: Wednesday, February 2, 2022 6:17 PM
To: swat@...
Cc: Glimsdale, Nathan <glimsdal@...>; Elberger, Richard <elberger@...>
Subject: RE: [EXTERNAL] [swat] meta-aws warnings: SRCREV_FORMAT variable must be set when multiple SCMs are used

CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe.



On Wed, Feb 02, 2022 at 11:04:57PM +0000, Richard Purdie wrote:
On Wed, 2022-02-02 at 23:54 +0100, Alexandre Belloni via
lists.yoctoproject.org
wrote:
On 02/02/2022 22:24:26+0000, Glimsdale, Nathan wrote:
I see this intermittently when building "aws-c-iot" directly. 3 of
5 attempts so far. The network restriction is also affected some
of the dependencies consistently.
Ah, interesting. Will you be tracking that on your side or should I
open a bug on bugzilla.yp.org?
I did just have a look at the aws-c-iot recipes:

https://git.yoctoproject.org/meta-aws/tree/recipes-sdk/aws-c-iot/aws-c
-iot_0.0.2.bb
https://git.yoctoproject.org/meta-aws/tree/recipes-sdk/aws-c-iot/aws-c
-iot_0.0.7.bb

and I notice they use git urls with tags.

This does not do what people expect since bitbake resolves the tag
each time over the network since people can change the tags (and do!).
I suspect that is why it is this recipe which is failing.
Ouch, no SRCREVs?

I'm guessing we need to extend yocto-check-layer with another test? :)

--
Denys




--
Alexandre Belloni, co-owner and COO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com


Re: meta-aws warnings: SRCREV_FORMAT variable must be set when multiple SCMs are used

elberger@...
 

The choice to use tags is a deliberate one.

For all AWS repos that meta-aws builds recipes we use release tagging to align with a specific release. While tags can move (aka the old school CVS floating label practice which is very frowned upon except for noting "stable") they do not for these repos. Of course, there is the exception when a release is about to be made and a late breaking change must be added to the release. Then the tag shifts to meet that demand. This happens very rarely and is not a contributor to the general concern of repo maintainers implementing the floating tag/label practice and the recipe using that tag.

Further, I don't think such a check in yocto-check-layer is appropriate. Rather, if a recipe author knows that a specific CM practice is in place for particular repos then have the opportunity to tell the various stages "don't check, I know what I'm doing". If YP/OE really is against using the tag mechanism then I suggest we deprecate it. I have found it very nice for maintainership when we know exactly how the development team handles the CM.

-----Original Message-----
From: Denys Dmytriyenko <denis@...>
Sent: Wednesday, February 2, 2022 6:17 PM
To: swat@...
Cc: Glimsdale, Nathan <glimsdal@...>; Elberger, Richard <elberger@...>
Subject: RE: [EXTERNAL] [swat] meta-aws warnings: SRCREV_FORMAT variable must be set when multiple SCMs are used

CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe.



On Wed, Feb 02, 2022 at 11:04:57PM +0000, Richard Purdie wrote:
On Wed, 2022-02-02 at 23:54 +0100, Alexandre Belloni via
lists.yoctoproject.org
wrote:
On 02/02/2022 22:24:26+0000, Glimsdale, Nathan wrote:
I see this intermittently when building "aws-c-iot" directly. 3 of
5 attempts so far. The network restriction is also affected some
of the dependencies consistently.
Ah, interesting. Will you be tracking that on your side or should I
open a bug on bugzilla.yp.org?
I did just have a look at the aws-c-iot recipes:

https://git.yoctoproject.org/meta-aws/tree/recipes-sdk/aws-c-iot/aws-c
-iot_0.0.2.bb
https://git.yoctoproject.org/meta-aws/tree/recipes-sdk/aws-c-iot/aws-c
-iot_0.0.7.bb

and I notice they use git urls with tags.

This does not do what people expect since bitbake resolves the tag
each time over the network since people can change the tags (and do!).
I suspect that is why it is this recipe which is failing.
Ouch, no SRCREVs?

I'm guessing we need to extend yocto-check-layer with another test? :)

--
Denys


Re: meta-aws warnings: SRCREV_FORMAT variable must be set when multiple SCMs are used

Denys Dmytriyenko
 

On Wed, Feb 02, 2022 at 11:04:57PM +0000, Richard Purdie wrote:
On Wed, 2022-02-02 at 23:54 +0100, Alexandre Belloni via lists.yoctoproject.org
wrote:
On 02/02/2022 22:24:26+0000, Glimsdale, Nathan wrote:
I see this intermittently when building "aws-c-iot" directly. 3 of 5
attempts so far. The network restriction is also affected some of the
dependencies consistently.
Ah, interesting. Will you be tracking that on your side or should I open
a bug on bugzilla.yp.org?
I did just have a look at the aws-c-iot recipes:

https://git.yoctoproject.org/meta-aws/tree/recipes-sdk/aws-c-iot/aws-c-iot_0.0.2.bb
https://git.yoctoproject.org/meta-aws/tree/recipes-sdk/aws-c-iot/aws-c-iot_0.0.7.bb

and I notice they use git urls with tags.

This does not do what people expect since bitbake resolves the tag each time
over the network since people can change the tags (and do!). I suspect that is
why it is this recipe which is failing.
Ouch, no SRCREVs?

I'm guessing we need to extend yocto-check-layer with another test? :)

--
Denys


Re: meta-aws warnings: SRCREV_FORMAT variable must be set when multiple SCMs are used

Richard Purdie
 

On Wed, 2022-02-02 at 23:54 +0100, Alexandre Belloni via lists.yoctoproject.org
wrote:
On 02/02/2022 22:24:26+0000, Glimsdale, Nathan wrote:
I see this intermittently when building "aws-c-iot" directly. 3 of 5
attempts so far. The network restriction is also affected some of the
dependencies consistently.
Ah, interesting. Will you be tracking that on your side or should I open
a bug on bugzilla.yp.org?
I did just have a look at the aws-c-iot recipes:

https://git.yoctoproject.org/meta-aws/tree/recipes-sdk/aws-c-iot/aws-c-iot_0.0.2.bb
https://git.yoctoproject.org/meta-aws/tree/recipes-sdk/aws-c-iot/aws-c-iot_0.0.7.bb

and I notice they use git urls with tags.

This does not do what people expect since bitbake resolves the tag each time
over the network since people can change the tags (and do!). I suspect that is
why it is this recipe which is failing.

I'm not sure why that is sometimes showing a warning though.

Cheers,

Richard


Re: meta-aws warnings: SRCREV_FORMAT variable must be set when multiple SCMs are used

Alexandre Belloni
 

On 02/02/2022 22:24:26+0000, Glimsdale, Nathan wrote:
I see this intermittently when building "aws-c-iot" directly. 3 of 5
attempts so far. The network restriction is also affected some of the
dependencies consistently.
Ah, interesting. Will you be tracking that on your side or should I open
a bug on bugzilla.yp.org?


On 2/2/22, 2:11 PM, "Richard Elberger" <elberger@...> wrote:

Hi Alexandre, I'm adding Nathan to see if there's anything he can see.

I am not familiar with the warning at all, it looks new to me. Which
build step has the commit hash for the repos we're checking out? That
way I can compare apples-apples.

-- rich

On 2/2/22 16:53, Alexandre Belloni wrote:
> CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe.
>
>
>
> Hello,
>
> It's been a few days that I'm struggling to understand what happened in
> this build. I've been looking at the fetcher changes, meta-aws changes
> but couldn't find anything relevant.
>
> https://autobuilder.yoctoproject.org/typhoon/#/builders/122/builds/713/steps/16/logs/warnings
>
> I would take any hint that you have ;)
>
> Regards,
>
> --
> Alexandre Belloni, co-owner and COO, Bootlin
> Embedded Linux and Kernel engineering
> https://bootlin.com


--
Richard Elberger
Principal Technologist
AWS IoT

+1 646.927.6897
+1 203.942.8039 – mobile
elberger@... – email

--
Alexandre Belloni, co-owner and COO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com


Re: meta-aws warnings: SRCREV_FORMAT variable must be set when multiple SCMs are used

Alexandre Belloni
 

On 02/02/2022 17:11:14-0500, Richard Elberger wrote:
Hi Alexandre, I'm adding Nathan to see if there's anything he can see.

I am not familiar with the warning at all, it looks new to me. Which build
step has the commit hash for the repos we're checking out? That way I can
compare apples-apples.
This was:

meta
meta-poky
meta-yocto-bsp = "master:9f8e74668730e6eb3eb44df187a40d0a0f8c7b0a"
meta-oe
meta-python
meta-networking = "master:a558d51fecda3e66ace21d02b57ab61bf122fdc1"
meta-aws = "master:8893e0cd4c0981eeda941eaa9ad2eb9359670502"

-- rich

On 2/2/22 16:53, Alexandre Belloni wrote:
CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe.



Hello,

It's been a few days that I'm struggling to understand what happened in
this build. I've been looking at the fetcher changes, meta-aws changes
but couldn't find anything relevant.

https://autobuilder.yoctoproject.org/typhoon/#/builders/122/builds/713/steps/16/logs/warnings

I would take any hint that you have ;)

Regards,

--
Alexandre Belloni, co-owner and COO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

--
Richard Elberger
Principal Technologist
AWS IoT

+1 646.927.6897
+1 203.942.8039 – mobile
elberger@... – email
--
Alexandre Belloni, co-owner and COO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com


Re: meta-aws warnings: SRCREV_FORMAT variable must be set when multiple SCMs are used

elberger@...
 

Hi Alexandre, I'm adding Nathan to see if there's anything he can see.

I am not familiar with the warning at all, it looks new to me. Which build step has the commit hash for the repos we're checking out? That way I can compare apples-apples.

-- rich

On 2/2/22 16:53, Alexandre Belloni wrote:
CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe.



Hello,

It's been a few days that I'm struggling to understand what happened in
this build. I've been looking at the fetcher changes, meta-aws changes
but couldn't find anything relevant.

https://autobuilder.yoctoproject.org/typhoon/#/builders/122/builds/713/steps/16/logs/warnings

I would take any hint that you have ;)

Regards,

--
Alexandre Belloni, co-owner and COO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
--
Richard Elberger
Principal Technologist
AWS IoT

+1 646.927.6897
+1 203.942.8039 – mobile
elberger@... – email


meta-aws warnings: SRCREV_FORMAT variable must be set when multiple SCMs are used

Alexandre Belloni
 

Hello,

It's been a few days that I'm struggling to understand what happened in
this build. I've been looking at the fetcher changes, meta-aws changes
but couldn't find anything relevant.

https://autobuilder.yoctoproject.org/typhoon/#/builders/122/builds/713/steps/16/logs/warnings

I would take any hint that you have ;)

Regards,

--
Alexandre Belloni, co-owner and COO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com


SWAT Rotation schedule

Alexandre Belloni
 

Hello everyone,

Following the winter break, I would like to organize SWAT duty rotation.
I've again prepared a schedule so it is easier for each one of you to
plan for SWAT duty.

Please check the table and let me know if you are not available for the
selected week. SWAT duty will be from Friday to Thursday and the goal is
to triage all the failures on swatbot before the weekly triage call
happening at 2:30pm UTC.

At the current failure rate, I would expect the task to take between a
half a day and a day of work.

┌───────────────────────────────┬──────┬────────────┐
│ │ Week │ Start │
├───────────────────────────────┼──────┼────────────┤
│ Anibal Limon │ 4 │ 28/01/2022 │
│ Saul Wold │ 5 │ 04/02/2022 │
│ Alejandro Hernandez Samaniego │ 6 │ 11/02/2022 │
│ Oleksiy Obitotskyy │ 7 │ 18/02/2022 │
│ Naveen Saini │ 8 │ 25/02/2022 │
│ Paul Eggleton │ 9 │ 04/03/2022 │
│ Christopher Larson │ 10 │ 11/03/2022 │
│ Jon Mason │ 11 │ 18/03/2022 │
│ Lee Chee Yang │ 12 │ 25/03/2022 │
│ Minjae Kim │ 13 │ 01/04/2022 │
│ Jaga │ 14 │ 08/04/2022 │
│ Leo Sandoval │ 15 │ 15/04/2022 │
│ Ross Burton │ 16 │ 22/04/2022 │
│ Valerii Chernous │ 17 │ 29/04/2022 │
└───────────────────────────────┴──────┴────────────┘

Anibal you would be the next one on the list, starting this Friday,
can you confirm you are available?

There are currently 2 failures to triage on swatbot, I'm going to take
care of those.

When filling in bugs or bug comments, please also add the worker name
and whether it has an SSD. Currently, we have:

┌──────────────────┬─────────┐
│ alma8-ty-1 │ SSD │
│ alma8-ty-2 │ SSD │
│ debian11-ty-3 │ SSD │
│ fedora35-ty-1 │ SSD │
│ fedora35-ty-2 │ SSD │
│ opensuse153-ty-1 │ SSD │
│ stream8-ty-1 │ SSD │
│ ubuntu2004-ty-1 │ SSD │
│ ubuntu2110-ty-2 │ SSD │
│ centos7-ty-4 │ Non-SSD │
│ debian11-ty-1 │ Non-SSD │
│ fedora34-ty-1 │ Non-SSD │
│ perf-centos7 │ Non-SSD │
│ perf-ubuntu1604 │ Non-SSD │
│ tumbleweed-ty-3 │ Non-SSD │
│ ubuntu1604-ty-1 │ Non-SSD │
│ ubuntu1804-arm-1 │ Non-SSD │
│ ubuntu1804-ty-3 │ Non-SSD │
│ ubuntu2004-arm-1 │ Non-SSD │
│ centos8-ty-1 │ Non-SSD │
│ centos8-ty-2 │ Non-SSD │
│ debian10-ty-1 │ Non-SSD │
│ debian11-ty-2 │ Non-SSD │
│ debian9-ty-2 │ Non-SSD │
│ opensuse154-ty-1 │ Non-SSD │
└──────────────────┴─────────┘

Thanks!

--
Alexandre Belloni, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com


Re: New warning in meta-aws

Alexandre Belloni
 

On 17/01/2022 23:05:18+0000, Elberger, Richard wrote:
Thank you - it is in our current sprint to fix, it was noted last week when the distutils warnings across openembedded really alarmed us.
Sure, it was kind of hidden in all those meta-oe warnings so I wanted to
make sure you knew about it.

It’s a simple fix so I don’t expect an extended CR :D

Sent from my iPhone

On Jan 17, 2022, at 17:44, Alexandre Belloni <alexandre.belloni@...> wrote:

CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe.



Hello Richard,

There is a new warning in meta-aws:

WARNING: /home/pokybuild/yocto-worker/meta-aws/build/meta-aws/recipes-iot/aws-iot-securetunneling-localproxy/aws-iot-securetunneling-localproxy_git.bb: URL: git://git@.../aws-samples/aws-iot-securetunneling-localproxy.git;protocol=ssh does not set any branch parameter. The future default branch used by tools and repositories is uncertain and we will therefore soon require this is set in all git urls.

I believe this is the only affected recipe.

--
Alexandre Belloni, co-owner and COO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
--
Alexandre Belloni, co-owner and COO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com


Re: New warning in meta-aws

elberger@...
 

Thank you - it is in our current sprint to fix, it was noted last week when the distutils warnings across openembedded really alarmed us.

It’s a simple fix so I don’t expect an extended CR :D

On Jan 17, 2022, at 17:44, Alexandre Belloni <alexandre.belloni@...> wrote:

CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe.



Hello Richard,

There is a new warning in meta-aws:

WARNING: /home/pokybuild/yocto-worker/meta-aws/build/meta-aws/recipes-iot/aws-iot-securetunneling-localproxy/aws-iot-securetunneling-localproxy_git.bb: URL: git://git@.../aws-samples/aws-iot-securetunneling-localproxy.git;protocol=ssh does not set any branch parameter. The future default branch used by tools and repositories is uncertain and we will therefore soon require this is set in all git urls.

I believe this is the only affected recipe.

--
Alexandre Belloni, co-owner and COO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com


New warning in meta-aws

Alexandre Belloni
 

Hello Richard,

There is a new warning in meta-aws:

WARNING: /home/pokybuild/yocto-worker/meta-aws/build/meta-aws/recipes-iot/aws-iot-securetunneling-localproxy/aws-iot-securetunneling-localproxy_git.bb: URL: git://git@.../aws-samples/aws-iot-securetunneling-localproxy.git;protocol=ssh does not set any branch parameter. The future default branch used by tools and repositories is uncertain and we will therefore soon require this is set in all git urls.

I believe this is the only affected recipe.

--
Alexandre Belloni, co-owner and COO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

41 - 60 of 287