Re: meta-aws warnings: SRCREV_FORMAT variable must be set when multiple SCMs are used


elberger@...
 

This makes a lot of sense to me and it's clear now. It didn't come across this way in prior responses. Thank you for taking the time to put this together.

We can't remove aws-c-common from the sources since ... it's a build quirk ... virtually all repos require it to be cloned in the source tree context.

Fixing this is going to take quite some time - will need to get the work scheduled. Hopefully will get it done over the next week.

-----Original Message-----
From: Jan-Simon Moeller <jsmoeller@...>
Sent: Thursday, February 3, 2022 9:41 AM
To: Elberger, Richard <elberger@...>
Subject: FW: [EXTERNAL] [swat] meta-aws warnings: SRCREV_FORMAT variable must be set when multiple SCMs are used

CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe.



Forgot to add all CC back ...
Best regards,
Jan-Simon

------
Jan-Simon Möller
AGL Release Manager
The Linux Foundation

Visit us at:
www.automotivegradelinux.org
lists.automotivelinux.org
www.linuxfoundation.org


---------- Forwarded message ---------
From: Jan-Simon Moeller <jsmoeller@...>
Date: Thu, Feb 3, 2022 at 3:40 PM
Subject: Re: [swat] meta-aws warnings: SRCREV_FORMAT variable must be set when multiple SCMs are used
To: <swat@...>
Cc: Denys Dmytriyenko <denis@...>, Glimsdale, Nathan <glimsdal@...>


So the main error is:
aws-c-auth-0.6.1-r0 do_packagedata_setscene: ExpansionError('SRCPV', '${@bb.fetch2.get_srcrev(d)}', FetchError('The SRCREV_FORMAT variable must be set when multiple SCMs are used.\nThe SCMs are:\ngit://github.com/awslabs/aws-c-common.git;protocol=https;branch=main;tag=v0.6.8;destsuffix=/home/pokybuild/yocto-worker/meta-aws/build/build/tmp/work/cortexa57-poky-linux/aws-c-auth/0.6.1-r0/git/aws-c-common;name=common\ngit://github.com/awslabs/aws-c-auth.git;protocol=https;branch=main;tag=v0.6.1;destsuffix=/home/pokybuild/yocto-worker/meta-aws/build/build/tmp/work/cortexa57-poky-linux/aws-c-auth/0.6.1-r0/git/aws-c-auth;name=auth',
None))


Let's try to dissect:

The SRCREV_FORMAT variable must be set when multiple SCMs are used.
The SCMs are:
git://github.com/awslabs/aws-c-common.git;protocol=https;branch=main;tag=v0.6.8;destsuffix=/home/pokybuild/yocto-worker/meta-aws/build/build/tmp/work/cortexa57-poky-linux/aws-c-auth/0.6.1-r0/git/aws-c-common;name=common
git://github.com/awslabs/aws-c-auth.git;protocol=https;branch=main;tag=v0.6.1;destsuffix=/home/pokybuild/yocto-worker/meta-aws/build/build/tmp/work/cortexa57-poky-linux/aws-c-auth/0.6.1-r0/git/aws-c-auth;name=auth


I read this like:
- we're using two git:// url's in the SRC_URI
- bitbake needs one single hash value for SRCREV for the recipe to be 'uniq'
- we cannot decide what to use as SRCREV based on this SRC_URI (with multiple git:// entries)

I found in meta-arm:
SRC_URI = "git://git.trustedfirmware.org/TF-A/trusted-firmware-a.git;protocol=https;name=tfa"
# Use TF-A for version
SRCREV_FORMAT = "tfa"
# ^^^^^^^^^^^^^^^^^^^^^^^^ !!!!!!!! tfa = name from above # TF-A v2.1 SRCREV_tfa = "e1286bdb968ee74fc52f96cf303a4218e1ae2950"
#
# mbed TLS source
# Those are used in trusted-firmware-a.inc if TFA_MBEDTLS is set to 1 SRC_URI_MBEDTLS = "git://github.com/ARMmbed/mbedtls.git;name=mbedtls;protocol=https;destsuffix=git/mbedtls;branch=mbedtls-2.16"
SRCREV_mbedtls = "d81c11b8ab61fd5b2da8133aa73c5fe33a0633eb"


So to apply this to meta-aws, you'd have to pick one which should be the canonical SRCREV of the two for this recipe.
E.g. (untested)

aws-c-auth_0.6.1.bb :
[...]
TAG ?= "v${PV}"
TAG_COMMON ?= "v0.6.8"
SRC_URI = "git://github.com/awslabs/aws-c-common.git;protocol=https;branch=${BRANCH};tag=${TAG_COMMON};destsuffix=${S}/aws-c-common;name=common
\
git://github.com/awslabs/aws-c-auth.git;protocol=https;branch=${BRANCH};tag=${TAG};destsuffix=${S}/aws-c-auth;name=auth
\
"
S= "${WORKDIR}/git"
BRANCH ?= "main"
# _____________________
SRCREV_FORMAT = "auth"
# ^^^^^^^^^^^^^^^^^^^^



But the main question here is: why do you need aws-c-common in SRC_URI ? . aws-c-common has its own recipe, so if it installs the headers properly it can be in DEPENDS and will be pulled automagically.

The optimization wrt tag and network / offline that Denis mentioned is important imho as well ! It is really better to use a SRCREV. But yes, it might affect workflow and means bumping SRCREV's.

TLDR:
a) If possible make aws-c-common a DEPENDS in the auth recipe
b) if above does not work, try SRCREV_FORMAT = "auth" (other recipes likewise)
c) tag should be optimized to a SRCREV to avoid the enforced network fetch


HTH.
Best,
Jan-Simon

------
Jan-Simon Möller
AGL Release Manager
The Linux Foundation

Visit us at:
www.automotivegradelinux.org
lists.automotivelinux.org
www.linuxfoundation.org

Best regards,
Jan-Simon

------
Jan-Simon Möller
AGL Release Manager
The Linux Foundation

Visit us at:
www.automotivegradelinux.org
lists.automotivelinux.org
www.linuxfoundation.org


On Thu, Feb 3, 2022 at 3:12 PM Alexandre Belloni via lists.yoctoproject.org <alexandre.belloni=bootlin.com@...> wrote:

On 03/02/2022 14:02:54+0000, elberger via lists.yoctoproject.org wrote:
The choice to use tags is a deliberate one.

For all AWS repos that meta-aws builds recipes we use release tagging to align with a specific release. While tags can move (aka the old school CVS floating label practice which is very frowned upon except for noting "stable") they do not for these repos. Of course, there is the exception when a release is about to be made and a late breaking change must be added to the release. Then the tag shifts to meet that demand. This happens very rarely and is not a contributor to the general concern of repo maintainers implementing the floating tag/label practice and the recipe using that tag.

Further, I don't think such a check in yocto-check-layer is appropriate. Rather, if a recipe author knows that a specific CM practice is in place for particular repos then have the opportunity to tell the various stages "don't check, I know what I'm doing". If YP/OE really is against using the tag mechanism then I suggest we deprecate it. I have found it very nice for maintainership when we know exactly how the development team handles the CM.
The main drawback is that then you are not able to build with
BB_NO_NETWORK = 1 and so there is no way to ensure you have locally
all the code you need to reproduce the build in case Amazon gets out
of business one day ;)

-----Original Message-----
From: Denys Dmytriyenko <denis@...>
Sent: Wednesday, February 2, 2022 6:17 PM
To: swat@...
Cc: Glimsdale, Nathan <glimsdal@...>; Elberger, Richard
<elberger@...>
Subject: RE: [EXTERNAL] [swat] meta-aws warnings: SRCREV_FORMAT
variable must be set when multiple SCMs are used

CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe.



On Wed, Feb 02, 2022 at 11:04:57PM +0000, Richard Purdie wrote:
On Wed, 2022-02-02 at 23:54 +0100, Alexandre Belloni via
lists.yoctoproject.org
wrote:
On 02/02/2022 22:24:26+0000, Glimsdale, Nathan wrote:
I see this intermittently when building "aws-c-iot" directly.
3 of
5 attempts so far. The network restriction is also affected
some of the dependencies consistently.
Ah, interesting. Will you be tracking that on your side or
should I open a bug on bugzilla.yp.org?
I did just have a look at the aws-c-iot recipes:

https://git.yoctoproject.org/meta-aws/tree/recipes-sdk/aws-c-iot/a
ws-c
-iot_0.0.2.bb
https://git.yoctoproject.org/meta-aws/tree/recipes-sdk/aws-c-iot/a
ws-c
-iot_0.0.7.bb

and I notice they use git urls with tags.

This does not do what people expect since bitbake resolves the tag
each time over the network since people can change the tags (and do!).
I suspect that is why it is this recipe which is failing.
Ouch, no SRCREVs?

I'm guessing we need to extend yocto-check-layer with another test?
:)

--
Denys




--
Alexandre Belloni, co-owner and COO, Bootlin Embedded Linux and Kernel
engineering https://bootlin.com




Join swat@lists.yoctoproject.org to automatically receive all group messages.