Date
1 - 11 of 11
meta-aws warnings: SRCREV_FORMAT variable must be set when multiple SCMs are used
|
Alexandre Belloni
Hello,
It's been a few days that I'm struggling to understand what happened in this build. I've been looking at the fetcher changes, meta-aws changes but couldn't find anything relevant. https://autobuilder.yoctoproject.org/typhoon/#/builders/122/builds/713/steps/16/logs/warnings I would take any hint that you have ;) Regards, -- Alexandre Belloni, co-owner and COO, Bootlin Embedded Linux and Kernel engineering https://bootlin.com |
|
|
|
elberger@...
Hi Alexandre, I'm adding Nathan to see if there's anything he can see.
toggle quoted message
Show quoted text
I am not familiar with the warning at all, it looks new to me. Which build step has the commit hash for the repos we're checking out? That way I can compare apples-apples. -- rich On 2/2/22 16:53, Alexandre Belloni wrote:
CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe. --
Richard Elberger Principal Technologist AWS IoT +1 646.927.6897 +1 203.942.8039 – mobile elberger@... – email |
|
|
|
Alexandre Belloni
On 02/02/2022 17:11:14-0500, Richard Elberger wrote:
Hi Alexandre, I'm adding Nathan to see if there's anything he can see.This was: meta meta-poky meta-yocto-bsp = "master:9f8e74668730e6eb3eb44df187a40d0a0f8c7b0a" meta-oe meta-python meta-networking = "master:a558d51fecda3e66ace21d02b57ab61bf122fdc1" meta-aws = "master:8893e0cd4c0981eeda941eaa9ad2eb9359670502" -- rich-- Alexandre Belloni, co-owner and COO, Bootlin Embedded Linux and Kernel engineering https://bootlin.com |
|
|
|
Alexandre Belloni
On 02/02/2022 22:24:26+0000, Glimsdale, Nathan wrote:
I see this intermittently when building "aws-c-iot" directly. 3 of 5Ah, interesting. Will you be tracking that on your side or should I open a bug on bugzilla.yp.org? On 2/2/22, 2:11 PM, "Richard Elberger" <elberger@...> wrote:-- Alexandre Belloni, co-owner and COO, Bootlin Embedded Linux and Kernel engineering https://bootlin.com |
|
|
|
Richard Purdie
On Wed, 2022-02-02 at 23:54 +0100, Alexandre Belloni via lists.yoctoproject.org
wrote: On 02/02/2022 22:24:26+0000, Glimsdale, Nathan wrote:I did just have a look at the aws-c-iot recipes:I see this intermittently when building "aws-c-iot" directly. 3 of 5Ah, interesting. Will you be tracking that on your side or should I open https://git.yoctoproject.org/meta-aws/tree/recipes-sdk/aws-c-iot/aws-c-iot_0.0.2.bb https://git.yoctoproject.org/meta-aws/tree/recipes-sdk/aws-c-iot/aws-c-iot_0.0.7.bb and I notice they use git urls with tags. This does not do what people expect since bitbake resolves the tag each time over the network since people can change the tags (and do!). I suspect that is why it is this recipe which is failing. I'm not sure why that is sometimes showing a warning though. Cheers, Richard |
|
|
|
Denys Dmytriyenko
On Wed, Feb 02, 2022 at 11:04:57PM +0000, Richard Purdie wrote:
On Wed, 2022-02-02 at 23:54 +0100, Alexandre Belloni via lists.yoctoproject.orgOuch, no SRCREVs? I'm guessing we need to extend yocto-check-layer with another test? :) -- Denys |
|
|
|
elberger@...
The choice to use tags is a deliberate one.
toggle quoted message
Show quoted text
For all AWS repos that meta-aws builds recipes we use release tagging to align with a specific release. While tags can move (aka the old school CVS floating label practice which is very frowned upon except for noting "stable") they do not for these repos. Of course, there is the exception when a release is about to be made and a late breaking change must be added to the release. Then the tag shifts to meet that demand. This happens very rarely and is not a contributor to the general concern of repo maintainers implementing the floating tag/label practice and the recipe using that tag. Further, I don't think such a check in yocto-check-layer is appropriate. Rather, if a recipe author knows that a specific CM practice is in place for particular repos then have the opportunity to tell the various stages "don't check, I know what I'm doing". If YP/OE really is against using the tag mechanism then I suggest we deprecate it. I have found it very nice for maintainership when we know exactly how the development team handles the CM. -----Original Message-----
From: Denys Dmytriyenko <denis@...> Sent: Wednesday, February 2, 2022 6:17 PM To: swat@... Cc: Glimsdale, Nathan <glimsdal@...>; Elberger, Richard <elberger@...> Subject: RE: [EXTERNAL] [swat] meta-aws warnings: SRCREV_FORMAT variable must be set when multiple SCMs are used CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe. On Wed, Feb 02, 2022 at 11:04:57PM +0000, Richard Purdie wrote: On Wed, 2022-02-02 at 23:54 +0100, Alexandre Belloni viaOuch, no SRCREVs? I'm guessing we need to extend yocto-check-layer with another test? :) -- Denys |
|
|
|
Alexandre Belloni
On 03/02/2022 14:02:54+0000, elberger via lists.yoctoproject.org wrote:
The choice to use tags is a deliberate one.The main drawback is that then you are not able to build with BB_NO_NETWORK = 1 and so there is no way to ensure you have locally all the code you need to reproduce the build in case Amazon gets out of business one day ;) -----Original Message------- Alexandre Belloni, co-owner and COO, Bootlin Embedded Linux and Kernel engineering https://bootlin.com |
|
|
|
So the main error is:
toggle quoted message
Show quoted text
aws-c-auth-0.6.1-r0 do_packagedata_setscene: ExpansionError('SRCPV', '${@bb.fetch2.get_srcrev(d)}', FetchError('The SRCREV_FORMAT variable must be set when multiple SCMs are used.\nThe SCMs are:\ngit://github.com/awslabs/aws-c-common.git;protocol=https;branch=main;tag=v0.6.8;destsuffix=/home/pokybuild/yocto-worker/meta-aws/build/build/tmp/work/cortexa57-poky-linux/aws-c-auth/0.6.1-r0/git/aws-c-common;name=common\ngit://github.com/awslabs/aws-c-auth.git;protocol=https;branch=main;tag=v0.6.1;destsuffix=/home/pokybuild/yocto-worker/meta-aws/build/build/tmp/work/cortexa57-poky-linux/aws-c-auth/0.6.1-r0/git/aws-c-auth;name=auth', None)) Let's try to dissect: The SRCREV_FORMAT variable must be set when multiple SCMs are used. The SCMs are: git://github.com/awslabs/aws-c-common.git;protocol=https;branch=main;tag=v0.6.8;destsuffix=/home/pokybuild/yocto-worker/meta-aws/build/build/tmp/work/cortexa57-poky-linux/aws-c-auth/0.6.1-r0/git/aws-c-common;name=common git://github.com/awslabs/aws-c-auth.git;protocol=https;branch=main;tag=v0.6.1;destsuffix=/home/pokybuild/yocto-worker/meta-aws/build/build/tmp/work/cortexa57-poky-linux/aws-c-auth/0.6.1-r0/git/aws-c-auth;name=auth I read this like: - we're using two git:// url's in the SRC_URI - bitbake needs one single hash value for SRCREV for the recipe to be 'uniq' - we cannot decide what to use as SRCREV based on this SRC_URI (with multiple git:// entries) I found in meta-arm: SRC_URI = "git://git.trustedfirmware.org/TF-A/trusted-firmware-a.git;protocol=https;name=tfa" # Use TF-A for version SRCREV_FORMAT = "tfa" # ^^^^^^^^^^^^^^^^^^^^^^^^ !!!!!!!! tfa = name from above # TF-A v2.1 SRCREV_tfa = "e1286bdb968ee74fc52f96cf303a4218e1ae2950" # # mbed TLS source # Those are used in trusted-firmware-a.inc if TFA_MBEDTLS is set to 1 SRC_URI_MBEDTLS = "git://github.com/ARMmbed/mbedtls.git;name=mbedtls;protocol=https;destsuffix=git/mbedtls;branch=mbedtls-2.16" SRCREV_mbedtls = "d81c11b8ab61fd5b2da8133aa73c5fe33a0633eb" So to apply this to meta-aws, you'd have to pick one which should be the canonical SRCREV of the two for this recipe. E.g. (untested) aws-c-auth_0.6.1.bb : [...] TAG ?= "v${PV}" TAG_COMMON ?= "v0.6.8" SRC_URI = "git://github.com/awslabs/aws-c-common.git;protocol=https;branch=${BRANCH};tag=${TAG_COMMON};destsuffix=${S}/aws-c-common;name=common \ git://github.com/awslabs/aws-c-auth.git;protocol=https;branch=${BRANCH};tag=${TAG};destsuffix=${S}/aws-c-auth;name=auth \ " S= "${WORKDIR}/git" BRANCH ?= "main" # _____________________ SRCREV_FORMAT = "auth" # ^^^^^^^^^^^^^^^^^^^^ But the main question here is: why do you need aws-c-common in SRC_URI ? . aws-c-common has its own recipe, so if it installs the headers properly it can be in DEPENDS and will be pulled automagically. The optimization wrt tag and network / offline that Denis mentioned is important imho as well ! It is really better to use a SRCREV. But yes, it might affect workflow and means bumping SRCREV's. TLDR: a) If possible make aws-c-common a DEPENDS in the auth recipe b) if above does not work, try SRCREV_FORMAT = "auth" (other recipes likewise) c) tag should be optimized to a SRCREV to avoid the enforced network fetch HTH. Best, Jan-Simon ------ Jan-Simon Möller AGL Release Manager The Linux Foundation Visit us at: www.automotivegradelinux.org lists.automotivelinux.org www.linuxfoundation.org Best regards, Jan-Simon ------ Jan-Simon Möller AGL Release Manager The Linux Foundation Visit us at: www.automotivegradelinux.org lists.automotivelinux.org www.linuxfoundation.org On Thu, Feb 3, 2022 at 3:12 PM Alexandre Belloni via lists.yoctoproject.org <alexandre.belloni=bootlin.com@...> wrote:
|
|
|
|
Richard Purdie
On Thu, 2022-02-03 at 14:02 +0000, elberger via lists.yoctoproject.org wrote:
The choice to use tags is a deliberate one.It may be a deliberate choice but I suspect there are some issues you aren't considering, specifically: a) it breaks anyone doing offline builds b) it means source archives don't function correct c) it means your code isn't reproducible For all AWS repos that meta-aws builds recipes we use release tagging to alignYou are basically making my point here for me. Our metadata is meant to describe a specific build exactly, at least "out the box" without changing things. By using tags and allowing them to shift even if it is "very rarely", it isn't explicit and therefore isn't reproducible. Further, I don't think such a check in yocto-check-layer is appropriate.I'm going to disagree, let me explain why. Rather, if a recipe author knows that a specific CM practice is in place forI have no issue with anyone using tags if they want to. Most people use them not realising there are serious consequences though (as listed above). I therefore think this is exactly the kind of thing yocto-check-layer should be flagging up. We don't want to "ban" them outright but we do want our best practises to say not to use them. If YP/OE really is against using the tag mechanism then I suggest weI agree it is "nice" for maintainership but it has several implications which at the very least need to be thought through. Are you happy that meta-aws isn't reproducible, buildable offline and can't work against source archives? Cheers, Richard |
|
|
|
elberger@...
This makes a lot of sense to me and it's clear now. It didn't come across this way in prior responses. Thank you for taking the time to put this together.
toggle quoted message
Show quoted text
We can't remove aws-c-common from the sources since ... it's a build quirk ... virtually all repos require it to be cloned in the source tree context. Fixing this is going to take quite some time - will need to get the work scheduled. Hopefully will get it done over the next week. -----Original Message-----
From: Jan-Simon Moeller <jsmoeller@...> Sent: Thursday, February 3, 2022 9:41 AM To: Elberger, Richard <elberger@...> Subject: FW: [EXTERNAL] [swat] meta-aws warnings: SRCREV_FORMAT variable must be set when multiple SCMs are used CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe. Forgot to add all CC back ... Best regards, Jan-Simon ------ Jan-Simon Möller AGL Release Manager The Linux Foundation Visit us at: www.automotivegradelinux.org lists.automotivelinux.org www.linuxfoundation.org ---------- Forwarded message --------- From: Jan-Simon Moeller <jsmoeller@...> Date: Thu, Feb 3, 2022 at 3:40 PM Subject: Re: [swat] meta-aws warnings: SRCREV_FORMAT variable must be set when multiple SCMs are used To: <swat@...> Cc: Denys Dmytriyenko <denis@...>, Glimsdale, Nathan <glimsdal@...> So the main error is: aws-c-auth-0.6.1-r0 do_packagedata_setscene: ExpansionError('SRCPV', '${@bb.fetch2.get_srcrev(d)}', FetchError('The SRCREV_FORMAT variable must be set when multiple SCMs are used.\nThe SCMs are:\ngit://github.com/awslabs/aws-c-common.git;protocol=https;branch=main;tag=v0.6.8;destsuffix=/home/pokybuild/yocto-worker/meta-aws/build/build/tmp/work/cortexa57-poky-linux/aws-c-auth/0.6.1-r0/git/aws-c-common;name=common\ngit://github.com/awslabs/aws-c-auth.git;protocol=https;branch=main;tag=v0.6.1;destsuffix=/home/pokybuild/yocto-worker/meta-aws/build/build/tmp/work/cortexa57-poky-linux/aws-c-auth/0.6.1-r0/git/aws-c-auth;name=auth', None)) Let's try to dissect: The SRCREV_FORMAT variable must be set when multiple SCMs are used. The SCMs are: git://github.com/awslabs/aws-c-common.git;protocol=https;branch=main;tag=v0.6.8;destsuffix=/home/pokybuild/yocto-worker/meta-aws/build/build/tmp/work/cortexa57-poky-linux/aws-c-auth/0.6.1-r0/git/aws-c-common;name=common git://github.com/awslabs/aws-c-auth.git;protocol=https;branch=main;tag=v0.6.1;destsuffix=/home/pokybuild/yocto-worker/meta-aws/build/build/tmp/work/cortexa57-poky-linux/aws-c-auth/0.6.1-r0/git/aws-c-auth;name=auth I read this like: - we're using two git:// url's in the SRC_URI - bitbake needs one single hash value for SRCREV for the recipe to be 'uniq' - we cannot decide what to use as SRCREV based on this SRC_URI (with multiple git:// entries) I found in meta-arm: SRC_URI = "git://git.trustedfirmware.org/TF-A/trusted-firmware-a.git;protocol=https;name=tfa" # Use TF-A for version SRCREV_FORMAT = "tfa" # ^^^^^^^^^^^^^^^^^^^^^^^^ !!!!!!!! tfa = name from above # TF-A v2.1 SRCREV_tfa = "e1286bdb968ee74fc52f96cf303a4218e1ae2950" # # mbed TLS source # Those are used in trusted-firmware-a.inc if TFA_MBEDTLS is set to 1 SRC_URI_MBEDTLS = "git://github.com/ARMmbed/mbedtls.git;name=mbedtls;protocol=https;destsuffix=git/mbedtls;branch=mbedtls-2.16" SRCREV_mbedtls = "d81c11b8ab61fd5b2da8133aa73c5fe33a0633eb" So to apply this to meta-aws, you'd have to pick one which should be the canonical SRCREV of the two for this recipe. E.g. (untested) aws-c-auth_0.6.1.bb : [...] TAG ?= "v${PV}" TAG_COMMON ?= "v0.6.8" SRC_URI = "git://github.com/awslabs/aws-c-common.git;protocol=https;branch=${BRANCH};tag=${TAG_COMMON};destsuffix=${S}/aws-c-common;name=common \ git://github.com/awslabs/aws-c-auth.git;protocol=https;branch=${BRANCH};tag=${TAG};destsuffix=${S}/aws-c-auth;name=auth \ " S= "${WORKDIR}/git" BRANCH ?= "main" # _____________________ SRCREV_FORMAT = "auth" # ^^^^^^^^^^^^^^^^^^^^ But the main question here is: why do you need aws-c-common in SRC_URI ? . aws-c-common has its own recipe, so if it installs the headers properly it can be in DEPENDS and will be pulled automagically. The optimization wrt tag and network / offline that Denis mentioned is important imho as well ! It is really better to use a SRCREV. But yes, it might affect workflow and means bumping SRCREV's. TLDR: a) If possible make aws-c-common a DEPENDS in the auth recipe b) if above does not work, try SRCREV_FORMAT = "auth" (other recipes likewise) c) tag should be optimized to a SRCREV to avoid the enforced network fetch HTH. Best, Jan-Simon ------ Jan-Simon Möller AGL Release Manager The Linux Foundation Visit us at: www.automotivegradelinux.org lists.automotivelinux.org www.linuxfoundation.org Best regards, Jan-Simon ------ Jan-Simon Möller AGL Release Manager The Linux Foundation Visit us at: www.automotivegradelinux.org lists.automotivelinux.org www.linuxfoundation.org On Thu, Feb 3, 2022 at 3:12 PM Alexandre Belloni via lists.yoctoproject.org <alexandre.belloni=bootlin.com@...> wrote:
|
|
|