Date
1 - 11 of 11
meta-aws warnings: SRCREV_FORMAT variable must be set when multiple SCMs are used
Alexandre Belloni
Hello,
It's been a few days that I'm struggling to understand what happened in
this build. I've been looking at the fetcher changes, meta-aws changes
but couldn't find anything relevant.
https://autobuilder.yoctoproject.org/typhoon/#/builders/122/builds/713/steps/16/logs/warnings
I would take any hint that you have ;)
Regards,
--
Alexandre Belloni, co-owner and COO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
It's been a few days that I'm struggling to understand what happened in
this build. I've been looking at the fetcher changes, meta-aws changes
but couldn't find anything relevant.
https://autobuilder.yoctoproject.org/typhoon/#/builders/122/builds/713/steps/16/logs/warnings
I would take any hint that you have ;)
Regards,
--
Alexandre Belloni, co-owner and COO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
elberger@...
Hi Alexandre, I'm adding Nathan to see if there's anything he can see.
I am not familiar with the warning at all, it looks new to me. Which build step has the commit hash for the repos we're checking out? That way I can compare apples-apples.
-- rich
toggle quoted message
Show quoted text
I am not familiar with the warning at all, it looks new to me. Which build step has the commit hash for the repos we're checking out? That way I can compare apples-apples.
-- rich
On 2/2/22 16:53, Alexandre Belloni wrote:
CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe.
Hello,
It's been a few days that I'm struggling to understand what happened in
this build. I've been looking at the fetcher changes, meta-aws changes
but couldn't find anything relevant.
https://autobuilder.yoctoproject.org/typhoon/#/builders/122/builds/713/steps/16/logs/warnings
I would take any hint that you have ;)
Regards,
--
Alexandre Belloni, co-owner and COO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
--
Richard Elberger
Principal Technologist
AWS IoT
+1 646.927.6897
+1 203.942.8039 – mobile
elberger@... – email
Richard Elberger
Principal Technologist
AWS IoT
+1 646.927.6897
+1 203.942.8039 – mobile
elberger@... – email
Alexandre Belloni
On 02/02/2022 17:11:14-0500, Richard Elberger wrote:
meta
meta-poky
meta-yocto-bsp = "master:9f8e74668730e6eb3eb44df187a40d0a0f8c7b0a"
meta-oe
meta-python
meta-networking = "master:a558d51fecda3e66ace21d02b57ab61bf122fdc1"
meta-aws = "master:8893e0cd4c0981eeda941eaa9ad2eb9359670502"
Alexandre Belloni, co-owner and COO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
Hi Alexandre, I'm adding Nathan to see if there's anything he can see.This was:
I am not familiar with the warning at all, it looks new to me. Which build
step has the commit hash for the repos we're checking out? That way I can
compare apples-apples.
meta
meta-poky
meta-yocto-bsp = "master:9f8e74668730e6eb3eb44df187a40d0a0f8c7b0a"
meta-oe
meta-python
meta-networking = "master:a558d51fecda3e66ace21d02b57ab61bf122fdc1"
meta-aws = "master:8893e0cd4c0981eeda941eaa9ad2eb9359670502"
-- rich--
On 2/2/22 16:53, Alexandre Belloni wrote:CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe.
Hello,
It's been a few days that I'm struggling to understand what happened in
this build. I've been looking at the fetcher changes, meta-aws changes
but couldn't find anything relevant.
https://autobuilder.yoctoproject.org/typhoon/#/builders/122/builds/713/steps/16/logs/warnings
I would take any hint that you have ;)
Regards,
--
Alexandre Belloni, co-owner and COO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
--
Richard Elberger
Principal Technologist
AWS IoT
+1 646.927.6897
+1 203.942.8039 – mobile
elberger@... – email
Alexandre Belloni, co-owner and COO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
Alexandre Belloni
On 02/02/2022 22:24:26+0000, Glimsdale, Nathan wrote:
a bug on bugzilla.yp.org?
Alexandre Belloni, co-owner and COO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
I see this intermittently when building "aws-c-iot" directly. 3 of 5Ah, interesting. Will you be tracking that on your side or should I open
attempts so far. The network restriction is also affected some of the
dependencies consistently.
a bug on bugzilla.yp.org?
On 2/2/22, 2:11 PM, "Richard Elberger" <elberger@...> wrote:--
Hi Alexandre, I'm adding Nathan to see if there's anything he can see.
I am not familiar with the warning at all, it looks new to me. Which
build step has the commit hash for the repos we're checking out? That
way I can compare apples-apples.
-- rich
On 2/2/22 16:53, Alexandre Belloni wrote:
> CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe.
>
>
>
> Hello,
>
> It's been a few days that I'm struggling to understand what happened in
> this build. I've been looking at the fetcher changes, meta-aws changes
> but couldn't find anything relevant.
>
> https://autobuilder.yoctoproject.org/typhoon/#/builders/122/builds/713/steps/16/logs/warnings
>
> I would take any hint that you have ;)
>
> Regards,
>
> --
> Alexandre Belloni, co-owner and COO, Bootlin
> Embedded Linux and Kernel engineering
> https://bootlin.com
--
Richard Elberger
Principal Technologist
AWS IoT
+1 646.927.6897
+1 203.942.8039 – mobile
elberger@... – email
Alexandre Belloni, co-owner and COO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
Richard Purdie
On Wed, 2022-02-02 at 23:54 +0100, Alexandre Belloni via lists.yoctoproject.org
wrote:
https://git.yoctoproject.org/meta-aws/tree/recipes-sdk/aws-c-iot/aws-c-iot_0.0.2.bb
https://git.yoctoproject.org/meta-aws/tree/recipes-sdk/aws-c-iot/aws-c-iot_0.0.7.bb
and I notice they use git urls with tags.
This does not do what people expect since bitbake resolves the tag each time
over the network since people can change the tags (and do!). I suspect that is
why it is this recipe which is failing.
I'm not sure why that is sometimes showing a warning though.
Cheers,
Richard
wrote:
On 02/02/2022 22:24:26+0000, Glimsdale, Nathan wrote:I did just have a look at the aws-c-iot recipes:I see this intermittently when building "aws-c-iot" directly. 3 of 5Ah, interesting. Will you be tracking that on your side or should I open
attempts so far. The network restriction is also affected some of the
dependencies consistently.
a bug on bugzilla.yp.org?
https://git.yoctoproject.org/meta-aws/tree/recipes-sdk/aws-c-iot/aws-c-iot_0.0.2.bb
https://git.yoctoproject.org/meta-aws/tree/recipes-sdk/aws-c-iot/aws-c-iot_0.0.7.bb
and I notice they use git urls with tags.
This does not do what people expect since bitbake resolves the tag each time
over the network since people can change the tags (and do!). I suspect that is
why it is this recipe which is failing.
I'm not sure why that is sometimes showing a warning though.
Cheers,
Richard
Denys Dmytriyenko
On Wed, Feb 02, 2022 at 11:04:57PM +0000, Richard Purdie wrote:
I'm guessing we need to extend yocto-check-layer with another test? :)
--
Denys
On Wed, 2022-02-02 at 23:54 +0100, Alexandre Belloni via lists.yoctoproject.orgOuch, no SRCREVs?
wrote:On 02/02/2022 22:24:26+0000, Glimsdale, Nathan wrote:I did just have a look at the aws-c-iot recipes:I see this intermittently when building "aws-c-iot" directly. 3 of 5Ah, interesting. Will you be tracking that on your side or should I open
attempts so far. The network restriction is also affected some of the
dependencies consistently.
a bug on bugzilla.yp.org?
https://git.yoctoproject.org/meta-aws/tree/recipes-sdk/aws-c-iot/aws-c-iot_0.0.2.bb
https://git.yoctoproject.org/meta-aws/tree/recipes-sdk/aws-c-iot/aws-c-iot_0.0.7.bb
and I notice they use git urls with tags.
This does not do what people expect since bitbake resolves the tag each time
over the network since people can change the tags (and do!). I suspect that is
why it is this recipe which is failing.
I'm guessing we need to extend yocto-check-layer with another test? :)
--
Denys
elberger@...
The choice to use tags is a deliberate one.
For all AWS repos that meta-aws builds recipes we use release tagging to align with a specific release. While tags can move (aka the old school CVS floating label practice which is very frowned upon except for noting "stable") they do not for these repos. Of course, there is the exception when a release is about to be made and a late breaking change must be added to the release. Then the tag shifts to meet that demand. This happens very rarely and is not a contributor to the general concern of repo maintainers implementing the floating tag/label practice and the recipe using that tag.
Further, I don't think such a check in yocto-check-layer is appropriate. Rather, if a recipe author knows that a specific CM practice is in place for particular repos then have the opportunity to tell the various stages "don't check, I know what I'm doing". If YP/OE really is against using the tag mechanism then I suggest we deprecate it. I have found it very nice for maintainership when we know exactly how the development team handles the CM.
toggle quoted message
Show quoted text
For all AWS repos that meta-aws builds recipes we use release tagging to align with a specific release. While tags can move (aka the old school CVS floating label practice which is very frowned upon except for noting "stable") they do not for these repos. Of course, there is the exception when a release is about to be made and a late breaking change must be added to the release. Then the tag shifts to meet that demand. This happens very rarely and is not a contributor to the general concern of repo maintainers implementing the floating tag/label practice and the recipe using that tag.
Further, I don't think such a check in yocto-check-layer is appropriate. Rather, if a recipe author knows that a specific CM practice is in place for particular repos then have the opportunity to tell the various stages "don't check, I know what I'm doing". If YP/OE really is against using the tag mechanism then I suggest we deprecate it. I have found it very nice for maintainership when we know exactly how the development team handles the CM.
-----Original Message-----
From: Denys Dmytriyenko <denis@...>
Sent: Wednesday, February 2, 2022 6:17 PM
To: swat@...
Cc: Glimsdale, Nathan <glimsdal@...>; Elberger, Richard <elberger@...>
Subject: RE: [EXTERNAL] [swat] meta-aws warnings: SRCREV_FORMAT variable must be set when multiple SCMs are used
CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe.
On Wed, Feb 02, 2022 at 11:04:57PM +0000, Richard Purdie wrote:
I'm guessing we need to extend yocto-check-layer with another test? :)
--
Denys
From: Denys Dmytriyenko <denis@...>
Sent: Wednesday, February 2, 2022 6:17 PM
To: swat@...
Cc: Glimsdale, Nathan <glimsdal@...>; Elberger, Richard <elberger@...>
Subject: RE: [EXTERNAL] [swat] meta-aws warnings: SRCREV_FORMAT variable must be set when multiple SCMs are used
CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe.
On Wed, Feb 02, 2022 at 11:04:57PM +0000, Richard Purdie wrote:
On Wed, 2022-02-02 at 23:54 +0100, Alexandre Belloni viaOuch, no SRCREVs?
lists.yoctoproject.org
wrote:On 02/02/2022 22:24:26+0000, Glimsdale, Nathan wrote:I did just have a look at the aws-c-iot recipes:I see this intermittently when building "aws-c-iot" directly. 3 ofAh, interesting. Will you be tracking that on your side or should I
5 attempts so far. The network restriction is also affected some
of the dependencies consistently.
open a bug on bugzilla.yp.org?
https://git.yoctoproject.org/meta-aws/tree/recipes-sdk/aws-c-iot/aws-c
-iot_0.0.2.bb
https://git.yoctoproject.org/meta-aws/tree/recipes-sdk/aws-c-iot/aws-c
-iot_0.0.7.bb
and I notice they use git urls with tags.
This does not do what people expect since bitbake resolves the tag
each time over the network since people can change the tags (and do!).
I suspect that is why it is this recipe which is failing.
I'm guessing we need to extend yocto-check-layer with another test? :)
--
Denys
Alexandre Belloni
On 03/02/2022 14:02:54+0000, elberger via lists.yoctoproject.org wrote:
BB_NO_NETWORK = 1 and so there is no way to ensure you have locally all
the code you need to reproduce the build in case Amazon gets out of
business one day ;)
Alexandre Belloni, co-owner and COO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
The choice to use tags is a deliberate one.The main drawback is that then you are not able to build with
For all AWS repos that meta-aws builds recipes we use release tagging to align with a specific release. While tags can move (aka the old school CVS floating label practice which is very frowned upon except for noting "stable") they do not for these repos. Of course, there is the exception when a release is about to be made and a late breaking change must be added to the release. Then the tag shifts to meet that demand. This happens very rarely and is not a contributor to the general concern of repo maintainers implementing the floating tag/label practice and the recipe using that tag.
Further, I don't think such a check in yocto-check-layer is appropriate. Rather, if a recipe author knows that a specific CM practice is in place for particular repos then have the opportunity to tell the various stages "don't check, I know what I'm doing". If YP/OE really is against using the tag mechanism then I suggest we deprecate it. I have found it very nice for maintainership when we know exactly how the development team handles the CM.
BB_NO_NETWORK = 1 and so there is no way to ensure you have locally all
the code you need to reproduce the build in case Amazon gets out of
business one day ;)
-----Original Message-------
From: Denys Dmytriyenko <denis@...>
Sent: Wednesday, February 2, 2022 6:17 PM
To: swat@...
Cc: Glimsdale, Nathan <glimsdal@...>; Elberger, Richard <elberger@...>
Subject: RE: [EXTERNAL] [swat] meta-aws warnings: SRCREV_FORMAT variable must be set when multiple SCMs are used
CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe.
On Wed, Feb 02, 2022 at 11:04:57PM +0000, Richard Purdie wrote:On Wed, 2022-02-02 at 23:54 +0100, Alexandre Belloni viaOuch, no SRCREVs?
lists.yoctoproject.org
wrote:On 02/02/2022 22:24:26+0000, Glimsdale, Nathan wrote:I did just have a look at the aws-c-iot recipes:I see this intermittently when building "aws-c-iot" directly. 3 ofAh, interesting. Will you be tracking that on your side or should I
5 attempts so far. The network restriction is also affected some
of the dependencies consistently.
open a bug on bugzilla.yp.org?
https://git.yoctoproject.org/meta-aws/tree/recipes-sdk/aws-c-iot/aws-c
-iot_0.0.2.bb
https://git.yoctoproject.org/meta-aws/tree/recipes-sdk/aws-c-iot/aws-c
-iot_0.0.7.bb
and I notice they use git urls with tags.
This does not do what people expect since bitbake resolves the tag
each time over the network since people can change the tags (and do!).
I suspect that is why it is this recipe which is failing.
I'm guessing we need to extend yocto-check-layer with another test? :)
--
Denys
Alexandre Belloni, co-owner and COO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
So the main error is:
aws-c-auth-0.6.1-r0 do_packagedata_setscene: ExpansionError('SRCPV',
'${@bb.fetch2.get_srcrev(d)}', FetchError('The SRCREV_FORMAT variable
must be set when multiple SCMs are used.\nThe SCMs
are:\ngit://github.com/awslabs/aws-c-common.git;protocol=https;branch=main;tag=v0.6.8;destsuffix=/home/pokybuild/yocto-worker/meta-aws/build/build/tmp/work/cortexa57-poky-linux/aws-c-auth/0.6.1-r0/git/aws-c-common;name=common\ngit://github.com/awslabs/aws-c-auth.git;protocol=https;branch=main;tag=v0.6.1;destsuffix=/home/pokybuild/yocto-worker/meta-aws/build/build/tmp/work/cortexa57-poky-linux/aws-c-auth/0.6.1-r0/git/aws-c-auth;name=auth',
None))
Let's try to dissect:
The SRCREV_FORMAT variable must be set when multiple SCMs are used.
The SCMs are:
git://github.com/awslabs/aws-c-common.git;protocol=https;branch=main;tag=v0.6.8;destsuffix=/home/pokybuild/yocto-worker/meta-aws/build/build/tmp/work/cortexa57-poky-linux/aws-c-auth/0.6.1-r0/git/aws-c-common;name=common
git://github.com/awslabs/aws-c-auth.git;protocol=https;branch=main;tag=v0.6.1;destsuffix=/home/pokybuild/yocto-worker/meta-aws/build/build/tmp/work/cortexa57-poky-linux/aws-c-auth/0.6.1-r0/git/aws-c-auth;name=auth
I read this like:
- we're using two git:// url's in the SRC_URI
- bitbake needs one single hash value for SRCREV for the recipe to be 'uniq'
- we cannot decide what to use as SRCREV based on this SRC_URI (with
multiple git:// entries)
I found in meta-arm:
SRC_URI = "git://git.trustedfirmware.org/TF-A/trusted-firmware-a.git;protocol=https;name=tfa"
# Use TF-A for version
SRCREV_FORMAT = "tfa"
# ^^^^^^^^^^^^^^^^^^^^^^^^ !!!!!!!! tfa = name from above
# TF-A v2.1
SRCREV_tfa = "e1286bdb968ee74fc52f96cf303a4218e1ae2950"
#
# mbed TLS source
# Those are used in trusted-firmware-a.inc if TFA_MBEDTLS is set to 1
SRC_URI_MBEDTLS =
"git://github.com/ARMmbed/mbedtls.git;name=mbedtls;protocol=https;destsuffix=git/mbedtls;branch=mbedtls-2.16"
SRCREV_mbedtls =
"d81c11b8ab61fd5b2da8133aa73c5fe33a0633eb"
So to apply this to meta-aws, you'd have to pick one which should be
the canonical SRCREV of the two for this recipe.
E.g. (untested)
aws-c-auth_0.6.1.bb :
[...]
TAG ?= "v${PV}"
TAG_COMMON ?= "v0.6.8"
SRC_URI = "git://github.com/awslabs/aws-c-common.git;protocol=https;branch=${BRANCH};tag=${TAG_COMMON};destsuffix=${S}/aws-c-common;name=common
\
git://github.com/awslabs/aws-c-auth.git;protocol=https;branch=${BRANCH};tag=${TAG};destsuffix=${S}/aws-c-auth;name=auth
\
"
S= "${WORKDIR}/git"
BRANCH ?= "main"
# _____________________
SRCREV_FORMAT = "auth"
# ^^^^^^^^^^^^^^^^^^^^
But the main question here is: why do you need aws-c-common in SRC_URI
? . aws-c-common has its own recipe, so if it installs the headers
properly it can be in DEPENDS and will be pulled automagically.
The optimization wrt tag and network / offline that Denis mentioned is
important imho as well ! It is really better to use a SRCREV. But yes,
it might affect workflow and means bumping SRCREV's.
TLDR:
a) If possible make aws-c-common a DEPENDS in the auth recipe
b) if above does not work, try SRCREV_FORMAT = "auth" (other recipes likewise)
c) tag should be optimized to a SRCREV to avoid the enforced network fetch
HTH.
Best,
Jan-Simon
------
Jan-Simon Möller
AGL Release Manager
The Linux Foundation
Visit us at:
www.automotivegradelinux.org
lists.automotivelinux.org
www.linuxfoundation.org
Best regards,
Jan-Simon
------
Jan-Simon Möller
AGL Release Manager
The Linux Foundation
Visit us at:
www.automotivegradelinux.org
lists.automotivelinux.org
www.linuxfoundation.org
On Thu, Feb 3, 2022 at 3:12 PM Alexandre Belloni via
lists.yoctoproject.org
<alexandre.belloni=bootlin.com@...> wrote:
toggle quoted message
Show quoted text
aws-c-auth-0.6.1-r0 do_packagedata_setscene: ExpansionError('SRCPV',
'${@bb.fetch2.get_srcrev(d)}', FetchError('The SRCREV_FORMAT variable
must be set when multiple SCMs are used.\nThe SCMs
are:\ngit://github.com/awslabs/aws-c-common.git;protocol=https;branch=main;tag=v0.6.8;destsuffix=/home/pokybuild/yocto-worker/meta-aws/build/build/tmp/work/cortexa57-poky-linux/aws-c-auth/0.6.1-r0/git/aws-c-common;name=common\ngit://github.com/awslabs/aws-c-auth.git;protocol=https;branch=main;tag=v0.6.1;destsuffix=/home/pokybuild/yocto-worker/meta-aws/build/build/tmp/work/cortexa57-poky-linux/aws-c-auth/0.6.1-r0/git/aws-c-auth;name=auth',
None))
Let's try to dissect:
The SRCREV_FORMAT variable must be set when multiple SCMs are used.
The SCMs are:
git://github.com/awslabs/aws-c-common.git;protocol=https;branch=main;tag=v0.6.8;destsuffix=/home/pokybuild/yocto-worker/meta-aws/build/build/tmp/work/cortexa57-poky-linux/aws-c-auth/0.6.1-r0/git/aws-c-common;name=common
git://github.com/awslabs/aws-c-auth.git;protocol=https;branch=main;tag=v0.6.1;destsuffix=/home/pokybuild/yocto-worker/meta-aws/build/build/tmp/work/cortexa57-poky-linux/aws-c-auth/0.6.1-r0/git/aws-c-auth;name=auth
I read this like:
- we're using two git:// url's in the SRC_URI
- bitbake needs one single hash value for SRCREV for the recipe to be 'uniq'
- we cannot decide what to use as SRCREV based on this SRC_URI (with
multiple git:// entries)
I found in meta-arm:
SRC_URI = "git://git.trustedfirmware.org/TF-A/trusted-firmware-a.git;protocol=https;name=tfa"
# Use TF-A for version
SRCREV_FORMAT = "tfa"
# ^^^^^^^^^^^^^^^^^^^^^^^^ !!!!!!!! tfa = name from above
# TF-A v2.1
SRCREV_tfa = "e1286bdb968ee74fc52f96cf303a4218e1ae2950"
#
# mbed TLS source
# Those are used in trusted-firmware-a.inc if TFA_MBEDTLS is set to 1
SRC_URI_MBEDTLS =
"git://github.com/ARMmbed/mbedtls.git;name=mbedtls;protocol=https;destsuffix=git/mbedtls;branch=mbedtls-2.16"
SRCREV_mbedtls =
"d81c11b8ab61fd5b2da8133aa73c5fe33a0633eb"
So to apply this to meta-aws, you'd have to pick one which should be
the canonical SRCREV of the two for this recipe.
E.g. (untested)
aws-c-auth_0.6.1.bb :
[...]
TAG ?= "v${PV}"
TAG_COMMON ?= "v0.6.8"
SRC_URI = "git://github.com/awslabs/aws-c-common.git;protocol=https;branch=${BRANCH};tag=${TAG_COMMON};destsuffix=${S}/aws-c-common;name=common
\
git://github.com/awslabs/aws-c-auth.git;protocol=https;branch=${BRANCH};tag=${TAG};destsuffix=${S}/aws-c-auth;name=auth
\
"
S= "${WORKDIR}/git"
BRANCH ?= "main"
# _____________________
SRCREV_FORMAT = "auth"
# ^^^^^^^^^^^^^^^^^^^^
But the main question here is: why do you need aws-c-common in SRC_URI
? . aws-c-common has its own recipe, so if it installs the headers
properly it can be in DEPENDS and will be pulled automagically.
The optimization wrt tag and network / offline that Denis mentioned is
important imho as well ! It is really better to use a SRCREV. But yes,
it might affect workflow and means bumping SRCREV's.
TLDR:
a) If possible make aws-c-common a DEPENDS in the auth recipe
b) if above does not work, try SRCREV_FORMAT = "auth" (other recipes likewise)
c) tag should be optimized to a SRCREV to avoid the enforced network fetch
HTH.
Best,
Jan-Simon
------
Jan-Simon Möller
AGL Release Manager
The Linux Foundation
Visit us at:
www.automotivegradelinux.org
lists.automotivelinux.org
www.linuxfoundation.org
Best regards,
Jan-Simon
------
Jan-Simon Möller
AGL Release Manager
The Linux Foundation
Visit us at:
www.automotivegradelinux.org
lists.automotivelinux.org
www.linuxfoundation.org
On Thu, Feb 3, 2022 at 3:12 PM Alexandre Belloni via
lists.yoctoproject.org
<alexandre.belloni=bootlin.com@...> wrote:
On 03/02/2022 14:02:54+0000, elberger via lists.yoctoproject.org wrote:The choice to use tags is a deliberate one.The main drawback is that then you are not able to build with
For all AWS repos that meta-aws builds recipes we use release tagging to align with a specific release. While tags can move (aka the old school CVS floating label practice which is very frowned upon except for noting "stable") they do not for these repos. Of course, there is the exception when a release is about to be made and a late breaking change must be added to the release. Then the tag shifts to meet that demand. This happens very rarely and is not a contributor to the general concern of repo maintainers implementing the floating tag/label practice and the recipe using that tag.
Further, I don't think such a check in yocto-check-layer is appropriate. Rather, if a recipe author knows that a specific CM practice is in place for particular repos then have the opportunity to tell the various stages "don't check, I know what I'm doing". If YP/OE really is against using the tag mechanism then I suggest we deprecate it. I have found it very nice for maintainership when we know exactly how the development team handles the CM.
BB_NO_NETWORK = 1 and so there is no way to ensure you have locally all
the code you need to reproduce the build in case Amazon gets out of
business one day ;)-----Original Message-------
From: Denys Dmytriyenko <denis@...>
Sent: Wednesday, February 2, 2022 6:17 PM
To: swat@...
Cc: Glimsdale, Nathan <glimsdal@...>; Elberger, Richard <elberger@...>
Subject: RE: [EXTERNAL] [swat] meta-aws warnings: SRCREV_FORMAT variable must be set when multiple SCMs are used
CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe.
On Wed, Feb 02, 2022 at 11:04:57PM +0000, Richard Purdie wrote:On Wed, 2022-02-02 at 23:54 +0100, Alexandre Belloni viaOuch, no SRCREVs?
lists.yoctoproject.org
wrote:On 02/02/2022 22:24:26+0000, Glimsdale, Nathan wrote:I did just have a look at the aws-c-iot recipes:I see this intermittently when building "aws-c-iot" directly. 3 ofAh, interesting. Will you be tracking that on your side or should I
5 attempts so far. The network restriction is also affected some
of the dependencies consistently.
open a bug on bugzilla.yp.org?
https://git.yoctoproject.org/meta-aws/tree/recipes-sdk/aws-c-iot/aws-c
-iot_0.0.2.bb
https://git.yoctoproject.org/meta-aws/tree/recipes-sdk/aws-c-iot/aws-c
-iot_0.0.7.bb
and I notice they use git urls with tags.
This does not do what people expect since bitbake resolves the tag
each time over the network since people can change the tags (and do!).
I suspect that is why it is this recipe which is failing.
I'm guessing we need to extend yocto-check-layer with another test? :)
--
Denys
Alexandre Belloni, co-owner and COO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
Richard Purdie
On Thu, 2022-02-03 at 14:02 +0000, elberger via lists.yoctoproject.org wrote:
considering, specifically:
a) it breaks anyone doing offline builds
b) it means source archives don't function correct
c) it means your code isn't reproducible
a specific build exactly, at least "out the box" without changing things. By
using tags and allowing them to shift even if it is "very rarely", it isn't
explicit and therefore isn't reproducible.
realising there are serious consequences though (as listed above). I therefore
think this is exactly the kind of thing yocto-check-layer should be flagging up.
We don't want to "ban" them outright but we do want our best practises to say
not to use them.
the very least need to be thought through. Are you happy that meta-aws isn't
reproducible, buildable offline and can't work against source archives?
Cheers,
Richard
The choice to use tags is a deliberate one.It may be a deliberate choice but I suspect there are some issues you aren't
considering, specifically:
a) it breaks anyone doing offline builds
b) it means source archives don't function correct
c) it means your code isn't reproducible
For all AWS repos that meta-aws builds recipes we use release tagging to alignYou are basically making my point here for me. Our metadata is meant to describe
with a specific release. While tags can move (aka the old school CVS floating
label practice which is very frowned upon except for noting "stable") they do
not for these repos. Of course, there is the exception when a release is about
to be made and a late breaking change must be added to the release. Then the
tag shifts to meet that demand. This happens very rarely and is not a
contributor to the general concern of repo maintainers implementing the
floating tag/label practice and the recipe using that tag.
a specific build exactly, at least "out the box" without changing things. By
using tags and allowing them to shift even if it is "very rarely", it isn't
explicit and therefore isn't reproducible.
Further, I don't think such a check in yocto-check-layer is appropriate.I'm going to disagree, let me explain why.
Rather, if a recipe author knows that a specific CM practice is in place forI have no issue with anyone using tags if they want to. Most people use them not
particular repos then have the opportunity to tell the various stages "don't
check, I know what I'm doing".
realising there are serious consequences though (as listed above). I therefore
think this is exactly the kind of thing yocto-check-layer should be flagging up.
We don't want to "ban" them outright but we do want our best practises to say
not to use them.
If YP/OE really is against using the tag mechanism then I suggest weI agree it is "nice" for maintainership but it has several implications which at
deprecate it. I have found it very nice for maintainership when we know
exactly how the development team handles the CM.
the very least need to be thought through. Are you happy that meta-aws isn't
reproducible, buildable offline and can't work against source archives?
Cheers,
Richard
elberger@...
This makes a lot of sense to me and it's clear now. It didn't come across this way in prior responses. Thank you for taking the time to put this together.
We can't remove aws-c-common from the sources since ... it's a build quirk ... virtually all repos require it to be cloned in the source tree context.
Fixing this is going to take quite some time - will need to get the work scheduled. Hopefully will get it done over the next week.
toggle quoted message
Show quoted text
We can't remove aws-c-common from the sources since ... it's a build quirk ... virtually all repos require it to be cloned in the source tree context.
Fixing this is going to take quite some time - will need to get the work scheduled. Hopefully will get it done over the next week.
-----Original Message-----
From: Jan-Simon Moeller <jsmoeller@...>
Sent: Thursday, February 3, 2022 9:41 AM
To: Elberger, Richard <elberger@...>
Subject: FW: [EXTERNAL] [swat] meta-aws warnings: SRCREV_FORMAT variable must be set when multiple SCMs are used
CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe.
Forgot to add all CC back ...
Best regards,
Jan-Simon
------
Jan-Simon Möller
AGL Release Manager
The Linux Foundation
Visit us at:
www.automotivegradelinux.org
lists.automotivelinux.org
www.linuxfoundation.org
---------- Forwarded message ---------
From: Jan-Simon Moeller <jsmoeller@...>
Date: Thu, Feb 3, 2022 at 3:40 PM
Subject: Re: [swat] meta-aws warnings: SRCREV_FORMAT variable must be set when multiple SCMs are used
To: <swat@...>
Cc: Denys Dmytriyenko <denis@...>, Glimsdale, Nathan <glimsdal@...>
So the main error is:
aws-c-auth-0.6.1-r0 do_packagedata_setscene: ExpansionError('SRCPV', '${@bb.fetch2.get_srcrev(d)}', FetchError('The SRCREV_FORMAT variable must be set when multiple SCMs are used.\nThe SCMs are:\ngit://github.com/awslabs/aws-c-common.git;protocol=https;branch=main;tag=v0.6.8;destsuffix=/home/pokybuild/yocto-worker/meta-aws/build/build/tmp/work/cortexa57-poky-linux/aws-c-auth/0.6.1-r0/git/aws-c-common;name=common\ngit://github.com/awslabs/aws-c-auth.git;protocol=https;branch=main;tag=v0.6.1;destsuffix=/home/pokybuild/yocto-worker/meta-aws/build/build/tmp/work/cortexa57-poky-linux/aws-c-auth/0.6.1-r0/git/aws-c-auth;name=auth',
None))
Let's try to dissect:
The SRCREV_FORMAT variable must be set when multiple SCMs are used.
The SCMs are:
git://github.com/awslabs/aws-c-common.git;protocol=https;branch=main;tag=v0.6.8;destsuffix=/home/pokybuild/yocto-worker/meta-aws/build/build/tmp/work/cortexa57-poky-linux/aws-c-auth/0.6.1-r0/git/aws-c-common;name=common
git://github.com/awslabs/aws-c-auth.git;protocol=https;branch=main;tag=v0.6.1;destsuffix=/home/pokybuild/yocto-worker/meta-aws/build/build/tmp/work/cortexa57-poky-linux/aws-c-auth/0.6.1-r0/git/aws-c-auth;name=auth
I read this like:
- we're using two git:// url's in the SRC_URI
- bitbake needs one single hash value for SRCREV for the recipe to be 'uniq'
- we cannot decide what to use as SRCREV based on this SRC_URI (with multiple git:// entries)
I found in meta-arm:
SRC_URI = "git://git.trustedfirmware.org/TF-A/trusted-firmware-a.git;protocol=https;name=tfa"
# Use TF-A for version
SRCREV_FORMAT = "tfa"
# ^^^^^^^^^^^^^^^^^^^^^^^^ !!!!!!!! tfa = name from above # TF-A v2.1 SRCREV_tfa = "e1286bdb968ee74fc52f96cf303a4218e1ae2950"
#
# mbed TLS source
# Those are used in trusted-firmware-a.inc if TFA_MBEDTLS is set to 1 SRC_URI_MBEDTLS = "git://github.com/ARMmbed/mbedtls.git;name=mbedtls;protocol=https;destsuffix=git/mbedtls;branch=mbedtls-2.16"
SRCREV_mbedtls = "d81c11b8ab61fd5b2da8133aa73c5fe33a0633eb"
So to apply this to meta-aws, you'd have to pick one which should be the canonical SRCREV of the two for this recipe.
E.g. (untested)
aws-c-auth_0.6.1.bb :
[...]
TAG ?= "v${PV}"
TAG_COMMON ?= "v0.6.8"
SRC_URI = "git://github.com/awslabs/aws-c-common.git;protocol=https;branch=${BRANCH};tag=${TAG_COMMON};destsuffix=${S}/aws-c-common;name=common
\
git://github.com/awslabs/aws-c-auth.git;protocol=https;branch=${BRANCH};tag=${TAG};destsuffix=${S}/aws-c-auth;name=auth
\
"
S= "${WORKDIR}/git"
BRANCH ?= "main"
# _____________________
SRCREV_FORMAT = "auth"
# ^^^^^^^^^^^^^^^^^^^^
But the main question here is: why do you need aws-c-common in SRC_URI ? . aws-c-common has its own recipe, so if it installs the headers properly it can be in DEPENDS and will be pulled automagically.
The optimization wrt tag and network / offline that Denis mentioned is important imho as well ! It is really better to use a SRCREV. But yes, it might affect workflow and means bumping SRCREV's.
TLDR:
a) If possible make aws-c-common a DEPENDS in the auth recipe
b) if above does not work, try SRCREV_FORMAT = "auth" (other recipes likewise)
c) tag should be optimized to a SRCREV to avoid the enforced network fetch
HTH.
Best,
Jan-Simon
------
Jan-Simon Möller
AGL Release Manager
The Linux Foundation
Visit us at:
www.automotivegradelinux.org
lists.automotivelinux.org
www.linuxfoundation.org
Best regards,
Jan-Simon
------
Jan-Simon Möller
AGL Release Manager
The Linux Foundation
Visit us at:
www.automotivegradelinux.org
lists.automotivelinux.org
www.linuxfoundation.org
On Thu, Feb 3, 2022 at 3:12 PM Alexandre Belloni via lists.yoctoproject.org <alexandre.belloni=bootlin.com@...> wrote:
From: Jan-Simon Moeller <jsmoeller@...>
Sent: Thursday, February 3, 2022 9:41 AM
To: Elberger, Richard <elberger@...>
Subject: FW: [EXTERNAL] [swat] meta-aws warnings: SRCREV_FORMAT variable must be set when multiple SCMs are used
CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe.
Forgot to add all CC back ...
Best regards,
Jan-Simon
------
Jan-Simon Möller
AGL Release Manager
The Linux Foundation
Visit us at:
www.automotivegradelinux.org
lists.automotivelinux.org
www.linuxfoundation.org
---------- Forwarded message ---------
From: Jan-Simon Moeller <jsmoeller@...>
Date: Thu, Feb 3, 2022 at 3:40 PM
Subject: Re: [swat] meta-aws warnings: SRCREV_FORMAT variable must be set when multiple SCMs are used
To: <swat@...>
Cc: Denys Dmytriyenko <denis@...>, Glimsdale, Nathan <glimsdal@...>
So the main error is:
aws-c-auth-0.6.1-r0 do_packagedata_setscene: ExpansionError('SRCPV', '${@bb.fetch2.get_srcrev(d)}', FetchError('The SRCREV_FORMAT variable must be set when multiple SCMs are used.\nThe SCMs are:\ngit://github.com/awslabs/aws-c-common.git;protocol=https;branch=main;tag=v0.6.8;destsuffix=/home/pokybuild/yocto-worker/meta-aws/build/build/tmp/work/cortexa57-poky-linux/aws-c-auth/0.6.1-r0/git/aws-c-common;name=common\ngit://github.com/awslabs/aws-c-auth.git;protocol=https;branch=main;tag=v0.6.1;destsuffix=/home/pokybuild/yocto-worker/meta-aws/build/build/tmp/work/cortexa57-poky-linux/aws-c-auth/0.6.1-r0/git/aws-c-auth;name=auth',
None))
Let's try to dissect:
The SRCREV_FORMAT variable must be set when multiple SCMs are used.
The SCMs are:
git://github.com/awslabs/aws-c-common.git;protocol=https;branch=main;tag=v0.6.8;destsuffix=/home/pokybuild/yocto-worker/meta-aws/build/build/tmp/work/cortexa57-poky-linux/aws-c-auth/0.6.1-r0/git/aws-c-common;name=common
git://github.com/awslabs/aws-c-auth.git;protocol=https;branch=main;tag=v0.6.1;destsuffix=/home/pokybuild/yocto-worker/meta-aws/build/build/tmp/work/cortexa57-poky-linux/aws-c-auth/0.6.1-r0/git/aws-c-auth;name=auth
I read this like:
- we're using two git:// url's in the SRC_URI
- bitbake needs one single hash value for SRCREV for the recipe to be 'uniq'
- we cannot decide what to use as SRCREV based on this SRC_URI (with multiple git:// entries)
I found in meta-arm:
SRC_URI = "git://git.trustedfirmware.org/TF-A/trusted-firmware-a.git;protocol=https;name=tfa"
# Use TF-A for version
SRCREV_FORMAT = "tfa"
# ^^^^^^^^^^^^^^^^^^^^^^^^ !!!!!!!! tfa = name from above # TF-A v2.1 SRCREV_tfa = "e1286bdb968ee74fc52f96cf303a4218e1ae2950"
#
# mbed TLS source
# Those are used in trusted-firmware-a.inc if TFA_MBEDTLS is set to 1 SRC_URI_MBEDTLS = "git://github.com/ARMmbed/mbedtls.git;name=mbedtls;protocol=https;destsuffix=git/mbedtls;branch=mbedtls-2.16"
SRCREV_mbedtls = "d81c11b8ab61fd5b2da8133aa73c5fe33a0633eb"
So to apply this to meta-aws, you'd have to pick one which should be the canonical SRCREV of the two for this recipe.
E.g. (untested)
aws-c-auth_0.6.1.bb :
[...]
TAG ?= "v${PV}"
TAG_COMMON ?= "v0.6.8"
SRC_URI = "git://github.com/awslabs/aws-c-common.git;protocol=https;branch=${BRANCH};tag=${TAG_COMMON};destsuffix=${S}/aws-c-common;name=common
\
git://github.com/awslabs/aws-c-auth.git;protocol=https;branch=${BRANCH};tag=${TAG};destsuffix=${S}/aws-c-auth;name=auth
\
"
S= "${WORKDIR}/git"
BRANCH ?= "main"
# _____________________
SRCREV_FORMAT = "auth"
# ^^^^^^^^^^^^^^^^^^^^
But the main question here is: why do you need aws-c-common in SRC_URI ? . aws-c-common has its own recipe, so if it installs the headers properly it can be in DEPENDS and will be pulled automagically.
The optimization wrt tag and network / offline that Denis mentioned is important imho as well ! It is really better to use a SRCREV. But yes, it might affect workflow and means bumping SRCREV's.
TLDR:
a) If possible make aws-c-common a DEPENDS in the auth recipe
b) if above does not work, try SRCREV_FORMAT = "auth" (other recipes likewise)
c) tag should be optimized to a SRCREV to avoid the enforced network fetch
HTH.
Best,
Jan-Simon
------
Jan-Simon Möller
AGL Release Manager
The Linux Foundation
Visit us at:
www.automotivegradelinux.org
lists.automotivelinux.org
www.linuxfoundation.org
Best regards,
Jan-Simon
------
Jan-Simon Möller
AGL Release Manager
The Linux Foundation
Visit us at:
www.automotivegradelinux.org
lists.automotivelinux.org
www.linuxfoundation.org
On Thu, Feb 3, 2022 at 3:12 PM Alexandre Belloni via lists.yoctoproject.org <alexandre.belloni=bootlin.com@...> wrote:
On 03/02/2022 14:02:54+0000, elberger via lists.yoctoproject.org wrote:The choice to use tags is a deliberate one.The main drawback is that then you are not able to build with
For all AWS repos that meta-aws builds recipes we use release tagging to align with a specific release. While tags can move (aka the old school CVS floating label practice which is very frowned upon except for noting "stable") they do not for these repos. Of course, there is the exception when a release is about to be made and a late breaking change must be added to the release. Then the tag shifts to meet that demand. This happens very rarely and is not a contributor to the general concern of repo maintainers implementing the floating tag/label practice and the recipe using that tag.
Further, I don't think such a check in yocto-check-layer is appropriate. Rather, if a recipe author knows that a specific CM practice is in place for particular repos then have the opportunity to tell the various stages "don't check, I know what I'm doing". If YP/OE really is against using the tag mechanism then I suggest we deprecate it. I have found it very nice for maintainership when we know exactly how the development team handles the CM.
BB_NO_NETWORK = 1 and so there is no way to ensure you have locally
all the code you need to reproduce the build in case Amazon gets out
of business one day ;)-----Original Message-------
From: Denys Dmytriyenko <denis@...>
Sent: Wednesday, February 2, 2022 6:17 PM
To: swat@...
Cc: Glimsdale, Nathan <glimsdal@...>; Elberger, Richard
<elberger@...>
Subject: RE: [EXTERNAL] [swat] meta-aws warnings: SRCREV_FORMAT
variable must be set when multiple SCMs are used
CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe.
On Wed, Feb 02, 2022 at 11:04:57PM +0000, Richard Purdie wrote:On Wed, 2022-02-02 at 23:54 +0100, Alexandre Belloni viaOuch, no SRCREVs?
lists.yoctoproject.org
wrote:On 02/02/2022 22:24:26+0000, Glimsdale, Nathan wrote:I did just have a look at the aws-c-iot recipes:I see this intermittently when building "aws-c-iot" directly.Ah, interesting. Will you be tracking that on your side or
3 of
5 attempts so far. The network restriction is also affected
some of the dependencies consistently.
should I open a bug on bugzilla.yp.org?
https://git.yoctoproject.org/meta-aws/tree/recipes-sdk/aws-c-iot/a
ws-c
-iot_0.0.2.bb
https://git.yoctoproject.org/meta-aws/tree/recipes-sdk/aws-c-iot/a
ws-c
-iot_0.0.7.bb
and I notice they use git urls with tags.
This does not do what people expect since bitbake resolves the tag
each time over the network since people can change the tags (and do!).
I suspect that is why it is this recipe which is failing.
I'm guessing we need to extend yocto-check-layer with another test?
:)
--
Denys
Alexandre Belloni, co-owner and COO, Bootlin Embedded Linux and Kernel
engineering https://bootlin.com