[PATCH] multiconfig: use diferent TMPDIR for each machine


Jose Quaresma
 

Bitbake manual [1] recomends to use one TMPDIR for each machine:

"Minimally, each configuration file must define the machine and the
temporary directory BitBake uses for the build.
Suggested practice dictates that you do not overlap the temporary
directories used during the builds."

[1] https://docs.yoctoproject.org/bitbake/bitbake-user-manual/bitbake-user-manual-intro.html#executing-a-multiple-configuration-build

Signed-off-by: Jose Quaresma <jose.quaresma@...>
---
meta-ti-bsp/conf/multiconfig/k3r5.conf | 2 ++
1 file changed, 2 insertions(+)

diff --git a/meta-ti-bsp/conf/multiconfig/k3r5.conf b/meta-ti-bsp/conf/multiconfig/k3r5.conf
index deb07210..0ba163ae 100644
--- a/meta-ti-bsp/conf/multiconfig/k3r5.conf
+++ b/meta-ti-bsp/conf/multiconfig/k3r5.conf
@@ -3,3 +3,5 @@ MAINMACHINE := "${MACHINE}"
DEPLOY_DIR_IMAGE = "${DEPLOY_DIR}/images/${MAINMACHINE}"

MACHINE:append = "-k3r5"
+
+TMPDIR = "${TOPDIR}/tmp-${MACHINE}"
--
2.36.1


Denys Dmytriyenko
 

On Mon, May 30, 2022 at 03:38:10PM +0100, Jose Quaresma wrote:
Bitbake manual [1] recomends to use one TMPDIR for each machine:

"Minimally, each configuration file must define the machine and the
temporary directory BitBake uses for the build.
Suggested practice dictates that you do not overlap the temporary
directories used during the builds."
What specific issue are you trying to fix with this?

I'm not aware of any immediate issues (because here multiconfig is used for
completely separate machines and those are well isolated inside the temp dir),
but I see few problems this will cause down the road, so I'm against this
change.


[1] https://docs.yoctoproject.org/bitbake/bitbake-user-manual/bitbake-user-manual-intro.html#executing-a-multiple-configuration-build

Signed-off-by: Jose Quaresma <jose.quaresma@...>
---
meta-ti-bsp/conf/multiconfig/k3r5.conf | 2 ++
1 file changed, 2 insertions(+)

diff --git a/meta-ti-bsp/conf/multiconfig/k3r5.conf b/meta-ti-bsp/conf/multiconfig/k3r5.conf
index deb07210..0ba163ae 100644
--- a/meta-ti-bsp/conf/multiconfig/k3r5.conf
+++ b/meta-ti-bsp/conf/multiconfig/k3r5.conf
@@ -3,3 +3,5 @@ MAINMACHINE := "${MACHINE}"
DEPLOY_DIR_IMAGE = "${DEPLOY_DIR}/images/${MAINMACHINE}"

MACHINE:append = "-k3r5"
+
+TMPDIR = "${TOPDIR}/tmp-${MACHINE}"
--
2.36.1


Jose Quaresma
 



Denys Dmytriyenko <denis@...> escreveu no dia terça, 31/05/2022 à(s) 20:33:
On Mon, May 30, 2022 at 03:38:10PM +0100, Jose Quaresma wrote:
> Bitbake manual [1] recomends to use one TMPDIR for each machine:
>
>  "Minimally, each configuration file must define the machine and the
>  temporary directory BitBake uses for the build.
>  Suggested practice dictates that you do not overlap the temporary
>  directories used during the builds."

What specific issue are you trying to fix with this?

The issue that I have is in the OE-core archiver.bbclasse that doesn't support multiconfig when
using the same TMPDIR, this because the shared recipes like gcc, kernel

It can be replicates in kirkstone using the am64xx-evm with the following:

# archive sources for target recipes in conf/local.conf 
INHERIT += "archiver"
COPYLEFT_RECIPE_TYPES = "target"
ARCHIVER_MODE[src] = "original"
ARCHIVER_MODE[diff] = "1"

# and run bitbake with:
bitbake mc:k3r5:gcc-source-11.2.0 gcc-source-11.2.0 -c do_unpack_and_patch -f

more info in following patch:


I'm not aware of any immediate issues (because here multiconfig is used for
completely separate machines and those are well isolated inside the temp dir),
but I see few problems this will cause down the road, so I'm against this
change.

What you are saying is that this change will introduce new  problems or regressions?

Jose
 


> [1] https://docs.yoctoproject.org/bitbake/bitbake-user-manual/bitbake-user-manual-intro.html#executing-a-multiple-configuration-build
>
> Signed-off-by: Jose Quaresma <jose.quaresma@...>
> ---
>  meta-ti-bsp/conf/multiconfig/k3r5.conf | 2 ++
>  1 file changed, 2 insertions(+)
>
> diff --git a/meta-ti-bsp/conf/multiconfig/k3r5.conf b/meta-ti-bsp/conf/multiconfig/k3r5.conf
> index deb07210..0ba163ae 100644
> --- a/meta-ti-bsp/conf/multiconfig/k3r5.conf
> +++ b/meta-ti-bsp/conf/multiconfig/k3r5.conf
> @@ -3,3 +3,5 @@ MAINMACHINE := "${MACHINE}"
>  DEPLOY_DIR_IMAGE = "${DEPLOY_DIR}/images/${MAINMACHINE}"

>  MACHINE:append = "-k3r5"
> +
> +TMPDIR = "${TOPDIR}/tmp-${MACHINE}"
> --
> 2.36.1


--
Best regards,

José Quaresma


Christopher Larson
 

Multi-machine builds in a single tmpdir has been supported by oe since before multiconfig existed, and certainly works fine with multiconfig, it's only other configuration changes, such as distro, that result in conflicts and failures when using multiconfig in a single tmpdir. I don't think it's a good idea to force the matter due to flaws in archiver, personally.


On Wed, Jun 1, 2022 at 2:17 AM Jose Quaresma <quaresma.jose@...> wrote:


Denys Dmytriyenko <denis@...> escreveu no dia terça, 31/05/2022 à(s) 20:33:
On Mon, May 30, 2022 at 03:38:10PM +0100, Jose Quaresma wrote:
> Bitbake manual [1] recomends to use one TMPDIR for each machine:
>
>  "Minimally, each configuration file must define the machine and the
>  temporary directory BitBake uses for the build.
>  Suggested practice dictates that you do not overlap the temporary
>  directories used during the builds."

What specific issue are you trying to fix with this?

The issue that I have is in the OE-core archiver.bbclasse that doesn't support multiconfig when
using the same TMPDIR, this because the shared recipes like gcc, kernel

It can be replicates in kirkstone using the am64xx-evm with the following:

# archive sources for target recipes in conf/local.conf 
INHERIT += "archiver"
COPYLEFT_RECIPE_TYPES = "target"
ARCHIVER_MODE[src] = "original"
ARCHIVER_MODE[diff] = "1"

# and run bitbake with:
bitbake mc:k3r5:gcc-source-11.2.0 gcc-source-11.2.0 -c do_unpack_and_patch -f

more info in following patch:


I'm not aware of any immediate issues (because here multiconfig is used for
completely separate machines and those are well isolated inside the temp dir),
but I see few problems this will cause down the road, so I'm against this
change.

What you are saying is that this change will introduce new  problems or regressions?

Jose
 


> [1] https://docs.yoctoproject.org/bitbake/bitbake-user-manual/bitbake-user-manual-intro.html#executing-a-multiple-configuration-build
>
> Signed-off-by: Jose Quaresma <jose.quaresma@...>
> ---
>  meta-ti-bsp/conf/multiconfig/k3r5.conf | 2 ++
>  1 file changed, 2 insertions(+)
>
> diff --git a/meta-ti-bsp/conf/multiconfig/k3r5.conf b/meta-ti-bsp/conf/multiconfig/k3r5.conf
> index deb07210..0ba163ae 100644
> --- a/meta-ti-bsp/conf/multiconfig/k3r5.conf
> +++ b/meta-ti-bsp/conf/multiconfig/k3r5.conf
> @@ -3,3 +3,5 @@ MAINMACHINE := "${MACHINE}"
>  DEPLOY_DIR_IMAGE = "${DEPLOY_DIR}/images/${MAINMACHINE}"

>  MACHINE:append = "-k3r5"
> +
> +TMPDIR = "${TOPDIR}/tmp-${MACHINE}"
> --
> 2.36.1


--
Best regards,

José Quaresma





--
Christopher Larson
chris_larson@..., chris.larson@..., kergoth@...
Principal Software Engineer, Embedded Linux Solutions, Siemens Digital Industries Software


Denys Dmytriyenko
 

On Wed, Jun 01, 2022 at 09:32:58AM -0700, Christopher Larson wrote:
Multi-machine builds in a single tmpdir has been supported by oe since
before multiconfig existed, and certainly works fine with multiconfig, it's
only other configuration changes, such as distro, that result in conflicts
and failures when using multiconfig in a single tmpdir. I don't think it's
a good idea to force the matter due to flaws in archiver, personally.
Yes, I agree that in this case it is best to fix the archiver class.

The reason everything else just works is because there's already per-machine
separation in many places - deploy, images, work, cache, stamps, etc.

There's also a great re-use of shared components between similar machines
built in the same tmpdir. As Chris says, this always worked even before
multiconfig - you just change MACHINE and re-run bitbake. And multiconfig
simply allows doing it in one invocation of bitbake, that's all.

Denys


On Wed, Jun 1, 2022 at 2:17 AM Jose Quaresma <quaresma.jose@...>
wrote:



Denys Dmytriyenko <denis@...> escreveu no dia terça, 31/05/2022
à(s) 20:33:

On Mon, May 30, 2022 at 03:38:10PM +0100, Jose Quaresma wrote:
Bitbake manual [1] recomends to use one TMPDIR for each machine:

"Minimally, each configuration file must define the machine and the
temporary directory BitBake uses for the build.
Suggested practice dictates that you do not overlap the temporary
directories used during the builds."
What specific issue are you trying to fix with this?
The issue that I have is in the OE-core archiver.bbclasse that doesn't
support multiconfig when
using the same TMPDIR, this because the shared recipes like gcc, kernel

It can be replicates in kirkstone using the am64xx-evm with the following:

# archive sources for target recipes in conf/local.conf
INHERIT += "archiver"
COPYLEFT_RECIPE_TYPES = "target"
ARCHIVER_MODE[src] = "original"
ARCHIVER_MODE[diff] = "1"

# and run bitbake with:
bitbake mc:k3r5:gcc-source-11.2.0 gcc-source-11.2.0 -c do_unpack_and_patch
-f

more info in following patch:
https://lists.openembedded.org/g/openembedded-core/message/166203


I'm not aware of any immediate issues (because here multiconfig is used
for
completely separate machines and those are well isolated inside the temp
dir),
but I see few problems this will cause down the road, so I'm against this
change.
What you are saying is that this change will introduce new problems or
regressions?

Jose




[1]
https://docs.yoctoproject.org/bitbake/bitbake-user-manual/bitbake-user-manual-intro.html#executing-a-multiple-configuration-build

Signed-off-by: Jose Quaresma <jose.quaresma@...>
---
meta-ti-bsp/conf/multiconfig/k3r5.conf | 2 ++
1 file changed, 2 insertions(+)

diff --git a/meta-ti-bsp/conf/multiconfig/k3r5.conf
b/meta-ti-bsp/conf/multiconfig/k3r5.conf
index deb07210..0ba163ae 100644
--- a/meta-ti-bsp/conf/multiconfig/k3r5.conf
+++ b/meta-ti-bsp/conf/multiconfig/k3r5.conf
@@ -3,3 +3,5 @@ MAINMACHINE := "${MACHINE}"
DEPLOY_DIR_IMAGE = "${DEPLOY_DIR}/images/${MAINMACHINE}"

MACHINE:append = "-k3r5"
+
+TMPDIR = "${TOPDIR}/tmp-${MACHINE}"
--
2.36.1

--
Best regards,

José Quaresma



--
Christopher Larson
chris_larson@..., chris.larson@..., kergoth@...
Principal Software Engineer, Embedded Linux Solutions, Siemens Digital
Industries Software
--
Regards,
Denys Dmytriyenko <denis@...>
PGP: 0x420902729A92C964 - https://denix.org/0x420902729A92C964
Fingerprint: 25FC E4A5 8A72 2F69 1186 6D76 4209 0272 9A92 C964


Jose Quaresma
 



Denys Dmytriyenko <denis@...> escreveu no dia quarta, 1/06/2022 à(s) 17:49:
On Wed, Jun 01, 2022 at 09:32:58AM -0700, Christopher Larson wrote:
> Multi-machine builds in a single tmpdir has been supported by oe since
> before multiconfig existed, and certainly works fine with multiconfig, it's
> only other configuration changes, such as distro, that result in conflicts
> and failures when using multiconfig in a single tmpdir. I don't think it's
> a good idea to force the matter due to flaws in archiver, personally.

I will investigate a bit more but for now the only issue I see is on the archiver.
 

Yes, I agree that in this case it is best to fix the archiver class.

I will try this approach instead.
 

The reason everything else just works is because there's already per-machine
separation in many places - deploy, images, work, cache, stamps, etc.

yes, everything seems to work except for the archiver class so for sure the problem is here.
 

There's also a great re-use of shared components between similar machines
built in the same tmpdir. As Chris says, this always worked even before
multiconfig - you just change MACHINE and re-run bitbake. And multiconfig
simply allows doing it in one invocation of bitbake, that's all.

Right and I totally agree.
Thanks for your input which always helps to better understand how this beast called bitbake works ;)

Jose
 

Denys


> On Wed, Jun 1, 2022 at 2:17 AM Jose Quaresma <quaresma.jose@...>
> wrote:
>
> >
> >
> > Denys Dmytriyenko <denis@...> escreveu no dia terça, 31/05/2022
> > à(s) 20:33:
> >
> >> On Mon, May 30, 2022 at 03:38:10PM +0100, Jose Quaresma wrote:
> >> > Bitbake manual [1] recomends to use one TMPDIR for each machine:
> >> >
> >> >  "Minimally, each configuration file must define the machine and the
> >> >  temporary directory BitBake uses for the build.
> >> >  Suggested practice dictates that you do not overlap the temporary
> >> >  directories used during the builds."
> >>
> >> What specific issue are you trying to fix with this?
> >>
> >
> > The issue that I have is in the OE-core archiver.bbclasse that doesn't
> > support multiconfig when
> > using the same TMPDIR, this because the shared recipes like gcc, kernel
> >
> > It can be replicates in kirkstone using the am64xx-evm with the following:
> >
> > # archive sources for target recipes in conf/local.conf
> > INHERIT += "archiver"
> > COPYLEFT_RECIPE_TYPES = "target"
> > ARCHIVER_MODE[src] = "original"
> > ARCHIVER_MODE[diff] = "1"
> >
> > # and run bitbake with:
> > bitbake mc:k3r5:gcc-source-11.2.0 gcc-source-11.2.0 -c do_unpack_and_patch
> > -f
> >
> > more info in following patch:
> > https://lists.openembedded.org/g/openembedded-core/message/166203
> >
> >
> >> I'm not aware of any immediate issues (because here multiconfig is used
> >> for
> >> completely separate machines and those are well isolated inside the temp
> >> dir),
> >> but I see few problems this will cause down the road, so I'm against this
> >> change.
> >>
> >
> > What you are saying is that this change will introduce new  problems or
> > regressions?
> >
> > Jose
> >
> >
> >>
> >>
> >> > [1]
> >> https://docs.yoctoproject.org/bitbake/bitbake-user-manual/bitbake-user-manual-intro.html#executing-a-multiple-configuration-build
> >> >
> >> > Signed-off-by: Jose Quaresma <jose.quaresma@...>
> >> > ---
> >> >  meta-ti-bsp/conf/multiconfig/k3r5.conf | 2 ++
> >> >  1 file changed, 2 insertions(+)
> >> >
> >> > diff --git a/meta-ti-bsp/conf/multiconfig/k3r5.conf
> >> b/meta-ti-bsp/conf/multiconfig/k3r5.conf
> >> > index deb07210..0ba163ae 100644
> >> > --- a/meta-ti-bsp/conf/multiconfig/k3r5.conf
> >> > +++ b/meta-ti-bsp/conf/multiconfig/k3r5.conf
> >> > @@ -3,3 +3,5 @@ MAINMACHINE := "${MACHINE}"
> >> >  DEPLOY_DIR_IMAGE = "${DEPLOY_DIR}/images/${MAINMACHINE}"
> >> >
> >> >  MACHINE:append = "-k3r5"
> >> > +
> >> > +TMPDIR = "${TOPDIR}/tmp-${MACHINE}"
> >> > --
> >> > 2.36.1
> >>
> >
> >
> > --
> > Best regards,
> >
> > José Quaresma
> >
> >
> >
> >
>
> --
> Christopher Larson
> chris_larson@..., chris.larson@..., kergoth@...
> Principal Software Engineer, Embedded Linux Solutions, Siemens Digital
> Industries Software

--
Regards,
Denys Dmytriyenko <denis@...>
PGP: 0x420902729A92C964 - https://denix.org/0x420902729A92C964
Fingerprint: 25FC E4A5 8A72 2F69 1186  6D76 4209 0272 9A92 C964





--
Best regards,

José Quaresma