IMAGE_INSTALL_append workflow


Trevor Woerner
 

This issue has been entered into the yocto bugzilla as:
https://bugzilla.yoctoproject.org/show_bug.cgi?id=3503


Robert P. J. Day
 

On Tue, 27 Nov 2012, Trevor Woerner wrote:

On Tue, Nov 27, 2012 at 12:24 PM, Robert P. J. Day
<rpjday@crashcourse.ca> wrote:
and ran the fetchall again. this time, no strace source was fetched,
so it looks like an issue with simple fetching.
Thank you for your testing and observations.

But I'd like to point out that in my tests, all the relevant files
have already been fetched. My DL_DIR already contains everything to
do with bash, trace-cmd, and various others that I have been
testing. The errors I am getting are not fetching-related errors,
but are errors because do_rootfs can't find any packages (since
packages relating to the recently-appended recipes have not been
built).

Somewhere in the guts of bitbake a modified IMAGE_INSTALL_append
variable is not being noticed; packages aren't being created (as
I've noticed) and sources aren't being downloaded (as you have just
noticed). But somehow do_rootfs is noticing the change and
complaining it can't find the necessary packages.
for what it's worth, if you're building an image that's a variation
of a "core" image, an alternate way of adding packages after the fact
is:

CORE_IMAGE_EXTRA_INSTALL = "sysfsutils strace bash"

i just tested it and, when i added a new entry in that list, at least
the fetching understood and fetched that new source and patched it.
so it doesn't solve the original problem with IMAGE_INSTALL_append,
but feel free to see if that variation works for you.

again, it's only valid if you're building a core image or some
variation thereof.

rday

--

========================================================================
Robert P. J. Day Ottawa, Ontario, CANADA
http://crashcourse.ca

Twitter: http://twitter.com/rpjday
LinkedIn: http://ca.linkedin.com/in/rpjday
========================================================================


Robert P. J. Day
 

On Tue, 27 Nov 2012, Trevor Woerner wrote:

On Tue, Nov 27, 2012 at 12:24 PM, Robert P. J. Day
<rpjday@crashcourse.ca> wrote:
and ran the fetchall again. this time, no strace source was fetched,
so it looks like an issue with simple fetching.
Thank you for your testing and observations.

But I'd like to point out that in my tests, all the relevant files
have already been fetched. My DL_DIR already contains everything to
do with bash, trace-cmd, and various others that I have been
testing. The errors I am getting are not fetching-related errors,
but are errors because do_rootfs can't find any packages (since
packages relating to the recently-appended recipes have not been
built).

Somewhere in the guts of bitbake a modified IMAGE_INSTALL_append
variable is not being noticed; packages aren't being created (as
I've noticed) and sources aren't being downloaded (as you have just
noticed). But somehow do_rootfs is noticing the change and
complaining it can't find the necessary packages.
ah, quite so ... but it's still useful to note that the problem
extends even to the act of fetching. weird.

rday

--

========================================================================
Robert P. J. Day Ottawa, Ontario, CANADA
http://crashcourse.ca

Twitter: http://twitter.com/rpjday
LinkedIn: http://ca.linkedin.com/in/rpjday
========================================================================


Eric Bénard <eric@...>
 

Le Tue, 27 Nov 2012 12:24:06 -0500 (EST),
"Robert P. J. Day" <rpjday@crashcourse.ca> a écrit :
and ran the fetchall again. this time, no strace source was fetched,
so it looks like an issue with simple fetching.
It's not fetched because the build system fails to see the new package
as a dependencie of the image so that's not a fetching issue but a
more global one.

Eric


Trevor Woerner
 

On Tue, Nov 27, 2012 at 12:24 PM, Robert P. J. Day
<rpjday@crashcourse.ca> wrote:
and ran the fetchall again. this time, no strace source was fetched,
so it looks like an issue with simple fetching.

Thank you for your testing and observations.

But I'd like to point out that in my tests, all the relevant files
have already been fetched. My DL_DIR already contains everything to do
with bash, trace-cmd, and various others that I have been testing. The
errors I am getting are not fetching-related errors, but are errors
because do_rootfs can't find any packages (since packages relating to
the recently-appended recipes have not been built).

Somewhere in the guts of bitbake a modified IMAGE_INSTALL_append
variable is not being noticed; packages aren't being created (as I've
noticed) and sources aren't being downloaded (as you have just
noticed). But somehow do_rootfs is noticing the change and complaining
it can't find the necessary packages.


Robert P. J. Day
 

On Tue, 27 Nov 2012, Eren Türkay wrote:

On Tue, Nov 27, 2012 at 08:40:26AM -0500, Trevor Woerner wrote:
Thank you, Eric, for the confirmation.

Just to clarify for anyone following along: the problem isn't with
bash or trace-cmd themselves. If I perform the same steps but use
'bash' first, then 'trace-cmd', adding 'bash' succeeds but later
adding 'trace-cmd' causes 'trace-cmd' to fail in the same way
'bash' did when it was added second.
Since it's confirmed from different people, I think it would be
better to report it using bugzilla. I guess that it's not an
intended behaviour/feature of bitbake. Please inform the bug number
so that anyone who is interested can follow.
here's a test i made based on nothing but fetching. using oe-core,
i selected qemuarm, added this line at the bottom of my local.conf:

IMAGE_INSTALL_append = " sysfsutils strace"

and ran

$ bitbake -c fetchall core-image-minimal

when that was done, i checked my downloads directory for the results
of fetching (and patching) those two packages:

$ ls strace*
strace-4.7.tar.xz strace-4.7.tar.xz.done
$ ls sysfs*
sysfs.sh.done sysfsutils-2.1.0.tar.gz
sysfsutils-2.0.0-class-dup.patch.done sysfsutils-2.1.0.tar.gz.done
$

so that seemed to work fine. so i started over with a clean build
and did it in two stages. first, i reduced the line to:

IMAGE_INSTALL_append = " sysfsutils"

ran the same "fetchall" operation again, and verified that that
content had been fetched:

$ ls sysfs*
sysfs.sh.done sysfsutils-2.1.0.tar.gz
sysfsutils-2.0.0-class-dup.patch.done sysfsutils-2.1.0.tar.gz.done
$

i then *changed* the line in local.conf to:

IMAGE_INSTALL_append = " strace sysfsutils"

and ran the fetchall again. this time, no strace source was fetched,
so it looks like an issue with simple fetching.

rday

--

========================================================================
Robert P. J. Day Ottawa, Ontario, CANADA
http://crashcourse.ca

Twitter: http://twitter.com/rpjday
LinkedIn: http://ca.linkedin.com/in/rpjday
========================================================================


Eren Türkay <eren@...>
 

On Tue, Nov 27, 2012 at 08:40:26AM -0500, Trevor Woerner wrote:
Thank you, Eric, for the confirmation.

Just to clarify for anyone following along: the problem isn't with
bash or trace-cmd themselves. If I perform the same steps but use
'bash' first, then 'trace-cmd', adding 'bash' succeeds but later
adding 'trace-cmd' causes 'trace-cmd' to fail in the same way 'bash'
did when it was added second.
Since it's confirmed from different people, I think it would be better to report
it using bugzilla. I guess that it's not an intended behaviour/feature of
bitbake. Please inform the bug number so that anyone who is interested can
follow.

--
. 73! DE TA1AET
http://erenturkay.com/


Eric Bénard <eric@...>
 

Le Tue, 27 Nov 2012 14:00:34 +0000,
Jonas Jonsson L <jonas.l.jonsson@ericsson.com> a écrit :

I can also confirm this.

A package (I've tried with diffutils & bash) must be built separately before it's added to an image via IMAGE_INSTALL_append.
that seems to work the first time you add something using
IMAGE_INSTALL_append.
Then it fails when you add packages later (bitbake wants to install
the package in the image but doesn't trigger the build of the package
before).

Eric


Jonas Jonsson L <jonas.l.jonsson@...>
 

I can also confirm this.

A package (I've tried with diffutils & bash) must be built separately before it's added to an image via IMAGE_INSTALL_append.

/Jonas - fetched poky today

-----Original Message-----
From: yocto-bounces@yoctoproject.org [mailto:yocto-bounces@yoctoproject.org] On Behalf Of Trevor Woerner
Sent: den 27 november 2012 03:21
To: Eren Türkay
Cc: yocto@yoctoproject.org
Subject: Re: [yocto] IMAGE_INSTALL_append workflow

On Mon, Nov 26, 2012 at 6:52 PM, Trevor Woerner <twoerner@gmail.com> wrote:
I haven't yet figured out if the "PREFERRED_PROVIDER_virtual/kernel =
"linux-yocto-rt"" line might be the culprit.
No, it is not related, this problem doesn't need any fancy configuration features. This problem is reproducible with just the plain qemux86 machine and the core-image-minimal image.
Also, before my latest tests I pulled
the latest git repo.

To reproduce this problem:

- use the latest git repositories
- run "source poky/oe-init-build-env qemux86"
- make no modifications to conf/bblayers.conf
- make no changes to conf/local.conf
- run "bitbake core-image-minimal"

After that completes successfully add the following line to your
conf/local.conf:

IMAGE_INSTALL_append = " trace-cmd"

- re-run the build "bitbake core-image-minimal"

This should complete successfully. When it does, modify the last line of conf/local.conf to be:

IMAGE_INSTALL_append = " trace-cmd bash"

- try re-running the build now "bitbake core-image-minimal". It fails for me as I have detailed in my previous emails.

If I manually do:

$ bitbake bash
$ bitbake core-image-minimal

everything succeeds and my image contains 'bash'. But something tells me there's a bug in there somewhere.
_______________________________________________
yocto mailing list
yocto@yoctoproject.org
https://lists.yoctoproject.org/listinfo/yocto


Trevor Woerner
 

On Tue, Nov 27, 2012 at 5:39 AM, Eric Bénard <eric@eukrea.com> wrote:
I confirm the problem when following this procedure step by step.

Thank you, Eric, for the confirmation.

Just to clarify for anyone following along: the problem isn't with
bash or trace-cmd themselves. If I perform the same steps but use
'bash' first, then 'trace-cmd', adding 'bash' succeeds but later
adding 'trace-cmd' causes 'trace-cmd' to fail in the same way 'bash'
did when it was added second.


Eric Bénard <eric@...>
 

Hi Trevor,

Le Mon, 26 Nov 2012 21:20:30 -0500,
Trevor Woerner <twoerner@gmail.com> a écrit :

On Mon, Nov 26, 2012 at 6:52 PM, Trevor Woerner <twoerner@gmail.com> wrote:
I haven't yet figured out if the "PREFERRED_PROVIDER_virtual/kernel =
"linux-yocto-rt"" line might be the culprit.
No, it is not related, this problem doesn't need any fancy
configuration features. This problem is reproducible
with just the plain qemux86 machine and the core-image-minimal image.
Also, before my latest tests I pulled
the latest git repo.

To reproduce this problem:

- use the latest git repositories
- run "source poky/oe-init-build-env qemux86"
- make no modifications to conf/bblayers.conf
- make no changes to conf/local.conf
- run "bitbake core-image-minimal"

After that completes successfully add the following line to your
conf/local.conf:

IMAGE_INSTALL_append = " trace-cmd"

- re-run the build "bitbake core-image-minimal"

This should complete successfully. When it does, modify the last line
of conf/local.conf to be:

IMAGE_INSTALL_append = " trace-cmd bash"

- try re-running the build now "bitbake core-image-minimal". It fails
for me as I have detailed in my previous emails.

If I manually do:

$ bitbake bash
$ bitbake core-image-minimal

everything succeeds and my image contains 'bash'. But something tells
me there's a bug in there somewhere.
I confirm the problem when following this procedure step by step.

with bitbake -e we see that bash is in RDEPENDS and in IMAGE_INSTALL
with bitbake -g we see that bash is not present in any of the
*depends.dot files

Eric


Trevor Woerner
 

On Mon, Nov 26, 2012 at 6:52 PM, Trevor Woerner <twoerner@gmail.com> wrote:
I haven't yet figured out if the "PREFERRED_PROVIDER_virtual/kernel =
"linux-yocto-rt"" line might be the culprit.
No, it is not related, this problem doesn't need any fancy
configuration features. This problem is reproducible
with just the plain qemux86 machine and the core-image-minimal image.
Also, before my latest tests I pulled
the latest git repo.

To reproduce this problem:

- use the latest git repositories
- run "source poky/oe-init-build-env qemux86"
- make no modifications to conf/bblayers.conf
- make no changes to conf/local.conf
- run "bitbake core-image-minimal"

After that completes successfully add the following line to your
conf/local.conf:

IMAGE_INSTALL_append = " trace-cmd"

- re-run the build "bitbake core-image-minimal"

This should complete successfully. When it does, modify the last line
of conf/local.conf to be:

IMAGE_INSTALL_append = " trace-cmd bash"

- try re-running the build now "bitbake core-image-minimal". It fails
for me as I have detailed in my previous emails.

If I manually do:

$ bitbake bash
$ bitbake core-image-minimal

everything succeeds and my image contains 'bash'. But something tells
me there's a bug in there somewhere.


Trevor Woerner
 

Thank you so much for having a crack at this. Your results have allowed me to dig a bit deeper into this issue.

I'm still trying to narrow down the exact issue but here's what I have so far.

I'm working with the git repository, but I've been playing around with this for about a week, so I'm not 100% up to date. Specifically I'm using:

master:c206007235df3e00c21a1cbf30e8a1bccc07a9e5

Starting with that repository I follow the regular steps:

$ poky/oe-init-build qemux86

then I edit the conf/local.conf to end up with the following (virtually no changes, just the lines at the end):

PARALLEL_MAKE = "-j 8"
MACHINE ??= "qemux86"
DISTRO ?= "poky"
PACKAGE_CLASSES ?= "package_rpm"
EXTRA_IMAGE_FEATURES = "debug-tweaks"
USER_CLASSES ?= "buildstats image-mklibs image-prelink"
PATCHRESOLVE = "noop"
CONF_VERSION = "1"

BB_DANGLINGAPPENDS_WARNONLY = "yes"
PREFERRED_PROVIDER_virtual/kernel = "linux-yocto-rt"

then I perform

$ bitbake core-image-minimal

everything goes well. I then add the following line to the end of my conf/local.conf

IMAGE_INSTALL_append = " trace-cmd"

and run

$ bitbake core-image-minimal

everything builds just fine.

Then I modify that last line as follows:

IMAGE_INSTALL_append = " trace-cmd bash"

now when I try to build I get:

Build Configuration:
BB_VERSION        = "1.16.0"
TARGET_ARCH       = "i586"
TARGET_OS         = "linux"
MACHINE           = "qemux86"
DISTRO            = "poky"
DISTRO_VERSION    = "1.3+snapshot-20121126"
TUNE_FEATURES     = "m32 i586"
TARGET_FPU        = ""
meta             
meta-yocto       
meta-yocto-bsp    = "master:c206007235df3e00c21a1cbf30e8a1bccc07a9e5"

NOTE: Resolving any missing task queue dependencies
NOTE: Preparing runqueue
NOTE: Executing SetScene Tasks
NOTE: Executing RunQueue Tasks
ERROR: Function failed: do_rootfs (see /home/trevor/devel/yocto/gitrepo-method/qemux86/tmp/work/qemux86-poky-linux/core-image-minimal/1.0-r0/temp/log.do_rootfs.19668 for further information)
ERROR: Logfile of failure stored in: /home/trevor/devel/yocto/gitrepo-method/qemux86/tmp/work/qemux86-poky-linux/core-image-minimal/1.0-r0/temp/log.do_rootfs.19668
Log data follows:
| DEBUG: Executing shell function do_rootfs
| Generating solve db for /home/trevor/devel/yocto/gitrepo-method/qemux86/tmp/deploy/rpm/qemux86...
| Generating solve db for /home/trevor/devel/yocto/gitrepo-method/qemux86/tmp/deploy/rpm/i586...
| Generating solve db for /home/trevor/devel/yocto/gitrepo-method/qemux86/tmp/deploy/rpm/all...
|    total:               1      0.000000 MB      2.930406 secs
|    fingerprint:      1104      0.032554 MB      0.063469 secs
|    install:           368      0.000000 MB      0.878790 secs
|    digest:            736      5.357010 MB      0.027508 secs
|    signature:         736      0.000000 MB      1.443277 secs
|    dbadd:             368      0.000000 MB      0.866237 secs
|    dbget:           12620      0.000000 MB      0.018020 secs
|    dbput:             368      3.063072 MB      0.541058 secs
|    readhdr:          3681      6.087218 MB      0.010347 secs
|    hdrload:          1929     10.076294 MB      0.012106 secs
|    hdrget:          70812      0.000000 MB      0.081749 secs
| Generating solve db for /home/trevor/devel/yocto/gitrepo-method/qemux86/tmp/deploy/rpm/all...
| Processing packagegroup-core-boot...
| Processing trace-cmd...
| Processing bash...
| Unable to resolve package bash
| ERROR: some packages were missing
| ERROR: Function failed: do_rootfs (see /home/trevor/devel/yocto/gitrepo-method/qemux86/tmp/work/qemux86-poky-linux/core-image-minimal/1.0-r0/temp/log.do_rootfs.19668 for further information)
ERROR: Task 7 (/home/trevor/devel/yocto/gitrepo-method/poky/meta/recipes-core/images/core-image-minimal.bb, do_rootfs) failed with exit code '1'
NOTE: Tasks Summary: Attempted 1443 tasks of which 1442 didn't need to be rerun and 1 failed.
No currently running tasks (1442 of 1444)

Summary: 1 task failed:
  /home/trevor/devel/yocto/gitrepo-method/poky/meta/recipes-core/images/core-image-minimal.bb, do_rootfs
Summary: There was 1 ERROR message shown, returning a non-zero exit code.

Can anyone reproduce this? It doesn't make sense.

Initially when I first saw this problem, I had been using building "core-image-rt" so I thought it had something to do with that image. But my tests from today seem to indicate that it doesn't since I was able to reproduce my problem with "core-image-minimal" too.

I haven't yet figured out if the "PREFERRED_PROVIDER_virtual/kernel = "linux-yocto-rt"" line might be the culprit.


Robert P. J. Day
 

On Mon, 26 Nov 2012, Aws Ismail wrote:

Have you also tried this?

IMAGE_INSTALL_append += "foo"
GAAAAAHHHHHH!!! :-)

rday

--

========================================================================
Robert P. J. Day Ottawa, Ontario, CANADA
http://crashcourse.ca

Twitter: http://twitter.com/rpjday
LinkedIn: http://ca.linkedin.com/in/rpjday
========================================================================


Aws Ismail <aws.ismail@...>
 

On 11/26/2012 02:04 PM, Eren Türkay wrote:
On Mon, Nov 26, 2012 at 10:47:31AM -0500, Trevor Woerner wrote:
I'm kinda surprised nobody jumped in on this one; either my question
is so silly it isn't worth anyone's time, or I'm the only one who
wants to add packages to an image after a full build. I know my
question has nothing to do with trace-cmd specifically, adding any
package after a full build results in the same problem.
I don't think you are the first person who wants to add a package to image after
the image has been built :)

The obvious problem would have been leading spaces in IMAGE_INSTALL_append but
you explicitly stated its correct. Normally, the class that's responsible for
image creation (image.bbclass) depends on the packages defined in IMAGE_INSTALL
so that the packages are built before the image creation begins.

image.bbclass, line 9
---------------------
RDEPENDS += "${IMAGE_INSTALL} ${LINGUAS_INSTALL} ${NORMAL_FEATURE_INSTALL}
${ROOTFS_BOOTSTRAP_INSTALL}"

On irc (#oe), Eric Benard tried IMAGE_INSTALL_append = " trace-cmd" after a full
build and reported that it built fine. I'm CCing him.
Have you also tried this?

IMAGE_INSTALL_append += "foo"

Aws\

I believe that it will be better to include your <image> file and the files that
you changed so that it will be easier to debug. Mentioning your layers,
configuration, version of Yocto you use will also be helpful.

Regards,
Eren



_______________________________________________
yocto mailing list
yocto@...
https://lists.yoctoproject.org/listinfo/yocto


-- 


Eren Türkay <eren@...>
 

On Mon, Nov 26, 2012 at 10:47:31AM -0500, Trevor Woerner wrote:
I'm kinda surprised nobody jumped in on this one; either my question
is so silly it isn't worth anyone's time, or I'm the only one who
wants to add packages to an image after a full build. I know my
question has nothing to do with trace-cmd specifically, adding any
package after a full build results in the same problem.
I don't think you are the first person who wants to add a package to image after
the image has been built :)

The obvious problem would have been leading spaces in IMAGE_INSTALL_append but
you explicitly stated its correct. Normally, the class that's responsible for
image creation (image.bbclass) depends on the packages defined in IMAGE_INSTALL
so that the packages are built before the image creation begins.

image.bbclass, line 9
---------------------
RDEPENDS += "${IMAGE_INSTALL} ${LINGUAS_INSTALL} ${NORMAL_FEATURE_INSTALL}
${ROOTFS_BOOTSTRAP_INSTALL}"

On irc (#oe), Eric Benard tried IMAGE_INSTALL_append = " trace-cmd" after a full
build and reported that it built fine. I'm CCing him.

I believe that it will be better to include your <image> file and the files that
you changed so that it will be easier to debug. Mentioning your layers,
configuration, version of Yocto you use will also be helpful.

Regards,
Eren

--
. 73! DE TA1AET
http://erenturkay.com/


Trevor Woerner
 

I'm kinda surprised nobody jumped in on this one; either my question
is so silly it isn't worth anyone's time, or I'm the only one who
wants to add packages to an image after a full build. I know my
question has nothing to do with trace-cmd specifically, adding any
package after a full build results in the same problem.

It would certainly seem odd that the yocto workflow assumes you always
know 100% ahead of time which packages you want in your final image. I
know when I was playing around with Hob this wasn't an issue: I would
simply add the new packages and Hob would build them and add them to
the image.

From what I have seen from skimming the documentation, the
documentation talks at length about how to add a new package to the
configuration, but it doesn't say what commands to issue after
modifying the config files. Obviously running "bitbake <image>" alone
doesn't work.

On Fri, Nov 23, 2012 at 1:27 PM, Trevor Woerner <twoerner@gmail.com> wrote:
There have been many emails and lots of documentation describing how
to add a package to a build, and that's all great.

Today I was playing around with an image I had already built and
decided I wanted to add a new package to the mix: trace-cmd. I added "
trace-cmd" (with the required leading space) to IMAGE_INSTALL_append
and re-ran my "bitbake <image>" command.

It failed:

Unable to resolve package trace-cmd

If I perform:

$ bitbake trace-cmd

and then

$ bitbake <image>

it works fine. What am I missing?

As it turns out, if I knew before my first build that I wanted
trace-cmd, I could have just added it as described above and
everything would have worked as well.

Best regards,
Trevor


Trevor Woerner
 

There have been many emails and lots of documentation describing how
to add a package to a build, and that's all great.

Today I was playing around with an image I had already built and
decided I wanted to add a new package to the mix: trace-cmd. I added "
trace-cmd" (with the required leading space) to IMAGE_INSTALL_append
and re-ran my "bitbake <image>" command.

It failed:

Unable to resolve package trace-cmd

If I perform:

$ bitbake trace-cmd

and then

$ bitbake <image>

it works fine. What am I missing?

As it turns out, if I knew before my first build that I wanted
trace-cmd, I could have just added it as described above and
everything would have worked as well.

Best regards,
Trevor