Hi,
We are trying to install packages on our arm64 architecture directly from debian.org.
Our sources.list contains "deb [ arch=arm64 ] http://ftp.de.debian.org/debian buster main".

1. When executing apt-get update we get the following output:

root@imx8mq-var-dart:~# apt-get update

Get:1 http://ftp.de.debian.org/debian buster InRelease [122 kB]

Ign:1 http://ftp.de.debian.org/debian buster InRelease

Get:2 http://ftp.de.debian.org/debian buster/main arm64 Packages [7737 kB]

Fetched 7858 kB in 5s (1424 kB/s)

Reading package lists... Done

W: GPG error: http://ftp.de.debian.org/debian buster InRelease: Unknown error executing apt-key

W: The repository 'http://ftp.de.debian.org/debian buster InRelease' is not signed.

N: Data from such a repository can't be authenticated and is therefore potentially dangerous to use.

N: See apt-secure(8) manpage for repository creation and user configuration details.

E: Failed to fetch /

E: Some index files failed to download. They have been ignored, or old ones used instead.

We did not worry about the missing gpg authentification since we were able to apt-cache search this source successfully.

2. When we tried to apt-get install python3-psutil we were promted with a large list of mainly perl related packages that were about to be removed:

12 upgraded, 66 newly installed, 531 to remove and 212 not upgraded

[complete output in attachments.]
Allowing these changes to be made led to dpkg trying to overwrite files that are also in other packages. This caused the command to fail.

3. We then tried to pass the "--force-overwrite" option to dpkg with the command:

sudo apt-get -o Dpkg::Options::="--force-overwrite" install python3-psutil

Is this a known issue? Are we missing an important configuration for apt within Yocto?
Is there another way to install and setup apt-get in our image that does not include Yocto?

Thanks,
Stefan

FOLDER_ID has already been assigned a defalut value "1", so there is no
need add 'or "1"' here.

Signed-off-by: Li Xiaoming <lixm.fnst@...>
---
classes/fossology-rest.bbclass | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/classes/fossology-rest.bbclass b/classes/fossology-rest.bbclass
index 5c5ef70..69f4998 100644
--- a/classes/fossology-rest.bbclass
+++ b/classes/fossology-rest.bbclass
@@ -230,7 +230,7 @@ def get_folder_id(d):
folder_name = d.getVar('FOLDER_NAME')
folder_id = create_folder(d, folder_name)
else:
- folder_id = (d.getVar('FOLDER_ID', True) or "1")
+ folder_id = d.getVar('FOLDER_ID', False)

bb.note("Folder Id = " + str(folder_id))
return str(folder_id)
--
2.17.1

To answer your others questions... see below

On 2020-03-28 8:26 a.m., Konrad Weihmann wrote:

Hi,

I'm facing the following error message sporadically on all branches I tried so far (master, zeus, warrior and thud)

The stack trace of python calls that resulted in this exception/failure was:
File: 'exec_python_func() autogenerated', lineno: 2, function: <module>
      0001:
  *** 0002:extend_recipe_sysroot(d)
      0003:
File: '/build/poky/meta/classes/staging.bbclass', lineno: 551, function: extend_recipe_sysroot
      0547:                    dest = newmanifest[l]
      0548:                    if l.endswith("/"):
      0549:                        staging_copydir(l, targetdir, dest, seendirs)
      0550:                        continue
  *** 0551:                    staging_copyfile(l, targetdir, dest, postinsts, seendirs)
      0552:
      0553:    bb.note("Installed into sysroot: %s" % str(msg_adding))
      0554:    bb.note("Skipping as already exists in sysroot: %s" % str(msg_exists))
      0555:
File: '/build/poky/meta/classes/staging.bbclass', lineno: 152, function: staging_copyfile
      0148:        os.symlink(linkto, dest)
      0149:        #bb.warn(c)
      0150:    else:
      0151:        try:
  *** 0152:            os.link(c, dest)
      0153:        except OSError as err:
      0154:            if err.errno == errno.EXDEV:
      0155:                bb.utils.copyfile(c, dest)
      0156:            else:
Exception: FileNotFoundError: [Errno 2] No such file or directory: '/build/poky/build/tmp/sysroots-components/x86_64/python3-msgcheck-native/usr/lib/python3.7/site-packages/msgcheck/__pycache__/__init__.cpython-37.pyc' -> '/build/poky/build/tmp/work/qemux86_64-mine-linux/core-image-minimal-mine/1.0-r0/recipe-sysroot-native/usr/lib/python3.7/site-packages/msgcheck/__pycache__/__init__.cpython-37.pyc'

I already had a look at the manifest

cat manifest-x86_64-python3-msgcheck-native.populate_sysroot
/build/poky/build/tmp/sysroots-components/x86_64/python3-msgcheck-native/usr/lib/python3.7/site-packages/msgcheck/__init__.py
/build/poky/build/tmp/sysroots-components/x86_64/python3-msgcheck-native/usr/lib/python3.7/site-packages/msgcheck/po.py
/build/poky/build/tmp/sysroots-components/x86_64/python3-msgcheck-native/usr/lib/python3.7/site-packages/msgcheck/msgcheck.py
/build/poky/build/tmp/sysroots-components/x86_64/python3-msgcheck-native/usr/lib/python3.7/site-packages/msgcheck/__pycache__/__init__.cpython-37.pyc
/build/poky/build/tmp/sysroots-components/x86_64/python3-msgcheck-native/usr/lib/python3.7/site-packages/msgcheck/__pycache__/po.cpython-37.pyc
/build/poky/build/tmp/sysroots-components/x86_64/python3-msgcheck-native/usr/lib/python3.7/site-packages/msgcheck/__pycache__/msgcheck.cpython-37.pyc
/build/poky/build/tmp/sysroots-components/x86_64/python3-msgcheck-native/usr/lib/python3.7/site-packages/msgcheck-2.8-py3.7.egg-info/dependency_links.txt
/build/poky/build/tmp/sysroots-components/x86_64/python3-msgcheck-native/usr/lib/python3.7/site-packages/msgcheck-2.8-py3.7.egg-info/requires.txt
/build/poky/build/tmp/sysroots-components/x86_64/python3-msgcheck-native/usr/lib/python3.7/site-packages/msgcheck-2.8-py3.7.egg-info/top_level.txt
/build/poky/build/tmp/sysroots-components/x86_64/python3-msgcheck-native/usr/lib/python3.7/site-packages/msgcheck-2.8-py3.7.egg-info/SOURCES.txt
/build/poky/build/tmp/sysroots-components/x86_64/python3-msgcheck-native/usr/lib/python3.7/site-packages/msgcheck-2.8-py3.7.egg-info/PKG-INFO
/build/poky/build/tmp/sysroots-components/x86_64/python3-msgcheck-native/usr/lib/python3.7/site-packages/msgcheck-2.8-py3.7.egg-info/entry_points.txt
/build/poky/build/tmp/sysroots-components/x86_64/python3-msgcheck-native/usr/bin/msgcheck
/build/poky/build/tmp/sysroots-components/x86_64/python3-msgcheck-native/sysroot-providers/python3-msgcheck-native
/build/poky/build/tmp/sysroots-components/x86_64/python3-msgcheck-native/usr/lib/python3.7/site-packages/msgcheck-2.8-py3.7.egg-info/
/build/poky/build/tmp/sysroots-components/x86_64/python3-msgcheck-native/usr/lib/python3.7/site-packages/msgcheck/__pycache__/
/build/poky/build/tmp/sysroots-components/x86_64/python3-msgcheck-native/usr/lib/python3.7/site-packages/msgcheck/
/build/poky/build/tmp/sysroots-components/x86_64/python3-msgcheck-native/usr/lib/python3.7/site-packages/
/build/poky/build/tmp/sysroots-components/x86_64/python3-msgcheck-native/usr/lib/python3.7/
/build/poky/build/tmp/sysroots-components/x86_64/python3-msgcheck-native/sysroot-providers/
/build/poky/build/tmp/sysroots-components/x86_64/python3-msgcheck-native/usr/share/
/build/poky/build/tmp/sysroots-components/x86_64/python3-msgcheck-native/usr/bin/
/build/poky/build/tmp/sysroots-components/x86_64/python3-msgcheck-native/usr/lib/
/build/poky/build/tmp/sysroots-components/x86_64/python3-msgcheck-native/usr/

which states the file should be there, but when doing

find /build/poky/build/tmp/sysroots-components/x86_64/python3-msgcheck-native/usr/
/build/poky/build/tmp/sysroots-components/x86_64/python3-msgcheck-native/usr/
/build/poky/build/tmp/sysroots-components/x86_64/python3-msgcheck-native/usr/lib
/build/poky/build/tmp/sysroots-components/x86_64/python3-msgcheck-native/usr/lib/python3.7
/build/poky/build/tmp/sysroots-components/x86_64/python3-msgcheck-native/usr/lib/python3.7/site-packages
/build/poky/build/tmp/sysroots-components/x86_64/python3-msgcheck-native/usr/lib/python3.7/site-packages/msgcheck
/build/poky/build/tmp/sysroots-components/x86_64/python3-msgcheck-native/usr/lib/python3.7/site-packages/msgcheck/__init__.py
/build/poky/build/tmp/sysroots-components/x86_64/python3-msgcheck-native/usr/lib/python3.7/site-packages/msgcheck/__pycache__
/build/poky/build/tmp/sysroots-components/x86_64/python3-msgcheck-native/usr/lib/python3.7/site-packages/msgcheck/__pycache__/po.cpython-37.pyc
/build/poky/build/tmp/sysroots-components/x86_64/python3-msgcheck-native/usr/lib/python3.7/site-packages/msgcheck/__pycache__/msgcheck.cpython-37.pyc
/build/poky/build/tmp/sysroots-components/x86_64/python3-msgcheck-native/usr/lib/python3.7/site-packages/msgcheck/po.py
/build/poky/build/tmp/sysroots-components/x86_64/python3-msgcheck-native/usr/lib/python3.7/site-packages/msgcheck/msgcheck.py
/build/poky/build/tmp/sysroots-components/x86_64/python3-msgcheck-native/usr/lib/python3.7/site-packages/msgcheck-2.8-py3.7.egg-info
/build/poky/build/tmp/sysroots-components/x86_64/python3-msgcheck-native/usr/lib/python3.7/site-packages/msgcheck-2.8-py3.7.egg-info/dependency_links.txt
/build/poky/build/tmp/sysroots-components/x86_64/python3-msgcheck-native/usr/lib/python3.7/site-packages/msgcheck-2.8-py3.7.egg-info/requires.txt
/build/poky/build/tmp/sysroots-components/x86_64/python3-msgcheck-native/usr/lib/python3.7/site-packages/msgcheck-2.8-py3.7.egg-info/top_level.txt
/build/poky/build/tmp/sysroots-components/x86_64/python3-msgcheck-native/usr/lib/python3.7/site-packages/msgcheck-2.8-py3.7.egg-info/SOURCES.txt
/build/poky/build/tmp/sysroots-components/x86_64/python3-msgcheck-native/usr/lib/python3.7/site-packages/msgcheck-2.8-py3.7.egg-info/PKG-INFO
/build/poky/build/tmp/sysroots-components/x86_64/python3-msgcheck-native/usr/lib/python3.7/site-packages/msgcheck-2.8-py3.7.egg-info/entry_points.txt
/build/poky/build/tmp/sysroots-components/x86_64/python3-msgcheck-native/usr/bin
/build/poky/build/tmp/sysroots-components/x86_64/python3-msgcheck-native/usr/bin/msgcheck
/build/poky/build/tmp/sysroots-components/x86_64/python3-msgcheck-native/usr/share

the file isn't there.

This happens to random python packages compiled as native (sometimes even for python-native itself), but (afaik) not for cross or target packages (I'm pretty sure because of the different packaging applied).
So I digged a little into the code classes/sstate.bbclass:sstate_install, which seems to create the sysroot-component dir and the manifest.
There is a gap between the manifest creation (line 285) and the hardlinking (line till 311).

Now my question is there any place where a file potentially could get lost? - at first glance there shouldn't be one - I have to admit that I don't fully understand all this subprocess magic in lib/oe/path.py:copyhardlinktree.
What I do to fix the issue is reopening the manifest and double check for missing files and remove them from the manifest, but this
feels wrong - so any advise is welcome...

Hope that someone more familiar with the topic could have a look.

Hi Konrad,

I'm not really familiar with that code but it's being run buy 1000s of
builder around the world so let's try to eliminate a few possibilities.

When did you start having this problem?

Since the start of the test distribution I'm working on. But also for plain poky builds if I forcefully inject all of the python-native site-packages via local.conf (DEPENDS_class-native += "..."), without actually using them in the recipe scope

How often do you think it's happening: 1 in 3 builds, 1 in 10?

See the other mail - looks like it heavily depends on the host

Tell us about your machine: OS,version, disk, CPUs, ram

See the other mail

Do you do anything special in your conf dir? Send local.conf perhaps.

No custom modification (just for testing the DEPENDS-injection)

Do you have any local bbappends or commits on top of poky or
in other layers?

No

Have you tried to simplify the build to eliminate problems
potentially caused by other layers?

I did - see above

Are you able to reproduce the problem on more than one build machine?

See the other mail

Are you able to reproduce the problem on a different Linux distro?

Not really - Debian 9 was fine all other Hosts are Ubuntu based

Are there other builds or users on the machine that may be causing
extra load?

No the hosts are just being poorly equipped - at least the ones that produce this issue

../Randy

Thanks

Konrad

HI,

sorry I didn't specify it further.
To me it seems to be a corner case scenario.

This happens when doing a from scratch build with a lot on native python libraries (as it is setup in a test distribution I'm working on).
Roughly it pulls in ~100 native python site packages.

The setup where I almost every time can reproduce the issue

- Ubuntu 16.04 or 18.04 container
- 2 VCPUs (can't say what it actually is)
- 14GB of hard disk (I guess it's some virtual storage)
- 12GB of RAM

-> as offered by GitHub's Actions Azure interface

The setup where it occasionally happens

- Ubuntu 19.04/19.10
- 8 cores i7-6th gen
- 16GB Ram
- some Samsung-SSD with more than enough space

The setup where it never happened so far

- Ubuntu 18.04
- 48 core threadripper
- lots of RAM and SSD

and

- Debian 9
- 24 core XEN guest
- >80GB of RAM
- HDD doesn't matter as it was a ramdisk only build

This doesn't happen (afaik) on differential builds when the initial one is good (same for sstate cache backed builds).

I just wanted to know if somebody ever encountered the same issue.
As the nature of this issue is hard to catch, I'm open to ideas...
What currently works for me is to remove all python cache files before they are packaged - at least none of my recent build failed so far.

Any ideas how to tackle this problem, any suggestions?

Best
Konrad

As folder_id can not be specified be in conf file, we do not need
getVar to read it.

Signed-off-by: Li Xiaoming <lixm.fnst@...>
---
classes/fossology-rest.bbclass | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/classes/fossology-rest.bbclass b/classes/fossology-rest.bbclass
index 5c5ef70..c8d29ff 100644
--- a/classes/fossology-rest.bbclass
+++ b/classes/fossology-rest.bbclass
@@ -230,7 +230,7 @@ def get_folder_id(d):
folder_name = d.getVar('FOLDER_NAME')
folder_id = create_folder(d, folder_name)
else:
- folder_id = (d.getVar('FOLDER_ID', True) or "1")
+ folder_id = FOLDER_ID

bb.note("Folder Id = " + str(folder_id))
return str(folder_id)
--
2.17.1

Signed-off-by: Yi Zhao <yi.zhao@...>
---
conf/layer.conf | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/conf/layer.conf b/conf/layer.conf
index cdee27a..da24359 100644
--- a/conf/layer.conf
+++ b/conf/layer.conf
@@ -23,7 +23,7 @@ BBFILE_PRIORITY_selinux = "5"
# cause compatibility issues with other layers
LAYERVERSION_selinux = "1"

-LAYERSERIES_COMPAT_selinux = "zeus"
+LAYERSERIES_COMPAT_selinux = "dunfell"

LAYERDEPENDS_selinux = " \
core \
--
2.17.1

On 2020-03-28 8:26 a.m., Konrad Weihmann wrote:

Hi,
I'm facing the following error message sporadically on all branches I tried so far (master, zeus, warrior and thud)
The stack trace of python calls that resulted in this exception/failure was:
File: 'exec_python_func() autogenerated', lineno: 2, function: <module>
     0001:
 *** 0002:extend_recipe_sysroot(d)
     0003:
File: '/build/poky/meta/classes/staging.bbclass', lineno: 551, function: extend_recipe_sysroot
     0547:                    dest = newmanifest[l]
     0548:                    if l.endswith("/"):
     0549:                        staging_copydir(l, targetdir, dest, seendirs)
     0550:                        continue
 *** 0551:                    staging_copyfile(l, targetdir, dest, postinsts, seendirs)
     0552:
     0553:    bb.note("Installed into sysroot: %s" % str(msg_adding))
     0554:    bb.note("Skipping as already exists in sysroot: %s" % str(msg_exists))
     0555:
File: '/build/poky/meta/classes/staging.bbclass', lineno: 152, function: staging_copyfile
     0148:        os.symlink(linkto, dest)
     0149:        #bb.warn(c)
     0150:    else:
     0151:        try:
 *** 0152:            os.link(c, dest)
     0153:        except OSError as err:
     0154:            if err.errno == errno.EXDEV:
     0155:                bb.utils.copyfile(c, dest)
     0156:            else:
Exception: FileNotFoundError: [Errno 2] No such file or directory: '/build/poky/build/tmp/sysroots-components/x86_64/python3-msgcheck-native/usr/lib/python3.7/site-packages/msgcheck/__pycache__/__init__.cpython-37.pyc' -> '/build/poky/build/tmp/work/qemux86_64-mine-linux/core-image-minimal-mine/1.0-r0/recipe-sysroot-native/usr/lib/python3.7/site-packages/msgcheck/__pycache__/__init__.cpython-37.pyc'
I already had a look at the manifest
cat manifest-x86_64-python3-msgcheck-native.populate_sysroot
/build/poky/build/tmp/sysroots-components/x86_64/python3-msgcheck-native/usr/lib/python3.7/site-packages/msgcheck/__init__.py
/build/poky/build/tmp/sysroots-components/x86_64/python3-msgcheck-native/usr/lib/python3.7/site-packages/msgcheck/po.py
/build/poky/build/tmp/sysroots-components/x86_64/python3-msgcheck-native/usr/lib/python3.7/site-packages/msgcheck/msgcheck.py
/build/poky/build/tmp/sysroots-components/x86_64/python3-msgcheck-native/usr/lib/python3.7/site-packages/msgcheck/__pycache__/__init__.cpython-37.pyc
/build/poky/build/tmp/sysroots-components/x86_64/python3-msgcheck-native/usr/lib/python3.7/site-packages/msgcheck/__pycache__/po.cpython-37.pyc
/build/poky/build/tmp/sysroots-components/x86_64/python3-msgcheck-native/usr/lib/python3.7/site-packages/msgcheck/__pycache__/msgcheck.cpython-37.pyc
/build/poky/build/tmp/sysroots-components/x86_64/python3-msgcheck-native/usr/lib/python3.7/site-packages/msgcheck-2.8-py3.7.egg-info/dependency_links.txt
/build/poky/build/tmp/sysroots-components/x86_64/python3-msgcheck-native/usr/lib/python3.7/site-packages/msgcheck-2.8-py3.7.egg-info/requires.txt
/build/poky/build/tmp/sysroots-components/x86_64/python3-msgcheck-native/usr/lib/python3.7/site-packages/msgcheck-2.8-py3.7.egg-info/top_level.txt
/build/poky/build/tmp/sysroots-components/x86_64/python3-msgcheck-native/usr/lib/python3.7/site-packages/msgcheck-2.8-py3.7.egg-info/SOURCES.txt
/build/poky/build/tmp/sysroots-components/x86_64/python3-msgcheck-native/usr/lib/python3.7/site-packages/msgcheck-2.8-py3.7.egg-info/PKG-INFO
/build/poky/build/tmp/sysroots-components/x86_64/python3-msgcheck-native/usr/lib/python3.7/site-packages/msgcheck-2.8-py3.7.egg-info/entry_points.txt
/build/poky/build/tmp/sysroots-components/x86_64/python3-msgcheck-native/usr/bin/msgcheck
/build/poky/build/tmp/sysroots-components/x86_64/python3-msgcheck-native/sysroot-providers/python3-msgcheck-native
/build/poky/build/tmp/sysroots-components/x86_64/python3-msgcheck-native/usr/lib/python3.7/site-packages/msgcheck-2.8-py3.7.egg-info/
/build/poky/build/tmp/sysroots-components/x86_64/python3-msgcheck-native/usr/lib/python3.7/site-packages/msgcheck/__pycache__/
/build/poky/build/tmp/sysroots-components/x86_64/python3-msgcheck-native/usr/lib/python3.7/site-packages/msgcheck/
/build/poky/build/tmp/sysroots-components/x86_64/python3-msgcheck-native/usr/lib/python3.7/site-packages/
/build/poky/build/tmp/sysroots-components/x86_64/python3-msgcheck-native/usr/lib/python3.7/
/build/poky/build/tmp/sysroots-components/x86_64/python3-msgcheck-native/sysroot-providers/
/build/poky/build/tmp/sysroots-components/x86_64/python3-msgcheck-native/usr/share/
/build/poky/build/tmp/sysroots-components/x86_64/python3-msgcheck-native/usr/bin/
/build/poky/build/tmp/sysroots-components/x86_64/python3-msgcheck-native/usr/lib/
/build/poky/build/tmp/sysroots-components/x86_64/python3-msgcheck-native/usr/
which states the file should be there, but when doing
find /build/poky/build/tmp/sysroots-components/x86_64/python3-msgcheck-native/usr/
/build/poky/build/tmp/sysroots-components/x86_64/python3-msgcheck-native/usr/
/build/poky/build/tmp/sysroots-components/x86_64/python3-msgcheck-native/usr/lib
/build/poky/build/tmp/sysroots-components/x86_64/python3-msgcheck-native/usr/lib/python3.7
/build/poky/build/tmp/sysroots-components/x86_64/python3-msgcheck-native/usr/lib/python3.7/site-packages
/build/poky/build/tmp/sysroots-components/x86_64/python3-msgcheck-native/usr/lib/python3.7/site-packages/msgcheck
/build/poky/build/tmp/sysroots-components/x86_64/python3-msgcheck-native/usr/lib/python3.7/site-packages/msgcheck/__init__.py
/build/poky/build/tmp/sysroots-components/x86_64/python3-msgcheck-native/usr/lib/python3.7/site-packages/msgcheck/__pycache__
/build/poky/build/tmp/sysroots-components/x86_64/python3-msgcheck-native/usr/lib/python3.7/site-packages/msgcheck/__pycache__/po.cpython-37.pyc
/build/poky/build/tmp/sysroots-components/x86_64/python3-msgcheck-native/usr/lib/python3.7/site-packages/msgcheck/__pycache__/msgcheck.cpython-37.pyc
/build/poky/build/tmp/sysroots-components/x86_64/python3-msgcheck-native/usr/lib/python3.7/site-packages/msgcheck/po.py
/build/poky/build/tmp/sysroots-components/x86_64/python3-msgcheck-native/usr/lib/python3.7/site-packages/msgcheck/msgcheck.py
/build/poky/build/tmp/sysroots-components/x86_64/python3-msgcheck-native/usr/lib/python3.7/site-packages/msgcheck-2.8-py3.7.egg-info
/build/poky/build/tmp/sysroots-components/x86_64/python3-msgcheck-native/usr/lib/python3.7/site-packages/msgcheck-2.8-py3.7.egg-info/dependency_links.txt
/build/poky/build/tmp/sysroots-components/x86_64/python3-msgcheck-native/usr/lib/python3.7/site-packages/msgcheck-2.8-py3.7.egg-info/requires.txt
/build/poky/build/tmp/sysroots-components/x86_64/python3-msgcheck-native/usr/lib/python3.7/site-packages/msgcheck-2.8-py3.7.egg-info/top_level.txt
/build/poky/build/tmp/sysroots-components/x86_64/python3-msgcheck-native/usr/lib/python3.7/site-packages/msgcheck-2.8-py3.7.egg-info/SOURCES.txt
/build/poky/build/tmp/sysroots-components/x86_64/python3-msgcheck-native/usr/lib/python3.7/site-packages/msgcheck-2.8-py3.7.egg-info/PKG-INFO
/build/poky/build/tmp/sysroots-components/x86_64/python3-msgcheck-native/usr/lib/python3.7/site-packages/msgcheck-2.8-py3.7.egg-info/entry_points.txt
/build/poky/build/tmp/sysroots-components/x86_64/python3-msgcheck-native/usr/bin
/build/poky/build/tmp/sysroots-components/x86_64/python3-msgcheck-native/usr/bin/msgcheck
/build/poky/build/tmp/sysroots-components/x86_64/python3-msgcheck-native/usr/share
the file isn't there.
This happens to random python packages compiled as native (sometimes even for python-native itself), but (afaik) not for cross or target packages (I'm pretty sure because of the different packaging applied).
So I digged a little into the code classes/sstate.bbclass:sstate_install, which seems to create the sysroot-component dir and the manifest.
There is a gap between the manifest creation (line 285) and the hardlinking (line till 311).
Now my question is there any place where a file potentially could get lost? - at first glance there shouldn't be one - I have to admit that I don't fully understand all this subprocess magic in lib/oe/path.py:copyhardlinktree.
What I do to fix the issue is reopening the manifest and double check for missing files and remove them from the manifest, but this
feels wrong - so any advise is welcome...
Hope that someone more familiar with the topic could have a look.

Hi Konrad,

I'm not really familiar with that code but it's being run buy 1000s of
builder around the world so let's try to eliminate a few possibilities.

When did you start having this problem?
How often do you think it's happening: 1 in 3 builds, 1 in 10?
Tell us about your machine: OS,version, disk, CPUs, ram
Do you do anything special in your conf dir? Send local.conf perhaps.
Do you have any local bbappends or commits on top of poky or
in other layers?
Have you tried to simplify the build to eliminate problems
potentially caused by other layers?
Are you able to reproduce the problem on more than one build machine?
Are you able to reproduce the problem on a different Linux distro?
Are there other builds or users on the machine that may be causing
extra load?


../Randy

Thanks
Konrad

--
# Randy MacLeod
# Wind River Linux

Could someone please review my patches ? Thanks.

Could someone please fix this simple piece?

Merged. Thank you.

On 4/1/20 2:20 PM, Joel Winarske wrote:

Hi Konrad,
Thanks, that helps a lot.  It works!  I used straight split() which prevented the first value from being null.
Yes, I agree less pythonic :)  The only recipe I've seen building gn based projects is OSSystem's meta-browser/chromium.  I followed it's pattern, as the Flutter Engine is a derivative of Chrome.
The issue with these large gn based projects is that they're self contained builds.  A build system within a build system.  I must admit though the gclient DEPS download scheme is efficient.  I initially looked at parsing out the DEPs file into a large SRC_URI list, but quickly hit Yocto serializing them.  Perhaps OE could benefit from tighter integration with gn.

perhaps a generic gn class might be helpful, there have been attemps to add depot_tools/glient to fetchers in past but there is no consistent apis there, it kept on changing in myriad different ways. gn is more or less same story, its too tightly integrated into the software its building, we should abstract it if we can.

I'm open to any other alternative approaches.
Cheers,
Joel
On Wed, Apr 1, 2020 at 1:09 PM Konrad Weihmann <kweihmann@... <mailto:kweihmann@...>> wrote:
Hi Joel,
okay, I see - although I have to admit the argparse idea looks very
weird, the error is that according to
https://docs.python.org/3/library/argparse.html#argparse.ArgumentParser.parse_args
the pkgconfig_args variable should be a list not a single string.
so doing pkgconfig_args = d.getVar("PACKAGECONFIG_CONFARGS").split("
") should fix the issue.
But from a bitbake perspective you should really think about
choosing a less pythonic version
Best
Konrad
On 01.04.20 22:00, Joel Winarske wrote:

Hi Konrad,

I left out the details of the path encoding :)  Effectively I was
attempting to duplicate get_out_dir() as found here:
https://github.com/flutter/engine/blob/master/tools/gn

My recipe is here:
https://github.com/jwinarske/meta-flutter/blob/zeus/recipes-graphics/flutter-engine/flutter-engine_git.bb

Cheers,
Joel

On Wed, Apr 1, 2020 at 10:46 AM Konrad Weihmann
<kweihmann@... <mailto:kweihmann@...>> wrote:

There is a single tick missing, but I'm sure you get the point

On 01.04.20 19:37, Konrad Weihmann wrote:

What about

do_configure() {

   cd ${@bb.utils.contains('PACKAGECONFIG_CONFARGS',
'--verbose', out/verbose', 'out', d)}

}

On 01.04.20 18:55, Joel Winarske wrote:

Hello,

I'm trying the below in a recipe.  Bitbake gives me: ERROR:
Unable to parse ...: Exited with "2"

inherit python3native

require utils.inc

do_configure() {

    cd ${@get_out_dir(d)}
}

(in utils.inc)
def get_out_dir(d):
    import os
    import argparse
    pkgconfig_args = d.getVar("PACKAGECONFIG_CONFARGS")

    parser = argparse.ArgumentParser()
    parser.add_argument("--verbose", help="increase output
verbosity",
                        action="store_true")
    args = parser.parse_args(pkgconfig_args)
    if args.verbose:
        return os.path.join('out', 'verbose')

    return 'out'



What am I doing wrong, or is there a better way?  Seems like
it doesn't like "args = parser.parse_args(pkgconfig_args)"

The output directory is a combination of the arguments.


Thanks!
Joel

Hi Joel,

okay, I see - although I have to admit the argparse idea looks very weird, the error is that according to
https://docs.python.org/3/library/argparse.html#argparse.ArgumentParser.parse_args
the pkgconfig_args variable should be a list not a single string.

so doing pkgconfig_args = d.getVar("PACKAGECONFIG_CONFARGS").split(" ") should fix the issue.
But from a bitbake perspective you should really think about choosing a less pythonic version

Best

Konrad

There is a single tick missing, but I'm sure you get the point

What about

do_configure() {

   cd ${@bb.utils.contains('PACKAGECONFIG_CONFARGS', '--verbose', out/verbose', 'out', d)}

}

Hello,

I'm trying the below in a recipe.  Bitbake gives me: ERROR: Unable to parse ...: Exited with "2"

inherit python3native

require utils.inc

do_configure() {

    cd ${@get_out_dir(d)}
}

(in utils.inc)
def get_out_dir(d):
    import os
    import argparse
    pkgconfig_args = d.getVar("PACKAGECONFIG_CONFARGS")

    parser = argparse.ArgumentParser()
    parser.add_argument("--verbose", help="increase output verbosity",
                        action="store_true")
    args = parser.parse_args(pkgconfig_args)
    if args.verbose:
        return os.path.join('out', 'verbose')

    return 'out'



What am I doing wrong, or is there a better way?  Seems like it doesn't like "args = parser.parse_args(pkgconfig_args)"

The output directory is a combination of the arguments.


Thanks!
Joel