Date   

Re: duplicate files in tmp

Ross Burton
 

On Tue, 31 Mar 2020 at 10:14, nus1998 <nus1998@yeah.net> wrote:
Well, Ubuntu doesn't tell any difference between a normal file and a hard link file on gui file explore, verified by "ls -i". thanks<br/><br/>B.R.<br/>Nus
Right. You'll need to use 'ls -l':

$ touch foo
$ ls -l foo
-rw-r--r-- 1 ross staff 0 Mar 31 11:46 foo
$ ln foo bar
$ ls -l foo bar
-rw-r--r-- 2 ross staff 0 Mar 31 11:46 bar
-rw-r--r-- 2 ross staff 0 Mar 31 11:46 foo

The second column is 'link count'. If you've never encountered hard
links before, look them up.
Ross


Re: duplicate files in tmp

nus1998 <nus1998@...>
 

Well, Ubuntu doesn't tell any difference between a normal file and a hard link file on gui file explore, verified by "ls -i". thanks<br/><br/>B.R.<br/>Nus

At 2020-03-31 16:58:26, "Ross Burton" <ross@...> wrote:
>On Tue, 31 Mar 2020 at 09:57, nus1998 <nus1998@...> wrote:
>> When I bitbaked the system, I found there are a lot of duplicate files in <build>/tmp, for example, there is a directory named "recipe-sysroot-native" in every package, which directory is serveral hundreds of MB in size. I'm confused that why don't use link file type to instead of duplicate copies, which can save tens of GB size. I know that disk capacity should not be a concern but it's always welcome to have a small size project.
>
>They are links, but hardlinks not softlinks for efficiency and even
>less disk usage.
>
>Ross


Re: duplicate files in tmp

Ross Burton
 

On Tue, 31 Mar 2020 at 09:57, nus1998 <nus1998@yeah.net> wrote:
When I bitbaked the system, I found there are a lot of duplicate files in <build>/tmp, for example, there is a directory named "recipe-sysroot-native" in every package, which directory is serveral hundreds of MB in size. I'm confused that why don't use link file type to instead of duplicate copies, which can save tens of GB size. I know that disk capacity should not be a concern but it's always welcome to have a small size project.
They are links, but hardlinks not softlinks for efficiency and even
less disk usage.

Ross


duplicate files in tmp

nus1998 <nus1998@...>
 

Hi All,

I'm newbie to yocto, currently I use imx-linux-zeus as the develop environment.

When I bitbaked the system, I found there are a lot of duplicate files in <build>/tmp, for example, there is a directory named "recipe-sysroot-native" in every package, which directory is serveral hundreds of MB in size. I'm confused that why don't use link file type to instead of duplicate copies, which can save tens of GB size. I know that disk capacity should not be a concern but it's always welcome to have a small size project.

B.R.
Nus


Re: [opkg-devel] [opkg-utils PATCH] Opkg.py : Support for tilde in version compare

Maarten van Megen
 

Hi Alejandro

Thanks for the review. Here is the revised version that addresses the issues with the original patch

Regards,
Maarten

---
opkg.py | 87 ++++++++++++++++++++++++++++++++++++++-------------------
1 file changed, 58 insertions(+), 29 deletions(-)
diff --git a/opkg.py b/opkg.py
index ba947c2..f1d1dcb 100644
--- a/opkg.py
+++ b/opkg.py
@@ -48,6 +48,20 @@ import tarfile
import textwrap
import collections

+
+def order(x):
+ if not x:
+ return 0
+ if x == "~":
+ return -1
+ if str.isdigit(x):
+ return 0
+ if str.isalpha(x):
+ return ord(x)
+
+ return 256 + ord(x)
+
+
class Version(object):
"""A class for holding parsed package version information."""
def __init__(self, epoch, version):
@@ -55,39 +69,53 @@ class Version(object):
self.version = version

def _versioncompare(self, selfversion, refversion):
+ """
+ Implementation below is a copy of the opkg version comparison algorithm
+ http://git.yoctoproject.org/cgit/cgit.cgi/opkg/tree/libopkg/pkg.c#n933
+ it alternates between number and non number comparisons until a difference is found
+ digits are compared by value. other characters are sorted lexically using the above method orderOfChar
+
+ One slight modification, the original version can return any value, whereas this one is limited to -1, 0, +1
+ """
if not selfversion: selfversion = ""
if not refversion: refversion = ""
- while 1:
- ## first look for non-numeric version component
- selfm = re.match('([^0-9]*)(.*)', selfversion)
- #print(('selfm', selfm.groups()))
- (selfalpha, selfversion) = selfm.groups()
- refm = re.match('([^0-9]*)(.*)', refversion)
- #print(('refm', refm.groups())
- (refalpha, refversion) = refm.groups()
- if (selfalpha > refalpha):
- return 1
- elif (selfalpha < refalpha):
- return -1
- ## now look for numeric version component
- (selfnum, selfversion) = re.match('([0-9]*)(.*)', selfversion).groups()
- (refnum, refversion) = re.match('([0-9]*)(.*)', refversion).groups()
- #print(('selfnum', selfnum, selfversion)
- #print(('refnum', refnum, refversion)
- if (selfnum != ''):
- selfnum = int(selfnum)
- else:
- selfnum = -1
- if (refnum != ''):
- refnum = int(refnum)
- else:
- refnum = -1
- if (selfnum > refnum):
+
+ value = list(selfversion)
+ ref = list(refversion)
+
+ while value or ref:
+ first_diff = 0
+ # alphanumeric comparison
+ while (value and not str.isdigit(value[0])) or (ref and not str.isdigit(ref[0])):
+ vc = order(value.pop(0) if value else None)
+ rc = order(ref.pop(0) if ref else None)
+ if vc != rc:
+ return -1 if vc < rc else 1
+
+ # comparing numbers
+ # start by skipping 0
+ while value and value[0] == "0":
+ value.pop(0)
+ while ref and ref[0] == "0":
+ ref.pop(0)
+
+ # actual number comparison
+ while value and str.isdigit(value[0]) and ref and str.isdigit(ref[0]):
+ if not first_diff:
+ first_diff = int(value.pop(0)) - int(ref.pop(0))
+ else:
+ value.pop(0)
+ ref.pop(0)
+
+ # the one that has a value remaining was the highest number
+ if value and str.isdigit(value[0]):
return 1
- elif (selfnum < refnum):
+ if ref and str.isdigit(ref[0]):
return -1
- if selfversion == '' and refversion == '':
- return 0
+ # in case of equal length numbers look at the first diff
+ if first_diff:
+ return 1 if first_diff > 0 else -1
+ return 0

def compare(self, ref):
if (self.epoch > ref.epoch):
@@ -592,6 +620,7 @@ if __name__ == "__main__":
assert Version(0, "1.2.2+cvs20070308").compare(Version(0, "1.2.2-r0")) == 1
assert Version(0, "1.2.2-r0").compare(Version(0, "1.2.2-r0")) == 0
assert Version(0, "1.2.2-r5").compare(Version(0, "1.2.2-r0")) == 1
+ assert Version(0, "1.1.2~r1").compare(Version(0, "1.1.2")) == -1

package = Package()

--

-----Original Message-----
From: Alejandro del Castillo <alejandro.delcastillo@ni.com>
Sent: Friday, March 27, 2020 9:34 PM
To: opkg-devel@googlegroups.com; Maarten van Megen <maarten.vanmegen@nedap.com>; yocto@yoctoproject.org
Subject: Re: [opkg-devel] [opkg-utils PATCH] Opkg.py : Support for tilde in version compare

Hi Marteen,

thanks for working on this, a couple of comments

On 3/19/20 6:26 AM, Maarten wrote:
Add support for the special tilde character. The current opkg version
allows specifying a version appended with a tilde e.g.
1.2.2~releasecandidate which is lower in version than 1.2.2 itself.
With this commit, this is now also supported by the opkg.py script.

Signed-off-by: Maarten <maarten.vanmegen@nedap.com>
---
opkg.py | 132 ++++++++++++++++++++++++++++----------------------------
1 file changed, 66 insertions(+), 66 deletions(-)

diff --git a/opkg.py b/opkg.py
index ba947c2..34af483 100644
--- a/opkg.py
+++ b/opkg.py
@@ -1,5 +1,4 @@
-#!/usr/bin/env python3
-# SPDX-License-Identifier: GPL-2.0-or-later
did you intentionally modified the shebang & removed the SPDX identifier?

+#!/usr/bin/env python
# Copyright (C) 2001 Alexander S. Guy <a7r@andern.org>
# Andern Research Labs
#
@@ -38,58 +37,86 @@ from __future__ import print_function
import tempfile
import os
import sys
-import glob
import hashlib
import re
import subprocess
from stat import ST_SIZE
import arfile
import tarfile
-import textwrap
import collections

+
+def order(x):
+ if not x:
+ return 0
+ if x == "~":
+ return -1
+ if str.isdigit(x):
+ return 0
+ if str.isalpha(x):
+ return ord(x)
+
+ return 256 + ord(x)
+
+
class Version(object):
"""A class for holding parsed package version information."""
+
def __init__(self, epoch, version):
self.epoch = epoch
self.version = version

def _versioncompare(self, selfversion, refversion):
+ """
+ Implementation below is a copy of the opkg version comparison algorithm
+ https://urldefense.com/v3/__http://git.yoctoproject.org/cgit/cgit.cgi/opkg/tree/libopkg/pkg.c*n933__;Iw!!FbZ0ZwI3Qg!5LDeOr__t_Ow1oYBFtvf1sVRXNIPxT_bNylibWik21KDrv8His3FsfSxBbFI88ZZzzTfeg$
+ it alternates between number and non number comparisons until a difference is found
+ digits are compared by value. other characters are sorted
+ lexically using the above method orderOfChar
+
+ One slight modification, the original version can return any value, whereas this one is limited to -1, 0, +1
+ """
if not selfversion: selfversion = ""
if not refversion: refversion = ""
- while 1:
- ## first look for non-numeric version component
- selfm = re.match('([^0-9]*)(.*)', selfversion)
- #print(('selfm', selfm.groups()))
- (selfalpha, selfversion) = selfm.groups()
- refm = re.match('([^0-9]*)(.*)', refversion)
- #print(('refm', refm.groups())
- (refalpha, refversion) = refm.groups()
- if (selfalpha > refalpha):
- return 1
- elif (selfalpha < refalpha):
- return -1
- ## now look for numeric version component
- (selfnum, selfversion) = re.match('([0-9]*)(.*)', selfversion).groups()
- (refnum, refversion) = re.match('([0-9]*)(.*)', refversion).groups()
- #print(('selfnum', selfnum, selfversion)
- #print(('refnum', refnum, refversion)
- if (selfnum != ''):
- selfnum = int(selfnum)
- else:
- selfnum = -1
- if (refnum != ''):
- refnum = int(refnum)
- else:
- refnum = -1
- if (selfnum > refnum):
+
+ value = list(selfversion)
+ ref = list(refversion)
+
+ while value or ref:
+ first_diff = 0
+ # alphanumeric comparison
+ while (value and not str.isdigit(value[0])) or (ref and not str.isdigit(ref[0])):
+ vc = order(value.pop(0) if value else None)
+ rc = order(ref.pop(0) if ref else None)
+ if vc != rc:
+ return -1 if vc < rc else 1
+
+ # comparing numbers
+ # start by skipping 0
+ while value and value[0] == "0":
+ value.pop(0)
+ while ref and ref[0] == "0":
+ ref.pop(0)
+
+ # actual number comparison
+ while value and str.isdigit(value[0]) and ref and str.isdigit(ref[0]):
+ if not first_diff:
+ first_diff = int(value.pop(0)) - int(ref.pop(0))
+ else:
+ value.pop(0)
+ ref.pop(0)
+
+ # the one that has a value remaining was the highest number
+ if value and str.isdigit(value[0]):
return 1
- elif (selfnum < refnum):
+ if ref and str.isdigit(ref[0]):
return -1
- if selfversion == '' and refversion == '':
- return 0
+ # in case of equal length numbers look at the first diff
+ if first_diff:
+ return 1 if first_diff > 0 else -1
+ return 0

def compare(self, ref):
+
if (self.epoch > ref.epoch):
return 1
elif (self.epoch < ref.epoch):
@@ -191,9 +218,6 @@ class Package(object):
if name == "md5":
self._computeFileMD5()
return self.md5
- elif name == "sha256":
- self._computeFileSHA256()
- return self.sha256
this seems unrelated to your patch, was it intentional to remove sha256 support?

elif name == 'size':
return self._get_file_size()
else:
@@ -213,20 +237,6 @@ class Package(object):
f.close()
self.md5 = sum.hexdigest()

- def _computeFileSHA256(self):
- # compute the SHA256.
- if not self.fn:
- self.sha256 = 'Unknown'
- else:
- f = open(self.fn, "rb")
- sum = hashlib.sha256()
- while True:
- data = f.read(1024)
- if not data: break
- sum.update(data)
- f.close()
- self.sha256 = sum.hexdigest()
-
same here

def _get_file_size(self):
if not self.fn:
self.size = 0;
@@ -259,8 +269,6 @@ class Package(object):
self.size = int(value)
elif name_lowercase == 'md5sum':
self.md5 = value
- elif name_lowercase == 'sha256sum':
- self.sha256 = value
and here

elif name_lowercase in self.__dict__:
self.__dict__[name_lowercase] = value
elif all_fields:
@@ -384,7 +392,6 @@ class Package(object):
error = subprocess.CalledProcessError(retcode, cmd)
error.output = output
raise error
- output = output.decode("utf-8")
return output

if not self.fn:
@@ -408,12 +415,8 @@ class Package(object):
return []
f = open(self.fn, "rb")
ar = arfile.ArFile(f, self.fn)
- try:
- tarStream = ar.open("data.tar.gz")
- tarf = tarfile.open("data.tar.gz", "r", tarStream)
- except IOError:
- tarStream = ar.open("data.tar.xz")
- tarf = tarfile.open("data.tar.xz", "r:xz", tarStream)
+ tarStream = ar.open("data.tar.gz")
+ tarf = tarfile.open("data.tar.gz", "r", tarStream)
this also seem to revert a newer commit (add xz support)

self.file_list = tarf.getnames()
self.file_list = [["./", ""][a.startswith("./")] + a for a
in self.file_list]

@@ -488,7 +491,7 @@ class Package(object):
ref.parsed_version = parse_version(ref.version)
return self.parsed_version.compare(ref.parsed_version)

- def print(self, checksum):
+ def __str__(self):
this change also looks unrelated

out = ""

# XXX - Some checks need to be made, and some exceptions @@
-505,10 +508,7 @@ class Package(object):
if self.section: out = out + "Section: %s\n" % (self.section)
if self.architecture: out = out + "Architecture: %s\n" % (self.architecture)
if self.maintainer: out = out + "Maintainer: %s\n" % (self.maintainer)
- if 'md5' in checksum:
- if self.md5: out = out + "MD5Sum: %s\n" % (self.md5)
- if 'sha256' in checksum:
- if self.sha256: out = out + "SHA256sum: %s\n" % (self.sha256)
+ if self.md5: out = out + "MD5Sum: %s\n" % (self.md5)
if self.size: out = out + "Size: %d\n" % int(self.size)
if self.installed_size: out = out + "InstalledSize: %d\n" % int(self.installed_size)
if self.filename: out = out + "Filename: %s\n" %
(self.filename) @@ -585,14 +585,14 @@ class Packages(object):
def __getitem__(self, key):
return self.packages[key]

-if __name__ == "__main__":

+if __name__ == "__main__":
assert Version(0, "1.2.2-r1").compare(Version(0, "1.2.3-r0")) == -1
assert Version(0, "1.2.2-r0").compare(Version(0, "1.2.2+cvs20070308-r0")) == -1
assert Version(0, "1.2.2+cvs20070308").compare(Version(0, "1.2.2-r0")) == 1
assert Version(0, "1.2.2-r0").compare(Version(0, "1.2.2-r0")) == 0
assert Version(0, "1.2.2-r5").compare(Version(0, "1.2.2-r0")) ==
1
-
+ assert Version(0, "1.1.2~r1").compare(Version(0, "1.1.2")) == -1
package = Package()

package.set_package("FooBar")
--
Cheers,

Alejandro


M+ & H bugs with Milestone Movements WW13

Stephen Jolley
 

All,

YP M+ or high bugs which moved to a new milestone in WW13 are listed below:

Priority

Bug ID

Short Description

Changer

Owner

Was

Became

Medium+

5322

Global DNS fallback mechanism not present in poky distro

kai.kang@...

kai.kang@...

3.1 M3

3.1 M4

 

8191

apply kernel config fragments to arbitrary kernels

randy.macleod@...

bruce.ashfield@...

3.1 M3

3.1 M4

 

10693

Add a testcase for multilib eSDK on the autobuilder

Qi.Chen@...

Qi.Chen@...

3.1 M3

3.2

 

10731

bitbake --observe-only doesn't work with memres

liezhi.yang@...

liezhi.yang@...

3.1 M3

3.2

 

11449

Allow overriding classes to override overridden classes

richard.purdie@...

richard.purdie@...

3.1 M3

3.1 M4

 

11746

oe-selftest: capture self.logger messages in XML output

richard.purdie@...

richard.purdie@...

3.1 M3

3.1 M4

 

11781

bitbake --observe-only may get KeyError

liezhi.yang@...

liezhi.yang@...

3.1 M3

3.2

 

11906

rpmbuild: Can not build packages on qemu target

hongxu.jia@...

hongxu.jia@...

3.1 M3

3.2

 

12279

enhance manifest not found warning

kai.kang@...

kai.kang@...

3.1 M3

3.2

 

12342

lib32-core-image-sato -ctestimage failed due to wrong package names

kai.kang@...

kai.kang@...

3.1 M3

3.1 M4

 

12868

archiver ARCHIVER_MODE[src] = "patched" and ="configured" fails to build.  Morty works.  Pyro, sumo, master etc fail

randy.macleod@...

pbarker@...

3.1 M3

3.1 M4

 

12917

Warnings from nightly-multilib builds (build-deps)

kai.kang@...

kai.kang@...

3.1 M3

3.1 M4

 

12937

Consistent naming scheme for deployed artifacts

Martin.Jansa@...

Martin.Jansa@...

3.1 M3

3.2

 

12963

nativesdk-opkg prefixes all internal paths with $SDKPATH and won't work

randy.macleod@...

denis@...

3.1 M3

3.2

 

12970

uninative file should be versionned

richard.purdie@...

richard.purdie@...

3.1 M3

3.1 M4

 

12986

Failed to expand SRCPV on updateding SRC_URI using pn overrides and BBCLASSEXTEND

richard.purdie@...

richard.purdie@...

3.1 M3

3.1 M4

 

13002

Create a set of tests to run after upgrading patchtest

randy.macleod@...

pbarker@...

3.1 M3

3.2

 

13004

Automate yocto-check-layer -m option

richard.purdie@...

richard.purdie@...

3.1 M3

3.1 M4

 

13008

toaster testing

david.reyna@...

david.reyna@...

3.1 M3

3.2

 

13009

Review the manual test cases and remove any which are obsolete or unneeded

akuster@...

akuster808@...

3.1 M3

3.1 M4

 

13010

Manual tests: place holder

akuster@...

akuster808@...

3.1 M3

3.1 M4

 

13016

How to run our test suites for boards like beaglebone remotely on a LAVA board farm?

randy.macleod@...

limon.anibal@...

3.1 M3

3.2

 

13039

fetch2: PREMIRROR and SRC_URI with type https and parameter downloadfilename yields invalid url

richard.purdie@...

richard.purdie@...

3.1 M3

3.1 M4

 

13051

allow Open Container compliant "images" to be generated from oe images (container or not)

randy.macleod@...

bruce.ashfield@...

3.1 M3

3.2

 

13052

Allow oe-core generated OCI containers to be executed/tested on the build host

randy.macleod@...

bruce.ashfield@...

3.1 M3

3.2

 

13053

oe-core OCI container compatibility / deployment and conversion

randy.macleod@...

bruce.ashfield@...

3.1 M3

3.2

 

13055

oe-selftest runcmd.RunCmdTests.test_stdin failure

Qi.Chen@...

Qi.Chen@...

3.1 M3

3.2

 

13062

oe-selftest PR server failure

liezhi.yang@...

liezhi.yang@...

3.1

3.2

 

13071

Multiconfig builds may try to execute events that don't exist for them

randy.macleod@...

alejandro@...

3.1 M3

3.2

 

13079

devtool documentation needs to mention oe-local-files

randy.macleod@...

bluelightning@...

3.1 M3

3.1 M4

 

13070

recipetool.RecipetoolTests.test_recipetool_load_plugin - Testcase 1640: FAILED

richard.purdie@...

richard.purdie@...

3.1 M3

3.1 M4

 

13109

Implement CPE to package to Release mapping

david.reyna@...

david.reyna@...

3.1 M3

3.2

 

13103

[Bug][QA 2.7 M1 rc1][Toaster] "Recipes" tableá and á"machines" table are not getting populated after clickingáon imported layer as well as after clicking Machines Tab on project page

david.reyna@...

david.reyna@...

3.1 M3

3.1 M4

 

13100

killed testimage does not cleanup properly

liezhi.yang@...

liezhi.yang@...

3.1 M4

3.2

 

13123

package.PackageTests.test_gdb_hardlink_debug failed

randy.macleod@...

randy.macleod@...

3.1 M3

3.1 M4

 

13176

yocto-check-layer can only check layers which aren't already in BBLAYERS

liezhi.yang@...

liezhi.yang@...

3.1 M3

3.2

 

13183

bitbake-layers crashes with incorrect layer configuration data is given (expected proper error printing and exit with error)

richard.purdie@...

richard.purdie@...

3.1 M3

3.1 M4

 

13182

Inconsistency detected by ld.so: dl-open.c: 274: dl_open_worker: Assertion xxx failed

Qi.Chen@...

Qi.Chen@...

3.1 M3

3.2

 

13190

RRS cannot handle multiple recipes with same PN

randy.macleod@...

bluelightning@...

3.1 M3

3.2

 

13199

devtool check-upgrade-status  : traceback when meta-openembedded added in bblayers

liezhi.yang@...

liezhi.yang@...

3.1 M3

3.2

 

13207

Toaster broken with recent SQLite

david.reyna@...

david.reyna@...

3.1 M3

3.1 M4

 

13218

classic SDK installer missing stuff?

liezhi.yang@...

liezhi.yang@...

3.1 M3

3.2

 

13226

Support out of tree modules for alternate kernels

randy.macleod@...

bruce.ashfield@...

3.1 M3

3.2

 

13229

ttm_bo_vm_open kernel warning

randy.macleod@...

jon.mason@...

3.1 M3

3.2

 

13230

Switch qemuarm (another others) to virtio graphics

randy.macleod@...

jon.mason@...

3.1 M3

3.2

 

13233

fetch2: try_premirror(): improve on updating repo from mirror

liezhi.yang@...

liezhi.yang@...

3.1 M3

3.2

 

13236

sstate for host native packages

liezhi.yang@...

liezhi.yang@...

3.1 M2

3.2

 

13251

Symlinks overridden when building multitple kernels

randy.macleod@...

bruce.ashfield@...

3.1 M3

3.2

 

13278

If git protocol doesn't work, you get a tar.gz clone from PREMIRROR which has git protocol origin

richard.purdie@...

richard.purdie@...

3.1 M3

3.1 M4

 

13279

Make sure users/groups exist for package_write_* tasks

liezhi.yang@...

liezhi.yang@...

3.1 M3

3.2

 

13281

build-perf-test cannot generate reports on new branches due to lack of comparison data

richard.purdie@...

richard.purdie@...

3.1 M3

3.1 M4

 

13285

YoctoProject Compatibility script improvements needed

randy.macleod@...

akuster808@...

3.1 M4

3.2

 

 

akuster@...

akuster808@...

3.1 M3

3.1 M4

 

13288

pseudo should not follow symlinks in /proc

randy.macleod@...

seebs@...

3.1 M3

3.2

 

13306

bitbake starts up to n^2 processes with n cpus

liezhi.yang@...

liezhi.yang@...

3.1 M3

3.2

 

13315

Migrate poky's WARN_TO_ERROR_QA settings to OE-Core defaults

liezhi.yang@...

liezhi.yang@...

3.1 M3

3.2

 

13320

Update license files to match current SPDX names and license contents

randy.macleod@...

newcomer@...

3.1 M3

3.2

 

13325

Add link to the output directory from LHS console view

richard.purdie@...

richard.purdie@...

3.1 M3

3.1 M4

 

13353

bitbake git fetcher does not honour BB_FETCH_PREMIRRORONLY

liezhi.yang@...

liezhi.yang@...

3.1 M3

3.2

 

13355

RDEPENDS does not work properly for native builds (only supports recipe names, not package names)

richard.purdie@...

richard.purdie@...

3.1 M3

3.1 M4

 

13360

kernel-dev: package full kernel source and headers for reference purposes

randy.macleod@...

bruce.ashfield@...

3.1 M3

3.2

 

13374

Determine 32bit guest support on arm64

randy.macleod@...

jon.mason@...

3.1 M3

3.2

 

13392

acl ptests fail for qemuarm64

randy.macleod@...

jon.mason@...

3.1 M3

3.1 M4

 

13421

File race when calling devtool.setUpModule

liezhi.yang@...

liezhi.yang@...

3.1 M3

3.2

 

13419

recipes that add users to groups cannot rely on other recipes creating those groups (when population from sstate happens)

richard.purdie@...

richard.purdie@...

3.1 M3

3.1 M4

 

13424

devupstream doesn't work with mutilib

richard.purdie@...

richard.purdie@...

3.1 M3

3.1 M4

 

13426

Loses track of data if file rename()d to same name

randy.macleod@...

seebs@...

3.1 M3

3.2

 

13448

bitbake master appears to expand variables it should not need to

richard.purdie@...

richard.purdie@...

3.1 M3

3.1 M4

 

13459

sdk: compiler targets glibc, even though rootfs uses musl-libc

randy.macleod@...

randy.macleod@...

3.1 M3

3.1 M4

 

13480

gcc-cross plugins feature cannot be used

randy.macleod@...

zhe.he@...

3.1 M3

3.2

 

13489

layer index: properly handle issues in first parse

randy.macleod@...

bluelightning@...

3.1 M3

3.2

 

13490

layer index: log failures during initial parse

randy.macleod@...

bluelightning@...

3.1 M3

3.2

 

13495

Support building out container image without init manager

Qi.Chen@...

Qi.Chen@...

3.1 M3

3.2

 

13520

many valgrind tests fail for arm64

randy.macleod@...

randy.macleod@...

3.1 M3

3.1 M4

 

13533

Devtool finish on _git package with SRCPV in PV points to wrong WORKDIR

randy.macleod@...

jaewon@...

3.1 M3

3.1 M4

 

13530

Add SPDX license headers to all source files for error-report-web

randy.macleod@...

mshah@...

3.1 M3

3.1 M4

 

13550

username/password specified to gitsm:// does not get propagated to submodules

randy.macleod@...

mark.hatle@...

3.1 M3

3.1 M4

 

13563

Can't migrate from django 1.x to 2.x

david.reyna@...

david.reyna@...

3.1 M3

3.2

 

13566

Write tests for multiconfig files in layers and document

randy.macleod@...

mostthingsweb@...

3.1 M3

3.1 M4

 

Thanks,

 

Stephen K. Jolley

Yocto Project Program Manager

(    Cell:                (208) 244-4460

* Email:              sjolley.yp.pm@...

 


Enhancements/Bugs closed WW12!

Stephen Jolley
 

All,

The below were the owners of enhancements or bugs closed during the last week!

Who

Count

richard.purdie@...

4

david.reyna@...

2

yang.wang@...

1

yi.zhao@...

1

hongxu.jia@...

1

kexin.hao@...

1

alex.kanavin@...

1

raj.khem@...

1

khoja.medhi@...

1

Qi.Chen@...

1

kai.kang@...

1

Grand Total

15

 

Thanks,

 

Stephen K. Jolley

Yocto Project Program Manager

(    Cell:                (208) 244-4460

* Email:              sjolley.yp.pm@...

 


Current Top developers on Yocto Project 3.1

Stephen Jolley
 

All,

Below is the list as of top 46 developers as of the end of WW13 of who have open medium or higher bugs and enhancements against YP 3.1.   There are 23 possible work days left until the final release candidates for YP 3.1 needs to be released.

Who

Count

richard.purdie@...

32

david.reyna@...

13

akuster808@...

13

mark.morton@...

11

randy.macleod@...

10

timothy.t.orling@...

9

bluelightning@...

9

kai.kang@...

8

ross@...

6

trevor.gamblin@...

6

michael@...

6

bruce.ashfield@...

5

chee.yang.lee@...

5

srifenbark@...

4

matthew.zeng@...

4

Qi.Chen@...

4

alex.kanavin@...

3

raj.khem@...

2

jon.mason@...

2

kexin.hao@...

2

alejandro@...

2

jaewon@...

2

pbarker@...

2

mark.hatle@...

2

ycnakajsph@...

2

kergoth@...

2

yang.wang@...

2

daisuke.yamane@...

1

jonathan.yong@...

1

changqing.li@...

1

sakib.sajal@...

1

maxime.roussinbelanger@...

1

anuj.mittal@...

1

matthias.schoepfer@...

1

bunk@...

1

scott.branden@...

1

hongxu.jia@...

1

amber.n.elliot@...

1

ee.peng.yeoh@...

1

mshah@...

1

mostthingsweb@...

1

jpuhlman@...

1

JPEWhacker@...

1

apoorv.sangal@...

1

kai.ruhnau@...

1

 

Thanks,

 

Stephen K. Jolley

Yocto Project Program Manager

(    Cell:                (208) 244-4460

* Email:              sjolley.yp.pm@...

 


Yocto Project Newcomer & Unassigned Bugs - Help Needed

Stephen Jolley
 

All,

 

The triage team is starting to try and collect up and classify bugs which a newcomer to the project would be able to work on in a way which means people can find them. They're being listed on the triage page under the appropriate heading:

 

https://wiki.yoctoproject.org/wiki/Bug_Triage#Newcomer_Bugs

 

The idea is these bugs should be straight forward for a person to help work on who doesn't have deep experience with the project.  If anyone can help, please take ownership of the bug and send patches!  If anyone needs help/advice there are people on irc who can likely do so, or some of the more experienced contributors will likely be happy to help too.

 

Also, the triage team meets weekly and does its best to handle the bugs reported into the Bugzilla. The number of people attending that meeting has fallen, as have the number of people available to help fix bugs. One of the things we hear users report is they don't know how to help. We (the triage team) are therefore going to start reporting out the currently 309 unassigned or newcomer bugs.

 

We're hoping people may be able to spare some time now and again to help out with these.  Bugs are split into two types, "true bugs" where things don't work as they should and "enhancements" which are features we'd want to add to the system.  There are also roughly four different "priority" classes right now, “3.1”, “3.2, "3.99" and "Future", the more pressing/urgent issues being in "3.1" and then “3.2”.

 

Please review this link and if a bug is something you would be able to help with either take ownership of the bug, or send me (sjolley.yp.pm@...) an e-mail with the bug number you would like and I will assign it to you (please make sure you have a Bugzilla account).  The list is at: https://wiki.yoctoproject.org/wiki/Bug_Triage_Archive#Unassigned_or_Newcomer_Bugs

 

Thanks,

 

Stephen K. Jolley

Yocto Project Program Manager

(    Cell:                (208) 244-4460

* Email:              sjolley.yp.pm@...

 


Re: Best way to mask bbappends based on Poky version to have a layer support multiple versions of Poky?

Matt Campbell
 

Thanks for the info. I think both of those will run into issues where the bbappend won't apply to the original recipe as that is fully done by path and file name and both of these methods mangle those. Please correct me if I'm wrong as there might be some cheese down that tunnel for me.

I've been thinking more about this over the weekend and have rethought the problem a bit. I think what I really want is to be able to conditionally skip a bbappend in recipe using python rather than having to do this centralized in the layer.conf with BBMASK. There are two reasons for this:

1) This puts the logic that skips the bbappend in the file itself which makes it more obvious what is going on when browsing/maintaining meta-data. The BBMASK method is centralized, but is hidden when you are just working with the metadata.
2) This allows for arbitrary and more complex logic for skipping bbappends.

I'm not super familiar with the internals of bitbake, but I think what I want is something similar to the bb.parse.SkipRecipe() exception. For example

file: test_%.bbappend
python() {
    # Skip based on poky version
    if d.getVar("DISTRO_CODENAME") != 'zeus':
        raise bb.parse.SkipBbappend("This bbappend only supports Zeus. Skipping this bbappend")

    # Skip based on package version
    if d.getVar("PV") != '1.0':
        raise bb.parse.SkipBbappend("This bbappend only supports version 1.0 of test. Skipping bbappend")
}

1) Does this seem like a good solution to the problem of dangling appends in layers that want to support two versions of Poky?
2) Any thoughts on how to go about implementing this?

~Matt

On Thu, Mar 26, 2020 at 4:09 PM Konrad Weihmann <kweihmann@...> wrote:

Hi,

I'll get your point.
Maybe this could be a solution to your problem https://www.yoctoproject.org/docs/current/bitbake-user-manual/bitbake-user-manual.html#var-BBVERSIONS.
Instead of having different bbappends have one and pick the right steps inside of the append.

If that is not working for you, you could also fake the behavior of BBFILES_DYNAMIC with putting the bbappend into separate subfolder, which than are
referenced by something like this

BBFILES += "${LAYERDIR}/dynamic-recipes/${BB_VERSION}/*.bbappend"

Regards
Konrad

On 26.03.20 20:56, Matt Campbell wrote:
I didn't know about  BB_DANGLINGAPPENDS_WARNONLY. That would mask the problem, but doesn't feel like a great solution. Either way I do appreciate you sharing that.

Further implementation and discussion with our team brought up another possible solution. We could wildcard all bbappends (_%.bbappend) and use some anonymous python inside our bbappend files that will error out if the package version isn't in a supported list. We could also easily roll this up into a bbclass to prevent the need to duplicate this everywhere.

python () {
    package_version = d.getVar("PV")
    if  package_version is not in ['3.14", "3.15"]:
        bb.error("This bbappend file isn't compatible with the version {}. You will need to add support to this bbappend for that version.".format(package_version))
}

This still seems more like we are fighting bitbake rather than working with it. Does anyone have any thoughts or suggestions on this?

Other upstreams seem to maintain different branches for different Poky releases. That is a road we would rather avoid if possible. Our goal is to be able to have an extra CI build against the version of Poky under development so we can continuously fix the upgrade issues as they come up rather than as a landslide when we upgrade. Making a separate branch for this would mean we would need to merge all active development into each branch to get the benefits of a poky next canary build plan. That said, I'd love to hear about a solution that lets us have our cake and eat it too.

~Matt

On Thu, Mar 26, 2020 at 9:55 AM Robert P. J. Day <rpjday@...> wrote:
On Thu, 26 Mar 2020, Matt Campbell wrote:

> HI All,

> We have a layer where we want to concurrently support two releases
> of Poky. There is an issue when we have bbappnds against recipes
> that have different versions in the two poky releases. for instance,
> imagine recipe foo that is version 1.0 in Zeus and 1.2 in Dunfell.
> If we had a bbappend in our layer `foo_1.0.bbappend` and tried to
> use our layer with Dunfell, bitbake will error out saying that
> `foo_1.0.bbappend` has no base recipe.

  not sure if this really solves the underlying issue, but you can
always turn those errors into warnings with:

  BB_DANGLINGAPPENDS_WARNONLY = "1"

in your local.conf, although i'm still skeptical as to whether that's
really the problem you're trying to solve.

rday


--
Matthew Campbell
Senior Embedded Systems Engineer

iZotope, Inc.


    


--
Matthew Campbell
Senior Embedded Systems Engineer

iZotope, Inc.


Re: Image size reduction

Mikko Rapeli
 

On Mon, Mar 30, 2020 at 03:00:05AM +0000, Ajam Ali wrote:
Hi khemraj,

Please suggest the way to reduce the image size. Your earlier suggestion is not that much helpful.
Actually suggestion to use buildhistory is the first critical step because without
this you can't easily see where the size comes from. buildhistory provides
for example the list of installed binary packages ordered by size for every image
built. Then it enables you to easily check the dependencies of variour packages
and detect, e.g. who brings perl or python to the image.

Cheers,

-Mikko


Re: Fw: Reducing rootfs size in yocto

Mikko Rapeli
 

Hi,

Generic approaches for reducing yocto target image size where buildhistory
is an important source of information:

* review and minimize distro features, enable only what you need
https://www.yoctoproject.org/docs/latest/mega-manual/mega-manual.html#ref-features-distro

* review and minimize image recipe, install only those image features and packages
which you need
https://www.yoctoproject.org/docs/latest/mega-manual/mega-manual.html#ref-features-image

* review the needed binary packages: traverse the tree of dependencies what you need,
review every single recipe and change PACKAGECONFIG or build flags to remove
features which you do not need. This it may be possible to remove large set
of dependencies from images, e.g. language bindings, debug tools. In some cases
one just needs a single binary executable or shared library but it is bundled
in a larger binary package with more complex dependencies. For these cases
add a bbappend to custom layers to change the packaging to only include what
you need.
https://www.yoctoproject.org/docs/latest/mega-manual/mega-manual.html#var-PACKAGECONFIG

* change compiler flags from default O2 to Os optimization. This can work for
some recipes which are not prerformance critical or to the whole build. Note
that many SW components break this in their own build scripts by adding
O2 or even O3 by default. Check from buildhistory that binary sizes go smaller
after switch from O2 to Os and review every important recipe which did not.
You may be able to save up to 15% in rootfs size this way, but all depends
on the details of SW components and image contents.
see poky/meta/conf/bitbake.conf

I've followed this for multiple products and found out that poky is nice to work with
but BSP layers add or depend on all kinds of extra things that products do not need.
Thus I have ended by BBMASK'ing away large parts of BSP layer recipes in distro
configs. This with multiple x86 and ARM BSP layers from various SoC vendors.

Also, RTFM :)

https://www.yoctoproject.org/docs/latest/mega-manual/mega-manual.html#building-a-tiny-system

Hope this helps,

-Mikko


[PATCH] Add localhost to noproxy list.

Li, Xiaoming
 

Signed-off-by: Li Xiaoming <lixm.fnst@cn.fujitsu.com>
---
classes/fossology-rest.bbclass | 16 +++++++++-------
1 file changed, 9 insertions(+), 7 deletions(-)

diff --git a/classes/fossology-rest.bbclass b/classes/fossology-rest.bbclass
index 48e16f3..000eefe 100644
--- a/classes/fossology-rest.bbclass
+++ b/classes/fossology-rest.bbclass
@@ -25,6 +25,8 @@ HOSTTOOLS_NONFATAL += "curl"

CREATOR_TOOL = "fossology-rest.bbclass in meta-spdxscanner"

+NO_PROXY = "localhost, 127.0.0.1"
+
# If ${S} isn't actually the top-level source directory, set SPDX_S to point at
# the real top-level directory.
SPDX_S ?= "${S}"
@@ -159,7 +161,7 @@ def get_folder_id_by_name(d, folder_name):

rest_api_cmd = "curl -k -s -S -X GET " + server_url + "/api/v1/folders" \
+ " -H \"Authorization: Bearer " + token + "\"" \
- + " --noproxy 127.0.0.1"
+ + " --noproxy " + NO_PROXY
bb.note("Invoke rest_api_cmd = " + rest_api_cmd )
try:
all_folder = subprocess.check_output(rest_api_cmd, stderr=subprocess.STDOUT, shell=True)
@@ -200,7 +202,7 @@ def create_folder(d, folder_name):
+ " -H \'parentFolder: 1\'" \
+ " -H \'folderName: " + folder_name + "\'" \
+ " -H \"Authorization: Bearer " + token + "\"" \
- + " --noproxy 127.0.0.1"
+ + " --noproxy " + NO_PROXY
bb.note("Invoke rest_api_cmd = " + rest_api_cmd)
try:
add_folder = subprocess.check_output(rest_api_cmd, stderr=subprocess.STDOUT, shell=True)
@@ -251,7 +253,7 @@ def has_upload(d, tar_file, folder_id):

rest_api_cmd = "curl -k -s -S -X GET " + server_url + "/api/v1/uploads" \
+ " -H \"Authorization: Bearer " + token + "\"" \
- + " --noproxy 127.0.0.1"
+ + " --noproxy " + NO_PROXY
bb.note("Invoke rest_api_cmd = " + rest_api_cmd )

try:
@@ -302,7 +304,7 @@ def upload(d, tar_file, folder):
+ " -H \'public: public\'" \
+ " -H \'Content-Type: multipart/form-data\'" \
+ " -F \'fileInput=@\"" + tar_file + "\";type=application/octet-stream\'" \
- + " --noproxy 127.0.0.1"
+ + " --noproxy " + NO_PROXY
bb.note("Upload : Invoke rest_api_cmd = " + rest_api_cmd )
while i < 10:
time.sleep(delaytime)
@@ -343,7 +345,7 @@ def analysis(d, folder_id, upload_id):
+ " -H \"Authorization: Bearer " + token + "\"" \
+ " -H \'Content-Type: application/json\'" \
+ " --data \'{\"analysis\": {\"bucket\": true,\"copyright_email_author\": true,\"ecc\": true, \"keyword\": true,\"mime\": true,\"monk\": true,\"nomos\": true,\"package\": true},\"decider\": {\"nomos_monk\": true,\"bulk_reused\": true,\"new_scanner\": true}}\'" \
- + " --noproxy 127.0.0.1"
+ + " --noproxy " + NO_PROXY
bb.note("Analysis : Invoke rest_api_cmd = " + rest_api_cmd )
while i < 10:
try:
@@ -389,7 +391,7 @@ def trigger(d, folder_id, upload_id):
+ " -H \"Authorization: Bearer " + token + "\"" \
+ " -H \"uploadId: " + str(upload_id) + "\"" \
+ " -H \'reportFormat: spdx2tv\'" \
- + " --noproxy 127.0.0.1"
+ + " --noproxy " + NO_PROXY
bb.note("trigger : Invoke rest_api_cmd = " + rest_api_cmd )
while i < 10:
time.sleep(delaytime)
@@ -431,7 +433,7 @@ def get_spdx(d, report_id, spdx_file):
rest_api_cmd = "curl -k -s -S -X GET " + server_url + "/api/v1/report/" + report_id \
+ " -H \'accept: text/plain\'" \
+ " -H \"Authorization: Bearer " + token + "\"" \
- + " --noproxy 127.0.0.1"
+ + " --noproxy " + NO_PROXY
bb.note("get_spdx : Invoke rest_api_cmd = " + rest_api_cmd )
while i < 10:
time.sleep(delaytime)
--
2.17.1


Re: Fw: Reducing rootfs size in yocto

Ajam Ali
 

Hi,

-dev version of boost is a supporting package which is getting installed by default
not by me using local.conf.

Thanks,
Ajam Ali


From: yocto@... <yocto@...> on behalf of Oliver Westermann via Lists.Yoctoproject.Org <oliver.westermann=cognex.com@...>
Sent: Sunday, March 29, 2020 8:16 PM
To: yocto@... <yocto@...>
Cc: yocto@... <yocto@...>
Subject: Re: [yocto] Fw: Reducing rootfs size in yocto
 

[CAUTION: This Email is from outside the Organization. Do not click links or open attachments unless you trust the sender.]

Do you need the -dev version of boost on the device?
::DISCLAIMER::

The contents of this e-mail and any attachment(s) are confidential and intended for the named recipient(s) only. E-mail transmission is not guaranteed to be secure or error-free as information could be intercepted, corrupted, lost, destroyed, arrive late or incomplete, or may contain viruses in transmission. The e mail and its contents (with or without referred errors) shall therefore not attach any liability on the originator or HCL or its affiliates. Views or opinions, if any, presented in this email are solely those of the author and may not necessarily reflect the views or opinions of HCL or its affiliates. Any form of reproduction, dissemination, copying, disclosure, modification, distribution and / or publication of this message without the prior written consent of authorized representative of HCL is strictly prohibited. If you have received this email in error please delete it and notify the sender immediately. Before opening any email and/or attachments, please check them for viruses and other defects.


Add a customised shell script to be invked after board bootso #yocto

Raghu Icecraft Software Trainings
 

Hello,
I have written a shell script to change the password for my root user.
I want to execute this script file after board boots
So i created .bb file as follows:
 
 
SUMMARY = "Initscript for enabling root pwd"

LICENSE = "GPLv2"
#LIC_FILES_CHKSUM = "file://${WORKDIR}/root_vpd;md5=bc4963ad2e7baa1c3bdfc9031d8453f0"


PR = "r0"

SRC_URI = "file://root_vpd"
S = "${WORKDIR}"

do_install() {
    install -d ${D}${sysconfdir}
    install -d ${D}${sysconfdir}/init.d
    install -m 0755 root_vpd ${D}${sysconfdir}/init.d
}

inherit update-rc.d allarch

INITSCRIPT_NAME = "root-vpd"
INITSCRIPT_PARAMS = "start 99 5 2 . stop 20 0 1 6 ."
 
 
 
and i copied the script file in the sam efolder 
BUt when i do bitbake of above recipew i get this error :
 
Parsing recipes: 100% |#######################################################################################################################| Time: 0:00:01
Parsing of 1992 .bb files complete (1982 cached, 10 parsed). 4688 targets, 384 skipped, 1 masked, 0 errors.
ERROR: Nothing PROVIDES  pwd.bb
 
 

 
--
Thanks,
raghu


Novice question on makefile in yocto environment

Raghu Icecraft Software Trainings
 

Hi,

  I have some general idea on makefile in C, C++ environment.
Related to my Yocto project having a very tough time in getting my makefile ready on shape after almost every development change.

Is there any specific Troubleshooting based guidance available some where ?
I have also tried looking for old posts in this group.
Please suggest if I have to post this question, somewhere else.


Re: Image size reduction

Ajam Ali
 

Hi khemraj, 

Please suggest the way to reduce the image size. Your earlier suggestion is not that much helpful. 

Thanks, 
Ajam Ali

Sent from Outlook Mobile


From: Khem Raj <raj.khem@...>
Sent: Monday, March 30, 2020 2:44:21 AM
To: Ajam Ali <AjamA@...>
Cc: yocto@... <yocto@...>
Subject: Re: [yocto] Image size reduction
 
[CAUTION: This Email is from outside the Organization. Do not click links or open attachments unless you trust the sender.]

On Sun, Mar 29, 2020 at 9:29 AM Ajam Ali <ajama@...> wrote:
>
> Hi All,
>
> Actually my current image size is 3.9GB because of heavy size packages required by my project and without project required packages my image size in Yocto is 800MB.
>
> I want to reduce the image size as maximum as possible.
> Please suggest the best possible way so that I could reduce the maximum possible size(desirable below 1.5 GB).

Add INHERIT += "buildhistory" to local.conf and then do a build

Start looking at the buildhistory folder especially the size file
images/<machine>/glibc/<image-name>/installed-package-sizes.txt

>
>
> Thanks in advance,
> Ajam Ali
>
>
> Sent from Outlook Mobile
> ::DISCLAIMER::
> ________________________________
> The contents of this e-mail and any attachment(s) are confidential and intended for the named recipient(s) only. E-mail transmission is not guaranteed to be secure or error-free as information could be intercepted, corrupted, lost, destroyed, arrive late or incomplete, or may contain viruses in transmission. The e mail and its contents (with or without referred errors) shall therefore not attach any liability on the originator or HCL or its affiliates. Views or opinions, if any, presented in this email are solely those of the author and may not necessarily reflect the views or opinions of HCL or its affiliates. Any form of reproduction, dissemination, copying, disclosure, modification, distribution and / or publication of this message without the prior written consent of authorized representative of HCL is strictly prohibited. If you have received this email in error please delete it and notify the sender immediately. Before opening any email and/or attachments, please check them for viruses and other defects.
> ________________________________
>


Re: Image size reduction

Khem Raj
 

On Sun, Mar 29, 2020 at 9:29 AM Ajam Ali <ajama@hcl.com> wrote:

Hi All,

Actually my current image size is 3.9GB because of heavy size packages required by my project and without project required packages my image size in Yocto is 800MB.

I want to reduce the image size as maximum as possible.
Please suggest the best possible way so that I could reduce the maximum possible size(desirable below 1.5 GB).
Add INHERIT += "buildhistory" to local.conf and then do a build

Start looking at the buildhistory folder especially the size file
images/<machine>/glibc/<image-name>/installed-package-sizes.txt



Thanks in advance,
Ajam Ali


Sent from Outlook Mobile
::DISCLAIMER::
________________________________
The contents of this e-mail and any attachment(s) are confidential and intended for the named recipient(s) only. E-mail transmission is not guaranteed to be secure or error-free as information could be intercepted, corrupted, lost, destroyed, arrive late or incomplete, or may contain viruses in transmission. The e mail and its contents (with or without referred errors) shall therefore not attach any liability on the originator or HCL or its affiliates. Views or opinions, if any, presented in this email are solely those of the author and may not necessarily reflect the views or opinions of HCL or its affiliates. Any form of reproduction, dissemination, copying, disclosure, modification, distribution and / or publication of this message without the prior written consent of authorized representative of HCL is strictly prohibited. If you have received this email in error please delete it and notify the sender immediately. Before opening any email and/or attachments, please check them for viruses and other defects.
________________________________


Re: Files get sporadically lost for native packages

Konrad Weihmann
 

some updates on the issue.
I did some further investigation and it turns out that only python cache files are affected - not only when the manifest is created but also when files are being staged.

Here is a dump from one of my runs

2020-03-29T03:09:17.3893569Z ERROR: simple-c-1.0-r0 do_prepare_recipe_sysroot: Error executing a python function in exec_python_func() autogenerated:
2020-03-29T03:09:17.3893931Z
2020-03-29T03:09:17.3894095Z The stack trace of python calls that resulted in this exception/failure was:
2020-03-29T03:09:17.3894483Z File: 'exec_python_func() autogenerated', lineno: 2, function: <module>
2020-03-29T03:09:17.3894608Z      0001:
2020-03-29T03:09:17.3894727Z  *** 0002:extend_recipe_sysroot(d)
2020-03-29T03:09:17.3894837Z      0003:
2020-03-29T03:09:17.3895143Z File: '/opt/build/poky/meta/classes/staging.bbclass', lineno: 570, function: extend_recipe_sysroot
2020-03-29T03:09:17.3895282Z      0566:                    if "/bin/" in l or "/sbin/" in l:
2020-03-29T03:09:17.3895415Z      0567:                        # defer /*bin/* files until last in case they need libs
2020-03-29T03:09:17.3895535Z      0568:                        binfiles[l] = (targetdir, dest)
2020-03-29T03:09:17.3895652Z      0569:                    else:
2020-03-29T03:09:17.3895780Z  *** 0570:                        staging_copyfile(l, targetdir, dest, postinsts, seendirs)
2020-03-29T03:09:17.3895901Z      0571:
2020-03-29T03:09:17.3896183Z      0572:    # Handle deferred binfiles
2020-03-29T03:09:17.3896313Z      0573:    for l in binfiles:
2020-03-29T03:09:17.3896429Z      0574:        (targetdir, dest) = binfiles[l]
2020-03-29T03:09:17.3896763Z File: '/opt/build/poky/meta/classes/staging.bbclass', lineno: 152, function: staging_copyfile
2020-03-29T03:09:17.3896904Z      0148:        os.symlink(linkto, dest)
2020-03-29T03:09:17.3897018Z      0149:        #bb.warn(c)
2020-03-29T03:09:17.3897129Z      0150:    else:
2020-03-29T03:09:17.3897233Z      0151:        try:
2020-03-29T03:09:17.3897341Z  *** 0152:            os.link(c, dest)
2020-03-29T03:09:17.3897455Z      0153:        except OSError as err:
2020-03-29T03:09:17.3897571Z      0154:            if err.errno == errno.EXDEV:
2020-03-29T03:09:17.3897691Z      0155:                bb.utils.copyfile(c, dest)
2020-03-29T03:09:17.3897807Z      0156:            else:
2020-03-29T03:09:17.3898436Z Exception: FileNotFoundError: [Errno 2] No such file or directory: '/opt/build/build/tmp/sysroots-components/x86_64/python3-vulture-native/usr/lib/python3.8/site-packages/vulture/whitelists/__pycache__/unittest_whitelist.cpython-38.pyc' -> '/opt/build/build/tmp/work/core2-32-scatest-linux/simple-c/1.0-r0/recipe-sysroot-native/usr/lib/python3.8/site-packages/vulture/whitelists/__pycache__/unittest_whitelist.cpython-38.pyc'
2020-03-29T03:09:17.3898588Z
2020-03-29T03:09:17.3898936Z ERROR: Logfile of failure stored in: /opt/build/build/tmp/work/core2-32-scatest-linux/simple-c/1.0-r0/temp/log.do_prepare_recipe_sysroot.122175
2020-03-29T03:09:17.3899233Z NOTE: recipe simple-c-1.0-r0: task do_prepare_recipe_sysroot: Failed
2020-03-29T03:09:17.3936565Z ERROR: Task (/opt/build/meta-sca/recipes-sca-test/code-from-elsewhere/simple-c.bb:do_prepare_recipe_sysroot) failed with exit code '1'
2020-03-29T03:09:17.3956520Z NOTE: Running task 3920 of 5451 (/opt/build/poky/meta/recipes-devtools/python/python3-git_3.0.5.bb:do_fetch)

After reading a bit about pyc-files(https://docs.python.org/3/library/py_compile.html#py_compile.PycInvalidationMode),
I suspect some of the files being invalidated, although I still haven't figured out how that is done.
After forcefully removing all cache files from all native packages, it seems to work - but still I think it just hides the general issue causing this.

Current summary:

- a lot of native python packages are required by multiple recipes (via DEPENDS)
- this effect is only visible under heavy load (on a 16+ core machine: never happened so far, 8 core: happens occasionally, 2 core: happens nearly every time)
- only python cache files (.pyc) are affected
- error occurs at random packages in do_prepare_recipe_sysroot
- some pyc files set in the package manifest are not present in the filesystem

On 28.03.20 13:26, Konrad Weihmann wrote:

Hi,

I'm facing the following error message sporadically on all branches I tried so far (master, zeus, warrior and thud)

The stack trace of python calls that resulted in this exception/failure was:
File: 'exec_python_func() autogenerated', lineno: 2, function: <module>
     0001:
 *** 0002:extend_recipe_sysroot(d)
     0003:
File: '/build/poky/meta/classes/staging.bbclass', lineno: 551, function: extend_recipe_sysroot
     0547:                    dest = newmanifest[l]
     0548:                    if l.endswith("/"):
     0549:                        staging_copydir(l, targetdir, dest, seendirs)
     0550:                        continue
 *** 0551:                    staging_copyfile(l, targetdir, dest, postinsts, seendirs)
     0552:
     0553:    bb.note("Installed into sysroot: %s" % str(msg_adding))
     0554:    bb.note("Skipping as already exists in sysroot: %s" % str(msg_exists))
     0555:
File: '/build/poky/meta/classes/staging.bbclass', lineno: 152, function: staging_copyfile
     0148:        os.symlink(linkto, dest)
     0149:        #bb.warn(c)
     0150:    else:
     0151:        try:
 *** 0152:            os.link(c, dest)
     0153:        except OSError as err:
     0154:            if err.errno == errno.EXDEV:
     0155:                bb.utils.copyfile(c, dest)
     0156:            else:
Exception: FileNotFoundError: [Errno 2] No such file or directory: '/build/poky/build/tmp/sysroots-components/x86_64/python3-msgcheck-native/usr/lib/python3.7/site-packages/msgcheck/__pycache__/__init__.cpython-37.pyc' -> '/build/poky/build/tmp/work/qemux86_64-mine-linux/core-image-minimal-mine/1.0-r0/recipe-sysroot-native/usr/lib/python3.7/site-packages/msgcheck/__pycache__/__init__.cpython-37.pyc'

I already had a look at the manifest

cat manifest-x86_64-python3-msgcheck-native.populate_sysroot
/build/poky/build/tmp/sysroots-components/x86_64/python3-msgcheck-native/usr/lib/python3.7/site-packages/msgcheck/__init__.py
/build/poky/build/tmp/sysroots-components/x86_64/python3-msgcheck-native/usr/lib/python3.7/site-packages/msgcheck/po.py
/build/poky/build/tmp/sysroots-components/x86_64/python3-msgcheck-native/usr/lib/python3.7/site-packages/msgcheck/msgcheck.py
/build/poky/build/tmp/sysroots-components/x86_64/python3-msgcheck-native/usr/lib/python3.7/site-packages/msgcheck/__pycache__/__init__.cpython-37.pyc
/build/poky/build/tmp/sysroots-components/x86_64/python3-msgcheck-native/usr/lib/python3.7/site-packages/msgcheck/__pycache__/po.cpython-37.pyc
/build/poky/build/tmp/sysroots-components/x86_64/python3-msgcheck-native/usr/lib/python3.7/site-packages/msgcheck/__pycache__/msgcheck.cpython-37.pyc
/build/poky/build/tmp/sysroots-components/x86_64/python3-msgcheck-native/usr/lib/python3.7/site-packages/msgcheck-2.8-py3.7.egg-info/dependency_links.txt
/build/poky/build/tmp/sysroots-components/x86_64/python3-msgcheck-native/usr/lib/python3.7/site-packages/msgcheck-2.8-py3.7.egg-info/requires.txt
/build/poky/build/tmp/sysroots-components/x86_64/python3-msgcheck-native/usr/lib/python3.7/site-packages/msgcheck-2.8-py3.7.egg-info/top_level.txt
/build/poky/build/tmp/sysroots-components/x86_64/python3-msgcheck-native/usr/lib/python3.7/site-packages/msgcheck-2.8-py3.7.egg-info/SOURCES.txt
/build/poky/build/tmp/sysroots-components/x86_64/python3-msgcheck-native/usr/lib/python3.7/site-packages/msgcheck-2.8-py3.7.egg-info/PKG-INFO
/build/poky/build/tmp/sysroots-components/x86_64/python3-msgcheck-native/usr/lib/python3.7/site-packages/msgcheck-2.8-py3.7.egg-info/entry_points.txt
/build/poky/build/tmp/sysroots-components/x86_64/python3-msgcheck-native/usr/bin/msgcheck
/build/poky/build/tmp/sysroots-components/x86_64/python3-msgcheck-native/sysroot-providers/python3-msgcheck-native
/build/poky/build/tmp/sysroots-components/x86_64/python3-msgcheck-native/usr/lib/python3.7/site-packages/msgcheck-2.8-py3.7.egg-info/
/build/poky/build/tmp/sysroots-components/x86_64/python3-msgcheck-native/usr/lib/python3.7/site-packages/msgcheck/__pycache__/
/build/poky/build/tmp/sysroots-components/x86_64/python3-msgcheck-native/usr/lib/python3.7/site-packages/msgcheck/
/build/poky/build/tmp/sysroots-components/x86_64/python3-msgcheck-native/usr/lib/python3.7/site-packages/
/build/poky/build/tmp/sysroots-components/x86_64/python3-msgcheck-native/usr/lib/python3.7/
/build/poky/build/tmp/sysroots-components/x86_64/python3-msgcheck-native/sysroot-providers/
/build/poky/build/tmp/sysroots-components/x86_64/python3-msgcheck-native/usr/share/
/build/poky/build/tmp/sysroots-components/x86_64/python3-msgcheck-native/usr/bin/
/build/poky/build/tmp/sysroots-components/x86_64/python3-msgcheck-native/usr/lib/
/build/poky/build/tmp/sysroots-components/x86_64/python3-msgcheck-native/usr/

which states the file should be there, but when doing

find /build/poky/build/tmp/sysroots-components/x86_64/python3-msgcheck-native/usr/
/build/poky/build/tmp/sysroots-components/x86_64/python3-msgcheck-native/usr/
/build/poky/build/tmp/sysroots-components/x86_64/python3-msgcheck-native/usr/lib
/build/poky/build/tmp/sysroots-components/x86_64/python3-msgcheck-native/usr/lib/python3.7
/build/poky/build/tmp/sysroots-components/x86_64/python3-msgcheck-native/usr/lib/python3.7/site-packages
/build/poky/build/tmp/sysroots-components/x86_64/python3-msgcheck-native/usr/lib/python3.7/site-packages/msgcheck
/build/poky/build/tmp/sysroots-components/x86_64/python3-msgcheck-native/usr/lib/python3.7/site-packages/msgcheck/__init__.py
/build/poky/build/tmp/sysroots-components/x86_64/python3-msgcheck-native/usr/lib/python3.7/site-packages/msgcheck/__pycache__
/build/poky/build/tmp/sysroots-components/x86_64/python3-msgcheck-native/usr/lib/python3.7/site-packages/msgcheck/__pycache__/po.cpython-37.pyc
/build/poky/build/tmp/sysroots-components/x86_64/python3-msgcheck-native/usr/lib/python3.7/site-packages/msgcheck/__pycache__/msgcheck.cpython-37.pyc
/build/poky/build/tmp/sysroots-components/x86_64/python3-msgcheck-native/usr/lib/python3.7/site-packages/msgcheck/po.py
/build/poky/build/tmp/sysroots-components/x86_64/python3-msgcheck-native/usr/lib/python3.7/site-packages/msgcheck/msgcheck.py
/build/poky/build/tmp/sysroots-components/x86_64/python3-msgcheck-native/usr/lib/python3.7/site-packages/msgcheck-2.8-py3.7.egg-info
/build/poky/build/tmp/sysroots-components/x86_64/python3-msgcheck-native/usr/lib/python3.7/site-packages/msgcheck-2.8-py3.7.egg-info/dependency_links.txt
/build/poky/build/tmp/sysroots-components/x86_64/python3-msgcheck-native/usr/lib/python3.7/site-packages/msgcheck-2.8-py3.7.egg-info/requires.txt
/build/poky/build/tmp/sysroots-components/x86_64/python3-msgcheck-native/usr/lib/python3.7/site-packages/msgcheck-2.8-py3.7.egg-info/top_level.txt
/build/poky/build/tmp/sysroots-components/x86_64/python3-msgcheck-native/usr/lib/python3.7/site-packages/msgcheck-2.8-py3.7.egg-info/SOURCES.txt
/build/poky/build/tmp/sysroots-components/x86_64/python3-msgcheck-native/usr/lib/python3.7/site-packages/msgcheck-2.8-py3.7.egg-info/PKG-INFO
/build/poky/build/tmp/sysroots-components/x86_64/python3-msgcheck-native/usr/lib/python3.7/site-packages/msgcheck-2.8-py3.7.egg-info/entry_points.txt
/build/poky/build/tmp/sysroots-components/x86_64/python3-msgcheck-native/usr/bin
/build/poky/build/tmp/sysroots-components/x86_64/python3-msgcheck-native/usr/bin/msgcheck
/build/poky/build/tmp/sysroots-components/x86_64/python3-msgcheck-native/usr/share

the file isn't there.

This happens to random python packages compiled as native (sometimes even for python-native itself), but (afaik) not for cross or target packages (I'm pretty sure because of the different packaging applied).
So I digged a little into the code classes/sstate.bbclass:sstate_install, which seems to create the sysroot-component dir and the manifest.
There is a gap between the manifest creation (line 285) and the hardlinking (line till 311).

Now my question is there any place where a file potentially could get lost? - at first glance there shouldn't be one - I have to admit that I don't fully understand all this subprocess magic in lib/oe/path.py:copyhardlinktree.
What I do to fix the issue is reopening the manifest and double check for missing files and remove them from the manifest, but this
feels wrong - so any advise is welcome...

Hope that someone more familiar with the topic could have a look.

Thanks

Konrad



    


Image size reduction

Ajam Ali
 

Hi All, 

Actually my current image size is 3.9GB because of heavy size packages required by my project and without project required packages my image size in Yocto is 800MB.

I want to reduce the image size as maximum as possible.
Please suggest the best possible way so that I could reduce the maximum possible size(desirable below 1.5 GB). 


Thanks in advance,
Ajam Ali


Sent from Outlook Mobile
::DISCLAIMER::

The contents of this e-mail and any attachment(s) are confidential and intended for the named recipient(s) only. E-mail transmission is not guaranteed to be secure or error-free as information could be intercepted, corrupted, lost, destroyed, arrive late or incomplete, or may contain viruses in transmission. The e mail and its contents (with or without referred errors) shall therefore not attach any liability on the originator or HCL or its affiliates. Views or opinions, if any, presented in this email are solely those of the author and may not necessarily reflect the views or opinions of HCL or its affiliates. Any form of reproduction, dissemination, copying, disclosure, modification, distribution and / or publication of this message without the prior written consent of authorized representative of HCL is strictly prohibited. If you have received this email in error please delete it and notify the sender immediately. Before opening any email and/or attachments, please check them for viruses and other defects.

4881 - 4900 of 53871