Date   

Re: yocto meta-intel preempt-rt

Leon Woestenberg
 

Hello Steve,

I have been running the PREEMPT_RT for Intel platforms for a few
releases, I remember it fetched sources from a GIT branch for -rt
(PREEMPT_RT). So it does not need to apply separate patches in that
case.

Could you check if this is the case for you as well?

Regards,

Leon.

p.s. Slightly off-topic, I moved away from PREEMPT_RT in favor of task
isolation mode, where we use one CPU core in isolated single task
mode, for *much* lower latencies than PREEMPT_RT can provide.


--
Leon Woestenberg
leon@...
T: +31 40 711 42 76
M: +31 6 472 30 372

Sidebranch Embedded Systems
Eindhoven, The Netherlands
http://www.sidebranch.com



On Tue, Nov 2, 2021 at 12:30 PM Monsees, Steven C (US) via
lists.yoctoproject.org
<steven.monsees=baesystems.com@...> wrote:




I have an Intel based platform, and was looking to implement preempt-rt on it to test.



For Intel, the meta-intel component recipe appears to supports “linux-intel-rt”, and I can build my intel based platform with this and it boots… My platform kernel is currently 4.19 based under the meta-inteI component I do not see the rt patch/patches being applied…



Is meta-intel component recipe building the full preempt-rt support ?, or do I still need to apply the patch ?



Thanks,

Steve






yocto meta-intel preempt-rt

Monsees, Steven C (US)
 

 

I have an Intel based platform, and was looking to implement preempt-rt on it to test.

 

For Intel, the meta-intel component recipe appears to supports “linux-intel-rt”, and I can build my intel based platform with this  and it boots…  My platform kernel is currently 4.19 based under the meta-inteI component I do not see the rt patch/patches being applied…

 

Is meta-intel component recipe building the full preempt-rt support ?, or do I still need to apply the patch ?

 

Thanks,

Steve

 


Re: #swupdate integration error. #swupdate

tomzy
 

Hi Vishal,

It looks like there is some problem with `u-boot-fw-utils` compilation. Are you
using U-Boot in your system? Which target do you try to build? As you can see
[here](https://github.com/sbabic/meta-swupdate/blob/f2d65d87485ada5a2d3a744fd7b9e46ec7e6b9f2/recipes-support/swupdate/swupdate.inc#L73)

building update image adds `u-boot-fw-utils` to DEPENDS when U-Boot is used. And
from the logs it looks like the compilation fails because of wrong
`UBOOT_MACHINE` set. See
[here](https://git.yoctoproject.org/cgit/cgit.cgi/poky/tree/meta/recipes-bsp/u-boot/u-boot-fw-utils_2018.01.bb?h=sumo&id=b39f4146de84d7b36861859ec669d9c8e2ca77c6#n13)

 

Cheers,
--
Tomasz Żyjewski
Embedded Systems Engineer
GPG: 5C495EA3EBEECA59
https://3mdeb.com | @3mdeb_com


[meta-zephyr][PATCH] README.txt: update for honister release

Naveen Saini
 

Signed-off-by: Naveen Saini <naveen.kumar.saini@...>
---
README.txt | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/README.txt b/README.txt
index 5a0ccc7..9889b01 100644
--- a/README.txt
+++ b/README.txt
@@ -9,9 +9,9 @@ Prerequisites:
==============

This layer depends on:
- Yocto distro (master)
+ Yocto distro (honister)
git://git.yoctoproject.org/poky
- Python layer (meta-openembedded/meta-python)
+ Python layer (meta-openembedded/meta-python) (honister)
git://git.openembedded.org/meta-openembedded

Modify local conf by adding:
--
2.17.1


Re: Specified SDKMACHINE value is not valid

Khem Raj
 

On 11/1/21 3:52 PM, Richard Purdie wrote:
On Mon, 2021-11-01 at 21:50 +0100, Josef Holzmayr wrote:
(re-adding list as I messed up)

Am Mo., 1. Nov. 2021 um 21:45 Uhr schrieb jchludzinski
<jchludzinski@...>:

I want to build this image for a Raspberry Pi, which means ARM.
MACHINE = "raspberrypi3"
(for example, pick your specific one like
http://git.yoctoproject.org/cgit/cgit.cgi/meta-raspberrypi/tree/conf/machine
)

Is that not possible?

That can't be true!
Building the SDK for an architecture other than x86 is not supported
at the moment.
Not quite true, aarch64 is supported and tested as an SDKMACHINE value.
at one point, I had ppc64le working as SDKMACHINE target as well.


But again, this is about the SDK. And you probably just
want to build the image, where the standard procedures apply. Add the
BSP layer, set MACHINE.
Right, this sounds like MACHINE is wanted for raspberrypi.
Cheers,
Richard


Re: Specified SDKMACHINE value is not valid

Richard Purdie
 

On Mon, 2021-11-01 at 21:50 +0100, Josef Holzmayr wrote:
(re-adding list as I messed up)

Am Mo., 1. Nov. 2021 um 21:45 Uhr schrieb jchludzinski
<jchludzinski@...>:

I want to build this image for a Raspberry Pi, which means ARM.
MACHINE = "raspberrypi3"
(for example, pick your specific one like
http://git.yoctoproject.org/cgit/cgit.cgi/meta-raspberrypi/tree/conf/machine
)

Is that not possible?

That can't be true!
Building the SDK for an architecture other than x86 is not supported
at the moment.
Not quite true, aarch64 is supported and tested as an SDKMACHINE value.

But again, this is about the SDK. And you probably just
want to build the image, where the standard procedures apply. Add the
BSP layer, set MACHINE.
Right, this sounds like MACHINE is wanted for raspberrypi.

Cheers,

Richard


Re: Specified SDKMACHINE value is not valid

Josef Holzmayr
 

(re-adding list as I messed up)

Am Mo., 1. Nov. 2021 um 21:45 Uhr schrieb jchludzinski
<jchludzinski@...>:

I want to build this image for a Raspberry Pi, which means ARM.
MACHINE = "raspberrypi3"
(for example, pick your specific one like
http://git.yoctoproject.org/cgit/cgit.cgi/meta-raspberrypi/tree/conf/machine
)

Is that not possible?

That can't be true!
Building the SDK for an architecture other than x86 is not supported
at the moment. But again, this is about the SDK. And you probably just
want to build the image, where the standard procedures apply. Add the
BSP layer, set MACHINE.

---John


On 2021-11-01 05:55, Josef Holzmayr wrote:
jchludzinski via lists.yoctoproject.org [1]
<jchludzinski=vivaldi.net@...> schrieb am Mo., 1.
Nov. 2021, 07:27:

NEWBIE question!

I tried building and I get: "_Specified SDKMACHINE value is not
valid_"

pi@raspberrypi ~/p/build> bitbake core-image-minimal
/usr/lib/python3/dist-packages/html5lib/_trie/_base.py:3:
DeprecationWarning: Using or importing the ABCs from 'collections'
instead of from 'collections.abc' is deprecated, and in 3.8 it will
stop working
from collections import Mapping
WARNING: Host distribution "raspbian-10" has not been validated with
this version of the build system; you may possibly experience
unexpected failures. It is recommended that you use a tested
distribution.
ERROR: OE-core's config sanity checker detected a potential
misconfiguration.
Either fix the cause of this error or at your own risk disable
the checker (see sanity.conf).
Following is the list of potential problems / advisories:

Specified SDKMACHINE value is not valid

Summary: There was 1 WARNING message shown.
Summary: There was 1 ERROR message shown, returning a non-zero exit
code.
I've tried setting SDKMACHINE to 'arm' and 'qemuarm' but get the
same message?
SDKMACHINE only accepts x86, in either 32 or 64 bit variants, see
https://docs.yoctoproject.org/ref-manual/variables.html#term-SDKMACHINE

Obviously I failed to do something (properly)?

Ideas?

Links:
------
[1] http://lists.yoctoproject.org


M+ & H bugs with Milestone Movements WW44

Stephen Jolley
 

All,

YP M+ or high bugs which moved to a new milestone in WW44 are listed below:

Priority

Bug ID

Short Description

Changer

Owner

Was

Became

Medium+

12917

Warnings from nightly-multilib builds (build-deps)

kai.kang@...

kai.kang@...

3.4 M4

3.5

 

13338

SDK  build fails if image contains bash

randy.macleod@...

unassigned@...

3.4 M4

3.5 M1

 

13766

Using TCLIB=musl results in SDKs producing incompatible binaries

randy.macleod@...

sakib.sajal@...

3.5 M3

3.5 M1

 

14126

resolvconf incompatible with busybox flock

randy.macleod@...

newcomer@...

3.4 M4

3.5 M2

 

14157

git fetcher: consider using different git commands for repo packing, eliminating "git pack-redundant"

randy.macleod@...

newcomer@...

3.4 M4

3.5 M1

 

14185

Git 2.30.0 defaults to main

randy.macleod@...

richard.purdie@...

3.4 M4

3.5 M1

 

14522

qemuppc doesn't shutdown within timeout (serial console issues)

randy.macleod@...

randy.macleod@...

3.6

3.5

 

14553

insane.bbclass: host-user-contaminated QA doesn't skip the home directory

randy.macleod@...

kiran.surendran@...

3.4 M4

3.5 M1

Thanks,

 

Stephen K. Jolley

Yocto Project Program Manager

(    Cell:                (208) 244-4460

* Email:              sjolley.yp.pm@...

 


Enhancements/Bugs closed WW44!

Stephen Jolley
 

All,

The below were the owners of enhancements or bugs closed during the last week!

Who

Count

randy.macleod@...

1

mickael.laventure+yocto@...

1

steve@...

1

Grand Total

3

 

Thanks,

 

Stephen K. Jolley

Yocto Project Program Manager

(    Cell:                (208) 244-4460

* Email:              sjolley.yp.pm@...

 


Current high bug count owners for Yocto Project 3.5

Stephen Jolley
 

All,

Below is the list as of top 50 bug owners as of the end of WW44 of who have open medium or higher bugs and enhancements against YP 3.5.   There are 123 possible work days left until the final release candidates for YP 3.5 needs to be released.

Who

Count

ross@...

37

michael.opdenacker@...

35

david.reyna@...

22

randy.macleod@...

19

bruce.ashfield@...

17

trevor.gamblin@...

16

timothy.t.orling@...

14

JPEWhacker@...

12

sakib.sajal@...

11

richard.purdie@...

10

mhalstead@...

7

kai.kang@...

7

bluelightning@...

6

Qi.Chen@...

5

kiran.surendran@...

5

saul.wold@...

4

hongxu.jia@...

4

chee.yang.lee@...

3

pgowda.cve@...

3

jon.mason@...

3

pokylinux@...

2

mshah@...

2

mingli.yu@...

2

alejandro@...

2

akuster808@...

2

angolini@...

1

weaverjs@...

1

alexandre.belloni@...

1

sangeeta.jain@...

1

kexin.hao@...

1

mostthingsweb@...

1

yoctoproject@...

1

anuj.mittal@...

1

diego.sueiro@...

1

Martin.Jansa@...

1

vinay.m.engg@...

1

open.source@...

1

TicoTimo@...

1

jaewon@...

1

mister_rs@...

1

nicolas.dechesne@...

1

john.kaldas.enpj@...

1

yi.zhao@...

1

ydirson@...

1

steve@...

1

yf3yu@...

1

jeanmarie.lemetayer@...

1

jay.shen.teoh@...

1

aehs29@...

1

raj.khem@...

1

matthewzmd@...

1

 

Thanks,

 

Stephen K. Jolley

Yocto Project Program Manager

(    Cell:                (208) 244-4460

* Email:              sjolley.yp.pm@...

 


Reminder: Yocto Project Technical Team Meeting @ Monthly from 8am on the first Tuesday (PDT)

Stephen Jolley
 

All,

 

Just a reminder we will hold the monthly Yocto Project Technical Meeting at 8am PST tomorrow. (11/2) 

 

Yocto Project Technical Team Meeting: We encourage people attending the meeting to logon and announce themselves on the Yocto Project IRC chancel during the meeting (optional):

Yocto IRC: https://web.libera.chat/#yocto 

Wiki: https://www.yoctoproject.org/public-virtual-meetings/

 

When            Monthly from 8am to 9am on the first Tuesday Pacific Time

Where           Zoom Meeting: https://zoom.us/j/990892712?pwd=cHU1MjhoM2x6ck81bkcrYjRrcmJsUT09

 

We are tracking the minutes at: https://docs.google.com/document/d/1ly8nyhO14kDNnFcW2QskANXW3ZT7QwKC5wWVDg9dDH4/edit?pli=1 Please request access if you want to assist in editing them.  The world should have view access.

 

Thanks,

 

Stephen K. Jolley

Yocto Project Program Manager

(    Cell:                (208) 244-4460

* Email:              sjolley.yp.pm@...

 


Re: Specified SDKMACHINE value is not valid

Ross Burton <ross@...>
 

As per https://docs.yoctoproject.org/ref-manual/variables.html#term-SDKMACHINE
the value isn't a target MACHINE, but a name that is present in
conf/machine-sdk:

https://git.openembedded.org/openembedded-core/tree/meta/conf/machine-sdk

If you want to build a SDK to run on an arm host, you most likely want
aarch64. A 32-bit Arm SDK hasn't been tested, but writing a new
configuration file for that shouldn't be that difficult.

Ross

On Mon, 1 Nov 2021 at 06:27, jchludzinski via lists.yoctoproject.org
<jchludzinski=vivaldi.net@...> wrote:

NEWBIE question!

I tried building and I get: "Specified SDKMACHINE value is not valid"

pi@raspberrypi ~/p/build> bitbake core-image-minimal
/usr/lib/python3/dist-packages/html5lib/_trie/_base.py:3: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated, and in 3.8 it will stop working
from collections import Mapping
WARNING: Host distribution "raspbian-10" has not been validated with this version of the build system; you may possibly experience unexpected failures. It is recommended that you use a tested distribution.
ERROR: OE-core's config sanity checker detected a potential misconfiguration.
Either fix the cause of this error or at your own risk disable the checker (see sanity.conf).
Following is the list of potential problems / advisories:

Specified SDKMACHINE value is not valid


Summary: There was 1 WARNING message shown.
Summary: There was 1 ERROR message shown, returning a non-zero exit code.

I've tried setting SDKMACHINE to 'arm' and 'qemuarm' but get the same message?

Obviously I failed to do something (properly)?

Ideas?



Specified SDKMACHINE value is not valid

jchludzinski
 

NEWBIE question!
 
I tried building and I get: "Specified SDKMACHINE value is not valid"

pi@raspberrypi ~/p/build> bitbake core-image-minimal
/usr/lib/python3/dist-packages/html5lib/_trie/_base.py:3: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated, and in 3.8 it will stop working
  from collections import Mapping
WARNING: Host distribution "raspbian-10" has not been validated with this version of the build system; you may possibly experience unexpected failures. It is recommended that you use a tested distribution.
ERROR:  OE-core's config sanity checker detected a potential misconfiguration.
    Either fix the cause of this error or at your own risk disable the checker (see sanity.conf).
    Following is the list of potential problems / advisories:
 
    Specified SDKMACHINE value is not valid
 
 
Summary: There was 1 WARNING message shown.
Summary: There was 1 ERROR message shown, returning a non-zero exit code.

I've tried setting SDKMACHINE to 'arm' and 'qemuarm' but get the same message?

Obviously I failed to do something (properly)?

Ideas?


[yocto-autobuilder-helper] [PATCH 4/4] shared-repos: Use tar instead of rsync for speed

Richard Purdie
 

The rysnc of 20,000 files (650MB) onto the nas is slow taking ~3 minutes
at idle and worse at load. This is due to the number of files which
is a pain point for NFS. This piece of the build is also a bottleneck
since the rest of a build depends on it happening.

If we switch to zstd compressed tar, it takes 2.49s. Other compression
methods were much slower but zstd seems 'accptable' and speeds things
up too.

Signed-off-by: Richard Purdie <richard.purdie@...>
---
scripts/prepare-shared-repos | 4 ++--
scripts/send-qa-email | 6 ++++--
scripts/shared-repo-unpack | 5 ++---
3 files changed, 8 insertions(+), 7 deletions(-)

diff --git a/scripts/prepare-shared-repos b/scripts/prepare-shared-repos
index f34ba99..d60ad32 100755
--- a/scripts/prepare-shared-repos
+++ b/scripts/prepare-shared-repos
@@ -39,5 +39,5 @@ with tempfile.TemporaryDirectory(prefix="shared-repo-temp-", dir="/home/pokybuil
if args.publish_dir:
utils.publishrepo(tempdir, repo, args.publish_dir)

- utils.printheader("Running rsync")
- subprocess.check_call("rsync -a " + tempdir + "/* " + args.sharedsrcdir, shell=True)
+ utils.printheader("Creating shared src tarball")
+ subprocess.check_call("tar -I zstd -cf " + args.sharedsrcdir.rstrip("/") + ".tar.zst ./*", shell=True, cwd=tempdir)
diff --git a/scripts/send-qa-email b/scripts/send-qa-email
index 1b69307..fc34f7e 100755
--- a/scripts/send-qa-email
+++ b/scripts/send-qa-email
@@ -45,9 +45,11 @@ buildtoolsdir = os.path.dirname(args.repojson) + "/build/buildtools"
if os.path.exists(buildtoolsdir):
utils.enable_buildtools_tarball(buildtoolsdir)

+repodir = os.path.dirname(args.repojson) + "/repos"
+
if 'poky' in repos and os.path.exists(resulttool) and args.results_dir:
# Need the finalised revisions (not 'HEAD')
- targetrepodir = "%s/poky" % (args.sharedrepodir)
+ targetrepodir = "%s/poky" % (repodir)
revision = subprocess.check_output(["git", "rev-parse", "HEAD"], cwd=targetrepodir).decode('utf-8').strip()
branch = repos['poky']['branch']
repo = repos['poky']['url']
@@ -116,7 +118,7 @@ if args.send.lower() != 'true' or not args.publish_dir or not args.release:
buildhashes = ""
for repo in sorted(repos.keys()):
# Need the finalised revisions (not 'HEAD')
- targetrepodir = "%s/%s" % (args.sharedrepodir, repo)
+ targetrepodir = "%s/%s" % (repodir, repo)
revision = subprocess.check_output(["git", "rev-parse", "HEAD"], cwd=targetrepodir).decode('utf-8').strip()
buildhashes += "%s: %s\n" % (repo, revision)

diff --git a/scripts/shared-repo-unpack b/scripts/shared-repo-unpack
index f08efa8..3dda5b3 100755
--- a/scripts/shared-repo-unpack
+++ b/scripts/shared-repo-unpack
@@ -50,11 +50,10 @@ needrepos_baseddirs = [r.split('/')[0] for r in needrepos]
for repo in sorted(repos.keys()):
if repo not in needrepos_baseddirs:
continue
- targetrepodir = "%s/%s" % (targetsubdir, repo)
if args.cache_dir:
utils.printheader("Copying in repo %s" % repo)
- utils.mkdir(targetrepodir)
- subprocess.check_call(["rsync", "-a", "%s/%s" % (args.cache_dir, repo), targetsubdir])
+ utils.mkdir(targetsubdir)
+ subprocess.check_call(["tar", "-I", "zstd", "-C", targetsubdir, "-xf", "%s.tar.zst" % args.cache_dir, "./" + repo])
else:
utils.printheader("Fetching repo %s" % repo)
utils.fetchgitrepo(targetsubdir, repo, repos[repo], stashdir)
--
2.32.0


[yocto-autobuilder-helper] [PATCH 3/4] prepare-shared-repo/utils: Limit HEAD clones to shallow depth to save time/space

Richard Purdie
 

By not syncing all the history is is possible to save some time/space
in the checkout process since we never use this data. This reduces data
from 650MB to 400MB or with the tarball, 416MB to 55MB.

The logic for the commands needs to be tweaked to handle this and as
written it can't work in non-HEAD revision case but that isn't a commonly
used situation.

Signed-off-by: Richard Purdie <richard.purdie@...>
---
scripts/prepare-shared-repos | 2 +-
scripts/utils.py | 18 +++++++++++++-----
2 files changed, 14 insertions(+), 6 deletions(-)

diff --git a/scripts/prepare-shared-repos b/scripts/prepare-shared-repos
index c221e69..f34ba99 100755
--- a/scripts/prepare-shared-repos
+++ b/scripts/prepare-shared-repos
@@ -35,7 +35,7 @@ stashdir = utils.getconfig("REPO_STASH_DIR", ourconfig)
with tempfile.TemporaryDirectory(prefix="shared-repo-temp-", dir="/home/pokybuild/tmp") as tempdir:
for repo in sorted(repos.keys()):
utils.printheader("Intially fetching repo %s" % repo)
- utils.fetchgitrepo(tempdir, repo, repos[repo], stashdir)
+ utils.fetchgitrepo(tempdir, repo, repos[repo], stashdir, depth=1)
if args.publish_dir:
utils.publishrepo(tempdir, repo, args.publish_dir)

diff --git a/scripts/utils.py b/scripts/utils.py
index 4c73f81..3c2622f 100644
--- a/scripts/utils.py
+++ b/scripts/utils.py
@@ -228,26 +228,34 @@ def runcmd(cmd):
return subprocess.check_output(cmd, stderr=subprocess.STDOUT)


-def fetchgitrepo(clonedir, repo, params, stashdir):
+def fetchgitrepo(clonedir, repo, params, stashdir, depth=None):
sharedrepo = "%s/%s" % (clonedir, repo)
branch = params["branch"]
revision = params["revision"]
+ if revision != "HEAD":
+ depth = None
+ fetchopt = []
+ depthopt = []
+ if depth:
+ fetchopt = ["--depth", str(depth), branch + ":origin/" + branch]
+ depthopt = ["--depth", str(depth), "--branch", branch]
print("Checking for stash at: " + stashdir + "/" + repo)
flush()
if os.path.exists(stashdir + "/" + repo):
print("Cloning from stash to %s..." % sharedrepo)
flush()
- subprocess.check_call(["git", "clone", "file://%s/%s" % (stashdir, repo), "%s/%s" % (clonedir, repo)])
+ subprocess.check_call(["git", "clone", "file://%s/%s" % (stashdir, repo), "%s/%s" % (clonedir, repo)] + depthopt)
subprocess.check_call(["git", "remote", "rm", "origin"], cwd=sharedrepo)
subprocess.check_call(["git", "remote", "add", "origin", params["url"]], cwd=sharedrepo)
print("Updating from origin...")
flush()
- subprocess.check_call(["git", "fetch", "origin"], cwd=sharedrepo)
- subprocess.check_call(["git", "fetch", "origin", "-t"], cwd=sharedrepo)
+ subprocess.check_call(["git", "fetch", "origin"] + fetchopt, cwd=sharedrepo)
+ if not depth:
+ subprocess.check_call(["git", "fetch", "origin", "-t"], cwd=sharedrepo)
else:
print("Cloning from origin to %s..." % sharedrepo)
flush()
- subprocess.check_call(["git", "clone", params["url"], sharedrepo])
+ subprocess.check_call(["git", "clone", params["url"], sharedrepo] + depthopt)

print("Updating checkout...")
flush()
--
2.32.0


[yocto-autobuilder-helper] [PATCH 2/4] prepare-shared-repos: Make it clear when rsync starts in logs

Richard Purdie
 

Signed-off-by: Richard Purdie <richard.purdie@...>
---
scripts/prepare-shared-repos | 1 +
1 file changed, 1 insertion(+)

diff --git a/scripts/prepare-shared-repos b/scripts/prepare-shared-repos
index 1573f85..c221e69 100755
--- a/scripts/prepare-shared-repos
+++ b/scripts/prepare-shared-repos
@@ -39,4 +39,5 @@ with tempfile.TemporaryDirectory(prefix="shared-repo-temp-", dir="/home/pokybuil
if args.publish_dir:
utils.publishrepo(tempdir, repo, args.publish_dir)

+ utils.printheader("Running rsync")
subprocess.check_call("rsync -a " + tempdir + "/* " + args.sharedsrcdir, shell=True)
--
2.32.0


[yocto-autobuilder-helper] [PATCH 1/4] scripts/prepare-shared-repos: Use tmpfs for speed

Richard Purdie
 

Signed-off-by: Richard Purdie <richard.purdie@...>
---
scripts/prepare-shared-repos | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/scripts/prepare-shared-repos b/scripts/prepare-shared-repos
index 8400d09..1573f85 100755
--- a/scripts/prepare-shared-repos
+++ b/scripts/prepare-shared-repos
@@ -32,7 +32,7 @@ with open(args.repojson) as f:

stashdir = utils.getconfig("REPO_STASH_DIR", ourconfig)

-with tempfile.TemporaryDirectory(prefix="shared-repo-temp-", dir="/tmp") as tempdir:
+with tempfile.TemporaryDirectory(prefix="shared-repo-temp-", dir="/home/pokybuild/tmp") as tempdir:
for repo in sorted(repos.keys()):
utils.printheader("Intially fetching repo %s" % repo)
utils.fetchgitrepo(tempdir, repo, repos[repo], stashdir)
--
2.32.0


[yocto-autobuilder2] [PATCH] builders: Fix quarantine handling

Richard Purdie
 

The way the canStartBuild code was written, it inserted a delay between
each build starting of 2 minutes unconditionally. We only want to do this
if the worker had run out of space so tweak the code accordingly.

Signed-off-by: Richard Purdie <richard.purdie@...>
---
builders.py | 15 ++++++++-------
1 file changed, 8 insertions(+), 7 deletions(-)

diff --git a/builders.py b/builders.py
index 20ad0bb2..e575a1a6 100644
--- a/builders.py
+++ b/builders.py
@@ -69,16 +69,17 @@ def canStartBuild(builder, wfb, request):
threshold = 60 # GB of space
if int(cmd.stdout) < threshold:
log.msg("Detected {0} GB of space available, less than threshold of {1} GB. Can't start build".format(cmd.stdout, threshold))
+ wfb.worker.quarantine_timeout = 2 * 60
wfb.worker.putInQuarantine()
return False
- else:
- log.msg("Detected {0} GB of space available, more than threshold of {1} GB. OK to build".format(cmd.stdout, threshold))
-
- wfb.worker.quarantine_timeout = 120
- wfb.worker.putInQuarantine()
-
- wfb.worker.resetQuarantine()

+ log.msg("Detected {0} GB of space available, more than threshold of {1} GB. OK to build".format(cmd.stdout, threshold))
+ if wfb.worker.isPaused:
+ # It was low on space so delay the builds starting a bit
+ wfb.worker.quarantine_timeout = 2 * 60
+ wfb.worker.putInQuarantine()
+ else:
+ wfb.worker.exitQuarantine()
return True

def create_builder_factory():
--
2.32.0


Re: [yocto-autobuilder2][PATCH] builders.py: subtract builder weight instead of add

Richard Purdie
 

On Fri, 2021-10-29 at 11:39 -0400, Trevor Gamblin wrote:
Signed-off-by: Trevor Gamblin <trevor.gamblin@...>
---
builders.py | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/builders.py b/builders.py
index f94d1dd..fc92e36 100644
--- a/builders.py
+++ b/builders.py
@@ -168,7 +168,7 @@ def prioritizeBuilders(master, builders):
time = max_time
else:
if bldr.name in builder_bonuses:
- time = time + builder_bonuses[bldr.name]
+ time = time - builder_bonuses[bldr.name]
defer.returnValue((time, bldr))

transformed = yield defer.gatherResults(
This didn't work (and wasn't merged) although I still think it is probably the
correct thing to do with that code.

I did some digging and the issue is that when we trigger builds, it triggers
them one at a time as part of a buildSet and the prioritisation function
therefore sees each build by itself and therefore gives it priority.

That raises the question of the order the triggers are run in. A list of builder
names is passed which is translated to a list of builderids. I think the order
is therefore determined by the database ID of the builders. I did wonder if
sorting the builders list would help but it does not.

There is code in schedulers/base.py which does:

builderids = list()
for bldr in (yield self.master.data.get(('builders', ))):
if bldr['name'] in builderNames:
builderids.append(bldr['builderid'])

which I think is where the sorting comes from.

I think we're going to have discuss this with upstream.

Cheers,

Richard


Re: [yocto-autobuilder2][PATCH] builders.py: Add canStartBuild disk space check

Richard Purdie
 

On Fri, 2021-10-29 at 09:22 -0400, Trevor Gamblin wrote:
We need a way to limit the builds for when a given worker has less than
a certain amount of disk space available. This implements a
canStartBuild method based on the example in the Buildbot docs and
blocks a build if the worker has less than 60GB of disk space available.
Unlike the example code, we want the stdout of the command so that we
can calculate the amount of disk space, rather than just relying on the
remote command's return code.

Docs: https://docs.buildbot.net/latest/manual/customization.html#canstartbuild-functions

[YOCTO #14591]

Signed-off-by: Trevor Gamblin <trevor.gamblin@...>
---
builders.py | 44 ++++++++++++++++++++++++++++++++++++++++----
1 file changed, 40 insertions(+), 4 deletions(-)

diff --git a/builders.py b/builders.py
index 5773950..0d1facc 100644
--- a/builders.py
+++ b/builders.py
@@ -8,6 +8,7 @@ from yoctoabb import config
from yoctoabb.steps.writelayerinfo import WriteLayerInfo
from yoctoabb.steps.runconfig import get_publish_dest, get_publish_resultdir, get_publish_name, RunConfigCheckSteps, TargetPresent
from buildbot.process.results import Results, SUCCESS, FAILURE, CANCELLED, WARNINGS, SKIPPED, EXCEPTION, RETRY
+from buildbot.process.remotecommand import RemoteCommand

from twisted.python import log
from twisted.internet import defer
@@ -45,6 +46,41 @@ def ensure_props_set(props):
"publish_destination": props.getProperty("publish_destination", "")
}

+@...
+def shell(command, worker, builder):
+ args = {
+ 'command': command,
+ 'workdir': worker.worker_basedir,
+ 'logEnviron': False,
+ 'want_stdout': True,
+ 'want_stderr': False,
+ }
+
+ cmd = RemoteCommand('shell', args, collectStdout=True, stdioLogName="stdio")
+ cmd.worker = worker
+ yield cmd.run(None, worker.conn, builder.name)
+ return cmd
+
+@...
+def canStartBuild(builder, wfb, request):
+ log.msg("Checking available disk space...")
+
+ cmd = yield shell("df -BG | grep $(findmnt -T . | awk '{print $2}' | sed -n 2p) | awk '{print $4}' | sed 's/[^0-9]*//g'", wfb.worker, builder)
+ threshold = 60 # GB of space
+ if int(cmd.stdout) < threshold:
+ log.msg("Detected {0} GB of space available, less than threshold of {1} GB. Can't start build".format(cmd.stdout, threshold))
+ wfb.worker.putInQuarantine()
+ return False
+ else:
+ log.msg("Detected {0} GB of space available, more than threshold of {1} GB. OK to build".format(cmd.stdout, threshold))
+
+ wfb.worker.quarantine_timeout = 120
+ wfb.worker.putInQuarantine()
+
+ wfb.worker.resetQuarantine()
+
+ return True
+
Unfortunately this quarantine piece is causing problems. It means that even if
there is free space on the builder and always was, a maximum of one build every
2 minutes can be started.

I'll probably drop the quarantine piece as it was a nice to have soft start for
recovery rather than an essential.

Cheers,

Richard

2641 - 2660 of 57815