Date   

Re: yocto preempt-rt

Leon Woestenberg
 

On Tue, Nov 2, 2021 at 5:26 PM Bruce Ashfield <bruce.ashfield@...> wrote:

Correct.

The rt patches are already integrated on the branches that that recipe will build out of the linux-yocto repository.
And adding to that, besides the Yocto maintained kernel, meta-intel
has it's own kernel GIT repo/branch for the kernel and -rt kernel
maintained by Intel.

(There might be some overlap in the maintainers, I am not aware of the
differences between those.).

I used the latter with success on Intel Atom 3950 for hard real-time purposes.

Regards,

Leon.


Re: yocto preempt-rt

Bruce Ashfield
 

Correct.

The rt patches are already integrated on the branches that that recipe will build out of the linux-yocto repository.

Bruce

On Tue, Nov 2, 2021 at 12:23 PM Monsees, Steven C (US) via lists.yoctoproject.org <steven.monsees=baesystems.com@...> wrote:

 

Is it true that no patch work is required if out under

…/poky/meta/recipes-kernel, there exists  a yocto-linux-rt_##.##.bb recipe that matches your kernel release?, and that it will build the full preemptive RT Kernel ?

 

Thanks,

Steve

 






--
- Thou shalt not follow the NULL pointer, for chaos and madness await thee at its end
- "Use the force Harry" - Gandalf, Star Trek II


yocto preempt-rt

Monsees, Steven C (US)
 

 

Is it true that no patch work is required if out under

…/poky/meta/recipes-kernel, there exists  a yocto-linux-rt_##.##.bb recipe that matches your kernel release?, and that it will build the full preemptive RT Kernel ?

 

Thanks,

Steve

 


Yocto Project Status WW43`21

Stephen Jolley
 

Current Dev Position: YP 3.5 M1

Next Deadline: 6th Dec. 2021 YP 3.5 M1 build

 

Next Team Meetings:

 

Key Status/Updates:

  • YP 3.4 has been released. Thanks to everyone who contributed!
  • YP 3.3.4 is due to build this week and will be the last planned release of the hardknott series.
  • YP 3.5 Planning document: https://docs.google.com/document/d/1OXw-NKoL_Vb9RWI6sRPs3zTcAn4hHPtG0Y2BIs7xIzo/edit?usp=sharing
  • Git’s default branch choice could potentially change in the future and we’re seeing service providers like github change policy too. To react to this variability the project really needs to start encoding the branch name used in SRC_URI rather than having a default of master. OE-Core has been converted and there is a script ( scripts/contrib/convert-srcuri.py) to help with conversions. Bitbake will start warning where this is unset soon (patch in master-next).
  • Github has announced that git protocol support will be dropped as of January. We use this in a number of our SRC_URIs. This has been discussed on the architecture list and we have a plan to warn (and later error) on problematic urls and magically map to the correct urls within bitbake. The latter change is easily backported to older bitbake releases and will allow older branches to continue to function without invasive changes. The conversion script mentioned above can also convert github urls.
  • We have seen a drop in the number of patches in “Pending” state, partly thanks to work from Richard to send libtool and gcc patches upstream or otherwise clean them up. Others are also stepping forward to try and reduce the number of patches in Pending state, help would be much appreciated as spread over a number of people this could be quickly handled and reduced.
  • Intermittent issues continue to rise and help is very much welcome on these issues. You can see the list of failures we’re continuing to see by searching for the “AB-INT” tag in bugzilla: https://bugzilla.yoctoproject.org/buglist.cgi?quicksearch=AB-INT

 

Ways to contribute:

 

YP 3.5 Milestone Dates:

  • YP 3.5 M1 build date 2021/12/06
  • YP 3.5 M1 Release date 2021/12/17
  • YP 3.5 M2 build date 2022/01/10
  • YP 3.5 M2 Release date 2022/1/21
  • YP 3.5 M3 build date 2022/2/21
  • YP 3.5 M3 Release date 2022/03/04
  • YP 3.5 M4 build date 2022/04/04
  • YP 3.5 M4 Release date 2022/04/29

 

Upcoming dot releases:

  • YP 3.3.4 build date 2021/11/01
  • YP 3.3.4 Release date 2021/11/12
  • YP 3.1.12 build date 2021/11/15
  • YP 3.1.12 Release date 2021/11/26
  • YP 3.4.1 build date 2021/11/22
  • YP 3.4.1 Release date 2021/12/03
  • YP 3.1.13 build date 2021/12/13
  • YP 3.1.13 Release date 2021/12/22
  • YP 3.1.14 build date 2022/01/24
  • YP 3.1.14 Release date 2022/02/04
  • YP 3.4.2 build date 2022/02/07
  • YP 3.4.2 Release date 2022/02/18
  • YP 3.1.15 build date 2022/03/14
  • YP 3.1.15 Release date 2022/03/25
  • YP 3.4.3 build date 2022/03/21
  • YP 3.4.3 Release date 2022/04/01
  • YP 3.1.16 build date 2022/04/25
  • YP 3.1.16 Release date 2022/05/06

 

Tracking Metrics:

 

The Yocto Project’s technical governance is through its Technical Steering Committee, more information is available at:

https://wiki.yoctoproject.org/wiki/TSC

 

The Status reports are now stored on the wiki at: https://wiki.yoctoproject.org/wiki/Weekly_Status

 

[If anyone has suggestions for other information you’d like to see on this weekly status update, let us know!]

 

Thanks,

 

Stephen K. Jolley

Yocto Project Program Manager

(    Cell:                (208) 244-4460

* Email:              sjolley.yp.pm@...

 


Re: yocto meta-intel preempt-rt

Monsees, Steven C (US)
 

I will check...

Would you have an example of a recipe that shows how to apply the rt path ?

Thanks,
Steve

-----Original Message-----
From: Leon Woestenberg <leon@...>
Sent: Tuesday, November 2, 2021 8:25 AM
To: Monsees, Steven C (US) <steven.monsees@...>
Cc: yocto@...
Subject: Re: [yocto] yocto meta-intel preempt-rt

External Email Alert

This email has been sent from an account outside of the BAE Systems network.

Please treat the email with caution, especially if you are requested to click on a link, decrypt/open an attachment, or enable macros. For further information on how to spot phishing, access “Cybersecurity OneSpace Page” and report phishing by clicking the button “Report Phishing” on the Outlook toolbar.


Hello Steve,

I have been running the PREEMPT_RT for Intel platforms for a few releases, I remember it fetched sources from a GIT branch for -rt (PREEMPT_RT). So it does not need to apply separate patches in that case.

Could you check if this is the case for you as well?

Regards,

Leon.

p.s. Slightly off-topic, I moved away from PREEMPT_RT in favor of task isolation mode, where we use one CPU core in isolated single task mode, for *much* lower latencies than PREEMPT_RT can provide.


--
Leon Woestenberg
leon@...
T: +31 40 711 42 76
M: +31 6 472 30 372

Sidebranch Embedded Systems
Eindhoven, The Netherlands
http://www.sidebranch.com



On Tue, Nov 2, 2021 at 12:30 PM Monsees, Steven C (US) via lists.yoctoproject.org <steven.monsees=baesystems.com@...> wrote:



I have an Intel based platform, and was looking to implement preempt-rt on it to test.



For Intel, the meta-intel component recipe appears to supports
“linux-intel-rt”, and I can build my intel based platform with this
and it boots… My platform kernel is currently 4.19 based under the
meta-inteI component I do not see the rt patch/patches being applied…



Is meta-intel component recipe building the full preempt-rt support ?, or do I still need to apply the patch ?



Thanks,

Steve






Re: yocto meta-intel preempt-rt

Leon Woestenberg
 

Hello Steve,

I have been running the PREEMPT_RT for Intel platforms for a few
releases, I remember it fetched sources from a GIT branch for -rt
(PREEMPT_RT). So it does not need to apply separate patches in that
case.

Could you check if this is the case for you as well?

Regards,

Leon.

p.s. Slightly off-topic, I moved away from PREEMPT_RT in favor of task
isolation mode, where we use one CPU core in isolated single task
mode, for *much* lower latencies than PREEMPT_RT can provide.


--
Leon Woestenberg
leon@...
T: +31 40 711 42 76
M: +31 6 472 30 372

Sidebranch Embedded Systems
Eindhoven, The Netherlands
http://www.sidebranch.com



On Tue, Nov 2, 2021 at 12:30 PM Monsees, Steven C (US) via
lists.yoctoproject.org
<steven.monsees=baesystems.com@...> wrote:




I have an Intel based platform, and was looking to implement preempt-rt on it to test.



For Intel, the meta-intel component recipe appears to supports “linux-intel-rt”, and I can build my intel based platform with this and it boots… My platform kernel is currently 4.19 based under the meta-inteI component I do not see the rt patch/patches being applied…



Is meta-intel component recipe building the full preempt-rt support ?, or do I still need to apply the patch ?



Thanks,

Steve






yocto meta-intel preempt-rt

Monsees, Steven C (US)
 

 

I have an Intel based platform, and was looking to implement preempt-rt on it to test.

 

For Intel, the meta-intel component recipe appears to supports “linux-intel-rt”, and I can build my intel based platform with this  and it boots…  My platform kernel is currently 4.19 based under the meta-inteI component I do not see the rt patch/patches being applied…

 

Is meta-intel component recipe building the full preempt-rt support ?, or do I still need to apply the patch ?

 

Thanks,

Steve

 


Re: #swupdate integration error. #swupdate

tomzy
 

Hi Vishal,

It looks like there is some problem with `u-boot-fw-utils` compilation. Are you
using U-Boot in your system? Which target do you try to build? As you can see
[here](https://github.com/sbabic/meta-swupdate/blob/f2d65d87485ada5a2d3a744fd7b9e46ec7e6b9f2/recipes-support/swupdate/swupdate.inc#L73)

building update image adds `u-boot-fw-utils` to DEPENDS when U-Boot is used. And
from the logs it looks like the compilation fails because of wrong
`UBOOT_MACHINE` set. See
[here](https://git.yoctoproject.org/cgit/cgit.cgi/poky/tree/meta/recipes-bsp/u-boot/u-boot-fw-utils_2018.01.bb?h=sumo&id=b39f4146de84d7b36861859ec669d9c8e2ca77c6#n13)

 

Cheers,
--
Tomasz Żyjewski
Embedded Systems Engineer
GPG: 5C495EA3EBEECA59
https://3mdeb.com | @3mdeb_com


[meta-zephyr][PATCH] README.txt: update for honister release

Naveen Saini
 

Signed-off-by: Naveen Saini <naveen.kumar.saini@...>
---
README.txt | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/README.txt b/README.txt
index 5a0ccc7..9889b01 100644
--- a/README.txt
+++ b/README.txt
@@ -9,9 +9,9 @@ Prerequisites:
==============

This layer depends on:
- Yocto distro (master)
+ Yocto distro (honister)
git://git.yoctoproject.org/poky
- Python layer (meta-openembedded/meta-python)
+ Python layer (meta-openembedded/meta-python) (honister)
git://git.openembedded.org/meta-openembedded

Modify local conf by adding:
--
2.17.1


Re: Specified SDKMACHINE value is not valid

Khem Raj
 

On 11/1/21 3:52 PM, Richard Purdie wrote:
On Mon, 2021-11-01 at 21:50 +0100, Josef Holzmayr wrote:
(re-adding list as I messed up)

Am Mo., 1. Nov. 2021 um 21:45 Uhr schrieb jchludzinski
<jchludzinski@...>:

I want to build this image for a Raspberry Pi, which means ARM.
MACHINE = "raspberrypi3"
(for example, pick your specific one like
http://git.yoctoproject.org/cgit/cgit.cgi/meta-raspberrypi/tree/conf/machine
)

Is that not possible?

That can't be true!
Building the SDK for an architecture other than x86 is not supported
at the moment.
Not quite true, aarch64 is supported and tested as an SDKMACHINE value.
at one point, I had ppc64le working as SDKMACHINE target as well.


But again, this is about the SDK. And you probably just
want to build the image, where the standard procedures apply. Add the
BSP layer, set MACHINE.
Right, this sounds like MACHINE is wanted for raspberrypi.
Cheers,
Richard


Re: Specified SDKMACHINE value is not valid

Richard Purdie
 

On Mon, 2021-11-01 at 21:50 +0100, Josef Holzmayr wrote:
(re-adding list as I messed up)

Am Mo., 1. Nov. 2021 um 21:45 Uhr schrieb jchludzinski
<jchludzinski@...>:

I want to build this image for a Raspberry Pi, which means ARM.
MACHINE = "raspberrypi3"
(for example, pick your specific one like
http://git.yoctoproject.org/cgit/cgit.cgi/meta-raspberrypi/tree/conf/machine
)

Is that not possible?

That can't be true!
Building the SDK for an architecture other than x86 is not supported
at the moment.
Not quite true, aarch64 is supported and tested as an SDKMACHINE value.

But again, this is about the SDK. And you probably just
want to build the image, where the standard procedures apply. Add the
BSP layer, set MACHINE.
Right, this sounds like MACHINE is wanted for raspberrypi.

Cheers,

Richard


Re: Specified SDKMACHINE value is not valid

Josef Holzmayr
 

(re-adding list as I messed up)

Am Mo., 1. Nov. 2021 um 21:45 Uhr schrieb jchludzinski
<jchludzinski@...>:

I want to build this image for a Raspberry Pi, which means ARM.
MACHINE = "raspberrypi3"
(for example, pick your specific one like
http://git.yoctoproject.org/cgit/cgit.cgi/meta-raspberrypi/tree/conf/machine
)

Is that not possible?

That can't be true!
Building the SDK for an architecture other than x86 is not supported
at the moment. But again, this is about the SDK. And you probably just
want to build the image, where the standard procedures apply. Add the
BSP layer, set MACHINE.

---John


On 2021-11-01 05:55, Josef Holzmayr wrote:
jchludzinski via lists.yoctoproject.org [1]
<jchludzinski=vivaldi.net@...> schrieb am Mo., 1.
Nov. 2021, 07:27:

NEWBIE question!

I tried building and I get: "_Specified SDKMACHINE value is not
valid_"

pi@raspberrypi ~/p/build> bitbake core-image-minimal
/usr/lib/python3/dist-packages/html5lib/_trie/_base.py:3:
DeprecationWarning: Using or importing the ABCs from 'collections'
instead of from 'collections.abc' is deprecated, and in 3.8 it will
stop working
from collections import Mapping
WARNING: Host distribution "raspbian-10" has not been validated with
this version of the build system; you may possibly experience
unexpected failures. It is recommended that you use a tested
distribution.
ERROR: OE-core's config sanity checker detected a potential
misconfiguration.
Either fix the cause of this error or at your own risk disable
the checker (see sanity.conf).
Following is the list of potential problems / advisories:

Specified SDKMACHINE value is not valid

Summary: There was 1 WARNING message shown.
Summary: There was 1 ERROR message shown, returning a non-zero exit
code.
I've tried setting SDKMACHINE to 'arm' and 'qemuarm' but get the
same message?
SDKMACHINE only accepts x86, in either 32 or 64 bit variants, see
https://docs.yoctoproject.org/ref-manual/variables.html#term-SDKMACHINE

Obviously I failed to do something (properly)?

Ideas?

Links:
------
[1] http://lists.yoctoproject.org


M+ & H bugs with Milestone Movements WW44

Stephen Jolley
 

All,

YP M+ or high bugs which moved to a new milestone in WW44 are listed below:

Priority

Bug ID

Short Description

Changer

Owner

Was

Became

Medium+

12917

Warnings from nightly-multilib builds (build-deps)

kai.kang@...

kai.kang@...

3.4 M4

3.5

 

13338

SDK  build fails if image contains bash

randy.macleod@...

unassigned@...

3.4 M4

3.5 M1

 

13766

Using TCLIB=musl results in SDKs producing incompatible binaries

randy.macleod@...

sakib.sajal@...

3.5 M3

3.5 M1

 

14126

resolvconf incompatible with busybox flock

randy.macleod@...

newcomer@...

3.4 M4

3.5 M2

 

14157

git fetcher: consider using different git commands for repo packing, eliminating "git pack-redundant"

randy.macleod@...

newcomer@...

3.4 M4

3.5 M1

 

14185

Git 2.30.0 defaults to main

randy.macleod@...

richard.purdie@...

3.4 M4

3.5 M1

 

14522

qemuppc doesn't shutdown within timeout (serial console issues)

randy.macleod@...

randy.macleod@...

3.6

3.5

 

14553

insane.bbclass: host-user-contaminated QA doesn't skip the home directory

randy.macleod@...

kiran.surendran@...

3.4 M4

3.5 M1

Thanks,

 

Stephen K. Jolley

Yocto Project Program Manager

(    Cell:                (208) 244-4460

* Email:              sjolley.yp.pm@...

 


Enhancements/Bugs closed WW44!

Stephen Jolley
 

All,

The below were the owners of enhancements or bugs closed during the last week!

Who

Count

randy.macleod@...

1

mickael.laventure+yocto@...

1

steve@...

1

Grand Total

3

 

Thanks,

 

Stephen K. Jolley

Yocto Project Program Manager

(    Cell:                (208) 244-4460

* Email:              sjolley.yp.pm@...

 


Current high bug count owners for Yocto Project 3.5

Stephen Jolley
 

All,

Below is the list as of top 50 bug owners as of the end of WW44 of who have open medium or higher bugs and enhancements against YP 3.5.   There are 123 possible work days left until the final release candidates for YP 3.5 needs to be released.

Who

Count

ross@...

37

michael.opdenacker@...

35

david.reyna@...

22

randy.macleod@...

19

bruce.ashfield@...

17

trevor.gamblin@...

16

timothy.t.orling@...

14

JPEWhacker@...

12

sakib.sajal@...

11

richard.purdie@...

10

mhalstead@...

7

kai.kang@...

7

bluelightning@...

6

Qi.Chen@...

5

kiran.surendran@...

5

saul.wold@...

4

hongxu.jia@...

4

chee.yang.lee@...

3

pgowda.cve@...

3

jon.mason@...

3

pokylinux@...

2

mshah@...

2

mingli.yu@...

2

alejandro@...

2

akuster808@...

2

angolini@...

1

weaverjs@...

1

alexandre.belloni@...

1

sangeeta.jain@...

1

kexin.hao@...

1

mostthingsweb@...

1

yoctoproject@...

1

anuj.mittal@...

1

diego.sueiro@...

1

Martin.Jansa@...

1

vinay.m.engg@...

1

open.source@...

1

TicoTimo@...

1

jaewon@...

1

mister_rs@...

1

nicolas.dechesne@...

1

john.kaldas.enpj@...

1

yi.zhao@...

1

ydirson@...

1

steve@...

1

yf3yu@...

1

jeanmarie.lemetayer@...

1

jay.shen.teoh@...

1

aehs29@...

1

raj.khem@...

1

matthewzmd@...

1

 

Thanks,

 

Stephen K. Jolley

Yocto Project Program Manager

(    Cell:                (208) 244-4460

* Email:              sjolley.yp.pm@...

 


Reminder: Yocto Project Technical Team Meeting @ Monthly from 8am on the first Tuesday (PDT)

Stephen Jolley
 

All,

 

Just a reminder we will hold the monthly Yocto Project Technical Meeting at 8am PST tomorrow. (11/2) 

 

Yocto Project Technical Team Meeting: We encourage people attending the meeting to logon and announce themselves on the Yocto Project IRC chancel during the meeting (optional):

Yocto IRC: https://web.libera.chat/#yocto 

Wiki: https://www.yoctoproject.org/public-virtual-meetings/

 

When            Monthly from 8am to 9am on the first Tuesday Pacific Time

Where           Zoom Meeting: https://zoom.us/j/990892712?pwd=cHU1MjhoM2x6ck81bkcrYjRrcmJsUT09

 

We are tracking the minutes at: https://docs.google.com/document/d/1ly8nyhO14kDNnFcW2QskANXW3ZT7QwKC5wWVDg9dDH4/edit?pli=1 Please request access if you want to assist in editing them.  The world should have view access.

 

Thanks,

 

Stephen K. Jolley

Yocto Project Program Manager

(    Cell:                (208) 244-4460

* Email:              sjolley.yp.pm@...

 


Re: Specified SDKMACHINE value is not valid

Ross Burton <ross@...>
 

As per https://docs.yoctoproject.org/ref-manual/variables.html#term-SDKMACHINE
the value isn't a target MACHINE, but a name that is present in
conf/machine-sdk:

https://git.openembedded.org/openembedded-core/tree/meta/conf/machine-sdk

If you want to build a SDK to run on an arm host, you most likely want
aarch64. A 32-bit Arm SDK hasn't been tested, but writing a new
configuration file for that shouldn't be that difficult.

Ross

On Mon, 1 Nov 2021 at 06:27, jchludzinski via lists.yoctoproject.org
<jchludzinski=vivaldi.net@...> wrote:

NEWBIE question!

I tried building and I get: "Specified SDKMACHINE value is not valid"

pi@raspberrypi ~/p/build> bitbake core-image-minimal
/usr/lib/python3/dist-packages/html5lib/_trie/_base.py:3: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated, and in 3.8 it will stop working
from collections import Mapping
WARNING: Host distribution "raspbian-10" has not been validated with this version of the build system; you may possibly experience unexpected failures. It is recommended that you use a tested distribution.
ERROR: OE-core's config sanity checker detected a potential misconfiguration.
Either fix the cause of this error or at your own risk disable the checker (see sanity.conf).
Following is the list of potential problems / advisories:

Specified SDKMACHINE value is not valid


Summary: There was 1 WARNING message shown.
Summary: There was 1 ERROR message shown, returning a non-zero exit code.

I've tried setting SDKMACHINE to 'arm' and 'qemuarm' but get the same message?

Obviously I failed to do something (properly)?

Ideas?



Specified SDKMACHINE value is not valid

jchludzinski
 

NEWBIE question!
 
I tried building and I get: "Specified SDKMACHINE value is not valid"

pi@raspberrypi ~/p/build> bitbake core-image-minimal
/usr/lib/python3/dist-packages/html5lib/_trie/_base.py:3: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated, and in 3.8 it will stop working
  from collections import Mapping
WARNING: Host distribution "raspbian-10" has not been validated with this version of the build system; you may possibly experience unexpected failures. It is recommended that you use a tested distribution.
ERROR:  OE-core's config sanity checker detected a potential misconfiguration.
    Either fix the cause of this error or at your own risk disable the checker (see sanity.conf).
    Following is the list of potential problems / advisories:
 
    Specified SDKMACHINE value is not valid
 
 
Summary: There was 1 WARNING message shown.
Summary: There was 1 ERROR message shown, returning a non-zero exit code.

I've tried setting SDKMACHINE to 'arm' and 'qemuarm' but get the same message?

Obviously I failed to do something (properly)?

Ideas?


[yocto-autobuilder-helper] [PATCH 4/4] shared-repos: Use tar instead of rsync for speed

Richard Purdie
 

The rysnc of 20,000 files (650MB) onto the nas is slow taking ~3 minutes
at idle and worse at load. This is due to the number of files which
is a pain point for NFS. This piece of the build is also a bottleneck
since the rest of a build depends on it happening.

If we switch to zstd compressed tar, it takes 2.49s. Other compression
methods were much slower but zstd seems 'accptable' and speeds things
up too.

Signed-off-by: Richard Purdie <richard.purdie@...>
---
scripts/prepare-shared-repos | 4 ++--
scripts/send-qa-email | 6 ++++--
scripts/shared-repo-unpack | 5 ++---
3 files changed, 8 insertions(+), 7 deletions(-)

diff --git a/scripts/prepare-shared-repos b/scripts/prepare-shared-repos
index f34ba99..d60ad32 100755
--- a/scripts/prepare-shared-repos
+++ b/scripts/prepare-shared-repos
@@ -39,5 +39,5 @@ with tempfile.TemporaryDirectory(prefix="shared-repo-temp-", dir="/home/pokybuil
if args.publish_dir:
utils.publishrepo(tempdir, repo, args.publish_dir)

- utils.printheader("Running rsync")
- subprocess.check_call("rsync -a " + tempdir + "/* " + args.sharedsrcdir, shell=True)
+ utils.printheader("Creating shared src tarball")
+ subprocess.check_call("tar -I zstd -cf " + args.sharedsrcdir.rstrip("/") + ".tar.zst ./*", shell=True, cwd=tempdir)
diff --git a/scripts/send-qa-email b/scripts/send-qa-email
index 1b69307..fc34f7e 100755
--- a/scripts/send-qa-email
+++ b/scripts/send-qa-email
@@ -45,9 +45,11 @@ buildtoolsdir = os.path.dirname(args.repojson) + "/build/buildtools"
if os.path.exists(buildtoolsdir):
utils.enable_buildtools_tarball(buildtoolsdir)

+repodir = os.path.dirname(args.repojson) + "/repos"
+
if 'poky' in repos and os.path.exists(resulttool) and args.results_dir:
# Need the finalised revisions (not 'HEAD')
- targetrepodir = "%s/poky" % (args.sharedrepodir)
+ targetrepodir = "%s/poky" % (repodir)
revision = subprocess.check_output(["git", "rev-parse", "HEAD"], cwd=targetrepodir).decode('utf-8').strip()
branch = repos['poky']['branch']
repo = repos['poky']['url']
@@ -116,7 +118,7 @@ if args.send.lower() != 'true' or not args.publish_dir or not args.release:
buildhashes = ""
for repo in sorted(repos.keys()):
# Need the finalised revisions (not 'HEAD')
- targetrepodir = "%s/%s" % (args.sharedrepodir, repo)
+ targetrepodir = "%s/%s" % (repodir, repo)
revision = subprocess.check_output(["git", "rev-parse", "HEAD"], cwd=targetrepodir).decode('utf-8').strip()
buildhashes += "%s: %s\n" % (repo, revision)

diff --git a/scripts/shared-repo-unpack b/scripts/shared-repo-unpack
index f08efa8..3dda5b3 100755
--- a/scripts/shared-repo-unpack
+++ b/scripts/shared-repo-unpack
@@ -50,11 +50,10 @@ needrepos_baseddirs = [r.split('/')[0] for r in needrepos]
for repo in sorted(repos.keys()):
if repo not in needrepos_baseddirs:
continue
- targetrepodir = "%s/%s" % (targetsubdir, repo)
if args.cache_dir:
utils.printheader("Copying in repo %s" % repo)
- utils.mkdir(targetrepodir)
- subprocess.check_call(["rsync", "-a", "%s/%s" % (args.cache_dir, repo), targetsubdir])
+ utils.mkdir(targetsubdir)
+ subprocess.check_call(["tar", "-I", "zstd", "-C", targetsubdir, "-xf", "%s.tar.zst" % args.cache_dir, "./" + repo])
else:
utils.printheader("Fetching repo %s" % repo)
utils.fetchgitrepo(targetsubdir, repo, repos[repo], stashdir)
--
2.32.0


[yocto-autobuilder-helper] [PATCH 3/4] prepare-shared-repo/utils: Limit HEAD clones to shallow depth to save time/space

Richard Purdie
 

By not syncing all the history is is possible to save some time/space
in the checkout process since we never use this data. This reduces data
from 650MB to 400MB or with the tarball, 416MB to 55MB.

The logic for the commands needs to be tweaked to handle this and as
written it can't work in non-HEAD revision case but that isn't a commonly
used situation.

Signed-off-by: Richard Purdie <richard.purdie@...>
---
scripts/prepare-shared-repos | 2 +-
scripts/utils.py | 18 +++++++++++++-----
2 files changed, 14 insertions(+), 6 deletions(-)

diff --git a/scripts/prepare-shared-repos b/scripts/prepare-shared-repos
index c221e69..f34ba99 100755
--- a/scripts/prepare-shared-repos
+++ b/scripts/prepare-shared-repos
@@ -35,7 +35,7 @@ stashdir = utils.getconfig("REPO_STASH_DIR", ourconfig)
with tempfile.TemporaryDirectory(prefix="shared-repo-temp-", dir="/home/pokybuild/tmp") as tempdir:
for repo in sorted(repos.keys()):
utils.printheader("Intially fetching repo %s" % repo)
- utils.fetchgitrepo(tempdir, repo, repos[repo], stashdir)
+ utils.fetchgitrepo(tempdir, repo, repos[repo], stashdir, depth=1)
if args.publish_dir:
utils.publishrepo(tempdir, repo, args.publish_dir)

diff --git a/scripts/utils.py b/scripts/utils.py
index 4c73f81..3c2622f 100644
--- a/scripts/utils.py
+++ b/scripts/utils.py
@@ -228,26 +228,34 @@ def runcmd(cmd):
return subprocess.check_output(cmd, stderr=subprocess.STDOUT)


-def fetchgitrepo(clonedir, repo, params, stashdir):
+def fetchgitrepo(clonedir, repo, params, stashdir, depth=None):
sharedrepo = "%s/%s" % (clonedir, repo)
branch = params["branch"]
revision = params["revision"]
+ if revision != "HEAD":
+ depth = None
+ fetchopt = []
+ depthopt = []
+ if depth:
+ fetchopt = ["--depth", str(depth), branch + ":origin/" + branch]
+ depthopt = ["--depth", str(depth), "--branch", branch]
print("Checking for stash at: " + stashdir + "/" + repo)
flush()
if os.path.exists(stashdir + "/" + repo):
print("Cloning from stash to %s..." % sharedrepo)
flush()
- subprocess.check_call(["git", "clone", "file://%s/%s" % (stashdir, repo), "%s/%s" % (clonedir, repo)])
+ subprocess.check_call(["git", "clone", "file://%s/%s" % (stashdir, repo), "%s/%s" % (clonedir, repo)] + depthopt)
subprocess.check_call(["git", "remote", "rm", "origin"], cwd=sharedrepo)
subprocess.check_call(["git", "remote", "add", "origin", params["url"]], cwd=sharedrepo)
print("Updating from origin...")
flush()
- subprocess.check_call(["git", "fetch", "origin"], cwd=sharedrepo)
- subprocess.check_call(["git", "fetch", "origin", "-t"], cwd=sharedrepo)
+ subprocess.check_call(["git", "fetch", "origin"] + fetchopt, cwd=sharedrepo)
+ if not depth:
+ subprocess.check_call(["git", "fetch", "origin", "-t"], cwd=sharedrepo)
else:
print("Cloning from origin to %s..." % sharedrepo)
flush()
- subprocess.check_call(["git", "clone", params["url"], sharedrepo])
+ subprocess.check_call(["git", "clone", params["url"], sharedrepo] + depthopt)

print("Updating checkout...")
flush()
--
2.32.0

2621 - 2640 of 57800