[PATCH yocto-autobuilder-helper] config.json: set ZSTD_THREADS like XZ_THREADS


Ross Burton <ross@...>
 

Signed-off-by: Ross Burton <ross.burton@...>
---
config.json | 1 +
1 file changed, 1 insertion(+)

diff --git a/config.json b/config.json
index 7203783..c0543d9 100644
--- a/config.json
+++ b/config.json
@@ -47,6 +47,7 @@
"PARALLEL_MAKE =3D '-j 16 -l 52'",
"XZ_MEMLIMIT =3D '5%'",
"XZ_THREADS =3D '8'",
+ "ZSTD_THREADS =3D '8'",
"BB_TASK_NICE_LEVEL =3D '5'",
"BB_TASK_NICE_LEVEL:task-testimage =3D '0'",
"BB_TASK_IONICE_LEVEL =3D '2.7'",
--=20
2.25.1


Alexander Kanavin
 

Is there a particular reason?

Alex


On Tue, 23 Nov 2021 at 16:03, Ross Burton <ross@...> wrote:
Signed-off-by: Ross Burton <ross.burton@...>
---
 config.json | 1 +
 1 file changed, 1 insertion(+)

diff --git a/config.json b/config.json
index 7203783..c0543d9 100644
--- a/config.json
+++ b/config.json
@@ -47,6 +47,7 @@
             "PARALLEL_MAKE = '-j 16 -l 52'",
             "XZ_MEMLIMIT = '5%'",
             "XZ_THREADS = '8'",
+            "ZSTD_THREADS = '8'",
             "BB_TASK_NICE_LEVEL = '5'",
             "BB_TASK_NICE_LEVEL:task-testimage = '0'",
             "BB_TASK_IONICE_LEVEL = '2.7'",
--
2.25.1





Ross Burton <ross@...>
 

My personal build machine has been crashing and dying due to overload
with the default ZSTD_THREADS, and the AB has had problems with xz due
to overload so I was being proactive and ensuring zstd is limited on
the AB the same way xz is.

Ross

On Tue, 23 Nov 2021 at 17:11, Alexander Kanavin <alex.kanavin@...> wrote:

Is there a particular reason?

Alex

On Tue, 23 Nov 2021 at 16:03, Ross Burton <ross@...> wrote:

Signed-off-by: Ross Burton <ross.burton@...>
---
config.json | 1 +
1 file changed, 1 insertion(+)

diff --git a/config.json b/config.json
index 7203783..c0543d9 100644
--- a/config.json
+++ b/config.json
@@ -47,6 +47,7 @@
"PARALLEL_MAKE = '-j 16 -l 52'",
"XZ_MEMLIMIT = '5%'",
"XZ_THREADS = '8'",
+ "ZSTD_THREADS = '8'",
"BB_TASK_NICE_LEVEL = '5'",
"BB_TASK_NICE_LEVEL:task-testimage = '0'",
"BB_TASK_IONICE_LEVEL = '2.7'",
--
2.25.1




Ernst Sjöstrand <ernst.sjostrand@...>
 

There was a long thread about this on oe-core recently:
https://lists.openembedded.org/g/openembedded-core/message/157999

//Ernst


Alexander Kanavin
 

But the AB has not been exhibiting any problems with zstd, and this will degrade performance. Let's only fix what is broken.

Alex


On Tue, 23 Nov 2021 at 20:53, Ross Burton <ross@...> wrote:
My personal build machine has been crashing and dying due to overload
with the default ZSTD_THREADS, and the AB has had problems with xz due
to overload so I was being proactive and ensuring zstd is limited on
the AB the same way xz is.

Ross

On Tue, 23 Nov 2021 at 17:11, Alexander Kanavin <alex.kanavin@...> wrote:
>
> Is there a particular reason?
>
> Alex
>
> On Tue, 23 Nov 2021 at 16:03, Ross Burton <ross@...> wrote:
>>
>> Signed-off-by: Ross Burton <ross.burton@...>
>> ---
>>  config.json | 1 +
>>  1 file changed, 1 insertion(+)
>>
>> diff --git a/config.json b/config.json
>> index 7203783..c0543d9 100644
>> --- a/config.json
>> +++ b/config.json
>> @@ -47,6 +47,7 @@
>>              "PARALLEL_MAKE = '-j 16 -l 52'",
>>              "XZ_MEMLIMIT = '5%'",
>>              "XZ_THREADS = '8'",
>> +            "ZSTD_THREADS = '8'",
>>              "BB_TASK_NICE_LEVEL = '5'",
>>              "BB_TASK_NICE_LEVEL:task-testimage = '0'",
>>              "BB_TASK_IONICE_LEVEL = '2.7'",
>> --
>> 2.25.1
>>
>>
>>
>>


Richard Purdie
 

On Wed, 2021-11-24 at 09:00 +0100, Alexander Kanavin wrote:
But the AB has not been exhibiting any problems with zstd, and this will
degrade performance. Let's only fix what is broken.
I'm not sure I agree with that. 

We have 60+ "intermittent" bugs and some of us are in weekly meetings trying to
do something about working out why these are failing. It feels like we're not
really getting too far with some subset of them and it is using up a lot of the
SWAT and bug triage time.

We've made a few changes to try and reduce the load spikes on the systems and
this fits with the other changes we've made.

Cheers,

Richard


Khem Raj
 

On Wed, Nov 24, 2021 at 2:18 AM Richard Purdie
<richard.purdie@...> wrote:

On Wed, 2021-11-24 at 09:00 +0100, Alexander Kanavin wrote:
But the AB has not been exhibiting any problems with zstd, and this will
degrade performance. Let's only fix what is broken.
I'm not sure I agree with that.

We have 60+ "intermittent" bugs and some of us are in weekly meetings trying to
do something about working out why these are failing. It feels like we're not
really getting too far with some subset of them and it is using up a lot of the
SWAT and bug triage time.

We've made a few changes to try and reduce the load spikes on the systems and
this fits with the other changes we've made.
From a different data point, we have clipped the parallelism for XZ
and ZSTD internally to very low ( 2 and 4 )
and it has in fact reduced unexpected failures and seen no impact on
build performance. Since parallelism settings
are myopic for these tools, they greedily take every CPU which is not
best for overall build thats doing many
things in parallel, so its best to curtail them to a conservative value.

Cheers,

Richard