|
From: Trevor Gamblin <trevor.gamblin@...> Add "-l 52" to PARALLEL_MAKE in config.json to limit Make and Ninja builds based on the detected system load. With this option added, if either tool has at least one job running and detects that the system load exceeds the given value, it will wait until either the system load average drops below that limit, or until all other jobs are finished before starting additional jobs. Since most autobuilder machines have 56 cores, this should help keep the system from being overloaded during builds. Reference: https://www.gnu.org/software/make/manual/html_node/Parallel.htmlSigned-off-by: Trevor Gamblin <trevor.gamblin@...> Signed-off-by: Richard Purdie <richard.purdie@...> (cherry picked from commit 5c5fc7bcd221427d34bbac80c9bad315fb6de4df) Signed-off-by: Anuj Mittal <anuj.mittal@...> --- config.json | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/config.json b/config.json index 33d36ad..d397365 100644 --- a/config.json +++ b/config.json @@ -43,7 +43,7 @@ "PREMIRRORS = ''", "BB_GENERATE_MIRROR_TARBALLS = '1'", "BB_NUMBER_THREADS = '16'", - "PARALLEL_MAKE = '-j 16'", + "PARALLEL_MAKE = '-j 16 -l 52'", "XZ_MEMLIMIT = '5%'", "XZ_THREADS = '8'", "BB_TASK_NICE_LEVEL = '5'", -- 2.31.1
|
|
|
Khem Raj
On 10/25/21 7:07 AM, Anuj Mittal wrote: From: Trevor Gamblin <trevor.gamblin@...>
Add "-l 52" to PARALLEL_MAKE in config.json to limit Make and Ninja
builds based on the detected system load. With this option added, if
either tool has at least one job running and detects that the system
load exceeds the given value, it will wait until either the system load
average drops below that limit, or until all other jobs are finished
before starting additional jobs.
Since most autobuilder machines have 56 cores, this should help keep the
system from being overloaded during builds.
Reference: https://www.gnu.org/software/make/manual/html_node/Parallel.html
Signed-off-by: Trevor Gamblin <trevor.gamblin@...>
Signed-off-by: Richard Purdie <richard.purdie@...>
(cherry picked from commit 5c5fc7bcd221427d34bbac80c9bad315fb6de4df)
Signed-off-by: Anuj Mittal <anuj.mittal@...>
---
config.json | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/config.json b/config.json
index 33d36ad..d397365 100644
--- a/config.json
+++ b/config.json
@@ -43,7 +43,7 @@
"PREMIRRORS = ''",
"BB_GENERATE_MIRROR_TARBALLS = '1'",
"BB_NUMBER_THREADS = '16'",
- "PARALLEL_MAKE = '-j 16'",
+ "PARALLEL_MAKE = '-j 16 -l 52'", We have seen a lot of hung builds when we used this option internally on builders, Additionally some packages like bison started to fail intermittently which seemed like a parallel build issue but never got to see why it would happen only with -l enabled. So this change would need some testing before we make it permanent into AB "XZ_MEMLIMIT = '5%'",
"XZ_THREADS = '8'",
"BB_TASK_NICE_LEVEL = '5'",
|
|
|
On Mon, Oct 25, 2021 at 6:09 AM Khem Raj <raj.khem@...> wrote:
On 10/25/21 7:07 AM, Anuj Mittal wrote:
From: Trevor Gamblin <trevor.gamblin@...>
Add "-l 52" to PARALLEL_MAKE in config.json to limit Make and Ninja
builds based on the detected system load. With this option added, if
either tool has at least one job running and detects that the system
load exceeds the given value, it will wait until either the system load
average drops below that limit, or until all other jobs are finished
before starting additional jobs.
Since most autobuilder machines have 56 cores, this should help keep the
system from being overloaded during builds.
Reference: https://www.gnu.org/software/make/manual/html_node/Parallel.html
Signed-off-by: Trevor Gamblin <trevor.gamblin@...>
Signed-off-by: Richard Purdie <richard.purdie@...>
(cherry picked from commit 5c5fc7bcd221427d34bbac80c9bad315fb6de4df)
Signed-off-by: Anuj Mittal <anuj.mittal@...>
---
config.json | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/config.json b/config.json
index 33d36ad..d397365 100644
--- a/config.json
+++ b/config.json
@@ -43,7 +43,7 @@
"PREMIRRORS = ''",
"BB_GENERATE_MIRROR_TARBALLS = '1'",
"BB_NUMBER_THREADS = '16'",
- "PARALLEL_MAKE = '-j 16'",
+ "PARALLEL_MAKE = '-j 16 -l 52'", We have seen a lot of hung builds when we used this option internally on
builders, Additionally some packages like bison started to fail
intermittently which seemed like a parallel build issue but never got to
see why it would happen only with -l enabled.
So this change would need some testing before we make it permanent into AB
Which branch did you test this with? I had some issues when I originally added it to dunfell but after the last patch series I am getting green a-full and meta-oe builds on the autobuilder. And master has been running with this for q Steve
"XZ_MEMLIMIT = '5%'",
"XZ_THREADS = '8'",
"BB_TASK_NICE_LEVEL = '5'",
|
|
|
On Mon, Oct 25, 2021 at 6:46 AM Steve Sakoman via lists.yoctoproject.org <steve=sakoman.com@...> wrote:
On Mon, Oct 25, 2021 at 6:09 AM Khem Raj <raj.khem@...> wrote:
On 10/25/21 7:07 AM, Anuj Mittal wrote:
From: Trevor Gamblin <trevor.gamblin@...>
Add "-l 52" to PARALLEL_MAKE in config.json to limit Make and Ninja
builds based on the detected system load. With this option added, if
either tool has at least one job running and detects that the system
load exceeds the given value, it will wait until either the system load
average drops below that limit, or until all other jobs are finished
before starting additional jobs.
Since most autobuilder machines have 56 cores, this should help keep the
system from being overloaded during builds.
Reference: https://www.gnu.org/software/make/manual/html_node/Parallel.html
Signed-off-by: Trevor Gamblin <trevor.gamblin@...>
Signed-off-by: Richard Purdie <richard.purdie@...>
(cherry picked from commit 5c5fc7bcd221427d34bbac80c9bad315fb6de4df)
Signed-off-by: Anuj Mittal <anuj.mittal@...>
---
config.json | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/config.json b/config.json
index 33d36ad..d397365 100644
--- a/config.json
+++ b/config.json
@@ -43,7 +43,7 @@
"PREMIRRORS = ''",
"BB_GENERATE_MIRROR_TARBALLS = '1'",
"BB_NUMBER_THREADS = '16'",
- "PARALLEL_MAKE = '-j 16'",
+ "PARALLEL_MAKE = '-j 16 -l 52'", We have seen a lot of hung builds when we used this option internally on
builders, Additionally some packages like bison started to fail
intermittently which seemed like a parallel build issue but never got to
see why it would happen only with -l enabled.
So this change would need some testing before we make it permanent into AB
Which branch did you test this with? I had some issues when I
originally added it to dunfell but after the last patch series I am
getting green a-full and meta-oe builds on the autobuilder.
And master has been running with this for q
Accidentally hit send :-) I intended to say that master has been running with this for quite some time now. Steve
"XZ_MEMLIMIT = '5%'",
"XZ_THREADS = '8'",
"BB_TASK_NICE_LEVEL = '5'",
|
|
|
Khem Raj
On Mon, Oct 25, 2021 at 9:46 AM Steve Sakoman <steve@...> wrote:
On Mon, Oct 25, 2021 at 6:09 AM Khem Raj <raj.khem@...> wrote:
On 10/25/21 7:07 AM, Anuj Mittal wrote:
From: Trevor Gamblin <trevor.gamblin@...>
Add "-l 52" to PARALLEL_MAKE in config.json to limit Make and Ninja
builds based on the detected system load. With this option added, if
either tool has at least one job running and detects that the system
load exceeds the given value, it will wait until either the system load
average drops below that limit, or until all other jobs are finished
before starting additional jobs.
Since most autobuilder machines have 56 cores, this should help keep the
system from being overloaded during builds.
Reference: https://www.gnu.org/software/make/manual/html_node/Parallel.html
Signed-off-by: Trevor Gamblin <trevor.gamblin@...>
Signed-off-by: Richard Purdie <richard.purdie@...>
(cherry picked from commit 5c5fc7bcd221427d34bbac80c9bad315fb6de4df)
Signed-off-by: Anuj Mittal <anuj.mittal@...>
---
config.json | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/config.json b/config.json
index 33d36ad..d397365 100644
--- a/config.json
+++ b/config.json
@@ -43,7 +43,7 @@
"PREMIRRORS = ''",
"BB_GENERATE_MIRROR_TARBALLS = '1'",
"BB_NUMBER_THREADS = '16'",
- "PARALLEL_MAKE = '-j 16'",
+ "PARALLEL_MAKE = '-j 16 -l 52'", We have seen a lot of hung builds when we used this option internally on
builders, Additionally some packages like bison started to fail
intermittently which seemed like a parallel build issue but never got to
see why it would happen only with -l enabled.
So this change would need some testing before we make it permanent into AB
Which branch did you test this with? I had some issues when I
originally added it to dunfell but after the last patch series I am
getting green a-full and meta-oe builds on the autobuilder.
it was dunfell primarily
And master has been running with this for q
Steve
"XZ_MEMLIMIT = '5%'",
"XZ_THREADS = '8'",
"BB_TASK_NICE_LEVEL = '5'",
|
|
|
On Mon, Oct 25, 2021 at 7:00 AM Khem Raj <raj.khem@...> wrote:
On Mon, Oct 25, 2021 at 9:46 AM Steve Sakoman <steve@...> wrote:
On Mon, Oct 25, 2021 at 6:09 AM Khem Raj <raj.khem@...> wrote:
On 10/25/21 7:07 AM, Anuj Mittal wrote:
From: Trevor Gamblin <trevor.gamblin@...>
Add "-l 52" to PARALLEL_MAKE in config.json to limit Make and Ninja
builds based on the detected system load. With this option added, if
either tool has at least one job running and detects that the system
load exceeds the given value, it will wait until either the system load
average drops below that limit, or until all other jobs are finished
before starting additional jobs.
Since most autobuilder machines have 56 cores, this should help keep the
system from being overloaded during builds.
Reference: https://www.gnu.org/software/make/manual/html_node/Parallel.html
Signed-off-by: Trevor Gamblin <trevor.gamblin@...>
Signed-off-by: Richard Purdie <richard.purdie@...>
(cherry picked from commit 5c5fc7bcd221427d34bbac80c9bad315fb6de4df)
Signed-off-by: Anuj Mittal <anuj.mittal@...>
---
config.json | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/config.json b/config.json
index 33d36ad..d397365 100644
--- a/config.json
+++ b/config.json
@@ -43,7 +43,7 @@
"PREMIRRORS = ''",
"BB_GENERATE_MIRROR_TARBALLS = '1'",
"BB_NUMBER_THREADS = '16'",
- "PARALLEL_MAKE = '-j 16'",
+ "PARALLEL_MAKE = '-j 16 -l 52'", We have seen a lot of hung builds when we used this option internally on
builders, Additionally some packages like bison started to fail
intermittently which seemed like a parallel build issue but never got to
see why it would happen only with -l enabled.
So this change would need some testing before we make it permanent into AB
Which branch did you test this with? I had some issues when I
originally added it to dunfell but after the last patch series I am
getting green a-full and meta-oe builds on the autobuilder.
it was dunfell primarily
In that case you might want to try again with the current dunfell head. It was a couple of weeks of pain, but I think I've eliminated all of the issues with using this patch. Steve
And master has been running with this for q
Steve
"XZ_MEMLIMIT = '5%'",
"XZ_THREADS = '8'",
"BB_TASK_NICE_LEVEL = '5'",
|
|
|
Khem Raj
On Mon, Oct 25, 2021 at 10:30 AM Steve Sakoman <steve@...> wrote:
On Mon, Oct 25, 2021 at 7:00 AM Khem Raj <raj.khem@...> wrote:
On Mon, Oct 25, 2021 at 9:46 AM Steve Sakoman <steve@...> wrote:
On Mon, Oct 25, 2021 at 6:09 AM Khem Raj <raj.khem@...> wrote:
On 10/25/21 7:07 AM, Anuj Mittal wrote:
From: Trevor Gamblin <trevor.gamblin@...>
Add "-l 52" to PARALLEL_MAKE in config.json to limit Make and Ninja
builds based on the detected system load. With this option added, if
either tool has at least one job running and detects that the system
load exceeds the given value, it will wait until either the system load
average drops below that limit, or until all other jobs are finished
before starting additional jobs.
Since most autobuilder machines have 56 cores, this should help keep the
system from being overloaded during builds.
Reference: https://www.gnu.org/software/make/manual/html_node/Parallel.html
Signed-off-by: Trevor Gamblin <trevor.gamblin@...>
Signed-off-by: Richard Purdie <richard.purdie@...>
(cherry picked from commit 5c5fc7bcd221427d34bbac80c9bad315fb6de4df)
Signed-off-by: Anuj Mittal <anuj.mittal@...>
---
config.json | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/config.json b/config.json
index 33d36ad..d397365 100644
--- a/config.json
+++ b/config.json
@@ -43,7 +43,7 @@
"PREMIRRORS = ''",
"BB_GENERATE_MIRROR_TARBALLS = '1'",
"BB_NUMBER_THREADS = '16'",
- "PARALLEL_MAKE = '-j 16'",
+ "PARALLEL_MAKE = '-j 16 -l 52'", We have seen a lot of hung builds when we used this option internally on
builders, Additionally some packages like bison started to fail
intermittently which seemed like a parallel build issue but never got to
see why it would happen only with -l enabled.
So this change would need some testing before we make it permanent into AB
Which branch did you test this with? I had some issues when I
originally added it to dunfell but after the last patch series I am
getting green a-full and meta-oe builds on the autobuilder.
it was dunfell primarily
In that case you might want to try again with the current dunfell
head. It was a couple of weeks of pain, but I think I've eliminated
all of the issues with using this patch.
OK, I think it will be a while before I can get to tip if dunfell though.
Steve
And master has been running with this for q
Steve
"XZ_MEMLIMIT = '5%'",
"XZ_THREADS = '8'",
"BB_TASK_NICE_LEVEL = '5'",
|
|
|
