[yocto-autobuilder-helper][PATCH] config.json: track system load with PARALLEL_MAKE


Trevor Gamblin
 

This adds the "-l" option to PARALLEL_MAKE in config.json with an
initial testing value of 100 (100% system load). This option is supported
by both Make and Ninja. However, we also require the "--debug=j" option
to be passed to Make in order for the latter to report perceived system
load in the do_compile logs, and since this option is not supported by
Ninja, also add EXTRA_OEMAKE to the EXTRAVARS so that we can determine if
the target load percentage needs to be adjusted.

Signed-off-by: Trevor Gamblin <trevor.gamblin@...>
---
config.json | 5 +++--
1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/config.json b/config.json
index f54081b..7fc89ea 100644
--- a/config.json
+++ b/config.json
@@ -44,7 +44,7 @@
"PREMIRRORS = ''",
"BB_GENERATE_MIRROR_TARBALLS = '1'",
"BB_NUMBER_THREADS = '16'",
- "PARALLEL_MAKE = '-j 16'",
+ "PARALLEL_MAKE = '-j 16 -l 100'",
"XZ_MEMLIMIT = '5%'",
"XZ_THREADS = '8'",
"BB_TASK_NICE_LEVEL = '5'",
@@ -61,7 +61,8 @@
"RUNQEMU_TMPFS_DIR = '/home/pokybuild/tmp'",
"BB_HEARTBEAT_EVENT = '60'",
"BB_LOG_HOST_STAT_ON_INTERVAL = '1'",
- "BB_LOG_HOST_STAT_CMDS_INTERVAL = 'oe-time-dd-test.sh 100'"
+ "BB_LOG_HOST_STAT_CMDS_INTERVAL = 'oe-time-dd-test.sh 100'",
+ "EXTRA_OEMAKE = ' --debug=j'"
]
},
"templates" : {
--
2.31.1


Alexander Kanavin
 

I seem to vaguely remember that -l is not actually taking a percentage, but an absolute value that is specific to the amount of CPU cores a system has. Have you verified your assumption?

Alex


On Mon, 12 Jul 2021 at 14:11, Trevor Gamblin <trevor.gamblin@...> wrote:
This adds the "-l" option to PARALLEL_MAKE in config.json with an
initial testing value of 100 (100% system load). This option is supported
by both Make and Ninja. However, we also require the "--debug=j" option
to be passed to Make in order for the latter to report perceived system
load in the do_compile logs, and since this option is not supported by
Ninja, also add EXTRA_OEMAKE to the EXTRAVARS so that we can determine if
the target load percentage needs to be adjusted.

Signed-off-by: Trevor Gamblin <trevor.gamblin@...>
---
 config.json | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/config.json b/config.json
index f54081b..7fc89ea 100644
--- a/config.json
+++ b/config.json
@@ -44,7 +44,7 @@
             "PREMIRRORS = ''",
             "BB_GENERATE_MIRROR_TARBALLS = '1'",
             "BB_NUMBER_THREADS = '16'",
-            "PARALLEL_MAKE = '-j 16'",
+            "PARALLEL_MAKE = '-j 16 -l 100'",
             "XZ_MEMLIMIT = '5%'",
             "XZ_THREADS = '8'",
             "BB_TASK_NICE_LEVEL = '5'",
@@ -61,7 +61,8 @@
             "RUNQEMU_TMPFS_DIR = '/home/pokybuild/tmp'",
             "BB_HEARTBEAT_EVENT = '60'",
             "BB_LOG_HOST_STAT_ON_INTERVAL = '1'",
-            "BB_LOG_HOST_STAT_CMDS_INTERVAL = 'oe-time-dd-test.sh 100'"
+            "BB_LOG_HOST_STAT_CMDS_INTERVAL = 'oe-time-dd-test.sh 100'",
+            "EXTRA_OEMAKE = ' --debug=j'"
         ]
     },
     "templates" : {
--
2.31.1





Trevor Gamblin
 


On 2021-07-12 12:04 p.m., Alexander Kanavin wrote:

[Please note: This e-mail is from an EXTERNAL e-mail address]

I seem to vaguely remember that -l is not actually taking a percentage, but an absolute value that is specific to the amount of CPU cores a system has. Have you verified your assumption?
You are right that it's not actually a percentage and is instead tied to the number of cores. I'll correct the patch body to reflect this when I inevitably send a v2. With the latest testing on the AB, my suspicion that "-l 100" is too generous for most machines is being confirmed...

Alex

On Mon, 12 Jul 2021 at 14:11, Trevor Gamblin <trevor.gamblin@...> wrote:
This adds the "-l" option to PARALLEL_MAKE in config.json with an
initial testing value of 100 (100% system load). This option is supported
by both Make and Ninja. However, we also require the "--debug=j" option
to be passed to Make in order for the latter to report perceived system
load in the do_compile logs, and since this option is not supported by
Ninja, also add EXTRA_OEMAKE to the EXTRAVARS so that we can determine if
the target load percentage needs to be adjusted.

Signed-off-by: Trevor Gamblin <trevor.gamblin@...>
---
 config.json | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/config.json b/config.json
index f54081b..7fc89ea 100644
--- a/config.json
+++ b/config.json
@@ -44,7 +44,7 @@
             "PREMIRRORS = ''",
             "BB_GENERATE_MIRROR_TARBALLS = '1'",
             "BB_NUMBER_THREADS = '16'",
-            "PARALLEL_MAKE = '-j 16'",
+            "PARALLEL_MAKE = '-j 16 -l 100'",
             "XZ_MEMLIMIT = '5%'",
             "XZ_THREADS = '8'",
             "BB_TASK_NICE_LEVEL = '5'",
@@ -61,7 +61,8 @@
             "RUNQEMU_TMPFS_DIR = '/home/pokybuild/tmp'",
             "BB_HEARTBEAT_EVENT = '60'",
             "BB_LOG_HOST_STAT_ON_INTERVAL = '1'",
-            "BB_LOG_HOST_STAT_CMDS_INTERVAL = 'oe-time-dd-test.sh 100'"
+            "BB_LOG_HOST_STAT_CMDS_INTERVAL = 'oe-time-dd-test.sh 100'",
+            "EXTRA_OEMAKE = ' --debug=j'"
         ]
     },
     "templates" : {
--
2.31.1