Dunfell, nodejs and typescript - short experience report


TRO
 

Hi Simon,
thank you - in my current solution I don't use npm bbclass at all.
I basically use npmsw://${THISDIR}/${BPN}/npm-shrinkwrap.json;dev=True
this will downloadall npm stuff including angular because of dev=True to $S/node_modules

do_compile () {
#    build frontend
     chmod -R a+w ${S}/node_modules/@angular
     chmod 755 ${S}/node_modules/@angular/cli/bin/ng
     cd ${S}/ && ./node_modules/@angular/cli/bin/ng build --prod
}

I'm dealing with that problem:
https://lists.yoctoproject.org/g/yocto/message/52515


Simon Vogl
 

Sure, please find the bbclass file attached to this mail. All it does is to inherit npm and overwrite configure with an extended copy .

Meanwhile, I saw another issue popping up: It seems that multiple indirect dependencies to a package with different versions create inconsitent license checksum entries. In my case, several packages depend on 'xtend' in versions 4.0.0 to 4.0.2, the license file picked is package.json which contains the version, of course, and causes conflicts. Oh my.

Simon



Am 24.02.21 um 12:02 schrieb TRO:

Hi Simon,
I'm dealing actually with the same problem. Would you like to share your  "configure in my own subclass"?

I'm also thinking there is a need for a bbclass which actually is not using gyp, instead it should be able to "npm run build".

There is alsa a patch for speeding up npm npmsw fetcher https://www.mail-archive.com/openembedded-core@.../msg142406.html
cheers Thomas




-- 
VoXel Interaction Design  |  www.voxel.at
DI Dr.techn. Simon Vogl   |  simon@...
Tomaschekweg 46           |  +43 650 2323 555
A-4040 Linz - Austria     |
Office address: Industriezeile 35, 4020 Linz (2nd floor)


TRO
 

Hi Simon,
I'm dealing actually with the same problem. Would you like to share your  "configure in my own subclass"?

I'm also thinking there is a need for a bbclass which actually is not using gyp, instead it should be able to "npm run build".

There is alsa a patch for speeding up npm npmsw fetcher https://www.mail-archive.com/openembedded-core@.../msg142406.html
cheers Thomas


Simon Vogl
 

Hi,


Am 12.02.21 um 11:12 schrieb Josef Holzmayr:
Howdy!

Thanks for sharing, a few comments inline.

Am Fr., 12. Feb. 2021 um 11:03 Uhr schrieb Simon Vogl <simon@...>:
I have some remarks and questions about the npm/nodejs support in dunfell that I wanted to share. We are creating nodejs-based IoT edge solutions and upgrading our build environments to Dunfell one by one. In the course of this, we are switching to the new npm-implementation and found a few small issues.

Firstly, the do_configure() task takes quite some time to complete. After a quick analysis, I saw that most of the time is being spent in creating the npmrc files while packing the dependent packages. I wrote a small workaround to directly create the file instead of calling 'npm config', which results in a 3x-4x speedup:

Signed-off-by: Simon Vogl <simon@...>
---
 lib/bb/fetch2/npm.py | 9 +++++++--
 1 file changed, 7 insertions(+), 2 deletions(-)

diff --git a/lib/bb/fetch2/npm.py b/lib/bb/fetch2/npm.py
index 4789850..2720d87 100644
--- a/lib/bb/fetch2/npm.py
+++ b/lib/bb/fetch2/npm.py
@@ -97,13 +97,18 @@ class NpmEnvironment(object):
                 cmd = "NPM_CONFIG_GLOBALCONFIG=%s " % cfgfile + cmd
                 return runfetchcmd(cmd, d, workdir=workdir)

+            cfg = open(cfgfile, "a")
             if self.configs:
                 for key, value in self.configs:
-                    _run("npm config set %s %s" % (key, shlex.quote(value)))
+                    cfg.write("%s=%s\n" % (key, shlex.quote(value)))
+                    #_run("npm config set %s %s" % (key, shlex.quote(value)))

             if configs:
                 for key, value in configs:
-                    _run("npm config set %s %s" % (key, shlex.quote(value)))
+                    cfg.write("%s=%s\n" % (key, shlex.quote(value)))
+                    # _run("npm config set %s %s" % (key, shlex.quote(value)))
+
+            cfg.close()

             if args:
                 for key, value in args:
--
2.7.4

Are there any side effects that I did not stumble over yet? And I'd LOVE to have these calls running in a thread-pool for better performance...
The main side effect is that you're effectively patching poky, which
is bad for maintenance.
I know, that's why I'm asking in the first place. But performance here is really really improvable.

      
Secondly, our projects are based on typescript, so a native compile step is necessary to create a compiled version for packing. We experimented with a separate release branch to check in compiled versions, but this is not easy to handle. I played around with npm.bbclass and found a way to extend configure (!) with a call to our build script before packaging:

diff --git a/meta/classes/npm.bbclass b/meta/classes/npm.bbclass
index 068032a1e5..31535098cf 100644
--- a/meta/classes/npm.bbclass
+++ b/meta/classes/npm.bbclass
@@ -170,6 +170,11 @@ python npm_do_configure() {

     # Configure the main package
     with tempfile.TemporaryDirectory() as tmpdir:
+        # install all (native) build dependencies, overrides npm cache:
+        ret = os.system("npm i")
+        # run build step:
+        env.run("npm run build", args=[], workdir=d.getVar("S"))
+
         tarball = npm_pack(env, d.getVar("S"), tmpdir)
         npm_unpack(tarball, d.getVar("NPM_PACKAGE"), d)

As we have plain JS packages as well, I put the modified configure() in a subclass and this works for us, but it does not look like a clean solution to me. How do you other IoT'ers address this situation?
Again, patching poky is a bad idea. Creating custom bbclasses is much
neater: you could create a base include, and pull that together with
npm.bbclass into two final bbclasses of yours, like npm-js-voxel and
npm-ts-voxel. The former would not have the compilation step. And,
putting the typescript/webpack invocation into configure is also not
exactly how things are meant to work. I know that the dependency
tracking of npm is not easily compatible with bitbake, but the aim
should be to
1) have a recipe that provides typescript-native
2) DEPEND on typescript-native in the recipe which you need to compile
3) add a do_compile stage that does the work.

Agreed, and I have a patched configure in my own subclass without changing the official codebase -- I just wanted to point where the modification needs to take place.

I actually tried the approach that you propose by playing around with configure_append / compile_prepend tasks, but these build steps are called after the package has already been packed --> the compiled data is not being  installed, I'd have to re-pack things.

Agreed, a typescript-native package would be nice, on the other hand this is where the npm-version-chaos comes in again: Many packages use different tsc versions,...

Simon


Greetz



-- 
VoXel Interaction Design  |  www.voxel.at
DI Dr.techn. Simon Vogl   |  simon@...
Tomaschekweg 46           |  +43 650 2323 555
A-4040 Linz - Austria     |
Office address: Industriezeile 35, 4020 Linz (2nd floor)


Josef Holzmayr
 

Howdy!

Thanks for sharing, a few comments inline.

Am Fr., 12. Feb. 2021 um 11:03 Uhr schrieb Simon Vogl <simon@voxel.at>:

I have some remarks and questions about the npm/nodejs support in dunfell that I wanted to share. We are creating nodejs-based IoT edge solutions and upgrading our build environments to Dunfell one by one. In the course of this, we are switching to the new npm-implementation and found a few small issues.

Firstly, the do_configure() task takes quite some time to complete. After a quick analysis, I saw that most of the time is being spent in creating the npmrc files while packing the dependent packages. I wrote a small workaround to directly create the file instead of calling 'npm config', which results in a 3x-4x speedup:

Signed-off-by: Simon Vogl <simon@voxel.at>
---
lib/bb/fetch2/npm.py | 9 +++++++--
1 file changed, 7 insertions(+), 2 deletions(-)

diff --git a/lib/bb/fetch2/npm.py b/lib/bb/fetch2/npm.py
index 4789850..2720d87 100644
--- a/lib/bb/fetch2/npm.py
+++ b/lib/bb/fetch2/npm.py
@@ -97,13 +97,18 @@ class NpmEnvironment(object):
cmd = "NPM_CONFIG_GLOBALCONFIG=%s " % cfgfile + cmd
return runfetchcmd(cmd, d, workdir=workdir)

+ cfg = open(cfgfile, "a")
if self.configs:
for key, value in self.configs:
- _run("npm config set %s %s" % (key, shlex.quote(value)))
+ cfg.write("%s=%s\n" % (key, shlex.quote(value)))
+ #_run("npm config set %s %s" % (key, shlex.quote(value)))

if configs:
for key, value in configs:
- _run("npm config set %s %s" % (key, shlex.quote(value)))
+ cfg.write("%s=%s\n" % (key, shlex.quote(value)))
+ # _run("npm config set %s %s" % (key, shlex.quote(value)))
+
+ cfg.close()

if args:
for key, value in args:
--
2.7.4

Are there any side effects that I did not stumble over yet? And I'd LOVE to have these calls running in a thread-pool for better performance...
The main side effect is that you're effectively patching poky, which
is bad for maintenance.

Secondly, our projects are based on typescript, so a native compile step is necessary to create a compiled version for packing. We experimented with a separate release branch to check in compiled versions, but this is not easy to handle. I played around with npm.bbclass and found a way to extend configure (!) with a call to our build script before packaging:

diff --git a/meta/classes/npm.bbclass b/meta/classes/npm.bbclass
index 068032a1e5..31535098cf 100644
--- a/meta/classes/npm.bbclass
+++ b/meta/classes/npm.bbclass
@@ -170,6 +170,11 @@ python npm_do_configure() {

# Configure the main package
with tempfile.TemporaryDirectory() as tmpdir:
+ # install all (native) build dependencies, overrides npm cache:
+ ret = os.system("npm i")
+ # run build step:
+ env.run("npm run build", args=[], workdir=d.getVar("S"))
+
tarball = npm_pack(env, d.getVar("S"), tmpdir)
npm_unpack(tarball, d.getVar("NPM_PACKAGE"), d)

As we have plain JS packages as well, I put the modified configure() in a subclass and this works for us, but it does not look like a clean solution to me. How do you other IoT'ers address this situation?
Again, patching poky is a bad idea. Creating custom bbclasses is much
neater: you could create a base include, and pull that together with
npm.bbclass into two final bbclasses of yours, like npm-js-voxel and
npm-ts-voxel. The former would not have the compilation step. And,
putting the typescript/webpack invocation into configure is also not
exactly how things are meant to work. I know that the dependency
tracking of npm is not easily compatible with bitbake, but the aim
should be to
1) have a recipe that provides typescript-native
2) DEPEND on typescript-native in the recipe which you need to compile
3) add a do_compile stage that does the work.

Greetz


Simon Vogl
 

I have some remarks and questions about the npm/nodejs support in dunfell that I wanted to share. We are creating nodejs-based IoT edge solutions and upgrading our build environments to Dunfell one by one. In the course of this, we are switching to the new npm-implementation and found a few small issues.

Firstly, the do_configure() task takes quite some time to complete. After a quick analysis, I saw that most of the time is being spent in creating the npmrc files while packing the dependent packages. I wrote a small workaround to directly create the file instead of calling 'npm config', which results in a 3x-4x speedup:

Signed-off-by: Simon Vogl <simon@...>
---
 lib/bb/fetch2/npm.py | 9 +++++++--
 1 file changed, 7 insertions(+), 2 deletions(-)

diff --git a/lib/bb/fetch2/npm.py b/lib/bb/fetch2/npm.py
index 4789850..2720d87 100644
--- a/lib/bb/fetch2/npm.py
+++ b/lib/bb/fetch2/npm.py
@@ -97,13 +97,18 @@ class NpmEnvironment(object):
                 cmd = "NPM_CONFIG_GLOBALCONFIG=%s " % cfgfile + cmd
                 return runfetchcmd(cmd, d, workdir=workdir)
 
+            cfg = open(cfgfile, "a")
             if self.configs:
                 for key, value in self.configs:
-                    _run("npm config set %s %s" % (key, shlex.quote(value)))
+                    cfg.write("%s=%s\n" % (key, shlex.quote(value)))
+                    #_run("npm config set %s %s" % (key, shlex.quote(value)))
 
             if configs:
                 for key, value in configs:
-                    _run("npm config set %s %s" % (key, shlex.quote(value)))
+                    cfg.write("%s=%s\n" % (key, shlex.quote(value)))
+                    # _run("npm config set %s %s" % (key, shlex.quote(value)))
+
+            cfg.close()
 
             if args:
                 for key, value in args:
-- 2.7.4

Are there any side effects that I did not stumble over yet? And I'd LOVE to have these calls running in a thread-pool for better performance...


Secondly, our projects are based on typescript, so a native compile step is necessary to create a compiled version for packing. We experimented with a separate release branch to check in compiled versions, but this is not easy to handle. I played around with npm.bbclass and found a way to extend configure (!) with a call to our build script before packaging: 

diff --git a/meta/classes/npm.bbclass b/meta/classes/npm.bbclass
index 068032a1e5..31535098cf 100644
--- a/meta/classes/npm.bbclass
+++ b/meta/classes/npm.bbclass
@@ -170,6 +170,11 @@ python npm_do_configure() {
 
     # Configure the main package
     with tempfile.TemporaryDirectory() as tmpdir:
+        # install all (native) build dependencies, overrides npm cache:
+        ret = os.system("npm i")
+        # run build step:
+        env.run("npm run build", args=[], workdir=d.getVar("S"))
+
         tarball = npm_pack(env, d.getVar("S"), tmpdir)
         npm_unpack(tarball, d.getVar("NPM_PACKAGE"), d)

As we have plain JS packages as well, I put the modified configure() in a subclass and this works for us, but it does not look like a clean solution to me. How do you other IoT'ers address this situation?

Simon


-- 
VoXel Interaction Design  |  www.voxel.at
DI Dr.techn. Simon Vogl   |  simon@...
Tomaschekweg 46           |  +43 650 2323 555
A-4040 Linz - Austria     |  
Office address: Industriezeile 35, 4020 Linz (2nd floor)