Re: [npm] duplicate code


Jean-Marie Lemetayer
 

Hi Stefan,

I thought about your idea of using Yocto to manage NPM package dependencies and I ran into an issue. NPM projects can have multiple dependencies on a single package, and sometimes with multiple versions. NPM will manage this by creating sub 'node_modules' tree.

Here is an example with a newly created angular application. The app depends on 3 different version of 'ansi-regex'. NPM will install the packages this way:
node_modules/ansi-regex @ 2.1.1
node_modules/cliui/node_modules/ansi-regex @ 3.0.0
node_modules/inquirer/node_modules/ansi-regex @ 4.1.0
node_modules/string-width/node_modules/ansi-regex @ 3.0.0
node_modules/@angular/compiler-cli/node_modules/ansi-regex @ 4.1.0

How can you handle that with Yocto ? I am not sure but I think it is not possible.

For that reason I think I will base my developments on the master branch.

----- Mail original -----
De: "Stefan Herbrechtsmeier" <stefan@...>
À: "Jean-marie Lemetayer" <jean-marie.lemetayer@...>
Cc: "Yocto-mailing-list" <yocto@...>, "Savoir-faire Linux Rennes" <rennes@...>
Envoyé: Vendredi 4 Octobre 2019 16:55:47
Objet: Re: [yocto] [npm] duplicate code
Hi Jean-Marie,

Am 04.10.19 um 14:37 schrieb Jean-marie Lemetayer:
I have recently worked on a yocto project using npm and I have seen some issues.
I have solved a few but only for bitbake:
https://git.yoctoproject.org/cgit/cgit.cgi/poky/commit/?id=eecba41822e86b69ebdb9cbc8fbfd512ad9a47d7
https://git.yoctoproject.org/cgit/cgit.cgi/poky/commit/?id=a34d0d539e5fdf341541fb628652d22289e80512
https://git.yoctoproject.org/cgit/cgit.cgi/poky/commit/?id=e90cd2ed61b93ee7e290e7e592f1f0242ab5c281
https://git.yoctoproject.org/cgit/cgit.cgi/poky/commit/?id=a35abe31dc23916fd06afdb3a5e22a81ef3df579

As I have more time now, I wanted to continue my work by fixing devtool /
recipetool.

I have also checked the bugzilla for issues that I could fix / that should be
tested again:
https://bugzilla.yoctoproject.org/show_bug.cgi?id=10515
https://bugzilla.yoctoproject.org/show_bug.cgi?id=10760
https://bugzilla.yoctoproject.org/show_bug.cgi?id=11028
https://bugzilla.yoctoproject.org/show_bug.cgi?id=11029
https://bugzilla.yoctoproject.org/show_bug.cgi?id=12534
https://bugzilla.yoctoproject.org/show_bug.cgi?id=13415

I ended up with this todo list:
  - merge the duplicate code between bitbake and recipetool
  - fix the npm package name handling in recipetool
  - fix the npm package version handling in recipetool
  - fix the lockdown.json file generation in recipetool
  - create an example nodejs application to test all these cases
  - update the wiki using this example application:
    https://wiki.yoctoproject.org/wiki/TipsAndTricks/NPM

Finally, in my recent project, we were using angular (https://angular.io) for
the front-end. I have planned to add the work done to support angular
applications in yocto (angular.bbclass) and update recipetool to handle them.
Do you want to build an angular application? In this case you have to
produce a machine (independent) package via a native npm build because
you need to run the angular compiler on the build machine.
Yes, I have an 'angular-cli_x.y.z.bb' and an 'angular.bbclass' which depends on 'angular-cli-native'. I have planned to upstream this work.



Your work sounds very interesting. The good point is that npm-shrinkwrap.json
and lockdown.json files (which have generation issues btw) will no longer be
required. But projects using npm can have a lot of dependencies (e.g. the
angular example app have 1053 dependencies).
A lot of dependencies are the problem at the moment. But many
dependencies are not update frequently and my current solution assume
that I can always use the latest version per major release version

Is recipetool will be handling the whole recipes creation in one time ?
That's the problem at the moment. I don't manage to build multiple
recipes from one recipetool run. Therefore I have a two stage approach.
I use recipetool to build exact one recipe. A second script generates a
list of all dependencies and call recipetool per package if the recipe
doesn't already exist. This works but the run time is very high.

Is it possible to see your work ? A public fork would be nice. I would gladly
help you / test your work / add my fixes if needed.
I only have a unfinished Proof of Concept. If you like I could clean it
up a bit and post it on github as a separate layer.
Like I said at the start of this email, I will not base my dev on yours. But I am still interested by seeing your work.

Best regards,
Jean-Marie

Join yocto@lists.yoctoproject.org to automatically receive all group messages.