yocto@lists.yoctoproject.org | Dealing with go dependencies in recipes - native docker-compose

Home Messages Hashtags Subgroups

Date Date 1 - 8 of 8

Dealing with go dependencies in recipes - native docker-compose

Konstantin Kletschke #58364 On Tue, Oct 18, 2022 at 01:39:34PM -0400, Bruce Ashfield wrote: root@qemux86-64-c9:~# /usr/lib/docker/cli-plugins/docker-compose version Docker Compose version v2.11.2 root@insidem2m:~/docker-nginx-demo# docker compose version Docker Compose version v2.11.2 I did a test run with a small docker-compose.yml for a nginx hello world demo. It worked flawlessly, downloading, installing and starting went really fine. Perfect, nice work! Kind Regards Konstantin -- INSIDE M2M GmbH Konstantin Kletschke Berenbosteler Straße 76 B 30823 Garbsen Telefon: +49 (0) 5137 90950136 Mobil: +49 (0) 151 15256238 Fax: +49 (0) 5137 9095010 konstantin.kletschke@... http://www.inside-m2m.de Geschäftsführung: Michael Emmert, Ingo Haase, Dr. Fred Könemann, Derek Uhlig HRB: 111204, AG Hannover
More All Messages By This Member
Bruce Ashfield #58358 On Tue, Oct 18, 2022 at 9:17 AM Bruce Ashfield via lists.yoctoproject.org <bruce.ashfield=gmail.com@...> wrote: On Tue, Oct 18, 2022 at 6:56 AM Konstantin Kletschke <konstantin.kletschke@...> wrote: On Sun, Oct 16, 2022 at 10:28:18PM -0400, Bruce Ashfield wrote: > FYI: Here's the very lightly tested RFC version of the recipe: > > https://git.yoctoproject.org/meta-virtualization/commit/?h=master-next&id=fa24cfc08b129de5c7ba36099985359cc0228630 Thanks for the kind notification. My stuff is based upon kirkstone, I stuffed the recipe from master-next into my kirkstone build. One package of this new docker-compose is depending upon go-1.19 (kirkstone provides go-1.17) so I pulled in go-1.19 from langdale also into my kirkstone build. The compiling and building works just fine! The recipe(s) for docker-compose look impressive, I would not have expected them being so complex. But they work. The amount of fetches is certainly impressive, but it is like that for any non-vendored go application, whether we can see it or not. When the recipes are simple, all of the fetching and source structuring takes place behind the scenes (I know, I'm stating the obvious), just in a manner we don't control. I just yank the complexity out to the front, and go directly to the upstream sources of all the go libraries/packages. If we let go fetch it, we have to both rely on the versions being in the fetch locations "forever", or the infrastructure always being available (it should, but you never know). We then have to setup our own go proxy/mirror infrastructure to get around those two issues, and have now become curators of our own repositories and infrastructure. That infrastructure isn't really hard to setup, but in a maintenance mode, updating it, and getting new versions of the code for CVEs, etc, could be challenging. By doing the fetches directly, all of the established infrastructure, archiving, mirrors, licensing, etc, "just work", and we can bump the SRCREV of any dependency if a CVE or issue is found. Definitely a balancing act, with no 'one true answer' yet, so I've been going with what I know works :) Two notes: After installing the package provides /bin/docker-compose. I realized other parties provide this as /usr/lib/docker/cli-plugins/docker-compose which make something like "docker compose version" working. For me this does not harm, only as remark. May be this is intentional. Completely random. I just put it there. I'll make that adjustment in the recipe. Upon investigating I realized the version reported by docker-compose is "dev" instead of the expected "v2.11.2". I saw the same thing, and was looking into fixing it, I figured it was worth getting the recipe out, before I did more tweaking. emux86-64-c9 login: root root@qemux86-64-c9:~# /usr/lib/docker/cli-plugins/docker-compose version Docker Compose version v2.11.2 Cheers, Bruce Great work, thank you! I appreciate the testing and the report, very helpful! Bruce Regards Konstantin -- - Thou shalt not follow the NULL pointer, for chaos and madness await thee at its end - "Use the force Harry" - Gandalf, Star Trek II -- - Thou shalt not follow the NULL pointer, for chaos and madness await thee at its end - "Use the force Harry" - Gandalf, Star Trek II
More All Messages By This Member
Bruce Ashfield #58355 On Tue, Oct 18, 2022 at 6:56 AM Konstantin Kletschke <konstantin.kletschke@...> wrote: On Sun, Oct 16, 2022 at 10:28:18PM -0400, Bruce Ashfield wrote: > FYI: Here's the very lightly tested RFC version of the recipe: > > https://git.yoctoproject.org/meta-virtualization/commit/?h=master-next&id=fa24cfc08b129de5c7ba36099985359cc0228630 Thanks for the kind notification. My stuff is based upon kirkstone, I stuffed the recipe from master-next into my kirkstone build. One package of this new docker-compose is depending upon go-1.19 (kirkstone provides go-1.17) so I pulled in go-1.19 from langdale also into my kirkstone build. The compiling and building works just fine! The recipe(s) for docker-compose look impressive, I would not have expected them being so complex. But they work. The amount of fetches is certainly impressive, but it is like that for any non-vendored go application, whether we can see it or not. When the recipes are simple, all of the fetching and source structuring takes place behind the scenes (I know, I'm stating the obvious), just in a manner we don't control. I just yank the complexity out to the front, and go directly to the upstream sources of all the go libraries/packages. If we let go fetch it, we have to both rely on the versions being in the fetch locations "forever", or the infrastructure always being available (it should, but you never know). We then have to setup our own go proxy/mirror infrastructure to get around those two issues, and have now become curators of our own repositories and infrastructure. That infrastructure isn't really hard to setup, but in a maintenance mode, updating it, and getting new versions of the code for CVEs, etc, could be challenging. By doing the fetches directly, all of the established infrastructure, archiving, mirrors, licensing, etc, "just work", and we can bump the SRCREV of any dependency if a CVE or issue is found. Definitely a balancing act, with no 'one true answer' yet, so I've been going with what I know works :) Two notes: After installing the package provides /bin/docker-compose. I realized other parties provide this as /usr/lib/docker/cli-plugins/docker-compose which make something like "docker compose version" working. For me this does not harm, only as remark. May be this is intentional. Completely random. I just put it there. I'll make that adjustment in the recipe. Upon investigating I realized the version reported by docker-compose is "dev" instead of the expected "v2.11.2". I saw the same thing, and was looking into fixing it, I figured it was worth getting the recipe out, before I did more tweaking. Great work, thank you! I appreciate the testing and the report, very helpful! Bruce Regards Konstantin -- - Thou shalt not follow the NULL pointer, for chaos and madness await thee at its end - "Use the force Harry" - Gandalf, Star Trek II
More All Messages By This Member
Konstantin Kletschke #58354 On Sun, Oct 16, 2022 at 10:28:18PM -0400, Bruce Ashfield wrote: FYI: Here's the very lightly tested RFC version of the recipe: https://git.yoctoproject.org/meta-virtualization/commit/?h=master-next&id=fa24cfc08b129de5c7ba36099985359cc0228630 Thanks for the kind notification. My stuff is based upon kirkstone, I stuffed the recipe from master-next into my kirkstone build. One package of this new docker-compose is depending upon go-1.19 (kirkstone provides go-1.17) so I pulled in go-1.19 from langdale also into my kirkstone build. The compiling and building works just fine! The recipe(s) for docker-compose look impressive, I would not have expected them being so complex. But they work. Two notes: After installing the package provides /bin/docker-compose. I realized other parties provide this as /usr/lib/docker/cli-plugins/docker-compose which make something like "docker compose version" working. For me this does not harm, only as remark. May be this is intentional. Upon investigating I realized the version reported by docker-compose is "dev" instead of the expected "v2.11.2". Great work, thank you! Regards Konstantin
More All Messages By This Member
Bruce Ashfield #58347 On Thu, Oct 13, 2022 at 10:05 AM Konstantin Kletschke <konstantin.kletschke@...> wrote: On Tue, Oct 11, 2022 at 11:23:31PM -0300, Bruce Ashfield wrote: > Adding the missing setuptools does get things working. Oh my, I was still looking for python3-distutils (deprecated, not available) and did not realize I now need setuptools. Thanks for clarifying, however, I investigated the gao approach then... > I actually have a prototype recipe for this that I was working on before > ELCe, but I didn't get it into meta-virtualization yet, as it had a few > rough edges. I suppose, those go recipes look extremly difficult to do. > If you give me a few days, I can post it to the meta-virtualization list, > but I'm on the road right now and don't have access to all my build > machines. Of course I have patience and I am very curious to test this out! Currently I have no urge but in future it will be extremely handy to have the native docker compose approach available. It is a bit smaller then the python approach (if python is only used by this docker-compose, a third disk space is used by native approach). > I also did a presentation at the yocto summit about "modern languages". Opps, interesting. No need to summarize here, I agree. I will dig this up in the internet. Interesting... > You can see the approach that I take for this in the k3s and nerdctl recipes > in meta-virtualization. My new docker-compose recipe is of similar format. As I vaguely mentioned above, those recipes look far more complex than I would have imagined when starting to dig into the go world... astonishing! FYI: Here's the very lightly tested RFC version of the recipe: https://git.yoctoproject.org/meta-virtualization/commit/?h=master-next&id=fa24cfc08b129de5c7ba36099985359cc0228630 Cheers, Bruce > If we just bypass the fetcher, offline builds, some of licensing and SBOM > and reproducible builds .. you can have a simple recipe like that as well :) Yea, and I already learned to lovae this reproducibility approach. -- INSIDE M2M GmbH Konstantin Kletschke Berenbosteler Straße 76 B 30823 Garbsen Telefon: +49 (0) 5137 90950136 Mobil: +49 (0) 151 15256238 Fax: +49 (0) 5137 9095010 konstantin.kletschke@... http://www.inside-m2m.de Geschäftsführung: Michael Emmert, Ingo Haase, Dr. Fred Könemann, Derek Uhlig HRB: 111204, AG Hannover -- - Thou shalt not follow the NULL pointer, for chaos and madness await thee at its end - "Use the force Harry" - Gandalf, Star Trek II
More All Messages By This Member
Konstantin Kletschke #58332 On Tue, Oct 11, 2022 at 11:23:31PM -0300, Bruce Ashfield wrote: Adding the missing setuptools does get things working. Oh my, I was still looking for python3-distutils (deprecated, not available) and did not realize I now need setuptools. Thanks for clarifying, however, I investigated the gao approach then... I actually have a prototype recipe for this that I was working on before ELCe, but I didn't get it into meta-virtualization yet, as it had a few rough edges. I suppose, those go recipes look extremly difficult to do. If you give me a few days, I can post it to the meta-virtualization list, but I'm on the road right now and don't have access to all my build machines. Of course I have patience and I am very curious to test this out! Currently I have no urge but in future it will be extremely handy to have the native docker compose approach available. It is a bit smaller then the python approach (if python is only used by this docker-compose, a third disk space is used by native approach). I also did a presentation at the yocto summit about "modern languages". Opps, interesting. No need to summarize here, I agree. I will dig this up in the internet. Interesting... You can see the approach that I take for this in the k3s and nerdctl recipes in meta-virtualization. My new docker-compose recipe is of similar format. As I vaguely mentioned above, those recipes look far more complex than I would have imagined when starting to dig into the go world... astonishing! If we just bypass the fetcher, offline builds, some of licensing and SBOM and reproducible builds .. you can have a simple recipe like that as well :) Yea, and I already learned to lovae this reproducibility approach. -- INSIDE M2M GmbH Konstantin Kletschke Berenbosteler Straße 76 B 30823 Garbsen Telefon: +49 (0) 5137 90950136 Mobil: +49 (0) 151 15256238 Fax: +49 (0) 5137 9095010 konstantin.kletschke@... http://www.inside-m2m.de Geschäftsführung: Michael Emmert, Ingo Haase, Dr. Fred Könemann, Derek Uhlig HRB: 111204, AG Hannover
More All Messages By This Member
Bruce Ashfield #58325 On Tue, Oct 11, 2022 at 2:51 PM Konstantin Kletschke <konstantin.kletschke@...> wrote: Today I realized, that the docker-compose in the meta-virtualization layer is the old python3 based one and a native solution is available. I tried to make a recipe for it to get rid of python3 (no other packages uses it in our setup) and the current python3-docker-compose is broken: https://lore.kernel.org/yocto-meta-virtualization/49660EA1-CD22-40DB-98C7-C43F38A72CCA@.../ Adding the missing setuptools does get things working. I tried to build the native solution proposed in https://github.com/docker/compose/tree/v2.11.2. I actually have a prototype recipe for this that I was working on before ELCe, but I didn't get it into meta-virtualization yet, as it had a few rough edges. If you give me a few days, I can post it to the meta-virtualization list, but I'm on the road right now and don't have access to all my build machines. So my current recipe docker-compose_2.11.2.bb looks like this: LICENSE = "Apache-2.0" inherit go inherit go-mod GO_IMPORT = "github.com/docker/compose" SRC_URI = "git://${GO_IMPORT};protocol=https;branch=v2" # v2.11.2 SRCREV = "616777eb4ad4d1101622d6727d9b7adaeb7943bb" # S = "${WORKDIR}/git" DEPENDS = "docker-ce" RDEPENDS:${PN} = "docker-ce-cli" With this I run into the issue, that go want's to download stuff while compiling. Which is forbidden and breaks the reliable build proposal. Totally understandable. Looks similair to this: dial tcp: lookup proxy.golang.org: Temporary failure in name resolution I read about a proposal allowing this by adding do_compile[network] = "1" to meta/classes/go.bb: https://lore.kernel.org/all/20220228235433.3948994-1-andrei@.../ Of course, if this works, the reliable build thingy is gone, but while Right, it works, but would never make it into core due to it bypassing many of the fetcher properties. trying I discovered go loads external stuff, compiles it too. In my case (I have kirkstone with go-1.17) I run into modules demanding a go more recent: ASH[build k8s.io/client-go/applyconfigurations/autoscaling/v2beta2]: "file containerresourcemetricsource.go Mas4-HIX5lGBEQNTIo58\n" # github.com/docker/compose/v2/pkg/utils pkg/utils/slices.go:23:6: missing function body pkg/utils/slices.go:23:14: syntax error: unexpected [, expecting ( note: module requires Go 1.19 HASH[build k8s.io/client-go/applyconfigurations/certificates/v1beta1] HASH[build k8s.io/client-go/applyconfigurations/certificates/v1beta1]: "go1.17.13" which could be worked around by moving to landsdale release. Which bothers me, because I too want to keep my build reliable. And I do not want to mess around outside our meta layer in the distribution! How do I do this properly? I see people pulling in each dependency by individual golang.org-x.bb recipes, how could such a recipe look like? There are many different discussions about how to generate go recipes, as well as similar discussions for ruby/rust, etc, you can find them on the openembedded-core and openembedded-architectures lists. I also did a presentation at the yocto summit about "modern languages". So I won't try and summarize all those discussions here, since they are archived and do a much better job than I could here. But to answer the specific question .. you can't really handle the individual go dependencies as recipes. They are far too sensitive to versions and individual git hashes. Plus you'll end up with thousands of recipes of varying versions. There have been several proposals about how to generate go recipes to deal with the dependencies. Some prototypes have been posted, but there hasn't been a lot of traffic on that topic for several months now. You can see the approach that I take for this in the k3s and nerdctl recipes in meta-virtualization. My new docker-compose recipe is of similar format. I read https://lore.kernel.org/all/8132db85-5881-636e-c091-d84c47efe4fe@.../T/ where Mike is not happy with this appraoch either and comes up with a working recipe I don't get why this could work at all. What am I missing here where is the missing link I did not get yet? Also, I am jealous about the buildroot guys sometimes, how do they do this in a 22 lines makefile including comments: https://git.busybox.net/buildroot/tree/package/docker-compose/docker-compose.mk If we just bypass the fetcher, offline builds, some of licensing and SBOM and reproducible builds .. you can have a simple recipe like that as well :) Bruce Regards Konstantin -- INSIDE M2M GmbH Konstantin Kletschke Berenbosteler Straße 76 B 30823 Garbsen Telefon: +49 (0) 5137 90950136 Mobil: +49 (0) 151 15256238 Fax: +49 (0) 5137 9095010 konstantin.kletschke@... http://www.inside-m2m.de Geschäftsführung: Michael Emmert, Ingo Haase, Dr. Fred Könemann, Derek Uhlig HRB: 111204, AG Hannover -- - Thou shalt not follow the NULL pointer, for chaos and madness await thee at its end - "Use the force Harry" - Gandalf, Star Trek II
More All Messages By This Member
Konstantin Kletschke #58323 Today I realized, that the docker-compose in the meta-virtualization layer is the old python3 based one and a native solution is available. I tried to make a recipe for it to get rid of python3 (no other packages uses it in our setup) and the current python3-docker-compose is broken: https://lore.kernel.org/yocto-meta-virtualization/49660EA1-CD22-40DB-98C7-C43F38A72CCA@ieee.org/ I tried to build the native solution proposed in https://github.com/docker/compose/tree/v2.11.2. So my current recipe docker-compose_2.11.2.bb looks like this: LICENSE = "Apache-2.0" inherit go inherit go-mod GO_IMPORT = "github.com/docker/compose" SRC_URI = "git://${GO_IMPORT};protocol=https;branch=v2" # v2.11.2 SRCREV = "616777eb4ad4d1101622d6727d9b7adaeb7943bb" # S = "${WORKDIR}/git" DEPENDS = "docker-ce" RDEPENDS:${PN} = "docker-ce-cli" With this I run into the issue, that go want's to download stuff while compiling. Which is forbidden and breaks the reliable build proposal. Totally understandable. Looks similair to this: dial tcp: lookup proxy.golang.org: Temporary failure in name resolution I read about a proposal allowing this by adding do_compile[network] = "1" to meta/classes/go.bb: https://lore.kernel.org/all/20220228235433.3948994-1-andrei@gherzan.com/ Of course, if this works, the reliable build thingy is gone, but while trying I discovered go loads external stuff, compiles it too. In my case (I have kirkstone with go-1.17) I run into modules demanding a go more recent: ASH[build k8s.io/client-go/applyconfigurations/autoscaling/v2beta2]: "file containerresourcemetricsource.go Mas4-HIX5lGBEQNTIo58\n" # github.com/docker/compose/v2/pkg/utils pkg/utils/slices.go:23:6: missing function body pkg/utils/slices.go:23:14: syntax error: unexpected [, expecting ( note: module requires Go 1.19 HASH[build k8s.io/client-go/applyconfigurations/certificates/v1beta1] HASH[build k8s.io/client-go/applyconfigurations/certificates/v1beta1]: "go1.17.13" which could be worked around by moving to landsdale release. Which bothers me, because I too want to keep my build reliable. And I do not want to mess around outside our meta layer in the distribution! How do I do this properly? I see people pulling in each dependency by individual golang.org-x.bb recipes, how could such a recipe look like? I read https://lore.kernel.org/all/8132db85-5881-636e-c091-d84c47efe4fe@gmail.com/T/ where Mike is not happy with this appraoch either and comes up with a working recipe I don't get why this could work at all. What am I missing here where is the missing link I did not get yet? Also, I am jealous about the buildroot guys sometimes, how do they do this in a 22 lines makefile including comments: https://git.busybox.net/buildroot/tree/package/docker-compose/docker-compose.mk Regards Konstantin -- INSIDE M2M GmbH Konstantin Kletschke Berenbosteler Straße 76 B 30823 Garbsen Telefon: +49 (0) 5137 90950136 Mobil: +49 (0) 151 15256238 Fax: +49 (0) 5137 9095010 konstantin.kletschke@... http://www.inside-m2m.de Geschäftsführung: Michael Emmert, Ingo Haase, Dr. Fred Könemann, Derek Uhlig HRB: 111204, AG Hannover
More All Messages By This Member

1 - 8 of 8

1

Previous Topic Next Topic

Home Hashtags Subgroups Terms