Fetch private gitlab repo using ssh with Yocto recipe #bitbake


Sourabh Hegde
 

I am trying to fetch a private gitlab repo within Yocto image recipe using SSH protocol. In my image recipe I have passed SRC_URI as:

SRC_URI = " \
        gitsm://git@...:2224/blah/blah/blah/blah;protocol=ssh;branch=master \
"
But this results in the error:

ERROR: Fetcher failure: Fetch command export PSEUDO_DISABLED=1; export PATH="/root /build-swu-v2/tmp/sysroots-uninative/x86_64-linux/usr/bin:/root/sources/poky/scripts: /root/build-swu-v2/tmp/work/cortexa7t2hf-neon-vfpv4-poky-linux-gnueabi/awsclient/1.4- r0/recipe-sysroot-native/usr/bin/arm-poky-linux-gnueabi:/root/build-swu-v2/tmp /work/cortexa7t2hf-neon-vfpv4-poky-linux-gnueabi/awsclient/1.4-r0/recipe-sysroot/usr/bin /crossscripts:/root/build-swu-v2/tmp/work/cortexa7t2hf-neon-vfpv4-poky-linux-gnueabi /awsclient/1.4-r0/recipe-sysroot-native/usr/sbin:/root/build-swu-v2/tmp/work/cortexa7t2hf- neon-vfpv4-poky-linux-gnueabi/awsclient/1.4-r0/recipe-sysroot-native/usr/bin:/root/build- swu-v2/tmp/work/cortexa7t2hf-neon-vfpv4-poky-linux-gnueabi/awsclient/1.4-r0/recipe-sysroot- native/sbin:/root/build-swu-v2/tmp/work/cortexa7t2hf-neon-vfpv4-poky-linux-gnueabi /awsclient/1.4-r0/recipe-sysroot-native/bin:/root/sources/poky/bitbake/bin:/root/build-swu- v2/tmp/hosttools"; export HOME="/root"; LANG=C git -c core.fsyncobjectfiles=0 clone --bare  --mirror "ssh://git@...:2224/iot/products/iot-device-suite_client/aws- iot-client" /root/downloads//git2/git@...:2224/blah/blah/blah/blah --progress failed with exit code 128, no output
ERROR: Bitbake Fetcher Error: FetchError('Unable to fetch URL from any source.',  'gitsm://git@...:2224/blah/blah/blah/blah;protocol=ssh;branch=master')
DEBUG: Python function base_do_fetch finished

But I am able to clone the repo using git clone.

SSH key is already added to the Gitlab. There is no config file in my ~/.ssh. Do I need to create a config file? What should be the content of the config file?

Can anyone please let me know how to resolve this?

Thanks in advance.


Nicolas Jeker
 

On Tue, 2022-01-25 at 23:16 -0800, hrsourabh011@... wrote:
I am trying to fetch a private gitlab repo within Yocto image recipe
using SSH protocol. In my image recipe I have passed SRC_URI as:

SRC_URI = " \
        gitsm://git@...:2224/blah/blah/blah/blah;protocol
=ssh;branch=master \
"
I use almost the same, just without submodules.

SRC_URI =
"git://git@...:1234/group/project.git;protocol=ssh"

It should "just work" if ssh is able to find your key. I often build in
a docker container, so I have to forward SSH_AGENT into it to be able
to fetch from internal projects without the need to mount the key into
the container. I don't really have any insight for builds outside
docker, if git clone works, the bitbake fetcher should too.

But this results in the error:
<snip>

But I am able to clone the repo using git clone.
SSH key is already added to the Gitlab. There is no config file in my
~/.ssh. Do I need to create a config file? What should be the content
of the config file?
You should not need a ssh config file.

Can anyone please let me know how to resolve this?
Thanks in advance.


Sourabh Hegde
 

Hi Nicolas,

Thanks for your answer.

That's great. Even I am building inside a docker container. I tried with creating a "config" file in .ssh directory. But I still have same issue.

Can you please let me know how to "forward SSH_AGENT into it to be able
to fetch from internal projects without the need to mount the key into the container."? I never did that before.

Thanks in advance.

On Fri, Jan 28, 2022, 10:42 Nicolas Jeker <n.jeker@...> wrote:
On Tue, 2022-01-25 at 23:16 -0800, hrsourabh011@... wrote:
> I am trying to fetch a private gitlab repo within Yocto image recipe
> using SSH protocol. In my image recipe I have passed SRC_URI as:
>
> SRC_URI = " \
>         gitsm://git@...:2224/blah/blah/blah/blah;protocol
> =ssh;branch=master \
> "

I use almost the same, just without submodules.

SRC_URI =
"git://git@...:1234/group/project.git;protocol=ssh"

It should "just work" if ssh is able to find your key. I often build in
a docker container, so I have to forward SSH_AGENT into it to be able
to fetch from internal projects without the need to mount the key into
the container. I don't really have any insight for builds outside
docker, if git clone works, the bitbake fetcher should too.

> But this results in the error:
>
<snip>
>
> But I am able to clone the repo using git clone.
> SSH key is already added to the Gitlab. There is no config file in my
> ~/.ssh. Do I need to create a config file? What should be the content
> of the config file?

You should not need a ssh config file.

> Can anyone please let me know how to resolve this?
> Thanks in advance.


VIVAVIS AG
 

Hi,

Von: yocto@... <yocto@...> Im Auftrag von Sourabh Hegde
Gesendet: Freitag, 28. Januar 2022 10:47

Can you please let me know how to "forward SSH_AGENT into it to be able
to fetch from internal projects without the need to mount the key into the container."? I never did that before.
I use the following options within the Docker run command:
-v $SSH_AUTH_SOCK:/ssh.socket \
-e SSH_AUTH_SOCK=/ssh.socket \

Furthermore, I had to mount the .ssh folder into the container to make it working (be aware of security risk).
Additionally, you should check that uid, gid of the user in the container is the same on the host.

Regards,

Carsten


Nicolas Jeker
 

On Fri, 2022-01-28 at 10:27 +0000, VIVAVIS AG wrote:
Hi,
 
Von: yocto@... <yocto@...> Im
Auftrag von Sourabh Hegde
Gesendet: Freitag, 28. Januar 2022 10:47

Can you please let me know how to "forward SSH_AGENT into it to be
able
to fetch from internal projects without the need to mount the key
into the container."? I never did that before.
I use the following options within the Docker run command:
  -v $SSH_AUTH_SOCK:/ssh.socket \
  -e SSH_AUTH_SOCK=/ssh.socket \
That's pretty much what I use.

Furthermore, I had to mount the .ssh folder into the container to
make it working (be aware of security risk).
Additionally, you should check that uid, gid of the user in the
container is the same on the host.
I do something similar, my "problem" was that ssh needs the
.ssh/known_hosts file with a matching entry in addition to your
key/agent, but mounting the .ssh folder was not possible for me because
of permissions. Currently, I just created a little script that wraps
"oe-init-build-env" and populates the known_hosts file accordingly.

mkdir -p ~/.ssh

cat <<EOF >> ~/.ssh/known_hosts
git.example.com ssh-ed25519 <base64key>
EOF

Regards,

Carsten


Erik Boto
 

On Fri, Jan 28, 2022 at 11:50 AM Nicolas Jeker <n.jeker@...> wrote:

On Fri, 2022-01-28 at 10:27 +0000, VIVAVIS AG wrote:
Hi,

Von: yocto@... <yocto@...> Im
Auftrag von Sourabh Hegde
Gesendet: Freitag, 28. Januar 2022 10:47

Can you please let me know how to "forward SSH_AGENT into it to be
able
to fetch from internal projects without the need to mount the key
into the container."? I never did that before.
I use the following options within the Docker run command:
-v $SSH_AUTH_SOCK:/ssh.socket \
-e SSH_AUTH_SOCK=/ssh.socket \
That's pretty much what I use.

Furthermore, I had to mount the .ssh folder into the container to
make it working (be aware of security risk).
Additionally, you should check that uid, gid of the user in the
container is the same on the host.
I do something similar, my "problem" was that ssh needs the
.ssh/known_hosts file with a matching entry in addition to your
key/agent, but mounting the .ssh folder was not possible for me because
of permissions. Currently, I just created a little script that wraps
"oe-init-build-env" and populates the known_hosts file accordingly.

mkdir -p ~/.ssh

cat <<EOF >> ~/.ssh/known_hosts
git.example.com ssh-ed25519 <base64key>
EOF
I use my own Dockerfile based on crops/poky where I do the following,
which might be helpful if you also use this. It sets up the config
changes in /etc/skel/ since it creates users "on the fly" with
matching uid.

# Remove strict host key checking for ssh
# This is needed since the build will pull source over git-ssh
RUN mkdir -p /etc/skel/.ssh/
COPY ci-scripts/docker-stuff/config /etc/skel/.ssh/
RUN echo 'export GIT_SSH_COMMAND="ssh -o UserKnownHostsFile=/dev/null
-o StrictHostKeyChecking=no"' >> /etc/skel/.bashrc


The ci-scripts/docker-stuff/config file contains:
Host *
StrictHostKeyChecking no
UserKnownHostsFile=/dev/null

Now it was ages ago I set this up, and right now I can't really
understand why I basically do the same thing twice. So you'd have to
check which of the two things that actually solves the issue :-)

Cheers,
Erik


Regards,

Carsten



Khem Raj
 

On Fri, Jan 28, 2022 at 2:27 AM VIVAVIS AG <embedded@...> wrote:

Hi,

Von: yocto@... <yocto@...> Im Auftrag von Sourabh Hegde
Gesendet: Freitag, 28. Januar 2022 10:47

Can you please let me know how to "forward SSH_AGENT into it to be able
to fetch from internal projects without the need to mount the key into the container."? I never did that before.
I use the following options within the Docker run command:
-v $SSH_AUTH_SOCK:/ssh.socket \
-e SSH_AUTH_SOCK=/ssh.socket \

Furthermore, I had to mount the .ssh folder into the container to make it working (be aware of security risk).
Additionally, you should check that uid, gid of the user in the container is the same on the host.
yeah something like that works, we use it for yoe which always uses
container to build
see

https://github.com/YoeDistro/yoe-distro/blob/master/envsetup.sh#L528-L541

Regards,

Carsten



Sourabh Hegde
 

Hello @Nicolas @Erik @Khem,

Update from my side:

After following some discussion from other posts, I added "config" file.

~/.ssh/config:

Host git.example.com
HostName git.example.com
User git
PreferredAuthentications publickey
IdentityFile ~/.ssh/id_ed25519.pub
# LogLevel DEBUG3

Then I did "eval `ssh-agent -s`"

Then doing "ssh-add ~/.ssh/id_ed25519.pub" results in:

@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@         WARNING: UNPROTECTED PRIVATE KEY FILE!          @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
Permissions 0644 for '/root/.ssh/id_ed25519.pub' are too open.
It is required that your private key files are NOT accessible by others.
This private key will be ignored.

Whereas the permissions are set as:

ls -l -a ~/.ssh

-rw-r--r-- 1 root root  157 Jan 31 10:48 config
-rw------- 1 root root  464 Jan 20 15:26 id_ed25519
-rw-r--r-- 1 root root  109 Jan 20 15:26 id_ed25519.pub
-rw-r--r-- 1 root root  888 Jan 26 08:43 known_hosts

"ssh-agent" is running

ssh-agent
SSH_AUTH_SOCK=/tmp/ssh-lcft54A4nriC/agent.2833; export SSH_AUTH_SOCK;
SSH_AGENT_PID=2834; export SSH_AGENT_PID;
echo Agent pid 2834;

After doing these changes, when I try to "ssh -v git.example.com" to test the connection before running bitbake, I get

OpenSSH_8.2p1 Ubuntu-4ubuntu0.4, OpenSSL 1.1.1f  31 Mar 2020
debug1: Reading configuration data /root/.ssh/config
debug1: /root/.ssh/config line 1: Applying options for git.example.com
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: include /etc/ssh/ssh_config.d/*.conf matched no files
debug1: /etc/ssh/ssh_config line 21: Applying options for *
debug1: Connecting to git.example.com [116.203.241.xxx] port 22.
debug1: connect to address 116.203.241.xxx port 22: Connection refused
ssh: connect to host git.example.com port 22: Connection refused

I don't understand what is the issue here.

@Nicolas Can you please let me know where and how to run below commands? Do I need to run them every time before fetching from gitlab?
  -v $SSH_AUTH_SOCK:/ssh.socket \
  -e SSH_AUTH_SOCK=/ssh.socket \

And also I already have "known_hosts" file with matching entries for key/agent pair.

Can you please let me know how to make this working?

Your help will be much appreciated.

Thanks in advance.


Nicolas Jeker
 

On Mon, 2022-01-31 at 02:54 -0800, Sourabh Hegde wrote:
Hello @Nicolas @Erik @Khem,
Hi!

Update from my side:

After following some discussion from other posts, I added "config"
file.
I think you're starting to mix various things together, you should
maybe try to not do everything at the same time. I added comments about
what is wrong with your config, but depending on your build
environment, the ssh config is maybe not the best choice.

~/.ssh/config:

Host git.example.com
HostName git.example.com
User git
PreferredAuthentications publickey
IdentityFile ~/.ssh/id_ed25519.pub
# LogLevel DEBUG3
You need to specify the private key with IdentityFile, not the public
key.

Then I did "eval `ssh-agent -s`"

Then doing "ssh-add ~/.ssh/id_ed25519.pub" results in:
Same here, you should be doing "ssh-add ~/.ssh/id_ed25519" (without the
.pub).

@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@         WARNING: UNPROTECTED PRIVATE KEY FILE!          @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
Permissions 0644 for '/root/.ssh/id_ed25519.pub' are too open.
It is required that your private key files are NOT accessible by
others.
This private key will be ignored.

Whereas the permissions are set as:

ls -l -a ~/.ssh

-rw-r--r-- 1 root root  157 Jan 31 10:48 config
-rw------- 1 root root  464 Jan 20 15:26 id_ed25519
-rw-r--r-- 1 root root  109 Jan 20 15:26 id_ed25519.pub
-rw-r--r-- 1 root root  888 Jan 26 08:43 known_hosts
Well, the permissions on id_ed25519 are correct, but you added the
public key as private key in your config / in your ssh-add command,
which doesn't have the required permissions for private keys (because
it's not).

"ssh-agent" is running

ssh-agent
SSH_AUTH_SOCK=/tmp/ssh-lcft54A4nriC/agent.2833; export SSH_AUTH_SOCK;
SSH_AGENT_PID=2834; export SSH_AGENT_PID;
echo Agent pid 2834;

After doing these changes, when I try to "ssh -v git.example.com" to
test the connection before running bitbake, I get

OpenSSH_8.2p1 Ubuntu-4ubuntu0.4, OpenSSL 1.1.1f  31 Mar 2020
debug1: Reading configuration data /root/.ssh/config
debug1: /root/.ssh/config line 1: Applying options for
git.example.com
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: include
/etc/ssh/ssh_config.d/*.conf matched no files
debug1: /etc/ssh/ssh_config line 21: Applying options for *
debug1: Connecting to git.example.com [116.203.241.xxx] port 22.
debug1: connect to address 116.203.241.xxx port 22: Connection
refused
ssh: connect to host git.example.com port 22: Connection refused

I don't understand what is the issue here.

@Nicolas Can you please let me know where and how to run below
commands? Do I need to run them every time before fetching from
gitlab?
  -v $SSH_AUTH_SOCK:/ssh.socket \
  -e SSH_AUTH_SOCK=/ssh.socket \
I think you should explain your build environment a bit better, as I
can just guess what you're doing. You should add these parameters when
starting your docker container. For example I use something along these
lines:

docker run -ti --rm -v ~/development/oe-build:/workdir -v
$SSH_AUTH_SOCK:$SSH_AUTH_SOCK -e SSH_AUTH_SOCK="$SSH_AUTH_SOCK"
crops/poky --workdir=/workdir

If you're forwarding the ssh agent like this, you don't need a key or
config file at all, only known_hosts.


On the other hand, if you're using e.g. GitLab pipelines with docker,
you should not do it like mentioned above, but follow their guide [1].

[1]:
https://docs.gitlab.com/ee/ci/ssh_keys/index.html#ssh-keys-when-using-the-docker-executor

And also I already have "known_hosts" file with matching entries for
key/agent pair.

Can you please let me know how to make this working?

Your help will be much appreciated.

Thanks in advance.