Topics

#yocto systemd not able to start sshd after a reboot #yocto


srijan.nandi@...
 

I am facing a peculiar problem with openssh. I have built openssh_8.0p1on zeus. 

The sshd.service file is as follows:
[Unit]
Description=OpenSSH server daemon
Documentation=man:sshd(8) man:sshd_config(5)
After=sshdgenkeys.service
Wants=sshdgenkeys.service
 
[Service]
Type=simple
PIDFile=/var/run/sshd.pid
EnvironmentFile=-/etc/default/sshd
ExecStart=/usr/sbin/sshd -D $OPTIONS
ExecReload=/bin/kill -HUP $MAINPID
ExecStop=/bin/kill $MAINPID
PermissionsStartOnly=true
KillMode=process
Restart=on-failure
StandardError=syslog
 
[Install]
WantedBy=multi-user.target

It starts without issues, if i do a systemctl start sshd.service. If I do a test of the config file it does not come up with any errors:

genericx86-64:~$ sudo /usr/sbin/sshd -t
genericx86-64:~$ 

Problem:
If I reboot the server, sshd does not start. There is no error on syslog. I have enabled debug logging, still no logs in syslog.

# Logging
SyslogFacility DAEMON
LogLevel DEBUG3

Even systemctl is-enabled sshd shows as enabled.

After a reboot, if I do a systemctl status sshd it shows:
Loaded: loaded  (/lib/systemd/system/sshd.service; enabled; vendor preset: enabled)
Active: inactive (dead)

If I manually run systemctl start sshd.service, everything works perfectly well without issues. sshd start on 0.0.0.0:2224 and I am able to ssh in as well.

It's just that systemctl is not able to start sshd after a reboot and there is no error that i can find or debug. Absolutely run out of ideas to resolve this. Any help will be greatly appreciated.

Thanks and Regards,
-=Srijan Nandi


Khem Raj
 

does something like this work?

[Unit]
Description=OpenSSH Daemon
Wants=sshdgenkeys.service
After=sshdgenkeys.service
After=network.target

[Service]
ExecStart=/usr/bin/sshd -D
ExecReload=/bin/kill -HUP $MAINPID
KillMode=process
Restart=always

[Install]
WantedBy=multi-user.target

On Fri, Sep 18, 2020 at 8:29 AM <srijan.nandi@...> wrote:

I am facing a peculiar problem with openssh. I have built openssh_8.0p1on zeus.

The sshd.service file is as follows:
[Unit]
Description=OpenSSH server daemon
Documentation=man:sshd(8) man:sshd_config(5)
After=sshdgenkeys.service
Wants=sshdgenkeys.service

[Service]
Type=simple
PIDFile=/var/run/sshd.pid
EnvironmentFile=-/etc/default/sshd
ExecStart=/usr/sbin/sshd -D $OPTIONS
ExecReload=/bin/kill -HUP $MAINPID
ExecStop=/bin/kill $MAINPID
PermissionsStartOnly=true
KillMode=process
Restart=on-failure
StandardError=syslog

[Install]
WantedBy=multi-user.target

It starts without issues, if i do a systemctl start sshd.service. If I do a test of the config file it does not come up with any errors:

genericx86-64:~$ sudo /usr/sbin/sshd -t
genericx86-64:~$

Problem:
If I reboot the server, sshd does not start. There is no error on syslog. I have enabled debug logging, still no logs in syslog.

# Logging
SyslogFacility DAEMON
LogLevel DEBUG3

Even systemctl is-enabled sshd shows as enabled.

After a reboot, if I do a systemctl status sshd it shows:
Loaded: loaded (/lib/systemd/system/sshd.service; enabled; vendor preset: enabled)
Active: inactive (dead)

If I manually run systemctl start sshd.service, everything works perfectly well without issues. sshd start on 0.0.0.0:2224 and I am able to ssh in as well.

It's just that systemctl is not able to start sshd after a reboot and there is no error that i can find or debug. Absolutely run out of ideas to resolve this. Any help will be greatly appreciated.

Thanks and Regards,
-=Srijan Nandi



srijan.nandi@...
 

Hello Khem,

With the above sshd.service file the sshd daemon fails to start. It gives an error "(code=exited, status=203/EXEC)".

Cannot figure out as to why the systemd for sshd fails to work, while the other systemd files are working perfectly fine.

-=Srijan Nandi


Zoran
 

Hello Srijan,

Did you recap/look into this sshd.service file?
https://lists.yoctoproject.org/g/yocto/message/49993

Zoran
_______

On Fri, Sep 18, 2020 at 8:07 PM <srijan.nandi@...> wrote:

Hello Khem,

With the above sshd.service file the sshd daemon fails to start. It gives an error "(code=exited, status=203/EXEC)".

Cannot figure out as to why the systemd for sshd fails to work, while the other systemd files are working perfectly fine.

-=Srijan Nandi


srijan.nandi@...
 

Hello All,

I finally got it to work!!!

There was a sshd.socket file in /lib/systemd/system which had the following line in it.

Conflicts=sshd.service

I remove it and added the following two lines:

After=network.target
Before=sshd.service

And that did the trick. Now sshd service starts on every boot.

Thanks,
-=Srijan Nandi


Zoran
 

Interesting... Here is what I have on Fedora32:

[root@fedora32-ssd system]# pwd
/lib/systemd/system
[root@fedora32-ssd system]# cat /lib/systemd/system/sshd.service
[Unit]
Description=OpenSSH server daemon
Documentation=man:sshd(8) man:sshd_config(5)
After=network.target sshd-keygen.target
Wants=sshd-keygen.target

[Service]
Type=notify
EnvironmentFile=-/etc/sysconfig/sshd-permitrootlogin
EnvironmentFile=-/etc/sysconfig/sshd
ExecStart=/usr/sbin/sshd -D $OPTIONS $PERMITROOTLOGIN
ExecReload=/bin/kill -HUP $MAINPID
KillMode=process
Restart=on-failure
RestartSec=42s

[Install]
WantedBy=multi-user.target

Seems that some leftovers from System V still reside in YOCTO... Correct???

Best Regards,
Zoran
_______


On Sat, Sep 19, 2020 at 3:37 PM <srijan.nandi@...> wrote:
Hello All,

I finally got it to work!!!

There was a sshd.socket file in /lib/systemd/system which had the following line in it.

Conflicts=sshd.service

I remove it and added the following two lines:

After=network.target
Before=sshd.service

And that did the trick. Now sshd service starts on every boot.

Thanks,
-=Srijan Nandi



srijan.nandi@...
 

Seems that some leftovers from System V still reside in YOCTO... Correct???

Not sure about that. 

The problem I faced was because there was a sshd.socket that had the following line in it. The sshd.socket comes with openssh. 

Conflicts=sshd.service

So I had two options. either to add the ExecStartPre in the sshd.service file or to remove the Conflicts line in sshd.socket. 

I choose to remove the Conflicts line and add the following in the sshd.socket file.

After=network.target
Before=sshd.service

Thanks and Regards,
-=Srijan Nandi


Zoran
 

There was a sshd.socket file in /lib/systemd/system which had the following line in it.
Interesting... Pushed/forced me to think.

There is no formal dependency between sshd.service and sshd.socket!

[vuser@fedora32-ssd systemd]$ systemctl list-dependencies sshd.service
| grep ssh
sshd.service
● ├─sshd-keygen.target
● │ ├─sshd-keygen@...
● │ ├─sshd-keygen@...
● │ └─sshd-keygen@...
[vuser@fedora32-ssd systemd]$ systemctl list-dependencies sshd.service
| grep socket
● ├─lvm2-lvmetad.socket
● ├─lvm2-lvmpolld.socket
[vuser@fedora32-ssd systemd]$ systemctl list-dependencies sshd.socket
| grep sshd
sshd.socket

Strange... Isn't it?!

Zoran
_______

On Sat, Sep 19, 2020 at 3:37 PM <srijan.nandi@...> wrote:

Hello All,

I finally got it to work!!!

There was a sshd.socket file in /lib/systemd/system which had the following line in it.

Conflicts=sshd.service

I remove it and added the following two lines:

After=network.target
Before=sshd.service

And that did the trick. Now sshd service starts on every boot.

Thanks,
-=Srijan Nandi


srijan.nandi@...
 

Seems strange to me too...I had been troubleshooting the sshd.service issue for two days. There was no logs, nothing. I was just hitting the wall. I tried a lot of combinations to get it to work but all failed.

The sshd.service was starting, if I manually did a systemctl start sshd.service. But always failed at startup. At times it would start and then get a signal 15 terminating and would close the daemon.

After not able to resolve the issue, I started checking all the other services. Every other service was starting at bootup just fine except sshd.

Not finding anything else to troubleshoot. I happened to stumble upon the sshd.socket and the Conflicts part of it. Did a hit and trial and it worked. Technically I still am not sure as to why..

-=Srijan Nandi


Khem Raj
 

On 9/21/20 5:17 AM, srijan.nandi@... wrote:
/Seems that some leftovers from System V still reside in YOCTO...
Correct???
Yocto project supports sysvinit as init system option as well so no there are no leftovers.

/
Not sure about that.
The problem I faced was because there was a sshd.socket that had the following line in it. The sshd.socket comes with openssh.
Conflicts=sshd.service
So I had two options. either to add the ExecStartPre in the sshd.service file or to remove the Conflicts line in sshd.socket.
I choose to remove the Conflicts line and add the following in the sshd.socket file.
After=network.target
Before=sshd.service
Usually you use socket activation for sshd then you would enable sshd.socket and not sshd.service, socket will be listening on incoming connections on ssh port ( 22 by defaault ) and launch sshd.service whenever there is incoming ssh connection request. I suggest you should perhaps follow this approach as well, its also efficient due to its on-demand launch nature.

Thanks and Regards,
-=Srijan Nandi


Zoran
 

Maybe this should be added to sshd.service [unit] part
(to have added dependency) to solve this problem:

> The sshd.service file is as follows:
> [Unit]
> Description=OpenSSH server daemon
> Documentation=man:sshd(8) man:sshd_config(5)
> After=sshdgenkeys.service
> Wants=sshdgenkeys.service
Requires=sshd.socket

Zoran
_______

On Fri, Sep 18, 2020 at 5:29 PM <srijan.nandi@...> wrote:
>
> I am facing a peculiar problem with openssh. I have built openssh_8.0p1on zeus.
>
> The sshd.service file is as follows:
> [Unit]
> Description=OpenSSH server daemon
> Documentation=man:sshd(8) man:sshd_config(5)
> After=sshdgenkeys.service
> Wants=sshdgenkeys.service
>  
> [Service]
> Type=simple
> PIDFile=/var/run/sshd.pid
> EnvironmentFile=-/etc/default/sshd
> ExecStart=/usr/sbin/sshd -D $OPTIONS
> ExecReload=/bin/kill -HUP $MAINPID
> ExecStop=/bin/kill $MAINPID
> PermissionsStartOnly=true
> KillMode=process
> Restart=on-failure
> StandardError=syslog
>  
> [Install]
> WantedBy=multi-user.target
>
> It starts without issues, if i do a systemctl start sshd.service. If I do a test of the config file it does not come up with any errors:
>
> genericx86-64:~$ sudo /usr/sbin/sshd -t
> genericx86-64:~$
>
> Problem:
> If I reboot the server, sshd does not start. There is no error on syslog. I have enabled debug logging, still no logs in syslog.
>
> # Logging
> SyslogFacility DAEMON
> LogLevel DEBUG3
>
> Even systemctl is-enabled sshd shows as enabled.
>
> After a reboot, if I do a systemctl status sshd it shows:
> Loaded: loaded  (/lib/systemd/system/sshd.service; enabled; vendor preset: enabled)
> Active: inactive (dead)
>
> If I manually run systemctl start sshd.service, everything works perfectly well without issues. sshd start on 0.0.0.0:2224 and I am able to ssh in as well.
>
> It's just that systemctl is not able to start sshd after a reboot and there is no error that i can find or debug. Absolutely run out of ideas to resolve this. Any help will be greatly appreciated.
>
> Thanks and Regards,
> -=Srijan Nandi
>
>
>