How to manage root password and emergency mode in RHEL6 and RHEL7

Easiest root password management is to not set it up at all and therefore not manage it.
Why? It may be just too big overhead to make sure its in the same time securely stored, quickly accessible for authorized people when needed and changed right after it has been used.

Okay. You don’t know root password, you have all the needed sudo rights set up but still in case some trouble, machine will not boot up. Instead it drops you into emergency shell demanding root password to continue and you cant get shell without it?

Workaround is to boot from some other Linux image, mount the disk and do whatever you need to do. Really easy if you have one or two machines.
But what if you just broke entire farm of machines? You are screwed if you have to boot up 100 machines manually from some image and fix them. Solution?

Create custom emergency shell, which starts in place of default one when you are dropped into emergency mode

Few ideas for implementing custom emergency shell:

  • Implement a custom emergency shell that does not need root password to provide you a root shell.
  • Implement a custom emergency shell that will ask some other (not root) password. This password cant be used when server is started normally and it will protect you a little against other people having access to console. As this password can be used only when dropped to emergency shell, you can share this password between relevant people.
  • You can implement custom emergency shell which will try to fix reason behind dropping into that shell f.e. run “fsck -y” against dirty file-systems, reboot and if dropped to emergency shell again, will wait for administrator intervention.

Below is explained how to set up custom emergency shell with custom password protection on RHEL6 and RHEL7.

Benefits:

  • Quick emergency access without knowing root password.
  • No need to boot from ISO images.
  • No need to manage root password, at least not for this emergency case.
  • You are still somewhat secured from other people who also have access to console (because no direct drop to root shell).
  • Nothing useful can be done with that shell/password, if server is started normally.
  • Configuration can be done with by just dropping 1-2 files into correct place. No change in existing files/procedures.

Negative sides:

  • You have to remember emergency mode password.
  • Fsck must still be executed manually.

Starting custom emergency shell instead of default one on RHEL6

RHEL6 – Override file is /etc/init/rcS-emergency.override . If it exists, its executed instead of default emergency shell /etc/init/rcS-emergency.conf .

So create file /etc/init/rcS-emergency.override with following contents:

# this is custom override file for emergency mode RC script
# located in /etc/init/rcS-emergency.conf

console owner

task

script
 . /etc/sysconfig/init
 plymouth --hide-splash || true
 [ -z "$EMERGENCY" ] && EMERGENCY=/sbin/sulogin

 /bin/echo "Enter emergency admin password to get shell"

 /root/emergency.sh

 exec $EMERGENCY
end script

Permissions 600 and owned by root:root is enough. When this file is in place,  your RHEL6 will start custom emergency shell, in this example “/root/emergency.sh” instead of “sulogin”.

Starting custom emergency shell instead of default one on RHEL7

RHEL7 – Override file is /etc/systemd/system/emergency.service . If it exists, its executed instead of default emergency service /usr/lib/systemd/system/emergency.service .

So create file /usr/lib/systemd/system/emergency.service with following contents:

# this is custom override file for emergency.service
# located in /usr/lib/systemd/system/emergency.service

[Unit]
Description=Emergency Shell
Documentation=man:sulogin(8)
DefaultDependencies=no
Conflicts=shutdown.target
Conflicts=rescue.service
Before=shutdown.target

[Service]
Environment=HOME=/root
WorkingDirectory=/root
ExecStartPre=-/bin/plymouth quit
#ExecStartPre=-/bin/echo -e 'Welcome to emergency mode! After logging in, type "journalctl -xb" to view\\nsystem logs, "systemctl reboot" to reboot, "systemctl default" or ^D to\\ntry again to boot into default mode.'
#ExecStart=-/bin/sh -c "/usr/sbin/sulogin; /usr/bin/systemctl --fail --no-block default"
ExecStartPre=-/bin/echo -e 'Enter emergency admin password to get shell'
ExecStart=-/bin/sh -c "/root/emergency.sh; /usr/bin/systemctl --fail --no-block default"
Type=idle
StandardInput=tty-force
StandardOutput=inherit
StandardError=inherit
KillMode=process
IgnoreSIGPIPE=no
SendSIGHUP=yes

Permissions 600 and owned by root:root is enough. When this file is in place,  your RHEL7 will start custom emergency shell, in this example “/root/emergency.sh” instead of “sulogin”.

Custom emergency shell

Here is example with very basic password-protected custom emergency shell which works both with RHEL6 and RHEL7.  Executing this shell is described above for both RHEL6 and RHEL7.

Create file, in this example “/root/emergency.sh” with following contents_

 while true; do

 read -s pass

 # hash
 hash=`/bin/echo "${pass}" | /usr/bin/sha256sum |/bin/awk '{print $1}'`
 # hash for comparison
 hash2="b5bb9d8014a0f9b1d61e21e796d78dccdf1352f23cd32812f4850b878ae4944c"
 
 if [ "${hash}" = "${hash2}" ];then

 # hash check ok, run shell
 /bin/echo "pass OK"
 exec /bin/bash

 else

 /bin/sleep 5
 /bin/echo "wrong pass, try again"

 fi

Permissions 700 and owned by root:root is good. Then others cant read the hash. Having such password hash comparison in the script is not a best coding example so feel free to customize. Just there are not so many solutions available if you are in emergency mode. Such emergency shell could be automatically updated on a regular basis like generated by puppet and applied monthly.

Usage

In order to emulate file-system issue and drop to the emergency shell, make small file-system somewhere, break it and add to /etc/fstab , /mnt mountpoint will do just fine. Reboot.

In the place of usual emergency shell prompt (Give root password or else!) you should see prompt “Enter emergency admin password to get shell”

In this example hash is made from the word “foo”, which gives you root shell. All other passwords will force you to wait 5 seconds and try again.

To reboot normally again, mount / filesystem r-w and comment out that line you added to /etc/fstab.

If you found this useful, say thanks, click on some banners or donate, I can always use some beer money.