Shellshock Warning: Even after patching, your old vulnerable bash binary could be “resurrected” from memory 10

I received an email today from a friend and fellow Linux sysadmin, Todd Lyons, informing me of a very sneaky way to exploit the Shellshock vulnerability under what he calls “just the wrong conditions.”

Here’s how his email starts out:

With your recent blog post, you might also want to advise people to further look around their systems to see if they need to restart any services. If non-trusted users have shell accounts, you can still access the old (replaced) bash binary under just the wrong conditions:

Todd went on to share with me bits of an IRC chat where he explains the exploit further:

I found a sneaky way, not good for uptime :-(, to exploit the shellshock vulnerability.
if you do lsof | grep bash, and any of them are state “deleted”, it’s simple to merely call it.
A good example is mysql, which gets started by “/bin/sh /usr/bin/mysqld_safe”. If your /bin/sh is symlinked to /bin/bash (yes
for RH/CentOS, no for Debian/Ubuntu, probably not for most unixes), then it works because the old deleted one is still in memory.
# lsof | grep bash | grep deleted
mysqld_sa 20470 root txt REG 8,1 903336 268382 /bin/bash (deleted)
# env var='() { ignore this;}; echo vulnerable’ /proc/20470/exe -c /bin/true

Todd finished up his email to me by stating that he wasn’t sure that it’s “exploitable” to get root elevation, but that if a sysadmin doesn’t know about this potential method of “resurrecting” the deleted bash binary that’s still in memory, the system could still technically be vulnerable, even after patching.

I tested this on my patched Fedora 12 test system, and found exactly what Todd predicted:

# lsof | grep bash
mysqld_sa  1714      root  txt       REG      253,0     861128   14303268 /bin/bash (deleted)
bash      24123      root  cwd       DIR      253,0      12288    3768321 /etc
bash      24123      root  rtd       DIR      253,0       4096          2 /
bash      24123      root  txt       REG      253,0    2280202   14303310 /bin/bash

Right there, as process 1714, the deleted /bin/bash binary was still in memory. And even though my system had been patched, I was able to produce a “vulnerable” output with:

# env var='() { ignore this;}; echo vulnerable' /proc/1714/exe -c /bin/true

Restarting the service (in this case, mysqld) did the trick:

# service mysqld restart
Stopping MySQL:                                            [  OK  ]
Starting MySQL:                                            [  OK  ]

# lsof | grep bash
mysqld_sa  7285      root  txt       REG      253,0  2280202   14303310 /bin/bash

I also checked on a production CentOS 6 box that I’d patched with the “official” patch from the repositories:

]# lsof | grep bash
sh         5941      root  txt       REG              253,0    938832     921278 /bin/bash (deleted)
mysqld_sa  7782      root  txt       REG              253,0    938832     921733 /bin/bash (deleted)
sh        14553      root  txt       REG              253,0    938832     921278 /bin/bash (deleted)

Yep! Three naughty binaries right at the top of the list. I had to restart MySQL on this server too, and kill and restart the shell scripts that were attached to the other two processes.

So, as Todd suggests, after patching your system against Shellshock, you should check to see if anything in memory is still “holding on” to the old binary. And if they are, kill or restart the process to banish your old vulnerable binary forever.

Another solution, if you’re willing to reset your uptime, is simply rebooting your server.

Of course, the level of threat of an external attack using this vector is unlikely. Todd notes that “it’s limited to users having local shell accounts already, so the remote aspect of it is removed. But the vulnerable version is still in memory, and a local vulnerability is bad too.”

  • Pingback: Shellshock: bash manuell patchen | slacc()

  • Pingback: Eric Stone | Eric M. Stone | Eric Michael Stone | Random Thoughts, Whims and Fancies()

  • Obviously elevated privliges are required to write to a mysql /proc/ in which case you can do anything you like if you have them, but thanks for sharing this info nonetheless.

    With regards to reloading a new “home brew” version of /bin/bash it would have been prudent to do some testing on your staging environment before releasing it into production, and considering the pervasiveness of the bash shell in Unix/Linux systems, a thorough test should include a reboot of the server which would have prevented this problem from occurring in the first place.

    Don’t want to nit-pick your work too much as the info here is very educational. Thanks.

    • Arthur: You’re right about staging environments and testing… but we both know that the majority of those who babysit servers probably don’t do it enough (or at all). And there’s so much pride associated with uptimes among some Linux admins, that many eschew reboots (I don’t – I upgrade kernels and reboot all the time). I don’t think you’re nit-picking at all. You’re making some very valid points. Unfortunately, there are too many who will ignore them. 🙂

  • Hi Steve,

    thanks for the great articles! 🙂 Now, regarding this one, there’s a cool tool for debian boxes, called “check-restart” (it’s in the package “debian-goodies”), and some others similar tools as well (both for RH and Debian-based systems), that enable you to check for this problem in a fast and simple way, along with listing the services that should be restarted. In short, it’s dead simple, with very little manual work required, so it scales well 🙂


    • Thanks, Milan. Great tip. For users of older Fedora systems, if they have the yum-utils package installed, they can run needs-restarting to see what needs it. On newer Fedora-based systems (including RHEL6) they can use yum-plugin-ps.

  • The danger of shellshock was that things were run by /bin/sh (actually /bin/bash) when accessed e.g. via the webserver and that could be exploited to execute things on the system as the webserver user ID. If you’re already logged in and able to do lsof to find deleted bash binaries, then you’re already superuser and you’re wasting your time by doing these things.

  • I do not think this is a security issue.
    Shellshock is a threat because it allows you to execute commands, where you previously couldn’t.
    If you already have access to choose to run bash from a non-default location, you already have the access that shellshock would give you. 

    Or in other words. Shellshock executes evil commands upon startup of the bash shell, in the context of the user who executes bash. If YOU execute the shell through /proc/*/exe, then you are only hacking yourself.
    To use this to hack some other user, you’d need to get that user to somehow execute /proc/*/exe instead of /bin/bash. And if you have a way to make the user execute a command of your choosing, why not just have them execute /tmp/evilbackdoor instead?.

  • Sreejiht

    Hey we have pushed the patch to many boxes, but now found in lsof many bash is in deleted state, do i have to worry about this, or should i go for reboot of all boxes where the patch has been pushed

    bash 16511 ubuntu txt REG 202,1 959120 393327 /bin/bash (deleted)
    bash 17965 system txt REG 202,1 959120 393327 /bin/bash (deleted)
    bash 20208 root txt REG 202,1 959120 393327 /bin/bash (deleted)
    bash 20473 system txt REG 202,1 959120 393327 /bin/bash (deleted)
    bash 20585 root txt REG 202,1 959120 393327 /bin/bash (deleted)
    bash 21922 ubuntu txt REG 202,1 959120 393327 /bin/bash (deleted)
    bash 22943 ubuntu txt REG 202,1 959120 393327 /bin/bash (deleted)

    Or go kill all such kind of process which are already in delete state, not sure will this affect any current process running in the box.

    • Rebooting is always recommended, and is safer than just killing processes.