Techniques for recovering almost-destroyed systems
Daniel Berrange
This is a clever way to recover when you’ve totally hosed your system, but still have a root shell around. Ok, in the “rm -rf /” scenario there’s not much you can usefullly do, but if, lets say, you’ve broken libc.so or some other critical part of the system, this could be a nice “get out of a jail free” card http://lambdaops.com/rm-rf-remains
Curtis Olson June 12, 2014 09:27
I knew about the echo * trick. Seemed like we’d periodically run into very unhappy machines back when I was a solaris admin. But there are some other fascinating hacks there too … hope I never need them. Back in the old days I compiled my linux kernel with finnish keyboard mappings (the default in that day) and then proceeded to edit (err try to edit) several critical system files resulting in near total system destruction … fortunately managed to trial and error and re-edit my way out of that one. I don’t remember much finnish any more though. Moral of the story, if you are an english speaker with an english keyboard, don’t try to run vi with finnish keyboard mappings, especially on files inside /etc, especially when you haven’t yet realized why everything is breaking so badly.
Rajeesh K V June 12, 2014 09:53
The most hairy and thrilling system recovery I ever read was by none other than Al Viro: http://yarchive.net/comp/linux/extreme_system_recovery.html
Eugene Crosser June 12, 2014 10:03
I sill occasionally need to use some (easier) of these tricks e.g. if root cannot be mounted and I end up in initramfs’s busybox.
Yet when it comes to netcat-ing the binaries, it’s time to give up and boot from some recovery media…
Edward Morbius June 12, 2014 22:52
So, I don’t feel like spinning up a VM to confirm, but wouldn’t those /proc/ exe entries keep the sources from being deleted?
Since the “problem” here was running ‘rm -rf /’, then any filehandles which were open **remain** open and should be accessible. Which means it might be useful to troll through /proc/<pid>/ and see what exectuables still remain.
A cheat, perhaps, but when you’re trying to recover from disaster, use everything you’ve got.
Imported from Google+ — content and formatting may not be reliable