Three Wise Men

Linux Tips Aggregated – 1

So far I posted ten Linux tips on Twitter as @BigWhale,  now reposting them here with a little more explanation and insight why things work that way. I am writing tips mostly out of my own experience about something I am currently working on. I will touch basics of most Unix like operating systems, bash shell and Unix utilities. All the tips I publish are tested on my computer and they work for me. Before using them on some data that you actually need, make sure that you test them on your computer with your operating system and with your environment. Data loss is something that you want to avoid and I don’t want to take any part in it. ;) All the feedback is appreciated and more than welcome.

Just to be sure, another friendly warning: Some of this commands are and can be used in a lot of very destructive ways. Make sure you are always working as non-root user (unless the task requires root access) and that you always double check what you wrote before you press enter. Seriously, first hand experience here … ;)

#1 – Creating remote disk images

There is a server running at some remote location and you need exact copy of the server on your computer. It is either 3AM in the morning and the place is not accessible or it is just too far away to go there. By using dd(1) and ssh(1) commands you can create an image of hard drive in the remote computer.

$ sudo dd if=/dev/sda | ssh user@host "dd of=backup_image.img"

How does this work? First you have to login to remote server¹. Then you execute dd command with sudo. We need sudo because dd requires read permission for physical device /dev/sda. Like most Unix commands dd works with standard input and standard output. We told dd to use /dev/sda as input file and since we didn’t specify the output file dd will dump all the data to standard output. Then we use pipe ( | ) to redirect this output to another program. In our case this is the ssh program which we gave instructions to connect to user@host and execute dd command telling it to use backup_image.img for output file and because we didn’t specify the input file dd will expect data on standard input. Pipe conveniently connected output of one program to the output of another program. The fact that this program is running on some other computer doesn’t really bother us. SSH is quite capable of piping input and output over the network.

#2 – Finding lost files without Permission Denied warnings for non-root users

Finding files on your hard drive can sometimes be painful. Our geek friend NixiePixel over at LinuxHaxor.com talked about this subject some time ago. The problem with find is that it will display a lot of error messages if it can’t access a directory so in the end you’ll need to look for your search result between all those error messages. Instead, do this:

$ find / -iname "niksiii_nekkid_*.jpg" 2> /dev/null

You successfully redirected all the error messages to /dev/null and your result will be easier to read. The secret lies just in two characters “2>”. Greater than character will redirect the output to some other file or device. By adding 2 in front of it you tell Bash shell that it should redirect only messages that are sent to standard error output. At last but not least, /dev/null is a special device that serves as a sinkhole. Everything you throw in it is lost forever.

#3 – Having trouble figuring out file sizes

Some thirty or forty years ago when Unix was still in its infancy and hard drives came in the sizes of an average washing machine or kitchen refrigerator, files were relatively small. Up to a megabyte or so. Everything that was more than that was considered big. One 5.25 inch floppy disk had capacity of 1.2 megabytes. Peanuts. So displaying file sizes in bytes was reasonable. Increase in file size by one kilobyte could mean that you had to carry an extra floppy disk with you. Nowadays nobody cares about bytes or kilobytes even megabytes are becoming insignificant. So if ls(1) told you that one file is 2734897245 bytes in length you’ll have to look very closely, perhaps you’ll even group the numbers in three to figure out how big this file really is. GNU conveniently added -h flag to some of the commands that are dealing with displaying sizes of files and disk drives.

So, why don’t you try the human readable format and add the -h switch, like this:

$ ls -lh

$ du -sh *

$ df -h

See the results for yourself, before:

$ df

/dev/sdb3            272575028 128962480 129766528  50% /home

/dev/sda2            204445328  40452624 163992704  20% /media/Pictures

/dev/sdc1            976643072 284948864 691694208  30% /media/Big

And after:

$ df -h

/dev/sdb3             260G  123G  124G  50% /home

/dev/sda2             195G   39G  157G  20% /media/Pictures

/dev/sdc1             932G  272G  660G  30% /media/Big

Pretty isn’t it? Random trivia: df has -H switch that will do the same thing but it will use the powers of 1000 instead of 1024 for calculating kilo, mega and gigabytes.

#4 – List top ten programs by memory usage

Something is eating and chewing away your memory and you are not sure what? Again, the power of the shell comes to the rescue.

$ ps --no-headers -eopid,rss,comm | sort -bk2,2nr | head -10

Wow, neat, but how? Easy, ps(1) lists running processes and with –no-headers switch we tell it not to display headers since they mess up sorting. Switch -e will select all processes and with -o we select what information to display about processes. For this case we selected process id, resident set size – physical non-swapped memory that process used in kilobytes and command name – the name of the executable file without any parameters. Then we piped that output to sort(1) which takes care of sorting. Switches for sort were: -b to tell sort that it should skip leading blank characters, -k to tell it which columns should be used as sort keys and -r to reverse the output. At last we pipe this output to head(1) telling it to display only first ten lines and discard the rest of the input. And there you have it: top ten programs by memory usage.

#5 – Shutting down your system

Did you know that you can shutdown your system without clicking? ;) Of course you can! That’s why we have shutdown(8) command.

$ shutdown -h -t now

When I posted this tip I was quickly corrected that this is not the correct command. I double checked and noticed that this form works on Slackware 12 (Yes, some people still run it, not me tho …) but for Ubuntu you’ll have to use the following command:

$ shutdown -h now

In both cases the -h switch will also power off your computer.

#6 – List all files in current directory and sort them by modification time

This one is fairly easy ls(1) provides us with all the switches we need.

$ ls -lat

The switches are from left to right: -l long display, -a display all files including hidden files², -t sort listing by file modification time, newest first.

#7 – Change directory and then change it back to previous directory

You just changed directory and realized that you need to go back to previous directory. Because you forgot to check something or whatever? See dee minus to the rescue!

bigwhale@thefish:/opt/test/new/software/with/a/lot/of/directories$ ls -l

total 0

-rw-r--r-- 1 root root 0 2009-09-12 00:55 new_file.txt

bigwhale@thefish:/opt/test/new/software/with/a/lot/of/directories$ cd /opt/test/other/software/with/less/directories/

bigwhale@thefish:/opt/test/other/software/with/less/directories$ ls -l

total 0

-rw-r--r-- 1 root root 0 2009-09-12 00:56 old_file.txt

bigwhale@thefish:/opt/test/other/software/with/less/directories$ cd -

/opt/test/new/software/with/a/lot/of/directories

bigwhale@thefish:/opt/test/new/software/with/a/lot/of/directories$ ls -l

total 0

-rw-r--r-- 1 root root 0 2009-09-12 00:55 new_file.txt

bigwhale@thefish:/opt/test/new/software/with/a/lot/of/directories$

What’s going on? When you run cd – bash checks if there is an OLDPWD environmental variable set and changes to that directory. When directory change is complete OLDPWD is rewritten with the previous directory. Handy, isn’t it? A bonus tip, typing just cd and pressing enter will take you to your home directory.

#8 – Finding files that were changed in the last 5 minutes

Recently some program changed a file in your home directory and you have no idea which file it was, you just know that it wasn’t more than five minutes ago. Easy life with the find(1) command.

$ find ~ -type f -mmin -5

Executing find with ~ as first parameter means that we are searching only in our home directory. Parameter -type f will tell find that we are searching only for regular files and -mmin -5 means that we are looking for files that were modified at most five minutes ago.

#9 – Listing all the shares on a remote computer

You have one more computer in the local network and you want to check if there are any SMB shares on that computer? The easiest way to do this is by using smblclient(1).

$ smbclient -L 192.168.1.1

SMB or CIFS shares are directories that are shared by Microsoft Windows or Samba.

#10 – Duplicating content of your computer to a remote location using rsync

When tip #1 is too much for you and you only need actual files from remote computer you might want to try rsync(1), it can do wonders.

$ sudo rsync -avHAXe ssh --exclude "/proc" --exclude "/dev" --exclude "/sys" / user@host:/target/dir

Running rsync with sudo gives us access to all the files on the computer. The parameters to rsync are: -a for transfering all the attributes together with file, -v to be more verbose and -HAX to preserve hard links, access control lists and extended attributes. -e ssh will tell rsync to use ssh for remote connection. We take all the files from root directory with the obvious excludes and sync them to user@host in /target/dir

Until next time, more tips are coming soon, just follow @BigWhale on Twitter.

1. This could also be done the other way around, but since we need root privileges on remote machine, this is probably easier way. [↩]
2. The dot files, files with dot as the first character in the name [↩]

Related posts:

1. Linux Tips Aggregated – 2
2. Linux Backup – Poor man’s solution
3. Cloning Ubuntu Linux – The easy way
4. Network boot and install of Ubuntu Linux
5. Losing your password in Ubuntu Linux