Do not let yourself be fooled, a shell is like any other interpreter-based language, such as perl
. The only real difference is that a shell is interactive while interpreters usually execute scripts, one-time. Thus, as with any programming language, constructs may seem similar yet the syntax may greatly differ - in which case some of the fuss provided on this page will work on some shells but not on others.
For example, Linux distributions are moving away from bash
as the default shell, towards dash
which is leaning towards POSIX
compatibility. Apparently, bash
does cover dash
features but dash
has some of its own dash
-centric features, such that Linux now starts to depend on dash
instead of bash
.
Furthermore, every Linux distribution also packs its own set of tools, just like commands in a programming language, such that the fuss listed here may work on some systems, but not on others.
Learning or preferring one shell over the other, given that they are all feature-packed (or, conversely, feature-less), becomes a matter of religious dispute - just like choosing a Linux distribution, which gradually tends to void any sense of pragmatism and becomes difficult to categorize.
Lastly, some rudimentary commands, such as ps
(used for retrieving the process table) use differing syntax, for different set of features and spanning across different versions.
########################################################################### ## Copyright (C) Wizardry and Steamworks 2024 - License: MIT ## ## Please see: https://opensource.org/license/mit/ for legal details, ## ## rights of fair usage, the disclaimer and warranty conditions. ## ###########################################################################
ps ax | grep '[a]fpd' | awk '{print $1}'
would grab the PID of a process named afpd.
Provided you are in the same directory as the access_log
file:
for i in `cat access_log | awk '{ print $1 }'`; do dig -x $i +noall +short +answer; done | grep -v 'googlebot\|cloudshare'
which prints all hostnames that that IPs resolve to except the hostnames matching googlebot
and cloudshare
.
for i in `cat access_log | awk '{ print $1 }'`; do dig -x $i +noall +short +answer; done | grep '\.edu\|\.mil\|\.gov'
looks for the funny guys (will match .milkshake
TLDs as well).
In cases where the /dev/null
device is unavailable, pipe stderr
and stdout
together to the command true
:
ps ax 2&>1 | true
will suppress the output of ps ax
Supposing that data.txt
contains a value on every line:
cat data.txt | while read i; do perl -e "print log($i)/log(2)" && echo -en "\n"; done
Surprisingly easy:
cat file.txt | tail -r
Supposing that combined_data.txt
contains lines such as:
10 blah 10000 blah blah
the following will sort the above example in descending order by the first column:
cat combined_data.txt | sort -s -n -r
The result is:
10000 blah blah 10 blah
Input:
echo "hello world" | tr [:lower:] [:upper:]
Output:
HELLO WORLD
find . -name "*.rar" -exec unrar x -ad '{}' \;
The ad
switch creates a folder named after the rar
file. For example, the files in My Arhive.rar
will create a folder My Archive/
and place the files there.
A good time-stamp is, in order:
because when it is used with files, the stamp takes precedence in chronological order. One can generate this timestamp with:
date +%Y%m%d%H%M | tr '\n' ' '
where tr
is used to delete the newline produced by date
and add a space.
Only a y
or n
option breaks out of the loop.
while [ "$OPT" != "y" ] && [ "$OPT" != "n" ]; do echo -n "Yes or no? (y/n) : " read OPT case "$OPT" in y) echo "yes!" ;; n) echo "no!" ;; *) ;; esac done
To prevent parallel execution of scripts, the following snippet can be used:
# Acquire a lock. LOCK_FILE='/var/lock/myscript' if mkdir $LOCK_FILE 2>&1 >/dev/null; then trap '{ rm -rf $LOCK_FILE; }' KILL QUIT TERM EXIT INT HUP else exit 0 fi
where:
LOCK_FILE
points to a location where a directory myscript
can be created.
The command mkdir
is guaranteed to be atomic, such that out of multiple concurrent processes only one will succeed in creating the directory at /var/lock/myscript
. Once the directory is created, the lock is established, and the trap
instruction will bind to the KILL
, QUIT
, TERM
, EXIT
, INT
and HUP
, essentially ensuring that the code { rm -rf $LOCK_FILE; }
will be executed once those signals are raised (this includes normal script termination - via QUIT
) thereby releasing the lock.
The snippet can be used directly in any script just by setting the value of LOCK_FILE
to some chosen path where the script will be able to create a lock directory.
for arg in "$@"; do echo $arg done
Input:
HELLO='hello' WORLD='world' GREET=$HELLO" "$WORLD echo $GREET
Output:
hello world
Introduced by Anonymous on 4chan. For each number to sort, spawn a thread (process) and make the thread sleep for that number of seconds and receive the output in order. Noted for further reference.
Example, call with:
./sleepsort.bash 5 3 6 3 6 3 1 4 7
to sort the numbers.
#!/bin/bash function f() { sleep "$1" echo "$1" } while [ -n "$1" ] do f "$1" & shift done wait
bash
, that would be seconds. One can perhaps further reduce the numbers by dividing them with a large constant. That way, the thread would sleep for a very short period of time. When the numbers arrive, multiply the number with the same large constant to obtain the original numbers in sorted order.
The bash
history of commands can be reveled by issuing:
history
The number on the left column represent an index, and the command is listed in the right column. To repeat a certain command, issue:
!<index>
where index
is a number from the left column. For example:
!602
will execute the command referenced by the index 602
.
The history can also be searched using the keyboard shortcut Ctr+R or using the !?
operator followed by a partial history match.
If you would write:
!!
then the first !
will be substituted for the last command and the second !
will be substituted for the command arguments. You can use !!
as a shortcut to repeating the previous command as an alternative to using the up-arrow and pressing enter.
To execute the parameters of the previous command:
!*
which helps in cases where two commands were issued by accident.
As an alternative, you could write:
!!:gs/foo/bar
which will replace foo
by bar
every time foo
appears (hence the g
; s
stands for substitute).
One other thing to do is:
echo "!!" > script.sh
which will take the last command and arguments and place them in a file called script.sh
.
pv
is an utility that can intercept pipes and show various statistics during transfer.
pv --format="Elapsed Time:%t Transferred:%b Rate:%r Average:%a" /dev/disk3 > disk3.img
Unless pv
opens a file directly, it will not be able to determine its file size and will not be able to display an ETA
or what is commonly understood by a progress bar. Even when reading from a block device such as /dev/disk3
in the example above, pv
does not seem able to determine the size of the associated drive. In case pv
is not able to determine the file size, what is commonly understood by a progress bar will be in fact an activity monitor. This is why the example above chooses to show only statistics instead of a progress bar.
Given a string variable var
, the following:
echo ${#var}
prints out the length of the string var
.
Requires the nasm
disassembler:
echo -ne "\xeb\xe0" | ndisasm -u -
history -c && rm -f ~/.bash_history
rename 'y/A-Z/a-z/' *
cat file | awk '{ print length, $0 }' | sort -n | cut -d" " -f2-
Suppose you have a text file containing files listed one-by-one such as:
# removed in 2014-05-05 lib/images/fileicons/audio.png lib/plugins/acl/lang/hi/lang.php lib/plugins/acl/lang/id-ni/lang.php
then, the following command can be used to delete them:
grep -Ev '^($|#)' delete.txt | xargs -n 1 rm -vf
The command ignores any lines starting with either $
or #
.
ls -R | grep ":$" | sed -e 's/:$//' -e 's/[^-][^\/]*\//--/g' -e 's/^/ /' -e 's/-/|/'
A reoccurring problem is to find executables using the terminal. One solution is to use find
and set the perm
mask accordingly. The following command will find files (-type f
) with any executable permissions (-perm /11
):
find . -type f -perm /111 -exec echo '{}' \;
/111
indicates that any of the bit fields of the mask 0111
are set. In other words, for an executable file if any of the following flags are set:
-rwxr-xr-x ^ ^ ^ | | | OR OR OR
then the find
will match.
When a binary is found, find
will print out the executable filename (-exec echo '{}
').
Note that the executable flag can be set for any file or directory and it does not mean that the file is really an executable.
We use the -delete
predicate:
find . -name "*.txt" -delete
Suppose you have a file containing the following lines:
a b c
and you wish to reverse the order so that you obtain the lines:
c b a
Then you can use store the lines in a file called file.txt
and then issue the command:
tail -r file.txt
which will reverse the lines.
Usually the nice
and renice
commands are used to execute commands with lowered priority. However, the following can be used to run a shell script with a modified priority. For example, this is what the header of a script looks like:
#!/bin/sh renice 19 -p $$
The renice
command will schedule that script with the lowest priority.
The following command searches for all the zip files in all the sub-directories, decompresses them in the directory they are in while overwriting any existing file and finally removes the zip file itself.
find . -name '*.zip' | parallel -j4 'cd {//}; unzip -o "$(basename {/} .zip)"; rm {/}'
Suppose you have a hierarchy:
+ Music | +- A | +- A-10 Tank Killer.zip | | | +- Abandoned Places.zip | +- B | +- Baal.zip | | | +- Back to the Future Part II.zip | ...
after running the command in the Music
directory, you want to obtain the following directory hierarchy:
+ Music | +- A | +- A-10 Tank Killer | | + | | | | | + sound.mod | | | | | + print.wav | | | +- Abandoned Places | + | | | + abd.trk | | | + flow.wav | | | + mus.mp3 | +- B | +- Baal | | + | | | | | + crest.wav | | | +- Back to the Future Part II | + | | | + music.mp3 | | | + instruments | + | | | + guitars.pro | | | + percussion.trk | ...
Thus, given a directory hierarchy that contains archives (in this case, zip files) stored within sub-directories, the following command will recursively unzip all the files in the hierarchy by:
The command unarchives the files in parallel (the -j4
indicates 4
different processes) and works for directory trees of any depth.
Using parallel:
find . -iname '*.zip' | parallel -j4 'cd {//}; mkdir -p "$(basename {/} .zip)"; unzip {/} -d "$(basename {/} .zip)"; rm {/}'
and without parallel:
find . -iname '*.zip' -print0 | xargs -0 -I{} sh -c 'd=$(basename "{}" .zip); mkdir -p "$d"; unzip "{}" -d "$d";'
To use RAR / unrar
instead, issue:
find . -name '*.rar' | parallel -j4 'cd {//}; mkdir "$(basename {/} .rar)"; unrar x {/} "$(basename {/} .rar)"; rm {/}'
To replace a string in multiple files you can use find
and perl
:
find . -type f -exec perl -p -i -e 's/was\.fm/grimore\.org/g' '{}' \;
this command will search all files in the path and replace the string was.fm
with grimore.org
.
This can be performed in several ways depending on the tools available.
This can be accomplished by using tar
and the pipe operator:
tar -cf - /path/to/dir | ssh -C remote_server 'tar -xvf - -C /path/to/remotedir'
where:
/path/to/dir
is the path to the directory to transfer on the local server/path/to/remotedir
is the destination directory on the remote server
Another alternative is to use nc
(netcat) and cpio
. First on the receiving machine, you would write:
nc -l -p <port> | gunzip | cpio -i -d -m
where <port>
represents any port number.
and then on the destination machine, you would write:
find . -type f | cpio -o | gzip -1 | nc <destination> <port>
where <destination>
is the hostname or IP address of the receiving machine and <port>
represents the port number that you used on the receiving machine.
The following command will remove all duplicate lines from input.txt
and output the result to output.txt
while preserving the order of the lines:
cat -n input.txt | sort -k2 -k1n | uniq -f1 | sort -nk1,1 | cut -f2- > output.txt
The command performs the following operation:
cat -n input.txt
- will open input.txt
and number the lines and pass on the result.sort -k2 -k1n
- will consider the characters from the second column to the end of the line first, then the first column but numerically when sorting.uniq -f1
- will ignore the first column.sort -nk1,1
- will sort the first column numerically while getting the sort order towhead it was originally. cut -f2-
will remove the line numbers that were inserted by cat
.
The perl rename
command combined with find
is the easiest way. The following command converts from uppercase to lowercase:
find . -depth -print -execdir rename -f 'y/A-Z/a-z/' '{}' \;
while the following command reverses the operation:
find . -depth -print -execdir rename -f 'y/a-z/A-Z/' '{}' \;
the rename
part is self-explanatory but the magic lies with find
: the -depth
parameter specifies that files should be changed before directories - this is very important on case-sensitive file-systems because one may end-up changing a directory name before the files inside it and while the command is recursing, the directory name may not be found and this is what -depth
is for.
The command:
paste codes.txt status.txt | awk -F"\t" '{print $1,$2}'
will print lines from codes.txt
and then lines from status.txt
delimited by a tab character. The sequence will then be piped into awk
that will use the tab character as separator and print the first and then the second column received from the paste
command.
Appending text to a file is usually achieved with:
echo "append this string to the file" >> test.txt
which adds the string at the end of the file.
In order to prepend some text to a file, the following can be used:
echo "prepend this string to the file" | cat - test.txt | cat - > test.txt
which adds the string at the top of the file.
The loop checks the date against the modification timestamp and sleeps while the difference is smaller than 10 seconds.
while [ $(( $(date +%s) - $(stat -c %Y FILENAME) )) -lt 10 ]; do sleep 1 done
The following command:
echo `yes +|head -50`|tr -d ' '
will repeat the +
character 50 times.
Executing:
time read
and then aborting with Ctrl+D will print out the amount of time waited.
In order to remove all files without an ace
, zip
or rar
extension, issue:
rm !(*.ace|*.zip|*.rar)
In oder to get the top RAM consuming processes:
ps -eo pmem,pcpu,pid,args | tail -n +2 | sort -rnk 1 | head
To get the top CPU consuming processes:
ps -eo pmem,pcpu,pid,args | tail -n +2 | sort -rnk 2 | head
The task is to copy a file, say /home/root/writeup.txt
to the directories:
/var/www/site_1.tld/data/
/var/www/site_2.tld/data/
/var/www/site_3.tld/data/
without looping - since the standard solution would be to loop over all the directories and then on every iteration copy the file.
This can be accomplished, in this scenario, using xargs
:
echo /var/www/*/data/* | xargs -n 1 cp /etc/root/writeup.txt
Goto / exception handling can be implemented in a shell by using trap
to install a handler and then deliver a signal to the current process:
#! /bin/sh trap '{ echo "Good day!"; }' HUP # do stuff kill -s HUP $$
The script will install the (inline) handler function { echo "Good day!"; }
and bind to the HUP signal. The kill
command will then deliver the HUP
signal to the shell process (itself) which will then call the handler function.
The mechanism can be used to implement some form of goto or exception continuations - for instance, one could just terminate the shell script inside the handler function instead of just printing out Good day!
. For instance, the following script:
#! /bin/sh trap '{ # Check if temporary file exists. if [ -f /tmp/tmp.EERfX5aeY8 ]; then echo "temporary file exists" # Terminate. exit fi }' HUP # Raise signal. kill -s HUP $$ # Code not reached. echo "END"
will:
HUP
signal that will process an inline function.HUP
signal and the inline function will execute.exit
, the last line echo "END"
will never executeThe goto trick is illustrated in the following scripts:
Given the filesystem structure:
top + | +---+ list.txt | +---+ stuff +---- f1.txt +---- f2.jpg +---- f4.gif +---- f5.doc
and a file containing:
f2.jpg f5.doc
and would like to remove all the files in the stuff
folder that are contained in the list file, then the following command ran from the top
directory:
find stuff -type f -print0 | grep -zZ -f list.txt | xargs -0 rm
where:
stuff
is the directory containing the files,-type f
specifies that only files should be matched,-print0
use a null byte as a terminator (in order to be compatible with files containing spaces),-zZ
makes grep treat all input as being terminated with null bytes,-f list.txt
will instruct grep to read the file list.txt
line-by-line,rm
removes the files
will remove all the files in the stuff
directory that match any entry line-by-line in list.txt
.
Conversely, if all the files in stuff
have to be removed that are not contained in the list file list.txt
, simply add the -v
option to grep in the filter chain.
The following command will generate an alpha-numeric string from the real random number device that would be suitable to be used as a password:
egrep -m 10 -ao '[\x20-\x7E]{1}$' </dev/random | tr -d '\n' | wc -c
where:
-m 10
matches printable characters (all ASCII characters starting from space 20
octal and up to tilde symbol 7E
in octal) from /dev/urandom
,-a
instructs grep to treat even binary input as characters,-o
instructs grep to only print matches,tr
will be used to delete space characters, andwc -c
will count the characters for verification.
On OSX you can prevent the password to be echoed to the terminal by enqueueing the pbcopy
command, ie:
egrep -m 20 -ao '[\x20-\x7E]{1}$' </dev/random | tr -d '\n' | pbcopy
which will then place the password in the clipboard.
More than often, as a command runs on the command line, it becomes perceivable that the command will take a long while to terminate. In such cases, in case the connection is severed, the command will be terminated as well since it is attached to the terminal being used for the connection.
In such situations, the solution is to background the process, disown the process from the current terminal and, perhaps, re-attach the process using a terminal multiplexer such as tmux
.
With a command running in the current terminal, the operations should be:
bg
to background the process,disown
command with the PID of the background command as a parameter, for instance disown 3142
At this point, the connection can be severed and the process will continue running in the background. However, there is no way of capturing the command output, if any. One solution is to use a terminal multiplexer to re-attach the command and then be able to receive output or send input:
tmux
to create a new session,tmux
, issue reptyr -s PID
where PID
is the process identifier of the command running in the backround
The command will now execute in the TMux session. Now, to background the entire tmux session with the process running, press the key-combination Ctrl+B to open up the tmux command line and type: :detach
.
Later on, to check up on the command's progress, issue the command tmux attach
.
echo 'AGsg4SKKs74s62#' | sed 's/./&\n/g' | shuf | tr -d "\n"
awk -v min=X -v max=Y 'BEGIN{srand(); print int(min+rand()*(max-min+1))}'
where:
X
is the low number,Y
is the high numberfind . -type f -exec grep -Iq . {} \; -print
find -name \*.txt | while read f; do tail -n1 $f | read -r _ || echo >> $f; done
where:
*.txt
is the pattern to search for.
Using su
, the following script can be ran as root
but will execute the remainder of the script after the if
clause as the user myanonamouse
:
# Restart the script as an user via su. if [ "$(id -u)" -eq 0 ]; then # exec replaces current shell exec su myanonamouse "$0" -- "$@" # this line will not be reached fi # this will run as user "myanonamouse"
Using opendns.com
that is programmed to return the IP address of the client performing the lookup:
dig +short myip.opendns.com @resolver1.opendns.com
The following command:
install -D -o www-data -g www-data /dev/null /var/log/razor.log
will create the file /var/log/razor.log
and set the ownership to the user www-data
and group www-data
. The -D
flag will ensure that in case the path /var/log/
is missing some directories, then the path will be created.
The following line will print 10 successive dots:
yes "." | head -n 10 | xargs | tr -d ' '
The Unix utility find
provides a results placeholder that holds the filename that has been found matching the filters provided to find. Here is an example that searches the current directory recursively for files ending in mp4
:
find . -type f -name '*.mp4' -exec echo "{}" \;
such that the command after the exec
parameter will be executed for all files, with the special curly-bracers symbol {}
expanding during execution to the full path of the file as reported by the find
utility.
Remarkably, as feature-packed as Unix tools are, the results placeholder is deceptively straightforward and without features. For example, one of the most common tasks is to perform some sort of string substitution on the results in case the file must be moved or manipulated in any way.
In order to use the file name without further bothering about the find
utility, simply pass the special placeholder symbol {}
as the argument to a shell, for example, a bash shell and then for the command being executed, the contents of {}
will be transferred to the first command-line argument $0
. Here is an example that changes the extension of all images recursively from the current directory from jpg
to png
:
find . -type f -name '*.jpg' -exec bash -c 'mv "$0" "${0//jpg/png}"' {} \;
Whilst the command invocation:
find . -type f -name '*.jpg' -exec ... \;
is responsible for finding files ending with the jpg
extension, for every file found, the following command is executed, that can be disected:
bash -c 'mv "$0" "${0//jpg/png}"' {} + + + + + | | | | | +-----+ | | | from find; expands to the path of the file execute a | | command | | | | bash substitution acting on parameter $0, | | changing "jpg" to "png" | | | $0 is the first parameter | passed to bash and holds | the exact contents of {}
Something that seems counter-intuitive is that the command passed to the bash -c
command-line parameter is quoted using single-quotes, which would imply no substitution:
find . -type f -name '*.jpg' -exec bash -c 'mv "$0" "${0//jpg/png}"' {} \;
Interestingly, if one were to issue the same command with double-quotes instead:
.................................. bash -c "mv \"$0\" \"${0//jpg/png}\"" {} \;
then the command would not run properly because while issuing the "find" command itself, the shell would substitute the parameters within the quotes. That being said, it only makes sense to execute the "inner-most" command whilst escaping it with single quotes:
.................................. bash -c 'mv "$0" "${0//jpg/png}"' {} \;
such that substitution only occurs when the command passed to the -c
bash parameter actually executes.
An equivalent to the alarm(2)
call in systems programming can be created by simply placing a process in the background and then killing the process in order to rearm the alarm or wait until the time elapses in order for the command to run.
# alarm(2) function alarm { sleep $1 # this is the command to be executed after the elapsed time echo "EXEC" } # ensures that any pending alarm is cleared upon termination of the current script ALARM_PID=0 trap '{ test $ALARM_PID = 0 || kill -KILL $ALARM_PID; }' KILL QUIT TERM EXIT INT HUP # the parent program consists in a process that runs indefinitely while "..."; do # when the alarm must be re-armed: # * kill the previous alarm # * reschedule the alarm into the future if [ -d /proc/"$ALARM_PID" ]; then kill -9 $ALARM_PID fi alarm "5" & ALARM_PID=$! done
The actual code to be executed is to be found within the alarm
function, namely anything after the sleep
line. Fortunately, SIGKILL
can interrupt sleep
and additionally abort the program such that the command will not be executed when the alarm is just rearmed.
Note that the expression $!
might be a bashism but there are ample equivalents such as using pidof
, etc, to determine the PID of the last backgrounded process in case the scheme has to be used within a strict POSIX shell.