Do not let yourself be fooled, a shell is like any other interpreter-based language, such as perl. The only real difference is that a shell is interactive while interpreters usually execute scripts, one-time. Thus, as with any programming language, constructs may seem similar yet the syntax may greatly differ - in which case some of the fuss provided on this page will work on some shells but not on others.

For example, Linux distributions are moving away from bash as the default shell, towards dash which is leaning towards POSIX compatibility. Apparently, bash does cover dash features but dash has some of its own dash-centric features, such that Linux now starts to depend on dash instead of bash.

Furthermore, every Linux distribution also packs its own set of tools, just like commands in a programming language, such that the fuss listed here may work on some systems, but not on others.

Learning or preferring one shell over the other, given that they are all feature-packed (or, conversely, feature-less), becomes a matter of religious dispute - just like choosing a Linux distribution, which gradually tends to void any sense of pragmatism and becomes difficult to categorize.

Lastly, some rudimentary commands, such as ps (used for retrieving the process table) use differing syntax, for different set of features and spanning across different versions.

###########################################################################
##  Copyright (C) Wizardry and Steamworks 2024 - License: MIT            ##
##  Please see: https://opensource.org/license/mit/ for legal details,   ##
##  rights of fair usage, the disclaimer and warranty conditions.        ##
###########################################################################

ps ax | grep '[a]fpd' | awk '{print $1}'

would grab the PID of a process named afpd.

Provided you are in the same directory as the access_log file:

for i in `cat access_log | awk '{ print $1 }'`; do 
  dig -x $i +noall +short +answer; 
done | grep -v 'googlebot\|cloudshare'

which prints all hostnames that that IPs resolve to except the hostnames matching googlebot and cloudshare.

for i in `cat access_log | awk '{ print $1 }'`; do 
  dig -x $i +noall +short +answer; 
done | grep '\.edu\|\.mil\|\.gov'

looks for the funny guys (will match .milkshake TLDs as well).

In cases where the /dev/null device is unavailable, pipe stderr and stdout together to the command true:

ps ax 2&>1 | true

will suppress the output of ps ax

Supposing that data.txt contains a value on every line:

cat data.txt | while read i; do perl -e "print log($i)/log(2)" && echo -en "\n"; done

Surprisingly easy:

cat file.txt | tail -r

Supposing that combined_data.txt contains lines such as:

10 blah
10000 blah blah

the following will sort the above example in descending order by the first column:

cat combined_data.txt | sort -s -n -r

The result is:

10000 blah blah
10 blah

Input:

echo "hello world" | tr [:lower:] [:upper:]

Output:

HELLO WORLD

find . -name "*.rar" -exec unrar x -ad '{}' \;

The ad switch creates a folder named after the rar file. For example, the files in My Arhive.rar will create a folder My Archive/ and place the files there.

A good time-stamp is, in order:

1. year
2. month
3. day
4. hour
5. minutes
6. seconds

because when it is used with files, the stamp takes precedence in chronological order. One can generate this timestamp with:

date +%Y%m%d%H%M | tr '\n' ' '

where tr is used to delete the newline produced by date and add a space.

Only a y or n option breaks out of the loop.

while [ "$OPT" != "y" ] && [ "$OPT" != "n" ]; do
  echo -n "Yes or no? (y/n) : "
  read OPT
  case "$OPT" in
    y)
      echo "yes!"
      ;;
    n)
      echo "no!"
      ;;
    *)
      ;;
  esac
done

To prevent parallel execution of scripts, the following snippet can be used:

# Acquire a lock.
LOCK_FILE='/var/lock/myscript'
if mkdir $LOCK_FILE 2>&1 >/dev/null; then
    trap '{ rm -rf $LOCK_FILE; }' KILL QUIT TERM EXIT INT HUP
else
    exit 0
fi

where:

• LOCK_FILE points to a location where a directory myscript can be created.

The command mkdir is guaranteed to be atomic, such that out of multiple concurrent processes only one will succeed in creating the directory at /var/lock/myscript. Once the directory is created, the lock is established, and the trap instruction will bind to the KILL, QUIT, TERM, EXIT, INT and HUP, essentially ensuring that the code { rm -rf $LOCK_FILE; } will be executed once those signals are raised (this includes normal script termination - via QUIT) thereby releasing the lock.

The snippet can be used directly in any script just by setting the value of LOCK_FILE to some chosen path where the script will be able to create a lock directory.

for arg in "$@"; do
  echo $arg
done

Input:

HELLO='hello'
WORLD='world'
GREET=$HELLO" "$WORLD
echo $GREET

Output:

hello world

Introduced by Anonymous on 4chan. For each number to sort, spawn a thread (process) and make the thread sleep for that number of seconds and receive the output in order. Noted for further reference.

Example, call with:

./sleepsort.bash 5 3 6 3 6 3 1 4 7

to sort the numbers.

#!/bin/bash
function f() {
    sleep "$1"
    echo "$1"
}
while [ -n "$1" ]
do
    f "$1" &
    shift
done
wait

• The algorithm sleeps for the numeric number of the values to sort - in bash, that would be seconds. One can perhaps further reduce the numbers by dividing them with a large constant. That way, the thread would sleep for a very short period of time. When the numbers arrive, multiply the number with the same large constant to obtain the original numbers in sorted order.
• This is an instance where a race-condition can be used in a beneficial way. Another example for race conditions that can be used in a beneficial way can be found on the project Alexandria page. The Oracle sends a trigger message on a public channel and the bookshelves are given a certain timeout to respond. Based on the time it takes for that answer to arrive from any bookshelf, the Oracle can tell whether the result is more relevant than other results.

The bash history of commands can be reveled by issuing:

history

The number on the left column represent an index, and the command is listed in the right column. To repeat a certain command, issue:

!<index>

where index is a number from the left column. For example:

!602

will execute the command referenced by the index 602.

The history can also be searched using the keyboard shortcut Ctr+R or using the !? operator followed by a partial history match.

If you would write:

!!

then the first ! will be substituted for the last command and the second ! will be substituted for the command arguments. You can use !! as a shortcut to repeating the previous command as an alternative to using the up-arrow and pressing enter.

To execute the parameters of the previous command:

!*

which helps in cases where two commands were issued by accident.

As an alternative, you could write:

!!:gs/foo/bar

which will replace foo by bar every time foo appears (hence the g; s stands for substitute).

One other thing to do is:

echo "!!" > script.sh

which will take the last command and arguments and place them in a file called script.sh.

pv is an utility that can intercept pipes and show various statistics during transfer.

pv --format="Elapsed Time:%t Transferred:%b Rate:%r Average:%a" /dev/disk3 > disk3.img

Unless pv opens a file directly, it will not be able to determine its file size and will not be able to display an ETA or what is commonly understood by a progress bar. Even when reading from a block device such as /dev/disk3 in the example above, pv does not seem able to determine the size of the associated drive. In case pv is not able to determine the file size, what is commonly understood by a progress bar will be in fact an activity monitor. This is why the example above chooses to show only statistics instead of a progress bar.

Given a string variable var, the following:

echo ${#var}

prints out the length of the string var.

Requires the nasm disassembler:

echo -ne "\xeb\xe0" | ndisasm -u -

history -c && rm -f ~/.bash_history

rename 'y/A-Z/a-z/' *

cat file | awk '{ print length, $0 }' | sort -n | cut -d" " -f2-

Suppose you have a text file containing files listed one-by-one such as:

# removed in 2014-05-05
lib/images/fileicons/audio.png
lib/plugins/acl/lang/hi/lang.php
lib/plugins/acl/lang/id-ni/lang.php

then, the following command can be used to delete them:

grep -Ev '^($|#)' delete.txt | xargs -n 1 rm -vf

The command ignores any lines starting with either $ or #.

ls -R | grep ":$" | sed -e 's/:$//' -e 's/[^-][^\/]*\//--/g' -e 's/^/ /' -e 's/-/|/'

A reoccurring problem is to find executables using the terminal. One solution is to use find and set the perm mask accordingly. The following command will find files (-type f) with any executable permissions (-perm /11):

find . -type f  -perm /111 -exec echo '{}' \;

/111 indicates that any of the bit fields of the mask 0111 are set. In other words, for an executable file if any of the following flags are set:

-rwxr-xr-x
   ^  ^  ^
   |  |  |
 OR OR OR

then the find will match.

When a binary is found, find will print out the executable filename (-exec echo '{}').

Note that the executable flag can be set for any file or directory and it does not mean that the file is really an executable.

We use the -delete predicate:

find . -name "*.txt" -delete

Suppose you have a file containing the following lines:

a
b
c

and you wish to reverse the order so that you obtain the lines:

c
b
a

Then you can use store the lines in a file called file.txt and then issue the command:

tail -r file.txt

which will reverse the lines.

Usually the nice and renice commands are used to execute commands with lowered priority. However, the following can be used to run a shell script with a modified priority. For example, this is what the header of a script looks like:

#!/bin/sh
 
renice 19 -p $$

The renice command will schedule that script with the lowest priority.

The following command searches for all the zip files in all the sub-directories, decompresses them in the directory they are in while overwriting any existing file and finally removes the zip file itself.

find . -name '*.zip' | parallel -j4 'cd {//}; unzip -o "$(basename {/} .zip)"; rm {/}'

Suppose you have a hierarchy:

+ Music
|
+- A
|  +- A-10 Tank Killer.zip
|  |
|  +- Abandoned Places.zip
|   
+- B
|  +- Baal.zip
|  |
|  +- Back to the Future Part II.zip
|
...

after running the command in the Music directory, you want to obtain the following directory hierarchy:

+ Music
|
+- A
|  +- A-10 Tank Killer
|  |  +
|  |  |
|  |  + sound.mod
|  |  |
|  |  + print.wav
|  |
|  +- Abandoned Places
|     +
|     |
|     + abd.trk
|     |
|     + flow.wav
|     |
|     + mus.mp3
| 
+- B
|  +- Baal
|  |  +
|  |  |
|  |  + crest.wav
|  |
|  +- Back to the Future Part II
|     +
|     |
|     + music.mp3
|     |
|     + instruments
|       +
|       |
|       + guitars.pro
|       |
|       + percussion.trk
|
...

Thus, given a directory hierarchy that contains archives (in this case, zip files) stored within sub-directories, the following command will recursively unzip all the files in the hierarchy by:

• creating a directory for the zip file,
• unzipping the files in that directory,
• removing the original zip file

The command unarchives the files in parallel (the -j4 indicates 4 different processes) and works for directory trees of any depth.

Using parallel:

find . -iname '*.zip' | parallel -j4 'cd {//}; mkdir -p "$(basename {/} .zip)"; unzip {/} -d "$(basename {/} .zip)"; rm {/}'

and without parallel:

find . -iname '*.zip' -print0 | xargs -0 -I{} sh -c 'd=$(basename "{}" .zip); mkdir -p "$d"; unzip "{}" -d "$d";'

To use RAR / unrar instead, issue:

find . -name '*.rar' | parallel -j4 'cd {//}; mkdir "$(basename {/} .rar)"; unrar x {/} "$(basename {/} .rar)"; rm {/}'

Command Line Aspect	Visual Mnemonic Graft
-p -i"" -e

perl -p -i"" -e 's/ORIGINAL/REPLACEMENT/g' file

where:

• -p calls print at the end of the command,
• -i"" will perform the change in-place and the additional quotes imply that no backups shall be made (""),
• -e means to execute the following command, which just happens to be a regular expression,
• ORIGINAL represents a regular expression,
• REPLACEMENT is the replacement text,
• file is a file or a shell expanded list of files, such as the wildcard operator *

The former can also be combined with find in order to recursively and selectively replace text in files:

find . -type f -exec perl -p -i'' -e 's/was\.fm/grimore\.org/g' '{}' \;

this command will search all files in the path and replace the string was.fm with grimore.org.

This can be performed in several ways depending on the tools available.

This can be accomplished by using tar and the pipe operator:

tar -cf - /path/to/dir | ssh -C remote_server 'tar -xvf - -C /path/to/remotedir'

where:

• /path/to/dir is the path to the directory to transfer on the local server
• /path/to/remotedir is the destination directory on the remote server

Another alternative is to use nc (netcat) and cpio. First on the receiving machine, you would write:

nc -l -p <port> | gunzip | cpio -i -d -m

where <port> represents any port number.

and then on the destination machine, you would write:

find . -type f | cpio -o | gzip -1 | nc <destination> <port>

where <destination> is the hostname or IP address of the receiving machine and <port> represents the port number that you used on the receiving machine.

The following command will remove all duplicate lines from input.txt and output the result to output.txt while preserving the order of the lines:

cat -n input.txt | sort -k2 -k1n  | uniq -f1 | sort -nk1,1 | cut -f2- > output.txt

The command performs the following operation:

• cat -n input.txt - will open input.txt and number the lines and pass on the result.
• sort -k2 -k1n - will consider the characters from the second column to the end of the line first, then the first column but numerically when sorting.
• uniq -f1 - will ignore the first column.
• sort -nk1,1 - will sort the first column numerically while getting the sort order towhead it was originally.
• cut -f2- will remove the line numbers that were inserted by cat.

The perl rename command combined with find is the easiest way. The following command converts from uppercase to lowercase:

find . -depth -print -execdir rename -f 'y/A-Z/a-z/' '{}' \;

while the following command reverses the operation:

find . -depth -print -execdir rename -f 'y/a-z/A-Z/' '{}' \;

the rename part is self-explanatory but the magic lies with find: the -depth parameter specifies that files should be changed before directories - this is very important on case-sensitive file-systems because one may end-up changing a directory name before the files inside it and while the command is recursing, the directory name may not be found and this is what -depth is for.

The command:

paste codes.txt status.txt | awk -F"\t" '{print $1,$2}'

will print lines from codes.txt and then lines from status.txt delimited by a tab character. The sequence will then be piped into awk that will use the tab character as separator and print the first and then the second column received from the paste command.

Appending text to a file is usually achieved with:

echo "append this string to the file" >> test.txt

which adds the string at the end of the file.

In order to prepend some text to a file, the following can be used:

echo "prepend this string to the file" | cat - test.txt | cat - > test.txt

which adds the string at the top of the file.

The loop checks the date against the modification timestamp and sleeps while the difference is smaller than 10 seconds.

while [ $(( $(date +%s) - $(stat -c %Y FILENAME) )) -lt 10 ]; do 
    sleep 1
done 

The following command:

echo `yes +|head -50`|tr -d ' '

will repeat the + character 50 times.

Executing:

time read

and then aborting with Ctrl+D will print out the amount of time waited.

In order to remove all files without an ace, zip or rar extension, issue:

rm !(*.ace|*.zip|*.rar)

In oder to get the top RAM consuming processes:

ps -eo pmem,pcpu,pid,args | tail -n +2 | sort -rnk 1 | head

To get the top CPU consuming processes:

ps -eo pmem,pcpu,pid,args | tail -n +2 | sort -rnk 2 | head

The task is to copy a file, say /home/root/writeup.txt to the directories:

• /var/www/site_1.tld/data/
• /var/www/site_2.tld/data/
• /var/www/site_3.tld/data/

without looping - since the standard solution would be to loop over all the directories and then on every iteration copy the file.

This can be accomplished, in this scenario, using xargs:

echo /var/www/*/data/* | xargs -n 1 cp /etc/root/writeup.txt

Goto / exception handling can be implemented in a shell by using trap to install a handler and then deliver a signal to the current process:

#! /bin/sh
 
trap '{ echo "Good day!"; }' HUP
 
# do stuff
 
kill -s HUP $$

The script will install the (inline) handler function { echo "Good day!"; } and bind to the HUP signal. The kill command will then deliver the HUP signal to the shell process (itself) which will then call the handler function.

The mechanism can be used to implement some form of goto or exception continuations - for instance, one could just terminate the shell script inside the handler function instead of just printing out Good day!. For instance, the following script:

#! /bin/sh
 
trap '{ 
 
# Check if temporary file exists.
if [ -f /tmp/tmp.EERfX5aeY8 ]; then
    echo "temporary file exists"
    # Terminate.
    exit
fi
 
}' HUP
 
# Raise signal.
kill -s HUP $$
# Code not reached.
echo "END"

will:

1. install a handler for the HUP signal that will process an inline function.
2. the script will then raise the HUP signal and the inline function will execute.
3. since the inline function contains a terminator exit, the last line echo "END" will never execute

The goto trick is illustrated in the following scripts:

• script for blocking attacks with firehol level 1 list

Given the filesystem structure:

 top
  +
  |
  +---+ list.txt     
  |         
  +---+ stuff
      +---- f1.txt
      +---- f2.jpg
      +---- f4.gif
      +---- f5.doc

and a file containing:

f2.jpg
f5.doc

and would like to remove all the files in the stuff folder that are contained in the list file, then the following command ran from the top directory:

find stuff -type f -print0 | grep -zZ -f list.txt | xargs -0 rm

where:

• stuff is the directory containing the files,
• -type f specifies that only files should be matched,
• -print0 use a null byte as a terminator (in order to be compatible with files containing spaces),
• -zZ makes grep treat all input as being terminated with null bytes,
• -f list.txt will instruct grep to read the file list.txt line-by-line,
• rm removes the files

will remove all the files in the stuff directory that match any entry line-by-line in list.txt.

Conversely, if all the files in stuff have to be removed that are not contained in the list file list.txt, simply add the -v option to grep in the filter chain.

The following command will generate an alpha-numeric string from the real random number device that would be suitable to be used as a password:

egrep -m 10 -ao '[\x20-\x7E]{1}$' </dev/random | tr -d '\n' | wc -c

where:

• -m 10 matches printable characters (all ASCII characters starting from space 20 octal and up to tilde symbol 7E in octal) from /dev/urandom,
• -a instructs grep to treat even binary input as characters,
• -o instructs grep to only print matches,
• tr will be used to delete space characters, and
• wc -c will count the characters for verification.

On OSX you can prevent the password to be echoed to the terminal by enqueueing the pbcopy command, ie:

egrep -m 20 -ao '[\x20-\x7E]{1}$' </dev/random | tr -d '\n' | pbcopy

which will then place the password in the clipboard.

More than often, as a command runs on the command line, it becomes perceivable that the command will take a long while to terminate. In such cases, in case the connection is severed, the command will be terminated as well since it is attached to the terminal being used for the connection.

In such situations, the solution is to background the process, disown the process from the current terminal and, perhaps, re-attach the process using a terminal multiplexer such as tmux.

With a command running in the current terminal, the operations should be:

1. press the key combination Ctrl+Z in order to suspend the process and return to the command line,
2. issue bg to background the process,
3. issue the disown command with the PID of the background command as a parameter, for instance disown 3142

At this point, the connection can be severed and the process will continue running in the background. However, there is no way of capturing the command output, if any. One solution is to use a terminal multiplexer to re-attach the command and then be able to receive output or send input:

1. issue tmux to create a new session,
2. in the new terminal opened by tmux, issue reptyr -s PID where PID is the process identifier of the command running in the backround

The command will now execute in the TMux session. Now, to background the entire tmux session with the process running, press the key-combination Ctrl+B to open up the tmux command line and type: :detach.

Later on, to check up on the command's progress, issue the command tmux attach.

echo 'AGsg4SKKs74s62#' | sed 's/./&\n/g' | shuf | tr -d "\n"

awk -v min=X -v max=Y 'BEGIN{srand(); print int(min+rand()*(max-min+1))}'

where:

• X is the low number,
• Y is the high number

find . -type f -exec grep -Iq . {} \; -print

find -name \*.txt | while read f; do tail -n1 $f | read -r _ || echo >> $f; done

where:

• *.txt is the pattern to search for.

Using su, the following script can be ran as root but will execute the remainder of the script after the if clause as the user myanonamouse:

# Restart the script as an user via su.
if [ "$(id -u)" -eq 0 ]; then
    # exec replaces current shell
    exec su myanonamouse "$0" -- "$@"
    # this line will not be reached
fi
 
# this will run as user "myanonamouse"

Using opendns.com that is programmed to return the IP address of the client performing the lookup:

dig +short myip.opendns.com @resolver1.opendns.com

The following command:

install -D -o www-data -g www-data /dev/null /var/log/razor.log

will create the file /var/log/razor.log and set the ownership to the user www-data and group www-data. The -D flag will ensure that in case the path /var/log/ is missing some directories, then the path will be created.

The following line will print 10 successive dots:

yes "." | head -n 10 | xargs | tr -d ' '

The Unix utility find provides a results placeholder that holds the filename that has been found matching the filters provided to find. Here is an example that searches the current directory recursively for files ending in mp4:

find . -type f -name '*.mp4' -exec echo "{}" \;

such that the command after the exec parameter will be executed for all files, with the special curly-bracers symbol {} expanding during execution to the full path of the file as reported by the find utility.

Remarkably, as feature-packed as Unix tools are, the results placeholder is deceptively straightforward and without features. For example, one of the most common tasks is to perform some sort of string substitution on the results in case the file must be moved or manipulated in any way.

In order to use the file name without further bothering about the find utility, simply pass the special placeholder symbol {} as the argument to a shell, for example, a bash shell and then for the command being executed, the contents of {} will be transferred to the first command-line argument $0. Here is an example that changes the extension of all images recursively from the current directory from jpg to png:

find . -type f -name '*.jpg' -exec bash -c 'mv "$0" "${0//jpg/png}"' {}  \;

Whilst the command invocation:

find . -type f -name '*.jpg' -exec ... \;

is responsible for finding files ending with the jpg extension, for every file found, the following command is executed, that can be disected:

bash -c 'mv "$0" "${0//jpg/png}"' {}
+     +      +          +        +
|     |      |          |        |
+-----+      |          |        | from find; expands to the path of the file
execute a    |          |
command      |          |
             |          | bash substitution acting on parameter $0,
             |          | changing "jpg" to "png"
             |
             |
             | $0 is the first parameter
             | passed to bash and holds
             | the exact contents of {}

Something that seems counter-intuitive is that the command passed to the bash -c command-line parameter is quoted using single-quotes, which would imply no substitution:

find . -type f -name '*.jpg' -exec bash -c 'mv "$0" "${0//jpg/png}"' {}  \;

Interestingly, if one were to issue the same command with double-quotes instead:

.................................. bash -c "mv \"$0\" \"${0//jpg/png}\"" {}  \;

then the command would not run properly because while issuing the "find" command itself, the shell would substitute the parameters within the quotes. That being said, it only makes sense to execute the "inner-most" command whilst escaping it with single quotes:

.................................. bash -c 'mv "$0" "${0//jpg/png}"' {}  \;

such that substitution only occurs when the command passed to the -c bash parameter actually executes.

An equivalent to the alarm(2) call in systems programming can be created by simply placing a process in the background and then killing the process in order to rearm the alarm or wait until the time elapses in order for the command to run.

# alarm(2)
function alarm {
    sleep $1
    # this is the command to be executed after the elapsed time
    echo "EXEC"
}
 
 
# ensures that any pending alarm is cleared upon termination of the current script
ALARM_PID=0
trap '{ test $ALARM_PID = 0 || kill -KILL $ALARM_PID; }' KILL QUIT TERM EXIT INT HUP
 
# the parent program consists in a process that runs indefinitely
while "..."; do
    # when the alarm must be re-armed:
    #   * kill the previous alarm
    #   * reschedule the alarm into the future
    if [ -d /proc/"$ALARM_PID" ]; then
        kill -9 $ALARM_PID
    fi
    alarm "5" &
    ALARM_PID=$!
done

The actual code to be executed is to be found within the alarm function, namely anything after the sleep line. Fortunately, SIGKILL can interrupt sleep and additionally abort the program such that the command will not be executed when the alarm is just rearmed.

Note that the expression $! might be a bashism but there are ample equivalents such as using pidof, etc, to determine the PID of the last backgrounded process in case the scheme has to be used within a strict POSIX shell.

Given two files A and B both with lines comparable under the same context, here are some of the operations that can be performed when considering the lines to be set elements and the files A and B sets. Note that this respects set theory operations in terms of sets accepting only the same kind of element once and only once (as opposed to the notion of "collections" where elements of the same kind might be accepted multiple times) - or, in short, both files A and ''B' contain lines unique to themselves.

"Lines in file A but not in file B" is a set subtraction (defined as all elements / lines in that are not in ).

To perform this operation, issue:

comm -23 <(sort A) <(sort B)

"Lines in file B but not in file A" is a set subtraction (defined as all elements / lines in that are not in ).

To perform this operation, issue:

comm -13 <(sort A) <(sort B)

"Lines in file A that are also in B" is a set intersection .

To perform this operation, issue:

comm -12 <(sort A) <(sort A)