About

 Oh no, not this shit again! Do not let yourself be fooled, a shell is like any other interpreter-based language, such as perl. The only real difference is that a shell is interactive while interpreters usually execute scripts, one-time. Thus, as with any programming language, constructs may seem similar yet the syntax may greatly differ - in which case some of the fuss provided on this page will work on some shells but not on others.

For example, Linux distributions are moving away from bash as the default shell, towards dash which is leaning towards POSIX compatibility. Apparently, bash does cover dash features but dash has some of its own dash-centric features, such that Linux now starts to depend on dash instead of bash.

Furthermore, every Linux distribution also packs its own set of tools, just like commands in a programming language, such that the fuss listed here may work on some systems, but not on others.

Learning or preferring one shell over the other, given that they are all feature-packed (or, conversely, feature-less), becomes a matter of religious dispute - just like choosing a Linux distribution, which gradually tends to void any sense of pragmatism and becomes difficult to categorize.

Lastly, some rudimentary commands, such as ps (used for retrieving the process table) use differing syntax, for different set of features and spanning across different versions.

Copyright

###########################################################################
##  Copyright (C) Wizardry and Steamworks 2019 - License: GNU GPLv3      ##
##  Please see: http://www.gnu.org/licenses/gpl.html for legal details,  ##
##  rights of fair usage, the disclaimer and warranty conditions.        ##
###########################################################################

Grab PID of Running Process

ps ax | grep '[a]fpd' | awk '{print $1}'

would grab the PID of a process named afpd.

Perform Reverse Lookups of IP Addresses in Apache Access Logs

Provided you are in the same directory as the access_log file:

for i in `cat access_log | awk '{ print $1 }'`; do 
  dig -x $i +noall +short +answer; 
done | grep -v 'googlebot\|cloudshare'

which prints all hostnames that that IPs resolve to except the hostnames matching googlebot and cloudshare.

for i in `cat access_log | awk '{ print $1 }'`; do 
  dig -x $i +noall +short +answer; 
done | grep '\.edu\|\.mil\|\.gov'

looks for the funny guys (will match .milkshake TLDs as well).

Suppress Command Output

In cases where the /dev/null device is unavailable, pipe stderr and stdout together to the command true:

ps ax 2&>1 | true

will suppress the output of ps ax

Logarithm Base-2 For Values Per Line

Supposing that data.txt contains a value on every line:

cat data.txt | while read i; do perl -e "print log($i)/log(2)" && echo -en "\n"; done

Reverse Lines in a File

Surprisingly easy:

cat file.txt | tail -r

Sort By First Column

Supposing that combined_data.txt contains lines such as:

10 blah
10000 blah blah

the following will sort the above example in descending order by the first column:

cat combined_data.txt | sort -s -n -r

The result is:

10000 blah blah
10 blah

Lowercase->Uppercase, Uppercase->Lowercase

Input:

echo "hello world" | tr [:lower:] [:upper:]

Output:

HELLO WORLD

Unrar Files Recursively

find . -name "*.rar" -exec unrar x -ad '{}' \;

The ad switch creates a folder named after the rar file. For example, the files in My Arhive.rar will create a folder My Archive/ and place the files there.

Timestamp

A good time-stamp is, in order:

  1. year
  2. month
  3. day
  4. hour
  5. minutes
  6. seconds

because when it is used with files, the stamp takes precedence in chronological order. One can generate this timestamp with:

date +%Y%m%d%H%M | tr '\n' ' '

where tr is used to delete the newline produced by date and add a space.

Yes/No Script

Only a y or n option breaks out of the loop.

while [ "$OPT" != "y" ] && [ "$OPT" != "n" ]; do
  echo -n "Yes or no? (y/n) : "
  read OPT
  case "$OPT" in
    y)
      echo "yes!"
      ;;
    n)
      echo "no!"
      ;;
    *)
      ;;
  esac
done

Lock

To prevent parallel execution of scripts, the following snippet can be used:

# Acquire a lock.
LOCK_FILE='/var/lock/myscript'
if mkdir $LOCK_FILE 2>&1 >/dev/null; then
    trap '{ rm -rf $LOCK_FILE; }' KILL QUIT TERM EXIT INT HUP
else
    exit 0
fi

where:

  • LOCK_FILE points to a location where a directory myscript can be created.

The command mkdir is guaranteed to be atomic, such that out of multiple concurrent processes only one will succeed in creating the directory at /var/lock/myscript. Once the directory is created, the lock is established, and the trap instruction will bind to the KILL, QUIT, TERM, EXIT, INT and HUP, essentially ensuring that the code { rm -rf $LOCK_FILE; } will be executed once those signals are raised (this includes normal script termination - via QUIT) thereby releasing the lock.

The snippet can be used directly in any script just by setting the value of LOCK_FILE to some chosen path where the script will be able to create a lock directory.

Loop Over Command Line Arguments

for arg in "$@"; do
  echo $arg
done

Concatenate Strings

Input:

HELLO='hello'
WORLD='world'
GREET=$HELLO" "$WORLD
echo $GREET

Output:

hello world

Sleep Sort

Introduced by Anonymous on 4chan. For each number to sort, spawn a thread (process) and make the thread sleep for that number of seconds and receive the output in order. Noted for further reference.

Example, call with:

./sleepsort.bash 5 3 6 3 6 3 1 4 7

to sort the numbers.

#!/bin/bash
function f() {
    sleep "$1"
    echo "$1"
}
while [ -n "$1" ]
do
    f "$1" &
    shift
done
wait

Notes

  • The algorithm sleeps for the numeric number of the values to sort - in bash, that would be seconds. One can perhaps further reduce the numbers by dividing them with a large constant. That way, the thread would sleep for a very short period of time. When the numbers arrive, multiply the number with the same large constant to obtain the original numbers in sorted order.
  • This is an instance where a race-condition can be used in a beneficial way. Another example for race conditions that can be used in a beneficial way can be found on the project Alexandria page. The Oracle sends a trigger message on a public channel and the bookshelves are given a certain timeout to respond. Based on the time it takes for that answer to arrive from any bookshelf, the Oracle can tell whether the result is more relevant than other results.

Shell History

The bash history of commands can be reveled by issuing:

history

The number on the left column represent an index, and the command is listed in the right column. To repeat a certain command, issue:

!<index>

where index is a number from the left column. For example:

!602

will execute the command referenced by the index 602.

The history can also be searched using the keyboard shortcut Ctr+R or using the !? operator followed by a partial history match.

If you would write:

!!

then the first ! will be substituted for the last command and the second ! will be substituted for the command arguments. You can use !! as a shortcut to repeating the previous command as an alternative to using the up-arrow and pressing enter.

To execute the parameters of the previous command:

!*

which helps in cases where two commands were issued by accident.

As an alternative, you could write:

!!:gs/foo/bar

which will replace foo by bar every time foo appears (hence the g; s stands for substitute).

One other thing to do is:

echo "!!" > script.sh

which will take the last command and arguments and place them in a file called script.sh.

Pipe Visualiser

pv is an utility that can intercept pipes and show various statistics during transfer.

pv --format="Elapsed Time:%t Transferred:%b Rate:%r Average:%a" /dev/disk3 > disk3.img

Unless pv opens a file directly, it will not be able to determine its file size and will not be able to display an ETA or what is commonly understood by a progress bar. Even when reading from a block device such as /dev/disk3 in the example above, pv does not seem able to determine the size of the associated drive. In case pv is not able to determine the file size, what is commonly understood by a progress bar will be in fact an activity monitor. This is why the example above chooses to show only statistics instead of a progress bar.

Length of String

Given a string variable var, the following:

echo ${#var}

prints out the length of the string var.

Convert Shellcode to OpCode

Requires the nasm disassembler:

echo -ne "\xeb\xe0" | ndisasm -u -

Purge Shell History

history -c && rm -f ~/.bash_history

Change all Files in a Directory to Lowercase

rename 'y/A-Z/a-z/' *

Sort File by Line Length

cat file | awk '{ print length, $0 }' | sort -n | cut -d" " -f2-

Delete a List of Files

Suppose you have a text file containing files listed one-by-one such as:

# removed in 2014-05-05
lib/images/fileicons/audio.png
lib/plugins/acl/lang/hi/lang.php
lib/plugins/acl/lang/id-ni/lang.php

then, the following command can be used to delete them:

grep -Ev '^($|#)' delete.txt | xargs -n 1 rm -vf

The command ignores any lines starting with either $ or #.

Show a Tree View of Folders, Subfolders and Files

ls -R | grep ":$" | sed -e 's/:$//' -e 's/[^-][^\/]*\//--/g' -e 's/^/ /' -e 's/-/|/'

Find Executables

A reoccurring problem is to find executables using the terminal. One solution is to use find and set the perm mask accordingly. The following command will find files (-type f) with any executable permissions (-perm /11):

find . -type f  -perm /111 -exec echo '{}' \;

/111 indicates that any of the bit fields of the mask 0111 are set. In other words, for an executable file if any of the following flags are set:

-rwxr-xr-x
   ^  ^  ^
   |  |  |
 OR OR OR

then the find will match.

When a binary is found, find will print out the executable filename (-exec echo '{}').

Note that the executable flag can be set for any file or directory and it does not mean that the file is really an executable.

Delete Files using Find

We use the -delete predicate:

find . -name "*.txt" -delete

Reverse Lines in a File

Suppose you have a file containing the following lines:

a
b
c

and you wish to reverse the order so that you obtain the lines:

c
b
a

Then you can use store the lines in a file called file.txt and then issue the command:

tail -r file.txt

which will reverse the lines.

Run Shell Script with Modified Priority

Usually the nice and renice commands are used to execute commands with lowered priority. However, the following can be used to run a shell script with a modified priority. For example, this is what the header of a script looks like:

#!/bin/sh
 
renice 19 -p $$

The renice command will schedule that script with the lowest priority.

Extract Files Recursively

The following command searches for all the zip files in all the sub-directories, decompresses them in the directory they are in while overwriting any existing file and finally removes the zip file itself.

find . -name '*.zip' | parallel -j4 'cd {//}; unzip -o "$(basename {/} .zip)"; rm {/}'

While Creating Folders

Suppose you have a hierarchy:

+ Music
|
+- A
|  +- A-10 Tank Killer.zip
|  |
|  +- Abandoned Places.zip
|   
+- B
|  +- Baal.zip
|  |
|  +- Back to the Future Part II.zip
|
...

after running the command in the Music directory, you want to obtain the following directory hierarchy:

+ Music
|
+- A
|  +- A-10 Tank Killer
|  |  +
|  |  |
|  |  + sound.mod
|  |  |
|  |  + print.wav
|  |
|  +- Abandoned Places
|     +
|     |
|     + abd.trk
|     |
|     + flow.wav
|     |
|     + mus.mp3
| 
+- B
|  +- Baal
|  |  +
|  |  |
|  |  + crest.wav
|  |
|  +- Back to the Future Part II
|     +
|     |
|     + music.mp3
|     |
|     + instruments
|       +
|       |
|       + guitars.pro
|       |
|       + percussion.trk
|
...

Thus, given a directory hierarchy that contains archives (in this case, zip files) stored within sub-directories, the following command will recursively unzip all the files in the hierarchy by:

  • creating a directory for the zip file,
  • unzipping the files in that directory,
  • removing the original zip file

The command unarchives the files in parallel (the -j4 indicates 4 different processes) and works for directory trees of any depth.

For ZIP

Using parallel:

find . -iname '*.zip' | parallel -j4 'cd {//}; mkdir -p "$(basename {/} .zip)"; unzip {/} -d "$(basename {/} .zip)"; rm {/}'

and without parallel:

find . -iname '*.zip' -print0 | xargs -0 -I{} sh -c 'd=$(basename "{}" .zip); mkdir -p "$d"; unzip "{}" -d "$d";'

For RAR

To use RAR / unrar instead, issue:

find . -name '*.rar' | parallel -j4 'cd {//}; mkdir "$(basename {/} .rar)"; unrar x {/} "$(basename {/} .rar)"; rm {/}'

Replace String in Files

To replace a string in multiple files you can use find and perl:

find . -type f -exec perl -p -i -e 's/was\.fm/grimore\.org/g' '{}' \;

this command will search all files in the path and replace the string was.fm with grimore.org.

Transfer a Directory between Servers

This can be performed in several ways depending on the tools available.

Using Tape Archives

This can be accomplished by using tar and the pipe operator:

tar -cf - /path/to/dir | ssh -C remote_server 'tar -xvf - -C /path/to/remotedir'

where:

  • /path/to/dir is the path to the directory to transfer on the local server
  • /path/to/remotedir is the destination directory on the remote server

Using Sockets

Another alternative is to use nc (netcat) and cpio. First on the receiving machine, you would write:

nc -l -p <port> | gunzip | cpio -i -d -m

where <port> represents any port number.

and then on the destination machine, you would write:

find . -type f | cpio -o | gzip -1 | nc <destination> <port>

where <destination> is the hostname or IP address of the receiving machine and <port> represents the port number that you used on the receiving machine.

Delete Duplicate Lines in a File while Preserving the Order

The following command will remove all duplicate lines from input.txt and output the result to output.txt while preserving the order of the lines:

cat -n input.txt | sort -k2 -k1n  | uniq -f1 | sort -nk1,1 | cut -f2- > output.txt

The command performs the following operation:

  • cat -n input.txt - will open input.txt and number the lines and pass on the result.
  • sort -k2 -k1n - will consider the characters from the second column to the end of the line first, then the first column but numerically when sorting.
  • uniq -f1 - will ignore the first column.
  • sort -nk1,1 - will sort the first column numerically while getting the sort order towhead it was originally.
  • cut -f2- will remove the line numbers that were inserted by cat.

Rename Files Recursively Changing Case

The perl rename command combined with find is the easiest way. The following command converts from uppercase to lowercase:

find . -depth -print -execdir rename -f 'y/A-Z/a-z/' '{}' \;

while the following command reverses the operation:

find . -depth -print -execdir rename -f 'y/a-z/A-Z/' '{}' \;

the rename part is self-explanatory but the magic lies with find: the -depth parameter specifies that files should be changed before directories - this is very important on case-sensitive file-systems because one may end-up changing a directory name before the files inside it and while the command is recursing, the directory name may not be found and this is what -depth is for.

Merge Files Line-by-Line

The command:

paste codes.txt status.txt | awk -F"\t" '{print $1,$2}'

will print lines from codes.txt and then lines from status.txt delimited by a tab character. The sequence will then be piped into awk that will use the tab character as separator and print the first and then the second column received from the paste command.

Prepend and Append Text to a File

Appending text to a file is usually achieved with:

echo "append this string to the file" >> test.txt

which adds the string at the end of the file.

In order to prepend some text to a file, the following can be used:

echo "prepend this string to the file" | cat - test.txt | cat - > test.txt

which adds the string at the top of the file.

Wait for File to Stop Changing

The loop checks the date against the modification timestamp and sleeps while the difference is smaller than 10 seconds.

while [ $(( $(date +%s) - $(stat -c %Y FILENAME) )) -lt 10 ]; do 
    sleep 1
done 

Repeat a Sequence

The following command:

echo `yes +|head -50`|tr -d ' '

will repeat the + character 50 times.

Stopwatch

Executing:

time read

and then aborting with Ctrl+D will print out the amount of time waited.

Delete Files that Do Not Match an Extension

In order to remove all files without an ace, zip or rar extension, issue:

rm !(*.ace|*.zip|*.rar)

Get Top RAM and CPU Consuming Processes

In oder to get the top RAM consuming processes:

ps -eo pmem,pcpu,pid,args | tail -n +2 | sort -rnk 1 | head

To get the top CPU consuming processes:

ps -eo pmem,pcpu,pid,args | tail -n +2 | sort -rnk 2 | head

Copy a File to Multiple Directories

The task is to copy a file, say /home/root/writeup.txt to the directories:

  • /var/www/site_1.tld/data/
  • /var/www/site_2.tld/data/
  • /var/www/site_3.tld/data/

without looping - since the standard solution would be to loop over all the directories and then on every iteration copy the file.

This can be accomplished, in this scenario, using xargs:

echo /var/www/*/data/* | xargs -n 1 cp /etc/root/writeup.txt

Goto and Exception Handling in Shell Scripts

Goto / exception handling can be implemented in a shell by using trap to install a handler and then deliver a signal to the current process:

#! /bin/sh
 
trap '{ echo "Good day!"; }' HUP
 
# do stuff
 
kill -s HUP $$

The script will install the (inline) handler function { echo "Good day!"; } and bind to the HUP signal. The kill command will then deliver the HUP signal to the shell process (itself) which will then call the handler function.

The mechanism can be used to implement some form of goto or exception continuations - for instance, one could just terminate the shell script inside the handler function instead of just printing out Good day!. For instance, the following script:

#! /bin/sh
 
trap '{ 
 
# Check if temporary file exists.
if [ -f /tmp/tmp.EERfX5aeY8 ]; then
    echo "temporary file exists"
    # Terminate.
    exit
fi
 
}' HUP
 
# Raise signal.
kill -s HUP $$
# Code not reached.
echo "END"

will:

  1. install a handler for the HUP signal that will process an inline function.
  2. the script will then raise the HUP signal and the inline function will execute.
  3. since the inline function contains a terminator exit, the last line echo "END" will never execute

Examples

The goto trick is illustrated in the following scripts:

Remove Files In or Not in a List

Given the filesystem structure:

 top
  +
  |
  +---+ list.txt     
  |         
  +---+ stuff
      +---- f1.txt
      +---- f2.jpg
      +---- f4.gif
      +---- f5.doc

and a file containing:

f2.jpg
f5.doc

and would like to remove all the files in the stuff folder that are contained in the list file, then the following command ran from the top directory:

find stuff -type f -print0 | grep -zZ -f list.txt | xargs -0 rm

where:

  • stuff is the directory containing the files,
  • -type f specifies that only files should be matched,
  • -print0 use a null byte as a terminator (in order to be compatible with files containing spaces),
  • -zZ makes grep treat all input as being terminated with null bytes,
  • -f list.txt will instruct grep to read the file list.txt line-by-line,
  • rm removes the files

will remove all the files in the stuff directory that match any entry line-by-line in list.txt.

Conversely, if all the files in stuff have to be removed that are not contained in the list file list.txt, simply add the -v option to grep in the filter chain.

Generate Random String (for Passwords, or Otherwise)

The following command will generate an alpha-numeric string from the real random number device that would be suitable to be used as a password:

egrep -m 10 -ao '[\x20-\x7E]{1}$' </dev/random | tr -d '\n' | wc -c

where:

  • -m 10 matches $10$ printable characters (all ASCII characters starting from space 20 octal and up to tilde symbol 7E in octal) from /dev/urandom,
  • -a instructs grep to treat even binary input as characters,
  • -o instructs grep to only print matches,
  • tr will be used to delete space characters, and
  • wc -c will count the characters for verification.

On OSX you can prevent the password to be echoed to the terminal by enqueueing the pbcopy command, ie:

egrep -m 20 -ao '[\x20-\x7E]{1}$' </dev/random | tr -d '\n' | pbcopy

which will then place the password in the clipboard.

Detach, Background and Re-Attach a Process

More than often, as a command runs on the command line, it becomes perceivable that the command will take a long while to terminate. In such cases, in case the connection is severed, the command will be terminated as well since it is attached to the terminal being used for the connection.

In such situations, the solution is to background the process, disown the process from the current terminal and, perhaps, re-attach the process using a terminal multiplexer such as tmux.

With a command running in the current terminal, the operations should be:

  1. press the key combination Ctrl+Z in order to suspend the process and return to the command line,
  2. issue bg to background the process,
  3. issue the disown command with the PID of the background command as a parameter, for instance disown 3142

At this point, the connection can be severed and the process will continue running in the background. However, there is no way of capturing the command output, if any. One solution is to use a terminal multiplexer to re-attach the command and then be able to receive output or send input:

  1. issue tmux to create a new session,
  2. in the new terminal opened by tmux, issue reptyr -s PID where PID is the process identifier of the command running in the backround

The command will now execute in the TMux session. Now, to background the entire tmux session with the process running, press the key-combination Ctrl+B to open up the tmux command line and type: :detach.

Later on, to check up on the command's progress, issue the command tmux attach.

Scramble a String

echo 'AGsg4SKKs74s62#' | sed 's/./&\n/g' | shuf | tr -d "\n"

Generate Random Numbers in Range using awk

awk -v min=X -v max=Y 'BEGIN{srand(); print int(min+rand()*(max-min+1))}'

where:

  • X is the low number,
  • Y is the high number

fuss/sh.txt ยท Last modified: 2020/03/30 06:18 by office

Access website using Tor Access website using i2p


For the copyright, license, warranty and privacy terms for the usage of this website please see the license, privacy and plagiarism pages.