Glob

Glob is a fast shell-based equivalent of regular expressions.

Glob Explanation
* Matches any string, of any length
foo* Matches any string beginning with foo
*x* Matches any string containing an x (beginning, middle or end)
*.tar.gz Matches any string ending with .tar.gz
foo? Matches foot or foo$ but not fools

Range

Glob Explanation
[abcd] Matches a, b, c and d
[a-d] The same as above, if your locale is C or POSIX. Otherwise, implementation-defined.
[!aeiouAEIOU] Matches any character except a, e, i, o, u and their uppercase counterparts
[[:alnum:]] Matches any alphanumeric character in the current locale (letter or number)
[[:space:]] Matches any whitespace character
[![:space:]] Matches any character that is not whitespace
[[:digit:]_.] Matches any digit, or _ or .

Increment in Bash s

array[$[${#array[@]}+1]] = 3

is homologous to:

array[elements_in(array)+1] = 3

String Substitute

Input:

T="test.svg"
echo ${T/svg/png}

Output:

test.png

Check if String Contains a Substring

CHECK_STRING="google"
if [[ "$CHECK_STRING" != *o* ]]; then
  echo "character o is in the string"
fi

Split a String to an and other Manipulations

The following code:

IFS=' ' read -ra VAR_ARRAY <<< "$input"

where:

  • IFS will switch the separator (the character to split on) to a space.
  • read will read-in the input
  • VAR_ARRAY is the array that will be filled with the values from the input split by the separator
  • $input is the input to split

will split the input to an array.

In order to iterate over the elements of the array, one would write:

for e in "${VAR_ARRAY[@]}"; do
    echo $e
done

which will print out all the elements in the array.

To get both the index and the value while iterating over the array, one would write:

for i in "${!VAR_ARRAY[@]}"; do
    echo "$i ${VAR_ARRAY[i]}"
done

where on every iteration $i is the index and ${VAR_ARRAY[i]} will expand to the array element at i.

To get the number of elements in the array, one would write:

echo "${VAR_ARRAY[@]}"

Get the CPU Core Temperature File

If you use coretemp to retrieve the CPU sensors temperature, a bunch of files are created in /sys/devices/platform/coretemp.*/hwmon/hwmon*/ such as temp1_label, temp2_label, etc… Each of them represent the temperature of one of the cores but there is no guarantee that the numbering will start at 1. The following bash function reads the files in /sys/devices/platform/coretemp.*/hwmon/hwmon*/ and returns the file corresponding to a certain core file:

###########################################################################
##  Copyright (C) Wizardry and Steamworks 2016 - License: GNU GPLv3      ##
###########################################################################
function getCoreTemperatureFile {                                          
    for i in /sys/devices/platform/coretemp.*/hwmon/hwmon*/*_label; do     
        IFS=' ' read -ra CORE <<< `cat $i`                                 
        if [ "${CORE[0]}" == "Core" ] && [ "${CORE[1]}" == $1 ]; then     
            echo "${i//label/input}"
        fi
    done
}

an example call, would then be:

CPU0_TEMPERATURE_FILE=$(getCoreTemperatureFile 0)
echo $CPU0_TEMPERATURE_FILE

in order to retrieve the temperature of the first core (0).

Get a List of Files by File Extension from an Array

Via a function listFiles:

# By: https://stackoverflow.com/users/1773798/renaud-pacalet
listFiles() {
    local -n _OUTPUT_="$1"
    local -n _DIRECTORY_="$2"
    local -n _TYPES_="$3"
    local _FILTER_
 
    _FILTER_="${_TYPES_[@]/#/ -o -name *.}"
    _FILTER_="${_FILTER_# -o }"
    while read -d $'\0' FILE; do
        _OUTPUT_+=( "$FILE" )
    done < <( find "$_DIRECTORY_" -type f \( $_FILTER_ \) -print0 )
}

Example call:

DIR_PATH=/mnt/Movies
TYPES=(mkv mp4 avi)
 
listFiles FILES DIR_PATH TYPES

Recursively Convert Video Files

As the title would imply, the following script will recursively convert video files to conform to the following parameters:

  • MPEG-4
  • 720p or less,
  • AVC (H.264),
  • AAC

Its main design was meant to convert recorded TV shows where a set quality should not necessarily exceed the mentioned parameters for the sake of storage conservation.

#!/bin/bash
###########################################################################
##  Copyright (C) Wizardry and Steamworks 2021 - License: GNU GPLv3      ##
###########################################################################
# The script is a way of converting all video files recursively in a      #
# directory tree in order to conform to the following parameters:         #
#  * MPEG-4 (MP4)                                                         #
#  * 480p or less                                                         # 
#  * AVC (H.264)                                                          #
#  * AAC                                                                  #
#                                                                         #
# Notes:                                                                  #
#  * There are some added optimizations such as switching to just         #
#    converting to MPEG-4 in case no more than a container change is      #
#    necessary  (ie: MKV to MP4).                                         #
#  * This is a long-running script and best ran within a detachable       #
#    terminal such as tmux or screen.                                     #
#  * This is a bash script that will not run on other shells.             #
###########################################################################
 
###########################################################################
##                            CONFIGURATION                              ##
###########################################################################
 
# The path to the ffmpeg binary.
FFMPEG='/usr/bin/ffmpeg'
# The ffmpeg flags to use when converting.
# veryfast vs. veryslow https://write.corbpie.com/ffmpeg-preset-comparison-x264-2019-encode-speed-and-file-size/
FFMPEG_OPTIONS=`tr -d '\012' <<'EOF'
-c:v libx264 \
-crf 23 \
-level 3.1 \
-preset veryfast \
-tune film \
-sws_flags lanczos \
-sn \
-c:a libfdk_aac \
-vbr 5 \
-q:a 2 \
-ac 2 \
-b:a 128k \
-c:s mov_text \
-map 0:v:0 \
-map 0:a:0 \
-map_metadata -1 \
-movflags +faststart \
-strict -2
EOF`
# The maximal height of the video files.
FFMPEG_HEIGHT=480
# The maximal integral CPU percentage that will be allowed to be used.
FFMPEG_CPU=600
# The file types to convert.
TYPES=(mp4 mkv avi)
 
###########################################################################
##                              INTERNALS                                ##
###########################################################################
 
if [ -z "$1" ] || [ ! -d "$1" ]; then
    echo "SYNTAX: $0 <DIRECTORY>"
    exit 1
fi
 
DIR_PATH="$1"
 
# By: https://stackoverflow.com/users/1773798/renaud-pacalet
listFiles() {
    local -n _OUTPUT_="$1"
    local -n _DIRECTORY_="$2"
    local -n _TYPES_="$3"
    local _FILTER_
 
    _FILTER_="${_TYPES_[@]/#/ -o -name *.}"
    _FILTER_="${_FILTER_# -o }"
    while read -d $'\0' FILE; do
        _OUTPUT_+=( "$FILE" )
    done < <( find "$_DIRECTORY_" -type f \( $_FILTER_ \) -print0 )
}
 
TEMP_FILE=""
 
# Cleanup on file termination.
trap '{
    rm -rf "$TEMP_FILE" 2>&1 2>/dev/null
    exit 0
}' KILL QUIT TERM EXIT INT HUP
 
# Iterate over all matching files.
FILES=()
listFiles FILES DIR_PATH TYPES
for FILE in "${FILES[@]}"; do
    FILE_NAME=`basename "$FILE"`
    DIR_NAME=`dirname "$FILE"`
    FILE_EXTENSION=`echo "$FILE" | awk -F'.' '{ print $NF }'`
    TEMP_FILE="/tmp/${FILE_NAME/$FILE_EXTENSION/mp4}"
    INFO=`mediainfo "$FILE" | awk -F':' '{ print $1":"$2 }' | awk '{$1=$1};1'`
    HAS_AVC=`echo "$INFO" | grep 'Format : AVC'`
    HAS_AAC=`echo "$INFO" | grep 'Format : AAC LC'`
    HAS_MP4=`echo "$INFO" | grep 'Format : MPEG-4'`
    HAS_2CH=`echo "$INFO" | grep 'Channel(s) : 2'`
    HAS_TXT=`mediainfo $FILE | grep "^Text"`
    HEIGHT=`mediainfo --Inform="Video;%Height%" "$FILE"`
 
    # If all criteria matches then proceed to the next file.
    if [[ $HEIGHT -le $FFMPEG_HEIGHT ]] && \
       [[ ! -z $HAS_AVC ]] && \
       [[ ! -z $HAS_AAC ]] && \
       [[ ! -z $HAS_MP4 ]] && \
       [[ ! -z $HAS_2CH ]] && \
       [[ -z $HAS_TXT ]]
    then
    	echo "Skipping $FILE_NAME..."
    	continue
    fi
 
    FFMPEG_ACTION=$FFMPEG_OPTIONS
    # Resize down to $FFMPEG_HEIGHT or less depending on the height of the file.
    if [[ $HEIGHT -gt $FFMPEG_HEIGHT ]]; then
        FFMPEG_SIZE="-vf scale=-2:$FFMPEG_HEIGHT"
    fi
 
    # Start the conversion.
    echo -n "Converting $FILE_NAME to $TEMP_FILE: "
    # -loglevel quiet
    "$FFMPEG" -hide_banner -loglevel quiet -i "$FILE" $FFMPEG_ACTION $FFMPEG_SIZE "$TEMP_FILE" </dev/null 2>/dev/null 2>&1 &
    FFMPEG_PID=$!
    cpulimit -q -z -e ffmpeg -l $FFMPEG_CPU >/dev/null 2>/dev/null 2>&1 &
    T1=`date +%s`
    wait $FFMPEG_PID
    T2=`date +%s`
    RUNTIME=$(( T2 - T1 ))
    T_H=$((RUNTIME / 3600))
    T_M=$((RUNTIME % 3600 / 60 ))
    T_S=$(((RUNTIME % 3600) % 60 ))
    if [[ "$?" -eq 0 ]]; then
        echo "Done ($T_H:$T_M:$T_S)."
        # Permute the files on successful conversion.
        rm "$FILE"
        cp "$TEMP_FILE" "$DIR_NAME"
        if [ $? -ne 0 ]; then
            echo "Failed to copy converted file."
            continue
        fi
        rm "$TEMP_FILE"
    else
        echo "Failed."
    fi
done

Netstat without Netstat

The following code allows a user to print out a list of established connections without having to resort to using the command-line tool netstat by reading and processing /proc/net/tcp. There are other versions out there that use awk extensively in order to convert the hex values from /proc/net/tcp and even smaller than the provided script albeit at the cost of readability.

#!/usr/bin/env bash
###########################################################################
##  Copyright (C) Wizardry and Steamworks 2022 - License: GNU GPLv3      ##
##  Please see: http://www.gnu.org/licenses/gpl.html for legal details,  ##
##  rights of fair usage, the disclaimer and warranty conditions.        ##
###########################################################################
#                                                                         #
# This script processes the TCP connections from /proc/net/tcp and will   #
# output a list of locally listening TCP IP addresses and ports (sockets) #
# mapped to a list of remotely connecting TCP IP addresses and ports.     #
#                                                                         #
# The script is written with a minimalistic environment in mind with just #
# a few tools available that should be present within most busyboxes.     #
#                                                                         #
# For best results, disable IPv6 on the executing machine (via the kernel #
# command line) because some TCP connections are processed through IPv6   #
# even if they are IPv4 such that /proc/net/tcp might not include them.   #
#                                                                         #
###########################################################################
 
decode () {
    DUO=$1
    LEN=`echo -n $DUO | wc -c`
    OUT=""
    for X in `seq 0 2 $(($LEN-2))`; do
        OUT=$((16#${DUO:$X:2}))"."$OUT
    done
    OUT=${OUT::-1}
    echo $OUT
}
 
tcp_state() {
    OUT=""
    for SET in $1; do
        IP_HEX=`echo $SET | awk -F':' '{ print $1 }'`
        PO_HEX=`echo $SET | awk -F':' '{ print $2 }'`
        IP=`decode $IP_HEX`
        PO=$((16#$PO_HEX))
        OUT=$OUT" "$IP:$PO
    done
    echo $OUT
}
 
TMP_STATE=/tmp/tcp_state
trap '{ rm -rf $TMP_STATE; }' KILL QUIT TERM EXIT INT HUP
cat /proc/net/tcp >$TMP_STATE
 
TCP_STATE_LEN=`cat $TMP_STATE | wc -l`
LISTEN_LOCAL=`cat $TMP_STATE | \
    awk 'NR==1 { for (i=1;i<=NF;++i) if ($i=="local_address") { n=i; break }} { print $n }' | \
    tail +2`
REMOTE_ADDRS=`cat $TMP_STATE | \
    awk 'NR==1 { for (i=1;i<=NF;++i) if ($i=="rem_address") { n=i; break }} { print $n }' | 
    tail +2`
 
LISTEN=`tcp_state "$LISTEN_LOCAL"`
REMOTE=`tcp_state "$REMOTE_ADDRS"`
 
for I in `seq 1 1 $(($TCP_STATE_LEN-1))`; do
    A=`echo $LISTEN | awk -v var=$I '{ print $var }'`
    B=`echo $REMOTE | awk -v var=$I '{ print $var }'`
    echo $A" - "$B
done

Publishing Incoming TCP Connections to MQTT Broker

The following script is meant to run in the background and poll /proc/net/tcp for incoming TCP connections on a configurable port and then publish a JSON message to an MQTT broker with the following structure:

server -> FQDN hostname of the machine that the script is running on,
this -> the listening IP address followed by the port separated by a colon 
that -> the connecting IP address followed by the port separated by a colon  
#!/usr/bin/env bash
###########################################################################
##  Copyright (C) Wizardry and Steamworks 2022 - License: GNU GPLv3      ##
##  Please see: http://www.gnu.org/licenses/gpl.html for legal details,  ##
##  rights of fair usage, the disclaimer and warranty conditions.        ##
###########################################################################
#                                                                         #
# This script publishes the state of TCP connections to the machine over  #
# a specified port to an MQTT broker. The script leverages polling        #
# /proc/net/tcp for TCP connections instead of using netstat and uses     #
# just the most basic commands making it suitable for various cases.      #
#                                                                         #
###########################################################################
 
###########################################################################
#                           CONFIGURATION                                 #
###########################################################################
 
# path to the mosquitto publishing tool
MOSQUITTO_PUBLISHER=/storage/.local/usr/local/bin/mosquitto_pub
# the host name of the MQTT broker
MOSQUITTO_HOST=iot.internal
# the MQTT topic to publish on
MOSQUITTO_TOPIC=arcade
# the TCP port to monitor for connections
MONITOR_PORT=55435
 
# the time to sleep between polling
SLEEP_TIME=1
 
###########################################################################
#                             INTERNALS                                   #
###########################################################################
 
decode () {
    DUO=$1
    LEN=`echo -n $DUO | wc -c`
    OUT=""
    for X in `seq 0 2 $(($LEN-2))`; do
        OUT=$((16#${DUO:$X:2}))"."$OUT
    done
    OUT=${OUT::-1}
    echo $OUT
}
 
tcp_state() {
    OUT=""
    for SET in $1; do
        IP_HEX=`echo $SET | awk -F':' '{ print $1 }'`
        PO_HEX=`echo $SET | awk -F':' '{ print $2 }'`
        IP=`decode $IP_HEX`
        PO=$((16#$PO_HEX))
        OUT=$OUT" "$IP:$PO
    done
    echo $OUT
}
 
 
TMP_STATE=/dev/shm/tcp_state
RUN=1
trap '{ rm -rf $TMP_STATE; RUN=0; }' KILL QUIT TERM EXIT INT HUP SIGUSR1
 
while [ $RUN -ne 0 ]; do
    cat /proc/net/tcp >$TMP_STATE
    TCP_STATE_LEN=`cat $TMP_STATE | wc -l`
    LISTEN_LOCAL=`cat $TMP_STATE | \
        awk 'NR==1 { for (i=1;i<=NF;++i) if ($i=="local_address") { n=i; break }} { print $n }' | \
        tail +2`
    REMOTE_ADDRS=`cat $TMP_STATE | \
        awk 'NR==1 { for (i=1;i<=NF;++i) if ($i=="rem_address") { n=i; break }} { print $n }' | 
        tail +2`
 
    LISTEN=`tcp_state "$LISTEN_LOCAL"`
    REMOTE=`tcp_state "$REMOTE_ADDRS"`
 
    for I in `seq 1 1 $(($TCP_STATE_LEN-1))`; do
        X=`echo $LISTEN | awk -v var=$I '{ print $var }'`
        Y=`echo $REMOTE | awk -v var=$I '{ print $var }'`
        NETPLAY=`echo $X $Y | grep $MONITOR_PORT | grep -v 0.0.0.0`
        if [ ! -z "$NETPLAY" ]; then
            THIS=`echo $NETPLAY | awk '{ print $1 }'`
            THAT=`echo $NETPLAY | awk '{ print $2 }'`
 
            $MOSQUITTO_PUBLISHER \
                -h $MOSQUITTO_HOST \
                -t $MOSQUITTO_TOPIC \
                -m "{ \"origin\": \"`hostname -f`\", \"this\": \"$THIS\", \"that\": \"$THAT\" }"
        fi
    done
    sleep $SLEEP_TIME
done

Text Activity Spinners in Bash

In order to use the text activity spinners in bash, here is an example script:

#!/usr/bin/env bash
 
ASCII_SPINNER=( ".oOo" "oOo." "Oo.o" "o.oO" )
while [ 1 = 1 ]; do
    CAR="${ASCII_SPINNER[0]}"
    CDR=("${ASCII_SPINNER[@]:1:${#ASCII_SPINNER[@]}}")
    ASCII_SPINNER=(${CDR[@]} ${CAR})
    echo -n -e "\r${CAR}"
done

The logic is pretty straight-forward:

  • the spinner is loaded as a bash array,
  • the array is shuffled by shifting the first element, keeping a copy it and then adding the first element to the end of the spinner array,
  • the first element is printed out, without any newlines and by prepending a carriage-return \r in order to make sure that the next element in the spinner array will overwrite the old element

fuss/bash.txt · Last modified: 2023/10/15 10:16 by office

Access website using Tor Access website using i2p Wizardry and Steamworks PGP Key


For the contact, copyright, license, warranty and privacy terms for the usage of this website please see the contact, license, privacy, copyright.