Differences

This shows you the differences between two versions of the page.

--- fuss:bash [2017/02/22 18:30] – external edit 127.0.0.1
+++ fuss:bash [2025/07/03 10:41] (current) – [Pipe Heredoc] office
@@ Line 22: / Line 22: @@
-====== Increment in Bash Arrays ======
+====== Increment in Bash s ======
 <code bash>
@@ Line 57: / Line 57: @@
 </code>
-====== Split a String to an Array and other Array Manipulations ======
+====== Split a String to an  and other  Manipulations ======
 The following code:
@@ Line 120: / Line 120: @@
 in order to retrieve the temperature of the first core (''0'').
+====== Get a List of Files by File Extension from an Array ======
+Via a function ''listFiles'':
+<code bash>
+# By: https://stackoverflow.com/users/1773798/renaud-pacalet
+listFiles() {
+    local -n _OUTPUT_="$1"
+    local -n _DIRECTORY_="$2"
+    local -n _TYPES_="$3"
+    local _FILTER_
+    _FILTER_="${_TYPES_[@]/#/ -o -name *.}"
+    _FILTER_="${_FILTER_# -o }"
+    while read -d $'\0' FILE; do
+        _OUTPUT_+=( "$FILE" )
+    done < <( find "$_DIRECTORY_" -type f \( $_FILTER_ \) -print0 )
+}
+</code>
+Example call:
+<code bash>
+DIR_PATH=/mnt/Movies
+TYPES=(mkv mp4 avi)
+listFiles FILES DIR_PATH TYPES
+</code>
+====== Recursively Convert Video Files ======
+As the title would imply, the following script will recursively convert video files to conform to the following parameters:
+  * MPEG-4
+  * 720p or less,
+  * AVC (H.264),
+  * AAC
+Its main design was meant to convert recorded TV shows where a set quality should not necessarily exceed the mentioned parameters for the sake of storage conservation.
+<code bash>
+#!/bin/bash
+###########################################################################
+##  Copyright (C) Wizardry and Steamworks 2021 - License: GNU GPLv3      ##
+###########################################################################
+# The script is a way of converting all video files recursively in a      #
+# directory tree in order to conform to the following parameters:         #
+#  * MPEG-4 (MP4)                                                         #
+#  * 480p or less                                                         #
+#  * AVC (H.264)                                                          #
+#  * AAC                                                                  #
+#                                                                         #
+# Notes:                                                                  #
+#  * There are some added optimizations such as switching to just         #
+#    converting to MPEG-4 in case no more than a container change is      #
+#    necessary  (ie: MKV to MP4).                                         #
+#  * This is a long-running script and best ran within a detachable       #
+#    terminal such as tmux or screen.                                     #
+#  * This is a bash script that will not run on other shells.             #
+###########################################################################
+###########################################################################
+##                            CONFIGURATION                              ##
+###########################################################################
+# The path to the ffmpeg binary.
+FFMPEG='/usr/bin/ffmpeg'
+# The ffmpeg flags to use when converting.
+# veryfast vs. veryslow https://write.corbpie.com/ffmpeg-preset-comparison-x264-2019-encode-speed-and-file-size/
+FFMPEG_OPTIONS=`tr -d '\012' <<'EOF'
+-c:v libx264 \
+-crf 23 \
+-level 3.1 \
+-preset veryfast \
+-tune film \
+-sws_flags lanczos \
+-sn \
+-c:a libfdk_aac \
+-vbr 5 \
+-q:a 2 \
+-ac 2 \
+-b:a 128k \
+-c:s mov_text \
+-map 0:v:0 \
+-map 0:a:0 \
+-map_metadata -1 \
+-movflags +faststart \
+-strict -2
+EOF`
+# The maximal height of the video files.
+FFMPEG_HEIGHT=480
+# The maximal integral CPU percentage that will be allowed to be used.
+FFMPEG_CPU=600
+# The file types to convert.
+TYPES=(mp4 mkv avi)
+###########################################################################
+##                              INTERNALS                                ##
+###########################################################################
+if [ -z "$1" ] || [ ! -d "$1" ]; then
+    echo "SYNTAX: $0 <DIRECTORY>"
+    exit 1
+fi
+DIR_PATH="$1"
+# By: https://stackoverflow.com/users/1773798/renaud-pacalet
+listFiles() {
+    local -n _OUTPUT_="$1"
+    local -n _DIRECTORY_="$2"
+    local -n _TYPES_="$3"
+    local _FILTER_
+    _FILTER_="${_TYPES_[@]/#/ -o -name *.}"
+    _FILTER_="${_FILTER_# -o }"
+    while read -d $'\0' FILE; do
+        _OUTPUT_+=( "$FILE" )
+    done < <( find "$_DIRECTORY_" -type f \( $_FILTER_ \) -print0 )
+}
+TEMP_FILE=""
+# Cleanup on file termination.
+trap '{
+    rm -rf "$TEMP_FILE" 2>&1 2>/dev/null
+    exit 0
+}' KILL QUIT TERM EXIT INT HUP
+# Iterate over all matching files.
+FILES=()
+listFiles FILES DIR_PATH TYPES
+for FILE in "${FILES[@]}"; do
+    FILE_NAME=`basename "$FILE"`
+    DIR_NAME=`dirname "$FILE"`
+    FILE_EXTENSION=`echo "$FILE" | awk -F'.' '{ print $NF }'`
+    TEMP_FILE="/tmp/${FILE_NAME/$FILE_EXTENSION/mp4}"
+    INFO=`mediainfo "$FILE" | awk -F':' '{ print $1":"$2 }' | awk '{$1=$1};1'`
+    HAS_AVC=`echo "$INFO" | grep 'Format : AVC'`
+    HAS_AAC=`echo "$INFO" | grep 'Format : AAC LC'`
+    HAS_MP4=`echo "$INFO" | grep 'Format : MPEG-4'`
+    HAS_2CH=`echo "$INFO" | grep 'Channel(s) : 2'`
+    HAS_TXT=`mediainfo $FILE | grep "^Text"`
+    HEIGHT=`mediainfo --Inform="Video;%Height%" "$FILE"`
+    # If all criteria matches then proceed to the next file.
+    if [[ $HEIGHT -le $FFMPEG_HEIGHT ]] && \
+       [[ ! -z $HAS_AVC ]] && \
+       [[ ! -z $HAS_AAC ]] && \
+       [[ ! -z $HAS_MP4 ]] && \
+       [[ ! -z $HAS_2CH ]] && \
+       [[ -z $HAS_TXT ]]
+    then
+    	echo "Skipping $FILE_NAME..."
+    	continue
+    fi
+    FFMPEG_ACTION=$FFMPEG_OPTIONS
+    # Resize down to $FFMPEG_HEIGHT or less depending on the height of the file.
+    if [[ $HEIGHT -gt $FFMPEG_HEIGHT ]]; then
+        FFMPEG_SIZE="-vf scale=-2:$FFMPEG_HEIGHT"
+    fi
+    # Start the conversion.
+    echo -n "Converting $FILE_NAME to $TEMP_FILE: "
+    # -loglevel quiet
+    "$FFMPEG" -hide_banner -loglevel quiet -i "$FILE" $FFMPEG_ACTION $FFMPEG_SIZE "$TEMP_FILE" </dev/null 2>/dev/null 2>&1 &
+    FFMPEG_PID=$!
+    cpulimit -q -z -e ffmpeg -l $FFMPEG_CPU >/dev/null 2>/dev/null 2>&1 &
+    T1=`date +%s`
+    wait $FFMPEG_PID
+    T2=`date +%s`
+    RUNTIME=$(( T2 - T1 ))
+    T_H=$((RUNTIME / 3600))
+    T_M=$((RUNTIME % 3600 / 60 ))
+    T_S=$(((RUNTIME % 3600) % 60 ))
+    if [[ "$?" -eq 0 ]]; then
+        echo "Done ($T_H:$T_M:$T_S)."
+        # Permute the files on successful conversion.
+        rm "$FILE"
+        cp "$TEMP_FILE" "$DIR_NAME"
+        if [ $? -ne 0 ]; then
+            echo "Failed to copy converted file."
+            continue
+        fi
+        rm "$TEMP_FILE"
+    else
+        echo "Failed."
+    fi
+done
+</code>
+====== Netstat without Netstat ======
+The following code allows a user to print out a list of established connections without having to resort to using the command-line tool ''netstat'' by reading and processing ''/proc/net/tcp''. There are other versions out there that use ''awk'' extensively in order to convert the hex values from ''/proc/net/tcp'' and even smaller than the provided script albeit at the cost of readability.
+<code bash>
+#!/usr/bin/env bash
+###########################################################################
+##  Copyright (C) Wizardry and Steamworks 2022 - License: GNU GPLv3      ##
+##  Please see: http://www.gnu.org/licenses/gpl.html for legal details,  ##
+##  rights of fair usage, the disclaimer and warranty conditions.        ##
+###########################################################################
+#                                                                         #
+# This script processes the TCP connections from /proc/net/tcp and will   #
+# output a list of locally listening TCP IP addresses and ports (sockets) #
+# mapped to a list of remotely connecting TCP IP addresses and ports.     #
+#                                                                         #
+# The script is written with a minimalistic environment in mind with just #
+# a few tools available that should be present within most busyboxes.     #
+#                                                                         #
+# For best results, disable IPv6 on the executing machine (via the kernel #
+# command line) because some TCP connections are processed through IPv6   #
+# even if they are IPv4 such that /proc/net/tcp might not include them.   #
+#                                                                         #
+###########################################################################
+decode () {
+    DUO=$1
+    LEN=`echo -n $DUO | wc -c`
+    OUT=""
+    for X in `seq 0 2 $(($LEN-2))`; do
+        OUT=$((16#${DUO:$X:2}))"."$OUT
+    done
+    OUT=${OUT::-1}
+    echo $OUT
+}
+tcp_state() {
+    OUT=""
+    for SET in $1; do
+        IP_HEX=`echo $SET | awk -F':' '{ print $1 }'`
+        PO_HEX=`echo $SET | awk -F':' '{ print $2 }'`
+        IP=`decode $IP_HEX`
+        PO=$((16#$PO_HEX))
+        OUT=$OUT" "$IP:$PO
+    done
+    echo $OUT
+}
+TMP_STATE=/tmp/tcp_state
+trap '{ rm -rf $TMP_STATE; }' KILL QUIT TERM EXIT INT HUP
+cat /proc/net/tcp >$TMP_STATE
+TCP_STATE_LEN=`cat $TMP_STATE | wc -l`
+LISTEN_LOCAL=`cat $TMP_STATE | \
+    awk 'NR==1 { for (i=1;i<=NF;++i) if ($i=="local_address") { n=i; break }} { print $n }' | \
+    tail +2`
+REMOTE_ADDRS=`cat $TMP_STATE | \
+    awk 'NR==1 { for (i=1;i<=NF;++i) if ($i=="rem_address") { n=i; break }} { print $n }' |
+    tail +2`
+LISTEN=`tcp_state "$LISTEN_LOCAL"`
+REMOTE=`tcp_state "$REMOTE_ADDRS"`
+for I in `seq 1 1 $(($TCP_STATE_LEN-1))`; do
+    A=`echo $LISTEN | awk -v var=$I '{ print $var }'`
+    B=`echo $REMOTE | awk -v var=$I '{ print $var }'`
+    echo $A" - "$B
+done
+</code>
+====== Publishing Incoming TCP Connections to MQTT Broker ======
+The following script is meant to run in the background and poll ''/proc/net/tcp'' for incoming TCP connections on a configurable port and then publish a JSON message to an MQTT broker with the following structure:
+<code>
+server -> FQDN hostname of the machine that the script is running on,
+this -> the listening IP address followed by the port separated by a colon
+that -> the connecting IP address followed by the port separated by a colon
+</code>
+<code bash>
+#!/usr/bin/env bash
+###########################################################################
+##  Copyright (C) Wizardry and Steamworks 2022 - License: GNU GPLv3      ##
+##  Please see: http://www.gnu.org/licenses/gpl.html for legal details,  ##
+##  rights of fair usage, the disclaimer and warranty conditions.        ##
+###########################################################################
+#                                                                         #
+# This script publishes the state of TCP connections to the machine over  #
+# a specified port to an MQTT broker. The script leverages polling        #
+# /proc/net/tcp for TCP connections instead of using netstat and uses     #
+# just the most basic commands making it suitable for various cases.      #
+#                                                                         #
+###########################################################################
+###########################################################################
+#                           CONFIGURATION                                 #
+###########################################################################
+# path to the mosquitto publishing tool
+MOSQUITTO_PUBLISHER=/storage/.local/usr/local/bin/mosquitto_pub
+# the host name of the MQTT broker
+MOSQUITTO_HOST=iot.internal
+# the MQTT topic to publish on
+MOSQUITTO_TOPIC=arcade
+# the TCP port to monitor for connections
+MONITOR_PORT=55435
+# the time to sleep between polling
+SLEEP_TIME=1
+###########################################################################
+#                             INTERNALS                                   #
+###########################################################################
+decode () {
+    DUO=$1
+    LEN=`echo -n $DUO | wc -c`
+    OUT=""
+    for X in `seq 0 2 $(($LEN-2))`; do
+        OUT=$((16#${DUO:$X:2}))"."$OUT
+    done
+    OUT=${OUT::-1}
+    echo $OUT
+}
+tcp_state() {
+    OUT=""
+    for SET in $1; do
+        IP_HEX=`echo $SET | awk -F':' '{ print $1 }'`
+        PO_HEX=`echo $SET | awk -F':' '{ print $2 }'`
+        IP=`decode $IP_HEX`
+        PO=$((16#$PO_HEX))
+        OUT=$OUT" "$IP:$PO
+    done
+    echo $OUT
+}
+TMP_STATE=/dev/shm/tcp_state
+RUN=1
+trap '{ rm -rf $TMP_STATE; RUN=0; }' KILL QUIT TERM EXIT INT HUP SIGUSR1
+while [ $RUN -ne 0 ]; do
+    cat /proc/net/tcp >$TMP_STATE
+    TCP_STATE_LEN=`cat $TMP_STATE | wc -l`
+    LISTEN_LOCAL=`cat $TMP_STATE | \
+        awk 'NR==1 { for (i=1;i<=NF;++i) if ($i=="local_address") { n=i; break }} { print $n }' | \
+        tail +2`
+    REMOTE_ADDRS=`cat $TMP_STATE | \
+        awk 'NR==1 { for (i=1;i<=NF;++i) if ($i=="rem_address") { n=i; break }} { print $n }' |
+        tail +2`
+    LISTEN=`tcp_state "$LISTEN_LOCAL"`
+    REMOTE=`tcp_state "$REMOTE_ADDRS"`
+    for I in `seq 1 1 $(($TCP_STATE_LEN-1))`; do
+        X=`echo $LISTEN | awk -v var=$I '{ print $var }'`
+        Y=`echo $REMOTE | awk -v var=$I '{ print $var }'`
+        NETPLAY=`echo $X $Y | grep $MONITOR_PORT | grep -v 0.0.0.0`
+        if [ ! -z "$NETPLAY" ]; then
+            THIS=`echo $NETPLAY | awk '{ print $1 }'`
+            THAT=`echo $NETPLAY | awk '{ print $2 }'`
+            $MOSQUITTO_PUBLISHER \
+                -h $MOSQUITTO_HOST \
+                -t $MOSQUITTO_TOPIC \
+                -m "{ \"origin\": \"`hostname -f`\", \"this\": \"$THIS\", \"that\": \"$THAT\" }"
+        fi
+    done
+    sleep $SLEEP_TIME
+done
+</code>
+====== Text Activity Spinners in Bash ======
+In order to use the [[/assets/databases/spinners/text|text activity spinners in bash]], here is an example script:
+<code bash>
+#!/usr/bin/env bash
+ASCII_SPINNER=( ".oOo" "oOo." "Oo.o" "o.oO" )
+while [ 1 = 1 ]; do
+    CAR="${ASCII_SPINNER[0]}"
+    CDR=("${ASCII_SPINNER[@]:1:${#ASCII_SPINNER[@]}}")
+    ASCII_SPINNER=(${CDR[@]} ${CAR})
+    echo -n -e "\r${CAR}"
+done
+</code>
+The logic is pretty straight-forward:
+  * the spinner is loaded as a bash array,
+  * the array is shuffled by shifting the first element, keeping a copy it and then adding the first element to the end of the spinner array,
+  * the first element is printed out, without any newlines and by prepending a carriage-return ''\r'' in order to make sure that the next element in the spinner array will overwrite the old element
+====== Snippet that Generates Subtitles by Reading a File ======
+The following script fragment:
+<code bash>
+COUNT=0 && while [ 1 ]; do while read i; do echo -e "$COUNT\n$(date -d@$COUNT -u +%H:%M:%S,000) -> $(date -d@$((COUNT+1)) -u +%H:%M:%S,000)\n$i" && COUNT=$((COUNT+1)) && sleep 1; done <noise.txt; done
+</code>
+will poll the contents of the file ''noise.txt'' in the current directory every second (''sleep 1'') and then output text formatted in SRT format to the console.
+Here is some example output:
+<code>
+:00:01,000 --> 00:00:02,000
+dB ± 1.5
+:00:02,000 --> 00:00:03,000
+dB ± 1.5
+:00:03,000 --> 00:00:04,000
+dB ± 1.5
+:00:04,000 --> 00:00:05,000
+dB ± 1.5
+:00:05,000 --> 00:00:06,000
+dB ± 1.5
+:00:06,000 --> 00:00:07,000
+dB ± 1.5
+:00:07,000 --> 00:00:08,000
+dB ± 1.5
+:00:08,000 --> 00:00:09,000
+dB ± 1.5
+:00:09,000 --> 00:00:10,000
+dB ± 1.5
+:00:10,000 --> 00:00:11,000
+dB ± 1.5
+:00:11,000 --> 00:00:12,000
+dB ± 1.5
+:00:12,000 --> 00:00:13,000
+dB ± 1.5
+:00:13,000 --> 00:00:14,000
+dB ± 1.5
+:00:14,000 --> 00:00:15,000
+dB ± 1.5
+:00:15,000 --> 00:00:16,000
+dB ± 1.5
+</code>
+The snippet is useful, for example, if some tool is running at the same time in the background (ie: ffmpeg or mencoder) that records some live stream, and then this script can be used to generate subtitles for the live stream being recorded.
+For example, if the file being generated by recording the live stream is named ''stream.mkv'' in the current directory, then the previous command can be piped to a similarly named file, ''stream.sub'' also in the current directory:
+<code bash>
+... > stream.sub
+</code>
+such that when ''stream.mkv'' is opened, most players will also automatically load ''stream.sub''.
+====== Pipe Heredoc ======
+Heredoc can be used to include block-text in a bash script listing verbatim and sometimes it is nifty to be able to pipe the included block-text to a command. For example, imagine that one would want to include a script inside a Dockerfile to be expanded when the image is built without involving any filesystem files to be included with the ''ADD'' directive. In such a case, the formula would be:
+  * write the script,
+  * encode the script to Base64,
+  * include the Base64 in the Dockerfile using heredoc and pipe it to a command that decodes the Base64 text and saves it to a file
+Here is what it would look like within a Dockerfile:
+<code>
+RUN mkdir -p /usr/local/bin/ && base64 -d >/usr/local/bin/run <<'EOT'
+IyEvdXNyL2Jpbi9lbnYgYmFzaAoKIyBkZWZpbmUgZGVmYXVsdCBwYXJhbWV0ZXJzClRPUl9TT0NL
+PSIke1RPUl9ETlNfUE9SVDotMC4wLjAuMDo5MDUzfSIKQ0hFQ0tfQ0lSQ1VJVF9QT1JUPSIke0NI
+RUNLX0NJUkNVSVRfUE9SVDotNzA1MH0iCkpBM19QT1JUPSIke0pBM19QT1JUOi05MDQwfSIKIyAx
+...
+EOT
+</code>
+which is meant to perform the following via the ''RUN'' Dockerfile command:
+  * creates the directory ''/usr/local/bin/''
+  * reads the large block of text with seemingly random characters using heredoc up to the ''EOT'' terminator as input for the ''base64'' command that, in turn, decodes the Base64 payload and saves the plaintext to ''/usr/local/bin/run''
+Counter-intuitively one would want to echo or ''cat'' the payload to ''base64'' but the trick here is that a call to ''cat'' would have been unnecessary and that heredoc is used to place the Base64 payload onto standard input for the ''base64'' command.
+Note that the quotes around the ''EOT'' keyword mean that any syntax that looks like variable definitions within the payload shall not be interpolated. Leaving the quotes out would have allowed variables to be used within the payload.