Differences

This shows you the differences between two versions of the page.

--- fuss:bash [2016/01/10 21:59] – [Get the CPU Core Temperature File] office
+++ fuss:bash [2023/10/15 10:16] (current) – [Publishing Incoming TCP Connections to MQTT Broker] office
@@ Line 1: / Line 1: @@
+====== Glob ======
+''Glob'' is a fast shell-based equivalent of regular expressions.
+^ Glob ^ Explanation ^
+| ''*'' | Matches any string, of any length |
+| ''foo*'' | Matches any string beginning with ''foo'' |
+| ''*x*'' | Matches any string containing an ''x'' (beginning, middle or end) |
+| ''*.tar.gz'' | Matches any string ending with ''.tar.gz'' |
+| ''foo?'' | Matches ''foot'' or ''foo$'' but not ''fools'' |
+===== Range =====
+^ Glob   ^ Explanation ^
+| ''[abcd]'' | Matches ''a'', ''b'', ''c'' and ''d'' |
+| ''[a-d]'' | The same as above, if your locale is ''C'' or ''POSIX''. Otherwise, implementation-defined. |
+| ''[!aeiouAEIOU]'' | Matches any character except ''a'', ''e'', ''i'', ''o'', ''u'' and their uppercase counterparts |
+| ''<nowiki>[[:alnum:]]</nowiki>'' | Matches any alphanumeric character in the current locale (letter or number) |
+| ''<nowiki>[[:space:]]</nowiki>'' | Matches any whitespace character |
+| ''<nowiki>[![:space:]]</nowiki>'' | Matches any character that is not whitespace |
+| ''<nowiki>[[:digit:]_.]</nowiki>'' | Matches any digit, or ''_'' or ''.'' |
+====== Increment in Bash s ======
+<code bash>
+array[$[${#array[@]}+1]] = 3
+</code>
+is homologous to:
+<code bash>
+array[elements_in(array)+1] = 3
+</code>
+====== String Substitute ======
+''Input'':
+<code bash>
+T="test.svg"
+echo ${T/svg/png}
+</code>
+''Output'':
+<code bash>
+test.png
+</code>
+====== Check if String Contains a Substring ======
+<code bash>
+CHECK_STRING="google"
+if [[ "$CHECK_STRING" != *o* ]]; then
+  echo "character o is in the string"
+fi
+</code>
+====== Split a String to an  and other  Manipulations ======
+The following code:
+<code bash>
+IFS=' ' read -ra VAR_ARRAY <<< "$input"
+</code>
+where:
+  * ''IFS'' will switch the separator (the character to split on) to a space.
+  * ''read'' will read-in the input
+  * ''VAR_ARRAY'' is the array that will be filled with the values from the input split by the separator
+  * ''$input'' is the input to split
+will split the input to an array.
+In order to iterate over the elements of the array, one would write:
+<code bash>
+for e in "${VAR_ARRAY[@]}"; do
+    echo $e
+done
+</code>
+which will print out all the elements in the array.
+To get both the index and the value while iterating over the array, one would write:
+<code bash>
+for i in "${!VAR_ARRAY[@]}"; do
+    echo "$i ${VAR_ARRAY[i]}"
+done
+</code>
+where on every iteration ''$i'' is the index and ''${VAR_ARRAY[i]}'' will expand to the array element at ''i''.
+To get the number of elements in the array, one would write:
+<code bash>
+echo "${VAR_ARRAY[@]}"
+</code>
+====== Get the CPU Core Temperature File ======
+If you use ''coretemp'' to retrieve the CPU sensors temperature, a bunch of files are created in ''/sys/devices/platform/coretemp.*/hwmon/hwmon*/'' such as ''temp1_label'', ''temp2_label'', etc... Each of them represent the temperature of one of the cores but there is no guarantee that the numbering will start at ''1''. The following bash function reads the files in ''/sys/devices/platform/coretemp.*/hwmon/hwmon*/'' and returns the file corresponding to a certain core file:
+<code bash>
+###########################################################################
+##  Copyright (C) Wizardry and Steamworks 2016 - License: GNU GPLv3      ##
+###########################################################################
+function getCoreTemperatureFile {
+    for i in /sys/devices/platform/coretemp.*/hwmon/hwmon*/*_label; do
+        IFS=' ' read -ra CORE <<< `cat $i`
+        if [ "${CORE[0]}" == "Core" ] && [ "${CORE[1]}" == $1 ]; then
+            echo "${i//label/input}"
+        fi
+    done
+}
+</code>
+an example call, would then be:
+<code bash>
+CPU0_TEMPERATURE_FILE=$(getCoreTemperatureFile 0)
+echo $CPU0_TEMPERATURE_FILE
+</code>
+in order to retrieve the temperature of the first core (''0'').
+====== Get a List of Files by File Extension from an Array ======
+Via a function ''listFiles'':
+<code bash>
+# By: https://stackoverflow.com/users/1773798/renaud-pacalet
+listFiles() {
+    local -n _OUTPUT_="$1"
+    local -n _DIRECTORY_="$2"
+    local -n _TYPES_="$3"
+    local _FILTER_
+    _FILTER_="${_TYPES_[@]/#/ -o -name *.}"
+    _FILTER_="${_FILTER_# -o }"
+    while read -d $'\0' FILE; do
+        _OUTPUT_+=( "$FILE" )
+    done < <( find "$_DIRECTORY_" -type f \( $_FILTER_ \) -print0 )
+}
+</code>
+Example call:
+<code bash>
+DIR_PATH=/mnt/Movies
+TYPES=(mkv mp4 avi)
+listFiles FILES DIR_PATH TYPES
+</code>
+====== Recursively Convert Video Files ======
+As the title would imply, the following script will recursively convert video files to conform to the following parameters:
+  * MPEG-4
+  * 720p or less,
+  * AVC (H.264),
+  * AAC
+Its main design was meant to convert recorded TV shows where a set quality should not necessarily exceed the mentioned parameters for the sake of storage conservation.
+<code bash>
+#!/bin/bash
+###########################################################################
+##  Copyright (C) Wizardry and Steamworks 2021 - License: GNU GPLv3      ##
+###########################################################################
+# The script is a way of converting all video files recursively in a      #
+# directory tree in order to conform to the following parameters:         #
+#  * MPEG-4 (MP4)                                                         #
+#  * 480p or less                                                         #
+#  * AVC (H.264)                                                          #
+#  * AAC                                                                  #
+#                                                                         #
+# Notes:                                                                  #
+#  * There are some added optimizations such as switching to just         #
+#    converting to MPEG-4 in case no more than a container change is      #
+#    necessary  (ie: MKV to MP4).                                         #
+#  * This is a long-running script and best ran within a detachable       #
+#    terminal such as tmux or screen.                                     #
+#  * This is a bash script that will not run on other shells.             #
+###########################################################################
+###########################################################################
+##                            CONFIGURATION                              ##
+###########################################################################
+# The path to the ffmpeg binary.
+FFMPEG='/usr/bin/ffmpeg'
+# The ffmpeg flags to use when converting.
+# veryfast vs. veryslow https://write.corbpie.com/ffmpeg-preset-comparison-x264-2019-encode-speed-and-file-size/
+FFMPEG_OPTIONS=`tr -d '\012' <<'EOF'
+-c:v libx264 \
+-crf 23 \
+-level 3.1 \
+-preset veryfast \
+-tune film \
+-sws_flags lanczos \
+-sn \
+-c:a libfdk_aac \
+-vbr 5 \
+-q:a 2 \
+-ac 2 \
+-b:a 128k \
+-c:s mov_text \
+-map 0:v:0 \
+-map 0:a:0 \
+-map_metadata -1 \
+-movflags +faststart \
+-strict -2
+EOF`
+# The maximal height of the video files.
+FFMPEG_HEIGHT=480
+# The maximal integral CPU percentage that will be allowed to be used.
+FFMPEG_CPU=600
+# The file types to convert.
+TYPES=(mp4 mkv avi)
+###########################################################################
+##                              INTERNALS                                ##
+###########################################################################
+if [ -z "$1" ] || [ ! -d "$1" ]; then
+    echo "SYNTAX: $0 <DIRECTORY>"
+    exit 1
+fi
+DIR_PATH="$1"
+# By: https://stackoverflow.com/users/1773798/renaud-pacalet
+listFiles() {
+    local -n _OUTPUT_="$1"
+    local -n _DIRECTORY_="$2"
+    local -n _TYPES_="$3"
+    local _FILTER_
+    _FILTER_="${_TYPES_[@]/#/ -o -name *.}"
+    _FILTER_="${_FILTER_# -o }"
+    while read -d $'\0' FILE; do
+        _OUTPUT_+=( "$FILE" )
+    done < <( find "$_DIRECTORY_" -type f \( $_FILTER_ \) -print0 )
+}
+TEMP_FILE=""
+# Cleanup on file termination.
+trap '{
+    rm -rf "$TEMP_FILE" 2>&1 2>/dev/null
+    exit 0
+}' KILL QUIT TERM EXIT INT HUP
+# Iterate over all matching files.
+FILES=()
+listFiles FILES DIR_PATH TYPES
+for FILE in "${FILES[@]}"; do
+    FILE_NAME=`basename "$FILE"`
+    DIR_NAME=`dirname "$FILE"`
+    FILE_EXTENSION=`echo "$FILE" | awk -F'.' '{ print $NF }'`
+    TEMP_FILE="/tmp/${FILE_NAME/$FILE_EXTENSION/mp4}"
+    INFO=`mediainfo "$FILE" | awk -F':' '{ print $1":"$2 }' | awk '{$1=$1};1'`
+    HAS_AVC=`echo "$INFO" | grep 'Format : AVC'`
+    HAS_AAC=`echo "$INFO" | grep 'Format : AAC LC'`
+    HAS_MP4=`echo "$INFO" | grep 'Format : MPEG-4'`
+    HAS_2CH=`echo "$INFO" | grep 'Channel(s) : 2'`
+    HAS_TXT=`mediainfo $FILE | grep "^Text"`
+    HEIGHT=`mediainfo --Inform="Video;%Height%" "$FILE"`
+    # If all criteria matches then proceed to the next file.
+    if [[ $HEIGHT -le $FFMPEG_HEIGHT ]] && \
+       [[ ! -z $HAS_AVC ]] && \
+       [[ ! -z $HAS_AAC ]] && \
+       [[ ! -z $HAS_MP4 ]] && \
+       [[ ! -z $HAS_2CH ]] && \
+       [[ -z $HAS_TXT ]]
+    then
+    	echo "Skipping $FILE_NAME..."
+    	continue
+    fi
+    FFMPEG_ACTION=$FFMPEG_OPTIONS
+    # Resize down to $FFMPEG_HEIGHT or less depending on the height of the file.
+    if [[ $HEIGHT -gt $FFMPEG_HEIGHT ]]; then
+        FFMPEG_SIZE="-vf scale=-2:$FFMPEG_HEIGHT"
+    fi
+    # Start the conversion.
+    echo -n "Converting $FILE_NAME to $TEMP_FILE: "
+    # -loglevel quiet
+    "$FFMPEG" -hide_banner -loglevel quiet -i "$FILE" $FFMPEG_ACTION $FFMPEG_SIZE "$TEMP_FILE" </dev/null 2>/dev/null 2>&1 &
+    FFMPEG_PID=$!
+    cpulimit -q -z -e ffmpeg -l $FFMPEG_CPU >/dev/null 2>/dev/null 2>&1 &
+    T1=`date +%s`
+    wait $FFMPEG_PID
+    T2=`date +%s`
+    RUNTIME=$(( T2 - T1 ))
+    T_H=$((RUNTIME / 3600))
+    T_M=$((RUNTIME % 3600 / 60 ))
+    T_S=$(((RUNTIME % 3600) % 60 ))
+    if [[ "$?" -eq 0 ]]; then
+        echo "Done ($T_H:$T_M:$T_S)."
+        # Permute the files on successful conversion.
+        rm "$FILE"
+        cp "$TEMP_FILE" "$DIR_NAME"
+        if [ $? -ne 0 ]; then
+            echo "Failed to copy converted file."
+            continue
+        fi
+        rm "$TEMP_FILE"
+    else
+        echo "Failed."
+    fi
+done
+</code>
+====== Netstat without Netstat ======
+The following code allows a user to print out a list of established connections without having to resort to using the command-line tool ''netstat'' by reading and processing ''/proc/net/tcp''. There are other versions out there that use ''awk'' extensively in order to convert the hex values from ''/proc/net/tcp'' and even smaller than the provided script albeit at the cost of readability.
+<code bash>
+#!/usr/bin/env bash
+###########################################################################
+##  Copyright (C) Wizardry and Steamworks 2022 - License: GNU GPLv3      ##
+##  Please see: http://www.gnu.org/licenses/gpl.html for legal details,  ##
+##  rights of fair usage, the disclaimer and warranty conditions.        ##
+###########################################################################
+#                                                                         #
+# This script processes the TCP connections from /proc/net/tcp and will   #
+# output a list of locally listening TCP IP addresses and ports (sockets) #
+# mapped to a list of remotely connecting TCP IP addresses and ports.     #
+#                                                                         #
+# The script is written with a minimalistic environment in mind with just #
+# a few tools available that should be present within most busyboxes.     #
+#                                                                         #
+# For best results, disable IPv6 on the executing machine (via the kernel #
+# command line) because some TCP connections are processed through IPv6   #
+# even if they are IPv4 such that /proc/net/tcp might not include them.   #
+#                                                                         #
+###########################################################################
+decode () {
+    DUO=$1
+    LEN=`echo -n $DUO | wc -c`
+    OUT=""
+    for X in `seq 0 2 $(($LEN-2))`; do
+        OUT=$((16#${DUO:$X:2}))"."$OUT
+    done
+    OUT=${OUT::-1}
+    echo $OUT
+}
+tcp_state() {
+    OUT=""
+    for SET in $1; do
+        IP_HEX=`echo $SET | awk -F':' '{ print $1 }'`
+        PO_HEX=`echo $SET | awk -F':' '{ print $2 }'`
+        IP=`decode $IP_HEX`
+        PO=$((16#$PO_HEX))
+        OUT=$OUT" "$IP:$PO
+    done
+    echo $OUT
+}
+TMP_STATE=/tmp/tcp_state
+trap '{ rm -rf $TMP_STATE; }' KILL QUIT TERM EXIT INT HUP
+cat /proc/net/tcp >$TMP_STATE
+TCP_STATE_LEN=`cat $TMP_STATE | wc -l`
+LISTEN_LOCAL=`cat $TMP_STATE | \
+    awk 'NR==1 { for (i=1;i<=NF;++i) if ($i=="local_address") { n=i; break }} { print $n }' | \
+    tail +2`
+REMOTE_ADDRS=`cat $TMP_STATE | \
+    awk 'NR==1 { for (i=1;i<=NF;++i) if ($i=="rem_address") { n=i; break }} { print $n }' |
+    tail +2`
+LISTEN=`tcp_state "$LISTEN_LOCAL"`
+REMOTE=`tcp_state "$REMOTE_ADDRS"`
+for I in `seq 1 1 $(($TCP_STATE_LEN-1))`; do
+    A=`echo $LISTEN | awk -v var=$I '{ print $var }'`
+    B=`echo $REMOTE | awk -v var=$I '{ print $var }'`
+    echo $A" - "$B
+done
+</code>
+====== Publishing Incoming TCP Connections to MQTT Broker ======
+The following script is meant to run in the background and poll ''/proc/net/tcp'' for incoming TCP connections on a configurable port and then publish a JSON message to an MQTT broker with the following structure:
+<code>
+server -> FQDN hostname of the machine that the script is running on,
+this -> the listening IP address followed by the port separated by a colon
+that -> the connecting IP address followed by the port separated by a colon
+</code>
+<code bash>
+#!/usr/bin/env bash
+###########################################################################
+##  Copyright (C) Wizardry and Steamworks 2022 - License: GNU GPLv3      ##
+##  Please see: http://www.gnu.org/licenses/gpl.html for legal details,  ##
+##  rights of fair usage, the disclaimer and warranty conditions.        ##
+###########################################################################
+#                                                                         #
+# This script publishes the state of TCP connections to the machine over  #
+# a specified port to an MQTT broker. The script leverages polling        #
+# /proc/net/tcp for TCP connections instead of using netstat and uses     #
+# just the most basic commands making it suitable for various cases.      #
+#                                                                         #
+###########################################################################
+###########################################################################
+#                           CONFIGURATION                                 #
+###########################################################################
+# path to the mosquitto publishing tool
+MOSQUITTO_PUBLISHER=/storage/.local/usr/local/bin/mosquitto_pub
+# the host name of the MQTT broker
+MOSQUITTO_HOST=iot.internal
+# the MQTT topic to publish on
+MOSQUITTO_TOPIC=arcade
+# the TCP port to monitor for connections
+MONITOR_PORT=55435
+# the time to sleep between polling
+SLEEP_TIME=1
+###########################################################################
+#                             INTERNALS                                   #
+###########################################################################
+decode () {
+    DUO=$1
+    LEN=`echo -n $DUO | wc -c`
+    OUT=""
+    for X in `seq 0 2 $(($LEN-2))`; do
+        OUT=$((16#${DUO:$X:2}))"."$OUT
+    done
+    OUT=${OUT::-1}
+    echo $OUT
+}
+tcp_state() {
+    OUT=""
+    for SET in $1; do
+        IP_HEX=`echo $SET | awk -F':' '{ print $1 }'`
+        PO_HEX=`echo $SET | awk -F':' '{ print $2 }'`
+        IP=`decode $IP_HEX`
+        PO=$((16#$PO_HEX))
+        OUT=$OUT" "$IP:$PO
+    done
+    echo $OUT
+}
+TMP_STATE=/dev/shm/tcp_state
+RUN=1
+trap '{ rm -rf $TMP_STATE; RUN=0; }' KILL QUIT TERM EXIT INT HUP SIGUSR1
+while [ $RUN -ne 0 ]; do
+    cat /proc/net/tcp >$TMP_STATE
+    TCP_STATE_LEN=`cat $TMP_STATE | wc -l`
+    LISTEN_LOCAL=`cat $TMP_STATE | \
+        awk 'NR==1 { for (i=1;i<=NF;++i) if ($i=="local_address") { n=i; break }} { print $n }' | \
+        tail +2`
+    REMOTE_ADDRS=`cat $TMP_STATE | \
+        awk 'NR==1 { for (i=1;i<=NF;++i) if ($i=="rem_address") { n=i; break }} { print $n }' |
+        tail +2`
+    LISTEN=`tcp_state "$LISTEN_LOCAL"`
+    REMOTE=`tcp_state "$REMOTE_ADDRS"`
+    for I in `seq 1 1 $(($TCP_STATE_LEN-1))`; do
+        X=`echo $LISTEN | awk -v var=$I '{ print $var }'`
+        Y=`echo $REMOTE | awk -v var=$I '{ print $var }'`
+        NETPLAY=`echo $X $Y | grep $MONITOR_PORT | grep -v 0.0.0.0`
+        if [ ! -z "$NETPLAY" ]; then
+            THIS=`echo $NETPLAY | awk '{ print $1 }'`
+            THAT=`echo $NETPLAY | awk '{ print $2 }'`
+            $MOSQUITTO_PUBLISHER \
+                -h $MOSQUITTO_HOST \
+                -t $MOSQUITTO_TOPIC \
+                -m "{ \"origin\": \"`hostname -f`\", \"this\": \"$THIS\", \"that\": \"$THAT\" }"
+        fi
+    done
+    sleep $SLEEP_TIME
+done
+</code>
+====== Text Activity Spinners in Bash ======
+In order to use the [[/assets/databases/spinners/text|text activity spinners in bash]], here is an example script:
+<code bash>
+#!/usr/bin/env bash
+ASCII_SPINNER=( ".oOo" "oOo." "Oo.o" "o.oO" )
+while [ 1 = 1 ]; do
+    CAR="${ASCII_SPINNER[0]}"
+    CDR=("${ASCII_SPINNER[@]:1:${#ASCII_SPINNER[@]}}")
+    ASCII_SPINNER=(${CDR[@]} ${CAR})
+    echo -n -e "\r${CAR}"
+done
+</code>
+The logic is pretty straight-forward:
+  * the spinner is loaded as a bash array,
+  * the array is shuffled by shifting the first element, keeping a copy it and then adding the first element to the end of the spinner array,
+  * the first element is printed out, without any newlines and by prepending a carriage-return ''\r'' in order to make sure that the next element in the spinner array will overwrite the old element