Differences

This shows you the differences between two versions of the page.

--- fuss:sh [2020/03/30 05:18] – [For ZIP] office
+++ fuss:sh [2025/06/10 03:18] (current) – [Replace Text in Files using Perl] office
@@ Line 14: / Line 14: @@
 <code bash>
 ###########################################################################
-##  Copyright (C) Wizardry and Steamworks 2019 - License: GNU GPLv3      ##
+##  Copyright (C) Wizardry and Steamworks 2025 - License: MIT            ##
-##  Please see: http://www.gnu.org/licenses/gpl.html for legal details,  ##
+##  Please see: https://opensource.org/license/mit/ for legal details,   ##
 ##  rights of fair usage, the disclaimer and warranty conditions.        ##
 ###########################################################################
@@ Line 165: / Line 165: @@
 ====== Lock ======
-To prevent parallel execution of scripts, the following snippet can be used:
+To prevent parallel execution of scripts, the following snippet can be placed within existing scripts at any point where the operations following the snippet must be guaranteed to be executed by a single script:
 <code bash>
 # Acquire a lock.
 LOCK_FILE='/var/lock/myscript'
-if mkdir $LOCK_FILE 2>&1 >/dev/null; then
+if mkdir "$LOCK_FILE" &>/dev/null; then
-    trap '{ rm -rf $LOCK_FILE; }' KILL QUIT TERM EXIT INT HUP
+    trap '{ rm -rf $LOCK_FILE; }' HUP INT QUIT TERM EXIT
 else
     exit 0
@@ Line 180: / Line 180: @@
   * ''LOCK_FILE'' points to a location where a directory ''myscript'' can be created.
-The command ''mkdir'' is guaranteed to be atomic, such that out of multiple concurrent processes only one will succeed in creating the directory at ''/var/lock/myscript''. Once the directory is created, the lock is established, and  the ''trap'' instruction will bind to the ''KILL'', ''QUIT'', ''TERM'', ''EXIT'', ''INT'' and ''HUP'', essentially ensuring that the code ''{ rm -rf $LOCK_FILE; }'' will be executed once those signals are raised (this includes normal script termination - via ''QUIT'') thereby releasing the lock.
+The command ''mkdir'' is guaranteed to be atomic, such that out of multiple concurrent processes only one will succeed in creating the directory at ''/var/lock/myscript''. Once the directory is created, the lock is established, and  the ''trap'' instruction will bind to signals like ''KILL'', ''QUIT'', ''TERM'', ''EXIT'', ''INT'' and ''HUP'', essentially ensuring that the code ''{ rm -rf $LOCK_FILE; }'' will be executed once those signals are raised (this includes normal script termination - via ''QUIT'' or ''EXIT'') thereby releasing the lock.  Note that some signals cannot be trapped, such as ''KILL'' but it is imperative to include the ''EXIT'' signal within the list of signals in order to ensure that the lock is cleared up after the script terminates.
-The snippet can be used directly in any script just by setting the value of ''LOCK_FILE'' to some chosen path where the script will be able to create a lock directory.
+The snippet can be used directly in any script just by setting the value of ''LOCK_FILE'' to some chosen path where the script will be able to create a directory.
 ====== Loop Over Command Line Arguments ======
@@ Line 279: / Line 279: @@
 </code>
 which will take the last command and arguments and place them in a file called ''script.sh''.
-====== Pipe Visualiser ======
+====== Pipe Visualizer ======
 ''pv'' is an utility that can intercept pipes and show various statistics during transfer.
@@ Line 516: / Line 517: @@
 </code>
-====== Replace String in Files ======
+====== Replace Text in Files using Perl ======
-To replace a string in multiple files you can use ''find'' and ''perl'':
+^ Command Line Aspect ^ Visual Mnemonic Graft ^
+| ''-p -i"" -e'' | {{fuss:fuss_linux_mnemonic_perl_pie.png?nolink&128}} |
 <code bash>
-find . -type f -exec perl -p -i -e 's/was\.fm/grimore\.org/g' '{}' \;
+perl -p -i"" -e 's/ORIGINAL/REPLACEMENT/g' file
+</code>
+where:
+  * ''-p'' calls print at the end of the command,
+  * ''-i""'' will perform the change in-place and the additional quotes imply that no backups shall be made (''""''),
+  * ''-e'' means to execute the following command, which just happens to be a regular expression,
+  * ''ORIGINAL'' represents a regular expression,
+  * ''REPLACEMENT'' is the replacement text,
+  * ''file'' is a file or a shell expanded list of files, such as the wildcard operator ''*''
+The former can also be combined with ''find'' in order to recursively and selectively replace text in files:
+<code bash>
+find . -type f -exec perl -p -i'' -e 's/was\.fm/grimore\.org/g' '{}' \;
 </code>
 this command will search all files in the path and replace the string ''was.fm'' with ''grimore.org''.
+====== Print Match Groups with Perl ======
+One of the advantages of using Perl is that it is integrated with most operating systems and also by default uses regular expressions.
+Using grep is not that useful given that in spite of benefiting from regular expressions via the POSIX flag ''-P'', it is not created to be able to extract match groups such that most solutions that would have used grep tend to use sed instead and cut out parts of the text.
+For instance, the following command:
+<code bash>
+ffmpeg \
+    -i a.mp4 \
+    -filter_complex ebur128=peak=true \
+    -f null - 2>&1 | \
+     tr -s ' ' | \
+     grep FTPK | \
+     perl -ne '/t: ([0-9\.]+).+?FTPK: ([\-0-9\.]+).+/ and print "$1 $2\n"'
+</code>
+will extract the time stamp and the noise peaks from an input file ''a.mp4'' using ffmpeg.
+The input that the perl command processes is the following:
+<code>
+[Parsed_ebur128_0 @ 00000000041e8c00] t: 28.5695 TARGET:-23 LUFS M: -55.3 S: -57.4 I: -33.2 LUFS LRA: 18.8 LU FTPK: -48.7 -48.7 dBFS TPK: -5.9 -5.9 dBFS
+</code>
+Using Perl also has the advantage that the output can be customized, via the ''print'' Perl function and in this case first the time is captured and stored to the first match group and after that the noise level is captured and stored to the second match group.
+<code bash>
+perl -ne '/t: ([0-9\.]+).+?FTPK: ([\-0-9\.]+).+/ and print "$1 $2\n"'
+</code>
+After that the first and second match group are printed out and a newline appended. Fortunately the ''print'' Perl function follows ''printf''-like semantics such that transforming the output is very convenient, especially when the command would have to be piped into a different command.
 ====== Transfer a Directory between Servers ======
 This can be performed in several ways depending on the tools available.
-===== Using Tape Archives =====
+===== Using Tape Archives (tar) =====
 This can be accomplished by using ''tar'' and the pipe operator:
@@ Line 807: / Line 849: @@
   * ''X'' is the low number,
   * ''Y'' is the high number
+====== Find Plaintext Files ======
+<code bash>
+find . -type f -exec grep -Iq . {} \; -print
+</code>
+====== Ensure Newline at End of File ======
+<code bash>
+find -name \*.txt | while read f; do tail -n1 $f | read -r _ || echo >> $f; done
+</code>
+where:
+  * ''*.txt'' is the pattern to search for.
+====== Execute Script as a Different User ======
+Using ''su'', the following script can be ran as ''root'' but will execute the remainder of the script after the ''if'' clause as the user ''myanonamouse'':
+<code bash>
+# Restart the script as an user via su.
+if [ "$(id -u)" -eq 0 ]; then
+    # exec replaces current shell
+    exec su myanonamouse "$0" -- "$@"
+    # this line will not be reached
+fi
+# this will run as user "myanonamouse"
+</code>
+====== Get External IP Address via DNS ======
+Using ''opendns.com'' that is programmed to return the IP address of the client performing the lookup:
+<code bash>
+dig +short myip.opendns.com @resolver1.opendns.com
+</code>
+====== Create a File under a Given Username and Group at Once ======
+The following command:
+<code bash>
+install -D -o www-data -g www-data /dev/null /var/log/razor.log
+</code>
+will create the file ''/var/log/razor.log'' and set the ownership to the user ''www-data'' and group ''www-data''. The ''-D'' flag will ensure that in case the path ''/var/log/'' is missing some directories, then the path will be created.
+====== Print a Character a Certain Number of Times ======
+The following line will print 10 successive dots:
+<code bash>
+yes "." | head -n 10 | xargs | tr -d ' '
+</code>
+====== Using the Results Placeholder for the find Utility ======
+The Unix utility ''find'' provides a results placeholder that holds the filename that has been found matching the filters provided to find. Here is an example that searches the current directory recursively for files ending in ''mp4'':
+<code bash>
+find . -type f -name '*.mp4' -exec echo "{}" \;
+</code>
+such that the command after the ''exec'' parameter will be executed for all files, with the special curly-bracers symbol ''{}'' expanding during execution to the full path of the file as reported by the ''find'' utility.
+Remarkably, as feature-packed as Unix tools are, the results placeholder is deceptively straightforward and without features. For example, one of the most common tasks is to perform some sort of string substitution on the results in case the file must be moved or manipulated in any way.
+In order to use the file name without further bothering about the ''find'' utility, simply pass the special placeholder symbol ''{}'' as the argument to a shell, for example, a bash shell and then for the command being executed, the contents of ''{}'' will be transferred to the first command-line argument ''$0''. Here is an example that changes the extension of all images recursively from the current directory from ''jpg'' to ''png'':
+<code bash>
+find . -type f -name '*.jpg' -exec bash -c 'mv "$0" "${0//jpg/png}"' {}  \;
+</code>
+Whilst the command invocation:
+<code bash>
+find . -type f -name '*.jpg' -exec ... \;
+</code>
+is responsible for finding files ending with the ''jpg'' extension, for every file found, the following command is executed, that can be disected:
+<code>
+bash -c 'mv "$0" "${0//jpg/png}"' {}
++     +      +          +        +
+|     |      |          |        |
++-----+      |          |        | from find; expands to the path of the file
+execute a    |          |
+command      |          |
+             |          | bash substitution acting on parameter $0,
+             |          | changing "jpg" to "png"
+             |
+             |
+             | $0 is the first parameter
+             | passed to bash and holds
+             | the exact contents of {}
+</code>
+Something that seems counter-intuitive is that the command passed to the bash ''-c'' command-line parameter is quoted using single-quotes, which would imply no substitution:
+<code bash>
+find . -type f -name '*.jpg' -exec bash -c 'mv "$0" "${0//jpg/png}"' {}  \;
+</code>
+Interestingly, if one were to issue the same command with double-quotes instead:
+<code bash>
+.................................. bash -c "mv \"$0\" \"${0//jpg/png}\"" {}  \;
+</code>
+then the command would not run properly because while issuing the "find" command itself, the shell would substitute the parameters within the quotes. That being said, it only makes sense to execute the "inner-most" command whilst escaping it with single quotes:
+<code bash>
+.................................. bash -c 'mv "$0" "${0//jpg/png}"' {}  \;
+</code>
+such that substitution only occurs when the command passed to the ''-c'' bash parameter actually executes.
+====== Shell alarm(2) Implementation ======
+An equivalent to the ''alarm(2)'' call in systems programming can be created by simply placing a process in the background and then killing the process in order to rearm the alarm or wait until the time elapses in order for the command to run.
+<code bash>
+# alarm(2)
+function alarm {
+    sleep $1
+    # this is the command to be executed after the elapsed time
+    echo "EXEC"
+}
+# ensures that any pending alarm is cleared upon termination of the current script
+ALARM_PID=0
+trap '{ test $ALARM_PID = 0 || kill -KILL $ALARM_PID; }' KILL QUIT TERM EXIT INT HUP
+# the parent program consists in a process that runs indefinitely
+while "..."; do
+    # when the alarm must be re-armed:
+    #   * kill the previous alarm
+    #   * reschedule the alarm into the future
+    if [ -d /proc/"$ALARM_PID" ]; then
+        kill -9 $ALARM_PID
+    fi
+    alarm "5" &
+    ALARM_PID=$!
+done
+</code>
+The actual code to be executed is to be found within the ''alarm'' function, namely anything after the ''sleep'' line. Fortunately, ''SIGKILL'' can interrupt ''sleep'' and additionally abort the program such that the command will not be executed when the alarm is just rearmed.
+Note that the expression ''$!'' might be a bashism but there are ample equivalents such as using ''pidof'', etc, to determine the PID of the last backgrounded process in case the scheme has to be used within a strict POSIX shell.
+====== Comparing Files Line-by-Line ======
+Given two files ''A'' and ''B'' both with lines comparable under the same context, here are some of the operations that can be performed when considering the lines to be set elements and the files ''A'' and ''B'' sets. Note that this respects //set theory// operations in terms of sets accepting only the same kind of element once and only once (as opposed to the notion of "collections" where elements of the same kind might be accepted multiple times) - or, in short, both files ''A'' and ''B'' contain lines unique to themselves.
+===== Lines in A but not in B =====
+"Lines in file A but not in file B" is a set subtraction $A \setminus B$ (defined as all elements / lines in $A$ that are not in $B$).
+$$
+\begin{venndiagram2sets}
+\fillANotB
+\end{venndiagram2sets}
+$$
+To perform this operation, issue:
+<code bash>
+comm -23 <(sort A) <(sort B)
+</code>
+===== Lines in B but not in A =====
+"Lines in file B but not in file A" is a set subtraction $B \setminus A$ (defined as all elements / lines in $B$ that are not in $A$).
+$$
+\begin{venndiagram2sets}
+\fillBNotA
+\end{venndiagram2sets}
+$$
+To perform this operation, issue:
+<code bash>
+comm -13 <(sort A) <(sort B)
+</code>
+===== Lines Common to Both A and B =====
+"Lines in file A that are also in B" is a set intersection $A \cap B$.
+$$
+\begin{venndiagram2sets}
+\fillACapB
+\end{venndiagram2sets}
+$$
+To perform this operation, issue:
+<code bash>
+comm -12 <(sort A) <(sort A)
+</code>
+====== Return Machine Utilization from Load Average ======
+On the [[/networking/leveraging_docker_swarm_for_high-availability_and_load-balancing_services|Docker swarm load average and high-availability page]] a program is shown that is able to return the CPU usage using ''vmstats'' but it seems too complex in terms of calls, such that the following alternative might be better:
+<code bash>
+awk '{ print int(100 - $1/$NF) }' <<< $(cat /proc/loadavg)" "$(nproc)
+</code>
+Note that the CPU utilization is computed relative to the 5minute usage over time returned from ''/proc/loadavg'' such that the result is not an instantaneous value compared to the version that uses ''vmstat''.