Differences

This shows you the differences between two versions of the page.

--- fuss:linux [2020/05/08 21:53] – [Automatically Add all RNDIS Devices to a Bridge] office
+++ fuss:linux [2020/07/08 04:19] – [Scraping a Site Automatically using SystemD] office
@@ Line 1143: / Line 1143: @@
 One usage case for this rule is to connect a bunch of RNDIS devices to an USB hub and have them join the network automatically as they are hotplugged; for instance, Raspberry Pis [[fuss/raspberry_pi#logging-in_via_ssh_over_usb|can be configured as USB gadgets]] and then connected to an USB hub.
+====== Scraping a Site Automatically using SystemD ======
+FTP sites can be scraped elegantly by using systemd and tmux on Linux. By starting a ''tmux'' detached terminal, ''wget'' can run in the background and download a website entirely whilst also allowing the user to check up on the progress by manually attaching and detaching from ''tmux''.
+The following script contains a few parameters underneath the ''Configuration'' comment and up to ''Internals'' in order to set:
+  * the download path (''DOWNLOAD_DIRECTORY''),
+  * the ''wget'' download URL (all protocols supported by ''wget'' such as FTP or HTTP) (''DOWNLOAD_URL''),
+  * a descriptive name for the ''tmux'' session (''TMUX_SESSION_NAME'')
+<code bash>
+[Unit]
+Description=Scrape FTP Site
+Requires=network.target local-fs.target remote-fs.target
+After=network.target local-fs.target remote-fs.target
+[Install]
+WantedBy=multi-user.target
+[Service]
+# Configuration
+Environment=DOWNLOAD_DIRECTORY="/path/to/directory"
+Environment=DOWNLOAD_URL="ftp://somesite.tld/somedirectory"
+Environment=TMUX_SESSION_NAME="somesite.tld-download"
+# Internals
+Type=oneshot
+KillMode=none
+User=root
+ExecStartPre = -/bin/mkdir -p \""$DOWNLOAD_DIRECTORY"\"
+ExecStart=/usr/bin/tmux new-session -d -c "\"$DOWNLOAD_DIRECTORY\"" -s "$TMUX_SESSION_NAME" -n "$TMUX_SESSION_NAME" "/usr/bin/wget -c -r \"$DOWNLOAD_URL\" -P \"$DOWNLOAD_DIRECTORY\""
+ExecStop=/usr/bin/tmux send-keys -t "$TMUX_SESSION_NAME" C-c
+RemainAfterExit=yes
+</code>
+The file should be placed under ''/etc/systemd/system'', then systemd has to be reloaded by issuing ''systemctl daemon-reload'', the service should then be loaded with ''systemctl enable SERVICE_FILE_NAME'' where ''SERVICE_FILE_NAME'' is the name of the file copied into ''/etc/systemd/system'' and finally started by issuing ''systemctl start SERVICE_FILE_NAME''.
+Upon every reboot, the service file will create a detached tmux terminal and start scraping files from the URL.
+====== Access Directory Underneath Mountpoint ======
+In order to access a directory underneath a mountpoint without unmounting, create a bind mount of the root filesystem to a directory and then access the content via the bind mount.
+Ie, to access the contents of the directory ''/mnt/usb'' on top of which a filesystem has been mounted, create a directory:
+<code bash>
+mkdir /mnt/root
+</code>
+and create a bind mount:
+<code bash>
+mount -o bind / /mnt/root
+</code>
+Finally access the original underlying content via the path ''/mnt/root/mnt/usb2''.