Bulk YAML / JSON Subtree Addition and Subtraction

Note that the following instructions require yq, preferrably from the source at github where binaries are also provided.

Suppose that a Docker swarm is meant to centrally log to a graylog instance running on a server. In order to do that, you have devised a small snippet of code that will have to be added to all compose files in order to instruct Docker to send the logs to the central server. The snippet looks like this:

logging:
  driver: gelf
  options:
    gelf-address: udp://docker.internal:12201
    tag: PLACEHOLDER

where:

• udp://docker.internal:12201 is the address to the GELF sink on graylog and,
• PLACEHOLDER is a descriptive tag for the service

Now, you would like to inject all the compose files in a directory with this snippet and also change the PLACEHOLDER dynamically to the name of the compose file because the name of the compose file describes the service.

Imagine a layout such as the following filesystem layout:

---+ general
        +
        |
        +---- a.yaml
        |
        +---- b.yaml
        |
        +---- c.yaml
        .
        .
        .

and that the snippet above will have to be injected into all Docker compose files under /general/ whilst minding to change the PLACEHOLDER string to a for a.yaml, b for b.yaml, etc.

This is one daunting task given sufficiently many compose files without any automation. However, this can be done, even in a single command as a one-liner. Here is the command:

for composeFile in `find * -name \*.yaml`; do echo "ICAgIGxvZ2dpbmc6CiAgICAgIGRyaXZlcjogZ2VsZgogICAgICBvcHRpb25zOgogICAgICAgIGdlbGYtYWRkcmVzczogdWRwOi8vZG9ja2VyLmludGVybmFsOjEyMjAxCiAgICAgICAgdGFnOiBncmF5bG9nCgo=" | base64 -d | yq --output-format yaml "(.logging.options.tag=\"${composeFile/.yaml/}\" | .logging)" | yq -i '(.services[].logging = load("/dev/stdin"))' $composeFile; done

The long Base64 string packs the snippet to be inserted conveniently into a single line of Base64 characters, that is then unpacked via base64 -d, after which the tag is updated to the name of the compose file without the .yaml suffix and then it is passed through the pipe again to be joined with the compose file and appended to all the services inside the compose file.

Out of all, the last yq pipe in the line is the most interesting:

yq -i '(.services[].logging = load("/dev/stdin"))' $composeFile

where yq modifies $composeFile in place via the provided -i option by appending to all services in $composeFile what it reads from stdin via load(/dev/stdin). This works around the fact that a pipe can contain only one stream of data, namely the antecedent that is passed to it, by using the load command and loading STDIN whilst the yq command actually operates on a file that exists on the filesystem, namely the file referenced by the variable $composeFile (that is iterated by find).

The converse, namely to remove the entire logging subtree from all files in a directory, is much simpler:

yq -yi 'del(.services[] | .logging)' *.yaml

The previous command just removes the logging key and everything else from all the files in the current directory ending in *.yaml.