Sometimes software needs to be compiled natively, especially when cross-toolchains cannot be made, such as the case of .NET programming that involves Windows Forms. In such a case, even though historically mono has provided a re-implementation of .NET, even decades later, there are many incompatibilities due to implementation differences between mono and the official .NET. dotnet is a Microsoft effort to bring .NET semantics to other platforms seamlessly but that comes at a cost that involves dropping some programming paradigms that would not have had to be dropped in case the projects created would have targeted the Microsoft platform exclusively.

For that purpose, it is sometimes necessary to compile a project directly on a Windows operating system in order to be able to use the official .NET framework. The following guide details some ideas on how that could be achieved by going through continuous integration solutions such as jenkins and use Docker and virtualization to start and stop a Windows operating system on demand.

An official image for Jenkins exists that can be used directly however more than likely it is necessary to create your own container for Jenkins in order to include the tools necessary for your build environment.

Since the introduction mentioned the .NET framwork, the Jenkins build file is based on the official Dockerfile for the Docker Jenkins container and then modified to install both mono, dotNET amongst other tools, while building on Linux. Note that the link is to the Dockerfile itself, but that the Docker file references a few scripts such as jdk-download.sh, jenkins-plugin-cli.sh, jdk-download-url.sh, jenkins.sh, jenkins-support that need to be pulled into the same directory where the Dockerfile is placed.

One peculiarity that must be mentioned is that for this guide, Docker itself should also be included in the Jenkins build file because Jenkins will use the Docker command-line tools to control Docker on the host machine. To that end, the following additions are made to the Dockerfile:

# install docker
RUN curl -fsSL https://download.docker.com/linux/debian/gpg -o /etc/apt/keyrings/docker.asc && \
    echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/debian bookworm stable" | tee /etc/apt/sources.list.d/docker.list > /dev/null && \
    apt-get update -y && \
    apt-get upgrade -y && \
    apt-get install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin

The snippet adds Docker to the Debian repositories, the Linux distribution that is used to build the Jenkins container, from the official Docker repositories and then installs Docker inside the container.

Building then is just a matter of running docker build and then uploading the image to a custom registry in order to be pulled and referenced later with Docker compose or on the command line.

As an example, here is a Jenkins compose file that will be used to run the Jenkins build created using the previous Dockerfile:

version: "3.9"
services:
  jenkins:
    image: machine.tld:5000/jenkins:latest
    user: root:root
    ports:
      - 50005:8080
      - 50000:50000
    environment:
      - JAVA_OPTS=-Xms512m -Xmx512m
    volumes:
      - /var/lib/jenkins:/var/jenkins_home
      - /var/run/docker.sock:/var/run/docker.sock
    stop_grace_period: 60s
    deploy:
      replicas: 1
      placement:
        max_replicas_per_node: 1

where:

• machine.tld:5000/jenkins:latest is the fully qualified reference to the Jenkins image stored within a registry running at the address and port machine.tld:5000,
• Jenkins itself runs on port 8080 with 50000 being an auxiliary that is supposed to accept connections from Jenkins agents,
• JAVA_OPTS=-Xms512m -Xmx512m is the infamous Java memory limiting line due to the Java allocator being very happy to gobble up as much RAM as possible from the operating system such that limiting its frenzy is wise,
• /var/lib/jenkins is a directory on the host that will contain all the Jenkins configuration files, workspaces and build files and it is also a directory that will be shared with the Windows machine,
• finally the Docker socket is passed through to the container via the line /var/run/docker.sock:/var/run/docker.sock in order to allow Jenkins to control the Docker daemon on the host file; for example, to start or stop the Windows container.

in principle using the Kernel virtualization Machine (KVM) on Linux, it is possible for Docker to bundle something like qemu or even virsh and then run an image file within the container by passing the KVM device (/dev/kvm) through to the container from the host machine. In that sense, the Docker container will just contain a scripted environment that takes an image file and runs it under qemu whilst using hardware virtualization from the host machine that is passed into the container but the container will not really be running Windows itself yet rather just the virtualization layer. In principle, both virtualization and emulation are specialized subsets of "containerization" such that combining these together should not be a problem.

Given the aforementioned the rest of the task should involve writing the necessary entry-scripts for a Docker container that contains virtualization tools such as qemu but fortunately there exist ready-made solutions out there provided by people that followed the same endeavors. One such solution is dockur/windows that implements all the necessary framework, scripts and tools in order to be able to run Windows. In fact, the project has lots of convenience features built in such as installing Windows itself by pulling the desired Windows ISO thereby making the process fairly seamless. The image also runs the usual OEM scripts in order to run post-install instructions that bootstrap the environment by creating a specified default user and password.

dockur/windows also bundles a web-based VNC viewer that connects to the Windows environment via the qemu VNC driver such that Windows itself can be viewed or used within a standard browser. However, the VNC viewer will not be used in the end, the whole environment needing to be accessed heartlessly from Jenkins and there will not be any need for a Desktop. The project even suggests using RDP to access Windows instead of via the VNC due to RDP being more optimized for Windows but all of that will not be necessary and the already-provided VNC viewer will be used to just set up the environment necessary to push builds from Jenkins.

Here is the command that was run to create a container and then let dockur/windows to install Windows 7 (best Windows to run in order to just be able to have access to .NET Framework, given that anything after Windows 7 is already built around dotNET):

/usr/bin/docker run --name windows \
    --rm \
    --interactive \
    --user 0:0 \
    --privileged \
    --cap-add NET_ADMIN \
    --device=/dev/kvm \
    --device=/dev/net/tun \
    --device=/dev/dri/renderD128 \
    --device=/dev/dri/card0 \
    -p 8006:8006 \
    -p 8022:22 \
    -e RAM_SIZE="1G" \
    -e CPU_CORES="1" \
    -e USERNAME="" \
    -e PASSWORD="" \
    -e LANGUAGE="English" \
    -e REGION="en-US" \
    -e KEYBOARD="en-US" \
    -e VERSION="7e" \
    -v "/mnt/data/windows/storage:/storage" \
    -v "/var/lib/jenkins:/data" \
    --stop-timeout 120 \
    dockurr/windows

with the following remarks:

• 8006 is the port number used for the VNC viewer that has to be used with the URL of the machine to access the Windows desktop (ie, via http://machine:8006,
• 8022 is the port number that is passed to port number 22 on the Windows machine where an SSH server will be listening and it will be used to remotely log into Windows via SSH,
• USERNAME and PASSWORD have to be filled in with an username and a password that will be used to create an account when Windows is installed,
• USERNAME references 7e which corresponds to Windows 7 Enterprise, the operating system that will be installed,
• /mnt/data/windows/storage is the path where the image created by the Windows 7 installation will be stored on the host and it has to be mapped to /storage within the container,
• the /dev/kvm device is passed for virtualization and /dev/tun for the network between the container and the Windows VM,
• the /dev/dri/renderD128 device along with /dev/dri/card0 are both devices that are passed to the docker container in the hopes that qemu can leverage them in order to use GPU acceleration on the Windows VM within the docker container; for this to work the dockur/windows documentation specifies that GPU="y" must be passed as an environment variable (ie: via -e) but all devices are shared such that there is no harm nor performance penalty to pass these device hooks into the container.

Lastly, /var/lib/jenkins is the path where Jenkins stores all its configuration files, workspaces and build environments and in this case the directory referenced by the Jenkins container, namely /var/lib/jenkins is a folder that has to be shared with the container that runs Jenkins.

An important remark pertains to the specification of the amount of RAM and CPU to pass through to the container and hence the Windows machine being virtualized. The aforementioned command specifies that the virtual machine should be limited to one CPU core (via CPU_CORES="1") and should be passed a share of of RAM (via RAM_SIZE="1G"). When Windows installs, the RAM should be adjusted to about in order to be able to run the virtual machine but after the installation, the value can be decreased to about .

Go figure that the whole reason for creating this on-demand system where Windows will be launched on demand relies on problems that are tough to deal with when using virtualization. In principle, any operating system is designed to make best use of its memory pool, with Linux itself, reveered for its low memory footprint, actually using up all the RAM that is unused for filesystem buffers in order to speed up I/O operations, also the slowest of the RAM-CPU and Disk trinity. The problem is that from the outside of the container there is no way of knowing how much RAM is "actually used" because if the operating system inside the container allocates all the RAM passed to it to, say, like Linux, speed up disk operations, then from the outside all the RAM that has been passed to the virtual machine is in fact all in use, even if the programs running within the virtual machine might just be using a fraction of the RAM allocated. With that said, when the Docker container running Windows starts, the whole RAM_SIZE amount of RAM will be marked "in use" on the host operating system such that even if the Windows virtualization will be idling and, in fact, just using up a megabytes, a whole slab of memory will vanish from the host. To order to avoid these problems, historically speaking, KVM drafted "memory ballooning" that is able to dynamically decrease or increase the amount of RAM allocated to the virtual machines and then either maximizing RAM to the VMs (as would be desired on, say, a bare metal virtualization environment where the host visualizer is irrelevant and yields all resources to VMs) or using other RAM scheduling options. However, unfortunately the ballooning draft was not given all that much attention and currently there is no canonical ready-made way to just specify an allocation scheduling option - instead, some third-party daemons exist, such as balloond written in Go, or even momd, both being viable solutions.

With the above made clear, the Windows VM will not run permanently in order to not steal such a large chunk of RAM from the machine running the Docker containers but instead the Windows container and VM will be started by Jenkins, on demand whenever a pipeline runs.

It is possible to install a Jenkins agent on the Windows machine and then interact with it from the Jenkins container, however, the interaction is fairly limited in terms of sophistication and will just end up requiring hand-made-scripts to be written in order to automate the process. For that purpose Cygwin is installed within the Windows machine that will erect an SSH shell ready for connection from the Jenkins container.

The openssh package has to be installed via Cygwin Setup and then the command ssh-host-config has to be run on the Cygwin shell in order to install SSHd as a Windows service such that an SSH connection is possible even without logging into Windows on bootup. Otherwise, any other GNU tools can be installed that might facilitate building programs when a shell is invoked from Jenkins.

It was decided that the best way to approach this in a given scenario was to start the Windows Docker container at the start of the pipeline and then tear down the container in a post-build action in order to ensure that the Windows VM is stopped. For concurrency, the scripts were written to wait for the Windows machine to be in a "stopped state", then ideally grab a lock, start the Windows machine, wait for the SSH port to be available, execute the necessary commands to build a project and then finally to symmetrically stop the Windows VM. The last post-build step is made to run regardless of whether the build itself, namely, the compilation part, either fails or succeeds in order to ensure both re-entrancy and concurrency safety with other projects being compiled after or at the same time that would need access to the Windows VM.

For that purpose, the following is a small script that is placed at the start of the pipeline and is designed to ensure that after the script runs, the Windows machine is up and running and ready to accept commands over SSH:

# Check if windows already running and wait if it is already running...
echo "Waiting for Docker container: "
while [ "$(docker ps --format={{.Names}} | grep -Poq '^windows$; echo $?')" -eq 0 ]; do
    echo -n "."
    sleep 1
done
echo " ok"
 
# Start windows
docker run --name windows \
    --detach \
    --rm \
    --user 0:0 \
    --privileged \
    --device=/dev/kvm \
    --device=/dev/net/tun \
    --device=/dev/dri/renderD128 \
    --device=/dev/dri/card0 \
    -p 8006:8006 \
    -p 8022:22 \
    -e RAM_SIZE="1G" \
    -e CPU_CORES="1" \
    -e USERNAME="" \
    -e PASSWORD="" \
    -e LANGUAGE="English" \
    -e REGION="en-US" \
    -e KEYBOARD="en-US" \
    -e VERSION="7e" \
    -v "/mnt/grimore/windows/storage:/storage" \
    -v "/mnt/grimore/jenkins:/data" \
    --stop-timeout 120 \
    dockurr/windows
 
# wait for windows to start up...
echo "Starting Windows: "
while [ "$(sshpass -p ... ssh -q -p 8022 -o StrictHostKeyChecking=no sys@docker exit; echo $?)" -ne 0 ]; do
    echo -n "."
    sleep 1
done
echo " ok"

As the comments would hint, the first loop continuously pools Docker on the host to make sure that the Windows machine is stopped, then the docker run command starts the Windows container on the host. Note that even if the script code runs inside the Jenkins container, the actual docker run command will run on the Docker host that runs Jenkins as well, due to the socket file that is passed within the Docker container.

The last loop runs the ssh command in a loop that is supposed to log into the Windows machine at host docker with username sys on port 8022 and just run the exit command. If this operation succeeds, then it means that Windows is up and running and that the SSH daemon is ready to accept commands in order to compile projects; for instance dotnet publish … commands that will run on the Windows machine within the Cygwin environment.

Note that sshpass, a tool that can pass a password on the command line to the ssh command in order to logon to the Windows machine without having to create trusted SSH keys. The environment itself will be sealed such that over-complicating the setup with keys of variable lifespan that might need to be changed, especially if the address changes, is undesirable.

Lastly, a plugin from the Jenkins repository is used to allow a script be ran as the post-build action of a pipeline and that script will consist in just one single command:

docker stop windows