These notes should be considered as complementary to the official guide on how to set up root on external storage mainly because the situation might change given OpenWRT updates. Furthermore, due to the heterogeneous environment with different device types or storage devices, this task cannot truly be automated without a user at least confirming or picking options. In order to make the guide universal, it seems that the official guide does not do too good of a job explaining what exactly is going on and why, such that this guide should follow the highlights of the official guide, provide alternative and also explain a little more why certain steps are performed.

In short, the problem is that devices with SoCs or embedded systems typically are not produced with lots of storage in RAM due to RAM being relatively expensive to the overall build of the system such that an alternative that many embedded devices provide is the ability to access or use external drives (even SD cards) that can be used as storage. By connecting external storage (as opposed to using it directly), the device can install more complex and size-heavy software packages in order to extend the capabilities of the device.

As follows, aside from connecting some external storage or SD card, the software solution can be summarized to:

• install the default OpenWRT firmware,
• SSH into the device,
• format the external storage,
• mount the external storage to /mnt,
• transfer everything from /overlay to the new external storage mounted at /mnt,
• alter fstab to set the /overlay mountpoint to the external storage and remap the old /overlay mount to a new path /rwm.
• commit the changes.
• reboot

After rebooting the root filesystem will be on the external storage such that installing packages or transferring files will go directly to the expanded storage.

The first step is to flash the latest OpenWRT firmware and, in doing so, preferably use a clean variant. Depending on the hardware, several versions might be available and the flashing procedure might even require intermediary steps if the device is older or refurbished.

Either way, the purpose is to get to LuCI, the web interface for OpenWRT and be able to log-in. After logging in, the password should be set or reset and access to SSH should be granted on all interfaces. Access can be restricted later but leaving SSH open during setup makes the job easier to accomplish.

For now, all memory is volatile, such that the package lists have to be updated using the OpenWRT package manager opkg:

opkg update

After which, package can be installed. You will find that, depending on the version, there is no direct search feature and that issuing:

opkg list | grep PACKAGE

where:

• PACKAGE is a package or command to look for

is the canonical way of looking for stuff using opkg. Later, this changes because LuCI, the web interface will be available and will make package management easier.

At this point the OpenWRT device is a blank slate more-or-less and even if you're accustomed to Linux systems, you will find that OpenWRT is so modular that it does not even include support for the default Linux filesystem such as ext2 or ext4. OpenWRT does have support for most filesystems but kernel module and the associated tools have to be installed. It might be difficult to imagine at this point, but now you will find yourself on a system that will be masked after the root is pivoted to the external storage, and that everything that is installed will not be accessible after the pivoting. Having that in mind, do not rush to install stuff but rather keep the list short and ideally just the bare minimal needed to deal with an external filesystem that will become the root of the device.

To make things easier, the following command will install a series of packages that will be used to accomplish the pivoting:

opkg install block-mount kmod-usb-storage fdisk f2fs-tools f2fsck kmod-fs-f2fs mkf2fs lsblk rsync

and here is a breakdown:

• block-mount is an OpenWRT script packages that realizes the pivoting,
• kmod-usb-storage is the kernel module that adds mass storage support,
• fdisk is a partitioning tool just like parted,
• f2fs-tools, f2fsck, mkf2fs are all tools to deal with F2FS filesystems and kmod-fs-f2fs is the F2FS filesystem kernel module,
• lsblk is a tool that displays a tree-like overview of the storage / block devices connected to the system,
• rsync is a file synchronization tool

Now the lsblk command can be issued to inspect the connected devices. Here is an example snippet:

NAME      MAJ:MIN RM  SIZE RO TYPE MOUNTPOINTS
sda         8:0    1 29.7G  0 disk
└─sda1      8:1    1 29.7G  0 part
mtdblock0  31:0    0  128M  0 disk
mtdblock1  31:1    0  256K  1 disk
mtdblock2  31:2    0   64K  0 disk
mtdblock3  31:3    0   64K  1 disk
...

As can be observed /dev/sda is a disk that corresponds to the small SD card inserted into the device. It seems that the disk is already partitioned and contains one single partition at /dev/sda1. When purchasing storage, typically the device is preformatted with some filesystem such as NTFS. However, we would like to use F2FS in order to leverage its features that are targetted at flash storage. In order to do that, the /dev/sda1 partition will be reformatted to F2FS:

mkfs.f2fs -l slate -i -O extra_attr,inode_checksum,sb_checksum,compression,encrypt /dev/sda1

where:

• slate is a partition label that can be anything,
• /dev/sda1 is the partition to be formatted,
• extra_attr,inode_checksum,sb_checksum,compression,encrypt are all optimizations such as turning on compression and allowing encyrption

With the partition ready, the following command is used to take a look around and see what the status of the block devices on the system is:

block info

The command should return something similar to the following:

/dev/mtdblock6: UUID="fe7105d6-ba33-4771-98c2-6090233d05d2" VERSION="4.0" MOUNT="/rom" TYPE="squashfs"
/dev/mtdblock7: MOUNT="/overlay" TYPE="jffs2"
/dev/sda1: UUID="8f78ed34-b469-44f1-a259-cfa1a8f66bfa" LABEL="slate" VERSION="1.16" TYPE="f2fs"

Vaguely (because it is not really important) the output shows that there are two partitions on the device itself, both /dev/mtdblock6 and /dev/mtdblock7, the former being the read-only ROM of the device and the later mounted at /overlay being the built-in storage of device that is read-write. Lastly, the partition that was just formatted shows within the output as /dev/sda1 and with an UUID displayed.

To settle any uncertainties: the /overlay mount is equivalent to the / root mount, such that all "default" reads and writes that do not access the device read-only ROM will always take place on /overlay, which is the root-equivalent for OpenWRT.

Everything seems ready for pivoting. To understand what is about to be set up, here is an illustration of the gambit:

       +----------+                     +----------+
       | /overlay | built-in storage    | /overlay | built-in storage
       +----------+                     +----------+
            ^                                |
            |                                +-------< read-only mount
            |                                
            | read /write               +----------+
            v                           | /overlay | external storage
                                        +----------+
                                             ^
                                             |
                                             | read /write
                                             v

In other words, before the pivot any writes would occur to the built-in flash storage and after the pivot any reads and writes will take place onto the external storage whilst the older flash storage mount will be mounted somewhere just for reference. Of course, in order to be able to pivot, the data that already exists on the device internal flash at /overlay must be transferred over to the new external device that has been just prepared:

mount /dev/sda1 /mnt/
rsync -SaxH --info=progress2 /overlay/* /mnt/

Here rsync is used to transfer the files, but the plain old cp should do just fine.

The next step is to modify fstab in order to pivot the /overlay from the internal flash to the external storage and then the older /overlay will be moved somewhere for reference.

The first sequence will modify the fstab to set the external storage to /overlay by referencing it by UUID:

uci -q delete fstab.extroot
uci set fstab.extroot="mount"
uci set fstab.extroot.uuid="8f78ed34-b469-44f1-a259-cfa1a8f66bfa"
uci set fstab.extroot.target="/overlay"
uci commit fstab

and the second step moves the older /overlay to some other mount point, in this case /rwm:

uci -q delete fstab.rwm
uci set fstab.rwm="mount"
uci set fstab.rwm.device="/dev/mtdblock7"
uci set fstab.rwm.target="/rwm"
uci commit fstab

With all that in place, the system can be rebooted.

After a reboot, some testing is required to be able to tell whether the pivot was successful.

Issue:

grep -e /overlay /etc/mtab

which should output the following:

/dev/sda1 /overlay f2fs rw,lazytime,relatime,background_gc=on,nodiscard,no_heap,user_xattr,inline_xattr,inline_data,inline_dentry,flush_merge,extent_cache,mode=adaptive,active_logs=6,alloc_mode=default,checkpoint_merge,fsync_mode=posix,discard_unit=block 0 0
overlayfs:/overlay / overlay rw,noatime,lowerdir=/,upperdir=/overlay/upper,workdir=/overlay/work 0 0

which means that the /dev/sda1 external device has been mapped to /overlay.

Issue:

df /overlay /

should output something like the following:

Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/sda1             31163904    361432  30802472   1% /overlay
overlayfs:/overlay    31163904    361432  30802472   1% /

that will confirm that the root is now on /dev/sda1.

When changes that pertain to the mounted external storage must be made, the base system must be accessed. In order to do this, the procedure is fairly simple:

• power down the device,
• remove the external storage (or sd card),
• boot the device back up,
• access the device using SSH,

To make sure it is the base system, issue:

df /overlay /

which should output something like the following:

Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/mtdblock7            9280      1832      7448  20% /overlay
overlayfs:/overlay        9280      1832      7448  20% /

where:

• /dev/mtdblock7 is the internal device flash

It is now confirmed that we are accessing the internal flash storage, such that the SD or external storage can be reattached. After attaching the storage, issue dmesg to see the latest kernel messages and hopefully see the sda device that was formerly used as external storage.

As the kernel confirms:

sda: detected capacity change from 0 to 62333952
sda: sda1

the device is automatically detected and ready to be used.

Issuing lsblk to check the device tree reveals:

NAME      MAJ:MIN RM  SIZE RO TYPE MOUNTPOINTS
sda         8:0    1 29.7G  0 disk
└─sda1      8:1    1 29.7G  0 part
mtdblock0  31:0    0  128M  0 disk
mtdblock1  31:1    0  256K  1 disk
mtdblock2  31:2    0   64K  0 disk
...
mtdblock6  31:6    0 13.1M  0 disk /rom
mtdblock7  31:7    0  9.1M  0 disk /overlay

which means that the /overlay is now back on the internal flash storage and the external storage has been detected after inserting it again into the device.

For example's sake, the F2FS partition mount options will be modified. First, check the existing options for the mount:

uci get fstab.extroot.options

which should output an error, along the lines of uci: Entry not found.

So, let's set some recommended options for F2FS, in particular, turning on compression:

uci set fstab.extroot.options="compress_algorithm=zstd:6,compress_chksum,atgc,gc_merge,lazytime"
uci commit fstab

Just to test, issue:

mount -t f2fs -o compress_algorithm=zstd:6,compress_chksum,atgc,gc_merge,lazytime /dev/sda1 /mnt/

Keep in mind that OpenWRT in its smallest form barely even has any kernel modules installed such that if f2fs would have had some dependencies, these would have had to be installed on the base system before attempting to mount the partition.

If it all works, issue

reboot

to reboot the device.