Encrypting Your Root File System on JETSON NANO - using LUKS & dm-crypt

Click here to view Encrypting Your Root File System on RASPBERRY PI.

WHY ENCRYPT?

There are many reasons to encrypt the Root File System (RFS) on the Nvidia Jetson Nano, from keeping WiFi credentials immutable to keeping proprietary software and sensitive data from being cloned. For many Jetson Nano configurations, many small partitions are created but the / file system including the /boot area is located here:

  • /dev/mmcblk0p1

It makes sense to encrypt the root partition as a way of encrypting everything.


INTRODUCING LUKS

LUKS (Linux Unified Key Setup) is the popular key management setup for dm-crypt, the de-facto standard for block device encryption with Linux.

LUKS provides a robust and flexible mechanism for multiple users (and services) to interface to and access Linux’s ‘dm-crypt’ infrastructure.

dm-crypt is a transparent disk encryption subsystem in Linux kernel versions 2.6 and later and is part of the device mapper infrastructure, and uses cryptographic routines from the kernel’s Crypto API. Both are widely used and understood in the IT community.

Weaknesses of single Master key

dm-crypt has a single Master Key that is used to encrypt / decrypt data in/out of the block. To ensure long term security and deal with changing authorized users/services, it would be necessary to change the Master Key frequently, and potentially share it with multiple users/services on a regular basis. Every new iteration of Master Key would require the underlying data block to be re-encrypted every time. In real systems, touched by different users/services, this is impractical.

Hierarchical key management

A more practical solution is to have a hierarchical key management setup in which users/services are given User Keys that are used to release the MasterKey. User Keys can be easily changed and revoked, without having to re-encrypt the underlying data block.
The management of such a hierarchical key managers is the role of LUKS.

In this post we show how to use Zymkey to lock a User Key, that is subsequently used to unlock the Master Key and provide access to the Root File System. If you’d like to learn more about LUKS see the References at the bottom of this post.


SECURE STORAGE OF LUKS USER KEYS

The security efficacy of your LUKS encrypted RFS is highly dependent upon how the User Keys are generated and where they are stored.

The SD Card is NOT a secure storage location

The growing Jetson Nano family is awesome and we love it! It is inexpensive, has an incredible amount of computing power for an embedded device and has a very robust software development ecosystem.

However, Jetson Nano has an Achilles heel: the SD card is the primary software deployment media and it can be very easily removed and manipulated.

The natural inclination would be to encrypt the file system using LUKS on dm-crypt, but for unattended use across many deployed units the obvious question is: where is the LUKS key stored? Of course, it’s the file system. Even if you try to obfuscate it through various programmatic means, the key is still very vulnerable to attack.

Securing LUKS User Key with Zymkey Security Module.

Zymkey provides a general “locking” service whereby a block of plaintext data is encrypted and signed.

When used with LUKS, the User Key is sent to the Zymkey to be locked (encrypted and signed) when the file system is created. When the system boots and needs to decrypt the root file system, the locked LUKS key is “unlocked” (signature verified and contents decrypted) and presented to dm-crypt. If the key was unlocked successfully, the boot process continues normally. Here is the boot sequence with a LUKS/dm-crypt filesystem where the key is protected by Zymkey:

  1. The kernel initializes initramfs
  2. initramfs presents the locked LUKS key to Zymkey
  3. Zymkey validates the signature and decrypts the key *
  4. The decrypted key is presented to LUKS and the root file system is then decrypted

*requires that Zymkey operational status is “secure”


Zymkey Security Module fitted to Jetson Nano

Zymkey Authenticates Host System Before Unlocking LUKS Key

One of the key features of Zymkey is to generate a unique Identity (ID) for the host system, based upon a fingerprint that measures specific system components. This fingerprinting process is used to “bind” together a specific Zymkey (root of trust, key store, crypto services), a specific host computer and a specific SD card. Once bound, these components form a permanent and immutable ID of the host system.

Each time the SBC boots, and at random intervals thereafter, the Zymkey rechecks the ID fingerprint. If any of the system components have changed the fingerprint changes and the system is deemed to have been compromised, authentication fails and all security services are shut down.

Using this ID / Authentication feature, Zymkey can be used to protect LUKS User Keys in unattended applications, where it might be easy to remove and copy SD card content. (Zymkey also has other physical security features which are also used to lock/enable security services)


WHERE TO STORE YOUR LUKS ENCRYPTED RFS

LUKS is very versatile and can be applied to both SD Card and external storage media. Lets review the pro’s and cons of each option:

Option 1 - Convert existing SD Card to LUKS

Converting the existing root file system on the SD card still requires an external device (e.g. USB flash drive) that is used as a temporary boot root file system: this provide an easier and lower risk means to convert and copy the original contents. The external devices needs to be a little larger than the existing root file system in order to store the old file system.

Pros:

  1. Less physical space requirements.
  2. Much less power required.

Cons:

  1. Conversion is more complex and time consuming than migrating to an external drive.
  2. Data space constraints.
  3. Write cycle constraints.
  4. Access speed constraints.

Process Steps:

  1. Make a tarball of the original root file system and store it on the external device. Separates out the /boot area
  2. Copy the original root file system files to the external device to form a temporary file system
  3. Boot to the temporary file system. Once booted, the temporary file system will:
    a. Create a LUKS key
    b. Lock the LUKS key with zymkey
    c. Create a LUKS volume on the SD card in a new partition. The standard Nano installation creates 14 partitions. In most cases, the new partition will be mmcblk0p15
    d. Create an ext4 partition on the LUKS volume on the new partition
    e. Untar the root file system tarball into the converted partition
    f. Untar the /boot area into the original SD card partition, mmcblk0p1

Option 2 - Migrate existing SD card to external LUKS storage device.

The existing root file system can be migrated to an external LUKS encrypted USB flash, hard drive or SSD.

Pros:

  1. External devices can hold much more data.
  2. Migration is easier and quicker than SD card conversion method.
  3. Some external devices have much faster data access than SD cards.
  4. Some external devices (e.g. HDD) can tolerate many more write cycles than an SD card.

Cons:

  1. For HDD and SSD and non-compact USB flash devices, there are additional power requirements.
  2. Except for compact USB flash devices, physical space requirements also increase.

Process Steps:

  1. Create the LUKS key
  2. Lock the LUKS key
  3. Create a LUKS volume on an external USB device
  4. Create an ext4 partition on the LUKS volume
  5. Move the existing root file system to the LUKS volume on the external device
  6. Copy the /boot area into the original SD card partition mmcblk0p1
  7. Boot to the new root file system

BUILDING YOUR LUKS ENCRYPTED RFS

Prerequisites

Make sure you have the Zymkey software suite already running and operational as well as insuring that your Zymkey is bound. Instructions [here]
(Getting Started: ZYMKEY4i with JETSON NANO).

Option 1 - Convert existing SD Card to LUKS

To convert your root file system to LUKS/dm-crypt, you will need to connect an external USB disk (as temporary storage). As mentioned previously, this is necessary because it is not possible to encrypt the partition in place, so the external disk is needed as temporary storage and a temporary root file system while the conversion takes place. The external disk needs to be at least twice as big as the root partition. Next, run the following script:

curl -G https://s3.amazonaws.com/zk-sw-repo/mk_encr_sd_rfs.sh | sudo bash

The script uses the following defaults:

  1. Original root file system located on /dev/mmcblk0p1
  2. Temporary root file system/storage for original root tarball located on /dev/sda
  3. Temporary root file system takes up entirety of new device

The very first run of this script on a new temporary external USB disk could take a long time. Also, two reboots are required before the script is complete.

One thing to note is that, if the external storage device has an ext4 formatted partition with the original root file system partition (e.g. /dev/mmcblk0p1) on it, this script will use what is already on the external storage device to convert the SD card. This cuts down time for converting lots of SBC root file systems and allows the script to be used in a mass production deployment.

The first run of this script can take upwards of 30 minutes to an hour to complete the first phase. The second phase takes around 15 minutes.

Based on the above, using the formatted external device to convert subsequent units should only take 15 minutes.

Option 2 - Migrate existing SD card to external LUKS storage device.

To migrate your root file system to an external USB device, you can run the following script:

curl -G https://s3.amazonaws.com/zk-sw-repo/mk_encr_ext_rfs.sh | sudo bash

This script is parameterized, so if you have special requirements, you can invoke in the following fashion:

curl -G https://s3.amazonaws.com/zk-sw-repo/mk_encr_ext_rfs.sh | sudo bash -s -- -x <path to external storage device (e.g. /dev/sdX> -p <destination partition number -s <max size of new root partition> -m <source partition number>

In the above invocation with no parameters, the defaults are:

  1. Original root file system located on /dev/mmcblk0p1
  2. Temporary root file system/storage for original root tarball located on /dev/sda
  3. Temporary root file system takes up entirety of new device

Please note that the new root file system should be at least a little larger in size than the original root partition

Running this script takes around 30-40 minutes. The Zymkey’s LED flashes rapidly until the process has completed.


INTEGRATING LUKS INTO VOLUME MANUFACTURING WORKFLOW

The examples above are designed to help you get up and running with single and low volume applications.

If you require support in developing a high volume manufacturing encryption workflow then please contact us to discuss our OEM engineering services.


REFERENCES

Hi, I tried this on an NVIDIA Xavier NX and it seems to work until

Mounting new crypto volume…done.
Copying files to crypto fs…
13,187,882,658 95% 13.82MB/s 0:15:09 (xfr#143436, to-chk=0/197453)
done.
Building initramfs…E: /etc/initramfs-tools/hooks/zymkey_cryptfs_cfg failed with return 2.
done.
Rebooting…

Any ideas are appreciated?

We only officially support the Nano. We have not tried on the Xavier. Our encryption script hard codes a file copy of the tegra21x_xusb_firmware library the Nano uses. I think the Xavier uses a different version, most likely tegra19x_xusb_firmware. We don’t have a Xavier to try, but If you’re willing to try a change, all that we think is needed is to change “21” to “19” in the encryption script.

Here’s how to do that:

Download our encryption script. I’m not sure if you were encrypting the SD card (Option 1) or using an External USB (Option 2):

Option 1 - Convert existing SD Card to LUKS
curl -G https://s3/amazonaws.com/zk-sw-repo/mk_encr_sd_rfs_nvidia.sh --output mk_encr_sd_rfs_nvidia.sh

Option 2 - Migrate existing SD card to external LUKS storage device
curl -G https://s3.amazonaws.com/zk-sw-repo/mk_encr_ext_rfs_nvidia.sh --output mk_encr_ext_rfs_nvidia.sh

Check to see if the version of the firmware is 19 on the Xavier:

ls /lib/firmware/tegra*_xusb_firmware

Assuming the filename references “19”, change 21 to 19 in the script. If it is something different, use that value:

Option 1
sed -i "s/tegra21x/tegra19x/" mk_encr_sd_rfs_nvidia.sh

Option 2
sed -i "s/tegra21x/tegra19x/" mk_encr_ext_rfs_nvidia.sh

Run our encryption script locally with sudo/root privilege:

sudo su

Option 1
./mk_encr_sd_rfs_nvidia.sh

Option 2
./mk_encr_ext_rfs_nvidia.sh

Please let me know if that works.

Bob

seems also the busid and gpio is changed on the Xavier
Ive hunted that part down
cat /var/lib/zymbit/zkenv.conf
ZK_GPIO_WAKE_PIN=436
ZK_RESTRICTED_I2C_BUS_LIST=8

modified with sed -i "s/tegra21x/tegra19x/" mk_encr_ext_rfs_nvidia.sh did the trick after setting the zkenv with the proper GPIO / I2C BUS … I can confirm it does now work!
./mk_encr_ext_rfs_nvidia.sh
No volume name (/dev/…) specified. Defaulting to /dev/sda…
Binary file /proc/device-tree/model matches
Exporting zkenv.conf
Installing necessary packages…
—SNIP----
Reading package lists… Done
Reading package lists… Done
Building dependency tree
Reading state information… Done
rsync is already the newest version (3.1.2-2.1ubuntu1.1).
zksaapps is already the newest version (1.0-13).
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
done.
Stopping udisks2…
done.
Formatting external storage media on /dev/sda…1+0 records in
1+0 records out
512 bytes copied, 0.0110623 s, 46.3 kB/s
Partition #1 contains a crypto_LUKS signature.
Stopping zkifc…done.
Creating LUKS key…done.
Created symlink /etc/systemd/system/multi-user.target.wants/encr_fs_cleanup.service → /etc/systemd/system/encr_fs_cleanup.service.
Formatting crypto file system on /dev/sda…done.
Creating ext4 partition on /dev/sda1…mke2fs 1.44.1 (24-Mar-2018)
done.
Mounting new crypto volume…done.
Copying files to crypto fs…
13,012,738,640 95% 13.32MB/s 0:15:31 (xfr#141141, to-chk=0/194659)
done.
Building initramfs…/sbin/ldconfig.real: Warning: ignoring configuration file that cannot be opened: /etc/ld.so.conf.d/aarch64-linux-gnu_EGL.conf: No such file or directory
/sbin/ldconfig.real: Warning: ignoring configuration file that cannot be opened: /etc/ld.so.conf.d/aarch64-linux-gnu_GL.conf: No such file or directory
done.
Rebooting…

Hi, I’m trying to get dm-crypt working on the Jetson Nano Developer Edition with Jetpack 4.3 as the OS in preparation for when our zymbits arrive. Given I don’t currently have the zymbit, I have modified the bash script, removing the zymbit-specific references, enabling logging and inserted a wait-for-user-confirmation so I can explore the system state after each change.

Original script from https://s3.amazonaws.com/zk-sw-repo/mk_encr_ext_rfs_nvidia.sh
Modifications to script mk_encr_ext_rfs_nvidia.sh to come (can’t upload more than 1 image - new users limitation).

I thought that I reached a point where it should work,

nvidia:~$ sudo ./mk_encr_ext_rfs.sh -x /dev/sda -s 20G

The following packages will be upgraded:
rsync
1 upgraded, 0 newly installed, 0 to remove and 446 not upgraded.
Need to get 308 kB of archives.
After this operation, 1,024 B of additional disk space will be used.
Get:1 http://ports.ubuntu.com/ubuntu-ports bionic-updates/main arm64 rsync arm64 3.1.2-2.1ubuntu1.1 [308 kB]
Fetched 308 kB in 2s (158 kB/s)
debconf: delaying package configuration, since apt-utils is not installed
(Reading database … 144666 files and directories currently installed.)
Preparing to unpack …/rsync_3.1.2-2.1ubuntu1.1_arm64.deb …
Unpacking rsync (3.1.2-2.1ubuntu1.1) over (3.1.2-2.1ubuntu1) …
Setting up rsync (3.1.2-2.1ubuntu1.1) …
Processing triggers for man-db (2.8.3-2ubuntu0.1) …
Processing triggers for systemd (237-3ubuntu10.29) …
done.
Stopping udisks2…
done.
Unmounting /dev/sda1…
Press Y to continue…
Yumount: /dev/sda1: not mounted.
Device cryptrfs is not active.
Formatting external storage media on /dev/sda…Press Y to continue…
Y34+0 records in
34+0 records out
17408 bytes (17 kB, 17 KiB) copied, 0.0221343 s, 786 kB/s
Partition #1 contains a crypto_LUKS signature.
Creating LUKS key…1+0 records in
1+0 records out
512 bytes copied, 0.000393654 s, 1.3 MB/s
done.
Created symlink /etc/systemd/system/multi-user.target.wants/encr_fs_cleanup.service → /etc/systemd/system/encr_fs_cleanup.service.
Formatting crypto file system on /dev/sda…Press Y to continue…
YCommand successful.
done.
Creating ext4 partition on /dev/sda1…mke2fs 1.44.1 (24-Mar-2018)
Creating filesystem with 5242368 4k blocks and 1310720 inodes
Filesystem UUID: ee110670-a192-4f8e-bc35-b00dab50aa28
Superblock backups stored on blocks:
32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
4096000

Allocating group tables: done
Writing inode tables: done
Creating journal (32768 blocks): done
Writing superblocks and filesystem accounting information: done

done.
Mounting new crypto volume…done.
Copying files to crypto fs…
Press Y to continue…
Y
12,867,339,196 98% 4.19MB/s 0:48:45 (xfr#128335, to-chk=0/182443)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1196) [sender=3.1.2]
done.
Building initramfs…/sbin/ldconfig.real: Warning: ignoring configuration file that cannot be opened: /etc/ld.so.conf.d/aarch64-linux-gnu_EGL.conf: No such file or directory
/sbin/ldconfig.real: Warning: ignoring configuration file that cannot be opened: /etc/ld.so.conf.d/aarch64-linux-gnu_GL.conf: No such file or directory
done.
Rebooting…
Press Y to continue…
qDid not receive ‘Y’, exiting…

but when I rebooted I received the following message:

Looking at that message suggests there are too many (3) root file systems, and /dev/mapper/cryptfs has failed to load. But I’m not sure if this is expected or not. I’m new to bootloading so any pointers / resources on how to proceed would be greatly appreciated.

e.g. does this suggest ${cbootargs} in /boot/extlinux/extlinux.conf needs modifying? Does that require a kernel image rebuild (rather than just initrd rebuild as this script does already)? etc.

Modifications to script mk_encr_ext_rfs_nvidia.sh.
visualPatch1 (only 1 image per post for new users).

Modifications to script mk_encr_ext_rfs_nvidia.sh.
visualPatch2 (only 1 image per post for new users).

@fazzle - The only thing the first script does is recognize you are on an NVIDIA product and then run the second script.
It’s difficult to accomplish anything very meaningful until you have a Zymkey. I would suggest waiting to receive your Zymkey.

Bob

Hey so we received our Zymkey and initially managed to get it working, but I’m at a point where I’m a bit stuck.
What I’ve done is saved stage 2 of the pipeline of mk_encr_sd_rfs_nvidia.sh (i.e. the external SD card / hard drive) to an image to try and speed things up. I boot to an internal SD card, that has the zymkey libs installed (by running install_zk_sw.sh) and then run mk_encr_sd_rfs_nvidia.sh which doesn’t do too much before it reboots (because it detects that the .tar files are already on the external SD card). However the cfg_SD_crfs.sh / service fails because zkgrifs and zklockifs hang. Both seem to be unresponsive whether zkifc.service is running or not. But the zymkey light responds to zkifc.service being switched on.
I should add that at one point I left zkgrifs running for perhaps 10min or so and it exited with ‘ERROR: no zymkeys installed’ (but zkifc.service prompted LED activity)
I tried deleting the /var/lib/zymkey/<UUID> directory and reconnecting the zymkey, deleting the /var/lib/zymkey/<UUID> directory and using a different zymkey, but I can’t seem to get zkgrifs to work.
In all cases the zymkey LED is responsive to the zkifc.service being switched on.
Any ideas before I go back to starting at stage 1?