This document provides a key management strategy using Yubico's YubiHSM2. You may use it to manage private Ed25519 keys for log servers and witnesses.
Basic understanding of how Yubico's YubiHSM 2 works. For orientation, take a look at Yubico's quickstart tutorial and the YubiHSM 2 user guide. Pay extra close attention to "Core Concepts" and "YubiHSM Command Reference".
The overall objectives are:
- Signing requires access to a localhost-only YubiHSM and a 128-bit secret generated while the device was (re)provisioned. This signing-oracle YubiHSM is plugged into a networked node at all times (i.e., a primary log server, a secondary log server, or a witness), and is only usable as a signing oracle if a YubiHSM authentication passphrase is made available. The passphrase is only accessible as root and only when the node is in active use.
- (Re)provisioning requires access to a backup YubiHSM and a different 128-bit secret. Backup YubiHSMs are never plugged into a networked node; similarly, backup passphrases are never entered into a networked node.
- A single trusted operator can gain access to all YubiHSMs and the secret passphrases needed to make use of them. However, backup YubiHSMs and their secret passphrases are never stored at rest in the same physical location.
These objectives imply a threat model where trusted operators are honest at all times. In other words, what we are trying to protect against are external threats and failures. Examples of threats include a device that gets physically stolen or remotely accessible as a result of a networked node being owned. Examples of failures include devices that break. We assume that at least one out of two backup YubiHSMs and one out of three backup USB thumb drives function at all times.
Randomness produced by a YubiHSM is assumed to be good, so that the generated keys and passphrases are computationally hard to brute force (128-bit security).
For key recovery, the attacker needs to gain access to a backup YubiHSM and its secret passphrase. This is assumed to be hard: two different physical security boundaries need to be breached at the same time for the attacker to succeed.
For signing-oracle access, a (local) attacker can steal a node's YubiHSM and breach another physical security perimeter to gain access to its passphrase. This, too, is assumed to be hard.
For signing-oracle access, a (remote) attacker may gain root access on an active node, or in the case of non-root access learn the secret passphrase by breaching a physical security perimeter that stores it. This may be the most likely attack scenario. It is also hard to detect if no physical boundary is breached.
Note how log servers have a larger (remote) attack surface than witnesses, simply because witnesses can be operated from less public vantage points.
For physical security and detection of breaches we rely on locks, alarms, safes, tamper-evident bags, etc. Detection of a breach is generally assumed to be rapid and obvious; but as a defense-in-depth we also do a few routine checks.
Secret passphrases are used for YubiHSM authentication and key wrapping. These secrets are always 32 hex-characters and generated by the YubiHSM's get-random.
Secret passphrases are stored redundantly on n = 2 USB thumb drives. Use
different models or vendors to reduce the likelihood of simultaneous failures.
The storage capacity requirements are modest -- at most a handful of secret
passphrases need to be stored on each USB thumb drive. We discourage n < 2
to get a reasonable level of reliability.
These USB thumb drives are stored in tamper-evident bags at separate locations. Make sure that a secret passphrase is not stored together with a YubiHSM for which the passphrase is valid. Use a safe for additional physical security. Record the tamper-evident bags' serial numbers in a git repository. Sync all changes to this git repository between the trusted operators out-of-band.
Below is an example of a table that could be tracked in a git repository. E.g., before opening a tamper-evident bag and subsequently replacing it, one would confirm with the others that they currently have and still get the same table.
| Date | Storage location | Device | Bag serial number | Notes |
|---|---|---|---|---|
| YYYY-MM-DD | Location A | USB-1 | 00 000 000 001 | Initial provisioning |
| YYYY-MM-DD | Location B | USB-2 | 00 000 000 002 | Initial provisioning |
| YYYY-MM-DD | Location A | USB-1 | 00 000 000 004 | Promote logsrv secondary |
If a tamper-evident bag is breached or stolen, all secret passphrases must be changed immediately. This involves all backup and signing-oracle YubiHSMs.
If a USB thumb drive is plugged into a system that is not a dedicated provisioning machine (introduced below), it should be considered compromised.
The above is essential for passphrases associated with backup YubiHSMs (described below). To make day-to-day operations easier, you may store passphrases associated with signing-oracle YubiHSMs (also described below) on separate USB thumb drives that are otherwise tracked in the same way as above with one main difference: they may be plugged into another machine than the one used for provisioning. If so, do not plug them back into the provisioning machine again as such USB thumb drives have been exposed to less safe places.
(This will make day-to-day operations much easier. It also reflects that signing-oracle passphrases will necessarily be exposed on other machines. There is a concrete example of how to do the split in the repository's README file.)
Routine: check safe and its tamper-evident bag once per month.
We have n = 2 backup YubiHSMs that are complete replicates of each other:
- An
authkeyobject with all capabilities, delegated capabilities, and domains enabled. This provides complete YubiHSM device access. - A
wrapkeyobject for all domains but only with capabilitiesimport-wrapped,export-wrappedand delegated capabilitiesexportable-under-wrap,sign-eddsa. This provides the ability to export and import private Ed25519 keys in encrypted form between YubiHSM devices. - An
asymmetric-keyobject with capabilitiesexportable-under-wrap,sign-eddsaand algorithmed25519. This can be used as the log server signing key. It is placed in its own domain. - Another
asymmetric-keyobject with capabilitiesexportable-under-wrap,sign-eddsaand algorithmed25519. This can be used as the witness signing key. It is placed in its own domain.
(Storing the two signing keys in different domains is not necessary when each networked node gets its own YubiHSM. Such a setup makes it easier to change this in the future though, i.e., if a signing-oracle YubiHSM becomes shared.)
These four objects are generated on an initial YubiHSM, then exported and
imported under wrap to the other backup. The secret passphrases associated with
authkey and wrapkey are stored on 2x USB thumb drives as described above.
All of this takes place on a dedicated provisioning machine without network, to avoid that the wrapped key material ever touches a networked node. In other words, always do the (wrapped) YubiHSM exporting and importing on this provisioning machine. Store it with the most accessible backup YubiHSM.
Store backup YubiHSMs in tamper-evident bags on different secure locations. Use
the same procedure to track locations and serial numbers as for USB thumb
drives. We discourage n < 2 to get a reasonable level of reliability.
If it is detected that a backup YubiHSM is breached (e.g., it is lost or somehow
broken), reprovision so that there is a new set of n = 2 backup YubiHSMs. Use
newly generated authkey and keywrap passphrases. Destroy old passphrases.
Routine: check that the most accessible backup YubiHSM (e.g., office) works every three months. Check all other backup YubiHSM replicas on a yearly basis.
The signing-oracle YubiHSMs get their key(s) imported under wrap while plugged
into the provisioning machine. After import, the wrap key and the default
authentication key is deleted in favor of an authentication key that only
permits signing: capability sign-eddsa and delegated capabilities none.
The domain can be set to all or only to match the log server or witness key; it
does not really matter as no new key can be imported without reprovisioning.
Each signing-oracle YubiHSM should have its own authkey passphrase.
Assuming a primary log server, a secondary log server, and a witness, this
results in another three secrets to be stored on the 2x USB thumb drives.
As noted earlier, day-to-day operations will be much easier if different USB thumb drives are used for this (so they can be plugged into a regular laptop).
Only transfer secret passphrases to networked nodes that are in active use. Do this manually, and store them in files that require root privileges to access.
If it is detected that a signing-oracle YubiHSM is breached (e.g., it is lost or somehow broken), reprovision a new one from backup. Destroy the old passphrase.
Routine: Automate checks that verify if a node's YubiHSM is plugged-in. Delete passphrases that are stored on disk if a node becomes inactive.
- 2x backup YubiHSMs
- 1x signing-oracle YubiHSM (primary log server)
- 1x signing-oracle YubiHSM (secondary log server)
- 1x signing-oracle YubiHSM (witness)
- 2x USB thumb drives for backup (different models/vendors)
- 2x USB thumb drives for operations (optional but recommended)
- May be the same model/vendor (no essential secrets wrt. key recovery)
- 1x dedicated provisioning machine configured with YubiHSM2 tooling, the scripts for automation (see README instructions), and no network access
- 1x git repository
- Many tamper-evident bags
- 2x physically secure sites for backup YubiHSMs and provisioning machine
- 2x physically secure locations for USB thumb drives
- Secure site(s) where the machines with online YubiHSMs are operated
The sites that contain backup and signing-oracle YubiHSMs may be the same.
Log in as a non-root user.
Change directory to /dev/shm. Erasing data from an SSD is very hard if at all possible due to wear leveling, garbage collection and probably more. Even if the provisioning process will not write any key material to disk it will write highly sensitive secrets. If you happen to write backup passphrases to disk anyway, don't store the provisioning machine together with any of the backup YubiHSM's.
Run all provision scripts and follow the instructions. Take note of which YubiHSM serial number is provisioned with what. Save stdout output to each USB thumb drive as it contains the above secret passphrases.
Put backup YubiHSMs and USB thumb drives into tamper-evident bags. Populate the git repository table with initial storage locations and serial numbers.
Transport the different devices to their distinct storage locations. Plug signing-oracle YubiHSMs into their respective networked nodes, or store them in the same physical location as a backup YubiHSM until they are ready to be used.
Transfer the respective authentication passphrases to nodes that are active.
Appoint someone to be responsible for following up on routine checks.
If it is desired to showcase to others that this provision ceremony was used, optionally invite one or more independent parties to witness and take notes.