Building and signing live patches

This page is for the owner of a HiTechCloud installation who has decided to produce live patches of their own, for kernels whose distribution publishes none (AlmaLinux, Rocky Linux, Debian, or an Ubuntu or RHEL server without the vendor service). It is the operating procedure for a pipeline whose software is built and tested, and whose operation is not: nothing on this page has been run on a real kernel, a real build host or a real mirror. Treat every command as unproven until the lab checklist has been done, and confirm the options of kpatch-build, sign-file and mokutil against the versions you have.

A live patch is code that runs in the kernel with the power to replace any kernel function. Whoever can get a patch loaded owns the machine, and every customer on it. That is why most of this page is about keys, review and what is refused.

What only the owner can do​

The software that ships does the checking and the packaging. It cannot, and must not, do any of the following. They are yours.

How a patch gets to a server​

1. A build host (one per target kernel build, never a customer server, no key on it) builds the live-patch module from the distribution's own source.
2. Two people review every CVE in it. The result is review.json.
3. hitechcloud-livepatch-tool bundle writes an unsigned bundle, and refuses unless the review is complete.
4. On an offline machine, a key (or a token) signs the manifest. The manifest names the module's SHA-256, the kernel release, the kernel build ID, the architecture, the CVEs, a serial, an expiry and the hash of the review, so the signature covers all of them.
5. The signed bundle is published into a mirror tree, an index of the tree is written and signed, and the tree is uploaded to the mirror host.
6. On a server, an administrator asks for the patch. The server checks the signatures, the kernel build and the review, and loads it.

Keys​

• One ed25519 key signs manifests and indexes. It is not the release key. A server refuses to trust a key that is the release key or an official module key, wherever the file is put.
• The key never goes in a repository, in a CI secret, on a build host or on a command line. The tool takes it as one line on standard input, or from a file that only its owner can read.
• A hardware token that never lets the key out is the recommended way. If the token supports ed25519 signatures, the tool prints the exact bytes to sign and accepts the answer; see Signing with a token.
• A kernel module signature (for Secure Boot) is a second, different key and certificate; see Secure Boot and MOK.
• Make the key on your own offline machine. This is the same format as the release key, with its own name:

umask 077
openssl genpkey -algorithm ed25519 -out hitechcloud-livepatch.pem
printf 'htc-ed25519-private %s\n' "$(openssl pkey -in hitechcloud-livepatch.pem -outform DER | tail -c 32 | base64 -w0)" > hitechcloud-livepatch.key
printf 'htc-ed25519 %s hitechcloud-livepatch-%s\n' "$(openssl pkey -in hitechcloud-livepatch.pem -pubout -outform DER | tail -c 32 | base64 -w0)" "$(date +%Y)" > hitechcloud-livepatch.pub
hitechcloud-livepatch-tool public-id --public hitechcloud-livepatch.pub

• The public line goes to each server as /etc/hitechcloud/livepatch/trusted-keys/<key id>.pub, in a directory owned by root and writable by nobody else. A server ignores a file whose name is not its key's id.
• To rotate, trust the new public key on servers first, sign with the new key, then remove the old key's file. If the private key may have leaked, remove its public file from every server at once: a stored index and every bundle it signed stop being accepted.

The tool​

hitechcloud-livepatch-tool is built from the cli/livepatch-tool directory of the hitechcloudcli repository (cargo build --release). It is run on the owner's own machines and is never installed on a server. It includes the same file that the servers' server-livepatch is built from, so a bundle it writes is judged by the code that judges it on a server.

hitechcloud-livepatch-tool help
hitechcloud-livepatch-tool review-guide
hitechcloud-livepatch-tool recipe rhel --kernel-release <uname -r of the target>

The human review​

A bundle needs a review.json that is complete and signed off. The tool writes a template in which every CVE is pending, every check is false and nobody has signed off:

hitechcloud-livepatch-tool review-template --module <module>.ko --id <patch id> --kernel-release <release> --build-id <build id> --arch <arch> --cve CVE-YYYY-NNNN --out review.json

The author and a different reviewer fill it in. The tool and every server refuse the bundle unless:

• every CVE in the manifest is in the review, and only those, each with a decision of patched and a written analysis
• the nine checks in review-guide are all true: the source is the distribution's exact source of the target build, the compiler and configuration are the build's, the change is the CVE's fix and nothing else, it is function-level or its data changes are reviewed, no patched function is in an init or an always-running path, and the module was booted, loaded, unloaded and loaded again, regression-tested and rolled back on the target build
• the review names this module's SHA-256, this kernel release, build ID and architecture
• an author and a reviewer who is not the author have signed off, each with a UTC time

This is a record, not a proof. The software checks that the record is complete and bound to the module; it cannot check that the people did the work. Signing is the owner's attestation that they did. Individual cryptographic signatures by each reviewer are not built (see Open decisions).

A CVE that cannot be live-patched is not put in a bundle. The answer for it is a reboot.

Build recipes​

Print the recipe and checklist for the target, with its kernel release filled in:

hitechcloud-livepatch-tool recipe rhel --kernel-release <release>
hitechcloud-livepatch-tool recipe ubuntu --kernel-release <release>
hitechcloud-livepatch-tool recipe debian --kernel-release <release>

What they say, in short:

• Every family. One build host per exact kernel build: the build's source (with the distribution's patches), its configuration, the matching debug-symbol vmlinux (several GB), and the same compiler and binutils the distribution used. A different compiler produces different code. The host is isolated, holds no key, and loads nothing but test modules on a disposable VM of the target build.
• RHEL, AlmaLinux, Rocky. The SRPM of the exact kernel, the matching kernel-debuginfo, and kpatch-build. Genuine RHEL has Red Hat's own kpatch-patch packages for selected CVEs, which is a reason not to build there. The EL kernels are heavily backported, so the patch is cut from the distribution's next kernel, not from upstream.
• Ubuntu. The source package that built that kernel flavour and version, the matching -dbgsym package, and kpatch-build. Canonical offers its own Livepatch service, which is a vendor service (decision D5). Ubuntu kernels are signed for Secure Boot by Canonical, so a module of ours needs our own module signature and MOK enrolment.
• Debian. The source package and the matching -dbg package. Debian publishes no live patches, and its own project reports that kpatch no longer supports the kernel of Debian 13. For Debian 11 and 12, try it in the lab first. For Debian 13, the realistic route is klp-build from upstream Linux 6.19 back-ported with its tooling, which is a project and not done.

The per-kernel-build checklist in every recipe has the same shape: build host, source, symbols, compiler, build, target facts (read from the target kernel itself, never from the build host), review, test on the target build, bundle, sign, publish. Do it again for every distribution kernel update, for every kernel your customers run.

Writing a bundle​

On the build host (no key), with the module from the build and the target's facts from server-livepatch status --json on a machine running that kernel:

hitechcloud-livepatch-tool next-serial --tree <mirror tree>
hitechcloud-livepatch-tool bundle --module <module>.ko --review review.json --out <bundle dir> --id <patch id> --kernel-release <release> --build-id <build id> --arch <arch> --cve CVE-YYYY-NNNN --serial <serial>

A bundle is exactly four files: manifest.json, manifest.sig (not yet), review.json and the module. The serial is one number across all bundles: a server refuses a patch whose serial is lower than the last one it applied, so take the next one from next-serial. A manifest is valid for 30 days by default and 90 at most.

Signing, offline​

Copy the bundle directory to the offline machine and sign it there. The tool re-checks everything a server will check before it signs.

hitechcloud-livepatch-tool sign --bundle <bundle dir> --forbid-key-id <release key id> < hitechcloud-livepatch.key
hitechcloud-livepatch-tool verify --bundle <bundle dir> --public hitechcloud-livepatch.pub

The key can also be a file given with --key-file, which the tool refuses unless only its owner can read it. A key as an argument is refused.

Signing with a token​

hitechcloud-livepatch-tool signing-input --bundle <bundle dir> --out manifest.signing-input
openssl pkeyutl -sign -rawin -inkey <token key URI> -in manifest.signing-input -out manifest.signature
hitechcloud-livepatch-tool attach --bundle <bundle dir> --public hitechcloud-livepatch.pub --signature manifest.signature

The middle command depends on the token and its PKCS#11 provider; it is an example, not a tested recipe, and many tokens do not support ed25519. The tool checks the signature against the public key before it writes it, and writes nothing when it is wrong.

The mirror​

The mirror is a static site: any web server that serves files over HTTPS. It only distributes. It cannot sign, and a compromised mirror cannot make a server load anything, because every file is checked against a signed index and the manifest's own signature. The layout is one directory per bundle:

livepatch/index.json
livepatch/index.sig
livepatch/<distribution>/<release>/<arch>/<kernel release>/<patch id>/manifest.json
livepatch/<distribution>/<release>/<arch>/<kernel release>/<patch id>/manifest.sig
livepatch/<distribution>/<release>/<arch>/<kernel release>/<patch id>/review.json
livepatch/<distribution>/<release>/<arch>/<kernel release>/<patch id>/<module>.ko

hitechcloud-livepatch-tool publish --tree <tree> --bundle <bundle dir> --distro ubuntu --release 24.04 --public hitechcloud-livepatch.pub
hitechcloud-livepatch-tool index --tree <tree> --public hitechcloud-livepatch.pub
hitechcloud-livepatch-tool sign --index <tree>/index.json < hitechcloud-livepatch.key
hitechcloud-livepatch-tool verify --tree <tree> --public hitechcloud-livepatch.pub

• The index lists every bundle with the size and SHA-256 of each file, its kernel release, build ID and architecture, and a list of withdrawn patches. It has its own serial, which only goes up, and an expiry (14 days by default, 30 at most): a stale index or an old copy replayed to a server is refused. Re-sign and re-upload before it expires.
• To withdraw a patch, run revoke --tree <tree> --id <patch id> --reason <text>, then index and sign --index again. Servers show the patch as withdrawn, and tell the administrator to roll it back if it is loaded.
• Upload the tree (including the new index.sig last) to the mirror host. Redirects are not followed, so serve the files directly. The address https://mirrors.hitechcloud.org is the project's own mirror, already listed in docs-internal/external-downloads.allow; a livepatch/ directory under it adds no new external host. The mirror's sync job does not write or prune a livepatch/ directory.

On a server​

All of this is root only, on the server's command line. The admin panel can show the result and ask for a refresh or an apply, and cannot change any of it.

install -d -m 0755 /etc/hitechcloud/livepatch/trusted-keys
install -m 0644 <key id>.pub /etc/hitechcloud/livepatch/trusted-keys/<key id>.pub
hitechcloudcli server-livepatch mirror set https://<mirror host>/livepatch
hitechcloudcli server-livepatch mirror refresh
hitechcloudcli server-livepatch mirror
hitechcloudcli server-livepatch fetch <patch id> --yes
hitechcloudcli server-livepatch rollback <module> --yes

The address is https only (plain http only for this machine, in a lab), with no user name or query. It is kept in a file only root can write. The default is none, and nothing is ever fetched until it is set. A server never fetches or applies by itself: no timer does it. A patch is not kept across a reboot.

Secure Boot and MOK​

Under Secure Boot, kernel lockdown or sig_enforce, the kernel loads only modules signed by a key it trusts. The manifest's ed25519 signature does not satisfy the kernel. The owner therefore needs, in addition:

• a module-signing key and X.509 certificate, kept offline like the manifest key
• every module signed with the kernel's own sign-file before it is bundled (the tool recognises the signature the kernel appends and records it in the manifest)
• the certificate enrolled on every Secure Boot server with mokutil --import, confirmed on the console at the next boot

A server that enforces signatures refuses an unsigned module before it is loaded, and says why. Without Secure Boot and without sig_enforce, the manifest signature is the only gate, which is why the review and the key matter so much.

Lab checklist​

Nobody has done this yet. Do it on disposable VMs and write the results into the design document.

1. Build hitechcloud-livepatch-tool, run its tests, and make a throwaway key.
2. On a VM of a real distribution kernel, build a trivial patch with kpatch-build on a separate build host, review it, bundle it, sign it with the throwaway key, and trust that key on the VM.
3. apply --bundle it. Check /sys/kernel/livepatch/<name>/enabled and transition, the kernel taint, rollback, and that a reboot removes it.
4. Check that each of these is refused: another build ID, a changed byte in the module, an unsigned manifest, an expired manifest, a lower serial, a changed review.
5. Publish it into a tree, index and sign, serve the tree over HTTPS, then mirror set, mirror refresh and fetch. Check that a stale index, a lower index serial, a swapped file and a withdrawn patch are each refused.
6. On a Secure Boot VM: an unsigned module is refused with the reason, and a module signed with an enrolled certificate loads.
7. A kernel thread pinned in a patched function: apply reverts after the limit and exits non-zero; rollback reports it stuck and never writes force.
8. Put the real public key on a spare server and repeat 3 with the real token.

Open decisions​

The owner's decisions D1 to D8 are in the design document. The ones this pipeline makes concrete:

• D1: who holds the signing key and the module-signing key, and where. The recommendation is a hardware token held by the owner.
• D2: whether to run a build operation at all, for which kernels, with which reviewers, and with what promise to customers. Start with one kernel line and "best effort, reboot otherwise".
• D4: whether to support MOK enrolment.
• D8: liability for a patch that panics a customer's server.
• Whether each reviewer should sign review.json with a key of their own, instead of the owner's signature attesting to the record. Not built.