Preliminary Call-For-Testing: Cross-DSO CFI

[Shawn Webb]

Call For Testing: Cross-DSO CFI in HardenedBSD
==============================================

Over the past year, HardenedBSD has been hard at work in integrating
the Cross-DSO CFI implementation in llvm. We have reached a point
where we can release an early public Call For Testing (CFT) of this
work.

For reasons which will be described below, we recommend this CFT be
used by those using root-on-ZFS with boot environments. We recommend
testing in a dedicated boot environment.

This initial round of testing is best suited for development server
installations. Production servers and desktops/laptops are not advised
for testing at this time. We're looking for feedback on what works and
doesn't work.

Introduction
------------

Control Flow Integrity, or CFI, is an exploit mitigation that aims to
make it harder for an attacker to hijack the control flow of an
executable image. llvm's CFI implementation provides forward-edge
protection, meaning it protects call sites and non-return code
branches. llvm includes basic and incomplete backward-edge protection
via SafeStack.

CFI in llvm consists of two flavors:

1. Non-Cross-DSO CFI
2. Cross-DSO CFI

For over a year now, HardenedBSD has adopted non-Cross-DSO CFI in
12-CURRENT/amd64. Support for non-Cross-DSO CFI was added for
12-CURRENT/arm64 on 01 July 2018. Non-Cross-DSO CFI applies CFI to the
applications themselves, but not on the shared objects they depend on.
Cross-DSO CFI applies CFI to both applications and shared objects,
enforcing CFI across shared object boundaries.

When an application or shared object is compiled, its source files
typically get compiled first to intermediate object files. Enabling
Cross-DSO CFI requires compiling and linking both static and shared
libraries with Link Time Optimization (LTO). When LTO is enabled,
these object files are no longer ELF object files, but rather LLVM IR
bitcode object files.

Linking applications that have been compiled with LTO generally only
requires ld.lld as the linker. Linking libraries that have been
compiled with LTO requires switching certain compiler toolchain
components to ones that understand LLVM IR bitcode. To prepare for
Cross-DSO CFI, we switched ar, ranlib, nm, and objdump to their
respective llvm compiler toolchain components. This gives us the
ability to use LTO across-the-board for the HardenedBSD userland, with
a few exceptions.

Note that because Cross-DSO CFI requires storing metadata regarding
the shared library boundaries at runtime, Cross-DSO CFI requires ASLR
and PaX NOEXEC at a minimum to be effective. If an attacker knows the
address of the metadata pages, the attacker can first perform
data-only attacks for later code execution/code reuse attacks.
Similarily, if an attacker is able to mark non-executable, yet
writable, pages as executable while still obeying CFI, (example: JIT
compiled code) the attacker can still gain perform execution/code
reuse attacks.

Known Issues And Limitations
----------------------------

There are a few known issues. Before we dive into the testing
procedure, I would like to talk a bit about known regressions. Note
that this list of known issues essentially also constitutes a
"work-in-progress" and every known issue will be fixed prior to the
official launch of Cross-DSO CFI.

It seems llvm does not like statically compiling applications with LTO
that have a mixture of C and C++ code. /sbin/devd is one of these
applications. As such, when Cross-DSO CFI is enabled, devd is compiled
as a Position-Independent Executable (PIE). Doing this breaks UFS systems
where /usr is on a separate partition. We are currently looking into
solving this issue to allow devd to be statically compiled again.

NO_SHARED is now unset in the tools build stage (aka, bootstrap-tools,
cross-tools). This is related to the static compilation issue above.
Unsetting NO_SHARED for to tools build stage is only a band-aid until
we can resolve static compliation with LTO.

One goal of our Cross-DSO CFI integration work is to be able to
support the cfi-icall scheme when dlopen(3) and dlsym(3)/dlfunc(3) is
used. This means the runtime linker (RTLD), must be enhanced to know
and care about the CFI runtime. This enhancement is not currently
implemented, but is planned.

When Cross-DSO CFI is enabled, SafeStack is disabled. This is because
compiling with Cross-DSO CFI brings in a second copy of the sanitizer
runtime, violating the One Definition Rule (ODR). Resolving this issue
should be straightforward: Unify the sanitizer runtime into a single
common library that both Cross-DSO CFI and SafeStack can link against.

As of 07 Jun 2018, libpmc and friends are receiving a lot of code
churn in upstream FreeBSD. The jevents application in
lib/libpmc/pmu-events is used as a build tool to generate code.
Enabling Cross-DSO CFI disables building PMC-related tools (libpmc and
friends) due to the jevents application segfaulting during the build
process.

When the installed world has Cross-DSO CFI enabled, performing a
buildworld with Cross-DSO CFI disabled fails. This is somewhat related
to the static compilation issue described above.

Linking with Cross-DSO CFI can cause lld to use an extremely large
amount of memory. For each parallel build job, budget around 15GB of
memory for the linker.

Due to the issues discussed above, this CFT is applicable to users who
either use ZFS or where /usr is contained within the root filesystem.

Procedure For Testing
---------------------

Use git to clone locally the HardenedBSD Playground repo. The
instructions below assume using /usr/src as the location for the
source tree. It also blows away your existing source tree, if it
exists. If you want to keep your existing source tree, feel free to
modify the steps below to your liking.

Due to the complexity of building Cross-DSO CFI, the buildworld step
must be completed twice: once without Cross-DSO CFI and a second time
with. The non-Cross-DSO CFI world must be installed prior to
performing the second build. As noted above, we will work to ensure
this doesn't need to happen later.

These instructions make the following assumptions:

1. The root filesystem is on ZFS, with the proper layout for ZFS Boot
Environments.
2. The beadm package is installed.
3. The existing installation is running 12-CURRENT on amd64 or arm64.

# cd /usr
# rm -rf src
# git clone https://github.com/HardenedBSD/hardenedBSD-playground.git \
src
# cd src
# git checkout -b hardened/current/cross-dso-cfi \
origin/hardened/current/cross-dso-cfi
# make -sj$(sysctl -n hw.ncpu) buildworld buildkernel
# beadm create cfi-01
# beadm mount cfi-01 /tmp/newbe
# make -s installworld installkernel DESTDIR=/tmp/newbe
# mergemaster -iFUD /tmp/newbe
# beadm umount cfi-01
# beadm activate cfi-01
# shutdown -r now

Binary Updates
--------------

We will provide binary updates in base for the
hardened/current/cross-dso-cfi feature branch until this work gets
merged into hardened/current/master. Take a look at Appendix A for a
sample hbsd-update.conf configuration file for the Cross-DSO CFI work.

Future Work
-----------

We're not done, yet! There's still plenty of work to do. Of upmost
importance is fixing static compilation with LTO enabled. Without it,
statically-linked applications will crash. devd can go back to being a
statically-linked application and users with /usr on a separate
non-ZFS filesystem will be able to take advantage of Cross-DSO CFI.

Secondly, we need to re-integrate SafeStack, giving us backward-edge
protections once again.

Third up is integration with the RTLD. Without it, we still need to
disable the cfi-icall scheme for applications that make use of
dlopen(3)+dlsym(3)/dlfunc(3).

Given that we're in uncharted territory, we will likely find other
issues. We will keep the community updated and informed. Once all
issues have been resolved, we will work on integration with ports.

We need to ensure buildworld works with the various CFI (MK_CFI and
MK_CROSS_DSO_CFI) options toggled, regardless of installed world
state.

Given the tremendous memory requirements, HardenedBSD may not be able
to apply Cross-DSO CFI across the entire package repository. The
current amd64 package building system is a dual Xeon system with
eight cores per CPU (sixteen cores total), 192GB RAM, and 64GB swap.
Without Cross-DSO CFI, building the package repo for amd64 takes
around 82 hours to complete using all sixteen cores. With Cross-DSO
CFI enabled, the package build server eventually runs out of swap
using only eight of the sixteen cores.

Though we plan to support Cross-DSO CFI in arm64, amd64 will be the
primary development platform until the major issues are worked out.
The sanitizer framework needs to be updated to take
FreeBSD/HardenedBSD into account on arm64. As of 14 July 2018, the
llvm sanitizer framework does not support FreeBSD/arm64. With time, we
plan to change that.

HardenedBSD may very well be the first UNIX-like operating system with
full Cross-DSO CFI integration across its entire base operating
system userland.

Appendix A - hbsd-update.conf
-----------------------------

# hbsd-update.conf
# Configuration settings for hbsd-update.
# This file is read in through a /bin/sh shell and uses that syntax.

# dnsrec:
# DNS TXT record to use when looking up the version info for the
# latest update.
#
# This record name seems redundant, but it provides the following
# information:
# 1) architecture
# 2) branch (hardened/current/master) in reverse form
# 3) repo (hardenedbsd)
dnsrec="$(uname -m).cross-dso-cfi.current.hardened.hardenedbsd-playground.updates.hardenedbsd.org"

# kernel:
# Which kernel to install.
# By default, this is intelligently detected by parsing `uname -v`
# output.
#kernel="HARDENEDBSD"

# capath:
# Location of the trusted root certificate store.
capath="/usr/share/keys/hbsd-update/trusted"

# branch:
# Which branch/tag we are pointing to. This option is only used in
# this file for the baseurl option below.
branch="hardened/current/cross-dso-cfi"

# baseurl:
# Where to get the update from.
baseurl="http://updates.hardenedbsd.org/pub/HardenedBSD-playground/updates/${branch}/$(uname -m)"

# dnssec:
# Use DNSSEC for validating the DNS TXT record. Default: yes
#dnssec="no"

Thanks,

--
Shawn Webb
Cofounder and Security Engineer
HardenedBSD

Tor-ified Signal: +1 443-546-8752
Tor+XMPP+OTR: lat...@is.a.hacker.sx
GPG Key ID: 0x6A84658F52456EEE
GPG Key Fingerprint: 2ABA B6BD EF6A F486 BE89 3D9E 6A84 658F 5245 6EEE