Running Envoy on CentOS 6/7

Envoy currently does not produce a binary that runs out of the box on CentOS 6/7. The main build container is based on Ubuntu and it produces a binary that requires glibc >= 2.18. Unfortunately, CentOS 7 is on 2.17 and CentOS 6 is on 2.12.

Rather than taking on ownership of a custom build, we found a more creative way to get Envoy running. There's a tool called PatchELF that allows you to modify an ELF binary's interpreter path. On a Linux system, this is generally /lib/ld-linux.so.2, but with PatchELF we can change that to a relative path containing a custom build of glibc.

The process for creating our modified Envoy has 2 main steps:

  1. Fetch the Envoy docker image for the target version and extract the binary.
  2. Build glibc 2.18 and patch the Envoy binary to use it from a relative path.

Fetching the Envoy binary

We have an update script that plucks the binary from the docker image and saves some metadata about which version we currently have:

#!/usr/bin/env bash
#
# Usage
# ./fetch_envoy.sh <ENVOY_SHA>

set -eu

DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
ENVOY_DIR="${DIR}/envoy"

ENVOY_SHA=$1

# Since envoy doesn't publish a standalone binary yet, we fetch their docker
# image and extract the binary from it.
ENVOY_CONTAINER_ID=$(docker create envoyproxy/envoy:${ENVOY_SHA})

docker cp ${ENVOY_CONTAINER_ID}:/usr/local/bin/envoy "${ENVOY_DIR}/envoy"
docker rm ${ENVOY_CONTAINER_ID}

echo "${ENVOY_SHA}" > "${ENVOY_DIR}/ENVOY_SHA"

Packaging the modified Envoy binary

A CentOS-based Dockerfile is then used to build glibc 2.18, prep Envoy to use it, and package it all together:

FROM centos:7

ARG glibc_version=2.18
ARG patchelf_version=0.9

RUN yum install -y autoconf automake gcc gcc-c++ git make wget

ADD glibc-${glibc_version}.tar.xz .
ADD patchelf-${patchelf_version}.tar.bz2 .

RUN mkdir glibc-${glibc_version}-build && \
    cd glibc-${glibc_version}-build && \
    ../glibc-${glibc_version}/configure --prefix=/opt/glibc-${glibc_version} && \
    make -j4 && \
    make install && \
    cd ..

RUN cd patchelf-${patchelf_version} && \
    ./configure && \
    make && \
    make install && \
    cd ..

WORKDIR /package
ADD envoy envoy

# Patch the envoy binary so that it uses our custom glibc build.
RUN cp -r /opt/glibc-${glibc_version}/lib . && patchelf --set-interpreter lib/ld-linux-x86-64.so.2 --set-rpath lib envoy

RUN tar cf envoy.tar *

# CMD was required because of the error & explanation here: https://github.com/hashicorp/vagrant/issues/4602
CMD ["/bin/bash"]

The Dockerfile assumes that glibc-2.18.tar.xz and patchelf-0.9.tar.bz2 exist along with it in the same directory as the raw Envoy binary.

You can run the build and extract the custom Envoy package with the following script:

#!/usr/bin/env bash
#
# Usage:
# ./package_centos_envoy.sh

set -eu

DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
ENVOY_DIR="${DIR}/envoy"

# Use our envoy packaging image to do our custom glibc build and patch the envoy binary to use it.
docker build -f "${ENVOY_DIR}/Dockerfile-package-envoy" -t centos-envoy-package:build "${ENVOY_DIR}"

PACKAGE_CONTAINER_ID=$(docker create centos-envoy-package:build)

docker cp ${PACKAGE_CONTAINER_ID}:/package/envoy.tar "${ENVOY_DIR}/centos-envoy.tar"
docker rm ${PACKAGE_CONTAINER_ID}

That should leave you with a centos-envoy.tar package containing the modified Envoy binary and its custom glibc 2.18 build.

It feels like a pretty nasty hack, but we've been using it without issues for close to a year now. Over that time, creating and maintaining a custom CentOS-friendly build of the raw binary would have involved a lot more effort than this approach (which has essentially worked without changes since we started doing it).