Envoy currently does not produce a binary that runs out of the box on CentOS 6 or CentOS 7. The main build container is based on Ubuntu and it produces a binary that requires glibc >= 2.18. Unfortunately, CentOS 7 is on 2.17 and CentOS 6 is on 2.12.
Rather than taking on ownership of a custom build, we found a more creative way to get Envoy running. There’s a tool called PatchELF that allows you to modify an ELF binary’s interpreter path. On a Linux system, this is generally /lib/ld-linux.so.2, but with PatchELF we can change that to a relative path containing a custom build of glibc.
The process for creating our modified Envoy has 2 main steps:
- Fetch the Envoy docker image for the target version and extract the binary.
- Build glibc 2.18 and patch the Envoy binary to use it from a relative path.
Fetching the Envoy binary
We have an update script that plucks the binary from the docker image and saves some metadata about which version we currently have:
#!/usr/bin/env bash
# Usage
# ./fetch_envoy.sh <ENVOY_SHA>
set -eu
DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
ENVOY_DIR="${DIR}/envoy"
ENVOY_SHA=$1
# Since envoy doesn't publish a standalone binary yet, we fetch their docker
# image and extract the binary from it.
ENVOY_CONTAINER_ID=$(docker create envoyproxy/envoy:${ENVOY_SHA})
docker cp ${ENVOY_CONTAINER_ID}:/usr/local/bin/envoy "${ENVOY_DIR}/envoy"
docker rm ${ENVOY_CONTAINER_ID}
echo "${ENVOY_SHA}" > "${ENVOY_DIR}/ENVOY_SHA"
Packaging the modified Envoy binary
A CentOS-based Dockerfile is then used to build glibc 2.18, prep Envoy to use it, and package it all together:
FROM centos:7
ARG glibc_version=2.18
ARG patchelf_version=0.9
RUN yum install -y autoconf automake gcc gcc-c++ git make wget
ADD glibc-${glibc_version}.tar.xz .
ADD patchelf-${patchelf_version}.tar.bz2 .
RUN mkdir glibc-${glibc_version}-build && \
cd glibc-${glibc_version}-build && \
../glibc-${glibc_version}/configure --prefix=/opt/glibc-${glibc_version} && \
make -j4 && \
make install && \
cd ..
RUN cd patchelf-${patchelf_version} && \
./configure && \
make && \
make install && \
cd ..
WORKDIR /package
ADD envoy envoy
# Patch the envoy binary so that it uses our custom glibc build.
RUN cp -r /opt/glibc-${glibc_version}/lib . && patchelf --set-interpreter lib/ld-linux-x86-64.so.2 --set-rpath lib envoy
RUN tar cf envoy.tar *
# CMD was required because of the error & explanation here: https://github.com/hashicorp/vagrant/issues/4602
CMD ["/bin/bash"]
The Dockerfile assumes that glibc-2.18.tar.xz and patchelf-0.9.tar.bz2 exist along with it in the same directory as the raw Envoy binary.
You can run the build and extract the custom Envoy package with the following script:
#!/usr/bin/env bash
# Usage:
# ./package_centos_envoy.sh
set -eu
DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
ENVOY_DIR="${DIR}/envoy"
# Use our envoy packaging image to do our custom glibc build and patch the envoy binary to use it.
docker build -f "${ENVOY_DIR}/Dockerfile-package-envoy" -t centos-envoy-package:build "${ENVOY_DIR}"
PACKAGE_CONTAINER_ID=$(docker create centos-envoy-package:build)
docker cp ${PACKAGE_CONTAINER_ID}:/package/envoy.tar "${ENVOY_DIR}/centos-envoy.tar"
docker rm ${PACKAGE_CONTAINER_ID}
That should leave you with a centos-envoy.tar package containing the modified Envoy binary and its custom glibc 2.18 build.
It feels like a pretty nasty hack, but we’ve been using it without issues for close to a year now. Over that time, creating and maintaining a custom CentOS-friendly build of the raw binary would have involved a lot more effort than this approach (which has essentially worked without changes since we started doing it).