Someone emailed me the other day and was having some trouble getting

their program to work under OPS.

He had downloaded a binary release from github and knowing that OPS only runs linux binaries chose the linux version. Further, there are numerous

examples of people running linux programs under ops on osx. Hell, just do a



ops pkg list

for a short list.

However, for some reason it just wasn't working for him.



user@users-MacBook-Pro Desktop % ops run solana-validator * errors.errorString libstdc++.so.6: file does not exist /Users/eyberg/go/src/github.com/nanovms/ops/lepton/ldd_darwin.go:100 ( 0x4c55779 ) /Users/eyberg/go/src/github.com/nanovms/ops/lepton/ldd_darwin.go:129 ( 0x4c5607d ) /Users/eyberg/go/src/github.com/nanovms/ops/lepton/image.go:261 ( 0x4c52d0e ) /Users/eyberg/go/src/github.com/nanovms/ops/lepton/image.go:24 ( 0x4c50f5f ) /Users/eyberg/go/src/github.com/nanovms/ops/cmd/run.go:15 ( 0x4ca4189 ) /Users/eyberg/go/src/github.com/nanovms/ops/cmd/run.go:118 ( 0x4ca4f75 ) /Users/eyberg/go/pkg/mod/github.com/spf13/cobra@v0.0.5/command.go:830 ( 0x4c8bf3e ) /Users/eyberg/go/pkg/mod/github.com/spf13/cobra@v0.0.5/command.go:914 ( 0x4c8cb5b ) /Users/eyberg/go/pkg/mod/github.com/spf13/cobra@v0.0.5/command.go:864 ( 0x4ca8d88 ) /Users/eyberg/go/pkg/mod/github.com/spf13/cobra@v0.0.5/command.go:864 ( 0x4ca8d83 ) /usr/local/go/src/runtime/proc.go:200 ( 0x402f57c ) main: return /usr/local/go/src/runtime/asm_amd64.s:1337 ( 0x405a671 ) goexit: GLOBL shifts< >( SB ) ,RODATA, $256 panic: libstdc++.so.6: file does not exist

After I saw the word mac and then libstdc++ I knew immediately what was wrong but I also realized that it might not be readily apparent to many people and so this is what this blogpost is about.

The binary in question is dynamically linked. What this means is that

there are several libraries that the dynamic loader (ld) will try and

load at runtime versus statically linked where all the libraries are

packaged inside the binary. This leads to a fatter binary but you can be

assured that everything is present when needed. I won't jump into a

static/dynamic flamewar as they each have their own usecases.

I should also point out that normally you can't run linux binaries directly on a mac - this is actually a uniquely interesting unikernel aspect as we don't need to boot up vagrant or a linux vm. Traditional virtualization software virtualizes the operating system. You can see using a tool like OPS as virtualizing the application instead.

One of the reasons I decided to write this post was not just the email but I've noticed in the past few years I've seen quite a lot of people that use containers and go to refer to their programs as statically linked when it is in fact absolutely not.

Let's look at this example go webserver:



package main import ( "fmt" "net/http" ) func main () { http.HandleFunc ( "/" , func ( w http.ResponseWriter, r * http.Request ) { fmt.Fprintf ( w, "Welcome to my website!" ) }) fs : = http.FileServer ( http.Dir ( "static/" )) http.Handle ( "/static/" , http.StripPrefix ( "/static/" , fs )) http.ListenAndServe ( ":8080" , nil ) }

Fairly simple - all it does is listen on 8080 and serve up requests. If we compile it:



go build

We see that by default it will be dynamically linked against a few libs:



eyberg@box:~/z$ ldd z linux-vdso.so.1 (0x00007ffe8b1db000) libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f2e091f4000) libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f2e08e03000) /lib64/ld-linux-x86-64.so.2 (0x00007f2e09413000) eyberg@box:~/z$ file z z: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/l, not stripped

VDSO gives us a fast clock. Libpthread gives us threads. Libc contains everything you'd find in section 3 of the man pages and finally LD is our loader.

You can link your go programs statically via something like this:



eyberg@box:~/z $ go build -buildmode = pie -ldflags "-linkmode external -extldflags -static" # _/home/eyberg/z /tmp/go-link-057347370/000004.o: In function ` _cgo_7e1b3c2abc8d_C2func_getaddrinfo ': /tmp/go-build/cgo-gcc-prolog:57: warning: Using ' getaddrinfo ' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking eyberg@box:~/z$ ldd z not a dynamic executable eyberg@box:~/z$ file z z: ELF 64-bit LSB executable, x86-64, version 1 (GNU/Linux), statically linked, for GNU/Linux 3.2.0, BuildID[sha1]=0671d22e52baf812ed9f500a8d9ee7848b8c809e, not stripped

Both ldd && file now report that it is statically linked. However, you'll notice (at least in go 1.12) it now outputs a message stating that getaddrinfo is still required! That's cause even though ldd output and file is telling us that it is indeed static we still rely on libnss for dns. You can force go to use its built in resolver instead.

Getaddrinfo is found here:



eyberg@box:~/s/solana-release/bin $ readelf --dyn-syms /lib/x86_64-linux-gnu/libc-2.27.so | grep getaddr 873: 0000000000107bc0 3261 FUNC GLOBAL DEFAULT 13 getaddrinfo@@GLIBC_2.2.5 1601: 00000000001413c0 43 FUNC GLOBAL DEFAULT 13 inet6_rth_getaddr@@GLIBC_2.5

You'll note that your libraries are probably stripped which means you can't use something like nm to find the symbols but readelf surfaces them up in the .dynsym section.

Also, even that is not enough - you'll probably make use of various functions found in these files as well:



eyberg@box:~/z$ ls /lib/x86_64-linux-gnu/libnss_files.so.2 /lib/x86_64-linux-gnu/libnss_files.so.2 eyberg@box:~/z$ ls /lib/x86_64-linux-gnu/libnss_dns.so.2 /lib/x86_64-linux-gnu/libnss_dns.so.2

The point here is that just sticking a go program in a container does not inherently make your binary 'statically linked' and the problem is that, that is not what a lot of people have been stating and this source of

confusion is probably what leads to misunderstandings such as this.

So let's go back to our example at the beginning of the article.

The program in question is the solana blockchain.

You can download and unzip like so:



wget https://github.com/solana-labs/solana/releases/download/v0.23.2/solana-release-x86_64-unknown-linux-gnu.tar.bz2 bunzip2 solana-release-x86_64-unknown-linux-gnu.tar.bz2 tar xf solana-release-x86_64-unknown-linux-gnu.tar

You can see from the ldd output that it is trying to load a library that is dynamically linked (libstdc++) and that's what our error in the trace was.



eyberg@box:~/s/solana-release/bin $ ldd solana linux-vdso.so.1 ( 0x00007ffe338df000 ) libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 ( 0x00007f06976a8000 ) /lib64/ld-linux-x86-64.so.2 ( 0x00007f06984a2000 ) libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 ( 0x00007f06974a4000 ) librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 ( 0x00007f069729c000 ) libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 ( 0x00007f069707d000 ) libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 ( 0x00007f0696e65000 ) libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 ( 0x00007f0696ac7000 )

As you can see this program works out of the box on linux.



eyberg@box:~/s/solana-release/bin $ ops run -c config.json solana-validator [ solana-validator --ledger /] booting /home/eyberg/.ops/images/solana-validator.img ... qemu-system-x86_64: warning: TCG doesn 't support requested feature: CPUID.01H:ECX.vmx [bit 5] assigned: 10.0.2.15 solana-validator 0.23.2 log file: solana-validator-7wgZy3ozTRyvLtRN5o3Xh6m5aEvEo2XBxP5yPfaCny1t-20200210-204507.log eyberg@box:~/s/solana-release/bin$ cat config.json { "Args": ["solana-validator", "--ledger", "/"] }

This is all fine but if you want to run it on osx you'll need to do one

of two things:

1) Either build from source and statically link it (so ldd doesn't show

output).

or

2) Download the libraries it's linked to and manually add them to the

filesystem.

The reason this works out of the box on linux is that OPS looks at and

try to load the libraries as we build the disk image. This isn't

possible on a mac cause mac uses mach-o which is a different format that Nanos does not and probably won't ever support as nanos is explicitly designed to run linux server-side programs.

For example when we build common packages like nginx or node, you'll see from the ops output that all required libraries are placed on the filesystem in a known location.



➜ ~ ops pkg contents node_v13.6.0 File :/node File :/package.manifest Dir :/sysroot Dir :/sysroot/lib Dir :/sysroot/lib/x86_64-linux-gnu File :/sysroot/lib/x86_64-linux-gnu/libc.so.6 File :/sysroot/lib/x86_64-linux-gnu/libdl.so.2 File :/sysroot/lib/x86_64-linux-gnu/libgcc_s.so.1 File :/sysroot/lib/x86_64-linux-gnu/libm.so.6 File :/sysroot/lib/x86_64-linux-gnu/libnss_dns.so.2 File :/sysroot/lib/x86_64-linux-gnu/libnss_files.so.2 File :/sysroot/lib/x86_64-linux-gnu/libpthread.so.0 Dir :/sysroot/lib64 File :/sysroot/lib64/ld-linux-x86-64.so.2 Dir :/sysroot/proc File :/sysroot/proc/meminfo Dir :/sysroot/usr Dir :/sysroot/usr/lib Dir :/sysroot/usr/lib/x86_64-linux-gnu File :/sysroot/usr/lib/x86_64-linux-gnu/libstdc++.so.6

Without getting into the semantics of building your own package which you can find here the easiest way to do this with your own app is in your project directory you can create a directory:



mkdir -p usr/lib

then include it in your config.json that you pass to ops when it builds your image.



{ "Dirs" :[ "usr" ] }

ops run -c config.json myprogram

After that you can always grab the image that was created in ~/.ops/images.

Hope this clears up any confusion.