brotli: my c2go experience

[Andy Balholm]

Over the last few months, I’ve been working (on and off) at translating the Brotli compression library into Go. (The result is at github.com/andybalholm/brotli.) I’d like to share what I’ve learned.

I tried various tools: rsc/c2go, elliotchance/c2go, Konstantin8105/c4go, and a tool that I developed myself (leaven). I kept coming back to rsc/c2go because it produces the cleanest, most readable output. But I was frustrated by its limitations; there are so many C constructs that it can’t handle.

Finally I realized that the only way a C-to-Go transpiler can produce readable output is to limit itself to the subset of C that maps to Go fairly cleanly. And the way to deal with that is to progressively refactor the C project into that subset of C.

In practice, it turned out to be a two-sided process. I worked on c2go to make it handle more of the constructs used in brotli, and I refactored the brotli codebase to get rid of things c2go couldn’t handle, until the two converged.

My fork of rsc/c2go is at github.com/andybalholm/c2go; the README contains some more of my thoughts on the transpilation process, and a general summary of how to go about translating a C project.

Andy

[Andy Balholm]

Now for some benchmarks. I wrote a simple benchmark program that compares my pure Go brotli package with cbrotli (the cgo wrapper in the standard brotli repo) and with compress/gzip. Here are the results for compressing Newton’s Opticks (from the testdata directory of the Go source tree):

brotli-0 37.9% 10.611264ms 51.0 MB/s

brotli-1 33.9% 12.440498ms 43.5 MB/s

brotli-2 32.2% 32.232898ms 16.8 MB/s

brotli-3 31.8% 32.788896ms 16.5 MB/s

brotli-4 31.0% 41.686392ms 13.0 MB/s

brotli-5 29.0% 52.108321ms 10.4 MB/s

brotli-6 28.4% 61.75893ms 8.8 MB/s

brotli-7 27.9% 77.842721ms 6.9 MB/s

brotli-8 27.7% 101.626643ms 5.3 MB/s

brotli-9 27.5% 118.482597ms 4.6 MB/s

brotli-10 25.6% 897.835046ms 0.6 MB/s

brotli-11 25.0% 2.13944223s 0.3 MB/s

cbrotli-0 37.9% 9.664506ms 56.0 MB/s

cbrotli-1 33.9% 11.348875ms 47.7 MB/s

cbrotli-2 32.2% 29.9876ms 18.0 MB/s

cbrotli-3 31.8% 36.219238ms 14.9 MB/s

cbrotli-4 31.0% 44.791383ms 12.1 MB/s

cbrotli-5 29.0% 57.329373ms 9.4 MB/s

cbrotli-6 28.4% 58.928917ms 9.2 MB/s

cbrotli-7 27.9% 76.804601ms 7.0 MB/s

cbrotli-8 27.7% 100.928546ms 5.4 MB/s

cbrotli-9 27.5% 122.621637ms 4.4 MB/s

cbrotli-10 25.6% 916.783042ms 0.6 MB/s

cbrotli-11 25.0% 2.091093825s 0.3 MB/s

gzip-1 38.5% 9.548712ms 56.6 MB/s

gzip-2 34.9% 11.672584ms 46.3 MB/s

gzip-3 34.1% 14.940402ms 36.2 MB/s

gzip-4 31.9% 15.40793ms 35.1 MB/s

gzip-5 30.9% 25.36402ms 21.3 MB/s

gzip-6 30.7% 31.563463ms 17.1 MB/s

gzip-7 30.6% 37.404353ms 14.5 MB/s

gzip-8 30.6% 45.647576ms 11.8 MB/s

gzip-9 30.6% 47.600027ms 11.4 MB/s