This part of library is (almost) unmaintained. There are no any documents and original author (of opensciencemap) does not participate. What I know so far:
Each theme instruction primitive drawing is controlled by its own bucket, they can be found in org.oscim.render.bucket package. Line drawing is controlled by one of two buckets: LineBucket and LineTexBucket. First one is used for simple solid color or stippled lines, second one - for textured or array-dashed lines (because array-dash is converted to bitmap texture). Line outlines are ugly implemented by simply drawing lines with outline color underneath lines with main color. This means that outlined lines can not be semi-transparent for example.
Each line bucket is supplemented by GLSL shader file which can be found in assets/shaders. Files contain both vertex shader and fragment shader separated by '$' symbol. These files contain outdated and unused parts of code with little comments. Simple line shader files also have two separate versions for orthogonal and slanted map views. The later has '_proj' suffix.