I just downloaded the jython installer. it is a runnable jar file, so make sure java is installed. then run the jar file, and click 'all everything including sources' when installation type appears. once its done, include the /bin folder into your Path because that's where jython.exe is. after that i installed RF. i don't know if the order matters, but at least that works for me.
then you can create your java library. it can be a normal class with some methods. build it into a jar file and include that into your CLASSPATH. start RF by something like this:
set CLASSPATH="..\..\lib\SampleLibrary.jar;%CLASSPATH%"
jybot --outputdir ..\..\..\output ..\..\..\implementation\testsuites\SampleLibraryTestsuite.html
see this tutorial