Getdown 1.6

[Michael Bayne]

I have released Getdown 1.6, it is making its way to Maven Central as I type.

This release introduces a new digest algorithm which fixes a few things people have requested. Specifically: things inside the META-INF directory of a jar file are included in its hash, the hash algorithm has been changed from MD5 to SHA1, and the algorithm used to sign the digest.txt file (if you're signing things) is changed from SHA1 to SHA256.

Because I cannot just change the digest algorithm without breaking old clients, this new version creates two digest files: digest.txt which uses the old algorithm and digest2.txt which uses the new algorithm. This ensures that when you ship an app built with this new Getdown, old versions of Getdown can still understand and upgrade your app. They will use the digest.txt file which is exactly as it has always been.

Assuming you have a process in place for Getdown to update itself inside your app's installation, then once the user has installed your update which was built with the new Getdown, your app will start using the new Getdown which will use the digest2.txt file going forward.

Since these files are small, we'll probably just generate both digest.txt files forever, because that means that someone who hasn't updated their app in a really long time will still be able to make this transition at any point in the future, but if there is demand, I will add a way to turn off the old digest.txt generation for people who know they're only ever going to have clients running the new Getdown.

The Ant task now generates both digests automatically, but the Maven plugin needs to be updated to call the new function that generates all the digests instead of just one.

This release also includes another pull request which ensures that non-versioned installations download update files to a separate location and only install them after validating that they've been completely and successfully downloaded. So they won't cause problems if the download fails halfway. This was never a problem for versioned installations, but a lot of people seem to be using non-versioned installations in production (which was never the intent, but hey apparently it's useful).

Another pull request is included which introduces a use_code_cache configuration parameter. If you set it to true, Getdown will copy all of the code files into a temporary directory before launching the application and set the classpath to reference those temporary files. This allows the original application to be updated even if old instances of the application are still running.

There are a lot of other small tweaks and fixes, check the commit history for all the details:

https://github.com/threerings/getdown/commits/master

-- m...@samskivert.com

[Dominique]

Hi!

I think Getdown 1.6 might not work correctly. I have used Getdown for the first time and started with Getdown 1.6 with a very simple setup. I generated digest.txt and digest2.txt using the Digester class:

% java -classpath getdown-1.6.jar com.threerings.getdown.tools.Digester myapp

It generated a digest2.txt file with checksums of all files. I uploaded everything to a webserver. However when I then tried to download everything again using getdown, it tried to download the first file again and again until it stopped after five attempts. The log file revealed that the computed checksum was different from the checksum in the digest2.txt file.

After trying this several times I gave up and tried the same with getdown-1.5.jar and without the digest2.txt file. With getdown-1.5.jar everything worked fine.

So it seems as if the function that computes the new checksums is either broken in the Digester class or in the class that verifies the download.

Can you please check that? Thank you very much!

Kind regards

Dominique

[kid_wonder]

I am seeing this same issue when trying to use 1.6

My digest and digest2 generated exactly the same thing (note that the digest2.txt has an entry for "digest.txt" not "digest2.txt" which seems odd)

However on the client side I get the following error:

java.io.IOException: m.invalid_digest_file|~|~a4c1d88b0a973f3a9e61a6c4863d25a8d11686e25c46314bcf36c1a839b74e38

at com.threerings.getdown.data.Digest.<init>(Unknown Source)

at com.threerings.getdown.data.Digest.<init>(Unknown Source)

at com.threerings.getdown.data.Application.verifyMetadata(Unknown Source)

at com.threerings.getdown.launcher.Getdown.getdown(Unknown Source)

at com.threerings.getdown.launcher.Getdown.run(Unknown Source)

When I rollback to getdown 1.5 it works as expected - even with the 1.6 generated digest.txt files

scott

[Michael Bayne]

On Tue, Dec 13, 2016 at 11:27 AM, kid_wonder <kscot...@gmail.com> wrote:

However on the client side I get the following error:

java.io.IOException: m.invalid_digest_file|~|~a4c1d88b0a973f3a9e61a6c4863d25a8d11686e25c46314bcf36c1a839b74e38

at com.threerings.getdown.data.Digest.<init>(Unknown Source)

at com.threerings.getdown.data.Digest.<init>(Unknown Source)

at com.threerings.getdown.data.Application.verifyMetadata(Unknown Source)

at com.threerings.getdown.launcher.Getdown.getdown(Unknown Source)

at com.threerings.getdown.launcher.Getdown.run(Unknown Source)

When I rollback to getdown 1.5 it works as expected - even with the 1.6 generated digest.txt files

Weird, I tested this all with the test app (https://github.com/threerings/getdown-testapp) and it all seemed to work fine.

I'll look into it when I get a chance, but unfortunately I'm kind of swamped at the moment, so it may be a few days.

-- m...@samskivert.com

[Michael Bayne]

On Tue, Dec 13, 2016 at 11:45 AM, Michael Bayne <m...@samskivert.com> wrote:

Weird, I tested this all with the test app (https://github.com/threerings/getdown-testapp) and it all seemed to work fine.

I'll look into it when I get a chance, but unfortunately I'm kind of swamped at the moment, so it may be a few days.

So I just tried out the testapp again and everything looks fine. I'm not sure what's going wrong for you guys.

When I generate the digests manually using "java -classpath getdown-1.6.jar com.threerings.getdown.tools.Digester testapp" or when I generate them using the Ant task, I get sensible looking digests:

digest.txt contains:

getdown.txt = 46a543a025a4d00cf54bdb130f834013

testapp-jar-with-dependencies.jar = 492bd0c3bc2bd991237a795d9fe2fc1d

getdown-new.jar = 7a4173a8c79d55b934fc81d2099300fd

digest.txt = 8e10ff2cb4e6f89374560da7041bca66

and digest2.txt contains:

getdown.txt = 034fbe68fb6e791bd1c2a6b28c19e2d6e07794593e61547d1a00b2087a28f339

testapp-jar-with-dependencies.jar = acc0ca91141b9fdcd99b97d8b086ee9ac50c6ad7e079b944e15dea9e80fe5bc0

getdown-new.jar = ceffcee81cd8581d5146958ff8231c4e64666068e8c5fcfa4912dcc9b58554d1

digest2.txt = 4183881f257c8b569528c892df5e8186631d742fcc50c2288cd4c5c9ecc7256d

If I run getdown as-is on the fully populated testapp directory, it runs just fine. If I delete everything from that directory and just create a getdown.txt that contains the single line:

appbase = http://samskivert.com/code/testapp

and then run Getdown, it downloads everything correctly and launches the app. Launched like so:

% java -jar ~/.m2/repository/com/threerings/getdown/1.6/getdown-1.6.jar testapp

the launcher.log reports:

2016/12/13 21:52:44:405 INFO n.a: ------------------ VM Info ------------------

2016/12/13 21:52:44:407 INFO n.a: -- OS Name: Mac OS X

2016/12/13 21:52:44:407 INFO n.a: -- OS Arch: x86_64

2016/12/13 21:52:44:407 INFO n.a: -- OS Vers: 10.12.1

2016/12/13 21:52:44:407 INFO n.a: -- Java Vers: 1.8.0_112

2016/12/13 21:52:44:407 INFO n.a: -- Java Home: /Library/Java/JavaVirtualMachines/jdk1.8.0_112.jdk/Contents/Home/jre

2016/12/13 21:52:44:408 INFO n.a: -- User Name: mdb

2016/12/13 21:52:44:408 INFO n.a: -- User Home: /Users/mdb

2016/12/13 21:52:44:408 INFO n.a: -- Cur dir: /Users/mdb/projects/getdown/testapp/target

2016/12/13 21:52:44:408 INFO n.a: ---------------------------------------------

2016/12/13 21:52:44:675 INFO n.a: Checking whether we need to use a proxy...

2016/12/13 21:52:44:839 INFO n.a: No proxy appears to be needed.

2016/12/13 21:52:44:840 INFO n.a: ---------------- Proxy Info -----------------

2016/12/13 21:52:44:840 INFO n.a: -- Proxy Host: null

2016/12/13 21:52:44:840 INFO n.a: -- Proxy Port: null

2016/12/13 21:52:44:840 INFO n.a: ---------------------------------------------

2016/12/13 21:52:44:841 WARNING n.a: Failed to initialize: java.io.IOException: m.missing_class

2016/12/13 21:52:44:841 INFO n.a: Attempting to refetch 'getdown.txt' from 'http://samskivert.com/code/testapp/getdown.txt'.

2016/12/13 21:52:44:923 INFO n.a: Able to lock for updates: true

2016/12/13 21:52:44:924 INFO n.a: Verifying application: http://samskivert.com/code/testapp/

2016/12/13 21:52:44:924 INFO n.a: Version: -1

2016/12/13 21:52:44:924 INFO n.a: Class: com.threerings.testapp.TestApp

2016/12/13 21:52:44:924 INFO n.a: Failed to load digest: testapp/digest2.txt (No such file or directory). Attempting recovery...

2016/12/13 21:52:44:924 INFO n.a: Attempting to refetch 'digest.txt' from 'http://samskivert.com/code/testapp/digest.txt'.

2016/12/13 21:52:44:980 INFO n.a: No signers, not verifying file [path=digest.txt]

2016/12/13 21:52:44:980 INFO n.a: Attempting to refetch 'digest2.txt' from 'http://samskivert.com/code/testapp/digest2.txt'.

2016/12/13 21:52:45:035 INFO n.a: No signers, not verifying file [path=digest2.txt]

2016/12/13 21:52:45:043 INFO n.a: Unversioned digest changed. Revalidating...

2016/12/13 21:52:45:045 INFO n.a: Resource failed digest check [rsrc=testapp-jar-with-dependencies.jar, error=java.io.FileNotFoundException: testapp/testapp-jar-with-dependencies.jar (No such file or directory)]

2016/12/13 21:52:45:045 INFO n.a: Resource failed digest check [rsrc=getdown-new.jar, error=java.io.FileNotFoundException: testapp/getdown-new.jar (No such file or directory)]

2016/12/13 21:52:45:045 INFO n.a: 2 of 2 rsrcs require update (0 assumed valid).

2016/12/13 21:52:45:174 INFO n.a: Downloading 362549 bytes...

2016/12/13 21:52:45:231 INFO n.a: Downloading resource [url=http://samskivert.com/code/testapp/testapp-jar-with-dependencies.jar, size=184090]

2016/12/13 21:52:45:473 INFO n.a: Downloading resource [url=http://samskivert.com/code/testapp/getdown-new.jar, size=178459]

2016/12/13 21:52:45:535 INFO n.a: Verifying application: http://samskivert.com/code/testapp/

2016/12/13 21:52:45:536 INFO n.a: Version: -1

2016/12/13 21:52:45:536 INFO n.a: Class: com.threerings.testapp.TestApp

2016/12/13 21:52:45:536 INFO n.a: Attempting to refetch 'digest.txt' from 'http://samskivert.com/code/testapp/digest.txt'.

2016/12/13 21:52:45:595 INFO n.a: No signers, not verifying file [path=digest.txt]

2016/12/13 21:52:45:596 INFO n.a: Attempting to refetch 'digest2.txt' from 'http://samskivert.com/code/testapp/digest2.txt'.

2016/12/13 21:52:45:652 INFO n.a: No signers, not verifying file [path=digest2.txt]

2016/12/13 21:52:45:689 INFO n.a: Resources verified.

2016/12/13 21:52:45:689 INFO n.a: Installing downloaded resources:

2016/12/13 21:52:45:690 INFO n.a: - testapp/testapp-jar-with-dependencies.jar_new

2016/12/13 21:52:45:690 INFO n.a: - testapp/getdown-new.jar_new

2016/12/13 21:52:45:690 INFO n.a: Install completed.

2016/12/13 21:52:45:695 INFO n.a: Didn't find any custom environment variables, not setting any.

2016/12/13 21:52:45:696 INFO n.a: Running /Library/Java/JavaVirtualMachines/jdk1.8.0_112.jdk/Contents/Home/jre/bin/java

  -classpath

  /Users/mdb/projects/getdown/testapp/target/testapp/.cache/ac/acc0ca91141b9fdcd99b97d8b086ee9ac50c6ad7e079b944e15dea9e80fe5bc0.jar

  -Xdock:icon=/Users/mdb/projects/getdown/testapp/target/testapp/../desktop.icns

  -Xdock:name=Getdown Test App

  -Dcom.threerings.getdown=true

  -Dusername=\mdb

  com.threerings.testapp.TestApp

  /Users/mdb/projects/getdown/testapp/target/testapp

2016/12/13 21:52:46:138 INFO n.a: Updating Getdown with /Users/mdb/projects/getdown/testapp/target/testapp/getdown-new.jar...2016/12/13 21:52:46:588 WARNING n.a: Failed to load image [path=testapp/background.png, error=javax.imageio.IIOException: Can't read input file!]

You can try creating a testapp directory, putting the above line in a getdown.txt file and running getdown.jar with that directory as I did and see if the testapp works for you.

Maybe you can compare your launcher.log to mine and get some idea of what's going wrong?

-- m...@samskivert.com