Re: Error when running "pio build" - "deduplicate: different file contents found"

[Donald Szeto]

Hi Jimmy,

This situation arises when the sbt-assembly plugin fails to deduplicate a file in the final assembled JAR. Please refer to https://github.com/sbt/sbt-assembly#merge-strategy for details in resolving this issue.

Regards,

Donald

On Monday, February 16, 2015 at 6:31:00 PM UTC-8, Jimmy Wu wrote:

I am trying to build my custom engine template after adding some of my own dependencies. My build.sbt file is here: https://github.com/jimmyywu/template-scala-parallel-vanilla-test/blob/master/build.sbt

Running pio build generates the following output. Any ideas?

2015-02-16 21:05:21,392 INFO  tools.Console$ - Using existing engine manifest JSON at /home/owner/template-scala-parallel-vanilla-test/manifest.json

2015-02-16 21:05:21,397 INFO  tools.Console$ - Using command '/home/owner/PredictionIO/sbt/sbt' at the current working directory to build.

2015-02-16 21:05:21,400 INFO  tools.Console$ - If the path above is incorrect, this process will fail.

2015-02-16 21:05:21,401 INFO  tools.Console$ - Uber JAR disabled. Making sure lib/pio-assembly-0.8.6.jar is absent.

2015-02-16 21:05:21,401 INFO  tools.Console$ - Going to run: /home/owner/PredictionIO/sbt/sbt  package assemblyPackageDependency

2015-02-16 21:06:00,180 ERROR tools.Console$ - [error] (*:assemblyPackageDependency) deduplicate: different file contents found in the following:

2015-02-16 21:06:00,181 ERROR tools.Console$ - [error] /home/owner/.ivy2/cache/org.springframework/spring-context/jars/spring-context-3.2.5.RELEASE.jar:META-INF/spring.tooling

2015-02-16 21:06:00,181 ERROR tools.Console$ - [error] /home/owner/.ivy2/cache/org.springframework/spring-aop/jars/spring-aop-3.2.5.RELEASE.jar:META-INF/spring.tooling

2015-02-16 21:06:00,182 ERROR tools.Console$ - [error] /home/owner/.ivy2/cache/org.springframework/spring-beans/jars/spring-beans-3.2.5.RELEASE.jar:META-INF/spring.tooling

2015-02-16 21:06:00,182 ERROR tools.Console$ - [error] Total time: 4 s, completed Feb 16, 2015 9:06:00 PM

2015-02-16 21:06:00,207 ERROR tools.Console$ - Return code of previous step is 1. Aborting.

[Benson Qiu]

Hi Donald,

We managed to resolve the deduplicate in 2 steps. Just want to check with you if it looks reasonable:

1. Upgrade sbt-assembly to version 1.2.0 in project/assembly.sbt (so we can use assemblyMergeStrategy in build.sbt).

2. Remove line 1-3 in build.sbt (version 1.2.0 don't need those), and add the following at the end of build.sbt:

"""

assemblyMergeStrategy in assembly := {

    case x => MergeStrategy.discard

}

[Donald Szeto]

Hi Benson,

Sorry for the late reply. Please make sure discarding all duplicates do not affect the functionality of your engine. In that case, you might need to treat each duplicate differently.

Regards,

Donald