Invalid signature file digest for Manifest main attributes

[Hossein Falaki]

Hi Spark developers,

When building version 0.7.2 using sbt, we end up with an invalid signature in the core jar file.

[spark-0.7.2]$ jarsigner -verify core/target/spark-core-assembly-0.7.2.jar 

jarsigner: java.lang.SecurityException: Invalid signature file digest for Manifest main attributes

The pre-built version, however, is unsigned. We did not face this problem with previous versions of Spark:

[spark-0.7.0]$ jarsigner -verify core/target/spark-core-assembly-0.7.0.jar 

jar is unsigned. (signatures missing or not parsable)

[spark-0.6.2]$ jarsigner -verify core/target/spark-core-assembly-0.6.2.jar 

jar is unsigned. (signatures missing or not parsable)

I suspect it has to do with the following messages when assembling the latest version.

[info] SHA-1: WrappedArray(-9, 25, -79, 97, 122, -57, 119, 43, -46, 77, 37, -12, -99, 84, -125, 82, -44, -122, -68, 56)

[info] Packaging /opt/spark-0.7.2/core/target/spark-core-assembly-0.7.2.jar ...

[info] SHA-1: WrappedArray(-26, -21, 68, -66, 56, -40, 62, -84, 79, 22, 1, -94, -100, -7, 4, -16, 53, -21, 94, 102)

[info] Packaging /opt/spark-0.7.2/streaming/target/spark-streaming-assembly-0.7.2.jar ...

[info] Done packaging.

[info] Done packaging.

[success] Total time: 32 s

Any idea how to prevent the signing process or to fix it?

Thanks,

--Hossein

[Mridul Muralidharan]

Hmm, I had fixed this in trunk as part of yarn merge.
Basically this is because some of the recent jars (netty I think)
include signature files - which sbt does not remove while creating the
assembled jar as part of assembly.
You can try maven - and that should work fine.

If you want to fix it, simply copy the line 222 from
https://github.com/mesos/spark/blob/master/project/SparkBuild.scala

'case m if m.toLowerCase.matches("meta-inf/.*\\.sf$") => MergeStrategy.discard'



Regards,
Mridul

> --
> You received this message because you are subscribed to the Google Groups
> "Spark Developers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to spark-develope...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>

[ahoyle]

Hi,
I also encounter this problem and I try to fix it by adding the line 'case m if m.toLowerCase.matches("meta-

inf/.*\\.sf$") => MergeStrategy.discard'

 

to sparkBuild.scala and then running 'sbt\sbt assembly'. However, the problem remains.
Could someone help please?
Thank you.

[Josh Rosen]

To manually delete the signature files from the assembly JAR, you can use 

jar xvf your-jar-assembly-0.1.jar
rm -rf your-jar-assembly-0.1.jar
rm -rf META-INF/*  # Substitute the correct command to delete the signature files; I forget their file extension
java cvf your-jar-assembly-0.1.jar *

I'm pretty sure that you can also do this with `zip -d YOUR-JAR.jar file/to/delete`.

The right fix is to exclude these signature files using sbt.  I'd unzip the assembly jars and manually check whether the signatures are excluded to determine why the sbt fix isn't working.

[Mridul Muralidharan]

I am not sure if this is a windows vs linux difference - just noticed
the sbt\sbt you used.
If yes, can you try
'case m if m.toLowerCase.matches("meta- inf.*\\.sf$") => MergeStrategy.discard'

(note, no / after meta-inf).


If this works for you, can you please submit a pull request to fix
this in mast too ? (or I can do it after you validate it).

Thanks !
Mridul

[Matei Zaharia]

Thanks for the fix, Mridul; I've included it in branch-0.7 and removed the forward slash in master as well.

Matei

[ahoyle]

solved.
Thank you.