Problem with podman v2

[Fin Steenbjerg]

I have experienced a problem when doing native builds. The problem is caused by an update to podman (going from version 1.9.3 to 2.0.1) at my laptop with OS fedora 32. When using podman 2.0.1 the native build fails (see output below) while it the same thing works at podman 1.9.3. My application uses version 1.5.2 of Quarkus. The native build is executed by the command './mvnw clean package -Pnative' and my application.properties contains these setting for the build stuff:

# Build
quarkus.native.builder-image=quay.io/quarkus/ubi-quarkus-native-image:20.1.0-java11
quarkus.native.container-runtime=podman
quarkus.package.uber-jar=true
quarkus.native.native-image-xmx=6g

The ouput when using podman version 2.0.1 is:

[INFO] [io.quarkus.deployment.pkg.steps.NativeImageBuildStep] Running Quarkus native-image plugin on GraalVM Version 20.1.0 (Java Version 11.0.7)
[INFO] [io.quarkus.deployment.pkg.steps.NativeImageBuildStep] podman run -v /opt/jb/udvikler/ws/jb-ocp.git/cluster-service/target/cluster-service-1.0.0-SNAPSHOT-native-image-source-jar:/project:z --env LANG=C --userns=keep-id --rm quay.io/quarkus/ubi-quarkus-native-image:20.1.0-java11 -J-Dsun.nio.ch.maxUpdateArraySize=100 -J-Djava.util.logging.manager=org.jboss.logmanager.LogManager -J-DCoordinatorEnvironmentBean.transactionStatusManagerEnable=false -J-Dvertx.logger-delegate-factory-class-name=io.quarkus.vertx.core.runtime.VertxLogDelegateFactory -J-Dvertx.disableDnsResolver=true -J-Dio.netty.leakDetection.level=DISABLED -J-Dio.netty.allocator.maxOrder=1 -J-Duser.language=en -J-Dfile.encoding=UTF-8 --initialize-at-build-time= -H:InitialCollectionPolicy=com.oracle.svm.core.genscavenge.CollectionPolicy$BySpaceAndTime -H:+JNI -jar cluster-service-1.0.0-SNAPSHOT-runner.jar -H:FallbackThreshold=0 -H:+ReportExceptionStackTraces -J-Xmx6g -H:-AddAllCharsets -H:-IncludeAllTimeZones -H:EnableURLProtocols=http,https --enable-all-security-services --no-server -H:-UseServiceLoaderFeature -H:+StackTrace cluster-service-1.0.0-SNAPSHOT-runner
-H:IncludeAllTimeZones and -H:IncludeTimeZones are now deprecated. Native-image includes all timezonesby default.
[cluster-service-1.0.0-SNAPSHOT-runner:20]    classlist:   8,646.31 ms,  1.19 GB
[cluster-service-1.0.0-SNAPSHOT-runner:20]        (cap):     548.25 ms,  1.19 GB
[cluster-service-1.0.0-SNAPSHOT-runner:20]        setup:   2,039.93 ms,  1.19 GB
08:38:42,885 INFO  [org.hib.val.int.uti.Version] HV000001: Hibernate Validator 6.1.5.Final
08:38:42,999 INFO  [org.hib.Version] HHH000412: Hibernate ORM core version 5.4.16.Final
08:38:43,003 INFO  [org.hib.ann.com.Version] HCANN000001: Hibernate Commons Annotations {5.1.0.Final}
08:38:43,025 INFO  [org.hib.dia.Dialect] HHH000400: Using dialect: io.quarkus.hibernate.orm.runtime.dialect.QuarkusPostgreSQL10Dialect
08:39:01,227 INFO  [org.jbo.threads] JBoss Threads version 3.1.1.Final
WARNING GR-10238: VarHandle for static field is currently not fully supported. Static field private static volatile java.lang.System$Logger jdk.internal.event.EventHelper.securityLogger is not properly marked for Unsafe access!
[cluster-service-1.0.0-SNAPSHOT-runner:20]     (clinit):   1,486.40 ms,  2.62 GB
[cluster-service-1.0.0-SNAPSHOT-runner:20]   (typeflow):  37,469.62 ms,  2.62 GB
[cluster-service-1.0.0-SNAPSHOT-runner:20]    (objects):  40,304.46 ms,  2.62 GB
[cluster-service-1.0.0-SNAPSHOT-runner:20]   (features):   1,399.72 ms,  2.62 GB
[cluster-service-1.0.0-SNAPSHOT-runner:20]     analysis:  86,173.14 ms,  2.62 GB
[cluster-service-1.0.0-SNAPSHOT-runner:20]     universe:   2,364.33 ms,  2.71 GB
[cluster-service-1.0.0-SNAPSHOT-runner:20]      (parse):   6,484.22 ms,  2.59 GB
[cluster-service-1.0.0-SNAPSHOT-runner:20]     (inline):  10,600.73 ms,  3.54 GB
[cluster-service-1.0.0-SNAPSHOT-runner:20]    (compile):  36,356.05 ms,  3.66 GB
[cluster-service-1.0.0-SNAPSHOT-runner:20]      compile:  56,831.78 ms,  3.67 GB
[cluster-service-1.0.0-SNAPSHOT-runner:20]        image:   5,832.32 ms,  3.68 GB
[cluster-service-1.0.0-SNAPSHOT-runner:20]        write:     486.62 ms,  3.68 GB
Fatal error:java.lang.RuntimeException: java.lang.RuntimeException: There was an error linking the native image: Linker command exited with 1


Linker command executed:
/usr/bin/gcc -z noexecstack -Wl,--gc-sections -Wl,--dynamic-list -Wl,/tmp/SVM-8281210852036372715/exported_symbols.list -Wl,--exclude-libs,ALL -Wl,-x -o /project/cluster-service-1.0.0-SNAPSHOT-runner /tmp/SVM-8281210852036372715/cluster-service-1.0.0-SNAPSHOT-runner.o /opt/graalvm/lib/svm/clibraries/linux-amd64/liblibchelper.a /opt/graalvm/lib/libnet.a /opt/graalvm/lib/svm/clibraries/linux-amd64/libstrictmath.a /opt/graalvm/lib/svm/clibraries/linux-amd64/libffi.a /opt/graalvm/lib/libextnet.a /opt/graalvm/lib/libnio.a /opt/graalvm/lib/libjava.a /opt/graalvm/lib/libsunec.a /opt/graalvm/lib/svm/clibraries/linux-amd64/libjvm.a /opt/graalvm/lib/libzip.a -v -L/tmp/SVM-8281210852036372715 -L/opt/graalvm/lib -L/opt/graalvm/lib/svm/clibraries/linux-amd64 -lstdc++ -lpthread -ldl -lz -lrt


Linker command ouput:
Using built-in specs.
COLLECT_GCC=/usr/bin/gcc
COLLECT_LTO_WRAPPER=/usr/libexec/gcc/x86_64-redhat-linux/8/lto-wrapper
OFFLOAD_TARGET_NAMES=nvptx-none
OFFLOAD_TARGET_DEFAULT=1
Target: x86_64-redhat-linux
Configured with: ../configure --enable-bootstrap --enable-languages=c,c++,fortran,lto --prefix=/usr --mandir=/usr/share/man --infodir=/usr/share/info --with-bugurl=http://bugzilla.redhat.com/bugzilla --enable-shared --enable-threads=posix --enable-checking=release --enable-multilib --with-system-zlib --enable-__cxa_atexit --disable-libunwind-exceptions --enable-gnu-unique-object --enable-linker-build-id --with-gcc-major-version-only --with-linker-hash-style=gnu --enable-plugin --enable-initfini-array --with-isl --disable-libmpx --enable-offload-targets=nvptx-none --without-cuda-driver --enable-gnu-indirect-function --enable-cet --with-tune=generic --with-arch_32=x86-64 --build=x86_64-redhat-linux
Thread model: posix
gcc version 8.3.1 20191121 (Red Hat 8.3.1-5) (GCC)
COMPILER_PATH=/usr/libexec/gcc/x86_64-redhat-linux/8/:/usr/libexec/gcc/x86_64-redhat-linux/8/:/usr/libexec/gcc/x86_64-redhat-linux/:/usr/lib/gcc/x86_64-redhat-linux/8/:/usr/lib/gcc/x86_64-redhat-linux/
LIBRARY_PATH=/usr/lib/gcc/x86_64-redhat-linux/8/:/usr/lib/gcc/x86_64-redhat-linux/8/../../../../lib64/:/lib/../lib64/:/usr/lib/../lib64/:/usr/lib/gcc/x86_64-redhat-linux/8/../../../:/lib/:/usr/lib/
COLLECT_GCC_OPTIONS='-z' 'noexecstack' '-o' '/project/cluster-service-1.0.0-SNAPSHOT-runner' '-v' '-L/tmp/SVM-8281210852036372715' '-L/opt/graalvm/lib' '-L/opt/graalvm/lib/svm/clibraries/linux-amd64' '-mtune=generic' '-march=x86-64'
 /usr/libexec/gcc/x86_64-redhat-linux/8/collect2 -plugin /usr/libexec/gcc/x86_64-redhat-linux/8/liblto_plugin.so -plugin-opt=/usr/libexec/gcc/x86_64-redhat-linux/8/lto-wrapper -plugin-opt=-fresolution=/tmp/ccXICVZ7.res -plugin-opt=-pass-through=-lgcc -plugin-opt=-pass-through=-lgcc_s -plugin-opt=-pass-through=-lc -plugin-opt=-pass-through=-lgcc -plugin-opt=-pass-through=-lgcc_s --build-id --no-add-needed --eh-frame-hdr --hash-style=gnu -m elf_x86_64 -dynamic-linker /lib64/ld-linux-x86-64.so.2 -o /project/cluster-service-1.0.0-SNAPSHOT-runner -z noexecstack /usr/lib/gcc/x86_64-redhat-linux/8/../../../../lib64/crt1.o /usr/lib/gcc/x86_64-redhat-linux/8/../../../../lib64/crti.o /usr/lib/gcc/x86_64-redhat-linux/8/crtbegin.o -L/tmp/SVM-8281210852036372715 -L/opt/graalvm/lib -L/opt/graalvm/lib/svm/clibraries/linux-amd64 -L/usr/lib/gcc/x86_64-redhat-linux/8 -L/usr/lib/gcc/x86_64-redhat-linux/8/../../../../lib64 -L/lib/../lib64 -L/usr/lib/../lib64 -L/usr/lib/gcc/x86_64-redhat-linux/8/../../.. --gc-sections --dynamic-list /tmp/SVM-8281210852036372715/exported_symbols.list --exclude-libs ALL -x /tmp/SVM-8281210852036372715/cluster-service-1.0.0-SNAPSHOT-runner.o /opt/graalvm/lib/svm/clibraries/linux-amd64/liblibchelper.a /opt/graalvm/lib/libnet.a /opt/graalvm/lib/svm/clibraries/linux-amd64/libstrictmath.a /opt/graalvm/lib/svm/clibraries/linux-amd64/libffi.a /opt/graalvm/lib/libextnet.a /opt/graalvm/lib/libnio.a /opt/graalvm/lib/libjava.a /opt/graalvm/lib/libsunec.a /opt/graalvm/lib/svm/clibraries/linux-amd64/libjvm.a /opt/graalvm/lib/libzip.a -lstdc++ -lpthread -ldl -lz -lrt -lgcc --as-needed -lgcc_s --no-as-needed -lc -lgcc --as-needed -lgcc_s --no-as-needed /usr/lib/gcc/x86_64-redhat-linux/8/crtend.o /usr/lib/gcc/x86_64-redhat-linux/8/../../../../lib64/crtn.o
/usr/bin/ld: cannot open output file /project/cluster-service-1.0.0-SNAPSHOT-runner: Permission denied
collect2: error: ld returned 1 exit status


 at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
 at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
 at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
 at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:490)
 at java.base/java.util.concurrent.ForkJoinTask.getThrowableException(ForkJoinTask.java:600)
 at java.base/java.util.concurrent.ForkJoinTask.get(ForkJoinTask.java:1006)
 at com.oracle.svm.hosted.NativeImageGenerator.run(NativeImageGenerator.java:463)
 at com.oracle.svm.hosted.NativeImageGeneratorRunner.buildImage(NativeImageGeneratorRunner.java:359)
 at com.oracle.svm.hosted.NativeImageGeneratorRunner.build(NativeImageGeneratorRunner.java:518)
 at com.oracle.svm.hosted.NativeImageGeneratorRunner.main(NativeImageGeneratorRunner.java:117)
 at com.oracle.svm.hosted.NativeImageGeneratorRunner$JDK9Plus.main(NativeImageGeneratorRunner.java:546)
Caused by: java.lang.RuntimeException: There was an error linking the native image: Linker command exited with 1


Linker command executed:
/usr/bin/gcc -z noexecstack -Wl,--gc-sections -Wl,--dynamic-list -Wl,/tmp/SVM-8281210852036372715/exported_symbols.list -Wl,--exclude-libs,ALL -Wl,-x -o /project/cluster-service-1.0.0-SNAPSHOT-runner /tmp/SVM-8281210852036372715/cluster-service-1.0.0-SNAPSHOT-runner.o /opt/graalvm/lib/svm/clibraries/linux-amd64/liblibchelper.a /opt/graalvm/lib/libnet.a /opt/graalvm/lib/svm/clibraries/linux-amd64/libstrictmath.a /opt/graalvm/lib/svm/clibraries/linux-amd64/libffi.a /opt/graalvm/lib/libextnet.a /opt/graalvm/lib/libnio.a /opt/graalvm/lib/libjava.a /opt/graalvm/lib/libsunec.a /opt/graalvm/lib/svm/clibraries/linux-amd64/libjvm.a /opt/graalvm/lib/libzip.a -v -L/tmp/SVM-8281210852036372715 -L/opt/graalvm/lib -L/opt/graalvm/lib/svm/clibraries/linux-amd64 -lstdc++ -lpthread -ldl -lz -lrt

The ouput when using podman version 1.9.3 is:

[INFO] [io.quarkus.deployment.pkg.steps.NativeImageBuildStep] podman run -v /opt/jb/udvikler/ws/jb-ocp.git/cluster-service/target/cluster-service-1.0.0-SNAPSHOT-native-image-source-jar:/project:z --env LANG=C --userns=keep-id --rm quay.io/quarkus/ubi-quarkus-native-image:20.1.0-java11 -J-Dsun.nio.ch.maxUpdateArraySize=100 -J-DCoordinatorEnvironmentBean.transactionStatusManagerEnable=false -J-Djava.util.logging.manager=org.jboss.logmanager.LogManager -J-Dvertx.logger-delegate-factory-class-name=io.quarkus.vertx.core.runtime.VertxLogDelegateFactory -J-Dvertx.disableDnsResolver=true -J-Dio.netty.leakDetection.level=DISABLED -J-Dio.netty.allocator.maxOrder=1 -J-Duser.language=en -J-Dfile.encoding=UTF-8 --initialize-at-build-time= -H:InitialCollectionPolicy=com.oracle.svm.core.genscavenge.CollectionPolicy$BySpaceAndTime -H:+JNI -jar cluster-service-1.0.0-SNAPSHOT-runner.jar -H:FallbackThreshold=0 -H:+ReportExceptionStackTraces -J-Xmx6g -H:-AddAllCharsets -H:-IncludeAllTimeZones -H:EnableURLProtocols=http,https --enable-all-security-services --no-server -H:-UseServiceLoaderFeature -H:+StackTrace cluster-service-1.0.0-SNAPSHOT-runner
-H:IncludeAllTimeZones and -H:IncludeTimeZones are now deprecated. Native-image includes all timezonesby default.
[cluster-service-1.0.0-SNAPSHOT-runner:20]    classlist:   9,410.79 ms,  1.19 GB
[cluster-service-1.0.0-SNAPSHOT-runner:20]        (cap):   1,051.49 ms,  1.19 GB
[cluster-service-1.0.0-SNAPSHOT-runner:20]        setup:   2,578.64 ms,  1.19 GB
08:19:37,585 INFO  [org.hib.val.int.uti.Version] HV000001: Hibernate Validator 6.1.5.Final
08:19:37,690 INFO  [org.hib.Version] HHH000412: Hibernate ORM core version 5.4.16.Final
08:19:37,696 INFO  [org.hib.ann.com.Version] HCANN000001: Hibernate Commons Annotations {5.1.0.Final}
08:19:37,723 INFO  [org.hib.dia.Dialect] HHH000400: Using dialect: io.quarkus.hibernate.orm.runtime.dialect.QuarkusPostgreSQL10Dialect
08:19:56,649 INFO  [org.jbo.threads] JBoss Threads version 3.1.1.Final
WARNING GR-10238: VarHandle for static field is currently not fully supported. Static field private static volatile java.lang.System$Logger jdk.internal.event.EventHelper.securityLogger is not properly marked for Unsafe access!
[cluster-service-1.0.0-SNAPSHOT-runner:20]     (clinit):   1,556.39 ms,  3.48 GB
[cluster-service-1.0.0-SNAPSHOT-runner:20]   (typeflow):  42,248.22 ms,  3.48 GB
[cluster-service-1.0.0-SNAPSHOT-runner:20]    (objects):  41,200.38 ms,  3.48 GB
[cluster-service-1.0.0-SNAPSHOT-runner:20]   (features):   1,480.44 ms,  3.48 GB
[cluster-service-1.0.0-SNAPSHOT-runner:20]     analysis:  91,881.87 ms,  3.48 GB
[cluster-service-1.0.0-SNAPSHOT-runner:20]     universe:   2,492.42 ms,  3.48 GB
[cluster-service-1.0.0-SNAPSHOT-runner:20]      (parse):   5,941.69 ms,  3.33 GB
[cluster-service-1.0.0-SNAPSHOT-runner:20]     (inline):   9,347.51 ms,  3.64 GB
[cluster-service-1.0.0-SNAPSHOT-runner:20]    (compile):  41,843.05 ms,  4.21 GB
[cluster-service-1.0.0-SNAPSHOT-runner:20]      compile:  60,463.18 ms,  4.21 GB
[cluster-service-1.0.0-SNAPSHOT-runner:20]        image:   5,459.18 ms,  4.28 GB
[cluster-service-1.0.0-SNAPSHOT-runner:20]        write:     846.21 ms,  4.28 GB
[cluster-service-1.0.0-SNAPSHOT-runner:20]      [total]: 173,504.19 ms,  4.28 GB
[INFO] [io.quarkus.deployment.QuarkusAugmentor] Quarkus augmentation completed in 179908ms

My dnf.log states:

020-07-03T05:21:22Z DEBUG ---> Package podman.x86_64 2:1.9.3-1.fc32 will be upgraded
2020-07-03T05:21:22Z DEBUG ---> Package podman.x86_64 2:2.0.1-1.fc32 will be an upgrade

Has anyone experienced the same kind of problems?

[Foivos Zakkak]

Hello,

FYI this is now being tracked at https://bugzilla.redhat.com/show_bug.cgi?id=1854738

Regards,
Foivos

[Fin Steenbjerg]

Super. Thanks for the information.