plugin.shadow.path not respected

[bobob...@gmail.com]

I'm running the Docker container with:

docker run -d -p 8091:8888 -v <photos>:/opt/picapport/photos:ro -v <data>:/opt/picapport/data --name picapport fionnb/picapport:6.3.06

Note the ":ro" after the photo volume. This indicates that the volume is read-only. I don't want Picapport modifying my photos or doing anything to the directories they're in.

My picapport.properties is:

server.port=8888
robot.root.0.path=/opt/picapport/photos
foto.jpg.usecache=2
foto.jpg.cache.path=/opt/picapport/data/cache
plugin.shadow.path=/opt/picapport/data/shadow

And the shadow directory exists.

When I start Picapport, I repeatedly get error messages like this:

ERROR@ 00:53:19.267 Exception Error PhotoInFileSystem.PhotoInFileSystem: java.lang.NullPointerException: theSuffixes is null
java.io.FileNotFoundException: /opt/picapport/photos/Astro/16-02-21-12:30:11.thumb.ppm (Read-only file system)
at java.io.FileOutputStream.open0(Native Method)
at java.io.FileOutputStream.open(FileOutputStream.java:270)
at java.io.FileOutputStream.<init>(FileOutputStream.java:213)
at java.io.FileOutputStream.<init>(FileOutputStream.java:162)
at de.contecon.picapport.plugin.dcraw.DcRawProcessWrapper$StreamHandlerBinaryFile.run(Unknown Source)
ERROR@ 00:53:19.990 de.contecon.picapport.plugin.dcraw.DcRawPlugin: Unecpected Exception in createJpegFile: Invalid RAW file: /opt/picapport/photos/Astro/16-02-21-12:30:11.arw
ERROR@ 00:53:19.990 Exception Error PhotoCrawler.DirectoryConsumer.createSubstituteIfReqired: java.nio.file.FileSystemException: /opt/picapport/photos/Astro/16-02-21-12:30:11.arw.$.jpg: Read-only file system

It appears that the dcraw plugin is trying to write files to my photo directory even though I've told it to write them elsewhere.

I'm using the dcraw plugin from https://www.picapport.de/en/plugins.php

[PicApport]

Hi
it seems that the plugin.shadow.path is not recognized. If you start PicApport with -DTRACE=DEBUG you should check the logfiles in .picapport/logfiles for the following information:

• The location of the picapport.properties (is it the one that you have edited?)
  
• The actual values loaded from picapport.properties

[bobob...@gmail.com]

Thank for the response.

It's definitely using the correct picapport.properties:

find: 'data/picapport.ks': No such file or directory
PicApport headless set default option -Dmemory.useUnsafe=false
Logging to file: /opt/picapport/data/logfiles/picapport.000
Copyright (c) 1997-2017 Contecon Software GmbH. All rights reserved.
2017-08-08 05:27:37
Tracelevel: ERROR
VER @ 05:27:37.855 PicApport Version 6.3 from 25.04.2017 loaded.
VER @ 05:27:37.856 de.contecon.picapport.PicApport.Version=V6.3.6 25.04.2017
VER @ 05:27:37.856 java.runtime.version: 1.8.0_131-8u131-b11-0ubuntu1.16.04.2-b11
MSG @ 05:27:37.856 PicApport home=/opt/picapport/data
MSG @ 05:27:37.856 PicApport logging to /opt/picapport/data/logfiles
MSG @ 05:27:37.856 Loading config File: /opt/picapport/data/picapport.properties
MSG @ 05:27:38.363 Init: picApportHome=/opt/picapport/data
MSG @ 05:27:38.376 Init: server.port=8888
MSG @ 05:27:38.376 Init: server.ssl=false
MSG @ 05:27:38.376 Init: foto.jpg.usecache=2
MSG @ 05:27:38.376 Init: foto.jpg.cache.path=/opt/picapport/data/cache
MSG @ 05:27:38.376 Init: plugin.shadow.path=/opt/picapport/data/shadow
MSG @ 05:27:38.376 Init: gui.enabled=false
MSG @ 05:27:38.376 Init: gui.hideintray=true
MSG @ 05:27:38.377 Init: robot.root.0.id=id0
MSG @ 05:27:38.377 Init: robot.root.0.path=/opt/picapport/photos
MSG @ 05:27:38.377 Init: robot.root.0.isuploadroot=true
MSG @ 05:27:38.377 Init: server.demomode=false

I also confirmed that the directory exists and is writable. It's actually being used for the JPEG versions of PNG, GIF and BMP files. I suspect this means there's something uniquely broken about the dcraw plugin.

[E. Schreiner]

Hi,
the working Gif and PNG plungin was a good hint.
can you give PicApoort write access to /opt/picapport/photos(just for test purposes)
and see if it works? I'm afraid that the dcraw creates a temporary file.

[bobob...@gmail.com]

Yep, you were right. When I gave it write access it worked so your temporary file idea seems to be it. Is this something you can fix?

Also, the generated thumbnails are all broken, I suspect this may be because I'm using a relatively new camera that's not supported by the version of dcraw you're using? Or does it use the system version? Can I replace the binary somehow?

[bobob...@gmail.com]

Ah, looking through the code, I see the problem. The only ways to get thumbnails out of dcraw are through stdout or by letting it write a temp file. If you let it write a temp file, it puts the extension on the end, which lets you know whether it's a JPEG or PPM. If you write it to stdout, it's harder to figure out which kind of file it is.

I made some changes I think will fix the problem, there's a diff below.

To explain a little, this change:

- Writes thumbnails to a temporary file provided by Java (which uses the OS' facilities)
- Looks at the first two bytes of the file and
- If the first bytes are 0xff, 0xd8 (the SOI of a JPEG file), move the temporary file to the output location
- If the first two bytes are "P3" or "P6", assume it's PPM and convert it to JPEG.
- Else throw an error and remove it.

6a7
> import java.io.RandomAccessFile;
90c91
< dcRawThumbCmd=Arrays.asList(new String[]{"dcraw","-e","rawfilename"});
---
> dcRawThumbCmd=Arrays.asList(new String[]{"dcraw","-e","-c","rawfilename"});
138c139,140
<
---
> File tempFile = File.createTempFile("dcrawthumb", "tmp");
>
140c142
< dcRawThumbCmd.set(2, otherFormatFile.getAbsolutePath());
---
> dcRawThumbCmd.set(3, otherFormatFile.getAbsolutePath());
145,152c147,163
< new DcRawProcessWrapper(logger, this.pluginDirectory, dcRawThumbCmd);
< if(jpgThumbNail.exists()) {
< // Thumbnail was supported
< jpegFileToCreate.delete();
< jpgThumbNail.renameTo(jpegFileToCreate);
< } else {
< if(ppmThumbNail.exists()) {
< cjpegCmd.set(1, ppmThumbNail.getAbsolutePath());
---
> new DcRawProcessWrapper(logger, this.pluginDirectory, dcRawThumbCmd, tempFile);
> if (tempFile.exists()) {
> RandomAccessFile thumbFile = new RandomAccessFile(tempFile, "r");
> int[] magic = new int[2];
> magic[0] = thumbFile.readUnsignedByte();
> magic[1] = thumbFile.readUnsignedByte();
> thumbFile.close();
>
> thumbFile.read(magic, 0, 2);
> // Look for JPEG SOI
> if (magic[0] == 0xff && magic[1] == 0xd8) {
> // Thumbnail was supported
> jpegFileToCreate.delete();
> tempFile.renameTo(jpegFileToCreate);
> // Look for PPM
> } else if (magic[0] == 0x50 && (magic[1] == 0x33 || magic[1] == 0x36)) {
> cjpegCmd.set(1, tempFile.getAbsolutePath());
154,155c165,166
< ppmThumbNail.delete();
< } else {
---
> tempFile.delete();
> } else {
158d168
< }
159a170
> }

[E. Schreiner]

did your change finally fix the problem?

[bobob...@gmail.com]

Oh sorry, I forgot to say that I have no idea how to compile the source code, so I was unable to test.

[bobob...@gmail.com]

Okay, I figured out how to compile and run.

Code is here: https://paste.fedoraproject.org/paste/94U8JtxnNmxjdDs7SAW~Mg

Right at the top of createJpegFile I added a line:

logger.LogErrorMessage("Creating JPEG file: " +
jpegFileToCreate.getAbsolutePath());

When I look at the logs, I see this:

ERROR@ 17:44:56.262 de.contecon.picapport.plugin.dcraw.DcRawPlugin:
Creating JPEG file:
/opt/picapport/photos/Astro/16-02-21-12:24:11.arw.$.jpg

PicApport is calling createJpegFile with the wrong path. That's the
photo directory, not the shadow directory.

[E. Schreiner]

Hi,
thanks for providing us the code and analysis. I'll check and fix as soon as we are ready with Version 7.
That will be end of September (Sorry but V7 keeps us very busy at the moment)

[bobob...@gmail.com]

Hi,

This is still very much an issue:

java.io.FileNotFoundException: /opt/picapport/photos/Photos/RAW/2014/2014-03-06/P3060010.thumb.ppm (Read-only file system)

at java.io.FileOutputStream.open0(Native Method)
at java.io.FileOutputStream.open(FileOutputStream.java:270)
at java.io.FileOutputStream.<init>(FileOutputStream.java:213)
at java.io.FileOutputStream.<init>(FileOutputStream.java:162)
at de.contecon.picapport.plugin.dcraw.DcRawProcessWrapper$StreamHandlerBinaryFile.run(Unknown Source)

ERROR@ 17:03:13.876 de.contecon.picapport.plugin.dcraw.DcRawPlugin: Unecpected Exception in createJpegFile: Invalid RAW file: /opt/picapport/photos/Photos/RAW/2014/2014-03-06/P3060010.dng