Transfers hanging in characterizefFile_v0.0

[Buddha Buck]

I am investigating a recurring problem we are seeing with transfers in Archivematica 1.6.1 (running on CentOS 7). The transfer gets "stuck" on the "characterize and extract metadata" microservice, and does not proceed past that point.

The test transfer I am using for this is a zipped bag containing a total of 87 files (some of them in nested ZIP archives) totalling 870MB. The transfer time before it hangs is less than 15 minutes.

While the characterization and extract metadata step is running (before it hangs), I note a persistent load average on my machine of 6-7, with several concurrent executions of characterizeFile.py. When it hangs, the load average drops down to under 1, and characterizeFile.py is no longer executing.

The output visible by clicking on the gear icon on the dashboard includes characterization information for 86 of the files, but lists the final PDF in the bag as still in progress (a task start time, but no end time).

The log files at /var/log/archivematica/MCPClient have log entries for characterizing all 87 files. The file which is hanging is not the last one in the logs.

Restarting the MCP client and server "fails" the hung characterization effort and restarts it. The 2nd time it goes much faster because the majority of the work has already been done. I get 86 listings of "won't recharacterize" before it hangs on the 87th.

"gearadmin --status" shows no running instances of characterizeFile

The "Files" table in the database shows the incomplete file, but the main_fpcommandoutput table has no outputs for it.

I did find a "line" in the /var/log/archivematica/MCPCServer.debug.log file reporting that the characterization task completed, and includes a copy of the output from FITS -- all 1.3 MB of it, most of it from JHove and Exiftool. There is also a stack trace from JHove included at the end of that long output.

My suspicion is that the size of the output is preventing the completion of the task to be recorded properly, so the transfer does not proceed to the next step.

I'm not sure where to proceed from here. Has anyone else seen anything like this? 

[Buddha Buck]

To follow on my own question, by default MariaDB as installed by Archivematica on CentOS 7 is configured to allow a maximum "packet" size of 1MB, which precludes the writing of the 1.3MB results to the DB. MariaDB quits listening and throws an error which got recorded in the MCPCServer.debug.log file.

Adding a "max_allowed_packet = 16M" setting to /etc/my.cnf.d/server.cnf solved the problem.