Comment Full Message longblob UTF-8

[Ankit Bharti]

Hi All

I facing a bit of problem with the Spring/JSP UTF-8 problem when trying to retrieve full message content. It seems the comments are getting stored perfectly in database but somehow method Comment.getFullMessage content is replaced with malformed bytes.I have CharacterEncoding filter set forced to UTF-8. Other parts are working well.

I tried almost everything, can somebody help me please !!

// all comments

for (HistoricTaskInstance taskInst : historyTaskList) {

Iterator<Comment> commentIterator = taskService.getTaskComments(taskInst.getId()).listIterator();

while (commentIterator.hasNext()) {

Comment comment = commentIterator.next();

User user = getUser(comment.getUserId());

CommentDto cDto = new CommentDto(comment, user);

System.out.println("Byte " + comment.getFullMessage());

commentListDto.add(cDto);

}

}

[thorben....@camunda.com]

Hi,

There is a related bug report [1] that comments are encoded with the default platform encoding. Perhaps that is the issue you face here. Please vote for the ticket if that is the case. It is not trivial for us to fix this issue since for the sake of backwards compatibility, we cannot assume that from now on all existing comments are encoding in UTF-8 or something similar.

Cheers,
Thorben

[1] https://app.camunda.com/jira/browse/CAM-3035

[Ankit Bharti]

Hi Thorben

Thanks for your swift reply,Upvoted the issue.
Until the issue is resolved or may be some other concrete alternative down the line , is there any other workaround for now to get the exact string value from database for comment full message ? Any recommended approach that i can adopt for now to get the full String ?

[Ankit Bharti]

A Strange behaviour while testing with Unicode character [1]

<c:forEach items="${requestScope.commentListDto}" var="c">

 ${c.comment.message} <br> ${c.comment.fullMessage}
</c:forEach>

comment.message is limiting to ~160 characters but displaying correct encoding values as in database. 
comment.fullMessage is displaying malformed String via Blob value.

I have even tried : String messageUtf = new String(comment.getFullMessage().getBytes(StandardCharsets.UTF_8)); but no Success.

[1]http://www.ltg.ed.ac.uk/~richard/unicode-sample.html

[thorben....@camunda.com]

Hi,

Unfortunately the bug is hidden behind the Camunda API and I'm not aware of a purely code-based workaround.
Perhaps you can set the JVM's default encoding to UTF-8 (or any other encoding that works for you) when you start the JVM with the parameter -Dfile.encoding=UTF-8 (see [1]). This may not solve the problem for existing comments in your database but should work for new comments. Note you'll have to set the parameter for every JVM that accesses comments.

Regarding your last post: The shortened message is not converted to bytes by the process engine but ends up in a VARCHAR field encoded by the database. That should be why the shortened message does not show the problem.

Cheers,
Thorben

[1] https://stackoverflow.com/questions/361975/setting-the-default-java-character-encoding

[Ankit Bharti]

Hi 
Just could not get it to work in Google App Engine. Tried almost every resource available .
By any chance can you please tell me whether i can increase the varchar limit of message field from 4k to 20k to support alteast 6rows of textarea in datatable ?.
Is there any sideaffect by doing so or is it permitted ?

One thing i was not able to understand when posting comment using  -  taskService.createComment(taskId, processInstanceId, message);
The entire message was truncated to first ~160 characeters or so. Can i somehow increase this limit ?

[thorben....@camunda.com]

Hi,

You can certainly change the database field length. It is just not a use case we test for, so you'll have to try it yourself. Permitted is anything ;)

Regarding the 160 characters limit, I am afraid this is a hard limitation in the code, see [1]. You could replace the entire command by doing the following:

* Subclass org.camunda.bpm.engine.impl.cmd.AddCommentCmd and make the truncation flexible (you can also submit this as a pull request to get it into the regular command)
* Subclass org.camunda.bpm.engine.impl.TaskServiceImpl and overwrite the method. Execute your custom command in that implementation.
* Implement a process engine plugin [2]. In its preInit method, call processEngineConfiguration.setRuntimeService(newInstanceOfYourCustomRuntimeService)
* Put the plugin on the engine's classpath

Cheers,
Thorben

[1] https://github.com/camunda/camunda-bpm-platform/blob/master/engine/src/main/java/org/camunda/bpm/engine/impl/cmd/AddCommentCmd.java#L62-L65
[2] https://docs.camunda.org/manual/7.3/guides/user-guide/#process-engine-process-engine-plugins

[Ankit Bharti]

Excellent. Worked out perfectly well. Such a relief. Not sure about performance issues but i changed datatype from varchar to Text Field as i need more space.
This solution has really helped me in countering the special characters too.