StackOverflowError from stock-management blocking requisition approvals

[Sebastian Brudziński]

Hi dev group,

I wanted to bring your attention to the issue that we have just started seeing in Malawi - the stock-management service is throwing StackOverflowError while processing StockEvent sent from the requisition service on final approval, and therefore blocking the ability to approve the requisition.

Apparently, this is not happening for all of the requisition approvals and I'm not exactly clear on the frequency of this. So far, I'm seeing this in two of our requisitions currently waiting for approval. I've spent some time debugging this, but was not able to find anything unusual about those two requisitions. I've determined the line that throws the error though (it also wasn't that obvious) and have put it in the ticket: https://openlmis.atlassian.net/browse/OLMIS-3813
The StackOverflowError is coming from the circular references in the toString() method, between StockEvent and StockEventLineItem (as can be determined from the logs attached in the ticket). This of course needs to be fixed, but I'm not sure this is a root cause of the problem (otherwise, why isn't this a problem for all approvals?)

Unfortunately, I'm not seeing any other workarounds than removing all of the stock cards from the stock-management for the given facility. Before taking this drastic step though, I wanted to see if anyone else has got any ideas where the problem might be or what better workaround exists.

Thanks,
Sebastian.

--

Sebastian Brudziński
Software Developer / Team Leader
sbrud...@soldevelo.com

SolDevelo Sp. z o.o. [LLC] / www.soldevelo.com
Al. Zwycięstwa 96/98, 81-451, Gdynia, Poland
Phone: +48 58 782 45 40 / Fax: +48 58 782 45 41

[Josh Zamor]

Hi Sebastian,

As I noted in the ticket, we really need steps to reproduce.

We looked at this here and while we can't find a way to reproduce what you're encountering, I have to ask why you zero'ed in on Stock Event and Stock Event Line items?

• Neither declare their own toString method - it's given by @Data.  This looks similar to Stock cards and their line items.
• The line of code you linked to appears to deal with stock event dtos, and not the similarly named domain class
• The first/second line in your log statement is actually for a ZonedDateTime class, why did you exclude that?
  

It'll be very difficult to help out with this without those reproduction steps.  Lets try to focus on that first.

Thanks.  The more information you can get us the more we'll be able to help.

Best,
Josh

[Brandon Bowersox-Johnson]

I agree with Josh.

 

Here are a few ideas for troubleshooting/reproducing (sounds like a tricky bug!):

 

• If it happens only with certain requisitions/submissions, copy that JSON into Postman so you can edit the data and try submitting again. You could “chop down” the data to remove products or zero out quantities and see when you reduce the data if it will submit successfully. That could help you narrow it down to a single line item or value that is causing the error. (Since you said most times it works, but it only fails on certain requisitions, this approach could help you find what is different in the data.)
• Copy the production database so you can reproduce this on a local dev environment. Since the bug is data-related, having a copy where you can locally troubleshoot, add Debug statements into your code, or use an interactive debugger to step through it.
• Once you do know which records get the error, and why the error happens, then ideally it would be possible to write down steps-to-reproduce on a plain vanilla v3.2.1 system. This will help any developer or QA person working with this ticket.

 

That is just a few ideas. I hope some of this helps!

 

-Brandon

--
You received this message because you are subscribed to the Google Groups "OpenLMIS Dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to openlmis-dev...@googlegroups.com.
To post to this group, send email to openlm...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/openlmis-dev/715b7f54-189b-4f7f-be4e-708c2655c735%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Sebastian Brudziński]

Hello.

Thanks for your input, everyone.

I eventually found out that the problem was with duplicative StockCards. The fix for duplicate Stock Cards being created was apparently made in https://openlmis.atlassian.net/browse/OLMIS-3533 but it didn't make it into the 3.2.1 release.

Even though the root cause appears to have been fixed, we should still resolve the issue with circular references in toString, equals etc. - I've modifed the ticket I mentioned in the original post to tackle this. Otherwise, we may end up with more difficult to track issues, because of this. My best guess is that a different error/exception/validation was originally triggered due to that duplicative StockCard and an attempt to print the object was actually failing and causing StackOverflow.

@Josh - As I've mentioned in the original post, I didn't believe the StackOverflow and this circular toString reference is a root cause of the problem. Lombok generates equals and toString methods using all fields by default though, and that's a problem, that I believe we should still fix. I agree that StockCards and StockCardLineItems have the same issue. The line I've marked actually lazily loads StockCard domain objects from the database, which has got references to StockCardLineItems and StockEvent. I've attached a screenshot from the debugger in the ticket.

Thanks again!
Sebastian.

To view this discussion on the web visit https://groups.google.com/d/msgid/openlmis-dev/18BB2727-4CB4-4C72-B1A8-E67CEC324326%40villagereach.org.

For more options, visit https://groups.google.com/d/optout.

--

Sebastian Brudziński
Software Developer / Team Leader
sbrud...@soldevelo.com

[Łukasz Lewczyński]

Hi,

I only wanted to mention that there is a circular dependency between StockEvent and StockEventLineItem domain classes. Also in the 3.2.1 release the DTO contains the StockEventLineItem domain class. This was changed to use DTO for line items. If the StackOverflow is still present perhaps we could remove the stockEvent field in line items but first we need to check if this field is used.

Regards,

Lukasz

Łukasz Lewczyński
Software Developer
llewc...@soldevelo.com

To unsubscribe from this group and stop receiving emails from it, send an email to openlmis-dev+unsubscribe@googlegroups.com.

To post to this group, send email to openlm...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/openlmis-dev/715b7f54-189b-4f7f-be4e-708c2655c735%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "OpenLMIS Dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to openlmis-dev+unsubscribe@googlegroups.com.

To post to this group, send email to openlm...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/openlmis-dev/18BB2727-4CB4-4C72-B1A8-E67CEC324326%40villagereach.org.
For more options, visit https://groups.google.com/d/optout.

--

Sebastian Brudziński
Software Developer / Team Leader
sbrud...@soldevelo.com

SolDevelo Sp. z o.o. [LLC] / www.soldevelo.com
Al. Zwycięstwa 96/98, 81-451, Gdynia, Poland
Phone: +48 58 782 45 40 / Fax: +48 58 782 45 41

--

You received this message because you are subscribed to the Google Groups "OpenLMIS Dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to openlmis-dev+unsubscribe@googlegroups.com.

To post to this group, send email to openlm...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/openlmis-dev/b460afde-0ca3-009c-7f8c-0b9f924901d1%40soldevelo.com.

For more options, visit https://groups.google.com/d/optout.

[Sebastian Brudziński]

Hi Łukasz,

it's not the bidirectional relation between StockEvent and StockEventLineItem (and similarly between StockCard and StockCardLineItems) itself that is the problem. I believe we can still keep it. The problem is that the equals/toString/hashCode methods of those classes use all the fields (meaning StockEvent uses StockEvenLineItem and StockEventLineItem uses StockEvent. Modifying those methods to only use relevant fields should be sufficient (I believe Lombok allows to specify which fields should be included in the generated methods). Anyways, I believe it's OK to continue discussion about possible fixes/implementation in the ticket linked in original post.

Best regards,
Sebastian.

[Łukasz Lewczyński]

I am not sure if (according to DDD) line items should have reference to the aggregate object (in this case StockEvent / StockCard). Pawel could you give more details about this?

Łukasz Lewczyński
Software Developer
llewc...@soldevelo.com

To view this discussion on the web visit https://groups.google.com/d/msgid/openlmis-dev/d7f291fe-2a9a-5dae-6fa2-32ff9e61adc5%40soldevelo.com.

For more options, visit https://groups.google.com/d/optout.

[Paweł Albecki]

Hi all,

Thanks for summon me, Łukasz. You are actually right, I can't find good reason why bidirectional relation should be used between Aggregate and Entity and looking at StockEventLineItem, reference to stockEvent is not even used anywhere (same for StockCardLineItem and StockCard). So I think it can be safely removed.

Best regards,

Paweł

--

Paweł Albecki
Software Developer
palb...@soldevelo.com

[Sebastian Brudziński]

Thanks Pawel,

do you want to add that note to the ticket as well? I'm afraid it will be lost here when the implementation kicks in.

[Paweł Albecki]

Sure, I will add that note.

To view this discussion on the web visit https://groups.google.com/d/msgid/openlmis-dev/ae081ef1-a429-587f-3b15-cd6649687c52%40soldevelo.com.

For more options, visit https://groups.google.com/d/optout.

--

Paweł Albecki
Software Developer
palb...@soldevelo.com