writing chunks in non-blocking mode

[Steve Barnsley]

I'm writing a Mojolicious application that will bulk load transactions into an

ERP system.  The user will upload an Excel spreadsheet with multiple worksheets in it.

Each worksheet will contain data needed to execute screen transactions in the ERP, typically

one transaction per worksheet row.  There are a dozen or so different worksheet types each

of which does different ERP transactions. A workbook can contain several different types

of worksheet.

There is code in the model that translates the spreadsheet contents into an array.  Each entry

in the array represents a worksheet and contains an array for the rows from that worksheet plus

other attributes like the appropriate template name etc.

There is a Mojolicious template for each worksheet type.  The templates contain code that

loop though the rows in the array entry whose address is passed via the stash, executing The

ERP screens as needed and writing transaction level status info to an output Excel workbook.

The controller loops through the data array sequentially calling render on the template name indicated 

in the array for each entry.

The ERP transactions can take a second or more to execute for each row in a worksheet and there

can be hundreds (occasionally thousands) of rows in a worksheet.

This would lead to long periods of time where the user would see nothing going on and, unless 

the timeout was set to some ridiculous value, timeouts will occur.  To get around that I thought

that I would present an html table for each worksheet, row by row, on the browser showing the worksheet name, row number

and transaction status as the process works its way through the array.  I added code to do that in each template.

That kinda works, however, the write_chunk('stuff',$cb) being non-blocking causes the controller to

zip through all of the arrays very quickly and all of the html tables get munged together instead of

appearing as separate tables presented chronologically as the workbook content is processed. Using

write_chunk without a callback causes the entire page to be rendered at the end of the process which

doesn't achieve my goal of keeping the user aware of how her transactions are progressing.

Is there any way to get Mojolicious to flush chunks of html to the browser without entering non-blocking

mode so that the output from the various templates can be rendered sequentially and chronologically in 'real time'?

Thanks, Steve.

[Heiko Jansen]

I´m not sure if I understand your workflow correctly but how about this: for every row processed send a small JSON documents to the browser which contains information on the worksheet, row and row status and then have some JavaScript code in the browser assemble and update the correct HTML table (inserting another row)?
I´d probably render a skeleton HTML page as response to the upload of the Excel file, then start a new AJAX request from that page which triggers the processing of the spreadsheet uploaded previously and receives the processing status as "server-sent events" (cf. http://www.w3.org/TR/eventsource/).

Heiko

[Steve Barnsley]

Thanks for the suggestion Heiko.  I've done something similar to what you propose in a prior project - send the entire table to the browser at the start with the 'status' column blank then open a websocket that sends the status value for each row to the browser and have a little javacript that plugs in the status value in the appropriate row.  I was hoping to avoid that with this project because it seems like overkill for such a simple project, but the eventsource method in the link that you sent looks much simpler to implement so I'll experiment with that.

I tried 'wrapping' the list of calls to the various templates in a $c->delay(@list); structure, this causes the first template (table) to execute in its entirety but it never starts the second entry in the list (even though I see the result from the very last statement in the sub in $list[0].) Is there some magic that I need to invoke to make delay start the next item in the list?    

[Heiko Jansen]

Am Freitag, 14. Oktober 2016 01:06:57 UTC+2 schrieb Steve Barnsley:

I tried 'wrapping' the list of calls to the various templates in a $c->delay(@list); structure, this causes the first template (table) to execute in its entirety but it never starts the second entry in the list (even though I see the result from the very last statement in the sub in $list[0].) Is there some magic that I need to invoke to make delay start the next item in the list?

Without any deeper knowledge of your code that´s a tough question.
@list is a list of subroutine references, right?
You do call $delay->begin in every callback in that list, don´t you?
Otherwise, like https://metacpan.org/pod/Mojo::IOLoop::Delay#steps says: "This chain will continue until ... a callback does not increment the event counter ..."