As an aside, I don't think I need to actually write in chunks, since Mojolicious' write() method also takes a callback to cause the output buffer to flush.
is there a pattern that I should be following in the example above? Can you point me to example code perhaps?
Your problem (if I'm reading your code right) is that you don't have anything to catch when the query has exhausted itself. Instead it looks like you're immediately calling write_chunk($final_json_stuff)
after the first callback returns, which then closes the JSON object being constructed, and when the other rows get sent, the most they'll do is trigger a syntax error at the far end (because they'll be outside of the }
that you sent in $final_json_stuff
) and if the far end is sufficiently badly coded, it'll just snarf enough of the stream to see a complete JSON object and ignore what follows, and then rows 2-n get silently dropped on the floor.
Meaning the if
in your write_results needs a else (i.e., $query->hash
returns false signalling that there are no more rows), and that is where you want to write your }
or whatever it is you're doing to close out the JSON thingie.
However, I also agree that this stuff really shouldn't be in the model. If we're really doing MVC-separation right, then the question of generating JSON is really the job of the Viewer, what the controller invokes after getting the result from the model....
At which point your problem is a model that returns partial results with a thunk to either get more or indicate we're done -- whether you want to just pass the Mojo::Pg::Results object around or have something more abstract wrapping it is up to you -- but, unless I'm missing something, Mojo doesn't seem to have a notion of "partial template" that can deal with these things (I'm not even sure what it would look like)...
... so you have to fake it in the controller, which I believe will go something like
# in the model sub get_data { ... return $pg->db->query('SELECT whatever...'); } # in the controller $qresult = $model->get_data(...); $self->write_chunk('{ ..., "rows": ['); my $cb; my $comma=''; $cb = sub { my $row = eval { $qresult->hash; }; if($@) { $controller->finish('], "status" : "error", "msg": '.to_json($@).'}'); } elsif ($row) { $controller->write_chunk($comma . to_json($row), $cb); $comma = ','; } else { $controller->finish('], "status" : "ok" }'); } } $cb->(); $self->render_later; # not sure if this is necessary # DO NOTHING FURTHER HERE
It is annoying not to have a way to just write to the output stream without resorting to callbacks, in this scenario.
# DO NOTHING FURTHER HERE, MAY BE IGNORED BY CLIENT
be added to the hello world example in the documentation as it might help folk understand that all subsequent output must be done in the callback to appear in the stream in the correct place.
The examples show what to do to write non-blocking via a callback but there's a higher level concept that's missing and IMHO needs further explanation.
Allan
can I suggest that something akin to Roger's comment:# DO NOTHING FURTHER HERE, MAY BE IGNORED BY CLIENTbe added to the hello world example in the documentation as it might help folk understand that all subsequent output must be done in the callback to appear in the stream in the correct place.
Using https://gist.github.com/AllanCochrane/1dd34f0fd3649ece7689 as an example, the callback code looks recursive (and you alluded to that in a comment above) but it actually isn't.
Hi,can I suggest that something akin to Roger's comment:# DO NOTHING FURTHER HERE, MAY BE IGNORED BY CLIENT
# DO NOTHING FURTHER HERE; WE HAVE TO WAIT FOR THE CALLBACKS TO ACTUALLY GET CALLED
No. What I actually meant was this:# DO NOTHING FURTHER HERE; WE HAVE TO WAIT FOR THE CALLBACKS TO ACTUALLY GET CALLED
> To my mind the callback runs and as part of its functionality schedules itself for later execution, it doesn't invoke itself directly as 'classical' recursion.This much is correct, i.e., you're only creating a closure once, and all calls to it except for the first are coming from the event loop (i.e., each time the write buffer is emptied, the callback is invoked). However there is still recursion in the sense that the function is referenced from within its own body, which means there's a loop of pointers ($cb refers to the sub and the sub refers to $cb) that the reference-counting garbage collector will never free up unless you do something to break it explicitly.(weaken($cb) but do it after that first call.)