Delay a Node Stream Transform process

[Kayode Odeyemi]

Hi,

I'm streaming data using the following data flow:

Source(Database) -> Transform (An HTTP request) -> Sink(Database)

Obviously the HTTP request task is a blocking task. At the moment, it seems the Source is faster than the Transform process.

How do I make the Transform process delay or better still work in tandem with the Source or much better make the source delay until the current transform task finishes?

Appreciate your help.

Thanks

[Zlatko]

Depending on how you implemented the transform stream, the pipeline should be balancing itself automatically. How did you implement the stream? Do you use object mode? Did you try to do something with highWaterMark?

[Kayode Odeyemi]

I'm using objectMode.

Do I need to do anything differently? I sometimes get a 502 error during the transform process, which proves to me that the web server handling the HTTP request is taking too much that it can handle. So, is this a performance tuning task that should be done on the web server side of things? - Like increasing the web server response timeout?

If I increase the web server response timeout, will the client wait for it to finish?

Thanks Zlatko

--
Job board: http://jobs.nodejs.org/
New group rules: https://gist.github.com/othiym23/9886289#file-moderation-policy-md
Old group rules: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
---
You received this message because you are subscribed to the Google Groups "nodejs" group.
To unsubscribe from this group and stop receiving emails from it, send an email to nodejs+un...@googlegroups.com.
To post to this group, send email to nod...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/nodejs/791f9e72-c9e6-458f-bad7-04496ffd9505%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.