Best way to make multiple http requests

[Francesco Pepe]

I need to create a script that should make multiple http requests.

I need to make 5 http requests each time. My script parse an xml to find something, if it found insert a value in database, if not make other 5 http request. And so on until it find what I'm looking for.

What's the best way to make this script? 

I should use cfthread? Will this cause timeout if it cannot find anything with the first 15 http requests?

[Francesco Pepe]

my firts try was to create a cfhttp tag nested inside a cfthread and make a loop to create about 10 cfthread and join all of them. I'm not sure this is the best way and I could not create a nested cfthread inside another cfthread if I cannot find what I was searching. So I should control if 

My other attempt is to create a loop that contains a simple cfhttp request until I found something, but in this case the request could timeout.

[Denard Springle]

You're best bet is to use a gateway instead... Gateways run in their own thread and can do what you need... Either through a CF gateway or a directory watch. I might even combine both and use the CF gateway to make http/s calls that get your XML and then a directory watch gateway to parse your XML downloaded by the CF gateway/http/s call.

--
You received this message because you are subscribed to the Google Groups "Object-Oriented Programming in ColdFusion" group.
To view this discussion on the web visit https://groups.google.com/d/msg/coldfusionoo/-/IVCexZN48AcJ.
To post to this group, send email to coldfu...@googlegroups.com.
To unsubscribe from this group, send email to coldfusionoo...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/coldfusionoo?hl=en.

[Francesco Pepe]

Are you taliking about event gateway?

http://livedocs.adobe.com/coldfusion/8/htmldocs/help.html?content=UseGateways_8.html 

To unsubscribe from this group, send email to coldfusionoo+unsubscribe@googlegroups.com.

[Denard Springle]

Yup. Event gateway is the way to go. I have an FTP app that uses the same process I described and been working like a charm for 2+ years :)

To view this discussion on the web visit https://groups.google.com/d/msg/coldfusionoo/-/kQ_yOpoFKcMJ.

To post to this group, send email to coldfu...@googlegroups.com.

To unsubscribe from this group, send email to coldfusionoo...@googlegroups.com.

[Francesco Pepe]

I need an enterprise license?

[Denard Springle]

No, event gateways are available in the Std edition.

On Jul 12, 2012, at 10:04 PM, Francesco Pepe <sgozz...@gmail.com> wrote:

I need an enterprise license?

--
You received this message because you are subscribed to the Google Groups "Object-Oriented Programming in ColdFusion" group.

To view this discussion on the web visit https://groups.google.com/d/msg/coldfusionoo/-/QE8CmFMyVQ4J.

[Francesco Pepe]

I should read a lot of tutorials...

[Francesco Pepe]

If I have understood well, with event gateway could make asynchronous http request and with directory watcher I will parse xml.

But I don't undrstand if after I started my loop of http requests, how can I stop the process with event gateway?

I don't want to waste my server resources. I don't want make http requests that are not necessary.

So my question is: I make an http request, parse the response with directorywatch, if I found what I'm looking for stop the process, otherwise make another http request. Is this possible?

Once I make an http request I should set a method to create a callback. Really don't find this in the documentation. Could someone help?

[Denard Springle]

Yup - you're on the right path. I would suggest using a scheduled task to determine if you have the data you need, and if not then kick off an CF Event gateway that makes the next X http requests (I believe you said 5 at a time). No call back is necessary really - the scheduled task recognizes if you have your data and if not kicks off the http requests, which download the XML to a directory where a directory watcher gateway then parses them for your value and puts it in the database. If it find it, the next time the scheduled task fires and checks for the value it'll find it, and no more http calls will be made. Assuming you are doing this a few times a day, a week, or a month - doesn't matter just make sure the scheduled task handles that logic and only fires http requests when needed. Make sense?

-- Denny
 

--

You received this message because you are subscribed to the Google Groups "Object-Oriented Programming in ColdFusion" group.

To view this discussion on the web visit https://groups.google.com/d/msg/coldfusionoo/-/Vu1x-DCmZJYJ.

[Francesco Pepe]

Yes, scheduled task is exactly what I've thought. But in this case, if I'm right I should schedule a task every about 2 min that checks if something is in the database.

Assuming I have 1000 of this scheduled tasks, is this good for performance?

Il giorno mercoledì 11 luglio 2012 21:33:28 UTC+2, Francesco Pepe ha scritto:

[Denard Springle]

Yikes! What are you building that needs 1000 scheduled tasks that run every 2 mins? 

--

You received this message because you are subscribed to the Google Groups "Object-Oriented Programming in ColdFusion" group.

To view this discussion on the web visit https://groups.google.com/d/msg/coldfusionoo/-/uSvleJQJwzMJ.

[Francesco Pepe]

I have multiple users, each one should insert an xml source, feed rss etc...

So if I schedule a task per users I have a lot of scheduled tasks. If I should check with scheduled task for each one that something is in the database, I will have a thousand of scheduled tasks....

This if I understand the way you suggested to use the scheduled task....

Il giorno mercoledì 11 luglio 2012 21:33:28 UTC+2, Francesco Pepe ha scritto:

[Francesco Pepe]

Probably, the best solution is to create a scheduled task for each user, scheduled task will check 10 http requests and put in the database. The scheduled task will run each half hour. I should set a maximum of 300 http request per user, so I will have only a scheduled task per user that will run each half hour. So each user will make maximum 300 http request. 

Il giorno mercoledì 11 luglio 2012 21:33:28 UTC+2, Francesco Pepe ha scritto:

[Denard Springle]

Ahh... no, you should write one scheduled task that executes one query to determine which users need refreshing, and then fire off CF event gateway requests only for those that need it - but yes, that is still a lot of traffic and you'd be best to run that on a separate server (or two, or three depending on your scale) than the site that serves pages to customers. So, yes, it's still a lot of bandwidth, and in that instance, you may actually have to have a scheduled task that fires a gateway to determine which users need refreshing and then have that gateway fire off additional instances as needed to gather data - that would scale better as there is still the timeout issue with scheduled tasks to contend with, as it really executes the same basic way as a <cfhttp> call does, and with 1000+ users you may find a timeout error.

--

You received this message because you are subscribed to the Google Groups "Object-Oriented Programming in ColdFusion" group.

To view this discussion on the web visit https://groups.google.com/d/msg/coldfusionoo/-/JOC9UgUbMHEJ.

[Francesco Pepe]

Just a question: the directory watcher have a timeout just like every cfm or cfc page?

Il giorno mercoledì 11 luglio 2012 21:33:28 UTC+2, Francesco Pepe ha scritto: