How to get data from e department sites

[Vartika Sinha]

Hello All,

I need some data from an E-department portal. This data is fetched using selection options on the site. It is then displayed onto the maps. 

I need the points that are being display on the maps, for an academic project I am working on. 

Is it advisable to send out of site multiple (could be in thousands) XHR calls to the site to get the response. If not, then how can I access the data?

Regards,

Vartika

[srinivas kodali]

Please post the website from which you need data. Each website design is different and no one can tell you how without checking it.

Regards,
Srinivas Kodali
www.lostprogrammer.com

--
Datameet is a community of Data Science enthusiasts in India. Know more about us by visiting http://datameet.org
---
You received this message because you are subscribed to the Google Groups "datameet" group.
To unsubscribe from this group and stop receiving emails from it, send an email to datameet+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Vartika Sinha]

http://45.114.143.92/CMRabiWebService/#/GMapReportTest

I want to use data from this site. I can write JavaScript to get lat long points for all districts mandal and villages. But I don't know if this is allowed. Or how should I get about it.

[srinivas kodali]

Make GET requests with changing parameters of districts, mandals and other ids. You will find them when you inspect the requests. 

http://45.114.143.92/CMRabiWebService/includes/areaSownDataObjects.php?page=GMapRepTest&selectedDist=2&selectedMand=53&selectedVill=1742

Regards,

Srinivas Kodali

www.lostprogrammer.com

On Wed, Apr 19, 2017 at 9:13 PM, Vartika Sinha <sinha.v...@gmail.com> wrote:

http://45.114.143.92/CMRabiWebService/#/GMapReportTest

I want to use data from this site. I can write JavaScript to get lat long points for all districts mandal and villages. But I don't know if this is allowed. Or how should I get about it.

[Vartika Sinha]

There would be a huge amount of calls. I just wanted to know if it is allowed? Although I will set an interval of around 2 minutes. Is it ok if I use the same IP. I am just concerned about the policies.

[srinivas kodali]

There are no policies. You can do frequent calls depending on the server response times. I look at that as an indicator whether the calls i make can get the server down.

Considering the response times from this server. I wouldn't think much about it. You won't be doing a DDoS attack to download this data.

Regards,
Srinivas Kodali
www.lostprogrammer.com

On Apr 20, 2017 7:19 AM, "Vartika Sinha" <sinha.v...@gmail.com> wrote:

There would be a huge amount of calls. I just wanted to know if it is allowed? Although I will set an interval of around 2 minutes. Is it ok if I use the same IP. I am just concerned about the policies.

[Vartika Sinha]

Thank you for pointing these things out. I will be careful in setting the interval of calls. It should not do anything much. But I thought it is good to be aware of such things. Thanks again!!!

To unsubscribe from this group and stop receiving emails from it, send an email to datameet+u...@googlegroups.com.