Toll Fare Calculator (Android App)

[Hongster]

There are very few localized android apps. I am creating this app,
hoping that it will be a useful tool and also encourage Android
development. I am thinking of creating a calculator for toll fare,
similar to http://www.plus.com.my/toll_calculator.asp.

I have difficulty extracting the (fare) data from Plus's website. I
tried to write a screen-scraping program, but it failed. Can anyone
give me some help?

[Leong Hean Hong]

This html scraping exercise is getting harder than I thought. I am
going to share some information here, hoping someone can help me.

The http://www.plus.com.my/toll_calculator.asp is developed in ASP,
hosted on a Window machine.

A number of <iframe>s are used. The actual calculator page is at
http://www.plus.com.my/tollfare/index.asp. You will notice that 2
variables, startloc and endloc, are POSTed to http://www.plus.com.my/tollfare/fare_details.asp.
The result is redirected to another <iframe> named "fare_details".

You can try http://www.plus.com.my/tollfare/fare_details.asp?startloc=ASS&endloc=ASU

It works if the http request is sent through browser. Problem occurs
when I tried to send a http request through my code. I kept getting
server error. I *suspect* cookie might be a possible cause for error.

On Sep 25, 1:00 am, Hongster <expertle...@gmail.com> wrote:
> There are very few localized android apps. I am creating this app,
> hoping that it will be a useful tool and also encourage Android
> development. I am thinking of creating a calculator for toll fare,

> similar tohttp://www.plus.com.my/toll_calculator.asp.

[-syah-]

Hi! How is things? Maybe only do a certain segment / stretch of the
PLUS h/way for the time being?

I am highly interested on the dataset, if you manage to scrap off the
site. Thanks..

Regards,
-syah-

On Sep 26, 11:03 am, Leong Hean Hong <expertle...@gmail.com> wrote:
> This html scraping exercise is getting harder than I thought. I am
> going to share some information here, hoping someone can help me.
>

> Thehttp://www.plus.com.my/toll_calculator.aspis developed in ASP,

> hosted on a Window machine.
>

> A number of <iframe>s are used. The actual calculator page is athttp://www.plus.com.my/tollfare/index.asp. You will notice that 2
> variables, startloc and endloc, are POSTed tohttp://www.plus.com.my/tollfare/fare_details.asp.

> The result is redirected to another <iframe> named "fare_details".
>

> You can tryhttp://www.plus.com.my/tollfare/fare_details.asp?startloc=ASS&endloc=ASU

[Leong Hean Hong]

Glad to hear your reply :)

I have solved the cookie problem. The solution is to get a session ID
(in cookie) from any site (e.g. http://www.plus.com.my/index.asp).
Submit the session ID, with any HTTP request you want, back to the
site.

This what I have done:
- Get the fare for any 2 plazas in a stretch. For example a trip from
Johor Bahru Causeway to Alor Setar (Utara) involves 3 stretches.
- Fare for all Open System.
- Total fare (including open systems and all stretches in a trip) for
any 2 points.

Todo:
- Record all stretches in a trip.
- Implement in Android.
- Resolve some minor issues (e.g. bug occurs when destination is
Lunas).

Currently I am extracting the data in Python. Tonight I will put my
Python code on snippt.net and upload my data (in csv) somewhere, to
share it with everyone.

On Oct 8, 7:00 am, -syah- <sazalins...@gmail.com> wrote:
> Hi! How is things? Maybe only do a certain segment / stretch of the
> PLUS h/way for the time being?
>
> I am highly interested on the dataset, if you manage to scrap off the
> site. Thanks..
>
> Regards,
> -syah-
>
> On Sep 26, 11:03 am, Leong Hean Hong <expertle...@gmail.com> wrote:
>
> > This html scraping exercise is getting harder than I thought. I am
> > going to share some information here, hoping someone can help me.
>

> > Thehttp://www.plus.com.my/toll_calculator.aspisdeveloped in ASP,

[Leong Hean Hong]

Hi, sorry for the late update. Streamyx was crawling like a snail over
the weekend :(

http://bit.ly/u7SES
This is the python script I used for extracting data from the mini
calculator (http://www.plus.com.my/miniquery/index.asp). I have sorted
the 87 toll plazas and grouped them into 3k+ unordered pairs.

http://ahhong.co.cc/data.csv
This is the result of the extraction. The fields are (Start,
Destination, Class1 Normal Rate, Class1 Off Peak Rate, Class1 Off Peak
Festive Rate, Class2 Rate, Class3 Rate, Class4 Rate, Class5 Rate,
Distance in km). There are 10 fields in total.

NOTE:
- The data is not complete.
-- Not all pairs are valid. For example, there is result for Jitra to
Bkt. Tambun (Seletan), no result for Bkt. Tambun (Seletan) to Jitra.
-- There were errors for certain pairs (e.g. BTT, JBC). The website
return some error when you try to find data for Bt. Tiga to Johor
Bahru Causeway.
- The tolls code are already sorted alphabetically. In future, I will
reverse the order and extract the data again. Because not all routes
are bidirectional, come are unidirectional (e.g. BTS, JTR).

I think this is enough for me to start working on the Android coding.

@syah Do you want to share how you will be using the data?

P.S. This set of data took me 5 hours to extract.

On Oct 9, 10:17 am, Leong Hean Hong <expertle...@gmail.com> wrote:
> Glad to hear your reply :)
>
> I have solved the cookie problem. The solution is to get a session ID

> (in cookie) from any site (e.g.http://www.plus.com.my/index.asp).

> > > Thehttp://www.plus.com.my/toll_calculator.aspisdevelopedin ASP,

[-syah-]

Hi!

Superb! Congrats on your effort. I was looking to have this kind of
datasets before, for students' projects. This would be great to have
for such projects, if you allow such use. Is it ok with you?

Also, the code should be able to be use for other sites as well, with
minor modifications - e.g. for LRT, bus, etc. Yes, I think the dataset
is enough to start on your Android project. All the best! Looking
forward for any updates on it. May I suggest your project be named:
plusDroid hehehe....

I have some Android ideas: such as on having the location of R&R into
such apps - it can show the nearest R&R from current location (might
need to know the current direction too :-D ), and also a simple Speed
Limit advisor using the data from the Android accelerometer and
current location. Anyway, those are just my ideas.. Thanks...

ps: can the admin remove the spam msgs on the groups' site, as it is
quite embarrassing... :-P

Kind regards,
-syah-

[Leong Hean Hong]

I am always supportive of sharing information/code for educational
purpose. Feel free to use the data and my python code anyway you want,
you can even use if for commercial purpose if you want ;)

Singapore developers have already developed Android apps for bus and
MRT services. This is still lacking in Msia. I hope to share my Toll
project with the rest of the CodeAndroidMY community, and encourage
more developers to develop localize apps. Actually I was thinking of
creating Astro Channel Schedule Android apps, after this project is
completed.

I have to *inform* you that the data may not be complete, as certain
route is unidirectional. Anyway, you can modify my code to extract
more data from Plus.com.my. For the time being, I will just use this
dataset for development purpose.

plusDroid sounds good! I also came out with a name called
"Antollroid", but I am still open to suggestions. What is "R&R"?

FYI: Accelerometer is a device for measuring rate of change in
acceleration (in gravity). It measures the gravitional acceleration
along the 3 axis, which is useful for measuring orientation/tilting of
device. I don't think it can measure velocity of a moving device.

Nazrul is the admin for this forum, I have forwarded your message (on
the spam issue) to him.

[-syah-]

That's great! Thanks.

The Astro idea sounds good. Maybe can have something similar to the
Beep apps in the Android Market, heee.. :D I sometimes watch the BBC
TV (Live and recorded) on my Android... sometimes ok, sometimes not.

R&R is Rehat & Rawat or Rest & Relax.. Sometimes called Service area,
as what they call it over here.

My mistake! Should have been, the GPS sensor data. Take two data
point, get the distance and over the time taken to get the speed.
Although, I do not know an iota of any way to do it, codewise :-P
Thanks to the admin...

regards,
-syah-