Web Component Cookies

[StudentWinna]

Hi I am a student and I am trying to create an app that displays the html of a webpage using the Web component. Right now the app does not display the correct html although it goes to the correct webpage in the browser on my computer using the same url. It goes to a page that says I have not given enough information on the app. I believe the issue is that I need to utilize cookies in order to save certain data that would allow me to get to the correct webpage. I have tried using the allow cookies component but it has not made a noticeable difference. I do not know if I am using the allow cookies method properly, also I am not sure if there is another way to help you connect to certain urls. Thanks, any suggestions would be very useful. 

[SteveJG]

Posting your blocks and web page link would help us understand what you want to do.

You want to "display(s) the html of a web page"  Does that mean you want to display the web page html code itself or the html 'image' (the entire web page)? If the latter, you should be using s WebViewer or ActivityStarter..

Are you attempting to 'scrape' specific data from a Web page perhaps?

Regards,

Steve

[SteveJG]

I now know a little more but no one else that can help you does Nicholas.  You responded directly to me and not in the Forum.  Please respond in the Forum..Thanks.

OK, you want to scrape a WebPage and think your issue is related to cookies. Perhaps.

Here is some general information from the Forum on scraping web pages  https://groups.google.com/forum/#!searchin/mitappinventortest/scrape$20web$20page%7Csort:date  What is in the links may help you.

Some web pages cannot be scraped, some can be scraped only for data posted directly in the html and some, that have links to data might be impossible to scrape. No one can really provide specific help unless we know a little about the url you want to scrape .  If you post an image of your relevant blocks so far, that will help someone provide specific advice.  Right now 

your question is like this question:  How long is a string?

Regards,

Steve

[StudentWinna]

Sorry for replying directly. It was an accident. 

I am scraping data from the web page. In my blocks I am saying Web1.url to (a url method that goes to a web page, which I am trying to retrieve data from). Then I simply change the text in a textbox to getResponsecontent which gets me the html. 

I am also doing Web1.allow cookies to true but that doesn't seem to be making a difference.

This is the url I am trying to connect to.

https://connection.naviance.com/family-connection/auth/login/authenticate?username=(insert my username)&password=(insert my password)&hsid=winnacunneths

I know that when you clear the cookies on the browser and put this link in it, it doesn't work without the hsid variable specified. But when you add the hsid it does work. Although on the web component in app inventor with that exact link it will not work.

(I am trying to access my own account and get the raw html code from it. I then plan on parsing it for data.)

[StudentWinna]

Also a piece of string is twice as long as half it's length. 

[SteveJG]

You are accessing  a private database.  The current issue is not scraping the web page, but getting access to the web page that has your information by using a password in the Web Component request

Setting to accept cookies is/was a good idea but did not work.

You may have to use 

text UriEncode(text text)

Encodes the given text value so that it can be used in a URL

and encode the url    .  Try that.   

It would be helpful to post the blocks you use (sans your passwords) so someone might comment on the possibility of a timing issue and see how you are using the Web1.GotText  block.

The naviance site might  require you to make one of the several PUT requests possible and sign in using a stepped approach.     We can not see your data (or even want to)  so you have to do the debugging.  Do you know how to use the AppInventor  DoIt debugger?  You might start by checking what response you get using the GotText block and it might provide clues.  The exact error messages , if you get any might be important clues.

[SteveJG]

also this might help  https://puravidaapps.com/filedownload.php  

or perhaps https://puravidaapps.com/postfile.php    .  These are probably not exactly what you need to do.  Taifun's web pages have some neat stuff.

[StudentWinna]

I tried UriEncode and it gave me an error message that the specified URL is not valid(When testing the prototype on my phone). I may be using it incorrectly. I have never heard of the PUT requests but that sounds very interesting. I am fairly new to appinventor and I have never used it's debugger. For the html returned by the gotText it led me to that page. I noticed it said if your school is not subscribed so the clue I got from it was that it did not recognize my hsid (highschool id)  and so I specified it. That did not fix the connection error on the web component but that was when it starting working on my browser with no cookies.  

I noticed in your downloading link the html variables were encoded and put in different blocks separate from the url or something along those lines. How would I go about doing that with this? I also noticed Base64 encoding the username and password. http://ostermiller.org/calc/encode.html is there anything I would use this for? The download from a restricted area is probably similar to taking html but i'm not sure if you go about it the exact same way. 

For the postfile link I did notice the concatenation which would be useful but in the blocks the site showed I didn't see anything specific to what I am doing about my connection error. Not sure what to look for. Thoughts?

Thanks again for the help.

 

[StudentWinna]

Just to clarify without using the UriEncode in the aaa.png blocks above, the html returned was of that page.png that is also above. 

[Taifun]

see the documentation

http://ai2.appinventor.mit.edu/reference/components/connectivity.html#Web

what is the purpose of the Delete method there?

it does not look like this makes sense to use that block while connecting, does it?

see also the purpose of the UriEncode method

text UriEncode(text text)

Encodes the given text value so that it can be used in a URL.

 

which means, it does not make much sense to use that block to encode a complete URL...

is there any API documentation of the page you like to access?

Taifun

[SteveJG]

this implies that you need to access the page  connection....myschoolnickname     where the school knows that but you do not perhaps. but you may already have the credentials necessary but 

are not using them properly.  

The advice here is may be what you need  (File download from restricted area)  https://puravidaapps.com/filedownload.php    or some variation.

Try using  the Web1. RequestHeaders block ... What you put there might need to be encoded using the  'encoder' here    http://ostermiller.org/calc/encode.html    .as shown in the example 

It is possible  the password and user might not have to be encoded so you should try both ways.  Note the List blocks used in the example ... it indicates the user and password might need to submitted as part of a list.

You do not do that in the blocks you posted.  You possibly would post the user name separately from the passcode

Again, just guessing.  

Regarding  DoIt ... look here  for DoIt  https://puravidaapps.com/learn.php  

[StudentWinna]

I got rid of the delete method (and the encoder part for now), it's original purpose was to retrieve the necessary cookies. I am not usre about API but I am now trying to pack and send a json to gain access to the url. 

The picture is of the java script that occurs when you login. 

I believe I need to pack and send a json string looking something like this: {type:"post",url:"https://connection.naviance.com/family-connection/auth/login/authenticate",password:"XXXX", is_ajax: !0,datatype: json, success (everything on the json after success although not sure what this part means).

Here is the link to the original source code.

https://connection.naviance.com/family-connection/auth/login/?hsid=winnacunneths

Also the json that is on the png is in the source code as this: /resources-fc/javascript/login/login.js

How would I pack this json and send it to the Url so that it is run? I could use help with the specific syntax if you guys know how. Not sure which blocks to use to do this or how.

Thanks!   

[StudentWinna]

I also found it very interesting how the hsid had to be specified on the browser of my computer and then it worked. But it is not specified in the java script. 

[StudentWinna]

There is an API for naviance, although I was on the right track with the json. 

This is it. It all adds up. 

• navigate to https://connection.naviance.com/family-connection/auth/login?hsid=<high school id>

$.ajax({ url: '/family-connection/auth/login/authenticate', type: 'post', dataType: 'json', data: { username: '<username>', password: '<password>', is_ajax: true }, success: function(d) { console.log(d); }});

• cookie sess (Session ID) set by Naviance even if not present
• navigate to d.url (expected /main if correct or /auth/login if not)

Which blocks would I use to access the html of the page in app inventor?