A Caché of Tips: HTTP

[Michael Cohen]

Caché provides many classes that provide useful APIs.  This week's Tip looks at the available %Net.HttpRequest and %XML.TextReader interfaces.

As discussed in my MultiValue eLearning topic and in my recent MV Tip on XML, the web is based on text files containing mark-up text to tell the browser how to format and display the text, where to find images, etc.

Browsers offer a View Source option to display the raw text including the mark-ups (instead of the formatted text).

One of Google's main activities is visiting every web site and page it can, and indexing the results.  It does not need to grab the text from a browser.  Rather, there are web APIs that will return the web page source directly.

Caché provides access to such an API via the %Net, package.

If a website provides some useful information you want to examine programmatically, there are a number of ways to capture and analyze the webpage.

For today's example, I will query Google's weather API (mentioned in my XML tip), and capture the current temperature in Boston (also provided is a 4-day forecast).

You could use similar code to visit a stock, auto or real estate site and, for example, determine the current price of an item of interest.

If I point my web browse to:

www.google.com/ig/api?weather=Boston+MA

the browser displays a nested display of the result.  But if I View Source, I see the unformatted text:

<?xml version="1.0"?><xml_api_reply version="1"><weather module_id="0" tab_id="0" mobile_row="0" mobile_zipped="1" row="0" section="0" ><forecast_information><city data="Boston, MA"/><postal_code data="Boston MA"/><latitude_e6 data=""/><longitude_e6 data=""/><forecast_date data="2012-04-23"/><current_date_time data="2012-04-23 19:17:00 +0000"/><unit_system data="US"/></forecast_information><current_conditions><condition data="Light rain"/><temp_f data="61"/><temp_c data="16"/><humidity data="Humidity: 82%"/><icon data="/ig/images/weather/mist.gif"/><wind_condition data="Wind: S at 15 mph"/></current_conditions><forecast_conditions><day_of_week data="Mon"/><low data="46"/><high data="70"/><icon data="/ig/images/weather/rain.gif"/><condition data="Rain"/></forecast_conditions><forecast_conditions><day_of_week data="Tue"/><low data="45"/><high data="57"/><icon data="/ig/images/weather/chance_of_rain.gif"/><condition data="Chance of Showers"/></forecast_conditions><forecast_conditions><day_of_week data="Wed"/><low data="37"/><high data="61"/><icon data="/ig/images/weather/chance_of_rain.gif"/><condition data="Chance of Showers"/></forecast_conditions><forecast_conditions><day_of_week data="Thu"/><low data="39"/><high data="66"/><icon data="/ig/images/weather/mostly_sunny.gif"/><condition data="Partly Sunny"/></forecast_conditions></weather></xml_api_reply>

It is one long string of text.

If I examine the original nested display, I see:

• ...
  
• <current_conditions>
  
•   <condition data="Light rain"/>
  
•   <temp_f data="61"/>
  
• ...
  

 

This is in the raw source text as:

...<current_conditions><condition data="Light rain"/><temp_f data="61"/>...

This is interpreted as XML element current_conditions contains element condition and element temp_f, condition contains attribute data with value "Light rain", and temp_f contains attribute data with value "61".

Once I have the string of text, I could use a simple text editor or expression parser to find this string and return the current temp.

But XML makes this more convenient.  There are many tools available, either from InterSystems or publicly, that can parse and/or transform XML or HTTP files.

For this Tip, I will use our %Net.HttpRequest class to grab this result from the internet and put it in a variable.  I will then use our %XML.TextReader class to walk thru the result, and find the current temp.

This MVBasic program gets the text, walks thru it, and prints the full result in an indented format to highlight that it understands the structure.

1. #PRAGMA ROUTINENAME=WEATHER
   
2. httprequest="%Net.HttpRequest"->%New()
   
3. httprequest->Server="www.google.com"
   
4. httprequest->SetParam("weather","Boston+MA")
   
5. httprequest->Get("/ig/api")
   
6. *Note:  the following line would output the unformatted result to the terminal
   
7. *httprequest->HttpResponse>OutputToDevice()
   
8. responseObject = httprequest->HttpResponse->Data
   
9. responseStream=responseObject->ReadLine()
   
10. *Create an instance of %XML.TextReader (returned by reference)
    
11. sc="%XML.TextReader"->ParseStream(responseStream,reader)
    
12. IF sc=1 THEN * success
    
13.     *Read all elements within the document
    
14.     level=-1
    
15.     LOOP WHILE (reader->Read())
    
16.         IF (reader->NodeType = "element") THEN
    
17.             level=level+1
    
18.                indent=SPACE(5*level)
    
19.             CRT indent:reader->NodeType:": ":reader->Name
    
20.             index=1
    
21.             level=level+1
    
22.                indent=SPACE(5*level)
    
23.                * Read all attributes of this element
    
24.             LOOP WHILE (reader->MoveToAttributeIndex(index))
    
25.                 CRT indent:reader->NodeType:": ":reader->Name:"=":reader->Value
    
26.                 index=index+1               
    
27.             REPEAT
    
28.             level=level-1
    
29.         END
    
30.                if (reader->NodeType = "endelement") THEN level=level-1
    
31.     REPEAT
    
32. END
    

 

This is the result.

• MV:WEATHER
  
• element: xml_api_reply
  
•      attribute: version=1
  
•      element: weather
  
•           attribute: module_id=0
  
•           attribute: tab_id=0
  
•           attribute: mobile_row=0
  
•           attribute: mobile_zipped=1
  
•           attribute: row=0
  
•           attribute: section=0
  
•           element: forecast_information
  
•                element: city
  
•                     attribute: data=Boston, MA
  
•                element: postal_code
  
•                     attribute: data=Boston+MA
  
•                element: latitude_e6
  
•                     attribute: data=
  
•                element: longitude_e6
  
•                     attribute: data=
  
•                element: forecast_date
  
•                     attribute: data=2012-04-23
  
•                element: current_date_time
  
•                     attribute: data=2012-04-24 00:54:00 +0000
  
•                element: unit_system
  
•                     attribute: data=US
  
•           element: current_conditions
  
•                element: condition
  
•                     attribute: data=Overcast
  
•                element: temp_f
  
•                     attribute: data=51
  
•                element: temp_c
  
•                     attribute: data=11
  
•                element: humidity
  
•                     attribute: data=Humidity: 80%
  
•                element: icon
  
•                     attribute: data=/ig/images/weather/cloudy.gif
  
•                element: wind_condition
  
•                     attribute: data=Wind: S at 13 mph
  
•           element: forecast_conditions
  
•                element: day_of_week
  
•                     attribute: data=Mon
  
•                element: low
  
•                     attribute: data=46
  
•                element: high
  
•                     attribute: data=70
  
•                element: icon
  
•                     attribute: data=/ig/images/weather/rain.gif
  
•                element: condition
  
•                     attribute: data=Rain
  
•           element: forecast_conditions
  
•                element: day_of_week
  
•                     attribute: data=Tue
  
•                element: low
  
•                     attribute: data=45
  
•                element: high
  
•                     attribute: data=57
  
•                element: icon
  
•                     attribute: data=/ig/images/weather/chance_of_rain.gif
  
•                element: condition
  
•                     attribute: data=Chance of Showers
  
•           element: forecast_conditions
  
•                element: day_of_week
  
•                     attribute: data=Wed
  
•                element: low
  
•                     attribute: data=37
  
•                element: high
  
•                     attribute: data=61
  
•                element: icon
  
•                     attribute: data=/ig/images/weather/chance_of_rain.gif
  
•                element: condition
  
•                     attribute: data=Chance of Showers
  
•           element: forecast_conditions
  
•                element: day_of_week
  
•                     attribute: data=Thu
  
•                element: low
  
•                     attribute: data=39
  
•                element: high
  
•                     attribute: data=66
  
•                element: icon
  
•                     attribute: data=/ig/images/weather/mostly_sunny.gif
  
•                element: condition
  
•                     attribute: data=Partly Sunny
  
• MV:
  

 

This MVBasic program gets the text as above, walks thru it, and reports just the current temp.

1. #PRAGMA ROUTINENAME=WEATHERF
   
2. httprequest="%Net.HttpRequest"->%New()
   
3. httprequest->Server="www.google.com"
   
4. httprequest->SetParam("weather","Boston+MA")
   
5. httprequest->Get("/ig/api")
   
6. responseObject = httprequest->HttpResponse->Data
   
7. responseStream=responseObject->ReadLine()
   
8. sc="%XML.TextReader"->ParseStream(responseStream,reader)
   
9. IF sc=1 THEN * success; *Read all elements within the document
   
10.     level=-1
    
11.     * look for element "current_conditions"
    
12.     LOOP WHILE (reader->Read())
    
13.         IF (reader->NodeType = "element") THEN
    
14.             IF (reader->Name = "current_conditions") THEN
    
15.                 CRT "found element current_conditions"
    
16.                 * look for sub-element "condition"
    
17.                 LOOP WHILE (reader->Read())
    
18.                     IF ((reader->NodeType = "element") AND (reader->Name="condition")) THEN
    
19.                         CRT "found sub-element condition"
    
20.                         * look for sub-sub-element "temp_f"
    
21.                         LOOP WHILE (reader->Read())
    
22.                             IF ((reader->NodeType = "element") AND (reader->Name="temp_f")) THEN
    
23.                                 CRT "found sub-sub-element temp_f"
    
24.                                 * look for attribute "data"
    
25.                                 IF (reader->MoveToAttributeName("data")) THEN
    
26.                                     CRT "current temp (F) is: ":reader->Value
    
27.                                     STOP
    
28.                                 END
    
29.                             END
    
30.                       REPEAT
    
31.                   END
    
32.               REPEAT
    
33.           END
    
34.       END
    
35.   REPEAT
    
36. END

Here is the result.

• MV:WEATHERF
  
• found element current_conditions
  
• found sub-element condition
  
• found sub-sub-element temp_f
  
• current temp (F) is: 61
  
• MV:
  

 

You could replace “Boston+MA” with your zip code (in double quotes) and display your data.

The big difference between general HTTP access and a web service, is that the web service describes the structure of the result (with a WSDL), while you are on your own with parsing general web pages.

This example captured the text as a variable. Methods are also provided to capture it as a string, stream or file.

Also, many methods are provided to navigate the resulting structure, depending on your needs and inclination.

The Caché Class Reference provides a number of other useful classes under %XML.

There is another useful example using the %XML.TextReader class in the docs under Caché XML Tutorial

Enjoy!

[JJ]

Very interesting...but would have been nice to see the code in COS,
using the class. Thx! JJ

[Dawn Wolthuis]

This is great! Thanks Michael. We have done a little bit with
consuming web services, but there is still a lot of mystery in it, so
this will be a good resource the next time we are working with it.

It is super to get such an example in mvbasic given that when we
convert from the cos examples in SAMPLES or documentation there are
usually still a few tough conversions for us (particularly macros
which should not be such a mystery for me at this point, but I still
haven't figured out how to convert them for mvbasic). It would be
great if examples like this could be provided in the SAMPLES namespace
in the future, but at least being able to find them here is terrific!

We will eventually be writing the server side of non-soap web services
too, providing an api as well as consuming them ourselves from Zen
pages. We have not yet dived into that, so maybe by the time we get
there, you guys might have provided examples of that too??

Keep up the good work with the tips. They are much appreciated. --dawn

> --
> You received this message because you are subscribed to the Google Groups
> "InterSystems: MV Community" group.
> To post to this group, send email to Cac...@googlegroups.com
> To unsubscribe from this group, send email to
> CacheMV-u...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/CacheMV?hl=en

--
Dawn M. Wolthuis

Take and give some delight today

[Lee Burstein]

Being the MultiValue group, we always provide code examples using MVBasic. There are many COS examples in the documentation and Samples.

On Apr 24, 2012, at 7:29 AM, JJ wrote:

Very interesting...but would have been nice to see the code in COS,
using the class.  Thx!  JJ

--
You received this message because you are subscribed to the Google Groups "InterSystems:  MV Community" group.
To post to this group, send email to Cac...@googlegroups.com
To unsubscribe from this group, send email to CacheMV-u...@googlegroups.com
For more options, visit this group at http://groups.google.com/group/CacheMV?hl=en

Lee H. Burstein

Technical Trainer

InterSystems
Office: 302-477-0180
Cell: 302-345-0810

[JJ]

Thanks! JJ

On Apr 24, 9:19 am, Lee Burstein <Lee.Burst...@intersystems.com>
wrote:

> Being the MultiValue group, we always provide code examples using MVBasic. There are many COS examples in the documentation and Samples.
>
> On Apr 24, 2012, at 7:29 AM, JJ wrote:
>
> Very interesting...but would have been nice to see the code in COS,
> using the class.  Thx!  JJ
>
> --
> You received this message because you are subscribed to the Google Groups "InterSystems:  MV Community" group.

> To post to this group, send email to Cac...@googlegroups.com<mailto:Cac...@googlegroups.com>
> To unsubscribe from this group, send email to CacheMV-u...@googlegroups.com<mailto:CacheMV-unsubscribe@googlegroup­s.com>
> For more options, visit this group athttp://groups.google.com/group/CacheMV?hl=en

[Roland]

Thanks for the tip.  Great work.  I have it working but I have another problem.  Is it possible to post xml getting a response?  I modified this code, looked at Cache documentation but could only come up with something like this:

xml="<ippay><TransactionType>PING</TransactionType><TerminalID>TESTTERMINAL</TerminalID></ippay>"
httprequest->Server="testgtwy.ippay.com"
httprequest->SetParam(xml)
*httprequest->ContentType = "text/xml"
*httprequest->Https = 1

httprequest->Send("Post","/ippay/") 

doesn't work.  I'm not sure where to put the xml code or if it even works.

In .NET C# it works like this:

            WebRequest request = WebRequest.Create("https://testgtwy.ippay.com/ippay/");            // Create a request using a URL that can receive a post. 

            request.Method = "POST";                                                                                         // Set the Method property of the request to POST.

            string xml =  "\n<ippay><TransactionType>PING</TransactionType><TerminalID>TESTTERMINAL</TerminalID></ippay>";            // Create POST data and convert it to a byte array.

            string postData = xml;

            byte[] byteArray = Encoding.UTF8.GetBytes(postData);

            request.ContentType = "text/xml";                                                                                 // Set the ContentType property of the WebRequest.

            request.ContentLength = byteArray.Length;                                                                    // Set the ContentLength property of the WebRequest.

            Stream dataStream = request.GetRequestStream();                                                       // Get the request stream.

            dataStream.Write(byteArray, 0, byteArray.Length);                                                        // Write the data to the request stream.

            dataStream.Close();                                                                                                    // Close the Stream object.

            WebResponse response = request.GetResponse();                                                        // Get the response.

            Console.WriteLine(((HttpWebResponse)response).StatusDescription);                            //Display the status.

            dataStream = response.GetResponseStream();                                                            // Get the stream containing content returned by the server.

            StreamReader reader = new StreamReader(dataStream);                                              // Open the stream using a StreamReader for easy access.

            string responseFromServer = reader.ReadToEnd();                                                       // Read the content.

            Console.WriteLine(responseFromServer);                                                                      // Display the content.

I think the XML data needs to be transformed into a stream but I'm not sure how this .NET C# code is transferred over to Cache code (ideally in MVBasic)

Any help would be appreciated.

[David Guest]

You need to use the EntityBody of %Net.HttpRequest

http://docs.intersystems.com/cache20122/csp/documatic/%25CSP.Documatic.cls?PAGE=CLASS&LIBRARY=DOCBOOK&CLASSNAME=%25Net.HttpRequest#EntityBody

Something like this should work.

xml="<ippay><TransactionType>PING</TransactionType><TerminalID>TESTTERMINAL</TerminalID></ippay>"

httprequest->Server="testgtwy.ippay.com"

httprequest->EntityBody->Write(xml)

*httprequest->ContentType = "text/xml"

*httprequest->Https = 1

httprequest->Send("Post","/ippay/") 

I have done something similar in the past but used the object to XML projections to generate a stream for the entity body.

David

[Roland]

David, thanks for the reply but I couldn't get it to work trying various forms of what you gave me. I did find a workaround using curl which I show below, but if anyone can get this to work it is better not to use an external command. These commands should return a valid response for anyone since this is a test server and a test ID.

*DOESN'T WORK in varying forms
httprequest="%Net.HttpRequest"->%New()

xml="<ippay><TransactionType>PING</TransactionType><TerminalID>TESTTERMINAL</TerminalID></ippay>"
httprequest->Server="testgtwy.ippay.com"
httprequest->EntityBody->Write(xml)

httprequest->ContentType = "text/xml"
*httprequest->Https = 1
httprequest->Send("Post","/ippay/")

stop


1 *WORKS by passing XML via STDIN (aka DATA command). Note the extra quotes needed in XML: double quotes need to be on inside.
  * I have curl.exe in the same directory as the namespace (on windows--should work on linux as well)

XML='"<ippay><TransactionType>PING</TransactionType><TerminalID>TESTTERMINAL</TerminalID></ippay>"'; *Whole XML surrounded by inner double quotes and outer single quotes
URL="https://testgtwy.ippay.com/ippay/"
DATA XML; EXECUTE "SH curl -k -d ":XML:" ":URL CAPTURING X; PRINT X
STOP


2; *WORKS by writing XML to file and then posting that file (file is in same directory here as curl.exe)
  * I have curl.exe in the same directory as the namespace (on windows--should work on linux as well)

XML="<ippay><TransactionType>PING</TransactionType><TerminalID>TESTTERMINAL</TerminalID></ippay>"; *XML surrounded only by double quotes
OPENPATH "c:\intersystems\trycache\mgr\national" TO FPATH
WRITE XML ON FPATH,"text.xml"
URL="https://testgtwy.ippay.com/ippay/"
EXECUTE "SH curl -k -...@text.xml ":URL CAPTURING X; PRINT X
STOP

Note that curl is free and is widely used.

[David Guest]

I think you are having problems because it is a SSL connection. You need to set the Port to 443 and you will also need a "SSL/TLS Configuration" setup from the management portal (just create one called HTTPS with all the default options). The following should then work.

     xml="<ippay><TransactionType>PING</TransactionType><TerminalID>TESTTERMINAL</TerminalID></ippay>"

 

     httprequest="%Net.HttpRequest"->%New()

 

     httprequest->Server = "testgtwy.ippay.com"

     httprequest->Port = "443"

     

     httprequest->ContentType = "text/xml"

 

     httprequest->Https = @TRUE

     httprequest->SSLConfiguration = "HTTPS"

             

     httprequest->EntityBody->Write(xml)

     httprequest->Post("/ippay/")

     

     PRINT httprequest->HttpResponse->Data->Read(1024)

David

[Roland]

That worked perfectly!  Thanks for the help, David!