Working with CREA RETS and need the download.php TODO code

1,334 views
Skip to first unread message

David Hollebone

unread,
Sep 30, 2014, 12:33:48 PM9/30/14
to phr...@googlegroups.com
I am new to the CREA RETS environment and not proficient in PHP - although I can hack things together and make it all work. With that proviso, I need to have a way to save the data generated by download.php. I'm not interested in the photos download, only the data download in COMPACT or CSV format. Can someone suggest a script that will complete the @TODO in the following code that will download the results as a file onto my local machine? Here's the code:

<?PHP

/* Script Variables */
// Lots of output, saves requests to a local file.
$debugMode = false; 
// Initially, you should set this to something like "-2 years". Once you have all day, change this to "-48 hours" or so to pull incremental data
$TimeBackPull = "-2 years";

/* RETS Variables */
require("PHRets_CREA.php");
$RETS = new PHRets();
$RETSUsername = "";
$RETSPassword = "";
$RETS->Connect($RETSURL, $RETSUsername, $RETSPassword);
$RETS->AddHeader("RETS-Version", "RETS/1.7.2");
$RETS->AddHeader('Accept', '/');
$RETS->SetParam('compression_enabled', true);
$RETS_PhotoSize = "LargePhoto";
$RETS_LimitPerQuery = 100;
if($debugMode /* DEBUG OUTPUT */)
{
//$RETS->SetParam("catch_last_response", true);
$RETS->SetParam("debug_file", "/var/web/CREA_Anthony.txt");
$RETS->SetParam("debug_mode", true);
}

function downloadPhotos($listingID)
{
global $RETS, $RETS_PhotoSize, $debugMode;
if(!$downloadPhotos)
{
if($debugMode) error_log("Not Downloading Photos");
return;
}

$photos = $RETS->GetObject("Property", $RETS_PhotoSize, $listingID, '*');
if(!is_array($photos))
{
if($debugMode) error_log("Cannot Locate Photos");
return;
}

if(count($photos) > 0)
{
$count = 0;
foreach($photos as $photo)
{
if(
(!isset($photo['Content-ID']) || !isset($photo['Object-ID']))
||
(is_null($photo['Content-ID']) || is_null($photo['Object-ID']))
||
($photo['Content-ID'] == 'null' || $photo['Object-ID'] == 'null')
)
{
continue;
}
$listing = $photo['Content-ID'];
$number = $photo['Object-ID'];
$destination = $listingID."_".$number.".jpg";
$photoData = $photo['Data'];
/* @TODO SAVE THIS PHOTO TO YOUR PHOTOS FOLDER
* Easiest option:
* file_put_contents($destination, $photoData);
*/
 
$count++;
}
if($debugMode)
error_log("Downloaded ".$count." Images For '".$listingID."'");
}
elseif($debugMode)
error_log("No Images For '".$listingID."'");
// For good measure.
if(isset($photos)) $photos = null;
if(isset($photo)) $photo = null;
}

/* NOTES
 * With CREA, You have to ask the RETS server for a list of IDs.
 * Once you have these IDs, you can query for 100 listings at a time
 * Example Procedure:
 * 1. Get IDs (500 Returned)
 * 2. Get Listing Data (1-100)
 * 3. Get Listing Data (101-200)
 * 4. (etc)
 * 5. (etc)
 * 6. Get Listing Data (401-500)
 *
 * Each time you get Listing Data, you want to save this data and then download it's images...
 */
 
error_log("-----GETTING ALL ID's-----");
$DBML = "(LastUpdated=" . date('Y-m-d', strtotime($TimeBackPull)) . ")";
$params = array("Limit" => 1, "Format" => "STANDARD-XML", "Count" => 1);
$results = $RETS->SearchQuery("Property", "Property", $DBML, $params);
$totalAvailable = $results["Count"];
error_log("-----".$totalAvailable." Found-----");
if(empty($totalAvailable) || $totalAvailable == 0)
error_log(print_r($RETS->GetLastServerResponse(), true));
for($i = 0; $i < ceil($totalAvailable / $RETS_LimitPerQuery); $i++)
{
$startOffset = $i*$RETS_LimitPerQuery;
error_log("-----Get IDs For ".$startOffset." to ".($startOffset + $RETS_LimitPerQuery).". Mem: ".round(memory_get_usage()/(1024*1024), 1)."MB-----");
$params = array("Limit" => $RETS_LimitPerQuery, "Format" => "STANDARD-XML", "Count" => 1, "Offset" => $startOffset);
$results = $RETS->SearchQuery("Property", "Property", $DBML, $params);
foreach($results["Properties"] as $listing)
{
$listingID = $listing["@attributes"]["ID"];
if($debugMode) error_log($listingID);
/* @TODO Handle $listing array. Save to Database? */
/* @TODO Uncomment this line to begin saving images. Refer to function at top of file */
//downloadPhotos($listingID);
}
}

$RETS->Disconnect();

/* This script, by default, will output something like this:

Connecting to RETS as '[YOUR RETS USERNAME]'...
-----GETTING ALL ID's-----
-----81069 Found-----
-----Get IDs For 0 to 100. Mem: 0.7MB-----
-----Get IDs For 100 to 200. Mem: 3.7MB-----
-----Get IDs For 200 to 300. Mem: 4.4MB-----
-----Get IDs For 300 to 400. Mem: 4.9MB-----
-----Get IDs For 400 to 500. Mem: 3.4MB-----
*/

?>

download.php

Jason Graham

unread,
Oct 1, 2014, 10:20:00 AM10/1/14
to phr...@googlegroups.com
David,

I'm not a PHP programmer myself, so can't help much with the actual client coding, but typically RETS/DDF clients would replace that TODO section with add/update/delete calls directly to their database, instead of writing to a file and then having a separate process parse that file.

If you did prefer to use PHRets just to create this file though, you should be able to just loop through the array and use a PHP command like FILE_PUT_CONTENTS to append each record to a local file.

Jason Graham
Manager, Application Development
CREA

David Hollebone

unread,
Oct 1, 2014, 3:10:14 PM10/1/14
to phr...@googlegroups.com
Thanks Jason,

If you know of any example(s) of this functioning as a direct-to-mySQL database script I would love to have it!

David

Eliot Jones

unread,
Jun 11, 2015, 10:03:46 AM6/11/15
to phr...@googlegroups.com
I have a working example that is pumping data into my database here.  However I can't seem to get all of the listings in my initial download.  I'm not sure if it is a memory issue or if I need to configure the offset differently but if anyone can help I would be grateful.

Tom Ashworth

unread,
Jul 7, 2015, 11:24:36 AM7/7/15
to phr...@googlegroups.com

did you ever find your example code?  I would like this also, especially for version 2.  Hard to find that info anywhere...

Brady Edgar

unread,
Sep 24, 2015, 12:11:04 PM9/24/15
to PHRETS
Hey Eliot, did you ever fix the issue of not getting all the listings to download. I have code very alike to your example you provided with the same issue! I will get ssay 1950 results however only 1400 will actually download into my BD.

Thanks

Patrick Lyver

unread,
Oct 13, 2015, 9:55:52 PM10/13/15
to PHRETS
Hey Brady, 

I'm in the same boat. I changed the $TimeBackPull variable to -30 days and I got one day's worth of listings from 30 days previous.

I'm wondering how this works as the documentation is lean.

Have you been able to sort this?

Peter van Schaik

unread,
Mar 29, 2017, 7:56:09 AM3/29/17
to PHRETS
hello!  does anyone have a working sample code to connect to CREA DDF?

I have used the sample code from realtorlink.ca, but it keeps returning "No records found"

Is there a setting in the DDF Dashboard that I am missing?

David Hollebone

unread,
Mar 29, 2017, 12:30:54 PM3/29/17
to phr...@googlegroups.com

Here's the script that I pieced together and has been working for my CREA DDF. It generates a csv (getdata_Property.csv) file in my root directory, which I then retreive manually right after I have called this script (I was not able to automate the uploading of the csv file to mySQL database, so I do it manually):

 

<?php require_once "../Connections/crealogon.php";

 

$rets_modtimestamp_field = "LastUpdated";

 

// use http://retsmd.com to help determine the names of the classes you want to pull.

// these might be something like RE_1, RES, RESI, 1, etc.

$property_classes = array("Property");

 

// DateTime which is used to determine how far back to retrieve records.

// using a really old date so we can get everything

$previous_start_time = "1980-01-01T00:00:00";

 

//////////////////////////////

require_once("phrets.php");

 

// start rets connection

$rets = new phRETS;

 

echo "+ Connecting to {$rets_login_url} as {$rets_username}<br>\n";

$connect = $rets->Connect($rets_login_url, $rets_username, $rets_password);

 

if ($connect) {

        echo "  + Connected<br>\n";

}

else {

        echo "  + Not connected:<br>\n";

        print_r($rets->Error());

        exit;

}

 

foreach ($property_classes as $class) {

 

        echo "+ Property:{$class}<br>\n";

 

        $file_name = "getdata_{$class}.csv";

        $fh = fopen($file_name, "w+");

 

        $maxrows = true;

        $offset = 1;

        $limit = 100;

        $fields_order = array();

 

        while ($maxrows) {

 

                $query = "({$rets_modtimestamp_field}={$previous_start_time}+)";

 

                // run RETS search

                echo "   + Query: {$query}  Limit: {$limit}  Offset: {$offset}<br>\n";

                $search = $rets->SearchQuery("Property", $class, $query, array('Limit' => $limit, 'Offset' => $offset, 'Format' => 'COMPACT-DECODED', 'Count' => 1));

 

                if ($rets->NumRows() > 0) {

 

                        if ($offset == 1) {

                                // print filename headers as first line

                                $fields_order = $rets->SearchGetFields($search);

                                fputcsv($fh, $fields_order, $delimiter = ',', $enclosure = '"');

                        }

 

                        // process results

                        while ($record = $rets->FetchRow($search)) {

                                $this_record = array();

                                foreach ($fields_order as $fo) {

                                        $this_record[] = $record[$fo];

                                }

                                fputcsv($fh, $this_record, $delimiter = ',', $enclosure = '"');

                        }

 

                        $offset = ($offset + $rets->NumRows());

 

                }

 

                $maxrows = $rets->IsMaxrowsReached();

                echo "    + Total found: {$rets->TotalRecordsFound()}<br>\n";

 

                $rets->FreeResult($search);

        }

 

        fclose($fh);

 

        echo "  - done<br>\n";

 

}

echo $search;

echo "+ Disconnecting<br>\n";

$rets->Disconnect();

--
You received this message because you are subscribed to a topic in the Google Groups "PHRETS" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/phrets/JVsjwmgrOjs/unsubscribe.
To unsubscribe from this group and all its topics, send an email to phrets+un...@googlegroups.com.
To post to this group, send email to phr...@googlegroups.com.
Visit this group at https://groups.google.com/group/phrets.
For more options, visit https://groups.google.com/d/optout.

Gifford Boake

unread,
Mar 29, 2017, 12:48:26 PM3/29/17
to PHRETS
If you are using the sample data login credentials instead of your own personal login credentials (for the feed that your created in realtorlink.ca) then the data is very old. You must go back to early 2013 to get any data and there are only about 960 records in total.

Also CREA does things quite differently then other RETS. IF you query for (ID=*) you will get every listing in the database but only its ListingKey and its ModificaitonTimeStamp properties. You can then use that master list to retrieve the full listing details to populate the database for the first time. After that you will want to have an automated query to retrieve all new and updated listings since your last request (you should be doing this every 24hours) You should not do what David is doing and re-downloading every listing with their full details every day. 

Peter van Schaik

unread,
Mar 29, 2017, 1:17:16 PM3/29/17
to PHRETS
Thank you!
Can you also post the contents of your crealogon.php

Gifford Boake

unread,
Mar 29, 2017, 1:29:17 PM3/29/17
to PHRETS
Sorry if you were asking me (disregard if you were talking to David) I have long ago stopped using Phrets (and PHP in general) and now use Node.js for most things.

Even still the login credentials will be the same (the format will probably be different) but this is what Im using.

var clientSettings = {
    username'CXLHfDVrziCfvwgCuL8nUahC',
    password'mFqMsCSPdnb5WO1gpEEtDCHH',
    version: 'RETS/1.7.2',
    userAgent: 'RETS node-client/4.x', //This will be different for Phrets!!!!
    method: 'GET' 
};

OR: If you are using the live feed

var clientSettings = {
    loginUrl: 'http://data.crea.ca/Login.svc/Login ',
    username: 'YOUR USERNAMEFROM realtorlink.ca',
    password: 'YOUR PASSWORD FROM realtorlink.ca',
    version: 'RETS/1.7.2',
    userAgent: 'RETS node-client/4.x',
    method: 'GET' 
};
Message has been deleted

Gifford Boake

unread,
Feb 25, 2019, 7:43:40 AM2/25/19
to phr...@googlegroups.com
You don't. You have to be a realtor or directly hired by a realtor in which case you can only provide the data feed on their site.

On Sun, Feb 24, 2019, 11:33 PM Manraj Jaswal <jaswal...@gmail.com> wrote:
How are you guy getting access to Realtor link assuming your not Realtors?  The CREA sample feed sucks.
--
You received this message because you are subscribed to the Google Groups "PHRETS" group.
To unsubscribe from this group and stop receiving emails from it, send an email to phrets+un...@googlegroups.com.
Reply all
Reply to author
Forward
0 new messages