I can't get PHPRets to work!!!
Terry Johnson
Joseph R. B. Taylor
On March 31 of this year the MLS Board I serve data from shut off
IDX and I didn't find out until about 3 weeks later.....
So.... I came back, installed Troy's RETS code and crossed my fingers.
Using the retsmd site and my credentials I was able to connect to
the server and that was good so I knew HIS stuff was working. The
problem was me. That part took a little while to grasp.
I didn't know how queries etc really worked and without understanding that, you're not gonna be able to get and use results from the server.
The flexmls site has a section on the query language/syntax. (the site is horrible it has some good code exmaples) The first key is knowing how to make a simple query that will actually return a result. If you can't do that well....
I fiddled with the phrets/index.php file until I got an example
that would return something. Start simple.
Once you CAN get a result from your server there's a couple things to think about.
Here's the contents of that old index.php file I left all my debugging comments in there so maybe you can your this to get a step further:
<?php
date_default_timezone_set('America/New_York');
require_once("vendor/autoload.php");
$log = new \Monolog\Logger('PHRETS');
$log->pushHandler(new
\Monolog\Handler\StreamHandler('php://stdout',
\Monolog\Logger::DEBUG));
$config = new \PHRETS\Configuration;
$config->setLoginUrl('http://capemay.rets.fnismls.com/rets/fnisrets.aspx/CAPEMAY/login?rets-version=rets/1.5')
->setUsername('{USERNAME}')
->setPassword('{PASSWORD}')
->setRetsVersion('1.5');
$rets = new \PHRETS\Session($config);
$rets->setLogger($log);
$connect = $rets->Login();
$system = $rets->GetSystemMetadata();
//echo '<pre>';
//print_r($system);
$resources = $system->getResources();
$classes = $resources->first()->getClasses();
//echo '<pre>';
//print_r($classes);
//exit;
$classes = $rets->GetClassesMetadata('Property');
//echo '<pre>';
//print_r($classes->first());
//exit;
$objects = $rets->GetObject('Property', 'Photo',
'00-1669', '*', 1);
//echo '<pre>';
//print_r($objects);
//exit;
$fields = $rets->GetTableMetadata('Property', 'RE_1');
echo '<pre>';
foreach ($fields as $field) {
//echo $field->getSystemName() . "<br>";
/* echo $field->getStandardName() . "\r\n";
echo $field->getDescription() . "\r\n";
echo $field->getType() . "\r\n";
echo $field->getLength() . "\r\n";
echo $field->getLookup() . "\r\n";*/
}
//$results = $rets->Search('Property', 'RE_1',
'(field_L_UpdateDate>2016-05-13T00:00:00)', ['Limit' => 3,
'Select' => 'L_ListingID']);
//$results = $rets->Search('Property', 'LD_2',
'(L_UpdateDate=2014-05-12T00:00:00+)', ['Limit' => 1,
'Offset' => 255]);
$results = $rets->Search('Media', 'CT_5',
'(L_UpdateDate=2014-05-12T00:00:00+),(L_ListingID=99706)');
echo '<pre>';
//file_put_contents('data/Property_' . $pc . '.csv',
$results->toCSV());
print_r($results->toArray());
exit;
$listings = $results->toArray();
foreach ($listings as $listing) {
print_r($listing);
echo '<hr>';
}
exit;
Chances are copying and pasting this won't work - I've got the
field names MY MLS uses in here, like RE_1 and field_L_UpdateDate
etc - all that stuff you need to discover using the retsmd site.
1. What to do with results? Store in database? Just hit the actual RETS server with every request?
I don't recommend using their server directly. My MLS has like 400 fields in each result record - and it kills my server to try and process that. I opted to stored into a mysql database.
Tables:
active_listings = all active records for use on client sites
sold_listings = same but sold data
photos = all active and sold photos
listings_raw = everything from the rets server
photos_raw = everything from the rets server
So what I do is run a "daily job" every couple hours that gets all the new stuff from today or so and dump that into the listings_raw table - then the same for photos into the photos_raw table.
Then I dedup that data using the MLS and updated_timestamp so I have the newest stuff only. Then I copy that over the active_listings and photos table.
Having the separate tables feeding client sites lets to do stuff
behind the scenes without causing them downtime - even purging the
entire database and downloading it over again which I've had to do
a bunch of times....
2. Strategy to get all the data the server has stored.
You can't just download the whole thing at once....At least not with PHP - it'll die. So you need to chunk up all the data as you import import it. Some people do with this limits and offsets. I decided to just do one day at a time starting on Jan 1, 2015 and moving forward from there. Yes, that's a bunch of days.
Once I had a good PHP script for pulling a series from records
from one date to the next I built a little front end application
with angular that would build a set of dates from then til now,
one day at a time, then slowly go through and store the records
one day at a time, not starting the next day til the previous one
was complete until I rebuilt my entire database.
First version took all night. Latest version takes about 30 minutes. Plan on coding and improving your stuff for a while.
After a while I got smarter and starting just querying the 60 or so fields I'd actually need which sped up working with the rets server TREMENDOUSLY.
3. Integrate the data with client's website(s)
Since I already had a bunch of client code utilizing IDX data
this was pretty painless for me.
4. Keep your data up to date and clean.
Here's an example of my daily routine:
In my controller, I have this function:
public function daily_import()
{
echo 'Running daily import
job<br>';
$start_date = date('Y-m-d');
$end_date = date('Y-m-d', strtotime("+1
day"));
//$start_date = '2016-05-20';
//$end_date = '2016-05-21';
$rets = self::rets_login();
echo 'login finished<br>';
self::import($rets, $start_date,
$end_date);
//echo 'now remove all
pending<Br>';
//self::purge_pendings($rets);
echo 'dedup next<br>';
self::dedup_data();
echo 'refresh next<br>';
self::refresh_client_data();
echo 'finished whole job';
}
You can see it's just calling a bunch of other functions from my controller:
private function rets_login()
{
date_default_timezone_set('America/New_York');
require_once("/home/sitesbyjoe2/rets.sitesbyjoe.com/phrets/vendor/autoload.php");
$log = new \Monolog\Logger('PHRETS');
$log->pushHandler(new
\Monolog\Handler\StreamHandler('php://stdout',
\Monolog\Logger::DEBUG));
$config = new \PHRETS\Configuration;
$config->setLoginUrl('http://capemay.rets.fnismls.com/rets/fnisrets.aspx/CAPEMAY/login?rets-version=rets/1.5')
->setUsername('')
->setPassword('')
->setRetsVersion('1.5');
$rets = new \PHRETS\Session($config);
$rets->setLogger($log);
$connect = $rets->Login();
return $rets;
}
This one is a doozy - but it lets you see how I manage all the fields coming in and then when storing them:
private function import($rets, $start_date, $end_date)
{
//echo 'Starting import for ' .
$start_date . ' to ' . $end_date . '<br>';
$date_range = '(L_UpdateDate=' .
$start_date . 'T00:00:00-' . $end_date . 'T00:00:00)';
$mls_numbers = array();
$total_photos = 0;
// loop through the property classes
$classes =
array('RE_1','LD_2','CI_3','MF_4','CT_5');
$all_fields =
'L_UpdateDate,L_ListingID,L_Class,L_StatusCatID,L_SaleRent,L_AskingPrice,L_AddressNumber,L_AddressStreet,L_Address2,L_City,L_State,L_Zip,LM_Dec_1,LR_remarks11,LR_remarks33,LA1_UserFirstName,LA1_UserLastName,LO1_OrganizationName,LM_char5_29,LM_Char10_21,LM_Char10_3,LM_Char10_5,L_PictureCount,L_ClosingDate,L_SoldPrice';
$re_fields =
'L_Keyword1,L_Keyword2,L_Keyword3,LFD_LOCATION_1,LFD_EXTERIOR_3,LFD_OUTSIDEFEATURES_4,LFD_PARKINGGARAGE_5,LFD_OTHERROOMS_6,LFD_INTERIORFEATURES_7,LFD_APPLIANCESINCLUDED_8,LFD_ALSOINCLUDED_9,LFD_BASEMENT_10,LFD_HEATING_11,LFD_COOLING_12,LFD_HOTWATER_13,LFD_WATER_14,LFD_SEWER_15,LM_Char10_19';
$ld_fields = 'LFD_LOCATION_18';
$ci_fields =
'LFD_LOCATION_24,LFD_EXTERIOR_25,LFD_EXTERIORFEATURES_26,LFD_PARKING_31,LFD_SPECIALTYROOMS_28,LFD_INTERIORFEATURES_27,LFD_HEATING_32,LFD_COOLING_33,LFD_HOTWATER_36,LFD_WATER_34,LFD_SEWER_35,LM_Char10_19';
$mf_fields =
'LFD_LOCATION_42,LFD_OUTSIDEFEATURES_44,LFD_ALSOINCLUDED_45,LFD_PARKING_47,LFD_HEATING_43,LFD_COOLING_51,LFD_HOTWATER_52,LFD_WATER_49,LFD_SEWER_50,LM_Char10_19';
$ct_fields =
'L_Keyword1,L_Keyword2,L_Keyword3,LFD_LOCATION_56,LFD_OTHERROOMS_58,LFD_UNITFEATURES_59,LFD_COMMONAREAFEATURES_60,LFD_APPLIANCESINCLUDED_61,LFD_ALSOINCLUDED_62,LFD_HEATING_63,LFD_WATER_65,LFD_SEWER_66,LFD_COOLING_67,LFD_HOTWATER_68,LFD_PARKING_69,LM_Char10_19';
foreach ($classes as $class)
{
switch ($class)
{
case 'LD_2':
$fields = $all_fields .
',' . $ld_fields;
break;
case 'CI_3':
$fields = $all_fields .
',' . $ci_fields;
break;
case 'MF_4':
$fields = $all_fields .
',' . $mf_fields;
break;
case 'CT_5':
$fields = $all_fields .
',' . $ct_fields;
break;
default:
$fields = $all_fields .
',' . $re_fields;
break;
}
//$results =
$rets->Search('Property', $class, $date_range, ['Limit' =>
2, 'Select' => $all_fields]);
$results =
$rets->Search('Property', $class, $date_range, ['Select'
=> $fields]);
$listings =
$results->toArray();
//echo count($listings) . '
found. ';
foreach ($listings as $listing)
{
// basic fields
$record = array(
'last_updated'
=> $listing['L_UpdateDate'],
'mls' =>
$listing['L_ListingID'],
'class' =>
$listing['L_Class'],
'status' =>
$listing['L_StatusCatID'],
'sale_rent' =>
$listing['L_SaleRent'],
'price' =>
$listing['L_AskingPrice'],
'address_1' =>
$listing['L_AddressNumber'],
'street' =>
$listing['L_AddressStreet'],
'address_2' =>
$listing['L_Address2'],
'city' =>
$listing['L_City'],
'state' =>
$listing['L_State'],
'zip' =>
$listing['L_Zip'],
'taxes' =>
$listing['LM_Dec_1'],
'addendum' =>
$listing['LR_remarks11'],
'remarks' =>
$listing['LR_remarks33'],
'agent_first_name' => $listing['LA1_UserFirstName'],
'agent_last_name'
=> $listing['LA1_UserLastName'],
'listing_office'
=> $listing['LO1_OrganizationName'],
'3rd_party_approval' => $listing['LM_char5_29'],
'bank_owned'
=> $listing['LM_Char10_21'],
'lot_number'
=> $listing['LM_Char10_3'],
'block_number'
=> $listing['LM_Char10_5'],
'num_photos'
=> $listing['L_PictureCount'],
'closing_date'
=> $listing['L_ClosingDate'],
'sold_price'
=> $listing['L_SoldPrice']
);
// class specific fields
if ($listing['L_Class']
== 'RESIDENTIAL')
{
$record['bedrooms'] = $listing['L_Keyword1'];
$record['full_baths'] = $listing['L_Keyword2'];
$record['half_baths'] = $listing['L_Keyword3'];
$record['location'] = $listing['LFD_LOCATION_1'];
$record['exterior'] = $listing['LFD_EXTERIOR_3'];
$record['outside_features'] = $listing['LFD_OUTSIDEFEATURES_4'];
$record['parking'] = $listing['LFD_PARKINGGARAGE_5'];
$record['other_rooms'] = $listing['LFD_OTHERROOMS_6'];
$record['interior_features'] =
$listing['LFD_INTERIORFEATURES_7'];
$record['appliances_included'] =
$listing['LFD_APPLIANCESINCLUDED_8'];
$record['also_included'] = $listing['LFD_ALSOINCLUDED_9'];
$record['basement'] = $listing['LFD_BASEMENT_10'];
$record['heating'] = $listing['LFD_HEATING_11'];
$record['cooling'] = $listing['LFD_COOLING_12'];
$record['hot_water'] = $listing['LFD_HOTWATER_13'];
$record['water']
= $listing['LFD_WATER_14'];
$record['sewer']
= $listing['LFD_SEWER_15'];
$record['new_construction'] = $listing['LM_Char10_19'];
}
if ($listing['L_Class']
== 'LOTS/LAND')
{
$record['location'] = $listing['LFD_LOCATION_18'];
}
if ($listing['L_Class']
== 'COMMERCIAL/INDUSTRIAL')
{
$record['location'] = $listing['LFD_LOCATION_24'];
$record['exterior'] = $listing['LFD_EXTERIOR_25'];
$record['outside_features'] =
$listing['LFD_EXTERIORFEATURES_26'];
$record['parking'] = $listing['LFD_PARKING_31'];
$record['other_rooms'] = $listing['LFD_SPECIALTYROOMS_28'];
$record['interior_features'] =
$listing['LFD_INTERIORFEATURES_27'];
$record['heating'] = $listing['LFD_HEATING_32'];
$record['cooling'] = $listing['LFD_COOLING_33'];
$record['hot_water'] = $listing['LFD_HOTWATER_36'];
$record['water']
= $listing['LFD_WATER_34'];
$record['sewer']
= $listing['LFD_SEWER_35'];
$record['new_construction'] = $listing['LM_Char10_19'];
}
if ($listing['L_Class']
== 'MULTI-FAMILY')
{
$record['location'] = $listing['LFD_LOCATION_42'];
$record['outside_features'] =
$listing['LFD_OUTSIDEFEATURES_44'];
$record['also_included'] = $listing['LFD_ALSOINCLUDED_45'];
$record['parking'] = $listing['LFD_PARKING_47'];
$record['heating'] = $listing['LFD_HEATING_43'];
$record['cooling'] = $listing['LFD_COOLING_51'];
$record['hot_water'] = $listing['LFD_HOTWATER_52'];
$record['water']
= $listing['LFD_WATER_49'];
$record['sewer']
= $listing['LFD_SEWER_50'];
$record['new_construction'] = $listing['LM_Char10_19'];
}
if ($listing['L_Class']
== 'CONDO/TOWNHOUSE')
{
$record['bedrooms'] = $listing['L_Keyword1'];
$record['full_baths'] = $listing['L_Keyword2'];
$record['half_baths'] = $listing['L_Keyword3'];
$record['location'] = $listing['LFD_LOCATION_56'];
$record['outside_features'] =
$listing['LFD_COMMONAREAFEATURES_60'];
$record['parking'] = $listing['LFD_PARKING_69'];
$record['other_rooms'] = $listing['LFD_OTHERROOMS_58'];
$record['interior_features'] = $listing['LFD_UNITFEATURES_59'];
$record['appliances_included'] =
$listing['LFD_APPLIANCESINCLUDED_61'];
$record['also_included'] = $listing['LFD_ALSOINCLUDED_62'];
$record['heating'] = $listing['LFD_HEATING_63'];
$record['cooling'] = $listing['LFD_COOLING_67'];
$record['hot_water'] = $listing['LFD_HOTWATER_68'];
$record['water']
= $listing['LFD_WATER_65'];
$record['sewer']
= $listing['LFD_SEWER_66'];
$record['new_construction'] = $listing['LM_Char10_19'];
}
//echo $record['mls'] .
'(' . $record['num_photos'] . '), ';
$this->db->insert('listings_raw', $record);
// any photos?
if
($listing['L_PictureCount'] > 0)
{
$mls_numbers[] =
$listing['L_ListingID'];
$mls_search =
'(L_ListingID=' . $listing['L_ListingID'] . ')';
$photo_results =
$rets->Search('Media', $class, $mls_search, ['Select' =>
'L_ListingID,MED_media_url,MED_sequence']);
$photos =
$photo_results->toArray();
//echo
count($photos) . ' photos found<br>';
$total_photos =
$total_photos + count($photos);
foreach ($photos
as $photo)
{
$photo_record = array(
'MLS_ID' => $photo['L_ListingID'],
'Image' => $photo['MED_media_url'],
'sequence' => $photo['MED_sequence']
);
$this->db->insert('photos_raw', $photo_record);
//print_r($photo_record);
}
}
}
}
echo count($listings) . ' listings saved.
';
echo $total_photos . ' photos saved.';
}
private function dedup_data()
{
echo '<hr>';
echo 'Deduping all raw data<br>';
// listings
$this->db->query("
DELETE FROM listings_raw
WHERE id NOT IN
(SELECT * FROM (SELECT MAX(id) FROM
listings_raw GROUP BY mls) x);
");
// photos
$this->db->query("
DELETE FROM photos_raw
WHERE id NOT IN
(SELECT * FROM (SELECT MAX(id) FROM
photos_raw GROUP BY Image) x);
");
echo 'All data deduped<br>';
}
private function refresh_client_data()
{
/*
delete from listings;
delete from images;
INSERT INTO listings SELECT * FROM
listing_listings where status = 'Active';
insert into images select * from listing_photos;
DELETE FROM listings WHERE id NOT IN (SELECT *
FROM (SELECT MAX(id) FROM listings GROUP BY mls) x);
DELETE FROM images WHERE id NOT IN (SELECT *
FROM (SELECT MAX(id) FROM images GROUP BY Image) x);
*/
// clear out all old data
$this->db->query("DELETE FROM
active_listings");
// repopulate from listings raw
$this->db->query("INSERT INTO
active_listings SELECT * FROM listings_raw where status =
'Active';");
$this->db->query("DELETE FROM photos");
$this->db->query("INSERT INTO photos
SELECT * FROM photos_raw;");
$this->db->query("DELETE FROM
sold_listings");
$this->db->query("INSERT INTO
sold_listings SELECT * FROM listings_raw where status =
'Sold';");
echo '<br>all client facing tables
updated<br>';
}
Hopefully this helps. So final thoughts:
Plan on this being a comprehensive system so it can eventually be hands off for you.
All MLSs are a little different. I got lucky and mine just gives your links to their CDN for photos - phew!
My code may not be helpful at all all.
Good luck!
Joseph R. B. Taylor Designer/Developer http://www.leasetool.com http://sitesbyjoe.com j...@sitesbyjoe.com 508-840-9657
--
You received this message because you are subscribed to the Google Groups "PHRETS" group.
To unsubscribe from this group and stop receiving emails from it, send an email to phrets+un...@googlegroups.com.
To post to this group, send email to phr...@googlegroups.com.
Visit this group at https://groups.google.com/group/phrets.
For more options, visit https://groups.google.com/d/optout.
Terry Johnson
David R
Terry Johnson
Gennadiy Kofman
http://stackoverflow.com/questions/13433946/how-to-check-if-curl-is-enabled-or-disabled
Sometimes shared hosting may have certain restrictions.
Joseph, it's great that you posted that. Other people are always looking for examples.
My works similar except I run more then one rets db and so my code is more abstract and database driven.
I also have raw data tables and a listing table that combines all my raw tables. You map the fields in the code, I actually have a table that maps them.
I think the hardest part of building the rets feed is the scrubbing. Meaning how to get rid of the listings that disappear from rets. Thats probably the slowest process for me and something I need to improve on. Because of that, I have it run nightly.