Strategies for implementation: How do you use RETS?

[Rus Miller]

I'm puzzling through the best way to gather data from RETS and display
it on our site in a searchable format. I'd love to hear how you use
RETS.

Do you do a live call for each page load based on search parameters
provided by the user?

Do you use a cron job to store RETS data in your local database? If
so, do you also store the images?

How do you efficiently add to and update your local database? Do you
pull all the properties from RETS, in essence mirroring the RETS DB?

Any other thoughts? I'm not looking for code help, just stories of
what works for you. Thanks for your time.

[Peter Carr]

- Live call would be very slow :)

- Store it locally, cron updating it nightly, there is a timestamp field on images, and you can use that to see if you need to update your images or not.

- You really can pull what you need, after looking at the metadata decide if you need everything, or if you only need specific fields. Then add/update at your discretion!

- Plan it all the way through before you start :D

Hope this helps!
-Pete

[Rus Miller]

Thanks Pete.

I thought as much about live calls, but wanted to see if anyone was
using this method and if it was working. It seemed to me that
effectively duplicating the MLSs database to my local server was maybe
not that efficient, unless the reason to do so is the slow speed at
which the data is returned. It does seem very slow but I wasn't sure
if that was just my experience.

The way I'm planning on doing it:

1. Foreach Property Classes as Class
a. Retrieve all properties for each class.
b. Foreach Properties as Property
i. Retrieve the photos for each property.
ii. Foreach Photos as Photo
a. Store the photo in the database as a BLOB field, since some
of our MLSs appear not to offer image URLs.
[This seems very resource-heavy. Should I store the binary
image data on the server as a JPG
and record the path to the image instead?]
iii. Before exiting the secondary loop, store the property in an
array for transfer to the database after the primary loop completes.
[Is there a more efficient way to do this? Should I just
check for the property's existence and modification
date and add/update the database during the loop instead of
waiting until after?]

I apologize for asking what may be elementary questions about
efficiency, but I've never had to deal with this much data before and
want to do it right.

[tim]

Hi Rus, I'm using this basic flow, but I'm using Amazon S3 with CloudFront CDN to push the images to, so it's cheap, redundant and fast. (http://aws.amazon.com/cloudfront/)

Storing them as blobs in your DB will eat up a lot of space pretty quick depending on how many feeds you're bringing in.

[Troy Davisson]

As someone who's tried storing MLS photos inside of a database, don't do it.  Just... don't.  :)

The 2 most common ways are to either 1) store them locally on the server with a database record that points to it, or 2) store them using a service such as Amazon S3 and create a local database record that points to it.

Also, if you change your looping around slightly to break out the photo retrieval from the data retrieval, you can do something like I described here: http://groups.google.com/group/phrets/msg/d16570387c3a2cb0

[bautes]

I used some kind of Rus strategy.
I created several tables with the following criteria:
Resource ID + "_" + Class Name.
So I get 5 tables for properties (Prop_RE_1, Prop_LL_2, ...), 5 for
ActiveAgents, etc..
Each one of them with his own field list.

For fields type "Lookup Multi" I created also a table with the p-key,
f-key.
And also a FieldsLookup table with the foreign long values.

> > On Oct 5, 10:23 am, Peter Carr <petro...@gmail.com> <petro...@gmail.com> wrote:
>
> >  - Live call would be very slow :)
>
> > - Store it locally, cron updating it nightly, there is a timestamp field on
> > images, and you can use that to see if you need to update your images or
> > not.
>
> > - You really can pull what you need, after looking at the metadata decide if
> > you need everything, or if you only need specific fields. Then add/update at
> > your discretion!
>
> > - Plan it all the way through before you start :D
>
> > Hope this helps!
> > -Pete
>