metadata

[pbass]

HI all--

I've been working on and off for a few months trying to get a simple
script written to do all the data imports and image transforms done to
get multiple RETS feeds into a single data store (since the broker I'm
working for has told me he wants to have five feeds coming into his
system eventually).

One thing that's been hindering me is that I only have access to a
single feed at the moment, so I can't really test my scripts against
anything but that one feed (and the NAR sample feed, though that's of
limited use!)

Would anyone be willing to send me metadata files for the servers
they're using? I don't need real data, just an idea of how different
the field names and types are across the different RETS systems.

If this somehow violates the NDAs signed to get access to the feeds,
then I apologize and forget I asked!

Thanks for any help.

[v1nce]

I was also wondering how a developer could test against different RETS
vendors? Do any of the MLS's provide test account access?

pbass - did you get anywhere with regards to your original request?

[Troy Davisson]

I know of no vendors who will provide test accounts.  CRT's demo RETS server is the only server I know about where the credentials are freely given out.

[tim]

Hi Vince-- no, I got nothing from the list.

I have 4-5 clients and beta testers who provided me metadata, but if I expand my offering then I'll have to do them on the fly, since you can't prepare for a new MLS by just downloading the metadata info. Not sure why the data is so under wraps, but I guess I don't understand this particular market too well at this point-- I don't see what harm making the metadata public could do unless they just want to limit access to vendors-- as usual, it seems like the policy is contrary to the benefit of the agents. Even some of the really big guys in the space have a limited number of MLS's they serve, probably for this very reason.

I'm interested in what the more experienced guys on the list might have to say however!

[Troy Davisson]

As someone who works for one, I can assure you it's not a MLS software vendor conspiracy but take from that what you will.  I can answer just about any questions you'd have about metadata but giving you a RETS account to each of our MLS customers means that you have access to their data and that's just not something I can do.  The field formats in every case are defined by our customers (the local associations or MLS's which are made up of or voted in by agent members) so it's something that belongs to them.

As I said, I can help to provide more specific answers if you have specific questions.  To summarize in a very short way:

Any time you have fields that apply to each listing, you'll find those as part of the main listing record (from the Property resource).  This would include the basic fields like address information or price but also often includes fields such as the features that could have multiple selections.  These are sometimes combined or sometimes split up as a bunch of boolean fields.  For example, "Fireplace" might be a general feature category.  You might have 1 field for "Fireplace" which contains a comma-delimited list of values if multiple were selected or you might have a separate field for each fireplace type with a true/false flag showing if it was selected.

When you get into getting room dimension or unit information, those are broken out (at least in our system) in a separate RETS resource (though we do provide limited room-related detail directly along with the listing).  Because 1 property could have 2 rooms and another could have 20, it's nearly impossible to come up with a sane way to provide that information in fixed column format bolted onto each listing.  Instead, you download 1 record per room per listing from a different section (so you get it in row format instead of fields).

[tim]

Hi Troy-- sorry if that seemed like I was suggesting a conspiracy, I wasn't. I appreciate your response, however. I totally understand the need of MLS systems to protect their proprietary data, though I have some reservations about the utility of it all. And after your explanation, I understand how the complexity of the lookups and all the variety of the data can make it difficult to provide that data easily. But since that data is structured and created by the vendor for the MLS during the intial setup, it seems kinda trivial to provide a "dummy" record of the main resource types for developer use, at the option of the MLS, obviously. For large MLS systems with thousands of agents and potentially dozens of vendors serving them, this could only lead to better services and products for the members of the MLS.

Just my two cents, and obviously the system is what it is, and I can deal with it.

[Troy Davisson]

All good points, although the MLS defines the data structure.  It's up to our system to adapt to their requirements.

[tim]

Understood, and thanks for the huge amounts of help you provide through your advice and the phRETS library-- your efforts help way more agents get quality service than they could ever know.

[Troy Davisson]

RETS covers 2 main technical levels: 1) transport of data, and 2) the data

I'd argue that RETS handles the transport portion pretty well.  There is a big push now to work on standardizing the data so that you get common field names and data types regardless of which RETS server you connect to.  It's a pretty big challenge trying to normalize 800+ unique field formats from each MLS into something that could work well for everyone but it's thought to be a much needed task so it's being taken on.

Hopefully the structure of RETS metadata will be much less important in the coming years as servers get into providing more common data payloads.  But it's a process.

[tim]

I agree. I've been following some of the discussion through NAR, and it looks like the wheels are slowly starting to turn in the right direction, not only on the RETS issues but on the realization that the models of data security and access that have worked for 30 years have got to change to meet the needs of the modern world.

As you say, it's a process, but it's encouraging that it happening.