record deletion using HUD HMIS XML 2.8
Eric Jahn
something lacking in the current HMIS XML 2.7 standard. I've been
considering a few different ways of implementing this, and I wanted to
get your feedback as to the best approach. The approaches so far
include:
1) A schema which allows for a deletion request for an entire
record with a particular timestamp/value for one of ClientHistorical,
ServiceEvent, Need, ROI, etc.. This approach wouldn't be very
granular, but it would be fairly simple I think;
2) Add a new submission type of "delete" to the existing options of add/retry to the
SOAP envelope. Then, any records in the enveloped XML will be searched
for and deleted from the receiving server/database.
3) Add an add/update/delete attribute to every element in the XML Schema. This
would be an extended type, most likely.
What do you all think? If I'm missing another way useful of implementing a record deletion
request, please let me know via this forum. Thank you!
--
Eric Jahn
Alexandria Consulting LLC
3126 8th Ave. N
St. Petersburg, FL 33713
941.321.1466
David Talbot
data warehousing. IE, you give me all your data as of now, and I combine it
with a few other people, we cleanse it and report on it. It's not designed
to be a transactional integration format. If we want to move it into
transactional integration territory we really need to address more than
deletes.
At present, our current usage scenarios around HUD XML are based on file
exports/imports for the purpose of data warehousing. For consolidated
reporting endeavors, doing DIFFs never works in achieving GOOD accuracy.
Thus, INSERT/DELETE/UPDATE request based data warehouses over time slowly
become farther and farther from accurate until it's almost a joke (the
tinyst difference in how a single contributor handles something overtime
builds a crack that turns into a fissure which eventually collapses the
whole building and it's never possible to fully iron it out to "perfect"),
then all the parties agree to do another whole dump/rebuild of the
consolidated reporting warehouse. For data warehousing, its just easier to
do a rebuild from a full dump each time and store your data cleansing rules
from the last time you did it.
For real-time transactional integration (which is better suited to
situations where operational data sharing is desired over consolidated
reporting), which seems to be what you're looking to do, at present we are
using a proprietary web service API. I would, of course, welcome a standard
here for transactional integration and to make the standard easier to
implement I'd of course like it to resemble at least in parts the current
HUD XML spec while more properly facilitating real time integration. With
real-time integration, I would recommend we do a series of CRUD APIs
covering the specific elements and add <xxxIdentifier></xxxIdentifier> to
the portions that don't currently have them. Internally in our system (and I
would assume others have already taken this approach or will once they start
doing real time) we keep a map of SourceDatabaseID + ClientIdentifier = My
System's native client id. Thus when you send me client 123456 at some point
in the future, I know you're talking about my clientid 99999.
I would hope the API would work something like this (web service based):
StatusObject CreateClient(MySourceDatabaseID, "<Client>HUD XML Client.
Either inclusive of sub elements client ClientHistorical, etc or without
them.</Client>");
StatusObject CreateService(MySourceDatabaseID, MyClientIdentifier,
"<ServiceEvent>...</ServiceEvent>");
StatusObject DeleteService(MySourceDatabaseID, MyClientIdentifier,
MyServiceIdentifier);
StatusObject would contain success/failure etc.
Basically, we take the main data structures as they are presently structured
but we allow the main portions of the entity to be decoupled from expressing
the sub-elements. We assume the destination system can handle correlating
the record identifiers to whatever their native record identifiers are.
Thoughts?
-David Talbot
Eric Jahn
Good points, as always. Responses below, inline.
On Wed, 2007-09-12 at 16:33 -0500, David Talbot wrote:
...
> With real-time integration, I would recommend we do a series of CRUD APIs
> covering the specific elements
So, as "borrowed" from the CARS WSDL, we currently have the following
operations:
reqSecToken
getSubmissionInfo (for feedback on a recent submission)
postSubmission (for sending XML attachments)
postSubmissionInline (for sending inline XML)
CARS apparently left off "Update", I believe, since they just wanted
agencies to submit full records for data warehousing; not operational
use.
postSubmission gives us a broad-brushed "Create", and any of the data in
the attached XML document would be added to the database. For "Read" I
suppose we should add an operation to getServiceInfo or getClientInfo
and one would provide the exact ServiceID, NeedID, PersonID, etc.
necessary to retrieve the record (which means we have to add ServiceID
and other IDs to the HMIS Schema). Same goes for "Delete" and "Get"
operations, which we'd need to define. "Update" operations probably
need to transmit the ID of the element to be updated, along with the
complete schema-defined type data for an update, so a subset of a post
submission. For now, I think it would be okay to restrict
postSubmission to having to post for "creation" fully valid HMIS Schema
docs.
Of course, the processing service would have to verify and check
everything before committing the transaction, but it seems like a good
idea for the sender'sto explicitly declare intent.
> and add <xxxIdentifier></xxxIdentifier> to
> the portions that don't currently have them.
Yes, I agree, so I think the next task with the Schema is to add unique
IDs to the major elements. I'll add those now. Ideally, it would be
nice to have IDs for most every element to allow for granular CRUD
operations, but this would be too messy (especially at this standard's
state of maturity). So, I propose to
> Internally in our system (and I
> would assume others have already taken this approach or will once they start
> doing real time) we keep a map of SourceDatabaseID + ClientIdentifier = My
> System's native client id. Thus when you send me client 123456 at some point
> in the future, I know you're talking about my clientid 99999.
Makes sense.
> I would hope the API would work something like this (web service based):
> StatusObject CreateClient(MySourceDatabaseID, "<Client>HUD XML Client.
> Either inclusive of sub elements client ClientHistorical, etc or without
> them.</Client>");
> StatusObject CreateService(MySourceDatabaseID, MyClientIdentifier,
> "<ServiceEvent>...</ServiceEvent>");
> StatusObject DeleteService(MySourceDatabaseID, MyClientIdentifier,
> MyServiceIdentifier);
>
> StatusObject would contain success/failure etc.
Makes sense also. Right now the operations that exist are defined to
have return message parts with error codes. But the new operations
should have "Get/Read" return XML. "Post/Create" and "Update" should
return Schema validation output and error codes. "Delete" should return
error codes.
> Basically, we take the main data structures as they are presently structured
> but we allow the main portions of the entity to be decoupled from expressing
> the sub-elements. We assume the destination system can handle correlating
> the record identifiers to whatever their native record identifiers are.
Agreed. And it would be nice to have a test server that stores SOAP/XML
from authorized testing communities/vendors, and provides error codes.
Any comments/feedback from the rest of the the group? Hopefully, I'll
see some of you in Minneapolis! -Eric
David Talbot
Clayton Flom (joined this listserv recently) joined or organization a while
back and has taken over the data integration capabilities of our product and
is hard at work on implementing CARS at the moment and will be handling
upgrading our HUD XML import/export to 3.1 (or whatever the version number
is that supports identifiers on everything :), we're temporarily paused on
2.7 and will probably jump from 2.7 to 3.1) and the eventual standard around
data transmission.
It would certainly save us implementation time if we just adopted the CARS
WSDL as close as possible. Not necessarily the "superior" standard for doing
it, but the cheapskate in me likes to get support for a second standard with
little additional work :)
As for Minneapolis, I won't be there unfortunately. At the moment,
ClientTrack has me focused on growing the technology team to accommodate our
customer base. So less trade shows for me for the foreseeable future which
I'll miss. I like getting to know folks out there working in this space and
hearing folks talk about their triumphs and challenges and I'm going to miss
out on some of it (at least for a while :( ).
-David Talbot
-----Original Message-----
From: HMIS_Data_...@googlegroups.com
[mailto:HMIS_Data_...@googlegroups.com] On Behalf Of Eric Jahn