SITREP XML import/export
Nuwan Waidyanatha
we need to add a function to Import and export the SITREP data from
table as XML; possibly add an icon with the rest of the stack xml,
pdf, ....; unless it already exists.
My thinking
1) Each import or export iteration should use a single xml file; i.e.
multiple records in a single file
2) export filter is done using the attributes shown in the SITREP home
page table
3) Can we undo imports? i.e. be able to review the records first
before committing to tables?
4) Is there a generic API, possibly through S3XML that we could use?
5) validated with EDXL-SITREP v 1.0
thanks,
Nuwan
Fran Boon
> we need to add a function to Import and export the SITREP data from
> table as XML; possibly add an icon with the rest of the stack xml,
> pdf, ....; unless it already exists.
I don't think we want a generic one for 'XML' as this would be native
S3XML which most users aren't interested in.
For resources with specific XML formats then the icon could be a
generic 'XML' one but the returned data wouldn't be the native S3XML
but the transformed XML suitable for this usecase - i.e. EDXL for this
case.
This toolbar is defined in views/formats.html
You could add an:
{{if "xml" in response.s3.formats:}}
with
href="{{=r.url(representation=response.s3.formats.xml)}}
> My thinking
> 1) Each import or export iteration should use a single xml file; i.e.
> multiple records in a single file
This is standard :)
> 2) export filter is done using the attributes shown in the SITREP home
> page table
I'm not sure what you mean here - you're wanting to add dynamic
variables or filters to the export based on settings on the screen?
Filtering can be done by URL:
http://eden.sahanafoundation.org/wiki/S3XRC/RESTfulAPI/URLFormat
- S3Search has this integrated into it's formats toolbar (by
specifying the IDs to export)
We have extra variables on import forms, but not currently any export
examples...however the underlying XML engine can support extra
variables being passed to the stylesheet.
> 3) Can we undo imports? i.e. be able to review the records first
> before committing to tables?
We have a 2-phase importer, yes :)
This is the sdtanrd when doing /eden/module/resourcename/import.xml
> 4) Is there a generic API, possibly through S3XML that we could use?
Definitely the way to go, yes
To get the basic SITREP export working, add a 'sitrep' format to
static/formats with your import/export stylesheets (can start with
just 1)
- use an existing set as an example...the 'edrm' might be closest (EDXL-RM)
Then you can simply export via /eden/module/resourcename.sitrep
> 5) validated with EDXL-SITREP v 1.0
I don't believe we have any automatic DTD validation currently - this
is up to the manual testing of the stylesheet...
F
Nuwan Waidyanatha
be able to filter what they want.
What you say is that the user should be ideally using the "views"
module to filter the data, then generate the XML form it?
Thanks for the pointers.
Nuwan
> --
> You received this message because you are subscribed to the Google Groups "Sahana-Eden" group.
> To post to this group, send email to sahan...@googlegroups.com.
> To unsubscribe from this group, send email to sahana-eden...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/sahana-eden?hl=en.
>
>
Fran Boon
> 2) yes, filter a set of records by dates for example The user should
> be able to filter what they want.
I suggest making an S3Search method for this then.
Here's an example of one with a date widget:
http://bazaar.launchpad.net/~flavour/sahana-eden/trunk/view/head:/models/08_inv.py#L175
Then you can use /module/resourcename/search to allow the user to set
the options & the formats toolbar will have the filter preset inside
it.
> What you say is that the user should be ideally using the "views"
> module to filter the data, then generate the XML form it?
Not quite - views/formats.html was just how to get the 'XML' button on
the list of export formats.
For searches, there is an extra one too:
views/_search.html
F
Dominic König
> I don't believe we have any automatic DTD validation currently - this
> is up to the manual testing of the stylesheet...
Yup, the lxml parser /can/ validate - it's just to enable it.
However, I didn't do that - for these reasons:
- most XML sources do not provide DOCTYPE information at all
- many XML sources use mismatching namespaces or similar "errors" that would
prevent successful DTD validation
- DTD validation needs to download the DTD which is extra network traffic
(*if* the network is available at all - maybe not e.g. in offline
deployments), and another possible point of failure
Question: what's the possible benefit? DTD validation does not check data
types or content, and we're transforming the XML structure anyway, so it has
absolutely no meaning for our data import.
It would, in doubt, just refuse to parse sources if they do not comply 100%
with the DTD - but we actually want to be able to import even from these
sources, even if they do not 100% comply with the DTD - so we're just adding
processing time and cutting off (possibly important) data sources.
From my experience with interoperating with other applications, I can
guarantee that they won't comply 100% with the DTD in the majority of the
cases, *especially* when it comes to EDXL.
But honestly, who cares? Shall we ask them to please correct their export API
functions in order to comply with the DTD and include proper declarations so
that *we* can import their data? Who are we that others will just go and "oh
yes, sorry - if Sahana needs this, we'll fix it immediately"? Come on - that's
not the reality.
For exports, we can do this validation once before publishing the stylesheet
(manually), but there's no need to do this on every export. It would only add
extra processing time for no reason - if it complies once, it will always
comply - this isn't handcrafted XML, it's machine output.
In general - in interoperability, you would rather try to avoid on-the-fly DTD
validation. We do not really care whether sources really comply with the XML
standard 100% as long as they implement the important structures - what we
really want is to get at their data no matter what.
Thus, what you really *have* to do is to validate and de-duplicate the data as
per our model, and that's what we're doing automatically (built-in in S3XML,
so you don't need to care for this separately).
Best regards,
Dominic
Dominic König
> Hello All,
>
> we need to add a function to Import and export the SITREP data from
> table as XML; possibly add an icon with the rest of the stack xml,
> pdf, ....; unless it already exists.
It does exist, except that there isn't quite an icon for SITREP or EDXL in
general.
> My thinking
>
> 1) Each import or export iteration should use a single xml file; i.e.
> multiple records in a single file
That is standard, just select the records and export them.
To implement that, do the following:
a) add a folder "sitrep" under static/formats
b) put a stylesheet "export.xsl" into this folder which transforms S3XML to
EDXL-SITREP (look into static/formats/have/export.xsl for an example)
c) put a stylesheet "import.xsl" into this folder which transforms SITREP into
S3XML (look into static/formats/have/import.xsl for an example)
d) You can use the static/formats/xml/edxl-de.xsl templates to wrap the output
in an EDXL-DE envelope if needed, or you can use the s3mgr.xml.envelope() for
the output of s3_rest_controller (the latter variant would allow you to
specify for parameters for the envelope, so this is the preferred method).
> 2) export filter is done using the attributes shown in the SITREP home
> page table
You can use URL variables, or a filter query in the controller, or both to
select records. Then just append ".sitrep" to the URL and you'll get your
sitrep.
> 3) Can we undo imports? i.e. be able to review the records first
> before committing to tables?
The 2-phase importer doesn't yet support XML imports (in general it does, but
it it still disabled there), but I'm currently working on that. So, in
principle: yes. By next week...
> 4) Is there a generic API, possibly through S3XML that we could use?
Yes, s3_rest_controller().
Check:
http://eden.sahanafoundation.org/wiki/S3XRC/RESTfulAPI
>
> 5) validated with EDXL-SITREP v 1.0
See my other mail.
Best regards,
Dominic
Dominic König
I would wait until SITREP is actually published as a standard before
validating against it. SITREP isn't an OASIS-approved EDXL standard yet, but
still a EIC/DHS draft in process, so peers may or may not refer to the same or
a different DTD. I didn't even find a standard common sitrep namespace
definition.
Which other applications do currently process or provide SITREP data?
Not that we end up in the same dilemma as with PFIF, which, mainly due to the
lack of a stakeholder consensus process, changed core concepts between two
versions and left implementations incompatible despite using a "standard".
Such a development would not really promote interoperability, but would rather
be just for reputation á la "Look at us, we have implemented a FEMA standard".
The reality could reply "Hmm nice, but what for?".
So what would we answer?
Dominic
Nuwan Waidyanatha
Thanks for all the explanations, it's quite a handful. Hopefully,
Kasun will get started on it soon and we'll come to you for help.
The SITREP schema we are using is what will be released soon; i.e.
v1.0. They are still working on perfecting the literature around it.
Nuwan
2011/11/27 Dominic König <dom...@nursix.org>:
kasun perera
I'm working on xml exporting
I have added following line on views/formats.html
<a href="{{=response.s3.formats.xml or r.url(representation="xml")}}" title="{{=T("Export in XML format")}}"><img src="/{{=request.application}}/static/img/xml_icon_small.jpeg"/></a>
and added export.xsl and import.xsl on static/formats/sitrep as Dominic has indicated.
I'm confused with finding where to write the implementation of the export function(for example saving as file on local machine). can you give some indications?
--
You received this message because you are subscribed to the Google Groups "Sahana-Eden" group.
To post to this group, send email to sahan...@googlegroups.com.
To unsubscribe from this group, send email to sahana-eden...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/sahana-eden?hl=en.
--
Regards
Kasun Dileepa Perera
Department of Computer Science and Engineering
University Of Moratuwa
Fran Boon
> I'm working on xml exporting
> I have added following line on views/formats.html
> <a href="{{=response.s3.formats.xml or r.url(representation="xml")}}"
> title="{{=T("Export in XML format")}}"><img
> src="/{{=request.application}}/static/img/xml_icon_small.jpeg"/></a>
I would make the whole line optional:
{{if "xml" in response.s3.formats:}}
I would also make response.s3.formats.xml into a Storage(), to handle
the exact XML representation desired for this resource, so:
<a href="{{=response.s3.formats.xml.url or
r.url(representation=response.s3.formats.xml.format)}}"
title="{{=T("Export in XML format")}}"><img
src="/{{=request.application}}/static/img/xml_icon_small.jpeg"/></a>
Note that to plug into Search, which is what I understood you wanted,
so that you can filter the export via GUI, then you also need to edit:
views/_search.html
> and added export.xsl and import.xsl on static/formats/sitrep as Dominic has
> indicated.
> I'm confused with finding where to write the implementation of the export
> function(for example saving as file on local machine). can you give some
> indications?
The export.xsl is what does all the transformation.
All the rest is done via the existing REST controller:
http://.../eden/module/resourcename.xmlformat
When you say 'local machine' - do you mean the client desktop or the server?
For the server you'd need to call lower level APIs...
Best Wishes,
Fran.
Dominic König
> If you additionally need the EDXL-DE wrapping, then I can quickly enable
> this in export_xml - would be quite the same sequence of function calls,
> just one additional parameter. Just ping me.
Err...if you include the EDXL-DE envelope in the sitrep/export.xsl stylesheet
like:
<xsl:stylesheet
xmlns:xsl="http://www.w3.org/1999/XSL/Transform" version="1.0">
<xsl:output method="xml"/>
<xsl:include href="../xml/de.xsl"/>
<xsl:template match="/">
<!-- wrap the content in EDXL-DE -->
<xsl:call-template name="EDXLDistribution">
<xsl:with-param name="template" select="'true'"/>
</xsl:call-template>
</xsl:template>
<xsl:template match="s3xml">
<!-- transformation of the content goes here -->
</xsl:template>
</xsl:stylesheet>
and then add the parameters for the envelope directly to export_xml():
r = s3mgr.parse_request("my", "resource")
resource = r.resource
sitrep = os.path.join(r.folder, "static", "formats", "sitrep", "export.xsl")
xml = resource.export_xml(stylesheet=sitrep,
distributionID="12345",
senderID="dom...@nursix.org",
distributionStatus="actual",
distributionType="report",
combinedConfidentiality="unclassified",
contentDescription="Site Report")
...then it will add the envelope right away.
I modified export_xml to accept a parameter as_tree=True/False, so you can
retrieve an ElementTree instead of a string object if you need it.
Of course you can extend the EDXL-DE envelope by modifying
static/formats/xml/de.xsl - I have only put in the required basic elements.
Best regards,
Dominic
Dominic König
that's what I try to say all the time: you do not need an XML export function.
XML export is already included in s3_rest_controller - all you need is to
(HTTP-)request the particular format you want from the API (that is, XML is
not a format, but a family of formats) - and that would have to be done in the
URL:
http://localhost:8000/eden/xxx/yyy = HTML format
http://localhost:8000/eden/xxx/yyy.have = EDXL-HAVE format
http://localhost:8000/eden/xxx/yyy.sitrep = SITREP format
So, if you have implemented a sitrep/export.xsl, then all you have to do is to
change the export link into:
<a href="{{=response.s3.formats.sitrep or r.url(representation="sitrep")}}"
title="{{=T("Export in SITREP format")}}"><img
src="/{{=request.application}}/static/img/sitrep_icon_small.jpeg"/></a>
Well, and of course you need a sitrep icon then.
I am not sure whether SITREP is a generic format for any data resource, or
rather tied to a particular controller (i.e. a particular set of database
tables). If it is not generic, then you should not add it to
views/formats.html, but only in the view of this particular controller.
Also I wonder how the stylesheet looks like - what do you transform into what?
---
If you need to retrieve the XML in a controller function (! that's btw the
answer to your question on "where" to implement it), e.g. in order to push it
somewhere (e.g. by messaging), then you can use the Python API of the same
function:
r = s3mgr.parse_request("my", "resource")
resource = r.resource
sitrep = os.path.join(r.folder, "static", "formats", "sitrep", "export.xsl")
xml = resource.export_xml(stylesheet=sitrep)
If you like to select particular records of the resource, you can use
add_filter, e.g.:
table = resource.table
query = table.modified_on > (datetime.utcnow() - datetime.timedelta(days=5))
resource.add_filter(query)
...before calling resource.export_xml().
If you additionally need the EDXL-DE wrapping, then I can quickly enable this
in export_xml - would be quite the same sequence of function calls, just one
additional parameter. Just ping me.
Dominic
Nuwan Waidyanatha
Thanks for the guidance. Kasun is handling the coding.
I see two use cases for the SITREP (keeping EDXL-SITREP standard in mind)
1) A Sahana user
1.1) saving a subset of the SITREP records in XML format to throw
on a USB to take somewhere else
1.2) bring a an XML file with several SITREP records on USB in
XML format, then importing it in to Eden after stagging them and
validating
2) Another software, let's say Ushahidi (incident manager) for
example, wants a webservice
2.1) to grap daily SITREP records from Sahana
2.2) share their incident records with Sahana but we convert them to SITREP
the other thread that Kasun is talking to you and Fran is tied in with
use case 1)
I know Eden has provision to accommodate 2) but Kasun is leaving this
project to join another Dec 1st. I'd be happy if he can completes 1)
fully for now.
What I want to test with 1) is, export the SITREP files from Eden then
validate it with EDXLSharp (http://edxlsharp.codeplex.com/). That
exercise would certify Eden SITREP to be EDXL-SITREP compliant; i.e.
we can boldly put a message on the wiki saying "Eden SITREP was tested
with EDXLSharp for interoperability".
Nuwan
2011/11/29 Dominic König <dom...@nursix.org>:
Fran Boon
REST controller...just that (1) can give the user a UI to generate the
URL which downloads their choice of which data.
2.2 is 'just' the import.xsl
Note that we already support Ushahidi's native API, so there is no
need for them to build a SITREP exporter.
The main problem with Ushahidi compatibility is that their categories
vary per-instance, so you really need a custom import.xsl for every
single deployment!
So a UI to do the category mapping & save the custom.xsl would be useful...
F
Nuwan Waidyanatha
as "other"; i.e. Ushahid's SITREP export should comply with EDXL and
handover Eden a XML file that is EDXL-SITREP compliant; of cause some
interfaces will need some customization on a case by case basis;
nevertheless, the generic functions should be in place.
Nuwan
Fran Boon
> If Ushahid's categories don't comply, then we can label those records
> as "other"; i.e. Ushahid's SITREP export should comply with EDXL and
> handover Eden a XML file that is EDXL-SITREP compliant; of cause some
> interfaces will need some customization on a case by case basis;
> nevertheless, the generic functions should be in place.
Are you aware of any plans for Ushahidi to implement SITREP?
F
Nuwan Waidyanatha
Nuwan Waidyanatha
Dominic König
> What is IRS?
The Sahana Eden Incident Report System (i.e. the pendant for Ushahidi).
> comes from the dedicated module
Ah - I see.
Is it possible for you to share the code, i.e. is there a branch somewhere so
I could take a look at it?
That'd be good in order to get a better understanding of what you're trying to
implement, and thus give better support in terms of how to code it.
Dominic
Nuwan Waidyanatha
Can you point Dominc to the branch that you created
Nuwan
2011/11/29 Dominic König <dom...@nursix.org>:
Fran Boon
> Is it possible for you to share the code, i.e. is there a branch somewhere so
> I could take a look at it?
https://code.launchpad.net/~kkasunperera/sahana-eden/sitrep
F
Pat Tressel
Are you working on static/formats/sitrep/import.xml ? We're at RHoK in Portland, and we're interested in EDXL-SITREP. We're just playing around with this, so nothing serious yet, but if you haven't started on an import.xml, but you do already have a sitrep table, maybe I could do the import.xml, or help with it...?
-- Pat
Nuwan Waidyanatha
Kasun has left the project and no longer will be working on Eden-SitRep.
Please feel free to build if you wish. The LP shared with you earlier
has the tables in that module.
does that help?
Nuwan