Re: StratML Query Service

[Owen Ambur]

Thanks for the progress report, Naval.

With respect to a subset of the files for use in the initial iteration of the prototype/demonstration, it would be good to focus on the two dozen 2022 updates to the U.S. federal agency plans listed at https://stratml.us/#WorkResults  The intent is to show how the discovery and presentation of the goals/objective in those plans can be made more citizen/stakeholder centered than currently is the case on the Performance.gov site, which is centered on the bureaucracy and making the administration look good rather than enabling users to see what is of direct and immediate interest to them.

Since Pradeep is particularly interested in DoD, the several dozen plans at https://stratml.us/drybridge/index.htm#DoD is another subset for potential focus.

As far as the search interface is concerned, the advanced query screen for MarkLogic's prototype comes fairly close.  It shows how the key elements of the schema can be selectively queried and something like it would be fine for the initial iteration.  Eventually, however, it might make sense to simplify and combine the query features for: a) goal and objective names, and b) goal and objective descriptions -- because the only difference between the meaning of those two elements is a matter of timing (goals focus on the longer term while objectives focus on the near term).

The key is that the query results listings should deliver direct links to the elements in which the queried terms appear, within the context of the plan, like this.  That's the benefit to be provided versus existing full-text query engines.  For goals, objectives, stakeholders, and values, the first two or three lines of their descriptions should probably be provided in the query results list along with the name of the organization compiling the plan.

Eventually, faceted querying of goal/objective names and stakeholder names should be provided.

Owen Ambur

https://www.linkedin.com/in/owenambur/

-----Original Message-----
From: Naval Sarda <nsa...@epicomm.net>
To: Owen Ambur <owen....@verizon.net>
Cc: pradee...@ictect.com <pradee...@ictect.com>
Sent: Thu, Sep 15, 2022 9:12 am
Subject: Re: StratML Query Service

Hi Owen,

We have downloaded BaseX database and are loading 50-60 XMLs into the same out of 5000+ XMLs in sitemap.

Then we will run queries against the database and will share screenshot will you.

So the initial scope of work seems to be loading entire XML sets in BaseX Database and search within the same. Do let us know what is your expectations about search interface.

Second part is allowing users to submit their XML, validate the XML and either include the same in the repository or email feedback about validation errors.

We hope to get back to early next week with screenshots of searches against loaded XMLs.

I have not provided estimates yet. Once we share screenshots of search, based on your responses about search expectations, we will share the estimates for this piece of work.

Naval Sarda

EpiComm Technologies

On 14/09/22 8:57 pm, Owen Ambur wrote:

Naval, I too would like to hear from Jorge again.  We were in close contact a couple years ago but I didn't hear from him at all for quite awhile.  Then I did get one message from him indicating that he thought he might be able to figure out how to deal with the difficulties he had encountered, but I haven't heard further from him.  I think he may have had health issues.  His GitHub repository hasn't had any activity since August 2021.  Nor has he been active on LinkedIn.

His initial thrust was to provide a more user-friendly authoring/editing interface than my XForms.  He called it a "wizard" interface.  I understand it was rendered inoperable by an upgrade to the platform on which he was hosting it but I captured screen shots of the five "stages" he designed in his version of the form.  From my perspective, he missed the real potential in that the complexity that could usefully be addressed is all in his 5th stage, where the goals, objectives, and performance indicators are to be documented.  It would have been better to combine his first five stages into one and enable goal and objective sections to be selectively expanded and collapsed to save users from having to do so much scrolling while also making it easier to see where they are in the form.

However, the primary point is that the BaseX query service was mostly an after-thought for him, and while his browse/query interface was relatively attractive and had some useful features, he wasn't able to get it to work properly.  For example, he wasn't able to figure out how to ingest addtional files, i.e., the entire StratML collection.  Nor was he able to get to the point of enabling queries based upon goal/objective, stakeholder, and value names.  If memory serves me correctly, the search results listings also linked to entire documents rather than to the element in which the queried term resides.  (That's the same issue Jason Lind had with the StratML query service he built on Azure.  Jason said he thought he knew how to fix it but ran into personal difficulties and lost access to the Azure platform.)

From my perspective, it seems like that issue should be easy to fix simply by leveraging the GUIDs associated with each goal and objective and pointing (hyperlinking) directly to them from the query results listings, like this.  (I'm not an XPath or XQuery expert but surely they must do something like that and native XML databases must take advantage of such capabilities.)  It seems to me the mistake other developers have been making thus far is to attempt to make the service the repository for the plans/reports themselves rather than of metadata about them.

Since StratML is XML, it seems like a no-brainer to use a native XML database, assuming any of them are mature enough to rely upon.

I'm not sure what you mean by "structure the query language based on format," but the service most definitely should leverage the semantics and structure of the StratML schema.  That is the essential requirement and what will differentiate StratML query services from other search engines that fail to take advantage of valid XML instance documents (and thus are lowly ranked by me).

BTW, I subscribed to the BaseX listserv for the purpose of trying to engage a developer(s) but was unable to connect with anyone willing to take on the project.  That makes me a bit skeptical about whether the database coding is mature enough to rely upon.

Owen Ambur

https://www.linkedin.com/in/owenambur/

-----Original Message-----
From: Naval Sarda <nsa...@epicomm.net>
To: Owen Ambur <owen....@verizon.net>
Cc: pradee...@ictect.com <pradee...@ictect.com>; jo...@vionta.net <jo...@vionta.net>
Sent: Wed, Sep 14, 2022 2:03 am
Subject: Re: StratML Query Service

Hi Owen,

I would like to hear from Jorge about his experiences with BaseX.

Amongst the XML databases, BaseX seems to be one of the best.

Amongst all databases choices, BaseX is ranked quite low and I suppose it is being used only for XML based applications.

If we are using XML based databases, then importing has no effort. It is just dumping the received XML into database without worrying about the format.

While searching we might want to structure the query language based on format.

So probably we need one field which identifies the XML format stored in the other field and based on the format, write the query to fetch the data.

Since Pradeep is also in this email chain, I would like to know his thought process as well.

May be I am missing something which is holding up. Let me know.

Naval Sarda

EpiComm Technologies

skype - navalsarda

On 14/09/22 9:19 am, Owen Ambur wrote:

Naval, since the schema has been specified, it has occurred to me that a SQL database could be used and I am open to considering that prospect.  However, there are actually three or even four StratML schemas: 

1) the pre-ISO version of Part 1, Strategic Plans;

2) ISO 17469-1, the official version of Part 1, the StratML core;

3) Part 2, Performance Plans & Reports (formerly ANSI/AIIM 22:2017); and

4) Part 3, Additional Elements, which was adopted as an AIIM best practice but apparently is no longer recognized as such.  It is more complex and unlikely to be implemented by anyone anytime soon anyway, and I certainly don't want to take on that much complexity right now.

However, the difference between the first two versions of the schema is minor:  In the ISO version, FirstName became GivenName and LastName became Surname.

In the initial iteration of the query service, I want to focus on Part 1 files, but at the same time, I also want to keep in mind the need to import Part 2 files and to accommodate potential extensions in the standard in the years ahead as well.  While that doesn't seem likely in the near future, if we can effectively demontrate the power of the standard, the need/opportunity for extensions may become apparent at some point a few years down the road.

The bottom line is that it seems to me that a NoSQL database should be used and preferably one that "understands" XML natively, e.g., BaseX or eXist.  On the other hand, the stability and supportability of the database also needs to be taken into account.

From my perspective, it would be great if the volume of usage of the service becomes large enough to justify more server power, but I don't expect that to be the case anytime soon nor would I want to pay much more for scalability right out of the gate.

In addition to Pradeep, I'm also copying Jorge in the event he may wish to chime in on the potential to use BaseX.  You can see screen shots of his effort via a link at https://stratml.us/#Vionta

Owen Ambur

https://www.linkedin.com/in/owenambur/

-----Original Message-----
From: Naval Sarda <nsa...@epicomm.net>
To: Owen Ambur <owen....@verizon.net>
Cc: pradee...@ictect.com <pradee...@ictect.com>
Sent: Tue, Sep 13, 2022 5:32 pm
Subject: Re: StratML Query Service

Hi Owen,

Thank you for providing additional information as below. This gives more of bigger picture which is essential to design architecture.

Yes AWS glue is not needed for importing structured data.

Elastic Search is one of the open source tool which is better to build query service as it indexes the database properly for faster results

Regarding database choices between databases - there are two types -

1) Relational Database like MYSQL, MariaDB

2) NoSQL - like MongoDB

Relational databases are good if structure is defined but they have to be vertically scaled i.e. you have to increase server size if load increases and might have to go with paid databases if you really want to take advantage of additional processors available on servers. There are ways to make relational database scale horizontally i.e. add more server by splitting data into multiple server and initiating request to all servers from front end. We did this for one project where data was split in 8 different servers and front end was sending request to 8 different servers. We used servers provided by Digital Ocean at that point of time.

NoSQL databases are scaled vertically i.e. you can add split the data into multiple servers and can build a cluster of servers. Front end does not need to designed to query all servers. It will do it by itself.

Depending on your work load and total database size estimations, you can think which database is right choice for you. Each server has harddisk and each harddisk has upper bottleneck to provide data.

Regarding Mocha host shared hosting plan, it will not provide needful facility to install additional softwares like installing elastic search. So you will get stuck at some point of time. You may go with other hosting plans of mochahost which provides ability to install software like elastic search for development needs.

Once the software is ready, then you will have to decide on hosting provider which provides facility to deploy load balancer to handle load for handling multiple request and forward it to multiple servers. But it all depends on workload you are envisioning. There are applications which are running on single server for years. It might be the case that workload is not much and you can continue with single server.

Also there are caching services which lets you cache the most frequently queried data. But I don't think queries are going to be same. But it is a big performance booster.

Naval

On 13/09/22 11:38 pm, Owen Ambur wrote:

Naval, this sounds very encouraging.  I want to invite Pradeep to chime in but here are my additional thoughts right now:

1. I understand why an ETL tool would be required to import XML files into a SQL database but don't know why that should be necessary with a NoSQL database.  From its description, Amazon GLUE does not seem to be what I need.  StratML files don't need cleansing or validation.  What value do you think a Amazon GLUE demonstration might provide?
2. Matt MacKenzie developed a StratML query service running on MongoDB so I know that it can be done.  What I don't know is why he abandoned it or whether it actually met the requirements.  https://stratml.us/#Uberity
3. Shashi Mudunuri has also recommended MongoDB as a less costly alternative.  He developed MarkLogic's StratML query prototype, which was the best implementation I've seen thus far.  https://stratml.us/#MarkLogic
4. The elements to be queried and presented in results listing are specified in the objectives under Goal 1: Query/Discovery and Goal 2: Presentation of my technical development plan.  Note that the GUIDs associated with each goal and objective enable direct referencing, which is what the query service should do.  
5. Please note also that the query service should be just that, i.e., a query service and not a repository of the plans and reports themselves.  In that sense, it should be considered to be more of a metadata repository than a content repository.  The authoritative versions of the files should reside elsewhere on the Web.
6. I am certainly willing to consider using AWS but am concerned about paying for the development of software that is tied to them.  I'd like the software to be open source and available for reuse and extension, e.g., on GitHub.  AWS didn't do itself any favors by deplatforming some of its users and I'm not going to risk my money on a platform that may decide to do that to me and leave me with no reasonable alternative.
7. My current hosting service is https://www.mochahost.com/  While they do support some databases, MongoDB is not among them:

Owen Ambur

https://www.linkedin.com/in/owenambur/

-----Original Message-----
From: Naval Sarda <nsa...@epicomm.net>
To: Owen Ambur <owen....@verizon.net>
Cc: pradee...@ictect.com <pradee...@ictect.com>
Sent: Tue, Sep 13, 2022 1:22 pm
Subject: Re: StratML Query Service

Hi Owen,

Thanks for getting back so quickly.

I am impressed how stratml.us have been able to assimilate information about my company in entirely segregated information on URL below

https://stratml.us/drybridge/index.htm#EPCM

And it is so amazing to know the standard you have worked on for years have been accepted as International Standard.

I have been reviewing the technical development plan.

3.1 Bulk upload says

"It would be good if the sitemap listing could be used to semi-automatically import the existing collection. https://stratml.us/sitemap.xml"

This is super easy I believe if we are using any ETL tool. Without ETL tool, it will take a lot of programming effort.

Amazon GLUE which I had proposed is an ETL tool. There are many more ETL tool in the market. Do you have any preference.

3.2 says - Adhoc Importing -
I think, the same ETL tool can be invoked whenever any new submission for import comes.

3.1.1 Wireframes for query and query results.

StratML has definite XML structure I believe, So the ETL tool extracts the data from XML files and loads into database of your choice. Then it is upto you to decide how you want to show the data. So this phase is more of understanding what is important for end users to come up with useful user interface.

3.2 Database preferable should be No SQL DB like Mongo DB considering that each field is going to have lot of data.

3.4.1 - Hosting plan is shared hosting. This won't suffice in any way. It is more suited for static website which does not need any server resoruces.

If you wish to see a live demo of import data in AWS account using Amazon GLUE, we can schedule a session. Let me know 24 hours in advance so that I can load the data from the sitemap XML feed and query the same. It is not critical to use same set of tools. We can use another tool for your application which can give similar features.

Let me know your thoughts. Whatever is my interpretation is based on the document plan you have provided. It can vary significantly as current status might be different as plan might have been developed at different point of time.

Let me know your thoughts.

Naval Sarda
skype - navalsarda

On 13/09/22 9:59 pm, Owen Ambur wrote:

Hey, Naval, thanks for getting back to me so promptly.

I don't have an AWS account and have been planning to host a StratML query service at https://aboutthem.info/  See my technical development plan there.

However, I'm pleased to know that you think you can develop such a service quickly and easily because it seems to me that someone with the necessary skills and capabilities should be able to do so.  The hard work has already been done.  The schema has been specified and adopted as an international standard and thousands of StratML files are available for ingestion and indexing.

The collection now includes Epicomm's about statement, at https://stratml.us/drybridge/index.htm#EPCM  While your site does not document any apparent experience with developing XML-enabled query services, I look forward to seeing a prototype.  When do you think you might be able to show me one?

BTW, I'm also glad to see the value you place upon long-term relationships.  I'm especially interested in engaging partners, like Pradeep, who are not merely looking to get paid for one-off work but who share the vision of the StratML standard -- a worldwide web of intentions, stakeholders, and results -- and are interested in helping to commercialize it.

Owen Ambur

https://www.linkedin.com/in/owenambur/

-----Original Message-----
From: Naval Sarda <nsa...@epicomm.net>
To: Owen....@verizon.net
Cc: Jim Young <j...@xforms.co>
Sent: Tue, Sep 13, 2022 6:03 am
Subject: Re: Fwd: New submission from XForms.co Website: Contact Us

Hi Owen,

Thanks for sending us your request through xforms platform.

You are primarly looking to develop a query service to StratML Schema as mentioned in email chain below. And there are more than 5k files to load and query as mentioned in sitemap URL.

We can definitely setup the same for you using AWS Glue and Amazon Athena combination in your AWS account.

Glue will fetch all the data, apply ETL and load the data and Athena can help you query the data. We can write pre defined queries so that you can run the queries whenever you want.

We can also write Glue jobs which can be used to fetch fresh data whenever you want.

Let me know if that works for you and we will send you a quotation. It is a quick job and good thing is that you don't need any server as such.

Naval Sarda

EpiComm Technologies

skype - navalsarda

On 13/09/22 2:25 pm, Jim Young wrote:

This isn’t something we will do but maybe u can build something for this guy 

Jim Young, PG, STS
j...@xforms.co
M: 919-809-4577
Any Form, on Any Device, Anywhere

Forms Platform | xformsmobile.com

Commissioning Platform | xformscx.com

---------- Forwarded message ----------
From: <j...@xforms.co>
Date: Monday, September 12 2022 at 11:04 PM EDT
Subject: New submission from XForms.co Website: Contact Us
To: <j...@xforms.co>

Name   Owen Ambur Organization   StratML Community, AboutThem.info, & ConnectedCommunity.net Email   Owen....@verizon.net Describe your vertical/interest in XForms   I'm looking for partners to develop a query service(s) leveraging the StratML schema, using the >5K files in the StratML collection for demonstration purposes. https://stratml.us/drybridge/index.htm All of their URLs are listed in sitemap format at https://stratml.us/sitemap.xml If you think your NoSQL datastore may be up to the challenge, I'd be pleased to hear from you. BTW, while it is not my primary focus right now, I'm also interested in seeing the PDF form that Ed Chase developed updated to accommodate minor changes that were incorporated in the ISO 17469-1 version of the schema. https://stratml.us/index.htm#PDFform

[Naval Sarda]

Hi Owen,

In one of your emails, you have mentioned that you want an ability where users can submit their own StratML XML.

We have designed few screens as attached for submission.

So first time users will fill basic information and then submit either StratML XML file or share URL with StratML XML data.

We will validate the XML file. If it is valid, then we will ask the users to verify their email address by clicking the link in the email and setup password/account after verification,

If XML is invalid, then show relevant error message.

I have been seeing your emails. I suppose there are revisions expected in XML format. But I believe we can still validate against the XML tags and data which have stabilzed and have been not been changing for very long time. Other additional tags, we can ignore as long basic needed tags are correct.

Please review attached mockup and share your feedback. You might be having different idea though regarding user data collection which you can share with us.

In the mockups, we have added sample XML format download option for users to download and also help link which can be either a webpage or pdf which shares all the details of the format needed.

We are still working on figuring out querying Xbase database. Some XML files have already been uploaded in Xbase database.

Whichever final set you provide, will be uploaded as per your needs. There are mutliple places you have mentioned. So you can consolidate and let us which is the final set.

Naval Sarda

EpiComm Technologies

[Owen Ambur]

Naval, on a quick glance, the mock-up screens look fine and I'm glad to know that you're thinking about how best to accommodate the submission process.  Before you put too much effort into that, however, please bear in mind it is not an essential feature of the initial iteration of the prototype.

As for which files to include, eventually all >5K files in the collection should be indexed and it is unnecesary to run the validation process against them.  However, as far as I am concerned, the highest priority subset is the twenty-four 2022 updates to the U.S. federal agency plans listed at https://stratml.us/#WorkResults

I defer to Pradeep to tell you if he'd like to make a pitch for focusing on the several dozen DoD plans listed at https://stratml.us/drybridge/index.htm#DoD

If Gaya is able to transform the South Carolina agency accountability reports from Excel into StratML, they would also become a priority for inclusion in the query service, along with the federal agency plans.  (I manually converted about a dozen of them but it is a very cumbersome process.)

Locally, here on Hilton Head Island, I am also anxious to include the town's and the county's plans but, again, I'd like to focus first on the U.S. federal agency plans.  (The PDF of the town's comprehensive plan is 404 pages long and, ironically, I just discovered that it is generating a 404 error at https://hiltonheadislandsc.gov/ourplan/  However, my link to it still works at https://connectedcommunity.net/HHI.html, where my StratML rendition of it also resides.)

As far as the functionality of the query service is concerned, one key capability is being able to discover goals and objectives based upon the stakeholders to whom they are applicable, regardless of which bureaucracy is responsible for them.

[Owen Ambur]

Sounds like good progress, Naval, but the screen shot you attached doesn't make much sense to me.  It's certainly not the kind of result I'm seeking.

I'm not sure I understand some of your questions but see my responses below in [brackets].

I'm copying Jorge in the event he may have suggestions based upon his experience in developing his StratML query service prototype.

Owen Ambur

https://www.linkedin.com/in/owenambur/

-----Original Message-----
From: Naval Sarda <nsa...@epicomm.net>
To: Owen Ambur <owen....@verizon.net>
Sent: Tue, Sep 20, 2022 9:15 am
Subject: Re: StratML Query Service

Hi Owen,

As stated earlier, we installed XBase, loaded all 5000+ XMLs. Search was not working initially.

Then we removed attributes in 7 XMLs from first tag as highlighted below

And search started working.

Right now search happens based on specific tags in XML.

[Yes, the query service definitely should and must leverage the semantics (tag names) and structure of the StratML schema.  That's the entire point.]

For example, you have to specify which tag you want to search

[See MarkLogic's advanced search screen.  A somewhat simplified version of it is what I'd like to have.  See the query/discovery requirements documented in my technical development plan and let me know if you have specific questions about them.  I intend to track progress on the project in that document.]

Tag can be /StrategicPlan/StrategicPlanCore/Goal/Objective/Stakeholder/Name

and say what you want to search in this tag and it will search accordingly.

[As I interpret what you are saying here, it seems to be what I'm looking for but I won't know for sure until I see it in action.]

Also you can combine XSD and Xquery to generate Webpage (HTML) result.

[Not sure what this means. I'm not looking to generate HTML.  What I want and already have is styled XML (plus, thanks to the Turnkey folks, PDF).  However, if you think having HTML too will add value, I look forward to learning why. To me, it seems like a step backward.]

There is lot of documentation available in Xbase. Dedepending on what searching you want to do, we can explore more.

[I'm not looking for anything very complicated or complex in the initial minimally viable query service, but if we can successfully complete the initial project, there will be a lot more potential to pursue.]

We have reached to a stage where we can say we can give estimations. We need final link to what XML needs to be loaded. We will have to remove attributes stated above before loading.

[Feel free to give me cost estimates at anytime but please keep in mind that you have not yet shown me that you can deliver the kind of capabilities I'm seeking.]

There are two things you have mentioned. One was XML urls in sitemap and there was another.

[I'm not sure what you are saying here.  Yes, I want all >5K of the files currently in the StratML collection to be imported and available for querying, and using my sitemap listing to import them is one way of getting them into the system.  However, with extra effort they could also be transmitted in .zip files and, of course, the stratml.us site could also be crawled to discover and index them, as the web search engines do.]

[Perhaps the second thing you are referencing is how best to enable ad hoc ingestion of files after the initial bulk upload is completed.  We have exchanged messages about one potential way of doing so but that is not necessarily a requirement for the initial iteration of the service.  So long as I can upload files into the system, that is all that is initially required.] 

What kind of UI/ interface/APIs you are expecting for searching?

[See MarkLogic's advance search screen.  I'd like a somewhat simplified version of it.]

[I'm not looking for any APIs.  Indeed, it seems to me API's are a poor substitute for making data available in open, standard, machine-readable format -- like StratML -- so that it can be directly accessed and used without the need for APIs.  I was pleased to see the folks at Ontola making that point with reference to the design of their Argu application.  I hope to engage them in building support for the StratML standard into Argu.]

There are lot of emails around different topics. But for this phase, it would be great if you could finalize the expectation for generating estimations for your request.

[From my perspective, I've already done that, at https://aboutthem.info/SQS.xml  Please focus on that StratML rendition of the project performance plan.  The next step is to flesh out the performance indicators and begin reporting results against them.  I intend to use that document to track progress on the project.]

See attached screenshot of Xbase UI client. On the right, there are 7 XML records we took as sample to query so that it is easier to verify results.

[As indicated above, the screen shot doesn't make much sense to me. It's definitely not what I'm looking for in a search results UI.  Jorge's initial browse screen comes much closer to what is needed in a query results listing.  The difference is that, say, if goals/objectives are being queried, the two column headings would be Goal/Objective and Description and all that would be required in the third column would be an icon with a link to the relevant element, within the context of the plan/report in which it occurs, like this.]

And in the left we wrote some code to find some text in those 7 XMLs.

[I definitely don't want to burden users with seeing such coding.]

Naval Sarda

EpiComm Technologies

[Owen Ambur]

Adnan & Kurt, base upon our previous exchanges, I listed you as stakeholders in the first draft of my technical development plan for a StratML query service.

However, I we haven't had any further contact about that in recent times.  Meanwhile, I've been engaged with Pradeep Jain and Naval Sarda. 

Some of our exchanges are documented in my Google groups discussion list supporting the plan.

Do you want to be involved or shall I remove your name from my plan?

Owen Ambur

https://www.linkedin.com/in/owenambur/

[Owen Ambur]

Naval, here are some of the primary use cases for StratML query services:

1. goals & objectives applicable to stakeholder groups
2. conversely, stakeholders associated with goals & objectives (including those for which we ourselves may be responsible)
3. goals & objectives commonly associated with values (that are important to us and thus may be worthy of our support)
4. common (and complementary) goals and objectives in the plans of prospective suppliers, customers, and partners (This use case may have the greatest commercialization potential, to revolutionize the advertising and marketing paradigm and capture revenue currently going to social networking services that are highly immature for business-quality networking purposes.)
5. political parties and candidates for elective office whose goals & objectives coincide with our own priorities and support our values

See also the broader listing of use cases for the StratML standard itself.  Query/discovery capabilities are implicit in all of them.  However, please focus on the three requirements outlined in my technical development plan, which I will be using to track progress on the project to develop a minimally viable StratML-enabled query service.

While I encourage you to apply your own creativity, I also encourage you to collaborate with Pradeep, Jorge, and perhaps Kranthi to share knowledge/experience and avoid making the same mistakes over and over again while reinventing the wheel.  I'd like for any software development for which I pay to be made readily available for reuse and extension as open source, e.g., on GitHub, as Jorge has done.

Owen Ambur

https://www.linkedin.com/in/owenambur/

-----Original Message-----
From: Naval Sarda <nsa...@epicomm.net>
To: Owen Ambur <owen....@verizon.net>
Sent: Wed, Sep 21, 2022 10:12 am
Subject: Re: StratML Query Service

Hi Owen,

Based on your feedback below, we are working on figuring out use cases of the query search service.

For example -

Somebody knows a stakeholder. He/she wants to know where this person is stakeholder and what is his/her role. One stakeholder might be involved in mutliple companies. So a complete picture is needed.

Another use case could be -

Somebody want to know about all the stakeholders in an organsation. So he seaches for organisation name and see list of stakeholder and they might also further drill down to know where all the stake holders are involved to decide if they can invest in the organisation or not.

There might be many more.

We will study in details about the possibile of use cases and then only can come up how the search screen needs to be structured.

If you wish to throw some light on use cases, it will be great.

Regarding XML capability, you can share some searches you wish to perform in the xml files and we can share the XMLs which matches your searches so that you get an idea.

The previous UI was not end user friendly. It was direct BaseX search UI tool ment for developers to develop the program.

Naval

[Naval Sarda]

Hi Owen,

We have drafted few screens for proposed search query.

See attached.

First screen is basic search with result listing

Second screen is detailed search with more fields for searching

Third screen is detailed results screen.

We can render details result screen as per URL below by reusing the styled-XML sheets

https://stratml.us/forms2/temp-stratml/temp.x

https://stratml.us/carmel/iso/EPCMwStyle.xml

We can push our search result to URL below and output will be rendered by URL below

https://stratml.us/forms2/out.xml

The choice of search fields is very subjected.

There are several use cases, for example

User might want to find CSR report for particular year.

Now there is no direct tag for year.

If year needs to be in search criteria, then we will need the XML tag/field in which program needs to search for a year.

So you can provide search criterias. We can even have different search all together for different stakeholder type.

Based on your feedback, we can figure out next steps.

Also in your technical development plan, it was mentioned hosting, domain, etc.
Major thing was UI wireframes. For bulk loading, we have already done loading in BaseX locally. For adhoc addition, we have provided a UI which could be phase 2.

Also this URL provide a user interface / form where user can type in data to generate XML

https://stratml.us/forms2/out.xml

We will wait for your feedback for next steps.

We now understand, then we can generate styled XML where XSD and XSL (XML style sheets) renders the UI and we just need to pass on the XML data to them.

So we have proposed to reuse the two UI URLs above as we can easily pass on output XML to their XML stylesheets.

Let me know your thoughts.

Naval Sarda

EpiComm Technologies

[Owen Ambur]

Excellent progress, Naval.  Both the query screen and the results screen are more complex than I've been thinking that I'd like to see in the initial, non-advanced version of the service.  Israel Condon has drafted some wire frames that come closer to what I had in mind.

However, I'm open to considering all options, and while I'd prefer if we could all work together to figure out how best to proceed, I also welcome competitive efforts.  If you can build a service that accommodates all of the complexity in your wire frames for the advanced search capability, I'd love to see how it works and particular whether others find it useable and useful.

I'm not sure what you mean by "Now there is no direct tag for year."  As shown in your wireframes, each document may be assigned a <StartDate>, <EndDate> & <SubmissionDate>.  However, those elements are not required and in many cases (e.g., website about us statements) the dates are open ended.  So they cannot be assumed to be available for querying nor do I want to focus on them in the initial iteration of the service.

Down the road, the most important dates will be those for <TargetResult>s & <ActualResult>s to be documented in StratML Part 2, Performance Plan/Report, format -- not merely for interested users but even more so for <Stakeholder>s of the performer type, i.e., those responsible for achieving them.  That's more complexity than I want to take on immediately, however.

BTW, Chris Fox's StratNavApp sends reminders based upon those dates and he would like to have us use his service to track progress in the version of our performance plan for development of a StratML query service(s) available in his app.  However, those who have agreed to be stakeholders in collaboratively updating and maintaining that rendition of the plan have not been active in using it and it takes a bit of effort to do so.  I'm cc'ing them in the event they may wish to offer any comments or suggestions.

Meanwhile, you seem to prefer using E-mail, Pradeep prefers televideo conferences, and it is easier for me to use my XForm to update the version of the technical development plan on the AboutThem.info site.  So I guess each of us will need to continue doing our best to muddle through and see what we can deliver in terms of results that may be useful to others.

Also, BTW, Chris' app includes a query feature and you can see how the results look in these screen shots.  However, it has a number of constraints that make it ill-suited for the kind of query services I want to provide at https://aboutthem.info/

[Naval Sarda]

Hi Owen,

Can you please list down the fields you want in simple search?  Also we can have video call to understand your needs about results screen. We are in GMT +5.5 time zone.

We can have either Microsoft Team's call, Google Meet or call on any other platform.

For advance search, we have checked the XML format and accordingly have choosen the fields to cover them. So it is doable to implement.

Naval

[Owen Ambur]

Kurt, for some reason your reply went to my spam folder, where I recently discovered it.

If "oXml frameworks" refers to oXygen's StratML plug-in, yes, I'm still very interested in making it available again.

As Pradeep knows, I'd also like to see support for StratML built into MS Word online.

Please let me know if you have suggestions on how we might move forward in either of those regards.

BTW, Naval Sarda and his colleagues seem to be making good progress on developing a StratML query service running on BaseX.  With luck, I hope to have a prototype up and running before the end of the year at https://aboutthem.info/  I'll look forward to learning what you mean by "integrating the procedural side."

Owen Ambur

https://www.linkedin.com/in/owenambur/

On Friday, September 30, 2022 at 06:30:37 PM EDT, Kurt Conrad <con...@sagebrushgroup.com> wrote:

Owen,

It was good to see you at Balisage this year. The progress is impressive.

To answer your question, directly, I'm not sure that I can be of much help on the query service topic. It's not my area of expertise and the reliance on written communications makes the conversations expensive to follow. Also, most of my interests around querying would relate to integrating the procedural side,

Our early discussions focused on modernizing the oXml frameworks. Is that still of interest?

Thanks

/s/ kwc 2022.09.30 15:30


At 2022-09-20 08:47, you wrote:

Content-Transfer-Encoding: 7bit

---

Kurt Conrad
<con...@SagebrushGroup.com>
SagebrushGroup.com

Facebook.com/KurtWConrad
LinkedIn .com/in/KurtWConrad
Zoom: 509 581 0832 / sagebrush

[Kurt Conrad]

Owen,

Since we're looking for change vectors, let me launch a friendly bitch-fest...

None of the existing oXml styles really satisfy me. Authoring is awkward, even with the UI tunings.

Looking at the  v24.1 StrategicPlan template:

a. Not sure why the identifiers are displayed, they don't look editable. IDs are usually stored as @id attributes. Why not here?

b. It reads somewhere between document and form. Perhaps separate stylesheets?

c. Section borders feel weird. I use top and right to show the nesting and still maintain an open flow. The enhanced stylesheet seems to be a response. Eh

d. The hints need editing

e. I really miss a wrapper around Stakeholder, Value, etc. The flat model, at least for me, may drive some of the awkwardness. Also, if you wrap Stakeholder in Stakeholders, you could format the info as a table and improve usability.

f. This gets to an issue that Jean Kaplansky has gotten me to appreciate: trying to treat content as data. She's got some interesting horror stories.

g. The description field is a good example of this. Allowing a single text() node is data-oriented. But in the real world (the Balisage doc is an example), the lack of any substructure creates unreadable data bricks. Large descriptions need to be broken into paragraphs, lists, etc. How about adding more XML layers instead of using funky punctuation patterns? This might also help with querying.

h. XML Schema  was developed for and is optimized for expressing rigorous data models. DTDs are still common in the documentation field as strong data-typing isn't as important and DTD syntax is also a modeling language. For many years, I've noticed that documents designed using XML Schema tend to be a little weird.

To get a better feel for the design patterns, I built an authoring DTD from your schemas. It raises a few interesting questions:

.1. Why the multiple switches (doctype, document element, and @Type attribute)? PerformancePlanOrReport kind of begs the question. How about a single doctype that supports three document elements: StrategicPlan, PerformancePlan, and PerformanceReport

.2. Element names are a bit long (e.g., PlanCore instead of StrategicPlanCore since its also used in the performance context?)

.3. Identifiers are typically stored as @id attributes, especially for machine-generated IDs

.4. There appears to be an error in MeasurementInstance. It allows unlimited children instead of a single Target/Actual pair.

.5. The Role/RoleType encoding is very unusual.


"Integrating the procedural side" goes to this data/document value optimization question.

The Community engagement process is carefully sequenced and plans should reflect the order of activities. The stratml data models are limited in both order and expressiveness (e.g., Activity-Based Management performance models).

Increasing flexibility in the schema would make it more suited for general use, including authoring of strategic plans and performance documents. This might also accelerate adoption. Is there a customization layer that I haven't noticed?

I'll keep playing with the authoring DTD to see if any other ideas come to mind.

Let me know what your priorities are for the update. The phrase "available again" confuses me, a bit. I think that stratml frameworks have been in every recent version of oXml.

Thanks

/s/ kwc 2022.10.23 15:10

[Owen Ambur]

Kurt, thanks a bunch for working on this.  I said oXygen's StratML framework hasn't been accessible because as far as I personally am concerned that has been true.  On the other hand, I haven't tried to access it in recent years because the last time I tried, I was unable to do so and my impression has been that I could no longer access it.  Nor have my entreaties to the Synchro Soft folks brought a response.  Their documentation on GitHub is nearly a decade old.

If and hopefully when I can gain access to it, I'll be better able to respond to the issues you've raised with respect to how it works.  However, in the meantime, see my responses below in [brackets].

Owen Ambur

https://www.linkedin.com/in/owenambur/

On Sunday, October 23, 2022 at 06:10:46 PM EDT, Kurt Conrad <con...@sagebrushgroup.com> wrote:

Owen,

Since we're looking for change vectors, let me launch a friendly bitch-fest...

None of the existing oXml styles really satisfy me. Authoring is awkward, even with the UI tunings.

Looking at the  v24.1 StrategicPlan template:

a. Not sure why the identifiers are displayed, they don't look editable. IDs are usually stored as @id attributes. Why not here?

[Most of the IDs are automatically assigned (e.g., by GUID generator invoked by my XForms), if they have not previously been populated.  However, in the case of the Relationship element, the Referent Identifier must be copied from the source of the referent.  Also, for the IDs associated with Goals and Objectives, they too must be copied from the orginial source when they are repeated (i.e., identical) in subsequent versions of the plans/reports in which they occur.  Moreover, to the degree that referents may be internal to a document, their IDs must be repeated within that document and thus are not unique in the sense of being limited to a single instance.]

b. It reads somewhere between document and form. Perhaps separate stylesheets?

[Not sure what you mean here.]

c. Section borders feel weird. I use top and right to show the nesting and still maintain an open flow. The enhanced stylesheet seems to be a response. Eh

[Again, I'm not exactly sure what you mean here.  However, one of the biggest shortcomings of my XForms is the inability to collapse and expand sections, particularly when goals have lots of objetives and objectives have many performance indicators.  In those cases, lots of scrolling is required and it is difficult to see where you are in the document.]

d. The hints need editing

[Will be better able to address this point when I am able to access the StratML framework in oXygen again.]

e. I really miss a wrapper around Stakeholder, Value, etc. The flat model, at least for me, may drive some of the awkwardness. Also, if you wrap Stakeholder in Stakeholders, you could format the info as a table and improve usability.

[It sounds like the framework may ignore the hierarchy of the schema.  Stakeholders may occur at three different layers of the schema -- Organization, Goal, and Objective.  Is that what you mean by "wrapper"?]

f. This gets to an issue that Jean Kaplansky has gotten me to appreciate: trying to treat content as data. She's got some interesting horror stories.

[StratML is mixed content, as I understand the meaning of that term.  The purest data is contained in the performance indicators of the numeric type but the names of goals, objectives, stakeholders, values, and performance indicators can also be treated as data or at least meta-data.  By contrast, the content of the Mission, Vision, and Description elements is unstructured text, preferably phrases, sentences, or single, short paragraph.  However, the schema does not require terseness in those elements.]  

g. The description field is a good example of this. Allowing a single text() node is data-oriented. But in the real world (the Balisage doc is an example), the lack of any substructure creates unreadable data bricks. Large descriptions need to be broken into paragraphs, lists, etc. How about adding more XML layers instead of using funky punctuation patterns? This might also help with querying.

[Yes, the Description elements may be populated with excessive text.  However, if we ever get StratML back on a standization track again and we find that certain types of content are showing up in those elements, it would be appropriate to consider specifying additional elements for those types of content.  In the meantime, they are a catch-all for complexities that were unknown and could not be fully addressed in our initial KISS versions of the schemas.]

h. XML Schema  was developed for and is optimized for expressing rigorous data models. DTDs are still common in the documentation field as strong data-typing isn't as important and DTD syntax is also a modeling language. For many years, I've noticed that documents designed using XML Schema tend to be a little weird.

[Again, I'm not sure what you mean here but I'll look forward to seeing how you might be able to improve oXygen's StratML authoring/editing interface, assuming that the output validates against the schemas.]

To get a better feel for the design patterns, I built an authoring DTD from your schemas. It raises a few interesting questions:

.1. Why the multiple switches (doctype, document element, and @Type attribute)?

PerformancePlanOrReport kind of begs the question. How about a single doctype that supports three document elements: StrategicPlan, PerformancePlan, and PerformanceReport

[That is certainly a possibility.  However, if one were to be chosen, it might be the schema for Part 3 and I believe it is both too complex as well as untested to take on that role.  The realities of the standardization process also supported, if not required breaking Parts 1 & 2 apart.  Otherwise I doubt Part 1 would have achieved ISO approval or that we would have gotten around to getting Part 2 approved by ANSI.]

.2. Element names are a bit long (e.g., PlanCore instead of StrategicPlanCore since its also used in the performance context?)

[I'm not sure I fully understand your point here but it may relate to the point above.  There are two distinct but closely related types of plans -- Strategic Plans and Performance Plans.  The differences between them are specified in the additions to the Part 2 schema.]

.3. Identifiers are typically stored as @id attributes, especially for machine-generated IDs

[As indicated above it is neither possible nor appropriateto machine-generate all of the IDs associated with StratML elements.]

.4. There appears to be an error in MeasurementInstance. It allows unlimited children instead of a single Target/Actual pair.

[Each performance indicator may and often should have multiple instances of measurements of Target and Actual results.  For example, results are often reported on a quarterly basis, and targets may be changed (re-baselined) based upon experience (evidence that they are either too low or too high).]

.5. The Role/RoleType encoding is very unusual.

[Not sure what you mean here.  The concepts are fairly simple and straightforward, at least to me, based upon two decades of experience with StratML.]

"Integrating the procedural side" goes to this data/document value optimization question.

[Will look forward to learning more about how you might propose to optimize the value of plans and reports in machine-readable format.  To the degree that it may be difficult, if not impossible in the foreseeable future to change the de jure standard, the potential always exists for entrepreneurs to extend and add value to it.]

The Community engagement process is carefully sequenced and plans should reflect the order of activities. The stratml data models are limited in both order and expressiveness (e.g., Activity-Based Management performance models).

[Sounds like a propostion for value-added intermediaries to me.]

Increasing flexibility in the schema would make it more suited for general use, including authoring of strategic plans and performance documents. This might also accelerate adoption. Is there a customization layer that I haven't noticed?

[Again, I'm not sure what you mean here but will look forward to learning what you have in mind.  With respect to customization, Extensible is XML's first name and so long as applications and services can ingest and output valid StratML instance documents, the sky is the limit.]

I'll keep playing with the authoring DTD to see if any other ideas come to mind.

Let me know what your priorities are for the update. The phrase "available again" confuses me, a bit. I think that stratml frameworks have been in every recent version of oXml.

[While better authoring/editing tools will always be nice to have, my priority right now is developing a query service that leveages the semantics and structure of the StratML schemas for Parts 1 & 2.  For my personal usage, my XForms work quite well, with the exceptions noted above.  I use the Part 1 form almost every day.  There are >5K files in the collection and it seems to me that demonstrating the value of the collection is more important than improving the authoring/editing tools right now.]