Re: Keyspaces &/or DynamoDB for StratML Query Services?

[Owen Ambur]

Naval, issues like this are beyond my level of technical expertise, which is why I need to rely upon the expertise of folks like you, Jorge, and Pradeep.

In my technical development plan, I'd like to document the most important factors upon which our decision is based, under the Database objective.

Owen Ambur

https://www.linkedin.com/in/owenambur/

-----Original Message-----
From: Naval Sarda <nsa...@epicomm.net>
To: Owen Ambur <am...@verizon.net>; pradee...@ictect.com <pradee...@ictect.com>; jo...@vionta.net <jo...@vionta.net>
Sent: Fri, Sep 23, 2022 12:31 am
Subject: Re: Keyspaces &/or DynamoDB for StratML Query Services?

Hi Owen

AWS keyspaces based on Apache Cassandra is ment for huge dataset in terrabytes.

Strat ML initial dataset is about 250MB. It is kind of overkill to go for AWS keyspaces.

Regarding DynamoDB, it is based on throughout coming out of database.

For provisioned IO and 1MB throughput / sec of writing, about $0.65 and 3MB throughput / sec of reading about $0.65

In nutshell, if we are using indexes on specific columns, then cost will be miniscule.

But index are limited per table.

https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/SecondaryIndexes.html

If you want to search each and every field, then it will scan whole database which means too many read IOs needed

To avoid whole database scan, indexes is only way out and loading same data into multiple tables in the only solution.

So we will have query multiple tables to find the data from indexes to save cost.

It is sort of manipulation.

Biggest advantage of Dynamo DB is that you can scale like anything and it might turn out cost to be near zero for low volumes of searches. So your application is up all the time without paying any money if volumes are low.

I have not used Dynamo DB in real life situation to give justice if Dynamo DB will charge a bomb or is very cost effective. So you might look for alternate opinion.

Theortically I know most of the Dynamo DB concepts as I am certified as  AWS certified developer associate and AWS certified sys ops associate.

Naval

On 23/09/22 1:41 am, Owen Ambur wrote:

Naval, Pradeep & Jorge, while I hope that we can build on experience with BaseX, I am also open to exploring other alternatives.

I don't understand the pricing implications of AWS Keyspaces.  However, I suspect that usage of a StratML query service might initially be pretty low, and if it scales up to become costly under those pricing conditions, that might indicate the potential for profitably enaging paying customers.

Keyspaces came to my attention via a posting on LinkedIn by Jason Hunter, who spent >17 years with MarkLogic and is now Principal Solution Architect for DynamoDB.  I don't understand its pricing implications for it either and would be hesitant to take on a "free" account without knowing what the costs might turn out to be.

Owen Ambur

https://www.linkedin.com/in/owenambur/

[Owen Ambur]

Naval, as summarized under the Query/Discovery goal in my technical development plan, the basic query service should enable searches of these elements of the schema:

1. Stakeholder Name & Description
2. Goal & Objective Name & Description
   
3. Value Name & Description

While those are the elements of the "basic" search capability, please note that it is not exactly "simple" because:

• The Stakeholder element occurs at three levels of the schema -- Organization, Goal & Objective.
  
• Since the only distinction between Goal and Objective is a matter of timing, it makes sense to query those elements together.
• Similarly, since the queried term may occur in either the Name or the Description, it might make sense to combine queries of those two elements together as well.  For the goal/objective query, that means searching four elements at once: 1) Goal Name, 2) Goal Description, 3) Objective Name, and 4) Objective Description.

More advanced query capabilities will enable such things as faceted searches, e.g., searching for goals/objectives associated with particular stakeholders.

Owen Ambur

https://www.linkedin.com/in/owenambur/

-----Original Message-----
From: Naval Sarda <nsa...@epicomm.net>
To: Owen Ambur <owen....@verizon.net>
Sent: Tue, Sep 27, 2022 12:28 am
Subject: Re: Keyspaces &/or DynamoDB for StratML Query Services?

Hi Owen,

For the search fields, we checked the plan again. But we were not able to figure out the search fields needed for basic and advance search. If you can just copy and paste the same in the email, it will be great.

Naval

On 24/09/22 8:24 pm, Owen Ambur wrote:

Naval, I always enjoy having the opportunity to virtually meet those with whom I have established potentially productive relationships and I'll be happy to join you in a televideo conference when convenient for both of us.

Please be aware, however, that my hearing is not great and I have a hard time understanding Pradeep on our televideo conferences.  Also, I wouldn't want progress to be held up until we can find a time that is mutually convenient for an synchronous meeting of the minds.

So from that perspective E-mail works better for me.  It also seems to me that your usage of it has enabled you to make more rapid progress than he has.

On the other hand, it would be great if we could all work together, with each of us adding the value accommodated by our own interests and expertise.  So I'd welcome the chance to have joint televideo conference with both of you.

I'll be especially interested to learn what it is about my plan that is still unclear.  To the degree that it needs to be further fleshed out, I'd rather do that than merely talk about it.  It contains mailto Comment links to facilitate such focused dialog.

BTW, you can check out what Pradeep has done in the past at https://stratml.us/#Ictect

Owen Ambur

https://www.linkedin.com/in/owenambur/

-----Original Message-----
From: Naval Sarda <nsa...@epicomm.net>
To: Owen Ambur <am...@verizon.net>; pradee...@ictect.com <pradee...@ictect.com>; jo...@vionta.net <jo...@vionta.net>
Sent: Fri, Sep 23, 2022 8:57 am
Subject: Re: Keyspaces &/or DynamoDB for StratML Query Services?

Hi Owen,

We need to get on call once you are back from your vacations so that we can understand your needs better.

Naval

On 23/09/22 1:41 am, Owen Ambur wrote:

[Naval Sarda]

Hi Owen,

We have designed another basic search screen. Please see attached.  We can further refined the same based on your instructions.

Thanks for providing the information below which tells us which all tags and section needs to be searched behind the scenes. It is really helpful.

We will get back to you about advance search screens.

Naval

[Owen Ambur]

Good progress, Naval.  Here are my initial thoughts on your updated screen shots/wire frames:

1. It seems to me that it would be good to combine each of the two search fields for Goal/Objective, Stakeholder, and Value into one so that the simple seach screen includes only three fields.
2. Ideally, it would be good if the user could decide to query only one or two or all three of those fields at once.  However, that may be more technical complexity than we should take on initially.
3. Regarding the query results screen, I assume the pencil icon indicates an edit link, which is not feature I'd like to include in the basic query service.  However, at some point, I would not be adverse to providing such feature that opens the file in my XForm for editing and saving locally.  I don't want to get into the business of managing UIDs & passwords for authorizing edit access to the database.
4. With respect to the column headings, we'll need to give them further thought but the share of screen space devoted to the Description element should be increased.  That information will be the primary means by which users will determine which of the results they may wish to view in context of the documents in which they occur.
5. Depending upon which of the elements the user has queried, it should not be necessary to show that element in the column heading.
6. I assume the green down arrow designates a download feature, which may be OK, but if there is an icon that indicates viewing/opening a hyperlink, that would be better.  
7. In that regard, one issue to be addressed is the fact that my sitemap listing references the plain XML text renditions whereas the styled renditions should be presented when clicking on links in query results listings.
8. In some cases, PDF renditions are also available but they do no permit direct referencing of the queried elements, which is the primary purpose of the query service.

BTW, Hilton Head Island is directly in the currently projected path of hurricane Ian.  While we're currently in the North Carolina mountains, we've been scheduled to head home on Friday but will probably need to delay our return and will be a bit unsettled for the next few days