Transferring from HandBase

[Sean Marshall]

Hi

I am looking at alternatives to the apparently moribund HandBase, which I have used for many years.  I have a large number of databases set up, several of which are in the 10-20,000 record territory, and most are in parent/child type relationship with links to several databases from a master database.  For example, I have an aircraft set, which has a main database with all the details (type, registration etc), then child databases linked from there for sightings, flights, photographs, history (all on a one to many basis).

I have had a short play with Memento, but find the documentation frankly dire.  The wiki for example gives no indication of how links work (they are not even mentioned as far as I can see), and there is no formal user manual (I am old school and lie to read things rather than have pretty pictures!).  Regarding data import, HandBase would directly import .csvs and set the database up, fields and all, allowing easy import of spreadsheets to form the base data; Memento appears to require that the fields are set up first - and have to match - before a .csv can be imported, with again no indication of how link fields, calculated fields etc are dealt with.

Memento looks like it may be a good product, but poor documentation of its features (I can't even find a complete, non ambiguous list of field types, with references to 13, 14 and 19 in various places) makes me very nervous of investing time in what will be a major undertaking.

Cheers

Sean

[Bill Crews]

Memento is a very capable database tool and is quite likely able to do most of what you want. As I recall hanDBase from olden days, it had extensive forms design capabilities, and Memento's such capabilities are much more modest, but its fundamental are strong.

It happens that the wiki is just a few weeks old and is in its infancy. During this formative period, the references to the User Manual and the FAQ on the Main page are very important. Whatever documentation existed before the wiki is there. I'll copy those references toward the top of the Main page to make them more prominent. But Memento users have been and I would guess will continue to be people who are able and willing to learn by trial with help through this group. The wiki could begin to change that over time.

In the future, the Manual and FAQ will be integrated into the wiki. Even with the User Manual and FAQ, though, it is still true that the documentation is sparse. With the wiki, that is beginning to change, but again, we're just getting started.

There is a chapter in the User Manual on the Calculation field type and also on the JavaScript field type. The Calculation field has been a part of Memento for quite a while, and the JavaScript field type is new; right now both exist, but in the long term, maybe Calculations will become deprecated; that's just my guess. There is also a chapter in the FAQ on Calculations, with a subchapter on JavaScript.

Keep in mind that references through links to non-key fields in the referenced library are made using Calculation or JavaScript field scripts.

Link To Entry fields have also been around a while, though just recently it was expanded to support not only one-to-many references, but also one-to-one and many-to-many references. I can speak to one-to-many references, and I am writing that wiki page right now. To write the others, I'll have to do some experimentation first.

Please ask questions to the group or privately to me as appropriate, and we'll get them answered for you.

[Sean Marshall]

Thanks Bill.

First, can I ask if you are part of the development team or not?

I have no issues with learning by trial and error, and will be playing with this (and other) products to find something which meets my needs.  My concern over the lack of documentation is two-fold; one, I am looking to move from a product which has effectively died as the developer lost interest, and two as I mentioned I have a large amount of data which I really don't want to corrupt :)

As regards the first, if a developer cannot, or chooses not to, develop the documentation alongside the product then the commitment issue is there; while documentation is not the "fun" side, it is important.  Also, and with no disrespect to anyone who is willing to help, self-help groups allow the developer to wash their hands of any problem - "Well, we didn't recommend you do this, so we have no responsibility for any data loss" - ie, not our fault pal.  Re the second, the only way of importing .csvs seems to be through Google sheets and Memento cloud, which limits without subscription to three libraries, which would mean a huge amount of data re-input.  Memento doesn't seem to be able to import "raw" .csvs (Handbase can), and set up databases which can then be edited.  The desktop version, where it is easier to manipulate large datasets also seems somewhat crippled in the Lite version.  Given the amount of data I have, these issues are significant to me.

My current back-up is via Dropbox (free), with files backed up automatically with Dropsync.  It is not clear if I could use this strategy with Memento, or be forced into a paid subscription for Memento cloud, which would be a negative.

As I said, I will continue to play around and see what I can achieve, but the thought of having to re-build several database groups each containing 10,000+ records manually is not one I relish :)

Regards

Sean

[Eugene Kartoyev]

1. This is a user group, not programmer's. But some users are advanced long-time users and programmers themselves - therefore they can help you with formulas/structures/techniques. Memento developers actively respond during the issue of new versionsto fix bugs and explain overall features. If you spot a bug, you can contact them via email. Currently, Memento is a living project. But you can never guarantee that developers don't lose interest in their project or don't die.
2. No developer would take reaponsibility for loss of your data. You use any application and device on your own risk, unless you manage somehow to force a developer to sign a contract with you to keep your data safe.
3. You can back up data (including attachments, like photos, pdfs, etc.) via a third-party backupper - onto Dropboxand others. You can fully restore the database on your device or on a different one. Data, settings, designs are all kept in an accessible folder, specified in the database options. No limits for that.
4.Google sheets is the best way for entering multiple data after you draw the library structure. It is also the best way to provide and edit URIs to external files (photos etc). CSV - is not the strong part of memento.
5. If you worked with databases before - you definitely shouldn't suffer from the lack of information. Memento's most functions are self explanatory and internal help is detailed enough to start using its full capacities after "playing" with it for a while. Alternatively, you can ask or search for information here. This forum is exhaustive and provides answers to 99.9% of questions about memento's functions.

[Bill Crews]

First, ditto on everything Eugene said. I am also just a user, but I've occasionally wondered if any developers might be reading or even posting in this user group, and my most recent guess was Eugene, but now I can cross him off the list. Vasya may occasionally read in here, but I really don't think so in general.

My guess/assessment for what it's worth, is as follows. This product was pretty young when I ran across it, and over the time since then, it has continued to be actively developed. Early on, the version I used was free, and the only attempt to monetize was the Pro edition (no cloud in those days), which is a one-time charge thing, so it was mostly a labor-of-love kind of thing. With the cloud and added Desktop Edition, he is now starting to monetize for real.

I say "he", because from our perspective, it appears as if Vasya is programming everything himself. I expect there is a team, but I don't know that, and it is clear he is the focal point of everything. He has his following, and he dedicates all his time to development, so documentation has so far been only as really needed, driven by user feedback.

It has been interesting to see the direction of development. He has developed along classic relational database lines showing real knowledge of that -- something you don't see much from mobile app authors -- but the features he implements are driven by usability for mobile users -- something you don't see much from real relational vendors. That's what I liked about hanDBase in the old days, though I ended up a SmartList user back then.

Memento is still in its high-growth stage, with features and platforms coming out pretty fast. But yes, that could change at any time. I gather he may be starting to make money off the Desktop Edition with business users, and that could help to keep this product development cycle running.

As for documentation, I also consider the Manual and FAQ to be pretty paltry, but most things that would be hard to figure out from trial and error do seem to find a mention there. Now, he has started the wiki, and for the moment, he has commissioned me to get that going; I'm sure there will be others contributing before long, and ultimately it may become a very user-developed thing, like Wikipedia, though with the Manual and FAQ integrated, he'll always have a proprietary interest in its quality and accuracy.

Finally, note that in certain areas, the product has embedded help built in. In scripting a Calculation field, for instance, press the buttons to enter a field or function, and you'll have assistance regarding the available functions, what arguments they take, and what results they return.

[Bob]

Sean,
I have been following Memento with great hopes that it would turn into a full featured relational database.  I have seen many features and functions added over the past 3 years.   I decided to post to your thread because you seem to have articulated my main question and problem with Mememto.  That problem being that while it may be "relational" in the sense that it can "link" entries together when manually records are manually selected it does not relate the tables on any user created and populated field.  Which means that every time you set up a new Library you must recreated all the "links" between records/tables.  In a traditional database, key fields are identified and joins are made between those fields.  Common data with exact matches of data in those fields is what creates the link and provides for a "one to many" or "one to one" relationship.  Inner and outer joins will determine if the resulting queries are inclusive or exclusive of non matching data in the Master table.   The data in those fields "key fields" is populated by the database creator or user. 

As far as I can tell, Memento must use a background set of fields to make the "links" between tables.  There seems to be no way to import already populated fields which would then be used as the key fields, (or "link to entry" in Memento parlance), which would be used to establish the joins between tables.  I have been toying with the idea of using Memento to develop a small business application for my sister's business.  The application would require importing Customers, Inventory and Delivery Routes into the database.  The relationships between those tables has already been established by common data in key fields (columns).  This would be an impossible task if we have to go into each record and establish the "link" again individually between records.

The real attraction of Memento is the mobile functionality and the fantastic set of features but the deal breaker is that there is no way to import an already developed set of data... similar with the problem you are going to have trying to implement your project using Memento. 

Bill or Eugene, I would welcome your insightful input here to point out if I am missing something and am wrong in my assessment of Memento.  This problem has been described in many different ways in this forum and I have never seen an answer...Maybe there is some way to use calculated fields to establish the joins on user created data?   I am really hopeful that someone here can come up with a solution to this problem.

[Bill Crews]

Bob, I believe you are correct about that. There may be some workaround involving CSV files and synchronization, but the overall situation is that the scope of JavaScript and Calculation fields is intrinsic to the field itself. You can reach across a link in a script to reference fields on the other side, but that's about it.

I don't know Vasya, but he seems to me to be driven by a combination of building the core UP so as to support more features and then implementing OUT those features. I think I know him well enough to make sure he reads this thread, and if I get any response, I'll share it, but I'm not sure there are enough of us fully-relational types out there to incent him to support these use cases. I'd focus on the use cases themselves, as you did in your post. He is moving in the direction of more JavaScript support.

[Eugene Kartoyev]

As I understood, the problem is about importing somebody else's linked data.
I wonder if this technique seems helpful. It will be easier for me to make an example:

Suppose I have customers,
who pay me in dollars.
For tax purposes, I need to calculate my income in rubles.
using the exchange rate of a certain date, which is valid for 30-40 days.

I build two libraries:

DATABASE1

Date of payment
Customer ID
Dollars paid
Exchange date --> link to (Database 2).date
Exchage rate <-- extracted from Database 2
Rubles taxable - calculated (dollars x exchange rate)

DATABASE2

Exchange date <-- key field looked up by Database1
Exchange rate <-- extracted by database1

My friend wants me to make the same table for him, and brings me his 10 000 pieces of data that I need to import it into this type database.

I'll do the following,

Sync both of my databases with Google sheets.

Populate DB1 and DB2 with my friend's data *in Google sheets*

Resync the DB2, and then DB1.

All the links between the records will be established automatically without me having to manually re-establish the link in each entry.

[Bob]

Yes, I agree that this direction is not one that the developer is interested in going.  I asked a similar question a few years ago and got nowhere...

It seems that the "link to entry" functionality does not make Memento a relational database but more like a Lookup Table in Excel where it just copies or displays data from another spreadsheet based on the cell location.   When you click on "Relationship" in Memento all it does is create a new "link to entry" field in your table.  Nothing happens in the other table.  Then when you enter a "link to entry" in that new field all you are doing is pointing to a particular record in the other table.  If you set the field to "one to one" it wont let you point to another record.  If you change it to "many to many" you can point to multiple records in the other table.  This allows you to jump between the tables to a particular records but you must manually identify the link.  The only way it could do this is by using some hidden fields containing record numbers in each table that keeps track of this link.  That is why you lose the links when you export the files.  If we had access to these fields we could then import common "key field" data from other sources. 

[Bob]

Eugene,
I love your positive optimism and examples!

What you are saying is that your join is between your "key field": Exchange Date.  And that this field is in both Libraries as a "link to entry" type field.   And that you are able to link those Libraries with Google sheets and the "link to entry" fields are populated with the Dates necessary to link the records together? 

Have you tried what you propose in your example?

Does it work?

[Eugene Kartoyev]

It did. 15 minutes ago. :) The extracted and calculated fields got populated automatically from the other table based on the key field (link to a unique FX field). Link in Memento was established from the data of Google sheets - I did not select them manually.

[Bob]

Eugene,
I have tried what you suggest with no results.

I set up 2 test libraries with a common field called "BoxID"
I made one of the fields a "link to entry" field and set BoxID in the other Library as the "Name" field.
I syncd with Google Sheets.
I then populated the sheets with data and syncd again... no luck.

I would like to be sure I understand your procedure.

When you say you did this:

"Exchange date  --> link to (Database 2).date"

How did you establish this link?  Using "link to entry" field type?  Or calculation?

When you say you did this:

Exchage rate <-- extracted from Database 2

Using "link to entry" field type?

[Eugene Kartoyev]

Just wait 5 minutes, I'll write you the step-by-step instruction. It worked perfectly for me.

[Eugene Kartoyev]

1. Create DB1
2. Create integer field called "ID1" (it will be your unique key)
3. Create String field called "Mydata" (it will be your data)

4. Create DB2
5. Create Link-to-entry field called "ID2" and set the link to DB1.
6. Create Calculated field called "extractedData"
A. Set the formula '#{id2.mydata}'(you can use the insert field button) --> it will use ID2 to extract field "mydata" from the reference therein
B. Don't forget to set the result output to string type

7. Sync DB1 to Google Sheets
8. Sync DB2 to Google Sheets

It will ask you to set unique ID for a linked field - agree.

9. Go to Google Sheets.
10. Look for your DB1 and DB2 there. You may have to update the view by swiping your finger from top to bottom of the screen, if you don't see the databases DB1 and DB2 there.

In GOOGLE SHEETS in cells
11. Populate DB1 as:
Id1= 1 Mydata = Hello
Id1= 2 Mydata = Bye
Id1= 4 Mydata = Wow

12. Populate DB2 as:
Id2=4
Id2=1
Id2=2

Don't touch MEMENTO_ID field or extracted data field.

13. Go back to Memento

14. Sync the keys first -- DB1
15. Sync the extracting base -- DB2

Open DB2 and see the contents of records. Links to records 4,2,1 and data exctracted therefrom will alraedy be there.

[Eugene Kartoyev]

I mean, when you populate cells, dont write "ID1 = 1"
"ID1=1" means, that you need to enter 1 in the cell under the column "ID1"...
I hope, my style of giving directions is not confusing

[Eugene Kartoyev]

I've played around with this method of importing third-party data, additional conclusions are:
1. The DB structure must be designed before import, or an existing db be used.
2. The referenced DB must have a unique key field and this unique key must be set as "entry name" (set in the library settings).
3. The structure and the data must be synced to a new google sheet table before adding new entries via google sheet.
4. After syncing - you may restructure you db output,e.g. hide the unique entry field, change the Entry name field etc.

And yes -- you can import tables with one or more links to other tables into Memento - no problem,
as long as the db structures are prepared and synced to new tables in Google Sheets before entry of new data,
and the key fields are set as entry names and unique and in referenced tables, where data is extracted from.

This method to me appears even better than in standard databases, because you can combine several fields by setting them as entry name - thus making a unique entry name (=key field) without needing to create a separate keyfield with a numeric code. Though, you can, if you want to.

Therefore, Memento is fully relational, but has a bit different (I'd say more universal) logic for setting up key fields for adding external data than other databases, making unique field identifiers optional in imported data.

There is a problem with some users who think that google sheets are for exporting memento tables to other phones.
They are not. Google tables are intended for 2 purposes -
1. adding new data from third party sources and
2. changing field types in existing tables (for example, change integer / calculate type to text type) - this used to be documented before. I don't know if it is in Wiki.
GT can also be used to enter custom data with URI's to photos, for example...

[Sean Marshall]

Thank you for your robust - some would say aggressive - response.  You must excuse me as a potential user from researching things before I commit to what would be a significant amount of work to migrate from HandBase to another product.  Clearly, that kind of approach is not appeciated by some.

To deal with your points :

1.  I am aware of that.  However, I queried Bill's role as (good) developers often take an active role in such groups, and I was simply trying to ascertain with whom I was dealing with.  I am also aware that all developers can lose interest; hence my concern at the paucity of documentation.

2. I wasn't expecting them to.  However, if acting on advice from a third party (ie an unoffical user group) screwed up the program to an extent, the developer could (quite rightly) say not my problem, rather than addressing any such issues to prevent a recurrence.  I back up data regularly; it is not the loss of data I was getting at, more the potential for preventing future issues.

3.  Thank you.  That is not clear from any of the online documentation.  However, this does not apply (as far as I can seen from experimentation) to the desktop version, which requires log-in to the Memento cloud and has no option for local or alternate storage.  It also has a limit of three libraries without subscription - I would need at least 8.

4.  So I would need to export to .csv, import to Google sheets, design the field layout of the library in Memento, export that to Google sheets, copy and paste the data, then re-import to Memento?  Not much potential for things to go wrong there, especially with between 3 and 10 links on my current databases (to multiple databases including ones which would be in separate libraries).

5. You seem to have an issue with my use of the word playing (even though you subsequently use it in your post of 20th August).  Maybe English is not your first language, but in colloquial English it is used as a synonym for trying out, experimenting, and infers no levity per se. In any event, while the group may provide answers, I still think it is incumant on a developer to provide   

Another issue I have subsequently come across with the desktop Lite version (ie used for trial purposes to ascertain suitability) is that the import from .csv is disabled.  Manipulation of large sheets, while possible on mobile devices, is far from ideal, but trial users cannot see how the more convenient desktop approach works.  I am unwilling to enter a subs agreement for storage of my data prior to confirming suitability of the software, nor with an apparent one man band with no kind of guarantee over data safety or security.

Given these issues, and the tone of your response to a potential user, I think I will be looking elsewhere.  Thanks for your time.

Sean

[Eugene Kartoyev]

Hello, Sean! Indeed, English is not my native language. I am Russian and we also use the word "playing" for "testing", but we use quote marks when we use words in their figurative sense. I did it reflexively when I enclosed "play" into quotes, so there was no intention to taunt you. But in any case – it doesn't sound mature to base your decision based on some user's seemingly aggressive tone. I'd try to separate the wheat from the chaff. 

Oops. Sorry for another offense. But, let's dismiss that as irrelevant to the issue of Memento's functionanlity

Before I chose Memento, I played with all dabases I could find on Google Play. Handbase was a "no" from the beginning because 1) I doesn't support Unicode and therefore you can only use English letters with it according to complaints of other users. This drawback limits Handbase's market to only English-speakers . 2) A trial version to test it does not exist in the Russian play maket domain. 3) Users say that Android version of Handbase is much inferior to that of iOs. 4) Besides, as you have mentioned, the project is moribund. It is another "no" for me.

The only adequate alternative to Memento I could find was Sail Forms. It has very generous form creation capacities, but 1) the database itself is hidden in the internal "data/data" folder, which is not accessible in an unrooted device. The developers declare that "it is our policy to keep the database data hidden from the user", without explanation why they'd do that. There is no way to backup data independently. 2) Because the data is in the internal memory, you can quickly run into "out-of-memory" issue, and you are not able to keep your database on a bigger external Sd memory. 3) If you construct a bit more complex forms - the database simply crushes. 

Absolutely same issues with MobiDB Database. It may crush for example, when you create text fields and fill it with data which it deems inappropriate. There is no way to export form designs to other devices, and you are bound to one device only.

All other databases I could find were nothing as close to Memento. So, I won't discuss them.

Memento: 

You can import linked databases using Google Sheets, but you must design the form first.

Library designs are standardised and will work correctly on any Android device.

You can store your database data wherever you want to and fully backup/restore data at any time on any device.

It is cheaper than other databases.

It offers a javascript capacity which makes you almost almighty with respect to data manipulation.

You can backup attached files together with the database – like photos / music / pdfs (!!!). You can Dropsync and stuff.

Memento provides more than 3000 preset databases that you can consult for techniques or modify to your needs. 

You can propose your ideas to the developers in Uservoice.

Memento has a strong user community that will help you with the implementation of functions that cannot be documented per se – like JS formulas, for example.

It is very unlikely that Memento is going to die in the near future, given that the developers started monetizing on it with additional cloud-based functions.

Paid cloud functions are realizable outside the cloud for free if you are able to migrate data via backup and import/export options.

The only weakness I found was a messy CSV import of unicode data. But, that's the problem of the CSV engine in most applications, which I fully understand, because the CSV format is a legacy, existing since my granny's youth, and doesn't support Unicode data. Another problem with CSV is keeping pictures... To cut it short – the good news is that Memento provides excellent opportunities of importing data with the help of Google sheets.

The desktop version of Memento is less than a year old. (Bill can correct me if I am wrong). It is an absolutely new feature and may be unstable and limited. The developers have asked to send all the reports from desktop because it is still under development. You can also contact developers via e-mail. In any case, Android is more important than Windows to me for keeping my mobile data always with me, and, just in case, I can again use Google Sheets to stuff my database with desktop data, so I don't use it.

As for data safety issues – it depends solely and exclusively on the user, no matter how many men there are in the band. Google addons – which is a crowd-band enterprise – is an example of how they just delete projects that they deem unprofitable by sending a 6-month short-notice to users that they need to migrate data or it would be permanently lost. People do lose data everywhere. Besides Google messes up designs in their projects with every change in their policy, – like, the latest one was limiting the number of colors in Google-Keep, because their material-design designer likes it so. You cannot roll back to the version you like.

To avoid such issues, you can backup my Memento data to a hard disk. You can extract the working Memento apk (install file) and backup it too, in case there is an untolerable bug in a new issue of Memento, and you can roll back to the version you like, provided you have the skills to do that. But, it may be somewhat hacky, so it won't be documented for you.

The information about Memento's functions is not well organized yet, because all the information is being systematized and migrated by Bill into a Wiki site. In any case, one can't complain that information is totally missing, because you can find answers to all questions in this forum. 

Be that as it may, no one will be able to convince you of Memento's unbeaten advantages if you keep up your negative stance only based on your surmises about what might befall Memento's project some day (cf. item 2 of your message) or on a user's tone. However, if you ever find an objectively better solution, I will be very grateful to you if you share your search outcomes.

[Bill Crews]

Thanks, Yevgeny, for a mature and helpful post! I'll try to refrain from such personal comments, in general. They distract from the task at hand.

After the wiki is mostly booted up, I hope still to be involved, that others will start joining in, and what I will start doing is mining the tips, techniques, and other information from the user group postings like this one and getting it into the wiki, as well, being careful to do version-shifting to make sure it is up to date, or at least documenting the age and version/platform of the information. In addition to the manual-ish parts, the wiki should contain such user-experienced wisdom, in my view.

[Eugene Kartoyev]

Thank you, Bill, for describing your plans. I seldom use help now, because I had read it all through along with this forum before wiki existed 

and I've even created a personal tips library for Memento.

I got somewhat disappointed when help was replaced with Wiki, but I do realize now that Wiki has improved considerably recently

and is rapidly becoming a full-fledged reference site. 

Including user tips (like how to calculate days from now(), how to import linked data, how to store db on external SD...) is a great idea.

When you start working on that, I will be happy to take part. Though, I wouldn't join directly because I'm not good

at formulating instructions in English. (I can end up with Russglish, when in a hurry). I'd rather contribute to translating, too. ;)

[Bill Crews]

Excellent! Most folks are not retired and without a life, like me. :-)

There's much to do. Working...