Trying to understand if there is a demand on automated data exchange using Memento API

[Alex Sk]

When using Memento Database and other database (ERP, CRM, Project Management or any business management software) at the same time, there is a need to exchange data between systems as usual. One of the possible ways to exchange data with Memento Database libraries is to import and export using CSV files. Despite Memento Database is just brilliant in many aspects, the regular Memento functionality with CSV files is quite limited, so I use an automatic exports and imports via CSV using API.

I am wondering if there is a need for this in the Memento community.

[Bill Crews]

Maybe, yes. What is it? A JSON export/import? Is it general or specific to a purpose?

On Thu, Jul 25, 2024, 1:37 PM Alex Sk <kuo...@gmail.com> wrote:

When using Memento Database and other database (ERP, CRM, Project Management or any business management software) at the same time, there is a need to exchange data between systems as usual. One of the possible ways to exchange data with Memento Database libraries is to import and export using CSV files. Despite Memento Database is just brilliant in many aspects, the regular Memento functionality with CSV files is quite limited, so I use an automatic exports and imports via CSV using API.

I am wondering if there is a need for this in the Memento community.

--
You received this message because you are subscribed to the Google Groups "mementodatabase" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mementodataba...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/mementodatabase/4e25a782-29a0-472f-a767-fc847594b297n%40googlegroups.com.

[Alex Sk]

Hi Bill,

First of all, I’m happy to see your response, as I’ve been reading the Memento group for several years (although, I’ve never participated in discussions) and respect your opinion and appreciate what you do for Wiki etc.

It’s not a JSON export/import but pure CSV file. I took a full compliance with the Memento Android export CSV file as a goal (with all its commas and dots), and my export files match fully, so if a third party software accepts Memento CSV, mine will suit without a fail. The same applies to imports. Honestly, the export is not a big deal, import is another thing, as long as it intends to deal with attached files, embedded objects etc as well.

Yes, it’s very general and works with all Memento field types, until it touches on some specific API features with two or three field types.

I will dare to list some features:

• Supports Memento Cloud API and Memento Private Server API.
  
• Scheduled exports and imports.
  
• All field types including embedded, attached files (image, signature, audio and file) field types.
  
• Smart import: only modified fields and entries update, not every entry and field of a library from incoming CSV file en masse. This avoids needless requests to the database.
  
• Exclude any field if necessary.
  
• Link-to-entry support when export and import.
  
• Logging and notification.
  
• Able to perform customized data processing and verification.
  

I would plan some improvements (field names mapping, filters, etc.), but for me they're not in need, only if there's any demand from anybody.

пятница, 26 июля 2024 г. в 05:30:04 UTC+3, bill....@gmail.com:

[Bill Crews]

That sounds excellent to me, as I know there are many people in the forum that depend heavily on CSV Import & export, though I'm not one of them. It's even more excellent that you are willing to share a product in this space that can help people. Some comments & questions...

• I've never used Memento Private Server. It's been "out there" for years, but is it a released product? What can depend on it? Can people who don't use Private Server use your product?
• The last time I looked (not long ago), the Memento Cloud API was still in beta testing status for a very long time (years). Is it released yet? Can we depend on it?
• There are many CSV file formats in the world, all similar, but not interworkable, some using double-quotes & some not, and other variations. I think most people are importing from either Microsoft Excel or Google Sheets, with Memento mainly supporting Google Sheets. What should people, especially those importing from other databases or other sources, know about the file format supported?
• There are many forum members located around the world, so regional/international settings & standards are important. In particular, date formats vary by region, and there are various currencies used. What is there for users to know about regional/international support?
• When I first started with Memento in 2015, there was no Help, no Wiki, but only out-of-date FAQ & "manual files". Now, Help is managed by Memento, and the wiki is precariously attempted to be managed by a user or two, maybe with some help from the developer (someone please save us!). The template catalog is largely unmanaged, as well. How will people learn about and understand how to use your product?
• What shall we call your product? Does it have a name? Where would it live? As a JavaScript Library that can be added in the Memento script editor?

It sounds to be excellent. I hope you continue to do this. I also hope others will add their comments & questions to this thread, as I know interested parties must be out there. Maybe some could be helpful to you. Thank you in advance.

To view this discussion on the web visit https://groups.google.com/d/msgid/mementodatabase/d0dcd77a-c637-4b65-b4d5-58233c937c40n%40googlegroups.com.

[Alex Sk]

To start with something, although this question does not sound from you — I do not plan to make any money from this for now especially here from people in the forum, in the same time I do not plan to share a source code. I developed and use this for several of my customers that use MDB for business purposes. It is hard for me to imagine now, if a demand has a potential to transform this to a web application with UI, subscription plans etc. Honestly, I’m not sure.

>> Does it have a name?

Not yet. I’m not an expert in naming, let’s call it SEM (from Scheduled Exchange for Memento; but this is beta, I wouldn't like to delay my answer to you because of a naming tortures), so far no one has come up here with a better name for it.

It’s a script that lives on a VPS and could be scheduled. A number of presets could be configured (for instance, for a customer, or a customer's project, or a somebody from the forum with Project1, Project2 etc), each include basic setup (database address, access token), library names intended to exchange and whether they are to be exported or/and imported. It uses FTP currently for export CSV files to and import from, so FTP-server access should be set up also (address, user and password). Plus, data exchange sessions schedule. If I haven't forgotten anything that is enough in general for a typical case – to export or import of all the fields of the libraries. Then, if there is a need in an import/export customization (to exclude some fields, attached fields directories etc.) it needs additional setup, but it is not too complicated.

SEM just exchanges data between CSV files and a Memento Database via API, it does not store any data from libraries. It deletes incoming import files from FTP server or could move them to archive directory if necessary. For a VPS I currently use a data centre in Germany, but nowadays it could be almost anywhere including the North America etc. There is no any limitation about FTP server – it could be a database owners’ or mine or a third party.

Yes, some sensitive data has to be set up, but a Memento Database administrator (there is no need for SEM to know admin or user logins) can always delete an access token and close FTP access for the given username.

>> I've never used Memento Private Server. It's been "out there" for years, but is it a released product? What can depend on it? Can people who don't use Private Server use your product?

The Memento Database developer sets the licensing rules and terms of use, I have nothing to do with it. I’ve got an experience of working with Memento Private Server as long as my customers acquired it legally. The Memento Cloud API and the Memento Private Server API are identical in terms of methods, responses etc. The Memento Cloud API has requests limits which could be found in the Memento documentation; the Private Server API does not have requests limits.

So, in general people who use Memento Cloud can use SEM.

>> The last time I looked (not long ago), the Memento Cloud API was still in beta testing status for a very long time (years). Is it released yet? Can we depend on it?

The Memento documentation says the Memento Cloud API is still beta and in my experience it has some limitations, but you can get along with them. The same applies to the Private Server.

>> What should people, especially those importing from other databases or other sources, know about the file format supported?

A field separator by default is comma (could be altered in SEM setup), Memento also uses a quotation marks “. People could just make an export to csv of any library and study the result file. Comma-separated values format has some specifications, please forgive me if I will not list them here.

>> regional/international settings & standards are important

Yes. Desired date format, decimal separator and currency delimiter are also customizable in SEM. Memento does not export a currency code in csv file, so SEM does not as well by default.

>> How will people learn about and understand how to use your product?

I've not composed neither readme nor Wiki yet, I’m just trying to understand if there is any demand on it.

>> Where would it live? As a JavaScript Library that can be added in the Memento script editor?

I tried to answer this question at the very beginning, from which I could assume it cannot live on the user's device — at least because it should be scheduled and not depend on the activity of the user's device itself.

I tried my best to respond you Bill, but there are two of us here for now.

пятница, 26 июля 2024 г. в 15:44:11 UTC+3, bill....@gmail.com:

[Bill Crews]

Very good response, Alex. Let's hope others respond to let you know there really is interest. I assume VPS means a virtual private server? I get that it's a server app running somewhere to be determined. Good luck with this. 

On Fri, Jul 26, 2024, 11:50 AM Alex Sk <kuo...@gmail.com> wrote:

To start with something, although this question does not sound from you — I do not plan to make any money from this for now especially here from people in the forum, in the same time I do not plan to share a source code. I developed and use this for several of my customers that use MDB for business purposes. It is hard for me to imagine now, if a demand has a potential to transform this to a web application with UI, subscription plans etc. Honestly, I’m not sure.

>> Does it have a name?

Not yet. I’m not an expert in naming, let’s call it SEM (from Scheduled Exchange for Memento; but this is beta, I wouldn't like to delay my answer to you because of a naming tortures), so far no one has come up here with a better name for it.

It’s a script that lives on a VPS and could be scheduled. A number of presets could be configured (for instance, for a customer, or a customer's project, or a somebody from the forum with Project1, Project2 etc), each include basic setup (database address, access token), library names intended to exchange and whether they are to be exported or/and imported. It uses FTP currently for export CSV files to and import from, so FTP-server access should be set up also (address, user and password). Plus, data exchange sessions schedule. If I haven't forgotten anything that is enough in general for a typical case – to export or import of all the fields of the libraries. Then, if there is a need in an import/export customization (to exclude some fields, attached fields directories etc.) it needs additional setup, but it is not too complicated.

SEM just exchanges data between CSV files and a Memento Database via API, it does not store any data from libraries. It deletes incoming import files from FTP server or could move them to archive directory if necessary. For a VPS I currently use a data centre in Germany, but nowadays it could be almost anywhere including the North America etc. There is no any limitation about FTP server – it could be a database owners’ or mine or a third party.

Yes, some sensitive data has to be set up, but a Memento Database administrator (there is no need for SEM to know admin or user logins) can always delete an access token and close FTP access for the given username.

>> I've never used Memento Private Server. It's been "out there" for years, but is it a released product? What can depend on it? Can people who don't use Private Server use your product?

The developer sets the licensing rules and terms of use, I have nothing to do with it. I’ve got an experience of working with Memento Private Server as long as my customers acquired it legally. The Memento Cloud API and the Memento Private Server API are identical in terms of methods, responses etc. The Memento Cloud API has requests limits which could be found in the Memento documentation; the Private Server API does not have requests limits.

So, in general people who use Memento Cloud can use SEM.

>> The last time I looked (not long ago), the Memento Cloud API was still in beta testing status for a very long time (years). Is it released yet? Can we depend on it?

The Memento documentation says the Memento Cloud API is still beta and in my experience it has some limitations, but you can get along with them. The same applies to the Private Server.

>> What should people, especially those importing from other databases or other sources, know about the file format supported?

A field separator by default is comma (could be altered in SEM setup), Memento also uses a quotation marks “. People could just make an export to csv of any library and study the result file. Comma-separated values format has some specifications, please forgive me if I will not list them here.

>> regional/international settings & standards are important

Yes. Desired date format, decimal separator and currency delimiter are also customizable in SEM. Memento does not export a currency code in csv file, so SEM does not as well by default.

>> How will people learn about and understand how to use your product?

I've not composed neither readme nor Wiki yet, I’m just trying to understand if there is any demand on it.

>> Where would it live? As a JavaScript Library that can be added in the Memento script editor?

I tried to answer this question at the very beginning, from which I could assume it cannot live on the user's device — at least because it should be scheduled and not depend on the activity of the user's device itself.

I tried my best to respond you Bill, but there are two of us here for now.

пятница, 26 июля 2024 г. в 15:44:11 UTC+3, bill....@gmail.com:

That sounds excellent to me, as I know there are many people in the forum that depend heavily on CSV Import & export, though I'm not one of them. It's even more excellent that you are willing to share a product in this space that can help people. Some comments & questions...

To view this discussion on the web visit https://groups.google.com/d/msgid/mementodatabase/ba8a1eb5-495e-4a68-8f6b-2e3353d6ebcbn%40googlegroups.com.

[Alex Sk]

Bill and other who may read this,

I’d like to clarify the point of altruism.

I definitely don't know what kind of demand I'm dealing with. For me not to become servicing dozens of setups (=time), tons of attached files (=server traffic) etc., I’d like to mean the completely free two libraries per project in basic exchange (all fields with no customization) with no fields of attached files file types two times a day. I would definitely handle this level, the rest will depend on the situation.

пятница, 26 июля 2024 г. в 19:19:38 UTC+3, bill....@gmail.com:

[Alex Sk]

>>   assume VPS means a virtual private server? 

Yes, that's right.

пятница, 26 июля 2024 г. в 20:14:50 UTC+3, Alex Sk:

[Alex Sk]

If I understand correctly, you are now considering copyright infringement as an altruism.

суббота, 27 июля 2024 г. в 14:10:51 UTC+3, Mmm:

Лучшим проявлением альтруизма был бы частный сервер для новых версий Memento. 

https://4pda.to/forum/index.php?forums=316&topics=184204&act=search&source=pst&query=%F1%E5%F0%E2%E5%F0

За него можно благодарить не только словами.

пятница, 26 июля 2024 г. в 20:15:31 UTC+3, kuo...@gmail.com:

[Mmm]

Не увидел причину нарушения авторских прав в своем предложении. 

1. Работа с частным сервером предусмотрена программой. 

2. Сервера для новых версий нет.

3. Отсутствие сервера не подразумевает его запрет. В противном случае смотри п.1.

С Вашей точки зрения и работа с API нарушение, автор же не предоставил этот функционал в своей программе.

Да, он работает над этим. Но поручал ли он Вам выполнять что-либо в этом направлении.

 

Еще скажите, что нельзя использовать дополнительные сторонние библиотеки  JavaScript , кроме встроенных.

Поэтому, если можете - сделайте. Если Вас беспокоят сомнения - пообщайтесь с автором. Возможно он не против (...).

суббота, 27 июля 2024 г. в 15:57:52 UTC+3, kuo...@gmail.com:

[Alex Sk]

It seems you are basing your assumptions about my motives on your own conjectures. Besides you want to motivate me to take some action with your manipulations.

I would like to end my communication with you by noting that your messages are not related to the topic I created.

суббота, 27 июля 2024 г. в 16:33:41 UTC+3, Mmm:

[Mmm]

>>> I will dare to list some features:

• Supports Memento Cloud API and Memento Private Server API.

Я предложил основываясь на Вашей записи. Как говорится - начнем "от печки".

Спасибо...

суббота, 27 июля 2024 г. в 17:00:18 UTC+3, kuo...@gmail.com: