Groups keyboard shortcuts have been updated
Dismiss
See shortcuts

Re: StratML Import file feature through FTP is ready

1 view
Skip to first unread message

Owen Ambur

unread,
Mar 7, 2023, 2:36:31 PM3/7/23
to Naval Sarda, aboutthe...@googlegroups.com
Naval, this isn't exactly what I was looking for.  I'd rather not have to take the additional time and effort to create a text file with a URL(s) in it for uploading via FTP -- particularly when I'm already maintaining a sitemap listing of the files.  

I'd prefer to be able just to enter the URL(s) in a field on the site itself.  That's what I thought you were working on.  It would be good if we could have a clearer understanding about matters like this before your developer spends much time on them.

However, I tried uploading a .txt file just now with the URLs for three files that I have been unable to get to appear in the query service in recent days, using the previous FTP method.  It seems like that method is not working anymore.  Has it been disabled?

Also, are the FTP sign-in credentials different for the .txt file listings of the URLs than for uploading the regular .xml files?

Does it matter what the .txt file is named?

The three files whose URLs I just upoaded don't seem to have been indexed yet.  Does the indexer still run every half hour?  Would it be a problem to have it run immediately when a URL(s) provided, particularly since I am the only one who should be using that feature in the near future?



On Monday, March 6, 2023 at 07:15:46 PM EST, Naval Sarda <nsa...@epicomm.net> wrote:


Please see below



-------- Forwarded Message --------
Subject: Re: Fwd: StratML Import file feature through FTP is ready
Date: Mon, 6 Mar 2023 21:53:01 +0530
From: Sudarshana <sudar...@epicomm.net>
To: Naval Sarda <nsa...@epicomm.net>
CC: jitendras <jite...@epicomm.net>



Owen ,

Import file feature with URL is ready and uploaded on server.
You can upload text file with standard file name on ftp given below. Add urls in file, each URL on separate line. (Sample file is attached)
File name should be fixed.
Program will fetch file from that URL and index file for searching. If that URL is already present then that file will get overwritten. If URL is different and filename is already present in indexing then separate copy of that file will get indexed.

Host - ***************
Port No - ************
Username - ************
Password - *************
FTP User Directory - /

On 1/11/2023 8:51 PM, Sudarshana wrote:

Owen,

With reference to your below email we can suggest you one solution.
Now the system is expecting files but now you want to add a URL so what we can do is, we can download a file from that URL and insert that URL with the indexing file, that download process is additional and that will take more 12 hours.        

 

So to accomplish this you can submit a  file which has URLs for the files which you want to import or which you want to update if the filename is already exists.

-Sudarshana

On 1/11/2023 7:36 PM, Naval Sarda wrote:




-------- Forwarded Message --------
Subject: Re: StratML Import file feature through FTP is ready
Date: Tue, 10 Jan 2023 19:39:42 +0000 (UTC)
From: Owen Ambur <owen....@verizon.net>
Reply-To: Owen Ambur <owen....@verizon.net>
To: Naval Sarda <nsa...@epicomm.net>


Naval, I'm getting concerned about the URL issue.  It sounds like the existing programming logic may be faulty.  I'd rather not pay more for the functionality that should have been built in from the start.

The URLs for the authoritative versions of the files are not "assigned" when they are indexed in the query service; the URLs already exist and point to where the files eside on the Web.  Those URLs should automatically be captured in the database when each of files is indexed.

It also sounds like we'll need to give more thought to the file name collision issue.  Presumably, if the URLs were properly treated in the database, that would not be an issue.

For example, when file collisons occur in my collection on the StratML.us site, I manually add a sequential number to the file name.  It is not a "random" number.  Again, however, that is only necessary because I am maintaining files for so many different organizations on my site.  If they were maintaining them on their own sites, at their own URLs, that would not be an issue.  It would be up to them to decide whether to over-write the previous versions or maintain them for historical purposes (which is my preference).  The desired end state is for each organization to maintain its own plans and reports on its own website, for indexing not only in my query service but many others as well.

I hope we can get these issues resolved.



On Tuesday, January 10, 2023 at 10:57:46 AM EST, Naval Sarda <nsa...@epicomm.net> wrote:


Please see below



-------- Forwarded Message --------
Subject: Re: Fwd: StratML Import file feature through FTP is ready
Date: Tue, 10 Jan 2023 20:59:10 +0530
From: Sudarshana <sudar...@epicomm.net>
To: Naval Sarda <nsa...@epicomm.net>, Balasaheb Pandarkar <balas...@epicomm.net>, kom...@epicomm.net, Jitendra Shende <jite...@epicomm.net>


Owen,

Please use port number 21 to upload files. Permission denied issue is now resolved.

Currently we have implemented basic import functionality with FTP, considering that this will be used by you for your internal purpose . As we are doing this through FTP, to update the existing files we have to use overwrite functionality if name of the files are same. If we assign random number after each file, then it won't update existing record and there will be lot of duplicates.

Along with file If you want to assign the URL for that file where it should open on View Hyperlink, we can provide one input control on web UI to provide that URL. We will then save that URL along with the uploaded files. It will take another 16 hours of efforts along with initial 16 hours of web UI to upload the files. That web UI will be visible after login only. This login will be only for you. Not for public. You can take call whether to implement in such way or not. Direct FTP won't map the file with the URL, hence web UI is needed.

Regarding public submissions of stratml files, we can assign random numbers to the files they are uploading and save the unique filename so generated in database, so that they can update the records they have uploaded using web UI.  This is essentially next phase and is not part of scope of current phase. 

-Sudarshana

On 1/10/2023 10:46 AM, Naval Sarda wrote:

See below



-------- Forwarded Message --------
Subject: Re: StratML Import file feature through FTP is ready
Date: Mon, 9 Jan 2023 22:39:07 +0000 (UTC)
From: Owen Ambur <owen....@verizon.net>
Reply-To: Owen Ambur <owen....@verizon.net>
To: Naval Sarda <nsa...@epicomm.net>
CC: aboutthe...@googlegroups.com <aboutthe...@googlegroups.com>


Naval, I was able to log on with the credentials you sent but unable to upload a file.  See error message in screen shot below.

Also, over-riding files with the same name may and almost certainly will become a problem when we open the service to submissions by others.  In the meantime, I've been dealing with filename collisions by adding numbers to files with the same name in my hperlinked index.  See, for example, https://stratml.us/drybridge/index.htm#OGP-USNAP3

I'm not sure what you mean by saying the the files must be uploaded to https://stratml.us/docs in order to be indexed in the query service.  If that's true, that's a problem that needs to be fixed.  The query service links should point to the files wherever they reside on the Web.





On Monday, January 9, 2023 at 05:01:14 PM EST, Naval Sarda <nsa...@epicomm.net> wrote:


Please see below



-------- Forwarded Message --------
Subject: Re: Fwd: StratML Import file feature through FTP is ready
Date: Mon, 9 Jan 2023 23:23:57 +0530
From: Sudarshana <sudar...@epicomm.net>
To: Naval Sarda <nsa...@epicomm.net>, Balasaheb Pandarkar <balas...@epicomm.net>, kom...@epicomm.net, Jitendra Shende <jite...@epicomm.net>


Owen,

We have changed startml folder name to stratml.

In another email we are sending you credentials for FTP, that will open one folder where you have to upload files.(/home/stratml/Uploaded_xml_Files)

Our program will run every 30 minute and picks up files from this folder. Then another program will do processing on that files and if files are perfect for indexing then add those files in indexing folder. Then those files will get deleted from the FTP folder.

After 30 minute when we search  on we application that files should be visible in search query.

If names are identical then file will get overwritten and if filename is different then new file gets added.

To explain this import feature I will explain you with one example.

I have taken one file 1MMMM.xml, renamed it with Test10.xml and imported with this feature.

When I searched by Stakeholder name "Hal Burrows", I got 2 records in result. When I click on view link of first record it redirect to "https://stratml.us/docs/1MMMM.xml" URL. As this file is present on https://stratml.us , that opens successfully. When I click on second link, that will redirect to "https://stratml.us/docs/Test10.xml" URL. As this file is not present on https://stratml.us, it cant open and gives error of "Resource Not found".

For this we have to upload that file on https://stratml.us also. As this location is external to us and located out of the server where the program is running.

So, while uploading file on FTP you have to upload it on https://stratml.us/docs location also. Please refer below images.





-Sudarshana
On 1/7/2023 11:08 AM, Naval Sarda wrote:


See below

-------- Forwarded Message --------
Subject: Re: StratML Import file feature through FTP is ready
Date: Fri, 6 Jan 2023 22:34:10 +0000 (UTC)
From: Owen Ambur <owen....@verizon.net>
Reply-To: Owen Ambur <owen....@verizon.net>
To: Naval Sarda <nsa...@epicomm.net>
CC: aboutthe...@googlegroups.com <aboutthe...@googlegroups.com>


Naval, there is a typo in the upload file pathname:  "startml" should  be "stratml".

Also, if I understand Sudarshana's assumption correctly, it is false.  Files may be uploaded to the query service from anywhere on the Web and not just from the stratml.us site.  Indeed, the entire collection on the stratml.us site is prototypical in nature.  Hopefully, someday all of the authoritative sources of the files will reside on the organizations' own websites and those on the stratml.us site can be deprecated.  However, that may be fairly far in the future.

When the pathname is corrected, I'll need the UID &  password to upload files.  I'll also need to know which we the most recent files that were imported in batch so that I can figure out where to pick up on uploading those that have been created since then.  If you can tell me where that break occurs in my sitemap listing, that would be helpful.  https://stratml.us/docs/sitemap.xml 



On Friday, January 6, 2023 at 01:24:33 PM EST, Naval Sarda <nsa...@epicomm.net> wrote:


Please see below



-------- Forwarded Message --------
Subject: StratML Import file feature through FTP is ready
Date: Fri, 6 Jan 2023 23:46:24 +0530
From: Sudarshana <sudar...@epicomm.net>
To: Naval Sarda <nsa...@epicomm.net>


Hi Owen

We have deployed application of Import File Feature through the FTP.

If you upload files in  /home/startml/Uploaded_xml_Files/  location on Server that files will get indexed after processing.

We are assuming that those files are getting uploaded externally on https://stratml.us/docs/ this location.


--
Thanks & Regards
Sudarshana
--
Thanks & Regards
Sudarshana
--
Thanks & Regards
Sudarshana
--
Thanks & Regards
Sudarshana
--
Thanks & Regards
Sudarshana

Owen Ambur

unread,
Mar 7, 2023, 3:03:27 PM3/7/23
to Naval Sarda, aboutthe...@googlegroups.com
Naval, the previous method doesn't seem to have been working for several days.  Three of the most recent files aren't showing up in the query service.

I guess I'll need to figure out again how to set up another account in my FTP client.  It's not particularly easy to do and the last time I did it access to my other, regular account for uploading to the stratml.us site was messed up for awhile.

Also, if the "web method" is for public use, does that mean others will set up FTP accounts?  

It doesn't seem like that should be necessary or the best way for occasional submitters to propose additions to the collection.  A simple HTTP get request form on the site itself seems like a better alternative, with fields for the URLs and the E-mail addresses of the submitters (for confirmation their submissions and acknowledgement of whether they have been accepted or not).



On Tuesday, March 7, 2023 at 02:49:45 PM EST, Naval Sarda <nsa...@epicomm.net> wrote:


Hi Owen,

There are two FTP accounts now. One for uploading URLs and another for previous import method. So both are operational. Please check if you were uploading to previous FTP method.

Please note this phase is for your internal use. The web method to import is for public use.

Naval

Reply all
Reply to author
Forward
0 new messages