Error in Step1 of Bulk Import

[Yvan Stg]

Hi again,

I'd like to set up a bulk import. Step 1 causes an error (attached). 

I'm trying to create the CSV for 48,000 files (28GB) located in 37 separate folders  on an external hard drive. The path I provide the Bulk Importer, is to the parent folder. I used a JVM heap size of 1500MB. Running on Windows 8, with Java x64 V. 8.301. 

I'm unsure why this is an issue with heap size since, based on your comments in this forum, 1500MB should amply sufficient for 50K pictures. 

Also, would it be possible to show a message once Step 1 has started? Since the button remains blue, it's unclear if it started or not. 

Cheers,

Yvan

[Yvan Stg]

It seems the error is because I'm trying to do a Step1 scan of a folder on an external hard drive. I tried with a smaller folder containing images on my computer's desktop and Step1 worked.

I'm guessing the issue is one of Camelot not having permission to access the folder. So I shared the folder with all users (see screenshot - two different folder names to show both "sharing" and "security" tabs). But that didn't solve the problem.

Does this look correct? Could you advise on best sharing practices?

[Chris Mann]

 Hi Yvan,

It looks like Camelot runs out of memory during scanning the network location.  Could you try increasing the JVM Heap Size setting in the Administration interface (https://camelot-project.readthedocs.io/en/latest/administration.html?highlight=heap%20size#system-configuration)?  This will need to be less than the amount of physical memory in the computer.

-Chris

[Yvan Stg]

Hi Chris, 

Thanks for getting back to me. 

I did try increasing the memory (to 1500MB) but even with a small test folder (400 pictures) the bulk import of that folder on the external hard drive (EHD) failed. 

I also have issues with Camelot not accessing a dataset (as in Admin/Datasets) on that EHD: I was able to upload all media (~70K pictures) to a 2nd dataset (1st being the default dataset) located on that disk (D:\Yvan\Camera trap\Dataset2), and animal detection started so I thought everything was fine. I closed Camelot but when trying to restart it, I had the typical issue of Camelot failing to locate the Database folder (despite the path in Admin/Dataset settings being correctly assigned to the parent folder of that Database folder). Just to say that it may be the EHD that's having issues - hence my trying to share it/update permissions. 

Yvan

[Chris Mann]

Hi Yvan,

Could you check whether the Java edition being ran is 32-bit or 64-bit?  Camelot only supports 64-bit Java, but while 32-bit works it often results in memory-related problems.

-Chris

[Yvan Stg]

Hi Chris,

I'm using the 64-bit version. 

Yvan

[Chris Mann]

Thanks Yvan,

Stepping back, sounds like there's two separate problems: one the external HDD, the other bulk import search.

Could you try running the bulk import again on the network location and sending through both launcher & server log files, so that I might investigate both the Camelot configuration & error message generated with that configuration?

Similarly, and separately, if you could have the dataset configured to the external HDD location, and then send me both sets of logs for that, that'd be appreciated.

Thanks,

-Chris

--
You received this message because you are subscribed to the Google Groups "Camelot Project" group.
To unsubscribe from this group and stop receiving emails from it, send an email to camelot-proje...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/camelot-project/af512ed6-e664-404d-80c9-51ab922ec747n%40googlegroups.com.

[Yvan Satge]

I'm not sure if I can send you any bulk import logs: I can't get Camelot to start because of the derby dataset issue, so I can't access the bulk import page. 

However, to me this is not the most pressing issue, as there is still the option of importing the media manually - not the most fun but can be done. 

--

Yvan Satgé

Independent researcher associated with the

South Carolina Cooperative Fish & Wildlife Research Unit

+33 782 649 373

ysa...@g.clemson.edu

atlanticseabirds.org

If your email bounces back, please contact me at: yvan....@gmail.com