Camelot not working - internet issue?

[Tori Peltier]

Hi everyone,

For about the past week, Camelot has not been working for me. I use it at school, so it uses our school network which is infamous for not being great. I figured I’d ask if anyone else has been having issues or if it is more likely a problem with our school internet that’s causing the issue.

[Chris Mann]

Hi Tori,

Camelot typically doesn't make much use of the Internet unless you're
accessing from a different computer than it's running from.

Are you able to describe the symptoms of the problem and attach the log
files if available? (https://camelot-project.readthedocs.io/en/latest/administration.html#system-logs)

--
Chris Mann

[Tori Peltier]

Hi Chris,

I’m actually off campus now. I’ve asked one of my lab mates to send me the system logs so I’ll get that to you ASAP.

In the meantime I actually have another question. We’ve also been having trouble downloading reports. The species statistics and summary statistics reports took about 10-15
minutes to download when previously they’d downloaded instantly. We tried downloading a full survey export which loaded for about 30 minutes before giving us a message saying: it couldn’t be accessed because of Java heap space. I’ve contacted out IT, thinking it was an internet or computer problem (the two lab computers are probably 10 years old). But they haven’t gotten back to us.

[Chris Mann]

Hi Tori,

Thanks - logs would be good. If the internet connection is poor, that
could explain a variety of problems. What sort of problems were you
experiencing?

Regarding reports:

the Java heap space error would explain why some reports suddenly take
significantly longer. Once the size of the data crosses the amount of
memory available to Camelot, it becomes *much* slower to produce
reports, and they may fail with this error.

This page gives some suggestions about how to configure Camelot when
working with larger datasets:

https://camelot-project.readthedocs.io/en/latest/scale.html#configuration

The other thing to check is the computer running Camelot has a 64-bit
version of Java installed. A 32-bit version is very restrictive in the
amount of memory that can be made available and can lead to this problem
on even "smaller" (<100k image) datasets.

--
Chris Mann

[Tori Peltier]

For problems, when we are using Camelot it’s usually slow. It can range from pretty slow to incredibly slow where trying to go from page to page in the library takes multiple minutes. We figured it’s because we now have more data uploaded (about 45-50,000 pictures). We’ve also had problems with Camelot not starting. Sometimes we’d just have to close out of the program and reopen it. Sometimes it wouldn’t work for maybe a day. But it’s always started working again.

I have the system logs now. Should I just email them to you in an attachment?

[Tori Peltier]

I also have another question. As I said I’m working remotely and have been communicating with my lab mate. We are not sure how to tell what version of Java we’re using, if it’s the 64 or 32 bit. Would you be able to direct us in how to figure that out?

And if we just downloaded the 64 bit, would that work, even if we already had the 64 bit version installed?

[Tori Peltier]

I've attached the system logs file. I had been responding on my phone so I didn't see that there was an option to attach a file.

[Tori Peltier]

I apologize for all of the comments! This is an updated systems log after IT came in and updated some things. 

[Chris Mann]

Hi Tori,

Let me check to make sure my understanding is right. It sounds like there is potentially a few issues here:

1. Looking at those logs, I see some errors making me believe Camelot may not be starting up. Is that the case?

2. I understand that Camelot is slow to show the library. On modern hardware, I would expect ~50k images to load in maybe ~5-10 seconds. Taking minutes suggests there's a problem.

3. Regarding 64-bit Java: this can be checked by looking for the Java entry in "Uninstall or change a program" in the Control Panel. I'd suggest doing this before any Java installation.

Regarding #2: aside from old hardware, the other possible problem is the location of the Camelot Database.  I see this is on the E:\ drive, which is often a network drive.  Camelot will perform much better if the database is on fast, local storage. You'd typically not want to run Camelot from a drive not having backups though.

Given there's a large database, needing fast storage, deserving backups, with Camelot needing the right version of Java, and sufficient RAM and CPU performance, and supporting remote access to boot, I'd suggest it may be best to work with IT to get Camelot onto a server.

The following may be useful towards that:

* Best practices around Camelot storage: https://camelot-project.readthedocs.io/en/latest/datasets.html#best-practices

* Memory configuration and expected performance: https://camelot-project.readthedocs.io/en/latest/scale.html

Hope that helps,

-Chris

--
You received this message because you are subscribed to the Google Groups "Camelot Project" group.
To unsubscribe from this group and stop receiving emails from it, send an email to camelot-proje...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/camelot-project/055bfec9-0d47-4230-a85a-7b27c6595b54%40googlegroups.com.

[Tori Peltier]

Hi Chris,

We had the 64 bit version of Java reinstalled as of yesterday so that’s no longer an issue.

And yes, Camelot is still not starting up. As for our database, it is an external hard drive we have connected to the computer. It has multiple terabytes of storage so storage space for it is not an issue.

[Tori Peltier]

Just to add, when starting up, Camelot will open, and then give the message that it’s taking longer to load then expected. We have left it to load overnight and nothing will happen.

[Tori Peltier]

I apologize for all of the add ons, but we have a deadline for all of this to get done, which is why we're acting so urgently right now. But I'm trying to include as much info as possible to aid you in figuring out what our issue/s are. As I said, Camelot got progressively slower as we added more data. We'd have times where it didn't work, but not for more than a day. But Friday, April 17, one of my colleagues was able to connect in the morning, and by late afternoon, another colleague wasn't able to. I went in that weekend and it never worked, and it hasn't worked since.

[Barry Brook]

Hi Chris,

Further to this, our group is using camelot 1.5.8, running with Linux (Ubuntu 18) on a server (multiple people access the database remotely using a VPN and their web browser). I have Java64 bit installed, have allocated 18GB RAM to java process (the server has 64 cores and 128 GB RAM in total --- it's a high-performance machine).

I just checked the latest camelot speed (there were no other people using it at this time):

• From Survey page to Library load of first 150 images (160K images in database) = 115 seconds
• To load a new page of 150 images = 110 seconds
• Search for specific terms (e.g. isolate to a species) = 115 seconds
• To load a full individual image after clicking on thumbnail = <1 sec (i.e. this part is OK)
• Return to Organisation page, or doing anything other than accessing the library = instantaneous

I am current using the "Other" (i.e., server version), not Linux, but this only seems to be 32 bit? Should I be using the Linux version instead? Multiple people access the database remotely.

I note that in your documentation, you say:

"To serve as a guide for how Camelot may perform for large datasets, below is the timing for working with a dataset of 2 million images using a high-end laptop in 2017 (Dell Precision 5510; 16GB RAM; Intel i7-6820HQ; SSD) with a maximum JVM heap size of 14GB:

Full Export report: 368 seconds

Library load time: 13 seconds

Library search time (basic search): 6.5 seconds

Library search time (full-text search): 28 seconds"

I'm clearly not getting that sort of performance! Any advice?

cheers,

Barry

[Barry Brook]

Just to clarify a few things from my previous post. 

First, the number I quoted were wrong, I was using min/sec, it should be: 75 sec for library load, 70 sec for loading a new page of 150 thumbnails, and 75 sec for a basic search. The media data is stored on a local 16 TB hard drive, and the database on a local SSD.

Next, for further information, I'm loading the .jar version (via crontab, on server startup) via the following:

@reboot java -Xms8g -Xmx8g -server -jar /home/xxx/camelot/camelot-1.5.8.jar

There is never an issue with it loading/working, or stability, that's all terrific.

A full export takes 121 seconds, which is in line (better because a only 160K images) with your quoted figures, so it's just the library loading / searching performance that's the issue.

[Chris Mann]

Hi Barry!

Thanks for the system details and timings. That configuration all sounds fine, though I certainly think we can get some better performance than that.

I'll dig into this over the weekend and see what can be done. 

Regarding which version to run: the server release (plain jar) is recommended for the use case you described. I expect it is running within a 64-bit 'java', in which case all is as it should be. 

-Chris

--

You received this message because you are subscribed to the Google Groups "Camelot Project" group.
To unsubscribe from this group and stop receiving emails from it, send an email to camelot-proje...@googlegroups.com.

To view this discussion on the web, visit https://groups.google.com/d/msgid/camelot-project/a388bb4c-8451-4d28-b156-67804b42a228%40googlegroups.com.

[Chris Mann]

Hi Barry,

I've just released Camelot 1.6.0, which should improve this quite a bit: https://groups.google.com/forum/#!topic/camelot-project/2XVYi9h54M4

I'd expect somewhere between a 10x and 100x improvement over 1.5.8.

-Chris

--

You received this message because you are subscribed to the Google Groups "Camelot Project" group.
To unsubscribe from this group and stop receiving emails from it, send an email to camelot-proje...@googlegroups.com.

To view this discussion on the web, visit https://groups.google.com/d/msgid/camelot-project/a388bb4c-8451-4d28-b156-67804b42a228%40googlegroups.com.

[Barry Brook]

Oh wow, thanks Chris! That's amazing. Can't wait to check this out.