What is the maximum FileSize that Autoplot can handle at once?

[Megha P]

Hi All,

I have around 130 netCDF files, each approximately 700 MB in size. My goal is to generate a continuous spectrogram using all these netCDF files. However, when attempting to load around 10 files, I encounter an error message. Is there a way to create a complete spectrogram?

Interestingly, some of my colleagues mentioned that they were previously successful in generating a full spectrogram of the same netCDF files using Autoplot. However, I now face errors. Can you provide guidance on how to overcome this issue and successfully generate the full spectrogram using Autoplot?"

[Jeremy]

In general the biggest individual file size Autoplot can read is 2GB.  The problem is as these files collect you are probably running out of memory which is available to the Java Virtual Machine.  (The JVM is an odd because it is limited to a certain size, which is less than what your computer  actually has available.) You can see this limit using the about dialog, shown using [menubar]->Help->"About Autoplot", then scroll down to "max memory".  I see on my Mac I get 4GB, which would probably be a problem for what you are trying to load in.  You can increase the amount of memory it gets if you use the "single jar" release, using the command line:

java -Xmx16G -jar autoplot.jar

I might be able to make the .dmg and other releases use more memory by default.  This issue came up a lot when Java was 32 bit and limited to 1GB, but much now.  But it would be nice if there were ways to get more memory with other launch methods, and I can look into that.

Last, there's a feature called "aggregation" which might be useful here.  Suppose one of your files is .../VIPER_H1_Spec_20220331_011628.nc.  You can instead plot .../VIPER_H1_Spec_$Y$m$d_$H$M$S.nc, and it will load whatever it needs to cover the interval you want to see.  I mention this because this also provides the feature that it can reduce the data as each file is read in. (Attached screenshot shows "reduce" button, which reduces the data to pixel resolution as each file is read in.)  This might also help out with your memory problems.  Let me know if this looks promising and I can help make a more precise aggregation spec.

Jeremy

[Megha]

Hi Jeremy,

Your reply is so much appreciated. It worked for me! Infact, I am able to load all the files and make a full spectrogram!
I truly value your response despite the long weekend.

Thanks again!

-Megha
 

--
You received this message because you are subscribed to a topic in the Google Groups "autoplot" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/autoplot/KnjblWQBPIY/unsubscribe.
To unsubscribe from this group and all its topics, send an email to autoplot+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/autoplot/74a809d8-1558-41f1-8a39-4f33b932860cn%40googlegroups.com.

--

-------------------------------------------------------------------

Megha Pandya

[Jeremy Faden CS]

Good to hear!  I bet it was the aggregation that worked for you, right?  The one risk of that is that if you have for example 10 seconds in each file, you might loose the first interval because .../VIPER_H1_Spec_$Y$m$d_$H$M$S.nc says that there is 1 second of data in each file.  If you were to use .../VIPER_H1_Spec_$Y$m$d_$H$M$(S;delta=10).nc, this would tell Autoplot that there is 10 seconds starting at the time indicated by the filename, and it would properly pick up the first file.  

Don't hesitate to ask if you have other questions!

Jeremy

You received this message because you are subscribed to the Google Groups "autoplot" group.
To unsubscribe from this group and stop receiving emails from it, send an email to autoplot+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/autoplot/CAFXPqAbgNj%2BnZ02qKOme%3DJgJSr-30__K%2BZjH0OYzvT%2B1XyLh_Q%40mail.gmail.com.

[faden cottagesystems.com]

It can plot binary data, but only 2GB at a time.  It would be difficult to implement something like this, since it really needs to reduce (by averaging) the data as it is read in, which is how the aggregation is working.  This is probably where you would use a Jython script to read data from the binary file and reduce it, but I'm not sure that would work any better than the NetCDF files.

Jeremy

---

From: Megha <megha.p...@gmail.com>
Sent: Thursday, November 16, 2023 3:36 PM
To: faden cottagesystems.com <fa...@cottagesystems.com>
Cc: auto...@googlegroups.com <auto...@googlegroups.com>
Subject: Re: [autoplot] Re: What is the maximum FileSize that Autoplot can handle at once?

 

Jeremy,
You are right. It was aggregation that did my job!
Well I had been also wondering if is it possible for Autoplot to plot binary data? This is the same aggregated 90 GB binary file.
-Megha

--

-------------------------------------------------------------------

Megha Pandya

[Megha]

Jeremy,
You are right. It was aggregation that did my job!
Well I had been also wondering if is it possible for Autoplot to plot binary data? This is the same aggregated 90 GB binary file.
-Megha

On Tue, Nov 14, 2023 at 5:25 PM Jeremy Faden CS <fa...@cottagesystems.com> wrote:

--

-------------------------------------------------------------------

Megha Pandya

[Megha]

Thanks for clarifying, Jeremy. I appreciate your quick response. I've found a way to get to this done via Python. :)

--

-------------------------------------------------------------------

Megha Pandya

[Megha]

Hi Team,

Thank you for helping me to plot a spectrogram from a bunch of heavy .nc files using aggregation technique.

Now I have power spectra with the x-tick labels as shown in the picture below:

However, the actual time stamps on the data file are 32, 33, 34, ...  in seconds.

The autoplot does not allow me to change the x-tick labels on the plot.
The picture of "axes" tab is as follows:

I cannot change the manual label here. It shows the following error.


Could you please suggest how to set actual time stamps on the data file are 32, 33, 34, ...  in seconds.?

-Megha

--

-------------------------------------------------------------------

Megha Pandya

[Jeremy Faden CS]

Hello Megha,

      I'm guessing that you can plot an individual file by specifying that its time-axis units are like "seconds since 2019-11-09T10:14:01Z," which means each .nc file must have a different time base (2019-11-09T10:14).  If the aggregation code knows the units as each file is loaded, it will convert them to a common time base.  Could you send a URI you are using so I can see how you are setting the time units?  Is the time base stored somewhere in each file?  You could send a file to me, too, directly to fa...@cottagesystems.com.

      It might be a jyds script is needed to handle the times.  Supposing that the time base is stored within the file, this script would load the time base as well and add it to the seconds, before returning data with timetags which can be aggregated.  I can show how this would be done.

      Alternatively, if you are creating the .nc files and can decide what is stored in each one, you might change the time variable so that its units state the time base.  The first file might have units of  "seconds since 2019-11-09T10:14:01Z" while another has units of "seconds since 2021-05-27T01:15:00Z".  Autoplot's aggregation code should detect these and properly combine them.  

                                 Jeremy

To view this discussion on the web visit https://groups.google.com/d/msgid/autoplot/CAFXPqAYELMsjjHesYzZoS1uu5K%3DngYx%3D88HsY3UKVBzKgmr37A%40mail.gmail.com.

[Megha]

Hi Jeremy,

I have got the desired plot along with the x-tick labels from the .nc file that I shared with you.
Another thing that I wish to add to this data set is a few vertical lines on the top of the existing spectrogram.
These vertical lines act as an indicator of the events "A", "B" etc.
Is it possible to add such lines with desired color and style on the top of the existing plot at a given location?
Please share your thoughts.

Best Regards,
Megha

On Wed, Jan 24, 2024 at 11:44 AM Jeremy Faden CS <fa...@cottagesystems.com> wrote:

That's great to hear.  Let me know if you need anything else.

Jeremy

On 1/24/24 10:42, Megha wrote:

Hi Jeremy,

Thank you so much for your quick reply.
Regarding,  "|setDepend0Units('s')" --> It worked!
I could get the X-Tick Labels in seconds now. That serves a great help! Thank you so much.

Also, thanks for reiterating the importance of the "reduce" tab. I remember you shared this with me while explaining "aggregation".
I always use it before I plot.

Thanks for being so helpful and saving my time.
Megha

On Mon, Jan 22, 2024 at 9:24 AM Jeremy Faden CS <fa...@cottagesystems.com> wrote:

Hi Megha,

      I think I finally follow you, and thanks for being patient with me.  You wish to simply show the time axis in seconds since rocket launch (seconds since 2023-11-09T10:14:01), right?  The code really wants to treat time locations like "seconds since ..." as fully-located times, and I should make sure there's an option to show the raw time locations.  You can force it to show the times as seconds instead of "seconds since 2023-11-09T10:14:01" by adding on the data tab the operation "|setDepend0Units('s')".  What this will do will be to replace the unit with just seconds.  See the screenshot below and let me know if you have trouble finding this.  (You can press the button on the right to get a dialog to find the function, or you can just type it in.) 

      So for the aggregation, are you able to combine multiple files together into one load?  I keep running out of memory, but there are tricks to this, too.  There's a button to have it "reduce" the data as it is loaded. (See the "reduce" checkbox in the image below.)  Each file of the aggregation is loaded, then successive points are combined to show an equivalent spectrogram.  When you zoom in, it may go back to the data file and reload it, leaving the data at higher resolution.   If you have more files from the launch I can experiment with, I'd be happy to look into it more. 

                             Jeremy

On 1/21/24 20:17, Megha wrote:

Hi Jeremy,

Thank you for your quick reply.

The URI is: file:///Users/mmpandya/Documents/BeamPIE/BP_HF_17JAN_FOLDED_32SEC_AGC/BEAMPIE_12__H1_HF-Spectrogram_$Y$m$d_v01-Folded_$Y-$m-$d_$H$M$S.nc?Power&timerange=2023-11-09+10:10+to+10:30&reduce=T

I do not know exactly but I assume you are asking for the above URI.

I agree that timestamps are stored in .nc file. I am sending you just the file with possibly no data in it (Blank data).
However, you will be able to follow the data structure and format. I hope this will work.

I appreciate your help and look forward to the suggestions.

Thanks again,

Megha

 d102628.nc

 

--

-------------------------------------------------------------------

Megha Pandya

--

-------------------------------------------------------------------

Megha Pandya

[Jeremy Faden CS]

Hi Megha,

     Have a look at annotations, which let you do everything from having an extra label next to a plot axis to pointing out a feature on a plot.  Right-click on a plot, for example, "Add Annotation", and then see if you can figure out the GUI.  These are admittedly too flexible, and I always have to experiment with the many controls to get what I want.  (Note the annotations are listed on the layout tab, it's easy to have them disappear when they are drawn off the screen.)  

     Also you can just plot an events dataset, which is another way to have a "renderer" draw lines instead of a spectrogram or line plot.  Try File->Add Plot From->Inline..., then Events List and "Add" a list of times on the "data" tab.  This will create a URI like "vap+inline:2014-01-01T01:01Z&RENDER_TYPE=eventsBar" which is drawn as a single vertical line.

     Hope that helps!

                           Jeremy

[Megha]

Hi Jeremy,

Thank you for your quick reply.

1) I had been playing around with the annotation tools as you suggested, but I think my skills with autoplot is really poor.
Well, I wanted to make blue and red lines like the one shown in the spectrogram below (This figure is edited in powerpoint to represent what I mean).

I was thinking if there is any way that I could give commands to autoplot to make a vertical line at
X = 10:11:35 ; Y = 0 to 10000 with blue color
and

X = 10:11:40 ; Y = 0 to 10000 with red color, dashed style.
Could you please suggest?

 

2) While checking for the scripts to make a code instead of plotting manually, I landed on http://autoplot.org/matlab
As you can see in the background picture, the matlab is throwing an error. Do I need any packages for running Section-8: complete matlab matlab examples as shown on page?
In that case, I can use hold on command and plot lines as required. I understand this is a vast topic, but any pdf explaining how to use it on Mac and Windows, will be helpful.

3) I was trying to regenerate the same spectrogram using autoplot with aggregation in Windows instead of Mac. For this, I downloaded:

 exe: a self-contained installer for Windows computers.

However, when I am running: java -Xmx16G -jar autoplot.jar
The following error comes up:


Which means Java is not on my computer. I do not have a licensed version of Java and I cannot install unlicensed JAVA on my NASA computer.
Do you have any suggestions to solve this?

-Megha

--

-------------------------------------------------------------------

Megha Pandya

[Jeremy Faden]

Hi Megha,

        I can respond to your last item quickly, others will have to wait until I'm back at my desk.  With Windows you should have downloaded an .exe program.   You should be able to run this (after accepting the security risks) and then it will install Autoplot on the computer.  

           Jeremy

To view this discussion on the web visit https://groups.google.com/d/msgid/autoplot/CAFXPqAaGN1-psBmyhmo68q_Obx_ogZFAqAj6KFnbEiAE%2BT8CHw%40mail.gmail.com.

[Jeremy Faden]

(To be more clear, the .exe has a java embedded within it, so you won't need Java to be on the Windows machine.)

[Harris, Bernard T. (GSFC-5870)]

How is Java included in the .exe file?  For example, the windows skteditor-1.3.6.msi file from https://spdf.gsfc.nasa.gov/skteditor/ includes Java but only for use by skteditor.  If you type java in a command window after installing the msi, you'll get same message Megha reported (java not found).  So, I wonder if autoplot's embedded java is like skteditor's.

 

If a "traditional" copy of Java is really needed on a nasa computer, it can be installed from the NASA-OCIO "Self Service Portal" (macs) or "Software Center" (windows).  I know it's still available for macs but it's been years since I've had access to a nasa windows computer to check.  I assume it's still available in the windows [nasa] "software center" (or whatever it's named now).  Note that if someone has a "traditional" installation of java on their computer, the skteditor msi won't touch it and skteditor won't use it.

 

Bernie

 

From: auto...@googlegroups.com <auto...@googlegroups.com> on behalf of Jeremy Faden <jerem...@gmail.com>
Date: Friday, February 2, 2024 at 1:16 PM
To: auto...@googlegroups.com <auto...@googlegroups.com>
Subject: [EXTERNAL] [BULK] Re: [autoplot] Re: What is the maximum FileSize that Autoplot can handle at once?

CAUTION: This email originated from outside of NASA.  Please take care when clicking links or opening attachments.  Use the "Report Message" button to report suspicious messages to the NASA SOC.

(To be more clear, the .exe has a java embedded within it, so you won't need Java to be on the Windows machine.)

 

On Fri, Feb 2, 2024 at 12:12 PM Jeremy Faden <jerem...@gmail.com> wrote:

Hi Megha,

        I can respond to your last item quickly, others will have to wait until I'm back at my desk.  With Windows you should have downloaded an .exe program.   You should be able to run this (after accepting the security risks) and then it will install Autoplot on the computer.  

           Jeremy

 

 

On Fri, Feb 2, 2024 at 11:49 AM Megha <megha.p...@gmail.com> wrote:

Hi Jeremy,

Thank you for your quick reply.

1) I had been playing around with the annotation tools as you suggested, but I think my skills with autoplot is really poor.
Well, I wanted to make blue and red lines like the one shown in the spectrogram below (This figure is edited in powerpoint to represent what I mean).

I was thinking if there is any way that I could give commands to autoplot to make a vertical line at
X = 10:11:35 ; Y = 0 to 10000 with blue color
and

X = 10:11:40 ; Y = 0 to 10000 with red color, dashed style.
Could you please suggest?

 

2) While checking for the scripts to make a code instead of plotting manually, I landed on http://autoplot.org/matlab
As you can see in the background picture, the matlab is throwing an error. Do I need any packages for running Section-8: complete matlab matlab examples as shown on page?
In that case, I can use hold on command and plot lines as required. I understand this is a vast topic, but any pdf explaining how to use it on Mac and Windows, will be helpful.

 

3) I was trying to regenerate the same spectrogram using autoplot with aggregation in Windows instead of Mac. For this, I downloaded:

 exe: a self-contained installer for Windows computers.

 

However, when I am running: java -Xmx16G -jar autoplot.jar
The following error comes up:

Which means Java is not on my computer. I do not have a licensed version of Java and I cannot install unlicensed JAVA on my NASA computer.
Do you have any suggestions to solve this?

-Megha

 

Megha

 d102628.ncError! Filename not specified.

 

 

On Fri, Jan 19, 2024 at 7:59 AM Jeremy Faden CS <fa...@cottagesystems.com> wrote:

Hello Megha,

      I'm guessing that you can plot an individual file by specifying that its time-axis units are like "seconds since 2019-11-09T10:14:01Z," which means each .nc file must have a different time base (2019-11-09T10:14).  If the aggregation code knows the units as each file is loaded, it will convert them to a common time base.  Could you send a URI you are using so I can see how you are setting the time units?  Is the time base stored somewhere in each file?  You could send a file to me, too, directly to fa...@cottagesystems.com.

      It might be a jyds script is needed to handle the times.  Supposing that the time base is stored within the file, this script would load the time base as well and add it to the seconds, before returning data with timetags which can be aggregated.  I can show how this would be done.

      Alternatively, if you are creating the .nc files and can decide what is stored in each one, you might change the time variable so that its units state the time base.  The first file might have units of  "seconds since 2019-11-09T10:14:01Z" while another has units of "seconds since 2021-05-27T01:15:00Z".  Autoplot's aggregation code should detect these and properly combine them.  

                                 Jeremy

 

On 1/18/24 21:09, Megha wrote:

Hi Team,

Thank you for helping me to plot a spectrogram from a bunch of heavy .nc files using aggregation technique.

Now I have power spectra with the x-tick labels as shown in the picture below:

However, the actual time stamps on the data file are 32, 33, 34, ...  in seconds.

The autoplot does not allow me to change the x-tick labels on the plot.
The picture of "axes" tab is as follows:

I cannot change the manual label here. It shows the following error.

-Megha

 

 

--

-------------------------------------------------------------------

Megha Pandya

 

 

--

-------------------------------------------------------------------

Megha Pandya

 

 

--

-------------------------------------------------------------------

Megha Pandya

 

 

--

-------------------------------------------------------------------

Megha Pandya

 

 

--

-------------------------------------------------------------------

Megha Pandya

 

To view this discussion on the web visit https://groups.google.com/d/msgid/autoplot/CAJMNx72qmRVuWA%2BN7pweood6XAZ5vb%3DMr7%3DBmrutoVC7p1%2BEJA%40mail.gmail.com.

[Jeremy Faden Gmail]

Install4J is used to build .dmg and .exe releases for Mac and Windows.  It embeds the Java within the release, and everything is unpacked in an install directory.  I believe Java 17 has similar abilities built in, but I found Install4J easier and well-documented.  If you are interested, I could build an install for you using the software so you could see if you like how it works.

Jeremy

To view this discussion on the web visit https://groups.google.com/d/msgid/autoplot/BLAPR09MB7315225B3D0CAA5018D76EA5BC422%40BLAPR09MB7315.namprd09.prod.outlook.com.

[Jeremy Faden Gmail]

I read your email a little too quickly.  The Java install is only used by it, so if you installed two software packages you would find that you have two Java runtimes (JREs) on your system.

Jeremy

To view this discussion on the web visit https://groups.google.com/d/msgid/autoplot/ee13de72-bff5-4f8e-878a-cdea97786c40%40gmail.com.

[Jeremy Faden CS]

Hi Megha,

       I'll make a more graphical tutorial or video sometime over the weekend, but to get to the two vertical strikes you can:

1. start with your spectrogram

2. File->Add Plot From->Inline...

3. Here you should have "Dataset Type" set to "Event List", then "Add..." on the right.

4. Enter your time, like "2023-11-09T10:11:40" and enter.

5. press "Overplot" (not "Plot").  You should see a black vertical line at 10:11:40.

6. click on the vertical line, right click, then "Plot Element Style Properties"

7. color to red, line thickness to "3px", line style to "DashFine", Opaque should be checked.

8. to add the second line, add another overplot and repeat the procedure.

This only supports the use case of the vertical line which occurs often.  Also not that I think the line thickness and line style are fairly new, so this probably only works with v2024a_1 and more recent releases.

      I'm going to have to check on Matlab next time I have a client available to me.  I can probably infer what's wrong with the Complete Example using the Python client, which uses the same interface, but I don't have time right now.  I'll let you know.

Jeremy

[Jeremy Faden CS]

Hi Megha,

     I verified operation with Matlab (R2021a).  I wonder if the problem is that I didn't have the javaaddpath() for the Autoplot jar file.  Each time Matlab is restarted this is needed.  I've updated this and each of those complete examples should work with a fresh Matlab session now.

             Jeremy

[Jeremy]

Here's a video showing these steps: https://www.youtube.com/watch?v=G0iuEwEWs6w

[Megha]

Hi Jeremy,

I deeply appreciate the time you invested in providing a detailed, step-by-step explanation and creating a tutorial video on overplotting lines on a spectrogram.

Your willingness is truly appreciated.

Currently, I am trying MATLAB example to automate the process, eliminating the need for manual plotting.

Your support and time are sincerely valued.

Thank you once again.

Megha

To view this discussion on the web visit https://groups.google.com/d/msgid/autoplot/46304acf-c03c-4bc5-95b8-4afbfa6e3a98n%40googlegroups.com.

--

-------------------------------------------------------------------

Megha Pandya