Sunpy offline mode working
93 views
Skip to first unread message
Anand MN
May 3, 2021, 6:47:21 AM5/3/21
to SunPy
hi all,
- I am using Sunpy for my solar optical image analysis and calibration. In the beginning, it was working smoothly, but one day I had to run it without internet and issues started. Now it is giving a warning on cache directory permissions and the solar parameters and rotation values are not calculating correctly. How to solve this issue?.
- How to plot the grid and solar coordinates against custom data which is taken using my own telescope? All the examples are showing AIA_171 images. Even limb detection is not working with that. Please help me with this.
regards,
Anand
Nabil Freij
May 3, 2021, 9:16:24 AM5/3/21
to SunPy
Hi Anand,
1. Can you show the output from the terminal/script with these issues?
2. I am unsure about this but I assume that you would have to create a custom WCS for your data for those plotting functions to work. Do you have an example of your data, that someone might be able to use to help you with?
Many thanks,
Nabil
Anand MN
May 5, 2021, 1:34:54 AM5/5/21
to SunPy
Hi Nabil,
- please see the attachments, the screenshots are the warnings I am getting while running the code. I am using Sunpy.Cordinates.sun package and calculating the P-value using the built-in function. this warning started after running the code without internet. Now even after connecting to the internet, this warning continues and the P-value is not correct. the JPG images are of the calibrated output files, you can compare them with the solar image taken from solarmonitor.org of the same date attached. There is a clear difference in the rotation. how to solve this issue?
Anand MN
May 5, 2021, 1:41:06 AM5/5/21
to SunPy
- The Fits files are the raw data I am working with. After calibration, I need to store them in fits and jpg formats. For the jpg image, I have to integrate the solar coordinate grid and mark the north pole, date and time of the observations, and details of the observatory. How to do that? (Since the files are of large size, I need to share you the link)
Ireland, Jack (GSFC-6710)
May 5, 2021, 9:25:03 AM5/5/21
to su...@googlegroups.com
Anand,
Even although the images you sent are from the same day, there is an approximate 20 hour difference between the SolarMonitor image and the times in the filenames of the images. Solar rotation at disk center as seen from the Earth is approximately 240 arcseconds per day. The time difference between the images and the subsequent solar rotation may be enough to explain why the sunspots in the SolarMonitor image are in a different location compared to the other images.
Jack
--
You received this message because you are subscribed to the Google Groups "SunPy" group.
To unsubscribe from this group and stop receiving emails from it, send an email to
sunpy+un...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/sunpy/d2629f31-136f-414b-be74-ffb0c32c0fadn%40googlegroups.com.
Albert Y. Shih
May 5, 2021, 10:08:41 AM5/5/21
to su...@googlegroups.com
Hi, Anand,
I'm guessing your warning message is due to your Astropy having trouble updating its leap-second table. That should have negligible impact for what you are trying to do here.
How was your telescope mounted? Your images do not appear to be taken from an equatorial mount, so the P angle alone would not represent the correct rotation angle. I'm guessing your telescope was aligned with azimuth-elevation, so you need to account for the orientation of the Sun relative to local zenith.
I've attached a script showing the bare minimum of additional header information to be able to plot the heliographic grid, as well as the plot output. I had to eyeball a number of the values. I've also attached the SolarMonitor image from the preceding day, which is only 4 hours different in time, so a better point of comparison. (Of course, you could use SunPy to grab the HMI image at the matching time, rather than being stuck with what is shown in SolarMonitor.)
Hope that helps!
Albert
To view this discussion on the web visit https://groups.google.com/d/msgid/sunpy/D17669E6-2103-400F-BEC2-1ADF39DA7C91%40nasa.gov.
Anand MN
May 6, 2021, 2:40:53 AM5/6/21
to SunPy
Dear Jack,
As you said in 20hr there can be a difference in the position. But that will be in the solar longitude only. Here the alignment of the group itself is wrong. If you compare the red lines marked, there is a difference in the slope. That should not happen. Even with the AR 12816 & 12819, the latitudes are different. That cannot be because of the time difference in the observation. this is the issue I am trying to solve.
Anand MN
May 6, 2021, 2:47:00 AM5/6/21
to SunPy
Dear Albert,
- As you said the warning msg is due to astropy. I am not using astropy coordinate library directly, I am using sunpy library.
- I am using a coelostat with an equatorial mount. There will be rotation created by this system, and I am correcting that in the calibration. the warning i am getting while calculating the P-value, not with coelostat parameters.
Anand MN
May 6, 2021, 2:52:39 AM5/6/21
to SunPy
Thank you for the script.
Now you are estimating the disk center by assumption. Is there any way we can find that using any function.?
On Wednesday, May 5, 2021 at 7:38:41 PM UTC+5:30 ays...@gmail.com wrote:
Anand MN
May 6, 2021, 6:47:56 AM5/6/21
to SunPy
Please see the attachments, for the download failed error thrown by the astropy. It is referring to some URL. Is astropy communicating with this URL every time to get the values? how to get it installed in the local system and run the code without internet?
Albert Y. Shih
May 6, 2021, 9:47:22 AM5/6/21
to su...@googlegroups.com
Hi, Anand,
I'm going to reply to your various emails in a single email:
> As you said in 20hr there can be a difference in the position.
To avoid the complications of solar rotation, I have updated my script to grab the HMI continuum image at the same time, for direct comparison. You can see that the correspondence is great.
> I am using a coelostat with an equatorial mount. There will be rotation created by this system, and I am correcting that in the calibration. the warning i am getting while calculating the P-value, not with coelostat parameters.
Given that this is an equatorial mount, I went back and realized that I put the P angle incorrectly into the CROTA2 keyword. See the updated script for how to use P angle correctly, and accounting for the fact that your data array is rotated by 90 degrees.
> Now you are estimating the disk center by assumption. Is there any way we can find that using any function.?
The updated script also shows one way to determine the disk center via centroiding. There are a number of other possible approaches.
> Is astropy communicating with this URL every time to get the values?
Astropy, which SunPy depends on, tries to download an updated leap-second table if its current leap-second table is over a month stale. Once the leap-second table is updated, Astropy shouldn't try to download it again for another month.
I think you are running an older version of Astropy, because those are out-of-date mirror URLs for the leap-second table. Try updating your Astropy. Regardless, for your purposes, you can pretty much ignore the warning since a slightly out-of-date leap-second table has negligible effect.
Albert
To view this discussion on the web visit https://groups.google.com/d/msgid/sunpy/6a70db91-3566-4ee5-9deb-4df1e4fcce46n%40googlegroups.com.
Anand MN
May 7, 2021, 2:29:42 AM5/7/21
to SunPy
If this is the issue with outdated astropy, then, once I create the Exe file and install it on another PC, I will have to update the package frequently. Am I correct?
Anand MN
May 7, 2021, 2:49:28 AM5/7/21
to SunPy
Dear Albert,
Currently, I am using the centroid method, but that is not the correct center of the disk. I am looking for something like circle/limb detection and center based on that.
Anand
Ireland, Jack (GSFC-6710)
May 7, 2021, 9:39:17 AM5/7/21
to su...@googlegroups.com
Anand,
There are many methods in the literature for automatically detecting circles in images, for example:
https://scikit-image.org/docs/dev/auto_examples/edges/plot_circular_elliptical_hough_transform.html
There are a lot of code snippets on line.
To view this discussion on the web visit
https://groups.google.com/d/msgid/sunpy/ba0937e9-907b-4b69-82b1-45536455691an%40googlegroups.com.
Albert Y. Shih
May 7, 2021, 1:34:28 PM5/7/21
to su...@googlegroups.com
Hi, Anand,
> If this is the issue with outdated astropy, then, once I create the Exe file and install it on another PC, I will have to update the package frequently. Am I correct?
You can turn off the auto-downloading of the IERS table through an Astropy configuration setting. See:
and
You may see warnings show up if the IERS table is too old, but you should be able to filter those warnings out.
> Currently, I am using the centroid method, but that is not the correct center of the disk. I am looking for something like circle/limb detection and center based on that.
Note that I perform the centroid on a Boolean array, not on the data itself, so it in fact uses a simple limb detection. In addition to Jack's suggestion of the Hough transform, I have also used convolutions or more general template matching (e.g., https://docs.opencv.org/master/d4/dc6/tutorial_py_template_matching.html). You'll have to figure out what method works best for you depending on considerations such as the needed accuracy, the needed performance, the needed robustness given your particular data, and your ability to test/maintain the implementation.
Albert
To view this discussion on the web visit https://groups.google.com/d/msgid/sunpy/1A396E16-014E-49DD-AD87-11420F944FD0%40nasa.gov.
Anand MN
Jun 7, 2021, 4:20:59 AM6/7/21
to SunPy
DearAlbert,
I tried your code. It is working with raw data, or data without all those header values. I included all those header values to my data during level 1 calibration of the data and tried running the code. Here is my new header a comparison with the header your program has created;
Keyword, value from your code, value from my code, comparison, types
SIMPLE True True True 4 4 <class 'bool'> <class 'bool'>
BITPIX -32 32 False 3 2 <class 'int'> <class 'int'>
NAXIS 2 2 True 1 1 <class 'int'> <class 'int'>
NAXIS1 2048 2048 False 4 4 <class 'int'> <class 'int'>
NAXIS2 2048 2048 False 4 4 <class 'int'> <class 'int'>
EXPTIME 0.28 0.28 False 4 4 <class 'float'> <class 'float'>
BSCALE Not found
BZERO Not found
CREATOR PCO 2000 PCO 2000 False 8 8 <class 'str'> <class 'str'>
DATE 20210223T022434Z 20210223T022434Z False 16 16 <class 'str'> <class 'str'>
DATE-OBS 2021-02-23T02:24:34Z 2021-02-23T02:24:34Z False 20 20 <class 'str'> <class 'str'>
CTYPE1 HPLN-TAN HPLN-TAN False 8 8 <class 'str'> <class 'str'>
CTYPE2 HPLT-TAN HPLT-TAN False 8 8 <class 'str'> <class 'str'>
CUNIT1 deg deg False 3 3 <class 'str'> <class 'str'>
CUNIT2 deg deg False 3 3 <class 'str'> <class 'str'>
CDELT1 0.00031 0.000323176 False 7 11 <class 'float'> <class 'float'>
CDELT2 0.00031 0.000323176 False 7 11 <class 'float'> <class 'float'>
CRPIX1 1113 1024 False 4 4 <class 'int'> <class 'int'>
CRPIX2 1036 1024 False 4 4 <class 'int'> <class 'int'>
CROTA2 70.03360090513783 -50.41824139313259 False 17 18 <class 'numpy.float64'> <class 'float'>
Though the values and sizes are matching it is showing false while comparing them. I don't know why.
When I am running your code, this is the error I am getting:
C:\Users\Anand\Anaconda3\python.exe C:/Users/Anand/Downloads/script_for_anand.py
C:\Users\Anand\Anaconda3\lib\site-packages\sunpy\util\decorators.py:378: SunpyMetadataWarning: Missing metadata for observer: assuming Earth-based observer.
new_val = prop(instance)
C:\Users\Anand\Anaconda3\lib\site-packages\sunpy\util\decorators.py:378: SunpyMetadataWarning: Missing metadata for observer: assuming Earth-based observer.
new_val = prop(instance)
WARNING: SunpyUserWarning: Integer input data has been cast to float64. [sunpy.image.transform]
C:\Users\Anand\Anaconda3\lib\site-packages\sunpy\util\decorators.py:378: SunpyMetadataWarning: Missing metadata for observer: assuming Earth-based observer.
new_val = prop(instance)
Traceback (most recent call last):
File "C:/Users/Anand/Downloads/script_for_anand.py", line 44, in <module>
m.draw_limb()
File "C:\Users\Anand\Anaconda3\lib\site-packages\sunpy\map\mapbase.py", line 1959, in draw_limb
radius = self.rsun_obs.to(u.deg).value
File "C:\Users\Anand\Anaconda3\lib\site-packages\astropy\units\core.py", line 987, in to
return self._get_converter(other, equivalencies=equivalencies)(value)
File "C:\Users\Anand\Anaconda3\lib\site-packages\astropy\units\core.py", line 918, in _get_converter
raise exc
File "C:\Users\Anand\Anaconda3\lib\site-packages\astropy\units\core.py", line 904, in _get_converter
self, other, self._normalize_equivalencies(equivalencies))
File "C:\Users\Anand\Anaconda3\lib\site-packages\astropy\units\core.py", line 888, in _apply_equivalencies
unit_str, other_str))
astropy.units.core.UnitConversionError: 'arcsec2' (solid angle) and 'deg' (angle) are not convertible
Process finished with exit code 1
Please tell me, what I am doing wrong? While running your code with data not having all these header values, and giving them in the program, it is running with the same warnings.
regards,
Albert Y. Shih
Jun 7, 2021, 9:48:57 AM6/7/21
to su...@googlegroups.com
Hi, Anand,
> Though the values and sizes are matching it is showing false while comparing them. I don't know why.
I don't know either without seeing your comparison code. You probably aren't comparing the values themselves, but rather some instances that don't have a simple equality check.
> Please tell me, what I am doing wrong?
My guess is that you have a header value for the apparent angular radius of the Sun that is already an Astropy Quantity (i.e., `arcsec`), and so when `self.rsun_obs` internally tacks on `arcsec`, you end up with `arcsec^2`, which of course raise an error when it gets used. Can you print out your entire FITS header?
> While running your code with data not having all these header values, and giving them in the program, it is running with the same warnings.
To be clear, the *warnings* are okay. Your header (presumably) doesn't have information about the location of the observer (e.g., `HGLT_OBS`, `HGLN_OBS`, `DSUN_OBS`), so SunPy assumes that the observer is centered at Earth. Does running my code with data not having your self-generated header values return the same *error*?
Albert
To view this discussion on the web visit https://groups.google.com/d/msgid/sunpy/7ac2ff48-8615-4f6a-8d08-267a655ad70dn%40googlegroups.com.
Anand MN
Jun 8, 2021, 1:53:58 AM6/8/21
to su...@googlegroups.com
Dear Albert,
Your doubt is correct, I am having the radius in arcsec. here is my complete header;
SIMPLE = T / conforms to FITS standard
BITPIX = 32 / array data type
NAXIS = 2 / number of array dimensions
NAXIS1 = 1024
NAXIS2 = 1024
EXPTIME = 0.28
CREATOR = 'ANDOR iXON'
DATE = '20210223T022517Z'
DATE-OBS= '2021-02-23T02:25:17Z'
CTYPE1 = 'HPLN-TAN'
CTYPE2 = 'HPLT-TAN'
CUNIT1 = 'deg '
CUNIT2 = 'deg '
CRPIX1 = 512
CRPIX2 = 512
CROTA2 = -50.41838454628599
RADIUS = '969.527arcsec'
PLATE-X = '2.1 arcsec/pixel'
PLATE-Y = '2.1 arcsec/pixel'
LAMBDA = '393.3 '
CDELT1 = 0.00032
CDELT2 = 0.00032
EXPOSURE= 0.28
END
Even when I run your code with RAW data without all these self-generated headers, it is giving the warning on observatory details.
What should be the format of location? what are these parameters - `HGLT_OBS`, `HGLN_OBS`, `DSUN_OBS`
You received this message because you are subscribed to a topic in the Google Groups "SunPy" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/sunpy/uywAFSl-YX4/unsubscribe.
To unsubscribe from this group and all its topics, send an email to sunpy+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/sunpy/CAB4tczg3CmyZPmoXeg76o78vqXJoXr%3DR3ANpNG5L3qOcUPNmMg%40mail.gmail.com.
Anand MN
Jun 8, 2021, 3:38:15 AM6/8/21
to su...@googlegroups.com
Dear Albert,
I corrected the radius value and now it is drawing the limb and grid. Still, the warning for the observatory is showing.
Then, the output I am unable to save as jpg and fits files. How to do that?
Albert Y. Shih
Jun 8, 2021, 9:18:29 AM6/8/21
to su...@googlegroups.com
Hi, Anand,
> Even when I run your code with RAW data without all these self-generated headers, it is giving the warning on observatory details.
What should be the format of location? what are these parameters - `HGLT_OBS`, `HGLN_OBS`, `DSUN_OBS`
This page succinctly explains those keywords: https://www.atnf.csiro.au/people/mcalabre/WCS/wcslib/structauxprm.html
> Then, the output I am unable to save as jpg and fits files. How to do that?
For JPG output, you could use either the Matplotlib save button in the GUI or call Matplotlib's `savefig()` function. For FITS output, you can use the `.save()` method for the `Map` instance you've created.
Albert
To view this discussion on the web visit https://groups.google.com/d/msgid/sunpy/CACvv_-Pt76dZ3%3DnibNoSXBA%3DswHcHHvf-88ViJ0yivGHoTo7Bw%40mail.gmail.com.
Anand MN
Jun 8, 2021, 12:51:44 PM6/8/21
to su...@googlegroups.com
Dear Albert,
I had seen that discussion on .save() option for map instance. But I couldn't find that option with my map. Don't know whether it is because older sunpy. I had updated last month. Anyway I will check that.
Regards,
Anand
To view this discussion on the web visit https://groups.google.com/d/msgid/sunpy/CAB4tczjES1K2gqg65eSi1KKoJHSqsZ2SL5OqLRraPKCs%2BOAYiw%40mail.gmail.com.
Anand MN
Oct 13, 2021, 1:32:35 AM10/13/21
to SunPy
Dear Albert,
Sorry for the delayed reply. the save() option worked, but unfortunately only for fits output, and the grid is not shown on the output file.
Is it possible to get an interactive map? I need to get the coordinates of an active region from these maps just by clicking on the display figure.
How to do it?
regards,
Anand
Reply all
Reply to author
Forward
0 new messages