Snappy Vms

[Karoline Oum]

Ihave come back to use snappy after some time, and I am seeing a slower processing of the snappy module when writing a product in a numpy array (10390x10390 pixels) , and also when trying to display a band with matplotlib , it do needs quite a longer time than before. Is this something common or it could only be something related to my system ( but I have not found any problem yet with it)

yes, I keep SNAP updated to the latest release. Actually the critical operation is to display the image 10980x10980 pixels with matplotlib, it really needs a lot of time. Do you think it could be caused by the heavy size of the image?

Hello,

I have a question about the performance of snappy compared to the performance of SNAP toolbox.

I am using SNAP toolbox for deriving a subsidence map. I decided to create a graph (the one that is shown below) to automate the procedure.

graph.png906258 20.8 KB

I also found convenient using the snappy in python for automating this procedure without having to start SNAP toolbox. The code i wrote is provided below. The code works perfectly fine but the only issue I encounter is the speed.

When I load the graph into SNAP, it takes no more than 1.5 hour for the whole process. On the other hand, when I run the code with snappy, it takes more than 4 hours. It does not make much sense to me because we are effectively accessing the JAVA modules using python as an interface. Hence, one expect to get results as fast as in SNAP toolbox

Is there anything particular in my code that might slow down the process? Do we expect snappy to be slower compared to SNAP?

It would be very useful to know why running snappy takes so much time for the process to complete.

I would appreciate any feedbacks

I have automated (pretty much) the whole procedure for producing a map subsidence using snappy.

I got stuck at the last step where we need to export the wrapped phase to snaphu format.

marpet, one of the developers kindly offered to help me with that by looking at my SAR data. Once this step is done, then i will upload the whole InSAR pipeline in this forum for everyone to use.

It will make our life easier

Python is much nicer than bash to prepare the workflow.

Yes I could use gpt+graph but since you paid for the development of snappy go all the way and make it work

You have plenty of users in this forum wanting to use Python and snappy.

I understand what you mean. I just used snappy because i wanted to automate some tasks. I found it very useful but i had issues with performance (hopefully, you do not run snappy on a jupyter notebook as it gets even slower).

To be honest, i gave up on snappy and used GPD and bash script instead and everything works just fine.

I wish we did not have such performance issues with snappy as the majority of people prefer using python.

On the tutorials page in the External Resources Category you find a link to some python examples. These will be extended in the future.

As snappy is just a wrapper around the SNAP Java API you can do almost everything what you can do in Java.

So you can also look at Java example code.

As @marpet said, in the external resources you can find the a link to a github repository with first basics tutorials about the usage of Snappy . From now on the repository will be quite regularly enriched with new tutorials that will deal with various functionalities of SNAP exploited in Python. As a little example the following RGB image has been obtained in Python as a result of a processing chain that resamples and extracted a subset of a full size Sentinel2 data product, with a contrast enhancing operation as the last step.

I work with osnaps turned on, and hold alt to temporarily disable. However I work with smooth gumball, since I usually use it to nudge things around. Occasionally I turn on snappy dragging, but holding alt would be simpler and expected behaviour.

Before everyone rushes to mark this as a duplicate, I have indeed read the other threads here and here, googled this, and visited the snappy homepage here searching to clear my confusion, and I couldn't. I ask here because I still remain confused. I understand that it is some sort of a minimal working ubuntu, and the spirit of this question is not what is "in it", rather what it is and how it works.

Will regular ubuntu applications work if I install snappy (the same thing that works on my normal ubuntu install using apt-get or by downloading a deb package), or does someone need to do something special to make an app work in snappy?

apt-get and other utilities for working with traditional .deb packages do not work to install and upgrade applications on a Snappy Ubuntu Core system. Instead you use the snappy utility. See the Snappy Tour for details.

And to install updates, you run sudo snappy update-versions (which corresponds to sudo apt-get update on a normal Ubuntu system), snappy versions to see what's newly available, and sudo snappy update ... to specify packages for updating (put their names in place of ...).

Because Snappy Ubuntu Core doesn't use .deb packages, Ubuntu packages created for regular Ubuntu systems will not work--the relationship between regular Ubuntu systems and Snappy Ubuntu Core when it comes to package files is, in effect, the same as the relationship between any two distributions that use totally different package managers. (For example: Ubuntu and Fedora.)

Like in just about any OS, you could manually install programs on Snappy Ubuntu Core provided you have or can obtain all the libraries (and any other dependencies) they need. However, just as the best way to install most software on regular Ubuntu systems is with a .deb package, the best way to install software on Snappy Ubuntu Core is with specially built snappy packages. Only a handful of these exist so far.

As for Raspberry Pi, those have ARM processors, which you're right are not the same as the usual 32-bit and 64-bit Intel and AMD processors most of us use on our traditional PCs. However, Raspberry Pi support does not appear to be the main point of Snappy Ubuntu Core. Instead, as you may be aware, the general aim of Snappy is for better speed, stability, and security, in circumstances where a minimal Ubuntu Core system is sufficient and appropriate.

Snappy is cloud-oriented ("Snappy Ubuntu Core is the perfect system for large-scale cloud container deployments..."), and explicitly supports many x86 (i.e., not ARM) platforms--see the list of options under "Try the new, snappy Ubuntu yourself!" on the Snappy home page.

Nonetheless, depending on your needs, Snappy Ubuntu Core may be a reasonable way to get a working Ubuntu Core system on a Raspberry Pi. Depending on which Raspberry Pi board you have, it may or may not support a fuller, more traditional installation of Ubuntu. Note, though, that there are other options that may be more Ubuntu-like in the ways that matter to most users of traditional Ubuntu systems:

Snappy Ubuntu Core is like a Lego system, each snap (app) is a block not affected by other snaps. So security is simpler and higher, dependencies are no longer an issue and the lifecycle of snaps are much easier. With a single command you can exchange versions, eliminate and upgrade any snapp and even the core (OS).

Instantly feed your hungry party of 5-10 people with our NEW group meal options, available for same-day pickup or delivery. Conveniently order online through our Snappy Rewards app or at snappysalads.com, and earn Reward points with your group meals. Need to feed more than 10 people? Schedule your catering order online at catering.snappysalads.com, call our catering concierge at 1-866-991-5006, or email cate...@snappysalads.com.

I am trying to read a snappy.parquet file using dask and later on, convert it to a dask array that I will be using to train my machine learning model. One of the columns is an object (supposed to be image that needs to be unpacked) and all the rest are of float64 type. Here is a dask dataframe that I am referring to.

I am having trouble to convert this object column into a dask array. When I executed data.compute() , the floats show but the objects in the X_jets column do not. As a matter of fact, it even returned a None value. Been stuck in this problem for already a day and I am planning to also convert it to dask array. Any help would be appreciated. Thanks

There is a pytorch implementation of it using some self.transform execution under a class for preprocessing, but I just dont want to copy paste it. I want to learn something new. To open the parquet file, it was done using pyarrow.parquet, with a code snippet below as

Hi @guillaumeeb! Thank you for the warm welcome and response. I tried using the pyarrow engine before and when the unpacking comes, it turns out that I am having an OOM (Out Of Memory) error. The snippet that I used was

The file is quite huge, so I attached a google drive link for this one (can be accessed through QCDToGGQQ_IMGjet_RH1all_jet0_run1_n47540.test.snappy.parquet - Google Drive) . On the pytorch implementation, colleagues simply used pyarrow.parquet to read the X_jets column (which is supposed to be a 125x125 image size composed of an array) as can be seen on the code below.

Edit: The X_jets I believe contains three sets of data, all of which are images of size 125x125. I think the names areTracks, Ecal, and HCal in order. What was done here is for HCal. The schema looks like:

X_jets: list child 0, item: list child 0, item: list child 0, item: double

Could you give us a more complete code snippet of what you are trying to achieve? The code above will just load all the chunks in main process memory one by one, and so probably overwhelm your machine if the file is huge. Do you really want to load all the data in memory at the same time?

3a8082e126