Archiving Images ?

[Sylvain LAFRASSE]

Dear all,

Thanks to Sébastien GARA, I currently play with HDB++ and co.
Am I missing something, or images can’t be archived this way ?
More generally, I can't find a document describing which attribute type can be archived or not.
Any insight on this issue ?

Best regards,
Sylvain LAFRASSE

[Sergi Rubio]

Well, I had previous experience archiving images on java HDB over mysql
and it presents lots of performance issues because the data was
effectively written as strings. In HDB++ each array value is archived
separately with an index, so data could be saved as a 1D spectrum and
later reconstructed by clients using the information stored in attr_conf
(of course, this forces you to use fixed-size images).

I don't know if HDB++ running on cassandra could provide a native type
for images.

You can see which types are supported just listing the tables in the schema:

| att_array_devboolean_ro |
| att_array_devboolean_rw |
| att_array_devdouble_ro |
| att_array_devdouble_rw |
| att_array_devencoded_ro |
| att_array_devencoded_rw |
| att_array_devfloat_ro |
| att_array_devfloat_rw |
| att_array_devlong64_ro |
| att_array_devlong64_rw |
| att_array_devlong_ro |
| att_array_devlong_rw |
| att_array_devshort_ro |
| att_array_devshort_rw |
| att_array_devstate_ro |
| att_array_devstate_rw |
| att_array_devstring_ro |
| att_array_devstring_rw |
| att_array_devuchar_ro |
| att_array_devuchar_rw |
| att_array_devulong64_ro |
| att_array_devulong64_rw |
| att_array_devulong_ro |
| att_array_devulong_rw |
| att_array_devushort_ro |
| att_array_devushort_rw |
| att_scalar_devboolean_ro |
| att_scalar_devboolean_rw |
| att_scalar_devdouble_ro |
| att_scalar_devdouble_rw |
| att_scalar_devencoded_ro |
| att_scalar_devencoded_rw |
| att_scalar_devfloat_ro |
| att_scalar_devfloat_rw |
| att_scalar_devlong64_ro |
| att_scalar_devlong64_rw |
| att_scalar_devlong_ro |
| att_scalar_devlong_rw |
| att_scalar_devshort_ro |
| att_scalar_devshort_rw |
| att_scalar_devstate_ro |
| att_scalar_devstate_rw |
| att_scalar_devstring_ro |
| att_scalar_devstring_rw |
| att_scalar_devuchar_ro |
| att_scalar_devuchar_rw |
| att_scalar_devulong64_ro |
| att_scalar_devulong64_rw |
| att_scalar_devulong_ro |
| att_scalar_devulong_rw |
| att_scalar_devushort_ro |
| att_scalar_devushort_rw |

Hope it helps,

Sergi Rubio, ALBA



"Earthquakes, in fact, are a really good example of a push based stream.
We're not polling the earth for earthquakes." - Erik Meijer

[Reynald Bourtembourg]

Hi Sergi and Sylvain,

Please find my comments below.

On 18/10/2016 09:30, Sergi Rubio wrote:

Well, I had previous experience archiving images on java HDB over mysql and it presents lots of performance issues because the data was effectively written as strings. In HDB++ each array value is archived separately with an index, so data could be saved as a 1D spectrum and later reconstructed by clients using the information stored in attr_conf (of course, this forces you to use fixed-size images).

In the case of HDB++ MySQL, it seems the dimensions of spectrums and images are stored in att_array_devxxx tables, not in att_conf table. So the support for images and spectrums having dynamic dimensions should already be supported by HDB++ MySQL. There is even a distinction between the dimensions of the read value and of the set value (dim_x_r, dim_y_r, dim_x_w, dim_y_r fields from att_array_devxxx_rw tables).
I do not know how the extraction GUIs are handling that though.

You can have a look at the table definitions from the HDB++ Design Guidelines document which can be found on the following web page:
http://www.tango-controls.org/community/project-docs/hdbplusplus/hdbplusplus-doc/hdbplusplus-design-guidelines

I don't know if HDB++ running on cassandra could provide a native type for images.

HDB++ cassandra is storing arrays and images using CQL (Cassandra Query Language) list type.
HDB++ Cassandra does not currently support correctly images because the dimensions of the images are currently not saved.
So images are stored into a 1D list but the dimensions are currently not saved. So there is no way to reconstruct the image correctly for the moment in the case where dim_y > 1.
This could be improved to support correctly small images.
I wrote "small images" because HDB++ Cassandra is not a good choice if you want to archive big images.
Cassandra is not a god fit to store images and big arrays.

Cassandra is a distributed database. So there is the notion of partition.
According to Cassandra specialists, a partition should not be bigger than about 100 MB on disk otherwise you will run into performance (and operational) issues.
In the current version (DB schema) of libhdb++cassandra, there is one partition per day and per attribute.
So if you want to archive an attribute which is a spectrum or an image, the disk space taken to save all the data for this attribute per day should ideally not exceed 100MB. Which is a big limitation in the case of images.

There will probably be evolutions of HDB++ Cassandra in the future to be able to create more than one partition per day for a given attribute, for example, having one partition per hour for some attributes but we do not intend to support the possibility to store big images at high frequency because Cassandra technology does not seem a good fit for this kind of use case.

About the supported types, please be aware that DevULong64 types are currently not supported correctly by HDB++ Cassandra, the data in this case is stored  and "casted" into 64-bit signed longs. This should be improved in the future.
Sorry for the lack of documentation for HDB++ Cassandra. This will be improved in the next months.

I hope it answers your questions.

Kind regards

Reynald