dcm4chee blob bug and huge MySQL

[andor]

Hi there!

I've found one of my installs has the famous blob bug that didn't filter private data before putting it in inst_attr table. I haven't noticed that until a new modality that sends LOTS of private data has made my MySQL database grow up to 250GB size.

I've been using the UpdateAttributesService but it's very slow with that big database, and I'm getting my hard disk full.

Also, that machine mysql is growing a huge ibdata file that wont shrink, as nobody noticed that innodb_file_per_table wasn't active there :P
Once I clean the database I'll have to dump it, activate innodb_file_per_table and import again.
Problems everywhere!

So, the quiestion is:
How safe is dropping the contents for inst_attrs for *all* the instances? Can I do it and then recreate it again later with UpdateAttributesService safely?

[Damien Evans]

I'm afraid that I don't have a good answer for you.  I've never run into this situation, don't use MySQL, and haven't used the UpdateAttributesService to completely restore purged blob fields.  My recommendation (what I would do) is clone this installation or do a fresh installation and test it out.  Sorry.  :(

--
You received this message because you are subscribed to the Google Groups "dcm4che" group.
To post to this group, send email to dcm...@googlegroups.com.
To unsubscribe from this group, send email to dcm4che+u...@googlegroups.com.
Visit this group at http://groups.google.com/group/dcm4che?hl=en.
 
 

[Alvaro G. [andor]]

Dumping and restoring a 250GB database is going to hurt... :P

I've read somewhere in the forum or wiki that inst_attrs is only used by clients searching for studies... Do you know if that is right?

El 13/11/12 17:44, Damien Evans escribió:

[Arnold Maderthaner]

Its used for searching but also when someone loads an object the dicom attributes in *_attrs will be put on top of the information from the file itself. 

[Alvaro G. [andor]]

So if it's used for searching but I only use patient name, id and date on searches, as that info t's already on the database, can I drop that safely? Can I recreate that with the UpdateAttributesService?

"the dicom attributes in *_attrs will be put on top of the information from the file itself"

You mean that, the phisical file on the hard disk will be overwritten or more info added to it?

Thanks!

El 19/11/12 11:32, Arnold Maderthaner escribió:

[Damien Evans]

I think that dropping that column would cause problems.  I think it could easily be tested with a smaller database.  Simply create a new installation with the old version, store some of this data with the large amount of private tags, and run your tests to see what happens.  You don't necessarily need to do it on a clone of your existing database to see what the behavior would be.  I am interested in this and would test it out myself, but I am preparing for the RSNA conference and am very short on time this week, and will be out of town next week.

Regarding Arnold's comment below, he meant that when the file is retrieved, the *_attrs columns are overlaid onto the DICOM object before it leaves the system over the network.  They aren't written to disk.  It is simply an efficient way of managing updates to the data without having to constantly read/write from/to the disk.  

 -- Damien

[Alvaro G. [andor]]

I've made a pair of tests:

• I can drop the column, and searches will work, but trying to get studies without that column will end on a Unknown Format error:

2012-11-20 17:30:33,606 ERROR OSIRIX->DCM4CHEE (Thread-29953) [org.dcm4chex.archive.dcm.qrscp.QueryRetrieveScpService] org.dcm4che.data.DcmParseException: Unknown Format
java.lang.IllegalArgumentException: org.dcm4che.data.DcmParseException: Unknown Format
        at org.dcm4chex.archive.common.DatasetUtils.fromByteArray(DatasetUtils.java:94)
        at org.dcm4chex.archive.common.DatasetUtils.fromByteArray(DatasetUtils.java:80)
        at org.dcm4chex.archive.dcm.qrscp.QueryRetrieveScpService.makeCStoreRQ(QueryRetrieveScpService.java:1436)
        at org.dcm4chex.archive.dcm.qrscp.GetTask.retrieveInstances(GetTask.java:285)
        at org.dcm4chex.archive.dcm.qrscp.GetTask.run(GetTask.java:237)
        at java.lang.Thread.run(Thread.java:662)

• Yes, executing the UpdateAttributesService over that studies will recover its blob.

I'm thinking on dropping that column on all the old offline studies, so I can reduce in one third or fourth the size of the db, then, upgrade MySQL server to 5.5, activate activate innodb_file_per_table (ahem), then regenerate them with UpdateAttributesService.

:P
Thanks!

El 21/11/12 07:17, Damien Evans escribió:

[Damien Evans]

Thanks for the update.  Buena suerte!