unexpected pickle.dump error

[Kristen Hunter-Cevera]

Hi Meren,

 We recently upgraded to Ubuntu 20.04 and Oligotyping v3.1 and ran into a python error surrounding pickle (see below for error messages) when trying to run decompose. We were able to get it working by changing the pickle open and load to expect bytes, but I'm not sure what else this hack may have altered (I'm not well-versed in Python!) If you have a better fix or advice for a remedy, that would be great (and I apologize if this ends up being a sole Python error on my side). Please let me know if you need any other information.

Thank you in advance for your help.

Kind regards,

Kristen

------------------------------------------------------------------------------------------------------------------------------------------

System: - Ubuntu 20.04

pip3 install oligotyping==3.1

Description Two consecutive errors with pickle.dump of dict of figures now being in bytes? So tried to allow pickle and unpickle of said bytes...

The Hack: Did a small local (hack) fix which lets `decompose` run through (but probably not right)

# change 1 of 2 ~/.local/lib/python3.8/site-packages/Oligotyping/lib/decomposer.py pickle.dump(figures_dict, open(figures_dict_file_path, 'wb'))

# change 2 of 2 ~/.local/lib/python3.8/site-packages/Oligotyping/utils/html/for_decomposition.py figures_dict = pickle.load(open(html_dict[html_dict_key],'rb'), encoding='bytes')

## full output of oligotyping not shown - only last few lines before errors ###########

Basic topology of MED nodes ......................................: /home/labuser/bham_files/Pseudoalteroseqs_for_oligotyping_pynast_aligned-m0.10-A0-M0-d4/TOPOLOGY.gexf
[10 Mar 21 16:43:04 Storing outliers] Storing reads removed due to "min_substantive_abundance_re (...[10 Mar 21 16:43:04 Storing outliers] Storing reads removed due to "maximum_variation_allowed_re (...Representative sequences per node ................................: /home/labuser/bham_files/Pseudoalteroseqs_for_oligotyping_pynast_aligned-m0.10-A0-M0-d4/NODE-REPRESENTATIVES.fasta
Read distribution among samples table ............................: /home/labuser/bham_files/Pseudoalteroseqs_for_oligotyping_pynast_aligned-m0.10-A0-M0-d4/READ-DISTRIBUTION.txt
End of run .......................................................: 10 Mar 21 16:43:04
GEXF file for network analysis ...................................: /home/labuser/bham_files/Pseudoalteroseqs_for_oligotyping_pynast_aligned-m0.10-A0-M0-d4/NETWORK.gexf
[10 Mar 21 16:43:04 Figures] NMDS Analysis "bray" ...                                               Traceback (most recent call last):
  File "/home/labuser/.local/bin/decompose", line 42, in <module>
    decomposer.decompose()
  File "/home/labuser/.local/lib/python3.8/site-packages/Oligotyping/lib/decomposer.py", line 365, in decompose
    self._generate_default_figures()
  File "/home/labuser/.local/lib/python3.8/site-packages/Oligotyping/lib/decomposer.py", line 1386, in _generate_default_figures
    pickle.dump(figures_dict, open(figures_dict_file_path, 'w'))
TypeError: write() argument must be str, not bytes

##==============================================================================
## AFTER ALLOWING BYTES in pickle.dump A NEW ERROR ON pickle.load BEING BYTES

...
Read distribution among samples table ............................: /home/labuser/bham_files/Pseudoalteroseqs_for_oligotyping_pynast_aligned-m0.10-A0-M0-d4/READ-DISTRIBUTION.txt                          
End of run .......................................................: 10 Mar 21 21:15:03
GEXF file for network analysis ...................................: /home/labuser/bham_files/Pseudoalteroseqs_for_oligotyping_pynast_aligned-m0.10-A0-M0-d4/NETWORK.gexf                                   
figures_dict_file_path ...........................................: /home/labuser/bham_files/Pseudoalteroseqs_for_oligotyping_pynast_aligned-m0.10-A0-M0-d4/FIGURES.cPickle                                
[10 Mar 21 21:15:03 HTML Output] Generating                                                                                                                                                               Traceback (most recent call last):
  File "/home/labuser/.local/bin/decompose", line 42, in <module>
    decomposer.decompose()
  File "/home/labuser/.local/lib/python3.8/site-packages/Oligotyping/lib/decomposer.py", line 385, in decompose
    self._generate_html_output()
  File "/home/labuser/.local/lib/python3.8/site-packages/Oligotyping/lib/decomposer.py", line 1357, in _generate_html_output
    index_page = generate_html_output(self.run.info_dict, html_output_directory = output_directory_for_html)
  File "/home/labuser/.local/lib/python3.8/site-packages/Oligotyping/utils/html/for_decomposition.py", line 218, in generate_html_output
    html_dict['figures_dict'] = get_figures_dict('figures_dict')
  File "/home/labuser/.local/lib/python3.8/site-packages/Oligotyping/utils/html/for_decomposition.py", line 203, in get_figures_dict
    figures_dict = pickle.load(open(html_dict[html_dict_key]), encoding='bytes')
  File "/usr/lib/python3.8/codecs.py", line 322, in decode
    (result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x80 in position 0: invalid start byte

[A. Murat Eren]

Dear Kristen,

Thank you very much for going after this error :) You clearly are quite well-versed in Python. Unfortunately I don't have means to test this at the moment, but I'm happy to send them to the repository on your behalf and update the release to v3.2 so it is available to everyone. Can you please confirm that these are the changes you propose:

Thank you again,

--

A. Murat Eren (Meren) | he/him
http://merenlab.org :: twitter

--
You received this message because you are subscribed to the Google Groups "Oligotyping and MED" group.
To unsubscribe from this group and stop receiving emails from it, send an email to oligotyping...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/oligotyping/0fde21f0-9b1d-4da8-b360-ccb996186221n%40googlegroups.com.

[Kristen Hunter-Cevera]

Hi Meren,

Thank you for the quick reply! Sorry for not including line numbers earlier, but yes, those are the correct locations, but the line in for_decomposition.py should read:

figures_dict = pickle.load(open(html_dict[html_dict_key],'rb'), encoding='bytes')

These changes worked for us, but I hadn't looked to see what else might be affected or might be missing downstream (and really - I'm not a python user - we just stackoverflowed our way through this!). Hope it helps though to start the troubleshooting or if others run into the same problem.

Thanks again,

Kristen

To view this discussion on the web visit https://groups.google.com/d/msgid/oligotyping/CAK5_Lad8j-UrbCFKECKszdZ0ybVHbFPnqRBPT-kSzehf5d9MJQ%40mail.gmail.com.

[A. Murat Eren]

Thank you, Kristen! These changes look appropriate to me and we will see if things break for others :)

--

A. Murat Eren (Meren) | he/him
http://merenlab.org :: twitter

To view this discussion on the web visit https://groups.google.com/d/msgid/oligotyping/CALekGtT6XbDU4ubjXUODjM1p6_eDu8sJTSncD-ZtDzmFfSe7vA%40mail.gmail.com.