2d linear prediction

[Luca Codutti]

Hi Everyone,
I am trying to remove some bad FIDs on my Bruker 2D spectra. To do that I would like to do use nmrglue lp2d function, by forward predicting each bad point (3 over 1600, which are randomly placed during the acquisition) and replacing them with the predicted points.
Do you have any example of using the 2Dlp function? I could not find anything online about it.
All the best!

[Troels Emtekær Linnet]

Hi Luca.

Have you tried nmrPipe for that ?

http://spin.niddk.nih.gov/NMRPipe/ref/nmrpipe/lp.html

Or you can consider using Compressed Sensing for solving bad points in qMDD?.

http://nmrwiki.org/wiki/index.php?title=MddNMR

Troels Emtekær Linnet

2014/1/17 Luca Codutti <lucac...@gmail.com>

--
You received this message because you are subscribed to the Google Groups "nmrglue-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to nmrglue-discu...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[Luca Codutti]

Hey Troels,
thanks for your quick reply. I wanted to avoid NMRPipe since I do not want to go into several conversions of my spectra (I need to go to Felix format in the end). I did not know about MddNMR and I will take a look. I was just curious if nmrglue could handle lp prediction.

I am now in this situation:
I do have my data read by nmrglue, which shape is (420,2048). So far so good.
Lets assume my bad FID is number 60
so what I would do is nmrglue.process.proc_lp.lp2d(data[:60],pred,P,M) as a trial where pred is the number of points I want to predict, P and M are the prediction matrix dimensions. Supposedly I would then transpose the predicted matrix and re-apply the prediction function.
The problem is that I end up with a memory error, eg with arguments (data[:60],32,16,8), and I don't know what I am doing wrong.
Any hints?

On Friday, January 17, 2014 7:19:32 PM UTC+1, Luca Codutti wrote:

[Troels Emtekær Linnet]

Hi Luca.

From the definition of your 420,2048 matrix, I guess that you have recorded

1024 points in the direct dimension with complex => 2048 points

For the 420 number I guess that you 210 increments in the in-direct dimension?

When you refer to FID number 60, I guess that increment 60 has gone wrong, and all 1024 points are rubbish.

Is this the case?

If this is the case, I do not believe that Forward or Backward prediction can help you.

There is no information on how the wave should look like.

If just some of the 1024 points are rubbish, then you might have luck with linear prediction.

If the whole increment 60 is rubbish, then it can be seen as your dataset is sparse.

"non-uniformly sampled NMR data cannot be processed directly using fast Fourier transform (FFT)"

http://www.ncbi.nlm.nih.gov/pubmed/20236843

And Linear Prediction is not made for this.

In qMDD, you can transform your spectrum defining that FID 60 is "gone".

Then either reconstruct with FT (Zero filling), or reconstruct the missing FID, from nearby information (CS  Compressed sensing).

In our Lab we routinely use qMDD for recording sparse 3D data.

We think it is well working, with a easy GUI, and easy to process.

We have tried several NLS software, and qMDD is by far the easiest we have tried yet.

The output format is nmrPipe, which should be possible to convert to Felix.

(http://qa.nmrwiki.org/question/207/converting-nmrpipe-processed-data-ft3-files-to-felix-matrix ?? )

Best

Troels

Troels Emtekær Linnet

2014/1/17 Luca Codutti <lucac...@gmail.com>

Hey Troels,

thanks for your quick reply. I wanted to avoid NMRPipe since I do not want to go into several conversions of my spectra (I need to go to Felix format in the end). I did not know about MddNMR and I will take a look. I was just curious if nmrglue could handle lp prediction.

I am now in this situation:
I do have my data read by nmrglue, which shape is (420,2048). So far so good.
Lets assume my bad FID is number 60
so what I would do is nmrglue.process.proc_lp.lp2d(data[:60],pred,P,M) as a trial where pred is the number of points I want to predict, P and M are the prediction matrix dimensions. Supposedly I would then transpose the predicted matrix and re-apply the prediction function.
The problem is that I end up with a memory error, eg with arguments (data[:60],32,16,8), and I don't know what I am doing wrong.
Any hints?

[Jonathan J. Helmus]

Luca,

The linear predictions routines in nmrglue are probably not a good option for processing real NMR data, as mentioned in the documentation for the proc_lp modules [1], the LP function are not numerically stable nor fast, the first of which may introduce undesired artifacts into real dataset.  In truth, the LP functions in nmrglue most likely  only work on small toy datasets and they have only been tested on such dataset.  I wrote them to teach myself how linear prediction works and kept them in nmrglue in case anyone else wanted to see how such methods work with actual code.  You would be much better suited using the LP routines on NMRPipe or RNMRTK, or using a non-Fourier bases processing method like Maximum Entropy or MDD (RNMRTK has the former, qMDD the later and I'm sure there are other package which have similar routines).

If an entire FID is corrupt is should be possible to use LP to predict the missing points along each indirect dimension vector, assuming that there is sufficient data prior to the corrupt FID.  This is not the typical use of LP and so I can't comment as to how well it would work.  Another option might be to replace the corrupt FID with one interpolated from the FIDs with the same quadrature just prior to and immediately after the corrupted one.  This might give you an idea of what the spectrum should look like but I wouldn't use the results for anything qualitative and would want to rerun the experiment if I was to include it in anything published.  

If still want to try out the lp2d function in NMRPipe read the documentation carefully [2] and reading the reference article would also be helpful.  The 2d data array should have time along the last axes and frequency along the leading axes.  This means you will need to perform a Fourier transform along at least one axes.  Predicted points are appended to the last dimension after the current data. Also your data must be collected with either no initial delay or a half-point delay along the dimension to be predicted. 

From the code you provided it looks as if you had the axes reversed, if you want to correct FID 60, you need to first process the direct dimension, then perform a hypercomplex transpose taking into account the indirect dimension encoding (State, TPPI, etc).  Then you can call the lp2d function.  Also note that the memory demands for the 2D LP algorithm are quite large, especially when P and M are large.  Unfortunately the larger P and M are the better prediction you typically get.  

Cheers,

- Jonathan Helmus

[1] http://jjhelmus.github.io/nmrglue/current/reference/proc_lp.html

[2] http://jjhelmus.github.io/nmrglue/current/reference/generated/nmrglue.process.proc_lp.lp2d.html

[Luca Codutti]

Thank you very much!
I will apply your suggestions.
Luca