Cross-correlation of two data sets

Questions
, ,
1

Hello all, I am very new to signal processing and my advisor has given me a job that I completely do not understand. I need to find the cross-correlation maximum between two spectrum data sets, each in the form of 2-D lists {{wavelength1 , intensity1} , {wavelength2 , intensity2}, etc.}. The wavelength steps are not the same between the two data sets, and for the first data set the wavelength steps change intermittently.

The instructions I received were:
-Create a new file that contains an offset of -.01 microns → .01 microns
-Fill in values in an array
-Apply offest for list #1, interpolate into list #2
-Calculate the Fourier Transform of the central maximum of the Cross-Correlation between list #1 and list #2.

I have very little idea what my advisor is telling me to do. Any help or suggestions would be greatly appreciated!

2

It might be helpful to know and think about what tools your advisor has used in the past for this kind of problem. The tools they used will probably inform how they describe the process that they’re looking for. I remember this being an issue for me with my internships. My advisors often communicated what they wanted using the process and format of old programs that they used. More often than not, the hardest part of the process was piecing together in formal terms, usually mathematical, what was needed.

I’m not sure exactly how accurately I can tell what needs to be done here. I’m pretty certain that you will have to go back and ask for more precise details about what should get done. More important than understanding the steps is understanding the goal. You will probably need to become decently familiar with programming in Mathematica to program a way to achieve that goal.

First , it looks like your data comprises of a pairs of wavelengths and insentities (amplitudes?). I am not sure how these pairs relate to the original signal you want to study. Before doing a Fourier Transform or anything, you’ll want to convert whatever form your data is in into a time series of the actual signal. You mentioned that the sampling isn’t even. If that is the case, you will probably want to do some kind of interpolation and then resample at an even sample rate. You can use Mathematica’s Interpolation function or write a custom interpolation scheme depending on your needs. Once you have both signals at the same sample rate, use ListConvolve to convolve the lists and compute an auto correlation. You can vary the offset used in the function to find the optimial offset.

3

I appreciate your advice. I figured out how to interpolate my data and generate new sample rates fairly easily. I got stuck on the ListConvolve function. The Mathematica Help tool gives its general template as ListConvolve[ker, list]. I am actually working with an observed spectrum (D) and a template spectrum (T), and I know that my Cross-Correlation Function (CCF) needs to be `CCF=(D*T)/(||D||.||T||), where * is a convolution, and ||D|| and ||T|| are the norms of each spectrum. Does this that mean that D needs to be the kernal in the ListConvolve function? Also, any insight on what an “offset” is and/or how to vary it in the ListConvolve function would be super helpful!

Based off the paper that I am trying to decipher, the idea is to calculate the position of the first zero of the Fourier Transform of the central maximum of the Cross Correlation Function between the object and template spectra. Thank you for your time!`

4

This tutorial shows more about how to do convolutions and correlations.

Yes the documentation shows Kernel, but that can just be another list. Both ListCorrelate and ListConvolve have an argument that changes the shift on the operation. Consider this code for example:

ListCorrelate[{a, b, c, d}, {w, x, y, z}, {1}, 0]

Which gives:

{a w + b x + c y + d z, a x + b y + c z, a y + b z, a z}
5

Thank you, I have made a lot of progress because of your help. Is there a way to convolve interpolated functions? My code so far looks like:

object = Import[ "C:\\Users\\John\\Documents\\Research\\Modeling\\interpolationtest.txt", "Table"]; 

template =   Import["C:\\Users\\John\\Documents\\Research\\Modeling\\interpolationtest2.txt", "Table"]; 

o = Interpolation[object,InterpolationOrder -> 3];

t = Interpolation[template,InterpolationOrder -> 3];

I am hoping to do something along the lines of:

Convolve[o,t]

But I cannot figure out what to do with all the variables and definitions.

6

Please take a quick look at the documentation for Convolve:

Convolve requires 4 arguments. Additionally the examples show that it is meant to take symbolic functions, not numerical functions.

I don’t want to complicate the math of this at all. But I should warn against using too many tools like interpolations if they are not needed. If you can resample the data set to keep it discrete and do a discrete convolution, I would recommend it instead of continuing down the path below.

Here’s the definition of a convolution. I will use this with a couple of example InterpolationFunction expressions I’ve constructed.

f = Interpolation[Table[{x, Sin[x]}, {x, 0, 10, 0.1}]];
g = Interpolation[Table[{x, Cos[E^x]}, {x, 0, 10, 0.1}]];

“f” and “g” are only defined from x = 0 to 10, so this will change the range of the integration. In short the integral is (I’m mostly sure):

Integrate[f[x]*g[y - x], {x, y, - y + 10}]

“y” of course must be between 0 and 10 for this to make sense.
Because f and g are not symbolic, they are numeric, we will have to use NIntegrate:

myConv[y_?NumericQ] := NIntegrate[f[x]*g[y - x], {x, y, y - 10}]

See this article for why the function was defined with ?NumericQ.