Group Abstract Group Abstract

Message Boards Message Boards

0
|
15.5K Views
|
7 Replies
|
8 Total Likes
View groups...
Share
Share this post:

How to smooth and filter the signal?

Posted 11 years ago
POSTED BY: Peter Parker
7 Replies
POSTED BY: EDITORIAL BOARD
Posted 11 years ago

Rather than doing an interpolation followed by an FFT, I encourage you to investigate the Lomb-Scargle approach. This will give you a periodogram without interpolation and the additional errors that involves. Google should help. Also, this book isn't bad: "Bayesian Logical Data Analysis for the Physical Sciences: A Comparative Approach with Mathematica Support".

POSTED BY: Kevin McCann
POSTED BY: Peter Parker
Posted 11 years ago

So, first a question: what do you know about the data? For example, is it a sinusoidal signal + exponential + noise? (Just a guess) If you know enough about the signal, you can do a least-squares fit to a model with parameters, e.g. frequency of the sinusoid, exponential coefficient, and then retrieve the coefficients. I would assume that the "noise" other than the sinusoid is Gaussian, but you may know otherwise.

BTW, a Fourier Transform is in fact a least-squares fit to the data by a sum of sinusoids.

POSTED BY: Kevin McCann
POSTED BY: Sean Clarke

The workflow for filtering with a fourier transform is:

  1. Apply Fourier to a time series
  2. Remove small frequency components
  3. Apply InverseFourier to get back a time series

The code below assumes your dataset is called data.

We just want the "y" values for the time series, so we'll be using data[[All,2]]. We don't really need the "x" values - the data points should be and are evenly spaced.

Step One:

fdata = Fourier[data[[All, 2]]];

Step Two:

fdataFiltered = Threshold[fdata, {"Hard", 0.0005}]

or

fdataFiltered = (fdata /. x_ /; Abs[x] < 0.0005 -> 0) 

Step Three:

smoothedData = InverseFourier[fdataFiltered]

Fourier transforms aren't the only way to filter time series. There are a lot of different options for different purposes. WeinerFilter works fine for many cases:

WienerFilter[data[[All, 2]], 3, 0.1]
POSTED BY: Sean Clarke

This is really great. Now I know and understand how to filter the signal from the noise. Thank you very much!

But what is about the other problem I have mentioned? As you can see, my signal is superimposed by an other sinus which has a slightly longer frequency. I would like to filter this out. As I mentioned, I have heard that I need to transform my wavelength-signal into a frequency-signal. Only the frequency signal can be later used for the Fourier and InverseFouier functions. Otherwhise I will change the results. I have transformed my data to frequency in my first post by simply using the equation c=lambda*frequency. But in the next step I have to apply some kind of array in order to correct the differente step sizes. As far as I know the FFT accepts data sets of any length, but they must be sampled regulary (same step size)!!! Do you know what I mean? This is what other usually do, as you can see here on page 3: https://www.phys.ksu.edu/personal/washburn/Teaching/Class%20Files/NQO/Tutorials/Tutorial8_FFT.pdf I don't really understand this and I don't know how to apply this to my problem.

What would you do?

Thanks in advance,

Peter

POSTED BY: Peter Parker
Reply to this discussion
Community posts can be styled and formatted using the Markdown syntax.
Reply Preview
Attachments
Remove
or Discard