Hi Jonathan

Thank you for your detailed steps of the wavelet transform, links and comments. Its a big help as a) I have very little prior experience with wavelet transforms and b) have been struggling to make any progress with either Haar and Shannon DWT using Wolfram and c) did not expect anyone to respond!

Taken together my sense of this paper is 1. DWT are not well suited to de noising financial time series indicators. 2. Similar to Fourier transforms they introduce yet another layer of complexity and possible side effects/interactions that increase model risk. Charts showing transformed series are one view of that. 3. Bandpass/Lowpass filters are still a very credible alternative for de-noising OHLCV by comparison. 4. Without the full code supplied by the authors for this paper, look ahead and peeking bias is the more likely driver of the very high predictive accuracy as opposed to the efficacy or parameterization of wavelet transfers. The generic set of indicators and exclusion of any volume or entropy based indicators adds weight to that. 5. Too good to be true is usually...

I'm not done with filters/de-noising so will post follow up work here even if its does not track this paper's workflow exactly.

Thank you again Jonathan. Your help on this has been incredible and really appreciate the time and energy taken to do this.

So this is a brief excursion into some of the rest of the methodology outlined in the paper. First up, we need to produce data for training, validation and testing. I am doing this for just the first batch of data. We would then move the window forward + 3 months, rinse and repeat.

Note that

(1) The data is being standardized. If you dont do this the outputs from the autoencoders is mostly just 1s and 0s. Same happens if you use Min/Max scaling.

(2) We use the mean and standard deviation from the training dataset to normalize the test dataset. This is a trap that too many researchers fall into - standardizing the test dataset using the mean and standard deviation of the test dataset is feeding forward information.

In[33]:= startISIdx = 1;
endISIdx = -1 + 
      Position[wlDates, x_ /; x > DateObject[{2009, 12, 31}, "Day"]] //
      First // First // Quiet;
endISDate = wlDates[[endISIdx]];

In[134]:= startValIdx = endISIdx + 1;
startValDate = wlDates[[startValIdx]];
endValDate = DatePlus[startValDate, Quantity[3, "Months"]];
endValIdx = 
  Position[wlDates, x_ /; x >= endValDate] // First // First // 
   Quiet;
endValDate = wlDates[[endValIdx]];
startTestIdx = endValIdx + 1;
startTestDate = wlDates[[startTestIdx]];
endTestDate = DatePlus[startTestDate, Quantity[3, "Months"]];
endTestIdx = 
  Position[wlDates, x_ /; x >= endTestDate] // First // First // Quiet;
endTestDate = wlDates[[endTestIdx]];

In[147]:= 
inputData = Standardize[transformedData[[startISIdx ;; endISIdx]]];
valData = 
  Standardize[transformedData[[startValIdx ;; endValIdx]], isRange];
testData = transformedData[[startTestIdx ;; endTestIdx]];
isMean = Mean[transformedData[[startISIdx ;; endISIdx]]] ;
isMean = ArrayReshape[isMean, Dimensions@testData, isMean];
isStDev = 
  StandardDeviation[transformedData[[startISIdx ;; endISIdx]]] ;
isStDev = ArrayReshape[isStDev, Dimensions@testData, isStDev];
testData = 
  isMean + 
   isStDev*Standardize[transformedData[[startTestIdx ;; endTestIdx]]];

In[116]:= Dimensions /@ {inputData, valData}

Out[116]= {{503, 17}, {63, 17}}

Now, admittedly, an argument can be made that a properly constructed LSTM model would outperform a simple gradient-boosted tree - but not by the amount that would be required to improve the prediction accuracy from around 50% to nearer 95%, the level claimed in the paper. At most I would expect to se a 1% to 5% improvement in forecast accuracy.

So what this suggests to me is that the researchers have got something wrong, allowing forward information to leak into the modeling process. The most likely culprits are:

1. Applying DWT transforms to the entire dataset, instead of the training and test sets individually
2. Standardzing the test dataset using the mean and standard deviation of the test dataset, instead of the training data set

Now we can produce the output from the autoencoder stack for both the training and test data:

encodedInputData = 
  TrainedEncoders[[4]][
   TrainedEncoders[[3]][
    TrainedEncoders[[2]][TrainedEncoders[[1]][inputData]]]];

encodedTestData = 
  TrainedEncoders[[4]][
   TrainedEncoders[[3]][
    TrainedEncoders[[2]][TrainedEncoders[[1]][testData]]]];

So this is a brief excursion into some of the rest of the methodology outlined in the paper. First up, we need to produce data for training, validation and testing. I am doing this for just the first batch of data. We would then move the window forward + 3 months, rinse and repeat.

Note that

(1) The data is being standardized. If you dont do this the outputs from the autoencoders is mostly just 1s and 0s. Same happens if you use Min/Max scaling.

(2) We use the mean and standard deviation from the training dataset to normalize the test dataset. This is a trap that too many researchers fall into - standardizing the test dataset using the mean and standard deviation of the test dataset is feeding forward information.

In[33]:= startISIdx = 1;
endISIdx = -1 + 
      Position[wlDates, x_ /; x > DateObject[{2009, 12, 31}, "Day"]] //
      First // First // Quiet;
endISDate = wlDates[[endISIdx]];

In[134]:= startValIdx = endISIdx + 1;
startValDate = wlDates[[startValIdx]];
endValDate = DatePlus[startValDate, Quantity[3, "Months"]];
endValIdx = 
  Position[wlDates, x_ /; x >= endValDate] // First // First // 
   Quiet;
endValDate = wlDates[[endValIdx]];
startTestIdx = endValIdx + 1;
startTestDate = wlDates[[startTestIdx]];
endTestDate = DatePlus[startTestDate, Quantity[3, "Months"]];
endTestIdx = 
  Position[wlDates, x_ /; x >= endTestDate] // First // First // Quiet;
endTestDate = wlDates[[endTestIdx]];

In[147]:= 
inputData = Standardize[transformedData[[startISIdx ;; endISIdx]]];
valData = 
  Standardize[transformedData[[startValIdx ;; endValIdx]], isRange];
testData = transformedData[[startTestIdx ;; endTestIdx]];
isMean = Mean[transformedData[[startISIdx ;; endISIdx]]] ;
isMean = ArrayReshape[isMean, Dimensions@testData, isMean];
isStDev = 
  StandardDeviation[transformedData[[startISIdx ;; endISIdx]]] ;
isStDev = ArrayReshape[isStDev, Dimensions@testData, isStDev];
testData = 
  (transformedData[[startTestIdx ;; endTestIdx]) - isMean) / isStDev;

In[116]:= Dimensions /@ {inputData, valData}

Out[116]= {{503, 17}, {63, 17}}

While introducing unwanted fluctuations in the US Dollar Index:

Well ok, perhaps I'll take a look.

So I think something like this to start, except that, as I said before, we want to do this on each in-sample dataset separately, not on the whole dataset. The variables closely match those described in the paper (see headers for details). Of course, I don't know the parameter values they used for most of the technical indicators, but it possibly doesn't matter all that much.

Note that I am applying DWT using the Haar wavelet twice: once to the original data and then again to the transformed data. This has the effect of filtering out higher frequency "noise" in the data, which is the object of the exercise. If follow this you will also see that the DWT actually adds noisy fluctuations to the USDollar index and 13-Wek TBill series. So I'm thinking these should probably be excluded from the de-noising process. We don't have sufficient detail in the paper to know exactly what the researchers did originally.

8-20-2022 Hi Jonathan and Warren, Very interesting and hope it is OK for me to submit a few questions re. validation. Would the application of AI neuron layers also involve the use of training data followed by analysis of future incoming data? Could another AI then be trained to analyze for errors and inconsistencies?

Hi Warren,

There is a syntax error, a missing comma, in the FinancialData expression. Try

data = FinancialData["SP500", "AdjustedOHLCV", {{2008, 7, 01}, {2016, 9, 30}}]