Hi Jonathan

There are a lot of undefined variables in your code such as: wlDates, TransformedData, dataDir, etc which make the evaluation of your code impossible to replicate

Regards Michael

Hi Warren,

How is your research progressing? I read the paper and I am deeply sceptical. The results are too good to be true, and suggest some kind of feed-forward of information.

Here is a link to the stem paper: https://tinyurl.com/4nuvwb5x As you can see, the approach is very similar.

I have no particular expertise in wavelet transforms, but I suspect the issue lies there. Neither the original research, nor this sequel, discussed the use of wavelet transforms in sufficient detail to be sure, but my guess is that they are preprocessing the entire dataset with wavelet transforms before applying it to the inputs of the stacked auto-encoders. This embeds future information into the wavelet transforms, since the coefficients are estimated using the entire dataset.

If I am right, this is an elementary error disguised by the use of a fancy denoising procedure. You sometimes see a similar error where researchers standardize the data using the mean and standard deviation of the entire dataset, which embeds future information about the process into the transformed data. This can lead to spuriously accurate prediction results.

The correct procedure, of course, is to de-noise the in-sample data only, and separately de-noise the out-of-sample data used for testing purposes. The researchers do use a set of rolling train/test datasets, which is fine, except that each training & test set should be de-noised individually.

That's my initial take. Perhaps someone with specific expertise on wavelet transforms can chime in and give us their view.

PS: This paper echos my concerns about the incorrect use of wavelets in a forecasting context and highlights the concerns I expressed above:

The incorrect development of these wavelet-based forecasting models occurs during wavelet decomposition (the process of extracting high- and low-frequency information into different sub-time series known as wavelet and scaling coefficients, respectively) and as a result introduces error into the forecast model inputs. The source of this error is due to the boundary condition that is associated with wavelet decomposition (and the wavelet and scaling coefficients) and is linked to three main issues: 1) using ‘future data’ (i.e., data from the future that is not available); 2) inappropriately selecting decomposition levels and wavelet filters; and 3) not carefully partitioning calibration and validation data.

I think what I am saying is that the whole methodology may be invalidated by the method used to de-noise the dataset. I don't see a way forward from there, other than to do the analysis properly, i.e. de-noise each individual in-sample and out-of-sample dataset independently. I rather doubt whether the results will be very interesting after that. I'm trying to summon the energy to do the analysis to demonstrate my point.

Well ok, perhaps I'll take a look.

So I think something like this to start, except that, as I said before, we want to do this on each in-sample dataset separately, not on the whole dataset. The variables closely match those described in the paper (see headers for details). Of course, I don't know the parameter values they used for most of the technical indicators, but it possibly doesn't matter all that much.

Note that I am applying DWT using the Haar wavelet twice: once to the original data and then again to the transformed data. This has the effect of filtering out higher frequency "noise" in the data, which is the object of the exercise. If follow this you will also see that the DWT actually adds noisy fluctuations to the USDollar index and 13-Wek TBill series. So I'm thinking these should probably be excluded from the de-noising process. We don't have sufficient detail in the paper to know exactly what the researchers did originally.

While introducing unwanted fluctuations in the US Dollar Index:

So this is a brief excursion into some of the rest of the methodology outlined in the paper. First up, we need to produce data for training, validation and testing. I am doing this for just the first batch of data. We would then move the window forward + 3 months, rinse and repeat.

Note that

(1) The data is being standardized. If you dont do this the outputs from the autoencoders is mostly just 1s and 0s. Same happens if you use Min/Max scaling.

(2) We use the mean and standard deviation from the training dataset to normalize the test dataset. This is a trap that too many researchers fall into - standardizing the test dataset using the mean and standard deviation of the test dataset is feeding forward information.

In[33]:= startISIdx = 1;
endISIdx = -1 + 
      Position[wlDates, x_ /; x > DateObject[{2009, 12, 31}, "Day"]] //
      First // First // Quiet;
endISDate = wlDates[[endISIdx]];

In[134]:= startValIdx = endISIdx + 1;
startValDate = wlDates[[startValIdx]];
endValDate = DatePlus[startValDate, Quantity[3, "Months"]];
endValIdx = 
  Position[wlDates, x_ /; x >= endValDate] // First // First // 
   Quiet;
endValDate = wlDates[[endValIdx]];
startTestIdx = endValIdx + 1;
startTestDate = wlDates[[startTestIdx]];
endTestDate = DatePlus[startTestDate, Quantity[3, "Months"]];
endTestIdx = 
  Position[wlDates, x_ /; x >= endTestDate] // First // First // Quiet;
endTestDate = wlDates[[endTestIdx]];

In[147]:= 
inputData = Standardize[transformedData[[startISIdx ;; endISIdx]]];
valData = 
  Standardize[transformedData[[startValIdx ;; endValIdx]], isRange];
testData = transformedData[[startTestIdx ;; endTestIdx]];
isMean = Mean[transformedData[[startISIdx ;; endISIdx]]] ;
isMean = ArrayReshape[isMean, Dimensions@testData, isMean];
isStDev = 
  StandardDeviation[transformedData[[startISIdx ;; endISIdx]]] ;
isStDev = ArrayReshape[isStDev, Dimensions@testData, isStDev];
testData = 
  (transformedData[[startTestIdx ;; endTestIdx]) - isMean) / isStDev;

In[116]:= Dimensions /@ {inputData, valData}

Out[116]= {{503, 17}, {63, 17}}

Now we can produce the output from the autoencoder stack for both the training and test data:

encodedInputData = 
  TrainedEncoders[[4]][
   TrainedEncoders[[3]][
    TrainedEncoders[[2]][TrainedEncoders[[1]][inputData]]]];

encodedTestData = 
  TrainedEncoders[[4]][
   TrainedEncoders[[3]][
    TrainedEncoders[[2]][TrainedEncoders[[1]][testData]]]];

Now, admittedly, an argument can be made that a properly constructed LSTM model would outperform a simple gradient-boosted tree - but not by the amount that would be required to improve the prediction accuracy from around 50% to nearer 95%, the level claimed in the paper. At most I would expect to se a 1% to 5% improvement in forecast accuracy.

So what this suggests to me is that the researchers have got something wrong, allowing forward information to leak into the modeling process. The most likely culprits are:

1. Applying DWT transforms to the entire dataset, instead of the training and test sets individually
2. Standardzing the test dataset using the mean and standard deviation of the test dataset, instead of the training data set

So this is a brief excursion into some of the rest of the methodology outlined in the paper. First up, we need to produce data for training, validation and testing. I am doing this for just the first batch of data. We would then move the window forward + 3 months, rinse and repeat.

Note that

(1) The data is being standardized. If you dont do this the outputs from the autoencoders is mostly just 1s and 0s. Same happens if you use Min/Max scaling.

(2) We use the mean and standard deviation from the training dataset to normalize the test dataset. This is a trap that too many researchers fall into - standardizing the test dataset using the mean and standard deviation of the test dataset is feeding forward information.

In[33]:= startISIdx = 1;
endISIdx = -1 + 
      Position[wlDates, x_ /; x > DateObject[{2009, 12, 31}, "Day"]] //
      First // First // Quiet;
endISDate = wlDates[[endISIdx]];

In[134]:= startValIdx = endISIdx + 1;
startValDate = wlDates[[startValIdx]];
endValDate = DatePlus[startValDate, Quantity[3, "Months"]];
endValIdx = 
  Position[wlDates, x_ /; x >= endValDate] // First // First // 
   Quiet;
endValDate = wlDates[[endValIdx]];
startTestIdx = endValIdx + 1;
startTestDate = wlDates[[startTestIdx]];
endTestDate = DatePlus[startTestDate, Quantity[3, "Months"]];
endTestIdx = 
  Position[wlDates, x_ /; x >= endTestDate] // First // First // Quiet;
endTestDate = wlDates[[endTestIdx]];

In[147]:= 
inputData = Standardize[transformedData[[startISIdx ;; endISIdx]]];
valData = 
  Standardize[transformedData[[startValIdx ;; endValIdx]], isRange];
testData = transformedData[[startTestIdx ;; endTestIdx]];
isMean = Mean[transformedData[[startISIdx ;; endISIdx]]] ;
isMean = ArrayReshape[isMean, Dimensions@testData, isMean];
isStDev = 
  StandardDeviation[transformedData[[startISIdx ;; endISIdx]]] ;
isStDev = ArrayReshape[isStDev, Dimensions@testData, isStDev];
testData = 
  isMean + 
   isStDev*Standardize[transformedData[[startTestIdx ;; endTestIdx]]];

In[116]:= Dimensions /@ {inputData, valData}

Out[116]= {{503, 17}, {63, 17}}

Hi Jonathan After trying a number of approaches and getting input from more experienced friends I cannot get anywhere close to these results without code to accompany the paper. Anecdotal feedback is that there may be an element of forward looking "peeking" even before getting to partitioning etc. so moving on to other projects. Warren