Per,

Thanks for you post. I suppose that large excursions from equilibrium are a risk with mean-reversion strategies. (The underlying statistics are not strictly a normal distribution, and so "fat tails" imply that large excursions occur more frequently than one might expect.)

In reply to your points:

1. I'm using the same ETF triplet for my trading because it has a high Johansen score (>> 99%), a modest half-life for mean-reversion, and the three ETFs are very liquid (which is also very important!). I can imagine that trading multiple portfolios (or perhaps larger portfolios of more than three ETFs) would likely reduce risk, but it would also increase transaction costs. My funds are limited (< $100,000), so I haven't pursued that option.

2. As to arbitrage between an ETF and its components, I would imagine that there would be only limited arbitrage opportunities (because the ETFs track their components pretty closely) which would limit profits, as you suggest. However, I would certainly expect high cointegration. It's just the small excursions from equilibrium that would limit profitability.

3. Combining strategies is probably a good idea. I hear that that's what the large quant hedge funds do. They have multiple quants pursuing different strategies, and when one strategy is not working, others are. I actually have a trend-following algorithm that I've been using with cryptocurrencies over the past seven months, so I suppose I am "combining strategies" -- even though my total investment in cryptocurrencies is small ($5,000). Unfortunately, the cryptocurrencies had an horrendous sell-off last year. Nevertheless, my algo limited my maximum drawdown to around 20% by mostly keeping me out of the market. I'm hoping that the period of relative stability in the cryptocurrencies in the past few months is a prelude to stronger prices. I'm actually starting to make a small amount of money in the cryptos.

I agree with your comments about Chan. I'm grateful that he's illuminated the basic concepts and strategies. His book inspired me to study how to best implement the Kalman filter when trading a cointegrated portfolio, which I decided to share with others. If you have difficulty implementing the Kalman filter strategy, let me know. I can help with explanations, but I won't post my Kalman filter code on a public forum. I put too much effort into that to just give it away. I'm sure you understand.

Amanda

Amanda - thanks for taking your time to write such elaborate answer. To me this is extremely interesting and I had a similar experience beginning of this year in one of my cointegration baskets with European stock index futures, where the basket wandered off on a really long adverse excursion before eventually reverting at a loss. I realized that this became a quite lengthy post so I apologize for that in advance.

There are a few core concepts associated with this type of trading that I'm constantly working on in addition to refining the mathematical procedure of constructing a stationary portfolio. It would be interesting to here your view on these as well:

1. Selecting the ETFs? Are you using the same ETFs or do you continuously screen for new combinations with potentially better cointegration statistics? I have relied on a basket with the same set of stock index futures, reasoning that the European economies are fundamentally interlinked at some level and indeed this can be validated statistically for extended periods. But not always…and there's the problem.

2. Arbitrage between the ETF and its constituents? This is also briefly described by Chan, but of course any practical implementation comes with a heap of issues not covered in the book. I alluded to this in my first post and I think it is a quite interesting approach. The point here is that the ETF is perfectly cointegrated with its portfolio of weighted constituents by construction and not by a hidden set of underlying factors et c. The task and the challenge here is to find a subset of constituents with high enough cointegration properties in combination with sufficient variance to overcome the transaction cost. I'm exploring this approach again for stock indices and their constituents, where I periodically reconstruct the constituent subset basket. I would imagine it to be quite straight forward to apply your existing model to this approach as well?

3. Combining strategies? Wether they are statistical flukes or cointegration breakdowns, one of the cures for painful or even catastrophic drawdowns is to maintain a portfolio of different strategies with limited correlation. I tend to focus more on methods to construct a portfolio of cointegratin baskets than going all in on one or a few of them. What are your thoughts in this?

Regarding Chan's book, I think the depth of your research is on a completely different level. You are rebuilding the concepts from scratch, finding and solving issues not even mentioned in the book. There are clearly shortcuts and maybe even mistakes in the book, but in all fairness I think Ernie's doing a great job explaining the basic concepts and setting the scene for further investigation.

Your post on StackExchange is what led me to the Wolfram site in the first place. I'm already working on incorporating your procedure in Python.

Hi Per. I got an email today notifying me of your post to this discussion. I've been trading this algorithm during this entire time, as well as having a lot of email exchanges with Jonathan Kinlay regarding how to implement both the Johansen code and the Kalman filter in a manner similar to Chan. The algorithm was working well for me until about 2 months ago when one of the components of my triplet started rising in price in a way that appears to violate the cointegration. The result was that I waited 9 weeks for mean reversion. The portfolio finally mean-reverted, but by that time I had taken a significant hit on my profits. As a result my total trading profit over this time is much smaller than it was before this event -- around 10% or so over the past 9 months.

The past 2 months are clearly an outlier as compared to past algorithm performance. I can tell that simply by looking at the variation of the z-score over the past 10 years. The average half-life for mean-reversion of the z-score was 1.17 weeks. On a few occasions it took as long as 4 or 5 weeks to mean revert. (Remember, I don't make a profit until the portfolio value -- and thus the z-score -- mean-reverts.) Over the past 2 months or so, it took 9 weeks for my portfolio to mean revert. Is the cointegration breaking down, or was this just a one-time statistical fluke?

The problem is that when I re-calculate the Kalman filter parameters each weekend, the z-score often shifts in such a way that losses can accumulate if the mean reversion is delayed for too long. For example, on the close Friday, I may show a z-score of around 1.0. Over the weekend I re-calculate the Johansen test + Kalman filter parameters, and then when I run the algo on Monday morning, the z-score is significantly lower (in the range 0.5 - 0.8), even if the market is essentially unchanged. Thus, even if mean reversion occurs that day, I don't make as much profit as I'd hoped. This isn't a big deal if the portfolio mean reverts within 3 weeks -- I still make a profit. But if mean reversion doesn't happen for more than 3 or 4 weeks, I may end up with a loss. Waiting 9 weeks for mean reversion accumulated a 12% loss, which was more than half my profit over the past 9 months.

I'm still trading with this algorithm, but I'm waiting for a higher z-score before I "pull the trigger" on my trades, in order to increase the probability of a quicker mean-reversion. This means I may miss some trades, but that's OK. Also, I scale in my buys. (Partial buy at, say, z-score = 1.0, and another partial buy at z-score = 1.5, etc.)

If you're interested in how I implement the Kalman filter -- which is significantly different than Chan -- I wrote a detailed post on StackExchange - Quantitative Finance:

Does Chan use the wrong state transition model in his Kalman filter code?

Using a careful analysis, I argued in my post that Chan uses the wrong state transition matrix, i.e., the identity matrix, in his Kalman filter. I showed how to calculate the correct state transition matrix for a cointegrated portfolio, as well as how to initialize the Kalman filter using an adaptive tuning method. I received positive feedback from the readers of the forum, and one reader emailed Ernie Chan. This precipitated an email exchange between Ernie and I. Ernie basically said that his treatment was meant to be more general and so he didn't assume cointegration. I didn't want to argue, so I let it go. (My method actually works when there's no cointegration -- you basically get the identity matrix solution that Chan uses in that case.) Chan replied that "a stationary [cointegrated] portfolio can be more profitable", and that "your analysis may be correct". I didn't press him any further. I know what I did is correct, and none of the readers of my StackExchange post criticized my analysis. I received a fair number of up-votes.

I hope this is helpful.

You're correct. I was thinking about the dynamic weights from the Kalman filter. However, when we use the static weights from the Johansen test, we lose the stationarity for out-of-sample data. So, for example, when I apply the unit root test to my weighted portfolio, using the Johansen (static) weights, I get:

In-sample data length = 289, Johansen weights
p = 0.0718

However, when I calculate the Johansen coefficients using only the first 189 data points, and then look at unit root test, I get:

In-sample data length = 189, Johansen weights
p = 0.109
Out-of-sample data length = 100, Johansen weights
p = 0.587

Clearly, the out-of-sample period cannot be considered stationary. The situation is not helped by going to a larger in-sample (smaller out-of-sample) period, as you point out.

Now, however, let's look at the same situation except using the dynamic weights from the Kalman filter. For the full sample length:

In-sample data length = 289, Kalman weights
p = 5.5 x 10^-11

Much higher confidence of stationarity! Now, for in-sample/out-of-sample:

In-sample data length = 189, Kalman weights
p = 1.76 x 10^-9
Out-of-sample data length = 100, Kalman weights
p = 3.722 x 10^-7

Still, very good. However, it may be argued that I'm cheating here because I used the entire array of data to calculate the Kalman filter parameters (transition matrix, noise variance/covariances, initial state covariance). So I re-calculated the in-sample/out-of-sample weights using only in-sample data to calculate these parameters:

In-sample data length = 189, Kalman weights, revised Kalman parameter calculations
p = 0.000211

Out-of-sample data length = 100, Kalman weights, revised Kalman parameter calculations
p = 0.0871

The out-of-sample p-value for the unit root test is not as good, but still what I would consider stationary. Furthermore, let's look at a smaller out-of-sample (larger in-sample) period:

In-sample data length = 239, Kalman weights, revised Kalman parameter calculations
p = 7.1 x 10^-10
Out-of-sample data length = 50, Kalman weights, revised Kalman parameter calculations
p = 0.0000548

Using the Kalman filter weights, the stationarity of the out-of-sample period appears to be dependent on the size of the in-sample/out-of-sample periods. A shorter out-of-sample period gives a much smaller p-value for the unit root test. Now, considering that I update my Kalman filter parameters once per week, my out-of-sample period is only 1 time step. Therefore, the loss in stationarity should be very small.

So I thought it might be useful to work through an example, to try to make the mechanics clear. I'll try to do this is stages so that others can jump in along the way, if they want to.

Start with some weekly data for an ETF triplet analyzed in Ernie Chan's book:

`tickers = {"EWA", "EWC", "IGE"};
period = "Week";
nperiods = 52*15;
finaldate = DatePlus[Today, {-1, "BusinessDay"}];`

After downloading the weekly close prices for the three ETFs we divide the data into 14 years of in-sample data and 1 year out of sample:

  stockdata = 
          FinancialData[#, 
             "Close", {DatePlus[finaldate, {-nperiods, period}], finaldate, 
              period}, "DateList"] & /@ tickers;
    stockprices = stockdata[[All, All, 2]];
    isprices = Take[stockprices, All, 52*14];
    osprices = Drop[stockprices, 0, 52*14];

We then apply Amanda's JohansenTest function:

JT = JohansenTest[isprices, 2, 1]

We find evidence of up to three cointegrating vectors at the 95% confidence level:

Let's take a look at the vector coefficients (laid out in rows, in Amanda's function):

We now calculate the in-sample and out-of-sample portfolio values using the first cointegrating vector:

isportprice = (JT[[2, 1]]*100).isprices;
osportprice = (JT[[2, 1]]*100).osprices;

The portfolio does indeed appear to be stationary, in-sample, and this is confirmed by the unit root test, which rejects the null hypothesis of a unit root:

ListLinePlot[isportprice]

UnitRootTest[isportprice]

0.000232746

Unfortunately (and this is typically the case) the same is not true for the out of sample period:

ListLinePlot[osportprice]

UnitRootTest[osportprice]    

0.121912

We fail to reject the null hypothesis of unit root in the portfolio process, out of sample.

I'll press pause here before we go on to the next stage, which is Kalman Filtering.

Amanda, I think you may have hit on something very important. As you point out, the determination of the variance/covariances is critical and the adaptive tuning procedure you recommend appears very successful in stabilizing the portfolio, making it suitable for a stat-arb strategy.

As you saw, I did not use MMA in my own implementation because I felt that Wolfram's approach was somewhat unsympathetic to the needs of the economic researcher (vs. say the requirements of an engineer), compared to the available alternatives. I see that I am not entirely alone in that assessment: here, for instance. So I am delighted that you have successfully implemented this in MMA, presumably using KalmanEstimator(?). Or did you build the model from scratch?

I will run a few tests on your Johansen code and attempt to build a KF model in MMA using some of the ETF pairs/triplets Ernie discusses in his book and compare the results.

Meanwhile, I wondered if you could comment on the following:

1) While the initial trading performance appear very encouraging, what kind of performance results did the backtest produce, out of sample?

2) You mention that you update the model using weekly data and then trade it intraday during the following week. So presumably you are getting real-time market data into MMA somehow: via the Finance Platform, perhaps? And do you trade the signals via that platform, or some other way (manually)?

3) One extension that i found quite useful in my own research was to fit a GARCH model to the residuals and use this to determine the trade entry/exit points. But that procedure was probably only useful because of the nonstationarity in the portfolio returns process. If you have succeeded in dealing with that key issue at a more fundamental level, a GARCH extension is probably superfluous.

Hi Amanda,

Your approach seems very promising.

On point 2: I made the assumption that you had to be getting (quasi) real-time data into MMA somehow and indeed this turns to be the case - a creative solution to the problem.

Of course, since you are only updating the model weekly you wouldn't need to use MMA at all during the week. Some trading platforms will allow you to place bids and offers for a synthetic contract according to a simple formula, where the betas are fixed (for the week). In other cases a simple api interface is provided to something like Excel. That would enable you to recalculate the entry/exit prices automatically tick-by-tick, if you wanted to, and would also eliminate the need for manual trading as the orders could be fired into the trading platform via the api.

There are the usual practical considerations that apply to any stat arb strategy. For instance, do you try to enter passively, posting orders on the bid and ask prices of the portfolio (treating it as a single synthetic security)? Another approach is to post resting orders for the individual ETF components at appropriate price levels then cross the spread on the other ETFs if you get filled on one of them. These execution strategies tend to apply more in the case of pairs trading. For more complex strategies involving multiple securities like yours they can be very tricky to implement and traders typically cross the spread on entry and exit, which is what is you are doing, I would guess.

Another question is how to treat open positions held over a w/e when models get updated. The original exit points will likely change. So you have some options there too: exit all positions by the end of the week; maintain the original exit prices (profit target and stop loss); or recalculate exit prices for existing positions once the models get updated.

Finally, one other important issue is whether to use prices or (log) returns in your cointegration model. I suspect you are using the former, as I did in my toy illustration. But the resulting portfolios are rarely dollar neutral and hence consume margin capital. On the other hand, if you use returns and create a dollar-neutral portfolio, rebalancing becomes more of an issue. In that case I suspect you would want to rebalance the portfolio at least once a day, or according to some more sophisticated rebalancing algorithm.

Nice work, Amanda.

Hopefully Wolfram will include more of these standard statistical tests in futures releases, to bring MMA to parity with comparable math/stats software packages.

I have written a few posts about using the Kalman filter approach in pairs trading (stat arb), for instance here and here.

I would certainly be interested to get your own take on the subject and any trading results you might care to share.