You're correct. I was thinking about the dynamic weights from the Kalman filter. However, when we use the static weights from the Johansen test, we lose the stationarity for out-of-sample data. So, for example, when I apply the unit root test to my weighted portfolio, using the Johansen (static) weights, I get:
In-sample data length = 289, Johansen weights
p = 0.0718
However, when I calculate the Johansen coefficients using only the first 189 data points, and then look at unit root test, I get:
In-sample data length = 189, Johansen weights
p = 0.109
Out-of-sample data length = 100, Johansen weights
p = 0.587
Clearly, the out-of-sample period cannot be considered stationary. The situation is not helped by going to a larger in-sample (smaller out-of-sample) period, as you point out.
Now, however, let's look at the same situation except using the dynamic weights from the Kalman filter. For the full sample length:
In-sample data length = 289, Kalman weights
p = 5.5 x 10^-11
Much higher confidence of stationarity! Now, for in-sample/out-of-sample:
In-sample data length = 189, Kalman weights
p = 1.76 x 10^-9
Out-of-sample data length = 100, Kalman weights
p = 3.722 x 10^-7
Still, very good. However, it may be argued that I'm cheating here because I used the entire array of data to calculate the Kalman filter parameters (transition matrix, noise variance/covariances, initial state covariance). So I re-calculated the in-sample/out-of-sample weights using only in-sample data to calculate these parameters:
In-sample data length = 189, Kalman weights, revised Kalman parameter calculations
p = 0.000211
Out-of-sample data length = 100, Kalman weights, revised Kalman parameter calculations
p = 0.0871
The out-of-sample p-value for the unit root test is not as good, but still what I would consider stationary. Furthermore, let's look at a smaller out-of-sample (larger in-sample) period:
In-sample data length = 239, Kalman weights, revised Kalman parameter calculations
p = 7.1 x 10^-10
Out-of-sample data length = 50, Kalman weights, revised Kalman parameter calculations
p = 0.0000548
Using the Kalman filter weights, the stationarity of the out-of-sample period appears to be dependent on the size of the in-sample/out-of-sample periods. A shorter out-of-sample period gives a much smaller p-value for the unit root test. Now, considering that I update my Kalman filter parameters once per week, my out-of-sample period is only 1 time step. Therefore, the loss in stationarity should be very small.