Hi Rohit, so, according to Dr.Kinlay's answer, i assume, that code for tsVTDSPX will be the same as for tsVTDStrategy, just without quantiles:

strategyreturns = tsSPXReturns["Values"];
strategyreturns[[bottompercentile]] = (1/2)*
   strategyreturns[[bottompercentile]];
strategyreturns[[toppercentile]] = 2*strategyreturns[[toppercentile]];
tsVTDstrategy = 
 TimeSeries[
  Transpose[{datelist, 1000*FoldList[Times, 1, 1 + strategyreturns]}]]
marketreturns = tsSPXReturns["Values"];
tsVTDSPX = 
 TimeSeries[
  Transpose[{datelist, 1000*FoldList[Times, 1, 1 + marketreturns]}]]

With the leverage factor 2. However it is not working for me - i get transpose error.

The quantiles are found as follows:

percentiles = Quantile[tsSIchange, {1/3, 2/3}];
bottompercentile = 
  Flatten[Position[tsSIchange["Values"], 
    x_ /; x < percentiles[[1]]]];
toppercentile = 
  Flatten[Position[tsSIchange["Values"], x_ /; x > percentiles[[2]]]]

So we set the strategy returns equal to the S&P 500 Index returns, except in the bottom 1/3 quantile of the sentiment indicator, where we reduce leverage by 1/leveragefactor, and in the top 1/3 quantile of the sentiment indicator, where we increase leverage and returns by multiplying by the leverage factor.

What we are saying is: we will only adjust our strategy away from the market portfolio when the sentiment indicator is in the bottom 1/3 or top 1/3 quantile. In those periods we halve or double our exposure, respectively.

Roman, without going through all the code I don't recall specifically the reason for using Rest@datelist. But I would imagine it's because the length of the returns vector is one less than the number of dates (because you are differencing the log price series).

increasing leverage by 2x means that you are borrowing 100% of your capital, as investing 2x the amount of capital, as in Regulation T (so $1,000 becomes $2,000). So you earn twice the return.

Decreasing leverage by 1/2 means that you deploy only 50% of your capital in that period, ( $1,000 becomes $500), so earning one half of the return in that period.

Put simply, the idea is to push more chips onto the table when market sentiment is very positive and reduce your exposure when its very negative.

Hi Rohit,

VTD stands for "Value of $1,000"

So VTDSPX is the compounded value of $1,000 if "invested" in the S&P 500 index. VTDStrategy is the compounded value of $1,000 is invested in the strategy.

ts(anything) stands for "time series". So tsVTDStrategy is VTDStrategy formulated as a time series.

Hope this helps.

Dear Dr. Kinlay, I am replicating the last part of your study now. If you will have time, could you please explain, what tsVTDSPX and tsVTDStrategy functions are?

period = QuantityMagnitude@
   DateDifference[First@datelist, Last@datelist, "Year"];
AnnStd = Sqrt[252]*
   StandardDeviation[
    Transpose@{tsSPXreturns["Values"], strategyreturns}];
cf = {tsVTDSPX[Last@datelist]/1000 - 1, 
   tsVTDstrategy[Last@datelist]/1000 - 1};
CAGR = -1 + (1 + cf)^(1/period);
IR = CAGR/AnnStd;

Print[Style["News Sentiment Strategy", "Subsection"]];
P1 = Style[
   NumberForm[
    TableForm[{CAGR, AnnStd, IR}, 
     TableHeadings -> {{"CAGR", "Ann. StDev.", 
        "IR"}, {Style["SP500 Index", Bold], 
        Style["Strategy", Bold]}}], {6, 2}], FontSize -> 14];
P2 = DateListPlot[{tsVTDSPX, tsVTDstrategy}, Filling -> Axis, 
   PlotLegends -> {"S&P500 Index", "Strategy"}, 
   PlotLabel -> Style["Value of $1,000", Bold], ImageSize -> Medium];
Print[P1];
Print[P2];

Hi Roman,

Glad you got it working. Rohit is right, the format changed (which happens quite frequently, with news sites, unfortunately).

Let me know if you run into other issues.

Jonathan

Hi Roman,

My bad. The second DateString format should be {"MonthNameShort", " ", "DayShort", ", ", "Year"}.

For eight days:

datelist = Table[DateObject[{2012, 1, n}], {n, 3, 10}];
WSJSI = Flatten[First@WSJSentimentIndicator[#] & /@ datelist]
tsWSJSI = TimeSeries[Transpose[{datelist, WSJSI}]]
Histogram[tsWSJSI, PlotLabel -> Style["Histogram of WSJ Sentiment indicator", Bold]]

Hi Roman,

Sorry for the late response.

Yes, that is the code I used. I copied it from your post into a notebook, evaluated and tested with

WSJSentimentIndicator[DateObject[{2012, 1, 3}]]

The first time I evaluated, it failed because Classify returned an empty association. Subsequent evaluations worked fine. Not sure why - perhaps it has to retrieve some data from Wolfram servers and that timed out? If it happens frequently, the code can easily be modified to retry the Classify.

Hi Roman,

I took a closer look at why it fails. It is the Import that occasionally returns partial results. The return value has the preamble text, the header text for the archive e.g. "News Archive for Jun 25, 2019" followed by blank lines, followed by "Most Popular Articles". The entire archive section is missing. It is not a bug in Import, I can reproduce it with a simple Ruby program.

require 'nokogiri'
require 'open-uri'

doc = Nokogiri::HTML(open("https://www.wsj.com/news/archive/20120105"))
txt = doc.xpath("//*[@id='root']/div/div/div/div[2]/div/div/div[2]/div[1]/div/div").text

puts txt

My guess is that the ads and other asynchronous JavaScript have to run before the archive section is rendered.

To work around this, use a helper function and retry the Import operation until it succeeds. A better way would be to use WebExecute if you are on version 12. You could experiment with that.

As far as performance, the Import is quite slow, taking ~15s to complete. With the workaround you can just increase maxRetries and leave it running overnight.

Workaround:

ClearAll[downloadArchiveWords, WSJSentimentIndicator]

downloadArchiveWords[date_] := 
  Module[{end = "Most Popular Articles", url, archive},
   url = StringJoin["https://www.wsj.com/news/archive/", DateString[date, {"Year", "Month", "Day"}]];
   archive = Import[url];
   archive = 
    StringDrop[archive, 
     StringPosition[archive, 
       DateString[date, {"MonthNameShort", " ", "DayShort", ", ", "Year"}]][[1, 2]]];
   archive = StringTake[archive, -1 + StringPosition[archive, end][[1, 1]]];
   {ToLowerCase[DeleteStopwords[TextWords[archive]]], archive}];

WSJSentimentIndicator[date_, maxRetries:_?Positive:5] := 
 Module[{retries, archivewords, archive, WSJSI},
  retries = 0;
  archivewords = "";
  While[Length@archivewords == 0 && retries < maxRetries,
   {archivewords, archive} =  downloadArchiveWords[date]; retries++; Pause[1]];
  WSJSI = #Positive/(#Negative + #Positive) &@Counts[Classify["Sentiment", archivewords]] // N;
  {WSJSI, archivewords, archive}];

Hello Rohit! The code which you proposed worked fine! I tried it on the WSJ articles for the last 3 years and out of 779 values, I had output for 710, with 69 Null values, which is fine. The rest I calculated manually afterwards, because complex computations too much time. I am now going through the last part of the paper - construction of the Trading Algorithm. Do you have any idea, what stands for tsVTDSPX and tsStrategy?

period = QuantityMagnitude@
   DateDifference[First@datelist, Last@datelist, "Year"];
AnnStd = Sqrt[252]*
   StandardDeviation[
    Transpose@{tsSPXreturns["Values"], strategyreturns}];
cf = {tsVTDSPX[Last@datelist]/1000 - 1, 
   tsVTDstrategy[Last@datelist]/1000 - 1};
CAGR = -1 + (1 + cf)^(1/period);
IR = CAGR/AnnStd;

Print[Style["News Sentiment Strategy", "Subsection"]];
P1 = Style[
   NumberForm[
    TableForm[{CAGR, AnnStd, IR}, 
     TableHeadings -> {{"CAGR", "Ann. StDev.", 
        "IR"}, {Style["SP500 Index", Bold], 
        Style["Strategy", Bold]}}], {6, 2}], FontSize -> 14];
P2 = DateListPlot[{tsVTDSPX, tsVTDstrategy}, Filling -> Axis, 
   PlotLegends -> {"S&P500 Index", "Strategy"}, 
   PlotLabel -> Style["Value of $1,000", Bold], ImageSize -> Medium];
Print[P1];
Print[P2];

In the next computation, Dr.Kinlay defines tsStrategy as tsSPXReturns, however, as for tsVTDSPX I have no idea.

Dear Dr Kinlay, i will be writing a course paper in international finance soon and I would like to replicate your market sentiment analysis but within the different timeframe and also with the use of twitter/reddit articles data. May I ask you, whether it is possible for me to use your code and reference your article in my paper? Thank you very much for your time. Roman

could you please explain, how did you understood in which format the date should be written.

Nothing fancy. Did a Google search for "wall street journal archives" and this showed up in the results. So the date format in the URL is YYYYMMDD which is {"Year", "Month", "Day"} in WL.

The page body has some text unrelated to the archive followed by "News Archive for Jun 25 2019". So the actual archive data follows a date matching {"MonthNameShort", " ", "DayShort", " ", "Year"}.

Archive data is followed by "Most Popular Articles" and other text unrelated to the archive. So the actual archive data is between those two strings. The code extracts that portion from the page body.

Thank you very much for your help, your code works perfectly!!!

If you have time, could you please explain, how did you understood in which format the date should be written.

And also, how exactly does this line work?

archive = 
 StringTake[
  archive, -1 + 
   StringPosition[archive, "Most Popular Articles"][[1, 1]]]

Does it somehow brings us into the "Most Popular Articles" section and gates titles from there?