Need help with Market Sentiment Analysis

romubay · June 24, 2019, 9:40pm

Dear Wolfram Users,
I am new to Mathematica and I wanted to replicate this study in order to learn how to do sentiment analysis on website data.

However, it seems to me, that the code is not working. Maybe it was an older Mathematica code?

For example:

archive = 
  Import[StringJoin["http://www.wsj.com/public/page/archive-", 
    DateString[
     datelist[[1]], {"Year", "-", "MonthShort", "-", "DayShort"}], 
    ".html"]];
archive = 
  StringDrop[archive, 
   StringPosition[archive, 
     DateString[
      datelist[[1]], {"MonthName", " ", "DayShort", ", ", 
       "Year"}]][[1, 2]]];
archive = 
  StringTake[
   archive, -1 + StringPosition[archive, "ARCHIVE FILTER"][[1, 1]]];
archivewords = ToLowerCase[DeleteStopwords[TextWords[archive]]];
TakeLargest[Counts[archivewords], 20]
WordCloud[archivewords]

This command should download news from the WSJ archive and format for Machine Learning purposes. But somehow it is not working. Can somebody help me to replicate this code in a new way?
I also think that one of the problems is that this code is not able to switch archive dated anymore.
Or maybe there is a newer version of the article?

Thank you very much.

rohitnamjoshi · June 26, 2019, 4:53am

Roman,

Looks like the URL for the archive as well as the format has changed. Try this

datelist = {DateObject[{2012, 1, 3}]};

archive = 
 Import[StringJoin["https://www.wsj.com/news/archive/", 
   DateString[datelist[[1]], {"Year", "Month", "Day"}]]];

archive = 
 StringDrop[archive, 
  StringPosition[archive, 
    DateString[
     datelist[[1]], {"MonthNameShort", " ", "DayShort", " ", 
      "Year"}]][[1, 2]]];

archive = 
 StringTake[
  archive, -1 + 
   StringPosition[archive, "Most Popular Articles"][[1, 1]]];

archivewords = ToLowerCase[DeleteStopwords[TextWords[archive]]];

TakeLargest[Counts[archivewords], 20]

<|“new” → 34, “2012” → 33, “markets” → 27, “business” → 24,
“year” → 22, “u.s.” → 21, “asia” → 18, “opinion” → 16, “2011” →
15, “said” → 12, “$” → 11, “york” → 10, “economic” → 10, “iowa”
→ 10, “news” → 10, “economy” → 10, “biggest” → 10, “day” → 9, “data” → 9, “december” → 9|>

WordCloud[archivewords]

romubay · June 26, 2019, 10:16pm

Thank you very much for your help, your code works perfectly!!!

If you have time, could you please explain, how did you understood in which format the date should be written.

And also, how exactly does this line work?

archive = 
 StringTake[
  archive, -1 + 
   StringPosition[archive, "Most Popular Articles"][[1, 1]]]

Does it somehow brings us into the “Most Popular Articles” section and gates titles from there?

rohitnamjoshi · June 27, 2019, 1:18am

could you please explain, how did you understood in which format the
date should be written.

Nothing fancy. Did a Google search for “wall street journal archives” and this showed up in the results. So the date format in the URL is YYYYMMDD which is {"Year", "Month", "Day"} in WL.

The page body has some text unrelated to the archive followed by “News Archive for Jun 25 2019”. So the actual archive data follows a date matching {"MonthNameShort", " ", "DayShort", " ", "Year"}.

Archive data is followed by “Most Popular Articles” and other text unrelated to the archive. So the actual archive data is between those two strings. The code extracts that portion from the page body.

jonathan3 · June 27, 2019, 9:35am

Hi Roman,

Glad you got it working. Rohit is right, the format changed (which happens quite frequently, with news sites, unfortunately).

Let me know if you run into other issues.

Jonathan

romubay · June 27, 2019, 9:52am

Thank you very much for your reply Jonathan,

I find your article very interesting and inspiring. I am not a proficient Wolfram user, unfortunately. However, I study finance now, and I would really like to learn how you did this sentiment analysis.

With the help of Rohit, I went through the first part of code, and now I have some troubles with the WSJSentimentIndicator:

WSJSentimentIndicator[date_] := 
 Module[{d = date, archive, archivewords, WSJSI}, 
  archive = 
   Import[StringJoin["http://www.wsj.com/public/page/archive-", 
     DateString[d, {"Year", "-", "MonthShort", "-", "DayShort"}], 
     ".html"]];
  archive = 
   StringDrop[archive, 
    StringPosition[archive, 
      DateString[d, {"MonthName", " ", "DayShort", ", ", "Year"}]][[1,
      2]]];
  archive = 
   StringTake[
    archive, -1 + StringPosition[archive, "ARCHIVE FILTER"][[1, 1]]];
  archivewords = ToLowerCase[DeleteStopwords[TextWords[archive]]];
  WSJSI = #Positive/(#Negative + #Positive) &@
     Counts[Classify["Sentiment", archivewords]] // N;
  {WSJSI, archivewords, archive}]

So, if we update the code, it should look like this:

WSJSentimentIndicator[date_ ] := 
 Module[{d = date, archive, archivewords, WSJSI}, 
archive = 
 Import[StringJoin["https://www.wsj.com/news/archive/", 
   DateString[d, {"Year", "Month", "Day"}]]];
archive = 
 StringDrop[archive, 
  StringPosition[archive, 
    DateString[
     d, {"MonthNameShort", " ", "DayShort", " ", 
      "Year"}]][[1, 2]]];
archive = 
 StringTake[
  archive, -1 + 
   StringPosition[archive, "Most Popular Articles"][[1, 1]]];
archivewords = ToLowerCase[DeleteStopwords[TextWords[archive]]];
 WSJSI = #Positive /(#Negative + #Positive) &@
     Counts[Classify["Sentiment", archivewords]] // N;
  {WSJSI, archivewords, archive}]

However, the code which returns us the histogram does not work for me:

WSJSI = Flatten[First@WSJSentimentIndicator[#]&/@datelist]
Histogram[tsWSJSI, PlotLabel -> Style["Histogram of WSJ Sentiment indicator",Bold]]

If you will have time, could you please help me solve this one? I have a feeling, that the WSJSI does not account for the ‘datelist’ correctly.

rohitnamjoshi · June 28, 2019, 1:29am

Hi Roman,

My bad. The second DateString format should be {"MonthNameShort", " ", "DayShort", ", ", "Year"}.

For eight days:

datelist = Table[DateObject[{2012, 1, n}], {n, 3, 10}];
WSJSI = Flatten[First@WSJSentimentIndicator[#] & /@ datelist]
tsWSJSI = TimeSeries[Transpose[{datelist, WSJSI}]]
Histogram[tsWSJSI, PlotLabel -> Style["Histogram of WSJ Sentiment indicator", Bold]]

romubay · June 30, 2019, 8:43pm

Thank you very much Rohit!

And the module code you did’t change?

It looks like that for you?

WSJSentimentIndicator[date_ ] := 
 Module[{d = date, archive, archivewords, WSJSI}, 
archive = 
 Import[StringJoin["https://www.wsj.com/news/archive/", 
   DateString[d, {"Year", "Month", "Day"}]]];
archive = 
 StringDrop[archive, 
  StringPosition[archive, 
    DateString[
     d, {"MonthNameShort", " ", "DayShort", ", ", 
      "Year"}]][[1, 2]]];
archive = 
 StringTake[
  archive, -1 + 
   StringPosition[archive, "Most Popular Articles"][[1, 1]]];
archivewords = ToLowerCase[DeleteStopwords[TextWords[archive]]];
 WSJSI = #Positive /(#Negative + #Positive) &@
     Counts[Classify["Sentiment", archivewords]] // N;
  {WSJSI, archivewords, archive}]

romubay · July 3, 2019, 12:24pm

Thank you very much Rohit!

And the module code you did’t change?

It looks like that for you?

WSJSentimentIndicator[date_ ] := 
 Module[{d = date, archive, archivewords, WSJSI}, 
archive = 
 Import[StringJoin["https://www.wsj.com/news/archive/", 
   DateString[d, {"Year", "Month", "Day"}]]];
archive = 
 StringDrop[archive, 
  StringPosition[archive, 
    DateString[
     d, {"MonthNameShort", " ", "DayShort", ", ", 
      "Year"}]][[1, 2]]];
archive = 
 StringTake[
  archive, -1 + 
   StringPosition[archive, "Most Popular Articles"][[1, 1]]];
archivewords = ToLowerCase[DeleteStopwords[TextWords[archive]]];
 WSJSI = #Positive /(#Negative + #Positive) &@
     Counts[Classify["Sentiment", archivewords]] // N;
  {WSJSI, archivewords, archive}]

rohitnamjoshi · July 3, 2019, 5:34pm

Hi Roman,

Sorry for the late response.

Yes, that is the code I used. I copied it from your post into a notebook, evaluated and tested with

WSJSentimentIndicator[DateObject[{2012, 1, 3}]]

The first time I evaluated, it failed because Classify returned an empty association. Subsequent evaluations worked fine. Not sure why - perhaps it has to retrieve some data from Wolfram servers and that timed out? If it happens frequently, the code can easily be modified to retry the Classify.

romubay · July 7, 2019, 6:29pm

Dear Dr Kinlay,
i will be writing a course paper in international finance soon and I would like to replicate your market sentiment analysis but within the different timeframe and also with the use of twitter/reddit articles data. May I ask you, whether it is possible for me to use your code and reference your article in my paper?
Thank you very much for your time.
Roman

romubay · July 7, 2019, 10:40pm

Thank you very much Rohit!
I now understand why I have problems with computation - Mathematica just gives me fractured output every time. For example:

{0.6781, #Positive/(#Negative + 
#Positive), #Positive/(#Negative + #Positive), 0.7890, 0.6905, #Positive/(#Negative + \
#Positive), #Positive/(#Negative + #Positive)}

I have to repeat calculation 5-6 times to get the full output. I just wanted to run the analysis based on the data of a couple of years but I barely can run it for 8 day (calculation takes around 5min and not always gives full output).
I think that the problem is either in my PC, even though I have i7 and 16 RAM, or maybe the code is too heavy.
Do you know, if it is possible to add some line of the code to

WSJSI = Flatten[First@WSJSentimentIndicator[#] & /@ datelist]
tsWSJSI = TimeSeries[Transpose[{datelist, WSJSI}]]
Histogram[tsWSJSI, PlotLabel -> Style["Histogram of WSJ Sentiment indicator", Bold]]

to make it repeat the calculation over and other until success?

Thank you very much for your time.

jonathan3 · July 8, 2019, 8:56am

Roman,
Yes sure, no problem.

rohitnamjoshi · July 9, 2019, 9:00pm

Hi Roman,

I took a closer look at why it fails. It is the Import that occasionally returns partial results. The return value has the preamble text, the header text for the archive e.g. “News Archive for Jun 25, 2019” followed by blank lines, followed by “Most Popular Articles”. The entire archive section is missing. It is not a bug in Import, I can reproduce it with a simple Ruby program.

require 'nokogiri'
require 'open-uri'

doc = Nokogiri::HTML(open("https://www.wsj.com/news/archive/20120105"))
txt = doc.xpath("//*[@id='root']/div/div/div/div[2]/div/div/div[2]/div[1]/div/div").text

puts txt

My guess is that the ads and other asynchronous JavaScript have to run before the archive section is rendered.

To work around this, use a helper function and retry the Import operation until it succeeds. A better way would be to use WebExecute if you are on version 12. You could experiment with that.

As far as performance, the Import is quite slow, taking ~15s to complete. With the workaround you can just increase maxRetries and leave it running overnight.

Workaround:

ClearAll[downloadArchiveWords, WSJSentimentIndicator]

downloadArchiveWords[date_] := 
  Module[{end = "Most Popular Articles", url, archive},
   url = StringJoin["https://www.wsj.com/news/archive/", DateString[date, {"Year", "Month", "Day"}]];
   archive = Import[url];
   archive = 
    StringDrop[archive, 
     StringPosition[archive, 
       DateString[date, {"MonthNameShort", " ", "DayShort", ", ", "Year"}]][[1, 2]]];
   archive = StringTake[archive, -1 + StringPosition[archive, end][[1, 1]]];
   {ToLowerCase[DeleteStopwords[TextWords[archive]]], archive}];

WSJSentimentIndicator[date_, maxRetries:_?Positive:5] := 
 Module[{retries, archivewords, archive, WSJSI},
  retries = 0;
  archivewords = "";
  While[Length@archivewords == 0 && retries < maxRetries,
   {archivewords, archive} =  downloadArchiveWords[date]; retries++; Pause[1]];
  WSJSI = #Positive/(#Negative + #Positive) &@Counts[Classify["Sentiment", archivewords]] // N;
  {WSJSI, archivewords, archive}];

romubay · July 9, 2019, 10:19pm

Thank you very much Rohit! I will try run the workaround solution tonight and I will check out the WebExecute!

romubay · July 17, 2019, 9:43pm

Dear Dr. Kinlay,
I am replicating the last part of your study now. If you will have time, could you please explain, what tsVTDSPX and tsVTDStrategy functions are?

period = QuantityMagnitude@
   DateDifference[First@datelist, Last@datelist, "Year"];
AnnStd = Sqrt[252]*
   StandardDeviation[
    Transpose@{tsSPXreturns["Values"], strategyreturns}];
cf = {tsVTDSPX[Last@datelist]/1000 - 1, 
   tsVTDstrategy[Last@datelist]/1000 - 1};
CAGR = -1 + (1 + cf)^(1/period);
IR = CAGR/AnnStd;

Print[Style["News Sentiment Strategy", "Subsection"]];
P1 = Style[
   NumberForm[
    TableForm[{CAGR, AnnStd, IR}, 
     TableHeadings -> {{"CAGR", "Ann. StDev.", 
        "IR"}, {Style["SP500 Index", Bold], 
        Style["Strategy", Bold]}}], {6, 2}], FontSize -> 14];
P2 = DateListPlot[{tsVTDSPX, tsVTDstrategy}, Filling -> Axis, 
   PlotLegends -> {"S&P500 Index", "Strategy"}, 
   PlotLabel -> Style["Value of $1,000", Bold], ImageSize -> Medium];
Print[P1];
Print[P2];

romubay · July 17, 2019, 10:02pm

Hello Rohit!
The code which you proposed worked fine! I tried it on the WSJ articles for the last 3 years and out of 779 values, I had output for 710, with 69 Null values, which is fine. The rest I calculated manually afterwards, because complex computations too much time.
I am now going through the last part of the paper - construction of the Trading Algorithm. Do you have any idea, what stands for tsVTDSPX and tsStrategy?

period = QuantityMagnitude@
   DateDifference[First@datelist, Last@datelist, "Year"];
AnnStd = Sqrt[252]*
   StandardDeviation[
    Transpose@{tsSPXreturns["Values"], strategyreturns}];
cf = {tsVTDSPX[Last@datelist]/1000 - 1, 
   tsVTDstrategy[Last@datelist]/1000 - 1};
CAGR = -1 + (1 + cf)^(1/period);
IR = CAGR/AnnStd;

Print[Style["News Sentiment Strategy", "Subsection"]];
P1 = Style[
   NumberForm[
    TableForm[{CAGR, AnnStd, IR}, 
     TableHeadings -> {{"CAGR", "Ann. StDev.", 
        "IR"}, {Style["SP500 Index", Bold], 
        Style["Strategy", Bold]}}], {6, 2}], FontSize -> 14];
P2 = DateListPlot[{tsVTDSPX, tsVTDstrategy}, Filling -> Axis, 
   PlotLegends -> {"S&P500 Index", "Strategy"}, 
   PlotLabel -> Style["Value of $1,000", Bold], ImageSize -> Medium];
Print[P1];
Print[P2];

In the next computation, Dr.Kinlay defines tsStrategy as tsSPXReturns, however, as for tsVTDSPX I have no idea.

rohitnamjoshi · July 18, 2019, 4:12pm

Hi Roman,

I am sorry, I have no idea what tsVTDSPX and tsVTDstrategy are. I could not find their definitions in Jonathan’s original post. You will have to wait for a response from him.

Rohit

jonathan3 · July 18, 2019, 4:57pm

Hi Rohit,

VTD stands for “Value of $1,000”

So VTDSPX is the compounded value of $1,000 if “invested” in the S&P 500 index.
VTDStrategy is the compounded value of $1,000 is invested in the strategy.

ts(anything) stands for “time series”. So tsVTDStrategy is VTDStrategy formulated as a time series.

Hope this helps.

jonathan3 · July 18, 2019, 5:00pm

So VTDStrategy and VTDSPX are not functions - they’re variables whose value is set using FoldList (as I recall).

Topic		Replies	Views
Market Sentiment Analysis Staff Picks data-science , finance , social-sciences , statistics-and-probability , machine-learning , staff-pick , computational-humanities , artificial-intelligence , societies-and-civilizations	3	37587	September 21, 2017
Real time market monitor with notebook using Schwab API Staff Picks data-science , economics , finance , external-interfaces , graphics-and-visualization , interface-construction , staff-pick , artificial-intelligence	3	7239	May 7, 2025
Impact of Reddit posts on Stock Market Price? Questions data-science , finance , curated-data , external-interfaces , interface-construction , machine-learning	1	11730	June 11, 2019
[WSC21] Analyzing COVID-19 vaccine sentiment over time Staff Picks data-science , medical-sciences , social-sciences , machine-learning , staff-pick , wolfram-high-school-summer-research-program , artificial-intelligence , living-systems , societies-and-civilizations	2	17186	August 25, 2021
Create a Correlation Table with financial data Questions finance	3	6875	January 17, 2026

Need help with Market Sentiment Analysis

Related topics