Group Abstract

Message Boards

FinBERT and FinRoBERTa models for financial sentiment analysis

Posted 2 years ago

CITE THIS NOTEBOOK: FinBERT and FinRoBERTa models for financial sentiment analysis by Jürgen Kanz. Wolfram Community JAN 18 2023.

Introduction

FinBert is an open source Natural Language Processing (NLP) model that has been trained on Financial data and outperforms almost all other NLP techniques for financial sentiment analysis.

Financial data sentiment analysis is more complicated than normal use cases. Let’s take a look at an example:

“Although the global market is experiencing a downturn, food companies are doing well thanks to Covid.”

Normal NLP techniques will not be able to determine whether the above sentence is good news or has a positive sentiment for food companies. This is because normal NLP techniques, like word2vec, look at each word separately and do not provide context for the words. In order to understand the sentiment of the above sentence, we would need to be aware of the context.

Context awareness is enabled by BERT for sentences. BERT is the acronym for Bidirectional Encoder Representation from Transformer. It is one of the most popular state-of-the-art text embedding models published by Google.

Domain Specific Fine-Tuning of BERT

Many domain specific models have emerged using BERT as the base and are being used for NLP tasks. Some of them are: FinBERT for Finance, BioBERT for Biomedical, VideoBERT for Video captioning categorization, ClinicalBERT for hospitals, and many more continue to evolve.

BERT, RoBERTa and the Wolfram Neural Net Repository

BERT is available in the Wolfram Neural Net Repository [1]. Furthermore, Wolfram provides domain specific fine-tuned versions of BERT like BioBERT, DistilBERT, Pre-trained Distill BERT, and SciBERT [2].
FinBERT is currently not available as a Wolfram Resource.

“The better BERT” is RoBERTa. RoBERTa is a robustly optimized BERT pretraining approach [3]. While BERT is trained on BookCorpus and Wikipedia Data, it is RoBERTa that is already trained on BookCorpus, English Wikipedia, CC-News, OpenWebText and Stories Datasets.

Consequently, the author decided to train BERT and RoBERTa on the enhanced Financial Phrase-bank.

Datasets

Financial PhraseBank

For training of the networks, the sentiment analysis dataset used is the Financial Phrase-bank [4] from Malo et al. (2014) [5]. The “Financial Phrase-bank” consists of, 4845 English sentences selected randomly from financial news found on the LexisNexis database. These sentences then were annotated by 16 people with background in finance and business. The annotators were asked to give labels according to how they think the information in the sentence might affect the mentioned company stock price. The dataset also includes information regarding the agreement levels on sentences among annotators. The distribution of agreement levels and sentiment labels can be seen on the following table.

Table 1: Distribution of sentiment labels and agreement levels in Financial Phrase-bank

FiQA - Financial Opinion Mining and Question Answering

On top, 997 selected sentences from the “FiQA - Financial Opinion Mining and Question Answering” challenge [6] have been used for training purposes.

Validation Measurements

On the repository webpage [10] in section “Train a classifier model with the subword embeddings”, we can find information about Validation Measurements for different net models.The following results are based on calculations with “BERT Trained on BookCorpus and Wikipedia Data”, “GloVe 300 - Dimensional Word Vectors Trained on Wikipedia and Gigaword 5 Data”, and finally “RoBERTa Trained on BookCorpus, English Wikipedia, CC-News, OpenWebText and Stories Datasets”. As input the FinancialPhrase-bank was used.

It becomes clear, that the given BERT net model does not deliver the best results regarding Loss and ErrorRate. Nevertheless, BERT is more or less the current standard transformer, which can be observed in the Python [7] [8] and Matlab community [9].The best results are provided by RoBERTa regarding Loss and Error Rate. Therefore, I decided to train both BERT versions simultaneously.

Training

The training process itself is not part of this notebook. Nevertheless, some general remarks:

◼

The Wolfram resource “BERT Trained on BookCorpus and Wikipedia Data” [10] is used to create FinBERT.

◼

To become FinRoBERTa, the Wolfram resource “RoBERTa Trained on BookCorpus, English Wikipedia, CC-News, OpenWebText and Stories Datasets”[11] is used.

◼

A balanced training set has been used. Which means that there are the same number of positive, neutral, and negative sentiments applied.

◼

The training has been completed with sentences with and without sentence punctuation.

Load FinBERT and FinRoBERTa

Import FinBERT and FinRoBERTa from the WolframCloud

In[]:=

finbert=Import["https://www.wolframcloud.com/obj/juergen.kanz/FinBERT.WLNET"]

Out[]=

NetChain



Input port:	string
Output port:	class



In[]:=

finroberta=Import["https://www.wolframcloud.com/obj/juergen.kanz/FinRoBERTa.WLNET"]

Out[]=

NetChain



Input port:	string
Output port:	class



Import local copies of FinBERT and FinRoBERTa

Let’s assume that this notebook, as well as FinBERT and FinRoBERTa are stored in the same directory.

In[]:=

SetDirectory[NotebookDirectory[]]

Out[]=

e:\Users\juerg\Documents\Mathematica\NLP

In[]:=

finbert=Import["FinBERT.WLNET"]

Out[]=

NetChain



Input port:	string
Output port:	class



In[]:=

finroberta=Import["FinRoBERTa.WLNET"]

Out[]=

NetChain



Input port:	string
Output port:	class



Sentiment Analysis

A simple example to show the application of both networks.

In[]:=

sentence="The company said that sales in the three months to the end of March slid to EUR86 .4 m US$ 113.4 m from EUR91 .2 m last year";

In[]:=

sentimentFinBERT=finbert[sentence]

Out[]=

negative

In[]:=

sentimentFinRoBERTa=finroberta[sentence]

Out[]=

negative

The answers are acceptable for both Sentiment Analysis.

Comparison with other BERTs

Let’s look at a prepared dataset for this step. To avoid further file import, the dataset is compressed.

In[]:=

ds=Uncompress["1:eJztXFtz28YZVe+Z6Uwv05k8f51m6hfJI8lJbKvtpLo6cixLI8p23tolsCA2XGCR3QUpZPq3+95zFqRsJbYjy4ouVl84XBIAwcPv+875Lss/D91h8YuFhYXwGzxsqaiCjsXf+cIv8fDEhFj8iqvf4mE9BJcZFY2ri5/PDzlsrR58hCcDXUddZ3rAs4/M+MiNJXN1MLn2QZ64One1qDoXF0vtxbosXSlI4byUWuXftspHHPu9a/8OTw66WOLsHVNvbB8eDXivtW6jV/Y1B++paNXwDAf/qT+41BXuJFNnOOPjH55x6HiOOnXSuyL2D2JvQq07ydoYROUSGl3nph4Jv7bK9NABTlWZXEbeTfnG0HWZi3FNXgwe/zhkH6XbG+FWJvrHMXvL0W8C7S2nvAW1U2e9K2y7NDSvjBVXSGVCICwvjNeZ8rnoYw0wcV2xMK4g0cmGtsq3YVG2YICDxuiRtuJ143y8aKs7B4LnMrv3AnCPAE5NhNdJUB0ML0aVjemvuZvW1qlc55IjKkjhXSVtQxS1GZVRalcvTbQ3hcEhKstcW99OEL/C4tndwV051E07tLg0YlrQeizWqKGxJnbSeBd11oc7FRAWvavVxMAWRcGlh8Ddwk6dq64BhFfgyY+weKGslUH0Wkd5QfjWSQprstOCMypVqxHtkrDhQ7TceZrMEJ6P25EnZkLn39LqZrIH0WhcMOdC7y7dWI01LjjRa/Kla4gFaTWWKsrzpVCqRoupRStf472rp9hT3/dyUDrCYntiZl5YAwrbwfOyMWNa4/VSA3GiK5OBLybaBuIVXKUlObfyWoU12UzvUcXsIOyF2ObdrXTYB1g8cm6EqDVkvCPB4hu/Gtio/RrjVdbRQ53v+ArOjrcSsK+x2LEpZMHcQjts/TBAw7kAvZcsrIStDVtjqfoWZdhG2dx/vru1tPIQh/uRFu1drntc24qMjWub6kyAvsXd3oDoG46+eETP7c/k3S1toU2mBsyhJg5M2hjrohStt64dlfDhRA+dqBF4hch7nbcZXLcytanaShrVrUmlK3eDQ+K5jfILLv56IJ8tLwvSjSAlDBSWBYmCG09525RqD4EOYFmqFUS+3r2nznujz8Ak7555nMP2LplLvqQ3H22e5Lfk21zzcngSXfMyZTvJQhKfKHhsBHqRinCoz5i7fYDhkBnvC5ialhxeS+VHFTdsLXh35L6QQ7wE64Lh8T3Y4/YzRMGqMlGGemTqayCTLz/TWMNiw0KxHDqoFlhQkeBQYJCZbtZ0VKjjlGVUyo813vOwRGjoa8ISl+yprBFsOYu8XwqYEnJ/3XoHsp0wukEdF671SxWUSZmi38zW8Bu5aBKSXoes1eCUs0D4AbrqIa3OudI5iZr++ewrpBjQxQOdeR0VhB1+mpAiGsNfViqolZBqMso2EIC4nZR9hLZpoLlxgPn+J94YKN+nzPcCi+3jDDya8jSYGUBBgBuDTQ3sbw/6rlKLsmHc06izclE2W6+fqyxJRvAIwiUcIZghZHcShzJRWWbqs9DwO8N5M3x7x3lq4QNd6WPI6QCfhi4h3abUDgDv6WOTuZQOj7wqoL+9URZxoGaxGZEAWUp8LSgfet2AVZfT5T8QRZSVe8snBT3oPxhdpEdHqGbAmXVDKsRUKARpGx6ixzfVnd8rMm70CxUdJTO8eA5mj2Nmlamok8d4aQpeaVjrgyuX2ja9x5cA8VZCd0DTY0ml9d2dIHt1m5UwLtfomggNraqhXei0I1weAa5vEwUwigWaQ4TQ9KLXkImFWKcuXBKeL4O75CT4b1js16Inmnw7BGq+bWLWLfY1hcL4CiK6dHDjlIrMmGbo6jbosHYtILtk1vgnXzUTZeGPPjLJiKyd0lchBx9pX6maNjnISme/EwrFl02lkCEDvnCyuBk+u7Mwa3RopL2u6iCoM6ZoZlSbgm2PKN4gDC7CHaEDvVN5X3HJiGG3Jrt7O0zoiuKiiwaX6arnhm8di23mHt+5WsMnFRILKSxsrk855jhJg+AnaqoMGHdRtr3J8AGsRauz9MYvpdF7yXUquuymK3UNtzxwpo73V2lZiS1eABU1RfQrmGUkkmByktTf7MWo1YX31W5EqLuPxW4tz6JCLqEk10N2zkI00L0prm2H6GoDc/MqN46ZGogEUiW+znRuTIQ7N17sh29zQqVvKn7TVkO3Jhue+W0p68ZPqZKR9Br67v1P70vfRcrbVFs+1f1AKDTXoEp1BYGOVWV2x+ieQe7sIduf1afuJKHGSgEysYqQ1a7u61gszKhAMgYBnzHMfWi1qsfpUBmaUa/eKOeCWOjgrIWjVmkuQ1vVsWvBxk+iWgBqkfemSlVFrNs6s7DLa0IVl2x8pIpnG4NF2UPqoGh4qra6g/htdMYUFkbHeSB57nJVkImPkjjePdhfA3O0PrvwYsqNoIr9hVO1qaOyg0mlbKKf+TlurPOsgvo0veeal7N7qYmZJg5MrWpKwqUzdtQ+RCemAe5maUZAGZ8qd5QimRqyeOL1NLV2x1o3EH8dYVMFawaFMpbmyXLgraSNbVpgyixM9lIMzw0OephzBPDoeVsjlqYeg51PGiKb2/u8mYsuqNyIOgA5d1uF7jHIIvXAAyt6fe+7VBB1NeVJXZich7OOt7m/L4kscGWZuLMovp+cLy598Hb3dNT7crCxicxMp2nRhr6IA5mejawbKiv629akikEqNINGJinY/TS0cTO8luHuiXUdINuAXJGQXDPVixn7TJ2xygf7g5YmbxQcWzFgjeS3Zy0HfIDNCwL36sijqpXtOAxkOTbQIAeTpzoW5vgOK+xqzJmhgKyCKIJ1dYYjroHLXoHJPT7ttAdIW2lqLK43DpaVucwp8KlPghkxrtHepCGCk4GqEgmHsNl21rb4NR1gOTeGD7HYRXzrq00wOjOTcMBvvY0QdJz/kWE3y3BXl1dXro8+vnSeYKenrwyL4rC2g7KblYKl4Y6NzDlL/AJcWFcBtJrDT/sebWhcTJN/t5IgCN0Pm7RP3HeuU6nynho+8y7sTAynHi5WJ+3bq4fu0m1uc4FzF17nqQ7wSgmgsK4fWs5Vt0Q/peHBjSOcF6aXt2kcbT7NfOXOegXRbZWL6E2jZWoQ3U7ahYWztt89Ne/o3KHvWquacNElzxuR9nMofpc783RgGTh2CGS1DJRl55rQZdaFWbeidJZBjblCP+tdR5/GqAw89ZN7MjTWMpnI9U/SILv+ku4eFqmRM2othwICUi8b5clge1EOdWFqXraf/FzsGzxpb9XVx7YrcFHOnjzpipj2XrgJ8lLJObfjZx3q1JvlRWB7Rtv8B/umXm6puqF++z7c8HSBs+8QHlmpZcMBNU1AcBOaW1um4nFKX51Dygq22Fr/mjW81gZCClqdsueNbGweB68FiJdcNGHFaUuZihMlrNMx55/vwECamlMYc0KvnO8BStoksP2dL812K6fa++3Mw/6FxVFJcjANBG/aLerFsHbCqW2kDnViD853cyCAOHLLitfkjolOxU+o5tQ745v4Abwaac4VcNvkmXYXfGiZxn66rFfpS74c5yY8EMRt8lkT572eZKhpfAV2euCJuznmGMsnsnpfqp6Qbx+I/5lZZuYqyOBOQhq8417IoKw+cepYclNQmv0OcxvUffsXP0FWSrD9wN72s8MHn8vdT6WSZ4NPZGXlXnqe9kTjvYcrcncVa8uiTXdrW2876dV2PDbyyLu2kUzZjEoo2S/Ah94pFQtcgAmwyadVPf8pZt0PcNg3IPXbid9/sdhQYQqpLZvONwiGvQCKzPf0cVaqGtHRa5sqquvtiBuJ7q3I6vLKMp1+5fO11c9kfgnOZTH2wiy5e5CqvJdWiqiDuup0+ZRl57gSp6L55xL44eaXSBvrcm2py2S3njhO07A8VM3uzNk2PeG2sE1X1xRkE254H2g/MWwFsjD3Y593K5MFSrj9RvM3Zrbeb+bxTLb6gCOfwTvmEUZW7z6QV7zFsSsmcwmCeLW6vPzgdvZbOQEb004yKA/FWVcfKSIASqZSTWkxmedUQxhr/i0IrGjWqgbyFx1qbsQo4jIW/BsZ2JDhHjtPeTYDKXULGza+OmJoDRKIHC59O63rLyfWxXiYu3YY59sk4IMnEw9Xb0WXvxvxD8S1n91K4HCU9cql5gVUMtJZf8TDUdfoQarn//s5iM351741iL7NZn+Sxc/iX2adBuo1KLzmu77pG735tk9/ZvEzrn5/+ubWQdXh1wt9NRTB7k3H9H8B9vHpN7Zr/s1AsoNTCL36s5wG/P+Xf/PlDf8WrVh4jZP9D+GMYRo="]

Out[]=

Sentence	Python FinBERT	Matlab FinBERT	Mathematica FinBERT	Mathematica FinRoBERTa
TikTok considers London and other locations for headquarters	neutral	neutral	neutral	neutral
Disney cuts ad spending on Facebook amid growing boycott: WSJ	negative	negative	negative	negative
Trail of missing Wirecard executive leads to Belarus, Der Spiegel reports	neutral	negative	neutral	negative
Twitter says attackers downloaded data from up to eight non-verified accounts	neutral	negative	neutral	negative
U.S. Republicans seek liability protections as coronavirus aid battle looms	neutral	negative	negative	negative
Wall Street Week Ahead: Fund managers navigate 'Night of the Living Dea	neutral	neutral	neutral	positive
Take Five: Hoping for that V-shape in earnings	neutral	neutral	positive	positive
Evictions nearly back to pre-pandemic levels in some U.S. areas: Cleveland Fed study	neutral	negative	negative	negative
Google bans ads on coronavirus conspiracy theory content	neutral	negative	negative	negative
Flight to suburbs boosts U.S. homebuilding, but COVID-19 surge erodes consumer sentiment	positive	positive	negative	positive
Delta will avoid pilot furloughs if they agree to reduced minimum pay: memo	neutral	neutral	positive	negative
S&P 500 ends higher as traders weigh stimulus and virus worries	negative	positive	positive	positive
FTC considering deposing top Facebook executives in antitrust probe: WSJ	neutral	negative	negative	negative
Where did all the bulls go? Rallies stall as EU summit begins	neutral	negative	neutral	negative
BlackRock profit beats Street estimates as markets rebound	positive	positive	positive	positive
Dollar falls, euro hovers at four-month high as EU negotiates rescue fund	neutral	negative	negative	negative
Boohoo tells UK Home Secretary itis probing charges of malpractice in supply chain	neutral	negative	negative	neutral
Exclusive: EU in talks with Modema, BioNtech, CureVac to secure possible COVID vaccines	neutral	positive	positive	positive
Former Pemex boss arrives back in Mexico for graft trial, enters hospital	neutral	neutral	neutral	positive
Twitter says about 130 accounts were targeted in cyber attack this week	neutral	negative	negative	negative
rows 1–20 of 60

The column “Sentence” contains 60 sentences from across the internet. The “Sentence” sentiments are listed in the columns “Mathematica FinBERT” and “Mathematica FinRoBERTa.” It is recognizable, that few sentiments differ among these two networks.
Sentiment analysis has been done with the “Huggingface FinBERT solution for Python” [7][8], and with the Matlab FinBERT solution [9]. A comparison of all the sentiments shows that there are differences between the sentiments.
The person who wrote the notebook is not a native English speaker, so some mistakes might happen during sentence interpretation. The sentiments calculated by Mathematica FinRoBERTA fit the expectations of most of the author. In general, the differences might be caused by a lack of training settings that are standard.
Therefore, the user is free to choose which solution he/she prefers.

References

[1] Wolfram Neural Net Repository, https://resources.wolframcloud.com/NeuralNetRepository

[2] Wolfram Neural Net Repository, Text Domain, https://resources.wolframcloud.com/NeuralNetRepository/inputdomain/Text/

[3] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, Veselin Stoyanov, 2019, RoBERTa: A Robustly Optimized BERT Pretraining Approach, https://arxiv.org/abs/1907.11692, arXiv:1907.11692v1

[4] The dataset of the Financial Phrase-Bank can be found here: https://www.researchgate.net/publication/251231364_FinancialPhraseBank-v10

[5] Pekka Malo, Ankur Sinha, Pekka Korhonen, Jyrki Wallenius, and Pyry Takala, 2014, Good debt or bad debt: Detecting semantic orientations in economic texts. Journal of the Association for Information Science and Technology 65, 4 (2014), 782–796. https://doi.org/10.1002/asi.23062 arXiv:arXiv:1307.5336v2

[6] selection of FiQA-2018 data, https://sites.google.com/view/fiqa/home

[7] https://github.com/yya518/FinBERT

[8] https://finbert.ai/sentiment

[9] https://github.com/matlab-deep-learning/transformer-models

[10] BERT Trained on BookCorpus and Wikipedia Data, https://resources.wolframcloud.com/NeuralNetRepository/resources/BERT-Trained-on-BookCorpus-and-Wikipedia-Data/

[11] RoBERTa Trained on BookCorpus, English Wikipedia, CC-News, OpenWebText and Stories Datasets, https://resources.wolframcloud.com/NeuralNetRepository/resources/RoBERTa-Trained-on-BookCorpus-English-Wikipedia-CC-News-OpenWebText-and-Stories-Datasets/

POSTED BY: Jürgen Kanz

4 Replies

Sort By:

Jürgen Kanz

Jürgen Kanz, @ JuergenKanz

Posted 2 years ago

I just recognized that it is no longer possible to download my transformer nets from the Wolfram Cloud. No problem, you can download the files in the following ways: Download FinBERT via the following link https://1drv.ms/u/s!Aty-DGt3XGT0hY8-DB2e1R5A6xzJVQ?e=zoONT8 Download FinRoBERTa via the following link https://1drv.ms/u/s!Aty-DGt3XGT0hY8_sXcFuRKgse7QZQ?e=CO2klO

POSTED BY: Jürgen Kanz

Jürgen Kanz

Jürgen Kanz, @ JuergenKanz

Posted 2 years ago

Michael, thank you for your kind words. Regards, Juergen

POSTED BY: Jürgen Kanz

Michael Kelly

Michael Kelly, Wolfram Research Inc.

Posted 2 years ago

Thanks Juergen for this wonderful example of the application of ML to financial sentiment. In my own experience with trading, our team always had different financial news channels on TVs and computers blaring throughout the day in case anything of interest had suddenly changed the market behavior. Most of the time it was just noise, but in this case you have provided a tool to filter through the noise to discover some real information. Regards Michael

POSTED BY: Michael Kelly

EDITORIAL BOARD

EDITORIAL BOARD, WOLFRAM

Posted 2 years ago

-- you have earned *Featured Contributor Badge* Your exceptional post has been selected for our editorial column *Staff Picks* http://wolfr.am/StaffPicks and Your Profile is now distinguished by a *Featured Contributor Badge* and is displayed on the Featured Contributor Board. Thank you!

POSTED BY: EDITORIAL BOARD

Reply to this discussion

Reply Preview

Attachments

Remove Add a file to this post

Follow this discussion

or Discard

Feedback