Message Boards

GROUPS:

Staff Picks Data Science Wolfram Language Natural Language Processing Wolfram High School Summer Camp Computational Humanities

Anika Karpurapu

[WSC21] Structuring and exploring TEI marked-up texts

Anika Karpurapu

Posted 17 days ago

985 Views

5 Replies

6 Total Likes

Follow this post

enter image description here

Structuring and exploring TEI marked-up texts
by Anika Karpurapu

Nowadays, play scripts are all over the internet, but they are often lengthy and difficult to navigate. These texts are often unstructured, so you may have a hard time quickly finding what you're looking for. Some play scripts are marked up in formats such as TEI and XML, making it slightly easier to traverse, but these texts are overwhelmed by tags and numerous descriptors. It takes a while to become accustomed to these formats and understand how everything is formatted. The goal of my project is to implement additional forms of text segmentation, specifically regarding plays. Given a marked up play, I am able to process it into a structured, hierarchical format, so that it may be easily interpreted by anyone. With this newly formatted text, I am able to conduct numerous analyses on different plays to find out information such as who the most prevalent characters are or what the main themes of each act are.

Introduction

Text Encoding Initiative, or TEI, is a variation of the well-known XML format. It currently acts as the standard for maintaining texts digitally, and is organized in more of a semantic matter. In TEI, everything from acts to individual words have their own tag, so everything is accounted for.

The tags mentioned in this project are as follows:

Methodology

Data Importing

The text used in this project is TEI-formatted and was downloaded from The Folger Shakespeare Library.

Importing the play Romeo and Juliet from the cloud.

In[]:=

playText=CloudImport[CloudObject[

https://www.wolframcloud.com/obj/anika.karpurapu/playText

]]

Out[]//Short=

XMLObject[Document][{XMLObject[ProcessingInstruction][xml-model,href="https://raw.githubusercontent.com/TEIC/TEI-Simple/master/teisimple.rng" type="application/xml" schematypens="http://relaxng.org/ns/structure/1.0"]},XMLElement[TEI,{{http://ww…000/xmlns/,…}…},{1,XMLElement[text,{},1]}],{}]

Text Processing Rules

By just looking at Romeo and Juliet in TEI format, it's difficult to figure out where acts begin and what lines are assigned to each character. In order to convert the play into a more legible and structured format, I created rules and patterns to process the data.

Word Processing

The wordProcessingRule deals with the most fundamental structure in our text — words. You can think of it as the base case to our recursive function. In a TEI-formatted document, a single sentence is broken down into words, punctuation characters, whitespace, and line beginnings. Each of these properties has it's own line, so you must read down several lines in order to fully read and understand a sentence. The following wordProcessingRule takes all of these attributes into account and allows us to label and sort through all of them in order to parse the text more easily later on.

Specifically, wordProcessingRule assigns the data into seven different categories, with the main categories being:

◼

Word: Either parses the <c> tag into whitespace or keeps track of the words and punctuation in a sentence.

◼

Type: Stores and renames the tag type into a more understandable label.

◼

ID: Each tag has a unique number/ID associated with it.

◼

Number: Each section in the play is broken down such that each corresponding group of tags has the same number.

◼

Lemma: Stores the root word(s) of a given word in the play.

If a certain data point doesn't fall into some of the categories, it is given a label of "Missing" for that specific property.

In[]:=

wordProcessingRule=XMLElement[type:("w"|"pc"|"c"|"lb"),attrib_List,word_]:>Module[{attassoc=Association@@attrib,w=Replace[word,{ww_}:>ww]},<|"Word"->Which[MatchQ[type,"c"]," ",True,w],"Type"->Replace[type,{"w"->"Word","pc"->"Punctuation","c"->"Whitespace","lb"->"Line Beginning"}],"ID"->Lookup[attassoc,Key[{"http://www.w3.org/XML/1998/namespace","id"}],Missing["NotAvailable"]],"Number"->Lookup[attassoc,"n",Missing["NotAvailable"]],"Lemma"->Lookup[attassoc,"lemma",Missing["NotAvailable"]],"Analysis"->Lookup[attassoc,"ana",Missing["NotAvailable"]],"Rendition"->Lookup[attassoc,"rendition",Missing["NotAvailable"]]|>];

The following is an example of a sentence that we would apply the wordProcessingRule to. The sentence “Gregory, on my word we’ll not carry coals.” is broken down into several different lines. Each line represents one line beginning, character, whitespace, or punctuation mark from the sentence.

In[]:=

exampleSentence=Cases[playText,XMLElement["p",{{"http://www.w3.org/XML/1998/namespace","id"}"p-0015"},_],Infinity]

Out[]=

{XMLElement[p,{{http://www.w3.org/XML/1998/namespace,id}p-0015},{XMLElement[lb,{{http://www.w3.org/XML/1998/namespace,id}ftln-0015,n1.1.1},{}],XMLElement[w,{{http://www.w3.org/XML/1998/namespace,id}fs-rom-0002620,n1.1.1,lemmaGregory,ana#n1-nn},{Gregory}],XMLElement[pc,{{http://www.w3.org/XML/1998/namespace,id}fs-rom-0002630,n1.1.1},{,}],XMLElement[c,{},{}],XMLElement[w,{{http://www.w3.org/XML/1998/namespace,id}fs-rom-0002650,n1.1.1,lemmaon,ana#acp-p},{on}],XMLElement[c,{},{}],XMLElement[w,{{http://www.w3.org/XML/1998/namespace,id}fs-rom-0002670,n1.1.1,lemmamy,ana#po},{my}],XMLElement[c,{},{}],XMLElement[w,{{http://www.w3.org/XML/1998/namespace,id}fs-rom-0002690,n1.1.1,lemmaword,ana#n1},{word}],XMLElement[c,{},{}],XMLElement[w,{{http://www.w3.org/XML/1998/namespace,id}fs-rom-0002710,n1.1.1,lemmawe|will,ana#pns|vmb},{we’ll}],XMLElement[c,{},{}],XMLElement[w,{{http://www.w3.org/XML/1998/namespace,id}fs-rom-0002730,n1.1.1,lemmanot,ana#xx},{not}],XMLElement[c,{},{}],XMLElement[w,{{http://www.w3.org/XML/1998/namespace,id}fs-rom-0002750,n1.1.1,lemmacarry,ana#vvi},{carry}],XMLElement[c,{},{}],XMLElement[w,{{http://www.w3.org/XML/1998/namespace,id}fs-rom-0002770,n1.1.1,lemmacoal,ana#n2},{coals}],XMLElement[pc,{{http://www.w3.org/XML/1998/namespace,id}fs-rom-0002780,n1.1.1},{.}]}]}

Applying the wordProcessingRule to previous sentence.

In[]:=

exampleSentence/.wordProcessingRule

Out[]=

{XMLElement[p,{{http://www.w3.org/XML/1998/namespace,id}p-0015},{Word{},TypeLine Beginning,IDftln-0015,Number1.1.1,LemmaMissing[NotAvailable],AnalysisMissing[NotAvailable],RenditionMissing[NotAvailable],WordGregory,TypeWord,IDfs-rom-0002620,Number1.1.1,LemmaGregory,Analysis#n1-nn,RenditionMissing[NotAvailable],Word,,TypePunctuation,IDfs-rom-0002630,Number1.1.1,LemmaMissing[NotAvailable],AnalysisMissing[NotAvailable],RenditionMissing[NotAvailable],Word ,TypeWhitespace,IDMissing[NotAvailable],NumberMissing[NotAvailable],LemmaMissing[NotAvailable],AnalysisMissing[NotAvailable],RenditionMissing[NotAvailable],Wordon,TypeWord,IDfs-rom-0002650,Number1.1.1,Lemmaon,Analysis#acp-p,RenditionMissing[NotAvailable],Word ,TypeWhitespace,IDMissing[NotAvailable],NumberMissing[NotAvailable],LemmaMissing[NotAvailable],AnalysisMissing[NotAvailable],RenditionMissing[NotAvailable],Wordmy,TypeWord,IDfs-rom-0002670,Number1.1.1,Lemmamy,Analysis#po,RenditionMissing[NotAvailable],Word ,TypeWhitespace,IDMissing[NotAvailable],NumberMissing[NotAvailable],LemmaMissing[NotAvailable],AnalysisMissing[NotAvailable],RenditionMissing[NotAvailable],Wordword,TypeWord,IDfs-rom-0002690,Number1.1.1,Lemmaword,Analysis#n1,RenditionMissing[NotAvailable],Word ,TypeWhitespace,IDMissing[NotAvailable],NumberMissing[NotAvailable],LemmaMissing[NotAvailable],AnalysisMissing[NotAvailable],RenditionMissing[NotAvailable],Wordwe’ll,TypeWord,IDfs-rom-0002710,Number1.1.1,Lemmawe|will,Analysis#pns|vmb,RenditionMissing[NotAvailable],Word ,TypeWhitespace,IDMissing[NotAvailable],NumberMissing[NotAvailable],LemmaMissing[NotAvailable],AnalysisMissing[NotAvailable],RenditionMissing[NotAvailable],Wordnot,TypeWord,IDfs-rom-0002730,Number1.1.1,Lemmanot,Analysis#xx,RenditionMissing[NotAvailable],Word ,TypeWhitespace,IDMissing[NotAvailable],NumberMissing[NotAvailable],LemmaMissing[NotAvailable],AnalysisMissing[NotAvailable],RenditionMissing[NotAvailable],Wordcarry,TypeWord,IDfs-rom-0002750,Number1.1.1,Lemmacarry,Analysis#vvi,RenditionMissing[NotAvailable],Word ,TypeWhitespace,IDMissing[NotAvailable],NumberMissing[NotAvailable],LemmaMissing[NotAvailable],AnalysisMissing[NotAvailable],RenditionMissing[NotAvailable],Wordcoals,TypeWord,IDfs-rom-0002770,Number1.1.1,Lemmacoal,Analysis#n2,RenditionMissing[NotAvailable],Word.,TypePunctuation,IDfs-rom-0002780,Number1.1.1,LemmaMissing[NotAvailable],AnalysisMissing[NotAvailable],RenditionMissing[NotAvailable]}]}

As we can see, the text looks a lot cleaner, which each attribute being properly labeled. If we remove all of the “Missing” characteristics, we can get a clearer view of the attributes and have a better idea of how the rule is working.

In[]:=

noMissing=DeleteCases[exampleSentence/.wordProcessingRule,Missing[_],Infinity]

Out[]=

{XMLElement[p,{{http://www.w3.org/XML/1998/namespace,id}p-0015},{Word{},TypeLine Beginning,IDftln-0015,Number1.1.1,WordGregory,TypeWord,IDfs-rom-0002620,Number1.1.1,LemmaGregory,Analysis#n1-nn,Word,,TypePunctuation,IDfs-rom-0002630,Number1.1.1,Word ,TypeWhitespace,Wordon,TypeWord,IDfs-rom-0002650,Number1.1.1,Lemmaon,Analysis#acp-p,Word ,TypeWhitespace,Wordmy,TypeWord,IDfs-rom-0002670,Number1.1.1,Lemmamy,Analysis#po,Word ,TypeWhitespace,Wordword,TypeWord,IDfs-rom-0002690,Number1.1.1,Lemmaword,Analysis#n1,Word ,TypeWhitespace,Wordwe’ll,TypeWord,IDfs-rom-0002710,Number1.1.1,Lemmawe|will,Analysis#pns|vmb,Word ,TypeWhitespace,Wordnot,TypeWord,IDfs-rom-0002730,Number1.1.1,Lemmanot,Analysis#xx,Word ,TypeWhitespace,Wordcarry,TypeWord,IDfs-rom-0002750,Number1.1.1,Lemmacarry,Analysis#vvi,Word ,TypeWhitespace,Wordcoals,TypeWord,IDfs-rom-0002770,Number1.1.1,Lemmacoal,Analysis#n2,Word.,TypePunctuation,IDfs-rom-0002780,Number1.1.1}]}

Although we would not remove any of the Missing values when working with the text in practice, this view of the sentence allows us to clearly see how each line is broken down.

Stage Directions

A play script consists of numerous stage directions, and each direction informs us about two main things—what characters are involved and a description of how the characters must proceed within a scene. In a TEI-formatted play, we have a <stage> tag to indicate that whatever information follows will be part of a stage direction.

One level beneath the <stage> tag, we will find our word processing tags (<w>, <pc>, <c>, <lb>). Thus, once we process our stage tag, we must go one level down and call the wordProcessingRule to fully process the given text. Whenever we need to go a couple of levels down and call a rule recursively, we place the referenced rule in the "Contents" attribute.

The stageDirectionRule processes any tag that begins with the tag <stage>.

In[]:=

stageDirectionRule=XMLElement[type:"stage",attrib_List,outType_]:>Module[{attassoc=Association@@attrib},<|"ID"->Lookup[attassoc,"id",Missing["NotAvailable"]],"Number"->Lookup[attassoc,"n",Missing["NotAvailable"]],"Type"->Lookup[attassoc,"type",Missing["NotAvailable"]],"Characters"->Lookup[attassoc,"who",Missing["NotAvailable"],StringSplit],"Contents"->(outType//.wordProcessingRule)|>];

Headings

Another tag we use is called the <head> tag. Essentially, this tag acts as a heading and oftentimes contains information about what scene or act is coming up.

The headRule searches for all of the <head> tags and begins processing from that level.

In[]:=

headRule=XMLElement[type:"head",attrib_List,outType_]:>Module[{attassoc=Association@@attrib},<|"Type"->Lookup[attassoc,"type",Missing["NotAvailable"]],"ID"->Lookup[attassoc,Key[{"http://www.w3.org/XML/1998/namespace","id"}],Missing["NotAvailable"]],"Contents"->(outType//.wordProcessingRule)|>];

Lines

Next, we have the lineRule, which accounts for the <l> tag. This tag represents each line in a play. If a certain character has multiple lines to speak in a row, each line will be broken down into <l> tags, and each <l> is further broken down into the word processing tags.

The lineRule processes all lines that include the <l>, or line, tag.

In[]:=

lineRule=XMLElement[type:"l",attrib_List,outType_]:>Module[{attassoc=Association@@attrib},<|"Type"->"Line","ID"->Lookup[attassoc,Key[{"http://www.w3.org/XML/1998/namespace","id"}],Missing["NotAvailable"]],"Number"->Lookup[attassoc,"n",Missing["NotAvailable"]],"Contents"->(outType//.wordProcessingRule)|>];

Speakers

The <speaker> tag in a play keeps track of the character(s) who perform next. A stage direction or a sentence usually follows a speaker rule, so to account for both of these possibilities, the speakerRule leads into both the stage direction and word processing rules.

The speakerRule starts processing from the "speaker" rule, and works its way down from there.

In[]:=

speakerRule=XMLElement[type:"speaker",attrib_List,outType_]:>Module[{attassoc=Association@@attrib},<|"Type"->"Speaker","ID"->Lookup[attassoc,Key[{"http://www.w3.org/XML/1998/namespace","id"}],Missing["NotAvailable"]],"Contents"->(outType//.{stageDirectionRule,wordProcessingRule})|>];

Paragraphs

In a prose play, the <p> tag divides speech into paragraphs. Each paragraph is filled with sentences, so we must call the wordProcessingRule next to drill down further.

Searching for each element beginning with a <p>, the pRule goes about processing the data.

In[]:=

pRule=XMLElement[type:"p",attrib_List,outType_]:>Module[{attassoc=Association@@attrib},<|"Type"->"Paragraph","ID"->Lookup[attassoc,Key[{"http://www.w3.org/XML/1998/namespace","id"}],Missing["NotAvailable"]],"Contents"->(outType//.wordProcessingRule)|>];

Speech

The next rule searches for the <sp> tag, which stands for speech. Along with paragraphs and lines, a single speech block contains stage directions and speakers. To account for all of these conditions, we make sure to call the line, speaker, paragraph, and stage direction rules after we process the information in the speech tag.

Here, we can see the spRule looking for all the <sp> tags and processing the data following this pattern.

In[]:=

speechRule=XMLElement[type:"sp",attrib_List,outType_]:>Module[{attassoc=Association@@attrib},<|"Type"->"Speech","ID"->Lookup[attassoc,Key[{"http://www.w3.org/XML/1998/namespace","id"}],Missing["NotAvailable"]],"Character"->Lookup[attassoc,"who",Missing["NotAvailable"],StringSplit],"Contents"->(outType//.{lineRule,speakerRule,pRule,stageDirectionRule})|>];

Main Rule

Finally, we have the mainRule. All of the previous rules culminate to this main rule, because it will process any tag in a play. We can apply this rule to any block of text and it will process it into a clean, readable format. The highest level tag in a play is the <div> tag, and the prologue, along with each act, each have this tag. Below this tag, we have more tags ranging from speech to stage directions to a heading, so those rules are called recursively in the "Contents" section.

The final, main rule that will take care of processing that play.

In[]:=

mainRule=XMLElement[type:"div",attrib_List,outType_]:>Module[{attassoc=Association@@attrib,w=outType},<|"Type"->Lookup[attassoc,"type",Missing["NotAvailable"]],"Number"->Lookup[attassoc,"n",Missing["NotAvailable"]],"Contents"->(outType//.{speechRule,stageDirectionRule,headRule})|>];

To see the main rule in action, we will navigate to Act 4 of the play and see how it breaks down each individual component.

In[]:=

act4=Cases[playText,XMLElement["div",{"type"->"act","n"->"4"},_],Infinity]/.mainRule

Out[]=

Typeact,Number4,Contents{

⋯1⋯

}

large output

show less

show all

set size limit...

Once again, if we view Act 4 without the Missing elements, we have a clear picture of what is going on.

In[]:=

DeleteCases[act4,Missing[_],Infinity]

Out[]=

Typeact,Number4,ContentsContents{WordACT,TypeWord,IDfs-rom-0380720,Word ,TypeWhitespace,Word4,TypeWord,IDfs-rom-0380740},

⋯4⋯

,XMLElementdiv,{

⋯1⋯

},Contents{WordScene,TypeWord,IDfs-rom-0428000,Word ,TypeWhitespace,Word5,TypeWord,IDfs-rom-0428020},

⋯1⋯

⋯53⋯

,TypeSpeech,IDsp-2731,Character

⋯1⋯

,ContentsTypeSpeaker,IDspk-2731,ContentsWordSECOND,TypeWord,IDfs-rom-0452470,

⋯1⋯

,WordMUSICIAN,TypeWord,IDfs-rom-0452490,

⋯1⋯

,

⋯1⋯



large output

show less

show all

set size limit...

Cast Rule

We can also create another rule to grab the character list from the play.

In[]:=

castRule=XMLElement[type:"castItem",attrib_List,outType_]:>Module[{attassoc=Association@@attrib},<|"Type"->"Cast Member","Character"->Lookup[attassoc,Key[{"http://www.w3.org/XML/1998/namespace","id"}],Missing["NotAvailable"]]|>];

Exploration

Now that the play is nicely formatted, it is a lot easier to analyze the text. I generated a created a couple of new functions and conducted some analyses about the text itself along with characters in the text.

Functions and Text Analysis

The following DisplayPrologue function can be used to display the prologue of any given TEI-formatted play. We can simply search for the "Contents" and "Word" labels under the "prologue" section in the processed text, and the prologue will be displayed.

In[]:=

DisplayPrologue[x_]:=(Flatten@Lookup[#,"Character"]->StringJoin@@Lookup[Flatten@Lookup[Flatten@Lookup[#,"Contents"],"Contents"],"Word",""]&/@(Select[#,#Type==="Speech"&]&/@Lookup[(Cases[x,XMLElement["div",{"type"->"prologue","n"->_},_],Infinity]/.mainRule),"Contents"]))[[1]][[2]]

In[]:=

DisplayPrologue[playText]

Out[]=

Two households, both alike in dignity(In fair Verona, where we lay our scene),From ancient grudge break to new mutiny,Where civil blood makes civil hands unclean.From forth the fatal loins of these two foesA pair of star-crossed lovers take their life;Whose misadventured piteous overthrowsDoth with their death bury their parents’ strife.The fearful passage of their death-marked loveAnd the continuance of their parents’ rage,Which, but their children’s end, naught could remove,Is now the two hours’ traffic of our stage;The which, if you with patient ears attend,What here shall mis