Message Boards

GROUPS:

Staff Picks Data Science Wolfram Language Computational Linguistics Natural Language Processing Wolfram High School Summer Camp Computational Humanities

Emily Do

[WSC22] Exploring the divergence in pronunciations of Chinese characters

Emily Do

Posted 21 days ago

400 Views

1 Reply

7 Total Likes

Follow this post

enter image description here

Exploring the divergence in pronunciations of Chinese characters
By Emily Do

In this project, I explored the divergence of the pronunciations of Chinese characters in the modern Chinese, Japanese, and Korean languages by creating a data table where each Chinese character was listed next to its Romanized pronunciations in all three languages (Romanized pronunciation meaning the closest equivalent sound to the actual pronunciation using the Roman alphabet). I did this by scraping online dictionaries of all three languages and parsing their pronunciation keys, and in the case of Korean, systematically converting Korean characters to their Romanized pronunciations. From the data table, I found the distances between each of the languages' pronunciations of Chinese characters, and compared all of the homonyms. Further patterns in the divergence of these pronunciations can be found using this work.

Introduction

Finding patterns in changes in pronunciations of Chinese characters between Chinese, Japanese, and Korean can help one better understand the connections between these languages. Such patterns can also make it easier for one to learn one of these languages if they already know another.

Background Information

Before learning about the history of Chinese, Japanese, and Korean, it is important to note that language is separate from writing system. The word “language” only refers to the communication in the form of speech rather than both spoken and written communication, which both the Japanese and Koreans had before they started using Chinese characters. However, they both initially used Chinese characters to express the sounds in their languages because they did not have their own writing systems.

Out of the Chinese, Japanese, and Korean writing systems, Chinese script is the oldest and heavily influenced both Japanese and Korean vocabularies which went along with the spread of their soft and hard power. [7] About 60% of the Japanese vocabulary and about 75% of the Korean vocabulary originated from Chinese. [36, 40] Chinese also greatly influenced the Vietnamese vocabulary. [18] The Chinese writing system is made up of more than 50,000 characters, each of which are logograms, meaning they each have individual meanings. [7] Some modern Chinese characters trace back to around 1200 BC during the Shang Dynasty, but the process of the creation of characters started even earlier. [38] However the Chinese writing system’s standard written form was only created in 1913. [7]

The Japanese first adopted Chinese characters for their own language, and later developed their own writing scripts, katakana and hiragana, created around the 9th century.[7, 23] Katakana and hiragana are both syllabaries, meaning each character represents a syllable. [23, 24]

Koreans initially also had their own spoken language but not writing system, so they wrote with Chinese characters. [7] Most of the ancient Korean texts were written in Chinese script. [18] In the 15th century, Korea created its own written language, but Chinese characters were still used for some time. [7] Vocabulary derived from Chinese characters is still heavily used, but now the sounds are expressed with the Korean script.

Although both the Japanese and Koreans now have their individual writing systems for expressing sounds, their vocabularies derived from Chinese are still in use and retain much of the original pronunciations. [7] In this project, I studied the Romanized versions of those pronunciations (romaji for Japanese), alongside the Romanized pronunciations of Chinese characters, called pinyin. [9]

Compiling a List of Traditional Chinese Characters and Pronunciations

First, I found an online Chinese dictionary, and I wanted to create a list of traditional Chinese characters together with their pronunciations. [10]
I took the standard URL for the website so that you could add on the English word you wanted the dictionary entry for, then import the data from that entry.

In[]:=

standardc="https://chinesedictionary.net/translate-english/";

The URL and data from entering the word "search":

In[]:=

examplec=StringJoin[standardc,"search"]

Out[]=

https://chinesedictionary.net/translate-english/search

examplecdata=Import[examplec,"Data"]

Then I needed to take the relevant data from each entry, namely, the translation in traditional Chinese characters and the pronunciation. There were often multiple translations for an English word, so I took the characters from all of them.

Taking the translations of the word "search" in traditional characters and pronunciations. The dictionary offered four translations.

In[]:=

Cases[examplecdata[[2,2]],{"Traditional",__}]

Out[]=

Traditional,搜查,Traditional,搜索,Traditional,探訪,Traditional,追尋

In[]:=

Cases[examplecdata[[2,2]],{"Pronunciation",__}]

Out[]=

{{Pronunciation,sōu chá},{Pronunciation,sōu suǒ},{Pronunciation,tàn fǎng},{Pronunciation,zhuī xún}}

So, the generalized functions for getting the characters and pronunciations were:

Taking the data:

In[]:=

datac[word_]:=Import[StringJoin[standardc,word],"Data"]

Taking the relevant information (without the headers of "Traditional" and "Pronunciation":

In[]:=

infoc[x_]:={Characters@#[[2]]&/@Cases[x[[2,2]],{"Traditional",__}]//Flatten,TextWords@#[[2]]&/@Cases[x[[2,2]],{"Pronunciation",__}]//Flatten}

Then, using both those functions to create a list of characters together with their pronunciations:

In[]:=

functionc[word_]:=Transpose[infoc[datac[word]]]

The final results from inputting the word "search":

In[]:=

functionc["search"]

Out[]=

搜,sōu,查,chá,搜,sōu,索,suǒ,探,tàn,訪,fǎng,追,zhuī,尋,xún

Taking the Japanese Pronunciation of a Character

Similarly to the Chinese dictionary, I created a standard URL for a Japanese dictionary such that I could replace the string "keyWord" with the Chinese character I wanted to look up. [22] The only difference was that I had to run the function URLEncode on the character so the website could take the URL, then I imported the data from that dictionary entry.

The standard URL:

In[]:=

urlj="https://www.japandict.com/kanji/keyWord?lang=en"

Out[]=

https://www.japandict.com/kanji/keyWord?lang=en

Inserting the encoded character so the website would understand the URL:

In[]:=

urlSearchj=StringReplace[urlj,"keyWord"->URLEncode["搜"]]

Out[]=

https://www.japandict.com/kanji/%E6%90%9C?lang=en

Importing the data from the dictionary entry:

In[]:=

webSource=Import[urlSearchj,"Source"];

Within the source data, I needed to find the string <dt>On’yomi</dt> as it would be near the pronunciation I needed. On'yomi is one of the two ways that the Japanese read Chinese characters (the other being "Kun'yomi). On'yomi is a way of reading derived from Chinese pronunciations, while Kun’yomi is the original Japanese way of reading Chinese characters. [8] The "On'yomi" readings are the pronunciations used in this project.

Finding the start index of the string position of <dt>On’yomi</dt>

In[]:=

start=StringPosition[webSource,"<dt>On'yomi</dt>"][[1,1]]

Out[]=

32453

Taking the segment including that string:

In[]:=

segment=StringTake[webSource,{start,start+220}]

Out[]=

<dt>On'yomi</dt><dd class="ms-3 mt-2">ソウシュ<span

Picking out the string ”title=\””, which would be next to the On'yomi pronunciation:

In[]:=

begin=StringPosition[segment,"title=\""][[1,2]]

Out[]=

113

Taking the segment after that string:

In[]:=

remainder=StringTake[segment,begin;;-1]

Out[]=

"sou">ソウシュ<span

Finding the position of the character that separated the On'yomi pronunciation from the rest of the segment:

In[]:=

position=StringPosition[remainder,"\">"][[All,1]][[1]]-2

Out[]=

Taking the On'yomi pronunciation:

In[]:=

StringTake[segment,{begin+1,begin+position}]

Out[]=

sou

Then I created functions for each step of the process, and consolidated that into one function that would return a Chinese character and its Japanese pronunciation.

In[]:=

importj[char_]:=Import[StringReplace[urlj,"keyWord"->URLEncode[char]],"Source"]segmentj[char_]:=StringTake[importj[char],{StringPosition[importj[char],"<dt>On'yomi</dt>"][[1,1]],StringPosition[importj[char],"<dt>On'yomi</dt>"][[1,1]]+220}]beginj[char_]:=StringPosition[segmentj[char],"title=\""][[1,2]]remainderj[char_]:=StringTake[segmentj[char],beginj[char];;-1]positionj[char_]:=StringPosition[remainderj[char],"\">"][[All,1]][[1]]-2takej[char_]:=StringTake[segmentj[char],{beginj[char]+1,beginj[char]+positionj[char]}]functionj[char_]:={char,takej[char]}

The final results from inputting the word “搜":

In[]:=

functionj["搜"]

Out[]=

搜,sou

Taking the Korean Pronunciation of a Character

There were two steps to taking the Korean pronunciation of a Chinese character. The Korean dictionary I used took a Chinese character and gave the Korean pronunciation of it, but in a Korean character. [43] So, the first step was getting that character.
I again used a standard URL with the string "searchKey" replaced with an encoded Chinese character. Then I imported the source data from the dictionary entry pulled up from the URL and looked for the string that would be next to the Korean character I wanted. Lastly, I took that Korean character.

In[]:=

urlk="https://dic.daum.net/search.do?q=searchKey&dic=hanja";koreandata[char_]:=Import[StringReplace[urlk,"searchKey"->URLEncode[char]],"Source"]koreanstring[char_]:=StringCases[koreandata[char],Shortest["data-tiara-action-name=\"훈음 클릭\">"~~x__~~","]:>x][[1]],StringCases[koreandata[char],Shortest["data-tiara-action-name=\"훈음 클릭\">"~~x__~~"<"]:>x][[1]]koreanpronunciationfunction[char_]:=MinimalBy[koreanstring[char],StringLength[#]&][[1]]koreanpronunciationfull[char_]:=StringSplit[koreanpronunciationfunction[char]," "][[-1]]

Getting the Korean pronunciation in Korean characters of a Chinese character:

In[]:=

koreanpronunciationfull["搜"]

Out[]=

수

As opposed to in the Japanese dictionary, from which I could take just the Romanized pronunciation of a Chinese character, I needed to convert the Korean character taken from the Korean dictionary into its Romanized pronunciation.
Each Korean character is made up of a starting consonant and a vowel, and sometimes an ending consonant. The unicode range of Korean characters includes all the possibilities created in this format, in an organized sequence. The functions below use this organization to map each consonant and vowel to a number. Then, each of those are mapped to their Romanized pronunciation, so that the Korean characters could be converted to their pronunciation.

In[]:=

charToCount[char_]:=ToCharacterCode[char][[1]]-ToCharacterCode["가"][[1]]consonantCount[char_]:=Quotient[charToCount[char],588]vowelCount[char_]:=Quotient[Mod[charToCount[char],588],28]endCount[char_]:=Mod[Mod[charToCount[char],588],28]rule1={0"g",1"gg",2"n",3"d",4"dd",5"r",6"m",7"b",8"vv",9"s",10"ss",11"",12"j",13"jj",14"ch",15"k",16"t",17"p",18"h"};rule2={0"a",1"ae",2"ya",3"yae",4"eo",5"e",6"yeo",7"ye",8"o",9"wa",10"wae",11"oi",12"yo",13"u",14"weo",15"we",16"wi",17"yu",18"eu",19"ui",20"i"};rule3={0"",1"g",2"gg",3"gs",4"n",5"nj",6"nh",7"d",8"l",9"rg",10"rm",11"rb",12"rs",13"rt",14"rp",15"rh",16"m",17"b",18"bs",19"s",20"ss",21"ng",22"j",23"ch",24"k",25"t",26"p",27"h"};koreanCode[char_]:=StringJoin@{consonantCount[char]/.rule1,vowelCount[char]/.rule2,endCount[char]/.rule3}

Converting a Korean character to its Romanized pronunciation:

In[]:=

koreanCode["수"]

Out[]=

Finally, a function that combines the two steps and returns a Chinese character with its Korean pronunciation:

In[]:=

functionk[char_]:={char,koreanCode[koreanpronunciationfull[char]]}

The final results from inputting the word “搜":

In[]:=

functionk["搜"]

Out[]=

搜,su

Creating a List of Characters and Pronunciations

After creating the above functions, I needed a word list to run them on. Due to the format of the Chinese dictionary function, the words all had to be in English, so I took a list of the most frequently used words from Wikipedia. [34]

In[]:=

commonwordsdata=Import["https://en.wikipedia.org/wiki/Most_common_words_in_English","Data"][[1,1,2]];

Getting just the list of words (the full data included information about each word):

In[]:=

wordlist=Drop[#[[1]]&/@commonwordsdata,1]

Out[]=

{the,be,to,of,and,a,in,that,have,I,it,for,not,on,with,he,as,you,do,at,this,but,his,by,from,they,we,say,her,she,or,an,will,my,one,all,would,there,their,what,so,up,out,if,about,who,get,which,go,me,when,make,can,like,time,no,just,him,know,take,people,into,year,your,good,some,could,them,see,other,than,then,now,look,only,come,its,over,think,also,back,after,use,two,how,our,work,first,well,way,even,new,want,because,any,these,give,day,most,us}

However, some of the words did not have dictionary entries in the Chinese dictionary, so I created a function to discard those.

In[]:=

cleanwordlist[list_]:=If[Quiet@functionc[#]==={},Nothing,#]&/@list

In[]:=

cleanwordlist[wordlist]

Out[]=

{be,to,of,and,in,that,have,it,not,on,with,he,as,you,at,this,but,his,by,from,they,we,her,she,or,an,will,my,one,all,there,their,what,so,up,out,if,about,who,get,which,me,when,make,can,like,time,no,just,him,know,take,people,year,good,some,see,other,than,then,now,look,only,over,think,back,use,two,how,work,first,well,way,even,new,because,any,these,give,most,us}

I then ran that edited word list through the Chinese dictionary function, and selected the results that came back with a character and pronunciation.

In[]:=

chinesewordlist=Quiet[functionc[#]&/@cleanwordlist[wordlist]]

In[]:=

chinesewordlistfull=Cases[Flatten[chinesewordlist,1],{__,__}]

Editing the Character List

I then wanted to take each of the Chinese characters from the list, and use the Japanese and Korean dictionaries to get their pronunciations in both languages. However, some of the Chinese characters did not have entries in the Japanese and Korean dictionaries, so I wanted to preemptively discard such characters in the character list.
As shown above, to get the pronunciation of Chinese characters in the Japanese and Korean dictionaries, I looked for specific strings in the source data of the dictionary entries of each character. If those strings did not show up, I then knew that the character did not have a pronunciation in the language, and discarded it.
For each list, I took the intersection of the results that it had in both functions, so that the character would have pronunciations in all three languages.
Finally, I created a function combining the three first functions that would take the format of the above list (both characters and pronunciations). It took the above list of Chinese characters and their pronunciations and returned only those with pronunciations in all the languages.

In[]:=

cleanjapaneselist[list_]:=DeleteDuplicates[If[StringPosition[importj[#],"<dt>On'yomi</dt>"]==={},Nothing,#]&/@list]cleankoreanlist[list_]:=DeleteDuplicates[If[StringCases[koreandata[#],Shortest["data-tiara-action-name=\"훈음 클릭\">"~~x__~~","]:>x]==={},Nothing,#]&/@list]cleantotallist[list_]:=Intersection[cleankoreanlist[list],cleanjapaneselist[list]]cleanchineselist[list_]:=If[Intersection[cleantotallist[#[[1]]&/@list],#]==={},Nothing,#]&/@list//DeleteDuplicates

I first tried running the function on the whole Chinese word list, but it was too big a list for the function to do all at once, so I split the list into groups of 10 sublists (each with a character and its pronunciation), and ran the function separately on each sublist.
For example, the first sublist was:

In[]:=

characters1=cleanchineselist[chinesewordlistfull[[1;;10]]]

Out[]=

係,xì,系,xì,至,zhì,於,yú,由,yóu,對,duì,向,xiàng,給,jǐ,為,wèi,的,dì

Creating a Data Table

There ended up being 64 "clean" character lists. I then grouped each of them into groups of 5, ending up with 12 larger character lists (the last one had 4 extra sublists).

The first character list:

In[]:=

characterlist1=Join[characters1,characters2,characters3,characters4,characters5]

Then I wanted to make each sublist with a Chinese character and its pronunciation into a list including its pronunciations in Korean and Japanese.

In[]:=

datalist[list_]:={#,functionj[#[[1]]],functionk[#[[1]]]}&/@list

The "datalist" function run on the first character list of 12:

In[]:=

datalist1=datalist[characterlist1]

The "finaldatalist" function makes sure everything in the list is the correct format and that there are no duplicates, so that everything can

formatlist[list_]:=DeleteDuplicates[Flatten[#]]&/@listfinaldatalist[list_]:=If[Length[#]!=4,Nothing,#]&/@formatlist[list]//DeleteDuplicates

The "finaldatalist" function run on the first data list of 12:

In[]:=

finaldatalist1=finaldatalist[datalist1]

Then I combined all the separately formatted data lists and sorted them by alphabetical order (in the Chinese pronunciation).

In[]:=

fulldatalist=SortBy[Join[finaldatalist1,finaldatalist2,finaldatalist3,finaldatalist4,finaldatalist5,finaldatalist6,finaldatalist7,finaldatalist8,finaldatalist9,finaldatalist10,finaldatalist11,finaldatalist12]//DeleteDuplicates,#[[2]]&]

Finally, I created a function that would put my full data list into a table.

In[]:=

chartlist[list_]:=GridBox[Insert[list,{"Character","Chinese Pronunciation","Japanese Pronunciation","Korean Pronunciation"},1],ColumnLines->True,RowLines->True]//DisplayForm//Framed

In[]:=

chartlist[fulldatalist]

Out[]=

Character	ChinesePronunciation	JapanesePronunciation	KoreanPronunciation
彼	bǐ	hi	pi
搬	bān	han	ban
般	bān	han	ban
報	bào	hou	bo
抱	bào	hou	po
本	běn	hon	bon
必	bì	hitsu	pil
邊	biān	hen	byeon
別	bié	betsu	byeol
並	bìng	hei	byeong
并	bìng	hei	byeong
不	bu	fu	bul
不	bù	fu	bul
此	cǐ	shi	cha
才	cái	sai	jae
纔	cái	san	jae
初	chū	sho	cho
辰	chen	shin	jin
啻	chì	shi	si
次	cì	ji	cha
從	cóng	juu	jong
打	dǎ	da	ta
睹	dǔ	to	do
大	dà	dai	dae
帶	dài	tai	dae
待	dài	tai	dae
殆	dài	tai	tae
單	dān	tan	dan
但	dàn	tan	dan
當	dāng	tou	dang
當	dàng	tou	dang
到	dào	tou	do
道	dào	dou	do
得	de	toku	deug
的	de	teki	jeog
得	dé	toku	deug
地	dì	chi	ji
的	dì	teki	jeog
第	dì	dai	je
頂	dǐng	chou	jeong
都	dōu	to	do
獨	dú	doku	dog
多	duō	ta	da
對	duì	tai	dae
娥	é	ga	a
而	ér	ji	i
二	èr	ni	i
爾	ěr	ji	i
法	fǎ	hou	beob
甫	fǔ	ho	bo
發	fā	hatsu	bal
凡	fán	bon	beom
方	fāng	hou	bang
放	fàng	hou	bang
風	fēng	fuu	pung
否	fǒu	hi	bu
弗	fú	futsu	bul
該	gāi	gai	hae
概	gài	gai	gae
剛	gāng	gou	gang
個	ge	ko	gae
個	gè	ko	gae
給	gěi	kyuu	geub
跟	gēn	kon	geun
公	gōng	kou	gong
工	gōng	kou	gong
姤	gòu	kou	gu
覯	gòu	kou	gu
果	guǒ	ka	gwa
關	guān	kan	gwan
罐	guàn	kan	gwan
光	guāng	kou	gwang
過	guò	ka	gwa
還	hái	kan	hwan
耗	hào	mou	mo
何	hé	ka	ha
和	hé	wa	hwa
曷	hé	katsu	gal
很	hěn	kon	heun
好	hǎo	kou	ho
候	hou	kou	hu
候	hòu	kou	hu
後	hòu	go	hu
話	huà	wa	hwa
活	huó	katsu	hwal
或	huò	waku	hog
給	jǐ	kyuu	geub
脊	jǐ	seki	cheog
及	jí	kyuu	geub
季	jì	ki	gye
既	jì	ki	gi
洎	jì	ki	gye
繼	jì	kei	gye
假	jiǎ	ka	ga
間	jiān	kan	gan
將	jiāng	shou	jang
交	jiāo	kou	gyo
價	jie	ka	ga
節	jié	setsu	jeol
接	jiē	setsu	jeob
皆	jiē	kai	gae
徑	jìng	kei	gyeong
淨	jìng	jou	jeong
逕	jìng	kei	gyeong
就	jiù	shuu	chwi
今	jīn	kon	geum
均	jūn	kin	gyun
僅	jǐn	kin	geun
緊	jǐn	kin	gin
丼	jǐng	ton	jeong
井	jǐng	sei	jeong
睊	juàn	en	gyeon
厥	jué	ketsu	gweol
看	kàn	kan	gan
靠	kào	kou	go
刻	kè	koku	gag
可	kě	ka	ga
考	kǎo	kou	go
賴	lài	rai	roi
類	lèi	rui	ryu
連	lián	ren	ryeon
良	liáng	ryou	ryang
嫽	liáo	rou	ro
撩	liáo	ryou	ryo
臉	liǎn	ken	geom
兩	liǎng	ryou	ryang
另	lìng	rei	ryeong
領	lǐng	ryou	ryeong
靡	mǐ	hi	mi
麼	me	bi	ma
沒	méi	botsu	mol
們	men	mon	mun
氓