For my first project, I\ncreated a database for historical lexicons that allows information to be\nextracted based on the date of the lexicon. We wanted to find word-entries that\nare in the historical dictionaries but not in the OED, as well as word-entries\nthat antedate or postdate the OED citations. OED cites the first and last\noccurrence of a headword, but historical dictionaries also provide their own\ndate of publication. By comparing these dates, we can determine whether a\nhistorical lexicon antedates or postdates headwords cited by the OED. The date\nof the first and last occurrence of a word in the OED can be simply extracted\nby a SQL command. To accomplish this, I extracted lexemes from word-entries in\nXML files of historical dictionaries. A unique LEME ID is also needed to create\nthe tables. At first, I tried to use Python XML parsing tools to extract the\ninformation. However, I found that not all files were well formatted so I\nneeded to validate and modify them before extracting word-entries. I had to\nwrite programs that automatically corrected invalid XML elements. Then, I\ndecided to consider the original XML files as regular TXT files, which means\nthat I did not use built-in XML functions in Python; I wrote my own functions\nto delete XML tags so that only word-entries and their part of speech remained in\nthe TXT files. As a result, I could easily add word-entries with parts of\nspeech to the database I built, and I used SQL queries to extract the\ninformation we needed. Once we added word-entries into the database, we could\ngather information through one simple command. For example, if we want to find\nword-entries that antedate the OED, we could use a simple SQL command such as <\/p>\n\n\n\n
SELECT\nleme.wordentry, leme.pos FROM leme inner join oed on\noed.headword=leme.wordentry and oed.pos=leme.pos where leme.date < oed,first_date.\n<\/p>\n\n\n\n
Given a historical\ndictionary file, the program could add word-entries into the database and\ngenerate report files.<\/p>\n\n\n\n
Furthermore, individual\nreport files for historical dictionaries are insufficient for forming any\nconclusions, but graphs may be helpful for comparing several lexicons across\ndifferent periods of time. The database allows us to discover the number of\nantedatings, postdatings, and LEME lexemes not in the OED during a specific\nperiod. For example, we could generate graphs of LEME lexemes not in the OED\n(Figure 1).<\/p>\n\n\n\n
![\"\"](\"https:\/\/leme.utoronto.ca\/wp-content\/uploads\/2019\/10\/fig1.png\")
I\nalso created a user interface (UI) for this program (Figure 2). The program has\ntwo features: one, creating tables for each dictionary; and two, creating\ngraphs. To create tables for each dictionary, we give the program the full path\nof the dictionary file and its LEME id, and then click \u201cCreate.\u201d Now, the\nprogram can automatically parse the file, add new word-entries into the\ndatabase, and generate report files into the same directory as the input. Once\nthe program finishes creating tables for dictionary files, a message would\ninform the user whether the process has successfully finished. Clicking \u201cPlot\u201d\nautomatically plots the graph. When given the desired output path, graphs would\nbe created in that directory. (Figure 3)<\/p>\n\n\n\n
![\"\"](\"https:\/\/leme.utoronto.ca\/wp-content\/uploads\/2019\/10\/fig2.png\")
![\"\"](\"https:\/\/leme.utoronto.ca\/wp-content\/uploads\/2019\/10\/fig3.png\")
The\nsecond project is a Natural Language Processing project on automatically\nlemmatizing lexicons based on the OED. I started with studying Python NLP tools\nand papers on string matching. I benefited from Natural Language Processing for Historical Texts<\/em>, edited by Graeme\nHirst. The book emphasizes that the spelling of words varies frequently,\ndepending on the grammar and historical period in which the word occurs. To\naddress the variance of spelling, word-entries need to be normalized by\nstandardizing grammar and spelling. In Python, there is a toolkit called NLTK,\nwhich provides text processing libraries for tokenization, stemming, tagging,\nparsing and so on.[2]<\/a>\nGiven a sentence, NLTK could automatically recognize the root word (Figure 4)\nor add part of speech tags (Figure 5). <\/p>\n\n\n\n However, historical\nspelling is not the same as modern spelling, as there are some rules and\nvariations. Referring to an OED blog post on early modern English pronunciation\nand spelling, I used regular expressions to implement its rules so that\nhistorical spelling could be recognized and modified.[3]<\/a> As a result, the program\ncan recognize some simple historical variations. For example, \u2018abiect\u2019 is\nrecognized as \u2018abject\u2019, and \u2018aboue\u2019 is recognized as \u2018above\u2019 (Figure 6).\nHowever, there are more variations than simple rules. Consequently, I tried to\nfind the most similar word in the dictionary by edit distance and longest\ncommon subsequence. Edit distance is way to quantify how dissimilar two strings\nare.[4]<\/a> It measures the minimum\nnumber of operations (including insertions, deletions, and substitutions)\nrequired to change one word to another. Instead of regular edit distance, I\nused Damerau-Levenshtein distance, which includes the transposition of two\nadjacent characters as an operation. Damerau-Levenshtein distance allows us to\naccommodate the shifting of historical spellings into modern spellings of\nwords. For example, the program registers that \u2018abel\u2019 could be a historical\nspelling for \u2018able\u2019. Longest common subsequence helps by checking the order of\nthe characters in the original strings. If I find more than one similar word in\nthe dictionary, I will list all of them. For example, although \u2018heereafter\u2019 is\nnot in the OED, the program can find \u2018thereafter, whereafter, hereafter\u2019 as similar\nwords.<\/p>\n\n\n\n I then created a UI to\nrun this program. It could lemmatize a single word or a dictionary file (Figure\n7).<\/p>\n\n\n\n However, this program runs slow for large files since it\nguesses the words by edit distance and compares the results with 97,800 OED\nword-entries. I came up with a way to improve the speed by running the program\nin multi-thread. In multi-thread, I can lemmatize several words at the same\ntime, which drastically reduces the time it takes for the program to run. Since\nlemmatizing a single word takes around three minutes, without multi-thread, lemmatizing twenty\nword-entries would take around one hour. With multi-thread, we could lemmatize five\nword-entries simultaneously, and the entire process would take around twelve\nminutes. We could also run the program on a more powerful computer or a virtual\nmachine to improve the efficiency. [1]<\/a>\n\u201cRelational database,\u201d Wikipedia, accessed June 6, 2019, https:\/\/en.wikipedia.org\/wiki\/Relational_database.<\/p>\n\n\n\n [2]<\/a>\nSteven Bird, Edward Loper and Ewan Klein, Natural Language Processing\nwith Python<\/em> (O\u2019Reilly Media Inc., 2009). <\/p>\n\n\n\n![\"\"](\"https:\/\/leme.utoronto.ca\/wp-content\/uploads\/2019\/10\/fig4.png\")
![\"\"](\"https:\/\/leme.utoronto.ca\/wp-content\/uploads\/2019\/10\/fig5.png\")
![\"\"](\"https:\/\/leme.utoronto.ca\/wp-content\/uploads\/2019\/10\/fig6.png\")
![\"\"](\"https:\/\/leme.utoronto.ca\/wp-content\/uploads\/2019\/10\/fig7.png\")
<\/p>\n\n\n\n
\n\n\n\n