This blog post has three main goals:
- Review potential historical source that can be used as digital inputs for your programs
- Establish how to open text digital input in your code
- Explore how to transform and clean up your text inputs (removing stopwords, cleaning up formatting, and other considerations)
This post attempts to provide a simple and digestible overview of what to consider when thinking of what historical sources could be used your programs. However, it does not cover the basic information regarding installing python3, setting up a virtual environment, installing libraries, or defintions for common terminology (lists, strings, etc.).
To conduct code-based research analysis, you need to find (or create) digital source inputs. The potential inputs you can use are really limitless. Analysis can be performed on text files, data tables, digitally-uploaded images, auditory files, and more modalities.
The following focuses on cleaning up text files. Many historical primary sources are already available online (via various efforts to digitize and archive cultural works such as Project Gutenberg, Fordham Universities Internet History Sourcebooks Project, the National Archives, etc.). For written materials, you can find and download sources as txt, csv (comma-separated values), html (HyperText Markup Language), or other formats. I found it easiest to work with the raw text files as they are easy to transform into strings, there are less notation additions, and, thus, there is less clean up required.
If you only have access to a certain format (like html), there are steps you can take to clean up your files and ensure that they can be used as an input. Explore other posts and forums to learn more about the numerous files types or inputs you can use in your analysis.
Download and add your text file to the workspace or virtual environment (venv) you are using and then you can use it in analysis.
For example, I imported the raw text of Wheeler Thackston's translation of the Jahangirnama made available by Freer Gallery of Art, Arthur M. Sackler Gallery, Smithsonian Institution (doi:10.5479/sil.849796.39088018028456 )
At this point, you can manually conduct some very high-level clean up efforts to ensure the text is ready for code-based analysis. This could include actions like manually deleting prefaces or indices at the end of the text you don't want to include in your analysis. After saving this file as jahangir_wheeler_thackston_translation.txt in my virtual environment, I am able to open it in other programs in the same environment.
filename = 'jahangir_wheeler_thackston_translation.txt'
file = open(filename, 'rt')
jahangirnama_text = file.read()
file.close()
Next, use the built-in python functionality and nltk library (ensure you have installed the library first: pip install nltk) to transform your text from a string to a list as well as generally clean up your input file. While there are many different approaches and different libraries that you can take to transform and clean up your input, the nltk library provide powerful, ready-made functions.
from nltk.corpus import stopwords
from nltk.tokenize import word_tokenize
from nltk.stem import PorterStemmer
tokens = word_tokenize(jahangirnama_text)
text_turned_into_list = [word for word in tokens if word.isalpha()]
After creating a list, you can transform your input further by making the list all lower case, removing stopwords (a predefined list of commonly used words built into the nltk library), editing punctuation issues, or other issues could hinder you analysis.
For example, you don't want a frequency function to tell you that "the" or "a" are the most frequently used words or think that 'Pearl' and 'pearl' are different words. Removing stopwords or capitalization allows your programs to dig more deeply into the meaning and data of the text.
text_lower = [parts.lower() for parts in text_turned_into_list]
stopwords = nltk.corpus.stopwords.words('english')
new_words = ['one', 'also', 'two', 'would']
for i in new_words:
stopwords.append(i)
final_text = [w for w in text_lower if not w in stopwords]
Additionally, you can stem the words. This may or may not be necessary depending on the analysis you want to perform. Stemming, as the word suggests, cuts the end of words or reduces the word to its root. There are different types of stemming or lemmatization and are built into the nltk library (PorterStemmer, LancasterStemmer, etc.).
ps = PorterStemmer()
stemmed_words=[]
for t in final_text:
stemmed_words.append(ps.stem(t))
Use print() command to test the results along the way. Given we usually use primary sources that are hundreds of pages long (the Jahangirnama is around 500-pages), practice using indices to cut what is shows in your terminal.
print(final_text[:100])
You can turn all this code into a neat, reusable function. See this post for more details on building functions. !
Example File Clean-up Function (no stemming):
- Function: function_name(filename)
- Purpose: When called, the function opens a file inputted as the parameter
filename and performs all necessary clean up tasks (i.e. tokenizes, removes stop words, etc.).
- Parameters:
filename: can be any source file saved in the same environment
def file_function (filename):
import nltk
from nltk.corpus import stopwords
from nltk.tokenize import word_tokenize
file = open(filename, 'rt')
text = file.read()
file.close()
tokens = word_tokenize(text)
text_turned_into_list = [word for word in tokens if word.isalpha()]
text_lower = [parts.lower() for parts in text_turned_into_list]
stopwords = nltk.corpus.stopwords.words('english')
new_words=['i', 'also', 'much', 'would', 'by', 'another', 'could', 'thou', 'do']
for i in new_words:
stopwords.append(i)
final_text = [w for w in text_lower if not w in stopwords]
#instead of print(), use the return() command in this function
return(final_text)
There are other commonly used functions that you can leverage to clean up your primary source (or other) inputs. Many blog post share how to clean up strings and list, though not many have a history focus! The above steps are the ones I followed to create lists from historical text that would allow me to perform analysis on textual primary sources. See this post for a more complex discussion regarding functions that augmented my analysis of the gems and pearls in early modern history.
Happy coding!
