The below link for files:
https://send.firefox.com/download/55d0645d54c7d27a/#nFxD5K_kxqYucDTaj6QJpg
In this exercise, you will write a script called "docdistances" that will calculate distances between pairs of text documents. These distances will be based on a vanilla version of term frequency-inverse document frequency (tf-idf). Your script will calculate the distances between 6 documents: 3 documents are synopses of fairy tales (Red Riding Hood, The Princess and the Pea, and Cinderella); the other 3 documents are the abstracts of papers related to protein function prediction (identified as CAFA1, CAFA2, and CAFA3). You will find these documents on the Moodle page (the file names are: RedRidingHood.txt, PrincessPea.txt, Cinderella.txt, CAFAl.txt, CAFA2.txt, CAFA3.txt).
Your script will:
1. For each document, calculate its tf-idf vector. The tf-idf vector of a document is a vector whose length is equal to the total number of different terms (words) that are present in the corpus (in this case, the corpus is the entire set of 6 documents). Each term is assigned a specific element of the vector, which is in the same position for the tf-idf vector of every document. For a given document d, the vector element corresponding to term t is calculated as the product of 2 values:
a) Term frequency: the number of times that term t appears in document d
b) Inverse document frequency: the logarithm base 10 of the inverse fraction of the documents that contain the term, i.e. number of documents in the corpus
2. Calculate the cosine distance between every pair of tf-idf vectors representing each document. (This is equal to 1 minus the cosine of the angle between the 2 vectors.)
3. Collect these distances into a 6x6 matrix where the value in the (i,j) element contains the distance between document i and document j. Then make a figure that displays the matrix. Your figure should look similar to the figure below (here I have used imagesc and set the colormap to gray):
Cosine Distance
RRH
0.9
0.8
PPea
0.7
Cinde
0.6
0.5
CAFA1
0.4
0.3
CAFA2
0.2
CAFA3
0.1
RRH PPea Cinde CAFA1 CAFA2 CAFA3
0
