School of Information Technology and Electrical Engineering

INFS3208 – Cloud Computing

Programming Assignment Task III (10 Marks)

Task description:

In this assignment, you are asked to write a piece of Spark code to count occurrences of verbs in the

UN debates and find the most similar debate contents. The returned result should be the top 10

verbs that are most frequently used in all debates and the debate that is most similar to the one

we provide. This assignment is to test your ability to use transformation and action operations in Spark

RDD programming and your understanding of Vector Database. You will be given three files,

including a UN General Debates dataset (un-general-debates.csv), a verb list (all_verbs.txt)

and a verb dictionary file (verb_dict.txt). These source files are expected to be stored in a HDFS.

You can choose either Scala or Python to complete this assignment in the Jupyter Notebook. There are

some technical requirements in your code submission as follows:

Objectives:

1. Read Source Files from HDFS and Create RDDs (1.5 marks):

? Read the UN General Debates dataset (un-general-debates.csv) from HDFS and

convert only the “text” column into an RDD. Details of un-general-debates.csv are

provided in the Preparation section below (1 mark).

 Read the verb list file (all_verbs.txt) and verb dictionary file (verb_dict.txt) from

HDFS and load them into separate RDDs (0.5 marks).

 Note: If you failed to read files from HDFS, you can still read them from the local file

system in work/nbs/ and complete the following tasks.

2. Use Learned RDD Operations to Preprocess the Debate Texts (3 marks):

 Remove empty lines (0.5 marks).

 Remove punctuations that could attach to the verbs (0.5 marks).

o E.g., “work,” and “work” will be counted differently, if you DO NOT remove the

punctuation.

 Change the capitalization or case of text (0.5 marks).

o E.g., “WORK”, “Work” and “work” will be counted as three different verbs, if you

DO NOT make all of them in lower-case.

Find all verbs in the RDD by matching the words in the given verb list (all_verbs.txt)

(0.5 mark).

 Convert all verbs in different tenses into the simple present tense by looking up the

verbs in the verb dictionary list (verb_dict.txt) (1 mark).

o E.g., regular verb: “work” - works”, “worked”, and “working”.

o E.g., irregular verb: “begin” - “begins”, “began”, and “begun”. o E.g., linking verb “be” and its various forms, including “is”, “am”, “are”, “was”,

“were”, “being” and “been”.

o E.g., (work, 100), (works,50), (working,150) should be counted as (work, 300).

3. Use learned RDD Operations to Count Verb Frequency (3 marks):

 Count the top 10 frequently used verbs in UN debates (2 marks).

 Display the results in the format (“verb1”, count1), (“verb2”, count2), … and in a

descending order of the counts (1 marks).