Complete Guide To Cracking MachineHack’s ‘Predict The News Category Hackathon'

Natural Language Processing (NLP) is one of the most explored and successful domains in machine learning. It is important because complex communication is one of the best signs of intelligence as we are trying to make machines communicate with humans effortlessly. In this article, we will do a hands-on NLP with Python to solve MachineHack’s Predict The News Category hackathon. 

Predict The News Category Hackathon 

MachineHack has launched its second Natural Language Processing challenge for its large Data Science and ML audience. The hackathon is about predicting the category or section of news from its content.The dataset consists of news pieces collected from a number of different sources along with the category or section of the news piece in which it was featured. Given below is the description of the dataset. Size of training set: 7,628 records Size of test set: 2,748 records FEATURES: STORY: A part of the main content of the article to be published as a piece of news. SECTION: The genre/category the STORY falls in. There are four distinct sections where each story may fall in to. The Sections are labelled as follows : Politics: 0 Technology: 1 Entertainment: 2 Business: 3

Getting The Datasets

Go to MachineHack, Sign Up as a user and click on the Predict The News Category Hackathon. Start the hackathon and find the dataset in the Attachment section. Click here to register for the hackathon  Without further ado, let's crack the Hackathon!

Solving The Hackathon

Let's break the solution into 6 parts as given below for better understanding.

1. Exploratory Data Analysis: A Simple analysis of Data 
2. Data cleaning 
3. Data preprocessing: Count Vectors and TF-IDF Vectors
4. Training the classifier
5. Predicting for the test set
6. Submitting your solution at MachineHack

Exploratory Data Analysis: A Simple analysis of Data

Let's start off with the usual drill and import all the necessary modules for our project. #Importing the libraries import pandas as pd import nltk from nltk.corpus import stopwords import string #Download the following modules once nltk.download('stopwords') nltk.download('wordnet') Let's do a simple analysis of the data in hand. #Importing the training set train_data = pd.read_excel("Datasets/Data_Train.xlsx") #Printing the top 5 rows print(train_data.head(5)) #Printing the dataset info print(train_data.info()) #Printing the shape of the dataset print(train_data.shape)] Out:(7628, 2) #Printing the group by description of each category train_data.groupby("SECTION").describe()

Data Cleaning 

#Removing duplicates to avoid overfitting train_data.drop_duplicates(inplace = True) #A punctuations string for reference (added other valid characters from the dataset) all_punctuations = string.punctuation + '‘’,:”][],'  #Method to remove punctuation marks from the data def punc_remover(raw_text): no_punct = "".join([i for i in raw_text if i not in all_punctuations]) return no_punct #Method to remove stopwords from the data def stopword_remover(no_punc_text): words = no_punc_text.split() no_stp_words = " ".join([i for i in words if i not in stopwords.words('english')]) return no_stp_words #Method to lemmatize the words in the data lemmer = nltk.stem.WordNetLemmatizer() def lem(words): return " ".join([lemmer.lemmatize(word,'v') for word in words.split()]) #Method to perform a complete cleaning def text_cleaner(raw): cleaned_text = stopword_remover(punc_remover(raw)) return lem(cleaned_text) #Testing the cleaner method text_cleaner("Hi!, this is a sample text to test the text cleaner method. Removes *@!#special characters%$^* and stopwords. And lemmatizes, go, going - run, ran, running") Out: 'Hi sample text test text cleaner method Removes special character stopwords And lemmatizes go go run run run' #Applying the cleaner method to the entire data train_data['CLEAN_STORY'] = train_data['STORY'].apply(text_cleaner) #Checking the new dataset print(train_data.values) 

Data Preprocessing: Count Vectors and TF-IDF Vectors

Creating Count vectors

#Importing sklearn’s Countvectorizer from sklearn.feature_extraction.text import CountVectorizer #Creating a bag-of-words dictionary of words from the data bow_dictionary = CountVectorizer().fit(train_data['CLEAN_STORY']) #Total number of words in the bow_dictionary len(bow_dictionary.vocabulary_) Out : 35189 #Using the bow_dictionary to create count vectors for the cleaned data. bow = bow_dictionary.transform(train_data['CLEAN_STORY']) #Printing the shape of the bag of words model print(bow.shape) Out: (7551, 35189)

Creating TF-IDF Vectors

#Importing TfidfTransformer from sklearn from sklearn.feature_extraction.text import TfidfTransformer #Fitting the bag of words data to the TF-IDF transformer tfidf_transformer = TfidfTransformer().fit(bow) #Transforming the bag of words model to TF-IDF vectors storytfidf = tfidf_transformer.transform(bow)

Training The Classifier

#Creating a Multinomial Naive Bayes Classifier from sklearn.naive_bayes import MultinomialNB #Fitting the training data to the classifier classifier = MultinomialNB().fit(storytfidf, train_data['SECTION'])

Predicting For The Test Set

#Importing and cleaning the test data test_data = pd.read_excel("Datasets/Data_Test.xlsx") test_data['CLEAN_STORY'] = test_data['STORY'].apply(text_cleaner) #Printing the cleaned data print(test_data.values)

Creating A Pipeline To Pre-Process The Data & Initialise The Classifier

#Importing the Pipeline module from sklearn from sklearn.pipeline import Pipeline #Initializing the pipeline with necessary transformations and the required classifier pipe = Pipeline([ ('bow', CountVectorizer()), ('tfidf', TfidfTransformer()), ('classifier', MultinomialNB())]) #Fitting the training data to the pipeline pipe.fit(train_data['CLEAN_STORY'], train_data['SECTION']) #Predicting the SECTION test_preds_mnb = pipe.predict(test_data['CLEAN_STORY']) #Writing the predictions to an excel sheet pd.DataFrame(test_preds_mnb, columns = ['SECTION']).to_excel("Predictions/predictions.xlsx")

Submitting Your Solution At MachineHack

Finally, head to MachineHack, and submit your excel fine at the Submission Deck of the hackathon.

• 1. Click on the Assignment

• 2. Browse to your file and select it.

 Note: Also provide a comment for your submission. Check your score on the Hackathon Leaderboard. The hackathon leaderboard will be updated within 2 minutes. And that's it. You have successfully found a solution. Now what's left to do is tweaking the model for performance. We will leave that part up to you. Tune the model, improve your accuracy, top the leaderboard and win exciting prizes. Happy Coding!