Introduction to the basics of NLP<\/h3>\n
Word2vec<\/a> is a group of related models that are used to produce word embeddings<\/strong><\/a>.\u00a0Word2vec takes as its input a large corpus of text and produces a\u00a0vector space, typically of several hundred\u00a0dimensions, with each unique word in the\u00a0corpus\u00a0being assigned a corresponding vector in the space.\u00a0Word vectors\u00a0are positioned in the vector space such that words that share common contexts in the corpus are located in close proximity to one another in the space [1].<\/p>\n Word embedding is the collective name for a set of language modelling and feature learning techniques in natural language processing (NLP) where words or phrases from the vocabulary are mapped to vectors of real numbers. Word and phrase embeddings, when used as the underlying input representation, have been shown to boost the performance in NLP tasks such as synthetic parsing and sentiment analysis.<\/p>\n Data was collected from various sources:<\/p>\n Download the text data of polish sentiment analysis <\/a>\u00a0form our Google Drive.<\/p>\n The polish word embeddings were\u00a0downloaded from the Polish Academy of Science<\/a>.<\/p>\n First of all, we will be defining all of the libraries and functions we will need:<\/p>\n Then load our dataset with simple preprocessing of data:<\/p>\n Split data into training (60%),\u00a0 test (20%) and validation (20%) set:<\/p>\n Print shape of X: train, test, validate and y: train, test, validate:<\/p>\n Load existing Polish Word2vec model taken from Polish Academy of Science:<\/p>\n Vectorize X_train and X_test to 2D tensor:<\/p>\n Long Short-Term Memory<\/a> networks \u2013 usually just called \u201cLSTMs\u201d \u2013 are a special kind of RNN, capable of learning long-term dependencies.<\/p>\n LSTMs are explicitly designed to avoid the long-term dependency problem. Remembering information for long periods of time is practically their default behaviour, not something they struggle to learn!\u00a0The key to LSTMs is the cell state, the horizontal line running through the top of the diagram.<\/p>\n The LSTM does have the ability to remove or add information to the cell state, carefully regulated by structures called gates.<\/p>\n We just need to compile the model and we will be ready to train it. When we compile the model, we declare the optimizer<\/strong><\/a> (Adam, SGD, etc.)\u00a0 and the loss function<\/strong><\/a>. To fit the model, all we have to do is declare the number of epochs <\/strong>and\u00a0the batch size<\/strong><\/a>.<\/p>\n The last step is to save our pre-trained model with word index. We can then use it later to predict new sentences in the future.<\/p>\n We can also control the overfitting<\/strong><\/a> using graphs. <\/p>\n As we can see in the charts above, the neural network learns quickly, getting good results.<\/p>\n Precision (also called positive predictive value) is the fraction of relevant instances among the retrieved instances.<\/p>\n Recall (also known as sensitivity) is the fraction of relevant instances that have been retrieved over the total amount of relevant instances.<\/p>\n Both precision and recall<\/a> are therefore based on an understanding and measure of relevance.<\/p>\n Confusion Matrix<\/a> also is known as an\u00a0error matrix<\/em>,\u00a0is a specific table layout that allows visualization of the performance of an algorithm, typically a\u00a0supervised learning<\/a>\u00a0one (in\u00a0unsupervised learning<\/a>\u00a0it is usually called a\u00a0matching matrix<\/em>). Each row of the matrix represents the instances in a predicted class while each column represents the instances in an actual class (or vice versa).<\/p>\n Word clouds (also known as text clouds or tag clouds) work in a simple way: the more a specific word appears in a source of textual data (such as a speech, blog post, or database), the bigger and bolder it appears in the word cloud.<\/p>\n <\/p>\n The ROC curve is created by plotting the\u00a0true positive rate<\/a>\u00a0(TPR)<\/strong> against the\u00a0false positive rate<\/a>\u00a0(FPR)<\/strong> at various threshold settings. The true-positive rate is also known as\u00a0sensitivity<\/a><\/strong>,\u00a0recall<\/a>.<\/strong><\/p>\n Test Accuracy: 97,65<\/strong>%<\/p>\n Test Loss: 6,56<\/strong>%<\/p>\n Recall score:\u00a00.976<\/strong>1942865880075<\/p>\n Precision score:\u00a00.975<\/strong>7310701772396<\/p>\n F1 score:\u00a00.975<\/strong>8853714533414<\/p>\n For epochs = 12; 220 minutes on MacBook Pro i7, 2.5GHz, 16 GB RAM, 512 GB SSD.<\/p>\n <\/p>\n After working through this post you learned:<\/p>\n You can download code from Ermlab GitHub repository<\/a>.<\/p>\n Go to README in project repository here<\/a>.<\/p>\n If you have any questions about the project or this post, please ask your question in the comments.<\/p>\n The post on the blog will be devoted to the analysis of sentimental Polish language, a problem in the category of natural language processing, implemented using machine learning techniques and recurrent neural networks.<\/p>\n","protected":false},"author":16,"featured_media":2554,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":[],"categories":[85],"tags":[87,86,18],"yoast_head":"\nWhy is sentiment analysis for the Polish language difficult?<\/h3>\n
\n
Data sources used for the project<\/h3>\n
\n
Let’s go to what we like the most – code…<\/h3>\n
import numpy as np\r\nimport pandas as pd\r\nimport matplotlib.pylab as plt\r\nfrom livelossplot import PlotLossesKeras\r\nnp.random.seed(7)\r\nfrom sklearn.model_selection import train_test_split\r\nfrom keras.preprocessing.text import Tokenizer\r\nfrom keras.models import Sequential\r\nfrom keras.layers import Dense, Dropout, Activation, LSTM\r\nfrom keras.layers.embeddings import Embedding\r\nfrom keras.utils import np_utils\r\nfrom keras.preprocessing import sequence\r\nfrom gensim.models import Word2Vec, KeyedVectors, word2vec\r\nimport gensim\r\nfrom gensim.utils import simple_preprocess\r\nfrom keras.utils import to_categorical\r\nimport pickle\r\nimport h5py\r\nfrom time import time<\/pre>\n
filename = 'Data\/Dataset.csv'\r\n\r\ndataset = pd.read_csv(filename, delimiter = \",\")\r\n\r\n# Delete unused column\r\ndel dataset['length']\r\n\r\n# Delete All NaN values from columns=['description','rate']\r\ndataset = dataset[dataset['description'].notnull() & dataset['rate'].notnull()]\r\n\r\n# We set all strings as lower case letters\r\ndataset['description'] = dataset['description'].str.lower()<\/pre>\n
X = dataset['description']\r\ny = dataset['rate']\r\n\r\nX_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)\r\n\r\nX_train, X_val, y_train, y_val = train_test_split(X_train, y_train, test_size=0.2, random_state=42)<\/pre>\n
print(\"X_train shape: \" + str(X_train.shape))\r\nprint(\"X_test shape: \" + str(X_test.shape))\r\nprint(\"X_val shape: \" + str(X_val.shape))\r\nprint(\"y_train shape: \" + str(y_train.shape))\r\nprint(\"y_test shape: \" + str(y_test.shape))\r\nprint(\"y_val shape: \" + str(y_val.shape))<\/pre>\n
word2vec_model = gensim.models.KeyedVectors.load_word2vec_format('nkjp.txt', binary=False)\r\n\r\nembedding_matrix = word2vec_model.wv.syn0\r\nprint('Shape of embedding matrix: ', embedding_matrix.shape)<\/pre>\ntop_words = embedding_matrix.shape[0]\r\nmxlen = 50\r\nnb_classes = 3\r\n\r\ntokenizer = Tokenizer(num_words=top_words)\r\ntokenizer.fit_on_texts(X_train)\r\nsequences_train = tokenizer.texts_to_sequences(X_train)\r\nsequences_test = tokenizer.texts_to_sequences(X_test)\r\nsequences_val = tokenizer.texts_to_sequences(X_val)\r\n\r\nword_index = tokenizer.word_index\r\nprint('Found %s unique tokens.' % len(word_index))\r\nprint(word_index)\r\n\r\nX_train = sequence.pad_sequences(sequences_train, maxlen=mxlen)\r\nX_test = sequence.pad_sequences(sequences_test, maxlen=mxlen)\r\nX_val = sequence.pad_sequences(sequences_val, maxlen=mxlen)\r\n\r\ny_train = np_utils.to_categorical(y_train, nb_classes)\r\ny_test = np_utils.to_categorical(y_test, nb_classes)\r\ny_val = np_utils.to_categorical(y_val, nb_classes)<\/pre>\nWe define our LSTM (Long Short-Term Memory)<\/h4>\n
batch_size = 32\r\nnb_epoch = 12\r\n\r\nembedding_layer = Embedding(embedding_matrix.shape[0],\r\n embedding_matrix.shape[1],\r\n weights=[embedding_matrix],\r\n trainable=False)\r\n\r\nmodel = Sequential()\r\nmodel.add(embedding_layer)\r\nmodel.add(LSTM(128, recurrent_dropout=0.5, dropout=0.5))\r\nmodel.add(Dense(nb_classes))\r\nmodel.add(Activation('softmax'))\r\nmodel.summary()<\/pre>\nmodel.compile(optimizer='adam', loss='categorical_crossentropy', metrics=['accuracy'])\r\nrnn = model.fit(X_train, Y_train, nb_epoch= nb_epoch, batch_size=batch_size, shuffle=True, validation_data=(X_val, y_val))\r\nscore = model.evaluate(X_val, y_val)\r\nprint(\"Test Loss: %.2f%%\" % (score[0]*100))\r\nprint(\"Test Accuracy: %.2f%%\" % (score[1]*100))<\/pre>\n
print('Save model...')\r\nmodel.save('Models\/finalsentimentmodel.h5')\r\nprint('Saved model to disk...')\r\n\r\nprint('Save Word index...')\r\noutput = open('Models\/finalwordindex.pkl', 'wb')\r\npickle.dump(word_index, output)\r\nprint('Saved word index to disk...')<\/pre>\n
\nPlots for training and testing process (loss and accuracy):<\/p>\nplt.figure(0)\r\nplt.plot(rnn.history['acc'],'r')\r\nplt.plot(rnn.history['val_acc'],'g')\r\nplt.xticks(np.arange(0, nb_epoch+1, nb_epoch\/5))\r\nplt.rcParams['figure.figsize'] = (8, 6)\r\nplt.xlabel(\"Num of Epochs\")\r\nplt.ylabel(\"Accuracy\")\r\nplt.title(\"Training vs Validation Accuracy LSTM l=10, epochs=20\") # for max length = 10 and 20 epochs\r\nplt.legend(['train', 'validation'])\r\n\r\nplt.figure(1)\r\nplt.plot(rnn.history['loss'],'r')\r\nplt.plot(rnn.history['val_loss'],'g')\r\nplt.xticks(np.arange(0, nb_epoch+1, nb_epoch\/5))\r\nplt.rcParams['figure.figsize'] = (8, 6)\r\nplt.xlabel(\"Num of Epochs\")\r\nplt.ylabel(\"Training vs Validation Loss LSTM l=10, epochs=20\") # for max length = 10 and 20 epochs\r\nplt.legend(['train', 'validation'])\r\nplt.show()<\/pre>\n
![](\"https:\/\/ermlab.com\/wp-content\/uploads\/2018\/02\/epoch12a.png\")
<\/p>\n![](\"https:\/\/ermlab.com\/wp-content\/uploads\/2018\/02\/epoch12b.png\")
<\/p>\nApply Precision-Recall<\/h4>\n
# Apply Precision-Recall\r\n\r\nfrom sklearn.metrics import recall_score\r\nfrom sklearn.metrics import precision_score\r\nfrom sklearn.metrics import f1_score\r\n\r\ny_pred = model.predict(X_test)\r\n\r\n# Convert Y_Test into 1D array\r\nyy_true = [np.argmax(i) for i in Y_test]\r\nprint(yy_true)\r\n\r\nyy_scores = [np.argmax(i) for i in y_pred]\r\nprint(yy_scores)\r\n\r\nprint(\"Recall: \" + str(recall_score(yy_true, yy_scores, average='weighted')))\r\nprint(\"Precision: \" + str(precision_score(yy_true, yy_scores, average='weighted')))\r\nprint(\"F1 Score: \" + str(f1_score(yy_true, yy_scores, average='weighted')))<\/pre>\n
Apply and Visualizing of the confusion matrix<\/h4>\n
# Apply Confusion matrix\r\n\r\nfrom sklearn.metrics import classification_report, confusion_matrix\r\nY_pred = model.predict(X_test, verbose=2)\r\ny_pred = np.argmax(Y_pred, axis=1)\r\n\r\nfor ix in range(3):\r\n print(ix, confusion_matrix(np.argmax(Y_test, axis=1), y_pred)[ix].sum())\r\ncm = confusion_matrix(np.argmax(Y_test, axis=1), y_pred)\r\nprint(cm)\r\n\r\n# Visualizing of confusion matrix\r\nimport seaborn as sn\r\n\r\ndf_cm = pd.DataFrame(cm, range(3), range(3))\r\nplt.figure(figsize=(10,7))\r\nsn.set(font_scale=1.4)\r\nsn.heatmap(df_cm, annot=False)\r\nsn.set_context(\"poster\")\r\nplt.xlabel(\"Predicted Label\")\r\nplt.ylabel(\"True Label\")\r\nplt.title(\"Confusion Matrix\")\r\nplt.savefig('Plots\/confusionMatrix.png')\r\nplt.show()<\/pre>\n![](\"https:\/\/ermlab.com\/wp-content\/uploads\/2018\/02\/confusionmatrix.png\")
<\/p>\nWordcloud<\/h4>\n
from wordcloud import WordCloud\r\nfrom many_stop_words import get_stop_words\r\n\r\nstop_words = get_stop_words('pl')\r\n\r\nwordcloud = WordCloud(\r\n background_color='white',\r\n stopwords=stop_words,\r\n max_words=200,\r\n max_font_size=40,\r\n random_state=42\r\n).generate(str(dataset['description']))\r\n\r\nprint(wordcloud)\r\nfig = plt.figure(1)\r\nplt.imshow(wordcloud)\r\nplt.axis('off')\r\nplt.show()<\/pre>\n![](\"https:\/\/ermlab.com\/wp-content\/uploads\/2018\/03\/wc.png\")
\nThe word cloud above shows the most common words in our dataset.<\/p>\n# ROC Curve\r\n\r\nfrom sklearn.metrics import roc_curve, auc\r\nfrom scipy import interp\r\nfrom itertools import cycle\r\n\r\n# Compute ROC curve and ROC area for each class\r\n\r\nn_classes = 3\r\nlw = 2\r\nfpr = dict()\r\ntpr = dict()\r\nroc_auc = dict()\r\n\r\nfor i in range(n_classes):\r\n fpr[i], tpr[i], _ = roc_curve(np.array(pd.get_dummies(yy_true))[:, i], np.array(pd.get_dummies(yy_scores))[:, i])\r\n roc_auc[i] = auc(fpr[i], tpr[i])\r\n\r\n\r\n# Compute micro-average ROC curve and ROC area\r\nfpr[\"micro\"], tpr[\"micro\"], _ = roc_curve(np.array(pd.get_dummies(yy_true))[:, i], np.array(pd.get_dummies(yy_scores))[:, i])\r\nroc_auc[\"micro\"] = auc(fpr[\"micro\"], tpr[\"micro\"])\r\n\r\n# Compute macro-average ROC curve and ROC area\r\n\r\n# First aggregate all false positive rates\r\nall_fpr = np.unique(np.concatenate([fpr[i] for i in range(n_classes)]))\r\n\r\n# Then interpolate all ROC curves at this points\r\nmean_tpr = np.zeros_like(all_fpr)\r\nfor i in range(n_classes):\r\n mean_tpr += interp(all_fpr, fpr[i], tpr[i])\r\n\r\n# Finally average it and compute AUC\r\nmean_tpr \/= n_classes\r\n\r\nfpr[\"macro\"] = all_fpr\r\ntpr[\"macro\"] = mean_tpr\r\nroc_auc[\"macro\"] = auc(fpr[\"macro\"], tpr[\"macro\"])\r\n\r\n# Plot all ROC curves\r\nplt.figure(figsize=(8,5))\r\nplt.plot(fpr[\"micro\"], tpr[\"micro\"],\r\n label='micro-average ROC curve (area = {0:0.2f})'\r\n ''.format(roc_auc[\"micro\"]),\r\n color='deeppink', linestyle=':', linewidth=4)\r\nplt.plot(fpr[\"macro\"], tpr[\"macro\"],\r\n label='macro-average ROC curve (area = {0:0.2f})'\r\n ''.format(roc_auc[\"macro\"]),\r\n color='green', linestyle=':', linewidth=4)\r\n\r\ncolors = cycle(['aqua', 'darkorange', 'cornflowerblue'])\r\nfor i, color in zip(range(n_classes), colors):\r\n plt.plot(fpr[i], tpr[i], color=color, lw=lw,\r\n label='ROC curve of class {0} (area = {1:0.2f})'\r\n ''.format(i, roc_auc[i]))\r\n\r\nplt.plot([0, 1], [0, 1], 'k--',color='red', lw=lw)\r\nplt.xlim([0.0, 1.0])\r\nplt.ylim([0.0, 1.05])\r\nplt.xlabel('False Positive Rate')\r\nplt.ylabel('True Positive Rate')\r\nplt.title('Receiver Operating Characteristic')\r\nplt.legend(loc=\"lower right\")\r\nplt.savefig('Plots\/ROCcurve.png')\r\nplt.show()<\/pre>\n![](\"https:\/\/ermlab.com\/wp-content\/uploads\/2018\/03\/roc3.png\")
<\/p>\nResults<\/h3>\n
Some statistics<\/h3>\n
\n
![](\"https:\/\/ermlab.com\/wp-content\/uploads\/2018\/02\/histogram.png\")
<\/p>\n\n
Summary<\/h3>\n
\n
How to run Our project?<\/h3>\n
Resources<\/h3>\n
\n