Effective Python Keyword Detection Techniques

You can detect keywords in a given text with Python using various techniques, starting from basic string operations to advanced methods. Basic techniques include direct matching, splitting and comparing text, which have limitations such as case sensitivity and lack of semantic understanding. For more robust detection, libraries like NLTK and spaCy are recommended, incorporating tokenization, stop word removal, and stemming. Advanced methods like TF-IDF, RAKE, YAKE!, and topic modeling provide deeper insights into keyword extraction. Each approach has its use case, from predefined keywords to identifying important terms based on frequency.

Note: The codes for this post are available on Colab.

Basic Techniques

1. Basic String Operations (Simple but Limited):

• Direct Matching: You can directly check if specific words or phrases exist in the text using the in operator or string methods like find() or count(). This is suitable if you have a predefined list of keywords.

text = "This document discusses Python programming and its applications in data science."
keywords = ["Python", "data science", "programming"]

found_keywords = [keyword for keyword in keywords if keyword.lower() in text.lower()]
print(f"Found keywords: {found_keywords}")

Output:

Found keywords: ['Python', 'data science', 'programming']

• Splitting and Comparing: You can split the text into words and compare them against your keyword list. Remember to handle case sensitivity and punctuation.

import re

text = "Python is a powerful programming language."
keywords = ["python", "language"]

words = re.findall(r'\b\w+\b', text.lower())  # Extract words using regex
found_keywords = [keyword for keyword in keywords if keyword in words]
print(f"Found keywords: {found_keywords}")

• Output:

Found keywords: ['python', 'language']

Limitations of Basic String Operations:

• Case Sensitivity: You need to handle different capitalizations.
• Punctuation: Punctuation marks can prevent exact matches.
• Word Variations: It won’t detect variations of words (e.g., “program” vs. “programming”).
• Semantic Meaning: It doesn’t understand the meaning of words or synonyms.

2. Using Libraries for Text Processing:

For more robust keyword detection, libraries like nltk (Natural Language Toolkit) and spaCy are highly recommended.

• NLTK (Natural Language Toolkit):
  In the following code, we will:
  • Tokenization: Splits the text into individual words (tokens).
  • Stop Word Removal: Removes common words like “the,” “is,” “a,” which usually don’t carry significant meaning.
  • Stemming/Lemmatization: Reduces words to their base or root form (e.g., “running” becomes “run,” “data” remains “data”). Lemmatization is generally more accurate but computationally more intensive.

import nltk
from nltk.tokenize import word_tokenize
from nltk.corpus import stopwords
from nltk.stem import PorterStemmer  # or other stemmers/lemmatizers

nltk.download('punkt', quiet=True)
nltk.download('punkt_tab', quiet = True)
nltk.download('stopwords', quiet=True)

text = "Analyzing text data with Python is a key skill for data scientists."
keywords = ["Python", "data", "science", "analyze"]

# Tokenize the text
tokens = word_tokenize(text.lower())

# Remove stop words
stop_words = set(stopwords.words('english'))
filtered_tokens = [w for w in tokens if not w in stop_words and w.isalnum()]

# Stemming (reducing words to their root form)
porter = PorterStemmer()
stemmed_tokens = [porter.stem(w) for w in filtered_tokens]

# Stem the keywords for comparison
stemmed_keywords = [porter.stem(k.lower()) for k in keywords]

found_keywords = [keywords[i] for i, sk in enumerate(stemmed_keywords) if sk in stemmed_tokens]
print(f"Found keywords (using NLTK): {found_keywords}")

• Output:

Found keywords (using NLTK): ['Python', 'data', 'analyze']

• spaCy: spaCy uses pre-trained language models to understand the context and linguistic features of the text.
  • Tokenization: Similar to NLTK, it splits the text into tokens.
  • Stop Word Removal: Identifies and removes stop words.
  • Lemmatization: Reduces words to their base form (lemma).
  • Part-of-Speech Tagging (Implicit): spaCy’s language model helps in more accurate lemmatization.

import spacy

nlp = spacy.load("en_core_web_sm")

text = "The fast brown fox jumped over the lazy dogs using Python."
keywords = ["fox", "dog", "Python", "jump"]

doc = nlp(text)
text_tokens = [token.lemma_.lower() for token in doc if not token.is_stop and token.is_alpha]
keyword_tokens = [nlp(keyword)[0].lemma_.lower() for keyword in keywords]

found_keywords = [keywords[i] for i, kw in enumerate(keyword_tokens) if kw in text_tokens]
print(f"Found keywords (using spaCy): {found_keywords}")

Output:

Found keywords (using spaCy): ['fox', 'dog', 'Python', 'jump']

More Advanced Techniques:

• TF-IDF (Term Frequency-Inverse Document Frequency): This technique assigns weights to words based on their frequency in a document and their inverse frequency across a collection of documents. Words with high TF-IDF scores are often considered important keywords.

from sklearn.feature_extraction.text import TfidfVectorizer

documents = [
    "This is the first document about Python.",
    "This document is the second document about the Python programming language.",
    "And this is the third document on Python data science."
]

vectorizer = TfidfVectorizer()
tfidf_matrix = vectorizer.fit_transform(documents)

# To get keywords for a specific document (e.g., the first one):
feature_names = vectorizer.get_feature_names_out()
doc_index = 0
tfidf_scores = tfidf_matrix[doc_index].toarray()[0]

# Get top N keywords
top_n = 3
top_indices = tfidf_scores.argsort()[-top_n:][::-1]
top_keywords = [feature_names[i] for i in top_indices]

print(f"Top {top_n} keywords for document 1: {top_keywords}")

• Output:

Top 3 keywords for document 1: ['first', 'about', 'this']

• Rake (Rapid Automatic Keyword Extraction): RAKE is a domain-independent keyword extraction algorithm that identifies keywords by analyzing the frequency of word occurrences and their co-occurrence within phrases. There are Python implementations available (e.g., rake-nltk).

# !pip install rake-nltk

from rake_nltk import Rake
import nltk
nltk.download('punkt_tab')
nltk.download('stopwords')

text ="Topic modeling is useful for understanding the underlying themes and identifying related keywords."

r = Rake()
r.extract_keywords_from_text(text)
print(r.get_ranked_phrases())  # Get extracted keywords

• Output:

['identifying related keywords', 'underlying themes', 'topic modeling', 'useful', 'understanding']

• YAKE! (Yet Another Keyword Extractor): YAKE! is a lightweight unsupervised keyword extraction method that relies on statistical features of words within the text. It’s often effective and doesn’t require training on large corpora. You can use the yake library in Python.

!pip install yake
import yake

text = "Artificial intelligence (AI) is the simulation of intelligence in machines that are programmed to think like humans and mimic their actions. The term may also be applied to any machine that exhibits traits associated with a human mind such as learning and problem-solving."

kw_extractor = yake.KeywordExtractor()
keywords = kw_extractor.extract_keywords(text)

print("Keywords extracted by YAKE!:", keywords[:5])

Output:

Keywords extracted by YAKE!: [('mimic their actions', np.float64(0.04131246820716694)), ('Artificial intelligence', np.float64(0.05441960865889508)), ('Artificial', np.float64(0.15831692877998726)), ('actions', np.float64(0.15831692877998726)), ('intelligence', np.float64(0.16299886812352193))]

YAKE’s keyword extraction output consists of tuples with two elements:

• The score represents the importance or relevance of the keyword, but in YAKE’s case, lower scores indicate higher relevance (the most important keywords will have the smallest scores).
• The score is computed based on statistical features like term frequency, word position, and how distinct the term is within the text.

In this example:

1. "mimic their actions" (Score: 0.041) → Most relevant phrase
2. "Artificial intelligence" (Score: 0.054) → Also important
3. "Artificial" (Score: 0.158) → Less relevant as a standalone word
4. "actions" (Score: 0.158) → Similar to “Artificial” in relevance
5. "intelligence" (Score: 0.162) → Least relevant in this set

• Topic Modeling (e.g., LDA – Latent Dirichlet Allocation): While not strictly keyword detection, topic modeling can identify the main topics discussed in a text, and the most frequent and relevant words within those topics can be considered keywords. Libraries like gensim are used for topic modeling.

Choosing the Right Method:

• For a small, predefined set of keywords, basic string operations might suffice.
• For more general keyword extraction, especially when you need to handle variations and common words, nltk or spaCy with stop word removal and stemming/lemmatization are good starting points.
• If you want to identify the most important terms based on their frequency and distribution across documents, TF-IDF is a powerful technique.
• RAKE and YAKE! are effective unsupervised methods that don’t require large datasets.
• Topic modeling is useful for understanding the underlying themes and identifying related keywords.