// Listen for the removal of the full notebook cells var notebookContainer = gd.closest(‘#notebook-container’); if (notebookContainer) {{ x.observe(notebookContainer, {childList: true}); }}

// Listen for the clearing of the current output cell var outputEl = gd.closest(‘.output’); if (outputEl) {{ x.observe(outputEl, {childList: true}); }}

                    })                };                });            </script>        </div>

Part 4: Data Analysis

Machine Learning (Topic Modeling)

Topic modeling is a type of statistical modeling for discovering the abstract topics that occur in a collection of documents. Essentially, it is a method for finding a group of words (i.e., a topic) from a collection of documents that best represents the information in the collection. It’s a form of text mining - a way to identify patterns in a large collection of text data, which we often refer to as a “corpus”.

Latent Dirichlet Allocation (LDA)

Latent Dirichlet Allocation (LDA) is an example of a topic model and is used to classify text in a document to a particular topic. It builds a topic per document model and words per topic model, modeled as Dirichlet distributions.

The LDA model discovers the different topics that the documents represent and how much of each topic is present in a document. In essence, it assumes that each document in a corpus is a combination of a finite number of topics, and each word in the document can be attributed to one of the document’s topics.

Why Use LDA for Analyzing the New York Times Articles on Special Purpose Acquisition Companies (SPACs)?

As we have a large number of articles from the New York Times relating to SPACs, it becomes nearly impossible to read through every article and understand the key themes or topics manually. That’s where the LDA model becomes particularly useful.

Using LDA for this task helps us automatically discover the main topics discussed in these articles, providing a high-level overview of the common themes. By applying the LDA model, we can find out which topics were most prevalent over time, helping us understand how the discussion around SPACs has evolved.

It can also highlight the context in which SPACs are being discussed, such as in relation to financial regulations, market trends, or specific industries. This information can be crucial in providing a more structured and deep understanding of the news corpus.

In conclusion, LDA is a powerful tool for analyzing large collections of text data, uncovering hidden thematic structures, and helping in the understanding of large-scale trends and patterns. By using LDA in conjunction with Python’s Scikit-Learn library, we can effectively analyze and interpret our corpus of New York Times articles.

Resources

• Topic Modeling with LDA
• Latent Dirichlet Allocation (LDA)

from sklearn.feature_extraction.text import CountVectorizer
from sklearn.decomposition import LatentDirichletAllocation
from sklearn.feature_extraction.text import ENGLISH_STOP_WORDS


# Combine the abstract and lead_paragraph columns into one column
full_df['text'] = full_df['abstract'] + '\n' + full_df['lead_paragraph']

# Want to exclude "New York City" from the LDA model along with the default english words
custom_stop_words = list(ENGLISH_STOP_WORDS)
custom_stop_words.extend(['new', 'york', 'city'])

# Transform the 'abstract' column into a document-term matrix
vectorizer = CountVectorizer(max_df=0.95, min_df=2, stop_words=custom_stop_words)
dtm = vectorizer.fit_transform(full_df['text'])

dtm.shape

(2290, 10322)

# Train LDA model
# Note: You might want to tune the number of topics (n_components parameter)
lda = LatentDirichletAllocation(n_components=10, random_state=42)
lda.fit(dtm)

LatentDirichletAllocation(random_state=42)

LatentDirichletAllocation(random_state=42)

topic_results = lda.transform(dtm)

top_words = 5  # Number of top words in each topic

feature_names = vectorizer.get_feature_names_out()

# For each document, get the top words in each of its top topics
top_words_for_each_doc = []
top_topic_for_each_doc = []
top_topic_prob_for_each_doc = []

for prob_dist in topic_results:
    # Get the probability of the top topic for this document
    top_topic_prob = np.sort(prob_dist)[-1]
    top_topic_prob_for_each_doc.append(top_topic_prob)
    # Get the top topic for this document
    top_topic = np.argsort(prob_dist)[-1]
    top_topic_for_each_doc.append(top_topic)
    # Get the top words for this topic
    top_words_for_topic = np.argsort(lda.components_[top_topic])[::-1][:top_words]
    top_words_for_each_doc.append([feature_names[i] for i in top_words_for_topic])

full_df['main_topic_index'] = top_topic_for_each_doc
full_df['main_topic_prob'] = top_topic_prob_for_each_doc
full_df['main_topic'] = top_words_for_each_doc
full_df['main_topic_str'] = full_df['main_topic'].apply(lambda x: ', '.join(x))

# Some of the entries have empty of not useful abstracts and lead paragraphs
# Remove those entries
clean_df = full_df[full_df['lead_paragraph'].notna()] # Remove rows where 'lead_paragraph' is NaN
clean_df = clean_df[clean_df['lead_paragraph'].str.strip() != ''] # Remove rows where 'lead_paragraph' is whitespace or empty string
clean_df = clean_df[clean_df['abstract'].str.split().str.len() >= 3]  # Keep rows where 'abstract' has at least three words

clean_df.shape

(1710, 25)

# Print a random sample of 5 articles and their main topic
for abstract, lead_paragraph, main_topic_prob, main_topic in clean_df[['abstract', 'lead_paragraph', 'main_topic_prob', 'main_topic']].sample(n=5).values:
    print("Abstract: %s" % abstract)
    print("Lead Paragraph: %s" % lead_paragraph)
    print("Main Topics: %s" % main_topic)
    print("Main Topic Probability: %s" % main_topic_prob)
    print()

Abstract: LEAD: Robert Maxwell comes lum-bering toward the waiting helicopter.
Lead Paragraph: Robert Maxwell comes lum-bering toward the waiting helicopter.
Main Topics: ['company', 'acquisition', 'deal', 'companies', 'said']
Main Topic Probability: 0.5209040459869437

Abstract:   "It will probably be the most interesting thing we have ever done," says Jules Kroll.   The world's most famous gumshoe is matter-of-fact about his company's latest major international assignment: the Republic of Russia has hired it to locate the money -- an estimated $6 billion to $8 billion in 1991 alone -- believed to have been spirited out of the country by the directors of state enterprises when they realized that privatization was inevitable. The Russian Government wants Kroll to get the money back.
Lead Paragraph: "It will probably be the most interesting thing we have ever done," says Jules Kroll.
Main Topics: ['com', 'designer', 'style', 'slide', 'times']
Main Topic Probability: 0.8852844100115668

Abstract: Nature Conservancy acquires 1,300-acre tract of beaches, dunes and rare wetlands on LI south fork near Hither Hills State Pk for $8.5-million; plans to sell tract to NYS at no profit; property was purchased from Hanson Properties Inc and Atlantic Processing Co; orgn spokesman says group borrowed money at special low-interest rates from Citibank; rare flora and fauna indigenous to tract noted; area map (M)
Lead Paragraph: PORT JEFFERSON L.I., Dec. 29—New York State will soon acquire a 1,300‐acre area of beaches, dunes and rare wetlands on Long Island's South Fork.
Main Topics: ['text', 'wall', 'street', 'says', 'state']
Main Topic Probability: 0.9849968701836254

Abstract: The Treasury secretary said that borrowing costs remained high despite government efforts.
Lead Paragraph: Despite huge government efforts to restore lending to normal, Treasury Secretary Timothy F. Geithner said Tuesday that borrowing costs remained high and credit was still not flowing normally.
Main Topics: ['bank', 'president', 'trump', 'media', 'social']
Main Topic Probability: 0.9709643287303419

Abstract: Following is the full text of the principal findings of the special committee investigating financial improprieties among Hollinger International Inc. executives. The report was filed today with the Securities and Exchange Commission.
Lead Paragraph:  Following is the full text of the principal findings of the special committee investigating financial improprieties among Hollinger International Inc. executives. The report was filed today with the Securities and Exchange Commission. 
Main Topics: ['company', 'acquisition', 'deal', 'companies', 'said']
Main Topic Probability: 0.49392439808855293

# Find the most common topics and their average probabilities
count_topic_df = clean_df \
    .groupby('main_topic_str') \
    .count()[['_id']] \
    .reset_index() \
    .rename(columns={'_id': 'Count', 'main_topic_str': 'Topic'}) \
    .sort_values('Count', ascending=False)

avg_topic_prob_df = clean_df \
    .groupby('main_topic_str') \
    .mean(numeric_only=True)[['main_topic_prob']] \
    .reset_index() \
    .rename(columns={'main_topic_prob': 'Avg. Probability', 'main_topic_str': 'Topic'}) \

count_topic_df = count_topic_df.merge(avg_topic_prob_df, on='Topic')

count_topic_df

As the above table shows, extracted from the article abstract and leading paragraphs, the topic all the most popular topics are related to business terms like “company,” “money,” and “stock.” The highest ranking topic with 467 articles that best fit that topic has the words “company, acquisition, deal, companies, said.” These words strongly match the phrase special purpose acquisition company, giving good confidence to the quality of our data.

NLP (Sentiment Analysis)

What is Natural Language Processing?

Natural Language Processing (NLP) is a subfield of artificial intelligence and computational linguistics that focuses on the interaction between computers and humans through natural language. The goal of NLP is to enable computers to understand, interpret, and generate human language in a valuable way.

NLP involves several tasks, including but not limited to:

• Text Analysis: Understanding the structure of the language, including grammar and syntax.
• Semantic Analysis: Understanding the meaning of the text.
• Sentiment Analysis: Determining the sentiment or emotion expressed in the text.
• Text Generation: Creating meaningful text based on certain inputs or requirements.

NLP in Our Project

It’s one thing to have a lot of articles published on a topic. It’s another thing to know if there articles are positive or negative. While some say that all publicity is good publicity, we are interested in finding out more about public awareness and public opinion on SPACs. Whether these articles have positive or negative sentiment to them matters for this case.

One of the columns of our NYT articles dataframe is web_url which contains the link to the published article on the New York Times website. In theory, we could request the HTML content of all these web pages, scrape the web page so we are only getting the text content, and then running NLP on the text for each article. In practice, this proves to be quite difficult.

1. Our data frame has roughly 2,500 rows, meaning we’d have to scrape 2,500 websites. Making that many requests to the New York Times would likely cause a rate limit.
2. The New York Times puts all its articles behind a paywall and does not load the article’s content unless its reader logs in.
3. Not all New York Times articles are guaranteed to put their article content in the same HTML tag

Resources

• What is Natural Language Processing?
• NLTK Docs

test_web_url = full_df.sort_values('pub_date', ascending=False).iloc[0]['web_url']
test_web_url

'https://www.nytimes.com/2023/01/20/business/media/vice-puts-itself-up-for-sale.html'

res = requests.get(test_web_url)
res.content

b'<html><head><title>nytimes.com</title><style>#cmsg{animation: A 1.5s;}@keyframes A{0%{opacity:0;}99%{opacity:0;}100%{opacity:1;}}</style></head><body style="margin:0"><p id="cmsg">Please enable JS and disable any ad blocker</p><script data-cfasync="false">var dd={\'rt\':\'c\',\'cid\':\'AHrlqAAAAAMAl5sIanRzcdYAgQJZKA==\',\'hsh\':\'499AE34129FA4E4FABC31582C3075D\',\'t\':\'bv\',\'s\':17439,\'e\':\'640143223d3583113180365187720914bb64b245f82387444835d77e1b96a39f\',\'host\':\'geo.captcha-delivery.com\'}</script><script data-cfasync="false" src="https://ct.captcha-delivery.com/c.js"></script></body></html>\n'

soup = BeautifulSoup(res.content, 'html.parser')
soup.find_all(string="Please enable JS and disable any ad blocker")

['Please enable JS and disable any ad blocker']

# The new york times publishes text content in the <article> tag
soup.find("article") is None

True

Since we can’t easily make HTTP requests to scrape NYT article text content, we should look to our DataFrame for columns which we can do NLP on. We’ll use the text column that we created for the Topic Modeling which is the combination of the abstract and lead paragraph.

import nltk
nltk.download('vader_lexicon')

[nltk_data] Downloading package vader_lexicon to
[nltk_data]     /Users/work/nltk_data...
[nltk_data]   Package vader_lexicon is already up-to-date!





True

from nltk.sentiment.vader import SentimentIntensityAnalyzer
sid = SentimentIntensityAnalyzer()

def get_sentiment(text):
    # Get the sentiment scores
    sentiment_scores = sid.polarity_scores(text)
    # Return the compound score
    return sentiment_scores['compound']

clean_df['sentiment'] = full_df['text'].apply(get_sentiment)

import plotly.graph_objects as go

fig = px.histogram(
    clean_df, 
    x='sentiment', 
    color_discrete_sequence=['lightblue'], 
    opacity=0.8,
    title='Sentiment Score Distribution of SPAC NYT Articles',
    labels={'sentiment':'Sentiment Score', 'count': 'Proportion'},
)
fig.update_layout(bargap=0.05)
fig.show()

                    })                };                });            </script>        </div>