Market Data & Web Scraping Intermediate

Financial News Headline Scraper & Sentiment Polarity Scorer in Python

Scrape live financial news RSS feeds in Python using BeautifulSoup and score bullish/bearish sentiment polarity for stock tickers.

#requests#beautifulsoup4#pandas

Problem Statement

Unexpected macro or earnings headlines can invalidate technical setups in seconds. This script pulls live RSS headlines for any ticker and scores keyword sentiment polarity.

Formula / Core Logic

Parses XML nodes from Yahoo/Google Finance RSS and computes net lexicon score: (Bull_Words - Bear_Words) / Total_Matches.

Installation

pip install requests beautifulsoup4 pandas

scrape-financial-news-sentiment-analysis-python.py

import requests
from bs4 import BeautifulSoup
import pandas as pd

BULL_WORDS = {'beat', 'surge', 'upgrade', 'record', 'profit', 'growth', 'rally', 'buy', 'bullish', 'outperform'}
BEAR_WORDS = {'miss', 'plunge', 'downgrade', 'loss', 'crash', 'lawsuit', 'cut', 'sell', 'bearish', 'debt'}

def score_ticker_news(ticker='AAPL'):
    url = f'https://feeds.finance.yahoo.com/rss/2.0/headline?s={ticker}®ion=US&lang=en-US'
    resp = requests.get(url, headers={'User-Agent': 'Mozilla/5.0'}, timeout=10)
    soup = BeautifulSoup(resp.content, 'xml')
    rows = []
    for item in soup.find_all('item')[:15]:
        title = item.title.text
        words = set(title.lower().replace(',', '').replace('.', '').split())
        score = len(words & BULL_WORDS) - len(words & BEAR_WORDS)
        rows.append({'Headline': title, 'Sentiment_Score': score})
    return pd.DataFrame(rows)

Get the Complete 100+ Python Quant & Automation Bundle

Instant lifetime access to all production-ready scripts, valuation models, and trading bots ($19).

Buy Full Bundle on Gumroad ($19) →

Frequently Asked Questions

Q: Why use RSS XML instead of scraping HTML pages?

RSS feeds are lightweight, standardized XML endpoints that rarely change DOM classes or trigger CAPTCHA blocks.