Project Overview
S.T.E.M (Stock Market Sentiment Analyzer) is an ML pipeline that matches qualitative news with market movements. The system crawls Wikipedia summaries for target companies, scores the sentiment using VADER, downloads historical Yahoo Finance stock prices, aligns them by date, and trains a Logistic Regression classifier to predict next-day market movements.
Problem & Motivation
The Problem
Analyzing market sentiment usually requires expensive news feeds and complex deep-learning setups. S.T.E.M was designed to test if a lightweight, open-source pipeline using logistic regression could yield high-confidence direction indicators from public wiki logs.
The Motivation
Financial markets react strongly to news cycles, but manual news triage is slow. S.T.E.M was built to explore if a simple, lightweight sentiment analysis pipeline could accurately predict qualitative trend directions using free, public news metadata from Wikipedia summaries.
System Architecture
A Python pipeline running a daily update task. The data collection module fetches stock indices (yfinance) and Wikipedia articles. The sentiment module runs NLP scoring. The data layer merges dates and fits a Scikit-Learn logistic classifier. A Streamlit web dashboard visualizes prediction probabilities and metrics.
Wikipedia summary parsing utilizing requests and beautifulsoup crawler frameworks.
Sentiment feature engineering using NLTK VADER (Valence Aware Dictionary and Sentiment Reasoner).
Historical data alignment using Pandas merge_asof joining sentiment dates with market opening times.
Daily automation pipeline running yfinance downloads and model refits at 6 PM.
Key Features & Capabilities
Automated Data Fetching
Daily stock and news indexing pipelines.
VADER NLP Sentiment Pipeline
Converts text logs into continuous sentiment scores.
Logistic Regression Predictor
Fits sentiment scores to binary stock movement targets.
Streamlit Visualizer
Plots stock graphs alongside model confidence values.
Engineering Challenges
Noisy Daily Sentiment Data
Wikipedia text updates are irregular, causing daily sentiment metrics to spikes erratically, leading to poor model fitting and high prediction variance.
Implemented a 7-day rolling window sentiment smoothing check. Instead of matching single-day scores, the model fits the average sentiment of the previous 7 days, smoothing noise and boosting classifier accuracy to 58%.
Development Timeline
Began modeling Wikipedia sentiment crawling pipelines.
Trained logistic regression classifiers and exported model.joblib.
Completed Streamlit UI and automated schedulers.
Lessons Learned
- Financial data is highly volatile; models must incorporate temporal constraints to avoid future lookahead bias.
- Feature scaling and data cleaning represent 80% of the effort in practical ML engineering.
Future Improvements
- Upgrade the sentiment classifier to fine-tuned BERT models for financial contexts.
- Incorporate deep learning time-series models (LSTMs) for multi-day opening price predictions.