NewsPulse AI: Databricks Generative AI Hackathon [1st place winner in Financial Services]
What It Does
This application is specifically designed to monitor and analyze the sentiment of the latest news articles regarding significant business events, such as layoffs, mergers and acquisitions, reorganizations, and disputes. These events can profoundly affect stock performance, making it vital for investors to stay informed.
Key Features
- Sentiment Analysis: Analyze sentiment by day and topic, with aggregated results.
- Stock Price vs Sentiment: A time series analysis to study the impact of news sentiment on stock performance.
- Chatbot: Provides Q&A capabilities using a vector search index and sourced information.
Data Acquisition Process
- News Articles: Uses the DuckDuckGo API to fetch recent news articles about selected companies.
- Content Scraping: Utilizes ScrapeGraphAI and GPT 3.5-Turbo to extract content from URLs.
- Sentiment Extraction: Applies DBRX Instruct and LangChain to determine sentiment from articles.
- RAG System: Articles are chunked, embedded using DBRX, and loaded into a Databricks vector store.
- Stock Data: Uses YahooQuery to gather historical stock price data from YahooFinance.
Automated Databricks jobs are supposed to run daily or multiple times a day to continuously update the database and vector store with new articles.
Tech Stack
- Databricks - Data Processing, Storage, Vector Database
- Streamlit - Frontend
- OpenAI - LLM
- DBRX - LLM
- Langchain - LLM wrapper
- DuckDuckGo - News API
- ScrapeGraphAI - Web Scraping
- Yahooquery - Yahoo Finance API
- Embedchain - RAG (used for demo as alternative to Databricks endpoint)
