AutomationSTATUS: Live 1 0

Competitor Tracking Engine

Headless web scraping and change detection engine

A FastAPI-based web scraping and market intelligence engine. It tracks target competitor websites, detects structural DOM changes, and generates intelligence reports on pricing shifts.

Inspect GitHub Source Code

01 // THE BUSINESS PROBLEM & SITUATION

Operational Context

Product teams lacked real-time visibility into rapid competitor pricing shifts and unannounced feature releases.

CLIENT / ENVIRONMENT: Built for marketing and product managers needing automated competitive intelligence.

02 // SYSTEM ARCHITECTURE & SOLUTION

What Was Built

An automated tracking pipeline using Playwright headless browsers to capture page snapshots, compute DOM diffs, and generate change summaries.

competitor-intel-monitor-ai.sysAutomation

VERIFIED ARCHITECTURECompetitor Tracking Engine

SYSTEMS_ARCHITECTURE_EXPLODER // LAYERED_STACKFULL-STACK SYSTEM DECOUPLING

Frontend Layer

Next.js 16 App Router, React 19, Tailwind CSS, TypeScript

API & Middleware

FastAPI, Pydantic Schema Validation, Rate Limiters

Orchestration

LangGraph Multi-Agent Supervisor & State Machines

Model Providers

OpenAI GPT-4o, Anthropic Claude 3.5, Vision AI

Data Stores

PostgreSQL (Prisma), Qdrant Vector Store, Redis Caching

Observability

LangSmith traces, Pydantic audit logs, Error retries

03 // ENGINEERING CONTRIBUTION & FEATURES

Exact Contribution

Developed Playwright scraping scripts bypassing basic anti-bot blocks
Implemented DOM diffing algorithms to isolate relevant text shifts
Integrated automated email alerts for pricing drops

Key Capabilities Built

✓ Headless Browser Page Scraping

✓ DOM Change Detection Engine

✓ Automated Pricing Alerts

✓ Weekly Digest Generation

DATA_PROVENANCE_TIMELINE // AUDIT_TRACETRACE_ID: #TR-992041

14:22:01.002Source Ingestion

Received webhook from lead submission formOK

14:22:01.045Schema Sanitization

Pydantic validated 12 extracted fieldsOK

14:22:01.210LangGraph Reasoning

Score intent = 0.94 (Tier A Lead)OK

14:22:01.350Human Audit Check

Confidence threshold passed (>0.85)PASSED

14:22:01.480CRM Commit

Created contact ID #884920 in PostgreSQLCOMMITTED

04 // EXECUTION PIPELINE

Step-by-Step Workflow

STEP 01Scheduled task triggers Playwright scrapers.

STEP 02Scrapers capture competitor page text.

STEP 03Engine compares new text against baselines.

STEP 04Alert dispatches if significant shifts are detected.

05 // ENGINEERING LESSONS & OUTCOMES

Technical Challenges Overcome

• Filtering out noise like timestamps and dynamic ad banners.
• Handling dynamic Single-Page Application rendering.

Measurable Outcomes

→ Provided automated visibility into competitor pricing adjustments.

→ Eliminated manual weekly checking of competitor websites.

Lessons & Engineering Rules

"Combining semantic HTML analysis with visual regression yields better change detection than brittle CSS selectors."

NEXT RELATED CASE STUDY

WhatsApp Automation Hub

View Study

PROJECT METADATA

ARSLAN'S ROLEFull-Stack Automation Developer

YEAR & STATUS2024 // Live

CATEGORYAutomation

TECHNOLOGY STACK

PythonFastAPIPlaywrightBeautifulSoupCron

GITHUB METRICS

Stars: 1

Forks: 0

Verified GitHub Update: 8/2/2026

Need a similar solution?

If your business faces similar data, automation, or software bottlenecks, let's discuss your requirements.

Start a Conversation