Skip to content

About

Comparing Claude, Gemini, and ChatGPT

Resources

Stars

0 stars

Watchers

0 watching

Forks

Latest commit

 

History

11 Commits

Folders and files

Repository files navigation

Chatbot Conversation Analysis

A Python toolkit that ingests your conversation exports from Claude, Gemini, and ChatGPT, normalizes them into a common schema, and runs linguistic and semantic analyses to compare how each platform writes, thinks, and talks.

The goal is insight over volume. Not just word counts, but patterns in hedging language, intellectual depth, conversational style, and how your own writing shifts depending on which bot you're talking to.


Analysis Modules

Module What it measures
Lexical Vocabulary richness (MATTR), TF-IDF distinctive words, readability scores, sentence statistics, markdown formatting density
Semantic BERTopic topic modeling (assistant + user turns), embedding self-similarity (formulaicness), sentiment distribution
Pragmatic Hedging language, question rates, disclaimer frequency, verbosity ratios, first-person usage, opening patterns
Comparative Cross-platform style fingerprints, topic-controlled comparisons, user language adaptation analysis
Temporal Monthly trends in response length, vocabulary richness, hedging rates, user effort, inflection point detection
Conversation Structure Turn dynamics, conversation depth, resolution patterns, rephrase detection via embedding similarity
User Behavior How your own writing shifts per platform — formality, prompt engineering patterns, vocabulary, message classification

Visualizations

21 plot types generated automatically, including: style fingerprint radar, vocabulary comparison, response length distributions, hedging breakdowns, verbosity ratios, opening patterns, activity timelines, user formality comparison, rephrase rates, and depth distribution. All saved to data/outputs/plots/.


Getting Started

Prerequisites

  • Python 3.11+
  • ~2 GB disk space for models (sentence-transformers, spaCy)

Installation

git clone https://github.com/YOUR_USERNAME/chatbot-analysis.git
cd chatbot-analysis

# Option A: use the setup script
chmod +x setup.sh && ./setup.sh

# Option B: manual
pip install -r requirements.txt
python -m spacy download en_core_web_sm
mkdir -p data/raw/claude data/raw/gemini data/raw/chatgpt data/processed data/outputs/plots

Preparing Your Data

Drop your export file into the corresponding directory:

  • Claude: claude.ai > Settings > Account > Export Data > extract conversations.json > data/raw/claude/
  • Gemini: takeout.google.com > Deselect All > select "My Activity" > under "Activity Records" select only "Gemini Apps" > extract MyActivity.html > data/raw/gemini/
  • ChatGPT: chatgpt.com > Settings > Data Controls > Export Data > extract conversations.json > data/raw/chatgpt/

Running the Pipeline

python main.py inspect                        # Preview raw data structure
python main.py parse                           # Normalize to common format
python main.py analyze                         # Run all analysis modules
python main.py analyze --module lexical        # Run a single module
python main.py visualize                       # Generate all charts
python main.py report                          # Generate narrative report

All commands read paths and parameters from config.yaml.


Architecture

chatbot-analysis/
├── src/
│   ├── parsers/              # Per-platform parsers -> common Conversation schema
│   │   ├── base.py           #   Turn/Conversation dataclasses + abstract BaseParser
│   │   ├── claude_parser.py  #   JSON: conversations with typed content blocks
│   │   ├── gemini_parser.py  #   HTML Takeout: markdownify conversion, time-gap threading
│   │   └── chatgpt_parser.py #   JSON: tree-walking parser for ChatGPT exports
│   │
│   ├── analysis/             # Independent modules, each exposing run() -> dict
│   │   ├── lexical.py        #   Surface-level language (TTR, TF-IDF, readability)
│   │   ├── semantic.py       #   Embeddings, BERTopic, self-similarity, sentiment
│   │   ├── pragmatic.py      #   Hedging, disclaimers, verbosity, opening patterns
│   │   ├── comparative.py    #   Style fingerprints, topic-controlled comparisons
│   │   ├── temporal.py       #   Metric trends over time, inflection detection
│   │   ├── conversation_structure.py  # Depth, rephrase, resolution patterns
│   │   └── user_behavior.py  #   User language adaptation across platforms
│   │
│   └── viz/
│       └── plots.py          # 21 visualization functions + registry
│
├── data/
│   ├── raw/                  # Your export files (gitignored)
│   ├── processed/            # Normalized parquet + embedding caches (gitignored)
│   └── outputs/              # Charts, reports, analysis JSON
│
├── main.py                   # CLI entrypoint (click): inspect, parse, analyze, visualize, report
└── config.yaml               # All configurable parameters

Each analysis module is self-contained: it takes a list[Conversation] and config dict, returns a results dict, and can run independently. Visualization logic is fully separated from analysis — plots.py consumes the JSON output files.


Methodology

Common schema normalization. All platforms export differently. Claude provides clean JSON with typed content blocks. Gemini exports as Google Takeout HTML, which is converted to Markdown via markdownify and threaded into conversations using 60-minute time-gap heuristics. ChatGPT exports as JSON with a tree structure of message nodes. The parsers normalize everything into a shared Conversation > Turn schema with source, role, content, and timestamps.

Known data limitations. Gemini auto-capitalizes user messages, making user-side capitalization analysis unreliable for that source. Gemini conversation boundaries are inferred (not native), so some conversations may be mis-segmented. Sentiment analysis uses a lexicon-based approach (VADER) that skews negative for technical content.

Privacy. All analysis runs locally using all-MiniLM-L6-v2 (sentence-transformers) and en_core_web_sm (spaCy). No data leaves your machine. data/raw/ and data/processed/ are gitignored by default.


Project Status

Platform Parser Analysis
Claude Done Done
Gemini Done Done
ChatGPT Done Done

License

MIT

About

Comparing Claude, Gemini, and ChatGPT

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages