Skip to content

Commit 30ad6d1

Browse files
HoneyTyagiiHoneyTyagii
committed
test+docs: test suite, CI, and docs
1 parent 06c094c commit 30ad6d1

17 files changed

Lines changed: 1154 additions & 63 deletions

‎.github/workflows/ci.yml‎

Lines changed: 48 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,48 @@
1+
name: CI
2+
3+
on:
4+
push:
5+
branches: [main, develop]
6+
pull_request:
7+
branches: [main]
8+
9+
jobs:
10+
lint:
11+
runs-on: ubuntu-latest
12+
steps:
13+
- uses: actions/checkout@v4
14+
- uses: actions/setup-python@v5
15+
with:
16+
python-version: "3.12"
17+
- run: pip install -e ".[dev]"
18+
- run: ruff check src tests
19+
- run: black --check src tests
20+
- run: mypy src/regmon
21+
22+
test:
23+
runs-on: ubuntu-latest
24+
strategy:
25+
matrix:
26+
python-version: ["3.10", "3.11", "3.12"]
27+
steps:
28+
- uses: actions/checkout@v4
29+
- uses: actions/setup-python@v5
30+
with:
31+
python-version: ${{ matrix.python-version }}
32+
- run: pip install -e ".[dev]"
33+
- run: pytest --cov=regmon --cov-report=xml -q
34+
- uses: actions/upload-artifact@v4
35+
if: matrix.python-version == '3.12'
36+
with:
37+
name: coverage-report
38+
path: coverage.xml
39+
40+
integration:
41+
runs-on: ubuntu-latest
42+
steps:
43+
- uses: actions/checkout@v4
44+
- uses: actions/setup-python@v5
45+
with:
46+
python-version: "3.12"
47+
- run: pip install -e ".[dev]"
48+
- run: pytest -m integration -q

‎README.md‎

Lines changed: 122 additions & 62 deletions
Original file line numberDiff line numberDiff line change
@@ -1,97 +1,157 @@
11
# Regulatory Monitoring Pipeline
22

33
A multi-agent system that monitors regulatory changes across jurisdictions
4-
(RBI, SEBI, FDA, EU AI Act, and more), determines business impact, assigns
5-
risk, and generates actionable follow-ups for the right teams.
4+
(RBI, SEBI, FDA, EU AI Act), determines business impact, assigns risk, and
5+
generates actionable follow-ups for the right teams — always with a
6+
human-in-the-loop approval gate.
67

7-
> Status: early development. This commit establishes project scaffolding and
8-
> tooling. Functionality lands incrementally over subsequent commits.
9-
10-
## Why
11-
12-
Compliance teams cannot manually track every circular, guideline, and rule
13-
change across regulators. This pipeline automates the tedious parts -
14-
collection, cleaning, deduplication, summarization - and augments human
15-
judgment for the parts that matter: impact analysis, risk assessment, and
16-
action planning, always with a human-in-the-loop approval gate.
17-
18-
## Architecture (target)
8+
## Architecture
199

2010
```
21-
Crawler Agent
22-
-> Document Parser
23-
-> Classification Agent
24-
-> RAG Search
25-
-> Risk Assessment Agent
26-
-> Action Planner
27-
-> Notification Agent
11+
┌─────────────────────────────────────────────────────────────────────┐
12+
│ Pipeline Orchestrator │
13+
│ │
14+
│ ┌──────────┐ ┌────────┐ ┌───────────┐ ┌──────────────────┐ │
15+
│ │ Crawler │──>│ Parser │──>│ Normalizer│──>│ Deduplication │ │
16+
│ │ Agent │ │ Agent │ │ │ │ Engine │ │
17+
│ └──────────┘ └────────┘ └───────────┘ └────────┬─────────┘ │
18+
│ │ │ │
19+
│ │ (adapters: RBI, SEBI, FDA, EU) unique docs │ │
20+
│ │ ▼ │
21+
│ ┌──────────┐ ┌────────────┐ ┌──────────┐ ┌──────────────┐ │
22+
│ │ Notifier │<──│ Approval │<──│ Action │<──│ Risk │ │
23+
│ │ Agent │ │ Gate (HITL)│ │ Planner │ │ Assessment │ │
24+
│ └──────────┘ └────────────┘ └──────────┘ └──────┬───────┘ │
25+
│ │ │ │
26+
│ │ (Slack, Email) │ │
27+
│ │ ┌──────────────┴───────┐ │
28+
│ │ │ Classification Agent │ │
29+
│ │ │ + Summarization │ │
30+
│ │ │ + RAG Indexing │ │
31+
│ │ └──────────────────────┘ │
32+
│ │ │
33+
│ ┌────┴────────────────────────────────────────────────────────┐ │
34+
│ │ Cross-cutting: SQLite/Postgres DB │ │
35+
│ │ Document Store │ Audit Log │ Pipeline Memory │ Approvals │ │
36+
│ └─────────────────────────────────────────────────────────────┘ │
37+
└─────────────────────────────────────────────────────────────────────┘
2838
```
2939

30-
Cross-cutting concerns: long-term memory, scheduling, human approval, and an
31-
append-only audit log.
32-
33-
## Project layout
34-
35-
```
36-
regulatory_monitoring_pipeline/
37-
├── src/regmon/ # application package (src layout)
38-
│ ├── __init__.py
39-
│ └── logging_config.py # structured logging setup
40-
├── tests/ # test suite
41-
├── pyproject.toml # build config, dependencies, tooling
42-
├── requirements.txt # runtime deps
43-
├── requirements-dev.txt # dev/test deps
44-
├── .env.example # configuration template
45-
├── .pre-commit-config.yaml
46-
├── Makefile
47-
└── README.md
48-
```
49-
50-
## Quickstart
40+
## Supported Jurisdictions
41+
42+
| Jurisdiction | Regulator | Sources |
43+
|-------------|-----------|---------|
44+
| RBI | Reserve Bank of India | Notifications, Press Releases |
45+
| SEBI | Securities and Exchange Board of India | Legal Circulars |
46+
| FDA | U.S. Food and Drug Administration | Press RSS, Federal Register |
47+
| EU AI Act | European Commission | AI Act Newsroom |
48+
49+
## Features
50+
51+
- **Async crawler** with per-host rate limiting, robots.txt respect, retry/backoff
52+
- **Source adapters** per regulator (HTML listing + RSS/Atom feed parsing)
53+
- **Document parser** supporting HTML and PDF (via `pypdf`)
54+
- **Content normalization** — encoding repair, boilerplate stripping, language detection
55+
- **Deduplication** — exact (SHA-256) + near-duplicate (SimHash) across crawl runs
56+
- **Embeddings + vector search** — chunking, mock/OpenAI embeddings, in-memory/FAISS/Chroma stores
57+
- **RAG search service** with citations, jurisdiction filtering, score thresholding
58+
- **Classification agent** — rule-based topic/function/urgency tagging
59+
- **Summarization** — structured LLM summaries (mock offline + OpenAI)
60+
- **Risk assessment** — weighted scoring (urgency, keywords, scope, penalties) + RAG context
61+
- **Action planner** — generates owner-assigned, deadline-driven follow-up items
62+
- **Human-in-the-loop approval** — risk-threshold gate with persisted decisions
63+
- **Notifications** — Slack webhook + SMTP email with digest formatting and alert dedup
64+
- **Pipeline orchestrator** — full graph wiring with persistent run memory
65+
- **Scheduler + CLI** — `regmon run|backfill|status|sources` commands
66+
- **Append-only audit log** across all pipeline events
67+
68+
## Quick Start
5169

5270
```bash
5371
# 1. Create and activate a virtual environment
5472
python -m venv .venv
5573
. .venv/Scripts/activate # Windows
5674
# source .venv/bin/activate # macOS / Linux
5775

58-
# 2. Install (with dev tooling)
76+
# 2. Install with dev tooling
5977
pip install -e ".[dev]"
6078

6179
# 3. Configure
6280
cp .env.example .env
6381

64-
# 4. Install git hooks
65-
pre-commit install
66-
67-
# 5. Run checks
82+
# 4. Run checks
6883
make lint
6984
make test
85+
86+
# 5. Run the pipeline
87+
regmon sources # list configured sources
88+
regmon run # one-shot pipeline run
89+
regmon run --loop --interval 60 # continuous mode
90+
regmon status # check pipeline state
91+
regmon backfill # re-process ignoring dedup
7092
```
7193

72-
## Make targets
94+
## Project Layout
7395

74-
| Target | Description |
75-
| ------------- | ---------------------------------------- |
76-
| `make install`| Install package with dev dependencies |
77-
| `make lint` | Run ruff + black (check) + mypy |
78-
| `make format` | Auto-format with black + ruff --fix |
79-
| `make test` | Run the test suite |
80-
| `make hooks` | Install pre-commit hooks |
81-
| `make clean` | Remove caches and build artifacts |
96+
```
97+
src/regmon/
98+
├── __init__.py, __main__.py, cli.py, scheduler.py
99+
├── config/ # Settings, secrets, source registry
100+
├── models/ # Pydantic domain models and enums
101+
├── crawler/ # Async fetcher, rate limiter, robots.txt, adapters
102+
├── parser/ # HTML/PDF extraction, metadata (dates, ref numbers)
103+
├── normalize/ # Encoding repair, boilerplate strip, language detect
104+
├── dedup/ # Content hashing, SimHash near-duplicate engine
105+
├── embeddings/ # Chunking, providers, vector stores, indexer
106+
├── rag/ # Semantic search service with citations
107+
├── classification/ # Rule-based + LLM topic/function classifier
108+
├── summarization/ # Structured LLM summaries (mock + OpenAI)
109+
├── risk/ # Risk scoring, rationale, assessment agent
110+
├── actions/ # Action planner with templates and due dates
111+
├── approval/ # Human-in-the-loop gate with persisted decisions
112+
├── notifications/ # Slack/email channels, digest formatting, dedup
113+
├── pipeline/ # Orchestrator, state tracking, run context
114+
└── db/ # SQLAlchemy engine, document store, audit log
115+
tests/ # Unit + integration test suite
116+
.github/workflows/ # CI pipeline (lint, test matrix, integration)
117+
```
82118

83119
## Configuration
84120

85121
All settings are read from environment variables (see `.env.example`). The
86122
pipeline ships with `mock` LLM and embedding providers so it runs fully
87-
offline by default, and a `REGMON_DRY_RUN` flag that disables outbound
88-
notifications during development.
123+
offline by default. Set `REGMON_DRY_RUN=true` to suppress outbound notifications.
124+
125+
### Optional Extras
126+
127+
```bash
128+
pip install -e ".[faiss]" # FAISS vector store backend
129+
pip install -e ".[chroma]" # Chroma vector store backend
130+
pip install -e ".[openai]" # OpenAI embeddings + LLM
131+
```
132+
133+
## Testing
134+
135+
```bash
136+
make test # unit tests
137+
pytest -m integration # integration tests
138+
pytest --cov=regmon # with coverage
139+
```
140+
141+
The test suite uses in-memory SQLite and `httpx.MockTransport` so no external
142+
services are needed. Integration tests are marked with `@pytest.mark.integration`.
89143

90-
## Roadmap
144+
## Make Targets
91145

92-
This project is built across 20 incremental commits, from scaffolding through
93-
ingestion, storage/RAG, the agent layer, orchestration, and hardening. See the
94-
commit history for the full progression.
146+
| Target | Description |
147+
|--------|-------------|
148+
| `make install` | Install package with dev dependencies |
149+
| `make lint` | Run ruff + black (check) + mypy |
150+
| `make format` | Auto-format with black + ruff --fix |
151+
| `make test` | Run the test suite |
152+
| `make test-cov` | Run tests with coverage report |
153+
| `make hooks` | Install pre-commit hooks |
154+
| `make clean` | Remove caches and build artifacts |
95155

96156
## License
97157

‎src/regmon/__init__.py‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -4,7 +4,7 @@
44
across jurisdictions.
55
"""
66

7-
from __future__ import annotations
7+
from __future__ import annotations
88

99
__version__ = "0.1.0"
1010

‎tests/conftest.py‎

Lines changed: 106 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,106 @@
1+
"""Shared test fixtures for the regulatory monitoring pipeline."""
2+
3+
from __future__ import annotations
4+
5+
import uuid
6+
7+
import pytest
8+
9+
from regmon.config.settings import CrawlSettings, Settings, StorageSettings
10+
from regmon.db import create_database
11+
from regmon.db.engine import Database
12+
from regmon.models import (
13+
ActionItem,
14+
ActionPriority,
15+
DocumentFormat,
16+
Jurisdiction,
17+
ParsedDocument,
18+
RawDocument,
19+
RiskAssessment,
20+
RiskLevel,
21+
)
22+
23+
24+
@pytest.fixture()
25+
def db() -> Database:
26+
"""In-memory SQLite database with all tables created."""
27+
# Import all record modules so Base.metadata knows every table.
28+
import regmon.approval.store
29+
import regmon.db.records
30+
import regmon.dedup.store
31+
import regmon.notifications.service
32+
import regmon.pipeline.state # noqa: F401
33+
34+
database = create_database("sqlite:///:memory:")
35+
database.create_all()
36+
yield database
37+
database.dispose()
38+
39+
40+
@pytest.fixture()
41+
def settings(tmp_path) -> Settings:
42+
"""Test-friendly settings with temp paths."""
43+
return Settings(
44+
crawl=CrawlSettings(rate_limit_per_sec=100, max_retries=1, backoff_factor=0.0),
45+
storage=StorageSettings(
46+
raw_storage_path=tmp_path / "raw",
47+
vectorstore_path=tmp_path / "vs",
48+
),
49+
)
50+
51+
52+
@pytest.fixture()
53+
def raw_document() -> RawDocument:
54+
"""A sample raw document."""
55+
return RawDocument(
56+
source_id="rbi-test",
57+
jurisdiction=Jurisdiction.RBI,
58+
url="https://rbi.example/doc",
59+
content="<html><body><h1>KYC Direction</h1><p>All banks must comply with KYC norms.</p></body></html>",
60+
content_format=DocumentFormat.HTML,
61+
http_status=200,
62+
)
63+
64+
65+
@pytest.fixture()
66+
def parsed_document() -> ParsedDocument:
67+
"""A sample parsed document."""
68+
return ParsedDocument(
69+
raw_document_id=uuid.uuid4(),
70+
source_id="rbi-test",
71+
jurisdiction=Jurisdiction.RBI,
72+
title="Master Direction on KYC",
73+
clean_text=(
74+
"All scheduled commercial banks must implement video-KYC for new "
75+
"customer onboarding with immediate effect. Non-compliance will "
76+
"attract penalty of Rs. 5 crore under Banking Regulation Act. "
77+
"This circular supersedes earlier guidance on the subject."
78+
),
79+
reference_number="RBI/2023-24/115",
80+
summary="RBI mandates video-KYC for banks.",
81+
)
82+
83+
84+
@pytest.fixture()
85+
def risk_assessment(parsed_document) -> RiskAssessment:
86+
"""A sample high-risk assessment."""
87+
return RiskAssessment(
88+
document_id=parsed_document.id,
89+
jurisdiction=Jurisdiction.RBI,
90+
risk_level=RiskLevel.HIGH,
91+
score=0.65,
92+
rationale="High risk due to penalty and immediate enforcement.",
93+
impacted_areas=["Compliance", "Technology"],
94+
)
95+
96+
97+
@pytest.fixture()
98+
def action_item(risk_assessment) -> ActionItem:
99+
"""A sample action item."""
100+
return ActionItem(
101+
assessment_id=risk_assessment.id,
102+
title="Review KYC circular",
103+
description="Review and implement video-KYC requirements.",
104+
priority=ActionPriority.HIGH,
105+
owner_team="Compliance",
106+
)

0 commit comments

Comments
 (0)