AI-Driven Policy & Compliance Intelligence System

An end-to-end NLP platform that automates the analysis of policy and regulatory documents. Users upload multiple PDFs or text files and the system runs an 8-stage pipeline — preprocessing, clause segmentation, knowledge extraction, embedding, semantic search, risk assessment, conflict detection, and AI summarization — behind a single Streamlit dashboard. The output is structured, explainable compliance intelligence rather than raw documents.

Compliance teams routinely wade through hundreds of pages of overlapping regulations, corporate policies, and legal manuals. Manual review is slow, obligations get missed, and conflicts across documents are almost impossible to catch by eye. The goal was to compress that workflow into an interactive tool that surfaces obligations, risks, and contradictions in minutes while remaining transparent about how it got there.

• 01Ingestion: PDF and TXT documents parsed with pdfplumber, normalized, and sentence-tokenized with NLTK.
• 02Clause segmentation: a rule-based engine detects headers and numbered clauses and splits documents into logical compliance units with preserved metadata.
• 03Knowledge extraction: keyword classification tags each clause as obligation, prohibition, penalty, or permission, combined with spaCy NER for organizations, laws, dates, and regulatory entities.
• 04Embeddings: every clause is encoded with all-MiniLM-L6-v2 (Sentence Transformers) into 384-dim vectors for semantic comparison.
• 05Semantic search & similarity: cosine similarity over cached embeddings powers natural-language queries and all-pairs clause comparison.
• 06Risk engine: hybrid rule-based + similarity signals classify clauses into Low / Medium / High risk with explainable reasoning.
• 07Conflict detection: identifies direct contradictions, overlapping obligations, missing requirements, and inconsistent regulations across documents.
• 08UI: multi-page Streamlit dashboard with resource caching, pagination, and persistent artifact storage (pickle) so embeddings and reports reload instantly.

• Python
• Streamlit
• Sentence Transformers
• spaCy
• NLTK
• Scikit-learn
• pdfplumber

• Multi-document PDF and text ingestion with automated preprocessing
• Clause-level segmentation and structured compliance metadata
• Named-entity extraction for organizations, laws, dates, and regulators
• Semantic search using Sentence Transformer embeddings and cosine similarity
• Explainable Low / Medium / High risk scoring for every clause
• Automated conflict detection across contradictions, overlaps, and gaps
• AI-generated document summaries and actionable recommendations
• Natural-language Q&A over the uploaded compliance corpus
• Streamlit dashboard with analytics, filters, and conflict visualization

• Clause boundaries in legal PDFs are inconsistent — solved with a layered segmentation strategy combining header detection, numbering patterns, and paragraph heuristics.
• Similarity thresholds needed careful calibration to separate true conflicts from paraphrased duplicates without flooding the dashboard with noise.
• Streamlit reruns on every widget change; introduced resource caching and persistent pickle artifacts so embeddings and analyses are computed once and reused.
• Keeping risk scoring explainable — combined rule signals with similarity evidence so every score carries a human-readable rationale.

• 8-stage NLP pipeline running end-to-end on multi-document uploads
• Semantic search and conflict detection working across mixed policy corpora
• Cached embeddings and lazy rendering keep the dashboard responsive on repeat runs
• Reusable extraction and risk modules structured for future rule additions

• For legal text, clause-level granularity beats document-level analysis — everything downstream gets sharper.
• Small, well-chosen embedding models (MiniLM) are more than enough when the retrieval surface is well-scoped.
• Explainability is a feature, not a nice-to-have — a risk score without a reason is useless to a compliance reviewer.
• Persisting intermediate artifacts is what makes a Streamlit app feel like a real product instead of a notebook.