Introduction

A startup client asked the question: Can an Agent run my compliance for me?

The client has the following challenges:

The business must comply with obligations across legislation, internal policies, procedures, contracts, standards, and ecosystem rules.
The current compliance process is too manual for a small organisation with limited resources.

An example obligation may be “Quarterly access reviews must be conducted for all production systems”

A corporate may have teams that can manage compliance.

graph LR
    LO[Legal Officer]
    CM[Compliance Manager]
    SH[Stakeholder]

    LD[Legal and Compliance Documents]
    VO[Versioned Obligations]

    LO -.->|creates| LD
    LD -.->|creates| VO
    LO -.->|extends| LD
    LD -.->|extends| VO
    SH -.->|queries| LD
    LO -.->|queries| LD
    CM -.->|manages| VO

    style LO fill:#dbeafe,stroke:#2563eb,color:#1e40af
    style CM fill:#dcfce7,stroke:#16a34a,color:#166534
    style SH fill:#f3e8ff,stroke:#9333ea,color:#6b21a8

However in a start up environment most people are wearing many hats.

graph LR
    LO[Analyst]
    SH[COO]

    LD[Legal and Compliance Documents]
    VO[Versioned Obligations]

    LO -.->|creates| LD
    LO -.->|extends| LD
    SH -.->|queries| LD
    LD -.->|creates| VO
    LD -.->|extends| VO
    SH -.->|queries| VO
    LO -.->|queries| LD
    LO -.->|manages| VO

    style LO fill:#dbeafe,stroke:#2563eb,color:#1e40af
    style SH fill:#f3e8ff,stroke:#9333ea,color:#6b21a8

The proposed solution is:

The business needs a highly automated solution that identifies obligations and supports an operational process for managing them.
The solution must provide a complete and transparent view of all obligations in one place.

These are the initial non-functional requirements:

The solution must minimise manual effort because the organisation has limited compliance capacity.
The solution must be highly automated to reduce dependency on repeated manual review.
The solution must be clear and transparent so users can trust and review extracted obligations.
The solution must produce output in Excel or Word because that is the required working format. The startup is a Microsoft shop.
The solution should support repeatable scheduled execution as part of normal operations.
The solution should handle multiple source types without requiring separate tracking processes.
The solution must maintain a high degree of privacy and use only local storage for sensitive documents and outputs.

The key non-functional themes are privacy, accuracy, trust, and familiar formats.

The proposed flow for a new “agent” might look like this:

graph TD

    SH[Human]

    LD[Legal and Compliance Documents]
    VO[Versioned Obligations]

    START --> SH
    SH -.->|creates| LD
    SH -.->|document questions| QUERY[Query Documents Tool]
    QUERY --> LD
    LD --> EXTRACT[Extract Obligations Tool]
    EXTRACT --> VO
    VO --> MANAGE[Manage Obligations Tool]
    MANAGE -.->|notifies obligations| SH
    SH -.->|reviews obligations| VO

    MANAGE --> END[End]

    style START fill:#22c55e,color:#fff

    style EXTRACT fill:#6366f1,color:#fff
    style QUERY fill:#6366f1,color:#fff
    style MANAGE fill:#6366f1,color:#fff


    style END fill:#22c55e,color:#fff
    style SH fill:#f3e8ff,stroke:#9333ea,color:#6b21a8

Proof of Concepts

The first step was to prototype various tools and workflows before making recommendations of what is technically viable given the evolving nature of AI.

Starting small and focusing on querying an IT compliance document step in the flow.

1. Retrieval Augmented Generation (RAG) with Web App

The first Rag Tutorial repo proof of concept was using a Retrieval Augmented Generation (RAG) to solve the problem of too much, relevant and sensitive data being loaded into an LLM.

In this repo, the RAG system reads a PDF document, retrieves the most relevant sections based on a user query using a vector ChromaDB in a web app.

The reason for using a web app is to build an end to end PoC assuming the user would prefer to interact with a chat interface. This assumption will be tested through all the PoCs.

See deployed app version here: https://compliance-rag.streamlit.app/

graph LR
    A[Retrieval] --> B[Augmentation]
    B --> C[Generation]

This system uses free LLM, free Hugging Face transformers library, free ChromaDB storage and runs locally.

However simply retrieving text snippets using vector search isn’t enough.

This led to the next PoC to attempt to provide better context in the system.

2. GraphRAG

GraphRAG is a variation of RAG where the underlying database used for retrieval is a knowledge graph or a graph database.

This allows the model to reason over entities and relationships rather than flat text chunks.

See this paper for further reading : A BENCHMARK TO UNDERSTAND THE ROLE OF KNOWLEDGE GRAPHS ON LARGE LANGUAGE MODEL’S ACCURACY FOR QUESTION ANSWERING ON ENTERPRISE SQL DATABASES

The graphRAG tutorial repo runs as a CLI interface rather than using a web app. It uses a Neo4j for Desktop and a python script running locally to view the quality and timing of the responses.

graph LR
    A[Graph Retrieval - Neo4j Query] --> B[Knowledge Augmentation - Graph Context]
    B --> C[Generation - Anthropic LLM]

The GraphRAG seems to give more semantic meaning than RAG. However both of these proof of concepts are simple abstractions.

There are libraries that provide high level abstractions to hide complexity while allowing customisation.

3. LlamaIndex

LlamaIndex is a complete toolkit for creating LLM-powered agents over your data using indexes and workflows. For this course we’ll focus on three main parts that help build agents in LlamaIndex: Components, Agents and Tools and Workflows.

LlamaIndex Tutorial repo

Although this is a RAG Application with ChromaDB and Agent Workflow, it can support GraphRAG.

However I encountered too many issues and errors during the PoC that I decided not to use LlamaIndex, e.g.:

Import & Dependency Issues
Session Management Issues
Extensive configuration overhead

4. LangGraph

Moving on to another popular abstraction.

This LangGraph tutorial repo demonstrates a LangGraph-powered policy compliance assistant. The system showcases workflow patterns using offline, local LLMs for document processing and compliance analysis.

Agent with conversational UI using LangGraph Studio which also acts as an IDE.

The system implements a ReAct (Reasoning-Action) workflow pattern as a progression from RAG:

Reasoning: The agent analyzes the current state and user request
Action: Executes appropriate tools based on reasoning
Observation: Processes tool outputs and updates state
Decision: Determines next steps (continue with tools or provide final response)

This cycle ensures systematic problem-solving with clear decision points and state tracking.

graph LR
    A[START] --> I[Initialise Node]
    I --> B[Assistant Node]
    B --> C{Tools Condition}
    C -->|Tool Calls| D[Tools Node]
    C -->|No Tools| E[END]
    D --> B
    B --> E

5. Python Scripts

Instead of using a new models to extract obligations, I created a python tool for extracting compliance obligations from PDF documents and exporting them to Excel format.

As a prototype it runs locally, but with AI coding agents it is relatively quick to write an update these kind of scripts.

6. A Data Governance and Readiness Tool

This idea was a left field one.

One of the barriers to making a leap to using a full agent was just getting started changing the process, given the other priorities in the business.

The concept behind this tool is a daily tip and reward for thinking about data governance and AI usage.

https://chex-wheat.vercel.app

It takes the requirements and groups these into categories:

document
metadata
governance
security
integration
ai-llm
reporting
pilot

The tool is still a PoC (the name still resembles a breakfast cereal) and is being trialled internally.

Lessons Learnt

Lessons so far categorised by the key themes above:

Data privacy concerns:

It is possible to run LLM models locally and not in the cloud to alleviate privacy concerns
Data governance is key whether running the process manually or automating it with AI.

Accuracy concerns:

The LLMs extraction of obligations are still viewed as not accurate compared to an analyst manually extracting the obligations to a spreadsheet

Trust concerns:

The more abstracted a package/tool the more the tool becomes a black box to the steps and decisions made by the tool
Web app chat UIs do not give enough transparency or visibility of underlying documentation or steps or decisions in the process

Familiar formats:

Excel is still preferred as a format for the output to manage the obligations
The team does not have any in house engineers so any solution would need to be as low code or no code as possible

Recommendation

The recommendation is to agentify the obligations management process only using least priviledge data access and limit AI access to the obligations, with a boundary to isolate the legal documents store. This means a cloud AI solution is more acceptable.

The obligations already live in Excel, which is the preferred format that is readable and auditable.

graph TD

    SH[Human]

    LD[Legal and Compliance Documents]
    VO[Versioned Obligations]

    START --> SH
    SH -.->|creates| LD
    SH -.->|document questions| LD
    LD -.->|extracts| VO
    VO --> MANAGE[Manage Obligations Tool]
    MANAGE -.->|notifies obligations| SH
    SH -.->|reviews obligations| VO

    MANAGE --> END[End]

    style START fill:#22c55e,color:#fff
    style MANAGE fill:#6366f1,color:#fff
    style END fill:#22c55e,color:#fff
    style SH fill:#f3e8ff,stroke:#9333ea,color:#6b21a8

Given the startup has no in-house engineers, the recommendation is two further proof of concepts :

Microsoft Power Automate + AI Builder. This has a visual workflow with prebuilt and custom models available.
Microsoft Copilot studio which is a platform for building and managing agents.

Although this doesn’t create an agent for the whole process yet, it is a step forward in automation alongside the data governance efforts.

Table of Contents