DataFog is a Python library for detecting and redacting personally identifiable information (PII).
It provides:
- Fast structured PII detection via regex
- An offline PII firewall for AI agents: a Claude Code hook and a LiteLLM gateway guardrail (introduced in 4.6)
- Optional NER support via spaCy and GLiNER
- A simple agent-oriented API for LLM applications
- Backward-compatible
DataFogandTextServiceclasses
DataFog includes two ready-made enforcement points that catch PII at the moment it would leave your machine — offline, in microseconds, with matched values never echoed into logs or transcripts:
-
Claude Code hook (
datafog-hook): gates agent tool calls (shell commands, web requests, file writes, MCP tools) and warns the model when prompts or tool results carry PII. ~105–110ms per invocation including process startup and the native engine load (DATAFOG_BACKEND=pythontrims ~20ms at the cost of detection coverage). Easiest install is the Claude Code plugin:/plugin marketplace add DataFog/datafog-claude-plugin /plugin install datafog@datafogManual hook setup and limitations: examples/claude_code_hook/.
-
LiteLLM guardrail (
DataFogGuardrail): redacts or blocks PII in requests and responses at the gateway, for any LiteLLM-proxied provider. In-process (tens of microseconds per message scanned on the native engine), no sidecar service. Setup: examples/litellm_guardrail/.
Both default to the high-precision entity set (EMAIL, PHONE,
CREDIT_CARD, SSN); noisier types are opt-in. Known-safe values can be
exempted with an allowlist: scan(text, allowlist=[...]) for exact values,
allowlist_patterns=[...] for full-match regexes (e.g. ^\d{10}$ to stop
unix timestamps matching as phone numbers) — available in both adapters and
the API. Presidio-style entity names (EMAIL_ADDRESS, PHONE_NUMBER,
US_SSN) are accepted as aliases for easy migration.
Every performance number above is reproducible with one command — methodology, pinned payloads, and comparisons against Presidio and spaCy NER live in benchmarks/.
# Core install: native engine, no extra packages needed
pip install datafog
# Add spaCy support
pip install datafog[nlp]
# Add GLiNER + spaCy support
pip install datafog[nlp-advanced]
# Add local OCR support
pip install datafog[ocr]
# Add Spark/distributed support
pip install datafog[distributed]
# Everything
pip install datafog[all]pip install datafog pulls datafog-core, which is the native engine. Wheels
exist for Linux (glibc and musl, x86_64/aarch64/s390x/ppc64le), macOS and
Windows (amd64/arm64); anywhere else the source build needs a Rust toolchain.
If Core cannot be imported, detection falls back to the pure-Python engine and
warns once — DATAFOG_BACKEND=python selects that path deliberately.
The nlp-advanced extra includes SentencePiece and protobuf for GLiNER's
multilingual tokenizer; installing the OCR extra is not required for GLiNER.
backend defaults to auto: the native engine when datafog-core is
importable, the pure-Python regex engine otherwise. DATAFOG_BACKEND sets the
default and backend="python" / backend="rust" override it per call.
Every engine uses this backend for structured PII. engine="spacy",
"gliner" and "smart" add model spans to the same structured findings;
selecting an ML engine does not disable native card or credential detection.
The native engine adds detectors the Python engine never had — API_KEY,
BEARER_TOKEN, JWT, PRIVATE_KEY, CREDENTIAL_URI, ACCOUNT_NUMBER,
US_ROUTING_NUMBER, NPI, UUID and a German set — and serves prose PERSON
and ADDRESS from an embedded model, so name and address detection needs no
spaCy and no model download. Findings keep engine="regex" provenance and
confidence=1.0 as compatibility values, not native confidence estimates.
Reviewed differences from the Python engine are listed in
tests/contracts/rust-1.5.0.json.
datafog.native exposes the native schema directly.
Python 3.13 support is certified for the core SDK, CLI, nlp,
nlp-advanced, and ocr install profiles. Donut OCR still requires a model
that is available locally before runtime use. distributed and all remain
optional, heavier profiles and are not part of the lightweight core path.
Python 3.14 support is certified for the core SDK and CLI. Optional profiles remain dependent on upstream Python 3.14 package availability until they are covered by the corresponding CI install-profile checks.
import datafog
text = "Contact [email protected] or call (555) 123-4567"
clean = datafog.sanitize(text, engine="regex")
print(clean)
# Contact [EMAIL_1] or call [PHONE_1]import datafog
# 1) Scan prompt text before sending to an LLM
prompt = "My SSN is 123-45-6789"
scan_result = datafog.scan_prompt(prompt, engine="regex")
if scan_result.entities:
print(f"Detected {len(scan_result.entities)} PII entities")
# 2) Redact model output before returning it
output = "Email me at [email protected]"
safe_result = datafog.filter_output(output, engine="regex")
print(safe_result.redacted_text)
# Email me at [EMAIL_1]
# 3) One-liner redaction
print(datafog.sanitize("Card: 4111-1111-1111-1111", engine="regex"))
# Card: [CREDIT_CARD_1]German structured PII is country-specific and opt-in. Use explicit locale selection or entity-type filtering when you want German VAT IDs, German IBANs, tax IDs, postal codes, passports, or residence permits.
import datafog
text = "Steuer-ID 12345678901 liegt vor."
print(datafog.scan(text, engine="regex", locales=["de"],
entity_types=["DE_TAX_ID"]).entities)
# [Entity(type='DE_TAX_ID', text='12345678901', ...)]import datafog
# Reusable guardrail object
guard = datafog.create_guardrail(engine="regex", on_detect="redact")
@guard
def call_llm() -> str:
return "Send to [email protected]"
print(call_llm())
# Send to [EMAIL_1]Use the engine that matches your accuracy and dependency constraints:
regex:- Fastest and always available.
- Best for default structured entities:
EMAIL,PHONE,SSN,CREDIT_CARD,IP_ADDRESS,DATE,ZIP_CODE(DOBandZIPare accepted as input aliases). - Use
locales=["de"]for German structured IDs such asDE_VAT_ID,DE_IBAN,DE_TAX_ID,DE_POSTAL_CODE, and passport or residence permit numbers.
spacy:- Requires
pip install datafog[nlp]. - Useful for unstructured entities like person and organization names.
- Requires
gliner:- Requires
pip install datafog[nlp-advanced]. - Stronger NER coverage than regex for unstructured text.
- Requires
smart:- Cascades regex with optional NER engines.
- If optional deps are missing, it degrades gracefully and warns.
The 4.x line keeps the main package story centered on lightweight text PII screening. OCR and Spark remain supported optional surfaces for users who already rely on them, but they are not required for the core import, default scan/redact helpers, or guardrail helpers.
- OCR:
- Install
datafog[ocr]for local image OCR helpers. - URL-based image downloading also needs
datafog[web,ocr]. - Tesseract usage requires the system
tesseractbinary. - Python 3.13 is validated for the OCR install profile, Pillow, pytesseract, and system Tesseract smoke checks.
- Donut OCR requires
datafog[nlp-advanced,ocr]and a model already available locally.
- Install
- Spark:
- Install
datafog[distributed]forSparkService. - Spark PII UDF helpers also require
datafog[nlp]and an installed spaCy model. - A Java runtime is required by PySpark.
- Install
DataFog deprecates OCR and Spark with visible use-time
warnings; their APIs and extras will be removed in 5.0. The last supported
version in this release train is 4.12.0 — pin it with
pip install "datafog==4.12.0" — and every
retired surface is tracked in the retirement schedule.
See the 4.9 migration guide for the transition plan,
and VERSIONING.md for what a version number promises.
The existing public API remains available.
from datafog import DataFog
result = DataFog().scan_text("Email [email protected]")
print(result["EMAIL"])from datafog.services import TextService
service = TextService(engine="regex")
result = service.annotate_text_sync("Call (555) 123-4567")
print(result["PHONE"])Install the command dependencies with pip install "datafog[cli]".
# Scan text
datafog scan-text "[email protected]"
# Redact text
datafog redact-text "[email protected]"
# Replace text with pseudonyms
datafog replace-text "[email protected]"
# Hash detected entities
datafog hash-text "[email protected]"
# Enable German regex identifiers
datafog redact-text "Steuer-ID 12345678901" --locale deDataFog telemetry is disabled by default.
To opt in:
export DATAFOG_TELEMETRY=1To force telemetry off:
export DATAFOG_NO_TELEMETRY=1
# or
export DO_NOT_TRACK=1Telemetry does not include input text or detected PII values.
The native engine is the default, with a warned Python fallback. The 4.12
review fixes preserve structured detection across every engine and legacy API.
DataFog 4.12 requires the reviewed Core 1.5 minor series
(datafog-core~=1.5.0) in the base package and the rust extra. The published
Core 1.5.0 wheel passed installed integration and compatibility qualification
with this release. Python and Core have independent release versions.
See release preparation.
Entity labels, locales and activation settings come from the installed Core.
get_supported_entities() reports that backend's supported inventory, including
opt-in labels; it does not mean every label is enabled by default. German
detection is opt-in through locale or entity selection. UUID, PERSON and
ADDRESS can be selected explicitly. Unsupported labels and invalid arguments
raise errors instead of silently disabling detection.
The 4.8.1 compatibility contract records published Python behavior for the Rust migration, with frozen fixtures and instructions for independently reproducing them from the release wheel.
git clone https://github.com/datafog/datafog-python
cd datafog-python
python -m venv .venv
source .venv/bin/activate # Windows: .venv\Scripts\activate
pip install -e ".[all,dev]"
pip install -r requirements-dev.txt
pytest tests/