cest is a fast-as-hell and less-greedy Certificate Transparency monitor built for put your task here, as a drop-in replacement for https://github.com/d-Rickyy-b/certstream-server-go.
Why cest?
- Cuz it has 26% less traffic consumption, keeping 99.2% of new domains, compared to
certstream-server-go. - It cuts off 99% of useless domains, making domains stream ~31% lighter out of the box.
- It's built with love and supercharged with Ukrainian energy.
Usage:
go install github.com/aels/cest/cmd/cest@latest # but the rusty imperfections that define us...Than
cest --daemonThen connect with a WebSocket client such as websocat:
websocat ws://127.0.0.1:4949/all | python3 any-of-your-tools.pyThe installed binary is placed in $(go env GOPATH)/bin. Make sure that directory is in your PATH, then run cest.
The cest is intentionally optimized for only one task: "give me only domains feed, fast". Selected high-overlap CT logs are denied, non-active logs are skipped, the default noise filter removes known high-volume domains, and an in-memory FIFO set keeps you from dupes. Use
--nofilterto find out how deep a rabbit hole is.
- Reduce duplicate noise. Wildcards are normalized and dupe domains are suppressed by a 100k FIFO set.
- Filter common trash by default. The default filter removes most large cloud, hosting, CDN, and platform domains:
\.(google|googleapis|amazon|amazonaws|microsoft|azure|cloudflare|akamai|fastly|github|facebook|apple|netflix|shopify|squarespace|wix|godaddy|wordpress\.com|herokuapp|vercel|netlify|pages\.dev|firebaseapp|cloudfront|letsencrypt|sectigo|digicert|globalsign|comodo|verisign|gov|edu|mil)|workers\.dev|quickconnect\.(to|cn)|icdn\.ru|aws\.dev|\.on\.aws|netcraze\.io|mybluehost\.me|sslip\.io|nip\.io|amplifyapp\.com|platformsh\.site|zendesk\.com|synology\.me|remotewd\.com|meraki\.direct|duckdns\.org|plesk\.page|workdaysuv|website-k\.com|technology2022\.com|altervista\.org|haplorrhini\.com|tunwg\.com|towerbrands\.asia|dpdns\.org|certsbridge\.com|myvolumio\.org|ui\.direct|mybluelantern\.com|sovcloud-core\.fr|hstgr\.cloud|myftpupload\.com|xmit\.dev|mywhc\.ca|sub-site\.eu|endl\.site|hostclusters\.com|server-he\.de|rakkoserver\.net|optum\.com|a2z\.eu|wholesale-nondurable\.org|trucking-warehousing\.org|ciscosecureaccess|sourcemanager\.dev|cpanel\.site|traefik\.me|stratoserver\.net|ts\.net|ccwu\.cc|battery\.cam|servi\.tn|testsoftwares\.site|uhc\.com|flowww\.net|luckysrv\.de|ccaiplatform\.com|skalator\.de|judikamuhoki|freedomsoftsites\.com|ekalamfoundation\.com|gaymonrealtygroup\.com|adison\.co|jnj\.com|pustekno\.id|verify-recruit\.work|beatplugs\.com|texastopn\.top|whm\.
That's a lot of trash (like a 30% offload). It can be disabled or extended with your own regular expression, btw.
- Create your own live feeds. Shops? VPNs? Any CMS, any app, really.
--sort wp:wp-contentexposes a/wpWebSocket containing domains whose HTTP or HTTPS response includeswp-content. - Use several classifiers at once. A response may match multiple rules, and the domain will be published to every matching route.
- Built-in DNS server, no bottlenecks. The built-in UDP resolver benchmarks public resolvers and rotates among the fastest available servers.
- Observe when shit goes wrong.
/statsreports CT, DNS, HTTP, queue, and per-route counters, including evicted sort tasks and client messages. - Run cheaply. Need only a pure domain list? -
/all. Without--sort, no DNS resolver & no HTTP worker pool are created.
The current stable Go release on Ubuntu or Debian is recommended.
git clone https://github.com/aels/cest.git
cd cest
go build -o cest ./cmd/cest
./cestBoth the short and long forms shown below are supported.
| Option | Default | Description |
|---|---|---|
-nf, --nofilter |
disabled | Disable only the built-in noise filter. A regex supplied with --filter remains active. |
-f REGEX, --filter REGEX |
none | Additional case-insensitive exclusion regex applied to normalized full domain names. May be specified once. Invalid expressions stop startup. |
-p PORT, --port PORT |
4949 |
Port for the HTTP and WebSocket server. |
-s PATH:LITERAL, --sort PATH:LITERAL |
none | Add an HTTP body classifier and WebSocket route. Repeatable. Search is a case-sensitive literal byte match. |
-dns LIST, --dns LIST |
automatic | Comma-separated public IPv4 DNS servers, for example 1.1.1.1,8.8.8.8. Port is always UDP/53. |
-expose, --expose |
disabled | Bind to 0.0.0.0 instead of 127.0.0.1. There is no built-in auth/TLS - share it with friends & enemies. |
-t N, --threads N |
2000 |
Maximum concurrent sort workers. The sort ring buffer holds N × 10 pending domains. |
--client-buffer-mb N |
10 |
Independent unsent-message buffer for each WebSocket client, in MiB. |
--dns-timeout DURATION |
1s |
Timeout for a UDP DNS query. Uses Go durations such as 500ms or 2s. |
--connect-timeout DURATION |
2s |
TCP connection timeout for HTTP and HTTPS checks. |
--response-timeout DURATION |
5s |
Total timeout for each HTTP or HTTPS attempt. |
--body-limit-kb N |
200 |
Maximum decompressed response bytes inspected per request. |
-d, --daemon |
disabled | Self-daemonize and run in the background. A service supervisor is still preferable for production. |
-h, --help |
— | Show command help. |
-v, --version |
— | Show the version. |
A sort rule uses PATH:LITERAL:
--sort 'wp:wp-content'
│ └─ case-sensitive string searched in the response body
└───────── creates WebSocket route /wp
The path may contain lowercase a-z, digits, -, _ and .. all and stats are reserved. Duplicate paths are rejected. Everything after the first : is part of the literal.
For every accepted domain, the sorter:
- publishes it to
/allimmediately; - resolves an A record with the built-in UDP resolver;
- requests
http://domain/without following redirects; - if HTTP does not return exactly
200 OK, trieshttps://domain/once; - scans only a successful response body and publishes the domain to every matching sort route.
Exclude names containing dev, staging or internal while keeping the built-in filter enabled:
cest --filter '(^|\.)(dev|staging|internal)(\.|$)'Disable the built-in filter but retain the custom expression:
cest --nofilter --filter '\.example\.com$'cest \
--sort 'wordpress:wp-content' \
--sort 'magento:Magento_' \
--sort 'shops:class="cart"'Consume the resulting feeds:
websocat ws://127.0.0.1:4949/wordpress
websocat ws://127.0.0.1:4949/magento
websocat ws://127.0.0.1:4949/shopsA domain may appear in more than one route when multiple literals match.
cest \
--sort 'wordpress:wp-content' \
--dns '1.1.1.1,8.8.8.8,9.9.9.9' \ # only 2 DNS servers will be for sure not enough to resolve 5000 domains/sec
--dns-timeout 750ms \
--connect-timeout 1500ms \
--response-timeout 4scest --expose --port 4200 --daemon --sort 'wordpress:wp-content'This exposes the unauth server on 0.0.0.0:4200. Extremely useful when you have only one server with 200Mb/s bandwidth cap to knock every domain of a stream over HTTP. Less powerful nodes can just listen for 10-20 events/sec sitting somewhere else.
curl http://127.0.0.1:4949/statsThe JSON snapshot is rebuilt every five seconds. Important fields include:
ct.active_logs,ct.entries,ct.source_errors;domains.extracted,domains.duplicates,domains.published_all;dns.errors,dns.timeouts,dns.private_address_drops;http.http_200,http.https_200,http.non_200,http.read_timeouts;queues.sort_queue_dropped;routes.<name>.clientsandroutes.<name>.client_dropped.
| Endpoint | Protocol | Purpose |
|---|---|---|
/all |
WebSocket | All accepted domains after filtering and global deduplication. |
/<sort-path> |
WebSocket | Domains whose successful HTTP/HTTPS body matched the route literal. |
/stats |
HTTP | Cached runtime statistics and health response. |
On /all, one WebSocket text frame may contain up to 100 domains separated by \n; treat every non-empty line as one domain. On every sort route, each WebSocket text frame contains exactly one domain. New clients receive no history.
- Google v3
logsandtiled_logsare refreshed daily. - Only
usableandqualifiedlogs are followed. - Every source starts at its current STH/checkpoint; no history or disk cursor is used.
- Both certificates and precertificates are processed.
- DNS SAN and legacy CN are extracted without full certificate enrichment.
- Wildcards become their base names, Unicode is converted with IDNA, and malformed names are dropped.
- The global 100,000-name FIFO does not refresh a duplicate's position.
- With sorting enabled, queue overflow evicts the oldest unprocessed domain and increments
sort_queue_dropped.
The sorter uses its own UDP/53 A resolver with EDNS(0), bounded DNS parsing, and at most three CNAME transitions. It does not use /etc/resolv.conf, DoH, DoT, DNS-over-TCP, DNSSEC, or a DNS cache.
Automatic resolver candidates:
1.1.1.1 1.0.0.1 8.8.8.8 8.8.4.4 9.9.9.9
149.112.112.112 208.67.222.222 208.67.220.220 94.140.14.14 94.140.15.15
76.76.2.0 76.76.10.0 185.228.168.9 185.228.169.9 64.6.64.6
64.6.65.6 77.88.8.8 77.88.8.1 156.154.70.1 156.154.71.1
195.46.39.39 195.46.39.40 84.200.69.80 84.200.70.40 45.90.28.0
They are benchmarked from the running machine against example.com, iana.org and cloudflare.com; up to ten successful resolvers are selected and refreshed hourly.
HTTP connects directly to the selected public IPv4 while preserving the domain in the Host header and TLS SNI.
- HTTPS fallback intentionally accepts invalid, expired, and hostname-mismatched certificates.
- Redirects and keep-alive are disabled.
- The output is optimized for discovery efficiency, not guaranteed complete CT coverage.
- CT signatures, inclusion proofs, and consistency proofs are not verified.
- State is not persisted; domains issued while the process is stopped are not recovered.
- DNS has no TCP fallback. A truncated UDP response is retried once through another resolver and then skipped.
- Only IPv4 A records (no AAAA) are used for HTTP sorting.
- The embedded server has no authentication or TLS.
- Recent-domain deduplication intentionally hides repeated certificate issuance until the domain leaves the 100k-entry FIFO.
go test ./...
go test -race ./internal/ctstream ./internal/dnsresolver \
./internal/ring ./internal/sorter ./internal/web ./internal/dedupCustom permissive, WTFPL-style license. See LICENSE; this project does not claim that the text is the official WTFPL wording.