Code behind Regime Shifts in Web Attack Traffic: Uncovering the Rise of AI-Crawler Impersonation. It detects behavioral regime shifts in HTTP(S) honeypot traffic from four observable daily signals, profiles each shift and attributes it to source populations by actor ablation, and checks claimed crawler identities against operators' published IP ranges and reverse DNS.
The request logs cannot be redistributed (they are processed in situ through controlled interfaces of the data owner). This repository contains the code, the configuration, and snapshots of the operators' public IP-range lists; it does not contain raw or processed traffic.
pipeline/ raw daily CSV -> cleaned, scanner-tagged Parquet
analysis/ one script per paper artifact, plus the modules they share (common.py, probe_defs.py, operators.py, ...)
config/ scanner signatures; the few example source addresses used in Tables 2 and 3 (actors.yaml, see the note in the file)
data/external/ operators' published IP ranges (snapshots), their provenance (sources.csv, fetch_log.txt), an industry list of forged-crawler IPs
tests/ check_reproduction.py (change units) and check_numbers.py (headline numbers of the paper)
run_all.sh runs the stages in order
Data layout expected by the scripts (all created by the pipeline; none distributed):
data/processed/date=YYYY-MM-DD/ 15 March 2025 - 18 September 2026, the study window
data/processed_after/date=YYYY-MM-DD/ 19 September - 18 October 2026, used only for the aftermath paragraph of Section 5.5
data/analysis/ every intermediate result and table
figures/ figures
pip install -r requirements.txt
RAW_DIR=/path/to/raw_csv ./run_all.sh preprocess # only with access to the raw daily files
./run_all.sh # dataset detect profile ablate identities verify after check
./run_all.sh detect # or any single stageverify resolves reverse-DNS records over the network; results are cached in data/external/rdns_cache.csv (not distributed), so verified shares
can differ slightly from the paper if DNS records have changed (the paper resolved them on 12--13 October 2026). check compares the regenerated
change units and numbers with the paper; several scripts also assert the counts they depend on and stop if the data differ.
| Paper | Stage | Scripts |
|---|---|---|
| Section 3, Figure 1 | dataset, detect | dataset_summary.py, plot_daily_overview.py |
| 4.1, 5.1, Table 1 | detect | build_daily_features.py, fit_baseline.py, detect_changepoints.py, fit_hmm.py (fast df=14 and slow df=6), changepoint_registry.py, table_units_latex.py; checks spline_df_sensitivity.py, section4_checks.py |
| 4.2, 5.2, Figure 2, Table 2 | profile | dimension_scan.py, case_events.py, case_coverage.py, case_table.py |
| 4.2, 5.3, Table 3 | ablate | ablation.py, ablation_effects.py, table_ablation.py |
| 4.3, 5.4, Table 4, Table 6 | verify | verify_rdns.py, rdns_name_check.py, rdns_identity_string.py, table4_verified.py, table4_paths.py, table4_latex.py, table_weak_rdns.py |
| 5.4, 5.5, Figures 3 and 4, Table 5 | identities | ai_names_table.py, ai_claims.py, ai_configs_by_category.py, ai_attribution.py, ai_independent.py, ai_figure.py, ai_timeline_figure.py, jul18_investigation.py, operator_claims.py, table_identity_catalogue.py |
| 5.5 (industry list, aftermath) | verify, after | greynoise_overlap.py, after_window.py |
Modules imported by the scripts above and not run on their own: common.py, probe_defs.py, operators.py, verify_openai.py, verify_operators.py,
dimension_matched.py, ua_analysis.py, spline_df_sensitivity.py. The changepoint dates are regenerated by the detect stage and compared with
tests/expected_registry.json; nothing is typed into the code.
data/external/*.json are the lists published by the operators (sources.csv gives the URL, retrieval date and note of each, fetch_log.txt the
fetches). They are snapshots: operators update their lists, so verified counts are lower bounds.