Skip to content

About

No description, website, or topics provided.

Resources

Stars

0 stars

Watchers

0 watching

Forks

Latest commit

 

History

2 Commits

Folders and files

Repository files navigation

Regime Shifts in Web Attack Traffic: code for reproduction

Code behind Regime Shifts in Web Attack Traffic: Uncovering the Rise of AI-Crawler Impersonation. It detects behavioral regime shifts in HTTP(S) honeypot traffic from four observable daily signals, profiles each shift and attributes it to source populations by actor ablation, and checks claimed crawler identities against operators' published IP ranges and reverse DNS.

The request logs cannot be redistributed (they are processed in situ through controlled interfaces of the data owner). This repository contains the code, the configuration, and snapshots of the operators' public IP-range lists; it does not contain raw or processed traffic.

Layout

pipeline/            raw daily CSV -> cleaned, scanner-tagged Parquet
analysis/            one script per paper artifact, plus the modules they share (common.py, probe_defs.py, operators.py, ...)
config/              scanner signatures; the few example source addresses used in Tables 2 and 3 (actors.yaml, see the note in the file)
data/external/       operators' published IP ranges (snapshots), their provenance (sources.csv, fetch_log.txt), an industry list of forged-crawler IPs
tests/               check_reproduction.py (change units) and check_numbers.py (headline numbers of the paper)
run_all.sh           runs the stages in order

Data layout expected by the scripts (all created by the pipeline; none distributed):

data/processed/date=YYYY-MM-DD/          15 March 2025 - 18 September 2026, the study window
data/processed_after/date=YYYY-MM-DD/    19 September - 18 October 2026, used only for the aftermath paragraph of Section 5.5
data/analysis/                           every intermediate result and table
figures/                                 figures

Run

pip install -r requirements.txt
RAW_DIR=/path/to/raw_csv ./run_all.sh preprocess     # only with access to the raw daily files
./run_all.sh                                         # dataset detect profile ablate identities verify after check
./run_all.sh detect                                  # or any single stage

verify resolves reverse-DNS records over the network; results are cached in data/external/rdns_cache.csv (not distributed), so verified shares can differ slightly from the paper if DNS records have changed (the paper resolved them on 12--13 October 2026). check compares the regenerated change units and numbers with the paper; several scripts also assert the counts they depend on and stop if the data differ.

Paper artifacts

Paper Stage Scripts
Section 3, Figure 1 dataset, detect dataset_summary.py, plot_daily_overview.py
4.1, 5.1, Table 1 detect build_daily_features.py, fit_baseline.py, detect_changepoints.py, fit_hmm.py (fast df=14 and slow df=6), changepoint_registry.py, table_units_latex.py; checks spline_df_sensitivity.py, section4_checks.py
4.2, 5.2, Figure 2, Table 2 profile dimension_scan.py, case_events.py, case_coverage.py, case_table.py
4.2, 5.3, Table 3 ablate ablation.py, ablation_effects.py, table_ablation.py
4.3, 5.4, Table 4, Table 6 verify verify_rdns.py, rdns_name_check.py, rdns_identity_string.py, table4_verified.py, table4_paths.py, table4_latex.py, table_weak_rdns.py
5.4, 5.5, Figures 3 and 4, Table 5 identities ai_names_table.py, ai_claims.py, ai_configs_by_category.py, ai_attribution.py, ai_independent.py, ai_figure.py, ai_timeline_figure.py, jul18_investigation.py, operator_claims.py, table_identity_catalogue.py
5.5 (industry list, aftermath) verify, after greynoise_overlap.py, after_window.py

Modules imported by the scripts above and not run on their own: common.py, probe_defs.py, operators.py, verify_openai.py, verify_operators.py, dimension_matched.py, ua_analysis.py, spline_df_sensitivity.py. The changepoint dates are regenerated by the detect stage and compared with tests/expected_registry.json; nothing is typed into the code.

Operator range snapshots

data/external/*.json are the lists published by the operators (sources.csv gives the URL, retrieval date and note of each, fetch_log.txt the fetches). They are snapshots: operators update their lists, so verified counts are lower bounds.

About

No description, website, or topics provided.

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages