Skip to content

Pool LargeObject database connections via engine.raw_connection() - #1748

Merged
Virv12 merged 1 commit into
cms-dev:mainfrom
Virv12:pool-large-object-connections
Nov 17, 2026
Merged

Virv12 merged 1 commit into
cms-dev:mainfrom
Virv12:pool-large-object-connections

Conversation

@Virv12

@Virv12 Virv12 commented Nov 17, 2026

Copy link
Copy Markdown
Contributor

Reuse SQLAlchemy's connection pool in LargeObject.init(), close(), and unlink() instead of opening a new unpooled psycopg2 connection via custom_psycopg2_connection() on every large object operation. Also fix the copy-paste error message in LargeObject.readinto().

Benchmarks on round1 DB (1,409 MB, 10,033 submissions, 11,182 blobs):

  • ExportSubmissions pattern (10,033 files): 68.62 s -> 4.53 s (15.1x)
  • Worker FileCacher cold cache (1,083 files, 1.44 GiB): 17.96 s -> 8.18 s (2.2x)
  • cmsDumpExporter -c 4 (11,182 files): 131.09 s -> 36.05 s (3.6x; files 122.16 s -> 26.70 s)

@Virv12
Virv12 force-pushed the pool-large-object-connections branch from e592856 to fabd8cf Compare November 17, 2026 17:53
@Virv12
Virv12 enabled auto-merge (rebase) November 17, 2026 17:59
@codecov

codecov Bot commented Nov 17, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 82.75862% with 5 lines in your changes missing coverage. Please review.
✅ Project coverage is 56.86%. Comparing base (6979344) to head (7fc42a8).
✅ All tests successful. No failed tests found.

Files with missing lines Patch % Lines
cms/db/fsobject.py 82.75% 5 Missing ⚠️
Additional details and impacted files
@@           Coverage Diff           @@
##             main    #1748   +/-   ##
=======================================
  Coverage   56.86%   56.86%           
=======================================
  Files         344      344           
  Lines       28364    28374   +10     
=======================================
+ Hits        16128    16134    +6     
- Misses      12236    12240    +4     
Flag Coverage Δ
functionaltests 0.00% <0.00%> (ø)
unittests 56.86% <82.75%> (+<0.01%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@Virv12
Virv12 disabled auto-merge November 17, 2026 18:06
Reuse SQLAlchemy's connection pool in LargeObject.__init__(), close(),
and unlink() instead of opening a new unpooled psycopg2 connection via
custom_psycopg2_connection() on every large object operation. Also fix
the copy-paste error message in LargeObject.readinto().

Benchmarks on round1 DB (1,409 MB, 10,033 submissions, 11,182 blobs):
- ExportSubmissions pattern (10,033 files): 68.62 s -> 4.53 s (15.1x)
- Worker FileCacher cold cache (1,083 files, 1.44 GiB): 17.96 s -> 8.18 s (2.2x)
- cmsDumpExporter -c 4 (11,182 files): 131.09 s -> 36.05 s (3.6x; files 122.16 s -> 26.70 s)
@Virv12
Virv12 enabled auto-merge (rebase) November 17, 2026 18:20
@Virv12
Virv12 force-pushed the pool-large-object-connections branch from fabd8cf to 7fc42a8 Compare November 17, 2026 18:20
@Virv12
Virv12 merged commit 95408fb into cms-dev:main Nov 17, 2026
4 checks passed
@Virv12
Virv12 deleted the pool-large-object-connections branch November 17, 2026 18:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

2 participants