bpo-41486: zlib uses an UINT32_MAX sliding window for the output buffer#26143
Conversation
These funtions have an initial output buffer size parameter: - zlib.decompress(data, /, wbits=MAX_WBITS, bufsize=DEF_BUF_SIZE) - zlib.Decompress.flush([length]) If the initial size > UINT32_MAX, use an UINT32_MAX sliding window, instead of clamping to UINT32_MAX. Speed up when (the initial size == the actual size).
Author
|
@gpshead See this comment for details. I had used half a month to polish this patch, mainly trying various code styles. The code has been tested with this script: import zlib, time, random
_1G = 1024*1024*1024
UINT32_MAX = 0xFFFF_FFFF
def test(DATA_SIZE, INIT_BUFF_SIZES):
b = random.choice((b'a', b'b', b'c', b'd'))
raw_dat = b * DATA_SIZE
t1=time.perf_counter()
compressed_dat = zlib.compress(raw_dat, 1)
t2=time.perf_counter()
del raw_dat
print(f'compressed size: {len(compressed_dat)}, time: {t2-t1:.5f}')
for init_size in INIT_BUFF_SIZES:
t1=time.perf_counter()
decompressed_dat = zlib.decompress(compressed_dat, bufsize=init_size)
t2=time.perf_counter()
assert len(decompressed_dat) == DATA_SIZE
assert decompressed_dat.count(b) == DATA_SIZE
del decompressed_dat
print(f'data size: {DATA_SIZE:>8}, init buff size: {init_size:>8}, time: {t2-t1:.5f}')
print()
SIZE = _1G
test(SIZE, (100, SIZE, SIZE+1, UINT32_MAX, UINT32_MAX+1))
SIZE = UINT32_MAX
test(SIZE, (100, SIZE, SIZE+1, 2*UINT32_MAX+1))
SIZE = 10*_1G
test(SIZE, (_1G, SIZE-1, SIZE, SIZE+1, UINT32_MAX, UINT32_MAX+1,
2*UINT32_MAX+1, 3*UINT32_MAX+1, 4*UINT32_MAX+1))
SIZE = 20*_1G
test(SIZE, (3*_1G, SIZE-1, SIZE, SIZE+1, UINT32_MAX, UINT32_MAX+1,
2*UINT32_MAX+1, 3*UINT32_MAX+1, 4*UINT32_MAX+1, 5*UINT32_MAX+1, 6*UINT32_MAX+1)) |
|
This PR is stale because it has been open for 30 days with no activity. |
gpshead
approved these changes
Jul 5, 2021
Contributor
|
Thanks @animalize for the PR, and @gpshead for merging it 🌮🎉.. I'm working now to backport this PR to: 3.10. |
miss-islington
pushed a commit
to miss-islington/cpython
that referenced
this pull request
Jul 5, 2021
…er (pythonGH-26143) * zlib uses an UINT32_MAX sliding window for the output buffer These funtions have an initial output buffer size parameter: - zlib.decompress(data, /, wbits=MAX_WBITS, bufsize=DEF_BUF_SIZE) - zlib.Decompress.flush([length]) If the initial size > UINT32_MAX, use an UINT32_MAX sliding window, instead of clamping to UINT32_MAX. Speed up when (the initial size == the actual size). This fixes a memory consumption and copying performance regression in earlier 3.10 beta releases if someone used an output buffer larger than 4GiB with zlib.decompress. Reviewed-by: Gregory P. Smith (cherry picked from commit a9a69bb) Co-authored-by: Ma Lin <animalize@users.noreply.github.com>
|
GH-27025 is a backport of this pull request to the 3.10 branch. |
miss-islington
added a commit
that referenced
this pull request
Jul 5, 2021
…er (GH-26143) * zlib uses an UINT32_MAX sliding window for the output buffer These funtions have an initial output buffer size parameter: - zlib.decompress(data, /, wbits=MAX_WBITS, bufsize=DEF_BUF_SIZE) - zlib.Decompress.flush([length]) If the initial size > UINT32_MAX, use an UINT32_MAX sliding window, instead of clamping to UINT32_MAX. Speed up when (the initial size == the actual size). This fixes a memory consumption and copying performance regression in earlier 3.10 beta releases if someone used an output buffer larger than 4GiB with zlib.decompress. Reviewed-by: Gregory P. Smith (cherry picked from commit a9a69bb) Co-authored-by: Ma Lin <animalize@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
These funtions have an initial output buffer size parameter:
If the initial size >
UINT32_MAX, use anUINT32_MAXsliding window, instead of clamping toUINT32_MAX.Speed up when (the initial size == the actual size).
https://bugs.python.org/issue41486