gh-154863: Fix iconv encoding of ISO-2022-CN-EXT returning empty bytes - #154899
Open
fedonman wants to merge 2 commits into
Open
gh-154863: Fix iconv encoding of ISO-2022-CN-EXT returning empty bytes#154899fedonman wants to merge 2 commits into
fedonman wants to merge 2 commits into
Conversation
…y bytes The flush that emits the pending shift sequence can report a nonreversible conversion, which the loop mistook for a substituted character. It then retried while still flushing, converted nothing and returned the empty output buffer.
macOS substitutes '?' for the Chinese character in ISO-2022-CN instead of encoding it, so check only that the surrounding ASCII survives, which is what the fix is about.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Encoding ends with an
iconv()call on a NULL input to flush the pending shift sequence.That call's return value counts nonreversible conversions, and glibc makes it 1 for
ISO-2022-CN-EXT. The loop read any positive count as "iconv substituted an unencodable
character" and restarted one code point at a time, but
flushingwas still set, so theretry fed no input, returned 0 and broke out with the output buffer already reset to
empty.
Checking
flushingbefore the count fixes it. The retry path cannot be reached whileflushing, so nothing else changes.
I compared the codec against raw iconv for all 1180 encodings
iconv -llists here.ISO-2022-CN-EXT and its alias were the only two that disagreed, and none do now. The test
also covers ISO-2022-CN and ISO-2022-JP, where the flush count is 0, so those pass either
way and act as controls. It fails without the change.
The
(b'', 1)consumed count in the report is not a second bug.iconv_encode()alwaysconsumes the whole string, so the length is correct.
The iconv codecs are new in 3.16, so there is no NEWS entry.