[3.15] gh-98820: Fix quadratic time in csv.Sniffer for quoted fields - #154867
Merged
Conversation
The regular expressions which look for a quoted field matched its body lazily, so a closing quote which was not followed by a delimiter was retried with every following quote, to the end of the sample. Match the body possessively instead: it ends at the first quote which is not doubled, as it does for a reader.
serhiy-storchaka
added
needs backport to 3.13
# Conflicts: # Lib/test/test_csv.py
sniff() now replaces \r\n and \r with \n before these regular expressions are used. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
Thanks @serhiy-storchaka for the PR 🌮🎉.. I'm working now to backport this PR to: 3.13, 3.14. |
|
GH-155117 is a backport of this pull request to the 3.14 branch. |
|
GH-155118 is a backport of this pull request to the 3.13 branch. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The four regular expressions which look for a quoted field match its body lazily. A closing quote which is not followed by a delimiter is therefore retried with every following quote, to the very end of the sample, and
findall()repeats that from every start position -- so the search is quadratic. On a single column of quoted fields none of them ever matches, so the whole cost is a failing scan.The body now ends at the first quote which is not doubled, which is the only closing quote a reader would accept, and is matched possessively.
A minimal reproducer needs no quoted fields at all --
',"x' * nis quadratic too, because two of the four patterns are anchored on a delimiter rather than a line.Being unambiguous is what makes it linear, so a few matches necessarily change: 16 of 560 files in the CSVsniffer corpora. Against their ground truth the net is positive -- delimiter 73.3% -> 74.5%, quotechar 81.3% -> 81.1%, doublequote unchanged.
main is not affected: the sniffer was rewritten there in gh-83273.