fix(workflow-operator): validate base64 in HF binary-column fallback … - #7243
fix(workflow-operator): validate base64 in HF binary-column fallback …#7243PG1204 wants to merge 1 commit into
Conversation
…so plain text isn't decoded to garbage
Backport auto-label reportThis
|
Automated Reviewer SuggestionsBased on the
|
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #7243 +/- ##
============================================
- Coverage 82.63% 82.63% -0.01%
+ Complexity 4079 4076 -3
============================================
Files 1162 1162
Lines 46275 46275
Branches 5160 5160
============================================
- Hits 38241 38238 -3
Misses 6343 6343
- Partials 1691 1694 +3
*This pull request uses carry forward flags. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
|
| config | throughput | MB/s | latency | max Δ latest / 7d | |
|---|---|---|---|---|---|
| 🟢 | bs=10 sw=10 sl=64 | 648 | 0.396 | 15,487/18,990/18,990 us | 🟢 -15.8% / 🔴 +24.2% |
| 🔴 | bs=100 sw=10 sl=64 | 1,192 | 0.728 | 82,819/107,204/107,204 us | 🔴 +21.6% / 🟢 +19.6% |
| 🔴 | bs=1000 sw=10 sl=64 | 1,509 | 0.921 | 665,845/730,229/730,229 us | 🔴 +6.2% / 🟢 +47.4% |
Baseline details
Latest main 33cc5d1 from same runner
| config | metric | PR | latest main | 7d avg | Δ latest | Δ 7d |
|---|---|---|---|---|---|---|
| bs=10 sw=10 sl=64 | throughput | 648 tuples/sec | 627 tuples/sec | 782.98 tuples/sec | +3.3% | -17.2% |
| bs=10 sw=10 sl=64 | MB/s | 0.396 MB/s | 0.383 MB/s | 0.478 MB/s | +3.4% | -17.1% |
| bs=10 sw=10 sl=64 | p50 | 15,487 us | 15,900 us | 12,467 us | -2.6% | +24.2% |
| bs=10 sw=10 sl=64 | p95 | 18,990 us | 22,553 us | 15,695 us | -15.8% | +21.0% |
| bs=10 sw=10 sl=64 | p99 | 18,990 us | 22,553 us | 18,950 us | -15.8% | +0.2% |
| bs=100 sw=10 sl=64 | throughput | 1,192 tuples/sec | 1,384 tuples/sec | 996.92 tuples/sec | -13.9% | +19.6% |
| bs=100 sw=10 sl=64 | MB/s | 0.728 MB/s | 0.844 MB/s | 0.608 MB/s | -13.7% | +19.6% |
| bs=100 sw=10 sl=64 | p50 | 82,819 us | 71,384 us | 100,899 us | +16.0% | -17.9% |
| bs=100 sw=10 sl=64 | p95 | 107,204 us | 88,139 us | 107,385 us | +21.6% | -0.2% |
| bs=100 sw=10 sl=64 | p99 | 107,204 us | 88,139 us | 115,226 us | +21.6% | -7.0% |
| bs=1000 sw=10 sl=64 | throughput | 1,509 tuples/sec | 1,588 tuples/sec | 1,024 tuples/sec | -5.0% | +47.4% |
| bs=1000 sw=10 sl=64 | MB/s | 0.921 MB/s | 0.969 MB/s | 0.625 MB/s | -5.0% | +47.4% |
| bs=1000 sw=10 sl=64 | p50 | 665,845 us | 627,160 us | 989,300 us | +6.2% | -32.7% |
| bs=1000 sw=10 sl=64 | p95 | 730,229 us | 697,069 us | 1,031,689 us | +4.8% | -29.2% |
| bs=1000 sw=10 sl=64 | p99 | 730,229 us | 697,069 us | 1,058,394 us | +4.8% | -31.0% |
Raw CSV
config_idx,batch_size,schema_width,string_len,num_batches,total_ms,total_tuples,total_bytes,tuples_per_sec,mb_per_sec,lat_p50_us,lat_p95_us,lat_p99_us
0,10,10,64,20,308.57,200,128000,648,0.396,15487.04,18989.69,18989.69
1,100,10,64,20,1677.48,2000,1280000,1192,0.728,82819.19,107204.45,107204.45
2,1000,10,64,20,13250.38,20000,12800000,1509,0.921,665844.80,730229.45,730229.45
What changes were proposed in this PR?
In the HuggingFace operator's
_read_binary_value, the final fallback decoded a column value withbase64.b64decode(val). Withoutvalidate=True,b64decodesilently ignores characters outside the base64 alphabet, so ordinary text "successfully" decodes into garbage bytes and the intendedval.encode("utf-8")fallback rarely fires and a plain-text cell could be turned into garbage and sent as image/audio bytes.Adding
validate=Truemakesb64decodereject non-base64 input (spaces, punctuation, newlines), so real text now correctly falls through to UTF-8. Only this fallback call changes; thedata:-URL decodes are left as-is (genuine base64 that can legitimately contain whitespace).Any related issues, documentation, discussions?
Closes #7200.
How was this PR tested?
sbt "WorkflowOperator/testOnly org.apache.texera.amber.operator.huggingFace.* org.apache.texera.amber.util.PythonCodeRawInvalidTextSpec": passes (125 tests). Added a test asserting the fallback usesvalidate=True;PythonCodeRawInvalidTextSpecpy-compiles the generated Python. scalafmt clean.Was this PR authored or co-authored using generative AI tooling?
This PP was co-authored with Claude Opus 4.8 in compliance with ASF policy.