Repository navigation
fix: let int() decide what an int is in handle_arg_string - #146
Open
abhishekcpp007 wants to merge 1 commit into
Open
abhishekcpp007 wants to merge 1 commit into
abhishekcpp007 wants to merge 1 commit into
Conversation
`handle_arg_string` used `str.isnumeric()` to decide whether a `--data_args`
value was an integer. That predicate disagrees with `int()` in both directions:
- `"-5".isnumeric()` is False, so a negative integer skipped the int branch and
came back from `float()` as `-5.0`. `simple_parse_args_string` feeds straight
into `load_dataset(**kwargs)`, so `--data_args num_proc=-1` silently handed a
float to a parameter that wants an int. No error, just the wrong type.
- `"½".isnumeric()` is True but `int("½")` raises, so a Unicode numeral raised an
unhandled ValueError. The same applies to `"²"` and to Roman numerals such as
`"Ⅷ"`. At the call site in `__main__.py` that ValueError is caught by the
`load_from_disk` handler and resurfaces as a dataset error, which points at
the wrong thing.
Trying `int()` and falling through on ValueError removes both. Values that
already parsed keep parsing the same way, including fullwidth digits.
Adds tests/test_utils.py covering signed integers, floats, Unicode numerals,
booleans, plain strings, and the end-to-end `simple_parse_args_string` path.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this fixes
handle_arg_stringinsparsify/utils.pyusesstr.isnumeric()to decide whether a value is aninteger. That predicate disagrees with
int()in both directions, so the function gets twodifferent things wrong.
A negative integer silently becomes a float.
"-5".isnumeric()isFalse, so-5skips theint branch and comes back out of
float()as-5.0.simple_parse_args_stringfeeds directlyinto
load_dataset(**kwargs)in__main__.py, so--data_args num_proc=-1hands a float to aparameter that wants an int. No error, just the wrong type.
A Unicode numeral raises.
"½".isnumeric()isTruebutint("½")raisesValueError, andnothing catches it on that branch. Same for
"²"and for Roman numerals such as"Ⅷ". At thecall site that
ValueErroris swallowed by theload_from_diskhandler and resurfaces as adataset error, which points at the wrong thing.
Current behaviour:
Fix
Try
int()and fall through onValueError, instead of askingisnumeric()first. The predicateand the parser are then the same thing, so they cannot disagree.
Values that already parsed keep parsing the same way, fullwidth digits (
"123"to123)included.
"3.5"still reachesfloat, and anything else still comes back as the original string.Tests
tests/test_utils.pycovers signed integers, floats including1e3, the three Unicode numeralsabove, booleans, plain strings, and the end-to-end
simple_parse_args_stringpath with a negativenum_proc. The signed-integer and Unicode cases fail onmain.The module it tests is pure stdlib, so the file needs no GPU and no model.
Checks I ran
ruff check sparsify/utils.py tests/test_utils.py— cleanblack --checkon both — no changespytestagainst the repo.Saying that rather than reporting a pass I did not see. The two functions under test are pure
stdlib, so I re-ran every assertion in the new test file against the patched implementations in
isolation and all 20 pass. CI is the real run.