Rendered at 15:48:44 GMT+0000 (Coordinated Universal Time) with Wasmer Edge.
dgrunwald 8 days ago [-]
Caution with these functions: in most cases you need to check not only the error code, but also the `ptr` in the result.
Otherwise you end up with `to_int("0x1234") == 0` instead of the expected `std::nullopt`, because these functions return success if they matched a number at the beginning of the string.
pilif 8 days ago [-]
how can this be the ~5th iteration of a very wide-spread use-case and still contain a footgun?
The API looks like it's following best-practice, with a specific result type that also contains specific error information and yet, that's not enough and you still end up with edge-cases where things look like they're fine when they aren't.
amelius 8 days ago [-]
I suppose the reason is that sometimes you want to parse more than just the number. For example, numbers separated by commas. In that case you will have to call the function repeatedly from your own parsing routine and advance the pointer.
gpderetta 8 days ago [-]
Yes, if you are parsing in-place numbers in an existing string you do not necessarily know a-priori where the number end. You could have extra logic to look ahead to the end of the number before passing it from_chars, but a) it would require an additional pass and b) you could end up replicating part of the implementation of from chars.
from_chars is supposed to be a low lever routine, it must be possible to use it correctly, but it is not necessarily the most ergonomic.
jdbdndj 8 days ago [-]
I don't see a foot gun, you just need to check if you have consumed the whole input. Which is the norm in nearly any streaming api
randomNumber7 8 days ago [-]
I didn't see a foot gun either, but somehow my foot went missing.
8 days ago [-]
pilif 8 days ago [-]
I get that this is a super low level API, but yet, my expectation about an API that parses a buffer with length to a number and which has a specific enum for error cases as its return type would be that when asked to parse "9not a number" that I would get an error response and not a "the number you asked me to parse is 9 and everything went well" as the result.
The whole reason for returning a result enum would be so I don't forget to also check somewhere else whether something could possibly have gone wrong.
Especially when there is a specific result type.
gpderetta 8 days ago [-]
But now you need a different API to compute where a number starts and end and that API must use exactly the same definition of number.
pilif 8 days ago [-]
There could be an enum value for "read a partial number"
mort96 8 days ago [-]
In what way is it streaming? It takes at least the entire numeric string as a string in memory, you can't stream in more data as needed
khwrht 8 days ago [-]
That is a good question. The C++ stdlib has some truly bizarre APIs. I wonder if they should freeze std and work on std2.
otabdeveloper4 8 days ago [-]
from_chars is the correct API here. When you're parsing decimal numbers you want to do it with streaming semantics.
mort96 8 days ago [-]
Hm but there's nothing streaming about it? You need the entire numeric string in memory
Leherenn 8 days ago [-]
I think they meant the other way around: you can have a single string in memory containing multiple numbers, and process that string as a stream.
captainmuon 8 days ago [-]
I wonder why it is called `from_chars` and not `to_number` or similar. It's obvious what you are converting from, because you have to write down the argument `from_chars(someString)`, but you don't see what is coming out.
Someone 8 days ago [-]
As you indicate, you do see what you’re putting in, and that includes an argument holding a reference to the result of the conversion.
What’s coming out is a std::from_chars_result: a status code plus an indicator how much of the data was consumed.
What to name this depends on how you see this function. As part of a function on number types, from_chars is a good names. As part of a function on strings, to_int/to_long/etc are good names. As freestanding functions, chars_to_int (ugly, IMO), parse_int (with parse sort-of implying taking a string) are options.
I can see why they went for from_chars. Implementations will be more dependent on the output type than on character pointers, it’s more likely there will be more integral types in the future than that there will be a different way to specify character sequences, and it means adding a single function name.
beyondCritics 8 days ago [-]
As a matter of fact, this overloads the name and hence gives less hassle for generic coding.
pavlov 8 days ago [-]
Maybe “number” is too ambiguous because they’d have to define that “in this context a number means a float or integer type only.” The C++ standard also includes support for complex numbers.
vaylian 8 days ago [-]
Example from the website:
const std::string str { "12345678901234" };
int value = 0;
std::from_chars(str.data(),str.data() + str.size(), value);
On the third line: Why can I just pass in `value` like this? Shouldn't I use `&value` to pass in the output variable as a reference?
_gabe_ 8 days ago [-]
I saw another commenter explain that it’s passed by reference, but I agree with you. The C++ Core Guidelines even mention that it’s better to use raw pointers (or pass by value and return a value) in cases like this to make the intent clear.
A pointer parameter can be null and it doesn't make sense for this parameter to be null, so IMO a reference is the better choice here.
A non-const reference is just as clear a signal that the parameter may be modified as a non-const pointer. If there's no modification const ref should be used.
jdashg 8 days ago [-]
It's about clarity of intent at the call site. Passing by mutable ref looks like `foo`, same as passing by value, but passing mutability of a value by pointer is textually readably different: `&foo`. That's the purpose of the pass by pointer style.
You could choose to textually "tag" passing by mutable ref by passing `&foo` but this can rub people the wrong way, just like chaining pointer outvars with `&out_foo`.
jcelerier 8 days ago [-]
If you want clarity of intent define dummy in and out macros but please don't make clean reference-taking APIs a mess by turning them into pointers for no good reason
8 days ago [-]
gpderetta 8 days ago [-]
In theory a from_char with an optional output parameter could be useful to validate that the next field is a number and/or discard it without needing to parse it; it might even be worth optimizing for that case.
cornstalks 8 days ago [-]
nit: I don't think the Core Guidelines actually suggests it's better. It's just "one style." There are pros and cons to both styles.
hardlianotion 8 days ago [-]
In the function signature, value is a reference. You can think of reference as being a pointer that points to a fixed location but with value semantics.
So you can dip value into the function call and the function can assign to it, as it could to data pointed to by a pointer.
trealira 8 days ago [-]
&value would be a pointer to that integer. Instead, it's using references, which also use the & character. References use different syntax, but they're like pointers that can't be reassigned. Examples:
int x = 0;
int &r = x;
r += 1;
assert(x == 1);
int y;
r = y; // won't compile
void inc(int &r) { r += 1; }
int x = 0;
inc(x);
assert(x == 1);
The equivalent using pointers, like in C:
int x = 0;
int *p = &x;
*p += 1;
assert(x == 1);
int y;
p = &y;
void inc(int *p) { *p += 1; }
int x = 0;
inc(&x);
assert(x == 1);
trealira 8 days ago [-]
Mistake:
r = y; // won't compile
This will compile. It will be effectively the same as "x = y". The pointer equivalent is *p = y".
My apologies, as it's been a while since I've used C++.
Dwedit 8 days ago [-]
What if you try to convert a French floating point number that uses a comma instead of a dot?
jeffbee 8 days ago [-]
Then the programmer has made a mistake, because the behavior is the same as `strtod` in C locale, i.e. it stops parsing at the first comma.
You should think of `from_chars` as a function that accepts the outputs of `to_chars`, not as a general text understander.
einpoklum 8 days ago [-]
Note that this won't work (AFAICT) with Unicode strings and non-western-arabic digits, e.g.:
will fail to compile due to pointer incompatibility.
jcelerier 8 days ago [-]
std::from_chars / std::to_chars are explicitly made to only operate in the C locale, so basically ASCII texts. It's not meant for parsing user-input strings but rather text protocols with maximal efficiency (and locale support prevents that).
E.g. "۱۲۳٤" isn't as far as I know a valid number to put in a json file, http content-length or CSS property. Maybe it'd be ok in a CSV but realistically have you ever seen a big data csv encoded in non-ascii numbers?
gpderetta 8 days ago [-]
This way your spreadsheet could interpret the data it loads from a CVS differently depending on on the locale of the machine it is running on.
Of course nobody would design an application like that.
einpoklum 8 days ago [-]
> have you ever seen a big data csv encoded in non-ascii numbers?
Let me ask you this: How much data have you processed which comes from human user input in Arabic-speaking countries?
jcelerier 8 days ago [-]
If your CSV is defined to contain straight, unparsed user input it's wrong no matter the context.
If it's defined to contain numbers then if at some point between [user input] and [csv output] you don't have a pass where the value is parsed, validated and converted to one of your programming language's actual number data types before being passed to your CSV writer, then your code is wrong.
cout 8 days ago [-]
What would you suggest instead?
8 days ago [-]
cherryteastain 8 days ago [-]
Wish they returned std::expected<T, std::errc> instead of the weird from_chars_result struct
Longhanks 8 days ago [-]
std::expected is new in C++23, std::from_chars was introduces in C++17. Obviously, 2023 features were not available in 2017. Changing the return type now would break everybody's code, with very little benefit, you can easily wrap std::from_chars.
jlarocco 8 days ago [-]
Having an inconsistent, special case way of doing something in the name of backwards compatibility is the defining characteristic of C++.
Night_Thastus 7 days ago [-]
It's a tradeoff.
For example: With Python Python breaking changes are more common, and everyone complains about how much they have to go and fix every time something changes.
Damned if you do, damned if you don't. Either have good backwards compatibility but sloppy older parts of the language - or have a 'cleaner' overall language at the cost of crushing the community every time you want to change anything.
jlarocco 5 days ago [-]
I'm fully aware, just pointing out that C++ is particularly afflicted with backward compatibility issues; far more than other languages.
"The Design and Evolution of C++" gives the impression that even back in the 80s, major concessions were being made in the name of compatibility. At that time it was with C; now it's with previous versions of C++.
criddell 8 days ago [-]
If returning std::expected makes more sense, why not make it the primary signature and create a wrapper to maintain compatibility with old code?
Thorrez 8 days ago [-]
Then you would have to use 2 names: the primary name and the wrapper name. What would they be? Using 2 names wastes more of the namespace, and will confuse people. If the wrapper name isn't from_chars, then people's code will break when upgrading.
criddell 8 days ago [-]
Oh right. A different return type isn't enough to differentiate one function from another.
cherryteastain 8 days ago [-]
They could add an overload like std::expected<T, std::errc> from_chars(std::string_view). That way, since the input arguments are different, there'd be no issues about overload resolution.
gpderetta 8 days ago [-]
But overloads providing slightly different interfaces are a bane to generic programming.
Overloads are already confusing, if they can't be used generically there is really no point in reusing the name.
eMSF 8 days ago [-]
from_chars_result closely matches the strtol line of functions we've had for decades. Not returning the end position would be weird here!
lynx23 8 days ago [-]
Whoever introduced the rule to automatically delete :: in titles on a hacker site should be made to rethink their decisions. Its a silly rule. It should go.
pjmlp 8 days ago [-]
Most of those silly rules can be overriden after submission, there is a timeout to enforce the desired title instead by editing it.
MathMonkeyMan 8 days ago [-]
It would be an interesting piece to learn why.
Maybe it was for ::reasons::.
lynx23 8 days ago [-]
I am sure the intern who wrote a rule for the Apple VoiceOver speech synthesizer to special case the number 18 being pronounced english while the synth voice is set to german imagined to have a good reason at the time as well. However, that desn't make ther decision less stupid. "Vierzehn Uhr", "Fünfzehn Uhr", "Sechzehn Uhr", "Siebzehn Uhr", "Eighteen Uhr".
8 days ago [-]
criddell 8 days ago [-]
The author lists sprintf as one of the ways you can convert a string to a numbers. How would that work?
murderfs 8 days ago [-]
It's not a list of ways to convert strings to numbers, it's a list of string conversion functions (i.e. including the other direction). to_string is also listed there.
randomNumber7 8 days ago [-]
Did you mean sscanf? You could for example parse a number into a double variable by using "sscanf(str, "%d", &x)"
You can even parse a whole line of a csv file with multiple numbers in one call "sscanf(str, "%d;%d;%d\n, &d1, &d2, &d3)"
criddell 8 days ago [-]
No, the article lists sprintf/snprintf.
Another person already correctly pointed out that the author was listing a bunch of functions for number and string conversions in general.
randomNumber7 7 days ago [-]
printf makes no sense in this context, because the conversion happens in the other direction.
The article doesn't change that. It also lists scanf...
userbinator 8 days ago [-]
Wasn’t the old stuff good enough? Why do we need new methods? In short: because from_chars is low-level, and offers the best possible performance.
That sounds like marketing BS, especially when most likely these functions just call into or are implemented nearly identically to the old C functions which are already going to "offers the best possible performance".
I did some benchmarks, and the new routines are blazing fast![...]around 4.5x faster than stoi, 2.2x faster than atoi and almost 50x faster than istringstream
Are you sure that wasn't because the compiler decided to optimise away the function directly? I can believe it being faster than istringstream, since that has a ton of additional overhead.
After all, the source is here if you want to look into the horse's mouth:
Apart from various nuts and bolts optimizations (eg not using locales, better cache friendless, etc...) it also uses a novel algorithm which is an order of magnitude quicker for many floating points tasks (https://github.com/ulfjack/ryu).
If you actually want to learn about this, then watch the video I linked earlier.
secondcoming 8 days ago [-]
You profiled the code in your head?
ftrobro 8 days ago [-]
Integers are simple to parse, but from_chars is a great improvement when parsing floats. It's more standardized on different platforms than the old solutions (no need to worry about the locale, for example whether to use comma or dot as decimals separator) but also has more reliable performance in different compilers. The most advanced approaches to parsing floats can be surprisingly much faster than intermediately advanced approaches. The library used by GCC since version 12 (and also used by Chrome) claims to be 4 - 10 times faster than old strtod implementations:
They're locale independent, which the C stol, stof, etc functions are not.
a1369209993 8 days ago [-]
Yes, exactly. Which means that, while the speed gains are real, they only apply in cases where your libc is dangerously defective.
OvbiousError 8 days ago [-]
I agree with some of this, and the author could've made a better case for from/to_chars:
- Afaik stoi and friends depend on the locale, so it's not hard to believe this introduced additional overhead. The implicit locale dependency is also often very surprising.
- std::stoi only accepts std::string as input, so you're forced to allocate a string to use it. std::from_chars does not.
- from/to_chars don't throw. As far as I know this won't affect performance if it doesn't happen, it does mean you can use these functions in environments where exceptions are disabled.
eptcyka 8 days ago [-]
Locale env stuff is inherently thread unsafe, which is the main reason to never rely on it.
deeringc 8 days ago [-]
There's also the new Ryu algorithm that is being used, which is probably the biggest speed up.
A few months ago I optimized the parsing of a file and did some micro benchmarks.
I observed a similar speed-up compared to stoi and atoi (didn't bother to look at stringstream). Others already commented, that it's probably due to not supporting locales.
dexen 8 days ago [-]
For sake of example: a "locale-aware" number conversion routine would be the worst possible choice for parsing incoming network traffic. Beyond the performance concerns, there's the significant semantic difference in number formatting across cultures. Different conventions of decimal or thousands coma easily leading to subtle data errors or even security concerns.
Lastly, having a simple and narrowly specified conversion routines allows one to create a small sub-set of C++ standard library fit for constrained environments like embedded systems.
doug_durham 8 days ago [-]
I get that. However then they should name the function and put highly visible disclaimers in the documentation. Something like "from_ascii" instead of "from_chars". Also the documentation, including this blog post should be very clear that this function is only suitable for parsing machine to machine communications and should never be used for human input data. There is clearly a place for this type of function, however this blog post miscommunicates this in a potentially harmful way. When I read the post I presumed that this was a replacement for atoi() even though it had a confusing "non-locale" bullet point.
blux 8 days ago [-]
Did you verify their claims or are you just calling BS and that's it? The new functions are in fact much faster than their C equivalent (and yes, I did verify that).
Your original claim "I've not checked but this guy, and by extension the C++ standards committee who worked on this new API, are probably full of shit" was pretty extraordinary.
userbinator 8 days ago [-]
Look at the compiler-generated instructions yourself if you don't believe the source that I linked; in the cases I've seen all the extra new stuff just adds another layer on top of existing functions and if the former are faster the latter must necessarily also be.
The standards committee's purpose is to justify their own existence by coming up with new stuff all the time. Of course they're going to try to spin it as better in some way.
j16sdiz 8 days ago [-]
How not?
It compiles from sources, can be better in-lined, benefits from dead code elimination when you don't use unusual radix.
It also don't do locale based things.
> Not surprisingly, under all those layers of abstraction-hell, there's just a regular accumulation loop.
Your dismissive answer sounds so much like the one of a typical old-C style programmer that underestimate by 2 order of magnitude what compiler inlining can do.
Abstraction, genericity and inlining on a function like from_chars is currently exactly what you want.
userbinator 8 days ago [-]
It's my experience that says inlining only looks great in microbenchmarks but is absolutely horrible for cache usage and causes other things to become slower.
adev_ 7 days ago [-]
Which is wrong on almost any modern architecture.
For small size functions inlining is almost always preferable because (1) the prefetcher actually love that and (2) a cache-miss due to a mis predicted jump is way more costly than anything a bit of bloat will ever cost you.
The API looks like it's following best-practice, with a specific result type that also contains specific error information and yet, that's not enough and you still end up with edge-cases where things look like they're fine when they aren't.
from_chars is supposed to be a low lever routine, it must be possible to use it correctly, but it is not necessarily the most ergonomic.
The whole reason for returning a result enum would be so I don't forget to also check somewhere else whether something could possibly have gone wrong.
Especially when there is a specific result type.
What’s coming out is a std::from_chars_result: a status code plus an indicator how much of the data was consumed.
What to name this depends on how you see this function. As part of a function on number types, from_chars is a good names. As part of a function on strings, to_int/to_long/etc are good names. As freestanding functions, chars_to_int (ugly, IMO), parse_int (with parse sort-of implying taking a string) are options.
I can see why they went for from_chars. Implementations will be more dependent on the output type than on character pointers, it’s more likely there will be more integral types in the future than that there will be a different way to specify character sequences, and it means adding a single function name.
https://isocpp.org/wiki/faq/references#call-by-reference
A non-const reference is just as clear a signal that the parameter may be modified as a non-const pointer. If there's no modification const ref should be used.
You could choose to textually "tag" passing by mutable ref by passing `&foo` but this can rub people the wrong way, just like chaining pointer outvars with `&out_foo`.
So you can dip value into the function call and the function can assign to it, as it could to data pointed to by a pointer.
My apologies, as it's been a while since I've used C++.
You should think of `from_chars` as a function that accepts the outputs of `to_chars`, not as a general text understander.
E.g. "۱۲۳٤" isn't as far as I know a valid number to put in a json file, http content-length or CSS property. Maybe it'd be ok in a CSV but realistically have you ever seen a big data csv encoded in non-ascii numbers?
Of course nobody would design an application like that.
Let me ask you this: How much data have you processed which comes from human user input in Arabic-speaking countries?
For example: With Python Python breaking changes are more common, and everyone complains about how much they have to go and fix every time something changes.
Damned if you do, damned if you don't. Either have good backwards compatibility but sloppy older parts of the language - or have a 'cleaner' overall language at the cost of crushing the community every time you want to change anything.
"The Design and Evolution of C++" gives the impression that even back in the 80s, major concessions were being made in the name of compatibility. At that time it was with C; now it's with previous versions of C++.
Overloads are already confusing, if they can't be used generically there is really no point in reusing the name.
Maybe it was for ::reasons::.
You can even parse a whole line of a csv file with multiple numbers in one call "sscanf(str, "%d;%d;%d\n, &d1, &d2, &d3)"
Another person already correctly pointed out that the author was listing a bunch of functions for number and string conversions in general.
The article doesn't change that. It also lists scanf...
That sounds like marketing BS, especially when most likely these functions just call into or are implemented nearly identically to the old C functions which are already going to "offers the best possible performance".
I did some benchmarks, and the new routines are blazing fast![...]around 4.5x faster than stoi, 2.2x faster than atoi and almost 50x faster than istringstream
Are you sure that wasn't because the compiler decided to optimise away the function directly? I can believe it being faster than istringstream, since that has a ton of additional overhead.
After all, the source is here if you want to look into the horse's mouth:
https://raw.githubusercontent.com/gcc-mirror/gcc/master/libs...
Not surprisingly, under all those layers of abstraction-hell, there's just a regular accumulation loop.
MSFT's one is totally standards compliant and it is a very different beast: https://github.com/microsoft/STL/blob/main/stl/inc/charconv
Apart from various nuts and bolts optimizations (eg not using locales, better cache friendless, etc...) it also uses a novel algorithm which is an order of magnitude quicker for many floating points tasks (https://github.com/ulfjack/ryu).
If you actually want to learn about this, then watch the video I linked earlier.
https://github.com/fastfloat/fast_float
For more historical context:
https://lemire.me/blog/2020/03/10/fast-float-parsing-in-prac...
- Afaik stoi and friends depend on the locale, so it's not hard to believe this introduced additional overhead. The implicit locale dependency is also often very surprising.
- std::stoi only accepts std::string as input, so you're forced to allocate a string to use it. std::from_chars does not.
- from/to_chars don't throw. As far as I know this won't affect performance if it doesn't happen, it does mean you can use these functions in environments where exceptions are disabled.
https://github.com/ulfjack/ryu
https://github.com/jk-jeon/dragonbox
Lastly, having a simple and narrowly specified conversion routines allows one to create a small sub-set of C++ standard library fit for constrained environments like embedded systems.
"Extraordinary claims require extraordinary evidence."
The standards committee's purpose is to justify their own existence by coming up with new stuff all the time. Of course they're going to try to spin it as better in some way.
It compiles from sources, can be better in-lined, benefits from dead code elimination when you don't use unusual radix. It also don't do locale based things.
Removed `std::atoi` from the benchmarks since it was performing so poorly; not a contender. Should be easy to verify.
Rough results (last column is #iterations):
Your dismissive answer sounds so much like the one of a typical old-C style programmer that underestimate by 2 order of magnitude what compiler inlining can do.
Abstraction, genericity and inlining on a function like from_chars is currently exactly what you want.
For small size functions inlining is almost always preferable because (1) the prefetcher actually love that and (2) a cache-miss due to a mis predicted jump is way more costly than anything a bit of bloat will ever cost you.