[C++][Python][Compute] String hex to numeric conversion and bit shifting

[asfimport]

Hi Apache Arrow Community,

thanks for the great work on this project - it is really a game changer. I've started to use it more frequently since more and more compute kernels became available.

However, I have a current requirement which I can't fulfill, yet. More concretely, I have hex values as strings. I need to convert them to numeric type (int) and apply bit shifting.

Currently, I can't find a way to do so. I tried type casts (string to int) like 

import pyarrow as pa

array = pa.array(["0x2001591", "0x2000848", "0x2000123"])
array.cast(pa.uint32())

However this results in ArrowInvalid: Failed to parse string: '0x2000123' as a scalar of type uint32. 

Moreover, I will need to apply bit shifting once converted. I'm not sure if there is anything comparable in the compute kernels, yet.

Thanks for any help in advance.
 
 

Reporter: Franz
Assignee: William Malpica / @wmalpica

Related issues:

• [C++] String algorithm library for StringArray/BinaryArray (is a child of)
• [C++][Python][Compute] Number to string hex conversion (is related to)

PRs and other links:

• GitHub Pull Request #11064
• GitHub Pull Request #11160
• GitHub Pull Request #11161

_{Note: This issue was originally created as ARROW-12657. Please see the migration documentation for further details.}

[asfimport]

Eduardo Ponce / @edponce:
Generally, compute functions in Arrow perform a single type of operation. This request can be resolved via two independent compute functions:

• Conversion of hex string to numeric output

• Bit shifting of numeric input

  Arrow already contains compute functions for bit shifting numeric inputs, so only the hex conversion is missing.

[asfimport]

Eduardo Ponce / @edponce:
The inverse operation, commonly called hex() converts an integer into a hex string.

There are several things to consider for the hex-to-number conversion:
Python/numpy use type casting for hex conversion to integers. Only int() supports hex conversion, but not int32(), int64(), etc.

int('0xa', base=16)      # 'base' option is mandatory
int('0xA', base=16)      # case-insensitive
int('a', base=16)        # '0x' is optional
int('000000a', base=16)  # if '0x' is not given, any number of '0's to the left of the first non-zero value are ignored
int('     a\t', base=16) # left/right whitespace are ignored

R uses a function for hex conversion

as.numeric('0xa')       # equivalently, as.integer('0xa')
as.numeric('0XA')       # case-insensitive
as.numeric('0xa')       # '0x' is mandatory, there is no 'base' option
as.numeric('000000xa')  # Warning: returns NA. '0's to the left are not supported
as.numeric('0xa')       # left/right whitespace are ignored

SQL uses a conversion function for hex-to-number

SELECT CONVERT(int, 0xa)
SELECT CONVERT(int, 0XA)     -- case-insensitive
SELECT CONVERT(int, 0xa)     -- '0x' is mandatory, there is no 'base' option
SELECT CONVERT(int, 0000xa)  -- Error. '0's to the left are not supported
SELECT CONVERT(int, '0xa')   -- Error. Literal strings are not supported

[asfimport]

Eduardo Ponce / @edponce:
Also, Arrow already supports some hex parsing support by HexEncode and ParseHexValue in util/string.cc (and related PR).

[asfimport]

William Malpica / @wmalpica:
The ParseHexValue function takes in only two char at time and outputs a uint_8. We could reuse this, but it gets fairly awkward if your string has an odd number of hex digits. I think it would be better to implement something more flexible

[asfimport]

Eduardo Ponce / @edponce:
Based on the different hex formats supported by other tools, I would suggest to support case-insensitive (including '0x') and make it optional. The reason for making it optional is that the function name already states that it is a hex value, so forcing '0x' seems redundant but acceptable.

[asfimport]

Eduardo Ponce / @edponce:
After reviewing the PR, noticed that the string parsing is done along the unsigned parsing, so not having prefix of '0x' would require to first traverse the string and validate it as a hex string. Also, in addition to the implicit hex parsing, we could consider adding a kernel hex_to_str that would basically be a wrapper over this (but will need to check/add '0x'). This would allow hex values stored without the prefix be parsed correctly.

[asfimport]

Antoine Pitrou / @pitrou:
Issue resolved by pull request 11161
#11161