[Python] support conversion to decimal type from floats?

[asfimport]

We currently allow constructing a decimal array from decimal.Decimal objects or from ints:

In [14]: pa.array([1, 0], type=pa.decimal128(2))                                                                                                              
Out[14]: 
<pyarrow.lib.Decimal128Array object at 0x7f51fa2da818>
[
  1,
  0
]

In [31]: pa.array([decimal.Decimal('0.1'), decimal.Decimal('0.2')], pa.decimal128(2, 1))                                                                      
Out[31]: 
<pyarrow.lib.Decimal128Array object at 0x7fce671172b0>
[
  0.1,
  0.2
]

but not from floats (or strings):

In [18]: pa.array([0.1, 0.2], pa.decimal128(2))                                                                                                               
...
ArrowTypeError: int or Decimal object expected, got float

Is this something we would like to support?

There are for sure precision issues you run into, but if the decimal type is fully specified, it seems clear what the user wants. In general, since decimal objects in pandas are not that easy to work with, many people might have plain float columns that they want to convert to decimal.

Reporter: Joris Van den Bossche / @jorisvandenbossche

Related issues:

• [C++] Support lossy casts from decimal128 to float32 and float64/double (relates to)
• [C++] Implement casts from float/double to decimal128 (is related to)

_{Note: This issue was originally created as ARROW-5905. Please see the migration documentation for further details.}

[asfimport]

Antoine Pitrou / @pitrou:
ARROW-7011 will allow making this by calling the Array.cast() method. Is it enough for the use case?

[asfimport]

Joris Van den Bossche / @jorisvandenbossche:
That certainly solves the immediate need/functionality for being able to convert floats to decimal type in Arrow.

I would personally say that it would still be nice to be able to do this already upon conversion to Arrow in pa.array (which would also ensure it works when converting eg a pandas DataFrame with a float column to a pyarrow Table with a given pyarrow schema).
But I suppose that once Decimal128::FromReal is added, it should also be possible to use this in the python_to_arrow.cc? (meaning, we could leave the issue open as a possible future enhancement, if we want this)

[asfimport]

Joris Van den Bossche / @jorisvandenbossche:
Actually, also strings are not accepted (while internally python decimal objects are converted to strings first to convert them to decimal type) :

In [12]: pa.array(["0.1", "0.2"], pa.decimal128(2, 1))                                                                                                                                                             
...
ArrowTypeError: int or Decimal object expected, got str

(and casting strings to decimal doesn't work yet, that's probably worth another JIRA?)

So it's maybe a more general question: what types of values do we want to accept to construct a decimal array? Now we accept Python decimal.Decimal objects, but also ints, so why not floats or strings? After ARROW-7011, I think it would be a relatively easy addition to also accept also those types in DecimalFromPyObject (

arrow/cpp/src/arrow/python/decimal.cc

Lines 148 to 164 in bcbb3e2

	Status DecimalFromPyObject(PyObject* obj, const DecimalType& arrow_type,
	Decimal128* out) {
	DCHECK_NE(obj, NULLPTR);
	DCHECK_NE(out, NULLPTR);
	if (IsPyInteger(obj)) {
	// TODO: add a fast path for small-ish ints
	std::string string;
	RETURN_NOT_OK(PyObject_StdStringStr(obj, &string));
	return DecimalFromStdString(string, arrow_type, out);
	} else if (PyDecimal_Check(obj)) {
	return DecimalFromPythonDecimal(obj, arrow_type, out);
	} else {
	return Status::TypeError("int or Decimal object expected, got ",
	Py_TYPE(obj)->tp_name);
	}
	}

).

One disadvantage might be that the object-by-object conversion in the DecimalConverter (involving Python) might be less efficient than a cast in case of a typed float array as input.

[asfimport]

Antoine Pitrou / @pitrou:
That sounds reasonable to me, yes.

[github-actions]

This issue has been marked as stale because it has had no activity in the past 365 days. Please remove the stale label or comment below, or this issue will be closed in 14 days. If this improvement is still desired but has no current owner, please add the 'Status: needs champion' label.