[C++][Python] Support ExtensionType arrays in more kernels

[asfimport]

From a quick test (through Python), it seems that slice and take work, but the following not:

• cast: it could rely on the casting rules for the storage type. Or do we want that you explicitly have to take the storage array before casting?
• dictionary_encode / unique

Reporter: Joris Van den Bossche / @jorisvandenbossche
Watchers: Rok Mihevc / @rok

Related issues:

• [C++] C++ array kernels framework and execution buildout (umbrella issue) (is a child of)
• [C++][CSV] Add support for Extention type in csv reader (relates to)
• [C++][Python] Support casting an Extension type to its storage type (relates to)

_{Note: This issue was originally created as ARROW-5890. Please see the migration documentation for further details.}

[asfimport]

Antoine Pitrou / @pitrou:
We should make a list of compute functions that could might support extension types in a generic way. cc @ianmcook

cast wouldn't work: if you have an extension type representing IPv4 addresses on top of Int32s, the "cast to string" function should output the IPv4 addresses in dotted format ("1.2.3.4"), not in raw decimal format.

[asfimport]

Joris Van den Bossche / @jorisvandenbossche:
Specifically for the cast case, would we want to have some extension point in the ExtensionType to control the casting?

[asfimport]

Joris Van den Bossche / @jorisvandenbossche:
In addition to the dictionary_encode/unique/value_counts from above, some other kernels that could be supported (but are not at the moment):

• "equal" / "not_equal" (other comparisons are trickier, as that would rely on the ordering of the storage type being meaningful or not)

• "if_else", "case_when", "choose", "coalesce"

• "fill_null", "replace_with_mask"

• "is_in", "index", "index_in"

  Already working:

• "filter", "take"

• "is_null"

• "make_struct"

  The sorting-related ones (sort_indices, partition_nth_indices, etc) are probably tricky, as there is not guaranteed a link between the order of the storage type and the logical type.

[asfimport]

Antoine Pitrou / @pitrou:

Specifically for the cast case, would we want to have some extension point in the ExtensionType to control the casting?

Perhaps, though I'm not sure what it should look like. Perhaps like this:

class ExtensionType {
  // ...
  bool CanCastTo(const DataType& to_type);

  Result<Datum> CastTo(const Datum& value, std::shared_ptr<DataType> to_type,
                   const CastOptions& options = CastOptions::Safe(),
                   ExecContext* ctx = NULLPTR);
}

cc @bkietz

[asfimport]

Antoine Pitrou / @pitrou:
cc @lidavidm

[asfimport]

David Li / @lidavidm:
Casts are implemented by registering cast_boolean, cast_int8, ..., right? We could perhaps detect a registered "cast_extension_ipv4" or things like that.

[asfimport]

Antoine Pitrou / @pitrou:
That requires implementers of extension type to implement and register their own cast function, which sounds a bit more annoying.

[asfimport]

Clark Zinzow:
@pitrou I'm working on a tensor column extension type similar to this one and was hoping to allow users to interpret Parquet columns containing bytes blobs (e.g. images) as tensors by having them provide a schema for those columns, where the column's dtype is a tensor array extension type instantiated with the requisite data (shape, dtype, etc.) to cast that column as a tensor array. Since there isn't a static conversion between the bytes blobs and the underlying extension array dtype (both the shape and the underlying element dtype is parameterizable), it'd be nice if an extension type could register a cast function so we could use the shape and dtype context to properly interpret those bytes blobs.

[asfimport]

Clark Zinzow:
Does allowing extension type implementers to register a cast function sound reasonable? I might be able to take a stab at this (just casting) in the coming months.

[github-actions]

This issue has been marked as stale because it has had no activity in the past 365 days. Please remove the stale label or comment below, or this issue will be closed in 14 days. If this improvement is still desired but has no current owner, please add the 'Status: needs champion' label.