[C++] Implement file reads for Azure filesystem

[Tom-Newton]

Describe the enhancement requested

Read support probably requires an Azure implementation for arrow::io::RandomAccessFile then that can be used to implement the OpenInputStream and OpenInputFile methods of the AzureFileSystem.

#12914 implemented all of these features so this will be largely a case of just extracting the relevant parts from there. One modification I would suggest compared to that would be to avoid branching logic based on whether the Azure storage account has the hierarchical namespace enabled. Utilising features of the hierarchical namespace can make renames and listing tasks faster but for just reading blobs it shouldn't make any difference.

If we want to use features of the hierarchical namespace that adds some complexities:

1. Makes things harder to test because its not supported by azurite Is there a plan to support AdlsGen2 (Datalake Storage) on top of blobstore emulator ?  Azure/Azurite#553
2. Its a bit difficult to query the storage account to determine if it supports hierarchical namespace. ServiceClient::GetAccountInfo() requires Storage Account Contributor permissions (https://learn.microsoft.com/en-us/rest/api/storageservices/get-blob-service-properties?tabs=azure-ad#authorization) which is quite significantly elevated. Hadoop solves this by essentially calling PathClient::GetAccessControlList() and if it raises an exception hierarchical namespace is not supported https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AzureBlobFileSystemStore.java#L356-L385.

Related Issues:

• [C++] Filesystem implementation for Azure Blob Storage #18014 (is a child of)

Component(s)

C++

[Tom-Newton]

cc @srilman since you mentioned you might be able to help out
If nobody else is able to pick it up I will probably start working on this within a couple of weeks.

[felipecrv]

@Tom-Newton are you taking over the work on #12914?

cc @zeroshade @bkietz

[Tom-Newton]

@Tom-Newton are you taking over the work on #12914?

I wouldn't say I'm taking over but I'm keen to push it along. So far @srilman and I have both merged PRs that implement a subset of what #12914 implemented.

[felipecrv]

@Tom-Newton are you taking over the work on #12914?

I wouldn't say I'm taking over but I'm keen to push it along. So far @srilman and I have both merged PRs that implement a subset of what #12914 implemented.

Thank you! Would you mind rebasing that PR to incorporate the work that has been done on the other PRs?

[Tom-Newton]

Thank you! Would you mind rebasing that PR to incorporate the work that has been done on the other PRs?

Plan was to keep merging small sections until it's feature complete. I think that's more likely to be done by extracting small sections from #12914 rather than rebasing it and trying to merge it all in one go. This was the approach taken for the GCS filesystem and recommend by @kou.

[felipecrv]

@Tom-Newton ok. This makes sense. Just make sure you make any work in progress you have as visible as possible so we can help along the way.

[Tom-Newton]

I'm going to start working on this.

[Tom-Newton]

I think the PR is ready for review #38269.
Hopefully what I've done makes sense, I'm still very inexperienced writing C++.