[C++] Filesystem implementation for Azure Blob Storage

[asfimport]

Subissues:

• [C++] Skeleton for Azure Filesystem Implementation #35903
• [C++] Configure azurite in preparation for testing Azure C++ filesystem #36886
• [C++] Implement file reads for Azure filesystem #37511
• [C++] Return filesystem properties not user defined metadata in Azure file reads #38330
• [C++] Implement file writes for Azure filesystem #38333
• [C++] Implement GetFileInfo for a single file in Azure filesystem #38335
• [C++] Implement GetFileInfo selector for Azure filesystem #38597
• [C++] Implement all important authentication methods for Azure filesystem #38598
  • [C++][FS][Azure] Default credential authentication #39262
  • [C++][FS][Azure] Workload Identity Authentication #39318
  • [C++][FS][Azure] Managed Identity authentication #39320
  • [C++][FS][Azure] Client secret authentication #39343
  • [C++][FS][Azure] Azure CLI authentation #39344
  • [C++][FS][Azure] Environment credential authentication #39345
  • [C++][FS] Use default Azure credentials implicitly and support anonymous credentials explictly #39449
• [C++][FS][Azure] Implement CreateDir() #38699
• [C++][FS][Azure] Implement DeleteDir() #38700
• [C++][FS][Azure] Implement DeleteDirContents() #38701
• [C++][FS][Azure] Implement DeleteRootDirContents() #38702
• [C++][FS][Azure] Implement DeleteFile() #38703
• [C++][FS][Azure] Implement DeleteFile() for flat-namespace storage accounts #40074
• [C++][FS][Azure] Implement Move() #38704
• [C++][FS][Azure] Implement CopyFile() #38705
• [C++][FS][Azure] Rename AzurePath to AzureLocation #38758
• [C++][FS][Azure] Should GetFileInfo() against a directory always return true without hierarchical namespace support? #38772
• [C++][FS][Azure] Add allow_container_deletion option #38999
• [C++][FS][Azure] Use the generic filesystem tests #39069
  • [C++][FS][Azure] Generic filesystem tests for DeleteDir/DeleteDirContents/GetFileInfoSelector failed with Azure #41034
• [C++][FS][Azure] Make mutations of test data explicit in azurefs_test.cc #39119
• [C++] Remove the AzureOptions::backend option #39292
• [C++] Inform caller of Azure Storage container not-existing when checking for HNS support #39297
• [C++][FS][Azure] Implement Move() for flat namespace storage accounts #40025
• [C++][FS][Azure] Include user set metadata in ObjectInputFile::ReadMetadata #40026
• [C++][FS][Azure] Add AzureFileSystem support to FileSystemFromUri() #40028
• [C++][FS][Azure] Expose parallel transfer config options available in the Azure SDK #40035
• [C++][FS][Azure] Optimise ObjectAppendStream::DoAppend in the case of many small appends #40036
• [C++][FS][Azure] Prevent reading or writing directory marker blobs #40037
• [C++][FS][Azure] Fix CreateDir and DeleteDir on hierarchical namespace accounts where the directory has a trailing / #40052
• [C++][FS][Azure] Consider how and if to fix disallowed characters in file metadata #40057
• ...

---

Reporter: Wes McKinney / @wesm
Assignee: Shefali Singh

Related issues:

• [C++] Filesystem implementation for Azure Data Lake (is cloned by)
• [R] Expose Azure Blob Storage filesystem (is depended upon by)

Original Issue Attachments:

• azfs.cc
• azfs.h

PRs and other links:

• GitHub Pull Request #12914

_{Note: This issue was originally created as ARROW-2034. Please see the migration documentation for further details.}

[asfimport]

Wes McKinney / @wesm:
I see that TileDB (MIT license) has built a C++ wrapper for Azure

https://github.com/TileDB-Inc/TileDB/blob/dev/tiledb/sm/filesystem/azure.cc

No this is not moved to fsspec, this is a C++ ticket

[asfimport]

Antoine Pitrou / @pitrou:
I'm not sure I understand the relationship between Blob Store and Data Lake. Is Data Lake a higher-level layer above Blob Store? Or are they two different services that would need separate filesystem implementations?

[asfimport]

Antoine Pitrou / @pitrou:
According to https://docs.microsoft.com/en-us/azure/storage/blobs/data-lake-storage-introduction ,

Data Lake Storage Gen2 converges the capabilities of Azure Data Lake Storage Gen1 with Azure Blob storage. For example, Data Lake Storage Gen2 provides file system semantics, file-level security, and scale. Because these capabilities are built on Blob storage, you'll also get low-cost, tiered storage, with high availability/disaster recovery capabilities.
I'm not sure this means the same C++ API can be used to access both, though.

[asfimport]

Uwe Korn / @xhochy:
It is as confusing in reality, here is what they all are (I'm though already 1 year outdated on this):

• Blob Store: Like S3, simple but limited API
• Data Lake Gen 1: HDFS-like deployment with different but more user-friendly API / attributes
• Data Lake Gen 2: Some improvements were made to the Blob Store so that there is no need for a special (more expensive) Data Lake service anymore, everything is now on the Blob Store. A new set of APIs was though released that exposes some nice features that the initial Blob Store API didn't have, probably for marketing purposes this was named Data Lake Gen 2 although technically Blob Store Gen 2 would have been more appropriate.

[asfimport]

Antoine Pitrou / @pitrou:
Ha, and here are some interesting resources:

• https://docs.microsoft.com/en-us/azure/storage/blobs/data-lake-storage-known-issues : "Blob APIs and Data Lake Storage Gen2 APIs can operate on the same data." but "You cannot use blob API and Data Lake Storage APIs to write to the same instance of a file" (sounds ok?)

• https://docs.microsoft.com/en-us/rest/api/storageservices/understanding-block-blobs--append-blobs--and-page-blobs : three different kinds of blobs. "Block blobs" sound most useful, though "append blobs" may work too.

• https://docs.microsoft.com/en-us/azure/storage/blobs/data-lake-storage-namespace : a Data Lake admin can configure a it with a hierarchical filesystem semantics.  Apparently it may enable different APIs (??).

   

   

[asfimport]

Yesh:
There is also https://github.com/Azure/azure-sdk-for-cpp which I’ve tested against adl gen2 .

[asfimport]

Antoine Pitrou / @pitrou:
Various pointers about the Azure C++ SDK:

https://devblogs.microsoft.com/azure-sdk/cppintro/#example-code-using-the-c-storage-blob-client-libraryhttps://devblogs.microsoft.com/azure-sdk/cppintro/

https://github.com/Azure/azure-sdk-for-cpp

https://github.com/Azure/azure-sdk-for-cpp/blob/master/sdk/storage/azure-storage-blobs/sample/blob_getting_started.cpp

https://azure.github.io/azure-sdk-for-cpp/ (note the API docs are segregated per component, with separate land pages for Core, Storage, Blog Storage, etc.)

Note that the SDK requires C++14.

 

[asfimport]

Tom Augspurger / @TomAugspurger:
Does Arrow support C+\14 features now (or more specifically, is the SDK being C\14 a problem?) From https://issues.apache.org/jira/browse/ARROW-13744 it seems like C\14 is at least tested, but https://github.com/apache/arrow/blame/master/docs/source/developers/cpp/building.rst#L40 says a "A C\+11-enabled compiler. " is required.

[asfimport]

Neal Richardson / @nealrichardson:
I'm not an expert here, but I think we could use C+14 if required and if the compiler supports it. If the compiler doesn't support C 14, we wouldn't be able to build the azure sdk. So the line would be that Arrow requires C 11 at a minimum, and some features are only available with C+14.

[asfimport]

Antoine Pitrou / @pitrou:
We require only C+11 in the codebase. We might add C+14-requiring optional components if desired, but that will add complication to the build setup.

[asfimport]

Shashanka Balakuntala Srinivasa:
hi @pitrou , we were looking into implementing this feature from our side. I did try to compile the whole arrow code base with c+14 and ran unit tests as well. Everything is passing in local and as mentioned before : [ARROW-13744] [CI] c+14 and 17 nightly job fails - ASF JIRA (apache.org) ticket mentions we have daily build run for validation which are passing. 

Since the azure sdks depend on c+14 features, and since we have the code compiled in c 14, can we look into upgrading the c+ version to 14? 
Let me know if there are any issues. I will be happy to take those and work on them.

[asfimport]

Antoine Pitrou / @pitrou:
[~balakuntala] We can probably require C+\14 for the Azure filesystem implementation only, but the rest of Arrow should remain C+11-compatible.

[asfimport]

Dean MacGregor:
If someone wants to work on this but doesn't have an Azure account let me know.  I can make a storage account for this development/testing

[asfimport]

Bipin Mathew:
I have a rudimentary implementation of this that supports import and export to Azure. I tried to align as closely as possible to the s3 implementation, however as I needed it only for a specific use case and have not had a chance to implement all the same methods. Depending on bandwidth, I could probably build out the implementation further. I have never contributed to this project. Do all endpoints need to be implemented before it can be merged into the code base? Can it be built out over time? I attach what I have so far here.

azfs.hazfs.cc

[asfimport]

Antoine Pitrou / @pitrou:
[~mathewb001] Well, there's a Github PR open already, did you take a look?