Is there a plan to support AdlsGen2 (Datalake Storage) on top of blobstore emulator ?

[Arnaud-Nauwynck]

Which service(blob, file, queue, table) does this issue concern?

not existing currently in V3 : AdlsGen2 (Datalake)

This is a question: Is there a plan to support AdlsGen2 (Datalake Storage) on top of blobstore emulator ?
If yes, when would it be available?

Which version of the Azurite was used?

V3, unusable yet, laking ADlsGen2 support

Where do you get Azurite? (npm, DockerHub, NuGet, Visual Studio Code Extension)

npm

What's the Node.js version?

What problem was encountered?

Steps to reproduce the issue?

Have you found a mitigation/solution?

no
not able to develop on local with emulator, forced to connect to azure

[XiaoningLiu]

Hi @Arnaud-Nauwynck, can you further description your usage scenarios? For example, are you using a datalake gen2 account with or without hierarchical namespace enabled? Do you use any datalake storage sdks? which datalake gen2 features you are most interested? Interop between blob or datalake gen2?

[rh99]

As for me, I'd like to Spark and hadoop-azure, as described here: https://stackoverflow.com/questions/65050695/how-can-i-read-write-data-from-azurite-using-spark

[yuranos]

Hi @XiaoningLiu
We are using Gen 2 with Hierarchical namespace enabled.
We write our apps with Akka Streams and need to write to and read from to Datalake Gen 2.
Akka Streams has HDFS connector and since Datalake Gen 2 is HDFS-compatible, so far so good.

However, we need to add test coverage to our code. And not just basic unit tests, but something that can verify that integration is Ok end to end, but without being tightly coupled to network for real calls to Azure.

We implemented what we needed with testcontainers, but hadoop inside docker is a bit tricky to use and requires quite unconventional setup(with host resolutions, some env vars, permissions, etc.) while all we need is the filesystem for tests.

[KlaudiuszBryjaRelativity]

When do you release ADLS feature?
BTW do you have plan to add support for file share?

[karimdabbagh]

Any updates on this?

[XiaoningLiu]

Hi guys, we get your feedbacks and the asks to support datalake gen2 on Azurite. We do think it's a valid ask and keep it open for collecting requirements and feedbacks. Unluckily, it's not our current priorities yet.

In the same time, if possible, please try to leverage your contacts with Azure, and reach to Azure AdlsGen2 team for your asks directly. We have once talked this ask with AdlsGen2 team, it's better they can get more direct feedbacks to better understand the scenario and importance.

[blueww]

Another request for datalake gen2 #909