Remove support for Hadoop <3.3

[Fokko]

Describe the enhancement requested

Remove Hadoop 2 support

There is fallback logic in case it needs to seek within a file.

Component(s)

No response

[steveloughran]

I'm going to propose this is done with a cull of the hadoop-2 profile, with other cleanup code done more incrementally.

Or would you want HadoopStreams cleaned up at the same time? It'd be the nice tangible "this is why it is worthwhile" change?

[Fokko]

Hey @steveloughran! As a first PR, I'd love to remove the Hadoop 2 profile and the error-prone reflection. Next, we can do incremental cleanup. The discussion has been open on the dev-list for some time now, let me conclude it over there.

[steveloughran]

+1; will submit both.
One thing to consider here is actually dropping the hadoop 3 version to 3.3.0 to guarantee all API/tests are against that version. Avoids any accidental use of newer classes/methods/constants etc.

[Fokko]

One thing to consider here is actually dropping the hadoop 3 version to 3.3.0 to guarantee all API/tests are against that version. Avoids any accidental use of newer classes/methods/constants etc.

Yes, I was also thinking about that. I like that idea (or testing against both 3.3.x and 3.4.x).

[steveloughran]

I've actually been thinking about having a format-test module in Hadoop, which contains basic Parquet, avro &c tests which and then we run against object stores through the S3a, abfs and gcs stores. That way we can identify regressions fast and test against the development branches against live cloud infrastructure. There is also the option of a mini in-process HDFS cluster to test file R/W there... that can be done in parquet today.

[steveloughran]

w.r.t format testing, got some more thoughts there which would actually be

• make some/all the benchmark tests subclass of the (stable) hadoop fs contract tests in hadoop-common-test. (we can add more stability flags on that if needed; needs junit4 on the classpath)
• default to run against local file:// with/without vector IO
• module to run against hadoop-latest release
• (somewhere) point at s3, abfs, maybe gcs, using the same xml binding as we do for the usual store contract tests.

Someone still needs to provide keys for the target stores, so can't be run in the public CI tests...recurrent PITA there