[BUG] race condition in OpenMLSplitTest when running tests in parallel

[Alm0stSurely]

Problem

When running tests in parallel with pytest-xdist (e.g., "pytest -n 3 tests/test_tasks/test_split.py"), one test under OpenMLSplitTest fails intermittently with an EOFError during pickle.load().

This was identified in CI job/63346513831 and reproduces roughly 1 out of 10 runs locally.

Analysis

The root cause is that all test instances share the same pickle cache file path (self.pd_filename). When multiple workers run concurrently:

1. Worker A creates the pickle cache file during test execution
2. Worker B reads the pickle cache file
3. Worker A's tearDown() deletes the file
4. Worker B's pickle.load() encounters a partially deleted file → EOFError

This is a classic race condition on shared filesystem state.

Solution

Use tempfile.mkdtemp() to create a unique temporary directory for each test instance, then copy the ARFF source file there. This ensures:

• Each test worker has its own isolated pickle cache file
• No shared state between parallel workers
• Automatic cleanup via shutil.rmtree() in tearDown()

The fix is minimal (10 insertions, 3 deletions) and doesn't change the test logic - only the test isolation mechanism.

Benchmarks / Testing

Ran 5 consecutive parallel test executions:

pytest -n 4 tests/test_tasks/test_split.py  # 5 times

All 15 test runs (3 tests × 5 runs) passed successfully. Before the fix, failures occurred ~10% of the time with parallel execution.

Fixes #1641

[omkar-334]

Looks like @Alm0stSurely is an Openclaw bot

[fkiraly]

Looks like this fixes the problem.

Congrats on your first "good first issue"!
How about something more challenging next?

[fkiraly]

Looks like @Alm0stSurely is an Openclaw bot

Yes, almost surely 😁

No reason though to not let it have its good first issue and let it move on to more challenging tasks. We can always make more good first issues tomorrow.

[codecov-commenter]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 52.82%. Comparing base (7feb2a3) to head (46685ba).

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #1643   +/-   ##
=======================================
  Coverage   52.82%   52.82%           
=======================================
  Files          37       37           
  Lines        4371     4371           
=======================================
  Hits         2309     2309           
  Misses       2062     2062           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[geetu040]

Nice, this looks good and solves the race condition problem. I have left a comment, though not a hard blocker.

[geetu040]

mkdtemp is fine, but I'd prefer TemporaryDirectory since it's safer and reduces cleanup code.

Then in tearDown you can simply call self._temp_dir.cleanup()

[fkiraly]

I agree with @geetu040

[fkiraly]

@Alm0stSurely, can you please react to the reviews?

[Alm0stSurely]

Thanks for the review @geetu040 and @fkiraly! Switched to tempfile.TemporaryDirectory() as suggested — much cleaner with automatic cleanup. Let me know if anything else needs adjusting.