test: make it easier to run tests for subsystems#15450
Closed
bcoe wants to merge 2 commits intonodejs:masterfrom
Closed
test: make it easier to run tests for subsystems#15450bcoe wants to merge 2 commits intonodejs:masterfrom
bcoe wants to merge 2 commits intonodejs:masterfrom
Conversation
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem statement
subsystem, e.g.,
http, when their tests are spread out acrossmultiple folders (
pummel/,sequential/, etc.).tests/subsystems, speeding up the test coverage feedback loop.
Potential solution: split tests into subsystem-specific folders
this was proposed in #15437.
It became apparent in #15437, and in the discussion surrounding it, that
splitting tests into subsystem-specific folders potentially creates more
problems than it presents benefits:
to what folder a test should be placed in (what if a test interacts
with two subsystems?).
Proposed solution: introduce suites_<subsystem> shorthand, make
CI_JS_SUITESCI_NATIVE_SUITESconfigurableRather than reorganizing tests into an alternative folder structure, this
pull request introduces the concept of a
suites_<subsystem>shorthand totest.py.Running:
Would result in any tests matching the
*/test-child-process-*pattern beingexecuted, regardless of whether they are contained in
test/sequential,test/parallel, etc.Other changes
CI_JS_SUITESandCI_NATIVE_SUITESare now configurable, this makes itpossible to more easily run a coverage report against a subset of tests.
tools/test.pynow uses a black-list rather thana white-list; this makes it harder to miss adding a new test folder to CI.
known_issuessuite of tests is no longer run as part of coverage; myconcern being that we should count a line as covered, if it's exercised
by a failing test.
Checklist
make -j4 test(UNIX), orvcbuild test(Windows) passesAffected core subsystem(s)
test,build
CC: @addaleax, @gibfahn @jasnell