Skip to content

test: make it easier to run tests for subsystems#15450

Closed
bcoe wants to merge 2 commits intonodejs:masterfrom
bcoe:test-refactor-pass-2
Closed

test: make it easier to run tests for subsystems#15450
bcoe wants to merge 2 commits intonodejs:masterfrom
bcoe:test-refactor-pass-2

Conversation

@bcoe
Copy link
Contributor

@bcoe bcoe commented Sep 17, 2017

Problem statement

  • it would be nice to be able to more easily run tests across a given
    subsystem, e.g., http, when their tests are spread out across
    multiple folders (pummel/, sequential/, etc.).
  • it would be nice to be able to report on coverage for a subset of
    tests/subsystems, speeding up the test coverage feedback loop.

Potential solution: split tests into subsystem-specific folders

this was proposed in #15437.

It became apparent in #15437, and in the discussion surrounding it, that
splitting tests into subsystem-specific folders potentially creates more
problems than it presents benefits:

  • you end up changing the git-history of 1000s of files.
  • at a glance, there's even more of a paradox of choice with regards
    to what folder a test should be placed in (what if a test interacts
    with two subsystems?).

Proposed solution: introduce suites_<subsystem> shorthand, make CI_JS_SUITES CI_NATIVE_SUITES configurable

Rather than reorganizing tests into an alternative folder structure, this
pull request introduces the concept of a suites_<subsystem> shorthand to
test.py.

Running:

$ python tools/test.py -J --mode=release suites_child-process

Would result in any tests matching the */test-child-process-* pattern being
executed, regardless of whether they are contained in test/sequential,
test/parallel, etc.

Other changes

  • CI_JS_SUITES and CI_NATIVE_SUITES are now configurable, this makes it
    possible to more easily run a coverage report against a subset of tests.
  • the default suites run by tools/test.py now uses a black-list rather than
    a white-list; this makes it harder to miss adding a new test folder to CI.
  • the known_issues suite of tests is no longer run as part of coverage; my
    concern being that we should count a line as covered, if it's exercised
    by a failing test.
Checklist
  • make -j4 test (UNIX), or vcbuild test (Windows) passes
  • documentation is changed or added
  • commit message follows commit guidelines
Affected core subsystem(s)

test,build

CC: @addaleax, @gibfahn @jasnell

Loading
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

build Issues and PRs related to build files or the CI. doc Issues and PRs related to the documentations. test Issues and PRs related to the tests. tools Issues and PRs related to the tools directory. windows Issues and PRs related to the Windows platform.

Projects

None yet

Development

Successfully merging this pull request may close these issues.