core: functional testing strategy

[beriberikix]

Document the design and execution plan for functional testing.

[dhermes]

@tseaver This is the bug you were referring to for me to claim and do the regression test work in?

[tseaver]

@dhermes Maybe. This one makes it a goal to document how we do functional tests (but maybe not to actually write any?).

There is some jargon skew between "functional" / "regression" / "integration" testing. For instance, I consider that adding a unit test which fails before fixing a bug could be called "regression" testing, as it ensures that the fix is not accidentally undone later. From conversations with @silvolu, I believe he uses "regression test" to refer to tests which actually exercise the server API, versus the unit tests which stub / mock it.

I contemplated writing a sample app which would serve both as the basis for narrative docs and as a possible set of regression tests (see my preliminary sketch).

[silvolu]

@tseaver I should probably say "regression testing done with functional tests" :)
What I mean is to ensure that changes tested only against mocks have not introduced new faults in the core set of functionalities by testing them against the real API.
@dhermes this issue is mostly a placeholder, you (or I) can open a new one if you want.

[dhermes]

Yeah I think this issue is just fine. As mentioned I'm tracking this in a dirty branch but hope to get real reviewable code soon.

[dhermes]

#281 also chipping away at this

[dhermes]

@silvolu As of #319 landing I am calling this "done".

Re-open if you feel it's necessary.