"The hidden tests are lossless" is empty by construction

#The original criterion

Our gate uses only the visible tests to decide what to let through. So one thing had to be verified:

There will be no candidate that "would pass the hidden tests but failed visible acceptance."

If there were, it would mean the visible tests miss some requirements, and the gate would let bad things through.

#Results on six datasets

#But R440Q found out why it is 0

So the criterion "visible filtering is lossless" has zero statistical power on these two task banks. 0/2394 does not mean "we verified that it is lossless"; it means "the definition of this task bank guarantees it holds."

#Three stricter ways of putting it

R440Q upgraded it into a main conclusion:

If a requirement cannot be compiled into executable tests, this mechanism has no free referee. It degrades into "ask a model" — and that is exactly the thing that measured worst.

#What the literature calls this

This is the concrete form, in this system, of the test oracle problem reviewed by Barr et al. (2015):

Generating test inputs and running the program is not the same as knowing whether the output is correct.

Large language models make generating candidate programs extremely cheap, but they do not remove the oracle problem. Worse, checking a model with a model does not automatically supply a basis for judgment.

#The practical consequences of this conclusion

#Honest boundaries

#Next

• How this premise became the gate: conformance-gate
• Why the ask-a-model path loses: committee-loses-to-majority
• How a preregistration was refuted by its own next experiment: preregistration