Tests which check for every possible values of the parameters (and state for stateful systems) are impractical or even impossible for all but the most simple systems. Instead, tests try cover enough common cases and all the corner cases.
The ALU has 6 1-bit and 2 16-bit inputs. All possible inputs would be 2^(6+16+16) = 2^38 = 274877906944 possible inputs. The ALU test define just 36 checks.
Most of the tests are fairly simple and are only checking for gross errors.
For example the ALU test checks that each of the documented functions generates correct results for two different pairs of values. It assumes that the underlying parts are correct. It does not test that your implementation handles the control bits exactly as specified in all combinations.
Some of the tests, particularly the Xxx16 parts where the HDL has many copies of the same line but with different bus indices, could do a better job of testing that every one of the indices is correct.
So is it possible to write a HDL program that passes the tests but that is not a correct implementation of the chip?
Easy to do this intentionally, less likely to do it accidentally if all the underlying parts are good, except that the zr status is not tested for every out bit handled correctly.
seems my problem with the ALU was that in the description of f ("if f then out x+y else out x&y) i confused that description with the behaviour of a multiplexor and made the first input of the multiplexor i used as the first output of that description, when it actually needs to be the other way around (for the second test, testing f(x,y) = 1). When i got those multiplexor inputs fixed (swapped around) the test ran through until computing f(x,y) = y-1, so it passed 10 tests just from one variable change, but it has not passed them all, which is the next problem i need to solve, which will be fun :) thanks for your inputs