代码之家 › 专栏 › 技术社区 › Christian Heigele

如果存在TestClassSetup,则在执行期间不会显示TAP结果

unit-testing matlab

Christian Heigele · 技术社区 · 8 年前

我有个问题。我们使用Matlab测试框架来分析我们的代码库。为了在我们的CI系统TeamCity中跟踪结果,我们使用TAP格式。这里我们有以下问题:

如果测试包含TestClassSetup部分,则TAP结果只显示在末尾,而不显示在执行过程中。这给我们带来了一些问题:

CI系统创建的时间戳可能不正确
如果信息输出是在测试用例中给出的,则它不会与断言语句一起显示。

我们使用以下(简化的)代码片段来标识出TestSuite并执行它:

testSuite = matlab.unittest.TestSuite.fromFolder('.');
runner = matlab.unittest.TestRunner.withNoPlugins();
runner.addPlugin(matlab.unittest.plugins.TAPPlugin.producingOriginalFormat());
results = runner.run(testSuite);

对于以下两个类别,问题是可重复的(内容当然是无意义的…)

classdef SomeTest < matlab.unittest.TestCase
   properties (TestParameter)
       param = {1, 2};
       param2 = {1, 2};
   end        
   methods (TestClassSetup)
       function someSetup(testCase)
           pause(0.1);
       end
   end    
   methods (Test)
       function testMethod(self, param, param2)
          fprintf('I''m here, with the params: %f/%f\n', param, param2);
          pause(0.1);
          self.assertGreaterThan(param, param2); 
       end
   end    
end

classdef SomeOtherTest < matlab.unittest.TestCase
   properties (TestParameter)
       param = {1, 2};
       param2 = {1, 2};
   end    

   methods (Test)
       function testMethod(self, param, param2)
          fprintf('I''m here, with the params: %f/%f\n', param, param2);
          pause(0.1);
          self.assertGreaterThan(param, param2); 
       end
   end

end

如果将所有三个文件复制到一个文件夹中,并执行runner,您将看到输出(断言被简化):

1..8
I'm here, with the params: 1.000000/1.000000
not ok 1 - SomeOtherTest/testMethod(param=1,param2=1)
# ================================================================================
# Assertion failed in SomeOtherTest/testMethod(param=1,param2=1) and it did not run to completion.
# ================================================================================
# 
I'm here, with the params: 1.000000/2.000000
not ok 2 - SomeOtherTest/testMethod(param=1,param2=2)
# ================================================================================
# Assertion failed in SomeOtherTest/testMethod(param=1,param2=2) and it did not run to completion.
# ================================================================================
# 
I'm here, with the params: 2.000000/1.000000
ok 3 - SomeOtherTest/testMethod(param=2,param2=1)
I'm here, with the params: 2.000000/2.000000
not ok 4 - SomeOtherTest/testMethod(param=2,param2=2)
# ================================================================================
# Assertion failed in SomeOtherTest/testMethod(param=2,param2=2) and it did not run to completion.
# ================================================================================
# 
I'm here, with the params: 1.000000/1.000000
I'm here, with the params: 1.000000/2.000000
I'm here, with the params: 2.000000/1.000000
I'm here, with the params: 2.000000/2.000000
not ok 5 - SomeTest/testMethod(param=1,param2=1)
# ================================================================================
# Assertion failed in SomeTest/testMethod(param=1,param2=1) and it did not run to completion.
# ================================================================================
# 
not ok 6 - SomeTest/testMethod(param=1,param2=2)
# ================================================================================
# Assertion failed in SomeTest/testMethod(param=1,param2=2) and it did not run to completion.
# ================================================================================
# 
ok 7 - SomeTest/testMethod(param=2,param2=1)
not ok 8 - SomeTest/testMethod(param=2,param2=2)
# ================================================================================
# Assertion failed in SomeTest/testMethod(param=2,param2=2) and it did not run to completion.
# ================================================================================

我期望的是,在第二种情况下,断言语句(和ok/notok TAP标志)也与fprintf语句对齐。

有人知道吗?

1 回复 | 直到 8 年前

Andy Campbell 8 年前

TestClassSetup的出现“推迟”TAP输出的打印是因为TAP输出是流式格式,如果有任何TestClassSetup代码,框架实际上还不知道测试是否通过。例如,如果在TestClassTeardown中失败(或者通过TestClassSetup中的addTeardown函数调用),最终结果是共享TestClassSetup代码的所有测试都将失败。

然而,考虑到它的流格式,TAPPLugin希望在知道结果后立即打印出结果。实际上,有一个TestRunnerPlugin方法专门为这种情况设计,即 reportFinalizedResult method .

这里的基本问题是,我建议您避免使用disp或fprintf打印到日志中。这不太理想,因为插件对使用fprintf打印的任何信息都没有任何洞察力。此外,除了matlab命令行之外,不能将此信息重定向到任何其他地方。

但是,如果您使用 testCase.log 方法将在正确的位置得到诊断,并且它将更加灵活。您可以在不同的级别登录,以便可以根据需要打开或关闭它,并控制是否要查看它。它不仅可以进入命令行,而且可以更方便地进入TAP流、junit xml和pdf/html测试报告等。对于您的情况,如下所示:

runner = matlab.unittest.TestRunner.withNoPlugins();
runner.addPlugin(matlab.unittest.plugins.TAPPlugin.producingOriginalFormat());
results = runner.run(testSuite);

第一次运行时,没有看到任何日志调用,因为它是以详细信息“3”记录的,并且默认值较低(我相信是级别1)

1..8
not ok 1 - SomeOtherTest/testMethod(param=value1,param2=value1)
# ================================================================================
# Assertion failed in SomeOtherTest/testMethod(param=value1,param2=value1) and it did not run to completion.
# ================================================================================
not ok 2 - SomeOtherTest/testMethod(param=value1,param2=value2)
# ================================================================================
# Assertion failed in SomeOtherTest/testMethod(param=value1,param2=value2) and it did not run to completion.
# ================================================================================
ok 3 - SomeOtherTest/testMethod(param=value2,param2=value1)
not ok 4 - SomeOtherTest/testMethod(param=value2,param2=value2)
# ================================================================================
# Assertion failed in SomeOtherTest/testMethod(param=value2,param2=value2) and it did not run to completion.
# ================================================================================
not ok 5 - SomeTest/testMethod(param=value1,param2=value1)
# ================================================================================
# Assertion failed in SomeTest/testMethod(param=value1,param2=value1) and it did not run to completion.
# ================================================================================
not ok 6 - SomeTest/testMethod(param=value1,param2=value2)
# ================================================================================
# Assertion failed in SomeTest/testMethod(param=value1,param2=value2) and it did not run to completion.
# ================================================================================
ok 7 - SomeTest/testMethod(param=value2,param2=value1)
not ok 8 - SomeTest/testMethod(param=value2,param2=value2)
# ================================================================================
# Assertion failed in SomeTest/testMethod(param=value2,param2=value2) and it did not run to completion.
# ================================================================================

但是,如果您将tap插件(或版本13 tap插件或报表插件等)配置为在第三级登录,则会看到这些诊断,并且它们也位于预期位置:

runner = matlab.unittest.TestRunner.withNoPlugins();
runner.addPlugin(matlab.unittest.plugins.TAPPlugin.producingOriginalFormat('Verbosity', 3));
results = runner.run(testSuite);

你看到了输出。也可以尝试TAPVersion 13,它提供的结构化输出可能会提供更好的结果。

1..8
not ok 1 - SomeOtherTest/testMethod(param=value1,param2=value1)
# ================================================================================
# [Detailed] Diagnostic logged (2018-08-09 16:47:18): I'm here, with the params: 1.000000/1.000000
# ================================================================================
# ================================================================================
# Assertion failed in SomeOtherTest/testMethod(param=value1,param2=value1) and it did not run to completion.
# ================================================================================
not ok 2 - SomeOtherTest/testMethod(param=value1,param2=value2)
# ================================================================================
# [Detailed] Diagnostic logged (2018-08-09 16:47:19): I'm here, with the params: 1.000000/2.000000
# ================================================================================
# ================================================================================
# Assertion failed in SomeOtherTest/testMethod(param=value1,param2=value2) and it did not run to completion.
# ================================================================================
ok 3 - SomeOtherTest/testMethod(param=value2,param2=value1)
# ================================================================================
# [Detailed] Diagnostic logged (2018-08-09 16:47:19): I'm here, with the params: 2.000000/1.000000
# ================================================================================
not ok 4 - SomeOtherTest/testMethod(param=value2,param2=value2)
# ================================================================================
# [Detailed] Diagnostic logged (2018-08-09 16:47:19): I'm here, with the params: 2.000000/2.000000
# ================================================================================
# ================================================================================
# Assertion failed in SomeOtherTest/testMethod(param=value2,param2=value2) and it did not run to completion.
# ================================================================================
not ok 5 - SomeTest/testMethod(param=value1,param2=value1)
# ================================================================================
# [Detailed] Diagnostic logged (2018-08-09 16:47:19): I'm here, with the params: 1.000000/1.000000
# ================================================================================
# ================================================================================
# Assertion failed in SomeTest/testMethod(param=value1,param2=value1) and it did not run to completion.
# ================================================================================
not ok 6 - SomeTest/testMethod(param=value1,param2=value2)
# ================================================================================
# [Detailed] Diagnostic logged (2018-08-09 16:47:19): I'm here, with the params: 1.000000/2.000000
# ================================================================================
# ================================================================================
# Assertion failed in SomeTest/testMethod(param=value1,param2=value2) and it did not run to completion.
# ================================================================================
ok 7 - SomeTest/testMethod(param=value2,param2=value1)
# ================================================================================
# [Detailed] Diagnostic logged (2018-08-09 16:47:20): I'm here, with the params: 2.000000/1.000000
# ================================================================================
not ok 8 - SomeTest/testMethod(param=value2,param2=value2)
# ================================================================================
# [Detailed] Diagnostic logged (2018-08-09 16:47:20): I'm here, with the params: 2.000000/2.000000
# ================================================================================
# ================================================================================
# Assertion failed in SomeTest/testMethod(param=value2,param2=value2) and it did not run to completion.
# ================================================================================

希望能有帮助!