Skip to content

Difference of values in duration in HTML report and xctest log #329

Description

@jbatra-wondermentapps

The HTML report and test-case logs have different values for duration

Is this a bug or some misunderstanding for interpretation?

Please route me to the right approach to read the logs.

Testing started
2023-05-26 12:24:58.207271+0530 PSSwiftExpressUITests-Runner[1191:259871] Running tests...
Test Suite 'Selected tests' started at 2023-05-26 12:24:58.269
Test Suite 'PSSwiftExpressUITests.xctest' started at 2023-05-26 12:24:58.270
Test Suite 'PerformanceTests' started at 2023-05-26 12:24:58.270
Test Case '-[PSSwiftExpressUITests.PerformanceTests testColdAppLaunchTime]' started.
    t =     0.00s Start Test at 2023-05-26 12:24:58.270
    t =     0.00s Set Up
executing: -[PerformanceTests testColdAppLaunchTime]
Battery before start:  0.96
RAM before start:  607993856
Free Disk space before start:  45.87 GB
    t =     0.01s Open com.adobe.PSMobile
    t =     0.01s     Launch com.adobe.PSMobile
    t =     0.01s         Terminate com.adobe.PSMobile:1168
    t =    21.39s         Setting up automation session
    t =    22.25s         Wait for com.adobe.PSMobile to idle
    t =    35.23s Open com.adobe.PSMobile
    t =    35.23s     Launch com.adobe.PSMobile
    t =    35.23s         Terminate com.adobe.PSMobile:1194
    t =    37.59s         Setting up automation session
    t =    37.70s         Wait for com.adobe.PSMobile to idle
    t =    49.34s Open com.adobe.PSMobile
    t =    49.34s     Launch com.adobe.PSMobile
    t =    49.34s         Terminate com.adobe.PSMobile:1198
    t =    51.61s         Setting up automation session
    t =    51.71s         Wait for com.adobe.PSMobile to idle
    t =    63.30s Open com.adobe.PSMobile
    t =    63.30s     Launch com.adobe.PSMobile
    t =    63.30s         Terminate com.adobe.PSMobile:1200
    t =    65.75s         Setting up automation session
    t =    65.85s         Wait for com.adobe.PSMobile to idle
    t =    77.44s Open com.adobe.PSMobile
    t =    77.44s     Launch com.adobe.PSMobile
    t =    77.44s         Terminate com.adobe.PSMobile:1203
    t =    79.95s         Setting up automation session
    t =    80.05s         Wait for com.adobe.PSMobile to idle
    t =    91.65s Open com.adobe.PSMobile
    t =    91.65s     Launch com.adobe.PSMobile
    t =    91.65s         Terminate com.adobe.PSMobile:1205
    t =    94.35s         Setting up automation session
    t =    94.37s         Wait for com.adobe.PSMobile to idle
/Users/usr/Documents/Adobe/Example/example-ios/PSSwiftExpressUITests/tests/PerformanceTests.swift:315: Test Case '-[PSSwiftExpressUITests.PerformanceTests testColdAppLaunchTime]' measured [Duration (AppLaunch), s] average: 0.436, relative standard deviation: 3.916%, values: [0.468968, 0.420271, 0.426677, 0.432267, 0.431298], performanceMetricID:com.apple.dt.XCTMetric_ApplicationLaunch-AppLaunch.duration, baselineName: "Local Baseline", baselineAverage: 2.786, polarity: prefers smaller, maxPercentRegression: 10.000%, maxPercentRelativeStandardDeviation: 10.000%, maxRegression: 0.000, maxStandardDeviation: 0.000
    t =   105.97s Tear Down
    t =   105.97s     -[PerformanceTests testColdAppLaunchTime]
    t =   107.15s     Terminate com.adobe.PSMobile:1206
Battery after end:  0.96
RAM after end:  590168064
Free Disk Space after end:  45.87 GB
Test Case '-[PSSwiftExpressUITests.PerformanceTests testColdAppLaunchTime]' passed (108.186 seconds).
Test Suite 'PerformanceTests' passed at 2023-05-26 12:26:46.458.
	 Executed 1 test, with 0 failures (0 unexpected) in 108.186 (108.188) seconds
Test Suite 'PSSwiftExpressUITests.xctest' passed at 2023-05-26 12:26:46.460.
	 Executed 1 test, with 0 failures (0 unexpected) in 108.186 (108.190) seconds
Test Suite 'Selected tests' passed at 2023-05-26 12:26:46.461.
	 Executed 1 test, with 0 failures (0 unexpected) in 108.186 (108.192) seconds
2023-05-26 12:26:46.379 xcodebuild[95428:2845548] [MT] IDETestOperationsObserverDebug: 113.146 elapsed -- Testing started completed.
2023-05-26 12:26:46.379 xcodebuild[95428:2845548] [MT] IDETestOperationsObserverDebug: 0.000 sec, +0.000 sec -- start
2023-05-26 12:26:46.379 xcodebuild[95428:2845548] [MT] IDETestOperationsObserverDebug: 113.146 sec, +113.146 sec -- end

Test session results, code coverage, and logs:
	/Users/usr/Documents/Adobe/Example/Example-ios/TestResults

** TEST SUCCEEDED **

usr@usr-MacBook-Pro Example-ios % xchtmlreport /Users/usr/Documents/Adobe/Example/Example-ios/TestResults.xcresult                                                     

Report successfully created at /Users/usr/Documents/Adobe/Example/Example-ios/index.html
usr@usr-MacBook-Pro Example-ios % xchtmlreport /Users/usr/Documents/Adobe/Example/Example-ios/TestResults.xcresult

HTML report

index 2.html.zip

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions