run_tests

Easily run tests of your iOS app (via scan)

The easiest way to run tests of your iOS and Mac app

scan makes it easy to run tests of your iOS and Mac app on a simulator or connected device.


FeaturesUsageScanfile


What's scan?

https://pbs.twimg.com/media/CURcEpuWoAArE3d.png:large

Before scan

xcodebuild \
  -workspace MyApp.xcworkspace \
  -scheme "MyApp" \
  -sdk iphonesimulator \
  -destination 'platform=iOS Simulator,name=iPhone 6,OS=8.1' \
  test

As the output will look like this

/Users/felixkrause/Library/Developer/Xcode/DerivedData/Example-fhlmxikmujknefgidqwqvtbatohi/Build/Intermediates/ArchiveIntermediates/Example/IntermediateBuildFilesPath/Example.build/Release-iphoneos/Example.build/Objects-normal/arm64/main.o Example/main.m normal arm64 objective-c com.apple.compilers.llvm.clang.1_0.compiler
    cd /Users/felixkrause/Developer/fastlane/gym/example/cocoapods
    export LANG=en_US.US-ASCII
    export PATH="/Applications/Xcode-beta.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/usr/bin:/Applications/Xcode-beta.app/Contents/Developer/usr/bin:/Users/felixkrause/.rvm/gems/ruby-2.2.0/bin:/Users/felixkrause/.rvm/gems/ruby-2.2.0@global/bin:/Users/felixkrause/.rvm/rubies/ruby-2.2.0/bin:/Users/felixkrause/.rvm/bin:/usr/local/heroku/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin"
    /Applications/Xcode-beta.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/clang -x objective-c -arch arm64 -fmessage-length=126 -fdiagnostics-show-note-include-stack -fmacro-backtrace-limit=0 -fcolor-diagnostics -std=gnu99 -fobjc-arc -fmodules -gmodules -fmodules-cache-path=/Users/felixkrause/Library/Developer/Xcode/DerivedData/ModuleCache -fmodules-prune-interval=86400 -fmodules-prune-after=345600 -fbuild-session-file=/Users/felixkrause/Library/Developer/Xcode/DerivedData/ModuleCache/Session.modulevalidation -fmodules-validate-once-per-build-session -Wnon-modular-include-in-framework-module -Werror=non-modular-include-in-framework-module -Wno-trigraphs -fpascal-strings -Os -fno-common -Wno-missing-field-initializers -Wno-missing-prototypes -Werror=return-type -Wunreachable-code -Wno-implicit-atomic-properties -Werror=deprecated-objc-isa-usage -Werror=objc-root-class -Wno-arc-repeated-use-of-weak -Wduplicate-method-match -Wno-missing-braces -Wparentheses -Wswitch -Wunused-function -Wno-unused-label -Wno-unused-parameter -Wunused-variable -Wunused-value -Wempty-body -Wconditional-uninitialized -Wno-unknown-pragmas -Wno-shadow -Wno-four-char-constants -Wno-conversion -Wconstant-conversion -Wint-conversion -Wbool-conversion -Wenum-conversion -Wshorten-64-to-32 -Wpointer-sign -Wno-newline-eof -Wno-selector -Wno-strict-selector-match -Wundeclared-selector -Wno-deprecated-implementations -DCOCOAPODS=1 -DNS_BLOCK_ASSERTIONS=1 -DOBJC_OLD_DISPATCH_PROTOTYPES=0 -isysroot /Applications/Xcode-beta.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS9.0.sdk -fstrict-aliasing -Wprotocol -Wdeprecated-declarations -miphoneos-version-min=9.0 -g -fvisibility=hidden -Wno-sign-conversion -fembed-bitcode -iquote /Users/felixkrause/Library/Developer/Xcode/DerivedData/Example-fhlmxikmujknefgidqwqvtbatohi/Build/Intermediates/ArchiveIntermediates/Example/IntermediateBuildFilesPath/Example.build/Release-iphoneos/Example.build/ExampleProductName-generated-files.hmap -I/Users/felixkrause/Library/Developer/Xcode/DerivedData/Example-fhlmxikmujknefgidqwqvtbatohi/Build/Intermediates/ArchiveIntermediates/Example/IntermediateBuildFilesPath/Example.build/Release-iphoneos/Example.build/ExampleProductName-own-target-headers.hmap -I/Users/felixkrause/Library/Developer/Xcode/DerivedData/Example-fhlmxikmujknefgidqwqvtbatohi/Build/Intermediates/ArchiveIntermediates/Example/IntermediateBuildFilesPath/Example.build/Release-iphoneos/Example.build/ExampleProductName-all-target-headers.hmap -iquote /Users/felixkrause/Library/Developer/Xcode/DerivedData/Example-fhlmxikmujknefgidqwqvtbatohi/Build/Intermediates/ArchiveIntermediates/Example/IntermediateBuildFilesPath/Example.build/Release-iphoneos/Example.build/ExampleProductName-project-headers.hmap -I/Users/felixkrause/Library/Developer/Xcode/DerivedData/Example-fhlmxikmujknefgidqwqvtbatohi/Build/Intermediates/ArchiveIntermediates/Example/BuildProductsPath/Release-iphoneos/include -I/Users/felixkrause/Developer/fastlane/gym/example/cocoapods/Pods/Headers/Public -I/Users/felixkrause/Developer/fastlane/gym/example/cocoapods/Pods/Headers/Public/HexColors -I/Users/felixkrause/Library/Developer/Xcode/DerivedData/Example-fhlmxikmujknefgidqwqvtbatohi/Build/Intermediates/ArchiveIntermediates/Example/IntermediateBuildFilesPath/Example.build/Release-iphoneos/Example.build/DerivedSources/arm64 -I/Users/felixkrause/Library/Developer/Xcode/DerivedData/Example-fhlmxikmujknefgidqwqvtbatohi/Build/Intermediates/ArchiveIntermediates/Example/IntermediateBuildFilesPath/Example.build/Release-iphoneos/Example.build/DerivedSources -F/Users/felixkrause/Library/Developer/Xcode/DerivedData/Example-fhlmxikmujknefgidqwqvtbatohi/Build/Intermediates/ArchiveIntermediates/Example/BuildProductsPath/Release-iphoneos -isystem /Users/felixkrause/Developer/fastlane/gym/example/cocoapods/Pods/Headers/Public -isystem /Users/felixkrause/Developer/fastlane/gym/example/cocoapods/Pods/Headers/Public/HexColors -MMD -MT dependencies -MF /Users/felixkrause/Library/Developer/Xcode/DerivedData/Example-fhlmxikmujknefgidqwqvtbatohi/Build/Intermediates/ArchiveIntermediates/Example/IntermediateBuildFilesPath/Example.build/Release-iphoneos/Example.build/Objects-normal/arm64/main.d --serialize-diagnostics /Users/felixkrause/Library/Developer/Xcode/DerivedData/Example-fhlmxikmujknefgidqwqvtbatohi/Build/Intermediates/ArchiveIntermediates/Example/IntermediateBuildFilesPath/Example.build/Release-iphoneos/Example.build/Objects-normal/arm64/main.dia -c /Users/felixkrause/Developer/fastlane/gym/example/cocoapods/Example/main.m -o /Users/felixkrause/Library/Developer/Xcode/DerivedData/Example-fhlmxikmujknefgidqwqvtbatohi/Build/Intermediates/ArchiveIntermediates/Example/IntermediateBuildFilesPath/Example.build/Release-iphoneos/Example.build/Objects-normal/arm64/main.o

you'll probably want to use something like xcpretty, which will look like this:

set -o pipefail &&
  xcodebuild \
    -workspace MyApp.xcworkspace \
    -scheme "MyApp" \
    -sdk iphonesimulator \
    -destination 'platform=iOS Simulator,name=iPhone 6,OS=8.1' \
    test \
  | xcpretty \
    -r "html" \
    -o "tests.html"

With scan

fastlane scan

Why scan?

scan uses the latest APIs and tools to make running tests plain simple and offer a great integration into your existing workflow, like fastlane or Jenkins.

scan Features
🏁 Beautiful inline build output while running the tests
🚠 Sensible defaults: Automatically detect the project, schemes and more
📊 Support for HTML, JSON and JUnit reports
🔎 Xcode duplicated your simulators again? scan will handle this for you
🔗 Works perfectly with fastlane and other tools
🚅 Don't remember any complicated build commands, just scan
🔧 Easy and dynamic configuration using parameters and environment variables
📢 Beautiful slack notifications of the test results
💾 Store common build settings in a Scanfile
📤 The raw xcodebuild outputs are stored in ~/Library/Logs/scan
💻 Supports both iOS and Mac applications
👱 Automatically switches to the travis formatter when running on Travis
📖 Helps you resolve common test errors like simulator not responding

scan uses a plain xcodebuild command, therefore keeping 100% compatible with xcodebuild. To generate the nice output, scan uses xcpretty. You can alway access the raw output in ~/Library/Logs/scan.

img/actions/scanScreenshot.png img/actions/slack.png img/actions/scanHTML.png img/actions/scanHTMLFailing.png

Usage

fastlane scan

That's all you need to run your tests. If you want more control, here are some available parameters:

fastlane scan --workspace "Example.xcworkspace" --scheme "AppName" --device "iPhone 6" --clean

If you need to use a different xcode install, use xcode-select or define DEVELOPER_DIR:

DEVELOPER_DIR="/Applications/Xcode6.2.app" scan

To run scan on multiple devices via fastlane, add this to your Fastfile:

scan(
  workspace: "Example.xcworkspace",
  devices: ["iPhone 6s", "iPad Air"]
)

For a list of all available parameters use

fastlane action scan

To access the raw xcodebuild output open ~/Library/Logs/scan

Scanfile

Since you might want to manually trigger the tests but don't want to specify all the parameters every time, you can store your defaults in a so called Scanfile.

Run fastlane scan init to create a new configuration file. Example:

scheme("Example")
devices(["iPhone 6s", "iPad Air"])

clean(true)

output_types("html")

Automating the whole process

scan works great together with fastlane, which connects all deployment tools into one streamlined workflow.

Using fastlane you can define a configuration like

lane :test do
  scan(scheme: "Example")
end

run_tests
Supported platforms ios, mac
Author @KrauseFx

6 Examples

run_tests
scan # alias for "run_tests"
run_tests(
  workspace: "App.xcworkspace",
  scheme: "MyTests",
  clean: false
)
# Build For Testing
run_tests(
   derived_data_path: "my_folder",
   build_for_testing: true
)
# run tests using derived data from prev. build
run_tests(
   derived_data_path: "my_folder",
   test_without_building: true
)
# or run it from an existing xctestrun package
run_tests(
   xctestrun: "/path/to/mytests.xctestrun"
)

Parameters

Key Description Default
workspace Path to the workspace file
project Path to the project file
device The name of the simulator type you want to run tests on (e.g. 'iPhone 6')
toolchain The toolchain that should be used for building the application (e.g. com.apple.dt.toolchain.Swift_2_3, org.swift.30p620160816a)
devices Array of devices to run the tests on (e.g. ['iPhone 6', 'iPad Air'])
scheme The project's scheme. Make sure it's marked as Shared
clean Should the project be cleaned before building it? false
code_coverage Should code coverage be generated? (Xcode 7 and up)
address_sanitizer Should the address sanitizer be turned on?
thread_sanitizer Should the thread sanitizer be turned on?
skip_build Should debug build be skipped before test build? false
output_directory The directory in which all reports will be stored *
output_style Define how the output should look like (standard, basic, rspec or raw)
output_types Comma separated list of the output types (e.g. html, junit, json-compilation-database) html,junit
output_files Comma separated list of the output files, corresponding to the types provided by :output_types (order should match). If specifying an output type of json-compilation-database with :use_clang_report_name enabled, that option will take precedence
buildlog_path The directory where to store the raw log *
include_simulator_logs If the logs generated by the app (e.g. using NSLog, perror, etc.) in the Simulator should be written to the output_directory false
formatter A custom xcpretty formatter to use
max_concurrent_simulators Constrain the number of simulator devices on which to test concurrently. Equivalent to -maximum-concurrent-test-simulator-destinations
disable_concurrent_testing Do not run test bundles in parallel on the specified destinations. Testing will occur on each destination serially. Equivalent to -disable-concurrent-testing false
test_without_building Test without building, requires a derived data path
build_for_testing Build for testing only, does not run tests
xctestrun Run tests using the provided .xctestrun file
derived_data_path The directory where build products and other derived data will go
should_zip_build_products Should zip the derived data build products and place in output path? false
result_bundle Should an Xcode result bundle be generated in the output directory false
sdk The SDK that should be used for building the application
open_report Should the HTML report be opened when tests are completed? false
configuration The configuration to use when building the app. Defaults to 'Release' *
destination Use only if you're a pro, use the other options instead
xcargs Pass additional arguments to xcodebuild. Be sure to quote the setting names and values e.g. OTHER_LDFLAGS="-ObjC -lstdc++"
xcconfig Use an extra XCCONFIG file to build your app
only_testing Array of strings matching Test Bundle/Test Suite/Test Cases to run
skip_testing Array of strings matching Test Bundle/Test Suite/Test Cases to skip
slack_url Create an Incoming WebHook for your Slack group to post results there
slack_channel #channel or @username
slack_message The message included with each message posted to slack
skip_slack Don't publish to slack, even when an URL is given false
slack_only_on_failure Only post on Slack if the tests fail false
use_clang_report_name Generate the json compilation database with clang naming convention (compile_commands.json) false
custom_report_file_name DEPRECATED! Use --output_files instead - Sets custom full report file name when generating a single report
fail_build Should this step stop the build if the tests fail? Set this to false if you're using trainer true

* = default value is dependent on the user's system


To show the documentation in your terminal, run

fastlane action run_tests

View source code


Back to actions