Introduction to TAF

TAF helps you do robust and maintainable test automation for web based systems, REST services and WPF clients. It has been developed with several decades of automation experience in hundreds of assignments. It's meant to work well for both developers, and technical testers and is prepared for and has proven itself well for use in CI/CD environments.

TAF provides logging, reporting and an exchangeable driver structure ensuring you that you'll have a test automation that will have a life span longer than for example Selenium or Windows.Automation.

Main concepts around TAF are:

• Never disable use of underlaying components. If a situation requires special intricate and creative coding, all shall be there for you to work with.
• Grab a lot of information - only display what's relevant at each given time.
• Ease of maintenance and being prepared for change. It's our understanding that most test automation doesn't fail due to technical problems, but to maintenance ones. Our goal is to make it as easy as possible to maintain the code base for test automation.
• Push the complexity downwards in the code. Test automation is a complex task with complex dependencies to development project as well as to the development process. It should be easy to do test automation.
• Reuse of effort. TAF is open source. When some overcomes difficult automation challenges others too should benefit from it. Hence TAF is constantly evolving for each new challenge.

Index

Pre-requisites

TAF currently has a dependency to the Microsoft MSTest framework. To override this dependency, see the chapter about how to do this.

Current support with TAF

Module dependencies diagram of TAF C#

Using TAF

To use TAF you first need to get the TAF code and its dependencies accessible from your own code. The easiest way is to use your nuget manager to include relevant TAF packages to your own solution. The different modules reference eachother so it makes sense only to reference the top level one, enabling a specific technology, for example through nuget packet manager console:

        
         PM> Install-Package TAF.WebSupport -Version 2.2.1
        

This will enable TAF.WebSupport by installing the nuget package and the secondary (transitive) dependencies needed.

Enabling TAF

TAF is created to be used with MS Test framework. The binding is loose, so you could substitute MS Test for NUnit or whatever, but then you'll have to adhere to the instructions section below. The easiest way to get started with TAF is to use MS Test.
You need to make sure to create your test project as a unit test project in Visual Studio, or include the MSTest.Framework and MSTest.TestAdapter nuget packages to use TAF.

For the most part TAF can be used with either NetCore or NetFramework. Some functionality, like desktop screenshots, are lacking from NetCore due to limitations built into NetCore.

When MSTest is available, make sure your test class implements TAF.Core.TestSet, like this:

        
        using Microsoft.VisualStudio.TestTools.UnitTesting;
        using TAF.Core;
        
        [TestClass]
        public class MyTests : TestSet //It's essential for TAF that the test class extends the TAF.Core.TestSet class
        {
            [TestMethod]
            public void MyFirstTafTest(){
            }
        }
        

This will enable a range of new functionality. The most common ones are:

•  AddReport()  - adding different types of reports
•  AddKnownError()  - registering known errors for the test set or test case
•  AddExecutionListener()  - registering a new execution listener for TAF (advanced)
•  CurrentTestCase  - enabling access to test case result, among other things.

Common TAF operations

Changing output folder

TAF comes with a range of setting for the test run. One of these is the test output folder. If this is not set explicitly output will be directed to a folder called TAF in the executing user home folder.

            TestRun.Instance.RunSettings.SetRunSetting(RunParameter.TestRunLogFolder, "C:\\Temp\\MyLogFolder");
        

Testing REST applications with TAF

TAF support testing towards REST API:s. This is enabled by adding a dependecy to TAF.RestSupport nuget package. This could be done with the nuget package management console:

        
         PM> Install-Package TAF.RestSupport -Version 3.2.6
        

When the package is installed the full functionality of TAF REST is enabled. A typical setup would look a bit like:

        
        using TAF.RestSupport;
        using TAF.Core;
        using System.Net;   
        using Microsoft.VisualStudio.TestTools.UnitTesting;
        
        namespace MySiteTests
        {
         [TestClass]
         [TestCategory("SystemTests.REST")]
         public class MyRestTests : TestSet
         {
          private RestInteraction _rest;
        
          [TestInitialize]
          public void Setup()
          {
           _rest = new RestInteraction(CurrentTestCase.TestCaseResult.LogPosts);
           AddReport(new TestCaseHtmlReport());
          }
        
          [TestMethod]
          public void FirstRestTest()
          {
           _rest.MakeDriverUnsafe();
           _rest.HttpGet("http://www.testsite.com")
            .Verify()
             .StatusCode(HttpStatusCode.OK)
             .BodyContains("Mydata");
          }
         }
        }
        

A REST response could contain anything but typically it is data that is serialized to JSON or REST. Hence operations for these two data formats are included as default. This could include getting specific data values, de-serialization to objects, verifications and so on.

Testing web applications with TAF

TAF has its own implementation for testing of web. It uses Selenium WebDriver to drive the tests, but has implemented a framework upon it to get more robustness, better logging and better testing. The dependency to Selenium WebDriver is a soft one and could be substituted at will, without changing your tests. Selenium is chosen due to its ability to support many types of browsers, and for its wide support among third party services.

TAF.WebSupport is enabled through a nuget package, installed from the nuget package manager console:

        
         PM> Install-Package TAF.WebSupport -Version 1.8.6
            

To enable web testing with TAF you need to reference TAF.WebSupport and instantiate an instance of WebInteractions.

        
        using TAF.Core;
        using Microsoft.VisualStudio.TestTools.UnitTesting;
        using TAF.WebSupport.SeleniumAdapter;
        using OpenQA.Selenium.Chrome;
        using TAF.WebSupport.Core;
        
        namespace MySiteTests
        {
            [TestClass]
            [TestCategory("SystemTests.GUI")]
            public class MyGuiTests : TestSet
            {
                private WebInteraction _web;
        
                [TestInitialize]
                public void Setup()
                {
                    _web = new WebInteractionSelenium(CurrentTestCase, new ChromeDriver());
                }
        
                [TestCleanup]
                public void TearDown()
                {
                    _web.MakeSureDriverIsClosed();
                }
        
                [TestMethod]
                public void LandingPageTest()
                {
                    _web.Navigate("http://mysite.mycompany.com");
                    _web.Verify().CurrentBrowserTabTitle("My site");
                }
            }
        }
            

If no Selenium driver is given to the constructor attempts will be made using common Selenium drivers.

For other drivers than Selenium, make sure they extend TafWebDriver and seed them into the WebInteraction constructor, like this:

        
            var web = new WebInteraction(CurrentTestCase, new TafWebDriverJasmine(CurrentTestCase));
        

The DomElement class

TAF.WebSupport uses a sub-class of the generic GuiObject to interact with web page content. Web pages are made up of a DOM (Document Object Model). The sub class of GUI objects for web hence is called DomElement. The DomElement is kind of a map to how you intend to identify the element at runtime.

Using DomElement to describe how to identify a runtime DOM element of a web page is done by chained By statements:

        
            public static DomElement SubmitButton = new DomElement(By
                .TagName("button")
                .AndByClass("formelement")
                .AndByExactText("Submit"));
        

In most cases the DomElement will get a name from the variable used, or you could provide one yourself in the constructor.
The element name is used for logging purposes.

You could manually construct your DomElements for the page, or you could have TAF construct the page class for you, mapping relevant elements for you. Mapping a page is done with the command

            _web.Navigate("http://www.thesite.com");
            _web.MapCurrentPage("C:\\temp\\landingpage.txt");
        

Test case flow control

Some commands are considered flow control commands. These are commands that drives the test forward. Examples are:

• Navigate()
• Click()
• Select()
• Exist()
• GetText()
• Write()

If a flow control command fails, for example from an element that cannot be identified, further script execution is halted - as opposed to verification commands where script execution generally continues.

Verifications

TAF is a testing framework. Verifications is essential, and TAF provides a wide range of pre-defined verifications. The TAF verifications are like asserts, but they continue execution and are typically chainable in a builder pattern.

Apart from the generic verifications linked directly to the test case (se below), different TAF modules enables verifications of their own.

Sample generic verifications

        
         CurrentTestCase.Verify().Text(myVariable.Text, "^MyText.*[1,2]", VerificationHelper.StringComparisonMethod.Regex);
         CurrentTestCase.Verify().Text(myVariable.Text, "MyText", VerificationHelper.StringComparisonMethod.ContainsIgnoreCase);
         CurrentTestCase.Verify().IsEqual(image1, image2);
        

Verifications in the TAF.WebSupport module

For the WebSupport module for TAF there are three different types of verifications. Browser Verifications check browser status, GUI Layout Verifications check appearance of a web page, and DomElement verifications check status of individual HTML elements.

Sample WebSupport browser verifications

         _web.Verify().BrowserConsoleIsEmpty();
         _web.Verify().PageSource("MyCompanyTitle", VerificationHelper.StringComparisonMethod.Contains);
         _web.Verify().CurrentUrl("MyExpectedUrl", VerificationHelper.StringComparisonMethod.ExactIgnoreCase);
         CurrentTestCase.Verify().Text(myVariable.Text, "^MyText.*[1,2]", VerificationHelper.StringComparisonMethod.RegEx);
         CurrentTestCase.Verify().Text(myVariable.Text, "MyText", VerificationHelper.StringComparisonMethod.ContainsIgnoreCase);
         CurrentTestCase.Verify().IsEqual(image1, image2);
        

Sample WebSupport GUI Layout verifications

         //Supplying element in Verify directly
         _web.Verify(SubmitButton).GuiLayout().VerifyIsBiggerThan(50, 50);
         //Supplying element in GuiLayout
         _web.Verify().GuiLayout(SubmitButton).VerifyIsBiggerThan(50, 50);
        

Sample DomElement verifications

         _web.Verify(SubmitButton)
          .IsFullyWithinView()
          .Color(ColorGroup.Blue)
          .IsEnabled();
        

Testing WPF and Windows Forms applications with TAF

WPF (Windows Presentation Foundation) is a .NET/C# GUI technology that replaces Windows.Forms.

Testing WPF applications, or Windows.Forms applications, is easiest done from a Windows client machine.

TAF support testing of both these GUI technologies from the TAF.WpfSupport nuget package. However, testing Windows.Forms might require some special hacks since the Windows.Forms runtime libraries are CPU architecture dependant.

Most of WPF testing works more or less exactly like testing web, but one key difference is that the WPF libraries don't just work with a specific browser, but any process on the execution computer. This means we always have to make sure we interact with a GUI element in the correct window. This makes Window, and WindowElement key classes to get to know.

Windows are identified by the title as regular expression.

Enabling WPF testing

To enable TAF support for WPF just reference the TAF.WpfSupport nuget package in its latest version.

         PM> Install-Package TAF.WpfSupport 
            

Complete code example for Windows Calculator:

        using System.Diagnostics;
        using Microsoft.VisualStudio.TestTools.UnitTesting;
        using TAF.Core;
        using TAF.WpfSupport;
        
        namespace WPFDemo
        {
            [TestClass]
            public class TestWpf : TestSet
            {
                private WindowsGuiInteractionMethods _win;
        
                [TestInitialize]
                public void Setup()
                {
                    _win = new WindowsGuiInteractionMethods(CurrentTestCase);
                    AddReport(new TestCaseHtmlReport());
                }
        
                [TestCleanup]
                public void TearDown()
                {
                    foreach (var process in Process.GetProcessesByName("Calculator"))
                    {
                        process.Kill();
                    }
                }
        
                [TestMethod]
                public void TestCalc01()
                {
                    Process.Start("Calc.exe");
                    var app = new Window("Calculator.*");
        
                    var calc = new WindowsElement(app, By.Name("Calculator"));
                    var buttonSix = new WindowsElement(app, By.AutomationId("num6Button"));
        
                    _win.BringElementWindowToFront(calc);
                    _win.Click(buttonSix);
                    _win.VerifyText(new WindowsElement(app, By.AutomationId("CalculatorResults")),
                        "Display is 6");
                }
            }
        }
        
            

There are a number of specific WPF testing utilities included in TAF. There are methods to print the titles of all windows. There are methods to map all elements of a specific Window to enable speedy test automation implementations.

Although the WPF support enables to interact with most client based Windows programs, one limitation of the WPF support package is that it currently cannot break into the Java JVM. If you are expecting to test a GUI based Java program you are better off with the Java version of TAF.

Performance testing capabilities of TAF.Core

To ease the left-shifting of performance tests in a CI/CD context some methods has been implemented in TAF.Core to ease the developer performance testing.

Normally not all types of performance tests are relevant to perform in a CI environment. For example, testing of infrastructure configuration and behavior with extreme workloads and full system data volumes generally require manual attendance during the workload to prevent damage to the system and the infrastructure.

However, testing concurrency and escalating load on individual components could be expected to be tested from a unit testing framework. To lower the threshold to implement that type of tests it's been implemented in TAF.

Currently the performance testing support is implemented with MS Test as test runner.

Parallel tests

To run a test in a number of parallel threads makes sure the tested functionality doesn't produce any type of deadlock situations.

To run a test in parallel, just substitute [TestMethod] with [ConcurrencyTest(6)], where the parameter of the ConcurrencyTest attribute is the number of parallel threads.

Examples

Take any regular MS Test, like the one below

        
        [TestMethod]
        public void Test1(){
            var web = new WebInteraction(CurrentTestCase);
            web.Navigate("http://www.zingtongroup.com")
                 .Verify().CurrentBrowserTabTitle("TAF Home Page");
        }
            

This test could easily be run in parallel by substituting the [TestMethod] with [ConcurrencyTest(4)].

        
        [ConcurrencyTest]
        public void Test1(){
            var web = new WebInteraction(CurrentTestCase);
            web.Navigate("http://www.zingtongroup.com")
                 .Verify().CurrentBrowserTabTitle("TAF Home Page");
        }
            

For any type of test to run in parallel it cannot use class level variables/fields/properties (unless they are syncronized static ones, and that might affect test outcome), due to normal OOP behavior.

Naturally any test of this kind would be better off by measuring response times. This could be done with normal TAF verification:

        
        [ConcurrencyTest(numberOfParallelThreads: 4)]
        public void Test1(){
            var web = new WebInteraction(CurrentTestCase);
            web.Navigate("http://www.zingtongroup.com")
               .Verify().VerifyResponseTime(2000);
        }
            

But if you want to measure response times for the whole test iteration this is another parameter in the ConcurrencyTest

        
        [ConcurrencyTest(numberOfParallelThreads: 4, maximumAcceptedResponseTimeInMilliseconds: 2000)]
        public void Test1(){
            var web = new WebInteraction(CurrentTestCase);
            web.Navigate("http://www.zingtongroup.com")
               .Verify().CurrentBrowserTabTitle("TAF Home Page");
        }
            

ConcurrencyTest is a convenient way of running the same test in parallel. It doesn't have to be a TAF test at all. Any MS Test will do. Like this:

        [ConcurrencyTest(numberOfParallelThreads: 4, maximumAcceptedResponseTimeInMilliseconds: 200)]
        public async Task Test1()
        {
            var client = new HttpClient();
            var response = await client.GetAsync("http://www.zingtongroup.com");
            Assert.IsTrue(response.StatusCode.Equals(HttpStatusCode.OK));
        }
            

This will run the assertion for each thread in its own right and then combine the results to a master test result.

Comparison of execution time of test run in parallel and single thread tests.

It's sometimes more relevant to assess if a test executed in parallel takes significantly longer than the same test run in a single thread. To make this assessment easy TAF.Core has implemented a relatively easy way of doing this.

        [ParallelSequentialDurationComparisonTest]
        public async Task Test1(){
            var client = new HttpClient();
            var response = await client.GetAsync("http://www.zingtongroup.com");
            Assert.IsTrue(response.StatusCode.Equals(HttpStatusCode.OK));
        }
            

This will execute the test in the following sequence:

1. Warm-up run without measurements with a single thread to populate any caches.
2. Single thread execution of the test, with recorded execution duration.
3. Test execution in three parallel threads, with recorded full execution duration.

If the execution duration for three parallel threads is more than 1.5 times the execution duration for a single thread the test will fail.

Off course all the parameters of the test can be tweaked:

        
        [ParallelSequentialDurationComparisonTest(
                                numberOfParallelThreadsInParallelExecution: 5, 
                                acceptedDurationRatioForParallelExecution: 2.0, 
                                executeUnBenchmarkedWarmupRunToPopulateCaches: false)]
        public async Task Test1(){
            var client = new HttpClient();
            var response = await client.GetAsync("http://www.zingtongroup.com");
            Assert.IsTrue(response.StatusCode.Equals(HttpStatusCode.OK));
        }
        

This is a pretty simple way of assessing if there are any code related issues to system parallelism.

Load test

Sometimes you want to slowly increase the load upon a system to make sure it doesn't break, and then let the test run for a while to make sure response times doesn't increase over time. To accomodate this TAF includes an attribute called [LoadTest].

        
        [LoadTest(numberOfParallelThreads: 50, 
                loadDurationInSeconds: 300, 
                rampUpTimeInSeconds: 20,
                maxIterationTimeInMilliseconds: 3000)]
        public async Task Test1(){
            var client = new HttpClient();
            var response = await client.GetAsync("http://www.zingtongroup.com");
            Assert.IsTrue(response.StatusCode.Equals(HttpStatusCode.OK));
        }
        

If any test iteration takes longer than the maxIterationTimeInMilliseconds value the test will fail.

Result reports

Referencing TAF.Core enables reporting in TAF. Test results are stored in a folder called TAF in the user home directory.

TAF has several built-in ways of displaying the test outcome, and more can be added.

Test case execution reports

At the end of a test case execution the log is analysed for highest log post status and known errors, thus giving it an outcome status.

The built-in report types for TAF are:

• TestCaseExecutionConsoleReport
• TestCaseHtmlReport

Enabling these reports are done by making sure you have your TAF.Core using directive and invoking the AddReport() method:

        
            [TestMethod]
            public void MyTestCase()
            {
                CurrentTestCase.AddReport(new TestCaseHtmlReport());
            }
        

Of course, this could be done for all test cases by using the [TestInitialize] mechanism of MSTest, like this (probably way more common):

        
            [TestInitialize]
            public void TestSetUp()
            {
                AddReport(new TestCaseHtmlReport());
            }
        

Recording video of test execution

To record a video of test execution with TAF you must reference the TAF.Logging.TestCaseVideoLog library.

There are two main ways of using the video recording feature of TAF.

Recording single test case

If TAF.Logging.TestCaseVideoLog is referenced you can make a single test case record a video by adding the TestCaseExecutionVideo report:

        
            [TestMethod]
            public void MyTestCase()
            {
                AddReport(new TestCaseExecutionVideo());
            }
        

This will save a recording of the test execution to the test output folder and make a note of it in other reports.

Recording all test cases

To enable recording of all test cases you use the command:

        
            [AssemblyInitialize]
            public static void AssemblyInit(TestContext testContext)
            {
                TestCaseVideoLogs.Activate();
            }
        

To save disk space there is an option to only save videos for test cases with failed test cases. This is used like this:

        
            [AssemblyInitialize]
            public static void AssemblyInit(TestContext testContext)
            {
                TestCaseVideoLogs.Activate(saveRecordingsForSuccessfulTestCaseExecutions: false);
            }
        

Deactivating video recording of test cases is done by the command  TestCaseVideoLogs.Deactivate(); .

To create your own test case report, you only have to implement the single method of the ITestCaseExecutionReport interface.

Test run CSV report

Reports can also be of the scope of the whole test run. Currently only one such report is pre-implemented:

• TestRunCsvReport

The TestRunCsvReport is added as a TafExecutionListener. It was initially developed for integration with Splunk, but the usage could be far wider.

        
                TestRun.Instance.AddExecutionListener(new TestRunCsvReport());
            

This report can be run with a few different behaviors:

• including file headings row or not
• Apending to output file or deleting pre-existing files
• Stating special file path to output file

If no file name is given to the TestRunCsvReport it will produce a file called TestResults.csv in the log folder of the test run.

Reporting test results to TFS

If TAF.TfsIntegration is added to your solution the test case results could be propagated to TFS/VSTS.

If you want to implement your own test case report, just make sure it implements the TAF.Core.ITestCaseExecutionReport interface and you are good to go, as described later in this document.

Reporting test results to Splunk

For organizations using dashboards in Splunk to display testing status, trends, and progress, there is a specific TestRunSplunkReport that produces a specific CSV file in a Splunk friendly format.

To produce this file simply add the following line anywhere in your test assembly.

        using Microsoft.VisualStudio.TestTools.UnitTesting;
        using TAF.Core;
        
        public class MyTestClass : TestSet
        {
            [ClassInitialize]
            public void ClassSetup()
            {
                TestRun.Instance.AddExecutionListenerIfNotAlreadyAdded(new TestRunSplunkReport());
            }
        }
        

Since this is implemented as a TestRun listener it doesn't matter where in your code it's added, but since test case results only are saved after this report is added it should preferably be in [AsseblyInitialize] or [ClassInitialize] methods.

This report comes with a number of parameters to customize behavior. You can set Splunk specific parameters like:

• index
• sourcetype
• source

• Outfile name and path
• Writing mode (streaming or at the end of test execution)
• If any pre-existing file should be overwritten or appended to

Known errors

Dealing with system level tests you sometimes have to live with the same errors for weeks at the time. A test automation that always report red loses in trustworthiness since errors should be something to really attend to. To mitigate this TAF has implemented a KnownError mechanism. Adding an error is easy. Known error can be added both at the test set level and the test case level.

In case of a log post about something that failed an extra log post is added explaining how to add this error as a known error. This explanation also includes a suggested string to paste into your test case code to register this error as a failed error. This row usually looks something like this:

        
        If you want to add this error as a known error, please add the following line to your test case:
         
           CurrentTestCase.AddKnownError("Your description of the error", ".*Expected 'Testus' but was '.*'.*");
        

Known errors are registered by using the method AddKnownError() at the TestSet or the TestCase. The scope of the known error depends on the placement.

If a log post with a plain text representation matching the regular expression string is encountered during execution it is reported as a known error. If it is not encountered a suggestion to check it out is made.

Adding a known error to a TestCase

Known errors are added to tests like this:

        
            [TestMethod]
            public void MyFirstTest()
            {
                CurrentTestCase.AddKnownError("IdentityService failing", ".*Cannot access service at '.*'.*");
            }
        

If a log post with a plain text representation matching the regular expression string is encountered during execution it is reported as a known error. If it is not encountered a suggestion to check it out is made.

Adding a known error to a TestSet

Since tests in a given test class often has a lot in common it's possible to register Known Errors to test sets too. A known error registered to a test set is treated like it is registered to all tests in the test class. A test set level known error is registered like this:

        
            [ClassInitialize]
            public void MyFirstTest()
            {
                AddKnownError("IdentityService failing", ".*Cannot access service at '.*'.*");
            }
        

The main difference is that since the test class is a subclass of TestSet the AddReport used without the CurrentTestCase object is registered to the TestSet itself.

TAF Core

The main purpose of TAF Core is to provide basic structures for test execution. TAF Core come with:

The relationship between the components of the TAF Core module could be described as follows:

As can be seen there is only a single TestRun, but it holds several TestSets (test classes) with potentially multiple TestCases within them.

Run settings usage

Build-in run settings

For convenience a number of pre-defined settings has been implemented. The pre-defined ones are:

These settings may be changed from the command line, if tests are started from the command line as described here, or programmatically if started in some other way (like TFS build server MSTest plugin).

Setting a built-in runSetting can be done like this:

        
            TestRun.Instance.RunSettings.SetRunSetting(RunParameter.TestRunLogFolder, "\\\\MyShare\\MyLogFolder");
        

Reading a value from a build-in run setting is done with a similar method:

        
            var logFolder = TestRun.Instance.RunSettings.GetRunSetting(RunParameter.TestRunLogFolder); //Returns value for TestRunLogFolder
        

Custom run parameters

You may add your own run parameters. These can be set from the command line, or programmatically - like the pre-defined ones.

Custom run parameters are suitable for configuration of your tests. For example, you might have differences between different environments (URLs, users, and so forth). For this purpose, the custom parameters are included in TAF.

To define a custom parameter, you can either use the command line to set these, as described here, or state them programmatically, like this:

        
            TestRun.Instance.RunSettings.SetCustomParameter("Environment", "Acc");
        

To get the value of a parameter that is set you use the corresponding get method:

        
            var currentEnvironment = TestRun.Instance.RunSettings.GetCustomParameter("Environment");  //Returns 'Acc' if used as in the example above
        

There is a special case where you use the command line interface of TAF to read whole objects into the solution at runtime. This is described in the chapter about the TAF command line interface.

Advanced TAF usage

Sub-sections:

1. Logging
2. Command line usage
3. Adding TFS (Team Foundation Server) reporting
4. Position based GUI element identification
5. Using TAF with NUnit or XUnit

Logging

If you use TAF as it is you rarely have to think twice about logging since TAF handles this for you. TAF internally performs logging to its own logging levels that are adapted to use with test automation. The logging levels of TAF.Logging are:

1. Debug
2. Info
3. Execution step
4. Passed verification
5. Verification problem
6. Failed verification
7. Execution problem

In case of a failing log row an addition suggestion for how to add it as a known error is written to the log.

The Info log post type has a subtype of FileLogPost that saves a file to the log.

Desktop screenshots

This log post type is also used by the TAF.Core.GenericGuiDriver module that enables to take screenshots of the desktop, for example when errors occur. Some modules, like TAF.WebSupport also produces browser screenshots upon errors.

If a test execution encounters an error so severe further execution seems futile a HaltFurtherExecution event is raised and test execution halted, jumping straight to test results evaluation for the test case.

Creating log posts

It's possible to add log entries yourself to the testCaseResult. Both the TestCase, the TestCaseResult, and the LogPostList has Log() methods.

The log posts are prepared for both HTML view for those types of reports, and plain text view for console view. The log message content is analysed and the various parts of the string is divided into their corresponding categories for formatting purposes and to present usable known-error suggestion strings.

There is one quick and sloppy way of producing log posts, and one more thorough. In most cases the quick one with a plain text string will do.

        
                CurrentTestCase.Log(new LogPost.LogPostInfo("No more text in element."));
        

Sometimes it's meaningful to use more detailed log post descriptions. Then a pattern like the one in the example below could be used.

        
            var logPostMessages = new List<LogPostMessagePartType>
            {
                new LogPostMessageText("Could not find the data "),
                new LogPostMessageData(testData),
                new LogPostMessageText(" in the string provided.")
            };
            CurrentTestCase.Log(new LogPost.LogPostDebug(logPostMessages));
        

Command line usage

TAF.Core can be used as a console application. If used from the command line it consumes the TAF arguments and sends the rest of the arguments on to the MSTest execution engine.

It's always possible to print the help text of the current version of TAF by invoking TAF.Core.exe without parameters.

        
        C:\TAF>TAF.Core.exe
        

TAF CLI is used to set run parameters like log folder, run name, and your own custom arguments. It is also meant to be able to use for environment specific test data management.

Run parameters

Setting run parameters from the command line can be useful for CI/CD integration. All regular TAF run parameters can be set from the command line, but also the run name parameter and the log folder parameter.

        
        C:\TAF>TAF.Core.exe runName=AccEnvSmokeTest logFolder=C:\logs\
        

A list of run parameters in TAF include:

• TestRunLogFolder (base folder for the test run logs)
• PathToReportLogo (an URI to an image file to use in report headers)
• TestSetLogFolder (if set overrides test run default log folder. Generally set by the framework itself.)
• HtmlReportsLinkPrefix (used to enable saving of HTML reports to a web server based file - making file references protocol http:// rather than file://)

Custom run parameters

If a command line argument is added that includes an equal sign ('=') it's interpreted as adding your own custom run parameter. For example, the command argument:

        
        C:\TAF>TAF.Core.exe env=acc
        

Could be retrieved at runtime with the normal run parameter getter:

        
            var currentEnvironment = TestRun.Instance.RunSettings.GetCustomParameter("env"); //Returns Acc, if used as above
        

This will retrieve the value of 'env' as a string.

Test data management

One straightforward way of using the CLI for test data management is to use parameters, as described above. However, this will not allow any more complex use. For more complex scenarios it's possible to read JSON formatted objects to your test execution. This could either be done by specifying a JSON string directly, or by stating a file name to a JSON object text file.

        
        C:\TAF>TAF.Core.exe json=\\MyTestDataShare\User.json
        

If the string after the json keyword is a valid JSON string it's interpreted and read as a JSON object itself:

        
        C:\TAF>TAF.Core.exe json={"myobject":"MyValue"}
        

The test data objects are then read runtime by TAF using:

         
            public User GetUser()
            {
                foreach (var testDataObject in TestRun.Instance.RunSettings.GetCustomObjects())
                {
                    if (testDataObject is User)
                        return (User)testDataObject;
                }
                return null;
            }
        

Adding TFS (Team Foundation Server) reporting

TAF can automatically report test execution results to test cases in TFS. The test result, the execution log, and saved files during execution will be saved to the TFS test run instance as well, even if the test run is performed in a CI/CD environment.

To enable reporting test results to TFS/VSTS you could use the normal Visual Studio TFS integration, but this require special Visual Studio licenses. Hence TAF includes an optional nuget package for license less TFS/VSTS integration.

To enable this package add a nuget dependency to the TAF.TfsIntegration package at nuget.org.

To propagate test case results to test cases in TFS or VSTS you first have to etablish a connection to the TFS/VSTS server, and for each test case state what TFS test case it corresponds to.

Setting up a TFS/VSTS connection could be done in two ways. The most common approach is using the static TfsConnection.Setup() method in a relevant class or assembly instansiation method of your test project. If you intend to interact with the TFS connection in your testing you could get the instance of it with var tfs = new TfsConnection() constructor. Either way you will state the URL to the TFS instance, the project name in TFS, the test suite id in TFS, and the TFS test plan id.

Example:

            using TAF.TfsIntegration;
            using TAF.Core;
            using Microsoft.VisualStudio.TestTools.UnitTesting;
        
            namespace MyTestingProject
            {
                public class MyTestClass : TestSet
                {
                    [AssemblyInitialize]
                    public static void AssemblySetup()
                    {
                        TfsConnection.Setup("http://tfs.mycompany.com:8080", "ExitingNewProductProject", 1234, 1235);
                    }
                }
            }
        
        

When the connection parameters are stated the test cases can be tagged for TFS. This is done by adding a test case execution report for TFS, like this:

           
            [TestMethod]
            public void MyTestCase()
            {
                AddReport(new TestCaseTfsReporter(12345));
                //... perform testing...
            }
        

Full example:

            using TAF.TfsIntegration;
            using TAF.Core;
            using Microsoft.VisualStudio.TestTools.UnitTesting;
        
            namespace MyTestingProject
            {
                public class MyTestClass : TestSet
                {
                    [AssemblyInitialize]
                    public static void AssemblySetup()
                    {
                        //Numbers below represents TFS Test suite ID and TFS test plan ID respectively.
                        TfsConnection.Setup("http://tfs.mycompany.com:8080", "ExitingNewProductProject", 1234, 1235);
                    }
           
                    [TestMethod]
                    public void MyTestCase()
                    {
                        AddReport(new TestCaseTfsReporter(12345)); //Number given is test case id in TFS
                        //... perform testing...
                    }
                }
            }
        
        

Position based GUI element identification

Sometimes identifying GUI elements from their IDs or other attributes is futile. Many type of GUI applications are built without relevant attributes, or dynamic attributes that only make sense internally.

For this purpose the generic GUI methods include PositionBasedIdentification. Basically this is used by having an initial collection of GUI element, and then filtering away other elements based on their location, until only the relevant element should remain.

Example below:

                [TestMethod]
                public void ElementCollectionTest1()
                {
                    var web = new WebInteractionSelenium(CurrentTestCase);
                    web.Navigate("file://TestPage.html");
        
                    DomElement passwordField = PositionBasedIdentificationSelenium
                        .OutOfAllElementsInCurrentBrowserTab(web)
                        .KeepOnlyObjectsAtTheSameLevelAs(LoginButton)
                        .TheObjectFurthestToTheLeft();
        
                    web.Write(passwordField, "TopSecretPassword");
                }
            

For some of the filtering methods offset margin tolerances may be applied, for example:

        
                    DomElement userNameField = PositionBasedIdentificationSelenium
                        .OutOfAllElementsInContainerElement(web, LoginBox)
                        .KeepTheBottomMostObjects(marginInPixels: 30)
                        .TheOnlyObjectThatRemains();
        
                    web.Write(userNameField, "BatCaveMaster");
            

There are a few typical GUI patterns where TAF include special methods to help.
For example it's quite common to have a label to the left of a field. This is solved in TAF with:

                    DomElement passwordField = PositionBasedIdentificationSelenium
                        .OutOfAllElementsInContainerElement(web, LoginBox)
                        .TheElementDirectlyToTheRightOf(PasswordLabel);
        
            

Similar methods of identifying GUI elements are implemented for both WPF and Java rich based clients as well as the web example above.

Using TAF with NUnit or XUnit

Some teams prefer NUnit or XUnit over MSTest. To use these rather than the built in MS Test support you'll need to make three adjustments. You use both TAF and NUnit like you normally would, but you add the following:

        
            [SetUp] //Run just before test start
            public void TestSetup(){
                SetupTestCase();
            }
        
            [TearDown] //Run just after test is finished
            public void TestCleanup(){
                TearDownTestCase();
            }
        
            [TestFixtureTearDown] //Run after all tests in this test class has finished
            public void ClassTearDown(){
                TestSetCleanUp();
            }
        

One way of doing this of course, is to create a base class for your test classes and add these method calls to it. Just make sure it is a sub-class of  TAF.Core.TestSet .

For XUnit, or any other test runner framework, just make sure the corresponding methods are called upon test start or test finish.

Developing TAF itself

Sub-sections:

1. Building your own test case report
2. TAF execution listener
3. Implement your own driver support
4. Updating the user experience in reports

Building your own test case report

One of the features of TAF is the logging possibilities. If you want to create your own report, the easiest way is to implement the ITestCaseExecutionReport interface. This interface only has one method, Report(TestCase testCase).

Sample custom TestCaseExecutionReport that saves the test results to a text file in the log folder:

        
            public class PureTextFileTestCaseExecutionReport : ITestCaseExecutionReport
            {
                public void Report(TestCaseResult testCaseResult)
                {
                    File.WriteAllText(
                        testCaseResult.LogPostList().GetLogFolder() +
                        "\\" + testCaseResult.LogPostList().TestCaseName + ".txt",
                        testCaseResult.ToString());
                }
            }
        

A test report is attached to a test case with the AddReport() method of a test set or a test case, in a fashion described below.

            [TestInitialize]
            public void TestSetUp(){
                AddReport(new PureTextFileTestCaseExecutionReport());
            }
        

TAF execution listener

To enable a wider use of TAF there is a TafExecutionListener implemented in TAF.Core. This could be extended to your own needs. Just create a subclass of the TafExecutionListener and override the methods available. Example below:

        
            public class MyExecutionListener : TafExecutionListener
            {
                public override void TestCaseExecutionStartedListener(TestCase testCase)
                {
                    Console.WriteLine(testCase.Name + " execution started");
                }
            }
        

The methods possible to override are:

•  TestCaseExecutionStartedListener(TestCase testCase) 
•  TestCaseExecutionEndedListener(TestCase testCase) 
•  TestCaseReported(TestCase testCase) 
•  TestRunExecutionStartedListener(TestRun testRun) 
•  TestRunExecutionEndedListener(TestRun testRun) 

You add the listener to the test solution by adding it to the TestRun instance, like this (example):

        
            TestRun.Instance.AddExecutionListenerIfNotAlreadyAdded(new MyExecutionListener());
        

Implement your own driver support

When implementing support for a new technology, or altering technology support, it's important to understand the structure of the code of TAF, and the terms used.

GUI elements

TAF declares the element for interaction before they are used, and at runtime the runtime object for these are attempted to be found by the Driver.GetRuntimeElement method, as described below. Benefits of declaring elements are to get better logging, and for potential caching (technology dependant).

The main component of GUI interaction is the IGuiObject. This represents e.g. a button, a frame, or an input field. Specific technology implementations use their own elements derived from IGuiObject. For example, the TAF.WebSupport declares a IDomElement, that also is a IGuiObject, but with additional parameters specific for web based technologies. The GetRuntimeElement() method for web, as implemented by the SeleniumTafDriver returns a IWebElement to be used by the TafGuiDriver (again, see below).

GUI object declaration relations

The IGuiDriver interface makes all implemented drivers implement the method public object GetRuntimeElement(IGuiObject guiobject); . Through polymorfism this method returns the relevant object type, like IWebElement for Selenium, or AutomationElement for Windows.

Driver

A driver is anything that enables us to interact with some interface. It could be RestSharp providing HTTP support, or Selenium providing support for interacting with browser elements.

Drivers for different GUI technologies have a lot in common, and it's a hassle coming up to speed with logging and robustness for driver usage. To make the most of technology and re-use the GUI driver structure of TAF is layered with a common and generic TafGuiDriver at its core. The TafGuiDriver uses a MethodInvoker and lists of common method names for interaction for different well-known drivers making it very versatile. The one method from the IGuiDriver interface that TafGuiDriver does not implement is the GetRuntimeElement, since the mechanism for this is very different depending on the driver and technology used.

Methods implemented by the TafGuiDriver, and their default call methods in attempt order:

• Click ("Click()", "Click", "Invoke", "FireEvent")
• Write ("SendKeys", "SendKeys(System.String)", "Write", "Type", "Submit", "set_Text")
• GetText ("Text", "GetText()", "GetText", "get_Text", "get_Text()", "System.String get_Text()", "Text()")
• Exist
• IsDisplayed ("Displayed", "IsDisplayed", "Displayed()", "IsDisplayed()")
• IsEnabled ("get_Enabled", "Enabled", "IsEnabled", "Enabled()", "IsEnabled()")
• QuitDriver
• GetAttributeValue ("GetAttribute", "GetAttribute(string)", "GetProperty(string)", "Attribute[string]", "GetAttribute[string]")
• LogDebuggingInfoAfterDeviation

GUI driver's declaration relations

For specific driver implementations there are two main ways these methods can be adapted; they may be overridden, and the method names may be substituted for more suitable ones for the underlaying driver used.

Drivers for specific technologies

To enable driver substitution there is a technology specific layer above the generic GUI driver. For example, the testing of web require interaction with cookies, browser console log, navigation, and JavaScript execution. This requirement doesn't change depending on the driver, but the technology. Therefore, technology specific interfaces and implementations extend the generic driver classes and interfaces. Hence the ITafWebDriver interface, implemented by the SeleniumTafDriver.

GuiInteraction

TAF introduces GUI interaction mechanisms separate from the driver. The GuiInteraction uses the driver, but is the one providing logging, retrying if conditions are not met, and mechanisms to gather info upon failure.

In a similar layered approach as with the driver, the interaction methods provide generic interaction methods at the core level (through the TafGuiInteraction class). These are extended and possibly overridden by technology specific interaction methods like the WebInteraction class, implementing the IWebInteractions interface, in turn extending the IGuiInteraction interface implemented by TafGuiInteraction.

Verification

As other parts there already are defined verification mechanisms in the TAF.GenericGuiDriver module. These can be overridden and extended, and uses the already declared interaction methods described above.

Two types of verification methods are included in the TAF.GenericGuiDriver module; generic verifications, implemented by the TafVerificationMethods (implementing the IGenericVerificationMethods interface), and element property verifications, implemented by TafGuiObjectVerificationMethods (implemented by its own interface). Technology specific verification methods extend or overrides these.

Updating the user experience in reports

The colour codes used in reports are stored in the TAF.UxParameters project files. If you update these the GUI will be updated too.

There are several other places where GUI information is stored. The TAF.Core.TestCaseHtmlReportJavascript.js file is the one producing the TestCaseHtmlReport output file. These contain CSS information used for formatting displays.

FAQ

1. What's the difference between the IGuiObject and the IDomElement?
   
   
   The IDomElement is a IGuiObject since it's a sub-instance of the IGuiObject. Whenever a method takes a IGuiObject parameter, you may safely enter a IDomElement, like for example a DomElement.
   
   
   
2. Why doesn't TAF come with Asserts?
   
   
   The classic Assert structure halts further test execution upon failure. To avoid confusion about behaviour TAF has a similar mechanism called  Verify()  that performs the check but continue execution. In TAF it's certain hindered flow control actions (like trying to click an element that can't be found that stops further test execution.
   
   
   
3. Why is TAF divided into so many confusing sub-modules?
   
   
   The dependency system makes it possible to keep a minimum dependent code base for each user. The true reason though is that it makes better code since the structure of dependencies is enforced upon the developers - and it makes it easy to unit test the solution thoroughly - giving the users a more stable and reliable product.
   
   
   
4. How do you test your own framework?
   
   
   With rigorous unit testing and a number of system level tests, and in some parts, we actually use TAF functionality to test parts of TAF - although it sounds counter-intuitive. But for example, we use the TAF.WebSupport to test the HTML based reports of TAF.Core. For each project/module in TAF there is a corresponding test project with tests. However, these are kept separate to minimize the size and dependency list of TAF. Since they are separate traces of them can be seen in the TAF functionality (e.g. you can state your file system in some components, and you can run the command line interface with the test mode argument).
   
   
   
5. It seems awkward not to interact with the elements themselves, but with an instance of the framework. Why is it like this?
   
   
   TAF was developed with rather technical, but not regularly coding testers, in mind. To people used to system testing rather than unit testing it's natural to use the system itself as the object, rather than its components. To them it's also beneficial with the auto-completion feature of modern IDE's to get inspiration what's possible to do in the test construction. Working at the system test level it's also great having the C# possibilities of 'named and optional arguments' for test data management and code maintenance reduction.
   
   
   
6. I've heard about a Java based TAF. What's the difference?
   
   
   Yes, there is a Java based version of TAF. The TAF Java is similar to TAF.NET. In some parts they differ however. The Java version doesn't support TFS, or WPF. Instead it supports Testlink as test management system, and Java client applications (AWT, Swing, JavaFX and so forth). The Java version also has a limited functionality GUI and has a driver for EyeAutomate for smart image recognition GUI interaction. But the C# version of TAF is developed later, and thus has a little better code structure. It's also easier to keep nuget dependencies up-to-date than maven ones... ;) The Java based TAF can be found at GitHub.
   
   
   
7. Why have you made it so counter-intuitive and cumbersome to initiate a new Selenium WebDriver?
   
   
   First of all: Sorry about that. Sincerely. On the positive side; we are attempting a better way, but it will break backward compatibility and there are users out there who we want to keep being friends with.
   
   
   
8. Why the hard dependency to MS Test? We'd rather use NUnit/XUnit.
   
   
   We had to pick a runner, at least until we've been able to find a way to hook into the framework execution methods for each driver. Also see the chapter on Using TAF with NUnit or XUnit. The reason is we need to initiate code just before each test execution, and after a test is finished, and the whole test run is finished.
   
   
   

Background of TAF

TAF was initiated 2016 when the test automation team of Zington got fed up with repeatedly having to spend months to produce robust and stable test automation solutions at different customers. During a workshop the idea of a re-usable and extendable architecture for test automation - regardless of the driver being used or level of testing performed. It was concluded that the logging, timing, and reporting structural problems were the same regardless of underlying tool set used to drive automation, or the high-level use like with Cucumber of Specflow.

The next time a test automator of the group got a new greenfield project to establish test automation for an organization it was agreed the framework should be developed with a separation of concerns pattern and that the framework would be kept property of Zington but open-sourced. The next initiative for test automation at another customer as expected came with new requirements for the automation and those were incorporated into the framework too. This kept up for a number of customers, each new one building to and adding to the framework - and made us constantly refactor TAF while the architecture slowly matured.

The first versions of TAF was built using Java and maven. It is still being developed and you can find it on GitHub (also see FAQ about differences between the Java and C# versions of TAF).

In 2017 it became increasingly apparent that many customers who developed software in the Microsoft development universe saw it as a great obstacle to bring Java based tools into their tool set, and a C# port of TAF was initialized. The first version of TAF C# (also known as TAF.NET) met its customers in early 2017, and since then TAF for C# has shared the same ideas and concepts as TAF for Java, but being developed separately.

During Q2 2018 the C# version of TAF underwent a major refactoring, breaking backward compatibility.