Automated UI testing of AngularJS applications using TFS – Part II

11 03 2015

In the first article “Automated UI testing of AngularJS applications using TFS – Part I” we started with the theory on running E2E-tests with protractor. This post and the third part will show the automation of this process. Inside this part the focus is the creation of the custom activity to achieve this.  Requirements for the automation of E2E-tests with protractor

Before we start with the creation of the activity, make sure to fulfil the necessary requirements, to get the E2E-tests up and running with protractor on an installed Team Foundation Server:

  • On the build controller
    • Visual Studio Premium or higher
    • nodejs
  • For development
    • Visual Studio

Create a custom activity

We start with the creation of a custom build activity to automate the E2E testing with protractor. Inside this activity we handle everything that is necessary to run the protractor tests and publish the test results. With the custom activity we are able to integrate the automated protractor tests very easy into different build templates. This improves the reusability in a great way.

Start with a new project and create an “Activity Library” in your preferred programming language.


Now we can start to create the custom activity. By default a new *.xaml file was created inside the new project. The first thing we drop from the toolbox to the activity is a sequence. This is the container for all activities needed for running protractor tests. The following picture shows the necessary arguments and variables which should be created on the level of the sequence.



To setup the arguments and variables additional references are needed. Please add / make sure that the following references are added to the project:

  • Microsoft.TeamFoundation.Build.Activities
  • Microsoft.TeamFoundation.Build.Client
  • Microsoft.TeamFoundation.Client
  • Microsoft.TeamFoundation.Build.Workflows
  • Microsoft.TeamFoundation.Deployment.Workflow
  • Microsoft.TeamFoundation.VersionControl.Client
  • Microsoft.TeamFoundation.VersionControl.Common
  • System.Drawing
  • System.Activities.Presentation
  • PresentationCore
  • PresentationFramework
  • WindowsBase

Note: This list depends on the Version of your installed Visual Studio. The following references are necessary for Visual Studio Premium 2013. Make sure that the installed Visual Studio version on the build controller is the same, you use for the activity creation, to avoid version problems.

The next step is, to retrieve the required values from the build, to setup the variables with values. With the help of the “GetEnvironmentVariable”-Activity reading the “WellKnownEnvironmentVariables.SourcesDirectory” the value can be retrieved. The retrieved result value should then be stored to the “SourcesDirectory” variable.


The “GetTeamProjectCollection”-Activity should store the data to the “TeamProjectCollection” variable. With the “GetBuildDetail”-Activity we receive the information we need to store inside the “BuildDetail” variable.

In case you cannot find these activities inside the toolbox you need to customize the toolbox. Click right into the toolbox pane and click on the “choose items” menu item. Inside the dialog you can add the Microsoft.TeamFoundation.Build.Workflow.dll. Depending on the Visual Studio version you are using, this can be found at a location similar to “C:\Program Files (x86)\Microsoft Visual Studio 12.0\Common7\IDE\PrivateAssemblies”

Go on with the nodejs package manager and install all dependencies. For this step we create an “InvokeProcess”-Activitiy. This activity is used to trigger a powershell script which is responsible for installing the required dependencies. The “InvokeProcess”-Activity should be configured that the standard output is written as a build message and all errors are written as build errors.


The properties of the “WriteBuildMessage”-Activity and the “WriteBuildError”-Activity should be setuped like you can see on the following pictures.



In case you want to log the complete console output of the invoked process to the build log, change the “BuildMessageImportance” to high.

The next thing is the configuration of the step by itself. A powershell script is executed which installs all dependencies (in case they aren’t already installed). Therefore the followings properties should be set on the “InvokeProcess”-Activity.


string.Format(CultureInfo.InvariantCulture, @" ""& '{0}' '{1}' "" ", SourcesDirectory + InstallDependenciesScript, SourcesDirectory + ProtractorConfigFile);



The complete script which installs all dependencies can be found here.
The next process step is the start of the selenium server (all protractor tests needs an up and running selenium server). Another “InvokeProcess”-Activity is needed to achieve this. Configure this activity like described before. Only change the argument property to the following:

string.Format(CultureInfo.InvariantCulture, @" ""& '{0}' "" ", SourcesDirectory + WebdriverManagerScript);

This “InvokeProcess”-Activity calls a powershell script which starts the selenium server in the background thru the webdriver-manager cmdlet. The complete powershell script can be downloaded here.

The next activity we need to add is a “Delay”. We need to add this because the selenium server needs some time to be up and running. A delay of 10 sec should be enough.

At this point everything is installed and up and running. With another “InvokeProcess”-Activity the test will be executed. Configure the activity as described before and change the arguments and filename properties.

"\"" + SourcesDirectory + ProtractorConfigFile + "\""



This activity calls protractor with the created protractor configuration file that should exists inside the source control of the TFS. Protractor is executed and runs all the tests provided thru the protractor configuration file.

Maybe you need to adjust the path to the protractor.cmd depending under which user the build controller is running and where npm installed the protractor package.

As described inside the first post the trx reporter is attached thru the protractor configuration. Therefore we found a generated trx file with all results inside the working folder. With mstest the results can be published and associated with the current build. This can be done by another “InvokeProcess”-Activity with the following properties.


string.Format(CultureInfo.InvariantCulture, @"/publish:""{0}"" /publishresultsfile:""{1}"" /publishbuild:""{2}"" /teamproject:""{3}"" /platform:""{4}"" /flavor:""{5}""", TeamProjectCollection.Uri, SourcesDirectory + @"\ProtractorTestResults.trx", BuildDetail.BuildNumber, BuildDetail.TeamProject, Plattform, Configuration)


"C:\Program Files (x86)\Microsoft Visual Studio 12.0\Common7\IDE\MSTest.exe"

Note: Depending on the Version of Visual Studio you have installed the mstest.exe has different arguments available. The publish arguments of mstest.exe are only available if you have installed Visual Studio Premium / Ultimate or Visual Studio Test Professional.
The next step is to read the test output from the trx file and evaluate the results. In case of an error while executing the test the build should fail. With an “Assign”-Activity the trx result output file is read and assigned to the “TestOutcome” variable. Set the following properties for the activity to achieve this.




System.IO.File.ReadAllText(SourcesDirectory + @"\ProtractorTestResults.trx")

To evaluate the “TestOutcome” variable an “If”-Activity is needed. If an error is written to the test results a build error will be published otherwise everything is fine and nothing needs to be done. The condition of the “If”-Activity should be the following:

!string.IsNullOrEmpty(TestOutcome) && !TestOutcome.ToUpper().Contains(@"RESULTSUMMARY OUTCOME=""PASSED""")

Now you are done with the creation of the custom activity. If you like you can download a version of the complete activity here.

In the next blog post the creation of a customized build template and the integration into a TFS Build is described.

Note: All the described steps and screenshots are from TFS 2013 and Visual Studio Premium 2013. On other versions some differences may arise.




Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s

%d bloggers like this: