Coding For Life

Running Saucelabs in a Geb, Spock, Groovy and Gradle stack

John Roach -

TL;TR; You can visit SauceLabs samples which is hosted on GitHub to which I contributed per this blog post.

Gradle is an open source polyglot build automation system. I find it more powerful than maven. The best features for me are its build reporting and its fully programmable build capabilities. You can do many complex stuff that was not possible with Maven.

SauceLabs is an awesome tool where one can run automated/manual tests on numerous different combinations of OS, browsers and devices. It only makes sense that functional tests written for a project to run on the configurations your customers use no matter how diverse.

Apache Groovy integrates smoothly with any Java program, and delivers powerful features, including scripting capabilities, Domain-Specific Language authoring, runtime and compile-time meta-programming and functional programming*. It allows developers to quickly prototype, which is perfect when writing tests for software that is constantly changing.

Spock is a testing and specification framework for Java and Groovy applications. What makes it stand out from the crowd is its beautiful and highly expressive specification language*. Spock gives us the capability to use the same language for both unit and functional tests. It also has awesome extensions that could be used. Such as; @Issue to point to JIRA issues or its very powerful Mocking capabilities. I hope to do a more detailed post later on regarding this.

Geb, is an awesome browser automation tool. It has a built in powerful approach called Page Object modeling. It also uses WebDriver, the elegance of jQuery for content selection and with Groovy it is simply the perfect match.

Ok, now we know what all the moving pieces do let us get cracking.

First order of business is to bring all the requirements to get SauceLabs working. Now the thing is I wouldn’t want to make QA/Developers beholden to Sauce only. So this will allow one to switch to using a local Firefox installation. You will also notice the usage of Xvfb. This will allow one to run the Firefox tests without the need of a screen which is much needed when you are running the tests on a CI or on a Vagrant server box.

    ext { 
        drivers = ["firefox", "sauce"] 
        ext { 
            groovyVersion = '2.4.7' 
            gebVersion = '0.13.1' 
            seleniumVersion = '2.53.1'
            ciSauceVersion = '1.116'
            sauceJavaCommonVersion = '2.1.21'
            sauceJunitVersion = '2.1.21'
            saucerestVersion = '1.0.33'
            sauceOndemandDriverVersion = '1.2'          
            seleniumClientFactoryVersion = '1.2'
            junitVersion = '4.12' 
        } 
     } 
   
    apply plugin: "groovy" 
    
    repositories {
        mavenCentral() 
    }
    dependencies {
        testCompile "org.gebish:geb-spock:$gebVersion" 
        testCompile("org.spockframework:spock-core:1.0-groovy-2.4") {
            exclude group: "org.codehaus.groovy" 
        }
        testCompile "org.codehaus.groovy:groovy-all:$groovyVersion" 
        testCompile "org.seleniumhq.selenium:selenium-firefox-driver:$seleniumVersion"
        testCompile("com.athaydes:spock-reports:1.2.12") {
            transitive = false // this avoids affecting our version of Groovy/Spock 
         } 
        // if you don't already have slf4j-api and an implementation of it in the classpath, add this! 
        testCompile 'org.slf4j:slf4j-api:1.7.13'
        testCompile 'org.slf4j:slf4j-simple:1.7.13'
        testCompile group: 'com.saucelabs', name: 'ci-sauce', version: "$ciSauceVersion" 
        testCompile group: 'com.saucelabs', name: 'sauce_java_common', version: "$sauceJavaCommonVersion"
        testCompile group: 'com.saucelabs', name: 'sauce_junit', version: "$sauceJunitVersion"
        testCompile group: 'com.saucelabs', name: 'saucerest', version: "$saucerestVersion" 
        testCompile group: 'com.saucelabs.selenium', name: 'sauce-ondemand-driver', version: "$sauceOndemandDriverVersion"
        testCompile group: 'com.saucelabs.selenium', name: 'selenium-client-factory', version: "$seleniumClientFactoryVersion"
        testCompile group: 'junit', name: 'junit', version: "$junitVersion" 
}

drivers.each { 
    driver -> task "${driver}Test"(type: Test) {
        if("${driver}" != "sauce") { dependsOn "startXvfb" }
        reports {
            html.destination = reporting.file("$name/tests")        
            junitXml.destination = file("$buildDir/test-results/$name")
         }
         outputs.upToDateWhen { false } // Always run tests systemProperty 
         "geb.build.reportsDir", reporting.file("$name/geb") 
         systemProperty "driverType", "${driver}"
         systemProperty "geb.env", System.properties['geb.env'] ?: driver environment 'DISPLAY', ':99' 
  }
} 

task startXvfb(type: Exec) {
    workingDir "$buildDir/../scripts"
    environment 'DISPLAY', ':99'
    commandLine "./runXvfb.sh"
    ignoreExitValue true
} 

test { dependsOn drivers.collect { tasks["${it}Test"] } enabled = false }

For those wondering what the runXvfb.sh script looks like here it is:

    #!/bin/bash
    export DISPLAY=:99
    echo "Starting Xvfb Server"
    /usr/bin/Xvfb :99 -ac &> /dev/null & echo "Running..."

Now the thing is that you can if you want create the SauceLabs driver GebConfig.groovy. However this causes issues especially with SauceLabs time limits. Once you create the driver SauceLabs will assume that is a single test. So if you have like 200-300 functional tests which might take 3 hours plus SauceLabs will timeout.

The best way we found out how this could be handled is by creating a BaseSpec where all other Specs extend from and create the driver on a Spec to Spec basis. This allowed us to both separate tests and have more segregated test results.

So now to the Gravy, I mean Groovy file. See what I did there… You know because what I am about to show is the gravy of  this whole post… I mean you saw it right? 😉

package com.ag.functionaltest.accelerator.specs
import com.ag.functionaltest.accelerator.utils.SpecialSauceOnDemandTestWatcher 
import com.saucelabs.common.SauceOnDemandAuthentication 
import com.saucelabs.common.SauceOnDemandSessionIdProvider
import com.saucelabs.common.Utils
import geb.spock.GebSpec
import groovy.json.JsonSlurper
import org.junit.Rule
import org.junit.rules.TestName
import org.junit.runner.Description
import org.openqa.selenium.firefox.FirefoxDriver
import org.openqa.selenium.firefox.FirefoxProfile
import org.openqa.selenium.remote.DesiredCapabilities
import org.openqa.selenium.remote.RemoteWebDriver

class BasePageGebSpec extends GebSpec implements SauceOnDemandSessionIdProvider {
    public String username = System.getenv("SAUCE_USERNAME")
    public String accessKey = System.getenv("SAUCE_ACCESS_KEY")
    /** * Represents the browser to be used as part of the test run. */
    private String browser 
    /** * Represents the operating system to be used as part of the test run. */
    private String os
    /** * Represents the version of the browser to be used as part of the test run. */
    private String version
    /** * Represents the deviceName of mobile device */ 
    private String deviceName
    /** * Represents the device-orientation of mobile device */ 
    private String deviceOrientation
    /** * Instance variable which contains the Sauce Job Id. */ 
    private String sessionId public 
    
    SauceOnDemandAuthentication authentication = new   
    SauceOnDemandAuthentication(username, accessKey) 
    
    @Rule
    public SpecialSauceOnDemandTestWatcher resultReportingTestWatcher = new SpecialSauceOnDemandTestWatcher(this, username, accessKey, true)
    @Rule
    public TestName name = new TestName() {
        public String getMethodName() {
           return super.getMethodName() 
        }
     } 
    
    /** * * @return the value of the Sauce Job id. */ 
    @Override
    public String getSessionId() { return sessionId } 
    
    public void setup() throws Exception { 
        Map<String, String> capMap
        String capabilityString = System.getProperty("geb.saucelabs.capabilities") 
        String driverType = System.getProperty("driverType")

        capabilityString = '{"browserName": "Firefox", "platform": "Windows 10", "version": "42"}'
        if (capabilityString && (driverType == "sauce")) {
            capMap = new JsonSlurper().parseText(capabilityString) 
            DesiredCapabilities capabilities = new DesiredCapabilities(capMap) 
            String methodName = name.getMethodName() 
            String specName = this.class.getSimpleName()
            capabilities.setCapability("name", String.format("%s.%s", specName, methodName))
            capabilities.setCapability("newCommandTimeout", 180)
            driver = new RemoteWebDriver( new URL("http://" + authentication.getUsername() + ":" + authentication.getAccessKey() + "@ondemand.saucelabs.com:80/wd/hub"), capabilities)
            this.sessionId = (((RemoteWebDriver) driver).getSessionId()).toString() 
         } else {
           FirefoxProfile profile = new FirefoxProfile() 
           driver = new FirefoxDriver(profile) 
          }
} 

@Override 
public void cleanup() throws Exception { driver.quit() } 

}

So you might notice that there is a special test watcher called SpecialSauceOnDemandTestWatcher the reason is simply due to how SauceLabs handles test results. We wanted to surface up the Spock test results in SauceLabs. It turned out to be a futile. More on that later.

So the TestWatcher allows us to surface up the results tests.

package com.ag.functionaltest.accelerator.utils

import com.saucelabs.common.SauceOnDemandSessionIdProvider
import com.saucelabs.common.Utils
import com.saucelabs.saucerest.SauceREST
import org.junit.rules.TestWatcher
import org.junit.runner.Description


class SpecialSauceOnDemandTestWatcher extends TestWatcher {
    private String watchedLog = ""
    
     /**
     * The underlying {@link com.saucelabs.common.SauceOnDemandSessionIdProvider} instance which contains the Selenium session id.  This is typically
     * the unit test being executed.
     */
    private final SauceOnDemandSessionIdProvider sessionIdProvider

    /**
     * The instance of the Sauce OnDemand Java REST API client.
     */
    private final SauceREST sauceREST

    /**
     * Boolean indicating whether to print the log messages to the stdout.
     */
    private boolean verboseMode = true


    public AGSauceOnDemandTestWatcher(SauceOnDemandSessionIdProvider sessionIdProvider, String username, String accessKey, boolean verboseMode) {
        this.sessionIdProvider = sessionIdProvider
        this.sauceREST = new SauceREST(username, accessKey)
        this.verboseMode = verboseMode
    }

    private void printSessionId(Description description) {
        if (verboseMode) {
            String message = String.format("SauceOnDemandSessionID=%1$s job-name=%2$s.%3$s", sessionIdProvider.getSessionId(), description.getClassName(), description.getMethodName())
            System.out.println(message)
        }
    }

    /*
     * This doesn't fully work due to https://github.com/spockframework/spock/issues/118
     * The new version of spock v.1.1 is supposed to fix this soon
     */
    protected void failed(Throwable e, Description description) {
        this.watchedLog+= description.getMethodName()
        if (sessionIdProvider!=null && sessionIdProvider.getSessionId()!=null) {
            printSessionId(description)
            updateTestStatus()
            if (verboseMode) {
                // get, and print to StdOut, the link to the job
                String authLink = sauceREST.getPublicJobLink(sessionIdProvider.getSessionId())
                System.out.println("Job link: " + authLink)
            }
        } else {
            if (verboseMode) {
                System.out.println("Test succeeded and Session ID doesn't exist!!")
            }
        }
    }

    protected void succeeded(Description description) {
        if (sessionIdProvider.getSessionId()!=null) {
            System.out.println("A test succeeded " + description.getMethodName())
            updateTestStatus()
        } else {
            if (verboseMode) {
                System.out.println("Test Failed and Session ID doesn't exist!!")
            }
        }
    }

    public boolean areTestsSuccessful() {
        boolean pass = true;
        if(this.watchedLog!=null && !this.watchedLog.equals("")) {
            pass = false
        }
        return pass
    }

    public void updateTestStatus() {
        Map<String, Object> updates = new HashMap<>()
        updates.put("passed", areTestsSuccessful())
        Utils.addBuildNumberToUpdate(updates)
        this.sauceREST.updateJobInfo(this.sessionIdProvider.getSessionId(), updates)
    }

}

Now as you might have seen in the comments stated in the code above there is a bug(feature?) in Spock that doesn’t correctly surface up errors. So even though the test fails the report in SauceLabs shows it as passed. Is this too big of an issue? Not really as Geb/Spock comes with its own reporting tools. And there are some solutions out there to create way better reports than what SauceLabs can provide.

Now how does all this tie in? Well below is a dummy test for the fun of it:

package com.ag.functionaltest.accelerator.specs

import geb.driver.CachingDriverFactory
import spock.lang.Issue
class SauceGebSpec extends BasePageGebSpec {
    def setup() { //setup goes here }
    def cleanupSpec() { CachingDriverFactory.clearCache() } 

    @Issue("http://jira.ag.com/AGT-101")
    def "Test that fails"() {
        when: "When I go to google"
        go "http://google.com"
        then: "Then test fails"
        assert false
     }
   
    @Issue(["http://jira.ag.com/AGT-101", "http://jira.ag.com/AGT-102"])
    def "Test that passes"() {
        when: "When I go to google"
        go "http://google.com"
        then: "Then test passes"
        assert true
     }
}

This was an exciting find. Looking through multiple documentations, issues, code examples provided by SauceLabs. Couldn’t find anything that worked with exactly with this stack. So I hope this post will help feature developers that want to go with this awesome stack.

As always have fun!