So.....Gradle

My previous post mentioned that we are currently using Maven as our build system but looking at moving across to Buck. Well, there has been a slight change of plan and we've moved to using Gradle. The main reason we opted for Gradle over Buck is its integration with Intellij, which was some what of a pain point when I was working at LMAX.

Migrating from Gradle to Maven was super easy, and has allowed us to simplify the various tests we had defined; Maven becomes very complicated if you try and do anything beyond their defined build tasks, which was very annoying for us as we have multiple test definitions being unit tests and integration tests (dao, acceptance, benchmark, perf, compatibility.....). This had resulted in some serious hackery (note: completely my fault).

As an example, this is how we ran DAO tests in the Maven world:

	local testDirectory=test-classes
	local testDirectoryLength=$((${#testDirectory} + 1))
	local mavenRepository=$(eval echo ~$USER)/.m2/repository
	#Find test classes, and convert them their package location (rather than file system)
	local testsClasses=$(find . -path *target/*DaoTest.class | grep -v "/DaoTest")
	local tests=""
	for test in ${testsClasses}; do
	tests=${tests}" "$(echo ${test:$(($(echo ${test} | grep -b -o ${testDirectory} | cut -d: -f1) + ${testDirectoryLength}))} | sed -e 's/\//\./g' | sed -e 's/.class//g')
	done
	#Build the class path....
	local jars=$(JARS=($(find deploy -name *.jar)); IFS=:; echo "${JARS[*]}")
	local clazzs=$(CLAZZ=($(find . -name ${testDirectory})); IFS=:; echo "${CLAZZ[*]}")
	local classpath=${mavenRepository}/junit/junit/4.12/junit-4.12.jar:${jars}:${clazzs}
	${OPT_DIR}/jdk/bin/java -cp ${classpath} org.junit.runner.JUnitCore ${tests}

view raw maven.sh hosted with ❤ by GitHub

And this is how we run them with Gradle:

Wrapper script:

${ROOT_DIR}/gradlew daoTest

view raw wrapper.sh hosted with ❤ by GitHub

build.gradle:

	task daoTest(type: Test) {
	include "**/*DaoTest.class"
	outputs.upToDateWhen { return false }
	}

view raw build.gradle hosted with ❤ by GitHub

Nice and simple!

Keep it Simple - Using a bootstrap script to wrap your build system

Do you know what's annoying when you start at a new company (or do a clean checkout of some code)? Getting your development environment set up; especially when it comes to making sure you have the right software in place to be able to build said code!

I'm sure there are lots of ways to solve these problems, but a simple solution we use at TransFICC is to use a bootstrap script that wraps our build system. This is called everytime you want to run something via the build system (in our case Maven [we haven't buck'd up]), and ensures all build dependencies are present. I should note that we only use this for build dependencies, not run time dependencies.

bootstrap.sh

	#!/usr/bin/env bash
	set -euo pipefail
	JDK="jdk1.8.0_74"
	EXPECTED_JDK_EXPLODED_SHA1="25d468495c8d44f71233cf240560b9d2c681d5e4"
	NODE="node-v4.3.2-linux-x64"
	EXPECTED_NODE_EXPLODED_SHA1="f1ecc0b12739e5ab1b4d03006f7f8b406fcaba07"
	BUCK="buck-344c05b"
	EXPECTED_BUCK_EXPLODED_SHA1="9bd0b11e2225d74f38d57c22dc404142e5d51ed0"
	MAVEN="apache-maven-3.3.9"
	EXPECTED_MAVEN_EXPLODED_SHA1="5b4c117854921b527ab6190615f9435da730ba05"
	#Runtime Dependencies
	PGSQL="pgsql-9.2.15"
	EXPECTED_PGSQL_EXPLODED_SHA1="8fad03ec139bc05cd7d3b7976e93f32ea5bc93d8"
	OPT_DIR=${ROOT_DIR}/opt
	function fetch_artifact() {
	local name=$1
	echo "Fetching dependency ${name} ..."
	local namedVersion=$2
	curl -J --progress-bar -o ${OPT_DIR}/${version}.tar.gz http://thearts.internal.transficc.com/artifacts/${name}/${namedVersion}.tar.gz
	}
	function ensure_dependency_extracted_and_linked() {
	local short_name=$1
	local name_version=$2
	local expectedSha1Hash=$3
	local fully_qualified_extracted_dir=${OPT_DIR}/${name_version}
	local fully_qualified_linked_dir=${OPT_DIR}/${short_name}
	local package_hash_file="$fully_qualified_extracted_dir.sha1"
	local actual_sha1_hash=""
	if [ -e "$package_hash_file" ]; then
	actual_sha1_hash=$(cat ${package_hash_file})
	fi
	if [ "$actual_sha1_hash" != "$expectedSha1Hash" ]; then
	echo "Hashes don't match"
	echo "actual: $actual_sha1_hash expected: $expectedSha1Hash"
	rm -rf ${fully_qualified_linked_dir}
	rm -rf ${fully_qualified_extracted_dir}
	fetch_artifact ${short_name} ${name_version}
	tar xzf "${OPT_DIR}/${name_version}.tar.gz" -C ${OPT_DIR}/
	ln -s ${fully_qualified_extracted_dir} ${fully_qualified_linked_dir}
	local sha1hash=$(sha1sum ${OPT_DIR}/${name_version}.tar.gz | awk '{ print $1 }')
	echo ${sha1hash} > "$fully_qualified_extracted_dir.sha1"
	rm "${OPT_DIR}/${name_version}.tar.gz"
	fi
	}
	function ensure_directory_exists() {
	mkdir -p ${OPT_DIR}
	}
	function bootstrap() {
	echo "Arguments: $@"
	ensure_directory_exists
	ensure_dependency_extracted_and_linked "jdk" ${JDK} ${EXPECTED_JDK_EXPLODED_SHA1}
	ensure_dependency_extracted_and_linked "node" ${NODE} ${EXPECTED_NODE_EXPLODED_SHA1}
	ensure_dependency_extracted_and_linked "buck" ${BUCK} ${EXPECTED_BUCK_EXPLODED_SHA1}
	ensure_dependency_extracted_and_linked "maven" ${MAVEN} ${EXPECTED_MAVEN_EXPLODED_SHA1}
	}
	boostrap

view raw bootstrap.sh hosted with ❤ by GitHub

Our build dependencies are Java, Node (well, npm), Buck, and Maven. This script assumes the dependencies live at http://thearts.internal.transficc.com/artifacts/ (e.g. the jdk is located at http://thearts.internal.transficc.com/artifacts/jdk/jdk1.8.0_74.tar.gz and the directory contained inside the tar.gz is jdk1.8.0_74). The checksums check for a change of dependencies rather than ensuring they are downloaded correctly.

This script could be used as follows.

transficc.sh

	#!/usr/bin/env bash
	set -euo pipefail
	export ROOT_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
	OPT_DIR=${ROOT_DIR}/opt
	./bootstrap.sh
	export JAVA_HOME=${OPT_DIR}/jdk
	mvn $@

view raw transficc.sh hosted with ❤ by GitHub

./transficc.sh compile