One of our projects has a really large test suite that isn't optimised to run with Laravel's parallel testing. It was starting to take more than 15 minutes to have the whole test suite run inside Github Actions, which is when I searched for a better way to do this.
This post by Ruby Yagi explains how this is done for a Ruby project and I've copied some of the information to this post, we'll take a look at how it's done for a Laravel/PHP one with PHPUnit.
GitHub Actions matrices#
GitHub Actions lets you define a strategy matrix which launches all the combinations of that matrix into parallel processes. We can use this process to run our tests in arbitrary parallel chunks.
Let's set up our Github Action workflow to generate these processes:
name: PHPUnit
on:
push:
jobs:
phpunit:
name: PHPUnit
runs-on: 'ubuntu-latest'
strategy:
fail-fast: false
matrix:
# Set N number of parallel jobs you want to run tests on.
# Use higher number if you have slow tests to split them on more parallel jobs.
# Remember to update ci_node_index below to 0..N-1
ci_node_total: [ 4 ]
# set N-1 indexes for parallel jobs
# When you run 2 parallel jobs then first job will have index 0, the second job will have index 1 etc
ci_node_index: [ 0, 1, 2, 3 ]
steps:
- uses: actions/checkout@v2
- name: Setup PHP
uses: shivammathur/setup-php@v2
with:
php-version: '8.1'
extensions: dom, curl, libxml, mbstring, zip, pcntl, pdo, sqlite, pdo_sqlite, bcmath, soap, intl, gd, exif, iconv, imagick, redis
tools: composer:v2
coverage: none
- name: Run composer install
run: composer install -n --prefer-dist
- name: Run PHPUnit
run: ./bin/ci
env:
# Specifies how many jobs you would like to run in parallel,
# used for partitioning
CI_NODE_TOTAL: ${{ matrix.ci_node_total }}
# Use the index from matrix as an environment variable
CI_NODE_INDEX: ${{ matrix.ci_node_index }}
DB_PASSWORD: root
QUEUE_CONNECTION: redis
The CI_NODE_TOTAL means the total number of parallel instances you want to spin up during the process. We are using 4 instances here so the value is [4]. If you would like to use more, or less instances, say 2 instances, then you can change the value to [2] :
ci_node_total: [2]
The CI_NODE_INDEX means the index of the parallel instances you spin up during the CI process, this should match the CI_NODE_TOTAL you have defined earlier.
For example, if you have 2 total nodes, your CI_NODE_INDEX should be [0, 1]. If you have 4 total nodes, your it should be [0, 1, 2, 3]. This is useful for when we write the script to split the tests later.
The fail-fast: false means that we want to continue running the test on other instances even if there is a failing test on one of the instances. The default value for fail-fast is true if we didn’t set it, which will stop all instances if there is even one failing test on one instance. This allows all test instances to run completely showing us all the possible failed tests.
Where normally you'd have something like run: ./vendor/bin/phpunit in your action to run the tests, we've replaced this with run: ./bin/ci, which will be our own script to split the tests and tell PHPUnit which tests to run.
Creating a script to split the tests#
First, create a file named ci (without any file extension), and place it in the “bin” folder inside your Laravel app and make sure it's executable:
touch bin/ci
chmod +x bin/ci
Next, we'll create the script:
#!/usr/bin/env php
<?php
/*
* This script assumes you're in a Laravel project that has access
* to the Str, Collection and Symfony's Process class.
*/
require_once 'vendor/autoload.php';
/**
* Lists PHPunit tests in the following format:
* - Tests\Support\UuidTest::it_can_create_a_uuid_from_a_string
* - Tests\Support\UuidTest::it_can_not_create_a_uuid_from_null
* - ...
*/
$process = new \Symfony\Component\Process\Process([__DIR__ . '/vendor/bin/phpunit', '--list-tests']);
$process->mustRun();
$tests = \Illuminate\Support\Str::of($process->getOutput())
->explode("\n") // Break the output from new lines into an array
->filter(fn (string $test) => str_contains($test, ' - ')) // Only lines with " - "
->map(fn (string $test) => addslashes(
\Illuminate\Support\Str::of($test)
->replace('- ', '') // Strip the "- "
->trim()
->explode('::') // Only the class, not the method
->get(0)
))
->filter(fn (string $test) => !empty($test)) // Make sure there are no empty lines
->unique() // We only need unique classes
->split((int) getenv('CI_NODE_TOTAL')) // Split it into equally sized chunks
->get((int) getenv('CI_NODE_INDEX')); // Get the index we need for this instance
/**
* Run phpunit with a filter:
* phpunit --filter 'TestClass|AnotherTestClass|...'
*/
$process = new \Symfony\Component\Process\Process(['./vendor/bin/phpunit', '--filter', $tests->join('|')], timeout: null);
$process->start();
// Make sure we have live data output
foreach ($process as $type => $data) {
echo $data;
}
$process->wait();
// Exit using PHPUnit's exit code to have the action pass/fail
exit($process->getExitCode());
Running the Github Action together with this ci script should result in parallel actions being ran with each their own set of tests.
Final result of this: our test suite that took 15 minutes now takes only 6 minutes!