Simulating Realistic Conditions: QA for Business-critical MySQL Clustering

After Tungsten components are built, we test their functionality in the most realistic conditions possible; this means simulating the dynamic environments running on customer servers in Production. To achieve this, we need tens of servers ready to deploy Tungsten software, perform many different tests and in all the different use cases. Thankfully we are living in the future, and we can use cloud technologies to reserve the computing power and quantity of virtual servers to fit our needs. For this purpose, we leverage Amazon EC2 instances in AWS.

To find out more about how our QA framework ensures software is battle-tested for business-critical use, please visit, Intro to Business Critical Software QA.

Always With a Clean Shield

Every time we want to build and test (or just test) our software, we create a fresh new group of instances. This proves to us, we always have a clean environment for testing and all packages are up to date and with latest versions. After testing is done, and we are no longer interested in those running instances, we terminate them. This way we are not wasting money for constantly running servers and, more importantly, we don’t face environmental issues or inconsistencies.

Developers are not dependent on one shared group of servers! We create groups of servers on demand; each group is dedicated to one build/testing round. Also, one testing process could create 1 to 6 parallel child processes, each having its own group of servers and running tests from a shared pool of test suites.

When I say “we do this, or that” I mean - these processes are fully automated! Starting servers, dependencies preparation, build software and then test software in thousands of different ways and finally terminating servers and reports creation. As folk wisdom says: “Never spend 10 minutes doing something by hand, when you can spend 10 hours failing to automate it!”

Keep It Lightweight

In the past, we put a lot of libraries and packages that were not defined as prerequisites for Tungsten Clustering, just to make our testing process more comfortable for us. We found out it’s not best practice because our customers don’t have all packages we use for testing installed. That made our testing environment “not clean enough”. Now, we install only the packages and third party software that we define in our Documentation as prerequisites for Tungsten.

And what about libraries that make test development easier? We basically removed all dependencies from tests and testing tools and did everything on our own. We have our own testing frameworks written from scratch, which fully fits and is targeted to testing only our products. Anyway, there is no public testing framework, fitting and able to test software like Tungsten Clustering and Replication. For DevOps we use tools like Ansible, but for testing we prefer to have it all under our own tight control.

Too Many Linux Distributions

At Continuent, we make sure that our Tungsten software is able to run on many different operating systems and environments. For this reason, we test on multiple different Linux distributions. In the article Automated MySQL Server Preparation Using Ansible you can find a description of the Ansible tool and source codes that we use for installation and configuration of prerequisites on our servers, including on different supported Linux distributions.

As every Linux distribution has a slightly different file system, we look for performance and security ‘holes’ by testing in different Linuxes. During ‘multi-linux’ testing we cover:

  • Amazon Linux 2
  • CentOS 7
  • CentOS 8
  • RHEL 7
  • RHEL 8
  • Debian 9
  • Debian 10
  • Ubuntu 18.04 LTS
  • SUSE Linux Enterprise Server 15

A Little Big Playground

When Continuent discusses Composite Active/Passive and Active/Active topologies, one of the use cases is Geo-scale multi-region MySQL Clustering. Communication between servers located around the world brings new unexpected environment conditions like high latency and other performance issues. For this reason, we test our composite topologies across multiple regions.

The solution is pretty ‘easy;’ we reserve a few servers in multiple regions, then we set up networking between them and run the tests. For example, when we are testing cross-region replication and clustering in a Composite Active/Passive topology, we prepare 3 instances per each of following regions in AWS:

  • US East (N. Virginia)
  • Europe (Ireland)
  • Asia Pacific (Singapore)

MySQL Versions

Last but not least we test our software against all possible versions of MySQL. Tungsten Clustering version 5.4.0 (released in 2019) brought full support of MySQL 8, during 2020 we certified our software for Percona Server distributions, and Tungsten Clustering version 7.0.0 will bring full certification for distributions of MariaDB.

At the moment, it is possible to test on servers with the following distributions of MySQL server:

  • MySQL versions 5.6, 5.7 and 8.0
  • Percona Server versions 5.6, 5.7 and 8.0
  • MariaDB versions 10.0, 10.1, 10.2, 10.3, 10.4 and 10.5

Environments Matrix

As described above, we are able to set up pretty various and dynamic environments for testing. 9 Linux distributions, multi-region, 12 MySQL versions in total… oh, and we are also interested in testing different Java versions! We are able to set servers on Java versions from 8 to 13, but most important are long term support Java versions 8 and 11.

Now, if we want to test all possible combinations of Linux, MySQL, Java and include multi-region testing, it will get us a massive multidimensional matrix of possible environments. Just for information, theoretically, with all our test suites, it takes weeks to finish tests on all possible environments we can prepare. Also I calculated with 6 parallel processes which could be busy working on this task. This is why we defined some combinations, they are necessary to be tested and also ‘levels’ of testing - from the easy smoke testing to advanced testing of release candidates.

For more information visit, “Intro to Business Critical Software QA”:

To learn more about Tungsten Clustering, visit this link:

Feel free to reach out to our team any time:

About the Author

Patrik Michalák
Manager of Quality Assurance

Patrik has been with Continuent for a few years, having previously worked as a full-stack web and mobile application developer. He’s a technology enthusiast, and has been awarded at country-level for a photography processing project, and has been involved as an IoT architect in a scooter sharing project in his country in 2019. Patrik is skilled in Perl, Python, JavaScript (ES6), C, Java, SQL, PHP, including technical skills in Linux administration and automation tools.

Add new comment