When is a TPS not a TPS? A guide to effectively measuring system performance

By Bethan Cowper | 29 April 2016

Payments system performance is a delicate subject as of late. System failures and downtime consistently make both industry and international headlines and paint the financial institution in question as inept at worst and clumsy at best. With more pressure than ever for well-established brick and mortar financial institutions to both adopt and adapt to today’s modern and often innovative requirements, system replacement and modernisation are the two most discussed topics du jour.

Evaluating system performance isn’t clear cut; FIs need to look at the bigger picture and read the small print when evaluating technology critical to the future success of their business. The most quotable performance metric is the Transaction Per Second (TPS) which is often used to represent system speed and capacity. However, without a clear understanding of the test environment, basing any decisions on the TPS is nonsensical from a business perspective. For example, the interpretation of a transaction can vary from vendor to vendor.

It is universally agreed that transactions are essentially collections of actions. For example, transferring funds between two bank accounts would constitute one transaction to the customer, however, behind the scenes, the source account is debited, the target account is credited, this action is recorded, checks are made on the source account funds availability, etc. The customer transaction is also known as a business transaction, whilst the actions that go on behind the scenes are database transactions. When looking at TPS, comparing a vendor quoting business transactions per second with another vendor quoting database transactions is akin to comparing a sugar cube to sugar granules and will not result in an accurate representation of the two systems against each other.

There are three key areas FIs should evaluate when considering system performance:

  1. Metrics
  2. Parameters
  3. Costs

Vendors should be happy to disclose the full results of any benchmark testing they have carried out using their software. Metrics should include statistics such as average authorisation time, percentage stability through peak load, time peak load was sustained and the percentage capacity of the server. Many vendors abstain from sharing details around how long peak load was sustained during stress testing. With intensive transaction periods becoming more common, there is an increased requirement for an FI’s payment infrastructure to remain stable at higher capacities for longer. Similarly figures that are presented based on the assumptive multiplication method, with terminology such as ‘the equivalent of’ or ‘this equates to’ can be misleading. Vendors multiplying TPS results to demonstrate the system’s capabilities over an hour aren’t realistically representing performance.

Performance testing is essentially used to determine how a system performs in terms of responsiveness and stability under a particular workload, whilst investigating, validating and verifying other attributes of the system such as scalability and reliability. It is therefore clear that measuring and promoting system performance using only TPS as an indicator is essentially redundant in understanding the mechanisms involved in generating these figures.

The parametres for benchmarking system performance aren’t uniform; the test environment is made up of multiple variables and it is essential to understand the components of this environment in order to understand what is being evaluated. Ideally the parametres of any system testing should make up a simulation that is as close to reality as possible; to provide performance indicators that would be achievable in real-time. However, this is not always the case; often test scenarios are simplistic and subsequently unrealistic.

It is not always possible to stage a test environment that accurately represents the variables that occur in real-life. However, test results that demonstrate different volumes of different types of data (for example issuing and acquiring data) can replicate how the system behaves across different situations. Parametres that should be clearly outlined in the test scenario include the hardware used, the volume of transaction data used, types of data sets (for example, overdue/non-overdue accounts, active/blocked cards, etc.), the number of customers, cards and accounts that the test processing system database holds respectively, the number of terminals, types of terminals, protocols used, the day (for example, statement day, end-of-da,y etc.), amongst others.

Ultimately, the test environment’s primary objective should be to help the vendor better understand the limits of their system. By stress testing the system, the vendor can uncover and solve problems before go-live. The test environment should help create better technology rather than provide a ‘version’ of performance that is favourable to the vendor.

The most effective measure of system performance for an FI is the Cost Per Transaction or CPT. This takes into account the hardware, software and operational costs; essentially the total system cost including maintenance normalised by the performance metric. Although TPS is an important measure of system performance that should not be overlooked, it is the CPT that represents the system efficiency in direct relation to outgoing cost.

Considerations when assessing performance versus cost:

  • Establishing the hidden costs: how many servers are running? How much do they cost? How many system administrators are required for maintenance?
  • Meeting current and future requirements: is the system scalable? How much will you need to pay for additional servers and support staff as your company grows? Are costs incremental over time?
  • Keeping cost relative: what systems do you need to run your day-to-day business? Will the value of a system upgrade offset the hardware/software/maintenance costs? Does your business need to run on high-end large scale systems or would something smaller be more efficient?

There is a lack of information, guidelines and advice to help FIs understand what supplier performance metrics really mean to their organisation. As a result, the majority of FIs simply don’t evaluate in terms of performance vs. cost and have a view of performance requirements that are neither realistic nor holistic. They risk paying excess for a payment system that does not meet their needs right now and spiralling incremental costs to meet performance requirements in the future.

By Bethan Cowper, Head of Global Marketing at Compass Plus.