Benchmarking

It is necessary to develop methods and tools to assess quantum computing technologies and their potential for scientific applications. To measure the performance of quantum computer benchmarks, on different levels of the system, have been suggested. All of these performance measurements have their pros and cons and are under discussion in the community.

Benchmarking on device and subsystem level (e.g. gate or circuit level protocols) is important to assess the properties and capacities of the different technologies. As we are currently in an era where these properties are difficult to reproduce and we are moving to pilot lines for bigger production, performing benchmarking at this level is important and might help with standardisation activities for the hardware technologies. Beyond gate or circuit level protocols application-centric benchmarks are necessary. However, at the current stage of technology and quantum algorithm development it is not clear the use of which algorithm will prove to be advantageous when applied to quantum and which algorithm should be developed further. Therefore, the characteristics of the most important benchmarks are not yet known and at the current stage it is important to promote a broad set of benchmarks, so that we do not remain constrained to a certain technology or algorithm too early.

Despite various initiatives, no general and widely adopted benchmark suite is currently available. A recent contribution is Atos Q-Score (Atos n.d.) (Martiel, Ayral and Allouche 2021), with a certain level of generality and agnosticism, and which also allows for the comparison of quantum versions to classical implementations of well-known algorithms in a quantitative way. This kind of effort with additional value to Europe, should be continued and broadened.

Such extensive benchmarking on the level of algorithms and applications is still an open problem and further research in this area must be strongly supported, in close links with the application communities. It has to be noted that benchmarking at the algorithm level tests the combination of the algorithm, software stack and the technology. An evaluation of algorithms and applications needs to be done on the basis of previously defined criteria, such as time to solution or accuracy of the results.

The progress in application developments and their experimentation must feed the definition of benchmarks, i.e. benchmarks will play an important role in the hardware-software co-design that is currently necessary. It would also be important to develop a widely acknowledged ranking system, mature and flexible enough to accommodate future technology evolutions, and European players can significantly contribute to this effort.