and the Future of Quantum Machine Learning
#QuantumComputer #QuantumComputing #ClassicalComputer #ArtificialIntelligence #AI #MachineLearning #ML #DeepLearning #DL #QuantumMachineLearning #IBMQ #IBMQISkit #IBMPOWER #IBMAC922 #IBMIC922 #IBMWMLA #IBMPowerAIVision #H2ODriverlessAI #IBMCloud #IBMWatsonStudio #IBMWatsonNeuralNetworkModeler #IBMCloudPakForData
For the introduction on Quantum Computing (local language: Indonesian), please refer to the previous article, posted by Andi Sama on June 27, 2019 “Meneropong Masa Depan: Quantum Computing”.
Quantum Supremacy, is a condition in which a Quantum Computer can solve the complex problem(s) that can not be practically done using a classical computer (even with the fastest supercomputer today, like IBM Summit). Most research activities are racing towards achieving quantum supremacy with universal Quantum Computer, i.e. the Quantum Computer with more than 50-qubits configuration to solve complex problems for example, as actively being researched in IBM Thomas J. Watson Research Center, Yorktown Heights, USA.
Quantum advantage refers to the moment when a quantum computer can compute hundreds or thousands of times faster than a classical computer (Insight Quantum Technology, 2020).
IBM has been becoming one of the leaders in this cutting-edge research for a better future. Even IBM Q System One, as a product that can be installed on-premise, has been recently announced since early 2019, while we can access IBM Quantum Experience through IBM Cloud since early 2016. Others that are doing research or committed to research this area are Google, Intel & Microsoft. More including startups includes Rigetti Computing, IonQ, and Quantum Circuits (Jeremy Liebman, March-April 2018).
On the other hand, the field of Artificial Intelligence (AI) especially in Machine Learning (ML) and Deep Learning (DL) has been improving quite fast in the last 8-years since the ImageNet competition in 2012, including the advancements in GPU-hardware (Graphics Processing Unit) and bigdata in the early 2010s.
Machine Learning and Deep Learning have been dealing with a lot of data points processing (the dataset) in high dimensions with millions of hyperparameters to optimize (during the training of dataset), to create models that represent kind of relationships with the expectation to achieve a high degree of accuracy by minimizing errors (usually computed through minimizing the differences between training dataset and validation dataset). Those datasets are represented in the form of large matrices, mostly contains floating-point numbers.
The trained-model then can be used to do prediction by giving it new data within the same domain (while the model can be updated to be better along the way as new data are coming).
There is a high expectation that at some point in time in the not-so-far future within the next 10-years or so, Quantum Computer with the right amount of qubits can significantly speed up some of the most complex processings in Machine Learning/Deep Learning.
Artificial Intelligence, Machine Learning & Deep Learning — Today
The field of AI is really exciting with lots of new things to explore and yet waiting to be invented. By following various directions of research alone, may not be enough to keep up with all those advancements. Things that we knew well in recent years, maybe outdated (or updated) soon along with new inventions.
For those experts who want to stay updated, practicing lifelong learning maybe one of the key success factors to keep ourselves updated. Lifelong learning may also fit for us that are really at the beginning or intermediate pathway to learn and understand some of the basic concepts and its practical implementations.
Nowadays — when we mention the term “AI”, it is synonymously associated mostly with Deep Learning (as part of Machine Learning, which is also as part of Artificial Intelligence) within the Computer Science discipline.
Machine learning offers the ability to extract certain knowledge and patterns from a series of observations. It’s all done through mathematical optimization by using models (pattern recognition or exploration of many possibilities). In supervised learning, for example, minimizing the errors (as feedbacks) is very important to get the best learning result possible.
Deep Learning involves learning directly from data through multiple neural network layers (depending on the level of granularity in learning, the number of layers may be just a few or going to hundreds of layers or more).
Supervised Learning, Unsupervised Learning, and Reinforcement Learning are three types of Machine Learning (which is in recent years, extended into Deep Learning, an approach in a neural network to process data through multiple neural network layers).
Illustration-1 shows the potential applications in each type of Machine Learning. The illustration also shows briefly on the evolution of Deep Learning, all the way back to the 1950s since the term “Artificial Intelligence” was first introduced.
Rapid advancements in AI within the last decade were made possible when Professor Geoffrey Hinton (now in Google) and his team came up with a neural network architecture called: Alexnet, implementing Deep Learning to process ImageNet dataset during ImageNet competition in 2012. The team achieved a significant improvement in doing Image Classifications based on around 1.2 million ImageNet dataset. The success was then quickly fueling the research community and triggering many more advancements in the subsequent years. And, it is still until now.
Thanks to the advancements in Graphics Processing Unit (GPU) hardware as well as the availability of huge datasets (a mix of structured, semi-structured, and unstructured data) and the processing capability of Bigdata (Hadoop, Spark). Those factors contributed significantly to massively parallel processing power and data readiness that is speeding up the modeling (a.k.a. training a dataset, in creating model) in Machine Learning & Deep Learning.
Illustration-2a and illustration-2b show a few practical implementations (mostly in computer vision) of Deep Learning from face recognition associated with the name, face generation, object detection, self-driving car, to AI-powered robotic arm that is implementing a trained-AI model to IoT device and natural language processing in a human-machine unmoderated debate.
Typical implementations in Machine Learning are doing classification and clustering.
Classification is a type of supervised learning. By given a bunch of input and its corresponding output, the algorithm will eventually learn to derive the pattern from the data, thus be able to make a prediction. Support Vector Machine (SVM) is an example of a classification algorithm. Clustering is a type of unsupervised learning. Given a dataset, the algorithm is not given any information about the relationship in a dataset like in supervised learning. The algorithm tries to group the data based on a target number of clusters that we specify in the beginning, for instance.
For a very high dimensions dataset, doing classification and clustering will be a time-consuming process, that can take forever. Let’s see an example of unsupervised learning.
Recall that in supervised learning, we do have labeled data (observations as input and responses as output, and its relationship as our dataset). When we do not have or do not know whether the dataset has a pair of observations/responses, one approach is to explore the dataset through unsupervised learning. We will take a look at one of the algorithms for Clustering, called k-means (in which k is the number of target clusters). The purpose of the algorithm is to partition the data into k number of mutually exclusive clusters.
The technique assigns each observation to a cluster by minimizing the distance from the data point to the mean or median location of its assigned cluster, respectively. We need to specify a distance metric for the algorithm: the common ones are L1 (Manhattan distance) and L2 (Euclidean distance).
By using Matlab version R2019a update 2 trial version (Matlab, 2019), first, we generate 1000-random data (illustration-3a) and applies k-means algorithm to do clustering with k=2, with distance metric sets to “cityblock” or known as Manhattan distance (the default is Euclidean distance metric).
The complete Matlab code is shown in illustration-4 (result is shown in Illustration-3b as two clustered data). Matlab documentation says the k-means algorithm performs k-means clustering to “partition the observations of the n-by-p data matrix X into k clusters and returns an n-by-1 vector (idx) containing cluster indices of each observation. Rows of X correspond to points and columns correspond to variables.”
Initially, quantum computing initiative has been based on decades of deep-advanced-research to build a Universal Quantum Computer. Along the way with continuous advancements & achievements, It is seen that it is ready to be made commercially available. The ambitious target has been to achieve 50-qubits (the stable and less noisy ones) at minimum, at it is believed that achieving 50-qubits is the first step to have a Universal Quantum Computer that is exceeding the fastest current classical available Supercomputer.
Key Industry Players in Quantum Computing
IBM-Q’s initiative to build a Universal Quantum Computer has been publicly announced since early 2019, however, we can access and have practical experience on the real IBM Quantum Computer a long way back since early 2016 (starting with 5-qubits, and has been available up to 14-qubits as of 2019), through IBM Cloud. More qubits as a Quantum Computer simulator is available from IBM at up to 32-qubits.
Illustration-5 shows the current IBM Quantum Experience welcome screen on IBM Cloud, as of November 26, 2019. Illustration-6 also shows the current status of two IBM real Quantum Computers in operation, each configured with 5-qubits and 14-qubits.
A superposition is a weighted sum or difference of two or more states; for example, the state of the air when two or more musical tones are sounding at once. Ordinary, or “classical,” superpositions commonly occur in everyday phenomena involving waves.
- IBM Quantum Experience, January 2019 -
Entanglement is a property of most quantum superpositions and does not occur in classical superpositions. In an entangled state, the whole system is in a definite state, even though the parts are not. Observing one of two entangled particles makes it behave randomly but tells the observer exactly how the other particle would act if a similar observation were made on it.
- IBM Quantum Experience, January 2019 -
Quantum Volume (QV) is a hardware-agnostic metric that IBM defined to measure the performance of a real quantum computer. QV takes into account the number of qubits, connectivity, and gate and measurement errors.
- IBM Research blog, January 2020 -
IBM said that it has achieved 53-qubits as of September 2019 (Edwin Pednault, John Gunnels, Dmitri Maslov, and Jay Gambetta, 2019) and Google was said to have achieved a similar one. More quantum initiatives have been around since then, such as (other includes Quantum Circuits and IonQ):
- Jan 2020: IBM announces to have achieved 32 Quantum Volume (IBM, 2020)
- Dec 2019: AWS Announces New Quantum Computing Service (Amazon Braket)
- Oct 2019: Google claims it has achieved ‘quantum supremacy’, challenged by IBM
- May 2019: Microsoft to open-source quantum development kit (Q# within Microsoft Visual Studio)
- February 2019: Microsoft announced Microsoft Quantum Network
- October 2018: D-Wave launched Leap, cloud-based quantum application environment
- September 2018: Rigetti introduced Quantum Cloud Services
- March 2018: Alibaba & CAS launched Superconducting Quantum Computing Cloud
- January 2018: Intel announced Tangle Lake, a quantum processor
- March 2017: IBM to open source Qiskit, open-source framework for quantum computing
- May 2016: IBM opens-up public access to 5-qubits quantum computer, through IBM Cloud
Since March 2017, IBM had started to provide QISkit, the Phyton-based Software Development Kit (SDK) for IBM Quantum Computer in the form of Application Programming Interface (API). It is available in Github.
QISKit is an Open Source Quantum Information Software Kit, designed to simplify the developer to work with OpenQASM (IBM Quantum Assembly Language) and IBM Q Quantum Processor Hardware. We can create a quantum program within Jupyter Notebook (an interactive modern application development environment) for example, compile & execute our program in the real or simulated Quantum Computer.
Quantum Superposition & Quantum Entanglement
The basic processing unit of a Quantum Computer is called Qubit (quantum bit). Qubit exists in either state ‘0’ or ‘1’ as in the bit (binary digit) in a Classical Computer. However, qubit’s state is more than just ‘0’ or ‘1’, it can also be in ‘0’ and ‘1’ at the same point in time, a condition called as Superposition.
Another one is called: Entanglement, which is a property of superposition and happens only in quantum mechanics, not in classical superposition. If we are observing two entangled particles, we will know exactly the state of another particle by observing only one entangled particle.
A Quick Look on selected Algorithms, and their Big Os
Let’s take a look at 3 algorithms that can significantly be improved (Seth Lloyd, 2016) by using a quantum computer, thus implementing quantum Machine Learning: 1. Principal Component Analysis (PCA), 2. Fast Fourier Transform (FFT) and 3. Finding Eigenvalues & Eigenvectors. We will see later their Big Os if processed in a classical computer, then see the Big Os if processed in Quantum Computer.
“Big O” Notation
Wikipedia, 2019b: “Big O notation is a mathematical notation that describes the limiting behavior of a function when the argument tends towards a particular value or infinity.” In this article, we see “Big O” as the indicator the complexity of an algorithm.
Principal Component Analysis (PCA)
PCA is an unsupervised Machine Learning algorithm for dimensionality reduction in Machine Learning.
A high dimensional dataset has a large number of features, in which some might not be relevant to be processed in the Machine Learning algorithm to generate a model. As an analogy like in Microsoft Excel spreadsheets, having a high dimensional dataset is like having a huge table with hundreds of columns with hundreds of thousands of rows as data. The columns in the Excel sheet are like features in Machine Learning.
By unsupervised, it means that we through the algorithm to the data and the algorithm will figure out to extract meaningful insight from the data. In this PCA, it means that it will extract “principal components” (significant features) that have higher variances, among other available features in the dataset.
By reducing “selecting” the features that are relevant to the Machine Learning algorithm, we can minimize the occurrence of overfitting. Overfitting reduces the capability for generalization. Dimensionality reduction also significantly reduces the needs for processing time, thus also the computing resources (Disk, Memory, CPU, and GPU) to process the dataset.
Fast Fourier Transform (FFT)
(Wikipedia, 2019a) “A fast Fourier transform (FFT) is an algorithm that computes the discrete Fourier transform (DFT) of a sequence, or its inverse (IDFT). Fourier analysis converts a signal from its original domain
(often time or space) to a representation in the frequency domain and vice versa. The DFT is obtained by decomposing a sequence of values into components of different frequencies”.
FFT can be an additional external subprocess in IBM PowerAI Vision (IBM PAIV) to process voice dataset for example. IBM PAIV (runs on IBM POWER AC922 hardware, Accelerated Computing, optimized for AI-training) is a visual-based AI data workflow for doing modeling in computer vision based on the dataset in the form of images and videos to find patterns in data to do prediction. Alternatively, there is H2O Driverless AI (H2O DAI), a kind-of simplified visual-based AI data workflow for Data Scientists and a more-flexible tool for experienced Data Scientists: IBM Watson Machine Learning Accelerator (WMLA), both can run on IBM POWER AC922.
Say if we have the voice data like for voice recognition, we can use FFT to convert the voice (frequency domain) to images (time or space domain), thus utilizing the same AI-data workflow with a different set of data type while it is not currently supported out-of-the-box within the IBM PAIV itself.
Note that we can code the AI workflow directly from scratch with IBM Watson Studio for example, or with a more flexible IBM visual tool on IBM Cloud: Watson Neural Network modeler.
Eigenvalues & Eigenvectors
In linear algebra, an eigenvector or characteristic vector of a linear transformation is a nonzero vector that changes at most by a scalar factor when that linear transformation is applied to it. The eigenvalue is the factor by which the eigenvector is scaled (Wikipedia, 2019c).
Face Recognition (224x224x3 pixels in one RGB-image alone which is usually implemented in image classification or object recognition would have incredibly high 150,528 dimensions) would be one of the applications of eigenvalues and eigenvectors by implementing what it is called as eigenfaces. Even if it is scaled down to 32x32x1 pixels (monochrome), it is still having very high 1024 dimensions.
Quantum Machine Learning
In its simplified definition, Quantum Machine Learning (Quantum ML) is the use of Quantum Computer to process Machine Learning (or Deep Learning) algorithm. It is expected that, by applying quantum-computing with enough quantum bits (qubits) to process a Machine Learning algorithm, the process will be exponentially faster than if it is processed in a conventional way using a classical computer.
Classical Computer (includes Super Computer)
In a classical computer with Machine Learning, we used to process program + data in CPU or GPU, by first loading it from external storage such as disk to Random Access Memory (RAM).
If using CPU without GPU, we are using the CPU’s RAM to transfer our program + data before processing them in CPU (e.g. in current typical high-end server-based systems, RAM may contain some Giga Bytes (GB) in size like 128GB, 512GB or 1024GB or more likely up to hundreds or thousands of TeraBytes (TB) in higher-end multi-servers multi-CPUs supercomputer systems).
With GPU, usually, CPU will transfer the program + data from CPU’s RAM to GPU’s RAM according to GPU’s RAM capacity, for the program + data to be processed in several of magnitudes faster as GPU implements high-degree of parallelism in which it has multiple processing units that can process the stream of data at once at one time, although with certain limitation on RAM capacity (e.g. typical single GPU RAM with 512 to 5120 processing elements has only 8GB, 16GB or 32GB RAM).
Even with GPU, the challenge comes also with the slow I/O performance when transferring data from CPU to GPU. This bottleneck is somewhat solved within the IBM POWER AC922 server with a high-speed NVlink I/O (Input/Output) interface implemented both between CPU-GPU, and GPU-GPU as well. Most servers in the market implement NVlink only for GPU-GPU communications. The NVlink between CPU-GPU gives AC922 a significant increase in I/O performance (about 5 times) over other similar servers available in the market.
Another challenge includes loading large dataset from disk (external storage) to the CPU’s RAM, this process takes a lot of time (very slow) for either CPU or GPU. The advancement of high-speed flash-based storage, for example, has contributed to significantly reduce the gap in this area.
A large highest-speed, number one world-class supercomputer since 2018 like IBM Summit for example (Top500.org, 2019), will still be seen as “slow” for a Quantum Computer sometime to be available in near future when a Quantum Computer has reached the certain number of stable qubits (has passed 50 qubits in 2019 and to achieve a few hundred or even thousands qubits in the next years to come).
IBM Summit has the computing capability of 200 PF (Petaflops), 9.216 IBM POWER9 CPU processor + 27.648 NVidia GPU (Graphics Processing Unit), RAM capacity >10 PB (Peta Bytes), storage capacity: 250 PB. An analogy (according to IBM, IBM Summit that is installed in Oak Ridge National Laboratory (Tennessee, Amerika Serikat), it has about one million more powerful than the most powerful laptop/notebook computers as of 2019). IBM Summit is a classical computer with a completely different architecture with a Quantum Computer like IBM Q.
Challenges in Applying Quantum Computation to Machine Learning Algorithms
The next near future to come will be really exciting indeed. There is a high expectation of having Quantum Computers with enough qubits that can solve complex problems a lot faster. Problems that can not be efficiently solved by a classical computer, including the most powerful supercomputer (if one can represent the classical problem in quantum-way, e.g. doing complex large scale optimization & simulation that would take unreasonable time <more than a lifespan of a typical human being>, like 100+ years for example).
Table-1 shows the comparison of Big O’s for each of three algorithms discussed before (PCA, FFT, and finding eigenvalues & eigenvectors), running those algorithms using classical computer vs quantum mechanically using quantum computing. We can see from graphics in illustration-7 that the improvements are exponentially significant for those 3 algorithms.
However, implementing a Quantum Computer with a large number of qubits is challenging. It requires a lot of physical qubits and a very challenging error correction handling as a quantum system will get noisier as we are adding more qubits.
In reply to a comment from Professor Joshua Bengio (Head of the Montreal Institute for Learning Algorithms (MILA): Peter Shor, a Professor of Applied Mathematics (MIT commented during MIT-IBM panel discussion (IBM Research, 2018) “Near term Quantum Computers are going to be very noisy, to have 50 logical qubits, for example, we may need to build real 1000+ physical qubits”. He continues, “rather than focusing the work in quantum error corrections, we may be able to use the noisy qubits to get a result”.
Joshua Bengio previously commented: “For things like continuous non-linear computation like our group is working right now with analog-based circuits, we focus on getting overall result statistically by tweaking the parameters during training. We do not focus on individual results.”
IBM Solutions for Data Science, Machine Learning & Quantum Machine Learning
There are several options with IBM Hardware & Software to work with Machine Learning algorithms, either for Modeling (develop model) or Inferencing (deploy/run model). Some of them are as follow (indicated by HW for Hardware, and SW for Software-based solutions):
- [HW] IBM Q, the IBM Quantum Computer accessible from IBM Cloud since early 2016. Since Jan 2019, IBM announced an on-premise Quantum Computer called [HW] IBM Q System One. [SW] IBM QISKit has been available recently since March 2017 with its Python interface as a means to access IBM Quantum Computer, while the assembly counterpart [SW] IBM QASM and [SW] IBM Quantum Composer (before QASM) had been available earlier since 2016.
- [HW] IBM POWER AC922 (Accelerated Computing) + IBM Watson Machine Learning Accelerator (WMLA, previously known as PowerAI on IBM POWER Systems + [SW] IBM Power AI Vision (PAIV) with custom model. This IBM PAIV is for Supervised Learning for either Classification, Object Detection (including Action Detection) with Vision (images, videos) as the supported dataset. POWER: Performance Optimization with Enhanced RISC (Reduced Instruction Set Computing).
WMLA (or WML CE for the Community Edition) is available as On-Premise solution for modeling & inference, and comes with very scalable configuration equipped with multiple NVidia high-end Tesla V100 GPUs. In the latest release in late 2019, Inferencing at Edge is available with NVidia Jetson TX2 and Jetson Nano. Previously, Xilink Alveo U200 Field Programmable Gate Array (FPGA) has been supported. WMLA & Power AI Vision is good for general users & non-experienced developers.
IBM IC922 is also available starting in Q1 2020, as the IBM offering for Inferencing Platform (runtime) for Machine Learning trained-model.
- IBM POWER AC922 + WMLA + [SW] H2O Driverless AI. This is also mostly for Supervised Learning for either Classification or Regression (text-based, images, videos) and more with extensive parameter settings and explainability. Good for advanced users & experienced developers or Data Scientists.
- [SW] IBM Watson OpenScale. The platform to choose when we want to do scalable model deployment (Inferencing), running a few or many models in parallel with capabilities to track performance in a production environment, model tuning as well as explainability (explain why the AI made certain decisions) — available in IBM Cloud.
- [SW] IBM Watson Data Platform (IBM WDP, available in Cloud or On-Premise, or On-Premise in combination with IBM WMLA or [SW] IBM Cloud Pak for Data. This is for those who would like to have deep customization (using IBM Watson Studio) and the greatest flexibility in implementing Machine Learning. The best fit for working in collaboration between experts of data analysts experienced Data Engineers and Data Scientists.
There are set of tools provided for Data pre-processing for preparing data in IBM WDP (such as Data Manipulation/Cleansing, Data Augmentations) before being processed by Machine Learning algorithms, as well as Data Catalog for managing data sets used to train the data using Machine Learning algorithms.
[SW] IBM Cloud Pak for Data (available in On-Premises as well as in Cloud) is a container-ready solution that enables users to put lots of data to work quickly and efficiently to use the data to generate meaningful insights that can help users avoid problems and reach the goals, while maintaining the data privately. The best fit for experienced Data Scientists/Analysts & experienced developers.
- [SW] IBM Watson Studio (part of IBM WDP, available in Cloud or On-Premise or On-Premise in combination with IBM WMLA or Cloud Pak for Data). This is for those who would like to have deep customization and the greatest flexibility in implementing Machine Learning. The best fit for expert or experienced developers as well as Data Scientists.
- [SW] IBM Neural Network Modeler (available in IBM Cloud). This is for those who would like to work with guided tools in implementing Machine Learning. Good for the advanced user or advanced developers as well as Data Scientists.
- Andi Sama, 2019a, “AI Model Inferencing, Practical deployment approaches & considerations”, SWG Insight, Edisi Q4 2019, page 3–9.
- Andi Sama, 2019b, “Quantum Superposition & Entanglement on IBMQ”.
- Andi Sama, 2019c, “Meneropong Masa Depan, IBM Q Quantum Computing”, SWG Insight, Edisi Q2 2019, page 12–26.
- Andi Sama et al., 2019a, “Think like a Data Scientist”.
- Andi Sama et.al, 2019b, “Guest Lecturing on AI: Challenges & Opportunity”, Lecture to FEBUI — University of Indonesia”.
- Andi Sama et al., 2017, “The Future of Machine Learning: The State of Advancements in Deep Learning”, SWG Insight, Edisi Q4 2017, page 6–17.
- Edwin Pednault, John Gunnels, Dmitri Maslov, and Jay Gambetta, 2019, “On Quantum Supremacy”, IBM, October 2019.
- Explaining Computer, 2019, “Quantum Computing 2019 Update”.
- IBM, 2020, “Quantum Takes Flight: Moving from Laboratory Demonstrations to Building Systems”, IBM Research blog, Jan 2020.
- IBM, 2019, “IBM Quantum Experience”.
- IBM Research, 2018, “What can quantum do for AI”, MIT-IBM Research, November 2018.
- Insight Quantum Technology, 2020, “Quantum Advantage & Quantum Supremacy Explained”.
- Jeremy Liebman, March-April 2018, “10 Breakthrough Technologies 2018: Serious Quantum Computers are finally here. What are we going to do with them?”, MIT Technology Review, February 2018.
- Karen Hao, 2019, “We analyzed 16,625 papers to figure out where AI is headed next”, MIT Technology Review, January 2019.
- Matlab, 2019, “Matlab Documentation”, R2019a, MathWorks.
- Pieter Abbeel, 2018, “Full Stack Deep Learning — Lecture 10: Research Directions”, Deep Learning Bootcamp, March 2019, Berkeley.
- Qiskit, “Open-source quantum computing framework”.
- Seth Lloyd, 2018, “The Future of Quantum Computing”, The Artificial Intelligence Channel, January 2018.
- Seth Lloyd, 2016, “Quantum Machine Learning”, Lecture in Keio University, June 2016.
- Top500.org, 2019, “TOP500 List — November 2019”, November 2019, Top500.org.
- Wikipedia, 2019a, “Fast Fourier Transform”.
- Wikipedia, 2019b, “Big O Notation”.
- Wikipedia, 2019c, “Eigenvalues and Eigenvectors”.