*For Finance to serve society, it needs to evolve beyond wizardry, guru’s guesswork and flawed toy models. Just as modern physics cannot advance without facilities like the Large Hadron Collider (LHC), a useful Finance will require machinery up to the task.*

**IN-SAMPLE DREAMS, OUT-OF-SAMPLE NIGHTMARES**

In a recent paper, I have argued that some of the most popular optimization techniques used in Finance are in fact detrimental. Take mean-variance optimization (MVO), the most commonly used portfolio construction technique, with its multiple upgrades and variations throughout the past 60 years. A great majority of academic papers apply MVO when the authors are faced with the dilemma of building a diversified portfolio. One would expect that such venerable technique would be among the best performing portfolio construction methods, right? Think again.

A number of studies have demonstrated that MVO portfolios underperform the so called “naïve portfolio”, that is the portfolio that splits assets equally among holdings (see here for example). And yet MVO is taught in every business school as one of the key results in Finance. Shouldn’t students be warned that MVO is detrimental, relative to a naïve allocation? How can a Nobel prize-winning theory lose to the most rudimentary scheme?

This is not a unique example. There are plenty of revered financial techniques that fail to perform as advertised. Cointegration models are known to lack robustness, in the sense that small changes on a few observations will lead to entirely different forecasts. This is particularly problematic in a discipline like Finance, where the signal-to-noise ratio is low and measurements are far from precise. Still, unstable econometric methods are routinely used by economists to forecast macro variables and by the Federal Reserve to inform their life-changing decisions.

In fairness, these methods were designed and vetted for academic consumption only. They are toy models, to be used for in-sample philosophical disquisitions, not in out-of-sample industrial applications (see here for the difference). Their authors never managed investors’ money, risked their own assets, or foresaw their widespread use outside academia. Real-world applications require a degree of complexity and robustness that simple models cannot satisfy.

**OLD MATH AND TFLOPS-PAUCITY**

Financial markets are complex networks of decision-making agents, interacting with each other. MVO is a very simple model, where every asset is a candidate to be replaced by another. A more sophisticated method would recognize the fact that some assets share features, such as their country of origin, sector, industry, default probability, liquidity, etc. In other words, like most applications of Geometry and Linear Algebra, MVO ignores hierarchical relationships. And because the method lacks structure, it lacks robustness. Conversely, graph theory embeds hierarchical information with minimal assumptions. But their development requires modern mathematical techniques absent from most economics/business curricula, hence graph-theory applications to Econometrics remain rare.

In many instances, simplicity is not a choice. Many researchers opt for dumb-down models because they lack the computational power required by more realistic ones. Should researchers have had access to more powerful computing machines, they would have proposed more realistic methods and theories.

**REFORMULATING PROBLEMS IN REALISTIC TERMS**

Portfolio construction techniques allocate assets in such a way that risk is minimized for a given return target. MVO can solve this problem as long as some *unrealistic* assumptions are met: First, returns must be IID Normal. Second, the covariance matrix must be invertible. But inverting a *NxN* covariance matrix with a minimal degree of success requires *N(N+1)/2* independent observations. For a small portfolio, of say size 50 instruments, that would involve at least 1,275 independent daily observations, or over 5 years of data. As every practitioner knows, correlations do not remain stable over such horizon with any meaningful degree of confidence. The implication is, MVO gives the precise answer to the wrong question.

Alternatively, we could study the hierarchical relations between the 50 instruments and build a tree-graph with 49 edges. This machine learning approach, called HRP, does not require covariance inversion, or a large number of independent observations. Results are robust to noisy observations and HRP delivers a 31% increase in out-of-sample Sharpe ratio relative to MVO. HRP is just one example of how modern mathematical approaches can help reformulate a problem in realistic terms. Interestingly, HRP does not require more data than MVO, so the performance improvement is all due to processing the available data better.

**HOW QUANTUM COMPUTING MAY SAVE FINANCE**

There are numerous instances in which machine learning methods deliver better results than classical calculus or linear algebra applications. But machine learning often deals with NP-complete or NP-hard problems, which demand overwhelming computational power. One possibility is to use proper heuristics (as HRP’s clustering algorithm does) to make the problem amenable to digital computers (DC). Carefully applying heuristics to solve a NP-complete problem is better than dumbing-down the problem to the point that a closed-form solution exists. A better way is to replace heuristics altogether, and explore a large portion of the feasibility region.

*[Image: A non-invertible returns covariance matrix of thousands of securities becomes invertible after running a ML clustering algorithm]*

This is the promise of financial quantum computing (QC). One day, in the near future, we will not need to dumb-down models, or rely on heuristics. We will develop models cognizant of reality’s complexity, and solve them in their NP-complete grandeur. Think about it: If HRP can improve your out-of-sample Sharpe ratio by 31% over MVO’s, what will the improvement be once you replace the DC+Heuristics tandem with QC+Completeness? Perhaps 50%? That means boosting your Sharpe ratio from 1.50 to 2.25, quite worth the management fee. Besides, QC can be used to save transaction costs and reduce the prevalence of high-frequency trading.

It is possible that one day Finance may offer real value to investors, beyond sales pitches, TV-show pundits and gurus’ guesswork. When cutting-edge technologies will be combined with realistic mathematical models to deliver strong, measurable benefits to savers and retirees. When financial firms will operate as research laboratories, and the alchemy of finance will be replaced with the chemistry of asset allocation. It will take a lot of qubits… but society will be better served.

Mark Dilweg, IBM Risk AnalyticsMay 28, 2016Excellent Marcos!

QC’s ability to change the financial world will be exciting to watch. Capturing reality’s complexness driven by infinite levels of scenarios and outcomes will have tremendous benefits to limiting current and even “new unseens” financial risks. Reduction of signal to noise ratios alone will provide advantages, but financial homogenisation also carries breakaway disruptive risks.

There in the core of the “unseen -unpredicted” risk lays the “black swan” which may actually be a manifestation of QC usage itself. Shear speed and cognitive magnitude will create a new natural laws and selection processes. With this power comes the emergence of outlier events. As cognitive agents learn and develop creative “thought” using the power of QC, some may enviably travel down a insane course of phantom realities. The Master-Slave flip over effect then becomes dangerous mathematical ground. Now add hyper-performance algo trading drones in the mix and cross your fingers.

Our ability as humans to remain on the Master side of that equation is the real test of our harnessing beneficial use of QC.

GiliMay 29, 2016That was an enjoyable read. I’ve often wondered what the bottleneck is right now, for financial practitioners, in improving their practices – is the problem that they don’t have enough computing power and knowledge at their hands, or is it rather that they are clinging to old practices by inertia and an inherent “late adopter” conservatism? With the cheap and widespread availability of cloud computing with thousands of cores / GPU’s, I’m definitely leaning towards the latter. The practical meaning is hopefully obvious: if most financial practitioners are clinging to their old ideas and practices, and the old ideas and practices have been shown to fail miserably and often, the ones who embrace the new ones have much to gain.

her latest blogJanuary 14, 2017I like this post, enjoyed this one thanks for putting up. “It is well to give when asked but it is better to give unasked, through understanding.” by Kahlil Gibran.

eebest8 fiverrMarch 2, 2017“Thanks for every other fantastic article. The place else could anyone get that kind of info in such an ideal means of writing? I have a presentation next week, and I am at the search for such info.”