As Artificial Intelligence becomes a widespread and mature technology, the investment industry is increasingly turning its tools to face the complexity of financial markets. From security analysis to portfolio construction, there are many fields in which AI has proved capable of adding scale and efficiency to the investment process and be a valuable resource for asset and investment managers.
Yet, despite the many touchpoints with traditional quantitative models, it is the ability to improve the signal-to-noise ratio and focus on a holistic – rather than narrow – understanding of market dynamics that set AI models apart. As we are going to see, this brings a new level of understanding of the inner workings of financial markets.
Knowledge is Power
Over the past decade, the tools of machine learning and Artificial Intelligence (AI) have led to an impressive technological revolution that has redefined – and in some cases disrupted – a wide array of industries. Among their benefits, these tools have allowed us to process data more efficiently and dramatically improved our ability to understand, interpret and adapt to the complexity of the real world.
For these reasons, apart from some of its well-known applications in fields like space exploration or medical imaging, AI is being increasingly adopted by the investment industry. Indeed, as financial markets represent a traditionally complex and dynamic environment, the opportunity offered by AI is twofold: improve the quality of investment decision making and quickly adapt to the gradual evolution of financial markets.
However, rather than a totally new framework, AI seems to stem out of the decades-long quantitative approach to investing. An evolution, in terms of power and efficiency, that is hardly going unnoticed by investment professionals as it overcomes the well-known limitations of traditional models and better frames the underlying dynamics of the economic machine.
In this sense, what goes under the name of “traditional quantitative models” is an ample category of statistical tools that aim at capturing mathematically the relationships between a set of financial variables, given very restrictive assumptions. In this way, although they might be a good first-approximation of the true relationships, they miss out on what is truly valuable for asset and investment managers: the capacity to quickly adapt when things change.
However, what appears to be the common pain point of traditional models is that they struggle to understand the structure of time-varying and complex relationships. Indeed, what happens is ideally represented by two limit-case scenarios.
On the one hand, under-complex models risk providing few insights, as they will leave on the table most of the valuable information. On the other hand, over-complex models would closely mirror past observations with a high risk of overfitting, that is, to fail in generalizing the relationship and have a high error rate on never-seen data.
From this standpoint, it clearly emerges how the new generation of AI models adds substantial value with respect to traditional ones. Indeed, thanks to their ability to uncover patterns and continuously learn from new data, asset and investment managers are increasingly adopting AI in their investment processes as a better tool to decipher the complex dynamics that drive financial markets.
Yet, as with any powerful tool, we cannot expect to generate value effortlessly. For this reason, as we will see, successfully applying AI to make better investment decisions requires finding the right mix of three ingredients: a clear understanding of the economic rationale, a thorough knowledge of the interactions between the financial variables, and a disciplined approach rooted in the scientific method.
Start integrating AI into your investment process with the most advanced no-code solution.
Book A Custom Demo
Take the Medicine. Not the Placebo.
During the years, Finance has heavily drawn from the body of knowledge developed in many other fields of study such as biology, physics and medicine, especially for what concerns the application of the scientific method to investment research.
Indeed, from the late 1970s, applying statistical techniques to study the behaviour of financial markets and eliminate the negative effects of emotions has become the backbone of the quantitative approach to investing.
However, using these techniques without a thorough understanding of the underlying mechanisms of financial markets may prove to be inadequate, not only to account for the high degree of complexity that they involve but also to keep pace with their constant evolution.
This concept is well summarized by saying that it is important to develop theories – instead of rules – to build successful investment strategies. Being able to effectively separate the signal from the noise is therefore essential to model and control the interaction between the many variables involved and obtain valuable insights.
A very useful example to understand the role of the scientific method in investment research is offered by looking at how pharmaceutical companies develop a new drug. Instead of trying out many different molecules and see if they work to cure a disease, the biggest effort for researchers is represented by understanding the cause-effect relationships between the drug composition and bacteria (or virus), so as to infer their interactions with human cells and justify why patients will get better after taking the drug.
Only at a later stage, when the final drug has passed several tests to account for the possible side effects, they will proceed with the clinical trial and the direct testing on the patients as we can see from Exhibit 1.
We can learn a lot from this example. Indeed, in a similar way, when we develop investment strategies we must follow the same path. Starting from the knowledge of the underlying dynamics of financial markets and their economic and behavioural rationale, we must use the scientific method to find out how it will play out under many different circumstances and have control over its execution.
Risk is What You Don’t See
Suppose we want to develop an investment strategy based on a potentially interesting market dynamic that we want to exploit.
From the previous example, we have learned that an uncritical trial and error process would not be enough, but rather we must seek to get a general and objective understanding of how the strategy works, like its fundamental drivers, its economic and behavioral rationale or the reasons why it should persist in the future.
We would then build a raw model to represent such behaviour and we would start to optimize it to reach, for example, a better efficiency as we try many different versions of it.
However, if we were to use traditional techniques for this job, we would eventually end up making some sort of assumptions to narrow them down to several (even hundreds) of those, as it would be cumbersome to explore how all the possible combinations of the model will play out.
Yet, being unable to explore all the possible combinations of a model is definitely one of the main shortcomings of traditional models and the reason why they risk being unstable as time goes by and markets change.
A risk that is not rewarded in any way. On the contrary, Artificial Intelligence tools – like neural networks – allow performing this task on scale, leveraging their high computational power to simultaneously process each possible combination and have a better understanding of the interactions between the different components of the model.
To a certain extent, we can imagine the difference between the two approaches as looking only at the top of a very big iceberg, in which the vast majority of its mass is hidden underwater, as we see from Exhibit 2.
What we understand is that traditional models only take into account a fraction of the relevant information available, scratching the surface of what is really needed to understand a given market dynamic.
AI instead works as a sophisticated and powerful radar scanning the depth of data, bringing to light the information that stays disguised behind the noise.
Eventually, with a wider and holistic understanding of the inner workings, it becomes possible to see all the opportunities (and risks) and make more informed investment decisions, reducing the risk of being hit by unexpected events that causes strategies to drift from expected results.
Overfitting: The Financial Elephant in The Room
Have you ever missed out on a crucial piece of information, even though it was clearly visible? Or noticed a blind spot only after someone told you?
If the answer is “yes”, you do not need to worry. Indeed, the tendency of our brain to selectively focalize our attention on a narrow portion of reality excluding a part of the background information is what psychologists call selective attention.
The first studies on the subject date back to 1958 when Donald E. Broadbent – an English experimental psychologist – expressed the idea that, at any given time, we are not able to correctly process all the inputs we receive, but only on a fraction of those.
Many other works have followed over the years, but one of the most famous (and curious) experiments on selective attention is definitely the one conducted by Chabris and Simons in 1999, when they were researchers at Harvard University.
They let a group of students watch a short video in which six people passed two basketballs around, asking them to count the number of passes they performed.
At some point during the video, a person dressed in a gorilla suit would enter the scene, face the camera, thump its chest and leave. Surprisingly, they discovered that nearly half of the students admitted to having missed the gorilla walking by, since they were too absorbed in counting the passes.
In the world of investing, the same idea of the “gorilla in the room” can be linked to what statisticians call overfitting, that is, to select models that too closely mimic the original data they were trained on, missing out on the underlying dynamics and thus useless when used on never-seen observations.
In particular, this tends to happen very frequently with financial data because it is hard to correctly identify the signal, the part that contains useful information, from the noise, which does not contain any.
The result is that overfitted financial models tend to be overly complex and case-specific, therefore failing to correctly generalize and provide useful insights. The concepts of overfitting and robust fitting – the desired result – are shown in Exhibit 3.
As we can see, a robust model (the green line) is able to generalize well the original data, capturing the underlying dynamic with a good level of approximation and a limited error rate.
In our example, it recognizes a linear relation between data that leaves out only one circle. Overfitting instead, means training a model to such a degree of specificity to the original dataset that it may appear a flawless – yet not useful – solution.
How AI Cuts Through Complexity
The reason why studying the financial markets is at the same time both fascinating and challenging is because they exhibit a very high level of complexity. This happens because every day they show us the result of billions of interactions between many different players, each one buying or selling securities according to their own strategy, sentiment and investment outlook.
This makes it very difficult to correctly identify (and model) the interactions between the different variables involved, be them macroeconomic or financial, and gain useful insights about their future evolution. However, especially when traditional techniques are involved, many times we need to turn to simplistic hypotheses or recursive patterns to interpret this complexity.
Yet, this approach poses a significant upper limit to our ability to understand the reality of financial markets. From this standpoint, it clearly emerges why (and how) Artificial Intelligence can add substantial value to help us navigate this complexity.
To a certain extent, we can visualize this difference by looking at how humans and AI would approach another notoriously complex problem: solving the Rubik’s cube.
Indeed, despite the cube has over 43 billions of billions of possible combinations, it has been mathematically proven that it is possible to solve it in about 22 moves.
However, since our brain is forced to rely on a limited set of moves to get each colored tile to a known position from which to eventually solve the cube, this creates an inefficient strategy that leads even the best players to take about 50 steps to complete the puzzle.
Here, we see exactly where AI tools can bring the biggest benefit. Thanks to their high computational power, they can avoid turning to pre-structured sequences or categories and find the solution to problems by learning from the structure of data.
From a financial point of view, this makes AI particularly suited to processing massive amounts of data and uncover complex interactions between variables by capturing non-linear relationships and recognizing structural market shifts.
In addition, thanks to their continuous learning, AI models adapt to new data, learning without any pre-established schemes, so to ensure that the knowledge acquired is always up to date, expanded and improved with new observations.
Finding The Sweet Spot
As we have seen so far, AI has the potential to add substantial value to the investment process. Indeed, we have learned that to build reliable investment strategies, we need to combine the systematic application of the scientific method to understand the real nature of financial market dynamics together with finding the right tradeoff between the complexity and the explanatory power of our model.
What we must avoid is relying on assumptions that make our models fragile to sudden changes or, worse, unstable as markets evolve. Instability is not only dangerous because it signals a lack of control over the underlying market dynamic, but because it makes us exposed to the risk of unexpected shocks that can severely impact the final performance achieved. Achieving stability should be seen not only as the desirable end of a long development process, but also for a more practical reason: because stable things tend to persist, and so be profitable over the long term.
Exhibit 4 depicts how we can use AI to study and reach stability. To understand how the different variables interact with each other and contribute to the final result, we should not consider each trial as an independent observation but instead as part of a much bigger surface that can be either stable or unstable.
AI models have the power to explore all possible interactions between variables and build a proper and more populated surface. AI can grasp what is beyond the surface, and go past the depths of the apparent reality. It is no longer a question of seeing a spurious relationship between different models, but a real universe of relationships and interconnections between multiple variables.
Ideally, by leveraging automated tools to identify promising model configurations, understanding the evolution of results over time and analyzing interactions with other variables, AI-driven models do the work of thousands of analysts, thus making scalable the quest for the statistical relevance of the investable signal hidden among data.
In particular, what asset and portfolio managers look for is the combination of model parameters that gives shape to a stable and robust surface that, in turn, is the result of a stable investment strategy.
Indeed, looking at the graph on the right of Exhibit 5, a smooth surface captures the investment signal with high consistency, meaning that it tends to display superior performances regardless of the unfolding market cycle.
On the contrary, the performance of the unstable surface on the left features many peaks and valleys, thus generating a highly volatile strategy.
So, using AI to gain control over market dynamics, asset and portfolio managers can develop investment strategies that are better positioned to turn the valuable pieces of information they collect into profitable and resilient portfolios that help them navigate market turmoil.
Unlock New Value with AI
In an era of unprecedented complexity in financial markets, asset and portfolio managers can leverage Artificial Intelligence to extrapolate more value from their investments and make better and more informed decisions.
As opposed to traditional techniques that tend to rely on restrictive assumptions, AI has dramatically improved our ability to analyze data and assess the full scale of interactions among the determinants of market dynamics.
By uncovering hidden and non-linear interconnections between variables, AI tools are essential in achieving a thorough understanding of why and how markets move. This helps to overcome overfitting, a problem that usually goes with traditional models and that causes poor performance and predictive power.
Eventually, this has opened new routes for navigating the complexity of financial markets. Asset and portfolio managers thoughtfully deploying AI can indeed improve the quality of investment decision making and quickly adapt to the gradual evolution of financial markets.
