An Application of Deep Reinforcement Learning to Algorithmic Trading

Théate, Thibaut; Ernst, Damien

doi:10.1016/j.eswa.2021.114632

Download

Article (Scientific journals)

An Application of Deep Reinforcement Learning to Algorithmic Trading

Théate, Thibaut; Ernst, Damien

2021 • In Expert Systems with Applications, 173

Peer Reviewed verified by ORBi

Permalink
https://hdl.handle.net/2268/246457

DOI
10.1016/j.eswa.2021.114632

Files (1)Send to Details Statistics Bibliography Similar publications

Files

Full Text

2004.06627.pdf

Author preprint (1.54 MB)

Arxiv Preprint (PDF)

Download

All documents in ORBi are protected by a user license.

Send to

RIS BibTex APA Chicago Permalink X Linkedin

Details

Keywords :

Artificial intelligence; Deep reinforcement learning; Algorithmic trading; Trading policy

Abstract :

[en] This scientific research paper presents an innovative approach based on deep reinforcement learning (DRL) to solve the algorithmic trading problem of determining the optimal trading position at any point in time during a trading activity in the stock market. It proposes a novel DRL trading policy so as to maximise the resulting Sharpe ratio performance indicator on a broad range of stock markets. Denominated the Trading Deep Q-Network algorithm (TDQN), this new DRL approach is inspired from the popular DQN algorithm and significantly adapted to the specific algorithmic trading problem at hand. The training of the resulting reinforcement learning (RL) agent is entirely based on the generation of artificial trajectories from a limited set of stock market historical data. In order to objectively assess the performance of trading strategies, the research paper also proposes a novel, more rigorous performance assessment methodology. Following this new performance assessment approach, promising results are reported for the TDQN algorithm.

Research center :

Montefiore Institute - Montefiore Institute of Electrical Engineering and Computer Science - ULiège

Disciplines :

Finance
Computer science

Author, co-author :

Théate, Thibaut ; Université de Liège - ULiège > Dép. d'électric., électron. et informat. (Inst.Montefiore) > Smart grids

Ernst, Damien ; Université de Liège - ULiège > Dép. d'électric., électron. et informat. (Inst.Montefiore) > Smart grids

Language :

English

Title :

An Application of Deep Reinforcement Learning to Algorithmic Trading

Publication date :

2021

Journal title :

Expert Systems with Applications

ISSN :

0957-4174

eISSN :

1873-6793

Publisher :

Elsevier, Oxford, United Kingdom

Volume :

173

Peer reviewed :

Peer Reviewed verified by ORBi

Additional URL :

https://arxiv.org/abs/2004.06627

Funders :

F.R.S.-FNRS - Fonds de la Recherche Scientifique [BE]

Available on ORBi :

since 07 April 2020

Statistics

Number of views

558 (45 by ULiège)

Number of downloads

474 (12 by ULiège)

More statistics

Scopus citations^®

Scopus citations^®
without self-citations

OpenCitations

Bibliography

Almahdi, S., Yang, S., A constrained portfolio trading system using particle swarm algorithm and recurrent reinforcement learning. Expert Systems With Applications 130 (2019), 145–156.
Arévalo, A., Niño, J., Hernández, G., Sandoval, J., High-frequency trading strategy based on deep neural networks. ICIC, 2016.
Arulkumaran, K., Deisenroth, M. P., Brundage, M., & Bharath, A. A. (2017). A brief survey of deep reinforcement learning. CoRR, abs/1708.05866.
Bailey, D.H., Borwein, J.M., de Prado, M.L., Zhu, Q.J., Pseudo-mathematics and financial charlatanism: the effects of backtest overfitting on out-of-sample performance. Notice of the American Mathematical Society, 2014, 458–471.
Bao, W.N., Yue, J., Rao, Y., A deep learning framework for financial time series using stacked autoencoders and long-short term memory. PloS one, 12, 2017.
Bellemare, M. G., Dabney, W., & Munos, R. (2017). A distributional perspective on reinforcement learning. CoRR, abs/1707.06887.
Bollen, J., Mao, H., & jun Zeng, X. (2011). Twitter mood predicts the stock market. Journal of Computer Science, 2, 1–8.
Boukas, I., Ernst, D., Théate, T., Bolland, A., Huynen, A., Buchwald, M., Wynants, C., & Cornélusse, B. (2020). A deep reinforcement learning framework for continuous intraday market bidding. ArXiv, abs/2004.05940.
Busoniu, L., Babuska, R., De Schutter, B., Ernst, D., Reinforcement learning and dynamic programming using function approximators. 2010, CRC Press.
Carapuço, J., Neves, R.F., Horta, N., Reinforcement learning applied to Forex trading. Applied Soft Computing 73 (2018), 783–794.
Chan, E.P., Quantitative trading: how to build your own algorithmic trading business. 2009, Wiley.
Chan, E.P., Algorithmic trading: winning strategies and their rationale. 2013, Wiley.
Dempster, M.A.H., Leemans, V., An automated FX trading system using adaptive reinforcement learning. Expert Systems With Applications 30 (2006), 543–552.
Deng, Y., Bao, F., Kong, Y., Ren, Z., Dai, Q., Deep direct reinforcement learning for financial signal representation and trading. IEEE Transactions on Neural Networks and Learning Systems 28 (2017), 653–664.
Fortunato, M., Azar, M. G., Piot, B., Menick, J., Hessel, M., Osband, I., Graves, A., Mnih, V., Munos, R., Hassabis, D., Pietquin, O., Blundell, C., & Legg, S. (2018). Noisy networks for exploration. CoRR, abs/1706.10295.
Goodfellow, I., Bengio, Y., Courville, A., Deep learning. 2016, MIT Press.
Goodfellow, I.J., Bengio, Y., Courville, A.C., Deep learning. Nature 521 (2015), 436–444.
Hausknecht, M. J., & Stone, P. (2015). Deep recurrent Q-Learning for partially observable MDPs. CoRR, abs/1507.06527.
Hendershott, T., Jones, C.M., Menkveld, A.J., Does algorithmic trading improve liquidity?. Journal of Finance 66 (2011), 1–33.
Hessel, M., Modayil, J., van Hasselt, H. P., Schaul, T., Ostrovski, G., Dabney, W., Horgan, D., Piot, B., Azar, M. G., & Silver, D. (2017). Rainbow: Combining improvements in deep reinforcement learning. CoRR, abs/1710.02298.
Ioannidis, J.P.A., Why most published research findings are false. PLoS Med, 2, 2005, 124.
Ioffe, S., & Szegedy, C. (2015). Batch normalization: Accelerating deep network training by reducing internal covariate shift. CoRR, abs/1502.03167.
Jeong, G., Kim, H., Improving financial trading decisions using Deep Q-Learning: Predicting the number of shares, action strategies, and transfer learning. Expert Systems With Applications 117 (2019), 125–138.
Kingma, D. P., & Ba, J. (2015). Adam: A method for stochastic optimization. CoRR, abs/1412.6980.
LeCun, Y., Bengio, Y., Hinton, G., Deep learning. Nature, 521, 2015.
Lei, K., Zhang, B., Li, Y., Yang, M., & Shen, Y. (2020). Time-driven Feature-aware jointly deep reinforcement learning for financial signal representation and algorithmic trading. Expert Systems With Applications, 140.
Leinweber, D., Sisk, J., Event-driven trading and the “New News”. The Journal of Portfolio Management 38 (2011), 110–124.
Li, Y. (2017). Deep reinforcement learning: An overview. CoRR, abs/1701.07274.
Mnih, V., Kavukcuoglu, K., Silver, D., Graves, A., Antonoglou, I., Wierstra, D., & Riedmiller, M. A. (2013). Playing Atari with deep reinforcement learning. CoRR, abs/1312.5602.
Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A.A., Veness, J., Bellemare, M.G., Graves, A., Riedmiller, M.A., Fidjeland, A., Ostrovski, G., Petersen, S., Beattie, C., Sadik, A., Antonoglou, I., King, H., Kumaran, D., Wierstra, D., Legg, S., Hassabis, D., Human-level control through deep reinforcement learning. Nature 518 (2015), 529–533.
Moody, J.E., Saffell, M., Learning to trade via direct reinforcement. IEEE Transactions on Neural Networks 12:4 (2001), 875–889.
Narang, R.K., Inside the black box. 2009, Wiley.
Nuij, W., Milea, V., Hogenboom, F., Frasincar, F., Kaymak, U., An automated framework for incorporating news into stock trading strategies. IEEE Transactions on Knowledge and Data Engineering 26 (2014), 823–835.
Nuti, G., Mirghaemi, M., Treleaven, P.C., Yingsaeree, C., Algorithmic trading. Computer 44 (2011), 61–69.
Paiva, F.D., Cardoso, R.T.N., Hanaoka, G.P., Duarte, W.M., Decision-making for financial trading: A fusion approach of machine learning and portfolio selection. Expert Systems With Applications 115 (2019), 635–655.
Park, H., Sim, M.K., Choi, D.G., An intelligent financial portfolio trading strategy using Deep Q-Learning. Expert Systems With Applications, 158, 2020, 113573.
Schaul, T., Quan, J., Antonoglou, I., & Silver, D. (2016). Prioritized experience replay. CoRR, abs/1511.05952.
Schulman, J., Wolski, F., Dhariwal, P., Radford, A., & Klimov, O. (2017). Proximal policy optimization algorithms. CoRR, abs/1707.06347.
Shao, K., Tang, Z., Zhu, Y., Li, N., & Zhao, D. (2019). A survey of deep reinforcement learning in video games. ArXiv, abs/1912.10944.
Silver, D., Huang, A., Maddison, C.J., Guez, A., Sifre, L., van den Driessche, G., Schrittwieser, J., Antonoglou, I., Panneershelvam, V., Lanctot, M., Dieleman, S., Grewe, D., Nham, J., Kalchbrenner, N., Sutskever, I., Lillicrap, T.P., Leach, M., Kavukcuoglu, K., Graepel, T., Hassabis, D., Mastering the game of go with deep neural networks and tree search. Nature 529 (2016), 484–489.
Sutton, R.S., Barto, A.G., Reinforcement learning: An introduction. 2nd ed., 2018, The MIT Press.
Szepesvari, C., Algorithms for reinforcement learning. 2010, Morgan and Claypool Publishers.
Treleaven, P.C., Galas, M., Lalchand, V., Algorithmic trading review. Communications of the ACM 56 (2013), 76–85.
van Hasselt, H. P., Guez, A., & Silver, D. (2015). Deep reinforcement learning with double Q-Learning. CoRR, abs/1509.06461.
Wang, Z., de Freitas, N., & Lanctot, M. (2015). Dueling network architectures for deep reinforcement learning. CoRR, abs/1511.06581.
Watkins, C.J.C.H., Dayan, P., Technical note: Q-Learning. Machine Learning 8 (1992), 279–292.
Zhang, C., Vinyals, O., Munos, R., & Bengio, S. (2018). A study on overfitting in deep reinforcement learning. CoRR, abs/1804.06893.