FinRL Multiple Stock Trading

This is my short note on me trying to run FinRL multiple stock trading. It’s hard to find ML for Multiple Stock Trading. After trying several tutorials, I can run FinRL for Multiple stock as of April 2021. But I didn’t have time to verify the result yet. I’ll check the prediction result for the models later, when I have time.

Useful Tutorials

DRL Multiple Stocks Trading - FinRL Library 1.0.1 documentation

FinRL for Quantitative Finance: Tutorial for Multiple Stock Trading

AI4Finance-LLC/FinRL-Library

I use Google Colab to train this model. First, Mount Google Drive.

from google.colab import drive
drive.mount(‘/content/drive/’)

Create Folder you want to put your model and logs in your Google Drive.

Then install FinRL Library.

!pip install git+https://github.com/AI4Finance-LLC/FinRL-Library.git

Install dependencies.

import os
import pathlib
import pkg_resources
import pip
installedPackages = {pkg.key for pkg in pkg_resources.working_set}
required = {'yfinance', 'pandas', 'matplotlib', 'stockstats','stable-baselines','gym','tensorflow'}
missing = required - installedPackages
if missing:
!pip install yfinance
!pip install pandas
!pip install matplotlib
!pip install stockstats
!pip install gym
!pip install stable-baselines[mpi]
!pip install tensorflow==1.15.4

Then you restart runtime.

Run again from the beginning.

Install more dependencies.

import pandas as pd
import matplotlib.pyplot as plt
import numpy as np
import matplotlib
matplotlib.use('Agg')
import datetime
import os
from finrl.config import config
from finrl.marketdata.yahoodownloader import YahooDownloader
from finrl.preprocessing.preprocessors import FeatureEngineer
from finrl.preprocessing.data import data_split
from finrl.env.env_stocktrading import StockTradingEnv
from finrl.model.models import DRLAgent
from finrl.trade.backtest import backtest_stats, backtest_plot
import sys
sys.path.append("../FinRL-Library")

Install pyfolio.

!pip install pyfolio

Create Directory on your Google Drive to keep model and logs.

PATH_TO_MODEL_DIR = 'drive/MyDrive/Colab Notebooks/FinRLManystock/'
print(PATH_TO_MODEL_DIR)
import os
if not os.path.exists(PATH_TO_MODEL_DIR + config.DATA_SAVE_DIR):
os.makedirs(PATH_TO_MODEL_DIR + config.DATA_SAVE_DIR)
if not os.path.exists(PATH_TO_MODEL_DIR + config.TRAINED_MODEL_DIR):
os.makedirs(PATH_TO_MODEL_DIR + config.TRAINED_MODEL_DIR)
if not os.path.exists(PATH_TO_MODEL_DIR + config.TENSORBOARD_LOG_DIR):
os.makedirs(PATH_TO_MODEL_DIR + config.TENSORBOARD_LOG_DIR)
if not os.path.exists(PATH_TO_MODEL_DIR + config.RESULTS_DIR):
os.makedirs(PATH_TO_MODEL_DIR + config.RESULTS_DIR)

Set Stock Data you want and download data from YahooDownloader.

dow_30_ticker = ['AAPL','MSFT','JPM','V','RTX','PG','GS','NKE','DIS','AXP',
'HD','INTC','WMT','IBM','MRK','UNH','KO','CAT','TRV','JNJ',
'CVX','MCD','VZ','CSCO','XOM','BA','MMM','PFE','WBA','DD']
data_df = YahooDownloader(start_date = '2015-01-01',
end_date = '2021-01-01',
ticker_list = dow_30_ticker).fetch_data()
data_df.head()

data_df.head()

Add Technical indicator.

tech_indicator_list=config.TECHNICAL_INDICATORS_LIST
## you can add more technical indicators
## visit https://github.com/jealous/stockstats for different names
tech_indicator_list=tech_indicator_list+['kdjk','open_2_sma','boll','close_10.0_le_5_c','wr_10','dma','trix']
print(tech_indicator_list)
fe = FeatureEngineer(
use_technical_indicator=True,
tech_indicator_list = tech_indicator_list,
use_turbulence=False,
user_defined_feature = False)
data_df = fe.preprocess_data(data_df)
data_df.head()

Your stock data with technical indicator.

Ass covariance.

df = data_df
# add covariance matrix as states
df=df.sort_values(['date','tic'],ignore_index=True)
df.index = df.date.factorize()[0]
cov_list = []
# look back is one year
lookback=252
for i in range(lookback,len(df.index.unique())):
data_lookback = df.loc[i-lookback:i,:]
price_lookback=data_lookback.pivot_table(index = 'date',columns = 'tic', values = 'close')
return_lookback = price_lookback.pct_change().dropna()
covs = return_lookback.cov().values
cov_list.append(covs)
df_cov = pd.DataFrame({'date':df.date.unique()[lookback:],'cov_list':cov_list})
df = df.merge(df_cov, on='date')
df = df.sort_values(['date','tic']).reset_index(drop=True)
df.head()

Split train and test data by date. Save to csv files.

train = data_split(df, start = '2015-01-01', end = '2019-01-01')
trade = data_split(df, start = '2019-01-01', end = '2021-01-01')
train.to_csv(PATH_TO_MODEL_DIR + config.DATA_SAVE_DIR + '/train_MULTI.csv',index=False)
trade.to_csv(PATH_TO_MODEL_DIR + config.DATA_SAVE_DIR + '/trade_MULTI.csv',index=False)

Set environment.

stock_dimension = len(train.tic.unique())
state_space = 1 + 2*stock_dimension + len(config.TECHNICAL_INDICATORS_LIST)*stock_dimension
#state_space = 156
print(f"Stock data Dimensions: {stock_dimension}, State Spaces: {state_space}")
env_kwargs = {
"hmax": 100,
"initial_amount": 1000000,
#"transaction_cost_pct": 0.001,
"buy_cost_pct":0.001,
"sell_cost_pct":0.001,
"state_space": state_space,
"stock_dim": stock_dimension,
"tech_indicator_list": config.TECHNICAL_INDICATORS_LIST,
"action_space": stock_dimension,
"reward_scaling": 1e-4}
e_train_gym = StockTradingEnv(df = train, **env_kwargs)
env_train, _ = e_train_gym.get_sb_env()
print(type(env_train))

Training time… (I use GPU runtime in Colab. It take around 10 minutes for total_timesteps=100000)

agent = DRLAgent(env = env_train)
A2C_PARAMS = {"n_steps": 5, "ent_coef": 0.005, "learning_rate": 0.0002}
model_a2c = agent.get_model(model_name="a2c",model_kwargs = A2C_PARAMS)
trained_a2c = agent.train_model(model=model_a2c,
tb_log_name='a2c',
total_timesteps=100000)

Save model.

trained_a2c.save(PATH_TO_MODEL_DIR + config.TRAINED_MODEL_DIR+'/trained_a2c.model')

Test

e_trade_gym = StockTradingEnv(df = trade, **env_kwargs)
df_account_value, df_actions = DRLAgent.DRL_prediction(model=trained_a2c, environment = e_trade_gym)

Check account value and action.

print(df_account_value)
df_actions.head()

Check Result.

from finrl.trade.backtest import backtest_stats, backtest_plot
print("==============Results===========")
now = datetime.datetime.now().strftime('%Y%m%d-%Hh%M')
perf_stats_all = backtest_stats(account_value=df_account_value)
perf_stats_all = pd.DataFrame(perf_stats_all)

Plot the result.

%matplotlib inline
backtest_plot(account_value=df_account_value, baseline_ticker = 'AAPL',
baseline_start = '2019-01-01', baseline_end = '2021-01-01')

That’s all for my short note. Source is in my github repo.

krisadas/finrl-multiple