Is it possible to avoid exception if a coin misses it's data while backtesting in FreqAI mode

[StephaneMangin]

Describe your environment

(if applicable)

• Operating system: ubuntu server
• Python Version: 3.9.0
• CCXT version: 4.1.74
• Freqtrade Version: tag 2023.12 from freqai docker image

Describe the enhancement

Context: An hybrid freqtrade instance using freqAI decisions on enter/exit signals (to make sure my signals have the maximum potential)

Is it possible to NOT thow exception while a crypto is missing data on backtest mode but "just" do not take in account this coin. I use all cryptos from binance during backtest (to measure performance of several versions of a strategy using bullish, bearish and stable trend over a ful list of crypto to make decisions on production) .

If i have to check each separate coin data before manually, it will take me ages to process and i want to automate this thing properly over time regrding market changes.

I have a python script that automate backtest over several versions of my strategy to ensure performance increase over each commit changes.
(i confess it takes days and days to process, but the result is quite amazing. FreqAI could be usefull to add a new brick about automated decision making)

For instance i have several exceptions occuring :

• One from missing data

backtest_freqai_training            | 2024-01-11 20:46:44,388 - freqtrade.freqai.base_models.BaseClassifierModel - INFO - -------------------- Starting training LQTY/USDT --------------------
backtest_freqai_training            | 2024-01-11 20:46:44,391 - freqtrade.freqai.data_kitchen - INFO - LQTY/USDT: dropped 0 training points due to NaNs in populated dataset 0.
backtest_freqai_training            | 2024-01-11 20:46:44,391 - freqtrade.freqai.freqai_interface - WARNING - Training LQTY/USDT raised exception OperationalException. Message: LQTY/USDT: all training data dropped due to NaNs. You likely did not download enough training data prior to your backtest timerange. Hint:
backtest_freqai_training            | https://www.freqtrade.io/en/stable/freqai-running/#downloading-data-to-cover-the-full-backtest-period, skipping.
backtest_freqai_training            | Traceback (most recent call last):
backtest_freqai_training            |   File "/freqtrade/freqtrade/freqai/freqai_interface.py", line 351, in start_backtesting
backtest_freqai_training            |     self.model = self.train(dataframe_train, pair, dk)
backtest_freqai_training            |                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
backtest_freqai_training            |   File "/freqtrade/freqtrade/freqai/base_models/BaseClassifierModel.py", line 41, in train
backtest_freqai_training            |     features_filtered, labels_filtered = dk.filter_features(
backtest_freqai_training            |                                          ^^^^^^^^^^^^^^^^^^^
backtest_freqai_training            |   File "/freqtrade/freqtrade/freqai/data_kitchen.py", line 248, in filter_features
backtest_freqai_training            |     raise OperationalException(
backtest_freqai_training            | freqtrade.exceptions.OperationalException: LQTY/USDT: all training data dropped due to NaNs. You likely did not download enough training data prior to your backtest timerange. Hint:
backtest_freqai_training            | https://www.freqtrade.io/en/stable/freqai-running/#downloading-data-to-cover-the-full-backtest-period
backtest_freqai_training            | 2024-01-11 20:46:44,392 - freqtrade.freqai.freqai_interface - INFO - Saving metadata to disk.
backtest_freqai_training            | 2024-01-11 20:46:44,392 - freqtrade - ERROR - Fatal exception!
backtest_freqai_training            | Traceback (most recent call last):
backtest_freqai_training            |   File "/freqtrade/freqtrade/main.py", line 42, in main
backtest_freqai_training            |     return_code = args['func'](args)
backtest_freqai_training            |                   ^^^^^^^^^^^^^^^^^^
backtest_freqai_training            |   File "/freqtrade/freqtrade/commands/optimize_commands.py", line 58, in start_backtesting
backtest_freqai_training            |     backtesting.start()
backtest_freqai_training            |   File "/freqtrade/freqtrade/optimize/backtesting.py", line 1392, in start
backtest_freqai_training            |     min_date, max_date = self.backtest_one_strategy(strat, data, timerange)
backtest_freqai_training            |                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
backtest_freqai_training            |   File "/freqtrade/freqtrade/optimize/backtesting.py", line 1309, in backtest_one_strategy
backtest_freqai_training            |     preprocessed = self.strategy.advise_all_indicators(data)
backtest_freqai_training            |                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
backtest_freqai_training            |   File "/freqtrade/freqtrade/strategy/interface.py", line 1346, in advise_all_indicators
backtest_freqai_training            |     return {pair: self.advise_indicators(pair_data.copy(), {'pair': pair}).copy()
backtest_freqai_training            |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
backtest_freqai_training            |   File "/freqtrade/freqtrade/strategy/interface.py", line 1346, in <dictcomp>
backtest_freqai_training            |     return {pair: self.advise_indicators(pair_data.copy(), {'pair': pair}).copy()
backtest_freqai_training            |                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
backtest_freqai_training            |   File "/freqtrade/freqtrade/strategy/interface.py", line 1378, in advise_indicators
backtest_freqai_training            |     return self.populate_indicators(dataframe, metadata)
backtest_freqai_training            |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
backtest_freqai_training            |   File "/freqtrade/user_data/strategies/abstract.py", line 289, in populate_indicators
backtest_freqai_training            |     df = super().populate_indicators(dataframe, metadata)
backtest_freqai_training            |          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
backtest_freqai_training            |   File "/freqtrade/user_data/strategies/freqai_predictor.py", line 168, in populate_indicators
backtest_freqai_training            |     dataframe = self.freqai.start(dataframe, metadata, self)
backtest_freqai_training            |                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
backtest_freqai_training            |   File "/freqtrade/freqtrade/freqai/freqai_interface.py", line 161, in start
backtest_freqai_training            |     dk = self.start_backtesting(dataframe, metadata, self.dk, strategy)
backtest_freqai_training            |          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
backtest_freqai_training            |   File "/freqtrade/freqtrade/freqai/freqai_interface.py", line 368, in start_backtesting
backtest_freqai_training            |     self.dd.save_metadata(dk)
backtest_freqai_training            |   File "/freqtrade/freqtrade/freqai/data_drawer.py", line 457, in save_metadata
backtest_freqai_training            |     dk.data["training_features_list"] = list(dk.data_dictionary["train_features"].columns)
backtest_freqai_training            |                                              ~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^
backtest_freqai_training            | KeyError: 'train_features'

• And one from a past model exception i don't understand quite well:

backtest_freqai_training            | 2024-01-11 20:46:44,388 - freqtrade.freqai.base_models.BaseClassifierModel - INFO - -------------------- Starting training LQTY/USDT --------------------
backtest_freqai_training            | 2024-01-11 20:46:44,391 - freqtrade.freqai.data_kitchen - INFO - LQTY/USDT: dropped 0 training points due to NaNs in populated dataset 0.
backtest_freqai_training            | 2024-01-11 20:46:44,391 - freqtrade.freqai.freqai_interface - WARNING - Training LQTY/USDT raised exception OperationalException. Message: LQTY/USDT: all training data dropped due to NaNs. You likely did not download enough training data prior to your backtest timerange. Hint:
backtest_freqai_training            | https://www.freqtrade.io/en/stable/freqai-running/#downloading-data-to-cover-the-full-backtest-period, skipping.
backtest_freqai_training            | Traceback (most recent call last):
backtest_freqai_training            |   File "/freqtrade/freqtrade/freqai/freqai_interface.py", line 351, in start_backtesting
backtest_freqai_training            |     self.model = self.train(dataframe_train, pair, dk)
backtest_freqai_training            |                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
backtest_freqai_training            |   File "/freqtrade/freqtrade/freqai/base_models/BaseClassifierModel.py", line 41, in train
backtest_freqai_training            |     features_filtered, labels_filtered = dk.filter_features(
backtest_freqai_training            |                                          ^^^^^^^^^^^^^^^^^^^
backtest_freqai_training            |   File "/freqtrade/freqtrade/freqai/data_kitchen.py", line 248, in filter_features
backtest_freqai_training            |     raise OperationalException(
backtest_freqai_training            | freqtrade.exceptions.OperationalException: LQTY/USDT: all training data dropped due to NaNs. You likely did not download enough training data prior to your backtest timerange. Hint:
backtest_freqai_training            | https://www.freqtrade.io/en/stable/freqai-running/#downloading-data-to-cover-the-full-backtest-period
backtest_freqai_training            | 2024-01-11 20:46:44,392 - freqtrade.freqai.freqai_interface - INFO - Saving metadata to disk.
backtest_freqai_training            | 2024-01-11 20:46:44,392 - freqtrade - ERROR - Fatal exception!
backtest_freqai_training            | Traceback (most recent call last):
backtest_freqai_training            |   File "/freqtrade/freqtrade/main.py", line 42, in main
backtest_freqai_training            |     return_code = args['func'](args)
backtest_freqai_training            |                   ^^^^^^^^^^^^^^^^^^
backtest_freqai_training            |   File "/freqtrade/freqtrade/commands/optimize_commands.py", line 58, in start_backtesting
backtest_freqai_training            |     backtesting.start()
backtest_freqai_training            |   File "/freqtrade/freqtrade/optimize/backtesting.py", line 1392, in start
backtest_freqai_training            |     min_date, max_date = self.backtest_one_strategy(strat, data, timerange)
backtest_freqai_training            |                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
backtest_freqai_training            |   File "/freqtrade/freqtrade/optimize/backtesting.py", line 1309, in backtest_one_strategy
backtest_freqai_training            |     preprocessed = self.strategy.advise_all_indicators(data)
backtest_freqai_training            |                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
backtest_freqai_training            |   File "/freqtrade/freqtrade/strategy/interface.py", line 1346, in advise_all_indicators
backtest_freqai_training            |     return {pair: self.advise_indicators(pair_data.copy(), {'pair': pair}).copy()
backtest_freqai_training            |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
backtest_freqai_training            |   File "/freqtrade/freqtrade/strategy/interface.py", line 1346, in <dictcomp>
backtest_freqai_training            |     return {pair: self.advise_indicators(pair_data.copy(), {'pair': pair}).copy()
backtest_freqai_training            |                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
backtest_freqai_training            |   File "/freqtrade/freqtrade/strategy/interface.py", line 1378, in advise_indicators
backtest_freqai_training            |     return self.populate_indicators(dataframe, metadata)
backtest_freqai_training            |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
backtest_freqai_training            |   File "/freqtrade/user_data/strategies/abstract.py", line 289, in populate_indicators
backtest_freqai_training            |     df = super().populate_indicators(dataframe, metadata)
backtest_freqai_training            |          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
backtest_freqai_training            |   File "/freqtrade/user_data/strategies/freqai_predictor.py", line 168, in populate_indicators
backtest_freqai_training            |     dataframe = self.freqai.start(dataframe, metadata, self)
backtest_freqai_training            |                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
backtest_freqai_training            |   File "/freqtrade/freqtrade/freqai/freqai_interface.py", line 161, in start
backtest_freqai_training            |     dk = self.start_backtesting(dataframe, metadata, self.dk, strategy)
backtest_freqai_training            |          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
backtest_freqai_training            |   File "/freqtrade/freqtrade/freqai/freqai_interface.py", line 368, in start_backtesting
backtest_freqai_training            |     self.dd.save_metadata(dk)
backtest_freqai_training            |   File "/freqtrade/freqtrade/freqai/data_drawer.py", line 457, in save_metadata
backtest_freqai_training            |     dk.data["training_features_list"] = list(dk.data_dictionary["train_features"].columns)
backtest_freqai_training            |                                              ~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^
backtest_freqai_training            | KeyError: 'train_features'

Could you enlight my candle about this please ?

Thanks a lot guys

[xmatthias]

FreqAI currently assumes that the same (full) timerange of data is available for each pair in your pairlist.

If data for one pair starts on 2022-04 - but you backtest from 20220101 onwards - then this will not be working.

This is a known limitation i've pointed out before (someplace) - and i think @robcaulk did start to look at this - but i'm not sure on the process, or if he got anywhere with this.