semi-endless loop around add_candle call #447
Comments
|
@movy please pull the new master version and try it out. can you also tell me. what tf your example was? the number of routes and the simulation start / end time? |
|
Thanks for prompt response, I import candles for ~6 months, then split them into 40 days periods and get candles for each period. Basic code: for index, date_range in enumerate(backtest_dates, start=0):
print(f"Importing candles for {date_range[0]} - {date_range[1]}...")
try:
candles_warmup, candles_period = research.get_candles(exchange, symbol, tf, int((datetime.strptime(date_range[0], '%Y-%m-%d')).timestamp() * 1000), int((datetime.strptime(date_range[1], '%Y-%m-%d')).timestamp() * 1000), warmup_candles_num=config["warm_up_candles"], caching=True, is_for_jesse=True)
period_candles = {
jh.key(exchange, symbol): {
'exchange': exchange,
'symbol': symbol,
# backtest candles tf is always 1m
'candles': candles_period
}
}
warmup_candles = {
jh.key(exchange, symbol): {
'exchange': exchange,
'symbol': symbol,
'candles': candles_warmup
}
}
candles_refs.append(ray.put(period_candles))
# print(candles_refs)
warmup_candles_refs.append(ray.put(warmup_candles))
[...]
for candles_ref in candles_refs:
result = backtest(config=ray.get(config_ref),
routes=ray.get(routes_refs[params["tf"]]),
extra_routes=ray.get(extra_routes_refs[anchor_timeframe(params["tf"])]),
candles=ray.get(candles_ref), # candles within backtest date range
warmup_candles=ray.get(warmup_candles_refs[candles_refs.index(candles_ref)]),
generate_csv=False,
generate_json=True,
hyperparameters=params,
fast_mode=False)Will try master branch now. |
|
the higher the timeframe you use the better theperformance of fast_mose. |
|
Hey Movy, I pushed the patch that Yakir is talking about. It is version 0.48.1. Please give it a try and see if it fixes your issue. |
|
So, good news and bad news: this regression was not caused by 0.48 (i.e. 0.48.1. did not fix it). The last version that does not hang is 0.45, so I'm sticking to it for now. I guess some change related to p.s. inconsistent params names 'warm_up_candles' vs 'warmup_candles' gave me some avoidable pain while I was refactoring |
|
@movy |
|
Yes, I cancel on every candle if order was not executed. This never was a problem prior to 0.46. |
|
Thank you, @yakir4123 , I tried your branch, unfort same result. I will try to create a small script that consistently reproduces problematic behaviour, will post it here later. |
|
@movy , Yeah im facing the same problemim trying to solve it. I know the root of the problem. I guess i need to go back to v0.46 to see what was there |
|
0.45 is the final version working w/o hanging, 0.46 introduced this bug |
|
@movy https://github.com/yakir4123/jesse/tree/yakir/feat/delete-orders |
|
@yakir4123 , still hangs unfort. |
|
@movy |
|
Last commit I tested was 5ed48de |
|
:/ I wonder what is it, please give me a simple strategy that repreduce that ill probably find the problem |
|
Here's one of Jesse's default strategies. I added trailing take-profit and offset-entries (i.e. entry not at current price, but at Tested on multiple coins and 4h candles for speed, same result. I run tests using Ray.tune (to enable more complex hyperparams perturbations and schedulers), but this particular code should run with jesse out of the box. |
Describe the bug
Until this week I'd been running 0.44 w/o any problems, but 0.48 added some welcome changes, so I switched my backtesting to 0.48 and encountered a very long loop around https://github.com/jesse-ai/jesse/blob/master/jesse/store/state_candles.py#L102 call, which sometimes causes a backtest to hang for 10-15 minutes. This happens sporadically, maybe one out of 100-200 backtests run, but it effectively it blocks the whole pipeline until problematic test is finished with this loop.
This happens on all symbols and tfs.
When I say 'around' it means impossible to track down exact line that hangs, but what I could gather via py-spy is:
I've added some logging:
And from these
prints we can see this method is called millions of times by a single backtest process:Eventually number of calls falls:
Which makes me think there's loop inside loop somewhere.
I could not instantly pinpoint recent changes responsible for this. Will roll back to 0.44 for now and slowly work my way to 0.48, maybe I can find where regression first appeared.
Any help from those familiar with recent changes is appreciated @yakir4123
The text was updated successfully, but these errors were encountered: