Questions on PST Backtesting

[anggs1996]

Hi all

I've spent the last couple of months reading AFTS and trying to set up PST. I've been trying to keep up with the updates implemented but have noticed that some segments of the document provided by Rob are outdated.

I believe that I have a decent understanding of the data implementation in PST (i.e. contract sampling, historical updates) and have already set these up. However, I would appreciate it if someone could answer a few of my questions regarding backtest and production:

1. Bad markets and duplicate/ignore instrument: Referring to this post. It seems that the 'old' method of determining highly corr instruments have been rescinded since the option is no longer available. Maybe I've missed something (I hope not), but can someone guide me on how I can update my list on my private_config.yaml ? I have close to 250 instruments, manual steps to update this portion is not desirable.

2. Fixed weights of instruments: Referencing to this link to the config.yaml in the provided rob_system. Based on my understanding (and from the doc), Rob periodically runs a dynamic estimates backtest system but typically deploys his strategy using fixed weights of instruments. Is there an easy way for me to do the same? (i.e. run my backtest via dynamic estimators, and update my fixed weights, scalar etc in the config file). I've tried to follow this portion of the backtesting document but the saved config file does not seem useful to recreate the weight estimate needed of the config yaml - all i got was a chunk of code that is a spit back from my initial yaml. Is there a way for me to recreate what rob did? And am i right in understanding that this will then be used by any one of these order/positions optimisors in creating an Live trading order?

3. Saving of backtesting system: Referring to this portion of the doc on saving cache. I've managed to saved my 'trained' system as a pickle file using the below code. However, the system reruns the portfolio daily pandl everytime. Is this normal? I've used the .get_itemnames_for_stage for the below. Are there any other stages that is saved in the cache?

    system = futures_system()
  
    # Read from pickle file
    system.cache.unpickle("private.systems.ewmac_strategy.cache.system.pck")
    system.cache.get_items_with_data() ## Cache is now populated. Any existing data in system instance would have been removed.
    
    system.cache.get_itemnames_for_stage("accounts")
    system.cache.get_itemnames_for_stage("portfolio")
    system.cache.get_itemnames_for_stage("percentageTdelayfillTroundpositionsT")

    print("\n########### PORTFOLIO STATS ############\n")
    print(system.accounts.portfolio().stats()) 

I recognise that I have covered quite a wide range of content. But would appreciate if someone could help me answer my queries!!

[bug-or-feature]

Lots of questions there, but I can answer the first one, as it is easy. Either

• run the Remove Markets Report and the Duplicate Markets Report, and follow the advice there, or
• look at Rob's version of those reports, which are auto published every day

[anggs1996]

Thanks a lot Andy. Got it! Knew I was missing something important.

[tgibson11]

I will answer the second one.

I've tried to follow this portion of the backtesting document but the saved config file does not seem useful to recreate the weight estimate needed of the config yaml - all i got was a chunk of code that is a spit back from my initial yaml.

It is only hinted at in the docs you linked to, but you need to do this before running the backtest:

system.config.use_forecast_scale_estimates = True
system.config.use_forecast_weight_estimates = True
system.config.use_forecast_div_mult_estimates = True
system.config.use_instrument_weight_estimates = True
system.config.use_instrument_div_mult_estimates = True

True will do estimates, False will use fixed values from config.

With 250 instruments, I hope you have a powerful machine (or a lot of patience).

Some other things that might help you:

• Despite having written a lot about estimating instrument weights, for his own system, I believe Rob has always used hand-crafted weights, not estimated. (I could be wrong though.)

• Rob is now using a dynamically optimized system, and you'll want to also with 250 instruments, unless you have multiple millions of trading capital. It's difficult to describe in a few words what that means. The code to do it is all in pysystemtrade. I'd recommend the following blog posts:

Rob's actual system (at least as of December 2021):
https://qoppac.blogspot.com/2021/12/my-trading-system.html

Dynamic optimization:
https://qoppac.blogspot.com/2021/10/mr-greedy-and-tale-of-minimum-tracking.html
https://qoppac.blogspot.com/2021/11/mr-greedy-and-tale-of-minimum-tracking.html

[anggs1996]

Hi @tgibson11

It is only hinted at in the docs you linked to, but you need to do this before running the backtest:

system.config.use_forecast_scale_estimates = True
system.config.use_forecast_weight_estimates = True
system.config.use_forecast_div_mult_estimates = True
system.config.use_instrument_weight_estimates = True
system.config.use_instrument_div_mult_estimates = True

True will do estimates, False will use fixed values from config.

My apologies. Maybe i've used the 'term' dynamic optimizor too interchangeably. Regarding the above portion - I am aware that these estimates have to be set to True for the estimator to run. What i was referring to was the config results of the estimators.

I.e. My question was rather - after i run the estimator - the system should have came up with a fixed value of scalar, weights and idm estimates (at least at that point) - which I reckon Rob extracts somehow. Is there an easy way to export the information similar to what rob has done here?

'#Example
forecast_scalars:
accel16: 7.8170710605387095
accel32: 5.563487137713779
accel64: 3.896720541225276
assettrend16: 3.549452858682833
assettrend2: 10.846520114531351
assettrend32: 2.3449234496490723
.....
and
instrument_div_multiplier: 2.75
risk_overlay:
max_risk_fraction_normal_risk: 1.75
max_risk_fraction_stdev_risk: 4.0
max_risk_limit_sum_abs_risk: 4.0
max_risk_leverage: 20.0
instrument_weights:
AEX: 0.00673
ALUMINIUM: 0.0114
AUD: 0.00509
AUDJPY: 0.00533

What I want to do is only run the estimator periodically (maybe weekly/adhoc basis) and keep a fixed weight and estimate for live trading production daily. I imagine Rob has probably came up with a way to extract this after each estimation (but I could be wrong).

P.s. Im not trading and backtesting on 250 instruments - I only loaded them and kept them up-to-date. I was hoping that my question in part 1 allow me to filter down these instruments.

[tgibson11]

I assume you are doing this:

sysdiag = systemDiag(system)
sysdiag.yaml_config_with_estimated_parameters('someyamlfile.yaml')

If so, someyamlfile.yaml will contain the estimated parameters, suitable for copying & pasting into your system config file. I just re-estimated some FDMs a few days ago, so it definitely works if everything is configured properly.

What you described "all i got was a chunk of code that is a spit back from my initial yaml" is what you would see if the flags I mentioned were set to false.

What I want to do is only run the estimator periodically (maybe weekly/adhoc basis) and keep a fixed weight and estimate for live trading production daily. I imagine Rob has probably came up with a way to extract this after each estimation (but I could be wrong).

P.s. Im not trading and backtesting on 250 instruments - I only loaded them and kept them up-to-date. I was hoping that my question in part 1 allow me to filter down these instruments.

This is possible, but weekly is (IMO) far too often to re-estimate parameters. It sounds like an attempt at over-optimization, but actually just a lot of unnecessary work, as the parameters are barely going to change on that timescale (unless you're using a very short window for the estimates, in which case it is definitely over-optimization).

I don't recall that Rob has ever said how often he updates his parameter estimates, but I suspect it is somewhere between "several years" and "never".

[anggs1996]

@tgibson11 Thanks a ton! Have investigating my code further, i realised i've saved the yaml file elsewhere in my script. So Q2 is resolved.

This is possible, but weekly is (IMO) far too often to re-estimate parameters. It sounds like an attempt at over-optimization, but actually just a lot of unnecessary work, as the parameters are barely going to change on that timescale (unless you're using a very short window for the estimates, in which case it is definitely over-optimization).

I don't recall that Rob has ever said how often he updates his parameter estimates, but I suspect it is somewhere between "several years" and "never".

That makes a lot of sense. I was imagining that the instrument weight is dynamic and could change on a weekly basis. But after second thought, it seems like the portfolio weight of the instrument is unlikely to change too drastically due to the length of the window (correct me if im wrong). This is very helpful.

[tgibson11]

after second thought, it seems like the portfolio weight of the instrument is unlikely to change too drastically due to the length of the window

Correct, but I think maybe you haven't realized the bigger (but related) problem: deciding which instruments to include in your system in the first place.

You'd like to include as many as possible for the diversification benefits, as well as just not missing big trends in some instrument that you didn't include. But the more instruments you include, the smaller position you can take in each instrument. And you need to be able to take a position of at least 1 full contract in each instrument (and I think Rob has concluded that ~3 contracts is actually the minimum before you should consider adding an instrument).

Dynamic optimization solves this by having an "ideal" portfolio with fixed weights for a large number of instruments. Then it figures out the "practical" portfolio that comes closest to tracking the "ideal", but taking into account existing positions, trading costs, and of course that you can't trade factional contracts.

[bug-or-feature]

No. 3: Yes, it's normal that daily pandl is recalculated each time. P&L values are not cached. See systems.accounts.accountInstruments.pandl_for_instrument() - it is decorated by @dont_cache

[anggs1996]

Gotcha, Thanks! @bug-or-feature

[bug-or-feature]

@anggs1996 it would be really helpful if you could note any areas where the docs are missing or outdated. Please add any notes to the ticket here