Error running Reports

[rorymac]

Hello All
I am on the latest stable version, running on parquet.
Everything is pretty stable but I have an error in running reports.
Can anyone shed any light on the error?

Your choice? <RETURN for Back> 61
End date for report?
: Enter date and time in format %Y-%m-%d eg '2024-04-04' OR '%Y-%m-%d %H:%M:%S' eg '2024-04-04 17:19:39'
 OR [Enter an integer to go back N calendar days]OR [Enter a string for period, eg 'YTD', '3M', '2B']OR <RETURN for now>
Start date for report? 
: Enter date and time in format %Y-%m-%d eg '2024-04-04' OR '%Y-%m-%d %H:%M:%S' eg '2024-04-04 17:19:39'
 OR [Enter an integer to go back N calendar days]OR [Enter a string for period, eg 'YTD', '3M', '2B']YTD
1: Print or 2: email or 3: file or 4: email and file? <RETURN for Print> 
2024-04-04 17:19:59 DEBUG Interactive-Diagnostics Running report P&L report sysproduction.reporting.pandl_report.pandl_report console {'calendar_days_back': 1, 'start_date': datetime.datetime(2024, 1, 1, 0, 0), 'end_date': datetime.datetime(2024, 4, 4, 17, 19, 50, 637614)}
2024-04-04 17:19:59 DEBUG config {'type': 'config', 'stage': 'config'} Adding config defaults
Getting p&l for CORN
Traceback (most recent call last):
  File "/home/rorym/pysystemtrade/sysproduction/reporting/api.py", line 360, in pandl_for_instruments_across_strategies
    pandl_for_instruments_across_strategies = getattr(
AttributeError: 'reportingApi' object has no attribute '_pandl_for_instruments_across_strategies'. Did you mean: 'pandl_for_instruments_across_strategies'?

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/rorym/pysystemtrade/sysproduction/reporting/data/pandl.py", line 196, in pandl_for_instrument_across_contracts
    pandl_for_instrument = pandl_store[instrument_code]
KeyError: 'CORN'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/rorym/.local/lib/python3.10/site-packages/pandas/core/indexes/base.py", line 3790, in get_loc
    return self._engine.get_loc(casted_key)
  File "index.pyx", line 152, in pandas._libs.index.IndexEngine.get_loc
  File "index.pyx", line 181, in pandas._libs.index.IndexEngine.get_loc
  File "pandas/_libs/hashtable_class_helper.pxi", line 7080, in pandas._libs.hashtable.PyObjectHashTable.get_item
  File "pandas/_libs/hashtable_class_helper.pxi", line 7088, in pandas._libs.hashtable.PyObjectHashTable.get_item
KeyError: datetime.datetime(2024, 1, 1, 0, 0)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/rorym/pysystemtrade/sysproduction/linux/scripts/run.py", line 66, in <module>
    func(*args, **kwargs)
  File "/home/rorym/pysystemtrade/sysproduction/interactive_diagnostics.py", line 78, in interactive_diagnostics
    menu.run_menu()
  File "/home/rorym/pysystemtrade/syscore/interactive/menus.py", line 64, in run_menu
    method_chosen(*self._args, **self._kwargs)
  File "/home/rorym/pysystemtrade/sysproduction/interactive_diagnostics.py", line 182, in pandl_report
    run_report(report_config, data=data)
  File "/home/rorym/pysystemtrade/sysproduction/reporting/reporting_functions.py", line 63, in run_report
    run_report_with_data_blob(report_config, data)
  File "/home/rorym/pysystemtrade/sysproduction/reporting/reporting_functions.py", line 75, in run_report_with_data_blob
    report_results = run_report_from_config(report_config=report_config, data=data)
  File "/home/rorym/pysystemtrade/sysproduction/reporting/reporting_functions.py", line 85, in run_report_from_config
    report_results = report_function(data, **report_kwargs)
  File "/home/rorym/pysystemtrade/sysproduction/reporting/pandl_report.py", line 52, in pandl_report
    formatted_output.append(func())
  File "/home/rorym/pysystemtrade/sysproduction/reporting/api.py", line 335, in table_pandl_for_instruments_across_strategies
    self.pandl_for_instruments_across_strategies()
  File "/home/rorym/pysystemtrade/sysproduction/reporting/api.py", line 366, in pandl_for_instruments_across_strategies
    self._get_pandl_for_instruments_across_strategies()
  File "/home/rorym/pysystemtrade/sysproduction/reporting/api.py", line 378, in _get_pandl_for_instruments_across_strategies
    self.pandl_calculator.get_ranked_list_of_pandl_by_instrument_all_strategies_in_date_range()
  File "/home/rorym/pysystemtrade/sysproduction/reporting/data/pandl.py", line 90, in get_ranked_list_of_pandl_by_instrument_all_strategies_in_date_range
    self.get_period_perc_pandl_for_all_instruments_all_strategies_in_date_range()
  File "/home/rorym/pysystemtrade/sysproduction/reporting/data/pandl.py", line 120, in get_period_perc_pandl_for_all_instruments_all_strategies_in_date_range
    list_pandl = [
  File "/home/rorym/pysystemtrade/sysproduction/reporting/data/pandl.py", line 123, in <listcomp>
    self.get_period_perc_pandl_for_instrument_all_strategies_in_date_range(
  File "/home/rorym/pysystemtrade/sysproduction/reporting/data/pandl.py", line 180, in get_period_perc_pandl_for_instrument_all_strategies_in_date_range
    pandl_across_contracts = self.pandl_for_instrument_across_contracts(
  File "/home/rorym/pysystemtrade/sysproduction/reporting/data/pandl.py", line 198, in pandl_for_instrument_across_contracts
    pandl_for_instrument = self._get_pandl_for_instrument_across_contracts(
  File "/home/rorym/pysystemtrade/sysproduction/reporting/data/pandl.py", line 221, in _get_pandl_for_instrument_across_contracts
    pandl_df = pandl_df_all_data[self.start_date : self.end_date]
  File "/home/rorym/.local/lib/python3.10/site-packages/pandas/core/frame.py", line 3876, in __getitem__
    return self._getitem_slice(key)
  File "/home/rorym/.local/lib/python3.10/site-packages/pandas/core/generic.py", line 4284, in _getitem_slice
    slobj = self.index._convert_slice_indexer(key, kind="getitem")
  File "/home/rorym/.local/lib/python3.10/site-packages/pandas/core/indexes/base.py", line 4285, in _convert_slice_indexer
    indexer = self.slice_indexer(start, stop, step)
  File "/home/rorym/.local/lib/python3.10/site-packages/pandas/core/indexes/base.py", line 6601, in slice_indexer
    start_slice, end_slice = self.slice_locs(start, end, step=step)
  File "/home/rorym/.local/lib/python3.10/site-packages/pandas/core/indexes/base.py", line 6818, in slice_locs
    start_slice = self.get_slice_bound(start, "left")
  File "/home/rorym/.local/lib/python3.10/site-packages/pandas/core/indexes/base.py", line 6743, in get_slice_bound
    raise err
  File "/home/rorym/.local/lib/python3.10/site-packages/pandas/core/indexes/base.py", line 6737, in get_slice_bound
    slc = self.get_loc(label)
  File "/home/rorym/.local/lib/python3.10/site-packages/pandas/core/indexes/base.py", line 3797, in get_loc
    raise KeyError(key) from err
KeyError: datetime.datetime(2024, 1, 1, 0, 0)

[frankcodespoorly]

Were you able to resolve this? I'm getting similar errors.

I recently updated to parquet and have had issues with migrating historic capital data into parquet. I was able to get the system running by only using fresh capital data pulled from IB from that point forward.

I saw your post in an Issue about how you got historic capital data into parquet by using backup csv data and converting to parquet. I did the same and noticed afterwards that reports involving capital (P&L report; P&L account curve) do not work with the hacked parquet file. The same reports however, do run with the "fresh" capital data parquet file.

[rorymac]

I have not managed to resolve this. I run the system with the pnl reports off in run_reports, not a huge problem as I can recreate from the brokers statement. I do miss the attribution though. I have recreated my pnl parquet files, very carefully, the dates are crucial, you need to have date time , otherwise you canot load into parquet. I have managed to get the pnl reports to run BUT it appears to be random some dates do and some dates don't ( apologies to the Rolling Stones) . I have not figured out a pattern to the dates that do, or don't. I am using the latest iteration of master. I have a clean install, hacked the backup csv files to get my parquet files. All the other reports run. The biggest issue for me is that I cannot stop the old server still using arctic as everything works. So I have had to make serious backups. That remains my production server. Rory Mackay ***@***.*** Sent from Proton Mail Android

…

-------- Original Message --------

On 6/16/24 12:56 PM, frankcodespoorly wrote: Were you able to resolve this? I'm getting similar errors. I recently updated to parquet and have had issues with migrating historic capital data into parquet. I was able to get the system running by only using fresh capital data pulled from IB from that point forward. I saw your post in an Issue about how you got historic capital data into parquet by using backup csv data and converting to parquet. I did the same and noticed afterwards that reports involving capital (P&L report; P&L account curve) do not work with the hacked parquet file. The same reports however, do run with the "fresh" capital data parquet file. — Reply to this email directly, [view it on GitHub](#1368 (comment)), or [unsubscribe](https://github.com/notifications/unsubscribe-auth/AA2DQXZ3VEDA2UDMGSEPKFLZHXGWTAVCNFSM6AAAAABFYDV3P2VHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM4TOOBXHEYTM). You are receiving this because you authored the thread.Message ID: ***@***.***>

[localempire]

Hello Rory,

I also do also a fresh installation of Headless on a Proxmox as a backup solution.
Mongo is also running on a different Kubernetes. It might not be the best setup.

I do get the same error as described but haven´t recovered from backups. Instead, I copied the entire data after two inital csv imports.
I wonder if a real clean installation can succeed without the history of the artic parquet export. It was difficult for me to find the clue on how to get the CSV files to parpuet (Apache Arrow can help?). But this is another story.  
Thanks for the hint about disabling report generation.

Best regards
Ralf

[rorymac]

Ralf If you need it , here is what I used: You do need to install pyarrow. Then in a Jupyter notebook ( to make it easy on yourself! ) load the csv into a df using pandas , then write the df you created into parquet also using pandas. Be sure to create the index as datetime in the df. I hope that helps. Rory Mackay ***@***.*** Sent from Proton Mail Android

…

-------- Original Message --------

On 6/17/24 6:39 AM, localempire wrote: Hello Rory, I also do also a fresh installation of Headless on a Proxmox as a backup solution. Mongo is also running on a different Kubernetes. It might not be the best setup. I do get the same error as described but haven´t recovered from backups. Instead, I copied the entire data after two inital csv imports. I wonder if a real clean installation can succeed without the history of the artic parquet export. It was difficult for me to find the clue on how to get the CSV files to parpuet (Apache Arrow can help?). But this is another story. Thanks for the hint about disabling report generation. Best regards Ralf — Reply to this email directly, [view it on GitHub](#1368 (reply in thread)), or [unsubscribe](https://github.com/notifications/unsubscribe-auth/AA2DQXY6HE5YCBG2D7OSQ7LZH3DIZAVCNFSM6AAAAABFYDV3P2VHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM4TOOJTGYYTM). You are receiving this because you authored the thread.Message ID: ***@***.***>