Caching Behavior for "actions"

[ozcanmetu]

Hello everyone, up until this method, I was caching the requests for info, history and history_metadata satisfactorily. When I try to adapt this method with the same mechanism that I'd done for info, history and history_metadata :

if use_session: 
                
                if use_rateLimiter:

                    session = CachedLimiterSession(
                        limiter=Limiter(RequestRate(2, Duration.SECOND * 5)),  
                        bucket_class=MemoryQueueBucket,
                        backend=SQLiteCache(full_cache_path),
                    )
                else:
                     backend = SQLiteCache(full_cache_path)
                     session = CachedSession(        
                         backend=backend)
                     

                session.headers['User-agent'] = 'my-program/1.0'
            else:
                session = None

It keeps increasing the size of the cache file everytime I execute the script. For others, it did not have such a behavior and was able to use pre-existing cache to fetch data provided that the data has not changed. What could be the reason for this ? I've looked up to this and its sub parts but I could not solve the issue :

requests-cache request-matching

Thanks for your support,

[ValueRaider]

requests_cache caches requests so probably parameters are changing. Run with yf.enable_debug_mode()

[ozcanmetu]

Some of the debug output is like below. I suspected that "period2" was causing the issue because it seemed that it uses current date and time (although my time zone is different from the time that is specified in the debug output) for each execution and I assumed that it may affect the request matching mechanism. I tried to use "ignored_parameters" argument of session with "period2" (and also for period1 afterwards), it had too little affect on execution time but the cache files were still increased in size. Besides, when I try to open the url outputted with debug mode such as : https://query2.finance.yahoo.com/v8/finance/chart/MSFT, it has no content related with what ticker.actions returns as a dataframe:

msft.actions

Dividends	Stock Splits

Date
1987-09-21 00:00:00-04:00 0.00 2.0
1990-04-16 00:00:00-04:00 0.00 2.0
1991-06-27 00:00:00-04:00 0.00 1.5
1992-06-15 00:00:00-04:00 0.00 1.5
1994-05-23 00:00:00-04:00 0.00 2.0
... ... ...
2023-02-15 00:00:00-05:00 0.68 0.0
2023-05-17 00:00:00-04:00 0.68 0.0
2023-08-16 00:00:00-04:00 0.68 0.0
2023-11-15 00:00:00-05:00 0.75 0.0
2024-02-14 00:00:00-05:00 0.75 0.0

The debug output is :

Script is running from VSCode.
Processing chunk 1 of 10 with 5 quotes
DEBUG Entering history()
DEBUG MSFT: Yahoo GET parameters: {'period1': '1925-05-29 20:00:00-04:00', 'period2': '2024-05-05 05:39:45-04:00', 'interval': '1d', 'includePrePost': False, 'events': 'div,splits,capitalGains'}
DEBUG Entering get()
DEBUG url=https://query2.finance.yahoo.com/v8/finance/chart/MSFT
DEBUG params={'period1': -1407196800, 'period2': 1714901985, 'interval': '1d', 'includePrePost': False, 'events': 'div,splits,capitalGains'}
DEBUG Entering _get_cookie_and_crumb()
DEBUG cookie_mode = 'basic'
DEBUG Entering _get_cookie_and_crumb_basic()
DEBUG reusing cookie
DEBUG reusing crumb
DEBUG Exiting _get_cookie_and_crumb_basic()
DEBUG Exiting _get_cookie_and_crumb()
DEBUG response code=200
DEBUG Exiting get()
DEBUG MSFT: yfinance received OHLC data: 1986-03-13 14:30:00 -> 2024-05-03 20:00:01
DEBUG MSFT: OHLC after cleaning: 1986-03-13 09:30:00-05:00 -> 2024-05-03 16:00:01-04:00
DEBUG MSFT: OHLC after combining events: 1986-03-13 00:00:00-05:00 -> 2024-05-03 00:00:00-04:00
DEBUG MSFT: yfinance returning OHLC: 1986-03-13 00:00:00-05:00 -> 2024-05-03 00:00:00-04:00
DEBUG Exiting history()
DEBUG Entering history()
DEBUG AAPL: Yahoo GET parameters: {'period1': '1925-05-29 20:00:00-04:00', 'period2': '2024-05-05 05:39:45-04:00', 'interval': '1d', 'includePrePost': False, 'events': 'div,splits,capitalGains'}
DEBUG Entering get()
DEBUG url=https://query2.finance.yahoo.com/v8/finance/chart/AAPL
DEBUG params={'period1': -1407196800, 'period2': 1714901985, 'interval': '1d', 'includePrePost': False, 'events': 'div,splits,capitalGains'}
DEBUG Entering _get_cookie_and_crumb()
DEBUG cookie_mode = 'basic'
DEBUG Entering _get_cookie_and_crumb_basic()
DEBUG reusing cookie
DEBUG reusing crumb
DEBUG Exiting _get_cookie_and_crumb_basic()
DEBUG Exiting _get_cookie_and_crumb()
DEBUG response code=200
DEBUG Exiting get()
DEBUG AAPL: yfinance received OHLC data: 1980-12-12 14:30:00 -> 2024-05-03 20:00:07
DEBUG AAPL: OHLC after cleaning: 1980-12-12 09:30:00-05:00 -> 2024-05-03 16:00:07-04:00
DEBUG AAPL: OHLC after combining events: 1980-12-12 00:00:00-05:00 -> 2024-05-03 00:00:00-04:00
DEBUG AAPL: yfinance returning OHLC: 1980-12-12 00:00:00-05:00 -> 2024-05-03 00:00:00-04:00
DEBUG Exiting history()
DEBUG Entering history()
DEBUG NVDA: Yahoo GET parameters: {'period1': '1925-05-29 20:00:00-04:00', 'period2': '2024-05-05 05:39:45-04:00', 'interval': '1d', 'includePrePost': False, 'events': 'div,splits,capitalGains'}
DEBUG Entering get()
DEBUG url=https://query2.finance.yahoo.com/v8/finance/chart/NVDA
DEBUG params={'period1': -1407196800, 'period2': 1714901985, 'interval': '1d', 'includePrePost': False, 'events': 'div,splits,capitalGains'}
DEBUG Entering _get_cookie_and_crumb()
DEBUG cookie_mode = 'basic'
DEBUG Entering _get_cookie_and_crumb_basic()
DEBUG reusing cookie
DEBUG reusing crumb
DEBUG Exiting _get_cookie_and_crumb_basic()
DEBUG Exiting _get_cookie_and_crumb()
DEBUG response code=200
DEBUG Exiting get()
DEBUG NVDA: yfinance received OHLC data: 1999-01-22 14:30:00 -> 2024-05-03 20:00:07
DEBUG NVDA: OHLC after cleaning: 1999-01-22 09:30:00-05:00 -> 2024-05-03 16:00:07-04:00
DEBUG NVDA: OHLC after combining events: 1999-01-22 00:00:00-05:00 -> 2024-05-03 00:00:00-04:00
DEBUG NVDA: yfinance returning OHLC: 1999-01-22 00:00:00-05:00 -> 2024-05-03 00:00:00-04:00
DEBUG Exiting history()
DEBUG Entering history()
DEBUG GOOGL: Yahoo GET parameters: {'period1': '1925-05-29 20:00:00-04:00', 'period2': '2024-05-05 05:39:45-04:00', 'interval': '1d', 'includePrePost': False, 'events': 'div,splits,capitalGains'}
DEBUG Entering get()
DEBUG url=https://query2.finance.yahoo.com/v8/finance/chart/GOOGL
DEBUG params={'period1': -1407196800, 'period2': 1714901985, 'interval': '1d', 'includePrePost': False, 'events': 'div,splits,capitalGains'}
DEBUG Entering _get_cookie_and_crumb()
DEBUG cookie_mode = 'basic'
DEBUG Entering _get_cookie_and_crumb_basic()
DEBUG reusing cookie
DEBUG reusing crumb
DEBUG Exiting _get_cookie_and_crumb_basic()
DEBUG Exiting _get_cookie_and_crumb()
DEBUG response code=200
DEBUG Exiting get()
DEBUG GOOGL: yfinance received OHLC data: 2004-08-19 13:30:00 -> 2024-05-03 20:00:07
DEBUG GOOGL: OHLC after cleaning: 2004-08-19 09:30:00-04:00 -> 2024-05-03 16:00:07-04:00
DEBUG GOOGL: OHLC after combining events: 2004-08-19 00:00:00-04:00 -> 2024-05-03 00:00:00-04:00
DEBUG GOOGL: yfinance returning OHLC: 2004-08-19 00:00:00-04:00 -> 2024-05-03 00:00:00-04:00
DEBUG Exiting history()
DEBUG Entering history()
DEBUG AMZN: Yahoo GET parameters: {'period1': '1925-05-29 20:00:00-04:00', 'period2': '2024-05-05 05:39:45-04:00', 'interval': '1d', 'includePrePost': False, 'events': 'div,splits,capitalGains'}
DEBUG Entering get()
DEBUG url=https://query2.finance.yahoo.com/v8/finance/chart/AMZN
DEBUG params={'period1': -1407196800, 'period2': 1714901985, 'interval': '1d', 'includePrePost': False, 'events': 'div,splits,capitalGains'}
DEBUG Entering _get_cookie_and_crumb()
DEBUG cookie_mode = 'basic'
DEBUG Entering _get_cookie_and_crumb_basic()
DEBUG reusing cookie
DEBUG reusing crumb
DEBUG Exiting _get_cookie_and_crumb_basic()
DEBUG Exiting _get_cookie_and_crumb()
DEBUG response code=200
DEBUG Exiting get()
DEBUG AMZN: yfinance received OHLC data: 1997-05-15 13:30:00 -> 2024-05-03 20:00:08
DEBUG AMZN: OHLC after cleaning: 1997-05-15 09:30:00-04:00 -> 2024-05-03 16:00:08-04:00
DEBUG AMZN: OHLC after combining events: 1997-05-15 00:00:00-04:00 -> 2024-05-03 00:00:00-04:00
DEBUG AMZN: yfinance returning OHLC: 1997-05-15 00:00:00-04:00 -> 2024-05-03 00:00:00-04:00
DEBUG Exiting history()
Processing chunk 2 of 10 with 5 quotes
DEBUG Entering history()
DEBUG META: Yahoo GET parameters: {'period1': '1925-05-29 20:00:00-04:00', 'period2': '2024-05-05 05:39:46-04:00', 'interval': '1d', 'includePrePost': False, 'events': 'div,splits,capitalGains'}
DEBUG Entering get()
DEBUG url=https://query2.finance.yahoo.com/v8/finance/chart/META
DEBUG params={'period1': -1407196800, 'period2': 1714901986, 'interval': '1d', 'includePrePost': False, 'events': 'div,splits,capitalGains'}
DEBUG Entering _get_cookie_and_crumb()
DEBUG cookie_mode = 'basic'
DEBUG Entering _get_cookie_and_crumb_basic()
DEBUG reusing cookie
DEBUG reusing crumb
DEBUG Exiting _get_cookie_and_crumb_basic()
DEBUG Exiting _get_cookie_and_crumb()
DEBUG response code=200
DEBUG Exiting get()
DEBUG META: yfinance received OHLC data: 2012-05-18 13:30:00 -> 2024-05-03 20:00:08
DEBUG META: OHLC after cleaning: 2012-05-18 09:30:00-04:00 -> 2024-05-03 16:00:08-04:00
DEBUG META: OHLC after combining events: 2012-05-18 00:00:00-04:00 -> 2024-05-03 00:00:00-04:00
DEBUG META: yfinance returning OHLC: 2012-05-18 00:00:00-04:00 -> 2024-05-03 00:00:00-04:00
DEBUG Exiting history()
DEBUG Entering history()
DEBUG BRK-B: Yahoo GET parameters: {'period1': '1925-05-29 20:00:00-04:00', 'period2': '2024-05-05 05:39:46-04:00', 'interval': '1d', 'includePrePost': False, 'events': 'div,splits,capitalGains'}
DEBUG Entering get()
DEBUG url=https://query2.finance.yahoo.com/v8/finance/chart/BRK-B
DEBUG params={'period1': -1407196800, 'period2': 1714901986, 'interval': '1d', 'includePrePost': False, 'events': 'div,splits,capitalGains'}
DEBUG Entering _get_cookie_and_crumb()
DEBUG cookie_mode = 'basic'
DEBUG Entering _get_cookie_and_crumb_basic()
DEBUG reusing cookie
DEBUG reusing crumb
DEBUG Exiting _get_cookie_and_crumb_basic()
DEBUG Exiting _get_cookie_and_crumb()
DEBUG response code=200
DEBUG Exiting get()
DEBUG BRK-B: yfinance received OHLC data: 1996-05-09 13:30:00 -> 2024-05-03 20:01:48
DEBUG BRK-B: OHLC after cleaning: 1996-05-09 09:30:00-04:00 -> 2024-05-03 16:01:48-04:00
DEBUG BRK-B: OHLC after combining events: 1996-05-09 00:00:00-04:00 -> 2024-05-03 00:00:00-04:00
DEBUG BRK-B: yfinance returning OHLC: 1996-05-09 00:00:00-04:00 -> 2024-05-03 00:00:00-04:00
DEBUG Exiting history()
DEBUG Entering history()
DEBUG LLY: Yahoo GET parameters: {'period1': '1925-05-29 20:00:00-04:00', 'period2': '2024-05-05 05:39:46-04:00', 'interval': '1d', 'includePrePost': False, 'events': 'div,splits,capitalGains'}
DEBUG Entering get()
DEBUG url=https://query2.finance.yahoo.com/v8/finance/chart/LLY
DEBUG params={'period1': -1407196800, 'period2': 1714901986, 'interval': '1d', 'includePrePost': False, 'events': 'div,splits,capitalGains'}
DEBUG Entering _get_cookie_and_crumb()
DEBUG cookie_mode = 'basic'
DEBUG Entering _get_cookie_and_crumb_basic()
DEBUG reusing cookie
DEBUG reusing crumb
DEBUG Exiting _get_cookie_and_crumb_basic()
DEBUG Exiting _get_cookie_and_crumb()
DEBUG response code=200
DEBUG Exiting get()
DEBUG LLY: yfinance received OHLC data: 1972-06-01 13:30:00 -> 2024-05-03 20:00:02
DEBUG LLY: OHLC after cleaning: 1972-06-01 09:30:00-04:00 -> 2024-05-03 16:00:02-04:00
DEBUG LLY: OHLC after combining events: 1972-06-01 00:00:00-04:00 -> 2024-05-03 00:00:00-04:00
DEBUG LLY: yfinance returning OHLC: 1972-06-01 00:00:00-04:00 -> 2024-05-03 00:00:00-04:00
DEBUG Exiting history()
DEBUG Entering history()
DEBUG TSM: Yahoo GET parameters: {'period1': '1925-05-29 20:00:00-04:00', 'period2': '2024-05-05 05:39:46-04:00', 'interval': '1d', 'includePrePost': False, 'events': 'div,splits,capitalGains'}
DEBUG Entering get()
DEBUG url=https://query2.finance.yahoo.com/v8/finance/chart/TSM
DEBUG params={'period1': -1407196800, 'period2': 1714901986, 'interval': '1d', 'includePrePost': False, 'events': 'div,splits,capitalGains'}
DEBUG Entering _get_cookie_and_crumb()
DEBUG cookie_mode = 'basic'
DEBUG Entering _get_cookie_and_crumb_basic()
DEBUG reusing cookie
DEBUG reusing crumb
DEBUG Exiting _get_cookie_and_crumb_basic()
DEBUG Exiting _get_cookie_and_crumb()
DEBUG response code=200
DEBUG Exiting get()
DEBUG TSM: yfinance received OHLC data: 1997-10-09 13:30:00 -> 2024-05-03 20:00:02
DEBUG TSM: OHLC after cleaning: 1997-10-09 09:30:00-04:00 -> 2024-05-03 16:00:02-04:00
DEBUG TSM: OHLC after combining events: 1997-10-09 00:00:00-04:00 -> 2024-05-03 00:00:00-04:00
DEBUG TSM: yfinance returning OHLC: 1997-10-09 00:00:00-04:00 -> 2024-05-03 00:00:00-04:00
DEBUG Exiting history()
DEBUG Entering history()
DEBUG AVGO: Yahoo GET parameters: {'period1': '1925-05-29 20:00:00-04:00', 'period2': '2024-05-05 05:39:46-04:00', 'interval': '1d', 'includePrePost': False, 'events': 'div,splits,capitalGains'}
DEBUG Entering get()
DEBUG url=https://query2.finance.yahoo.com/v8/finance/chart/AVGO
DEBUG params={'period1': -1407196800, 'period2': 1714901986, 'interval': '1d', 'includePrePost': False, 'events': 'div,splits,capitalGains'}
DEBUG Entering _get_cookie_and_crumb()
DEBUG cookie_mode = 'basic'
DEBUG Entering _get_cookie_and_crumb_basic()
DEBUG reusing cookie
DEBUG reusing crumb
DEBUG Exiting _get_cookie_and_crumb_basic()
DEBUG Exiting _get_cookie_and_crumb()
DEBUG response code=200
DEBUG Exiting get()
DEBUG AVGO: yfinance received OHLC data: 2009-08-06 13:30:00 -> 2024-05-03 20:00:01
DEBUG AVGO: OHLC after cleaning: 2009-08-06 09:30:00-04:00 -> 2024-05-03 16:00:01-04:00
DEBUG AVGO: OHLC after combining events: 2009-08-06 00:00:00-04:00 -> 2024-05-03 00:00:00-04:00
DEBUG AVGO: yfinance returning OHLC: 2009-08-06 00:00:00-04:00 -> 2024-05-03 00:00:00-04:00
DEBUG Exiting history()

Note:

If I enable debug mode while executing for info, history or history_metadata, the debug output instantly appears on vscode's interactive window and by looking this, it seems to be able to use existing cache file as intended. However, for "actions", it is obvious that it processes something and it outputs the regarding information in pieces. Thus, I could not find the root cause of this.