Custom cache for time series

[bekirisgor]

Hey all,

Currently, I am working on an analytic dashboard and i use heavily time series depend on picked date range.
For Example, request

I wonder if custom cache is possible. If not what would you do for this situation,

I have already fetched 01.08.23 - 01.11.23 and as user action add this range to data 01.07.23- 01.08.2023

[zdenkolini]

Kind of a late answer, but found this discussion only recently.

I implemented this functionality by having a main query for fetching initial data, then expanding the data based on datefrom and dateto parameters.
This data is then returned with a useQueries that segregates the data by month and permanently saves to cache.

The code isn't ideal (there are some edge cases that come to mind), but I think that it would just be better to rewrite the expanding fetch mechanism.

One caveat to my implementation is that if the user selects a range that doesn't include the previous data, the main query will fetch data between the new specified range, and the previous date params. This is an optimization that I felt like implementing, as skipping the data between would require additional metadata regarding what is currently in the cache to be stored somewhere, and additional code to determine which chunks of data are missing from said cache.

Any comments on this code are highly encouraged.

import {
  QueryKey,
  UseQueryOptions,
  useQueries,
  useQuery,
  useQueryClient,
} from "@tanstack/react-query";
import { endOfMonth, startOfMonth } from "date-fns";
import addMonths from "date-fns/addMonths";
import differenceInCalendarMonths from "date-fns/differenceInCalendarMonths";
import isAfter from "date-fns/isAfter";
import isBefore from "date-fns/isBefore";
import { useMemo } from "react";

import { Api } from "@/lib/api";
import {
  BillingCostReportParams,
  BillingCostReportResponse,
} from "@/lib/api/dto";
import { removeDateOffset } from "@/lib/dateUtils";

import { billingReportKeys } from "./queryKeys";

interface UseFetchBillingCostReportQueryOptions {
  params?: Partial<BillingCostReportParams>;
}

const fetchBillingCost = ({
  datefrom,
  dateto,
  subscriptionid,
}: Pick<BillingCostReportParams, "datefrom" | "dateto" | "subscriptionid">) => {
  return !datefrom || !dateto || !subscriptionid
    ? Promise.resolve([])
    : Api.billingReports
        .costReport({
          datefrom: removeDateOffset(startOfMonth(datefrom)).getTime(),
          dateto: removeDateOffset(endOfMonth(dateto)).getTime(),
          subscriptionid,
        })
        .then((res) => res.data);
};

export const useFetchBillingCostReportQuery = <
  T = BillingCostReportResponse[]
>({
  params,
}: UseFetchBillingCostReportQueryOptions) => {
  const queryClient = useQueryClient();
  const parsedDateFrom = useMemo(
    () =>
      params?.datefrom &&
      removeDateOffset(startOfMonth(params.datefrom)).getTime(),
    [params?.datefrom]
  );
  const parsedDateTo = useMemo(
    () =>
      params?.dateto && removeDateOffset(endOfMonth(params.dateto)).getTime(),
    [params?.dateto]
  );
  const previousQuery = useMemo(() => {
    const previousQueries = queryClient.getQueriesData<
      BillingCostReportResponse[]
    >({
      queryKey: billingReportKeys.costReportBase(),
    });

    return previousQueries.reduce<
      [QueryKey, BillingCostReportResponse[] | undefined] | null
    >((prevVal, currVal) => {
      const currQueryKeyParams = currVal[0][1] as BillingCostReportParams;
      if (!prevVal) {
        return currQueryKeyParams.subscriptionid === params?.subscriptionid
          ? currVal
          : prevVal;
      }

      const prevQueryState = queryClient.getQueryState(prevVal[0]);
      const currQueryState = queryClient.getQueryState(currVal[0]);

      if (prevQueryState?.dataUpdatedAt && currQueryState?.dataUpdatedAt) {
        return prevQueryState.dataUpdatedAt > currQueryState.dataUpdatedAt
          ? prevVal
          : currVal;
      }

      return prevVal;
    }, null);
  }, [params?.subscriptionid, queryClient]);

  const previousQueryParams = useMemo(() => {
    return previousQuery?.[0][1] as BillingCostReportParams;
  }, [previousQuery]);

  const newParams = useMemo(
    () => ({
      datefrom: Math.min(
        parsedDateFrom!,
        previousQueryParams?.datefrom || Infinity
      ),
      dateto: Math.max(parsedDateTo!, previousQueryParams?.dateto || -Infinity),
      subscriptionid: params?.subscriptionid,
    }),
    [parsedDateFrom, previousQueryParams, parsedDateTo, params?.subscriptionid]
  );

  const fullDataQuery = useQuery({
    queryKey: billingReportKeys.costReport(newParams),
    queryFn: async ({ queryKey: [_, keyParams] }) => {
      if (
        !keyParams?.datefrom ||
        !keyParams?.dateto ||
        !keyParams?.subscriptionid
      ) {
        return Promise.resolve(null);
      }

      const retVal =
        (await queryClient.ensureQueryData<BillingCostReportResponse[]>({
          queryKey: billingReportKeys.costReport({
            datefrom: previousQueryParams.datefrom,
            dateto: previousQueryParams.dateto,
            subscriptionid: keyParams.subscriptionid,
          }),
          queryFn: () =>
            fetchBillingCost({
              datefrom: previousQueryParams.datefrom!,
              dateto: previousQueryParams.dateto!,
              subscriptionid: keyParams.subscriptionid!,
            }),
        })) || [];

      const [previousData, nextData] = await Promise.all([
        keyParams.datefrom !== previousQueryParams.datefrom &&
        isBefore(keyParams.datefrom, previousQueryParams.datefrom)
          ? fetchBillingCost({
              datefrom: keyParams.datefrom,
              dateto: previousQueryParams.datefrom - 1,
              subscriptionid: keyParams.subscriptionid,
            })
          : null,
        keyParams.dateto !== previousQueryParams.dateto &&
        isAfter(keyParams.dateto, previousQueryParams.dateto)
          ? fetchBillingCost({
              datefrom: previousQueryParams.dateto + 1,
              dateto: keyParams.dateto,
              subscriptionid: keyParams.subscriptionid,
            })
          : null,
      ]);

      if (previousData) {
        retVal.unshift(...previousData);
      }

      if (nextData) {
        retVal.push(...nextData);
      }

      return retVal;
    },
  });

  const monthsArray = Array.from(
    { length: differenceInCalendarMonths(parsedDateTo!, parsedDateFrom!) },
    (_, index) => {
      const date = addMonths(parsedDateFrom!, index);
      const month = date.getMonth() + 1;
      const year = date.getFullYear();

      return {
        month: month.toString(),
        year: year.toString(),
      };
    }
  );

  return useQueries({
    queries: monthsArray.map<CostReportFormMonthOptions>(({ month, year }) => {
      const queryKey = billingReportKeys.costReportForMonth({
        month,
        year,
        subscriptionid: params!.subscriptionid!,
      });

      return {
        enabled: fullDataQuery.isSuccess,
        staleTime: Infinity,
        queryKey,
        queryFn: ({ queryKey: [_, keyParams] }) =>
          fullDataQuery.data?.filter(
            (item) =>
              item.year === keyParams.year && item.month === keyParams.month
          ) || [],
      };
    }),
  });

  type CostReportFormMonthOptions = UseQueryOptions<
    BillingCostReportResponse[],
    Error,
    T,
    ReturnType<typeof billingReportKeys.costReportForMonth>
  >;
};

[spigelli]

There are so many features Ive found myself looking for when it comes to time series.

What if we have multiple resolutions for a metric and want to automatically return low resolution initial data.

Or always start with lowest resolution sampling and always progressively become more detailed (just like map tiles).

What if we want to automatically batch overlapping ranges or ranges that are only separated by at most some duration threshold?

What if we want to take in a time range for some metric and automatically divide it into parallelized queries based on some configurable logic.

Some metrics might never become stale (append only) and others might reset.

Automatic windowing based on the actual pixel width of some graph that gets displayed.

There are also some time series paradigms (counters vs gauges). Where some library could provide additional helpers.

Time series also really lends itself to streaming so what if we wanted these things to support streaming in records one at a time as they update state (currently experimental in react query).