Table manger creating invalid tables for old configs

[littlepangdi]

Describe the bug

Premise：

Table manager provides a periodic table schema, which allows to have multiple non-overlapping schema configs over the time.

In our project, we find that if you put new entry to schema, retention_period does not work as expected, and table manager will create tables for old config. As result, retention_period * num_of_config will be retained.

To Reproduce
Steps to reproduce the behavior:

1. Started Loki with config below(only list key parameters here)

schema_config:
  configs:
    - from: 2021-01-01
      store: pegasus
      object_store: fds
      schema: v11
      index:
        prefix: loki_sgp_online_
        period: 24h
    - from: 2022-04-29
      store: pegasus
      object_store: fds
      schema: v11
      index:
        prefix: loki_sgp_online_twomonth_
        period: 1440h  #two month

...
table_manager_config:
    retention_deletes_enabled: false
    retention_period: 1440h
    poll_interval: 1h
    creation_grace_period: 7h

2. During initiating, table manager calculate result:

level=info ts=2022-06-06T06:37:24.050918162Z caller=table_manager.go:322 msg="synching tables" expected_tables=63
level=info ts=2022-06-06T07:37:24.051268311Z caller=table_manager.go:322 msg="synching tables" expected_tables=63

3. After initiating, old invalid tables are created by table manager.

level=info ts=2022-04-29T03:37:03.013873477Z caller=table_manager.go:473 msg="creating table" table=loki_mos_online_19051
level=info ts=2022-06-07T08:37:24.052232362Z caller=table_manager.go:473 msg="creating table" table=loki_mos_online_19058
level=info ts=2022-06-07T08:37:49.098392763Z caller=table_manager.go:473 msg="creating table" table=loki_mos_online_19063
level=info ts=2022-06-07T08:38:19.147631991Z caller=table_manager.go:473 msg="creating table" table=loki_mos_online_19057
level=info ts=2022-06-07T08:38:48.191893498Z caller=table_manager.go:473 msg="creating table" table=loki_mos_online_19061
level=info ts=2022-06-07T08:38:48.194352974Z caller=table_manager.go:473 msg="creating table" table=loki_mos_online_19069
level=info ts=2022-06-07T08:39:18.242200251Z caller=table_manager.go:473 msg="creating table" table=loki_mos_online_19060
level=info ts=2022-06-07T08:39:48.299557946Z caller=table_manager.go:473 msg="creating table" table=loki_mos_online_19087
level=info ts=2022-06-07T08:40:18.345171691Z caller=table_manager.go:473 msg="creating table" table=loki_mos_online_19086
level=info ts=2022-06-07T08:40:48.394609006Z caller=table_manager.go:473 msg="creating table" table=loki_mos_online_19071
level=info ts=2022-06-07T08:41:18.439606089Z caller=table_manager.go:473 msg="creating table" table=loki_mos_online_19076
level=info ts=2022-06-07T08:41:48.484749065Z caller=table_manager.go:473 msg="creating table" table=loki_mos_online_19053
level=info ts=2022-06-07T08:42:18.533293191Z caller=table_manager.go:473 msg="creating table" table=loki_mos_online_19079
level=info ts=2022-06-07T08:42:48.584298975Z caller=table_manager.go:473 msg="creating table" table=loki_mos_online_19083
level=info ts=2022-06-07T08:43:18.62813697Z caller=table_manager.go:473 msg="creating table" table=loki_mos_online_19078
level=info ts=2022-06-07T08:43:48.671717054Z caller=table_manager.go:473 msg="creating table" table=loki_mos_online_19075
level=info ts=2022-06-07T08:44:18.717133728Z caller=table_manager.go:473 msg="creating table" table=loki_mos_online_19084
level=info ts=2022-06-07T08:44:18.717623933Z caller=table_manager.go:473 msg="creating table" table=loki_mos_online_19052
level=info ts=2022-06-07T08:44:48.7624188Z caller=table_manager.go:473 msg="creating table" table=loki_mos_online_19085
level=info ts=2022-06-07T08:45:18.806052265Z caller=table_manager.go:473 msg="creating table" table=loki_mos_online_19088
level=info ts=2022-06-07T08:45:48.849086904Z caller=table_manager.go:473 msg="creating table" table=loki_mos_online_19054
level=info ts=2022-06-07T08:45:48.849435543Z caller=table_manager.go:473 msg="creating table" table=loki_mos_online_19077
level=info ts=2022-06-07T08:45:48.849829869Z caller=table_manager.go:473 msg="creating table" table=loki_mos_online_19082
level=info ts=2022-06-07T08:45:48.850188131Z caller=table_manager.go:473 msg="creating table" table=loki_mos_online_19067
level=info ts=2022-06-07T08:45:48.850494606Z caller=table_manager.go:473 msg="creating table" table=loki_mos_online_19074
level=info ts=2022-06-07T08:45:48.850775738Z caller=table_manager.go:473 msg="creating table" table=loki_mos_online_19056
level=info ts=2022-06-07T08:45:48.851043158Z caller=table_manager.go:473 msg="creating table" table=loki_mos_online_19059
level=info ts=2022-06-07T08:45:48.8512787Z caller=table_manager.go:473 msg="creating table" table=loki_mos_online_19068
level=info ts=2022-06-07T08:45:48.85151786Z caller=table_manager.go:473 msg="creating table" table=loki_mos_online_19070
level=info ts=2022-06-07T08:45:48.851766738Z caller=table_manager.go:473 msg="creating table" table=loki_mos_online_19072
level=info ts=2022-06-07T08:45:48.852152382Z caller=table_manager.go:473 msg="creating table" table=loki_mos_online_19073
level=info ts=2022-06-07T08:46:18.894559528Z caller=table_manager.go:473 msg="creating table" table=loki_mos_online_19065
level=info ts=2022-06-07T08:46:18.894972542Z caller=table_manager.go:473 msg="creating table" table=loki_mos_online_19080
level=info ts=2022-06-07T08:46:18.895306397Z caller=table_manager.go:473 msg="creating table" table=loki_mos_online_19062
level=info ts=2022-06-07T08:46:18.895638348Z caller=table_manager.go:473 msg="creating table" table=loki_mos_online_19064
level=info ts=2022-06-07T08:46:18.895909443Z caller=table_manager.go:473 msg="creating table" table=loki_mos_online_19066
level=info ts=2022-06-07T08:46:18.896318486Z caller=table_manager.go:473 msg="creating table" table=loki_mos_online_19055

Expected behavior

After the new configuration takes effect(after 2022-04-29)，old tables in first config can be removed.
Since retention_period=1440h,

1. expected_tables should be less than 30.
2. there should be no create actions for old configs since new config already in work.

[stale]

Hi! This issue has been automatically marked as stale because it has not had any
activity in the past 30 days.

We use a stalebot among other tools to help manage the state of issues in this project.
A stalebot can be very useful in closing issues in a number of cases; the most common
is closing issues or PRs where the original reporter has not responded.

Stalebots are also emotionless and cruel and can close issues which are still very relevant.

If this issue is important to you, please add a comment to keep it open. More importantly, please add a thumbs-up to the original issue entry.

We regularly sort for closed issues which have a stale label sorted by thumbs up.

We may also:

• Mark issues as revivable if we think it's a valid issue but isn't something we are likely
  to prioritize in the future (the issue will still remain closed).
• Add a keepalive label to silence the stalebot if the issue is very common/popular/important.

We are doing our best to respond, organize, and prioritize all issues but it can be a challenging task,
our sincere apologies if you find yourself at the mercy of the stalebot.