Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Reorg tiering policy sections into manage tiering #3524

Open
wants to merge 21 commits into
base: latest
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 15 commits
Commits
Show all changes
21 commits
Select commit Hold shift + click to select a range
20822a0
draft
atovpeko Oct 22, 2024
e44aadc
draft
atovpeko Oct 22, 2024
09f9863
draft
atovpeko Oct 23, 2024
e897b89
Merge branch 'latest' into 3508-docs-rfc-reorg-tiering-policy-section…
atovpeko Oct 23, 2024
6ad34ad
review comments
atovpeko Oct 30, 2024
373f6c5
Merge branch 'latest' into 3508-docs-rfc-reorg-tiering-policy-section…
atovpeko Oct 30, 2024
c572d39
Update use-timescale/data-tiering/enabling-data-tiering.md
atovpeko Oct 30, 2024
be82728
Update use-timescale/data-tiering/enabling-data-tiering.md
atovpeko Oct 30, 2024
c34591a
Update use-timescale/data-tiering/enabling-data-tiering.md
atovpeko Oct 30, 2024
0a643a2
Merge branch '3508-docs-rfc-reorg-tiering-policy-sections-into-manage…
atovpeko Oct 30, 2024
ca3f4df
review comment
atovpeko Oct 30, 2024
6b90d0b
pricing widget
atovpeko Oct 30, 2024
ee380c4
chore: add the plan widget to the index page.
billy-the-fish Oct 30, 2024
d44a4a4
chore: tiny cleanup for clickthroughs.
billy-the-fish Oct 30, 2024
6d574e3
Merge branch 'latest' into 3508-docs-rfc-reorg-tiering-policy-section…
billy-the-fish Oct 30, 2024
9d2b86e
chore: updates on review.
billy-the-fish Oct 31, 2024
dd53546
removed tiered storage tour
atovpeko Nov 1, 2024
8fd8769
Merge branch 'latest' into 3508-docs-rfc-reorg-tiering-policy-section…
billy-the-fish Nov 1, 2024
e4e384a
review comments
atovpeko Nov 4, 2024
6213d93
Merge branch 'latest' into 3508-docs-rfc-reorg-tiering-policy-section…
atovpeko Nov 11, 2024
b65c2a7
Merge branch 'latest' into 3508-docs-rfc-reorg-tiering-policy-section…
atovpeko Nov 13, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions _troubleshooting/slow-tiering-chunks.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@ products: [cloud]
topics: [data tiering]
keywords: [tiered storage]
tags: [tiered storage]
plans: [scale, enterprise]
---


Expand Down
1 change: 1 addition & 0 deletions use-timescale/data-tiering/about-data-tiering.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@ tags: [storage, data management]
cloud_ui:
path:
- [services, :serviceId, overview]
plans: [scale, enterprise]
---

# About the object storage tier
Expand Down
223 changes: 199 additions & 24 deletions use-timescale/data-tiering/enabling-data-tiering.md
Original file line number Diff line number Diff line change
@@ -1,46 +1,221 @@
---
title: Enabling the object storage tier
excerpt: How to enable the object storage tier
title: Manage tiering
excerpt: How to enable and use object storage tiering
products: [cloud]
keywords: [tiered storage]
tags: [storage, data management]
cloud_ui:
path:
- [services, :serviceId, overview]
plans: [scale, enterprise]
---

# Tier data to the object storage tier
# Manage tiering
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe the title should explain more clearly what we explain. Manage automatic and manual tiering?


Enable tiered storage to begin migrating rarely used data from Timescale's standard high-performance storage tier
to the object storage tier to save on storage costs.
You use tiered storage to save on storage costs. Specifically, you can migrate rarely used data from Timescale's standard high-performance storage to the object storage. After you [enable tiered storage](#enable-tiered-storage), you then either [create automated tiering policies](#automate-tiering-with-policies) or [manually tier and untier data](#manually-tier-and-untier-chunks).
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
You use tiered storage to save on storage costs. Specifically, you can migrate rarely used data from Timescale's standard high-performance storage to the object storage. After you [enable tiered storage](#enable-tiered-storage), you then either [create automated tiering policies](#automate-tiering-with-policies) or [manually tier and untier data](#manually-tier-and-untier-chunks).
You use tiered storage to save on storage costs. Specifically, you can migrate rarely used data from Timescale's standard high-performance storage to object storage. After you [enable tiered storage](#enable-tiered-storage), you then either [create automated tiering policies](#automate-tiering-with-policies) or [manually tier and untier data](#manually-tier-and-untier-chunks).


## Enabling the object storage tier
You can query the data on the object storage tier, but you cannot modify it. Make sure that you are not tiering data that needs to be **actively modified**.

You can enable tiered storage from the Services Overview page in the Timescale
console.
## Enable tiered storage

You enable tiered storage from the `Overview` tab in Console.

<Procedure>

### Enabling tiered storage
1. **In [Timescale Console][console], select the service to modify**.

You see the `Overview` section.

1. **Scroll down, then click `Enable tiered storage`**.

1. In the Timescale console, from the `Services` list, click the name of
the service you want to modify.
1. In the `Overview` tab, locate the `Tiered Storage` card, and click
`Enable tiered storage`. Confirm the action.
1. Tiered storage can take a few seconds to turn on and once activated shows the amount of
data that has been tiered. Once enabled, data can be tiered by manually tiering
a chunk or by creating a tiering policy.
![Enable tiered storage](https://assets.timescale.com/docs/images/console-enable-tiered-storage.png)

<img class="main-content__illustration"
src="https://assets.timescale.com/docs/images/enable-data-tiering-ga.png"
width={1375} height={944}
alt="The Timescale Console showing tiered storage enabled" />
When tiered storage is enabled, you see the amount of data in the tiered object storage.

<Highlight type="note">
Data tiering is available in [Scale and Enterprise](/about/latest/pricing-and-account-management/) pricing plans only.
</Highlight>

</Procedure>

After tiered storage is enabled you must either [manually tier data][manual-tier-chunk] or [setup a tiering policy][creating-data-tiering-policy]
to begin tiering data from your hypertables.
## Automate tiering with policies

A tiering policy automatically moves any chunks that only contain data
older than the `move_after` threshold to the object storage tier. This works similarly to a
[data retention policy][data-retention], but chunks are moved rather than deleted.

A tiering policy schedules a job that runs periodically to asynchronously migrate eligible chunks to object storage. Chunks are considered tiered once they appear in the `timescaledb_osm.tiered_chunks` view.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
A tiering policy schedules a job that runs periodically to asynchronously migrate eligible chunks to object storage. Chunks are considered tiered once they appear in the `timescaledb_osm.tiered_chunks` view.
A tiering policy schedules a job that runs periodically to asynchronously migrate eligible chunks to object storage. After chunks are tiered, they appear in the `timescaledb_osm.tiered_chunks` view.


You can add tiering policies to [hypertables][hypertable], including [continuous aggregates][caggs]. To manage tiering policies, [connect to your service][connect-to-service] and run the queries below in the data mode, the SQL editor, or using `psql`.

### Add a tiering policy

To add a tiering policy, call `add_tiering_policy`:
atovpeko marked this conversation as resolved.
Show resolved Hide resolved

```sql
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

remove

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

remove what?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The bits that are moved to the updates i suggested previously.

SELECT add_tiering_policy(hypertable REGCLASS, move_after INTERVAL, if_not_exists BOOL = false);
```

For example, to tier chunks that are more than three days old in the `example` [hypertable][hypertable]:

```sql
SELECT add_tiering_policy('example', INTERVAL '3 days');
```

### Remove a tiering policy

To remove an existing tiering policy, call `remove_tiering_policy`:

```sql
SELECT remove_tiering_policy(hypertable REGCLASS, if_exists BOOL = false);
```

For example, to remove the tiering policy from the `example` hypertable:

```sql
SELECT remove_tiering_policy('example');
```

If you remove a tiering policy, the remaining scheduled chunks are not tiered. However, chunks in tiered storage are not untiered. You [untier chunks manually](#manually-tier-and-untier-chunks) to local storage.

## Manually tier and untier chunks

If tiering policies do not meet your current needs, you can tier and untier chunks manually. To do so, [connect to your service][connect-to-service] and run the queries below in the data mode, the SQL editor, or using `psql`.

### Tier chunks

Tiering a chunk is an asynchronous process that schedules the chunk to be tiered. In the following example, you tier chunks older than three days in the `example` hypertable. You then list the tiered chunks.

<Procedure>

1. **Select all chunks in `example` that are older than three days:**

```sql
SELECT show_chunks('example', older_than => INTERVAL '3 days');
```

This returns a list of chunks. Take a note of the chunk names:

```sql
|1|_timescaledb_internal_hyper_1_2_chunk|
|2|_timescaledb_internal_hyper_1_3_chunk|
```

1. **Call `tier_chunk` to manually tier each chunk:**

```sql
SELECT tier_chunk( '_timescaledb_internal_hyper_1_2_chunk');
```

1. **Repeat for all chunks you want to tier.**

Tiering a chunk schedules it for migration to the object storage tier, but the migration won't happen immediately. Chunks are tiered one at a time in order to minimize database resource consumption. You can continue to query a chunk during migration.

1. **To see which chunks are tiered into the object storage tier, use the `tiered_chunks` informational view:**

```sql
SELECT * FROM timescaledb_osm.tiered_chunks;
```

</Procedure>

To see which chunks are scheduled for tiering either by policy or by a manual call, but have not yet been tiered, use this view:

```sql
SELECT * FROM timescaledb_osm.chunks_queued_for_tiering ;
```

### Untier chunks

To update data in a tiered chunk, move it back to the standard high-performance storage tier in $CLOUD_LONG. Untiering chunks is a synchronous process. Chunks are renamed when the data is untiered.

To untier a chunk, call the `untier_chunk` stored procedure.

<Procedure>

1. **Check which chunks are currently tiered:**

```sql
SELECT * FROM timescaledb_osm.tiered_chunks ;
```

Sample output:

```sql
hypertable_schema | hypertable_name | chunk_name | range_start | range_end
-------------------+-----------------+------------------+------------------------+------------------------
public | sample | _hyper_1_1_chunk | 2023-02-16 00:00:00+00 | 2023-02-23 00:00:00+00
(1 row)
```

1. **Call `untier_chunk`**:

```sql
CALL untier_chunk('_hyper_1_1_chunk');
```

1. **See the details of the chunk with `timescaledb_information.chunks`**:

```sql
SELECT * FROM timescaledb_information.chunks;
```

Sample output:

```sql
-[ RECORD 1 ]----------+-------------------------
hypertable_schema | public
hypertable_name | sample
chunk_schema | _timescaledb_internal
chunk_name | _hyper_1_4_chunk
primary_dimension | ts
primary_dimension_type | timestamp with time zone
range_start | 2023-02-16 00:00:00+00
range_end | 2020-03-23 00:00:00+00
range_start_integer |
range_end_integer |
is_compressed | f
chunk_tablespace |
data_nodes |
```

</Procedure>


## Disable tiering

If you no longer want to use tiered storage for a particular hypertable, drop the associated metadata by calling `disable_tiering`.

<Procedure>

1. **To drop all tiering policies associated with a table, call `remove_tiering_policy`**.

1. **Make sure that there is no tiered data associated with this hypertable**:

1. List the tiered chunks associated with this hypertable:

```sql
select * from timescaledb_osm.tiered_chunks
```

1. If you have any tiered chunks, either untier this data, or drop these chunks from tiered storage.

1. **Use `disable_tiering` to drop all tiering-related metadata for the hypertable**:

```sql
select disable_tiering('my_hypertable_name');
```

1. **Verify that tiering has been disabled by listing the hypertables that have tiering enabled**:

```sql
select * from timescaledb_osm.tiered_hypertables;
```

</Procedure>

[manual-tier-chunk]: /use-timescale/:currentVersion:/data-tiering/manual-tier-chunk/
[creating-data-tiering-policy]: /use-timescale/:currentVersion:/data-tiering/creating-data-tiering-policy/
[data-retention]: /use-timescale/:currentVersion:/data-retention/
[console]: https://console.cloud.timescale.com/dashboard/services
[hypertable]: /use-timescale/:currentVersion:/hypertables/
[connect-to-service]: /getting-started/:currentVersion:/services/#connect-to-your-service
[caggs]: /use-timescale/:currentVersion:/continuous-aggregates/
35 changes: 19 additions & 16 deletions use-timescale/data-tiering/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,16 +4,16 @@ excerpt: Save on storage costs by tiering older data to a low-cost bottomless ob
products: [cloud]
keywords: [tiered storage]
tags: [storage, data management]
plans: [scale, enterprise]
---

# Tiered storage

Tiered storage is Timescale's [hierarchical storage management architecture](https://en.wikipedia.org/wiki/Hierarchical_storage_management).
Engineered for infinite low-cost scalability, tiered storage is available for the
[Time series and analytics](https://www.timescale.com/products) instances you create in
[Timescale](https://console.cloud.timescale.com/).
Tiered storage is a [hierarchical storage management architecture](https://en.wikipedia.org/wiki/Hierarchical_storage_management) for
[Time series and analytics][create-service] services you create in [$CLOUD_LONG](https://console.cloud.timescale.com/).

Engineered for infinite low-cost scalability, tiered storage consists of the:

Tiered storage consists of the:
* **High-performance tier**: rapid access to the most recent, and frequently accessed data.

* **Object storage tier**: store data that is rarely accessed and has lower performance requirements.
Expand Down Expand Up @@ -42,30 +42,33 @@ solutions to offload data to secondary storage and fetch it back in when needed.
we do the work for you.

<Highlight type="info">

Tiered storage is only available for the [Time series and analytics](https://www.timescale.com/products)
instances you create in [Timescale](https://console.cloud.timescale.com/).
instances you create in [$CLOUD_LONG](https://console.cloud.timescale.com/).

Tiered storage **DOES NOT** work on Self-hosted TimescaleDB or Managed Service for TimescaleDB.
</Highlight>

<!-- vale Google.SmartQuotes = YES -->

In this section you can:
* [Learn about the object storage tier][about-data-tiering] before you start using tiered storage.
* Take a [tour of tiered storage features][tour-data-tiering].
* [Learn how to enable the object storage tier][enabling-data-tiering] on your service.
* Manually [tier chunks][manual-tier-chunk] to schedule individual chunks to be tiered.
* Create a [Tiering Policy][creating-data-tiering-policy] to automatically schedule chunks to be tiered.
* [Learn how to query tiered data][querying-tiered-data].
* Manually [untier chunks][untier-data] to move data back to the high-performance local storage tier.
* [Disable tiering on a hypertable][disabling-data-tiering] on an individual table if you no longer want to associate it with tiered storage.
This section explains the following:
* [Learn about the object storage tier][about-data-tiering]: understand tiered storage before you
[Manage tiering][enabling-data-tiering].
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should remove this.

* [Tour tiered storage][tour-data-tiering]: see the different features in tiered storage.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you remove this link please.

* [Manage tiering][enabling-data-tiering]: enable and disable data tiering, automate tiering with
policies or tier and untier manually.
* [Query tiered data][querying-tiered-data]: query and performance for tiered data.
* [Replicas and forks with tiered data][replicas-and-forks]: billing and tiered storage.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
* [Replicas and forks with tiered data][replicas-and-forks]: billing and tiered storage.
* [Replicas and forks with tiered data][replicas-and-forks]: How does tiered storage work with forks and replicas.



[about-data-tiering]: /use-timescale/:currentVersion:/data-tiering/about-data-tiering/
[tour-data-tiering]: /use-timescale/:currentVersion:/data-tiering/tour-data-tiering/
[enabling-data-tiering]: /use-timescale/:currentVersion:/data-tiering/enabling-data-tiering/
[replicas-and-forks]: /use-timescale/:currentVersion:/data-tiering/tiered-data-replicas-forks/
[manual-tier-chunk]: /use-timescale/:currentVersion:/data-tiering/manual-tier-chunk/
[disabling-data-tiering]: /use-timescale/:currentVersion:/data-tiering/disabling-data-tiering/
[creating-data-tiering-policy]: /use-timescale/:currentVersion:/data-tiering/creating-data-tiering-policy/
[creating-data-tiering-policy]: /use-timescale/:currentVersion:/data-tiering/enabling-data-tiering/#automate-tiering-with-policies
[querying-tiered-data]: /use-timescale/:currentVersion:/data-tiering/querying-tiered-data/
[untier-data]: /use-timescale/:currentVersion:/data-tiering/untier-data/
[add-retention-policies]: /api/:currentVersion:/continuous-aggregates/add_policies/
[create-service]: /getting-started/:currentVersion:/services/
1 change: 1 addition & 0 deletions use-timescale/data-tiering/querying-tiered-data.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@ excerpt: How to query tiered data
product: [ cloud ]
keywords: [ tiered storage, tiering ]
tags: [ storage, data management ]
plans: [scale, enterprise]
---

# Querying tiered data
Expand Down
1 change: 1 addition & 0 deletions use-timescale/data-tiering/tiered-data-replicas-forks.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@ excerpt: How tiered data works on replicas and forks
product: [cloud]
keywords: [tiered storage]
tags: [storage, data management]
plans: [scale, enterprise]
---

# How tiered data works on replicas and forks
Expand Down
1 change: 1 addition & 0 deletions use-timescale/data-tiering/tour-data-tiering.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@ excerpt: A quick tour of tiered storage
product: [cloud]
keywords: [tiered storage]
tags: [storage, data management]
plans: [scale, enterprise]
---

# Tiered Storage
Expand Down
Loading
Loading