-
Notifications
You must be signed in to change notification settings - Fork 99
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Reorg tiering policy sections into manage tiering #3524
base: latest
Are you sure you want to change the base?
Changes from 15 commits
20822a0
e44aadc
09f9863
e897b89
6ad34ad
373f6c5
c572d39
be82728
c34591a
0a643a2
ca3f4df
6b90d0b
ee380c4
d44a4a4
6d574e3
9d2b86e
dd53546
8fd8769
e4e384a
6213d93
b65c2a7
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change | ||||
---|---|---|---|---|---|---|
@@ -1,46 +1,221 @@ | ||||||
--- | ||||||
title: Enabling the object storage tier | ||||||
excerpt: How to enable the object storage tier | ||||||
title: Manage tiering | ||||||
excerpt: How to enable and use object storage tiering | ||||||
products: [cloud] | ||||||
keywords: [tiered storage] | ||||||
tags: [storage, data management] | ||||||
cloud_ui: | ||||||
path: | ||||||
- [services, :serviceId, overview] | ||||||
plans: [scale, enterprise] | ||||||
--- | ||||||
|
||||||
# Tier data to the object storage tier | ||||||
# Manage tiering | ||||||
|
||||||
Enable tiered storage to begin migrating rarely used data from Timescale's standard high-performance storage tier | ||||||
to the object storage tier to save on storage costs. | ||||||
You use tiered storage to save on storage costs. Specifically, you can migrate rarely used data from Timescale's standard high-performance storage to the object storage. After you [enable tiered storage](#enable-tiered-storage), you then either [create automated tiering policies](#automate-tiering-with-policies) or [manually tier and untier data](#manually-tier-and-untier-chunks). | ||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
|
||||||
|
||||||
## Enabling the object storage tier | ||||||
You can query the data on the object storage tier, but you cannot modify it. Make sure that you are not tiering data that needs to be **actively modified**. | ||||||
|
||||||
You can enable tiered storage from the Services Overview page in the Timescale | ||||||
console. | ||||||
## Enable tiered storage | ||||||
|
||||||
You enable tiered storage from the `Overview` tab in Console. | ||||||
|
||||||
<Procedure> | ||||||
|
||||||
### Enabling tiered storage | ||||||
1. **In [Timescale Console][console], select the service to modify**. | ||||||
|
||||||
You see the `Overview` section. | ||||||
|
||||||
1. **Scroll down, then click `Enable tiered storage`**. | ||||||
|
||||||
1. In the Timescale console, from the `Services` list, click the name of | ||||||
the service you want to modify. | ||||||
1. In the `Overview` tab, locate the `Tiered Storage` card, and click | ||||||
`Enable tiered storage`. Confirm the action. | ||||||
1. Tiered storage can take a few seconds to turn on and once activated shows the amount of | ||||||
data that has been tiered. Once enabled, data can be tiered by manually tiering | ||||||
a chunk or by creating a tiering policy. | ||||||
![Enable tiered storage](https://assets.timescale.com/docs/images/console-enable-tiered-storage.png) | ||||||
|
||||||
<img class="main-content__illustration" | ||||||
src="https://assets.timescale.com/docs/images/enable-data-tiering-ga.png" | ||||||
width={1375} height={944} | ||||||
alt="The Timescale Console showing tiered storage enabled" /> | ||||||
When tiered storage is enabled, you see the amount of data in the tiered object storage. | ||||||
|
||||||
<Highlight type="note"> | ||||||
Data tiering is available in [Scale and Enterprise](/about/latest/pricing-and-account-management/) pricing plans only. | ||||||
</Highlight> | ||||||
|
||||||
</Procedure> | ||||||
|
||||||
After tiered storage is enabled you must either [manually tier data][manual-tier-chunk] or [setup a tiering policy][creating-data-tiering-policy] | ||||||
to begin tiering data from your hypertables. | ||||||
## Automate tiering with policies | ||||||
|
||||||
A tiering policy automatically moves any chunks that only contain data | ||||||
older than the `move_after` threshold to the object storage tier. This works similarly to a | ||||||
[data retention policy][data-retention], but chunks are moved rather than deleted. | ||||||
|
||||||
A tiering policy schedules a job that runs periodically to asynchronously migrate eligible chunks to object storage. Chunks are considered tiered once they appear in the `timescaledb_osm.tiered_chunks` view. | ||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
|
||||||
|
||||||
You can add tiering policies to [hypertables][hypertable], including [continuous aggregates][caggs]. To manage tiering policies, [connect to your service][connect-to-service] and run the queries below in the data mode, the SQL editor, or using `psql`. | ||||||
|
||||||
### Add a tiering policy | ||||||
|
||||||
To add a tiering policy, call `add_tiering_policy`: | ||||||
atovpeko marked this conversation as resolved.
Show resolved
Hide resolved
|
||||||
|
||||||
```sql | ||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. remove There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. remove what? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. The bits that are moved to the updates i suggested previously. |
||||||
SELECT add_tiering_policy(hypertable REGCLASS, move_after INTERVAL, if_not_exists BOOL = false); | ||||||
``` | ||||||
|
||||||
For example, to tier chunks that are more than three days old in the `example` [hypertable][hypertable]: | ||||||
|
||||||
```sql | ||||||
SELECT add_tiering_policy('example', INTERVAL '3 days'); | ||||||
``` | ||||||
|
||||||
### Remove a tiering policy | ||||||
|
||||||
To remove an existing tiering policy, call `remove_tiering_policy`: | ||||||
|
||||||
```sql | ||||||
SELECT remove_tiering_policy(hypertable REGCLASS, if_exists BOOL = false); | ||||||
``` | ||||||
|
||||||
For example, to remove the tiering policy from the `example` hypertable: | ||||||
|
||||||
```sql | ||||||
SELECT remove_tiering_policy('example'); | ||||||
``` | ||||||
|
||||||
If you remove a tiering policy, the remaining scheduled chunks are not tiered. However, chunks in tiered storage are not untiered. You [untier chunks manually](#manually-tier-and-untier-chunks) to local storage. | ||||||
|
||||||
## Manually tier and untier chunks | ||||||
|
||||||
If tiering policies do not meet your current needs, you can tier and untier chunks manually. To do so, [connect to your service][connect-to-service] and run the queries below in the data mode, the SQL editor, or using `psql`. | ||||||
|
||||||
### Tier chunks | ||||||
|
||||||
Tiering a chunk is an asynchronous process that schedules the chunk to be tiered. In the following example, you tier chunks older than three days in the `example` hypertable. You then list the tiered chunks. | ||||||
|
||||||
<Procedure> | ||||||
|
||||||
1. **Select all chunks in `example` that are older than three days:** | ||||||
|
||||||
```sql | ||||||
SELECT show_chunks('example', older_than => INTERVAL '3 days'); | ||||||
``` | ||||||
|
||||||
This returns a list of chunks. Take a note of the chunk names: | ||||||
|
||||||
```sql | ||||||
|1|_timescaledb_internal_hyper_1_2_chunk| | ||||||
|2|_timescaledb_internal_hyper_1_3_chunk| | ||||||
``` | ||||||
|
||||||
1. **Call `tier_chunk` to manually tier each chunk:** | ||||||
|
||||||
```sql | ||||||
SELECT tier_chunk( '_timescaledb_internal_hyper_1_2_chunk'); | ||||||
``` | ||||||
|
||||||
1. **Repeat for all chunks you want to tier.** | ||||||
|
||||||
Tiering a chunk schedules it for migration to the object storage tier, but the migration won't happen immediately. Chunks are tiered one at a time in order to minimize database resource consumption. You can continue to query a chunk during migration. | ||||||
|
||||||
1. **To see which chunks are tiered into the object storage tier, use the `tiered_chunks` informational view:** | ||||||
|
||||||
```sql | ||||||
SELECT * FROM timescaledb_osm.tiered_chunks; | ||||||
``` | ||||||
|
||||||
</Procedure> | ||||||
|
||||||
To see which chunks are scheduled for tiering either by policy or by a manual call, but have not yet been tiered, use this view: | ||||||
|
||||||
```sql | ||||||
SELECT * FROM timescaledb_osm.chunks_queued_for_tiering ; | ||||||
``` | ||||||
|
||||||
### Untier chunks | ||||||
|
||||||
To update data in a tiered chunk, move it back to the standard high-performance storage tier in $CLOUD_LONG. Untiering chunks is a synchronous process. Chunks are renamed when the data is untiered. | ||||||
|
||||||
To untier a chunk, call the `untier_chunk` stored procedure. | ||||||
|
||||||
<Procedure> | ||||||
|
||||||
1. **Check which chunks are currently tiered:** | ||||||
|
||||||
```sql | ||||||
SELECT * FROM timescaledb_osm.tiered_chunks ; | ||||||
``` | ||||||
|
||||||
Sample output: | ||||||
|
||||||
```sql | ||||||
hypertable_schema | hypertable_name | chunk_name | range_start | range_end | ||||||
-------------------+-----------------+------------------+------------------------+------------------------ | ||||||
public | sample | _hyper_1_1_chunk | 2023-02-16 00:00:00+00 | 2023-02-23 00:00:00+00 | ||||||
(1 row) | ||||||
``` | ||||||
|
||||||
1. **Call `untier_chunk`**: | ||||||
|
||||||
```sql | ||||||
CALL untier_chunk('_hyper_1_1_chunk'); | ||||||
``` | ||||||
|
||||||
1. **See the details of the chunk with `timescaledb_information.chunks`**: | ||||||
|
||||||
```sql | ||||||
SELECT * FROM timescaledb_information.chunks; | ||||||
``` | ||||||
|
||||||
Sample output: | ||||||
|
||||||
```sql | ||||||
-[ RECORD 1 ]----------+------------------------- | ||||||
hypertable_schema | public | ||||||
hypertable_name | sample | ||||||
chunk_schema | _timescaledb_internal | ||||||
chunk_name | _hyper_1_4_chunk | ||||||
primary_dimension | ts | ||||||
primary_dimension_type | timestamp with time zone | ||||||
range_start | 2023-02-16 00:00:00+00 | ||||||
range_end | 2020-03-23 00:00:00+00 | ||||||
range_start_integer | | ||||||
range_end_integer | | ||||||
is_compressed | f | ||||||
chunk_tablespace | | ||||||
data_nodes | | ||||||
``` | ||||||
|
||||||
</Procedure> | ||||||
|
||||||
|
||||||
## Disable tiering | ||||||
|
||||||
If you no longer want to use tiered storage for a particular hypertable, drop the associated metadata by calling `disable_tiering`. | ||||||
|
||||||
<Procedure> | ||||||
|
||||||
1. **To drop all tiering policies associated with a table, call `remove_tiering_policy`**. | ||||||
|
||||||
1. **Make sure that there is no tiered data associated with this hypertable**: | ||||||
|
||||||
1. List the tiered chunks associated with this hypertable: | ||||||
|
||||||
```sql | ||||||
select * from timescaledb_osm.tiered_chunks | ||||||
``` | ||||||
|
||||||
1. If you have any tiered chunks, either untier this data, or drop these chunks from tiered storage. | ||||||
|
||||||
1. **Use `disable_tiering` to drop all tiering-related metadata for the hypertable**: | ||||||
|
||||||
```sql | ||||||
select disable_tiering('my_hypertable_name'); | ||||||
``` | ||||||
|
||||||
1. **Verify that tiering has been disabled by listing the hypertables that have tiering enabled**: | ||||||
|
||||||
```sql | ||||||
select * from timescaledb_osm.tiered_hypertables; | ||||||
``` | ||||||
|
||||||
</Procedure> | ||||||
|
||||||
[manual-tier-chunk]: /use-timescale/:currentVersion:/data-tiering/manual-tier-chunk/ | ||||||
[creating-data-tiering-policy]: /use-timescale/:currentVersion:/data-tiering/creating-data-tiering-policy/ | ||||||
[data-retention]: /use-timescale/:currentVersion:/data-retention/ | ||||||
[console]: https://console.cloud.timescale.com/dashboard/services | ||||||
[hypertable]: /use-timescale/:currentVersion:/hypertables/ | ||||||
[connect-to-service]: /getting-started/:currentVersion:/services/#connect-to-your-service | ||||||
[caggs]: /use-timescale/:currentVersion:/continuous-aggregates/ |
Original file line number | Diff line number | Diff line change | ||||
---|---|---|---|---|---|---|
|
@@ -4,16 +4,16 @@ excerpt: Save on storage costs by tiering older data to a low-cost bottomless ob | |||||
products: [cloud] | ||||||
keywords: [tiered storage] | ||||||
tags: [storage, data management] | ||||||
plans: [scale, enterprise] | ||||||
--- | ||||||
|
||||||
# Tiered storage | ||||||
|
||||||
Tiered storage is Timescale's [hierarchical storage management architecture](https://en.wikipedia.org/wiki/Hierarchical_storage_management). | ||||||
Engineered for infinite low-cost scalability, tiered storage is available for the | ||||||
[Time series and analytics](https://www.timescale.com/products) instances you create in | ||||||
[Timescale](https://console.cloud.timescale.com/). | ||||||
Tiered storage is a [hierarchical storage management architecture](https://en.wikipedia.org/wiki/Hierarchical_storage_management) for | ||||||
[Time series and analytics][create-service] services you create in [$CLOUD_LONG](https://console.cloud.timescale.com/). | ||||||
|
||||||
Engineered for infinite low-cost scalability, tiered storage consists of the: | ||||||
|
||||||
Tiered storage consists of the: | ||||||
* **High-performance tier**: rapid access to the most recent, and frequently accessed data. | ||||||
|
||||||
* **Object storage tier**: store data that is rarely accessed and has lower performance requirements. | ||||||
|
@@ -42,30 +42,33 @@ solutions to offload data to secondary storage and fetch it back in when needed. | |||||
we do the work for you. | ||||||
|
||||||
<Highlight type="info"> | ||||||
|
||||||
Tiered storage is only available for the [Time series and analytics](https://www.timescale.com/products) | ||||||
instances you create in [Timescale](https://console.cloud.timescale.com/). | ||||||
instances you create in [$CLOUD_LONG](https://console.cloud.timescale.com/). | ||||||
|
||||||
Tiered storage **DOES NOT** work on Self-hosted TimescaleDB or Managed Service for TimescaleDB. | ||||||
</Highlight> | ||||||
|
||||||
<!-- vale Google.SmartQuotes = YES --> | ||||||
|
||||||
In this section you can: | ||||||
* [Learn about the object storage tier][about-data-tiering] before you start using tiered storage. | ||||||
* Take a [tour of tiered storage features][tour-data-tiering]. | ||||||
* [Learn how to enable the object storage tier][enabling-data-tiering] on your service. | ||||||
* Manually [tier chunks][manual-tier-chunk] to schedule individual chunks to be tiered. | ||||||
* Create a [Tiering Policy][creating-data-tiering-policy] to automatically schedule chunks to be tiered. | ||||||
* [Learn how to query tiered data][querying-tiered-data]. | ||||||
* Manually [untier chunks][untier-data] to move data back to the high-performance local storage tier. | ||||||
* [Disable tiering on a hypertable][disabling-data-tiering] on an individual table if you no longer want to associate it with tiered storage. | ||||||
This section explains the following: | ||||||
* [Learn about the object storage tier][about-data-tiering]: understand tiered storage before you | ||||||
[Manage tiering][enabling-data-tiering]. | ||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. should remove this. |
||||||
* [Tour tiered storage][tour-data-tiering]: see the different features in tiered storage. | ||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Can you remove this link please. |
||||||
* [Manage tiering][enabling-data-tiering]: enable and disable data tiering, automate tiering with | ||||||
policies or tier and untier manually. | ||||||
* [Query tiered data][querying-tiered-data]: query and performance for tiered data. | ||||||
* [Replicas and forks with tiered data][replicas-and-forks]: billing and tiered storage. | ||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
|
||||||
|
||||||
|
||||||
[about-data-tiering]: /use-timescale/:currentVersion:/data-tiering/about-data-tiering/ | ||||||
[tour-data-tiering]: /use-timescale/:currentVersion:/data-tiering/tour-data-tiering/ | ||||||
[enabling-data-tiering]: /use-timescale/:currentVersion:/data-tiering/enabling-data-tiering/ | ||||||
[replicas-and-forks]: /use-timescale/:currentVersion:/data-tiering/tiered-data-replicas-forks/ | ||||||
[manual-tier-chunk]: /use-timescale/:currentVersion:/data-tiering/manual-tier-chunk/ | ||||||
[disabling-data-tiering]: /use-timescale/:currentVersion:/data-tiering/disabling-data-tiering/ | ||||||
[creating-data-tiering-policy]: /use-timescale/:currentVersion:/data-tiering/creating-data-tiering-policy/ | ||||||
[creating-data-tiering-policy]: /use-timescale/:currentVersion:/data-tiering/enabling-data-tiering/#automate-tiering-with-policies | ||||||
[querying-tiered-data]: /use-timescale/:currentVersion:/data-tiering/querying-tiered-data/ | ||||||
[untier-data]: /use-timescale/:currentVersion:/data-tiering/untier-data/ | ||||||
[add-retention-policies]: /api/:currentVersion:/continuous-aggregates/add_policies/ | ||||||
[create-service]: /getting-started/:currentVersion:/services/ |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Maybe the title should explain more clearly what we explain. Manage automatic and manual tiering?