Reorg tiering policy sections into manage tiering #3524

atovpeko · 2024-10-23T08:36:15Z

No description provided.

github-actions · 2024-10-23T08:36:27Z

Allow 10 minutes from last push for the staging site to build. If the link doesn't work, try using incognito mode instead. For internal reviewers, check web-documentation repo actions for staging build status. Link to build for this PR: http://docs-dev.timescale.com/docs-3508-docs-rfc-reorg-tiering-policy-sections-into-manage-tiering

…s-into-manage-tiering

billy-the-fish

Few comments, good stuff.

use-timescale/data-tiering/enabling-data-tiering.md

…s-into-manage-tiering

Co-authored-by: Iain Cox <[email protected]> Signed-off-by: atovpeko <[email protected]>

…-tiering' of github.com:timescale/docs into 3508-docs-rfc-reorg-tiering-policy-sections-into-manage-tiering

use-timescale/data-tiering/enabling-data-tiering.md

billy-the-fish · 2024-10-30T13:55:00Z

@gayyappan can you have a look at https://docs-dev.timescale.com/docs-3508-docs-rfc-reorg-tiering-policy-sections-into-manage-tiering/use-timescale/3508-docs-rfc-reorg-tiering-policy-sections-into-manage-tiering/data-tiering/enabling-data-tiering/ please. Any comments in this PR.

@atovpeko: I have bad news for you. Now we have https://docs-dev.timescale.com/docs-3508-docs-rfc-reorg-tiering-policy-sections-into-manage-tiering/use-timescale/3508-docs-rfc-reorg-tiering-policy-sections-into-manage-tiering/data-tiering/enabling-data-tiering/ , I don't think we need https://docs-dev.timescale.com/docs-3508-docs-rfc-reorg-tiering-policy-sections-into-manage-tiering/use-timescale/3508-docs-rfc-reorg-tiering-policy-sections-into-manage-tiering/data-tiering/tour-data-tiering/ any more. @gayyappan, what do you think?

Do we have a corresponding redirects PR?

…s-into-manage-tiering

gayyappan · 2024-10-30T18:42:52Z

use-timescale/data-tiering/index.md

-* [Disable tiering on a hypertable][disabling-data-tiering] on an individual table if you no longer want to associate it with tiered storage.
+This section explains the following:
+* [Learn about the object storage tier][about-data-tiering]: understand tiered storage before you 
+  [Manage tiering][enabling-data-tiering].


should remove this.

gayyappan · 2024-10-30T18:44:30Z

use-timescale/data-tiering/index.md

+* [Manage tiering][enabling-data-tiering]: enable and disable data tiering, automate tiering with 
+   policies or tier and untier manually.
+* [Query tiered data][querying-tiered-data]: query and performance for tiered data.
+* [Replicas and forks with tiered data][replicas-and-forks]: billing and tiered storage. 


Suggested change

* [Replicas and forks with tiered data][replicas-and-forks]: billing and tiered storage.

* [Replicas and forks with tiered data][replicas-and-forks]: How does tiered storage work with forks and replicas.

gayyappan · 2024-10-30T18:46:24Z

use-timescale/data-tiering/enabling-data-tiering.md

+older than the `move_after` threshold to the object storage tier. This works similarly to a
+[data retention policy][data-retention], but chunks are moved rather than deleted. 
+
+A tiering policy schedules a job that runs periodically to asynchronously migrate eligible chunks to object storage. Chunks are considered tiered once they appear in the `timescaledb_osm.tiered_chunks` view. 


Suggested change

A tiering policy schedules a job that runs periodically to asynchronously migrate eligible chunks to object storage. Chunks are considered tiered once they appear in the `timescaledb_osm.tiered_chunks` view.

A tiering policy schedules a job that runs periodically to asynchronously migrate eligible chunks to object storage. After chunks are tiered, they appear in the `timescaledb_osm.tiered_chunks` view.

gayyappan

minor edits suggested.
The overall "Manage tiering" section looks good!

billy-the-fish

These pages are really coming together.

billy-the-fish · 2024-11-01T15:56:45Z

use-timescale/data-tiering/index.md

-* [Disable tiering on a hypertable][disabling-data-tiering] on an individual table if you no longer want to associate it with tiered storage.
+This section explains the following:
+* [Learn about the object storage tier][about-data-tiering]: understand tiered storage.
+* [Tour tiered storage][tour-data-tiering]: see the different features in tiered storage. 


Can you remove this link please.

billy-the-fish · 2024-11-01T15:57:38Z

use-timescale/data-tiering/enabling-data-tiering.md

 ---

-# Tier data to the object storage tier
+# Manage tiering


Maybe the title should explain more clearly what we explain. Manage automatic and manual tiering?

billy-the-fish · 2024-11-01T15:59:54Z

use-timescale/data-tiering/about-data-tiering.md

 ---

 # About the object storage tier

-The tiered storage architecture complements Timescale's standard high-performance storage tier with a low-cost object storage tier.
+The Timescale's tiered storage architecture includes a standard high-performance storage tier and a low-cost object storage tier built on Amazon S3. You can use the standard tier for data that requires quick access, and the object tier for rarely used historical data. Chunks from a single hypertable, including compressed chunks, can stretch across these two storage tiers. A compressed chunk uses a different storage representation after tiering.


Suggested change

The Timescale's tiered storage architecture includes a standard high-performance storage tier and a low-cost object storage tier built on Amazon S3. You can use the standard tier for data that requires quick access, and the object tier for rarely used historical data. Chunks from a single hypertable, including compressed chunks, can stretch across these two storage tiers. A compressed chunk uses a different storage representation after tiering.

Timescale's tiered storage architecture includes a standard high-performance storage tier, and a low-cost object storage tier built on Amazon S3. You use the standard tier for data that requires quick access, and the object tier for rarely used historical data. Chunks from a single hypertable, including compressed chunks, can stretch across these two storage tiers. A compressed chunk uses a different storage representation after tiering.

billy-the-fish · 2024-11-01T16:01:23Z

use-timescale/data-tiering/about-data-tiering.md

-build views on tiered data, and even define continuous aggregates on tiered data.
-In fact, because the implementation of continuous aggregates also use hypertables, 
-they can be tiered to low-cost storage as well.
+In the standard storage, chunks are stored in the block format. In the object storage, they are stored in a compressed, columnar format. This format is different from that of the internals of the database, for better interoperability across various platforms. It allows for more efficient columnar scans across longer time periods, and Timescale uses other metadata and query optimizations to reduce the amount of data that needs to be fetched from the object storage tier to satisfy a query. 


Suggested change

In the standard storage, chunks are stored in the block format. In the object storage, they are stored in a compressed, columnar format. This format is different from that of the internals of the database, for better interoperability across various platforms. It allows for more efficient columnar scans across longer time periods, and Timescale uses other metadata and query optimizations to reduce the amount of data that needs to be fetched from the object storage tier to satisfy a query.

In high-performance storage, chunks are stored in the block format. In the object storage, they are stored in a compressed, columnar format. For better interoperability across various platforms, this format is different from that of the internals of the database. It allows for more efficient columnar scans across longer time periods, and Timescale Cloud uses other metadata and query optimizations to reduce the amount of data that needs to be fetched from the object storage tier to satisfy a query.

billy-the-fish · 2024-11-01T16:33:32Z

use-timescale/data-tiering/about-data-tiering.md

-an object store built on Amazon S3.
-There, it's stored in the Apache Parquet format, which is a compressed
-columnar format well-suited for S3. Data remains accessible both during and after the migration.
+The tiered storage backend works by periodically and asynchronously moving older chunks to the object storage tier. 


Suggested change

The tiered storage backend works by periodically and asynchronously moving older chunks to the object storage tier.

The tiered storage backend works by periodically and asynchronously moving older chunks from high-performance storage to the object storage tier.

billy-the-fish · 2024-11-01T16:36:02Z

use-timescale/data-tiering/about-data-tiering.md

-
-The result is transparent queries across standard PostgreSQL storage and S3
-storage, so your queries fetch the same data as before.
+* Chunk pruning - exclude the chunks that fall outside the query time window.


Can you put Chunk pruning: etc in bold to match the other lists in the page please.

billy-the-fish · 2024-11-01T16:38:04Z

use-timescale/data-tiering/about-data-tiering.md

+* Row group pruning - identify the row groups within the Parquet object that satisfy the query.
+* Column pruning - fetch only columns that are requested by the query.
+
+The result is transparent queries across standard PostgreSQL storage and S3 storage, so your queries fetch the same data as before.


Suggested change

The result is transparent queries across standard PostgreSQL storage and S3 storage, so your queries fetch the same data as before.

The result is transparent queries across high-performance storage and S3 object storage , so your queries fetch the same data as before.

billy-the-fish · 2024-11-01T16:39:39Z

use-timescale/data-tiering/enabling-data-tiering.md


-Enable tiered storage to begin migrating rarely used data from Timescale's standard high-performance storage tier
-to the object storage tier to save on storage costs. 
+You use tiered storage to save on storage costs. Specifically, you can migrate rarely used data from Timescale's standard high-performance storage to the object storage. After you [enable tiered storage](#enable-tiered-storage), you then either [create automated tiering policies](#automate-tiering-with-policies) or [manually tier and untier data](#manually-tier-and-untier-chunks).


Suggested change

You use tiered storage to save on storage costs. Specifically, you can migrate rarely used data from Timescale's standard high-performance storage to the object storage. After you [enable tiered storage](#enable-tiered-storage), you then either [create automated tiering policies](#automate-tiering-with-policies) or [manually tier and untier data](#manually-tier-and-untier-chunks).

You use tiered storage to save on storage costs. Specifically, you can migrate rarely used data from Timescale's standard high-performance storage to object storage. After you [enable tiered storage](#enable-tiered-storage), you then either [create automated tiering policies](#automate-tiering-with-policies) or [manually tier and untier data](#manually-tier-and-untier-chunks).

billy-the-fish · 2024-11-01T16:45:02Z

use-timescale/data-tiering/querying-tiered-data.md

@@ -23,95 +21,170 @@ sessions.
 With tiered reads enabled, you can query your data normally even when it's distributed across different storage tiers.
 Your hypertable is spread across the tiers, so queries and `JOIN`s work and fetch the same data as usual.

-<!-- vale Google.Acronyms = YES -->
-
 <Highlight type="warning">


I'd make this into a sentence without the warning and link to the performance section. if you must, make it an info admomition.

…s-into-manage-tiering

atovpeko · 2024-11-04T10:35:18Z

@iroussos could you please take a look at all those pages in Manage tiering, rather than just the diff, to get a bigger picture?

iroussos · 2024-11-06T15:13:38Z

@iroussos could you please take a look at all those pages in Manage tiering, rather than just the diff, to get a bigger picture?

Makes sense, thank you @atovpeko!

…s-into-manage-tiering

atovpeko added 3 commits October 22, 2024 12:25

draft

20822a0

draft

e44aadc

draft

09f9863

atovpeko requested review from billy-the-fish and gayyappan October 23, 2024 08:36

atovpeko linked an issue Oct 23, 2024 that may be closed by this pull request

[Docs RFC] Reorg tiering policy sections into Manage tiering #3508

Open

Merge branch 'latest' into 3508-docs-rfc-reorg-tiering-policy-section…

e897b89

…s-into-manage-tiering

billy-the-fish requested changes Oct 29, 2024

View reviewed changes

atovpeko and others added 6 commits October 30, 2024 12:15

review comments

6ad34ad

Merge branch 'latest' into 3508-docs-rfc-reorg-tiering-policy-section…

373f6c5

…s-into-manage-tiering

Update use-timescale/data-tiering/enabling-data-tiering.md

c572d39

Co-authored-by: Iain Cox <[email protected]> Signed-off-by: atovpeko <[email protected]>

Update use-timescale/data-tiering/enabling-data-tiering.md

be82728

Co-authored-by: Iain Cox <[email protected]> Signed-off-by: atovpeko <[email protected]>

Update use-timescale/data-tiering/enabling-data-tiering.md

c34591a

Co-authored-by: Iain Cox <[email protected]> Signed-off-by: atovpeko <[email protected]>

Merge branch '3508-docs-rfc-reorg-tiering-policy-sections-into-manage…

0a643a2

…-tiering' of github.com:timescale/docs into 3508-docs-rfc-reorg-tiering-policy-sections-into-manage-tiering

billy-the-fish reviewed Oct 30, 2024

View reviewed changes

atovpeko and others added 4 commits October 30, 2024 13:52

review comment

ca3f4df

pricing widget

6b90d0b

chore: add the plan widget to the index page.

ee380c4

chore: tiny cleanup for clickthroughs.

d44a4a4

Merge branch 'latest' into 3508-docs-rfc-reorg-tiering-policy-section…

6d574e3

…s-into-manage-tiering

gayyappan reviewed Oct 30, 2024

View reviewed changes

billy-the-fish and others added 2 commits October 31, 2024 15:36

chore: updates on review.

9d2b86e

removed tiered storage tour

dd53546

atovpeko requested a review from billy-the-fish November 1, 2024 13:37

billy-the-fish reviewed Nov 1, 2024

View reviewed changes

billy-the-fish and others added 2 commits November 1, 2024 17:47

Merge branch 'latest' into 3508-docs-rfc-reorg-tiering-policy-section…

8fd8769

…s-into-manage-tiering

review comments

e4e384a

atovpeko requested a review from billy-the-fish November 5, 2024 08:13

atovpeko added 2 commits November 11, 2024 09:53

Merge branch 'latest' into 3508-docs-rfc-reorg-tiering-policy-section…

6213d93

…s-into-manage-tiering

Merge branch 'latest' into 3508-docs-rfc-reorg-tiering-policy-section…

b65c2a7

…s-into-manage-tiering

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Reorg tiering policy sections into manage tiering #3524

Reorg tiering policy sections into manage tiering #3524

atovpeko commented Oct 23, 2024

github-actions bot commented Oct 23, 2024

billy-the-fish left a comment

billy-the-fish commented Oct 30, 2024

gayyappan Oct 30, 2024

gayyappan Oct 30, 2024

gayyappan Oct 30, 2024

gayyappan left a comment

billy-the-fish left a comment

billy-the-fish Nov 1, 2024

billy-the-fish Nov 1, 2024

billy-the-fish Nov 1, 2024

billy-the-fish Nov 1, 2024

billy-the-fish Nov 1, 2024

billy-the-fish Nov 1, 2024

billy-the-fish Nov 1, 2024

billy-the-fish Nov 1, 2024

billy-the-fish Nov 1, 2024

atovpeko commented Nov 4, 2024

iroussos commented Nov 6, 2024

	* [Replicas and forks with tiered data][replicas-and-forks]: billing and tiered storage.
	* [Replicas and forks with tiered data][replicas-and-forks]: How does tiered storage work with forks and replicas.

	A tiering policy schedules a job that runs periodically to asynchronously migrate eligible chunks to object storage. Chunks are considered tiered once they appear in the `timescaledb_osm.tiered_chunks` view.
	A tiering policy schedules a job that runs periodically to asynchronously migrate eligible chunks to object storage. After chunks are tiered, they appear in the `timescaledb_osm.tiered_chunks` view.

	The tiered storage backend works by periodically and asynchronously moving older chunks to the object storage tier.
	The tiered storage backend works by periodically and asynchronously moving older chunks from high-performance storage to the object storage tier.

	The result is transparent queries across standard PostgreSQL storage and S3 storage, so your queries fetch the same data as before.
	The result is transparent queries across high-performance storage and S3 object storage , so your queries fetch the same data as before.

	You use tiered storage to save on storage costs. Specifically, you can migrate rarely used data from Timescale's standard high-performance storage to the object storage. After you [enable tiered storage](#enable-tiered-storage), you then either [create automated tiering policies](#automate-tiering-with-policies) or [manually tier and untier data](#manually-tier-and-untier-chunks).
	You use tiered storage to save on storage costs. Specifically, you can migrate rarely used data from Timescale's standard high-performance storage to object storage. After you [enable tiered storage](#enable-tiered-storage), you then either [create automated tiering policies](#automate-tiering-with-policies) or [manually tier and untier data](#manually-tier-and-untier-chunks).

Reorg tiering policy sections into manage tiering #3524

Are you sure you want to change the base?

Reorg tiering policy sections into manage tiering #3524

Conversation

atovpeko commented Oct 23, 2024

github-actions bot commented Oct 23, 2024

billy-the-fish left a comment

Choose a reason for hiding this comment

billy-the-fish commented Oct 30, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gayyappan left a comment

Choose a reason for hiding this comment

billy-the-fish left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

atovpeko commented Nov 4, 2024

iroussos commented Nov 6, 2024