Add DR with active-passive and external storage #3679

aneta-petrova · 2025-02-18T07:20:02Z

What changes are you introducing?

Adding a new disaster recovery scenario: Active and passive server with external storage.

Why are you introducing these changes? (Explanation, links to references, issues, etc.)

A follow-up to #3558
The new scenario follows the structure established in #3615 for consistency

Anything else to add? (Considerations, potential downsides, alternative solutions you have explored, etc.)

Now that I'm working on the second scenario, I can already see opportunities for reuse or snippets. However, I think it will be better to deal with this after all the planned scenarios are in place.

Checklists

I am okay with my commits getting squashed when you merge this PR.
I am familiar with the contributing guidelines.

Please cherry-pick my commits into:

Foreman 3.14/Katello 4.16
Foreman 3.13/Katello 4.15 (EL9 only)
Foreman 3.12/Katello 4.14 (Satellite 6.16)
Foreman 3.11/Katello 4.13 (orcharhino 6.11 on EL8 only; orcharhino 7.0 on EL8+EL9)
Foreman 3.10/Katello 4.12
Foreman 3.9/Katello 4.11 (Satellite 6.15; orcharhino 6.8/6.9/6.10)
Foreman 3.8/Katello 4.10
Foreman 3.7/Katello 4.9 (Satellite 6.14)
We do not accept PRs for Foreman older than 3.7.

github-actions · 2025-02-18T07:22:53Z

The PR preview for f95ffc1 is available at theforeman-foreman-documentation-preview-pr-3679.surge.sh

The following output files are affected by this PR:

Administering_Project/index-satellite.html

show diff

show diff as HTML

aneta-petrova · 2025-02-18T07:28:57Z

@ehelms Can you please review another DR scenario? I am once again making a lot of barely-educated guesses and I added several questions as comments so please keep that in mind and read carefully :) Also, I'm not sure what level of detail we are aiming for with this scenario: Do we want to keep things high-level like the virtualization scenario or can we/should we provide more detailed instructions?

...dules/con_disaster-recovery-with-active-and-passive-project-server-and-external-storage.adoc

...ring-for-disaster-recovery-with-active-and-passive-project-server-with-external-storage.adoc

rh-max

Very clear and well-structured overall, I have only a few suggestions and considerations. 👍

Also, I'll just paste here what I found in section 21.1. Overview of recommended disaster recovery plans:

Choose a disaster recovery plan that best helps ensure the continuity of Satellite services in your deployment.

Virtualizing your Satellite Server ...

This communicates the idea "my disaster recovery plan is virtualizing the Satellite Server", but virtualizing by itself is not the solution. I'd make it more precise:

Snapshots of virtualized Satellite Server

This also aligns better with the non-action style of the subsequent plan suggestions.

This method is suitable if you can run Satellite on top of virtualization.

This is partly subjective, but to me choosing this phrasing:

run Satellite on top of virtualization (phrasing 1)

instead of something like this:

run Satellite virtualized (phrasing 2a)

or

run Satellite in a virtual machine (phrasing 2b)

implies that scenario described in phrasing 1 has some additional layer between Satellite and virtualization, and scenario described in phrasings 2a and 2b does not. But maybe most readers have more context, which rules out that extra non-existent layer.

...dules/con_disaster-recovery-with-active-and-passive-project-server-and-external-storage.adoc

...equisites-disaster-recovery-with-active-and-passive-project-server-and-external-storage.adoc

...ring-for-disaster-recovery-with-active-and-passive-project-server-with-external-storage.adoc

...c_recovering-from-disaster-with-active-and-passive-project-server-with-external-storage.adoc

...dules/con_disaster-recovery-with-active-and-passive-project-server-and-external-storage.adoc

aneta-petrova · 2025-02-28T18:17:34Z

Also, I'll just paste here what I found in section 21.1. Overview of recommended disaster recovery plans:

Let's track these in #3704.

...dules/con_disaster-recovery-with-active-and-passive-project-server-and-external-storage.adoc

...equisites-disaster-recovery-with-active-and-passive-project-server-and-external-storage.adoc

...ring-for-disaster-recovery-with-active-and-passive-project-server-with-external-storage.adoc

...c_recovering-from-disaster-with-active-and-passive-project-server-with-external-storage.adoc

aneta-petrova · 2025-02-28T20:32:09Z

@ehelms @evgeni Thanks for all your feedback so far! I implemented most of it, except for one comment thread that remains open with a follow-up question. Can you please re-review? This scenario feels more complex to me than the virtualization one, so I can't really tell whether we are getting close enough to the desired outcome.

aneta-petrova · 2025-02-28T21:03:04Z

~~It looks like the preview in #3679 (comment) didn't get updated after the latest set of commits. Just FYI for potential reviewers.~~

...ring-for-disaster-recovery-with-active-and-passive-project-server-with-external-storage.adoc

Co-authored-by: Eric Helms <[email protected]>

aneta-petrova · 2025-03-05T12:47:31Z

Feedback from style review implemented, which means style review done.

Rebased, which pulled in the change in #3705 and allowed me to include the procedure on retrieving status of services cleanly.

aneta-petrova · 2025-03-05T12:49:28Z

@ehelms and/or @evgeni, do you have any further comments?

...ring-for-disaster-recovery-with-active-and-passive-project-server-with-external-storage.adoc

evgeni · 2025-03-05T15:39:42Z

...ring-for-disaster-recovery-with-active-and-passive-project-server-with-external-storage.adoc

+. Clone your active {ProjectServer}.
+See xref:cloning_satellite_server[].


I just realized: the cloning process involves a DB-restore, which is probably not what people want.
What would probably work is: clone with /var/lib/pgsql being local, then turn things off and replace it with shared storage.
But that's fragile :/

When you say that it's "fragile", you're actually kind of saying that that's probably not what we want either :) So what would be a good solution?

Yeah… The deployment consists of three things: packages (rpms), config (/etc) and data (/var/lib/). To get a working deployment you need all of them. As the data is quickly moving, we place it on shared storage so it can be picked up from various places. The packages and config are not, so we can keep them on "traditional" storage, but they still need to match.

The easiest way to create a matching set is a "clone" (in the general meaning of the term), but satellite-clone does more than cloning rpms/configs, it also does data.

How we can get to a good clone of the slowly moving parts only: I need to think more about it :D

I created issue #3714 where we can track this. According to #3679 (review), this open item doesn't block merging.

Co-authored-by: Evgeni Golov <[email protected]>

ehelms

I am good to merge the current state.

--------- Co-authored-by: Eric Helms <[email protected]> Co-authored-by: Evgeni Golov <[email protected]> (cherry picked from commit 79a2600)

aneta-petrova · 2025-03-06T08:57:27Z

Merged to "master" and cherry-picked:

a8cd1ff..f564f69 3.14 -> 3.14

pr-processor bot added the Not yet reviewed label Feb 18, 2025

github-actions bot added Needs tech review Requires a review from the technical perspective Needs style review Requires a review from docs style/grammar perspective Needs testing Requires functional testing labels Feb 18, 2025

aneta-petrova removed the Needs testing Requires functional testing label Feb 18, 2025

ehelms reviewed Feb 19, 2025

View reviewed changes

...dules/con_disaster-recovery-with-active-and-passive-project-server-and-external-storage.adoc Outdated Show resolved Hide resolved

pr-processor bot removed the Not yet reviewed label Feb 19, 2025

ehelms reviewed Feb 19, 2025

View reviewed changes

...dules/con_disaster-recovery-with-active-and-passive-project-server-and-external-storage.adoc Outdated Show resolved Hide resolved

ehelms reviewed Feb 19, 2025

View reviewed changes

...ring-for-disaster-recovery-with-active-and-passive-project-server-with-external-storage.adoc Outdated Show resolved Hide resolved

ehelms reviewed Feb 19, 2025

View reviewed changes

...ring-for-disaster-recovery-with-active-and-passive-project-server-with-external-storage.adoc Outdated Show resolved Hide resolved

ehelms reviewed Feb 19, 2025

View reviewed changes

...ring-for-disaster-recovery-with-active-and-passive-project-server-with-external-storage.adoc Outdated Show resolved Hide resolved

aneta-petrova requested a review from rh-max February 19, 2025 10:28

rh-max suggested changes Feb 26, 2025

View reviewed changes

pr-processor bot added the Waiting on contributor Requires an action from the author label Feb 26, 2025

evgeni reviewed Feb 27, 2025

View reviewed changes

...dules/con_disaster-recovery-with-active-and-passive-project-server-and-external-storage.adoc Outdated Show resolved Hide resolved

aneta-petrova mentioned this pull request Feb 28, 2025

Suggestions to improve disaster recovery overview #3704

Closed

aneta-petrova commented Feb 28, 2025

View reviewed changes

pr-processor bot added Needs re-review and removed Waiting on contributor Requires an action from the author Needs re-review labels Feb 28, 2025

aneta-petrova requested review from ehelms and evgeni February 28, 2025 20:32

aneta-petrova mentioned this pull request Feb 28, 2025

Move parent-context from master.adoc to assembly #3705

Merged

10 tasks

ehelms reviewed Mar 3, 2025

View reviewed changes

...ring-for-disaster-recovery-with-active-and-passive-project-server-with-external-storage.adoc Show resolved Hide resolved

aneta-petrova marked this pull request as ready for review March 5, 2025 12:13

pr-processor bot added the Waiting on contributor Requires an action from the author label Mar 5, 2025

aneta-petrova and others added 7 commits March 5, 2025 13:23

Add DR with active-passive and external storage

652a028

Update NAS reference

a790e6a

Co-authored-by: Eric Helms <[email protected]>

Add link to RHEL storage docs

22f81c0

Specify how to mimic failure

806520f

Address feedback from tech and style review

e63e39f

Remove a few questions in comments

bfe0868

Clarify verification/testing steps

3ceb94f

aneta-petrova force-pushed the dr-2-external branch from 5b988b8 to 306f0f9 Compare March 5, 2025 12:42

pr-processor bot added Needs re-review and removed Waiting on contributor Requires an action from the author Needs re-review labels Mar 5, 2025

aneta-petrova added style review done No issues from docs style/grammar perspective and removed Needs style review Requires a review from docs style/grammar perspective labels Mar 5, 2025

evgeni reviewed Mar 5, 2025

View reviewed changes

...ring-for-disaster-recovery-with-active-and-passive-project-server-with-external-storage.adoc Outdated Show resolved Hide resolved

evgeni reviewed Mar 5, 2025

View reviewed changes

aneta-petrova and others added 3 commits March 5, 2025 18:08

Rename DR intro module and assembly

decca91

Add 'check service status' proc to DR with ext storage

5107d3c

Clarify intro to DR with external storage

f95ffc1

Co-authored-by: Evgeni Golov <[email protected]>

aneta-petrova force-pushed the dr-2-external branch from ede6556 to f95ffc1 Compare March 5, 2025 17:08

ehelms approved these changes Mar 6, 2025

View reviewed changes

aneta-petrova mentioned this pull request Mar 6, 2025

Figure out the details of cloning for DR with external storage #3714

Open

aneta-petrova added tech review done No issues from the technical perspective and removed Needs tech review Requires a review from the technical perspective labels Mar 6, 2025

aneta-petrova merged commit 79a2600 into theforeman:master Mar 6, 2025
8 of 9 checks passed

aneta-petrova deleted the dr-2-external branch March 6, 2025 08:55

aneta-petrova added a commit that referenced this pull request Mar 6, 2025

Add DR with active-passive and external storage (#3679)

f564f69

--------- Co-authored-by: Eric Helms <[email protected]> Co-authored-by: Evgeni Golov <[email protected]> (cherry picked from commit 79a2600)

		. Clone your active {ProjectServer}.
		See xref:cloning_satellite_server[].

Add DR with active-passive and external storage #3679

Add DR with active-passive and external storage #3679

Uh oh!

Conversation

aneta-petrova commented Feb 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes are you introducing?

Why are you introducing these changes? (Explanation, links to references, issues, etc.)

Anything else to add? (Considerations, potential downsides, alternative solutions you have explored, etc.)

Checklists

Uh oh!

github-actions bot commented Feb 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

aneta-petrova commented Feb 18, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

rh-max left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

aneta-petrova commented Feb 28, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

aneta-petrova commented Feb 28, 2025

Uh oh!

aneta-petrova commented Feb 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

aneta-petrova commented Mar 5, 2025

Uh oh!

aneta-petrova commented Mar 5, 2025

Uh oh!

Uh oh!

evgeni Mar 5, 2025

Choose a reason for hiding this comment

Uh oh!

aneta-petrova Mar 5, 2025

Choose a reason for hiding this comment

Uh oh!

evgeni Mar 5, 2025

Choose a reason for hiding this comment

Uh oh!

aneta-petrova Mar 6, 2025

Choose a reason for hiding this comment

Uh oh!

ehelms left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

aneta-petrova commented Mar 6, 2025

Uh oh!

Uh oh!

aneta-petrova commented Feb 18, 2025 •

edited

Loading

github-actions bot commented Feb 18, 2025 •

edited

Loading

aneta-petrova commented Feb 28, 2025 •

edited

Loading