Skip to content

Use msgrv2 always for all ODF modes(internal,external,provider) #2793

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
Oct 17, 2024
Merged

Use msgrv2 always for all ODF modes(internal,external,provider) #2793

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
c80db15
Use msgrv2 always for all ODF modes(internal,external,provider)
malayparida2000 Sep 11, 2024
52899eb
Fix unit tests
malayparida2000 Sep 11, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 4 additions & 6 deletions controllers/storagecluster/cephcluster.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -605,13 +605,11 @@ func getNetworkSpec(sc ocsv1.StorageCluster) rookCephv1.NetworkSpec {
// respect both the old way and the new way for enabling HostNetwork
networkSpec.HostNetwork = networkSpec.HostNetwork || sc.Spec.HostNetwork

// If it's not an external and not a provider cluster always require msgr2
if !sc.Spec.AllowRemoteStorageConsumers && !sc.Spec.ExternalStorage.Enable {
if networkSpec.Connections == nil {
networkSpec.Connections = &rookCephv1.ConnectionsSpec{}
}
networkSpec.Connections.RequireMsgr2 = true
if networkSpec.Connections == nil {
networkSpec.Connections = &rookCephv1.ConnectionsSpec{}
}
// Always require msgr2 for all modes of ODF
networkSpec.Connections.RequireMsgr2 = true
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am not sure we can blindly change that for provider mode. What happens to existing deployments?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For existing deployments the mons after the upgrade would have the v1 port still available, so there would be no problem for the existing volumes that are already mounted. For any new volumes that are created after the upgrade, the volume would use the v2 port.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@malayparida2000 I am not sure that will work out of the box, don't we need to inform the CSI running on the clients that we moved to v2? If so how do we do that in provider mode? the CSI is running on many different clusters

Copy link
Contributor Author

@malayparida2000 malayparida2000 Oct 8, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When the ODF version upgrade happens from 4.17 to 4.18 where this change will come in, The subscription channel would change for ocs-operator, which would trigger a change in the hash for the clients & would trigger a reconciliation for them based on this

ocs-operator/services/provider/server/server.go

Line 860 in f80f43b

channelName,

When that happens the client would now get the new info from the provider cluster. That's my understanding.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

would now get the new info from the provider cluster

  • this config will be pushed to clients right after provider gets upgraded but not when the client gets upgraded
  • nevertheless, based on all the info presented even 4.17 clients will have v2 endpoint which should be supported by kernel on client clusters

don't we need to inform the CSI running on the clients that we moved to v2

  • two changes will happen after provider upgrade, new v2 endpoints and mount option ms_mode=prefer-crc will be pushed to client clusters and IIUC they can be changed on the fly w/o any issues for both existing and new workloads.


return networkSpec
}
Expand Down
8 changes: 7 additions & 1 deletion controllers/storagecluster/cephcluster_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1290,7 +1290,9 @@ func TestCephClusterNetworkConnectionsSpec(t *testing.T) {
},
ccSpec: rookCephv1.ClusterSpec{
Network: rookCephv1.NetworkSpec{
Connections: &rookCephv1.ConnectionsSpec{},
Connections: &rookCephv1.ConnectionsSpec{
RequireMsgr2: true,
},
},
},
},
Expand All @@ -1308,6 +1310,7 @@ func TestCephClusterNetworkConnectionsSpec(t *testing.T) {
ccSpec: rookCephv1.ClusterSpec{
Network: rookCephv1.NetworkSpec{
Connections: &rookCephv1.ConnectionsSpec{
RequireMsgr2: true,
Encryption: &rookCephv1.EncryptionSpec{
Enabled: true,
},
Expand All @@ -1329,6 +1332,7 @@ func TestCephClusterNetworkConnectionsSpec(t *testing.T) {
ccSpec: rookCephv1.ClusterSpec{
Network: rookCephv1.NetworkSpec{
Connections: &rookCephv1.ConnectionsSpec{
RequireMsgr2: true,
Compression: &rookCephv1.CompressionSpec{
Enabled: true,
},
Expand All @@ -1353,6 +1357,7 @@ func TestCephClusterNetworkConnectionsSpec(t *testing.T) {
ccSpec: rookCephv1.ClusterSpec{
Network: rookCephv1.NetworkSpec{
Connections: &rookCephv1.ConnectionsSpec{
RequireMsgr2: true,
Encryption: &rookCephv1.EncryptionSpec{
Enabled: true,
},
Expand Down Expand Up @@ -1380,6 +1385,7 @@ func TestCephClusterNetworkConnectionsSpec(t *testing.T) {
ccSpec: rookCephv1.ClusterSpec{
Network: rookCephv1.NetworkSpec{
Connections: &rookCephv1.ConnectionsSpec{
RequireMsgr2: true,
Encryption: &rookCephv1.EncryptionSpec{
Enabled: false,
},
Expand Down
17 changes: 1 addition & 16 deletions controllers/util/csi_options.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,22 +13,7 @@ func GetCephFSKernelMountOptions(sc *ocsv1.StorageCluster) string {
return "ms_mode=secure"
}

// If Encryption is not enabled, but Compression or RequireMsgr2 is enabled, use prefer-crc mode
if sc.Spec.Network != nil && sc.Spec.Network.Connections != nil &&
((sc.Spec.Network.Connections.Compression != nil && sc.Spec.Network.Connections.Compression.Enabled) ||
sc.Spec.Network.Connections.RequireMsgr2) {
return "ms_mode=prefer-crc"
}

// Network spec always has higher precedence even in the External or Provider cluster. so they are checked first above

// None of Encryption, Compression, RequireMsgr2 are enabled on the StorageCluster
// If it's an External or Provider cluster, We don't require msgr2 by default so no mount options are needed
if sc.Spec.ExternalStorage.Enable || sc.Spec.AllowRemoteStorageConsumers {
return "ms_mode=legacy"
}
// If none of the above cases apply, We set RequireMsgr2 true by default on the cephcluster
// so we need to set the mount options to prefer-crc
// If encryption is not enabled, use prefer-crc mode
return "ms_mode=prefer-crc"
}

Expand Down
2 changes: 1 addition & 1 deletion services/provider/server/server_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -769,7 +769,7 @@ func TestOCSProviderServerGetStorageClaimConfig(t *testing.T) {
CephFs: &csiopv1a1.CephFsConfigSpec{
SubVolumeGroup: "cephFilesystemSubVolumeGroup",
KernelMountOptions: map[string]string{
"ms_mode": "legacy",
"ms_mode": "prefer-crc",
},
},
},
Expand Down
Loading