Implement all required bits to C/R MPTCP connections from user-space (CRIU)

[avagin]

Pre-requisites

• A similar idea has not been reported before.
• mptcp.dev website does not cover my case.
• The wiki doesn't cover my case.

Description

As MPTCP gains wider adoption, we start getting more requests to support MPTCP socket in the CRIU project.

CRIU (Checkpoint/Restore In Userspace) is a tool that freezes a running application and checkpoints its state to disk. This allows the application to be restored and run exactly as it was at the time of the freeze. CRIU enables use cases like live migration, snapshotting, etc.

Solution

CRIU can dump/restore TCP connections and now we need to implement something similar to support MPTCP.
Here is an initial patchset for tcp sockets: https://lore.kernel.org/netdev/4F901572.4040009@parallels.com/
The tcp user-space part is here: https://github.com/checkpoint-restore/criu/tree/criu-dev/soccr

[matttbe]

Hi Andrei,

Thank you for having opened this ticket.

Currently, I confirm MPTCP sockets don't support any of the TCP_REPAIR* socket options. This has not been implemented mainly because it looked too complex, and we were not sure who would need it.

On the other hand, I don't think anybody was "against them". If someone knowing well this feature is happy to look at that, I'm sure this can be accepted in MPTCP. More info needs to be exported and imported, e.g. the keys. By chance, do you know someone interested to look at that? :)

(Not directly related to TCP_REPAIR* socket options, but with MPTCP, we could imagine a different way to do the migrations without interruptions: the current subflows could be closed (TCP-FIN) while not closing the MPTCP connection (no DATA_FIN), then the MPTCP socket could be duplicated elsewhere, and create/accept a new subflow (path) from the other peer or the previous socket to continue the transfer. That should in theory work, but can also be complex)

[avagin]

By chance, do you know someone interested to look at that? :)

I don't. I will let you know if I found someone.

(Not directly related to TCP_REPAIR* socket options, but with MPTCP, we could imagine a different way to do the migrations without interruptions: the current subflows could be closed (TCP-FIN) while not closing the MPTCP connection (no DATA_FIN), then the MPTCP socket could be duplicated elsewhere, and create/accept a new subflow (path) from the other peer or the previous socket to continue the transfer. That should in theory work, but can also be complex)

I mentioned TCP_REPAIR because I thought MPTCP should be similar to TCP. I could be wrong. We are open to any solution that will allow us to restore MPTCP connections.

[matttbe]

By chance, do you know someone interested to look at that? :)

I don't. I will let you know if I found someone.

👍

(Not directly related to TCP_REPAIR* socket options, but with MPTCP, we could imagine a different way to do the migrations without interruptions: the current subflows could be closed (TCP-FIN) while not closing the MPTCP connection (no DATA_FIN), then the MPTCP socket could be duplicated elsewhere, and create/accept a new subflow (path) from the other peer or the previous socket to continue the transfer. That should in theory work, but can also be complex)

I mentioned TCP_REPAIR because I thought MPTCP should be similar to TCP. I could be wrong. We are open to any solution that will allow us to restore MPTCP connections.

MPTCP tries to be as closed as possible to TCP, not to have to modify applications (except at socket creation time, to request an MPTCP one instead of TCP). But in the case of TCP_REPAIR, I guess it cannot be as transparent, because an MPTCP connection has more state to transfer: keys, multiple subflows (= path = TCP connection with extra TCP options). Maybe something else would be needed, maybe a dedicated socket option under the SOL_MPTCP level?

[milantracy]

(Not directly related to TCP_REPAIR* socket options, but with MPTCP, we could imagine a different way to do the migrations without interruptions: the current subflows could be closed (TCP-FIN) while not closing the MPTCP connection (no DATA_FIN), then the MPTCP socket could be duplicated elsewhere, and create/accept a new subflow (path) from the other peer or the previous socket to continue the transfer. That should in theory work, but can also be complex)

Comparing the solution to TCP_REPAIR, I realized that it is equally complex to TCP_REPAIR, I am also curious about the feasibility for the checkpoint/restore case.

In order to checkpoint and restore, IIUC, it has to leave MPTCP open, a MPTCP socket will be checkpoint and restored as a normal TCP socket like what https://criu.org/TCP_connection describes. However, since the peer has not received no DATA_FIN, how does it deal with the remaining data?

[matttbe]

(Not directly related to TCP_REPAIR* socket options, but with MPTCP, we could imagine a different way to do the migrations without interruptions: the current subflows could be closed (TCP-FIN) while not closing the MPTCP connection (no DATA_FIN), then the MPTCP socket could be duplicated elsewhere, and create/accept a new subflow (path) from the other peer or the previous socket to continue the transfer. That should in theory work, but can also be complex)

Comparing the solution to TCP_REPAIR, I realized that it is equally complex to TCP_REPAIR, I am also curious about the feasibility for the checkpoint/restore case.

With TCP_REPAIR and MPTCP, there will be more info to export: info about the MPTCP socket, but also each subflows. It sounds cleaner/easier to close all subflows without closing the MPTCP connection, only deal with the MPTCP socket, and re-create new subflows later. Even more info could be exported for each subflows, but I don't think that's worth it.

Also, with this technique, and thanks to MPTCP, it is possible to switch to new IP addresses, no need to recreate the exact same 5-tuple like with TCP, and thus, no need to change the network config around to route packets to the new place.

In order to checkpoint and restore, IIUC, it has to leave MPTCP open, a MPTCP socket will be checkpoint and restored as a normal TCP socket like what https://criu.org/TCP_connection describes.

Indeed, it looks similar, just different pieces of data to export. Everything specific to MPTCP, and needed to recreate the same state (token, etc.)

However, since the peer has not received no DATA_FIN, how does it deal with the remaining data?

If the peer has no subflows to send anything, it will simply wait before sending data. Similar to the case where the other peer announces a zero window.

Note that when there is no more subflows attached to an MPTCP connection, there is a timeout before closing the connection, controlled by net.mptcp.close_timeout sysctl knob. By default, it is 1 minute once all subflows have been marked as closed, but this can easily be changed.

To implement this feature, it might be easier to add MPTCP-specific socket options, in SOL_MPTCP level.