EcChronos uses LOCAL_QUORUM consistency causing failures

[Thiagoag]

EcChronos fails to perform operations when using LOCAL_QUORUM consistency level in datacenters with only one Cassandra node, as LOCAL_QUORUM requires multiple nodes within the same datacenter.

Operations fail with consistency level errors when datacenter has insufficient replicas to satisfy LOCAL_QUORUM requirements.

com.datastax.oss.driver.api.core.servererrors.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM

Cannot achieve consistency level LOCAL_QUORUM with 1 nodes

[tommystendahl]

Hi @Thiagoag,
The error message indicates that you have a replication factor of more then one but you say that you only have one node deployed so thats not going to work. If you only have one Cassandra node you must use a replication factor of 1 otherwise Cassandra will not work, this is not a ecchronos issue, this is your Cassandra cluster that is configured incorrectly.
But I'm not sure how you manage to set a rplication factor of more then 1 with a cluster of only one node, I thought Cassandra would throw an error for that.

[Thiagoag]

Hi @tommystendahl .
I am deploying one datacenter with two replicas. So I have two cassandra pods in that dc. Sorry I forgot to add that information.

[tommystendahl]

Well, you specificly said in the description that you had one Cassandra node, but if you have two how do you set the replication for the ecchronos keyspace?
Still you describe no problems in ecchronos, what you describe is a problem in the Cassandra cluster.

[Thiagoag]

It is setting based on the number of replicas configured, since we have two, it sets the replication factor to 2. Why is that a problem in Cassandra cluster?

[tommystendahl]

I'm not saying the RF2 is wrong, but something seams to be wrong with your cluster. There could be a node down or the nodes don't communicate or something else.

But I don't see anything here related to ecchronos. You have one DC with two nodes and RF2, that's fine. And ecchronos connect to the cassandra cluster, the request is sent to the coordinator and there is a response from the coordinator. But the response say that cassandra can't achieve the consistence level, that's some thing going wrong in cassandra and you need to investigate your cluster.

[Thiagoag]

That issue happened during an upgrade operation. After the cassandra pods conclude the operation, Ecchronos agent was able to connect to them. But during the process, ecchronos restart once or twice, logging that error.

[tommystendahl]

Ok, I think that explains everything. The problem is that with 2 nodes and RF=2 you have no redundancy in the cluster and both nodes must always be up and available for LOCAL_QUORUM to work.
If you setup three nodes with RF=3 and try again it should work, and 3 nodes with RF=3 is a much more realistic setup.

[Thiagoag]

Hi Tommy. I've tested with 3 replicas and it worked fine like you suggested. But I wonder if there is a reason for EcChronos to use LOCAL_QUORUM hardcoded. Or, could you make it configurable? This way we could use LOCAL_ONE when needed.

[tommystendahl]

Well, why not. Its probably very much a nisch use case with very limited use, even in a test environment, but it should be easy to implement. The risks with using it would have to be properly documented. Its not something needed for the first release and it can be added later so I can open a separate issue for that.

If you have no further questions around this I think we can close this issue?

[Thiagoag]

Another option, instead of making consistency configurable, would be to implement a retry mechanism for this operation that is failing. What do you think? That way we could keep the actual consistency and make ecchronos startup more robust.

[tommystendahl]

Since I don't know what request that is failing here its hard to say exactly. But there are retries and failed repair job are picked up later and will be done when the Cassandra cluster is in a working order. Its also important to remember that if you leave ecchrons running during an upgrade like this Cassandra will not be able to do any repair and repair jobs are expected to fail, but they will be picked up later. The consistency level (LOCAL_ONE or LOCAL_QUORUM) will not change this.

You have not reported any errors or unexpected behavior in ecchronos here, only a failed request in Cassandra, so I'm not sure what you are actually after here. Unless you have an ecchronos issue I think we should close this issue.