Trying to achieve HA with two monitoring nodes and Loki simple scalable

[kuckenberg]

Hi everyone,

I am using the simple scalable helm chart of Loki and have two nodes in my monitoring kubernetes cluster.
So far so good, this config is working fine, from the official grafana docs:

loki:
  schemaConfig:
    configs:
      - from: "2024-04-01"
        store: tsdb
        object_store: s3
        schema: v13
        index:
          prefix: loki_index_
          period: 24h
  ingester:
    chunk_encoding: snappy
  tracing:
    enabled: true
  querier:
    max_concurrent: 4


deploymentMode: SimpleScalable

backend:
  replicas: 2
read:
  replicas: 2
write:
  replicas: 2


minio:
  enabled: true
  replicas: 2



singleBinary:
  replicas: 0

ingester:
  replicas: 0
querier:
  replicas: 0
queryFrontend:
  replicas: 0
queryScheduler:
  replicas: 0
distributor:
  replicas: 0
compactor:
  replicas: 0
indexGateway:
  replicas: 0
bloomCompactor:
  replicas: 0
bloomGateway:
  replicas: 0 

Now I was trying test if i everything stays up and running if i kill a node in the cluster, so I did and I kept getting some error along these lines: "too many unhealthy instances in the ring". After reading into the replication factor I realized it probably happens because I am running only 2 instead of 3 replicas each and the default factor is 3.

I have tried setting replication_factor=1 but that seems to still need 2 active replicas for a successful write. According to the docs the factor results from ( replication_factor / 2 ) + 1.
A factor of 0 is not allowed.

My question is: is there a way of making my setup work?

I appreciate any input.

Thanks

[kuckenberg]

I have now also tried using the deploymentMode: SingleBinary and replicas: 2 for it. It also does not work with either settings for replication_factor, the default 3 nor 1.

Is there no way of running Loki in a highly available matter with just 2 replicas?

[gjrtimmer]

I have the same issue, two replicas with replication_factor: 3 like the docs says, does not work.

[JStickler]

Configuration questions have a better chance of being answered if you ask them on the community forums.