[raft/ci] CI Check with Raft cluster

[MariemBaccari]

Adds a CI check for Raft (probe, qualify and security locally) that uses 3 nodes and a load balancer to distribute requests across them.

Implements #1703

[mickmis]

I'm a bit conflicted by this: I see the value in running those tests on the clustered raft nodes in the CI of this repo, especially since now with raft it is significantly easier to do (no separate DB service or whatever).
However now it looks like by doing it that way we are just recreating little by little the stack of the monitoring repository and duplicating our testing stacks, which are getting pretty complex on top of that.

Given that we already have part of the DSS CI relying on the monitoring image for running the USS qualifier and prober, maybe it could make sense relying on it too (or the repo) for the deployment of the full testing stack...
WDYT? Interested in your input on this @BenjaminPelletier @barroco.

[MariemBaccari]

@mickmis Just to add to what you already mentioned, I think having a way to test a cluster on the CI is important since we implement a consensus layer that can "easily" be broken. This was not the case with CRDB.

[barroco]

Given that we already have part of the DSS CI relying on the monitoring image for running the USS qualifier and prober, maybe it could make sense relying on it too (or the repo) for the deployment of the full testing stack...
WDYT?

The monitoring repository only contains test suites, which should cover as much as possible any DSS implementation. As we introduce the more specialized testing described in the design document, we will need additional infrastructure here, since it is highly implementation-specific.

[mickmis]

FTR, discussed offline: for now let's duplicate the tooling from monitoring in this repo so that the two may stay relatively aligned while still benefiting from running the CI test on multi-nodes raft in this repo. In parallel discuss the strategy regarding the testing stack to come up with something that makes sense.

[mickmis]

Note: partial review. IMO that's a bit too much changes and complexification on the existing scripts. I think we should be able to do that a bit differently and not have too refactor as much the existing scripts.
I'm not sure we need to keep the ports exposed strictly separated, they are not already as of now. I think it's reasonable to expect stopping down a stack before starting the other and that both cannot be fully live at the same time. However they should not interfere with each other when stopped: so a different docker compose project should be enough.

[mickmis]

Not for this PR, but those ports are required because of each service (rid, scd, aux) right?
IMO we should try to reduce that to one port, that really simplifies deployment. Could you open an issue regarding that?

[MariemBaccari]

Opened #1735 to track this although I'm not sure yet how it would work exactly.

[mickmis]

LGTM modulo minor comments.
I will make a commit with those changes.