diff --git a/build/dev/README.md b/build/dev/README.md index 0ce67de06c..c7c83cd8b1 100644 --- a/build/dev/README.md +++ b/build/dev/README.md @@ -46,6 +46,10 @@ When scaling up the local ecosystem by increasing `NUM_USS` or `NUM_NODES`, ther ### Host Port Mapping Limit (Max 99 Total Nodes) The script maps container ports to host ports using a two-digit padded index (`PADDED_NODE_IDX` from `01` to `99`). So, NUM_USS*NUM_NODES may not exceed 99 without adjusting run_locally.sh. +### Static IPs of DSS and DB containers limits (Max 128 Nodes) +In the `dss_internal_network` network the `dss` and `crdb` or `ybdb` containers get assigned a static IP based on their USS and node IDs. In the last byte of the IP address, the first bit is used as a flag distinguishing the DSS from the DB container. E.g. for USS 3 node 2 the IPs would be `172.27.3.2` for the DSS container and `172.27.3.130` for the DB one. I.e. 7 bits only are available for the IP in the subnet, limiting to 128 the maximum number of nodes. Do note that in any case the previous limitation of 99 nodes would be reached before. + + ### Dynamic IP Limit (Max ~250 Total Nodes) The `dss_internal_network` is created with a dynamic IP pool range of `172.27.0.0/24`. All DSS containers, bootstrap, and init containers are assigned dynamic IPs from this pool. Since `/24` has **253** usable host IPs (with `172.27.0.1` as gateway), starting more than ~250 containers requiring dynamic IPs will cause Docker to run out of IPs. In addition to 2 containers per node, there are also bootstrap containers. diff --git a/build/dev/docker-compose.yaml b/build/dev/docker-compose.yaml index 42a4887992..a1d6186ed0 100644 --- a/build/dev/docker-compose.yaml +++ b/build/dev/docker-compose.yaml @@ -15,7 +15,7 @@ services: hostname: db${USS_NODE_IDX:?}.uss${USS_IDX:?}.localutm networks: dss_internal_network: - ipv4_address: 172.27.${USS_IDX:?}.${USS_NODE_IDX:?} + ipv4_address: ${DB_IP:?} ports: - "81${PADDED_NODE_IDX:?}:8080" - "262${PADDED_NODE_IDX:?}:26257" @@ -43,7 +43,7 @@ services: hostname: db${USS_NODE_IDX:?}.uss${USS_IDX:?}.localutm networks: dss_internal_network: - ipv4_address: 172.27.${USS_IDX:?}.${USS_NODE_IDX:?} + ipv4_address: ${DB_IP:?} ports: - "70${PADDED_NODE_IDX:?}:7000" - "90${PADDED_NODE_IDX:?}:9000" @@ -171,8 +171,9 @@ services: required: false hostname: dss${USS_NODE_IDX:?}.uss${USS_IDX:?}.localutm networks: - - dss_internal_network - - interop_ecosystem_network + dss_internal_network: + ipv4_address: ${DSS_IP:?} + interop_ecosystem_network: ports: - "40${PADDED_NODE_IDX:?}:4000" - "80${PADDED_NODE_IDX:?}:80" diff --git a/build/dev/run_locally.sh b/build/dev/run_locally.sh index 84b7af11d7..80ce058c58 100755 --- a/build/dev/run_locally.sh +++ b/build/dev/run_locally.sh @@ -80,6 +80,11 @@ for ((i=1; i<=NUM_USS; i++)); do export COMPOSE_PROFILES=${COMPOSE_PROFILES},bootstrap-${DB_TYPE} fi + # keep the DSS and the DB in the same subnet by using the first bit of the last byte of the IP + # e.g. for USS 3 node 2 the IPs would be 172.27.3.2 for the DSS container and 172.27.3.130 for the DB container + export DSS_IP="172.27.$USS_IDX.$USS_NODE_IDX" + export DB_IP="172.27.$USS_IDX.$((2#10000000 | USS_NODE_IDX))" + # shellcheck disable=SC2086 docker compose -f docker-compose.yaml -p "local_infra_${USS_IDX}-${USS_NODE_IDX}" $DC_COMMAND $DC_OPTIONS & sleep 0.1 # reduce probability of race condition in joining network at container start