Taurus and mainnet infra changes and refactor #352

DaMandal0rian · 2024-10-23T16:21:47Z

PR Type

enhancement, configuration changes

Description

Refactored and added new AWS instance resources, including bootstrap_node_evm, bootstrap_node_autoid, rpc_indexer_node and nova_indexer_node.
Added provisioning scripts for nova subql, nova indexer, farmer, RPC subql, RPC indexer, RPC, and bootstrap nodes.
Configured network infrastructure with VPC, subnets, and security groups.
Set up Cloudflare DNS records for various nodes, including A and AAAA records.
Updated and renamed node configuration variables and instance counts.
change CLI parameters for taurus and mainnet

Changes walkthrough 📝

Relevant files

Enhancement

11 files

instances.tf `Refactor and add new AWS instance resources` templates/terraform/network-primitives-archive/gemini-3h/instances.tf Replaced `rpc_indexer_node` with `bootstrap_node_evm`. Added `bootstrap_node_autoid` and `rpc_squid_node` resources. Updated tags and dependencies for new resources.	+206/-206
instances.tf `Modify instance configurations and add RPC indexer node` templates/terraform/network-primitives/instances.tf Commented out `bootstrap_node_evm` and `bootstrap_node_autoid`. Added `rpc_indexer_node` resource. Updated instance type and volume configurations.	+206/-206
nova_squid_node_provisioner.tf `Add nova squid node provisioning script` templates/terraform/network-primitives-archive/gemini-3h/nova_squid_node_provisioner.tf Added provisioning for nova squid nodes. Included setup, prune, and start resources. Configured SSH connections and Docker setup.	+186/-1
nova_indexer_node_provisioner.tf `Add nova indexer node provisioning script` templates/terraform/network-primitives/nova_indexer_node_provisioner.tf Added provisioning for nova indexer nodes. Included setup, prune, and start resources. Configured SSH connections and Docker setup.	+186/-1
farmer_node_provisioner.tf `Add farmer node provisioning script` templates/terraform/network-primitives-archive/gemini-3h/farmer_node_provisioner.tf Added provisioning for farmer nodes. Included setup, prune, and start resources. Configured SSH connections and Docker setup.	+175/-1
rpc_squid_node_provisioner.tf `Add RPC squid node provisioning script` templates/terraform/network-primitives-archive/gemini-3h/rpc_squid_node_provisioner.tf Added provisioning for RPC squid nodes. Included setup, prune, and start resources. Configured SSH connections and Docker setup.	+175/-1
rpc_indexer_node_provisioner.tf `Add RPC indexer node provisioning script` templates/terraform/network-primitives/rpc_indexer_node_provisioner.tf Added provisioning for RPC indexer nodes. Included setup, prune, and start resources. Configured SSH connections and Docker setup.	+175/-1
rpc_node_provisioner.tf `Add RPC node provisioning script` templates/terraform/network-primitives-archive/gemini-3h/rpc_node_provisioner.tf Added provisioning for RPC nodes. Included setup, prune, and start resources. Configured SSH connections and Docker setup.	+175/-1
domain_node_provisioner.tf `Refactor EVM node provisioning to domain nodes` templates/terraform/network-primitives-archive/gemini-3h/domain_node_provisioner.tf Refactored EVM node provisioning to domain nodes. Updated resource names and configurations. Configured SSH connections and Docker setup.	+29/-29
domain_node_provisioner.tf `Refactor domain node provisioning to EVM nodes` templates/terraform/network-primitives/domain_node_provisioner.tf Refactored domain node provisioning to EVM nodes. Updated resource names and configurations. Configured SSH connections and Docker setup.	+29/-29
bootstrap_node_provisioner.tf `Add bootstrap node provisioning script` templates/terraform/network-primitives-archive/gemini-3h/bootstrap_node_provisioner.tf Added provisioning for bootstrap nodes. Included setup, prune, and start resources. Configured SSH connections and Docker setup.	+161/-1

Configuration changes

4 files

network.tf `Add network infrastructure configurations for VPC and subnets` templates/terraform/network-primitives-archive/gemini-3h/network.tf Added VPC, subnet, and security group configurations. Configured route tables and internet gateways. Set up ingress and egress rules for various ports.	+223/-1
main.tf `Configure Taurus module with gemini-3h network settings` resources/taurus/main.tf Added module configuration for `gemini-3h`. Defined various node configurations and parameters. Included Cloudflare and AWS credentials.	+172/-0
dns.tf `Add Cloudflare DNS configuration for network nodes` templates/terraform/network-primitives-archive/gemini-3h/dns.tf Added Cloudflare DNS records for various nodes. Configured A and AAAA records for IPv4 and IPv6. Set up DNS for bootstrap, RPC, and domain nodes.	+108/-1
variables.tf `Update and rename node configuration variables` templates/terraform/network-primitives-archive/gemini-3h/variables.tf Updated instance count defaults for various nodes. Renamed node configuration variables. Adjusted variable descriptions and types.	+14/-30

Additional files (token-limit)

50 files

variables.tf `...` resources/taurus/variables.tf ...	+132/-0
outputs.tf `...` templates/terraform/network-primitives-archive/gemini-3h/outputs.tf ...	+26/-26
outputs.tf `...` templates/terraform/network-primitives/outputs.tf ...	+26/-26
variables.tf `...` templates/terraform/network-primitives/variables.tf ...	+30/-14
bootstrap_node_autoid_provisioner.tf `...` templates/terraform/network-primitives-archive/gemini-3h/bootstrap_node_autoid_provisioner.tf ...	+3/-3
bootstrap_node_evm_provisioner.tf `...` templates/terraform/network-primitives-archive/gemini-3h/bootstrap_node_evm_provisioner.tf ...	+3/-3
bootstrap_node_autoid_provisioner.tf `...` templates/terraform/network-primitives/bootstrap_node_autoid_provisioner.tf ...	+3/-3
bootstrap_node_evm_provisioner.tf `...` templates/terraform/network-primitives/bootstrap_node_evm_provisioner.tf ...	+3/-3
outputs.tf `...` resources/taurus/outputs.tf ...	+51/-0
autoid_node_provisioner.tf `...` templates/terraform/network-primitives-archive/gemini-3h/autoid_node_provisioner.tf ...	+2/-2
autoid_node_provisioner.tf `...` templates/terraform/network-primitives/autoid_node_provisioner.tf ...	+2/-2
provider.tf `...` templates/terraform/network-primitives-archive/gemini-3h/provider.tf ...	+32/-1
main.tf `...` resources/gemini-3h/main.tf ...	+3/-3
ami.tf `...` templates/terraform/network-primitives-archive/gemini-3h/ami.tf ...	+21/-1
main.tf `...` resources/mainnet/main.tf ...	+3/-3
common.tf `...` resources/taurus/common.tf ...	+20/-0
backend.tf `...` resources/taurus/backend.tf ...	+9/-0
create_domain_node_compose_file.sh `...` templates/terraform/network-primitives-archive/gemini-3h/scripts/create_domain_node_compose_file.sh ...	+183/-0
create_bootstrap_node_domain_compose_file.sh `...` templates/terraform/network-primitives-archive/gemini-3h/scripts/create_bootstrap_node_domain_compose_file.sh ...	+182/-0
create_rpc_node_compose_file.sh `...` templates/terraform/network-primitives-archive/gemini-3h/scripts/create_rpc_node_compose_file.sh ...	+147/-0
create_bootstrap_node_compose_file.sh `...` templates/terraform/network-primitives-archive/gemini-3h/scripts/create_bootstrap_node_compose_file.sh ...	+151/-0
create_farmer_node_compose_file.sh `...` templates/terraform/network-primitives-archive/gemini-3h/scripts/create_farmer_node_compose_file.sh ...	+137/-0
create_bootstrap_node_domain_compose_file.sh `...` templates/scripts/create_bootstrap_node_domain_compose_file.sh ...	+8/-8
create_bootstrap_node_autoid_compose_file.sh `...` testing-framework/ec2/base/scripts/create_bootstrap_node_autoid_compose_file.sh ...	+8/-8
create_bootstrap_node_evm_compose_file.sh `...` testing-framework/ec2/base/scripts/create_bootstrap_node_evm_compose_file.sh ...	+8/-8
create_domain_node_compose_file.sh `...` templates/scripts/create_domain_node_compose_file.sh ...	+6/-6
create_autoid_node_compose_file.sh `...` testing-framework/ec2/base/scripts/create_autoid_node_compose_file.sh ...	+6/-6
create_domain_node_compose_file.sh `...` testing-framework/ec2/base/scripts/create_domain_node_compose_file.sh ...	+6/-6
create_bootstrap_node_compose_file.sh `...` testing-framework/ec2/base/scripts/create_bootstrap_node_compose_file.sh ...	+4/-4
create_bootstrap_node_compose_file.sh `...` templates/scripts/create_bootstrap_node_compose_file.sh ...	+4/-4
installer.sh `...` templates/terraform/network-primitives-archive/gemini-3h/scripts/installer.sh ...	+27/-0
create_rpc_node_compose_file.sh `...` templates/scripts/create_rpc_node_compose_file.sh ...	+4/-4
create_node_compose_file.sh `...` testing-framework/ec2/base/scripts/create_node_compose_file.sh ...	+4/-4
create_farmer_node_compose_file.sh `...` templates/scripts/create_farmer_node_compose_file.sh ...	+2/-4
create_farmer_node_compose_file.sh `...` testing-framework/ec2/base/scripts/create_farmer_node_compose_file.sh ...	+2/-4
create_rpc_node_compose_file.sh `...` templates/terraform/network-primitives-archive/gemini-3g/scripts/create_rpc_node_compose_file.sh ...	+1/-2
create_farmer_node_compose_file.sh `...` templates/terraform/network-primitives-archive/gemini-3g/scripts/create_farmer_node_compose_file.sh ...	+1/-1
create_full_node_compose_file.sh `...` templates/terraform/network-primitives-archive/gemini-3g/scripts/create_full_node_compose_file.sh ...	+1/-1
create_domain_node_compose_file.sh `...` templates/terraform/network-primitives-archive/gemini-3g/scripts/create_domain_node_compose_file.sh ...	+1/-1
acme.sh `...` templates/terraform/network-primitives-archive/gemini-3h/scripts/acme.sh ...	+14/-0
prune_docker_system.sh `...` templates/terraform/network-primitives-archive/gemini-3h/scripts/prune_docker_system.sh ...	+9/-0
docker-compose-bootstrap-autoid.yml `...` ansible/network/files/docker-compose-bootstrap-autoid.yml ...	+14/-14
docker-compose-bootstrap-domain.yml `...` ansible/network/files/docker-compose-bootstrap-domain.yml ...	+14/-14
docker-compose-autoid.yml `...` ansible/network/files/docker-compose-autoid.yml ...	+10/-10
docker-compose-domain.yml `...` ansible/network/files/docker-compose-domain.yml ...	+10/-10
docker-compose-rpc.yml `...` ansible/network/files/docker-compose-rpc.yml ...	+6/-6
docker-compose-bootstrap.yml `...` ansible/network/files/docker-compose-bootstrap.yml ...	+6/-6
docker-compose-farmer.yml `...` ansible/network/files/docker-compose-farmer.yml ...	+2/-6
terrafrom.tfvars.example `...` resources/taurus/terrafrom.tfvars.example ...	+17/-0
prometheus.yml `...` templates/terraform/network-primitives-archive/gemini-3h/configs/prometheus.yml ...	+8/-0

💡 PR-Agent usage: Comment /help "your question" on any pull request to receive relevant information

github-actions · 2024-10-23T16:24:17Z

PR Reviewer Guide 🔍

Here are some key observations to aid the review process:

⏱️ Estimated effort to review: 4 🔵🔵🔵🔵⚪
🧪 No relevant tests
🔒 No security concerns identified
⚡ Recommended focus areas for review Configuration Consistency Ensure that the instance configurations and resource dependencies are consistent and correctly set across different environments and node types. This includes checking security group settings, subnet configurations, and AMI IDs. Resource Naming Verify that the naming conventions for resources like instances, security groups, and tags are consistent and meaningful across the entire infrastructure codebase to avoid confusion and potential conflicts. Security Groups Review the security group assignments and rules for each instance type to ensure that only the necessary ports are exposed and that the configurations adhere to the principle of least privilege. Provisioning Scripts Check the provisioning scripts and remote-exec blocks for potential errors or inefficiencies in script execution, such as redundant commands or incorrect file paths. Instance Configuration Ensure that the instance configurations such as AMI, instance type, and EBS settings are optimized for cost and performance, and are aligned with the application requirements.

github-actions · 2024-10-23T16:25:14Z

PR Code Suggestions ✨

Explore these optional code suggestions:

Category	Suggestion	Score
Security	Improve security by tightening file permissions Consider using a more restrictive file permission setting for sensitive directories and files to enhance security. templates/terraform/network-primitives-archive/gemini-3h/nova_squid_node_provisioner.tf [35] -"sudo chown -R ${var.ssh_user}:${var.ssh_user} /home/${var.ssh_user}/subspace/ && sudo chmod -R 750 /home/${var.ssh_user}/subspace/" +"sudo chown -R ${var.ssh_user}:${var.ssh_user} /home/${var.ssh_user}/subspace/ && sudo chmod -R 700 /home/${var.ssh_user}/subspace/" Suggestion importance[1-10]: 9 Why: Tightening file permissions from 750 to 700 enhances security by restricting access to sensitive directories and files. This is a significant improvement for protecting sensitive data.	9
Possible bug	Improve the reliability of shell command variable expansion Ensure that the `count.index` variable is properly escaped in the `sed` command to prevent issues with variable expansion in shell commands. templates/terraform/network-primitives-archive/gemini-3h/nova_squid_node_provisioner.tf [172] -"echo NODE_KEY=$(sed -nr 's/NODE_${count.index}_KEY=//p' /home/${var.ssh_user}/subspace/node_keys.txt) >> /home/${var.ssh_user}/subspace/.env" +"echo NODE_KEY=$(sed -nr 's/NODE_'"${count.index}"'_KEY=//p' /home/${var.ssh_user}/subspace/node_keys.txt) >> /home/${var.ssh_user}/subspace/.env" Suggestion importance[1-10]: 8 Why: The suggestion addresses a potential issue with variable expansion in shell commands, which can lead to incorrect behavior. Properly escaping the variable ensures the command works as intended, enhancing reliability.	8
Enhancement	Enhance error handling in script execution to improve debugging and reliability Add error handling for the `remote-exec` provisioner commands to ensure that the script stops on errors and provides meaningful error messages. templates/terraform/network-primitives-archive/gemini-3h/nova_squid_node_provisioner.tf [60-61] +"set -e", "sudo bash /home/${var.ssh_user}/subspace/installer.sh", "bash /home/${var.ssh_user}/subspace/acme.sh", Suggestion importance[1-10]: 7 Why: Adding error handling with "set -e" improves script robustness by stopping execution on errors, which aids in debugging and ensures reliability. This is a valuable enhancement for maintaining script integrity.	7
Enhancement	Increase configuration flexibility by using a variable for SSH timeouts Replace the hard-coded SSH connection timeout with a variable to allow easy adjustment across different environments and scenarios. templates/terraform/network-primitives-archive/gemini-3h/nova_squid_node_provisioner.tf [28] -timeout = "300s" +timeout = var.ssh_timeout Suggestion importance[1-10]: 6 Why: Using a variable for SSH timeouts allows for easier adjustments across different environments, improving flexibility and maintainability. This is a beneficial enhancement for configuration management.	6

taurus module changes

DaMandal0rian added 8 commits October 23, 2024 18:39

changes to cli parameters ansible

ff6aac9

gemini-3h path changes

c7c9bf2

change cli parameters testing framework

8b641b7

change cli parameters for scripts

365b98f

delete mainnet primitives and use new network primitives

592b8fe

update network primitives folder

fdbbb5e

gemini-3h path changes

702ef20

mainnet path changes module

9a2c520

github-actions bot added the Review effort [1-5]: 4 label Oct 23, 2024

DaMandal0rian added 4 commits October 23, 2024 19:25

add taurus testnet module

4e3bc26

taurus module changes

ansible change cli parameters farmer

855ac80

edits to paths gemini-3g

503cca7

move gemini-3h primitives to archive

d472849

DaMandal0rian force-pushed the feat/taurus branch from 6545531 to d472849 Compare October 23, 2024 16:25

DaMandal0rian merged commit d4dbc32 into main Oct 23, 2024
1 check passed

DaMandal0rian deleted the feat/taurus branch October 23, 2024 16:29

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Taurus and mainnet infra changes and refactor #352

Taurus and mainnet infra changes and refactor #352

Uh oh!

DaMandal0rian commented Oct 23, 2024 •

edited

Loading

Uh oh!

github-actions bot commented Oct 23, 2024

Uh oh!

github-actions bot commented Oct 23, 2024

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Taurus and mainnet infra changes and refactor #352

Taurus and mainnet infra changes and refactor #352

Uh oh!

Conversation

DaMandal0rian commented Oct 23, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

PR Type

Description

Changes walkthrough 📝

Uh oh!

github-actions bot commented Oct 23, 2024

PR Reviewer Guide 🔍

Uh oh!

github-actions bot commented Oct 23, 2024

PR Code Suggestions ✨

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

DaMandal0rian commented Oct 23, 2024 •

edited

Loading