Adjust reporter partition and filesystem size

The root and data volumes attached to the CyHy reporter EC2 instance can occasionally fill up. Until we resolve #4 manage reporting disk space and warn on nearly-full, you can use the instructions in this document to increase the sizes of the root and/or data volumes.

Determine which volumes require resizing

After sshing into the CyHy reporter EC2 instance, you can use the df command to see how much free space is left on the volumes:

$ df -h
Filesystem      Size  Used Avail Use% Mounted on
<snip>
/dev/nvme0n1p1   20G  7.7G   12G  41% /
<snip>
/dev/nvme1n1    200G  200G   20K   0% /var/cyhy/reports/output
<snip>

(I have omitted irrelevant lines.)

Note that /dev/nvme0n1p1 is the root volume while /dev/nvme1n1 is the data volume mounted at /var/cyhy/reports/output. In this case, the data volume is in obvious need of resizing.

Increase volume size in Terraform

In the cyhy_amis repository, increase the size(s) specified in terraform/cyhy_reporter_ec2.tf, which provisions the CyHy reporter EC2 instance.

resource "aws_instance" "cyhy_reporter" {
  ...
  root_block_device {
    volume_type           = "gp2"
    # Increase this volume size if the root volume requires resizing
    volume_size           = 50
    delete_on_termination = true
  }
  ...
}
...

resource "aws_ebs_volume" "cyhy_reporter_data" {
  availability_zone = "${var.aws_region}${var.aws_availability_zone}"
  type              = "io1"
  # Increase this volume size if the data volume requires resizing
  size              = local.production_workspace ? 500 : 5
  iops              = 100
  encrypted         = true
...

After saving these changes, apply them in production via Terraform:

terraform apply -var-file=prod-a.tfvars -target=aws_ebs_volume.cyhy_reporter_data -target=aws_instance.cyhy_reporter

For an example PR of these changes, visit cyhy_amis#288 Increase volume sizes on reporter instance.

Resize filesystems on the reporter instance

After the Terraform apply completes successfully, ssh into the reporter instance to resize the filesystems on the root and/or data volumes.

Reference documentation: AWS: Extending a Linux file system after resizing a volume

To resize the (ext4) filesystem on the reporter instance's root volume:

$ sudo growpart /dev/nvme0n1 1   # Grow root volume partition
CHANGED: partition=1 start=2048 old: size=41940959 end=41943007 new: size=104855519,end=104857567
$ sudo resize2fs /dev/nvme0n1p1  # Resize file system on root volume
resize2fs 1.43.4 (31-Jan-2017)
Filesystem at /dev/nvme0n1p1 is mounted on /; on-line resizing required
old_desc_blocks = 3, new_desc_blocks = 7
The filesystem on /dev/nvme0n1p1 is now 13106939 (4k) blocks long.

To resize the (XFS) filesystem on the reporter instance's data volume:

$ sudo xfs_growfs -d /var/cyhy/reports/output  # Grow file system on data volume
meta-data=/dev/nvme1n1           isize=512    agcount=4, agsize=13107200 blks
         =                       sectsz=512   attr=2, projid32bit=1
         =                       crc=1        finobt=1 spinodes=0 rmapbt=0
         =                       reflink=0
data     =                       bsize=4096   blocks=52428800, imaxpct=25
         =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=4096   ascii-ci=0 ftype=1
log      =internal               bsize=4096   blocks=25600, version=2
         =                       sectsz=512   sunit=0 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0
data blocks changed from 52428800 to 131072000

Note that you can use the command lsblk to view all block devices (including drives) attached to the instance. You can also use df -h after resizing to verify that the mounted filesystems are indeed increased in size.