Automate GCP persistent disk initialization

Question

Are there any scripts that automate persistent disks formatting and attaching to the Google Cloud VM instance, instead of doing formatting & mounting steps?

The persistent disk is created with Terraform, which also creates a VM and attaches the disk to it with the attached_disk command.

I am hoping to run a simple script on the VM instance start that would:

check if the attached disk is formatted, and format if needed with ext4
check if the disk is mounted, and mount if not
do nothing otherwise

I'm not sure what you mean by instead of doing formatting & mounting steps when that's the process you need to do to be able to use the disk. Are you saying you want something that just does it for you instead of having to write a script (or basically copy it from the linked docs)? — ydaetskcoR
– ydaetskcoR, Commented Nov 5, 2018 at 23:07
@ydaetskcoR the process could be manual (me copy/pasting the commands), and automated (terraform formatting the disk, startup script auto-mounting it). I prefer the later, and was hoping someone has already done that, instead of every person reinventing the wheel :) P.S. in docs, edit /etc/fstab instructions are incorrect - it gets reset on instance reboot. — Yuri Astrakhan
– Yuri Astrakhan, Commented Nov 6, 2018 at 3:23
The docs you linked to also mention why the file might be reset: “On Container-Optimized OS, modifications to /etc/fstab do not persist across reboots. To ensure the device is checked and mounted during boot, run the fsck and mount operations on the persistent disk from your cloud-config’s bootcmd section. See ‘Mounting and formatting disks’ in the Container-Optimized OS documentation.” — Dan
– Dan, Commented Nov 6, 2018 at 5:48
Unfortunately everybody under the sun uses a different OS configuration tool. For a basic shell script you should just copy-paste from the docs, but to integrate it with your other config scripts in Puppet / Chef / Ansible / Salt / ... a bit more work will be required. — Dan
– Dan, Commented Nov 6, 2018 at 5:51

neilH · Accepted Answer · 2018-11-14 12:28:23Z

10

Have you considered using a startup script on the instance (I presume you can also add a startup-script with Terraform)? You could use an if loop to discover if the disk is formatted, then if not, you could try running the formatting/mounting commands in the documentation you linked (I realise you have suggested you do not want to follow the manual steps in the documentation, but these can be integrated into the startup script to achieve the desired result).

Running the following outputs and empty string if the disk is not formatted:

 sudo blkid /dev/sdb

You could therefore use this in a startup script to discover if the disk is formatted, then perform formatting/mounting if that is not the case. For example, you could use something like this (Note*** If the disk is formatted but not mounted this could be dangerous and should not be used if your use case could involve existing disks which may have already been formatted):

#!/bin/bash


if sudo blkid /dev/sdb;then 
        exit
else 
        sudo mkfs.ext4 -m 0 -F -E lazy_itable_init=0,lazy_journal_init=0,discard /dev/sdb; \
        sudo mkdir -p /mnt/disks/newdisk
        sudo mount -o discard,defaults /dev/sdb /mnt/disks/newdisk
fi

edited Nov 14, 2018 at 12:28

answered Nov 6, 2018 at 12:46

neilH

3,4582 gold badges23 silver badges40 bronze badges

Sign up to request clarification or add additional context in comments.

4 Comments

Yuri Astrakhan Over a year ago

Thanks! Does GCP guarantee /dev/sdb to always be the second attached disk, sda == boot? Also, the formatting you suggested is dangerous - the disk could be formatted but not mounted.

neilH Over a year ago

In terms of your question about /dev/sdb, no it doesn't guarantee this this, although it's highly likely if this is the first disk you attach it would be /dev/sdb. In regards to your point about the risk of a disk being formatted but not mounted, I've added a note to my response to warn of the risk. I would suggest adding the attempts you've made in terms of your script to your original question (if you've now tried scripting this yourself), so that people can troubleshoot why your attempts are failing.

JE42 Over a year ago

@yurik you can assign an id to the disk. google then provides a symlink /dev/disk/by-id/google-<my-id>

Raj Saxena Over a year ago

I have provided a script that works by assigning a disk-name & then using that to identify the correct device at /dev/sd* - stackoverflow.com/a/59646741/3074229

runofthemill · Accepted Answer · 2024-04-22 11:26:09Z

9

The marked answer did not work for me as the sudo blkid /dev/sdb part always returned a value (hence, true) and the script would exit.

I updated the script to check for the entry in fstab and added safety options to the script.

#!/bin/bash
set -uxo pipefail

MOUNT_DIR=/mnt/disks/persistent_storage
DISK_NAME=my-disk

# Check if entry exists in fstab
grep -q "$MOUNT_DIR" /etc/fstab
if [[ $? -eq 0 ]]; then # Entry exists
    exit
else
    set -e # The grep above returns non-zero for no matches & we don't want to exit then.

    # Find persistent disk's drive value, prefixed by `google-`
    DEVICE_NAME="/dev/$(basename $(readlink /dev/disk/by-id/google-${DISK_NAME}))"

    sudo mkfs.ext4 -m 0 -F -E lazy_itable_init=0,lazy_journal_init=0,discard $DEVICE_NAME
    sudo mkdir -p $MOUNT_DIR
    sudo mount -o discard,defaults $DEVICE_NAME $MOUNT_DIR

    # Add fstab entry
    echo UUID=$(sudo blkid -s UUID -o value $DEVICE_NAME) $MOUNT_DIR ext4 discard,defaults,nofail 0 2 | sudo tee -a /etc/fstab
fi

Here's the gist if you want to download it - https://gist.github.com/raj-saxena/3dcaa5c0ba0be88ed91ef3fb50d3ce85

edited Apr 22, 2024 at 11:26

runofthemill

811 silver badge7 bronze badges

answered Jan 8, 2020 at 13:17

Raj Saxena

87210 silver badges20 bronze badges

2 Comments

FreshMike Over a year ago

This does sound dangerous, we should check if there is a partition already and if it has a filesystem, otherwise you might redo the filesystem.

Raj Saxena Over a year ago

@FreshMike yeah, you are right. I have updated the script & instead of the folder, I am checking for an entry in the fstab. How would you have done?

ayuuk ja'ay · Accepted Answer · 2023-01-12 15:56:59Z

Formatting, mounting and adding entry in /etc/fstab is necessary almost all the time. Here is a solution I came up with and might help others. This can also, for sure, be improved. I added echo commands to explain what each block does.

About disk name you could add device_name on your terraform code when you attach your disks to the instance(s) like mentioned here: https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/compute_attached_disk

device_name - (Optional) Specifies a unique device name of your choice that is reflected into the /dev/disk/by-id/google- tree of a Linux operating system running within the instance. This name can be used to reference the device for mounting, resizing, and so on, from within the instance.*

#!/bin/bash

DISKS_PATH=/dev/disk/by-id

DISKS=(disk1 disk2)

check_disks () {
    for disk in "${DISKS[@]}"; do
        MOUNT_DIR="/$disk"
        echo "$MOUNT_DIR"
        if sudo blkid $DISKS_PATH/google-${disk}; then
            echo "$disk is already formatted, nothing to do"
            echo "checking if $disk is present in fstab"
            UUID=$(sudo blkid -s UUID -o value $DISKS_PATH/google-${disk})
            grep -q "UUID=${UUID} $MOUNT_DIR" /etc/fstab
            if [[ $? -eq 0 ]]; then
                echo "$disk already present in fstab, continuing with checking mount"
                echo "Now checking if $disk is already mounted"
                grep -qs "$MOUNT_DIR" /proc/mounts
                if [[ $? -eq 0 ]]; then
                    echo "$disk is already mounted, so doing nothing  with mount"
                else
                    echo "$disk is not mounted, so mounting it"
                    sudo mkdir -p $MOUNT_DIR
                    sudo mount -o discard,defaults $DISKS_PATH/google-${disk} $MOUNT_DIR
                fi
            elif [[ $? -ne 0 ]]; then
                echo "$disk not present in fstab, so adding it"
                echo UUID="$UUID" $MOUNT_DIR ext4 discard,defaults,nofail 0 2 | sudo tee -a /etc/fstab
                echo "Now checking if $disk is already mounted"
                grep -qs "$MOUNT_DIR" /proc/mounts
                if [[ $? -eq 0 ]]; then
                    echo "$disk is already mounted, so doing nothing  with mount"
                else
                    echo "$disk is not mounted, so mounting it"
                    sudo mkdir -p $MOUNT_DIR
                    sudo mount -o discard,defaults $DISKS_PATH/google-${disk} $MOUNT_DIR
                fi
            fi
        else
            echo "Formatting ${disk}"
            sudo mkfs.ext4 $DISKS_PATH/google-${disk};
            echo "Creating directory for ${disk} on $MOUNT_DIR"
            sudo mkdir -p $MOUNT_DIR
            echo "adding  $disk in fstab"
            UUID=$(sudo blkid -s UUID -o value $DISKS_PATH/google-${disk})
            echo UUID="$UUID" $MOUNT_DIR ext4 discard,defaults,nofail 0 2 | sudo tee -a /etc/fstab
            echo "Mounting $disk"
            sudo mount -o discard,defaults $DISKS_PATH/google-${disk} $MOUNT_DIR
        fi
    done
}

check_disks

Collectives™ on Stack Overflow

Automate GCP persistent disk initialization

3 Answers 3

4 Comments

2 Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

4 Comments

2 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related