1

I have a Python Script that gets the details of the unused security groups. I want that to write into a CSV file and upload to S3 Bucket.

When I test it in local machine it writes to CSV in the local machine. But when I execute that as a lambda function, it needs a place to save the CSV. So I am using s3. 

import boto3
import csv

ses = boto3.client('ses')

def lambda_handler(event, context):
    with open('https://unused******- 
    1.amazonaws.com/Unused.csv', 'w') as csvfile:
        writer = csv.writer(csvfile)
        writer.writerow([
            'Account Name',
            'Region',
            'Id'
        ])
        ec2 = boto3.resource('ec2')
        sgs = list(ec2.security_groups.all())
        insts = list(ec2.instances.all())

        all_sgs = set([sg.group_id for sg in sgs])
        all_inst_sgs = set([sg['GroupId'] for inst in insts for sg in
        inst.security_groups])

        unused_sgs = all_sgs - all_inst_sgs


        for elem in unused_sgs:
            writer.writerow([
                Account_Name,
                region,
                elem
                ])

I want to write the result of "elem" into csv file and upload to S3 Bucket. Kindly advice.

Improve this question

4 Answers 4

Reset to default
4

By using StringIO(), you don't need to save the csv to local and just upload the IO to S3. Try my code and let me know if something wrong because I can't test the code but it was worked for other cases.

import boto3
import csv
import io

s3 = boto3.client('s3')
ses = boto3.client('ses')

def lambda_handler(event, context):
    csvio = io.StringIO()
    writer = csv.writer(csvio)
    writer.writerow([
        'Account Name',
        'Region',
        'Id'
    ])

    ec2 = boto3.resource('ec2')
    sgs = list(ec2.security_groups.all())
    insts = list(ec2.instances.all())

    all_sgs = set([sg.group_id for sg in sgs])
    all_inst_sgs = set([sg['GroupId'] for inst in insts for sg in
    inst.security_groups])

    unused_sgs = all_sgs - all_inst_sgs

    for elem in unused_sgs:
        writer.writerow([
            Account_Name,
            region,
            elem
            ])

    s3.put_object(Body=csvio.getvalue(), ContentType='application/vnd.ms-excel', Bucket='bucket', Key='name_of.csv') 
    csvio.close()
Improve this answer
Sign up to request clarification or add additional context in comments.

7 Comments

Thanks for that @Lamanus. Getting an Error. unicode argument expected, got 'str': TypeError Traceback (most recent call last): File "/var/task/lambda_function.py", line 13, in lambda_handler 'Id' TypeError: unicode argument expected, got 'str' END RequestId: b56f2c6f-4c56-4793-a828-bbefd9cf017b REPORT RequestId: b56f2c6f-4c56-4793-a828-bbefd9cf017b Duration: 0.64 ms Billed Duration: 100 ms Memory Size: 128 MB Max Memory Used: 65 MB
Please debug the code and let me know what line gives the error. Btw, I added a line number 5 for s3 client definition which is missing before.
I made some fixes and it works fine now.. Yes I defined s3 already.. StringIO() should be to Bytes(IO). It works cool now..
Also I have to attach this file(csv) to email and send via ses. Please help me how to read this file and attach.
I think it is better to upload a new question. I am not good for ses and raw_email. Check this first.
|
0

If the CSV file will be small, write it to the /tmp folder, then upload that file to S3. If it's large (say, larger than ~200MB) then you should probably stream it to S3.

Read the boto3 documents for the relevant S3 client methods.

Improve this answer

2 Comments

The /tmp/ directory provides 512MB of storage. Make sure you delete files at the end of the function because the container might be used again for subsequent calls.
@jarmod Thanks for the answer.. The File size is only in KB.
0

Follow jarmod advice if your csv file is small, otherwise you could use lambda to spin up a temporary ec2 instance (you can go for xlarge size for better performance) with user_data in it. The user_data will do all the csv processes on a strong and healthy ec2, though remember to terminate the instance (the termination command can be also included in the user_data) once the process is done.

Improve this answer

2 Comments

Using ec2 to handle heavy workload offers better performance than relying on lambda which has memory, CPU and storage limitation.
The File Size is very small. In that case, I guess Lambda should be fine.
0

Generate an inventory from multiple accounts, and push it to the s3 bucket using the Lambda function.

Create SSM parameter store to put the IAM assume roles of the Accounts;

Name: 'rolearnlist'

Type: 'StringList'

Values: 'arn::::::,arn:::::'

Create a Lambda function as below;

import boto3
import json
import datetime
import csv

lambda_client = boto3.client('lambda')
ssm_client = boto3.client('ssm')
s3_client = boto3.resource("s3")
sts_client = boto3.client('sts')

def lambda_handler(event, context):

time = datetime.datetime.now().strftime ('%Y-%m-%d-%H-%M-%S')
bucket = s3_client.Bucket('expo2020-core-master-me-south-1-agent-bucket')
file_name = ('backup_job_weekly_report_' + time + '.csv')
s3_path = 'Inventory/Weekly/' + file_name

fieldnames = ['Account Id','Backup Job Id', 'Backup State', 'Resource Arn', 'Resource Type', 'Start By','Creation Date']

rolearnlist = []
rolearnlist_from_ssm = ssm_client.get_parameter(Name='rolearnlist')
rolearnlist_from_ssm_list = rolearnlist_from_ssm['Parameter']['Value'].split(",")
rolearnlist = rolearnlist_from_ssm_list

with open('/tmp/file_name', 'w', newline='') as csvFile:
    w = csv.writer(csvFile, dialect='excel')
    w.writerow(fieldnames)

    for rolearn in rolearnlist:
        awsaccount = sts_client.assume_role(
            RoleArn=rolearn,    
            RoleSessionName='awsaccount_session'
        )

        ACCESS_KEY = awsaccount['Credentials']['AccessKeyId']
        SECRET_KEY = awsaccount['Credentials']['SecretAccessKey']
        SESSION_TOKEN = awsaccount['Credentials']['SessionToken']

        backup = boto3.client('backup', aws_access_key_id=ACCESS_KEY, aws_secret_access_key=SECRET_KEY, aws_session_token=SESSION_TOKEN)
        response = backup.list_backup_jobs()

        for i in response['BackupJobs']:
            AccountId = i.get('AccountId')
            BackupJobId = i.get('BackupJobId')
            BackupState = i.get('State')
            ResourceArn = i.get('ResourceArn')
            ResourceType = i.get('ResourceType')
            StartBy = i.get('StartBy')
            CreationDate = i.get('CreationDate')

            raw =   [
                    AccountId,
                    BackupJobId,
                    BackupState,
                    ResourceArn,
                    ResourceType,
                    StartBy,
                    CreationDate,
                    ]

            w.writerow(raw)
            raw = []
csvFile.close()

bucket.upload_file('/tmp/file_name', s3_path)
Improve this answer

Comments

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.