How do i use AWS-lambda to create csv file from json data?

Question

Problem:

I have a json data filtered out from security groups with respect to my requirements which i want to save into .csv file.

My Research effort:

I have done it in pycharm where i am storing it to my local drive using the following code :

def store_details_in_csv(jsondata):
    file_path = os.path.dirname(__file__) + "/security.csv"
    with open(file_path, "a") as export:
        names = ['Account Number', 'Account Name', 'Region', 'SG', 'Inbound port', 'Inbound IP', 'Assessment']
        writer = csv.DictWriter(export, fieldnames=names)
        writer.writeheader()
        for security_group in jsondata['SecurityGroups']:
            groupId = security_group["GroupId"]
            for inboundEntry in security_group["InboundDetails"]:
                inboundPort = inboundEntry['port']
                inboundIP = inboundEntry['source']
                assessment = inboundEntry['assessment']
                writer.writerow({'Account Number': ACCOUNT_NUMBER, 'Account Name': ACCOUNT_NAME, 'Region': REGION,
                                 'SG': groupId, 'Inbound port': inboundPort,
                                 'Inbound IP': inboundIP, 'Assessment': assessment})

NOW,

How do i use AWS-lambda to create csv file from json data ?

So, what do you want exactly? save a csv to where? A bit confused. — Daeho Ro
– Daeho Ro, Commented Sep 3, 2019 at 9:14
Actually the data which i am getting after doing filtering on Security groups is huge and saving that to any S3 is not required for my case, so I want that file to get downloaded to my local. If that is not possible then storing into S3 bucket can also help me out — YouBee
– YouBee, Commented Sep 3, 2019 at 9:42

Daeho Ro · Accepted Answer · 2019-09-03 10:39:46Z

4

Here, I attach the lambda code for saving csv into S3 without temp files.

import boto3
import csv
import io

s3 = boto3.client('s3')
ec2 = boto3.client('ec2')

def lambda_handler(event, context):
    csvio = io.StringIO()
    writer = csv.writer(csvio)
    writer.writerow(['id', 'name'])

    paginator = ec2.get_paginator('describe_security_groups').paginate()

    for page in paginator:
        for item in page['SecurityGroups']:
            identity = item['GroupId']
            name = item['GroupName']

            writer.writerow([identity, name])

    s3.put_object(Body=csvio.getvalue(), ContentType='text/csv', Bucket='<bucket>', Key='<filename.csv>') 
    csvio.close()

You can modify this for your usa case.

edited Sep 3, 2019 at 10:39

answered Sep 3, 2019 at 9:17

Daeho Ro

13.7k4 gold badges25 silver badges50 bronze badges

Sign up to request clarification or add additional context in comments.

5 Comments

YouBee Over a year ago

sure will try this and let you know.

YouBee Over a year ago

Thanks Lamanus. This has worked and is storing the csv file into the S3 bucket. I have added some more code to get it downloaded and that also is working. Now the thing is I want it to get downloaded without saving into s3 bucket. Is it possible ?

Daeho Ro Over a year ago

I misread your comment. Can you update your code into the question? Then I will take a look.

Daeho Ro Over a year ago

Lambda is also a server and it cannot specify the saving path of another server without ssh or FTP. If you open those service and allow to connect from lambda then it can be downloaded to your local. Without that, save to s3 and download from s3 is reasonable.

YouBee Over a year ago

Thanka Lamanus. I am thinking of sending the csv file as an email using SES service. Your solution was most helpful so marking it as Accepted :)

Collectives™ on Stack Overflow

How do i use AWS-lambda to create csv file from json data?

NOW,

1 Answer 1

5 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

NOW,

1 Answer 1

5 Comments

Your Answer

Sign up or log in

Post as a guest

Related