In this blog, we will run a Demo of creating an AWS Lambda Function which will convert a csv file into a json file.
Prerequisite
- Clone this git repo
- Refer this blog for basic understanding of AWS Lambda
For this we will use S3 bucket as a source where we will store the csv file that will trigger the lambda function and convert the file in json and store it in another bucket.
We will be creating below resources. You can refer the git repo
- Create source S3 bucket
- Create target S3 bucket
- Create a lambda function in python
- Zip the file
- Upload a zip file in source S3 bucket
- Create a lambda function
- Create an IAM role and add permission
- Create a trigger
- Test it
Create source and target bucket
cd Lambda
python create_s3_bkt1.py
python create_s3_bkt2.py
Now we will create a Lambda function which will do the parsing and store the new file in the target bucket
zip lamda_function.zip lambda_function.pyaws s3 cp lamda_function.zip s3://
devops4solutions-source
We have successfully uploaded the zip file in our source bucket
Create an IAM role
- which gives access to the lambda function with policy
AWSLambdaExecute
aws iam create-role --role-name PayrollProcessingLambdaRole1 --description "Provides AWS Lambda with access to s3 and cloudwatch to execute the PayrollProcessing function" --assume-role-policy-document file://lambda-trust-policy.json
aws iam attach-role-policy --role-name PayrollProcessingLambdaRole1 --policy-arn arn:aws:iam::aws:policy/AWSLambdaExecute
Create a Lambda function
You can use the aws-cli or can create directly from GUI also
aws lambda create-function --function-name PayrollProcessing --runtime python3.7 --role
arn:aws:iam::936379345511:role/PayrollProcessingLambdaRole--handler lambda_function.lambda_handler --description "Converts Payroll CSVs to JSON and puts the results in an s3 bucket." --timeout 3 --memory-size 128 --code S3Bucket=
devops4solutions-source,S3Key=lamda_function.zip --tags Environment="Production",Application="Payroll" --region us-east-2
Add permission
aws lambda add-permission --function-name PayrollProcessing --statement-id lambdas3permission --action lambda:InvokeFunction --principal s3.amazonaws.com --source-arn arn:aws:s3:::
devops4solutions-source--source-account
936379345511--region us-east-2
Now add trigger whenever an object is put in the S3 bucket
aws s3api put-bucket-notification-configuration ––bucket
devops4solutions-source––notification-configuration file://notification-config.json
Sometimes you see this error when you are trying to use the same bucket. To fix this issue, go to S3 bucket -> Properties -> Event -> Delete Event Notification
Test it
Copy csv file in the source bucket using below command
aws s3 cp input-payroll-data.csv s3://
Now you can see that in the target ,a new json file is uploaded
References
- Safari Book