Read and write to/from s3 using python — boto3 and pandas (s3fs)!

!pip install -m boto3
!pip install -m pandas "s3fs<=0.4"
import io
import os
import boto3
import pandas as pd
AWS_S3_BUCKET = "samplebucket1303"
AWS_ACCESS_KEY_ID = "accesskey"
AWS_SECRET_ACCESS_KEY = "secretkey"
s3_client = boto3.client(
"s3",
aws_access_key_id=AWS_ACCESS_KEY_ID,
aws_secret_access_key=AWS_SECRET_ACCESS_KEY,
)
df = pd.read_csv("Language Detection.csv")
with io.StringIO() as csv_buffer:
df.to_csv(csv_buffer, index=False)
response = s3_client.put_object(
Bucket=AWS_S3_BUCKET, Key="Language_Detection_boto3.csv", Body=csv_buffer.getvalue()
)

status = response.get("ResponseMetadata", {}).get("HTTPStatusCode")
if status == 200:
print(f"Successful S3 put_object response. Status - {status}")
else:
print(f"Unsuccessful S3 put_object response. Status - {status}")
df.to_csv(
f"s3://{AWS_S3_BUCKET}/Language_Detection_s3fs.csv",
index=False,
storage_options={
"key": AWS_ACCESS_KEY_ID,
"secret": AWS_SECRET_ACCESS_KEY,
}
)
response = s3_client.get_object(Bucket=AWS_S3_BUCKET, Key="Language_Detection_boto3.csv")
status = response.get("ResponseMetadata", {}).get("HTTPStatusCode")if status == 200:
print(f"Successful S3 get_object response. Status - {status}")
books_df = pd.read_csv(response.get("Body"))
print(books_df)
else:
print(f"Unsuccessful S3 get_object response. Status - {status}")
df = pd.read_csv(
f"s3://{AWS_S3_BUCKET}/Language_Detection_s3fs.csv",
storage_options={
"key": AWS_ACCESS_KEY_ID,
"secret": AWS_SECRET_ACCESS_KEY,
}
)
df.head()

--

--

--

Data Scientist with a majors in Computer Vision. Love to blog and share the knowledge with the data community.

Love podcasts or audiobooks? Learn on the go with our new app.

Recommended from Medium

My learnings during the third week in the Encora’s Apprentice program

Three (Not So New) Technology Trends in Enterprise Networking Part 1: Software Defined Networking…

Matrix 3 Walkthrough

HackTheBox — Knife, WriteUp

A Complete guide to write test cases

Effective Java! In Public Classes, Use Accessors, Not Public Fields

Running Effective Bug Bashes That Your Team Won’t Dread

Elliot Cloud is helping São Paulo to manage its water resources network

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Shekhar Khandelwal

Shekhar Khandelwal

Data Scientist with a majors in Computer Vision. Love to blog and share the knowledge with the data community.

More from Medium

Reading CSV file from amazon S3 bucket using csv module in Python

Working with python boto3 library — Environment setup

Creating an AWS Lambda Function for Matplotlib Data Visualizations

Automatic data quality validations with Great Expectations: An Introduction to DQVT