[Tips] Handle Athena with Python

Introduction

This article is for anyone who wants to query Athena and analyze the results in pandas. I think it's especially useful when analyzing with Jupyter notebook.

Installation

Install PyAthena.

pip install PyAthena

How to use

Use the connect function. Specify the AWS key and the path of S3 that spits out the result of querying with Athena. If you execute it using the function pd.read_sql, you can get the execution result in the form of pandas.

from pyathena import connect
import pandas as pd
aws_access_key_id = 'Your aws access key id'
aws_secret_access_key = 'Your aws secret access key'

conn = connect(aws_access_key_id=aws_access_key_id,
                 aws_secret_access_key=aws_secret_access_key,
                 s3_staging_dir='Your s3 path',
                 region_name='ap-northeast-1')

df = pd.read_sql("SELECT * FROM sample", conn)

Recommended Posts

[Tips] Handle Athena with Python
Handle Excel with python
Handle rabbimq with python
python tips
Python Tips
Python tips
Handle Base91 keys with python + redis.
Easily handle lists with python + sqlite3
Handle Excel CSV files with Python
Easily handle databases with Python (SQLite3)
Python Conda Tips
Statistics with python
Python with Go
Python debugging tips
Tips for dealing with binaries in Python
Twilio with Python
Python click tips
Integrate with Python
Play with 2016-Python
Unexpectedly (?) Python tips
AES256 with python
Tested with Python
python starts with ()
Tips for using python + caffe with TSUBAME
Trying to handle SQLite3 with Python [Note]
with syntax (Python)
Bingo with python
Zundokokiyoshi with python
Excel with Python
Microcomputer with Python
Cast with python
Handle zip files with Japanese filenames in Python 3
[Rust / Python] Handle numpy with PyO3 (August 2020 version)
[Python] How to handle Japanese characters with openCV
~ Tips for Python beginners from Pythonista with love ② ~
Zip, unzip with python
Django 1.11 started with Python3.6
Primality test with Python
Python with eclipse + PyDev.
Scraping with Python (preparation)
Try scraping with Python.
Python and numpy tips
Learning Python with ChemTHEATER 03
Sequential search with Python
"Object-oriented" learning with python
Run Python with VBA
Handling yaml with python
Serial communication with python
Learning Python with ChemTHEATER 05-1
Learn Python with ChemTHEATER
Run prepDE.py with python3
1.1 Getting Started with Python
Collecting tweets with Python
Binarization with OpenCV / Python
3. 3. AI programming with Python
Kernel Method with Python
Non-blocking with Python + uWSGI
Scraping with Python + PhantomJS
Posting tweets with python
Drive WebDriver with python
Use mecab with Python3