[PYTHON] Extract data from S3

For some reason, when I took out the data uploaded to S3, I had to get the entire list once before I could get it. The prefix specification was useless.

Sample code

#!/usr/bin/env python
import boto.s3.connection

def main():
  s3con = boto.s3.connection.S3Connection('Something like an access key', 'Something secret')
  bucket_con = s3con.get_bucket('logs-many')
  for k in bucket_con.list(prefix="access_log/2015/02/10/"):
    fn = k.name.split('/')[-1]
    body = k.get_contents_to_filename('log/' + fn)
    print 'Got! ' + fn

if __name__ == '__main__':      
  main()

So, what was wrong was that I had specified `` `/ logs-many``` in the prefix.

Somehow, when I read the document, it says / foo /, so I thought it was necessary.

Recommended Posts

Extract data from S3
Extract specific data from complex JSON
Extract features (features) from sentences.
Aggregate AWS S3 data
Extract table from wikipedia
Extract data from a web page with Python
Export 3D data from QGIS
Extract Twitter data with CSV
Extract specific languages from Wiktionary
Extract redirects from Wikipedia dumps
Extract csv data and calculate
[Python] Extract text data from XML data of 10GB or more.
Follow Blender's data structure and extract vertex coordinates from fbx
The story of copying data from S3 to Google's TeamDrive
Extract classification information etc. from genbank data in xml format
From Elasticsearch installation to data entry
Python: Exclude tags from html data
Hit treasure data from Python Pandas
Get data from Quandl in Python
[AWS] Search and acquire necessary data from S3 files with S3 Select
Persistent data structure created from scratch
Extract a page from a Wikipedia dump
Extract text from images in Python
[Data science basics] Data acquisition from API
Get data from Twitter using Tweepy
Extract Pokemon GO Pokemon data and skill data
Extract strings from files in Python
Extract and plot the latest population data from the PDF data provided by the city
Copy data from Amazon S3 to Google Cloud Storage with Python (boto)
Try to extract specific data from JSON format data in object storage Cloudian/S3
Extract images from cifar and CUCUMBER-9 datasets
I tried scraping conversation data from Askfm
Receive textual data from mysql with python
SIGNATE Quest ① From data reading to preprocessing
Use PostgreSQL data type (jsonb) from Python
Python: Reading JSON data from web API
Download the file from S3 using boto.
Acquisition of plant growth data Acquisition of data from sensors
Get data from Cloudant with Bluemix flask
# 5 [python3] Extract characters from a character string
Extract Japanese text from PDF with PDFMiner
Rescue data from HDD taken from ReadyNAS RN212
[Python] Web application from 0! Hands-on (4) -Data molding-
[TensorFlow] Extract features from trained model Inception-v3
Get data from an oscilloscope with pyVISA
[Python] (Line) Extract values from graph images
Use PIL in Python to extract only the data you want from Exif
[AWS; Introduction to Lambda] 2nd; Extract sentences from json file and save S3 ♬
Extract periods that match a particular pattern from pandas time series qualitative data