・ GCP account is ready -There must be a table that can connect to BigQuery
ec2 -pyenv -naconda3-4.0.0 -pandas -httplib2 -google-api-python-client
sudo su -
pyenv global naconda3-4.0.0
pip install httplib2
pip install google-api-python-client
pyenv global system
curl https://sdk.cloud.google.com | bash
~/google-cloud-sdk/bin/gcloud auth login
import pandas as pd
query = 'SELECT * FROM dataset_name.table_name'
pd.read_gbq(query, 'your_project_ID')
→ Authentication runs with pd.read_gbq, but since EC2 is the execution source, I can't receive the authentication URL to listen on localhost
from oauth2client.client import OAuth2WebServerFlow
from oauth2client.file import Storage
flow = OAuth2WebServerFlow(client_id='your_client_id',
client_secret='your_client_secret',
scope='https://www.googleapis.com/auth/bigquery',
redirect_uri='urn:ietf:wg:oauth:2.0:oob')
storage = Storage('bigquery_credentials.dat')
authorize_url = flow.step1_get_authorize_url()
print 'Go to the following link in your browser: ' + authorize_url
code = raw_input('Enter verification code: ')
credentials = flow.step2_exchange(code)
storage.put(credentials)
After executing the above process, "bigquery_credentials.dat" is created in the current directory. → pandas.read_gbq uses the above as authentication information
import pandas as pd
query = 'SELECT * FROM dataset_name.table_name'
pd.read_gbq(query, 'your_project_ID')
https://developers.google.com/api-client-library/python/guide/aaa_oauth http://stackoverflow.com/questions/37792709/accessing-big-query-from-cloud-datalab-using-pandas
Recommended Posts