A memo that implements the job of loading a GCS file into BigQuery in Python

Sample code


#If you do not set the service account, you will get an authentication error.
# export GOOGLE_APPLICATION_CREDENTIALS=/path/to/credential

from google.cloud import bigquery
client = bigquery.Client('project')

table_ref = client.dataset('dataset').table('table$20200101') #partition
    
job_config = bigquery.LoadJobConfig()
job_config.write_disposition = bigquery.WriteDisposition.WRITE_TRUNCATE #Overwrite
job_config.source_format = bigquery.SourceFormat.NEWLINE_DELIMITED_JSON # JSON
# job_config.ignore_unknown_values = True #Uncomment if unknown value is allowed
    
uri = 'gs://bucket/path/*.gz' #Specify everything under the directory
load_job = client.load_table_from_uri(
        uri, table_ref, job_config=job_config
        )     
load_job.result()
print("Job finished.")

reference

Load CSV data into table (https://cloud.google.com/bigquery/docs/loading-data-cloud-storage-csv?hl=ja#loading_csv_data_into_a_table)

Recommended Posts

A memo that implements the job of loading a GCS file into BigQuery in Python
[Note] Import of a file in the parent directory in Python
A function that measures the processing time of a method in python
A memo organized by renaming the file names in the folder with python
Test & Debug Tips: Create a file of the specified size in Python
I made a program to check the size of a file in Python
A memo that I wrote a quicksort in Python
Get the caller of a function in Python
Make a copy of the list in Python
Output in the form of a python array
Get the update date of the Python memo file.
Various ways to read the last line of a csv file in Python
[Python] A program that counts the number of valleys
Python that merges a lot of excel into one excel
A memo that I touched the Datastore with python
A reminder about the implementation of recommendations in Python
[Python] A program that compares the positions of kangaroos.
A python script that imports a dated csv file into BigQuery as a time partition table
[Python] A program that finds the shortest number of steps in a game that crosses clouds
[Python] A notebook that translates and downloads the ipynb file on GitHub into Japanese.
Download the file in Python
A function that divides iterable into N pieces in Python
[Memo] The mystery of cumulative assignment statements in Python functions
Change the standard output destination to a file in Python
Get the number of specific elements in a python list
How to develop in a virtual environment of Python [Memo]
A memo of writing a basic function in Python using recursion
Google search for the last line of the file in Python
A set of script files that do wordcloud in Python3
Find the eigenvalues of a real symmetric matrix in Python
A Python script that compares the contents of two directories
[Python] Programming to find the number of a in a character string that repeats a specified number of times.
A memo that reproduces the slide show (gadget) of Windows 7 on Windows 10.
[Python] A memo of frequently used phrases (by myself) in Python scripts
Process the contents of the file in order with a shell script
Play a sound in Python assuming that the keyboard is a piano keyboard
How to determine the existence of a selenium element in Python
A memo that handles double-byte double quotes in Python regular expressions
Save the binary file in Python
A story that struggled to handle the Python package of PocketSphinx
How to check the memory size of a variable in Python
Read the standard output of a subprocess line by line in Python
Create a binary file in Python
How to check the memory size of a dictionary in Python
The story of the "hole" in the file
Read a file in Python with a relative path from the program
The idea of feeding the config file with a python file instead of yaml
Get the formula in an excel file as a string in Python
[python] A note that started to understand the behavior of matplotlib.pyplot
The story of making a module that skips mail with python
The story of creating a bot that displays active members in a specific channel of slack with python
[Python] A program that rotates the contents of the list to the left
Get the number of readers of a treatise on Mendeley in Python
Trial of writing the configuration file in Python instead of .ini etc.
Get a capture of the entire web page in Selenium Python VBA
What is the XX file at the root of a popular Python project?
If you want a singleton in python, think of the module as a singleton
[Python] A program that calculates the number of socks to be paired
Check the in-memory bytes of a floating point number float in Python
View the full path (absolute path) of a file in a directory in Linux Bash
Receive a list of the results of parallel processing in Python with starmap