[PYTHON] Write data in HDF format

1. This article is

This is an introduction of how to save DataFrame type data in HDF format to speed up data reading.

2. Contents

Save: store.put ('where to put the data in the h5 file', DataFrame name) Read: store ('Specify the location to read the data in the h5 file')

sample.py


import pandas as pd
import numpy as np
DATA_STORE = './data/asset.h5'

dat = [
    ['2019-07-01','9997','740'],
    ['2019-07-02','9997','749'],
    ['2019-07-03','9997','757'],
    ['2019-07-04','9997','769'],
    ['2019-07-05','9997','762'],
    ['2019-07-08','9997','860']
]
df4 = pd.DataFrame(dat,columns=["A","B","C"])
print("df4",df4)

# DATA_Asset created in STORE.Write DataFrame df4 in HDF format to h5 file.
#Format: store.put( 'Where to put the data in the h5 file' ,DataFrame name)

with pd.HDFStore(DATA_STORE) as store:
   store.put('general/test', df4)

# DATA_Asset created in STORE.Read data from h5 file.
#Format: store( 'Specify the location to read the data in the h5 file')

with pd.HDFStore(DATA_STORE) as store:
    df5 = store['general/test']    
print(df5)

output


df4
            A     B    C
0  2019-07-01  9997  740
1  2019-07-02  9997  749
2  2019-07-03  9997  757
3  2019-07-04  9997  769
4  2019-07-05  9997  762
5  2019-07-08  9997  860

df5
            A     B    C
0  2019-07-01  9997  740
1  2019-07-02  9997  749
2  2019-07-03  9997  757
3  2019-07-04  9997  769
4  2019-07-05  9997  762
5  2019-07-08  9997  860

Recommended Posts

Write data in HDF format
Export DB data in json format
Handle NetCDF format data in Python
format in python
Write Pulumi in Go
Write DCGAN in Keras
Write decorator in class
Sampling in imbalanced data
Write Python in MySQL
Image format in Python
I want to write in Python! (1) Code format check
Handle Ambient data in Python
Easily format JSON in Python
Write Pandoc filters in Python
Data handling 3 (development) About data format
Write beta distribution in Python
Data Manipulation in Python-Try Pandas_plyr
Write python in Rstudio (reticulate)
Write Spigot in VS Code
Extract classification information etc. from genbank data in xml format
Write Spider tests in Scrapy
Concisely write operations on every pair in your data using broadcast
Get Leap Motion data in Python.
Write a binary search in Python
Write a table-driven test in C
Read Protocol Buffers data in Python3
Write JSON Schema in Python DSL
How to write soberly in pandas
Python Application: Data Handling Part 3: Data Format
Automatically format Python code in Vim
Write an HTTP / 2 server in Python
Handle GDS II format in Python
Write A * (A-star) algorithm in Python
Data visualization in Python-draw cool heatmaps
Parsing CSV format data using SQL
Store RSS data in Zabbix (Zabbix sender)
Try to put data in MongoDB
Data prediction competition in 3 steps (titanic)
Write foreign key constraints in Django
Write selenium test code in python
Hashing data in R and Python
Write a pie chart in Python
Write a vim plugin in Python
Write a depth-first search in Python
Machine learning in Delemas (data acquisition)
Check the data summary in CASTable
Preprocessing in machine learning 2 Data acquisition
Write tests in GO language + gin
Write C unit tests in Python
Merge JSON format data with Ansible
Preprocessing in machine learning 4 Data conversion
Try to extract specific data from JSON format data in object storage Cloudian/S3