[python] Frequently used techniques in machine learning

Data frame filled with all 0s

import pandas as pd
import numpy as np

rows = 4
cols = 4 
df = pd.DataFrame(np.zeros((rows, cols)))

numpy.float Get the number of decimal places of an array element of type 64

I used it when I wanted to split the element extracted from the numpy array and check the number of decimal places. Since split is a method of type str, convert it to str and then get it.

import numpy as np

narr = np.array([1.3334, 5.3343, 7.3322])

n_to_str = np.array2string(x[0])

print(len(n_to_str.split('.')[-1]))
# 4

Concatenate data frames

When you want to connect to the far right


a = [1,2,3]
b = [1,2,3]

bf = pd.DataFrame()
bf['a'] = a
bf['b'] = b
	a	b
0	1	1
1	2	2
2	3	3
a = [1,2,3]
b = [1,2,3]

bf = pd.DataFrame(a, columns=['a'])
bf.insert(1, "b", b)
	a	b
0	1	1
1	2	2
2	3	3

Connected together



a = [1,2,3]
b = [1,2,3]
c = [1,2,3]

cols = {
    "a":a,
    "b":b,
    "c":c
}

da = pd.DataFrame(cols)

#Connected horizontally
print(pd.concat([da,da], axis=1, ignore_index=True))
print()
#Connected vertically
# ignore_If index is not specified as True, it will be 012012.
print(pd.concat([da,da], axis=0, ignore_index=True))

   0  1  2  3  4  5
0  1  1  1  1  1  1
1  2  2  2  2  2  2
2  3  3  3  3  3  3

   a  b  c
0  1  1  1
1  2  2  2
2  3  3  3
3  1  1  1
4  2  2  2
5  3  3  3

Convert strings to datetime, timestamp format

It is a technique to convert to a format that can be handled by python when calculating date and time data.

Converting to timestamp makes it easier to handle when seconds, minutes, and hours move forward.

An example of trying to calculate without doing anything


#Example) 1:59:50 + 0:00::20 = 2:00:10
#When not converting to timestamp

hour = 1
minute = 59
sec = 50

add_hour = 0
add_minute = 0
add_sec = 20

if sec + add_sec > 60 or minute + add_minute > 60 or hour + add_hour > 24:
    #Carrying process
else:
    #Processing when it does not move up

Datetime string,Convert to timestamp


from datetime import datetime

s = "2020.07.10 22:59:59"

date = datetime.strptime(s, "%Y.%m.%d %H:%M:%S")
print(date)
# 2020-07-10 22:59:59
 
timestamp = datetime.timestamp(date)
print(timestamp)
# 1594389599.0

Return the timestamp value put in the list etc. to the character string

Put the timestamp in a list and then return it to a string


from datetime import datetime

t_list = [1594177973.0,1594177974.0,1594177975.0]
 
print(type(t_list[0]))
# float

#Convert to datetime type and then to string
datetime.fromtimestamp(t_list[0]).strftime("%H:%M:%S")
# '12:12:53'

Addition and subtraction of time

You can add and subtract to a datetime.datetime object using datetime.timedelta

Supports calculations for weeks, days, hours, minutes, seconds

import datetime

#Current time
now = datetime.datetime.now()
print(now)
# 2020-07-11 18:19:47.149523

#Add 10 seconds to the current time
now_add_10sec = now + datetime.timedelta(seconds=10)
print(now_add_10sec)
# 2020-07-11 18:19:57.149523

moving average

Use numpy convolve. It's faster than pandas rollin, and the code is short and easy to understand.

period = 3 #Average period
data = np.arange(1,10)
print(data)
print(len(data))

# mode = { same, valid, full }You can choose from
# same:The number of elements in the calculation result is the same size(not recommended)
# valid:There are many considerations, but the size is(period-1)Be scraped(Recommendation)
# full:Increased size (deprecated))

average = np.convolve(data, np.ones(period)/period, mode='valid')
print(average)
print(len(average))
print([np.nan]*(period-1) + average.tolist()) #Put Nan in the scraped area
[1 2 3 4 5 6 7 8 9]
9
[2. 3. 4. 5. 6. 7. 8.]
7
[nan, nan, 2.0, 3.0, 3.9999999999999996, 5.0, 6.0, 7.0, 8.0]

Recommended Posts

[python] Frequently used techniques in machine learning
Used in machine learning EDA
Python: Preprocessing in Machine Learning: Overview
Python: Preprocessing in machine learning: Data acquisition
[Python] Saving learning results (models) in machine learning
Python: Preprocessing in machine learning: Data conversion
[Machine learning] List of frequently used packages
Techniques often used in python short coding (Notepad)
Full disclosure of methods used in machine learning
Summary of evaluation functions used in machine learning
Get a glimpse of machine learning in Python
Machine learning in Delemas (practice)
Build an interactive environment for machine learning in Python
Machine learning / classification related techniques
Tool MALSS (application) that supports machine learning in Python
Coursera Machine Learning Challenges in Python: ex2 (Logistic Regression)
Tool MALSS (basic) that supports machine learning in Python
Machine learning with Python! Preparation
List of main probability distributions used in machine learning and statistics and code in python
[Python] Frequently used library code
Techniques for sorting in Python
Python Machine Learning Programming> Keywords
Coursera Machine Learning Challenges in Python: ex1 (Linear Regression)
Python frequently used code snippets
Attempt to include machine learning model in python package
Frequently used commands in virtualenv
Beginning with Python machine learning
MALSS, a tool that supports machine learning in Python
The result of Java engineers learning machine learning in Python www
Coursera Machine Learning Challenges in Python: ex7-2 (Principal Component Analysis)
Implement stacking learning in Python [Kaggle]
Machine learning with python (1) Overall classification
Machine learning summary by Python beginners
Automate routine tasks in machine learning
[Python] Basic knowledge used in AtCoder
Widrow-Hoff learning rules implemented in Python
Classification and regression in machine learning
<For beginners> python library <For machine learning>
Machine learning in Delemas (data acquisition)
Implemented Perceptron learning rules in Python
Preprocessing in machine learning 2 Data acquisition
Random seed research in machine learning
"Scraping & machine learning with Python" Learning memo
Preprocessing in machine learning 4 Data conversion
[Python] A memo of frequently used phrases (by myself) in Python scripts
How about Anaconda for building a machine learning environment in Python?
Coursera Machine Learning Challenges in Python: ex5 (Adjustment of Regularization Parameters)
Machine learning
python learning
A class that summarizes frequently used methods in twitter api (python)
Python & Machine Learning Study Memo: Environment Preparation
Notes on PyQ machine learning python grammar
Summary of frequently used commands in matplotlib
Python + Selenium Frequently used operation method summary
Use machine learning APIs A3RT from Python
Machine learning with python (2) Simple regression analysis
I installed Python 3.5.1 to study machine learning
Why Python is chosen for machine learning
"Python Machine Learning Programming" Summary Note (Jupyter)
[Shakyo] Encounter with Python for machine learning
[Python] First data analysis / machine learning (Kaggle)