[PYTHON] Time series analysis 3 Preprocessing of time series data

Aidemy　2020/10/29

Introduction

Hello, it is Yope! I am a liberal arts student, but I was interested in the possibilities of AI, so I went to the AI-specialized school "Aidemy" to study. I would like to share the knowledge gained here with you, and I am summarizing it on Qiita. I am very happy that many people have read the previous summary article. Thank you! This is the third post in time series analysis. Nice to meet you.

• This article is a summary of what you learned in "Aidemy" "in your own words". It may contain mistakes and misunderstandings. Please note.

What to learn this time ・ Handle time series data with pandas ・ How to make time series data stationary

Handle time series data with pandas

Loading and displaying data

-Although the ultimate goal is to analyze time-series data with SARIMA, it is necessary to perform some preprocessing on the data passed at this time. -If time series data is given as a CSV file, reading will be done. Use __pd.read_csv ("file path") __.

Convert time information to index

-When analyzing time-series data, convert time information (Hour, Month, etc.) into an index to make it easier to handle. -The conversion procedure is as follows. (1) Define index information with __pd.date_range ("start", "end", freq = "interval") __. (2) Substitute the defined information in the index of the original data. ③ Delete the time information of the original data.

-At the start and end of the original data entered in ①, the interval can be confirmed with __df.head () __ and __df.tail () __. -As for the interval, if the data is composed in seconds, pass "S", minutes for "min", hours for "H", days for "D", and months for "M".

-Code

・ Result

Causes when data is not stationary

-The causes when time-series data does not have constantity include __ "trend" __ and "__ seasonal variation" __. -The expected value should be constant when there is stationarity, but if there is a __positive trend, it means that the expected value is on an upward trend __, so it cannot be said that there is stationarity. -Similarly, the autocorrelation coefficient should be constant when there is stationarity, but the autocorrelation coefficient is constant for data with seasonal fluctuations, that is, data in which the value suddenly increases or decreases only for a period of the year. It cannot be said that.

-In such a case, it is possible to obtain stationary data by performing __trend and conversion __ that removes seasonal fluctuations. -After creating a model with this steady-state data, the trend and seasonal fluctuations are combined again to build a model of the original series.

Make time series data stationary

Elimination of trends and seasonal fluctuations

-The following four methods can be mentioned to eliminate trends and seasonal fluctuations and to maintain stationarity. Details will be described later. ・ Uniform variance of fluctuation with __logarithmic transformation __ ・ Take __moving average __ to estimate the trend and remove it ・ Convert to __staff series __ (general) ・ Perform __seasonally adjusted __

Logarithmic transformation

-As seen in "Time Series Analysis 1", the change in data value can be moderated by performing __logarithmic conversion __. -By using this, the autocovariance can be made uniform for data with sudden changes in values such as seasonal fluctuations. That is, __seasonal fluctuations can be removed __. -However, __trend cannot be removed by this method __, so it is necessary to perform processing to remove the trend in addition to logarithmic conversion. -Logarithmic conversion can be done with __np.log (data) __.

・ Code

・ Result

moving average

-__ Moving average __ is __ to take the average of a certain section while moving the section __. -The moving average allows the data to be smoothed while retaining the characteristics of the original data. This makes it possible to remove __seasonal fluctuations and extract trends __. -As an example, when monthly data has seasonal fluctuations, seasonal fluctuations can be removed by taking 12 moving averages. The extracted trend can also be removed by "(original series)-(moving average)".

・ The moving average can be calculated as follows. __Data .rolling (window = number of moving averages) .mean () __

-Code (CO2 concentration data, moving average every 51 weeks (1 year)) -northeast-1.amazonaws.com/0/698700/3acdfc2c-6d10-a226-0f5c-740246fdfd67.png)

・ Result

Floor difference series

-As seen in "Time Series Analysis 1", handling data by taking the difference from the previous value is called scale difference series. -Trends and seasonal fluctuations can be eliminated by using a difference series. It is the most common way to maintain stationarity because it is easy to do. -To find the difference series, you can find it with __data.diff () __. ・ The one that obtains the first-order difference series is called the primary difference series, and the one that obtains the difference series of the primary difference series is called the secondary difference series.

・ Code

・ Result

Seasonally adjusted

-As seen in "Time Series Analysis 1", by making the series seasonally adjusted, trends and seasonal fluctuations can be removed from the original series. -As a mechanism of removal, the original series can be decomposed as "original series = trend + seasonal variation + residual", but if this is converted to "residual = original series- (trend + seasonal variation)" The residual is the original series minus trends and seasonal fluctuations. I am using this. -To make a seasonally adjusted series, perform __sm.tsa.seasonal_decompose (data, freq = interval specification) __.

-Code

・ Result

Summary

-Before performing time series analysis, index the time information of the time series data. -Since the data to be passed to the time series model must have stationarity, it is necessary to perform preprocessing such as making a difference series for the data without stationarity.

This time is over. Thank you for reading until the end.