[PYTHON] Smoothing of time series and waveform data 3 methods (smoothing)

What is this

I sometimes deal with time series and waveform data, so I tried some smoothing. I will write down the three methods together as a memorandum.

Waveform data generation

Generate the waveform data to be used this time

import numpy as np
import matplotlib.pyplot as plt


N = 1024            #The number of samples
dt = 0.001          #Sampling cycle[s]
f1, f2 = 30, 90    #frequency[Hz]
a1, a2 = 1.5, 0.8 #amplitude

t = np.arange(0, N*dt, dt) #time[s]
wave = a1*np.sin(2*np.pi*f1*t) + a2*np.sin(2*np.pi*f2*t) + 0.3 * np.random.randn(N) #signal


fig, ax = plt.subplots(figsize=(14.0, 6.0))
ax.plot(t, wave)
ax.set_xlabel("Time [s]")
ax.set_ylabel("Signal")
ax.grid()
plt.title("wave")
plt.show()

fig, ax = plt.subplots(figsize=(14.0, 6.0))
ax.plot(t, wave)
ax.set_xlim(0, 0.1)
ax.set_xlabel("Time [s]")
ax.set_ylabel("Signal")
ax.grid()
plt.title("enlarged view")
plt.show()

wave.png

1. Moving average

window = 5 #Moving average range
w = np.ones(window)/window

x = np.convolve(wave, w, mode='same')

fig, ax = plt.subplots(figsize=(14.0, 6.0))
ax.plot(t, x, label='moving average')
ax.plot(t, wave, alpha=0.3, label='wave')
ax.set_xlabel("Time [s]")
ax.set_ylabel("Signal")
ax.grid()
plt.legend()
plt.title("moving average window=5")
plt.show()

fig, ax = plt.subplots(figsize=(14.0, 6.0))
ax.plot(t, x, label='moving average')
ax.plot(t, wave, alpha=0.3, label='wave')
ax.set_xlim(0, 0.1)
ax.set_xlabel("Time [s]")
ax.set_ylabel("Signal")
ax.grid()
plt.legend()
plt.title("enlarged view")
plt.show()

mooving_avg.png

  1. Savitzky-Golay filter

2.1 Non-differentiating pattern

import scipy.signal

#Degree of polynomial: 2
#Frame length: 5
x = scipy.signal.savgol_filter(wave, 5, 2, deriv=0)

fig, ax = plt.subplots(figsize=(14.0, 6.0))
ax.plot(t, x, label='Savitzky-Golay')
ax.plot(t, wave, alpha=0.3, label='wave')
ax.set_xlabel("Time [s]")
ax.set_ylabel("Signal")
ax.grid()
plt.legend()
plt.title("Savitzky-Golay deriv=0")
plt.show()

fig, ax = plt.subplots(figsize=(14.0, 6.0))
ax.plot(t, x, label='Savitzky-Golay')
ax.plot(t, wave, alpha=0.3, label='wave')
ax.set_xlim(0, 0.1)
ax.set_xlabel("Time [s]")
ax.set_ylabel("Signal")
ax.grid()
plt.legend()
plt.title("enlarged view")
plt.show()

sav0.png

2.2 First derivative

x = scipy.signal.savgol_filter(wave, 5, 2, deriv=1)

fig, ax = plt.subplots(figsize=(14.0, 6.0))
ax.plot(t, x)
ax.set_xlabel("Time [s]")
ax.set_ylabel("Signal")
ax.grid()
plt.title("Savitzky-Golay deriv=1")
plt.show()

fig, ax = plt.subplots(figsize=(14.0, 6.0))
ax.plot(t, x)
ax.set_xlim(0, 0.1)
ax.set_xlabel("Time [s]")
ax.set_ylabel("Signal")
ax.grid()
plt.title("enlarged view")
plt.show()

sav1.png

2.3 Second derivative

x = scipy.signal.savgol_filter(wave, 5, 2, deriv=2)

fig, ax = plt.subplots(figsize=(14.0, 6.0))
ax.plot(t, x)
ax.set_xlabel("Time [s]")
ax.set_ylabel("Signal")
ax.grid()
plt.title("Savitzky-Golay deriv=2")
plt.show()

fig, ax = plt.subplots(figsize=(14.0, 6.0))
ax.plot(t, x)
ax.set_xlim(0, 0.1)
ax.set_xlabel("Time [s]")
ax.set_ylabel("Signal")
ax.grid()
plt.title("enlarged view")
plt.show()

sav2.png

3. Fourier transform

3.1 Fast Fourier Transform

x = np.fft.fft(wave)
x = np.abs(x) #Convert complex numbers to absolute values
x = x / N * 2 #Amplitude adjustment
fq = np.linspace(0, 1.0/dt, N) #Frequency adjustment

fig, ax = plt.subplots(figsize=(14.0, 6.0))
ax.plot(fq[: int(N / 2)], x[: int(N / 2)])
ax.set_xlabel("Frequency [Hz]")
ax.set_ylabel("Swing")
ax.grid()
plt.title("Fast Fourier Transform")
plt.show()

#Frequency 30,90, amplitude 1.5, 0.Peak is seen around 8

fft.png

3.2 Inverse Fast Fourier Transform without Noise

threshold = 0.6 #Amplitude threshold

x = np.fft.fft(wave)
x_abs = np.abs(x)
x_abs = x_abs / N * 2
x[x_abs < threshold] = 0

x = np.fft.ifft(x)
x = x.real #Extract only the real part from a complex number

fig, ax = plt.subplots(figsize=(14.0, 6.0))
ax.plot(t, x, label='IFFT')
ax.plot(t, wave, alpha=0.3, label='wave')
ax.set_xlabel("Time [s]")
ax.set_ylabel("Signal")
ax.grid()
plt.legend()
plt.title("Inverse Fast Fourier Transform")
plt.show()

fig, ax = plt.subplots(figsize=(14.0, 6.0))
ax.plot(t, x, label='IFFT')
ax.plot(t, wave, alpha=0.3, label='wave')
ax.set_xlim(0, 0.1)
ax.set_xlabel("Time [s]")
ax.set_ylabel("Signal")
ax.grid()
plt.legend()
plt.title("enlarged view")
plt.show()

ifft.png

When I have time, I would like to add the theory part and describe other methods.

Recommended Posts

Smoothing of time series and waveform data 3 methods (smoothing)
About time series data and overfitting
Differentiation of time series data (discrete)
Time series analysis 3 Preprocessing of time series data
Reading OpenFOAM time series data and sets data
Comparison of time series data predictions between SARIMA and Prophet models
Power of forecasting methods in time series data analysis Semi-optimization (SARIMA) [Memo]
Acquisition of time series data (daily) of stock prices
View details of time series data with Remotte
[Python] Plot time series data
[Latest method] Visualization of time series data and extraction of frequent patterns using Pan-Matrix Profile
Graph time series data in Python using pandas and matplotlib
A story about clustering time series data of foreign exchange
"Measurement Time Series Analysis of Economic and Finance Data" Solving Chapter End Problems with Python
Calculation of time series customer loyalty
Python: Time Series Analysis: Preprocessing Time Series Data
How to extract features of time series data with PySpark Basics
When plotting time series data and getting a matplotlib Overflow Error
Forecasting time series data with Simplex Projection
Predict time series data with neural network
Example of using class variables and class methods
[Python] Accelerates loading of time series CSV
Time series analysis 4 Construction of SARIMA model
Time series data anomaly detection for beginners
Separation of design and data in matplotlib
Plot CSV of time series data with unixtime value in Python (matplotlib)
Conversion of time data in 25 o'clock notation
Easy introduction of python3 series and OpenCV3
[Kaggle] I tried feature engineering of multidimensional time series data using tsfresh.
How to handle time series data (implementation)
Time series analysis # 6 Spurious regression and cointegration
Format and display time series data with different scales and units with Python or Matplotlib
List of frequently used built-in functions and methods
Kaggle Kernel Method Summary [Table Time Series Data]
Organize the meaning of methods, classes and objects
Frequently used methods of Selenium and Beautiful Soup
How to read time series data in PyTorch
Data cleansing 3 Use of OpenCV and preprocessing of image data
Basic operation of Python Pandas Series and Dataframe (1)
I implemented "Basics of Time Series Analysis and State Space Model" (Hayamoto) with pystan
Reading, summarizing, visualizing, and exporting time series data to an Excel file with Python
[Introduction to element decomposition] Let's arrange time series analysis methods in R and python ♬
Time Series Decomposition
Implementation of clustering k-shape method for time series data [Unsupervised learning with python Chapter 13]
I just wanted to extract the data of the desired date and time with Django
It's time to seriously think about the definition and skill set of data scientists
Analysis of financial data by pandas and its visualization (2)
Full-width and half-width processing of CSV data in Python
Features that can be extracted from time series data
List of Python libraries for data scientists and data engineers
Analysis of financial data by pandas and its visualization (1)
Knowledge and study methods required for future data analysts
relation of the Fibonacci number series and the Golden ratio
Statistical hypothesis test of A/B test and required number of data
Visualize data and understand correlation at the same time
Overview and tips of seaborn with statistical data visualization
Summary of Pandas methods used when extracting data [Python]
Story of image analysis of PDF file and data extraction
Time series data prediction by AutoML (automatic machine learning)
Analysis of measurement data ②-Histogram and fitting, lmfit recommendation-
Visualization method of data by explanatory variable and objective variable