Convert from Pandas DataFrame to System.Data.DataTable using Python for .NET

Python and .NET table data exchange

When exchanging data frames with Python-> .NET in the first place, the .NET side has a standard library class called DataTable, which is suitable. It doesn't have the high-performance data processing power of Pandas and dplyr, but it can be mapped for the time being. If you want to call it from Python on BayesServer, it seems that it is recommended to access the Java library using JPype as shown below.

Setup Python

On the other hand, it says that pythonnet is fine, and I'm not sure why I have to go through a Java wrapper even though it's native to .NET, so I decided to go there. The R wrapper also goes through the Java wrapper. With this, you have to look at two types of API references. (Because I will modify the .NET library anyway and rewrap it) I should have used Python from the beginning. .. ..


pythonnet is a Python to .NET access library called Python for .NET. You can use it both when calling .NET from Python and when calling Python from .NET.

Python + .NET may remind you of IronPython, but it's a CLI language that runs on .NET. With this, the support for libraries that have a good reputation in data science etc. is limited. Or rather, numpy doesn't work.

Since pythonnet only connects languages, you can use it while keeping the good points of both. It also supports the latest .NET Core and so on. Speaking sensuously

import clr
from System.Windows.Forms import Form

There is a taste of import clr.

dynamic np = Py.Import("numpy");
var x = np.cos(np.pi * 2);

There is an article by @hogegex in this direction.

Call python from .NET (C #)

Possible uses include calling .NET-specific libraries (business systems, etc.) from Python, and throwing numerical calculation tasks from .NET. It seems good to be able to create an app using the .NET GUI library on Python. However, my purpose is to call the BayesServer library from Python, so if I can do the following, I'm almost done.

Python DataFrame-> .NET DataTable conversion

The BayesServer reference provides a dataframe mapping helper function using jpype1 as a helper class.

Pandas DataFrame helper functions

Probably, conversion using jpype1 has a habit, so I think that they have prepared such a thing, but pythonnet can write it more straightforwardly. I almost rewrote it for .NET with copy and paste.

# %%
import numpy as np
import pandas as pd
import clr
from System.Data import *
# %%
def _to_net_class(data_type):
    Converts numpy data type to equivalent .NET class
    :param data_type: the numpy data type
    :return: The Net Class
    if data_type == np.int32:
        return clr.GetClrType(Int32)  
    if data_type == np.int64:
        return clr.GetClrType(Int64)  
    if data_type == np.float32:
        return clr.GetClrType(Single) 
    if data_type == np.float64:
        return clr.GetClrType(Double) 
    if data_type == np.bool:
        return clr.GetClrType(Boolean) 
    if data_type == np.object:
        return clr.GetClrType(Object) 

    raise ValueError('dtype [{}] not currently supported'.format(data_type))

# %%
def to_data_table(df):
    data_table = DataTable()

    for name, data_type in df.dtypes.iteritems():
        net_class = _to_net_class(data_type)

    for index, row in df.iterrows():
        xs = [None if pd.isnull(x) else x for x in row]

    return data_table

With this, pandas data can be brought into the .NET library, so it may be easier to exchange with a GUI such as a so-called Excel table or a database. Think of clr.GetClrType () as an alternative to typeof (or Object.GetType ()).

Recommended Posts

Convert from Pandas DataFrame to System.Data.DataTable using Python for .NET
Python pandas: Search for DataFrame using regular expressions
[python] Create table from pandas DataFrame to postgres
[Python] Convert list to Pandas [Pandas]
Convert list to DataFrame with python
[Python] Random data extraction / combination from DataFrame using random and pandas
Convert DataFrame column names from Japanese to English using Googletrans
From Python to using MeCab (and CaboCha)
Convert from katakana to vowel kana [python]
[Python] Add total rows to Pandas DataFrame
Python> Output numbers from 1 to 100, 501 to 600> For csv
Create a dataframe from excel using pandas
Convert from Markdown to HTML in Python
From preparation for morphological analysis with python using polyglot to part-of-speech tagging
[Python / Tkinter] Search for Pandas DataFrame → Create a simple search form to display
[Python] Convert from DICOM to PNG or CSV
[Python pandas] Create an empty DataFrame from an existing DataFrame
I want to email from Gmail using Python.
Convert STL to Voxel mesh using Python VTK
Convert pandas dataframe elements to regular string type
[Python learning part 3] Convert pandas DataFrame, Series, and standard List to each other
I tried to execute Python code from .Net using Pythonnet (Hallo World edition)
Convert 202003 to 2020-03 with pandas
Changes from Python 3.0 to Python 3.5
Changes from Python 2 to Python 3.0
Python application: Pandas # 3: Dataframe
How to convert Python # type for Python super beginners: str
Push notifications from Python to Android using Google's API
Summary of pre-processing practices for Python beginners (Pandas dataframe)
Convert comma-separated numeric strings to numbers in Pandas DataFrame
MessagePack-Call Python (or Python to Ruby) methods from Ruby using RPC
How to convert JSON file to CSV file with Python Pandas
[Python] Summary of table creation method using DataFrame (pandas)
Copy S3 files from Python to GCS using GSUtil
[Introduction to Python] How to write repetitive statements using for statements
Query from python to Amazon Athena (using named profile)
A real way for people using python 3.8.0-2 from windows to work with multibyte characters
Flatten using Python yield from
[python] Convert date to string
Post from Python to Slack
Convert numpy int64 to python int
Use IvyFEM (Finite Element Method Library for .NET) from Python
Cheating from PHP to Python
Compile Tesseract for Tess4J to transcribe from images using CentOS
Try to operate an Excel file using Python (Pandas / XlsxWriter) ①
Try to operate an Excel file using Python (Pandas / XlsxWriter) ②
Anaconda updated from 4.2.0 to 4.3.0 (python3.5 updated to python3.6)
Post to Twitter using Python
Start to Selenium using python
Convert Scratch project to Python
[Python] Convert Shift_JIS to UTF-8
How to convert Python # type for Python super beginners: int, float
[For beginners] Script within 10 lines (4. Connection from python to sqlite3)
Introduction to Python for VBA users-Calling Python from Excel with xlwings-
Switch from python2.7 to python3.6 (centos7)
Connect to sqlite from python
100 Pandas knocks for Python beginners
[Python] Introduction to graph creation using coronavirus data [For beginners]
~ Tips for beginners to Python ③ ~
Data analysis using python pandas
Export pandas dataframe to excel