[PYTHON] Data engineers learn DevOps with a view to MLOps. ① Getting started

tl; dr A series of systematically mastering DevOps / MLOps practices for the future In my case, I thought it would be a good idea to start by connecting the front-end language Typescript (ts) with the back-end language Scala / Python (with Spark ecosystem).

Reason for judgment:

1)Front-end developer(js/ts user)Python is friendly
　※js/It seems that one of the standard programming languages used by ts users is python ★ 1
2)High affinity between ts model definition and spark model definition
* Typescript interface definition and case class for Spark are especially similar.
3)Mutual calling between ts and spark is becoming easier (the magic of "jsii").
* For example, using jsii, test a nodejs app written in typescript in Scala./Now you can do it in Python.
4) terraform/ansible/docker/My challenge with hashi stack is the modern front end.

★ 1 Source: "Second programming language" used by JavaScript users, first place is Python

[1] Maturity of the Typescript ecosystem on nodejs.

Typescript is a typed js compatible language. Typescript may be a safer option for engineers who write their backends in a typed language (C # / Scala etc.), although it may have a higher threshold than js. It seems that know-how for running Typescript on nodejs, which is an indispensable option for web development, has also been accumulated.

In the following, I will take the Nest.js framework, which seems to be one of the recent solutions of Typescript on nodejs, as an example. Will continue to be.) Reference Nest.js is wonderful

[2] Between the back end and the front end.

Typescript model definitions with interfaces are compatible with C # and Scala. Let me give you an example. ↑ Nest.js Great article, [Creating a User Interface with Nest.js](https://qiita.com/kmatae/items/5aacc8375f71105ce0e4#user%E3%82%A4%E3%83%B3%E3%82 % BF% E3% 83% BC% E3% 83% 95% E3% 82% A7% E3% 83% BC% E3% 82% B9% E3% 81% AE% E4% BD% 9C% E6% 88% 90 The interface user.interface.ts of the User model created in) is as follows:

typescript:user.interface.ts

export interface IUser {
    id: string;
    name: string;
    kana: string;
    email: string;
    postcode: string;
    address: string;
    phone: string;
    password: string;
    admin: boolean;
    createdAt: Date;
    updatedAt: Date;
}

In contrast, the corresponding scala classes for Spark are, for example:

IUser.scala

case class IUser (
    id: String,
    name: String,
    kana: String,
    email: String,
    postcode: String,
    address: String,
    phone: String,
    password: String,
    admin: Boolean,
    createdAt:TimeStamp,
    updatedAt:TimeStamp
)

It is a level that can be done by mechanical replacement (I used scala here, but it is the same with Python (PySpark) with type annotation like Typscript). It is practically easy to use only basic types such as String / Int / Boolean / TimeStamp for properties that are persisted to Spark. If the front end side can write safely in the same way, the story is quick (nest.js seems to be safe and productive to write ... Do you actually write it yourself? Please or not). If you can generate json from node.js written in Typescript and pour it into spark (with Kafka etc. in between or semi-stream processing), the rest will be irrelevant. On the backend side, Spark is one of the standard solutions for describing data hubs between datastores (connecting csv, cassandra, hive, neo4j, machine learning infrastructure etc.).

[3] Typescripe on nodejs can be tested in Python / Scala via jsii.

The jsii made by AWS that I learned from the following article. Introducing the magical "jsii" that runs programs written in TypeScript with Python etc.

jsii means that TypeScript code can be called directly from the Python / JVM language (Scala) / C #. This means you can write Typescript test code in Python or Scala. In the system that has entered actual operation, the Typescript model will change depending on the circumstances on the front end side (service expansion, etc.). It's nice to be able to continuously test the transfer of data as the model changes (testing when bringing untyped data (eg CSV) into the spark world is quite cumbersome).

I will move my hand from the next time.

Recently, I have personally studied how to quickly create the back side (data infrastructure) of a system with a web front end that pursues performance. I'm not familiar with front-end technology, but I've been able to get a feel for it by touching Svelte / typescript / nodejs / firebase ... in order. In the future, I would like to try connecting with the backend spark side while moving my hands with nestjs / SSR / ts2elm .... I hope that github action, which can CI / CD targeting nodejs and Docker, will be useful.