[PYTHON] I participated in PyData Tokyo Meetup # 2

I participated in PyData Tokyo Meetup # 2-New Data Analysis Infrastructure! The following is a memo that I summarized while listening, so it is not particularly organized.

• For PyData Tokyo itself, please refer to the following page. http://pydatatokyo.connpass.com/

Contents

Introduction and use of Amazon Kinesis

Amazon Kinesis is a fully managed service that can process high-volume continuous data in real time.

• Use Case --Real-time status grasp of services and systems --Anomaly detection (e.g. Unauthorized access detection) --Service improvement (e.g. Real-time service using social data, recommendation based on recent actions)
• Constitution --Create a Stream, which consists of one or more Shards --Shard has the capacity of 1MB / sec, 1000 TPS on the data input side and 2MB / sec, 5 TPS on the data processing side. --Data can be entered with PutRecord API (** Python ** can of course be used (boto)) --Since it is distributed to Shard based on the partition key, the partition key design is important for Sharding. --Kinesis assigns a unique sequence number in Stream, so you can get data as many times as you like with the sequence number (within 24 hours → you may be able to set the time more flexibly in the future) --GetShardIteator API to get the data in Shard and GetRecords API to input the data

Kinesis Client Library (KCL) now supports ** Python **

• Kinesis Client Library for Python --Using KCL for Java'MultiLangDeamon'as a resident process, you can write the main logic of data processing in Python. --Data processing is started as a subprocess --Data communication between'MultiLangDaemon'and subprocesses is done using STDIN / STDOUT with the defined protocol.

The story of AWS + Jubatus was personally interesting. There wasn't much talk about machine learning at AWS re: Invent, so this talk raised my expectations considerably.

Highlights from PyData NYC

This is a story I went to PyData NYC 2014.

――How do you handle big data using Python? ――It seems that PySpark was very hot --Separate data and operations --Unified interface to DB --You can put the data extracted from Mongo into Spark -Advanced Scikit-Learn: A fairly thick tutorial -Beaker Notebook: Notebook that can use various languages at the same time (runs locally) --Python → You can pass variables with JavaScript

The story of SymPy was quite interesting. Is there a demand for Python → Fortran? Actually, I haven't used it yet, but Caffe seems to be very interesting. Let's use it.

Impressions

This was my first time participating, but I really liked PyData, so it was fun \ (^ o ^) / There are many stories that I can use in my work, so I will try various things. And PyData NYC, I want to go someday. I would also like to participate in the next and subsequent events mm Thank you!

Now that it's a social gathering, you can drink!