This entry is a sequel to the following two entries.
Regarding the data analysis environment, there are various ways of thinking regarding the number of analysts, the type of business of the company, the sense of scale, etc., and it is annoying every time I hear a story. I have a feeling for the future of Datalab, but in conclusion, unfortunately, Datalab alone does not seem to be functional enough. However, now that the new analysis technology called deep learning has become the de facto standard to some extent, I think it is meaningful to reorganize the analysis environment, so I will write it down.
Retty's case became a hot topic, and I myself found it helpful. Deep learning has become popular these days, so in practice it will settle in this direction. Our analysis environment is created in a similar direction.
Also, as seen in the social game industry a long time ago, I improved the game by data mining and made ○ billion yen, but it does not have as much impact, but Mr. Retty mentioned above recovery more than the investment seems to have been completed, and it seems that there is no point in investing in an analytical environment.
As for the story from here, the following is assumed.
--There are multiple analysts (a reasonable number) --There are a wide variety of themes to analyze --It also requires resources such as GPU that are a little more expensive (for deep learning, etc.)
I will write it along with the transition of the analysis environment.
At the beginning of the analysis, it will look like this.
--Each person builds an analysis environment as needed ――The sources of data are also different, and it feels like each person picks up what they need for analysis. ――Since it is not clear what data you can use and what data you can use
I think that it is unavoidable if the number of people is small and the cost for analysis cannot be spent.
As the next step, I think you want to have some data sources.
--Prepare shared storage, cloud storage, etc., and aggregate data sources there --Introduce commercial DWH, Hadoop, etc. to perform data processing and primary processing ――Each person brings the processing result to the analysis environment at hand and analyzes it.
As usual, the analysis environment itself seems to be different, but I think there are many points that have been improved by unifying the data sources. However, due to the lack of an analysis environment, it is expensive to perform analysis as a team, and each person remains at the level of independent analysis work.
The next step is to collaborate on analytical work. At this stage, if the analysis environment is not complete, it will be difficult to divide the analysis work and do things like OJT, so there will be a need to prepare an analysis environment.
――In most cases, you will leave the installation procedure etc. in the document, and if you are new to the site, you will have to build your own analysis environment. --Using Jupyter to prepare Python versions, like ... ――The nice thing is, for example, you can provide a VM with an analysis environment built, or prepare an execution environment at the docker image level and share a notebook (ipynb) file.
Gradually, it will be an important phase to standardize and share the analysis environment from the engineer's point of view. Before the advent of deep learning, it can be said that a somewhat satisfactory analysis environment was created at this stage.
There are issues in operating it as an analysis environment with the configuration up to this point.
That is, because the data processing part (DWH, Hadoop / Hive, Redshift, etc.) is shared, queries that bother other people and the administrator (fucking queries) are created. The administrator of the DWH part needs to keep an eye on the fucking query and kill it.
However, from the analyst's point of view, if there are no clear rules, there is a tendency to proceed with data processing with queries as much as possible, and newly joined people will issue fucking queries without maliciousness and become apt. I will. The fucking query problem is generally a cat-and-mouse game and the perception that there is no fundamental solution.
Then, as a recent trend, there is a trend to introduce GPU in order to tackle deep learning properly. In general, GPU machines are a bit more expensive to procure than CPU machines, and are still not cheap and easy to use in the cloud. Also, instead of wanting a large amount of GPU resources, there is a tendency to introduce high-performance and several GPUs for analysis on a trial basis at best, and there is a case where the environment itself where several GPUs can be used is created on-premise. Seems to be most.
In that case, GPU machines are often shared, and in that case it seems that Docker images are often used. The same is true for Retty's case. Since deep learning requires a lot of data, it is natural to attach a large disk to the GPU machine and place it nearby.
Of course, the GPU is shared, so exchanges such as borrowing the GPU and returning it will begin to occur in chats and the like. To make matters worse, deep learning takes time to learn, so once you borrow it, you can't easily return it. When it happens that the work that wants to use the GPU overlaps, the resources are exhausted immediately, and conversely, when it is not used, it is not used at all. Since the data is also on-premise, backup and capacity problems may occur.
How does a Datalab-centric workflow fit in here? I think it will be for the following.
--Leave it to BigQuery and solve fucking queries with brute force
――You can stand up for those who use GCE --Resources that are not shared with others can be prepared immediately with a single command! --You can raise or lower the instance specifications as needed.
--Datalab has access to GCS, so capacity issues are likely to be virtually resolved
--Can work with your Google account! ――Since it can be restricted appropriately in cooperation with IAM, it seems to be relatively easy to manage.
So, Datalab, let's all use it! I'd like to say, but it collapsed because GPU Instant can not be used (laugh) It's regrettable, regrettable. Also, as some pointed out, Datalab also has Python2 problems.
There are still many issues to be solved, but I personally would like to pay attention to the data analysis environment on GCP centered on Datalab.
The story is covered with Cloud NEXT Extended [Away study session to be held on April 1](https:: //algyan.connpass.com/event/52494/).