[LINUX] I tried reading data from a file using Node.js.

Introduction

This article is recorded by a student studying JS instead of a memo. Don't expect the content.

Open a virtual environment

This time it will be done on Ubuntu, so start the virtual environment with iTerm2.

Where it started

  1. Virtual Box 2.Vagrant I use Ubuntu in a virtual environment that uses two software. cd ~/vagrant/ubuntu vagrant up vagrant ssh

Move to the directory where Ubuntu is installed. vagrant up is a command to start Ubuntu installed on a virtual PC, and vagrant ssh connects to SSH with the Vagrant virtual machine set.

1. Write a program to read from a file


'use strict';
const filesystem=require('fs');
const readline=require('readline');
const rs =fs.createReadStream('file name');
const rl=readline.createInterface({input:rs, output:{}});
const prefectureDataMap = new Map();
rl.on('line', lines =>{
  console.log(lines);
});

The first line is the description for using JS in strict mode.

The second and third lines call the module in Node.js with require. require is an npm (Node Package Manager) module that allows you to use extensions. You can assign a module to a variable and use it in JS with the following description.

The module called this time is fs (filesystem) for handling the file and readline for reading the file line by line.

The fourth line uses a stream to read the file and assign it to a variable called rs. In addition to this, the fs module also has functions such as fs.readFileSync that read data synchronously.

The 5th line is set as follows using a module (readline) that can read the stream line by line.

const rl = readline.createInterface({
//Setting the stream you want to read
  input: rs,
//Setting the stream you want to export
  output: {}
});

The 6th line uses a Map object to handle a combination of keys and values. Here, the Map object is assigned to the variable and the Map object is newly initialized.

The 7th line is an object called rl, which fires the following function when an event called line occurs. This time, the argument lines is displayed on the console.

Change the function on line 2.6

The function passed to the on method on the 6th line has been changed to the following.

rl.on('line', lines => {


  const cutline = lines.split(',');
  const year = parseInt(cutline[0]);
  const prefecture = cutline[1];
  const popu = parseInt(cutline[3]);
 if (year === 2010 || year === 2015) {
    let vofmap = prefectureDataMap.get(prefecture);
    if (!vofmap) {
      vofmap = {
        p10: 0,
        p15: 0,
        change: null
      };
    }
    if (year === 2010) {
      vofmap.p10 = popu;
    }
    if (year === 2015) {
      vofmao.p15 = popu;
    }
    prefectureDataMap.set(prefecture, value);
  }
});

The split () method on the second line splits the String into an array of strings by splitting it by the specified delimiter string. The method is used as follows.

String.sprit(Specified delimiter)


From the 3rd line, a part of the array created from the character string is assigned to the variable. parseInt () parses a string argument and returns an integer value of the specified radix (base of mathematical notation). Simply put, what was a character string value has been changed to a number (integer value) to make it easier to handle.

From the 6th line, the following program is running only when the variable has the value specified by using the if statement.

The 7th line fetches the key value while specifying the key on the map. See below for how to write

Map name.get(Key name);


In the 8th line, if the value specified for the key name does not exist, the value corresponding to the key name is assigned for the first time.

I will explain this part after the 9th line.

      vofmap.p10 = popu;
    }
    if (year === 2015) {
      vofmao.p15 = popu;
    }
prefectureDataMap.set(prefecture, value);
  }
});

Here, it is assigned to each object of the value corresponding to the key name created in the Map object earlier. After that, the key and value set is registered in the previous map. See below for how to write


Map name.set(Key name,value)

3. Write a function to fire with close

The following contents are described at the end.

rl.on('close', () => {


  for (let [key, vofmap] of prefectureDataMap) {
    vofmap.change = vofmap.popu15 / vofmap.popu10;
  }
  const rankingArray = Array.from(prefectureDataMap).sort((pair1, pair2) => {
    return pair2[1].change - pair1[1].change;
  });
  const rankingStrings = rankingArray.map(([key, value]) => {
    return (
      key +
      ': ' +
      value.popu10 +
      ':' +
      value.popu15 +
       ':'+
      value.change
    );
  });
  console.log(rankingStrings);
});

The first line is the same as before, and when the event called close occurs in the object called rl, the following function is fired.

The 2nd to 4th lines loop the key name and the corresponding value for the map called prefectureDataMap with the for ~ of syntax. After that, a new object called change is created in the map and the applicable value is assigned.

The 5th line is converted to an array after performing the function. This time, sort is used to change the order according to the size of the value of the callback function. See below for the description method.

Array.from(Conversion target,function{})


The 9th line uses the map function. It is different from the one that uses the key name etc. The map function transforms each element of the Array into the content to which the given function is applied. This time, everything is converted as a character string.

node JS file name

If you execute the above and no error occurs in REPL, it is successful. Thank you for your hard work! !!

What is a stream?

When dealing with very large data, if all the data is read into the memory at once, the memory will be tight and the program performance may be degraded. In such cases, use streams. A stream is one that can gradually read, write, and transform data.

Recommended Posts

I tried reading data from a file using Node.js.
I tried running python etc. from a bat file
I tried collecting data from a website with Scrapy
I tried to get data from AS / 400 quickly using pypyodbc
I tried scraping conversation data from Askfm
I tried playing a ○ ✕ game using TensorFlow
I tried using YOUTUBE Data API V3
I tried drawing a line using turtle
I tried using UnityCloudBuild API from Python
I tried using Headless Chrome from Selenium
I tried using pipenv, so a memo
I tried to get data from AS / 400 quickly using pypyodbc Preparation 1
I tried to perform a cluster analysis of customers using purchasing data
I tried using Pythonect, a dataflow programming language.
Run a Python file from html using Django
I tried DBM with Pylearn 2 using artificial data
I tried using a database (sqlite3) with kivy
I tried to make a ○ ✕ game using TensorFlow
I tried using parameterized
I tried using argparse
I tried using mimesis
I tried using anytree
I tried using Summpy
I tried using coturn
I tried using Pipenv
I tried using matplotlib
I tried using "Anvil".
I tried using Hubot
I tried using ESPCN
I tried using openpyxl
I tried using Ipython
I tried using PyCaret
I tried using cron
I tried using ngrok
I tried using face_recognition
I tried using Jupyter
I tried using PyCaret
I tried using Heapq
I tried using doctest
I tried using folium
I tried using jinja2
I tried using folium
I tried using time-window
I tried to make a suspicious person MAP quickly using Geolonia address data
Python-Read data from a numeric data file and calculate covariance
I tried hosting a Pytorch sample model using TorchServe
I tried clustering ECG data using the K-Shape method
I tried using the API of the salmon data project
[Python] I tried running a local server using flask
I tried drawing a pseudo fractal figure using Python
I tried using PySpark from Jupyter 4.x on EMR
I tried using Python (3) instead of a scientific calculator
PyTorch Learning Note 2 (I tried using a pre-trained model)
I tried to draw a configuration diagram using Diagrams
I tried to make a function to retrieve data from database column by column using sql with sqlite3 of python [sqlite3, sql, pandas]
[I tried using Pythonista 3] Introduction
I tried using easydict (memo).
I tried face recognition using Face ++
I tried using Random Forest
I tried using BigQuery ML
Create a dummy data file