[PYTHON] Compete for file IO in various languages and compare speeds

I read / write a CSV file of about 1GB in multiple programming languages and time it!

Common rules

--Check using the same file in any language --Read the file line by line and write line by line --All Docker Images should use base (Ubuntu 18.04 LTS) to eliminate OS differences.

First of all, the result

language Average time run 5 times(Seconds)
C-lang 6.2962
C++ 6.0538
C# 17.1798
Golang 10.3966
Java 21.0840
Python 32.6949
Ruby 30.7153
Scala 21.8364

After all C / C ++ is by far the fastest! !! Go language in second place

Implementation sample

I wrote C ++ for the first time in 5 years, so I don't think it's beautiful, but since it was the fastest, I'll post it as a sample.

I implemented it in other languages based on the following behavior.

--Calculate the average time executed 5 times --Read one line and write one line

#include <iostream>
#include <fstream>
#include <time.h>
#include <iomanip>

using namespace std;

int main(int argc, char *argv[])
{
    cout << "START!!!" << endl;
    clock_t start, end;
    double time, sum = 0;
    int count = 5;

    for (int i = 0; i < count; i++)
    {
        start = clock();

        ifstream ifs("/fixtures/sample.csv");
        ofstream ofs("./sample.csv");

        const int SIZE = 1024;
        char buffer[SIZE];

        while (ifs.getline(buffer, SIZE))
        {
            // ofs << buffer << endl;It's really slow
            ofs << buffer << "\n";
        }

        end = clock();
        time = (double)(end - start) / CLOCKS_PER_SEC;
        cout << "Time Result: ";
        cout << fixed << setprecision(4) << time << endl;
        sum += time;
    }

    cout << "C++ Average: ";
    cout << fixed << setprecision(4) << sum / count << endl;

    return 0;
}

As mentioned in the comment in the source, in C ++, writing with ʻofs << buffer << endl; `has a problem that the speed drops significantly, so I added the line feed code myself.

Impressions

As a result, as expected, C / C ++ was by far the best. Since Ruby and Python are scripting languages, I thought it would be slow, but it was still slow. (Maybe it's a matter of my writing) The Go language was so fast that I had never used it at work, but I was very interested in it!

Other

Actually I wanted to make Rust, but I implemented it halfway due to lack of knowledge and was frustrated, so I excluded it this time. We are always waiting for PR from Rust love people!

Also, if there is a way of writing that can be done faster than my implementation and the result will be different from this ranking, I am waiting for PR! (I would appreciate it if the basic concept remains the same)

The repository created this time

https://github.com/MasanoriIwakura/multi-lang-battle

Recommended Posts

Compete for file IO in various languages and compare speeds
HMAC in various languages
I implemented N-Queen in various languages and measured the speed
[AOJ] Descending sort in various languages
What are the "pipeline" and "{...}" in the Jenkins Pipeline pipeline {...} (for Groovy beginners, for experienced languages)
Write O_SYNC file in C and Python
I wrote unit tests for various languages