Be careful if you find SHIFT-JIS in Java

Background

There was a problem that "~" was garbled in the WEB application created by Java. Since the DB character code was UTF-8 and the CSV output was MS932, it was a simpler story when I was investigating whether it was garbled due to conversion of different character codes. I wasted a lot of time, so I'll leave a note as well.

Cause

Even if the DB is UTF-8, there is no problem here because it is UTF16, which is an internal representation in Java. When I was following the process, after acquiring the data from the DB, it was converted to SHIFT-JIS by Java processing and then converted to MS932. I thought it was a conversion problem of SHIFT-JIS, MS932 that appears in "~", but it was a conversion problem of SHIFT-JIS, UTF16 (internal representation of Java).

Character conversion example

I created a simple source code and verified it. (java: 1.8.0_121) When a Java character string is generated after converting to a byte array of SHIFT-JIS and MS932, only SHIFT-JIS is garbled.

        String org = "～";

        byte[] sjBytes = org.getBytes("SHIFT-JIS");
        byte[] ms932Bytes = org.getBytes("MS932");

        String sj = new String(sjBytes, "SHIFT-JIS");
        String ms932 = new String(ms932Bytes, "MS932");

        String fmt = "%s\t string:%s,Byte array:%s";
        System.out.println(String.format(fmt, "Original character", org, DatatypeConverter.printHexBinary(org.getBytes())));
        System.out.println(String.format(fmt, "SHIFT-JIS", sj, DatatypeConverter.printHexBinary(sjBytes)));
        System.out.println(String.format(fmt, "MS932", ms932, DatatypeConverter.printHexBinary(ms932Bytes)));

Output result

Original string: ~, byte array: EFBD9E SHIFT-JIS string:?, Byte array: 3F MS932 String: ~, Byte array: 8160

Summary

There is no problem if you use UTF-8 in the first place, but it is difficult to handle because the specifications will change. If you really want to use SHIFT-JIS, MS932 is enough, so don't use SHIFT-JIS.

Table of Contents