[PYTHON] When I try to divide with Bert Japanese Tokenizer of Hugging Face, it fails with initializing of MeCab or even with encode.

I'll write it down as a note so that people who stumble on the same error will have less time to look up.

The following is running on Google Colab.

Install MeCab and huggingface transformers on Colab by referring to here.

!apt install aptitude swig
!aptitude install mecab libmecab-dev mecab-ipadic-utf8 git make curl xz-utils file -y
!pip install mecab-python3
!pip install transformers

Try to divide with the tokenizer of BERT for Japanese.

from transformers.tokenization_bert_japanese import BertJapaneseTokenizer

#Declared tokenizer for Japanese BERT
tokenizer = BertJapaneseTokenizer.from_pretrained('cl-tohoku/bert-base-japanese-whole-word-masking')

text = "Natural language processing is a lot of fun."

wakati_ids = tokenizer.encode(text, return_tensors='pt')
print(tokenizer.convert_ids_to_tokens(wakati_ids[0].tolist()))
print(wakati_ids)

I got the following error.

----------------------------------------------------------

Failed initializing MeCab. Please see the README for possible solutions:

    https://github.com/SamuraiT/mecab-python3#common-issues

If you are still having trouble, please file an issue here, and include the
ERROR DETAILS below:

    https://github.com/SamuraiT/mecab-python3/issues

You don't have to write the issue in English.

------------------- ERROR DETAILS ------------------------
arguments: 
error message: [ifs] no such file or directory: /usr/local/etc/mecabrc
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-3-f828f6470517> in <module>()
      2 
      3 #Declared tokenizer for Japanese BERT
----> 4 tokenizer = BertJapaneseTokenizer.from_pretrained('cl-tohoku/bert-base-japanese-whole-word-masking')
      5 
      6 text = "Natural language processing is a lot of fun."

4 frames
/usr/local/lib/python3.6/dist-packages/MeCab/__init__.py in __init__(self, rawargs)
    122 
    123         try:
--> 124             super(Tagger, self).__init__(args)
    125         except RuntimeError:
    126             error_info(rawargs)

RuntimeError: 

Kindly ask me to look at here in the error output, so install mecab-python3 according to the instructions at the URL. When you do

pip install unidic-lite

If you also execute, MeCab's initializing will no longer fail. But this time, I got angry with ʻencode saying Value Error: too many values to unpack (expected 2) `.

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-5-f828f6470517> in <module>()
      6 text = "Natural language processing is a lot of fun."
      7 
----> 8 wakati_ids = tokenizer.encode(text, return_tensors='pt')
      9 print(tokenizer.convert_ids_to_tokens(wakati_ids[0].tolist()))
     10 print(wakati_ids)

8 frames
/usr/local/lib/python3.6/dist-packages/transformers/tokenization_bert_japanese.py in tokenize(self, text, never_split, **kwargs)
    205                 break
    206 
--> 207             token, _ = line.split("\t")
    208             token_start = text.index(token, cursor)
    209             token_end = token_start + len(token)

ValueError: too many values to unpack (expected 2)

About this error here, the developer of mecab-python3? As mentioned by, it was solved by specifying the version of mecab-python3 as 0.996.5.

In summary, when installing pip, I think that it will not be an error if you declare it as follows.

!apt install aptitude swig
!aptitude install mecab libmecab-dev mecab-ipadic-utf8 git make curl xz-utils file -y
!pip install mecab-python3==0.996.5
!pip install unidic-lite
!pip install transformers

If you have already installed the latest version of mecab-python3 with pip before running ↑, don't forget to reconnect your colab session once. You can disconnect the session from the session management by clicking ▼ on the top right of the colab screen, such as RAM or disk.

from transformers.tokenization_bert_japanese import BertJapaneseTokenizer

#Declared tokenizer for Japanese BERT
tokenizer = BertJapaneseTokenizer.from_pretrained('cl-tohoku/bert-base-japanese-whole-word-masking')

text = "Natural language processing is a lot of fun."

wakati_ids = tokenizer.encode(text, return_tensors='pt')
print(tokenizer.convert_ids_to_tokens(wakati_ids[0].tolist()))
print(wakati_ids)

#Downloading: 100%
#258k/258k [00:00<00:00, 1.58MB/s]
#
#['[CLS]', 'Nature', 'language', 'processing', 'Is', 'very much', 'pleasant', '。', '[SEP]']
#tensor([[    2,  1757,  1882,  2762,     9,  8567, 19835,     8,     3]])

I was able to successfully divide it with BertJapaneseTokenizer.

end

Recommended Posts

When I try to divide with Bert Japanese Tokenizer of Hugging Face, it fails with initializing of MeCab or even with encode.
When I try to divide a list with MeCab, I get'TypeError: in method'Tagger_parse', argument 2 of type'char const *''
When I try to push with heroku, it doesn't work
I wrote a class that makes it easier to divide by specifying part of speech when using Mecab in python
When I try to make Apache SSL, it doesn't start.