[PYTHON] [TensorFlow] I tried mass-producing "posthumous judgment" style messages with LSTM

Introduction

I tried sentence generation using LSTM in TensorFlow. Completely nth decoction. That's normal, so I'd like to study using the message of the Christ sign that is affixed to the hut in the mountains, "Meet the judgment after death", and make a sentence of ** "Non-existent Christ sign". .. ** **

** (Addition) I uploaded an article that tried Seq2Seq. Please also include this. ** ** [\ TensorFlow ] I felt that I was able to mass-produce "posthumous judgment" style messages using Seq2Seq --Qiita

Christ sign?

Like this.

There is an article covering the people who make it. Christ signboard, seeing the moment when it is pasted, closely related to the missionary activities of the Bdljapan Association

If you would like to see more of the signboards, please go to ↓ (promotion. I make it). Christ sign image bot (@christsignbot) / Twitter

Refuse in advance

** This article was written by an individual who has nothing to do with Bdljapan, and is not intended to express a position on any particular religion. ** Please see it as a "tried" report.

Verification environment

• Ubuntu 18.04
• Python 3.6.9
• TensorFlow 2.1.0 (CPU)

Based sample

Basically, it is a diversion of Keras sample. I am learning LSTM that when I enter a character string (a sequence of characters), the next character is predicted. When generating a sentence, give a few characters at the beginning, and then randomly generate the following sentences character by character. keras/lstm_text_generation.py at master · keras-team/keras

In the sample, it seems that Nietzsche's sentences are used for learning, but it is somewhat unstudied, and even if you look at the generated output, it is not clear whether it is a Nietzsche-like sentence. I hope that the Christ signboard will give you a better understanding of the generated results.

There was a commentary article on this sample. If you are interested, please. Explanation of Keras Single-LSTM character generation sample code --Qiita

Due to the nature of the data used this time, the following points have been changed from the original.

--Since one sentence of the Christ signboard is short, the number of characters used for prediction has been reduced (3 characters → predict the next 1 character) --The characters given as seeds when generating a sentence are always taken from the beginning of the original sentence (the original is taken from the middle of the sentence). --The length of the generated sentence is not fixed (the maximum number of characters is decided for the time being), and the generation ends when the end of the sentence is predicted.

Learning code

The learning data is a copy of the original text of the signboard.

lstm_text_generation_train.py

from tensorflow.keras.callbacks import LambdaCallback
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense
from tensorflow.keras.layers import LSTM
from tensorflow.keras.optimizers import RMSprop
from tensorflow.keras.callbacks import ModelCheckpoint
import numpy as np
import random
import sys

text = """
Oh, listen to the world, the world, the world, the word of God
The reward of evil is death
The lustful person does not recognize God
Those who sow evil reap disaster
Your god is alone
Get ready to meet your maker
Remember your maker
Jesus Christ is your Creator
Jesus Christ gives eternal hope
Jesus Christ is the only Son of God
Jesus Christ is the Son of God
Jesus Christ is the only God
Jesus Christ judges the world correctly
Those who call on Jesus Christ will be saved
eternal life
Source of eternal life
Eternal god
Source of eternal salvation
Hope for eternity
Stand before God in the last days
In the last days man stands before God
The day when God judges humanity is near
Reconcile with God
Repent of your sins against God
Seek the kingdom of God and justice of God
Seek the kingdom of God and justice
The kingdom of God is approaching
The kingdom of God is approaching repentance.
Those who refuse the word of God prefer death
God's judgment comes suddenly
The day of God's righteous judgment is near
Jesus Christ, the only Son of God, is the Messiah
God says it's not the rule to die here
God sees the heart
God punishes sin
God sent Christ the only Son to the world
God sent His Son Christ to the world
God is the only
God made the wisdom of the world foolish
God set the day to judge the world
Repent of your attitude toward God
Awe of god
Fear God
Those who move away from God enter the path of evil
Acknowledge God
Seek god
Think about the destination after death
Today is the day of salvation
There is no salvation other than Christ
The day when Christ will judge people is near.
Christ is the true god
Christ is the way, the truth, the life
The Second Coming of Christ is near.
Purify the blood sin of Christ
The blood of Christ cleanses sin
The blood of Christ purifies sin
The blood of Christ purifies sin.
The blood of Christ removes sin
There is no god but Christ
The resurrection of Christ is a confirmation of salvation
Christ gives you eternal life
Christ justifies you
Christ frees you from sin
Christ gives eternal life
Christ is the Son of God
Christ sinned on the cross
Christ is the true alter ego of God
Christ will come soon
Christ will come soon
Christ cancels sin and gives life
Christ revokes sin
Christ forgives sin and gives eternal life
Christ rose from the tomb
Christ has risen from the tomb
Christ sinned on behalf of man
Christ will come again and judge the world
Christ will come again and judge the world
Christ comes again
Christ will come again
Christ was resurrected
Christ is the true god
Christ is the true alter ego of God
Christ sinned in his place
Christ is the way, the truth, the life
Christ resurrected and gives eternal life
Christ revived and overcame death
Those who believe in Christ will be saved
Those who believe in Christ have eternal life
Those who call on Christ will be saved
Repent
Repent
Idolatry is a sin
Fortunately for those with a pure heart
Believe in God from the bottom of my heart
God judges the sin of the heart
Escape the indelible fire of hell
Hell is eternal suffering
Hell is the second death
There is a judgment after death
Meet the judgment after death
Meet the judgment after death.
Think about your destination after death
Think about your destination after death
God sees private life
There is a way of death and a way of life
Death is the reward of sin
The Second Coming of the Lord Jesus Christ is near
The resurrection of the Lord Jesus Christ is a confirmation of salvation
The Lord Jesus Christ is the Creator of all things
Lord's day comes suddenly
Life is terrible, heaven is long
No one is right
Just believe
Believe me
Corrupted society does not recognize God
The earth and the person belong to God
Don't worship "made" things
Fortunately those who have been cleansed of their sins
Fortunately those who have been cleansed from sin
Enlightenment about sin, justice, and judgment
If you die with sin, you will go to eternal hell
If you die of sin, you will go to eternal hell
The reward of sin is death
The reward of sin is death, the gift of God is eternal life
The reward of sin is death, the gift of God is eternal life in Christ
Get forgiveness of sin
Get forgiveness of sins
Ask for forgiveness of sins
God punishes sin
Fortunately those who have been cleansed
Repent of your sins
Repent of your sins
Heaven or hell or your destination
Heaven is eternal life Hell is the sea of fire
Even if the heavens and the earth perish, my words will not come apart
Creator of all things
Heaven or hell or all people will revive
The kingdom of heaven repents of near sins
Watch out for fake
In the beginning God created the heavens and the earth
Man-made things are not gods
Get rid of human evil
Sin dwells in people
Get rid of people's sins
God sees people's ways and deeds
The pond of fire is the second death
Redeem iniquity
God judges immorality and adultery
God judges adultery and adultery
There is a way of death and a way of life
Don't recognize God in the crooked times
True God loves people and removes their sins
Believe in the true god
Believe in the true God.
Way, truth, life
Behold, I will come soon
The world and the greed of the world are dying
The end of the world is near
The end of the world comes suddenly
The resurrected Christ gives eternal life
Judge the world correctly
A person who says that evil is good
Those who worship idols
The hero of drinking alcohol
Those who bother to wear
I am the way, the truth, the life
Eternal life in my words
I am life
I'm the bread of life
I'm the bread of life
I have the keys to death and hell
I am the truth
I am the truth
I am the light of the world
Those who believe in me have eternal life
Those who believe in me have eternal life
Those who believe in me have eternal life
Those who believe in me live even if they die
I am the truth and the life
I remember your Creator.
There is no salvation other than Jesus Christ.
Call on God before it's too late.
In the last days man stands before God.
Christ will judge the hidden things.
Christ sent by God is the savior
If you return to God, God will forgive you abundantly.
Repent of your sins against God.
The kingdom of God is near. Repent.
The day of God's judgment is near.
The gift of God is eternal life
God loves people and removes their sins.
Fear God and obey the word.
Recognize God, be afraid, and leave sin.
Christ has the word of eternal life.
Accept the salvation of Christ.
The blood of Christ purifies sin.
Christ will come again and judge the world.
Christ was guilty of man on the cross.
Christ saves sinners.
Christ gives man eternal life.
Christ was punished on behalf of man.
Christ frees people from sin.
Believe in Christ and be saved.
Those who call on Christ will be saved.
Repent and believe in the gospel.
Repent.
Those who repent are redeemed by Christ.
Be renewed from the bottom of your heart.
Think about your destination after death
Fortunately those who have been cleansed from their sins.
The reward of sin is death
Please forgive me for my sins.
Get rid of sin.
Acknowledge your sins and return to God.
Christ was resurrected because man was justified.
God sees people's hearts and thoughts.
People meet afterlife judgment.
The world and the desires of the world are gone.
There is no god other than me.
I am the way, the truth, the life.
Those who believe in me have eternal life.
"""
lines = [s + "\n" for s in text.strip().split("\n")]
print('corpus length:', sum(len(s) for s in lines))

chars = list(sorted(set("".join(lines))))
print('total chars:', len(chars))
char_indices = dict((c, i) for i, c in enumerate(chars))
indices_char = dict((i, c) for i, c in enumerate(chars))
print(chars)

# cut the text in semi-redundant sequences of maxlen characters
maxlen = 3
step = 1
sentences = []
next_chars = []
for s in lines:
  for i in range(0, len(s) - maxlen, step):
      sentences.append(s[i: i + maxlen])
      next_chars.append(s[i + maxlen])
print('nb sequences:', len(sentences))

print('Vectorization...')
x = np.zeros((len(sentences), maxlen, len(chars)), dtype=np.bool)
y = np.zeros((len(sentences), len(chars)), dtype=np.bool)
for i, sentence in enumerate(sentences):
    for t, char in enumerate(sentence):
        x[i, t, char_indices[char]] = 1
    y[i, char_indices[next_chars[i]]] = 1

# build the model: a single LSTM
print('Build model...')
model = Sequential()
model.add(LSTM(128, input_shape=(maxlen, len(chars))))
model.add(Dense(len(chars), activation='softmax'))
model.summary()

optimizer = RMSprop(learning_rate=0.01)
model.compile(loss='categorical_crossentropy', optimizer=optimizer)


def sample(preds, temperature=1.0):
    # helper function to sample an index from a probability array
    preds = np.asarray(preds).astype('float64')
    preds = np.log(preds) / temperature
    exp_preds = np.exp(preds)
    preds = exp_preds / np.sum(exp_preds)
    probas = np.random.multinomial(1, preds, 1)
    return np.argmax(probas)


def on_epoch_end(epoch, _):
    # Function invoked at end of each epoch. Prints generated text.
    print()
    print('----- Generating text after Epoch: %d' % epoch)
    line_index = random.randint(0, len(lines) - 1)
    start_index = 0
    for diversity in [0.2, 0.5, 1.0, 1.2]:
        print('----- diversity:', diversity)

        generated = ''
        sentence = lines[line_index][start_index: start_index + maxlen]
        generated += sentence
        print('----- Generating with seed: "' + sentence + '"')
        sys.stdout.write(generated)

        for i in range(100):
            x_pred = np.zeros((1, maxlen, len(chars)))
            for t, char in enumerate(sentence):
                x_pred[0, t, char_indices[char]] = 1.

            preds = model.predict(x_pred, verbose=0)[0]
            next_index = sample(preds, diversity)
            next_char = indices_char[next_index]

            if next_char == "\n":
              sys.stdout.flush()
              break
            sentence = sentence[1:] + next_char

            sys.stdout.write(next_char)
            sys.stdout.flush()
        print()

print_callback = LambdaCallback(on_epoch_end=on_epoch_end)

cp_cb = ModelCheckpoint(
    filepath="model.{epoch:02d}.hdf5",
    verbose=1,
    mode="auto")
model.fit(x, y,
          batch_size=32,
          epochs=60,
          callbacks=[print_callback, cp_cb])

Execution result

The first is tattered.

----- Generating text after Epoch: 0
----- diversity: 0.2
----- Generating with seed: "God's Hi"
God
----- diversity: 0.5
----- Generating with seed: "God's Hi"
God's day is God's good.
----- diversity: 1.0
----- Generating with seed: "God's Hi"
God's closeness
----- diversity: 1.2
----- Generating with seed: "God's Hi"
A stalking statue of the death and life-giving stalking of the gods

It gradually becomes more like a sentence. I feel like the original text is as it is.

----- Generating text after Epoch: 11
----- diversity: 0.2
----- Generating with seed: "Kilis"
Christ frees people from sin.
----- diversity: 0.5
----- Generating with seed: "Kilis"
Christ has eternal life for those who believe in true God
----- diversity: 1.0
----- Generating with seed: "Kilis"
Christ will come soon
----- diversity: 1.2
----- Generating with seed: "Kilis"
The blood of Christ removes sin

↓ It feels good. Actually, none of them are sentences that do not exist in the learning data.

----- Generating text after Epoch: 36
----- diversity: 0.2
----- Generating with seed: "Trust me"
Those who believe in me will be saved.
----- diversity: 0.5
----- Generating with seed: "Trust me"
Those who believe in me are eternal life
----- diversity: 1.0
----- Generating with seed: "Trust me"
Those who believe in me will be saved
----- diversity: 1.2
----- Generating with seed: "Trust me"
Save those who believe in me.

The end is like this. Since the characters are taken randomly according to the probability, the sentences may occasionally collapse.

Epoch 60/60
----- Generating text after Epoch: 59
----- diversity: 0.2
----- Generating with seed: "Is it heaven?"
Heaven or hell or your maker
----- diversity: 0.5
----- Generating with seed: "Is it heaven?"
Heaven or hell or your destination
----- diversity: 1.0
----- Generating with seed: "Is it heaven?"
Heaven or hell or you can't be cleansed from sin
----- diversity: 1.2
----- Generating with seed: "Is it heaven?"
Heaven or hell or everyone revives

Mass production of non-existent sentences

I used to save the model for each epoch, so I used this. If you give 3 letters, it will make 10 patterns of the following sentences.

lstm_text_generation_exec.py

from tensorflow.keras.models import load_model
import numpy as np
import sys

# generated in train phase
chars = list('\n ,. "" Aiue Oka Kigiku Guke Kosazaji Suzu Suzu Suzu Suzu Suzu Suzu Suzu Suzu Suzu Suzu Suzu Suzu Suzu Suzu Suzu Suzu Suzu Suzu Suzu Suzu Suzu Suzu Suzu Suzu Shimofuyosei Chief Two Dead People Imadaikai Shinroku Evening Statue Sinkou Iriuchi Re-division First Mae Sokatsu Toshiki Takeshi Called Japanese Goods Yuizen Kokuchi Koho Takaki's return, obedience, revenge, regret, regret, foolishness, refusal to worship, relieve, relieve, rebirth, new date and time Prison Risei Reborn Shinchi Kakusha Shinfuku Private Standing Final Sin Punishment Considerer Hearing Suffering Britain Fallen Leaves Bloodline Judgment Opinion Testimony Proof Deafening top drink')
char_indices = dict((c, i) for i, c in enumerate(chars))
indices_char = dict((i, c) for i, c in enumerate(chars))
maxlen = 3

# load the model
model = load_model("model.60.hdf5")

def sample(preds, temperature=1.0):
    # helper function to sample an index from a probability array
    preds = np.asarray(preds).astype('float64')
    preds = np.log(preds) / temperature
    exp_preds = np.exp(preds)
    preds = exp_preds / np.sum(exp_preds)
    probas = np.random.multinomial(1, preds, 1)
    return np.argmax(probas)

def generate(model, first_chars):
    diversity = 1.0
    for trial in range(10):
        generated = first_chars
        sentence = first_chars[-maxlen:]
        sys.stdout.write(first_chars)

        for i in range(100):
            x_pred = np.zeros((1, maxlen, len(chars)))
            for t, char in enumerate(sentence):
                x_pred[0, t, char_indices[char]] = 1.

            preds = model.predict(x_pred, verbose=0)[0]
            next_index = sample(preds, diversity)
            next_char = indices_char[next_index]

            if next_char == "\n":
                sys.stdout.flush()
                break
            sentence = sentence[1:] + next_char
            sys.stdout.write(next_char)
            sys.stdout.flush()

        print()

for l in iter(sys.stdin.readline, ""):
    l = l.rstrip()
    if l == "":
        continue
    if len(l) < 3:
        print("ERROR: input must be >= 3 characters", file=sys.stderr)
        continue
    if not all(c in chars for c in l):
        print("ERROR: invalid charater", file=sys.stderr)
        continue
    generate(model, l)

Execution example.

>>>Jesus
Jesus Christ justifies you
Jesus Christ is your righteousness
The blood of Jesus Christ purifies sin.
Jesus Christ the source of eternal life
Jesus Christ loves people and their source
Believe in Jesus Christ
Jesus Christ is your Creator
Jesus Christ gives you eternal life
The blood of Jesus Christ purifies sin.
Jesus Christ is the destination
>>>Kilis
Christ resurrected and gives eternal life
Christ sinned in his place.
Those who believe in Christ have eternal life
Purify the blood sin of Christ
Christ is the way, the truth, the life
Christ is the savior
Have eternal life in Christ.
Christ cleanses sin
There is no salvation other than Christ
Christ was sent to the world
>>>you
Your maker
Free you from sin.
Remember your Creator.
You have been cleansed from sin.
Justify you
Remember your Creator.
Justify you
Remember your Creator.
Your maker
Remember your Creator.
>>>True god
True God's alter ego
There is no true god
The true God loved man and was guilty of it.
True God's alter ego
True God's alter ego
True god
Those who believe in the true God give eternal life
The true god.
Those who believe in the true God live even if they die
True god

"Those who believe in the true God live even if they die", although there is no same sentence in the learning data, it usually makes sense and I personally like it. (Lol)

Task

--Only the characters that appear in the learning data can be output. ――In this program, for example, ** "Reconcile with cats" will never appear ** ――Is it good to create a word-by-word model and use word2vec to use words with similar meanings? ――Isn't it possible to utilize these learned parameters? → Kyubyong / wordvectors: Pre-trained word vectors of 30+ languages --There are quite a few sentences that are exactly the same as the learning data. ――Sentences that are grammatically strange or have strange meanings appear ――For example, the correspondence between the subject and the predicate is strange, saying that "those who believe in the true God give eternal life." --SeqGAN seems to solve this problem → Generate text (novel synopsis) using SeqGAN --Qiita --First, let's study the basic Seq2Seq. .. .. -** (Addition: I tried) ** [\ [TensorFlow ] I felt that I was able to mass-produce "posthumous judgment" style messages using Seq2Seq --Qiita](https://qiita.com / everylittle / items / d4efdafd7b73db965945) --Learning data itself is very small ――There is nothing you can do about this ...

Summary

Resurrection of the Christ Sign Generator (v2): This is also made by me (also advertised).