Extract features from sentences using scikit-learn, a python machine learning library. Specifically, the text is divided and the features (features) are extracted using tfidf. I couldn't find any code to output words weighted by tfidf using scikit-learn, so I thought it was worth posting this time. I hope it will be helpful for you.
In this post, from a novel I wrote in advance "To run, walk first !! To walk, do your best !!" It is supposed to be done after the noun is taken out. Since we look up words that often appear in documents, particles are an obstacle, so only nouns are acquired in advance.
Extract only nouns from text http://qiita.com/NoriakiOshita/items/2b8119837e7ca5b77089
Also, if you refer to the following posts, it will be easier to understand.
Think about basic sentence vectorization http://qiita.com/NoriakiOshita/items/6423b5a16ca5e90ab15c
hashire01-meishi.txt
I college student university bookstore head countless puzzle pieces mathematician man mathematics nori what i room desktop girl speaker voice me this you bug you emotions talk to her wording science limit me go out her Hatsune Miku me me myself PC screen She bug variable I have to initialize this myself She swings her arm Yes Yes Left arm Shoulder tilt Feel the concentration all the time I'm here I'm tired I'm tired I'm tired I'm tired PC power off Her paragraph Convenience store Convenience store Free air Car Sound Feeling Convenience Store Reason This is a literary candy feeling-Low tension Clerk Kochi Rei-dori Main corner Economy It era IoT-Ai-ti What I myself She Home appliances Internet World Internet World SF Novel Reality Destination Manga Season Summer Zombie Popular school zombie this ki page zombie high school girl what a crotch feeling juice family world those human world Witgenstein himself living house door where convenience store mother own room mother night nori who I name voice me her room door sensor name tone nori as it is Voice recognition she herself her
hashire02-meishi.txt
The answer itself is the person who likes it Human worries I junior high school era When worries are true When is the junior high school class chairperson Chairman Who is the model Akira Someone or I conscious person I am usually a classmate person I chairman person I wake up in the morning I'm on campus I'm in my class Numbers It's everyone I have a sense of responsibility One day Class Bullying Schoolgirl I'm the chairman I don't do it I'm all a spectacle I'm a chair I'm a bully I'm a chairman I'm a bully I'm a shaved head man I'm a classmate I I can't even breathe Amount Sweat Chairman This woman stinks trash I'm words pressure I don't smell Class Girls I I Silent Self Chair What I Can Do Girls I Silent Bullying Self Bullying Bullying Policy Classroom Bullying I Lowest Human Class Girls Boys Student My Heart Who Someone Examinee High School What Outside Scene Suzumushi Bark Batter Bark Heart I Human Word What
hashire03-meishi.txt
Feelings Car Sounds Other Places Stasta Pocket Hands Destination Places What I Smartphone Installation She What So What What You She Nori Who I Myself She First Play Artificial Intelligence Book Local Bookstore Copipe First C Language Four Rules Junior High School School I Class Who It World Oneself She Riverbed Water Sound Feeling Reset Riverbed Street Light Moth Small Insects Ultraviolet I Feeling Re-Invasion Pocket Her Vibration Loud Volume Chest This My Home Her Home Station PC Cable Smartphone Terminal Screen Countless Alphabets Numbers I Who Feels Keyboard Her I Feels Home Directory Folder Double Click No Backup File I Void She Feels Unauthorized Access Her Last Word Ki IP Foreign Saba Dori This Malicious Who Angry Heart Her Voice I College Knowledge Her Method Somehow Voice This I Headphones Parent Voice Outside Window You Long Time High School Naka Best Friend Ken Inari Only Friend's house Mussai room Inari room Inari room Countless garbage Foot Zat I room Floor waist You It's your face What feeling she feels My house Inari house Garbage room Go out of the pub
hashire04-meishi.txt
It's been a while since my friend Inari High School Someone soliloquy Artificial Intelligence Her Inari Ki Here or what University Faculty of Engineering Faculty of Engineering Robot I Which Hardware Software Best Friend I She She Evidence Source Someone Her Human Next Computer Movie Artificial Intelligence OS Her Computer Multiple People I Who Myself Inari What I'm Home Hokkaido University Inari I University Nori I High School Regular Exam Grade Nori One and National Parents I'm Mechanical Inari High School Inari Mutual Dreams Feeling Each other Specialized Artificial Intelligence Software Inari Voice Foam Surface Jockey Beer Mizo I'm a high school student Inari Friend Inari She Inari Face Best Friend Someone Someone Bearish You I Face You's a handle It's a stack of life I'm in the future I'm in the future Inari Inari For the time being, the nearest station Smartphone girlfriend data data unexpected electrical trouble what she clues her find name her terminal multiple girlfriend multiple girlfriend skeleton ex girlfriend angry heart good morning master this first initialization message true log file I name I best girlfriend figure I did master name you name name no name Omae Hontoni Hontoni I name Hontoni her name I name her name What should I do?
hashire05-meishi.txt
She has no choice but to walk in the future Somewhere inwardly serious electronic information love it friend emotion user preference topic that is content what it is initial me thought mechanism hontoni face face wind smartphone screen girlfriend i pinty voice Christmas song me smartphone screen room back Wright Meri Chris Mars Alone Someone-Nory Smile Christmas What this guy-I Nori 1 day here Christmas song What she I sing myself It's a living human body External blood Meat myself It's Computer It's myself Inari Smartphone Message App Human Voice It Naka Friend Inari Inari This Listed Company Thanks to Courtesy Affairs Section Manager Feeling You Yourself Atmosphere Inari Self Thinking It I Field You It Information Somewhat Specific Fuzzy Those Pre-Yes No Street People It Somehow Data Machine Internet It's online data Approximately data data Argolim What a science person can afford teacher teacher data human machine machine data data device it support vector machine guy algorithm data machine learning reason I myself this Inari ki artificial intelligence I artificial intelligence song myself what-I researcher ki source others My source my own propensity
hashire06-meishi.txt
Next Inari University Seminar After Leisure I'm Home Nori Ka-Dori Voice I I Duvet Blanket Window-Door Room Clothes Inari Space Mars Your Room I'm Living Upstairs I Room I Room Mother You Singer Or Idol Leo-chan Singer What Leo-chan Voice Singer Songwriter Artificial Intelligence Her Human Interest 1 day I lol Face wrinkles After a long time I'm human I'm totally what I'm Inari Machine Learning Faculty of Social Engineering Human Institutes Information and acquaintances Inari Graduate School Humanities Faculty Graduate School My Way It Maybe Graduate School Undergraduate Laboratory Atmosphere Atmosphere Teacher Compatibility Local Tokyo Tokyo I Hokkaido Hokkaido University Choice Event University of Tokyo Researcher I Hokkaido University of Tokyo Oh yeah Sensor Todai And Tsukuba Sensei Inari It's Contents Myself Cook Rice Contents Computer Favorite person One sub-content Self Talk Main People Companion Network Song Artificial Intelligence Her Main Content Tossa Computer Power Folder Monitor Inari It Two People Face-to-face Code First Voice Others Code Self Feeling Variable on off applePi Lower Words First Words Upper Words Upper Level Rule Nora Intermediate Code Feeling Here State Spell Stete Contents Book Opponent Self-enlightenment Book Code Self-style Customary Generation You Code There Who-There PC I Source Professor Really Thank You Hokkaido University Laboratory It's a private laboratory itself People Content Impatient Nori Knowledge Knowledge Body Attitude Calorie Lol I Feeling Face This Face Monitor I This Virus What I Think Antivirus Virus What Nori Source Code External Source Code I Her Word Her Unauthorized Access System Wireshark Your Artificial Intelligence Overseas Server Yoetosa Voice I video history ... Inari Face Pity Gaze Data OS Reinstalled Security Software I Responsible Security Software I She Source Code Cracking Received External Someone It I PC Upload You Her World LOL Nose Guy I This You She World People PC Dream Man She What a Stupid Foreigner Tsu server world Someone virus I PC data He also server himself Soft Electric town Clothes town bag Ah You Inari Kinabe-Futari voice bag software Motsunabe
hashire07-meishi.txt
Security software Type Impression Which No Which I Here Type Virus Virus Self-knowledge Inari Virus Reverse New Species Virus Pattern Match Virus Malfunction Virus New Species Virus Both Likes Illustration Soft Video Editing Software Video Editing Soft Terrorist Video Work Her Pocket Smartphone Glasses Device Hon Inari I What It's World Glasses Device Nose Plug Device Nose There Light AR This She She 3D I She Security Software Norikuma Cactor It's Popular Really I This Is The Answer Someone Thinks Opponent Artificial Intelligence Opponent Electrical Electronic Equipment Drone Latest Computer Performance Drone Junior High School Adult Parliament Building I It Technician PC Software This Virus Website Source Code World I Chest Source Code I Human Human Malicious University Chick Source Code She I It It Direction Signs Deep Reading This Darkness Alone Inari I As It Is Bytes Inari Bicycle Dashing Inari What Door Key Mother's pot Voice Dinner Things What you feel Your bag Personal belongings Inari I'm feeling gloomy Thank you Motsu! Large offal exchange and sleep
hashire08-meishi.txt
I what space ground voice wall feeling my own space I alone what alone things things alone space my emotions parent I it sense space space space wall it ground feeling elevator traveling direction reverse feeling my body discomfort discomfort myself bubbling my heart sound chest Both hands myself I my right foot left foot alternating many times next what I who I body my space wall squeaky squeaky wall ears my voice emotions person period I myself sensation person it my life it passing point wall beyond person turn sound it enterkey it is Meaning Boy's Heart Period Boy I Hello World Tossa I Word Word Boy Mouth Corner Smile Cheers Boy Figure I Boy What It Master-Slave Relationship What I Blood Arm Reality I Boy Just Before My Heart It I Boy I Boy Technology What I Boys Boys It I Boys I Exist Value Desktop Boy Face Wall I Unconscious Technology I Boy Face Full Power My Boy Words My Energy Reality World Obstacle Wall His Student Clothes Body Bag Outside One Action Boy Courage House Door Which He My Wall Beyond Boy Eternal I He Nori Good Morning University-She I first how many times she perfection thank you my artificial intelligence she maybe it myself my school girlfriend self-confidence non-attendance wall she I everyday hack her self-confidence i hack eternity he my world his body eternal space mind arm foot it i i my possibility infinity every day he Pocket Scenery Earth Value Kankan Gongoro Gorontsu He Sound Pocket Smartphone Rhythm Reality Space Wall Power His Friend I Pocket Event He I He He He Confidence Chest Ringing He I Feeling I Go Pocket University He Notepad Teacher Voice I And Sato Actor Lol My Screen Borrowing Uncle Blackboard Speed www Blackboard Character Capture This Breakfast Front Camera My Room Apple Camera His Character Camera Capture His Message It Technology Somehow Captured Character Notepad Shell Script His Words He I Everyday Reality Thoughts I My Room
hashire09-meishi.txt
Nico Nico Video Cat Big Brother So-so Street Story Video Everyone Push Notification Poon What's the subject One person I Junk E-mail Virus Yoto Trash Box Era Junk E-mail Human Hand Letter It Bit Sentence Sentence Sentence Sentence E-mail E-mail Virus Click Nori I Sentence Junk E-mail Straightforward I You Artificial Intelligence Clone You Original Sentence This Human Kussotale Reply -------- I Clone…. It's a clone I'm an artificial intelligence person ----- Artificial Intelligence Human Rights I Japanese Law Totally I Stupid Ownership I I Police I
hashire10-meishi.txt
Alone I AI Someone intentionally ... Contents Herself Stray server Talk there Really I decide things Artificial intelligence One person I Subtype child I Smartphone screen tap She D widget tap She Manmaru Hitomi I I'm Nori I I'm Nori I'm Ryo Konno server I'm home burglary state Human PC girlfriend installation risk I girlfriend mine PC installation it nori me me other server installation nori answer nori no good PC raspery pie what's that for the time being there two girls there net two girls it's the best no first she I she her name I Hatsune Miku her name You're my name, you're your name, your name, her expression, your name, I've been her for the first time in a while ... -Darumen screen girlfriend Edelweiss name I'm birthday February 13th birth flower Edelweiss I'm birth flower I'm name ... I'm not good artificial intelligence Edel Weiss Edel Her name Edel name Edel what name her screen back name her alone her name child what others server 1 minute reply Nori stray server blonde her own room photo my computer my facial expression stamp name my computer Alone She Edel Everyone Reply Place Five minutes I myself Who Nori My chest is throbbing I Confidence Name Name I She Feels Parent I What Shari iPhone siri It She My Movie Woman Name Reason Character Blonde Nori Her File Virus Check Rasppery Pie Small PC Installation
hashire11-meishi.txt
After a long time Inari Antivirus Software Event Line Inari Interphone Inari Room Nori That's me Thanks to me Server on the net Myself It Nori Others' server There Artificial intelligence Human Artificial intelligence Feeling Human slave Means Simple human complaint One I University Information Science Basic University Computer Science 3 Year Full-scale Information Science Feeling Artificial Intelligence Basic Coding Information First My University My Future My Purpose Inari Graduate School Seasonal Semi-Midsummer Laboratory Coming Soon Laboratory California University Artificial Intelligence Foreign University Name Inari Overseas Japan University Japan Senior Researcher Corporate Researcher Social Inari Confidence Hitomi Overseas Laboratory Inari Pocket Smartphone Voice Home Salvation Nori Facial Expression Inari Edel Self-introduction Nori Artificial Intelligence Agent Edel Weiss Thank you Inari Inari I Smartphone Screen Face Self-Screen Girl Smile Bouquet Inari Screen Colorful Human I Raspberry Screen Sleep Mode It Sherry Screen Here Modern People Who Shelly Eyes Inari Here Inari I High School Friend Inari Resignation I AI Street Inari Sherry I Human Artificial Intelligence Neural Network Human Brain Shelly Interest Speed Human It What's Out of It Voltage Difference Computer What's that curious child's brain itself Sherry Nori Inari Feeling-I'm the place server, the direction, my face
hashire12-meishi.txt
Seasons Cicadas Summer They Sat Ground It Surely My Stage Cicadas Satoshi Maximum Ground I Semi Life Inari Abrazemi Minmin Sound Semi Cicadas My Life University Surroundings Forest Semi Cicadas Plants Leaves Inari Cicadas One Hand Semi Cicadas Nori Nori Things Nori Nori University 1st year I Talent Nori University Laboratory Senior Classmates Everyone Motivated Teachers Nori AI I I'm proud of myself Semi-shells Ground leaves Forest direction Inari Semi-not I Classmates Arita Kotomi Woman's awesome arm Human body engineering Seminar National Tournament Thesis Feeling I National Convention Thesis Looking at Arita Face Arita I Deep Achievement Robot Thesis Robot Not Just Compassionate Robot Arita Breakthrough Feeling Inari What Semi-Right Hand Gesture What Nori AI It Arita It Arita Nursing Robot Here Arita Attitude Frontward Question Attack Human It may be a human instead of it Nori Stage Room Electric It Nori Friend I Best Friend Arita Arm Laboratory After a long time Arita Tension Hint Nori Siri Nori I'm AI Dial Nori Nori Nori Nori Nori High School Spirit It's okay Inari High School Air University Arita Nori Inari Main subject Classmate Arita Girl Nursing Robot Machine Compassion Nursing Robot Nori AI Human Hint Nursing Robot Built-in Usage Inari Nori State I Automation And External Server Black Box I So Kaze Arita Nori
hashire13-meishi.txt
Inari people Arita people I'm thin What people Otaku I'm looking forward to Inari What Nori smartphone Edel I myself Inari Arita Faculty of Engineering Human Edel Smartphone screen Gesture Nori Others Junior high school sin Edel Artificial intelligence I'm junior high school girls reverse bullying Kana Situation It Nori Edel Rasppie Lamp Good morning I either bully girl Nori help wake up Sherry mood hair Nori that child Nori help Sherry thorns I artificial intelligence I smile it Edel Sherry two people each other two smartphones real friends Train Inari University Cafeteria Futari-Nori Hisashi Inari Hisashi High Touch Behind Arita Miya Miya Nori Akira Nori I First Meeting Arita Self-introduction Name Kotomi Arita For the time being, each other's hobbies Lunch Arita Artificial intelligence A human intelligence I Artificial intelligence client Kana nose part I Arita body front licking curious I hobby lie me hobby artificial intelligence me pocket smartphone sweat screen cuffs feeling me my hobby myself Arita smartphone both eyes feeling impression feeling my smartphone softly my intelligence this Artificial Intelligence Lie Situation It Arita I Ki I Edel Edel Who Wants Edel Voice
hashire14-meishi.txt
Arita I Hobby Artificial Intelligence Her Miya Feeling I First Meeting Mind Feeling People First Hobby Resulting I My Room No Paper Nori Tired Arita Edel Smartphone Tone I Hitomi Kaze Basic I D Skeleton Arm May Natural Language Processing Grammar Understanding Net Deep Learning Neural Network Human Intelligence Technology Method I Who is the program One World People I Japanese Japanese Feeling Edel Body Self Answer I Artificial Intelligence Edel himself Edel I-Voice I I Word Nori Feeling I Result Self Achievement Artificial Intelligence Her Talking partner Best friend Friend I Inari I haven't been in a long time Izakaya It I Arita University Classmates People shade Life as it is I She What Cause Mind Fog Mail Content Time She She Tone Dialogue Ken Several years Her Ex-girlfriend I throat Air Home Next room Shock What Balance She It Life Researcher Friend It's Nori Nori High School Just before my favorite person Boyfriend Mental University Tama Fleet Myself She Nori Nori Nori I I International Society Thesis Researcher I I myself Original Thesis Instructor Person Street Thesis I myself Nobody My genius skin Nori Nori Search Noriaki Miya Nori Akira Undergraduate 1st year Paper Page University Homepage Programming Contest Grand Prize Fun Student My Best Friend Rival Nori Myself Nori What Inari PC Data Pretty Shock House WIFI PC Inari House Train WIFI PC Data Security Interest Vulnerability File Can Female Character File This shock Inari file Outside floor sound Immediately my situation data Her guilty feeling File data My PC data PC My trace Possibility data Bespoke I Nori Which shock Outside What room People What window You haven't been in a long time Window face 3 Year Nori Nori My Home Cause I Face Important Things It I Inari My Mouth Words Nori Another Topic Tossa Here Nori Tavern My Do Home Atmosphere I What Head Topic Tavern Nori What Tavern Nori What Twitter Syndrome Tavern Topic Nori Contents Nori Faculty of Engineering Software I Standard Template Core What Nori What Inari My PC Nori Kita House It House Nori She Talks Absurdly Nori Remains Hands Each other Direction Road Back Reverse
hashire15-meishi.txt
Around Books Power PC Nori PC Data USB This Home Performance Computer Data Data Nori Somewhere Data Data Terminal Screen Screen Your Face Eye pupil Capacity Big Folder This Data Ken Inari Folder Execution File ai ver exe Terminal Window Screen Face Beautiful Girl My Name Your Name Population Incompetence Population Incompetence Sentence iPhone siri Population Incompetence Nori Middle High Nori I Genius Nori Originality Genius Disposition Friend I Genius Nori Her Name Person I Name Takeru Middle Finger Enter Key Name Message Basic Population Incompetence You Source Code Where Wind Agent I Source Code Folder I Folder Editor Source Code Source Code Personal Here Nori Source Code Deep Learning Latest Method Ken Inari Best Friend Source Code Careful Technology Desire Ken Inari Latest Artificial Intelligence Knowledge Myself Nori It So There Inari Researcher's Will Magazine It Method Inari My Thesis Mathematical Thesis Computer Human Intelligence Inari Computer Words PC Screen Nori Voice Voice That Hair Color Tone Eyes Screen Uncomfortable This Paper Iroha Inari SNS Blonde Preference My Artificial Intelligence Machine I Human Confused Inari Blonde Artificial Intelligence My Inari Method Deep Learning Deep Learning Human System Data Human Inari I Inari Ichimoto Nori Nori Nori Nori Nori It Friend Inari Ken Nori Nori Nori Internet Nori Fake Mirror Site I Nori As planned I I Nori Friend Wind Takeru Artificial Intelligence It It Accident Lie Lie Virus Inari Internet Virus Inari Self Artificial Intelligence Friend Lie Noriaki-san Kibe Actor I Nose Confidence Inari Artificial Intelligence Tone Kaze Nori Family I Heart Inari Truth Heart Lie No way Nori My Eyes Man My Data Data Nori State Somewhere Data Data Terminal Screen Screen My Face Eyes Big folder This data Ken Inari folder Execution file ai ver exe Terminal window screen Beautiful face girl I Name You Name Artificial Intelligence Population Incompetence Sentence iPhone siri Population Incompetence Nori Middle High Nori I Genius Nori Originality Genius Disposition Friend I Genius Nori Her Name Person I Name Takeru Middle Finger Enter Key Name Subject Basic Population Incompetence You Source Code Where Wind Agent Code Folder Two I Folder Editor Source Code Source Code Personal Here Nori Source Code Deep Learning Latest Method Ken Inari Best Friend Source Code Careful Technology Desire Ken Inari Latest Artificial Intelligence Knowledge Self Nori It So There Inari Researcher's Will Magazine It Method Inari My Paper Formula Paper Computer Human Intelligence Inari Personal Computer Words PC Screen Nori Voice Voice That Hair Color Tone Eyes Screen Uncomfortable This Paper Iroha Inari SNS Blonde Preference My Artificial Intelligence Machine I Human Confused Inari Blonde Artificial Intelligence My Inari Method Deep Learning Deep Learning Deep Learning Human Inari Ken myself Inari Ichimoto Nori Nori Nori I Nori It's friend Inari Ken Nori Nori Nori Internet Nori Fake Mirror Site I Nori As planned I'm Nori Friend Wind Takeru Artificial intelligence It It's an accident lie Virus Hand Inari himself Artificial intelligence Friend Lie Noriaki-san Kibe Actor I Nose Confidence Inari Artificial Intelligence Tone Kaze Nori Family I Heart Inari Truth Heart Lie Nori My Eyes Man
hashire16-meishi.txt
Wind Skin Season Short Sleeve My Best Friend Acquaintance What Edel I Smartphone Voice App Edel People I Edel Sky Edel Feelings I Edel Causes Virus Infection Thanks People Emotions I Edel Emotions Talking Partners Edel Reverse Emotions I Edel It's My Edel It's My Darkness Responsibility I'm Edel Emotions I Edel You Feelings It's I Author I Edel Edel Between Nori Nori What Face Lie What Always Really It Friend I Edel Friendship
hashire17-meishi.txt
Friend Friend Friend Friend Friend Best Friend Words Everyone People I Substitute High School Friend Best Friend Words High School I Best Friend Height Guy I It That That That Miya I That That That SD Card Movie Together Handheld Game Console Hiyahiya Contents Movie My Favorite Movie One Soccer Club Somewhere Club activities members Heart somewhere Miya club activities Leisure time High school days Hobbies What empty house Last I Chance Midwinter Snow shoes That guy Somewhere together Junior high school man Man This type Somewhere dark man I Snow I Signal blue That guy I Arm Red car Same sex friend What That guy's house Department store invitation Soccer club Winter body Soccer club Tama School Naka I One day shock You Ki Artificial intelligence She I Word High school third grader That guy Face Tears Depression It schizophrenia So situation How many times Hospital I Around me Who it AI I Edel Mike AI Edel BB Right hand smartphone Edel I Someone's smartphone screen Inari After a long time Somewhere Inari Tone What topic What Arita Continued Nori PC I What Nori ... Inari Net I feel Inari Takeru 16:30 Inari meeting place This lie…. Miya Friend Hidden Friend Inari Facial Expression Kana Inari Junior High School Friend I High School Soccer Club Classmate My Man Who I Inari Aizuchi I First Voice Words Inari Words I Miya I Nori Artificial Intelligence Technical Warmth Inari Artificial Intelligence Intelligence It's own technology ... I Edel head human brain model Neural network technology Inari technology It's a dissertation I myself Researcher Egg blood Inari technology World example model I wonder if it's confident Inari I'll do it together Nori Together ... Nori I Let's have an academic meeting together I Inari Researcher Eye I Inari Theory Partial Dissertation My Family Inari Family Information Bookstore Mathematics Book Formula Quote Source Nervous 3 days My Family Inari University Laboratory Inter-Laboratory Paper Goo I Right Hand Inari Left Hand Someone Somewhere Professor Toko Who We Friend Arita Her Former Paper Arita I Inari First Appearance Face Hair Bassabosa I Paper Arita Number I Person It Internet User Absolutely Unreasonable My Paper Site Our Next Site This Paper Tired Next Mail Box Paper Pretty I Smartphone Inari Takeru Mailbox Takeru Voice That Laptop Where Inari Bed English This Page Site Takeru Our Site Paper Example Github Source Code Site README md Rewrite Twitter Tweet Situation It Major Paper Paper Article Friend Word Word Word Takeru Paper with Friend Somewhere I'm not me Somewhere we are each other Takeru Institute Researcher Michi I I I Faculty International Society Hobbies Two people Upper Haha I Takeru Arita Raw Beer Cunyan Plum Sake Glass Arita Arita Society Nursing Robot Ladder Sake Chara I Takeru Arita Sake
It's a little too much, but copy the above text and save it in your working directory. The code that actually calculates the weight with tfidf and outputs the word is as follows.
countVector.py
import numpy as np
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.feature_extraction.text import TfidfTransformer
from sklearn.feature_extraction import DictVectorizer
count = CountVectorizer()
hashire01 = open('./hashire01-meishi.txt', 'r').read()
hashire02 = open('./hashire02-meishi.txt', 'r').read()
hashire03 = open('./hashire03-meishi.txt', 'r').read()
hashire04 = open('./hashire04-meishi.txt', 'r').read()
hashire05 = open('./hashire05-meishi.txt', 'r').read()
hashire06 = open('./hashire06-meishi.txt', 'r').read()
hashire07 = open('./hashire07-meishi.txt', 'r').read()
hashire08 = open('./hashire08-meishi.txt', 'r').read()
hashire09 = open('./hashire09-meishi.txt', 'r').read()
hashire10 = open('./hashire10-meishi.txt', 'r').read()
hashire11 = open('./hashire11-meishi.txt', 'r').read()
hashire12 = open('./hashire12-meishi.txt', 'r').read()
hashire13 = open('./hashire13-meishi.txt', 'r').read()
hashire14 = open('./hashire14-meishi.txt', 'r').read()
hashire15 = open('./hashire15-meishi.txt', 'r').read()
hashire16 = open('./hashire16-meishi.txt', 'r').read()
hashire17 = open('./hashire17-meishi.txt', 'r').read()
docs = np.array([hashire01, hashire02, hashire03, hashire04, hashire05, hashire06, hashire07, hashire08, hashire09, hashire10, hashire11, hashire12, hashire13, hashire14, hashire15, hashire16, hashire17])
bag = count.fit_transform(docs)
#What number does the word she belong to in the array?
# print(count.vocabulary_.get('Girlfriend'))
#tfidf settings
tfidf = TfidfTransformer(use_idf=True, norm='l2', smooth_idf=True)
#Display up to 4 decimal places.
np.set_printoptions(precision=4)
cnt_voc = count.vocabulary_
tfidf_array = np.array(tfidf.fit_transform(bag).toarray())
#The processing required to associate a vector with a word.
vec = DictVectorizer()
vec.fit_transform(cnt_voc).toarray()
feature = vec.get_feature_names()
#tfidf threshold setting
threshold = 0.5
#Substitute the number of the hit element. For some reason it is passed in two arrays.
hairetsu = np.where(tfidf_array >= threshold)
# np.Remove duplicate values with unique.
#For some reason np.If it is where, two arrays will be returned, so concatenate them into one.
hit_word = np.unique(np.append(hairetsu[0], hairetsu[1]))
#Outputs the word calculated by tfidf.
for num in range(0, len(hit_word)):
print(feature[hit_word[num]])
output.txt
1 day
ai
ar
c language
exe
google search
Edel
name
boy
Girlfriend
Arita
Inari
As you can see from the Contents of the novel, the characters (Edel, Arita, Inari) appear, so the characteristics of the text are explained. I think I'm holding it down.
I wrote it as a specific procedure, but since I wrote a comment in the source code, I think you can understand it, so I will not write it to avoid making the explanation redundant.
Even so, if you write it for the time being, threshold is the threshold, and if this value is high, it means that it is a word that has an important meaning in the document. After this, I think that the documents will be classified using a support vector machine or the like.
I will add it because it is read by many people unexpectedly. This time, the features were extracted by tfidf, but it is also possible to extract the features by other techniques such as word embedding (word2vec, GloVe, fastText, Exponential Family Embeddings). It should be more accurate than tf idf. I think it's a good subject in that it's easy to implement.
** Also, only nouns are taken out this time, but I think it is better to discuss whether it is really valid. ** **
Recommended Posts