########## cell 1 (exec 1) :: ########## cell 2 (exec 2) :: # [Data loading changed] Only the data-loading part is modified (allow downloaded questions-words.txt ########## cell 3 (exec 3) :: file_name = "questions-words" ########## cell 4 (exec 4) :: for entry in data[:10]: : capital-common-countries Athens Greece Baghdad Iraq Athens Greece Bangkok Thailand Athens Greece Beijing China Athens Greece Berlin Germany Athens Greece Bern Switzerland Athens Greece Cairo Egypt Athens Greece Canberra Australia Athens Greece Hanoi Vietnam Athens Greece Havana Cuba ########## cell 5 (exec 5) :: # TODO1: Write your code here for processing data to pd.DataFrame 19544 questions ########## cell 6 (exec 6) :: df = pd.DataFrame( ########## cell 7 (exec 7) :: Question Category SubCategory 0 Athens Greece Baghdad Iraq Semantic : capital-common-countries 1 Athens Greece Bangkok Thailand Semantic : capital-common-countries 2 Athens Greece Beijing China Semantic : capital-common-countries 3 Athens Greece Berlin Germany Semantic : capital-common-countries 4 Athens Greece Bern Switzerland Semantic : capital-common-countries ########## cell 8 (exec 8) :: ########## cell 10 (exec 9) :: import numpy as np ########## cell 11 (exec 10) :: ########## cell 12 (exec 11) :: # You can try other models. The Gensim model loaded successfully! ########## cell 13 (exec 12) :: # Do predictions and preserve the gold answers (word_D) OOV questions: 0 ########## cell 14 (exec 13) :: Category: Semantic, Accuracy: 65.3399481339497% Category: Syntactic, Accuracy: 61.255269320843084% Sub-Category: capital-common-countries, Accuracy: 93.87351778656127% Sub-Category: capital-world, Accuracy: 88.94783377541998% Sub-Category: currency, Accuracy: 14.203233256351039% Sub-Category: city-in-state, Accuracy: 30.806647750304013% Sub-Category: family, Accuracy: 81.62055335968378% Sub-Category: gram1-adjective-to-adverb, Accuracy: 24.39516129032258% Sub-Category: gram2-opposite, Accuracy: 20.073891625615765% Sub-Category: gram3-comparative, Accuracy: 79.12912912912913% Sub-Category: gram4-superlative, Accuracy: 54.278074866310156% Sub-Category: gram5-present-participle, Accuracy: 69.50757575757575% Sub-Category: gram6-nationality-adjective, Accuracy: 87.86741713570981% Sub-Category: gram7-past-tense, Accuracy: 55.44871794871795% Sub-Category: gram8-plural, Accuracy: 71.996996996997% Sub-Category: gram9-plural-verbs, Accuracy: 58.39080459770115% ########## cell 15 (exec 14) :: # Collect words from Google Analogy dataset [image cell15_0.png]
########## cell 18 (exec 15) :: # [Data loading changed] Only the data-loading part is modified (allow data\wiki_texts_part_0.txt.gz 1.51 GB data\wiki_texts_part_1.txt.gz 0.84 GB data\wiki_texts_part_2.txt.gz 0.67 GB data\wiki_texts_part_3.txt.gz 0.60 GB data\wiki_texts_part_4.txt.gz 0.57 GB data\wiki_texts_part_5.txt.gz 0.58 GB data\wiki_texts_part_6.txt.gz 0.56 GB data\wiki_texts_part_7.txt.gz 0.53 GB data\wiki_texts_part_8.txt.gz 0.55 GB data\wiki_texts_part_9.txt.gz 0.55 GB data\wiki_texts_part_10.txt.gz 0.00 GB ########## cell 19 (exec 16) :: # [Data loading changed] Only the data-loading part is modified (allow all parts downloaded ########## cell 20 (exec 17) :: # [Data loading changed] Only the data-loading part is modified (allow 11 parts: ['wiki_texts_part_0.txt.gz', 'wiki_texts_part_1.txt.gz', 'wiki_texts_part_2.txt.gz', 'wiki_texts_part_3.txt.gz', 'wiki_texts_part_4.txt.gz', 'wiki_texts_part_5.txt.gz', 'wiki_texts_part_6.txt.gz', 'wiki_texts_part_7.txt.gz', 'wiki_texts_part_8.txt.gz', 'wiki_texts_part_9.txt.gz', 'wiki_texts_part_10.txt.gz'] ########## cell 21 (exec 18) :: # [Data loading changed] Only the data-loading part is modified (allow total compressed size: 6.93 GB ########## cell 22 (exec 19) :: # [Data loading changed] Only the data-loading part is modified (allow 0 anarchism is political philosophy and movement that is against all forms of authority and seeks to abolish the institutions it claims maintain unnecessary coercion and hierarchy typically including th ... 1 albedo change in greenland the map shows the difference between the amount of sunlight greenland reflected in the summer of versus the average percent it reflected between and some areas reflect close ... 2 or is the first letter and the first vowel letter of the latin alphabet used in the modern english alphabet and others worldwide its name in english is pronounced plural aes it is similar in shape to ... 3 alabama is state in the southeastern region of the united states it borders tennessee to the north georgia to the east florida and the gulf of mexico to the south and mississippi to the west alabama i ... 4 in greek mythology achilles or achilleus was hero of the trojan war who was known as being the greatest of all the greek warriors the central character in homer iliad he was the son of the nereid thet ... 5 abraham lincoln february april was an american lawyer politician and statesman who served as the th president of the united states from until his assassination in he led the united states through the ... 6 aristotle aristotélēs bc was an ancient greek philosopher and polymath his writings cover broad range of subjects spanning the natural sciences philosophy linguistics economics politics psychology and ... 7 an american in paris is jazz influenced symphonic poem or tone poem for orchestra by american composer george gershwin first performed in it was inspired by the time that gershwin had spent in paris a ... 8 the academy award for best production design recognizes achievement for art direction in film the category original name was best art direction but was changed to its current name in for the th academ ... 9 the academy awards of merit commonly known as the oscars or academy awards are awards for artistic and technical merit given for excellence within the american and international film industry they are ... ########## cell 24 (exec 20) :: # Now you need to do sampling because the corpus is too big. articles: total 5,623,655, sampled 1,124,733 (20.00%) wiki_sampled_5.txt 1.04 GB wiki_sampled_10.txt 2.08 GB wiki_sampled_20.txt 4.16 GB ########## cell 25 (exec 21) :: # TODO5: Train your own word embeddings with the sampled articles wiki_sampled_20.txt: removed non-[a-z] tokens 4,828,696 of 670,881,822 (0.72%) 20% wiki tokens after preprocessing: 666,053,126 trained on wiki_sampled_20_clean.txt: 37.9 min, vocab 299,405, effective min_count 30 ########## cell 26 (exec 22) :: ########## cell 27 (exec 23) :: # Do predictions and preserve the gold answers (word_D) OOV questions: 0 Category: Semantic, Accuracy: 56.70312323824558% Category: Syntactic, Accuracy: 41.48009367681499% Sub-Category: capital-common-countries, Accuracy: 77.86561264822134% Sub-Category: capital-world, Accuracy: 72.81167108753316% Sub-Category: currency, Accuracy: 10.277136258660507% Sub-Category: city-in-state, Accuracy: 35.508715038508306% Sub-Category: family, Accuracy: 74.30830039525692% Sub-Category: gram1-adjective-to-adverb, Accuracy: 15.92741935483871% Sub-Category: gram2-opposite, Accuracy: 16.00985221674877% Sub-Category: gram3-comparative, Accuracy: 49.0990990990991% Sub-Category: gram4-superlative, Accuracy: 24.331550802139038% Sub-Category: gram5-present-participle, Accuracy: 31.344696969696972% Sub-Category: gram6-nationality-adjective, Accuracy: 80.48780487804879% Sub-Category: gram7-past-tense, Accuracy: 43.84615384615385% Sub-Category: gram8-plural, Accuracy: 41.66666666666667% Sub-Category: gram9-plural-verbs, Accuracy: 40.91954022988506% ########## cell 28 (exec 24) :: # Collect words from Google Analogy dataset [image cell28_0.png]
########## cell 31 (exec 25) :: # [Generative AI] The code in this cell was generated by Claude (Anthr GloVe (pre-trained): overall 63.11% (OOV questions 0) Wiki 20%: overall 48.39% (OOV questions 0) ########## cell 32 (exec 26) :: # 第 3 題：換一份「不是維基百科」的語料。 keys: ['RelComments', 'RelQuestion', 'THREAD_SEQUENCE'] | question keys: ['RELQ_DATE', 'RELQ_ID', 'RelQSubject', 'RELQ_USERNAME', 'RELQ_USERID', 'RelQBody', 'RELQ_CATEGORY'] subject: Thailand:IT Minsitry blocks CNN; Facebook; first comment: have they blocked porn??? ########## cell 33 (exec 27) :: # 用跟維基百科一樣的規則前處理：先拿掉網頁語法和網址，再小寫、只留 2～15 個字母的英文字（WikiCorpus 的預設）。 forum: 189,941 threads, 2,118,254 posts, 72,919,300 tokens wiki_sampled_5.txt: removed non-[a-z] tokens 1,211,749 of 167,406,150 (0.72%) 5% wiki tokens: 166,194,401 wiki control: 72,920,923 tokens ########## cell 34 (exec 28) :: forum_model = train_w2v("forum_clean.txt") trained on forum_clean.txt: 3.9 min, vocab 100,310, effective min_count 5 trained on wiki_control_clean.txt: 3.9 min, vocab 224,050, effective min_count 5 Forum (same size): overall 25.78% (OOV questions 3510) Wiki control (same size): overall 41.46% (OOV questions 169) Wiki control (same size) Forum (same size) \ Overall 41.46 25.78 Semantic 45.77 12.84 Syntactic 37.87 36.53 : capital-common-countries 76.48 33.60 : capital-world 54.00 12.51 : currency 6.93 1.27 : city-in-state 33.81 2.92 : family 66.21 63.24 : gram1-adjective-to-adverb 10.28 13.10 : gram2-opposite 8.87 14.04 : gram3-comparative 53.08 64.94 : gram4-superlative 17.20 40.55 : gram5-present-participle 31.82 51.33 : gram6-nationality-adjective 76.55 21.70 : gram7-past-tense 40.96 33.78 : gram8-plural 36.26 41.97 : gram9-plural-verbs 32.99 41.49 OOV questions 169.00 3510.00 Wiki 20% Overall 48.39 Semantic 56.70 Syntactic 41.48 : capital-common-countries 77.87 : capital-world 72.81 : currency 10.28 : city-in-state 35.51 : family 74.31 : gram1-adjective-to-adverb 15.93 : gram2-opposite 16.01 : gram3-comparative 49.10 : gram4-superlative 24.33 : gram5-present-participle 31.34 : gram6-nationality-adjective 80.49 : gram7-past-tense 43.85 : gram8-plural 41.67 : gram9-plural-verbs 40.92 OOV questions 0.00 coverage: wiki control coverage: forum SubCategory : capital-common-countries 100.0 100.0 : capital-world 98.3 48.6 : currency 86.8 39.0 : city-in-state 100.0 68.3 : family 100.0 91.3 : gram1-adjective-to-adverb 100.0 100.0 : gram2-opposite 100.0 100.0 : gram3-comparative 100.0 100.0 : gram4-superlative 94.1 100.0 : gram5-present-participle 100.0 100.0 : gram6-nationality-adjective 100.0 95.1 : gram7-past-tense 100.0 100.0 : gram8-plural 100.0 100.0 : gram9-plural-verbs 100.0 100.0 Accuracy on questions whose 4 words are in BOTH vocabularies: Wiki control (same size) Forum (same size) questions used 15684.00 15684.00 Overall 42.80 32.12 Semantic 52.80 22.10 Syntactic 37.91 37.01 : capital-common-countries 76.48 33.60 : capital-world 62.03 25.74 : currency 11.26 3.64 : city-in-state 36.34 4.28 : family 70.13 69.26 : gram1-adjective-to-adverb 10.28 13.10 : gram2-opposite 8.87 14.04 : gram3-comparative 53.08 64.94 : gram4-superlative 18.28 42.90 : gram5-present-participle 31.82 51.33 : gram6-nationality-adjective 77.12 22.81 : gram7-past-tense 40.96 33.78 : gram8-plural 36.26 41.97 : gram9-plural-verbs 32.99 41.49 ########## cell 35 (exec 29) :: # 同一個字，在論壇和維基（一樣大小）學到的鄰居差在哪 word corpus \ 0 visa Wiki control (same size) 1 visa Forum (same size) 2 salary Wiki control (same size) 3 salary Forum (same size) 4 doha Wiki control (same size) 5 doha Forum (same size) 6 family Wiki control (same size) 7 family Forum (same size) 8 king Wiki control (same size) 9 king Forum (same size) 10 paris Wiki control (same size) 11 paris Forum (same size) 12 computer Wiki control (same size) 13 computer Forum (same size) top 5 0 visas, passport, bdtc, passports, zwartendijk 1 rp, viza, iqama, residance, vissa 2 salaries, remuneration, wages, repayment, paid 3 slary, salry, salery, sallary, salaray 4 qatar, manama, abbasiyyin, dhabi, dubai 5 qatar, doah, qata, qatat, qutar 6 father, relatives, phyllanthaceae, grandparents, parents 7 familly, fmaily, rescidence, pernament, husbant 8 queen, prince, monarch, harthacnut, eystein 9 queen, ypsilanti, edsel, bhumibol, edshel 10 fontainebleau, marseille, bercy, brussels, reims 11 amsterdam, cancun, france, bangkok, london 12 computers, software, mainframe, hardware, computing 13 soundcard, desktop, pc, computers, hardware ########## cell 36 (exec 30) :: # 第 3 題補充證據：同樣的字，在兩份一樣大的教材裡各出現幾次（每 100 萬字）。 Wiki control (per 1M): 72,920,923 tokens Forum (per 1M): 72,919,300 tokens Wiki control (per 1M) Forum (per 1M) forum / wiki better 116.98 990.40 8.47 best 553.01 988.65 1.79 cheaper 7.78 80.27 10.32 cheapest 0.96 27.50 28.64 bigger 13.12 53.58 4.08 biggest 37.26 59.97 1.61 looking 56.31 680.74 12.09 going 104.22 914.05 8.77 working 187.26 583.48 3.12 costs 43.99 77.13 1.75 goes 66.47 235.33 3.54 wife 212.38 605.50 2.85 husband 104.85 418.87 3.99 he 5122.78 3669.63 0.72 she 1266.80 1885.25 1.49 capital 160.38 43.23 0.27 illinois 90.02 1.10 0.01 albanian 13.37 0.75 0.06 brazilian 42.43 33.46 0.79 kwanza 0.19 0.01 0.07 ########## cell 38 (exec 31) :: # 第 2 題：5%、10%、20%（助教說這題不用附程式，這裡附上供參考） wiki_sampled_10.txt: removed non-[a-z] tokens 2,417,608 of 335,470,319 (0.72%) 10% wiki tokens: 333,052,711 trained on wiki_sampled_5_clean.txt: 9.1 min, vocab 278,954, effective min_count 9 Wiki 5%: overall 46.12% (OOV questions 107) trained on wiki_sampled_10_clean.txt: 19.7 min, vocab 294,612, effective min_count 16 Wiki 10%: overall 47.15% (OOV questions 0) Wiki 5% Wiki 10% Wiki 20% Overall 46.12 47.15 48.39 Semantic 55.35 55.60 56.70 Syntactic 38.45 40.13 41.48 : capital-common-countries 81.82 76.48 77.87 : capital-world 69.83 71.07 72.81 : currency 7.62 9.58 10.28 : city-in-state 37.37 35.47 35.51 : family 68.77 73.32 74.31 : gram1-adjective-to-adverb 12.70 10.99 15.93 : gram2-opposite 11.82 14.29 16.01 : gram3-comparative 44.14 46.92 49.10 : gram4-superlative 17.02 22.37 24.33 : gram5-present-participle 29.92 34.28 31.34 : gram6-nationality-adjective 78.42 76.11 80.49 : gram7-past-tense 44.29 44.87 43.85 : gram8-plural 38.59 40.77 41.67 : gram9-plural-verbs 37.82 41.49 40.92 OOV questions 107.00 0.00 0.00 ########## cell 40 (exec 32) :: # 第 4 題：挑字，各找 5 個最像的字。同時看現成的 GloVe 和自己訓練的 Wiki 20%。 word model \ 0 king GloVe (pre-trained) 1 king Wiki 20% 2 cat GloVe (pre-trained) 3 cat Wiki 20% 4 apple GloVe (pre-trained) 5 apple Wiki 20% 6 bank GloVe (pre-trained) 7 bank Wiki 20% 8 good GloVe (pre-trained) 9 good Wiki 20% 10 taiwan GloVe (pre-trained) 11 taiwan Wiki 20% 12 computer GloVe (pre-trained) 13 computer Wiki 20% top 5 (similarity) 0 prince (0.77), queen (0.75), son (0.70), brother (0.70), monarch (0.70) 1 nangklao (0.74), chlothar (0.72), queen (0.70), suryavarman (0.70), bodawpaya (0.70) 2 dog (0.88), rabbit (0.74), cats (0.73), monkey (0.73), pet (0.72) 3 rabbit (0.73), dog (0.72), sourpuss (0.72), mouse (0.71), pet (0.70) 4 microsoft (0.74), ibm (0.68), intel (0.68), software (0.68), dell (0.67) 5 blackberry (0.79), iphone (0.70), goldieblox (0.69), tvos (0.68), raspberry (0.68) 6 banks (0.81), banking (0.75), credit (0.70), investment (0.69), financial (0.68) 7 guaranty (0.77), savings (0.77), ameriprise (0.77), indymac (0.76), onewest (0.76) 8 better (0.89), sure (0.83), really (0.83), kind (0.83), very (0.83) 9 sure (0.71), decent (0.71), lovely (0.70), bad (0.70), thankful (0.70) 10 mainland (0.86), china (0.83), taiwanese (0.79), taipei (0.79), hong (0.77) 11 taipei (0.83), china (0.82), guangdong (0.82), hainan (0.79), fujian (0.79) 12 computers (0.88), software (0.84), technology (0.76), pc (0.74), hardware (0.73) 13 computing (0.83), computers (0.79), software (0.79), mainframe (0.77), hardware (0.74) ########## cell 41 (exec 33) :: # 第 4 題補充證據 king -> [('nangklao', np.int64(34)), ('chlothar', np.int64(76)), ('queen', np.int64(92609)), ('suryavarman', np.int64(53)), ('bodawpaya', np.int64(77))] cat -> [('rabbit', np.int64(7844)), ('dog', np.int64(36766)), ('sourpuss', np.int64(64)), ('mouse', np.int64(13524)), ('pet', np.int64(9983))] apple -> [('blackberry', np.int64(1285)), ('iphone', np.int64(2800)), ('goldieblox', np.int64(33)), ('tvos', np.int64(102)), ('raspberry', np.int64(1710))] bank -> [('guaranty', np.int64(330)), ('savings', np.int64(8467)), ('ameriprise', np.int64(49)), ('indymac', np.int64(86)), ('onewest', np.int64(38))] word top 5 among the 50k most frequent words 0 king queen, throne, prince, reigned, ruler 1 cat rabbit, dog, mouse, pet, mug 2 apple blackberry, iphone, raspberry, ipad, android 3 bank savings, bancorp, banking, lenders, jpmorgan 4 good sure, decent, lovely, bad, thankful 5 taiwan taipei, china, guangdong, hainan, fujian 6 computer computing, computers, software, mainframe, hardware apple top 15: ['blackberry', 'iphone', 'goldieblox', 'tvos', 'raspberry', 'xelibri', 'iigs', 'visicalc', 'magix', 'airis', 'cloudflare', 'roxio', 'ipad', 'livescribe', 'android'] similarity(apple, banana) = 0.41 similarity(apple, fruit) = 0.37 similarity(apple, microsoft) = 0.65 similarity(bank, river) = 0.49 similarity(bank, money) = 0.40 similarity(good, bad) = 0.70 ... of guard dog bushy doozy best friend and hero to tom tom ms lulu the pet psychic sourpuss owner sourpuss owner and friend of millie animals bip and bop two nutty squirrel ... ... toon characters including fanny zilch mighty mouse heckle and jeckle gandy goose sourpuss dinky duck little roquefort the terry bears dimwit and luno terry pre existing c ... ... k puck in hockey almost certainly from irish poc according to the oed puss as in sourpuss comes from irish pus pouting mouth rapparee an irish highwayman from ropaire sta ... ... n year title role notes hangin in episode the princess and the pea today special sourpuss sal episode smiles murielle episode three monkeys of bah roghar part episode thr ... ... ack flag discharge and flipper in few days before her th birthday her first band sourpuss played set at australia summersault festival where she met tim armstrong frontma ... ########## cell 43 (exec 34) :: # 第 5 題 (a)：同樣設定、只換亂數種子，結果會晃多少？——先量雜訊，後面的比較才知道哪些差距算數。 Wiki 5% seed=1: overall 46.05% (OOV questions 107) Wiki 5% seed=2: overall 47.10% (OOV questions 107) Wiki 5% seed=3: overall 46.34% (OOV questions 107) seed 42 seed 1 seed 2 seed 3 max - min Overall 46.12 46.05 47.10 46.34 1.05 Semantic 55.35 54.18 55.98 54.72 1.80 Syntactic 38.45 39.30 39.72 39.37 1.26 : capital-common-countries 81.82 80.04 82.81 81.82 2.77 : capital-world 69.83 67.44 69.43 68.21 2.39 : currency 7.62 7.39 8.89 8.31 1.50 : city-in-state 37.37 38.27 39.04 37.74 1.66 : family 68.77 67.39 72.13 69.17 4.74 : gram1-adjective-to-adverb 12.70 11.29 14.21 11.19 3.02 : gram2-opposite 11.82 10.84 11.08 13.05 2.22 : gram3-comparative 44.14 46.10 47.82 48.87 4.73 : gram4-superlative 17.02 17.83 18.98 19.79 2.76 : gram5-present-participle 29.92 29.73 31.53 34.47 4.73 : gram6-nationality-adjective 78.42 80.11 78.86 78.42 1.69 : gram7-past-tense 44.29 44.81 44.55 42.37 2.44 : gram8-plural 38.59 40.62 39.86 37.99 2.63 : gram9-plural-verbs 37.82 39.77 38.97 37.70 2.07 ########## cell 44 (exec 35) :: # 第 5 題 (b)：為什麼不移除停用詞？先看考卷，再真的做實驗比較。 questions containing a stop word: 206 SubCategory : family 86 : gram1-adjective-to-adverb 62 : currency 58 Name: count, dtype: int64 ['Algeria dinar Korea won', 'Angola kwanza Korea won', 'Argentina peso Korea won', 'Armenia dram Korea won', 'Brazil real Korea won'] trained on wiki_sampled_5_nostop.txt: 7.0 min, vocab 278,810, effective min_count 9 Wiki 5% (stop words removed): overall 45.25% (OOV questions 284) Wiki 5% Wiki 5% (stop words removed) diff \ Overall 46.12 45.25 -0.87 Semantic 55.35 55.41 0.06 Syntactic 38.45 36.81 -1.65 : capital-common-countries 81.82 83.99 2.17 : capital-world 69.83 70.87 1.04 : currency 7.62 8.89 1.27 : city-in-state 37.37 37.82 0.45 : family 68.77 53.95 -14.82 : gram1-adjective-to-adverb 12.70 12.80 0.10 : gram2-opposite 11.82 9.98 -1.85 : gram3-comparative 44.14 37.01 -7.13 : gram4-superlative 17.02 16.31 -0.71 : gram5-present-participle 29.92 30.87 0.95 : gram6-nationality-adjective 78.42 80.61 2.19 : gram7-past-tense 44.29 41.09 -3.21 : gram8-plural 38.59 41.89 3.30 : gram9-plural-verbs 37.82 26.55 -11.26 noise (max-min of 4 seeds) beyond noise? Overall 1.05 False Semantic 1.80 False Syntactic 1.26 True : capital-common-countries 2.77 False : capital-world 2.39 False : currency 1.50 False : city-in-state 1.66 False : family 4.74 True : gram1-adjective-to-adverb 3.02 False : gram2-opposite 2.22 False : gram3-comparative 4.73 True : gram4-superlative 2.76 False : gram5-present-participle 4.73 False : gram6-nationality-adjective 1.69 True : gram7-past-tense 2.44 True : gram8-plural 2.63 True : gram9-plural-verbs 2.07 True ########## cell 45 (exec 36) :: # 第 5 題 (c)：超參數比較（都用 5% 維基，一次只改一個設定，其他跟主模型一樣） Wiki 5% CBOW (sg=0): trained in 9.3 min Wiki 5% CBOW (sg=0): overall 52.58% (OOV questions 107) Wiki 5% window=10: trained in 15.2 min Wiki 5% window=10: overall 41.75% (OOV questions 107) Wiki 5% vector_size=300: trained in 17.9 min Wiki 5% vector_size=300: overall 57.44% (OOV questions 107) Wiki 5% Wiki 5% CBOW (sg=0) diff \ Overall 46.12 52.58 6.46 Semantic 55.35 58.60 3.25 Syntactic 38.45 47.58 9.12 : capital-common-countries 81.82 79.45 -2.37 : capital-world 69.83 71.86 2.03 : currency 7.62 8.43 0.81 : city-in-state 37.37 43.66 6.28 : family 68.77 77.87 9.09 : gram1-adjective-to-adverb 12.70 19.15 6.45 : gram2-opposite 11.82 11.45 -0.37 : gram3-comparative 44.14 69.14 25.00 : gram4-superlative 17.02 37.61 20.59 : gram5-present-participle 29.92 40.81 10.89 : gram6-nationality-adjective 78.42 76.61 -1.81 : gram7-past-tense 44.29 46.03 1.73 : gram8-plural 38.59 49.55 10.96 : gram9-plural-verbs 37.82 48.16 10.34 noise (max-min of 4 seeds) beyond noise? Overall 1.05 True Semantic 1.80 True Syntactic 1.26 True : capital-common-countries 2.77 False : capital-world 2.39 False : currency 1.50 False : city-in-state 1.66 True : family 4.74 True : gram1-adjective-to-adverb 3.02 True : gram2-opposite 2.22 False : gram3-comparative 4.73 True : gram4-superlative 2.76 True : gram5-present-participle 4.73 True : gram6-nationality-adjective 1.69 True : gram7-past-tense 2.44 False : gram8-plural 2.63 True : gram9-plural-verbs 2.07 True Wiki 5% Wiki 5% window=10 diff \ Overall 46.12 41.75 -4.37 Semantic 55.35 51.22 -4.13 Syntactic 38.45 33.87 -4.58 : capital-common-countries 81.82 72.53 -9.29 : capital-world 69.83 68.24 -1.59 : currency 7.62 7.85 0.23 : city-in-state 37.37 31.01 -6.36 : family 68.77 50.59 -18.18 : gram1-adjective-to-adverb 12.70 11.09 -1.61 : gram2-opposite 11.82 6.90 -4.93 : gram3-comparative 44.14 32.88 -11.26 : gram4-superlative 17.02 9.80 -7.22 : gram5-present-participle 29.92 27.46 -2.46 : gram6-nationality-adjective 78.42 78.80 0.38 : gram7-past-tense 44.29 39.49 -4.81 : gram8-plural 38.59 33.56 -5.03 : gram9-plural-verbs 37.82 33.22 -4.60 noise (max-min of 4 seeds) beyond noise? Overall 1.05 True Semantic 1.80 True Syntactic 1.26 True : capital-common-countries 2.77 True : capital-world 2.39 False : currency 1.50 False : city-in-state 1.66 True : family 4.74 True : gram1-adjective-to-adverb 3.02 False : gram2-opposite 2.22 True : gram3-comparative 4.73 True : gram4-superlative 2.76 True : gram5-present-participle 4.73 False : gram6-nationality-adjective 1.69 False : gram7-past-tense 2.44 True : gram8-plural 2.63 True : gram9-plural-verbs 2.07 True Wiki 5% Wiki 5% vector_size=300 diff \ Overall 46.12 57.44 11.32 Semantic 55.35 68.54 13.19 Syntactic 38.45 48.22 9.76 : capital-common-countries 81.82 95.45 13.64 : capital-world 69.83 79.55 9.73 : currency 7.62 7.62 0.00 : city-in-state 37.37 63.15 25.78 : family 68.77 73.72 4.94 : gram1-adjective-to-adverb 12.70 13.51 0.81 : gram2-opposite 11.82 14.90 3.08 : gram3-comparative 44.14 64.11 19.97 : gram4-superlative 17.02 26.74 9.71 : gram5-present-participle 29.92 38.35 8.43 : gram6-nationality-adjective 78.42 86.05 7.63 : gram7-past-tense 44.29 49.81 5.51 : gram8-plural 38.59 54.28 15.69 : gram9-plural-verbs 37.82 52.53 14.71 noise (max-min of 4 seeds) beyond noise? Overall 1.05 True Semantic 1.80 True Syntactic 1.26 True : capital-common-countries 2.77 True : capital-world 2.39 True : currency 1.50 False : city-in-state 1.66 True : family 4.74 True : gram1-adjective-to-adverb 3.02 False : gram2-opposite 2.22 True : gram3-comparative 4.73 True : gram4-superlative 2.76 True : gram5-present-participle 4.73 True : gram6-nationality-adjective 1.69 True : gram7-past-tense 2.44 True : gram8-plural 2.63 True : gram9-plural-verbs 2.07 True ########## cell 46 (exec 37) :: # 第 5 題 (d)：前 k 名有猜中就算對，正確率會變多少？（老師說 queen 可能不是第一名，而是第二、三名） top-1 top-3 top-5 top-10 GloVe (pre-trained) 63.11 73.68 77.73 82.01 Wiki 20% 48.39 61.53 66.32 71.74 ########## cell 47 (exec 38) :: # 第 5 題 (e)：錯題分析——電腦答錯時，它到底猜了什麼？ wrong answers: 10087 of 19544 OOV (question word not in vocab): 0 gold answer was 2nd-10th: 4563 : capital-world Question gold pred rank 508 Abuja Nigeria Amman Jordan jordan kuwait NaN 509 Abuja Nigeria Ankara Turkey turkey sakarya 3.0 510 Abuja Nigeria Antananarivo Madagascar madagascar senegal 8.0 511 Abuja Nigeria Apia Samoa samoa tonga 2.0 513 Abuja Nigeria Asmara Eritrea eritrea indonesia 2.0 516 Abuja Nigeria Baghdad Iraq iraq syria 2.0 : currency Question gold pred rank 5030 Algeria dinar Angola kwanza kwanza centavos NaN 5032 Algeria dinar Armenia dram dram hryvnia NaN 5033 Algeria dinar Brazil real real peso NaN 5034 Algeria dinar Bulgaria lev lev hryvnia NaN 5035 Algeria dinar Cambodia riel riel kyat NaN 5036 Algeria dinar Canada dollar dollar loonie NaN : family Question gold pred rank 8366 boy girl father mother mother stepmother 2.0 8376 boy girl nephew niece niece granddaughter 6.0 8391 brother sister groom bride bride bridesmaid 2.0 8394 brother sister husband wife wife widowed 4.0 8396 brother sister man woman woman sexless 10.0 8397 brother sister nephew niece niece aunt NaN : gram3-comparative Question gold pred rank 10677 bad worse cool cooler cooler fresher 6.0 10682 bad worse great greater greater britain 7.0 10683 bad worse hard harder harder aisam 3.0 10686 bad worse hot hotter hotter peaking NaN 10687 bad worse large larger larger smaller 2.0 10688 bad worse long longer longer shorter 4.0 : gram6-nationality-adjective Question gold pred \ 14183 Albania Albanian Argentina Argentinean argentinean argentinian 14186 Albania Albanian Belarus Belorussian belorussian belarusian 14199 Albania Albanian Greece Greek greek cypriot 14217 Albania Albanian Slovakia Slovakian slovakian slovak 14224 Argentina Argentinean Belarus Belorussian belorussian russian 14229 Argentina Argentinean China Chinese chinese guangdong rank 14183 8.0 14186 NaN 14199 2.0 14217 3.0 14224 NaN 14229 2.0 : gram8-plural Question gold pred rank 17342 banana bananas bird birds birds waterfowl 8.0 17344 banana bananas building buildings buildings renovating 2.0 17345 banana bananas car cars cars truck 2.0 17346 banana bananas cat cats cats dogs 9.0 17347 banana bananas child children children childbearing 5.0 17348 banana bananas cloud clouds clouds lidar NaN ########## cell 48 (exec 39) :: # 第 5 題 (f)：換一種「算答案」的公式。3CosAdd 是 b - a + c；3CosMul 用乘除。 Wiki 20% (3CosMul): overall 44.66% (OOV questions 0) Wiki 20% Wiki 20% (3CosMul) Overall 48.39 44.66 Semantic 56.70 54.12 Syntactic 41.48 36.80 : capital-common-countries 77.87 77.27 : capital-world 72.81 69.30 : currency 10.28 9.35 : city-in-state 35.51 33.73 : family 74.31 71.34 : gram1-adjective-to-adverb 15.93 12.00 : gram2-opposite 16.01 13.18 : gram3-comparative 49.10 43.62 : gram4-superlative 24.33 22.19 : gram5-present-participle 31.34 24.43 : gram6-nationality-adjective 80.49 76.05 : gram7-past-tense 43.85 39.55 : gram8-plural 41.67 34.23 : gram9-plural-verbs 40.92 37.36 OOV questions 0.00 0.00 ########## cell 49 (exec 40) :: # 第 5 題 (g)：同一批 family 字，用 PCA 和 t-SNE 兩種方法壓成 2 維，圖長得不一樣嗎？ [image cell49_0.png] ########## cell 50 (exec 41) :: # 第 5 題 (h)：各小類「四個字都在字典裡」的比例：GloVe vs Wiki 20% GloVe Wiki 20% SubCategory : capital-common-countries 100.0 100.0 : capital-world 100.0 100.0 : currency 100.0 100.0 : city-in-state 100.0 100.0 : family 100.0 100.0 : gram1-adjective-to-adverb 100.0 100.0 : gram2-opposite 100.0 100.0 : gram3-comparative 100.0 100.0 : gram4-superlative 100.0 100.0 : gram5-present-participle 100.0 100.0 : gram6-nationality-adjective 100.0 100.0 : gram7-past-tense 100.0 100.0 : gram8-plural 100.0 100.0 : gram9-plural-verbs 100.0 100.0 ########## cell 52 (exec 42) :: # 全部結果總表（報告引用的數字都來自這裡） GloVe (pre-trained) Wiki 20% \ Overall 63.11 48.39 Semantic 65.34 56.70 Syntactic 61.26 41.48 : capital-common-countries 93.87 77.87 : capital-world 88.95 72.81 : currency 14.20 10.28 : city-in-state 30.81 35.51 : family 81.62 74.31 : gram1-adjective-to-adverb 24.40 15.93 : gram2-opposite 20.07 16.01 : gram3-comparative 79.13 49.10 : gram4-superlative 54.28 24.33 : gram5-present-participle 69.51 31.34 : gram6-nationality-adjective 87.87 80.49 : gram7-past-tense 55.45 43.85 : gram8-plural 72.00 41.67 : gram9-plural-verbs 58.39 40.92 OOV questions 0.00 0.00 Forum (same size) Wiki control (same size) \ Overall 25.78 41.46 Semantic 12.84 45.77 Syntactic 36.53 37.87 : capital-common-countries 33.60 76.48 : capital-world 12.51 54.00 : currency 1.27 6.93 : city-in-state 2.92 33.81 : family 63.24 66.21 : gram1-adjective-to-adverb 13.10 10.28 : gram2-opposite 14.04 8.87 : gram3-comparative 64.94 53.08 : gram4-superlative 40.55 17.20 : gram5-present-participle 51.33 31.82 : gram6-nationality-adjective 21.70 76.55 : gram7-past-tense 33.78 40.96 : gram8-plural 41.97 36.26 : gram9-plural-verbs 41.49 32.99 OOV questions 3510.00 169.00 Wiki 5% Wiki 10% Wiki 5% seed=1 \ Overall 46.12 47.15 46.05 Semantic 55.35 55.60 54.18 Syntactic 38.45 40.13 39.30 : capital-common-countries 81.82 76.48 80.04 : capital-world 69.83 71.07 67.44 : currency 7.62 9.58 7.39 : city-in-state 37.37 35.47 38.27 : family 68.77 73.32 67.39 : gram1-adjective-to-adverb 12.70 10.99 11.29 : gram2-opposite 11.82 14.29 10.84 : gram3-comparative 44.14 46.92 46.10 : gram4-superlative 17.02 22.37 17.83 : gram5-present-participle 29.92 34.28 29.73 : gram6-nationality-adjective 78.42 76.11 80.11 : gram7-past-tense 44.29 44.87 44.81 : gram8-plural 38.59 40.77 40.62 : gram9-plural-verbs 37.82 41.49 39.77 OOV questions 107.00 0.00 107.00 Wiki 5% seed=2 Wiki 5% seed=3 \ Overall 47.10 46.34 Semantic 55.98 54.72 Syntactic 39.72 39.37 : capital-common-countries 82.81 81.82 : capital-world 69.43 68.21 : currency 8.89 8.31 : city-in-state 39.04 37.74 : family 72.13 69.17 : gram1-adjective-to-adverb 14.21 11.19 : gram2-opposite 11.08 13.05 : gram3-comparative 47.82 48.87 : gram4-superlative 18.98 19.79 : gram5-present-participle 31.53 34.47 : gram6-nationality-adjective 78.86 78.42 : gram7-past-tense 44.55 42.37 : gram8-plural 39.86 37.99 : gram9-plural-verbs 38.97 37.70 OOV questions 107.00 107.00 Wiki 5% (stop words removed) \ Overall 45.25 Semantic 55.41 Syntactic 36.81 : capital-common-countries 83.99 : capital-world 70.87 : currency 8.89 : city-in-state 37.82 : family 53.95 : gram1-adjective-to-adverb 12.80 : gram2-opposite 9.98 : gram3-comparative 37.01 : gram4-superlative 16.31 : gram5-present-participle 30.87 : gram6-nationality-adjective 80.61 : gram7-past-tense 41.09 : gram8-plural 41.89 : gram9-plural-verbs 26.55 OOV questions 284.00 Wiki 5% CBOW (sg=0) Wiki 5% window=10 \ Overall 52.58 41.75 Semantic 58.60 51.22 Syntactic 47.58 33.87 : capital-common-countries 79.45 72.53 : capital-world 71.86 68.24 : currency 8.43 7.85 : city-in-state 43.66 31.01 : family 77.87 50.59 : gram1-adjective-to-adverb 19.15 11.09 : gram2-opposite 11.45 6.90 : gram3-comparative 69.14 32.88 : gram4-superlative 37.61 9.80 : gram5-present-participle 40.81 27.46 : gram6-nationality-adjective 76.61 78.80 : gram7-past-tense 46.03 39.49 : gram8-plural 49.55 33.56 : gram9-plural-verbs 48.16 33.22 OOV questions 107.00 107.00 Wiki 5% vector_size=300 Wiki 20% (3CosMul) Overall 57.44 44.66 Semantic 68.54 54.12 Syntactic 48.22 36.80 : capital-common-countries 95.45 77.27 : capital-world 79.55 69.30 : currency 7.62 9.35 : city-in-state 63.15 33.73 : family 73.72 71.34 : gram1-adjective-to-adverb 13.51 12.00 : gram2-opposite 14.90 13.18 : gram3-comparative 64.11 43.62 : gram4-superlative 26.74 22.19 : gram5-present-participle 38.35 24.43 : gram6-nationality-adjective 86.05 76.05 : gram7-past-tense 49.81 39.55 : gram8-plural 54.28 34.23 : gram9-plural-verbs 52.53 37.36 OOV questions 107.00 0.00 ########## cell 54 (exec 43) :: import sys, platform, gensim, sklearn Python 3.12.3 (tags/v3.12.3:f6650f9, Apr 9 2024, 14:05:25) [MSC v.1938 64 bit (AMD64)] Windows-11-10.0.26200-SP0 | Intel64 Family 6 Model 141 Stepping 1, GenuineIntel gensim 4.4.0 | numpy 2.5.3 | pandas 3.0.6 | scikit-learn 1.9.1