請用此 Handle URI 來引用此文件：
http://tdr.lib.ntu.edu.tw/jspui/handle/123456789/92873完整後設資料紀錄
| DC 欄位 | 值 | 語言 |
|---|---|---|
| dc.contributor.advisor | 蘇黎 | zh_TW |
| dc.contributor.advisor | Li Su | en |
| dc.contributor.author | 彭約博 | zh_TW |
| dc.contributor.author | Yueh-Po Peng | en |
| dc.date.accessioned | 2024-07-02T16:23:37Z | - |
| dc.date.available | 2024-07-03 | - |
| dc.date.copyright | 2024-07-02 | - |
| dc.date.issued | 2024 | - |
| dc.date.submitted | 2024-07-01 | - |
| dc.identifier.citation | [1] Tomoya Nakai, Naoko Koide-Majima, and Shinji Nishimoto. Music genre neu roimaging dataset. Data in Brief, 40:107675, 2022.
[2] Tomoya Nakai, Naoko Koide-Majima, and Shinji Nishimoto. Correspondence of categorical and feature-based representations of music in the human brain. Brain and behavior, 11(1):e01936, 2021. [3] Sean Paulsen and Michael Casey. Self-supervised pretraining on paired sequences of fmri data for transfer learning to brain decoding tasks, 2023. [4] Armin Thomas, Christopher Ré, and Russell Poldrack. Self-supervised learning of brain dynamics from broad neuroimaging data. Advances in neural information processing systems, 35:21255–21269, 2022. [5] Josue Ortega Caro, Antonio Henrique Oliveira Fonseca, Christopher Averill, Syed A Rizvi, Matteo Rosati, James L Cross, Prateek Mittal, Emanuele Zappala, Daniel Levine, Rahul M Dhodapkar, et al. Brainlm: A foundation model for brain activity recordings. bioRxiv, pages 2023–09, 2023. [6] Chenwei Shi, Yanming Wang, Yueyang Wu, Shishuo Chen, Rongjie Hu, Min Zhang, Bensheng Qiu, and Xiaoxiao Wang. Self-supervised pretraining improves the performance of classification of task functional magnetic resonance imaging. Frontiers in Neuroscience, 17:1199312, 2023. [7] Peter Kim, Junbeom Kwon, Sunghwan Joo, Sangyoon Bae, Donggyu Lee, Yoonho Jung, Shinjae Yoo, Jiook Cha, and Taesup Moon. Swift: Swin 4d fmri transformer. Advances in Neural Information Processing Systems, 36, 2024. [8] John-Dylan Haynes. A primer on pattern-based approaches to fmri: principles, pit falls, and perspectives. Neuron, 87(2):257–270, 2015. [9] John-Dylan Haynes and Geraint Rees. Decoding mental states from brain activity in humans. Nature reviews neuroscience, 7(7):523–534, 2006. [10] Bing Du, Xiaomu Cheng, Yiping Duan, and Huansheng Ning. fmri brain decod ing and its applications in brain–computer interface: A survey. Brain Sciences, 12(2):228, 2022. [11] Simon B Eickhoff, BT Yeo, and Sarah Genon. Imaging-based parcellations of the human brain. Nature Reviews Neuroscience, 19(11):672–686, 2018. [12] Stefan Czoschke, Cora Fischer, Tara Bahador, Christoph Bledowski, and Jochen Kaiser. Decoding concurrent representations of pitch and location in auditory work ing memory. Journal of Neuroscience, 41(21):4658–4666, 2021. [13] Kelly H Chang, Jessica M Thomas, Geoffrey M Boynton, and Ione Fine. Re constructing tone sequences from functional magnetic resonance imaging blood oxygen level dependent responses within human primary auditory cortex. Frontiers in Psychology, 8:1983, 2017. [14] Michael A Casey. Music of the 7ts: Predicting and decoding multivoxel fmri responses with acoustic, schematic, and categorical music features. Frontiers in psychology, 8:1179, 2017. [15] Sébastien Paquette, Sylvain Takerkart, Shinji Saget, Isabelle Peretz, and Pascal Be lin. Cross-classification of musical and vocal emotions in the auditory cortex. Annals of the New York Academy of Sciences, 1423(1):329–337, 2018. [16] Hedwig Eisenbarth, Luke J Chang, and Tor D Wager. Multivariate brain prediction of heart rate and skin conductance responses to social threat. Journal of Neuroscience, 36(47):11987–11998, 2016. [17] Stefan Koelsch, Vincent KM Cheung, Sebastian Jentschke, and John-Dylan Haynes. Neocortical substrates of feelings evoked with music in the acc, insula, and so matosensory cortex. Scientific reports, 11(1):1–11, 2021. [18] Stephen José Hanson and Yaroslav O Halchenko. Brain reading using full brain support vector machines for object recognition: there is no “face” identification area. Neural Computation, 20(2):486–503, 2008. [19] Jesse Engel, Cinjon Resnick, Adam Roberts, Sander Dieleman, Mohammad Norouzi, Douglas Eck, and Karen Simonyan. Neural audio synthesis of musical notes with WaveNet autoencoders. In Doina Precup and Yee Whye Teh, editors, Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pages 1068–1077. PMLR, 06–11 Aug 2017. [20] William D Penny, Karl J Friston, John T Ashburner, Stefan J Kiebel, and Thomas E Nichols. Statistical parametric mapping: the analysis of functional brain images. Elsevier, 2011. [21] Jacob S Prince, John A Pyles, Michael J Tarr, and Kendrick N Kay. Glmsingle: a turnkey solution for accurate single-trial fmri response estimates. Journal of Vision, 21(9):2831–2831, 2021. [22] Scott M Lundberg and Su-In Lee. A unified approach to interpreting model predic tions. Advances in neural information processing systems, 30, 2017. [23] Leo Breiman. Random forests. Machine learning, 45(1):5–32, 2001. [24] Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278–2324, 1998. [25] F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blon del, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825–2830, 2011. [26] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556, 2014. [27] BT Thomas Yeo, Fenna M Krienen, Jorge Sepulcre, Mert R Sabuncu, Danial Lashkari, Marisa Hollinshead, Joshua L Roffman, Jordan W Smoller, Lilla Zöllei, Jonathan R Polimeni, et al. The organization of the human cerebral cortex estimated by intrinsic functional connectivity. Journal of neurophysiology, 2011. [28] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A sim ple framework for contrastive learning of visual representations. In International conference on machine learning, pages 1597–1607. PMLR, 2020. [29] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference on computer vision, pages 10012–10022, 2021. [30] Deanna M Barch, Gregory C Burgess, Michael P Harms, Steven E Petersen, Bradley L Schlaggar, Maurizio Corbetta, Matthew F Glasser, Sandra Curtiss, Sachin Dixit, Cindy Feldt, et al. Function in the human connectome: task-fmri and individ ual differences in behavior. Neuroimage, 80:169–189, 2013. [31] Maedbh King, Carlos R Hernandez-Castillo, Russell A Poldrack, Richard B Ivry, and Jörn Diedrichsen. Functional boundaries in the human cerebellum revealed by a multi-domain task battery. Nature neuroscience, 22(8):1371–1378, 2019. [32] Leland McInnes, John Healy, and James Melville. Umap: Uniform manifold ap proximation and projection for dimension reduction, 2020. [33] G. Tzanetakis and P. Cook. Musical genre classification of audio signals. IEEE Transactions on Speech and Audio Processing, 10(5):293–302, 2002. [34] Vincent K.M. Cheung, Yueh-Po Peng, Jing-Hua Lin, and Li Su. Decoding musical pitch from human brain activity with automatic voxel-wise whole-brain fmri feature selection. In ICASSP 2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1–5, 2023. [35] Tomoya Nakai, Naoko Koide-Majima, and Shinji Nishimoto. Encoding and decod ing of music-genre representations in the human brain. In 2018 IEEE International Conference on Systems, Man, and Cybernetics (SMC), pages 584–589. IEEE, 2018. [36] Bob L Sturm. The gtzan dataset: Its contents, its faults, their effects on evaluation, and its future use. arXiv preprint arXiv:1306.1461, 2013. [37] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 16000–16009, 2022. [38] Christoph Feichtenhofer, Haoqi Fan, Yanghao Li, and Kaiming He. Masked au toencoders as spatiotemporal learners. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022. [39] Zijiao Chen, Jiaxin Qing, Tiange Xiang, Wan Lin Yue, and Juan Helen Zhou. Seeing beyond the brain: Masked modeling conditioned diffusion model for human vision decoding. In arXiv, November 2022. [40] Emily J Allen, Ghislain St-Yves, Yihan Wu, Jesse L Breedlove, Jacob S Prince, Logan T Dowdle, Matthias Nau, Brad Caron, Franco Pestilli, Ian Charest, et al. A massive 7t fmri dataset to bridge cognitive neuroscience and artificial intelligence. Nature neuroscience, 25(1):116–126, 2022. | - |
| dc.identifier.uri | http://tdr.lib.ntu.edu.tw/jspui/handle/123456789/92873 | - |
| dc.description.abstract | 腦部解碼旨在解釋神經活動以理解大腦功能。本論文探討了各種全腦特徵選擇方法及先進深度學習技術在功能性磁共振成像（fMRI）數據中的應用，旨在提高腦部解碼能力。我們提出了一種新的兩階段特徵選擇方法，將大腦體積分成非重疊的立方體，有效識別高貢獻的體素。這種方法顯著提高了來自個體內數據的音高解碼準確性。
利用自監督學習，我們運用對比學習在人類連接組計劃（HCP）中的多個任務fMRI資料集上，以學習可轉移的特徵。我們的方法在解碼各種心理狀態方面顯示出顯著改進，相較於現有方法，尤其在數據有限的情況下，我們的方法與稀疏字典方法比較下具有競爭力。通過在預訓練階段使用多樣的任務fMRI資料，我們提升了下游任務的性能。 此外，我們探討了學習到的特徵在類似條件下的可轉移性，例如解碼音樂類型。我們的實驗顯示，任務相關的預訓練在數據有限的情況下更加高效和有效。我們發現，古典音樂和嘻哈音樂更易區分，這可能是由於音樂片段中的歌詞影響了語言處理區域。雖然我們的方法受益於多個受試者數據的聚合，傳統特徵選擇方法在某些任務中仍具有優勢。總的來說，本論文強調了使用全腦fMRI數據和自監督學習在推進神經影像分析和解碼複雜神經活動方面的潛力。 | zh_TW |
| dc.description.abstract | Brain decoding seeks to interpret neural activity to identify mental states. This thesis explores various whole-brain feature selection methods and the application of advanced deep learning techniques to enhance brain decoding capabilities using functional magnetic resonance imaging (fMRI) data. We propose a novel two-stage feature selection method that divides brain volumes into non-overlapping cubes, effectively identifying high-contribution voxels. This approach significantly improves decoding accuracy for musical pitch information from within-subject data.
Leveraging self-supervised learning, we apply contrastive learning to multiple task fMRI datasets from the Human Connectome Project (HCP) to learn transferable features. Our method demonstrates substantial improvements in decoding various mental states compared to existing approaches and proves competitive with sparse dictionary methods, particularly when data is limited. By using diverse task fMRI data during pretraining, we enhance downstream task performance. Additionally, we investigate the transferability of learned features to similar conditions, such as decoding musical genres. Our experiments reveal that task-related pretraining can be more efficient and effective, especially with limited data. We find that classical and hip-hop music are more distinguishable, likely due to the presence of lyrics affecting language processing regions. While our method benefits from aggregating data from multiple subjects, traditional feature selection methods still excel in certain tasks. Overall, this thesis underscores the potential of using whole-brain fMRI data and self-supervised learning to advance neuroimaging analysis and decode complex neural activities. | en |
| dc.description.provenance | Submitted by admin ntu (admin@lib.ntu.edu.tw) on 2024-07-02T16:23:37Z No. of bitstreams: 0 | en |
| dc.description.provenance | Made available in DSpace on 2024-07-02T16:23:37Z (GMT). No. of bitstreams: 0 | en |
| dc.description.tableofcontents | Acknowledgements i
摘要 ii Abstract iii Contents v List of Figures ix List of Tables xiii Chapter 1 Introduction 1 Chapter 2 Previous Work 4 2.1 ROI-based method . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 2.2 Parcellation-based method . . . . . . . . . . . . . . . . . . . . . . . 6 2.3 Whole-brain method . . . . . . . . . . . . . . . . . . . . . . . . . . 8 Chapter 3 fMRI Overview 10 3.1 Preprocessing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11 3.1.1 Correction for Slice Timing . . . . . . . . . . . . . . . . . . . . . . 11 3.1.2 Realignment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11 3.1.3 Segmentation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 3.1.4 Co-registration . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 3.1.5 Normalization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13 3.1.6 Smoothing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13 3.2 Advanced Data Processing . . . . . . . . . . . . . . . . . . . . . . . 14 3.2.1 Detrending . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14 3.2.2 Z-Normalization . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 3.2.3 Precision Reduction to fp16 . . . . . . . . . . . . . . . . . . . . . . 15 3.2.4 Storage as PyTorch Tensors (.pt Format) . . . . . . . . . . . . . . . 15 Chapter 4 Automatic fMRI Feature Selection for Decoding Musical Pitch 17 4.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18 4.2 Dataset . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19 4.2.1 Experimental stimuli and fMRI data acquisition . . . . . . . . . . . 19 4.2.2 fMRI data preprocessing . . . . . . . . . . . . . . . . . . . . . . . 20 4.3 Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21 4.3.1 Proposed approach . . . . . . . . . . . . . . . . . . . . . . . . . . 21 4.3.2 Feature importance . . . . . . . . . . . . . . . . . . . . . . . . . . 21 4.3.3 Experimental implementation . . . . . . . . . . . . . . . . . . . . . 22 4.4 Results and Discussion . . . . . . . . . . . . . . . . . . . . . . . . . 24 4.4.1 Cube-wise decoding . . . . . . . . . . . . . . . . . . . . . . . . . . 24 4.4.2 SHAP value-based voxel selection . . . . . . . . . . . . . . . . . . 26 4.4.3 Pooled decoding performance . . . . . . . . . . . . . . . . . . . . . 27 4.5 Conclusion and Future work . . . . . . . . . . . . . . . . . . . . . . 28 Chapter 5 Self-Supervised Pretraining for Decoding Mental State 29 5.1 Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30 5.1.1 Self-supervised pretraining . . . . . . . . . . . . . . . . . . . . . . 30 5.1.2 Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33 5.1.3 Data augmentation . . . . . . . . . . . . . . . . . . . . . . . . . . 36 5.2 Dataset . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38 5.2.1 Human Connectome Project (HCP) 3T task fMRI . . . . . . . . . . 38 5.2.1.1 Working memory task . . . . . . . . . . . . . . . . . . 38 5.2.1.2 Gambling task . . . . . . . . . . . . . . . . . . . . . . 39 5.2.1.3 Motor task . . . . . . . . . . . . . . . . . . . . . . . . 40 5.2.1.4 Language processing task . . . . . . . . . . . . . . . . 40 5.2.1.5 Social cognition task . . . . . . . . . . . . . . . . . . . 40 5.2.1.6 Relational processing task . . . . . . . . . . . . . . . . 41 5.2.1.7 Emotion processing task . . . . . . . . . . . . . . . . . 41 5.2.2 Multi-Domain Task Battery (MDTB) . . . . . . . . . . . . . . . . . 42 5.3 Experimental implementation . . . . . . . . . . . . . . . . . . . . . 47 5.3.1 Pretraining . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47 5.3.2 Transfer learning to held out HCP tasks . . . . . . . . . . . . . . . 48 5.3.3 Transfer learning to MDTB . . . . . . . . . . . . . . . . . . . . . . 48 5.4 Results and Discussion . . . . . . . . . . . . . . . . . . . . . . . . . 50 5.4.1 Transfer learning to held-out HCP tasks . . . . . . . . . . . . . . . 50 5.4.2 Transfer learning to MDTB . . . . . . . . . . . . . . . . . . . . . . 52 5.4.2.1 Impact of training data on performance metrics . . . . . 52 5.4.2.2 Comparison of proposed method with parcellation-based method . . . . . . . . . . . . . . . . . . . . . . . . . . 53 5.4.2.3 Confusion matrix analysis . . . . . . . . . . . . . . . . 56 5.4.3 Features learned from self-supervised learning . . . . . . . . . . . . 56 Chapter 6 Self-Supervised Pretraining for Decoding Musical Genre 65 6.1 Dataset . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66 6.2 Method and Experiments . . . . . . . . . . . . . . . . . . . . . . . . 67 6.3 Results and Discussion . . . . . . . . . . . . . . . . . . . . . . . . . 68 6.3.1 Analysis of Music Genre Decoding Performance . . . . . . . . . . 68 6.3.2 Confusion Matrix Analysis . . . . . . . . . . . . . . . . . . . . . . 70 6.3.3 Comparison of Different Baselines . . . . . . . . . . . . . . . . . . 72 Chapter 7 Discussion 74 7.1 General Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . 74 7.2 Limitations and Future Work . . . . . . . . . . . . . . . . . . . . . . 76 Chapter 8 Conclusion 79 References 81 Appendix A — Decoding Mental State 87 A.1 Confusion Matrix of MDTB . . . . . . . . . . . . . . . . . . . . . . 87 A.1.1 Fine-tuned with 1 subject and pretrained with different tasks . . . . 87 A.1.2 Fine-tuned with 11 subjects and pretrained with different tasks . . . 94 A.2 Repetition of Experiments for transfer learning to MDTB . . . . . . . 100 Appendix B — Decoding Musical Genre 102 B.1 Confusion Matrix of GTZAN . . . . . . . . . . . . . . . . . . . . . 102 | - |
| dc.language.iso | en | - |
| dc.subject | 腦部解碼 | zh_TW |
| dc.subject | 功能性磁振造影 | zh_TW |
| dc.subject | 特徵選擇 | zh_TW |
| dc.subject | 自監督式學習 | zh_TW |
| dc.subject | 對比學習 | zh_TW |
| dc.subject | self-supervised learning | en |
| dc.subject | brain decoding | en |
| dc.subject | fMRI | en |
| dc.subject | feature selection | en |
| dc.subject | contrastive learning | en |
| dc.title | 功能性磁振造影解碼之全腦特徵選擇方法 | zh_TW |
| dc.title | Whole-Brain Feature Selection Methods for Decoding from fMRI Data | en |
| dc.type | Thesis | - |
| dc.date.schoolyear | 112-2 | - |
| dc.description.degree | 碩士 | - |
| dc.contributor.coadvisor | 楊鈞澔 | zh_TW |
| dc.contributor.coadvisor | Chun-Hao Yang | en |
| dc.contributor.oralexamcommittee | 楊奕軒;張家銘 | zh_TW |
| dc.contributor.oralexamcommittee | Yi-Hsuan Yang;Vincent Ka Ming Cheung | en |
| dc.subject.keyword | 腦部解碼,功能性磁振造影,特徵選擇,自監督式學習,對比學習, | zh_TW |
| dc.subject.keyword | brain decoding,fMRI,feature selection,self-supervised learning,contrastive learning, | en |
| dc.relation.page | 107 | - |
| dc.identifier.doi | 10.6342/NTU202401348 | - |
| dc.rights.note | 同意授權(限校園內公開) | - |
| dc.date.accepted | 2024-07-02 | - |
| dc.contributor.author-college | 電機資訊學院 | - |
| dc.contributor.author-dept | 資料科學學位學程 | - |
| 顯示於系所單位： | 資料科學學位學程 | |
文件中的檔案：
| 檔案 | 大小 | 格式 | |
|---|---|---|---|
| ntu-112-2.pdf 授權僅限NTU校內IP使用（校園外請利用VPN校外連線服務） | 8.59 MB | Adobe PDF |
系統中的文件，除了特別指名其著作權條款之外，均受到著作權保護，並且保留所有的權利。