Week 8. Feb. 28: Auto-encoders, Network & Table Learning - Possibilities #19
Comments
Data Sources:
Analysis Plan: |
|
Perozzi, B., Al-Rfou, R., & Skiena, S. (2014, August). Deepwalk: Online learning of social representations. In Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining (pp. 701-710). DeepWalk introduces a novel method for learning latent representations of network nodes by leveraging truncated random walks and applying unsupervised learning techniques inspired by language modeling. By treating random walks as sequences analogous to sentences, DeepWalk embeds graph structures into a continuous vector space, enabling improved performance on multi-label classification tasks, particularly in sparse-label environments. The approach significantly outperforms traditional network classification baselines such as spectral clustering and modularity-based methods, demonstrating up to 10% higher F1 scores while requiring substantially less labeled data. Additionally, DeepWalk is scalable and parallelizable, making it applicable to large social networks. This method could extend social science analysis by providing a scalable and effective way to model and analyze social influence, community structures, and opinion dynamics. By embedding individuals in a continuous space based on their network interactions, DeepWalk enables researchers to capture latent group memberships, detect ideological clusters, and predict behavioral trends without requiring explicit label supervision. Moreover, its adaptability to evolving networks allows for longitudinal studies of social change, polarization, and diffusion of beliefs. To pilot this approach in social science, I would use the World Values Survey (WVS) Wave 7 dataset, which captures social attitudes, political beliefs, and demographic characteristics of individuals across diverse countries. By constructing a network graph where nodes represent respondents and edges capture ideological similarity (based on cosine similarity of responses or self-reported social ties), DeepWalk could generate embeddings reflecting ideological proximity. These embeddings could then be used to predict opinion shifts, polarization, and demographic correlates of belief structures, providing a richer understanding of how social attitudes cluster and evolve over time. |
|
Djeddou, Messaoud, Jehad Al Dallal, Aouatef Hellal, Ibrahim A. Hameed, and Xingang Zhao. 2024. “Hybrid Deep Convolutional Neural Networks Combined with Autoencoders and Augmented Data to Predict the Look-Up Table 2006.” Pp. 1–5 in 2024 International Conference on Decision Aid Sciences and Applications (DASA). 1. Summary of the Article 2. Extending Social Science Analysis 3. Social Data for Pilot Implementation |
|
https://arxiv.org/abs/2407.03568
To illustrate the latter of the two ideas - I think social media data like in this paper would suffice, although the tag would have to be the list of items purchased instead of MBTI or other personality traits measures. The researcher could go on and create a detailed profile of each person with LLM, create embeddings, builda hypergraph like this paper, train the model on whether the item of interest was in the purchased item of the user, and test the model efficiency on the test data. |
|
Detecting coalitions by optimally partitioning signed networks of political collaboration Aref and Neal (2020) propose a mathematical optimization model for partitioning signed political collaboration networks into cohesive coalitions. Their approach minimizes intra-group negative and inter-group positive edges, solving the NP-hard problem of optimal partitioning using bounds and linear programming techniques. They apply this model to the U.S. Congress from 1979 to 2016, demonstrating that increasing polarization, as reflected in partisan coalitions, has been a protective factor in maintaining legislative effectiveness. Their findings challenge the assumption that polarization necessarily reduces legislative productivity, instead showing that ideologically homogeneous coalitions can streamline bill passage in the House of Representatives. This method provides a powerful tool for analyzing political and social networks beyond legislative contexts. It could be applied to other domains where polarization and coalition formation play a role, such as corporate governance (e.g., boardroom alliances), international relations (e.g., diplomatic blocs in the United Nations), or social media discourse (e.g., ideological echo chambers). By identifying cohesive opposition groups, we can assess the impact of factionalization on decision-making efficiency, cooperation, and conflict resolution. Additionally, this approach could be integrated into research on organizational behavior, identifying workplace factions that impact collaboration or conflict, or in studies of activist movements, determining how ideological divides influence the effectiveness of policy advocacy. In particular, this research highlights the potential of integrating the notion of signed network into the analysis of elite institutions. This conceptual construct can be easily adapted to the foreign policy decision-making scenario, where groups of policymakers constitute a signed elite network where nodes represent individuals and edge weights indicate homogeneity in policy positions. Signed graph embedding might offer a viable measure to the potential influential power of an individual, for instance, a respectful advisor who enjoys many positive in-degrees might be particularly persuasive in the decision-making process. Additionally, the methodology proposed in this paper can also be possibly used to identify coalitions in a foreign policy administration. The presence of a coalition at a policy meeting might find it easier to steer the policy decision in their own favor. |
|
Can Graphical Neural Networks Learn Language with Extremely Weak Text Supervision? The paper explores the challenge of adapting Graph Neural Networks (GNNs) to learn from language with extremely weak text supervision. While contrastive learning models such as CLIP have shown success in vision-language alignment, GNNs struggle due to data scarcity, diverse task levels, and conceptual mismatches across domains. The authors introduce Morpher, a novel multi-modal prompt learning approach that aligns pre-trained GNN representations with Large Language Model (LLM) embeddings without fine-tuning either model. Morpher employs a graph-text multi-modal prompt learning paradigm, leveraging cross-modal projectors to map graph embeddings into the LLM’s semantic space. The paper uses network data in Chemistry and Biology to demonstrate the effectiveness of their method. The paper offers several applications for social science research, particularly in network analysis, digital humanities, and computational sociology. In particular, if we were to consider data such as political discourse networks (e.g.,legislative voting records, citations in legal texts, or Twitter interactions among policymakers), we could be embedded the data in the same space as language models trained on political theory texts. This would enable semantic comparisons between network clusters and ideological constructs—helping researchers automatically classify emerging political alliances or detect shifts in ideological positioning, which provide a better platform to contextualize where the data is coming from. For the goal above social network data (such as connections in political parties or social connections on social media) might be a good start, and use pre-trained LLM models familiar with the political landscape to support the task. |
|
https://arxiv.org/html/2402.17905v1 The article "Using Graph Neural Networks to Predict Local Culture" by Thiago H. Silva and Daniel Silver explores the application of GNNs to model and predict cultural attributes of urban neighborhoods. The authors address the challenge of understanding the dynamic and interconnected nature of neighborhoods by integrating multiple data sources, including demographic information, historical data, and movement patterns of groups between areas. Utilizing a large-scale dataset from Yelp, they demonstrate that GNNs can effectively capture the structural connectedness of neighborhoods, leading to accurate predictions of local cultural characteristics. Notably, their findings suggest that both local demographic data and group profiles derived from user reviews are equally effective in predicting cultural attributes. This highlights the potential of leveraging automatically extracted online data, especially in scenarios where traditional local information is scarce or unavailable. The methodology presented in this study offers a valuable extension to social science research by providing a framework to quantitatively analyze the relational dynamics of neighborhoods. Traditional social science approaches often face limitations in capturing the complex interactions and temporal evolutions within urban settings due to data and methodological constraints. By employing GNNs, researchers can model neighborhoods as interconnected entities, allowing for the incorporation of both spatial and temporal data. This approach enables the examination of how movements of people, businesses, and ideas influence cultural and social transformations within urban areas. To pilot this approach, one could utilize data from platforms like Yelp, which provide user-generated reviews and check-in information for various venues across different neighborhoods. This data can be enriched with demographic and socio-economic information from census databases to create a multifaceted representation of each neighborhood. |
|
Exploring Auto-Encoders, Network Learning, and Table Learning in Social Science For this week's "possibility" reading, I examined an article that explores how auto-encoders, network learning, and table learning can be applied to social science research. The paper highlights the use of auto-encoders for dimensionality reduction, table learning for structured data analysis, and network learning for uncovering relationships in complex systems like social networks or cultural shifts. Key applications in social science include:
Potential data sources for applying these methods include:
By integrating network learning for relationships and table learning for structured data, these methods offer new perspectives on complex social behaviors. |
|
I have selected the paper titled "RecVAE: a New Variational Autoencoder for Top-N Recommendations with Implicit Feedback" for reflection. Reflection: The paper "RecVAE: a New Variational Autoencoder for Top-N Recommendations with Implicit Feedback" introduces an innovative approach to enhancing recommendation systems using variational autoencoders (VAEs). Traditional recommendation systems often struggle with effectively capturing user preferences, especially when dealing with implicit feedback data. This study addresses these challenges by proposing RecVAE, a model that incorporates several novel techniques to improve recommendation accuracy. One of the standout features of RecVAE is the introduction of a composite prior distribution for the latent codes. This approach allows the model to better capture the complex patterns in user-item interactions, leading to more personalized recommendations. Additionally, the authors present a new method for setting the β hyperparameter within the β-VAE framework, which balances the trade-off between reconstruction accuracy and the capacity of the latent space. This careful tuning is crucial for modeling the nuances of user behavior. The paper also details a unique training methodology based on alternating updates, which enhances the model's convergence and performance. Through extensive experiments on classical collaborative filtering datasets, RecVAE demonstrated significant improvements over existing autoencoder-based models, including Mult-VAE and RaCT. The comprehensive ablation study provided offers valuable insights into the contributions of each component of the proposed model. In summary, this paper contributes to the field of recommendation systems by leveraging advanced VAE techniques to address the limitations of previous models. The proposed RecVAE model showcases the potential of combining deep learning architectures with probabilistic modeling to achieve state-of-the-art performance in top-N recommendations with implicit feedback. |
|
“Why do tree-based models still outperform deep learning on typical tabular data?” 2022. Grinsztajn, L., Oyallon, E., & Varoquaux, G. Advances in neural information processing systems, 35, 507-520. This paper tries to understand why tree-based models perform better than deep learning on tabular data by first, establish a benchmark of 45 datasets for tabular learning and second, empirically investigate the different inductive biases of tree-based models and neural networks. The inductive biases tested and their results are: 1) when smoothing the target function, tree-based models’ performance decrease, which suggests that their performance are based on the inductive bias produced by a non-smooth target function, while neural networks would struggle to fit a non-smooth function; 2) by adding and removing uninformative features, the authors find that Multi-Layer Perceptron (MLP) like architectures are not robust to uninformative features; 3) another difference between tree-based models and neural networks is that in tree-based models are not rotationally invariant, which suits the properties of tabular data. The findings of this research might provide useful guidances for tabular deep-learning research. I am curious how can further development in neural networks potentially make it achieve better performance than tree-based models, and what limitations of tree-based models can possibly be overcome by neural networks. It also seems to me intuitively that the structures of tree-based models and neural networks simulate different human cognitive functions, namely classification and understanding. While the former focuses on setting boundaries, the latter emphasizes finding learnable patterns in the data. I am curious about testing this hypothesis empirically. For example, we can find datasets that researchers traditionally used to do “understanding”, or pattern extraction tasks, such as text data, and transform them into tabular data and compare the performance of tree-based models and neural networks on these datasets, roughly in the same way provided by this paper. The results should help us understand different cognition patterns that people intuitively choose in different cognition tasks. |
|
“Disrupted routines anticipate musical exploration Links to an external site.,” 2024. Kim, Khwan, Noah Askin, and James A. Evans. PNAS. The study reveals how cultural tastes evolve as individuals’ experiences change, especially when daily life is disrupted. Analyzing millions of global music streaming plays, the study found that individuals’ music preferences diversified after traveling or experiencing events that disrupt daily routines, such as COVID-19 lockdowns. Travelers’ music choices align with the culture of the destination, while pandemic lockdowns have led people to prefer regional music rather than global average music preferences. The effects of this exploratory consumption are long-lasting, suggesting that music tastes reflect rather than compensate for changes in life, shaping long-term cultural consumption patterns. Extending social science analysis: Possible dataset: |
|
A Novel Deep Auto-Encoder Based Linguistics Clustering Model for Social Text This paper presents a novel deep auto-encoder based approach for clustering short text in Urdu, a widely spoken but computationally under-resourced language. The authors created a standard dataset of 137,181 Urdu news headlines from multiple categories and developed a method that automatically learns feature representations while simultaneously determining cluster assignments. Their approach uses a deep neural network to transform high-dimensional textual features into a lower-dimensional space where clustering can be performed more effectively. The model iteratively refines cluster centroids by optimizing a KL divergence loss between soft assignments and an auxiliary target distribution. Through extensive experiments on Urdu news headlines, they demonstrated that their deep learning approach significantly outperforms traditional clustering methods like K-means, achieving 67.7% accuracy compared to 39.5% with standard methods. The work addresses the particular challenges of short text clustering, where limited contextual clues and sparse features make traditional clustering approaches less effective. This deep auto-encoder based clustering approach could be extended to analyze cross-cultural communication patterns in multilingual social media environments. The method's ability to learn representations directly from text data without human-engineered features makes it particularly valuable for comparative analysis of discourse across languages with different structural properties. This could help detect emerging cross-cultural narratives, identify culturally-specific framing of global events, and trace how ideas evolve as they move between linguistic communities. To pilot the deep auto-encoder based clustering model for analyzing cross-national narratives about LLMs, I would use a comprehensive multilingual dataset of public discourse collected from multiple sources. The dataset would comprise social media posts, news articles, and public comments about large language models gathered from five major language communities: English, Mandarin, Spanish, Arabic, and Japanese. |
|
Sparse Auto-Encoder Interprets Linguistic Features in Large Language Models This paper explores how large language models (LLMs) encode and process linguistic knowledge by using Sparse Autoencoders (SAEs) to extract interpretable linguistic features. The study proposes SAELING, a framework that systematically decomposes LLMs’ hidden states into sparse high-dimensional representations, capturing distinct linguistic features across phonetics, phonology, morphology, syntax, semantics, and pragmatics. Through causal intervention experiments, the study introduces two key evaluation metrics—Feature Representation Confidence (FRC) and Feature Intervention Confidence (FIC)—which assess how well these extracted features correspond to actual linguistic structures and whether modifying them can steer model outputs. Their findings suggest that LLMs encode genuine structured linguistic knowledge and that targeted interventions can effectively manipulate model behavior. I am also wondering what we can infer from their finding of the presence of inherent linguistic knowledge within LLMs? Their study can advance cognitive science research by uncovering how linguistic features are encoded and processed in LLMs, offering insights into language comprehension and production. For instance, it can be used to examine how metaphor, syntax, or pragmatics emerge in neural representations, aligning with theories of cognitive architecture and linguistic cognition. By intervening in model activations, researchers can simulate controlled experiments to test hypotheses about how language structure influences meaning-making and decision-making. A cognitive science pilot could analyze how metaphor processing differs between LLMs and human cognition. Using datasets of human-written metaphors (e.g., from literature, news, or experimental studies), we could extract metaphor-related features via SAELING and compare them to neural activation patterns in fMRI or EEG studies on metaphor comprehension. This approach would help evaluate whether LLMs represent metaphors in ways that align with human cognitive processing and identify key differences in abstraction and context integrations. |
Post a link for a "possibility" reading of your own on the topic of Auto-encoders, Network & Table Learning [Week 8], accompanied by a 300-400 word reflection that:
The text was updated successfully, but these errors were encountered: