Deep Dynamic Language Models

Abstract

This thesis investigates the domain of deep dynamic language models, focusing on the integration of temporal dynamics to enhance language modeling and its application in various tasks, such as text generation, recommendation systems, and predicting post popularity. Temporal content change, i.e., trends and themes that change with time featured in document collections such as academic journals, news articles and social media, make the traditional static language models (LMs) not an optimal solution. In order to address this limitation, several approaches to develop dynamic LMs are proposed and explored in this thesis. <br /> Initially, the impact of incorporating temporal information is explored, specifically in the context of modeling online communities. For the analysis of temporal content change in Yelp — a crowd-sourced review platform — an instantaneous language model is proposed. This model combines a temporal point process (TPP) for modeling review creation times and a LM to capture textual aspects. Empirical evaluations demonstrate that this model significantly improves the performance of LMs in terms of both language modeling and prediction of review creation time. <br /> Building upon the success of the instantaneous LM, the research in this thesis is extended to more application oriented task, such as recommender systems. Recognizing that user preferences and item reviews change over time, the proposed model here leverages users’ reviews to enhance rating predictions. By developing time-interval aware representations, the proposed model outperforms several state-of-the-art recommender systems models in real-world datasets. <br /> Additionally, the integration of dynamic topic models into LMs is explored. First, the problem of skewed topic distributions in topic modeling is addressed, which can cause models to learn more general topics present in the majority of documents, rather than rare topics present in only a few documents. A neural dynamic focused topic model is proposed as a solution, which decouples topic activities from topic proportions in documents using sequences of Bernoulli random variables. Experimental evaluations show that this model outperforms state-of-the-art topic models in generalization tasks, while employing a comparable number of parameters and converging two times faster. <br /> Furthermore, the performance of large pre-trained language models (LPLMs) in dynamic environments is explored. The empirical analysis on Reddit datasets reveals significant performance drops when predicting the popularity of future posts due to temporal distribution shifts in data. To mitigate this issue, a model is proposed that combines neural variational dynamic topic models and attention mechanisms to infer temporal LM representations. The proposed model exhibit improved performance while utilizing only a fraction of the parameters of LPLMs, and providing interpretable representations that offer insights into real-world events. <br /> In summary, this thesis emphasizes the significance of incorporating temporal dynamics into LMs and explores their application in various tasks.