Week Ending 12.24.2023
RESEARCH WATCH: 12.24.2023
SPONSORED BY
Digimarc digital watermarks invisibly guard your digital assets to protect against misuse, prove copyright ownership, and verify authenticity. In an era of artificial intelligence, don’t leave your images and other digital content exposed. Demand superior content protection and maintain trust in your brand with Digimarc.
Checkout Digimarc - https://www.digimarc.com/
Domain-Specific Fine-Tuning of Large Language Models for Interactive Robot Programming
The paper proposes a natural language system to assist with programming industrial robots. This could enable more accessible and efficient robot programming, expanding robotics applications.
Authors: Benjamin Alt, Urs Keßner, Aleksandar Taranovic, Darko Katic, Andreas Hermann, Rainer Jäkel, Gerhard Neumann
Link: https://arxiv.org/abs/2312.13905v1
Date: 2023-12-21
Summary:
Industrial robots are applied in a widening range of industries, but robot programming mostly remains a task limited to programming experts. We propose a natural language-based assistant for programming of advanced, industrial robotic applications and investigate strategies for domain-specific fine-tuning of foundation models with limited data and compute.
--------------------------------------------------------------------------------------------------------
Empowering Few-Shot Recommender Systems with Large Language Models -- Enhanced Representations
The paper explores using large language models to enhance few-shot recommender systems. This could improve recommendations in sparse data scenarios, benefiting platforms with limited user data.
Authors: Zhoumeng Wang
Link: https://arxiv.org/abs/2312.13557v1
Date: 2023-12-21
Summary:
Recommender systems utilizing explicit feedback have witnessed significant advancements and widespread applications over the past years. However, generating recommendations in few-shot scenarios remains a persistent challenge. Recently, large language models (LLMs) have emerged as a promising solution for addressing natural language processing (NLP) tasks, thereby offering novel insights into tackling the few-shot scenarios encountered by explicit feedback-based recommender systems. To bridge recommender systems and LLMs, we devise a prompting template that generates user and item representations based on explicit feedback. Subsequently, we integrate these LLM-processed representations into various recommendation models to evaluate their significance across diverse recommendation tasks. Our ablation experiments and case study analysis collectively demonstrate the effectiveness of LLMs in processing explicit feedback, highlighting that LLMs equipped with generative and logical reasoning capabilities can effectively serve as a component of recommender systems to enhance their performance in few-shot scenarios. Furthermore, the broad adaptability of LLMs augments the generalization potential of recommender models, despite certain inherent constraints. We anticipate that our study can inspire researchers to delve deeper into the multifaceted dimensions of LLMs's involvement in recommender systems and contribute to the advancement of the explicit feedback-based recommender systems field.
--------------------------------------------------------------------------------------------------------
Benchmarking and Defending Against Indirect Prompt Injection Attacks on Large Language Models
The paper analyzes vulnerabilities of large language models to indirect prompt injection attacks. Defenses proposed could improve model security, preventing malicious usage.
Authors: Jingwei Yi, Yueqi Xie, Bin Zhu, Keegan Hines, Emre Kiciman, Guangzhong Sun, Xing Xie, Fangzhao Wu
Link: https://arxiv.org/abs/2312.14197v1
Date: 2023-12-21
Summary:
Recent remarkable advancements in large language models (LLMs) have led to their widespread adoption in various applications. A key feature of these applications is the combination of LLMs with external content, where user instructions and third-party content are combined to create prompts for LLM processing. These applications, however, are vulnerable to indirect prompt injection attacks, where malicious instructions embedded within external content compromise LLM's output, causing their responses to deviate from user expectations. Despite the discovery of this security issue, no comprehensive analysis of indirect prompt injection attacks on different LLMs is available due to the lack of a benchmark. Furthermore, no effective defense has been proposed. In this work, we introduce the first benchmark, BIPIA, to measure the robustness of various LLMs and defenses against indirect prompt injection attacks. Our experiments reveal that LLMs with greater capabilities exhibit more vulnerable to indirect prompt injection attacks for text tasks, resulting in a higher ASR. We hypothesize that indirect prompt injection attacks are mainly due to the LLMs' inability to distinguish between instructions and external content. Based on this conjecture, we propose four black-box methods based on prompt learning and a white-box defense methods based on fine-tuning with adversarial training to enable LLMs to distinguish between instructions and external content and ignore instructions in the external content. Our experimental results show that our black-box defense methods can effectively reduce ASR but cannot completely thwart indirect prompt injection attacks, while our white-box defense method can reduce ASR to nearly zero with little adverse impact on the LLM's performance on general tasks. We hope that our benchmark and defenses can inspire future work in this important area.
--------------------------------------------------------------------------------------------------------
Secure Authentication Mechanism for Cluster based Vehicular Adhoc Network (VANET): A Survey
The paper surveys authentication mechanisms for vehicular networks. Standardized secure authentication would enable expanded intelligent transportation applications.
Authors: Rabia Nasir, Humaira Ashraf, NZ Jhanjhi
Link: https://arxiv.org/abs/2312.12925v1
Date: 2023-12-20
Summary:
Vehicular Ad Hoc Networks (VANETs) play a crucial role in Intelligent Transportation Systems (ITS) by facilitating communication between vehicles and infrastructure. This communication aims to enhance road safety, improve traffic efficiency, and enhance passenger comfort. The secure and reliable exchange of information is paramount to ensure the integrity and confidentiality of data, while the authentication of vehicles and messages is essential to prevent unauthorized access and malicious activities. This survey paper presents a comprehensive analysis of existing authentication mechanisms proposed for cluster-based VANETs. The strengths, weaknesses, and suitability of these mechanisms for various scenarios are carefully examined. Additionally, the integration of secure key management techniques is discussed to enhance the overall authentication process. Cluster-based VANETs are formed by dividing the network into smaller groups or clusters, with designated cluster heads comprising one or more vehicles. Furthermore, this paper identifies gaps in the existing literature through an exploration of previous surveys. Several schemes based on different methods are critically evaluated, considering factors such as throughput, detection rate, security, packet delivery ratio, and end-to-end delay. To provide optimal solutions for authentication in cluster-based VANETs, this paper highlights AI- and ML-based routing-based schemes. These approaches leverage artificial intelligence and machine learning techniques to enhance authentication within the cluster-based VANET network. Finally, this paper explores the open research challenges that exist in the realm of authentication for cluster-based Vehicular Adhoc Networks, shedding light on areas that require further investigation and development.
--------------------------------------------------------------------------------------------------------
The paper analyzes integrating deep learning and computer vision. These techniques could drive progress in intelligent machine vision systems across industrial applications.
Authors: Bo Liu, Liqiang Yu, Chang Che, Qunwei Lin, Hao Hu, Xinyu Zhao
Link: https://arxiv.org/abs/2312.12872v1
Date: 2023-12-20
Summary:
This paper focuses on the analysis of the application effectiveness of the integration of deep learning and computer vision technologies. Deep learning achieves a historic breakthrough by constructing hierarchical neural networks, enabling end-to-end feature learning and semantic understanding of images. The successful experiences in the field of computer vision provide strong support for training deep learning algorithms. The tight integration of these two fields has given rise to a new generation of advanced computer vision systems, significantly surpassing traditional methods in tasks such as machine vision image classification and object detection. In this paper, typical image classification cases are combined to analyze the superior performance of deep neural network models while also pointing out their limitations in generalization and interpretability, proposing directions for future improvements. Overall, the efficient integration and development trend of deep learning with massive visual data will continue to drive technological breakthroughs and application expansion in the field of computer vision, making it possible to build truly intelligent machine vision systems. This deepening fusion paradigm will powerfully promote unprecedented tasks and functions in computer vision, providing stronger development momentum for related disciplines and industries.
--------------------------------------------------------------------------------------------------------
Optimizing Distributed Training on Frontier for Large Language Models
The paper investigates distributed training techniques to optimize large language model training on the Frontier supercomputer. Efficient scaling could enable training trillion parameter models to advance language capabilities.
Authors: Sajal Dash, Isaac Lyngaas, Junqi Yin, Xiao Wang, Romain Egele, Guojing Cong, Feiyi Wang, Prasanna Balaprakash
Link: https://arxiv.org/abs/2312.12705v2
Date: 2023-12-21
Summary:
Large language models (LLMs) have demonstrated remarkable success as foundational models, benefiting various downstream applications through fine-tuning. Recent studies on loss scaling have demonstrated the superior performance of larger LLMs compared to their smaller counterparts. Nevertheless, training LLMs with billions of parameters poses significant challenges and requires considerable computational resources. For example, training a one trillion parameter GPT-style model on 20 trillion tokens requires a staggering 120 million exaflops of computation. This research explores efficient distributed training strategies to extract this computation from Frontier, the world's first exascale supercomputer dedicated to open science. We enable and investigate various model and data parallel training techniques, such as tensor parallelism, pipeline parallelism, and sharded data parallelism, to facilitate training a trillion-parameter model on Frontier. We empirically assess these techniques and their associated parameters to determine their impact on memory footprint, communication latency, and GPU's computational efficiency. We analyze the complex interplay among these techniques and find a strategy to combine them to achieve high throughput through hyperparameter tuning. We have identified efficient strategies for training large LLMs of varying sizes through empirical analysis and hyperparameter tuning. For 22 Billion, 175 Billion, and 1 Trillion parameters, we achieved GPU throughputs of $38.38\%$, $36.14\%$, and $31.96\%$, respectively. For the training of the 175 Billion parameter model and the 1 Trillion parameter model, we achieved $100\%$ weak scaling efficiency on 1024 and 3072 MI250X GPUs, respectively. We also achieved strong scaling efficiencies of $89\%$ and $87\%$ for these two models.
--------------------------------------------------------------------------------------------------------
A Challenger to GPT-4V? Early Explorations of Gemini in Visual Expertise
The paper explores visual capabilities of the new Gemini model, finding it comparable to GPT-4V on vision tasks. This indicates Gemini's potential as a vision-capable model despite limitations.
Authors: Chaoyou Fu, Renrui Zhang, Zihan Wang, Yubo Huang, Zhengye Zhang, Longtian Qiu, Gaoxiang Ye, Yunhang Shen, Mengdan Zhang, Peixian Chen, Sirui Zhao, Shaohui Lin, Deqiang Jiang, Di Yin, Peng Gao, Ke Li, Hongsheng Li, Xing Sun
Link: https://arxiv.org/abs/2312.12436v2
Date: 2023-12-20
Summary:
The surge of interest towards Multi-modal Large Language Models (MLLMs), e.g., GPT-4V(ision) from OpenAI, has marked a significant trend in both academia and industry. They endow Large Language Models (LLMs) with powerful capabilities in visual understanding, enabling them to tackle diverse multi-modal tasks. Very recently, Google released Gemini, its newest and most capable MLLM built from the ground up for multi-modality. In light of the superior reasoning capabilities, can Gemini challenge GPT-4V's leading position in multi-modal learning? In this paper, we present a preliminary exploration of Gemini Pro's visual understanding proficiency, which comprehensively covers four domains: fundamental perception, advanced cognition, challenging vision tasks, and various expert capacities. We compare Gemini Pro with the state-of-the-art GPT-4V to evaluate its upper limits, along with the latest open-sourced MLLM, Sphinx, which reveals the gap between manual efforts and black-box systems. The qualitative samples indicate that, while GPT-4V and Gemini showcase different answering styles and preferences, they can exhibit comparable visual reasoning capabilities, and Sphinx still trails behind them concerning domain generalizability. Specifically, GPT-4V tends to elaborate detailed explanations and intermediate steps, and Gemini prefers to output a direct and concise answer. The quantitative evaluation on the popular MME benchmark also demonstrates the potential of Gemini to be a strong challenger to GPT-4V. Our early investigation of Gemini also observes some common issues of MLLMs, indicating that there still remains a considerable distance towards artificial general intelligence. Our project for tracking the progress of MLLM is released at https://github.com/BradyFU/Awesome-Multimodal-Large-Language-Models.
--------------------------------------------------------------------------------------------------------
Control Aspects for Using RIS in Latency-Constrained Mobile Edge Computing
The paper studies control signaling for reconfigurable intelligent surfaces in edge computing. Optimizing this could improve latency for time-sensitive edge applications.
Authors: Fabio Saggese, Victor Croisfelt, Francesca Costanzo, Junya Shiraishi, Radosław Kotaba, Paolo Di Lorenzo, Petar Popovski
Link: https://arxiv.org/abs/2312.12025v1
Date: 2023-12-19
Summary:
This paper investigates the role and the impact of control operations for dynamic mobile edge computing (MEC) empowered by Reconfigurable Intelligent Surfaces (RISs), in which multiple devices offload their computation tasks to an access point (AP) equipped with an edge server (ES), with the help of the RIS. While usually ignored, the control aspects related to channel estimation (CE), resource allocation (RA), and control signaling play a fundamental role in the user-perceived delay and energy consumption. In general, the higher the resources involved in the control operations, the higher their reliability; however, this introduces an overhead, which reduces the number of resources available for computation offloading, possibly increasing the overall latency experienced. Conversely, a lower control overhead translates to more resources available for computation offloading but impacts the CE accuracy and RA flexibility. This paper establishes a basic framework for integrating the impact of control operations in the performance evaluation of the RIS-aided MEC paradigm, clarifying their trade-offs through theoretical analysis and numerical simulations.
--------------------------------------------------------------------------------------------------------
Climate Change from Large Language Models
The paper evaluates climate change knowledge in large language models. Addressing gaps found could improve public education on this critical issue.
Authors: Hongyin Zhu, Prayag Tiwari
Link: https://arxiv.org/abs/2312.11985v2
Date: 2023-12-20
Summary:
Climate change presents significant challenges to the global community, and it is imperative to raise widespread awareness of the climate crisis and educate users about low-carbon living. Artificial intelligence, particularly large language models (LLMs), have emerged as powerful tools in mitigating the climate crisis, leveraging their extensive knowledge, broad user base, and natural language interaction capabilities. However, despite the growing body of research on climate change, there is a lack of comprehensive assessments of climate crisis knowledge within LLMs. This paper aims to resolve this gap by proposing an automatic evaluation framework. We employ a hybrid approach to data acquisition that combines data synthesis and manual collection to compile a diverse set of questions related to the climate crisis. These questions cover various aspects of climate change, including its causes, impacts, mitigation strategies, and adaptation measures. We then evaluate the model knowledge through prompt engineering based on the collected questions and generated answers. We propose a set of comprehensive metrics to evaluate the climate crisis knowledge, incorporating indicators from 10 different perspectives. Experimental results show that our method is effective in evaluating the knowledge of LLMs regarding the climate crisis. We evaluate several state-of-the-art LLMs and find that their knowledge falls short in terms of timeliness.
--------------------------------------------------------------------------------------------------------
Fluctuation-based Adaptive Structured Pruning for Large Language Models
The paper proposes a structured pruning method to compress large language models. This could enable more efficient deployment, expanding access and usage.
Authors: Yongqi An, Xu Zhao, Tao Yu, Ming Tang, Jinqiao Wang
Link: https://arxiv.org/abs/2312.11983v1
Date: 2023-12-19
Summary:
Network Pruning is a promising way to address the huge computing resource demands of the deployment and inference of Large Language Models (LLMs). Retraining-free is important for LLMs' pruning methods. However, almost all of the existing retraining-free pruning approaches for LLMs focus on unstructured pruning, which requires specific hardware support for acceleration. In this paper, we propose a novel retraining-free structured pruning framework for LLMs, named FLAP (FLuctuation-based Adaptive Structured Pruning). It is hardware-friendly by effectively reducing storage and enhancing inference speed. For effective structured pruning of LLMs, we highlight three critical elements that demand the utmost attention: formulating structured importance metrics, adaptively searching the global compressed model, and implementing compensation mechanisms to mitigate performance loss. First, FLAP determines whether the output feature map is easily recoverable when a column of weight is removed, based on the fluctuation pruning metric. Then it standardizes the importance scores to adaptively determine the global compressed model structure. At last, FLAP adds additional bias terms to recover the output feature maps using the baseline values. We thoroughly evaluate our approach on a variety of language benchmarks. Without any retraining, our method significantly outperforms the state-of-the-art methods, including LLM-Pruner and the extension of Wanda in structured pruning. The code is released at https://github.com/CASIA-IVA-Lab/FLAP.
--------------------------------------------------------------------------------------------------------
LLM-ARK: Knowledge Graph Reasoning Using Large Language Models via Deep Reinforcement Learning
The paper proposes a reinforcement learning approach to improve knowledge graph reasoning by large language models. This could enable more capable AI assistants and chatbots leveraging knowledge bases.
Authors: Yuxuan Huang
Link: https://arxiv.org/abs/2312.11282v1
Date: 2023-12-18
Summary:
With the evolution of pre-training methods, large language models (LLMs) have exhibited exemplary reasoning capabilities via prompt engineering. However, the absence of Knowledge Graph (KG) environment awareness and the challenge of engineering viable optimization mechanisms for intermediary reasoning processes, constrict the performance of LLMs on KG reasoning tasks compared to smaller models. We introduce LLM-ARK, a LLM grounded KG reasoning agent designed to deliver precise and adaptable predictions on KG paths. LLM-ARK utilizes Full Textual Environment (FTE) prompts to assimilate state information for each step-sized intelligence. Leveraging LLMs to richly encode and represent various types of inputs and integrate the knowledge graph further with path environment data, before making the final decision. Reframing the Knowledge Graph (KG) multi-hop inference problem as a sequential decision-making issue, we optimize our model using the Proximal Policy Optimization (PPO) online policy gradient reinforcement learning algorithm which allows the model to learn from a vast array of reward signals across diverse tasks and environments. We evaluate state-of-the-art LLM(GPT-4) and our method which using open-source models of varying sizes on OpenDialKG dataset. Our experiment shows that LLaMA7B-ARK provides excellent results with a performance rate of 48.75% for the target@1 evaluation metric, far exceeding the current state-of-the-art model by 17.64 percentage points. Meanwhile, GPT-4 accomplished a score of only 14.91%, further highlighting the efficacy and complexity of our methodology. Our code is available on GitHub for further access.
--------------------------------------------------------------------------------------------------------
The paper develops an optimization framework combining machine learning and evolutionary algorithms to maximize throughput of industrial mills. Such hybrid intelligence systems could be applied to optimize manufacturing processes.
Authors: Zahra Ghasemi, Mehdi Neshat, Chris Aldrich, John Karageorgos, Max Zanin, Frank Neumann, Lei Chen
Link: https://arxiv.org/abs/2312.10992v1
Date: 2023-12-18
Summary:
In mineral processing plants, grinding is a crucial step, accounting for approximately 50 percent of the total mineral processing costs. Semi-autogenous grinding mills are extensively employed in the grinding circuit of mineral processing plants. Maximizing SAG mill throughput is of significant importance considering its profound financial outcomes. However, the optimum process parameter setting aimed at achieving maximum mill throughput remains an uninvestigated domain in prior research. This study introduces a hybrid intelligent framework leveraging expert knowledge, machine learning techniques, and evolutionary algorithms to address this research need. In this study, we utilize an extensive industrial dataset comprising 36743 records and select relevant features based on the insights of industry experts. Following the removal of erroneous data, a comprehensive evaluation of 17 diverse machine learning models is undertaken to identify the most accurate predictive model. To improve the model performance, feature selection and outlier detection are executed. The resultant optimal model, trained with refined features, serves as the objective function within three distinct evolutionary algorithms. These algorithms are employed to identify parameter configurations that maximize SAG mill throughput while adhering to the working limits of input parameters as constraints. Notably, our analysis revealed that CatBoost, as an ensemble model, stands out as the most accurate predictor. Furthermore, differential evolution emerges as the preferred optimization algorithm, exhibiting superior performance in both achieving the highest mill throughput predictions and ensuring robustness in predictions, surpassing alternative methods.
--------------------------------------------------------------------------------------------------------
SCoTTi: Save Computation at Training Time with an adaptive framework
The paper introduces a method to reduce computation in on-device model training for edge devices. Enabling efficient on-device training could facilitate privacy-preserving personalized ML applications.
Authors: Ziyu Lin, Enzo Tartaglione, Van-Tam Nguyen
Link: https://arxiv.org/abs/2312.12483v1
Date: 2023-12-19
Summary:
On-device training is an emerging approach in machine learning where models are trained on edge devices, aiming to enhance privacy protection and real-time performance. However, edge devices typically possess restricted computational power and resources, making it challenging to perform computationally intensive model training tasks. Consequently, reducing resource consumption during training has become a pressing concern in this field. To this end, we propose SCoTTi (Save Computation at Training Time), an adaptive framework that addresses the aforementioned challenge. It leverages an optimizable threshold parameter to effectively reduce the number of neuron updates during training which corresponds to a decrease in memory and computation footprint. Our proposed approach demonstrates superior performance compared to the state-of-the-art methods regarding computational resource savings on various commonly employed benchmarks and popular architectures, including ResNets, MobileNet, and Swin-T.
--------------------------------------------------------------------------------------------------------
The Problem of Coherence in Natural Language Explanations of Recommendations
The paper analyzes issues with coherence in natural language recommendation explanations. Improving coherence could make recommendation systems more interpretable and useful.
Authors: Jakub Raczyński, Mateusz Lango, Jerzy Stefanowski
Link: https://arxiv.org/abs/2312.11356v1
Date: 2023-12-18
Summary:
Providing natural language explanations for recommendations is particularly useful from the perspective of a non-expert user. Although several methods for providing such explanations have recently been proposed, we argue that an important aspect of explanation quality has been overlooked in their experimental evaluation. Specifically, the coherence between generated text and predicted rating, which is a necessary condition for an explanation to be useful, is not properly captured by currently used evaluation measures. In this paper, we highlight the issue of explanation and prediction coherence by 1) presenting results from a manual verification of explanations generated by one of the state-of-the-art approaches 2) proposing a method of automatic coherence evaluation 3) introducing a new transformer-based method that aims to produce more coherent explanations than the state-of-the-art approaches 4) performing an experimental evaluation which demonstrates that this method significantly improves the explanation coherence without affecting the other aspects of recommendation performance.
--------------------------------------------------------------------------------------------------------
Localization and Discrete Beamforming with a Large Reconfigurable Intelligent Surface
The paper studies localization and beamforming using large intelligent surfaces for wireless systems. Precise localization could enable applications like autonomous vehicles and augmented reality.
Authors: Baojia Luo, Yili Deng, Miaomiao Dong, Zhongyi Huang, Xiang Chen, Wei Han, Bo Bai
Link: https://arxiv.org/abs/2312.12358v1
Date: 2023-12-19
Summary:
In millimeter-wave (mmWave) cellular systems, reconfigurable intelligent surfaces (RISs) are foreseeably deployed with a large number of reflecting elements to achieve high beamforming gains. The large-sized RIS will make radio links fall in the near-field localization regime with spatial non-stationarity issues. Moreover, the discrete phase restriction on the RIS reflection coefficient incurs exponential complexity for discrete beamforming. It remains an open problem to find the optimal RIS reflection coefficient design in polynomial time. To address these issues, we propose a scalable partitioned-far-field protocol that considers both the near-filed non-stationarity and discrete beamforming. The protocol approximates near-field signal propagation using a partitioned-far-field representation to inherit the sparsity from the sophisticated far-field and facilitate the near-field localization scheme. To improve the theoretical localization performance, we propose a fast passive beamforming (FPB) algorithm that optimally solves the discrete RIS beamforming problem, reducing the search complexity from exponential order to linear order. Furthermore, by exploiting the partitioned structure of RIS, we introduce a two-stage coarse-to-fine localization algorithm that leverages both the time delay and angle information. Numerical results demonstrate that centimeter-level localization precision is achieved under medium and high signal-to-noise ratios (SNR), revealing that RISs can provide support for low-cost and high-precision localization in future cellular systems.
--------------------------------------------------------------------------------------------------------
The paper provides feedback on digital tools for diabetes screening in response to an FDA request. Wider deployment of inclusive screening tools could improve prevention and management.
Authors: Manuel Cossio
Link: https://arxiv.org/abs/2312.11226v1
Date: 2023-12-18
Summary:
This document provides responses to the FDA's request for public comments (Docket No FDA 2023 N 4853) on the role of digital health technologies (DHTs) in detecting prediabetes and undiagnosed type 2 diabetes. It explores current DHT applications in prevention, detection, treatment and reversal of prediabetes, highlighting AI chatbots, online forums, wearables and mobile apps. The methods employed by DHTs to capture health signals like glucose, diet, symptoms and community insights are outlined. Key subpopulations that could benefit most from remote screening tools include rural residents, minority groups, high-risk individuals and those with limited healthcare access. Capturable high-impact risk factors encompass glycemic variability, cardiovascular parameters, respiratory health, blood biomarkers and patient reported symptoms. An array of non-invasive monitoring tools are discussed, although further research into their accuracy for diverse groups is warranted. Extensive health datasets providing immense opportunities for AI and ML based risk modeling are presented. Promising techniques leveraging EHRs, imaging, wearables and surveys to enhance screening through AI and ML algorithms are showcased. Analysis of social media and streaming data further allows disease prediction across populations. Ongoing innovation focused on inclusivity and accessibility is highlighted as pivotal in unlocking DHTs potential for transforming prediabetes and diabetes prevention and care.
--------------------------------------------------------------------------------------------------------
Estimation of individual causal effects in network setup for multiple treatments
The paper proposes a method to estimate individual causal effects using networked observational data. This could improve analysis of treatment effects in social networks and online communities.
Authors: Abhinav Thorat, Ravi Kolla, Niranjan Pedanekar, Naoyuki Onoe
Link: https://arxiv.org/abs/2312.11573v1
Date: 2023-12-18
Summary:
We study the problem of estimation of Individual Treatment Effects (ITE) in the context of multiple treatments and networked observational data. Leveraging the network information, we aim to utilize hidden confounders that may not be directly accessible in the observed data, thereby enhancing the practical applicability of the strong ignorability assumption. To achieve this, we first employ Graph Convolutional Networks (GCN) to learn a shared representation of the confounders. Then, our approach utilizes separate neural networks to infer potential outcomes for each treatment. We design a loss function as a weighted combination of two components: representation loss and Mean Squared Error (MSE) loss on the factual outcomes. To measure the representation loss, we extend existing metrics such as Wasserstein and Maximum Mean Discrepancy (MMD) from the binary treatment setting to the multiple treatments scenario. To validate the effectiveness of our proposed methodology, we conduct a series of experiments on the benchmark datasets such as BlogCatalog and Flickr. The experimental results consistently demonstrate the superior performance of our models when compared to baseline methods.
--------------------------------------------------------------------------------------------------------
"Paraphrasing The Original Text" Makes High Accuracy Long-Context QA
The paper shows paraphrasing training data can improve question answering over long contexts. Enabling reasoning over lengthy texts could benefit legal, scientific, and conversational applications.
Authors: Yijiong Yu
Link: https://arxiv.org/abs/2312.11193v3
Date: 2023-12-20
Summary:
Although LLMs continue to iterate and improve, most open-source models still have a context window of no more than 4k, limiting their ability to handle long-context problems. Most existing open-source models for long-context chat still lack satisfactory accuracy. To address this issue, I approach it from the perspective of training data and theoretically prove that training the capability to handle long contexts requires "effective" rather than "long" data. Based on this, I propose using the "original text paraphrase" task, and successfully extend the context window of the existing model to 32k by a low-cost and effective method, achieving extremely high accuracy in multi-document-QA and surpassing all existing open-source models of the same scale. The model and training data have been open-sourced on HuggingFace(https://huggingface.co/yuyijiong/Qwen-14b-chat-yarn-32k) and WiseModel(https://wisemodel.cn/models/yuyijiong/Qwen-14b-chat-yarn-32k).
--------------------------------------------------------------------------------------------------------
Graph Invariant Learning with Subgraph Co-mixup for Out-Of-Distribution Generalization
The paper introduces a technique to improve graph neural network generalization on out-of-distribution data. This could expand applicability of graph learning methods to diverse real-world networks.
Authors: Tianrui Jia, Haoyang Li, Cheng Yang, Tao Tao, Chuan Shi
Link: https://arxiv.org/abs/2312.10988v1
Date: 2023-12-18
Summary:
Graph neural networks (GNNs) have been demonstrated to perform well in graph representation learning, but always lacking in generalization capability when tackling out-of-distribution (OOD) data. Graph invariant learning methods, backed by the invariance principle among defined multiple environments, have shown effectiveness in dealing with this issue. However, existing methods heavily rely on well-predefined or accurately generated environment partitions, which are hard to be obtained in practice, leading to sub-optimal OOD generalization performances. In this paper, we propose a novel graph invariant learning method based on invariant and variant patterns co-mixup strategy, which is capable of jointly generating mixed multiple environments and capturing invariant patterns from the mixed graph data. Specifically, we first adopt a subgraph extractor to identify invariant subgraphs. Subsequently, we design one novel co-mixup strategy, i.e., jointly conducting environment Mixup and invariant Mixup. For the environment Mixup, we mix the variant environment-related subgraphs so as to generate sufficiently diverse multiple environments, which is important to guarantee the quality of the graph invariant learning. For the invariant Mixup, we mix the invariant subgraphs, further encouraging to capture invariant patterns behind graphs while getting rid of spurious correlations for OOD generalization. We demonstrate that the proposed environment Mixup and invariant Mixup can mutually promote each other. Extensive experiments on both synthetic and real-world datasets demonstrate that our method significantly outperforms state-of-the-art under various distribution shifts.
--------------------------------------------------------------------------------------------------------
DeRDaVa: Deletion-Robust Data Valuation for Machine Learning
The paper develops a data valuation method robust to deletions, avoiding expensive recomputations. Accounting for deletions could make data valuation fairer as personal data control increases.
Authors: Xiao Tian, Rachael Hwee Ling Sim, Jue Fan, Bryan Kian Hsiang Low
Link: https://arxiv.org/abs/2312.11413v1
Date: 2023-12-18
Summary:
Data valuation is concerned with determining a fair valuation of data from data sources to compensate them or to identify training examples that are the most or least useful for predictions. With the rising interest in personal data ownership and data protection regulations, model owners will likely have to fulfil more data deletion requests. This raises issues that have not been addressed by existing works: Are the data valuation scores still fair with deletions? Must the scores be expensively recomputed? The answer is no. To avoid recomputations, we propose using our data valuation framework DeRDaVa upfront for valuing each data source's contribution to preserving robust model performance after anticipated data deletions. DeRDaVa can be efficiently approximated and will assign higher values to data that are more useful or less likely to be deleted. We further generalize DeRDaVa to Risk-DeRDaVa to cater to risk-averse/seeking model owners who are concerned with the worst/best-cases model utility. We also empirically demonstrate the practicality of our solutions.
--------------------------------------------------------------------------------------------------------