Fine-tuning large pre-trained models is an effective transfer mechanism in<br>NLP. However, in the presence of many downstream tasks, fine-tuning is<br>parameter inefficient: an entire new model is required for every task. As an<br>alternative, we propose transfer with adapter modules. Adapter modules yield a<br>compact and extensible model; they add only a few trainable parameters per<br>task, and new tasks can be added without revisiting previous ones. The<br>parameters of the original network remain fixed, yielding a high degree of<br>parameter sharing. To demonstrate adapter's effectiveness, we transfer the<br>recently proposed BERT Transformer model to 26 diverse text classification<br>tasks, including the GLUE benchmark. Adapters attain near state-of-the-art<br>performance, whilst adding only a few parameters per task. On GLUE, we attain<br>within 0.4% of the performance of full fine-tuning, adding only 3.6% parameters<br>per task. By contrast, fine-tuning trains 100% of the parameters per task. <<<

The AlphaZero algorithm has been successfully applied in a range of discrete<br>domains, most notably board games. It utilizes a neural network, that learns a<br>value and policy function to guide the exploration in a Monte-Carlo Tree<br>Search. Although many search improvements have been proposed for Monte-Carlo<br>Tree Search in the past, most of them refer to an older variant of the Upper<br>Confidence bounds for Trees algorithm that does not use a policy for planning.<br>We introduce a new, improved search algorithm for AlphaZero which generalizes<br>the search tree to a directed acyclic graph. This enables information flow<br>across different subtrees and greatly reduces memory consumption. Along with<br>Monte-Carlo Graph Search, we propose a number of further extensions, such as<br>the inclusion of Epsilon-greedy exploration, a revised terminal solver and the<br>integration of domain knowledge as constraints. In our evaluations, we use the<br>CrazyAra engine on chess and crazyhouse as examples to show that these changes<br>bring significant improvements to AlphaZero. <<<

This paper introduces the Stoch BiRo, a cost-effective bipedal robot designed<br>with a modular mechanical structure having point feet to navigate uneven and<br>unfamiliar terrains. The robot employs proprioceptive actuation in abduction,<br>hips, and knees, leveraging a Raspberry Pi4 for control. Overcoming<br>computational limitations, a Learning-based Linear Policy controller manages<br>balance and locomotion with only 3 degrees of freedom (DoF) per leg, distinct<br>from the typical 5DoF in bipedal systems. Integrated within a modular control<br>architecture, these controllers enable autonomous handling of unforeseen<br>terrain disturbances without external sensors or prior environment knowledge.<br>The robot's policies are trained and simulated using MuJoCo, transferring<br>learned behaviors to the Stoch BiRo hardware for initial walking validations.<br>This work highlights the Stoch BiRo's adaptability and cost-effectiveness in<br>mechanical design, control strategies, and autonomous navigation, promising<br>diverse applications in real-world robotics scenarios. <<<

Transformers achieve remarkable performance in several tasks but due to their<br>quadratic complexity, with respect to the input's length, they are<br>prohibitively slow for very long sequences. To address this limitation, we<br>express the self-attention as a linear dot-product of kernel feature maps and<br>make use of the associativity property of matrix products to reduce the<br>complexity from $\mathcal{O}\left(N^2\right)$ to $\mathcal{O}\left(N\right)$,<br>where $N$ is the sequence length. We show that this formulation permits an<br>iterative implementation that dramatically accelerates autoregressive<br>transformers and reveals their relationship to recurrent neural networks. Our<br>linear transformers achieve similar performance to vanilla transformers and<br>they are up to 4000x faster on autoregressive prediction of very long<br>sequences. <<<

This paper aims to generate materials for 3D meshes from text descriptions.<br>Unlike existing methods that synthesize texture maps, we propose to generate<br>segment-wise procedural material graphs as the appearance representation, which<br>supports high-quality rendering and provides substantial flexibility in<br>editing. Instead of relying on extensive paired data, i.e., 3D meshes with<br>material graphs and corresponding text descriptions, to train a material graph<br>generative model, we propose to leverage the pre-trained 2D diffusion model as<br>a bridge to connect the text and material graphs. Specifically, our approach<br>decomposes a shape into a set of segments and designs a segment-controlled<br>diffusion model to synthesize 2D images that are aligned with mesh parts. Based<br>on generated images, we initialize parameters of material graphs and fine-tune<br>them through the differentiable rendering module to produce materials in<br>accordance with the textual description. Extensive experiments demonstrate the<br>superior performance of our framework in photorealism, resolution, and<br>editability over existing methods. Project page: https://zju3dv.github.io/MaPa <<<

Conventional deformable registration methods aim at solving an optimization<br>model carefully designed on image pairs and their computational costs are<br>exceptionally high. In contrast, recent deep learning based approaches can<br>provide fast deformation estimation. These heuristic network architectures are<br>fully data-driven and thus lack explicit geometric constraints, e.g.,<br>topology-preserving, which are indispensable to generate plausible<br>deformations. We design a new deep learning based framework to optimize a<br>diffeomorphic model via multi-scale propagation in order to integrate<br>advantages and avoid limitations of these two categories of approaches.<br>Specifically, we introduce a generic optimization model to formulate<br>diffeomorphic registration and develop a series of learnable architectures to<br>obtain propagative updating in the coarse-to-fine feature space. Moreover, we<br>propose a novel bilevel self-tuned training strategy, allowing efficient search<br>of task-specific hyper-parameters. This training strategy increases the<br>flexibility to various types of data while reduces computational and human<br>burdens. We conduct two groups of image registration experiments on 3D volume<br>datasets including image-to-atlas registration on brain MRI data and<br>image-to-image registration on liver CT data. Extensive results demonstrate the<br>state-of-the-art performance of the proposed method with diffeomorphic<br>guarantee and extreme efficiency. We also apply our framework to challenging<br>multi-modal image registration, and investigate how our registration to support<br>the down-streaming tasks for medical image analysis including multi-modal<br>fusion and image segmentation. <<<

Pretraining Large Language Models (LLMs) on large corpora of textual data is<br>now a standard paradigm. When using these LLMs for many downstream<br>applications, it is common to additionally bake in new knowledge (e.g.,<br>time-critical news, or private domain knowledge) into the pretrained model<br>either through RAG-based-prompting, or fine-tuning. However, the optimal<br>methodology for the model to gain such new knowledge remains an open question.<br>In this paper, we present Retrieval Augmented FineTuning (RAFT), a training<br>recipe that improves the model's ability to answer questions in a "open-book"<br>in-domain settings. In RAFT, given a question, and a set of retrieved<br>documents, we train the model to ignore those documents that don't help in<br>answering the question, which we call, distractor documents. RAFT accomplishes<br>this by citing verbatim the right sequence from the relevant document that<br>would help answer the question. This coupled with RAFT's chain-of-thought-style<br>response helps improve the model's ability to reason. In domain-specific RAG,<br>RAFT consistently improves the model's performance across PubMed, HotpotQA, and<br>Gorilla datasets, presenting a post-training recipe to improve pre-trained LLMs<br>to in-domain RAG. RAFT's code and demo are open-sourced at<br>github.com/ShishirPatil/gorilla. <<<

The conformational landscape of proteins is crucial to understanding their<br>functionality in complex biological processes. Traditional physics-based<br>computational methods, such as molecular dynamics (MD) simulations, suffer from<br>rare event sampling and long equilibration time problems, hindering their<br>applications in general protein systems. Recently, deep generative modeling<br>techniques, especially diffusion models, have been employed to generate novel<br>protein conformations. However, existing score-based diffusion methods cannot<br>properly incorporate important physical prior knowledge to guide the generation<br>process, causing large deviations in the sampled protein conformations from the<br>equilibrium distribution. In this paper, to overcome these limitations, we<br>propose a force-guided SE(3) diffusion model, ConfDiff, for protein<br>conformation generation. By incorporating a force-guided network with a mixture<br>of data-based score models, ConfDiff can can generate protein conformations<br>with rich diversity while preserving high fidelity. Experiments on a variety of<br>protein conformation prediction tasks, including 12 fast-folding proteins and<br>the Bovine Pancreatic Trypsin Inhibitor (BPTI), demonstrate that our method<br>surpasses the state-of-the-art method. <<<

There is a growing body of work that leverages features extracted via<br>topological data analysis to train machine learning models. While this field,<br>sometimes known as topological machine learning (TML), has seen some notable<br>successes, an understanding of how the process of learning from topological<br>features differs from the process of learning from raw data is still limited.<br>In this work, we begin to address one component of this larger issue by asking<br>whether a model trained with topological features learns internal<br>representations of data that are fundamentally different than those learned by<br>a model trained with the original raw data. To quantify ``different'', we<br>exploit two popular metrics that can be used to measure the similarity of the<br>hidden representations of data within neural networks, neural stitching and<br>centered kernel alignment. From these we draw a range of conclusions about how<br>training with topological features does and does not change the representations<br>that a model learns. Perhaps unsurprisingly, we find that structurally, the<br>hidden representations of models trained and evaluated on topological features<br>differ substantially compared to those trained and evaluated on the<br>corresponding raw data. On the other hand, our experiments show that in some<br>cases, these representations can be reconciled (at least to the degree required<br>to solve the corresponding task) using a simple affine transformation. We<br>conjecture that this means that neural networks trained on raw data may extract<br>some limited topological features in the process of making predictions. <<<

We propose an efficient design of Transformer-based models for multivariate<br>time series forecasting and self-supervised representation learning. It is<br>based on two key components: (i) segmentation of time series into<br>subseries-level patches which are served as input tokens to Transformer; (ii)<br>channel-independence where each channel contains a single univariate time<br>series that shares the same embedding and Transformer weights across all the<br>series. Patching design naturally has three-fold benefit: local semantic<br>information is retained in the embedding; computation and memory usage of the<br>attention maps are quadratically reduced given the same look-back window; and<br>the model can attend longer history. Our channel-independent patch time series<br>Transformer (PatchTST) can improve the long-term forecasting accuracy<br>significantly when compared with that of SOTA Transformer-based models. We also<br>apply our model to self-supervised pre-training tasks and attain excellent<br>fine-tuning performance, which outperforms supervised training on large<br>datasets. Transferring of masked pre-trained representation on one dataset to<br>others also produces SOTA forecasting accuracy. Code is available at:<br>https://github.com/yuqinie98/PatchTST. <<<

Curriculum learning and imitation learning have been leveraged extensively in<br>the robotics domain. However, minimal research has been done on leveraging<br>these ideas on control tasks over highly stochastic time-series data. Here, we<br>theoretically and empirically explore these approaches in a representative<br>control task over complex time-series data. We implement the fundamental ideas<br>of curriculum learning via data augmentation, while imitation learning is<br>implemented via policy distillation from an oracle. Our findings reveal that<br>curriculum learning should be considered a novel direction in improving<br>control-task performance over complex time-series. Our ample random-seed<br>out-sample empirics and ablation studies are highly encouraging for curriculum<br>learning for time-series control. These findings are especially encouraging as<br>we tune all overlapping hyperparameters on the baseline -- giving an advantage<br>to the baseline. On the other hand, we find that imitation learning should be<br>used with caution. <<<

Autonomous systems are soon to be ubiquitous, from manufacturing autonomy to<br>agricultural field robots, and from health care assistants to the entertainment<br>industry. The majority of these systems are developed with modular<br>sub-components for decision-making, planning, and control that may be<br>hand-engineered or learning-based. While these existing approaches have been<br>shown to perform well under the situations they were specifically designed for,<br>they can perform especially poorly in rare, out-of-distribution scenarios that<br>will undoubtedly arise at test-time. The rise of foundation models trained on<br>multiple tasks with impressively large datasets from a variety of fields has<br>led researchers to believe that these models may provide common sense reasoning<br>that existing planners are missing. Researchers posit that this common sense<br>reasoning will bridge the gap between algorithm development and deployment to<br>out-of-distribution tasks, like how humans adapt to unexpected scenarios. Large<br>language models have already penetrated the robotics and autonomous systems<br>domains as researchers are scrambling to showcase their potential use cases in<br>deployment. While this application direction is very promising empirically,<br>foundation models are known to hallucinate and generate decisions that may<br>sound reasonable, but are in fact poor. We argue there is a need to step back<br>and simultaneously design systems that can quantify the certainty of a model's<br>decision, and detect when it may be hallucinating. In this work, we discuss the<br>current use cases of foundation models for decision-making tasks, provide a<br>general definition for hallucinations with examples, discuss existing<br>approaches to hallucination detection and mitigation with a focus on decision<br>problems, and explore areas for further research in this exciting field. <<<

Within clinical, biomedical, and translational science, an increasing number<br>of projects are adopting graphs for knowledge representation. Graph-based data<br>models elucidate the interconnectedness between core biomedical concepts,<br>enable data structures to be easily updated, and support intuitive queries,<br>visualizations, and inference algorithms. However, knowledge discovery across<br>these "knowledge graphs" (KGs) has remained difficult. Data set heterogeneity<br>and complexity; the proliferation of ad hoc data formats; poor compliance with<br>guidelines on findability, accessibility, interoperability, and reusability;<br>and, in particular, the lack of a universally-accepted, open-access model for<br>standardization across biomedical KGs has left the task of reconciling data<br>sources to downstream consumers. Biolink Model is an open source data model<br>that can be used to formalize the relationships between data structures in<br>translational science. It incorporates object-oriented classification and<br>graph-oriented features. The core of the model is a set of hierarchical,<br>interconnected classes (or categories) and relationships between them (or<br>predicates), representing biomedical entities such as gene, disease, chemical,<br>anatomical structure, and phenotype. The model provides class and edge<br>attributes and associations that guide how entities should relate to one<br>another. Here, we highlight the need for a standardized data model for KGs,<br>describe Biolink Model, and compare it with other models. We demonstrate the<br>utility of Biolink Model in various initiatives, including the Biomedical Data<br>Translator Consortium and the Monarch Initiative, and show how it has supported<br>easier integration and interoperability of biomedical KGs, bringing together<br>knowledge from multiple sources and helping to realize the goals of<br>translational science. <<<

Flow-based propagation and spatiotemporal Transformer are two mainstream<br>mechanisms in video inpainting (VI). Despite the effectiveness of these<br>components, they still suffer from some limitations that affect their<br>performance. Previous propagation-based approaches are performed separately<br>either in the image or feature domain. Global image propagation isolated from<br>learning may cause spatial misalignment due to inaccurate optical flow.<br>Moreover, memory or computational constraints limit the temporal range of<br>feature propagation and video Transformer, preventing exploration of<br>correspondence information from distant frames. To address these issues, we<br>propose an improved framework, called ProPainter, which involves enhanced<br>ProPagation and an efficient Transformer. Specifically, we introduce<br>dual-domain propagation that combines the advantages of image and feature<br>warping, exploiting global correspondences reliably. We also propose a<br>mask-guided sparse video Transformer, which achieves high efficiency by<br>discarding unnecessary and redundant tokens. With these components, ProPainter<br>outperforms prior arts by a large margin of 1.46 dB in PSNR while maintaining<br>appealing efficiency. <<<

Large Multimodal Model (LMM) GPT-4V(ision) endows GPT-4 with visual grounding<br>capabilities, making it possible to handle certain tasks through the Visual<br>Question Answering (VQA) paradigm. This paper explores the potential of<br>VQA-oriented GPT-4V in the recently popular visual Anomaly Detection (AD) and<br>is the first to conduct qualitative and quantitative evaluations on the popular<br>MVTec AD and VisA datasets. Considering that this task requires both<br>image-/pixel-level evaluations, the proposed GPT-4V-AD framework contains three<br>components: 1) Granular Region Division, 2) Prompt Designing, 3)<br>Text2Segmentation for easy quantitative evaluation, and have made some<br>different attempts for comparative analysis. The results show that GPT-4V can<br>achieve certain results in the zero-shot AD task through a VQA paradigm, such<br>as achieving image-level 77.1/88.0 and pixel-level 68.0/76.6 AU-ROCs on MVTec<br>AD and VisA datasets, respectively. However, its performance still has a<br>certain gap compared to the state-of-the-art zero-shot method, e.g., WinCLIP<br>ann CLIP-AD, and further research is needed. This study provides a baseline<br>reference for the research of VQA-oriented LMM in the zero-shot AD task, and we<br>also post several possible future works. Code is available at<br>\url{https://github.com/zhangzjn/GPT-4V-AD}. <<<