Classical optimization and learning-based methods are the two reigningparadigms in deformable image registration. While optimization-based methodsboast generalizability across modalities and robust performance, learning-basedmethods promise peak performance, incorporating weak supervision and amortizedoptimization. However, the exact conditions for either paradigm to perform wellover the other are shrouded and not explicitly outlined in the existingliterature. In this paper, we make an explicit correspondence between themutual information of the distribution of per-pixel intensity and labels, andthe performance of classical registration methods. This strong correlationhints to the fact that architectural designs in learning-based methods isunlikely to affect this correlation, and therefore, the performance oflearning-based methods. This hypothesis is thoroughly validated withstate-of-the-art classical and learning-based methods. However, learning-basedmethods with weak supervision can perform high-fidelity intensity and labelregistration, which is not possible with classical methods. Next, we show thatthis high-fidelity feature learning does not translate to invariance to domainshift, and learning-based methods are sensitive to such changes in the datadistribution. Finally, we propose a general recipe to choose the best paradigmfor a given registration problem, based on these observations. <<<

Sequential fine-tuning and multi-task learning are methods aiming toincorporate knowledge from multiple tasks; however, they suffer fromcatastrophic forgetting and difficulties in dataset balancing. To address theseshortcomings, we propose AdapterFusion, a new two stage learning algorithm thatleverages knowledge from multiple tasks. First, in the knowledge extractionstage we learn task specific parameters called adapters, that encapsulate thetask-specific information. We then combine the adapters in a separate knowledgecomposition step. We show that by separating the two stages, i.e., knowledgeextraction and knowledge composition, the classifier can effectively exploitthe representations learned from multiple tasks in a non-destructive manner. Weempirically evaluate AdapterFusion on 16 diverse NLU tasks, and find that iteffectively combines various types of knowledge at different layers of themodel. We show that our approach outperforms traditional strategies such asfull fine-tuning as well as multi-task learning. Our code and adapters areavailable at AdapterHub.ml. <<<

Transformer models have demonstrated impressive capabilities when trained atscale, excelling at difficult cognitive tasks requiring complex reasoning andrational decision-making. In this paper, we explore the application oftransformer models to chess, focusing on the critical role of the positionencoding within the attention mechanism. We show that in chess, transformersendowed with a sufficiently versatile position encoding can match existingchess-playing models at a fraction of the computational cost. Our architecturesignificantly outperforms AlphaZero at 8x fewer FLOPS and matches priorgrandmaster-level transformer-based agents at 30x fewer FLOPS. <<<

Recent advancements in quantum computing have opened new avenues for tacklinglong-standing complex combinatorial optimization problems that are intractablefor classical computers. Predicting secondary structure of mRNA is one suchnotoriously difficult problem that can benefit from the ever-increasingmaturity of quantum computing technology. Accurate prediction of mRNA secondarystructure is critical in designing RNA-based therapeutics as it dictatesvarious steps of an mRNA life cycle, including transcription, translation, anddecay. The current generation of quantum computers have reached utility-scale,allowing us to explore relatively large problem sizes. In this paper, weexamine the feasibility of solving mRNA secondary structures on a quantumcomputer with sequence length up to 60 nucleotides representing problems in thequbit range of 10 to 80. We use Conditional Value at Risk (CVaR)-based VQEalgorithm to solve the optimization problems, originating from the mRNAstructure prediction problem, on the IBM Eagle and Heron quantum processors. Toour encouragement, even with ``minimal'' error mitigation and fixed-depthcircuits, our hardware runs yield accurate predictions of minimum free energy(MFE) structures that match the results of the classical solver CPLEX. Ourresults provide sufficient evidence for the viability of solving mRNA structureprediction problems on a quantum computer and motivate continued research inthis direction. <<<

Fine-tuning large pre-trained models is an effective transfer mechanism inNLP. However, in the presence of many downstream tasks, fine-tuning isparameter inefficient: an entire new model is required for every task. As analternative, we propose transfer with adapter modules. Adapter modules yield acompact and extensible model; they add only a few trainable parameters pertask, and new tasks can be added without revisiting previous ones. Theparameters of the original network remain fixed, yielding a high degree ofparameter sharing. To demonstrate adapter's effectiveness, we transfer therecently proposed BERT Transformer model to 26 diverse text classificationtasks, including the GLUE benchmark. Adapters attain near state-of-the-artperformance, whilst adding only a few parameters per task. On GLUE, we attainwithin 0.4% of the performance of full fine-tuning, adding only 3.6% parametersper task. By contrast, fine-tuning trains 100% of the parameters per task. <<<

The AlphaZero algorithm has been successfully applied in a range of discretedomains, most notably board games. It utilizes a neural network, that learns avalue and policy function to guide the exploration in a Monte-Carlo TreeSearch. Although many search improvements have been proposed for Monte-CarloTree Search in the past, most of them refer to an older variant of the UpperConfidence bounds for Trees algorithm that does not use a policy for planning.We introduce a new, improved search algorithm for AlphaZero which generalizesthe search tree to a directed acyclic graph. This enables information flowacross different subtrees and greatly reduces memory consumption. Along withMonte-Carlo Graph Search, we propose a number of further extensions, such asthe inclusion of Epsilon-greedy exploration, a revised terminal solver and theintegration of domain knowledge as constraints. In our evaluations, we use theCrazyAra engine on chess and crazyhouse as examples to show that these changesbring significant improvements to AlphaZero. <<<

This paper introduces ZeusAI, an artificial intelligence system developed toplay the board game 7 Wonders Duel. Inspired by the AlphaZero reinforcementlearning algorithm, ZeusAI relies on a combination of Monte Carlo Tree Searchand a Transformer Neural Network to learn the game without human supervision.ZeusAI competes at the level of top human players, develops both known andnovel strategies, and allows us to test rule variants to improve the game'sbalance. This work demonstrates how AI can help in understanding and enhancingboard games. <<<

This paper introduces the Stoch BiRo, a cost-effective bipedal robot designedwith a modular mechanical structure having point feet to navigate uneven andunfamiliar terrains. The robot employs proprioceptive actuation in abduction,hips, and knees, leveraging a Raspberry Pi4 for control. Overcomingcomputational limitations, a Learning-based Linear Policy controller managesbalance and locomotion with only 3 degrees of freedom (DoF) per leg, distinctfrom the typical 5DoF in bipedal systems. Integrated within a modular controlarchitecture, these controllers enable autonomous handling of unforeseenterrain disturbances without external sensors or prior environment knowledge.The robot's policies are trained and simulated using MuJoCo, transferringlearned behaviors to the Stoch BiRo hardware for initial walking validations.This work highlights the Stoch BiRo's adaptability and cost-effectiveness inmechanical design, control strategies, and autonomous navigation, promisingdiverse applications in real-world robotics scenarios. <<<

Transformers achieve remarkable performance in several tasks but due to theirquadratic complexity, with respect to the input's length, they areprohibitively slow for very long sequences. To address this limitation, weexpress the self-attention as a linear dot-product of kernel feature maps andmake use of the associativity property of matrix products to reduce thecomplexity from $\mathcal{O}\left(N^2\right)$ to $\mathcal{O}\left(N\right)$,where $N$ is the sequence length. We show that this formulation permits aniterative implementation that dramatically accelerates autoregressivetransformers and reveals their relationship to recurrent neural networks. Ourlinear transformers achieve similar performance to vanilla transformers andthey are up to 4000x faster on autoregressive prediction of very longsequences. <<<

This paper aims to generate materials for 3D meshes from text descriptions.Unlike existing methods that synthesize texture maps, we propose to generatesegment-wise procedural material graphs as the appearance representation, whichsupports high-quality rendering and provides substantial flexibility inediting. Instead of relying on extensive paired data, i.e., 3D meshes withmaterial graphs and corresponding text descriptions, to train a material graphgenerative model, we propose to leverage the pre-trained 2D diffusion model asa bridge to connect the text and material graphs. Specifically, our approachdecomposes a shape into a set of segments and designs a segment-controlleddiffusion model to synthesize 2D images that are aligned with mesh parts. Basedon generated images, we initialize parameters of material graphs and fine-tunethem through the differentiable rendering module to produce materials inaccordance with the textual description. Extensive experiments demonstrate thesuperior performance of our framework in photorealism, resolution, andeditability over existing methods. Project page: https://zju3dv.github.io/MaPa <<<

Conventional deformable registration methods aim at solving an optimizationmodel carefully designed on image pairs and their computational costs areexceptionally high. In contrast, recent deep learning based approaches canprovide fast deformation estimation. These heuristic network architectures arefully data-driven and thus lack explicit geometric constraints, e.g.,topology-preserving, which are indispensable to generate plausibledeformations. We design a new deep learning based framework to optimize adiffeomorphic model via multi-scale propagation in order to integrateadvantages and avoid limitations of these two categories of approaches.Specifically, we introduce a generic optimization model to formulatediffeomorphic registration and develop a series of learnable architectures toobtain propagative updating in the coarse-to-fine feature space. Moreover, wepropose a novel bilevel self-tuned training strategy, allowing efficient searchof task-specific hyper-parameters. This training strategy increases theflexibility to various types of data while reduces computational and humanburdens. We conduct two groups of image registration experiments on 3D volumedatasets including image-to-atlas registration on brain MRI data andimage-to-image registration on liver CT data. Extensive results demonstrate thestate-of-the-art performance of the proposed method with diffeomorphicguarantee and extreme efficiency. We also apply our framework to challengingmulti-modal image registration, and investigate how our registration to supportthe down-streaming tasks for medical image analysis including multi-modalfusion and image segmentation. <<<

Pretraining Large Language Models (LLMs) on large corpora of textual data isnow a standard paradigm. When using these LLMs for many downstreamapplications, it is common to additionally bake in new knowledge (e.g.,time-critical news, or private domain knowledge) into the pretrained modeleither through RAG-based-prompting, or fine-tuning. However, the optimalmethodology for the model to gain such new knowledge remains an open question.In this paper, we present Retrieval Augmented FineTuning (RAFT), a trainingrecipe that improves the model's ability to answer questions in a "open-book"in-domain settings. In RAFT, given a question, and a set of retrieveddocuments, we train the model to ignore those documents that don't help inanswering the question, which we call, distractor documents. RAFT accomplishesthis by citing verbatim the right sequence from the relevant document thatwould help answer the question. This coupled with RAFT's chain-of-thought-styleresponse helps improve the model's ability to reason. In domain-specific RAG,RAFT consistently improves the model's performance across PubMed, HotpotQA, andGorilla datasets, presenting a post-training recipe to improve pre-trained LLMsto in-domain RAG. RAFT's code and demo are open-sourced atgithub.com/ShishirPatil/gorilla. <<<

The conformational landscape of proteins is crucial to understanding theirfunctionality in complex biological processes. Traditional physics-basedcomputational methods, such as molecular dynamics (MD) simulations, suffer fromrare event sampling and long equilibration time problems, hindering theirapplications in general protein systems. Recently, deep generative modelingtechniques, especially diffusion models, have been employed to generate novelprotein conformations. However, existing score-based diffusion methods cannotproperly incorporate important physical prior knowledge to guide the generationprocess, causing large deviations in the sampled protein conformations from theequilibrium distribution. In this paper, to overcome these limitations, wepropose a force-guided SE(3) diffusion model, ConfDiff, for proteinconformation generation. By incorporating a force-guided network with a mixtureof data-based score models, ConfDiff can can generate protein conformationswith rich diversity while preserving high fidelity. Experiments on a variety ofprotein conformation prediction tasks, including 12 fast-folding proteins andthe Bovine Pancreatic Trypsin Inhibitor (BPTI), demonstrate that our methodsurpasses the state-of-the-art method. <<<

There is a growing body of work that leverages features extracted viatopological data analysis to train machine learning models. While this field,sometimes known as topological machine learning (TML), has seen some notablesuccesses, an understanding of how the process of learning from topologicalfeatures differs from the process of learning from raw data is still limited.In this work, we begin to address one component of this larger issue by askingwhether a model trained with topological features learns internalrepresentations of data that are fundamentally different than those learned bya model trained with the original raw data. To quantify ``different'', weexploit two popular metrics that can be used to measure the similarity of thehidden representations of data within neural networks, neural stitching andcentered kernel alignment. From these we draw a range of conclusions about howtraining with topological features does and does not change the representationsthat a model learns. Perhaps unsurprisingly, we find that structurally, thehidden representations of models trained and evaluated on topological featuresdiffer substantially compared to those trained and evaluated on thecorresponding raw data. On the other hand, our experiments show that in somecases, these representations can be reconciled (at least to the degree requiredto solve the corresponding task) using a simple affine transformation. Weconjecture that this means that neural networks trained on raw data may extractsome limited topological features in the process of making predictions. <<<

We propose an efficient design of Transformer-based models for multivariatetime series forecasting and self-supervised representation learning. It isbased on two key components: (i) segmentation of time series intosubseries-level patches which are served as input tokens to Transformer; (ii)channel-independence where each channel contains a single univariate timeseries that shares the same embedding and Transformer weights across all theseries. Patching design naturally has three-fold benefit: local semanticinformation is retained in the embedding; computation and memory usage of theattention maps are quadratically reduced given the same look-back window; andthe model can attend longer history. Our channel-independent patch time seriesTransformer (PatchTST) can improve the long-term forecasting accuracysignificantly when compared with that of SOTA Transformer-based models. We alsoapply our model to self-supervised pre-training tasks and attain excellentfine-tuning performance, which outperforms supervised training on largedatasets. Transferring of masked pre-trained representation on one dataset toothers also produces SOTA forecasting accuracy. Code is available at:https://github.com/yuqinie98/PatchTST. <<<

Curriculum learning and imitation learning have been leveraged extensively inthe robotics domain. However, minimal research has been done on leveragingthese ideas on control tasks over highly stochastic time-series data. Here, wetheoretically and empirically explore these approaches in a representativecontrol task over complex time-series data. We implement the fundamental ideasof curriculum learning via data augmentation, while imitation learning isimplemented via policy distillation from an oracle. Our findings reveal thatcurriculum learning should be considered a novel direction in improvingcontrol-task performance over complex time-series. Our ample random-seedout-sample empirics and ablation studies are highly encouraging for curriculumlearning for time-series control. These findings are especially encouraging aswe tune all overlapping hyperparameters on the baseline -- giving an advantageto the baseline. On the other hand, we find that imitation learning should beused with caution. <<<

Autonomous systems are soon to be ubiquitous, from manufacturing autonomy toagricultural field robots, and from health care assistants to the entertainmentindustry. The majority of these systems are developed with modularsub-components for decision-making, planning, and control that may behand-engineered or learning-based. While these existing approaches have beenshown to perform well under the situations they were specifically designed for,they can perform especially poorly in rare, out-of-distribution scenarios thatwill undoubtedly arise at test-time. The rise of foundation models trained onmultiple tasks with impressively large datasets from a variety of fields hasled researchers to believe that these models may provide common sense reasoningthat existing planners are missing. Researchers posit that this common sensereasoning will bridge the gap between algorithm development and deployment toout-of-distribution tasks, like how humans adapt to unexpected scenarios. Largelanguage models have already penetrated the robotics and autonomous systemsdomains as researchers are scrambling to showcase their potential use cases indeployment. While this application direction is very promising empirically,foundation models are known to hallucinate and generate decisions that maysound reasonable, but are in fact poor. We argue there is a need to step backand simultaneously design systems that can quantify the certainty of a model'sdecision, and detect when it may be hallucinating. In this work, we discuss thecurrent use cases of foundation models for decision-making tasks, provide ageneral definition for hallucinations with examples, discuss existingapproaches to hallucination detection and mitigation with a focus on decisionproblems, and explore areas for further research in this exciting field. <<<