Transformers achieve remarkable performance in several tasks but due to theirquadratic complexity, with respect to the input's length, they areprohibitively slow for very long sequences. To address this limitation, weexpress the self-attention as a linear dot-product of kernel feature maps andmake use of the associativity property of matrix products to reduce thecomplexity from $\mathcal{O}\left(N^2\right)$ to $\mathcal{O}\left(N\right)$,where $N$ is the sequence length. We show that this formulation permits aniterative implementation that dramatically accelerates autoregressivetransformers and reveals their relationship to recurrent neural networks. Ourlinear transformers achieve similar performance to vanilla transformers andthey are up to 4000x faster on autoregressive prediction of very longsequences. <<<

This paper aims to generate materials for 3D meshes from text descriptions.Unlike existing methods that synthesize texture maps, we propose to generatesegment-wise procedural material graphs as the appearance representation, whichsupports high-quality rendering and provides substantial flexibility inediting. Instead of relying on extensive paired data, i.e., 3D meshes withmaterial graphs and corresponding text descriptions, to train a material graphgenerative model, we propose to leverage the pre-trained 2D diffusion model asa bridge to connect the text and material graphs. Specifically, our approachdecomposes a shape into a set of segments and designs a segment-controlleddiffusion model to synthesize 2D images that are aligned with mesh parts. Basedon generated images, we initialize parameters of material graphs and fine-tunethem through the differentiable rendering module to produce materials inaccordance with the textual description. Extensive experiments demonstrate thesuperior performance of our framework in photorealism, resolution, andeditability over existing methods. Project page: https://zju3dv.github.io/MaPa <<<

Conventional deformable registration methods aim at solving an optimizationmodel carefully designed on image pairs and their computational costs areexceptionally high. In contrast, recent deep learning based approaches canprovide fast deformation estimation. These heuristic network architectures arefully data-driven and thus lack explicit geometric constraints, e.g.,topology-preserving, which are indispensable to generate plausibledeformations. We design a new deep learning based framework to optimize adiffeomorphic model via multi-scale propagation in order to integrateadvantages and avoid limitations of these two categories of approaches.Specifically, we introduce a generic optimization model to formulatediffeomorphic registration and develop a series of learnable architectures toobtain propagative updating in the coarse-to-fine feature space. Moreover, wepropose a novel bilevel self-tuned training strategy, allowing efficient searchof task-specific hyper-parameters. This training strategy increases theflexibility to various types of data while reduces computational and humanburdens. We conduct two groups of image registration experiments on 3D volumedatasets including image-to-atlas registration on brain MRI data andimage-to-image registration on liver CT data. Extensive results demonstrate thestate-of-the-art performance of the proposed method with diffeomorphicguarantee and extreme efficiency. We also apply our framework to challengingmulti-modal image registration, and investigate how our registration to supportthe down-streaming tasks for medical image analysis including multi-modalfusion and image segmentation. <<<

Pretraining Large Language Models (LLMs) on large corpora of textual data isnow a standard paradigm. When using these LLMs for many downstreamapplications, it is common to additionally bake in new knowledge (e.g.,time-critical news, or private domain knowledge) into the pretrained modeleither through RAG-based-prompting, or fine-tuning. However, the optimalmethodology for the model to gain such new knowledge remains an open question.In this paper, we present Retrieval Augmented FineTuning (RAFT), a trainingrecipe that improves the model's ability to answer questions in a "open-book"in-domain settings. In RAFT, given a question, and a set of retrieveddocuments, we train the model to ignore those documents that don't help inanswering the question, which we call, distractor documents. RAFT accomplishesthis by citing verbatim the right sequence from the relevant document thatwould help answer the question. This coupled with RAFT's chain-of-thought-styleresponse helps improve the model's ability to reason. In domain-specific RAG,RAFT consistently improves the model's performance across PubMed, HotpotQA, andGorilla datasets, presenting a post-training recipe to improve pre-trained LLMsto in-domain RAG. RAFT's code and demo are open-sourced atgithub.com/ShishirPatil/gorilla. <<<

The conformational landscape of proteins is crucial to understanding theirfunctionality in complex biological processes. Traditional physics-basedcomputational methods, such as molecular dynamics (MD) simulations, suffer fromrare event sampling and long equilibration time problems, hindering theirapplications in general protein systems. Recently, deep generative modelingtechniques, especially diffusion models, have been employed to generate novelprotein conformations. However, existing score-based diffusion methods cannotproperly incorporate important physical prior knowledge to guide the generationprocess, causing large deviations in the sampled protein conformations from theequilibrium distribution. In this paper, to overcome these limitations, wepropose a force-guided SE(3) diffusion model, ConfDiff, for proteinconformation generation. By incorporating a force-guided network with a mixtureof data-based score models, ConfDiff can can generate protein conformationswith rich diversity while preserving high fidelity. Experiments on a variety ofprotein conformation prediction tasks, including 12 fast-folding proteins andthe Bovine Pancreatic Trypsin Inhibitor (BPTI), demonstrate that our methodsurpasses the state-of-the-art method. <<<

There is a growing body of work that leverages features extracted viatopological data analysis to train machine learning models. While this field,sometimes known as topological machine learning (TML), has seen some notablesuccesses, an understanding of how the process of learning from topologicalfeatures differs from the process of learning from raw data is still limited.In this work, we begin to address one component of this larger issue by askingwhether a model trained with topological features learns internalrepresentations of data that are fundamentally different than those learned bya model trained with the original raw data. To quantify ``different'', weexploit two popular metrics that can be used to measure the similarity of thehidden representations of data within neural networks, neural stitching andcentered kernel alignment. From these we draw a range of conclusions about howtraining with topological features does and does not change the representationsthat a model learns. Perhaps unsurprisingly, we find that structurally, thehidden representations of models trained and evaluated on topological featuresdiffer substantially compared to those trained and evaluated on thecorresponding raw data. On the other hand, our experiments show that in somecases, these representations can be reconciled (at least to the degree requiredto solve the corresponding task) using a simple affine transformation. Weconjecture that this means that neural networks trained on raw data may extractsome limited topological features in the process of making predictions. <<<

We propose an efficient design of Transformer-based models for multivariatetime series forecasting and self-supervised representation learning. It isbased on two key components: (i) segmentation of time series intosubseries-level patches which are served as input tokens to Transformer; (ii)channel-independence where each channel contains a single univariate timeseries that shares the same embedding and Transformer weights across all theseries. Patching design naturally has three-fold benefit: local semanticinformation is retained in the embedding; computation and memory usage of theattention maps are quadratically reduced given the same look-back window; andthe model can attend longer history. Our channel-independent patch time seriesTransformer (PatchTST) can improve the long-term forecasting accuracysignificantly when compared with that of SOTA Transformer-based models. We alsoapply our model to self-supervised pre-training tasks and attain excellentfine-tuning performance, which outperforms supervised training on largedatasets. Transferring of masked pre-trained representation on one dataset toothers also produces SOTA forecasting accuracy. Code is available at:https://github.com/yuqinie98/PatchTST. <<<

Curriculum learning and imitation learning have been leveraged extensively inthe robotics domain. However, minimal research has been done on leveragingthese ideas on control tasks over highly stochastic time-series data. Here, wetheoretically and empirically explore these approaches in a representativecontrol task over complex time-series data. We implement the fundamental ideasof curriculum learning via data augmentation, while imitation learning isimplemented via policy distillation from an oracle. Our findings reveal thatcurriculum learning should be considered a novel direction in improvingcontrol-task performance over complex time-series. Our ample random-seedout-sample empirics and ablation studies are highly encouraging for curriculumlearning for time-series control. These findings are especially encouraging aswe tune all overlapping hyperparameters on the baseline -- giving an advantageto the baseline. On the other hand, we find that imitation learning should beused with caution. <<<

Autonomous systems are soon to be ubiquitous, from manufacturing autonomy toagricultural field robots, and from health care assistants to the entertainmentindustry. The majority of these systems are developed with modularsub-components for decision-making, planning, and control that may behand-engineered or learning-based. While these existing approaches have beenshown to perform well under the situations they were specifically designed for,they can perform especially poorly in rare, out-of-distribution scenarios thatwill undoubtedly arise at test-time. The rise of foundation models trained onmultiple tasks with impressively large datasets from a variety of fields hasled researchers to believe that these models may provide common sense reasoningthat existing planners are missing. Researchers posit that this common sensereasoning will bridge the gap between algorithm development and deployment toout-of-distribution tasks, like how humans adapt to unexpected scenarios. Largelanguage models have already penetrated the robotics and autonomous systemsdomains as researchers are scrambling to showcase their potential use cases indeployment. While this application direction is very promising empirically,foundation models are known to hallucinate and generate decisions that maysound reasonable, but are in fact poor. We argue there is a need to step backand simultaneously design systems that can quantify the certainty of a model'sdecision, and detect when it may be hallucinating. In this work, we discuss thecurrent use cases of foundation models for decision-making tasks, provide ageneral definition for hallucinations with examples, discuss existingapproaches to hallucination detection and mitigation with a focus on decisionproblems, and explore areas for further research in this exciting field. <<<

Deepak R. Unni, Sierra A. T. Moxon, Michael Bada, Matthew Brush, Richard Bruskiewich, Paul Clemons, Vlado Dancik, Michel Dumontier, Karamarie Fecho, Gustavo Glusman, Jennifer J. Hadlock, Nomi L. Harris, Arpita Joshi, Tim Putman, Guangrong Qin, Stephen A. Ramsey, Kent A. Shefchek, Harold Solbrig, Karthik Soman, Anne T. Thessen, Melissa A. Haendel, Chris Bizon, Christopher J. Mungall, the Biomedical Data Translator Consortium <<<

Within clinical, biomedical, and translational science, an increasing numberof projects are adopting graphs for knowledge representation. Graph-based datamodels elucidate the interconnectedness between core biomedical concepts,enable data structures to be easily updated, and support intuitive queries,visualizations, and inference algorithms. However, knowledge discovery acrossthese "knowledge graphs" (KGs) has remained difficult. Data set heterogeneityand complexity; the proliferation of ad hoc data formats; poor compliance withguidelines on findability, accessibility, interoperability, and reusability;and, in particular, the lack of a universally-accepted, open-access model forstandardization across biomedical KGs has left the task of reconciling datasources to downstream consumers. Biolink Model is an open source data modelthat can be used to formalize the relationships between data structures intranslational science. It incorporates object-oriented classification andgraph-oriented features. The core of the model is a set of hierarchical,interconnected classes (or categories) and relationships between them (orpredicates), representing biomedical entities such as gene, disease, chemical,anatomical structure, and phenotype. The model provides class and edgeattributes and associations that guide how entities should relate to oneanother. Here, we highlight the need for a standardized data model for KGs,describe Biolink Model, and compare it with other models. We demonstrate theutility of Biolink Model in various initiatives, including the Biomedical DataTranslator Consortium and the Monarch Initiative, and show how it has supportedeasier integration and interoperability of biomedical KGs, bringing togetherknowledge from multiple sources and helping to realize the goals oftranslational science. <<<

Flow-based propagation and spatiotemporal Transformer are two mainstreammechanisms in video inpainting (VI). Despite the effectiveness of thesecomponents, they still suffer from some limitations that affect theirperformance. Previous propagation-based approaches are performed separatelyeither in the image or feature domain. Global image propagation isolated fromlearning may cause spatial misalignment due to inaccurate optical flow.Moreover, memory or computational constraints limit the temporal range offeature propagation and video Transformer, preventing exploration ofcorrespondence information from distant frames. To address these issues, wepropose an improved framework, called ProPainter, which involves enhancedProPagation and an efficient Transformer. Specifically, we introducedual-domain propagation that combines the advantages of image and featurewarping, exploiting global correspondences reliably. We also propose amask-guided sparse video Transformer, which achieves high efficiency bydiscarding unnecessary and redundant tokens. With these components, ProPainteroutperforms prior arts by a large margin of 1.46 dB in PSNR while maintainingappealing efficiency. <<<

Topological Data Analysis is a recent and fast growing field providing a setof new topological and geometric tools to infer relevant features for possiblycomplex data. This paper is a brief introduction, through a few selectedtopics, to basic fundamental and practical aspects of \tda\ for non experts. <<<

Large Multimodal Model (LMM) GPT-4V(ision) endows GPT-4 with visual groundingcapabilities, making it possible to handle certain tasks through the VisualQuestion Answering (VQA) paradigm. This paper explores the potential ofVQA-oriented GPT-4V in the recently popular visual Anomaly Detection (AD) andis the first to conduct qualitative and quantitative evaluations on the popularMVTec AD and VisA datasets. Considering that this task requires bothimage-/pixel-level evaluations, the proposed GPT-4V-AD framework contains threecomponents: 1) Granular Region Division, 2) Prompt Designing, 3)Text2Segmentation for easy quantitative evaluation, and have made somedifferent attempts for comparative analysis. The results show that GPT-4V canachieve certain results in the zero-shot AD task through a VQA paradigm, suchas achieving image-level 77.1/88.0 and pixel-level 68.0/76.6 AU-ROCs on MVTecAD and VisA datasets, respectively. However, its performance still has acertain gap compared to the state-of-the-art zero-shot method, e.g., WinCLIPann CLIP-AD, and further research is needed. This study provides a baselinereference for the research of VQA-oriented LMM in the zero-shot AD task, and wealso post several possible future works. Code is available at\url{https://github.com/zhangzjn/GPT-4V-AD}. <<<

Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan-Yen Lo, Piotr Dollár, Ross Girshick <<<

We introduce the Segment Anything (SA) project: a new task, model, anddataset for image segmentation. Using our efficient model in a data collectionloop, we built the largest segmentation dataset to date (by far), with over 1billion masks on 11M licensed and privacy respecting images. The model isdesigned and trained to be promptable, so it can transfer zero-shot to newimage distributions and tasks. We evaluate its capabilities on numerous tasksand find that its zero-shot performance is impressive -- often competitive withor even superior to prior fully supervised results. We are releasing theSegment Anything Model (SAM) and corresponding dataset (SA-1B) of 1B masks and11M images at https://segment-anything.com to foster research into foundationmodels for computer vision. <<<

Andreas Fürst, Elisabeth Rumetshofer, Johannes Lehner, Viet Tran, Fei Tang, Hubert Ramsauer, David Kreil, Michael Kopp, Günter Klambauer, Angela Bitto-Nemling, Sepp Hochreiter <<<

CLIP yielded impressive results on zero-shot transfer learning tasks and isconsidered as a foundation model like BERT or GPT3. CLIP vision models thathave a rich representation are pre-trained using the InfoNCE objective andnatural language supervision before they are fine-tuned on particular tasks.Though CLIP excels at zero-shot transfer learning, it suffers from anexplaining away problem, that is, it focuses on one or few features, whileneglecting other relevant features. This problem is caused by insufficientlyextracting the covariance structure in the original multi-modal data. Wesuggest to use modern Hopfield networks to tackle the problem of explainingaway. Their retrieved embeddings have an enriched covariance structure derivedfrom co-occurrences of features in the stored embeddings. However, modernHopfield networks increase the saturation effect of the InfoNCE objective whichhampers learning. We propose to use the InfoLOOB objective to mitigate thissaturation effect. We introduce the novel "Contrastive Leave One Out Boost"(CLOOB), which uses modern Hopfield networks for covariance enrichment togetherwith the InfoLOOB objective. In experiments we compare CLOOB to CLIP afterpre-training on the Conceptual Captions and the YFCC dataset with respect totheir zero-shot transfer learning performance on other datasets. CLOOBconsistently outperforms CLIP at zero-shot transfer learning across allconsidered architectures and datasets. <<<

Diffusion models have significantly advanced the fields of image, audio, andvideo generation, but they depend on an iterative sampling process that causesslow generation. To overcome this limitation, we propose consistency models, anew family of models that generate high quality samples by directly mappingnoise to data. They support fast one-step generation by design, while stillallowing multistep sampling to trade compute for sample quality. They alsosupport zero-shot data editing, such as image inpainting, colorization, andsuper-resolution, without requiring explicit training on these tasks.Consistency models can be trained either by distilling pre-trained diffusionmodels, or as standalone generative models altogether. Through extensiveexperiments, we demonstrate that they outperform existing distillationtechniques for diffusion models in one- and few-step sampling, achieving thenew state-of-the-art FID of 3.55 on CIFAR-10 and 6.20 on ImageNet 64x64 forone-step generation. When trained in isolation, consistency models become a newfamily of generative models that can outperform existing one-step,non-adversarial generative models on standard benchmarks such as CIFAR-10,ImageNet 64x64 and LSUN 256x256. <<<

This paper presents $\textbf{R}$epresentation-$\textbf{C}$onditioned image$\textbf{G}$eneration (RCG), a simple yet effective image generation frameworkwhich sets a new benchmark in class-unconditional image generation. RCG doesnot condition on any human annotations. Instead, it conditions on aself-supervised representation distribution which is mapped from the imagedistribution using a pre-trained encoder. During generation, RCG samples fromsuch representation distribution using a representation diffusion model (RDM),and employs a pixel generator to craft image pixels conditioned on the sampledrepresentation. Such a design provides substantial guidance during thegenerative process, resulting in high-quality image generation. Tested onImageNet 256$\times$256, RCG achieves a Frechet Inception Distance (FID) of3.31 and an Inception Score (IS) of 253.4. These results not only significantlyimprove the state-of-the-art of class-unconditional image generation but alsorival the current leading methods in class-conditional image generation,bridging the long-standing performance gap between these two tasks. Code isavailable at https://github.com/LTH14/rcg. <<<