Large language models (LLMs) are central to modern natural languageprocessing, delivering exceptional performance in various tasks. However, theirintensive computational and memory requirements present challenges, especiallyfor devices with limited DRAM capacity. This paper tackles the challenge ofefficiently running LLMs that exceed the available DRAM capacity by storing themodel parameters on flash memory but bringing them on demand to DRAM. Ourmethod involves constructing an inference cost model that harmonizes with theflash memory behavior, guiding us to optimize in two critical areas: reducingthe volume of data transferred from flash and reading data in larger, morecontiguous chunks. Within this flash memory-informed framework, we introducetwo principal techniques. First, "windowing'" strategically reduces datatransfer by reusing previously activated neurons, and second, "row-columnbundling", tailored to the sequential data access strengths of flash memory,increases the size of data chunks read from flash memory. These methodscollectively enable running models up to twice the size of the available DRAM,with a 4-5x and 20-25x increase in inference speed compared to naive loadingapproaches in CPU and GPU, respectively. Our integration of sparsity awareness,context-adaptive loading, and a hardware-oriented design paves the way foreffective inference of LLMs on devices with limited memory. <<<

Licheng Wen, Xuemeng Yang, Daocheng Fu, Xiaofeng Wang, Pinlong Cai, Xin Li, Tao Ma, Yingxuan Li, Linran Xu, Dengke Shang, Zheng Zhu, Shaoyan Sun, Yeqi Bai, Xinyu Cai, Min Dou, Shuanglu Hu, Botian Shi, Yu Qiao <<<

The pursuit of autonomous driving technology hinges on the sophisticatedintegration of perception, decision-making, and control systems. Traditionalapproaches, both data-driven and rule-based, have been hindered by theirinability to grasp the nuance of complex driving environments and theintentions of other road users. This has been a significant bottleneck,particularly in the development of common sense reasoning and nuanced sceneunderstanding necessary for safe and reliable autonomous driving. The advent ofVisual Language Models (VLM) represents a novel frontier in realizing fullyautonomous vehicle driving. This report provides an exhaustive evaluation ofthe latest state-of-the-art VLM, GPT-4V(ision), and its application inautonomous driving scenarios. We explore the model's abilities to understandand reason about driving scenes, make decisions, and ultimately act in thecapacity of a driver. Our comprehensive tests span from basic scene recognitionto complex causal reasoning and real-time decision-making under varyingconditions. Our findings reveal that GPT-4V demonstrates superior performancein scene understanding and causal reasoning compared to existing autonomoussystems. It showcases the potential to handle out-of-distribution scenarios,recognize intentions, and make informed decisions in real driving contexts.However, challenges remain, particularly in direction discernment, trafficlight recognition, vision grounding, and spatial reasoning tasks. Theselimitations underscore the need for further research and development. Projectis now available on GitHub for interested parties to access and utilize:\url{https://github.com/PJLab-ADG/GPT4V-AD-Exploration} <<<

Variational autoencoders are powerful algorithms for identifying dominantlatent structure in a single dataset. In many applications, however, we areinterested in modeling latent structure and variation that are enriched in atarget dataset compared to some background---e.g. enriched in patients comparedto the general population. Contrastive learning is a principled framework tocapture such enriched variation between the target and background, butstate-of-the-art contrastive methods are limited to linear models. In thispaper, we introduce the contrastive variational autoencoder (cVAE), whichcombines the benefits of contrastive learning with the power of deep generativemodels. The cVAE is designed to identify and enhance salient latent features.The cVAE is trained on two related but unpaired datasets, one of which hasminimal contribution from the salient latent features. The cVAE explicitlymodels latent features that are shared between the datasets, as well as thosethat are enriched in one dataset relative to the other, which allows thealgorithm to isolate and enhance the salient latent features. The algorithm isstraightforward to implement, has a similar run-time to the standard VAE, andis robust to noise and dataset purity. We conduct experiments across diversetypes of data, including gene expression and facial images, showing that thecVAE effectively uncovers latent structure that is salient in a particularanalysis. <<<

We propose an effective denoising diffusion model for generatinghigh-resolution images (e.g., 1024$\times$512), trained on small-size imagepatches (e.g., 64$\times$64). We name our algorithm Patch-DM, in which a newfeature collage strategy is designed to avoid the boundary artifact whensynthesizing large-size images. Feature collage systematically crops andcombines partial features of the neighboring patches to predict the features ofa shifted image patch, allowing the seamless generation of the entire image dueto the overlap in the patch feature space. Patch-DM produces high-quality imagesynthesis results on our newly collected dataset of nature images(1024$\times$512), as well as on standard benchmarks of smaller sizes(256$\times$256), including LSUN-Bedroom, LSUN-Church, and FFHQ. We compare ourmethod with previous patch-based generation methods and achievestate-of-the-art FID scores on all four datasets. Further, Patch-DM alsoreduces memory complexity compared to the classic diffusion models. <<<

Dynamic feature selection, where we sequentially query features to make accurate predictions with a minimal budget, is a promising paradigm to reduce feature acquisition costs and provide transparency into the prediction process. The problem is challenging, however, as it requires both making predictions with arbitrary feature sets and learning a policy to identify the most valuable selections. Here, we take an information-theoretic perspective and prioritize features based on their mutual information with the response variable. The main challenge is learning this selection policy, and we design a straightforward new modeling approach that estimates the mutual information in a discriminative rather than generative fashion. Building on our learning approach, we introduce several further improvements: allowing variable feature budgets across samples, enabling non-uniform costs between features, incorporating prior information, and exploring modern architectures to handle partial input information. We find that our method provides consistent gains over recent state-of-the-art methods across a variety of datasets. <<<

Deep learning in computer vision has achieved great success with the price of large-scale labeled training data. However, exhaustive data annotation is impracticable for each task of all domains of interest, due to high labor costs and unguaranteed labeling accuracy. Besides, the uncontrollable data collection process produces non-IID training and test data, where undesired duplication may exist. All these nuisances may hinder the verification of typical theories and exposure to new findings. To circumvent them, an alternative is to generate synthetic data via 3D rendering with domain randomization. We in this work push forward along this line by doing profound and extensive research on bare supervised learning and downstream domain adaptation. Specifically, under the well-controlled, IID data setting enabled by 3D rendering, we systematically verify the typical, important learning insights, e.g., shortcut learning, and discover the new laws of various data regimes and network architectures in generalization. We further investigate the effect of image formation factors on generalization, e.g., object scale, material texture, illumination, camera viewpoint, and background in a 3D scene. Moreover, we use the simulation-to-reality adaptation as a downstream task for comparing the transferability between synthetic and real data when used for pre-training, which demonstrates that synthetic data pre-training is also promising to improve real test results. Lastly, to promote future research, we develop a new large-scale synthetic-to-real benchmark for image classification, termed S2RDA, which provides more significant challenges for transfer from simulation to reality. The code and datasets are available at this https URL. <<<

Deep convolutional networks have demonstrated the state-of-the-art performance on various medical image computing tasks. Leveraging images from different modalities for the same analysis task holds clinical benefits. However, the generalization capability of deep models on test data with different distributions remain as a major challenge. In this paper, we propose the PnPAdaNet (plug-and-play adversarial domain adaptation network) for adapting segmentation networks between different modalities of medical images, e.g., MRI and CT. We propose to tackle the significant domain shift by aligning the feature spaces of source and target domains in an unsupervised manner. Specifically, a domain adaptation module flexibly replaces the early encoder layers of the source network, and the higher layers are shared between domains. With adversarial learning, we build two discriminators whose inputs are respectively multi-level features and predicted segmentation masks. We have validated our domain adaptation method on cardiac structure segmentation in unpaired MRI and CT. The experimental results with comprehensive ablation studies demonstrate the excellent efficacy of our proposed PnP-AdaNet. Moreover, we introduce a novel benchmark on the cardiac dataset for the task of unsupervised cross-modality domain adaptation. We will make our code and database publicly available, aiming to promote future studies on this challenging yet important research topic in medical imaging. <<<

Structure-based drug design (SBDD) aims to design small-molecule ligands that bind with high affinity and specificity to pre-determined protein targets. In this paper, we formulate SBDD as a 3D-conditional generation problem and present DiffSBDD, an SE(3)-equivariant 3D-conditional diffusion model that generates novel ligands conditioned on protein pockets. Comprehensive in silico experiments demonstrate the efficiency and effectiveness of DiffSBDD in generating novel and diverse drug-like ligands with competitive docking scores. We further explore the flexibility of the diffusion framework for a broader range of tasks in drug design campaigns, such as off-the-shelf property optimization and partial molecular design with inpainting. <<<

Deformable image registration is one of the fundamental tasks in medical imaging. Classical registration algorithms usually require a high computational cost for iterative optimizations. Although deep-learning-based methods have been developed for fast image registration, it is still challenging to obtain realistic continuous deformations from a moving image to a fixed image with less topological folding problem. To address this, here we present a novel diffusion-model-based image registration method, called DiffuseMorph. DiffuseMorph not only generates synthetic deformed images through reverse diffusion but also allows image registration by deformation fields. Specifically, the deformation fields are generated by the conditional score function of the deformation between the moving and fixed images, so that the registration can be performed from continuous deformation by simply scaling the latent feature of the score. Experimental results on 2D facial and 3D medical image registration tasks demonstrate that our method provides flexible deformations with topology preservation capability. <<<

Large language models (LLMs) are shown to possess a wealth of actionable knowledge that can be extracted for robot manipulation in the form of reasoning and planning. Despite the progress, most still rely on pre-defined motion primitives to carry out the physical interactions with the environment, which remains a major bottleneck. In this work, we aim to synthesize robot trajectories, i.e., a dense sequence of 6-DoF end-effector waypoints, for a large variety of manipulation tasks given an open-set of instructions and an open-set of objects. We achieve this by first observing that LLMs excel at inferring affordances and constraints given a free-form language instruction. More importantly, by leveraging their code-writing capabilities, they can interact with a visual-language model (VLM) to compose 3D value maps to ground the knowledge into the observation space of the agent. The composed value maps are then used in a model-based planning framework to zero-shot synthesize closed-loop robot trajectories with robustness to dynamic perturbations. We further demonstrate how the proposed framework can benefit from online experiences by efficiently learning a dynamics model for scenes that involve contact-rich interactions. We present a large-scale study of the proposed method in both simulated and real-robot environments, showcasing the ability to perform a large variety of everyday manipulation tasks specified in free-form natural language. Project website: this https URL <<<

We present high quality image synthesis results using diffusion probabilistic models, a class of latent variable models inspired by considerations from nonequilibrium thermodynamics. Our best results are obtained by training on a weighted variational bound designed according to a novel connection between diffusion probabilistic models and denoising score matching with Langevin dynamics, and our models naturally admit a progressive lossy decompression scheme that can be interpreted as a generalization of autoregressive decoding. On the unconditional CIFAR10 dataset, we obtain an Inception score of 9.46 and a state-of-the-art FID score of 3.17. On 256x256 LSUN, we obtain sample quality similar to ProgressiveGAN. Our implementation is available at this https URL <<<

Multivariate time series data comprises various channels of variables. The multivariate forecasting models need to capture the relationship between the channels to accurately predict future values. However, recently, there has been an emergence of methods that employ the Channel Independent (CI) strategy. These methods view multivariate time series data as separate univariate time series and disregard the correlation between channels. Surprisingly, our empirical results have shown that models trained with the CI strategy outperform those trained with the Channel Dependent (CD) strategy, usually by a significant margin. Nevertheless, the reasons behind this phenomenon have not yet been thoroughly explored in the literature. This paper provides comprehensive empirical and theoretical analyses of the characteristics of multivariate time series datasets and the CI/CD strategy. Our results conclude that the CD approach has higher capacity but often lacks robustness to accurately predict distributionally drifted time series. In contrast, the CI approach trades capacity for robust prediction. Practical measures inspired by these analyses are proposed to address the capacity and robustness dilemma, including a modified CD method called Predict Residuals with Regularization (PRReg) that can surpass the CI strategy. We hope our findings can raise awareness among researchers about the characteristics of multivariate time series and inspire the construction of better forecasting models. <<<

Diffusion-weighted magnetic resonance imaging (DW-MRI) is a non-invasive way of imaging white matter tracts in the human brain. DW-MRIs are usually acquired using echo-planar imaging (EPI) with high gradient fields, which could introduce severe geometric distortions that interfere with further analyses. Most tools for correcting distortion require two minimally weighted DW-MRI images (B0) acquired with different phase-encoding directions, and they can take hours to process per subject. Since a great amount of diffusion data are only acquired with a single phase-encoding direction, the application of existing approaches is limited. We propose a deep learning-based registration approach to correct distortion using only the B0 acquired from a single phase-encoding direction. Specifically, we register undistorted T1-weighted images and distorted B0 to remove the distortion through a deep learning model. We apply a differentiable mutual information loss during training to improve inter-modality alignment. Experiments on the Human Connectome Project dataset show the proposed method outperforms SyN and VoxelMorph on several metrics, and only takes a few seconds to process one subject. <<<

The pervasive uncertainty and dynamic nature of real-world environments present significant challenges for the widespread implementation of machine-driven Intelligent Decision-Making (IDM) systems. Consequently, IDM should possess the ability to continuously acquire new skills and effectively generalize across a broad range of applications. The advancement of Artificial General Intelligence (AGI) that transcends task and application boundaries is critical for enhancing IDM. Recent studies have extensively investigated the Transformer neural architecture as a foundational model for various tasks, including computer vision, natural language processing, and reinforcement learning. We propose that a Foundation Decision Model (FDM) can be developed by formulating diverse decision-making tasks as sequence decoding tasks using the Transformer architecture, offering a promising solution for expanding IDM applications in complex real-world situations. In this paper, we discuss the efficiency and generalization improvements offered by a foundation decision model for IDM and explore its potential applications in multi-agent game AI, production scheduling, and robotics tasks. Lastly, we present a case study demonstrating our FDM implementation, DigitalBrain (DB1) with 1.3 billion parameters, achieving human-level performance in 870 tasks, such as text generation, image captioning, video game playing, robotic control, and traveling salesman problems. As a foundation decision model, DB1 represents an initial step toward more autonomous and efficient real-world IDM applications. <<<

Diffusion probabilistic models have been shown to generate state-of-the-art results on several competitive image synthesis benchmarks but lack a low-dimensional, interpretable latent space, and are slow at generation. On the other hand, standard Variational Autoencoders (VAEs) typically have access to a low-dimensional latent space but exhibit poor sample quality. We present DiffuseVAE, a novel generative framework that integrates VAE within a diffusion model framework, and leverage this to design novel conditional parameterizations for diffusion models. We show that the resulting model equips diffusion models with a low-dimensional VAE inferred latent code which can be used for downstream tasks like controllable synthesis. The proposed method also improves upon the speed vs quality tradeoff exhibited in standard unconditional DDPM/DDIM models (for instance, FID of 16.47 vs 34.36 using a standard DDIM on the CelebA-HQ-128 benchmark using T=10 reverse process steps) without having explicitly trained for such an objective. Furthermore, the proposed model exhibits synthesis quality comparable to state-of-the-art models on standard image synthesis benchmarks like CIFAR-10 and CelebA-64 while outperforming most existing VAE-based methods. Lastly, we show that the proposed method exhibits inherent generalization to different types of noise in the conditioning signal. For reproducibility, our source code is publicly available at this https URL. <<<

Dimension reduction techniques for multivariate time series decompose the observed series into a few useful independent/orthogonal univariate components. We develop a spectral domain method for multivariate second-order stationary time series that linearly transforms the observed series into several groups of lower-dimensional multivariate subseries. These multivariate subseries have non-zero spectral coherence among components within a group but have zero spectral coherence among components across groups. The observed series is expressed as a sum of frequency components whose variances are proportional to the spectral matrices at the respective frequencies. The demixing matrix is then estimated using an eigendecomposition on the sum of the variance matrices of these frequency components and its asymptotic properties are derived. Finally, a consistent test on the cross-spectrum of pairs of components is used to find the desired segmentation into the lower-dimensional subseries. The numerical performance of the proposed method is illustrated through simulation examples and an application to modeling and forecasting wind data is presented. <<<

In addition to the shared weights of the synaptic connections, we proposed a new neural network that includes the synaptic effective range weights for both the forward and back propagation. And lots of simulations were used which RNN cannot be achieved. The simulations of PNN fit very well in experiments and hypotheses of 6 papers CNS Journals, 6 papers of CNS family Journals and 1 paper top Physics Journal [14-26]. The brain plasticity in positive or negative memory may be quantum and produce short-term memory, and exhibits an exponential decay in the wave function over a period of time, produced in the hippocampus. And exponential decay occurs due to barriers, and barriers can refer to astrocytes. Brain plasticity in working memory flows through the brain, from the hippocampus to the cortex, through directional derivatives. The strong working memory brain plasticity turns to long-term memory means maximum of directional derivatives, and maximum of directional derivatives is gradient. Thus, long-term memory signifies the gradient of brain plasticity in working memory. The process of short-term memory turns to long-term memory is the process of non-classically turns to classically. Astrocytic cortex memory persistence factor also inhibits local synaptic accumulation, and the model inspires experiments. This could be the process of astrocytes phagocytose synapses is driven by both positive and negative memories of plasticity in the brain. In simulation, it is possible that thicker cortices and more diverse individuals within the brain could have high IQ, but thickest cortices and most diverse individuals may have low IQ in simulation. PSO considers global solution or best previous solution, but also considers relatively good and relatively inferior solution. And PNN modified ResNet to consider memory gradient. The simple PNN only considers astrocytes phagocytosed synapses. <<<

Despite the success of Transformer models in vision and language tasks, they often learn knowledge from enormous data implicitly and cannot utilize structured input data directly. On the other hand, structured learning approaches such as graph neural networks (GNNs) that integrate prior information can barely compete with Transformer models. In this work, we aim to benefit from both worlds and propose a novel Multimodal Graph Transformer for question answering tasks that requires performing reasoning across multiple modalities. We introduce a graph-involved plug-and-play quasi-attention mechanism to incorporate multimodal graph information, acquired from text and visual data, to the vanilla self-attention as effective prior. In particular, we construct the text graph, dense region graph, and semantic graph to generate adjacency matrices, and then compose them with input vision and language features to perform downstream reasoning. Such a way of regularizing self-attention with graph information significantly improves the inferring ability and helps align features from different modalities. We validate the effectiveness of Multimodal Graph Transformer over its Transformer baselines on GQA, VQAv2, and MultiModalQA datasets. <<<

The amygdala is assumed to contribute to a bottom-up attentional bias during visual processing of emotional faces. Still, how its response to emotion interacts with top-down attention is not fully understood. It is also unclear if amygdala activity and scalp EEG respond to emotion and attention in a similar way. Therefore, we studied the interaction of emotion and attention during face processing in oscillatory gamma-band activity (GBA) in the amygdala and on the scalp. Amygdala signals were recorded via intracranial EEG (iEEG) in 9 patients with epilepsy. Scalp recordings were collected from 19 healthy participants. Three randomized blocks of angry, neutral, and happy faces were presented, and either negative, neutral, or positive expressions were denoted as targets. Both groups detected happy faces fastest and most accurately. In the amygdala, the earliest effect was observed around 170 ms in high GBA (105-117.5 Hz) when neutral faces served as targets. Here, GBA was higher for emotional than neutral faces. During attention to negative faces, low GBA (< 90 Hz) increased specifically for angry faces both in the amygdala and over posterior scalp regions, albeit earlier on the scalp (60 ms) than in the amygdala (210 ms). From 570 ms, amygdala high GBA (117.5-145 Hz) was also increased for both angry and neutral, compared to happy, faces. When positive faces were the targets, GBA did not differentiate between expressions. The present data reveal that attention-independent emotion detection in amygdala high GBA may only occur during a neutral focus of attention. Top-down threat vigilance coordinates widespread low GBA, biasing stimulus processing in favor of negative faces. These results are in line with a multi-pathway model of emotion processing and help specify the role of GBA in this process by revealing how attentional focus can tune timing and amplitude of emotional GBA responses. <<<

Deformable image registration is one of the fundamental tasks in medical imaging. Classical registration algorithms usually require a high computational cost for iterative optimizations. Although deep-learning-based methods have been developed for fast image registration, it is still challenging to obtain realistic continuous deformations from a moving image to a fixed image with less topological folding problem. To address this, here we present a novel diffusion-model-based image registration method, called DiffuseMorph. DiffuseMorph not only generates synthetic deformed images through reverse diffusion but also allows image registration by deformation fields. Specifically, the deformation fields are generated by the conditional score function of the deformation between the moving and fixed images, so that the registration can be performed from continuous deformation by simply scaling the latent feature of the score. Experimental results on 2D facial and 3D medical image registration tasks demonstrate that our method provides flexible deformations with topology preservation capability. <<<