Vision-language-action (VLA) models have emerged as the next generation ofmodels in robotics. However, despite leveraging powerful pre-trainedVision-Language Models (VLMs), existing end-to-end VLA systems often lose keycapabilities during fine-tuning as the model adapts to specific robotic tasks.We argue that a generalizable VLA model should retain and expand upon the VLM'score competencies: 1) Open-world embodied reasoning - the VLA should inheritthe knowledge from VLM, i.e., recognize anything that the VLM can recognize,capable of solving math problems, possessing visual-spatial intelligence, 2)Reasoning following - effectively translating the open-world reasoning intoactionable steps for the robot. In this work, we introduce ChatVLA-2, a novelmixture-of-expert VLA model coupled with a specialized three-stage trainingpipeline designed to preserve the VLM's original strengths while enablingactionable reasoning. To validate our approach, we design a math-matching taskwherein a robot interprets math problems written on a whiteboard and pickscorresponding number cards from a table to solve equations. Remarkably, ourmethod exhibits exceptional mathematical reasoning and OCR capabilities,despite these abilities not being explicitly trained within the VLA.Furthermore, we demonstrate that the VLA possesses strong spatial reasoningskills, enabling it to interpret novel directional instructions involvingpreviously unseen objects. Overall, our method showcases reasoning andcomprehension abilities that significantly surpass state-of-the-art imitationlearning methods such as OpenVLA, DexVLA, and pi-zero. This work represents asubstantial advancement toward developing truly generalizable roboticfoundation models endowed with robust reasoning capacities. <<<

Despite recent progress in general purpose robotics, robot policies still lagfar behind basic human capabilities in the real world. Humans interactconstantly with the physical world, yet this rich data resource remains largelyuntapped in robot learning. We propose EgoZero, a minimal system that learnsrobust manipulation policies from human demonstrations captured with ProjectAria smart glasses, $\textbf{and zero robot data}$. EgoZero enables: (1)extraction of complete, robot-executable actions from in-the-wild, egocentric,human demonstrations, (2) compression of human visual observations intomorphology-agnostic state representations, and (3) closed-loop policy learningthat generalizes morphologically, spatially, and semantically. We deployEgoZero policies on a gripper Franka Panda robot and demonstrate zero-shottransfer with 70% success rate over 7 manipulation tasks and only 20 minutes ofdata collection per task. Our results suggest that in-the-wild human data canserve as a scalable foundation for real-world robot learning - paving the waytoward a future of abundant, diverse, and naturalistic training data forrobots. Code and videos are available at https://egozero-robot.github.io. <<<

Many real-world applications require the prediction of long sequencetime-series, such as electricity consumption planning. Long sequencetime-series forecasting (LSTF) demands a high prediction capacity of the model,which is the ability to capture precise long-range dependency coupling betweenoutput and input efficiently. Recent studies have shown the potential ofTransformer to increase the prediction capacity. However, there are severalsevere issues with Transformer that prevent it from being directly applicableto LSTF, including quadratic time complexity, high memory usage, and inherentlimitation of the encoder-decoder architecture. To address these issues, wedesign an efficient transformer-based model for LSTF, named Informer, withthree distinctive characteristics: (i) a $ProbSparse$ self-attention mechanism,which achieves $O(L \log L)$ in time complexity and memory usage, and hascomparable performance on sequences' dependency alignment. (ii) theself-attention distilling highlights dominating attention by halving cascadinglayer input, and efficiently handles extreme long input sequences. (iii) thegenerative style decoder, while conceptually simple, predicts the longtime-series sequences at one forward operation rather than a step-by-step way,which drastically improves the inference speed of long-sequence predictions.Extensive experiments on four large-scale datasets demonstrate that Informersignificantly outperforms existing methods and provides a new solution to theLSTF problem. <<<

This paper introduces a novel trajectory planner for autonomous robots,specifically designed to enhance navigation by incorporating dynamic obstacleavoidance within the Robot Operating System 2 (ROS2) and Navigation 2 (Nav2)framework. The proposed method utilizes Model Predictive Control (MPC) with afocus on handling the uncertainties associated with the movement prediction ofdynamic obstacles. Unlike existing Nav2 trajectory planners which primarilydeal with static obstacles or react to the current position of dynamicobstacles, this planner predicts future obstacle positions using a stochasticVector Auto-Regressive Model (VAR). The obstacles' future positions arerepresented by probability distributions, and collision avoidance is achievedthrough constraints based on the Mahalanobis distance, ensuring the robotavoids regions where obstacles are likely to be. This approach considers therobot's kinodynamic constraints, enabling it to track a reference path whileadapting to real-time changes in the environment. The paper details theimplementation, including obstacle prediction, tracking, and the constructionof feasible sets for MPC. Simulation results in a Gazebo environmentdemonstrate the effectiveness of this method in scenarios where robots mustnavigate around each other, showing improved collision avoidance capabilities. <<<

Generative AI technologies have been deployed in many places, such as(multimodal) large language models and vision generative models. Theirremarkable performance should be attributed to massive training data andemergent reasoning abilities. However, the models would memorize and generatesensitive, biased, or dangerous information originated from the training dataespecially those from web crawl. New machine unlearning (MU) techniques arebeing developed to reduce or eliminate undesirable knowledge and its effectsfrom the models, because those that were designed for traditionalclassification tasks could not be applied for Generative AI. We offer acomprehensive survey on many things about MU in Generative AI, such as a newproblem formulation, evaluation methods, and a structured discussion on theadvantages and limitations of different kinds of MU techniques. It alsopresents several critical challenges and promising directions in MU research. Acurated list of readings can be found:https://github.com/franciscoliu/GenAI-MU-Reading. <<<

Large Language Models (LLMs) and LLM-based agents show great promise inaccelerating scientific research. Existing benchmarks for measuring thispotential and guiding future development continue to evolve from pure recalland rote knowledge tasks, towards more practical work such as literature reviewand experimental planning. Bioinformatics is a domain where fully autonomousAI-driven discovery may be near, but no extensive benchmarks for measuringprogress have been introduced to date. We therefore present the BioinformaticsBenchmark (BixBench), a dataset comprising over 50 real-world scenarios ofpractical biological data analysis with nearly 300 associated open-answerquestions designed to measure the ability of LLM-based agents to explorebiological datasets, perform long, multi-step analytical trajectories, andinterpret the nuanced results of those analyses. We evaluate the performance oftwo frontier LLMs (GPT-4o and Claude 3.5 Sonnet) using a custom agent frameworkwe open source. We find that even the latest frontier models only achieve 17%accuracy in the open-answer regime, and no better than random in amultiple-choice setting. By exposing the current limitations of frontiermodels, we hope BixBench can spur the development of agents capable ofconducting rigorous bioinformatic analysis and accelerate scientific discovery. <<<

Image inversion is a fundamental task in generative models, aiming to mapimages back to their latent representations to enable downstream applicationssuch as editing, restoration, and style transfer. This paper provides acomprehensive review of the latest advancements in image inversion techniques,focusing on two main paradigms: Generative Adversarial Network (GAN) inversionand diffusion model inversion. We categorize these techniques based on theiroptimization methods. For GAN inversion, we systematically classify existingmethods into encoder-based approaches, latent optimization approaches, andhybrid approaches, analyzing their theoretical foundations, technicalinnovations, and practical trade-offs. For diffusion model inversion, weexplore training-free strategies, fine-tuning methods, and the design ofadditional trainable modules, highlighting their unique advantages andlimitations. Additionally, we discuss several popular downstream applicationsand emerging applications beyond image tasks, identifying current challengesand future research directions. By synthesizing the latest developments, thispaper aims to provide researchers and practitioners with a valuable referenceresource, promoting further advancements in the field of image inversion. Wekeep track of the latest works at https://github.com/RyanChenYN/ImageInversion <<<

Large Vision-Language-Action (VLA) models, leveraging powerful pre trainedVision-Language Models (VLMs) backends, have shown promise in robotic controldue to their impressive generalization ability. However, the success comes at acost. Their reliance on VLM backends with billions of parameters leads to highcomputational costs and inference latency, limiting the testing scenarios tomainly quasi-static tasks and hindering performance in dynamic tasks requiringrapid interactions. To address these limitations, this paper proposes HiRT, aHierarchical Robot Transformer framework that enables flexible frequency andperformance trade-off. HiRT keeps VLMs running at low frequencies to capturetemporarily invariant features while enabling real-time interaction through ahigh-frequency vision-based policy guided by the slowly updated features.Experiment results in both simulation and real-world settings demonstratesignificant improvements over baseline methods. Empirically, in static tasks,we double the control frequency and achieve comparable success rates.Additionally, on novel real-world dynamic ma nipulation tasks which arechallenging for previous VLA models, HiRT improves the success rate from 48% to75%. <<<

To train generalist robot policies, machine learning methods often require asubstantial amount of expert human teleoperation data. An ideal robot forhumans collecting data is one that closely mimics them: bimanual arms anddexterous hands. However, creating such a bimanual teleoperation system withover 50 DoF is a significant challenge. To address this, we introduce Bidex, anextremely dexterous, low-cost, low-latency and portable bimanual dexterousteleoperation system which relies on motion capture gloves and teacher arms. Wecompare Bidex to a Vision Pro teleoperation system and a SteamVR system andfind Bidex to produce better quality data for more complex tasks at a fasterrate. Additionally, we show Bidex operating a mobile bimanual robot for in thewild tasks. The robot hands (5k USD) and teleoperation system (7k USD) isreadily reproducible and can be used on many robot arms including two xArms(16k USD). Website at https://bidex-teleop.github.io/ <<<

Denoising diffusion probabilistic models (DDPM) are powerful hierarchicallatent variable models with remarkable sample generation quality and trainingstability. These properties can be attributed to parameter sharing in thegenerative hierarchy, as well as a parameter-free diffusion-based inferenceprocedure. In this paper, we present Few-Shot Diffusion Models (FSDM), aframework for few-shot generation leveraging conditional DDPMs. FSDMs aretrained to adapt the generative process conditioned on a small set of imagesfrom a given class by aggregating image patch information using a set-basedVision Transformer (ViT). At test time, the model is able to generate samplesfrom previously unseen classes conditioned on as few as 5 samples from thatclass. We empirically show that FSDM can perform few-shot generation andtransfer to new datasets. We benchmark variants of our method on complex visiondatasets for few-shot learning and compare to unconditional and conditionalDDPM baselines. Additionally, we show how conditioning the model on patch-basedinput set information improves training convergence. <<<

Hex and Counter Wargames are adversarial two-player simulations of realmilitary conflicts requiring complex strategic decision-making. Unlikeclassical board games, these games feature intricate terrain/unit interactions,unit stacking, large maps of varying sizes, and simultaneous move and combatdecisions involving hundreds of units. This paper introduces a novel systemdesigned to address the strategic complexity of Hex and Counter Wargames byintegrating cutting-edge advancements in Recurrent Neural Networks withAlphaZero, a reliable modern Reinforcement Learning algorithm. The systemutilizes a new Neural Network architecture developed from existing research,incorporating innovative state and action representations tailored to thesespecific game environments. With minimal training, our solution has shownpromising results in typical scenarios, demonstrating the ability to generalizeacross different terrain and tactical situations. Additionally, we explore thesystem's potential to scale to larger map sizes. The developed system is openlyaccessible, facilitating continued research and exploration within thischallenging domain. <<<

DeepSeek-AI, Aixin Liu, Bei Feng, Bin Wang, Bingxuan Wang, Bo Liu, Chenggang Zhao, Chengqi Dengr, Chong Ruan, Damai Dai, Daya Guo, Dejian Yang, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Hanwei Xu, Hao Yang, Haowei Zhang, Honghui Ding, Huajian Xin, Huazuo Gao, Hui Li, Hui Qu, J. L. Cai, Jian Liang, Jianzhong Guo, Jiaqi Ni, Jiashi Li, Jin Chen, Jingyang Yuan, Junjie Qiu, Junxiao Song, Kai Dong, Kaige Gao, Kang Guan, Lean Wang, Lecong Zhang, Lei Xu, Leyi Xia, Liang Zhao, Liyue Zhang, Meng Li, Miaojun Wang, Mingchuan Zhang, Minghua Zhang, Minghui Tang, Mingming Li, Ning Tian, Panpan Huang, Peiyi Wang, Peng Zhang, Qihao Zhu, Qinyu Chen, Qiushi Du, R. J. Chen, R. L. Jin, Ruiqi Ge, Ruizhe Pan, Runxin Xu, Ruyi Chen, S. S. Li, Shanghao Lu, Shangyan Zhou, Shanhuang Chen, Shaoqing Wu, Shengfeng Ye, Shirong Ma, Shiyu Wang, Shuang Zhou, Shuiping Yu, Shunfeng Zhou, Size Zheng, T. Wang, Tian Pei, Tian Yuan, Tianyu Sun, W. L. Xiao, Wangding Zeng, Wei An, Wen Liu, Wenfeng Liang, Wenjun Gao, Wentao Zhang, X. Q. Li, Xiangyue Jin, Xianzu Wang, Xiao Bi, Xiaodong Liu, Xiaohan Wang, Xiaojin Shen, Xiaokang Chen, Xiaosha Chen, Xiaotao Nie, Xiaowen Sun, Xiaoxiang Wang, Xin Liu, Xin Xie, Xingkai Yu, Xinnan Song, Xinyi Zhou, Xinyu Yang, Xuan Lu, Xuecheng Su, Y. Wu, Y. K. Li, Y. X. Wei, Y. X. Zhu, Yanhong Xu, Yanping Huang, Yao Li, Yao Zhao, Yaofeng Sun, Yaohui Li, Yaohui Wang, Yi Zheng, Yichao Zhang, Yiliang Xiong, Yilong Zhao, Ying He, Ying Tang, Yishi Piao, Yixin Dong, Yixuan Tan, Yiyuan Liu, Yongji Wang, Yongqiang Guo, Yuchen Zhu, Yuduan Wang, Yuheng Zou, Yukun Zha, Yunxian Ma, Yuting Yan, Yuxiang You, Yuxuan Liu, Z. Z. Ren, Zehui Ren, Zhangli Sha, Zhe Fu, Zhen Huang, Zhen Zhang, Zhenda Xie, Zhewen Hao, Zhihong Shao, Zhiniu Wen, Zhipeng Xu, Zhongyu Zhang, Zhuoshu Li, Zihan Wang, Zihui Gu, Zilin Li, Ziwei Xie <<<

We present DeepSeek-V2, a strong Mixture-of-Experts (MoE) language modelcharacterized by economical training and efficient inference. It comprises 236Btotal parameters, of which 21B are activated for each token, and supports acontext length of 128K tokens. DeepSeek-V2 adopts innovative architecturesincluding Multi-head Latent Attention (MLA) and DeepSeekMoE. MLA guaranteesefficient inference through significantly compressing the Key-Value (KV) cacheinto a latent vector, while DeepSeekMoE enables training strong models at aneconomical cost through sparse computation. Compared with DeepSeek 67B,DeepSeek-V2 achieves significantly stronger performance, and meanwhile saves42.5% of training costs, reduces the KV cache by 93.3%, and boosts the maximumgeneration throughput to 5.76 times. We pretrain DeepSeek-V2 on a high-qualityand multi-source corpus consisting of 8.1T tokens, and further performSupervised Fine-Tuning (SFT) and Reinforcement Learning (RL) to fully unlockits potential. Evaluation results show that, even with only 21B activatedparameters, DeepSeek-V2 and its chat versions still achieve top-tierperformance among open-source models. <<<

NP-hard problems regularly come up in video games, with interestingconnections to real-world problems. In the game Minecraft, players placetorches on the ground to light up dark areas. Placing them in a way thatminimizes the total number of torches to save resources is far from trivial. Inthis paper, we use Quantum Computing to approach this problem. To this end, wederive a QUBO formulation of the torch placement problem, which we uncover tobe very similar to another NP-hard problem. We employ a solution strategy thatinvolves learning Lagrangian weights in an iterative process, adding to theever growing toolbox of QUBO formulations. Finally, we perform experiments onreal quantum hardware using real game data to demonstrate that our approachyields good torch placements. <<<

This article is about the neural conundrum behind the slowness of humanbehavior. The information throughput of a human being is about 10 bits/s. Incomparison, our sensory systems gather data at ~10^9 bits/s. The stark contrastbetween these numbers remains unexplained and touches on fundamental aspects ofbrain function: What neural substrate sets this speed limit on the pace of ourexistence? Why does the brain need billions of neurons to process 10 bits/s?Why can we only think about one thing at a time? The brain seems to operate intwo distinct modes: the "outer" brain handles fast high-dimensional sensory andmotor signals, whereas the "inner" brain processes the reduced few bits neededto control behavior. Plausible explanations exist for the large neuron numbersin the outer brain, but not for the inner brain, and we propose new researchdirections to remedy this. <<<

Weixin Liang, Zachary Izzo, Yaohui Zhang, Haley Lepp, Hancheng Cao, Xuandong Zhao, Lingjiao Chen, Haotian Ye, Sheng Liu, Zhi Huang, Daniel A. McFarland, James Y. Zou <<<

We present an approach for estimating the fraction of text in a large corpuswhich is likely to be substantially modified or produced by a large languagemodel (LLM). Our maximum likelihood model leverages expert-written andAI-generated reference texts to accurately and efficiently examine real-worldLLM-use at the corpus level. We apply this approach to a case study ofscientific peer review in AI conferences that took place after the release ofChatGPT: ICLR 2024, NeurIPS 2023, CoRL 2023 and EMNLP 2023. Our results suggestthat between 6.5% and 16.9% of text submitted as peer reviews to theseconferences could have been substantially modified by LLMs, i.e. beyondspell-checking or minor writing updates. The circumstances in which generatedtext occurs offer insight into user behavior: the estimated fraction ofLLM-generated text is higher in reviews which report lower confidence, weresubmitted close to the deadline, and from reviewers who are less likely torespond to author rebuttals. We also observe corpus-level trends in generatedtext which may be too subtle to detect at the individual level, and discuss theimplications of such trends on peer review. We call for futureinterdisciplinary work to examine how LLM use is changing our information andknowledge practices. <<<

This paper shows that masked autoencoders (MAE) are scalable self-supervisedlearners for computer vision. Our MAE approach is simple: we mask randompatches of the input image and reconstruct the missing pixels. It is based ontwo core designs. First, we develop an asymmetric encoder-decoder architecture,with an encoder that operates only on the visible subset of patches (withoutmask tokens), along with a lightweight decoder that reconstructs the originalimage from the latent representation and mask tokens. Second, we find thatmasking a high proportion of the input image, e.g., 75%, yields a nontrivialand meaningful self-supervisory task. Coupling these two designs enables us totrain large models efficiently and effectively: we accelerate training (by 3xor more) and improve accuracy. Our scalable approach allows for learninghigh-capacity models that generalize well: e.g., a vanilla ViT-Huge modelachieves the best accuracy (87.8%) among methods that use only ImageNet-1Kdata. Transfer performance in downstream tasks outperforms supervisedpre-training and shows promising scaling behavior. <<<

Most robotic hands and grippers rely on actuators with large gearboxes andforce sensors for controlling gripping force. However, this might not be idealfor tasks that require the robot to interact with an unstructured and unknownenvironment. In this paper, we introduce a novel quasi-direct-drivetwo-fingered robotic hand with variable impedance control in the joint spaceand Cartesian space. The hand has a total of four degrees of freedom,backdrivable differential gear trains, and four brushless direct current (BLDC)motors. Motor torque is controlled through Field-Oriented Control (FOC) withcurrent sensing. Variable impedance control enables the robotic hand to executedexterous manipulation tasks safely during environment-robot and human-robotinteractions. The quasi-direct-drive actuators eliminate the need for complextactile/force sensors or precise motion planning when handling environmentalcontact. A majority-3D-printed assembly makes this a low-cost research platformbuilt with affordable, readily available off-the-shelf components. Experimentalvalidation demonstrates the robotic hand's capability for stable force-closureand form-closure grasps in the presence of disturbances, reliable in-handmanipulation, and safe dynamic manipulations despite contact with theenvironment. <<<

This paper proposes a new game-search algorithm, PN-MCTS, which combinesMonte-Carlo Tree Search (MCTS) and Proof-Number Search (PNS). These twoalgorithms have been successfully applied for decision making in a range ofdomains. We define three areas where the additional knowledge provided by theproof and disproof numbers gathered in MCTS trees might be used: final moveselection, solving subtrees, and the UCB1 selection mechanism. We test allpossible combinations on different time settings, playing against vanilla UCTon several games: Lines of Action ($7$$\times$$7$ and $8$$\times$$8$ boardsizes), MiniShogi, Knightthrough, and Awari. Furthermore, we extend this newalgorithm to properly address games with draws, like Awari, by adding anadditional layer of PNS on top of the MCTS tree. The experiments show thatPN-MCTS is able to outperform MCTS in all tested game domains, achieving winrates up to 96.2% for Lines of Action. <<<

This work considers the category distribution heterogeneity in federatedlearning. This issue is due to biased labeling preferences at multiple clientsand is a typical setting of data heterogeneity. To alleviate this issue, mostprevious works consider either regularizing local models or fine-tuning theglobal model, while they ignore the adjustment of aggregation weights andsimply assign weights based on the dataset size. However, based on ourempirical observations and theoretical analysis, we find that the dataset sizeis not optimal and the discrepancy between local and global categorydistributions could be a beneficial and complementary indicator for determiningaggregation weights. We thus propose a novel aggregation method, FederatedLearning with Discrepancy-aware Collaboration (FedDisco), whose aggregationweights not only involve both the dataset size and the discrepancy value, butalso contribute to a tighter theoretical upper bound of the optimization error.FedDisco also promotes privacy-preservation, communication and computationefficiency, as well as modularity. Extensive experiments show that our FedDiscooutperforms several state-of-the-art methods and can be easily incorporatedwith many existing methods to further enhance the performance. Our code will beavailable at https://github.com/MediaBrain-SJTU/FedDisco. <<<

This paper shows that masked autoencoders (MAE) are scalable self-supervisedlearners for computer vision. Our MAE approach is simple: we mask randompatches of the input image and reconstruct the missing pixels. It is based ontwo core designs. First, we develop an asymmetric encoder-decoder architecture,with an encoder that operates only on the visible subset of patches (withoutmask tokens), along with a lightweight decoder that reconstructs the originalimage from the latent representation and mask tokens. Second, we find thatmasking a high proportion of the input image, e.g., 75%, yields a nontrivialand meaningful self-supervisory task. Coupling these two designs enables us totrain large models efficiently and effectively: we accelerate training (by 3xor more) and improve accuracy. Our scalable approach allows for learninghigh-capacity models that generalize well: e.g., a vanilla ViT-Huge modelachieves the best accuracy (87.8%) among methods that use only ImageNet-1Kdata. Transfer performance in downstream tasks outperforms supervisedpre-training and shows promising scaling behavior. <<<