Large Vision-Language-Action (VLA) models, leveraging powerful pre trainedVision-Language Models (VLMs) backends, have shown promise in robotic controldue to their impressive generalization ability. However, the success comes at acost. Their reliance on VLM backends with billions of parameters leads to highcomputational costs and inference latency, limiting the testing scenarios tomainly quasi-static tasks and hindering performance in dynamic tasks requiringrapid interactions. To address these limitations, this paper proposes HiRT, aHierarchical Robot Transformer framework that enables flexible frequency andperformance trade-off. HiRT keeps VLMs running at low frequencies to capturetemporarily invariant features while enabling real-time interaction through ahigh-frequency vision-based policy guided by the slowly updated features.Experiment results in both simulation and real-world settings demonstratesignificant improvements over baseline methods. Empirically, in static tasks,we double the control frequency and achieve comparable success rates.Additionally, on novel real-world dynamic ma nipulation tasks which arechallenging for previous VLA models, HiRT improves the success rate from 48% to75%.