Ensuring that large language models (LLMs) generate outputs aligned with<br>human preferences is important for safe and effective AI interactions. While<br>Direct Preference Optimization (DPO) employs an implicit reward function to<br>optimize the policy model, however, it and its related variants overlook the<br>differential importance of individual tokens and are sensitive to judgment<br>noise in preference datasets during generation. Although recent methods attempt<br>to assess the important weight of tokens via probability prediction or<br>simplistic weighting schemes, these evaluation methods are prone to biases and<br>still cannot fully address these issues. To solve this problem, we propose the<br>Token-Importance Guided Direct Preference Optimization (TI-DPO), which<br>introduces two key innovations: the gradient-based token-importance weights<br>that dynamically prioritize critical tokens, and a triple loss that explicitly<br>guides model outputs to approach human-preferred responses and stay away from<br>non-preferred responses. Experimental results show that TI-DPO achieves higher<br>accuracy and stronger generative diversity, providing more stable and<br>computationally efficient solutions compared with DPO and other RLHF methods.