Ensuring that large language models (LLMs) generate outputs aligned withhuman preferences is important for safe and effective AI interactions. WhileDirect Preference Optimization (DPO) employs an implicit reward function tooptimize the policy model, however, it and its related variants overlook thedifferential importance of individual tokens and are sensitive to judgmentnoise in preference datasets during generation. Although recent methods attemptto assess the important weight of tokens via probability prediction orsimplistic weighting schemes, these evaluation methods are prone to biases andstill cannot fully address these issues. To solve this problem, we propose theToken-Importance Guided Direct Preference Optimization (TI-DPO), whichintroduces two key innovations: the gradient-based token-importance weightsthat dynamically prioritize critical tokens, and a triple loss that explicitlyguides model outputs to approach human-preferred responses and stay away fromnon-preferred responses. Experimental results show that TI-DPO achieves higheraccuracy and stronger generative diversity, providing more stable andcomputationally efficient solutions compared with DPO and other RLHF methods.