Maintenance scheduling of stack replacement is indispensable for achieving the durable and reliable operation of fuel cell systems. However, the related maintenance actions for minimizing long-term risk and cost in fuel cells constitute a complex optimization problem. Moreover, the complicated stochastic degradation behavior of fuel cell stacks hid the development of maintenance scheduling policies. Thus, the action phase of maintenance scheduling in fuel cell prognostics and health management (PHM) studies and stochastic degradation modeling are identified as two major research gaps. This paper investigates the maintenance scheduling of a stochastically deteriorating fuel cell stack exposed to deterioration heterogeneity. The maintenance problem consists in finding preventive control limits and inspection intervals that minimize total maintenance costs. A proximal policy optimization (PPO) based method is used to develop the maintenance strategy by describing the maintenance scheduling problem as a Markov decision process (MDP). Investigation results show that a steeper preventive control limit and more conservative inspection intervals are needed when heterogeneity exists. The proposed PPO-based method can solve the MDP and obtain more general maintenance policies considering the stochastic deterioration of fuel cell systems. The formulation of a condition-based maintenance scheduling problem, the reinforcement learning-based solution, and the investigation of stack-to-stack degradation heterogeneity’s influence constitute this work’s main novelties. This study provides useful insights for the development of fuel cell PHM studies, particularly within the action phase, helping to enhance the durability and reliability of fuel cell systems by better operating them. So the paper could be useful for different fuel cell applications. Meanwhile, the framework discussed in the paper, i.e., developing reinforcement learning for maintenance planning, is generic enough to be referred to in other related applications.