Dynamic Power Allocation Strategy for Hybrid Energy Storage System of Urban Rail Trains Based on Improved SAC Algorithm

doi:10.3969/j.issn.1008-0198.2024.04.002

Abstract

Abstract: To smooth out voltage fluctuations in the traction network of urban rail trains, a power dynamic allocation strategy based on a soft actor-critic (SAC) with reinforcement learning is proposed based on the use of on-board supercapacitor and ground hybrid energy storage system. It is used to improve the energy-saving voltage stabilization characteristics of DC traction network and realize the life protection of on-board supercapacitor. Firstly, an urban rail train dynamics model is established, and the PEC-SAC algorithm is proposed to address the problems of long training time and slow convergence of SAC algorithm in urban rail dynamic power allocation. The algorithm combines prioritized experience replay, emphasizing recent experience and cosine annealing, and improves the learning rate by increasing the sampling probability of the recent experience and dynamically adjusting the learning rate, which improves the training efficiency and convergence speed. Then the state space, action space, and reward function are set up to realize that the train learns the optimal energy control strategy for the hybrid energy storage system in interaction with the simulation environment. The simulation platform is built through the joint simulation of MATLAB/Simulink and PYTHON, and the results show that the method improves the voltage stabilization by 0.36% and reduces the energy consumption by 4.52% compared to the SAC algorithm.

Key words: urban rail train, regenerative braking energy, hybrid energy storage system, power dynamic allocation, deep reinforcement learning

CLC Number:

TM911

HE Qingchen, QIN Bin. Dynamic Power Allocation Strategy for Hybrid Energy Storage System of Urban Rail Trains Based on Improved SAC Algorithm[J]. Hunan Electric Power, 2024, 44(4): 11-19.

References

[1] 周浪雅,王亦乐,谢余晨,等.站城融合背景下高速铁路综合枢纽短时客流预测研究[J].铁道学报,2023,45(4):1-7.
[2] QI L F,PAN H Y,PAN Y J,et al.A review of vibration energy harvesting in rail transportation field[J]. iScience, 2022,25(3):103849.
[3] 侯秀芳,冯晨,燕汉民,等.2023年中国内地城市轨道交通运营线路概况[J].都市快轨交通,2024, 37(1):10-16.
[4] 李志强,胡海涛,陈俊宇,等.城轨交通混合型再生制动能量利用系统及其控制策略[J].电力自动化设备,2023,43(11):1-14.
[5] 沈小军,曹戈.城轨交通制动能量回收超级电容储能阵列配置方法对比分析[J].电工技术学报, 2020,35(23):4988-4997.
[6] 郑文奇.城市轨道交通车载混合储能系统的控制策略及容量优化配置研究[D].南昌:华东交通大学,2020.
[7] KHODAPARASTAN M, MOHAMED A A, BRANDAUER W. Recuperation of regenerative braking energy in electric rail transit systems[J]. IEEE Transactions on Intelligent Transportation Systems, 2019, 20(8): 2831-2847.
[8] 章宝歌,李萍,张振,等.应用于城轨列车混合储能系统的能量管理策略[J].储能科学与技术,2020, 9(1):204-210.
[9] 朱志强,王欣,秦斌.城市轨道交通供电-牵引-混合储能系统设计与仿真[J].湖南电力,2024, 44(1):24-31.
[10] WANG X,LUO Y B,QIN B, et al. Power dynamic allocation strategy for urban rail hybrid energy storage system based on iterative learning control [J]. Energy, 2022, 245: 123263.
[11] 杨浩丰,刘冲,李彬,等.基于列车运行工况的城轨地面式混合储能系统控制策略研究[J].电工技术学报,2021,36(S1):168-178.
[12] LANCTOT M, LOCKHART E, LESPIAU J B, et al. Openspiel: a framework for reinforcement learning in games[EB/OL].[ 2020-09-29].https://arxiv.org/abs/1908.09453.
[13] LEVINE S, FINN C, DARRELL T, et al. End-to-end training of deep visuomotor policies[J]. The Journal of Machine Learning Research, 2016, 17(1): 1334-1373.
[14] YANG Z, ZHU F, LIN F. Deep-reinforcement-learning-based energy management strategy for supercapacitor energy storage systems in urban rail transit [J]. IEEE Transactions on Intelligent Transportation Systems, 2021, 22(2): 1150-1160.
[15] ABDELHEDI R,LAHYANI A,AMMARI A C,et al.Reinforcement learning-based power sharing between batteries and supercapacitors in electric vehicles[C] //2018 IEEE International Conference on Industrial Technology (ICIT).Lyon,France.IEEE,2018:2072-2077.
[16] FUJIMOTO S,VAN HOOF H,MEGER D.Addressing function approximation error in actor-critic methods [C] //35th International Conference on Machine Learning,Stockholm,Sweden.PMLR,2018,80:1587-1596 .
[17] WANG X,LUO Y,QIN B,et al.Power allocation strategy for urban rail hess based on deep reinforcement learning sequential decision optimization[J]. IEEE Transactions on Transportation Electrification, 2023, 9(2): 2693-2710.
[18] HAARNOJA T,ZHOU A,HARTIKAINEN K,et al.Soft actor-critic algorithms and applications[EB/OL].[2019-01-29].https://arxiv.org/pdf/1812.05905/1000.
[19] HAARNOJA T,ZHOU A,ABBEEL P,et al.Soft actor-critic: off-policy maximum entropy deep reinforcement learning with a stochastic actor[C] //35th International Conference on Machine Learning,Stockholm,Sweden.PMLR,2018,80:1861-1870.