As infrastructure systems such as transportation and water distribution networks deteriorate due to aging and corrosion, decision-makers need to assess the system-level risk to devise an appropriate operation plan to minimize the losses caused, by system failure over the lifecycle. Recently, Markov Decision Processes (MDP) has been utilized to identify optimal, decision-making policies efficiently. However, in complex systems consisting of many components, it is often intractable, to find the best solutions because the numbers of state and action spaces increase exponentially. To overcome the curse, of dimensionality, this study develops a multi-agent deep reinforcement learning framework termed Clustering-based, Cooperative Multi-Agent (CCMA) Deep Q-Network. CCMA takes a divide-and-conquer strategy, which identifies, multiple subsystems by clustering and assigns an agent to each subsystem. Each agent observes states of the structures, within the corresponding cluster to pursue appropriate actions and share information about the cluster with other agents. A numerical example demonstrates that the proposed method outperforms conventional maintenance schemes and, subsystem-level optimal policies.