At present, there is a strong drive towards development of autonomous ships and other marine systems. Some commercial, large size, long voyage, highly autonomous ships are under development and plan to operate in coming years. Many discussions exist about the human factors in autonomous ships because humans are expected to continue to play some roles in ship operations. However, the increased autonomy will challenge the performance of human operators. In this paper, we have identified issues impacting on operators' performance through the whole process from problem recognition, decision-making to action implementation. Those issues include, but are not limited to, level of autonomy design, remote operation, transition of operating modes, collaboration between crews and autonomy, teamwork between crews, goal and sub-goal omission, operation rules following, uncertainties, mental constraints and characteristics, and changes during operations. Those issues are integrated in a descriptive, holistic representation to illustrate the complication of human factor issues in the operation phase at the same time provides guidance to solutions. Both design-based solutions and operation-based solutions are identified in order to resolve the human factor challenges.