The utilization of Unmanned Aerial Vehicles (UAV) has rapidly been expanding globally. Due to their affordability and ease of operation, UAV are increasingly prevalent and provide innovative solutions across various industrial sectors. However, as their usage expands, security concerns regarding UAV have been critical, especially in business and military applications where safety and security are essential. Additionally, the development of artificial intelligence (AI) technologies (ex, machine learning and computer vision) significantly contributes to the autonomous flight and operational capabilities of UAV. Yet, these advancements also cause new security vulnerabilities within AI systems, and the vulnerabilities could be exploited by malicious attackers to infiltrate and manipulate UAS, leading to severe consequences. This paper investigates the current state of security attacks targeting AI in UAV and explores countermeasures to address these challenges. Therefore, utilizing the results of this paper can contribute to laying the foundation for safe and reliable UAV operation.