In the modern world of IT technologies, there is a trend of ever-increasing flow of network traffic, network connections and, consequently, a growing number of vulnerabilities of centralized and decentralized systems. The urgency of the research lies in the necessity to modernize and improve existing mechanisms for better malicious traffic detection and enhanced security of the entire network infrastructure. The paper presents a new approach to network traffic research. The advantages of the proposed techniques are given in comparison with modern intrusion detection system based on standard algorithms and intelligent methods. The article indicates the direction in the area of modernization and improvement of algorithms for detection of network anomalies and network intrusions. The main features of the network traffic classification subsystem and the logic of work of each stage are displayed, the results of the system research and testing are presented, recommendations on the application and practical significance of the developed algorithm are described.