maltrail

Maltrail is an open-source malicious traffic detection system developed by Marin Kaluža. Designed for network administrators and security analysts, it helps detect and analyze network anomalies, including suspicious or malicious traffic patterns. Maltrail uses threat intelligence from a variety of sources, including public blacklists, malicious domains, IP addresses, and behavioral signatures, to identify potential threats within network traffic.

Key Features of Maltrail

1. Threat Intelligence Sources: Utilizes various threat intelligence feeds from public blacklists and custom rules, helping detect known malicious IPs, domains, and behaviors.
2. Traffic Monitoring: Real-time monitoring of network traffic to detect potential intrusions or anomalies, including unusual DNS, HTTP, and IP connections.
3. Behavioral Analysis: Tracks suspicious behaviors like IP scanning, DDoS attacks, or potential malware communications.
4. Web-Based Interface: Offers an intuitive web-based dashboard that displays alerts, logs, and real-time data visualization.
5. Anomaly Detection: Detects unknown threats using heuristic and anomaly detection methods, flagging unusual network activities.

How Maltrail Works

Maltrail is typically deployed on a network sensor, such as a server that monitors traffic through a network tap or SPAN port. It analyzes raw traffic and compares it against its threat intelligence sources and behavior-based signatures. When suspicious or known malicious traffic is detected, Maltrail generates alerts that are accessible through the web-based UI or as JSON logs.

Deployment and Setup

To set up Maltrail, you’ll need:

1. Python Environment: Maltrail is written in Python, so a Python 3 environment is required.
2. Network Access: Access to the network where it will monitor traffic (e.g., through a mirrored or tapped port).
3. Configuration: Basic configuration for interfaces, threat feeds, logging preferences, and the admin credentials for the web UI.

Example Usage

To start Maltrail, simply clone the repository and run the main script:

# Clone the repository
git clone https://github.com/stamparm/maltrail.git
cd maltrail

# Start the server
sudo python3 sensor.py

For the server, you can access the Maltrail dashboard by starting the server.py script:

sudo python3 server.py

You can then access the dashboard via a web browser at http://<your_server_ip>:8338.

sensor.py and Server.py

In the context of Maltrail, sensor.py and server.py are core scripts responsible for data collection and user interface management:

1. sensor.py

The sensor.py script functions as the network traffic monitor and analyzer. It acts as the sensor component in the Maltrail architecture, capturing network packets, analyzing them for potential threats, and logging suspicious activity.

Key Responsibilities:

• Packet Capturing: sensor.py captures network packets from specified interfaces (e.g., Ethernet) and analyzes the traffic for malicious or suspicious patterns.
• Threat Detection: It uses multiple data sources, including public blacklists, community-reported IPs, and behavioral heuristics, to check if the traffic matches known indicators of compromise (IOCs) or suspicious patterns.
• Logging: Logs detected threats and anomalies in JSON format, making it easier for server.py to display them on the web interface.
• Data Sources: Regularly updates its list of malicious indicators by fetching from multiple feeds. These feeds may include IP blacklists, domain reputation lists, and malware signatures.

Usage:
You run sensor.py on a device within the network that has access to monitor traffic. This device could be a server connected to a mirror (SPAN) port on a switch, allowing it to passively analyze traffic from other devices in the network.

Running sensor.py:

sudo python3 sensor.py

Important Parameters in sensor.py

• INTERFACE: Specifies the network interface to monitor (e.g., eth0).
• LOG_FILE: Defines where logs are stored.
• SENSOR_NAME: Can be set to identify the specific sensor instance (useful in multi-sensor deployments).

2. server.py

The server.py script is the server component in the Maltrail architecture. It provides a web-based user interface and handles the backend for log management and alert visualization.

Key Responsibilities:

• Web-Based Interface: Hosts a dashboard accessible via a web browser where users can view logs, alerts, and statistics in real-time. The default access URL is http://<server_ip>:8338.
• Data Aggregation: Reads JSON logs created by sensor.py and displays them in a structured format, showing information about detected threats and suspicious activities.
• User Authentication: Allows the setup of user authentication, securing access to the Maltrail interface with admin credentials.
• Alert Management: Filters and sorts alerts based on threat severity, source IP, type of attack, etc., making it easier to manage and prioritize responses.

Usage:
After running sensor.py to collect and log data, you run server.py on the same or a different machine that has access to these logs. Users can then access the Maltrail web dashboard to analyze the detected threats.

Running server.py:

sudo python3 server.py

Important Configuration Options in server.py

• PORT: The default port is 8338, but this can be changed in the configuration file (maltrail.conf) if needed.
• USER_AUTHENTICATION: Supports user authentication with default credentials (set in the configuration file). The default username and password are admin:admin.
• ALERT_THRESHOLD: Controls when alerts should be triggered based on the severity of threats.

How sensor.py and server.py Work Together

1. Data Collection: sensor.py captures network packets, analyzes them for threats, and logs detected anomalies in a JSON format.
2. Log Aggregation: server.py reads these logs and aggregates the data, making it accessible to users.
3. User Interface: server.py hosts a dashboard where users can view and interact with logs, filter by date, threat type, and severity, and gain insights into the security status of their network.

This modular design allows Maltrail to be flexible; you can deploy multiple sensors across different network segments while using a single server instance to manage alerts and monitor traffic across your network.

Configurations

The maltrail.conf file configures Maltrail’s sensor and server components for monitoring, detection, and reporting. Let’s break down each section and key parameter to understand how it influences Maltrail’s operation:

[Server] Section

The [Server] section configures settings for the reporting server, which hosts the Maltrail web interface, manages user authentication, and handles log collection.

1. HTTP_ADDRESS: Defines the IP address on which the Maltrail server listens. The default setting (0.0.0.0) allows it to accept connections from any network interface.

   • Examples commented out include IPv6 addresses (::, or a specific interface).

2. HTTP_PORT: Specifies the port for the HTTP server, with a default of 8338.

3. USE_SSL: When set to true, it enables SSL/TLS for encrypted connections. false means the server runs without encryption.

4. SSL_PEM: Location of the SSL certificate (PEM file) if SSL is enabled.

5. USERS: Defines user accounts for accessing the Maltrail web interface. Users are defined by:

   • Username: E.g., admin.
   • Password (hashed): SHA256 hash of the password.
   • UID: User ID. Admin users typically have UID=0; non-admin users (UID>=1000) can only view results.
   • Filter Netmask(s): Optional IP filters that limit users’ access based on IP ranges.

6. ENABLE_MASK_CUSTOM: When true, hides custom trail names for non-admin users, providing additional privacy for custom monitoring rules.

7. UDP_ADDRESS and UDP_PORT: These (if enabled) allow the server to accept logs from remote sensors via UDP. Default is commented out.

8. USE_SERVER_UPDATE_TRAILS: When true, lets the server download trail updates, which sensors can then access.

9. IP_ALIASES: Allows defining aliases for specific IP addresses to make logs easier to interpret, e.g., naming 8.8.8.8 as google.

10. HEADER_LOGO: Allows customizing the top-left logo in the web interface with an image or text.

11. FAIL2BAN_REGEX: A regex pattern used for Fail2Ban integration to identify attacker IPs.

12. BLACKLIST: Provides rules for custom blacklists, e.g., blocking specific IPs or traffic based on IP ranges and ports.

[Sensor] Section

The [Sensor] section defines parameters for traffic monitoring and detection, specifying how and what Maltrail sensors capture and analyze.

1. PROCESS_COUNT: Number of processes used for packet capture and analysis. Increasing this number can improve performance on multi-core systems.

2. DISABLE_CPU_AFFINITY: When true, prevents the sensor from setting CPU affinity, which can be useful on systems with multiple high-demand processes.

3. USE_FEED_UPDATES: If true, enables automatic trail updates from various feeds (blacklists, known malware IPs, etc.).

4. DISABLED_FEEDS: Lists feeds that Maltrail won’t use, as they might generate excessive false positives (e.g., turris, ciarmy).

5. IP_MINIMUM_FEEDS: Controls the minimum number of feeds an IP must appear on to be flagged, which reduces false positives.

6. UPDATE_PERIOD: Frequency of trail updates in seconds, with 86400 (24 hours) as the default.

7. CUSTOM_TRAILS_URL and CUSTOM_TRAILS_DIR: Specifies URLs or directories for custom trails, allowing custom monitoring rules or sources.

8. CAPTURE_BUFFER: Defines the ring buffer size for network capture, set as a percentage of system memory or bytes, e.g., 10%.

9. MONITOR_INTERFACE: Specifies the network interface for monitoring (e.g., eth0, or any for all interfaces).

10. CAPTURE_FILTER: Packet capture filter to reduce data to only relevant traffic types. Current setting focuses on UDP, ICMP, and specific TCP flags or ports.

11. SENSOR_NAME: Sets the sensor’s name in logs; $HOSTNAME auto-fills the device hostname.

12. LOG_SERVER, SYSLOG_SERVER, LOGSTASH_SERVER: Configures remote servers for log storage (using UDP, Syslog, or JSON formats), useful in distributed setups.

13. REMOTE_SEVERITY_REGEX: Regular expression defining log severity levels for external log storage.

14. DISABLE_LOCAL_LOG_STORAGE: When true, disables local log storage and sends logs exclusively to remote servers.

15. USE_HEURISTICS: Enables heuristic-based detection methods for enhanced threat identification.

16. CHECK_MISSING_HOST and CHECK_HOST_DOMAINS: If enabled, checks HTTP Host headers for anomalies, which can increase detection rates but may lead to more false positives.

17. USER_WHITELIST and USER_IGNORELIST: Paths to files with whitelisted entries or ignored events, allowing selective filtering of specific IPs or events.

[All] Section

Applies configurations globally across the sensor and server.

1. SHOW_DEBUG: When true, enables detailed console output for debugging purposes.

2. LOG_DIR: Specifies the directory for log storage; $SYSTEM_LOG_DIR/maltrail is a system path variable.

3. PROXY_ADDRESS: Optional proxy for external connections (e.g., fetching trails).

4. DISABLE_CHECK_SUDO: When true, skips the check for sudo/administrator privileges, useful if permissions are set through CAP_NET_RAW or similar.

5. TRAILS_FILE: Sets a custom path for trail storage, defaulting to ~/.maltrail/trails.csv.

more info

In Maltrail, you can configure it to display only malicious IPs or to include all IPs, depending on your needs. By default, Maltrail is designed to highlight suspicious or known malicious activities based on its trails (feeds) of IPs associated with malware, scanners, spam, and other types of unwanted activity. Here’s how to manage the display options:

Display Only Malicious IPs

To focus on malicious IPs, you can adjust the trail and filter configurations to only show IPs with identified malicious activity. Here are steps to configure it accordingly:

1. Filter Results for Non-Admin Users:
   In the [Server] section, use the ENABLE_MASK_CUSTOM directive, which is already set to true in your configuration. This masks custom trail names for non-admin users (UID >= 1000), so only malicious and suspicious IPs are shown. Ensure all users who should see all IPs are set with a UID lower than 1000.

   ENABLE_MASK_CUSTOM true
   

2. Customizing Trail Filters:
   Use the CAPTURE_FILTER option to limit traffic to specific protocols or malicious traffic:

   CAPTURE_FILTER udp or icmp or (tcp and (tcp[tcpflags] == tcp-syn or port 80 or port 1080 or port 3128 or port 8000 or port 8080 or port 8118))
   

   This filter restricts the capture to specific protocols commonly associated with suspicious traffic.

3. Blacklist Rules:
   Use BLACKLIST rules if you want to further refine what IPs are considered malicious by creating filters that only log certain types of IPs or ports. For example:

   BLACKLIST
       src_ip !~ ^192.168. and dst_port ~ ^22$
   

   This configuration restricts results to exclude internal network IPs (e.g., 192.168.x.x) and includes only those with suspicious destination ports (like SSH on port 22).

Display All IPs

If you want to see all IPs, including those without malicious activity, you can:

1. Remove Filters:
   Adjust the CAPTURE_FILTER to capture all IPs without filtering:

   CAPTURE_FILTER ip or ip6
   

2. Disable Specific Trails:
   If you want to log all traffic but don’t want specific feeds marked as malicious, you can add them to DISABLED_FEEDS:

   DISABLED_FEEDS turris, ciarmy, policeman, myip, alienvault
   

   This prevents trails from these specific feeds from flagging IPs as suspicious.

3. User-Specific Views:
   Grant users the ability to see all IPs by setting their UID to less than 1000. This way, all IPs are displayed without the filtering applied for non-admins.

Summary of showing ip’s options

To focus on only malicious IPs, keep the ENABLE_MASK_CUSTOM as true, set appropriate CAPTURE_FILTER and BLACKLIST rules, and limit trails to those with confirmed malicious activity. For all IPs, relax the filtering options, disable trail-specific feeds if needed, and set user UIDs accordingly.