Sandfly 4.4.0 - Agentless Linux Password Auditing and Data De-Duplication
S
Steve Busko
started a topic
12 months ago
Sandfly 4.4.0 has two major new features we are excited about sharing:
Agentless password auditor that works across all Linux distributions.
De-duplicating events resulting in a 99%+ reduction in data in the database and sent to SIEM tools such as Splunk.
Agentless Password Auditor
Stolen and weak credentials on Linux are perhaps the major way hosts are compromised. We pay very special attention to SSH credentials with our SSH Hunter, but today we are closing the loop on username/password use with our agentless password auditor.
Weak passwords on Linux systems can be exploited by malware or dedicated attackers, leading to immediate system compromises. The challenge lies in effectively testing weak passwords across all Linux systems, CPU types, and doing so securely without the need to centralize password hashes for cracking.
We are introducing a mechanism where we bring the password auditing directly to each host. We check for vulnerable passwords on systems regardless of Linux distribution. We even do it on embedded Linux devices.
Password Brute Force Threat Model
The primary threat model under consideration involves external attackers attempting to gain access to hosts through channels such as SSH, rather than individuals who have obtained password hashes and employ specialized password crackers. By focusing on this threat model, highly-targeted password auditing can be implemented, effectively shifting the attack window beyond the capabilities of most external intruders.
Password Brute Force Attack Window
External attackers attempting brute force attacks are constrained by several factors, which limit the number of attempts they can execute:
Network latency
Time delays due to invalid authentication attempts
Auto-banning of IP addresses
Detection by security teams due to increased attack noise
Given these limitations, attackers without access to password hashes are unable to carry out extensive brute force attacks, as they would be restricted to dozens or, at most, hundreds of attempts before potentially being blocked.
As a result, a targeted audit of the most vulnerable password types on the host itself proves to be highly efficient and offers significant security improvements against simple compromises on Linux. Since attackers are more likely to begin with the weakest passwords, focusing on these vulnerabilities addresses the most probable threats while recognizing the limitations faced by external attackers.
On Host Password Auditing Advantages
On-host password auditing for Linux offers numerous benefits, which include:
Compliance with organizational or industry-specific security policies such as GDPR, PCI, or HIPAA by avoiding the transfer of password hashes off-host for auditing.
Compatibility with a wide range of Linux distributions, including legacy systems up to a decade old.
Support for various CPU types, including Intel, AMD, Arm, and MIPS.
Capability to audit challenging situations, such as embedded Linux devices, for weak passwords.
Customization options allowing users to define specific passwords to be banned from use.
Facilitating security teams in searching for passwords linked to ongoing incident response.
Scalability through the deployment of the password auditing engine on each endpoint, enabling independent audits.
Rapid auditing of the entire Linux infrastructure for weak passwords, often completed within seconds.
By employing on-host password auditing for Linux systems, organizations can enhance security, maintain compliance, and ensure comprehensive password assessments across diverse infrastructure.
Password Auditing Sandfly Modules
We have included several built-in password auditing modules:
user_password_auditor_password_is_username - Username is the password. This is a major risk and is enabled by default.
policy_user_password_auditor_top_100- User has a password in the Top 100 worst passwords.
policy_user_password_auditor_top_500 - User has a password in the Top 500 worst passwords.
policy_user_password_auditor_linux_common - User has a password that is a common Linux user or service name.
policy_user_password_auditor_custom_password_check - Customer defined template of passwords they want to make sure are not being used anywhere.
Username is Password
This module finds a very common and serious issue (especially on embedded devices): A password that is identical to the username. This is the primary brute force method used by many kinds of Linux malware families, including most cryptominers and botnets.
For example Sandfly will find accounts such as:
admin/admin
ubuntu/ubuntu
root/root
etc.
This module is enabled by default as it executes in a split second even on embedded systems. Below you can see an example where a Raspberry Pi has a username that is identical to the password.
Top 100 and 500 Worst Passwords
Additional modules are available to audit users utilizing the Top 100 and 500 worst passwords. The Top 100 checks require minimal time on most systems, making them highly valuable. The Top 500 checks may take longer but could be worthwhile as less frequent checks for many organizations.
These modules are not enabled by default. To activate them, review the policy sandfly area and enable the desired modules or run them manually.
Top Linux Usernames as Passwords
We also have a module that looks for common Linux usernames (e.g. "nagios") as the password. This also finds notorious combinations such as "root/superuser", "www/administrator", and more. This module takes about as much time as the Top 100 worst password module discussed above to run. This module will again need to be enabled or run it manually.
Custom Password Auditing
As with all Sandfly modules, the password auditing modules can be cloned and modified to search for custom parameters. For example, some organizations may have shared passwords across accounts (commonly referred to as the infamous corporate login credential). These passwords pose a particularly high risk, as they are rarely changed, their usage is difficult to track, and they often persist even after employees depart.
Sandfly enables users to tailor searches for passwords of this nature. With a small list of passwords, the check is nearly instantaneous and is a valuable addition to organizations facing this issue.
Below we see a custom password auditing module example. This module can be enabled and it then will provide continuous monitoring to make sure these banned passwords do not show up on any accounts once weeded out. Any account showing up with these passwords will generate an alert instantly.
Performance Benchmarks
When performing password auditing on the host, it is natural to inquire about potential CPU impacts. It is essential to clarify that the auditing process does not involve conventional brute force attacks with tens of millions of passwords. Instead, it focuses on a small, targeted set of high-risk weak passwords, resulting in a considerably lower impact than traditional password auditing methods.
To assess the potential impacts, the auditing modules were tested on various low-specification Linux systems. The highest-performing system was a low-end Intel i5. Additional tests were conducted on low-end shared CPU cloud VMs, including Amazon Arm EC2 cloud instances. Lastly, the modules were tested on embedded Arm and MIPS Power-Over-Ethernet (POE) network devices with limited processing power.
The password auditing modules, including username/password matching, Top 100 and 500 worst passwords, and common Linux usernames as passwords, have been tested to determine the time required to audit each user. The chart provided illustrates the time taken for each module to audit a single user. For instance, if an audit took approximately 5 seconds, that is for one user only; for 10 users, the total audit time would be around 50 seconds.
The username/password matching check is nearly instantaneous, even on low-powered embedded devices. Due to its low impact and critical importance, this check is enabled by default.
Top 100 and Linux usernames checks exhibit similar performance metrics. On modern CPUs, they take a few seconds per user. On low-power devices, such as Arm and MIPS64 embedded systems, each check takes approximately 10 seconds per user.
On Power-Over-Ethernet (POE) network devices, the impact on speed becomes more noticeable. Audits per user can take over 100 seconds for the Top 100 and Top Linux usernames. However, conducting periodic audits on these devices is recommended, as they often have limited users (typically 1-2), and the risk they pose is significant.
Regarding the Top 500 worst passwords, the time taken is still reasonable for bare metal and VMs (even low-end). For embedded devices, there is a time penalty that scales with the number of users. Despite this, occasional spot-audits of embedded systems may still be worthwhile if the number of users checked is kept low.
Limiting Performance Impacts
The password auditing engine incorporates a parameter called 'max_random_users_to_attempt,' which determines the maximum number of users to be audited in a random manner. By default, this parameter is set to 10 users.
For example, if there are five users with a password hash, all of them will be audited. However, if a system has 21 users, Sandfly will randomly select 10 users for auditing. In subsequent audits, the random selection process is repeated, ensuring that all users are eventually audited over time.
Another valuable feature is the 'password_is_username' parameter, which instructs Sandfly to check if the username and password are identical. This check takes only a split second to run, and it is recommended to keep it enabled on all audit modules for optimal security.
The password auditing engine incorporates standard timeout values, ensuring that audits do not exceed a specified duration (default of 360 seconds). This feature guarantees that the CPU will not be overwhelmed with auditing tasks for an indefinite period.
Additionally, since the password auditing process operates on a single core and at low priority, it is unlikely to have a significant impact on the vast majority of Linux systems where it is employed.
Audit Policies Not Enabled by Default
The password auditing policy modules are not enabled by default; therefore, users must activate them to utilize them in a scheduled rotation. To assess their performance, users can run the modules manually before activating and deploying them across their Linux fleet.
Won't Cause Banning
Sandfly's password auditing process examines user password hashes directly, rather than conducting brute force attacks against network or authentication services. As a result, external auto-banning mechanisms, such as fail2ban, and network monitoring systems are not triggered by excessive login attempts.
Users can confidently employ Sandfly's password auditing feature without concerns of being locked out of their hosts or overwhelming their security team with unnecessary alerts.
Don't Use Username/Passwords? Are you sure?
Implementing Sandfly's password auditing checks is highly recommended for most Linux deployments, even when username/password authentication is not believed to be in use. There have been instances where customers predominantly employed SSH keys, but username/password authentication was still present on some systems.
For embedded Linux systems, the likelihood of username/password usage is considerably higher, and it is crucial to perform regular checks to maintain security.
Limitations
The current password auditor supports Linux crypt MD5, SHA-256 and SHA-512 key hashes. Sandfly will be implementing yescrypt (which is part of the latest Debian/Fedora distros) in the near future.
Password Auditing Free for All Users
The password auditing features are accessible to all Sandfly users, including those with free licenses. Custom checks, however, are exclusive to licensed users. It is highly recommended that everyone utilize Sandfly to audit their Linux hosts for basic username/password vulnerabilities to prevent potential security breaches.
To illustrate the importance of these audits, a honeypot Ubuntu system was exposed to the Internet with the login credentials 'ubuntu/ubuntu'. It took a mere 2 minutes and 13 seconds for automated attackers to compromise the system. Conducting regular password audits can significantly reduce the risk of such intrusions.
Data De-Duplication
A significant feature introduced in Sandfly 4.4.0 is data de-duplication, which reduces the amount of data stored by Sandfly by over 99%. This major optimization offers numerous benefits:
Substantial reduction in database storage requirements.
Decreased events sent to external SIEMs like Splunk, leading to lower data ingestion impacts on licenses.
Improved database performance and increased UI responsiveness, even when monitoring tens of thousands of hosts.
Enhanced ability to store events for extended periods without concerns of purging expired data.
Sandfly 4.4.0 can handle extremely large host loads, allowing a modest server to manage thousands of systems. For customers exporting events to external SIEM tools, this feature results in significantly less data transfer, enabling faster queries, longer event storage, and reduced fees for inbound data costs.
Old Results Purged
With the introduction of the data de-duplication feature in Sandfly 4.4.0, please be aware that all prior results will be removed during the upgrade due to internal formatting changes. This may result in the loss of up to 3 days of events, which is the default expiration time for most customers' results.
For those without alerts, this should not cause any significant issues. However, if you have active alerts under investigation, it is recommended to resolve them before proceeding with the upgrade.
Following the upgrade, you can extend the data retention time beyond the default 3 days. The significant data reduction provided by the de-duplication feature allows for longer retention periods without compromising performance. Feel free to adjust the retention period as needed and evaluate its impact on your system.
UI Improvements
With the latest data de-duplication feature, the user interface has seen significant improvements in alert views. In previous versions, receiving a new alert would add it to the alert list, even if the same alert had been identified in prior scans. As a result, the alert list would continue to grow with each new scan.
With the implementation of data de-duplication, the system now updates the count and date for existing alerts instead of adding new ones. Alerts will be marked as "latest" when updated. This change ensures that the alert list remains consistent, showing the same alerts until they are resolved, rather than expanding each time a system is re-scanned and the same issue is identified.
Get a Free License Today
The password auditing feature holds significant value for Linux administrators and security teams. This feature is accessible in our free product, as we believe it is crucial for users to check for these issues. To obtain a free license, please visit the following link:
Users with paid licenses receive the same features as well as the added ability to create customized password audit sweeps according to their needs.
Upgrading Sandfly
Sandfly 4.4.0 represents a significant upgrade in multiple aspects. The data de-duplication provides substantial performance and usability enhancements. Moreover, the agentless password auditor is a distinct feature that can swiftly identify critical issues across your Linux systems.
Customers wishing to upgrade can follow the instructions here:
Steve Busko
Sandfly 4.4.0 has two major new features we are excited about sharing:
Agentless password auditor that works across all Linux distributions.
De-duplicating events resulting in a 99%+ reduction in data in the database and sent to SIEM tools such as Splunk.
Agentless Password Auditor
Stolen and weak credentials on Linux are perhaps the major way hosts are compromised. We pay very special attention to SSH credentials with our SSH Hunter, but today we are closing the loop on username/password use with our agentless password auditor.
Weak passwords on Linux systems can be exploited by malware or dedicated attackers, leading to immediate system compromises. The challenge lies in effectively testing weak passwords across all Linux systems, CPU types, and doing so securely without the need to centralize password hashes for cracking.
We are introducing a mechanism where we bring the password auditing directly to each host. We check for vulnerable passwords on systems regardless of Linux distribution. We even do it on embedded Linux devices.
Password Brute Force Threat Model
The primary threat model under consideration involves external attackers attempting to gain access to hosts through channels such as SSH, rather than individuals who have obtained password hashes and employ specialized password crackers. By focusing on this threat model, highly-targeted password auditing can be implemented, effectively shifting the attack window beyond the capabilities of most external intruders.
Password Brute Force Attack Window
External attackers attempting brute force attacks are constrained by several factors, which limit the number of attempts they can execute:
Network latency
Time delays due to invalid authentication attempts
Auto-banning of IP addresses
Detection by security teams due to increased attack noise
Given these limitations, attackers without access to password hashes are unable to carry out extensive brute force attacks, as they would be restricted to dozens or, at most, hundreds of attempts before potentially being blocked.
As a result, a targeted audit of the most vulnerable password types on the host itself proves to be highly efficient and offers significant security improvements against simple compromises on Linux. Since attackers are more likely to begin with the weakest passwords, focusing on these vulnerabilities addresses the most probable threats while recognizing the limitations faced by external attackers.
On Host Password Auditing Advantages
On-host password auditing for Linux offers numerous benefits, which include:
Compliance with organizational or industry-specific security policies such as GDPR, PCI, or HIPAA by avoiding the transfer of password hashes off-host for auditing.
Compatibility with a wide range of Linux distributions, including legacy systems up to a decade old.
Support for various CPU types, including Intel, AMD, Arm, and MIPS.
Capability to audit challenging situations, such as embedded Linux devices, for weak passwords.
Customization options allowing users to define specific passwords to be banned from use.
Facilitating security teams in searching for passwords linked to ongoing incident response.
Scalability through the deployment of the password auditing engine on each endpoint, enabling independent audits.
Rapid auditing of the entire Linux infrastructure for weak passwords, often completed within seconds.
By employing on-host password auditing for Linux systems, organizations can enhance security, maintain compliance, and ensure comprehensive password assessments across diverse infrastructure.
Password Auditing Sandfly Modules
We have included several built-in password auditing modules:
user_password_auditor_password_is_username - Username is the password. This is a major risk and is enabled by default.
policy_user_password_auditor_top_100- User has a password in the Top 100 worst passwords.
policy_user_password_auditor_top_500 - User has a password in the Top 500 worst passwords.
policy_user_password_auditor_linux_common - User has a password that is a common Linux user or service name.
policy_user_password_auditor_custom_password_check - Customer defined template of passwords they want to make sure are not being used anywhere.
Username is Password
This module finds a very common and serious issue (especially on embedded devices): A password that is identical to the username. This is the primary brute force method used by many kinds of Linux malware families, including most cryptominers and botnets.
For example Sandfly will find accounts such as:
admin/admin
ubuntu/ubuntu
root/root
etc.
This module is enabled by default as it executes in a split second even on embedded systems. Below you can see an example where a Raspberry Pi has a username that is identical to the password.
Top 100 and 500 Worst Passwords
Additional modules are available to audit users utilizing the Top 100 and 500 worst passwords. The Top 100 checks require minimal time on most systems, making them highly valuable. The Top 500 checks may take longer but could be worthwhile as less frequent checks for many organizations.
These modules are not enabled by default. To activate them, review the policy sandfly area and enable the desired modules or run them manually.
Top Linux Usernames as Passwords
We also have a module that looks for common Linux usernames (e.g. "nagios") as the password. This also finds notorious combinations such as "root/superuser", "www/administrator", and more. This module takes about as much time as the Top 100 worst password module discussed above to run. This module will again need to be enabled or run it manually.
Custom Password Auditing
As with all Sandfly modules, the password auditing modules can be cloned and modified to search for custom parameters. For example, some organizations may have shared passwords across accounts (commonly referred to as the infamous corporate login credential). These passwords pose a particularly high risk, as they are rarely changed, their usage is difficult to track, and they often persist even after employees depart.
Sandfly enables users to tailor searches for passwords of this nature. With a small list of passwords, the check is nearly instantaneous and is a valuable addition to organizations facing this issue.
Below we see a custom password auditing module example. This module can be enabled and it then will provide continuous monitoring to make sure these banned passwords do not show up on any accounts once weeded out. Any account showing up with these passwords will generate an alert instantly.
Performance Benchmarks
When performing password auditing on the host, it is natural to inquire about potential CPU impacts. It is essential to clarify that the auditing process does not involve conventional brute force attacks with tens of millions of passwords. Instead, it focuses on a small, targeted set of high-risk weak passwords, resulting in a considerably lower impact than traditional password auditing methods.
To assess the potential impacts, the auditing modules were tested on various low-specification Linux systems. The highest-performing system was a low-end Intel i5. Additional tests were conducted on low-end shared CPU cloud VMs, including Amazon Arm EC2 cloud instances. Lastly, the modules were tested on embedded Arm and MIPS Power-Over-Ethernet (POE) network devices with limited processing power.
The password auditing modules, including username/password matching, Top 100 and 500 worst passwords, and common Linux usernames as passwords, have been tested to determine the time required to audit each user. The chart provided illustrates the time taken for each module to audit a single user. For instance, if an audit took approximately 5 seconds, that is for one user only; for 10 users, the total audit time would be around 50 seconds.
The username/password matching check is nearly instantaneous, even on low-powered embedded devices. Due to its low impact and critical importance, this check is enabled by default.
Top 100 and Linux usernames checks exhibit similar performance metrics. On modern CPUs, they take a few seconds per user. On low-power devices, such as Arm and MIPS64 embedded systems, each check takes approximately 10 seconds per user.
On Power-Over-Ethernet (POE) network devices, the impact on speed becomes more noticeable. Audits per user can take over 100 seconds for the Top 100 and Top Linux usernames. However, conducting periodic audits on these devices is recommended, as they often have limited users (typically 1-2), and the risk they pose is significant.
Regarding the Top 500 worst passwords, the time taken is still reasonable for bare metal and VMs (even low-end). For embedded devices, there is a time penalty that scales with the number of users. Despite this, occasional spot-audits of embedded systems may still be worthwhile if the number of users checked is kept low.
Limiting Performance Impacts
The password auditing engine incorporates a parameter called 'max_random_users_to_attempt,' which determines the maximum number of users to be audited in a random manner. By default, this parameter is set to 10 users.
For example, if there are five users with a password hash, all of them will be audited. However, if a system has 21 users, Sandfly will randomly select 10 users for auditing. In subsequent audits, the random selection process is repeated, ensuring that all users are eventually audited over time.
Another valuable feature is the 'password_is_username' parameter, which instructs Sandfly to check if the username and password are identical. This check takes only a split second to run, and it is recommended to keep it enabled on all audit modules for optimal security.
The password auditing engine incorporates standard timeout values, ensuring that audits do not exceed a specified duration (default of 360 seconds). This feature guarantees that the CPU will not be overwhelmed with auditing tasks for an indefinite period.
Additionally, since the password auditing process operates on a single core and at low priority, it is unlikely to have a significant impact on the vast majority of Linux systems where it is employed.
Audit Policies Not Enabled by Default
The password auditing policy modules are not enabled by default; therefore, users must activate them to utilize them in a scheduled rotation. To assess their performance, users can run the modules manually before activating and deploying them across their Linux fleet.
Won't Cause Banning
Sandfly's password auditing process examines user password hashes directly, rather than conducting brute force attacks against network or authentication services. As a result, external auto-banning mechanisms, such as fail2ban, and network monitoring systems are not triggered by excessive login attempts.
Users can confidently employ Sandfly's password auditing feature without concerns of being locked out of their hosts or overwhelming their security team with unnecessary alerts.
Don't Use Username/Passwords? Are you sure?
Implementing Sandfly's password auditing checks is highly recommended for most Linux deployments, even when username/password authentication is not believed to be in use. There have been instances where customers predominantly employed SSH keys, but username/password authentication was still present on some systems.
For embedded Linux systems, the likelihood of username/password usage is considerably higher, and it is crucial to perform regular checks to maintain security.
Limitations
The current password auditor supports Linux crypt MD5, SHA-256 and SHA-512 key hashes. Sandfly will be implementing yescrypt (which is part of the latest Debian/Fedora distros) in the near future.
Password Auditing Free for All Users
The password auditing features are accessible to all Sandfly users, including those with free licenses. Custom checks, however, are exclusive to licensed users. It is highly recommended that everyone utilize Sandfly to audit their Linux hosts for basic username/password vulnerabilities to prevent potential security breaches.
To illustrate the importance of these audits, a honeypot Ubuntu system was exposed to the Internet with the login credentials 'ubuntu/ubuntu'. It took a mere 2 minutes and 13 seconds for automated attackers to compromise the system. Conducting regular password audits can significantly reduce the risk of such intrusions.
Data De-Duplication
A significant feature introduced in Sandfly 4.4.0 is data de-duplication, which reduces the amount of data stored by Sandfly by over 99%. This major optimization offers numerous benefits:
Substantial reduction in database storage requirements.
Decreased events sent to external SIEMs like Splunk, leading to lower data ingestion impacts on licenses.
Improved database performance and increased UI responsiveness, even when monitoring tens of thousands of hosts.
Enhanced ability to store events for extended periods without concerns of purging expired data.
Sandfly 4.4.0 can handle extremely large host loads, allowing a modest server to manage thousands of systems. For customers exporting events to external SIEM tools, this feature results in significantly less data transfer, enabling faster queries, longer event storage, and reduced fees for inbound data costs.
Old Results Purged
With the introduction of the data de-duplication feature in Sandfly 4.4.0, please be aware that all prior results will be removed during the upgrade due to internal formatting changes. This may result in the loss of up to 3 days of events, which is the default expiration time for most customers' results.
For those without alerts, this should not cause any significant issues. However, if you have active alerts under investigation, it is recommended to resolve them before proceeding with the upgrade.
Following the upgrade, you can extend the data retention time beyond the default 3 days. The significant data reduction provided by the de-duplication feature allows for longer retention periods without compromising performance. Feel free to adjust the retention period as needed and evaluate its impact on your system.
UI Improvements
With the latest data de-duplication feature, the user interface has seen significant improvements in alert views. In previous versions, receiving a new alert would add it to the alert list, even if the same alert had been identified in prior scans. As a result, the alert list would continue to grow with each new scan.
With the implementation of data de-duplication, the system now updates the count and date for existing alerts instead of adding new ones. Alerts will be marked as "latest" when updated. This change ensures that the alert list remains consistent, showing the same alerts until they are resolved, rather than expanding each time a system is re-scanned and the same issue is identified.
Get a Free License Today
The password auditing feature holds significant value for Linux administrators and security teams. This feature is accessible in our free product, as we believe it is crucial for users to check for these issues. To obtain a free license, please visit the following link:
Get Sandfly
Users with paid licenses receive the same features as well as the added ability to create customized password audit sweeps according to their needs.
Upgrading Sandfly
Sandfly 4.4.0 represents a significant upgrade in multiple aspects. The data de-duplication provides substantial performance and usability enhancements. Moreover, the agentless password auditor is a distinct feature that can swiftly identify critical issues across your Linux systems.
Customers wishing to upgrade can follow the instructions here:
Upgrading Sandfly
If you have any questions, please reach out to us.
Thank you for using Sandfly.