Sandfly 3.1 has been released with major changes enhancing performance while reducing the resources needed to run the system.
Goodbye Elasticsearch. Hello PostgreSQL.
Sandfly server used to rely on Elasticsearch for our internal database. This is OK for unstructured data, but it required a lot of RAM and CPU resources to run. The result was under previous versions we required 8GB minimum RAM to run just to handle the basic requirements of Elasticsearch.
We now have replaced the internal database with PostgreSQL and this has resulted in dramatically lower RAM requirements while providing a big boost in performance. While in the past we recommended a minimum of 8-12GB to run, we now very comfortably run on systems with 2-4GB or less of RAM on smaller deployments. This means many customers can run on extremely inexpensive cloud instances to protect their Linux systems without sacrificing any performance.
This update still allows you to export events to Elasticsearch externally however (see below).
Export Events to Elasticsearch and Splunk
The updated internal database does not affect the ability to export events to your own Elasticsearch cluster or Splunk. The changes only affec internal database operations which have always been masked from direct user access.
We in fact made Elasticsearch export to external clusters easier by including it directly in the server configuration inside the UI:
The main server has been ported to Golang to match our agentless forensic engines which have been in Golang for some time. This has resulted in a faster server using fewer resources for better performance. The old server could already handle heavy loads, but the new server takes us to the next level of performance and scalability. We can easily handle massive Linux host loads with our agentless platform on very modest hardware.
Improved Filtering and Host Views
We have improved results filtering to make it faster with more options. We also have enhanced the host view to show more information such as hostnames by default plus general optimizations:
Improved Syslog Output for Graylog and Other SIEM Tools
For customers using Sandfly to send syslog output, we have improved the formatting in the result data. For syslog receivers such as Graylog that support building fields from structured syslog messages, the Sandfly messages now have a more consistent and useful structured data to build queries or alerts.
Faster Dashboard and General Operation
Updated caching along with database optimization has resulted in a much faster dashboard and general operation in the UI. All operations are sped up such as alert viewing, filtering and forensic data views.
Sandfly Hunter Optimized
Fully licensed users get access to Sandfly Hunter and it too has been sped up. Searching across hosts for Linux forensic artifacts is faster and more efficient. Use Sandfly Hunter to search for files, processes, users, log entries, SSH keys and more as part of an incident to help quickly find and isolate compromised hosts.
New Sandfly Checks for CronRAT and More
We have added in new checks for malware tactics used in CronRAT Linux malware and more. We could find this type of malware already, but have added new hunting modules for suspicious cron command entries and more to make detection even more complete.
Seamless Upgrade from 3.0 to 3.1
The new upgrade automatically ports your existing data from Elasticsearch over to the new database. Please see the upgrade documentation for how to quickly and easily upgrade.
Upgrade For Free
All free and paid customers can upgrade today. Please follow the directions below:
Steve Busko
Sandfly 3.1 has been released with major changes enhancing performance while reducing the resources needed to run the system.
Goodbye Elasticsearch. Hello PostgreSQL.
Sandfly server used to rely on Elasticsearch for our internal database. This is OK for unstructured data, but it required a lot of RAM and CPU resources to run. The result was under previous versions we required 8GB minimum RAM to run just to handle the basic requirements of Elasticsearch.
We now have replaced the internal database with PostgreSQL and this has resulted in dramatically lower RAM requirements while providing a big boost in performance. While in the past we recommended a minimum of 8-12GB to run, we now very comfortably run on systems with 2-4GB or less of RAM on smaller deployments. This means many customers can run on extremely inexpensive cloud instances to protect their Linux systems without sacrificing any performance.
This update still allows you to export events to Elasticsearch externally however (see below).
Export Events to Elasticsearch and Splunk
The updated internal database does not affect the ability to export events to your own Elasticsearch cluster or Splunk. The changes only affec internal database operations which have always been masked from direct user access.
We in fact made Elasticsearch export to external clusters easier by including it directly in the server configuration inside the UI:
Splunk users can still use our certified app over at Splunkbase to send all Sandfly events directly to Splunk.
Server is Now Golang
The main server has been ported to Golang to match our agentless forensic engines which have been in Golang for some time. This has resulted in a faster server using fewer resources for better performance. The old server could already handle heavy loads, but the new server takes us to the next level of performance and scalability. We can easily handle massive Linux host loads with our agentless platform on very modest hardware.
Improved Filtering and Host Views
We have improved results filtering to make it faster with more options. We also have enhanced the host view to show more information such as hostnames by default plus general optimizations:
Improved Syslog Output for Graylog and Other SIEM Tools
For customers using Sandfly to send syslog output, we have improved the formatting in the result data. For syslog receivers such as Graylog that support building fields from structured syslog messages, the Sandfly messages now have a more consistent and useful structured data to build queries or alerts.
Faster Dashboard and General Operation
Updated caching along with database optimization has resulted in a much faster dashboard and general operation in the UI. All operations are sped up such as alert viewing, filtering and forensic data views.
Sandfly Hunter Optimized
Fully licensed users get access to Sandfly Hunter and it too has been sped up. Searching across hosts for Linux forensic artifacts is faster and more efficient. Use Sandfly Hunter to search for files, processes, users, log entries, SSH keys and more as part of an incident to help quickly find and isolate compromised hosts.
New Sandfly Checks for CronRAT and More
We have added in new checks for malware tactics used in CronRAT Linux malware and more. We could find this type of malware already, but have added new hunting modules for suspicious cron command entries and more to make detection even more complete.
Seamless Upgrade from 3.0 to 3.1
The new upgrade automatically ports your existing data from Elasticsearch over to the new database. Please see the upgrade documentation for how to quickly and easily upgrade.
Upgrade For Free
All free and paid customers can upgrade today. Please follow the directions below:
Upgrading Sandfly
Free License to Protect Your Linux Systems
We still offer a free license to use Sandfly immediately to help monitor and protect your Linux fleet. Get an instant no-obligation license right now.
Thank you for using Sandfly. Please contact us with any questions or comments.
Original post: https://www.sandflysecurity.com/blog/sandfly-3-1-released/