File Operations Monitoring With IBM Spectrum Scale File Audit Logging

With the release of Spectrum Scale 5.0, IBM is now offering File Audit Logging capability. Spectrum Scale File Audit Logging takes locally generated file system events and puts them on a multi-node message queue from which they are consumed and written to a retention enabled fileset. These events, called lightweight events, occur at the file system level. They are protocol agnostic, which means that they capture all access to a monitored file system from protocol exports to even root access that occurs directly on nodes. Spectrum Scale File Audit Logging is integrated into the system health infrastructure. Alerts are generated for elements of the message queue and the processes that consume the events and create the audit logs.

Key features of Spectrum Scale File Audit Logging

• Creates an audit trail of every file access in a Spectrum Scale filesystem.
• Allows monitoring of file access by every user including super user.
• Monitors all file access via Native GPFS (directly on the Spectrum Scale node) or SMB and NFS from Protocol Nodes.
• Enables companies to implement strict Corporate security policies with a wide range of Spectrum Scale ACLs and File Audit Logging events.

Use Cases

Data Security and Monitoring Data Access is increasingly becoming vital for companies to protect business critical data from insider threats and security breaches.
While the IBM Spectrum Scale Encryption process helps to secure the data at  rest, the new File Audit Logging feature helps customers monitor business critical data access at a more granular level.
As an IBM Business Partner, ATS Group provides an end-to-end Spectrum Scale cluster solutions architecture, implementation and managed support services to our Financial and Genomic Research customers.
One of our customers uses Spectrum Scale – SAS Grid cluster to store data for their economic, financial and strategic consulting services. They are currently using native Redhat Linux auditing process to monitor the data access from a Spectrum Scale filesystem. They intend to switch from native Redhat Linux Auditing process to Spectrum Scale File Audit Logging in order to monitory their business critical data access. Their business consultants access the data using many Analytical applications and also from Windows desktops via SMB. They asked us to evaluate the new Spectrum Scale File Audit Logging features in depth and come up with a detailed report about its capabilities, requirements, and performance.
Majority of clustered filesystems, which are POSIX compliant and support NFS/SMB ACLs have some kind of file access auditing process where as other filesystems which are not POSIX compliant like HDFS usually does not provide file access auditing functionality.  File operations on Windows based filesystems like NTFS can be monitored by locally or by enabling Windows file auditing policy on the Active Directory server. Lustre which is an open source clustered filesystem, also offers file auditing capability. Filesystem OneFS, used by EMC Isilon, also has built-in File Auditing capability. Filesystem CEPH which is primarily used as an object store does not provide File access auditing. Though few of the clustered filesystem have some kind of auditing functionality, Spectrum Scale File Audit Logging offers more flexibility and scalability. File operations across the entire cluster can be monitored by configuring Spectrum Scale File Audit logging on only three quorum nodes in the cluster. Spectrum Scale File Audit Logging captures all file operations in the cluster, when multiple applications and users access files on any node within the cluster. There are a wide variety of commercial software packages available to analyze, report and generate alerting events from the Spectrum Scale File Audit Logging information.

Spectrum Scale File Audit Logging – Requirements

• Spectrum Scale Advanced or Data Management Edition
• x86 or Power-8 Little Endian (Not supported on Big Endian)
• RHEL 7.x or Ubuntu 16.04 and above.
• 3 x Spectrum Scale Quorum Nodes running RHEL 7.x or Ubuntu that act as Apache – Zookeeper Nodes
• 3 x Spectrum Scale Nodes running RHEL 7.x or Ubuntu with a minimum of 5 GB local disk space that act as Apache – Kafka Message Queue Servers (Brokers)
• Spectrum Scale Nodes that act as Zookeeper and Kafka message queue servers need ports 2181, 9092 and 9093 along with port range 2888:3888 open for communication.

Spectrum Scale File Audit Logging Proof of Concept

This document describes a complete Proof of Concept of Spectrum Scale File Audit Logging completed on IBM Power-8 servers at ATS Group Innovation Center.

PoC – Environment

1. 2x Power-8 S822 Servers
2. PowerVM
3. 500 GB of IBM Flash Storage
4. 3x PowerVM LPARs running CentOS 7.4 Power-8 Little Endian Kernel
5. IBM Spectrum Scale 5.0.0.2 Advanced Edition

PoC – Objectives

1. Create a 3 -node Spectrum Scale cluster: 2x NSD Servers which also serve as Protocol Nodes, 1x NSD Client, All 3 nodes act as File Audit Logging Zookeeper Nodes and Kafka Message brokers.
2. Create a 250 GB Spectrum Scale filesystem on 5 x 50 GB NSDs from IBM flash storage.
3. Install and Configure and enable File Audit Logging on the Spectrum Scale Filesystem.
4. Access the Spectrum Scale filesystem on Windows Server via SMB.
5. Verify the entries and events generated by FAL File Audit Logging in the audit log.

PoC – Spectrum Scale Cluster Install and File Audit Logging Configuration

1. Download IBM Spectrum Scale 5.0.0.2 Advanced Edition package SpectrumScaleProto_ADV500PWRleLNX.tar.gz from IBM passport website.
2. Extract the RPMs and installation script from the package.
3. Enable File Audit Logging
4. Define Cluster name, Cluster Nodes, NSD servers, Protocol Nodes and configuration using the spectrumscale command.
5. Define Spectrum Scale filesystems.
6. Run Spectrum Scale Installation pre-requisite check:
7. Proceed with Spectrum Scale RPMs install.
8. Define filesystem for Audit Logging, Fileset for Audit logs and retention time.
9. Run Spectrum Scale deploy pre-requisite check
10. Deploy Spectrum Cluster and File Audit Logging which will install Apache-Zookeeper and Kafka RPMs.
11. Verify Audit Logging is enabled on the filesystem.
12. Verify Audit Logging Message Queues.
13. Verify the date-wise Audit Logs in Audit Log Fileset.

Spectrum Scale File Audit Logging Functionality Testing – Native GPFS File Access

At this point we are ready to test some basic File Audit logging functionality.

1. Let’s create a test file, read it , change its permissions, rename and delete it as shown.
2. Verify the entries in the Audit Log.

Spectrum Scale File Audit Logging  Functionality Testing – SMB Access on Windows

1. Create a test GPFS fileset and link the fileset.
2. Create a SMB Share:
3. Map the SMB Share on to a Windows Desktop and create a test folder.
4. Create few test files using Windows fsutil or any other methods.
5. Verify the entries in the Audit file log.
6. Test a ACL Change on the file from Windows.
7. Verify the Windows ACL Change in the Audit log.

Spectrum Scale File Audit Logging – Limitations

• Supported for filesystems created on Spectrum Scale 5.0.0 or later. Filesystems created on older versions of Spectrum Scale need to be upgraded to 5.0.0 or latest after upgrading all cluster nodes.
• Requires at least three Linux Quorum cluster nodes with x86/Power 8 Little Endian.
• Protocol Nodes or NSD servers can function as File Audit Logging nodes, but might add additional load on the nodes.
• AIX and Power 8 Big-Endian are not currently supported for File Audit Logging. So, if your cluster consists of these nodes, then data access from these nodes is not monitored.
• Available with Spectrum Scale Advanced and Data-management editions and not with Standard edition.
• Needs additional data for storing audit logs within the Spectrum Scale filesystem. These logs can occupy large space depending on filesystem activity, retention time, number of users/applications etc.
• Needs additional software for analyzing the audit logs and creating reports.
• Events generated by non-Linux and SLES nodes are not audited currently.
• Enabling File Audit Logging can impact Filesystem I/O performance.

Summary and Conclusion

IBM is continuously enhancing Spectrum Scale by adding more and more features and functionality that truly make it a Enterprise class clustered filesystem. Spectrum Scale File Audit Logging adds additional capability to Spectrum Scale meet the data security standards required that help companies protect and monitor their business critical data from a wide variety of security threats. Spectrum Scale File Audit Logging can be configured quickly on new Spectrum Scale 5.0.0 clusters and can replace native OS audit logging which do not has the capability of monitoring events generated from via SMB or NFS protocols. File Audit logging monitors all filesystem data access just from three nodes and obsoletes the need to configure native OS Audit logging on every node in the cluster.