Leveraging Hammerscript and the HSTK for Data Management

Hammerscript is the proprietary query language used to query the metadata of files stored on the Hammerscript platform. These queries are used for a variety of purposes including defining data orchestration policies and for any manner of file system reporting.

In this document, we will provide a variety of examples of how you use the Hammerspace Toolkit and Hammerscript to query and manipulate file system metadata.

Some of the examples provided in this document include:

Identify data older than X years, add it to a Hammerspace collection, and then delete it.
List the number of files and total space used, per file owner
Quickly output the file count and space used for each of your user home directories
Quickly output the file count and space used for each file owner
Output the file count and space used for all your Hammerspace shares (in one view)
Output the top 10 files by space used for all your Hammerspace shares (in one view)
Output the file instance count and space used for each cluster volume, on a per share basis

What Is the Hammerspace Toolkit (HSTK)?

The Hammerspace Toolkit (HSTK) is a collection of Python-based extensions developed for Linux Bash. However, it can also be used in the Windows command line (CMD or PowerShell) if either Python 3 or Windows Services for Linux (WSL) installed. Furthermore, the Hammerspace Toolkit additionally offers the ability to simplify common commands using the system wide bash completions outlined in the documentation.

There are many possible uses for the HSTK. Some of the examples featured in this document include:

Developing highly customized filesystem reports, which are delivered quickly and use real time data
Reviewing and managing file and directory metadata
Reviewing and managing file and directory objectives
Offloading resource intensive filesystem operations to the cluster directly

What Is the Hammerspace Metadata Plugin for Windows File Explorer?

The Hammerspace Metadata Plugin for Windows File Explorer enables users to right click a file or directory on a Hammerspace share, and then access and manipulate the more common custom metadata values without the need to use the HSTK. For the full power of Hammerspace advanced metadata, the HSTK should be used.

The Hammerspace Metadata Plugin can be downloaded from the Hammerspace licensing portal.
Additional information about the plugin is available in the Hammerspace Metadata Plugin for Windows section of this document.

What Is Metadata?

Simply put, metadata is data or information about other data. Anybody who has used a modern computer and operating system is familiar with concepts of files and directories within a file system. Directories and file names are the most basic metadata that we use daily, and are the most basic structures of a POSIX file system. POSIX also includes basic file properties like:

Size
Ownership (user and group)
Permissions (Unix style rwx or an Access Control List)
Timestamps including last access time (atime), last time the file contents were modified (mtime), and last time the file metadata or inode (think fileheader) was changed (ctime). Some also include a birth or create time.

Hammerspace manages and orchestrates unstructured data by separating metadata from data. Metadata is stored in a database on an Anvil node, rather than in inode, directory, and other structures in a volume as with other filesystems. Data, or file contents, is stored as an instance in one or more volumes on any or a combination of DSX nodes, NFS servers, or object storage.

Key Terms

Term	Definition
Anvil	A Hammerspace metadata server.
Clone	A duplicate of a file or a share.
DSX	Hammerspace data services node, which could be a direct storage or just as a data portal and a mover.
Hammerscript	A language for querying and manipulating metadata and related concepts and features of Hammerspace.
HSTK	Hammerspace Toolkit - Command line extensions that enable access to and manipulation of Hammerspace metadata.
Inode	A data structure in a Unix-style file system that describes a file-system object such as a file or a directory.
Instance	When a file is placed on a Hammerspace share, one or more instances will be written to cluster volumes based on the objective requirements. In a global file share, the only instances may be on a volume at another site.
NAS	Network attached storage.
Objective	Uses metadata criteria to orchestrate data.
Retention policy	Specified length of time to keep a file system snapshot or file clone.
Schedule	Specified time and frequency to take a snapshot or make a file clone.
Shadow Namespace	The shadow Namespace provides hidden access to Hammerspace metadata and functionality through a mounted file system.
Share	A file system presented for file access by clients via NFS, SMB, and/or S3 protocols. When a client writes a file to a Hammespace share, the cluster then uses objectives to determine where to place the file.
SLO	Service-level objective. A target value or ranges of values measured by service-level indicators. In Hammerscript, SLO often refers to an objective to place file instances on a specific volume, volume group, or storage system.
SMB	Server message block. A network communication protocol for providing shared access to files, printers, and serial ports between nodes on a network.
Snapshot	A record of the state of the file system at a specific point in time.
Volume	A storage container used by Hammerspace to store instances of files. May be a disk device on a DSX node, an export on a third-party storage system, or a bucket in cloud or on-premises object storage. Volumes are selected based on the objectives that apply to the file.

Term

Definition

Anvil

A Hammerspace metadata server.

Clone

A duplicate of a file or a share.

DSX

Hammerspace data services node, which could be a direct storage or just as a data portal and a mover.

Hammerscript

A language for querying and manipulating metadata and related concepts and features of Hammerspace.

HSTK

Hammerspace Toolkit - Command line extensions that enable access to and manipulation of Hammerspace metadata.

Inode

A data structure in a Unix-style file system that describes a file-system object such as a file or a directory.

Instance

When a file is placed on a Hammerspace share, one or more instances will be written to cluster volumes based on the objective requirements. In a global file share, the only instances may be on a volume at another site.

NAS

Network attached storage.

Objective

Uses metadata criteria to orchestrate data.

Retention policy

Specified length of time to keep a file system snapshot or file clone.

Schedule

Specified time and frequency to take a snapshot or make a file clone.

Shadow Namespace

The shadow Namespace provides hidden access to Hammerspace metadata and functionality through a mounted file system.

Share

A file system presented for file access by clients via NFS, SMB, and/or S3 protocols. When a client writes a file to a Hammespace share, the cluster then uses objectives to determine where to place the file.

SLO

Service-level objective. A target value or ranges of values measured by service-level indicators. In Hammerscript, SLO often refers to an objective to place file instances on a specific volume, volume group, or storage system.

SMB

Server message block. A network communication protocol for providing shared access to files, printers, and serial ports between nodes on a network.

Snapshot

A record of the state of the file system at a specific point in time.

Volume

A storage container used by Hammerspace to store instances of files. May be a disk device on a DSX node, an export on a third-party storage system, or a bucket in cloud or on-premises object storage. Volumes are selected based on the objectives that apply to the file.