Divya Muntimadugu

Anjana Suparna Sriram

Red Hat Storage 2.0

2.0 Release Notes

Release Notes for Red Hat Storage 2.0
Edition 1

Red Hat Storage 2.0 2.0 Release Notes

Release Notes for Red Hat Storage 2.0
Edition 1

Divya Muntimadugu
Red Hat Engineering Co ntent Services
[e-mail] m

Anjana Suparna Sriram
Red Hat Engineering Co ntent Services
[e-mail] m

Legal Notice

Copyright © 2012 Red Hat Inc.

This document is licensed by Red Hat under the

Creative Commons Attribution-ShareAlike 3.0 Unported

License

. If you distribute this document, or a modified version of it, you must provide attribution to Red

Hat, Inc. and provide a link to the original. If the document is modified, all Red Hat trademarks must be
removed.

Red Hat, as the licensor of this document, waives the right to enforce, and agrees not to assert, Section
4d of CC-BY-SA to the fullest extent permitted by applicable law.

Red Hat, Red Hat Enterprise Linux, the Shadowman logo, JBoss, MetaMatrix, Fedora, the Infinity Logo,
and RHCE are trademarks of Red Hat, Inc., registered in the United States and other countries.

Linux ® is the registered trademark of Linus Torvalds in the United States and other countries.

Java ® is a registered trademark of Oracle and/or its affiliates.

XFS ® is a trademark of Silicon Graphics International Corp. or its subsidiaries in the United States
and/or other countries.

MySQL ® is a registered trademark of MySQL AB in the United States, the European Union and other
countries.

Node.js ® is an official trademark of Joyent. Red Hat Software Collections is not formally related to or
endorsed by the official Joyent Node.js open source or commercial project.

The OpenStack ® Word Mark and OpenStack Logo are either registered trademarks/service marks or
trademarks/service marks of the OpenStack Foundation, in the United States and other countries and
are used with the OpenStack Foundation's permission. We are not affiliated with, endorsed or
sponsored by the OpenStack Foundation, or the OpenStack community.

All other trademarks are the property of their respective owners.

Abstract

This Release Notes introduces Red Hat Storage and provides information including key features,
installing/launching and managing the software.

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Table of Contents

1. Introducing Red Hat Storage

2. What is New in this Release?

3. Installing Red Hat Storage

4 . Known Issues in Red Hat Storage v2.0

5. Product Support

6. Product Documentation

A. Revision History

2

2

5

5

6

6

8

Table of Contents

1

1. Introducing Red Hat Storage

Red Hat Storage is software only, scale-out storage that provides flexible and affordable unstructured
data storage for the enterprise. Red Hat Storage 2.0 provides new opportunities to unify data storage
and infrastructure, increase performance, and improve availability and manageability in order to meet a
broader set of an organization’s storage challenges and needs.

GlusterFS, a key building block of Red Hat Storage, is based on a stackable user space design and can
deliver exceptional performance for diverse workloads. GlusterFS aggregates various storage servers
over network interconnects into one large parallel network file system. The POSIX compatible GlusterFS
servers, which use XFS file system format to store data on disks, can be accessed using industry
standard access protocols including NFS and CIFS.

Red Hat Storage can be deployed in the private cloud or datacenter using Red Hat Storage Server for
On-premise. Red Hat Storage can be installed on commodity servers and storage hardware resulting in
a powerful, massively scalable, and highly available NAS environment. Additionally, Red Hat Storage can
be deployed in the public cloud using Red Hat Storage Server for Public Cloud, for example, within the
Amazon Web Services (AWS) cloud. It delivers all the features and functionality possible in a private
cloud or datacenter to the public cloud by providing massively scalable and high available NAS in the
cloud.

Red Hat Storage Server for On-Premise

Red Hat Storage Server for On-Premise enables enterprises to treat physical storage as a virtualized,
scalable, and centrally managed pool of storage by using commodity server and storage hardware.

Red Hat Storage Server for Public Cloud

Red Hat Storage Server for Public Cloud packages GlusterFS as an Amazon Machine Image (AMI) for
deploying scalable NAS in the AWS public cloud. This powerful storage server provides a highly
available, scalable, virtualized, and centrally managed pool of storage for Amazon users.

2. What is New in this Release?

This section describes the key features available in Red Hat Storage. The following is a list of feature
highlights of this new version of the Red Hat Storage software:

Unified File and Object Storage
Unified File and Object Storage (UFO) unifies NAS and object storage technology. It provides a
system for data storage that enables users to access the same data, both as an object and as a file,
thus simplifying management and controlling storage costs.
Replicate Improvements (Pro-active Self-heal)
In replicate module, previously you had to manually trigger a self-heal when a brick goes offline and
comes back online, to bring all the replicas in sync. Now the pro-active self-heal daemon runs in the
background, diagnoses issues and automatically initiates self-healing when the brick comes on-line.
You can view the list of files that need healing, the list of files which are recently healed, list of files
which are in split-brain state, and you can manually trigger self-heal on the entire volume or only on
the files which need healing.
Network Lock Manager
Red Hat Storage includes network lock manager (NLM) v4. NLM is a standard and an extension to
NFSv3 which allows NFSv3 clients to lock on files across the network. NLM is required to make
applications running on top of NFSv3 mount points to use the standard fcntl() (POSIX) and flock()

Red Hat Storage 2.0 2.0 Release Notes

2

(BSD) lock system calls to synchronize access across clients.
Volume Statedump
Statedump is a mechanism through which you can get details of all internal variables and state of the
glusterfs process at the time of issuing the command.You can perform statedumps of the brick
processes and nfs server process of a volume using the statedump command. The statedump
information is useful while debugging.
Volume Status and Brick Information
You can display the status information about a specific volume, brick or all volumes, as needed.
Volume status information includes memory usage, memory pool details of the bricks, inode tables of
the volume, pending calls of the volume and other statistics. This information can be used to
understand the current status of the brick, nfs processes, and overall file system. Status information
can also be used to monitor and debug the volume information.
Automated NFS and CIFS IP Failover
In replicated volume environment, you can configure Cluster Trivial Database (CTDB) to provide high
availability for NFS and CIFS exports. CTDB adds virtual IP addresses (VIPs) and a heartbeat service
to each Red Hat Storage Server.
When a node in a cluster fails, CTDB enables a different node to take over the IP address of the
failed node. This ensures the IP addresses for the services provided are always available.
Geo-replication Enhancements

Configuring Secure Geo-replication Slave
Now you can configure a secure slave using SSH so that master is granted a restricted access.
You need not specify configuration parameters regarding the slave on the master-side
configuration. You can also rotate the log file of a particular master-slave session, all sessions of
a master volume, and all geo-replication sessions, as needed. You can also set ignore-deletes
option to 1 so that the file deleted on the master will not trigger a delete operation on the slave.
Hence, the slave will remain as a superset of the master and can be used to recover the master
in case of crash and/or accidental delete.
Geo-replication Failback and Failover
Red Hat Storage 2.0 supports Geo-Replication failover and failback. If the master goes down, you
can trigger a failover procedure so that the slave can be replaced as the master. During this time,
all I/O operations including writes and reads are done on the slave. When the master is back
online, you can trigger a failback procedure so that the slave syncs the delta back to the master.
Geo-replication Checkpointing
Red Hat Storage 2.0 introduces a new introspection feature, Geo-replication Checkpointing. Using
Checkpointing, you can get information on the progress of replication. By setting a checkpoint, the
actual time is recorded as a reference timepoint, and from then on, enhanced synchronization
information is available on whether the data on master as of the reference timepoint has been
replicated on slave.

Mount Server Fail-over
Now there is an option to add backup volfile server while mounting fuse client. When the first volfile
server fails, then the server specified in backup volfile-server option is used as volfile server to
mount the client. You can also specify the number of attempts to fetch while mounting glusterFS
server. This option is useful when you mount a server with multiple IPs.
Debugging Locks
You can use statedump command to list the locks held on files. The statedump output also provides
information on each lock with its range, basename, PID of the application holding the lock, and so on.
You can analyze and know which locks are valid and relevant at a point of time. After ensuring that
no application is using the file, you can clear the lock using the clear lock command.
Change in Working Directory

1. Introducing Red Hat Storage

3

The working directory of glusterd has changed to /var/lib/glusterd from /etc/glusterd.
Gluster Volume Life-Cycle Extensions
Red Hat Storage allows you to define custom actions for volume events such as volume start, stop,
create, set, delete, and add brick. You can define both pre and post event actions. The actions can
be present as executables or scripts in the defined directory structure.
Agile Provisioning

Remove Brick Enhancements
Previously, remove-bick command was used to remove a brick that is inaccessible due to
hardware or network failure and as a clean-up operation to remove dead server details from the
volume configuration. Now remove-brick command can migrate data to existing bricks before
deleting given brick.
Rebalance Enhancements
Red Hat Storage 2.0 supports open file rebalance and files that have hardlinks. Rebalance is now
enchanced to be more efficient with respect to network usage, completion time, and amount of
data movement. It starts migration of data immediately without waiting for directory layout to be
fixed.
Dynamic Alteration of Volume Type
You can now change the type of the volume from Distributed volume to Distributed Replicated
Volume when performing add-brick and remove-brick operation. You must specify the replica
count parameter to increase the number of replicas to change it to distributed replicated volume.

Note

Currently, changing of stripe count while changing volume configurations is not supported.

Hadoop Compatible Storage (Technology Preview)
Red Hat Storage provides compatibility for Apache Hadoop and it uses the standard file system APIs
available in Hadoop to provide a new storage option for Hadoop deployments. Existing MapReduce
based applications can use Red Hat Storage seamlessly. This new functionality opens up data within
Hadoop deployments to any file-based or object-based application.

Important

Technology Preview features are not fully supported under Red Hat subscription level
agreements (SLAs), may not be functionally complete, and are not intended for production use.
However, these features provide early access to upcoming product innovations, enabling
customers to test functionality and provide feedback during the development process. As Red
Hat considers making future iterations of Technology Preview features generally available, we
will provide commercially reasonable efforts to resolve any reported issues that customers
experience when using these features.

Red Hat Storage Console (Technology Preview)
Red Hat Storage Console is a powerful and simple web based Graphical User Interface for managing
a Red Hat Storage 2.0 environment. It helps Storage Administrators to easily create and manage
multiple storage pools. This includes features like elastically expanding / shrinking a cluster, creating
and managing volumes.
Granual Locking for Large Files (Technology Preview)
Enables using Red Hat Storage as a backing store for preserving large files like virtual machine
images. Granualar locking enables internal file operations (like self-heal) without blocking user level

Red Hat Storage 2.0 2.0 Release Notes

4

file operations. The latency for user I/O is reduced during self-heal operation.
Read-only Volume (Technology Preview)
Red Hat Storage enables you to mount volumes as read-only. While mounting the client, you can
mount a volume as read-only and you can also make the entire volume as read-only, which applies
for all the clients (including NFS clients) using volume set option.
RDMA (Remote Direct Memory Access) Support (Technology Preview)
You can optionally configure Red Hat Storage to work over RDMA.

3. Installing Red Hat Storage

For step-by-step instructions to install Red Hat Storage, see Red Hat Storage Installation Guide.

4. Known Issues in Red Hat Storage v2.0

This chapter provides a list of known issues at the time of release:

yum update command may fail when it is executed for the first time in Red Hat Virtual Storage
Appliance 3.2 with the following error "GPG key retrieval failed: [Errno 14] Could not open/read
file:///etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-auxiliary " .
Work Around: Run yum update command again and it works fine on subsequent runs.

Issues related to Rebalancing Volumes:

Rebalance does not happen if bricks are down.
Currently while running rebalance, make sure all the bricks are in operating or connected state.
After rebalancing a volume, if you run rm -rf command at the mount point to remove all contents
of the current working directory recursively without prompting, you may get "Directory not Empty"
error message.
Rebalance operation fails to migrate data in distributed striped volume.
Rebalance operation on geo-replicated volumes in the slave can lead to data inconsistency on
the slave till files are updated on master.

glusterfsd - Error return code is always 0 after daemonizing the process.
Due to this, scripts that mount glusterfs or start glusterfs process must not depend on its return
value.
After # gluster volume replace-brick VOLNAME Brick New-Brick commit command is
issued, the file system operations on that particular volume, which are in transit will fail.
RDMA transport is not supported in Red Hat Storage 2.0 release.
Issues related to Directory Quota:

Some writes can appear to pass even though the quota limit is exceeded (write returns success).
This is because they could be cached in write-behind. However disk-space would not exceed the
quota limit, since when writes to backend happen, quota does not allow them. Hence it is advised
that applications should check for return value of close call.
If a user has done cd into a directory on which the administrator is setting the limit, even though
the command succeeds and the new limit value will be applicable to all the users except for those
users’ who has done cd in to that particular directory. The old limit value will be applicable until
the user has cd out of that directory.
Rename operation (that is, removing oldpath and creating newpath) requires additional disk
space equal to file size. This is because, during rename, it subtracts the size on oldpath after
rename operation is performed, but it checks whether quota limit is exceeded on parents of
newfile before rename operation.

3. Installing Red Hat Storage

5

With striped volumes, Quota feature is not available.
When renaming a file, if the available free size is less than the size of the file, quota displays
"Disk limit exceeded" error without renaming the file.

Issues related to POSIX ACLs:

Even though POSIX ACLs are set on the file or directory, the + (plus) sign in the file permissions
will not be displayed. This is for performance optimization and will be fixed in a future release.
When glusterfs is mounted with -o acl, directory read performance can be bad. Commands like
recursive directory listing can be slower than normal.
When POSIX ACLs are set and multiple NFS clients are used, there could be inconsistency in the
way ACLs are applied due to attribute caching in NFS. For a consistent view of POSIX ACLs in a
multiple client setup, use -o noac option on NFS mount to switch off attribute caching. This could
have a performance impact on operations involving attributes.

If you have enabled Gluster NLM, you cannot mount kernel NFS client on your storage nodes.
Due to enhancements in Graphs, you may experience excessive memory usage with this release.
After you restart the NFS server, the unlock within the grace-period may fail and previously held locks
may not be reclaimed.
fcntl locking (NLM) does not work over IPv6.
You cannot perform NFS mount on a machine on which glusterfs-NFS process is already running
unless you use the NFS mount -o nolock option. This is because glusterfs-NFS has already
registered NLM port with portmapper.
If the NFS client is behind a firewall such as NAT (Network Address Translation) router, the locking
behavior is unpredictable. The current implementation of NLM assumes there are no NAT happening
to client's IP.
nfs.m ount-udp option is disabled by default. You must enable it if you want to use posix-locks on
solaris when you NFS mount the gluster volume. If you enable nfs.mount-udp option, while
mounting a subdirectory (exported using nfs.export-dir option) on linux, you must mount using -
o proto=tcp option.
For NLM to function properly, you must ensure that all the servers and clients have resolvable
hostnames. That is, servers must be able to resolve client names and clients must be able to resolve
server hostnames.
After replace-brick operation, the stat information is different on NFS mount and FUSE mount. This
happens due to internal time stamp changes when the replace-brick operation is performed.
GET and PUT commands fails on the large files while using Unified File and Object Storage.
Work Around: You must ensure to add node_timeout=60 variable in proxy, container, and object
server configuration files.
In Red Hat Storage 2.0, Object Expiration feature of Swift-1.4.8 version is not supported.
Excessive logging while deleting files when Quota or gsync-indexing options are enabled.
Geo-replication uses rsync to sync files from master to slave, but rsync does not sync mknod and
pipe files.
The following is a known missing (minor) feature:

locks - mandatory locking is not supported.

5. Product Support

You can reach support at

https://access.redhat.com/

.

6. Product Documentation

Red Hat Storage 2.0 2.0 Release Notes

6

Product documentation of Red Hat Storage is available at

https://access.redhat.com/knowledge/docs/Red_Hat_Storage/

.

5. Product Support

7

A. Revision History

Revision 1-15

Tue Dec 31 2013

Pavithra Srinivasan

Updated Known Issues chapter.

Revision 1-13

Tue Mar 19 2013

Divya Muntimadugu

Updated Known Issues chapter.

Revision 1-12

Tue Mar 12 2013

Divya Muntimadugu

Changed product support URL.

Revision 1-11

Tue Sep 18 2012

Divya Muntimadugu

Changed references from docs.redhat.com to access.redhat.com

Revision 1-10

Wed Jul 18 2012

Anthony Towns

Rebuild for Publican 3.0

Revision 1-8

Tue Jun 26 2012

Divya Muntimadugu

Version for 2.0 GA release

Revision 1-7

Tue May 22 2012

Divya Muntimadugu

Bug fixes

Revision 1-6

Thu May 10 2012

Divya Muntimadugu

Bug fixes

Revision 1-5

Thu May 03 2012

Divya Muntimadugu

Beta 2 updates and bug fixes

Revision 1-4

Fri Mar 30 2012

Divya Muntimadugu

Updated new features list in chapter 2

Revision 1-3

Thu Mar 22 2012

Divya Muntimadugu

Technical review comments incorporation

Revision 1-2

Tue Mar 20 2012

Divya Muntimadugu

Red Hat Storage Beta 1 release

Revision 1-1

Thu Mar 1 2012

Divya Muntimadugu

Red Hat Storage Alpha release

Revision 1-0

Mon Feb 27 2012

Divya Muntimadugu

Draft

Red Hat Storage 2.0 2.0 Release Notes

8