Release Notes 3.2
Release Notes for iRODS 3.2, released on October 3, 2012.
The following is a brief summary of the additions and improvements made to iRODS 3.2 compared to the previous version, 3.1.
As usual, this release contains many new features developed in response to needs expressed by the user community and which supplement the base iRODS functionality in various ways. Each is very important to some sites but may be unneeded by others and can be configured and used as needed. Also see:
- Release Notes history of iRODS
- Release Notes 3.1
- Jargon Release Notes 3.2
- Jargon-core Release Notes 3.2
- iDrop Release Notes 3.2
Major New Features
- Workflow Objects (Workflow Structured Objects (WSO)). This feature provides a means to execute an iRODS workflow and realize the results of the execution as an active object. This elevates the workflow to an object-level and 'igetting' a workflow results in its automatic execution. The execution is based on a parameter file which has directives about the input files and parameters and output objects that are created during the execution. It also provides directives about caching files (including the workflow file and executables if needed) into iRODS and for versioning earlier execution results from the same parameter file. More information about this feature can be found at Workflow Objects (WSO).
- PAM/LDAP Authentication. System passwords can now be used for iRODS authentication via PAM (Pluggable Authentication Modules). PAM can be configured to interact with various authentication systems, including LDAP (Lightweight Directory Access Protocol). In this mode, for additional security SSL is enabled to protect the password exchange ('iinit') and subsequent to that an iRODS-generated short term (2 week) password is used (for other i-commands). See PAM Authentication for more information. Thanks to Chris Smith of Distrbuted Bio for collaboration in the design and development of this.
- Additional NETCDF development. The integration of NETCDF/Opendap functionality into iRODS continued under the DCF project where it is being used to access Ocean Observatories Initiative (OOI) data. In iRODS 3.1, we had added 7 API functions and 12 micro-services to wrap most read-only type NETCDF API calls. In iRODS 3.2, the following have been added:
- More API functions and micro-services including API calls to handle "groups", subsetting, etc.
- Two new iCommands:
- inc - print NETCDF header info (attributions, dimension and variables); subsetting operation.
- incattr - extract attributions from NETCDF files and register them in the iCAT as AVU; list AVU; query for files using AVU.
- Three new drivers have been added to allow iRODS to register and access NETCDF data served by Thredds Data Server (TDS), ERDDAP and Pydap servers using Opendap protocol.
- Direct Access Resources. Direct access resources refer to iRODS data resources that are accessible both through iRODS and through the filesystem. A typical usage scenario would be an environment in which there is a shared high performance file system mounted on a compute cluster via NFS, and on which iRODS has the files from this file system registered in order to provide meta-data annotation for the files in this file system (i.e. iRODS acts as an "overlay" for the UNIX file system). Thanks to Chris Smith for this extension.
- File System Meta-data collection. This feature, when enabled, collects file system meta-data (i.e. 'stat' system call information) to be stored alongside the collection and data object information (in a new ICAT table). This information is then available via general-query calls. Currently this feature is only supported on UNIX/Linux type systems. Thanks to Chris Smith for this extension.
- New storageadmin role This mode is best used when you want to let somebody add a storage server to your zone, but don't want to give them rodsadmin privilege. It allows a site to run an iRODS data server (non-ICAT server) with very limited privilege by running the server using user credentials for a user of type 'storageadmin'. When running a server with this privilege level, a user can connect to the server to perform operations of data objects (i.e. put and get), but the server can only relay client connections to the zone's ICAT server with NO_USER_AUTH privilege level, except for limited operations (GEN_QUERY_AN). Users need to connect to the ICAT-Enabled Server as their irodsHost. Thanks to Chris Smith for this feature (done in close collaboration with DICE).
- Language integrated general-query Support for has been added for general-query syntax in the rule language. A gen query expression evaluates to an object of type genQueryInp_t * qenQueryOut_t.
- Optional use of system-installed PostgreSQL/ODBC. The 'irodssetup' script can now optionally use system-installed Postgresql and ODBC (i.e. installed via a package manager). See System-installed Postgres/ODBC for more information.
- 'ils -A' groups not expanded. The listing of ACLs (ils -A) will now optionally no longer expand user-groups to individual users in those groups too (and 'g:' prepended to the group names). This can be important to sites at are using large numbers of user ACLs (such as 'iPlant'). To enable this, a particular specific-query needs to be defined via an 'iadmin' command. See Non-expanding Group ACLs for more information.
- Case-insensitive Queries. ICAT queries (via the general-query API call) can now optionally be done in a case-insensitive manner, converting columns contents to upper case. See 'imeta -h' and 'iquest -h' for information on how to utilize this in those clients.
- iadmin modrescdatapath There is a new iadmin sub-command to update data-obj paths. This is sometimes needed after a resource path is updated (if a resource has been moved). See the 'iadmin h modrescdatapath' for more information.
- Better 'ichmod -r' performance Changes were made in the SQL used for 'ichmod' operations to improve performance in instances where large numbers of access control items (ACLs) are being used (users/groups). Logic was modified to make use of SQL 'like' instead of 'substr' which allows the DBMS to utilize the indexes. (svn revision r5164)
- Support for Postgres 9.1+ Additions were made to the installation scripts to handle Postgres 9.1.0 and later. The logic updates the postgres config file if the default setting is not the way iRODS needs it (that is, it is version 9.1.0 or later). The 'standard_conforming_strings' needs to be off but is on by default in 9.1+. (r4941)
- Support for newer GSI versions and untyped installs irodssetup will now allow 'none' as an GSI installation type and then use the globus library names without a trailing '_' and type. For example, globus_gss_assist instead of globus_gss_assist_$(GSI_INSTALL_TYPE) (e.g. globus_gss_assist_gcc32dbg). There's also a change in a variable type that may be needed in newer GSI versions. (r5209)
- Rule Engine log files (reLog) files renewed. An extension has been made that allows reLog files to be renewed every 7 days, closing the existing one and opening a new one with a refreshed name (based on the date). Thanks to Thomas Ledoux of the French National Library for this.
- irmtrash --age option. A new option has been added to allow irmtrash to only remove data-objects older than a specified age (similar to itrim --age). In support of this, when a collection or data-object is moved, its modify-time is updated. See 'irmtrash -h' for more information.
- irsync --age option. A new option has been added to allow irsync to synchronize only files younger than the specified age. See 'irsync -h' for more information.
- Error stack printed by all i-commands. All i-commands will now print the error stack (specific error messages returned by the server), if present. This allows the showing of customized error messages that are added with msiExit, or in other ways added to the stack. Previously, only some i-commands did this, for example 'iadmin', which allowed more detailed error messages from the server (ICAT) to be displayed. Thanks to Jean-Yves Nief for this extension.
- Universal MSS Driver handles blanks in file names The Universal Mass Storage System driver was extended to handle file names that contain special characters such as blanks. Thanks to Jean-Yves for this extension.
- irodssetup for Kerberos. Extensions were made so that irodssetup will ask and optionally enable Kerberos and set the Kerberos location (previously this was done by editing make-config files) and also eliminate some compiler warnings for format specifiers in ikrb.c. These were provided by Martin Pollard of the Wellcome Trust Sanger Institute (thanks!).
- irodssetup for UNICODE. An advanced option was added to the irodssetup/configure system to enable UNICODE (previously done manually) as well as various advice messages concerning upgrades and manual options.
- Retries of large file transfers supported. Extensions were made so that --retries to work with the --lfrestart option of iput/iget so that large file transfers can be retried until they are finished.
- unregister option added to msiDataObjUnlink. An 'unregister" option to msiDataObjUnlink was added. This allows it to function like 'irm -U' and remove a data-object without attempting to remove the physical file.
- irysnc -K option. A '-K' option was added to irsync to verify the checksum value.
- Implementation of exclude patterns for ireg -C There is a new option to ireg, '--exclude-from', which takes as an argument a file of pathname pattern matches evaluated by fnmatch(3). During a recursive registration, any pathname that matches one of the patterns will not be registered. The --exclude-from file must be readable on the data server where the files will be registered , and the file format is one pattern per line. Lines starting with '#' are treated as comments. This to Chris Smith for this extension.
- Timestamp added to rule engine audit xmessages The format of the message header will now be 'iaudit:<timestamp>:<call label>'. Thanks to Chris Smith for this enhancement.
- Test suite improvmements and more platforms tested/supported. Various improvements and bug fixes were made to test scripts and more platforms tested on the new NMI BaTLab and elsewhere (see Testing iRods). Fixes were made for FreeBSD, Solaris, and Solaris_PC and for using Oracle_Xe as the ICAT DBMS. A test script bug was fixed that was found on a Mac OSX 10.8 platform.
- New scan server log script There is now a script that can be executed remotely to return lines from server log files bounded by input start and end times. See the help text in server/bin/cmd/readRodsLog.py for more. Thanks to Adil Hasan/KEK for this enhancement. (r5017)
- Pluggable modules started. In collaboration with RENCI, the pluggable modules features of E-iRODS are being integrated, starting with the pluggable micro-services. Some of this is now part of the iRODS source code but disabled (ifdef'ed out) by default. The next release will have some of this functionality available.
Bug Fixes (partial list) and Additional Improvements
- Fix for deleted rules. A bug was fixed that was causing the reServer to delete delayed repeated rules even though the rule executed successfully (svn revision r4893).
- msiStoreVersionWithTS. The msiStoreVersionWithTS micro-service was added, which can be used to create a timestamped backup version of a DataObj. (r4898)
- ichksum -a on compound resource A problem was corrected in 'icksum -a' (all replicas) where the checksum calculation was not performed for copies on the compound resource.
- imeta mod bug fix. A bug was fixed in handling 'imeta mod' where a new null value for units was not taken as new and so was defaulted to the old value. This was noticed in some new tests developed by DICE. This bug probably existed since 'imeta mod' was added in 2.3. (r4900).
- KEEPALIVE. KEEPALIVE was added to setsockopt for iRODS sockets so the system will periodically send a message on the connection even when idle. (r4908)
- symlink checks. Additional symlink checks were added for handling mounted collections. (r4910 and r4906).
- iput/iget --lfrestart bug fixed.. A problem was corrected related to the use of the --lfrestart option of iput/iget where the transfer could fail even when the transfer was 100% complete. icp now works when the source and target are in different remote zones. The problem was due to proxy a user privilege issue and was resolved by adding a new singleL1Copy function to copy at the L1 level. (r4909)
- msiSysMetaModify bug fix A bug was fixed in msiSysMetaModify where comparisons on input options could fail due to extra characters in a temporary variable, avoiding a failure in certain cases.
- msiSendMail bug fix A correction was made to avoid a segfault when a subject line is very long.
- data_mode duplicated in replicas chlRegReplica was corrected to duplicate the data_mode field when a replica is created. Thanks to Howard Lander of RENCI for this fix.
- cross-zone icp icp now works when the source and target are in different remote zones. The problem was due to proxy a user privilege issue and was resolved by adding a new singleL1Copy function to copy at the L1 level.
- Administrator Suicide Prevented A check is now made disallowing the admin from deleting their own user account as it is probably unintended and likely problematic. Admins can still delete each other, but there will always be one left standing.
- More checks in msiSendMail and related msiSendMail and msiAutoReplicateService (UnixSendEmail) now more carefully check the input strings before performing the call to execute 'mail' (r5162).
- Timestamps corrected for 'iput -f', etc . An ICAT function (chlModDataObjMeta) now ensures that the modify-time value is the iRODS-standard 11 digits in length so that comparisons will work properly. This was a problem in 'iput -f' (overwriting a data-object), and perhaps other cases, where a numerically correct but 10-digit value was inserted. (r4989)
- imeta mod with a new null unit field handled. A bug was fixed in handling 'imeta mod' where a new null string for units was not taken as new and so was defaulted to the old value (r4900)
- irsync/icp mounted-collection over-write. A bug in dealing with mounted-collections when over-writing existing files was fixed. The error would prevent the writing of the data-object and return an CAT_UNKNOWN_FILE error. (r5191)
- parseRodsPath. A problem with overlapping rstrcpy strings was corrected. This could cause occasional errors (incorrect strings converted from user input), at least on scientific linux and perhaps in other environments. (r5195)
Note: a long-standing issue has been discovered in the use of Boost libraries in iRODS which will be resolved in a subsequent release.
See the Release Notes for a history of iRODS via descriptions of each release.
Upgrading from 3.1 to 3.2: Note that a patch to the iCAT database is required in support of the new features. The 'irodssetup --upgrade' (equivalent to 'irodsupgrade') script will warn and prompt you about this. The 3.2 clients will work properly with a 3.1 server except, of course, when a new feature is involved. To install, unpack the tar file, cd iRODS and run './irodssetup' or './irodsupgrade'.
Added well after the release (May 24): For all releases, we recommend you upgrade all servers in your zone when you do the upgrade. For most releases, mixing of Server/Agents with previous versions will interoperate fine, but for 3.2 if part of your zone will be running 3.2, we recommend you upgrade the ICAT-Enabled Server (IES) to 3.2. If you attempt to install a new 3.2 non-IES (a resource server) while still using an 3.1 IES, it will fail due to some changes in the way certain types of structures are encoded (packed) in the protocol messages (Native and XML). We believe the 3.2 servers are backward compatible with pervious versions, but not, for 3.2, when a new Server/Agent is communicating with an old one (3.1 or earlier).