System z Batch Optimization: Another Pipes Option?

Over the last 20 years or so I have encountered many sites looking for solutions to streamline their batch processing, only to find that sometimes they are their own worst enemy, because their cautious Change Management approach means they will not change or even recompile COBOL application source, unless absolutely forced to do so.  Sometimes VSAM file tuning is the answer, sometimes identifying the batch critical path, and on occasion, finding that key file or database that is processed on several or more occasions, which might benefit from parallelism is the answer.

BatchPipes was first introduced with MVS/ESA, allowing for data (E.g. BSAM, QSAM) to be piped between several jobs, allowing concurrent job processing, reducing the combined elapsed time of the associated job stream.  BatchPipes maintains a queue of records that are passed between a writer and reader.  The writer adds records to the back of a pipe queue and the reader processes them from the front.  This record level processing approach avoids any potential data set serialization issues when attempting to concurrently write and read records from the same physical data set.

The IBM BatchPipes feature has evolved somewhat and BMC have offered similar functionality with their initial Data Accelerator and Batch Accelerator offering, subsequently superseded by MainView Batch Optimizer Job Optimizer Pipes.  It seems patently obvious that to derive the parallelism benefit offered by BatchPipes, the reader and writer jobs need to be processed together.  For many, such a consideration has been an issue that has eliminated any notion of BatchPipes implementation.  Other considerations include a job failure in the BatchPipes process, where restart and recovery might include several jobs, as opposed to one.  Therefore widespread usage of BatchPipes has been seemingly limited.

The first step for any BatchPipes consideration is identifying whether there is any benefit.  IBM provide a BatchPipes SMF analysis tool to determine the estimated time savings and benefits that can be achieved with BatchPipes.  This tool reads SMF record types 14, 15 and 30 (Subtypes 1, 4 and 5) to analyse data set read and write activity, reconciling with the associated processing job.  As an observation, sometimes a data source might have a different data set name, be both permanent and temporary, while consuming significant I/O and CPU resource for processing.  Such data source reconciliation can easily be achieved, as the record and associated I/O count for such a data source is the same, for entire data set processing operations.  The analysis tool will identify the heavy I/O jobs and be a great starting point for any analysis activities.

UNIX users will be very familiar with the concept of pipes, where a UNIX pipeline is a sequence of processes chained together by their standard streams, where the output of each process (stdout) feeds directly as input (stdin) to the next one.  Wouldn’t it be good if there was a hybrid approach to BatchPipes, using a combination of standard z/OS and extended UNIX Systems Services (USS)?

With z/OS 2.2, JES2 introduced new functions to facilitate the scheduling of dependent batch jobs.  These functions comprise Job Execution Control (JEC) and can be utilized by making use of the new JOBGROUP and related Job Control Language (JCL) statements.  The primary goal of JEC is to provide an easy-to-use control mechanism, allowing complex batch jobs to be processed in inter-related constituent pieces.  Presuming that these constituent pieces can be run in parallel, improved throughput can be achieved by exploiting the concurrency functions provided by JEC.

UNIX named pipes can be used to pass data between simultaneously executing jobs, where the UNIX pipe can either be temporary or permanent.  One or more processes can connect to a UNIX named pipe, write to it, and read from it, as and when required.  Unlike most types of z/OS UNIX files, data written to a named pipe is always appended to existing data rather than replacing existing data.  Therefore, the STOR command is equivalent to the APPE command when UNIXFILETYPE=FIFO is configured.  This UNIX pipe facility, managed by the JES2 JEC functions can be leveraged to provide benefit for multiple step job processing and concurrent job processing, with the overall benefit of a reduction in overall batch stream elapsed time.

In conclusion, the advancement in JES2 JEC processing simplifies the batch scheduling and restart configuration processing, while the usage of UNIX named pipes leverages from existing z/OS USS functionality, safeguarding good performance using a tried and tested process.

Finally, returning full circle to my initial observation of Change Management considerations when performing batch optimization initiatives; recently I worked with a customer I visited in 2001, where they considered and dismissed BatchPipes Version 2.  We piloted this new UNIX pipe facility in Q4 2016, in readiness for their Year End processing, where they finally delivered a much needed ~2 Hour reduction in their ~9 Hour Critical Path Year End batch process.  Sometimes patience is a virtue, assisted by a slight implementation tweak…

Mainframe ISV Software: Is Continuous Product Improvement Always Evident?

Ken Venturi once said “I don’t believe you have to be better than everybody else.  I believe you have to be better than you ever thought you could be”.

Wouldn’t it be great if every CTO and/or Product manager had this same philosophy for their Mainframe software solution?  One such example I have experienced over the years is (E)JES from Phoenix Software International (PSI).  Of course it’s really important to have Day 1 support for the latest release of Operating System, z/OS 2.1 being the latest example, but what about actually exploiting the latest functionality available with the latest zSeries Mainframe Enterprise Servers and z/OS Operating Systems?

To drive maximum bang from you’re your buck, optimal performance and robust cost optimization can only be possible by recognizing and exploiting the latest Mainframe function ASAP, as and when appropriate.  Furthermore, listening to your customers, analysing their feedback, actively participating in User Organizations such as SHARE, and so on, will all help in continuous product development and innovation.

Here are some of the reasons why (E)JES has succeeded over a 30+ year period, recognizing and exploiting new z/OS function, as and when the updated z/OS is released for General Availability (GA).  Even today, with Version 5.3 supporting z/OS 2.1 as of day 1, (E)JES continues to offer value-added function for the seasoned, inexperienced and in fact, all IBM Mainframe technicians:

  • 64-bit performance optimizations (I.E. MEMLIMIT: above-the-bar) for both (E)JES client and server components, safeguarding minimal z/OS resource usage.
  • Nearly all (E)JES JES subsystem processing routines are eligible for zIIP redirection, delivering software cost savings for all (E)JES users.  Sub-Capacity System z processor users experience improved (E)JES performance because zIIP engines always run at full speed.  This behaviour differs from that of General Purpose CPs, “throttled” with Sub-Capacity deployments.
  • (E)JES code executes faster via its inbuilt High Performance Routine (HPR) facility, specifically developed to make (E)JES code execute faster while accessing data in JES control blocks.  HPRs have a shorter instruction path length than previous coding techniques, avoiding delays in modern z Series CPU instruction pipelines.
  • If High Performance FICON (zHPF) is available, (E)JES uses Transport Mode channel programs for JES Spool I/O.  When zHPF is not available, or when a CAS server performs I/O against the global data set, (E)JES uses the highest-performing Command Mode channel programs currently available.  These channel programs perform I/O significantly faster than “ordinary” channel programs.
  • The use of 24-bit (captured) UCBs puts a strain on the 24-bit virtual storage resource.  The use of ordinary (non-extended) TIOT entries puts a limit on the total number of allocations that can exist simultaneously in an address space.  (E)JES supports and uses 31-bit (uncaptured) UCBs and the extended TIOT (XTIOT) function (I.E. NON_VSAM_XTIOT=YES in DEVSUPxx PARMLIB)
  • (E)JES supports placement of JES spool data sets in the cylinder-managed area of an Extended Address Volume (EAV).  Of course, as of z/OS 1.12, EAV increases 3390 DASD capacity to ~1 TB.
  • (E)JES Pattern Utility Matching uses the SRST hardware instruction.  Empirical measurements show this technique is far faster on modern System z processors than alternatives such as the TRT instruction or “brute force” matching techniques using CLI/CLC.

One of the primary benefits of upgrading IBM z/OS software is the overall system performance benefit and associated cost reduction, but of course, IBM can only deliver the function and ability, while it’s incumbent upon the ISV community to upgrade their software products accordingly.  A key goal for any good ISV software product is to try to provide a value-add in the area of performance.  This has been one of the primary areas of focus for (E)JES since its introduction in 1978. 

Most spool display and management products tend to rely on the most resource-intensive interface available, namely the JES subsystem provided SSI 80.  (E)JES benchmarking tests against the most readily-available JES SSI 80 exploiters demonstrates significant CPU savings when deploying (E)JES.

Software products also need to deliver continuous improvements with regard to usability, presentation and in-built function, increasing user and system administrator productivity.  Without doubt, optimization encompasses not just hardware, but software, services, systems management disciplines and “best practices” that tie it all together.  Here are some of the usability enhancements that (E)JES has incorporated:

  • ISPF users running a 3270 emulator on a programmable workstation can now search IBM Eclipse-based InfoCenters via (E)JES.  Although (E)JES fully supports BookManager format documentation, BookManager READ/MVS is now obsolete, beginning with z/OS 2.1, BookManager softcopy books are no longer delivered by IBM.  IBM has stated that InfoCenters, and eventually KnowledgeCenters, are their strategic direction for online documentation.
  • (E)JES Web is a new, browser-based interface to (E)JES.  The associated RESTful API delivering this web enabled technology provides a framework for the creation of Eclipse plug-ins, mobile applications, and other web services clients.  This facility will provide a “rapid learning” type facility for users (E)JES users, both new and old that might be uncomfortable navigating traditional 3270 interfaces.
  • (E)JES provides a Java Application Programming Interface (API), complementing other in-built APIs for REXX and procedural languages.  By using an (E)JES API, a user can harness the versatility of their preferred programming language to interface and interact with (E)JES.  This support provides an interface to deliver nearly all of the capabilities available to an interactive (E)JES user.
  • (E)JES incorporates context sensitive help function, with point-and-shoot/pop-up dialogs, helping educate users on (E)JES, JES and z/OS while they work.  Users can get pop-up explanations of columns, input choices for unprotected fields, and a list of line commands.  Smart pop-ups explain the contents of certain columns, such as system abend codes.

The latest (E)JES Release Information Manual eloquently details the product enhancements over the last 5 releases or so, providing a good Product Roadmap reference point.

So, whether the ISV software product you deploy has been available for several years or several decades, do you safeguard maximum business benefit for optimal cost by considering:

  • Does the ISV deploy the latest zSeries server (I.E. zBC12, zEC12) for software interoperability and full hardware function exploitation; or an emulation (I.E. zPDT) technique?
  • Does the ISV deliver value-added z/OS related function on Day 1 or even within a year of the latest z/OS release?
  • Does the ISV deliver meaningful function to assist your users deploy said function, while simplifying environment management for system administrators?
  • Does your ISV product optimize cost, with Sub-Capacity pricing in MSU increments, aggregated MSU costs for your entire zSeries Mainframe environment, as opposed to specific workloads (E.g. CPC’s, LPAR’s, et al)?
  • Does your ISV product optimize cost by offloading the majority of its CPU function to zIIP specialty engines, which run at maximum speed, and where software “runs for free”?

Of course, only you can ask and potentially answer these questions during your day-to-day activities of maintaining currency and optimal performance for your Mainframe software portfolio.

Sometimes the hardest questions anybody can ask are the questions they ask themselves, which are never rhetorical questions!  Extracted verbatim from the latest (E)JES Release Information Manual:

Team (E)JES took advantage of the Phoenix Software International zHISR performance analysis product to discover performance “hot spots” in  the (E)JES product.  Sometimes the simplest, least conspicuous piece of code turns out to be a major CPU contributor.  See below for some of the most embarrassing “surprise” hot spots we discovered using zHISR in a z/OS 2.1 LPAR:

  • Over 30% of the CPU used during a Spool Data Browse FIND operation, against a multi-million-line SYSOUT in JES2, turned out to be code that was clearing a record buffer to blanks using MVCL.  This clearing code was eliminated and some minor adjustments were made in other code to compensate for this change.
  • 27% of the CPU used to produce the Activity display in JES2 turned out to be in a routine that manages an internal resource called the “Job Positions Table.”  The algorithm was improved (to work more like its JES3 counterpart) and that routine is no longer a significant CPU contributor.
  • 9% of (E)JES session start-up was a 26-year-old “brute force” prime number generator used to compute the size of a hash table.  That code was totally reworked and now accounts for approximately .02% of session start-up CPU.
  • A 6% performance penalty was observed when sorting a tabular display with a moderate number of rows. The hot spot turned out to be the code that cleared the work area for the sort service to zeros (another MVCL). This overhead was reduced to .04%.

Mea culpa and humility, never a bad thing, but you have to be honest with yourself and ask yourself the right questions!  So going back full circle and quoting Ken Venturi once again, “I don’t believe you have to be better than everybody else.  I believe you have to be better than you ever thought you could be”.  You must draw your own conclusions as to whether such an observation applies to the (E)JES team at Phoenix Software International (PSI)…

Why not ask them yourself?  Ed Jaffe, the (E)JES CTO will be available at the forthcoming UK GSE Annual Conference, 5-6 November 2013, speaking about (E)JES System Management Software: More With Less For Less, For The z/OS Mainframe and z/OS 2.1 User Experiences.