Is The Mainframe A Good Repository For Enterprise Wide User Passwords?

Posted on 04/02/2014 by zman

The subject matter of creating and maintaining passwords is arguably infinite and for the purposes of this article, we will provide a concise review…

In an ideal world, strong multiple factor authentication techniques would be deployed for every user authentication access attempt, including:

Biometrics – Unique measurable attribute (E.g. Voice, Fingerprint, Retina, et al)
Tokens – A physical device (E.g. Smart Card, One Time Password, et al)
User Secret – Something you know (E.g. Password, Phrase, PIN, et al)

Obviously the more authentication techniques used in combination, the stronger the authentication process becomes!

Primarily due to cost and complexity, passwords remain the most pervasive form of user authentication. This simple fact in itself exposes the human being as the primary vulnerability in safeguarding access to business systems.

However, passwords are simply just words, phrases or a string of characters that can be easily remembered by the user. As such, passwords can be compromised in numerous scenarios, for example:

Hardcopy – The written word; users write them down and/or share them with others.
Cracking – Passwords can be guessed; typically a simple program designed to try many possibilities in rapid succession. Simple passwords might be guessed by another human being.
Unsecure Transmission – Passwords no matter how complex are transmitted over an unsecure network in a simplistic (E.g. text) form, or with basic encoding, which can be easily converted to text.
Inappropriate Storage – Passwords are stored on a server, fixed or removable media storage, in a simplistic (E.g. text) form, or with basic encoding, which can be easily converted to text.

These potential vulnerabilities generate possibilities for somebody to obtain a password and subsequently access a business system as the user associated with their password. The potential consequences are obvious, depending on the importance of the user…

However, if password systems are implemented to deny malicious attacks, inspection or decryption of passwords being transmitted over the network, or at rest on fixed or removable storage media; passwords can be very secure. Therefore a combination of technology and good practice is required, safeguarding compliant and latest technology systems are deployed, educating users not to be the point of vulnerability, by allowing others to easily access their password.

There might be some urban myths as to whether the IBM Mainframe is a good platform for enterprise wide password management, for example:

Sniffing For Mainframe Passwords (This scenario depends on the lack of an SSL infrastructure)
CRACF (This Mainframe password cracking utility identifies simple user/password/group vulnerabilities)

Both of these scenarios are examples of whether “reverse engineering” thinking is good practice. So let’s pose as a potential hacker and see if we can obtain a user and associated password. These scenarios highlight the combined requirement of deploying a secure environment and safeguarding that user’s don’t and indeed are not allowed to create simplistic (low strength) passwords.

Ultimately password strength is governed by password length and associated combination of characters, including alphanumeric, upper/lower case, special characters, et al. There are also some other urban myths regarding the IBM Mainframe, regarding the maximum length of password (E.g. 8 Characters) and the type of character supported (E.g. only alphanumeric uppercase). For many years, RACF has supported the password phrase extension to the password rules, increasing password length to 100 characters:

Maximum length: 100 characters
Minimum length: 9 characters, when ICHPWX11 is present and allows the new value or 14 characters, when ICHPWX11 is not present
The user ID (as sequential upper case characters or sequential lower case characters) is not part of the password phrase
At least 2 alphabetic characters are specified (A – Z, a – z)
At least 2 non-alphabetic characters are specified (I.E. numeric, punctuation, special characters, blanks)
No more than 2 consecutive characters are identical

The use of high strength passwords is required because although human beings might give up after trying tens or maybe hundreds of password guesses, automated programs can achieve millions of password access attempts in a second, for example:

L0phtCrack (Windows, UNIX and Linux)
John the Ripper (Windows, Mac OS, UNIX and Linux)

There will always be a debate as to whether Single Sign On (SSO) or password synchronization is the best solution for maintaining password integrity and both solutions have their merits. Once again, a multiple authentication factor solution increases the security strength of either solution.

Passwords are most vulnerable when they’re forgotten and intervention is required to reinstate the password. Traditionally password resets were performed by an IT Support resource (human being) and this human interaction process generates what are termed “social engineering” challenges. Let’s explore a typical scenario, while considering any exposure and circumvention techniques:

Password Reset: IT Support Process

User has forgotten or mistyped their password (log-in denial/intruder alert)
User contacts IT support function (might encounter a no response or queue waiting scenario)
IT support asks user for credentials (E.g. name, department, et al)
IT Support authenticates this information with some on-line resource/authenticates user
IT support resets password or not, depending on whether user is “manually” authenticated
User might be prompted to immediately change their password on first successful log-in attempt

The security weaknesses associated with this process are numerous and prone to human error, for example:

Obvious Security Weaknesses: Business Exposure

IT Support forgets to authenticate the user
On-line resources for authenticating the user are not available
User credentials are widely available and so “social engineering” exposes the system
Password reset authority is granted to many non-IT personnel, for work productivity reasons
Password reset activity is not tracked and so is not auditable, accountable or traceable
IT support now knows the user password

Having identified the potential simplistic vulnerabilities, we implement processes to eradicate them, for example:

Implementing Controls

IT support training to safeguard user authentication occurs for each and every password reset request
Safeguard sufficient and secure user authentication information is available to IT support personnel
Implement a password reset solution/process (E.g. software) to eliminate non-IT personnel password reset personnel (I.E. for non-standard scenarios)
Implement a self-service solution (E.g. software) that allows the user to change their passwords, based on previously supplied “security challenge” questions and answers

Where user authentication depends on a password, eliminating “human” intervention touch points wherever possible is mandatory, minimizing the opportunity for “social engineering” techniques to compromise security. We have also identified that the IBM Mainframe does offer a secure environment for retaining passwords with ultra-high-strength security and that as always, the IBM Mainframe remains difficult to hack…

There are many software products to assist password reset scenarios, some that are platform specific and some that don’t support the IBM Mainframe. For those customers with an IBM Mainframe, Vanguard PasswordReset is an enterprise wide self-help password reset solution.

Vanguard PasswordReset addresses the common problem of forgotten or expired passwords, allowing authorized users to quickly and securely change their passwords at any time without help desk intervention.

Easy to install and use Vanguard PasswordReset does not require any software on user workstations or any additional hardware, with a rigorous set of checks and balances to ensure that only authorized users can initiate password reset requests.

Users register with the Vanguard PasswordReset website by typing a series of questions and answers or answering a set of predefine questions. When users want to change their passwords, they log on to the Vanguard PasswordReset website, type the answers to the questions and reset their passwords. For increased security, Vanguard PasswordReset allows system administrators to set the number of questions that must be answered and other characteristics of the answers.

A self-service password solution such as delivers Vanguard PasswordReset the following benefits:

Eliminates lost productivity when users are unable to access computer applications.
Provides improved help-desk productivity by allowing support staff to concentrate on solving other issues rather than time-consuming password resets.
Enhances enterprise security by standardizing password reset activities and eliminating human error.
Reduces IT support costs by automating costly password resetting activities.
Helps retain customers by making it easier for them to access extranet and e-business environments.
Virtually eliminates actual or hidden costs associated with installing, administering, maintaining and retiring thin-client software on user work stations.

In conclusion, maintaining passwords for user authentication purposes is a complex, costly and all-encompassing activity. Eradicating human intervention and touch points wherever possible, minimizes the impact of “social engineering” attacks, while deploying highly secure software solutions further increases the integrity of the primary access method to mission-critical business data, namely user access via authentication.

The IBM Mainframe – 50 Years & Counting

Posted on 07/01/2014 by zman

On 7 April 1964 IBM announced the System/360, which is now recognized as the first IBM Mainframe computer system. IBM Board Chairman Thomas J. Watson Jr. called the event the most important product announcement in the company’s history. At a press conference at the IBM Poughkeepsie facilities, Mr. Watson said:

“System/360 represents a sharp departure from concepts of the past in designing and building computers. It is the product of an international effort in IBM’s laboratories and plants and is the first time IBM has redesigned the basic internal architecture of its computers in a decade. The result will be more computer productivity at lower cost than ever before. This is the beginning of a new generation, not only of computers, but of their application in business, science and government.”

More than 100,000 businessmen in 165 American cities today attended meetings at which System/360 was announced. 50 years later, I wonder whether there are 100,000 people that work with the IBM Mainframe in The USA and maybe globally…

During this 50 year evolution, the IBM Mainframe has seen opinion polarize, sometimes from the same person:

In March 1991, Stewart Alsop stated “I predict that the last mainframe will be unplugged on March 15, 1996.”
In February 2002, Stewart Alsop stated “It’s clear that corporate customers still like to have centrally controlled, very predictable, reliable computing systems, exactly the kind of systems that IBM specializes in.”

Obviously the IBM Mainframe server is still here and just like in 1964, in the early 1990’s it did evolve into just another server on the distributed network and the use of routers, incorporating POSIX compliance and so on…

As we all know, the IBM Mainframe has always evolved, continues to evolve and in theory, and often in real-life, can run any workload.

Let’s reprise some of the notable IBM Mainframe models and associated functions since April 1964:

*Family Name*	*Announced*	*Notable Function Introduction*
S/360	April 1964	24-bit addressing (32-bit architecture)
S/360	August 1965	Virtual storage
S/360	January 1968	High speed cache
S/370	June 1970	Disk & printer support
S/370	August 1972	Virtual storage & multi-processor support
S/370 XA	June 1983	Extended storage 24-bit/31-bit addressing
S/390 ESA	September 1990	ESA & OS/390 operating systems
zSeries (zArchitecture)	October 2000	z operating systems, 24/31/64-bit addressing supported concurrently
zSeries z9 EC	July 2005	zIIP specialty engine
zSeries z10 EC	February 2008	High capacity/performance (quad core CPU chip)
z196 (zEnterprise)	July 2010	96-way core design & distributed systems integration (zBX)
zEC12	August 2012	Integrated platform for cloud computing, integrated OLTP & data warehousing

It’s interesting to note that the purchase price of an IBM mainframe is about the same, comparing 1964 to 2014, let’s say~$100,000. Of course, you can’t compare the feeds and speeds of these machines, they’re exponentially different. However, just as the S/360 in 1964 played a pivotal part in shaping data processing for that decade, subsequent evolutions of the IBM Mainframe follow in that tradition, lowering the cost of IT and simplifying business management.

I’m sure a lot of us have enjoyed our time working with the IBM Mainframe server and long may that be the case, for future generations of IT professionals.

Mainframe Server Planning: Vendor Interaction

Posted on 08/12/2013 by zman

In the last few weeks I have encountered a couple of scenarios regarding Mainframe Server upgrades that have surprised me somewhat. The first was at the annual UK GSE conference during November 2013, where one of the largest UK Mainframe customers stated “we had problems regarding the capacity sizing of the IBM Mainframe server installed and our vendor was not very helpful in resolving this challenge with us”. The second was a European customer with 2 aging servers deployed, z9 BC, and they had asked their IBM Mainframe server vendor to provide an upgrade quotation. The server vendor duly replied, providing a like-for-like upgrade quotation, 2 new zBC12 servers, which at first glance seemed to be a valid configuration.

The one thing in common for these 2 vastly different Mainframe customers, the first very large, the second quite small, is that inadvertently they didn’t necessarily engage their respective vendors with the best set of questions or indeed terms of reference; while the vendors might say “ask me no questions and I’ll tell you no lies”…

For the 2^nd scenario, I was asked to quickly review the configuration provided. My first observation was to consolidate both workloads on 1 server. The customer confirmed, there was no business reason to have 2 servers, it was historic, and there wasn’t even a SYSPLEX between the 2 z9 BC servers. The historic reason for the 2 z9 BC servers was the number of General Purpose (GP) engines supported. My second observation was that software licensing could be simplified and optimized with aggregated MSU and use of the AEWLC pricing model. So within ~1 hour, the customer had a significant potential to dramatically reduce costs.

We then suggested an analysis of their configuration with 2 software products, PerfTechPro for z/OS and zDynaCap. They already had the SMF data, so using the simulation abilities of these products, the customer quickly confirmed they could consolidate their workloads onto 1 zBC12, deploy zIIP processors to offload ~15% CPU usage from GP, and control MSU allocation with zDynaCap, saving another ~10% of CPU. For this customer, a small investment in software products reduced their server upgrade costs by ~€400,000 in year 1, with similar software savings, each and every year forever more. Although they didn’t have the skills in-house from a Mainframe Capacity Planning and software licensing viewpoint, this customer did eventually ask the right questions, and the rest as they say is history!

No man or indeed Mainframe customer is an island, so don’t be afraid to ask questions of your vendors or business partners!

From a cost viewpoint, both long-term (TCO) and day 1 (TCA), the requirement to deploy the optimum Mainframe server configuration from a capacity viewpoint cannot be under estimated, both in terms of hardware costs, but more importantly, associated software costs. It therefore follows that Mainframe Capacity Planning and Mainframe Software Licensing knowledge is imperative, but I’m not so sure there are that many Mainframe customers that have clearly defined job roles for such disciplines.

To generalize, always a dangerous thing, typically the larger Mainframe customer does have skilled and seasoned personnel for the Capacity Planning discipline, while the smaller Mainframe user might rely on a generic Systems Programmer or maybe even rely on their vendor to size their Mainframe servers. From a Mainframe software licensing viewpoint, there seems to be no general rule-of-thumb, as sometimes the smaller customer has significant knowledge and experience, whereas the larger Mainframe customer might not. Bottom Line: If the Mainframe customer doesn’t allocate the optimum capacity and associated software licensing metrics for their installation, problems will arise, probably for several years or more!

Are there any simple solutions or processes that can assist Mainframe customers?

The first and most simple observation is to engage your vendor and safeguard that they generate the final Mainframe server configuration that is used for Purchase Order activities. For sure, the customer will have their capacity plan and perhaps a “draft” server configuration, but even in these instances, the vendor should QA this data, refining the bill of materials (E.g. Hardware) accordingly. Therefore an iterative process occurs between customer and vendor, but the vendor is the one that confirms the agreed configuration is fit for purpose. In the unlikely event there are challenges in the future, the customer can work with their vendor to find a solution, as opposed to the example stated above where the vendor left their customer somewhat isolated.

The second observation is leverage from the tools and processes that are available, both generally available and internal for vendor pre sales personnel. Seemingly everybody likes something for nothing and so the ability to deploy “free” tools will appeal to most.

For Mainframe Capacity Planning, in addition to the standard in-house processes, whether bespoke (E.g. SAS, MXG, MICS based) or a packaged product, there are other additional tools available, primarily from IBM:

zPCR (Processor Capacity Reference) is a generally available Windows PC based tool, designed to provide capacity planning insight for IBM System z processors running various z/OS, z/VM, z/VSE, Linux, zAware, and CFCC workload environments on partitioned hardware. Capacity results are based on IBM’s most recently published LSPR data for z/OS. Capacity is presented relative to a user-selected Reference-CPU, which may be assigned any capacity scaling-factor and metric.

zCP3000 (Performance Analysis and Capacity Planning) is an IBM internal tool, Windows PC based, designed to for performance analysis and capacity planning simulations for IBM System z processors, running various SCP and workload environments. It can also be used to graphically analyse logically partitioned processors and DASD configurations. Input normally comes from the customer’s system logs via a separate tool (I.E. z/OS SMF via CP2KEXTR, VM Monitor via CP3KVMXT, VSE CPUMON via VSE2EDF).

zPSG (Processor Selection Guide) is an IBM internal tool, Windows PC based, designed to provide sizing approximations for IBM System z processors intended to host a new application, implemented using popular, commercially available software products (E.g. WebSphere, DB2, ODM, Linux Apache Server).

zSoftCap (Software Migration Capacity Planning Aid) is a generally available Windows PC based tool, designed to assess the effect on IBM System z processor capacity, when planning to upgrade to a more current operating system version and/or major subsystems versions (E.g. Batch, CICS, DB2, IMS, Web and System). zSoftCap assumes that the hardware configuration remains constant while the software version or release changes. The capacity implication of an upgrade for the software components can be assessed independently or in any combination.

zBNA (System z Batch Network Analysis) is a generally available Windows PC based tool, designed to understand the batch window, for example:

Perform “what if” analysis and estimate the CPU upgrade effect on batch window
Identify job time sequences based on a graphical view
Filter jobs by attributes like CPU time / intensity, job class, service class, et al
Review the resource consumption of all the batch jobs
Drill down to the individual steps to see the resource usage
Identify candidate jobs for running on different processors
Identify jobs with speed of engine concerns (top tasks %)

BWATOOL (Batch Workload Analysis Tool) is an IBM internal tool, Windows PC based, designed to analyse SMF type 30 and 70 data, producing a report showing how long batch jobs run on the currently installed processor. Both CPU time and elapsed time are reported. Similar results can then be projected for any IBM System z processor model. Basic questions that can be answered by BWATOOL include:

What jobs are good candidates for running on any given processor?
How much would jobs benefit from running on a faster processor?
For jobs within a critical path (batch window), what overall change in elapsed time might occur with a new processor?

zMCAT (Migration Capacity Analysis Tool) is an IBM internal tool, Windows PC based, designed to compare the performance of production workloads before and after migration of the system image to a new processor, even if the number of engines on the processor has changed. Workloads for which performance is to be analysed must be carefully chosen because the power comparison may vary considerably due to differing use of system services, I/O rate, instruction mix, storage reference patterns, et al. This is why customer experiences are unique from an internal throughput ratio (ITRR) based on LSPR benchmark data.

zTPM (Tivoli Performance Modeler) is an IBM internal tool, Windows PC based designed to let you build a model of a z/OS based IBM System z processor, and then run various “what if scenarios”. zTPM uses simulation techniques to let you model the impact of changes on individual workload performance. zTPM uses RMF or CMF reports as input. Based on these reports, zTPM can create summary charts showing LPAR as well as workload utilization. An automated Build function lets you build a model that represents the system for any reporting interval. Once the model is built, you can make changes to see the impact on workload performance. zTPM is also available as an IBM software product offering.

Therefore there are numerous tools available from IBM to assist their customers determine optimum Mainframe server capacity requirements. Some of these tools are generally available without engaging the IBM account team, but others are internal to IBM, and for that reason alone, Mainframe customers must engage their IBM Mainframe account team to participate in their capacity planning activities. Additionally, as the only supplier of Mainframe Servers, IBM have a wealth of knowledge and indeed a responsibility and generally a willingness to assist their customers deploy the right Mainframe server configuration from day 1.

As a customer, don’t be afraid to engage external 3^rd parties to perform a sanity check of your thinking and activities, clearly IBM as they will be fulfilling your IBM Mainframe server order. However, consider engaging other capacity/performance and software licensing specialists as their experience incorporates many customers, as opposed to an insular view. Moreover, such 3^rd parties probably utilize their own software tools or products to assist in this most important of disciplines.

In conclusion, as always, the worst question is the one not asked, and for this most fundamental of processes, not collaborating with your vendor and the wider community, might leave you as an individual exposed and isolated, and your company exposed to the consequences of an undersized or oversized Mainframe sever configuration…

Cloudy With A Chance Of Mainframe?

Posted on 01/11/2013 by zman

With the advent of Computer Generated Imagery (CGI) there is seemingly no end to the number of books, especially “children’s” books that can be encapsulated and delivered in animated movie format. I’m always surprised and arguably never surprised by the messaging in these stories; supposedly written for the younger person, but invariably delivering a message of good morals, ethics and human qualities, typically finding creative solutions to a myriad of problems. Of course, we’re all human, and typically as human beings, we’re responsible for the majority of our problems, either knowingly, or not.

Cloudy with a Chance of Meatballs is a book based on a town named Chewandswallow characterized by its strange daily meteorological pattern, providing townsfolk with all of their required daily meals by raining food. Although the residents of the town enjoy a lifestyle devoid of any grocery shopping or cookery, the weather unexpectedly and inexplicably takes a turn for the worse, devastating the local community with destructive and uncontrollable storms of either unpleasant or dangerously oversized foods, resulting in unstoppable catastrophes for the townspeople. Their lives endangered by the threats of the storms, they relocate to a different community of average meteorological patterns, safe from the hazards that once were presented by raining meals. However, they are forced to learn how to obtain food the normal way.

So what? Continuing with the creativity thought, the ethos of this story might be somewhat analogous to the sometimes polarized opinion between Distributed Systems and Mainframe computing. So depending on your philosophical bent or which side-of-the-fence you sit, there is only one choice, even if this seemingly perfect and de facto world is generating significant challenges…

Recently, z/OS 2.1 became Generally Available (GA) and most notably from my viewpoint was its continued and demonstrable ability to participate in cloud computing environments. So is the IBM Mainframe ready for the cloud? Wasn’t it always!

The fundamental ethos of the Mainframe environment is virtualization and was forever thus. The Mainframe has always shared the basic IT architecture components, including CPU, Memory, Storage, Networking and other peripherals, originally in a physical single-image structure, but since the late 1990’s in a shared (SYSPLEX) complex of interconnected physical servers (CPCs). So the Mainframe is and always has been ready for “Prime Time Cloud”!

z/OS V2.1 is a platform designed to dynamically respond and scale to workload change with enhancements to scalability and performance that cover operations, I/O, virtual storage constraint relief, memory management, and more. These enhancements are suitable for organizations that would like to catalyse a journey to highly scalable virtualized solutions like cloud.

IBM delivers improved scalability and performance for outstanding throughput and service within existing Mainframe environments. Smarter scalability can better prepare the user for growth and spikes in workloads while maintaining the qualities of service and balanced design that customers have come to expect of the IBM mainframe.

As customers consider all the components of downtime, the true costs can be surprising, which is why superior availability continues to remain a key factor in platform selection. With z/OS V2.1, IBM introduces new capabilities designed to improve upon the already legendary z/OS system availability. The industry-leading resiliency and high availability of System z remain key reasons why organizations keep their most critical processing on System z. With its attention to outage reduction, the availability of System z and z/OS is well recognized in the industry. In z/OS V2.1, IBM continues enhancements that improve critical IT systems availability, helping achieve an even higher level of service for customers.

Some of the “cloud friendly” z/OS 2.1 benefits include:

Support for Shared Memory Communications-RDMA (SMC-R), for low latency, application transparent communications to help you move data quickly between z/OS images on the same CPC or between CPCs.
Flash Express support for certain coupling facility list structures, such as IBM WebSphere MQ for z/OS, V7 (5655-R36), in order to strengthen resiliency for enterprise messaging workload spikes.
For zEC12 or zBC12 systems, shared engine coupling facilities can be used in many production environments, for improved economics by offering a high level of performance without requiring the use of dedicated CF engines.
EXCP support for System z High-Performance FICON (zHPF) is designed to help improve I/O start rates and improve bandwidth for more workloads on existing hardware and fabric.
Usability and performance improvements for z/OS FICON Discovery and Auto Configuration (zDAC), including discovery of directly attached devices.
Serial Coupling Facility structure rebuild processing, designed to help improve performance and availability by rebuilding coupling facility structures more quickly and in priority order.
100-way symmetric multiprocessing (SMP) support in a single LPAR on IBM zEC12 or zBC12 systems. Support for an architectural limit of 4 TB of real memory per LPAR.
Support for 2 GB pages is provided on zEC12 and zBC12 systems. This feature is designed to reduce memory management overhead and improve overall system performance by enabling middleware to use 2 GB pages. These improvements are expected due to improved effective translation lookaside buffer (TLB) coverage and a reduction in the number of steps the system must perform to translate a 2 GB page virtual address.
Capacity Provisioning is designed to provide support for manual and policy-based management of Defined Capacity and Group Capacity. This function broadens the range of automatic, policy-based responses available to help manage capacity shortage conditions when WLM cannot meet your workload policy goals.

There are numerous new and enhanced functions delivered with z/OS 2.1, too numerous to mention, but categorised as Quality Of Service, Availability, Networking, Security, Data Usability, Integrity, Systems Management, Application Development, Simplification & Usability, International Standards Compliance, et al.

So let’s not forget, this foundation and support for an IT infrastructure and its supporting eco (software) system is in one scalable, secure and “zero” downtime environment!

So maybe for us open-minded and enlightened generation of parents (oops, I forgot, Grandparents for us Dinosaur Mainframe folk!) that can now “access” children’s stories, even if it’s in the form of a CGI animated movie, maybe we can be dispassionate enough to consider all platforms, Distributed and Mainframe for our evolving business and associated IT requirements.

So you decide, can it be Cloudy With A Chance Of Mainframe? To overlook such an option, might be an oversight, just as overlooking the abundance of human stories, classified as children’s books or not…

Mainframe ISV Software: Is Continuous Product Improvement Always Evident?

Posted on 09/10/2013 by zman

Ken Venturi once said “I don’t believe you have to be better than everybody else. I believe you have to be better than you ever thought you could be”.

Wouldn’t it be great if every CTO and/or Product manager had this same philosophy for their Mainframe software solution? One such example I have experienced over the years is (E)JES from Phoenix Software International (PSI). Of course it’s really important to have Day 1 support for the latest release of Operating System, z/OS 2.1 being the latest example, but what about actually exploiting the latest functionality available with the latest zSeries Mainframe Enterprise Servers and z/OS Operating Systems?

To drive maximum bang from you’re your buck, optimal performance and robust cost optimization can only be possible by recognizing and exploiting the latest Mainframe function ASAP, as and when appropriate. Furthermore, listening to your customers, analysing their feedback, actively participating in User Organizations such as SHARE, and so on, will all help in continuous product development and innovation.

Here are some of the reasons why (E)JES has succeeded over a 30+ year period, recognizing and exploiting new z/OS function, as and when the updated z/OS is released for General Availability (GA). Even today, with Version 5.3 supporting z/OS 2.1 as of day 1, (E)JES continues to offer value-added function for the seasoned, inexperienced and in fact, all IBM Mainframe technicians:

64-bit performance optimizations (I.E. MEMLIMIT: above-the-bar) for both (E)JES client and server components, safeguarding minimal z/OS resource usage.
Nearly all (E)JES JES subsystem processing routines are eligible for zIIP redirection, delivering software cost savings for all (E)JES users. Sub-Capacity System z processor users experience improved (E)JES performance because zIIP engines always run at full speed. This behaviour differs from that of General Purpose CPs, “throttled” with Sub-Capacity deployments.
(E)JES code executes faster via its inbuilt High Performance Routine (HPR) facility, specifically developed to make (E)JES code execute faster while accessing data in JES control blocks. HPRs have a shorter instruction path length than previous coding techniques, avoiding delays in modern z Series CPU instruction pipelines.
If High Performance FICON (zHPF) is available, (E)JES uses Transport Mode channel programs for JES Spool I/O. When zHPF is not available, or when a CAS server performs I/O against the global data set, (E)JES uses the highest-performing Command Mode channel programs currently available. These channel programs perform I/O significantly faster than “ordinary” channel programs.
The use of 24-bit (captured) UCBs puts a strain on the 24-bit virtual storage resource. The use of ordinary (non-extended) TIOT entries puts a limit on the total number of allocations that can exist simultaneously in an address space. (E)JES supports and uses 31-bit (uncaptured) UCBs and the extended TIOT (XTIOT) function (I.E. NON_VSAM_XTIOT=YES in DEVSUPxx PARMLIB)
(E)JES supports placement of JES spool data sets in the cylinder-managed area of an Extended Address Volume (EAV). Of course, as of z/OS 1.12, EAV increases 3390 DASD capacity to ~1 TB.
(E)JES Pattern Utility Matching uses the SRST hardware instruction. Empirical measurements show this technique is far faster on modern System z processors than alternatives such as the TRT instruction or “brute force” matching techniques using CLI/CLC.

One of the primary benefits of upgrading IBM z/OS software is the overall system performance benefit and associated cost reduction, but of course, IBM can only deliver the function and ability, while it’s incumbent upon the ISV community to upgrade their software products accordingly. A key goal for any good ISV software product is to try to provide a value-add in the area of performance. This has been one of the primary areas of focus for (E)JES since its introduction in 1978.

Most spool display and management products tend to rely on the most resource-intensive interface available, namely the JES subsystem provided SSI 80. (E)JES benchmarking tests against the most readily-available JES SSI 80 exploiters demonstrates significant CPU savings when deploying (E)JES.

Software products also need to deliver continuous improvements with regard to usability, presentation and in-built function, increasing user and system administrator productivity. Without doubt, optimization encompasses not just hardware, but software, services, systems management disciplines and “best practices” that tie it all together. Here are some of the usability enhancements that (E)JES has incorporated:

ISPF users running a 3270 emulator on a programmable workstation can now search IBM Eclipse-based InfoCenters via (E)JES. Although (E)JES fully supports BookManager format documentation, BookManager READ/MVS is now obsolete, beginning with z/OS 2.1, BookManager softcopy books are no longer delivered by IBM. IBM has stated that InfoCenters, and eventually KnowledgeCenters, are their strategic direction for online documentation.
(E)JES Web is a new, browser-based interface to (E)JES. The associated RESTful API delivering this web enabled technology provides a framework for the creation of Eclipse plug-ins, mobile applications, and other web services clients. This facility will provide a “rapid learning” type facility for users (E)JES users, both new and old that might be uncomfortable navigating traditional 3270 interfaces.
(E)JES provides a Java Application Programming Interface (API), complementing other in-built APIs for REXX and procedural languages. By using an (E)JES API, a user can harness the versatility of their preferred programming language to interface and interact with (E)JES. This support provides an interface to deliver nearly all of the capabilities available to an interactive (E)JES user.
(E)JES incorporates context sensitive help function, with point-and-shoot/pop-up dialogs, helping educate users on (E)JES, JES and z/OS while they work. Users can get pop-up explanations of columns, input choices for unprotected fields, and a list of line commands. Smart pop-ups explain the contents of certain columns, such as system abend codes.

The latest (E)JES Release Information Manual eloquently details the product enhancements over the last 5 releases or so, providing a good Product Roadmap reference point.

So, whether the ISV software product you deploy has been available for several years or several decades, do you safeguard maximum business benefit for optimal cost by considering:

Does the ISV deploy the latest zSeries server (I.E. zBC12, zEC12) for software interoperability and full hardware function exploitation; or an emulation (I.E. zPDT) technique?
Does the ISV deliver value-added z/OS related function on Day 1 or even within a year of the latest z/OS release?
Does the ISV deliver meaningful function to assist your users deploy said function, while simplifying environment management for system administrators?
Does your ISV product optimize cost, with Sub-Capacity pricing in MSU increments, aggregated MSU costs for your entire zSeries Mainframe environment, as opposed to specific workloads (E.g. CPC’s, LPAR’s, et al)?
Does your ISV product optimize cost by offloading the majority of its CPU function to zIIP specialty engines, which run at maximum speed, and where software “runs for free”?

Of course, only you can ask and potentially answer these questions during your day-to-day activities of maintaining currency and optimal performance for your Mainframe software portfolio.

Sometimes the hardest questions anybody can ask are the questions they ask themselves, which are never rhetorical questions! Extracted verbatim from the latest (E)JES Release Information Manual:

Team (E)JES took advantage of the Phoenix Software International zHISR performance analysis product to discover performance “hot spots” in the (E)JES product. Sometimes the simplest, least conspicuous piece of code turns out to be a major CPU contributor. See below for some of the most embarrassing “surprise” hot spots we discovered using zHISR in a z/OS 2.1 LPAR:

Over 30% of the CPU used during a Spool Data Browse FIND operation, against a multi-million-line SYSOUT in JES2, turned out to be code that was clearing a record buffer to blanks using MVCL. This clearing code was eliminated and some minor adjustments were made in other code to compensate for this change.
27% of the CPU used to produce the Activity display in JES2 turned out to be in a routine that manages an internal resource called the “Job Positions Table.” The algorithm was improved (to work more like its JES3 counterpart) and that routine is no longer a significant CPU contributor.
9% of (E)JES session start-up was a 26-year-old “brute force” prime number generator used to compute the size of a hash table. That code was totally reworked and now accounts for approximately .02% of session start-up CPU.
A 6% performance penalty was observed when sorting a tabular display with a moderate number of rows. The hot spot turned out to be the code that cleared the work area for the sort service to zeros (another MVCL). This overhead was reduced to .04%.

Mea culpa and humility, never a bad thing, but you have to be honest with yourself and ask yourself the right questions! So going back full circle and quoting Ken Venturi once again, “I don’t believe you have to be better than everybody else. I believe you have to be better than you ever thought you could be”. You must draw your own conclusions as to whether such an observation applies to the (E)JES team at Phoenix Software International (PSI)…

Why not ask them yourself? Ed Jaffe, the (E)JES CTO will be available at the forthcoming UK GSE Annual Conference, 5-6 November 2013, speaking about (E)JES System Management Software: More With Less For Less, For The z/OS Mainframe and z/OS 2.1 User Experiences.

21st Century Mainframe Capacity Planning Requirements

Posted on 06/09/2013 by zman

With nearly 5 decades of longevity the IBM Mainframe has changed beyond recognition in terms of CPU capacity and performance capability. The Capacity Planning discipline for the IBM Mainframe server became more advanced and proactive in the early 1990’s, perhaps coinciding with the introduction of Parallel Sysplex structures associated with the MVS/ESA operating system. Therefore the requirement to measure and model the impact of workload movement between LPAR and CPC structures became important, if not mandatory.

The fundamental building-block for Mainframe CPU usage analysis is SMF Type 7n records (I.E. RMF or CMF), where this data was typically processed by MXG, MICS and maybe CIMS (acquired by IBM), generally using SAS for reporting purposes. Other tools, including but not limited to, BEST/1 (acquired by BMC) and PERFMAN (acquired by ASG) also offered capacity planning and performance management solutions. Therefore, for 20+ years the fundamental Mainframe CPU usage data and associated tools have remained largely the same. However, maybe the IBM Mainframe server has changed, both in terms of underlying CPU chip technology and customer workload deployment…

I often hear capacity planners state something along the lines of “I can report on the past with 100% accuracy, but predicting the future might prove to be a little more difficult”! Once again, going back to the early 1990’s, the IBM Mainframe had a typical if not generic workload profile deployment, namely On-Line Transaction Processing (E.g. CICS, IMS DC) and related Database Management Subsystems (E.g. DB2, IMS DB) with Batch Processing. This somewhat limited workload profile simplified the Capacity Planning process, applying estimates of growth based on current usage. However, when the Mainframe became more pervasive, taking on new workloads, how was the capacity planner supposed to estimate CPU requirements for their new business application workload?

IBM introduced the Large Systems Performance Reference (LSPR) methodology, designed to provide relative processor capacity data for IBM System/370, System/390 and z/Architecture processors. All LSPR data is based on a set of measured benchmarks and analysis, covering a variety of System Control Program (SCP) and workload environments. LSPR data is intended to be used to estimate the capacity expectation for a production workload when considering a move to a new processor. Although LSPR data is provided on an “as is” basis, with no warranty, it at least provides the Mainframe Capacity Planner with some insight into their CPU sizing challenge. For many years, LSPR provided the only other data source, as well as RMF (CMF) for Mainframe CPU sizing. However, is there a more accurate data source, perhaps based on real-life customer data?

With the introduction of the IBM System z10 server (February 2008), a new function CPU MF (CPU Measurement Facility) was incorporated. Let’s not forget, z10 is now an n-2 technology, having been superseded by the z196/z114 and the latest zBC12/zEC12 generation of servers. So each and every committed Mainframe customer should be positioned to benefit from the CPU MF function.

CPU MF provides optional hardware assisted collections of information about logical CPU activity executed over a specified interval in selected Logical Partitions (LPARs). The CPU MF counters function is intended to be run on a constant basis to collect long-term performance data (I.E. SMF Record 113), in a similar manner to how you collect other performance data. Therefore this data source can be deployed to further refine the accuracy of Mainframe CPU capacity planning projections. Let’s not forget:

The primary on-going requirement for Mainframe Capacity Planning is to minimize any over or under capacity provision from forecast predictions, used for Mainframe server acquisition purposes”

Mainframe chip technology has also changed in complexity, especially with the latest iterations of CPU chips associated with the z10 server (E.g. POWER 6) onwards, incorporating many layers of cache memory. Workload capacity performance will be quite sensitive to how deep into the memory hierarchy the processor must go to retrieve the workload’s instructions and data for execution. Best performance occurs when the instructions and data are found in the cache(s) nearest the processor so that little time is spent waiting prior to execution; as instructions and data must be retrieved from farther out in the hierarchy, the processor spends more time waiting for their arrival.

As workloads are moved between processors with different memory hierarchy designs, performance will vary as the average time to retrieve instructions and data from within the memory hierarchy will vary. Additionally, once on a processor this component will continue to vary significantly as the location of a workload’s instructions and data within the memory hierarchy is affected by many factors including; locality of reference, IO rate, competition from other resources (E.g. Applications, LPARs, et al), and so on…

The most performance sensitive area of the memory hierarchy is the activity to the memory nest, namely, the distribution of activity to the shared caches and memory. IBM introduced new terminology, namely Relative Nest Intensity (RNI), indicating the level of activity to this part of the memory hierarchy. Using data from CPU MF, the RNI of the workload running in an LPAR may be calculated. The higher the RNI, the deeper into the memory hierarchy the processor must go to retrieve the instructions and data for that workload.

Therefore the Mainframe Capacity Planner does have various data sources available to forecast how an existing or new workload might perform on an upgraded processor (CPC), further refining their CPU capacity requirement forecast. As always, the final stage in a Mainframe Capacity Planning process is to input the forecast data into the IBM Processor Capacity Reference (zPCR) tool, to determine the exact model and associated resource configuration options for their unique business workload mix.

To summarize, does your Mainframe Capacity Planning process incorporate all of these CPU sizing data sources, in an easy-to-use and cost efficient manner?

Founded by former IBM staffers and capacity planning and performance management industry veterans William Shelden, PhD, and William Hart, PerfTechPro is designed to deliver sophisticated, affordable, easy-to-use solutions for IT management professionals looking for fast, insightful help without high-cost, complex and time-consuming purchasing and licensing requirements.

PerfTechPro for z/OS is a Capacity Planning and Performance Measurement tool specifically designed for the cost conscious and savvy 21^st Century data centre. PerfTechPro for z/OS is the next evolution in Mainframe Capacity Planning tools, having been architected from ground zero using the latest techniques. PerfTechPro for z/OS provides sophisticated capacity and performance management capabilities, affordable by any sized data centre:

Clean, intuitive, easy-to-use interface and graphical representations, for example:
- Consolidated instance lists guide users to make informed selections
- Descriptive dialog boxes detail your configuration
- Anticipates, pre-loads data to speed retrieval, reporting and analysis
- Automated data management
Forecasting and modelling
Non-proprietary database, enabling data use outside of PerfTechPro
Capable of automated collection, analysis and reporting of SMF 113 records produced by the IBM CPU Measurement Facility (CPU MF)
Supports measurement, management of zAAP & zIIP Specialty Engines
Automated analysis and management of all key capacity and performance metrics, for example:
- GPP Utilization of All LPARs
- MIPS Usage by CPU
- DASD Response Times
- Address Spaces Dispatched and Waiting

PerfTechPro for z/OS also simplifies the data management process associated with Mainframe Capacity Planning. Using a streamlined process on the z/OS host, PerfTechPro extracts and formats the data required from various SMF sources (E.g. SMF Type 7n, Type 113); delivering an optimized Performance Data Base (PDB) for use by the Windows based GUI. This optimized file safeguards fast processing during the reporting and forecasting activities, while simplifying any data aggregation processes (E.g. Weekly, Monthly, et al). Moreover, PerfTechPro allows this data to be stored in non-proprietary (E.g. Microsoft Access, SQL Server, MySQL, Oracle) and multiple database structures, as and if required.

PerfTechPro for z/OS is a simple-to-use and cost-efficient solution, allowing customers to quickly save time and money from their Capacity Planning and Performance Measurement solution. Ultimately the bottom line objective for PerfTechPro for z/OS is to provide a best-of-breed solution for a very competitive cost. PerfTechPro for z/OS delivers business value by:

Ensuring enterprise zSeries Mainframe server resources are being used efficiently
Maximizing opportunities for cost-savings
Anticipating & responding to increased demand on resources
Reducing costs by exploiting periods of lower resource demand
Discerning underlying causes of performance and capacity issues
Eliminating time-consuming manual tracking, recording and analysis
Implementing disciplined management of valuable business resources

In conclusion, the Mainframe Capacity Planning process continues to evolve, forever striving to reduce any discrepancies in CPU requirements forecasting, which of course, have a high associated cost consideration. Integrating CPU MF (SMF Type 113) must be a mandatory requirement, safeguarding that CPU Sizing, Forecasting, Modelling and Correlation Analysis activities are optimized. Additionally, the actual process of Mainframe Capacity Planning is an activity that requires great skill and considerable associated responsibility. A modern day solution such as PerfTechPro for z/OS is worthy of consideration, having been designed by a team with a heritage in delivering Mainframe Capacity Planning solutions, architecting function compatible with modern day functionality, while considering the latest technology zSeries CPU chip design considerations.

A Tale of Two Twittees

Posted on 06/05/2013 by zman

It was the best of times, it was the worst of times, it was the age of wisdom, it was the age of foolishness, it was the epoch of belief, it was the epoch of incredulity, it was the season of Light, it was the season of Darkness, it was the spring of hope, it was the winter of despair, we had everything before us, we had nothing before us…

OK, so I’m not Charles Dickens, but recently I was reminded of this most notable of opening lines from a novel, these incredibly wise words, seemingly timeless, when assisting a recruitment consultant fulfil Mainframe technical positions for one of their clients.

So who are the Two Twittees? As my social networking is limited to LinkedIn, I’m not sure I understand all the terminology, but my analogy is based upon the inexperienced young entry level recruit to the Mainframe world, and the seasoned Mainframe professional with several decades of real-life vocational experience. Can and should these vastly different class of technical resources work together in today’s technology driven and seemingly endless social media resource aware world?

The simple answer was forever thus, yes, they must work together, but when we reflect upon that opening line from Charles Dickens, the gap between the young fresher and the seasoned sage perhaps has never been wider. Why? Perhaps largely attributable to the question of “common sense”, where perhaps the more experienced sage doesn’t answer questions instantaneously, doesn’t publish all sorts of information on social media sites various, and is used to being a problem solver, and when their career started, perhaps didn’t have instantaneous access to a wealth of electronically stored information (E.g. Technical Manuals, ISV/IHV Problem/Solution Resources, Forums, Blogs, et al). Conversely the inexperienced fresher has access to all of this information, without having done the job first, and so maybe they can be inclined to think that they can be an expert in weeks and months. This clearly is not the reality, regardless of IT platform, including the IBM Mainframe.

Recently I observed a quote from such a person posted on LinkedIn, something along the lines of “paper manuals, I heard that they existed, I always thought it was an urban myth”! Flippant or tongue in cheek maybe, but that’s the way we worked in the 1980’s, paper based, evolving slowly into IBM BookManager resources, initially via PC and CD-ROM! Additionally, you can witness numerous examples of potential job candidates asking how to prepare for an interview via social networking sites; what crib sheets and FAQ type resources can assist them in gaining a job! Surely, if you don’t know your subject matter, you perhaps shouldn’t be applying for the job? However, OK, secure an interview, but “tell the world” that you don’t know your subject matter? Surely, a modicum of common sense will tell you that this is not the way to inspire confidence in a potential employer, and thus perhaps you’re the one that’s impacting your employment chances, whether as an employee or sub-contractor…

It seems somewhat of a paradox that at 50 years old, most Mainframe people are considered to be too old, maybe being offered early retirement, or perhaps not being considered for any new job positions, because of their age. One must draw one’s own conclusions accordingly. Conversely, the younger and eager, perhaps recently graduated student, will be perceived as the future for a potential employer, but they have no experience. Additionally, such a person might have inadvertently or otherwise harmed their reputation by what they might or might not have posted on social networking sites such as Twitter, Facebook, LinkedIn, et al. It is somewhat bemusing to this social networking luddite, why any individual, regardless of age, can’t comprehend that recruitment consultants, company Human Resource departments and largely anybody involved in the recruitment/employment process will perform an Internet and social media search to determine the suitability of a candidate.

In this instance, the experience and perhaps inability of a fat fingered mature Mainframe person, who perhaps can’t or won’t tweet, text, connect, uses their common sense, does their job, and in fact their experience, both as a human being and a Mainframe technician, is best deployed by passing on these attributes to the new Mainframe fresher’s. Does this always happen? Once again, ad nauseam, if the experienced Mainframe resource is dismissed or overlooked at the age of 50 or older, how can they pass on such experience?

Shortly and on 16^th May 2013, the UK GSE 101 working group is having its inaugural working group meeting at the University of Bedfordshire, Luton. The UK GSE 101 working group is a new group aimed at those new to system z, discussing a wide range of topics for those new to the environment. What a fantastic stake in the ground for the IBM Mainframe newbie/fresher to meet with peers and industry colleagues, hopefully both young and old. I wish them well and hope to see this group go from strength to strength over the years.

So, back to my recent activity with the recruitment consultant! Their thought processes and tick boxes, pseudo or otherwise for vetting people must be recognised. Whatever age you are, be really careful how you use social networking sites, but this seemingly is more pertinent for the younger person. The older person, and seemingly being 50 or over, does have a major challenge in the employment world, but perhaps their best opportunity is being allowed to transfer their knowledge to newer Mainframe recruits. Therefore, perhaps they have to evolve, participate in movements such as the UK GSE 101 working group, but ideally, employers and those folks in the recruitment industry might look to work together and consider leveraging from the experience of more mature IT personnel, expediting the training of Mainframe fresher’s, but more importantly, transfer their “common sense” aptitude…

So even in 1859, Charles Dickens knew exactly what he was talking about, two sides to the coin, yin and yang, and so on. For some it can be the best of times, and for some it could be the worst of times, but by combining youth with experience, perhaps these current times can be good for all age demographics in the Mainframe workplace, where everybody wins, and yes that does include the Employer!

FICON (Fibre Connection Channel): 15 Years of Mainframe I/O Improvements

Posted on 09/04/2013 by zman

In 1998, IBM introduced FICON channels for enhanced I/O connectivity and performance for their 9672 G5 processors, delivering significant capability when compared to its predecessor, ESCON. Let’s not forget that ESCON (Enterprise Systems – S/390) was the first iteration of Fibre Channel for the IBM Mainframe, delivering significant capability, when compared with the previous technology of heavy, large and costly copper based bus & tag parallel (S/370) channels.

ESCON channels were first introduced in the early 1990’s, but after less than a decade, the data and associated storage device explosion was exposing the technical capabilities of ESCON, for example:

Mainframe Server Channel Support: One IBM Mainframe processor could only support 256 ESCON channels, whereas FICON was offering a ~5-8:1 reduction in channel requirements. Put another way, a customer could expect to consolidate the number of channels required from ~200 ESCON to ~30-40 FICON.
Device Support: One ESCON channel could support up to 1024 devices (sub-channel/device numbers), channel, whereas a 9672 FICON channel increased support by 16 fold, up to 16,384 (16 K) devices.
Distance: The performance of ESCON dropped off significantly when the distance between the channel and associated Control Unit was greater than ~9 KM. FICON increased this distance separation to ~100 KM, paving the way for the Geographically Dispersed Parallel Sysplex (GDPS) topologies we take for granted today.
Performance: ESCON performance was limited to 17 MB/S, whereas the first evolution of FICON channels delivered 100 MB/S full-duplex performance.

Clearly the first iteration of FICON technology delivered significant benefit to the IBM Mainframe User, and arguably is the primary Mainframe evolution that has sustained data growth and the adoption of Disaster Recovery and Business Continuity resiliency. So, what does FICON offer today, some 15 years later?

Just as FICON superseded ESCON, FICON Express has now superseded FICON, offering a technology base that can continue to deliver benefit for many years to come. FICON Express continues the tradition of offering more capabilities with each new generation of FICON channel. The features were designed with the future in mind, while remembering the past, by supporting the data serving leadership of System z and enabling improved data access using High Performance functions (I.E. zHPF), while providing backwards compatibility, being able to auto-negotiate the link data rates of 2, 4 or 8 Gbps, namely the various FICON Expressn iterations (2/4/8).

High Performance FICON for System z (zHPF) is a data transfer protocol that is optionally deployed for accessing data from IBM Mainframe compatible storage subsystems (E.g. IBM DS8000, EMC Symmetrix V-Max, HDS USP, et al) and other subsystems. Initially the data types supported were DB2, PDSE, VSAM, zFS and Extended Format SAM, and more latterly, legacy access methods including QSAM, BPAM and BSAM are now supported. zHPF leverages the potential of FICON channels to deliver significant performance enhancements, and can help reduce the infrastructure costs for System z I/O by efficiently utilizing I/O resources, minimizing CHPID (Channels), Fiber (Cables), Switch Ports (E.g. Cisco, Brocade) and Control Unit (E.g. Disk Subsystem) resource requirements. zHPF also compliments the Extended Address Volumes (EAV) strategy for growth, increasing I/O rate capability as the associated disk volume size increases.

The latest generation FICON Express8S channel has two possible modes of operation designed for connectivity to servers, switches/directors, disks, tapes and printers:

CHPID Type FC: FICON, zHPF, and channel-to-channel (CTC) traffic for the z/OS, z/VM, z/VSE, z/TPF, and Linux on System z environments
CHPID Type FCP: Fibre Channel Protocol (FCP) for attachment to SCSI devices for the z/VM, z/VSE, and Linux on System z environments

With FCP channel full fabric support, multiple switches/directors can be placed between the System z server and SCSI device, allowing many “hops” through a storage area network (SAN) and providing improved utilization of intersite-connected resources and infrastructure. This may help to provide more choices for storage solutions or the ability to use existing storage devices and can help facilitate the consolidation of Distributed Systems servers (E.g. UNIX, Wintel) onto System z servers, protecting investments in SCSI-based storage.

I/O performance improvement rates for the initial iterations of FICON when compared to ESCON and then FICON Express when compared to FICON, and more latterly zHPF have been significant. Using like-for-like benchmark performance studies, we can see significant performance improvements:

I/O Driver @ 4K Block Size – ~ I/Os Per Second

*Channel Type*	*#I/Os per Sec*	*n:1 Increase*
ESCON	1200	N/A
FICON Express 2/4 Native	14000	11.7
FICON Express 2/4 zHPF	31000	2.3
FICON Express 8 Native	20000	1.5
FICON Express 8 zHPF	52000	2.6
FICON Express 8S Native	23000	1.2
FICON Express 8S zHPF	92000	1.8

NB. Maximum performance is server related (E.g. z10, z114, z196, zEC2).

Compared to ESCON, the latest 8 Gbps FICON channel leveraging from zHPF function delivers ~76 times more I/O throughput compared to ESCON, while significantly increasing throughput, by at least 50% from generation to generation.

I/O Driver Mixed Read/Write – ~ MBs Per Second

*Channel Type*	*#MBs per Sec*	*n:1 Increase*
ESCON	12	N/A
FICON Express 4 Native	350	29.2
FICON Express 4 zHPF	620	1.8
FICON Express 8 Native	620	1.8
FICON Express 8 zHPF	770	1.3
FICON Express 8S Native	620	1.0
FICON Express 8S zHPF	1600	2.1

NB. Maximum performance is server related (E.g. z10, z114, z196, zEC2).

Compared to ESCON, the latest 8 Gbps FICON channel leveraging from zHPF function delivers ~133 times more I/O throughput compared to ESCON, while significantly increasing throughput, by at least 100% from generation to generation.

Once again, the backwards compatibility capability of the IBM Mainframe server is highlighted by the evolution of the FICON channel, and in particular, Disk Subsystems IHV’s, obviously IBM themselves, but notably EMC, HDS and Oracle (StorageTek) in evolving their offering to support the latest FICON technologies.

We sometimes might take for granted how much data can be stored by a single footprint IBM Mainframe and how much performance and throughput capability is available to process this data. However, we shouldn’t under estimate what role FICON has played in allowing this significant data (I/O) processing capability to grow, often rapidly, sometimes exponentially.

If there is a downside, such performance attributes might have eradicated the skills required to tune I/O subsystems, but that’s perhaps a subject matter for another day…

IBM Mainframe: Workload License Charges (WLC) Pros & Cons

Posted on 07/03/2013 by zman

It is estimated that less than half of eligible IBM Mainframe customers deploy the VWLC pricing mechanism, which in theory, is the lowest cost IBM software pricing metric. Why? In the first instance, let’s review the terminology…

Workload License Charges (WLC) is a monthly software license pricing metric applicable to IBM System z servers running z/OS or z/TPF in z/Architecture (64-bit) mode. The fundamental ethos of WLC is a “pay for what you use” mechanism, allowing a lower cost of incremental growth and the potential to manage software cost by managing associated workload utilization.

WLC charges are either VWLC (Variable) or FWLC (Flat). Not all IBM Mainframe software products are classified as VWLC eligible, but the major software is, including z/OS, CICS, DB2, IMS and WebSphere MQ, where these products are the most expensive, per MSU. What IBM consider to be legacy products, are classified as FWLC. More recently a modification to the VWLC mechanism was announced, namely AWLC (Advanced), strictly aligned with the latest generation of zSeries servers, namely zEC12, z196 and z114. For the smaller user, the EWLC (Entry) mechanism applies, where AEWLC would apply for the z114 server. There is a granular cost structure based on MSU (CPU) capacity that applies to VWLC and associated pricing mechanisms:

*Band*	*MSU Range*
Base	0-3 MSU
Level 0	4-45 MSU
Level 1	46-175 MSU
Level 2	176-315 MSU
Level 3	316-575 MSU
Level 4	576-875 MSU
Level 5	876-1315 MSU
Level 6	1316-1975 MSU
Level 7	1976+ MSU

Put simply, as the MSU band increases, the related cost per MSU decreases.

IBM Mainframe users can further implement cost control by specifying how much MSU resource they use by deploying Sub-Capacity and Soft Capping techniques. Defined Capacity (DC) allows the sizing of an LPAR in MSU, and so said LPAR will not exceed this MSU amount. Group Capacity Limit (GCL) extends the Defined Capacity principle for a single LPAR to a group of LPARs, and so allowing MSU resource to be shared accordingly. A potential downside of GCL is that is one LPAR of the group can consume all available MSU due to a rogue transaction (E.g. loop).

Sub-Capacity software charges are based upon LPAR hardware utilization, where the product runs, measured in hourly intervals. To smooth out isolated usage peaks, a Rolling 4-Hour Average (R4HA) is calculated for each LPAR combination, and so software charges are based on the Monthly R4HA peak of appropriate LPAR combinations (I.E. where the software product runs) and not based on individual product measurement.

Once a Defined Capacity LPAR is deployed, this informs WLM (Workload Manager) to monitor the R4HA utilization of that LPAR. If the LPAR R4HA utilization is less than the Defined Capacity, nothing happens. If the LPAR R4HA utilization exceeds the Defined Capacity, then WLM signals to PR/SM and requests that Soft Capping be initiated, constraining the LPAR workload to the Defined Capacity level.

If a user chooses a Sub-Capacity WLC pricing mechanism, they will be required by IBM to submit a monthly Sub-Capacity Reporting Tool (SCRT) report. Monthly WLC invoices are based upon hourly utilization metrics of LPAR hardware utilization, where the software product executes. The cumulative R4HA and bottom line WLC billing metric is calculated for each product and associated LPAR group and not based on individual product measurement.

Bottom Line: From a Soft Capping viewpoint, the customer only pays for WLC software based upon the Defined Capacity (DC) or Rolling 4-Hour Average (R4HA), whichever is the lowest. So whether a customer uses Soft Capping or not, in all likelihood, there will be occasions when their workload R4HA is lower than their zSeries server MSU capacity.

So, at first glance, VWLC seems to provide a compelling pricing metric, based upon Sub-Capacity and a pay for what you use ethos, and so why wouldn’t an IBM Mainframe user deploy this pricing metric?

The IBM Planning for Sub-Capacity Pricing (SA22-7999-0n) manual states “For IBM System z10 BC and System z9 BC environments, and z890 servers, EWLC pricing is the default for z/OS systems, and Sub-Capacity pricing is always the best option. For IBM zEnterprise 114, environments, AEWLC pricing is the default for z/OS systems, and Sub-Capacity pricing is always the best option. For IBM zEnterprise 196, System z10 EC and System z9 EC environments, and other zSeries servers, Sub-Capacity pricing is cost-effective for many, but not all, customers. You might even find that Sub-Capacity pricing is cost effective for some of your CPCs, but not others (although if you want pricing aggregation, you must always use the same pricing for all the CPCs in the same sysplex)”.

Conclusion: For all small Mainframe users qualifying for the EWLC (AEWLC) pricing metric, arguably this pricing mechanism is mandatory. For the majority of larger Mainframe users, the same applies, although a granularity of adoption might be required. IBM also have a disclaimer “Once you decide to use Sub-Capacity pricing for a specific operating system family, you cannot return to the alternative pricing methods for that operating system family on that CPC. For example, once you select WLC you may not switch back to PSLC without prior IBM approval”. However, the requisite contractual exit clause option does exist; the customer can switch back to the PSLC pricing metric.

Some IBM Mainframe users might object to a notion of Soft Capping, relying upon their tried and tested methodology of LPAR management via the number of CPs allocated and associated PR/SM Weight. This is seemingly a valid notion and requirement, prioritizing performance ahead of cost optimization.

Conclusion: As previously indicated, with VWLC, SCRT invoices are generated upon a premise of the customer only pays for WLC software based upon the Defined Capacity (DC) or Rolling 4-Hour Average (R4HA), whichever is the lowest. So the VWLC pricing mechanism should deliver a granularity of cost savings, typically higher for a Soft Capping environment.

Some IBM Mainframe users might just believe that nothing can match their Parallel Sysplex Licensing Charge (PSLC) mechanism, first available in the late 1990’s, which might be attributable to other 3^rd party ISV’s who cannot and will not allow for their software to be priced on a Sub-Capacity basis. In reality, adopting the VWLC pricing mechanism delivers ~5% cost savings when compared with PSLC, as indicated by the IBM Planning for Sub-Capacity Pricing Manual (SA22-7999-0n) and related Sub-Capacity Planning Tool (SCPT).

Conclusion: Adopting Sub-Capacity based pricing metrics can only be a good thing. If your 3^rd party ISV supplier doesn’t recognise Sub-Capacity pricing, whether MIPS or MSU based, perhaps you should consider your relationship with them. Regardless, the z10 server was the last IBM Mainframe to incorporate the “Technology Dividend” solely based on faster CPU chips. The lower cost WLC pricing metric is now only available with the AWLC and related (E.g. AEWLC) pricing metrics, as per the z196, z114 and zEC12 servers.

Some customers might state that there is a lack of function or granularity of policy definition for IBM supplied Soft Capping (E.g. DC, GCL) or Workload Management (WLM) techniques. To some extent this is a valid argument, but wasn’t it forever thus with IBM function? Sub-Capacity implementation is possible via IBM, as is Workload Management (WLM), Soft Capping or not, but should the customer require extra functionality, 3^rd party software solutions are available.

The zDynaCap software solution from zIT Consulting delivers a “Capacity Balancing” mechanism, integrating with R4HA and WLM methodologies, but constantly monitoring MSU usage to determine whether CPU resource can be reallocated to Mission & Time Critical workloads, based upon granular customer policies. The only guarantee in a multiple LPAR environment, for a Mission & Time Critical LPAR to receive all available MSU resource, Soft Capping or not, is to inactivate all other LPARs! Clearly this is not an acceptable policy for any installation, and so a best endeavours policy applies for PR/SM DC, GCL and Weight settings.

Conclusion: z/OS workloads change constantly, whether the time of day (E.g. On-Line, Batch) or period of the year (E.g. Weekly, Monthly, Quarterly, Yearly) or just by customer demand (E.g. 24 Hour Transaction Application). Therefore a dynamic MSU management solution such as zDynaCap is arguably mandatory, implementing the optimum MSU management policy, whether for purely performance reasons, safeguarding the Mission & Time Critical workload isn’t impacted by lower priority workloads, or for cost reasons, optimizing MSU usage for the best possible monthly WLC cost.

In conclusion, not considering and arguably not implementing z/OS VWLC related pricing mechanisms is impractical, because:

The VWLC and AWLC related pricing metrics deliver the lowest cost per MSU for eligible z/OS software
When compared with PSLC, VWLC related pricing mechanisms deliver conservative ~5% cost savings
A pay for what you use and therefore Sub-Capacity pricing mechanism, not the installed MSU capacity
If extra MSU policy management granularity is required, consider 3^rd party software such as zDynaCap

Software cost savings are not just for the privileged; they’re for everyone!

IBM Mainframe – Enterprise Software License Agreements Pros & Cons

Posted on 05/02/2013 by zman

An often quoted phrase in the Mainframe user base is “why are our Mainframe software costs so high”? Sometimes we might have to look closer to home when finding the answers to our questions…

Over the years, Mainframe software portfolios in the customer environment might have become unwieldy, with duplication of software function, unused software, unsupported software products, and so on. Typically this scenario occurs due to Merger & Acquisition (M&A) activity, where in an ideal world, a standard LPAR (image) with an optimally configured software portfolio would be deployed, which inevitably will generate the requirement for a modicum of migration activity, from one software product to another. The complexity of software migration can change dramatically from a simple change, generally associated with Systems Management (E.g. Monitors) products to enormously complex, generally involving Database Subsystems (E.g. Adabas, DB2, IDMS, et al) and Programming Languages (E.g. COBOL, PLI, et al) while there is some middle ground with some Systems Management products (E.g. Security, Storage, Scheduling) that maintain metadata (policy data). Therefore only the truly committed Mainframe user will adopt and fully commit to this standard LPAR methodology, benefitting to some extent from lower software costs.

Similarly over the last 20 years or so, the perceived requirement for Enterprise Software License Agreements has increased, where the fundamental premise is that such agreements make life easier for both the customer and ISV alike. An interesting notion indeed, and one must draw one’s own conclusions as to whether such a utopia can exist; therefore as always, the caveat emptor (let the buyer beware) term must apply!

However, with such fully encompassing requirements and associated pricing mechanisms, the need for each and every major ISV to have a fully rounded software portfolio has ensued. Therefore we have witnessed a lot of M&A activity in the Mainframe ISV market place, where several dominant players have emerged, in no particular order, BMC (Advantage), CA (FlexSelect, MLP, OLP) and IBM (ESSO, ELA), while some might say ASG should be included in this list. Generally it seems to be the norm that each and every Mainframe customer will have at least one Enterprise Software License Agreement in place, typically with IBM because of the need to deploy the z/OS (z/VM, z/VSE, zLinux) operating system, generally in conjunction one other, whether ASG, BMC or CA.

The advantages of an Enterprise Software License Agreement are primarily:

Simplified license management via many products from one supplier
A several (3-5) year license agreement, only requiring periodic review and negotiation
Perceived cost benefit, with discount based upon volume, both in terms of software and CPU power
Perceived deployment benefit, treating Distributed and Mainframe platforms equally
Simplified support, as each and every software product should have the same look and feel

However, for a balanced review, we must identify the potential disadvantages, for example:

Is each and every software product from this single supplier the best for our business?
How do we renegotiate this agreement, because our business requirements have unexpectedly changed?
How do we exit this agreement, because our relationship with this supplier has failed?
How do we calculate a tangible cost and value for each and every product we deploy?

As always, the devil is in the detail, and although most pros and cons seem fairly innocuous at first glance, the considerations generated regarding contract termination or renegotiation are significant. For example, if the Mainframe user chooses a 3 year Enterprise Software License Agreement, do they need to decide at least 18 Months before contract expiration that they must migrate to alternative software products, to terminate their relationship with a supplier? So at first glance, volume discount and simplification look good, but how expensive and disruptive will contract termination be?

In real-life human terms, this is somewhat analogous to Marriage, a long-term relationship between two parties that choose to declare significant commitment to one another, but perhaps, the realm of possibility exists that said relationship will fail, and of course, in the absence of a bulletproof pre-nuptial, complications occur, and exit from the relationship is both financially expensive and disruptive. Hmmm, so where is the equivalent of a pre-nuptial for the Enterprise Software License Agreement? In an ideal world, the commercially savvy customer will have planned for such a possibility, but whether they have or have not, the supplier will have been paid for their software, and the customer may not have any choice but to renew or extend their agreement! So which party is the winner and which one is the loser in such a scenario? Does one party benefit from a heads we win and tails you lose proposition?

How does the Mainframe customer choose the best software product for their business requirement? In an ideal world, they document their business requirement, collect information on market place offerings, review pricing options, generate a shortlist of suitable products, and eventually choose the “best-of-breed” product. How is such a structured and balanced approach possible when deploying the Enterprise Software License Agreement? The first thought must be cost based, as software has already been paid for, so if there’s a product in the portfolio we could use, we need to use it, whether it’s the best product or not.

If a Mainframe user is using an internal chargeback system for computing use, how can they fairly cost the pricing metric, if they don’t know the price of software products used? Equally, how can the Mainframe user attempt to identify single product pricing when Enterprise Software License Agreements detail no granularity of pricing information? Perhaps a modicum of research might help, where some global Government regulations dictate that contract details must be published for public scrutiny. Therefore ISV Mainframe software list pricing details can be identified, for example, IBM and BMC.

One must draw one’s own conclusions, where some Mainframe customers may perform a structured review of the market place, and even though the technical recommendation might be for a product not covered by an Enterprise Software License Agreement, typically from a smaller ISV, the product chosen is one already paid for, or at least available from the Enterprise Software License Agreement. This generates several issues, including but not limited to, alienating the smaller ISV community, having used them for expediency, and not delivering the best solution for your business…

So does the self-fulfilling prophecy ensue, where the Mainframe customer questions the cost of Mainframe software, but perhaps implicitly or unknowingly, said Mainframe user has contributed to such an environment, where a limited number Mainframe ISV’s control the Mainframe software market?

Isn’t it somewhat of a paradox that in The UK, the monopolies commission would review the merits of an M&A between two major grocery supermarket or energy supplier companies, and yet whether in The UK or globally, there are several major ISV’s (E.g. ASG, BMC, CA, IBM) dominating the Mainframe software market, primarily via Enterprise Software License Agreements? Can this really be a good thing for the Mainframe user, limited supplier choice and therefore a lack of healthy competition?

Perhaps it is the responsibility of the Mainframe user to actually choose software impartially, and from time-to-time choose the best product, regardless of ISV. This might generate a more active market place for software choice, while it was forever thus, the larger ISV is so big that they can easily acquire the smaller ISV who has developed and sold a good product, but at least the Mainframe ISV market place continues to evolve. In this case, it seems somewhat logical that the Mainframe user is in control of their destiny, but only by safeguarding that their default option is not the Enterprise Software License Agreement. They encourage an active and impartial ISV software market by dispassionately reviewing the open market and choosing the best Mainframe software product for their business!

Lewis Carroll once said “integrity is doing the right thing, even when no one is watching”! When was the last time a major ISV declared an open book policy for your business, offering you flexible options to benefit from their Enterprise Software License Agreement, while allowing you to choose a best-of-breed software product, but not from their software portfolio, giving you a discount (credit note) for their software product that didn’t match your business requirement?

Value-4IT Blog

zWorld Thoughts & Updates

Tag Archives: Mainframe