Simplifying Db2 for z/OS CPU Optimization: Eradicating Inefficient SQL Processing

Without doubt the IBM Z Mainframe server is recognised as the de facto choice for storing mission critical System of record (SOR) data in database repositories for 92 of the top 100 global banks, 23 of the 25 top global airlines; the top 10 global insurers & ~70% of all Fortune 500 companies. ~80% of mission critical data is hosted by IBM Z Mainframe servers, processing 30+ Billion transactions per day, including ~90% of all credit card transactions. This data is accessed by ~1.3 Million CICS transactions per second, compared with a Google (mostly search) processing rate of ~70,000 transactions per second. Interestingly enough, despite processing so many mission critical transactions the IBM Z Mainframe server platform is only accountable for ~6.2% of global IT spend. One must draw one’s own conclusions as to why some IT professionals perceive the IBM Z Mainframe server as being a legacy platform, not worthy of consideration as a strategic IT server platform…

The digital transformation has delivered an exponential growth of data, typically classified as Cloud, Mobile & Social based. This current & ever-growing data source requires intelligent analytics to deliver meaningful business decisions, requiring agile application software delivery to gain competitive edge. This digital approach can sometimes deliver a myriad of micro business application changes, personalised for each & every customer, often delivering “pop-up” applications…

IBM Z Mainframe software costs are often criticized as being a major barrier to maintaining or indeed commissioning the platform. IBM have tried to minimize these costs with numerous sub-capacity pricing options over the last 30 years or so, but this is perceived by many as being overly complicated; although with a modicum of knowledge, a specialized personnel resource can easily control software costs. All that said, IBM have introduced Tailored Fit Pricing for IBM Z, in an attempt to simplify software cost management. A recent blog reviewed the Tailored Fit Pricing for IBM Z offering & whether you decide whether this IBM Z pricing mechanism is suitable for your organization, optimizing IBM Z CPU MSU/MIPS usage is mandatory. Recognizing that the IBM Z Mainframe server is the de facto database server for System of Record data, primarily via the Db2 subsystem, clearly optimizing Db2 CPU usage, whether OLTP transactions, typically via CICS, or the batch window, has been & always will be, worthwhile…

All too often, many IT disciplines can be classified with a generic 80/20 rule & typically data can be classified accordingly, where 80% of data is accessed 20% of the time & 20% of data is accessed 80% of the time. The challenge with such a blunt Rule of Thumb (ROT) is that it’s static, but it’s a good starting point. Ideally for any large data source, there would be a dynamic sampling mechanism that would identify the most active data, loading this into the highest speed memory resource to reduce I/O access times & therefore CPU usage. Dynamic management of such a data buffer would render the 80/20 rule extraneous to requirements, as each & every business has their own data access profile. However, a simple cost benefit & therefore Proof of Value (POV) analysis could ensue.

From a Db2 viewpoint, pre-defined structures such as buffer pools offer some relief in storing highly referenced data in a high-speed server memory resource, but this has a finite capacity versus performance benefit, not necessarily using the fastest memory structures available nor dynamically caching the most accessed data. The business considerations of not optimizing Db2 data access are:

  • Elongated Batch Processing: With ever increasing amounts of data to process & greater demands for 247365 availability & real-time access, data access optimization is fundamental for optimized service delivery, often measured by mission critical SLA & KPI metrics. Optimized batch processing is a fundamental requirement for acceptable customer facing business service delivery.
  • Slow Transaction Response Times: As the nature of customer requirements change, mobile device applications exponentially increasing the number of daily transactions, overall system resource capacity constraints are often stressed during peak hours. Optimized transaction response time is a fundamental requirement, being the most transparent service delivered to each & every end customer.

An easy but very expensive solution to remediate batch processing & transaction response issues is to provide more resources via a CPU server upgrade activity. A more sensible approach is to optimize the currently deployed resources, safeguarding that frequently accessed data is mostly if not always high speed cache resident, reducing the I/O processing overhead, reducing CPU usage, which in turn will optimize batch processing & transaction response times, while controlling associated IBM Z Mainframe server hardware & software costs.

The ubiquitous Db2 data access method is Structured Query Language (SQL) based, where IBM has their own implementation, SQL for Db2 for z/OS, which could be via the commonly used COBOL (EXEC SQL) programming language or a Db2 Connect API (E.g. ADO.NET, CLI, Embedded SQL, JDBC, ODBC, OLE DB, Perl, PHP, pureQuery, Python, Ruby, SQLJ). For Db2 Connect, there are 2 types of embedded SQL processing, static & dynamic SQL. Static SQL minimizes execution time by processing in advance. Though some relief is provided by Dynamic Statement Cache, dynamic SQL is processed when the SQL statement is submitted to the IBM Z Db2 server. Dynamic SQL is more flexible, but potentially slower. The decision to use static or dynamic SQL is typically made by the application programmer. There is a danger that Dynamic Statement Cache might be considered as a panacea for SQL CPU performance optimization, but as per any other performance activity, reviewing any historical changes is a good idea. The realm of possibility exists for the Db2 Subject Matter Expert (SME) to be pleasantly surprised that more often than not, there are still significant SQL CPU optimization opportunities…

From a generic Db2 viewpoint, with static SQL, you cannot change the form of SQL statements unless you make changes to the program. However, you can increase the flexibility of static statements by using host variables. Obviously, application program changes are not always desirable.

Dynamic SQL provides flexibility, if an application program needs to process many data types & structures, dictating that the program cannot define a model for each one, dynamic SQL overcomes this challenge. Dynamic SQL processing is facilitated by Query Management Facility (QMF), SQL Processing Using File Input (SPUFI) or the UNIX Systems Services (USS) Command Line Processor (CLP). Not all SQL statements are supported when using dynamic SQL. A Db2 application program that processes dynamic SQL accepts as input, or generates, an SQL statement in the form of a character string. Programming is simplified when you can structure programs not to use SELECT statements, or to use only those that return a known number of values of known types.

For Db2 data access, SQL statement processing requires an access path. The major SQL statement performance factors to consider are the amount of time that Db2 uses to determine the access path at run time & whether the access path is efficient. Db2 determines the SQL statement access path either when you bind the plan or package that contains the SQL statement or when the SQL statement executes. The repeating cost of preparing a dynamic SQL statement can make the performance worse when compared with static SQL statements. However, if you execute the same SQL statement often, using the dynamic SQL statement cache decreases the number of times dynamic statements must be prepared.

Typically, organizations have embraced static SQL over dynamic because static is more predictable, showing little or no change, while dynamic implies ever changing & unpredictable. Db2 performance optimization functions have been incorporated into base Db2 (E.g. Buffer Pools) & software products (E.g. IBM Db2 AI for z/OS, IBM Db2 for z/OS Optimizer, IBM Db2 Analytics Accelerator, IBM Z Table Accelerator, IZTA), with varying levels of benefit & cost. Ultimately IBM Z Mainframe customers need simple cost-efficient off-the-shelf solutions of a plug & play variety & without doubt, optimizing static SQL data processing is a pragmatic option for reducing Db2 subsystem CPU usage.

In Db2 Version 10, support for 64-bit run time was introduced, providing Virtual Storage Constraint Relief (VSCR), improving the vertical scalability of Db2 subsystems. With Db2 Version 11, the key z/Architecture benefit of 64-bit virtual addressing support was finally introduced, increasing capacity of central memory & virtual address spaces from 2 GB to 16 EB (Exabytes), eliminating most storage constraints. It therefore follows that any Db2 CPU performance optimization solution should also exploit the z/Architecture 64-bit feature, to support the ever-increasing data storage requirements of today’s digital workloads.

As we have identified, Db2 can consume significant amounts of z/OS CPU accessing & retrieving the same static frequently used data elements repetitively. Upon analysis, these static frequently used data elements are typically identified originating from a small percentage of Db2 tablespaces. Typically, at first glance these simple SQL programs are considered as low risk, but are repeatedly processed, often in peak processing times, consuming excessive CPU & increasing processing cost accordingly, typically z/OS Monthly Licence Charges (MLC) related. Db2 optimization tools for access path or buffer pool management provide some benefit, but this is not always significant & may require application changes. Patently there is a clear & present requirement for a simple plug & play solution, transparent to Db2 processing, maintaining an optimized high-performance in-memory cache of frequently used Db2 data, safeguarding data integrity in environments various, including SYSPLEX, Data Sharing, et al…

QuickSelect is a plug-in solution dynamically activated in a batch or OLTP environment (I.E. CICS, IMS/TM) intercepting repetitive SQL statements from Db2 application programs, storing the most active result set, not necessarily the entire tablespace, in a high-performance in-memory cache, returning to applications the same result set as per Db2, but much faster & using less CPU accordingly. QuickSelect is completely transparent to z/OS applications, eliminating any requirement to change/recompile/relink application source or rebind packages. QuickSelect processing can be switched on or off using a single keystroke, either defaulting to standard Db2 SQL processing or to benefit from the QuickSelect high-speed cache for optimized CPU resource usage.

The 64-bit QuickSelect server, implemented as a started task, intelligently caching data in self-managed memory above the bar, supporting up to 16 EB of memory, eliminating concerns of using any other commonly used storage areas (E.g. ECSA). The intelligent caching mechanism safeguards that only highly active data is retained, optimizing the associated cache memory size required.

QuickSelect caches frequently requested Db2 SQL result sets, returning these results to the application from QuickSelect cache, when a repetition of the same SQL is encountered. For data integrity purposes, QuickSelect immediately invalidates result sets upon detection of changes to underlying tables, implicitly validating each cache resident SQL result set. Changes to Db2 data by application programs are captured by a standard Db2 VALIDPROC process, attached to the typically small subset of frequently accessed tables of interest to QuickSelect. Db2 automatically activates the VALIDPROC routine whenever the table contents are changed by INSERT, DELETE, UPDATE or TRUNCATE statements, invalidating cached data from the updated tables automatically. For standard Db2 utilities such as LOAD/REPLACE, REORG/DISCARD & RECOVER, table-level changes are identified by a QuickSelect utility-trap, invalidating cached data from the updated tables automatically. QuickSelect also supports SYSPLEX & Data Sharing environments, supporting update activity via the same XCF functions & processes used by Db2.

QuickSelect delivers the following benefits:

  • CPU Savings: Meaningful reduction (E.g. 20%) in the Db2 SQL direct processing; 10%+ peak time CPU reduction is not uncommon.
  • Faster Processing: Optimized CPU usage delivers shorter batch processing & OLTP transaction response times, for related SLA & KPI objective compliance.
  • Transparent Implementation: No application changes required, source code, load module or Db2 package.
  • Survey Mode: Unobtrusive & minimal Db2 workload overhead data sampling to identify potential CPU savings from repetitive SQL & tables of interest, before implementation.
  • Staggered Deployment: Granular criteria (E.g. Job, Program, Table, Transaction, Etc.) implementation ability.
  • Reporting & Analytics: Extensive information detailing cache usage for Db2 programs & tables.

Since 1993 Db2 has evolved dramatically, in line with the evolution of the IBM Z Mainframe server. When considering today’s requirement for a digital world, processing ever increasing amounts of mission critical data, a base requirement to optimize CPU processing for Db2 SQL data access is mandatory. In a hybrid support environment where today’s IBM Z Mainframe support resource requires an even blend of technical & business skills, plug & play, easy-to-use & results driven solutions are required to optimize CPU usage, transparent to the subsystem & related application programs. QuickSelect is such a solution, fully exploiting 64-bit z/Architecture for ultimate scalability, identifying & resolving a common CPU consuming data access problem, for a mission critical resource, namely the Db2 subsystem, maintaining mission-critical System of Record data.

z/OS CPU optimization is a mandatory requirement for every organization, to reduce associated software & hardware costs & in theory, as a mandatory pre requisite for deploying the Tailored Fit Pricing for IBM Z pricing mechanism. Tailored Fit Pricing uses the previous 12 Months SCRT submissions to establish a baseline for MSU charging over a contracted period, typically 3 years. If there are any unused MSU resources, these are carried forward to the next year, but if those MSU resources remain unused at the end of the contracted period, they are lost, meaning the organization has paid too much. If the MSU resource exceeds the agreed Tailored Fit Pricing, excess MSU resources are charged at a discounted rate. Clearly achieving an optimal MSU baseline before embarking on a Tailored Fit Pricing contract is arguably mandatory & it therefore follows that optimizing CPU forever more, safeguards optimal z/OS MLC charging during the Tailored Fit Pricing contract. QuickSelect for Db2 is a seamless CPU optimization product that will perpetually deliver benefit, assisting organizations minimize their z/OS MLC costs, whether they continue to proactively manage the R4HA, submitting monthly SCRT reports or they embark on a Tailored Fit Pricing contract…

DB2: Internal Subsystem Security vs. External Security Manager (ESM)?

With the ever increasing requirement for regulatory compliance and the clear and present danger associated with cybersecurity attacks, isn’t now the best time to safeguard your organizations most important asset, namely business data?  Various industry analyst quotes state that ~80%+ of Mainframe data resides in databases and associated data sources and ~80%+ of global corporate data originates or resides on IBM Mainframes.  Depending on your viewpoint, rightly or wrongly DB2 is the most pervasive of database subsystems, offering two mechanisms for security, either internal subsystem or External Security Manager (ESM) based via ACF2, RACF or Top Secret.  When DB2 was first released in 1983, Mainframe security was in its infancy and perhaps even an afterthought, and so implementing internal DB2 security might have been the typical approach for many years.  Some several decades later, asking that age old rhetorical question; what is the best security solution for my mission critical and priceless data?  I’m not sure it is a rhetorical question, the answer is patently obvious, external security!

RACF and DB2 security integration was introduced in 1997 with OS/390 2.4 and DB2 Version 6 and so a ~14 year period where DB2 internal security was the only option!  Personally, ~20 years ago I was involved with an internal DB2 to RACF security migration project, part of a larger Operating System, DB2 and indeed CICS upgrade.  Basically the DB2 DBA team stated “we would have never implemented internal DB2 security if the RACF option was available; can you migrate to RACF for us”?  The simple reality being that Security Management is not a core DBA skill and such a process is ideally delivered by a Subject Matter Expert (SME).  Of course, DB2 was somewhat straightforward ~20 years ago, as were its security features, but in the last ~20 years, DB2 has become more complex and enterprise wide, while I’m often surprised by the number of organizations I encounter, both small and large, still deploying internal DB2 security…

Recognizing a ~20 year longevity period of RACF security support for DB2, maybe even the most conservative of organizations might concede that the technology is proven and works?  From a business viewpoint, such a migration from DB2 internal to an External Security Manager (ESM) is the proverbial “no brainer”, because:

  • Subject Matter Expert (SME): Clearly all IBM Mainframe organizations now have dedicated security professionals who are ideally placed to implement enterprise wide security policies. A DB2 DBA is a highly skilled SME in their own discipline, most likely welcoming the migration of security from DB2 to ACF2, RACF or Top Secret.
  • Compliance: A plethora of industry regulations, including but not limited to GLB, SOX, PCI-DSS, et al, dictate that a hybrid of technical skills and business policy knowledge is required. This has generated a requirement for the executive level CISO role and associated security certifications (E.g. CISA, CISM, CISSP) for SME resources.
  • Auditability: From a board level CxO viewpoint, which technical resource would you want responsible for your security policy, the CISO/CIO and their security engineers or a DB2 DBA?
  • Hacking-Penetration Testing: Rightly or wrongly, rightly in my opinion, Penetration (Pen) Testing is a methodology to try and hack a system in order to highlight security vulnerabilities, supplementing the traditional periodic audit processes. Once again, high levels of security expertise are required for such activities.

From a technical viewpoint, what is the complexity of performing a DB2 internal to RACF external security migration?

From a DB2 viewpoint, internal security rules are stored in DB2 catalog tables with the SYSIBM.SYSxxxAUTH naming convention.  Therefore these data repositories can be processed with a simplistic DB2 to RACF security migration tool (RACFDB2).  As per any migration activity, Garbage In, Garbage Out (GIGO) applies, and this golden rule dictates the requirement for a collaborative team effort to execute a DB2 to RACF security migration process.  Of course, the most important resources will be the DB2 DBA(s) responsible for maintaining DB2 security and a RACF SME.  Between them, these 2 resources have all of the skills required to perform this migration process, if not the experience.

From a documentation viewpoint, there are several resources that can be referenced to simplify this process:

The purpose of this blog post is a “call to action”, for those sites still deploying DB2 internal security, to migrate to their External Security Manager (ESM), whether ACF2, RACF or Top Secret.  There are also options for the migration of internal DB2 security to CA ACF2 and Top Secret respectively.

As previously stated, the DB2 DBA will be ideally placed to review the existing internal DB2 security environment, performing any clean-up and rationalization before the actual migration process.  The initial pass of the migration process will inevitably produce a one:one (1:1) mapping of rules, generating numerous security definitions extraneous to requirements.  This is where the ACF2, RACF or Top Secret SME can collaborate with their DB2 DBA, applying grouping, masking and generic filters to vastly reduce and simplify the number of security definitions required.  As with any migration, perform on the lowest level non-Production environment first, apply the lessons learned, and use common sense, issuing warning messages for inadvertent security policy errors, as opposed to denying security access for Production migrations!  Therefore allowing for the smooth transition from DB2 internal to ESM based security.

In my opinion, each and every IBM Mainframe organization has the ability to initiate this DB2 internal to external ACF2, RACF or Top Secret migration project.  Leveraging from 3rd party organizations also makes sense and in no particular order, other than alphabetical, I would suggest IBM Global Services, millennia, RSM Partners or Vanguard.

In conclusion, the IBM System z External Security Manager (ESM), whether ACF2, RACF or Top Secret is an ever-evolving solution with highly advanced security functionality and the de facto central repository for IBM Mainframe security policy management.  From a Security Information & Event Management (SIEM) integration viewpoint, any IBM Mainframe security policy violations will be reported upon in near real-time, while being managed by IBM Mainframe security experts.  Without doubt, if DB2 was implemented before 1997, internal security would have applied, but there has been a ~20 year period where migration to the ACF2, RACF or Top Secret ESM could have happened.  If such a migration activity applies to your organization, I would hope it’s a high priority item, given the potential security risk and priceless value of your business data!