Datastage
11.5 New features for IIS common components
This
post will help you to understand the changes/enhancement made on IIS 11.5 for connectivity
services, administration services, and deployment services to support its data
integration and data governance capabilities. As part of IIS V11.5, the
following common platform enhancements have been made.
Engine tier execution within Apache
Hadoop
This
feature enables data integration, data cleansing, and data profiling and
analysis workloads to run on the data nodes of a Hadoop cluster.
# The Information Server engine can be installed on
the edge node of a Hadoop cluster and the required binaries can be made
immediately available to all nodes of the cluster through three different
mechanisms: HDFS distribution, dynamic runtime copy, or NFS. Organizations can
choose from among these models to best fit their architecture and security
requirements.
# A new Application Master component submits parallel
job run requests to YARN to secure resource allocation on the data nodes of the
cluster as appropriate to the job run requirements. Users can leverage features
such as Hadoop Node Labels to dedicate processing to specific nodes, and YARN
schedule queues to ensure appropriate workload management control.
# The stages within
the job then operate on data partitions that are available for data on those
nodes (Data can be shipped to specific data nodes based on YARN's resource
selections).
# The IIS engine
automatically uses in-memory features such as data pipelining, data
partitioning, and dynamic repartitioning to minimize job runtimes.
# Support for Kerberos-enabled clusters.
Hadoop
version currency
# As with previous
releases, the product supports Apache Hadoop distributions that conform to the
Open Data Platform (ODP) requirements, such as IBM Open Platform, IBM
BigInsights, and HortonWorks, as well as other non-ODP distributions, such as
Cloudera.
# Hadoop distribution
is supported for reading and writing of data from non-Hadoop IIS deployments as
well as Hadoop engine tier deployment: IBM Open Platform V4.x, IBM BigInsights
V4.x, HortonWorks V2.2, V2.3, and Cloudera V5.3, V5.4.
Updated
Repository and Application Server options
# Repository Tier
supports Oracle 12g (including RAC) and SQL Server 2014.
# Services Tier
supports IBM WebSphere Application Server 8.5.5.
Data
integration
Information
Server now includes the following additional features for data integration:
Data
integration for Hadoop
All
connectivity, transformation, and data delivery features are now able to
execute on the data nodes of the Hadoop cluster.
Hadoop
file connectivity
# Expanded support for Hadoop files, including new
data formats, additional character sets, and additional data types.
# Available on all supported operating systems,
including Microsoft™ Windows™, AIX®, Linux™ on System z®, and Linux (both
Hadoop and non-Hadoop).
Embedded
protection of sensitivity data in-flight
Information
Server Enterprise Edition licensing now includes Optim data masking libraries
for protecting sensitive information in a DataStage® parallel job.
Push-down
processing into relational database
# The DataStage Balanced Optimization features that
allow users to execute their data integration workloads within a relational
database by automatically rewriting the job design as SQL is now included in
the InfoSphere DataStage license and several products that include InfoSphere
DataStage as a supporting program.
# Balanced Optimization has also been extended to
support push-down into an IDAA configuration on System Z.
Data
quality
Information
Server now includes the following additional features for data quality:
Data
classification
# Support for data privacy, data masking and test data
management initiatives by identifying where Personally Identifiable Information
(PII), sensitive and other classes of data are stored.
# Quick time to value by identifying the type of data
contained within a column using three dozen predefined, data classes including
credit card, taxpayer IDs, US phone number, and others.
# Extensible classifications enable you to create and
customize three types of data classes: valid values list, regular expression
(regex), and Java™ class.
# Column analysis results provide suggested data
classification based on data values. Users can review these classifications and
select the data class that best represents the data stored in each column.
Performance
enhancements
# Improved scalability and reduced resource
consumption during data profiling (column analysis).
# A new option to limit the distinct values stored can
dramatically reduce the size of the results database (IADB).
# Improved scalability for cross domain and foreign
key analysis. The new algorithm does not require a full column analysis with
capture of all distinct values.
Certifications
# Data profiling, classification, investigation,
standardization, matching, survivorship, address verification, and monitoring
are now supported running directly inside a Hadoop cluster.
# USAC and AVI address cleansing and validation is
also supported running on the Hadoop cluster.
# Oracle Connector certification for IADB.
InfoSphere
Information Governance Catalog
Information
Server now includes the following additional features for data governance:
Governance
Catalog Extensible Framework
Allow
customers to define and register new assets within the catalog, setting their
display and structural definitions, allowing users to recognize and understand
the information best. Further, these assets may be fully governed and mapped
for purposes of lineage.
Data
class definitions
#.
Allow for the detected Data Classifications to display and be searched within
Information Governance Catalog.
# Allow
Administrators to create and manage Data Class definitions, including their
regular expressions, valid values or range.
Multilanguage
support
Enables
business terms to be defined in different languages and associated to each
other to allow the organization to manage business definitions that are related.
XML
Schema Definition support
Enables
you to import an XML Schema definition file, and browse, or share the included
entities, attributes, and types across the enterprise, and further, governing
and annotating the entities, attributes, and types.
Hadoop
file metadata import
You
can now import metadata for Hadoop files.
Column-level
lineage for Hadoop files
Job
lineage now includes column-level lineage for the file connector when used for
files in Hadoop or on other operating systems.
Asset
Interchange support for Extend Lineage content
Enables
you to easily and seamlessly migrate all Extended Data Source and Extension
Mapping content from previous versions of Metadata Workbench or between
versions of Information Governance Catalog. This capability was not previously
available.DK®
No comments:
Post a Comment