Green House Gas Emission (Hero’s and Culprit’s)

MIDS 209 – Data Visualization and Communication

Team Members

• Megan Pera
• Dustin Burt (http://linkedin.com/in/dustin-matthew-burt-760b112)
• Tarun Chopra (https://www.linkedin.com/in/tarun-chopra-78125842)
• XT Nguyen

Did You Know?

“1 Million metric ton of CO2 emission could fill up a balloon the size of a football field”.

Visualization Goals

Green House Gases (GHG) are gases such as CO2, Methane, Nitrous Oxide and 
Flurorinated gases that trap heat in the atmosphere.

Over time, this causes the earth to get warmer and leads to rapid climate
change. Purpose of this visualization is to analyze the GHG that is being 
produced in the United States over the last 25 years, identify major
contributors and determine if we are doing enough as a country to 
control the emission of GHG in the earth's atmosphere.

GHG Visualization will take the users through 10 phases of the storyboard
described below:

     Step 1 - Introduction to Green House Gases.

     Step 2 - Quantification of the basis metric for GHG 1 million 
              metric tons of CO2 and will form the base unit for the rest 
              of graphics.

     Step 3 - Total US GHG Emissions from 1990 - 2011.

     Step 4 - Total US GHG Emissions from 1990 - 2011 as compared with 
              population growth over the same years.

     Step 5 - Break down of US GHG Emissions from 1990 - 2011 by sectors 
              (Energy, Agriculture, Fuels, Industrial, Waste).

     Step 6 - As Energy is the biggest culprit for US GHG emissions, 
              further break down of Energy sector into various sub sectors
              (Commercial, Electric Power, Fugitive Emission, Industrial,
              Residential) from 1990 - 2011.

     Step 7 - Impact of Land Use and Forestry (LUCF) as a carbon sink and 
              the removal of atmospheric CO2 through LUCF from 1990 - 2011.
     
     Step 8 - State by State comparison for GHG Emission, Population, 
              Energy and Land use forestry.

     Step 9 - What can we do as citizens to reduce individual carbon foot
              print?

GHG Emission in US over last 25 years (Dashboard)

GHG Infographic

Artifacts

GHG story board for usability tests.

Final Project Presentation

Tools

D3, Java Script, HTML,  Tableau, Adobe Illustrator

Data Sources

Green House Gas Equivancy Calculator

CAIT – US States GreenHouse GAS Emissions

Green House Gas Inventory Data Explorer

Green House Gas Emissions, World Bank

Major Land Uses – USDA

Agriculture Live Stock – USDA

Data Science at the Intersection of Security and Privacy

Government organizations have a difficult task: effectively balancing privacy and security requirements in complex systems processing large amounts of sensitive and personal data. Continuous advancement in data science,  and techniques such as statistical analysis, predictive analytic’s and machine learning make the job  even more difficult. By leveraging advances in data science, it is easier for attackers to de-anonymize datasets and expose sensitive information long after the dataset has been released.

Traditional system design and implementation have emphasized security and privacy while the data are contained in the Government organization’s system. The organization complies with FISMA; follows FIPS Pubs 199 and 200; uses security controls specified in NIST Special Publication 800-53, and follows the Risk Management Framework.  The system and data are properly protected during collection, storage and processing, and the problem has been addressed. But that is often no longer true as “Open Government” is changing the rules of the game.

The “Open Government” push to make all datasets publicly available by default has many advantages. It emphasizes Government transparency. It puts data paid for by taxpayers into the hands of those taxpayers. Allowing downstream uses of the data that were never envisioned at the time the data were first collected leads to better governance and accountability.

However, there are problems with the “release everything” philosophy.  Many datasets are released in isolation, without regard to the data sets that are currently available or will become available in the future. Most of the cases of improper data exposure have resulted from cross-referencing between “sanitized” datasets and other public datasets. The sanitized dataset hid the identities of subjects, but de-anonymization can be easily achieved through correlation with other available information. Additionally, the ability to store and quickly process large quantities of information from different sources exacerbates this problem. It is easier than ever for “average attackers” without special skills or tools to correlate data across datasets and figure out hidden information.

Suppose that a law enforcement agency periodically releases its crime records to the public anonymizing all personal sensitive information. However, the crime records might indicate that a Domestic Assault was reported at 123 Main Street on May 1. Who might the victim be? Most states make available on-line a database of their property tax information. This will show who owns the residence at 123 Main Street. If the owner occupies the property, that will be a clue as to the identity of the victim. But what if it’s a rental? Most states make available for a small fee their voter registration databases, listing the home addresses of all registered voters. If a renter occupies the property and is registered to vote, that will provide a good clue. If that approach fails, there are other databases that can be used to gain information.

Two UC-Berkeley graduate students, Alfred Arsenault and Tarun Chopra of the Masters in Information and Data Science (MIDS) program, recently studied this problem and provided a series of recommendations for Government organizations.

Arsenault and Chopra recommended a set of procedures for an organization to follow when collecting, processing and later releasing large amounts of data. Recommendations are geared towards balancing the requirements for both privacy and security. These include having an advocate for protecting privacy involved; understand how long the information in or behind the released dataset may be sensitive; examining other publicly available datasets to determine what data correlations may be possible; protecting the metadata associated with the dataset; and understanding what data science tools now exist and what they can provide.

They also provided recommendations to the data scientists, who must understand the distinction between privacy and security, and be an advocate for data sharing where it doesn’t harm the privacy and security for the data subjects. The data scientist must take accountability and responsibility for understanding the data, how it is being protected for privacy and security threats and the impact of releasing data. Before releasing the data for broader analysis, she should implement a testing procedure for technical solutions that are being used to anonymize and de-identify the data and a process to store the documented results. Data scientists will also need to continuously monitor and keep pace with ever changing laws and leverage technology in order to keep ahead.

Finally, and most importantly, the key trait that will be most useful is the willingness to adapt and reach to changing policies, laws, regulations and security threats.

Full Paper.

z13 – Reminiscing the fond memories and The Penguins Are Coming

Today, I will be presenting at #SHARE in Orlando on z13 performance  and during preparation, I was reminiscing a lot of fond memories around design, development and launch of this amazing piece of technology . It was both a pleasure and honor to be around the creation of z13 platform with over $1B of investment over 3+ years but more importantly to work with amazing set of people that so deeply and passionately care about the mainframe platform and the value it bring to our customers. z13 was launched in the 1st Quarter of 2015 and the results so far have been very impressive with 15% growth in the last quarter. Our vision, while building the z13 platform was to bring a game changing system in the market place that is capable of meeting the various challenges brought together by the confluence of trends like Mobile, Cloud, Data Explosion and Security. By 2018, according to IDC, we are expecting 3.8B mobile users around the world (50% growth), 24 Zetabyte of data (4x growth), 40B connected devices (2x growth) and 77B data center cores (2x growth). We need technologies and platform that have a proven track record of meeting the world challenges and anticipate the future needs. There is no better technology platform than z Systems with its proven track record of over 50 years of technology renaissance, yes over 50 years and the ever present desire to evolve and stay ahead of the times.

z13, built for the the mobile economy has been designed with performance, scale, intelligent I/O and security enhancements to support transaction growth in the mobile world. From a system innovation perspective, the list goes on……

• Up to 141 Processor Cores with 5GHz performance and unprecedented scales for data and transaction growth.

• Up to 320 Separate Channels of Dedicated I/O for massive data and transaction throughput.

• Up to 10TB RAIM Memory delivers up to 50% better response time.

• Up to 30% Better Capacity for Linux and Java with Simultaneous Multi-Threading (SMT).

• Accelerated Analytic’s for Numeric-Intensive Workloads with Single Instruction Multiple Dataset (SIMD).

• CP Assist for Cryptographic Function (CPACF) ) has been optimized to provide up to 2x faster encryption functions on z13.

• Crypto Express5S providing dedicated cryptographic processing for security of transactions and data, faster.

• zEDC accelerated data compression to reduce data transfer volumes & storage costs by up to 75%.

• Up to 17x Faster Analytics than the Competition with IBM DB2 Analytics Accelerator.

Listening to customer testimonies around the business value z13 platform has brought to their enterprises, only strengthen our resolve to build technologies that can not only impact businesses but transform societies. Oh by the way, if you think we are done shaking the business world, please check out the amazing video and tune in at #LinuxCon on August 17th, 2015 and “Yes” The Penguins are coming. 

Going back to preparing for the pitch and the final last minute changes to the presentation 🙂

To learn more about the amazing platform and what it can do for you business following links can provide a good intro.

IBM Redbooks and Red papers introducing the IBM z13.

IBM z13 overview.

IBM z13 features and benefits.

IBM z13 specifications.

Mainframe and Java – Tale of constant innovation.

Recently, Mainframe platform celebrated 50 years of technical heritage serving as an engine of innovation around the world.  Things that amazes me about this wonderful  journey is how in this ever changing world of technology, Mainframe platform can not only exist but keep thriving serving as the backbone of modern enterprise computing. There is no better demonstration of Mainframe innovation over the last 50 years than the tale of Java and how it has evolved over the last 50 years. With over 9 million of Java developers around the world and with 80% of the world’s corporate data either residing or originating on the Mainframe platform makes for a very powerful and relevant system performance tales.

With zEC12 (latest mainframe platform), Java on mainframe platform continues to be industry leading by continuously exploiting hardware features tailored and co-designed with Java.

• Exploitation of Hardware Transaction Memory (HTM) resulting in better concurrency for multithreaded applications.

• Exploitation of 2GB pages frame for improved performance targeting 64-bit heaps and addressing growing storage trends.

• Leveraging innovative runtime instrumentation facility utilized by JVM to gather trace execution and heavy event data (d-cache miss, branch miss etc) at run time. JVM can use this type of information to adapt to the behavior of applications.

• Utilization of 1 meg pageable large pages through flash express providing much o f the same run time performance benefits as non-pageable  1 M pages, with additional benefits of managing memory to improve system availability and responsiveness.

• Another key example of mainframe and Java synergies, is the exploitation of Java 7R1 (java.util.zip.GZIPOutput.Stream) for zEDC (compression) hardware engine resulting in a 91% reduction  in CPU time using zEDC hardware vs mainframe software compression techniques.

• Integration of SMC-R technology (automatically/transparently exploiting RDMA/RoCE for socket based TCP application) with IBM Java 7R1 resulting in better response time for applications.

All of the above improvements are a small subset of key synergies that have been built over the years between mainframe and Java resulting in optimum performance along with the business resiliency provided by the mainframe platform.  Mainframe and Java will continue to evolve in the years to come in order to better serve enterprise computing demands along with meeting strict performance and security requirements.

Key References :-

zOS Java Website

IBM SDK Java Technology Edition Version 7 Information Center

System z and Large Memory – Infrastructure Matters

Recently, there were two white papers published by System z performance team in collaboration with system z ecosystem regarding the benefits of exploiting large memory on System z platform for higher performance gains. First white paper, “System z: Advantages of configuring more memory for DB2 buffer pools” highlights performance benefits for both online bank workload and online database transaction workload. Second white paper, “Performance reporting on exploiting large memory for DB2 buffer pools with SAP” talks more in depth about the performance benefits for SAP workloads while leveraging large amounts of system memory. In both cases, CPU savings and applications response time reduction has been demonstrated in in double digit potentially resulting in huge savings and higher customer satisfaction. Encourage all System z customers to read the above two white papers and determine potential benefits as they are very work load and customer environment sensitive.

On the memory footprint front, System z have been continuously raising the bar going all the way from 64GB available on z900 series to 3TB in latest zEC12. Along with the hardware availability, software stack have also been keeping pace to exploit large memory for optimized application performance. DB2 10 and 11 for zOS allows up to 1TB of memory to be used for all buffer pools in a given member. Recently, a lot of buzz is being created around in-memory computing or revolutions (as some articles would suggest) and how applications are leveraging large amounts of memory in order to reduce total cost of ownership but also solve complex business problems in a quick amount of time. Concept of in-memory computing is not very new but certain technology and cost trends like availability of cheap amounts of large memory and big data revolution is making the technology both easily consumable along with rapid adoption.

A lot of in memory computing discussion was originally centered around analytics, big data, no-sql columnar data base but recently discussion have started to shape along re-modernization of ERP processes and application to leverage large amounts of memory. In order for ERP processes to get benefits from large amounts of memory, legacy applications might need to go under transformation or middle ware/software stack needs to be updated so that system can scale for large memory consumption. With the above two cases mentioned in white paper, applications can easily get the performance benefits by simply allocating large amount of memory to DB2 buffer pools. These two use cases demonstrates the power of balanced System z design (CPU, IO, Memory, Capacity) over various generations resulting in the ability for applications to scale seamlessly and consume greater available system resources.

With all the needs for in memory computing, several technology companies like SAP or Oracle are coming with specific systems to leverage this wave like SAP HANA or announcement of Oracle in memory applications. The drawbacks of some of the above mentioned approaches can be their build for specific purpose and hence may be narrow in applicability. With System z holistic view and balanced system approach, memory is one of the key components of the ecosystem but not the only thing and hence the benefits can be far reaching. We are in the early stages of exploiting large amounts of memory on System z platform and the early results have been promising. One of the interesting and high valued aspect of exploiting large amounts of memory on System z platform is that both the traditional world of ERP application along with the new world of big data/analytics can benefit under one System z umbrella long with world class qualities of service.

Hybrid Computing Model – Bridging computing worlds

As Cloud Computing has gone mainstream, concepts like Infrastructure as a Service, Platform as a Service and Software as a Service are becoming part of current computing industry paradigm. With major enterprises focusing their attention on various cloud computing models and how enterprises can benefit from the agility of cloud computing, discussion are getting heated up on various cloud computing deployment models (Public, Private, Hybrid). Initial journey of cloud computing took its flight from the paradigm of public clouds where vendors like Amazon Web Services, Microsoft Azure or SoftLayer predominantly offered computing resources like servers, storage and network computes through a public computing infrastructure. As enterprises have started to explore cloud computing model, a quick realization has been setting that existing public cloud models can’t be the only form of computing consumption. Let’s look at some of the observations from the market place driving hypothesis of hybrid clouds gaining market attention.

“50% of companies will run on Hybrid Cloud by 2017” – Virginia Rometty, IBM Chairman and CEO, Pulse 2014
“Customer have outgrown the Public Cloud” – John Engates, Rackspace, Chief Technology Officer
“More than 70% of the enterprises plan to complement their in-house server and storage resources with IaaS resources from public cloud providers for primary or peak workloads” – Forrester
“In 2014, $7 billion hybrid cloud opportunity enabling customers to consume public and private clouds, with more workload and greater scale.” – TBR Cloud Program, “Hybrid Cloud Consumer Report”.

As hybrid clouds are getting market traction, lets explore some of key reasons on why hybrid models can bridge computing worlds of public clouds and existing IT enterprises.

1) Existing Investments: Enterprises over the years have invested a lot in their IT infrastructure and have fine tuned it to their business needs and business processes. The notion of hybrid clouds can potentially let enterprises to continue reaping the benefits of their existing IT infrastructure but also take advantage of rapid innovation in public clouds. This kind of arrangement can enable enterprises lower their overall TCO by balancing business requirements between their existing IT infrastructure and public clouds, possibly opening more doors to enhance customer experiences with deeper insight and rapid agility gained from public clouds.

2) Application Performance: Applications demanding sub second performance might find it difficult to entirely move to a public cloud infrastructure with out significant investments in application re-structure and re-write. Over the years, enterprises have optimized their entire IT computing stack from hardware, firmware, hypervisors, operating systems, middleware and applications in order to meet customer stringent service level agreements. Simply migrating existing IT applications, with out changing the underlying application structures will not yield desired performance results. One thing to also remember is that in traditional large enterprise environments a lot of legacy applications are built with scale up architecture where as cloud computing paradigm is very much based on scale out architecture further complicating the migration of applications. There is a whole separate discussion centered around bottoms up application development, keeping cloud architecture in mind and Platform As a Service (PaaS) is a prime example of the new kind of application development and deployment model in a cloud environment.

3) Compliance Issues: Enterprises have to adhere with various regulatory compliance restricting the switch to a complete public cloud computing model. For example, financial institutions dealing with credit and debit card payments must adhere to Payment Card Industry Data Security Standard (PCI DSS). Financial institutions also have to adhere with Gramm-Leach-Biley Act (GLBA) protecting customers financial and personal information. Health care organizations handling sensitive medical records must adhere to Health Insurance Portability and Accounting Act (HIPAA) in order to protect patient privacy. With regard to above mentioned compliance hurdles, a lot of cloud vendors have started to provide both private and hybrid cloud infrastructures to meet stringent client needs.

4) Inter-Geographic issues: With recent NSA revelations or so called Snowden effect, multinational regions are very sensitive about where data resides and who is handling sensitive data. In today’s environment, each individual country laws prohibits sensitive data leaving out of the country and hence we are seeing cloud leaders setting up data centers in various parts of the world. Also, in a lot of growing markets, investment models are cofounded with govt being the major stakeholders and hence still the need to set up private data centers. Even in those scenarios, enterprises still will like to leverage public clouds for its agility and cost effectiveness, hence arising the needs for hybrid clouds.

5) Skills and Cultural Paradigm: Another softer side of the business that is fueling the demand for hybrid clouds is cultural and skills aspect with in existing enterprises. Traditional IT models are in better position to understand various cloud models and how does their existing investments and skills can be leveraged in this new cloud paradigm. Traditional IT skills needs to evolve and also existing employees in the enterprises needs to get comfortable about what cloud means for their traditional roles. Hence, Hybrid Clouds can bridge skills and cultural inhibitors by allowing enterprises to leverage current set of investments in skills and infrastructure while explore the benefits of public clouds and relevancy in their existing business processes.  

Going forward, all three deployment models Public, Private and Hybrid model should continue to co-exist and bring varied value proposition in varied customer scenarios. Even cloud vendors have begin to realize that there is no one fit model and are continuously evolving in order to meet ever changing customer requirements. Hybrid computing models will also need continuous innovation in the area of security, common management controls, networking and application development. In the next series of blog, I will attempt to raise some of the issues with hybrid cloud adoption and also spell out some of the architectural concepts like cloud bursting, extending existing applications or leveraging rapid application development environment in public cloud to provide richer set of consumer services and also extending private enterprise infrastructure.

Useful links:

Has Hybrid Clouds Arrived? Part 1

How Cloud Computing Affects Regulatory Compliance?

Open, Hybrid, Interoperable: The Cloud Future

New Hybrid Clouds Model Emerging

System z and Business Analytic – A Perfect Tango

” Data is the next Intel inside “ by Tim Oriely is ubiquitous quote of our times. We have all heard about the explosion of data growth in recent times with the internet of things and emergence of social media ( Facebook, Twitter, LinkedIn, Tumblr, Instagram) or some might say it Web 3.0 revolution. According to IDC, amount of digital information created, replicated and consumed worldwide will grow exponentially from 0.8 trillion gigabytes in 2010 to 40 trillion gigabytes in 2020.  Also according to IDC, many organizations are expected to experience a doubling in the volume of data across their enterprises approximately every 24 months and are investing heavily to scale their data storage and management platforms to accommodate this growth.  There are a lot of big data, mobile and cloud growth statistics available on the web, stressing the importance of how data analysis and insights drawn from it is becoming a competitive edge for businesses around the world. Generally, when we talk about explosive data growth, discussion is more geared towards unstructured form of data. By definition, unstructured data is heterogeneous and variable in nature arising from sources like Twitter, Facebook, Instagram or even huge amount of email exchanges. Although most of data growth is around unstructured format but there is still a lot of structured data around in corporate warehouses that can be a potential gold mine for competitive advantage.

Recently, stumbled across a very informative and interesting blog by Paul Dimarzio on relevance of System z in today’s analytics environment and the importance of analyzing structured data (transaction processing) in today’s IT environment. This is where I like to pick some pieces and explain why System z and business analytic forms a perfect tango. Enterprises and technology has evolved from business analytic being a skunk project to being a truly integral part of how business decisions gets made. An enterprise will get maximum benefit of business analytic, if it is seamlessly integrated into existing technology infrastructure and part of day to day business transactions. Given below are key points that an enterprise should think in order to maximize business analytic benefits in their business environment.

1) Seamless integration with existing business processes and technology infrastructure.

2) A common management interface across its business IT processes and analytic capabilities.

3) Scalable analytic infrastructure that can meet demands of heterogeneous, compute intensive analytic applications and databases.

4) Secure analytic infrastructure environment that can meet strict demands of an enterprise business governance model.

Business analytic on System z can provide enterprises with a solution leveraging traditional strength of z platform (built in virtualization, 24*7 availability, EAL5 security certification, ability to run heterogeneous workload) coupled with integration of analytic appliance like Netezza along with SPSS and Congnos software stack. Given below are the key technical capabilities that have been added to z platform for analytic workload;

1) For business intelligence and predictive analytic, IBM analyticssoftware Cognos, SPSS has been tightly integrated with DB2 and web sphere to provide customers key insights into their data.

2) IBM zEnterprise analytic system 9700 and 9710 provides an integrated data ware housing facility where Netezza query accelerator has been integrated with DB2 and z/OS to provide customer a pre configured manageable environment to provide quick analytics operations. In certain cases, queries that in the past could have taken 5 hours of processing can be completed in just 20 seconds, quite astonishing.

3) On IBM zEnterprise analytic system 9700 and 9710, there are also add-on packs available to meet complex business demands. Data Analysis Pack for business intelligence and predictive analytic include IBM Cognos Business Intelligence, SPSS Modeller W/Scoring Adapter, Data Integration pack includes all the tools needed for integrating and transforming data from various heterogeneous sources and Fast Start service pack can be leverage for customized tailored solution.

4) DB2 11 for zOS provides integration with IBM InfoSphere BigInsights (Hadoop implementation). A typical use case scenario will be to leverage

Above points clearly highlight, how System z is innovating itself to be a premier enterprise analytic platform. As I discussed in my previous blog, driving maximum utilization from current and future IT investments will be at the forefront of every business strategy. For existing System z customers, the enhancements and integration of analytic technology in to the z platform further provides customer with the avenue of lowering down the TCO and TCA barrier. For new customers, System z with its rich innovative history of enterprise grade usage coupled with tight integration of world class analytic s technology should offer a very competitive technology platform choice but also offer fresh perspective.  Cheers to the next 50 years of System z innovation!!!!!

Useful links to further learn more about System z and Analytics:

http://www-01.ibm.com/software/os/systemz/badw/
http://www.redbooks.ibm.com/abstracts/redp5062.html?Open
http://www-03.ibm.com/systems/z/solutions/dataserving/index.html
http://www-03.ibm.com/systems/z/solutions/data.html

System z Accelerator and Appliance – A Tale of Constant Innovation

In 2014, System z will be celebrating 50 years of technology heritage and innovation A quite remarkable achievement for a technology that was invented in 1964 and has been pronounced obsolete or dead more than a couple of times. Remember Stewart Alsop famous prediction in 1991 of the last mainframe being unplugged by 1996. The secret of mainframe cherished heritage lies in the DNA of innovation and constant evolution with changing times. A very good read on mainframe survivor spirit was featured in New York times (Why Old Technologies are still kicking?). From the transition of Bipolar to CMOS, introduction of 64 bit architecture in 2000 and the arrival of Linux in 2001, System z or mainframe is always at the forefront of constant reinvention. Recently, System z BC12 has been recognized as one of the coolest server of 2013 . Sprinkling holiday cheers and wishing every body ChriZtmas, I will like to spend some time on a topic that is at the forefront of next generation of innovation on System z platform. Accelerators and Appliance model is not new to the industry with various companies either try to build appliance and integrated model servers like EXADATA from Oracle or UCS from Cisco. or even servers that are accelerated for a particular purpose like Netezza . System z had introduced accelerator and appliance based computing to compliment and extend traditional strengths for OLTP based work load. Combine with industry famous Reliability, Availability and Security of the z box, appliances and accelerators are further extending value proposition of System z to enterprise customers.

In last couple of years, System z has delivered on key technologies based on accelerator and appliance model like introduction of Flash express, zEDC (compression accelerator) and zAWARE (appliance based predictive availability modeling). In this blog, lets take a closer look at some of these technologies and potential benefits to customers.

zEDC: IBM zEnterprise Data Compression (zEDC) was introduced with zEC12 allowing high performance, low latency compression with little overhead. Software exploitation was provided through the release of z/OS V2.1 and the zEDC technology is seamlessly integrated with existing System z compression technology. Hardware compression had existed and is still present on System z but while utilizing hardware compression, a trade-off must be considered between CPU cycles usage and savings in storage consumption. WIth the introduction of zEDC, large files can be compressed by using zEDC compression card where as smaller set of files can still keep on leveraging CPU compression. zEDC potential use cases can range from Java, managing large sequential data sets to efficiently delivering compression when back up and restoring data (available at a later data, see IBM announce material). In certain scenarios, utilization of zLib compression through zEDC can lead up to 118x reduction in CPU and 24x throughput improvements.

For more information on zEC12, please refer to IBM techdoc: zEnterprise Data Compression for zOS

zAWARE:  With the launch of zEC12 in 2012, IBM zAWARE was also introduced as a self learning, IT analytics solution for z/OS in an integrated appliance format. Goal of this cool technology is to help IT operations quickly identify and cause of an issue that can ultimately lead towards faster recovery of service and in terms improve overall system availability. Problem analysis is based on sophisticated machine learning algorithms that has the ability to self learn a given system behavior, patterns and highlight deviation before they become issues in a typical system environment. Another neat part of the technology is its integration with other automation tools like Netview or Omegamon. By offering zAWARE in an appliance format and tightly integrated with System z as a stand alone LPAR, enterprise customers can quickly experience the benefit of zAware to simplify and modernize IT availability environment.

zFlash Express: An optional feature introduced with zEC12 focused on providing increased availability for critical business workloads. Flash express on System z is initially being used for paging and capturing large machine service dumps. Utilizing flash for faster page ins can reduced stand alone dump time by approximately 19% (varying with customer environments), which in turn can enhance the availability of the system. Again key attribute of technology, is its integration with existing ecosystem hiding the complexities of how to leverage Flash express for paging and dumps from customer applications. This seamless integration of technology into existing application infrastructure while providing greater value is a key recipe for success in today’s enterprise computing environment. A good example of this seamless integration is the encryption of all data stored in flash express memory. Hardware encryption is supported through the use of smart card and a unique encryption key stored on the support element.

For more information on Flash Express, please refer to IBM TechDocs: Flash Express – A Performance Snapshot

With the emergence of Cloud Computing model, driving maximum utilization from current and future IT investments will be at the forefront of every business strategy. Also with each customer closely looking at Total Cost of Ownership (TCO) and Total Cost of Application (TCA), System z is continuing to make investments in technology like accelerators and appliances that can further lower down TCO and TCA barrier. System z is uniquely positioned to serve both in traditional and cloud market where enterprise customers can enjoy the cloud in a box kind of attributes of tremendous scalability, deep virtualization and world-renowned reliability, availability and serviceability. Cheers to the next 50 years of System z innovation!!!!!

Evolving Cloud – Differentiation in growing segment

As Cloud Computing is gaining traction in market place, customers are getting more knowledgable about cloud computing concepts and what it can do for their business. Discussions are starting to shift from basic rudimentary questions related to what is cloud and how cloud can help enterprise grow. Customers are raising key questions related to performance, reliability, availability and scalability of cloud and how enterprise can be sure of their business integrity running on cloud. To this end an interesting evolution is starting to happen in cloud computing where managed service providers are starting to look into ways on how to differentiate from each other. Different offerings and services are starting to appear in cloud computing consumption model and cloud is starting to evolve from basics of IaaS, PaaS and SaaS. Let’s talk about four recent examples where this evolution is being highlighted.

Cognition as a Service (CaaS): Recently, there was a very interesting announcement from IBM about opening up WATSON (providing Cognition as a Service) as a cloud application to spur growth and innovation around WATSON ecosystem. With this announcement IBM is aiming to attract entrepreneurs from different fields to leverage Watson cognitive abilities in various disciplines. IBM is planning to offer this service as a Cloud offering like Platform as a Service to develop applications around Watson. With Watson cloud developers will get access to a development toolkit, Watson APIs, educational material and an application marketplace.  The key here is with cloud maturing as a business model it is facilitating an open and collaborative way to perform business. With cloud technologies IBM is planning to make Watson system available to general public with out huge initial cost for potential users.

IaaS offering of a different kind: Amazon recently announced that it will be offering NVIDIA GPU instances as an EC2 service in Amazon public cloud.  With this offering Amazon is trying to capture customers that demands high compute processing for workloads such as game creation, 3D visualization, video streaming and graphic intensive applications. Other vendors such as Softlayer (an IBM company) have also started to offer HPC and NVIDIA blades in order to meet the demands of high cpu intensive applications. In future as cloud gets more predominant in enterprise computing space further opportunities will arise where vendors can specifically provide hardware instances tuned to OLTP or IO bound workload. In short, IaaS will continue to evolve from a general CPU offering to a much more refined instance of customized compute capabilities.

Performance Optimized Cloud: At Cloud expo 2013, Verizon which had acquired Terramark introduced a new cloud with key differentiation around guaranteed performance. This is another evolution of a tradition cloud hosting environment where managed service providers are trying to find differentiate in their offerings and starting to address customer pain points. For Verizon, having an IaaS offering based on guaranteed performance makes sense as Verizon can leverage existing vast investments in broadband infrastructure. As more and more entrants are starting to come in the MSP space established players like Amazon, Softlayer, Rackspace will look to continue differentiate by offering advanced offering to its customers and also addressing key pain points like performance, security, availability and reliability.

Rackspace and Private Clouds: Another dominant player in cloud hosting space Rackspace has started to make a big push around private clouds, merging and managing boundaries between public and private clouds. With today’s enterprise, security of data and application is the main concerns while evaluating cloud computing environment. Hence, Private or Hybrid clouds have started to gain more prominence as it can offer enterprise their own dedicated set up in an MSP environment. According to IDC, private cloud services are expected to see a compounded annual growth rate of more than 50% from 2012 to 2016.

All the above examples are highlighting the continuous evolution of cloud computing and how competition is adapting to the changing landscape by offering customized offerings. In future, further customization to cloud will continue to happen in order to address challenges like security, performance and reliability posed by growth in consumption of cloud computing.

Cloud Expo 2013 Thoughts

Day 2-3 of  Cloud Computing session was focused a lot on how the traditional IT is changing from being an operator model to more of an innovator model. There were quite a few terms used for describing IT services – operator, brokers, innovators and at one of the sessions it was quite funny that no body could pin point what role IT services will play in the future. Also, second day got more focused on issues facing Cloud Computing like system performance, network performance, security, regulator and compliance issues. These are all issues that will require future innovation but there is a lot of momentum in cloud computing area and key innovation will be soon following in order to address critical issues.

On Cloud Performance side, network latency was highlighted but key that was missing is how system performance is maximized with in cloud data center. Cloud computing focus is very much driven by pushing as much physical resources as possible on the work load but enterprise applications will demand a combination of CPU, Memory and IO and it will be interesting to see how they will scale in current cloud IT environment. Some of these similar views are starting to be realized by various vendors where next generation of hardware infrastructure is being created keeping in mind effective pooling of hardware resources for example Open Compute project.

A lot of focus is also being driven around PaaS environment and lot of innovation is also happening in this space, saw a cool demo from IBM related to their BlueMix technology and it got me wondering if IBM can also leverage this environment for System z development environment.