Software development processes generally follow easily identifiable stages and become increasingly more challenging.

They differ from traditional manufacturing project stages in various ways, which make them very risky

and uncertain. Traditional software risk management practices are more focused on qualitative judgment and experiences;

however, with exponential growth in number and size, software companies require effective scientific

methodology for risk management. The main objective of this study is to increase the effectiveness of risk detection

and mitigation practices in the software development process. Another objective is to analyze these practices,

identify the areas for improvement, and develop a mechanism for their quantification. We also aim to identify the

processes, which cause problems and suggest strategy to eliminate or reduce the harmful effect of these processes

in minimum possible cost and time. Analytical hierarchy process and fuzzy set theory are suggested as effective

tools to achieve this objective.

Software development processes generally follow easily identifiable stages like project

planning, requirement definition, design, development, testing, integration, installation, acceptance

and support. However, they differ from traditional manufacturing project stages in

various ways. Firstly, we need to give time and cost estimates to the customers in advance

without actually knowing its exact nature, which make software projects very risky and uncertain.

Secondly, to judge the requirements in diverse domain areas with almost the same

set of manpower requires a lot of flexibility on the part of employees and management.

Then fast rate of changes in technologies, customer requirements, possibility of unexpected

number of employees leaving the organization make the task even more complicated. Integrating

the complete solution, implementing it in successful ways, and making the end-user

understand the software and changes in the traditional processes are all part of the software

development life cycle. More and more global companies are outsourcing their software

development projects to offshore software development destinations like India which are

626 NIRAJ KUMAR AND K. R. SRIVATHSAN

throwing new challenges in terms of requirements specifications, cultural differences, effective

communication, security of valuable information, more rigorous legal, governance and

compliance standards.

Because of challenges posed by software development risks many approaches have been

proposed to improve the development process. These approaches include various computeraided

software engineering (CASE) tools, enterprise project management tools, conceptual

tools like various quality, coding, process standards, models, and frameworks, and various

software engineering models like traditional waterfall and spiral model.

In some cases, modeling and design tools are widely used and include Rational Rose [1,

2], versioning control tools like CVS, various testing and bug-tracking tools like Bugzilla,

integrated development environment and automatic code generation tools like Visual Studio,

Jcreator, Dreamweaver and others. Requirements development and management have

always been critical in the implementation of software systems. Some automated tools are

also available to support requirements management. The use of these tools not only provides

support in the definition and tracing of requirements, but also opens the door to effective

use of metrics in characterizing and assessing testing. Metrics are important because of

the benefits associated with early detection and correction of problems with requirements

[3].

Enterprise project management tools are found to provide a wide range of functions.

Among these are scheduling, resource allocation, and cost estimating, budgeting, and collaborating.

It is important to note that these tools emphasize project performance relative to

resource consumption within a given set of time constraints (i.e. progress and dates). Some

of the popular enterprise project management tools include Welcome product suite, Microsoft

project 2002, Primavera P3e suite. While interest and investment in CASE and project

management tools are rising steadily, actual experiences with tools have exhibited more

ambiguity. There has been no systematic examination or formulation of the organizational

changes surrounding CASE tools [4]. The major challenge is to develop quality software in

a reliable and repeatable manner while improving productivity [5].

On the conceptual front all these challenges gradually led to development of capability

maturity model (CMM) and concept of 'Balanced Scorecard'. The concept of 'Balanced

Scorecard' [6] developed in the early 1990s by Kaplan and Norton represented an advance

in the field of measuring enterprise performance, providing a framework for companies to

evaluate both financial and 'non-financial', or 'extra-financial' measures such as quality,

customer and employee satisfaction. This framework was widely used for enterprise-level

planning, control and monitoring by software companies. However, it suffers from limitations

of over reliance on expert judgments for decision making.

CMM [7] was developed in the early 1990s by Carnegie Mellon's Software Engineering

Institute (SEI). Since then many versions of this model appeared and are widely accepted as

most rigorous and best standard by software industry throughout the world. What led to instant

success of this model is its ability to focus not only on external processes but also

provide a framework for continuous improvements of internal processes in the company.

ENTERPRISE RISK EVALUATION AND CONTINUOUS MITIGATION 627

CMM Level 2 KPAs CMM Level 3 KPAs CMM Level 4 KPAs CMM Level 5 KPAs

Requirements management Requirement development,

Technical solution,

product integration

Quantitative process

management

Defect prevention

Software project planning Verification, validation Software quality

management

Technology change

management

Software project monitoring

and control

Organization process

definition and focus

X Process change

management

Supplier agreement

management

Integrated software

management

X Continous improvement

and optimization

Measurement and analysis Organizational training,

integrated project

management

X X

Software configuration

management

Risk management,

integrated teaming

X X

Process and product quality

assurance

Integrated supplier

management, decision

analysis and resolution,

integrated supplier management,

integrated teaming

X X

The CMM establishes an yardstick against which it is possible to judge, in a repeatable

way, the maturity of an organization’s software process and compare it to the state of the

practice of the industry. The CMM can also be used by an organization to plan improvements

to its software development processes.

However, implementing the CMM framework is a very challenging problem for three

fundamental reasons. First is the issue of recognizing all input/output parameters in real

time, which influence various key process areas. Second, methodologies for quantification,

optimization, and continuous improvement of these processes are vague. Third is the issue

of synchronization of these processes with overall organizational objectives, profitability,

efficiency, and growth.

Our primary focus in this study is to develop a model to analyze risk management (CMM

level 3), quality management (CMM level 4), quantification of processes (CMM level 4),

and defect prevention, continuous improvement and optimization (CMM level 5). Then

based on analysis we also suggest improvement into various processes, which should be

cost effective and suited for particular organization culture.

Software development process analysis is an important first step towards making the

process more efficient and profitable. However, analysis is difficult not only due to its

complex nature because of large number of subsystems involved and dynamic interaction

and influence of one over another, but also because it is difficult to quantify their contribution

and influence on the software development process as a whole. For example, technical

ing any software company. However, technical expertise in itself can play an important role

in the quality of software delivered, at the same time it may also lead to cost and time overrun,

which may have negative consequences for the project. Similarly, the quality of software

also depends upon the processes adopted for quality control and on the amount of time

and funds available for the project. Also, the resources of an enterprise are limited so not all

sources of risk can be immediately eliminated and priorities need to be established. Studying

the various processes of a software development life cycle and measuring their impact

in quantitative terms is one of the important objectives of this study. Again identification of

key sources of risk and the ability to measure the level of its harmfulness on the system as a

whole is another important objective of this study. In this paper, various risk sources for a

software enterprise have been identified and a new approach based on analytical hierarchy

process [8] and fuzzy set theory [9'11] has been suggested as effective tool for enterprise

risk evaluation and continuous mitigation.

Many researchers have tackled problems related to software development processes, practices

and tools, multicriteria and multiattribute decision-making, risk management and

fuzzy set theory.

Wiegers [12] points out unstated expectations as major source of software project failure

which can lead to erroneous assumptions, unfulfilled dependencies, unexpected risks and

disappointed customers. He emphasizes the importance of documenting the requirements

thoroughly, precisely and without ambiguity. Mall [13] gives an overview of software engineering

practices and covers almost every aspect of software engineering in brief.

Ming and Smidts [14] deal with ranking of software engineering measures based on expert

opinion. Over reliance on expert opinion is a major limitation of this study. These theories,

methodologies and tools need to be subjected to rigorous test of practices to live up to

their perceived expectations and promises.

Many methodologies are available [9, 10] in the literature for multicriteria and multiattribute

decision-making including data envelopment analysis [15], analytical hierarchy

process [16], multiattribute utility theory [17], and Bayesian analysis and outranking methods

[19]. The AHP, designed to solve complex problems of multiple criteria involving both

qualitative and quantitative parameters, was proposed by Saaty in early 1970s. The application

of AHP is based on four basic principles, namely, decomposition, prioritization, synthesis

[22] have demonstrated the effectiveness of AHP to solve real-life industrial problems.

Kumar and colleagues [23'25] have tackled the problem of enterprise-level planning and

control with multiple criteria which include parameters like capacity, productivity, profitability,

environment and safety involving a number of decision-making units (DMUs).

The acceptance of term risk is universal and it penetrates every discipline of the society.

Each discipline visualizes risk in its own understanding. The concept of risk management

varies from macro to micro level. Several authors have highlighted the objectives of establishing

risk assessment and mitigation process [26, 27]. However, due to the dynamic nature

of interaction between various risk sources understanding of their relationship is

ENTERPRISE RISK EVALUATION AND CONTINUOUS MITIGATION 629

imprecise which can be equated with fuzziness. Further, they require far more detailed description

than is usually available for analysis. Application of the fuzzy set theory is an effective

tool to handle these kinds of real-life situations.

The theory of fuzzy set proposed by Zadeh [11] provides a strict mathematical framework

in which vague conceptual phenomena can be precisely and rigorously studied. It has

led to a paradigm shift in the way problems are solved in various disciplines. It can also be

considered as a modeling language well suited for situations in which fuzzy relations, criteria

and phenomena exist.

Most software development projects fail to deliver acceptable systems on time and within

budget. Many studies were conducted to study software project failure rate at global level

including the Conference Board Survey 2001. Their key findings suggest that 40% of software

projects failed to achieve their business case within one year of going live and project

costs were found to be on average 25% over budget from the original estimates. Traditionally,

user requirements, inadequate user documentation, excessive schedule pressure,

low quality, low user satisfaction, cost overruns were considered some of the major risk

sources for software projects. However, increasing concern over security, legal problems,

rising cost of software development, and HR-related problems necessitates that these factors

should also be taken into account in risk management [4, 7, 18].

Much of the failure could be avoided by managers proactively planning for dealing with

risk factors rather than waiting for problems to occur and then trying to react. Usually, this

reaction is too little and too late, because by the time the problem is fully recognized, the

schedule has already slipped, a considerable investment has been made, and the product

quality has suffered due to introduction of errors or workaround. Risk management is an

important tool to provide insight into potential problem areas and to identify, address, and

eliminate them before they derail the project. Software risk management is important because

it helps avoid disasters, rework, and overkill, but more importantly because it stimulates

win'win situations [3]. A typical risk assessment and mitigation system may follow

the following cycle (Fig. 1):

Typical internal and external factors causing risk in the software industry include the following:

630 NIRAJ KUMAR AND K. R. SRIVATHSAN

Sources of external and

internal risk in an enterprise

Risk identification Risk analysis and

prioritization Risk mitigation

New risk sources

Mitigated and

redundant risk

sources

ENTERPRISE RISK EVALUATION AND CONTINUOUS MITIGATION 631

exchange fluctuations, risk from increasing software development cost, risk from litigation

expense.

All the factors listed above are important sources of risk in the software industry. It is

also important to understand that some of these sources are critical and their impact is profound

across various processes of software development. Proper evaluation of all these factors

and identification of key risk sources and subsequently their reduction or elimination

with minimum cost and resources is necessary to make system more efficient and profitable

and to gain competitive advantage in the market.

Methodology to be adopted for this study is four fold?first to identify the major risk factors,

then able to quantify most of them, prioritize the factors based on their potential for

causing risk to the organization and finally developing a general-purpose risk management

software. These four stages can be summarized as

The constraints of time and cost make it impossible to eliminate all forms of risk in any enterprise,

so an effective way to eliminate key sources of risk and continuous improvement

of risk control process is required. Identifying and quantifying the key sources of risk that

should be eliminated is an important first step towards achieving this. Figure 2 displays

various potential risk sources identified and classified for a typical software enterprise.

Then harmfulness of each type of risk on the system as a whole is proposed to be determined.

Some factors like risk from project rejection by customers, risk due to theft or loss of

valuable information, risk from high cost of security management, risk from excessive inventory

level, financial risk, risk from currency fluctuations, risk from increasing software

development cost are relatively easy to quantify and estimate in monetary terms and their

harmfulness on the system as a whole can be determined. For example, consider risk due to

security-related problems. The cost of security-related problems on a software enterprise

ent of the software developed like cost of maintaining firewalls, gateways, etc. These costs

can result in increased capital and operating costs for software enterprise. Variable costs are

those that vary directly with the software developed. Methodology that examines the cost

impact of security-related problems can be based on crude estimates of the order of magnitude

of fixed and variable costs. For this, reliability analysis of failure due to securityrelated

problems is done and hazard function is determined.

Then cost model of security-related problems on software system can be given as:

= × + ×

ENTERPRISE RISK EVALUATION AND CONTINUOUS MITIGATION 633

where,

After evaluation of software development process and identification of the various forms of

risk and its degree of harmfulness, fuzzy clustering is used to cluster more harmful forms of

risk on the basis of their fuzzy correlation and categorize the various risk sources into key

634 NIRAJ KUMAR AND K. R. SRIVATHSAN

risk sources (Fig. 3). Expert grading method is used to estimate the correlation between objects.

The correlation between two risk sources is (0, 1). The bigger the number, the

stronger the correlation is. After generating tolerance matrix and transforming it into fuzzy

equivalence matrix, more harmful sources of risk that are strongly correlated can be clustered

into a single risk source. This stage is likely to bring down the number of unmanageable

risk sources into manageable few.

The priorities by which key forms of risk should be eliminated are ranked based on the following

important principles:

User-friendly computer software can be developed, which can quantify, cluster and rank the

various sources of risk by giving suitable input. Accordingly, management will be able to

generate useful information for elimination of risk sources and their effect on the software

development process.

ENTERPRISE RISK EVALUATION AND CONTINUOUS MITIGATION 635

Choosing the

most harmful forms

of risk sources

Identifying and classifying all forms of risk in software process

Evaluating the harmfulness of each type of risk

Clustering to identify the key sources of risk (Fuzzy clustering)

Dynamic

recycling for

total risk

Determining the primary elimination measure for each key risk source elimination

Ranking the key risk sources (Fuzzy comprehensive evaluation)

Identifying the eliminated object'the most harmful key risk source

Making systematic plan to eliminate this key risk source

Yes

No

Validation and implementation of the model to the scale, which is required for this kind of

research, is yet to be fully realized. However, preliminary attempt of its validation based on

information collected from a middle-level consultant of a fast-growing CMM-level 5 company

has been done. This company is situated at Technopark, Thiruvananthapuram, and it

relies on audits, reviews and testing for risk and quality management. However, no specific

information regarding quantification of quality and customer satisfaction level has been

provided.

Based on information, which is related with breakdown structure for software enterprise

in Fig. 2, data was collected in April 2005, which identifies two most important risk sources

636 NIRAJ KUMAR AND K. R. SRIVATHSAN

Category Two most important risk sources

in each category (Table II). However, these results are not conclusive, as the authors did not

verify practices and processes of the company.

A web-based system for risk evaluation, quantification and multicriteria decision-making

using Java-based technologies (JSP/Servlets), Mysql database and Apache tomcat application

server were already started (see Fig. 4 for screenshot of one such interface) and plan to

make it available in public domain after its completion.

Enterprise risk management is one of the most important strategic business tools to manage

effectively a variety of risks to gain competitive advantage and add value to the firm. This

study has identified the various external and internal risk sources in a software enterprise.

Authors have proposed a scientific model based on AHP and fuzzy set theory to prioritize

various risk sources and developed a framework for their continuous elimination by adopting

enterprise-specific strategy. The paper also emphasizes the dynamic interactions between

these factors and their suggested importance, quantification to judge the impact on

software enterprise as a whole. This model is highly flexible and customizable according to

specific enterprise need. Fuzzy clustering is proposed to cluster risk factors, which are

closely correlated to each other and are likely to have common source of problems to enable

their effective mitigation. However, quantifying (preferably in monetary terms) various risk

sources and improvements required for their effective mitigation can be some of the potential

directions in which this work can be extended.

Based on preliminary attempt of model validation with a CMM level-5 company it can be

said that uncertain requirements, change in requirements, reactive approach of security,

nonconformance with specifications and ineffective utilization of available resources are

some of the major risk sources for this enterprise. Software enterprises can apply this new

approach in their project and enterprise risk management to improve efficiency, performance,

profitability and to meet rising enterprise management challenges.

The authors would like to acknowledge Prof. Ashis Bhattacherjee, Department of Mining

Engineering, Indian Institute of Technology, Kharagpur, and anonymous reviewers for their

invaluable suggestions and encouragement. Thanks are also due to Mr K. Sreenivasa Rao,

1. Object Management Group, http://www.omg.org/gettingstarted/what is uml.htm

2. Rational Software Corporation, http://www.ibm.com/developerworks/rational/library/253.html

(J. L. Cochrane and M. Eeny, eds), University of South Carolina Press, Columbia, SC, USA, pp. 179'

201 (1973).

20. C. Rong, K. Takahashi, and J. Wang, Enterprise waste evaluation using the analytic hierarchy process and

The main objective of this whitepaper is to develop a framework for next

The Web can be considered as world's biggest data source and just textual data

amounts to at least hundreds of terabytes. The growth rate of Web is even more

dramatic with its size doubling every two years. However, the content of the Web

changes very fast and many of the past links and resources become dead while

many more newer one getting added every day. Aside from these newly created

pages, the existing pages are continuously updated. For example, in a study at

Stanford University of over half a million pages over 4 months, it was found that

about 23%

of pages changed daily. In the .com domain 40% of the pages changed daily,

and the half-life of pages is about 10 days (in 10 days half of the pages are gone,

i.e., their URLs are no longer valid) [Arasu et. al.].

Most of the modern science and technological advancement were driven by

latest development in sciences and Information Technology and particularly web

based system is going to play very important role. With technical development of

applied sciences currently facing many limitations and ever increasing

complexities of problems, increasing use of computer to solve problems is only

natural. Today, Scientific and research portals domain covers wide areas from

physical, organic, inorganic to biochemistry, molecular modeling, biology, drug

design, Geosciences, Applied engineering, and many more. The scientific

community is globally distributed with culture of sharing and rapid dissemination

of information. Each separate area of science generates its own data and

information sources.

A large amount of scientific data is distributed over the Internet (for example: The

Cambridge Crystallographic Data Center, NIST, The Protein Data Bank .

Typically, these information is accessible only through custom web based query

interfaces. Each of these sources and databases have different structures,

contents, query languages and retrieved data in different format. Furthermore,

they are prone to having their interfaces and formats updated without warning

[Buttler et. al.]. To facilitate scientist, students and industrial users, a large

number of specialist interrogation, modeling, and software analysis tools are

available (for example: GAMESS, Gaussian, GAP, Ghemical, COLUMBUS,

COSMO-RS etc).

When scientific resource users require information from multiple sources, they

must pose the appropriate queries at each source individually then explicitly

integrate the result. This solution may be acceptable for a small number of

sources, but it quickly becomes an overwhelming burden for users as the number

of sources grow [Buttler et. al.]. Currently there are thousands of scientific

resources and databases, making it infeasible to manually gather required and

relevant data from these sources. Our proposed system aims to provide a user

interface where they can enter their query, then it should be able to perform the

following query formulation and execution tasks :

Many researchers have tackled problems related to information extraction and integration

from the Web. These go from developing toolkits to add in building wrappers manually

and wrapper induction to the extraction of relational data from large collections of web

documents or extraction of symbolic knowledge. Some of wrapper construction

methods are manual, while others are semiautomatic and automatic. However manually

coding of wrappers become entirely impossible in current scenarios. Methodologies

employed to develop wrappers vary widely from finding pattern in HTML pages using

tree structure to finite state based approach to fuzzy set, artificial intelligence, and neural

networks based learning and training approach. Some of the well known research groups

and products in these areas are: ANDES, WysiWyg Web Wrapper Factory (W4F),

Ariadne, Garlic, TSIMMIS, XWRAP, Mostrare Project, STALKER, TAMBIS,

SoftMealy, FASTUS, HLRT Wrappers, Jedi etc.

XWRAP [Liu et. al.]: XWRAP is a semi-automatic XML-enabled wrapper construction

system for Web sources. The architecture of XWRAP consists of four components:

Syntactical structure normalization, Information extraction, Code generation, Program

Testing and packaging. XWRAP was developed in Java. By XML-enabling, it means that

the wrapper programs generated by XWRAP can transform an HTML document into an

XML document and deliver the extracted data content in XML format with a DTD.

STALKER [Muslea et. al.]: STALKER, developed in University of Southern California,

is a wrapper induction algorithm that generates extraction rules for semi-structured Web

based information sources using landmark automata. Based on just a few training

examples STALKER learns extraction rules for documents with multiple level of

embedding.

InfoSleuth [Bayardo et. al.]: The InfoSleuth project at MCC exploit and synthesize

new technologies into a unified system that retrieves and processes information

in an ever changing network of information resources. This is scalable and

portable and accomplished through the use of collaborative agents, and it uses

Java as a common wrapper agent.

TAMBIS [Baker et. al.]: The TAMBIS project at University of Manchester, UK, is

a three layer madiator/wrapper architecture which aims to provide transparent

access to various disparate biological databases and analysis tools. The use of

knowledge base and wrapped resources removes the need for user to know

which are the appropriate resources and how to access them. It greatly reduces

time taken to analyze their data. TAMBIS aims to use CORBA wrapped services.

Garlic: Developed at IBM Almaden Research Center, Garlic is a middleware

system that provides an integrated view of heterogeneous legacy data without

changing how or where data is stored. It provides a unified schema and common

interface for new applications without disturbing existing applications. This relies

on wrappers that encapsulate the underlying data and mediate between data

source and middleware.

Other projects which specifically aims at diverse and heterogeneous databases

are SINGAPORE, TSIMMIS, DISCO etc.

Problems related with crawling hidden Web and developing search engine were

addressed by Raghvan et. al., Brin et. al. among others.

Methodology to be adopted for this study is to develop a web crawler specific to

scientific areas which able to crawl on the Web for available database resources. For

start we do not propose to crawl all the Web resources but try to stick to four or five

sources. But as most of the databases available are hidden and they have their own

data retrieval mechanism and user interfaces, we need to develop a crawler taking

into account all these factors. Then based on these sources we try to cluster

information into one based on their structural similarity. As each of these databases

have their own format but closely related one as each of them have data about

molecular structure of chemical compounds, so We propose a generic wrapper

CHEMWRAP which able to filter required information from HTML pages covert

them into a common format (say XML) and extract the required information and

convert them into format supported by some scientific software program. We will

develop whole our system in Java, XML, COM/CORBA and other Java based Web

technologies.

Decomposition of Web Information extraction task: The Wrapper generation

process is so complex that it is not possible to consider the construction process

occurring in a one single step. For this reason we have partition the CHEMWRAP

construction process into six phases (Figure 2). The interaction and information

exchange between any two of the phases needs to be performed . After the

preprocessing of sources, information extraction is started. The main task of the

information extraction component is to explore and specify the structure of the

retrieved document (page object) in a declarative extraction rule language. For an

HTML document, the information extraction phase takes as input a parse tree

generated by the syntactical normalizer. It first interacts with the user to identify the

semantic tokens (a group of syntactic tokens that logically belong together) and the

important hierarchical structure. Then it annotates the tree nodes with semantic tokens

in comma-delimited format and nesting hierarchy in context-free grammar. More

concretely, the information extraction process involves three steps; each step

generates a set of extractions rules to be used by the code generation phase to

generate wrapper program code.

Step 1: Identifying region of interest on the Page

Step 2: Identifying Semantics token of interest on the page

Step 3: Determining the nesting hierarchy for the content presentation of the page

Scientific Portal

Client 1 Client 2 Client 3

HTTP

request

Cambridge databank NIST Databank Protein databank

Wrapper

Wrapper

wrapper

CHEMWRAP

Extracted Data

Data Integrator

XML form Data

Some Scientific software Format

Data

Result calculation by scientific

software

HTTP Query Building

Enter asset of URL (s)

Fetching and Repairing Source document

Clustering Pages of Same Structure if

needed

Required Source Document

Generating a Parse Tree

Information Extraction

XML-enabled Wrapper Code Generator

Code Testing and Integration With

ChemCraft

Wrapper Code

Extraction rule

CHEMW

RAP

Niraj Kumar

Traditional computer architecture and integration mechanism are more biased towards

tightly coupled client-server architecture and centralized databases. However with

phenomenal growth in web technologies and emergence of Web as world biggest

database has pushed human and organizations ability to utilize these resources effectively

to limits. Computer scientists, researchers and organizations throughout the world trying

to develop mechanism to make effective utilization of these distributed and

heterogeneous resources to gain competitive advantage in market. In this study, we

propose to develop a grid-enabled web services through web based interfaces, which may

considerably enhance our ability to share distributed and heterogeneous resources and

services among the institutions and organizations throughout the world.

To fulfill challenges posed by competitive world there is need to develop a system, which

can Integrate various distributed databases and web resources on the WWW and bring

these heterogeneous sources of data into common platform or in the form required by the

user. Also there is need to provide mechanism to allow sharing of resources and services

among various Organization / Institutions. Clients through portal interfaces should be

able to get required information on real time basis by seamlessly integrating various

resources/services spread across WWW and various Organization/ institutions and hiding

its implementation details from the user. Required system should be highly flexible and

scalable and services should be added and deleted at any time without affecting the

2

performance of the system. This required that Client and Servers must be independent of

each other.

The organizations/ institutions of todays need to share and integrate various resources and

services to provide task specific requirements of a particular users and applications.

However, this is very challenging task because each of these resources and services have

different structures, contents, query languages and retrieved data in different format and

supported by different underlying hardware and network support. Furthermore, they are

prone to having their interfaces and formats updated without warning. Due to increasing

complexities of problems any user specific task typically requires to interact with

number of resources and services distributed over geographically distant locations and

under the control of different agencies. Many of them have security concerns and want to

share only limited resources with others. They also need flexibility in adding or deleting

resources and our proposed system should able to run even if some resources and services

will be no longer available for use or new resources and services get added to the system.

This requires that machines must be able to communicate with one another without much

human intervention.

Our current web applications follow the traditional client-server model of software

architecture and they are closely interrelated with one other, inflexible and tightly

coupled. This makes whole system dysfunctional if any changes is done in client or

server architecture independently. Also, our traditional database and information

integration systems (like Enterprise Resource Planning) are biased towards centralized

system where various resources of the organization are integrated using centralized

databases and mostly with proprietary software. This is contrary to the spirit of the Web

- a loose, open confederation of resources held together by simple protocols. So, to

address these problems many distributed environment technologies and standards like

DCOM, CORBA, RMI etc were came up during late 1990's. However, in reality these

are not suitable for the Internet, and introduce a degree of dependency and/or platform

issues. They are not able to completely eliminate need to write client application without

having to know anything about the architecture of participating distributed objects. In the

meantime advances in Servlet and Java Server Page (JSP) technologies and emergence of

standard like J2EE and .Net from Sun Microsystems and Microsoft respectively made

possible development of fast and relatively secure Web based application within the

realm of reality.

This set the stage for XML based Web services, which is an exciting new technology

standard that enables communication between heterogeneous computer systems. Web

services emerged as standard only in last 3 years. At its core, the technology is simply

XML moving from one computer to another in a form that each computer can reliably

process. It is a significant improvement to traditional systems integrations and it has

significant implication for any organization. Web services facilitates the ability to expand

computer to computer communication. They are developed supporting three primary

3

internet standards - Simple object access protocol (SOAP), Web Services Description

Language (WSDL), and Universal Description, Discovery, and Integration (UDDI)

directory. Currently these standard were supported by all major software technology

vendors and approved by the W3C. Grid based computer applications can be considered

as next level in this chain of events, which make possible even heavy duty task solvable

by using diversified and heterogeneous resources and services.

In this case study, we propose to develop a XML-based Web services and Grid based

solution using various Java based technologies, which can facilitate transfer and sharing

of various resources across the Organizations/ Institutions of the world through Web

based interfaces on the portal. We also want to make it possible to share and transfer not

only light weight resources and services, but also heavy weight resources and services.

Our aim is to develop a full fledged independent software product and methodology,

which can be directly usable to any portals which needs to share and integrate resources

over the Internet or among the enterprises and their partners and can be marketable as an

independent product in its own right.

Imagine a scenario where just with an interface anybody will be able to run any program

without downloading any softwares or all barriers of platforms, databases and

networks vanishes . Grid computing system of the future should able to provide solution

to these problems. It is also expected that autonomic computing and smart network

technologies should emerge which should automatically able to detect changes in the

systems and accordingly able to take appropriate action. It is likely to provide user

friendly interfaces for remote job monitoring and show the status of the result computed

at each nodes in real time. More and more web based interfaces is likely to be added for

all activities related with grid computing. The primary reasons for these are because grid

systems require dynamic discovery and composition of services in heterogeneous

environments necessitates mechanisms for registering and discovering interface

definitions and endpoint implementation descriptions and for dynamically generating

proxies based on (potentially multiple) bindings for specific interfaces. WSDL supports

this requirement by providing a standard mechanism for defining interface definitions

separately from their embodiment within a particular binding (transport protocol and data

encoding format). Second, already numerous tools and support for WSDL processors

that can generate language bindings for a variety of languages and platforms. Third,

using http protocol for communication allows to communicate with any system sitting

behind firewalls as usually firewalls don't likely to block this port. Fourth, because any

computational intensive tasks require to interact with more than one computers at

geographically distributed locations, databases etc which requires algorithms for defining

work flows. Web Services Flow Language (WSFL) and MS XLANG, which is an XML

language to describe workflow processes among distributed and heterogeneous

environment offers excellent potential for this purpose.

4

The web services framework is likely to integrated with the grid computing system of the future

and already attempts in this direction is being made, however it is yet to matured to stage where

We need to develop a mechanism to send and receive communication to remote services

and resources in grid environment. Web services expose objects method via SOAP.

Following steps needed to be followed:

Once the resources are discovered, Work flow in grid can be established using Web

Services Flow Language (WSFL) and MS XLANG, which is an XML language to

describe workflow processes and spawn them. WSFL specifies how a Web Service is

interfaced with another. With it, we can determine whether the Web Services should be

treated as an activity in one workflow or as a series of activities. While WSFL

complements WSDL (Web Services Definition Language) and is transition-based,

XLANG is an extension of WSDL and block-structured based. WSFL supports two

model types: flow and global models. The flow model describes business processes that a

collection of Web Services needs to achieve. The global model describes how Web

Services interact with one another. XLANG, on the other hand, allows orchestration of

Web Services into business processes and composite Web Services. WSFL is strong on

model presentation while XLANG does well with the long-running interaction of Web

Services. Web Services and resources can be declared as private or public.

In a complex system like the grid, monitoring is essential for understanding its operation,

debugging, failure detection and for performance optimization. The monitoring system

must be able to provide information about the current state of various grid entities, such

as grid resources and running jobs, as well as to provide notifications when certain events

(e.g. system failures, performance problems) occur. Monitoring jobs require

interoperation between the monitoring system and other grid services. The running

application consists of processes running on hosts constituting the grid resource.

Processes are identified locally by the operating system by process identifiers

(PIDs). The local resource management system (LRMS) controls jobs running on hosts

belonging to a grid resource. It allocates hosts to jobs, starts and stops jobs on user

request and possibly restarts jobs in case of an error. It may also checkpoint and migrate

jobs between hosts of the resource which can be considered as a special case of job

startup. The LRMS identifies the job it manages by a local job identifier (LJID). To

monitor a job the monitoring system has to know the relation between LJIDs and PIDs.

There are various ways to accomplice this task and each grid system implements this in

different ways. The future generation grid systems should provide job submission, job

monitoring, job status and job output through web based interfaces to make it accessible

to common man.

6

Our architecture focuses on providing virtual integrated environment among

organizations/institutions through web based interfaces, which are easy to use, flexible

and secure. It provides mechanism to share large number of resources like video

lectures on real time and on demand, scientific databases, query to partner institutions,

lecture notes etc though our primary focus is on providing computation resources for

computation intensive scientific and engineering tasks. Our architecture aims to provide

support for platform independent and heterogeneous resources.

In the beginning our focus is on providing support for Java, C++/C and Fortan languages.

For scripting we intends to provide support for PERL, Shell Script, Dos Script. As far as

operating system is concerned we intend to provide support for windows and Unix based

platforms. To make this possible our architecture is XML based and intend to combine

web services concepts with grid computing concepts. Our architecture is based on

autonomic computing concepts and also intended to integrate intelligent network

technology concepts like Jini Technology with grid computing to make it possible for us

to able to dynamically sense changes in network environment and system should able to

take appropriate action automatically. Our architecture is capable of taking into

consideration any number of systems and services added/removed from the system in

real time. Besides it is capable of displaying status of jobs, output of the jobs at each

nodes in real time.

The job is submitted by the client to the web portal through a graphical user interface

(GUI). The web portal delegates the management of jobs to schedulers. A scheduler

divides a job into smaller tasks (in the case of an independent job, a task refers to the

subset of parameters that can be executed independently) and sends the tasks to the

resources for execution. Ease of use is achieved by encapsulating the system with an easy

and a well defined interface. The execution service provided by the resources is wrapped

inside a web service interface, hence it can be consumed easily by any user. The

scheduler encapsulates the complexity of the job scheduling into a web service interface.

This approach of using web service interface allows easy client side implementation. The

web portal provides GUIs for job submission and management, hence allowing the client

to submit and monitor jobs easily. The parameter file can contain a range or a list of input

values. The scheduler parses the parameter file and splits the input values so it can

distribute the job to many resource machines with different range or list of input values,

which is a subset of the submitted input values.

The status of the each subparts of the job submitted to each nodes is displayed through

web based interface at real time. It is also proposed to display the status about

performance of each nodes with respect to particular job to the respective clients through

web based interface. It will also display the available memory at each nodes, current load

at each nodes, average CPU performance of each nodes. After computation is complete

the final result is generated and displayed in real time basis through web based interface

and automatically an event is generated to send output to the clients through email or to

the computer directory of the client.

7

Communication Protocols(like HTTP/ HTTPS etc)

Information flow

Between two institutions

(XML over HTTP/

XML over FTP/

XML over HTTPS)

Data Data

Web Server and

Application Server

Database Server

Web Server and

Application Server

Database Server

WWW

Web Portal

Other resources

(Computers, Video

Lectures, Files etc

Other resources

(Computers, Video

Lectures, Files etc)

8

Result

Find the Appropriate Nodes

Transfer the Data

Submit the Jobs

Collect the result

Web Portal

Client 1 Client 2 Client 3

HTTP

request

Authenticate with the

Server

Connect to the Server

Transfer the File

(XML over FTP)

Close the Connection

Integrate the result

9

Design methodology to be adopted for this study is in stages. Each stages can be

considered as separate software modules. Primary challenges when designing Grid-based

Web services are to look beyond the traditional Client- Server paradigm, where client is

tightly coupled with server. Here we have to design client completely independent of

server, so that even if some changes take place in the server side (In this case which is not

in our control as many services can be withdrawn, while many new services can be added

without our knowledge). Then on server side, this system must able to give choices to the

other partner institutions regarding type of services and resources they want to share and

at any point of time able to withdrawn as well as add new services/resources. So this

kind of system design should be based on following design principles:

In this stage we propose to design user interfaces keeping into view that clients should

able to minimize the number of pages needed to be viewed and this should be

independent from the available services and resources.

10

In this stage we have developed elementary and small grid computing systems for

elementary computations. We have evaluated various available grid computing systems

like Globus, Unicore, Candore, JCGrid, Jgrid, Optimal Grid etc. We also propose to test

all these grid systems by running sample jobs among number of windows and Unix based

environment. Idea is just to make a suitable decision about which of these grid based

systems are based suitable for our purpose. Here we need to take into account computing

facilities available or likely to be available in the future for this purpose, Which kinds of

operating systems all these runs, technological level of people involved in this process

and their commitment, type of network facilities available and security, firewalls and

other administrative decisions about how to allow access to our facilities to the clients as

well as partner institutions. Also we need to take into consideration all these

environments of our partner institutions and general state of affairs and understanding

about grid based system in the country, its current and future potential, and requirements

for such a system .

In this stage we also propose to develop web based interfaces for this kind of system for

job submission, job monitoring, output condition monitoring, output displayed specific

for the grid system we plan to adopt considering our suitability and capabilities. Once this

interface come up user should start the server and submit their jobs . Similar interfaces

will be provided to the partner institutions with the difference that no public interfaces

should be provided but only to transfer their resources to IIITMK or to submit the

services/resources to the registry.

In this stage we implement the logic of web interface implementation using web services

concepts and suitable grid system. We also customize that system according to our

requirements and feasibility. We also combine computing power of various PCs in

windows and Unix platforms available. Make it fully operational and providing facilities

for remote jobs and status monitoring. Provide single web interface to submit any

computational intensive jobs. We also provide support for various available languages

and scripts as well as integrate the whole system with partner institutions and industry.

We also implement autonomic grid computing concepts and intelligent network concepts

with suitable technologies to see that our system should able to withstand the requirement

of future generation grid systems. We also optimize our whole system to get best possible

throughput and CPU utilization of available nodes for the purpose in geographically

distributed locations. We also aim to enhance our capabilities to apply web services

concepts for grid computing purpose. We also train people and students about these

technologies and systems and considerably enhance our abilities to use this system with

maturity.

In this stage we develop web based interfaces for many other grid computing systems as

well as continuously improve and upgrade our system as newer technologies for these

11

will be developed. In this stage we also try to make grid computing technologies

available to common man which requires very less technical knowledge of computers.

As future systems are likely to develop monitors of different kinds and capabilities in

terms of memory requirements, CPU capabilities, we aim to provide grid facilities

through all those devices. We also look into possibilities of developing cutting edge

technology and products in these areas as well as developing some algorithms for

defining work flows, job monitoring etc in the distributed environment. We also try to

look into developing the possibility of system where any body without downloading

softwares can able to use these softwares by sending appropriate program to them.

Summary and Conclusion

In this study, we have presented a framework for sharing distributed and heterogeneous

resources and services among the organizations/institutions. We have also presented

various modules of our framework. We have overviewed some current grid system

available and their usefulness through sample case studies taking into account strength

and weakness of the organizations. We have implemented sample SOAP, AXIS and Jini

based web services and made it available through our web based interfaces. We have

empathizes the importance of combining autonomic, intelligent networks, and web

services concepts with current grid computing systems to make it more effective,

efficient and reachable to the common man.

In summary, a simple grid computing system combining the power of Web Services,

autonomic computing, and intelligent network concepts with capabilities of combining

the computing power of twenty ' thirty computers of different platforms through web

based interfaces, which may be geographically distributed and hiding the complexities

of the implementations from its user, is recommended as grid computing system for the

future.

In this section we try to point out potential challenges and future research directions of

this study in stages.

(1) IBM Developer Works

http://www-128.ibm.com/developerworks/webservices

(2) IBM OptimalGrid

http://www.alphaworks.ibm.com/tech/optimalgrid

(3) IBM TSpaces

http://www.almaden.ibm.com/cs/TSpaces

(4) JCGrid Website

http://jcgrid.sourceforge.net/

(5) PovRay Website

http://www.povray.org/

(6) Foster Ian, "What is Grid? A Three Point Checklist", Argonne National Laboratory &

University of Chicago, 2002, PP: 1- 4 .

(7) Foster Ian, Kesselman Carl, Nick J., Tuecke, "The Physiology of the Grid - An Open

Grid Services Architecture for Distributed Systems Integration ", OGSA draft documents,

Version: 6/22/2002, PP: 1-31.

(8) Balaton Zoltan, Gombas Gabor, "Resource and Job Monitoring in the Grid", MTA

SZTAKI Computer and Automation Research Institute 2003, PP: 1-8.

13

(9) Tantra J. W., Thu M. M., Heng F. C., "A Framework for secure execution of java

jobs in grid computing", Executive Summary, 2004, PP: 1-7.

(10)Jgrid Website

www.gini.org

(12) Sun MicroSystem Web Services

http://java.sun.com/webservices

(13) Apache Website

http://apache.org

(14) Globus Grid Computing Site

http://www.globus.org/

(15) Unicore Grid Computing Site

www.unicore.org

(16) Gridbus, Australia Site

www.gridbus.org

(17) MIT Thesis Web Site

http://theses.mit.edu/

(18) MIT Open Course Site

http://ocw.mit.edu

(19) Java World Site

www.javaworld.com

(20) Apache Tomcat Site

http://tomcat.apache.org

(21) Apache Axis Site

http://ws.apache.org/axis/

(22) Apache Soap Site

http://ws.apache.org/soap/

(23) VideoLAN Project site

http://www.videolan.org/

(24) Streaming Media World Site

http://streamingmediaworld.com

14

(25) OpenSSH Website

http://www.openssh.com/

(26) Marty Hall CoreServlet Site

www.coreservlets.com

(27)W3 School Website

http://www.w3schools.com

(28) IIT Kharagpur E-library Site

http://www.library.iitkgp.ernet.in/

(29) IIT Kanpur Online Thesis Site

http://www.iitk.ac.in/

(30)Computational Chemistry portal, IIITMK

http://comchem.in

(31) W3 consortium Site

http://www.w3.org/

(32) Professional XML, Wrox Press, 2000, PP: 797-835.

(33)Microsoft Website

http://www.microsoft.com/

(34) Kumar Niraj, Srivathsan K. R., "Enterprise risk evaluation and continuous

mitigation using the Fuzzy-Multi-attribute decision making ' A conceptual approach",

Under review by IISc Journal, 2005, pp-1-25.

(35) Kumar N., Bhattacherjee A. and Sarkar D., " Performance appraisal of coal mines

using Data Envelopment Analysis and Fuzzy Set Theory", Mintech, 2002, Volume 23,

No. 5, pp. 18-25.

(36) Kumar N., Bhattacherjee A., Chakravarty D. and Sarkar D., “Efficiency

measurement of mines using DEA and AHP”, TAMSEM, I.I.T. Kharagpur, February,

2004.

(37) Biomer Website

http://www.es.embnet.org/Services/MolBio/B/

(38)Reddy J. N, “An Introduction to Finite Element Method”, McGRAW-HILL

International Editions, 1993.

(39) Condore Grid Computing Site

http://www.cs.wisc.edu/condor

15