A

A

DAPT

DAPT

IST-2001-37126

IST-2001-37126

Clustering Implementation in JBoss

Clustering Implementation in JBoss

Jakša Vučković

Jakša Vučković

Università di Bologna

Università di Bologna

Clustering Implementation in JBoss – Jakša Vučković, Università di
Bologna

2

A

DAPT

Clustering J2EE

Clustering J2EE

•

Replicating Resources

Replicating Resources

•

Communication

Communication

•

Concurrency Control

Concurrency Control

•

Failure Handling

Failure Handling

•

Controlling Access to External Resources

Controlling Access to External Resources

Clustering Implementation in JBoss – Jakša Vučković, Università di
Bologna

3

A

DAPT

Replicating Resources

Replicating Resources

•

Stateful Session Bean Replication

Stateful Session Bean Replication

•

Entity Bean Replication

Entity Bean Replication

•

HTTP Session Replication

HTTP Session Replication

•

JNDI Tree Replication

JNDI Tree Replication

Clustering Implementation in JBoss – Jakša Vučković, Università di
Bologna

4

A

DAPT

Sticky Sessions and Homogeneous

Sticky Sessions and Homogeneous

Deployment

Deployment

•

JBoss’ staff recommends the use of sticky

JBoss’ staff recommends the use of sticky

sessions and homogeneous deployment

sessions and homogeneous deployment

–

Using sticky sessions means that all requests of a

Using sticky sessions means that all requests of a

client arrive to the same node during a session

client arrive to the same node during a session

–

Homogeneous deployment means that each

Homogeneous deployment means that each

application component is replicated an all the

application component is replicated an all the

nodes, so requests don’t need to span on

nodes, so requests don’t need to span on

different nodes

different nodes

•

Sticky sessions + homogeneous deployment

Sticky sessions + homogeneous deployment

imply that each request will be entirely

imply that each request will be entirely

processed on one node

processed on one node

Clustering Implementation in JBoss – Jakša Vučković, Università di
Bologna

5

A

DAPT

Stateful Session Beans

Stateful Session Beans

•

Each instance of a SFSB is associated to a

Each instance of a SFSB is associated to a

client

client

•

The lifetime of a SFSB is determined by a

The lifetime of a SFSB is determined by a

timeout from the last client access

timeout from the last client access

•

EJBs cannot generate threads

EJBs cannot generate threads

•

All access to a SFSB instance should be

All access to a SFSB instance should be

serialized

serialized

•

SFSBs are non-reentrant

SFSBs are non-reentrant

Clustering Implementation in JBoss – Jakša Vučković, Università di
Bologna

6

A

DAPT

SFSB Replication

SFSB Replication

•

After each invocation of a SFSB, if the bean

After each invocation of a SFSB, if the bean

changed, a message is broadcast to the

changed, a message is broadcast to the

cluster

cluster

•

The message contains the serialized state

The message contains the serialized state

of the session bean

of the session bean

•

Before broadcasting the message a

Before broadcasting the message a

distributed lock is acquired

distributed lock is acquired

•

The broadcast operation is synchronous

The broadcast operation is synchronous

(blocking) with acknowledgements from all

(blocking) with acknowledgements from all

the nodes of the cluster

the nodes of the cluster

Clustering Implementation in JBoss – Jakša Vučković, Università di
Bologna

7

A

DAPT

RMI Clients

RMI Clients

•

The clients downloads an “intelligent” stub

The clients downloads an “intelligent” stub

from the server and uses it to access the

from the server and uses it to access the

bean

bean

•

The stub encapsulates the load balancing

The stub encapsulates the load balancing

policy and the failover mechanism

policy and the failover mechanism

•

The stubs maintains a list of server

The stubs maintains a list of server

addresses which is updated on each

addresses which is updated on each

request

request

Clustering Implementation in JBoss – Jakša Vučković, Università di
Bologna

9

A

DAPT

RMI Clients - Failover

RMI Clients - Failover

•

If a call to a node fails the client stub fails

If a call to a node fails the client stub fails

over to another server or raises an exception

over to another server or raises an exception

to the client application

to the client application

•

The stub fails over to another server when:

The stub fails over to another server when:

–

there is a communication exception

there is a communication exception

–

The server does not responds because it crashed

The server does not responds because it crashed

or a network partition occurred

or a network partition occurred

•

The stub reports an exception when:

The stub reports an exception when:

–

There are no more servers to fail over to

There are no more servers to fail over to

–

The server reported a GenericClusteringException

The server reported a GenericClusteringException

with the flag COMPLETED_YES or

with the flag COMPLETED_YES or

COMPLETED_MAYBE

COMPLETED_MAYBE

Clustering Implementation in JBoss – Jakša Vučković, Università di
Bologna

10

A

DAPT

RMI Clients - Failover

RMI Clients - Failover

•

If a client makes a request to a node and does

If a client makes a request to a node and does

not receive the response four scenarios are

not receive the response four scenarios are

possible:

possible:

1.

1.

The node crashed before receiving the request –

The node crashed before receiving the request –

failover safe

failover safe

2.

2.

The node received the request but crashed before

The node received the request but crashed before

broadcasting the state – failover safe

broadcasting the state – failover safe

3.

3.

The node received and processed the request,

The node received and processed the request,

broadcast the state and then crashed before

broadcast the state and then crashed before

sending the reply – the failover node must

sending the reply – the failover node must

recognize a duplicate request

recognize a duplicate request

•

The client indicates in the request message if

The client indicates in the request message if

it is a failover or the original request

it is a failover or the original request

Clustering Implementation in JBoss – Jakša Vučković, Università di
Bologna

11

A

DAPT

Partial Replication Problem

Partial Replication Problem

•

We have two beans S1 and S2 on two

We have two beans S1 and S2 on two

nodes A and B

nodes A and B

Client

S1

S2

S2

node A

node B

S1

S2

S2

S1

Clustering Implementation in JBoss – Jakša Vučković, Università di
Bologna

12

A

DAPT

Partial Replication Problem

Partial Replication Problem

•

The client invokes S1 on node A

The client invokes S1 on node A

Client

S1

S2

S2

node A

node B

S1

S2

S2

S1

Clustering Implementation in JBoss – Jakša Vučković, Università di
Bologna

13

A

DAPT

Partial Replication Problem

Partial Replication Problem

•

S1 invokes S2

S1 invokes S2

Client

S1

S2

S2

node A

node B

S1

S2

S2

S1

Clustering Implementation in JBoss – Jakša Vučković, Università di
Bologna

14

A

DAPT

Partial Replication Problem

Partial Replication Problem

•

Invocation of S2 is completed and its state

Invocation of S2 is completed and its state

is replicated on node B

is replicated on node B

Client

S1

S2

S2

node A

node B

S1

S2

S2

S1

Clustering Implementation in JBoss – Jakša Vučković, Università di
Bologna

15

A

DAPT

Partial Replication Problem

Partial Replication Problem

•

S1 tries to make another call to S2 but

S1 tries to make another call to S2 but

node A crashes

node A crashes

Client

S1

S2

S2

node A

node B

S1

S2

S2

S1

Clustering Implementation in JBoss – Jakša Vučković, Università di
Bologna

16

A

DAPT

Partial Replication Problem

Partial Replication Problem

•

Now we have an inconsistent state in node

Now we have an inconsistent state in node

B

B

Client

S1

S2

node B

Clustering Implementation in JBoss – Jakša Vučković, Università di
Bologna

17

A

DAPT

Partial Replication Problem

Partial Replication Problem

•

There are two possible solutions to this

There are two possible solutions to this

problem

problem

:

:

–

A replication aware Transaction Manager would

A replication aware Transaction Manager would

need to roll back the state of S2 in B also

need to roll back the state of S2 in B also

–

The updated state is broadcast to other nodes

The updated state is broadcast to other nodes

only when committing the whole transaction

only when committing the whole transaction

Clustering Implementation in JBoss – Jakša Vučković, Università di
Bologna

18

A

DAPT

Network Partitions

Network Partitions

•

A network failure could divide the cluster

A network failure could divide the cluster

in two or more partitions that cannot

in two or more partitions that cannot

communicate

communicate

Node A

Node C

Node B

Node D

No shared
data
(Entity
Beans,
DB) is
accessed!!
!

Clustering Implementation in JBoss – Jakša Vučković, Università di
Bologna

19

A

DAPT

Network Partitions

Network Partitions

•

The client stub maintains a list of available

The client stub maintains a list of available

servers

servers

•

If a client is not able to contact the server

If a client is not able to contact the server

instance it is bound to, it tries to contact

instance it is bound to, it tries to contact

the next from the list until it reaches an

the next from the list until it reaches an

instance that is its own partition

instance that is its own partition

Node A

Node C

Node B

Node D

client

No shared
data
(Entity
Beans,
DB) is
accessed!!
!

Clustering Implementation in JBoss – Jakša Vučković, Università di
Bologna

20

A

DAPT

Network Partitions

Network Partitions

•

The client stub maintains a list of available

The client stub maintains a list of available

servers

servers

•

If a client is not able to contact the server

If a client is not able to contact the server

instance it is bound to, it tries to contact

instance it is bound to, it tries to contact

the next from the list until it reaches an

the next from the list until it reaches an

instance that is its own partition

instance that is its own partition

Node A

Node C

Node B

Node D

client

No shared
data
(Entity
Beans,
DB) is
accessed!!
!

Clustering Implementation in JBoss – Jakša Vučković, Università di
Bologna

21

A

DAPT

Network Partitions

Network Partitions

•

The client stub maintains a list of available

The client stub maintains a list of available

servers

servers

•

If a client is not able to contact the server

If a client is not able to contact the server

instance it is bound to, it tries to contact

instance it is bound to, it tries to contact

the next from the list until it reaches an

the next from the list until it reaches an

instance that is its own partition

instance that is its own partition

Node A

Node C

Node B

Node D

client

No shared
data
(Entity
Beans,
DB) is
accessed!!
!

Clustering Implementation in JBoss – Jakša Vučković, Università di
Bologna

22

A

DAPT

Network Partitions

Network Partitions

•

Each SFSB is associated to a client, so

Each SFSB is associated to a client, so

there will be no other clients accessing the

there will be no other clients accessing the

instance on another partition

instance on another partition

Client

2

Client

1

s2

s3

s1

s2

s3

s1

s2

s3

s1

s2

s3

s1

A

B

C

D

No shared
data
(Entity
Beans,
DB) is
accessed!!
!

Clustering Implementation in JBoss – Jakša Vučković, Università di
Bologna

23

A

DAPT

Network Partitions

Network Partitions

•

Each clients list of available servers will be

Each clients list of available servers will be

updated to contain only the nodes on its

updated to contain only the nodes on its

partition

partition

Client

2

Client

1

s2

s3

s1

s2

s3

s1

s2

s3

s1

s2

s3

s1

A

B

C

D

No shared
data
(Entity
Beans,
DB) is
accessed!!
!

Clustering Implementation in JBoss – Jakša Vučković, Università di
Bologna

24

A

DAPT

Network Partitions Merging

Network Partitions Merging

•

When the cluster merges the clients will

When the cluster merges the clients will

still have the old list of nodes

still have the old list of nodes

Client

2

Client

1

s2

s3

s1

s2

s3

s1

s2

s3

s1

s2

s3

s1

A

B

C

D

No shared
data
(Entity
Beans,
DB) is
accessed!!
!

Clustering Implementation in JBoss – Jakša Vučković, Università di
Bologna

25

A

DAPT

Network Partitions Merging

Network Partitions Merging

•

Only when the client makes the next

Only when the client makes the next

request, the bean gets replicated on the

request, the bean gets replicated on the

other nodes and the client’s list is updated

other nodes and the client’s list is updated

Client

2

Client

1

s2

s3

s1

s2

s3

s1

s2

s3

s1

s2

s3

s1

A

B

C

D

No shared
data
(Entity
Beans,
DB) is
accessed!!
!

Clustering Implementation in JBoss – Jakša Vučković, Università di
Bologna

26

A

DAPT

Network Partitions Merging

Network Partitions Merging

•

S3 is not updated so the client must

S3 is not updated so the client must

maintain a separate list of nodes for each

maintain a separate list of nodes for each

SFSB

SFSB

Client

2

Client

1

s2

s3

s1

s2

s3

s1

s2

s3

s1

s2

s3

s1

A

B

C

D

No shared
data
(Entity
Beans,
DB) is
accessed!!
!

Clustering Implementation in JBoss – Jakša Vučković, Università di
Bologna

27

A

DAPT

Network Partitions Merging

Network Partitions Merging

Problem

Problem

•

Suppose node C crashes now

Suppose node C crashes now

Client

2

Client

1

s2

s3

s1

s2

s3

s1

s2

s3

s1

A

B

D

No shared
data
(Entity
Beans,
DB) is
accessed!!
!

Clustering Implementation in JBoss – Jakša Vučković, Università di
Bologna

28

A

DAPT

Network Partitions Merging

Network Partitions Merging

Problem

Problem

•

Client 2 wants to invoke s2 and fails over

Client 2 wants to invoke s2 and fails over

to node A

to node A

Client

2

Client

1

s2

s3

s1

s2

s3

s1

s2

s3

s1

A

B

D

No shared
data
(Entity
Beans,
DB) is
accessed!!
!

Clustering Implementation in JBoss – Jakša Vučković, Università di
Bologna

29

A

DAPT

Network Partitions Merging

Network Partitions Merging

Problem

Problem

•

If s2 invokes s3 we find an “out of date”

If s2 invokes s3 we find an “out of date”

state on this node

state on this node

Client

2

Client

1

s2

s3

s1

s2

s3

s1

s2

s3

s1

A

B

D

No shared
data
(Entity
Beans,
DB) is
accessed!!
!

Clustering Implementation in JBoss – Jakša Vučković, Università di
Bologna

30

A

DAPT

Entity Beans

Entity Beans

•

Entity Beans are persistent

Entity Beans are persistent

•

Unlike Session Beans, Entity Beans are not

Unlike Session Beans, Entity Beans are not

associated to clients

associated to clients

•

Can be accessed by multiple clients, but

Can be accessed by multiple clients, but

not concurrently

not concurrently

•

If an Entity Bean is declared as reentrant it

If an Entity Bean is declared as reentrant it

can be accessed concurrently but only from

can be accessed concurrently but only from

the same transaction

the same transaction

Clustering Implementation in JBoss – Jakša Vučković, Università di
Bologna

31

A

DAPT

Entity Beans

Entity Beans

•

JBoss does not replicate Entity Beans,

JBoss does not replicate Entity Beans,

replication is left to the database tier

replication is left to the database tier

•

JBoss controls concurrent access to the

JBoss controls concurrent access to the

database

database

•

All JBoss nodes share the same database

All JBoss nodes share the same database

•

It is transparent to JBoss if the database is

It is transparent to JBoss if the database is

clustered or not

clustered or not

Clustering Implementation in JBoss – Jakša Vučković, Università di
Bologna

32

A

DAPT

Entity Beans

Entity Beans

•

Entity Bean state is loaded from the DB

Entity Bean state is loaded from the DB

before each business method invocation

before each business method invocation

and stored after the invocation (Commit

and stored after the invocation (Commit

option B and C)

option B and C)

•

To each J2EE transaction is associated a

To each J2EE transaction is associated a

DB transaction

DB transaction

•

There are to policies to deal with

There are to policies to deal with

concurrency:

concurrency:

–

Optimistic

Optimistic

–

Pessimistic

Pessimistic

Clustering Implementation in JBoss – Jakša Vučković, Università di
Bologna

33

A

DAPT

Optimistic Policy

Optimistic Policy

•

Transactions are executed concurrently

Transactions are executed concurrently

•

Each Transaction has it’s own instance of

Each Transaction has it’s own instance of

the Entity Bean

the Entity Bean

•

Shared row locking is used in the database

Shared row locking is used in the database

•

Before committing the container verifies if

Before committing the container verifies if

the Bean changed in the database

the Bean changed in the database

–

If yes the transaction is rolled back and re-

If yes the transaction is rolled back and re-

executed

executed

–

If not the new value is stored in the DB

If not the new value is stored in the DB

Clustering Implementation in JBoss – Jakša Vučković, Università di
Bologna

34

A

DAPT

Pessimistic Policy

Pessimistic Policy

•

Transactions are not executed concurrently

Transactions are not executed concurrently

•

Exclusive Row locking at the DB is used

Exclusive Row locking at the DB is used

•

A transaction blocks if it tries to access a

A transaction blocks if it tries to access a

locked row until the row is released

locked row until the row is released

•

A Distributed Deadlock Detection

A Distributed Deadlock Detection

algorithm is used to detect Deadlocks

algorithm is used to detect Deadlocks

Clustering Implementation in JBoss – Jakša Vučković, Università di
Bologna

35

A

DAPT

Entity Bean Replication Failures

Entity Bean Replication Failures

•

If the node owner of a transaction crashes

If the node owner of a transaction crashes

the DB rolls back the transaction

the DB rolls back the transaction

•

Network partitions must be handled by the

Network partitions must be handled by the

DB clustering mechanism

DB clustering mechanism

•

With a non replicated database, the

With a non replicated database, the

approach of a primary partition is used

approach of a primary partition is used

where the primary partition is defined to

where the primary partition is defined to

be the one containing the database

be the one containing the database

Clustering Implementation in JBoss – Jakša Vučković, Università di
Bologna

36

A

DAPT

Entity Bean Replication

Entity Bean Replication

•

Relies intensively on DB

Relies intensively on DB

–

Maximizes DB I/O

Maximizes DB I/O

•

If we don’t want a single point of failure,

If we don’t want a single point of failure,

we must use a replicated database

we must use a replicated database

•

Entity Beans are not used as cache

Entity Beans are not used as cache

•

A distributed cache with distributed locks

A distributed cache with distributed locks

is under development

is under development

Clustering Implementation in JBoss – Jakša Vučković, Università di
Bologna

38

A

DAPT

HTTP Session Replication

HTTP Session Replication

•

An HTTP Session is implemented as a serializable

An HTTP Session is implemented as a serializable

object

object

•

The HTTP Session state is NOT broadcast to the

The HTTP Session state is NOT broadcast to the

other nodes but replication is implemented

other nodes but replication is implemented

through an Entity Bean

through an Entity Bean

•

The HTTP Session is a cmp-field of that bean

The HTTP Session is a cmp-field of that bean

•

The CMP engine serializes the HTTP Session

The CMP engine serializes the HTTP Session

object along with its contents

object along with its contents

•

The HTTP Session contains also references

The HTTP Session contains also references

(remote objects) of SFSBs

(remote objects) of SFSBs

•

SFSBs do not get serialized with the HTTP Session

SFSBs do not get serialized with the HTTP Session

Clustering Implementation in JBoss – Jakša Vučković, Università di
Bologna

39

A

DAPT

HTTP Session Replication

HTTP Session Replication

•

Failures are handled by the EJB Replication

Failures are handled by the EJB Replication

mechanism

mechanism

•

Client fail over behavior depends on the

Client fail over behavior depends on the

HTTP dispatcher

HTTP dispatcher

•

A JBoss HTTP Dispatcher is under

A JBoss HTTP Dispatcher is under

development

development

Clustering Implementation in JBoss – Jakša Vučković, Università di
Bologna

40

A

DAPT

JNDI Tree Replication

JNDI Tree Replication

•

Each node has a local JNDI Tree

Each node has a local JNDI Tree

•

All nodes share a replicated HA-JNDI Tree

All nodes share a replicated HA-JNDI Tree

•

When a client looks up an object on node N

When a client looks up an object on node N

–

The replicated HA-JNDI service is invoked first

The replicated HA-JNDI service is invoked first

–

If it cannot find it there it looks in the local

If it cannot find it there it looks in the local

JNDI Tree

JNDI Tree

–

If it’s not there it asks all other nodes if they

If it’s not there it asks all other nodes if they

have it in their local JNDI Tree

have it in their local JNDI Tree

–

If it hasn’t been found a

If it hasn’t been found a

NameNotFoundException is thrown

NameNotFoundException is thrown

Clustering Implementation in JBoss – Jakša Vučković, Università di
Bologna

41

A

DAPT

JNDI Tree Replication

JNDI Tree Replication

•

Unlike clients, EJBs access only the local

Unlike clients, EJBs access only the local

JNDI Tree

JNDI Tree

•

This solution was chosen because of

This solution was chosen because of

–

Compatibility with existing applications

Compatibility with existing applications

–

Distinction between local and replicated objects

Distinction between local and replicated objects

–

Low network traffic in homogeneous clusters

Low network traffic in homogeneous clusters

Clustering Implementation in JBoss – Jakša Vučković, Università di
Bologna

42

A

DAPT

Distributed Cache

Distributed Cache

•

Deadline for June (JavaONE)

Deadline for June (JavaONE)

•

Based on JavaGroups

Based on JavaGroups

•

Will be a Jboss Service

Will be a Jboss Service

•

Highly Configurable

Highly Configurable

•

Will be applied to Entity Bean Clustering

Will be applied to Entity Bean Clustering

and Session Bean Clustering

and Session Bean Clustering

•

Will enable replication of the Database

Will enable replication of the Database

Clustering Implementation in JBoss – Jakša Vučković, Università di
Bologna

43

A

DAPT

Distributed Cache Semantics

Distributed Cache Semantics

•

Three semantics:

Three semantics:

–

Asynchronous

Asynchronous

•

broadcast update and return immediately

broadcast update and return immediately

–

Synchronous

Synchronous

•

broadcast update and wait for acknowledgements

broadcast update and wait for acknowledgements

–

Serialized Synchronous

Serialized Synchronous

•

acquires locks before updating a bean

acquires locks before updating a bean

•

ensures ordering but not consistency

ensures ordering but not consistency

•

consistency is obtained at another level

consistency is obtained at another level

Clustering Implementation in JBoss – Jakša Vučković, Università di
Bologna

44

A

DAPT

Cache Contents

Cache Contents

•

The cache will implement the XAResource

The cache will implement the XAResource

interface

interface

•

This will enable the use of the cache in a

This will enable the use of the cache in a

Two Phase Commit protocol

Two Phase Commit protocol

•

The mapping of the XAResource semantics

The mapping of the XAResource semantics

to the cluster is not a trivial problem

to the cluster is not a trivial problem

•

This could enable to do the DB update,

This could enable to do the DB update,

cluster replication and sending the

cluster replication and sending the

response to the client an atomic operation

response to the client an atomic operation

Clustering Implementation in JBoss – Jakša Vučković, Università di
Bologna

45

A

DAPT

Conclusions

Conclusions

•

JBoss clustering algorithm does not handle all failure

JBoss clustering algorithm does not handle all failure

patterns correctly

patterns correctly

•

Simply changing the GC semantics will not solve the

Simply changing the GC semantics will not solve the

problems

problems

•

To make the existing mechanism work correctly

To make the existing mechanism work correctly

–

A replication aware Transaction Manager needs to be

A replication aware Transaction Manager needs to be

implemented

implemented

–

During state merging the “out-of-date” beans must be marked

During state merging the “out-of-date” beans must be marked

•

An alternative would be to re-implement the clustering

An alternative would be to re-implement the clustering

from scratch

from scratch

•

The development of the Distributed Cache by JBoss

The development of the Distributed Cache by JBoss

authors is a good opportunity to redefine the clustering

authors is a good opportunity to redefine the clustering

algorithm

algorithm