Hazelcast IMDG Reference Manual 3.11.1

User Manual:

Open the PDF directly: View PDF .
Page Count: 693

Download
Open PDF In Browser	View PDF

Hazelcast IMDG 3.11.1 Reference Manual
Hazelcast IMDG Reference Manual . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
Preface. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
Hazelcast IMDG Editions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
Hazelcast IMDG Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
Hazelcast IMDG Plugins . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
Licensing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
Trademarks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
Customer Support . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
Release Notes. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
Contributing to Hazelcast IMDG . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
Partners . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
Phone Home . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
1. Document Revision History . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
2. Getting Started . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
2.1. Installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
2.1.1. Installing Hazelcast IMDG. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
2.1.2. Installing Hazelcast IMDG Enterprise . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
2.1.3. Setting the License Key . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
License Key Format . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
2.1.4. License Information . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
JMX . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
REST . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
Logs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
2.2. Supported Java Virtual Machines . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
2.3. Running in Modular Java . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
2.4. Upgrading from 3.x . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
2.5. Upgrading from 2.x . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
2.6. Starting the Member and Client . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
2.7. Using the Scripts In The Package . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
2.8. Deploying On Amazon EC2. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
2.9. Deploying On Microsoft Azure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
2.10. Deploying On Pivotal Cloud Foundry . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
2.11. Deploying using Docker . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
3. Hazelcast Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
3.1. Sharding in Hazelcast . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
3.2. Hazelcast Topology. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
3.3. Why Hazelcast? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24
3.4. Data Partitioning. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26

3.4.1. How the Data is Partitioned . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28
3.4.2. Partition Table . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28
3.4.3. Repartitioning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29
3.5. Use Cases. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29
3.6. Resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30
4. Understanding Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30
4.1. Configuring Declaratively. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30
4.1.1. Composing Declarative Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31
4.2. Configuring Programmatically . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33
4.3. Configuring with System Properties . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34
4.4. Configuring within Spring Context . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35
4.5. Dynamically Adding Data Structure Configuration on a Cluster . . . . . . . . . . . . . . . . . . . . . . . . . . 36
4.5.1. Handling Configuration Conflicts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37
4.6. Checking Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37
4.7. Configuration Pattern Matcher . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38
4.8. Using Wildcards . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38
4.9. Using Variables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40
4.10. Variable Replacers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41
4.10.1. EncryptionReplacer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42
4.10.2. PropertyReplacer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43
4.10.3. Implementing Custom Replacers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44
5. Setting Up Clusters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44
5.1. Discovery Mechanisms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44
5.1.1. TCP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44
5.1.2. Multicast . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44
5.1.3. AWS Cloud Discovery. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45
5.1.4. GCP Cloud Discovery . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45
5.1.5. Apache jclouds® Cloud Discovery . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45
5.1.6. Azure Cloud Discovery . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45
5.1.7. Zookeeper Cloud Discovery . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45
5.1.8. Consul Cloud Discovery . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45
5.1.9. etcd Cloud Discovery . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45
5.1.10. Hazelcast for PCF . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46
5.1.11. Hazelcast OpenShift Integration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46
5.1.12. Eureka Cloud Discovery . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46
5.1.13. Heroku Cloud Discovery . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46
5.1.14. Kubernetes Cloud Discovery . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46
5.2. Discovering Members by TCP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47
5.3. Discovering Members by Multicast . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48
5.4. Discovering Native Clients . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49
5.5. Creating Cluster Groups . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50

5.5.1. Cluster Groups before Hazelcast 3.8.2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51
5.6. Member User Code Deployment - BETA . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51
5.6.1. Configuring User Code Deployment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52
5.6.2. Example for Filtering Members . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53
5.7. Client User Code Deployment - BETA . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54
5.7.1. Configuring Client User Code Deployment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54
Important to Know . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55
5.7.2. Adding User Library to CLASSPATH . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56
5.8. Partition Group Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58
5.8.1. Grouping Types . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59
HOST_AWARE . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59
CUSTOM. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59
PER_MEMBER . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60
ZONE_AWARE . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61
SPI . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61
5.9. Logging Configuration. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62
5.9.1. Example Log4j2 Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64
5.9.2. Example Log4j Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65
5.10. Other Network Configurations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66
5.10.1. Public Address . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66
5.10.2. Port . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67
5.10.3. Outbound Ports . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67
5.10.4. Reuse Address . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68
5.10.5. Join . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 69
multicast element . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70
tcp-ip element . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70
aws element . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71
discovery-strategies element . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71
5.10.6. AWSClient Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71
5.10.7. Interfaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72
5.10.8. IPv6 Support. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73
5.10.9. Member Address Provider SPI . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74
5.11. Failure Detector Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75
5.11.1. Deadline Failure Detector . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76
5.11.2. Phi Accrual Failure Detector . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77
5.11.3. Ping Failure Detector . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 78
Requirements and Linux/Unix Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79
6. Rolling Member Upgrades . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82
6.1. Terminology. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82
6.2. Hazelcast Members Compatibility Guarantees . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82
6.3. Rolling Upgrade Procedure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83

6.4. Network Partitions and Rolling Upgrades . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 84
6.5. Rolling Upgrade FAQ . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 84
7. Distributed Data Structures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 85
7.1. Overview of Hazelcast Distributed Objects . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 86
7.1.1. Loading and Destroying a Distributed Object . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 87
7.1.2. Controlling Partitions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 88
7.1.3. Common Features of all Hazelcast Data Structures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 89
7.1.4. Example Distributed Object Code . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 89
7.2. Map . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 90
7.2.1. Getting a Map and Putting an Entry . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 90
7.2.2. Creating A Member for Map Backup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91
7.2.3. Backing Up Maps. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92
Creating Sync Backups. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93
Creating Async Backups . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93
Enabling Backup Reads . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 94
7.2.4. Map Eviction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 94
Understanding Map Eviction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 94
Configuring Map Eviction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95
Example Eviction Configurations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 98
Evicting Specific Entries . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 99
Evicting All Entries . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 99
Forced Eviction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 100
Custom Eviction Policy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 101
7.2.5. Setting In-Memory Format . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 102
7.2.6. Using High-Density Memory Store with Map. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103
Required configuration changes when using NATIVE . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 104
7.2.7. Loading and Storing Persistent Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 105
Using Read-Through Persistence. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 107
Setting Write-Through Persistence. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 107
Setting Write-Behind Persistence . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 107
Storing Entries to Multiple Maps . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 109
Initializing Map on Startup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 110
Loading Keys Incrementally . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 111
Forcing All Keys To Be Loaded . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 111
Post-Processing Objects in Map Store . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 112
Accessing a Database Using Properties . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 112
7.2.8. Creating Near Cache for Map. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113
7.2.9. Locking Maps. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113
Pessimistic Locking . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 114
Optimistic Locking . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 115
Pessimistic vs. Optimistic Locking . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 116

Solving the ABA Problem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 117
Lock Split-Brain Protection with Pessimistic Locking . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 117
7.2.10. Accessing Entry Statistics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 118
7.2.11. Map Listener . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 118
7.2.12. Listening to Map Entries with Predicates . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 118
7.2.13. Removing Map Entries in Bulk with Predicates . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 121
7.2.14. Adding Interceptors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 121
7.2.15. Preventing Out of Memory Exceptions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 123
Setting Query Result Size Limit . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 124
Local Pre-check . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 124
Scope of Result Size Limit . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 124
Configuring Query Result Size . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 125
7.3. Queue . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 125
7.3.1. Getting a Queue and Putting Items. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 125
7.3.2. Creating an Example Queue. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 126
Putting Items on the Queue . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 126
Taking Items off the Queue . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 127
Balancing the Queue Operations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 127
ItemIDs When Offering Items . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 128
7.3.3. Setting a Bounded Queue . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 128
7.3.4. Queueing with Persistent Datastore. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 129
7.3.5. Split-Brain Protection for Queue. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 131
7.3.6. Configuring Queue . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 132
7.4. MultiMap . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 133
7.4.1. Getting a MultiMap and Putting an Entry. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 134
7.4.2. Configuring MultiMap . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 134
7.4.3. Split-Brain Protection for MultiMap and TransactionalMultiMap . . . . . . . . . . . . . . . . . . . . 135
7.5. Set. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 137
7.5.1. Getting a Set and Putting Items . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 137
7.5.2. Configuring Set . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 138
7.5.3. Split-Brain Protection for ISet and TransactionalSet . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 138
7.6. List . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 139
7.6.1. Getting a List and Putting Items . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 140
7.6.2. Configuring List. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 140
7.6.3. Split-Brain Protection for IList and TransactionalList. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 141
7.7. Ringbuffer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 143
7.7.1. Getting a Ringbuffer and Reading Items . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 143
7.7.2. Adding Items to a Ringbuffer. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 143
7.7.3. IQueue vs. Ringbuffer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 143
7.7.4. Configuring Ringbuffer Capacity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 144
7.7.5. Backing Up Ringbuffer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 144

7.7.6. Configuring Ringbuffer Time-To-Live . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 144
7.7.7. Setting Ringbuffer Overflow Policy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 145
7.7.8. Ringbuffer with Persistent Datastore . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 146
7.7.9. Configuring Ringbuffer In-Memory Format . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 148
7.7.10. Configuring Split-Brain Protection for Ringbuffer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 148
7.7.11. Adding Batched Items . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 149
7.7.12. Reading Batched Items . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 149
7.7.13. Using Async Methods. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 150
7.7.14. Ringbuffer Configuration Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 151
7.8. Topic . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 151
7.8.1. Getting a Topic and Publishing Messages . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 151
7.8.2. Getting Topic Statistics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 152
7.8.3. Understanding Topic Behavior . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 153
Ordering Messages as Published. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 153
Ordering Messages for Members . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 153
Keeping Generated and Published Order the Same . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 154
7.8.4. Configuring Topic . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 154
7.9. Reliable Topic . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 155
7.9.1. Slow Consumers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 157
7.9.2. Configuring Reliable Topic . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 157
7.10. Lock . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 158
7.10.1. Using Try-Catch Blocks with Locks. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 158
7.10.2. Releasing Locks with tryLock Timeout . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 159
7.10.3. Avoiding Waiting Threads with Lease Time . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 159
7.10.4. Understanding Lock Behavior. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 160
ILock vs. IMap.lock . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 160
7.10.5. Synchronizing Threads with ICondition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 160
7.10.6. Split-Brain Protection for Lock . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 161
7.10.7. Lock Configuration. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 162
7.11. IAtomicLong . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 163
7.11.1. Sending Functions to IAtomicLong . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 163
7.11.2. Executing Functions on IAtomicLong . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 164
7.11.3. Reasons to Use Functions with IAtomic . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 165
7.11.4. Split-Brain Protection for IAtomicLong . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 165
7.12. ISemaphore . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 166
7.12.1. Controlling Thread Counts with Permits . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 166
7.12.2. Example Semaphore Code . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 166
7.12.3. Configuring Semaphore . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 168
7.12.4. Split-Brain Protection for ISemaphore . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 168
7.13. IAtomicReference. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 169
7.13.1. Sending Functions to IAtomicReference. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 169

7.13.2. Using IAtomicReference . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 170
7.13.3. Split-Brain Protection for IAtomicReference . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 170
7.14. ICountDownLatch . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 171
7.14.1. Gate-Keeping Concurrent Activities . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 171
7.14.2. Recovering From Failure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 172
7.14.3. Using ICountDownLatch . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 173
7.14.4. Split-Brain Protection for ICountDownLatch. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 173
7.15. PN Counter. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 173
7.15.1. Configuring PN Counter . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 175
7.15.2. Configuring the CRDT replication mechanism . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 176
7.16. IdGenerator . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 177
7.16.1. Generating Cluster-Wide IDs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 177
7.16.2. Unique IDs and Duplicate IDs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 178
7.16.3. Migrating to FlakeIdGenerator . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 178
7.17. FlakeIdGenerator . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 178
7.17.1. Generating Cluster-Wide IDs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 178
7.17.2. Performance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 179
7.17.3. Example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 179
7.17.4. Node ID Assignment. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 179
Node ID Overflow . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 180
7.18. Replicated Map . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 180
7.18.1. Replicating Instead of Partitioning. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 180
7.18.2. Example Replicated Map Code . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 181
7.18.3. Considerations for Replicated Map . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 181
7.18.4. Configuration Design for Replicated Map . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 182
7.18.5. Configuring Replicated Map . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 182
In-Memory Format on Replicated Map . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 184
7.18.6. Using EntryListener on Replicated Map . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 184
Difference in EntryListener on Replicated Map . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 184
Example of Replicated Map EntryListener . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 184
7.18.7. Split-Brain Protection for Replicated Map . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 185
7.19. Cardinality Estimator Service . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 186
7.19.1. Split-Brain Protection for Cardinality Estimator . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 187
7.20. Event Journal. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 189
7.20.1. Interaction with Evictions and Expiration for IMap . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 189
7.20.2. Configuring Event Journal Capacity. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 189
7.20.3. Event Journal Partitioning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 191
7.20.4. Configuring Event Journal time-to-live . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 191
8. Distributed Events . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 191
8.1. Cluster Events . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 192
8.1.1. Listening for Member Events . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 192

Registering Membership Listeners. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 193
8.1.2. Listening for Distributed Object Events . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 194
Registering Distributed Object Listeners . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 195
8.1.3. Listening for Migration Events . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 195
Registering Migration Listeners . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 196
8.1.4. Listening for Partition Lost Events. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 197
Writing a Partition Lost Listener Class . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 197
Registering Partition Lost Listeners . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 198
8.1.5. Listening for Lifecycle Events . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 198
Registering Lifecycle Listeners . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 199
8.1.6. Listening for Clients . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 200
8.2. Distributed Object Events. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 201
8.2.1. Listening for Map Events . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 201
Catching a Map Event . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 201
8.2.2. Listening for Lost Map Partitions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 203
Registering Map Listeners . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 204
Map Listener Attributes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 205
8.2.3. Listening for MultiMap Events . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 205
Registering MultiMap Listeners . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 206
MultiMap Listener Attributes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 207
8.2.4. Listening for Item Events . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 207
Registering Item Listeners . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 207
Item Listener Attributes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 208
8.2.5. Listening for Topic Messages . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 208
Registering Message Listeners. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 209
8.3. Event Listeners for Hazelcast Clients . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 210
8.4. Global Event Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 210
9. Hazelcast Jet . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 211
9.1. Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 211
9.1.1. How You Can Use It . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 211
9.1.2. Where You Can Use It . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 211
9.1.3. Data Processing Styles . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 212
9.2. Relationship with Hazelcast IMDG. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 212
9.3. Hazelcast IMDG Computing vs. Hazelcast Jet . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 213
10. Distributed Computing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 214
10.1. Executor Service. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 214
10.1.1. Implementing a Callable Task . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 214
Executing a Callable Task . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 216
10.1.2. Implementing a Runnable Task . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 217
Executing a Runnable Task . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 217
10.1.3. Scaling The Executor Service. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 218

10.1.4. Executing Code in the Cluster . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 218
10.1.5. Canceling an Executing Task . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 220
Example Task to Cancel. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 220
Example Method to Execute and Cancel the Task . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 220
10.1.6. Callback When Task Completes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 221
Example Task to Callback . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 221
Example Method to Callback the Task . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 222
10.1.7. Selecting Members for Task Execution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 223
10.1.8. Configuring Executor Service . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 224
10.1.9. Split-Brain Protection for IExecutorService. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 224
10.2. Durable Executor Service. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 225
10.2.1. Configuring Durable Executor Service . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 226
10.2.2. Split-Brain Protection for Durable Executor Service . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 227
10.3. Scheduled Executor Service. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 227
10.3.1. Configuring Scheduled Executor Service . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 229
10.3.2. Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 230
10.3.3. Split-Brain Protection for IScheduled Executor Service . . . . . . . . . . . . . . . . . . . . . . . . . . . . 230
10.4. Entry Processor. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 231
10.4.1. Performing Fast In-Memory Map Operations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 232
Using Indexes. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 232
Using OBJECT In-Memory Format . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 232
Processing Entries. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 232
Respecting Locks on Single Keys . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 233
Processing Backup Entries . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 233
10.4.2. Creating an Entry Processor . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 233
10.4.3. Abstract Entry Processor . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 234
10.4.4. Entry Processor Performance Optimizations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 235
Offloadable Entry Processor . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 235
ReadOnly Entry Processor . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 236
ReadOnly and Offloadable Entry Processor . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 237
11. Distributed Query . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 238
11.1. How Distributed Query Works . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 238
11.1.1. Employee Map Query Example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 238
11.1.2. Querying with Criteria API. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 239
Predicates Class Operators . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 240
Combining Predicates with AND, OR, NOT . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 240
Simplifying with PredicateBuilder . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 241
11.1.3. Querying with SQL . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 241
Supported SQL Syntax . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 241
Querying Entry Keys with Predicates . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 242
11.1.4. Filtering with Paging Predicates . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 243

11.1.5. Filtering with Partition Predicate. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 244
11.1.6. Indexing Queries . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 244
Indexing Ranged Queries . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 245
Configuring IMap Indexes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 245
Copying Indexes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 246
Indexing Attributes with ValueExtractor . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 246
Using "this" as an Attribute . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 246
11.1.7. Configuring Query Thread Pool . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 247
Query Requests from Clients . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 247
11.2. Querying in Collections and Arrays . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 248
11.2.1. Indexing in Collections and Arrays . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 249
11.2.2. Corner cases. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 250
11.3. Custom Attributes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 250
11.3.1. Implementing a ValueExtractor . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 251
ValueExtractor with Portable Serialization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 251
Returning Multiple Values from a Single Extraction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 251
11.3.2. Extraction Arguments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 252
11.3.3. Configuring a Custom Attribute Programmatically . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 252
11.3.4. Configuring a Custom Attribute Declaratively . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 252
11.3.5. Indexing Custom Attributes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 253
11.4. MapReduce . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 253
11.4.1. Understanding MapReduce . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 254
MapReduce Workflow Example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 254
MapReduce Phases . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 255
Additional MapReduce Resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 256
11.4.2. Using the MapReduce API. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 257
Retrieving a JobTracker Instance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 257
Creating a Job. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 258
Creating Key-Value Input Sources with KeyValueSource . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 259
Implementing Mapping Logic with Mapper . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 260
Minimizing Cluster Traffic with Combiner . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 261
Doing Algorithm Work with Reducer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 262
Modifying the Result with Collator. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 263
Preselecting Keys with KeyPredicate . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 264
Job Monitoring with TrackableJob . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 264
Configuring JobTracker . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 265
11.4.3. Hazelcast MapReduce Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 266
Member Interoperation Example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 266
Internal MapReduce Packages. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 268
MapReduce Job Walk-Through . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 268
11.4.4. MapReduce Deprecation. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 269

Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 269
Built-In Aggregations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 269
Distributed Computation with Jet . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 270
Jet Compared with New Aggregations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 275
11.5. Aggregators . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 275
11.5.1. Aggregations Basics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 275
Aggregations and Map Interfaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 275
Aggregations and Java . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 276
Aggregations and Java 6 or Java 7 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 276
Aggregations and Java 8 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 276
Aggregations and the MapReduce Framework . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 277
11.5.2. Using the Aggregations API . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 277
Supplier . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 277
Defining the Aggregation Operation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 280
Extracting Attribute Values with PropertyExtractor . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 281
Configuring Aggregations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 282
11.5.3. Aggregations Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 283
Setting up the Data Model. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 283
Average Aggregation Example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 285
Map Join Example. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 285
Grouping Example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 286
Simple Count Example. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 287
11.5.4. Implementing Aggregations. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 287
Aggregation Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 288
11.6. Fast-Aggregations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 288
11.6.1. Aggregator API . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 288
11.6.2. Fast-Aggregations and Map Interfaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 289
11.6.3. Sample Implementation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 289
11.6.4. Built-In Aggregations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 290
11.6.5. Configuration Options . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 291
11.7. Projections . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 291
11.7.1. Projection API . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 291
Projections and Map Interfaces. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 291
11.7.2. Sample implementation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 292
11.7.3. Built-In Projections. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 292
11.8. Continuous Query Cache . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 293
11.8.1. Keeping Query Results Local and Ready. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 293
11.8.2. Accessing Continuous Query Cache from Member . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 293
11.8.3. Accessing Continuous Query Cache from Client Side . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 293
11.8.4. Features of Continuous Query Cache . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 294
11.8.5. Configuring Continuous Query Cache . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 294

12. Transactions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 296
12.1. Creating a Transaction Interface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 296
12.1.1. Queue/Set/List vs. Map/Multimap . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 298
12.1.2. ONE_PHASE vs. TWO_PHASE . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 298
12.2. Providing XA Transactions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 298
13. Hazelcast JCache . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 299
13.1. JCache Overview. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 299
13.1.1. Supported JCache Versions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 300
13.1.2. Upgrading from JCache 1.0.0 to 1.1.0 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 300
13.2. JCache Setup and Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 300
13.2.1. Setting up Your Application . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 301
Activating Hazelcast as JCache Provider . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 301
Connecting Clients to Remote Member . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 302
13.2.2. Example JCache Application . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 302
Getting the Hazelcast JCache Implementation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 303
Setting up the JCache Entry Point . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 303
Configuring the Cache Before Creating It . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 303
Creating the Cache . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 303
get, put and getAndPut . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 303
13.2.3. Configuring for JCache . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 303
JCache Declarative Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 303
JCache Programmatic Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 305
13.3. JCache Providers. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 306
13.3.1. Configuring JCache Provider . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 306
13.3.2. Configuring JCache with Client Provider . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 307
13.3.3. Configuring JCache with Server Provider . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 307
13.4. JCache API . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 308
13.4.1. JCache API Application Example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 308
Creating User Class Example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 308
Creating DAO Interface Example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 308
Configuring JCache Example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 309
13.4.2. JCache Base Classes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 310
13.4.3. Implementing Factory and FactoryBuilder . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 311
13.4.4. Implementing CacheLoader. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 311
Cache read-through . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 312
CacheLoader Example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 312
13.4.5. CacheWriter. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 313
13.4.6. Implementing EntryProcessor . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 315
13.4.7. CacheEntryListener . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 316
13.4.8. ExpiryPolicy. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 318
13.5. JCache - Hazelcast Instance Integration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 319

13.5.1. JCache and Hazelcast Instance Awareness . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 320
13.6. Hazelcast JCache Extension - ICache . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 320
13.6.1. Scoping to Join Clusters. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 321
Examples. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 321
Applying Configuration Scope . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 323
Binding to a Named Instance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 325
Binding to an Existing Hazelcast Instance Object. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 327
13.6.2. Namespacing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 328
13.6.3. Retrieving an ICache Instance. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 328
13.6.4. ICache Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 329
13.6.5. ICache Async Methods. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 330
13.6.6. Defining a Custom ExpiryPolicy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 332
13.6.7. JCache Eviction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 333
Eviction and Runtime . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 333
Cache Types . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 333
Configuring Eviction Policies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 334
Eviction Strategy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 336
Eviction Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 337
13.6.8. JCache Near Cache . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 340
13.6.9. ICache Convenience Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 340
13.6.10. Implementing BackupAwareEntryProcessor . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 340
13.6.11. ICache Partition Lost Listener . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 342
13.7. Testing for JCache Specification Compliance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 343
14. Integrated Clustering . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 344
14.1. Integration with Hibernate Second Level Cache . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 344
14.2. Web Session Replications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 344
14.3. Integration with Java EE. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 344
14.4. Integration with Spring . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 345
14.4.1. Configuring Spring . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 345
Enabling Spring Integration. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 345
Troubleshooting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 345
Declaring Beans by Spring beans Namespace . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 346
Declaring Beans by hazelcast Namespace . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 346
Supported Configurations with hazelcast Namespace . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 347
14.4.2. Enabling SpringAware Objects . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 350
SpringAware Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 350
14.4.3. Adding Caching to Spring . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 353
Declarative Spring Cache Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 354
Defining Timeouts for Cache Read Operation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 354
Declarative Hazelcast JCache Based Caching Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . 355
Annotation-Based Spring Cache Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 355

14.4.4. Configuring Hibernate Second Level Cache. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 356
14.4.5. Configuring Hazelcast Transaction Manager . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 357
Sample Configuration for Hazelcast Transaction Manager . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 357
Sample Transactional Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 358
14.4.6. Best Practices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 358
15. Storage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 359
15.1. High-Density Memory Store . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 359
15.1.1. Configuring High-Density Memory Store . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 360
15.2. Sizing Practices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 362
15.3. Hot Restart Persistence . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 362
15.3.1. Hot Restart Persistence Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 362
15.3.2. Hot Restart Types . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 363
15.3.3. Restart Process . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 363
Restart of a Member in Running Cluster. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 364
15.3.4. Force Start . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 364
15.3.5. Partial Start . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 365
15.3.6. Configuring Hot Restart . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 366
Global Hot Restart Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 366
Per Data Structure Hot Restart Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 367
Hot Restart Configuration Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 367
15.3.7. Moving/Copying Hot Restart Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 369
15.3.8. Hot Restart Persistence Design Details . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 369
15.3.9. Concurrent, Incremental, Generational GC . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 370
I/O Minimization Scheme . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 370
Cost-Benefit Factor . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 371
15.3.10. Hot Restart Performance Considerations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 371
Performance on a Physical Server . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 371
Performance on AWS R3 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 372
15.3.11. Hot Backup. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 373
Configuring Hot Backup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 373
Using Hot Backup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 374
Starting the Cluster From a Hot Backup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 375
Achieving High Consistency of Backup Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 376
Achieving High Performance of Backup Process . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 376
Backup Process Progress and Completion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 377
Backup Task Interruption and Cancellation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 377
15.4. Hazelcast Striim Hot Cache . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 378
16. Hazelcast Clients . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 379
16.1. Java Client . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 379
16.1.1. Getting Started with Java Client . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 380
Client API . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 380

Java Client Operation Modes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 381
Handling Failures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 381
Using Supported Distributed Data Structures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 383
Using Client Services . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 384
Client Listeners . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 386
Client Transactions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 386
Async Start and Reconnect Modes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 386
16.1.2. Configuring Java Client . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 387
Client Network. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 388
Configuring Client Load Balancer. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 397
Configuring Client Listeners . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 397
Configuring Client Connection Strategy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 398
Configuring Client Connection Retry . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 399
Other Configurations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 400
16.1.3. Java Client Failure Detectors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 401
Client Deadline Failure Detector . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 402
Client Ping Failure Detector . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 403
16.1.4. Client System Properties . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 405
16.1.5. Using High-Density Memory Store with Java Client. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 408
16.2. C++ Client . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 410
16.3. .NET Client . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 410
16.4. REST Client. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 411
16.4.1. REST Client GET/POST/DELETE Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 411
Creating/Updating Entries in a Map for REST Client . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 412
Retrieving Entries from a Map for REST Client . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 412
Removing Entries from a Map for REST Client . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 412
Offering Items on a Queue for REST Client. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 413
Retrieving Items from a Queue for REST Client . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 413
Getting the size of the queue for REST Client. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 414
16.4.2. Checking the Status of the Cluster for REST Client . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 414
16.5. Memcache Client . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 415
16.5.1. Memcache Client Code Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 415
16.5.2. Unsupported Operations for Memcache. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 417
16.6. Python Client . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 417
16.7. Node.js Client. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 417
16.8. Go Client . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 417
16.9. Scala. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 417
17. Serialization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 417
17.1. Serialization Interface Types. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 418
17.2. Comparing Serialization Interfaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 419
17.3. Implementing Java Serializable and Externalizable . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 420

17.3.1. Implementing Java Externalizable. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 421
17.4. Implementing DataSerializable . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 422
17.4.1. Reading and Writing and DataSerializable . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 422
17.4.2. IdentifiedDataSerializable . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 424
getID and getFactoryId Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 424
Implementing IdentifiedDataSerializable . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 424
Registering EmployeeDataSerializableFactory . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 426
17.5. Implementing Portable Serialization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 426
17.5.1. Portable Serialization Example Code . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 427
17.5.2. Registering the Portable Factory. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 429
17.5.3. Versioning for Portable Serialization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 429
Example Portable Versioning Scenarios . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 430
17.5.4. Ordering Consistency for writePortable . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 431
17.5.5. Null Portable Serialization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 432
17.5.6. DistributedObject Serialization. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 432
17.6. Custom Serialization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 432
17.6.1. Implementing StreamSerializer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 432
StreamSerializer Example Code 1. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 433
StreamSerializer Example Code 2. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 434
Configuring StreamSerializer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 435
17.6.2. Implementing ByteArraySerializer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 436
Configuring ByteArraySerializer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 436
17.7. Global Serializer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 437
17.7.1. Sample Global Serializer. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 437
17.8. Implementing HazelcastInstanceAware. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 438
17.9. Untrusted Deserialization Protection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 439
17.10. Serialization Configuration Wrap-Up . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 441
18. Management . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 443
18.1. Getting Member Statistics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 443
18.1.1. Map Statistics. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 443
18.1.2. Map Index Statistics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 444
18.1.3. Near Cache Statistics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 445
18.1.4. Multimap Statistics. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 445
18.1.5. Queue Statistics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 446
18.1.6. Topic Statistics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 447
18.1.7. Executor Statistics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 447
18.2. JMX API per Member. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 448
18.3. Monitoring with JMX . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 455
18.3.1. MBean Naming for Hazelcast Data Structures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 456
18.3.2. Connecting to JMX Agent . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 456
18.4. Cluster Utilities . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 457

18.4.1. Getting Member Events and Member Sets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 457
18.4.2. Managing Cluster and Member States. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 458
Cluster States . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 458
Cluster Member States. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 460
18.4.3. Using the Script cluster.sh . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 460
Example Usages for cluster.sh . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 461
18.4.4. Using REST API for Cluster Management . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 463
18.4.5. Enabling Lite Members . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 465
Configuring Lite Members . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 465
Promoting Lite Members to Data Member . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 465
18.4.6. Defining Member Attributes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 465
18.4.7. Safety Checking Cluster Members . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 467
Ensuring Safe State with PartitionService . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 467
18.5. Diagnostics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 468
18.5.1. Enabling Diagnostics Logging . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 468
18.5.2. Diagnostics Log File . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 468
18.5.3. Diagnostics Plugins . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 469
BuildInfo . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 469
SystemProperties . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 469
ConfigProperties . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 470
Metrics. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 470
SlowOperations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 470
Invocations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 470
HazelcastInstance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 470
SystemLog. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 471
StoreLatency . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 471
OperationHeartbeats . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 472
MemberHeartbeats . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 473
OperationThreadSamples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 473
WanDiagnostics. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 474
18.6. Health Check and Monitoring . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 475
18.6.1. Health Check . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 475
18.6.2. Health Check Script . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 476
18.6.3. Health Monitor . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 476
18.6.4. Using Health Check on F5 BIG-IP LTM. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 477
Monitor Types . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 477
Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 478
18.7. Management Center . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 479
18.7.1. Toggle Scripting Support. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 479
18.8. Clustered JMX and REST via Management Center . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 479
19. Security. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 480

19.1. Enabling JAAS Security . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 480
19.2. Socket Interceptor . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 480
19.3. Security Interceptor. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 483
19.4. Encryption . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 483
19.5. TLS/SSL . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 485
19.5.1. TLS/SSL for Hazelcast Members . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 485
Other Property Configuration Options . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 487
19.5.2. TLS/SSL for Hazelcast Clients. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 488
19.5.3. Mutual Authentication . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 488
19.5.4. TLS/SSL Performance Improvements for Java . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 490
19.5.5. TLS/SSL Debugging. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 491
19.5.6. TLS/SSL for Hazelcast Management Center . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 491
19.6. Integrating OpenSSL / BoringSSL . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 491
19.6.1. Netty Libraries . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 491
19.6.2. Using BoringSSL . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 492
19.6.3. Using OpenSSL . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 492
19.6.4. Configuring Hazelcast for OpenSSL . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 493
19.6.5. Configuring Cipher Suites. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 494
19.6.6. Other Ways of Configuring Properties . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 495
19.7. Credentials . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 496
19.8. Validating Secrets Using Strength Policy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 497
19.8.1. Using a Custom Secret Strength Policy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 498
19.8.2. Enforcing the Secret Strength Policy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 498
19.9. ClusterLoginModule . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 499
19.9.1. Enterprise Integration. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 501
19.10. Cluster Member Security . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 501
19.11. Native Client Security . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 503
19.11.1. Authentication . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 503
19.11.2. Authorization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 504
19.11.3. Permissions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 506
20. Performance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 511
20.1. Data Affinity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 511
20.1.1. PartitionAware . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 511
20.1.2. PartitioningStrategy. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 515
20.2. Running in EC2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 516
20.3. Back Pressure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 517
20.3.1. Member Side . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 517
20.3.2. Client Side. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 518
20.4. Threading Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 518
20.4.1. I/O Threading. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 518
20.4.2. Event Threading . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 519

20.4.3. IExecutor Threading . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 520
20.4.4. Operation Threading . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 520
Partition-aware Operations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 520
Non-Partition-aware Operations. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 521
Priority Operations. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 522
Operation-response and Invocation-future . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 522
Local Calls . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 522
20.5. SlowOperationDetector . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 522
20.5.1. Logging of Slow Operations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 523
20.5.2. Purging of Slow Operation Logs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 523
20.6. Near Cache . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 523
20.6.1. Hazelcast Data Structures with Near Cache Support. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 524
20.6.2. Configuring Near Cache . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 525
20.6.3. Near Cache Configuration Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 528
Near Cache Example for IMap. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 528
Near Cache Example for JCache Clients . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 530
Example for Near Cache with High-Density Memory Store . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 531
20.6.4. Near Cache Eviction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 532
20.6.5. Near Cache Expiration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 532
20.6.6. Near Cache Invalidation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 532
20.6.7. Near Cache Consistency . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 533
Eventual Consistency . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 533
Locally Initiated Changes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 533
20.6.8. Near Cache Preloader . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 534
20.7. Caching Deserialized Values . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 534
21. Hazelcast Simulator . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 535
22. WAN . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 535
22.1. WAN Replication . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 535
22.1.1. Defining WAN Replication . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 536
Defining WAN Replication Using Static Endpoints . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 536
Defining WAN Replication Using Discovery SPI . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 539
22.1.2. WanBatchReplication Implementation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 544
22.1.3. Configuring WAN Replication for IMap and ICache . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 545
22.1.4. Batch Size . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 547
22.1.5. Batch Maximum Delay . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 548
22.1.6. Response Timeout . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 549
22.1.7. Queue Capacity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 550
22.1.8. Queue Full Behavior . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 551
22.1.9. Event Filtering API . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 551
22.1.10. Acknowledgment Types . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 552
22.1.11. Synchronizing WAN Target Cluster . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 553

22.1.12. WAN Replication Failure Detection and Recovery . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 554
WAN Target Endpoint List . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 554
WAN Failure Detection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 554
WAN Endpoint Recovery . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 554
22.1.13. WAN Replication Additional Information . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 555
22.2. Delta WAN Synchronization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 555
22.2.1. Requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 556
22.2.2. Using Delta WAN Synchronization. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 556
22.2.3. Configuring Delta WAN Synchronization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 557
22.2.4. The Process . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 558
22.2.5. Memory Consumption. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 559
22.2.6. Defining the Depth . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 559
22.2.7. REST API . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 560
22.2.8. Statistics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 561
22.3. Hazelcast WAN Replication with Solace . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 561
22.3.1. Enabling Integration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 561
22.3.2. Configuring Publisher . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 562
22.3.3. Configuring Consumer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 563
22.3.4. Additional Information . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 563
23. OSGI. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 564
23.1. OSGI Support . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 564
23.2. API . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 564
23.3. Configuring Hazelcast OSGI Support. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 564
23.4. Design . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 565
23.5. Using Hazelcast OSGI Service . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 566
23.5.1. Getting Hazelcast OSGI Service Instances . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 566
23.5.2. Managing and Using Hazelcast instances. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 566
24. Extending Hazelcast . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 567
24.1. User Defined Services . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 567
24.1.1. Creating the Service Class. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 568
24.1.2. Enabling the Service Class . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 569
24.1.3. Adding Properties to the Service . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 570
24.1.4. Starting the Service . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 570
24.1.5. Placing a Remote Call via Proxy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 570
Making Counter a Distributed Object . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 571
Implementing ManagedService and RemoteService . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 571
Implementing CounterProxy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 572
Dealing with Exceptions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 574
Implementing the PartitionAwareOperation Interface. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 574
Running the Code . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 575
24.1.6. Creating Containers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 576

Integrating the Container in the CounterService . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 577
Connecting the IncOperation.run Method to the Container. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 579
Running the Sample Code. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 580
24.1.7. Partition Migration. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 582
Transferring migrationData. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 583
Letting Hazelcast Know CounterService Can Do Partition Migrations . . . . . . . . . . . . . . . . . . . 584
Running the Sample Code. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 586
24.1.8. Creating Backups . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 588
Performing the Backup with IncBackupOperation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 589
Running the Sample Code. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 590
24.2. OperationParker. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 591
24.3. Discovery SPI. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 592
24.3.1. Discovery SPI Interfaces and Classes. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 592
DiscoveryStrategy: Implement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 592
AbstractDiscoveryStrategy: Abstract Class . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 592
DiscoveryStrategyFactory: Factory Contract . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 592
DiscoveryNode: Describe a Member . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 593
SimpleDiscoveryNode: Default DiscoveryNode. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 593
NodeFilter: Filter Members . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 593
DiscoveryService: Support In Integrator Systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 593
DiscoveryServiceProvider: Provide a DiscoveryService . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 593
DiscoveryServiceSettings: Configure DiscoveryService . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 593
DiscoveryMode: Member or Client. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 593
24.3.2. Discovery Strategy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 594
Discovery Strategy Example. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 594
Configuring Site Domain. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 594
Creating Discovery . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 594
Implementing Discovery Strategy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 595
Extending The AbstractDiscoveryStrategy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 596
Overriding Discovery Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 598
Implementing Lookup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 598
Mapping to `DiscoveryNode`s . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 599
Configuring DiscoveryStrategy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 600
24.3.3. DiscoveryService (Framework integration) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 601
24.4. Config Properties SPI. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 602
24.4.1. Config Properties SPI Classes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 602
PropertyDefinition: Define a Single Property . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 602
SimplePropertyDefinition: Basic PropertyDefinition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 602
PropertyTypeConverter: Set of TypeConverters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 602
ValueValidator and ValidationException . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 602
24.4.2. Config Properties SPI Example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 603

Defining a Config PropertyDefinition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 603
Providing a value in XML . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 603
Retrieving a PropertyDefinition Value . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 603
25. Hazelcast Plugins . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 604
25.1. Cloud Discovery Plugins . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 604
25.1.1. Hazelcast jclouds® . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 604
25.1.2. Hazelcast AWS. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 604
25.1.3. Hazelcast GCP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 604
25.1.4. Hazelcast Azure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 605
25.1.5. Hazelcast Consul . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 605
25.1.6. Hazelcast etcd . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 605
25.1.7. Hazelcast Eureka . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 605
25.1.8. Hazelcast IMDG and Jet for PCF . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 606
25.1.9. Hazelcast OpenShift . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 606
25.1.10. Hazelcast Heroku . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 606
25.1.11. Hazelcast Kubernetes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 606
25.1.12. Hazelcast Zookeeper . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 607
25.2. Integration Plugins . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 607
25.2.1. Spring Data Hazelcast . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 607
25.2.2. Spring Integration Extension for Hazelcast . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 607
25.2.3. Hazelcast JCA Resource Adapter. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 608
Integrating with MuleSoft. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 608
25.2.4. Hazelcast Grails. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 608
25.2.5. Hazelcast Hibernate 2LC. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 608
25.2.6. Hazelcast DynaCache. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 609
25.2.7. Hazelcast Connector for Kafka . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 609
25.2.8. Openfire . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 609
25.2.9. SubZero . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 609
25.3. Web Sessions Clustering Plugins . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 609
25.3.1. Filter Based Web Session Replication . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 610
25.3.2. Tomcat Based Web Session Replication . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 610
25.3.3. Jetty Based Web Session Replication . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 610
25.4. Big Data Plugins . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 610
26. Consistency and Replication Model. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 611
26.1. A Brief Overview of Consistency and Replication in Distributed Systems. . . . . . . . . . . . . . . . 611
26.2. Hazelcast’s Replication Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 611
26.2.1. Best-Effort Consistency . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 612
26.3. Invocation Lifecycle . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 613
26.4. Exactly-once, At-least-once or At-most-once Execution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 614
26.5. IndeterminateOperationStateException. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 614
27. Network Partitioning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 615

27.1. Split-Brain Syndrome . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 615
27.2. Dealing with Network Partitions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 615
27.3. Split-Brain Protection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 615
27.3.1. Time Window for Split-Brain Protection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 616
27.3.2. Configuring Split-Brain Protection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 617
Member Count Quorum . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 617
Probabilistic Quorum Function. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 618
Recently-Active Quorum Function . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 620
Quorum Configuration Reference . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 620
27.3.3. Configuring Quorum Listeners . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 621
27.3.4. Querying Quorum Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 622
27.4. Split-Brain Recovery . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 623
27.4.1. Merge Policies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 624
27.4.2. Supported Data Structures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 624
27.4.3. Configuring Merge Policies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 625
Declarative Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 625
Programmatic Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 626
27.4.4. Custom Merge Policies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 627
Merge Types . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 627
Accessing Deserialized Values . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 630
Accessing Hazelcast UserContext . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 634
Merge Policies With Multiple Merge Types . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 636
Merge Policies For Specific Data Structures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 637
Best Practices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 639
Appendix A: System Properties. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 640
Appendix B: Common Exception Types . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 651
Appendix C: License Questions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 652
C.1. Embedded Dependencies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 652
C.2. Runtime Dependencies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 652
Appendix D: Frequently Asked Questions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 653
Glossary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 664

Hazelcast IMDG Reference Manual
Version 3.11.1

Preface
Welcome to the Hazelcast IMDG (In-Memory Data Grid) Reference Manual. This manual includes
concepts, instructions and samples to guide you on how to use Hazelcast and build Hazelcast IMDG
applications.
As the reader of this manual, you must be familiar with the Java programming language and you
should have installed your preferred Integrated Development Environment (IDE).

Hazelcast IMDG Editions
This Reference Manual covers all editions of Hazelcast IMDG. Throughout this manual:
• Hazelcast or Hazelcast IMDG refers to the open source edition of Hazelcast in-memory data
grid middleware. Hazelcast is also the name of the company (Hazelcast, Inc.) providing the
Hazelcast product.
• Hazelcast IMDG Enterprise is a commercially licensed edition of Hazelcast IMDG which
provides high-value enterprise features in addition to Hazelcast IMDG.
• Hazelcast IMDG Enterprise HD is a commercially licensed edition of Hazelcast IMDG which
provides High-Density (HD) Memory Store and Hot Restart Persistence features in addition to
Hazelcast IMDG Enterprise.

Hazelcast IMDG Architecture
You can see the features for all Hazelcast IMDG editions in the following architecture diagram.

1

You can see small "HD" boxes for some features in the above diagram. Those



features can use High-Density (HD) Memory Store when it is available. It means if
you have Hazelcast IMDG Enterprise HD, you can use those features with HD
Memory Store.

For more information on Hazelcast IMDG’s Architecture, please see the white paper An Architect’s
View of Hazelcast.

Hazelcast IMDG Plugins
You can extend Hazelcast IMDG’s functionality by using its plugins. These plugins have their own
lifecycles. Please see Plugins page to learn about Hazelcast plugins you can use. Hazelcast plugins
are marked with

label throughout this manual. See also the Hazelcast Plugins chapter for

more information.

Licensing
Hazelcast IMDG and Hazelcast Reference Manual are free and provided under the Apache License,
Version 2.0. Hazelcast IMDG Enterprise and Hazelcast IMDG Enterprise HD is commercially
licensed by Hazelcast, Inc.
For more detailed information on licensing, please see the License Questions appendix.

2

Trademarks
Hazelcast is a registered trademark of Hazelcast, Inc. All other trademarks in this manual are held
by their respective owners.

Customer Support
Support for Hazelcast is provided via GitHub, Mail Group and StackOverflow.
For information on the commercial support for Hazelcast IMDG and Hazelcast IMDG Enterprise,
please see hazelcast.com.

Release Notes
Please refer to the Release Notes document for the new features, enhancements and fixes
performed for each Hazelcast IMDG release.

Contributing to Hazelcast IMDG
You can contribute to the Hazelcast IMDG code, report a bug, or request an enhancement. Please
see the following resources.
• Developing with Git: Document that explains the branch mechanism of Hazelcast and how to
request changes.
• Hazelcast Contributor Agreement form: Form that each contributing developer needs to fill and
send back to Hazelcast.
• Hazelcast on GitHub: Hazelcast repository where the code is developed, issues and pull requests
are managed.

Partners
Hazelcast partners with leading hardware and software technologies, system integrators, resellers
and OEMs including Amazon Web Services, Vert.x, Azul Systems, C2B2. Please see the Partners page
for the full list of and information on our partners.

Phone Home
Hazelcast uses phone home data to learn about usage of Hazelcast IMDG.
Hazelcast IMDG member instances call our phone home server initially when they are started and
then every 24 hours. This applies to all the instances joined to the cluster.
What is sent in?
The following information is sent in a phone home:
• Hazelcast IMDG version
3

• Local Hazelcast IMDG member UUID
• Download ID
• A hash value of the cluster ID
• Cluster size bands for 5, 10, 20, 40, 60, 100, 150, 300, 600 and > 600
• Number of connected clients bands of 5, 10, 20, 40, 60, 100, 150, 300, 600 and > 600
• Cluster uptime
• Member uptime
• Environment Information:
◦ Name of operating system
◦ Kernel architecture (32-bit or 64-bit)
◦ Version of operating system
◦ Version of installed Java
◦ Name of Java Virtual Machine
• Hazelcast IMDG Enterprise specific:
◦ Number of clients by language (Java, C++, C#)
◦ Flag for Hazelcast Enterprise
◦ Hash value of license key
◦ Native memory usage
• Hazelcast Management Center specific:
◦ Hazelcast Management Center version
◦ Hash value of Hazelcast Management Center license key
Phone Home Code
The phone home code itself is open source. Please see here.
Disabling Phone Homes
Set the hazelcast.phone.home.enabled system property to false either in the config or on the Java
command line. Please see the System Properties section for information on how to set a property.
You can also disable the phone home using the environment variable HZ_PHONE_HOME_ENABLED. Simply
add the following line to your .bash_profile:

export HZ_PHONE_HOME_ENABLED=false
Phone Home URLs
For versions 1.x and 2.x: http://www.hazelcast.com/version.jsp.
For versions 3.x up to 3.6: http://versioncheck.hazelcast.com/version.jsp.

4

For versions after 3.6: http://phonehome.hazelcast.com/ping.

1. Document Revision History
This chapter lists the changes made to this document from the previous release.



Please refer to the Release Notes for the new features, enhancements and fixes
performed for each Hazelcast release.*

Table 1. Revision History

Chapter

Section

Getting Started

Description
Added Running in Modular Java
as a new section.
Added Supported Java Virtual
Machines as a new section to
summarize the compatibility
between Hazelcast IMDG and
various JVMs.
Added License Information as a
new section to explain how you
can retrieve your Hazelcast
IMDG license details.

Setting Up Clusters

Partition Group Configuration

Added a note to the
ZONE_AWARE type content
suggesting to have equal
number of members in each
Availability Zone.

Distributed Data Structures

Reliable Topic

Enhanced the content to clarify
the relation between Reliable
Topic and Ringbuffer data
structures.

Map

Added content for the newly
introduced maxIdle parameter
for the put operation. See the
Evicting Specific Entries section.

Distributed Query

Hazelcast JCache

Added Using "this" as an
Attribute as a new section to
explain how you can use the
keyword "this" in your queries.
Defining a Custom ExpiryPolicy Updated the content to mention
about the method
setExpiryPolicy which
associates certain keys with
custom expiry policies.
JCache API

Added content related to eager
expiration.

5

Storage

Hot Backup

Added Starting the Cluster From
a Hot Backup as a new section
to explain how you can start
your cluster with data from a
hot backup.

Hazelcast Clients

Java Client

Added Configuring Client
Connection Retry as a new
section.

Serialization

Added Untrusted
Deserialization Protection as a
new section.

Management

Added Map Index Statistics as a
new section describing how to
access map index statistics.
Diagnostics

Added content for the new
OperationThreadSamples
plugin.
Added Toggle Scripting Support
as a new section.

Security

TLS/SSL

Performed a full review and
updated the content to include
the new properties keyStoreType
and trustStoreType.
Added Using BoringSSL as a
new section.

WAN

Added URL information to
synchronize all the maps in
source and target clusters.
Added Delta WAN
Synchronization as a new
section.
Configuring Consumer and
Event Filtering API

Added content related to the
new event type LOADED and
configuration element persistwan-replicated-data.
Added the REST URL for
clearing the event queues. See
the Queue Capacity section.

Hazelcast Plugins

Added as a new chapter to
describe the plugins using
which you can extend Hazelcast
IMDG’s functionalities.

Network Partitioning

Enhanced the Merge Types
section by adding the
descriptions of all merge types.

6

System Properties

Added definitions for the
following system properties:
* hazelcast.socket.buffer.direct

2. Getting Started
This chapter explains how to install Hazelcast and start a Hazelcast member and client. It describes
the executable files in the download package and also provides the fundamentals for configuring
Hazelcast and its deployment options.

2.1. Installation
The following sections explain the installation of Hazelcast IMDG and Hazelcast IMDG Enterprise. It
also includes notes and changes to consider when upgrading Hazelcast.

2.1.1. Installing Hazelcast IMDG
You can find Hazelcast in standard Maven repositories. If your project uses Maven, you do not need
to add additional repositories to your pom.xml or add hazelcast-.jar file into your
classpath (Maven does that for you). Just add the following lines to your pom.xml:



com.hazelcast
hazelcast
Hazelcast IMDG Version To Be Installed


As an alternative, you can download and install Hazelcast IMDG yourself. You only need to:
• Download

the

package

hazelcast-.zip

or

hazelcast-.tar.gz

from

hazelcast.org.
• Extract the downloaded hazelcast-.zip or hazelcast-.tar.gz.
• Add the file hazelcast-.jar to your classpath.

2.1.2. Installing Hazelcast IMDG Enterprise
There are two Maven repositories defined for Hazelcast IMDG Enterprise:

7


Hazelcast Private Snapshot Repository
https://repository-hazelcast-l337.forge.cloudbees.com/snapshot/


Hazelcast Private Release Repository
https://repository-hazelcast-l337.forge.cloudbees.com/release/

Hazelcast IMDG Enterprise customers may also define dependencies, a sample of which is shown
below.


com.hazelcast
hazelcast-enterprise
Hazelcast IMDG Enterprise Version To Be Installed


com.hazelcast
hazelcast-enterprise-all
Hazelcast IMDG Enterprise Version To Be Installed


2.1.3. Setting the License Key
Hazelcast IMDG Enterprise offers you two types of licenses: Enterprise and Enterprise HD. The
supported features differ in your Hazelcast setup according to the license type you own.
• Enterprise license: In addition to the open source edition of Hazelcast, Enterprise features are
the following:
◦ Security
◦ WAN Replication
◦ Clustered REST
◦ Clustered JMX
◦ Striim Hot Cache
◦ Rolling Upgrades
• Enterprise HD license: In addition to the Enterprise features, Enterprise HD features are the
following:
◦ High-Density Memory Store
◦ Hot Restart Persistence
To use Hazelcast IMDG Enterprise, you need to set the provided license key using one of the
configuration methods shown below.

8



Hazelcast IMDG Enterprise license keys are required only for members. You do not
need to set a license key for your Java clients for which you want to use IMDG
Enterprise features.

Declarative Configuration:
Add the below line to any place you like in the file hazelcast.xml. This XML file offers you a
declarative way to configure your Hazelcast. It is included in the Hazelcast download package.
When you extract the downloaded package, you will see the file hazelcast.xml under the /bin
directory.


...
Your Enterprise License Key
...

Programmatic Configuration:
Alternatively, you can set your license key programmatically as shown below.

Config config = new Config();
config.setLicenseKey( "Your Enterprise License Key" );
Spring XML Configuration:
If you are using Spring with Hazelcast, then you can set the license key using the Spring XML
schema, as shown below.


...
Your Enterprise License Key
...

JVM System Property:
As another option, you can set your license key using the below command (the "-D" command line
option).

-Dhazelcast.enterprise.license.key=Your Enterprise License Key

License Key Format
License keys have the following format:

9

##
The strings before the  is the human readable part. You can use your license key with
or without this human readable part. So, both the following example license keys are valid:

HazelcastEnterpriseHD#2Nodes#1q2w3e4r5t

1q2w3e4r5t

2.1.4. License Information
License information is available through the following Hazelcast APIs.
JMX
The MBean HazelcastInstance.LicenseInfo holds all the relative license details and can be accessed
through Hazelcast’s JMX port (if enabled).
• maxNodeCountAllowed: Maximum members allowed to form a cluster under the current license.
• expiryDate: Expiration date of the current license.
• typeCode: Type code of the current license.
• type: Type of the current license.
• ownerEmail: Email of the current licence’s owner.
• companyName: Company name on the current license.
Following is the list of license `type`s and `typeCode`s:

MANAGEMENT_CENTER(1, "Management Center"),
ENTERPRISE(0, "Enterprise"),
ENTERPRISE_SECURITY_ONLY(2, "Enterprise only with security"),
ENTERPRISE_HD(3, "Enterprise HD"),
CUSTOM(4, "Custom");

REST
You can access the license details by issuing a GET request through the REST API (if enabled) on the
/license resource, as shown below.

curl -v -X GET -H "Content-Type: text/plain" http://localhost:5701/hazelcast/licence
Its output is similar to the following:

10

Note: Unnecessary use of -X or --request, GET is already inferred.
*
Trying 127.0.0.1...
* TCP_NODELAY set
* Connected to localhost (127.0.0.1) port 5701 (#0)
> GET /hazelcast/licence HTTP/1.1
> Host: localhost:5701
> User-Agent: curl/7.58.0
> Accept: */*
> Content-Type: text/plain
>
< HTTP/1.1 200 OK
< Content-Type: text/plain
< Content-Length: 105
<
licenseInfo{"expiryDate":1560380399161,"maxNodeCount":10,"type":1,"companyName":"ExampleCompany","ownerEmail":"ExampleOwner"}

Logs
Besides the above approaches (JMX and REST) to access the license details, Hazelcast also starts to
log a license information banner into the log files when the license expiration is approaching.
During the last two months prior to the expiration, this license information banner is logged daily,
as a reminder to renew your license to avoid any interruptions. Once the expiration is due to a
month, the frequency of logging this banner becomes hourly (instead of daily). Lastly, when the
expiration is due in a week, this banner will be printed every 30 minutes.



Similar alerts are also present on the Hazelcast Management Center.

The banner has the following format:

@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ WARNING @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
HAZELCAST LICENSE WILL EXPIRE IN 29 DAYS.
Your Hazelcast cluster will stop working after this time.
Your license holder is customer@example-company.com, you should have them contact
our license renewal department, urgently on info@hazelcast.com
or call us on +1 (650) 521-5453
Please quote license id CUSTOM_TEST_KEY
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@



Please pay attention to the license warnings to prevent any possible interruptions
in the operation of your Hazelcast applications.

11

2.2. Supported Java Virtual Machines
Following table summarizes the version compatibility between Hazelcast IMDG and various
vendors' Java Virtual Machines (JVMs).
Table 2. Supported JVMs

Hazelcast IMDG Version

JDK
Oracle JDK
Version

IBM SDK,
Java
Technology
Edition

Azul Zing
JDK

Azul
Zulu
OpenJD
K

Up to 4.0

6









Up to current

7









Up to current

8









Fully 9









(JDK 6 support will be dropped
with the release of Hazelcast
IMDG 4.0)

• 3.11

and

newer:

supported.
• 3.10

and

older:

(JDK not
(JDK not
available yet) available yet)

Partially

supported.
• 3.11

and

newer:

Fully 10



supported.
• 3.10

and

older:

and

newer:

Fully 11

supported.
• 3.10

and

older:

Partially

supported.







(JDK not
(JDK not
available yet) available yet)

Partially

supported.
• 3.11











(JDK not
(JDK not
(JDK not
available yet) available yet) available yet)

Hazelcast IMDG 3.10 and older releases are not fully tested on JDK 9 and newer, so
there may be some features that are not working properly.
See the following sections for the details of Hazelcast IMDG supporting JDK 9 and
newer:



• Running in Modular Java: Talks about the new module system present in Java 9
and newer and how you can run a Hazelcast application on it.
• TLS/SSL for Hazelcast Members: Lists TLSv1.3, which comes with Java 11, as a
supported TLS version.

12

2.3. Running in Modular Java
Java project Jigsaw brought a new Module System into Java 9 and newer. Hazelcast supports
running in the modular environment. If you want to run your application with Hazelcast libraries
on the modulepath, use following module names:
• com.hazelcast.core for hazelcast-.jar and hazelcast-enterprise-.jar
• com.hazelcast.client for hazelcast-client-.jar and hazelcast-enterprise-client.jar
Don’t

use

hazelcast-all-.jar

or

hazelcast-enterprise-all-.jar

on

the

modulepath as it could lead to problems in module dependencies for your application. You can still
use them on the classpath.
The Java Module System comes with stricter visibility rules. It affects Hazelcast which uses internal
Java API to reach the best performance results.
Hazelcast needs java.se module and access to the following Java packages for a proper work:
• java.base/jdk.internal.ref
• java.base/java.nio (reflective access)
• java.base/sun.nio.ch (reflective access)
• java.base/java.lang (reflective access)
• jdk.management/com.sun.management.internal (reflective access)
• java.management/sun.management (reflective access)
You can provide the access to the above mentioned packages by using --add-exports and --add
-opens (for the reflective access) Java arguments.
Example: Running a member on the classpath

java --add-modules java.se \
--add-exports java.base/jdk.internal.ref=ALL-UNNAMED \
--add-opens java.base/java.lang=ALL-UNNAMED \
--add-opens java.base/java.nio=ALL-UNNAMED \
--add-opens java.base/sun.nio.ch=ALL-UNNAMED \
--add-opens java.management/sun.management=ALL-UNNAMED \
--add-opens jdk.management/com.sun.management.internal=ALL-UNNAMED \
-jar hazelcast-.jar
Example: Running a member on the modulepath

13

java --add-modules java.se \
--add-exports java.base/jdk.internal.ref=com.hazelcast.core \
--add-opens java.base/java.lang=com.hazelcast.core \
--add-opens java.base/java.nio=com.hazelcast.core \
--add-opens java.base/sun.nio.ch=com.hazelcast.core \
--add-opens java.management/sun.management=com.hazelcast.core \
--add-opens jdk.management/com.sun.management.internal=com.hazelcast.core \
--module-path lib \
--module com.hazelcast.core/com.hazelcast.core.server.StartServer
This example expects hazelcast-.jar placed in the lib directory.

2.4. Upgrading from 3.x
• Upgrading from 3.6.x to 3.7.x when using JCache: Hazelcast 3.7 introduced changes in JCache
implementation which broke compatibility of 3.6.x clients to 3.7-3.7.2 cluster members and vice
versa, so 3.7-3.7.2 clients are also incompatible with 3.6.x cluster members. This issue only
affects Java clients which use JCache functionality.
Starting with Hazelcast 3.7.3, a compatibility option is provided which can be used to ensure
backwards compatibility with 3.6.x clients.
In order to upgrade a 3.6.x cluster and clients to 3.7.3 (or later), you will need to use this
compatibility option on either the member or the client side, depending on which one is
upgraded first:
◦ first

upgrade

your

cluster

members

to

3.7.3,

adding

property

hazelcast.compatibility.3.6.client=true to your configuration; when started with this
property, cluster members are compatible with 3.6.x and 3.7.3+ clients but not with 3.7-3.7.2
clients. Once your cluster is upgraded, you may upgrade your applications to use client
version 3.7.3+.
◦ upgrade

your

clients

from

3.6.x

to

3.7.3,

adding

property

hazelcast.compatibility.3.6.server=true to your Hazelcast client configuration. A 3.7.3
client started with this compatibility option is compatible with 3.6.x and 3.7.3+ cluster
members but incompatible with 3.7-3.7.2 cluster members. Once your clients are upgraded,
you may then proceed to upgrade your cluster members to version 3.7.3 or later.
You may use any of the supported ways as described in the System Properties section to
configure the compatibility option. When done upgrading your cluster and clients, you may
remove the compatibility property from your Hazelcast member configuration.
• Upgrading from 3.6.x to 3.8.x EE when using JCache: Due to a compatibility problem
CacheConfig serialization may not work if your member is 3.8.x where x < 5. Hence, you will
need to use the 3.8.5 or higher version where the problem is being fixed.
• Introducing

the

spring-aware

element:

Before

the

release

3.5,

Hazelcast

uses

SpringManagedContext to scan SpringAware annotations by default. This may cause some
performance overhead for the users who do not use SpringAware. This behavior has been

14

changed with the release of Hazelcast 3.5. SpringAware annotations are disabled by default. By
introducing the spring-aware element, now it is possible to enable it by adding the  tag to the configuration. Please see the Spring Integration section.
• Introducing new configuration options for WAN replication: Starting with Hazelcast 3.6,
WAN replication related system properties, which are configured on a per member basis, can
now be configured per target cluster. The 4 system properties below are no longer valid.
◦ hazelcast.enterprise.wanrep.batch.size, please see the WAN Replication Batch Size.
◦ hazelcast.enterprise.wanrep.batchfrequency.seconds, please see the WAN Replication Batch
Maximum Delay.
◦ hazelcast.enterprise.wanrep.optimeout.millis, please see the WAN Replication Response
Timeout.
◦ hazelcast.enterprise.wanrep.queue.capacity, please see the WAN Replication Queue
Capacity.
• Removal of deprecated getId() method: The method getId() in the interface DistributedObject
has been removed. Please use the method getName() instead.
• Change in the Custom Serialization in the C++ Client Distribution: Before, the method
getTypeId() was used to retrieve the ID of the object to be serialized. Now, the method
getHazelcastTypeId() is used and you give your object as a parameter to this new method. Also,
getTypeId() was used in your custom serializer class, now it has been renamed to
getHazelcastTypeId() too. Note that, these changes also apply when you want to switch from
Hazelcast 3.6.1 to 3.6.2 too.
• Important note about Hazelcast System Properties: Even Hazelcast has not been
recommending the usage of GroupProperties.java class while benefiting from System Properties,
there has been a change to inform to the users who have been using this class. Starting with
Hazelcast 3.7, the class GroupProperties.java has been replaced by GroupProperty.java. In this
new class, system properties are instances of the newly introduced HazelcastProperty object.
You can access the names of these properties by calling getName() method of HazelcastProperty.
• Removal of WanNoDelayReplication: WanNoDelayReplication implementation of Hazelcast’s
WAN Replication has been removed starting with Hazelcast 3.7. You can still achieve this
behavior by setting the batch size to 1 while configuring the WanBatchReplication. Please refer
to the Defining WAN Replication section for more information.
• Introducing  element: Starting with Hazelcast 3.8, the configuration element


is

replaced

with

the

element



in

WAN

replication

configuration.
• WaitNotifyService interface has been renamed as OperationParker.
• Synchronizing WAN Target Cluster: Starting with Hazelcast 3.8 release, the URL for the REST
call

has

been

changed

from

http://member_ip:port/hazelcast/rest/wan/sync/map

to

http://member_ip:port/hazelcast/rest/mancenter/wan/sync/map.

2.5. Upgrading from 2.x
• Removal of deprecated static methods: The static methods of Hazelcast class reaching
Hazelcast data components have been removed. The functionality of these methods can be

15

reached from the HazelcastInstance interface. You should replace the following:

Map customers = Hazelcast.getMap( "customers" );
with

HazelcastInstance hazelcastInstance = Hazelcast.newHazelcastInstance();
// or if you already started an instance named "instance1"
// HazelcastInstance hazelcastInstance = Hazelcast.getHazelcastInstanceByName(
"instance1" );
Map customers = hazelcastInstance.getMap( "customers" );
• Renaming "instance" to "distributed object": Before 3.0 there was confusion about the term
"instance": it was used for both the cluster members and the distributed objects (map, queue,
topic, etc. instances). Starting with Hazelcast 3.0, the term instance will be only used for
Hazelcast instances, namely cluster members. We will use the term "distributed object" for map,
queue, etc. instances. You should replace the related methods with the new renamed ones. 3.0
clients are smart clients in that they know in which cluster member the data is located, so you
can replace your lite members with native clients.

public static void main( String[] args ) throws InterruptedException {
HazelcastInstance hazelcastInstance = Hazelcast.newHazelcastInstance();
IMap map = hazelcastInstance.getMap( "test" );
Collection instances = hazelcastInstance.getInstances();
for ( Instance instance : instances ) {
if ( instance.getInstanceType() == Instance.InstanceType.MAP ) {
System.out.println( "There is a map with name: " + instance.getId() );
}
}
}
with

public static void main( String[] args ) throws InterruptedException {
HazelcastInstance hazelcastInstance = Hazelcast.newHazelcastInstance();
IMap map = hz.getMap( "test" );
Collection objects = hazelcastInstance.getDistributedObjects(
);
for ( DistributedObject distributedObject : objects ) {
if ( distributedObject instanceof IMap ) {
System.out.println( "There is a map with name: " + distributedObject.getName
() );
}
}
}

16

• Package structure change: PartitionService has been moved to package com.hazelcast.core
from com.hazelcast.partition.
• Listener API change: Before 3.0, removeListener methods were taking the Listener object as a
parameter. But this caused confusion because same listener object may be used as a parameter
for different listener registrations. So we have changed the listener API. addListener methods
returns a unique ID and you can remove a listener by using this ID. So you should do the
following replacement if needed:

IMap map = hazelcastInstance.getMap( "map" );
map.addEntryListener( listener, true );
map.removeEntryListener( listener );
with

IMap map = hazelcastInstance.getMap( "map" );
String listenerId = map.addEntryListener( listener, true );
map.removeEntryListener( listenerId );
• IMap changes:
◦ tryRemove(K key, long timeout, TimeUnit timeunit) returns boolean indicating whether
operation is successful.
◦ tryLockAndGet(K key, long time, TimeUnit timeunit) is removed.
◦ putAndUnlock(K key, V value) is removed.
◦ lockMap(long time, TimeUnit timeunit) and unlockMap() are removed.
◦ getMapEntry(K key) is renamed as getEntryView(K key). The returned object’s type, MapEntry
class is renamed as EntryView.
◦ There is no predefined names for merge policies. You just give the full class name of the
merge policy implementation:

com.hazelcast.map.merge.PassThroughMergePolicy
Also MergePolicy interface has been renamed to MapMergePolicy and also returning null
from the implemented merge() method causes the existing entry to be removed.
• IQueue changes: There is no change on IQueue API but there are changes on how IQueue is
configured. With Hazelcast 3.0 there will be no backing map configuration for queue. Settings
like backup count will be directly configured on queue config. For queue configuration details,
please see the Queue section.
• Transaction API change: In Hazelcast 3.0, transaction API is completely different. Please see
the Transactions chapter.
• ExecutorService API change: Classes MultiTask and DistributedTask have been removed. All
the functionality is supported by the newly presented interface IExecutorService. Please see the

17

Executor Service section.
• LifeCycleService API: The lifecycle has been simplified. pause(), resume(), restart() methods
have been removed.
• AtomicNumber: AtomicNumber class has been renamed to IAtomicLong.
• ICountDownLatch: await() operation has been removed. We expect users to use await()
method with timeout parameters.
• ISemaphore API: The ISemaphore has been substantially changed. attach(), detach() methods
have been removed.
◦ In 2.x releases, the default value for max-size eviction policy was cluster_wide_map_size. In
3.x releases, default is PER_NODE. After upgrading, the max-size should be set according to
this new default, if it is not changed. Otherwise, it is likely that OutOfMemory exception may
be thrown.

2.6. Starting the Member and Client
Having installed Hazelcast, you can get started.
In this short tutorial, you perform the following activities.
1. Create a simple Java application using the Hazelcast distributed map and queue.
2. Run our application twice to have a cluster with two members (JVMs).
3. Connect to our cluster from another Java application by using the Hazelcast Native Java Client
API.
Let’s begin.
• The following code starts the first Hazelcast member and creates and uses the customers map
and queue.

Config cfg = new Config();
HazelcastInstance instance = Hazelcast.newHazelcastInstance(cfg);
Map mapCustomers = instance.getMap("customers");
mapCustomers.put(1, "Joe");
mapCustomers.put(2, "Ali");
mapCustomers.put(3, "Avi");
System.out.println("Customer with key 1: "+ mapCustomers.get(1));
System.out.println("Map Size:" + mapCustomers.size());
Queue queueCustomers = instance.getQueue("customers");
queueCustomers.offer("Tom");
queueCustomers.offer("Mary");
queueCustomers.offer("Jane");
System.out.println("First customer: " + queueCustomers.poll());
System.out.println("Second customer: "+ queueCustomers.peek());
System.out.println("Queue size: " + queueCustomers.size());

18

• Run this GettingStarted class a second time to get the second member started. The members
form a cluster and the output is similar to the following.

Members {size:2, ver:2} [
Member [127.0.0.1]:5701 - e40081de-056a-4ae5-8ffe-632caf8a6cf1 this
Member [127.0.0.1]:5702 - 93e82109-16bf-4b16-9c87-f4a6d0873080
]
Here, you can see the size of your cluster (size) and member list version (ver). The member list
version will be incremented when changes happen to the cluster, e.g., a member leaving from
or joining to the cluster.
The above member list format is introduced with Hazelcast 3.9. You can enable the legacy
member list format, which was used for the releases before Hazelcast 3.9, using the system
property

hazelcast.legacy.memberlist.format.enabled.

Please

see

the

System

Properties

appendix. The following is an example for the legacy member list format:

Members [2] {
Member [127.0.0.1]:5701 - c1ccc8d4-a549-4bff-bf46-9213e14a9fd2 this
Member [127.0.0.1]:5702 - 33a82dbf-85d6-4780-b9cf-e47d42fb89d4
}
• Now, add the hazelcast-client-.jar library to your classpath. This is required to use a
Hazelcast client.
• The following code starts a Hazelcast Client, connects to our cluster, and prints the size of the
customers map.

public class GettingStartedClient {
public static void main( String[] args ) {
ClientConfig clientConfig = new ClientConfig();
HazelcastInstance client = HazelcastClient.newHazelcastClient( clientConfig
);
IMap map = client.getMap( "customers" );
System.out.println( "Map Size:" + map.size() );
}
}
• When you run it, you see the client properly connecting to the cluster and printing the map size
as 3.
Hazelcast also offers a tool, Management Center, that enables you to monitor your cluster. You can
download it from Hazelcast website’s download page. You can use it to monitor your maps, queues
and other distributed data structures and members. Please see the Hazelcast Management Center
Reference Manual for usage explanations.

19

By default, Hazelcast uses multicast to discover other members that can form a cluster. If you are
working with other Hazelcast developers on the same network, you may find yourself joining their
clusters under the default settings. Hazelcast provides a way to segregate clusters within the same
network when using multicast. Please see the Creating Cluster Groups for more information.
Alternatively, if you do not wish to use the default multicast mechanism, you can provide a fixed
list of IP addresses that are allowed to join. Please see the Join configuration section for more
information.




Multicast mechanism is not recommended for production since UDP is often
blocked in production environments and other discovery mechanisms are more
definite. Please see the Discovery Mechanisms section.
You can also check the video tutorials here.

2.7. Using the Scripts In The Package
When you download and extract the Hazelcast ZIP or TAR.GZ package, you will see the following
scripts under the /bin folder that provide basic functionalities for member and cluster
management.
The following are the names and descriptions of each script:
• start.sh / start.bat: Starts a Hazelcast member with default configuration in the working
directory.
• stop.sh / stop.bat: Stops the Hazelcast member that was started in the current working
directory.
• cluster.sh: Provides basic functionalities for cluster management, such as getting and changing
the cluster state, shutting down the cluster or forcing the cluster to clean its persisted data and
make a fresh start. Please refer to the Using the Script cluster.sh section to learn the usage of
this script.



start.sh / start.bat scripts lets you start one Hazelcast instance per folder. To start



You can also use the start scripts to deploy your own library to a Hazelcast

a new instance, please unzip Hazelcast ZIP or TAR.GZ package in a new folder.

member. See the Adding User Library to CLASSPATH section.

2.8. Deploying On Amazon EC2
You can deploy your Hazelcast project onto an Amazon EC2 environment using Third Party tools
such as Vagrant and Chef.
You can find a sample deployment project (amazon-ec2-vagrant-chef) with step-by-step instructions
in the hazelcast-integration folder of the hazelcast-code-samples package, which you can
download at hazelcast.org. Please refer to this sample project for more information.

20

2.9. Deploying On Microsoft Azure
You can deploy your Hazelcast cluster onto a Microsoft Azure environment. For this, your cluster
should make use of Hazelcast Discovery Plugin for Microsoft Azure. You can find information about
this plugin on its GitHub repository at Hazelcast Azure.
For information on how to automatically deploy your cluster onto Azure, please see the
Deployment section of Hazelcast Azure plugin repository.

2.10. Deploying On Pivotal Cloud Foundry
Starting with Hazelcast 3.7, you can deploy your Hazelcast cluster onto Pivotal Cloud Foundry. It is
available as a Pivotal Cloud Foundry Tile which you can download at here. You can find the
installation

and

usage

instructions

and

the

release

notes

documents

at

https://docs.pivotal.io/partners/hazelcast/index.html.

2.11. Deploying using Docker
You can deploy your Hazelcast projects using the Docker containers. Hazelcast has the following
images on Docker:
• Hazelcast IMDG
• Hazelcast IMDG Enterprise
• Hazelcast Management Center
• Hazelcast OpenShift
After you pull an image from the Docker registry, you can run your image to start the Management
Center or a Hazelcast instance with Hazelcast’s default configuration. All repositories provide the
latest stable releases but you can pull a specific release too. You can also specify environment
variables when running the image.
If you want to start a customized Hazelcast instance, you can extend the Hazelcast image by
providing your own configuration file.
This feature is provided as a Hazelcast plugin. Please see its own GitHub repo at Hazelcast Docker
for details on configurations and usages.

3. Hazelcast Overview
Hazelcast is an open source In-Memory Data Grid (IMDG). It provides elastically scalable
distributed In-Memory computing, widely recognized as the fastest and most scalable approach to

21

application performance. Hazelcast does this in open source. More importantly, Hazelcast makes
distributed computing simple by offering distributed implementations of many developer-friendly
interfaces from Java such as Map, Queue, ExecutorService, Lock and JCache. For example, the Map
interface provides an In-Memory Key Value store which confers many of the advantages of NoSQL
in terms of developer friendliness and developer productivity.
In addition to distributing data In-Memory, Hazelcast provides a convenient set of APIs to access
the CPUs in your cluster for maximum processing speed. Hazelcast is designed to be lightweight
and easy to use. Since Hazelcast is delivered as a compact library (JAR) and since it has no external
dependencies other than Java, it easily plugs into your software solution and provides distributed
data structures and distributed computing utilities.
Hazelcast is highly scalable and available. Distributed applications can use Hazelcast for distributed
caching, synchronization, clustering, processing, pub/sub messaging, etc. Hazelcast is implemented
in Java and has clients for Java, C/C++, .NET, REST, Python, Go and Node.js. Hazelcast also speaks
Memcached protocol. It plugs into Hibernate and can easily be used with any existing database
system.
If you are looking for in-memory speed, elastic scalability and the developer friendliness of NoSQL,
Hazelcast is a great choice.
Hazelcast is Simple
Hazelcast is written in Java with no other dependencies. It exposes the same API from the familiar
Java util package, exposing the same interfaces. Just add hazelcast.jar to your classpath and you
can quickly enjoy JVMs clustering and start building scalable applications.
Hazelcast is Peer-to-Peer
Unlike many NoSQL solutions, Hazelcast is peer-to-peer. There is no master and slave; there is no
single point of failure. All members store equal amounts of data and do equal amounts of
processing. You can embed Hazelcast in your existing application or use it in client and server
mode where your application is a client to Hazelcast members.
Hazelcast is Scalable
Hazelcast is designed to scale up to hundreds and thousands of members. Simply add new
members and they will automatically discover the cluster and will linearly increase both memory
and processing capacity. The members maintain a TCP connection between each other and all
communication is performed through this layer.
Hazelcast is Fast
Hazelcast stores everything in-memory. It is designed to perform very fast reads and updates.
Hazelcast is Redundant
Hazelcast keeps the backup of each data entry on multiple members. On a member failure, the data
is restored from the backup and the cluster will continue to operate without downtime.

22

3.1. Sharding in Hazelcast
Hazelcast shards are called Partitions. By default, Hazelcast has 271 partitions. Given a key, we
serialize, hash and mod it with the number of partitions to find the partition which the key belongs
to. The partitions themselves are distributed equally among the members of the cluster. Hazelcast
also creates the backups of partitions and distributes them among members for redundancy.



Please refer to the Data Partitioning section for more information on how
Hazelcast partitions your data.

3.2. Hazelcast Topology
You can deploy a Hazelcast cluster in two ways: Embedded or Client/Server.
If you have an application whose main focal point is asynchronous or high performance computing
and lots of task executions, then Embedded deployment is the preferred way. In Embedded
deployment, members include both the application and Hazelcast data and services. The advantage
of the Embedded deployment is having a low-latency data access.
See the below illustration.

In the Client/Server deployment, Hazelcast data and services are centralized in one or more server
members and they are accessed by the application through clients. You can have a cluster of server
members that can be independently created and scaled. Your clients communicate with these
members to reach to Hazelcast data and services on them.
See the below illustration.

23

Hazelcast provides native clients (Java, .NET and C++), Memcache and REST clients, Scala, Python
and Node.js client implementations.
Client/Server deployment has advantages including more predictable and reliable Hazelcast
performance, easier identification of problem causes and, most importantly, better scalability.
When you need to scale in this deployment type, just add more Hazelcast server members. You can
address client and server scalability concerns separately.
Note that Hazelcast member libraries are available only in Java. Therefore, embedding a member
to a business service, it is only possible with Java. Applications written in other languages (.NET,
C++, Node.js, etc.) can use Hazelcast client libraries to access the cluster. See the Hazelcast Clients
chapter for information on the clients and other language implementations.
If you want low-latency data access, as in the Embedded deployment, and you also want the
scalability advantages of the Client/Server deployment, you can consider defining Near Caches for
your clients. This enables the frequently used data to be kept in the client’s local memory. Please
refer to the Configuring Client Near Cache section.

3.3. Why Hazelcast?
A Glance at Traditional Data Persistence
Data is at the core of software systems. In conventional architectures, a relational database persists
and provides access to data. Applications are talking directly with a database which has its backup
as another machine. To increase performance, tuning or a faster machine is required. This can cost
a large amount of money or effort.
There is also the idea of keeping copies of data next to the database, which is performed using
technologies like external key-value stores or second level caching that help offload the database.
However, when the database is saturated or the applications perform mostly "put" operations

24

(writes), this approach is of no use because it insulates the database only from the "get" loads
(reads). Even if the applications are read-intensive there can be consistency problems - when data
changes, what happens to the cache and how are the changes handled? This is when concepts like
time-to-live (TTL) or write-through come in.
In the case of TTL, if the access is less frequent than the TTL, the result will always be a cache miss.
On the other hand, in the case of write-through caches, if there are more than one of these caches
in a cluster, we again will have consistency issues. This can be avoided by having the members
communicate with each other so that entry invalidations can be propagated.
We can conclude that an ideal cache would combine TTL and write-through features. There are
several cache servers and in-memory database solutions in this field. However, these are standalone single instances with a distribution mechanism that is provided by other technologies to an
extent. So, we are back to square one; we experience saturation or capacity issues if the product is a
single instance or if consistency is not provided by the distribution.
And, there is Hazelcast
Hazelcast, a brand new approach to data, is designed around the concept of distribution. Hazelcast
shares data around the cluster for flexibility and performance. It is an in-memory data grid for
clustering and highly scalable data distribution.
One of the main features of Hazelcast is that it does not have a master member. Each cluster
member is configured to be the same in terms of functionality. The oldest member (the first
member created in the cluster) automatically performs the data assignment to cluster members. If
the oldest member dies, the second oldest member takes over.
You can come across with the term "master" or "master member" in some sections



of this manual. They are used for contextual clarification purposes; please
remember that they refer to the "oldest member" which is explained in the above
paragraph.

Another main feature of Hazelcast is that the data is held entirely in-memory. This is fast. In the
case of a failure, such as a member crash, no data will be lost since Hazelcast distributes copies of
the data across all the cluster members.
As shown in the feature list in the Distributed Data Structures chapter, Hazelcast supports a
number of distributed data structures and distributed computing utilities. These provide powerful
ways of accessing distributed clustered memory and accessing CPUs for true distributed computing.
Hazelcast’s Distinctive Strengths
• Hazelcast is open source.
• Hazelcast is only a JAR file. You do not need to install software.
• Hazelcast is a library, it does not impose an architecture on Hazelcast users.
• Hazelcast provides out-of-the-box distributed data structures, such as Map, Queue, MultiMap,
Topic, Lock and Executor.
• There is no "master," meaning no single point of failure in a Hazelcast cluster; each member in

25

the cluster is configured to be functionally the same.
• When the size of your memory and compute requirements increase, new members can be
dynamically joined to the Hazelcast cluster to scale elastically.
• Data is resilient to member failure. Data backups are distributed across the cluster. This is a big
benefit when a member in the cluster crashes as data will not be lost.
• Members are always aware of each other unlike in traditional key-value caching solutions.
• You can build your own custom-distributed data structures using the Service Programming
Interface (SPI) if you are not happy with the data structures provided.
Finally, Hazelcast has a vibrant open source community enabling it to be continuously developed.
Hazelcast is a fit when you need:
• analytic applications requiring big data processing by partitioning the data.
• to retain frequently accessed data in the grid.
• a cache, particularly an open source JCache provider with elastic distributed scalability.
• a primary data store for applications with utmost performance, scalability and low-latency
requirements.
• an In-Memory NoSQL Key Value Store.
• publish/subscribe communication at highest speed and scalability between applications.
• applications that need to scale elastically in distributed and cloud environments.
• a highly available distributed cache for applications.
• an alternative to Coherence and Terracotta.

3.4. Data Partitioning
As you read in the Sharding in Hazelcast section, Hazelcast shards are called Partitions. Partitions
are memory segments that can contain hundreds or thousands of data entries each, depending on
the memory capacity of your system. Each Hazelcast partition can have multiple replicas, which are
distributed among the cluster members. One of the replicas becomes the primary and other replicas
are called backups. Cluster member which owns primary replica of a partition is called partition
owner. When you read or write a particular data entry, you transparently talk to the owner of the
partition that contains the data entry.
By default, Hazelcast offers 271 partitions. When you start a cluster with a single member, it owns
all of 271 partitions (i.e., it keeps primary replicas for 271 partitions). The following illustration
shows the partitions in a Hazelcast cluster with single member.

26

When you start a second member on that cluster (creating a Hazelcast cluster with two members),
the partition replicas are distributed as shown in the illustration here.
Partition distributions in the below illustrations are shown for the sake of
simplicity and for descriptive purposes. Normally, the partitions are not



distributed in any order, as they are shown in these illustrations, but are
distributed randomly (they do not have to be sequentially distributed to each
member). The important point here is that Hazelcast equally distributes the
partition primaries and their backup replicas among the members.

In the illustration, the partition replicas with black text are primaries and the partition replicas
with blue text are backups. The first member has primary replicas of 135 partitions (black) and
each of these partitions are backed up in the second member (i.e., the second member owns the
backup replicas) (blue). At the same time, the first member also has the backup replicas of the
second member’s primary partition replicas.
As you add more members, Hazelcast moves some of the primary and backup partition replicas to
the new members one by one, making all members equal and redundant. Thanks to the consistent
hashing algorithm, only the minimum amount of partitions will be moved to scale out Hazelcast.
The following is an illustration of the partition replica distributions in a Hazelcast cluster with four
members.

27

Hazelcast distributes partitions' primary and backup replicas equally among the members of the
cluster. Backup replicas of the partitions are maintained for redundancy.



Your data can have multiple copies on partition primaries and backups, depending
on your backup count. Please see the Backing Up Maps section.

Starting with Hazelcast 3.6, lite members are introduced. Lite members are a new type of members
that do not own any partition. Lite members are intended for use in computationally-heavy task
executions and listener registrations. Although they do not own any partitions, they can access
partitions that are owned by other members in the cluster.



Please refer to the Enabling Lite Members section.

3.4.1. How the Data is Partitioned
Hazelcast distributes data entries into the partitions using a hashing algorithm. Given an object key
(for example, for a map) or an object name (for example, for a topic or list):
• the key or name is serialized (converted into a byte array)
• this byte array is hashed
• the result of the hash is mod by the number of partitions
The result of this modulo - MOD(hash result, partition count) - is the partition in which the data
will be stored, that is the partition ID. For ALL members you have in your cluster, the partition ID
for a given key will always be the same.

3.4.2. Partition Table
When you start a member, a partition table is created within it. This table stores the partition IDs
and the cluster members to which they belong. The purpose of this table is to make all members
(including lite members) in the cluster aware of this information, making sure that each member
knows where the data is.
The oldest member in the cluster (the one that started first) periodically sends the partition table to

28

all members. In this way each member in the cluster is informed about any changes to partition
ownership. The ownerships may be changed when, for example, a new member joins the cluster, or
when a member leaves the cluster.



If the oldest member of the cluster goes down, the next oldest member sends the
partition table information to the other ones.

You can configure the frequency (how often) that the member sends the partition table the
information by using the hazelcast.partition.table.send.interval system property. The property is
set to every 15 seconds by default.

3.4.3. Repartitioning
Repartitioning is the process of redistribution of partition ownerships. Hazelcast performs the
repartitioning in the following cases:
• When a member joins to the cluster.
• When a member leaves the cluster.
In these cases, the partition table in the oldest member is updated with the new partition
ownerships. Note that if a lite member joins or leaves a cluster, repartitioning is not triggered since
lite members do not own any partitions.

3.5. Use Cases
Hazelcast can be used:
• to share server configuration/information to see how a cluster performs.
• to cluster highly changing data with event notifications, e.g., user based events, and to queue
and distribute background tasks.
• as a simple Memcache with Near Cache.
• as a cloud-wide scheduler of certain processes that need to be performed on some members.
• to share information (user information, queues, maps, etc.) on the fly with multiple members in
different installations under OSGI environments.
• to share thousands of keys in a cluster where there is a web service interface on an application
server and some validation.
• as a distributed topic (publish/subscribe server) to build scalable chat servers for smartphones.
• as a front layer for a Cassandra back-end.
• to distribute user object states across the cluster, to pass messages between objects and to share
system data structures (static initialization state, mirrored objects, object identity generators).
• as a multi-tenancy cache where each tenant has its own map.
• to share datasets, e.g., table-like data structure, to be used by applications.
• to distribute the load and collect status from Amazon EC2 servers where the front-end is
developed using, for example, Spring framework.
29

• as a real-time streamer for performance detection.
• as storage for session data in web applications (enables horizontal scalability of the web
application).

3.6. Resources
• Hazelcast source code can be found at Github/Hazelcast.
• Hazelcast API can be found at Hazelcast.org/docs/Javadoc.
• Code samples can be downloaded from Hazelcast.org/download.
• More use cases and resources can be found at Hazelcast.com.
• Questions and discussions can be posted at the Hazelcast mail group.

4. Understanding Configuration
This chapter describes the options to configure your Hazelcast applications and explains the
utilities which you can make use of while configuring. You can configure Hazelcast using one or
mix of the following options:
• Declarative way
• Programmatic way
• Using Hazelcast system properties
• Within the Spring context
• Dynamically adding configuration on a running cluster

4.1. Configuring Declaratively
This is the configuration option where you use an XML configuration file. When you download and
unzip hazelcast-.zip, you will see the following files present in /bin folder, which are
standard XML-formatted configuration files:
• hazelcast.xml: Default declarative configuration file for Hazelcast. The configuration in this
XML file should be fine for most of the Hazelcast users. If not, you can tailor this XML file
according to your needs by adding/removing/modifying properties.
• hazelcast-full-example.xml: Configuration file which includes all Hazelcast configuration
elements and attributes with their descriptions. It is the "superset" of hazelcast.xml. You can use
hazelcast-full-example.xml as a reference document to learn about any element or attribute, or
you can change its name to hazelcast.xml and start to use it as your Hazelcast configuration file.
A part of hazelcast.xml is shown as an example below.

30


dev

http://localhost:8080/mancenter

5701


0



224.2.2.3
54327


127.0.0.1

127.0.0.1






4.1.1. Composing Declarative Configuration
You can compose the declarative configuration of your Hazelcast member or Hazelcast client from
multiple declarative configuration snippets. In order to compose a declarative configuration, you
can use the  element to load different declarative configuration files.
Let’s say you want to compose the declarative configuration for Hazelcast out of two
configurations: development-group-config.xml and development-network-config.xml. These two
configurations are shown below.
development-group-config.xml:



dev



31

development-network-config.xml:



5701


224.2.2.3
54327




To get your example Hazelcast declarative configuration out of the above two, use the 
element as shown below.





This feature also applies to the declarative configuration of Hazelcast client. Please see the
following examples.
client-group-config.xml:



dev


client-network-config.xml:




127.0.0.1:7000



To get a Hazelcast client declarative configuration from the above two examples, use the 
element as shown below.

32








Use  element on top level of the XML hierarchy.

Using the element , you can also load XML resources from classpath and file system:


 
 

The element  supports variables too. Please see the following example snippet:








You can refer to the Using Variables section to learn how you can set the
configuration elements with variables.

4.2. Configuring Programmatically
Besides declarative configuration, you can configure your cluster programmatically. For this you
can create a Config object, set/change its properties and attributes and use this Config object to
create a new Hazelcast member. Following is an example code which configures some network and
Hazelcast Map properties.

Config config = new Config();
config.getNetworkConfig().setPort( 5900 )
.setPortAutoIncrement( false );
MapConfig mapConfig = new MapConfig();
mapConfig.setName( "testMap" )
.setBackupCount( 2 )
.setTimeToLiveSeconds( 300 );
To create a Hazelcast member with the above example configuration, pass the configuration object
as shown below:

33

HazelcastInstance hazelcast = Hazelcast.newHazelcastInstance( config );

The Config must not be modified after the Hazelcast instance is started. In other



words, all configuration must be completed before creating the HazelcastInstance.
Certain additional configuration elements can be added at runtime as described in
the Dynamically Adding Data Structure Configuration on a Cluster section.

You can also create a named Hazelcast member. In this case, you should set instanceName of Config
object as shown below:

Config config = new Config();
config.setInstanceName( "my-instance" );
Hazelcast.newHazelcastInstance( config );
To retrieve an existing Hazelcast member by its name, use the following:

Hazelcast.getHazelcastInstanceByName( "my-instance" );
To retrieve all existing Hazelcast members, use the following:

Hazelcast.getAllHazelcastInstances();

Hazelcast performs schema validation through the file hazelcast-config-



.xsd which comes with your Hazelcast libraries. Hazelcast throws a
meaningful exception if there is an error in the declarative or programmatic
configuration.

If you want to specify your own configuration file to create Config, Hazelcast supports several ways
including filesystem, classpath, InputStream and URL:
• Config cfg = new XmlConfigBuilder(xmlFileName).build();
• Config cfg = new XmlConfigBuilder(inputStream).build();
• Config cfg = new ClasspathXmlConfig(xmlFileName);
• Config cfg = new FileSystemXmlConfig(configFilename);
• Config cfg = new UrlXmlConfig(url);
• Config cfg = new InMemoryXmlConfig(xml);

4.3. Configuring with System Properties
You can use system properties to configure some aspects of Hazelcast. You set these properties as
name and value pairs through declarative configuration, programmatic configuration or JVM
system property. Following are examples for each option.

34

Declaratively:

....

value
....


Programmatically:

Config config = new Config() ;
config.setProperty( "hazelcast.property.foo", "value" );
Using JVM’s System class or -D argument:
System.setProperty( "hazelcast.property.foo", "value" );
or
java -Dhazelcast.property.foo=value
You will see Hazelcast system properties mentioned throughout this Reference Manual as required
in some of the chapters and sections. All Hazelcast system properties are listed in the System
Properties appendix with their descriptions, default values and property types as a reference for
you.

4.4. Configuring within Spring Context
If you use Hazelcast with Spring you can declare beans using the namespace hazelcast. When you
add the namespace declaration to the element beans in the Spring context file, you can start to use
the namespace shortcut hz to be used as a bean declaration. Following is an example Hazelcast
configuration when integrated with Spring:








10.10.1.2, 10.10.1.3






35

Please see the Integration with Spring section for more information on Hazelcast-Spring
integration.

4.5. Dynamically Adding Data Structure Configuration
on a Cluster
As described above, Hazelcast can be configured in a declarative or programmatic way;
configuration must be completed before starting a Hazelcast member and this configuration cannot
be altered at runtime, thus we refer to this as static configuration.
Starting with Hazelcast 3.9, it is possible to dynamically add configuration for certain data
structures at runtime; these can be added by invoking one of the Config.add*Config methods on the
Config object obtained from a running member’s HazelcastInstance.getConfig() method. For
example:

Config config = new Config();
MapConfig mapConfig = new MapConfig("sessions");
config.addMapConfig(mapConfig);
HazelcastInstance instance = Hazelcast.newHazelcastInstance(config);
MapConfig noBackupsMap = new MapConfig("dont-backup").setBackupCount(0);
instance.getConfig().addMapConfig(noBackupsMap);
Dynamic configuration elements must be fully configured before the invocation of add*Config
method: at that point, the configuration object will be delivered to every member of the cluster and
added to each member’s dynamic configuration, so mutating the configuration object after the
add*Config invocation will have no effect.
As dynamically added data structure configuration is propagated across all cluster members,
failures may occur due to conditions such as timeout and network partition. The configuration
propagation mechanism internally retries adding the configuration whenever a membership
change is detected. However if an exception is thrown from add*Config method, the configuration
may have been partially propagated to some cluster members and adding the configuration should
be retried by the user.
Adding new dynamic configuration is supported for all add*Config methods except:
• JobTracker which has been deprecated since Hazelcast 3.8
• QuorumConfig: new quorum configuration cannot be dynamically added but other configuration
can reference quorums configured in the existing static configuration
• WanReplicationConfig: new WAN replication configuration cannot be dynamically added,
however existing static ones can be referenced from other configurations, e.g., a new dynamic
MapConfig may include a WanReplicationRef to a statically configured WAN replication config.
• ListenerConfig: listeners can be instead added at runtime via other API such as
HazelcastInstance.getCluster().addMembershipListener
HazelcastInstance.getPartitionService().addMigrationListener.

36

and

4.5.1. Handling Configuration Conflicts
Attempting to add a dynamic configuration, when a static configuration for the same element
already exists, will throw ConfigurationException. For example, assuming we start a member with
the following fragment in hazelcast.xml configuration:


Then adding a dynamic configuration for a map with the name sessions will throw a
ConfigurationException:

HazelcastInstance instance = Hazelcast.newHazelcastInstance();
MapConfig sessionsMapConfig = new MapConfig("sessions");
// this will throw ConfigurationException:
instance.getConfig().addMapConfig(sessionsMapConfig);
When attempting to add dynamic configuration for an element for which dynamic configuration
has already been added, then if a configuration conflict is detected a ConfigurationException will be
thrown. For example:

HazelcastInstance instance = Hazelcast.newHazelcastInstance();
MapConfig sessionsMapConfig = new MapConfig("sessions").setBackupCount(0);
instance.getConfig().addMapConfig(sessionsMapConfig);
MapConfig sessionsWithBackup = new MapConfig("sessions").setBackupCount(1);
// throws ConfigurationException because the new MapConfig conflicts with existing one
instance.getConfig().addMapConfig(sessionsWithBackup);
MapConfig sessionsWithoutBackup = new MapConfig("sessions").setBackupCount(0);
// does not throw exception: new dynamic config is equal to existing dynamic config of
same name
instance.getConfig().addMapConfig(sessionsWithoutBackup);

4.6. Checking Configuration
When you start a Hazelcast member without passing a Config object, as explained in the
Configuring Programmatically section, Hazelcast checks the member’s configuration as follows:
• First, it looks for the hazelcast.config system property. If it is set, its value is used as the path.
This is useful if you want to be able to change your Hazelcast configuration; you can do this
because it is not embedded within the application. You can set the config option with the

37

following command:

-Dhazelcast.config=`*`
The path can be a regular one or a classpath reference with the prefix classpath:.
• If the above system property is not set, Hazelcast then checks whether there is a hazelcast.xml
file in the working directory.
• If not, it then checks whether hazelcast.xml exists on the classpath.
• If none of the above works, Hazelcast loads the default configuration (hazelcast.xml) that comes
with your Hazelcast package.
Before configuring Hazelcast, please try to work with the default configuration to see if it works for
you. This default configuration should be fine for most of the users. If not, you can consider to
modify the configuration to be more suitable for your environment.

4.7. Configuration Pattern Matcher
You can give a custom strategy to match an item name to a configuration pattern. By default
Hazelcast uses a simplified wildcard matching. See Using Wildcards section for this. A custom
configuration pattern matcher can be given by using either member or client config objects, as
shown below:

// Setting a custom config pattern matcher via member config object
Config config = new Config();
config.setConfigPatternMatcher(new ExampleConfigPatternMatcher());
And the following is an example pattern matcher:

class ExampleConfigPatternMatcher extends MatchingPointConfigPatternMatcher {
@Override
public String matches(Iterable configPatterns, String itemName) throws
ConfigurationException {
String matches = super.matches(configPatterns, itemName);
if (matches == null) throw new ConfigurationException("No config found for " +
itemName);
return matches;
}
}

4.8. Using Wildcards
Hazelcast supports wildcard configuration for all distributed data structures that can be configured
using Config, that is, for all except IAtomicLong, IAtomicReference. Using an asterisk (*) character in

38

the name, different instances of maps, queues, topics, semaphores, etc. can be configured by a
single configuration.
A single asterisk (*) can be placed anywhere inside the configuration name.
For instance, a map named com.hazelcast.test.mymap can be configured using one of the following.








Or a queue com.hazelcast.test.myqueue:


...



...


39

• You can use only a single asterisk as a wildcard for each data structure
configuration.
• If you have matching wildcard configurations for a data structure, the most
specific (longest) one is used when configuring it. Let’s say you have a map
named mymap.customer.name and you have map configurations mymap.* and



mymap.customer.*. Hazelcast uses mymap.customer.* to configure this map.
As another example, assume you have a map named mymap.customer.name and
you have map configurations mymap.*.name and mymap.customer.*. Hazelcast
uses mymap.customer.* to configure this map. As you see, the longest character
length before the asterisk makes it the most specific, so it wins the
configuration.

4.9. Using Variables
In your Hazelcast and/or Hazelcast Client declarative configuration, you can use variables to set the
values of the elements. This is valid when you set a system property programmatically or you use
the command line interface. You can use a variable in the declarative configuration to access the
values of the system properties you set.
For example, see the following command that sets two system properties.

-Dgroup.name=dev
Let’s get the values of these system properties in the declarative configuration of Hazelcast, as
shown below.



${group.name}


This also applies to the declarative configuration of Hazelcast Client, as shown below.



${group.name}


If you do not want to rely on the system properties, you can use the XmlConfigBuilder and explicitly
set a Properties instance, as shown below.

40

Properties properties = new Properties();
// fill the properties, e.g., from database/LDAP, etc.
XmlConfigBuilder builder = new XmlConfigBuilder();
builder.setProperties(properties);
Config config = builder.build();
HazelcastInstance hz = Hazelcast.newHazelcastInstance(config);

4.10. Variable Replacers
Variable replacers are used to replace custom strings during loading the configuration, e.g., they
can be used to mask sensitive information such as usernames and passwords. Of course their usage
is not limited to security related information.
Variable replacers implement the interface com.hazelcast.config.replacer.spi.ConfigReplacer and
they are configured only declaratively: in the Hazelcast’s declarative configuration files, i.e.,
hazelcast.xml and hazelcast-client.xml. You can refer to ConfigReplacer s Javadoc for basic
information on how a replacer works.
Variable replacers are configured within the element  under , as
shown below.


...



value
...




...

As you can see,  is the parent element for your replacers, which are declared
using the  sub-elements. You can define multiple replacers under the .
Here are the descriptions of elements and attributes used for the replacer configuration:
• fail-if-value-missing: Specifies whether the loading configuration process stops when a
replacement value is missing. It is an optional attribute and its default value is true.
• class-name: Full class name of the replacer.
• : Contains names and values of the properties used to configure a replacer. Each
property is defined using the  sub-element. All of the properties are explained in the

41

upcoming sections.
The following replacer classes are provided by Hazelcast as example implementations of the
ConfigReplacer interface. Note that you can also implement your own replacers.
• EncryptionReplacer
• PropertyReplacer



There is also a ExecReplacer which runs an external command and uses its
standard output as the value for the variable. Please refer to its code sample.

Each example replacer is explained in the below sections.

4.10.1. EncryptionReplacer
This example EncryptionReplacer replaces encrypted variables by its plain form. The secret key for
encryption/decryption is generated from a password which can be a value in a file and/or
environment specific values, such as MAC address and actual user data.
Its full class name is com.hazelcast.config.replacer.EncryptionReplacer and the replacer prefix is
ENC. Here are the properties used to configure this example replacer:
• cipherAlgorithm: Cipher algorithm used for the encryption/decryption. Its default value is AES.
• keyLengthBits: Length of the secret key to be generated in bits. Its default value is 128 bits.
• passwordFile: Path to a file whose content should be used as a part of the encryption password.
When the property is not provided no file is used as a part of the password. Its default value is
null.
• passwordNetworkInterface: Name of network interface whose MAC address should be used as a
part of the encryption password. When the property is not provided no network interface
property is used as a part of the password. Its default value is null.
• passwordUserProperties: Specifies whether the current user properties (user.name and user.home)
should be used as a part of the encryption password. Its default value is true.
• saltLengthBytes: Length of a random password salt in bytes. Its default value is 8 bytes.
• secretKeyAlgorithm: Name of the secret-key algorithm to be associated with the generated secret
key. Its default value is AES.
• secretKeyFactoryAlgorithm: Algorithm used to generate a secret key from a password. Its default
value is PBKDF2WithHmacSHA256.
• securityProvider: Name of a Java Security Provider to be used for retrieving the configured
secret key factory and the cipher. Its default value is null.



Older Java versions may not support all the algorithms used as defaults. Please use
the property values supported your Java version.

As a usage example, let’s create a password file and generate the encrypted strings out of this file.
1 - Create the password file: echo '/Za-uG3dDfpd,5.-' > /opt/master-password

42

2 - Define the encrypted variables:

java -cp hazelcast-*.jar \
-DpasswordFile=/opt/master-password \
-DpasswordUserProperties=false \
com.hazelcast.config.replacer.EncryptionReplacer \
"aGroup"
$ENC{Gw45stIlan0=:531:yVN9/xQpJ/Ww3EYkAPvHdA==}
java -cp hazelcast-*.jar \
-DpasswordFile=/opt/master-password \
-DpasswordUserProperties=false \
com.hazelcast.config.replacer.EncryptionReplacer \
"aPasswordToEncrypt"
$ENC{wJxe1vfHTgg=:531:WkAEdSi//YWEbwvVNoU9mUyZ0DE49acJeaJmGalHHfA=}
3 - Configure the replacer and put the encrypted variables into the configuration:





/opt/master-password
false




$ENC{Gw45stIlan0=:531:yVN9/xQpJ/Ww3EYkAPvHdA==}

$ENC{wJxe1vfHTgg=:531:WkAEdSi/YWEbwvVNoU9mUyZ0DE49acJeaJmGalHHfA=}


4 - Check if the decryption works:

java -jar hazelcast-*.jar
Apr 06, 2018 10:15:43 AM com.hazelcast.config.XmlConfigLocator
INFO: Loading 'hazelcast.xml' from working directory.
Apr 06, 2018 10:15:44 AM com.hazelcast.instance.AddressPicker
INFO: [LOCAL] [aGroup] [3.10-SNAPSHOT] Prefer IPv4 stack is true.
As you can see in the logs, the correctly decrypted group name value ("aGroup") is used.

4.10.2. PropertyReplacer
The PropertyReplacer replaces variables by properties with the given name. Usually the system

43

properties are used, e.g., ${user.name}. There is no need to define it in the declarative configuration
files.
Its full class name is com.hazelcast.config.replacer.PropertyReplacer and the replacer prefix is
empty string ("").

4.10.3. Implementing Custom Replacers
You can also provide your own replacer implementations. All replacers have to implement the
interface com.hazelcast.config.replacer.spi.ConfigReplacer. A simple snippet is shown below.

public interface ConfigReplacer {
void init(Properties properties);
String getPrefix();
String getReplacement(String maskedValue);
}

5. Setting Up Clusters
This chapter describes Hazelcast clusters and the methods cluster members and native clients use
to form a Hazelcast cluster.

5.1. Discovery Mechanisms
A Hazelcast cluster is a network of cluster members that run Hazelcast. Cluster members
automatically join together to form a cluster. This automatic joining takes place with various
discovery mechanisms that the cluster members use to find each other.
Please note that, after a cluster is formed, communication between cluster members is always via
TCP/IP, regardless of the discovery mechanism used.
Hazelcast uses the following discovery mechanisms.



You can refer to the Hazelcast IMDG Deployment and Operations Guide for advice
on the best discovery mechanism to use.

5.1.1. TCP
You can configure Hazelcast to be a full TCP/IP cluster. Please see the Discovering Members by TCP
section for configuration details.

5.1.2. Multicast
Multicast mechanism is not recommended for production since UDP is often blocked in production
environments and other discovery mechanisms are more definite.
With this mechanism, Hazelcast allows cluster members to find each other using multicast

44

communication. Please see the Discovering Members by Multicast section.

5.1.3. AWS Cloud Discovery
Hazelcast supports EC2 auto-discovery. It is useful when you do not want to provide or you cannot
provide the list of possible IP addresses. This discovery feature is provided as a Hazelcast plugin.
Please see its documentation for information on configuring and using it.

5.1.4. GCP Cloud Discovery
Hazelcast supports discovering members in the GCP Compute Engine environment. You can easily
configure Hazelcast members discovery, WAN replication, and Hazelcast Client to work seamlessly
on the native GCP VM Instances. This discovery feature is provided as a Hazelcast plugin. Please see
its documentation for information on configuring and using it.

5.1.5. Apache jclouds® Cloud Discovery
Hazelcast members and native clients support jclouds® for discovery. This mechanism allows
applications to be deployed in various cloud infrastructure ecosystems in an infrastructureagnostic way. This discovery feature is provided as a Hazelcast plugin. Please see its documentation
for information on configuring and using it.

5.1.6. Azure Cloud Discovery
Hazelcast offers a discovery strategy for your Hazelcast applications running on Azure. This
strategy provides all of your Hazelcast instances by returning the virtual machines within your
Azure resource group that are tagged with a specified value. This discovery feature is provided as a
Hazelcast plugin. Please see its documentation for information on configuring and using it.

5.1.7. Zookeeper Cloud Discovery
This discovery mechanism provides a service based discovery strategy by using Apache Curator to
communicate with your Zookeeper server. You can use this plugin with Discovery SPI enabled
Hazelcast 3.6.1 and higher applications. This is provided as a Hazelcast plugin. Please see its
documentation for information on configuring and using it.

5.1.8. Consul Cloud Discovery
Consul is a highly available and distributed service discovery and key-value store designed with
support for the modern data center to make distributed systems and configuration easy. This
mechanism provides a Consul based discovery strategy for Hazelcast enabled applications
(Hazelcast 3.6 and higher) and enables Hazelcast members to dynamically discover one another via
Consul. This discovery feature is provided as a Hazelcast plugin. Please see its documentation for
information on configuring and using it.

5.1.9. etcd Cloud Discovery
This mechanism provides an etcd based discovery strategy for Hazelcast enabled applications
(Hazelcast 3.6 and higher). This is an easy to configure plug-and-play Hazelcast discovery strategy
45

that will optionally register each of your Hazelcast members with etcd and enable Hazelcast
members to dynamically discover one another via etcd. This discovery feature is provided as a
Hazelcast plugin. Please see its documentation for information on configuring and using it.

5.1.10. Hazelcast for PCF
Using a clickable Hazelcast Tile for Pivotal Cloud Foundry (PCF), you can deploy your Hazelcast
cluster on PCF. This feature is provided as a Hazelcast plugin. Please see its documentation on how
to install, configure and use the plugin Hazelcast for PCF.

5.1.11. Hazelcast OpenShift Integration
Hazelcast can run inside OpenShift benefiting from its cluster management software Kubernetes
for discovery of members. Using Hazelcast Docker images, templates and default configuration
files, you can deploy Hazelcast IMDG, Hazelcast IMDG Enterprise and Management Center onto
OpenShift. Please see the documentation:
• Hazelcast IMDG and Hazelcast IMDG Enterprise
• Management Center
Please also see the Hazelcast for OpenShift guide, which presents how to set up local OpenShift
environment, start Hazelcast cluster, configure Management Center and finally run a sample client
application.

5.1.12. Eureka Cloud Discovery
Eureka is a REST based service that is primarily used in the AWS cloud for locating services for the
purpose of load balancing and failover of middle-tier servers. Hazelcast supports Eureka V1
discovery; Hazelcast members within EC2 Virtual Private Cloud can discover each other using this
mechanism. This discovery feature is provided as a Hazelcast plugin. Please see its documentation.

5.1.13. Heroku Cloud Discovery
Heroku is a platform as a service (PaaS) with which you can build, run and operate applications
entirely in the cloud. It is a cloud platform based on a managed container system, with integrated
data services and a powerful ecosystem. Hazelcast offers a discovery plugin that looks for IP
addresses of other members by resolving service names against the Heroku DNS Discovery in
Heroku Private Spaces. This discovery feature is provided as a Hazelcast plugin. Please see its
documentation.

5.1.14. Kubernetes Cloud Discovery
Kubernetes is an open source system for automating deployment, scaling and management of
containerized applications. Hazelcast provides Kubernetes discovery mechanism that looks for IP
addresses of other members by resolving the requests against a Kubernetes Service Discovery
system. It supports two different options of resolving against the discovery registry: (i) a request to
the REST API, (ii) DNS Lookup against a given DNS service name. This discovery feature is provided
as a Hazelcast plugin. Please see its documentation for information on configuring and using it.

46

5.2. Discovering Members by TCP
If multicast is not the preferred way of discovery for your environment, then you can configure
Hazelcast to be a full TCP/IP cluster. When you configure Hazelcast to discover members by TCP/IP,
you must list all or a subset of the members' hostnames and/or IP addresses as cluster members.
You do not have to list all of these cluster members, but at least one of the listed members has to be
active in the cluster when a new member joins.
To set your Hazelcast to be a full TCP/IP cluster, set the following configuration elements. Please
refer to the tcp-ip element section for the full description of the TCP/IP discovery configuration
elements.
• Set the enabled attribute of the multicast element to "false".
• Set the enabled attribute of the aws element to "false".
• Set the enabled attribute of the tcp-ip element to "true".
• Set your member elements within the tcp-ip element.
The following is an example declarative configuration.


...

...




machine1
machine2
machine3:5799
192.168.1.0-7
192.168.1.21

...

...

...

As shown above, you can provide IP addresses or hostnames for member elements. You can also give
a range of IP addresses, such as 192.168.1.0-7.
Instead of providing members line-by-line as shown above, you also have the option to use the
members element and write comma-separated IP addresses, as shown below.
192.168.1.0-7,192.168.1.21

47

If you do not provide ports for the members, Hazelcast automatically tries the ports 5701, 5702 and
so on.
By default, Hazelcast binds to all local network interfaces to accept incoming traffic. You can
change this behavior using the system property hazelcast.socket.bind.any. If you set this property
to false, Hazelcast uses the interfaces specified in the interfaces element (please refer to the
Interfaces Configuration section). If no interfaces are provided, then it will try to resolve one
interface to bind from the member elements.

5.3. Discovering Members by Multicast
With the multicast auto-discovery mechanism, Hazelcast allows cluster members to find each other
using multicast communication. The cluster members do not need to know the concrete addresses
of the other members, as they just multicast to all the other members for listening. Whether
multicast is possible or allowed depends on your environment.
To set your Hazelcast to multicast auto-discovery, set the following configuration elements. Please
refer to the multicast element section for the full description of the multicast discovery
configuration elements.
• Set the enabled attribute of the multicast element to "true".
• Set multicast-group, multicast-port, multicast-time-to-live, etc. to your multicast values.
• Set the enabled attribute of both tcp-ip and aws elements to "false".
The following is an example declarative configuration.


...

...


224.2.2.3
54327
32
2

192.168.1.102








Pay attention to the multicast-timeout-seconds element. multicast-timeout-seconds specifies the

48

time in seconds that a member should wait for a valid multicast response from another member
running in the network before declaring itself the leader member (the first member joined to the
cluster) and creating its own cluster. This only applies to the startup of members where no leader
has been assigned yet. If you specify a high value to multicast-timeout-seconds, such as 60 seconds,
it means that until a leader is selected, each member will wait 60 seconds before moving on. Be
careful when providing a high value. Also, be careful not to set the value too low, or the members
might give up too early and create their own cluster.



Multicast auto-discovery is not supported for Hazelcast native clients yet.
However, we offer Multicast Discovery Plugin for this purpose. Please refer to the
Discovering Native Clients section.

5.4. Discovering Native Clients
Hazelcast members and native Java clients can find each other with multicast discovery plugin.
This plugin is implemented using Hazelcast Discovery SPI. You should configure the plugin both at
Hazelcast members and Java clients in order to use multicast discovery.
To configure your cluster to have the multicast discovery plugin, follow these steps:
• Disable the multicast and TCP/IP join mechanisms. To do this, set the enabled attributes of the
multicast and tcp-ip elements to false in your hazelcast.xml configuration file
• Set the enabled attribute of the hazelcast.discovery.enabled property to true.
• Add multicast discovery strategy configuration to your XML file, i.e., 
element.
The following is an example declarative configuration.

49

...

true

....








224.2.2.3
54327




...
The following are the multicast discovery plugin configuration properties with their descriptions.
• group: String value that is used to set the multicast group, so that you can isolate your clusters.
• port: Integer value that is used to set the multicast port.

5.5. Creating Cluster Groups
You can create cluster groups. To do this, use the group configuration element.
You can separate your clusters in a simple way by specifying group names. Example groupings can
be by development, production, test, app, etc. The following is an example declarative
configuration.



production

...

You can also define the cluster groups using the programmatic configuration. A JVM can host
multiple Hazelcast instances. Each Hazelcast instance can only participate in one group. Each
Hazelcast instance only joins to its own group and does not interact with other groups. The
following code example creates three separate Hazelcast instances--h1 belongs to the production
cluster, while h2 and h3 belong to the development cluster.

50

Config configProd = new Config();
configProd.getGroupConfig().setName( "production" );
Config configDev = new Config();
configDev.getGroupConfig().setName( "development" );
HazelcastInstance h1 = Hazelcast.newHazelcastInstance( configProd );
HazelcastInstance h2 = Hazelcast.newHazelcastInstance( configDev );
HazelcastInstance h3 = Hazelcast.newHazelcastInstance( configDev );

5.5.1. Cluster Groups before Hazelcast 3.8.2
If you have a Hazelcast release older than 3.8.2, you need to provide also a group password along
with the group name. The following are the configuration examples with the password element:



production
prod-pass

...


Config configProd = new Config();
configProd.getGroupConfig().setName( "production" ).setPassword( "prod-pass" );
Config configDev = new Config();
configDev.getGroupConfig().setName( "development" ).setPassword( "dev-pass" );
HazelcastInstance h1 = Hazelcast.newHazelcastInstance( configProd );
HazelcastInstance h2 = Hazelcast.newHazelcastInstance( configDev );
HazelcastInstance h3 = Hazelcast.newHazelcastInstance( configDev );
Starting with 3.8.2, there is no need for a group password, both on member and client sides.

5.6. Member User Code Deployment - BETA
Hazelcast can dynamically load your custom classes or domain classes from a remote class
repository, which typically includes lite members. For this purpose Hazelcast offers a distributed
dynamic class loader.
Using this dynamic class loader, you can control the local caching of the classes loaded from other
members, control the classes to be served to other members and create blacklists or whitelists of
classes and packages. When you enable this feature, you will not have to deploy your classes to all
cluster members.

51

The following is the brief working mechanism of the User Code Deployment feature:
1. Dynamic class loader first checks the local classes, i.e., your classpath, for your custom class. If it
is there, Hazelcast does not try to load it from the remote class repository.
2. Then, it checks the cache of classes loaded from the remote class repository (for this, caching
should have been enabled in your local, please refer to Configuring User Code Deployment
section). If your class is found here, again, Hazelcast does not try to load it from the remote class
repository.
3. Finally, dynamic class loader checks the remote class repository. If a member in this repository
returns the class, it means your class is found and will be used. You can also put this class into
your local class cache as mentioned in the previous step.

5.6.1. Configuring User Code Deployment
User Code Deployment feature is not enabled by default. You can configure this feature
declaratively or programmatically. Following are example configuration snippets:
Declarative Configuration


ETERNAL
LOCAL_CLASSES_ONLY
com.foo
com.bar.MyClass
HAS_ATTRIBUTE:lite

Programmatic Configuration

Config config = new Config();
UserCodeDeploymentConfig distCLConfig = config.getUserCodeDeploymentConfig();
distCLConfig.setEnabled( true )
.setClassCacheMode( UserCodeDeploymentConfig.ClassCacheMode.ETERNAL )
.setProviderMode( UserCodeDeploymentConfig.ProviderMode.LOCAL_CLASSES_ONLY )
.setBlacklistedPrefixes( "com.foo" )
.setWhitelistedPrefixes( "com.bar.MyClass" )
.setProviderFilter( "HAS_ATTRIBUTE:lite" );
User Code Deployment has the following configuration elements and attributes:
• enabled: Specifies whether dynamic class loading is enabled or not. Its default value is "false"
and it is a mandatory attribute.
• : Controls the local caching behavior for the classes loaded from the remote
class repository. Available values are as follows:
◦ ETERNAL: Cache the loaded classes locally. This is the default value and suitable when you
load long-living objects, such as domain objects stored in a map.

52

◦ OFF: Do not cache the loaded classes locally. It is suitable for loading runnables, callables,
entry processors, etc.
• : Controls how the classes are served to the other cluster members. Available
values are as follows:
◦ LOCAL_AND_CACHED_CLASSES: Serve classes loaded from both local classpath and from other
members. This is the default value.
◦ LOCAL_CLASSES_ONLY: Serve classes from the local classpath only. Classes loaded from other
members will be used locally, but they are not served to other members.
◦ OFF: Never serve classes to other members.
• : Comma separated name prefixes of classes/packages to be prevented
from dynamic class loading. For example, if you set it as "com.foo", remote loading of all classes
from the "com.foo" package will be blacklisted, including the classes from all its sub-packages. If
you set it as "com.foo.Class", then the "Class" and all classes having the "Class" as prefix in the
"com.foo" package will be blacklisted. There are some built-in prefixes which are blacklisted by
default. These are as follows:
◦ javax.
◦ java.
◦ sun.
◦ com.hazelcast.
• : Comma separated name prefixes of classes/packages only from which the
classes will be loaded. It allows to quickly configure remote loading only for classes from
selected packages. It can be used together with blacklisting. For example, you can whitelist the
prefix "com.foo" and blacklist the prefix "com.foo.secret".
• : Filter to constraint members to be used for a class loading request when a
class is not available locally. The value is in the format "HAS_ATTRIBUTE:foo". When it is set as
"HAS_ATTRIBUTE:foo", the class loading request will only be sent to the members which have
"foo" as a member attribute. Setting this to null will allow to load classes from all members.
Please see an example in the below section.

5.6.2. Example for Filtering Members
As described above, the configuration element provider-filter is used to constrain a member to
load classes only from a subset of all cluster members. The value of the provider-filter must be set
as a member attribute in the desired members from which the classes will be loaded. Please see the
following example usage provided as programmatic configurations.
The below example configuration will allow the Hazelcast member to load classes only from
members with the class-provider attribute set. It will not ask any other member to provide a locally
unavailable class:

53

Config hazelcastConfig = new Config();
DistributedClassloadingConfig distributedClassloadingConfig = hazelcastConfig
.getDistributedClassloadingConfig();
distributedClassloadingConfig.setProviderFilter("HAS_ATTRIBUTE:class-provider");
HazelcastInstance instance = Hazelcast.newHazelcastInstance(hazelcastConfig);
And the below example configuration sets the attribute class-provider for a member. So, the above
member will load classes from the members who have the attribute class-provider:

Config hazelcastConfig = new Config();
MemberAttributeConfig memberAttributes = hazelcastConfig.getMemberAttributeConfig();
memberAttributes.setAttribute("class-provider", "true");
HazecastInstance instance = Hazelcast.newHazelcastInstance(hazelcastConfig);

5.7. Client User Code Deployment - BETA
You can use the User Code Deployment at the client side for the following situations:
1. You have objects that will run on the cluster via the clients such as Runnable, Callable and Entry
Processors.
2. You have new or amended user domain objects (in-memory format of the IMap set to Object)
which need to be deployed into the cluster.
When this feature is enabled, the clients will deploy these classes to the members. By this way,
when a client adds a new class, the members will not require restarts to include the new classes in
classpath.
You can also use the client permission policy to specify which clients are permitted to use User Code
Deployment. Please see the Permissions.

5.7.1. Configuring Client User Code Deployment
Client User Code Deployment feature is not enabled by default. You can configure this feature
declaratively or programmatically. Following are example configuration snippets:
Declarative Configuration
In your hazelcast-client.xml:

54



/User/sample/sample.jar
sample.jar 
https://com.sample.com/sample.jar
file://Users/sample/sample.jar



sample.ClassName
sample.ClassName2


Programmatic Configuration

ClientConfig clientConfig = new ClientConfig();
ClientUserCodeDeploymentConfig clientUserCodeDeploymentConfig = new
ClientUserCodeDeploymentConfig();
clientUserCodeDeploymentConfig.addJar("/User/sample/sample.jar");
clientUserCodeDeploymentConfig.addJar("https://com.sample.com/sample.jar");
clientUserCodeDeploymentConfig.addClass("sample.ClassName");
clientUserCodeDeploymentConfig.addClass("sample.ClassName2");
clientUserCodeDeploymentConfig.setEnabled(true);
clientConfig.setUserCodeDeploymentConfig(clientUserCodeDeploymentConfig);

Important to Know
Note that User Code Deployment should also be enabled on the members to use this feature.

Config config = new Config();
UserCodeDeploymentConfig userCodeDeploymentConfig = config.
getUserCodeDeploymentConfig();
userCodeDeploymentConfig.setEnabled( true );
Please refer to the Member User Code Deployment section for more information on enabling it on
the member side and its configuration properties.
For the property class-cache-mode, Client User Code Deployment supports only the ETERNAL mode,
regardless of the configuration set at the member side (which can be ETERNAL and OFF).
For

the

property,

provider-mode,

Client

User

Code

Deployment

supports

only

the

LOCAL_AND_CACHED_CLASSES mode, regardless of the configuration set at the member side (which can
be LOCAL_AND_CACHED_CLASSES, LOCAL_CLASSES_ONLY and OFF).

55

The remaining properties, which are blacklist-prefixes, whitelist-prefixes and provider-filter
configured at the member side, will effect the client user code deployment’s behavior too. For
example, assuming that you provide com.foo as a blacklist prefix at the member side, the member
will discard the classes with the prefix com.foo loaded by the client.

5.7.2. Adding User Library to CLASSPATH
When you want to use a Hazelcast feature in a non-Java client, you need to make sure that the
Hazelcast member recognizes it. For this, you can use the /user-lib directory that comes with the
Hazelcast package and deploy your own library to the member. Let’s say you use Hazelcast Node.js
client and want to use an entry processor. This processor should be IdentifiedDataSerializable or
Portable in the Node.js client. You need to implement the Java equivalents of the processor and its
factory on the member side, and put these compiled class or JAR files into the /user-lib directory.
Then you can run the start.sh script which adds them to the classpath.
The following is an example code which can be the Java equivalent of entry processor in Node.js
client:

56

import com.hazelcast.map.AbstractEntryProcessor;
import com.hazelcast.nio.ObjectDataInput;
import com.hazelcast.nio.ObjectDataOutput;
import com.hazelcast.nio.serialization.IdentifiedDataSerializable;
import java.io.IOException;
import java.util.Map;
public class IdentifiedEntryProcessor extends AbstractEntryProcessor
implements IdentifiedDataSerializable {
static final int CLASS_ID = 1;
private String value;
public IdentifiedEntryProcessor() {
}
@Override
public int getFactoryId() {
return IdentifiedFactory.FACTORY_ID;
}
@Override
public int getId() {
return CLASS_ID;
}
@Override
public void writeData(ObjectDataOutput out) throws IOException {
out.writeUTF(value);
}
@Override
public void readData(ObjectDataInput in) throws IOException {
value = in.readUTF();
}
@Override
public Object process(Map.Entry entry) {
entry.setValue(value);
return value;
}
}
You can implement the above processor’s factory as follows:

57

import com.hazelcast.nio.serialization.DataSerializableFactory;
import com.hazelcast.nio.serialization.IdentifiedDataSerializable;
public class IdentifiedFactory implements DataSerializableFactory {
public static final int FACTORY_ID = 5;
@Override
public IdentifiedDataSerializable create(int typeId) {
if (typeId == IdentifiedEntryProcessor.CLASS_ID) {
return new IdentifiedEntryProcessor();
}
return null;
}
}
And the following is the configuration for the above factory:





IdentifiedFactory




Then, you can start your Hazelcast member by using the start scripts (start.sh or start.bat) in the
/bin directory. The start scripts automatically adds your class and JAR files to the classpath.

5.8. Partition Group Configuration
Hazelcast distributes key objects into partitions using the consistent hashing algorithm. Multiple
replicas are created for each partition and those partition replicas are distributed among Hazelcast
members. An entry is stored in the members that own replicas of the partition to which the entry’s
key is assigned. The total partition count is 271 by default; you can change it with the configuration
property hazelcast.partition.count. Please see the System Properties appendix.
Hazelcast member that owns the primary replica of a partition is called as partition owner. Other
replicas are called backups. Based on the configuration, a key object can be kept in multiple
replicas of a partition. A member can hold at most one replica of a partition (ownership or backup).
By default, Hazelcast distributes partition replicas randomly and equally among the cluster
members, assuming all members in the cluster are identical. But what if some members share the
same JVM or physical machine or chassis and you want backups of these members to be assigned to
members in another machine or chassis? What if processing or memory capacities of some
members are different and you do not want an equal number of partitions to be assigned to all
members?

58

To deal with such scenarios, you can group members in the same JVM (or physical machine) or
members located in the same chassis. Or you can group members to create identical capacity. We
call these groups partition groups. Partitions are assigned to those partition groups instead of
individual members. Backup replicas of a partition which is owned by a partition group are located
in other partition groups.

5.8.1. Grouping Types
When you enable partition grouping, Hazelcast presents the following choices for you to configure
partition groups.
HOST_AWARE
You can group members automatically using the IP addresses of members, so members sharing the
same network interface will be grouped together. All members on the same host (IP address or
domain name) will be a single partition group. This helps to avoid data loss when a physical server
crashes, because multiple replicas of the same partition are not stored on the same host. But if there
are multiple network interfaces or domain names per physical machine, that will make this
assumption invalid.
Following are declarative and programmatic configuration snippets that show how to enable
HOST_AWARE grouping.



Config config = ...;
PartitionGroupConfig partitionGroupConfig = config.getPartitionGroupConfig();
partitionGroupConfig.setEnabled( true )
.setGroupType( MemberGroupType.HOST_AWARE );

CUSTOM
You can do custom grouping using Hazelcast’s interface matching configuration. This way, you can
add different and multiple interfaces to a group. You can also use wildcards in the interface
addresses. For example, the users can create rack-aware or data warehouse partition groups using
custom partition grouping.
Following are declarative and programmatic configuration examples that show how to enable and
use CUSTOM grouping.

59



10.10.0.*
10.10.3.*
10.10.5.*


10.10.10.10-100
10.10.1.*
10.10.2.*



Config config = new Config();
PartitionGroupConfig partitionGroupConfig = config.getPartitionGroupConfig();
partitionGroupConfig.setEnabled( true )
.setGroupType( PartitionGroupConfig.MemberGroupType.CUSTOM );
MemberGroupConfig memberGroupConfig = new MemberGroupConfig();
memberGroupConfig.addInterface( "10.10.0.*" )
.addInterface( "10.10.3.*" ).addInterface("10.10.5.*" );
MemberGroupConfig memberGroupConfig2 = new MemberGroupConfig();
memberGroupConfig2.addInterface( "10.10.10.10-100" )
.addInterface( "10.10.1.*").addInterface( "10.10.2.*" );
partitionGroupConfig.addMemberGroupConfig( memberGroupConfig );
partitionGroupConfig.addMemberGroupConfig( memberGroupConfig2 );

While your cluster was forming, if you configured your members to discover each



other by their IP addresses, you should use the IP addresses for the 
element. If your members discovered each other by their hostnames, you should
use the hostnames.

PER_MEMBER
You can give every member its own group. Each member is a group of its own and primary and
backup partitions are distributed randomly (not on the same physical member). This gives the least
amount of protection and is the default configuration for a Hazelcast cluster. This grouping type
provides good redundancy when Hazelcast members are on separate hosts. However, if multiple
instances run on the same host, this type is not a good option.
Following are declarative and programmatic configuration snippets that show how to enable
PER_MEMBER grouping.



60

Config config = ...;
PartitionGroupConfig partitionGroupConfig = config.getPartitionGroupConfig();
partitionGroupConfig.setEnabled( true )
.setGroupType( MemberGroupType.PER_MEMBER );

ZONE_AWARE
You can use ZONE_AWARE configuration with Hazelcast AWS, Hazelcast GCP, Hazelcast jclouds or
Hazelcast Azure Discovery Service plugins.
As discovery services, these plugins put zone information to the Hazelcast member attributes map
during the discovery process. When ZONE_AWARE is configured as partition group type, Hazelcast
creates the partition groups with respect to member attributes map entries that include zone
information. That means backups are created in the other zones and each zone will be accepted as
one partition group.



When using the ZONE_AWARE partition grouping, a Hazelcast cluster spanning
multiple AZs should have an equal number of members in each AZ. Otherwise, it
will result in uneven partition distribution among the members.

This is the list of supported attributes which is set by Discovery Service plugins during a Hazelcast
member start-up:
• hazelcast.partition.group.zone: For the zones in the same area.
• hazelcast.partition.group.rack: For different racks in the same zone.
• hazelcast.partition.group.host: For a shared physical member if virtualization is used.
hazelcast-jclouds offers rack or host information in addition to zone information



based on cloud provider. In such cases, Hazelcast looks for zone, rack and host
information in the given order and create partition groups with available
information*

Following are declarative and programmatic configuration snippets that show how to enable
ZONE_AWARE grouping.



Config config = ...;
PartitionGroupConfig partitionGroupConfig = config.getPartitionGroupConfig();
partitionGroupConfig.setEnabled( true )
.setGroupType( MemberGroupType.ZONE_AWARE );

SPI
You can provide your own partition group implementation using the SPI configuration. To create

61

your partition group implementation, you need to first extend the DiscoveryStrategy class of the
discovery

service

plugin,

getPartitionGroupStrategy()

and

override
return

the
the

method

public

PartitionGroupStrategy

PartitionGroupStrategy
configuration

in

that

overridden method.
Following is a sample code covering the implementation steps mentioned in the above paragraph:

public class CustomDiscovery extends AbstractDiscoveryStrategy {
public CustomDiscovery(ILogger logger, Map properties) {
super(logger, properties);
}
@Override
public Iterable discoverNodes() {
Iterable iterable = //your implementation
return iterable;
}
@Override
public PartitionGroupStrategy getPartitionGroupStrategy() {
return new CustomPartitionGroupStrategy();
}
private class CustomPartitionGroupStrategy implements PartitionGroupStrategy {
@Override
public Iterable getMemberGroups() {
Iterable iterable = //your implementation
return iterable;
}
}
}

5.9. Logging Configuration
Hazelcast has a flexible logging configuration and does not depend on any logging framework
except JDK logging. It has built-in adapters for a number of logging frameworks and it also supports
custom loggers by providing logging interfaces.
To use built-in adapters, set the hazelcast.logging.type property to one of the predefined types
below.
• jdk: JDK logging (default)
• log4j: Log4j
• log4j2: Log4j2
• slf4j: Slf4j
• none: disable logging

62

You can set hazelcast.logging.type through declarative configuration, programmatic configuration,
or JVM system property.



If you choose to use log4j, log4j2, or slf4j, you should include the proper
dependencies in the classpath.

Declarative Configuration

....

log4j
....


Programmatic Configuration

Config config = new Config() ;
config.setProperty( "hazelcast.logging.type", "log4j" );
System Property
• Using JVM parameter: java -Dhazelcast.logging.type=slf4j
• Using System class: System.setProperty( "hazelcast.logging.type", "none" );
If the provided logging mechanisms are not satisfactory, you can implement your own using the
custom logging feature. To use it, implement the com.hazelcast.logging.LoggerFactory and
com.hazelcast.logging.ILogger interfaces and set the system property hazelcast.logging.class as
your custom LoggerFactory class name.

-Dhazelcast.logging.class=foo.bar.MyLoggingFactory
You can also listen to logging events generated by Hazelcast runtime by registering LogListeners to
LoggingService.

LogListener listener = new LogListener() {
public void log( LogEvent logEvent ) {
// do something
}
};
HazelcastInstance instance = Hazelcast.newHazelcastInstance();
LoggingService loggingService = instance.getLoggingService();
loggingService.addLogListener( Level.INFO, listener );
Through the LoggingService, you can get the currently used ILogger implementation and log your

63

own messages too.



If you are not using command line for configuring logging, you should be careful
about Hazelcast classes. They may be defaulted to jdk logging before newly
configured logging is read. When logging mechanism is selected, it will not change.

Below are example configurations for Log4j2 and Log4j. Note that Hazelcast does not recommend
any specific logging library, these examples are provided only to demonstrate how to configure the
logging. You can use your custom logging as explained above.

5.9.1. Example Log4j2 Configuration
Specify the logging type as Log4j2 and a separate logging configuration file as shown below.
Using JVM arguments:

-Dhazelcast.logging.type=log4j2
-Dlog4j.configurationFile=/path/to/properties/log4j2.properties
Using declarative configuration (hazelcast.xml):


...

log4j2
/path/to/properties/log4j2.properties<
/property>
...

...

Following is an example log4j2.properties file:

64

rootLogger=file
rootLogger.level=info
property.filepath=/path/to/log/files
property.filename=hazelcast
appender.file.type=RollingFile
appender.file.name=RollingFile
appender.file.fileName=${filepath}/${filename}.log
appender.file.filePattern=${filepath}/${filename}-%d{yyyy-MM-dd}-%i.log.gz
appender.file.layout.type=PatternLayout
appender.file.layout.pattern = %d{yyyy-MM-dd HH:mm:ss} %-5p %c{1}:%L - %m%n
appender.file.policies.type=Policies
appender.file.policies.time.type=TimeBasedTriggeringPolicy
appender.file.policies.time.interval=1
appender.file.policies.time.modulate=true
appender.file.policies.size.type=SizeBasedTriggeringPolicy
appender.file.policies.size.size=50MB
appender.file.strategy.type=DefaultRolloverStrategy
appender.file.strategy.max=100
rootLogger.appenderRefs=file
rootLogger.appenderRef.file.ref=RollingFile
#Hazelcast specific logs.
#log4j.logger.com.hazelcast=debug
#log4j.logger.com.hazelcast.cluster=debug
#log4j.logger.com.hazelcast.partition=debug
#log4j.logger.com.hazelcast.partition.InternalPartitionService=debug
#log4j.logger.com.hazelcast.nio=debug
#log4j.logger.com.hazelcast.hibernate=debug
To enable the debug logs for all Hazelcast operations uncomment the below line in the above
configuration file:

log4j.logger.com.hazelcast=debug
If you do not need detailed logs, the default settings is enough. Using the Hazelcast specific lines in
the above configuration file, you can select to see specific logs (cluster, partition, hibernate, etc.) in
desired levels.

5.9.2. Example Log4j Configuration
Its configuration is similar to that of Log4j2. Below is the JVM argument way of specifying the
logging type and configuration file:

65

-Dhazelcast.logging.type=log4j
-Dlog4j.configuration=file:/path/to/properties/log4j.properties
Following is an example log4j.properties file:

log4j.rootLogger=INFO,file
log4j.appender.file=org.apache.log4j.RollingFileAppender
log4j.appender.file.File=/path/to/log/files/hazelcast.log
log4j.appender.file.layout=org.apache.log4j.PatternLayout
log4j.appender.file.layout.ConversionPattern=%d{yyyy-MM-dd HH:mm:ss} %p [%c{1}] - %m%n
log4j.appender.file.maxFileSize=50MB
log4j.appender.file.maxBackupIndex=100
log4j.appender.file.threshold=DEBUG
#log4j.logger.com.hazelcast=debug
#log4j.logger.com.hazelcast.cluster=debug
#log4j.logger.com.hazelcast.partition=debug
#log4j.logger.com.hazelcast.partition.InternalPartitionService=debug
#log4j.logger.com.hazelcast.nio=debug
#log4j.logger.com.hazelcast.hibernate=debug

5.10. Other Network Configurations
All network related configurations are performed via the network element in the Hazelcast XML
configuration file or the class NetworkConfig when using programmatic configuration. Following
subsections describe the available configurations that you can perform under the network element.

5.10.1. Public Address
public-address overrides the public address of a member. By default, a member selects its socket
address as its public address. But behind a network address translation (NAT), two endpoints
(members) may not be able to see/access each other. If both members set their public addresses to
their defined addresses on NAT, then that way they can communicate with each other. In this case,
their public addresses are not an address of a local network interface but a virtual address defined
by NAT. It is optional to set and useful when you have a private cloud. Note that, the value for this
element should be given in the format host IP address:port number. See the following examples.
Declarative:


11.22.33.44:5555

Programmatic:

66

Config config = new Config();
config.getNetworkConfig()
.setPublicAddress( "11.22.33.44:5555" );

5.10.2. Port
You can specify the ports that Hazelcast will use to communicate between cluster members. Its
default value is 5701. The following are example configurations.
Declarative:


5701

Programmatic:

Config config = new Config();
config.getNetworkConfig().setPort( 5701 )
.setPortAutoIncrement( true ).setPortCount( 20 );
According to the above example, Hazelcast will try to find free ports between 5701 and 5720.
port has the following attributes.
• port-count: By default, Hazelcast will try 100 ports to bind. Meaning that, if you set the value of
port as 5701, as members are joining to the cluster, Hazelcast tries to find ports between 5701
and 5801. You can choose to change the port count in the cases like having large instances on a
single machine or willing to have only a few ports to be assigned. The parameter port-count is
used for this purpose, whose default value is 100.
• auto-increment: In some cases you may want to choose to use only one port. In that case, you can
disable the auto-increment feature of port by setting auto-increment to false. The port-count
attribute is not used when auto-increment feature is disabled.

5.10.3. Outbound Ports
By default, Hazelcast lets the system pick up an ephemeral port during socket bind operation. But
security policies/firewalls may require you to restrict outbound ports to be used by Hazelcastenabled applications. To fulfill this requirement, you can configure Hazelcast to use only defined
outbound ports. The following are example configurations.
Declarative:

67




33000-35000

37000,37001,37002,37003
38000,38500-38600


Programmatic:

...
NetworkConfig networkConfig = config.getNetworkConfig();
// ports between 35000 and 35100
networkConfig.addOutboundPortDefinition("35000-35100");
// comma separated ports
networkConfig.addOutboundPortDefinition("36001, 36002, 36003");
networkConfig.addOutboundPort(37000);
networkConfig.addOutboundPort(37001);
...



You can use port ranges and/or comma separated ports.

As shown in the programmatic configuration, you use the method addOutboundPort to add only one
port. If you need to add a group of ports, then use the method addOutboundPortDefinition.
In the declarative configuration, the element ports can be used for both single and multiple port
definitions. When you set this element to 0 or *, your operating system (not Hazelcast) will select a
free port from the ephemeral range.

5.10.4. Reuse Address
When you shutdown a cluster member, the server socket port will be in the TIME_WAIT state for the
next couple of minutes. If you start the member right after shutting it down, you may not be able to
bind it to the same port because it is in the TIME_WAIT state. If you set the reuse-address element to
true, the TIME_WAIT state is ignored and you can bind the member to the same port again.
The following are example configurations.
Declarative:


true

Programmatic:
68

...
NetworkConfig networkConfig = config.getNetworkConfig();
networkConfig.setReuseAddress( true );
...

5.10.5. Join
The join configuration element is used to discover Hazelcast members and enable them to form a
cluster. Hazelcast provides multicast, TCP/IP, EC2 and jclouds® discovery mechanisms. These
mechanisms are explained the Discovery Mechanisms section. This section describes all the subelements and attributes of join element. The following are example configurations.
Declarative:




224.2.2.3
54327
32
2

192.168.1.102



192.168.1.104
192.168.1.104
192.168.1.105,192.168.1.106


my-access-key
my-secret-key
us-west-1
ec2.amazonaws.com
hazelcast-sg
type
hz-members






Programmatic:

69

Config config = new Config();
NetworkConfig network = config.getNetworkConfig();
JoinConfig join = network.getJoin();
join.getMulticastConfig().setEnabled( false )
.addTrustedInterface( "192.168.1.102" );
join.getTcpIpConfig().addMember( "10.45.67.32" ).addMember( "10.45.67.100" )
.setRequiredMember( "192.168.10.100" ).setEnabled( true );
The join element has the following sub-elements and attributes.
multicast element
The multicast element includes parameters to fine tune the multicast join mechanism.
• enabled: Specifies whether the multicast discovery is enabled or not, true or false.
• multicast-group: The multicast group IP address. Specify it when you want to create clusters
within the same network. Values can be between 224.0.0.0 and 239.255.255.255. Its default value
is 224.2.2.3.
• multicast-port: The multicast socket port that the Hazelcast member listens to and sends
discovery messages through. Its default value is 54327.
• multicast-time-to-live: Time-to-live value for multicast packets sent out to control the scope of
multicasts. See more information here.
• multicast-timeout-seconds: Only when the members are starting up, this timeout (in seconds)
specifies the period during which a member waits for a multicast response from another
member. For example, if you set it as 60 seconds, each member will wait for 60 seconds until a
leader member is selected. Its default value is 2 seconds.
• trusted-interfaces: Includes IP addresses of trusted members. When a member wants to join to
the cluster, its join request will be rejected if it is not a trusted member. You can give an IP
addresses range using the wildcard (*) on the last digit of IP address, e.g., 192.168.1.\* or
192.168.1.100-110.
tcp-ip element
The tcp-ip element includes parameters to fine tune the TCP/IP join mechanism.
• enabled: Specifies whether the TCP/IP discovery is enabled or not. Values can be true or false.
• required-member: IP address of the required member. Cluster will only formed if the member
with this IP address is found.
• member: IP address(es) of one or more well known members. Once members are connected to
these well known ones, all member addresses will be communicated with each other. You can
also give comma separated IP addresses using the members element.


70

tcp-ip element also accepts the interface parameter. Please refer to the
Interfaces element description.

• connection-timeout-seconds: Defines the connection timeout in seconds. This is the maximum
amount of time Hazelcast is going to try to connect to a well known member before giving up.
Setting it to a too low value could mean that a member is not able to connect to a cluster. Setting
it to a too high value means that member startup could slow down because of longer timeouts,
for example when a well known member is not up. Increasing this value is recommended if you
have many IPs listed and the members cannot properly build up the cluster. Its default value is
5 seconds.
aws element
The aws element includes parameters to allow the members to form a cluster on the Amazon EC2
environment.
• enabled: Specifies whether the EC2 discovery is enabled or not, true or false.
• access-key, secret-key: Access and secret keys of your account on EC2.
• region: The region where your members are running. Its default value is us-east-1. You need to
specify this if the region is other than the default one.
• host-header: The URL that is the entry point for a web service. It is optional.
• security-group-name: Name of the security group you specified at the EC2 management console.
It is used to narrow the Hazelcast members to be within this group. It is optional.
• tag-key, tag-value: To narrow the members in the cloud down to only Hazelcast members, you
can set these parameters as the ones you specified in the EC2 console. They are optional.
• connection-timeout-seconds: The maximum amount of time, in seconds, Hazelcast will try to
connect to a well known member before giving up. Setting this value too low could mean that a
member is not able to connect to a cluster. Setting the value too high means that member
startup could slow down because of longer timeouts (for example, when a well known member
is not up). Increasing this value is recommended if you have many IPs listed and the members
cannot properly build up the cluster. Its default value is 5 seconds.
If you are using a cloud provider other than AWS, you can use the programmatic configuration to
specify a TCP/IP cluster. The members will need to be retrieved from that provider, e.g., jclouds.
discovery-strategies element
The discovery-strategies element configures internal or external discovery strategies based on the
Hazelcast Discovery SPI. For further information, please refer to the Discovery SPI section and the
vendor documentation of the used discovery strategy.

5.10.6. AWSClient Configuration
To make sure EC2 instances are found correctly, you can use the AWSClient class. It determines the
private IP addresses of EC2 instances to be connected. Give the AWSClient class the values for the
parameters that you specified in the aws element, as shown below. You will see whether your EC2
instances are found.

71

public static void main( String[] args )throws Exception{
AwsConfig config = new AwsConfig();
config.setSecretKey( ... ) ;
config.setSecretKey( ... );
config.setRegion( ... );
config.setSecurityGroupName( ... );
config.setTagKey( ... );
config.setTagValue( ... );
config.setEnabled( true );
AWSClient client = new AWSClient( config );
Collection ipAddresses = client.getPrivateIpAddresses();
System.out.println( "addresses found:" + ipAddresses );
for ( String ip: ipAddresses ) {
System.out.println( ip );
}
}

5.10.7. Interfaces
You can specify which network interfaces that Hazelcast should use. Servers mostly have more than
one network interface, so you may want to list the valid IPs. Range characters ('*' and '-') can be
used for simplicity. For instance, 10.3.10.\* refers to IPs between 10.3.10.0 and 10.3.10.255. Interface
10.3.10.4-18 refers to IPs between 10.3.10.4 and 10.3.10.18 (4 and 18 included). If network interface
configuration is enabled (it is disabled by default) and if Hazelcast cannot find a matching interface,
then it will print a message on the console and will not start on that member.
The following are example configurations.
Declarative:


...

...

10.3.16.*
10.3.10.4-18
192.168.1.3


...

Programmatic:

72

Config config = new Config();
NetworkConfig network = config.getNetworkConfig();
InterfacesConfig interfaceConfig = network.getInterfaces();
interfaceConfig.setEnabled( true )
.addInterface( "192.168.1.3" );

5.10.8. IPv6 Support
Hazelcast supports IPv6 addresses seamlessly (This support is switched off by default, please see the
note at the end of this section).
All you need is to define IPv6 addresses or interfaces in the network configuration. The only
current limitation is that you cannot define wildcard IPv6 addresses in the TCP/IP join
configuration (tcp-ip element). Interfaces configuration does not have this limitation, you can
configure wildcard IPv6 interfaces in the same way as IPv4 interfaces.


...

5701


FF02:0:0:0:0:0:0:1
54327


[fe80::223:6cff:fe93:7c7e]:5701
192.168.1.0-7
192.168.1.*
fe80:0:0:0:45c5:47ee:fe15:493a



10.3.16.*
10.3.10.4-18
fe80:0:0:0:45c5:47ee:fe15:*
fe80::223:6cff:fe93:0-5555

...

...

JVM has two system properties for setting the preferred protocol stack (IPv4 or IPv6) as well as the
preferred address family types (inet4 or inet6). On a dual stack machine, IPv6 stack is preferred by
default, you can change this through the java.net.preferIPv4Stack= system property.
When querying name services, JVM prefers IPv4 addresses over IPv6 addresses and will return an
IPv4 address if possible. You can change this through java.net.preferIPv6Addresses=

73

system property.
Also see additional details on IPv6 support in Java.
IPv6 support has been switched off by default, since some platforms have issues



using the IPv6 stack. Some other platforms such as Amazon AWS have no support
at

all.

To

enable

IPv6

support,

just

set

configuration

property

hazelcast.prefer.ipv4.stack to false. Please refer to the System Properties
appendix for details.

5.10.9. Member Address Provider SPI



This SPI is not intended to provide addresses of other cluster members with which
the Hazelcast instance will form a cluster. To do that, refer to the other network
configuration sections above.

By default, Hazelcast chooses the public and bind address. You can influence on the choice by
defining a public-address in the configuration or by using other properties mentioned above. In
some cases, though, these properties are not enough and the default address picking strategy will
choose wrong addresses. This may be the case when deploying Hazelcast in some cloud
environments, such as AWS, when using Docker or when the instance is deployed behind a NAT
and the public-address property is not enough (please see the Public Address section).
In these cases, it is possible to configure the bind and public address in a more advanced way. You
can provide an implementation of the com.hazelcast.spi.MemberAddressProvider interface which
will provide the bind and public address. The implementation may then choose these addresses in
any way - it may read from a system property or file or even invoke a web service to retrieve the
public and private address.
The details of the implementation depend heavily on the environment in which Hazelcast is
deployed. As such, we will demonstrate how to configure Hazelcast to use a simplified custom
member address provider SPI implementation. An example of an implementation is shown below:

public static final class SimpleMemberAddressProvider implements MemberAddressProvider
{
@Override
public InetSocketAddress getBindAddress() {
// determine the address using some configuration, calling an API, ...
return new InetSocketAddress(hostname, port);
}
@Override
public InetSocketAddress getPublicAddress() {
// determine the address using some configuration, calling an API, ...
return new InetSocketAddress(hostname, port);
}
}

74

Note that if the bind address port is 0 then it will use a port as configured in the Hazelcast network
configuration (see the Port section). If the public address port is set to 0 then it will broadcast the
same port that it is bound to. If you wish to bind to any local interface, you may return new
InetSocketAddress((InetAddress) null, port) from the getBindAddress() address.
The following configuration examples contain properties that will be provided to the constructor of
the provided class. If you do not provide any properties, the class may have either a no-arg
constructor or a constructor accepting a single java.util.Properties instance. On the other hand, if
you do provide properties in the configuration, the class must have a constructor accepting a single
java.util.Properties instance.
Declarative:



SimpleMemberAddressProvider

prop1-value
prop2-value




Programmatic:

Config config = new Config();
MemberAddressProviderConfig memberAddressProviderConfig = config.getNetworkConfig()
.getMemberAddressProviderConfig();
memberAddressProviderConfig
.setEnabled(true)
.setClassName(MemberAddressProviderWithStaticProperties.class.getName());
Properties properties = memberAddressProviderConfig.getProperties();
properties.setProperty("prop1", "prop1-value");
properties.setProperty("prop2", "prop2-value");
config.getNetworkConfig().getJoin().getMulticastConfig().setEnabled(false);
// perform other configuration
Hazelcast.newHazelcastInstance(config);

5.11. Failure Detector Configuration
A failure detector is responsible to determine if a member in the cluster is unreachable or crashed.
The most important problem in failure detection is to distinguish whether a member is still alive
but slow or has crashed. But according to the famous FLP result, it is impossible to distinguish a

75

crashed member from a slow one in an asynchronous system. A workaround to this limitation is to
use unreliable failure detectors. An unreliable failure detector allows a member to suspect that
others have failed, usually based on liveness criteria but it can make mistakes to a certain degree.
Hazelcast has two built-in failure detectors; Deadline Failure Detector and Phi Accrual Failure
Detector.
Since 3.9.1, Hazelcast provides yet another failure detector, Ping Failure Detector, that, if enabled,
works in parallel with the above ones, but identifies failures on OSI Layer 3 (Network Layer). This
detector is by default disabled.
Note that, Hazelcast also offers failure detectors for its Java client. Please refer to the Client Failure
Detectors section for more information.

5.11.1. Deadline Failure Detector
Deadline Failure Detector uses an absolute timeout for missing/lost heartbeats. After timeout, a
member is considered as crashed/unavailable and marked as suspected.
Deadline Failure Detector has two configuration properties:
• hazelcast.heartbeat.interval.seconds: This is the interval at which member heartbeat messages
are sent to each other.
• hazelcast.max.no.heartbeat.seconds: This is the timeout which defines when a cluster member
is suspected because it has not sent any heartbeats.
To use Deadline Failure Detector configuration property hazelcast.heartbeat.failuredetector.type
should be set to "deadline".


[...]

deadline
5
120
[...]

[...]


Config config = ...;
config.setProperty("hazelcast.heartbeat.failuredetector.type", "deadline");
config.setProperty("hazelcast.heartbeat.interval.seconds", "5");
config.setProperty("hazelcast.max.no.heartbeat.seconds", "120");
[...]


76

Deadline Failure Detector is the default failure detector in Hazelcast.

5.11.2. Phi Accrual Failure Detector
This is the failure detector based on The Phi Accrual Failure Detector' by Hayashibara et al.
Phi Accrual Failure Detector keeps track of the intervals between heartbeats in a sliding window of
time and measures the mean and variance of these samples and calculates a value of suspicion
level (Phi). The value of phi will increase when the period since the last heartbeat gets longer. If the
network becomes slow or unreliable, the resulting mean and variance will increase, there will need
to be a longer period for which no heartbeat is received before the member is suspected.
hazelcast.heartbeat.interval.seconds and hazelcast.max.no.heartbeat.seconds properties will still
be used as period of heartbeat messages and deadline of heartbeat messages. Since Phi Accrual
Failure

Detector

is

adaptive

to

network

conditions,

a

much

lower

hazelcast.max.no.heartbeat.seconds can be defined than Deadline Failure Detector's timeout.
Additional to above two properties, Phi Accrual Failure Detector has three more configuration
properties:
• hazelcast.heartbeat.phiaccrual.failuredetector.threshold: This is the phi threshold for
suspicion. After calculated phi exceeds this threshold, a member is considered as unreachable
and marked as suspected. A low threshold allows to detect member crashes/failures faster but
can generate more mistakes and cause wrong member suspicions. A high threshold generates
fewer mistakes but is slower to detect actual crashes/failures.
phi = 1 means likeliness that we will make a mistake is about 10%. The likeliness is about 1% with
phi = 2, 0.1% with phi = 3 and so on. Default phi threshold is 10.
• hazelcast.heartbeat.phiaccrual.failuredetector.sample.size: Number of samples to keep for
history. Its default value is 200.
• hazelcast.heartbeat.phiaccrual.failuredetector.min.std.dev.millis:

Minimum

standard

deviation to use for the normal distribution used when calculating phi. Too low standard
deviation might result in too much sensitivity.
To

use

Phi

Accrual

Failure

Detector,

configuration

property

hazelcast.heartbeat.failuredetector.type should be set to "phi-accrual".

77


[...]

phi-accrual
1
60
10<
/property>
200

100
[...]

[...]


Config config = ...;
config.setProperty("hazelcast.heartbeat.failuredetector.type", "phi-accrual");
config.setProperty("hazelcast.heartbeat.interval.seconds", "1");
config.setProperty("hazelcast.max.no.heartbeat.seconds", "60");
config.setProperty("hazelcast.heartbeat.phiaccrual.failuredetector.threshold", "10");
config.setProperty("hazelcast.heartbeat.phiaccrual.failuredetector.sample.size", "200
");
config.setProperty("hazelcast.heartbeat.phiaccrual.failuredetector.min.std.dev.millis"
, "100");
[...]

5.11.3. Ping Failure Detector
The Ping Failure Detector may be configured in addition to one of Deadline and Phi Accrual Failure
Detectors. It operates at Layer 3 of the OSI protocol and provides much quicker and more
deterministic detection of hardware and other lower level events. This detector may be configured
to perform an extra check after a member is suspected by one of the other detectors, or it can work
in parallel, which is the default. This way hardware and network level issues will be detected more
quickly.
This failure detector is based on InetAddress.isReachable(). When the JVM process has enough
permissions to create RAW sockets, the implementation will choose to rely on ICMP Echo requests.
This is preferred.
If there are not enough permissions, it can be configured to fallback on attempting a TCP Echo on
port 7. In the latter case, both a successful connection or an explicit rejection will be treated as
"Host is Reachable". Or, it can be forced to use only RAW sockets. This is not preferred as each call
creates a heavy weight socket and moreover the Echo service is typically disabled.
For the Ping Failure Detector to rely only on ICMP Echo requests, there are some criteria that need
to be met.
78

Requirements and Linux/Unix Configuration
• Supported OS: as of Java 1.8 only Linux/Unix environments are supported. This detector
relies on ICMP, i.e., the protocol behind the ping command. It tries to issue the ping attempts
periodically, and their responses are used to determine the reachability of the remote member.
However, you cannot simply create an ICMP Echo Request because these type of packets do not
rely on any of the preexisting transport protocols such as TCP. In order to create such a request,
you must have the privileges to create RAW sockets (please see https://linux.die.net/man/7/raw).
Most operating systems allow this to the root users, however Unix based ones are more flexible
and allow the use of custom privileges per process instead of requiring root access. Therefore,
this detector is supported only on Linux.
• The Java executable must have the cap_net_raw capability. As described in the above
requirement, on Linux, you have the ability to define extra capabilities to a single process,
which would allow the process to interact with the RAW sockets. This interaction is achieved via
the capability cap_net_raw (please see https://linux.die.net/man/7/capabilities). To enable this
capability run the following command:
sudo setcap cap_net_raw=+ep /jre/bin/java
• When running with custom capabilities, the dynamic linker on Linux will reject loading
libs from untrusted paths. Since you have now cap_net_raw as a custom capability for a
process, it becomes suspicious to the dynamic linker and it will throw an error: java: error
while loading shared libraries: libjli.so: cannot open shared object file: No such file or
directory
◦ To overcome this rejection, the /jre/lib/amd64/jli/ path needs to be added in the
ld.conf. Run the following command to do this: echo "/jre/lib/amd64/jli/" >>
/etc/ld.so.conf.d/java.conf && sudo ldconfig
• ICMP

Echo

Requests

must

not

be

blocked

by

the

receiving

hosts.

/proc/sys/net/ipv4/icmp_echo_ignore_all set to 0. Run the following command:
echo 0 > /proc/sys/net/ipv4/icmp_echo_ignore_all
If any of the above criteria isn’t met, then the isReachable will always fallback on TCP Echo attempts
on port 7.
To be able to use the Ping Failure Detector, please add the following properties in your Hazelcast
declarative configuration file:

79


[...]

true
true
1000
3
1000
0
[...]

[...]

• hazelcast.icmp.enabled (default false) - Enables legacy ICMP detection mode, works
cooperatively with the existing failure detector and only kicks-in after a pre-defined period has
passed with no heartbeats from a member.
• hazelcast.icmp.parallel.mode (default true) - Enabling the parallel ping detector, works
separately from the other detectors.
• hazelcast.icmp.timeout (default 1000) - Number of milliseconds until a ping attempt is
considered failed if there was no reply.
• hazelcast.icmp.max.attempts (default 3) - The maximum number of ping attempts before the
member/node gets suspected by the detector.
• hazelcast.icmp.interval (default 1000) - The interval, in milliseconds, between each ping
attempt. 1000ms (1 sec) is also the minimum interval allowed.
• hazelcast.icmp.ttl (default 0) - The maximum number of hops the packets should go through or
0 for the default.
In the above configuration, the Ping detector will attempt 3 pings, one every second and will wait
up-to 1 second for each to complete. If after 3 seconds, there was no successful ping, the member
will get suspected.
To enforce the Requirements, the property hazelcast.icmp.echo.fail.fast.on.startup can also be
set to true, in which case, if any of the requirements isn’t met, Hazelcast will fail to start.
Below is a summary table of all possible configuration combinations of the ping failure detector.
Table 3. Ping Failure Detector Possible Configuration Combinations

ICMP

Parallel

Fail-Fast

Description Linux

Windows

macOS

false

false

false

Completely
disabled

N/A

N/A

80

N/A

ICMP

Parallel

Fail-Fast

Description Linux

Windows

macOS

true

false

false

Legacy ping
mode. This
works handto-hand with
the OSI
Layer 7
failure
detector (see.
phi or
deadline in
sections
above). Ping
in this mode
will only
kick in after
a period
when there
are no
hearbeats
received, in
which case
the remote
Hazelcast
member will
be pinged
up-to 5
times. If all 5
attempts fail,
the member
gets
suspected.

Supported
Supported
ICMP Echo if TCP Echo on
available port 7
Falls back on
TCP Echo on
port 7

Supported
ICMP Echo if
available Falls back on
TCP Echo on
port 7

true

true

false

Parallel ping
detector,
works in
parallel with
the
configured
failure
detector.
Checks
periodically
if members
are live (OSI
Layer 3) and
suspects
them
immediately,
regardless of
the other
detectors.

Supported
Supported
ICMP Echo if TCP Echo on
available port 7
Falls back on
TCP Echo on
port 7

Supported
ICMP Echo if
available Falls back on
TCP Echo on
port 7

81

ICMP

Parallel

Fail-Fast

Description Linux

true

true

true

Parallel ping
detector,
works in
parallel with
the
configured
failure
detector.
Checks
periodically
if members
are live (OSI
Layer 3) and
suspects
them
immediately,
regardless of
the other
detectors.

Windows

Supported - Not
Requires OS Supported
Configuratio
n Enforcing
ICMP Echo if
available No start up if
not available

macOS
Not
Supported Requires
root
privileges

6. Rolling Member Upgrades
Hazelcast IMDG Enterprise
This chapter explains the procedure of upgrading the version of Hazelcast members in a running
cluster without interrupting the operation of the cluster.

6.1. Terminology
• Minor version: A version change after the decimal point, e.g., 3.8 and 3.9.
• Patch version: A version change after the second decimal point, e.g., 3.8.1 and 3.8.2.
• Member codebase version: The major.minor.patch version of the Hazelcast binary on which
the member executes. For example, when running on hazelcast-3.8.jar, your member’s
codebase version is 3.8.0.
• Cluster version: The major.minor version at which the cluster operates. This ensures that
cluster members are able to communicate using the same cluster protocol and determines the
feature set exposed by the cluster.

6.2. Hazelcast Members Compatibility Guarantees
Hazelcast members operating on binaries of the same major and minor version numbers are
compatible regardless of patch version. For example, in a cluster with members running on version
3.7.1, it is possible to perform a rolling upgrade to 3.7.2 by shutting down, upgrading to hazelcast3.7.2.jar binary and starting each member one by one. Patch level compatibility applies to both
Hazelcast IMDG and Hazelcast IMDG Enterprise.

82

Starting with Hazelcast IMDG Enterprise 3.8, each next minor version released will be compatible
with the previous one. For example, it will be possible to perform a rolling upgrade on a cluster
running Hazelcast IMDG Enterprise 3.8 to Hazelcast IMDG Enterprise 3.9 whenever that is released.
Rolling upgrades across minor versions is a Hazelcast IMDG Enterprise feature, starting with version
3.8.
The compatibility guarantees described above are given in the context of rolling member upgrades
and only apply to GA (general availability) releases. It is never advisable to run a cluster with
members running on different patch or minor versions for prolonged periods of time.

6.3. Rolling Upgrade Procedure



The version numbers used in the paragraph below are only used as an example.

Let’s assume a cluster with four members running on codebase version 3.8.0 with cluster version
3.8, that should be upgraded to codebase version 3.9.0 and cluster version 3.9. The rolling upgrade
process for this cluster, i.e., replacing existing 3.8.0 members one by one with an upgraded one at
version 3.9.0, includes the following steps which should be repeated for each member:
• Shutdown gracefully an existing 3.8.0 member.
• Wait until all partition migrations are completed; during migrations, membership changes
(member joins or removals) are not allowed.
• Update the member with the new 3.9.0 Hazelcast binaries.
• Start the member and wait until it joins the cluster. You should see something like the following
in your logs:

...
INFO: [192.168.2.2]:5701 [cluster] [3.9] Hazelcast 3.9 (20170630 - a67dc3a)
starting at [192.168.2.2]:5701
...
INFO: [192.168.2.2]:5701 [cluster] [3.9] Cluster version set to 3.8

The version in brackets [3.9] still denotes the member’s codebase version (running on the
hypothetical hazelcast-3.9.jar binary). Once the member locates existing cluster members, it sends
its join request to the master. The master validates that the new member is allowed to join the
cluster and lets the new member know that the cluster is currently operating at 3.8 cluster version.
The new member sets 3.8 as its cluster version and starts operating normally.

At this point all members of the cluster have been upgraded to codebase version
`3.9.0` but the cluster still operates at cluster version `3.8`. In order to use `3.9`
features
the cluster version must be changed to `3.9`. There are two ways to accomplish this:
• Use Management Center.

83

• Use the command line cluster.sh script.
You need to upgrade your Management Center version before upgrading the
member version if you want to change cluster version using Management Center.
Management Center is compatible with the previous minor version of Hazelcast,



starting with version 3.9. For example, Management Center 3.9 works with both
Hazelcast IMDG 3.8 and 3.9. To change your cluster version to 3.9, you need
Management Center 3.9.

6.4. Network Partitions and Rolling Upgrades
In the event of network partitions which split your cluster into two subclusters, split-brain handling
works as explained in the Network Partitioning chapter, with the additional constraint that two
subclusters will only merge as long as they operate on the same cluster version. This is a
requirement to ensure that all members participating in each one of the subclusters will be able to
operate as members of the merged cluster at the same cluster version.
With regards to rolling upgrades, the above constraint implies that if a network partition occurs
while a change of cluster version is in progress, then with some unlucky timing, one subcluster may
be upgraded to the new cluster version and another subcluster may have upgraded members but
still operate at the old cluster version.
In order for the two subclusters to merge, it is necessary to change the cluster version of the
subcluster that still operates at the old cluster version, so that both subclusters will be operating at
the same, upgraded cluster version and will be able to merge as soon as the network partition is
fixed.

6.5. Rolling Upgrade FAQ
The following provide answers to the frequently asked questions related to rolling member
upgrades.
How is the cluster version set?
When a new member starts, it is not yet joined to a cluster; therefore its cluster version is still
undetermined. In order for the cluster version to be set, one of the following must happen:
• the member cannot locate any members of the cluster to join or is configured without a joiner:
in this case, the member will appoint itself as the master of a new single-member cluster and its
cluster version will be set to the major.minor version of its own codebase version. So a
standalone member running on codebase version 3.8.0 will set its own cluster version to 3.8.
• the member that is starting locates members of the cluster and identifies which is the master: in
this case, the master will validate that the joining member’s codebase version is compatible
with the current cluster version. If it is found to be compatible, then the member joins and the
master sends the cluster version, which is set on the joining member. Otherwise, the starting
member fails to join and shuts down.
What

84

if

a

new

Hazelcast

minor

version

changes

fundamental

cluster

protocol

communication, like join messages?



The version numbers used in the paragraph below are only used as an example.

On startup, as answered in the above question (How is the cluster version set?), the cluster version
is not yet known to a member that has not joined any cluster. By default the newly started member
will use the cluster protocol that corresponds to its codebase version until this member joins a
cluster (so for codebase 3.9.0 this means implicitly assuming cluster version 3.9). If, hypothetically,
major changes in discovery & join operations have been introduced which do not allow the
member to join a 3.8 cluster, then the member should be explicitly configured to start assuming a
3.8 cluster version.
Do I have to upgrade clients to work with rolling upgrades?
Starting with Hazelcast 3.6, the Hazelcast Open Binary Client Protocol was introduced. Clients
which implement the Open Binary Client Protocol are compatible with Hazelcast version 3.6 and
newer minor versions. Thus older client versions will be compatible with next minor versions.
Newer clients connected to a cluster will operate at the lower version of capabilities until all
members are upgraded and the cluster version upgrade occurs.
Can I stop and start multiple members at once during a rolling member upgrade?
It is not recommended due to potential network partitions. It is advised to always stop and start one
member in each upgrade step.
Can I upgrade my business app together with Hazelcast while doing a rolling member
upgrade?
Yes, but make sure to make the new version of your app compatible with the old one since there
will be a timespan when both versions interoperate. Checking if two versions of your app are
compatible includes verifying binary and algorithmic compatibility and some other steps.
It is worth mentioning that a business app upgrade is orthogonal to a rolling member upgrade. A
rolling business app upgrade may be done without upgrading the members.

7. Distributed Data Structures
As mentioned in the Overview section, Hazelcast offers distributed implementations of Java
interfaces. Below is the list of these implementations with links to the corresponding sections in this
manual.
• Standard utility collections
◦ Map is the distributed implementation of java.util.Map. It lets you read from and write to a
Hazelcast map with methods such as get and put.
◦ Queue is the distributed implementation of java.util.concurrent.BlockingQueue. You can add
an item in one member and remove it from another one.
◦ Ringbuffer is implemented for reliable eventing system.

85

◦ Set is the distributed and concurrent implementation of java.util.Set. It does not allow
duplicate elements and does not preserve their order.
◦ List is similar to Hazelcast Set. The only difference is that it allows duplicate elements and
preserves their order.
◦ Multimap is a specialized Hazelcast map. It is a distributed data structure where you can
store multiple values for a single key.
◦ Replicated Map does not partition data. It does not spread data to different cluster members.
Instead, it replicates the data to all members.
◦ Cardinality Estimator is a data structure which implements Flajolet’s HyperLogLog
algorithm.
• Topic is the distributed mechanism for publishing messages that are delivered to multiple
subscribers. It is also known as the publish/subscribe (pub/sub) messaging model. Please see the
Topic section for more information. Hazelcast also has a structure called Reliable Topic which
uses the same interface of Hazelcast Topic. The difference is that it is backed up by the
Ringbuffer data structure. Please see the Reliable Topic section.
• Concurrency utilities
◦ Lock is the distributed implementation of java.util.concurrent.locks.Lock. When you use
lock, the critical section that Hazelcast Lock guards is guaranteed to be executed by only one
thread in the entire cluster.
◦ ISemaphore is the distributed implementation of java.util.concurrent.Semaphore. When
performing concurrent activities, semaphores offer permits to control the thread counts.
◦ IAtomicLong is the distributed implementation of java.util.concurrent.atomic.AtomicLong.
Most of AtomicLong’s operations are available. However, these operations involve remote
calls and hence their performances differ from AtomicLong, due to being distributed.
◦ IAtomicReference

is

the

distributed

implementation

of

java.util.concurrent.atomic.AtomicReference. When you need to deal with a reference in a
distributed environment, you can use Hazelcast IAtomicReference.
◦ IdGenerator is used to generate cluster-wide unique identifiers. ID generation occurs almost
at the speed of AtomicLong.incrementAndGet(). This feature is deprecated, please use
FlakeIdGenerator instead.
◦ ICountdownLatch is the distributed implementation of java.util.concurrent.CountDownLatch.
Hazelcast CountDownLatch is a gate keeper for concurrent activities. It enables the threads
to wait for other threads to complete their operations.
◦ PN counter is a distributed data structure where each Hazelcast instance can increment and
decrement the counter value and these updates are propagated to all replicas.
• Event Journal is a distributed data structure that stores the history of mutation actions on map
or cache.

7.1. Overview of Hazelcast Distributed Objects
Hazelcast has two types of distributed objects in terms of their partitioning strategies:

86

1. Data structures where each partition stores a part of the instance, namely partitioned data
structures.
2. Data structures where a single partition stores the whole instance, namely non-partitioned data
structures.
Partitioned Hazelcast data structures are:
• Map
• MultiMap
• Cache (Hazelcast JCache implementation)
• PN Counter
• Event Journal
Non-partitioned Hazelcast data structures are:
• Queue
• Set
• List
• Ringbuffer
• Lock
• ISemaphore
• IAtomicLong
• IAtomicReference
• FlakeIdGenerator
• ICountdownLatch
• Cardinality Estimator
Besides these, Hazelcast also offers the Replicated Map structure as explained in the above
Standard utility collections list.

7.1.1. Loading and Destroying a Distributed Object
Hazelcast offers a get method for most of its distributed objects. To load an object, first create a
Hazelcast instance and then use the related get method on this instance. Following example code
snippet creates an Hazelcast instance and a map on this instance.

HazelcastInstance hazelcastInstance = Hazelcast.newHazelcastInstance();
Map customers = hazelcastInstance.getMap( "customers" );
As to the configuration of distributed object, Hazelcast uses the default settings from the file
hazelcast.xml that comes with your Hazelcast download. Of course, you can provide an explicit
configuration in this XML or programmatically according to your needs. Please see the

87

Understanding Configuration section.
Note that, most of Hazelcast’s distributed objects are created lazily, i.e., a distributed object is
created once the first operation accesses it.
If you want to use an object you loaded in other places, you can safely reload it using its reference
without creating a new Hazelcast instance (customers in the above example).
To destroy a Hazelcast distributed object, you can use the method destroy. This method clears and
releases all resources of the object. Therefore, you must use it with care since a reload with the
same object reference after the object is destroyed creates a new data structure without an error.
Please see the following example code where one of the queues are destroyed and the other one is
accessed.

HazelcastInstance hz1 = Hazelcast.newHazelcastInstance();
HazelcastInstance hz2 = Hazelcast.newHazelcastInstance();
IQueue q1 = hz1.getQueue("q");
IQueue q2 = hz2.getQueue("q");
q1.add("foo");
System.out.println("q1.size: "+q1.size()+ " q2.size:"+q2.size());
q1.destroy();
System.out.println("q1.size: " + q1.size() + " q2.size:" + q2.size());
If you start the Member above, the output will be as shown below:

q1.size: 1 q2.size:1
q1.size: 0 q2.size:0
As you see, no error is generated and a new queue resource is created.

7.1.2. Controlling Partitions
Hazelcast uses the name of a distributed object to determine which partition it will be put. Let’s
load two semaphores as shown below:

HazelcastInstance hazelcastInstance = Hazelcast.newHazelcastInstance();
ISemaphore s1 = hazelcastInstance.getSemaphore("s1");
ISemaphore s2 = hazelcastInstance.getSemaphore("s2");
Since these semaphores have different names, they will be placed into different partitions. If you
want to put these two into the same partition, you use the @ symbol as shown below:

HazelcastInstance hazelcastInstance = Hazelcast.newHazelcastInstance();
ISemaphore s1 = hazelcastInstance.getSemaphore("s1@foo");
ISemaphore s2 = hazelcastInstance.getSemaphore("s2@foo");

88

Now, these two semaphores will be put into the same partition whose partition key is foo. Note that
you can use the method getPartitionKey to learn the partition key of a distributed object. It may be
useful when you want to create an object in the same partition of an existing object. Please see its
usage as shown below:

String partitionKey = s1.getPartitionKey();
ISemaphore s3 = hazelcastInstance.getSemaphore("s3@"+partitionKey);

7.1.3. Common Features of all Hazelcast Data Structures
• If a member goes down, its backup replica (which holds the same data) will dynamically
redistribute the data, including the ownership and locks on them, to the remaining live
members. As a result, there will not be any data loss.
• There is no single cluster master that can be a single point of failure. Every member in the
cluster has equal rights and responsibilities. No single member is superior. There is no
dependency on an external 'server' or 'master'.

7.1.4. Example Distributed Object Code
Here is an example of how you can retrieve existing data structure instances (map, queue, set, lock,
topic, etc.) and how you can listen for instance events, such as an instance being created or
destroyed.

SampleDOL sample = new SampleDOL();
Config config = new Config();
HazelcastInstance hazelcastInstance = Hazelcast.newHazelcastInstance(config);
hazelcastInstance.addDistributedObjectListener(sample);
Collection distributedObjects = hazelcastInstance
.getDistributedObjects();
for (DistributedObject distributedObject : distributedObjects) {
System.out.println(distributedObject.getName());
}
}
@Override
public void distributedObjectCreated(DistributedObjectEvent event) {
DistributedObject instance = event.getDistributedObject();
System.out.println("Created " + instance.getName());
}
@Override
public void distributedObjectDestroyed(DistributedObjectEvent event) {
DistributedObject instance = event.getDistributedObject();
System.out.println("Destroyed " + instance.getName());
}

89

7.2. Map
Hazelcast Map (IMap) extends the interface java.util.concurrent.ConcurrentMap and hence
java.util.Map. It is the distributed implementation of Java map. You can perform operations like
reading and writing from/to a Hazelcast map with the well known get and put methods.

IMap data structure can also be used by Hazelcast Jet for Real-Time Stream
Processing (by enabling the Event Journal on your map) and Fast Batch Processing.



Hazelcast Jet uses IMap as a source (reads data from IMap) and as a sink (writes
data to IMap). Please see the Fast Batch Processing and Real-Time Stream
Processing use cases for Hazelcast Jet. Please also see here in the Hazelcast Jet
Reference Manual to learn how Jet uses IMap, i.e., how it can read from and write
to IMap.

7.2.1. Getting a Map and Putting an Entry
Hazelcast will partition your map entries and their backups, and almost evenly distribute them
onto all Hazelcast members. Each member carries approximately "number of map entries * 2 * 1/n"
entries, where n is the number of members in the cluster. For example, if you have a member with
1000 objects to be stored in the cluster and then you start a second member, each member will both
store 500 objects and back up the 500 objects in the other member.
Let’s create a Hazelcast instance and fill a map named Capitals with key-value pairs using the
following code. Use the HazelcastInstance getMap method to get the map, then use the map put
method to put an entry into the map.

HazelcastInstance hzInstance = Hazelcast.newHazelcastInstance();
Map capitalcities = hzInstance.getMap( "capitals" );
capitalcities.put( "1", "Tokyo" );
capitalcities.put( "2", "Paris" );
capitalcities.put( "3", "Washington" );
capitalcities.put( "4", "Ankara" );
capitalcities.put( "5", "Brussels" );
capitalcities.put( "6", "Amsterdam" );
capitalcities.put( "7", "New Delhi" );
capitalcities.put( "8", "London" );
capitalcities.put( "9", "Berlin" );
capitalcities.put( "10", "Oslo" );
capitalcities.put( "11", "Moscow" );
...
capitalcities.put( "120", "Stockholm" );
When you run this code, a cluster member is created with a map whose entries are distributed
across the members' partitions. See the below illustration. For now, this is a single member cluster.

90

Please note that some of the partitions will not contain any data entries since we



only have 120 objects and the partition count is 271 by default. This count is
configurable

and

can

be

changed

using

the

system

property

hazelcast.partition.count. Please see the System Properties appendix.

7.2.2. Creating A Member for Map Backup
Now let’s create a second member by running the above code again. This will create a cluster with
two members. This is also where backups of entries are created - remember the backup partitions
mentioned in the Hazelcast Overview section. The following illustration shows two members and
how the data and its backup is distributed.

As you see, when a new member joins the cluster, it takes ownership and loads some of the data in

91

the cluster. Eventually, it will carry almost "(1/n * total-data) + backups" of the data, reducing the
load on other members.
HazelcastInstance.getMap() returns an instance of com.hazelcast.core.IMap which extends the
java.util.concurrent.ConcurrentMap interface. Methods like ConcurrentMap.putIfAbsent(key,value)
and ConcurrentMap.replace(key,value) can be used on the distributed map, as shown in the example
below.

public class BasicMapOperations {
private HazelcastInstance hazelcastInstance = Hazelcast.newHazelcastInstance();
public Customer getCustomer(String id) {
ConcurrentMap customers = hazelcastInstance.getMap(
"customers");
Customer customer = customers.get(id);
if (customer == null) {
customer = new Customer(id);
customer = customers.putIfAbsent(id, customer);
}
return customer;
}
public boolean updateCustomer(Customer customer) {
ConcurrentMap customers = hazelcastInstance.getMap(
"customers");
return (customers.replace(customer.getId(), customer) != null);
}
public boolean removeCustomer(Customer customer) {
ConcurrentMap customers = hazelcastInstance.getMap(
"customers");
return customers.remove(customer.getId(), customer);
}
}
All ConcurrentMap operations such as put and remove might wait if the key is locked by another
thread in the local or remote JVM. But, they will eventually return with success. ConcurrentMap
operations never throw a java.util.ConcurrentModificationException.
Also see:
• Data Affinity section.
• Map Configuration with wildcards.

7.2.3. Backing Up Maps
Hazelcast distributes map entries onto multiple cluster members (JVMs). Each member holds some
portion of the data.

92

Distributed maps have one backup by default. If a member goes down, your data is recovered using
the backups in the cluster. There are two types of backups as described below: sync and async.
Creating Sync Backups
To provide data safety, Hazelcast allows you to specify the number of backup copies you want to
have. That way, data on a cluster member will be copied onto other member(s).
To create synchronous backups, select the number of backup copies using the backup-count
property.




When this count is 1, a map entry will have its backup on one other member in the cluster. If you
set it to 2, then a map entry will have its backup on two other members. You can set it to 0 if you do
not want your entries to be backed up, e.g., if performance is more important than backing up. The
maximum value for the backup count is 6.
Hazelcast supports both synchronous and asynchronous backups. By default, backup operations
are synchronous and configured with backup-count. In this case, backup operations block
operations until backups are successfully copied to backup members (or deleted from backup
members in case of remove) and acknowledgements are received. Therefore, backups are updated
before a put operation is completed, provided that the cluster is stable. Sync backup operations
have a blocking cost which may lead to latency issues.
Creating Async Backups
Asynchronous backups, on the other hand, do not block operations. They are fire & forget and do
not require acknowledgements; the backup operations are performed at some point in time.
To create asynchronous backups, select the number of async backups with the async-backup-count
property. An example is shown below.




See Consistency and Replication Model for more detail.



Backups increase memory usage since they are also kept in memory.

93



A map can have both sync and async backups at the same time.

Enabling Backup Reads
By default, Hazelcast has one sync backup copy. If backup-count is set to more than 1, then each
member will carry both owned entries and backup copies of other members. So for the
map.get(key) call, it is possible that the calling member has a backup copy of that key. By default,
map.get(key) will always read the value from the actual owner of the key for consistency.
To enable backup reads (read local backup entries), set the value of the read-backup-data property
to true. Its default value is false for consistency. Enabling backup reads can improve performance
but on the other hand it can cause stale reads while still preserving monotonic-reads property.




This feature is available when there is at least one sync or async backup.
Please note that if you are performing a read from a backup, you should take into account that your
hits to the keys in the backups are not reflected as hits to the original keys on the primary
members. This has an impact on IMap’s maximum idle seconds or time-to-live seconds expiration.
Therefore, even though there is a hit on a key in backups, your original key on the primary member
may expire.

7.2.4. Map Eviction



Starting with Hazelcast 3.7, Hazelcast Map uses a new eviction mechanism which
is based on the sampling of entries. Please see the Eviction Algorithm section for
details.

Unless you delete the map entries manually or use an eviction policy, they will remain in the map.
Hazelcast supports policy-based eviction for distributed maps. Currently supported policies are LRU
(Least Recently Used) and LFU (Least Frequently Used).
Understanding Map Eviction
Hazelcast Map performs eviction based on partitions. For example, when you specify a size using
the PER_NODE attribute for max-size (please see Configuring Map Eviction), Hazelcast internally
calculates the maximum size for every partition. Hazelcast uses the following equation to calculate
the maximum size of a partition:

94

partition-maximum-size = max-size * member-count / partition-count



If the partition-maximum-size is less than 1 in the equation above, it will be set to 1
(otherwise, the partitions would be emptied immediately by eviction due to the
exceedance of max-size being less than 1).

The eviction process starts according to this calculated partition maximum size when you try to put
an entry. When entry count in that partition exceeds partition maximum size, eviction starts on
that partition.
Assume that you have the following figures as examples:
• Partition count: 200
• Entry count for each partition: 100
• max-size (PER_NODE): 20000
The total number of entries here is 20000 (partition count * entry count for each partition). This
means you are at the eviction threshold since you set the max-size to 20000. When you try to put an
entry
1. the entry goes to the relevant partition;
2. the partition checks whether the eviction threshold is reached (max-size);
3. only one entry will be evicted.
As a result of this eviction process, when you check the size of your map, it is 19999. After this
eviction, subsequent put operations will not trigger the next eviction until the map size is again
close to the max-size.



The above scenario is simply an example that describes how the eviction process
works. Hazelcast finds the most optimum number of entries to be evicted
according to your cluster size and selected policy.

Configuring Map Eviction
The following is an example declarative configuration for map eviction.

95




Let’s describe each element:
• time-to-live-seconds: Maximum time in seconds for each entry to stay in the map (TTL). It
limits the lifetime of the entries relative to the time of the last write access performed on them.
If it is not 0, the entries whose lifetime exceeds this period (without any write access performed
on them during this period) are expired and evicted automatically. An individual entry may
have its own lifetime limit by using one of the methods accepting a TTL; see Evicting Specific
Entries section. If there is no TTL value provided for the individual entry, it inherits the value
set for this element. Valid values are integers between 0 and Integer.MAX VALUE. Its default value
is 0, which means infinite (no expiration and eviction). If it is not 0, entries are evicted
regardless of the set eviction-policy described below.
• max-idle-seconds: Maximum time in seconds for each entry to stay idle in the map. It limits the
lifetime of the entries relative to the time of the last read or write access performed on them.
The entries whose idle period exceeds this limit are expired and evicted automatically. An entry
is idle if no get, put, EntryProcessor.process or containsKey is called on it. Valid values are
integers between 0 and Integer.MAX VALUE. Its default value is 0, which means infinite.



Both time-to-live-seconds and max-idle-seconds may be used simultaneously
on the map entries. In that case, the entry is considered expired if at least one
of the policies marks it as expired.

• eviction-policy: Eviction policy to be applied when the size of map grows larger than the value
specified by the max-size element described below. Valid values are:
◦ NONE: Default policy. If set, no items will be evicted and the property max-size described
below will be ignored. You still can combine it with time-to-live-seconds and max-idleseconds.
◦ LRU: Least Recently Used.
◦ LFU: Least Frequently Used.
Apart from the above values, you can also develop and use your own eviction policy. Please
see the Custom Eviction Policy section.
• max-size: Maximum size of the map. When maximum size is reached, the map is evicted based
on the policy defined. Valid values are integers between 0 and Integer.MAX VALUE. Its default
value is 0, which means infinite. If you want max-size to work, set the eviction-policy property

96

to a value other than NONE. Its attributes are described below.
◦ PER_NODE: Maximum number of map entries in each cluster member. This is the default
policy.
5000
◦ PER_PARTITION: Maximum number of map entries within each partition. Storage size depends
on the partition count in a cluster member. This attribute should not be used often. For
instance, avoid using this attribute with a small cluster. If the cluster is small, it will be
hosting more partitions, and therefore map entries, than that of a larger cluster. Thus, for a
small cluster, eviction of the entries will decrease performance (the number of entries is
large).
27100
◦ USED_HEAP_SIZE: Maximum used heap size in megabytes per map for each Hazelcast instance.
Please note that this policy does not work when in-memory format is set to OBJECT, since the
memory footprint cannot be determined when data is put as OBJECT.
4096
◦ USED_HEAP_PERCENTAGE: Maximum used heap size percentage per map for each Hazelcast
instance. If, for example, a JVM is configured to have 1000 MB and this value is 10, then the
map entries will be evicted when used heap size exceeds 100 MB. Please note that this policy
does not work when in-memory format is set to OBJECT, since the memory footprint cannot
be determined when data is put as OBJECT.
10
◦ FREE_HEAP_SIZE: Minimum free heap size in megabytes for each JVM.
512
◦ FREE_HEAP_PERCENTAGE: Minimum free heap size percentage for each JVM. If, for example, a
JVM is configured to have 1000 MB and this value is 10, then the map entries will be evicted
when free heap size is below 100 MB.
10
◦ USED_NATIVE_MEMORY_SIZE: (Hazelcast IMDG Enterprise HD) Maximum used native memory
size in megabytes per map for each Hazelcast instance.
1024
◦ USED_NATIVE_MEMORY_PERCENTAGE: (Hazelcast IMDG Enterprise HD) Maximum used native
memory size percentage per map for each Hazelcast instance.
65
◦ FREE_NATIVE_MEMORY_SIZE: (Hazelcast IMDG Enterprise HD) Minimum free native memory
size in megabytes for each Hazelcast instance.
256

97

◦ FREE_NATIVE_MEMORY_PERCENTAGE: (Hazelcast IMDG Enterprise HD) Minimum free native
memory size percentage for each Hazelcast instance.
5
As of Hazelcast 3.7, the elements eviction-percentage and min-eviction-check-



millis are deprecated. They will be ignored if configured since map eviction is
based on the sampling of entries. Please see the Eviction Algorithm section for
details.

To put it briefly, Hazelcast maps have no restrictions on the size and may grow arbitrarily large, by
default. When it comes to reducing the size of a map, there are two concepts: expiration and
eviction.
Expiration puts a limit on the maximum lifetime of an entry stored inside the map. When the entry
expires it cannot be retrieved from the map any longer and at some point in time it will be cleaned
out from the map to free up the memory. Expiration, and hence the eviction based on the
expiration, can be configured using the element time-to-live-seconds- and `max-idle-seconds as
described above.
Eviction puts a limit on the maximum size of the map. If the size of the map grows larger than the
maximum allowed size, an eviction policy decides which item to evict from the map to reduce its
size. The maximum allowed size can be configured using the element max-size and the eviction
policy can be configured using the element eviction-policy as described above.
Eviction and expiration can be used together. In this case, the expiration configurations (time-tolive-seconds- and `max-idle-seconds) continue to work as usual cleaning out the expired entries
regardless of the map size. Note that locked map entries are not the subjects for eviction and
expiration.
Example Eviction Configurations


In the above example, documents map starts to evict its entries from a member when the map size
exceeds 10000 in that member. Then the entries least recently used will be evicted. The entries not
used for more than 60 seconds will be evicted as well.
And the following is an example eviction configuration for a map having NATIVE as the in-memory
format:

98



Evicting Specific Entries
The eviction policies and configurations explained above apply to all the entries of a map. The
entries that meet the specified eviction conditions are evicted.
If you want to evict some specific map entries, you can use the ttl and ttlUnit parameters of the
method map.put(). An example code line is given below.
myMap.put( "1", "John", 50, TimeUnit.SECONDS )
The map entry with the key "1" will be evicted 50 seconds after it is put into myMap.
You may also use map.setTTL method to alter the time-to-live value of an existing entry. It is done as
follows:
myMap.setTTL( "1", 50, TimeUnit.SECONDS )
In addition to the ttl, you may also specify a maximum idle timeout for specific map entries using
the maxIdle and maxIdleUnit parameters:
myMap.put( "1", "John", 50, TimeUnit.SECONDS, 40, TimeUnit.SECONDS )
Here ttl is set as 50 seconds and maxIdle is set as 40 seconds. The entry is considered to be evicted if
at least one of these policies marks it as expired. If you want to specify only the maxIdle parameter,
you need to set ttl as 0 seconds.
Evicting All Entries
To evict all keys from the map except the locked ones, use the method evictAll(). If a MapStore is
defined for the map, deleteAll is not called by evictAll. If you want to call the method deleteAll,
use clear().
An example is given below.

99

final int numberOfKeysToLock = 4;
final int numberOfEntriesToAdd = 1000;
HazelcastInstance node1 = Hazelcast.newHazelcastInstance();
HazelcastInstance node2 = Hazelcast.newHazelcastInstance();
IMap map = node1.getMap( "map" );
for (int i = 0; i < numberOfEntriesToAdd; i++) {
map.put(i, i);
}
for (int i = 0; i < numberOfKeysToLock; i++) {
map.lock(i);
}
// should keep locked keys and evict all others.
map.evictAll();
System.out.printf("# After calling evictAll...\n");
System.out.printf("# Expected map size\t: %d\n", numberOfKeysToLock);
System.out.printf("# Actual map size\t: %d\n", map.size());



Only EVICT_ALL event is fired for any registered listeners.

Forced Eviction
Hazelcast IMDG Enterprise
Hazelcast may use forced eviction in the cases when the eviction explained in Understanding Map
Eviction is not enough to free up your memory. Note that this is valid if you are using Hazelcast
IMDG Enterprise and you set your in-memory format to NATIVE.
Forced eviction mechanism is explained below as steps in the given order:
• When the normal eviction is not enough, forced eviction is triggered and first it tries to evict
approx. 20% of the entries from the current partition. It retries this five times.
• If the result of above step is still not enough, forced eviction applies the above step to all maps.
This time it might perform eviction from some other partitions too, provided that they are
owned by the same thread.
• If that is still not enough to free up your memory, it evicts not the 20% but all the entries from
the current partition.
• if that is not enough, it will evict all the entries from the other data structures; from the
partitions owned by the local thread.
Finally, when all the above steps are not enough, Hazelcast throws a Native Out of Memory
Exception.

100

Custom Eviction Policy



This section is valid for Hazelcast 3.7 and higher releases.

Apart from the policies such as LRU and LFU, which Hazelcast provides out-of-the-box, you can
develop and use your own eviction policy.
To achieve this, you need to provide an implementation of MapEvictionPolicy as in the following
OddEvictor example:

public class MapCustomEvictionPolicy {
public static void main(String[] args) {
Config config = new Config();
config.getMapConfig("test")
.setMapEvictionPolicy(new OddEvictor())
.getMaxSizeConfig()
.setMaxSizePolicy(PER_NODE).setSize(10000);
HazelcastInstance instance = Hazelcast.newHazelcastInstance(config);
IMap map = instance.getMap("test");
final Queue oddKeys = new ConcurrentLinkedQueue();
final Queue evenKeys = new ConcurrentLinkedQueue();
map.addEntryListener(new EntryEvictedListener() {
@Override
public void entryEvicted(EntryEvent event) {
Integer key = event.getKey();
if (key % 2 == 0) {
evenKeys.add(key);
} else {
oddKeys.add(key);
}
}
}, false);
// wait some more time to receive evicted-events
parkNanos(SECONDS.toNanos(5));
for (int i = 0; i < 15000; i++) {
map.put(i, i);
}
String msg = "IMap uses sampling based eviction. After eviction is completed,
we are expecting "
+ "number of evicted-odd-keys should be greater than number of
evicted-even-keys"
+ "\nNumber of evicted-odd-keys = %d, number of evicted-even-keys =
%d";

101

out.println(format(msg, oddKeys.size(), evenKeys.size()));
instance.shutdown();
}
/**
* Odd evictor tries to evict odd keys first.
*/
private static class OddEvictor extends MapEvictionPolicy {
@Override
public int compare(EntryView o1, EntryView o2) {
Integer key = (Integer) o1.getKey();
if (key % 2 != 0) {
return -1;
}
return 1;
}
}
}
Then you can enable your policy by setting it via the method MapConfig.setMapEvictionPolicy()
programmatically or via XML declaratively. Following is the example declarative configuration for
the eviction policy OddEvictor implemented above:


If you Hazelcast with Spring, you can enable your policy as shown below.





7.2.5. Setting In-Memory Format
IMap (and a few other Hazelcast data structures, such as ICache) has an in-memory-format
configuration option. By default, Hazelcast stores data into memory in binary (serialized) format.
Sometimes it can be efficient to store the entries in their object form, especially in cases of local
processing, such as entry processor and queries.
To set how the data will be stored in memory, set in-memory-format in the configuration. You have
the following format options:
102

• BINARY (default): The data (both the key and value) will be stored in serialized binary format.
You can use this option if you mostly perform regular map operations, such as put and get.
• OBJECT: The data will be stored in deserialized form. This configuration is good for maps where
entry processing and queries form the majority of all operations and the objects are complex,
making the serialization cost comparatively high. By storing objects, entry processing will not
contain the deserialization cost. Note that when you use OBJECT as the in-memory format, the
key will still be stored in binary format and the value will be stored in object format.
• NATIVE: (Hazelcast IMDG Enterprise HD) This format behaves the same as BINARY, however,
instead of heap memory, key and value will be stored in the off-heap memory.
Regular operations like get rely on the object instance. When the OBJECT format is used and a get is
performed, the map does not return the stored instance, but creates a clone. Therefore, this whole
get operation first includes a serialization on the member owning the instance and then a
deserialization on the member calling the instance. When the BINARY format is used, only a
deserialization is required; BINARY is faster.
Similarly, a put operation is faster when the BINARY format is used. If the format was OBJECT, the map
would create a clone of the instance, and there would first be a serialization and then a
deserialization. When BINARY is used, only a deserialization is needed.
If a value is stored in OBJECT format, a change on a returned value does not affect
the stored instance. In this case, the returned instance is not the actual one but a



clone. Therefore, changes made on an object after it is returned will not reflect on
the actual stored data. Similarly, when a value is written to a map and the value is
stored in OBJECT format, it will be a copy of the put value. Therefore, changes made
on the object after it is stored will not reflect on the stored data.

7.2.6. Using High-Density Memory Store with Map
Hazelcast IMDG Enterprise HD
Hazelcast instances are Java programs. In case of BINARY and OBJECT in-memory formats, Hazelcast
stores your distributed data into the heap of its server instances. Java heap is subject to garbage
collection (GC). In case of larger heaps, garbage collection might cause your application to pause for
tens of seconds (even minutes for really large heaps), badly affecting your application performance
and response times.
As the data gets bigger, you either run the application with larger heap, which would result in
longer GC pauses or run multiple instances with smaller heap which can turn into an operational
nightmare if the number of such instances becomes very high.
To overcome this challenge, Hazelcast offers High-Density Memory Store for your maps. You can
configure your map to use High-Density Memory Store by setting the in-memory format to NATIVE.
The following snippet is the declarative configuration example.

103


Keep in mind that you should have already enabled the High-Density Memory Store usage for your
cluster. Please see Configuring High-Density Memory Store section.
Required configuration changes when using NATIVE
Note that the eviction mechanism is different for NATIVE in-memory format. The new eviction
algorithm for map with High-Density Memory Store is similar to that of JCache with High-Density
Memory Store and is described here.
• Eviction percentage has no effect.


• These IMap eviction policies for max-size cannot be used: FREE_HEAP_PERCENTAGE, FREE_HEAP_SIZE,
USED_HEAP_PERCENTAGE, USED_HEAP_SIZE.
• Near Cache eviction configuration is also different for NATIVE in-memory format. For a Near
Cache configuration with in-memory format set to BINARY:


the equivalent configuration for NATIVE in-memory format would be similar to the following:


• Near Cache eviction policy ENTRY_COUNT cannot be used for max-size-policy.

104



Please refer to the High-Density Memory Store section for more information.

7.2.7. Loading and Storing Persistent Data
Hazelcast allows you to load and store the distributed map entries from/to a persistent data store
such as a relational database. To do this, you can use Hazelcast’s MapStore and MapLoader interfaces.
When you provide a MapLoader implementation and request an entry (IMap.get()) that does not exist
in memory, MapLoader's load method will load that entry from the data store. This loaded entry is
placed into the map and will stay there until it is removed or evicted.
When a MapStore implementation is provided, an entry is also put into a user defined data store.



Data store needs to be a centralized system that is accessible from all Hazelcast



Also note that the MapStore interface extends the MapLoader interface as you can see



Starting

members. Persistence to a local file system is not supported.

in the interface code.
with

Hazelcast

IMDG

3.11,

all

loads

can

be

listened

via

EntryLoadedListener.

Following is a MapStore example.

public class PersonMapStore implements MapStore {
private final Connection con;
private final PreparedStatement allKeysStatement;
public PersonMapStore() {
try {
con = DriverManager.getConnection("jdbc:hsqldb:mydatabase", "SA", "");
con.createStatement().executeUpdate(
"create table if not exists person (id bigint not null, name
varchar(45), primary key (id))");
allKeysStatement = con.prepareStatement("select id from person");
} catch (SQLException e) {
throw new RuntimeException(e);
}
}
public synchronized void delete(Long key) {
System.out.println("Delete:" + key);
try {
con.createStatement().executeUpdate(
format("delete from person where id = %s", key));
} catch (SQLException e) {
throw new RuntimeException(e);

105

}
}
public synchronized void store(Long key, Person value) {
try {
con.createStatement().executeUpdate(
format("insert into person values(%s,'%s')", key, value.getName()
));
} catch (SQLException e) {
throw new RuntimeException(e);
}
}
public synchronized void storeAll(Map map) {
for (Map.Entry entry : map.entrySet()) {
store(entry.getKey(), entry.getValue());
}
}
public synchronized void deleteAll(Collection keys) {
for (Long key : keys) {
delete(key);
}
}
public synchronized Person load(Long key) {
try {
ResultSet resultSet = con.createStatement().executeQuery(
format("select name from person where id =%s", key));
try {
if (!resultSet.next()) {
return null;
}
String name = resultSet.getString(1);
return new Person(key, name);
} finally {
resultSet.close();
}
} catch (SQLException e) {
throw new RuntimeException(e);
}
}
public synchronized Map loadAll(Collection keys) {
Map result = new HashMap();
for (Long key : keys) {
result.put(key, load(key));
}
return result;
}

106

public Iterable loadAllKeys() {
return new StatementIterable(allKeysStatement);
}
}

During the initial loading process, MapStore uses a thread different from the



partition threads that are used by the ExecutorService. After the initialization is
completed, the map.get method looks up any nonexistent value from the database
in a partition thread, or the map.put method looks up the database to return the
previously associated value for a key also in a partition thread.



For more MapStore/MapLoader code samples, please see here.

Hazelcast supports read-through, write-through and write-behind persistence modes, which are
explained in the subsections below.
Using Read-Through Persistence
If an entry does not exist in memory when an application asks for it, Hazelcast asks the loader
implementation to load that entry from the data store. If the entry exists there, the loader
implementation gets it, hands it to Hazelcast, and Hazelcast puts it into memory. This is readthrough persistence mode.
Setting Write-Through Persistence
MapStore can be configured to be write-through by setting the write-delay-seconds property to 0.
This means the entries will be put to the data store synchronously.
In this mode, when the map.put(key,value) call returns:
• MapStore.store(key,value) is successfully called so the entry is persisted.
• In-Memory entry is updated.
• In-Memory backup copies are successfully created on other cluster members (if backup-count is
greater than 0).
If MapStore throws an exception then the exception is propagated to the original put or remove call in
the form of RuntimeException.



There is a key difference in the behaviors of map.remove(key) and map.delete(key),
i.e., the latter results in MapStore.delete(key) to be invoked whereas the former
only removes the entry from IMap.

Setting Write-Behind Persistence
You can configure MapStore as write-behind by setting the write-delay-seconds property to a value
bigger than 0. This means the modified entries will be put to the data store asynchronously after a
configured delay.

107

In write-behind mode, Hazelcast coalesces updates on a specific key by default,



which means it applies only the last update on that key. However, you can set
MapStoreConfig.setWriteCoalescing() to FALSE and you can store all updates
performed on a key to the data store.
When you set MapStoreConfig.setWriteCoalescing() to FALSE, after you reached pernode maximum write-behind-queue capacity, subsequent put operations will fail



with ReachedMaxSizeException. This exception will be thrown to prevent
uncontrolled grow of write-behind queues. You can set per-node maximum
capacity using the system property hazelcast.map.write.behind.queue.capacity.
Please refer to the System Properties section for information on this property and
how to set the system properties.

In write-behind mode, when the map.put(key,value) call returns:
• In-Memory entry is updated.
• In-Memory backup copies are successfully created on other cluster members (if backup-count is
greater than 0).
• The entry is marked as dirty so that after write-delay-seconds, it can be persisted with
MapStore.store(key,value) call.
• For fault tolerance, dirty entries are stored in a queue on the primary member and also on a
back-up member.
The same behavior goes for the map.remove(key), the only difference is that MapStore.delete(key) is
called when the entry will be deleted.
If MapStore throws an exception, then Hazelcast tries to store the entry again. If the entry still
cannot be stored, a log message is printed and the entry is re-queued.
For batch write operations, which are only allowed in write-behind mode, Hazelcast will call
MapStore.storeAll(map) and MapStore.deleteAll(collection) to do all writes in a single call.
If a map entry is marked as dirty, meaning that it is waiting to be persisted to the



MapStore in a write-behind scenario, the eviction process forces the entry to be
stored. This way you have control over the number of entries waiting to be stored,
and thus you can prevent a possible OutOfMemory exception.



MapStore

or

implementations

should

not

use

Hazelcast

Map/Queue/MultiMap/List/Set operations. Your implementation should only work
with your data store. Otherwise, you may get into deadlock situations.

Here is a sample configuration:

108

MapLoader


...


The following are the descriptions of MapStore configuration elements and attributes:
• class-name: Name of the class implementing MapLoader and/or MapStore.
• write-delay-seconds: Number of seconds to delay to call the MapStore.store(key, value). If the
value is zero then it is write-through so MapStore.store(key, value) will be called as soon as the
entry is updated. Otherwise it is write-behind so updates will be stored after write-delayseconds value by calling Hazelcast.storeAll(map). Its default value is 0.
• write-batch-size: Used to create batch chunks when writing map store. In default mode, all map
entries will be tried to be written in one go. To create batch chunks, the minimum meaningful
value for write-batch-size is 2. For values smaller than 2, it works as in default mode.
• write-coalescing: In write-behind mode, Hazelcast coalesces updates on a specific key by
default; it applies only the last update on it. You can set this element to false to store all updates
performed on a key to the data store.
• enabled: True to enable this map-store, false to disable. Its default value is true.
• initial-mode: Sets the initial load mode. LAZY is the default load mode, where load is
asynchronous. EAGER means load is blocked till all partitions are loaded. Please see the
Initializing Map on Startup section for more details.
Storing Entries to Multiple Maps
A configuration can be applied to more than one map using wildcards (see Using Wildcards),
meaning that the configuration is shared among the maps. But MapStore does not know which
entries to store when there is one configuration applied to multiple maps.
To store entries when there is one configuration applied to multiple maps, use Hazelcast’s
MapStoreFactory interface. Using the MapStoreFactory interface, `MapStore`s for each map can be
created when a wildcard configuration is used. Example code is shown below.

109

Config config = new Config();
MapConfig mapConfig = config.getMapConfig( "*" );
MapStoreConfig mapStoreConfig = mapConfig.getMapStoreConfig();
mapStoreConfig.setFactoryImplementation( new MapStoreFactory() {
@Override
public MapLoader newMapStore( String mapName, Properties properties
) {
return null;
}
});
To initialize the MapLoader implementation with the given map name, configuration properties and
the Hazelcast instance, implement the MapLoaderLifecycleSupport interface. This interface has the
methods init() and destroy().
The method init() initializes the MapLoader implementation. Hazelcast calls this method when the
map is first used on the Hazelcast instance. The MapLoader implementation can initialize the
required resources for implementing MapLoader such as reading a configuration file or creating a
database connection.
Hazelcast calls the method destroy() before shutting down. You can override this method to
cleanup the resources held by this MapLoader implementation, such as closing the database
connections.
Initializing Map on Startup
To

pre-populate

the

in-memory

map

when

the

map

is

first

touched/used,

use

the

MapLoader.loadAllKeys API.
If MapLoader.loadAllKeys returns NULL, then nothing will be loaded. Your MapLoader.loadAllKeys
implementation can return all or some of the keys. For example, you may select and return only the
keys which are most important to you that you want to load them while initializing the map.
MapLoader.loadAllKeys is the fastest way of pre-populating the map since Hazelcast will optimize the
loading process by having each cluster member load its owned portion of the entries.
The InitialLoadMode configuration parameter in the class MapStoreConfig has two values: LAZY and
EAGER. If InitialLoadMode is set to LAZY, data is not loaded during the map creation. If it is set to EAGER,
all the data is loaded while the map is created and everything becomes ready to use. Also, if you
add indices to your map with the MapIndexConfig class or the addIndex method, then
InitialLoadMode is overridden and MapStoreConfig behaves as if EAGER mode is on.
Here is the MapLoader initialization flow:
1. When getMap() is first called from any member, initialization will start depending on the value
of InitialLoadMode. If it is set to EAGER, initialization starts on all partitions as soon as the map is
touched, i.e., all partitions will be loaded when getMap is called. If it is set to LAZY, data will be
loaded partition by partition, i.e., each partition will be loaded with its first touch.
2. Hazelcast will call MapLoader.loadAllKeys() to get all your keys on one of the members.

110

3. That member will distribute keys to all other members in batches.
4. Each member will load values of all its owned keys by calling MapLoader.loadAll(keys).
5. Each member puts its owned entries into the map by calling IMap.putTransient(key,value).
If the load mode is LAZY and the clear() method is called (which triggers



MapStore.deleteAll()), Hazelcast will remove ONLY the loaded entries from your
map and datastore. Since all the data is not loaded in this case (LAZY mode), please
note that there may still be entries in your datastore.*
If you do not want the MapStore start to load as soon as the first cluster member



starts, you can use the system property hazelcast.initial.min.cluster.size. For
example, if you set its value as 3, loading process will be blocked until all three
members are completely up.*



The return type of loadAllKeys() is changed from Set to Iterable with the release
of Hazelcast 3.5. MapLoader implementations from previous releases are also
supported and do not need to be adapted.

Loading Keys Incrementally
If the number of keys to load is large, it is more efficient to load them incrementally rather than
loading them all at once. To support incremental loading, the MapLoader.loadAllKeys() method
returns an Iterable which can be lazily populated with the results of a database query.
Hazelcast iterates over the Iterable and, while doing so, sends out the keys to their respective
owner members. The Iterator obtained from MapLoader.loadAllKeys() may also implement the
Closeable interface, in which case Iterator is closed once the iteration is over. This is intended for
releasing resources such as closing a JDBC result set.
Forcing All Keys To Be Loaded
The method loadAll loads some or all keys into a data store in order to optimize the multiple load
operations. The method has two signatures; the same method can take two different parameter
lists. One signature loads the given keys and the other loads all keys. Please see the example code
below.

111

final
final
final
final
final

int numberOfEntriesToAdd = 1000;
String mapName = LoadAll.class.getCanonicalName();
Config config = createNewConfig(mapName);
HazelcastInstance node = Hazelcast.newHazelcastInstance(config);
IMap map = node.getMap(mapName);

populateMap(map, numberOfEntriesToAdd);
System.out.printf("# Map store has %d elements\n", numberOfEntriesToAdd);
map.evictAll();
System.out.printf("# After evictAll map size\t: %d\n", map.size());
map.loadAll(true);
System.out.printf("# After loadAll map size\t: %d\n", map.size());

Post-Processing Objects in Map Store
In some scenarios, you may need to modify the object after storing it into the map store. For
example, you can get an ID or version auto-generated by your database and then need to modify
your object stored in the distributed map, but not to break the synchronization between the
database and the data grid.
To post-process an object in the map store, implement the PostProcessingMapStore interface to put
the modified object into the distributed map. This will trigger an extra step of Serialization, so use
it only when needed. (This is only valid when using the write-through map store configuration.)
Here is an example of post processing map store:

class ProcessingStore implements MapStore, PostProcessingMapStore {
@Override
public void store( Integer key, Employee employee ) {
EmployeeId id = saveEmployee();
employee.setId( id.getId() );
}
}



Please note that if you are using a post processing map store in combination with
entry processors, post-processed values will not be carried to backups.

Accessing a Database Using Properties
You can prepare your own MapLoader to access a database such as Cassandra and MongoDB. For this,
you can first declaratively specify the database properties in your hazelcast.xml configuration file
and then implement the MapLoaderLifecycleSupport interface to pass those properties.
You can define the database properties, such as its URL and name, using the properties
configuration element. The following is a configuration example for MongoDB:

112


After you specified the database properties in your configuration, you need to implement the
MapLoaderLifecycleSupport interface and give those properties in the init() method, as shown
below:

public class YourMapStoreImplementation implements MapStore,
MapLoaderLifecycleSupport {
private MongoClient mongoClient;
private MongoCollection collection;
public YourMapStoreImplementation() {
}
@Override
public void init(HazelcastInstance hazelcastInstance, Properties properties,
String mapName) {
String mongoUrl = (String) properties.get("mongo.url");
String dbName = (String) properties.get("mongo.db");
String collectionName = (String) properties.get("mongo.collection");
this.mongoClient = new MongoClient(new MongoClientURI(mongoUrl));
this.collection = mongoClient.getDatabase(dbName).getCollection(
collectionName);
}
You can refer to the full example here.

7.2.8. Creating Near Cache for Map
The Hazelcast distributed map supports a local Near Cache for remotely stored entries to increase
the performance of local read operations. Please refer to the Near Cache section for a detailed
explanation of the Near Cache feature and its configuration.

7.2.9. Locking Maps
Hazelcast Distributed Map (IMap) is thread-safe to meet your thread safety requirements. When
these requirements increase or you want to have more control on the concurrency, consider the
Hazelcast solutions described here.

113

Let’s work on a sample case as shown below.

public class RacyUpdateMember {
public static void main( String[] args ) throws Exception {
HazelcastInstance hz = Hazelcast.newHazelcastInstance();
IMap map = hz.getMap( "map" );
String key = "1";
map.put( key, new Value() );
System.out.println( "Starting" );
for ( int k = 0; k < 1000; k++ ) {
if ( k % 100 == 0 ) System.out.println( "At: " + k );
Value value = map.get( key );
Thread.sleep( 10 );
value.amount++;
map.put( key, value );
}
System.out.println( "Finished! Result = " + map.get(key).amount );
}
static class Value implements Serializable {
public int amount;
}
}
If the above code is run by more than one cluster member simultaneously, a race condition is likely.
You can solve this condition with Hazelcast using either pessimistic locking or optimistic locking.
Pessimistic Locking
One way to solve the race issue is by using pessimistic locking - lock the map entry until you are
finished with it.
To perform pessimistic locking, use the lock mechanism provided by the Hazelcast distributed map,
i.e., the map.lock and map.unlock methods. See the below example code.

114

public class PessimisticUpdateMember {
public static void main( String[] args ) throws Exception {
HazelcastInstance hz = Hazelcast.newHazelcastInstance();
IMap map = hz.getMap( "map" );
String key = "1";
map.put( key, new Value() );
System.out.println( "Starting" );
for ( int k = 0; k < 1000; k++ ) {
map.lock( key );
try {
Value value = map.get( key );
Thread.sleep( 10 );
value.amount++;
map.put( key, value );
} finally {
map.unlock( key );
}
}
System.out.println( "Finished! Result = " + map.get( key ).amount );
}
static class Value implements Serializable {
public int amount;
}
}
The IMap lock will automatically be collected by the garbage collector when the lock is released and
no other waiting conditions exist on the lock.
The IMap lock is reentrant, but it does not support fairness.
Another way to solve the race issue is by acquiring a predictable Lock object from Hazelcast. This
way, every value in the map can be given a lock, or you can create a stripe of locks.
Optimistic Locking
In Hazelcast, you can apply the optimistic locking strategy with the map’s replace method. This
method compares values in object or data forms depending on the in-memory format
configuration. If the values are equal, it replaces the old value with the new one. If you want to use
your defined equals method, in-memory-format should be OBJECT. Otherwise, Hazelcast serializes
objects to BINARY forms and compares them.
See the below example code.



The below example code is intentionally broken.

115

public class OptimisticMember {
public static void main( String[] args ) throws Exception {
HazelcastInstance hz = Hazelcast.newHazelcastInstance();
IMap map = hz.getMap( "map" );
String key = "1";
map.put( key, new Value() );
System.out.println( "Starting" );
for ( int k = 0; k < 1000; k++ ) {
if ( k % 10 == 0 ) System.out.println( "At: " + k );
for (; ; ) {
Value oldValue = map.get( key );
Value newValue = new Value( oldValue );
Thread.sleep( 10 );
newValue.amount++;
if ( map.replace( key, oldValue, newValue ) )
break;
}
}
System.out.println( "Finished! Result = " + map.get( key ).amount );
}
static class Value implements Serializable {
public int amount;
public Value() {
}
public Value( Value that ) {
this.amount = that.amount;
}
public boolean equals( Object o ) {
if ( o == this ) return true;
if ( !( o instanceof Value ) ) return false;
Value that = ( Value ) o;
return that.amount == this.amount;
}
}
}

Pessimistic vs. Optimistic Locking
The locking strategy you choose will depend on your locking requirements.
Optimistic locking is better for mostly read-only systems. It has a performance boost over
pessimistic locking.
Pessimistic locking is good if there are lots of updates on the same key. It is more robust than
optimistic locking from the perspective of data consistency.

116

In Hazelcast, use IExecutorService to submit a task to a key owner, or to a member or members.
This is the recommended way to perform task executions, rather than using pessimistic or
optimistic locking techniques. IExecutorService will have fewer network hops and less data over
wire, and tasks will be executed very near to the data. Please refer to the Data Affinity section.
Solving the ABA Problem
The ABA problem occurs in environments when a shared resource is open to change by multiple
threads. Even if one thread sees the same value for a particular key in consecutive reads, it does not
mean that nothing has changed between the reads. Another thread may change the value, do work
and change the value back, while the first thread thinks that nothing has changed.
To prevent these kind of problems, you can assign a version number and check it before any write
to be sure that nothing has changed between consecutive reads. Although all the other fields will be
equal, the version field will prevent objects from being seen as equal. This is the optimistic locking
strategy; it is used in environments that do not expect intensive concurrent changes on a specific
key.
In Hazelcast, you can apply the optimistic locking strategy with the map replace method.
Lock Split-Brain Protection with Pessimistic Locking
Locks can be configured to check the number of currently present members before applying a
locking operation. If the check fails, the lock operation will fail with a QuorumException (see SplitBrain Protection). As pessimistic locking uses lock operations internally, it will also use the
configured lock quorum. This means that you can configure a lock quorum with the same name or
a pattern that matches the map name. Note that the quorum for IMap locking actions can be
different from the quorum for other IMap actions.
The following actions will then check for lock quorum before being applied:
• IMap.lock(K) and IMap.lock(K, long, java.util.concurrent.TimeUnit)
• IMap.isLocked()
• IMap.tryLock(K), IMap.tryLock(K, long, java.util.concurrent.TimeUnit) and IMap.tryLock(K,
long, java.util.concurrent.TimeUnit, long, java.util.concurrent.TimeUnit)
• IMap.unlock()
• IMap.forceUnlock()
• MultiMap.lock(K) and MultiMap.lock(K, long, java.util.concurrent.TimeUnit)
• MultiMap.isLocked()
• MultiMap.tryLock(K),
MultiMap.tryLock(K,
MultiMap.tryLock(K,
long,
java.util.concurrent.TimeUnit)

long,
java.util.concurrent.TimeUnit)
java.util.concurrent.TimeUnit,

and
long,

• MultiMap.unlock()
• MultiMap.forceUnlock()
An example of declarative configuration:

117



map-lock-actions-quorum

Here the configured map will use the map-lock-actions-quorum quorum for map lock actions and the
map-actions-quorum quorum for other map actions.

7.2.10. Accessing Entry Statistics
Hazelcast keeps statistics about each map entry, such as creation time, last update time, last access
time, number of hits and version. To access the map entry statistics, use an IMap.getEntryView(key)
call. Here is an example.

HazelcastInstance hz = Hazelcast.newHazelcastInstance();
EntryView entry = hz.getMap( "quotes" ).getEntryView( "1" );
System.out.println ( "size in memory : " + entry.getCost() );
System.out.println ( "creationTime
: " + entry.getCreationTime() );
System.out.println ( "expirationTime : " + entry.getExpirationTime() );
System.out.println ( "number of hits : " + entry.getHits() );
System.out.println ( "lastAccessedTime: " + entry.getLastAccessTime() );
System.out.println ( "lastUpdateTime : " + entry.getLastUpdateTime() );
System.out.println ( "version
: " + entry.getVersion() );
System.out.println ( "key
: " + entry.getKey() );
System.out.println ( "value
: " + entry.getValue() );

7.2.11. Map Listener
Please refer to the Listening for Map Events section.

7.2.12. Listening to Map Entries with Predicates
You can listen to the modifications performed on specific map entries. You can think of it as an
entry listener with predicates. Please see the Listening for Map Events section for information on
how to add entry listeners to a map.
The default backwards-compatible event publishing strategy only publishes
UPDATED events when map entries are updated to a value that matches the



predicate with which the listener was registered. This implies that when using the
default event publishing strategy, your listener will not be notified about an entry
whose value is updated from one that matches the predicate to a new value that
does not match the predicate.

Since

118

version

3.7,

when

you

configure

Hazelcast

members

with

property

hazelcast.map.entry.filtering.natural.event.types set to true, handling of entry updates
conceptually treats value transition as entry, update or exit with regards to the predicate value
space. The following table compares how a listener is notified about an update to a map entry value
under

the

default

backwards-compatible

Hazelcast

behavior

(when

property

hazelcast.map.entry.filtering.natural.event.types is not set or is set to false) versus when set to
true:
Default

hazelcast.map.entry.filtering.
natural.event.types = true

When old value matches
predicate, new value does not
match predicate

No event is delivered to entry
listener

REMOVED event is delivered to
entry listener

When old value matches
predicate, new value matches
predicate

UPDATED event is delivered to
entry listener

UPDATED event is delivered to
entry listener

When old value does not match No event is delivered to entry
predicate, new value does not
listener
match predicate

No event is delivered to entry
listener

When old value does not match UPDATED event is delivered to
predicate, new value matches
entry listener
predicate

ADDED event is delivered to entry
listener

As an example, let’s listen to the changes made on an employee with the surname "Smith". First,
let’s create the Employee class.

public class Employee implements Serializable {
private final String surname;
public Employee(String surname) {
this.surname = surname;
}
@Override
public String toString() {
return "Employee{" +
"surname='" + surname + '\'' +
'}';
}
}
Then, let’s create a listener with predicate by adding a listener that tracks ADDED, UPDATED and REMOVED
entry events with the surname predicate.

119

public class ListenerWithPredicate {
public static void main(String[] args) {
Config config = new Config();
config.setProperty("hazelcast.map.entry.filtering.natural.event.types", "true
");
HazelcastInstance hz = Hazelcast.newHazelcastInstance(config);
IMap map = hz.getMap("map");
map.addEntryListener(new MyEntryListener(),
new SqlPredicate("surname=smith"), true);
System.out.println("Entry Listener registered");
}
static class MyEntryListener
implements EntryAddedListener,
EntryUpdatedListener,
EntryRemovedListener {
@Override
public void entryAdded(EntryEvent event) {
System.out.println("Entry Added:" + event);
}
@Override
public void entryRemoved(EntryEvent event) {
System.out.println("Entry Removed:" + event);
}
@Override
public void entryUpdated(EntryEvent event) {
System.out.println("Entry Updated:" + event);
}
}
}
And now, let’s play with the employee "smith" and see how that employee will be listened to.

120

public class Modify {
public static void main(String[] args) {
Config config = new Config();
config.setProperty("hazelcast.map.entry.filtering.natural.event.types", "true
");
HazelcastInstance hz = Hazelcast.newHazelcastInstance(config);
IMap map = hz.getMap("map");
map.put("1", new Employee("smith"));
map.put("2", new Employee("jordan"));
System.out.println("done");
System.exit(0);
}
}
When you first run the class ListenerWithPredicate and then run Modify, you will see output similar
to the listing below.

entryAdded:EntryEvent {Address[192.168.178.10]:5702} key=1,oldValue=null,
value=Person{name= smith }, event=ADDED, by Member [192.168.178.10]:5702



Please refer to Continuous Query Cache for more information.

7.2.13. Removing Map Entries in Bulk with Predicates
You can remove all map entries that match your predicate. For this, Hazelcast offers the method
removeAll(). Its syntax is as follows:

void removeAll(Predicate predicate);
Normally the map entries matching the predicate are found with a full scan of the map. If the
entries are indexed, Hazelcast uses the index search to find them. With index, you can expect that
finding the entries is faster.



When removeAll() is called, ALL entries in the caller member’s Near Cache are also
removed.

7.2.14. Adding Interceptors
You can add intercept operations and execute your own business logic synchronously blocking the
operations. You can change the returned value from a get operation, change the value in put, or
cancel operations by throwing an exception.
Interceptors are different from listeners. With listeners, you take an action after the operation has

121

been completed. Interceptor actions are synchronous and you can alter the behavior of operation,
change its values, or totally cancel it.
Map interceptors are chained, so adding the same interceptor multiple times to the same map can
result in duplicate effects. This can easily happen when the interceptor is added to the map at
member initialization, so that each member adds the same interceptor. When you add the
interceptor in this way, be sure to implement the hashCode() method to return the same value for
every instance of the interceptor. It is not strictly necessary, but it is a good idea to also implement
equals() as this will ensure that the map interceptor can be removed reliably.
The IMap API has two methods for adding and removing an interceptor to the map: addInterceptor
and removeInterceptor. Please also refer to the MapInterceptor interface to see the methods used to
intercept the changes in a map.
The following is an example usage.

122

public class MapInterceptorMember {
public static void main(String[] args) {
HazelcastInstance hz = Hazelcast.newHazelcastInstance();
IMap map = hz.getMap("themap");
map.addInterceptor(new MyMapInterceptor());
map.put("1", "1");
System.out.println(map.get("1"));
}
private static class MyMapInterceptor implements MapInterceptor {
@Override
public Object interceptGet(Object value) {
return value + "-foo";
}
@Override
public void afterGet(Object value) {
}
@Override
public Object interceptPut(Object oldValue, Object newValue) {
return null;
}
@Override
public void afterPut(Object value) {
}
@Override
public Object interceptRemove(Object removedValue) {
return null;
}
@Override
public void afterRemove(Object value) {
}
}
}

7.2.15. Preventing Out of Memory Exceptions
It is very easy to trigger an out of memory exception (OOME) with query-based map methods,
especially with large clusters or heap sizes. For example, on a cluster with five members having 10
GB of data and 25 GB heap size per member, a single call of IMap.entrySet() fetches 50 GB of data
and crashes the calling instance.

123

A call of IMap.values() may return too much data for a single member. This can also happen with a
real query and an unlucky choice of predicates, especially when the parameters are chosen by a
user of your application.
To prevent this, you can configure a maximum result size limit for query based operations. This is
not a limit like SELECT * FROM map LIMIT 100, which you can achieve by a Paging Predicate. A
maximum result size limit for query based operations is meant to be a last line of defense to
prevent your members from retrieving more data than they can handle.
The Hazelcast component which calculates this limit is the QueryResultSizeLimiter.
Setting Query Result Size Limit
If the QueryResultSizeLimiter is activated, it calculates a result size limit per partition. Each
QueryOperation runs on all partitions of a member, so it collects result entries as long as the member
limit is not exceeded. If that happens, a QueryResultSizeExceededException is thrown and propagated
to the calling instance.
This feature depends on an equal distribution of the data on the cluster members to calculate the
result

size

limit

per

member.

Therefore,

there

is

a

minimum

value

defined

in

QueryResultSizeLimiter.MINIMUM_MAX_RESULT_LIMIT. Configured values below the minimum will be
increased to the minimum.
Local Pre-check
In addition to the distributed result size check in the QueryOperations, there is a local pre-check on
the calling instance. If you call the method from a client, the pre-check is executed on the member
that invokes the QueryOperations.
Since the local pre-check can increase the latency of a QueryOperation, you can configure how many
local partitions should be considered for the pre-check, or you can deactivate the feature
completely.
Scope of Result Size Limit
Besides the designated query operations, there are other operations that use predicates internally.
Those method calls will throw the QueryResultSizeExceededException as well. Please see the
following matrix to see the methods that are covered by the query result size limit.

124

Configuring Query Result Size
The query result size limit is configured via the following system properties.
• hazelcast.query.result.size.limit: Result size limit for query operations on maps. This value
defines the maximum number of returned elements for a single query result. If a query exceeds
this number of elements, a QueryResultSizeExceededException is thrown.
• hazelcast.query.max.local.partition.limit.for.precheck: Maximum value of local partitions to
trigger local pre-check for TruePredicate query operations on maps.
Please refer to the System Properties appendix to see the full descriptions of these properties and
how to set them.

7.3. Queue
Hazelcast distributed queue is an implementation of java.util.concurrent.BlockingQueue. Being
distributed, Hazelcast distributed queue enables all cluster members to interact with it. Using
Hazelcast distributed queue, you can add an item in one cluster member and remove it from
another one.

7.3.1. Getting a Queue and Putting Items
Use the Hazelcast instance’s getQueue method to get the queue, then use the queue’s put method to
put items into the queue.

125

HazelcastInstance hazelcastInstance = Hazelcast.newHazelcastInstance();
BlockingQueue queue = hazelcastInstance.getQueue( "tasks" );
queue.put( new MyTask() );
MyTask task = queue.take();
boolean offered = queue.offer( new MyTask(), 10, TimeUnit.SECONDS );
task = queue.poll( 5, TimeUnit.SECONDS );
if ( task != null ) {
//process task
}
FIFO ordering will apply to all queue operations across the cluster. The user objects (such as MyTask
in the example above) that are enqueued or dequeued have to be Serializable.
Hazelcast distributed queue performs no batching while iterating over the queue. All items will be
copied locally and iteration will occur locally.
Hazelcast distributed queue uses ItemListener to listen to the events that occur when items are
added to and removed from the queue. Please refer to the Listening for Item Events section for
information on how to create an item listener class and register it.

7.3.2. Creating an Example Queue
The following example code illustrates a distributed queue that connects a producer and consumer.
Putting Items on the Queue
Let’s put one integer on the queue every second, 100 integers total.

public class ProducerMember {
public static void main( String[] args ) throws Exception {
HazelcastInstance hz = Hazelcast.newHazelcastInstance();
IQueue queue = hz.getQueue( "queue" );
for ( int k = 1; k < 100; k++ ) {
queue.put( k );
System.out.println( "Producing: " + k );
Thread.sleep(1000);
}
queue.put( -1 );
System.out.println( "Producer Finished!" );
}
}
Producer puts a -1 on the queue to show that the `put`s are finished.

126

Taking Items off the Queue
Now, let’s create a Consumer class to take a message from this queue, as shown below.

public class ConsumerMember {
public static void main( String[] args ) throws Exception {
HazelcastInstance hz = Hazelcast.newHazelcastInstance();
IQueue queue = hz.getQueue( "queue" );
while ( true ) {
int item = queue.take();
System.out.println( "Consumed: " + item );
if ( item == -1 ) {
queue.put( -1 );
break;
}
Thread.sleep( 5000 );
}
System.out.println( "Consumer Finished!" );
}
}
As seen in the above example code, Consumer waits five seconds before it consumes the next
message. It stops once it receives -1. Also note that Consumer puts -1 back on the queue before the
loop is ended.
When you first start Producer and then start Consumer, items produced on the queue will be
consumed from the same queue.
Balancing the Queue Operations
From the above example code, you can see that an item is produced every second and consumed
every five seconds. Therefore, the consumer keeps growing. To balance the produce/consume
operation, let’s start another consumer. This way, consumption is distributed to these two
consumers, as seen in the sample outputs below.
The second consumer is started. After a while, here is the first consumer output:

...
Consumed 13
Consumed 15
Consumer 17
...
Here is the second consumer output:

127

...
Consumed 14
Consumed 16
Consumer 18
...
In the case of a lot of producers and consumers for the queue, using a list of queues may solve the
queue bottlenecks. In this case, be aware that the order of the messages sent to different queues is
not guaranteed. Since in most cases strict ordering is not important, a list of queues is a good
solution.



The items are taken from the queue in the same order they were put on the queue.
However, if there is more than one consumer, this order is not guaranteed.

ItemIDs When Offering Items
Hazelcast gives an itemId for each item you offer, which is an incrementing sequence identification
for the queue items. You should consider the following to understand the itemId assignment
behavior:
• When a Hazelcast member has a queue and that queue is configured to have at least one
backup, and that member is restarted, the itemId assignment resumes from the last known
highest itemId before the restart; itemId assignment does not start from the beginning for the
new items.
• When the whole cluster is restarted, the same behavior explained in the above consideration
applies if your queue has a persistent data store (QueueStore). If the queue has QueueStore, the
itemId for the new items are given, starting from the highest itemId found in the IDs returned by
the method loadAllKeys. If the method loadAllKeys does not return anything, the `itemId`s will
started from the beginning after a cluster restart.
• The above two considerations mean there will be no duplicated `itemId`s in the memory or in
the persistent data store.

7.3.3. Setting a Bounded Queue
A bounded queue is a queue with a limited capacity. When the bounded queue is full, no more
items can be put into the queue until some items are taken out.
To turn a Hazelcast distributed queue into a bounded queue, set the capacity limit with the max-size
property. You can set the max-size property in the configuration, as shown below. max-size specifies
the maximum size of the queue. Once the queue size reaches this value, put operations will be
blocked until the queue size goes below max-size, which happens when a consumer removes items
from the queue.
Let’s set 10 as the maximum size of our example queue in Creating an Example Queue.

128


...

10

...

When the producer is started, ten items are put into the queue and then the queue will not allow
more put operations. When the consumer is started, it will remove items from the queue. This
means that the producer can put more items into the queue until there are ten items in the queue
again, at which point the put operation again becomes blocked.
In this example code, the producer is five times faster than the consumer. It will effectively always
be waiting for the consumer to remove items before it can put more on the queue. For this example
code, if maximum throughput is the goal, it would be a good option to start multiple consumers to
prevent the queue from filling up.

7.3.4. Queueing with Persistent Datastore
Hazelcast allows you to load and store the distributed queue items from/to a persistent datastore
using the interface QueueStore. If queue store is enabled, each item added to the queue will also be
stored at the configured queue store. When the number of items in the queue exceeds the memory
limit, the subsequent items are persisted in the queue store, they are not stored in the queue
memory.
The QueueStore interface enables you to store, load and delete queue items with methods like store,
storeAll, load and delete. The following example class includes all of the QueueStore methods.

129

public class TheQueueStore implements QueueStore {
@Override
public void delete(Long key) {
System.out.println("delete");
}
@Override
public void store(Long key, Item value) {
System.out.println("store");
}
@Override
public void storeAll(Map map) {
System.out.println("store all");
}
@Override
public void deleteAll(Collection keys) {
System.out.println("deleteAll");
}
@Override
public Item load(Long key) {
System.out.println("load");
return null;
}
@Override
public Map loadAll(Collection keys) {
System.out.println("loadALl");
return null;
}
@Override
public Set loadAllKeys() {
System.out.println("loadAllKeys");
return null;
}
}
Item must be serializable. The following is an example queue store configuration.

130


com.hazelcast.QueueStoreImpl

false
1000
500


Let’s explain the queue store properties.
• Binary: By default, Hazelcast stores the queue items in serialized form, and before it inserts the
queue items into the queue store, it deserializes them. If you are not reaching the queue store
from an external application, you might prefer that the items be inserted in binary form. Do this
by setting the binary property to true: then you can get rid of the deserialization step, which is a
performance optimization. The binary property is false by default.
• Memory Limit: This is the number of items after which Hazelcast will store items only to the
datastore. For example, if the memory limit is 1000, then the 1001st item will be put only to the
datastore. This feature is useful when you want to avoid out-of-memory conditions. If you want
to always use memory, you can set it to Integer.MAX_VALUE. The default number for memory-limit
is 1000.
• Bulk Load: When the queue is initialized, items are loaded from QueueStore in bulks. Bulk load
is the size of these bulks. The default value of bulk-load is 250.

7.3.5. Split-Brain Protection for Queue
Queues can be configured to check for a minimum number of available members before applying
queue operations (see Split-Brain Protection). This is a check to avoid performing successful queue
operations on all parts of a cluster during a network partition.
Following is a list of methods that now support Split-Brain Protection checks. The list is grouped by
quorum type.
• WRITE, READ_WRITE
◦ Collection.addAll()
◦ Collection.removeAll(), Collection.retainAll()
◦ BlockingQueue.offer(), BlockingQueue.add(), BlockingQueue.put()
◦ BlockingQueue.drainTo()
◦ IQueue.poll(), Queue.remove(), IQueue.take()
◦ BlockingQueue.remove()
• READ, READ_WRITE
◦ Collection.clear()
◦ Collection.containsAll(), BlockingQueue.contains()
◦ Collection.isEmpty()

131

◦ Collection.iterator(), Collection.toArray()
◦ Queue.peek(), Queue.element()
◦ Collection.size()
◦ BlockingQueue.remainingCapacity()

7.3.6. Configuring Queue
The following are examples of queue configurations. It includes the QueueStore configuration, which
is explained in the Queueing with Persistent Datastore section.
Declarative:


0
1
0
-1

com.hazelcast.examples.ItemListener

true

com.hazelcast.QueueStoreImpl

false
10000
500


quorumname

Programmatic:

Config config = new Config();
QueueConfig queueConfig = config.getQueueConfig("default");
queueConfig.setName("MyQueue")
.setBackupCount(1)
.setMaxSize(0)
.setStatisticsEnabled(true)
.setQuorumName("quorumname");
queueConfig.getQueueStoreConfig()
.setEnabled(true)
.setClassName("com.hazelcast.QueueStoreImpl")
.setProperty("binary", "false");
config.addQueueConfig(queueConfig);

132

Hazelcast distributed queue has one synchronous backup by default. By having this backup, when a
cluster member with a queue goes down, another member having the backup of that queue will
continue. Therefore, no items are lost. You can define the number of synchronous backups for a
queue using the backup-count element in the declarative configuration. A queue can also have
asynchronous backups: you can define the number of asynchronous backups using the asyncbackup-count element.
To set the maximum size of the queue, use the max-size element. To purge unused or empty queues
after a period of time, use the empty-queue-ttl element. If you define a value (time in seconds) for
the empty-queue-ttl element, then your queue will be destroyed if it stays empty or unused for the
time in seconds that you give.
The following is the full list of queue configuration elements with their descriptions.
• max-size: Maximum number of items in the queue. It is used to set an upper bound for the
queue. You will not be able to put more items when the queue reaches to this maximum size
whether you have a queue store configured or not.
• backup-count: Number of synchronous backups. Queue is a non-partitioned data structure, so all
entries of a queue reside in one partition. When this parameter is '1', it means there will be one
backup of that queue in another member in the cluster. When it is '2', two members will have
the backup.
• async-backup-count: Number of asynchronous backups.
• empty-queue-ttl: Used to purge unused or empty queues. If you define a value (time in seconds)
for this element, then your queue will be destroyed if it stays empty or unused for that time.
• item-listeners: Adds listeners (listener classes) for the queue items. You can also set the
attribute include-value to true if you want the item event to contain the item values. You can set
local to true if you want to listen to the items on the local member.
• queue-store: Includes the queue store factory class name and the properties binary, memory
limit and bulk load. Please refer to Queueing with Persistent Datastore.
• statistics-enabled: If set to true, you can retrieve statistics for this queue using the method
getLocalQueueStats().
• quorum-ref : Name of quorum configuration that you want this queue to use.

7.4. MultiMap
Hazelcast MultiMap is a specialized map where you can store multiple values under a single key. Just
like any other distributed data structure implementation in Hazelcast, MultiMap is distributed and
thread-safe.
Hazelcast MultiMap is not an implementation of java.util.Map due to the difference in method
signatures. It supports most features of Hazelcast Map except for indexing, predicates and
MapLoader/MapStore. Yet, like Hazelcast Map, entries are almost evenly distributed onto all cluster
members. When a new member joins the cluster, the same ownership logic used in the distributed
map applies.

133

7.4.1. Getting a MultiMap and Putting an Entry
The following example creates a MultiMap and puts items into it. Use the HazelcastInstance
getMultiMap method to get the MultiMap, then use the MultiMap put method to put an entry into the
MultiMap.

HazelcastInstance hazelcastInstance = Hazelcast.newHazelcastInstance();
MultiMap map = hazelcastInstance.getMultiMap( "map" );
map.put( "a", "1" );
map.put( "a", "2" );
map.put( "b", "3" );
System.out.println( "PutMember:Done" );
Now let’s print the entries in this MultiMap.

HazelcastInstance hazelcastInstance = Hazelcast.newHazelcastInstance();
MultiMap map = hazelcastInstance.getMultiMap("map");
map.put("a", "1");
map.put("a", "2");
map.put("b", "3");
System.out.printf("PutMember:Done");
for (String key: map.keySet()){
Collection values = map.get(key);
System.out.printf("%s -> %s\n", key, values);
}
After you run the first code sample, run the PrintMember sample. You will see the key a has two
values, as shown below.
b → [3]
a → [2, 1]
Hazelcast MultiMap uses EntryListener to listen to events which occur when entries are added to,
updated in or removed from the MultiMap. Please refer to the Listening for MultiMap Events
section for information on how to create an entry listener class and register it.

7.4.2. Configuring MultiMap
When using MultiMap, the collection type of the values can be either Set or List. Configure the
collection type with the valueCollectionType parameter. If you choose Set, duplicate and null values
are not allowed in your collection and ordering is irrelevant. If you choose List, ordering is
relevant and your collection can include duplicate and null values.
You can also enable statistics for your MultiMap with the statisticsEnabled parameter. If you
enable statisticsEnabled, statistics can be retrieved with getLocalMultiMapStats() method.

134



Currently, eviction is not supported for the MultiMap data structure.

The following are the example MultiMap configurations.
Declarative:


0
1
SET

com.hazelcast.examples.EntryListener

quorumname

Programmatic:

MultiMapConfig mmConfig = new MultiMapConfig();
mmConfig.setName( "default" )
.setBackupCount( 0 ).setAsyncBackupCount( 1 )
.setValueCollectionType( "SET" )
.setQuorumName( "quorumname" );
The following are the configuration elements and their descriptions:
• backup-count: Defines the number of synchronous backups. For example, if it is set to 1, backup
of a partition will be placed on one other member. If it is 2, it will be placed on two other
members.
• async-backup-count: The number of asynchronous backups. Behavior is the same as that of the
backup-count element.
• statistics-enabled: You can retrieve some statistics such as owned entry count, backup entry
count, last update time and locked entry count by setting this parameter’s value as "true". The
method for retrieving the statistics is getLocalMultiMapStats().
• value-collection-type: Type of the value collection. It can be SET or LIST.
• entry-listeners: Lets you add listeners (listener classes) for the map entries. You can also set the
attribute include-value to true if you want the item event to contain the entry values. You can
set local to true if you want to listen to the entries on the local member.
• quorum-ref: Name of quorum configuration that you want this MultiMap to use. Please see the
Split-Brain Protection for MultiMap and TransactionalMultiMap section.

7.4.3. Split-Brain Protection for MultiMap and TransactionalMultiMap
MultiMap & TransactionalMultiMap can be configured to check for a minimum number of

135

available members before applying their operations (see Split-Brain Protection). This is a check to
avoid performing successful queue operations on all parts of a cluster during a network partition.
Following is a list of methods that now support Split-Brain Protection checks. The list is grouped by
quorum type.
MultiMap:
• WRITE, READ_WRITE:
◦ clear
◦ forceUnlock
◦ lock
◦ put
◦ remove
◦ tryLock
◦ unlock
• READ, READ_WRITE:
◦ containsEntry
◦ containsKey
◦ containsValue
◦ entrySet
◦ get
◦ isLocked
◦ keySet
◦ localKeySet
◦ size
◦ valueCount
◦ values
TransactionalMultiMap:
• WRITE, READ_WRITE:
◦ put
◦ remove
• READ, READ_WRITE:
◦ size
◦ get
◦ valueCount
Configuring Split-Brain Protection
Split-Brain protection for MultiMap can be configured programmatically using the method
setQuorumName(), or declaratively using the element quorum-ref. Following is an example declarative
configuration:

136


...
quorumname
...

The value of quorum-ref should be the quorum configuration name which you configured under the
quorum element as explained in the Split-Brain Protection section.

7.5. Set
Hazelcast Set is a distributed and concurrent implementation of java.util.Set.
• Hazelcast Set does not allow duplicate elements.
• Hazelcast Set does not preserve the order of elements.
• Hazelcast Set is a non-partitioned data structure—all the data that belongs to a set will live on
one single partition in that member.
• Hazelcast Set cannot be scaled beyond the capacity of a single machine. Since the whole set lives
on a single partition, storing a large amount of data on a single set may cause memory pressure.
Therefore, you should use multiple sets to store a large amount of data. This way, all the sets
will be spread across the cluster, sharing the load.
• A backup of Hazelcast Set is stored on a partition of another member in the cluster so that data
is not lost in the event of a primary member failure.
• All items are copied to the local member and iteration occurs locally.
• The equals method implemented in Hazelcast Set uses a serialized byte version of objects, as
opposed to java.util.HashSet.

7.5.1. Getting a Set and Putting Items
Use the HazelcastInstance getSet method to get the Set, then use the add method to put items into
the Set.

HazelcastInstance hz = Hazelcast.newHazelcastInstance();
ISet set = hz.getSet("set");
set.add("Tokyo");
set.add("Paris");
set.add("London");
set.add("New York");
System.out.println("Putting finished!");
Hazelcast Set uses ItemListener to listen to events that occur when items are added to and removed
from the Set. Please refer to the Listening for Item Events section for information on how to create
an item listener class and register it.

137

7.5.2. Configuring Set
The following are the example set configurations.
Declarative:


1
0
10

com.hazelcast.examples.ItemListener

quorumname

Programmatic:

Config config = new Config();
CollectionConfig collectionSet = config.getSetConfig("MySet");
collectionSet.setBackupCount(1)
.setMaxSize(10)
.setQuorumName("quorumname");
Set configuration has the following elements.
• statistics-enabled: True (default) if statistics gathering is enabled on the Set, false otherwise.
• backup-count: Count of synchronous backups. Set is a non-partitioned data structure, so all
entries of a Set reside in one partition. When this parameter is '1', it means there will be one
backup of that Set in another member in the cluster. When it is '2', two members will have the
backup.
• async-backup-count: Count of asynchronous backups.
• max-size: The maximum number of entries for this Set. It can be any number between 0 and
Integer.MAX_VALUE. Its default value is 0, meaning there is no capacity constraint.
• item-listeners: Lets you add listeners (listener classes) for the list items. You can also set the
attributes include-value to true if you want the item event to contain the item values. You can
set local to true if you want to listen to the items on the local member.
• quorum-ref: Name of quorum configuration that you want this Set to use. Please refer to the
Split-Brain Protection for ISet and TransactionalSet section.

7.5.3. Split-Brain Protection for ISet and TransactionalSet
ISet & TransactionalSet can be configured to check for a minimum number of available members
before applying queue operations (see Split-Brain Protection). This is a check to avoid performing
successful queue operations on all parts of a cluster during a network partition.

138

Following is a list of methods that now support Split-Brain Protection checks. The list is grouped by
quorum type.
ISet:
• WRITE, READ_WRITE:
◦ add
◦ addAll
◦ clear
◦ remove
◦ removeAll
• READ, READ_WRITE:
◦ contains
◦ containsAll
◦ isEmpty
◦ iterator
◦ size
◦ toArray
TransactionalSet:
• WRITE, READ_WRITE:
◦ add
◦ remove
• READ, READ_WRITE:
◦ size
Configuring Split-Brain Protection
Split-Brain protection for ISet can be configured programmatically using the method
setQuorumName(), or declaratively using the element quorum-ref. Following is an example declarative
configuration:


...
quorumname
...

The value of quorum-ref should be the quorum configuration name which you configured under the
quorum element as explained in the Split-Brain Protection section.

7.6. List
Hazelcast List (IList) is similar to Hazelcast Set, but Hazelcast List also allows duplicate elements.

139

• Besides allowing duplicate elements, Hazelcast List preserves the order of elements.
• Hazelcast List is a non-partitioned data structure where values and each backup are
represented by their own single partition.
• Hazelcast List cannot be scaled beyond the capacity of a single machine.
• All items are copied to local and iteration occurs locally.

While IMap and ICache are the recommended data structures to be used by
Hazelcast Jet, IList can also be used by it for unit testing or similar non-production



situations. Please see here in the Hazelcast Jet Reference Manual to learn how Jet
can use IList, e.g., how it can fill IList with data, consume it in a Jet job and drain
the results to another IList. Please also see the Fast Batch Processing and Real-Time
Stream Processing use cases for Hazelcast Jet.

7.6.1. Getting a List and Putting Items
Use the HazelcastInstance getList method to get the List, then use the add method to put items into
the List.

HazelcastInstance hz = Hazelcast.newHazelcastInstance();
IList list = hz.getList("list");
list.add("Tokyo");
list.add("Paris");
list.add("London");
list.add("New York");
System.out.println("Putting finished!");
Hazelcast List uses ItemListener to listen to events that occur when items are added to and removed
from the List. Please refer to the Listening for Item Events section for information on how to create
an item listener class and register it.

7.6.2. Configuring List
The following are example list configurations.
Declarative:

140


1
0
10


com.hazelcast.examples.ItemListener


quorumname

Programmatic:

Config config = new Config();
CollectionConfig collectionList = config.getListConfig("MyList");
collectionList.setBackupCount(1)
.setMaxSize(10)
.setQuorumName("quorumname");
List configuration has the following elements.
• statistics-enabled: True (default) if statistics gathering is enabled on the list, false otherwise.
• backup-count: Number of synchronous backups. List is a non-partitioned data structure, so all
entries of a List reside in one partition. When this parameter is '1', there will be one backup of
that List in another member in the cluster. When it is '2', two members will have the backup.
• async-backup-count: Number of asynchronous backups.
• max-size: The maximum number of entries for this List.
• item-listeners: Lets you add listeners (listener classes) for the list items. You can also set the
attribute include-value to true if you want the item event to contain the item values. You can set
the attribute local to true if you want to listen the items on the local member.
• quorum-ref: Name of quorum configuration that you want this List to use. Please see the SplitBrain Protection for IList and TransactionalList section.

7.6.3. Split-Brain Protection for IList and TransactionalList
IList & TransactionalList can be configured to check for a minimum number of available members
before applying queue operations (see Split-Brain Protection). This is a check to avoid performing
successful queue operations on all parts of a cluster during a network partition.
Following is a list of methods that now support Split-Brain Protection checks. The list is grouped by
quorum type.
IList:
• WRITE, READ_WRITE:

141

◦ add
◦ addAll
◦ clear
◦ remove
◦ removeAll
◦ set
• READ, READ_WRITE:
◦ add
◦ contains
◦ containsAll
◦ get
◦ indexOf
◦ isEmpty
◦ iterator
◦ lastIndexOf
◦ listIterator
◦ size
◦ subList
◦ toArray
TransactionalList:
• WRITE, READ_WRITE:
◦ add
◦ remove
• READ, READ_WRITE:
◦ size
Configuring Split-Brain Protection
Split-Brain protection for IList can be configured programmatically using the method
setQuorumName(), or declaratively using the element quorum-ref. Following is an example declarative
configuration:


...
quorumname
...

The value of quorum-ref should be the quorum configuration name which you configured under the
quorum element as explained in the Split-Brain Protection section.

142

7.7. Ringbuffer
Hazelcast Ringbuffer is a replicated but not partitioned data structure that stores its data in a ringlike structure. You can think of it as a circular array with a given capacity. Each Ringbuffer has a tail
and a head. The tail is where the items are added and the head is where the items are overwritten
or expired. You can reach each element in a Ringbuffer using a sequence ID, which is mapped to
the elements between the head and tail (inclusive) of the Ringbuffer.

7.7.1. Getting a Ringbuffer and Reading Items
Reading from Ringbuffer is simple: get the Ringbuffer with the HazelcastInstance getRingbuffer
method, get its current head with the headSequence method and start reading. Use the method
readOne to return the item at the given sequence; readOne blocks if no item is available. To read the
next item, increment the sequence by one.

HazelcastInstance hz = Hazelcast.newHazelcastInstance();
Ringbuffer ringbuffer = hz.getRingbuffer("rb");
long sequence = ringbuffer.headSequence();
while(true){
String item = ringbuffer.readOne(sequence);
sequence++;
// process item
}
By exposing the sequence, you can now move the item from the Ringbuffer as long as the item is
still available. If the item is not available any longer, StaleSequenceException is thrown.

7.7.2. Adding Items to a Ringbuffer
Adding an item to a Ringbuffer is also easy with the Ringbuffer add method:

Ringbuffer ringbuffer = hz.getRingbuffer("SampleRB");
ringbuffer.add("someitem");
Use the method add to return the sequence of the inserted item; the sequence value will always be
unique. You can use this as a very cheap way of generating unique IDs if you are already using
Ringbuffer.

7.7.3. IQueue vs. Ringbuffer
Hazelcast Ringbuffer can sometimes be a better alternative than an Hazelcast IQueue. Unlike
IQueue, Ringbuffer does not remove the items, it only reads items using a certain position. There
are many advantages to this approach:
• The same item can be read multiple times by the same thread. This is useful for realizing
semantics of read-at-least-once or read-at-most-once.

143

• The same item can be read by multiple threads. Normally you could use an IQueue per thread
for the same semantic, but this is less efficient because of the increased remoting. A take from
an IQueue is destructive, so the change needs to be applied for backup also, which is why a
queue.take() is more expensive than a ringBuffer.read(…).
• Reads are extremely cheap since there is no change in the Ringbuffer. Therefore no replication
is required.
• Reads and writes can be batched to speed up performance. Batching can dramatically improve
the performance of Ringbuffer.

7.7.4. Configuring Ringbuffer Capacity
By default, a Ringbuffer is configured with a capacity of 10000 items. This creates an array with a
size of 10000. If a time-to-live is configured, then an array of longs is also created that stores the
expiration time for every item. In a lot of cases you may want to change this capacity number to
something that better fits your needs.
Below is a declarative configuration example of a Ringbuffer with a capacity of 2000 items.


2000

Currently, Hazelcast Ringbuffer is not a partitioned data structure; its data is stored in a single
partition and the replicas are stored in another partition. Therefore, create a Ringbuffer that can
safely fit in a single cluster member.

7.7.5. Backing Up Ringbuffer
Hazelcast Ringbuffer has a single synchronous backup by default. You can control the Ringbuffer
backup just like most of the other Hazelcast distributed data structures by setting the synchronous
and asynchronous backups: backup-count and async-backup-count. In the example below, a
Ringbuffer is configured with no synchronous backups and one asynchronous backup:


0
1

An asynchronous backup will probably give you better performance. However, there is a chance
that the item added will be lost when the member owning the primary crashes before the backup
could complete. You may want to consider batching methods if you need high performance but do
not want to give up on consistency.

7.7.6. Configuring Ringbuffer Time-To-Live
You can configure Hazelcast Ringbuffer with a time-to-live in seconds. Using this setting, you can

144

control how long the items remain in the Ringbuffer before they are expired. By default, the timeto-live is set to 0, meaning that unless the item is overwritten, it will remain in the Ringbuffer
indefinitely. If you set a time-to-live and an item is added, then, depending on the Overflow Policy,
either the oldest item is overwritten, or the call is rejected.
In the example below, a Ringbuffer is configured with a time-to-live of 180 seconds.


180


7.7.7. Setting Ringbuffer Overflow Policy
Using the overflow policy, you can determine what to do if the oldest item in the Ringbuffer is not
old enough to expire when more items than the configured Ringbuffer capacity are being added.
The below options are currently available.
• OverflowPolicy.OVERWRITE: The oldest item is overwritten.
• OverflowPolicy.FAIL: The call is aborted. The methods that make use of the OverflowPolicy
return -1 to indicate that adding the item has failed.
Overflow policy gives you fine control on what to do if the Ringbuffer is full. You can also use the
overflow policy to apply a back pressure mechanism. The following example code shows the usage
of an exponential backoff.

Random random = new Random();
HazelcastInstance hz = Hazelcast.newHazelcastInstance();
Ringbuffer rb = hz.getRingbuffer("rb");
long i = 100;
while (true) {
long sleepMs = 100;
for (; ; ) {
long result = rb.addAsync(i, OverflowPolicy.FAIL).get();
if (result != -1) {
break;
}
TimeUnit.MILLISECONDS.sleep(sleepMs);
sleepMs = min(5000, sleepMs * 2);
}
// add a bit of random delay to make it look a bit more realistic
Thread.sleep(random.nextInt(10));
System.out.println("Written: " + i);
i++;
}

145

7.7.8. Ringbuffer with Persistent Datastore
Hazelcast allows you to load and store the Ringbuffer items from/to a persistent datastore using the
interface RingbufferStore. If a Ringbuffer store is enabled, each item added to the Ringbuffer will
also be stored at the configured Ringbuffer store.
If the Ringbuffer store is configured, you can get items with sequences which are no longer in the
actual Ringbuffer but are only in the Ringbuffer store. This will probably be much slower but still
allow you to continue consuming items from the Ringbuffer even if they are overwritten with
newer items in the Ringbuffer.
When a Ringbuffer is being instantiated, it will check if the Ringbuffer store is configured and will
request the latest sequence in the Ringbuffer store. This is to enable the Ringbuffer to start with
sequences larger than the ones in the Ringbuffer store. In this case, the Ringbuffer is empty but you
can still request older items from it (which will be loaded from the Ringbuffer store).
The Ringbuffer store will store items in the same format as the Ringbuffer. If the BINARY in-memory
format is used, the Ringbuffer store must implement the interface RingbufferStore meaning
that the Ringbuffer will receive items in the binary format. If the OBJECT in-memory format is used,
the Ringbuffer store must implement the interface RingbufferStore, where K is the type of item
being stored (meaning that the Ringbuffer store will receive the deserialized object).
When adding items to the Ringbuffer, the method storeAll allows you to store items in batches.
The following example class includes all of the RingbufferStore methods.

146

public class TheRingbufferObjectStore implements RingbufferStore {
@Override
public void store(long sequence, Item data) {
System.out.println("Object store");
}
@Override
public void storeAll(long firstItemSequence, Item[] items) {
System.out.println("Object store all");
}
@Override
public Item load(long sequence) {
System.out.println("Object load");
return null;
}
@Override
public long getLargestSequence() {
System.out.println("Object get largest sequence");
return -1;
}
}
Item must be serializable. The following is an example of a Ringbuffer with the Ringbuffer store
configured and enabled.


10000
30
1
0
BINARY

com.hazelcast.RingbufferStoreImpl


Below are the explanations for the Ringbuffer store configuration elements:
• class-name: Name of the class implementing the `RingbufferStore interface.
• factory-class-name: Name of the class implementing the RingbufferStoreFactory interface. This
interface allows a factory class to be registered instead of a class implementing the
RingbufferStore interface.
Either the class-name or the factory-class-name element should be used.

147

7.7.9. Configuring Ringbuffer In-Memory Format
You can configure Hazelcast Ringbuffer with an in-memory format that controls the format of the
Ringbuffer’s stored items. By default, BINARY in-memory format is used, meaning that the object is
stored in a serialized form. You can select the OBJECT in-memory format, which is useful when
filtering is applied or when the OBJECT in-memory format has a smaller memory footprint than
BINARY.
In the declarative configuration example below, a Ringbuffer is configured with the OBJECT inmemory format:


OBJECT


7.7.10. Configuring Split-Brain Protection for Ringbuffer
Ringbuffer can be configured to check for a minimum number of available members before
applying Ringbuffer operations. This is a check to avoid performing successful Ringbuffer
operations on all parts of a cluster during a network partition and can be configured using the
element quorum-ref. You should set this element’s value as the quorum’s name, which you
configured under the quorum element as explained in the Split-Brain Protection section. Following is
an example snippet:


quorumname

Following is a list of methods that now support Split-Brain Protection checks. The list is grouped by
quorum type.
• WRITE, READ_WRITE:
◦ add
◦ addAllAsync
◦ addAsync
• READ, READ_WRITE:
◦ capacity
◦ headSequence
◦ readManyAsync
◦ readOne
◦ remainingCapacity
◦ size
◦ tailSequence

148

7.7.11. Adding Batched Items
In the previous examples, the method ringBuffer.add() is used to add an item to the Ringbuffer.
The problems with this method are that it always overwrites and that it does not support batching.
Batching can have a huge impact on the performance. You can use the method addAllAsync to
support batching.
Please see the following example code.

List items = Arrays.asList("1","2","3");
ICompletableFuture f = rb.addAllAsync(items, OverflowPolicy.OVERWRITE);
f.get();
In

the

above

case,

three

strings

are

added

to

the

Ringbuffer

using

the

policy

OverflowPolicy.OVERWRITE. Please see the Overflow Policy section for more information.

7.7.12. Reading Batched Items
In the previous example, the readOne method read items from the Ringbuffer. readOne is simple but
not very efficient for the following reasons:
• readOne does not use batching.
• readOne cannot filter items at the source; the items need to be retrieved before being filtered.
The method readManyAsync can read a batch of items and can filter items at the source.
Please see the following example code.

ICompletableFuture> readManyAsync(
long startSequence,
int minCount,
int maxCount,
IFunction filter);
The meanings of the readManyAsync arguments are given below.
• startSequence: Sequence of the first item to read.
• minCount: Minimum number of items to read. If you do not want to block, set it to 0. If you want
to block for at least one item, set it to 1.
• maxCount: Maximum number of the items to retrieve. Its value cannot exceed 1000.
• filter: A function that accepts an item and checks if it should be returned. If no filtering should
be applied, set it to null.
A full example is given below.

149

long sequence = rb.headSequence();
for(;;) {
ICompletableFuture> f = rb.readManyAsync(sequence, 1, 10,
null);
ReadResultSet rs = f.get();
for (String s : rs) {
System.out.println(s);
}
sequence+=rs.readCount();
}
Please take a careful look at how your sequence is being incremented. You cannot always rely on
the number of items being returned if the items are filtered out.

7.7.13. Using Async Methods
Hazelcast Ringbuffer provides asynchronous methods for more powerful operations like batched
writing or batched reading with filtering. To make these methods synchronous, just call the method
get() on the returned future.
Please see the following example code.

ICompletableFuture f = ringbuffer.addAsync(item, OverflowPolicy.FAIL);
f.get();
However, you can also use ICompletableFuture to get notified when the operation has completed.
The advantage of ICompletableFuture is that the thread used for the call is not blocked till the
response is returned.
Please see the below code as an example of when you want to get notified when a batch of reads
has completed.

ICompletableFuture> f = rb.readManyAsync(sequence, min, max,
someFilter);
f.andThen(new ExecutionCallback>() {
@Override
public void onResponse(ReadResultSet response) {
for (String s : response) {
System.out.println("Received:" + s);
}
}
@Override
public void onFailure(Throwable t) {
t.printStackTrace();
}
});

150

7.7.14. Ringbuffer Configuration Examples
The following shows the declarative configuration of a Ringbuffer called rb. The configuration is
modeled after the Ringbuffer defaults.


10000
1
0
0
BINARY
quorumname

You can also configure a Ringbuffer programmatically. The following is a programmatic version of
the above declarative configuration.

Config config = new Config();
RingbufferConfig rbConfig = config.getRingbufferConfig("myRB");
rbConfig.setCapacity(10000)
.setBackupCount(1)
.setAsyncBackupCount(0)
.setTimeToLiveSeconds(0)
.setInMemoryFormat(InMemoryFormat.BINARY)
.setQuorumName("quorumname");

7.8. Topic
Hazelcast provides a distribution mechanism for publishing messages that are delivered to multiple
subscribers. This is also known as a publish/subscribe (pub/sub) messaging model. Publishing and
subscribing operations are cluster wide. When a member subscribes to a topic, it is actually
registering for messages published by any member in the cluster, including the new members that
joined after you add the listener.



Publish operation is async. It does not wait for operations to run in remote
members; it works as fire and forget.

7.8.1. Getting a Topic and Publishing Messages
Use the HazelcastInstance’s getTopic method to get the topic, then use the topic’s publish method to
publish your messages. The following is a sample publisher:

151

public class TopicPublisher {
public static void main(String[] args) {
HazelcastInstance hz = Hazelcast.newHazelcastInstance();
ITopic topic = hz.getTopic("topic");
topic.publish(new Date());
}
}
And here is a sample subscriber:

public class TopicSubscriber {
public static void main(String[] args) {
HazelcastInstance hz = Hazelcast.newHazelcastInstance();
ITopic topic = hz.getTopic("topic");
topic.addMessageListener(new MessageListenerImpl());
System.out.println("Subscribed");
}
private static class MessageListenerImpl implements MessageListener {
public void onMessage(Message m) {
System.out.println("Received: " + m.getMessageObject());
}
}
}
Hazelcast Topic uses the MessageListener interface to listen for events that occur when a message is
received. Please refer to the Listening for Topic Messages section for information on how to create a
message listener class and register it.

7.8.2. Getting Topic Statistics
Topic has two statistic variables that you can query. These values are incremental and local to the
member.

HazelcastInstance hazelcastInstance = Hazelcast.newHazelcastInstance();
ITopic