RecoverPoint for VMs Deployment Automation – How to Deploy Complete RP4VMs Systems in Minutes
Wed, 21 Sep 2022 22:02:31 -0000
|Read Time: 0 minutes
I’m a firm believer in automating EVERYTHING, so in this blog we are going to show how we automate just about everything in RecoverPoint for VMs 😊.
In RP4VMs land, we have VM protection automation solutions, per-tag and per-cluster protection, use-case driven tasks, and so on. And now, I’m honored to present you a complete deployment automation solution.
So, first things first, what does it do?
- Automatically provisions vRPAs and the Plugin Server from OVA
- Automatically deploys an RP4VMs cluster based on the JSON configuration file
- Configures the Plugin Server (Optional)
- Connects to existing RP4VMs environments (Optional)
What is it?
It’s a Python-based script that operates with the RecoverPoint for VMs Deployment REST API and vCenter.
In terms of requirements, the following list should help:
- The script requires Python 3.x and supports every platform on which Python is supported.
- Requires the Python requests module. It can be installed using pip with the command: “pip install requests” or “python -m pip install requests”.
- RecoverPoint for VMs 5.3.x.
- Connectivity from the host on which the script is running to vCenter, vRPAs. It also requires connectivity to the plugin server and peer clusters if applicable.
- RP4VMs 5.3.x vRPA and Plugin Server OVA images must be located on the host that is running the script.
How do I use it?
The script accepts one mandatory parameter -configfile or --config-file and two optional parameters: (1) pluginserver and (2) connect to prevent configuration of the plugin server and to skip connection to other clusters accordingly.
Here is the full script syntax:
# python rp4vms_deploy.py -h
Usage: rp4vms_deploy.py [-h] -configfile CONFIGFILE [-pluginserver] [-connect] Script to automate RP4VMs deployment options: -h, --help show this help message and exit -configfile CONFIGFILE, --config-file CONFIGFILE Full path to the JSON config file -pluginserver, --config-plugin-server Optionally prevents Plugin Server configuration -connect, --connect-another-cluster Optionally prevents connection to a different cluster
Use cases
I might be biased but there are so many cases where this solution can come in handy. Let’s look at some common use cases:
- Greenfield deployment of a two-cluster system:
In this case, we would run the script twice in the following manner:
a) Deploying the first cluster:
# python rp4vms_deploy.py -configfile clusterA_config.json -connect
b) Deploying the second cluster including cluster connect:
# python rp4vms_deploy.py -configfile clusterB_config.json
2. Adding a new cluster to an existing system:
# python rp4vms_deploy.py -configfile cluster_config.json
3. Scale out – adding new clusters to a separate system on the same vCenter and plugin server:
a) Deploying the first cluster:
# python rp4vms_deploy.py -configfile clusterA_config.json -pluginserver -connect
b) Deploying the second cluster including cluster connect:
# python rp4vms_deploy.py -configfile clusterB_config.json -pluginserver
Script output
# python rp4vms_deploy.py -configfile config.json -> Provisioning Plugin Server from OVA Opening OVA source: C:\Users\idan\Downloads\RecoverPoint-PluginServer_v5.3.3_282.ova Opening VI target: vi://Administrator%40vsphere.local@10.0.0.10:443/Hopkinton/host/Hop_HA1/ Deploying to VI: vi://Administrator%40vsphere.local@10.0.0.10:443/Hopkinton/host/Hop_HA1/ Transfer Completed Powering on VM: Manchester_Plugin-Server Task Completed Completed successfully ---> OVA deployment completed successfully -> Provisioning vRPA1 from OVA Opening OVA source: C:\Users\idan\Downloads\EMC_RecoverPoint_vRPA-RP4VMs_rel5.3.SP3_m.348.ova Opening VI target: vi://Administrator%40vsphere.local@10.0.0.10:443/Hopkinton/host/Hop_HA1/ Deploying to VI: vi://Administrator%40vsphere.local@10.0.0.10:443/Hopkinton/host/Hop_HA1/ Transfer Completed Powering on VM: Manchester_vRPA1 Task Completed Completed successfully ---> OVA deployment completed successfully -> Provisioning vRPA2 from OVA Opening OVA source: C:\Users\idan\Downloads\EMC_RecoverPoint_vRPA-RP4VMs_rel5.3.SP3_m.348.ova Opening VI target: vi://Administrator%40vsphere.local@10.0.0.10:443/Hopkinton/host/Hop_HA1/ Deploying to VI: vi://Administrator%40vsphere.local@10.0.0.10:443/Hopkinton/host/Hop_HA1/ Transfer Completed Powering on VM: Manchester_vRPA2 Task Completed Completed successfully ---> OVA deployment completed successfully -> Checking connectivity to vRPAs ---> vRPA Mgmt IP 10.247.8.118 is reachable ---> vRPA Mgmt IP 10.247.8.119 is reachable ---> All vRPAs are reachable -> Running deployment pre-validation checks ---> Transaction LOCAL_1 RUNNING 0% ---> Pre-installation validation passed successfully -> Timezone detected: Europe/London -> Searching for available vRPAs ---> Transaction LOCAL_2 RUNNING 0% ---> Transaction LOCAL_2 RUNNING 50% ---> Transaction LOCAL_2 RUNNING 50% ---> Transaction LOCAL_2 RUNNING 50% -> DS for Repository Volume detected -> Mgmt Network detected -> Building deployment configuration -> Deploying RP4VMs cluster ---> Transaction DEPLOY_3 RUNNING 0% ---> Transaction DEPLOY_3 RUNNING 52% ---> Transaction DEPLOY_3 RUNNING 52% ---> Transaction DEPLOY_3 RUNNING 52% ---> Transaction DEPLOY_3 RUNNING 53% ---> Transaction DEPLOY_3 RUNNING 53% ---> Transaction DEPLOY_3 RUNNING 84% ---> Transaction DEPLOY_3 RUNNING 84% ---> Transaction DEPLOY_3 RUNNING 84% ---> Transaction DEPLOY_3 RUNNING 84% -> Cluster deployed successfully -> Checking connectivity to cluster Mgmt IP ---> vRPA Cluster is reachable -> Checking connectivity to Plugin Server ---> Plugin Server is reachable -> Configuring Plugin Server -> Plugin Server configured successfully -> Running connectivity checks to peer cluster ---> Transaction LOCAL_28 RUNNING 0% ---> Transaction LOCAL_28 RUNNING 14% ---> Transaction LOCAL_28 RUNNING 44% ---> Transaction LOCAL_28 RUNNING 48% ---> Transaction LOCAL_28 RUNNING 52% ---> Transaction LOCAL_28 RUNNING 84% ---> Connectivity checks to peer cluster completed successfully -> Connecting clusters ---> Transaction LOCAL_29 RUNNING 0% ---> Transaction LOCAL_29 RUNNING 51% ---> Transaction LOCAL_29 RUNNING 51% ---> Clusters Connect completed successfully -> All tasks completed successfully
Where can I find it?
The script and the config file can be found at GitHub:
https://github.com/IdanKen/Dell-EMC-RecoverPoint4VMs
How can I get help?
You are more than welcome to file an issue in GitHub or reach out to me via email:
Idan.kentor@dell.com
Thanks for reading!
Idan
Author: Idan Kentor
Related Blog Posts
RecoverPoint for VMs Automation – Advanced VM Protection
Wed, 21 Feb 2024 21:39:22 -0000
|Read Time: 0 minutes
In the spirit of automating everything, this blog will discuss a new automation solution in the RecoverPoint for VMs (RP4VMs) collection of automation solutions.
We have a variety of automation solutions for RP4VMs, including per-tag and per-cluster VM protection and use-case driven tasks, as well as a complete deployment automation solution. Now, I would like to present a new automation solution – Advanced VM Protection.
Let’s take a closer look at this exciting new solution.
What does the solution do?
The RecoverPoint for VMs advanced VM protection solution automates VM protection in RP4VMs with a wide variety of options:
- Automates VM protection based on pre-defined parameters in a JSON configuration file:
- VM name
- RP4VMs cluster name
- Plugin server IP or FQDN
- vCenter user/password or path to credentials file
- Production journal capacity (GB)
- Replica journal capacity (GB)
- Required RPO (sec)
- Failover networks per vNIC
- Performs and monitors mass VM protection
- Protects VMs for a specific RP4VMs cluster (optional)
- Performs VM protection operations on a specific plugin server (optional)
- Includes an option to skip the monitoring of VM protection preparation tasks
- Configures failover networks on a per network adapter basis as a post-protection operation
What is the solution?
It is a Python-based script that exclusively leverages the RP4VMs REST API.
Here is the list of prerequisites:
- Python 3.x (The script supports every platform Python is supported on)
- Python requests module, which can be installed using pip with the command:
pip install requests or python -m pip install requests
- RP4VMs 5.3.x and later
- Connectivity from the host running the script to the RP4VMs plugin server(s), specifically on tcp port 443
How do I use the script?
The script accepts the following parameters:
- One mandatory parameter, file, for a full path to the JSON configuration file.
- The optional parameters, rpvmcluster and server, limit script execution only for VM protection on a specified RP4VMs cluster and/or plugin server accordingly.
- The no-monitor parameter skips monitors of VM protection preparation task.
Here is the full script syntax:
# python advprotectvm.py -h usage: advprotectvm.py [-h] -file CONFIG_FILE [-cl RPVM_CLUSTER] [-s SERVER] [-nmonitor] Scripts advanced VM Protection in RecoverPoint for VMs options: -h, --help show this help message and exit -file CONFIG_FILE, --vm-config-file CONFIG_FILE Path to VM config file -cl RPVM_CLUSTER, --rpvmcluster RPVM_CLUSTER Optionally specify the RP4VMs cluster -s SERVER, --server SERVER Optionally specify RP4VMs Plugin Server DNS/IP -nmonitor, --no-monitor Optionally prevents protection monitoring
Use Cases and Examples
Let’s look at some common use cases for RP4VMs automated advanced VM protection:
- RP4VMs mass VM protection for onboarding of a new application:
# python advprotectvm.py -file idan-vms.json
- Batch VM protection only for a specific RP4VMs cluster:
# python advprotectvm.py -file idan-vms.json -cl Tel-Aviv
- Mass VM protection for a specific vCenter/ plugin or onboarding of a new datacenter:
# python advprotectvm.py -file vms.json -s pluginserver.idan.dell.com
Script output
# python advprotectvm.py -file vms.json -> Protecting VM prodwebsrv1 ---> Protection of VM prodwebsrv1 initiated -> Protecting VM prodappsrv1 ---> Protection of VM prodappsrv1 initiated -> Protecting VM proddbsrv1 ---> Protection of VM proddbsrv1 initiated -> VM protection initiated, monitoring ---> Protection of VM: prodwebsrv1, Transaction: d6783e2d-55be-47db-a082-de1d251c2375, Status: RUNNING ---> Protection of VM: prodappsrv1, Transaction: 808ab022-e79a-4ad1-a633-cc86e17644f2, Status: RUNNING ---> Protection of VM: proddbsrv1, Transaction: c7895dce-f3e6-4e70-872e-9d0b104d6273, Status: RUNNING ---> Protection of VM: prodwebsrv1, Transaction: d6783e2d-55be-47db-a082-de1d251c2375, Status: RUNNING ---> Protection of VM: prodappsrv1, Transaction: 808ab022-e79a-4ad1-a633-cc86e17644f2, Status: RUNNING ---> Protection of VM: proddbsrv1, Transaction: c7895dce-f3e6-4e70-872e-9d0b104d6273, Status: RUNNING ---> Protection of VM: prodwebsrv1, Transaction: d6783e2d-55be-47db-a082-de1d251c2375, Status: RUNNING ---> Protection of VM: prodappsrv1, Transaction: 808ab022-e79a-4ad1-a633-cc86e17644f2, Status: RUNNING ---> Protection of VM: proddbsrv1, Transaction: c7895dce-f3e6-4e70-872e-9d0b104d6273, Status: RUNNING ---> Protection of VM: prodwebsrv1, Transaction: d6783e2d-55be-47db-a082-de1d251c2375, Status: COMPLETED ---> Protection of VM: prodappsrv1, Transaction: 808ab022-e79a-4ad1-a633-cc86e17644f2, Status: COMPLETED ---> Protection of VM: proddbsrv1, Transaction: c7895dce-f3e6-4e70-872e-9d0b104d6273, Status: COMPLETED -> Configuring failover networks ---> Skipping failover network config for VM: prodwebsrv1 ---> Failover networks config is not required for VM: prodappsrv1 ---> Failover network config is successful for VM: proddbsrv1
Where can I find it?
The script and the config file can be found on GitHub: https://github.com/IdanKen/Dell-EMC-RecoverPoint4VMs.
Resources
- The Dell developer site provides comprehensive online API documentation, including full API references, tutorials, and use cases for the RP4VMs REST API.
- The RP4VMs REST API offers self-documentation – Swagger UI running on the plugin server itself – https://{plugin-server}/ui
- RecoverPoint for VMs GitHub repository
- RecoverPoint for VMs 5.3 – New RESTful API Demo
How can I get help?
For additional support, you are more than welcome to raise an issue in GitHub or reach out to me by email: Idan.kentor@dell.com
Thanks for reading!
Idan
Author: Idan Kentor
PowerProtect Data Manager Automation – Lifecycle Management
Mon, 20 Nov 2023 15:42:26 -0000
|Read Time: 0 minutes
In this installment of the PowerProtect Data Manager (PPDM) automation series of blogs, we will focus on a new solution that automates PowerProtect Data Manager life cycle management.
For PPDM automation, we have auto-policy creation and ad-hoc VM backup solutions, use-case driven tasks, complete PPDM deployment automation, and so on - all available in the official PowerProtect Data Manager GitHub repository. And now, I am proud to present to you the PPDM life cycle automation solution.
So, let’s take a closer look at the solution.
What does the solution do?
The PowerProtect Data Manager automated life cycle management automates PPDM upgrades with a wide variety of options:
- Upgrade with an upload package
- Performs pre-upgrade checks
- Continuously monitors pre-upgrade checks and upgrade processes
- Includes an option to skip the upload of the upgrade package and use the existing upgrade package
- Can perform only the pre-checks, without performing the actual upgrade
- Supports monitoring only phase where a current upgrade can be monitored
- Allows to skip PPDM VM snapshot. In any case, that snapshot will not be taken if a hosting vCenter is not configured
What is the solution?
It is a Python-based script that operates with the PPDM REST API.
Here is the list of prerequisites:
- Python 3.x (The script supports every platform Python is supported on)
- Python requests module, which can be installed using pip with the command: “pip install requests” or “python -m pip install requests”
- PowerProtect Data Manager 19.13 and later
- Connectivity from the host running the script to PPDM, specifically on tcp ports 8443 and 14443
How do I use the script?
The script accepts the following parameters:
- The PPDM host is represented by the mandatory parameter server and username (defaults to user admin) and a mandatory password parameter
- The parameter file for a full path to the upgrade package or, alternatively the skipupload parameter with the release parameter to determine the PPDM version to be applied
- The parameter onlyprecheck allows to perform precheck and exit without performing the actual upgrade
- Specify skipsnapshot to prevent VM snapshot from being taken on the PPDM VM
- The onlymonitor parameter performs monitoring of active upgrades
Here is the full script syntax:
# python ppdm_upgrade.py -h usage: ppdm_upgrade.py [-h] -s SERVER [-u USERNAME] -p PASSWORD [-f UPGFILE] [-onlyprecheck] [-skipupload] [-release PPDMRELEASE] [-skipsnapshot] [-onlymonitor]
Script to automate PowerProtect Data Manager lifecycle management
options: -h, --help show this help message and exit -s SERVER, --server SERVER PPDM server FQDN or IP -u USERNAME, --username USERNAME Optionally provide the PPDM username -p PASSWORD, --password PASSWORD PPDM password -f UPGFILE, --file UPGFILE Full path to upgrade package -onlyprecheck, --only-pre-check Optionally stops after pre-check -skipupload, --skip-file-upload Optionally skips file upload -release PPDMRELEASE, --ppdm-release PPDMRELEASE Provide PPDM version if skipping package upload -skipsnapshot, --skip-snapshot Optionally skips PPDM VM snapshot -onlymonitor, --only-monitor Optionally only monitor running upgrade |
Use Cases and Examples
Let’s look at some common use cases for automated PPDM life cycle management:
1. PPDM automated upgrade, including file upload:
# python ppdm_upgrade.py -s 10.0.0.1 -p "myTempPwd!" -f /home/idan/dellemc-ppdm-upgrade-sw-19.14.0-27.pkg
2. For cases where there is a need to prepare for an upgrade by uploading the package and run the precheck. It is possible to perform the automated upgrade in two phases.
a. First, only file upload and precheck:
# python ppdm_upgrade.py -s 10.0.0.1 -p "myTempPwd!" -f /home/idan/dellemc-ppdm-upgrade-sw-19.14.0-27.pkg -onlyprecheck
b. Second, perform the upgrade itself:
# python ppdm_upgrade.py -s 10.0.0.1 -p "myTempPwd!" -skipupload -release 19.14.0-27
3. Monitoring a running upgrade from a different workstation:
# python ppdm_upgrade.py -s 10.0.0.1 -p "myTempPwd!" -onlymonitor
Script output
# python ppdm_upgrade.py -s 10.0.0.1 -p "myTempPwd!" -f /home/idan/dellemc-ppdm-upgrade-sw-19.14.0-27.pkg -> Obtaining PPDM configuration information ---> PPDM is upgrade ready -> Performing pre-upgrade version checks ---> Current PPDM version: 19.13.0-20 ---> Checking upgrade to PPDM version: 19.14.0-27 -> Uploading PPDM upgrade package ---> Upload completed successfully in 3 mins and 34 secs -> Monitoring upgrade ID 636cc6a3-2e84-4fb8-bb40-87aefd0f7b96 ---> Monitoring state AVAILABLE -> Performing pre-upgrade checks -> Monitoring upgrade ID 636cc6a3-2e84-4fb8-bb40-87aefd0f7b96 ---> Monitoring state PROCESSING ---> Monitoring state PROCESSING ---> Monitoring state PROCESSING ---> Monitoring state AVAILABLE -> Upgrading PPDM to release 19.14.0-27 ---> Monitoring PPDM upgrade ---> Upgrade status: PENDING ---> Upgrade status: RUNNING 3% ----> Upgrade info: current component: eCDM, description: Taking Update Snapshot 3% ----> Upgrade info: seconds elapsed / remaining: 11 / 2477 ---> Upgrade status: RUNNING 3% ----> Upgrade info: current component: eCDM, description: Taking Update Snapshot 3% ----> Upgrade info: seconds elapsed / remaining: 41 / 2447 ---> Upgrade status: RUNNING 3% ----> Upgrade info: current component: eCDM, description: Taking Update Snapshot 3% ----> Upgrade info: seconds elapsed / remaining: 53 / 2435 ---> Upgrade status: RUNNING 3% ----> Upgrade info: current component: eCDM, description: Taking Update Snapshot 3% ----> Upgrade info: seconds elapsed / remaining: 64 / 2424 ---> Upgrade status: RUNNING 10% ----> Upgrade info: current component: eCDM, description: Shutting Down Components 10% ----> Upgrade info: seconds elapsed / remaining: 211 / 2004 ---> Upgrade status: RUNNING 10% ----> Upgrade info: current component: eCDM, description: Shutting Down Components 10% ----> Upgrade info: seconds elapsed / remaining: 222 / 1993 ----> Upgrade info: seconds elapsed / remaining: 266 / 1949 ---> Upgrade status: RUNNING 21% ----> Upgrade info: current component: eCDM, description: Updating The RPMs 21% ----> Upgrade info: seconds elapsed / remaining: 277 / 1720 ---> Upgrade status: RUNNING 21% ----> Upgrade info: current component: eCDM, description: Updating The RPMs 21% ----> Upgrade info: seconds elapsed / remaining: 288 / 1709 ---> Upgrade status: RUNNING 21% ----> Upgrade info: current component: eCDM, description: Updating The RPMs 21% ----> Upgrade info: seconds elapsed / remaining: 299 / 1698 ---> Upgrade status: RUNNING 21% ----> Upgrade info: current component: eCDM, description: Updating The RPMs 21% ----> Upgrade info: seconds elapsed / remaining: 563 / 1486 ---> Polling timed out, retrying... ---> Polling timed out, retrying... ---> Polling timed out, retrying... ---> Upgrade status: RUNNING 32% ----> Upgrade info: current component: eCDM, description: Starting Components 32% ----> Upgrade info: seconds elapsed / remaining: 688 / 1167 ---> Upgrade status: RUNNING 35% ----> Upgrade info: current component: eCDM, description: Migrating Data 50% ----> Upgrade info: seconds elapsed / remaining: 776 / 1038 ---> Upgrade status: RUNNING 52% ----> Upgrade info: current component: eCDM, description: Data Migration Completed 52% ----> Upgrade info: seconds elapsed / remaining: 787 / 1038 ---> Upgrade status: RUNNING 55% ----> Upgrade info: current component: eCDM, description: Data Migration Completed 55% ----> Upgrade info: seconds elapsed / remaining: 940 / 1038 ---> Upgrade status: RUNNING 55% ----> Upgrade info: current component: eCDM, description: Data Migration Completed 55% ----> Upgrade info: seconds elapsed / remaining: 951 / 1038 ---> Upgrade status: RUNNING 57% ----> Upgrade info: current component: eCDM, description: Data Migration Completed 57% ----> Upgrade info: seconds elapsed / remaining: 962 / 1038 ---> Upgrade status: RUNNING 66% ----> Upgrade info: current component: eCDM, description: Data Manager Core Services Update Completed. Waiting for Other Components to Update... 71% ----> Upgrade info: seconds elapsed / remaining: 1027 / 1038 ---> Upgrade status: RUNNING 71% ----> Upgrade info: current component: eCDM, description: Data Manager Core Services Update Completed. Waiting for Other Components to Update... 71% ----> Upgrade info: seconds elapsed / remaining: 1038 / 1038 ---> Upgrade status: RUNNING 71% ----> Upgrade info: current component: eCDM, description: Data Manager Core Services Update Completed. Waiting for Other Components to Update... 71% ----> Upgrade info: seconds elapsed / remaining: 1049 / 1038 ---> Upgrade status: RUNNING 71% ----> Upgrade info: current component: eCDM, description: Data Manager Core Services Update Completed. Waiting for Other Components to Update... 71% ----> Upgrade info: seconds elapsed / remaining: 1060 / 1038 ---> Upgrade status: RUNNING 71% ----> Upgrade info: current component: eCDM, description: Data Manager Core Services Update Completed. Waiting for Other Components to Update... 71% ----> Upgrade info: seconds elapsed / remaining: 1071 / 1038 ---> Upgrade status: RUNNING 71% ----> Upgrade info: current component: eCDM, description: Data Manager Core Services Update Completed. Waiting for Other Components to Update... 71% ----> Upgrade info: seconds elapsed / remaining: 1083 / 1038 ---> Upgrade status: RUNNING 71% ----> Upgrade info: current component: eCDM, description: Data Manager Core Services Update Completed. Waiting for Other Components to Update... 71% ----> Upgrade info: seconds elapsed / remaining: 1095 / 1038 ---> Upgrade status: RUNNING 71% ----> Upgrade info: current component: eCDM, description: Data Manager Core Services Update Completed. Waiting for Other Components to Update... 71% ----> Upgrade info: seconds elapsed / remaining: 1108 / 1038 ---> Upgrade status: RUNNING 71% ----> Upgrade info: current component: eCDM, description: Data Manager Core Services Update Completed. Waiting for Other Components to Update... 71% ----> Upgrade info: seconds elapsed / remaining: 1120 / 1038 ---> Upgrade status: RUNNING 71% ----> Upgrade info: seconds elapsed / remaining: 1350 / 1038 ---> Upgrade status: RUNNING 76% ----> Upgrade info: current component: TSDM, description: Transparent Snapshot Data Mover Update Started 76% ----> Upgrade info: seconds elapsed / remaining: 1361 / 1034 ---> Upgrade status: RUNNING 94% ----> Upgrade info: current component: TSDM, description: Transparent Snapshot Data Mover Update Completed 94% ----> Upgrade info: seconds elapsed / remaining: 1372 / 138 ---> Upgrade status: COMPLETED 100% ----> Upgrade completed in 22 mins and 56 seconds -> PPDM upgraded successfully -> Making sure PPDM is up and running ---> PPDM is available ---> PPDM is operational on version 19.14.0-27 -> All tasks completed successfully |
Where can I find it?
You can find the script in the official PowerProtect GitHub repository:
https://github.com/dell/powerprotect-data-manager
Resources
Other than the official PPDM repo on GitHub, developer.dell.com provides comprehensive online API documentation, including full API reference, tutorials, and use cases for PPDM REST API.
How can I get help?
For additional support, you are more than welcome to raise an issue in GitHub or reach out to me by email:
Idan.kentor@dell.com
Thanks for reading!
Idan