Exception handling for create_snapshot and remove_image methods

docs/Beginners_guide_docs/BMI_image_creation.md

@@ -0,0 +1,47 @@
+
+Generally BMI images are centos, we may be requested for debian based images (Ubuntu) at which point we will need to create BMI ubuntu image as follows.
+
+1. Download latest requested (Ubuntu) cloud image or server image.
+
+2. Install dracut 
+```
+      sudo apt install dracut-core
+```
+
+3. Modify dracut.conf to ensure hostonly=“no” . This ensures that the image is generic, if not the image created would contain only dracut modules/filesystems which are needed to boot the current machine. Dracut configuration file could be found in below path in ubuntu case.
+
+```
+ /etc/dracut.conf.d/10-debian.conf
+ ```
+
+ 4. If cloud image was taken, disable cloud.init as follows (this ensures cloud based customizations that take longer boot time are disabled), if not skip this step
+
+ ```
+
+ 		echo 'datasource_list: [ None ]' | sudo -s tee /etc/cloud/cloud.cfg.d/90_dpkg.cfg
+		sudo apt-get purge cloud-init
+		sudo rm -rf /etc/cloud/; sudo rm -rf /var/lib/cloud/
+		reboot
+
+```
+
+or 
+
+```
+		touch /etc/cloud/cloud-init.disabled
+```
+
+5. Create image by following command
+
+```
+		Dracut —force
+```
+
+6. Next add Intel 10 Gig NIC. The following link contains the good information for the same (ubuntu).
+
+```
+		http://ask.xmodulo.com/download-install-ixgbe-driver-ubuntu-debian.html
+```
+
+
+

docs/Beginners_guide_docs/BMI_installation.md

@@ -0,0 +1,173 @@
+
+**Prerequisties**<br />
+Ceph (cephargs, ceph.conf, client.bmi.key) and HIL project and network information is already available.
+
+**Description** <br />
+Installing BMI on a machine would mean to enable a machine to provide BMI services in conjunction with Ceph and HIL (these are already configued and available as mentioned in pre-requisites). To learn more about BMI architecture and services visit https://github.com/CCI-MOC/ims/blob/master/README.md. To install BMI on a machine ansible software/ansible-playbook script is used. The script installs all the necessary software’s (on the given machine only) to orchestrate the BMI services in conjunction with Ceph and HIL. The following modifications need to be taken care of before ansible script execution. To know more about Ansible scripting/template visit http://docs.ansible.com/. 
+
+**Pre-step**<br />
+Modify ims/constants/common.py file so that the BMI_ADMIN_PROJECT is set to your project (HIL project) to when bootstrapping the database. Perform the following as sudo su.
+
+1.    Get the latest Ansible playbook scripts repository from Git. The general repository is ~/ims/scripts/install/production/
+
+ ``` 
+	     git clone https://github.com/CCI-MOC/ims.git 
+
+	     Eg: The latest scripts at the time of documentation was PR 153.
+	     git fetch origin pull/153/head:pr-153
+	     git checkout pr-153
+```
+
+2.    Install Ansible 
+
+  - For Ubuntu:
+```
+      	sudo apt-get update
+      	sudo apt-get install software-properties-common
+      	sudo apt-add-repository ppa:ansible/ansible
+      	sudo apt-get update
+      	sudo apt-get install ansible
+```
+  - For Centos/RHEL:
+```
+      	sudo yum install ansible
+```
+
+3.    Add your host to the ansible hosts file (/etc/ansible/hosts), i.e:
+
+```
+	# Ex 1: Ungrouped hosts, specify before any group headers.   
+	192.168.122.76
+
+	Since in this case, we are installing on the local machine itself, the following could be done:
+
+	# Ex 1: Ungrouped hosts, specify before any group headers.   
+	localhost ansible_connection=local
+```
+
+4.    Modify the following sections in bmi_config.cfg under ~/ims   
+
+   -    To have a new UID as follows:
+```	
+	[bmi]
+	uid = <unique number>
+	service = <true or false>
+
+	EG:    
+	[bmi]
+        uid = vj-test-
+        service = true
+```
+
+  -    To match your HIL setup as follows:
+
+```
+	[fs]
+	id = <id in ceph>
+	pool = <the ceph pool to use>
+	conf_file = <location of ceph config file
+	keyring = <location of ceph key ring>
+
+	EG: 
+	id = bmi
+	pool = bmi
+	conf_file = /etc/ceph/ceph.conf
+	keyring = /etc/ceph/client.bmi.key
+	(Note: Modify the id, pool name and paths accordingly.)
+```
+
+  -    To match your Ceph setup as follows:
+
+```
+	# This section is for network isolator (HIL) related config
+	[net_isolator]
+	url = <base url for hil>
+
+	EG:
+	[net_isolator]
+	url = http://192.168.100.210:80/
+	(Note: Modify the url according to your HIL endpoint accordingly)
+```
+
+5.    Modify dnsmasq.conf within roles/dhcp/tasks/main.yml. The DHCP range and interface needs to be changed accordingly to match your requirements (below is an example).
+
+```
+	- name: Add DHCP configuration to dnsmasq.conf
+		lineinfile:
+          path: /etc/dnsmasq.conf
+	  line: "{{ item }}"
+          become: true
+          with_items:
+	      - 'interface=eth2'
+	      - 'dhcp-range=10.10.10.50,10.10.10.100,7d'
+	      - 'dhcp-boot=pxelinux.0'
+	      - 'enable-tftp'
+	      - 'tftp-root=/var/lib/tftpboot'
+	      - 'dhcp-userclass=set:ENH,iPXE'
+```
+
+Note : Also configure the interface (eth2 in the above ex) to have a gateway IP related to the range of IPs chosen above. Considering the above case, ifcfg-eth2 could have static IP of say 10.10.10.1.
+
+
+6.    In roles/bmi/tasks/main.yml:
+
+  -   Modify Ceph and HIL credentials in to the the correct username and password for your configuration. This includes the  CEPH_ARGS and HIL_ENDPOINT (below is an example).
+
+```
+	- name: Add Ceph and HIL credentials to bashrc
+		lineinfile:
+	  path: ~/.bashrc
+	  line: "{{ item }}"
+	  become: true
+	  with_items:
+	      - 'export CEPH_ARGS="--keyring /etc/ceph/client.bmi.key --id bmi --pool bmi"'
+	      - 'export HIL_USERNAME=vj'
+	      - 'export HIL_PASSWORD=redhat'
+	      - 'export HIL_ENDPOINT="http://192.168.100.210:80"'
+	      - 'export BMI_CONFIG=/etc/bmi/bmiconfig.cfg'
+
+```
+
+  -   Modify the project project and network from 'bmi_infra' and 'bmi_network' to the project and network you created within HIL (below is an example).
+
+```
+	- name: Bootstrap the database
+          command: "{{ item }}"
+	  environment:
+	      HIL_USERNAME: vj
+	      HIL_PASSWORD: redhat
+	   with_items:
+	       - bmi db ls
+	       - sqlite3 /etc/bmi/bmi.db "insert into project values (1, 'vj', 'bmi-provision-vj')"
+	         when: db.stat.size == 0
+```
+
+7.    Comment out any of the roles that you don’t want to execute in site.yml.
+
+8.    Run ansible-playbook site.yml.
+
+ ```    
+       ansible-playbook site.yml
+```
+
+9.   The install playbook modifies ~/.bashrc. Make sure to refresh your shell after it is run.
+
+
+Possible errors one may face:
+------------------------------
+
+```
+fatal: [localhost]: FAILED! => {"changed": false, "cmd": "/usr/bin/git clone --origin origin https://github.com/fujita/tgt /home/vj/ims/scripts/install/production/tgt", "msg": "fatal: could not create work tree dir '/home/vj/ims/scripts/install/production/tgt'.: Permission denied", "rc": 128, "stderr": "fatal: could not create work tree dir '/home/vj/ims/scripts/install/production/tgt'.: Permission denied\n", "stderr_lines": ["fatal: could not create work tree dir '/home/vj/ims/scripts/install/production/tgt'.: Permission denied"], "stdout": "", "stdout_lines": []}[WARNING]: Could not create retry file '/home/vj/ims/scripts/install/production/site.retry'.         [Errno 13] Permission denied: u'/home/vj/ims/scripts/install/production/site.retry'
+
+```
+
+Possible reasons: Not executing as sudo.
+
+
+```
+Error 4 \nwarning: failed to load external entity \"http://docbook.sourceforge.net/release/xsl/current/manpages/docbook.xsl\" 
+```
+
+Possible reasons: The sourceforge site could be offline or down. Could try again after a while.
+(Note: This site is frequently known to be down!. Could think about having these docs stored locally.)
+

docs/Beginners_guide_docs/BMI_node_provisioning.md

@@ -0,0 +1,59 @@
+The execution involves the following steps. Please note all the steps below are to be executed as root user.
+
+1. Start the Picasso (frontend) server as follows:
+
+```
+	>> screen 
+	>> picasso_server
+
+	If successful it should show something like below
+	* Running on http://10.10.10.1:1513/ (Press CTRL+C to quit)
+
+```
+2. Start the Einstein (backend) server as follows:
+```
+	>> screen
+	>> einstein_server
+
+	If successful it should show something like below
+	Broadcast server running on 0.0.0.0:9091
+	NS running on 10.10.10.1:9893 (10.10.10.1)
+	Warning: HMAC key not set. Anyone can connect to this server!
+	URI = PYRO:[email protected]:9893
+```
+
+Note : Few errors one could face at this phase<br />
+ - Address already in use : socket error -> This is a well known socket error, killing the process and restarting should resolve it.
+
+```
+ 		 ps aux | grep picasso/einstein
+ 		 kill -9 <process number>
+```
+ - Permission denied : Probably not executing it as root user.
+
+
+3. bmi --help will list down all the bmi commands
+
+4. To provision a node the command is as follows
+
+```
+		Usage: bmi pro [OPTIONS] PROJECT NODE IMG NETWORK NIC
+
+		EG: bmi pro vj dell-8 centos_67_e1 bmi-provision-vj em1
+```
+
+
+Possible errors one may face:
+------------------------------
+
+Enable debug and verbose in bmi_config.cfg. The logs can be found under path "var/log/bmi/"
+
+1. Connection reject :
+	Better to try again.
+
+2. Got status code 409 from HIL with message :
+	Node not in project : The node assigned to be provisioned has been freed and is no longer part of the (your) project. Need to reacquire it. The link https://github.com/CCI-MOC/kumo-leasing/blob/master/instructions.md has all the necessary information for the same.
+
+3. Couldn't connect to HIL : 
+	Check logs for reason for HIL connection exception. Test HIL_ENDPOINT connection through ping and curl. If thats fine check further from logs to see where the exception occurred. Some common mistakes include additional or wrong quotes in url in bmi_config.cfg, which result in request.gets error.
+

ims/common/constants.py

@@ -1,4 +1,6 @@
 # Config Flags
+CLONE = 1
+
 PICASSO_CONFIG_FLAG = 'picasso'
 EINSTEIN_CONFIG_FLAG = 'einstein'
 

ims/einstein/ceph.py

@@ -102,8 +102,9 @@ def create_image(self, img_id, img_size):
             raise file_system_exceptions.FunctionNotSupportedException()
 
     @log
-    def clone(self, parent_img_name, parent_snap_name, clone_img_name):
+    def clone(self, parent_img_name, parent_snap_name, clone_img_name, flag=None):
         try:
+            flag[0] = constants.CLONE
             parent_context = child_context = self.context
             self.rbd.clone(parent_context, parent_img_name, parent_snap_name,
                            child_context, clone_img_name, features=1)

ims/einstein/operations.py

@@ -332,6 +332,7 @@ def deprovision(self, node_name, network, nic):
     @log
     def create_snapshot(self, node_name, snap_name):
         try:
+            error_flag = [0]
             self.hil.validate_project(self.proj)
 
             ceph_img_name = self.__get_ceph_image_name(node_name)
@@ -344,7 +345,7 @@ def create_snapshot(self, node_name, snap_name):
                                  is_snapshot=True)
             snap_ceph_name = self.__get_ceph_image_name(snap_name)
             self.fs.clone(ceph_img_name, constants.DEFAULT_SNAPSHOT_NAME,
-                          snap_ceph_name)
+                          snap_ceph_name, flag=error_flag)
             self.fs.flatten(snap_ceph_name)
             self.fs.snap_image(snap_ceph_name, constants.DEFAULT_SNAPSHOT_NAME)
             self.fs.snap_protect(snap_ceph_name,
@@ -355,8 +356,38 @@ def create_snapshot(self, node_name, snap_name):
                                     constants.DEFAULT_SNAPSHOT_NAME)
             return self.__return_success(True)
 
-        except (HILException, DBException, FileSystemException) as e:
+        except HILException as e:
+            # Nothing more to do, if HIL validation fails
+            logger.exception('')
+            return self.__return_error(e)
+
+        except DBException as e:
+            # Failure has occured at db insert, unprotect and remove snapshot of ceph img
             logger.exception('')
+            self.fs.snap_unprotect(ceph_img_name,
+                                   constants.DEFAULT_SNAPSHOT_NAME)
+            self.fs.remove_snapshot(ceph_img_name,
+                                    constants.DEFAULT_SNAPSHOT_NAME)
+            time.sleep(constants.HIL_CALL_TIMEOUT)
+            return self.__return_error(e)
+
+        except FileSystemException as e:
+            logger.exception('')
+            # If file system exception occurs:
+
+            # 1. During initial snap creation or snap protect of ceph, then nothing more to do.
+
+            # 2. During clone or beyond such as flattening, snap image or protect of snap ceph image,
+            #    then undo cloning, undo db insert for the given snap name, unprotect and remove
+            #    snapshot of ceph img.
+            if error_flag == constants.CLONE:
+                self.fs.remove(snap_ceph_name)
+                self.db.image.delete_with_name_from_project(snap_name, self.proj)
+                self.fs.snap_unprotect(ceph_img_name,
+                                       constants.DEFAULT_SNAPSHOT_NAME)
+                self.fs.remove_snapshot(ceph_img_name,
+                                        constants.DEFAULT_SNAPSHOT_NAME)
+            time.sleep(constants.HIL_CALL_TIMEOUT)
             return self.__return_error(e)
 
     # Lists snapshot for the given image img_name
@@ -388,7 +419,25 @@ def remove_image(self, img_name):
             self.fs.remove(ceph_img_name)
             self.db.image.delete_with_name_from_project(img_name, self.proj)
             return self.__return_success(True)
-        except (HILException, DBException, FileSystemException) as e:
+
+        except HILException as e:
+            # Nothing more to do, if HIL validation fails
+            logger.exception('')
+            return self.__return_error(e)
+
+        except DBException as e:
+            # Failure has occured at db delete, undo unprotect and remove snapshot of ceph img
+            logger.exception('')
+            self.fs.clone(ceph_img_name)
+            self.fs.snap_image(ceph_img_name,
+                               constants.DEFAULT_SNAPSHOT_NAME)
+            self.fs.snap_protect(ceph_img_name,
+                                 constants.DEFAULT_SNAPSHOT_NAME)
+            time.sleep(constants.HIL_CALL_TIMEOUT)
+            return self.__return_error(e)
+
+        except FileSystemException as e:
+            # Nothing more to do, if file system exception occurs
             logger.exception('')
             return self.__return_error(e)