Version 1.1.001 release docs

SathiyaCV · SathiyaCV · commit 2e8a47ef84c8 · 2020-05-21T09:47:07.000Z
diff --git a/README.md b/README.md
@@ -8,7 +8,8 @@
 
  - [Zipkin](https://zipkin.io/)
  - [Kafka](https://kafka.apache.org/)
- - [Elasticsearch](https://www.elastic.co/products/elasticsearch)
+ - [Elasticsearch](https://www.elastic.co/products/elasticsearch) 
+ - [Prometheus](https://prometheus.io/docs/introduction/overview/)
 
 ## Overview
 
@@ -26,6 +27,10 @@ Currently, you can monitor performance at the application level using Citrix ADC
 
 Citrix Observability Exporter supports collecting transactions and streaming them to endpoints. Currently, Citrix Observability Exporter supports Elasticsearch and Kafka as transaction endpoints.
 
+### Time series data support
+
+Citrix Observability Exporter supports collecting time series data (metrics) from Citrix ADC instances and exports them to Prometheus. Prometheus is a monitoring solution for storing time series data like metrics. You can then add Prometheus as a data source to Grafana and graphically view the Citrix ADC metrics and analyze the metrics.
+
 ## How does Citrix Observability Exporter work
 
 ### Distributed tracing with Zipkin using Citrix Observability Exporter
@@ -42,14 +47,19 @@ When Elasticsearch is specified as the transaction endpoint, Citrix Observabilit
 
 When Kafka is specified as the transaction endpoint, Citrix Observability Exporter converts the transaction data to [Avro](http://avro.apache.org/docs/current/Avro) format and streams them to Kafka.
 
+### Citrix Observability Exporter with Prometheus as the endpoint for time series data
+
+When Prometheus is specified as the format for time series data, Citrix Observability Exporter collects various metrics from Citrix ADCs and converts them to appropriate Prometheus format and exports them to the Prometheus server. These metrics include counters of the virtual servers, services to which the analytics profile is bound and global counters of HTTP, TCP and so on.
+
 ## Deployment
 
-You can deploy Citrix Observability Exporter using Kubernetes YAML. To deploy Citrix Observability Exporter using Kubernetes YAML, see [Deployment](deployment/README.md).
+You can deploy Citrix Observability Exporter using Kubernetes YAML or Helm charts. To deploy Citrix Observability Exporter using Kubernetes YAML, see [Deployment](deployment/README.md).
 
 ## Questions
 
 For questions and support, the following channels are available:
 
+
 -  [Citrix Discussion Forum](https://discussions.citrix.com/)
 -  [Citrix ADC Cloud Native Slack Channel](https://citrixadccloudnative.slack.com/)
   
diff --git a/custom-header/README.md b/custom-header/README.md
@@ -0,0 +1,27 @@
+# Custom header logging
+
+This feature enables logging of all HTTP headers of a transaction. As part of this feature, a new option called `allHttpHeaders` is introduced under the Web Insight analytics profile. When this option is set, Citrix ADC uploads all the request and response headers under `httpAllReqHdrs` and `httpAllResHdrs` respectively. This option is disabled by default. Custom header logging is currently supported only for the Kafka endpoint.
+
+Now, Citrix Observability Exporter transaction records contain two additional fields:
+
+- `httpAllReqHdrs`: This field is a string containing all the request header lines separated by \r\n.
+- `httpAllResHdrs`: This field is a string containing all the response header lines separated by \r\n.
+
+## Citrix ADC configuration to enable custom header logging
+
+### Using the Citrix ADC command line
+
+Use the following command from the Citrix ADC command line.
+
+    set analytics profile ns_analytics_default_http_profile -allHttpHeaders  ENABLED
+
+### Using the Citrix ingress controller
+
+To enable `allHttpHeaders` using the Citrix ingress controller, you should set the analytics profile as annotations on the Ingress which is applied on the Citrix ADC.
+
+
+    ingress.citrix.com/analyticsprofile: '{"webinsight": {"httpurl":"ENABLED", "httpuseragent":"ENABLED", "httphost":"ENABLED", “allhttpheaders”:”ENABLED”, "httpmethod":"ENABLED", "httpcontenttype":"ENABLED"}, "tcpinsight": {"tcpBurstReporting":"DISABLED"}}'
+
+Following is a sample transaction record snapshot:
+
+    {"http_transid": "avro_HTTP_TF_0_c222660a_0_0_T_1577278789_1", "recType": "HTTP_A", "actualtemplatecode": 51, "httpReqMethod": "GET", "httpReqUrl": "/status/500", "httpReqUserAgent": "curl/7.58.0", "httpContentType": "", "httpReqHost": "10.102.34.201", "httpReqAuthorization": "", "httpReqCookie": "", "httpReqReferer": "", "httpResSetCookie": "", "httpAllReqHdrs": "Test: test500get\r\nAccept: */*\r\nHost: 10.102.34.201\r\nUser-Agent: curl/7.58.0\r\n", "httpAllResHdrs": "Connection: keep-alive\r\nContent-Length: 0\r\nContent-Type: text/html; charset=utf-8\r\nDate: Wed, 25 Dec 2019 13:16:39 GMT\r\nServer: gunicorn/19.9.0\r\nAccess-Control-Allow-Origin: *\r\nAccess-Control-Allow-Credentials: true\r\n", "icContGrpName": "", "icFlags": 0, "icNostoreFlags": 0, "icPolicyName": "", "responseMediaType": 0,  
diff --git a/deployment/README.md b/deployment/README.md
@@ -19,13 +19,14 @@ The following diagram shows a deployment of Citrix Observability Exporter with a
    ensure that you have the following docker images installed in the Kubernetes cluster:
     - [Zipkin](https://zipkin.io/)
     - (Optional) [Elasticsearch](https://www.elastic.co/products/elasticsearch) as back-end for Zipkin. Elasticsearch is required if you want to visualize your tracing data in [Kibana](https://www.elastic.co/products/kibana). You can also use Elasticsearch as an endpoint for transactions.
-    - (Optional) [Kibana](https://www.elastic.co/products/kibana) is required to visualize your tracing data.
+    - [Kibana](https://www.elastic.co/products/kibana) is required to visualize your tracing data.
 
 **Note:**
     You can use [zipkin.yaml](../examples/zipkin.yaml), [elasticsearch.yaml](../examples/elasticsearch.yaml), and [kibana.yaml](../examples/kibana.yaml) for installing Zipkin, Elasticsearch, and Kibana.
 
 - If Elasticsearch is used as the endpoint for transactions, ensure that you have Elasticsearch installed and configured.
 - If Kafka is used as the endpoint for transactions, ensure that Kafka server is installed and configured.
+- If Prometheus is used as the endpoint for time series data, ensure that Prometheus is installed and configured.
 
 ## Deploy Citrix Observability Exporter using YAML
 
@@ -38,6 +39,7 @@ To deploy Citrix Observability Exporter using Kubernetes YAML, perform the follo
 2. If you use Citrix ADC VPX or MPX in the deployment, create the necessary login credentials.
 
         kubectl create secret  generic nslogin --from-literal=username='nsroot' --from-literal=password='nsroot'
+
 3. Create a Kubernetes ConfigMap, Deployment, and Service with Log stream configuration for the required endpoint:
 
 - For Citrix Observability Exporter with Zipkin tracing support:
@@ -76,6 +78,19 @@ You can specify the tracing server in ConfigMap using environment variables in t
 
    Enable the Kafka endpoint by setting the value of `EnableKafka` as `yes`. Also, set Kafka broker details in `KafkaBroker` and topic details in `KafkaTopic`. You also must specify the Kafka cluster host IP mapping under HostAliases in the [Kubernetes Pod specification](https://kubernetes.io/docs/concepts/services-networking/add-entries-to-pod-etc-hosts-with-host-aliases/#adding-additional-entries-with-hostaliases).
 
+- For Citrix Observability Exporter with Prometheus as the endpoint for time series data:
+
+   You can enable Prometheus support by specifying the following annotations in the YAML files to deploy Zipkin, Kafka, or Elasticsearch and exposing the time series port.
+
+        prometheus.io/scrape: "true"
+        prometheus.io/port: "5563"
+
+   The following command deploys Citrix Observability Exporter with both Elasticsearch and Prometheus as endpoints, using the [coe-es-prometheus.yaml](coe-es-prometheus.yaml) file. In this YAML file, annotations for Prometheus support are enabled and port 5563 is exposed which is used for the time series data. 
+
+        kubectl create -f coe-es-prometheus.yaml
+
+   You should configure Prometheus to scrape the data from the Citrix Observability Exporter time series port. For enabling time series data processing, there is no specific configuration required on Citrix Observability Exporter. By default, if time series data is pushed to Citrix Observability Exporter, it is processed automatically. The time series port is enabled by default.
+
 **Note:**
    Once you deploy a Citrix Observability Exporter instance with a specific endpoint, you cannot modify it. For changing the endpoint, you must bring down the Citrix Observability Exporter instance and deploy it again with the new endpoint.
 
@@ -89,9 +104,6 @@ In this procedure, a Citrix ADC CPX is deployed with the Citrix ingress controll
 
 Depending on the endpoint you are using, you can choose the YAML file for deploying Citrix ADC CPX. These YAML files include the configuration required for Citrix Observability Exporter.
 
-**Note:**
-  Any usage of the environment variable ``NS_LOGPROXY`` in this procedure refers to ``Citrix Observability Exporter`` only.
-
 Perform the following steps to deploy a Citrix ADC CPX instance with Citrix Observability Exporter support enabled.
 
 1. Download the YAML file for deploying Citrix ADC CPX according to the endpoint.
@@ -100,34 +112,9 @@ Perform the following steps to deploy a Citrix ADC CPX instance with Citrix Obse
     - For Elasticsearch as the transaction endpoint: [cpx-ingress-es.yaml](../examples/elasticsearch/cpx-ingress-es.yaml)
     - For Kafka as the transaction endpoint: [cpx-ingress-kafka.yaml](../examples/kafka/cpx-ingress-kafka.yaml)
 
-2. Edit the YAML file and specify the environment variables in the Citrix ingress controller configuration according to the endpoint you are using:
-    - For tracing support with Zipkin:
-  
-                -name: "NS_LOGPROXY"
-                 value: "<abc.com>"
-                -name: "NS_DISTRIBUTED_TRACING"
-                 value: "yes"
-    - For Elasticsearch or Kafka as the transaction endpoint:
-
-                - name: "NS_LOGPROXY"
-                 value: "<abc.com>"
-        
-        **Note:**
-        Using [smart annotations](https://developer-docs.citrix.com/projects/citrix-k8s-ingress-controller/en/latest/configure/annotations/), you can define specific parameters you must import by specifying it in the YAML file for deploying Citrix ADC CPX.
-
-        For example:
-
-        
-                ingress.citrix.com/analyticsprofile: '{"webinsight": {"httpurl":"ENABLED", "httpuseragent":"ENABLED", "httphost":"ENABLED", "httpmethod":"ENABLED", "httpcontenttype":"ENABLED"}, "tcpinsight": {"tcpBurstReporting":"DISABLED"}}'
-  
-
-        **Note:**
-        You can also define the parameters to import using smart annotations for service. You can specify the parameters in the YAML for deploying Citrix Observability Exporter. However, you can use service annotations only when the service type is `LoadBalancer`.
+2. Create and deploy a ConfigMap with the required key-value pairs in the ConfigMap. You can use the [cic-configmap.yaml](../examples/cic-configmap.yaml) file.
 
-        For example:
-        
-                service.citrix.com/analyticsprofile: '{"<service name>":'{"webinsight": {"httpurl":"ENABLED", "httpuseragent":"ENABLED"}
-        
+             kubectl create -f cic-configmap.yaml
 
 3. Deploy Citrix ADC CPX with the Citrix ingress controller as a sidecar using the following command.
 
@@ -141,6 +128,22 @@ Perform the following steps to deploy a Citrix ADC CPX instance with Citrix Obse
 
                  kubectl create -f cpx-ingress-kafka.yaml 
 
+**Note:**
+  Using [smart annotations](https://developer-docs.citrix.com/projects/citrix-k8s-ingress-controller/en/latest/configure/annotations/), you can define specific parameters you must import by specifying it in the YAML file for deploying Citrix ADC CPX.
+
+  For example:
+
+        
+                ingress.citrix.com/analyticsprofile: '{"webinsight": {"httpurl":"ENABLED", "httpuseragent":"ENABLED", "httphost":"ENABLED", "httpmethod":"ENABLED", "httpcontenttype":"ENABLED"}, "tcpinsight": {"tcpBurstReporting":"DISABLED"}}'
+  
+
+**Note:**
+  You can also define the parameters to import using smart annotations for service. You can specify the parameters in the YAML for deploying Citrix Observability Exporter. However, you can use service annotations only when the service type is `LoadBalancer`.
+
+  For example:
+        
+                service.citrix.com/analyticsprofile: '{"<service name>":'{"webinsight": {"httpurl":"ENABLED", "httpuseragent":"ENABLED"}
+
 ### Deploy the Citrix ingress controller with Citrix Observability Exporter support for Citrix ADC MPX or VPX
 
  In this deployment, the Citrix ingress controller is deployed as a standalone pod in the Kubernetes cluster. It controls the Citrix ADC MPX or VPX appliance deployed outside the cluster. The Citrix Observability Exporter support is enabled in the Citrix ingress controller configuration.
@@ -150,24 +153,12 @@ Perform the following steps to deploy a Citrix ADC CPX instance with Citrix Obse
 You need to complete the [prerequisites](https://developer-docs.citrix.com/projects/citrix-k8s-ingress-controller/en/latest/deploy/deploy-cic-yaml/#prerequisites) for deploying the Citrix ingress controller as a standalone pod.
 
 1. Download the [vpx-ingress.yaml](../examples/vpx-ingress.yaml) file.
-2. Edit the `vpx-ingress.yaml` file and modify the values for the environmental variables as provided in [deploying the Citrix ingress controller](https://developer-docs.citrix.com/projects/citrix-k8s-ingress-controller/en/latest/deploy/deploy-cic-yaml/#deploy-citrix-ingress-controller-as-a-pod).
-
-3. Specify environment variables for Citrix Observability Exporter in the Citrix ingress controller configuration.
-
-   - For tracing support with Zipkin:
-  
 
-                - name: "NS_LOGPROXY"
-                  value: "<abc.com>"
-                - name: "NS_DISTRIBUTED_TRACING"
-                  value: "yes"
+1.  Create and deploy a ConfigMap with the required key-value pairs in the ConfigMap. You can use the [cic-configmap.yaml](../examples/cic-configmap.yaml) file.
 
-   - For ElasticSearch or Kafka as the transaction endpoint:
+             kubectl create -f cic-configmap.yaml
 
-                - name: "NS_LOGPROXY"
-                  value: "<abc.com>:5557"
-
-4. Once you update the environment variables, save the [vpx-ingress.yaml](../examples/vpx-ingress.yaml) file and deploy it using the following command.
+2. Deploy the [vpx-ingress.yaml](../examples/vpx-ingress.yaml) file using the following command.
 
         kubectl create -f vpx-ingress.yaml -n tracing
 
@@ -221,6 +212,7 @@ file. This sample web application is added as a service in the Ingress.
 1. Get `NodePort` information for `cpx-service` using the following command.
 
         kubectl describe service cpx-service
+
 2. Access `http://www.samplewebserver.com:NodePort` from a web browser to open the sample web application.
 
 3.  Send multiple requests to the application as shown in the following sample image.
@@ -230,7 +222,7 @@ file. This sample web application is added as a service in the Ingress.
     **Note:**
           You can generate different types of response status codes for different HTTP methods (for example, GET, POST, DELETE, and so on).
 
-1. All transactions are uploaded to the Elasticsearch server and you can view them using the Kibana dashboard. 
+1. All transactions are uploaded to the Elasticsearch server and you can view them using the Kibana dashboard.
 
    You can use the following sample Kibana dashboard to visualize transactions.
 
@@ -240,3 +232,11 @@ file. This sample web application is added as a service in the Ingress.
    You can import the Kibana dashboard template from [dashboards](../dashboards/KibanaAppTrans.ndjson).
    Before importing the Kibana dashboard, you must define an index pattern named `*http*` using the information in the [Kibana User Guide](https://www.elastic.co/guide/en/kibana/current/tutorial-define-index.html).
 
+### Sample Grafana dashboard for Prometheus
+
+Following is a sample Grafana dashboard which visualizes time series data from Prometheus. Kafka is used as the transaction endpoint.
+
+ ![Grafana-dashboard](../media/COE-GrafanaDashboard.png)
+
+  **Note:**
+   You can import the Grafana dashboard template from [dashboards](../dashboards).
diff --git a/media/citrix-observability-exporter-deploy.png b/media/citrix-observability-exporter-deploy.png
