Update polling config and paging config with updated information

Author: Srikar Shastry

articles/sentinel/data-connector-connection-rules-reference.md

@@ -354,6 +354,7 @@ The request section defines how the CCF data connector sends requests to your da
 | ---- | ---- | ---- | ---- |
 | **ApiEndpoint** | True | String | URL for remote server. Defines the endpoint to pull data from. |
 | **RateLimitQPS** |  | Integer | Defines the number of calls or queries allowed in a second. |
+| **RateLimitConfig** |  | Object | Defines the rate limit configuration for the RESTful API. See [example](#ratelimitconfig-example). |
 | **QueryWindowInMin** |  | Integer | Defines the available query window in minutes. Minimum is 1 minute. Default is 5 minutes.|
 | **HttpMethod** |  | String | Defines the API method: `GET`(default) or `POST` |
 | **QueryTimeFormat** |  | String | Defines the date and time format the endpoint (remote server) expects. The CCF uses the current date and time wherever this variable is used. Possible values are the constants: `UnixTimestamp`, `UnixTimestampInMills` or any other valid representation of date time, for example: `yyyy-MM-dd`, `MM/dd/yyyy HH:mm:ss`<br>default is ISO 8601 UTC |
@@ -395,7 +396,40 @@ Consider this example:
 - `QueryTimeIntervalDelimiter` = `..`
 - `ApiEndpoint` = `https://www.example.com`
 
-The query sent to the remote server is: `https://www.example.com?interval=time:{QueryTimeFormat}..{QueryTimeFormat + QueryWindowInMin}`	
+The query sent to the remote server is: `https://www.example.com?interval=time:{QueryTimeFormat}..{QueryTimeFormat + QueryWindowInMin}`
+
+### RateLimitConfig example
+
+Consider this example:
+- `ApiEndpoint` = `https://www.example.com`
+```json
+"rateLimitConfig": {
+  "evaluation": {
+    "checkMode": "OnlyWhen429"
+  },
+  "extraction": {
+    "source": "CustomHeaders",
+    "headers": {
+      "limit": {
+        "name": "X-RateLimit-Limit",
+        "format": "Integer"
+      },
+      "remaining": {
+        "name": "X-RateLimit-Remaining",
+        "format": "Integer"
+      },
+      "reset": {
+        "name": "X-RateLimit-RetryAfter",
+        "format": "UnixTimeSeconds"
+      }
+    }
+  },
+  "retryStrategy": {
+    "useResetOrRetryAfterHeaders": true
+  }
+}
+```
+When the response includes rate limit headers, the connector can use this information to adjust its request rate.
 
 ### Request examples using Microsoft Graph as data source API
 
@@ -555,6 +589,8 @@ The most common paging type is when a server data source API provides URLs to th
 | **LinkHeaderTokenJsonPath** | False | String | Use this property to indicate where to get the value in the response body.<br><br>For example, if the data source returns the following JSON: `{ nextPage: "foo", value: [{data}]}` then `LinkHeaderTokenJsonPath` will be `$.nextPage` |
 | **PageSize** | False | Integer | How many events per page |
 | **PageSizeParameterName** | False | String | Query parameter name for the page size |
+| **PagingInfoPlacement** | False | String | How paging info is populated. Accepts either "QueryString" or "RequestBody" |
+| **PagingQueryParamOnly** | False | Boolean | If set to true, will omit all other query parameters except paging query parameters. |
 
 Here are some examples:
 
@@ -587,6 +623,8 @@ Paging: {
 | **HasNextFlagJsonPath** | False | String | Defines the path to the HasNextPage flag attribute |
 | **NextPageRequestHeader** | False | String | Determines the next page header name in the request. |
 | **NextPageUrlQueryParametersTemplate** | False | String | Only if the connector is for Coralogix API |
+| **PagingInfoPlacement** | False | String | How paging info is populated. Accepts either "QueryString" or "RequestBody" |
+| **PagingQueryParamOnly** | False | Boolean | If set to true, will omit all other query parameters except paging query parameters. |
 
 Example:
 
@@ -615,6 +653,8 @@ Paging: {
 | **NextPageParaName** | False | string | Determines the next page name in the request. |
 | **HasNextFlagJsonPath** | False | string | Defines the path to a **HasNextPage** flag attribute when determining if more pages are left in the response.	|
 | **NextPageRequestHeader** | False | string | Determines the next page header name in the request. |
+| **PagingInfoPlacement** | False | String | How paging info is populated. Accepts either "QueryString" or "RequestBody" |
+| **PagingQueryParamOnly** | False | Boolean | If set to true, will omit all other query parameters except paging query parameters. |
 
 Examples:
 
@@ -643,14 +683,18 @@ Paging: {
 | **PageSize** | False | Integer | How many events per page |
 | **PageSizeParameterName** | False | String | Query parameter name for the page size |
 | **OffsetParaName** | False | String | The next request query parameter name. The CCF calculates the offset value for each request, (all events ingested + 1) |
+| **PagingInfoPlacement** | False | String | How paging info is populated. Accepts either "QueryString" or "RequestBody" |
+| **PagingQueryParamOnly** | False | Boolean | If set to true, will omit all other query parameters except paging query parameters. |
 
 Example:
 ```json
 Paging: {
-   
-       "pagingType": "Offset", 
-        "offsetParaName": "offset" 
- }
+  "pagingType": "Offset", 
+  "offsetParaName": "offset",
+  "pageSize": 50,
+  "pagingQueryParamOnly": true,
+  "pagingInfoPlacement": "QueryString"
+}
 ```
 
 #### Configure CountBasedPaging
@@ -666,6 +710,8 @@ Paging: {
 | **TotalResultsJsonPath** | False | String | JSON path of total number of results in HTTP response payload |
 | **PageNumberJsonPath** | False | String | Required if totalResultsJsonPath is provided. JSON path of page number in HTTP response payload |
 | **PageCountJsonPath** | False | String | Required if totalResultsJsonPath is provided. JSON path of page count in HTTP response payload |
+| **PagingInfoPlacement** | False | String | How paging info is populated. Accepts either "QueryString" or "RequestBody" |
+| **PagingQueryParamOnly** | False | Boolean | If set to true, will omit all other query parameters except paging query parameters. |
 
 Example:
 
@@ -713,8 +759,33 @@ Here's an example of all the components of the CCF data connector JSON together.
       "request": {
          "apiEndpoint": "https://rest.contoso.com/example",
          "rateLimitQPS": 10,
+         "rateLimitConfig": {
+            "evaluation": {
+              "checkMode": "OnlyWhen429"
+            },
+            "extraction": {
+              "source": "CustomHeaders",
+              "headers": {
+                "limit": {
+                  "name": "X-RateLimit-Limit",
+                  "format": "Integer"
+                },
+                "remaining": {
+                  "name": "X-RateLimit-Remaining",
+                  "format": "Integer"
+                },
+                "reset": {
+                  "name": "X-RateLimit-RetryAfter",
+                  "format": "UnixTimeSeconds"
+                }
+              }
+            },
+            "retryStrategy": {
+              "useResetOrRetryAfterHeaders": true
+            }
+         },
          "queryWindowInMin": 5,
-         "httpMethod": "GET",
+         "httpMethod": "POST",
          "queryTimeFormat": "UnixTimestamp",
          "startTimeAttributeName": "t0",
          "endTimeAttributeName": "t1",
@@ -726,7 +797,9 @@ Here's an example of all the components of the CCF data connector JSON together.
          } 
       },
       "paging": {
-         "pagingType": "LinkHeader"
+         "pagingType": "LinkHeader",
+         "pagingInfoPlacement": "RequestBody",
+         "pagingQueryParamOnly": true
       },
       "response": {
          "eventsJsonPaths": ["$"]