Merge branch 'master' into 54_AppSettings-config-with-colon

Author: StephenMolloy

README.md

@@ -19,7 +19,7 @@ Version 2 is here with some new features:
   * Base Optional Tag - The [optional](#optional) tag that some of the builders in this project employed in V1 has been moved into the base class and is now available
 		on all key/value config builders.
   * Section Handlers - This feature allows users to develop extensions that will apply key/value config to sections other than `appSettings` and `connectionStrings`
-		if desired. Read more about this feature in the [Section Handlers](#sectionhandlers) segment below.
+		if desired. Read more about this feature in the [Section Handlers](#section-handlers) segment below.
 
 ## Key/Value Config Builders
 
@@ -69,6 +69,13 @@ key name before being inserted into AppSettings. `stripPrefix` is a boolean valu
 This setting is a boolean that specified whether to avoid throwing exceptions when the backing configuration source cannot be found or connected.
 The default default value is `true`, though some config builders (such as the Azure-based builders) will use a different default.
 
+#### escapeExpandedValues
+.Net configuration is XML-based in it's raw form. While these config builders work on `ConfigurationSection` objects in `Strict` and `Greedy` modes,
+when operating in `Expand` mode, tokens in the raw XML input are directly replaced with values. Applications that use `Expand` mode may do so because
+they need to inject additional XML rather than just a string value. But for the cases when a simple string replacement is the goal, unescaped XML
+characters in replacement values may result in invalid XML. In these cases, simply set the `escapeExpandedValues` attribute to `true` and the
+config builder will escape special XML characters before replacing tokens in `Expand` mode. The default value is `false`.
+
 #### tokenPattern
 This is a setting that is shared between all KeyValueConfigBuilder-derived builders is `tokenPattern`. When describing the `Expand` behavior of these builders
 above, it was mentioned that the raw xml is searched for tokens that look like __`${token}`__. This is done with a regular expression. `@"\$\{(\w+)\}"` to be exact.
@@ -119,7 +126,7 @@ preceded with an '@' symbol.
 ### EnvironmentConfigBuilder
 ```xml
 <add name="Environment"
-    [mode|@prefix|@stripPrefix|tokenPattern|@optional=true]
+    [mode|@prefix|@stripPrefix|tokenPattern|@escapeExpandedValues|@optional=true]
     type="Microsoft.Configuration.ConfigurationBuilders.EnvironmentConfigBuilder, Microsoft.Configuration.ConfigurationBuilders.Environment" />
 ```
 This is the most basic of the config builders. It draws its values from Environment, and it does not have any additional configuration options.
@@ -132,7 +139,7 @@ This is the most basic of the config builders. It draws its values from Environm
 ### UserSecretsConfigBuilder
 ```xml
 <add name="UserSecrets"
-    [mode|@prefix|@stripPrefix|tokenPattern|@optional=true]
+    [mode|@prefix|@stripPrefix|tokenPattern|@escapeExpandedValues|@optional=true]
     (@userSecretsId="12345678-90AB-CDEF-1234-567890" | @userSecretsFile="~\secrets.file")
     type="Microsoft.Configuration.ConfigurationBuilders.UserSecretsConfigBuilder, Microsoft.Configuration.ConfigurationBuilders.UserSecrets" />
 ```
@@ -168,7 +175,7 @@ and currently exposes the format of the file which, as mentioned above, should b
 ### AzureAppConfigurationBuilder
 ```xml
 <add name="AzureAppConfig"
-    [mode|@prefix|@stripPrefix|tokenPattern|@optional=false]
+    [mode|@prefix|@stripPrefix|tokenPattern|@escapeExpandedValues|@optional=false]
     (@vaultName="MyVaultName" | @uri="https://MyVaultName.vault.azure.net")
     [@connectionString="connection string"]
     [@version="secrets version"]
@@ -211,7 +218,7 @@ entries: `item1` and `item2`.
 ### AzureKeyVaultConfigBuilder
 ```xml
 <add name="AzureKeyVault"
-    [mode|@prefix|@stripPrefix|tokenPattern|@optional=false]
+    [mode|@prefix|@stripPrefix|tokenPattern|@escapeExpandedValues|@optional=false]
     (@vaultName="MyVaultName" | @uri="https://MyVaultName.vault.azure.net")
     [@connectionString="connection string"]
     [@version="secrets version"]
@@ -249,7 +256,7 @@ entries: `item1` and `item2`.
 ### KeyPerFileConfigBuilder
 ```xml
 <add name="KeyPerFile"
-    [mode|@prefix|@stripPrefix|tokenPattern|@optional=false]
+    [mode|@prefix|@stripPrefix|tokenPattern|@escapeExpandedValues|@optional=false]
 	(@directoryPath="PathToSourceDirectory")
     [@ignorePrefix="ignore."]
     [keyDelimiter=":"]
@@ -267,7 +274,7 @@ their orchestrated windows containers in this key-per-file manner.
 ### SimpleJsonConfigBuilder
 ```xml
 <add name="SimpleJson"
-    [mode|@prefix|@stripPrefix|tokenPattern|@optional=true]
+    [mode|@prefix|@stripPrefix|tokenPattern|@escapeExpandedValues|@optional=true]
     @jsonFile="~\config.json"
     [@jsonMode="(Flat|Sectional)"]
     type="Microsoft.Configuration.ConfigurationBuilders.SimpleJsonConfigBuilder, Microsoft.Configuration.ConfigurationBuilders.Json" />
@@ -339,7 +346,25 @@ collection. As an example, here is what their explicit declaration would look li
   </handlers>
 </Microsoft.Configuration.ConfigurationBuilders.SectionHandlers>
 ```
-When adding additional handlers, the name of the section must be 'Microsoft.Configuration.ConfigurationBuilders.SectionHandlers' so key/value config builders can find it.
+When adding additional handlers, the name of this section must be 'Microsoft.Configuration.ConfigurationBuilders.SectionHandlers' so key/value config builders can find it.
+Also note that a more qualified type will be required so the runtime can determine which assembly contains the new handler type. When working
+with ASP.Net applications, it is hit and miss regarding whether its able to define new section handlers in `App_Code` or not. Some configuration
+sections (such as `appSettings`) get loaded by ASP.Net before `App_Code` is compiled, so handlers for those sections will need to be
+compiled in a separate assembly in the 'bin' directory. For example:
+```xml
+<configSections>
+  <section name="Microsoft.Configuration.ConfigurationBuilders.SectionHandlers" type="Microsoft.Configuration.ConfigurationBuilders.SectionHandlersSection, Microsoft.Configuration.ConfigurationBuilders.Base" restartOnExternalChanges="false" requirePermission="false" />
+</configSections>
+
+<Microsoft.Configuration.ConfigurationBuilders.SectionHandlers>
+  <handlers>
+    <remove name="DefaultAppSettingsHandler" />
+    <add name="DefaultAppSettingsHandler" type="MyCustomAppSettingsSectionHandler, RefAssemblyInBin" />
+    <remove name="DefaultConnectionStringsHandler" />
+    <add name="DefaultConnectionStringsHandler" type="MyCustomConnectionStringsSectionHandler, App_Code" />
+  </handlers>
+</Microsoft.Configuration.ConfigurationBuilders.SectionHandlers>
+```
 
 ## Implementing More Key/Value Config Builders
 

samples/SampleWebApp/App_Data/expandTest.json

@@ -0,0 +1,3 @@
+{
+  "expandTestCS": "A & really ' bad \" unescaped < connection > string."
+}

samples/SampleWebApp/App_Data/settings.json

@@ -21,7 +21,7 @@
     "jsonComplex": {
       "setting1": "Complex Setting 1",
       "setting2": "Complex Setting 2",
-      "jsonArrayOfSettings": [ "one", "two", "three"]
+      "jsonArrayOfSettings": [ "one", "two", "three" ]
     }
   }
 }

samples/SampleWebApp/SampleWebApp.csproj

@@ -91,6 +91,7 @@
   </ItemGroup>
   <ItemGroup>
     <Content Include="App_Data\settings.json" />
+    <Content Include="App_Data\expandTest.json" />
     <None Include="packages.config" />
     <None Include="Web.Debug.config">
       <DependentUpon>Web.config</DependentUpon>

samples/SampleWebApp/Web.config

@@ -22,18 +22,19 @@
 
   <configBuilders>
     <builders>
-      <add name="Environment" type="Microsoft.Configuration.ConfigurationBuilders.EnvironmentConfigBuilder, Microsoft.Configuration.ConfigurationBuilders.Environment, Version=2.0.0.0, Culture=neutral" />
-      <add name="Secrets" userSecretsFile="~/App_Data/secrets.xml" mode="Greedy" type="Microsoft.Configuration.ConfigurationBuilders.UserSecretsConfigBuilder, Microsoft.Configuration.ConfigurationBuilders.UserSecrets, Version=2.0.0.0, Culture=neutral" />
-      <add name="Json" jsonFile="${JSONConfigFile}" optional="true" type="Microsoft.Configuration.ConfigurationBuilders.SimpleJsonConfigBuilder, Microsoft.Configuration.ConfigurationBuilders.Json, Version=2.0.0.0, Culture=neutral" />
-      <add name="KeyPerFile" directoryPath="~/../KeyPerFileSampleRoot" mode="Strict" keyDelimiter="--" type="Microsoft.Configuration.ConfigurationBuilders.KeyPerFileConfigBuilder, Microsoft.Configuration.ConfigurationBuilders.KeyPerFile, Version=2.0.0.0, Culture=neutral" />
+      <add name="Environment" type="Microsoft.Configuration.ConfigurationBuilders.EnvironmentConfigBuilder, Microsoft.Configuration.ConfigurationBuilders.Environment, Version=2.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35" />
+      <add name="Secrets" userSecretsFile="~/App_Data/secrets.xml" mode="Greedy" type="Microsoft.Configuration.ConfigurationBuilders.UserSecretsConfigBuilder, Microsoft.Configuration.ConfigurationBuilders.UserSecrets, Version=2.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35" />
+      <add name="Json" jsonFile="${JSONConfigFile}" optional="true" type="Microsoft.Configuration.ConfigurationBuilders.SimpleJsonConfigBuilder, Microsoft.Configuration.ConfigurationBuilders.Json, Version=2.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35" />
+      <add name="KeyPerFile" directoryPath="~/../KeyPerFileSampleRoot" mode="Strict" keyDelimiter="--" type="Microsoft.Configuration.ConfigurationBuilders.KeyPerFileConfigBuilder, Microsoft.Configuration.ConfigurationBuilders.KeyPerFile, Version=2.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35" />
 
-      <add name="KV1" vaultName="${ConfigBuilderTestKeyVaultName}" type="Microsoft.Configuration.ConfigurationBuilders.AzureKeyVaultConfigBuilder, Microsoft.Configuration.ConfigurationBuilders.Azure, Version=2.0.0.0, Culture=neutral" />
-      <add name="KV2" vaultName="${ConfigBuilderTestKeyVaultName}" version="d14197de791c4ffe8e79bc7fc0f766b0" type="Microsoft.Configuration.ConfigurationBuilders.AzureKeyVaultConfigBuilder, Microsoft.Configuration.ConfigurationBuilders.Azure, Version=2.0.0.0, Culture=neutral" />
-      <add name="KV3" vaultName="${ConfigBuilderTestKeyVaultName}" mode="Greedy" type="Microsoft.Configuration.ConfigurationBuilders.AzureKeyVaultConfigBuilder, Microsoft.Configuration.ConfigurationBuilders.Azure, Version=2.0.0.0, Culture=neutral" />
-      <add name="KV4" vaultName="${ConfigBuilderTestKeyVaultName}" mode="Greedy" version="0de51928e49144ce86eb1de9056ac937" type="Microsoft.Configuration.ConfigurationBuilders.AzureKeyVaultConfigBuilder, Microsoft.Configuration.ConfigurationBuilders.Azure, Version=2.0.0.0, Culture=neutral" />
+      <add name="KV1" vaultName="${ConfigBuilderTestKeyVaultName}" type="Microsoft.Configuration.ConfigurationBuilders.AzureKeyVaultConfigBuilder, Microsoft.Configuration.ConfigurationBuilders.Azure, Version=2.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35" />
+      <add name="KV2" vaultName="${ConfigBuilderTestKeyVaultName}" version="d14197de791c4ffe8e79bc7fc0f766b0" type="Microsoft.Configuration.ConfigurationBuilders.AzureKeyVaultConfigBuilder, Microsoft.Configuration.ConfigurationBuilders.Azure, Version=2.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35" />
+      <add name="KV3" vaultName="${ConfigBuilderTestKeyVaultName}" mode="Greedy" type="Microsoft.Configuration.ConfigurationBuilders.AzureKeyVaultConfigBuilder, Microsoft.Configuration.ConfigurationBuilders.Azure, Version=2.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35" />
+      <add name="KV4" vaultName="${ConfigBuilderTestKeyVaultName}" mode="Greedy" version="0de51928e49144ce86eb1de9056ac937" type="Microsoft.Configuration.ConfigurationBuilders.AzureKeyVaultConfigBuilder, Microsoft.Configuration.ConfigurationBuilders.Azure, Version=2.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35" />
 
-      <add name="AS_Sub_Test" optional="${Boolean}" type="Microsoft.Configuration.ConfigurationBuilders.EnvironmentConfigBuilder, Microsoft.Configuration.ConfigurationBuilders.Environment, Version=2.0.0.0, Culture=neutral" />
-      <add name="AS_Sub_Test2" optional="${app~Settings_Colon-and$friends@super+duper,awesome#cool:Test.}" type="Microsoft.Configuration.ConfigurationBuilders.EnvironmentConfigBuilder, Microsoft.Configuration.ConfigurationBuilders.Environment, Version=2.0.0.0, Culture=neutral" />
+      <add name="AS_Sub_Test" optional="${Boolean}" type="Microsoft.Configuration.ConfigurationBuilders.EnvironmentConfigBuilder, Microsoft.Configuration.ConfigurationBuilders.Environment, Version=2.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35" />
+      <add name="AS_Sub_Test2" optional="${app~Settings_Colon-and$friends@super+duper,awesome#cool:Test.}" type="Microsoft.Configuration.ConfigurationBuilders.EnvironmentConfigBuilder, Microsoft.Configuration.ConfigurationBuilders.Environment, Version=2.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35" />
+      <add name="ExpTest" mode="Expand" escapeExpandedValues="true" jsonFile="~/App_Data/expandTest.json" jsonMode="Flat" type="Microsoft.Configuration.ConfigurationBuilders.SimpleJsonConfigBuilder, Microsoft.Configuration.ConfigurationBuilders.Json, Version=2.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35" />
     </builders>
   </configBuilders>
 
@@ -64,7 +65,9 @@
     <!-- key="Secret3" value="Will be added by KV3:Latest3. IFF already added, will be "updated" by KV1 to Latest3. KV2 and KV4 don't have versions matching this secret." -->
   </appSettings>
 
-  <connectionStrings configBuilders="Json,AS_Sub_Test2">
+  <connectionStrings configBuilders="Json,ExpTest,AS_Sub_Test2">
+    <add name="expansionTest" connectionString="${expandTestCS}" />
+    <add name="expandTestCS" connectionString="Only replaced in Strict/Greedy modes. Not Expand."/>
     <add name="jsonConnectionString1" connectionString="Will be replaced by 'Json' in 'Flat' AND 'Sectional' jsonModes, but with different values." />
     <add name="connectionStrings:jsonConnectionString1" connectionString="Will only be replaced by 'Json' in 'Flat' jsonMode." />
     <add name="jsonConnectionString2" connectionString="Will only be replaced by 'Json' in 'Sectional' jsonMode." />

src/Azure/AzureKeyVaultConfigBuilder.cs

@@ -143,14 +143,33 @@ public override ICollection<KeyValuePair<string, string>> GetAllValues(string pr
             return d;
         }
 
+        /// <summary>
+        /// Transform the given key to an intermediate format that will be used to look up values in backing store.
+        /// </summary>
+        /// <param name="key">The string to be mapped.</param>
+        /// <returns>The key string to be used while looking up config values..</returns>
+        public override string MapKey(string key)
+        {
+            if (String.IsNullOrEmpty(key))
+                return key;
+
+            // Colons and underscores are common in appSettings keys, but not allowed in key vault key names.
+            // It's likely that apps will want to lookup config values with these characters in their name in
+            // key vault. More extensive key mapping can be done with subclasses. But let's handle the most
+            // most common case here.
+            key = key.Replace(':', '-');
+            key = key.Replace('_', '-');
+            return key;
+        }
+
         /// <summary>
         /// Makes a determination about whether the input key is valid for this builder and backing store.
         /// </summary>
         /// <param name="key">The string to be validated. May be partial.</param>
         /// <returns>True if the string is valid. False if the string is not a valid key.</returns>
         public override bool ValidateKey(string key)
         {
-            // Key Vault only allows alphanumerics and '-'. This builder also allows for one '/'.
+            // Key Vault only allows alphanumerics and '-'. This builder also allows for one '/' for versioning.
             return Regex.IsMatch(key, "^[a-zA-Z0-9-]+(/?[a-zA-Z0-9-]+)?$");
         }
 

src/AzureAppConfig/AzureAppConfigurationBuilder.cs

@@ -5,6 +5,7 @@
 using System.Collections.Generic;
 using System.Collections.Specialized;
 using System.Linq;
+using System.Text.RegularExpressions;
 using System.Threading;
 using System.Threading.Tasks;
 using Microsoft.Azure.AppConfiguration.Azconfig;
@@ -45,9 +46,9 @@ protected override void LazyInitialize(string name, NameValueCollection config)
 
             base.LazyInitialize(name, config);
 
-            _keyFilter = config[keyFilterTag];
-            _labelFilter = config[labelFilterTag];
-            _dateTimeFilter = DateTimeOffset.TryParse(config[dateTimeFilterTag], out _dateTimeFilter) ? _dateTimeFilter : DateTimeOffset.MinValue;
+            _keyFilter = UpdateConfigSettingWithAppSettings(keyFilterTag);
+            _labelFilter = UpdateConfigSettingWithAppSettings(labelFilterTag);
+            _dateTimeFilter = DateTimeOffset.TryParse(UpdateConfigSettingWithAppSettings(dateTimeFilterTag), out _dateTimeFilter) ? _dateTimeFilter : DateTimeOffset.MinValue;
 
             // Place some restrictions on label filter, similar to the .net core provider.
             // The idea is to restrict queries to one label, and one label only. Even if that
@@ -102,8 +103,21 @@ protected override void LazyInitialize(string name, NameValueCollection config)
         /// <returns>True if the string is valid. False if the string is not a valid key.</returns>
         public override bool ValidateKey(string key)
         {
-            // Azure App Config does not restrict key names, although a couple characters have special meaning if not escaped.
-            // We may want to restrict using those characters unescaped in a key name in the future.
+            // From - https://docs.microsoft.com/en-us/azure/azure-app-configuration/concept-key-value
+            // You can use any unicode character in key names entered into App Configuration except for *, ,, and \. These characters are
+            // reserved.If you need to include a reserved character, you must escape it by using \{ Reserved Character}.
+            if (String.IsNullOrWhiteSpace(key))
+                return false;
+
+            if (key.Contains('*') || key.Contains(','))
+                return false;
+
+            // We don't want to completely disallow '\' since it is used for escaping. But writing a full parser for someone elses
+            // naming format could be error prone. If we see a '\' followed by a '{', just call it good. Don't bother with the Regex
+            // if there aren't any backslashes though.
+            if (key.Contains('\\'))
+                return !Regex.IsMatch(key, @"\\[^{]");
+
             return true;
         }