Skip to content

Commit e861ff8

Browse files
davidalpertdavidalpert
authored
Introduce ConvertConsulKVPairToConfig option on IConsulConfigurationSource (#103)
The current strategy for parsing consul KVPairs into IConfiguration data makes a few key assumptions: - IConfigurationParser implementations can parse consul Values without any knowledge of the consul Key; and - IConfigurationParser implementations are not concerned with parsing the consul Keys themselves - The only manipulation of consul Keys required is the optional removal of a prefix (e.g. KeyToRemove) By pulling a Func<KVPair, IEnumerable<KeyValuePair<string,string>>> property up to ConsulConfigurationSource we now allow clients to bypass these assumptions and fully customize the process of providing IConfiguration data from a Consul KV store.
1 parent aceae35 commit e861ff8

9 files changed

Lines changed: 201 additions & 51 deletions

File tree

‎README.md‎

Lines changed: 58 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -7,6 +7,16 @@
77

88
Adds support for configuring .NET Core applications using Consul. Works great with [git2consul](https://github.com/Cimpress-MCP/git2consul).
99

10+
- [Installation](#installation)
11+
- [Usage](#usage)
12+
- [Minimal Setup](#minimal-setup)
13+
- [Options](#options)
14+
- [Configure Parsing Options](#configure-parsing-options)
15+
- [Consul values are JSON](#consul-values-are-json)
16+
- [Consul values are scalars](#consul-values-are-scalars)
17+
- [Consul values are a mix of JSON and scalars](#consul-values-are-a-mix-of-json-and-scalars)
18+
- [Customizing the `ConvertConsulKVPairToConfig` strategy](#customizing-the-ConvertConsulKVPairToConfig-strategy)
19+
1020
## Installation
1121

1222
Add `Winton.Extensions.Configuration.Consul` to your project's dependencies, either via the NuGet package manager or as a `PackageReference` in the csproj file.
@@ -70,7 +80,18 @@ Assuming the application is running in the 'Development' environment and the app
7080
A `bool` indicating whether to reload the config when it changes in Consul.
7181
If `true` it will watch the configured key for changes. When a change occurs the config will be asynchronously reloaded and the `IChangeToken` will be triggered to signal that the config has been reloaded. Defaults to `false`.
7282

73-
## Storing Config as Expanded Keys In Consul
83+
* **`ConvertConsulKVPairToConfig`**
84+
85+
A `Func<KVPair, IEnumerable<KeyValuePair<string, string>>>` which gives you complete control over the parsing of fully qualified consul keys and raw consul values; the default implementation will:
86+
87+
- Use the configured `Parser` to parse consul values
88+
- Remove the configured `KeyToRemove` prefix from consul keys
89+
90+
When setting this member, however, you bypass the default key and value processing and `Parser` and `KeyToRemove` have no effect unless your `ConvertConsulKVPairToConfig` function uses them.
91+
92+
## Configure Parsing Options
93+
94+
### Consul values are JSON
7495

7596
By default this configuration provider will load all key-value pairs from Consul under the specified root key, but by default it assumes that the values of the leaf keys are encoded as JSON.
7697

@@ -102,6 +123,8 @@ var configuration = builder
102123

103124
The resultant configuration would contain sections for `auth` and `logging`. As a concrete example `configuration.GetValue<string>("logging:level")` would return `"warn"` and `configuration.GetValue<string>("auth:claims:0")` would return `"email"`.
104125

126+
### Consul values are scalars
127+
105128
Sometimes however, config in Consul is stored as a set of expanded keys. For instance, tools such as `consul-cli` load config in this format.
106129

107130
The config in this case can be thought of as a tree under a specific root key in Consul. For instance, continuing with the example above, the config would be stored as:
@@ -135,6 +158,8 @@ builder
135158

136159
The `SimpleConfigurationParser` expects to encounter a scalar value at each leaf key in the tree.
137160

161+
### Consul values are a mix of JSON and scalars
162+
138163
If you need to support both expanded keys and JSON values then this can be achieved by putting them under different root keys and adding multiple configuration sources. For example:
139164

140165
```csharp
@@ -147,3 +172,35 @@ builder
147172
})
148173
.AddConsul("myApp/jsonValues", cancellationToken);
149174
```
175+
176+
### Customizing the `ConvertConsulKVPairToConfig` strategy
177+
178+
Sometimes you may need more control over the conversion of raw consul KV pairs into `IConfiguration` data. In this case you can set a custom `ConvertConsulKVPairToConfig` function:
179+
180+
```csharp
181+
builder
182+
.AddConsul(
183+
"myApp",
184+
options =>
185+
{
186+
options.ConvertConsulKVPairToConfig = kvPair =>
187+
{
188+
var normalizedKey = kvPair.Key
189+
.Replace("base/key", string.Empty)
190+
.Replace("__", "/")
191+
.Replace("/", ":")
192+
.Trim('/');
193+
194+
using Stream valueStream = new MemoryStream(kvPair.Value);
195+
using var streamReader = new StreamReader(valueStream);
196+
var parsedValue = streamReader.ReadToEnd();
197+
198+
return new Dictionary<string, string>()
199+
{
200+
{ normalizedKey, parsedValue }
201+
};
202+
};
203+
});
204+
```
205+
206+
> :warning: Caution: by customizing this `ConvertConsulKVPairToConfig` strategy you bypass any automatic invocation of the configured `Parser` and `KeyToRemove` so it becomes your responsibility to use them as needed by your scenario.

‎src/Winton.Extensions.Configuration.Consul/ConsulConfigurationProvider.cs‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -156,7 +156,7 @@ private async Task PollingLoop(CancellationToken cancellationToken)
156156

157157
private void SetData(QueryResult<KVPair[]> result)
158158
{
159-
Data = result.ToConfigDictionary(_source.KeyToRemove, _source.Parser);
159+
Data = result.ToConfigDictionary(_source.ConvertConsulKVPairToConfig);
160160
}
161161

162162
private void SetLastIndex(QueryResult result)

‎src/Winton.Extensions.Configuration.Consul/ConsulConfigurationSource.cs‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2,9 +2,11 @@
22
// Licensed under the Apache License, Version 2.0. See LICENSE in the project root for license information.
33

44
using System;
5+
using System.Collections.Generic;
56
using System.Net.Http;
67
using Consul;
78
using Microsoft.Extensions.Configuration;
9+
using Winton.Extensions.Configuration.Consul.Extensions;
810
using Winton.Extensions.Configuration.Consul.Parsers;
911

1012
namespace Winton.Extensions.Configuration.Consul
@@ -22,6 +24,7 @@ public ConsulConfigurationSource(string key)
2224

2325
Key = key;
2426
Parser = new JsonConfigurationParser();
27+
ConvertConsulKVPairToConfig = DefaultConvertConsulKVPairToConfigStrategy;
2528
}
2629

2730
public Action<ConsulClientConfiguration>? ConsulConfigurationOptions { get; set; }
@@ -38,6 +41,8 @@ public string KeyToRemove
3841
set => _keyToRemove = value;
3942
}
4043

44+
public Func<KVPair, IEnumerable<KeyValuePair<string, string>>> ConvertConsulKVPairToConfig { get; set; }
45+
4146
public Action<ConsulLoadExceptionContext>? OnLoadException { get; set; }
4247

4348
public Func<ConsulWatchExceptionContext, TimeSpan>? OnWatchException { get; set; }
@@ -55,5 +60,10 @@ public IConfigurationProvider Build(IConfigurationBuilder builder)
5560
var consulClientFactory = new ConsulClientFactory(this);
5661
return new ConsulConfigurationProvider(this, consulClientFactory);
5762
}
63+
64+
private IEnumerable<KeyValuePair<string, string>> DefaultConvertConsulKVPairToConfigStrategy(KVPair consulKvPair)
65+
{
66+
return consulKvPair.ConvertToConfig(this.KeyToRemove, this.Parser);
67+
}
5868
}
5969
}

‎src/Winton.Extensions.Configuration.Consul/Extensions/KVPairQueryResultExtensions.cs‎

Lines changed: 2 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -22,12 +22,11 @@ internal static bool HasValue(this QueryResult<KVPair[]> result)
2222

2323
internal static Dictionary<string, string> ToConfigDictionary(
2424
this QueryResult<KVPair[]> result,
25-
string keyToRemove,
26-
IConfigurationParser parser)
25+
Func<KVPair, IEnumerable<KeyValuePair<string, string>>> convertConsulKVPairToConfig)
2726
{
2827
return (result.Response ?? new KVPair[0])
2928
.Where(kvp => kvp.HasValue())
30-
.SelectMany(kvp => kvp.ConvertToConfig(keyToRemove, parser))
29+
.SelectMany(convertConsulKVPairToConfig)
3130
.ToDictionary(kvp => kvp.Key, kvp => kvp.Value, StringComparer.OrdinalIgnoreCase);
3231
}
3332
}

‎src/Winton.Extensions.Configuration.Consul/IConsulConfigurationSource.cs‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,7 @@
22
// Licensed under the Apache License, Version 2.0. See LICENSE in the project root for license information.
33

44
using System;
5+
using System.Collections.Generic;
56
using System.Net.Http;
67
using System.Threading;
78
using Consul;
@@ -23,6 +24,19 @@ public interface IConsulConfigurationSource : IConfigurationSource
2324
/// </summary>
2425
Action<ConsulClientConfiguration>? ConsulConfigurationOptions { get; set; }
2526

27+
/// <summary>
28+
/// Gets or sets a function taking a Consul <see cref="KVPair"/> to one or more key/value
29+
/// pairs which are injected into the Microsoft <see cref="IConfiguration"/> system.
30+
/// </summary>
31+
/// <remarks>
32+
/// The default ConvertConsulKVPairToConfig strategy is to remove the
33+
/// <see cref="KeyToRemove"/> portion of the Consul Key and then apply the configured
34+
/// <see cref="Parser"/> to parse the Consul value. Note that if you customize this strategy
35+
/// there are some requirements on the final format of
36+
/// <see href="https://docs.microsoft.com/en-us/aspnet/core/fundamentals/configuration/?view=aspnetcore-3.1#configuration-keys-and-values">Configuration Keys and Values</see>.
37+
/// </remarks>
38+
Func<KVPair, IEnumerable<KeyValuePair<string, string>>> ConvertConsulKVPairToConfig { get; set; }
39+
2640
/// <summary>
2741
/// Gets or sets an <see cref="Action" /> to be applied to the <see cref="HttpClientHandler" />
2842
/// during construction of the <see cref="IConsulClient" />.

‎test/Winton.Extensions.Configuration.Consul.Test/ConsulConfigurationProviderTests.cs‎

Lines changed: 91 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -3,6 +3,7 @@
33
using System.IO;
44
using System.Linq;
55
using System.Net;
6+
using System.Text;
67
using System.Threading;
78
using System.Threading.Tasks;
89
using Consul;
@@ -674,5 +675,95 @@ private async Task ShouldWatchForChangesIfSourceReloadOnChangesIsTrue()
674675
Times.Exactly(2));
675676
}
676677
}
678+
679+
public sealed class CustomizeConvertConsulKVPairToConfig : ConsulConfigurationProviderTests
680+
{
681+
public CustomizeConvertConsulKVPairToConfig()
682+
{
683+
_source.ReloadOnChange = false;
684+
}
685+
686+
[Fact]
687+
private void ShouldSetData()
688+
{
689+
_parser
690+
.Setup(cp => cp.Parse(It.IsAny<MemoryStream>()))
691+
.Throws(new Exception("Should not get here..."));
692+
_kvEndpoint
693+
.Setup(kv => kv.List("Test", It.IsAny<QueryOptions>(), It.IsAny<CancellationToken>()))
694+
.ReturnsAsync(
695+
new QueryResult<KVPair[]>
696+
{
697+
Response = new[]
698+
{
699+
new KVPair("Test/key__with__double__underscores") { Value = Encoding.UTF8.GetBytes("Value") }
700+
},
701+
StatusCode = HttpStatusCode.OK
702+
});
703+
704+
_source.ConvertConsulKVPairToConfig = kvPair =>
705+
{
706+
var normalizedKey = kvPair.Key
707+
.Replace("__", ":")
708+
.Replace(_source.KeyToRemove, string.Empty)
709+
.Trim('/');
710+
711+
using Stream valueStream = new MemoryStream(kvPair.Value);
712+
using var streamReader = new StreamReader(valueStream);
713+
var parsedValue = streamReader.ReadToEnd();
714+
715+
return new Dictionary<string, string>()
716+
{
717+
{ normalizedKey, parsedValue }
718+
};
719+
};
720+
721+
_provider.Load();
722+
723+
_provider.TryGet("key:with:double:underscores", out var value);
724+
value.Should().Be("Value");
725+
}
726+
727+
[Fact]
728+
private void ShouldSetDataUsingDefinedParser()
729+
{
730+
_parser
731+
.Setup(cp => cp.Parse(It.IsAny<MemoryStream>()))
732+
.Returns(new Dictionary<string, string> { { string.Empty, "Value" } });
733+
_kvEndpoint
734+
.Setup(kv => kv.List("Test", It.IsAny<QueryOptions>(), It.IsAny<CancellationToken>()))
735+
.ReturnsAsync(
736+
new QueryResult<KVPair[]>
737+
{
738+
Response = new[]
739+
{
740+
new KVPair("Test/key__with__double__underscores") { Value = Encoding.UTF8.GetBytes("Value") }
741+
},
742+
StatusCode = HttpStatusCode.OK
743+
});
744+
745+
_source.ConvertConsulKVPairToConfig = kvPair =>
746+
{
747+
var normalizedKey = kvPair.Key
748+
.Replace("__", ":")
749+
.Replace(_source.KeyToRemove, string.Empty)
750+
.Trim('/');
751+
752+
using Stream valueStream = new MemoryStream(kvPair.Value);
753+
var parsedPairs = _source.Parser.Parse(valueStream);
754+
return parsedPairs.Select(parsedPair =>
755+
{
756+
return new KeyValuePair<string, string>(
757+
$"{normalizedKey}/{parsedPair.Key}".Trim('/'),
758+
parsedPair.Value);
759+
});
760+
};
761+
762+
_provider.Load();
763+
764+
_provider.TryGet("key:with:double:underscores", out var value);
765+
value.Should().Be("Value");
766+
}
767+
}
677768
}
678769
}

‎test/Winton.Extensions.Configuration.Consul.Test/ConsulConfigurationSourceTests.cs‎

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,8 @@
11
using System;
2+
using System.Collections.Generic;
3+
using System.Linq;
4+
using System.Text;
5+
using Consul;
26
using FluentAssertions;
37
using Winton.Extensions.Configuration.Consul.Parsers;
48
using Xunit;
@@ -60,6 +64,18 @@ private void ShouldThrowIfKeyIsInvalid(string key)
6064

6165
constructing.Should().Throw<ArgumentNullException>().And.Message.Should().Contain("key");
6266
}
67+
68+
[Fact]
69+
private void ShoulSetDefaultConvertConsulKVPairToConfigStrategy()
70+
{
71+
var source = new ConsulConfigurationSource("Key");
72+
73+
var consulKVPair = new KVPair("key") { Value = Encoding.UTF8.GetBytes("{\"a\": \"b\", \"c\": \"d\"}") };
74+
75+
var result = source.ConvertConsulKVPairToConfig(consulKVPair);
76+
result.Should()
77+
.BeEquivalentTo(new Dictionary<string, string> { { "key:a", "b" }, { "key:c", "d" } });
78+
}
6379
}
6480
}
6581
}

‎test/Winton.Extensions.Configuration.Consul.Test/Extensions/KVPairExtensionsTests.cs‎

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -11,16 +11,16 @@ namespace Winton.Extensions.Configuration.Consul.Extensions
1111
{
1212
public class KVPairExtensionsTests
1313
{
14-
public sealed class ConvertToConfig : KVPairExtensionsTests
14+
public sealed class ConvertConsulKVPairToConfig : KVPairExtensionsTests
1515
{
1616
private readonly Mock<IConfigurationParser> _parserMock;
1717

18-
public ConvertToConfig()
18+
public ConvertConsulKVPairToConfig()
1919
{
2020
_parserMock = new Mock<IConfigurationParser>(MockBehavior.Strict);
2121
}
2222

23-
public static IEnumerable<object[]> ConvertToConfigTestCases => new List<object[]>
23+
public static IEnumerable<object[]> ConvertConsulKVPairToConfigTestCases => new List<object[]>
2424
{
2525
new object[]
2626
{
@@ -136,7 +136,7 @@ public ConvertToConfig()
136136
};
137137

138138
[Theory]
139-
[MemberData(nameof(ConvertToConfigTestCases))]
139+
[MemberData(nameof(ConvertConsulKVPairToConfigTestCases))]
140140
private void ShouldConvertKVPairToConfigCorrectly(
141141
string rootKey,
142142
string kvPairKey,

0 commit comments

Comments
 (0)