diff --git a/CapriKit.slnx b/CapriKit.slnx index 818c7eb..4c79f7b 100644 --- a/CapriKit.slnx +++ b/CapriKit.slnx @@ -32,6 +32,12 @@ + + + + + + diff --git a/Research/AssetPipeline.md b/Research/AssetPipeline.md new file mode 100644 index 0000000..5345d53 --- /dev/null +++ b/Research/AssetPipeline.md @@ -0,0 +1,27 @@ +# Asset Pipeline +Now that I have support for encoding textures offline and then loading/transcoding them when the game runs (see CapriKit.SuperCompressed) it is time to start work on a real asset pipeline. I want you to help me create a high-level architecture for it. The expected output (when we are ready for that) is a markdown file in the `Research` folder. + +# Goals +1. Encoding/compile/compress assets into formats that makes them easy to ship and easy load +2. Load the assets when requested by the game into ready-to-use objects (for example, when done a texture is uploaded to the GPU and ready to be referenced) + +## Subgoals +1. Flexible enough to support textures, shaders, models and audio in a similar process +2. Support for detecting changes on-disk and then rebuilding and hot-reloading them +3. Loading a groups of assets (bundles) in parallel while the main loop keeps running uninterrupted. + 3a. I am thinking of a process similar to the parallel loading of test screens in the background in `C:\projects\csharp\CapriKit\source\CapriKit.Tests.Tool\Program.cs` with a `LightweightChannel` `C:\projects\csharp\CapriKit\source\CapriKit.Concurrency\Primitives\LightweightChannel.cs` + 3b. I wonder if I need two steps here, full background work and then coordination with the main loop to finish the loading of things like textures that need to be uploaded to GPU memory, which I think you cannot do in parallel/without a DeviceContext and help of the main thread. + 3c. I need a way to identify assets (maybe just a record `Asset` or `Asset` which holds the relative path to the asset). If I load a bundle I should be able to wait for the bundle to be fully loaded and then get the concrete Texture, Model, etc... object from the bundle using that record. + 3d. Somehow that Texture, model, etc.. object needs to have a point of indirection to support that hot reloading + + +I am saying bundle here as a helpful abstraction in code, but on disk I want each file to stay independent. I know it is a common optimization to compress multiple assets into one file and load those together (better disk IO) but I think that will complicate hot reloading and other IO code. (Happy to be proven wrong though, so maybe discuss it as an extra option but assume for now we are not doing it). + +## Assumptions +1. Assume assets do not need to carry extra information for the content pipeline. For example, if a texture should be loaded as normal map, srgb or linear texture is determined by the caller at run-time, not via an extra config file. +2. Assume that if an asset is loaded it stays loaded, we're more going for games like Factorio, Stationeers, Satisfactory and not for level-based games. + +## Examples +I have created a similar system before. See `C:\projects\csharp\MiniEngine3\src\Mini.Engine.Content\` and especially `C:\projects\csharp\MiniEngine3\src\Mini.Engine.Content\ContentManager.cs` and `C:\projects\csharp\MiniEngine3\src\Mini.Engine.Content\ContentProcessor.cs` but there were a couple of problems +- Background loading was super hacky +- A lot of boilerplate code, for example in the folder `C:\projects\csharp\MiniEngine3\src\Mini.Engine.Content\Shaders\` you can see that I needed 3 classes for each type of shader. I really want to have to use a lot less code. diff --git a/Research/AssetPipelineArchitecture.md b/Research/AssetPipelineArchitecture.md new file mode 100644 index 0000000..fd0ff6d --- /dev/null +++ b/Research/AssetPipelineArchitecture.md @@ -0,0 +1,234 @@ +# Asset Pipeline — High-Level Architecture + +Companion to `AssetPipeline.md` (the brief). Status: draft for discussion, 2026-07-10. + +## Decisions + +| Decision | Choice | Rationale | +|---|---|---| +| Build timing | **Hybrid** | Dev builds compile stale assets on demand into a cache; a CLI step produces the same output for shipping. Shipped games load precompiled assets only and carry no compiler code. | +| Asset relationships | **Flat** | No cross-asset references. Game code composes (loads a model's geometry and its textures separately). Kills the dependency-graph machinery that made MiniEngine3 complex. | +| GPU coupling | **Direct DX11 dependency** | The pipeline references `CapriKit.DirectX11` and produces GPU resources itself. Fewer abstractions to understand and maintain. | +| Shaders | **Offline bytecode** | HLSL → DXBC at compile time. `CapriKit.Generators.HLSL` keeps generating the C#-side metadata (structs, input element descriptions, entry points) unchanged; the pipeline only takes over bytecode compilation. | + +## Big Picture + +``` + source tree compiled tree runtime + (loose files) (loose files, mirrored) + ┌──────────────┐ compile ┌──────────────┐ load ┌───────────────┐ + │ grass.png │ ─────────► │ grass.cka │ ─────────► │ Texture2D │ + │ basic.hlsl │ IAsset- │ basic.cka │ IAsset- │ VertexShader..│ + │ tree.gltf │ Compiler │ tree.cka │ Loader │ Model │ + │ click.wav │ │ click.cka │ │ AudioClip │ + └──────────────┘ └──────────────┘ └───────────────┘ + dev machine only dev cache == ship content game process +``` + +Two phases, two families of pluggable pieces: + +- **Compilers** (dev machine / build server): source format → compiled container. Slow, thorough, offline. +- **Loaders** (game process): compiled container → ready-to-use object (texture on the GPU, playable audio clip). Fast, allocation-conscious. + +An asset is identified by its **source-relative path**, and every source file maps 1:1 to +one compiled file with the same relative path (different extension). This 1:1 rule is what +keeps identity, staleness checks, and file watching trivial. + +## Phase 1: Compiling + +```csharp +public interface IAssetCompiler +{ + IReadOnlySet SupportedExtensions { get; } + int Version { get; } + + // Reads source via a *tracking* file system so every file touched + // (e.g. #included HLSL) is recorded as an input of this asset. + void Compile(AssetId id, ITrackingReadOnlyFileSystem source, Stream output); +} +``` + +### Compiled container (`.cka` — "CapriKit Asset") + +One small binary envelope shared by all types, so staleness and loading logic is written once: + +- Header: magic, container version, compiler type + version, hashes of all input files. +- Body: one or more named blobs (a KTX2 payload; a DXBC blob per shader entry point; vertex + index buffers). + +### Staleness & the hybrid model + +An asset is stale when the compiled file is missing, or the stored input hashes / compiler +version no longer match. The check is identical everywhere; only *who runs it* differs: + +- **Dev**: the game process checks staleness when an asset is first requested and recompiles + inline (on the worker thread that is loading it) before loading. First run is slow, then it's cache hits. +- **Ship**: the CLI tool walks the source tree, compiles everything stale, and the resulting + compiled tree *is* the shipped content folder. Shipped builds skip the staleness check entirely + and never reference the compiler assemblies. + +### Per-type mapping + +| Type | Source | Compile step | Compiled body | +|---|---|---|---| +| Texture | png, jpg, ... | `CapriKit.SuperCompressed` encoder (mips baked in) | KTX2 | +| Shader | hlsl | DXC/FXC per `#pragma`-marked entry point | DXBC blob per entry point | +| Model | tbd (gltf?) | parse, triangulate, build interleaved buffers | vertex + index blobs | +| Audio | wav, ogg | tbd (likely near pass-through) | PCM or compressed blob | + +Note that input-file tracking (an `.hlsl` including `common.hlsli`) is a *build* dependency, +not a runtime asset reference — it exists only so staleness and hot reload know that touching +`common.hlsli` dirties every shader that included it. The "flat assets" decision is untouched. + +## Phase 2: Loading + +### Identity and typed access + +```csharp +public sealed record Asset(string Path); // relative path, forward slashes, lower case + +public static class GameAssets // game code declares what it uses +{ + public static readonly Asset Grass = new("textures/grass.png"); + // snip +} +``` + +Per assumption 1, interpretation (sRGB vs linear, transcode target, ...) is supplied by the +caller **at request time** as an optional per-type options record; nothing is persisted. + +### Bundles and background loading + +This generalizes the pattern already proven in `CapriKit.Tests.Tool/Program.cs`: +`BackgroundWorker` jobs write results (or one exception) into a `LightweightChannel`, +and the main loop drains it once per frame. + +```csharp +var bundle = assets.StartLoading([GameAssets.Grass, GameAssets.Tree /* snip */]); + +// in the main loop, once per frame: +assets.Update(); // drains channels, publishes finished assets, applies hot reloads + +if (bundle.IsLoaded) // all jobs finished (faulted bundles rethrow here) +{ + AssetRef grass = bundle.Get(GameAssets.Grass); +} +``` + +`bundle.Progress` (loaded/total) falls out for free for loading screens. + +### Threading: answering "do I need two steps?" (brief 3b) + +Mostly no, on D3D11. `ID3D11Device` is **free-threaded**: creating textures, buffers, and +shaders — including uploading initial data — is legal from any thread. Only the +**immediate `DeviceContext`** is single-threaded. Since mips are baked offline, nothing in +the current asset set needs the context at load time, so a worker thread can hand back a +fully GPU-resident texture. + +The main-thread coordination point still exists, but it is just `assets.Update()` draining +the channel: finished objects become *visible* to game code only on the main thread, which is +also the only place the registry is ever mutated — no locks anywhere. If a future asset type +does need context work (e.g. runtime `GenerateMips`), it can queue an action that `Update()` +executes; that seam costs nothing today. + +### Indirection for hot reload (3d) + +The registry maps `Asset` → `AssetRef`, created once and permanent (assumption 2: +loaded stays loaded, so no lifetime/refcount management at all). + +```csharp +public sealed class AssetRef +{ + public T Value { get; } // a field read, no lookup cost per use + public int Version { get; } // bumped on hot reload + // snip: internal setter used by the registry during Update() +} +``` + +Game code holds the `AssetRef` and reads `Value` when used. Derived objects (an input +layout built from a vertex shader blob) either poll `Version` or subscribe to a sparse +`Changed` event — needed rarely, mostly for shaders. + +## Hot Reload (dev only) + +``` +FileSystemWatcher (source tree) + → debounce (editors save in bursts) + → look up which assets have that file as an input (from stored input lists) + → recompile + reload on a worker, same code path as a normal load + → assets.Update() swaps AssetRef.Value on the main thread, bumps Version + → old object disposed (D3D11 keeps GPU resources alive while in flight, so this is safe) +``` + +Failures (syntax error in a shader) write the exception to the channel; `Update()` reports +it and keeps the old value live — a typo never kills the running game. + +## Project Layout + +| Project | Contents | Referenced by | +|---|---|---| +| `CapriKit.Assets` | `Asset`, registry, bundles, loaders, hot-reload swap. Refs `DirectX11`, `SuperCompressed` (transcode), `Concurrency`, `IO`. | game, always | +| `CapriKit.Assets.Compilers` | `IAssetCompiler` implementations, staleness, watcher. Refs `SuperCompressed` (encode), shader compiler. | game in dev builds; CLI tool | +| `CapriKit.Assets.Tool` | thin CLI over Compilers for the shipping build step | build scripts | + +Boilerplate budget per new asset type: **one compiler class + one loader class** +(MiniEngine3 needed processor + content wrapper + settings + serialization per type; the +shared container format and the no-settings/no-lifetime assumptions eliminate the rest). + +## Deliberately Out of Scope + +- Unloading / reference counting (assumption 2). +- Cross-asset references and cascading reloads (flat-assets decision). +- Per-asset metadata files (assumption 1). +- Packed archives — see appendix. + +## Risks & Open Points + +1. **Encoder settings vs assumption 1.** Normal maps and albedo textures ideally encode + differently (UASTC vs ETC1S, linear vs perceptual metrics). If one default proves + insufficient, a filename convention (`*_n.png`) is the escape hatch that doesn't violate + "no metadata files". +2. **Model and audio source formats** are undecided; the architecture only assumes "some + compiler produces blobs". Worth a separate research note before implementing those compilers. +3. **Input tracking in `CapriKit.IO`.** Compilers need a tracking read-only file system + (MiniEngine3's `TrackingVirtualFileSystem` is the precedent); check what `CapriKit.IO` + is missing. +4. **Native binary size.** The SuperCompressed native DLL contains encoder *and* transcoder; + shipped games carry encoder code they never call. Acceptable for now, splittable later. + +## Appendix: Packed Archives (not now, maybe later) + +Skipped because loose files make hot reload, staleness, and debugging trivially simple, and +the classic motivations (seek latency, file-handle overhead) matter little on modern SSDs. +Remaining real benefits: fewer/smaller patch artifacts, whole-archive compression, and light +obfuscation. + +The door stays open cheaply: loaders read compiled containers through +`IReadOnlyVirtualFileSystem`, so a future `PackedFileSystem` implementation (one archive = +one mounted file system) would slot in without touching any pipeline code. If it ever +happens, pack as a *post-step* over the compiled tree so the compile pipeline never knows. + +## Remark: uploading textures from a worker thread + +`UpdateSubresource`/`Map` need the immediate context (main thread), but they are only for +writing into a resource that *already exists* (`Default`/`Dynamic` usage). Asset textures +instead pass their pixels as initial data to `ID3D11Device::CreateTexture2D`, which is +free-threaded: one `SubresourceData` entry per subresource (mip × array slice), copied by +the driver *during* the create call. Use `ResourceUsage.Immutable` — it requires initial +data at creation, forbids later updates, and lets the driver optimize. + +```csharp +var subresources = new SubresourceData[mipCount]; +for (var mip = 0; mip < mipCount; mip++) +{ + // for BC formats: rowPitch = Math.Max(1, (mipWidth + 3) / 4) * bytesPerBlock + subresources[mip] = new SubresourceData(pointerToMipData, rowPitch); +} +var desc = new Texture2DDescription(format, width, height, mipLevels: mipCount, /* snip */ usage: ResourceUsage.Immutable); +var texture = device.CreateTexture2D(desc, subresources); +// pointers only need to stay pinned until CreateTexture2D returns — the copy is synchronous +``` + +Caveats: free-threaded means thread-safe, not necessarily concurrent — without +`DriverConcurrentCreates` (see `CheckFeatureSupport(D3D11_FEATURE_THREADING)`) creates +serialize on a device-wide lock, which is still correct. And none of this holds if the +device were created with `D3D11_CREATE_DEVICE_SINGLETHREADED` (ours is not, see `Device.cs`). diff --git a/source/CapriKit.AssetPipeline.DirectX11/CapriKit.AssetPipeline.DirectX11.csproj b/source/CapriKit.AssetPipeline.DirectX11/CapriKit.AssetPipeline.DirectX11.csproj new file mode 100644 index 0000000..c02ac0d --- /dev/null +++ b/source/CapriKit.AssetPipeline.DirectX11/CapriKit.AssetPipeline.DirectX11.csproj @@ -0,0 +1,11 @@ + + + $(TargetFramework)-windows + + + + + + + + diff --git a/source/CapriKit.AssetPipeline.DirectX11/Shaders/ShaderTranscoder.cs b/source/CapriKit.AssetPipeline.DirectX11/Shaders/ShaderTranscoder.cs new file mode 100644 index 0000000..2891395 --- /dev/null +++ b/source/CapriKit.AssetPipeline.DirectX11/Shaders/ShaderTranscoder.cs @@ -0,0 +1,29 @@ +using CapriKit.DirectX11.Resources.Shaders; +using CapriKit.IO.Buffers; +using System.Buffers; +using System.Collections.Frozen; + +namespace CapriKit.AssetPipeline.DirectX11.Shaders; + +internal static class ShaderTranscoder +{ + public static IReadOnlySet SupportedExtensions { get; } = new HashSet([".hlsl"]).ToFrozenSet(); + + public static void WriteCommon(ShaderByteCode shader, IBufferWriter writer) + { + writer.Write(shader.EntryPoint); + writer.Write(shader.Name); + writer.Write(shader.Bytes.Length); + writer.Write(shader.Bytes); + } + + public static ShaderByteCode ReadCommon(ref SequenceReader reader) + { + var entryPoint = reader.ReadString(); + var name = reader.ReadString(); + var length = reader.ReadInt32(); + var bytes = reader.ReadBytes(length); + + return new ShaderByteCode(bytes, entryPoint, name); + } +} diff --git a/source/CapriKit.AssetPipeline.DirectX11/Shaders/VertexShaderTranscoder.cs b/source/CapriKit.AssetPipeline.DirectX11/Shaders/VertexShaderTranscoder.cs new file mode 100644 index 0000000..1aef183 --- /dev/null +++ b/source/CapriKit.AssetPipeline.DirectX11/Shaders/VertexShaderTranscoder.cs @@ -0,0 +1,32 @@ +using CapriKit.DirectX11; +using CapriKit.DirectX11.Resources.Shaders; +using CapriKit.IO; +using System.Buffers; + +namespace CapriKit.AssetPipeline.DirectX11.Shaders; + +public sealed class VertexShaderTranscoder(Device device) : IAssetEncoder, IAssetDecoder +{ + public IReadOnlySet SupportedExtensions => ShaderTranscoder.SupportedExtensions; + public Guid Id => Guid.Parse("{CA3CB37D-9880-4B61-AB09-EBC17E7533E6}"); + public int Version => 1; + + public async Task Encode(AssetId id, IReadOnlyVirtualFileSystem fileSystem, IBufferWriter writer) + { + var source = await fileSystem.ReadAllText(id.Path); + var includePath = id.Path.Directory; + var bytes = ShaderCompiler.CompileVertexShader(fileSystem, includePath, source, id.Key, id.ToString()); + ShaderTranscoder.WriteCommon(bytes.Common, writer); + } + + public IVertexShader Decode(AssetId id, ref SequenceReader reader) + { + var common = ShaderTranscoder.ReadCommon(ref reader); + return ShaderCompiler.CreateVertexShader(new VertexShaderByteCode(common), device); + } + + public void HotSwap(IVertexShader instance, IVertexShader replacement) + { + instance.HotSwap(replacement); + } +} diff --git a/source/CapriKit.AssetPipeline/Asset.cs b/source/CapriKit.AssetPipeline/Asset.cs new file mode 100644 index 0000000..5cbcfbb --- /dev/null +++ b/source/CapriKit.AssetPipeline/Asset.cs @@ -0,0 +1,5 @@ +using CapriKit.IO; + +namespace CapriKit.AssetPipeline; + +internal sealed record Asset(T Value, IReadOnlySet Dependencies); diff --git a/source/CapriKit.AssetPipeline/AssetDecoder.cs b/source/CapriKit.AssetPipeline/AssetDecoder.cs new file mode 100644 index 0000000..952fed5 --- /dev/null +++ b/source/CapriKit.AssetPipeline/AssetDecoder.cs @@ -0,0 +1,105 @@ +using CapriKit.IO; +using CapriKit.IO.Buffers; +using System.Buffers; + +using static CapriKit.AssetPipeline.AssetUtilities; + +namespace CapriKit.AssetPipeline; + +public interface IAssetDecoder +{ + Guid Id { get; } + int Version { get; } + +} + +public interface IAssetDecoder : IAssetDecoder +{ + // Synchronous by design: the envelope owns all file IO and hands the decoder an + // in-memory payload. The reader's buffer is only valid for the duration of the call, + // decoders must copy out anything they want to keep. + TAsset Decode(AssetId id, ref SequenceReader reader); + void HotSwap(TAsset instance, TAsset replacement); +} + +internal static class AssetDecoder +{ + public static async Task> Decode(AssetId id, IVirtualFileSystem fileSystem, IAssetDecoder decoder) + { + var inputPath = ToEncodedFilePath(id.Path); + ThrowOnFileNotFound(inputPath, fileSystem); + + using var input = fileSystem.OpenRead(inputPath); + + var payloadLength = await ReadHeader(input, decoder, inputPath); + var asset = await ReadPayload(input, payloadLength, decoder, id); + var dependencies = await ReadDependencies(input); + + return new Asset(asset, dependencies); + } + + private static async Task ReadHeader(Stream input, IAssetDecoder decoder, FilePath path) + { + // Encoder id (16 bytes) + encoder version (4 bytes) + payload length (4 bytes) + const int HeaderSizeInBytes = 24; + var buffer = ArrayPool.Shared.Rent(HeaderSizeInBytes); + try + { + await input.ReadExactlyAsync(buffer.AsMemory(0, HeaderSizeInBytes)); + var reader = SequenceReaders.Create(buffer, 0, HeaderSizeInBytes); + var id = reader.ReadGuid(); + var version = reader.ReadInt32(); + var payloadLength = reader.ReadInt32(); + + if (id != decoder.Id || version != decoder.Version) + { + throw new InvalidDataException( + $"Cannot decode {path}, it was encoded by {id} v{version} but the decoder is {decoder.Id} v{decoder.Version}"); + } + + return payloadLength; + } + finally + { + ArrayPool.Shared.Return(buffer); + } + } + + private static async Task ReadPayload(Stream input, int payloadLength, IAssetDecoder decoder, AssetId id) + { + var buffer = ArrayPool.Shared.Rent(payloadLength); + try + { + await input.ReadExactlyAsync(buffer.AsMemory(0, payloadLength)); + var reader = SequenceReaders.Create(buffer, 0, payloadLength); + return decoder.Decode(id, ref reader); + } + finally + { + ArrayPool.Shared.Return(buffer); + } + } + + private static async Task> ReadDependencies(Stream input) + { + var length = (int)(input.Length - input.Position); + var buffer = ArrayPool.Shared.Rent(length); + try + { + await input.ReadExactlyAsync(buffer.AsMemory(0, length)); + var reader = SequenceReaders.Create(buffer, 0, length); + + var count = reader.ReadInt32(); + var dependencies = new HashSet(count); + for (var i = 0; i < count; i++) + { + dependencies.Add(reader.ReadString()); + } + return dependencies; + } + finally + { + ArrayPool.Shared.Return(buffer); + } + } +} diff --git a/source/CapriKit.AssetPipeline/AssetEncoder.cs b/source/CapriKit.AssetPipeline/AssetEncoder.cs new file mode 100644 index 0000000..735ec54 --- /dev/null +++ b/source/CapriKit.AssetPipeline/AssetEncoder.cs @@ -0,0 +1,61 @@ +using CapriKit.IO; +using CapriKit.IO.Buffers; +using System.Buffers; +using System.IO.Pipelines; +using static CapriKit.AssetPipeline.AssetUtilities; + +namespace CapriKit.AssetPipeline; + +public interface IAssetEncoder +{ + IReadOnlySet SupportedExtensions { get; } + Guid Id { get; } + int Version { get; } + + Task Encode(AssetId id, IReadOnlyVirtualFileSystem fileSystem, IBufferWriter writer); +} + +internal sealed class AssetEncoder +{ + public static async Task Encode(AssetId id, IVirtualFileSystem fileSystem, IAssetEncoder encoder) + { + ThrowOnFileNotFound(id.Path, fileSystem); + var outputPath = ToEncodedFilePath(id.Path); + + using var output = fileSystem.CreateReadWrite(outputPath); + var writer = PipeWriter.Create(output); + var spy = fileSystem.SpyOn(); + + WriteHeader(writer, encoder); // payload length is written in WritePayload + await WritePayload(id, encoder, writer, spy); + WriteDependencies(spy, writer); + + await writer.FlushAsync(); + await writer.CompleteAsync(); + } + + private static void WriteHeader(PipeWriter writer, IAssetEncoder encoder) + { + writer.Write(encoder.Id); + writer.Write(encoder.Version); + } + + private static async Task WritePayload(AssetId id, IAssetEncoder encoder, PipeWriter writer, VirtualFileSystemSpy spy) + { + var payload = new ArrayBufferWriter(); + await encoder.Encode(id, spy, payload); + writer.Write(payload.WrittenCount); + writer.Write(payload.WrittenSpan); + } + + private static void WriteDependencies(VirtualFileSystemSpy spy, PipeWriter writer) + { + writer.Write(spy.OpenedFiles.Count); + foreach (var dependency in spy.OpenedFiles) + { + writer.Write(dependency); + } + } + + +} diff --git a/source/CapriKit.AssetPipeline/AssetId.cs b/source/CapriKit.AssetPipeline/AssetId.cs new file mode 100644 index 0000000..d41a2ec --- /dev/null +++ b/source/CapriKit.AssetPipeline/AssetId.cs @@ -0,0 +1,5 @@ +using CapriKit.IO; + +namespace CapriKit.AssetPipeline; + +public record AssetId(string Key, FilePath Path); diff --git a/source/CapriKit.AssetPipeline/AssetManager.cs b/source/CapriKit.AssetPipeline/AssetManager.cs new file mode 100644 index 0000000..5349312 --- /dev/null +++ b/source/CapriKit.AssetPipeline/AssetManager.cs @@ -0,0 +1,30 @@ +namespace CapriKit.AssetPipeline; + +public class AssetManager +{ + + private readonly Dictionary Encoders = []; + private readonly Dictionary Decoders = []; + + public void RegisterTranscoder(IAssetEncoder encoder, IAssetDecoder decoder) + { + var typeKey = typeof(T); + Encoders[typeKey] = encoder; + Decoders[typeKey] = decoder; + } + + public T Decode(AssetId id) + { + var typeKey = typeof(T); + var encoder = Encoders[typeKey]; + var decoder = Decoders[typeKey]; + + var extension = new string(id.Path.Extension); + if (encoder.SupportedExtensions.Contains(extension)) + { + + } + + throw new NotImplementedException(); + } +} diff --git a/source/CapriKit.AssetPipeline/AssetUtilities.cs b/source/CapriKit.AssetPipeline/AssetUtilities.cs new file mode 100644 index 0000000..5999b10 --- /dev/null +++ b/source/CapriKit.AssetPipeline/AssetUtilities.cs @@ -0,0 +1,19 @@ +using CapriKit.IO; + +namespace CapriKit.AssetPipeline; + +internal static class AssetUtilities +{ + public static FilePath ToEncodedFilePath(FilePath path) + { + return path + ".cka"; + } + + public static void ThrowOnFileNotFound(FilePath path, IVirtualFileSystem fileSystem) + { + if (!fileSystem.Exists(path)) + { + throw new FileNotFoundException(null, path); + } + } +} diff --git a/source/CapriKit.AssetPipeline/CapriKit.AssetPipeline.csproj b/source/CapriKit.AssetPipeline/CapriKit.AssetPipeline.csproj new file mode 100644 index 0000000..4e4824a --- /dev/null +++ b/source/CapriKit.AssetPipeline/CapriKit.AssetPipeline.csproj @@ -0,0 +1,5 @@ + + + + + diff --git a/source/CapriKit.DirectX11/Resources/Shaders/ComputeShader.cs b/source/CapriKit.DirectX11/Resources/Shaders/ComputeShader.cs index e615f9b..581c23a 100644 --- a/source/CapriKit.DirectX11/Resources/Shaders/ComputeShader.cs +++ b/source/CapriKit.DirectX11/Resources/Shaders/ComputeShader.cs @@ -4,7 +4,22 @@ namespace CapriKit.DirectX11.Resources.Shaders; public interface IComputeShader : IDisposable { - internal ID3D11ComputeShader ID3D11ComputeShader { get; } + internal uint NumThreadsX { get; set; } + internal uint NumThreadsY { get; set; } + internal uint NumThreadsZ { get; set; } + internal ID3D11ComputeShader ID3D11ComputeShader { get; set; } + + + public void HotSwap(IComputeShader replacement) + { + NumThreadsX = replacement.NumThreadsX; + NumThreadsY = replacement.NumThreadsY; + NumThreadsZ = replacement.NumThreadsZ; + + var oldShader = ID3D11ComputeShader; + ID3D11ComputeShader = replacement.ID3D11ComputeShader; + oldShader.Dispose(); + } /// /// The shader kernel defines how big the groups for each dimension are. @@ -16,27 +31,49 @@ public interface IComputeShader : IDisposable internal sealed class ComputeShader : IComputeShader { - private readonly uint NumThreadsX; - private readonly uint NumThreadsY; - private readonly uint NumThreadsZ; - private readonly ID3D11ComputeShader Shader; + private uint numThreadsX; + private uint numThreadsY; + private uint numThreadsZ; + private ID3D11ComputeShader shader; internal ComputeShader(ID3D11ComputeShader shader, uint numThreadsX, uint numThreadsY, uint numThreadsZ) { - Shader = shader; - NumThreadsX = numThreadsX; - NumThreadsY = numThreadsY; - NumThreadsZ = numThreadsZ; + this.shader = shader; + this.numThreadsX = numThreadsX; + this.numThreadsY = numThreadsY; + this.numThreadsZ = numThreadsZ; + } + + ID3D11ComputeShader IComputeShader.ID3D11ComputeShader + { + get { return shader; } + set { shader = value; } } - ID3D11ComputeShader IComputeShader.ID3D11ComputeShader => Shader; + uint IComputeShader.NumThreadsX + { + get { return numThreadsX; } + set { numThreadsX = value; } + } + + uint IComputeShader.NumThreadsY + { + get { return numThreadsY; } + set { numThreadsY = value; } + } + + uint IComputeShader.NumThreadsZ + { + get { return numThreadsZ; } + set { numThreadsZ = value; } + } /// public (uint X, uint Y, uint Z) GetDispatchSize(uint dimX, uint dimY, uint dimZ) { - var x = GetDispatchSize(NumThreadsX, dimX); - var y = GetDispatchSize(NumThreadsY, dimY); - var z = GetDispatchSize(NumThreadsZ, dimZ); + var x = GetDispatchSize(numThreadsX, dimX); + var y = GetDispatchSize(numThreadsY, dimY); + var z = GetDispatchSize(numThreadsZ, dimZ); return (x, y, z); } @@ -48,6 +85,6 @@ private static uint GetDispatchSize(uint numThreads, uint dim) public void Dispose() { - Shader.Dispose(); + shader.Dispose(); } } diff --git a/source/CapriKit.DirectX11/Resources/Shaders/PixelShader.cs b/source/CapriKit.DirectX11/Resources/Shaders/PixelShader.cs index f0146e4..8eae3f1 100644 --- a/source/CapriKit.DirectX11/Resources/Shaders/PixelShader.cs +++ b/source/CapriKit.DirectX11/Resources/Shaders/PixelShader.cs @@ -4,20 +4,30 @@ namespace CapriKit.DirectX11.Resources.Shaders; public interface IPixelShader : IDisposable { - internal ID3D11PixelShader ID3D11PixelShader { get; } + internal ID3D11PixelShader ID3D11PixelShader { get; set; } + + public void HotSwap(IPixelShader replacement) + { + var oldShader = ID3D11PixelShader; + ID3D11PixelShader = replacement.ID3D11PixelShader; + oldShader.Dispose(); + } } internal sealed class PixelShader : IPixelShader { - private readonly ID3D11PixelShader Shader; + private ID3D11PixelShader Shader; internal PixelShader(ID3D11PixelShader shader) { Shader = shader; } - ID3D11PixelShader IPixelShader.ID3D11PixelShader => Shader; - + ID3D11PixelShader IPixelShader.ID3D11PixelShader + { + get { return Shader; } + set { Shader = value; } + } public void Dispose() { Shader.Dispose(); diff --git a/source/CapriKit.DirectX11/Resources/Shaders/ShaderCompiler.cs b/source/CapriKit.DirectX11/Resources/Shaders/ShaderCompiler.cs index a400951..5ea5b9f 100644 --- a/source/CapriKit.DirectX11/Resources/Shaders/ShaderCompiler.cs +++ b/source/CapriKit.DirectX11/Resources/Shaders/ShaderCompiler.cs @@ -5,13 +5,29 @@ namespace CapriKit.DirectX11.Resources.Shaders; -public abstract record ShaderByteCode(byte[] Bytes, string EntryPoint, string Name); +public record ShaderByteCode(byte[] Bytes, string EntryPoint, string Name); -public sealed record VertexShaderByteCode(byte[] Bytes, string EntryPoint, string Name) : ShaderByteCode(Bytes, EntryPoint, Name); +public sealed record VertexShaderByteCode(ShaderByteCode Common) +{ + public byte[] Bytes => Common.Bytes; + public string EntryPoint => Common.EntryPoint; + public string Name => Common.Name; +} -public sealed record PixelShaderByteCode(byte[] Bytes, string EntryPoint, string Name) : ShaderByteCode(Bytes, EntryPoint, Name); +public sealed record PixelShaderByteCode(ShaderByteCode Common) +{ + public byte[] Bytes => Common.Bytes; + public string EntryPoint => Common.EntryPoint; + public string Name => Common.Name; +} -public sealed record ComputeShaderByteCode(byte[] Bytes, uint NumThreadsX, uint NumThreadsY, uint NumThreadsZ, string EntryPoint, string Name) : ShaderByteCode(Bytes, EntryPoint, Name); + +public sealed record ComputeShaderByteCode(ShaderByteCode Common, uint NumThreadsX, uint NumThreadsY, uint NumThreadsZ) +{ + public byte[] Bytes => Common.Bytes; + public string EntryPoint => Common.EntryPoint; + public string Name => Common.Name; +} public static class ShaderCompiler { @@ -21,14 +37,14 @@ public static class ShaderCompiler public static IVertexShader CompileVertexShader(IReadOnlyVirtualFileSystem fileSystem, DirectoryPath includePath, Device device, string source, string entryPoint, string name) { - var byteCode = CompileVertexShader(fileSystem, includePath, source, entryPoint, name); - return CreateVertexShader(byteCode, device); + var bytes = CompileVertexShader(fileSystem, includePath, source, entryPoint, name); + return CreateVertexShader(bytes, device); } public static VertexShaderByteCode CompileVertexShader(IReadOnlyVirtualFileSystem fileSystem, DirectoryPath includePath, string source, string entryPoint, string name) { - var bytes = Compile(fileSystem, includePath, source, entryPoint, name, VERTEX_SHADER_PROFILE); - return new VertexShaderByteCode(bytes, entryPoint, name); + var byteCode = Compile(fileSystem, includePath, source, entryPoint, name, VERTEX_SHADER_PROFILE); + return new VertexShaderByteCode(byteCode); } public static IVertexShader CreateVertexShader(VertexShaderByteCode byteCode, Device device) @@ -47,7 +63,7 @@ public static IPixelShader CompilePixelShader(IReadOnlyVirtualFileSystem fileSys public static PixelShaderByteCode CompilePixelShader(IReadOnlyVirtualFileSystem fileSystem, DirectoryPath includePath, string source, string entryPoint, string name) { var bytes = Compile(fileSystem, includePath, source, entryPoint, name, PIXEL_SHADER_PROFILE); - return new PixelShaderByteCode(bytes, entryPoint, name); + return new PixelShaderByteCode(bytes); } public static IPixelShader CreatePixelShader(PixelShaderByteCode byteCode, Device device) { @@ -58,15 +74,15 @@ public static IPixelShader CreatePixelShader(PixelShaderByteCode byteCode, Devic public static IComputeShader CompileComputeShader(IReadOnlyVirtualFileSystem fileSystem, DirectoryPath includePath, Device device, string source, string entryPoint, string name) { - var byteCode = CompileComputeShader(fileSystem, includePath, source, entryPoint, name); - return CreateComputeShader(byteCode, device); + var common = CompileComputeShader(fileSystem, includePath, source, entryPoint, name); + return CreateComputeShader(common, device); } public static ComputeShaderByteCode CompileComputeShader(IReadOnlyVirtualFileSystem fileSystem, DirectoryPath includePath, string source, string entryPoint, string name) { - var bytes = Compile(fileSystem, includePath, source, entryPoint, name, COMPUTE_SHADER_PROFILE); - var (x, y, z) = QueryNumThreads(bytes, name); - return new ComputeShaderByteCode(bytes, x, y, z, entryPoint, name); + var common = Compile(fileSystem, includePath, source, entryPoint, name, COMPUTE_SHADER_PROFILE); + var (x, y, z) = QueryNumThreads(common.Bytes, name); + return new ComputeShaderByteCode(common, x, y, z); } public static IComputeShader CreateComputeShader(ComputeShaderByteCode byteCode, Device device) @@ -76,7 +92,7 @@ public static IComputeShader CreateComputeShader(ComputeShaderByteCode byteCode, return new ComputeShader(shader, byteCode.NumThreadsX, byteCode.NumThreadsY, byteCode.NumThreadsZ); } - private static byte[] Compile(IReadOnlyVirtualFileSystem fileSystem, DirectoryPath includePath, string source, string entryPoint, string name, string profile) + private static ShaderByteCode Compile(IReadOnlyVirtualFileSystem fileSystem, DirectoryPath includePath, string source, string entryPoint, string name, string profile) { using var includeResolver = new ShaderIncludeResolver(fileSystem, includePath); @@ -92,7 +108,7 @@ private static byte[] Compile(IReadOnlyVirtualFileSystem fileSystem, DirectoryPa var bytes = blob.AsBytes(); blob.Dispose(); - return bytes; + return new ShaderByteCode(bytes, entryPoint, name); } /// diff --git a/source/CapriKit.DirectX11/Resources/Shaders/VertexShader.cs b/source/CapriKit.DirectX11/Resources/Shaders/VertexShader.cs index 39ccfb7..31e9fa4 100644 --- a/source/CapriKit.DirectX11/Resources/Shaders/VertexShader.cs +++ b/source/CapriKit.DirectX11/Resources/Shaders/VertexShader.cs @@ -4,32 +4,52 @@ namespace CapriKit.DirectX11.Resources.Shaders; public interface IVertexShader : IDisposable { - internal ID3D11VertexShader ID3D11VertexShader { get; } + internal byte[] Blob { get; set; } + internal ID3D11VertexShader ID3D11VertexShader { get; set; } - IInputLayout CreateInputLayout(Device device, InputElementDescription[] elements); // TODO: How to do this without exposing types from Vortice? + IInputLayout CreateInputLayout(Device device, InputElementDescription[] elements); + + public void HotSwap(IVertexShader replacement) + { + Blob = replacement.Blob; + + var oldShader = ID3D11VertexShader; + ID3D11VertexShader = replacement.ID3D11VertexShader; + oldShader.Dispose(); + } } internal sealed class VertexShader : IVertexShader { - private readonly byte[] Blob; - private readonly ID3D11VertexShader Shader; + private byte[] blob; + private ID3D11VertexShader shader; internal VertexShader(byte[] blob, ID3D11VertexShader shader) { - Blob = blob; - Shader = shader; + this.blob = blob; + this.shader = shader; } - ID3D11VertexShader IVertexShader.ID3D11VertexShader => Shader; + byte[] IVertexShader.Blob + { + get { return blob; } + set { blob = value; } + } + + ID3D11VertexShader IVertexShader.ID3D11VertexShader + { + get { return shader; } + set { shader = value; } + } public IInputLayout CreateInputLayout(Device device, InputElementDescription[] elements) { - var inputLayout = device.ID3D11Device.CreateInputLayout(elements, Blob); + var inputLayout = device.ID3D11Device.CreateInputLayout(elements, blob); return new InputLayout(inputLayout); } public void Dispose() { - Shader.Dispose(); + shader.Dispose(); } } diff --git a/source/CapriKit.IO/Buffers/IBufferWriterExtensions.cs b/source/CapriKit.IO/Buffers/IBufferWriterExtensions.cs new file mode 100644 index 0000000..f84b3f6 --- /dev/null +++ b/source/CapriKit.IO/Buffers/IBufferWriterExtensions.cs @@ -0,0 +1,60 @@ +using System.Buffers; +using System.Buffers.Binary; +using System.Runtime.CompilerServices; +using System.Text; + +namespace CapriKit.IO.Buffers; + +public static class BufferWriterExtensions +{ + /// + /// Writes a length prefixed string. The prefix is the byte count of the encoded string as a + /// 7 bit encoded integer, identical to + /// + /// The writer to write the string to + /// The string to write + /// Defaults to UTF8 + public static void Write(this IBufferWriter writer, string value, Encoding? encoding = null) + { + encoding = encoding ?? Encoding.UTF8; + var length = encoding.GetByteCount(value); + writer.Write7BitEncodedInt(length); + encoding.GetBytes(value, writer); + } + + /// + /// Writes an integer seven bits at a time, using the eighth bit to indicate that another byte + /// follows. The format is identical to + /// + public static void Write7BitEncodedInt(this IBufferWriter writer, int value) + { + const int MaxLengthInBytes = 5; // ceil(32 bits / 7 bits per byte) + + var span = writer.GetSpan(MaxLengthInBytes); + var written = 0; + + var uValue = (uint)value; + while (uValue > 0x7Fu) + { + span[written++] = (byte)(uValue | ~0x7Fu); + uValue >>= 7; + } + span[written++] = (byte)uValue; + + writer.Advance(written); + } + + public static void Write(this IBufferWriter writer, int value) + { + var span = writer.GetSpan(sizeof(int)); + BinaryPrimitives.WriteInt32LittleEndian(span, value); + writer.Advance(sizeof(int)); + } + + public static void Write(this IBufferWriter writer, Guid guid) + { + var span = writer.GetSpan(Unsafe.SizeOf()); + guid.TryWriteBytes(span, false, out var written); + writer.Advance(written); + } +} diff --git a/source/CapriKit.IO/Buffers/SequenceReaderExtensions.cs b/source/CapriKit.IO/Buffers/SequenceReaderExtensions.cs new file mode 100644 index 0000000..3ea73b9 --- /dev/null +++ b/source/CapriKit.IO/Buffers/SequenceReaderExtensions.cs @@ -0,0 +1,112 @@ +using System.Buffers; +using System.Runtime.CompilerServices; +using System.Text; + +namespace CapriKit.IO.Buffers; + +public static class SequenceReaders +{ + public static SequenceReader Create(byte[] bytes, int start, int length) + { + var sequence = new ReadOnlySequence(bytes, start, length); + return new SequenceReader(sequence); + } +} + +public static class SequenceReaderExtensions +{ + + + /// + /// Reads a length prefixed string written by + /// . + /// The format is identical to so both can be mixed + /// freely, as long as both sides use the same encoding + /// + /// The reader to read the string from + /// Defaults to UTF8 + public static string ReadString(this ref SequenceReader reader, Encoding? encoding = null) + { + encoding = encoding ?? Encoding.UTF8; + var length = reader.Read7BitEncodedInt(); + if (!reader.TryReadExact(length, out var sequence)) + { + throw new EndOfStreamException(); + } + + return encoding.GetString(in sequence); + } + + /// + /// Reads an integer written seven bits at a time by + /// + /// or + /// + public static int Read7BitEncodedInt(this ref SequenceReader reader) + { + const int MaxBytesWithoutOverflow = 4; + + var result = 0u; + for (var shift = 0; shift < MaxBytesWithoutOverflow * 7; shift += 7) + { + var current = ReadByte(ref reader); + result |= (current & 0x7Fu) << shift; + if (current <= 0x7Fu) + { + return (int)result; + } + } + + // The fifth byte can only hold the 4 remaining bits of a 32 bit integer + var last = ReadByte(ref reader); + if (last > 0b_1111u) + { + throw new FormatException("Invalid 7 bit encoded integer"); + } + + result |= (uint)last << (MaxBytesWithoutOverflow * 7); + return (int)result; + } + + public static int ReadInt32(this ref SequenceReader reader) + { + if (!reader.TryReadLittleEndian(out int value)) + { + throw new EndOfStreamException(); + } + + return value; + } + + public static Guid ReadGuid(this ref SequenceReader reader) + { + Span bytes = stackalloc byte[Unsafe.SizeOf()]; + if (!reader.TryCopyTo(bytes)) + { + throw new EndOfStreamException(); + } + + reader.Advance(bytes.Length); + return new Guid(bytes, bigEndian: false); + } + + public static byte[] ReadBytes(this ref SequenceReader reader, int length) + { + if (!reader.TryReadExact(length, out var sequence)) + { + throw new EndOfStreamException(); + } + + return sequence.ToArray(); + } + + public static byte ReadByte(ref SequenceReader reader) + { + if (!reader.TryRead(out var value)) + { + throw new EndOfStreamException(); + } + + return value; + } +} diff --git a/source/CapriKit.IO/CapriKit.IO.csproj b/source/CapriKit.IO/CapriKit.IO.csproj index c369e11..2ae79d6 100644 --- a/source/CapriKit.IO/CapriKit.IO.csproj +++ b/source/CapriKit.IO/CapriKit.IO.csproj @@ -1,3 +1 @@ - - - + diff --git a/source/CapriKit.IO/FileSystem.cs b/source/CapriKit.IO/FileSystem.cs index 49ad138..750e919 100644 --- a/source/CapriKit.IO/FileSystem.cs +++ b/source/CapriKit.IO/FileSystem.cs @@ -5,7 +5,14 @@ public class FileSystem : IVirtualFileSystem public Stream AppendWrite(FilePath file) { var absoluteFile = FindOrThrow(file); - return absoluteFile.Open(FileMode.Append, FileAccess.Write, FileShare.Read); + var options = new FileStreamOptions() + { + Mode = FileMode.Append, + Access = FileAccess.Write, + Share = FileShare.Read, + Options = FileOptions.Asynchronous + }; + return new FileStream(absoluteFile.FullName, options); } public Stream CreateReadWrite(FilePath file) @@ -16,8 +23,14 @@ public Stream CreateReadWrite(FilePath file) var directory = absoluteFile.DirectoryName ?? string.Empty; Directory.CreateDirectory(directory); } - - return absoluteFile.Open(FileMode.Create, FileAccess.ReadWrite, FileShare.Read); + var options = new FileStreamOptions() + { + Mode = FileMode.Create, + Access = FileAccess.ReadWrite, + Share = FileShare.Read, + Options = FileOptions.Asynchronous + }; + return new FileStream(absoluteFile.FullName, options); } public void Delete(FilePath file) @@ -41,7 +54,14 @@ public DateTime LastWriteTime(FilePath file) public Stream OpenRead(FilePath file) { var absoluteFile = FindOrThrow(file); - return absoluteFile.Open(FileMode.Open, FileAccess.Read, FileShare.Read); + var options = new FileStreamOptions() + { + Mode = FileMode.Open, + Access = FileAccess.Read, + Share = FileShare.Read, + Options = FileOptions.Asynchronous + }; + return new FileStream(absoluteFile.FullName, options); } public long SizeInBytes(FilePath file) @@ -65,11 +85,6 @@ public IReadOnlyList List(DirectoryPath directory) return filePaths; } - public FileSystemEventListener Watch(DirectoryPath directory, bool includeSubDirectories = true) - { - return new FileSystemEventListener(this, directory, includeSubDirectories); - } - private FileInfo FindOrThrow(FilePath file) { var info = GetFileInfo(file); @@ -81,18 +96,18 @@ private FileInfo FindOrThrow(FilePath file) throw new FileNotFoundException(null, file.ToString()); } - internal FilePath GetFilePath(string path) + internal static FilePath GetFilePath(string path) { return new FilePath(path); } - internal FileInfo GetFileInfo(FilePath file) + internal static FileInfo GetFileInfo(FilePath file) { var absolutePath = file.IsAbsolute ? file : file.ToAbsolute(); return new FileInfo(absolutePath.ToString()); } - internal DirectoryInfo GetDirectoryInfo(DirectoryPath path) + internal static DirectoryInfo GetDirectoryInfo(DirectoryPath path) { var absolutePath = path.IsAbsolute ? path : path.ToAbsolute(); return new DirectoryInfo(absolutePath.ToString()); diff --git a/source/CapriKit.IO/FileSystemEventListener.cs b/source/CapriKit.IO/FileSystemEventListener.cs index edc9f0f..241928a 100644 --- a/source/CapriKit.IO/FileSystemEventListener.cs +++ b/source/CapriKit.IO/FileSystemEventListener.cs @@ -11,17 +11,13 @@ public enum FileSystemChangeKind public sealed class FileSystemEventListener : IDisposable { - private readonly FileSystem FileSystem; private readonly FileSystemWatcher Watcher; private event FileSystemEventHandler? onFileChanged; - public FileSystemEventListener(FileSystem fileSystem, DirectoryPath directory, bool includeSubDirectories = true) + public FileSystemEventListener(DirectoryPath directory, bool includeSubDirectories = true) { - FileSystem = fileSystem; Directory = directory; - - // Take the directory info since all file system types will point that to the true absolute path var directoryInfo = FileSystem.GetDirectoryInfo(directory); if (!directoryInfo.Exists) { diff --git a/source/CapriKit.Tests/AssetPipeline/AssetDecoderTests.cs b/source/CapriKit.Tests/AssetPipeline/AssetDecoderTests.cs new file mode 100644 index 0000000..dea0922 --- /dev/null +++ b/source/CapriKit.Tests/AssetPipeline/AssetDecoderTests.cs @@ -0,0 +1,25 @@ +using CapriKit.AssetPipeline; +using CapriKit.IO; + +namespace CapriKit.Tests.AssetPipeline; + +internal class AssetDecoderTests +{ + + [Test] + public async Task Decode() + { + var fileSystem = new InMemoryFileSystem(); + await fileSystem.WriteAllText("hello.txt", "héllo"); + var transcoder = new DummyTranscoder(); + var id = new AssetId("Main", "hello.txt"); + + await AssetEncoder.Encode(id, fileSystem, transcoder); + var envelope = await AssetDecoder.Decode(id, fileSystem, transcoder); + + FilePath expectedDependency = "hello.txt"; + await Assert.That(envelope.Value).IsEqualTo("HÉLLO"); + await Assert.That(envelope.Dependencies.Count).IsEqualTo(1); + await Assert.That(envelope.Dependencies.First()).IsEqualTo(expectedDependency); + } +} diff --git a/source/CapriKit.Tests/AssetPipeline/AssetEncoderTests.cs b/source/CapriKit.Tests/AssetPipeline/AssetEncoderTests.cs new file mode 100644 index 0000000..df49367 --- /dev/null +++ b/source/CapriKit.Tests/AssetPipeline/AssetEncoderTests.cs @@ -0,0 +1,36 @@ +using CapriKit.AssetPipeline; +using CapriKit.IO; +using CapriKit.IO.Buffers; +using System.Buffers; + +namespace CapriKit.Tests.AssetPipeline; + +internal class AssetEncoderTests +{ + [Test] + public async Task Encode() + { + var fileSystem = new InMemoryFileSystem(); + await fileSystem.WriteAllText("hello.txt", "héllo"); + var transcoder = new DummyTranscoder(); + var id = new AssetId("Main", "hello.txt"); + + await AssetEncoder.Encode(id, fileSystem, transcoder); + var bytes = await fileSystem.ReadAllBytes("hello.txt.cka"); + + var reader = new SequenceReader(new ReadOnlySequence(bytes)); + var encoderId = reader.ReadGuid(); + var encoderVersion = reader.ReadInt32(); + var payloadLength = reader.ReadInt32(); + reader.Advance(payloadLength); + var dependencyCount = reader.ReadInt32(); + var dependency = reader.ReadString(); + var end = reader.End; + + await Assert.That(encoderId).IsEqualTo(transcoder.Id); + await Assert.That(encoderVersion).IsEqualTo(transcoder.Version); + await Assert.That(dependencyCount).IsEqualTo(1); + await Assert.That(dependency).IsEqualTo("hello.txt"); + await Assert.That(end).IsTrue(); + } +} diff --git a/source/CapriKit.Tests/AssetPipeline/DummyTranscoder.cs b/source/CapriKit.Tests/AssetPipeline/DummyTranscoder.cs new file mode 100644 index 0000000..c29a769 --- /dev/null +++ b/source/CapriKit.Tests/AssetPipeline/DummyTranscoder.cs @@ -0,0 +1,32 @@ +using CapriKit.AssetPipeline; +using CapriKit.IO; +using CapriKit.IO.Buffers; +using System.Buffers; + +namespace CapriKit.Tests.AssetPipeline; + +/// +/// Uppercases a text file +/// +internal sealed class DummyTranscoder : IAssetEncoder, IAssetDecoder +{ + public IReadOnlySet SupportedExtensions { get; } = new HashSet([".txt"]); + public Guid Id => Guid.Parse("{B87F41E3-6C33-46E4-802A-3E1E82800E7A}"); + public int Version => 1; + + public async Task Encode(AssetId id, IReadOnlyVirtualFileSystem fileSystem, IBufferWriter writer) + { + var text = await fileSystem.ReadAllText(id.Path); + writer.Write(text.ToUpperInvariant()); + } + + public string Decode(AssetId id, ref SequenceReader reader) + { + return reader.ReadString(); + } + + public void HotSwap(string instance, string replacement) + { + throw new NotImplementedException(); + } +} diff --git a/source/CapriKit.Tests/CapriKit.Tests.csproj b/source/CapriKit.Tests/CapriKit.Tests.csproj index c907671..d345c16 100644 --- a/source/CapriKit.Tests/CapriKit.Tests.csproj +++ b/source/CapriKit.Tests/CapriKit.Tests.csproj @@ -15,6 +15,7 @@ + @@ -25,7 +26,4 @@ - - - diff --git a/source/CapriKit.Tests/DirectX11/Buffers/GenericBufferTests.cs b/source/CapriKit.Tests/DirectX11/Buffers/GenericBufferTests.cs index 4c3729d..ec6f046 100644 --- a/source/CapriKit.Tests/DirectX11/Buffers/GenericBufferTests.cs +++ b/source/CapriKit.Tests/DirectX11/Buffers/GenericBufferTests.cs @@ -11,7 +11,7 @@ internal class GenericBufferTests { /// /// Tests StructuredBuffer, RWStructuredBuffer and StagingBuffer through a round-trip of data. - /// + /// [Test] public async Task Mix_Upload_Modify_Download_Staging() { diff --git a/source/CapriKit.Tests/IO/Buffers/IBufferWriterExtensionsTests.cs b/source/CapriKit.Tests/IO/Buffers/IBufferWriterExtensionsTests.cs new file mode 100644 index 0000000..f0bd49d --- /dev/null +++ b/source/CapriKit.Tests/IO/Buffers/IBufferWriterExtensionsTests.cs @@ -0,0 +1,76 @@ +using CapriKit.IO.Buffers; +using System.Buffers; +using System.Buffers.Binary; +using System.Text; + +namespace CapriKit.Tests.IO.Buffers; + +internal class IBufferWriterExtensionsTests +{ + [Test] + public async Task Write_Int32() + { + var writer = new ArrayBufferWriter(); + + writer.Write(-12345); + + var bytes = writer.WrittenMemory.ToArray(); + var value = BinaryPrimitives.ReadInt32LittleEndian(bytes); + + await Assert.That(bytes.Length).IsEqualTo(sizeof(int)); + await Assert.That(value).IsEqualTo(-12345); + } + + [Test] + public async Task Write_String() + { + var writer = new ArrayBufferWriter(); + var value = "héllo"; // 'é' encodes to two bytes, so the prefix must count bytes, not chars + + writer.Write(value); + + // The format promises to be identical to BinaryWriter/BinaryReader's + using var stream = new MemoryStream(writer.WrittenMemory.ToArray()); + using var reader = new BinaryReader(stream, Encoding.UTF8); + var text = reader.ReadString(); + + await Assert.That(text).IsEqualTo(value); + await Assert.That(stream.Position).IsEqualTo(stream.Length); + } + + [Test] + public async Task Write7BitEncodedInt() + { + int[] values = [0, 127, 128, 300, int.MaxValue, -1]; + var writer = new ArrayBufferWriter(); + foreach (var value in values) + { + writer.Write7BitEncodedInt(value); + } + + // The format promises to be identical to BinaryWriter/BinaryReader's + using var stream = new MemoryStream(writer.WrittenMemory.ToArray()); + using var reader = new BinaryReader(stream, Encoding.UTF8); + foreach (var value in values) + { + await Assert.That(reader.Read7BitEncodedInt()).IsEqualTo(value); + } + + await Assert.That(stream.Position).IsEqualTo(stream.Length); + } + + [Test] + public async Task Write_Guid() + { + var writer = new ArrayBufferWriter(); + var guid = Guid.NewGuid(); + + writer.Write(guid); + + var bytes = writer.WrittenMemory.ToArray(); + var roundTripped = new Guid(bytes, bigEndian: false); + + await Assert.That(bytes.Length).IsEqualTo(16); + await Assert.That(roundTripped).IsEqualTo(guid); + } +} diff --git a/source/CapriKit.Tests/IO/Buffers/SequenceReaderExtensionsTests.cs b/source/CapriKit.Tests/IO/Buffers/SequenceReaderExtensionsTests.cs new file mode 100644 index 0000000..bb7ef01 --- /dev/null +++ b/source/CapriKit.Tests/IO/Buffers/SequenceReaderExtensionsTests.cs @@ -0,0 +1,109 @@ +using CapriKit.IO.Buffers; +using System.Buffers; +using System.Text; + +namespace CapriKit.Tests.IO.Buffers; + +internal class SequenceReaderExtensionsTests +{ + // SequenceReader is a ref struct, so all reading happens before the first + // await and only plain locals cross into the assertions + + [Test] + public async Task ReadInt32() + { + var writer = new ArrayBufferWriter(); + writer.Write(-12345); + + var reader = new SequenceReader(new ReadOnlySequence(writer.WrittenMemory)); + var value = reader.ReadInt32(); + var end = reader.End; + + await Assert.That(value).IsEqualTo(-12345); + await Assert.That(end).IsTrue(); + } + + [Test] + public async Task ReadString() + { + var writer = new ArrayBufferWriter(); + writer.Write("héllo"); // 'é' encodes to two bytes, exercising the bytes-not-chars length prefix + + var reader = new SequenceReader(new ReadOnlySequence(writer.WrittenMemory)); + var value = reader.ReadString(); + var end = reader.End; + + await Assert.That(value).IsEqualTo("héllo"); + await Assert.That(end).IsTrue(); + } + + [Test] + public async Task ReadString_WrittenByBinaryWriter() + { + using var stream = new MemoryStream(); + using (var writer = new BinaryWriter(stream, Encoding.UTF8, leaveOpen: true)) + { + writer.Write("héllo"); + } + + var reader = new SequenceReader(new ReadOnlySequence(stream.ToArray())); + var value = reader.ReadString(); + var end = reader.End; + + await Assert.That(value).IsEqualTo("héllo"); + await Assert.That(end).IsTrue(); + } + + [Test] + public async Task Read7BitEncodedInt() + { + int[] values = [0, 127, 128, 300, int.MaxValue, -1]; + using var stream = new MemoryStream(); + using (var writer = new BinaryWriter(stream, Encoding.UTF8, leaveOpen: true)) + { + foreach (var value in values) + { + writer.Write7BitEncodedInt(value); + } + } + + var reader = new SequenceReader(new ReadOnlySequence(stream.ToArray())); + var results = new List(); + while (!reader.End) + { + results.Add(reader.Read7BitEncodedInt()); + } + + await Assert.That(results.SequenceEqual(values)).IsTrue(); + } + + [Test] + public async Task ReadGuid() + { + var guid = Guid.NewGuid(); + var writer = new ArrayBufferWriter(); + writer.Write(guid); + + var reader = new SequenceReader(new ReadOnlySequence(writer.WrittenMemory)); + var value = reader.ReadGuid(); + var end = reader.End; + + await Assert.That(value).IsEqualTo(guid); + await Assert.That(end).IsTrue(); + } + + [Test] + public async Task ReadBytes() + { + var bytes = new byte[] { 1, 2, 3, 4, 5 }; + var writer = new ArrayBufferWriter(); + writer.Write(bytes); + + var reader = new SequenceReader(new ReadOnlySequence(writer.WrittenMemory)); + var value = reader.ReadBytes(bytes.Length); + var end = reader.End; + + await Assert.That(value.SequenceEqual(bytes)).IsTrue(); + await Assert.That(end).IsTrue(); + } +} diff --git a/source/CapriKit.Tests/IO/FileSystemEventListenerTests.cs b/source/CapriKit.Tests/IO/FileSystemEventListenerTests.cs index d991854..cbfd980 100644 --- a/source/CapriKit.Tests/IO/FileSystemEventListenerTests.cs +++ b/source/CapriKit.Tests/IO/FileSystemEventListenerTests.cs @@ -30,10 +30,9 @@ public async Task OnFileChanged() var changed = new TaskCompletionSource(TaskCreationOptions.RunContinuationsAsynchronously); var deleted = new TaskCompletionSource(TaskCreationOptions.RunContinuationsAsynchronously); - var fileSystem = new FileSystem(); var scopedFileSystem = new ScopedFileSystem(fileSystem, TempDirectory); - using var watcher = fileSystem.Watch(TempDirectory); + using var watcher = new FileSystemEventListener(TempDirectory, false); watcher.OnFileChanged += (s, e) => { if (e.reason == FileSystemChangeKind.Created && e.target.FileName.Equals(fileName, StringComparison.OrdinalIgnoreCase))