CP: Allow the app to react when app-provided buffer is insufficient (#5347)

* Allow the app to react when app-provided buffer is insufficient (#5313)

* Notify the app when not enough receive buffer is present
* Remove constrains on flow control
* Add test
* Update documentation

Author: guhetier

docs/PreviewFeatures.md

@@ -35,8 +35,7 @@ TODO
 
 TODO
 
-### StreamProvideReceiveBuffers
+### App-provided receive buffers
 
-[StreamProvideReceiveBuffers](StreamProvideReceiveBuffers.md)
-
-TODO
+- [StreamProvideReceiveBuffers](api/StreamProvideReceiveBuffers.md)
+- [QUIC_API_ENABLE_PREVIEW_FEATURES](api/QUIC_STREAM_EVENT.md#quic_stream_event_receive_buffer_needed)

docs/Streams.md

@@ -170,9 +170,16 @@ The application regains full ownership of a buffer after it get a receive notifi
 If the application accepts all the buffer's bytes **inline** from the receive notification, by returning `QUIC_STATUS_SUCCESS` and setting `TotalBufferLength` appropriately,
 it can free or reuse the buffer while in the notification handler.
 
+If more data is received on the stream than buffer space was provided by the application, MsQuic will emit a `QUIC_STREAM_EVENT_RECEIVE_BUFFER_NEEDED` notification.
+When receiving this notification, the app can:
+- provide a sufficient amount of buffer space **inline** from the callback using [`StreamProvideReceiveBuffers`](./api/StreamProvideReceiveBuffers.md)
+- shutdown the stream receive direction of the stream **inline** by calling [`StreamShutdown`](api/StreamShutdown.md) with the `QUIC_STREAM_SHUTDOWN_FLAG_INLINE` flag
+
+When the callback returns, if the stream has not been shutdown and a sufficient amount of memory is not available, the connection is closed abortively.
+Providing memory in reaction to `QUIC_STREAM_EVENT_RECEIVE_BUFFER_NEEDED` can impact performances negatively.
+
 For an application, providing receive buffers can improve performances by saving a copy: MsQuic places data directly in its final destination.
-However, it comes with a large complexity overhead for the application, both in term of memory management and in term of flow control:
-an application providing too much or too little buffer space could negatively impact performances.
+However, it comes with a large complexity overhead for the application, both in term of memory management and in term of flow control: an application providing too much or too little buffer space could negatively impact performances.
 Because of this, app-owned mode should be considered an advanced feature and used with caution.
 
 > **Note**: As of now, app-owned buffer mode is not compatible with multi-receive mode. If multi-receive mode is enabled for the connection and app-owned mode is enabled on a stream, that specific stream will behave as if multi-receive mode was disabled. This may change in the future.

docs/api/QUIC_STREAM_EVENT.md

@@ -17,6 +17,9 @@ typedef enum QUIC_STREAM_EVENT_TYPE {
     QUIC_STREAM_EVENT_IDEAL_SEND_BUFFER_SIZE    = 8,
     QUIC_STREAM_EVENT_PEER_ACCEPTED             = 9,
     QUIC_STREAM_EVENT_CANCEL_ON_LOSS            = 10,
+#ifdef QUIC_API_ENABLE_PREVIEW_FEATURES
+    QUIC_STREAM_EVENT_RECEIVE_BUFFER_NEEDED = 11,
+#endif
 } QUIC_STREAM_EVENT_TYPE;
 ```
 
@@ -69,6 +72,11 @@ typedef struct QUIC_STREAM_EVENT {
         struct {
             /* out */ QUIC_UINT62 ErrorCode;
         } CANCEL_ON_LOSS;
+#ifdef QUIC_API_ENABLE_PREVIEW_FEATURES
+        struct {
+            /* in */  uint64_t BufferLengthNeeded;
+        } RECEIVE_BUFFER_NEEDED;
+#endif
     };
 } QUIC_STREAM_EVENT;
 ```
@@ -261,6 +269,23 @@ The application can supply an error code in this struct to be sent to the peer.
 
 The application can set this 62 bit error code to communicate to the peer about the stream shutdown, which is received by the peer as a `QUIC_STREAM_EVENT_PEER_SEND_ABORTED` event on its stream object.
 
+## QUIC_STREAM_EVENT_RECEIVE_BUFFER_NEEDED
+
+This event is raised when a stream using app-provided receive buffers runs out of receive buffer space.
+
+`BufferLengthNeeded`
+
+The number of bytes MsQuic needs to be able to store the received data.
+
+When receiving this notification, the app can:
+- provide a sufficient amount of buffer space **inline** from the callback using [`StreamProvideReceiveBuffers`](./StreamProvideReceiveBuffers.md)
+- shutdown the stream receive direction of the stream **inline** by calling [`StreamShutdown`](./StreamShutdown.md)
+    with the `QUIC_STREAM_SHUTDOWN_FLAG_INLINE` flag
+
+Otherwise, the connection will be closed abortively.
+
+See [App-Owned Buffer Mode](../Streams.md#App-Owned_Buffer_Mode) for further details.
+
 # See Also
 
 [Streams](../Streams.md)<br>

docs/api/StreamProvideReceiveBuffers.md

@@ -28,7 +28,12 @@ QUIC_STATUS
 # Remarks
 
 This is an asynchronous API but it can run inline if called in a callback.
-If called inline when handling a `QUIC_CONNECTION_EVENT_PEER_STREAM_STARTED` event, it will convert the stream to app-owned buffer mode.
+
+If called inline when handling a `QUIC_CONNECTION_EVENT_PEER_STREAM_STARTED` event, it will convert
+the stream to app-owned buffer mode.
+
+If called inline when handling a `QUIC_STREAM_EVENT_RECEIVE_BUFFER_NEEDED` event, the provided
+memory buffers will be used to store the received data.
 
 # See also
 

src/core/crypto.c

@@ -1234,7 +1234,7 @@ QuicCryptoProcessDataFrame(
 {
     QUIC_STATUS Status;
     QUIC_CONNECTION* Connection = QuicCryptoGetConnection(Crypto);
-    uint64_t FlowControlLimit = UINT16_MAX;
+    uint64_t FlowControlQuota = UINT16_MAX;
 
     *DataReady = FALSE;
 
@@ -1267,14 +1267,18 @@ QuicCryptoProcessDataFrame(
         // Write the received data (could be duplicate) to the stream buffer. The
         // stream buffer will indicate if there is data to process.
         //
+        uint64_t BufferSizeNeeded = 0;
         Status =
             QuicRecvBufferWrite(
                 &Crypto->RecvBuffer,
                 Crypto->RecvEncryptLevelStartOffset + Frame->Offset,
                 (uint16_t)Frame->Length,
                 Frame->Data,
-                &FlowControlLimit,
-                DataReady);
+                FlowControlQuota,
+                &FlowControlQuota,
+                DataReady,
+                &BufferSizeNeeded);
+        CXPLAT_DBG_ASSERT(BufferSizeNeeded == 0);
         if (QUIC_FAILED(Status)) {
             if (Status == QUIC_STATUS_BUFFER_TOO_SMALL) {
                 QuicTraceEvent(