Imposter call verification

[pg94au]

Hi, we've been starting to work with Mountebank as a means of replacing a dependency on a real OAuth2 identity provider in our tests. We've got some code that is going to need to interact with an identity provider to perform an exchange for a token. We've been able to set up an imposter to respond appropriately given that the required conditions are met (correct action, resource, headers and query string parameters).

In addition to seeing our code successfully execute by externally observing its results, we are wondering about our ability to assert that a correct interaction was made with our 'identity provider' imposter behind the scenes. With mocking frameworks like Moq for example, we could specify that we wish to verify that a method was called using the provided Verify method. I see that if we specify --debug when starting Mountebank, we can inspect an imposter and determine which of its stubs were called, along with the original request/response pairs. It seems, based on this behavior being associated with the --debug option, that it was added only for the purpose of debugging Mountebank itself. Is it possible to get an indication of a stub matching for the purpose I describe, without having to enable debug behavior? (We don't even need the recorded requests and responses, only the match counts.)

I note also that we can specify that we would like to record all interaction with our imposter and work from there, examining recorded requests to see what has occurred. This appears less attractive, as it means we must replicate the predicates that we've already encoded in our imposter's stub configuration into our test's assertions on a recorded request. For example, we'd need to configure a stub to match on a given verb, path, etc., and then we'd have to also write those same criteria into separate assertions about a recorded request (that it matched the same verb, path, etc. that are set in the stub). It would seem to be much easier if we could just configure stubs and then check to see if Mountebank matched a request to a given stub.

Would it be possible to only enable match counts without having to record or run in debug mode? Any thoughts on this?

[bbyars]

Hi Paul, You are correct that the best way to do that today is by setting recordRequests to true and then querying the requests <https://www.mbtest.org/docs/api/mocks>. I haven't yet wrapped my brain around your point that this requires replicating the predicates from the stub configuration. Yes, it does mean doing it in your test code instead of mountebank, but I also don't think that, in most cases, the predicates would be the same. The mountebank predicates define how to match the request to determine the response; the verification test has no need to be coupled to those predicates. You could match on the path, for example, which suffices for response generation, but need to verify something about the body. I'm definitely open to suggestions and pull requests that challenge that assumption, but my current thinking is that, to do verification inside mountebank, we would need a different construct than the predicates. Let me know if I'm thinking about it wrong. -Brandon

…

On Thu, Aug 10, 2023 at 11:20 AM Paul Grebenc ***@***.***> wrote: Hi, we've been starting to work with Mountebank as a means of replacing a dependency on a real OAuth2 identity provider in our tests. We've got some code that is going to need to interact with an identity provider to perform an exchange for a token. We've been able to set up an imposter to respond appropriately given that the required conditions are met (correct action, resource, headers and query string parameters). In addition to seeing our code successfully execute by externally observing its results, we are wondering about our ability to assert that a correct interaction was made with our 'identity provider' imposter behind the scenes. With mocking frameworks like Moq for example, we could specify that we wish to verify that a method was called using the provided Verify method. I see that if we specify --debug when starting Mountebank, we can inspect an imposter and determine which of its stubs were called, along with the original request/response pairs. It seems, based on this behavior being associated with the --debug option, that it was added only for the purpose of debugging Mountebank itself. Is it possible to get an indication of a stub matching for the purpose I describe, without having to enable debug behavior? (We don't even need the recorded requests and responses, only the match counts.) I note also that we can specify that we would like to record all interaction with our imposter and work from there, examining recorded requests to see what has occurred. This appears less attractive, as it means we must replicate the predicates that we've already encoded in our imposter's stub configuration into our test's assertions on a recorded request. For example, we'd need to configure a stub to match on a given verb, path, etc., and then we'd have to also write those same criteria into separate assertions about a recorded request (that it matched the same verb, path, etc. that are set in the stub). It would seem to be much easier if we could just configure stubs and then check to see if Mountebank matched a request to a given stub. Would it be possible to only enable match counts without having to record or run in debug mode? Any thoughts on this? — Reply to this email directly, view it on GitHub <#737>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAARFPZPRFQSDXYJXHTETQLXUUC5FANCNFSM6AAAAAA3LX3XYU> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[pg94au]

Hi Brandon,

Every test case may be somewhat unique in terms of what you wish to assert, but imagine that I have created an imposter with a stub that has the following predicates:

1. Method is POST
2. Resource path is /api/foo
3. Body is "hello"

Only then would I consider the request valid, and in this case, I have set it up to respond with a 200 and a "Hello" body, let's say.

If I execute a test that calls some code that is expected to make a request that matches this stub, and I wish to verify that the request was made correctly, I can tell Mountebank to record the request and make the following assertions against what was recorded in my test code:

1. Method is POST
2. Resource path is /api/foo
3. Body is "hello"

I might care to assert less in some circumstances, but if I do want to know that everything required of a correct request matched, I will have one-for-one duplicated the predicates of my stub in my test code's assertions. In cases like this, it would be nice if it were possible to not have to record anything (or enable debug), and instead simply check that this stub's matched count is 1 (if the stub's predicates matched, my request was as expected). Would it be possible for a case like this to keep track of match counts even when recording and debug are not enabled?

Regards,
Paul

[bbyars]

Everything's possible, but I still think it'd require a different structure. For example, most mocking libraries have some form of a negative assert -- assert that this function was *not *called. It seems to me that making a general purpose verification system would best be served by a separate endpoint. -Brandon

…

On Tue, Aug 15, 2023 at 2:15 PM Paul Grebenc ***@***.***> wrote: Hi Brandon, Every test case may be somewhat unique in terms of what you wish to assert, but imagine that I have created an imposter with a stub that has the following predicates: 1. Method is POST 2. Resource path is /api/foo 3. Body is "hello" Only then would I consider the request valid, and in this case, I have set it up to respond with a 200 and a "Hello" body, let's say. If I execute a test that calls some code that is expected to make a request that matches this stub, and I wish to verify that the request was made correctly, I can tell Mountebank to record the request and make the following assertions against what was recorded in my test code: 1. Method is POST 2. Resource path is /api/foo 3. Body is "hello" I might care to assert less in some circumstances, but if I do want to know that everything required of a correct request matched, I will have one-for-one duplicated the predicates of my stub in my test code's assertions. In cases like this, it would be nice if it were possible to not have to record anything (or enable debug), and instead simply check that this stub's matched count is 1 (if the stub's predicates matched, my request was as expected). Would it be possible for a case like this to keep track of match counts even when recording and debug are not enabled? Regards, Paul — Reply to this email directly, view it on GitHub <#737 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAARFPZ5F5E3FHYUCRUS5OLXVPDFZANCNFSM6AAAAAA3LX3XYU> . You are receiving this because you commented.Message ID: ***@***.***>

[pg94au]

I'm not sure if I'm missing something, but wouldn't that already be covered if match counts were exposed? If a stub's match count is zero, you know that a matching request was not initiated.

-Paul

[bbyars]

Yep, it sounds like that works in your scenario, but it's not a great general purpose solution. For instance, it's pretty easy to imagine a test case which is simply about verification and needs no stubs (just a defaultResponse), but would want to verify that a certain endpoint was or was not called. This is why I don't think an appropriate verification solution should be coupled to the stubs or predicates; they serve different purposes. -Brandon

…

On Wed, Aug 16, 2023 at 2:52 PM Paul Grebenc ***@***.***> wrote: I'm not sure if I'm missing something, but wouldn't that already be covered if match counts were exposed? If a stub's match count is zero, you know that a matching request was not initiated. -Paul — Reply to this email directly, view it on GitHub <#737 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAARFP5EJ6C7WQ2WXYNJ2L3XVUQJHANCNFSM6AAAAAA3LX3XYU> . You are receiving this because you commented.Message ID: ***@***.***>

[pg94au]

Perhaps this scenario is already handled by the original premise that you can just record exchanges and make assertions about what ends up there in your test code? In that case, you can have a default response, assuming the code you are testing is fine with it, and assert whatever you wish about the recordings as verification.

Is the question whether all matching on predicates should be done only by Mountebank and not the assertion framework utilized by the tests that are interacting with Mountebank? I see verification as dependent upon predicates as well, in the form of matching criteria identical to the matching criteria upon which stub behavior is predicated. Is it just putting those same predicates either in place to determine responses to be returned, or putting them in place to determine what types of requests are being made?

-Paul

[bbyars]

Hi Paul, I have no qualms with the addition of verification to mountebank instead of having to do it client-side. It's the coupling of the existing predicates to the predicates needed for verification that I'm uncomfortable with. That's not how mocking frameworks that provide call verification work. As an example, I just pulled up the mockito site. Here's the equivalent to mountebank's stub predicates/responses: when(mockedList.get(anyInt())).thenReturn("element"); And here's an example verification: verify(mockedList).get(0); It is true both when() and verify() use a type of predicates, but any general purpose verification mechanism will use *different *predicates than the stubbing predicates used for result generation. I understand that's not your case, but in my experience of maintaining mountebank's API for 10 years, it's a mistake to change the API interface in a tactical way that doesn't solve the general purpose problem, because those tactical solutions add up to significant complexity over time. -Brandon

…

On Fri, Aug 18, 2023 at 11:13 AM Paul Grebenc ***@***.***> wrote: Perhaps this scenario is already handled by the original premise that you can just record exchanges and make assertions about what ends up there in your test code? In that case, you can have a default response, assuming the code you are testing is fine with it, and assert whatever you wish about the recordings as verification. Is the question whether all matching on predicates should be done only by Mountebank and not the assertion framework utilized by the tests that are interacting with Mountebank? I see verification as dependent upon predicates as well, in the form of matching criteria identical to the matching criteria upon which stub behavior is predicated. Is it just putting those same predicates either in place to determine responses to be returned, or putting them in place to determine what types of requests are being made? -Paul — Reply to this email directly, view it on GitHub <#737 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAARFP5GC2TS7RMM4KM6D3TXV6IBJANCNFSM6AAAAAA3LX3XYU> . You are receiving this because you commented.Message ID: ***@***.***>

[pg94au]

Hi Brandon,

Sorry for not getting back to you sooner. As you stated, Mockito does have you define predicates utilizing the same matchers in the org.mockito.matchers namespace, whether you are defining a predicate for a stub or for verification. That is what I am getting at, that predicates for both purposes are defined by the same matchers (they can go in either place), rather than being different creatures entirely. I'm not suggesting that the actual "use case" predicates defined for stubbing and verification are one in the sense that they cannot differ.

Please don't misunderstand my replies as pushing for a particular implementation solely for my own use. That is not true at all, and I would be only too happy that you might consider any solution. The work you have put into Mountebank has been very beneficial to the work my team has been doing, and we are most grateful.

-Paul