Micrometer Observation instrumentation #4980

marcingrzejszczak · 2023-06-22T14:26:09Z

fixes #4659

CLAassistant · 2023-06-22T14:26:15Z

All committers have signed the CLA.

jrhee17

Sorry about the delay! Left some questions and comments but honestly I think this PR is already close to completion 🚀

Also pushed some small commits which

Fixes failing tests
Hides some public classes

Feel free to revert any changes I made, or just let me know and I'll revert it back 😄

jrhee17 · 2023-06-27T13:18:19Z

...tion/src/main/java/com/linecorp/armeria/server/observation/MicrometerObservationService.java

+        if (!ctx.config().transientServiceOptions().contains(TransientServiceOption.WITH_TRACING) &&
+            !ctx.config().transientServiceOptions().contains(
+                    TransientServiceOption.WITH_METRIC_COLLECTION)) {


Transient services signify services that may not be of interest for most users. For instance, HealthCheckService is a transient service where users may not be interested in recording metrics/logs.

By default, user defined services have all TransientServiceOptions enabled so most times I guess this won't be a problem.

armeria/core/src/main/java/com/linecorp/armeria/server/ServiceConfig.java

Line 152 in ea3b063

return TransientServiceOption.allOf();

I guess we don't want to collect observation traces/metrics for transient services, so what do you think of just disabling if either option doesn't exist?
Even if WITH_TRACING may be disabled, it is currently possible that traces are still recordeddue to WITH_METRIC_COLLECTION being enabled.

Suggested change

if (!ctx.config().transientServiceOptions().contains(TransientServiceOption.WITH_TRACING) &&

!ctx.config().transientServiceOptions().contains(

TransientServiceOption.WITH_METRIC_COLLECTION)) {

if (!ctx.config().transientServiceOptions().contains(TransientServiceOption.WITH_TRACING) ||

!ctx.config().transientServiceOptions().contains(

TransientServiceOption.WITH_METRIC_COLLECTION)) {

I realize the behavior depends on how users customize their ObservationRegistry, but I guess we don't have a good way to determine what kind of handlers users added at the moment 😅

jrhee17 · 2023-06-27T13:22:09Z

...tion/src/main/java/com/linecorp/armeria/server/observation/MicrometerObservationService.java

+        enrichObservation(ctx, httpServerContext, observation);
+
+        return observation.scopedChecked(
+                () -> unwrap().serve(ctx, req)); // TODO: Maybe we should observation stopping here


// TODO: Maybe we should observation stopping here

As it stands, the most reliable way to know whether a response is finished is to subscribe to ctx.log in armeria.
I think the current approach is fine, but let me know if I misunderstood your comment 😄

jrhee17 · 2023-06-27T13:23:51Z

...tion/src/main/java/com/linecorp/armeria/server/observation/MicrometerObservationService.java

+               // TODO: ClientConnectionTimings - no hook to be there at the
+               //  moment of those things actually hapenning


ClientConnectionTimings represents timing information for client-side, whereas this class is for server side.
Would it be OK to remove this comment?

observation/src/main/java/com/linecorp/armeria/client/observation/HttpClientContext.java

jrhee17 · 2023-06-27T14:43:05Z

...rc/main/java/com/linecorp/armeria/client/observation/HttpClientObservationDocumentation.java

+            return DefaultHttpClientObservationConvention.class;
+        }
+
+        // TODO: Figure what should be low and what high cardinality


I guess HTTP_METHOD, STATUS_CODE can be low cardinality and we can probably add more keys later if needed.

For reference, here are some tags that are added by default in Armeria's MetricCollecting[Service|Client]

armeria/core/src/main/java/com/linecorp/armeria/internal/common/metric/DefaultMeterIdPrefixFunction.java

Lines 53 to 66 in ea3b063

public MeterIdPrefix activeRequestPrefix(MeterRegistry registry, RequestOnlyLog log) {

/* hostname.pattern, method, service */

final Builder<Tag> tagListBuilder = ImmutableList.builderWithExpectedSize(3);

addActiveRequestPrefixTags(tagListBuilder, log);

return new MeterIdPrefix(name, tagListBuilder.build());

}

@Override

public MeterIdPrefix completeRequestPrefix(MeterRegistry registry, RequestLog log) {

/* hostname.pattern, http.status, method, service */

final Builder<Tag> tagListBuilder = ImmutableList.builderWithExpectedSize(4);

addCompleteRequestPrefixTags(tagListBuilder, log);

return new MeterIdPrefix(name, tagListBuilder.build());

}

I guess SESSION_PROTOCOL and SERIALIZATION_FORMAT can also be considered low cardinality

jrhee17 · 2023-06-27T15:09:14Z

...n/src/main/java/com/linecorp/armeria/client/observation/HttpClientObservationConvention.java

+import io.micrometer.observation.Observation.Context;
+import io.micrometer.observation.ObservationConvention;
+
+interface HttpClientObservationConvention extends ObservationConvention<HttpClientContext> {


Question) Is it possible to remove this interface and just add supportContext directly into DefaultHttpClientObservationConvention?

You can do whatever you want :) This is how we're suggesting that the users should proceed https://micrometer.io/docs/observation#_observation_observationconvention_example

jrhee17 · 2023-06-27T15:21:14Z

observation/src/main/java/com/linecorp/armeria/client/observation/HttpClientContext.java

+
+import io.micrometer.observation.transport.RequestReplySenderContext;
+
+final class HttpClientContext extends RequestReplySenderContext<RequestHeadersBuilder, RequestLog> {


I didn't see much value in exposing HttpClientContext so I pushed a commit which hides this class to package-private along with some other classes (Sorry, I probably should've asked for your opinion before doing so! 😅 Let me know if you feel like it's better to expose this though )

If users want to add their own handlers/conventions, I guess they can store additional information via. Observation.Context.put(key, val) to access from handlers/conventions.

I think it would also be awesome if there was an interface for Observation.Context so that we could do something like:

// this is public public interface ArmeriaObservationContext implements Observation.Context { RequestContext requestContext(); } // this is package-private class HttpClientContext extends RequestReplySenderContext<RequestHeadersBuilder, RequestLog> implements ArmeriaObservationContext { } // and then users can do something like this... registry.observationConfig().observationHandler(new ObservationHandler<ArmeriaObservationContext>()

I didn't see much value in exposing HttpClientContext so I pushed a commit which hides this class to package-private along with some other classes (Sorry, I probably should've asked for your opinion before doing so! Let me know if you feel like it's better to expose this though )

If you make that package-private users will not be able to see it and won't be easily be able to provide their own customizations (here you have an example https://micrometer.io/docs/observation#_observation_observationconvention_example)

observation/src/main/java/com/linecorp/armeria/client/observation/HttpClientContext.java

jrhee17 · 2023-06-27T15:23:06Z

...ation/src/main/java/com/linecorp/armeria/client/observation/MicrometerObservationClient.java

+                                        @Nullable HttpClientObservationConvention
+                                                httpClientObservationConvention) {
+        super(delegate);
+        this.observationRegistry = observationRegistry;


Can you do a null check for public APIs so that users can know right away when their input is incorrect?

Suggested change

this.observationRegistry = observationRegistry;

this.observationRegistry = requireNonNull(observationRegistry, "observationRegistry");

jrhee17 · 2023-06-27T15:48:48Z

...tion/src/main/java/com/linecorp/armeria/server/observation/MicrometerObservationService.java

+            // Make the span the current span and run scope decorators when the ctx is pushed.
+            ctxExtension.hook(observation::openScope);


jrhee17 · 2023-06-27T15:55:04Z

...ation/src/main/java/com/linecorp/armeria/client/observation/MicrometerObservationClient.java

+               // TODO: ClientConnectionTimings - no hook to be there
+               //  at the moment of those things actually hapening


I think this information isn't really critical to merge this PR, but still is useful for users going through traces.

Would it be difficult to also provide the option to pass a timestamp such that observation.event(Event, long) like brave and otel do? (maybe DefaultMeterObservationHandler can just ignore the timestamp)
~~Adding such hooks require some degree of synchronization which imposes a performance penalty, and I'm not sure if it's feasible to add hooks for all events we want to add in the future 😅~~

Actually, I guess this doesn't make much sense if users want to just collect metrics... let me think about this a little more

marcingrzejszczak

I've added comments to your comments and will upload some code very soon

marcingrzejszczak · 2023-06-28T08:42:48Z

observation/src/main/java/com/linecorp/armeria/client/observation/HttpClientContext.java

+
+import io.micrometer.observation.transport.RequestReplySenderContext;
+
+final class HttpClientContext extends RequestReplySenderContext<RequestHeadersBuilder, RequestLog> {


I didn't see much value in exposing HttpClientContext so I pushed a commit which hides this class to package-private along with some other classes (Sorry, I probably should've asked for your opinion before doing so! Let me know if you feel like it's better to expose this though )

If you make that package-private users will not be able to see it and won't be easily be able to provide their own customizations (here you have an example https://micrometer.io/docs/observation#_observation_observationconvention_example)

observation/src/main/java/com/linecorp/armeria/client/observation/HttpClientContext.java

marcingrzejszczak · 2023-06-28T09:09:56Z

...n/src/main/java/com/linecorp/armeria/client/observation/HttpClientObservationConvention.java

+import io.micrometer.observation.Observation.Context;
+import io.micrometer.observation.ObservationConvention;
+
+interface HttpClientObservationConvention extends ObservationConvention<HttpClientContext> {


You can do whatever you want :) This is how we're suggesting that the users should proceed https://micrometer.io/docs/observation#_observation_observationconvention_example

marcingrzejszczak · 2023-06-28T12:50:44Z

I've also added support for Micrometer Docs Generation https://micrometer.io/docs/observation#_automated_documentation_generation. Under observation/build you will find adoc files (spans, metrics, conventions) that you can include in your documentation

marcingrzejszczak · 2023-06-29T08:54:26Z

...tion/src/main/java/com/linecorp/armeria/server/observation/MicrometerObservationService.java

@@ -82,7 +69,7 @@ public HttpResponse serve(ServiceRequestContext ctx, HttpRequest req) throws Exc

        final HttpServerContext httpServerContext = new HttpServerContext(ctx, req);
        final Observation observation = ServiceObservationDocumentation.OBSERVATION.observation(
-                this.serviceObservationConvention, DefaultServiceObservationConvention.INSTANCE,
+                null, DefaultServiceObservationConvention.INSTANCE,


So I understand that you're not allowing users to modify the default tagging by injecting the custom convention?

On second thought, I think it's a good idea to allow users to inject custom conventions.
Ended up making HttpClientContext, HttpServerContext public 😄

jrhee17

I'll take another look in the morning but I think this pull request is pretty much done!

I've pushed some changes which

Makes HttpClientContext, HttpServerContext public along with some test code
- Also renamed some variables
Removed the default Http[Client|Server]ObservationConvention interfaces
Added some more documentation

Let me know if any changes don't make sense 😄
Also, what do you think of changing the status of this PR to ready for review now?

jrhee17 · 2023-06-29T16:56:33Z

observation/src/main/java/com/linecorp/armeria/client/observation/HttpClientContext.java

+        this.clientRequestContext = clientRequestContext;
+        this.httpRequest = httpRequest;
+        setCarrier(carrier);
+        updateRemoteEndpoint(this, clientRequestContext);


On a second look, I think it's not possible to know what the remote address is at this stage unless the client is explicitly connecting to an ip address from the start. (this is before a connection is acquired/determined)

We can know this information for sure later when ctx.log().whenComplete() is invoked.

Is there any way we can update remote endpoint information later and have it exposed in the spans?
If not, I think it might be better to just remove this since we can't reliably supply an ip here 😅

jrhee17 · 2023-06-29T17:12:42Z

site/src/pages/docs/advanced-zipkin.mdx

@@ -47,7 +47,7 @@ Now, you can specify <type://BraveService> using [Decorating a service](/docs/se
 ```java
    import com.linecorp.armeria.common.HttpResponse;
    import com.linecorp.armeria.server.Server;
-    import com.linecorp.armeria.server.brave.BraveService;
+    import com.linecorp.armeria.server.observation.BraveService;


Can you revert this since the class doesn't exist? 😅
I'll be adding a separate page on tracing later

marcingrzejszczak · 2023-06-30T10:18:22Z

OK, I've removed setting of the remote address - and applied all suggested changes. Also, the PR is now officially ready for review :)

jrhee17

Thanks @marcingrzejszczak 🙇 👍 🚀

...ation/src/main/java/com/linecorp/armeria/client/observation/MicrometerObservationClient.java

marcingrzejszczak · 2023-07-03T09:16:04Z

Thank you, my pleasure @jrhee17 ! :)

trustin · 2023-07-05T07:54:29Z

...ation/src/main/java/com/linecorp/armeria/client/observation/MicrometerObservationClient.java

+               // TODO: ClientConnectionTimings - there is no way to record events
+               // with a specific timestamp for an observation


Could you elaborate on this comment? What prevents us from recording connection timings?

Micrometer Observation doesn't have the API to record an event with a timestamp. Current Armeria API gives you information about a fact happening at a given timestamp. With Micrometer Observation you can react to a situation happening at a given point in time. So Armeria API would have to give a handle to call observation.event(...).

OK. Any chance Micrometer Observation to have such an API in the future? We worked with Brave team to add such a feature in the past.

We could add that but the problem I see is that we can't set the timestamp for a counter. So whenever we have an event with Micrometer Observation what we do for metrics is we create a counter and for tracing we annotate the span. What we could do is ignore the timestamp information for the metrics and simply increment the counter when the event method was called. For tracing we would set an event with a given timestamp. Does it make sense? cc @shakuzen @jonatan-ivanov

For tracing we would set an event with a given timestamp.

Yeah, this is probably what we want.

trustin · 2023-07-05T07:55:05Z

...ation/src/main/java/com/linecorp/armeria/client/observation/MicrometerObservationClient.java

+           .whenAvailable(RequestLogProperty.RESPONSE_FIRST_BYTES_TRANSFERRED_TIME)
+           .thenAccept(requestLog -> {
+               if (requestLog.responseFirstBytesTransferredTimeNanos() != null) {
+                   observation.event(Events.WIRE_RECEIVE);


Is there a way to specify the timestamp? I'm pretty sure there will be latency between the actual recording and observation.event().

No there is none. That's why this (#4980 (comment)) is not possible to be done

.../src/test/java/com/linecorp/armeria/server/observation/MicrometerObservationServiceTest.java

trustin · 2023-07-05T08:01:49Z

settings.gradle

@@ -63,6 +63,7 @@ includeWithFlags ':kafka',                               'java', 'publish', 'rel
 includeWithFlags ':kotlin',                              'java', 'publish', 'relocate', 'kotlin'
 includeWithFlags ':logback',                             'java', 'publish', 'relocate'
 includeWithFlags ':oauth2',                              'java', 'publish', 'relocate'
+includeWithFlags ':observation',                         'java', 'publish', 'relocate'


Should we keep this as a separate module or make it part of the core? We already have Micrometer in our core, so I think we can move it into core if it doesn't pull in other dependencies such as Brave.

Makes sense. We always release Micrometer Observation with Micrometer Core (that's the same project) and Core depends on Observation.

Talked with the team and moving the implementation to the :core module

marcingrzejszczak · 2023-07-10T10:51:11Z

Is there anything else I should to merge this?

jrhee17 · 2023-07-10T10:52:43Z

Sorry about the delay 😅 Let me update the PR to address @trustin's comments tonight, and then ping the other maintainers

marcingrzejszczak · 2023-07-10T11:13:50Z

Oh absolutely no problem, I was just wondering if someone is waiting on me and I have missed the message. Take your time!

ikhoon

Overall looks good! Left comments for code style, micro optimizations, and suggestions.

core/src/main/java/com/linecorp/armeria/client/observation/HttpClientContext.java

core/src/main/java/com/linecorp/armeria/client/observation/MicrometerObservationClient.java

core/src/main/java/com/linecorp/armeria/client/observation/HttpClientContext.java

...rc/main/java/com/linecorp/armeria/server/observation/HttpServerObservationDocumentation.java

core/src/main/java/com/linecorp/armeria/server/observation/MicrometerObservationService.java

marcingrzejszczak · 2023-07-14T10:15:22Z

Hey I think I've applied all the changes. Tell me if I missed sth

ikhoon

High-quality work! Many thanks, @marcingrzejszczak. 🚀🙇‍♂️

...ain/java/com/linecorp/armeria/client/observation/DefaultHttpClientObservationConvention.java

core/src/main/java/com/linecorp/armeria/client/observation/ClientObservationContext.java

core/src/main/java/com/linecorp/armeria/client/observation/ObservationClient.java

...c/main/java/com/linecorp/armeria/server/observation/DefaultServiceObservationConvention.java

core/src/main/java/com/linecorp/armeria/server/observation/ObservationService.java

…ion/MicrometerObservationClient.java

…rometerObservationService.java Co-authored-by: Ikhun Um <ih.pert@gmail.com>

…rometerObservationClient.java Co-authored-by: Ikhun Um <ih.pert@gmail.com>

Co-authored-by: Ikhun Um <ih.pert@gmail.com>

…aultHttpClientObservationConvention.java Co-authored-by: Ikhun Um <ih.pert@gmail.com>

…entObservationContext.java Co-authored-by: Ikhun Um <ih.pert@gmail.com>

…ervationClient.java Co-authored-by: Ikhun Um <ih.pert@gmail.com>

…aultServiceObservationConvention.java Co-authored-by: Ikhun Um <ih.pert@gmail.com>

…ervationService.java Co-authored-by: Ikhun Um <ih.pert@gmail.com>

Co-authored-by: minux <songmw725@gmail.com>

trustin

Thanks a lot for your patience, @marcingrzejszczak. Looks great to me. It'd be awesome if Micrometer Observation exposes an API that allows us to specify timestamps explicitly, though. Any chance we can have them in the future?

marcingrzejszczak · 2023-08-16T12:20:01Z

Thanks! I left a comment on that comment :) I'd like @shakuzen and @jonatan-ivanov to chime in on that cause if we don't care that the moment in which the counter got incremented is not the same as the one from the timestamp then we should be alright and could add this feature in the future

trustin · 2023-08-17T08:25:25Z

Thanks a lot for your high quality contribution and patience, @marcingrzejszczak! Let us stay tuned to micrometer-metrics/micrometer#4032 🙇

marcingrzejszczak · 2023-08-17T09:09:22Z

Sure @trustin, we had a conversation inside the team about this and we see no problem in adding that feature to Micrometer . Since I have your attention here, what do you think of netty/netty#8546 ? :)

marcingrzejszczak changed the title ~~Observation~~ Micrometer Observation instrumentation Jun 22, 2023

jrhee17 added the new feature label Jun 23, 2023

jrhee17 added this to the 1.25.0 milestone Jun 23, 2023

jrhee17 reviewed Jun 27, 2023

View reviewed changes

marcingrzejszczak commented Jun 28, 2023

View reviewed changes

marcingrzejszczak force-pushed the observation branch from 145bc0c to a21e66a Compare June 28, 2023 12:48

jrhee17 force-pushed the observation branch from 1f54f38 to 58dfd49 Compare June 29, 2023 08:37

marcingrzejszczak commented Jun 29, 2023

View reviewed changes

jrhee17 reviewed Jun 29, 2023

View reviewed changes

marcingrzejszczak marked this pull request as ready for review June 30, 2023 10:17

marcingrzejszczak requested review from ikhoon, minwoox and trustin as code owners June 30, 2023 10:17

jrhee17 approved these changes Jul 3, 2023

View reviewed changes

...ation/src/main/java/com/linecorp/armeria/client/observation/MicrometerObservationClient.java Outdated Show resolved Hide resolved

trustin reviewed Jul 5, 2023

View reviewed changes

ikhoon reviewed Jul 14, 2023

View reviewed changes

jrhee17 mentioned this pull request Jul 17, 2023

Support micrometer-tracing client-side #4883

Closed

marcingrzejszczak mentioned this pull request Jul 19, 2023

[Netty 5] Visibility / Metrics netty/netty#8546

Open

marcingrzejszczak force-pushed the observation branch from f55ad42 to 3b3ed20 Compare July 20, 2023 10:25

ikhoon approved these changes Jul 30, 2023

View reviewed changes

jrhee17 and others added 22 commits August 16, 2023 12:24

Update observation/src/main/java/com/linecorp/armeria/client/observat…

c9d2e8d

…ion/MicrometerObservationClient.java

move observation to the core module

e5d0a17

fix build failure

ff965e8

remove unused test

399bf53

Update core/src/main/java/com/linecorp/armeria/server/observation/Mic…

0394860

…rometerObservationService.java Co-authored-by: Ikhun Um <ih.pert@gmail.com>

Update core/src/main/java/com/linecorp/armeria/client/observation/Mic…

340ce15

…rometerObservationClient.java Co-authored-by: Ikhun Um <ih.pert@gmail.com>

Apply suggestions from code review

c7691b6

Co-authored-by: Ikhun Um <ih.pert@gmail.com>

Applied changes following the review

9a2cee0

fix failing build

103f564

lint

d8721a6

lint

ea6930c

Update core/src/main/java/com/linecorp/armeria/client/observation/Def…

a176311

…aultHttpClientObservationConvention.java Co-authored-by: Ikhun Um <ih.pert@gmail.com>

Update core/src/main/java/com/linecorp/armeria/client/observation/Cli…

ee94f5f

…entObservationContext.java Co-authored-by: Ikhun Um <ih.pert@gmail.com>

Update core/src/main/java/com/linecorp/armeria/client/observation/Obs…

c3fb07f

…ervationClient.java Co-authored-by: Ikhun Um <ih.pert@gmail.com>

Update core/src/main/java/com/linecorp/armeria/client/observation/Obs…

792b26e

…ervationClient.java Co-authored-by: Ikhun Um <ih.pert@gmail.com>

Update core/src/main/java/com/linecorp/armeria/server/observation/Def…

26c3af5

…aultServiceObservationConvention.java Co-authored-by: Ikhun Um <ih.pert@gmail.com>

Update core/src/main/java/com/linecorp/armeria/server/observation/Obs…

4e2500f

…ervationService.java Co-authored-by: Ikhun Um <ih.pert@gmail.com>

Update core/src/main/java/com/linecorp/armeria/server/observation/Obs…

6d75d8b

…ervationService.java Co-authored-by: Ikhun Um <ih.pert@gmail.com>

Apply suggestions from code review

202a97a

Co-authored-by: minux <songmw725@gmail.com>

Added some javadocs + reformatted the code

9653ad0

Updated toml entry for micrometer tracing

581ef5d

fix build

868c8ea

marcingrzejszczak force-pushed the observation branch from 67d7a72 to 868c8ea Compare August 16, 2023 10:24

trustin approved these changes Aug 16, 2023

View reviewed changes

marcingrzejszczak mentioned this pull request Aug 16, 2023

Add a timestamp to Event in Observation micrometer-metrics/micrometer#4032

Closed

trustin merged commit 8351737 into line:main Aug 17, 2023
12 of 13 checks passed

marcingrzejszczak deleted the observation branch August 17, 2023 09:38

		// TODO: ClientConnectionTimings - no hook to be there at the
		// moment of those things actually hapenning

	public MeterIdPrefix activeRequestPrefix(MeterRegistry registry, RequestOnlyLog log) {
	/* hostname.pattern, method, service */
	final Builder<Tag> tagListBuilder = ImmutableList.builderWithExpectedSize(3);
	addActiveRequestPrefixTags(tagListBuilder, log);
	return new MeterIdPrefix(name, tagListBuilder.build());
	}

	@Override
	public MeterIdPrefix completeRequestPrefix(MeterRegistry registry, RequestLog log) {
	/* hostname.pattern, http.status, method, service */
	final Builder<Tag> tagListBuilder = ImmutableList.builderWithExpectedSize(4);
	addCompleteRequestPrefixTags(tagListBuilder, log);
	return new MeterIdPrefix(name, tagListBuilder.build());
	}


		import io.micrometer.observation.transport.RequestReplySenderContext;

		final class HttpClientContext extends RequestReplySenderContext<RequestHeadersBuilder, RequestLog> {

	this.observationRegistry = observationRegistry;
	this.observationRegistry = requireNonNull(observationRegistry, "observationRegistry");

		// Make the span the current span and run scope decorators when the ctx is pushed.
		ctxExtension.hook(observation::openScope);

		// TODO: ClientConnectionTimings - no hook to be there
		// at the moment of those things actually hapening

		// TODO: ClientConnectionTimings - there is no way to record events
		// with a specific timestamp for an observation

Micrometer Observation instrumentation #4980

Micrometer Observation instrumentation #4980

Conversation

marcingrzejszczak commented Jun 22, 2023

CLAassistant commented Jun 22, 2023 • edited

jrhee17 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jrhee17 Jun 27, 2023 • edited

Choose a reason for hiding this comment

marcingrzejszczak left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

marcingrzejszczak commented Jun 28, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jrhee17 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

marcingrzejszczak commented Jun 30, 2023

jrhee17 left a comment

Choose a reason for hiding this comment

marcingrzejszczak commented Jul 3, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

trustin Aug 16, 2023 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

marcingrzejszczak commented Jul 10, 2023

jrhee17 commented Jul 10, 2023

marcingrzejszczak commented Jul 10, 2023

ikhoon left a comment

Choose a reason for hiding this comment

marcingrzejszczak commented Jul 14, 2023

ikhoon left a comment

Choose a reason for hiding this comment

trustin left a comment

Choose a reason for hiding this comment

marcingrzejszczak commented Aug 16, 2023

trustin commented Aug 17, 2023

marcingrzejszczak commented Aug 17, 2023

CLAassistant commented Jun 22, 2023 •

edited

jrhee17 Jun 27, 2023 •

edited

trustin Aug 16, 2023 •

edited