Router OOMs

[montykamath]

Describe the bug

When you execute the query (see query.graphql), the query always times out.
If you execute the query in parallel about 4 or 5 times, it will grow router memory until router OOMs. On my laptop memory consumption grows to >250 GB and then crashes.

The query is:

query Feed {
  postInterface1(id: "") {
    __typename
    feed {
      ...relatedPostFragment
      ... on Recommendation {
        postInterface1 {
          ...relatedPostFragment
        }
      }
    }
  }
}

fragment crosspostFragment on PostInterface2 {
  crosspost {
    postInterface1 {
      ... on PostInterface2 {
        __typename
        gallery {
          __typename
          items {
            __typename
            id
            name
          }
        }
      }
    }
  }
}

fragment relatedPostFragment on PostInterface1 {
  __typename
  ... on PostInterface2 {
    ...crosspostFragment
  }
}

To Reproduce

Steps to reproduce the behavior:

1. Clone this repo: https://github.com/montykamath/apollo-router-ir-20708-oom
2. See the readme for stalling router and rover
3. Compose the supergraph with make supergraph
4. Run the service with make run
5. Execute the query in query.graphql (also above)

Expected behavior

The query should execute and complete but it does not.

Output

When you execute the query (see query.graphql), the query always times out.
If you execute the query in parallel about 4 or 5 times, it will grow router memory until router OOMs. On my laptop memory consumption grows to >250 GB and then crashes.

Desktop (please complete the following information):

• OS: Mac OS
• Version Tahoe 26.0.1

Additional context

NA

[aaronArinder]

@montykamath: thanks for the repro! this is super interesting and we're looking at it (alongside other investigations into v2.4.0 that line up with this)

[montykamath-reddit]

@montykamath: thanks for the repro! this is super interesting and we're looking at it (alongside other investigations into v2.4.0 that line up with this)

@aaronArinder: I put up a PR to try to fix the problem. Let me know what you think: #8453

[montykamath-reddit]

Checking back in here to see if there is any progress on a fix?

[aaronArinder]

@montykamath: no significant progress yet! I'm going to spend some time today with your draft pr to see what we can learn from it; it'd be a much longer, broader conversation to get that pr merged (one we'd need to have with some internal teams on the hook for query planning), but I'm hoping it points us in the right direction and funds that conversation with something tangible. I'll try to reach back out once we know more, but if I don't in the next few days, please ping me (I keep a list of todos and will make sure responding here is on it but that list only every grows longer, never shorter!)

[montykamath-reddit]

@aaronArinder - Checking back in again to see if you are making any progress on a fix or considering incorporating any of my suggestions from my PR as a potential fix.

[montykamath-reddit]

@aaronArinder - Checking in again to see if you are making any progress here on this infinite loop that causes OOMs. This is critical and blocking for us at Reddit. Can anything be done to expedite investigation on your side?

[aaronArinder]

hey @montykamath: we've identified the core problem and have a two-pronged approach we're experimenting with (one for router-side mem safety and one for federation-side mem safety; the latter is important because it's the crate that actually does the query planning, but we want to have both in place for overall safety--no commitments, promises, and all the normal caveats of setting no expectations). In the short run, your best bet is to not send operations to the router that might spin out and cause OOMkills; I know that's probably unrealistic (I'm assuming you have containers getting OOMkilled, so you won't be able to identify the problematic operations to tell some specific client team to stop sending bad queries--that, or maybe you have a public api and can't control what's being sent over). There's a feature for allowlisting queries that might be useful, but requires a license to avoid getting rate-limited on the free tier and would take a good chunk of time to identify all those operations you want to allow (in the long run it does provide nice, fine-grained control to prevent situations like this)

alternatively, if you want to experiment in a lower env than prod, there's a nightly build with this operation-logging code that might help you identify the problematic query. There's an environment variable you can set (in bytes) for how much buffer to give before beginning to log out operations when nearing memory exhaustion: APOLLO_MEMORY_HEADROOM, which would look something like APOLLO_MEMORY_HEADROOM=524288000

[montykamath-reddit]

we've identified the core problem and have a two-pronged approach we're experimenting with (one for router-side mem safety and one for federation-side mem safety; the latter is important because it's the crate that actually does the query planning, but we want to have both in place for overall safety--no commitments, promises, and all the normal caveats of setting no expectations)

hey @aaronArinder thanks for the reply! thanks for offering suggestions on find the bad queries. those ideas might help when we encounter this problem again and get OOMing pods again. we're also blocked on some migrations because when we do migrate we get OOMs. great to know you are exploring a fix! please keep me posted!

[montykamath-reddit]

@aaronArinder - Following up again here. This is really important to us so I hope you are making progress toward a publicly available fix. Any updates?

[aaronArinder]

@montykamath: we're still making progress! it's slow because we have other customer interrupts; so, it'd actually be super helpful if you'd reach out to either your account rep or open a support ticket to get more weight behind this (it'll help us plan/prioritize, especially across teams)