Support redis queue expiration

[ffje]

Redis Queue Expiration Feature

Description

This PR adds support for automatically expiring Redis queues after a period of inactivity. This is particularly useful for managing ephemeral queues that should be automatically cleaned up when they're no longer being used.

The implementation adds the following capabilities:

• Support for the x-expires queue argument for Redis transport, specifying milliseconds after which an inactive queue is deleted
• Support for the simpler expires parameter on Queue objects (in seconds) which is translated to x-expires internally
• Automatic TTL refresh on both queue read and write operations

Implementation

The implementation adds two key methods to the Redis transport Channel class:

• _maybe_update_queues_expire: Updates TTL on queue keys when there is activity
• _get_queue_expire: Extracts the expiration time from queue arguments

These methods ensure that Redis queue keys have TTL set appropriately based on the queue definition, and that the TTL gets refreshed on both get and put operations.

Testing

Added comprehensive integration tests that verify:

• Expiration is correctly set on queue declaration
• TTL is reset when publishing new messages
• TTL is reset when consuming messages
• Expiration works correctly with prioritized queues

Documentation

Added documentation of the x-expires option to the module docstring in the Transport Options section.

Usage Example

# Queue that expires after 60 seconds of inactivity
queue = kombu.Queue('temporary_queue', expires=60) 

[codecov]

Codecov Report

Attention: Patch coverage is 91.66667% with 2 lines in your changes missing coverage. Please review.

Project coverage is 81.57%. Comparing base (1645812) to head (6ae9a96).

Files with missing lines	Patch %	Lines
kombu/transport/redis.py	91.66%	1 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #2251      +/-   ##
==========================================
+ Coverage   81.55%   81.57%   +0.02%     
==========================================
  Files          77       77              
  Lines        9541     9565      +24     
  Branches     1162     1165       +3     
==========================================
+ Hits         7781     7803      +22     
- Misses       1568     1569       +1     
- Partials      192      193       +1     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[Nusnus]

You can run tox -e lint locally to check the lint errors instead of waiting for the CI

[ffje]

@Nusnus would appreciate feedback and guideline on the further actions. Should I cover by unit tests the code additionally? Since it seems covered in integration tests but codecov is not aware of those

[Nusnus]

@ffje

@Nusnus would appreciate feedback and guideline on the further actions. Should I cover by unit tests the code additionally? Since it seems covered in integration tests but codecov is not aware of those

I will need to properly review this change which might take some time (days to week or two).

In the meantime, and until I'll add it to Kombu's CI later this year, you can check manually if the entire test suits of Celery are passing with a patched Kombu running this change.

Unit, integration and smoke tests.

You don't need to add any new test, just make sure nothing is broken due to the changes of this PR.

Let me know if you need any assistance running the CI locally or configuring tox for your patched Kombu.

TL;DR remove Kombu from all req.txt files in Celery and add to the deps section in tox.ini to install with -e relative/path/to/kombu & check python-package.yml for running the tests.

[ffje]

@Nusnus I was able to run Celery test suite for python3.9 and it's looking good. I'm actually not exactly sure that my local Kombu version was used for that (but the -e statement was visible during pip install, so my assumption that it went well), but I indeed followed your guide (removing from req Kombu and adding -e local/path to deps of tox.ini).

Please, let me know if any further actions are required from my side. Thanks

[ffje]

hey @auvipy @Nusnus could you please rerun 3.11 Unit workflow? It hang in a middle of the execution.
would appreciate any updates on the PR as well, thanks in advance!

[Nusnus]

hey @auvipy @Nusnus could you please rerun 3.11 Unit workflow? It hang in a middle of the execution.
would appreciate any updates on the PR as well, thanks in advance!

Done.
Apologies for the delay with the review 🙏

[ffje]

Hey @Nusnus @auvipy hope you're having a great week. Do you have any timelines in mind when it would possible to get the PR reviewed? Thanks in advance!

[auvipy]

this is slated for kombu 5.6 so wait a little bit, we are in the middle of finaliizing v5.5 release

[auvipy]

hey @auvipy do you need any help from my side or it's going alright?

yes, I need your help, it would be great if you can re check and make the CI green again. thanks for your patience.

[ffje]

should be good now, please approve workflows @auvipy

[auvipy]

sadly the integration tests are also failing now! may be they also need some adjustment?

[ffje]

@auvipy has to be fixed now 🙂 are there any updates regarding 5.7.0?

[auvipy]

@auvipy has to be fixed now 🙂 are there any updates regarding 5.7.0?

I can merge it by the end of may, but a release need some more qualified PR's to be merged by then. you can still use it from github

[ffje]

Hey @auvipy are we still on track for the end of May?

[Nusnus]

Hey @auvipy are we still on track for the end of May?

Basically yeah, I'm planning on finalizing the last issues by then and dedicate the last week of May for the next release, so we'll be able to move forward with this change as well 🙏

EDIT:
Just to clarify, I'm talking about v5.6 for Celery

[auvipy]

Hey @auvipy are we still on track for the end of May?

there is no need for rushing for this pr. as this is planned for v5.7 it will be for sure. just we are ironing down new bugs popping up from the new release. so no worries.

[ffje]

Hey folks, hope you're doing great @auvipy @Nusnus. Do you have any updates on timelines by any chance?

[ffje]

Ping @auvipy @Nusnus

[Nusnus]

@ffje

Hey folks, hope you're doing great @auvipy @Nusnus. Do you have any updates on timelines by any chance?

Apologies on the delays, I'm preparing something big for the community so my capacity is a bit low these weeks.

TL;DR
The general plan is to work towards our next minor release in the upcoming weeks, so we can stop supporting Python 3.8, and then we can progress with new features.

Our general priorities are:

1. Quorum queues bug fixes above all.
2. Get rid of Python 3.8
3. Start adding new features.

With an emphasis on small iterative releases, to avoid collecting technical debt from our latest updates.

[auvipy]

Hey @auvipy are we still on track for the end of May?

there is no need for rushing for this pr. as this is planned for v5.7 it will be for sure. just we are ironing down new bugs popping up from the new release. so no worries.

this should be self explanatory. we got another intermediatary release of v5.6, after that we are all n for 5.7... and this is on the review q....