Feature Request: GitLab Code Quality Reporting and --exit-zero for ruff format

[peterfarrell]

Currently, ruff check offers an -output-format option (e.g., gitlab) to generate reports for CI platforms. This is incredibly useful for integrating Ruff's linting results directly into GitLab's Code Quality reports.

We propose adding a similar --output-format=gitlab option to the ruff format command. This would allow users to report files with formatting issues as GitLab Code Quality findings. Crucially, the report would flag the entire file as an issue, rather than specific lines, aligning with how formatting violations are typically viewed.

Use Case for --output-format in ruff format

Our existing GitLab CI/CD setup for ruff check looks something like this:

"ruff_check":
  image:
    name: $RUFF_IMAGE
  variables:
    RUFF_CODE_QUALITY_REPORT: $CI_PROJECT_DIR/gl-code-quality-report-ruff_check.json
  stage: "lint"
  dependencies: []
  interruptible: true
  allow_failure:
    exit_codes:
      - 1 # allow the job to "fail" with "warnings"
  script:
    - ruff check $[[ inputs.ruff-cli-opts ]] --output-format=gitlab > $RUFF_CODE_QUALITY_REPORT
  artifacts:
    name: "Ruff-check Reports"
    when: always
    access: "developer"
    reports:
      codequality: $RUFF_CODE_QUALITY_REPORT

Extending this functionality to ruff format would enable us to create similar Code Quality reports for formatting inconsistencies, providing a more comprehensive overview of code health within GitLab.

Request for --exit-zero in ruff format

Additionally, for improved testing of our GitLab CI/CD component for Ruff, we request the inclusion of a --exit-zero option for ruff format.

Currently, when testing our format job, there's no native way to force ruff format to exit with a 0 status code, even if formatting issues are found. This complicates automated testing scenarios where we might want to check for issues without failing the CI job directly. Our current workaround involves trapping the exit code and handling it conditionally:

    # Capture ruff output and exit code separately
    # ruff format does not operate like ruff check and does not have an option to exit 0 always
    - |
      set +e  # Disable exit on error
      ruff format --check $[[ inputs.ruff-cli-opts ]] > $RUFF_OUTPUT 2>&1
      exit_code=$?
      set -e  # Re-enable exit on error (must come after capturing the exit code)
      echo "Ruff exit code: $exit_code"
    # Display the output and save to report file
    - |
      cat $RUFF_OUTPUT
      cp $RUFF_OUTPUT $RUFF_CODE_QUALITY_REPORT
    # Handle the feedback depending on the exit_code
    - |
      if [ $exit_code -eq 0 ]; then
        echo "✅ No formatting issues found"
      elif [ $exit_code -eq 1 ]; then
        echo "⚠️ Formatting issues found"
        if [[ "$[[ inputs.test-mode ]]" == "true" ]]; then
          echo "🧪 TEST MODE ENABLED - Treating formatting violations as success"
          exit 0
        else
          echo "❌ Formatting violations detected"
          exit 1
        fi
      else
        echo "❌ Unexpected error occurred"
        exit $exit_code  # Fail for unexpected errors
      fi

Implementing --exit-zero would significantly simplify our testing workflows, allowing for cleaner and more direct validation of our format jobs without complex shell logic.

We believe these additions would greatly enhance Ruff's utility within CI/CD pipelines, particularly for GitLab users. We welcome any thoughts or feedback on these proposals.

[ntBre]

Thanks for the thorough write-up!

We've been working on some new internal diagnostic infrastructure that will hopefully make --output-format=gitlab work pretty easily with the formatter.

--exit-zero also makes sense to me, but @MichaReiser might have more context on previous discussion/design around that.

[peterfarrell]

That's fantastic news about the internal diagnostic infrastructure!

--output-format=gitlab for ruff format

Adding --output-format=gitlab to ruff format would be a significant improvement for us. Currently, we have to pipe ruff format --check output to a text file and then use a custom bash script to generate a GitLab Code Quality report. This is quite complex, so a native output format would greatly simplify our CI/CD jobs.

--exit-zero for ruff format

The --exit-zero option would be incredibly valuable for our pipeline that tests our custom GitLab CI/CD Component for Ruff. We have a lint:format_fail job designed to intentionally produce formatting issues. However, we want this job to appear "green" in our pipeline because a subsequent verify:lint_format_fail job is responsible for analyzing its output (both the raw text and our generated GitLab Code Quality report) to confirm our component works as expected. --exit-zero would enable this specific testing scenario without resorting to complex workarounds.

GitLab Code Quality Report Severity

On a related note, we currently set the severity for formatting issues in our GitLab Code Quality reports to major. This works for our team, but it's worth considering if info might be a more universally accepted severity for formatting violations, as per the GitLab Code Quality report format specification.

Here's an example of our current report structure:

[
  {
    "description": "File is not formatted according to Ruff style.",
    "check_name": "ruff-format-violation",
    "fingerprint": "$fingerprint",
    "severity": "major",
    "location": {
      "path": "$reported_path",
      "lines": {
        "begin": 1
      }
    }
  }
]

Thanks again for considering these improvements!

[MichaReiser]

This is sort of the opposite of #8191

I'm not opposed to it but it also seems fairly easy to work around.

[peterfarrell]

@MichaReiser To clarify, does your "seems fairly easy to work around" comment apply only to the --exit-zero suggestion, or also to the --output-format?

I agree that --exit-zero has a possible trivial workaround, even if it lacks the consistency/parity of ruff check. However, making ruff format output parsable for CI is a much larger effort. In our case, this involves a substantial bash script (~150 lines), a custom CI image, and significant testing.

[MichaReiser]

@MichaReiser To clarify, does your "seems fairly easy to work around" comment apply only to the --exit-zero suggestion, or also to the --output-format?

Only the exit code. The output format is tracked in #10430