[Feature] useGithubAPI by default when using github checkver

[Lutra-Fs]

Feature Request

Is your feature request related to a problem? Please describe.

Scoop's GitHub checkver currently has inconsistent behavior that relies on fragile HTML scraping. When using the
GitHub matcher with additional properties like regex, it falls back to parsing HTML pages instead of using the GitHub
API, leading to:

• Brittle regex patterns that match against HTML URLs instead of clean version strings
• Performance issues due to downloading full HTML pages vs compact JSON API responses
• Maintenance burden as manifest authors must understand GitHub's HTML structure
• Reliability problems when GitHub changes their HTML layout, breaking existing manifests

Describe the solution you'd like

Modify Scoop's checkver logic to use GitHub API by default when using the GitHub matcher, regardless of additional
properties. The proposed solution would:

1. Enable jsonpath with GitHub matcher: Allow jsonpath to work with github property
2. Default to API mode: Use GitHub API for all GitHub matcher usage, not just simple cases
3. Simplify regex patterns: Extract version directly from tag_name field, then clean with simple regex

Examples of improved manifest structure:

  // Current (HTML scraping)
  "checkver": {
      "github": "https://github.com/wez/atomicparsley",
      "regex": "/releases/tag/(\\d+\\.\\d+\\.[a-f0-9]+)"
  }

  // Proposed (API + jsonpath)
  "checkver": {
      "github": "https://github.com/wez/atomicparsley",
      "jsonpath": "$.tag_name",
      "regex": "([\\d]+\\.[\\d]+\\.[\\d]+\\.[a-f0-9]+)"
  }

  // Simple cases remains the same
  "checkver": {
      "github": "https://github.com/owner/repo"
  }

We can also share the result between checkver and hash extraction, since it is using github API for github assets now.

Describe alternatives you've considered

One option could be introducing more controls within the checkver property itself. However, in most cases, it's primarily used to match a tag, so additional flexibility may not offer substantial benefit.

[Lutra-Fs]

cc: @z-Fng @stevehipwell

[stevehipwell]

@Lutra-Fs I've been thinking about this same problem and came to a very similar conclusion. I think my only concern would be backwards compatibility, I was thinking of a new primary key like githubjson. But with your proposal the initial implementation could keep the existing behaviour if a regex but no jsonpath was specified?

[z-Fng]

Scoop's GitHub checkver currently has inconsistent behavior that relies on fragile HTML scraping. When using the GitHub matcher with additional properties like regex, it falls back to parsing HTML pages instead of using the GitHub API.

According to the code, checkver uses the GitHub API in the following situations:

1. The checkver field is set to the string "github". e.g.,

"checkver" : "github"

2. The checkver field contains only a "github" subkey. (Restriction not working) e.g.,

"checkver" : [
    "github": "https://github.com/..."
]

3. URL matches the GitHub API regex pattern. e.g.,

"checkver" : [
    "url": "https://api.github.com/..."
]

I'm not sure exactly which situation you are referring to. Are you referring to the second case?

If so, in my tests, it seems that the restriction -- where the checkver field contains only a "github" subkey -- does not take effect.
In other words, in practice, the GitHub API is being used in most cases. And the current regex works just fine.

We may need to check with @rashil2000 (#4557) and remove this logic if we confirm it is unnecessary.

Scoop/bin/checkver.ps1

Lines 159 to 163 in b588a06

	if ($json.checkver.github) {
	$url = $json.checkver.github.TrimEnd('/') + '/releases/latest'
	$regex = $githubRegex
	if ($json.checkver.PSObject.Properties.Count -eq 1) { $useGithubAPI = $true }
	}

e.g., https://github.com/ScoopInstaller/Extras/blob/master/bucket/jstock.json

    "checkver": {
        "github": "https://github.com/yccheok/jstock/",
        "regex": "release_(\\d+)-(\\d+)-(\\d+)-(\\d+)",
        "replace": "${1}.${2}.${3}.${4}"
    },

cc: @rashil2000

[Lutra-Fs]

I think using JSON path would be easier than our approach by using regex releases/tag/?

[Lutra-Fs]

Error log:

    "checkver": {
        "github": "https://github.com/wez/atomicparsley",
        "jsonpath": "$.tag_name",
        "regex": "([\\d]+\\.[\\d]+\\.[\\d]+\\.[a-f0-9]+)"
    },

main on  github-checkver [✘!?]
❯ .\bin\checkver.ps1 atomicparsley -u -f
ConvertFrom-Json: D:\Applications\Scoop\apps\scoop\current\lib\json.ps1:130
Line |
 130 |      $result = $json | ConvertFrom-Json -ea stop
     |                        ~~~~~~~~~~~~~~~~~~~~~~~~~
     | Conversion from JSON failed with error: Unexpected character encountered while parsing value: <. Path '', line
     | 0, position 0.
atomicparsley: couldn't find '$.tag_name' in https://github.com/wez/atomicparsley/releases/latest

So I checked and assumed if ($json.checkver.PSObject.Properties.Count -eq 1) { $useGithubAPI = $true } is suspicious.

We can always go back to using checkver with api.github.com at url, but I think providing a unified approach should be better.

[z-Fng]

Did you configure a GitHub token?

Scoop/bin/checkver.ps1

Lines 221 to 224 in b588a06

	if ($useGithubAPI -and ($null -ne $GitHubToken)) {
	$url = $url -replace '//(www\.)?github.com/', '//api.github.com/repos/'
	$wc.Headers.Add('Authorization', "token $GitHubToken")
	}

It works fine on my side.

PS> jq '{checkver}' .\bucket\atomicparsley.json                                                                                           
{
  "checkver": {
    "github": "https://github.com/wez/atomicparsley",
    "jsonpath": "$.tag_name",
    "regex": "([\\d]+\\.[\\d]+\\.[\\d]+\\.[a-f0-9]+)"
  }
}
PS> .\bin\checkver.ps1 -App atomicparsley -f                                                                                              
https://api.github.com/repos/wez/atomicparsley/releases/latest
atomicparsley: couldn't match '([\d]+\.[\d]+\.[\d]+\.[a-f0-9]+)' in https://api.github.com/repos/wez/atomicparsley/releases/latest

[Lutra-Fs]

I think I might misconfigure it with the wrong config name ;; checking it now

[z-Fng]

The behavior when no GitHub token is configured is inconsistent with when a token is set, and there's not even any related logs. This is very counterintuitive for contributors. Should we consider changing this behavior?

[z-Fng]

We can also share the result between checkver and hash extraction, since it is using github API for github assets now.

I completely agree with this. When refactoring webpage fetching for hash extraction in #6535, I also planned to cache the results to avoid hitting the GitHub API multiple times.

[Lutra-Fs]

I think a warning might need to be added.

[Lutra-Fs]

I am currently debugging why the regex does not work

[z-Fng]

Looks like the regex has one unnecessary group. It should be ([\d]+.[\d]+.[\d]+.[a-f0-9]+)

P.S, More debug logs are needed; Scoop is really hard to debug.

[Lutra-Fs]

Yeah, I really agree with that. I have fixed that either

[Lutra-Fs]

I am currently thinking on reuse the API result between checkver and autoupdate hash extraction, depending on how we go with ScoopInstaller/GithubActions#64 and #6535. If we keep hitting the rate limit, I think a reuse here could be better, since it is containing calculated digest as well.