You need to place a PacletSite.mz file which lists the available versions in the git. Next to that, you need a folder "Paclets" that contains the paclets.

The PacletSite.mz is a zipped .m file that basically contains a list of the PacletInfo.wl for each version but then slightly differently formatted.

    PacletSite[
        Paclet["Name" -> "QMRITools", "Version" -> "3.8.0", 
          "WolframVersion" -> "13.0+", "SystemID" -> All, 
          "Description" -> "Toolbox for Quantitative MRI.", 
          "Extensions" -> {...}],
        Paclet["Name" -> "QMRITools", "Version" -> "3.9.0", 
          "WolframVersion" -> "13.3+", "SystemID" -> All, 
          "Description" -> "Toolbox for Quantitative MRI.", 
          "Extensions" -> {...}]
    ]

for more information see here

It would be more useful if Mathematica would recognize the github.com/.../releases page a valid paclet location instead of the "Paclets" folder.