[Bug] Compiling version 8 fails with a no-git-repo

[nilason]

Describe the bug
Compiling with tarball, eg. https://grass.osgeo.org/grass80/source/grass-8.0.0.tar.gz fails, while works with a git repo.

if [ "" != "" ] ; then GISRC=/Users/nilason/Downloads/grass-8.0.0/dist.x86_64-apple-darwin21.2.0/demolocation/.grassrc80 GISBASE=/Users/nilason/Downloads/grass-8.0.0/dist.x86_64-apple-darwin21.2.0 PATH="/Users/nilason/Downloads/grass-8.0.0/dist.x86_64-apple-darwin21.2.0/bin:/Users/nilason/Downloads/grass-8.0.0/dist.x86_64-apple-darwin21.2.0/bin:/Users/nilason/Downloads/grass-8.0.0/dist.x86_64-apple-darwin21.2.0/scripts:$PATH" PYTHONPATH="/Users/nilason/Downloads/grass-8.0.0/dist.x86_64-apple-darwin21.2.0/etc/python:/Users/nilason/Downloads/grass-8.0.0/dist.x86_64-apple-darwin21.2.0/gui/wxpython:$PYTHONPATH" LD_RUN_PATH="/Users/nilason/Downloads/grass-8.0.0/dist.x86_64-apple-darwin21.2.0/bin:/Users/nilason/Downloads/grass-8.0.0/dist.x86_64-apple-darwin21.2.0/bin:/Users/nilason/Downloads/grass-8.0.0/dist.x86_64-apple-darwin21.2.0/scripts:/Users/nilason/Downloads/grass-8.0.0/dist.x86_64-apple-darwin21.2.0/lib:/Users/nilason/Downloads/grass-8.0.0/dist.x86_64-apple-darwin21.2.0/lib:" LC_ALL=C LANG=C LANGUAGE=C  --html-description < /dev/null | grep -v '</body>\|</html>' > vectorascii.tmp.html ; fi
VERSION_NUMBER=8.0.0 VERSION_DATE=2022 MODULE_TOPDIR=../.. \
        python3 /Users/nilason/Downloads/grass-8.0.0/dist.x86_64-apple-darwin21.2.0/utils/mkhtml.py vectorascii > /Users/nilason/Downloads/grass-8.0.0/dist.x86_64-apple-darwin21.2.0/docs/html/vectorascii.html
Traceback (most recent call last):
  File "/Users/nilason/Downloads/grass-8.0.0/dist.x86_64-apple-darwin21.2.0/utils/mkhtml.py", line 141, in download_git_commit
    response = urlopen(url, *args, **kwargs)
  File "/Users/nilason/Downloads/grass-8.0.0/dist.x86_64-apple-darwin21.2.0/utils/mkhtml.py", line 113, in urlopen
    return urlrequest.urlopen(request, *args, **kwargs)
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/urllib/request.py", line 214, in urlopen
    return opener.open(url, data, timeout)
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/urllib/request.py", line 523, in open
    response = meth(req, response)
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/urllib/request.py", line 632, in http_response
    response = self.parent.error(
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/urllib/request.py", line 561, in error
    return self._call_chain(*args)
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/urllib/request.py", line 494, in _call_chain
    result = func(*args)
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/urllib/request.py", line 641, in http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 403: rate limit exceeded

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/nilason/Downloads/grass-8.0.0/dist.x86_64-apple-darwin21.2.0/utils/mkhtml.py", line 658, in <module>
    git_commit = get_last_git_commit(
  File "/Users/nilason/Downloads/grass-8.0.0/dist.x86_64-apple-darwin21.2.0/utils/mkhtml.py", line 232, in get_last_git_commit
    response = download_git_commit(
  File "/Users/nilason/Downloads/grass-8.0.0/dist.x86_64-apple-darwin21.2.0/utils/mkhtml.py", line 168, in download_git_commit
    gs.warning(
AttributeError: 'NoneType' object has no attribute 'warning'
make[3]: *** [/Users/nilason/Downloads/grass-8.0.0/dist.x86_64-apple-darwin21.2.0/docs/html/vectorascii.html] Error 1
test -d OBJ.x86_64-apple-darwin21.2.0 || mkdir -p OBJ.x86_64-apple-darwin21.2.0

[nilason]

If this is not only me, we need a quick fix, and an equally quick 8.0.1.

[neteler]

Mhh, and in RC2 it was still compiling?

[nilason]

I don't know the reason for:

HTTP Error 403: rate limit exceeded

in

grass/utils/mkhtml.py

Line 131 in 65675c8

def download_git_commit(url, response_format, *args, **kwargs):

but

grass/utils/mkhtml.py

Line 51 in 65675c8

gs = None

causes failure for:

grass/utils/mkhtml.py

Line 168 in 65675c8

gs.warning(

[nilason]

I have only worked with git based repo.

[ninsbl]

Yes, I am sure ths is the same issue as #2138. And sorry, I am afraid I am invoved in the reason for this.

In the change you reference, the date of the latest change is taken from the

1. the git log, if that fails,
2. the github API, if that fails,
3. file attributes.

The 403 comes likely from the rate limit of the GitHub API (there is a limited number of allowed calls I think). So, 2) was probebly not such a great idea I had.

It did not show up in CI, because CI is git based...

Sorry for all that. Maybe 432ceb7 should be reverted as a quick fix?

[nilason]

Replacing those gs.warning() with print(), just to get pass that failure, resulted in a new point of failure:

VERSION_NUMBER=8.0.0 VERSION_DATE=2022 MODULE_TOPDIR=../.. \
        python3 /Users/nilason/Downloads/grass-8.0.0/dist.x86_64-apple-darwin21.2.0/utils/mkhtml.py vectorascii > /Users/nilason/Downloads/grass-8.0.0/dist.x86_64-apple-darwin21.2.0/docs/html/vectorascii.html
VERSION_NUMBER=8.0.0 /Users/nilason/Downloads/grass-8.0.0/dist.x86_64-apple-darwin21.2.0/utils/g.html2man.py "/Users/nilason/Downloads/grass-8.0.0/dist.x86_64-apple-darwin21.2.0/docs/html/vectorascii.html" "/Users/nilason/Downloads/grass-8.0.0/dist.x86_64-apple-darwin21.2.0/docs/man/man1/vectorascii.1"
Traceback (most recent call last):
  File "/Users/nilason/Downloads/grass-8.0.0/dist.x86_64-apple-darwin21.2.0/utils/g.html2man.py", line 39, in main
    p.feed(line)
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/html/parser.py", line 110, in feed
    self.goahead(0)
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/html/parser.py", line 170, in goahead
    k = self.parse_starttag(i)
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/html/parser.py", line 344, in parse_starttag
    self.handle_starttag(tag, attrs)
  File "/Users/nilason/Downloads/grass-8.0.0/dist.x86_64-apple-darwin21.2.0/utils/ghtml.py", line 281, in handle_starttag
    while not self.is_allowed(tag):
  File "/Users/nilason/Downloads/grass-8.0.0/dist.x86_64-apple-darwin21.2.0/utils/ghtml.py", line 277, in is_allowed
    return tag not in self.excluded and tag in allowed[self.top()]
KeyError: None

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/nilason/Downloads/grass-8.0.0/dist.x86_64-apple-darwin21.2.0/utils/g.html2man.py", line 74, in <module>
    main()
  File "/Users/nilason/Downloads/grass-8.0.0/dist.x86_64-apple-darwin21.2.0/utils/g.html2man.py", line 43, in main
    % (infile, err.lineno, err.offset, err.msg)
AttributeError: 'KeyError' object has no attribute 'lineno'
make[3]: *** [/Users/nilason/Downloads/grass-8.0.0/dist.x86_64-apple-darwin21.2.0/docs/man/man1/vectorascii.1] Error 1

[update] looks like this: #2100 (comment)

[nilason]

For testing purposes I downloaded https://github.com/OSGeo/grass/tree/04e3de8fa85f462b3f0acbf217cad08268c837a8 (release branch 8, a few commits ago).
Compilation was successful also without being a git repo.

[nilason]

FYI, reverting #2100 backport related commits (listed below), on release branch 8 solves compiling issues.

Revert "utils/mkhtml.py: fix black formatting"
f6dd02f31dce0bcd0ac0e56c3a6a6a4f822d5f94

Revert "utils/mkhtml.py: fix get core modules commit from local git repo and remote GitHub API server (#2113)"
8bfae091e4cae9479fa7618f02d652d9362aa679

Revert "utils/mkhtml.py: fix git cherry-pick commit 223dc88 (#2111)"
60e6b56db86342473d0acccb3b35367eba512c01

Revert "utils/mkhtml.py: get and set git commit and commit date in addon manual page (#2100)"
223dc8824fbc37333ec3c2e1d3afd0223168e3b4

[tmszi]

I'm working on a fix this bug.

[tmszi]

Replacing those gs.warning() with print(), just to get pass that failure, resulted in a new point of failure:

VERSION_NUMBER=8.0.0 VERSION_DATE=2022 MODULE_TOPDIR=../.. \
        python3 /Users/nilason/Downloads/grass-8.0.0/dist.x86_64-apple-darwin21.2.0/utils/mkhtml.py vectorascii > /Users/nilason/Downloads/grass-8.0.0/dist.x86_64-apple-darwin21.2.0/docs/html/vectorascii.html
VERSION_NUMBER=8.0.0 /Users/nilason/Downloads/grass-8.0.0/dist.x86_64-apple-darwin21.2.0/utils/g.html2man.py "/Users/nilason/Downloads/grass-8.0.0/dist.x86_64-apple-darwin21.2.0/docs/html/vectorascii.html" "/Users/nilason/Downloads/grass-8.0.0/dist.x86_64-apple-darwin21.2.0/docs/man/man1/vectorascii.1"
Traceback (most recent call last):
  File "/Users/nilason/Downloads/grass-8.0.0/dist.x86_64-apple-darwin21.2.0/utils/g.html2man.py", line 39, in main
    p.feed(line)
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/html/parser.py", line 110, in feed
    self.goahead(0)
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/html/parser.py", line 170, in goahead
    k = self.parse_starttag(i)
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/html/parser.py", line 344, in parse_starttag
    self.handle_starttag(tag, attrs)
  File "/Users/nilason/Downloads/grass-8.0.0/dist.x86_64-apple-darwin21.2.0/utils/ghtml.py", line 281, in handle_starttag
    while not self.is_allowed(tag):
  File "/Users/nilason/Downloads/grass-8.0.0/dist.x86_64-apple-darwin21.2.0/utils/ghtml.py", line 277, in is_allowed
    return tag not in self.excluded and tag in allowed[self.top()]
KeyError: None

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/nilason/Downloads/grass-8.0.0/dist.x86_64-apple-darwin21.2.0/utils/g.html2man.py", line 74, in <module>
    main()
  File "/Users/nilason/Downloads/grass-8.0.0/dist.x86_64-apple-darwin21.2.0/utils/g.html2man.py", line 43, in main
    % (infile, err.lineno, err.offset, err.msg)
AttributeError: 'KeyError' object has no attribute 'lineno'
make[3]: *** [/Users/nilason/Downloads/grass-8.0.0/dist.x86_64-apple-darwin21.2.0/docs/man/man1/vectorascii.1] Error 1

[update] looks like this: #2100 (comment)

The print function cannot be used here, because the whole html page is printed to stdout, and if you used print, this message would be part of html page.