Add hyperelliptic curves using the smooth model

[GiacomoPope]

Overview

This PR includes a whole new module into schemes which implements hyperelliptic curves over the smooth model, and focuses on implementing decent arithmetic for Jac(H) for all reasonable cases, something which is not done properly in the current implementation due to limitations of the curve model used.

The idea is that hyperelliptic_curves_sm will be available with the old model for some time, until someone decides we should depreciate the old model in favour of this new implementation.

Motivation

The current implementation of hyperelliptic curves in SageMath uses the projective plane model. Although this works nicely enough for imaginary curves with only one point at infinity, it is not descriptive enough for the real models. My gut feeling is the projective plane model was used as in early Sage days this was easier to do and allowed hyperelliptic curves to be supported at all. Mathematically, I believe it makes much more sense to use a weighted projective model to have this new description but it requires more tools which we only have thanks to the work of others in sage since it was released.

This PR is a total rewrite of the hyperelliptic curve classes to instead use the smooth model for hyperelliptic curves which facilitates implementing arithmetic of Jacobians of hyperelliptic curves for (almost) all cases. In particular, now if one tries to perform arithmetic on Jac(C) one either gets the correct value or a well-handled error instead of just returning something which is wrong.

The hope is that with this new model for the curves, more recent research in the area of hyperelliptic curves and their jacobians can be more easily integrated. As a personal example, being able to compute arithmetic in Jac(H) for the real model unconditionally would help with the implementation of genus two isogenies which are "in vogue" right now in the cryptography world.

Content of PR

This PR looks WAY bigger than it is, as it duplicates ALL code from hyperelliptic_curves and then has modifications to allow this to work in the new curve model. I believe the best thing to do in the long run is to depreciate the old hyperelliptic curve impl and have this replace it, but this will take years(?) so I think we'll just have to have both side by side for a while.

This PR still needs a lot of work including:

• more tests
• better comments and docstrings
• potentially refactoring of which files belong where

but the code has sat at this stage for a long time without much more progress from the three of us, so I think the best thing to do is open up the PR and try and get some extra attention on getting this code ready to be included.

One benefit is that all this code is "new" in that it should not conflict with anyone else's work, so we can get this code into a good position and keep adding to it with more interesting maths once the base layer is in place

Example of better arithmetic

For example, this code fixed the following issue:

#32024

sage: R.<x> = QQ[]
sage: f = 144*x^6 - 240*x^5 + 148*x^4 + 16*x^3 - 16*x^2 - 4*x + 1
sage: H = HyperellipticCurveSmoothModel(f)
sage: J = Jacobian(H)
sage: P = J(H(0,1))-J(H(0,-1))
sage: (5*P).is_zero()
False
sage: 
sage: H = HyperellipticCurve(f)
sage: J = Jacobian(H)
sage: P = J(H(0,1))-J(H(0,-1))
sage: (5*P).is_zero()
True

and is also related to the issues #37109 #37093 #37101 #37626

Help!

This code needs more work, and lots of time spent on the review which I don't think is a reasonable thing to do for one person. So if this area interests you any help would be appreciated (either in the review of some of the code or some extra commits which tidy up aspects)

[github-actions]

Documentation preview for this PR (built with commit 7c7138a; changes) is ready! 🎉
This preview will update shortly after each push to this PR.

[GiacomoPope]

Is this all held up by 40070 atm?

[vincentmacri]

Is this all held up by 40070 atm?

If it depends on #40070 then yes. I commented there that I want the docbuild to be working before I try to review it further (if it works locally but CI is broken then fine, just let me know).

As for reviewing, I will have limited time to review this until early/mid September (when I defend my Master's thesis on curve arithmetic). I have the time for higher-level discussions before then, but I won't have the time to really look at the code and give a proper review until after I defend. I expect I'll have time for a detailed review after September 8th, maybe a few days earlier or later depending on how many revisions I'm asked to make. The first thing I'll ask when I review it is to make sure all the tests pass and that the documentation builds successfully, so if you can get that working by September 8th, it would help me a lot in reviewing this faster.

[vincentmacri]

Can the duplicated code be moved into a file of utility functions or something like that? Would it be possible to change the old implementation to act as a wrapper for your new one so we don't have two implementations of basically the same thing lying around until we deprecate the old interface?

I don't see how this can be done. As mentioned, our implementation is incompatible with the existing implementation. Indeed, the main reason why this PR exists at all is because the existing implementation is wrong in the even degree model case (e.g. #32024 as referenced in the PR description). We fixed this by using a completely separate representation of points (IIRC) which is why it's incompatible.

If the existing implementation is mathematically wrong then I'm comfortable breaking existing code that uses it (because I would consider that code to be already broken), but we should double-check what our formal deprecation policy is for situations like this where we have a public API that is wrong for certain inputs in a way that cannot be fixed without an API change. For imaginary models (which are not broken in Sage), isn't what we do now just weighted projective space with the same weight on all three coordinates? That would seem to be a special case of the weighted projective models you're implementing.

In the meantime, you could add a stopgap warning and deprecation warning for real models to speed up that deprecation.

[vincentmacri]

How should we handle this PR? I propose we can PR e.g. "file by file", or PR the class structure first and actual useful methods later.

I've thought more about this. I see that the PR has code for finite fields, the rational field, and p-adic fields. You could split it up to add one at a time. Unless there are technical reasons preventing this, my preference would be finite fields, then rational field, then p-adic fields just because that's the order of my familiarity with the subject so it will be easiest for me to review it in that order. I won't complain if you want to do the rational field before finite fields, but if you do p-adic fields first then I'll have to simultaneously read up on p-adic hyperelliptic arithmetic and get familiar with your code which will slow down how quickly I can review it.

[vincentmacri]

I have time to start looking at this now.

I understand there is this PR and the related #40070. Are there any others? Where should I start? Which are ready for review?

[vincentmacri]

@GiacomoPope Are there any PRs related to this work that are ready for review?

[GiacomoPope]

As far as this code is considered I think the only PR above it is one by @grhkm21 but I've not checked how that's going. Sorry.

[vincentmacri]

As far as this code is considered I think the only PR above it is one by @grhkm21 but I've not checked how that's going. Sorry.

Okay. @user202729 has revived that work in #41170 and I'm reviewing it. If you could merge the latest develop into this PR, and make sure the linter and tests are passing, then I can review this PR after #41170 is merged. You'll probably need to merge develop again once #41170 is merged since this PR depends on it.

[GiacomoPope]

I see that progress has been made with the projective model, so I'm going to open this up for review -- there's probably a lot to adjust and there might be some necessary changes but I think it's best to work with this as some enormous PR rather than lots of small ones as the time I have to work on this is fleeting and as it's just a lot of new code it should hopefully only help research rather than break any existing code.

[vincentmacri]

#41170 has been approved. Once #41170 is merged into develop, merge develop again into this PR. I'll ignore the code here for the time being about weighted projective curves/points since that's all in #41170.

To speed up the review, make sure you've done the following:

• Linter and (relevant) tests are passing
  • I'm not going to review a PR this big until the linter is passing and all relevant tests are passing. You can of course ping me if you want to discuss how to resolve some failure. Most linter failures can be fixed by ruff.
• Docstrings for all modules, classes, and methods
  • AUTHORS block for module docstrings
  • Link to relevant papers/textbooks for where algorithms come from (especially GHM balanced divisors)
• Copyright headers on all files
• Sufficient tests
  • You can test an objects G in Sage with TestSuite(G).run(). TestSuite(G).run(verbose=True) is useful for debugging. This will run a generic test suite for whatever type of object you've implemented, as long as you've inherited from the correct classes and have the correct categories. So if you've implemented an abelian group (like the Jacobian) it will run a full test suite that all abelian group axioms hold for a few elements. You need to implement some_elements() and (I think) _an_element_() to provide inputs to TestSuite, but that should be done anyway. random_element is also useful to have, and can be used to implement some_elements and _an_element_. Ideally, every object should run its TestSuite at least once in the doctests.

I think it's best to work with this as some enormous PR rather than lots of small ones

I think that makes sense. It will probably be faster than separate smaller PRs, but it will still take a while to review a PR of this size.

the time I have to work on this is fleeting

Just to clarify, do you mean you will soon have no more time to work on this, or that you only have small amounts of time here and there to work on this, but with no upcoming endpoint where you can't work on it anymore?

My next 3 weeks are quite busy, until December 15. Would you prefer I do many small passes over different parts of your code looking for different things each time over the next few weeks as I am able, or would you prefer a large review of everything at once in about 3-4 weeks?

[GiacomoPope]

Sorry for the delay in response. I have small amounts of time semi-infrequently rather than lost of free time, which almost never happens now. I am grateful for any work you do on the review here though and will try and help however I can to get this reviewed efficiently

[vincentmacri]

Okay. I do really want this to get into Sage (I think I have my own implementation of balanced divisor arithmetic for spit hyperelliptic curves lying around somewhere, but it's somewhat hacky and not remotely in a state to be made into a PR). I'll take a deep dive into your code in about 3 weeks when I'll have more time. If you could get this to a state where all relevant CI tests passing in the meantime, that would be a big help.

So I know which sources to look at when reviewing, does your balanced divisors implementation align closer to the description in the GHM paper or the description in Galbraith's book? Or a bit of both? I find the notation in Galbraith's book to be a bit cleaner, but it also has some non-obvious typos (which I know you know: when I sent some corrections to Galbraith a while back he forwarded me an email you sent him with some other corrections).

[fchapoton]

I allowed myself a little fix about the linter warnings (partial fix only)

[GiacomoPope]

I allowed myself a little fix about the linter warnings (partial fix only)

Thanks so much @fchapoton, I'll attempt to fix the rest, now we have uv this is much easier for me to do (shows how long ive been ignoring sage that im only trying uv now...)

So I know which sources to look at when reviewing, does your balanced divisors implementation align closer to the description in the GHM paper or the description in Galbraith's book? Or a bit of both?

So a bit of both, probably for the same reasons as you. I like Galbraith's book, but the typos meant i needed the paper to fix a few pieces so in the end there's an amalgamation of both resources.

(Aside) did you find a correction to the proof with the typo? I meant to, but never got around to it.

[GiacomoPope]

It looks like the documentation build is broken in other PR, so maybe this is unrelated to this branch?

[vincentmacri]

(Aside) did you find a correction to the proof with the typo? I meant to, but never got around to it.

It's been a while since I looked at it, I can't recall.

[vincentmacri]

It looks like the documentation build is broken in other PR, so maybe this is unrelated to this branch?

Yeah, doc build is indeed broken on CI right now.