[Bug]: Royalty payments overflow if NFT + AH royalties exceed 100%

[KartikSoneji]

Which package is this bug report for?

auction-house

Which Type of Package is this bug report for?

Rust Contract

Issue description

Steps to reproduce:

1. Create auction house with sfbp of 200 = 2% royalty.
2. List NFT with sfbp of 9801 = 98.01% royalty.
3. Attempt to buy nft.
4. The sale will fail with an error.

The Auction house currently pays both the nft and ah royalty on the total sale amount.
Eg: if the nft has 5% royalty and the ah has 2% royalty, for a 1 sol sale:

• ah gets 2% of 1 sol = 0.02 sol
• nft creators get 5% of 1 sol = 0.05 sol
• seller gets the balance = 0.93 sol

The issue with this is if the nft and ah's royalty % combined exceed 100% then it will cause the sale to fail
Eg: if the nft has 60% royalty and the ah has 60% royalty, for a 1 sol sale:

• ah gets 60% of 1 sol = 0.6 sol
• nft creators get 60% of 1 sol = 0.6 sol
• the selling price was 1 sol but 0.6 + 0.6 = 1.2 > 1; the sale fails

I can think of three potential solutions, but none of them are perfect.
Consider: an nft with 50% royalty and an ah with 70% royalty, for a 1 sol sale:

1. Pay creators first, then charge auction house fees on remaining amount
   Downside: ah fees depends on royalty %, no ah fees for nfts with 100% royalty

   • nft creators get 50% of 1 sol = 0.5 sol
   • ah gets 70% of (1 sol - 0.5 sol) = 70% of 0.5 sol = 0.35 sol
   • seller gets 1 sol - 0.5 sol - 0.35 sol = 0.15 sol

2. Pay ah first fees, then pay creators royalty on remaining amount
   Downside: creator royalty depends on ah fees, no creator royalty for ah with 100% royalty

   • ah gets 70% of 1 sol = 0.7 sol
   • nft creators get 50% of (1 sol - 0.7 sol) = 70% of 0.3 sol = 0.21 sol
   • seller gets 1 sol - 0.7 sol - 0.21 sol = 0.09 sol

3. Cap ah + nft royalty to 100%, with the loss borne proportionally by the creator and auction house
   Downside: ah fee and creator royalty amount will vary based on nft royalty %

   • normalize ah and nft fees to 100%
     • total sfbp = ah sfbp + nft sfbp = 12000
     • new ah sfbp = (ah sfbp) * 10000/(total sfbp) = 5833 = 58.33%
     • new nft sfbp = (nft sfbp) * 10000/(total sfbp) = 4167 = 41.67%
   • ah gets 58.33% of 1 sol = 0.5833 sol
   • nft creators get 41.67% of 1 sol = 0.4167 sol
   • seller gets 1 sol - 0.5833 sol - 0.4167 sol = 0 sol

Relevant log output

No response

Priority this issue should have

Medium (should be fixed soon)