Upgrade Ruff to 0.6.8 #1588

DimitriPapadopoulos · 2024-06-29T09:06:07Z

warning: RUF025 has been remapped to C420.
Selection of deprecated rule UP027 is not allowed when preview is enabled.

DylanLukes · 2024-08-26T22:19:32Z

backend/src/hatchling/metadata/spec.py

@@ -174,7 +174,7 @@ def project_metadata_from_core_metadata(core_metadata: str) -> dict[str, Any]:
                    left, _, right = marker
                    if left.value == 'extra':
                        extra = right.value
-                        del markers[i]
+                        del markers[i]  # noqa: B909


This is safe currently because there is always a break after the deletion, but it would be preferable to do this safely rather than suppress the warning?

I guess we could simply extract i in the loop, then use ioutside the loop.

Can I change that in a different PR? Somehow I cannot get the fix right for now.

Let me take a quick crack at it...

I will PR your branch soon later today with a rewrite, I have something working but I have some meetings.

DylanLukes

There are numerous rules enabled here that are in preview only. These should probably be disabled?


warning: Selection `E112` has no effect because preview is not enabled.
warning: Selection `E113` has no effect because preview is not enabled.
warning: Selection `E115` has no effect because preview is not enabled.
warning: Selection `E116` has no effect because preview is not enabled.
warning: Selection `E201` has no effect because preview is not enabled.
warning: Selection `E202` has no effect because preview is not enabled.
warning: Selection `E203` has no effect because preview is not enabled.
warning: Selection `E211` has no effect because preview is not enabled.
warning: Selection `E221` has no effect because preview is not enabled.
warning: Selection `E222` has no effect because preview is not enabled.
warning: Selection `E223` has no effect because preview is not enabled.
warning: Selection `E224` has no effect because preview is not enabled.
warning: Selection `E225` has no effect because preview is not enabled.
warning: Selection `E226` has no effect because preview is not enabled.
warning: Selection `E227` has no effect because preview is not enabled.
warning: Selection `E228` has no effect because preview is not enabled.
warning: Selection `E231` has no effect because preview is not enabled.
warning: Selection `E241` has no effect because preview is not enabled.
warning: Selection `E242` has no effect because preview is not enabled.
warning: Selection `E251` has no effect because preview is not enabled.
warning: Selection `E252` has no effect because preview is not enabled.
warning: Selection `E261` has no effect because preview is not enabled.
warning: Selection `E262` has no effect because preview is not enabled.
warning: Selection `E265` has no effect because preview is not enabled.
warning: Selection `E266` has no effect because preview is not enabled.
warning: Selection `E271` has no effect because preview is not enabled.
warning: Selection `E272` has no effect because preview is not enabled.
warning: Selection `E273` has no effect because preview is not enabled.
warning: Selection `E274` has no effect because preview is not enabled.
warning: Selection `E275` has no effect because preview is not enabled.
warning: Selection `E502` has no effect because preview is not enabled.
warning: Selection `W391` has no effect because preview is not enabled.
warning: Selection `B909` has no effect because preview is not enabled.
warning: Selection `FURB110` has no effect because preview is not enabled.
warning: Selection `FURB113` has no effect because preview is not enabled.
warning: Selection `FURB116` has no effect because preview is not enabled.
warning: Selection `FURB118` has no effect because preview is not enabled.
warning: Selection `FURB131` has no effect because preview is not enabled.
warning: Selection `FURB132` has no effect because preview is not enabled.
warning: Selection `FURB142` has no effect because preview is not enabled.
warning: Selection `FURB145` has no effect because preview is not enabled.
warning: Selection `FURB148` has no effect because preview is not enabled.
warning: Selection `FURB152` has no effect because preview is not enabled.
warning: Selection `FURB157` has no effect because preview is not enabled.
warning: Selection `FURB164` has no effect because preview is not enabled.
warning: Selection `FURB166` has no effect because preview is not enabled.
warning: Selection `FURB171` has no effect because preview is not enabled.
warning: Selection `FURB180` has no effect because preview is not enabled.
warning: Selection `FURB192` has no effect because preview is not enabled.
warning: Selection `PLC0415` has no effect because preview is not enabled.
warning: Selection `PLC1901` has no effect because preview is not enabled.
warning: Selection `PLC2701` has no effect because preview is not enabled.
warning: Selection `PLC2801` has no effect because preview is not enabled.
warning: Selection `PLE0304` has no effect because preview is not enabled.
warning: Selection `PLE1141` has no effect because preview is not enabled.
warning: Selection `PLE4703` has no effect because preview is not enabled.
warning: Selection `PLR0202` has no effect because preview is not enabled.
warning: Selection `PLR0203` has no effect because preview is not enabled.
warning: Selection `PLR1733` has no effect because preview is not enabled.
warning: Selection `PLR6104` has no effect because preview is not enabled.
warning: Selection `PLR6201` has no effect because preview is not enabled.
warning: Selection `PLR6301` has no effect because preview is not enabled.
warning: Selection `PLW0108` has no effect because preview is not enabled.
warning: Selection `PLW0177` has no effect because preview is not enabled.
warning: Selection `PLW1514` has no effect because preview is not enabled.
warning: Selection `PLW1641` has no effect because preview is not enabled.
warning: Selection `PLW3201` has no effect because preview is not enabled.
warning: Selection `PYI059` has no effect because preview is not enabled.
warning: Selection `RUF021` has no effect because preview is not enabled.
warning: Selection `RUF022` has no effect because preview is not enabled.
warning: Selection `RUF023` has no effect because preview is not enabled.
warning: Selection `RUF027` has no effect because preview is not enabled.
warning: Selection `RUF028` has no effect because preview is not enabled.
warning: Selection `RUF029` has no effect because preview is not enabled.
warning: Selection `S401` has no effect because preview is not enabled.
warning: Selection `S402` has no effect because preview is not enabled.
warning: Selection `S403` has no effect because preview is not enabled.
warning: Selection `S405` has no effect because preview is not enabled.
warning: Selection `S406` has no effect because preview is not enabled.
warning: Selection `S407` has no effect because preview is not enabled.
warning: Selection `S408` has no effect because preview is not enabled.
warning: Selection `S409` has no effect because preview is not enabled.
warning: Selection `S411` has no effect because preview is not enabled.
warning: Selection `S412` has no effect because preview is not enabled.
warning: Selection `S413` has no effect because preview is not enabled.
warning: Selection `S415` has no effect because preview is not enabled.
warning: Selection `UP042` has no effect because preview is not enabled.

DimitriPapadopoulos · 2024-08-27T08:32:48Z

I ran scripts/update_ruff.py which added them automatically. Doesn't it do the right thing?

Besides, most of them have not been added in this PR. I believe the intent has always been to run ruff check --preview.

DylanLukes · 2024-08-27T15:10:32Z

I ran scripts/update_ruff.py which added them automatically. Doesn't it do the right thing?

Besides, most of them have not been added in this PR. I believe the intent has always been to run ruff check --preview.

Ah okay! Makes sense they came from the script. The issue I was seeing is that they appear when I run hatch fmt --check which did not occur before?

I believe the intent has always been to run ruff check --preview.

If this is the case maybe something else is going on?

DylanLukes · 2024-08-27T20:03:14Z

There's actually a more subtle issue with this code as well. The _markers are a tree, not a list.

hatch/backend/src/hatchling/metadata/spec.py

Lines 161 to 196 in 8f1a1d6

    
           if (requirements := message.get_all('Requires-Dist')) is not None: 
        
               from packaging.requirements import Requirement 
        
               dependencies = [] 
        
               for requirement in requirements: 
        
                   req = Requirement(requirement) 
        
                   if req.marker is None: 
        
                       dependencies.append(str(req)) 
        
                       continue 
        
                   markers = req.marker._markers  # noqa: SLF001 
        
                   for i, marker in enumerate(markers): 
        
                       if isinstance(marker, tuple): 
        
                           left, _, right = marker 
        
                           if left.value == 'extra': 
        
                               extra = right.value 
        
                               del markers[i]  # noqa: B909 
        
                               # If there was only one marker then there will be an unnecessary 
        
                               # trailing semicolon in the string representation 
        
                               if not markers: 
        
                                   req.marker = None 
        
                               # Otherwise we need to remove the preceding `and` operation 
        
                               else: 
        
                                   del markers[i - 1] 
        
                               optional_dependencies.setdefault(extra, []).append(str(req)) 
        
                               break 
        
                   else: 
        
                       dependencies.append(str(req)) 
        
               metadata['dependencies'] = dependencies 
        
           if optional_dependencies: 
        
               metadata['optional-dependencies'] = optional_dependencies 
        
           return metadata

It cannot handle cases such as the one described in Marker's own documentation where there are more complicated nestings.

class Marker:
    def __init__(self, marker: str) -> None:
        # Note: We create a Marker object without calling this constructor in
        #       packaging.requirements.Requirement. If any additional logic is
        #       added here, make sure to mirror/adapt Requirement.
        try:
            self._markers = _normalize_extra_values(_parse_marker(marker))
            # The attribute `_markers` can be described in terms of a recursive type:
            # MarkerList = List[Union[Tuple[Node, ...], str, MarkerList]]
            #
            # For example, the following expression:
            # python_version > "3.6" or (python_version == "3.6" and os_name == "unix")
            #
            # is parsed into:
            # [
            #     (<Variable('python_version')>, <Op('>')>, <Value('3.6')>),
            #     'and',
            #     [
            #         (<Variable('python_version')>, <Op('==')>, <Value('3.6')>),
            #         'or',
            #         (<Variable('os_name')>, <Op('==')>, <Value('unix')>)
            #     ]
            # ]
        except ParserSyntaxError as e:
            raise InvalidMarker(str(e)) from e
        
        ...

We have this example in the tests:

hatch/tests/backend/metadata/test_spec.py

Lines 205 to 230 in 8f1a1d6

    
               def test_dependencies(self): 
        
                   core_metadata = f"""\ 
        
           Metadata-Version: {LATEST_METADATA_VERSION} 
        
           Name: My.App 
        
           Version: 0.1.0 
        
           Requires-Dist: bar==5 
        
           Requires-Dist: foo==1 
        
           Provides-Extra: feature1 
        
           Requires-Dist: bar==5; (python_version < '3') and extra == 'feature1' 
        
           Requires-Dist: foo==1; extra == 'feature1' 
        
           Provides-Extra: feature2 
        
           Requires-Dist: bar==5; extra == 'feature2' 
        
           Requires-Dist: foo==1; (python_version < '3') and extra == 'feature2' 
        
           Provides-Extra: feature3 
        
           Requires-Dist: baz@ file:///path/to/project ; extra == 'feature3' 
        
           """ 
        
                   assert project_metadata_from_core_metadata(core_metadata) == { 
        
                       'name': 'My.App', 
        
                       'version': '0.1.0', 
        
                       'dependencies': ['bar==5', 'foo==1'], 
        
                       'optional-dependencies': { 
        
                           'feature1': ['bar==5; python_version < "3"', 'foo==1'], 
        
                           'feature2': ['bar==5', 'foo==1; python_version < "3"'], 
        
                           'feature3': ['baz@ file:///path/to/project'], 
        
                       }, 
        
                   }

But the current code cannot actually handle something like:

Requires-Dist: bar==5; ((python_version < '3') and extra == 'feature1') or ((python_version >= '3') and extra == 'feature2')

This is probably uncommon enough that no one has really run into it? Still, it seems like in general it should be handled. Some sort of visitor is probably the appropriate approach.

Or it might make the most sense to just handle the easy case and throw an error stating that the Requirement is too complex to handle otherwise.

DylanLukes · 2024-08-27T22:32:04Z

Actually handling all of the possible cases for an arbitrary PEP 508 requirement marker is messy and a rabbit hole. Might be best to just leave it as is until it becomes an issue down the line. It seemingly hasn't yet.

DimitriPapadopoulos · 2024-09-10T10:00:20Z

I think the new CI failures are unrelated to this PR.

DylanLukes · 2024-10-02T21:37:34Z

Whatever is causing those CI failures, I hope they can get resolved soon. For the time being I'm just not using hatch fmt...

ofek · 2024-10-02T21:41:34Z

I'll look into this soon, sorry

DylanLukes · 2024-11-07T22:43:21Z

After this is merged in, I have a PR just about ready to go for Ruff 0.7.x. I'm just working through the details of the changelog and migration guide to make sure there's nothing huge...

DylanLukes · 2024-11-07T22:44:40Z

docs/history/hatch.md

 ***Added:***

 - Add ability to control the source of Python distributions
- Upgrade Ruff to 0.4.5
+- Upgrade Ruff to 0.6.8
 - Upgrade PyApp to 0.22.0 for binary builds

 ***Fixed:***


This history is no longer in the right version, should be moved to the top of the file.

I removed this so please update in your PR

docs/history/hatch.md

DylanLukes · 2024-11-07T23:02:06Z

docs/history/hatch.md

@@ -34,7 +34,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 ***Added:***

 - Add ability to control the source of Python distributions
- Upgrade Ruff to 0.6.8
+- Upgrade Ruff to 0.4.5


Er, to clarify I meant that the "Upgrade Ruff to 0.6.8" just shouldn't be under the 1.12.0 heading, but rather Unreleased.

Yes thank you for clarifying, I did understand but couldn't easily change that within the UI so I choose to remove it

Co-authored-by: Ofek Lev <[email protected]> 750b498

DimitriPapadopoulos force-pushed the ruff_0.5.0 branch 2 times, most recently from efa619a to 5f92bc7 Compare July 2, 2024 18:56

DimitriPapadopoulos force-pushed the ruff_0.5.0 branch from 5f92bc7 to c16692a Compare July 13, 2024 06:38

DimitriPapadopoulos changed the title ~~Upgrade Ruff to 0.5.0~~ Upgrade Ruff to 0.5.1 Jul 13, 2024

DimitriPapadopoulos force-pushed the ruff_0.5.0 branch from c16692a to c9b8b25 Compare July 13, 2024 06:40

DimitriPapadopoulos changed the title ~~Upgrade Ruff to 0.5.1~~ Upgrade Ruff to 0.5.0 Jul 13, 2024

DimitriPapadopoulos force-pushed the ruff_0.5.0 branch from c9b8b25 to 10a0958 Compare July 13, 2024 06:41

DimitriPapadopoulos force-pushed the ruff_0.5.0 branch 2 times, most recently from 33004bc to bc6de3f Compare August 7, 2024 07:46

DimitriPapadopoulos changed the title ~~Upgrade Ruff to 0.5.0~~ Upgrade Ruff to 0.5.6 Aug 7, 2024

DimitriPapadopoulos force-pushed the ruff_0.5.0 branch 2 times, most recently from 09e84ea to f17dac6 Compare August 7, 2024 07:51

DimitriPapadopoulos force-pushed the ruff_0.5.0 branch from f17dac6 to 3b040a8 Compare August 20, 2024 11:58

DimitriPapadopoulos changed the title ~~Upgrade Ruff to 0.5.6~~ Upgrade Ruff to 0.6.1 Aug 20, 2024

DimitriPapadopoulos force-pushed the ruff_0.5.0 branch from 3b040a8 to 8f1a1d6 Compare August 24, 2024 14:02

DylanLukes mentioned this pull request Aug 26, 2024

Hatch 1.12.0 ruff defaults invalid for Ruff 0.6.2 (match upstream flake8-trio/flake8-async) #1695

Open

DylanLukes reviewed Aug 26, 2024

View reviewed changes

DylanLukes reviewed Aug 27, 2024

View reviewed changes

DimitriPapadopoulos changed the title ~~Upgrade Ruff to 0.6.1~~ Upgrade Ruff to 0.6.2 Aug 27, 2024

DimitriPapadopoulos force-pushed the ruff_0.5.0 branch 4 times, most recently from 1e523c1 to 8f1a1d6 Compare August 27, 2024 13:59

DimitriPapadopoulos force-pushed the ruff_0.5.0 branch from 8f1a1d6 to c21114c Compare September 10, 2024 09:44

DimitriPapadopoulos force-pushed the ruff_0.5.0 branch from c21114c to 3659d38 Compare September 29, 2024 11:19

DimitriPapadopoulos force-pushed the ruff_0.5.0 branch from 3659d38 to 068a182 Compare September 29, 2024 11:21

Upgrade Ruff to 0.6.8

f792352

DimitriPapadopoulos force-pushed the ruff_0.5.0 branch from 068a182 to f792352 Compare September 29, 2024 11:22

DylanLukes referenced this pull request Nov 7, 2024

Fix failing test due to unsorted iterdir (#1761)

9650a1d

Merge branch 'master' into ruff_0.5.0

e42d96d

DylanLukes reviewed Nov 7, 2024

View reviewed changes

ofek reviewed Nov 7, 2024

View reviewed changes

docs/history/hatch.md Outdated Show resolved Hide resolved

Apply suggestions from code review

eb51a32

ofek approved these changes Nov 7, 2024

View reviewed changes

ofek changed the title ~~Upgrade Ruff to 0.6.2~~ Upgrade Ruff to 0.6.8 Nov 7, 2024

DylanLukes reviewed Nov 7, 2024

View reviewed changes

ofek merged commit 750b498 into pypa:master Nov 7, 2024
50 checks passed

github-actions bot pushed a commit that referenced this pull request Nov 7, 2024

Upgrade Ruff to 0.6.8 (#1588)

0c591e9

Co-authored-by: Ofek Lev <[email protected]> 750b498

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Upgrade Ruff to 0.6.8 #1588

Upgrade Ruff to 0.6.8 #1588

DimitriPapadopoulos commented Jun 29, 2024 •

edited

Loading

DylanLukes Aug 26, 2024

DimitriPapadopoulos Aug 27, 2024

DimitriPapadopoulos Aug 27, 2024

DylanLukes Aug 27, 2024

DylanLukes Aug 27, 2024

DylanLukes left a comment •

edited

Loading

DimitriPapadopoulos commented Aug 27, 2024 •

edited

Loading

DylanLukes commented Aug 27, 2024 •

edited

Loading

DylanLukes commented Aug 27, 2024 •

edited

Loading

DylanLukes commented Aug 27, 2024

DimitriPapadopoulos commented Sep 10, 2024

DylanLukes commented Oct 2, 2024 •

edited

Loading

ofek commented Oct 2, 2024

DylanLukes commented Nov 7, 2024 •

edited

Loading

DylanLukes Nov 7, 2024

ofek Nov 7, 2024

DylanLukes Nov 7, 2024

ofek Nov 7, 2024

Upgrade Ruff to 0.6.8 #1588

Upgrade Ruff to 0.6.8 #1588

Conversation

DimitriPapadopoulos commented Jun 29, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

DylanLukes left a comment • edited Loading

Choose a reason for hiding this comment

DimitriPapadopoulos commented Aug 27, 2024 • edited Loading

DylanLukes commented Aug 27, 2024 • edited Loading

DylanLukes commented Aug 27, 2024 • edited Loading

DylanLukes commented Aug 27, 2024

DimitriPapadopoulos commented Sep 10, 2024

DylanLukes commented Oct 2, 2024 • edited Loading

ofek commented Oct 2, 2024

DylanLukes commented Nov 7, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

DimitriPapadopoulos commented Jun 29, 2024 •

edited

Loading

DylanLukes left a comment •

edited

Loading

DimitriPapadopoulos commented Aug 27, 2024 •

edited

Loading

DylanLukes commented Aug 27, 2024 •

edited

Loading

DylanLukes commented Aug 27, 2024 •

edited

Loading

DylanLukes commented Oct 2, 2024 •

edited

Loading

DylanLukes commented Nov 7, 2024 •

edited

Loading