New input type for semgrep-core allows taking scanning roots instead of target files

[mjambon]

This is used by https://github.com/semgrep/semgrep-proprietary/pull/2878.

• I ran make setup && make to update the generated code after editing a .atd file (TODO: have a CI check)
• I made sure we're still backward compatible with old versions of the CLI.
  For example, the Semgrep backend need to still be able to consume data
  generated by Semgrep 1.50.0.
  See https://atd.readthedocs.io/en/latest/atdgen-tutorial.html#smooth-protocol-upgrades
  Note that the types related to the semgrep-core JSON output or the
  semgrep-core RPC do not need to be backward compatible!

[github-actions]

Backwards compatibility summary:

Checking backward compatibility of semgrep_output_v1.atd against past version v1.100.0
Checking backward compatibility of semgrep_output_v1.atd against past version v1.101.0
Skipping v1.102.0 because commit 1c82453e89e0b569630e48ddde015e201df0e5f9 has already been checked
Checking backward compatibility of semgrep_output_v1.atd against past version v1.103.0
Checking backward compatibility of semgrep_output_v1.atd against past version v1.49.0
Skipping v1.50.0 because commit 857682f41eb09e0b330a247ff1adf3bfeaf9d9ca has already been checked
Checking backward compatibility of semgrep_output_v1.atd against past version v1.52.0
Checking backward compatibility of semgrep_output_v1.atd against past version v1.53.0
Skipping v1.54.0 because commit 3b72d494260258497e796d094b1a4916501a6df1 has already been checked
Skipping v1.54.1 because commit 3b72d494260258497e796d094b1a4916501a6df1 has already been checked
Checking backward compatibility of semgrep_output_v1.atd against past version v1.54.2
Skipping v1.54.3 because commit 9f1c50383a9a9969e2fe7a5f9bff9ca0a7c837bb has already been checked
Checking backward compatibility of semgrep_output_v1.atd against past version v1.55.0
Skipping v1.55.1 because commit 6dffeaa692153fd33b4f154fddaefde1f2f1ae27 has already been checked
Skipping v1.55.2 because commit 6dffeaa692153fd33b4f154fddaefde1f2f1ae27 has already been checked
Checking backward compatibility of semgrep_output_v1.atd against past version v1.56.0
Checking backward compatibility of semgrep_output_v1.atd against past version v1.57.0
Skipping v1.58.0 because commit 4cc11b00d411c02fc611aa8c78a336520438fb48 has already been checked
Checking backward compatibility of semgrep_output_v1.atd against past version v1.59.0
Checking backward compatibility of semgrep_output_v1.atd against past version v1.59.1
Checking backward compatibility of semgrep_output_v1.atd against past version v1.60.0
Skipping v1.60.1 because commit eed58a091fd7d19e402a6d4cf2d287e137215d03 has already been checked
Checking backward compatibility of semgrep_output_v1.atd against past version v1.61.0
Skipping v1.61.1 because commit bbfd1c5b91bd411bceffc3de73f5f0b37f04433d has already been checked
Skipping v1.62.0 because commit bbfd1c5b91bd411bceffc3de73f5f0b37f04433d has already been checked
Checking backward compatibility of semgrep_output_v1.atd against past version v1.63.0
Checking backward compatibility of semgrep_output_v1.atd against past version v1.64.0
Checking backward compatibility of semgrep_output_v1.atd against past version v1.65.0
Skipping v1.66.0 because commit 3e7bbafa2b7e722d893303a7fb90a83dab6737a7 has already been checked
Checking backward compatibility of semgrep_output_v1.atd against past version v1.66.1
Skipping v1.66.2 because commit 215a54782174de84f97188632b4a37e35ba0f827 has already been checked
Checking backward compatibility of semgrep_output_v1.atd against past version v1.67.0
Checking backward compatibility of semgrep_output_v1.atd against past version v1.68.0
Skipping v1.69.0 because commit d5b91fa4f6a03240db31e9bbbc5376a99bc8eeea has already been checked
Checking backward compatibility of semgrep_output_v1.atd against past version v1.70.0
Checking backward compatibility of semgrep_output_v1.atd against past version v1.71.0
Skipping v1.72.0 because commit 75abf193687b84ab341d8267d865ad68d81a89c9 has already been checked
Checking backward compatibility of semgrep_output_v1.atd against past version v1.73.0
Skipping v1.74.0 because commit 9f38254957c50c68ea402eebae0f7aa40dd01cbf has already been checked
Checking backward compatibility of semgrep_output_v1.atd against past version v1.75.0
Skipping v1.76.0 because commit 9102031608aa4154e1c37f557550ec4eabc8780c has already been checked
Checking backward compatibility of semgrep_output_v1.atd against past version v1.77.0
Skipping v1.78.0 because commit dcb5d77b420ddee61f58aadd3c2c7aef38778154 has already been checked
Checking backward compatibility of semgrep_output_v1.atd against past version v1.79.0
Checking backward compatibility of semgrep_output_v1.atd against past version v1.80.0
Checking backward compatibility of semgrep_output_v1.atd against past version v1.81.0
Skipping v1.82.0 because commit 9e0f3bec26b07b4fb6753a32cb75277f45f2572c has already been checked
Skipping v1.83.0 because commit 9e0f3bec26b07b4fb6753a32cb75277f45f2572c has already been checked
Checking backward compatibility of semgrep_output_v1.atd against past version v1.84.0
Skipping v1.84.1 because commit 3daef49297ada205359cc1d2996354c94b628b0d has already been checked
Checking backward compatibility of semgrep_output_v1.atd against past version v1.85.0
Checking backward compatibility of semgrep_output_v1.atd against past version v1.86.0
Checking backward compatibility of semgrep_output_v1.atd against past version v1.87.0
Skipping v1.88.0 because commit 512c0bd97db59c48a5705b2741662a338776e438 has already been checked
Skipping v1.89.0 because commit 512c0bd97db59c48a5705b2741662a338776e438 has already been checked
Checking backward compatibility of semgrep_output_v1.atd against past version v1.90.0
Checking backward compatibility of semgrep_output_v1.atd against past version v1.91.0
Skipping v1.92.0 because commit 2351c5e528cb7430422208dc66707894c066b508 has already been checked
Checking backward compatibility of semgrep_output_v1.atd against past version v1.93.0
Checking backward compatibility of semgrep_output_v1.atd against past version v1.94.0
Checking backward compatibility of semgrep_output_v1.atd against past version v1.95.0
Checking backward compatibility of semgrep_output_v1.atd against past version v1.96.0
Checking backward compatibility of semgrep_output_v1.atd against past version v1.97.0
Checking backward compatibility of semgrep_output_v1.atd against past version v1.98.0
Skipping v1.99.0 because commit 60809032a2e39742f42910d46b3e5dd305b8b8cf has already been checked

[aryx]

BTW would be great after all this work to merge the semgrep-interfaces repo in semgrep-pro :) Those double PRs
are becoming annoying (just like double PRs we used to have with change in semgrep-oss and semgrep-pro).

[mjambon]

This is annoying:

+[0db7c783] Incompatibility in both directions:
+Incompatible kinds of types: list/array is now a sum type or enum.
+The following types are affected:
+  targets

It's a legitimate incompatibility report but it's about the semgrep-core interface that we can break at will. It wouldn't happen if we could split the ATD file into multiple modules each concerned with a different interface. Maybe for now we could have some kind of hack to work around this. Atddiff offers an option to only check certain types so we could use this but it's not great for the long term (since adding a new interface would require adding the new root types to the list of types to check). By increasing difficulty (and cleanliness), we have:

1. use some shell script hack to silence reports about semgrep-core interface changes
2. use atddiff with the --types option to only check the root types involved in semgrep's external interfaces (= not semgrep-core)
3. modify atddiff to ensure any root type is listed on the command line by either --types or --ignore-types so as to ensure we don't forget to check new types we should check
4. implement and ship atdml with module support

[aryx]

Yes I've asked a few times for the ability to white list a few roots to not bother to check anything that descend from RPC or semgrep-core stuff.

[aryx]

whitelist or blacklist. In this case we probably want atddiff --do-not-bother-about-those-roots 'function_call,function_return,core_output'

Note that core_output also deeply use cli_output_extra, but this cli_output_extra is also used by cli_output so we should
run the compatibility check on cli_output_extra (but not on types that are referenced only by core_output).

[mjambon]

whitelist or blacklist. In this case we probably want atddiff --do-not-bother-about-those-roots 'function_call,function_return,core_output'

Note that core_output also deeply use cli_output_extra, but this cli_output_extra is also used by cli_output so we should
run the compatibility check on cli_output_extra (but not on types that are referenced only by core_output).

I added an option to atddiff to ensure that we don't miss future type definitions. atd PR coming soon.

[mjambon]

@aryx you wrote at ahrefs/atd#422 (comment):

the list for --types and --ignore seems wrong.
I don't see cli_output for example, which should be in --types I guess, and
function_call and function_return should be in --ignore as they don't have to be backward compatible.

cli_output is used by function_call and the latter wasn't in --types. I'll make this change.