17 of the sample rules fail to run `devskim verify`

[JaneX8]

Some sample rules from DevSkim and ApplicationInspector repo fail the verify command.

Here is the python script I used to quickly test them all:

import os
import subprocess
import shutil

# Directory to search for JSON files
directory = r".\test"

# Locate devskim executable
devskim_path = shutil.which("devskim")

if not devskim_path:
    print("Error: 'devskim' is not found in PATH. Ensure it's installed and accessible.")
    exit(1)

def run_devskim_on_json_files(directory):
    # Traverse directory recursively
    for root, _, files in os.walk(directory):
        for file in files:
            if file.endswith(".json"):  # Only process JSON files
                file_path = os.path.join(root, file)
                # print(f"Testing file: {file_path}")
                try:
                    # Run the devskim command silently
                    result = subprocess.run(
                        [devskim_path, "verify", "-x", "Verbose", "-r", file_path],
                        stdout=subprocess.DEVNULL,  # Suppress successful output
                        stderr=subprocess.DEVNULL,  # Suppress error output
                        text=True,
                        shell=True
                    )
                    # If tests fail, show output and preserve colors
                    if result.returncode != 0:
                        print(f"\nTest failed for file: {file_path}")
                        subprocess.run(
                            [devskim_path, "verify", "-x", "Verbose", "-r", file_path],
                            shell=True  # To preserve colors
                        )
                        print("\n")  # Add a blank line after output
                except Exception as e:
                    print(f"Error running devskim on {file_path}: {e}\n")

if __name__ == "__main__":
    run_devskim_on_json_files(directory)

Here are the results:

Test failed for file: .\test\ai\generative_ai_libraries.json
[11:16:35 ERR] Rule AI070003 failed from unrecognized language VSProject specified
[11:16:35 ERR] Rules failed validation. 

Test failed for file: .\test\cloud_services\data_storage.json
[11:16:37 ERR] Rule AI002600 failed from unrecognized language pom.xml specified
[11:16:37 ERR] Rules failed validation. 

Test failed for file: .\test\cryptography\external_libraries.json
[11:16:40 ERR] Rule AI008000 failed from unrecognized language kotlin specified
[11:16:41 ERR] Rule AI008000 failed from unrecognized language scala specified
[11:16:41 ERR] Rules failed validation.
[11:16:41 ERR] Rule AI008000 failed from unrecognized language kotlin specified
[11:16:41 ERR] Rule AI008000 failed from unrecognized language scala specified

Test failed for file: .\test\data_handling\database.json
[11:16:43 ERR] Rule AI011700 failed from unrecognized language pom.xml specified
[11:16:43 ERR] Rules failed validation.

Test failed for file: .\test\device_permissions\android_intents.json
[11:16:46 ERR] Rule AI016705 failed from unrecognized language kotlin specified
[11:16:46 ERR] Rule AI016706 failed from unrecognized language kotlin specified
[11:16:46 ERR] Rule AI016707 failed from unrecognized language kotlin specified
[11:16:46 ERR] Rule AI016708 failed from unrecognized language kotlin specified
[11:16:46 ERR] Rule AI016709 failed from unrecognized language kotlin specified
[11:16:46 ERR] Rule AI016710 failed from unrecognized language kotlin specified
[11:16:46 ERR] Rule AI016711 failed from unrecognized language kotlin specified
[11:16:46 ERR] Rule AI016712 failed from unrecognized language kotlin specified
[11:16:46 ERR] Rule AI016713 failed from unrecognized language kotlin specified
[11:16:46 ERR] Rule AI016714 failed from unrecognized language kotlin specified
[11:16:46 ERR] Rule AI016715 failed from unrecognized language kotlin specified
[11:16:46 ERR] Rule AI016716 failed from unrecognized language kotlin specified
[11:16:46 ERR] Rule AI016717 failed from unrecognized language kotlin specified
[11:16:46 ERR] Rule AI016718 failed from unrecognized language kotlin specified
[11:16:46 ERR] Rule AI016719 failed from unrecognized language kotlin specified
[11:16:46 ERR] Rule AI016720 failed from unrecognized language kotlin specified
[11:16:46 ERR] Rule AI016721 failed from unrecognized language kotlin specified
[11:16:46 ERR] Rule AI016722 failed from unrecognized language kotlin specified
[11:16:46 ERR] Rule AI016723 failed from unrecognized language kotlin specified
[11:16:46 ERR] Rule AI016724 failed from unrecognized language kotlin specified
[11:16:46 ERR] Rule AI016725 failed from unrecognized language kotlin specified
[11:16:46 ERR] Rule AI016726 failed from unrecognized language kotlin specified
[11:16:46 ERR] Rule AI016727 failed from unrecognized language kotlin specified
[11:16:46 ERR] Rule AI016728 failed from unrecognized language kotlin specified
[11:16:46 ERR] Rule AI016729 failed from unrecognized language kotlin specified
[11:16:46 ERR] Rule AI016730 failed from unrecognized language kotlin specified
[11:16:46 ERR] Rules failed validation.
[11:16:46 ERR] Rule AI016705 failed from unrecognized language kotlin specified

Test failed for file: .\test\device_permissions\UWP.json
[11:16:47 ERR] Rule AI016745 failed from unrecognized language Package.appxmanifest specified
[11:16:47 ERR] Rule AI016746 failed from unrecognized language Package.appxmanifest specified
[11:16:47 ERR] Rule AI016747 failed from unrecognized language Package.appxmanifest specified
[11:16:47 ERR] Rule AI016748 failed from unrecognized language Package.appxmanifest specified
[11:16:47 ERR] Rule AI016749 failed from unrecognized language Package.appxmanifest specified
[11:16:47 ERR] Rule AI016750 failed from unrecognized language Package.appxmanifest specified
[11:16:47 ERR] Rule AI016751 failed from unrecognized language Package.appxmanifest specified
[11:16:47 ERR] Rule AI016752 failed from unrecognized language Package.appxmanifest specified
[11:16:47 ERR] Rule AI016753 failed from unrecognized language Package.appxmanifest specified
[11:16:47 ERR] Rule AI016754 failed from unrecognized language Package.appxmanifest specified
[11:16:47 ERR] Rule AI016755 failed from unrecognized language Package.appxmanifest specified
[11:16:47 ERR] Rule AI016756 failed from unrecognized language Package.appxmanifest specified
[11:16:47 ERR] Rules failed validation.
[11:16:47 ERR] Rule AI016745 failed from unrecognized language Package.appxmanifest specified

Test failed for file: .\test\frameworks\build.json
[11:16:47 ERR] Rule AI016800 failed from unrecognized language pom.xml specified
[11:16:47 ERR] Rule AI016900 failed from unrecognized language build.xml specified
[11:16:47 ERR] Rule AI017000 failed from unrecognized language build.gradle specified
[11:16:47 ERR] Rule AI017100 failed from unrecognized language jenkins specified
[11:16:47 ERR] Rule AI017200 failed from unrecognized language sbt specified
[11:16:47 ERR] Rule AI017600 failed from unrecognized language build.xml specified
[11:16:47 ERR] Rule AI017700 failed from unrecognized language project.clj specified
[11:16:47 ERR] Rule AI017800 failed from unrecognized language VSSolution specified
[11:16:47 ERR] Rule AI017800 failed from unrecognized language VSProject specified
[11:16:47 ERR] Rules failed validation.
[11:16:47 ERR] Rule AI016800 failed from unrecognized language pom.xml specified

Test failed for file: .\test\frameworks\javascript.json
[11:16:48 ERR] Rule AI021000 failed from unrecognized language html specified
[11:16:48 ERR] Rule AI021200 failed from unrecognized language html specified
[11:16:48 ERR] Rule AI021300 failed from unrecognized language html specified
[11:16:48 ERR] Rule AI021400 failed from unrecognized language html specified
[11:16:48 ERR] Rule AI021500 failed from unrecognized language html specified
[11:16:48 ERR] Rules failed validation.

Test failed for file: .\test\general\platforms.json
[11:16:51 ERR] Rule AI028500 failed from unrecognized language VSProject specified
[11:16:51 ERR] Rule AI028600 failed from unrecognized language VSProject specified
[11:16:51 ERR] Rule AI028700 failed from unrecognized language VSProject specified
[11:16:51 ERR] Rules failed validation. 
[11:16:51 ERR] Rule AI028500 failed from unrecognized language VSProject specified

Test failed for file: .\test\general\solutioninfo.json
[11:16:51 ERR] Rule AI030100 failed from unrecognized language VSsolution specified
[11:16:51 ERR] Rule AI030100 failed from unrecognized language VSProject specified
[11:16:51 ERR] Rule AI030200 failed from unrecognized language VSProject specified
[11:16:51 ERR] Rule AI030210 failed from unrecognized language VSProject specified
[11:16:51 ERR] Rule AI030220 failed from unrecognized language VSProject specified
[11:16:51 ERR] Rule AI030230 failed from unrecognized language VSProject specified
[11:16:51 ERR] Rule AI030300 failed from unrecognized language VSProject specified
[11:16:51 ERR] Rule AI030800 failed from unrecognized language pom.xml specified
[11:16:51 ERR] Rule AI030800 failed from unrecognized language build.gradle specified
[11:16:51 ERR] Rule AI030800 failed from unrecognized language build.make.xml specified
[11:16:51 ERR] Rule AI030900 failed from unrecognized language package.json specified
[11:16:51 ERR] Rule AI031000 failed from unrecognized language package.json specified
[11:16:51 ERR] Rule AI031200 failed from unrecognized language package.json specified
[11:16:51 ERR] Rule AI031400 failed from unrecognized language VSProject specified
[11:16:51 ERR] Rules failed validation.
[11:16:51 ERR] Rule AI030100 failed from unrecognized language VSsolution specified
[11:16:51 ERR] Rule AI030100 failed from unrecognized language VSProject specified

Test failed for file: .\test\infrastructure\hashicorp_terraform_tmpl.json
[11:16:52 ERR] Rule AI031566 failed from unrecognized language terraform specified
[11:16:52 ERR] Rules failed validation.
[11:16:52 ERR] Rule AI031566 failed from unrecognized language terraform specified

Test failed for file: .\test\networkcomms\outbound_network.json
[11:16:53 ERR] Rule AI032300 failed from unrecognized language html specified
[11:16:53 ERR] Rules failed validation.

Test failed for file: .\test\os\acl.json
[11:16:53 ERR] Rule AI034700 failed from unrecognized language package.json specified
[11:16:53 ERR] Rules failed validation. 

Test failed for file: .\test\test_frameworks\javascript_testing.json
[11:17:09 ERR] Rule AI041800 failed from unrecognized language package.json specified
[11:17:09 ERR] Rules failed validation.
[11:17:09 ERR] Rule AI041800 failed from unrecognized language package.json specified

Test failed for file: .\test\webapp\comms.json
[11:17:11 ERR] Rule AI054500 failed from unrecognized language html specified
[11:17:11 ERR] Rule AI054600 failed from unrecognized language html specified
[11:17:11 ERR] Rule AI054700 failed from unrecognized language html specified
[11:17:11 ERR] Rules failed validation.
[11:17:11 ERR] Rule AI054500 failed from unrecognized language html specified

Test failed for file: .\test\webapp\media.json
[11:17:11 ERR] Rule AI055900 failed from unrecognized language html specified
[11:17:11 ERR] Rule AI056000 failed from unrecognized language html specified
[11:17:11 ERR] Rules failed validation.
[11:17:11 ERR] Rule AI055900 failed from unrecognized language html specified

Test failed for file: .\test\webapp\storage.json
[11:17:12 ERR] Rule AI057000 failed from unrecognized language html specified
[11:17:12 ERR] Rule AI057100 failed from unrecognized language html specified
[11:17:12 ERR] Rule AI057200 failed from unrecognized language html specified
[11:17:12 ERR] Rule AI057300 failed from unrecognized language html specified
[11:17:12 ERR] Rule AI057400 failed from unrecognized language html specified
[11:17:12 ERR] Rule AI057500 failed from unrecognized language html specified
[11:17:12 ERR] Rule AI057600 failed from unrecognized language html specified
[11:17:12 ERR] Rules failed validation.
[11:17:12 ERR] Rule AI057000 failed from unrecognized language html specified

[gfs]

I think this would boil down to the languages.json configuration differing between DevSkim and Application inspector. These rules pass the AppInspector validator (and would pass the DevSkim validator if used with the AppInspector languages file).

For example, AppInspector defines html as a valid language here:

ApplicationInspector/AppInspector.RulesEngine/Resources/languages.json

Lines 50 to 56 in c84796a

	"name": "html",
	"extensions": [
	".html",
	".htm",
	".cshtml",
	".tmpl"
	],

But DevSkim doesn't define html as a language: https://github.com/microsoft/DevSkim/blob/9fa526953c914cfbbb883e106b52441d73f360dc/DevSkim-DotNet/Microsoft.DevSkim/resources/languages.json#L4

It probably makes sense to make the default language set the same for each.