Provide a better testing methodology and documentation for User Plugin

[sluongng]

Currently developing User Plugin has many issues:

• Testing documentation is not available. You can follow some of the provided example in this repo to write a test but you cannot execute the test by yourself (at least there is no documentation on how to run it)

• Testing are mostly using the artifactory API, which assume that you have a working Pro Licensed instance up and running for testing, which cost money. Testing should not cost money and you should not have to test your User Plugin on a production instance.

Please provide a proper way to test these plugins for easier customization and development.

[DarthFennec]

I agree with you.

You can follow some of the provided example in this repo to write a test but you cannot execute the test by yourself

Our old testing and development platform, artifactory-user-plugins-devenv, used to be able to run most tests (sometimes with some manual steps required, though). However, we haven't updated it in a very long time, and it may or may not still work as well. Meanwhile, our current testing platform, Lilypad, is currently not available to the public. I think making Lilypad publicly available is the solution to this problem, and I hope that this can happen sooner rather than later.

Testing are mostly using the artifactory API, which assume that you have a working Pro Licensed instance up and running for testing, which cost money.

While I do agree, I seriously doubt this is going to change. All plugin tests need to run against an Artifactory instance that can run plugins, which already limits you to licensed instances anyway. So a Pro license would still be required, even if the REST API wasn't used for testing, because the instance still needs to be able to run the plugin being tested. The assumption here is that you won't be testing plugins unless you plan to use them, which means you're already a paying customer, or you at least have a free trial license.

[sluongng]

Instead of testing these test scripts via actual REST api, why dont you just provide us with an executable script that can send a mock payload and trigger the plugin?

Then we can write and run the test without artifactory at all.

If i recall correctly, the testing instance require a license which is an additional license we have to buy. So unless we taking down a production instance for testing, its a bit difficult to develop and debug these.

Also the fact that primary instance may replicate these plugins without a cleanup mechanism is just another point adding to why i dont wana test it on prod

[DarthFennec]

Instead of testing these test scripts via actual REST api, why dont you just provide us with an executable script that can send a mock payload and trigger the plugin?

Then we can write and run the test without artifactory at all.

I'm not so sure what you mean. The test scripts do send a mock payload and trigger the plugin, that's how they work. The thing is, for the plugin to get triggered it has to be running on an Artifactory instance that supports plugins. Unless you're suggesting that the tests mock out Artifactory entirely? That would be nice to have, but I don't think it would be practical. Many plugins need to use a lot of the Artifactory public and internal APIs to function at all, and mocking all that for each and every plugin would be an unreasonable amount of extra effort. More importantly, nine times out of ten when I see a test fail and the plugin needs fixing, it's because Artifactory has been changed in some way that breaks the plugin. If Artifactory is entirely mocked out, this no longer happens, making testing mostly useless.

If i recall correctly, the testing instance require a license which is an additional license we have to buy.

I definitely agree that this is not optimal. I personally would very much like to allow user plugins and REST APIs in the OSS version, but it simply isn't in the cards. Perhaps you can contact JFrog and explain your situation, you might be able to net yourself a free trial license you can use in the short run, who knows. For a long-term solution though, I expect they'll try to sell you the additional license. As far as I'm aware, it's recommended (and common practice) to buy an extra license or two specifically for development and testing purposes (at least for Enterprise customers, I don't know if the recommendation is the same for Pro customers).

Also the fact that primary instance may replicate these plugins without a cleanup mechanism is just another point adding to why i dont wana test it on prod

Yes, this is a long-standing technical issue that has caused me plenty of headaches, and I'd very much like to see it resolved. In the mean time, I don't think there's much you can do here. If this does happen, deleting the file manually from all nodes and then restarting all nodes should work.

[meiswjn]

Hey,
Are there any updates on this? Would be nice to run tests without requiring a license.

[sluongng]

I have switched jobs recently an is no longer administering an Artifactory instance. But here would be a quick description regarding how I managed to setup some tests for our custom groovy plugins in previous job:

Each "plugin implementation" is written into a function which is then used by 2 actual plugins.

The first plugin would always be REST-based and can only be triggered from localhost. We used this plugin to call into the implementation to ensure that the logic inside the plugin is correct. We setup logger that print out message from the plugin for easier dev/debugging. The REST-plugin often call into the implementation function with mock data, and the "implementation function" would detect these mock data to avoid triggering actual writes.

The second plugin would be the actual plugin used, which only implement the basic minimum interface of that plugin type before passing all the needed data into the "implementation function".

Testing these plugins were still a pain in the ass, as it involved us SSH into production env and test against the instances of artifactory we have licenses for. Avoiding/working around the plugin automatic propagations by artifactory were also another PITA. But at least, these tests gave us some confident in shipping out a new version of custom plugin into a black box application such as Artifactory.