Provide a way to run a subset of tests

[KurtPfeifle]

After meanwhile quite a few ipp-1.1.test runs and being entangled in related debugging activities, I think the following would be beneficial for future testers:

• Split all the individual tests making up the IPP-1.1 test suite into separate files.
• (Re-)Create a new ipp-1.1.test suite by using INCLUDE statements for the individual files.

This way it would be more easy to run specific tests individually, especially when it comes to debugging the respective parts of the now biiiig ipp-1.1.test file.

It would also allow to make a meaningful use of ipptool's capability to test with multiple test files within one run as its usage hint suggests:

 Usage: ipptool [options] URI filename [ ... filenameN ] 

[wifiprintguy]

I think this is a good idea and I would also recommend a similar enhancement to ippeveselfcert's ipp-tests.test and document-tests.test

[michaelrsweet]

The problem with breaking them all up is that many tests depend on the results of prior tests - not something we can easily deal with using includes... :/

[KurtPfeifle]

A few weeks ago, for my private usage and experimentation, I've broken up the big ipp-1.1.test into small units already.

They all already work by themselves, individually, without falling over syntax errors.

I've just not yet put all the complete dependencies in place yet, which Mike mentioned. Maybe I get round to it at the weekend. The thing is I first have to grok them all myself first. Which is not so easy if you do not deal with IPP Everywhere every day, everywhere... :-)

Here is a list of the names I chose for them with their respective file sizes:

kp@stretch:~>  ls -l *.test
-rw-r--r-- 1 kp kp     793 Nov  9 14:33 00-define-uri-regex.test
-rw-r--r-- 1 kp kp     418 Nov  9 15:03 01-rfc-2911-section-3.1.1-bad-request-id-value-0.test
-rw-r--r-- 1 kp kp     350 Nov  9 15:03 02-rfc-2911-section-3.1.4-no-operation-attributes.test
-rw-r--r-- 1 kp kp     252 Nov  9 15:04 03-rfc-2911-section-3.1.4-attributes-charset.test
-rw-r--r-- 1 kp kp     275 Nov  9 15:05 04-rfc-2911-section-3.1.4-attributes-natural-language.test
-rw-r--r-- 1 kp kp     335 Nov  9 15:05 05-rfc-2911-section-3.1.4-attributes-natural-language-+-attributes-charset.test
-rw-r--r-- 1 kp kp     404 Nov  9 16:08 06-rfc-2911-section-3.1.4-attributes-charset-+-attributes-natural-language.test
-rw-r--r-- 1 kp kp     454 Nov  9 15:06 07-rfc-2911-section-3.1.8-unsupported-ipp-version-0.0.test
-rw-r--r-- 1 kp kp     408 Nov  9 15:07 08-rfc-2911-section-3.2-no-printer-uri-operation-attribute.test
-rw-r--r-- 1 kp kp    1333 Nov  9 16:08 09-rfc-2911-section-3.2.1-print-job-operation.test
-rw-r--r-- 1 kp kp     552 Nov  9 15:08 10-rfc-2911-section-3.2.3-validate-job-operation.test
-rw-r--r-- 1 kp kp   10584 Nov  9 16:09 11-rfc-2911-section-3.2.5-get-printer-attributes-operation-default.test
-rw-r--r-- 1 kp kp     714 Nov  9 16:09 12-rfc-2911-section-3.2.5-get-printer-attributes-operation-requested-attributes.test
-rw-r--r-- 1 kp kp    1593 Nov  9 16:09 13-rfc-2911-section-3.2.6-get-jobs-operation-default.test
-rw-r--r-- 1 kp kp    4073 Nov  9 16:09 14-rfc-2911-section-3.2.6-get-jobs-operation-requested-attributes.test
-rw-r--r-- 1 kp kp    1658 Nov  9 16:09 15-rfc-2911-section-3.2.6-get-jobs-operation-my-jobs.test
-rw-r--r-- 1 kp kp    1668 Nov  9 15:16 16-rfc-2911-section-3.2.6-get-jobs-operation-my-jobs-different-user.test
-rw-r--r-- 1 kp kp    1687 Nov  9 16:09 17-rfc-2911-section-3.2.6-get-jobs-operation-which-jobs-not-completed.test
-rw-r--r-- 1 kp kp     522 Nov  9 15:17 18-get-job-attributes-until-job-complete.test
-rw-r--r-- 1 kp kp    1639 Nov  9 16:09 19-rfc-2911-section-3.2.6-get-jobs-operation-which-jobs-completed.test
-rw-r--r-- 1 kp kp    4109 Nov  9 16:09 20-rfc-2911-section-3.2.6-get-jobs-operation-which-jobs-requested-attributes.test
-rw-r--r-- 1 kp kp     418 Nov  9 15:19 21-rfc-2911-section-3.3.3-cancel-job-operation-completed-job.test
-rw-r--r-- 1 kp kp    1265 Nov  9 16:09 22-rfc-2911-section-3.2.1-print-job-operation.test
-rw-r--r-- 1 kp kp     449 Nov  9 15:20 23-rfc-2911-section-3.3.3-cancel-job-operation-pending-processing-job.test
-rw-r--r-- 1 kp kp    4021 Nov  9 16:09 24-rfc-2911-section-3.3.4-get-job-attributes-operation.test
-rw-r--r-- 1 kp kp    1263 Nov  9 16:09 25-rfc-2911-section-3.2.2-print-uri-operation.test
-rw-r--r-- 1 kp kp     763 Nov  9 15:24 26-print-uri-with-bad-uri-print-uri-operation.test
-rw-r--r-- 1 kp kp    1114 Nov  9 16:09 27-rfc-2911-section-3.2.4-create-job-operation.test
-rw-r--r-- 1 kp kp     685 Nov  9 15:25 28-rfc-2911-section-3.3.1-send-document-operation.test
-rw-r--r-- 1 kp kp    1146 Nov  9 16:10 29-send-document-missing-last-document-create-job-operation.test
-rw-r--r-- 1 kp kp     591 Nov  9 15:28 30-send-document-missing-last-document-send-document-operation.test
-rw-r--r-- 1 kp kp     467 Nov  9 16:43 31-rfc-2911-section-3.3.3-cancel-job-operation.test
-rw-r--r-- 1 kp kp    1104 Nov  9 16:44 32-rfc-2911-section-3.2.4-create-job-operation.test
-rw-r--r-- 1 kp kp     628 Nov  9 16:45 33-rfc-2911-section-3.3.2-send-uri-operation.test
-rw-r--r-- 1 kp kp    1079 Nov  9 16:45 34-send-uri-with-bad-uri-create-job-operation.test
-rw-r--r-- 1 kp kp     675 Nov  9 15:44 35-send-uri-with-bad-uri-send-uri-operation-bad-uri.test
-rw-r--r-- 1 kp kp     461 Nov  9 15:44 36-send-uri-with-bad-uri-cancel-job-operation.test
-rw-r--r-- 1 kp kp    1291 Nov  9 16:10 37-print-job-with-copies.test
-rw-r--r-- 1 kp kp    1334 Nov  9 16:10 38-print-job-with-a4-pdf.test
-rw-r--r-- 1 kp kp    1375 Nov  9 16:10 39-print-job-with-a4-pdf-duplex.test
-rw-r--r-- 1 kp kp    1346 Nov  9 16:10 40-print-job-with-us-letter-pdf.test
-rw-r--r-- 1 kp kp    1439 Nov  9 16:10 41-print-job-with-us-letter-pdf-duplex.test
-rw-r--r-- 1 kp kp    1372 Nov  9 16:10 42-print-job-with-a4-postscript.test
-rw-r--r-- 1 kp kp    1427 Nov  9 16:10 43-print-job-with-a4-postscript-duplex.test
-rw-r--r-- 1 kp kp    1373 Nov  9 16:10 44-print-job-with-us-letter-postscript.test
-rw-r--r-- 1 kp kp    1443 Nov  9 16:10 45-print-job-with-us-letter-postscript-duplex.test
-rw-r--r-- 1 kp kp    1329 Nov  9 16:10 46-print-job-with-color-jpeg-on-a4.test
-rw-r--r-- 1 kp kp    1332 Nov  9 16:11 47-print-job-with-color-jpeg-on-us-letter.test
-rw-r--r-- 1 kp kp    1314 Nov  9 16:11 48-print-job-with-color-jpeg-on-4x6.test
-rw-r--r-- 1 kp kp    1316 Nov  9 16:11 49-print-job-with-grayscale-jpeg-on-a4.test
-rw-r--r-- 1 kp kp    1338 Nov  9 16:11 50-print-job-with-grayscale-jpeg-on-us-letter.test
-rw-r--r-- 1 kp kp    1320 Nov  9 16:11 51-print-job-with-grayscale-jpeg-on-4x6.test
-rw-r--r-- 1 kp kp    1459 Nov  9 16:11 52-print-job-with-a4-pdf-and-standard-sheet.test
-rw-r--r-- 1 kp kp    1453 Nov  9 16:11 53-print-job-with-us-letter-pdf-and-standard-sheet.test
-rw-r--r-- 1 kp kp    1443 Nov  9 16:11 54-print-job-with-a4-pdf-and-standard-sheet.test
-rw-r--r-- 1 kp kp    1465 Nov  9 16:11 55-print-job-with-us-letter-pdf-and-standard-sheet.test
-rw-r--r-- 1 kp kp    1412 Nov  9 16:11 56-print-job-with-a4-pdf-2-up.test
-rw-r--r-- 1 kp kp    1418 Nov  9 16:11 57-print-job-with-us-letter-pdf-2-up.test
-rw-r--r-- 1 kp kp    1400 Nov  9 16:11 58-print-job-with-a4-pdf-2-up.test
-rw-r--r-- 1 kp kp    1430 Nov  9 16:11 59-print-job-with-us-letter-pdf-2-up.test
-rw-r--r-- 1 kp kp    1442 Nov  9 16:11 60-print-job-with-jpeg-on-4x6-draft-quality.test
-rw-r--r-- 1 kp kp    1409 Nov  9 16:11 61-print-job-with-jpeg-on-4x6-normal-quality.test
-rw-r--r-- 1 kp kp    1403 Nov  9 16:11 62-print-job-with-jpeg-on-4x6-high-quality.test
-rw-r--r-- 1 kp kp    1408 Nov  9 16:11 63-print-job-with-a4-pdf-draft-quality.test
-rw-r--r-- 1 kp kp    1438 Nov  9 16:11 64-print-job-with-us-letter-pdf-draft-quality.test
-rw-r--r-- 1 kp kp    1288 Nov  9 16:11 65-print-job-with-job-hold-until.test
-rw-r--r-- 1 kp kp     364 Nov  9 15:30 66-release-job.test
-rw-r--r-- 1 kp kp    4356 Nov  9 16:12 get-jobs-operation-requested-attr.test
-rw-r--r-- 1 kp kp    7720 Nov  8 22:50 ipp-2.0.test
-rw-r--r-- 1 kp kp    4792 Nov  7 12:30 ipp-2.1.test
-rw-r--r-- 1 kp kp    4111 Nov  7 12:30 ipp-2.2.test
-rw-r--r-- 1 kp kp   17972 Nov  7 16:35 ipp-3d.test
-rw-r--r-- 1 kp kp  318194 Nov  7 16:35 ipp-everywhere.test
-rw-r--r-- 1 kp kp    4052 Nov  9 16:53 new-ipp-1.1.test

And this is the content of my new-ipp-1.1.test

kp@stretch:>  cat new-ipp-1.1.test
INCLUDE 00-define-uri-regex.test
INCLUDE 01-rfc-2911-section-3.1.1-bad-request-id-value-0.test
INCLUDE 02-rfc-2911-section-3.1.4-no-operation-attributes.test
INCLUDE 03-rfc-2911-section-3.1.4-attributes-charset.test
INCLUDE 04-rfc-2911-section-3.1.4-attributes-natural-language.test
INCLUDE 05-rfc-2911-section-3.1.4-attributes-natural-language-+-attributes-charset.test
INCLUDE 06-rfc-2911-section-3.1.4-attributes-charset-+-attributes-natural-language.test
INCLUDE 07-rfc-2911-section-3.1.8-unsupported-ipp-version-0.0.test
INCLUDE 08-rfc-2911-section-3.2-no-printer-uri-operation-attribute.test
INCLUDE 09-rfc-2911-section-3.2.1-print-job-operation.test
INCLUDE 10-rfc-2911-section-3.2.3-validate-job-operation.test
INCLUDE 11-rfc-2911-section-3.2.5-get-printer-attributes-operation-default.test
INCLUDE 12-rfc-2911-section-3.2.5-get-printer-attributes-operation-requested-attributes.test
INCLUDE 13-rfc-2911-section-3.2.6-get-jobs-operation-default.test
INCLUDE 14-rfc-2911-section-3.2.6-get-jobs-operation-requested-attributes.test
INCLUDE 15-rfc-2911-section-3.2.6-get-jobs-operation-my-jobs.test
INCLUDE 16-rfc-2911-section-3.2.6-get-jobs-operation-my-jobs-different-user.test
INCLUDE 17-rfc-2911-section-3.2.6-get-jobs-operation-which-jobs-not-completed.test
INCLUDE 18-get-job-attributes-until-job-complete.test
INCLUDE 19-rfc-2911-section-3.2.6-get-jobs-operation-which-jobs-completed.test
INCLUDE 20-rfc-2911-section-3.2.6-get-jobs-operation-which-jobs-requested-attributes.test
INCLUDE 21-rfc-2911-section-3.3.3-cancel-job-operation-completed-job.test
INCLUDE 22-rfc-2911-section-3.2.1-print-job-operation.test
INCLUDE 23-rfc-2911-section-3.3.3-cancel-job-operation-pending-processing-job.test
INCLUDE 24-rfc-2911-section-3.3.4-get-job-attributes-operation.test
INCLUDE 25-rfc-2911-section-3.2.2-print-uri-operation.test
INCLUDE 26-print-uri-with-bad-uri-print-uri-operation.test
INCLUDE 27-rfc-2911-section-3.2.4-create-job-operation.test
INCLUDE 28-rfc-2911-section-3.3.1-send-document-operation.test
INCLUDE 29-send-document-missing-last-document-create-job-operation.test
INCLUDE 30-send-document-missing-last-document-send-document-operation.test
INCLUDE 31-rfc-2911-section-3.3.3-cancel-job-operation.test
INCLUDE 32-rfc-2911-section-3.2.4-create-job-operation.test
INCLUDE 33-rfc-2911-section-3.3.2-send-uri-operation.test
INCLUDE 34-send-uri-with-bad-uri-create-job-operation.test
INCLUDE 35-send-uri-with-bad-uri-send-uri-operation-bad-uri.test
INCLUDE 36-send-uri-with-bad-uri-cancel-job-operation.test
INCLUDE 37-print-job-with-copies.test
INCLUDE 38-print-job-with-a4-pdf.test
INCLUDE 39-print-job-with-a4-pdf-duplex.test
INCLUDE 40-print-job-with-us-letter-pdf.test
INCLUDE 41-print-job-with-us-letter-pdf-duplex.test
INCLUDE 42-print-job-with-a4-postscript.test
INCLUDE 43-print-job-with-a4-postscript-duplex.test
INCLUDE 44-print-job-with-us-letter-postscript.test
INCLUDE 45-print-job-with-us-letter-postscript-duplex.test
INCLUDE 46-print-job-with-color-jpeg-on-a4.test
INCLUDE 47-print-job-with-color-jpeg-on-us-letter.test
INCLUDE 48-print-job-with-color-jpeg-on-4x6.test
INCLUDE 49-print-job-with-grayscale-jpeg-on-a4.test
INCLUDE 50-print-job-with-grayscale-jpeg-on-us-letter.test
INCLUDE 51-print-job-with-grayscale-jpeg-on-4x6.test
INCLUDE 52-print-job-with-a4-pdf-and-standard-sheet.test
INCLUDE 53-print-job-with-us-letter-pdf-and-standard-sheet.test
INCLUDE 54-print-job-with-a4-pdf-and-standard-sheet.test
INCLUDE 55-print-job-with-us-letter-pdf-and-standard-sheet.test
INCLUDE 56-print-job-with-a4-pdf-2-up.test
INCLUDE 57-print-job-with-us-letter-pdf-2-up.test
INCLUDE 58-print-job-with-a4-pdf-2-up.test
INCLUDE 59-print-job-with-us-letter-pdf-2-up.test
INCLUDE 60-print-job-with-jpeg-on-4x6-draft-quality.test
INCLUDE 61-print-job-with-jpeg-on-4x6-normal-quality.test
INCLUDE 62-print-job-with-jpeg-on-4x6-high-quality.test
INCLUDE 63-print-job-with-a4-pdf-draft-quality.test
INCLUDE 64-print-job-with-us-letter-pdf-draft-quality.test
INCLUDE 65-print-job-with-job-hold-until.test
INCLUDE 66-release-job.test

As far as I remember currently (that was 5 weeks ago -- I have to confirm again), using this test file yielded the same results for me as did using the original ipp-1.1.test.

Provided I can confirm that my new-ipp-1.1.test really is equivalent to the original ipp-1.1.test, this collection of files could already serve as a drop-in replacement for the current test setup.

What is left then? AFAIU, it is to get as many of the simple XX-*-*.test tests work individually too, by including the respective prior tests into each one. That shouldn't be too difficult, once I grok all the inter-dependencies.

[KurtPfeifle]

To better be able to show what I did, I forked your repo.

• https://github.com/KurtPfeifle/ippsample

This commit created a new 'examples2' directory with my new 'atomic' *.test files (keeping the original 'examples' sub-directory in place + un-changed):

• KurtPfeifle@ea0c581

[michaelrsweet]

Ugh, that's a maintenance nightmare.

The purpose of these files was to have a simple way to sanity-check an implementation, not to provide packaged, isolated testing scripts. And given the dependencies, it is not always feasible to use the separate tests.

The examples directory already contains a number of isolated/example tests for a single functionality - I would rather expand/enhance those tests than split up all of the individual simple test suites. If there is a particular set of tests that might need to be repeated during development of a product, then IMHO that should be the focus of any work and not on a wholesale separation of every possible test that makes it impossible to maintain.

[wifiprintguy]

My own interest in this derives from a desire to run one or more of the tests in isolation without running all the tests in the .test file. One way I thought of doing this, without having to modify the ipptool code, was to put the separate tests into discrete .test files, as @KurtPfeifle had proposed here. It seems another way to do this might be to use a bunch of DEFINE statements at the top, such as:

DEFINE TEST-1 true
DEFINE TEST-2 true
...

and then turn them off by passing -d arguments to ipptool. (Just a hypothesis - I haven't tried this out yet to see how awkward that might be.)

[michaelrsweet]

That also seems pretty cumbersome - maybe an option to specify a range of tests to run?

[wifiprintguy]

If ipptool were extended to allow one or more tests to be specified by name as an option, and ipptool would "do the dependency math" to ensure that the dependencies were handled, that would be the best of both worlds. But tracking and resolving the dependencies smells like "work". But if you are interested in authoring that I'm 100% supportive!

As an extension of that, if it could list the names of the tests before running them that would also be cool!

[KurtPfeifle]

You can play with my 'atomic' .test files when you use a newly created .AppImage made with the @AppImage toolbox created by @probonopd & friends. Everything needed is embedded in one single file, and can be put to use easily.

Get it from here:

• https://github.com/KurtPfeifle/ippsample/releases
• https://github.com/KurtPfeifle/ippsample/releases/download/continuous/ippsample-xperimental-x86_64.AppImage

Tutorial as an ASCIIcast

Also, I've created a little tutorial about the IPP Sample Software using this AppImage and recorded an ASCIIcast with it, using @sickill's @asciinema:

(Click image above to watch it.)

List all embedded '.test' files

Run the AppImage to enumerate all embedded .test files:

$>  ./ippsample-xperimental-x86_64.AppImage --listtest

You should see the following:

00-define-uri-regex.test                                                              52-print-job-with-a4-pdf-and-standard-sheet.test
01-rfc-2911-section-3.1.1-bad-request-id-value-0.test                                 53-print-job-with-us-letter-pdf-and-standard-sheet.test
02-rfc-2911-section-3.1.4-no-operation-attributes.test                                54-print-job-with-a4-pdf-and-standard-sheet.test
03-rfc-2911-section-3.1.4-attributes-charset.test                                     55-print-job-with-us-letter-pdf-and-standard-sheet.test
04-rfc-2911-section-3.1.4-attributes-natural-language.test                            56-print-job-with-a4-pdf-2-up.test
05-rfc-2911-section-3.1.4-attributes-natural-language-+-attributes-charset.test       57-print-job-with-us-letter-pdf-2-up.test
06-rfc-2911-section-3.1.4-attributes-charset-+-attributes-natural-language.test       58-print-job-with-a4-pdf-2-up.test
07-rfc-2911-section-3.1.8-unsupported-ipp-version-0.0.test                            59-print-job-with-us-letter-pdf-2-up.test
08-rfc-2911-section-3.2-no-printer-uri-operation-attribute.test                       60-print-job-with-jpeg-on-4x6-draft-quality.test
09-rfc-2911-section-3.2.1-print-job-operation.test                                    61-print-job-with-jpeg-on-4x6-normal-quality.test
10-rfc-2911-section-3.2.3-validate-job-operation.test                                 62-print-job-with-jpeg-on-4x6-high-quality.test
11-rfc-2911-section-3.2.5-get-printer-attributes-operation-default.test               63-print-job-with-a4-pdf-draft-quality.test
12-rfc-2911-section-3.2.5-get-printer-attributes-operation-requested-attributes.test  64-print-job-with-us-letter-pdf-draft-quality.test
13-rfc-2911-section-3.2.6-get-jobs-operation-default.test                             65-print-job-with-job-hold-until.test
14-rfc-2911-section-3.2.6-get-jobs-operation-requested-attributes.test                66-release-job.test
15-rfc-2911-section-3.2.6-get-jobs-operation-my-jobs.test                             cancel-current-job.test
16-rfc-2911-section-3.2.6-get-jobs-operation-my-jobs-different-user.test              create-job-3d.test
17-rfc-2911-section-3.2.6-get-jobs-operation-which-jobs-not-completed.test            create-job-format.test
18-get-job-attributes-until-job-complete.test                                         create-job-sheets.test
19-rfc-2911-section-3.2.6-get-jobs-operation-which-jobs-completed.test                create-job.test
20-rfc-2911-section-3.2.6-get-jobs-operation-which-jobs-requested-attributes.test     create-job-timeout.test
21-rfc-2911-section-3.3.3-cancel-job-operation-completed-job.test                     create-printer-subscription.test
22-rfc-2911-section-3.2.1-print-job-operation.test                                    get-completed-jobs.test
23-rfc-2911-section-3.3.3-cancel-job-operation-pending-processing-job.test            get-job-attributes.test
24-rfc-2911-section-3.3.4-get-job-attributes-operation.test                           get-jobs.test
25-rfc-2911-section-3.2.2-print-uri-operation.test                                    get-job-template-attributes.test
26-print-uri-with-bad-uri-print-uri-operation.test                                    get-notifications.test
27-rfc-2911-section-3.2.4-create-job-operation.test                                   get-printer-attributes-2.0.test
28-rfc-2911-section-3.3.1-send-document-operation.test                                get-printer-attributes.test
29-send-document-missing-last-document-create-job-operation.test                      get-printer-description-attributes.test
30-send-document-missing-last-document-send-document-operation.test                   get-subscriptions.test
31-rfc-2911-section-3.3.3-cancel-job-operation.test                                   identify-printer-display.test
32-rfc-2911-section-3.2.4-create-job-operation.test                                   identify-printer-multiple.test
33-rfc-2911-section-3.3.2-send-uri-operation.test                                     identify-printer.test
34-send-uri-with-bad-uri-create-job-operation.test                                    ipp-1.1-new.test
35-send-uri-with-bad-uri-send-uri-operation-bad-uri.test                              ipp-1.1.test
36-send-uri-with-bad-uri-cancel-job-operation.test                                    ipp-2.0.test
37-print-job-with-copies.test                                                         ipp-2.1.test
38-print-job-with-a4-pdf.test                                                         ipp-2.2.test
39-print-job-with-a4-pdf-duplex.test                                                  ipp-3d.test
40-print-job-with-us-letter-pdf.test                                                  ipp-everywhere.test
41-print-job-with-us-letter-pdf-duplex.test                                           print-job-and-wait.test
42-print-job-with-a4-postscript.test                                                  print-job-deflate.test
43-print-job-with-a4-postscript-duplex.test                                           print-job-gzip.test
44-print-job-with-us-letter-postscript.test                                           print-job-hold.test
45-print-job-with-us-letter-postscript-duplex.test                                    print-job-media-col.test
46-print-job-with-color-jpeg-on-a4.test                                               print-job-password.test
47-print-job-with-color-jpeg-on-us-letter.test                                        print-job.test
48-print-job-with-color-jpeg-on-4x6.test                                              print-uri.test
49-print-job-with-grayscale-jpeg-on-a4.test                                           set-attrs-hold.test
50-print-job-with-grayscale-jpeg-on-us-letter.test                                    validate-job.test
51-print-job-with-grayscale-jpeg-on-4x6.test

Run a combo of embedded 'atomic' .test files

Those .test files named with leading numbers are my 'atomic' ones. To test a combination of them, you could run:

$>  HOST_PORT=my.ippserver.local.:22222
$>
$>  ./ippsample-xperimental-x86_64.AppImage  ipptool                \
            -tv                                                     \
             ipp://${HOST_PORT}/ipp/print                           \
             02-rfc-2911-section-3.1.4-no-operation-attributes.test \
             03-rfc-2911-section-3.1.4-attributes-charset.test      \
             04-rfc-2911-section-3.1.4-attributes-natural-language.test 

"/tmp/.mount_ippsamIEDv0p/usr/share/cups/ipptool/02-rfc-2911-section-3.1.4-no-operation-attributes.test":
    Get-Printer-Attributes:
    RFC 2911 section 3.1.4: No Operation Attributes                      [PASS]
        RECEIVED: 119 bytes in response
        status-code = client-error-bad-request (No attributes in request.)
        attributes-charset (charset) = utf-8
        attributes-natural-language (naturalLanguage) = en-us
        status-message (textWithoutLanguage) = No attributes in request.
"/tmp/.mount_ippsamIEDv0p/usr/share/cups/ipptool/03-rfc-2911-section-3.1.4-attributes-charset.test":
    Get-Printer-Attributes:
        attributes-charset (charset) = utf-8
        printer-uri (uri) = ipp://lenjessie2.fritz.box:22222/ipp/print
    RFC 2911 section 3.1.4: attributes-charset                           [PASS]
        RECEIVED: 122 bytes in response
        status-code = client-error-bad-request (Missing required attributes.)
        attributes-charset (charset) = utf-8
        attributes-natural-language (naturalLanguage) = en-us
        status-message (textWithoutLanguage) = Missing required attributes.
"/tmp/.mount_ippsamIEDv0p/usr/share/cups/ipptool/04-rfc-2911-section-3.1.4-attributes-natural-language.test":
    Get-Printer-Attributes:
        attributes-natural-language (naturalLanguage) = en
        printer-uri (uri) = ipp://lenjessie2.fritz.box:22222/ipp/print
    RFC 2911 section 3.1.4: attributes-natural-language                  [PASS]
        RECEIVED: 122 bytes in response
        status-code = client-error-bad-request (Missing required attributes.)
        attributes-charset (charset) = utf-8
        attributes-natural-language (naturalLanguage) = en-us
        status-message (textWithoutLanguage) = Missing required attributes.

Summary: 3 tests, 3 passed, 0 failed, 0 skipped
Score: 100%

Extract embedded .test files and hack them however you like

Here's how:

$>  ./ippsample-xperimental-x86_64.AppImage --cat                    \
          02-rfc-2911-section-3.1.4-no-operation-attributes.test     \
          > local-02.test
$>
$>  ./ippsample-xperimental-x86_64.AppImage --cat                    \
          03-rfc-2911-section-3.1.4-attributes-charset.test          \
          > local-03.test
$>
$>  ./ippsample-xperimental-x86_64.AppImage --cat                    \
          04-rfc-2911-section-3.1.4-attributes-natural-language.test \
          > local-04.test

Now edit these local-0{2,3,4}.test files to your heart's contents (or leave them alone, if you just want to verify if the AppImage'd tools can process "external" test files). Just give a relative or absolute PATH to the files:

$>  ./ippsample-xperimental-x86_64.AppImage  ipptool \
            -tv                                      \
             ipp://${HOST_PORT}/ipp/print            \
             ./local-02.test                         \
             ./local-03.test                         \
             ./local-04.test

[KurtPfeifle]

I got asked in private mail by a user, why above output shows lines like:

"/tmp/.mount_ippsamIEDv0p/usr/share/cups/ipptool/02-rfc-2911-section-3.1.4-no-operation-attributes.test":

This is an AppImage-y thing. AppImages, as was explained earlier, are designed according to the "One App = One File" paradigm (in THIS case rather "Multiple Apps == One File".

This one .AppImage file never gets extracted during run time. It works because (upon starting the AppImage) the embedded mini-"runtime" binary automatically creates a temporary mount-point as /tmp/.mount_<something>, mounts the AppImage there and lets the embedded "AppRun" binary execute the payload application. Whenever the run time of the embedded binary ends and exits, the AppImage gets un-mounted again, and the temporary mount point is removed.

Upon next startup, a new temp mount point with a new name is in effect. (So you have to keep in mind that those absolute paths will never be the same across multiple restarts of AppImages. Do not copy'n'paste these absolute paths and rely on that mount point to be persistent.)

[michaelrsweet]

I've renamed this issue to track the functionality we are looking for, namely to be able to run a subset of tests and not just to break up this one test file. I also added it to the potential GSoC 2018 work for ipptool enhancements... :)

[michaelrsweet]

Tracking as libcups issue 13...