Query API for Ubuntu `linux` packages

[dodys]

Describe the bug
Not sure if I'm actually missing something, but I'm getting a timeout when running:

$ curl -d '{"package": {"name": "linux-oracle-5.15", "ecosystem": "Ubuntu:20.04:LTS"}, "version": "5.15.0-1065.71~20.04.1"}' "https://api.osv.dev/v1/query"
{"code":504,"message":"upstream request timeout"}

And I get an empty answer when I run with a different version:

$ curl -d '{"package": {"name": "linux-oracle-5.15", "ecosystem": "Ubuntu:20.04:LTS"}, "version": "5.15.0-1065"}' "https://api.osv.dev/v1/query"
{}

What I would like to retrieve is some information like this:
https://osv.dev/vulnerability/USN-6957-1

Is this a problem with the data or the API?

To Reproduce
Steps to reproduce the behaviour:
Run the commands above and check the output.

Expected behaviour
The following data should be retrieved:
https://osv.dev/vulnerability/USN-6957-1

[hogo6002]

It seems that linux-oracle-5.15 shows in a lot of Ubuntu CVEs. This makes the query take a long time. Each CVE also lists information about a lot of other packages, which makes the query even slower and causes timeout.

Other Ubuntu queries, like curl -d '{"package": {"name": "curl", "ecosystem": "Ubuntu"}, "version": "8.2.1-1ubuntu3.1"}' https://api.osv.dev/v1/query, work fine.

We're currently working on a new way to make our API queries better and will think about this case. Thanks for reporting this.

[dodys]

It seems that linux-oracle-5.15 shows in a lot of Ubuntu CVEs. This makes the query take a long time. Each CVE also lists information about a lot of other packages, which makes the query even slower and causes timeout.

Other Ubuntu queries, like curl -d '{"package": {"name": "curl", "ecosystem": "Ubuntu"}, "version": "8.2.1-1ubuntu3.1"}' https://api.osv.dev/v1/query, work fine.

We're currently working on a new way to make our API queries better and will think about this case. Thanks for reporting this.

Thanks Holly!

I do have a question from your example. Passing the ecosystem as Ubuntu or Ubuntu:20.04:LTS, the expectation is for both to be accepted?

Also, what did you mean by non-enumerated ecosystem in #2401 ? Is it something that we could help with for Ubuntu?

[hogo6002]

Hi @dodys

I do have a question from your example. Passing the ecosystem as Ubuntu or Ubuntu:20.04:LTS, the expectation is for both to be accepted?

Passing Ubuntu will give you entries from all Ubuntu releases, and passing Ubuntu:20.04:LTS will only give you vulnerabilities from that release.

Also, what did you mean by non-enumerated ecosystem in #2401 ? Is it something that we could help with for Ubuntu?

For some ecosystems like Debian, we maintain a custom enumerate_versions() function to calculate all affected versions. We then use these enumerated versions to perform version queries for those ecosystems. But scaling this approach to all ecosystems is hard and has other limitations. So we're working on a new method for version querying that doesn't require enumerating all affected versions (#2401) and might replace the current enumeration query way.

The affected versions received from Ubuntu are very helpful, thanks for that! And nothing else we need from Ubuntu at this stage to support the new query method.

[oliverchang]

@another-rex is working on addressing the timeout issue this week. Thanks for reporting this @dodys !

[another-rex]

Hey @dodys, while implementing a fix for these queries, I noticed that

$ curl -d '{"package": {"name": "linux-oracle-5.15", "ecosystem": "Ubuntu:20.04:LTS"}, "version": "5.15.0-1065.71~20.04.1"}' "https://api.osv.dev/v1/query"

will return an empty array rather than https://osv.dev/vulnerability/USN-6957-1 like you are expecting.

This is because 5.15.0-1065.71~20.04.1 is not an affected version on that entry, but is the fixed version. Is this what you intended? Or should the fixed version in that record be last_affected instead?

[dodys]

Hey @dodys, while implementing a fix for these queries, I noticed that

$ curl -d '{"package": {"name": "linux-oracle-5.15", "ecosystem": "Ubuntu:20.04:LTS"}, "version": "5.15.0-1065.71~20.04.1"}' "https://api.osv.dev/v1/query"

will return an empty array rather than https://osv.dev/vulnerability/USN-6957-1 like you are expecting.

This is because 5.15.0-1065.71~20.04.1 is not an affected version on that entry, but is the fixed version. Is this what you intended? Or should the fixed version in that record be last_affected instead?

Oh, so the idea is to pass a "vulnerable" version then to the API call?

[andrewpollock]

Oh, so the idea is to pass a "vulnerable" version then to the API call?

Not necessarily. The idea is to pass to it what you want to know the vulnerability status of. Your current usage is perfectly valid.

Linux kernel related packages are particularly pathologically expensive to evaluate due to the number of possible vulnerabilities to the filter out. What happens on the backend of the API is first the entire set of vulnerabilities are retrieved for the package/ecosystem combination, and then they're scanned for falling within the vulnerable version range.

You can follow the code from

osv.dev/gcp/api/server.py

Line 619 in f66ffcf

def do_query(query, context: QueryContext, include_details=True):

[andrewpollock]

Reviewing this,

curl -d '{"package": {"name": "linux-oracle-5.15", "ecosystem": "Ubuntu:20.04:LTS"}, "version": "5.15.0-1065"}' "https://api.osv.dev/v1/query"

now appears to yield a result (although it does not include USN-6957-1, and I would have expected it to). 🤔

[dsever]

Have similar issue, /v1/query for the linux package

{"version": "5.4.0-205.225", "package": {"name": "linux", "ecosystem": "Ubuntu:20.04:LTS"}

So when executing query, it dumps complete DB for package even from 2012. So at the end received 1925 vulnerabilities in total.

Any recommendation how to cope with it?

[hogo6002]

for package even from 2012. So at the end received 1925 vulnerabilities in total.

Hi @dsever, among all the vulnerabilities, do you find any inaccurate ones? the linux package does contain a lot of vulnerabilities. If you just want a list of vulnerability IDs instead of the details, you can use our batch query v1/querybatch.