Skip to content

120000000 rows, when using ORDER BY + OFFSET will make duckdb hang for-ever : NEVER STOP, and 800% CPU #149

Open
@navegador5

Description

@navegador5

sql to trigger the error:
`

await conn.all(select * from pass_vehicle ORDER BY pass_time,flowid LIMIT 1 OFFSET 2000000);

进程号 USER PR NI VIRT RES SHR %CPU %MEM TIME+ COMMAND
268710 root 20 0 5943028 3.9g 74548 S 880.5 6.2 4:46.17 node
进程号 USER PR NI VIRT RES SHR %CPU %MEM TIME+ COMMAND
268710 root 20 0 7253748 5.2g 74548 S 905.9 8.3 7:10.05 node
进程号 USER PR NI VIRT RES SHR %CPU %MEM TIME+ COMMAND
268710 root 20 0 26.7g 23.5g 74548 S 1086 37.5 25:55.07 node

`

datas:
`
total : 120000000 rows. each row has 20 column

storage using local-file:

DUCKDB# ls -l | egrep hbj_vehicle_monitor
-rw-r--r-- 1 root root 2972725248 1月 19 20:06 hbj_vehicle_monitor
-rw-r--r-- 1 root root 3805 1月 19 20:06 hbj_vehicle_monitor.wal

{
column_name: 'flowid',
column_type: 'VARCHAR',
null: 'YES',
key: null,
default: null,
extra: null
}
{
column_name: 'pass_time',
column_type: 'TIMESTAMP',
null: 'YES',
key: null,
default: null,
extra: null
}

`

nodejs:
Node.js v22.13.0.

machine env:
`

cat /proc/cpuinfo | egrep "processor|model name"

processor : 0
model name : 12th Gen Intel(R) Core(TM) i9-12900K
processor : 1
model name : 12th Gen Intel(R) Core(TM) i9-12900K
processor : 2
model name : 12th Gen Intel(R) Core(TM) i9-12900K
processor : 3
model name : 12th Gen Intel(R) Core(TM) i9-12900K
processor : 4
model name : 12th Gen Intel(R) Core(TM) i9-12900K
processor : 5
model name : 12th Gen Intel(R) Core(TM) i9-12900K
processor : 6
model name : 12th Gen Intel(R) Core(TM) i9-12900K
processor : 7
model name : 12th Gen Intel(R) Core(TM) i9-12900K
processor : 8
model name : 12th Gen Intel(R) Core(TM) i9-12900K
processor : 9
model name : 12th Gen Intel(R) Core(TM) i9-12900K
processor : 10
model name : 12th Gen Intel(R) Core(TM) i9-12900K
processor : 11
model name : 12th Gen Intel(R) Core(TM) i9-12900K
processor : 12
model name : 12th Gen Intel(R) Core(TM) i9-12900K
processor : 13
model name : 12th Gen Intel(R) Core(TM) i9-12900K
processor : 14
model name : 12th Gen Intel(R) Core(TM) i9-12900K
processor : 15
model name : 12th Gen Intel(R) Core(TM) i9-12900K

DUCKDB# cat /proc/meminfo | egrep "Hugepagesize|Mem"
MemTotal: 65849100 kB
MemFree: 3627328 kB
MemAvailable: 35407524 kB
Hugepagesize: 2048 kB

DUCKDB# uname -a
Linux debian 5.10.0-32-amd64 #1 SMP Debian 5.10.223-1 (2024-08-10) x86_64 GNU/Linux

`

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions