You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I am using stream to move a big data set, but it seems there is a cap of ~460 iterations that causes an error i dont understand.
This is my code:
leti=0;conststream=connection.query('SELECT * FROM schedules ORDER BY id ASC').stream();stream.on('data',result=>{stream.pause();db.collection('packages').deleteOne({id: result.id}).then(()=>{db.collection('packages').insertOne(result).then(()=>{i++;if(i%1000===0){console.log('Processed',i);}stream.resume();});});});stream.on('end',()=>{connection.end();mongoClient.close();console.log('Finished streaming',i);});
at readLengthCodedNumberExt (/home/bun/app/node_modules/mysql2/lib/packets/packet.js:231:13)
at readLengthCodedString (/home/bun/app/node_modules/mysql2/lib/packets/packet.js:381:22)
at next (32:68)
at row (/home/bun/app/node_modules/mysql2/lib/commands/query.js:242:29)
at execute (/home/bun/app/node_modules/mysql2/lib/commands/command.js:45:22)
at handlePacket (/home/bun/app/node_modules/mysql2/lib/connection.js:481:34)
at /home/bun/app/node_modules/mysql2/lib/connection.js:97:12
at executePayload (/home/bun/app/node_modules/mysql2/lib/packet_parser.js:139:14)
at /home/bun/app/node_modules/mysql2/lib/connection.js:104:25
227 | res = resNumber.toString() === resString ? resNumber : resString;
228 |return bigNumberStrings ? resString : res;
229 | }
230 | // eslint-disable-next-line no-console
231 |console.trace();
232 | throw new Error(`Should not reach here: ${tag}`);
^
error: Should not reach here: undefined
at readLengthCodedNumberExt (/home/bun/app/node_modules/mysql2/lib/packets/packet.js:232:11)
at readLengthCodedString (/home/bun/app/node_modules/mysql2/lib/packets/packet.js:381:22)
at next (32:68)
at row (/home/bun/app/node_modules/mysql2/lib/commands/query.js:242:29)
at execute (/home/bun/app/node_modules/mysql2/lib/commands/command.js:45:22)
at handlePacket (/home/bun/app/node_modules/mysql2/lib/connection.js:481:34)
at /home/bun/app/node_modules/mysql2/lib/connection.js:97:12
at executePayload (/home/bun/app/node_modules/mysql2/lib/packet_parser.js:139:14)
at /home/bun/app/node_modules/mysql2/lib/connection.js:104:25
at emit (node:events:183:48)
if i use async/await instead of thenables i have the same result, only when i dont do both it runs through, but then 3mio db operations stuck up in memory and make dev ops kill it.
all i want is to do this in the background a little slower.
The text was updated successfully, but these errors were encountered:
I am using stream to move a big data set, but it seems there is a cap of ~460 iterations that causes an error i dont understand.
This is my code:
if i use async/await instead of thenables i have the same result, only when i dont do both it runs through, but then 3mio db operations stuck up in memory and make dev ops kill it.
all i want is to do this in the background a little slower.
The text was updated successfully, but these errors were encountered: