Backport memory optimization for palette mode from dav1d 1.3.0 #4117

Job	Run time
test-on-macos-latest	5m 2s
test-on-ubuntu-latest (x86_64-unknown-linux-gnu, release, --release --no-default-features --featu...	2m 2s
test-on-ubuntu-latest (x86_64-unknown-linux-gnu, release, --release --no-default-features --featu...	2m 12s
test-on-ubuntu-latest (x86_64-unknown-linux-gnu, release, --release --no-default-features --featu...	1m 46s
test-on-ubuntu-latest (x86_64-unknown-linux-gnu, opt-dev, --profile opt-dev, 2)	2m 40s
test-on-ubuntu-latest (x86_64-unknown-linux-gnu, opt-dev, --profile opt-dev, 2, -f 1)	2m 37s
test-on-ubuntu-latest (x86_64-unknown-linux-gnu, opt-dev, --profile opt-dev, 2, -f 2)	3m 34s
test-on-ubuntu-latest (i686-unknown-linux-gnu, release, --release --no-default-features --feature...	2m 29s
test-on-ubuntu-latest (i686-unknown-linux-gnu, release, --release --no-default-features --feature...	2m 26s
test-on-ubuntu-latest (i686-unknown-linux-gnu, release, --release --no-default-features --feature...	2m 5s
test-on-ubuntu-latest (i686-unknown-linux-gnu, opt-dev, --profile opt-dev, 2)	2m 4s
test-on-ubuntu-latest (i686-unknown-linux-gnu, opt-dev, --profile opt-dev, 2, -f 1)	2m 4s
test-on-ubuntu-latest (i686-unknown-linux-gnu, opt-dev, --profile opt-dev, 2, -f 2)	3m 2s
	34m 3s

Provide feedback