V's Veb in Web Framework Benchmarks

[brmassa]

There is a "competition" for Web Framework called Web Framework Benchmarks. It's redesigned to compare different languages and frameworks in "common" scenarios.

I've created a sample server using V's Veb, DB and ORM standard libraries. I will submit it to the benchmark organizers, but before, it would be great to have outside opinion and improvements.

https://github.com/brmassa/VTechEmpowerFrameworkBenchmarks

NOTE: as of 2025-03-24, the cached query test is not correctly implemented

NOTE: the only V alternative already present in the benchmark is the pico.v project, but it does not use the standard Veb server nor have any database connections, which "disqualifies" it from the final compiled benchmark results.

[medvednikov]

Veb* :)

Ved is the editor.

[brmassa]

I've written ALL the references wrong! Thanks Alex. fixed

[vandergrafgenerator]

Keep working on it, if you manage to obtain very high performance in every category it will attract massive interest to V and Veb.

[Avey777]

Keep working on it, if you manage to obtain very high performance in every category it will attract massive interest to V and Veb.

Agree with this view

[medvednikov]

I'll have a look and get back to you tomorrow.

[Avey777]

Looking forward to seeing how VEB compares to other frameworks

[brmassa]

Someone submitted a similar code to the competition before I did. It's about the same code, but only uses Postgres and some ORM. I will ()

I' ran the benchmark locally for H2O (a C web server), pico.v (a simple V server) and Veb. The results are the following:

NOTE: the benchmark uses 100% of my CPU and MEM, even being a powerful machine. because i had some apps opened at the same time, some results might be a bit affected

Test Name	Parameter	h2o	pico.v	veb
cached-query	1	8,390,848	-	-
cached-query	10	7,940,732	-	-
cached-query	20	7,414,285	-	-
cached-query	50	4,971,790	-	-
cached-query	100	4,067,494	-	-
db	16	2,005,309	-	764,114
db	32	2,582,701	-	1,019,224
db	64	3,208,533	-	1,181,013
db	128	3,554,495	-	1,183,047
db	256	3,995,039	-	1,210,602
db	512	4,182,292	-	1,192,207
fortune	16	2,043,143	-	849,001
fortune	32	2,599,302	-	1,124,590
fortune	64	3,104,106	-	1,169,592
fortune	128	3,602,634	-	1,168,337
fortune	256	3,904,273	-	1,166,060
fortune	512	4,125,815	-	1,137,132
json	16	6,013,156	6,246,700	2,559,335
json	32	6,729,767	7,229,221	3,007,447
json	64	7,673,903	7,922,773	3,286,791
json	128	7,929,046	8,397,069	3,314,099
json	256	8,093,027	6,391,806	3,341,491
json	512	8,276,644	2,123,336	3,403,816
plaintext	256	13,942,678	73,871,184	252
plaintext	1024	13,998,770	74,252,688	1,020
plaintext	4096	13,638,048	63,782,384	4,092
plaintext	16384	15,149,226	58,123,648	12,216
query	1	4,144,811	-	1,173,840
query	5	1,310,959	-	319,793
query	10	698,518	-	173,051
query	15	472,026	-	126,349
query	20	359,862	-	103,081
update	1	2,268,008	-	651,970
update	5	866,615	-	169,460
update	10	520,634	-	99,612
update	15	371,870	-	68,980
update	20	288,195	-	51,766

Some thoughts:

• Veb's bootstrapping is slow. Since picov also uses the same json library, it's all in the server account
• the plaintext results are strange. picov did 50k+ request (i doubt it) and Veb a minuscule number.
• by looking the results, Veb will be approximately the mid-position in the benchmark (considering that the "cache-queries" is eventually implemented, since it's generally half the performance of H2O)

I will now config and run my code from the repository I've presented and try to implement a simple DB code in pico.v to check the impact of database connection.

[brmassa]

Update: I've just finished the Veb + Postgres benchmarks. MySql ones will come later.

Scenarios

1. veb_brmassa: Veb + Postgres with all manual SQL commands with compilation as v -prod -cc clang -cflags '-std=gnu11 -Wall -O3 -march=native -mtune=native -flto' -o veb .
2. veb_brmassa-pg_orm: Veb + Postgres with all manual SQL commands
3. veb_brmassa-pure: same as the Add a gitter channel please #1, but with the compilation without any flag v -prod -o veb .
4. veb_brmassa-run: same as Add a gitter channel please #1, but using the following script to create multiple instances of the server to allow using all memory/cpu possible (it's the same script used by pico.v and veb tests I did before)

#!/bin/sh

for i in $(seq 0 $(($(nproc --all)-1))); do
  taskset -c $i ./veb &
done

wait

Thoughts

• the plaintext results are still super weird using Veb. The code could not be simpler (maybe a caching mechanism is gettings triggered?):

pub fn (app &App) plaintext(mut ctx Context) veb.Result {
	return ctx.text('Hello, World!')
}

• using the run.sh script gives 3~6x the performance. All other V frameworks use it, but I think it's kinda cheating. The web server was supposed to scale automatically.
• using ORM and sql commands made about 10% slower than using manual SQL calls. The only real benefit of ORM/SQL is the variable injection. It is as verbose as normal SQL, has fewer options (I could not run a DROP TABLE IF EXISTS World because the IF EXISTS is not implemented)
• compiling with the flags basically did not help. Gains of less than 3%.
• My having a lot of problems making code that runs both in Postgres and MySQL. They already have small internal differences, but V db library APIs differ. I could not use exec_param_many in MySQL.
• I believe V db library alters the SQL commands, specially the case of the table names.
• V fmt forces structs with uppercase and fields with lowercase. It's a source of errors if we want to match with external sources, like SQL and JSON.
• Again, the JSON test is 3x slower than using pico.v (which is basically the only test we can compare them), so it's the Veb weight here.

Benchmark (in my computer)

Test Name	Parameter	veb_brmassa	veb_brmassa-pg_orm	veb_brmassa-pure	veb_brmassa-run
db	16	251,303	222,056	250,211	969,312
db	32	251,834	222,852	244,302	1,082,666
db	64	250,073	221,053	248,050	1,290,204
db	128	248,702	219,919	249,013	1,323,779
db	256	248,046	218,752	247,258	1,328,911
db	512	247,794	217,044	247,522	1,303,981
fortune	16	252,144	227,706	246,093	910,254
fortune	32	251,201	225,544	247,043	1,146,077
fortune	64	250,692	224,981	245,589	1,222,037
fortune	128	247,680	224,572	243,259	1,245,039
fortune	256	248,101	223,015	243,573	1,233,299
fortune	512	246,154	219,963	244,023	1,205,231
json	16	875,621	871,501	874,893	2,614,082
json	32	873,514	881,079	878,880	3,115,502
json	64	899,823	900,664	888,969	3,309,067
json	128	893,217	885,784	888,698	3,359,048
json	256	872,403	869,308	859,932	3,429,851
json	512	856,434	855,121	855,800	3,443,567
plaintext	256	252	252	252	252
plaintext	1024	1,018	1,018	1,018	1,020
plaintext	4096	1,018	1,018	1,018	4,092
plaintext	16384	1,018	1,018	1,018	12,216
query	1	237,420	212,948	237,554	1,250,395
query	5	70,232	60,403	70,599	354,368
query	10	37,190	31,919	37,302	196,479
query	15	24,958	21,444	25,117	141,661
query	20	19,319	16,583	19,284	113,068
update	1	132,773	113,912	125,451	672,965
update	5	33,143	28,385	31,581	191,829
update	10	17,293	14,692	17,103	115,140
update	15	11,730	10,029	11,549	78,993
update	20	8,945	7,567	8,812	59,998

[brmassa]

I've removed the run.sh from pico.v test suit, so it calls a single ./main server instead multiple. the json results are 4x faster than Veb, and even faster with Veb using the script.

Test Name	Parameter	pico.v	pico.v-optimized
json	16	2,221,350	6,290,867
json	32	2,242,464	6,996,772
json	64	2,253,675	7,866,492
json	128	2,242,673	8,094,886
json	256	2,219,851	8,390,891
json	512	2,218,001	8,482,685
plaintext	256	20,609,488	74,460,352
plaintext	1024	19,209,936	73,739,808
plaintext	4096	18,998,880	63,747,984
plaintext	16384	18,249,952	57,245,568

[brmassa]

I've rewritten the pico.v code using current V (0.4.10), which already have pico.ev and picohttpparser modules (the current server code uses the C libraries with wrappers).

Test Name	Parameter	picoev	picoev-optimized
json	16	2,243,410	6,105,958
json	32	2,255,334	7,060,124
json	64	2,283,806	7,828,496
json	128	2,275,415	8,090,793
json	256	2,231,578	8,331,815
json	512	2,211,809	8,400,509
plaintext	256	504	504
plaintext	1024	2,038	2,040
plaintext	4096	2,038	8,184
plaintext	16384	2,038	24,456

Thoughts

• plaintext still suffering. So it's a problem on either pico.ev or picohttpparser V reimplementation.
• slightly slower than original, but it could be my setup (other programs opened in my pc during the benchmark). Negligible for the time being.
• the script that creates multiple server instances is still giving 4x performance. It means that pico.ev is either slow or not using the best configuration
• I will submit the new code to the competition too, because it reflects the current V state. It would be better to fix the plain

Code

import json
import picoev
import picohttpparser

struct Message {
	message string
}

@[inline]
fn json_response(mut res picohttpparser.Response) string {
	msg := Message{
		message: 'Hello, World!'
	}
	header_ok(mut res)
	res.json()
	return json.encode(msg)
}

@[inline]
fn hello_response(mut res picohttpparser.Response) string {
	header_ok(mut res)
	res.plain()
	return 'Hello, World!'
}

@[inline]
fn index_response(mut res picohttpparser.Response) string {
	header_ok(mut res)
	res.html()
	return 'Hello!\n<a href="/plaintext">plaintext</a>\n<a href="/json">json</a>'
}

fn callback(data voidptr, req picohttpparser.Request, mut res picohttpparser.Response) {
	if req.method == 'GET' {
		if req.path == '/plaintext' {
			res.body(hello_response(mut res))
		} else if req.path == '/json' {
			res.body(json_response(mut res))
		} else {
			res.body(index_response(mut res))
		}
	} else {
		res.http_404()
	}
	res.end()
}

fn header_ok(mut res picohttpparser.Response) {
	res.http_ok()
	res.header_server()
	res.header_date()
}

fn main() {
	config := picoev.Config{
		cb: callback
	}
	println('Starting webserver on http://localhost:${config.port}/ ...')
	mut server := picoev.new(config)!
	server.serve()
}

[JalonSolov]

"* V fmt forces structs with uppercase and fields with lowercase. It's a source of errors if we want to match with external sources, like SQL and JSON."

That's why V has attributes to allow you to specify the SQL/JSON names to match with the V names. Go has the same facility, since they also have restrictions on how things are named.

[brmassa]

1. what is the reason for the restrictions?
2. should we add multiple attributes, for each external integration (SQL, JSON, XML, YAML, TOML...)? Wouldn't it be easier (and faster to parse) if this hard-coded style was enforced by convention only (and usage of .editorconfig files)?

[JalonSolov]

The reason for the restrictions is that both V and Go are opinionated in their formatting. Certain things must be named certain ways, or the compiler won't compile your code. This is built into both languages at a fundamental level.

[brmassa]

Yes, I know.

Yet, I don't understand the reasoning for this decision. It makes the life harder for everyone.

pros:

• welcoming for Go devs
• consistency for 3rd party code

cons:

• not-welcoming for any Python, C, C++, Java, Javascript, C#, Rust...
• harder to write its own compiler
• harder to integrate with C (V's major selling point)
• harder to integrate with any other system and technologies
• 99% consistency == fail to compile

Being flexible would allow enforcing locally, in each project, with .editorconfig and proper code style documentation. The inverse is simply not possible (we cannot adopt other style at all). Forcing a style for everybody seems to have a negative net value.

---

Anyway, it's given. I've just noted because it has made my life a bit harder.

[vandergrafgenerator]

I guess it is to produce human readable C. A code format tool for proper style as seen in many languages should be ok. Or even better, check out Perl Critic, that tool is fantastic in practice and i have never seen equivalent in any other language.

Yes, I know.

Yet, I don't understand the reasoning for this decision. It makes the life harder for everyone.

pros:

• welcoming for Go devs
• consistency for 3rd party code

cons:

• not-welcoming for any Python, C, C++, Java, Javascript, C#, Rust...
• harder to write its own compiler
• harder to integrate with C (V's major selling point)
• harder to integrate with any other system and technologies
• 99% consistency == fail to compile

Being flexible would allow enforcing locally, in each project, with .editorconfig and proper code style documentation. The inverse is simply not possible (we cannot adopt other style at all). Forcing a style for everybody seems to have a negative net value.

Anyway, it's given. I've just noted because it has made my life a bit harder.