Implement Caddy JSON log parser #730
Conversation
Do you know of any other software that uses JSON for logs? In principle this could be done by scanning to a map[string]any, and then you can map JSON fields to things to import with something like --format=json:.. But if nothing else uses JSON then that's probably not worth it. |
|
I only know of Traefik but have not used it. |
DavidVentura
force-pushed
the
implement-caddy
branch
from
6bd2788 to
cf8c8e3
Compare
|
Okay, it'll be fine to just add a Caddy-specific method. I don't really feel like coming up and implementing a generic approach, especially when it's unclear that's needed. Looking at goaccess, it doesn't support JSON at all, and when people ask for it it's mostly for Caddy. A more generic approach can always be added later if it's needed – just accepting the "caddy" format doesn't prevent that. |
|
I merged your other PR – if you rebase on master I'll have a detailed look at this one. |
|
The test file only has a single request – it should have a few requests, to test that reading more than one request works. I also wonder if/how |
DavidVentura
force-pushed
the
implement-caddy
branch
from
4d46203 to
57a9aa4
Compare
| func (l CaddyLogEntry) Timing() time.Duration { | ||
| // TODO: `Second` should depend on the log format | ||
| // {seconds, nano, string} where string in {1m32.05s, 6.31ms} | ||
| return time.Duration(l.Duration * float64(time.Second)) |
There was a problem hiding this comment.
I wasn't clear on the best way of implementing this
There was a problem hiding this comment.
I think this can probably be auto-detected based on the type? Otherwise probably need to add a new flag for it or something. Need to look a bit at what the options are here – if you add a (failing) testcase for all possible options then I would have a better idea.
|
Caddy does indeed print one request per line, so that wouldn't be necessary. I've been using this branch with my Caddy instance since last week 😁 I've rebased from master and will add a test in a bit |
|
Right so – would be good to have logfiles as Caddy outputs them. Manually twiddling compacted JSON is a bit of a pain, but otherwise you're not really testing the real-world scenario. |
|
My use case is also caddy. What can unblock this PR? |
|
I think it's basically fine; IIRC I just wanted to write a bit of docs, but never really got around it. Basically, I just forgot 😅 I'll try and find some time for things this week or so. |
|
I forgot about this PR 😬 I have some time this week, let me know if there's anything you'd like me to do here |
|
The only open comment is: #730 (comment) I do remember now: what I wanted to do (but didn't) was to run Caddy and play around with possible options and see what does and doesn't work. I never really used Caddy and don't really have a good insight on how "complete" this PR is and what options Caddy does or doesn't have. |
This should be based on top of the
split-regexparserbranch but I'm not entirely sure how to do that.I understand that this does not allow for custom formats, only the base/default Caddy formatting. I'm not sure if a custom query language is desired or not.