Feature: Webcal support

[real-or-random]

It would be great to have support for "webcal" (often confused with CalDAV) to subscribe to publicly available calendars, e.g., for seminars, match schedules of sport events, etc. Webcal is nothing more than subscribing to an ICS file served over HTTP(S), i.e., it's a basically a subset of what's already supported:

• Fetching over GET with HTTP(S), no WebDAV extensions necessary
• sync only cal->org
  So even though it's a new feature, it should be relatively simple to add.

---

As a possible alternative, some people have apparently managed to import ics files into the diary, which is then displayed in org-agenda:

• https://emacs.stackexchange.com/questions/50137/show-apple-calendar-events-in-org-mode
• https://www.reddit.com/r/emacs/comments/ioenk2/ical_import_in_emacs_calendar/

But that's very different from what org-caldav can do.

[jackkamm]

Yes I think this would be good, and it would also be good to extract the ical->org conversion to a standalone module.

Relatedly there is the venerable ical2org.awk script:
https://orgmode.org/worg/org-tutorials/org-google-sync.html

As well as Python and Go ports:
https://github.com/rjhorniii/ical2org
https://github.com/ical2org-py/ical2org.py

But I think it is best to do this in elisp, and the current implementation in org-caldav is a good starting point.

EDIT to add: Note there is alsoorg-caldav-import-ics-to-org, but I don't think it is much used, and it just appends to the org-caldav-inbox rather than overwriting it. And only works with a local filepath, not a URL.

[jackkamm]

I've made a start on this with 619eb89

In particular, the new function org-caldav-import-ics-buffer--intern imports all events from an ICS file into an inbox file.

By passing a tmpfile as the inbox and then extracting the contents, it can be used to convert an ICS file into a standalone Org file/buffer. The new function org-caldav-convert-ics-to-datetree works this way.

The existing function org-caldav-import-ics-buffer-to-org has been modified to use org-caldav-import-ics-buffer--intern. So, it can now import from ICS files with multiple events.

This doesn't handle TODO items yet but I plan to add that in future. Possibly, we could also add more convenient interfaces/interactive commands to access this functionality. Long term, I'd like to separate out this functionality to a standalone ical2org.el module, but that will require a lot of refactoring first.

Currently, this only adds entries from iCalendar to Org, it never overwrites existing Org entries. For your use case, would it be useful to overwrite/update individual entries? Or do you envision overwriting the entire Org file on every update?

[real-or-random]

Importing is a nice feature to have! Though it's not really what I'd like to see.

Currently, this only adds entries from iCalendar to Org, it never overwrites existing Org entries. For your use case, would it be useful to overwrite/update individual entries? Or do you envision overwriting the entire Org file on every update?

My use case is really not just importing but "subscribing" to a Webcal calendar on the web. When the ICS file changes on the web, the entries in the inbox should be overwritten. This is really what org-caldav already does in the case of CalDAV calendars, so I think the CalDAV syncing logic should be reused for Webcal. The only technical differences are:

• The file is fetched via HTTPS instead of WebDAV.
• org->cal is simply not done for these calendars.

(Not sure if it's a good idea to overwrite the entire file. I guess we should do whatever the current flow does.)

As for config, there could be a simple variable org-webdav-urls, which is a list of calendars to fetch, and :url now indicates a simple HTTP URL instead of a WebDAV URL, and all the other keys are supported, e.g., :inbox as in org-caldav-calendars (except :sync-direction, which is implied). Or alternatively, there could be an additional entry type in the org-caldav-calendars, which distinguishes the two fetching methods using :webcal-url vs :url.

It is far more likely to have multiple servers in the Webcal case, as a calendar is simply identified via its URL and there is no concept of having multiple calendars on the same server. (Of course, there may be multiple calendars with different URLs on the same host, but nothing in the protocol would really care about this, I think.)

[grauschnabel]

That would be a great feature, but it is not really one of the goals in this project, I think. Maybe this could be in another package, because this can be confusing.

You have one calendar that is read only from the web and org entries should be overwritten. And than you have other calendar where org-mode entries are prioritized.

So an org-webcal package could be created. The goal would be to show webcal events in the org-agenda, so there would not be any need of managing org files by hand - such a package could handle that in the background (and a list of webcal urls should be enough in the config).

I don't need it because I use that on my phone together with nextcloud and Google, which both support that. But I would use it if it would exist.

[real-or-random]

So an org-webcal package could be created. The goal would be to show webcal events in the org-agenda, so there would not be any need of managing org files by hand - such a package could handle that in the background (and a list of webcal urls should be enough in the config).

This is how I use cal org-caldav. I use only cal->org one-way sync, even for my CalDAV calendar.

You have one calendar that is read only from the web and org entries should be overwritten. And than you have other calendar where org-mode entries are prioritized.

Okay sure, it will be confusing to have a different conflict resolution mechanism. But this is not necessary to make the feature work. Using the same conflicting resolution as currently would also work: As long as you don't touch the org entries, you receive updates from the web. If you want to edit your org entries, you're free to do it, but then you don't get updates for these entries. I think this model makes sense. IIUC, it's exactly what cal->org does currently.

[grauschnabel]

Maybe you can use it this way, but originally it was intended both ways and org-mode wins (if both were changed). Everything else came later. So yes, you can use it that way today.

[jackkamm]

I'd also like to note that the 1-way cal->org sync has a major flaw -- in the current implementation, all events in the eventdb are marked as deleted on Org side. So then if the calendar is switched to 2-way sync, this will cause all events to be deleted in the calendar! This has bitten me before, so I always avoid the 1-way sync mode now.

And IMO, doing 1-way sync with CalDav seems like an antipattern. For 1-way cal->org, it makes more sense to import the calendar wholesale using ical2org.awk or similar (hence my interest to extract out a standalone ical2org.el module from org-caldav). And in the other direction 1-way org->cal, one can simply use ox-icalendar.el, there is no need for CalDav either.

In a parallel example (non-Org), the excellent DAVx5 recommends the separate ICSx5 for 1-way subscription of ICS into Android (https://www.davx5.com/faq/subscribe-ics-file), as there is no need for CalDav in such case.

For these reasons I'm tempted to deprecate the 1-way conversion. But for now, I've just added a warning to the manual to avoid switching between 1-way and 2-way sync on the same calendar (086a8f3).

Using the same conflicting resolution as currently would also work: As long as you don't touch the org entries, you receive updates from the web. If you want to edit your org entries, you're free to do it, but then you don't get updates for these entries.

Could you verify whether the cal->org sync works this way? I was under the impression that the imported Org entries never receive updates, whether they have been edited or not. But I am not sure about this.

Finally, I'm not sure it's really so simple to adapt the org-caldav flow for WebCal. We don't pull the full ICS file, instead we query the CalDav server to determine which events have changed and need to be updated, then we pull those individually. In particular I'm unsure whether it's possible to determine whether an individual entry needs to be updated in WebCal. Possibly one could use the SEQUENCE property, but I'm not sure how robust that is, and is not how org-caldav determines whether to update an entry (we use ETag, which I don't think would exist for WebCal).

[real-or-random]

And IMO, doing 1-way sync with CalDav seems like an antipattern. For 1-way cal->org, it makes more sense to import the calendar wholesale using ical2org.awk or similar (hence my interest to extract out a standalone ical2org.el module from org-caldav).

Not if the calendar changes daily. I don't want to reimport every time. My workflow is that I (and collaborators) edit the calendars only outside of org. I simply want to get a read-only view in org to make sure that events appear in my org agenda so I can look at them and make sure that I don't schedule tasks at the same time.

Perhaps that makes it clearer why I want to do the same for Webcal: In this model, read-only subscribing to my (CalDAV) family calendar is not that different to a (Webcal) seminar calendar...

For these reasons I'm tempted to deprecate the 1-way conversion. But for now, I've just added a warning to the manual to avoid switching between 1-way and 2-way sync on the same calendar (086a8f3).

It may be philosophically inconsistent, but let me stress that it works perfectly fine for me. So as long as it doesn't cause a maintenance burden, please don't deprecate it...

Could you verify whether the cal->org sync works this way? I was under the impression that the imported Org entries never receive updates, whether they have been edited or not. But I am not sure about this.

Okay, I tried it again. We were both wrong: The org entry receives an update when the CalDAV entry changes. This is true even if first edit the org entry manually. Here, CalDAV always wins. And this makes sense if you think about: In fact, it's precisely what I need for my use case.

[jackkamm]

OK, thanks for the info and prompt response. We won't deprecate the 1-way sync. I may try to add in some guardrails to prevent users switching between 1-way and 2-way sync.

Indeed, I can see how it's useful to move around and update individual entries (as in org-caldav), rather than overwriting wholesale (as in ical2org). But it's still unclear to me whether this would be feasible with WebCal -- is there any mechanism to determine whether an individual entry changed, and is it conducive to the existing org-caldav machinery? If not, then ical2org may be the only feasible way.

[real-or-random]

But it's still unclear to me whether this would be feasible with WebCal -- is there any mechanism to determine whether an individual entry changed, and is it conducive to the existing org-caldav machinery? If not, then ical2org may be the only feasible way.

Okay, I see. This may indeed be a problem then. Since Webcal is really nothing else than an *.ics file over HTTP(S), I'm pretty sure there's no way of asking the server which entries have changed. Of course, it will always be possible to fetch and update all entries, but I can see that this can be a bit slow (i.e., useful only for small calendars) and also a bit inelegant. And if I understand correctly, that's almost what you've now implemented in org-caldav-import-ics-buffer-to-org, except the fetching part. Perhaps there are faster ways of doing it, e.g., storing a hash of each entry along with the entry, but this makes things complicated. So, yep, I agree, that's a real difference to CalDAV then.

[jackkamm]

In the case of updating from WebCal or plain ICS instead of CalDav, we could use SEQUENCE property instead of ETag to update:
https://icalendar.org/iCalendar-RFC-5545/3-8-7-4-sequence-number.html
We already store SEQUENCE in the eventdb, so the database structure wouldn't have to change at least.

org-caldav-import-ics-buffer--intern might be a starting point for this. It would need to be changed to:

• Determine whether event already exists and whether it needs to be updated
• Update existing events rather than importing events as new always
• Update the eventdb. (This may need to be handled differently depending on whether we're using the iCalendar as a backend as in WebCal, or whether we're just importing events from an email attachment to be sync'd with CalDav later)
• Delete events that were removed (Again, this should only happen in WebCal case, not the email attachment case)