"context deadline exceeded" when working with firewall address-lists

[javierbertoli]

Describe the bug
I have a RB750Gr3 with routeros v7.19.1 where I have ~30k records in the IPv4 firewall address lists. There are just a few records being managed by us (less than 30). The rest, are dynamically added by an external tool, populating blacklists.

We're in the process of managing the address lists with terraform, so we added the code to manage the static resources we manage. I created them with routeros_ip_firewall_addr_list and imported them.

The issue I'm noticing now is that, when I run a terraform plan, I get the following error multiple times in a run, while trying to compare the address lists:

│ Error: Get "https://my.firewall.local/rest/ip/firewall/address-list?.id=*4": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
│
│   with routeros_ip_firewall_addr_list.ros
│   on firewall_v4.tf line 2, in resource "routeros_ip_firewall_addr_list" "ros":
│    2: resource "routeros_ip_firewall_addr_list" "ros" {
│

Getting the router to list the whole address lists with /ip/firewall/address-list/print without-paging takes over 20 seconds, so I guess that the provider is trying to get the whole lists to compare, for each resource it has to update? I can tell that the issue is not the router running out of resources, as I can monitor them while I run the plan, and its not even getting to use half of its capacity.

If this is the case, perhaps either getting the whole lists of addresses once and comparing locally, or adding timeouts to the resources might help?

To Reproduce

1. Add over 20k addresses with routeros_ip_firewall_addr_list
2. Modify a few
3. Try to run a plan

Expected behavior
Terraform should be able to manage the address lists without failing

[vaerh]

I will try to reproduce the problem soon and see what happens. There is a limitation related to timeout by MT: any REST request should not exceed 60 seconds. In order to get around this, I had to change the logic of one of the resources, if I remember correctly.

[javierbertoli]

I wonder if these api calls can benefit of using a filter, like !dynamic?

[javierbertoli]

@vaerh did you have a chance to give this issue a look?

I'm working around it using -targets for other resources, but it's not a nice thing 😬

(Not pressing, just asking. I know you do this on your own time and for free, so would not even consider demanding a LoC)

[vaerh]

I'm so sorry!
Things have gotten busy, I'll try to take a look soon.
I apologize again!