Call for volunteers: Help improve the NetBox plugins framework!

[jeremystretch]

Hey folks! One of our initiatives for the upcoming v3.2 release is to extend the plugins framework. Specifically, we're working on refactoring and improving NetBox's generic view classes, which are currently used only internally, so that they're suitable for use by plugins. (This initiative was originally proposed under #6221, but has been redefined under #8334.)

Of course, there's plenty more we can do to extend the current framework and make the development of NetBox plugins more efficient and convenient. Plugins were introduced in NetBox v2.8, and not much work has been done to expand on that implementation since. But that's about to change! Following on the success of working group 3979 last year, which helped inform the implementation of wireless network modeling, the maintainers have decided to launch a working group focused on improving the plugins framework.

What is a working group? This is a group of volunteers with experience developing NetBox plugins who will engage in discussion to share ideas for improving the plugins framework. Anyone with experience developing plugins (whether privately or as open source) is invited to participate. Participants will collaborate with the NetBox maintainers to identify common pain points and propose solutions. As consensus is reached, these proposals will be converted into NetBox feature requests and assigned release milestones.

If you're interested in participating, please comment below and share your experience working with NetBox plugins so far. We're setting up a separate GitHub repository to serve as a dedicated forum for the group to ensure discussions aren't missed in the hustle of the core NetBox project, and will be adding people as they volunteer. I'm excited to see what we come up with!

[hagbarddenstore]

As someone who have a custom Netbox with 7 custom Django applications in Netbox because the current plug-in support is too cumbersome to work with, I have loads of ideas on how to improve it. I want to be in this group! :)

…

On 12 Jan 2022, at 20:06, Jeremy Stretch ***@***.***> wrote:  Hey folks! One of our initiatives for the upcoming v3.2 release is to extend the plugins framework. Specifically, we're working on refactoring and improving NetBox's generic view classes, which are currently used only internally, so that they're suitable for use by plugins. (This initiative was originally proposed under #6221, but has been redefined under #8334.) Of course, there's plenty more we can do to extend the current framework and make the development of NetBox plugins more efficient and convenient. Plugins were introduced in NetBox v2.8, and not much work has been done to expand on that implementation since. But that's about to change! Following on the success of working group 3979 last year, which helped inform the implementation of wireless network modeling, the maintainers have decided to launch a working group focused on improving the plugins framework. What is a working group? This is a group of volunteers with experience developing NetBox plugins who will engage in discussion to share ideas for improving the plugins framework. Anyone with experience developing plugins (whether privately or as open source) is invited to participate. Participants will collaborate with the NetBox maintainers to identify common pain points and propose solutions. As consensus is reached, these proposals will be converted into NetBox feature requests and assigned release milestones. If you're interested in participating, please comment below and share your experience working with NetBox plugins so far. We're setting up a separate GitHub repository to serve as a dedicated forum for the group to ensure discussions aren't missed in the hustle of the core NetBox project, and will be adding people as they volunteer. I'm excited to see what we come up with! — Reply to this email directly, view it on GitHub, or unsubscribe. Triage notifications on the go with GitHub Mobile for iOS or Android. You are receiving this because you are subscribed to this thread.

[DanSheps]

Just curious, what all did you customize?

[hagbarddenstore]

One application to manage firewalls / virtual firewalls, both their clusters, their managers, the actual firewalls, rules etc.
One application to manage load balancers, be it physical clusters, virtual instances, or a mix of both.
One application to manage database clusters.
One application to manage secrets stored in Hashicorp Vault (This could be a plugin, but I really dislike the forced-upon URL scheme)
One application to manage storage allocations, be it from centralised storage (SAN), cloud providers or a mix.
One application to manage installed software on Device/VirtualMachine, like which version of SQL Server is installed.
One application to manage VPN servers, be it physical appliances or virtual ones.

We have scheduled to create more applications to manage licenses, certificates and patch schedules.

[rstambach1]

I would like to help with this group too.

[DanSheps]

Have you written any plugins for NetBox?

[rstambach1]

Daniel,
Yes, we have been using and customizing NetBox for DICM in our labs for about 3 years now. We also have used NetBox in our discovery process with LOC (Lenovo Open Cloud). Early last year we converted our lab customization into a plugin format. The current focus of our plugin is for DCIMA.

[ryanmerolle]

I went through the netbox-plugins published on github, and I want to make sure those maintainers see the discussion. I likely missed people and only went by those whose plugin was tagged using the netbox-plugin topic.

Any interest to join this NetBox plugin framework working group to shape both the user experience, their development, and the interaction of plugins with NetBox core?

@991jo
@FlxPeters
@MajesticFalcon
@N-Multifibra
@PieterL75
@alsigna
@artyomovs
@candlerb
@devon-mar
@drygdryg
@fabi125
@goebelmeier
@hbasria
@hiddenman
@hoanhan101
@iDebugAll
@jeremyschulman
@jinaloo7
@k01ek
@mattieserver
@minitriga
@mraerino
@peteeckel
@progala
@robertcsapo
@rodvand

Apologies for the spam if you are not interested in volunteering.

[peteeckel]

I'd definitely be interested as in my experience one has the choice between an extremely limited interface or instability with respect to Netbox version upgrades at the moment. Glad to help if and when I can.

[ryanmerolle]

I think that is totally a fair point. Its an issue users really have no understanding of, nor do they actually care about the reasons, but they get super annoyed.

Plugins may not work because of a version compatibility check in the plugin and nothing to do with actual integration issues. Its just a mechanism plugin maintainers put in to avoid users installing plugins with new version of NetBox they have not tested against.

[Jenjen1324]

That's great timing! I'm starting a bachelors thesis/project in February where the goal is to provide configuration management for hardware through NetConf/YANG and/or NAPALM. I've already worked with lots of django applications and have some experience developing a Netbox plugin from scratch.

A few points of interest for me in the plugin interface:

• Ability to extend the GraphQL API from within plugins
• More possibilities to extend standard UI elements from Plugins (e.g. Tabs within Models and such, edit views, etc.) - I'm aware that caution is required when adding such features to not impair maintainability
  • I was planning to attempt reopening Tab support for plugins #5556 for this reason

While my key motivation is to aide our network engineers in automating their configuration through templates and thus attempt to make available data in the templates more discoverable, anything that would ease the development for plugin maintainers is an added benefit.

[DanSheps]

More possibilities to extend standard UI elements from Plugins (e.g. Tabs within Models and such, edit views, etc.) - I'm aware that caution is required when adding such features to not impair maintainability

This is great as it is going to be one of the focus' of the group is to find where to be able to extend the standard UI. For example, I think it would be nice for plugins to be able to add tabs to a device page, instead of being limited to just the standard left/right/full.

[PieterL75]

That section does need attention.
I'm pleased to see this getting traction!

What I miss, is the possibility to merge plugins to the UI, do that they become 'invisible'.
Extend the current menus, create a new block on a fixed position (not only at the bottom, but also on the top; and even replace an existing one)
Extend fixed dropdowns (device status,...)
Allow extension of existing functions in models, so that we can code extra features on top

I'm not a real developer, rather a scripter that tries very hard.
So my input will only in words an not in code...

Good luck all!

[mraerino]

I think i could provide some input too.

The vault plugin i made is a primarily frontend plugin and i'm curious to see whether we can improve the experience when writing a lot of custom JS for a plugin.

[jeremypng]

I would like to be a part. I have definitely felt the pain of the UI limitations without access to the internal views. My plugin for DNA Center, ISE, VManage, and NS1 is still stuck in 2.11 land because of that. I have plans to update it, but if improvements can be made for the 3.2 release it might better to see how those shake out.

[MajesticFalcon]

I'm definitely interested. I built several plugins for our business specifically revolving around fiber optic ISP and fixed wireless ISP operations. I ported a lot of what was not exposed to the plugins into the plug-in environment. Things like button generators, select2 wrappers, site engine functions, etc. I'll be happy to work on this with everyone.

[devon-mar]

I'd be interested in joining! One thing that I would like to see is the ability for plugins to take advantage of the the new dynamic configuration feature of v3.1.

[peteeckel]

Let me summarise some of the obstacles I rain into while developing for the auroraresearchlab/netbox-dns project. I assume they are quite typical for a plugin project.

Internal APIs

Many of the APIs providing useful functionality and a homogenous UX across Netbox and Netbox plugins are private and according to the development guidelines can't be used (or at least they can't be used when one doesn't want to test against each and every minor release and fix quirks).

The ones I stumbled across so far were Tags and Export Templates, but Custom Fields, Webhooks and - maybe the most promising feature for some purposes involving automation with Netbox as a data source - Config Context would also be very useful.

HTML Templates

For example with tags there is not only the problem that the Netbox classes in plugins isn't supported, but the template functionality is internal as well and did change in the last few releases (i.e. extras/inc/tags_panel.html moved to inc/panels/tags.html), which requires quite ugly version checks in the plugin's own templates to keep compatible with at least the last few versions of NetBox.

It would be really useful to have stable and documented APIs and templates for

• Tags
• Custom Fields
• Export Templates
• Journal Entries
• Webhooks

Testing

Another thing I came across is the test framework provided by Netbox. There are some really useful prepared test classes (i.e. APITestCase, APIViewTestCases.*, ViewTestCases.*), but using them usually requires using non-supported Netbox base classes as well. So one has the choice of either doing that or writing all test code from scratch.

Here it would be useful to be able to switch some test features on or off per test class when the underlying functionality is not required by the plugin code.

Using plugin data in Netbox core

A bit more involved and probably nothing that can be achieved in the short run, but also an interesting feature would be an interface that could be used to refer to plugin data within Netbox core.

In the 'netbox-dns'-plugin for example there are DNS record data mapping host names to IP addresses. In the Netbox core IPAM section, a DNS host name can be assigned to an IP address.

It would be a very sensible feature to be able to override a 'hook' class in Netbox that is normally used to enter the DNS name for an IP address and override it with a class that provides the functionality of looking up the IP address in the DNS model for DNS records and provides the user with a drop down menu with the possible resolutions for the address.

[jeremystretch]

These are great insights @peteeckel! Now that we have the WG8333 forum opened up, would you mind creating an issue for each of these topics?

[jeremystretch]

Thanks to everyone who has expressed so far! If you have a 👍 from me on your reply in this thread, you should have an invitation to the netbox-wg8333 repo that we'll use for discussion. (Please let me know if you have any issues accessing the repo.)

Of course, we'll continue to add people who volunteer, so please comment here if you're interested!

[MajesticFalcon]

@jeremystretch i didn't not accept my invite quick enough and it had expired. Would you mind resending?

[jeremystretch]

GitHub showed it as still pending, but I removed you and re-added, so you should see an invite now.

[jeremyschulman]

I am interested as well, thank you for the invite!

…

On Mon, Jan 17, 2022, 10:37 AM Jeremy Stretch ***@***.***> wrote: Thanks to everyone who has expressed so far! If you have a 👍 from me on your reply in this thread, you should have an invitation to the netbox-wg8333 repo that we'll use for discussion. (Please let me know if you have any issues accessing the repo.) Of course, we'll continue to add people who volunteer, so please comment here if you're interested! — Reply to this email directly, view it on GitHub <#8338 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAT4WVAVLMNTUHOMNRWYWLLUWQZU7ANCNFSM5LZYJVHA> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>. You are receiving this because you were mentioned.Message ID: ***@***.*** com>

[robertcsapo]

I agree with @peteeckel on the difficulty to navigate between official supported netbox-plugin interfaces and internal netbox core interfaces, while creating the ciscodnacnetbox plugin.

Like getting the plugin to follow the netbox theme across the ui views. (StaticSelect2 vs StaticSelect, base.html vs base/layout.html)

from utilities.forms import BootstrapMixin, StaticSelect

{% extends 'base/layout.html' %}
{% load buttons %}

Another thing I've struggle with was the cacheops (netbox v2), when creating objects.

[jeremystretch]

Thanks, but we're not doing feedback in this thread. Would you like to join the working group?

[robertcsapo]

Sure

[minitriga]

Ill be up for helping out.

[candlerb]

I would be happy to lurk. My integrations with Netbox have been via REST API and database rather than via plugins. However I do have some experience with Salesforce which is the grand-daddy of extensible application platforms.

[artyomovs]

Hello Jeremy. I would be happy to join to this Avengers team!
Long time ago I created a plugin for comparing changes in devices configuration like git diff style.
Unfortunately, I didn't have enough time to make it support Netbox 3.x. Will do it asap.

Here is a video about the plugin.

[hbasria]

I would be happy to join this team! it would be a good work for our ongoing project netbox-dns and planned projects

[hatsat32]

Hi all! I am one of the developers of netbox-dns plugin and I am very excited about improving netbox plugin framework. Thanks.

[jcralbino]

I would like to be involved
Thanks

[DanSheps]

What sort of plugins have you developed? As per the original call for volunteers, we are looking for those with NetBox plugin development experience.

[jcralbino]

Hello Dan , i have tested the plug-in framework to improve our integration with the automaton pipeline. I have found that custom scripts are limited in the output information, after and during execution of the script, they can provide and we are migrating them as a plug-in so we can improve its usability.
The goal is to deploy something closer to a self service portal for automated use cases like ( configuring aci interfaces, removal of vlans from aci, validating information in netbox vs operational state in infrastructure)

[iDebugAll]

Hi all! I am interested and would like to join. I’m a creator and maintainer of two feature-rich and pretty popular (300+ GH stars and 36k+ PyPI downloads in total) NetBox plugins: NextBox-UI-plugin and PhoneBox-plugin. The first one has been maintained since very first plugin-enabled NetBox versions so I’m familiar with the NetBox internals and codebase.

[dvaccarosenna]

Hi, I would be interested in joining the working group if invitations are still available.

I've been working on private plugins in my employment to support a variety of use cases, including:

• Network Functions Virtualization Infrastructure (NFVI)
• 5G Slicing
• Integration with a homemade authentication and authorization provider

We took the unrecommended decision to use many of NetBox's internal view/util classes and recently suffered when migrating them to work all the way up to NetBox v3.x (we started at NetBox v2.8).

I've stumbled on some of the limitations of the framework already identified by others in this discussion, including:

• needing to add tabs to models in the UI
• needing to add new navigation menus or items to the home page
• needing the GraphQL API to include models from my plugin

Often I've either worked around them or produced patches, so it'd be great to contribute in discussions and/or with code!

[jeremystretch]

Added

[raddessi]

Hi, I would love to join this group.

My personal experience with plugin development has been on a currently-private load-balancing plugin that lets us model our LB services in netbox. I'm trying to get it open sourced but that is slow going. Others in this thread have commented about most of the issues I had building it. Im glad to see the community is so active :)

It seems that you have probably fixed most of the issues myself and many others have found at the moment but I would very much appreciate being in the steering group. Thanks!

[jeremystretch]

Added

[emersonfelipesp]

If it's not late, I would like to join the working group also!

My main plugin is netbox-proxbox, although I have some other private working going on!

[jeremystretch]

Added

[emersonfelipesp]

Hi @jeremystretch! It would be great if you also add @marcosvella! He's helping me a lot on netbox-plugins-store and has private plugins too.

[ITJamie]

Is it too late to get added to this for future discussions? Im working on two internal plugins. One will open sourced this month.

[jeremystretch]

Actually I think I'm going to disband the working group and move plugin framework discussions into general feature discussion. The intent behind creating a closed group was to ensure that the initial work was driven by people with firsthand experience with plugin development, and we knocked out quite a bit for v3.2. Now that the foundation of the framework has been laid, I don't think the working group is necessary anymore.

[ryanmerolle]

One will open sourced this month.

What is the general idea of the one being open sourced?