Support COM port discovery via USB VID/PID

[MCUdude]

Apparently, @mariusgreuel's Avrdude fork for windows supports COM port discovery via USB VID/PID. This is actually very neat if you're using a UART-based programmer that has a tendency to bump the COM port number. I have a similar issue on my mac as well; The serial port (/dev/cu.*) gets a different name depending on which USB port I connect it to.

Would it be possible, within a reasonable amount of time, to make this work on non-windows systems as well?

[mariusgreuel]

The crucial part is figuring out the link between the USB device and the serial device. I never looked into non-Windows implementations, but on Windows there are two parts of metadata one can use: 1) You can enumerate through the device tree, and find the parent of the serial device, which will be the USB device. That is tricky if the USB device is a composite device. 2) Windows actually maintains a bunch of properties for every device, and Portname does what I needed.

[dl8dtl]

For a completely different example, on FreeBSD, they can only be examined by sysctl.
Here's the example of the tty port of a curiosity nano board:

# sysctl dev.umodem
dev.umodem.0.ttyports: 1
dev.umodem.0.ttyname: U3
dev.umodem.0.%parent: uhub12
dev.umodem.0.%pnpinfo: vendor=0x03eb product=0x2175 devclass=0xef devsubclass=0x02 devproto=0x01 sernum="MCHP3372031800002359" release=0x0100 mode=host intclass=0x02 intsubclass=0x02 intprotocol=0x01 ttyname=U3 ttyports=1
dev.umodem.0.%location: bus=4 hubaddr=4 port=4 devaddr=5 interface=1 ugen=ugen4.5
dev.umodem.0.%driver: umodem
dev.umodem.0.%desc: Microchip Technology Incorporated nEDBG CMSIS-DAP, class 239/2, rev 2.00/1.00, addr 5
dev.umodem.%parent: 

These attach to the umodem driver (aka. CDC device), and the respective device here is /dev/cuaU3. Alas, one would have to walk across all possible drivers in addition to umodem (FTDI, PL230x, CH3xx).

So question is to find out how to get this information on MacOS.

Downside: it adds a lot of per-OS code to AVRDUDE.

[MCUdude]

Thank you for the details! I'll do a bit of research this evening to see if I can figure it out on MacOS.

Downside: it adds a lot of per-OS code to AVRDUDE.

Good point. But on the other side, this (and probably the "Arduino Leonardo style") is IMO functionality that should be considered even though OSes handles this differently. The end-user won't notice anything, and it's very convenient!

I'll agree that adding functionality to one "OS build" that wouldn't work for the others is a bad idea, but if we can figure out how to do it for Windows, MacOS, FreeBSD and other UNIX compatible OSes, I think we should consider it for the sake of the convenience this functionality adds.

[dl8dtl]

I guess finding a way for Linux will be possible as well.
It would make sense to somehow abstract an API for that, and put that into its own implementation file, a bit like that whereami.[ch] stuff that has recently been added to find out about the location of the executable.

[MCUdude]

... a bit like that whereami.[ch] stuff that has recently been added to find out about the location of the executable.

Is whereami only needed for Windows builds?

I agree that it would make sense to "separate" OS-specific code into a separate API in order to not "pollute" existing code. The good this about this is that it makes it easier to add more OS-specific code later on. We should, however, be restrictive and not just throw in all the bells whistles just because we can. At the moment I can't think of other "nice to have" features that rely on OS-specific code other than port discovery via VID/PID and "1200bps touch" used on various Arduino boards.

EDIT:
Another wild thought. Instead of always having to "manually" find the VID/PID how about a keyword in avrdude.conf that could hold the VID, PID, and other information about a serial device as well?

serialadapter
  id        = "ch340";
  desc      = "CH340* USB to serial adapter";
  usbvid    = 0x1A86;
  usbpid    = 0x7523;
;

serialadapter
  id        = "cp2102";
  desc      = "CP2102* USB to serial adapter";
  usbvid    = 0x10C4;
  usbpid    = 0xEA60;
;

serialadapter
  id        = "cp2104";
  desc      = "CP2104* USB to serial adapter";
  usbvid    = 0x10C4;
  usbpid    = 0xEA60;
;

$ avrdude -C avrdude.conf -p avr128da48 -c serialupdi -P ch340 -v -t

[MCUdude]

@dl8dtl It looks like sysctl is included in MacOS, but I couldn't figure out how to get it to work.
However, Google reveals that there is at least two commands that does what we want. Currently, I have a CH340N USB to serial adapter and a PicKit4 connected to my mac.

First command:

$ ioreg -p IOUSB 
+-o Root  <class IORegistryEntry, id 0x100000100, retain 15>
  +-o AppleUSBXHCI Root Hub Simulation@14000000  <class AppleUSBRootHubDevice, id 0x100000340, registered, matched, ac$
    +-o Bluetooth USB Host Controller@14300000  <class AppleUSBDevice, id 0x10000046b, registered, matched, active, bu$
    +-o USB2.0-Serial@14100000  <class AppleUSBDevice, id 0x10000069e, registered, matched, active, busy 0 (2 ms), ret$
    +-o MPLAB PICkit 4 CMSIS-DAP@14200000  <class AppleUSBDevice, id 0x1000006b0, registered, matched, active, busy 0 $

Second command:

$ system_profiler SPUSBDataType
2022-03-22 20:58:58.311 system_profiler[2351:23194] SPUSBDevice: IOCreatePlugInInterfaceForService failed 0xe00002be
USB:

    USB 3.0 Bus:

      Host Controller Driver: AppleUSBXHCIWPT
      PCI Device ID: 0x9cb1 
      PCI Revision ID: 0x0003 
      PCI Vendor ID: 0x8086 

        Bluetooth USB Host Controller:

          Product ID: 0x8290
          Vendor ID: 0x05ac (Apple Inc.)
          Version: 1.68
          Manufacturer: Broadcom Corp.
          Location ID: 0x14300000

        MPLAB PICkit 4 CMSIS-DAP:

          Product ID: 0x2177
          Vendor ID: 0x03eb  (Atmel Corporation)
          Version: 1.00
          Serial Number: BUR204071896
          Speed: Up to 480 Mb/sec
          Manufacturer: Microchip Technology Incorporated
          Location ID: 0x14200000 / 6
          Current Available (mA): 500
          Current Required (mA): 500
          Extra Operating Current (mA): 0

        USB2.0-Serial:

          Product ID: 0x7523
          Vendor ID: 0x1a86
          Version: 2.63
          Speed: Up to 12 Mb/sec
          Location ID: 0x14100000 / 5
          Current Available (mA): 500
          Current Required (mA): 98
          Extra Operating Current (mA): 0

[dl8dtl]

Now, form that as an algorithm in C ;-)

[MCUdude]

Now, form that as an algorithm in C ;-)

Challenge accepted! Here's a proof of concept. Lots of code borrowed from here. Note that it will only recognize "modems", so from my understanding, serial devices only. I've tried to plug in various other USB equipment, but my USB to serial devices are the only ones that show up.

Output:

$ ./usb_vid_pid_test 
Found modem. Port: /dev/cu.usbserial-1410 USB VID: 0x1A86 PID: 0x7523
Found modem. Port: /dev/cu.SLAB_USBtoUART USB VID: 0x10C4 PID: 0xEA60

Build command:

gcc -framework CoreServices -framework IOKit -o usb_vid_pid_test usb_vid_pid_test.c

usb_vid_pid_test.c

#include <stdio.h>
#include <string.h>
#include <unistd.h>
#include <fcntl.h>
#include <sys/ioctl.h>
#include <errno.h>
#include <paths.h>
#include <termios.h>
#include <sysexits.h>
#include <sys/param.h>
#include <sys/select.h>
#include <sys/time.h>
#include <time.h>
#include <CoreFoundation/CoreFoundation.h>
#include <IOKit/IOKitLib.h>
#include <IOKit/serial/IOSerialKeys.h>
#include <IOKit/serial/ioss.h>
#include <IOKit/IOBSD.h>


// Function prototypes
static kern_return_t findModems(io_iterator_t *matchingServices);
static kern_return_t getModemPath(io_iterator_t serialPortIterator, char *bsdPath, CFIndex maxPathSize);

// Returns an iterator across all known modems. Caller is responsible for
// releasing the iterator when iteration is complete.
static kern_return_t findModems(io_iterator_t *matchingServices)
{
  kern_return_t           kernResult;
  CFMutableDictionaryRef  classesToMatch;

  // Serial devices are instances of class IOSerialBSDClient.
  // Create a matching dictionary to find those instances.
  classesToMatch = IOServiceMatching(kIOSerialBSDServiceValue);
  if (classesToMatch == NULL) {
    printf("IOServiceMatching returned a NULL dictionary.\n");
  }
  else {
    // Look for devices that claim to be modems.
    CFDictionarySetValue(classesToMatch,
                          CFSTR(kIOSerialBSDTypeKey),
                          CFSTR(kIOSerialBSDAllTypes));
  }

  // Get an iterator across all matching devices.
  kernResult = IOServiceGetMatchingServices(kIOMasterPortDefault, classesToMatch, matchingServices);
  if (KERN_SUCCESS != kernResult)
    printf("IOServiceGetMatchingServices returned %d\n", kernResult);
  }

  return kernResult;
}

static kern_return_t getModemPath(io_iterator_t serialPortIterator, char *bsdPath, CFIndex maxPathSize)
{
  io_object_t   modemService;
  kern_return_t kernResult = KERN_FAILURE;
  bool          modemFound = false;
  int           vid;
  int           pid;

  // Initialize the returned path
  *bsdPath = '\0';

  // Iterate across all modems found. In this example, we bail after finding the first modem.
  while ((modemService = IOIteratorNext(serialPortIterator))) {
    
    // Variable declaration
    int pid, vid;
    CFTypeRef bsdPathAsCFString, cf_vendor, cf_product;    

    cf_vendor = IORegistryEntrySearchCFProperty(modemService, kIOServicePlane,
                                                CFSTR("idVendor"),
                                                kCFAllocatorDefault,
                                                kIORegistryIterateRecursively
                                                | kIORegistryIterateParents);

    cf_product = IORegistryEntrySearchCFProperty(modemService, kIOServicePlane,
                                                CFSTR("idProduct"),
                                                kCFAllocatorDefault,
                                                kIORegistryIterateRecursively
                                                | kIORegistryIterateParents);
    
    bsdPathAsCFString = IORegistryEntryCreateCFProperty(modemService,
                                                        CFSTR(kIOCalloutDeviceKey),
                                                        kCFAllocatorDefault,
                                                        0);

    // Decode & print port, VID & PID
    if (cf_vendor && cf_product && bsdPathAsCFString &&
        CFNumberGetValue(cf_vendor , kCFNumberIntType, &vid)  &&
        CFNumberGetValue(cf_product, kCFNumberIntType, &pid) &&
        CFStringGetCString(bsdPathAsCFString, bsdPath, maxPathSize, kCFStringEncodingUTF8)) {
        printf("Found modem. Port: %s USB VID: 0x%04X PID: 0x%04X\n", bsdPath, vid, pid);
        modemFound = true;
        kernResult = KERN_SUCCESS;
    }

    // Release CFTypeRef
    if (cf_vendor)  CFRelease(cf_vendor);
    if (cf_product) CFRelease(cf_product);
    if (bsdPathAsCFString) CFRelease(bsdPathAsCFString);
  }

  return kernResult;
}

int main(int argc, const char * argv[])
{
    kern_return_t kernResult;
    io_iterator_t serialPortIterator;
    char          bsdPath[MAXPATHLEN];

    kernResult = findModems(&serialPortIterator);
    if (KERN_SUCCESS != kernResult) {
        printf("No modems were found.\n");
    }

    kernResult = getModemPath(serialPortIterator, bsdPath, sizeof(bsdPath));
    if (KERN_SUCCESS != kernResult) {
        printf("Could not get path for modem.\n");
    }

    IOObjectRelease(serialPortIterator);

    return EX_OK;
}

[MCUdude]

Here's another example where the USB VID and PID is passed as arguments, and the program will tell if the device is present or not:

Output:

.$ /search_usb_vid_pid 0x1a86 0x7523
USB device found. Port: /dev/cu.usbserial-1420 USB VID: 0x1A86 PID: 0x7523

$ ./search_usb_vid_pid 0x10c4 0xea60
USB device found. Port: /dev/cu.SLAB_USBtoUART USB VID: 0x10C4 PID: 0xEA60

# Looking for an FT231X that's not currenty plugged in:
$ ./search_usb_vid_pid 0x0403 0x6015
USB device not found.

# After connecting the FT231X:
$ ./search_usb_vid_pid 0x0403 0x6015
USB device found. Port: /dev/cu.usbserial-DN02SHCS USB VID: 0x0403 PID: 0x6015

Build command:

gcc -framework CoreServices -framework IOKit -o search_usb_vid_pid search_usb_vid_pid.c 

Source:

#include <stdio.h>
#include <string.h>
#include <unistd.h>
#include <fcntl.h>
#include <sys/ioctl.h>
#include <errno.h>
#include <paths.h>
#include <termios.h>
#include <sysexits.h>
#include <sys/param.h>
#include <sys/select.h>
#include <sys/time.h>
#include <time.h>
#include <CoreFoundation/CoreFoundation.h>
#include <IOKit/IOKitLib.h>
#include <IOKit/serial/IOSerialKeys.h>
#include <IOKit/serial/ioss.h>
#include <IOKit/IOBSD.h> 

// Function prototypes
static kern_return_t findModems(io_iterator_t *matchingServices);
static bool getModemPath(io_iterator_t serialPortIterator, char *bsdPath, CFIndex maxPathSize, int usb_vid, int usb_pid);

// Returns an iterator across all known modems. Caller is responsible for
// releasing the iterator when iteration is complete.
static kern_return_t findModems(io_iterator_t *matchingServices)
{
  kern_return_t           kernResult;
  CFMutableDictionaryRef  classesToMatch;

  // Serial devices are instances of class IOSerialBSDClient.
  // Create a matching dictionary to find those instances.
  classesToMatch = IOServiceMatching(kIOSerialBSDServiceValue);
  if (classesToMatch == NULL) {
      printf("IOServiceMatching returned a NULL dictionary.\n");
  }
  else {
      // Look for devices that claim to be modems.
      CFDictionarySetValue(classesToMatch,
                           CFSTR(kIOSerialBSDTypeKey),
                           CFSTR(kIOSerialBSDAllTypes));
  }

  // Get an iterator across all matching devices.
  kernResult = IOServiceGetMatchingServices(kIOMasterPortDefault, classesToMatch, matchingServices);
  if (KERN_SUCCESS != kernResult) {
      printf("IOServiceGetMatchingServices returned %d\n", kernResult);
  }

  return kernResult;
}

static bool getModemPath(io_iterator_t serialPortIterator, char *bsdPath, CFIndex maxPathSize, int usb_vid, int usb_pid)
{
  io_object_t     modemService;
  int             vid;
  int             pid;

  // Initialize the returned path
  *bsdPath = '\0';

  // Iterate across all modems found. In this example, we bail after finding the first modem.
  while ((modemService = IOIteratorNext(serialPortIterator))) {
    
    // Variable declaration
    int pid, vid;
    CFTypeRef bsdPathAsCFString, cf_vendor, cf_product;    

    cf_vendor = IORegistryEntrySearchCFProperty(modemService, kIOServicePlane,
                                                CFSTR("idVendor"),
                                                kCFAllocatorDefault,
                                                kIORegistryIterateRecursively
                                                | kIORegistryIterateParents);

    cf_product = IORegistryEntrySearchCFProperty(modemService, kIOServicePlane,
                                                CFSTR("idProduct"),
                                                kCFAllocatorDefault,
                                                kIORegistryIterateRecursively
                                                | kIORegistryIterateParents);
    
    bsdPathAsCFString = IORegistryEntryCreateCFProperty(modemService,
                                                        CFSTR(kIOCalloutDeviceKey),
                                                        kCFAllocatorDefault,
                                                        0);

    
    // Decode & print port, VID & PID
    if (cf_vendor && cf_product && bsdPathAsCFString &&
        CFNumberGetValue(cf_vendor , kCFNumberIntType, &vid)  &&
        CFNumberGetValue(cf_product, kCFNumberIntType, &pid) &&
        CFStringGetCString(bsdPathAsCFString, bsdPath, maxPathSize, kCFStringEncodingUTF8)){
        
        if(usb_vid == vid && usb_pid == pid) {
          printf("USB device found. Port: %s USB VID: 0x%04X PID: 0x%04X\n", bsdPath, vid, pid);
          return true;
        }
    }

    // Release CFTypeRef
    if (cf_vendor)  CFRelease(cf_vendor);
    if (cf_product) CFRelease(cf_product);
    if (bsdPathAsCFString) CFRelease(bsdPathAsCFString);
  }

  return false;
}

int main(int argc, const char * argv[])
{
    kern_return_t   kernResult;
    io_iterator_t   serialPortIterator;
    char            bsdPath[MAXPATHLEN];

    char * end_ptr;  
    int usb_vid;
    int usb_pid;
    
    char* token;
    
    usb_vid = strtol(argv[1], &end_ptr, 0);
    if (*end_ptr || (end_ptr == argv[1])) {
      printf("Could not parse argument %s\n", argv[1]);
      return 0;
    }
    usb_pid = strtol(argv[2], &end_ptr, 0);
    if (*end_ptr || (end_ptr == argv[1])) {
      printf("Could not parse argument %s\n", argv[2]);
      return 0;
    }
    
    kernResult = findModems(&serialPortIterator);

    if(KERN_SUCCESS != kernResult || !getModemPath(serialPortIterator, bsdPath, sizeof(bsdPath), usb_vid, usb_pid)) {
      printf("USB device not found.\n");
    }

    IOObjectRelease(serialPortIterator);

    return EX_OK;
}

[dl8dtl]

Cool

I wouldn't consider it for v7.0 though, there are enough other things still on the plate, and I'd rather have a release anytime soon than deferring it further by having to implement and debug a completely new feature.

[MCUdude]

I wouldn't consider it for v7.0 though, there are enough other things still on the plate, and I'd rather have a release anytime soon than deferring it further by having to implement and debug a completely new feature.

Very well, let's continue the discussion after 7.0 is released. Is there anything I or we can do to help out preparing for a release?

[dl8dtl]

Is there anything I or we can do to help out preparing for a release?

Well, you already did a great job by walking through the old issues.
As I already wrote you, I'd like to have a look at that WiFi-based programming tool, and try making the timeout tweaks a little less hacky and more generic, but otherwise I'm fine with the current state.

[mcuee]

I remember python-serial has already got this done.
https://pyserial.readthedocs.io/en/latest/tools.html

serial.tools.list_ports.comports(include_links=False)
Platform:	Posix (/dev files)
Platform:	Linux (/dev files, sysfs)
Platform:	OSX (iokit)
Platform:	Windows (setupapi, registry)

Example run log under Windows.

(py39x64venv) PS C:\work\python\pyserial\serial\tools> python -m serial.tools.list_ports -v
COM14
    desc: Arduino Leonardo (COM14)
    hwid: USB VID:PID=2341:8036 SER=5&586B51A&0&1 LOCATION=1-1:x.0
1 ports found

Unfortunately it does not work under FreeBSD, just as what the document says.

(mypy38venv) mcuee@freebsdx64vm:~/build $ python -m serial.tools.list_ports -v
/dev/cuaU0          
    desc: n/a
    hwid: n/a
1 ports found