m2snano-llm

Application for the analysis of nanoparticles using image processing with an LLM layer to provide insights in the analysis.

• Github repository: https://github.com/icn2/m2snano-llm/

---

Getting started with your project

First, clone the github repository:

git clone [email protected]:icn2-ai/m2snano-llm.git

We manage the project dependencies with Poetry. Make sure you have it installed. You can find the installation instructions here. It also uses exiftool to handle image metadata. You can find the installation instructions here or try to install it in linux with:

sudo apt install libimage-exiftool-perl

The LLM runs locally with ollama. You can find the installation instructions here. To install:

curl -fsSL https://ollama.com/install.sh | sh

Concretely, we use the llama3.2 model, which you can download once ollama is installed with:

ollama pull llama3.2

Install the environment and the pre-commit hooks with:

make install

The last thing you have to do is to download the per-trained models for particle detection and segmentation. Since that part of the project is based on ArdaGen's project Automated S/TEM-Nanoparticle-Analysis-YOLOv8-SAM, you can find the link to download the weights in their README file. They should be placed in the project in the following paths:

• m2snano_llm/weights/detection/best12x.pt
• m2snano_llm/weights/segmentation/mobile_sam.pt
• m2snano_llm/weights/segmentation/sam_vit_b_01ec64.pth

You are now ready to analyze images with the help of an LLM! To run the application, use:

source .venv/bin/activate  # Activate the virtual environment
python -m m2snano_llm.main

---

How to use the app

Selecting the root directory

When you run the application, it will first ask you to select the root directory where the images to analyze are stored. To do so, go to the File menu on the top left and select Choose Root Directory.

Once that is done, a list of all subdirectories containing images will be shown in the first dropdown menu. Select one, and a list of images in that directory will be shown in the second dropdown menu. Select one image to analyze.

Analyzing the image

The image will be shown in the main panel. To its left, we can see the metadata extracted from it, which allows us to know the real size of the nanoparticles. To the right, we have our tools to analyze the nanoparticles themselves.

Important: the automatic extraction of the metadata is tailored for images from specific microscopes. Because of that reason, the length of the scale bar in pixels is extracted from this specific types of images, from the image directly. It might not work with other images. If the Real scale of the bar (nm) field is empty or incorrect, please fill it manually.

To analyze them, we first need to detect them. To do so, click on the Find particles button. This will run the detection model and show the detected particles as bounding boxes in the image.

Here, you can do several things:

• You can delete a bounding box by clicking on it, in case you want to remove a false positive.
• You can add a bounding box by Shift + Click and dragging the mouse to create a new bounding box, in case you want to add a false negative.
• You can filter them by size, using the Min diameter and Max diameter fields and the ones that might be really close to the margins of the image with the Margin field.

Once you are satisfied with the detected particles, click on the Segment particles button. This will run the segmentation model and show the segmented particles in the image.

Finally, to store the results of the analysis, click on the Calculate and save button.

Results

For each subfolder, a particle_data.xlsx file will be created. Each sheet of the file will contain the results of the analysis for each image in that folder. In our case, the results contain information about:

• Equivalent circular diameter (ECD): an approximation of the diameter of each particle.
• Roughness: a measure of how rough the surface of the particle is.
• Shell thickness: the thickness of the shell of the particle (nm), if applicable.
• Dip to belly contrast: a measure to determine wether the particle is hollow or not. The higher the value, the clearer its intensity profile becomes in the center, the more likely it is to be hollow.

One extra sheet will be created ("Mean"), with information about the mean values and standard deviations for parameter for each image.

This excel will be stored in the following manner:

• Root directory: /path/to/root/directory/raw/
• Current folder: /path/to/root/directory/raw/image/folder_1/ (here's where there are images stored)
• Excel path: /path/to/root/directory/processed/image/folder_1/particle_data.xlsx (Note that they are not stored in the raw data directory, but in a directory beside it called processed)

LLM integration

The application also integrates an LLM layer to provide insights about the analysis. Once the particle_data.xlsx file is created, by analyzing one of the images in the subfolder, it is possible to ask the LLM for a summary of the results. It will analyze the information in the "Mean" sheet of the excel and provide insights on it.

To do so, click on the Summarize Data of the current folder button, and a summary will be shown below it:

---

Particle detection and segmentation was based on ArdaGen's project: Automated S/TEM-Nanoparticle-Analysis-YOLOv8-SAM.

Repository initiated with fpgmaas/cookiecutter-poetry.