Skip to content

ModelConvInst

Jump to bottom
Ricardo Ferraz Leal edited this page Apr 1, 2019 · 11 revisions

Note: this page describes the process of converting models from SasView 3.x to sasmodels. For writing new models, see the user manual ​http://www.sasview.org/docs/user/sasgui/perspectives/fitting/plugin.html

INTRODUCTION

This wiki page was originally written by our MOST experienced developers, but has subsequently been edited by our LEAST experienced developer who felt some instructions could have been clearer, and learnt one or two things that were missing altogether! But they succeeded in converting a model that passed testing, so there is no reason why you should not be able to do the same.

In the new sasmodels package there are three ways of writing models:

The documentation for the sasmodels package can be found at ​http://www.sasview.org/sasmodels/

Before following the conversion instructions below, it is worth reading the documentation at ​http://www.sasview.org/sasmodels/api/generate.html#module-sasmodels.generate which describes the structure of a model.

CREATE NEW MODEL FILES .

If you have not already done so, download the sasmodels package from the repo at ​https://github.com/SasView/sasmodels .

In the sasmodels/models directory, copy the appropriate files (using the examples above as templates) to mymodel.py (and mymodel.c, etc) as required, where "mymodel" is the NEW name for the model you are converting/creating. The OLD model files can be found in sasview/src/sas/models/ directory. Please follow these new naming rules:

  • No capitalization and thus no CamelCase. If necessary use underscore to separate (i.e. barbell not BarBell or broad_peak not BroadPeak)
  • Remove "model" from the name (i.e. barbell not BarBellModel)

EDIT NEW MODEL FILES

The model interface definition is in the .py file. This file contains:

  • MODEL DOCUMENTATION (also see Documentation Checks )
  • doc string . The .py starts with an r (for raw) and three sets of quotes to start the doc string and ends with a second set of 3 quotes. This is where the FULL documentation for the model goes (to be picked up by Sphinx). Paste the model documentation from the appropriate section of model_functions.rst (found at ​src/sas/models/media/model_functions.rst ) here. DO NOT use the old html documentation. For example, for the BarBellModel, search for BarBellModel in the rst file then copy everything after the *2.x.x. ModelNameModel**. Then edit the result as follows:
    • the *2.x.x.x Model** section with its brief description of the model becomes the model title; it feels odd that the documentation string does not start with name and brief description but it is automatically added by the model loader.
    • replace *2.x.x.x. Definition** with just Definition followed by 10 dashes on the line below.
    • wherever possible, try to replace images of math functions with Latex equivalents. You can use the live demo Mathjax page ( ​http://www.mathjax.org/ ) to make sure the equation looks as expected. Also a lot of the Latex code can be taken from (or edited) from the PDF document created by Paul Kienzle: ​http://sasview.org/attachment/wiki/SasModels%20Work%20Package/Equations.docx.pdf .
    • remove the table of parameters which will get autogenerated.
    • copy non-math figures from the sasview/src/sas/models/media/img subdirectory to the sasmodels/sasmodels/models/img subdirectory and rename them to something sensible. For example, image183.jpg became power_law_1d.jpg.
    • figure captions should be indented relative to the ..image:: tag. This allows us to, for example, change the font of the caption through CSS.
    • remove the figure number since this will be generated automatically.
    • make sure all references to model parameters are appropriately renamed, and that the relationship between equation variables and model parameters are obvious.
  • name = "mymodel". This is the name of the model that is shown to the user. Use the name of the file as described above in 1. There will be less confusion if the model name matches the file name, but they could be different.
  • title = "short description". This is a new string to be used for a tooltip, for example. It can probably be pulled directly from the model_functions.rst file.
  • description = """doc string""". Cut and paste the self.description string from the (old) SasView model header in ​src/sas/models/include . This gives a brief description of the equation and the parameters without the need to read the entire model documentation. Make sure the parameter names match the current model definition.
  • PARAMETERS

    • parameters = [["name", "units", default, [min,max], "type", "tooltip"],...]. This is where the parameters get defined. The syntax should be obvious from the default. Copy the parameters from the model header file ​src/sas/models/include .

    • the order of the parameters in the definition will be the order of the parameters in the user interface and the order of the parameters in Iq(), Iqxy() and form_volume().

    • VERY IMPORTANT: We are trying to make the model parameters more consistent between models. So sld_solvent, for example, should have exactly the same name in every model.

      • NOTE: There is no need to specify 'scale' or 'background', these are implicit to all models.

      • to see the list of parameters currently used across all models, use

        python -m sasmodels.list_pars

    • the parameter default values are used to auto-generate a plot of the model function in the documentation .

    • lower and upper limits can be any number, or -inf or inf.

      • add limits where required the (old) sasview models don't usually define them.
      • the limits will show up as the default limits for the fit making it easy, for example, to force the radius to always be greater than zero.

    • also note that as of Code Camp IV, sld's should be specified in units of 10 ^-6^ Ang ^-2^ and not simply Ang ^-2^ . This is because sasmodels now parses the parameter name looking for "sld" to decide if a parameter is an sld and automatically adjusts the volume normalisation factor accordingly!!!

    • "type" can take 3 values at this time: "", "volume", or "orientation".

      • "volume" parameters are passed to Iq(), Iqxy(), and form_volume(), and have polydispersity loops generated automatically.
      • "orientation" parameters are only passed to Iqxy().

    • "units" is the string that is displayed to the user in the user interface. This is not latex markup. If you want the manual to show fancy markup, do the following:

      • check RST_UNITS variable near the top of ​sasmodels/generate.py for units with special markup
      • if you don't see the units you need there, make a new entry in the RST_UNITS table
      • use the macros defined in ​doc/rst_prolog , or add your own if needed
      • if the markup is trivial, or if the it is really a one-off, then you can put the restructured text commands into RST_UNITS directly (I'm not sure if $latex$ works in this context)
      • if there is enough demand, we can set units to "display string|markup string" and have the model loader/documentation generator select the correct version for whomever is asking
  • THE COMPUTATION

    • Pure Python

      • def Iq (this section does not exist for c models. It is replaced by the list of c files to call)
        • Iq.vectorized = True or False. This is used to tag the definition as accepting a Q vector, or having to compute each Q individually.
          • use functions from numpy rather than math
          • don't use if statements. Instead do tricks like "a = x*(q>0) + y*(q⇐0)" which sets a to x if q is positive or y if q is zero or negative
          • return np.NaN if the parameters are not valid (e.g., cap_radius < radius for barbell)
      • def Iqxy. If this model does not have an oriented form just set it equal to the square root of the sum of Iq(x) squared and Iq(y) squared.
      • ​src/sas/models contains the pure python models, mixed together with the python + C models.

    • Python + C

      • model Iq, Iqxy are defined in ​src/sas/models/c_extension/c_models

      • supporting scattering calculations are defined in ​src/sas/models/c_extension/libigor

      • special functions are defined in ​src/sas/models/c_extension/cephes . With model definitions you are limited to the simple math functions defined in ​opencl

      • common functions should be moved into their own file, such as lib/J1.c for the bessel function J1.

      • in place of the above methods point to the C source files needed. This will be a list of any lib files needed, followed by the model.c file, where model is the same name as given to the python model.py. For example:

            source = ["lib/J1.c", "lib/gauss76.c", "my_model.c"]

      • the c file must contain the form_volume, Iq, Iqxy methods. These and any other functions defined (e.g. the _cyl() helper function) must be defined as doubles in the first lines of the file.

      • return NAN if the parameters are not valid (e.g., cap_radius < radius for barbell)

    • NOTE: for certain models, such as those that can be multiplied by a structure factor, the ER attribute should be set to the Equivalent Radius (of a sphere).

    • NOTE: for certain models, namely core-shell type models, the VR attribute returns the volume ratio for the core-shell - no they don't - but is irrelevant as for 4.0 we are going to remove the feature that causes volume and radius parameters to disappear.

    • volume normalization is now provided by the caller, so individual forms no longer need to be normalized by volume. This generally means you can remove the volume calculation from the Iq/Iqxy functions and move it to the form_volume function instead.

  • demo is a dictionary containing the value of each parameter as given in the rst documentation. Make sure to enter the appropriate values from that documentation. This will then be used to generate the example curve in said documentation.
    • include polydispersity parameters (e.g., radius_pd=0.2, radius_pd_n=9) so that you can compare the polydispersity calculations against those from sasview; use -mono on the compare.sh command line to see the direct calculation without polydispersity. For speed you can use small numbers for _pd_n. If you don't want polydispersity on a parameter by default in the demo, leave a reasonable value for _pd, but set _pd_n to 0.
  • TRANSITION CHECKS
    • oldname is the name for this model used in SasView. Make sure to put the correct original name. This is required for the transition to allow compatibility and to test that the models are equivalent.
    • oldpars is the name of the parameters as used in SasView. For each parameter give the name of the parameter as used in the original SasView code version of this model. Again this is required for the transition.
  • UNIT TEST(S)

    • THESE ARE VERY IMPORTANT. Include at least one test for each model and PLEASE make sure that the answer value is correct (i.e. not a random number).

    • Existing tests can be found by hunting through the existing test files in the sasview code : ​test/sasmodels/test

    • test is a list of lists. Each list is one test and contains, in order:

      • a dictionary of parameter values. This can be {} using the default parameters, or filled with some parameters that will be different from the default ({'radius':10.0, 'sld':4}). Unlisted parameters will be given the default values.
      • the input q value or tuple of qx, qy values.
      • the output Iq or Iqxy expected of the model for the parameters and q value given.
      • inputs and outputs can also be lists if you have several values to test for the same model parameters.

    • for ER and VR, give the inputs as 'ER' and 'VR' respectively.

    • to summarize:

          tests = [
        [ {parameters}, q, I(q)],
        [ {parameters}, [q], [I(q)] ],
        [ {parameters}, [q1, q2, ...], [I(q1), I(q2), ...]],
        [ {parameters}, (qx, qy), I(qx, Iqy)],
        [ {parameters}, [(qx1, qy1), (qx2, qy2), ...],
                        [I(qx1,qy1), I(qx2,qy2), ...]],
        [ {parameters}, 'ER', ER(pars) ],
        [ {parameters}, 'VR', VR(pars) ],
    ]

where

    parameters = {'k1': v1, 'k2': v2, ...}

TEST YOUR NEW/NEWLY-CONVERTED MODEL

Test your new model by running compare.py to verify that the converted model is giving the same results as it did in SasView prior to conversion. In order to do this, you need either a version of SasView installed in your python path, or a locally-built SasView in a checked-out directory at the same level as the checked-out sasmodels repository; i.e. you have the file structure:

  drive:\some_folder

contains

  \bumps
  \periodictable
  \sasview
  \sasmodels

If using a locally-built SasView, you need to add drive:\some_folder\sasview to your PYTHONPATH environment variable, or add drive:\some_folder to your User PATH environment variable and edit \sasmodels\compare.sh so that the global SASVIEW points to some_folder :

  SASVIEW=(../some_folder/build/lib.*)

Also remember, if using a locally-built SasView, that if you have done a pull from the repo you will need to re-build before continuing!

The first thing to test is that you are getting the same answer as SasView for the 1D version of the model. These tests use the demo= values in your model file. To run the tests use:

./compare.sh modelname -1d

This will result in some comparison metrics between the OpenCL implementation (if installed - it should revert to using ctypes if no OpenCL is installed) by way of a plot of the two calculations and a comparison plot; ie, three plots. If you do not see three plots, see the note on OpenCL issues at the bottom of the page.

If the model has 2D orientational calculation, then you should additionally test with:

./compare.sh modelname -2d

Brief help for the comparison script can be obtained by just running ./compare.sh

If you are on Windows, just cd into the /sasmodel folder and omit the './'

If you have the error

ValueError: Model does not contain parameter ...

then a parameter in sasmodels is not a parameter in sasview. Scale and background are the usual culprits, though polydispersity parameters can also be a problem, or where a converted model has parameters not present in the original SasView model. You will need to update the namelist= section of sasmodels.convert.revert_pars and/or sasmodels.convert.constrain_new_to_old to handle the differences.

Run a number of random parameter sets through your model to make sure that your default values aren't special:

./multi_compare.sh modelname 200 1d100 0 double > modelname-1d.csv

and for 2d calculations:

./multi_compare.sh modelname 20 2d32 0 double > modelname-2d.csv

Particularly for sasview with polydispersity, the 2D multi_compare may be much too slow, and you will want to use the following instead:

./multi_compare.sh modelname 20 2d32 mono double > modelname-2d.csv

The output is a comma separated value text which you can view with a spreadsheet program. Broken values (those which differ by more than 1e-14) are shown along with a random seed. You can reproduce these parameters in compare.sh using -random=SEED.

For the random models,

  • sld will be in(-0.5,10.5),
  • angles (theta, phi, psi) will be in (-180,180),
  • angular dispersion will be in (0,45),
  • polydispersity will be in (0,1)
  • other values will be in (0, 2*v) where v is the value of the parameter in demo.

Dispersion parameters n, sigma and type will be unchanged from demo so that run times are predictable.

Now run the unit tests that you have added. These tests use the test= values in your model file . To run the unit tests use:

python -m sasmodels.model_test modelname

CLEAN LINT

Run the lint check with:

python -m pylint --rcfile=extra/pylint.rc sasmodels/models/modelname.py

We are not aiming for zero lint just yet, only keeping it to a minimum. For now, don't worry too much about invalid-name. If you really want a variable name Rg for example because Rg is the right name for the model parameter then ignore the lint errors. Also, ignore missing-docstring for standard model functions Iq, Iqxy, etc.

We will have delinting sessions at the code camp, where we can decide on standards for model files, parameter names, etc.

For now, you can tell pylint to ignore things. For example, to align you parameters in blocks:

# pylint: disable=bad-whitespace,line-too-long
#   ["name",                  "units", default, [lower, upper], "type", "description"],
parameters = [
    ["contrast_factor",       "barns",    10.0,  [-inf, inf], "", "Contrast factor of the polymer"],
    ["bjerrum_length",        "Ang",       7.1,  [0, inf],    "", "Bjerrum length"],
    ["virial_param",          "1/Ang^2",  12.0,  [-inf, inf], "", "Virial parameter"],
    ["monomer_length",        "Ang",      10.0,  [0, inf],    "", "Monomer length"],
    ["salt_concentration",    "mol/L",     0.0,  [-inf, inf], "", "Concentration of monovalent salt"],
    ["ionization_degree",     "",          0.05, [0, inf],    "", "Degree of ionization"],
    ["polymer_concentration", "mol/L",     0.7,  [0, inf],    "", "Polymer molar concentration"],
    ]
# pylint: enable=bad-whitespace,line-too-long

Don't put in too many pylint statements, though, since they make the code ugly.

CHECK THE DOCS

To build the docs you will need:

Make sure C:\Program Files (x86)\!GnuWin32\bin and C:\Python27 are on your path.

Build the docs by changing path to /sasmodels/doc and typing "make html".

NOTE: The build also copies the images to be included in the documentation from drive:\some_folder\sasmodels\sasmodels\models\img to drive:\some_folder\sasmodels\doc\model\img. The script should create the destination folder, but if it cannot the docs build will fail. If this happens, just create the destination folder manually and then try the make again.

Navigate to path/to/sasmodels/doc/_build/html in your file browser and open index.html.

NOTE: You will need to use firefox, chrome or safari since internet explorer doesn't seem to support mathjax from a local file.

FINALLY

Once compare and the unit test(s) pass properly and everything is done, commit your new model to the repo and then edit the models table at http://trac.sasview.org/wiki/ListofModels to indicate that the conversion is complete.

Also complete the documentation check table Checks .

OPENCL ISSUES If the compare.sh modelname -1d test only generates one plot instead of three, or the compare.sh modelname -2d test complains that there is "No module named pyopencl", then you may not have OpenCL installed or set-up correctly. There are two parts to satisfying this:

  1. Make sure you have an OpenCL driver installed that is compatible with your graphics card:
  1. Make sure you have pyopencl installed:

    • Windows binaries are available at ​http://www.lfd.uci.edu/~gohlke/pythonlibs/#pyopencl

    • Note: the link above distributes packages as 'wheels' (.whl).

    • To install a wheel (this example being for a 32-bit Python 2.7 installation with no additional packages), cd to the download folder and then:

      python -m pip install pyopencl-2015.1-cp27-none-win32.whl
Clone this wiki locally