Quick Question Regarding OpSim3_61DBObject Queries

[JPGlaser]

Hello Everyone (@danielsf & @SimonKrughoff),

I have been working with the ObSim, CatSim, and PhoSims lately in an effort to generate test data for our AGN survey group here at Drexel. As such, I am currently investigating the ObSim integration in master-gce0eaba1be version of sims_catUtils.

Essentially, I'd like to do a double query of the ObSim 3.61 database, first filtering out airmass values that aren't in a specified range and then filtering out pointings that don't use a specific set of filters (say the r & g filters). Is this currently possible? From looking into the python classes, it seems that obsMD.executeConstrainedQuery only accepts a singular constraint ...

In addition, can someone clear up the exact nature of the output of these queries? It seems that it basically returns a trimmed array of matching ObsHistID's and so my script (attached below) would generate a catalog for the first matching pointing. Is this a correct understanding of the data structure?

Thanks in advance!

~ Joe G.

"""
This script creates an instance catalog of Galaxies and AGN for use in PhoSim,
subject to certain pre-defined constraints.

"""

from __future__ import with_statement
from lsst.sims.catalogs.measures.instance import InstanceCatalog
from lsst.sims.catalogs.generation.db import ObservationMetaData, CatalogDBObject
from lsst.sims.catUtils.baseCatalogModels import OpSim3_61DBObject
from lsst.sims.catUtils.exampleCatalogDefinitions.phoSimCatalogExamples import \
        PhoSimCatalogPoint, PhoSimCatalogSersic2D, PhoSimCatalogZPoint
from lsst.sims.catUtils.baseCatalogModels import *

# Define File Name
fileName = "InstCat_GalAGN_r2deg.txt"

# Define Desired Observing Parameters
ra = 1.5
dec = -1.5
tol = 2.0
radiusDeg = 1.0

# Loads in the OpSim 3.61 database
obsMD = OpSim3_61DBObject()

# Prepares to find the Spatial Bounds of the night sky.
skyBounds = SpatialBounds.getSpatialBounds('box', ra, dec, tol)

# Sets the SQL query constraint, in this case a specific airmass.
queryConstraint = "airmass = 1.1"

# Generates the constrained query on the Opsim 3.61 database.
query = obsMD.executeConstrainedQuery(skyBounds, constraint=queryConstraint)

# Converts the query into observation meta data for use in a catalog.
obsMetaData = obsMD.getObservationMetaData(query['Opsim_obshistid'][0],
 radiusDeg, makeCircBounds=True)

# Load in Galaxy Bulge objects, create their 2D sersic profiles, and append them
# to our output file subject to our observational meta-data.
gals = CatalogDBObject.from_objid('galaxyBulge')
galaxy_phoSim = PhoSimCatalogSersic2D(gals, obs_metadata=obs_metadata)
galaxy_phoSim.write_catalog("phoSim_example.txt",write_mode='a',write_header=True,chunk_size=20000)

# Load in Galaxy Disk objects, create their 2D sersic profiles, and append them
# to our output file subject to our observational meta-data.
gals = CatalogDBObject.from_objid('galaxyDisk')
galaxy_phoSim = PhoSimCatalogSersic2D(gals, obs_metadata=obs_metadata)
galaxy_phoSim.write_catalog("phoSim_example.txt",write_mode='a',write_header=False,chunk_size=20000)

# Load in Galaxy AGN objects, create their extragalatic point source profiles, and
# append them to our output file subject to our observational meta-data.
gals = CatalogDBObject.from_objid('galaxyAgn')
galaxy_phoSim = PhoSimCatalogZPoint(gals, obs_metadata=obs_metadata)
galaxy_phoSim.write_catalog("phoSim_example.txt",write_mode='a',write_header=False,chunk_size=20000)

[danielsf]

Hi Joe,

Is it absolutely necessary that you use OpSim 3.61? I ask because there are more up-to-date OpSim runs which can be found here

https://confluence.lsstcorp.org/display/SIM/OpSim+Datasets+for+Cadence+Workshop+LSST2015

and more up-to-date python code to perform exactly the queries you want. If you are willing to abandon OpSim 3.61, the thing you will want to do is install the most up-to-date lsst sims stack (so: do a fresh 'eups distrib install lsst_sims -t sims') and then look at the class ObservationMetaDataGenerator found in

sims_catUtils/python/lsst/sims/catUtils/utils/ObservationMetaDataGenerator.py

You can also look at the tutorial iPython notebook

CatSimTutorial_SimulationsAHM_1503.ipynb

which is in the CatSim directory of the git hub repository

uwssg/LSST-Tutorials.git

If you would rather keep going with OpSim 3.61 (or if what I said doesn't make much sense), go ahead and email me again and I will see what I can do to help you.

-- Scott Daniel

[darkojev]

Hi Joe,

singular constraint can have many parts (sql syntax) - this is
an example from alertsim...

opsim_constraint="(night=10 and rawseeing<0.6) and filter like 'i' "

The best thing is to pass constraint as an argument to the code (much
less typing)

Hope this helps.

Cheers
D.

On Mon, 11 May 2015, Joseph Glaser wrote:

Hello everyone,

I have been working with the ObSim, CatSim, and PhoSims lately in an effort to generate test data for our AGN survey group
here at Drexel. As such, I am currently investigating the ObSim integration in master-gce0eaba1be version of sims_catUtils.

Essentially, I'd like to do a double query of the ObSim 3.61 database, first filtering out airmass values that aren't in a
specified range and then filtering out pointings that don't use a specific set of filters (say the r & g filters). Is this
currently possible? From looking into the python classes, it seems that obsMD.executeConstrainedQuery only accepts a
singular constraint ...

In addition, can someone clear up the exact nature of the output of these queries? It seems that it basically returns a
trimmed array of matching ObsHistID's and so my script (attached below) would generate a catalog for the first matching
pointing. Is this a correct understanding of the data structure?

Thanks in advance!

~ Joe G.

"""
This script creates an instance catalog of Galaxies and AGN for use in PhoSim,
subject to certain pre-defined constraints.

"""

from future import with_statement
from lsst.sims.catalogs.measures.instance import InstanceCatalog
from lsst.sims.catalogs.generation.db import ObservationMetaData, CatalogDBObject
from lsst.sims.catUtils.baseCatalogModels import OpSim3_61DBObject
from lsst.sims.catUtils.exampleCatalogDefinitions.phoSimCatalogExamples import
PhoSimCatalogPoint, PhoSimCatalogSersic2D, PhoSimCatalogZPoint
from lsst.sims.catUtils.baseCatalogModels import *

Define File Name

fileName = "InstCat_GalAGN_r2deg.txt"

Define Desired Observing Parameters

ra = 1.5
dec = -1.5
tol = 2.0
radiusDeg = 1.0

Loads in the OpSim 3.61 database

obsMD = OpSim3_61DBObject()

Prepares to find the Spatial Bounds of the night sky.

skyBounds = SpatialBounds.getSpatialBounds('box', ra, dec, tol)

Sets the SQL query constraint, in this case a specific airmass.

queryConstraint = "airmass = 1.1"

Generates the constrained query on the Opsim 3.61 database.

query = obsMD.executeConstrainedQuery(skyBounds, constraint=queryConstraint)

Converts the query into observation meta data for use in a catalog.

obsMetaData = obsMD.getObservationMetaData(query['Opsim_obshistid'][0],
radiusDeg, makeCircBounds=True)

Load in Galaxy Bulge objects, create their 2D sersic profiles, and append them

to our output file subject to our observational meta-data.

gals = CatalogDBObject.from_objid('galaxyBulge')
galaxy_phoSim = PhoSimCatalogSersic2D(gals, obs_metadata=obs_metadata)
galaxy_phoSim.write_catalog("phoSim_example.txt",write_mode='a',write_header=True,chunk_size=20000)

Load in Galaxy Disk objects, create their 2D sersic profiles, and append them

to our output file subject to our observational meta-data.

gals = CatalogDBObject.from_objid('galaxyDisk')
galaxy_phoSim = PhoSimCatalogSersic2D(gals, obs_metadata=obs_metadata)
galaxy_phoSim.write_catalog("phoSim_example.txt",write_mode='a',write_header=False,chunk_size=20000)

Load in Galaxy AGN objects, create their extragalatic point source profiles, and

append them to our output file subject to our observational meta-data.

gals = CatalogDBObject.from_objid('galaxyAgn')
galaxy_phoSim = PhoSimCatalogZPoint(gals, obs_metadata=obs_metadata)
galaxy_phoSim.write_catalog("phoSim_example.txt",write_mode='a',write_header=False,chunk_size=20000)

—
Reply to this email directly or view it on GitHub.[AF4sd-CBVYnS3q1Ymv03sD4U1xhlv-mPks5oISYggaJpZM4EXK0G.gif]

[JPGlaser]

@danielsf,
Thanks very much for informing me about the ObservationMetaDataGenerator.py. My current installation of LSST and lsst_sims seems to be the most recent. However, the Confluence documentation that I was using to generate instance catalogs for PhoSim used OpSim3_61DBObject, so I had been just keeping in line with that. So there isn't much of a reason to continue using OpSim 3.61 if the newer ones can generate the meta-data PhoSim needs.

Currently, our cluster has been white-listed for access to Fatboy's database, so I guess my question now resides on how to I can get a connection established for these newer OpSim runs to run queries on them?

Also, I noticed that GalSim is a suggested alternative to PhoSim for image simulation. Do you know what the specific advantage it has over PhoSim?

Thanks very much for the help in advance.

~ Joe

PS: You can email me at [email protected] if you feel more comfortable discussing connection matters over that instead.

[danielsf]

To get the new OpSim databases, you just need to download them from the link I included in the message above (they're about 2G each). This will give you sqlite database files of the form enigma_1189.db

You can then connect to them by instantiating the ObservationMetaDataGeneratory with something like

myGen = ObservationMetaDataGenerator(address='sqlite:///path_to_your_database_file.db')

[danielsf]

Regarding GalSim vs. PhoSim: eventually GalSim will be faster (if less rigorous in modeling things like chip noise and PSF) than PhoSim. At the moment that is not the case. This is due to a combination of issues dealing with our implementation of GalSim in the stack. It is currently a work in progress.

[JPGlaser]

@danielsf,
Thanks for the info on how to access the SQL databases. I will look into getting that into my code later today. Also, its good to know I've been correct in using PhoSim for the moment and haven't been missing out on GalSim being a better option. However, its nice to see that it may eventually surpass PhoSim for quick generation as current runs can be quite time consuming even on our cluster.

Now, just as a bit of clarification, if I were to query a database subject to visits of the same field of view and airmass, would the generated catalogs take into account the variability of say, AGNs imported from CatalogDBObject.from_objid('galaxyAgn')? I ask as it seems that all AGNs utilize the same SED, just redshifted based on distance. Again, my end goal would be to run the processed PhoSim images from pointings with similar observation parameters through a source detector to generate source catalogs with measurement errors and magnitude variability.

As an additional note, is there any documentation on the information provided in the astronomical object classes?

~ Joe

[danielsf]

Currently, all AGNs in our model have the same SED. AGNs are stored in our database with a series of parameters that implement variability so that each AGN varies differently (to see the actual implementation, go to sims_photUtils/python/lsst/sims/photUtils/Variability.py and look at the method applyAgn). Unfortunately, we do not yet have the code in place to propagate that variability into PhoSim.

What you could do is generate a catalog which include quasar variability (see here

https://confluence.lsstcorp.org/display/SIM/Variability+models+in+the+Catalogs+Simulation+Framework

for an example of how to include variability in a catalog. You will want to use the mixin VariabilityGalaxies defined in sims_photUtils/python/lsst/sims/photUtils/Variability.py. See also here

https://confluence.lsstcorp.org/display/SIM/Getters+provided+by+Catalog+Simulations+mixins
). You could then take the variable g-band magnitudes from your catalogs and use them as the normalizing magnitudes in your PhoSim instance catalogs (since PhoSim expects normalization at 500nm, which falls in the g band). We will be updating the Variability model in the next few months, so hopefully this process will get easier soon.

"As an additional note, is there any documentation on the information provided in the astronomical object classes?"

I'm not sure what you mean by "astronomical object classes"? Can you give me an example of one of the classes to which you are referring?

[JPGlaser]

@danielsf,

Thanks for clearing up some confusion with this in regards to variability. However, the thing I am still confused about is that if I called CatalogDBObject.from_objid('galaxyAgn'), it pulls from the Fatboy database, which you say has variability built into each AGN. If that is the case, why do I still have to use applyAGN on the database to get the lsst_g band magnitudes out at a specific MJD?

In addition, looking through the Variability.py file, I don't find the mixin, VariabilityGalaxies. I only find get_galaxy_variability(self), which I am a bit unsure on how I would go about actually using to extract magNormVarOut (which I am assuming has the magnitude I want to place in norm_mag column for the PhoSim catalog).

As for my question on astronomical object classes, I was referring to the classes with names like 'galaxyAgn', 'galaxyBulge', 'galaxyDisk', etc.

~ Joe

[danielsf]

A mixin is a python class that does not have an __init__() method. It exists solely so that other classes can inherit from it and use its methods. get_galaxy_variability is a method defined in the class VariabilityGalaxies. Do a search for "class VaribilityGalaxies". That should turn it up.

I think it might be best for you to clone the repository

https://github.com/uwssg/LSST-Tutorials.git

and work through the iPython notebook CatSim/CatSimTutorial_SimulationsAHM_1503.ipynb

That notebook is probably the most up-to-date description we have of how the Catalog Simulation stack works.

Regarding your question on variability: fatboy stores all of the parameters describing a given AGN's variability. However, because we want the option to 'observe' each AGN at multiple times, fatboy does not actually implement that variability. It just store the magnitude of the AGN at some reference time t_0. In order to see how the magnitude varies with time, you actually have to run the AGN through the Catalog Simulation software.

'galaxyBulge', 'galaxyDisk' etc. are all python classes written to interface with tables in the fatboy databases. Documentation concerning them can be found at these pages

https://confluence.lsstcorp.org/display/SIM/Database+Schema
https://confluence.lsstcorp.org/display/SIM/Framework+Overview+--+Catalog+Simulations
https://confluence.lsstcorp.org/display/SIM/Database+Object+classes+in+the+Catalogs+Simulation+Framework

I'm sorry this is so poorly documented at the moment. Our attention is poorly divided between extending the code and documenting what is already there (and we are developing at a rapid enough pace that we keep getting ahead of whatever documentation already exists). Thank you for your patience.

[JPGlaser]

Hello Daniel,
Thanks for the input regarding all of this. The tutorial was very helpful in filling in a few of the gaps that were apparent. I also noticed their was a tutorial called generateAgnCatalog_150309.ipynb. I believe this details the same steps you were discussing in an earlier comment. My current focus is to output the catalogs for set amounts of observations (which I now know how to query) and then read them in to create a light-curve for the AGNs. I'll update here when I have that script running.

One last question: What model are you using for your AGN variability. It looks like just a damped random walk model. Is this correct?

~ Joe

[danielsf]

Hi Joe

Yes, my understanding is that the AGN variability model is a damped random walk (the method was written before I joined the group so I'm only 85% certain). If you want to see the source code, look for the method applyAgn in

sims_photUtils/python/lsst/sims/photUtils/Variability.py

Let me know if that is not illuminating,

-- Scott (Daniel is my surname; one day I will meet Daniel Scott and we will buy each other beers)

[JPGlaser]

Hello Scott,
Thank you for that confirmation with error bars. XD
Like I said, I will update here when I get things up and running on my side. Thanks for all the help thus far!

Warm Regards,
Joseph Glaser

PS: I probably should have asked that before I made an assumption. My apologies, Scott. I'll have to owe you one as well someday given how much help you've been on this.

[SimonKrughoff]

Just to follow up on this. The reference for the AGN model is:
http://adsabs.harvard.edu/abs/2012ApJ...753..106M

We got distributions of model parameters per band from Ivezic through private communication.

[JPGlaser]

Hey Scott (@danielsf),
So I am working through the iPython Notebooks provided earlier in this topic and I must say, they are very good at explaining what exactly is going on with CatSim. Its filled in a few holes in my knowledge just far and cleared up some questions. However, I noticed that in the AGN Variability notebook, running the following command generates an error:

variableObsMetadata = ObservationMetaData(unrefractedRA=25.0, unrefractedDec=30.0,
                                          boundType='circle', boundLength=0.05,
                                          mjd=57086)

variableAgn = variableAgnCatalog(galaxyDB, obs_metadata=variableObsMetadata)
variableAgn.write_catalog('variable_agn.txt')
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-15-7cca2531f2c5> in <module>()
      3                                           mjd=57086)
      4 
----> 5 variableAgn = variableAgnCatalog(galaxyDB, obs_metadata=variableObsMetadata)
      6 variableAgn.write_catalog('variable_agn.txt')

/Users/jglaser/lsst/DarwinX86/sims_catalogs_measures/10.1+4/python/lsst/sims/catalogs/measures/instance/InstanceCatalog.pyc in __init__(self, db_obj, obs_metadata, column_outputs, constraint, specFileMap)
    274         self._column_origins_switch = True
    275 
--> 276         self._check_requirements()
    277 
    278     def _all_columns(self):

/Users/jglaser/lsst/DarwinX86/sims_catalogs_measures/10.1+4/python/lsst/sims/catalogs/measures/instance/InstanceCatalog.pyc in _check_requirements(self)
    381             if len(nodefault) > 0:
    382                 raise ValueError("Required columns missing from database: "
--> 383                                  "({0})".format(', '.join(nodefault)))
    384 
    385         if self.verbose:

ValueError: Required columns missing from database: (rAgn_var, zAgn_var, gAgn_var, uAgn_var, yAgn_var, iAgn_var)

Looking at the output for the list of columns in GalaxyTileObj(), it doesn't have any of these in the database, however shouldn't these be created by the Variability mixin? Any thoughts?

~ Joe

[danielsf]

Hi Joe,

Sorry about that. That was a case of the demo notebooks lagging behind the development of the actual code. I just fixed it. If you do a git pull on the LSST-Tutorials repository, you should get something that works.

-- Scott

[danielsf]

to be more explicit:

you are right, those columns should have been made by the Variability mixin. However, in the last few months, we split Variability into

VariabilityStars
VariabilityGalaxies

so the InstanceCatalog daughter class needed to be updated to inherit from VariabilityGalaxies, instead of just Variability

[JPGlaser]

Hey Scott ( @danielsf ),
Thanks for that, I was hoping that I wasn't just missing something fundamental. While we are on the topic of column outputs, I am trying to get VariabilityGalaxies to spit out the sigma_[]Agn_var coloumn for each filter by just adding them to the column_outputs variable. However, upon asking for the catalog I get the following error:

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-4-323445607f52> in <module>()
      1 variableAgn = variableAgnCatalogCheat(galaxyDB, obs_metadata=obsMDresults[0])
----> 2 variableAgn.write_catalog('variable_agn_bigcat_obs_metadata.txt')

/Users/jglaser/lsst/DarwinX86/sims_catalogs_measures/10.1+4/python/lsst/sims/catalogs/measures/instance/InstanceCatalog.pyc in write_catalog(self, filename, chunk_size, write_header, write_mode)
    437                           if col in self.transformations.keys() else
    438                           self.column_by_name(col)
--> 439                           for col in self.iter_column_names()]
    440 
    441             # Create the template with the first chunk

/Users/jglaser/lsst/DarwinX86/sims_catalogs_measures/10.1+4/python/lsst/sims/catalogs/measures/instance/InstanceCatalog.pyc in column_by_name(self, column_name, *args, **kwargs)
    337                 self._column_origins[column_name] = self._get_class_that_defined_method(function)
    338 
--> 339             compound_column = function(*args, **kwargs)
    340             return compound_column[column_name]
    341         elif isinstance(self._current_chunk, _MimicRecordArray) or column_name in self._current_chunk.dtype.names:

/Users/jglaser/lsst/DarwinX86/sims_catalogs_measures/10.1+4/python/lsst/sims/catalogs/measures/instance/decorators.pyc in new_f(self, *args, **kwargs)
     22             result = self._column_cache[colname]
     23         else:
---> 24             result = f(self, *args, **kwargs)
     25             self._column_cache[colname] = result
     26         return result

/Users/jglaser/lsst/DarwinX86/sims_catalogs_measures/10.1+4/python/lsst/sims/catalogs/measures/instance/decorators.pyc in new_f(self, *args, **kwargs)
     47         @wraps(f)
     48         def new_f(self, *args, **kwargs):
---> 49             results = f(self, *args, **kwargs)
     50             return OrderedDict(zip(colnames, results))
     51         new_f._compound_column = True

/Users/jglaser/lsst/DarwinX86/sims_photUtils/1.0.0-81-g0c80041+14/python/lsst/sims/photUtils/Variability.pyc in get_galaxy_variability_uncertainties(self)
    582 
    583         output = self.calculateMagnitudeUncertainty(magnitudes, obs_metadata=self.obs_metadata,
--> 584                                                     sig2sys=self.sig2sys)
    585 
    586         magnitudes = numpy.array([self.column_by_name('uAgn_var'),

/Users/jglaser/lsst/DarwinX86/sims_photUtils/1.0.0-81-g0c80041+14/python/lsst/sims/photUtils/Photometry.pyc in calculateMagnitudeUncertainty(self, magnitudes, obs_metadata, sig2sys)
    437 
    438         snr = self.calculateFluxSignalToNoise(numpy.power(10.0, -0.4*magnitudes),
--> 439                                               obs_metadata=obs_metadata, sig2sys=sig2sys)
    440 
    441         #see www.ucolick.org/~bolte/AY257/s_n.pdf section 3.1

/Users/jglaser/lsst/DarwinX86/sims_photUtils/1.0.0-81-g0c80041+14/python/lsst/sims/photUtils/Photometry.pyc in calculateFluxSignalToNoise(self, fluxes, obs_metadata, sig2sys)
    417 
    418         snr, gamma = calcSNR_gamma(fluxes, self.bandpassDict.values(), self._m5List, gamma=self._gammaList,
--> 419                                    sig2sys=sig2sys)
    420 
    421         return snr

/Users/jglaser/lsst/DarwinX86/sims_photUtils/1.0.0-81-g0c80041+14/python/lsst/sims/photUtils/noiseUtilities.pyc in calcSNR_gamma(fluxes, bandpasses, m5, gamma, sig2sys, expTime, nexp, gain, effarea)
    381             sigmaSq = (0.04-gg)*fluxRatio+gg*fluxRatio*fluxRatio
    382 
--> 383         noise.append(numpy.sqrt(sigmaSq))
    384 
    385     return 1.0/numpy.array(noise), gamma

AttributeError: 'float' object has no attribute 'sqrt'

Ideas?

~ Joe

[danielsf]

Hi Joe,

This one is going to take a little more time to diagnose. It is a "real" bug (as opposed to the last one, which is something I should have been aware of but was blithely ignoring). I will keep you posted.

Sorry for the inconvenience, and thanks for the heads-up

-- Scott

[JPGlaser]

Hey Scott,

Thanks for confirming that. I look forward to getting the fix. Not sure why its saying that you can't apply a sqrt to a float, quite weird indeed.

On the bright side, everything else seems to be working fantastically. Here's an example output plot using the green-filters variable magnitude.

~ Joe

[JPGlaser]

Also a small additional note:

1. I have been noticing this error popping up for when I create the catalog:

/Users/jglaser/lsst/DarwinX86/sims_photUtils/1.0.0-81-g0c80041+14/python/lsst/sims/photUtils/Variability.py:531: FutureWarning: comparison to `None` will result in an elementwise object comparison in the future.
  if deltaMagNorm != None and (not numpy.isnan(magNormAgn[i])):

Not sure what exactly this means.

2. I have also been getting this error a lot upon querying the database:

/Users/jglaser/lsst/DarwinX86/anaconda/master-g68783b1848/lib/python2.7/site-packages/sqlalchemy/dialects/mssql/base.py:1294: SAWarning: Did not recognize type 'geography' of column 'geopoint'
  return fn(*arg, **kw)

[danielsf]

Hi Joe,

None of those warnings should be cause for concern. The first one is just telling us that we are not taking the full advantage of what numpy is capable of and may end up slowing ourselves down. That should probably be fixed, but is not critical.

The second is a weird disconnect between python and MSSQL. I don't see that one going away any time soon, but it is totally innocuous.

Regarding variability: There were some weird type casts going on in python. Things that should have been numpy arrays of floats were actually being returned as numpy arrays of objects. However, we have done a major refactor of the variability code, and that seems to have fixed it. Unfortunately, it was a MAJOR refactor of the variability code, so a lot of what the AGN iPython notebook has been telling you will soon be wrong. I will update it when we release the new code (early next week if not this weekend).

Thanks for all of your help exercising the code. Sorry to hit you with such a major change (though I do think that it will make the code easier to use in general).

-- Scott

[SimonKrughoff]

@danielsf Regarding the SAWarnings we see consistently, can't we set up a warning filter to get rid of those? I think that's done elsewhere in the code. The user should be able to turn them back on with a verbosity flag. What do you think?

[JPGlaser]

Hey Scott (@danielsf),

This is Joe. Thanks for confirming that those warnings aren't anything to be too concerned about. I figured that it wasn't that big of a deal, but you never know. Anyway, regarding the coming major overhaul of the variability code, let me know when it goes live. I look forward to working with the new code.

In news regarding the old code, I have polished up this iPy Notebook which expands upon the variable AGN tutorial you linked to earlier in this thread. You can find it here: https://github.com/JPGlaser/LSST_CatGen/blob/master/Tests/genVariableAGNCat.ipynb

Good news is everything looks like it is outputting how it should, so I think I have finally gripped the material (course just in time for a new update to come out XD).

~ Joe

[JPGlaser]

Hey All,

Looks like I'm having new problems with the OpSim generation. I decided I would try importing a different OpSim database as the included one seems to be a very trimmed down version of the planned survey. It seems that when I try to use any database other than opsimblitz1_1133_sqlite.db, I don't return any observations. I have included the code below and am open to any suggestions. Thanks in advance.

~ Joe

import eups
import os
from lsst.sims.catUtils.utils import ObservationMetaDataGenerator
from lsst.sims.catalogs.generation.db import ObservationMetaData
#help(ObservationMetaDataGenerator)
#print obsMDresults[0].__dict__

opsimdb = os.path.join(eups.productDir('sims_data'),'OpSimData','enigma_1189_sqlite.db')
gen = ObservationMetaDataGenerator(driver='sqlite', database=opsimdb)
SimObData = gen.getObservationMetaData(boundType='circle', boundLength=0.01, fieldRA=(40.0,60.0), fieldDec=(-40.0,-60.0))

for o in SimObData:
    print o.unrefractedRA, o.unrefractedDec, o.phoSimMetaData['airmass'][0], o.mjd

[danielsf]

Hi Joe

you have reversed the min and max of fieldDec (-40 > -60)

ObservationMetaDataGenerator.getObservationMetaData does not have an automatic check to make sure your bounds make sense. It probably should.

-- Scott

PS I have updated the Confluence documentation and the LSST-Tutorials example scripts, though you seem to have kept up with the new API pretty well on your own

[JPGlaser]

Hey @danielsf,

Thanks for getting back to me. Sorry for the slow reply, but I was away all of last week and most of the week before.

Can't believe that didn't dawn on me and correcting it fixed all of my issues when using enigma_1189_sqlite.db. And yeah, I'm finding it easier to work with now and have been brushing up on your edits when I run into errors so thanks for updating them.

I do have a question regarding errors in magnitude measurements. Has the old bug that we found been fixed yet for the variability mixin? It would be really nice to add those error bars into the light curves and source catalogs I am generating.

Current version of my script can be found here: https://github.com/JPGlaser/LSST_CatGen/blob/master/Tests/genLightCurves.ipynb

~ Joe

[danielsf]

Hi Joe,

I was unable to recreate the bug after the variability mixin was re-factored. As I recall, the problem was that python was doing some odd type casting (i.e. not all of the magnitudes were being cast as floats). I believe that has been fixed. You should be able to resume what you were doing. Please let me know if that is not the case.

FYI: I'm traveling until July 15th and have limited email access, so this conversation might be a little slower that usual.

Cheers,

Scott

[JPGlaser]

Hey @danielsf,

Yeah that was the problem and it is good to hear that the bug seems to go away after the reworking of the variability code.

Just to double check, if I wanted the magnitude of the AGN through the u-filter and its observational error, I would have the catalog return 'uAgn' and 'sigma_uAgn'?

~ Joe

[danielsf]

Yes. sigma_uAgn will be the uncertainty (in magnitudes) of the AGN component of a galaxy in the u band.

[JPGlaser]

Thanks @danielsf, that seems to be working fine. Question regarding the output of the catalog generator: I would only like the catalog to output the data for the magnitude of the band observed, instead of all the bands. Is there a way of doing this within the code, or should I just write if statements for the column_outputs variable?

~ Joe

[danielsf]

Look at the method setupPhotometryCatalog defined in sims_catUtils/python/lsst/sims/catUtils/utils/CatalogSetupFunctions.py

Does that do what you want?

[JPGlaser]

More or less. I'll have to make a few changes, but that does help me figure out the syntax I'd have to use. Another quick question: say I want to make a hard cut of objects that have calculated mags higher than 24, is that possible to do when I define my instance catalog daughter class?

~ Joe

[danielsf]

I don't think that is possible. Most of the ways to impose constraints on the contents of catalogs happen at the SQL level, and variable magnitudes are not stored on the database.

One thing you could do (this will not save you any time, but I do not think it will slow you down) is define a column "filtered_magnitude". The getter for this could look something like

def get_filtered_magnitude(self):
    raw_magnitude = self.column_by_name('uAgn')
    some code that converts all magntiudes in raw_magnitude gerater than 24 to None
    return raw_magnitude

and then have cannot_be_null = ['filtered_magnitude']

[JPGlaser]

That could definitely work well as a backup. However, I know the database has magNormAgn as a mapped column, so shouldn't I be able to write a SQL constraint that filters out any of these that have magNormAgn>24.0? For example, using CHECK magNormAgn > 24.0?

~ Joe

[danielsf]

magNormAgn is defined against a rather artificial bandpass (a delta function at 500nm). I don't think it will correspond to something that is actually at 24th magnitude.

[JPGlaser]

Hm, alright ... thanks for that clarification. What about making a cut with lsst_g, or is this a similarly defined variable before the PhotometryGalaxies mixin is added? If it is the latter, is there a variable in the database that might be used to make a similar cut?

The primary purpose is that due to our limiting mag for a single visit being around 24 mag (27 mag for multi-visit stacks, which doesn't help for variability), it would probably be a good idea to just get rid of those objects at the SQL level for speed since they aren't practical and are likely not able to be modeled in the same way as brighter AGNs.

~ Joe

[danielsf]

you can try comparing agn_sfru to uAgn and see how they compare (the contents of the database pre-date my involvement with the project, so I'm not 100% sure what everything is). I can tell you that, even if they are supposed to be the same thing, the magnitudes won't totally line up. All magnitudes stored in the database were calculated against hold versions of the filter throughputs. The filter specs have gone through a few iterations in the past five years and the database contents have not been kept up-to-date in that respect.

[JPGlaser]

Yeah I understand; kinda wish the database had a descriptor file somewhere to give us a hint at what some of it is. I can try running a cut with CHECK agn_sfu < 24.0 added to the constraint as see what happens. Any specific reason why you are suggesting that entry (besides it being in the u-band)?

~ Joe

[danielsf]

I meant agn_sf[u,g,r,i,z,y], depending on what band you were interested in

[JPGlaser]

Okay, thanks for that. I am going to try to run it with the following constraint to test things out:
'(sedname_agn IS NOT NULL) AND (agn_sfu < 24.0)'

Now, regarding the problem of outputting the column of the observed bandpass's magnitude I origionally had this included in the main section of my class:

# Only Append the Column of AGN Magnitudes in the Observed Bandpass
    if obs_metadata.bandpass is not None:
        if hasattr(obs_metadata.bandpass, '__iter__'):
            for Filter in obs_metadata.bandpass:
                column_outputs.append(Filter+'Agn')
                column_outputs.append('sigma_'+Filter+'Agn')
        else:
            Filter = obs_metadata.bandpass
            column_outputs.append(Filter+'Agn')
            column_outputs.append('sigma_'+Filter+'Agn')

It seems to be throwing up errors regarding the fact that obs_metadata is not defined. I assume that this is because the class doesn't have the definition syntax of classname(self, InstanceCatalog, ...)? Or am I missing something here. I could define a new getter function that would grab the column, though I am fuzzy on the details. I believe it would be something like:

@compound(ObsAgn, sigma_ObsAgn):
def get_ObsAgn(self):
     #Some Function that applies variability to the magnitude and finds the error given a bandpass
     return ObsAgn

Is this more or less correct?

~ Joe

[danielsf]

I think the problem above is that obs_metadata should be self.obs_metdata.

[JPGlaser]

So I tried that, but it returned the same error except that self was not defined, which doesn't make much sense ...

~ Joe

[danielsf]

Can you email me the python script that is causing the error? My email address is scottvalscott at gmail

[JPGlaser]

Email sent. For archival purposes, people can find the script in question (in whatever version it is) at this location: https://github.com/JPGlaser/LSST_CatGen/blob/master/Tests/genLSST_AGNData.ipynb

Will also post the solution we figure this out.

~ Joe

[JPGlaser]

So this is what I ended up making after trying out a few things that Scott suggested. Basically, my InstanceCatalog daughter class looks like this:

from lsst.sims.catalogs.measures.instance import InstanceCatalog, compound
from lsst.sims.photUtils import PhotometryGalaxies, VariabilityGalaxies

class FastVarAgnCat(InstanceCatalog, PhotometryGalaxies, VariabilityGalaxies):

    cannot_be_null = ['uAgn'] #Locating only the AGNs in the FoV.

    column_outputs = ['AGNID', 'redshift', 'raJ2000', 'decJ2000', 'ObsAgn', 'sigma_ObsAgn']

    transformations = {'raJ2000':numpy.degrees, 'decJ2000':numpy.degrees}

    #Only Append the Column of AGN Magnitudes in the Observed Bandpass
    @compound('ObsAgn', 'sigma_ObsAgn')
    def get_PhotometryCols(self):
        mag = None
        sigma = None
        if self.obs_metadata.bandpass is not None:
            if not hasattr(self.obs_metadata.bandpass, '__iter__'):
                Filter = self.obs_metadata.bandpass
                mag = self.column_by_name(Filter+'Agn')
                sigma = self.column_by_name('sigma_'+Filter+'Agn')
        return np.array([mag, sigma])

    @compound('sedFilenameBulge', 'sedFilenameDisk')
    def get_QuickSED(self):
        ra = self.column_by_name('raJ2000') #Finding how many objects are in the column.
        names = []
        for r in ra:
            names.append('None') #Tricking the catalog into thinking these galaxies don't have bulges or disks.
        return np.array([names, names])

def ObsHeader(IC, file_handle):
    ObsHeaderTransformations = {'Unrefracted_RA':numpy.degrees, 'Unrefracted_Dec':numpy.degrees,
                               'Opsim_moonra':numpy.degrees, 'Opsim_moondec':numpy.degrees,
                               'Opsim_rotskypos':numpy.degrees, 'Opsim_rottelpos':numpy.degrees,
                               'Opsim_sunalt':numpy.degrees, 'Opsim_moonalt':numpy.degrees,
                               'Opsim_dist2moon':numpy.degrees, 'Opsim_altitude':numpy.degrees,
                               'Opsim_azimuth':numpy.degrees
                               }
    md = IC.obs_metadata.phoSimMetaData
    for k in md:
        if k in ObsHeaderTransformations.keys():
            file_handle.write(str(k)+" "+str(ObsHeaderTransformations[k](md[k][0]))+"\n")
        else:
            file_handle.write(str(k)+" "+str(md[k][0])+"\n")

The getter get_PhotometryCols is called when I ask for ObsAGN and sigma_ObsAgn. We do a simple check to make sure that this is an LSST observation (AKA: only one filter is being used at a time) and then pulls the correct _Agn and sigma__Agn cols for the observation. Since I have a custom header writer for later use I don't mind not having the col name be perfectly descriptive (the filter used is in the header anyway).

Overall its a decent fix that shaves 2 seconds off of the time to generate 5589 AGNs for one observation (taking into account SQL constraints), with a final time of 32.123646 seconds.

~ Joe