Neo4j 2.0

[atomos]

Hi,

I'm just starting out using Neo4j and I'd like to use 2.0 (I have 2.0.1 community installed). I see that neo4django was only tested against neo4j 1.8.2-1.9.4, but have people gotten it working with 2.x? I installed the gremlin plugin but can't create or query through neo4django.

create:

In [8]: NeoProfile.objects.create(profile_id=1234)
[INFO] requests.packages.urllib3.connectionpool#214: Resetting dropped connection: localhost
---------------------------------------------------------------------------
StatusException                           Traceback (most recent call last)
/Users/atomos/workspace/Project-Vitamin/lib/python2.7/site-packages/django/core/management/commands/shell.pyc in <module>()
----> 1 NeoProfile.objects.create(profile_id=1234)

/Users/atomos/workspace/Project-Vitamin/src/neo4django/neo4django/db/models/manager.pyc in create(self, **kwargs)
     41
     42     def create(self, **kwargs):
---> 43         return self.get_query_set().create(**kwargs)
     44
     45     def filter(self, *args, **kwargs):

/Users/atomos/workspace/Project-Vitamin/src/neo4django/neo4django/db/models/query.pyc in create(self, **kwargs)
   1295         if 'id' in kwargs or 'pk' in kwargs:
   1296             raise FieldError("Neo4j doesn't allow node ids to be assigned.")
-> 1297         return super(NodeQuerySet, self).create(**kwargs)
   1298
   1299     #TODO would be awesome if this were transactional

/Users/atomos/workspace/Project-Vitamin/lib/python2.7/site-packages/django/db/models/query.pyc in create(self, **kwargs)
    375         obj = self.model(**kwargs)
    376         self._for_write = True
--> 377         obj.save(force_insert=True, using=self.db)
    378         return obj
    379

/Users/atomos/workspace/Project-Vitamin/src/neo4django/neo4django/db/models/base.pyc in save(self, using, **kwargs)
    315
    316     def save(self, using=DEFAULT_DB_ALIAS, **kwargs):
--> 317         return super(NodeModel, self).save(using=using, **kwargs)
    318
    319     @alters_data

/Users/atomos/workspace/Project-Vitamin/lib/python2.7/site-packages/django/db/models/base.pyc in save(self, force_insert, force_update, using)
    461         if force_insert and force_update:
    462             raise ValueError("Cannot force both insert and updating in model saving.")
--> 463         self.save_base(using=using, force_insert=force_insert, force_update=force_update)
    464
    465     save.alters_data = True

/Users/atomos/workspace/Project-Vitamin/src/neo4django/neo4django/db/models/base.pyc in save_base(self, raw, cls, origin, force_insert, force_update, using, *args, **kwargs)
    331
    332         is_new = self.id is None
--> 333         self._save_neo4j_node(using)
    334         self._save_properties(self, self.__node, is_new)
    335         self._save_neo4j_relationships(self, self.__node)

/Users/atomos/workspace/Project-Vitamin/src/neo4django/neo4django/db/models/base.pyc in _save_neo4j_node(self, using)

/Users/atomos/workspace/Project-Vitamin/src/neo4django/neo4django/db/models/base.pyc in trans_method(func, *args, **kw)
     95                 #TODO this is where generalized transaction support will go,
     96                 #when it's ready in neo4jrestclient
---> 97                 ret = func(*args, **kw)
     98                 #tx.commit()
     99                 return ret

/Users/atomos/workspace/Project-Vitamin/src/neo4django/neo4django/db/models/base.pyc in _save_neo4j_node(self, using)
    359             self.__node = conn.gremlin_tx(script, types=type_hier_props,
    360                                           indexName=self.index_name(),
--> 361                                           typesToIndex=type_names_to_index)
    362         return self.__node
    363

/Users/atomos/workspace/Project-Vitamin/src/neo4django/neo4django/neo4jclient.pyc in gremlin_tx(self, script, **params)
    177         will be wrapped in a transaction.
    178         """
--> 179         return self.gremlin(script, tx=True, **params)
    180
    181     def cypher(self, query, **params):

/Users/atomos/workspace/Project-Vitamin/src/neo4django/neo4django/neo4jclient.pyc in gremlin(self, script, tx, raw, **params)
    166             try:
    167                 return send_script(include_unloaded_libraries(lib_script),
--> 168                                    params)
    169             except LibraryCouldNotLoad:
    170                 if i == 0:

/Users/atomos/workspace/Project-Vitamin/src/neo4django/neo4django/neo4jclient.pyc in send_script(s, params)
    151             if raw:
    152                 execute_kwargs['returns'] = RETURNS_RAW
--> 153             script_rv = ext.execute_script(s, params=params, **execute_kwargs)
    154             if isinstance(script_rv, basestring):
    155                 if LIBRARY_ERROR_REGEX.match(script_rv):

/Users/atomos/workspace/Project-Vitamin/src/neo4j-rest-client/neo4jrestclient/client.py in __call__(self, *args, **kwargs)
   2313             except (ValueError, AttributeError, KeyError, TypeError):
   2314                 pass
-> 2315             raise StatusException(response.status_code, msg)
   2316
   2317     def __repr__(self):

StatusException: Code [400]: Bad Request. Bad request syntax or unsupported method.
Invalid data sent: javax.script.ScriptException: groovy.lang.MissingMethodException: No signature of method: groovy.lang.MissingMethodException.setMaxBufferSize() is applicable for argument types: () values: []

query:

In [9]: NeoProfile.objects.filter(profile_id=1234)
Out[9]: ---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
/Users/atomos/workspace/Project-Vitamin/lib/python2.7/site-packages/django/core/management/commands/shell.pyc in <module>()
----> 1 NeoProfile.objects.filter(profile_id=1234)

/Users/atomos/workspace/Project-Vitamin/lib/python2.7/site-packages/IPython/core/displayhook.pyc in __call__(self, result)
    236             self.start_displayhook()
    237             self.write_output_prompt()
--> 238             format_dict = self.compute_format_data(result)
    239             self.write_format_data(format_dict)
    240             self.update_user_ns(result)

/Users/atomos/workspace/Project-Vitamin/lib/python2.7/site-packages/IPython/core/displayhook.pyc in compute_format_data(self, result)
    148             MIME type representation of the object.
    149         """
--> 150         return self.shell.display_formatter.format(result)
    151
    152     def write_format_data(self, format_dict):

/Users/atomos/workspace/Project-Vitamin/lib/python2.7/site-packages/IPython/core/formatters.pyc in format(self, obj, include, exclude)
    124                     continue
    125             try:
--> 126                 data = formatter(obj)
    127             except:
    128                 # FIXME: log the exception

/Users/atomos/workspace/Project-Vitamin/lib/python2.7/site-packages/IPython/core/formatters.pyc in __call__(self, obj)
    445                 type_pprinters=self.type_printers,
    446                 deferred_pprinters=self.deferred_printers)
--> 447             printer.pretty(obj)
    448             printer.flush()
    449             return stream.getvalue()

/Users/atomos/workspace/Project-Vitamin/lib/python2.7/site-packages/IPython/lib/pretty.pyc in pretty(self, obj)
    358                             if callable(meth):
    359                                 return meth(obj, self, cycle)
--> 360             return _default_pprint(obj, self, cycle)
    361         finally:
    362             self.end_group()

/Users/atomos/workspace/Project-Vitamin/lib/python2.7/site-packages/IPython/lib/pretty.pyc in _default_pprint(obj, p, cycle)
    478     if getattr(klass, '__repr__', None) not in _baseclass_reprs:
    479         # A user-provided repr.
--> 480         p.text(repr(obj))
    481         return
    482     p.begin_group(1, '<')

/Users/atomos/workspace/Project-Vitamin/lib/python2.7/site-packages/django/db/models/query.pyc in __repr__(self)
     70
     71     def __repr__(self):
---> 72         data = list(self[:REPR_OUTPUT_SIZE + 1])
     73         if len(data) > REPR_OUTPUT_SIZE:
     74             data[-1] = "...(remaining elements truncated)..."

/Users/atomos/workspace/Project-Vitamin/lib/python2.7/site-packages/django/db/models/query.pyc in __len__(self)
     85                 self._result_cache = list(self.iterator())
     86         elif self._iter:
---> 87             self._result_cache.extend(self._iter)
     88         if self._prefetch_related_lookups and not self._prefetch_done:
     89             self._prefetch_related_objects()

/Users/atomos/workspace/Project-Vitamin/src/neo4django/neo4django/db/models/query.pyc in iterator(self)
   1274         using = self.db
   1275         if not self.query.can_filter():
-> 1276             for model in self.query.execute(using):
   1277                 yield model
   1278         else:

/Users/atomos/workspace/Project-Vitamin/src/neo4django/neo4django/db/models/query.pyc in execute(self, using)
   1161         conn = connections[using]
   1162
-> 1163         groovy, params = self.as_groovy(using)
   1164
   1165         raw_result_set = conn.gremlin_tx(groovy, **params) if groovy is not None else []

/Users/atomos/workspace/Project-Vitamin/src/neo4django/neo4django/db/models/query.pyc in as_groovy(self, using)
    925         # add the typeNodeId param, either for type verification or initial
    926         # type tree traversal
--> 927         cypher_params['typeNodeId'] = self.model._type_node(using).id
    928
    929         type_restriction_expr = """

/Users/atomos/workspace/Project-Vitamin/src/neo4django/neo4django/db/models/base.pyc in _type_node(cls, using)
    411             return cls.__type_node_memoized(using)
    412         else:
--> 413             return cls.__type_node_classmethod(using)
    414
    415     @classmethod

/Users/atomos/workspace/Project-Vitamin/src/neo4django/neo4django/db/models/base.pyc in __type_node(cls, using)
    394             script_rv = conn.gremlin_tx(script, types=type_hier_props)
    395         except Exception, e:
--> 396             raise RuntimeError(error_message, e)
    397         if not hasattr(script_rv, 'properties'):
    398             raise RuntimeError(error_message + '\n\n%s' % script_rv)

RuntimeError: ('The type node for class NeoProfile could not be created in the database.', StatusException())

My model is incredibly complex:

class NeoProfile(neomodels.NodeModel):
    profile_id = neomodels.IntegerProperty(indexed=True)

[mlent]

I'm also trying to upgrade to Neo4j 2. Now that the Gremlin plugin is installed, I get the same error that you end up with (from the API):

{
error_message: "('The type node for class Word could not be created in the database.', StatusException())",
traceback: "Traceback (most recent call last): File "/opt/phaidra/env/src/django-tastypie/tastypie/resources.py", line 195, in wrapper response = callback(request, *args, **kwargs) File "/opt/phaidra/env/src/django-tastypie/tastypie/resources.py", line 426, in dispatch_list return self.dispatch('list', request, **kwargs) File "/opt/phaidra/env/src/django-tastypie/tastypie/resources.py", line 458, in dispatch response = method(request, **kwargs) File "/opt/phaidra/api/api.py", line 421, in get_list if len(words) < 1: File "/opt/phaidra/env/local/lib/python2.7/site-packages/django/db/models/query.py", line 106, in __len__ self._result_cache = list(self.iterator()) File "/opt/phaidra/env/src/neo4django/neo4django/db/models/query.py", line 1283, in iterator piece = list(clone.execute(using)) File "/opt/phaidra/env/src/neo4django/neo4django/db/models/query.py", line 1162, in execute groovy, params = self.as_groovy(using) File "/opt/phaidra/env/src/neo4django/neo4django/db/models/query.py", line 926, in as_groovy cypher_params['typeNodeId'] = self.model._type_node(using).id File "/opt/phaidra/env/src/neo4django/neo4django/db/models/base.py", line 411, in _type_node return cls.__type_node_classmethod(using) File "/opt/phaidra/env/src/neo4django/neo4django/db/models/base.py", line 394, in __type_node raise RuntimeError(error_message, e) RuntimeError: ('The type node for class Word could not be created in the database.', StatusException()) "
}

Namely:
RuntimeError: ('The type node for class Word could not be created in the database.', StatusException()

Have you made any progress on this?

[atomos]

Sorry, I gave up on neo4django. I don't remember seeing that error.

[techdragon]

I'll be working on getting some progress here, ill post any notes as i go.
I think it wont be too difficult since the library were using to talk to neo4j has been working on 2.0 for 6 months now. We may just need to use the ParserVersion feature to inform neo4j that were going to query it with 1.8 or 1.9 query semantics. (at least i hope its that easy lol)

[techdragon]

Ive got some progress on 2.0 and 2.1 in my fork here: techdragon@fac398b

Ive been chipping away at the errors that caused the issues. After updating the neo4jrestclient library, adding detection code to catch when the version of neo4j is 2.0 or higher, and then creating a secondary groovy script to be called when we we are on 2.0 or higher in order to be able to try and develop something compatible with the old way as well as 2.0... I have been able to steadily comment my way through the groovy script and pinpoint a few of the problems with gettingneo4django working on Neo4j 2.x with versions of the Gremlin Plugin for 2.x ... All of which are in the groovy script code.

@mhluongo You're the only person thats ever touched the groovy code from the look of it, so while I'm sure ill eventually work it out, this may be much more efficient if you look at the issue. It appears that in order to be compatible with neo4j 2.x we need to remove all the transaction handling code and make some changes to how it queries and creates nodes. Like the changes and comments in the following links:
dbunskoek/bulbs@02c861b
https://groups.google.com/forum/#!topic/gremlin-users/OUkpukj8YYY
https://github.com/tinkerpop/blueprints/blob/master/doc/The-Major-Differences-Between-Blueprints-1.x-and-2.x.textile

I've tried to work out some of this groovy code stuff myself but it seems from your blog post you may be doing some advanced groovy code magic ( http://mattluongo.com/post/client-side-gremlin-libraries-in-neo4j ) so while I'll keep working on this, I'm pretty sure you may be able to either point me in the right direction so i can finish fixing my fork for a PR, or post some code i can use that will help.

edit:
It also seems to be related to the "type node" behaviour neo4django appears to either implicitly depend on or rely on as part of its use of gremlin

edit2:
I think you may have already logged an issue for this... hard to tell without a description ;-) #170

edit3:
I've tracked it back down to the assumption of neo4j having a node(0) "reference node" and the lack of the reference node is causing all kinds of trouble. Since the reference node assumption is a major roadblock on my ability to further tinker with the gremlin pathway, incase this 1.x reference node assumption proves to be a major hurdle to overcome, I'm going to take a shot at building a pure cypher implementation on top of the new GSOC 2014 django meta API in trunk (as mentioned here #252) and try to keep its schema as close to the one provided by neo4django as i can.

[mhluongo]

@techdragon thanks for your work so far! I've done some client work where I used a similar approach (Groovy + Gremlin + Cypher) against Neo4j 2.0+. I'll check if that's applicable.

Heh, yeah, that's the issue. It's actually pretty easy to just create a "reference node", and always use that by pulling from an index, instead of relying on the old node(0) business. I think that'd be the quickest fix. That said, actually using Neo4j 2.0 types instead of the neo4django-constructed type tree would be sweet.

I'd caution you on going all-Cypher- Cypher is great, but there are some very basic things it doesn't yet support that neo4django has had to work around with Gremlin / client-side index craziness. It'd be a shame to no longer have the complex filtering done against the indices first.

[techdragon]

@mhluongo I'm keeping an eye on this quite closely as my current app progresses so hopefully some of that client work code proves fruitful.

From what I've been building so far, the type/label system is quite powerful and does make creating node types a lot more strait forward.

I'm holding back on starting work on the all cypher neo4django option at the moment for a couple of reasons, one of which is the still evolving state for the new django meta api improvements, the other of which is a general uncertainty as to wether to drop down to direct communication to/from neo4j or not, so my current project is being built using neo4j 2.1 format raw cypher constructed using the helpful https://github.com/bruth/cypher tool and (at the moment) py2neo. I'm taking full advantage of the ability to just "remodel the schema as i go" and letting things evolve for a while before trying to build any kind of tool with it as a primary use case.