Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

WIP: Storm db manager #481

Closed
wants to merge 91 commits into from
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
91 commits
Select commit Hold shift + click to select a range
0c3b595
Added the StormDBManager class
lfdversluis Jul 1, 2016
2394f63
Added StormDBManager tests
lfdversluis Jul 1, 2016
1b5e564
Refactored community to handle the deferreds returned by StormDBManag…
lfdversluis Jul 1, 2016
ab9dfdf
Refactored community to handle the deferreds returned by StormDBManag…
lfdversluis Jul 1, 2016
c464dbb
Refactored conversion to handle deferreds returned by StormDBManager …
lfdversluis Jul 1, 2016
c4197f7
Refactored database to make use of the stormdb and become asynchronous
lfdversluis Jul 1, 2016
ed3142e
Refactored dispersydatabase to use the StormDBManager and become asyn…
lfdversluis Jul 1, 2016
549e931
Refactored message to handle deferreds returned by StormDBManager and…
lfdversluis Jul 1, 2016
a81fe9e
Refactored endpoint to handle deferreds returned by StormDBManager an…
lfdversluis Jul 1, 2016
62f29de
Refactored statistics to handle deferreds returned by StormDBManager …
lfdversluis Jul 1, 2016
6048009
Refactored decorator to handle async functions passed to it
lfdversluis Jul 1, 2016
719dfcb
Disabled decorators because otherwise these functions will start retu…
lfdversluis Jul 1, 2016
399ecdf
Refactored the discovery community to handle the deferreds returned b…
lfdversluis Jul 1, 2016
47c5fe9
Refactored dispersy to handle the deferreds returned by StormDBManage…
lfdversluis Jul 1, 2016
77e5972
Refactored the debug community to handle the deferreds returned by St…
lfdversluis Jul 1, 2016
a8fe1bc
Refactored the tracker community the handle the deferreds returned by…
lfdversluis Jul 1, 2016
587e069
Refactored the tracker plugin to handle the now asynchronous functions
lfdversluis Jul 1, 2016
299ebb7
Refactored tool to handle the now asynchronous functions
lfdversluis Jul 1, 2016
71f3908
refactored node to handle the asynchronous functions in dispersy
lfdversluis Jul 1, 2016
c4b0d9c
Refactored the dispersy testclass to handle the async functions and a…
lfdversluis Jul 1, 2016
5176418
Refactored test_batch to handle the now asynchronous behavior of Disp…
lfdversluis Jul 1, 2016
edeacd7
Refactored test_bootstrap to handle the now asynchronous behavior of …
lfdversluis Jul 1, 2016
78123b0
Refactored test_candidates to handle the now asynchronous behavior of…
lfdversluis Jul 1, 2016
71db1e5
Refactored test_classification to handle the now asynchronous behavio…
lfdversluis Jul 1, 2016
94a5bad
Refactored test_double_signature to handle the now asynchronous behav…
lfdversluis Jul 1, 2016
b99f183
Refactored test_dynamicsettings to handle the now asynchronous behavi…
lfdversluis Jul 1, 2016
1e5d46b
Refactored test_crypto to handle the now asynchronous behavior of Dis…
lfdversluis Jul 1, 2016
85f7462
Refactored test_destroycommunity to handle the now asynchronous behav…
lfdversluis Jul 1, 2016
7a11121
Refactored test_discovery to handle the now asynchronous behavior of …
lfdversluis Jul 1, 2016
22c98a4
Refactored test_identicalpayload to handle the now asynchronous behav…
lfdversluis Jul 1, 2016
2920eab
Refactored test_member to handle the now asynchronous behavior of Dis…
lfdversluis Jul 1, 2016
791885e
Refactored test_missingidentity to handle the now asynchronous behavi…
lfdversluis Jul 1, 2016
bf34526
Refactored test_missingmessage to handle the now asynchronous behavio…
lfdversluis Jul 1, 2016
dc6778b
Refactored test_nat_detection to handle the now asynchronous behavior…
lfdversluis Jul 1, 2016
e67dfad
Refactored test_neighbourhood to handle the now asynchronous behavior…
lfdversluis Jul 1, 2016
27a70e3
Refactored test_overlay to handle the now asynchronous behavior of Di…
lfdversluis Jul 1, 2016
7082fe0
Refactored test_pruning to handle the now asynchronous behavior of Di…
lfdversluis Jul 1, 2016
fb3d3a7
Refactored test_requestcache to handle the now asynchronous behavior …
lfdversluis Jul 1, 2016
a037591
Refactored test_sequence to handle the now asynchronous behavior of D…
lfdversluis Jul 1, 2016
af43242
Refactored test_signature to handle the now asynchronous behavior of …
lfdversluis Jul 1, 2016
a8a1b9d
Refactored test_sync to handle the now asynchronous behavior of Dispersy
lfdversluis Jul 1, 2016
528fbb2
Refactored test_timeline to handle the now asynchronous behavior of D…
lfdversluis Jul 1, 2016
1bbce24
Refactored test_undo to handle the now asynchronous behavior of Dispersy
lfdversluis Jul 1, 2016
1e636ca
Refactored test_walker to handle the now asynchronous behavior of Dis…
lfdversluis Jul 1, 2016
3242baf
Added tests that test the upgrade procedure of Dispersy
lfdversluis Jul 1, 2016
0f4deb1
Added tests that test the null endpoint
lfdversluis Jul 1, 2016
c006a73
Added three empty database files to test the upgrade process with
lfdversluis Jul 1, 2016
b3df402
Added pylintrc
lfdversluis Jul 7, 2016
71220c4
Added Jenkins file to run the code on the das5
lfdversluis Jul 7, 2016
fb80900
Updated Jenkinsfile
lfdversluis Jul 7, 2016
be3554c
Initializing the statistics on the tracker plugin.
lfdversluis Jul 11, 2016
a456f18
Added a missing yield
lfdversluis Jul 11, 2016
7e7ffd7
Refactored function to ensure it can handle deferreds returned
lfdversluis Jul 11, 2016
f5ffbf7
Refactored _load_persistent_storage
lfdversluis Jul 11, 2016
b89c967
Added missing yield
lfdversluis Jul 11, 2016
8af50f1
fixup! Refactored community to handle the deferreds returned by Storm…
lfdversluis Jul 11, 2016
c51ea1f
fixup! Refactored conversion to handle deferreds returned by StormDBM…
lfdversluis Jul 11, 2016
ed09981
fixup! Refactored dispersy to handle the deferreds returned by StormD…
lfdversluis Jul 11, 2016
1cb10b3
fixup! Refactored community to handle the deferreds returned by Storm…
lfdversluis Jul 12, 2016
53b1c40
fixup! Refactored dispersy to handle the deferreds returned by StormD…
lfdversluis Jul 12, 2016
8451580
fixup! Refactored dispersy to handle the deferreds returned by StormD…
lfdversluis Jul 12, 2016
1455e81
Removed unnecessary database commit
lfdversluis Jul 12, 2016
3f5b642
Added missing yields
lfdversluis Jul 13, 2016
16bf76a
fixup! Refactored community to handle the deferreds returned by Storm…
lfdversluis Jul 14, 2016
40a8813
fixup! Refactored community to handle the deferreds returned by Storm…
lfdversluis Jul 14, 2016
7a01762
fixup! Refactored dispersy to handle the deferreds returned by StormD…
lfdversluis Jul 14, 2016
a339f30
fixup! Refactored dispersy to handle the deferreds returned by StormD…
lfdversluis Jul 14, 2016
397cc1a
fixed a bug, added yields and raised the timeout on the introduction …
lfdversluis Jul 15, 2016
0aee86d
more debugging
lfdversluis Jul 15, 2016
709b701
Do not accept messages from localhost, else they will arrive twice
lfdversluis Jul 15, 2016
d67b1d6
DROPME: run sql on the mainthread test
lfdversluis Jul 15, 2016
f6d6d5d
Added missing yields
lfdversluis Jul 16, 2016
27384cf
DROPME: more debug
lfdversluis Jul 17, 2016
42fbb48
DROPME: more debugging
lfdversluis Jul 17, 2016
dbe2f4e
DROPME: even more debugging
lfdversluis Jul 17, 2016
850d84d
DROPME more debugging
lfdversluis Jul 18, 2016
21acaa8
DROPME debug
lfdversluis Jul 18, 2016
9e5d28a
Removed some debug stuff'
lfdversluis Jul 19, 2016
229244d
DROPME debug
lfdversluis Jul 20, 2016
08702c3
added drop logs
lfdversluis Jul 21, 2016
20c7353
added yield
lfdversluis Jul 21, 2016
52b5a6f
Changed storm to use one connection and one cursor object
lfdversluis Jul 21, 2016
aae6ea6
updated database to match the new situation in stormdb
lfdversluis Jul 21, 2016
05f7c73
updated stormDBManager and dispersy
lfdversluis Jul 21, 2016
445e815
Reenabled commits
lfdversluis Jul 21, 2016
c3be413
removed ignorecommit case
lfdversluis Jul 21, 2016
19f4a4a
fixed some bugs
lfdversluis Jul 21, 2016
9222ce6
closing stormdb is probably better
lfdversluis Jul 21, 2016
0b991d9
debug cleanup
lfdversluis Jul 22, 2016
fd50353
unloading communities need to be yielded too
lfdversluis Jul 22, 2016
06f77d9
Added the IgnoreCommits structure to StormDBManager as its used by Tr…
lfdversluis Jul 25, 2016
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
386 changes: 386 additions & 0 deletions .pylintrc
Original file line number Diff line number Diff line change
@@ -0,0 +1,386 @@
[MASTER]

# Specify a configuration file.
#rcfile=

# Python code to execute, usually for sys.path manipulation such as
# pygtk.require().
#init-hook=

# Add files or directories to the blacklist. They should be base names, not
# paths.
ignore=.git,pymdht,libnacl,data

# Pickle collected data for later comparisons.
persistent=yes

# List of plugins (as comma separated values of python modules names) to load,
# usually to register additional checkers.
load-plugins=pylint_common,pylint.extensions.check_elif,pylint.extensions.check_docs

# To install required plugins:
# sudo apt install python-pylint-common python-enchant python-pylint-plugin-utils


# Use multiple processes to speed up Pylint.
jobs=8

# Allow loading of arbitrary C extensions. Extensions are imported into the
# active Python interpreter and may run arbitrary code.
unsafe-load-any-extension=no

# A comma-separated list of package or module names from where C extensions may
# be loaded. Extensions are loading into the active Python interpreter and may
# run arbitrary code
extension-pkg-whitelist=libtorrent

# Allow optimization of some AST trees. This will activate a peephole AST
# optimizer, which will apply various small optimizations. For instance, it can
# be used to obtain the result of joining multiple strings with the addition
# operator. Joining a lot of strings can lead to a maximum recursion error in
# Pylint and this flag can prevent that. It has one side effect, the resulting
# AST will be different than the one from reality.
optimize-ast=yes


[MESSAGES CONTROL]

# Only show warnings with the listed confidence levels. Leave empty to show
# all. Valid levels: HIGH, INFERENCE, INFERENCE_FAILURE, UNDEFINED
confidence=

# Enable the message, report, category or checker with the given id(s). You can
# either give multiple identifier separated by comma (,) or put this option
# multiple time. See also the "--disable" option for examples.
#enable=

# Disable the message, report, category or checker with the given id(s). You
# can either give multiple identifiers separated by comma (,) or put this
# option multiple times (only on the command line, not in the configuration
# file where it should appear only once).You can also use "--disable=all" to
# disable everything first and then reenable specific checks. For example, if
# you want to run only the similarities checker, you can use "--disable=all
# --enable=similarities". If you want to run only the classes checker, but have
# no Warning level messages displayed, use"--disable=all --enable=classes
# --disable=W"
#disable=import-star-module-level,old-octal-literal,oct-method,print-statement,unpacking-in-except,parameter-unpacking,backtick,old-raise-syntax,old-ne-operator,long-suffix,dict-view-method,dict-iter-method,metaclass-assignment,next-method-called,raising-string,indexing-exception,raw_input-builtin,long-builtin,file-builtin,execfile-builtin,coerce-builtin,cmp-builtin,buffer-builtin,basestring-builtin,apply-builtin,filter-builtin-not-iterating,using-cmp-argument,useless-suppression,range-builtin-not-iterating,suppressed-message,no-absolute-import,old-division,cmp-method,reload-builtin,zip-builtin-not-iterating,intern-builtin,unichr-builtin,reduce-builtin,standarderror-builtin,unicode-builtin,xrange-builtin,coerce-method,delslice-method,getslice-method,setslice-method,input-builtin,round-builtin,hex-method,nonzero-method,map-builtin-not-iterating
disable=C0321,W0142,invalid-name,missing-docstring,missing-type-doc


[REPORTS]

# Set the output format. Available formats are text, parseable, colorized, msvs
# (visual studio) and html. You can also give a reporter class, eg
# mypackage.mymodule.MyReporterClass.
output-format=text

# Put messages in a separate file for each module / package specified on the
# command line instead of printing them on stdout. Reports (if any) will be
# written in a file name "pylint_global.[txt|html]".
files-output=no

# Tells whether to display a full report or only the messages
reports=no

# Python expression which should return a note less than 10 (10 is the highest
# note). You have access to the variables errors warning, statement which
# respectively contain the number of errors / warnings messages and the total
# number of statements analyzed. This is used by the global evaluation report
# (RP0004).
evaluation=10.0 - ((float(5 * error + warning + refactor + convention) / statement) * 10)

# Template used to display messages. This is a python new-style format string
# used to format the message information. See doc for all details
msg-template="{path}:{line}: [{msg_id}({symbol}), {obj}] {msg}"


[VARIABLES]

# Tells whether we should check for unused import in __init__ files.
init-import=no

# A regular expression matching the name of dummy variables (i.e. expectedly
# not used).
dummy-variables-rgx=_$|dummy

# List of additional names supposed to be defined in builtins. Remember that
# you should avoid to define new builtins when possible.
additional-builtins=

# List of strings which can identify a callback function by name. A callback
# name must start or end with one of those strings.
callbacks=cb_,_cb,on_


[TYPECHECK]

# Tells whether missing members accessed in mixin class should be ignored. A
# mixin class is detected if its name ends with "mixin" (case insensitive).
ignore-mixin-members=yes

# List of module names for which member attributes should not be checked
# (useful for modules/projects where namespaces are manipulated during runtime
# and thus existing member attributes cannot be deduced by static analysis. It
# supports qualified module names, as well as Unix pattern matching.
ignored-modules=

# List of classes names for which member attributes should not be checked
# (useful for classes with attributes dynamically set). This supports can work
# with qualified names.
ignored-classes=SQLObject,twisted.internet.reactor,ephem,libtorrent

# List of members which are set dynamically and missed by pylint inference
# system, and so shouldn't trigger E1101 when accessed. Python regular
# expressions are accepted.
generated-members=


[SPELLING]

# Spelling dictionary name. Available dictionaries: none. To make it working
# install python-enchant package.
# spelling-dict=en # Disable it for now, breaks pylint due to non-ascii chars.

# To install required dictionary:
# sudo apt install aspell-en

# List of comma separated words that should not be checked.
spelling-ignore-words=Tribler,dispersy,pymdht

# A path to a file that contains private dictionary; one word per line.
spelling-private-dict-file=

# Tells whether to store unknown words to indicated private dictionary in
# --spelling-private-dict-file option instead of raising a message.
spelling-store-unknown-words=no


[SIMILARITIES]

# Minimum lines number of a similarity.
min-similarity-lines=4

# Ignore comments when computing similarities.
ignore-comments=yes

# Ignore docstrings when computing similarities.
ignore-docstrings=yes

# Ignore imports when computing similarities.
ignore-imports=no


[MISCELLANEOUS]

# List of note tags to take in consideration, separated by a comma.
notes=FIXME,XXX,TODO


[LOGGING]

# Logging modules to check that the string format arguments are in logging
# function parameter format
logging-modules=logging


[FORMAT]

# Maximum number of characters on a single line.
max-line-length=120

# Regexp for a line that is allowed to be longer than the limit.
ignore-long-lines=^\s*(# )?<?https?://\S+>?$

# Allow the body of an if to be on the same line as the test if there is no
# else.
single-line-if-stmt=no

# List of optional constructs for which whitespace checking is disabled. `dict-
# separator` is used to allow tabulation in dicts, etc.: {1 : 1,\n222: 2}.
# `trailing-comma` allows a space between comma and closing bracket: (a, ).
# `empty-line` allows space-only lines.
no-space-check=trailing-comma,dict-separator

# Maximum number of lines in a module
max-module-lines=2000

# String used as indentation unit. This is usually " " (4 spaces) or "\t" (1
# tab).
indent-string=' '

# Number of spaces of indent required inside a hanging or continued line.
indent-after-paren=4

# Expected format of line ending, e.g. empty (any line ending), LF or CRLF.
expected-line-ending-format=LF


[BASIC]

# List of builtins function names that should not be used, separated by a comma
bad-functions=map,filter,input

# Good variable names which should always be accepted, separated by a comma
good-names=i,j,k,ex,Run,_,d

# Bad variable names which should always be refused, separated by a comma
bad-names=foo,bar,baz,toto,tutu,tata

# Colon-delimited sets of names that determine each other's naming style when
# the name regexes allow several styles.
name-group=

# Include a hint for the correct naming format with invalid-name
include-naming-hint=yes

# Regular expression matching correct function names
function-rgx=[a-z_][a-z0-9_]{2,30}$

# Naming hint for function names
function-name-hint=[a-z_][a-z0-9_]{2,30}$

# Regular expression matching correct variable names
variable-rgx=[a-z_][a-z0-9_]{2,30}$

# Naming hint for variable names
variable-name-hint=[a-z_][a-z0-9_]{2,30}$

# Regular expression matching correct constant names
const-rgx=(([A-Z_][A-Z0-9_]*)|(__.*__))$

# Naming hint for constant names
const-name-hint=(([A-Z_][A-Z0-9_]*)|(__.*__))$

# Regular expression matching correct attribute names
attr-rgx=[a-z_][a-z0-9_]{2,30}$

# Naming hint for attribute names
attr-name-hint=[a-z_][a-z0-9_]{2,30}$

# Regular expression matching correct argument names
argument-rgx=[a-z_][a-z0-9_]{2,30}$

# Naming hint for argument names
argument-name-hint=[a-z_][a-z0-9_]{2,30}$

# Regular expression matching correct class attribute names
class-attribute-rgx=([A-Za-z_][A-Za-z0-9_]{2,30}|(__.*__))$

# Naming hint for class attribute names
class-attribute-name-hint=([A-Za-z_][A-Za-z0-9_]{2,30}|(__.*__))$

# Regular expression matching correct inline iteration names
inlinevar-rgx=[A-Za-z_][A-Za-z0-9_]*$

# Naming hint for inline iteration names
inlinevar-name-hint=[A-Za-z_][A-Za-z0-9_]*$

# Regular expression matching correct class names
class-rgx=[A-Z_][a-zA-Z0-9]+$

# Naming hint for class names
class-name-hint=[A-Z_][a-zA-Z0-9]+$

# Regular expression matching correct module names
module-rgx=(([a-z_][a-z0-9_]*)|([A-Z][a-zA-Z0-9]+))$

# Naming hint for module names
module-name-hint=(([a-z_][a-z0-9_]*)|([A-Z][a-zA-Z0-9]+))$

# Regular expression matching correct method names
method-rgx=[a-z_][a-z0-9_]{2,30}$

# Naming hint for method names
method-name-hint=[a-z_][a-z0-9_]{2,30}$

# Regular expression which should only match function or class names that do
# not require a docstring.
no-docstring-rgx=^_

# Minimum line length for functions/classes that require docstrings, shorter
# ones are exempt.
docstring-min-length=5


[ELIF]

# Maximum number of nested blocks for function / method body
max-nested-blocks=5


[IMPORTS]

# Deprecated modules which should not be used, separated by a comma
deprecated-modules=regsub,TERMIOS,Bastion,rexec

# Create a graph of every (i.e. internal and external) dependencies in the
# given file (report RP0402 must not be disabled)
import-graph=

# Create a graph of external dependencies in the given file (report RP0402 must
# not be disabled)
ext-import-graph=

# Create a graph of internal dependencies in the given file (report RP0402 must
# not be disabled)
int-import-graph=


[DESIGN]

# Maximum number of arguments for function / method
max-args=5

# Argument names that match this expression will be ignored. Default to name
# with leading underscore
ignored-argument-names=_.*

# Maximum number of locals for function / method body
max-locals=15

# Maximum number of return / yield for function / method body
max-returns=6

# Maximum number of branch for function / method body
max-branches=12

# Maximum number of statements in function / method body
max-statements=50

# Maximum number of parents for a class (see R0901).
max-parents=7

# Maximum number of attributes for a class (see R0902).
max-attributes=7

# Minimum number of public methods for a class (see R0903).
min-public-methods=2

# Maximum number of public methods for a class (see R0904).
max-public-methods=20

# Maximum number of boolean expressions in a if statement
max-bool-expr=5


[CLASSES]

# List of method names used to declare (i.e. assign) instance attributes.
defining-attr-methods=__init__,__new__,setUp

# List of valid names for the first argument in a class method.
valid-classmethod-first-arg=cls

# List of valid names for the first argument in a metaclass class method.
valid-metaclass-classmethod-first-arg=mcs

# List of member names, which should be excluded from the protected access
# warning.
exclude-protected=_asdict,_fields,_replace,_source,_make


[EXCEPTIONS]

# Exceptions that will emit a warning when being caught. Defaults to
# "Exception"
overgeneral-exceptions=Exception
Loading