Introduce cache database and reindex command #4650
Conversation
barnabasdomozi
force-pushed
the
cachedb
branch
from
ba4dcf6 to
110a261
Compare
bruntib
left a comment
bruntib
left a comment
There was a problem hiding this comment.
Some tests would also be useful.
|
|
||
|
|
||
| def __process_file(file_path: str) -> Tuple[str, List[str]]: | ||
| with open(file_path, 'rb') as fp: |
There was a problem hiding this comment.
| with open(file_path, 'rb') as fp: | |
| with open(file_path, 'rb', encoding='utf-8', errors='ignore') as fp: |
Conventionally we're using utf-8 encoding for every open(). We should get rid of this in the future by setting encoding globally.
There was a problem hiding this comment.
We are opening the file in binary mode, where encoding and errors options shouldn't be specified, see docs: https://docs.python.org/3/library/functions.html#open
For XML parsing, the plistparser uses lxml, see (tools/report-converter/codechecker_report_converter/report/parser/plist.py, line 104). It accepts the data in binary form, thus we are reading the file in binary mode.
|
|
||
| def __create_tables(self): | ||
| if not self.__table_exists("plist_lookup"): | ||
| self.__cur.execute("CREATE TABLE plist_lookup(plist, source)") |
There was a problem hiding this comment.
Shouldn't column types be indicated here? The default type is BLOB (https://www.sqlite.org/datatype3.html: 3.1)
| plist_files = list(filter(lambda f: f.endswith( | ||
| plistparser.EXTENSION), os.listdir(report_dir))) | ||
| plist_files = list(map(lambda f: os.path.abspath( | ||
| os.path.join(report_dir, f)), plist_files)) | ||
| plist_files = list(filter(lambda f: f not in indexed_files, plist_files)) |
There was a problem hiding this comment.
List generation is not necessary. I think, the generator objects could be nested into each other.
barnabasdomozi
force-pushed
the
cachedb
branch
from
110a261 to
6b7cce9
Compare
Thanks for the review! I will add tests in a separate commit. |
barnabasdomozi
force-pushed
the
cachedb
branch
from
6b7cce9 to
c5467c8
Compare
barnabasdomozi
force-pushed
the
cachedb
branch
from
c5467c8 to
f05de3e
Compare
|
@bruntib @Discookie |
3 participants
No description provided.