Skip to content

Allow writing materialized views during index build #2693

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
ullingerc wants to merge 6 commits into
base: master
Choose a base branch
from
Open

Allow writing materialized views during index build #2693

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
02a27b2
write materialized views in qlever-index
ullingerc Feb 3, 2026
c06346b
refactor, nice code
ullingerc Feb 3, 2026
5df3fc3
Unit test
ullingerc Feb 3, 2026
1309ec9
improve help text
ullingerc Feb 3, 2026
d099e09
move
ullingerc Feb 3, 2026
de4425e
fix unit test
ullingerc Feb 3, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
39 changes: 38 additions & 1 deletion src/index/IndexBuilderMain.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,7 @@
#include "libqlever/Qlever.h"
#include "util/ProgramOptionsHelpers.h"
#include "util/ReadableNumberFacet.h"
#include "util/json.h"

using std::string;

Expand Down Expand Up @@ -142,6 +143,36 @@ auto getFileSpecifications = [](const auto& filetype, auto& inputFile,
return fileSpecs;
};

// Helper to convert the JSON given for writing materialized views to a proper
// `WriteMaterializedViews` vector.
qlever::IndexBuilderConfig::WriteMaterializedViews parseMaterializedViewsJson(
std::string materializedViewsJson) {
Copy link
Collaborator

@RobinTF RobinTF Feb 4, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think using const std::string& or std::string_view should be completely sufficient here, with no change in behaviour for the json parser.

qlever::IndexBuilderConfig::WriteMaterializedViews views;
if (!materializedViewsJson.empty()) {
AD_LOG_INFO << "Writing materialized views..." << std::endl;
try {
auto viewsJson = nlohmann::json::parse(materializedViewsJson);
if (!viewsJson.is_object()) {
throw std::runtime_error(
"The --write-materialized-views option must be a JSON object "
"mapping view names to SPARQL queries.");
}
for (auto& [viewName, query] : viewsJson.items()) {
if (!query.is_string()) {
throw std::runtime_error(absl::StrCat(
"Query for materialized view '", viewName,
"' must be a string, but got type: ", jsonToTypeString(query)));
}
views.push_back({std::move(viewName), query.get<std::string>()});
}
} catch (const nlohmann::json::exception& e) {
throw std::runtime_error(
absl::StrCat("Failed to parse materialized views JSON: ", e.what()));
}
}
return views;
}

// Main function.
int main(int argc, char** argv) {
// Copy the git hash and datetime of compilation (which require relinking)
Expand All @@ -159,6 +190,7 @@ int main(int argc, char** argv) {
std::vector<string> inputFile;
std::vector<string> defaultGraphs;
std::vector<bool> parseParallel;
std::string materializedViewsJson;

boost::program_options::options_description boostOptions(
"Options for qlever-index");
Expand Down Expand Up @@ -251,6 +283,10 @@ int main(int argc, char** argv) {
"large enough to hold a single input triple. Default: 10 MB.");
add("keep-temporary-files,k", po::bool_switch(&config.keepTemporaryFiles_),
"Do not delete temporary files from index creation for debugging.");
add("write-materialized-views", po::value(&materializedViewsJson),
"Write materialized views after index building. Takes a JSON object "
"mapping view names to SELECT queries for writing the view, for example: "
R"({"view1": "SELECT ...", "view2": "SELECT ..."})");

// Process command line arguments.
po::variables_map optionsMap;
Expand Down Expand Up @@ -280,9 +316,10 @@ int main(int argc, char** argv) {
try {
config.inputFiles_ = getFileSpecifications(filetype, inputFile,
defaultGraphs, parseParallel);
config.writeMaterializedViews_ =
parseMaterializedViewsJson(std::move(materializedViewsJson));
Copy link
Collaborator

@RobinTF RobinTF Feb 4, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In case you apply my suggestion, you don't need to move here.

config.validate();
qlever::Qlever::buildIndex(config);

} catch (std::exception& e) {
AD_LOG_ERROR << "Creating the index for QLever failed with the following "
"exception: "
Expand Down
12 changes: 12 additions & 0 deletions src/libqlever/Qlever.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -105,6 +105,18 @@ void Qlever::buildIndex(IndexBuilderConfig config) {
"version of QLever");
#endif
}

// Build materialized views if requested.
if (!config.writeMaterializedViews_.empty()) {
AD_LOG_INFO
<< "Loading the new index to execute materialized view write queries..."
<< std::endl;
Qlever engine{EngineConfig{config}};
for (auto& [viewName, query] : config.writeMaterializedViews_) {
engine.writeMaterializedView(viewName, query);
}
AD_LOG_INFO << "All materialized views written successfully." << std::endl;
}
}

// ___________________________________________________________________________
Expand Down
5 changes: 5 additions & 0 deletions src/libqlever/Qlever.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -137,6 +137,11 @@ struct IndexBuilderConfig : CommonConfig {
float bScoringParam_ = 0.75;
float kScoringParam_ = 1.75;

// Materialized views to be written after normal index build is complete.
using WriteMaterializedViews =
std::vector<std::pair<std::string, std::string>>;
WriteMaterializedViews writeMaterializedViews_;

// Assert that the given configuration is valid.
void validate() const;

Expand Down
8 changes: 8 additions & 0 deletions test/libqlever/QleverTest.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -36,7 +36,12 @@ TEST(LibQlever, buildIndexAndRunQuery) {
::testing::HasSubstr("buffer size"));

c.parserBufferSize_ = std::nullopt;

// Test materialized views to be written at index build time.
c.writeMaterializedViews_ = {{"demoView", "SELECT ?s { ?s <p> <o> }"}};

EXPECT_NO_THROW(Qlever::buildIndex(c));

{
EngineConfig ec{c};
Qlever engine{ec};
Expand Down Expand Up @@ -78,6 +83,9 @@ TEST(LibQlever, buildIndexAndRunQuery) {
engine.clearNamedResultCache();
AD_EXPECT_THROW_WITH_MESSAGE(engine.query(serviceQuery), notPinned);
AD_EXPECT_THROW_WITH_MESSAGE(engine.query(serviceQuery2), notPinned);

// Test that the requested materialized view exists.
EXPECT_NO_THROW(engine.loadMaterializedView("demoView"));
}

#ifndef QLEVER_REDUCED_FEATURE_SET_FOR_CPP17
Expand Down
Loading