How to avoid unnecessary heap allocations when working with DynObject?

[evroon]

My question in short: What is the most performant way to iterate over a BTreeMap enscapulated by Value in a filter?

Consider this piece of code to filter items from a BTreeMap by key (from the minimal example below):

// filter unwanted fields
// `value` is a `minijinja::Value` that represents a `BTreeMap`
// `negate == true`: keep all keys that are in `fields`
// `negate == false`: keep all keys that are not in `fields`
let result = value
    .as_object()
    .unwrap()
    .try_iter_pairs()
    .unwrap()
    .filter(|(key, _)| {
        if negate {
            !fields.contains(key)
        } else {
            fields.contains(key)
        }
    })
    .collect();

So this does try_iter_pairs() which clones the value (although I think the DynObject is stored in an Arc so that might not have much of a performance cost), but it also calls get_value which I think does this and therefore clones the individual values and then converts them from V into Value:

minijinja/minijinja/src/value/object.rs

Lines 839 to 841 in d1ce496

	fn get_value(self: &Arc<Self>, key: &Value) -> Option<Value> {
	self.get(key).cloned().map(|v| v.into())
	}

What is the best/most performant way to iterate over a Value containing a dictionary in a filter with less heap allocations? I think the value is actually a BTreeMap somewhere underneath if I look at this and the profiling I did:

minijinja/minijinja/src/value/mod.rs

Line 246 in d1ce496

pub(crate) type ValueMap = std::collections::BTreeMap<Value, Value>;

But can I get access to that raw BTreeMap? That way, I could filter on keys more efficiently.

I ran this profile using samply if that helps:

To demonstrate my question I tried to make an example as minimal as possible

use std::{fs::File, io::BufReader};

use minijinja::{
    context,
    value::{from_args, Kwargs, Rest},
    Environment, Error, Value,
};
use serde_json::Deserializer;

/// Returns `true` if value is considered undefined, `false` otherwise
pub fn undefined_field(value: Value) -> Result<bool, minijinja::Error> {
    Ok(!value.try_iter()?.nth(1).unwrap().is_true())
}

/// Filter fields (`fields`) by key in a dictionary `value`.
///
/// `args` contains `negate` as keyword argument, with a default of `false`
/// if `negate == true`, the filter is reversed: specified `fields` get excluded
/// from the output instead of included.
pub fn select_fields(value: Value, fields: Value, args: Rest<Value>) -> Result<Value, Error> {
    let fields: Vec<Value> = fields.as_object().unwrap().try_iter().unwrap().collect();

    let (_args, kwargs) = from_args::<(&[Value], Kwargs)>(&args)?;
    let negate = kwargs.get::<bool>("negate").unwrap_or(false);

    // filter unwanted fields
    let result = value
        .as_object()
        .unwrap()
        .try_iter_pairs()
        .unwrap()
        .filter(|(key, _)| {
            if negate {
                !fields.contains(key)
            } else {
                fields.contains(key)
            }
        })
        .collect();

    Ok(result)
}

const TEMPLATE: &str = r#"
<?xml version='1.0' encoding='utf-8'?>
<channel>
    <title>
        test of select_fields filter
    </title>
    {% for item in items %}
    <item>
        {% for field, values in item|select_fields(['key1', 'key2'], negate=True)|dictsort|reject('undefined_field') %}
            {% for value in values %}
                {{ value }}
            {% endfor %}
        {% endfor %}
    </item>
    {% endfor %}
</channel>
</rss>
"#;

fn main() {
    let mut env = Environment::new();
    env.add_template("template", TEMPLATE).unwrap();
    env.add_filter("select_fields", select_fields);
    env.add_test("undefined_field", undefined_field);
    let items_buffer = BufReader::new(File::open("items-200k.json").unwrap());

    let des = Deserializer::from_reader(items_buffer).into_iter::<serde_json::Value>();

    let result = env.get_template("template")
        .unwrap()
        .render(context!(items => Value::make_one_shot_iterator(des.map(|item| Value::from_serialize(item.unwrap())))));

    result.unwrap();
}

[mitsuhiko]

So this does try_iter_pairs() which clones the value (although I think the DynObject is stored in an Arc so that might not have much of a performance cost), but it also calls get_value which I think does this and therefore clones the individual values and then converts them from V into Value:

If the map has a Value as value then clone will only increment the refcount (or something equally cheap). If however the map contains something like a String then it will do something less optimal.

But can I get access to that raw BTreeMap? That way, I could filter on keys more efficiently.

If you know what the value is you can downcast it:

let map = Value::from_object({
    let mut m = BTreeMap::<String, i32>::new();
    m.insert("foo".into(), 2);
    m
});
let btree_map = map.downcast_object_ref::<BTreeMap<String, i32>>().unwrap();