FE computation for Categoricals fails when the parameter vector has 0 entries

[MagnusKoudahl]

When running an HMM, FE computation fails due to 0 entries. For example in the model

@model function t_maze(A,D,B,T)
Ac = constvar(A)
z_0 ~ Categorical(D)
z = randomvar(T)
x = datavar(Vector{Float64}, T)
z_prev = z_0
        for t in 1:T
                z[t] ~ Transition(z_prev,B[t])
                x[t] ~ Transition(z[t], Ac)
                z_prev = z[t]
        end
end

We are unable to compute FE if D contains 0's

[bartvanerp]

Could be related to #75

[bvdmitri]

@ismailsenoz @ThijsvdLaar

The reason for that is because we need to compute mean(log, q_p) in average energy function for Categorical node here. If q_p is a PointMass and has zero entries, this operation silently returns Inf.

Would it be reasonable to clamp values of parameter vector between tiny and 1? Do we allow this? From math point of view zero entry in p probably means that you don't need this category, so current behaviour is not really a bug IMO, but I would like to know your opinion.

[bvdmitri]

For reference we already have a similar clamp procedure for the Transition node: here

[ismailsenoz]

I think clamping and renormalizing the vector should resolve the problem. By definition, the logarithm function can not accept a zero-entry, so this is not a bug in my view. ForneyLab clamp log for entropy computation for a categorical variable (https://github.com/biaslab/ForneyLab.jl/blob/fca07c3bcb3a5abe7393dfb275a2e47e54fd68aa/src/factor_nodes/categorical.jl#L136).