Code silently fails after running model.fit() #720
Description
I am building a recommender system with LightFM. When the line of code model.fit() is executed, the code is immediately stopped. Sometimes, I run the code again and then it works and the model is trained (without having changed any parameters !)
I tried to change some parameters :
- How sparse my matrix is (from 50% to 99.99%)
- The number of epochs
- The number of latent factors
- The loss that I use
I've set the random.seed(42) and the random_state=42.
My interaction list is [(user_id, item_id, score)]
I build my dataset properly, and I don't have any Nan or Inf.
But when i run the code, it sometimes work, and it sometimes don't. Without any error message! I've set the verbose=True :
- When it works i see the Epoch bar going to 100%
- When it does not work, I see the Epoch bar at 0% and the code is exited / stopped.
Do you have any idea why it is happening ?
I am on Windows and using Visual Studio Code with LightFM 1.7
Here is my code :
def matrix_factorization(self, interaction_matrix: pd.DataFrame):
dataset = LightfmMatrixFactorizer.build_dataset(interaction_matrix=interaction_matrix)
interaction_list = [
(user_id, item_id, score)
for user_id in interaction_matrix.index
for item_id in interaction_matrix.columns
if (score := interaction_matrix.loc[user_id, item_id]) > self.threshold
]
interactions, weights = dataset.build_interactions(interaction_list)
sparsity = 1 - (interactions.nnz / (interactions.shape[0] * interactions.shape[1]))
print(f"Sparsity: {sparsity:.2%}")
# Initialize LightFM model (e.g., warp method for implicit feedback)
model = LightFM(no_components=self.n_factors, loss=self.loss, random_state=42)
# Train the LightFM model
try:
model.fit(interactions=interactions, sample_weight=weights, epochs=self.epochs, num_threads=self.num_threads, verbose = True)
except Exception as e:
print(f"Error occurred: {e}")
Main :
lightfm3 = LightfmMatrixFactorizer(score_percentile=70, n_factors=5, loss='bpr', epochs=10, num_threads=1)
im = pd.read_csv("data/interaction_matrix/global_interaction_matrix.csv", index_col=0)
print(im.head(3))
user_factors, item_factors, approx_matrix = lightfm3.matrix_factorization(interaction_matrix=im)