streams over paginate loop over the same record over and over again

[jessfraz]

Describe the bug

• create an invoice item
• list pending invoice items
• use the stream pagination
• watch as it loops endlessly over the same record
  some sample code:

#[tracing::instrument(skip(self))]
    async fn list_pending_invoice_items(&self, customer_id: &str) -> Result<Vec<crate::payments::types::InvoiceItem>> {
        // List all the invoice items that are PENDING for the customer.
        let mut invoice_items: Vec<crate::payments::types::InvoiceItem> = Default::default();

        let params = stripe::ListInvoiceItems {
            customer: Some(stripe::CustomerId::from_str(customer_id)?),
            limit: Some(100),
            pending: Some(true),
            ..Default::default()
        };
        let list = stripe::InvoiceItem::list(&self.client, &params).await?.paginate(params);
        let mut stream = list.stream(&self.client);
        // For each invoice in the stream add it to our array.
        while let Some(Ok(next)) = stream.next().await {
            invoice_items.push(next.into());
        }

        Ok(invoice_items)
    }

To Reproduce

see above sorry!

Expected behavior

Previous we were on version 0.21 now on 0.26 we see this bug. Without the code changing it appeared. So i would expect the stream to stop after its gotten all the records

Code snippets

No response

OS

macos

Rust version

1.75.0

Library version

0.26.0

API version

whatever the default is for the library?

Additional context

No response

[jessfraz]

ah okay so 0.25 works, 0.26 doesnt and I think it has to do with what is noted in the changelog about changes to List

[mzeitlin11]

Thanks @jessfraz - hoping this will be fixed by pagination changes in #452. If possible, it would be extremely helpful if you could capture the sequence of stripe responses to your sample code (or just debug printing the first 2 returned List I guess)? Then I can add as a simple regression test that doesn't need to hit the stripe API

[jessfraz]

yes will get you some json from our "test mode" today! thank you!

[jessfraz]

you know what is weird... it just started working fine on 0.26, my test that was failing is now passing, when i tried to get you the debug output

[jessfraz]

maybe it was a server side stripe thing because I swear its the same code

[mzeitlin11]

Thanks for looking into it - that's strange! Think this is good to leave open regardless because I wouldn't be surprised if it's a weird pagination edge case.

[jessfraz]

Okay it is def a weird edge case, in that previously it was happening on listing invoice items, but now it is happening on listing invoices.

here is the debug ooutput of listing invoices when it happens

This is in our stripe test mode so nothing is bad or a secret.

list: ListPaginator { page: List { data: [Invoice { id: InvoiceId("in_1OWRfIGC2bHVgrBXnRrx3rka"), account_country: Some("US"), account_name: Some("Zoo Corporation"), account_tax_ids: None, amount_due: Some(0), amount_paid: Some(0), amount_remaining: Some(0), amount_shipping: Some(0), application: None, application_fee_amount: None, attempt_count: Some(0), attempted: Some(false), auto_advance: Some(true), automatic_tax: Some(AutomaticTax { enabled: false, status: None }), billing_reason: Some(Manual), charge: None, collection_method: Some(ChargeAutomatically), created: Some(1704754288), currency: Some(USD), custom_fields: None, customer: Some(Id(CustomerId("cus_PJGZy9fkkikP3k"))), customer_address: None, customer_email: Some("test@create_invoice_item_beta.com"), customer_name: Some("Test User"), customer_phone: None, customer_shipping: None, customer_tax_exempt: Some(None), customer_tax_ids: Some([]), default_payment_method: None, default_source: None, default_tax_rates: Some([]), deleted: false, description: Some("I CAD to bill you."), discount: None, discounts: Some([]), due_date: None, effective_at: None, ending_balance: None, footer: None, from_invoice: None, hosted_invoice_url: None, invoice_pdf: None, last_finalization_error: None, latest_revision: None, lines: List { data: [InvoiceLineItem { id: InvoiceLineItemIdWebhook(InvoiceLineItemIdWebhook("il_1OWRfHGC2bHVgrBXdZPrmDTs")), amount: 150, amount_excluding_tax: Some(150), currency: USD, description: Some("12e4a6f1-8292-47bd-8e09-4438aa6abd84 GET /ws/modeling/commands 200 OK: 3 minutes"), discount_amounts: Some([DiscountsResourceDiscountAmount { amount: 150, discount: Id(DiscountId("di_1OWRfHGC2bHVgrBXJk0OkEe8")) }]), discountable: true, discounts: Some([Id(DiscountId("di_1OWRfHGC2bHVgrBXJk0OkEe8"))]), invoice_item: Some(Id(InvoiceItemId("ii_1OWRfHGC2bHVgrBXLFmhySEG"))), livemode: false, metadata: {"api_call_id": "12e4a6f1-8292-47bd-8e09-4438aa6abd84", "api_call_minutes": "3", "beta": "true", "user_id": "3e147226-5f6d-490d-888c-507bd7df4daa", "api_call_method": "GET", "api_call_endpoint": "/ws/modeling/commands", "api_call_status_code": "200 OK"}, period: Some(Period { end: Some(1704754287), start: Some(1704754287) }), plan: None, price: Some(Price { id: PriceId("price_1OWRfHGC2bHVgrBXv5rcq2qo"), active: Some(false), billing_scheme: Some(PerUnit), created: Some(1704754287), currency: Some(USD), currency_options: None, custom_unit_amount: None, deleted: false, livemode: Some(false), lookup_key: None, metadata: Some({}), nickname: None, product: Some(Id(ProductId("prod_PL7vbJk7ufq4zP"))), recurring: None, tax_behavior: Some(Inclusive), tiers: None, tiers_mode: None, transform_quantity: None, type_: Some(OneTime), unit_amount: Some(150), unit_amount_decimal: Some("150") }), proration: false, proration_details: Some(InvoicesResourceLineItemsProrationDetails { credited_items: None }), quantity: Some(1), subscription: None, subscription_item: None, tax_amounts: Some([]), tax_rates: Some([]), type_: InvoiceItem, unit_amount_excluding_tax: Some("150") }], has_more: false, total_count: Some(1), url: "/v1/invoices/in_1OWRfIGC2bHVgrBXnRrx3rka/lines" }, livemode: Some(false), metadata: Some({}), next_payment_attempt: Some(1704757888), number: None, on_behalf_of: None, paid: Some(false), paid_out_of_band: Some(false), payment_intent: None, payment_settings: Some(InvoicesPaymentSettings { default_mandate: None, payment_method_options: None, payment_method_types: None }), period_end: Some(1704754288), period_start: Some(1704754288), post_payment_credit_notes_amount: Some(0), pre_payment_credit_notes_amount: Some(0), quote: None, receipt_number: None, rendering_options: None, shipping_cost: None, shipping_details: None, starting_balance: Some(0), statement_descriptor: None, status: Some(Draft), status_transitions: Some(InvoicesStatusTransitions { finalized_at: None, marked_uncollectible_at: None, paid_at: None, voided_at: None }), subscription: None, subscription_details: Some(SubscriptionDetailsData { metadata: None }), subscription_proration_date: None, subtotal: Some(0), subtotal_excluding_tax: Some(0), tax: None, test_clock: None, threshold_reason: None, total: Some(0), total_discount_amounts: Some([DiscountsResourceDiscountAmount { amount: 150, discount: Id(DiscountId("di_1OWRfHGC2bHVgrBXJk0OkEe8")) }]), total_excluding_tax: Some(0), total_tax_amounts: Some([]), transfer_data: None, webhooks_delivered_at: Some(1704754289) }, Invoice { id: InvoiceId("in_1L8zBKGC2bHVgrBXetniCtUq"), account_country: Some("US"), account_name: Some("Zoo Corporation"), account_tax_ids: None, amount_due: Some(0), amount_paid: Some(0), amount_remaining: Some(0), amount_shipping: Some(0), application: None, application_fee_amount: None, attempt_count: Some(0), attempted: Some(false), auto_advance: Some(false), automatic_tax: Some(AutomaticTax { enabled: false, status: None }), billing_reason: Some(Manual), charge: None, collection_method: Some(ChargeAutomatically), created: Some(1654834246), currency: Some(USD), custom_fields: None, customer: Some(Id(CustomerId("cus_Lqg9WtmoSWfZhj"))), customer_address: None, customer_email: Some("[email protected]"), customer_name: Some("test f name test l name"), customer_phone: Some("test-phone"), customer_shipping: None, customer_tax_exempt: Some(None), customer_tax_ids: Some([]), default_payment_method: None, default_source: None, default_tax_rates: Some([]), deleted: false, description: None, discount: None, discounts: Some([]), due_date: None, effective_at: None, ending_balance: None, footer: None, from_invoice: None, hosted_invoice_url: None, invoice_pdf: None, last_finalization_error: None, latest_revision: None, lines: List { data: [], has_more: false, total_count: Some(0), url: "/v1/invoices/in_1L8zBKGC2bHVgrBXetniCtUq/lines" }, livemode: Some(false), metadata: Some({}), next_payment_attempt: None, number: None, on_behalf_of: None, paid: Some(false), paid_out_of_band: Some(false), payment_intent: None, payment_settings: Some(InvoicesPaymentSettings { default_mandate: None, payment_method_options: None, payment_method_types: None }), period_end: Some(1654834246), period_start: Some(1654834246), post_payment_credit_notes_amount: Some(0), pre_payment_credit_notes_amount: Some(0), quote: None, receipt_number: None, rendering_options: None, shipping_cost: None, shipping_details: None, starting_balance: Some(0), statement_descriptor: None, status: Some(Draft), status_transitions: Some(InvoicesStatusTransitions { finalized_at: None, marked_uncollectible_at: None, paid_at: None, voided_at: None }), subscription: None, subscription_details: Some(SubscriptionDetailsData { metadata: None }), subscription_proration_date: None, subtotal: Some(0), subtotal_excluding_tax: Some(0), tax: None, test_clock: None, threshold_reason: None, total: Some(0), total_discount_amounts: Some([]), total_excluding_tax: Some(0), total_tax_amounts: Some([]), transfer_data: None, webhooks_delivered_at: Some(1654834246) }], has_more: false, total_count: None, url: "/v1/invoices" }, params: ListInvoices { collection_method: None, created: None, customer: None, due_date: None, ending_before: None, expand: [], limit: Some(100), starting_after: None, status: Some(Draft), subscription: None } }

[jessfraz]

I think it happens when there is only one item in the list?

[sjmiller609]

I see the same and based on the above tip, this workaround is helping

    let mut result: Vec<Customer> = Default::default();

    let list = Customer::list(&client, &params).await?;
    if list.has_more {
        let mut stream = list.paginate(params).stream(&client);
        while let Some(Ok(next)) = stream.next().await {
            result.push(next);
        }
    } else {
        result = list.data;
    }

EDIT: actually not working 100% of the time

[arlyon]

I am hoping this issue will be nuked once the rewrite lands. If you are able to produce a unit test or even a set of json requests / responses I will try to understand what is wrong.