Skip to content

Pathway's connector issue in Jsonlines #53

Open
@Boburmirzo

Description

@Boburmirzo

Flatten data structures in a Jsonline file can not be mapped to structured schemas automatically.

For example, list_price and current_price mapping to the scheme is failing:

{"position": 1, "link": "https://www.amazon.com/Avia-Resistant-Restaurant-Service-Sneakers/dp/B0BJY1FN8F", "asin": "B0BJXSKK9L", "is_lightning_deal": false, "deal_type": "BEST_DEAL", "is_prime_exclusive": false, "starts_at": "2023-08-14T07:00:08.270Z", "ends_at": "2023-08-21T06:45:08.270Z", "type": "multi_item", "title": "Avia Anchor SR Mesh Slip On Black Non Slip Shoes for Women, Comfortable Water Resistant Womens Food Service Sneakers - Black, Blue, or White Med or Wide Restaurant, Slip Resistant Work Shoes Women", "image": "https://m.media-amazon.com/images/I/3195IpEIRpL._SY500_.jpg", "deal_price": 39.98, "list_price": {"value": 59.98, "currency": "USD", "symbol": "$", "raw": "59.98", "name": "List Price"}, "current_price": {"value": 39.98, "currency": "USD", "symbol": "$", "raw": "39.98", "name": "Current Price"}, "merchant_name": "Galaxy Active", "free_shipping": false, "is_prime": true, "is_map": false, "deal_id": "34f3da97", "seller_id": "A3GMJQO0HY62S", "description": "Avia Anchor SR Mesh Slip On Black Non Slip Shoes for Women, Comfortable Water Resistant Womens Food Service Sneakers - Black, Blue, or White Med or Wide Restaurant, Slip Resistant Work Shoes Women", "rating": 4.16, "ratings_total": 1148, "old_price": 59.98, "currency": "USD"}

In this data schema:

class Price(pw.Schema):
    value: float
    currency: str
    symbol: str
    raw: str
    name: str


class DealResult(pw.Schema):
    position: int
    link: str
    asin: str
    is_lightning_deal: bool
    deal_type: str
    is_prime_exclusive: bool
    starts_at: str
    ends_at: str
    type: str
    title: str
    image: str
    deal_price: Price
    list_price: Price
    current_price: Price
    merchant_name: str
    free_shipping: bool
    is_prime: bool
    is_map: bool

The error I got:

Read data parsed unsuccessfully. field deal_price with no JsonPointer path specified is absent in

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions