Internal error: Physical input schema should be the same as the one converted from logical input schema.

[alamb]

Describe the bug

When writing queries on parquet files with field metadata and not stripping that
metadata, DataFusion errors out with the above error.

To Reproduce

Repro

-- First, ensure that parquet metadata is not skipped (it is skipped by default)
> set datafusion.execution.parquet.skip_metadata = false;

SELECT
  'foo' AS name,
  COUNT(
    CASE
      WHEN prev_value = false AND value = TRUE THEN 1
      ELSE NULL
      END
     ) AS count_true_rises
FROM
  (
    SELECT
      value,
      LAG(value) OVER (ORDER BY time ) AS prev_value
    FROM
      'repro.parquet'
);

Results in

Internal error: Physical input schema should be the same as the one converted from logical input schema. Differences: .
This issue was likely caused by a bug in DataFusion's code. Please help us to resolve this by filing a bug report in our issue tracker: https://github.com/apache/datafusion/issues

I made the parquet file available here:

parquet-with-metadata.zip

Here is the code to generate the parquet file (I am not sure how to create parquet files with metadata otherwise):

Details

use std::collections::HashMap;
use std::fs::File;
use std::sync::Arc;
use arrow::array::{BooleanArray, RecordBatch, TimestampNanosecondArray};
use arrow::datatypes::{DataType, Field, Schema, SchemaRef, TimeUnit};

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    // write a parquet file which has a metadata
    let mut metadata = HashMap::new();
    metadata.insert(String::from("year"), String::from("2015"));
    let schema: SchemaRef = Arc::new(Schema::new(vec![
        Field::new("time", DataType::Timestamp(TimeUnit::Nanosecond, None), false),
        Field::new("value", DataType::Boolean, false)
            .with_metadata(metadata),
    ]));

    let time = TimestampNanosecondArray::from(vec![1_420_070_400_000_000_000i64, 1_420_070_401_000_000_000i64]);
    let value = BooleanArray::from(vec![true, false]);
    let batch = RecordBatch::try_new(schema.clone(), vec![
        Arc::new(time),
        Arc::new(value),
    ])?;


    println!("Writing parquet file with metadata repro.parquet...");
    let writer = File::create("repro.parquet")?;
    let mut arrow_writer = parquet::arrow::ArrowWriter::try_new(
        writer,
        schema.clone(),
        None,
    )?;
    arrow_writer.write(&batch)?;
    arrow_writer.close()?;

    Ok(())
}

Note this is all the more confusing because the error lists no differences

...  converted from logical input schema. Differences: . <-- no differences are listed!!!

The difference is the metadata on the value field.

Expected behavior

I expect the query to pass without error

Additional context

No response

[VedantSawant616]

hi @alamb, the root cause seems to be that when datafusion.execution.parquet.skip_metadata = false, the physical plan reads and keeps the field metadata, but the logical plan drops it somewhere during schema reconstruction (likely in Projection or WindowExec). this creates a mismatch that triggers the internal consistency error.

the blank Differences: . output happens because the schema diff code doesn’t know how to display metadata only differences ,a minor formatting bug.

workaround: leave skip_metadata at its default (true).
permanent fix: make DataFusion’s schema handling fully metadata-aware, either by consistently propagating field metadata across all planning stages or normalizing it before comparison. that would eliminate these logical–physical schema mismatches once and for all.

[alamb]

permanent fix: make DataFusion’s schema handling fully metadata-aware, either by consistently propagating field metadata across all planning stages or normalizing it before comparison. that would eliminate these logical–physical schema mismatches once and for all.

Exactly -- I think this ticke tracks the permanant fix

[alamb]

I spent some time today debugging this issue more

• Here is a reproducer: WIP: Fix internal error "Physical input schema should be the same as the one converted from logical input schema." #18412

I added some more debugging and I found

	DataFusion error: Internal error: Internal error planning LogicalPlan::Aggregate. Physical input schema should be the same as the one converted from logical input schema. Differences:
	- field metadata at indexl 1 [prev_value]: (physica) {"metadata_key": "the nonnull_name field"} vs (logical) {}.

You can see the actual input plans here:

[2025-10-31T17:30:07Z DEBUG datafusion::physical_planner] Input plan Projection: table_with_metadata.nonnull_name AS value, lag(table_with_metadata.nonnull_name) ORDER BY [table_with_metadata.ts ASC NULLS LAST] RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW AS prev_value
      WindowAggr: windowExpr=[[lag(table_with_metadata.nonnull_name) ORDER BY [table_with_metadata.ts ASC NULLS LAST] RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW]]
        TableScan: table_with_metadata projection=[ts, nonnull_name]

[2025-10-31T17:30:07Z DEBUG datafusion::physical_planner] Resulting exec: ProjectionExec: expr=[nonnull_name@1 as value, lag(table_with_metadata.nonnull_name) ORDER BY [table_with_metadata.ts ASC NULLS LAST] RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW@2 as prev_value]
      BoundedWindowAggExec: wdw=[lag(table_with_metadata.nonnull_name) ORDER BY [table_with_metadata.ts ASC NULLS LAST] RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW: Field { "lag(table_with_metadata.nonnull_name) ORDER BY [table_with_metadata.ts ASC NULLS LAST] RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW": nullable Utf8, metadata: {"metadata_key": "the nonnull_name field"} }, frame: RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW], mode=[Sorted]
        DataSourceExec: partitions=1, partition_sizes=[1]

So that basically says the logical plan lost the metadata on the field but the physical plan preserves it which is sort of counter intuitive.

However, then I looked at the code that determines the type of window functions in https://github.com/apache/datafusion/blob/4e0596d0d7a7598762e397778110a58bab5363b9/datafusion/expr/src/expr_schema.rs#L662-L661 and I would say "its complicated"