QC findings - Pilot 5 vs original CDISC data

[camillacalmasini]

QC findings from pilot 5 vs original CDISC data comparisons.

Updated based on PR #103: qc_findings_p5_vs_orig_cdisc.html

---

[OLD] qc_findings_p5_vs_orig_cdisc.html

[bms63]

oh this looks like the epoch issue. can you check to see if the rds files have 2014 or 2024?

[camillacalmasini]

@bms63 yes! just checked and the RDS file has 2014 for both those variables (so same as the JSON)

Also was wondering - what is the epoch issue?

[robertdevine]

@camillacalmasini - Linux, like most operating systems, uses an epoch — the current upper limit for systems tracking time and use 32-bit signed integers is 03:14:07 UTC on January 19, 2038. This is the moment when Unix time on 32-bit systems will overflow into negative values. Some refer to this looming issue as the "epoch issue" or "Y2038". It's a very real concern especially in medical research. Impacts might include system crashes, data corruption, loss of functionality, global scope - very serious issue.

[robertdevine]

@bms63 @parmsam-pfizer @camillacalmasini @RConsortium/pilot5 - as noted at the general meeting, the p21 community validation results for p5 does include detailed issues. Further details can be shared at Friday's meeting. Nothing too major but worth reporting for comparison with p3 validation results.

[bms63]

@bms63 yes! just checked and the RDS file has 2014 for both those variables (so same as the JSON)

Also was wondering - what is the epoch issue?

The programming language SAS uses a epoch start of 1960 whereas R uses the unix epoch which is 1970. Hence the 10 year gap of 2014 and 2024. It is interesting this has occurred. I will dig in more today.

[bms63]

@bms63 @parmsam-pfizer @camillacalmasini @RConsortium/pilot5 - as noted at the general meeting, the p21 community validation results for p5 does include detailed issues. Further details can be shared at Friday's meeting. Nothing too major but worth reporting for comparison with p3 validation results.

@robertdevine can you share the report here or in slack please

[robertdevine]

p3 pinnacle validation summary first - [back from p3 days with Joel, Lei, Kangjie and the p3 team]

pinnacle21-p3-report-2023-05-19T10-33-03-557.xlsx

p5 pinnacle validation summary -

[bms63]

@camillacalmasini regarding the 2024 issues you spotted - where are you pulling that data from?

I don't see it in our pilot3 data folder

[bms63]

For labels - I think the mutate_if is stripping off the labels - I've observed that some dplyr functions do this unfortunately.

[bms63]

Oh I think the CDISC Pilot Factory data is incorrect. That is the one that has the 2024. Can we use the pilot3 data as the compare instead. phew my heart almost stopped :)

[bms63]

Regarding the decimals issue. Could you try datasetjson::read_dataset_json("adsl.json", decimals_as_floats = TRUE) to see if that fixes our issues?

[bms63]

@mstackhouse and @nicholas-masel

We are comparing the datasetjson back to pilot3 data.

Two Questions on datasetjson behavior.

We are noticing that some variables that are written out as numeric, but when read back in are converted to integer. Is there something happening in the package that is doing or a setting we need to adjust?

We are also noticing truncation/rounding in some of the variables, which we have not done in our programs. Is there something happening in the package that is doing or a setting we need to adjust?

Can you advise us if possible? We were trying to send to FDA by end of next week.