Memory Footprint Calculation

• Default PartUploadSize = 50 Mb
• Memory Requirements:
  • For 2 core /4 vcpu
  • (4 + 3) x 50
  • 350 Mb
• Memory Calculation:
  • csv buffer size = 50 Mb
  • gzip buffer size = 50 Mb
  • gzip Upload queue = 4 x gzipped buffers + use 1 temporary buffer
  • goroutines = 4 for gzipping + 4 for uploading.
  • sql batch = 4096 x number of columns x string size /time.Time size
  • Memory = csv[50Mb] + gzip[250Mb] + sqlBatch[~1Mb] + Inbuilt buffers/vars[?] + goroutines[32K+?]
  • Memory = 301Mb + Inbuilt buffers/vars[?] + goroutines[32K+?]
  • Memory => 350Mb should be enough