Control over commit time period

[vinnymac]

First of all, very nice project you've all built. Love the documentation too!

I have a large project that I am running this on, but have noticed how resource intensive it is. Currently it has to checkout and analyze so many different commits that I can't go back more than a few weeks, and have to rerun the process again. I'm trying to workaround this by limiting the data to a single week, and then slowly increase that value as it builds up a cache.

Do you think it would be feasible to configure a setting that only runs static analysis for a single commit, every week per project, or even every day? (Bringing down the number of commits to analyze to 52 or 365 commits per project per year)

Open to ideas on how to improve performance, but wanted to ask if this has already been considered before attempting anything.

Thank you

[smoogly]

Hello Vincent!

At the moment the commits are analyzed on a weekly basis, only 1 commit as of Saturday morning each week:
https://github.com/rangle/radius-tracker/blob/a6a975c267701c421c2bdb8b8dced95cb2024b9f/src/lib/cli/timelines/getTimelineForOneRepo.ts#L82C33-L82C33

If you don't mind me asking, what happens if you leave tracker running for a long while? Trying to understand what causes the need to slowly increase the duration.

Long-term, I'd prefer to do analysis only on diff between commits.

[vinnymac]

Hi Arseny,

I did not realize it was already optimized around the time period, that's great! Perhaps I need something even less aggressive, such as on a monthly or quarterly basis 🤔

To provide some more details: I am using this on seven different projects, and when I try to run it on my local computer, for around ~2y period, it eventually just stops. No errors or anything interesting, usually just the last log is something along the lines of [Thread 7 - (Project Name) - 9s] Setting up ts-morph project. I am using an M1 Pro with 32GB of memory.

Reducing it to a single week for a single project, I can run it in under 10m. I also see a single week runs well for my CI service. The codebase I am targeting is very large, and for some projects, even monolothic. Meaning I have to target the same projects for varying subpaths.

Running all seven projects at the same time for a single week on my local does eventually complete, but takes around 2.5 hours.

[smoogly]

I'm not sure if you already tried, could you try running with NODE_OPTIONS="--max-old-space-size=16384 npx radius-tracker timelines <path-to-config>?

[vinnymac]

I'll give that a try, thanks!

Also here is an obfsucated config in case you see anything weird.

const defaultTargets = {
  'design-system': /^@pkg\/design-system/,
  radix: /^@radix-ui\/react-accordion/,
  spring: /^@react-spring\/web/,
};

const myTargets = {
  ...defaultTargets,
  'components': /^@platform\/components/,
};

const isIgnoredFile =
  /((\.(tests?|specs?|stories|story)\.)|(\/(tests?|specs?|stories|story)\/)|(\/node_modules\/)|(\/__mocks__\/)|(\.d\.ts$))/;

const configDefaults = {
  since: new Date('2022-01-01'),
  isIgnoredFile,
  isTargetImport: (imp) => imp.type !== 'cjs-import',
  isValidUsage: () => true,
};

const hostname = 'ci.hostname.com';

module.exports = [
  {
    ...configDefaults,
    repoUrl: `${hostname}project/repository1`,
    displayName: 'Project1',
    isTargetModuleOrPath: myTargets,
    subprojectPath: 'somePath',
    tsconfigPath: '../tsconfig.json',
  },

  {
    ...configDefaults,
    repoUrl: `${hostname}project/repository1`,
    displayName: 'Project1',
    isTargetModuleOrPath: myTargets,
    subprojectPath: 'someOtherPath',
    tsconfigPath: './tsconfig.json',
  },

  {
    ...configDefaults,
    repoUrl: `${hostname}project/repository2`,
    displayName: 'Project2',
    isTargetModuleOrPath: {
      'design-system': /^@pkg\/design-system/,
      hooks: /^@pkg\/hooks/,
      utils: /^@pkg\/utils/,
    },
    subprojectPath: '/',
    tsconfigPath: './packages/tsconfig/base.json',
    since: new Date('2023-09-08'),
  },

  {
    ...configDefaults,
    repoUrl: `${hostname}project/repository3`,
    displayName: 'Project3',
    isTargetModuleOrPath: defaultTargets,
    subprojectPath: 'somePath',
    tsconfigPath: '../tsconfig.json',
  },

  {
    ...configDefaults,
    repoUrl: `${hostname}project/repository4`,
    displayName: 'Project4',
    isTargetModuleOrPath: defaultTargets,
    subprojectPath: '/',
    tsconfigPath: './tsconfig.json',
    since: new Date('2022-07-20'),
  },

  {
    ...configDefaults,
    repoUrl: `${hostname}project/repository5`,
    displayName: 'Project5',
    isTargetModuleOrPath: defaultTargets,
    subprojectPath: 'src',
    tsconfigPath: '../tsconfig.json',
  },

  {
    ...configDefaults,
    repoUrl: `${hostname}project/repository6`,
    displayName: 'Project6',
    isTargetModuleOrPath: defaultTargets,
    subprojectPath: 'somePath',
    tsconfigPath: './tsconfig.json',
  },
];

[smoogly]

Config looks okay to me, I don't immediately see anything suspicious.

[vinnymac]

I've been running it with the node setting for most of the day on my local computer, still seems to be chugging along slowly.

Opened up activity monitor though, and saw a wild amount of swap. Maybe each child process is now trying to use as much memory as it needs?

I am thinking I might try tweaking the weekly commits to be quarterly and see if that helps build a report in a shorter time period.