natalie/indefero
1
0
Fork 0
Code Issues Pull Requests Releases Wiki Activity

Natalie Adams 2021-04-08 20:34:34 +00:00
parent 4be11c86cc
commit 3d960842a7
1 changed files with 12 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

12
GitLargeRepositories.md Normal file

@ -0,0 +1,12 @@
# Description
Limitations with large git repositories (Kernel size)
The memory usage is not anymore a problem but we need to deal with some performance problems. See issue 103 for details.
See issue 93 for some context information.
With large git repositories, it is consuming a lot of memory to know at which commit a file has been created. This is used normally for each file in the tree view. To avoid completely crashing the system, indefero is only going back to the latest 5000 commits when looking at historical data to find the origin of a file.
What should be done is that we should build over the time a database cache with the details. Maybe using only the sha1 for reference and simply storing in a compressed row the details to save space. That way the cache is built over the time and at the end we have all the data. An auxiliary script could be used to regenerate this cache.
Note, **a large repository like the Linux kernel will require about 35MB**. This is due to the ls-tree git command which basically is listing the complete git tree and we just need a part of it to display the current folder. One way to avoid that would be to pipe the output to grep, but that would require the availability of grep. I we consider that git is anyway a posix tool, we can consider that to be ok.