Joshua Redstone | 3 Feb 15:20 2012

Git performance results on a large repository

Hi Git folks,

We (Facebook) have been investigating source control systems to meet our
growing needs.  We already use git fairly widely, but have noticed it
getting slower as we grow, and we want to make sure we have a good story
going forward.  We're debating how to proceed and would like to solicit
people's thoughts.

To better understand git scalability, I've built up a large, synthetic
repository and measured a few git operations on it.  I summarize the
results here.

The test repo has 4 million commits, linear history and about 1.3 million
files.  The size of the .git directory is about 15GB, and has been
repacked with 'git repack -a -d -f --max-pack-size=10g --depth=100
--window=250'.  This repack took about 2 days on a beefy machine (I.e.,
lots of ram and flash).  The size of the index file is 191 MB. I can share
the script that generated it if people are interested - It basically picks
2-5 files, modifies a line or two and adds a few lines at the end
consisting of random dictionary words, occasionally creates a new file,
commits all the modifications and repeats.

I timed a few common operations with both a warm OS file cache and a cold
cache.  i.e., I did a 'echo 3 | tee /proc/sys/vm/drop_caches' and then did
the operation in question a few times (first timing is the cold timing,
the next few are the warm timings).  The following results are on a server
with average hard drive (I.e., not flash)  and > 10GB of ram.

'git status' :   39 minutes cold, and 24 seconds warm.

(Continue reading)

Ævar Arnfjörð Bjarmason | 3 Feb 15:56 2012
Picon

Re: Git performance results on a large repository

On Fri, Feb 3, 2012 at 15:20, Joshua Redstone <joshua.redstone <at> fb.com> wrote:

> We (Facebook) have been investigating source control systems to meet our
> growing needs.  We already use git fairly widely, but have noticed it
> getting slower as we grow, and we want to make sure we have a good story
> going forward.  We're debating how to proceed and would like to solicit
> people's thoughts.

Where I work we also have a relatively large Git repository. Around
30k files, a couple of hundred thousand commits, clone size around
half a GB.

You haven't supplied background info on this but it really seems to me
like your testcase is converting something like a humongous Perforce
repository directly to Git.

While you /can/ do this it's not a good idea, you should split up
repositories at the boundaries code or data doesn't directly cross
over, e.g. there's no reason why you need HipHop PHP in the same
repository as Cassandra or the Facebook chat system, is there?

While Git could better with large repositories (in particular applying
commits in interactive rebase seems to be to slow down on bigger
repositories) there's only so much you can do about stat-ing 1.3
million files.

A structure that would make more sense would be to split up that giant
repository into a lot of other repositories, most of them probably
have no direct dependencies on other components, but even those that
do can sometimes just use some other repository as a submodule.
(Continue reading)

Joshua Redstone | 3 Feb 18:00 2012

Re: Git performance results on a large repository

Hi Ævar,


Thanks for the comments.  I've included a bunch more info on the test repo
below.  It is based on a growth model of two of our current repositories
(I.e., it's not a perforce import). We already have some of the easily
separable projects in separate repositories, like HPHP.   If we could
split our largest repos into multiple ones, that would help the scaling
issue.  However, the code in those repos is rather interdependent and we
believe it'd hurt more than help to split it up, at least for the
medium-term future.  We derive a fair amount of benefit from the code
sharing and keeping things together in a single repo, so it's not clear
when it'd make sense to get more aggressive splitting things up.

Some more information on the test repository:   The working directory is
9.5 GB, the median file size is 2 KB.  The average depth of a directory
(counting the number of '/'s) is 3.6 levels and the average depth of a
file is 4.6.  More detailed histograms of the repository composition is
below:

------------------------

Histogram of depth of every directory in the repo (dirs=`find . -type d` ;
(for dir in $dirs; do t=${dir//[^\/]/}; echo ${#t} ; done) |
~/tmp/histo.py)
* The .git directory itself has only 161 files, so although included,
doesn't affect the numbers significantly)

[0.0 - 1.3): 271
[1.3 - 2.6): 9966
(Continue reading)

Sam Vilain | 3 Feb 23:40 2012
Picon

Re: Git performance results on a large repository

Joshua,

You have an interesting use case.

If I were you I'd consider investigating the git fast-import protocol. 
It has become bi–directional, and is essentially socket access to a git 
repository with read and transactional update capability.  With a few 
more commands implemented, it may even be capable of providing all 
functionality required for command–line git use.

It is already possible that the ".git" directory can be a file: this 
case is used for submodules in git 1.7.8 and higher.  For this use case, 
there would be an extra field to the ".git" file which is created.  It 
would indicate the hostname (and port) to connect its internal 
'fast-import' stream to.  'clone' would consist of creating this file, 
and then getting the server to stream the objects from its pack to the 
client.

With the hard–working part of git on the other end of a network service, 
you could back it by a re–implementation of git which is written to be 
distributed in Hadoop.  There are at least two similar implementations 
of git that are like this: one for cassandra which was written by github 
as a research project, and Google's implementation on top of their 
BigTable/GFS/whatever.  As the git object storage model is write–only 
and content–addressed, it should git this kind of scaling well.

There have also been designs at various times for sparse check–outs; ie 
check–outs where you don't check out the root of the repository but a 
sub–tree.

(Continue reading)

Sam Vilain | 3 Feb 23:57 2012
Picon

Re: Git performance results on a large repository

On 2/3/12 2:40 PM, Sam Vilain wrote:
> As the git object storage model is write–only and content–addressed,
> it should git this kind of scaling well.
             ^^^

Could have sworn I typed 'suit' there.  My fingers have auto–correct ;-)

Sam

Nguyen Thai Ngoc Duy | 7 Feb 02:19 2012
Picon

Re: Git performance results on a large repository

On Sat, Feb 4, 2012 at 5:40 AM, Sam Vilain <sam <at> vilain.net> wrote:
> There have also been designs at various times for sparse check–outs; ie
> check–outs where you don't check out the root of the repository but a
> sub–tree.

There is a sparse checkout feature in git (hopefully from one of the
designs you mentioned) and it can checkout subtrees. The only problem
in this case is it maintains full index. So it only solves half of the
problem (stat calls), reading/writing large index just slows
everything down.
--

-- 
Duy
Matt Graham | 4 Feb 00:05 2012
Picon

Re: Git performance results on a large repository

Hi Josh,

On Fri, Feb 3, 2012 at 17:00, Joshua Redstone <joshua.redstone <at> fb.com> wrote:
> Thanks for the comments.  I've included a bunch more info on the test repo
> below.  It is based on a growth model of two of our current repositories
> (I.e., it's not a perforce import). We already have some of the easily
> separable projects in separate repositories, like HPHP.   If we could
> split our largest repos into multiple ones, that would help the scaling
> issue.  However, the code in those repos is rather interdependent and we
> believe it'd hurt more than help to split it up, at least for the
> medium-term future.  We derive a fair amount of benefit from the code
> sharing and keeping things together in a single repo, so it's not clear
> when it'd make sense to get more aggressive splitting things up.
>
> Some more information on the test repository:   The working directory is
> 9.5 GB, the median file size is 2 KB.  The average depth of a directory
> (counting the number of '/'s) is 3.6 levels and the average depth of a
> file is 4.6.  More detailed histograms of the repository composition is
> below:

Do you have a histogram of the types of files in the repo?
And as suggested earlier, is svn working for you now because it allows
sparse checkout?  I imagine the stats for svn on the full repo would
be comparable or worse to what you measured with git?
Evgeny Sazhin | 4 Feb 02:25 2012
Picon

Re: Git performance results on a large repository


On Feb 3, 2012, at 9:56 AM, Ævar Arnfjörð Bjarmason wrote:

> On Fri, Feb 3, 2012 at 15:20, Joshua Redstone <joshua.redstone <at> fb.com> wrote:
> 
>> We (Facebook) have been investigating source control systems to meet our
>> growing needs.  We already use git fairly widely, but have noticed it
>> getting slower as we grow, and we want to make sure we have a good story
>> going forward.  We're debating how to proceed and would like to solicit
>> people's thoughts.
> 
> Where I work we also have a relatively large Git repository. Around
> 30k files, a couple of hundred thousand commits, clone size around
> half a GB.
> 
> You haven't supplied background info on this but it really seems to me
> like your testcase is converting something like a humongous Perforce
> repository directly to Git.
> 
> While you /can/ do this it's not a good idea, you should split up
> repositories at the boundaries code or data doesn't directly cross
> over, e.g. there's no reason why you need HipHop PHP in the same
> repository as Cassandra or the Facebook chat system, is there?
> 
> While Git could better with large repositories (in particular applying
> commits in interactive rebase seems to be to slow down on bigger
> repositories) there's only so much you can do about stat-ing 1.3
> million files.
> 
> A structure that would make more sense would be to split up that giant
(Continue reading)

Chris Lee | 4 Feb 00:35 2012
Picon

Re: Git performance results on a large repository

On Fri, Feb 3, 2012 at 6:20 AM, Joshua Redstone <joshua.redstone <at> fb.com> wrote:
> [snip]
>
> The git performance we observed here is too slow for our needs.  So the
> question becomes, if we want to keep using git going forward, what's the
> best way to improve performance.  It seems clear we'll probably need some
> specialized servers (e.g., to perform git-blame quickly) and maybe
> specialized file system integration to detect what files have changed in a
> working tree.

Have you considered upgrading all of engineering to SSDs? 200+GB SSDs
are under $400USD nowadays.

-clee
Zeki Mokhtarzada | 4 Feb 01:01 2012

Re: Git performance results on a large repository


> The test repo has 4 million commits, linear history and about 1.3 million
> files.  The size of the .git directory is about 15GB, and has been
> repacked with 'git repack -a -d -f --max-pack-size=10g --depth=100
> --window=250'.  This repack took about 2 days on a beefy machine (I.e.,
> lots of ram and flash).  The size of the index file is 191 MB. I can share

Are you willing to give up all or part of your history in your working
repository?  I've heard of larger projects starting from scratch (i.e. copy all
of your files into a brand new repo.)  You can keep your old repo around for
archival purposes.  Also, how much of your repo is code, versus static assets. 
You could move all of your static assets (images, css, maybe some js?) into
another repo, and then merge the two repo's together at build time if you
absolutely need them deployed together.

Here are a couple strategies for doing a partial truncate:

http://stackoverflow.com/questions/4515580/how-do-i-remove-the-old-history-from-a-git-repository
http://bogdan.org.ua/2011/03/28/how-to-truncate-git-history-sample-script-included.html

-Zeki

Joey Hess | 4 Feb 06:07 2012
Picon

Re: Git performance results on a large repository

Joshua Redstone wrote:
> The test repo has 4 million commits, linear history and about 1.3 million
> files.

Have you tried separating these two factors, to see how badly each is
affecting performance?

If the number of commits is the problem (seems likely for git blame at
least), a shallow clone would avoid that overhead.

I think that git often writes .git/index inneficiently when staging
files (though your `git add` is pretty fast) and committing. It rewrites
the whole file to .git/index.lck and the renames it over .git/index at
the end. I have code that keeps a journal of changes to avoid rewriting
the index repeatedly, but it's application specific. Fixing git to write
the index more intelligently is something I'd like to see.

Hint for git status: `git status .` in a smaller subdirectory will be much
faster than the default that stats everything.

--

-- 
see shy jo
Nguyen Thai Ngoc Duy | 4 Feb 07:53 2012
Picon

Re: Git performance results on a large repository

On Fri, Feb 3, 2012 at 9:20 PM, Joshua Redstone <joshua.redstone <at> fb.com> wrote:
> I timed a few common operations with both a warm OS file cache and a cold
> cache.  i.e., I did a 'echo 3 | tee /proc/sys/vm/drop_caches' and then did
> the operation in question a few times (first timing is the cold timing,
> the next few are the warm timings).  The following results are on a server
> with average hard drive (I.e., not flash)  and > 10GB of ram.
>
> 'git status' :   39 minutes cold, and 24 seconds warm.
>
> 'git blame':   44 minutes cold, 11 minutes warm.
>
> 'git add' (appending a few chars to the end of a file and adding it):   7
> seconds cold and 5 seconds warm.
>
> 'git commit -m "foo bar3" --no-verify --untracked-files=no --quiet
> --no-status':  41 minutes cold, 20 seconds warm.  I also hacked a version
> of git to remove the three or four places where 'git commit' stats every
> file in the repo, and this dropped the times to 30 minutes cold and 8
> seconds warm.

Have you tried "git update-index --assume-unchaged"? That should
reduce mass lstat() and hopefully improve the above numbers. The
interface is not exactly easy-to-use, but if it has significant gain,
then we can try to improve UI.

On the index size issue, ideally we should make minimum writes to
index instead of rewriting 191 MB index. An improvement we could do
now is to compress it, reduce disk footprint, thus disk I/O. If you
compress the index with gzip, how big is it?
--

-- 
(Continue reading)

Joshua Redstone | 4 Feb 19:05 2012

RE: Git performance results on a large repository

[ wanted to reply to my initial msg, but wasn't subscribed to the list at time of mailing, so replying to most
recent post instead ]

Thanks to everyone for the questions and suggestions.  I'll try to respond here.  One high-level
clarification - this synthetic repo for which I've reported perf times is representative of where we
think we'll be in the future.  Git is slow but marginally acceptable for today.  We want to start planning now
for any big changes we need to make going forward.

Evgeny Sazhin, Slinky and Ævar Arnfjörð Bjarmason suggested splitting up the repo into multiple,
smaller repos.  I indicated before that we have a lot of cross-dependencies.  Our largest repo by number of
files and commits is the repo containing the front-end server.  It is a large code base in which the tight
integration of various components results in many of the cross dependencies.  We are working slowly to
split things up more, for example into services, but that is a long-term process.

To get a bit abstract for a moment, in an ideal world, it doesn't seem like performance constraints of a
source-control-system should dictate how we choose to structure our code.  Ideally, seems like we should
be able to choose to structure our code in whatever way we feel maximizes developer productivity.  If
development and code/release management seem easier in a single repo, than why not make an SCM that can
handle it?  This is one reason I've been leaning towards figuring out an SCM approach that can work well with
our current practices rather than changing them as a prerequisite for good SCM performance.

Sam Vilain:  Thanks for the pointer, i didn't realize that fast-import was bi-directional.  I used it for
generating the synthetic repo.  Will look into using it the other way around.  Though that still won't speed
up things like git-blame, presumably?  The sparse-checkout issue you mention is a good one.  There is a good
question of how to support quick checkout, branch switching, clone, push and so forth.  I'll look into the
approaches you suggest.  One consideration is coming up with a high-leverage approach - i.e. not doing
heavy dev work if we can avoid it.  On the other hand, it would be nice if we (including the entire community :)
) improve git in areas that others that share similar issues benefit from as well.

Matt Graham:  I don't have file stats at the moment.  It's mostly code files, with a few larger data files here
(Continue reading)

Nguyen Thai Ngoc Duy | 5 Feb 04:47 2012
Picon

Re: Git performance results on a large repository

On Sun, Feb 5, 2012 at 1:05 AM, Joshua Redstone <joshua.redstone <at> fb.com> wrote:
> It's also conceivable that, if there were an external interface in git to attach other
> systems to efficiently report which files have changed (e.g., via file-system integration),
> it's possible that we could omit managing the index in many cases.
> I know that would be a big change, but the benefits are intriguing.

The "interface to report which files have changed" is exactly "git
update-index --[no-]assume-unchanged" is for. Have a look at the man
page. Basically you can mark every file "unchanged" in the beginning
and git won't bother lstat() them. What files you change, you have to
explicitly run "git update-index --no-assume-unchanged" to tell git.

Someone on HN suggested making assume-unchanged files read-only to
avoid 90% accidentally changing a file without telling git. When
assume-unchanged bit is cleared, the file is made read-write again.
--

-- 
Duy
Joey Hess | 6 Feb 16:40 2012
Picon

Re: Git performance results on a large repository

Nguyen Thai Ngoc Duy wrote:
> The "interface to report which files have changed" is exactly "git
> update-index --[no-]assume-unchanged" is for. Have a look at the man
> page. Basically you can mark every file "unchanged" in the beginning
> and git won't bother lstat() them. What files you change, you have to
> explicitly run "git update-index --no-assume-unchanged" to tell git.
> 
> Someone on HN suggested making assume-unchanged files read-only to
> avoid 90% accidentally changing a file without telling git. When
> assume-unchanged bit is cleared, the file is made read-write again.

That made me think about using assume-unchanged with git-annex since it
already has read-only files. 

But, here's what seems a misfeature... If an assume-unstaged file has
modifications and I git add it, nothing happens. To stage a change, I
have to explicitly git update-index --no-assume-unchanged and only then
git add, and then I need to remember to reset the assume-unstaged bit
when I'm done working on that file for now. Compare with running git mv
on the same file, which does stage the move despite assume-unstaged. (So
does git rm.)

--

-- 
see shy jo
Nguyen Thai Ngoc Duy | 7 Feb 14:43 2012
Picon

Re: Git performance results on a large repository

On Mon, Feb 6, 2012 at 10:40 PM, Joey Hess <joey <at> kitenet.net> wrote:
>> Someone on HN suggested making assume-unchanged files read-only to
>> avoid 90% accidentally changing a file without telling git. When
>> assume-unchanged bit is cleared, the file is made read-write again.
>
> That made me think about using assume-unchanged with git-annex since it
> already has read-only files.
>
> But, here's what seems a misfeature...

because, well.. assume-unchanged was designed to avoid stat() and
nothing else. We are basing a new feature on top of it.

> If an assume-unstaged file has
> modifications and I git add it, nothing happens. To stage a change, I
> have to explicitly git update-index --no-assume-unchanged and only then
> git add, and then I need to remember to reset the assume-unstaged bit
> when I'm done working on that file for now. Compare with running git mv
> on the same file, which does stage the move despite assume-unstaged. (So
> does git rm.)

This is normal in the lock-based "checkout/edit/checkin" model. mv/rm
operates on directory content, which is not "locked - no edit allowed"
(in our case --assume-unchanged) in git. But lock-based model does not
map really well to git anyway. It does not have the index (which may
make things more complicated). Also at index level, git does not
really understand directories.

I think we could add a protection layer to index, where any changes
(including removal) to an index entry are only allowed if the entry is
(Continue reading)

Joshua Redstone | 9 Feb 22:06 2012

Re: Git performance results on a large repository

Hi Nguyen,
I like the notion of using --assume-unchanged to cut down the set of
things that git considers may have changed.
It seems to me that there may still be situations that require operations
on the order of the # of files in the repo and hence may still be slow.
Following is a list of potential candidates that occur to me.

1. Switching branches, especially if you switch to an old branch.
Sometimes I've seen branch switching taking a long time for what I thought
was close to where HEAD was.

2. Interactive rebase in which you reorder a few commits close to the tip
of the branch (I observed this taking a long time, but haven't profiled it
yet).  I include here other types of cherry-picking of commits.

3. Any working directory operations that fail part-way through and make
you want to do a 'git reset --hard' or at least a full 'git-status'.  That
is, when you have reason to believe that files with 'assume-unchange' may
have accidentally changed.

4. Operations that require rewriting the index - I think git-add is one?

If the working-tree representation is the full set of all files
materialized on disk and it's the same as the representation of files
changed, then I'm not sure how to avoid some of these without playing file
system games or using wrapper scripts.

What do you (or others) think?

Josh
(Continue reading)

Nguyen Thai Ngoc Duy | 10 Feb 08:12 2012
Picon

Re: Git performance results on a large repository

On Fri, Feb 10, 2012 at 4:06 AM, Joshua Redstone <joshua.redstone <at> fb.com> wrote:
> Hi Nguyen,
> I like the notion of using --assume-unchanged to cut down the set of
> things that git considers may have changed.
> It seems to me that there may still be situations that require operations
> on the order of the # of files in the repo and hence may still be slow.
> Following is a list of potential candidates that occur to me.
>
> 1. Switching branches, especially if you switch to an old branch.
> Sometimes I've seen branch switching taking a long time for what I thought
> was close to where HEAD was.
>
> 2. Interactive rebase in which you reorder a few commits close to the tip
> of the branch (I observed this taking a long time, but haven't profiled it
> yet).  I include here other types of cherry-picking of commits.
>
> 3. Any working directory operations that fail part-way through and make
> you want to do a 'git reset --hard' or at least a full 'git-status'.  That
> is, when you have reason to believe that files with 'assume-unchange' may
> have accidentally changed.

All these involve unpack_trees(), which is full tree operation. The
bigger your worktree is, the slower it is. Another good reason to
split unrelated parts into separate repositories.

> 4. Operations that require rewriting the index - I think git-add is one?
>
> If the working-tree representation is the full set of all files
> materialized on disk and it's the same as the representation of files
> changed, then I'm not sure how to avoid some of these without playing file
(Continue reading)

Christian Couder | 10 Feb 10:39 2012
Picon

Re: Git performance results on a large repository

Hi,

On Fri, Feb 10, 2012 at 8:12 AM, Nguyen Thai Ngoc Duy <pclouds <at> gmail.com> wrote:
>
> All these involve unpack_trees(), which is full tree operation. The
> bigger your worktree is, the slower it is. Another good reason to
> split unrelated parts into separate repositories.

Maybe having different "views" would be enough to make a smaller
worktree and history, so that things are much faster for a developper?

(I already suggested "views" based on "git replace" in this thread:
http://thread.gmane.org/gmane.comp.version-control.git/177146/focus=177639)

Best regards,
Christian.
Nguyen Thai Ngoc Duy | 10 Feb 13:24 2012
Picon

Re: Git performance results on a large repository

On Fri, Feb 10, 2012 at 4:39 PM, Christian Couder
<christian.couder <at> gmail.com> wrote:
> Hi,
>
> On Fri, Feb 10, 2012 at 8:12 AM, Nguyen Thai Ngoc Duy <pclouds <at> gmail.com> wrote:
>>
>> All these involve unpack_trees(), which is full tree operation. The
>> bigger your worktree is, the slower it is. Another good reason to
>> split unrelated parts into separate repositories.
>
> Maybe having different "views" would be enough to make a smaller
> worktree and history, so that things are much faster for a developper?
>
> (I already suggested "views" based on "git replace" in this thread:
> http://thread.gmane.org/gmane.comp.version-control.git/177146/focus=177639)

That's more or less what I did with the subtree clone series [1] and
ended up doing narrow clone [2]. The only difference between the two
are how to handle partial worktree/index. The former uses git-replace
to seal any holes, the latter tackles at pathspec level and is
generally more elegant.

The worktree part from that work should be usable in full clone too. I
am reviving the series and going to repost it soon. Have a look [3] if
you are interested.

[1] http://thread.gmane.org/gmane.comp.version-control.git/152347
[2] http://thread.gmane.org/gmane.comp.version-control.git/155427
[3] https://github.com/pclouds/git/commits/narrow-clone
--

-- 
(Continue reading)

David Mohs | 6 Feb 08:10 2012

Re: Git performance results on a large repository

Joshua Redstone <joshua.redstone <at> fb.com> writes:

> To get a bit abstract for a moment, in an ideal world, it doesn't seem like
> performance constraints of a source-control-system should dictate how we
> choose to structure our code. Ideally, seems like we should be able to choose
> to structure our code in whatever way we feel maximizes developer
> productivity. If development and code/release management seem easier in a
> single repo, than why not make an SCM that can handle it? This is one reason
> I've been leaning towards figuring out an SCM approach that can work well with
> our current practices rather than changing them as a prerequisite for good SCM
> performance.

I certainly agree with this perspective---that our tools should support our
use cases and not the other way around. However, I'd like you to consider that
the size of this hypothetical repository might be giving you some useful
information on the health of the code it contains. You might consider creating
separate repositories simply to promote good modularization. It would involve
some up-front effort and certainly some pain, but this work itself might be
beneficial to your codebase without even considering the improved performance
of the version control system.

My concern here is that it may be extremely difficult to make a single piece
of software scale for a project that can grow arbitrarily large. You may add
some great performance improvements to git to then find that your bottleneck
is the filesystem. That would enlarge the scope of your work and would likely
make the project more difficult to manage.

If you are able to prove me wrong, the entire software community will benefit
from this work. However, before you embark upon a technical solution to your
problem, I would urge you to consider the possible benefits of a non-technical
(Continue reading)

Matt Graham | 6 Feb 17:23 2012
Picon

Re: Git performance results on a large repository

On Sat, Feb 4, 2012 at 18:05, Joshua Redstone <joshua.redstone <at> fb.com> wrote:
> [ wanted to reply to my initial msg, but wasn't subscribed to the list at time of mailing, so replying to most
recent post instead ]
>
> Matt Graham:  I don't have file stats at the moment.  It's mostly code files, with a few larger data files
here and there.    We also don't do sparse checkouts, primarily because most people use git (whether on
top of SVN or not), which doesn't support it.

This doesn't help your original goal, but while you're still working
with git-svn, you can do sparse checkouts. Use --ignore-paths when you
do the original clone and it will filter out directories that are not
of interest.

We used this at Etsy to keep git svn checkouts manageable when we
still had a gigantic svn repo.  You've repeatedly said you don't want
to reorganize your repos but you may find this writeup informative
about how Etsy migrated to git (which included a health amount of repo
manipuation).
http://codeascraft.etsy.com/2011/12/02/moving-from-svn-to-git-in-1000-easy-steps/
Joshua Redstone | 6 Feb 21:50 2012

Re: Git performance results on a large repository

Hi all,

Nguyen, thanks for pointing out the assume-unchanged part.  That, and
especially the suggestion of making assume-unchanged files read-only is
interesting.  It does require explicit specification of what's changed.
Hmm, I wonder if that could be a candidate API through which something
like  CoW file system could let git know what's changed.  Btw, I think you
asked earlier, but the index compresses from 158MB to 58MB - keep in mind
that the majority of file names in the repo are synthetic, so take with
big grain of salt.

Joey, it sounds like it might be good if git-mv and other commands where
consistent in how they treat the assume-unchanged bit.

David Mohs:  Yeah, it's an open question whether we'd be better off
somehow forcing the repos the split apart more.  As a practical matter,
what may happen is that we incrementally solve our problem by addressing
pain points as they come up (e.g., git status being slow).  One risk with
that approach is that it leads to overly short-term thinking and we get
stuck in a local minimum.  I totally agree that good modularization and
code health is valuable.  I think sometimes that getting to good
modularization does involve some technical work - like maybe moving
functionality between systems so they split apart better, having some
notion of versioning and dependency and managing that, and so forth.    I
suppose the other aspect to the problem is that we want to make sure we
have a good source-control story even if the modularization effort takes a
long time - we'd rather not end up in a race between long-term
modularization efforts and source-control performance going south too
fast.  I suppose this comes back to the desire that modularization not be
a prerequisite for good source-control performance.  Oh, and in case I
(Continue reading)

Greg Troxel | 6 Feb 22:07 2012
Picon

Re: Git performance results on a large repository


Joshua Redstone <joshua.redstone <at> fb.com> writes:

> Greg,  'git commit' does some stat'ing of every file, even with all those
> flags - for example, I think one instance it does it is, just in case any
> pre-commit hooks touched any files, it re-stats everything.

That seems ripe for skipping.  If I understand correctly, what's being
committed is the index, not the working dir contents, so it would follow
that a pre-commit hook changing a file is a bug.

> Regarding the perf numbers, I ran it on a beefy linux box.  Have you
> tried doing your measurements with the drop_caches trick to make sure
> the file cache is totally cold?

On NetBSD, there should be a clear cache command for just this reason,
but I'm not sure there is.  So I did

  sysctl -w kern.maxvnodes=1000 # seemed to take a while
  ls -lR # wait for those to be faulted in
  sysctl -w kern.maxvnodes=500000

Then, git status on my repo churned the disk for a long time.

  real    2m7.121s
  user    0m3.086s
  sys     0m7.577s

and then again right away

(Continue reading)

david | 7 Feb 02:28 2012

Re: Git performance results on a large repository

On Mon, 6 Feb 2012, Joshua Redstone wrote:

> David Lang and David Barr, I generated the pack files by doing a repack:
> "git repack -a -d -f --max-pack-size=10g --depth=100 --window=250"  after
> generating the repo.

how many pack files does this end up creating?

I think that doing a full repack the way you did will group all revisions 
of a given file into a pack.

while what I'm saying is that if you create the packs based on time, 
rather than space efficiency of the resulting pack files, you may end up 
not having to go through as much date when doing things like a git blame.

what you did was

initialize repo
4M commits
repack

what I'm saying is

initialize repo
loop
    500K commits
    repack (and set pack to .keep so it doesn't get overwritten)

so you will end up with ~8 sets of pack files, but time based so that when 
you only need recent information you only look at the most recent pack 
(Continue reading)

Sam Vilain | 6 Feb 22:17 2012
Picon

Re: Git performance results on a large repository

 > Sam Vilain: Thanks for the pointer, i didn't realize that
 > fast-import was bi-directional.  I used it for generating the
 > synthetic repo.  Will look into using it the other way around.
 > Though that still won't speed up things like git-blame,
 > presumably?

It could, because blame is an operation which primarily works on
the source history with little reference to the working copy.  Of
course this will depend on the quality of the implementation
server-side.  Blame should suit distribution over a cluster, as
it is mostly involved with scanning candidate revisions for
string matches which is the compute intensive part.  Coming up
with candidate revisions has its own cost and can probably also
be distributed, but just working on the lowest loop level might
be a good place to start.

What it doesn't help with is local filesystem operations.  For
this I think a different approach is required, if you can tie
into fam or a similar inode change notification system, then you
should be able to avoid the entire recursive stat on 'git
status'.  I'm not sure --assume-unchanged on its own is a good
idea, you could easily miss things.  Those stat's are useful.

Making the index able to hold just changes to the checked-out
tree, as others have mentioned, would also save the massive reads
and writes you've identified.  Perhaps a more high performance
back-end could be developed.

 > The sparse-checkout issue you mention is a good one.

(Continue reading)

Joshua Redstone | 4 Feb 21:05 2012

RE: Git performance results on a large repository

One more follow-on thought.  I imagine that most consumers of git are nowhere near the scale of the test repo
that I described.  They may still enjoy benefit from efforts to improve git support for large repos.  A few
possible reasons:

1. The performance improvements should speed things up for smaller repos as well.
2. They may find their repos growing to a 'large scale' at some point in the future.
3. Any code cleanup as part of an effort to support git scalability is good for code base health and e.g.,
would facilitate future modifications that may more directly affect them.

Cheers,
Josh
________________________________________
From: Nguyen Thai Ngoc Duy [pclouds <at> gmail.com]
Sent: Friday, February 03, 2012 10:53 PM
To: Joshua Redstone
Cc: git <at> vger.kernel.org
Subject: Re: Git performance results on a large repository

On Fri, Feb 3, 2012 at 9:20 PM, Joshua Redstone <joshua.redstone <at> fb.com> wrote:
> I timed a few common operations with both a warm OS file cache and a cold
> cache.  i.e., I did a 'echo 3 | tee /proc/sys/vm/drop_caches' and then did
> the operation in question a few times (first timing is the cold timing,
> the next few are the warm timings).  The following results are on a server
> with average hard drive (I.e., not flash)  and > 10GB of ram.
>
> 'git status' :   39 minutes cold, and 24 seconds warm.
>
> 'git blame':   44 minutes cold, 11 minutes warm.
>
> 'git add' (appending a few chars to the end of a file and adding it):   7
(Continue reading)

Tomas Carnecky | 5 Feb 16:01 2012

Re: Git performance results on a large repository

On 2/4/12 7:53 AM, Nguyen Thai Ngoc Duy wrote:
> On Fri, Feb 3, 2012 at 9:20 PM, Joshua Redstone<joshua.redstone <at> fb.com>  wrote:
>> I timed a few common operations with both a warm OS file cache and a cold
>> cache.  i.e., I did a 'echo 3 | tee /proc/sys/vm/drop_caches' and then did
>> the operation in question a few times (first timing is the cold timing,
>> the next few are the warm timings).  The following results are on a server
>> with average hard drive (I.e., not flash)  and>  10GB of ram.
>>
>> 'git status' :   39 minutes cold, and 24 seconds warm.
>>
>> 'git blame':   44 minutes cold, 11 minutes warm.
>>
>> 'git add' (appending a few chars to the end of a file and adding it):   7
>> seconds cold and 5 seconds warm.
>>
>> 'git commit -m "foo bar3" --no-verify --untracked-files=no --quiet
>> --no-status':  41 minutes cold, 20 seconds warm.  I also hacked a version
>> of git to remove the three or four places where 'git commit' stats every
>> file in the repo, and this dropped the times to 30 minutes cold and 8
>> seconds warm.
> Have you tried "git update-index --assume-unchaged"? That should
> reduce mass lstat() and hopefully improve the above numbers. The
> interface is not exactly easy-to-use, but if it has significant gain,
> then we can try to improve UI.
>
> On the index size issue, ideally we should make minimum writes to
> index instead of rewriting 191 MB index. An improvement we could do
> now is to compress it, reduce disk footprint, thus disk I/O. If you
> compress the index with gzip, how big is it?
If you're not afraid to add filesystem-specific code to git, you could 
(Continue reading)

Nguyen Thai Ngoc Duy | 5 Feb 16:17 2012
Picon

Re: Git performance results on a large repository

On Sun, Feb 5, 2012 at 10:01 PM, Tomas Carnecky <tom <at> dbservice.com> wrote:
> On 2/4/12 7:53 AM, Nguyen Thai Ngoc Duy wrote:
>>
>> On Fri, Feb 3, 2012 at 9:20 PM, Joshua Redstone<joshua.redstone <at> fb.com>
>>  wrote:
>>>
>>> I timed a few common operations with both a warm OS file cache and a cold
>>> cache.  i.e., I did a 'echo 3 | tee /proc/sys/vm/drop_caches' and then
>>> did
>>> the operation in question a few times (first timing is the cold timing,
>>> the next few are the warm timings).  The following results are on a
>>> server
>>> with average hard drive (I.e., not flash)  and>  10GB of ram.
>>>
>>> 'git status' :   39 minutes cold, and 24 seconds warm.
>>>
>>> 'git blame':   44 minutes cold, 11 minutes warm.
>>>
>>> 'git add' (appending a few chars to the end of a file and adding it):   7
>>> seconds cold and 5 seconds warm.
>>>
>>> 'git commit -m "foo bar3" --no-verify --untracked-files=no --quiet
>>> --no-status':  41 minutes cold, 20 seconds warm.  I also hacked a version
>>> of git to remove the three or four places where 'git commit' stats every
>>> file in the repo, and this dropped the times to 30 minutes cold and 8
>>> seconds warm.
>>
>> Have you tried "git update-index --assume-unchaged"? That should
>> reduce mass lstat() and hopefully improve the above numbers. The
>> interface is not exactly easy-to-use, but if it has significant gain,
(Continue reading)

slinky | 4 Feb 09:57 2012
Picon
Picon

Re: Git performance results on a large repository

Joshua Redstone <joshua.redstone <at> fb.com> writes:

> The git performance we observed here is too slow for our needs.  So the
> question becomes, if we want to keep using git going forward, what's the
> best way to improve performance.  It seems clear we'll probably need some
> specialized servers (e.g., to perform git-blame quickly) and maybe
> specialized file system integration to detect what files have changed in a
> working tree.

Hi Joshua,

sounds like you have everything in a single .git. Split up the massive
repository to separate smaller .git repositories.

For example, Android code base is quite big. They use the repo tool to manage a
number of separate .git repositories as one big aggregate "repository".

Cheers,
Slinky

Greg Troxel | 4 Feb 22:42 2012
Picon

Re: Git performance results on a large repository


Joshua Redstone <joshua.redstone <at> fb.com> writes:

> The test repo has 4 million commits, linear history and about 1.3 million
> files.  The size of the .git directory is about 15GB, and has been
> repacked with 'git repack -a -d -f --max-pack-size=10g --depth=100
> --window=250'.  This repack took about 2 days on a beefy machine (I.e.,
> lots of ram and flash).  The size of the index file is 191 MB. I can share
> the script that generated it if people are interested - It basically picks
> 2-5 files, modifies a line or two and adds a few lines at the end
> consisting of random dictionary words, occasionally creates a new file,
> commits all the modifications and repeats.

I have a repository with about 500K files, 3.3G checkout, 1.5G .git, and
about 10K commits.  (This is a real repository, not a test case.)  So
not as many commits by a lot, but the size seems not so far off.

> I timed a few common operations with both a warm OS file cache and a cold
> cache.  i.e., I did a 'echo 3 | tee /proc/sys/vm/drop_caches' and then did
> the operation in question a few times (first timing is the cold timing,
> the next few are the warm timings).  The following results are on a server
> with average hard drive (I.e., not flash)  and > 10GB of ram.
>
> 'git status' :   39 minutes cold, and 24 seconds warm.

Both of these numbers surprise me.  I'm using NetBSD, whose stat
implementation isn't as optimized as Linux (you didn't say, but
assuming).   On a years-old desktop, git status seems to be about a
minute semi-cold and 5s warm (once I set the vnode cache big over 500K,
vs 350K default for a 2G ram machine).
(Continue reading)

david | 5 Feb 05:30 2012

Re: Git performance results on a large repository

On Fri, 3 Feb 2012, Joshua Redstone wrote:

> The test repo has 4 million commits, linear history and about 1.3 million
> files.  The size of the .git directory is about 15GB, and has been
> repacked with 'git repack -a -d -f --max-pack-size=10g --depth=100
> --window=250'.  This repack took about 2 days on a beefy machine (I.e.,
> lots of ram and flash).  The size of the index file is 191 MB.

This may be a silly thought, but what if instead of one pack file of your 
entire history (4 million commits) you create multiple packs (say every 
half million commits) and mark all but the most recent pack as .keep (so 
that they won't be modified by a repack)

that way things that only need to worry about recent history (blame, etc) 
will probably never have to go past the most recent pack file or two

I may be wrong, but I think that when git is looking for 'similar files' 
for delta compression, it limits it's search to the current pack, so this 
will also keep you from searching the entire project history.

David Lang
David Barr | 5 Feb 12:24 2012
Picon
Picon

Re: Git performance results on a large repository

On Sun, Feb 5, 2012 at 3:30 PM,  <david <at> lang.hm> wrote:
> On Fri, 3 Feb 2012, Joshua Redstone wrote:
>
>> The test repo has 4 million commits, linear history and about 1.3 million
>> files.  The size of the .git directory is about 15GB, and has been
>> repacked with 'git repack -a -d -f --max-pack-size=10g --depth=100
>> --window=250'.  This repack took about 2 days on a beefy machine (I.e.,
>> lots of ram and flash).  The size of the index file is 191 MB.
>
>
> This may be a silly thought, but what if instead of one pack file of your
> entire history (4 million commits) you create multiple packs (say every half
> million commits) and mark all but the most recent pack as .keep (so that
> they won't be modified by a repack)
>
> that way things that only need to worry about recent history (blame, etc)
> will probably never have to go past the most recent pack file or two
>
> I may be wrong, but I think that when git is looking for 'similar files' for
> delta compression, it limits it's search to the current pack, so this will
> also keep you from searching the entire project history.

I don't know if there is an easy way to determine with the with the
current tools
in git but one useful statistic for tuning packing performance is the
size of the
largest component in the delta-chain graph. The significance of this number is
that the product of window-size and maximum depth need not be larger than it.
I've found that with some older repositories I could have a depth as low as 3
and still get good performance from a moderate window size.
(Continue reading)

Emanuele Zattin | 7 Feb 09:58 2012
Picon

Re: Git performance results on a large repository

Joshua Redstone <joshua.redstone <at> fb.com> writes:

> 
> Hi Git folks,
> 

Hello everybody! 

I would just like to contribute a small set of blog posts 
about this issue and a possible solution. 
Sorry for the tone in which I wrote those posts, 
but I think there are some valid points in there.

https://gist.github.com/1758346

BR,

Emanuele Zattin


Gmane