Olav Vitters | 17 Feb 10:48 2011
Picon

Git: Do not use http to access Git!

Please note that to checkout a Git repository you should use:
  git clone git://git.gnome.org/[project]
or (if you have commit access):
  git clone ssh://[login <at> ]git.gnome.org/git/[project]

Do NOT try to checkout via http. It is not something we support!

Some people have been using the cgit interface to checkout a lot of git
modules. This cause a very high load on our server. Please use the git
protocol instead. For now I've blocked git from accessing cgit as too
many people have been using the http interface and it negatively impacts
performance (very high loads over long periods).

--

-- 
Regards,
Olav
Danielle Madeley | 17 Feb 13:25 2011
Picon

Re: Git: Do not use http to access Git!

On Thu, 2011-02-17 at 10:48 +0100, Olav Vitters wrote:

> Some people have been using the cgit interface to checkout a lot of git
> modules. This cause a very high load on our server. Please use the git
> protocol instead. For now I've blocked git from accessing cgit as too
> many people have been using the http interface and it negatively impacts
> performance (very high loads over long periods).

Is it possible these people are on ridiculous corporate networks that
only permit HTTP traffic (via a proxy) and allow no other outgoing
connections beyond their network?

I have been on one of these before, http was the only way I could
checkout sources (from Subversion at the time).

If this is the case, git being awesome as it is, perhaps someone could
set up a remote that offers http cloning that updates once a day or so.
This would let people clone a repo, and still submit their work via
format-patch etc.

--danni

--

-- 
Danielle Madeley
Software Developer, Collabora Ltd.                  Melbourne, Australia

www.collabora.co.uk
Juan Jesús Ojeda Croissier | 17 Feb 13:53 2011
Picon

Re: Git: Do not use http to access Git!

On Thu, Feb 17, 2011 at 1:25 PM, Danielle Madeley
<danielle.madeley <at> collabora.co.uk> wrote:
> On Thu, 2011-02-17 at 10:48 +0100, Olav Vitters wrote:
>
>> Some people have been using the cgit interface to checkout a lot of git
>> modules. This cause a very high load on our server. Please use the git
>> protocol instead. For now I've blocked git from accessing cgit as too
>> many people have been using the http interface and it negatively impacts
>> performance (very high loads over long periods).
>
> Is it possible these people are on ridiculous corporate networks that
> only permit HTTP traffic (via a proxy) and allow no other outgoing
> connections beyond their network?
>
> I have been on one of these before, http was the only way I could
> checkout sources (from Subversion at the time).
>
> If this is the case, git being awesome as it is, perhaps someone could
> set up a remote that offers http cloning that updates once a day or so.
> This would let people clone a repo, and still submit their work via
> format-patch etc.
>

I agree. That has happened to me as well.
I understand the performance issues, but not to have a http acces to
the clones, could be blocker issue for some people.

Greetings.

--

-- 
(Continue reading)

Olav Vitters | 17 Feb 14:07 2011
Picon

Re: Git: Do not use http to access Git!

On Thu, Feb 17, 2011 at 01:53:12PM +0100, Juan Jesús Ojeda Croissier wrote:
> I agree. That has happened to me as well.
> I understand the performance issues, but not to have a http acces to
> the clones, could be blocker issue for some people.

Just wondering, and please see my other reply regarding http..

Why is a http checkout still handy if you cannot commit? Shouldn't
everything be much easier when using a distributed vcs like Git?

--

-- 
Regards,
Olav

PS: With impacts performance I mean the machine goes into swap and the
load is at 30+. I've limited it somewhat by severely reducing the number
of Apache processes, but load still easily reached 8.
Maciej Piechotka | 17 Feb 14:22 2011
Picon

Re: Git: Do not use http to access Git!

On 17/02/2011 13:07, Olav Vitters wrote:
> On Thu, Feb 17, 2011 at 01:53:12PM +0100, Juan Jesús Ojeda Croissier wrote:
>> I agree. That has happened to me as well.
>> I understand the performance issues, but not to have a http acces to
>> the clones, could be blocker issue for some people.
> 
> Just wondering, and please see my other reply regarding http..
> 
> Why is a http checkout still handy if you cannot commit? Shouldn't
> everything be much easier when using a distributed vcs like Git?
> 

If you don't have write access it doesn't matter for you anyway (you
clone to backport patch or to produce patch for bugzilla).

Regards
Olav Vitters | 17 Feb 14:04 2011
Picon

Re: Git: Do not use http to access Git!

On Thu, Feb 17, 2011 at 11:25:05PM +1100, Danielle Madeley wrote:
> On Thu, 2011-02-17 at 10:48 +0100, Olav Vitters wrote:
> 
> > Some people have been using the cgit interface to checkout a lot of git
> > modules. This cause a very high load on our server. Please use the git
> > protocol instead. For now I've blocked git from accessing cgit as too
> > many people have been using the http interface and it negatively impacts
> > performance (very high loads over long periods).
> 
> Is it possible these people are on ridiculous corporate networks that
> only permit HTTP traffic (via a proxy) and allow no other outgoing
> connections beyond their network?
> 
> I have been on one of these before, http was the only way I could
> checkout sources (from Subversion at the time).

Could be, no idea. Just know that we didn't intend http access for git.
That it worked at all is just by sheer luck (cgit supports a bad
method) and it is affecting our server (seems like entire jhbuilds by
multiple people, not just 1 checkout).

> If this is the case, git being awesome as it is, perhaps someone could
> set up a remote that offers http cloning that updates once a day or so.
> This would let people clone a repo, and still submit their work via
> format-patch etc.

I've noticed the following after sending the initial mail:
http://progit.org/2010/03/04/smart-http.html
http://www.kernel.org/pub/software/scm/git/docs/git-http-backend.html

(Continue reading)

Arun Raghavan | 17 Feb 14:32 2011
Picon

Re: Git: Do not use http to access Git!

On Thu, 2011-02-17 at 14:04 +0100, Olav Vitters wrote:
[...]
> > If this is the case, git being awesome as it is, perhaps someone could
> > set up a remote that offers http cloning that updates once a day or so.
> > This would let people clone a repo, and still submit their work via
> > format-patch etc.
> 
> I've noticed the following after sending the initial mail:
> http://progit.org/2010/03/04/smart-http.html
> http://www.kernel.org/pub/software/scm/git/docs/git-http-backend.html
> 
> Basically: If the client has git 1.6.6+, there is a possibility to be as
> (almost as?) efficient as the git protocol using http. I've reconfigured
> all git repos to have:
>   http.getanyfile false
>   http.receivepack false
> Meaning: Only want efficient Git.
> 
> My intend is to allow the git 1.6.6+ method, and explicitly disallow the
> previous method.
> 
> This requires some sysadmin work to setup (newer git, newer cgit, etc).
> Doing that atm, not sure when I'll finish.

Thanks! A number of colleges here in India block access to all ports and
allow HTTP over a proxy, so that's a whole bunch of people who will
benefit from this.

Cheers,
Arun
(Continue reading)

Olav Vitters | 17 Feb 15:33 2011
Picon

Use git 1.6.6+ when using http (was: Git: Do not use http to access Git!)

On Thu, Feb 17, 2011 at 02:04:02PM +0100, Olav Vitters wrote:
> This requires some sysadmin work to setup (newer git, newer cgit, etc).
> Doing that atm, not sure when I'll finish.

This is really efficient. Cloning gnome-terminal only results in 2 hits:
62.195.108.47 - - [17/Feb/2011:14:24:14 +0000] "GET
/browse/gnome-terminal/info/refs?service=git-upload-pack HTTP/1.1" 200 21538 "-" "git/1.7.4"
62.195.108.47 - - [17/Feb/2011:14:24:15 +0000] "POST /browse/gnome-terminal/git-upload-pack
HTTP/1.1" 200 12218190 "-" "git/1.7.4"

And the speed is nice too:

$ git clone http://git.gnome.org/browse/gnome-terminal
Cloning into gnome-terminal...
remote: Counting objects: 22113, done.
remote: Compressing objects: 100% (6749/6749), done.
remote: Total 22113 (delta 17558), reused 19020 (delta 15302)
Receiving objects: 100% (22113/22113), 11.64 MiB | 1.35 MiB/s, done.
Resolving deltas: 100% (17558/17558), done.

I've blocked Git 1.6.5 and earlier (hopefully).

--

-- 
Regards,
Olav
Behdad Esfahbod | 17 Feb 15:41 2011

Re: Use git 1.6.6+ when using http

On 02/17/11 09:33, Olav Vitters wrote:
> 62.195.108.47 - - [17/Feb/2011:14:24:15 +0000] "POST /browse/gnome-terminal/git-upload-pack
HTTP/1.1" 200 12218190 "-" "git/1.7.4"

What's the POST about?

So, if I understand it correctly, the refs? URLs are passed from cgit to git now?

behdad
Olav Vitters | 17 Feb 16:25 2011
Picon

Re: Use git 1.6.6+ when using http

On Thu, Feb 17, 2011 at 09:41:47AM -0500, Behdad Esfahbod wrote:
> On 02/17/11 09:33, Olav Vitters wrote:
> > 62.195.108.47 - - [17/Feb/2011:14:24:15 +0000] "POST /browse/gnome-terminal/git-upload-pack
HTTP/1.1" 200 12218190 "-" "git/1.7.4"
> 
> What's the POST about?

No idea :) I know the first hit is special, and then git knows the
server supports the efficient format. I assume/guess within the POST the
git client tells git.gnome.org what stuff to send.

> So, if I understand it correctly, the refs? URLs are passed from cgit
> to git now?

More like s/from/instead of/. I'm using the configuration adapted from 
http://www.kernel.org/pub/software/scm/git/docs/git-http-backend.html

More or less the following:
| SetEnv GIT_PROJECT_ROOT /git
| 
| AliasMatch ^/browse/(.*/objects/[0-9a-f]{2}/[0-9a-f]{38})$          /git/$1
| AliasMatch ^/browse/(.*/objects/pack/pack-[0-9a-f]{40}.(pack|idx))$ /git/$1
| ScriptAliasMatch \
|         "(?x)^/git/(.*/(HEAD | \
|                         info/refs | \
|                         objects/info/[^/]+ | \
|                         git-(upload|receive)-pack))$" \
|         /usr/bin/git-http-backend/$1
| ScriptAlias /git/ /var/www/cgit/cgit.cgi

(Continue reading)

Behdad Esfahbod | 17 Feb 16:49 2011

Re: Use git 1.6.6+ when using http

On 02/17/11 10:25, Olav Vitters wrote:
> directly linking 100+ of
> different images to cgit on a popular website (one of our artists),

Can we cache those somehow?  Or that would require a new squid layer?

behdad
Olav Vitters | 18 Feb 12:23 2011
Picon

Re: Use git 1.6.6+ when using http

On Thu, Feb 17, 2011 at 10:49:15AM -0500, Behdad Esfahbod wrote:
> On 02/17/11 10:25, Olav Vitters wrote:
> > directly linking 100+ of
> > different images to cgit on a popular website (one of our artists),
> 
> Can we cache those somehow?  Or that would require a new squid layer?

Website has been disabled, so that relieves a lot of stress. Caching is
still a possible good idea as server should cope better.

I've changed some of the expire settings in cgit. We could have apache
cache this (http://httpd.apache.org/docs/2.2/caching.html).
First waiting for the effect of the longer expire times on Google (used
to be 5min for everything, now an individual commit is set to 31 days;
others to 2 hours). Unfortunately bit difficult to set the expire times
exactly as I want it given the options provided by cgit.

Fwiw, cgit itself can also do caching, but Apache will still execute the
cgit process. Better to have the caching done by Apache IMO.

--

-- 
Regards,
Olav
Nicolas Trangez | 17 Feb 14:06 2011

Re: Git: Do not use http to access Git!

On Thu, 2011-02-17 at 23:25 +1100, Danielle Madeley wrote: 
> On Thu, 2011-02-17 at 10:48 +0100, Olav Vitters wrote:
> 
> > Some people have been using the cgit interface to checkout a lot of git
> > modules. This cause a very high load on our server. Please use the git
> > protocol instead. For now I've blocked git from accessing cgit as too
> > many people have been using the http interface and it negatively impacts
> > performance (very high loads over long periods).
> 
> Is it possible these people are on ridiculous corporate networks that
> only permit HTTP traffic (via a proxy) and allow no other outgoing
> connections beyond their network?
> 
> I have been on one of these before, http was the only way I could
> checkout sources (from Subversion at the time).
> 
> If this is the case, git being awesome as it is, perhaps someone could
> set up a remote that offers http cloning that updates once a day or so.
> This would let people clone a repo, and still submit their work via
> format-patch etc.

If this load issue concerns initial 'git clone' calls over HTTP, and not
further updates ('git fetch'), a solution might be to create baseline
tarballs (e.g. weekly) containing a packed HTTP clone, and put these
available for download. HTTP users can download and unpack these, then
use the repository as usual. This should reduce load on the server, as
well as reduce the time a developer needs to get his repository clone up
and running.

This method was/is used by several other projects using
(Continue reading)

Simos Xenitellis | 17 Feb 14:45 2011

Re: Git: Do not use http to access Git!

On Thu, Feb 17, 2011 at 3:06 PM, Nicolas Trangez <eikke <at> eikke.com> wrote:
> On Thu, 2011-02-17 at 23:25 +1100, Danielle Madeley wrote:
>> On Thu, 2011-02-17 at 10:48 +0100, Olav Vitters wrote:
>>
>> > Some people have been using the cgit interface to checkout a lot of git
>> > modules. This cause a very high load on our server. Please use the git
>> > protocol instead. For now I've blocked git from accessing cgit as too
>> > many people have been using the http interface and it negatively impacts
>> > performance (very high loads over long periods).
>>
>> Is it possible these people are on ridiculous corporate networks that
>> only permit HTTP traffic (via a proxy) and allow no other outgoing
>> connections beyond their network?
>>
>> I have been on one of these before, http was the only way I could
>> checkout sources (from Subversion at the time).
>>
>> If this is the case, git being awesome as it is, perhaps someone could
>> set up a remote that offers http cloning that updates once a day or so.
>> This would let people clone a repo, and still submit their work via
>> format-patch etc.
>
> If this load issue concerns initial 'git clone' calls over HTTP, and not
> further updates ('git fetch'), a solution might be to create baseline
> tarballs (e.g. weekly) containing a packed HTTP clone, and put these
> available for download. HTTP users can download and unpack these, then
> use the repository as usual. This should reduce load on the server, as
> well as reduce the time a developer needs to get his repository clone up
> and running.
>
(Continue reading)

Olav Vitters | 18 Feb 12:11 2011
Picon

Re: Git: Do not use http to access Git!

On Thu, Feb 17, 2011 at 03:45:55PM +0200, Simos Xenitellis wrote:
> On Thu, Feb 17, 2011 at 3:06 PM, Nicolas Trangez <eikke <at> eikke.com> wrote:
> > If this load issue concerns initial 'git clone' calls over HTTP, and not
> > further updates ('git fetch'), a solution might be to create baseline
> > tarballs (e.g. weekly) containing a packed HTTP clone, and put these
> > available for download. HTTP users can download and unpack these, then
[..]
> This would be amazing to have. It has been mentioned before.
> I understand though it might be difficult to set it up and get it working.

The fetch likely also impacted performance (fetch+clone is pretty much
the same I guess). Now with the new method, I think all problems should
be gone.

Regarding unreliable internet connections, anyone is free to setup
something (tarballs/mirrors). Or clearly show there is a great need for
this (gnome-i18n?).

--

-- 
Regards,
Olav
Tshepang Lekhonkhobe | 18 Feb 06:46 2011
Picon

Re: Git: Do not use http to access Git!

On Thu, Feb 17, 2011 at 09:48, Olav Vitters <olav <at> vitters.nl> wrote:
> Please note that to checkout a Git repository you should use:
>  git clone git://git.gnome.org/[project]
> or (if you have commit access):
>  git clone ssh://[login <at> ]git.gnome.org/git/[project]
>
> Do NOT try to checkout via http. It is not something we support!
>
> Some people have been using the cgit interface to checkout a lot of git
> modules. This cause a very high load on our server. Please use the git
> protocol instead. For now I've blocked git from accessing cgit as too
> many people have been using the http interface and it negatively impacts
> performance (very high loads over long periods).

Do you mind explaining what's causing bad performance when using
http... why is it less efficient than using git protocol?
_______________________________________________
desktop-devel-list mailing list
desktop-devel-list <at> gnome.org
http://mail.gnome.org/mailman/listinfo/desktop-devel-list
Germán Póo-Caamaño | 18 Feb 08:48 2011
Picon

Re: Git: Do not use http to access Git!

On Fri, 2011-02-18 at 05:46 +0000, Tshepang Lekhonkhobe wrote:
> On Thu, Feb 17, 2011 at 09:48, Olav Vitters <olav <at> vitters.nl> wrote:
> > Please note that to checkout a Git repository you should use:
> >  git clone git://git.gnome.org/[project]
> > or (if you have commit access):
> >  git clone ssh://[login <at> ]git.gnome.org/git/[project]
> >
> > Do NOT try to checkout via http. It is not something we support!
> >
> > Some people have been using the cgit interface to checkout a lot of git
> > modules. This cause a very high load on our server. Please use the git
> > protocol instead. For now I've blocked git from accessing cgit as too
> > many people have been using the http interface and it negatively impacts
> > performance (very high loads over long periods).
> 
> Do you mind explaining what's causing bad performance when using
> http... why is it less efficient than using git protocol?

This is documented in:
http://progit.org/2010/03/04/smart-http.html
http://progit.org/book/ch4-1.html
http://progit.org/book/ch9-6.html

--

-- 
Germán Póo-Caamaño
http://www.gnome.org/~gpoo/
_______________________________________________
desktop-devel-list mailing list
(Continue reading)

Olav Vitters | 18 Feb 09:44 2011
Picon

Re: Git: Do not use http to access Git!

On Fri, Feb 18, 2011 at 05:46:06AM +0000, Tshepang Lekhonkhobe wrote:
> Do you mind explaining what's causing bad performance when using
> http... why is it less efficient than using git protocol?

In addition to the answer by gpoo, I've finished the smart http setup
yesterday. I'll send an announce shortly.

--

-- 
Regards,
Olav

Gmane