The Subversion Version Control System Development

TheBlueSky .Net | 7 Feb 2012 20:49

Error While Checking out Git Repository

Hello,

I was checking out a Git repository (https://github.com/NuGet/PoliteCaptcha.git) when I encountered the following error near the end of getting all the files:

Subversion encountered a serious problem.

Please take the time to report this on the Subversion mailing list

with as much information as possible about what

you were trying to do.

But please first search the mailing list archives for the error message

to avoid reporting the same problem repeatedly.

You can find the mailing list archives at

http://subversion.apache.org/mailing-lists.html

Subversion reported the following

(you can copy the content of this dialog

to the clipboard using Ctrl-C):

In file

'D:\Development\SVN\Releases\TortoiseSVN-1.7.1\ext\subversion\subversion\libsvn_wc\wc_db.c'

line 1588: assertion failed (SVN_IS_VALID_REVNUM(changed_rev))

And if someone is wondering, checking out from github using SVN client is supported.

Thanks.

headers

C. Michael Pilato | 7 Feb 2012 21:20

Re: Error While Checking out Git Repository

On 02/07/2012 02:49 PM, TheBlueSky .Net wrote:
> Hello,
> 
> I was checking out a Git repository
> (https://github.com/NuGet/PoliteCaptcha.git) when I encountered the
> following error near the end of getting all the files:
> 
> Subversion encountered a serious problem.
> Please take the time to report this on the Subversion mailing list
> with as much information as possible about what
> you were trying to do.
> But please first search the mailing list archives for the error message
> to avoid reporting the same problem repeatedly.
> You can find the mailing list archives at
> http://subversion.apache.org/mailing-lists.html
>
> Subversion reported the following
> (you can copy the content of this dialog
> to the clipboard using Ctrl-C):
> 
> In file
>  'D:\Development\SVN\Releases\TortoiseSVN-1.7.1\ext\subversion\subversion\libsvn_wc\wc_db.c'
>  line 1588: assertion failed (SVN_IS_VALID_REVNUM(changed_rev))
> 
> And if someone is wondering, checking out from github using SVN client is
> supported.

Well, supported by whom?  

This community has only signed up to support Subversion's command-line
client checking out from a Subversion mod_dav_svn or svnserve server.  I
must assume that github has put together their own Subversion-esque server
software which communicates with git repositories.  Obviously, we expect our
client to behave properly against any server that wields our protocols
correctly, but first we need to establish that GitHub's server is such a server.

At first blush, it would appear that the GitHub server didn't transmit a
valid svn:entry:committed-rev property value for one of the directories your
checkout is attempting to register with Subversion's working copy
administration library.  If this problem is repeatable, it would be great to
get a network trace of the failing checkout.

--

-- 
C. Michael Pilato <cmpilato <at> collab.net>
CollabNet   <>   www.collab.net   <>   Distributed Development On Demand

headers

Nick Hengeveld | 8 Feb 2012 20:10

Re: Error While Checking out Git Repository

C. Michael Pilato <cmpilato <at> collab.net> writes:

>
> Well, supported by whom?  
> 
> At first blush, it would appear that the GitHub server didn't transmit a
> valid svn:entry:committed-rev property value for one of the directories your
> checkout is attempting to register with Subversion's working copy
> administration library.  If this problem is repeatable, it would be great to
> get a network trace of the failing checkout.
> 

I'm happy to take a look at this from the GitHub side - can the person who
reported this contact support <at> github.com?

FWIW, I just tried a checkout with a 1.6.16 client and didn't have any
problems.  I'll see if I can reproduce it with a 1.7 client.

--Nick

headers

C. Michael Pilato | 8 Feb 2012 20:47

Re: Error While Checking out Git Repository

I'm happy to try it with 1.7, too, but couldn't easily figure out the
correct checkout URL:

$ svn co https://github.com/NuGet/PoliteCaptcha.git
svn: E175009: Unable to connect to a repository at URL
'https://github.com/NuGet/PoliteCaptcha.git'
svn: E175009: XML parsing failed: (411 Length Required)
$

Ah!!  That was with ra_serf.  With ra_neon, I get the reported behavior:

$ svn co https://github.com/NuGet/PoliteCaptcha.git
A    PoliteCaptcha.git/branches
A    PoliteCaptcha.git/tags
A    PoliteCaptcha.git/tags/v0.1
[...]
A    PoliteCaptcha.git/tags/v0.1/Sample/Web.Release.config
A    PoliteCaptcha.git/tags/v0.1/Sample/Web.config
A    PoliteCaptcha.git/tags/v0.1/Sample/packages.config
svn: E235000: In file 'subversion/libsvn_wc/wc_db.c' line 1608: assertion
failed (SVN_IS_VALID_REVNUM(changed_rev))
Aborted
$

I'll take a peek with Neon's debugging enabled and see what I can see.

(But please do note that it appears GitHub appears to be incompatible with
Subversion clients which use the newer ra_serf library.)

On 02/08/2012 02:10 PM, Nick Hengeveld wrote:
> C. Michael Pilato <cmpilato <at> collab.net> writes:
> 
>>
>> Well, supported by whom?  
>>
>> At first blush, it would appear that the GitHub server didn't transmit a
>> valid svn:entry:committed-rev property value for one of the directories your
>> checkout is attempting to register with Subversion's working copy
>> administration library.  If this problem is repeatable, it would be great to
>> get a network trace of the failing checkout.
>>
> 
> 
> I'm happy to take a look at this from the GitHub side - can the person who
> reported this contact support <at> github.com?
> 
> FWIW, I just tried a checkout with a 1.6.16 client and didn't have any
> problems.  I'll see if I can reproduce it with a 1.7 client.
> 
> --Nick
> 

--

-- 
C. Michael Pilato <cmpilato <at> collab.net>
CollabNet   <>   www.collab.net   <>   Distributed Development On Demand

headers

Greg Stein | 8 Feb 2012 21:51

Re: Error While Checking out Git Repository

On Wed, Feb 8, 2012 at 14:47, C. Michael Pilato <cmpilato <at> collab.net> wrote:
> I'm happy to try it with 1.7, too, but couldn't easily figure out the
> correct checkout URL:
>
> $ svn co https://github.com/NuGet/PoliteCaptcha.git
> svn: E175009: Unable to connect to a repository at URL
> 'https://github.com/NuGet/PoliteCaptcha.git'
> svn: E175009: XML parsing failed: (411 Length Required)
> $

This looks like the problem where ra_serf starts out with an HTTP/1.1
request: specifically, a chunked request. This is typically caused by
a non-1.1 proxy in front of the server, such as nginx.

Ideally, github would switch to a newer/better proxy such as haproxy.

But the core problem is on our side -- issue 3979
(http://subversion.tigris.org/issues/show_bug.cgi?id=3979).

>...

Cheers,
-g

headers

Greg Stein | 8 Feb 2012 21:59

Re: Error While Checking out Git Repository

On Wed, Feb 8, 2012 at 15:51, Greg Stein <gstein <at> gmail.com> wrote:
>...
> This looks like the problem where ra_serf starts out with an HTTP/1.1
> request: specifically, a chunked request. This is typically caused by
> a non-1.1 proxy in front of the server, such as nginx.
>
> Ideally, github would switch to a newer/better proxy such as haproxy.

Let me clarify this a bit: sticking to a 1.0 proxy obviates a huge
number of client advantages. Subversion is built using a "streaming"
model where we don't know the request length in advance. If the client
is forced to perform an HTTP/1.0 request, then it would have to shove
the entire request onto disk. Check the length, and then build a 1.0
request.

ra_neon doesn't allow streaming like this (everything gets shoved to
disk first), which is why it can stick to 1.0 requests, and why it is
painful for the client.

Further, I don't think that HTTP/1.0 allows for gzip/deflate on the
requests, which means we have more network traffic.

So... this isn't really so much about "svn must allow for HTTP/1.0"
(yes, we'll work on it), but that github could also provide better
service overall by updating its proxy.

Cheers,
-g

headers

Nick Hengeveld | 10 Feb 2012 02:29

Re: Error While Checking out Git Repository

On Wed, Feb 8, 2012 at 12:59 PM, Greg Stein <gstein <at> gmail.com> wrote:

So... this isn't really so much about "svn must allow for HTTP/1.0"
(yes, we'll work on it), but that github could also provide better
service overall by updating its proxy.

I've been looking into what it would take to update our proxy. I have an nginx

configuration in my dev environment that handles requests with chunked

transfer-encoding and we could feasibly deploy something like this to

production. However, I'm now getting a different error on the client:

svn: E235000: In file 'subversion/libsvn_ra_serf/update.c' line 1769: internal malfunction

Aborted

From what I can see at https://github.com/apache/subversion/blob/trunk/subversion/libsvn_ra_serf/update.c#L1698-1771

it looks like the client is trying to process an add/open file. The checkout

I'm testing is small and contains only one file, and this is what the server is returning for that file:

<S:add-file name="README.txt" >

<D:checked-in><D:href>/github/sample/!svn/ver/3/trunk/README.txt</D:href></D:checked-in>

<S:set-prop name="svn:entry:committed-rev">3</S:set-prop>

<S:set-prop name="svn:entry:uuid">faa06a63-5b49-ecd5-b7fc-ce57b7db7bff</S:set-prop>

<S:set-prop name="svn:entry:committed-date">2012-02-09T22:12:11.000000Z</S:set-prop>

<S:set-prop name="svn:entry:last-author">subversion.user</S:set-prop>

<S:txdelta>

U1ZO

AQAAMQs4AnicazAEAAEzALIxeJwLycgsVgCixDyFotTElNxUHYXSgpTEktQUhbLMRIVEBUM9c4XknMzUvBI9rsh8PS4AnrgP

sw==

</S:txdelta>

<S:prop><V:md5-checksum>5fa5200f0f095d120c13870e107dfb7a</V:md5-checksum></S:prop>

</S:add-file>

I'm using a Linux client installed from CollabNetSubversion-client-1.7.2-1.x86_64.rpm.

svn --version reports: svn, version 1.7.2 (r1207936)

What's the best way to track down issues like this? I'm not currently familiar

with svn client debugging but would be happy to learn.

For a successful technology, reality must take precedence over public
relations, for nature cannot be fooled.

headers

Daniel Shahaf | 10 Feb 2012 03:53

Re: Error While Checking out Git Repository

Nick Hengeveld wrote on Thu, Feb 09, 2012 at 17:29:14 -0800:
> On Wed, Feb 8, 2012 at 12:59 PM, Greg Stein <gstein <at> gmail.com> wrote:
> 
> So... this isn't really so much about "svn must allow for HTTP/1.0"
> > (yes, we'll work on it), but that github could also provide better
> > service overall by updating its proxy.
> >
> 
> I've been looking into what it would take to update our proxy.  I have an
> nginx
> configuration in my dev environment that handles requests with chunked
> transfer-encoding and we could feasibly deploy something like this to
> production.  However, I'm now getting a different error on the client:
> 
> svn: E235000: In file 'subversion/libsvn_ra_serf/update.c' line
> 1769: internal malfunction
> Aborted
> 

As it happens I found and fixed this particular issue a few hours ago;
the fix is nominated for backport towards 1.7.3.

> From what I can see at
> https://github.com/apache/subversion/blob/trunk/subversion/libsvn_ra_serf/update.c#L1698-1771
> it looks like the client is trying to process an add/open file.  The
> checkout
> I'm testing is small and contains only one file, and this is what the
> server is returning for that file:
> 
> <S:add-file name="README.txt" >
> <D:checked-in><D:href>/github/sample/!svn/ver/3/trunk/README.txt</D:href></D:checked-in>
> <S:set-prop name="svn:entry:committed-rev">3</S:set-prop>
> <S:set-prop
> name="svn:entry:uuid">faa06a63-5b49-ecd5-b7fc-ce57b7db7bff</S:set-prop>
> <S:set-prop
> name="svn:entry:committed-date">2012-02-09T22:12:11.000000Z</S:set-prop>
> <S:set-prop name="svn:entry:last-author">subversion.user</S:set-prop>
> <S:txdelta>

As you probably know from the part of libsvn_ra_serf you cited, the
<S:txdelta/> tag is the one that causes the blow-up.

A quick grep through the libsvn_ra_serf and mod_dav_svn sources leads me
to conjecture that ra_serf requests a "send-all"=false REPORT, but
receives from your server a "send-all"=true response.

> U1ZO
> AQAAMQs4AnicazAEAAEzALIxeJwLycgsVgCixDyFotTElNxUHYXSgpTEktQUhbLMRIVEBUM9c4XknMzUvBI9rsh8PS4AnrgP
> sw==
> </S:txdelta>
> <S:prop><V:md5-checksum>5fa5200f0f095d120c13870e107dfb7a</V:md5-checksum></S:prop>
> </S:add-file>
> 
> I'm using a Linux client installed
> from CollabNetSubversion-client-1.7.2-1.x86_64.rpm.
> svn --version reports: svn, version 1.7.2 (r1207936)
> 
> What's the best way to track down issues like this?  I'm not currently
> familiar
> with svn client debugging but would be happy to learn.
> 

For svn client debugging, the first thing to do is to build
--enable-maintainer-mode; this enables stricter compiler warnings
and internal debugging checks (guarded by '#ifdef SVN_DEBUG').

> -- 
> For a successful technology, reality must take precedence over public
> relations, for nature cannot be fooled.

headers

Greg Stein | 10 Feb 2012 03:56

Re: Error While Checking out Git Repository

On Thu, Feb 9, 2012 at 21:53, Daniel Shahaf <danielsh <at> elego.de> wrote:
> Nick Hengeveld wrote on Thu, Feb 09, 2012 at 17:29:14 -0800:
>...
>> What's the best way to track down issues like this?  I'm not currently
>> familiar
>> with svn client debugging but would be happy to learn.
>>
>
> For svn client debugging, the first thing to do is to build
> --enable-maintainer-mode; this enables stricter compiler warnings
> and internal debugging checks (guarded by '#ifdef SVN_DEBUG').

In this case, the server-side work would seem to be two-fold:

1) look at the send-all property. ra_serf simply omits it, so it
should default to "false"
2) ensure the server responds well to pipelined GET requests. these
can be full-text, or (preferably) delta'd responses

Cheers,
-g

headers

Greg Stein | 10 Feb 2012 03:53

Re: Error While Checking out Git Repository

Hey Nick!

Thanks for looking into this.

On Thu, Feb 9, 2012 at 20:29, Nick Hengeveld <nickh <at> github.com> wrote:
> On Wed, Feb 8, 2012 at 12:59 PM, Greg Stein <gstein <at> gmail.com> wrote:
>
>> So... this isn't really so much about "svn must allow for HTTP/1.0"
>> (yes, we'll work on it), but that github could also provide better
>> service overall by updating its proxy.
>
>
> I've been looking into what it would take to update our proxy.  I have an
> nginx
> configuration in my dev environment that handles requests with chunked
> transfer-encoding and we could feasibly deploy something like this to
> production.

Awesome!

>  However, I'm now getting a different error on the client:
>
> svn: E235000: In file 'subversion/libsvn_ra_serf/update.c' line
> 1769: internal malfunction
> Aborted
>
> From what I can see at
> https://github.com/apache/subversion/blob/trunk/subversion/libsvn_ra_serf/update.c#L1698-1771
> it looks like the client is trying to process an add/open file.  The
> checkout
> I'm testing is small and contains only one file, and this is what the server
> is returning for that file:
>
> <S:add-file name="README.txt" >
> <D:checked-in><D:href>/github/sample/!svn/ver/3/trunk/README.txt</D:href></D:checked-in>
> <S:set-prop name="svn:entry:committed-rev">3</S:set-prop>
> <S:set-prop
> name="svn:entry:uuid">faa06a63-5b49-ecd5-b7fc-ce57b7db7bff</S:set-prop>
> <S:set-prop
> name="svn:entry:committed-date">2012-02-09T22:12:11.000000Z</S:set-prop>
> <S:set-prop name="svn:entry:last-author">subversion.user</S:set-prop>
> <S:txdelta>
> U1ZO
> AQAAMQs4AnicazAEAAEzALIxeJwLycgsVgCixDyFotTElNxUHYXSgpTEktQUhbLMRIVEBUM9c4XknMzUvBI9rsh8PS4AnrgP
> sw==
> </S:txdelta>
> <S:prop><V:md5-checksum>5fa5200f0f095d120c13870e107dfb7a</V:md5-checksum></S:prop>
> </S:add-file>

The S:txdelta element was not requested by the client, so it freaks
out when it sees the unknown element.

Control over whether the txdelta should be sent/not is within the
update-report that the client sends to the server. Take a look at the
"send-all" attribute on the S:update-report in the request. The
txdelta element should only be delivered to the client when
send-all=true.

ra_neon wants the embedded txdelta within one giant response. ra_serf
only wants a barebones "what changed", and then it will issue multiple
GET calls to fetch the deltas individually. serf will pipeline these
GET requests over several connections.

>...

Cheers,
-g

headers

Nick Hengeveld | 14 Feb 2012 06:01

Re: Error While Checking out Git Repository

On Thu, Feb 9, 2012 at 6:53 PM, Greg Stein <gstein <at> gmail.com> wrote:

The S:txdelta element was not requested by the client, so it freaks
out when it sees the unknown element.

Makes sense, we just deployed an update that fixes that and a few other bits that turned up while testing with ra_serf.

The chunked request encoding support has also been deployed, and I've been able to check out and manage repos from 1.7 clients using both ra_neon and ra_serf.

Thanks for all the help, and feel free to follow up if something doesn't look right on our end.

--
For a successful technology, reality must take precedence over public
relations, for nature cannot be fooled.

headers

Greg Stein | 14 Feb 2012 06:08

Re: Error While Checking out Git Repository

On Feb 14, 2012 12:01 AM, "Nick Hengeveld" <nickh <at> github.com> wrote:
>
> On Thu, Feb 9, 2012 at 6:53 PM, Greg Stein <gstein <at> gmail.com> wrote:
>
>> The S:txdelta element was not requested by the client, so it freaks
>> out when it sees the unknown element.
>
>
> Makes sense, we just deployed an update that fixes that and a few other bits that turned up while testing with ra_serf.
>
> The chunked request encoding support has also been deployed, and I've been able to check out and manage repos from 1.7 clients using both ra_neon and ra_serf.
>
> Thanks for all the help, and feel free to follow up if something doesn't look right on our end.

Awesome, Nick! Thanks for the super timely response! I know that svn is a bit fringe for you, which makes it all the better :-)

(now if only codeplex had anywhere near your kickass-ness...)

Cheers,
-g

headers

Nick Hengeveld | 8 Feb 2012 21:34

Re: Error While Checking out Git Repository

Thanks for checking - we want to be fully compatible so any information like this helps.

The checkout URL is the same as the HTTP clone URL, and the .git suffix is optional for both git and svn clients.

I may have misunderstood what's up with the ra_serf library, I thought that was related to HTTP protocol changes introduced in 1.7 and that 1.7 clients would fall back to old behavior for <1.7 servers. I'll read up on this, if you have any pointers to documentation related to ra_serf that would be helpful.

I'm seeing this XML (sans txdelta) in a debug session with a copy of that repo:

<S:add-file name="packages.config" >

<D:checked-in><D:href>/NuGet/PoliteCaptcha/!svn/ver/25/tags/v0.1/Sample/packages.config</D:href></D:checked-in>

<S:set-prop name="svn:entry:committed-rev">25</S:set-prop>

<S:set-prop name="svn:entry:uuid">73fd378b-872a-2eee-a48e-ce57b7db7bff</S:set-prop>

<S:set-prop name="svn:entry:committed-date">2012-02-07T16:47:44.000000Z</S:set-prop>

<S:set-prop name="svn:entry:last-author">anglicangeek</S:set-prop>

<S:txdelta>...</S:txdelta>

<S:prop><V:md5-checksum>f0315b5b2e0ba6a4cb3cde943d8ae9df</V:md5-checksum></S:prop>

</S:add-file>

It's possible that production is returning something different, I'll look into that next.

On Wed, Feb 8, 2012 at 11:47 AM, C. Michael Pilato <cmpilato <at> collab.net> wrote:

I'm happy to try it with 1.7, too, but couldn't easily figure out the
correct checkout URL:

$ svn co https://github.com/NuGet/PoliteCaptcha.git
svn: E175009: Unable to connect to a repository at URL
'https://github.com/NuGet/PoliteCaptcha.git'
svn: E175009: XML parsing failed: (411 Length Required)
$

Ah!! That was with ra_serf. With ra_neon, I get the reported behavior:

$ svn co https://github.com/NuGet/PoliteCaptcha.git
A PoliteCaptcha.git/branches
A PoliteCaptcha.git/tags
A PoliteCaptcha.git/tags/v0.1
[...]
A PoliteCaptcha.git/tags/v0.1/Sample/Web.Release.config
A PoliteCaptcha.git/tags/v0.1/Sample/Web.config
A PoliteCaptcha.git/tags/v0.1/Sample/packages.config
svn: E235000: In file 'subversion/libsvn_wc/wc_db.c' line 1608: assertion
failed (SVN_IS_VALID_REVNUM(changed_rev))
Aborted
$

I'll take a peek with Neon's debugging enabled and see what I can see.

(But please do note that it appears GitHub appears to be incompatible with
Subversion clients which use the newer ra_serf library.)

--
For a successful technology, reality must take precedence over public
relations, for nature cannot be fooled.

headers

C. Michael Pilato | 8 Feb 2012 21:41

Re: Error While Checking out Git Repository

The GitHub server is definitely trying to transmit the svn:entry: property
set ("svn:entry:committed-rev", "svn:entry:committed-date",
"svn:entry:last-author"), as can be seen in your output below.  But there
are some places where it fails to do so.

For example, the directory
"/home/cmpilato/tests/PoliteCaptcha.git/tags/v0.1" is coming down the pipe
without any "svn:entry:" properties at all.  Now, it's permissible for the
author and date to be missing, because that scenario can occur normally
within Subversion due to the user lacking certain access privileges.  But
the 1.7 client codebase now requires that the last-changed revision be valid.

On 02/08/2012 03:34 PM, Nick Hengeveld wrote:
> Thanks for checking - we want to be fully compatible so any information like
> this helps.
> 
> The checkout URL is the same as the HTTP clone URL, and the .git suffix is
> optional for both git and svn clients.
> 
> I may have misunderstood what's up with the ra_serf library, I thought that
> was related to HTTP protocol changes introduced in 1.7 and that 1.7 clients
> would fall back to old behavior for <1.7 servers.  I'll read up on this, if
> you have any pointers to documentation related to ra_serf that would be helpful.
> 
> I'm seeing this XML (sans txdelta) in a debug session with a copy of that repo:
> 
> <S:add-file name="packages.config" >
>    
> <D:checked-in><D:href>/NuGet/PoliteCaptcha/!svn/ver/25/tags/v0.1/Sample/packages.config</D:href></D:checked-in>
>     <S:set-prop name="svn:entry:committed-rev">25</S:set-prop>
>     <S:set-prop
> name="svn:entry:uuid">73fd378b-872a-2eee-a48e-ce57b7db7bff</S:set-prop>
>     <S:set-prop
> name="svn:entry:committed-date">2012-02-07T16:47:44.000000Z</S:set-prop>
>     <S:set-prop name="svn:entry:last-author">anglicangeek</S:set-prop>
>     <S:txdelta>...</S:txdelta>
>    
> <S:prop><V:md5-checksum>f0315b5b2e0ba6a4cb3cde943d8ae9df</V:md5-checksum></S:prop>
> </S:add-file>
> 
> It's possible that production is returning something different, I'll look
> into that next.
> 
> On Wed, Feb 8, 2012 at 11:47 AM, C. Michael Pilato <cmpilato <at> collab.net
> <mailto:cmpilato <at> collab.net>> wrote:
> 
>     I'm happy to try it with 1.7, too, but couldn't easily figure out the
>     correct checkout URL:
> 
>     $ svn co https://github.com/NuGet/PoliteCaptcha.git
>     svn: E175009: Unable to connect to a repository at URL
>     'https://github.com/NuGet/PoliteCaptcha.git'
>     svn: E175009: XML parsing failed: (411 Length Required)
>     $
> 
>     Ah!!  That was with ra_serf.  With ra_neon, I get the reported behavior:
> 
>     $ svn co https://github.com/NuGet/PoliteCaptcha.git
>     A    PoliteCaptcha.git/branches
>     A    PoliteCaptcha.git/tags
>     A    PoliteCaptcha.git/tags/v0.1
>     [...]
>     A    PoliteCaptcha.git/tags/v0.1/Sample/Web.Release.config
>     A    PoliteCaptcha.git/tags/v0.1/Sample/Web.config
>     A    PoliteCaptcha.git/tags/v0.1/Sample/packages.config
>     svn: E235000: In file 'subversion/libsvn_wc/wc_db.c' line 1608: assertion
>     failed (SVN_IS_VALID_REVNUM(changed_rev))
>     Aborted
>     $
> 
>     I'll take a peek with Neon's debugging enabled and see what I can see.
> 
>     (But please do note that it appears GitHub appears to be incompatible with
>     Subversion clients which use the newer ra_serf library.)
> 
> 
> -- 
> For a successful technology, reality must take precedence over public
> relations, for nature cannot be fooled.

--

-- 
C. Michael Pilato <cmpilato <at> collab.net>
CollabNet   <>   www.collab.net   <>   Distributed Development On Demand

headers

Nick Hengeveld | 8 Feb 2012 23:06

Re: Error While Checking out Git Repository

That pointed me in the right direction, thanks.

It was a bug related to tag versions, and we just deployed a fix. I was able to do a checkout with a 1.7 client, let me know if you have any problems.

On Wed, Feb 8, 2012 at 12:41 PM, C. Michael Pilato <cmpilato <at> collab.net> wrote:

The GitHub server is definitely trying to transmit the svn:entry: property
set ("svn:entry:committed-rev", "svn:entry:committed-date",
"svn:entry:last-author"), as can be seen in your output below. But there
are some places where it fails to do so.

For example, the directory
"/home/cmpilato/tests/PoliteCaptcha.git/tags/v0.1" is coming down the pipe
without any "svn:entry:" properties at all. Now, it's permissible for the
author and date to be missing, because that scenario can occur normally
within Subversion due to the user lacking certain access privileges. But
the 1.7 client codebase now requires that the last-changed revision be valid.

--
For a successful technology, reality must take precedence over public
relations, for nature cannot be fooled.

headers

Philip Martin | 9 Feb 2012 01:04

Re: Error While Checking out Git Repository

Nick Hengeveld <nickh <at> github.com> writes:

> It was a bug related to tag versions, and we just deployed a fix.  I was
> able to do a checkout with a 1.7 client, let me know if you have any
> problems.

Oooh!  Nasty client crash:

A    wc/tags/v0.3/3rdParty/xunit/xunit.extensions.xml
A    wc/tags/v0.3/3rdParty/xunit/xunit.gui.clr4.exe
*** glibc detected *** /home/pm/sw/subversion/obj/subversion/svn/.libs/lt-svn: corrupted
double-linked list: 0x0000000001efb360 ***
======= Backtrace: =========
/lib/libc.so.6(+0x71bd6)[0x7facf7254bd6]
/lib/libc.so.6(+0x736a0)[0x7facf72566a0]
/lib/libc.so.6(cfree+0x6c)[0x7facf725994c]
/usr/local/httpd-2.2/lib/libapr-1.so.0(+0x1abb9)[0x7facf79a4bb9]
/usr/local/httpd-2.2/lib/libapr-1.so.0(+0x1acfe)[0x7facf79a4cfe]
/home/pm/sw/subversion/obj/subversion/libsvn_delta/.libs/libsvn_delta-1.so.0(+0x1182e)[0x7facf828282e]

valgrind shows:

A    wc/tags/v0.1/3rdParty/xunit/xunit.extensions.dll
A    wc/tags/v0.1/3rdParty/xunit/xunit.extensions.xml
==3089== Invalid write of size 1
==3089==    at 0x5A16FC1: decode_line (svn_base64.c:384)
==3089==    by 0x5A17079: decode_bytes (svn_base64.c:421)
==3089==    by 0x5A1721C: decode_data (svn_base64.c:465)
==3089==    by 0x5A0F316: svn_stream_write (stream.c:161)
==3089==    by 0x70BE98A: cdata_handler (fetch.c:1974)
==3089==    by 0x70D22CF: wrapper_cdata_cb (util.c:1116)
==3089==    by 0x60AB5FB: ??? (in /usr/lib/libneon.so.27.2.3)
==3089==    by 0x95C14AE: xmlParseCharData (in /usr/lib/libxml2.so.2.7.8)
==3089==    by 0x95CBC34: ??? (in /usr/lib/libxml2.so.2.7.8)
==3089==    by 0x95CC30A: xmlParseChunk (in /usr/lib/libxml2.so.2.7.8)
==3089==    by 0x60AB75E: ne_xml_parse (in /usr/lib/libneon.so.27.2.3)
==3089==    by 0x70D286E: cancellation_callback (util.c:1241)
==3089==  Address 0xbff3700 is 0 bytes after a block of size 3,936 alloc'd
==3089==    at 0x4C244E8: malloc (vg_replace_malloc.c:236)
==3089==    by 0x5E7C130: pool_alloc (apr_pools.c:1463)
==3089==    by 0x5A17851: my__realloc (svn_string.c:55)
==3089==    by 0x5A18262: svn_stringbuf_ensure (svn_string.c:468)
==3089==    by 0x5A17049: decode_bytes (svn_base64.c:413)
==3089==    by 0x5A1721C: decode_data (svn_base64.c:465)
==3089==    by 0x5A0F316: svn_stream_write (stream.c:161)
==3089==    by 0x70BE98A: cdata_handler (fetch.c:1974)
==3089==    by 0x70D22CF: wrapper_cdata_cb (util.c:1116)
==3089==    by 0x60AB5FB: ??? (in /usr/lib/libneon.so.27.2.3)
==3089==    by 0x95C14AE: xmlParseCharData (in /usr/lib/libxml2.so.2.7.8)
==3089==    by 0x95CBC34: ??? (in /usr/lib/libxml2.so.2.7.8)

This solves the problem:

Index: ../src/subversion/libsvn_subr/svn_base64.c
===================================================================
--- ../src/subversion/libsvn_subr/svn_base64.c  (revision 1242045)
+++ ../src/subversion/libsvn_subr/svn_base64.c  (working copy)
 <at>  <at>  -410,7 +410,7  <at>  <at> 
   /* Resize the stringbuf to make room for the (approximate) size of
      output, to avoid repeated resizes later.
      The optimizations in decode_line rely on no resizes being necessary! */
-  svn_stringbuf_ensure(str, str->len + (len / 4) * 3 + 3);
+  svn_stringbuf_ensure(str, str->len + (len / 4) * 3 + 4);

   while ( !*done && p < end )
     {

but I don't really understand the comment.  There doesn't seem to be
anything "approximate" about this calculation?

--

-- 
uberSVN: Apache Subversion Made Easy
http://www.uberSVN.com

headers

Peter Samuelson | 9 Feb 2012 19:01

Re: Error While Checking out Git Repository


[Philip Martin]
> --- ../src/subversion/libsvn_subr/svn_base64.c  (revision 1242045)
> +++ ../src/subversion/libsvn_subr/svn_base64.c  (working copy)
>  <at>  <at>  -410,7 +410,7  <at>  <at> 
>    /* Resize the stringbuf to make room for the (approximate) size of
>       output, to avoid repeated resizes later.
>       The optimizations in decode_line rely on no resizes being necessary! */
> -  svn_stringbuf_ensure(str, str->len + (len / 4) * 3 + 3);
> +  svn_stringbuf_ensure(str, str->len + (len / 4) * 3 + 4);
>  
>    while ( !*done && p < end )
>      {
> 
> but I don't really understand the comment.  There doesn't seem to be
> anything "approximate" about this calculation?

It is approximate because the encoded length is always a multiple of 4,
even if the input length was not a multiple of 3.  (This is why so many
base64 strings end in = or ==; those are padding to a 4-byte output
group.)  The decoder doesn't yet know if the final group of 4 bytes
will decode to 1, 2 or 3 bytes of plaintext.

headers

Philip Martin | 9 Feb 2012 19:08

Re: Error While Checking out Git Repository

Peter Samuelson <peter <at> p12n.org> writes:

> [Philip Martin]
>> --- ../src/subversion/libsvn_subr/svn_base64.c  (revision 1242045)
>> +++ ../src/subversion/libsvn_subr/svn_base64.c  (working copy)
>>  <at>  <at>  -410,7 +410,7  <at>  <at> 
>>    /* Resize the stringbuf to make room for the (approximate) size of
>>       output, to avoid repeated resizes later.
>>       The optimizations in decode_line rely on no resizes being necessary! */
>> -  svn_stringbuf_ensure(str, str->len + (len / 4) * 3 + 3);
>> +  svn_stringbuf_ensure(str, str->len + (len / 4) * 3 + 4);
>>  
>>    while ( !*done && p < end )
>>      {
>> 
>> but I don't really understand the comment.  There doesn't seem to be
>> anything "approximate" about this calculation?
>
> It is approximate because the encoded length is always a multiple of 4,
> even if the input length was not a multiple of 3.  (This is why so many
> base64 strings end in = or ==; those are padding to a 4-byte output
> group.)  The decoder doesn't yet know if the final group of 4 bytes
> will decode to 1, 2 or 3 bytes of plaintext.

I believe approximate is still wrong.  It's supposed to be an accurate
calculation of the maximum output length.  If some approximate value is
used and the output is longer than that then the code will crash.

--

-- 
uberSVN: Apache Subversion Made Easy
http://www.uberSVN.com

headers

Stefan Fuhrmann | 12 Feb 2012 13:56

Re: Error While Checking out Git Repository

On 09.02.2012 19:08, Philip Martin wrote:
> Peter Samuelson<peter <at> p12n.org>  writes:
>
>> [Philip Martin]
>>> --- ../src/subversion/libsvn_subr/svn_base64.c  (revision 1242045)
>>> +++ ../src/subversion/libsvn_subr/svn_base64.c  (working copy)
>>>  <at>  <at>  -410,7 +410,7  <at>  <at> 
>>>     /* Resize the stringbuf to make room for the (approximate) size of
>>>        output, to avoid repeated resizes later.
>>>        The optimizations in decode_line rely on no resizes being necessary! */
>>> -  svn_stringbuf_ensure(str, str->len + (len / 4) * 3 + 3);
>>> +  svn_stringbuf_ensure(str, str->len + (len / 4) * 3 + 4);

Thanks for fixing that!
>>>
>>>     while ( !*done&&  p<  end )
>>>       {
>>>
>>> but I don't really understand the comment.  There doesn't seem to be
>>> anything "approximate" about this calculation?
>> It is approximate because the encoded length is always a multiple of 4,
>> even if the input length was not a multiple of 3.  (This is why so many
>> base64 strings end in = or ==; those are padding to a 4-byte output
>> group.)  The decoder doesn't yet know if the final group of 4 bytes
>> will decode to 1, 2 or 3 bytes of plaintext.
> I believe approximate is still wrong.  It's supposed to be an accurate
> calculation of the maximum output length.  If some approximate value is
> used and the output is longer than that then the code will crash.
>
Sorry about my imprecise commentary in the code.
It should have read "close upper boundary" instead
of "approximate".

As already discussed on IRC, the real cause here
is that _ensure() should follow the first invariant listed
in svn_string.h by allocating *additional* space for
the terminating 0. In particular svn_stringbuf_ensure()
is inconsistent with svn_stringbuf_create_ensure()
in this respect.

Fixing that is somewhat hard as all callers need to
be inspected for potential side-effects. _ensure2()
would probably be the better solution.

-- Stefan^2.

headers

Julian Foad | 14 Feb 2012 13:29

Re: Error While Checking out Git Repository

Stefan Fuhrmann wrote:

> [...] As already discussed on IRC, the real cause here
> is that _ensure() should follow the first invariant listed
> in svn_string.h by allocating *additional* space for
> the terminating 0. In particular svn_stringbuf_ensure()
> is inconsistent with svn_stringbuf_create_ensure()
> in this respect.
> 
> Fixing that is somewhat hard as all callers need to
> be inspected for potential side-effects. _ensure2()
> would probably be the better solution.

I'm following up with a patch in a new thread "[PATCH] String creation -- ensuring space for the NUL", <http://svn.haxx.se/dev/archive-2012-02/0415.shtml>.

- Julian

headers

Daniel Shahaf | 8 Feb 2012 22:47

Re: Error While Checking out Git Repository

Nick Hengeveld wrote on Wed, Feb 08, 2012 at 12:34:52 -0800:
> Thanks for checking - we want to be fully compatible so any information
> like this helps.
> 
> The checkout URL is the same as the HTTP clone URL, and the .git suffix is
> optional for both git and svn clients.
> 
> I may have misunderstood what's up with the ra_serf library, I thought that
> was related to HTTP protocol changes introduced in 1.7 and that 1.7 clients
> would fall back to old behavior for <1.7 servers.  I'll read up on this, if

Yes.  1.7 clients speak the 1.6 protocol with servers that have
"SVNAdvertiseV2Protocol off" in their config.

(ra_neon supports a subset of the HTTPv2 features that ra_serf does.)

> you have any pointers to documentation related to ra_serf that would be
> helpful.

There is an overview in the 1.7 release notes (look for "serf" and
"HTTPv2"), and a more technical overview might be found in
notes/http-and-webdav/.  (I'm not sure how up-to-date the latter is
wrt the implementation.)

HTH

headers

Nick Hengeveld | 8 Feb 2012 23:08

Re: Error While Checking out Git Repository

I think the document that I'd read previously was this:

http://svn.apache.org/repos/asf/subversion/trunk/notes/http-and-webdav/http-protocol-v2.txt

In particular, this section:

ra_serf will send an OPTIONS request when creating a new ra_session. mod_dav_svn will send back what it already sends now, but will also return new information as custom headers in the OPTIONS response: SVN-Youngest-Rev: REV SVN-Me-Resource: /REPOS-ROOT/!svn/me Additionally, this response will contain some new URL stub values: SVN-Rev-Stub: /REPOS-ROOT/!svn/rev SVN-Rev-Root-Stub: /REPOS-ROOT/!svn/rvr SVN-Txn-Stub: /REPOS-ROOT/!svn/txn SVN-Txn-Root-Stub: /REPOS-ROOT/!svn/txr SVN-VTxn-Stub: /REPOS-ROOT/!svn/vtxn SVN-VTxn-Root-Stub: /REPOS-ROOT/!svn/vtxr The presence of these new stubs (which can be appended to by the client to create full-fledged resource URLs) tells ra_serf that this is a new server, and that the new streamlined HTTP protocol can be used. ra_serf then caches them in the ra_session object. If these new OPTIONS responses are not returned, ra_serf falls back to 'classic' DeltaV protocol.

We're not returning the new custom headers, so I'd expected that ra_serf would fall back.

Do you know whether there's something else that the "SVNAdvertiseV2Protocol off" config directive does to OPTIONS or other responses?

On Wed, Feb 8, 2012 at 1:47 PM, Daniel Shahaf <danielsh <at> elego.de> wrote:

Nick Hengeveld wrote on Wed, Feb 08, 2012 at 12:34:52 -0800:
> Thanks for checking - we want to be fully compatible so any information
> like this helps.
>
> The checkout URL is the same as the HTTP clone URL, and the .git suffix is
> optional for both git and svn clients.
>
> I may have misunderstood what's up with the ra_serf library, I thought that
> was related to HTTP protocol changes introduced in 1.7 and that 1.7 clients
> would fall back to old behavior for <1.7 servers. I'll read up on this, if

Yes. 1.7 clients speak the 1.6 protocol with servers that have
"SVNAdvertiseV2Protocol off" in their config.

(ra_neon supports a subset of the HTTPv2 features that ra_serf does.)

> you have any pointers to documentation related to ra_serf that would be
> helpful.

There is an overview in the 1.7 release notes (look for "serf" and
"HTTPv2"), and a more technical overview might be found in
notes/http-and-webdav/. (I'm not sure how up-to-date the latter is
wrt the implementation.)

HTH

--
For a successful technology, reality must take precedence over public
relations, for nature cannot be fooled.

headers

Philip Martin | 9 Feb 2012 01:00

Re: Error While Checking out Git Repository

Nick Hengeveld <nickh <at> github.com> writes:

> We're not returning the new custom headers, so I'd expected that ra_serf
> would fall back.

As Greg explained serf doesn't get this far, probably because the
request with "Transfer-encoding: chunked" is rejected by the server.
Using neon, which also has v2 support, does work and does fall back on
the v1 protocol.

--

-- 
uberSVN: Apache Subversion Made Easy
http://www.uberSVN.com

headers

TheBlueSky .Net | 9 Feb 2012 21:19

Re: Error While Checking out Git Repository

Sorry if I have posted this in the not-right place :)

It's not like someone is going to die if I didn't checked out a git repo from SVN client; I was only testing it as other people have done it successfully.

Anyway, the only reason I posted in this mailing list is because the error message told me so... and to be nice, I did :)

Take care

On 8 February 2012 00:20, C. Michael Pilato <cmpilato <at> collab.net> wrote:

On 02/07/2012 02:49 PM, TheBlueSky .Net wrote:
> Hello,
>
> I was checking out a Git repository
> (https://github.com/NuGet/PoliteCaptcha.git) when I encountered the
> following error near the end of getting all the files:
>
> Subversion encountered a serious problem.
> Please take the time to report this on the Subversion mailing list
> with as much information as possible about what
> you were trying to do.
> But please first search the mailing list archives for the error message
> to avoid reporting the same problem repeatedly.
> You can find the mailing list archives at
> http://subversion.apache.org/mailing-lists.html
>
> Subversion reported the following
> (you can copy the content of this dialog
> to the clipboard using Ctrl-C):
>
> In file
> 'D:\Development\SVN\Releases\TortoiseSVN-1.7.1\ext\subversion\subversion\libsvn_wc\wc_db.c'
> line 1588: assertion failed (SVN_IS_VALID_REVNUM(changed_rev))
>
> And if someone is wondering, checking out from github using SVN client is
> supported.

Well, supported by whom?

This community has only signed up to support Subversion's command-line
client checking out from a Subversion mod_dav_svn or svnserve server. I
must assume that github has put together their own Subversion-esque server
software which communicates with git repositories. Obviously, we expect our
client to behave properly against any server that wields our protocols
correctly, but first we need to establish that GitHub's server is such a server.

At first blush, it would appear that the GitHub server didn't transmit a
valid svn:entry:committed-rev property value for one of the directories your
checkout is attempting to register with Subversion's working copy
administration library. If this problem is repeatable, it would be great to
get a network trace of the failing checkout.

--
C. Michael Pilato <cmpilato <at> collab.net>
CollabNet <> www.collab.net <> Distributed Development On Demand

headers

Peter Samuelson | 9 Feb 2012 22:48

Re: Error While Checking out Git Repository


[TheBlueSky .Net]
> Sorry if I have posted this in the not-right place :)

That's fine.  It was an assertion failure which, by definition, is a
bug in the Subversion client.  The bug was triggered by a bug on the
Github server side, but it's still a bug in the client.

Also, happily, someone at Github fixed their server-side bug as a
result of your report.

So, thanks!
--

-- 
Peter Samuelson | org-tld!p12n!peter | http://p12n.org/