headers
Luke Kenneth Casson Leighton | 1 Sep 2010 18:40
Favicon

Distributed Debian Distribution Development

around three years ago i wrote an article recommending that debian
move step-by-step towards distributed peer-to-peer infrastructure,
thus reducing the reliance on server infrastructure, thus potentially
allowing sponsorship funds and resources to be retargetted to other
areas which would improve the debian distribution.

mostly that was words, not actual code, and, before getting all upset
at how little progress has been made (cameron dale's fantastic apt-p2p
work literally being the only exception ) i decided a few days ago to
do something about that situation, so that i wouldn't come across as
being a complete spongeing whining knob.

my contribution is to prove that a combination of git and bittorrent
is actually possible:
http://gitorious.org/python-libbittorrent/pybtlib

(anyone wishing to help / contribute, do contact me: i'll happily give
you access to the repository if you can spot the irony of this
statement)

it's turned out to be much, much simpler than i thought, by the simple
expedient of turning git commits into a "virtual filesystem" where the
git-pack-objects are stored as files, named by their commit reference.
 hooking into the bittornado client's "file close" operation is enough
to fire off a test for whether the pack object is valid (check its
signature at the beginning, and the SHA-1 signature at the end), and
if it is, to run a "git unpack" operation.

what are the implications, and why is combining git with bittorrent a
big hairy deal?
(Continue reading)

Permalink | Reply |
headers
Yaroslav Halchenko | 1 Sep 2010 19:47
Gravatar

Re: Distributed Debian Distribution Development

Just - Wow... thanks!  

Hopefully digesting of this tasty post would not cause too much of farting ;-)

seems might be worth adding (if I am not missing the point), then the
concept of "derivatives" would then converge finally to a more
digestible, more manageable, and thus more robust mechanism of
branches... ?

On Wed, 01 Sep 2010, Luke Kenneth Casson Leighton wrote:

> ghandi said "be the change you want to see in the world".  this
> translates, roughly, to "eat your own dogfood".

> so - something to think about.

--

-- 
                                  .-.
=------------------------------   /v\  ----------------------------=
Keep in touch                    // \\     (yoh <at> |www.)onerussian.com
Yaroslav Halchenko              /(   )\               ICQ#: 60653192
                   Linux User    ^^-^^    [175555]
Permalink | Reply |
headers
Luke Kenneth Casson Leighton | 2 Sep 2010 12:55
Favicon

Re: Distributed Debian Distribution Development

On Wed, Sep 1, 2010 at 6:47 PM, Yaroslav Halchenko
<debian <at> onerussian.com> wrote:
> Just - Wow... thanks!
>
> Hopefully digesting of this tasty post would not cause too much of farting ;-)

 :)

> seems might be worth adding (if I am not missing the point), then the
> concept of "derivatives" would then converge finally to a more
> digestible, more manageable, and thus more robust mechanism of
> branches... ?

 ahh, you're still using "git": i'll be doing nothing more fancy than
creating a "git-remote-helper" which will add a protocol gitp2p:// or
gpp:// or something like that.

 so you'll still need to understand the concept of branches and
branching: it's just that you'd be able to grab them from peers rather
than being reliant on a central server.

 imagine a situation where you meet a fellow debian developer(s) and
you both(all) happen to be on holiday or just... somewhere where
connectivity is patchy.  i envisage a situation where both (or all)
publish their own trackers, and they can simply share whatever bits of
git repositories they (individually) happened to be "most interested
in, personally", with everyone else.

for example, one person happens to have an OCD-fueled interest in
keeping up-to-date with every single mailing list, whilst another
(Continue reading)

Permalink | Reply |
headers
Oscar Morante | 2 Sep 2010 14:44
Picon

Re: Distributed Debian Distribution Development

Have you seen this project [1]? It looks like they have been already
thinking about the git+bittorrent idea.

[1] http://code.google.com/p/gittorrent/

--

-- 
Oscar Morante
"Self-education is, I firmly believe, the only kind of education there is."
                                                          -- Isaac Asimov.
Permalink | Reply |
headers
Luke Kenneth Casson Leighton | 2 Sep 2010 15:29
Favicon

Re: Distributed Debian Distribution Development

On Thu, Sep 2, 2010 at 1:44 PM, Oscar Morante <spacepluk <at> gmail.com> wrote:
> Have you seen this project [1]? It looks like they have been already
> thinking about the git+bittorrent idea.
>
> [1] http://code.google.com/p/gittorrent/

 yes.  it's effectively shelved.  the name "gittorrent" was abandoned
and the name "mirrorsync" selected, because the people working on it
decided that bittorrent was an inappropriate protocol to use.  i got
them some slashdot coverage.  but, ultimately, i disagreed with them
that bittorrent isn't an appropriate protocol, so that's why i did the
thingy and proved that it's an effective transfer / distribution
mechanism.  thingy.  haven't even picked a name for it! :)
suggestions welcome.

 i want to see how far i can get away with leaving the bittorrent
protocol _entirely_ unchanged, by virtue of doing everything through
this "vfs layer" jobbie.  only if it becomes absolutely absolutely
necessary, _then_ start making changes.

good reasons to make changes:

a) the 256k chunk size becomes blindingly obviously completely
unacceptable.  for example: cameron dale did a study of the .deb
archives and found that a very large percentage are under 32k in size.
 this was the primary reason why he abandoned the bittorrent protocol
(baby? bathwater?? *sigh*...) even after modifying it to be able to
negotiate chunk sizes and he designed apt-p2p instead.

b) ISPs start doing packet-filtering (fuckers) so it becomes necessary
(Continue reading)

Permalink | Reply |
headers
Boyd Stephen Smith Jr. | 1 Sep 2010 20:20
X-Face
Face
Favicon

Re: Distributed Debian Distribution Development

In <AANLkTinehMUuBSmLyazux1KaKHfaqGuDrNg0q+aznNkb <at> mail.gmail.com>, Luke 
Kenneth Casson Leighton wrote:
>around three years ago i wrote an article recommending that debian
>move step-by-step towards distributed peer-to-peer infrastructure,
>thus reducing the reliance on server infrastructure, thus potentially
>allowing sponsorship funds and resources to be retargetted to other
>areas which would improve the debian distribution.
>
>mostly that was words, not actual code, and, before getting all upset
>at how little progress has been made (cameron dale's fantastic apt-p2p
>work literally being the only exception ) i decided a few days ago to
>do something about that situation, so that i wouldn't come across as
>being a complete spongeing whining knob.
>
>my contribution is to prove that a combination of git and bittorrent
>is actually possible:
>http://gitorious.org/python-libbittorrent/pybtlib
>
>(anyone wishing to help / contribute, do contact me: i'll happily give
>you access to the repository if you can spot the irony of this
>statement)
>
>it's turned out to be much, much simpler than i thought, by the simple
>expedient of turning git commits into a "virtual filesystem" where the
>git-pack-objects are stored as files, named by their commit reference.
> hooking into the bittornado client's "file close" operation is enough
>to fire off a test for whether the pack object is valid (check its
>signature at the beginning, and the SHA-1 signature at the end), and
>if it is, to run a "git unpack" operation.
(Continue reading)

Permalink | Reply |
headers
Arthur Machlas | 1 Sep 2010 20:29
Picon
Gravatar

Re: Distributed Debian Distribution Development

I'm glad this was cross posted otherwise I would've missed it. Even if
there are technical hurdles it's an exciting idea and I'm looking
forward to reading the devel mailing list for follow-ups. The point
about eating your own dog food is well made i thought, though whether
there is any interest in moving this forward duing a squeeze freeze
will be interesting to see. Hopefully this also makes it into the
debian news to give it some more reach for discussion.

Personally though, as much as I find the idea interesting, I won't be
having a torrent application running all the time to assist with the
project. Nor would I want to get mail through torrent, or a wiki... I
guess us "users" will have a better idea how such things can look once
diaspora is alpha released in October - November or so.
Permalink | Reply |
headers
Olivier Berger | 2 Sep 2010 16:17
Picon
Gravatar

Distributed bugtracker - Was: Re: Distributed Debian Distribution Development

Hi.

Le mercredi 01 septembre 2010 à 17:40 +0100, Luke Kenneth Casson
Leighton a écrit :

> 
> * again: ikiwiki or something similar could well be used as the basis
> for a distributed bugtracker.
> 

I'd suggest you have a look at the SD tool and the incoming debbugs
support for it developped by Christine Spang. See for instance a report
I've made at [0].

I'm not sure the distribution involved with SD involves basic
replication as in P2P systems like bittorrent, but there are strong
links.

[0] : http://www-public.it-sudparis.eu/~berger_o/weblog/2010/08/12/great-talk-distributed-bug-tracking-for-debian-with-sd-at-debconf/

Best regards,
--

-- 
Olivier BERGER <olivier.berger <at> it-sudparis.eu>
http://www-public.it-sudparis.eu/~berger_o/ - OpenPGP-Id: 2048R/5819D7E8
Ingénieur Recherche - Dept INF
Institut TELECOM, SudParis (http://www.it-sudparis.eu/), Evry (France)
Permalink | Reply |
headers
Olivier Berger | 2 Sep 2010 16:38
Picon
Gravatar

Re: Distributed Debian Distribution Development

Hi.

Le mercredi 01 septembre 2010 à 17:40 +0100, Luke Kenneth Casson
Leighton a écrit :

> i leave you with this: the idea of the "Freedom Box" caused quite a
> stir at debconf2010, but i honestly doubt that, without any experience
> of getting *yourselves* off of the client-server paradigm, there will
> be any success in getting *the average person* off of the
> client-server paradigm.  so i believe that by converting the debian
> development process and infrastructure to peer-to-peer distributed
> infrastructure *first*, and by creating aggregation and migration
> infrastructure for *yourselves*, the people who do that will be in a
> much stronger position to help convert other types of internet
> services to a distributed architecture.  and, will likely have written
> a good proportion of the necessary infrastructure along the way.

Regarding distributed development tools / forges, I'd suggest to also
have a look at developments happening on Savane, which will be rewritten
in a way which seems to be inspired by Moglen's vision also :
http://2010.rmll.info/A-new-Savane.html (slides 10+)

A video recording of the talk will be available soon.

Best regards,
--

-- 
Olivier BERGER <olivier.berger <at> it-sudparis.eu>
http://www-public.it-sudparis.eu/~berger_o/ - OpenPGP-Id: 2048R/5819D7E8
Ingénieur Recherche - Dept INF
Institut TELECOM, SudParis (http://www.it-sudparis.eu/), Evry (France)
(Continue reading)

Permalink | Reply |

Gmane