headers
ken cochrane | 15 Apr 2012 15:07
Picon
Gravatar

PyPI mirrors are all up to date

It looks like it took a little while, but all of the PyPI mirrors are
now up to date. Thank you to everyone that helped get those back up to
date. If you want to check out the current status I created this
little website to make it easier to see where things stand mirror
wise. I'm going to be adding a few more things, but it is fully useful
right now. If you have any suggestions, please let me know.

http://www.pypi-mirrors.org

If you run an unofficial PyPI mirror and want it added to the list,
just let me know.

Thanks,
Ken Cochrane
 <at> KenCochrane
Permalink | Reply |
headers
Yuval Greenfield | 15 Apr 2012 16:35
Picon
Gravatar

Re: PyPI mirrors are all up to date

What's the ranking between Excellent, Great and Awesome?

On Sun, Apr 15, 2012 at 4:07 PM, ken cochrane <kencochrane <at> gmail.com> wrote:
It looks like it took a little while, but all of the PyPI mirrors are
now up to date. Thank you to everyone that helped get those back up to
date. If you want to check out the current status I created this
little website to make it easier to see where things stand mirror
wise. I'm going to be adding a few more things, but it is fully useful
right now. If you have any suggestions, please let me know.

http://www.pypi-mirrors.org


If you run an unofficial PyPI mirror and want it added to the list,
just let me know.

Thanks,
Ken Cochrane
<at> KenCochrane
_______________________________________________
Catalog-SIG mailing list
Catalog-SIG <at> python.org
http://mail.python.org/mailman/listinfo/catalog-sig

_______________________________________________
Catalog-SIG mailing list
Catalog-SIG <at> python.org
http://mail.python.org/mailman/listinfo/catalog-sig
Permalink | Reply |
headers
Ken Cochrane | 15 Apr 2012 16:44
Picon
Gravatar

Re: PyPI mirrors are all up to date

Yeah I need to clean that up and make it more standard but this is how it works now.

Age < 5 min = excellent
Age > 5 min and age < 15 min = awesome
Age > 15 min and age < 1 hour = great.
Age > 1 hour and age <  6 hour = good
Age > 6 hours and age < 12 hour = OK
Age > 12 hours and age < 1 day = getting stale
Age > 1 day = out of date

Feel free to make suggestions on how to improve.

Ken

Sent from my iPhone

On Apr 15, 2012, at 10:35 AM, Yuval Greenfield <ubershmekel <at> gmail.com> wrote:

What's the ranking between Excellent, Great and Awesome?

On Sun, Apr 15, 2012 at 4:07 PM, ken cochrane <kencochrane <at> gmail.com> wrote:
It looks like it took a little while, but all of the PyPI mirrors are
now up to date. Thank you to everyone that helped get those back up to
date. If you want to check out the current status I created this
little website to make it easier to see where things stand mirror
wise. I'm going to be adding a few more things, but it is fully useful
right now. If you have any suggestions, please let me know.

http://www.pypi-mirrors.org


If you run an unofficial PyPI mirror and want it added to the list,
just let me know.

Thanks,
Ken Cochrane
<at> KenCochrane
_______________________________________________
Catalog-SIG mailing list
Catalog-SIG <at> python.org
http://mail.python.org/mailman/listinfo/catalog-sig

_______________________________________________
Catalog-SIG mailing list
Catalog-SIG <at> python.org
http://mail.python.org/mailman/listinfo/catalog-sig
Permalink | Reply |
headers
Hanno Schlichting | 15 Apr 2012 17:11
Picon
Gravatar

Re: PyPI mirrors are all up to date

On Sun, Apr 15, 2012 at 4:44 PM, Ken Cochrane <kencochrane <at> gmail.com> wrote:
> Yeah I need to clean that up and make it more standard but this is how it
> works now.
>
> Age < 5 min = excellent
> Age > 5 min and age < 15 min = awesome
> Age > 15 min and age < 1 hour = great.
> Age > 1 hour and age <  6 hour = good
> Age > 6 hours and age < 12 hour = OK
> Age > 12 hours and age < 1 day = getting stale
> Age > 1 day = out of date
>
> Feel free to make suggestions on how to improve.

My suggestion, keep it simple:

Age < 15 minutes - green (fresh)
Age < 1 day - yellow (oldish)
Age > 1 day - red (old)

Apache and CPAN mirrors are much more forgiving.

http://www.apache.org/mirrors/#age-histogram

< 30 hours - green
< 54 hours - yellow
> 54 hours - red

http://mirrors.cpan.org/#age-histogram

< 2 days green
< 4days - yellow
> 4 days - red

Hanno
_______________________________________________
Catalog-SIG mailing list
Catalog-SIG <at> python.org
http://mail.python.org/mailman/listinfo/catalog-sig
Permalink | Reply |
headers
ken cochrane | 16 Apr 2012 04:09
Picon
Gravatar

Re: PyPI mirrors are all up to date

I have taken your advice and I have updated it so that it is a little
clearer. I changed the times a little but I stuck to the 3 statuses
that you proposed.

I also added a bunch of other features to the site, so feel free to
check it out and let me know if you can think of anything else.

http://www.pypi-mirrors.org

On Sun, Apr 15, 2012 at 11:11 AM, Hanno Schlichting <hanno <at> hannosch.eu> wrote:
> On Sun, Apr 15, 2012 at 4:44 PM, Ken Cochrane <kencochrane <at> gmail.com> wrote:
>> Yeah I need to clean that up and make it more standard but this is how it
>> works now.
>>
>> Age < 5 min = excellent
>> Age > 5 min and age < 15 min = awesome
>> Age > 15 min and age < 1 hour = great.
>> Age > 1 hour and age <  6 hour = good
>> Age > 6 hours and age < 12 hour = OK
>> Age > 12 hours and age < 1 day = getting stale
>> Age > 1 day = out of date
>>
>> Feel free to make suggestions on how to improve.
>
> My suggestion, keep it simple:
>
> Age < 15 minutes - green (fresh)
> Age < 1 day - yellow (oldish)
> Age > 1 day - red (old)
>
> Apache and CPAN mirrors are much more forgiving.
>
> http://www.apache.org/mirrors/#age-histogram
>
> < 30 hours - green
> < 54 hours - yellow
>> 54 hours - red
>
> http://mirrors.cpan.org/#age-histogram
>
> < 2 days green
> < 4days - yellow
>> 4 days - red
>
> Hanno
Permalink | Reply |
headers
Andreas Jung | 16 Apr 2012 05:33

Re: PyPI mirrors are all up to date


This here is weird

c.pypi.python.org	Russian Federation

That's my server it it is running in Germany :)

-aj

ken cochrane wrote:
> I have taken your advice and I have updated it so that it is a
> little clearer. I changed the times a little but I stuck to the 3
> statuses that you proposed.
> 
> I also added a bunch of other features to the site, so feel free to 
> check it out and let me know if you can think of anything else.
> 
> http://www.pypi-mirrors.org
> 
> 
> On Sun, Apr 15, 2012 at 11:11 AM, Hanno Schlichting
> <hanno <at> hannosch.eu> wrote:
>> On Sun, Apr 15, 2012 at 4:44 PM, Ken Cochrane
>> <kencochrane <at> gmail.com> wrote:
>>> Yeah I need to clean that up and make it more standard but this
>>> is how it works now.
>>> 
>>> Age < 5 min = excellent Age > 5 min and age < 15 min = awesome 
>>> Age > 15 min and age < 1 hour = great. Age > 1 hour and age <  6
>>> hour = good Age > 6 hours and age < 12 hour = OK Age > 12 hours
>>> and age < 1 day = getting stale Age > 1 day = out of date
>>> 
>>> Feel free to make suggestions on how to improve.
>> My suggestion, keep it simple:
>> 
>> Age < 15 minutes - green (fresh) Age < 1 day - yellow (oldish) Age
>> > 1 day - red (old)
>> 
>> Apache and CPAN mirrors are much more forgiving.
>> 
>> http://www.apache.org/mirrors/#age-histogram
>> 
>> < 30 hours - green < 54 hours - yellow
>>> 54 hours - red
>> http://mirrors.cpan.org/#age-histogram
>> 
>> < 2 days green < 4days - yellow
>>> 4 days - red
>> Hanno
> _______________________________________________ Catalog-SIG mailing
> list Catalog-SIG <at> python.org 
> http://mail.python.org/mailman/listinfo/catalog-sig

--

-- 
ZOPYX Limited           | zopyx group
Charlottenstr. 37/1     | The full-service network for Zope & Plone
D-72070 Tübingen        | Produce & Publish
www.zopyx.com           | www.produce-and-publish.com
------------------------------------------------------------------------
E-Publishing, Python, Zope & Plone development, Consulting
Attachment (lists.vcf): text/x-vcard, 310 bytes
_______________________________________________
Catalog-SIG mailing list
Catalog-SIG <at> python.org
http://mail.python.org/mailman/listinfo/catalog-sig
Permalink | Reply |
headers
ken cochrane | 16 Apr 2012 21:39
Picon
Gravatar

Re: PyPI mirrors are all up to date

Sorry about that, I'm getting the location from http://ipinfodb.com
feel free to update their location for that IP with the correct one.

On Sun, Apr 15, 2012 at 11:33 PM, Andreas Jung <lists <at> zopyx.com> wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> This here is weird
>
> c.pypi.python.org       Russian Federation
>
> That's my server it it is running in Germany :)
>
> - -aj
>
> ken cochrane wrote:
>> I have taken your advice and I have updated it so that it is a
>> little clearer. I changed the times a little but I stuck to the 3
>> statuses that you proposed.
>>
>> I also added a bunch of other features to the site, so feel free to
>> check it out and let me know if you can think of anything else.
>>
>> http://www.pypi-mirrors.org
>>
>>
>> On Sun, Apr 15, 2012 at 11:11 AM, Hanno Schlichting
>> <hanno <at> hannosch.eu> wrote:
>>> On Sun, Apr 15, 2012 at 4:44 PM, Ken Cochrane
>>> <kencochrane <at> gmail.com> wrote:
>>>> Yeah I need to clean that up and make it more standard but this
>>>> is how it works now.
>>>>
>>>> Age < 5 min = excellent Age > 5 min and age < 15 min = awesome
>>>> Age > 15 min and age < 1 hour = great. Age > 1 hour and age <  6
>>>> hour = good Age > 6 hours and age < 12 hour = OK Age > 12 hours
>>>> and age < 1 day = getting stale Age > 1 day = out of date
>>>>
>>>> Feel free to make suggestions on how to improve.
>>> My suggestion, keep it simple:
>>>
>>> Age < 15 minutes - green (fresh) Age < 1 day - yellow (oldish) Age
>>> > 1 day - red (old)
>>>
>>> Apache and CPAN mirrors are much more forgiving.
>>>
>>> http://www.apache.org/mirrors/#age-histogram
>>>
>>> < 30 hours - green < 54 hours - yellow
>>>> 54 hours - red
>>> http://mirrors.cpan.org/#age-histogram
>>>
>>> < 2 days green < 4days - yellow
>>>> 4 days - red
>>> Hanno
>> _______________________________________________ Catalog-SIG mailing
>> list Catalog-SIG <at> python.org
>> http://mail.python.org/mailman/listinfo/catalog-sig
>
> - --
> ZOPYX Limited           | zopyx group
> Charlottenstr. 37/1     | The full-service network for Zope & Plone
> D-72070 Tübingen        | Produce & Publish
> www.zopyx.com           | www.produce-and-publish.com
> - ------------------------------------------------------------------------
> E-Publishing, Python, Zope & Plone development, Consulting
>
>
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.11 (Darwin)
> Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
>
> iQGUBAEBAgAGBQJPi5MhAAoJEADcfz7u4AZjNNkLwI4YhWH6JND0j4OKdQJEqk63
> caeRq+h8q+Pt4eAE+MiXkLxyx3VRK0tSEesGLFTC39aGKuSMKfpbKzYtc5dT8qIX
> +WAhzrHw1MMzO7s6ShATv2VDYqcSCrsJx2KcHexxdpawpQTh9G53dnHRnyT6/Cq/
> CaQHXnIyTgCbh3xK8ruZWJ098lvr6Nqj2mS4JFILb0qf6N+4nNR+6FeEITqFf7HS
> Hemwx6i5m2ZIx1gvPIsoqFnNNvLBSCMdluFPVESkIPf6Ocm19ykHYKTOY+6I7iCV
> zUNkKaucA6sTOW3OMQ8MR4/Km2tBy+PPjRx/0wY99SaPDAenvTGovq62koaYTBs7
> NE4Cd5P1ydUkNVXFIyVc/ZiAWAJCGg5FL7nxKZRbQ2fb9L7YGYYVIhy3AaOziL6T
> qBFT35uz3ZWh0linYUPNpHplJ7V6hdEazA5AgVaVaeNyor86JIYtuxZOC8UBHQQl
> hk686sVopjJUCzu3UZOT8eoZlIOqpIA=
> =O4Ej
> -----END PGP SIGNATURE-----
Permalink | Reply |
headers
Andreas Jung | 16 Apr 2012 21:53

Re: PyPI mirrors are all up to date

The service tells me 

DE; State/Province : NORDRHEIN-WESTFALEN

-aj

ken cochrane wrote:
> Sorry about that, I'm getting the location from http://ipinfodb.com
> feel free to update their location for that IP with the correct one.
>
>
>
> On Sun, Apr 15, 2012 at 11:33 PM, Andreas Jung <lists <at> zopyx.com> wrote:
>> -----BEGIN PGP SIGNED MESSAGE-----
>> Hash: SHA1
>>
>> This here is weird
>>
>> c.pypi.python.org       Russian Federation
>>
>> That's my server it it is running in Germany :)
>>
>> - -aj
>>
>> ken cochrane wrote:
>>> I have taken your advice and I have updated it so that it is a
>>> little clearer. I changed the times a little but I stuck to the 3
>>> statuses that you proposed.
>>>
>>> I also added a bunch of other features to the site, so feel free to
>>> check it out and let me know if you can think of anything else.
>>>
>>> http://www.pypi-mirrors.org
>>>
>>>
>>> On Sun, Apr 15, 2012 at 11:11 AM, Hanno Schlichting
>>> <hanno <at> hannosch.eu> wrote:
>>>> On Sun, Apr 15, 2012 at 4:44 PM, Ken Cochrane
>>>> <kencochrane <at> gmail.com> wrote:
>>>>> Yeah I need to clean that up and make it more standard but this
>>>>> is how it works now.
>>>>>
>>>>> Age < 5 min = excellent Age > 5 min and age < 15 min = awesome
>>>>> Age > 15 min and age < 1 hour = great. Age > 1 hour and age <  6
>>>>> hour = good Age > 6 hours and age < 12 hour = OK Age > 12 hours
>>>>> and age < 1 day = getting stale Age > 1 day = out of date
>>>>>
>>>>> Feel free to make suggestions on how to improve.
>>>> My suggestion, keep it simple:
>>>>
>>>> Age < 15 minutes - green (fresh) Age < 1 day - yellow (oldish) Age
>>>>> 1 day - red (old)
>>>> Apache and CPAN mirrors are much more forgiving.
>>>>
>>>> http://www.apache.org/mirrors/#age-histogram
>>>>
>>>> < 30 hours - green < 54 hours - yellow
>>>>> 54 hours - red
>>>> http://mirrors.cpan.org/#age-histogram
>>>>
>>>> < 2 days green < 4days - yellow
>>>>> 4 days - red
>>>> Hanno
>>> _______________________________________________ Catalog-SIG mailing
>>> list Catalog-SIG <at> python.org
>>> http://mail.python.org/mailman/listinfo/catalog-sig
>> - --
>> ZOPYX Limited           | zopyx group
>> Charlottenstr. 37/1     | The full-service network for Zope & Plone
>> D-72070 Tübingen        | Produce & Publish
>> www.zopyx.com           | www.produce-and-publish.com
>> - ------------------------------------------------------------------------
>> E-Publishing, Python, Zope & Plone development, Consulting
>>
>>
>> -----BEGIN PGP SIGNATURE-----
>> Version: GnuPG v1.4.11 (Darwin)
>> Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
>>
>> iQGUBAEBAgAGBQJPi5MhAAoJEADcfz7u4AZjNNkLwI4YhWH6JND0j4OKdQJEqk63
>> caeRq+h8q+Pt4eAE+MiXkLxyx3VRK0tSEesGLFTC39aGKuSMKfpbKzYtc5dT8qIX
>> +WAhzrHw1MMzO7s6ShATv2VDYqcSCrsJx2KcHexxdpawpQTh9G53dnHRnyT6/Cq/
>> CaQHXnIyTgCbh3xK8ruZWJ098lvr6Nqj2mS4JFILb0qf6N+4nNR+6FeEITqFf7HS
>> Hemwx6i5m2ZIx1gvPIsoqFnNNvLBSCMdluFPVESkIPf6Ocm19ykHYKTOY+6I7iCV
>> zUNkKaucA6sTOW3OMQ8MR4/Km2tBy+PPjRx/0wY99SaPDAenvTGovq62koaYTBs7
>> NE4Cd5P1ydUkNVXFIyVc/ZiAWAJCGg5FL7nxKZRbQ2fb9L7YGYYVIhy3AaOziL6T
>> qBFT35uz3ZWh0linYUPNpHplJ7V6hdEazA5AgVaVaeNyor86JIYtuxZOC8UBHQQl
>> hk686sVopjJUCzu3UZOT8eoZlIOqpIA=
>> =O4Ej
>> -----END PGP SIGNATURE-----
Attachment (lists.vcf): text/x-vcard, 325 bytes
_______________________________________________
Catalog-SIG mailing list
Catalog-SIG <at> python.org
http://mail.python.org/mailman/listinfo/catalog-sig
Permalink | Reply |
headers
Terry Reedy | 16 Apr 2012 07:54
Picon
Favicon

Re: PyPI mirrors are all up to date

On 4/15/2012 10:09 PM, ken cochrane wrote:
> I have taken your advice and I have updated it so that it is a little
> clearer. I changed the times a little but I stuck to the 3 statuses
> that you proposed.
>
> I also added a bunch of other features to the site, so feel free to
> check it out and let me know if you can think of anything else.
>
> http://www.pypi-mirrors.org

Looks pretty nice, except I suggest 'aging' rather than 'oldish', which 
is not a real word.

>
> On Sun, Apr 15, 2012 at 11:11 AM, Hanno Schlichting<hanno <at> hannosch.eu>  wrote:
>> On Sun, Apr 15, 2012 at 4:44 PM, Ken Cochrane<kencochrane <at> gmail.com>  wrote:
>>> Yeah I need to clean that up and make it more standard but this is how it
>>> works now.
>>>
>>> Age<  5 min = excellent
>>> Age>  5 min and age<  15 min = awesome
>>> Age>  15 min and age<  1 hour = great.
>>> Age>  1 hour and age<    6 hour = good
>>> Age>  6 hours and age<  12 hour = OK
>>> Age>  12 hours and age<  1 day = getting stale
>>> Age>  1 day = out of date
>>>
>>> Feel free to make suggestions on how to improve.
>>
>> My suggestion, keep it simple:
>>
>> Age<  15 minutes - green (fresh)
>> Age<  1 day - yellow (oldish)
>> Age>  1 day - red (old)
>>
>> Apache and CPAN mirrors are much more forgiving.
>>
>> http://www.apache.org/mirrors/#age-histogram
>>
>> <  30 hours - green
>> <  54 hours - yellow
>>> 54 hours - red
>>
>> http://mirrors.cpan.org/#age-histogram
>>
>> <  2 days green
>> <  4days - yellow
>>> 4 days - red
>>
>> Hanno

--

-- 
Terry Jan Reedy
Permalink | Reply |
headers
Yuval Greenfield | 16 Apr 2012 09:00
Picon
Gravatar

Re: PyPI mirrors are all up to date

Looks very nice. I especially like the graphs. If we're into the bike shedding stage then here are a few suggestions:
  • Make columns JS sortable
  • Ping is relative - that column should state "Response time from Florida" or something to that effect as I got very different results from those listed.
  • "Oldish" is ok though I think "Ok", "Medium", "Aging", "Freshish" or "So so" would be fine as well.

Yuval

On Mon, Apr 16, 2012 at 8:54 AM, Terry Reedy <tjreedy <at> udel.edu> wrote:
On 4/15/2012 10:09 PM, ken cochrane wrote:
I have taken your advice and I have updated it so that it is a little
clearer. I changed the times a little but I stuck to the 3 statuses
that you proposed.

I also added a bunch of other features to the site, so feel free to
check it out and let me know if you can think of anything else.

http://www.pypi-mirrors.org

Looks pretty nice, except I suggest 'aging' rather than 'oldish', which is not a real word.



On Sun, Apr 15, 2012 at 11:11 AM, Hanno Schlichting<hanno <at> hannosch.eu>  wrote:
On Sun, Apr 15, 2012 at 4:44 PM, Ken Cochrane<kencochrane <at> gmail.com>  wrote:
Yeah I need to clean that up and make it more standard but this is how it
works now.

Age<  5 min = excellent
Age>  5 min and age<  15 min = awesome
Age>  15 min and age<  1 hour = great.
Age>  1 hour and age<    6 hour = good
Age>  6 hours and age<  12 hour = OK
Age>  12 hours and age<  1 day = getting stale
Age>  1 day = out of date

Feel free to make suggestions on how to improve.

My suggestion, keep it simple:

Age<  15 minutes - green (fresh)
Age<  1 day - yellow (oldish)
Age>  1 day - red (old)

Apache and CPAN mirrors are much more forgiving.

http://www.apache.org/mirrors/#age-histogram

<  30 hours - green
<  54 hours - yellow
54 hours - red

http://mirrors.cpan.org/#age-histogram

<  2 days green
<  4days - yellow
4 days - red

Hanno


--
Terry Jan Reedy


_______________________________________________
Catalog-SIG mailing list
Catalog-SIG <at> python.org
http://mail.python.org/mailman/listinfo/catalog-sig

_______________________________________________
Catalog-SIG mailing list
Catalog-SIG <at> python.org
http://mail.python.org/mailman/listinfo/catalog-sig
Permalink | Reply |
headers
ken cochrane | 16 Apr 2012 21:42
Picon
Gravatar

Re: PyPI mirrors are all up to date

Thanks for the feedback, I'll think about the sortable tables, it
might be overkill for such a small dataset, but I'll keep it in mind.

I added a disclaimer that the response times are from Virginia, US.

Status has been changed to Fresh, Aging and Old.

Ken

On Mon, Apr 16, 2012 at 3:00 AM, Yuval Greenfield <ubershmekel <at> gmail.com> wrote:
> Looks very nice. I especially like the graphs. If we're into the bike
> shedding stage then here are a few suggestions:
>
> Make columns JS sortable
> Ping is relative - that column should state "Response time from Florida" or
> something to that effect as I got very different results from those listed.
> "Oldish" is ok though I think "Ok", "Medium", "Aging", "Freshish" or "So so"
> would be fine as well.
>
>
> Yuval
>
> On Mon, Apr 16, 2012 at 8:54 AM, Terry Reedy <tjreedy <at> udel.edu> wrote:
>>
>> On 4/15/2012 10:09 PM, ken cochrane wrote:
>>>
>>> I have taken your advice and I have updated it so that it is a little
>>> clearer. I changed the times a little but I stuck to the 3 statuses
>>> that you proposed.
>>>
>>> I also added a bunch of other features to the site, so feel free to
>>> check it out and let me know if you can think of anything else.
>>>
>>> http://www.pypi-mirrors.org
>>
>>
>> Looks pretty nice, except I suggest 'aging' rather than 'oldish', which is
>> not a real word.
>>
>>
>>>
>>> On Sun, Apr 15, 2012 at 11:11 AM, Hanno Schlichting<hanno <at> hannosch.eu>
>>>  wrote:
>>>>
>>>> On Sun, Apr 15, 2012 at 4:44 PM, Ken Cochrane<kencochrane <at> gmail.com>
>>>>  wrote:
>>>>>
>>>>> Yeah I need to clean that up and make it more standard but this is how
>>>>> it
>>>>> works now.
>>>>>
>>>>> Age<  5 min = excellent
>>>>> Age>  5 min and age<  15 min = awesome
>>>>> Age>  15 min and age<  1 hour = great.
>>>>> Age>  1 hour and age<    6 hour = good
>>>>> Age>  6 hours and age<  12 hour = OK
>>>>> Age>  12 hours and age<  1 day = getting stale
>>>>> Age>  1 day = out of date
>>>>>
>>>>> Feel free to make suggestions on how to improve.
>>>>
>>>>
>>>> My suggestion, keep it simple:
>>>>
>>>> Age<  15 minutes - green (fresh)
>>>> Age<  1 day - yellow (oldish)
>>>> Age>  1 day - red (old)
>>>>
>>>> Apache and CPAN mirrors are much more forgiving.
>>>>
>>>> http://www.apache.org/mirrors/#age-histogram
>>>>
>>>> <  30 hours - green
>>>> <  54 hours - yellow
>>>>>
>>>>> 54 hours - red
>>>>
>>>>
>>>> http://mirrors.cpan.org/#age-histogram
>>>>
>>>> <  2 days green
>>>> <  4days - yellow
>>>>>
>>>>> 4 days - red
>>>>
>>>>
>>>> Hanno
>>
>>
>>
>> --
>> Terry Jan Reedy
>>
>>
>> _______________________________________________
>> Catalog-SIG mailing list
>> Catalog-SIG <at> python.org
>> http://mail.python.org/mailman/listinfo/catalog-sig
>
>
>
> _______________________________________________
> Catalog-SIG mailing list
> Catalog-SIG <at> python.org
> http://mail.python.org/mailman/listinfo/catalog-sig
>
Permalink | Reply |
headers
ken cochrane | 16 Apr 2012 21:41
Picon
Gravatar

Re: PyPI mirrors are all up to date

Thanks,

Change has been made, it will be available in next update.

Ken

On Mon, Apr 16, 2012 at 1:54 AM, Terry Reedy <tjreedy <at> udel.edu> wrote:
> On 4/15/2012 10:09 PM, ken cochrane wrote:
>>
>> I have taken your advice and I have updated it so that it is a little
>> clearer. I changed the times a little but I stuck to the 3 statuses
>> that you proposed.
>>
>> I also added a bunch of other features to the site, so feel free to
>> check it out and let me know if you can think of anything else.
>>
>> http://www.pypi-mirrors.org
>
>
> Looks pretty nice, except I suggest 'aging' rather than 'oldish', which is
> not a real word.
>
>
>>
>> On Sun, Apr 15, 2012 at 11:11 AM, Hanno Schlichting<hanno <at> hannosch.eu>
>>  wrote:
>>>
>>> On Sun, Apr 15, 2012 at 4:44 PM, Ken Cochrane<kencochrane <at> gmail.com>
>>>  wrote:
>>>>
>>>> Yeah I need to clean that up and make it more standard but this is how
>>>> it
>>>> works now.
>>>>
>>>> Age<  5 min = excellent
>>>> Age>  5 min and age<  15 min = awesome
>>>> Age>  15 min and age<  1 hour = great.
>>>> Age>  1 hour and age<    6 hour = good
>>>> Age>  6 hours and age<  12 hour = OK
>>>> Age>  12 hours and age<  1 day = getting stale
>>>> Age>  1 day = out of date
>>>>
>>>> Feel free to make suggestions on how to improve.
>>>
>>>
>>> My suggestion, keep it simple:
>>>
>>> Age<  15 minutes - green (fresh)
>>> Age<  1 day - yellow (oldish)
>>> Age>  1 day - red (old)
>>>
>>> Apache and CPAN mirrors are much more forgiving.
>>>
>>> http://www.apache.org/mirrors/#age-histogram
>>>
>>> <  30 hours - green
>>> <  54 hours - yellow
>>>>
>>>> 54 hours - red
>>>
>>>
>>> http://mirrors.cpan.org/#age-histogram
>>>
>>> <  2 days green
>>> <  4days - yellow
>>>>
>>>> 4 days - red
>>>
>>>
>>> Hanno
>
>
>
> --
> Terry Jan Reedy
>
>
> _______________________________________________
> Catalog-SIG mailing list
> Catalog-SIG <at> python.org
> http://mail.python.org/mailman/listinfo/catalog-sig
Permalink | Reply |
headers
Sebastien Douche | 16 Apr 2012 11:15
Picon
Gravatar

Re: PyPI mirrors are all up to date

On Mon, Apr 16, 2012 at 04:09, ken cochrane <kencochrane <at> gmail.com> wrote:
> I also added a bunch of other features to the site, so feel free to
> check it out and let me know if you can think of anything else.
>
> http://www.pypi-mirrors.org

Nice! Thanks Ken. Missing only an information imho: the number of
packages on each mirror (to known if the mirror is really up to date).

--

-- 
Sebastien Douche <sdouche <at> gmail.com>
Twitter:  <at> sdouche / G+: +sdouche
Permalink | Reply |
headers
ken cochrane | 16 Apr 2012 21:46
Picon
Gravatar

Re: PyPI mirrors are all up to date

Is it possible to get the number of packages on each mirror? If so,
can you let me know and I'll look into adding it.

On Mon, Apr 16, 2012 at 5:15 AM, Sebastien Douche <sdouche <at> gmail.com> wrote:

> Nice! Thanks Ken. Missing only an information imho: the number of
> packages on each mirror (to known if the mirror is really up to date).
>
Permalink | Reply |
headers
Donald Stufft | 16 Apr 2012 21:52
Picon
Gravatar

Re: PyPI mirrors are all up to date

On Monday, April 16, 2012 at 3:46 PM, ken cochrane wrote:
Is it possible to get the number of packages on each mirror? If so,
can you let me know and I'll look into adding it.
Not without parsing the simple api. If you just want to know the total number of
names, you can parse the index page for number of links. If you want the total number
of files then you'll need to descend into all 20kish pages as well and pull links out of there.

On Mon, Apr 16, 2012 at 5:15 AM, Sebastien Douche <sdouche <at> gmail.com> wrote:

Nice! Thanks Ken. Missing only an information imho: the number of
packages on each mirror (to known if the mirror is really up to date).
_______________________________________________
Catalog-SIG mailing list
 

_______________________________________________
Catalog-SIG mailing list
Catalog-SIG <at> python.org
http://mail.python.org/mailman/listinfo/catalog-sig
Permalink | Reply |
headers
ken cochrane | 16 Apr 2012 21:58
Picon
Gravatar

Re: PyPI mirrors are all up to date

Ouch, does that take a while to gather the number of files?
Is there any open source code that already does this, that I can look at?
I'm thinking packages might be find for now.

Thanks,
Ken

On Mon, Apr 16, 2012 at 3:52 PM, Donald Stufft <donald.stufft <at> gmail.com> wrote:
> On Monday, April 16, 2012 at 3:46 PM, ken cochrane wrote:
>
> Is it possible to get the number of packages on each mirror? If so,
> can you let me know and I'll look into adding it.
>
> Not without parsing the simple api. If you just want to know the total
> number of
> names, you can parse the index page for number of links. If you want the
> total number
> of files then you'll need to descend into all 20kish pages as well and pull
> links out of there.
>
>
> On Mon, Apr 16, 2012 at 5:15 AM, Sebastien Douche <sdouche <at> gmail.com> wrote:
>
> Nice! Thanks Ken. Missing only an information imho: the number of
> packages on each mirror (to known if the mirror is really up to date).
>
> _______________________________________________
> Catalog-SIG mailing list
> Catalog-SIG <at> python.org
> http://mail.python.org/mailman/listinfo/catalog-sig
>
>
Permalink | Reply |
headers
ken cochrane | 16 Apr 2012 22:43
Picon
Gravatar

Re: PyPI mirrors are all up to date

OK, Thanks to Donald's help, I know show the number of packages for each mirror.

Ken

On Mon, Apr 16, 2012 at 3:58 PM, ken cochrane <kencochrane <at> gmail.com> wrote:
> Ouch, does that take a while to gather the number of files?
> Is there any open source code that already does this, that I can look at?
> I'm thinking packages might be find for now.
>
> Thanks,
> Ken
>
>
> On Mon, Apr 16, 2012 at 3:52 PM, Donald Stufft <donald.stufft <at> gmail.com> wrote:
>> On Monday, April 16, 2012 at 3:46 PM, ken cochrane wrote:
>>
>> Is it possible to get the number of packages on each mirror? If so,
>> can you let me know and I'll look into adding it.
>>
>> Not without parsing the simple api. If you just want to know the total
>> number of
>> names, you can parse the index page for number of links. If you want the
>> total number
>> of files then you'll need to descend into all 20kish pages as well and pull
>> links out of there.
>>
>>
>> On Mon, Apr 16, 2012 at 5:15 AM, Sebastien Douche <sdouche <at> gmail.com> wrote:
>>
>> Nice! Thanks Ken. Missing only an information imho: the number of
>> packages on each mirror (to known if the mirror is really up to date).
>>
>> _______________________________________________
>> Catalog-SIG mailing list
>> Catalog-SIG <at> python.org
>> http://mail.python.org/mailman/listinfo/catalog-sig
>>
>>
Permalink | Reply |
headers
Tarek Ziadé | 16 Apr 2012 22:48
Favicon
Gravatar

Re: PyPI mirrors are all up to date

On 4/16/12 9:58 PM, ken cochrane wrote:
> Ouch, does that take a while to gather the number of files?
> Is there any open source code that already does this, that I can look at?
> I'm thinking packages might be find for now.
the other option would be to change the mirroring protocol to have that 
total in a static page, like the timestamp

>
> Thanks,
> Ken
>
>
> On Mon, Apr 16, 2012 at 3:52 PM, Donald Stufft<donald.stufft <at> gmail.com>  wrote:
>> On Monday, April 16, 2012 at 3:46 PM, ken cochrane wrote:
>>
>> Is it possible to get the number of packages on each mirror? If so,
>> can you let me know and I'll look into adding it.
>>
>> Not without parsing the simple api. If you just want to know the total
>> number of
>> names, you can parse the index page for number of links. If you want the
>> total number
>> of files then you'll need to descend into all 20kish pages as well and pull
>> links out of there.
>>
>>
>> On Mon, Apr 16, 2012 at 5:15 AM, Sebastien Douche<sdouche <at> gmail.com>  wrote:
>>
>> Nice! Thanks Ken. Missing only an information imho: the number of
>> packages on each mirror (to known if the mirror is really up to date).
>>
>> _______________________________________________
>> Catalog-SIG mailing list
>> Catalog-SIG <at> python.org
>> http://mail.python.org/mailman/listinfo/catalog-sig
>>
>>
> _______________________________________________
> Catalog-SIG mailing list
> Catalog-SIG <at> python.org
> http://mail.python.org/mailman/listinfo/catalog-sig
Permalink | Reply |
headers
ken cochrane | 16 Apr 2012 22:57
Picon
Gravatar

Re: PyPI mirrors are all up to date

That would be nice, you could include other stats information as well
(not sure what yet).

Ken

On Mon, Apr 16, 2012 at 4:48 PM, Tarek Ziadé <tarek <at> ziade.org> wrote:
> On 4/16/12 9:58 PM, ken cochrane wrote:
>>
>> Ouch, does that take a while to gather the number of files?
>> Is there any open source code that already does this, that I can look at?
>> I'm thinking packages might be find for now.
>
> the other option would be to change the mirroring protocol to have that
> total in a static page, like the timestamp
>
>
>>
>> Thanks,
>> Ken
>>
>>
>> On Mon, Apr 16, 2012 at 3:52 PM, Donald Stufft<donald.stufft <at> gmail.com>
>>  wrote:
>>>
>>> On Monday, April 16, 2012 at 3:46 PM, ken cochrane wrote:
>>>
>>> Is it possible to get the number of packages on each mirror? If so,
>>> can you let me know and I'll look into adding it.
>>>
>>> Not without parsing the simple api. If you just want to know the total
>>> number of
>>> names, you can parse the index page for number of links. If you want the
>>> total number
>>> of files then you'll need to descend into all 20kish pages as well and
>>> pull
>>> links out of there.
>>>
>>>
>>> On Mon, Apr 16, 2012 at 5:15 AM, Sebastien Douche<sdouche <at> gmail.com>
>>>  wrote:
>>>
>>> Nice! Thanks Ken. Missing only an information imho: the number of
>>> packages on each mirror (to known if the mirror is really up to date).
>>>
>>> _______________________________________________
>>> Catalog-SIG mailing list
>>> Catalog-SIG <at> python.org
>>> http://mail.python.org/mailman/listinfo/catalog-sig
>>>
>>>
>> _______________________________________________
>> Catalog-SIG mailing list
>> Catalog-SIG <at> python.org
>> http://mail.python.org/mailman/listinfo/catalog-sig
>
>
> _______________________________________________
> Catalog-SIG mailing list
> Catalog-SIG <at> python.org
> http://mail.python.org/mailman/listinfo/catalog-sig
Permalink | Reply |
headers
Tarek Ziadé | 16 Apr 2012 23:05
Favicon
Gravatar

Re: PyPI mirrors are all up to date

On 4/16/12 10:57 PM, ken cochrane wrote:
> That would be nice, you could include other stats information as well
> (not sure what yet).
sounds like a nice "checksum" addition to PEP 381- that'd avoid 
client-side parsing just to get the number of mirrored packages.

>
> Ken
>
>
> On Mon, Apr 16, 2012 at 4:48 PM, Tarek Ziadé<tarek <at> ziade.org>  wrote:
>> On 4/16/12 9:58 PM, ken cochrane wrote:
>>> Ouch, does that take a while to gather the number of files?
>>> Is there any open source code that already does this, that I can look at?
>>> I'm thinking packages might be find for now.
>> the other option would be to change the mirroring protocol to have that
>> total in a static page, like the timestamp
>>
>>
>>> Thanks,
>>> Ken
>>>
>>>
>>> On Mon, Apr 16, 2012 at 3:52 PM, Donald Stufft<donald.stufft <at> gmail.com>
>>>   wrote:
>>>> On Monday, April 16, 2012 at 3:46 PM, ken cochrane wrote:
>>>>
>>>> Is it possible to get the number of packages on each mirror? If so,
>>>> can you let me know and I'll look into adding it.
>>>>
>>>> Not without parsing the simple api. If you just want to know the total
>>>> number of
>>>> names, you can parse the index page for number of links. If you want the
>>>> total number
>>>> of files then you'll need to descend into all 20kish pages as well and
>>>> pull
>>>> links out of there.
>>>>
>>>>
>>>> On Mon, Apr 16, 2012 at 5:15 AM, Sebastien Douche<sdouche <at> gmail.com>
>>>>   wrote:
>>>>
>>>> Nice! Thanks Ken. Missing only an information imho: the number of
>>>> packages on each mirror (to known if the mirror is really up to date).
>>>>
>>>> _______________________________________________
>>>> Catalog-SIG mailing list
>>>> Catalog-SIG <at> python.org
>>>> http://mail.python.org/mailman/listinfo/catalog-sig
>>>>
>>>>
>>> _______________________________________________
>>> Catalog-SIG mailing list
>>> Catalog-SIG <at> python.org
>>> http://mail.python.org/mailman/listinfo/catalog-sig
>>
>> _______________________________________________
>> Catalog-SIG mailing list
>> Catalog-SIG <at> python.org
>> http://mail.python.org/mailman/listinfo/catalog-sig
Permalink | Reply |
headers
Martin v. Löwis | 16 Apr 2012 23:23
Picon
Gravatar

Re: PyPI mirrors are all up to date

Am 16.04.2012 23:05, schrieb Tarek Ziadé:
> On 4/16/12 10:57 PM, ken cochrane wrote:
>> That would be nice, you could include other stats information as well
>> (not sure what yet).
> sounds like a nice "checksum" addition to PEP 381- that'd avoid
> client-side parsing just to get the number of mirrored packages.

I'm not sure what purpose this would serve. I already don't see the
point in computing the number of mirrored packages or the number of
mirrored files (other than to rejoice the PyPI maintainer: wow,
we have 20545 packages in PyPI! It was only 10,000 just a short time
ago).

I think the original requester of this feature suggested that as a
sanity check, apparently not trusting the last-modified information.
If it's really this kind of distrust, any generated stats information
should be just as distrusted. For example, it may be that mirrors
failed to correctly copy one of the files, or show other
inconsistencies. No mirror-generated statistics would indicate this
problem.

So before extending the protocol, there should be some serious
motivation for it.

Regards,
Martin
Permalink | Reply |
headers
Tarek Ziadé | 16 Apr 2012 23:45
Favicon
Gravatar

Re: PyPI mirrors are all up to date

On 4/16/12 11:23 PM, "Martin v. Löwis" wrote:
> Am 16.04.2012 23:05, schrieb Tarek Ziadé:
>> On 4/16/12 10:57 PM, ken cochrane wrote:
>>> That would be nice, you could include other stats information as well
>>> (not sure what yet).
>> sounds like a nice "checksum" addition to PEP 381- that'd avoid
>> client-side parsing just to get the number of mirrored packages.
> I'm not sure what purpose this would serve. I already don't see the
> point in computing the number of mirrored packages or the number of
> mirrored files (other than to rejoice the PyPI maintainer: wow,
> we have 20545 packages in PyPI! It was only 10,000 just a short time
> ago).
>
> I think the original requester of this feature suggested that as a
> sanity check, apparently not trusting the last-modified information.
Yeah that's what I understood too. So I used the term 'checksum'
> If it's really this kind of distrust, any generated stats information
> should be just as distrusted. For example, it may be that mirrors
> failed to correctly copy one of the files, or show other
> inconsistencies. No mirror-generated statistics would indicate this
> problem.
the number of packages is a good indication of a successful mirroring - 
I think that was the point.

I can definitely see a mirroring implementation where the last-modified 
field is updated at the end
while some packages are not copied over at the end for whatever network 
issue.

Maybe a better checksum would be a global hash calculated differently ?

>
> So before extending the protocol, there should be some serious
> motivation for it.
>
> Regards,
> Martin
Permalink | Reply |
headers
Martin v. Löwis | 16 Apr 2012 23:57
Picon
Gravatar

Re: PyPI mirrors are all up to date

> Maybe a better checksum would be a global hash calculated differently ?

Define a protocol, and I present you with an implementation that
conforms to the protocol, and still has inconsistent data, and not
in a malicious manner, but due to bugs/race conditions/unexpected
events. It's pointless. Ultimately, clients will need to verify the
data that they receive (if they suspect issues), and fall back gracefully.

> I can definitely see a mirroring implementation where the
> last-modified field is updated at the end while some packages are not
> copied over at the end for whatever network issue.

That mirroring implementation would violate the principle that
last-modified should only be updated when the mirroring run was
completed successfully.

Regards,
Martin
Permalink | Reply |
headers
Tarek Ziadé | 17 Apr 2012 00:09
Favicon
Gravatar

Re: PyPI mirrors are all up to date

On 4/16/12 11:57 PM, "Martin v. Löwis" wrote:
>> Maybe a better checksum would be a global hash calculated differently ?
> Define a protocol, and I present you with an implementation that
> conforms to the protocol, and still has inconsistent data, and not
> in a malicious manner, but due to bugs/race conditions/unexpected
> events. It's pointless.
if you calculate a checksum with all mirrored files - you can guarantee 
that the bits are the same
on both side, no ? then you have the md5 checksum per file for the 
download so the client can
be sure he got a non corrupted file.

> Ultimately, clients will need to verify the
> data that they receive (if they suspect issues), and fall back gracefully.
how can they know if version 1.3 of package foo never made it to the 
mirror they use ?

They can't. They have to trust the last modified date and make the 
assumption that the mirror
is fresh enough, for foo 1.3 to be present in both the master and the 
mirror.

>
>> I can definitely see a mirroring implementation where the
>> last-modified field is updated at the end while some packages are not
>> copied over at the end for whatever network issue.
> That mirroring implementation would violate the principle that
> last-modified should only be updated when the mirroring run was
> completed successfully.
like a file that claims in its metadata it's 512 kb long and only has 2 
kb in reality because something went wrong ?

I think the idea of the checksum is to double-check that kind of claim.  
But maybe that's overkill ?
maybe the mirroring code should check file by file that everything was 
copied correctly ?

Cheers
Tarek

>
> Regards,
> Martin
Permalink | Reply |
headers
Donald Stufft | 17 Apr 2012 00:14
Picon
Gravatar

Re: PyPI mirrors are all up to date


On Monday, April 16, 2012 at 6:09 PM, Tarek Ziadé wrote:

On 4/16/12 11:57 PM, "Martin v. Löwis" wrote:
Maybe a better checksum would be a global hash calculated differently ?
Define a protocol, and I present you with an implementation that
conforms to the protocol, and still has inconsistent data, and not
in a malicious manner, but due to bugs/race conditions/unexpected
events. It's pointless.
if you calculate a checksum with all mirrored files - you can guarantee
that the bits are the same
on both side, no ? then you have the md5 checksum per file for the
download so the client can
be sure he got a non corrupted file.

Ultimately, clients will need to verify the
data that they receive (if they suspect issues), and fall back gracefully.
how can they know if version 1.3 of package foo never made it to the
mirror they use ?

They can't. They have to trust the last modified date and make the
assumption that the mirror
is fresh enough, for foo 1.3 to be present in both the master and the
mirror.


I can definitely see a mirroring implementation where the
last-modified field is updated at the end while some packages are not
copied over at the end for whatever network issue.
That mirroring implementation would violate the principle that
last-modified should only be updated when the mirroring run was
completed successfully.
like a file that claims in its metadata it's 512 kb long and only has 2
kb in reality because something went wrong ?

I think the idea of the checksum is to double-check that kind of claim.
But maybe that's overkill ?
maybe the mirroring code should check file by file that everything was
copied correctly ?
fwiw Crate verifies the md5 hashes that the simple api gives for each package. I think not
doing that should be considered wrong. (it's considered important for clients to check the
checksum of packages they download, but mirrors that are going to be redistributing their
files to clients this isn't important? Seems to be a disconnect between the thoughts).

Cheers
Tarek


Regards,
Martin

_______________________________________________
Catalog-SIG mailing list
 

_______________________________________________
Catalog-SIG mailing list
Catalog-SIG <at> python.org
http://mail.python.org/mailman/listinfo/catalog-sig
Permalink | Reply |
headers
Martin v. Löwis | 17 Apr 2012 00:22
Picon
Gravatar

Re: PyPI mirrors are all up to date

> fwiw Crate verifies the md5 hashes that the simple api gives for
> each package. I think not doing that should be considered wrong.
> (it's considered important for clients to check the checksum of
> packages they download, but mirrors that are going to be 
> redistributing their files to clients this isn't important? Seems to
> be a disconnect between the thoughts).

I think this is a quality-of-implementation issue. They could verify
the md5; they could also verify the serversig. Some do, some don't.
Or they could claim they do but actually don't. Ultimately, clients
either need to trust the mirror, or do the verification themselves.

Regards,
Martin
Permalink | Reply |
headers
Donald Stufft | 17 Apr 2012 00:31
Picon
Gravatar

Re: PyPI mirrors are all up to date

On Monday, April 16, 2012 at 6:22 PM, "Martin v. Löwis" wrote:
fwiw Crate verifies the md5 hashes that the simple api gives for
each package. I think not doing that should be considered wrong.
(it's considered important for clients to check the checksum of
packages they download, but mirrors that are going to be
redistributing their files to clients this isn't important? Seems to
be a disconnect between the thoughts).

I think this is a quality-of-implementation issue. They could verify
the md5; they could also verify the serversig. Some do, some don't.
Or they could claim they do but actually don't. Ultimately, clients
either need to trust the mirror, or do the verification themselves.
Yea ultimately the clients will need to do the verification themselves, I
only meant to have somewhat more useful mirrors (corrupted data,  bugs
in scripts, etc) it would make sense for mirrors to verify that the data they
are getting is the data they are expecting so they don't serve bad data. While
a verifying client won't be affected by that bad data, it does lower the overall
availability for that file.

Regards,
Martin
 

_______________________________________________
Catalog-SIG mailing list
Catalog-SIG <at> python.org
http://mail.python.org/mailman/listinfo/catalog-sig
Permalink | Reply |
headers
Martin v. Löwis | 17 Apr 2012 00:19
Picon
Gravatar

Re: PyPI mirrors are all up to date

Am 17.04.2012 00:09, schrieb Tarek Ziadé:
> On 4/16/12 11:57 PM, "Martin v. Löwis" wrote:
>>> Maybe a better checksum would be a global hash calculated differently ?
>> Define a protocol, and I present you with an implementation that
>> conforms to the protocol, and still has inconsistent data, and not
>> in a malicious manner, but due to bugs/race conditions/unexpected
>> events. It's pointless.
> if you calculate a checksum with all mirrored files - you can guarantee
> that the bits are the same
> on both side, no ?

How exactly would you calculate that checksum? Would you really require
concatenation of all files? That could take a few hours per change. It
would also raise the question in what order the files ought to be
concatenated.

> how can they know if version 1.3 of package foo never made it to the
> mirror they use ?
> 
> They can't. They have to trust the last modified date and make the
> assumption that the mirror
> is fresh enough, for foo 1.3 to be present in both the master and the
> mirror.

How could they do so using your protocol?

> I think the idea of the checksum is to double-check that kind of claim. 
> But maybe that's overkill ?

I think it's both overkill, and it doesn't help.

> maybe the mirroring code should check file by file that everything was
> copied correctly ?

If you also assume malicious mirrors, then you definitely need to check
every file, as specified in

http://www.python.org/dev/peps/pep-0381/#mirror-authenticity

However, if a mirror claims it is up-to-date, and that verification
fails, my recommendation would be to give up in the tool and have
the user submit a bug report, in order to eliminate the mirror from
the mirror list.

Regards,
Martin
Permalink | Reply |
headers
Tarek Ziadé | 17 Apr 2012 00:48
Favicon
Gravatar

Re: PyPI mirrors are all up to date

On 4/17/12 12:19 AM, "Martin v. Löwis" wrote:
> Am 17.04.2012 00:09, schrieb Tarek Ziadé:
>> On 4/16/12 11:57 PM, "Martin v. Löwis" wrote:
>>>> Maybe a better checksum would be a global hash calculated differently ?
>>> Define a protocol, and I present you with an implementation that
>>> conforms to the protocol, and still has inconsistent data, and not
>>> in a malicious manner, but due to bugs/race conditions/unexpected
>>> events. It's pointless.
>> if you calculate a checksum with all mirrored files - you can guarantee
>> that the bits are the same
>> on both side, no ?
> How exactly would you calculate that checksum?
by calculating the grand hash of each file hash.
>   Would you really require
> concatenation of all files?
I did not say that. You are claiming it in a rhetorical question.

> That could take a few hours per change.
why that ? you don't calculate the checksum of a file your already have 
twice.

Even if you do, it's very fast to call md5.

try it:

$ find mirror | xargs md5

this takes a few seconds at most on the whole mirror

> It
> would also raise the question in what order the files ought to be
> concatenated.
Anything reproductible, a sorted list.  In bash I *suspect* the 
calculation of the grand hash of the mirror is a one-liner that takes 
less than a minute.

I am going to stop here anyways because I don't see the point of 
discussing implementation details at this stage, since we were
barely starting to talk about the idea of a checksum - and that seems to 
be going nowhere.

Cheers
Tarek
Permalink | Reply |
headers
martin | 17 Apr 2012 11:57
Picon
Gravatar

Re: PyPI mirrors are all up to date


>>> if you calculate a checksum with all mirrored files - you can guarantee
>>> that the bits are the same
>>> on both side, no ?
>> How exactly would you calculate that checksum?
> by calculating the grand hash of each file hash.

In this case, the checksum would not be a reliable indication that the
files are actually up-to-date. For example, a mirror may keep updating
files into the wrong location (not the location that is then used to
serve the files), so that the files being served are from a stale copy.
This is not theoretical - it actually happened in my mirror setup at one
time.

>> That could take a few hours per change.
> why that ? you don't calculate the checksum of a file your already  
> have twice.
>
> Even if you do, it's very fast to call md5.
>
> try it:
>
> $ find mirror | xargs md5
>
> this takes a few seconds at most on the whole mirror

I tried it, and on my mirror, it took 27 minutes and 7 seconds.
So not exactly hours, but not "a few seconds" either.

Regards,
Martin
Permalink | Reply |
headers
Tarek Ziadé | 17 Apr 2012 12:50
Favicon
Gravatar

Re: PyPI mirrors are all up to date

On 4/17/12 11:57 AM, martin <at> v.loewis.de wrote:
> > by calculating the grand hash of each file hash.
>
> In this case, the checksum would not be a reliable indication that the
> files are actually up-to-date. For example, a mirror may keep updating
> files into the wrong location (not the location that is then used to
> serve the files), so that the files being served are from a stale copy.
> This is not theoretical - it actually happened in my mirror setup at one
> time.
>
So you were updating a directory but serving another directory ?

But then updating the right last-modified page people were seeing ?

In that case, updating the checksum would have revealed you were on the 
wrong set of files.

Unless you script was updating everything on a stale copy that was not 
published ?

>>> That could take a few hours per change.
>> why that ? you don't calculate the checksum of a file your already 
>> have twice.
>>
>> Even if you do, it's very fast to call md5.
>>
>> try it:
>>
>> $ find mirror | xargs md5
>>
>> this takes a few seconds at most on the whole mirror
>
> I tried it, and on my mirror, it took 27 minutes and 7 seconds.
> So not exactly hours, but not "a few seconds" either.
oops sorry I ran it on the wrong directory, it's true that it takes more 
time !

So on my centos 5 VM - which is quite slow and doing many other stuff 
like running Jenkins jobs, running the "md5deep" program like this : 
http://tarek.pastebin.mozilla.org/1574557

It took 15minutes and 1 second.   It can be optimized of course, since 
most directories are done quickly and everything is in /source. That 
time can be divided by 2 at least with the proper load balancing between 
a few md5 runners.

But that just to be run *once*. You would not compute it on every mirror 
update but keep all md5 values somewhere.

So, recalculating the grand hash on every mirror update should takes a 
few seconds because it would just consist of calculating the hash for 
the new files, then
calculating the grand hash -- a loop that updates a md5 hash with 20k 
hashes takes less than a second if I don't count the file reading.

(see http://tarek.pastebin.mozilla.org/1574574)

I am not sure why we're having this discussion since it's implementation 
details, but it's fun :)

If there's interest I can write a multiprocess-based script that keeps a 
md5 database up-to-date

Cheers
Tarek

>
> Regards,
> Martin
>
>
Permalink | Reply |
headers
Donald Stufft | 17 Apr 2012 12:54
Picon
Gravatar

Re: PyPI mirrors are all up to date

On Tuesday, April 17, 2012 at 6:50 AM, Tarek Ziadé wrote:
On 4/17/12 11:57 AM, martin <at> v.loewis.de wrote:
by calculating the grand hash of each file hash.

In this case, the checksum would not be a reliable indication that the
files are actually up-to-date. For example, a mirror may keep updating
files into the wrong location (not the location that is then used to
serve the files), so that the files being served are from a stale copy.
This is not theoretical - it actually happened in my mirror setup at one
time.
So you were updating a directory but serving another directory ?

But then updating the right last-modified page people were seeing ?

In that case, updating the checksum would have revealed you were on the
wrong set of files.

Unless you script was updating everything on a stale copy that was not
published ?


That could take a few hours per change.
why that ? you don't calculate the checksum of a file your already
have twice.

Even if you do, it's very fast to call md5.

try it:

$ find mirror | xargs md5

this takes a few seconds at most on the whole mirror

I tried it, and on my mirror, it took 27 minutes and 7 seconds.
So not exactly hours, but not "a few seconds" either.
oops sorry I ran it on the wrong directory, it's true that it takes more
time !

So on my centos 5 VM - which is quite slow and doing many other stuff
like running Jenkins jobs, running the "md5deep" program like this :

It took 15minutes and 1 second. It can be optimized of course, since
most directories are done quickly and everything is in /source. That
time can be divided by 2 at least with the proper load balancing between
a few md5 runners.

But that just to be run *once*. You would not compute it on every mirror
update but keep all md5 values somewhere.

So, recalculating the grand hash on every mirror update should takes a
few seconds because it would just consist of calculating the hash for
the new files, then
calculating the grand hash -- a loop that updates a md5 hash with 20k
hashes takes less than a second if I don't count the file reading.


I am not sure why we're having this discussion since it's implementation
details, but it's fun :)

If there's interest I can write a multiprocess-based script that keeps a
md5 database up-to-date
I'd be interested ;) Although i'd prefer sha256 personally. 

Cheers
Tarek


Regards,
Martin

_______________________________________________
Catalog-SIG mailing list
 

_______________________________________________
Catalog-SIG mailing list
Catalog-SIG <at> python.org
http://mail.python.org/mailman/listinfo/catalog-sig
Permalink | Reply |
headers
Tarek Ziadé | 17 Apr 2012 23:59
Favicon
Gravatar

Re: PyPI mirrors are all up to date

On 4/17/12 12:54 PM, Donald Stufft wrote:


If there's interest I can write a multiprocess-based script that keeps a
md5 database up-to-date
I'd be interested ;) Although i'd prefer sha256 personally. 

well, this is not really for security and I don't think a collision can happen that often with md5 :D

here's a raw script: http://tarek.pastebin.mozilla.org/1575563

The grand digest is done like a derived secret: I loop on the hash and do

   grand hash = hash(n & n+1) for n in hashes

I've run it against the 111,196 files I currently have in my mirror

- First *full* run from scratch - 15m32s  (not sure why I don't have better here, maybe Python's md5 is slower than md5deep)

- Second *full* run, md5 database filled - 2m33s - it scans the mirror, and adds missing md5s + build the grand digest.

- Just the digest, against a synced MD5 DB - 1m1s  (I just commented the first part that builds/updates the md5 db)

In a real mirror, once the first full run is done, the md5 db would be updated continuously everytime a new file is added
in the mirror, so the only extra load is recalculating the digest again.

So it would take around a minute each time, not a few seconds as I said previously. But that seems ok if a mirror is updated
for example every 5 minutes, 4 minutes can be spent to sync the files, and 1 minute to do the checksum I guess




Cheers
Tarek


Regards,
Martin

_______________________________________________
Catalog-SIG mailing list
 


_______________________________________________
Catalog-SIG mailing list
Catalog-SIG <at> python.org
http://mail.python.org/mailman/listinfo/catalog-sig
Permalink | Reply |
headers
martin | 17 Apr 2012 15:45
Picon
Gravatar

Re: PyPI mirrors are all up to date

> So you were updating a directory but serving another directory ?
>
> But then updating the right last-modified page people were seeing ?

It probably would have updated an unpublished last-modified as well.

For a similar issue, in appengine, I had duplicate File objects in the
database, and always served the one that GQL happened to return first.
In that case, both last-modified and the checksum might update correctly,
but still, the wrong file might get served.

> I am not sure why we're having this discussion since it's  
> implementation details, but it's fun :)

I'm still trying to prove my claim that it's not feasible to increase
the trustworthiness of a mirror by computing some kind of checksum.
If the mirror has some systematic or random error, it may well be that
the checksum is as-expected, yet the mirror is inconsistent.

> If there's interest I can write a multiprocess-based script that  
> keeps a md5 database up-to-date

That's besides the point. The question is whether doing so would practically
help to improve the consistency, and I believe the answer is: no. It may help
to increase people's trust (which is a subjective manner), which may be
worthwhile itself, but may also backfire if they download inconsistent files
despite the mirror giving "proof" that it is consistent.

Regards,
Martin
Permalink | Reply |
headers
Donald Stufft | 17 Apr 2012 19:57
Picon
Gravatar

Re: PyPI mirrors are all up to date

On Tuesday, April 17, 2012 at 9:45 AM, martin <at> v.loewis.de wrote:
So you were updating a directory but serving another directory ?

But then updating the right last-modified page people were seeing ?

It probably would have updated an unpublished last-modified as well.

For a similar issue, in appengine, I had duplicate File objects in the
database, and always served the one that GQL happened to return first.
In that case, both last-modified and the checksum might update correctly,
but still, the wrong file might get served.

I am not sure why we're having this discussion since it's
implementation details, but it's fun :)

I'm still trying to prove my claim that it's not feasible to increase
the trustworthiness of a mirror by computing some kind of checksum.
If the mirror has some systematic or random error, it may well be that
the checksum is as-expected, yet the mirror is inconsistent.
As I thought more about this, I think it is actually feasible. It's not feasible to
increase it to 100%, but in this case, assuming a bug over malevolence, the
bug would have to affect both the updating of the last-modified and the checksum
generation.

It's essentially the same thing as asking another person to check over some work,
that person could miss something, or also be wrong, but it decreases the likelihood
by adding another piece of verification. 

If there's interest I can write a multiprocess-based script that
keeps a md5 database up-to-date

That's besides the point. The question is whether doing so would practically
help to improve the consistency, and I believe the answer is: no. It may help
to increase people's trust (which is a subjective manner), which may be
worthwhile itself, but may also backfire if they download inconsistent files
despite the mirror giving "proof" that it is consistent.

Regards,
Martin


_______________________________________________
Catalog-SIG mailing list
 

_______________________________________________
Catalog-SIG mailing list
Catalog-SIG <at> python.org
http://mail.python.org/mailman/listinfo/catalog-sig
Permalink | Reply |
headers
Donald Stufft | 17 Apr 2012 00:10
Picon
Gravatar

Re: PyPI mirrors are all up to date

On Monday, April 16, 2012 at 5:57 PM, "Martin v. Löwis" wrote:
Maybe a better checksum would be a global hash calculated differently ?

Define a protocol, and I present you with an implementation that
conforms to the protocol, and still has inconsistent data, and not
in a malicious manner, but due to bugs/race conditions/unexpected
events. It's pointless. Ultimately, clients will need to verify the
data that they receive (if they suspect issues), and fall back gracefully.

I can definitely see a mirroring implementation where the
last-modified field is updated at the end while some packages are not
copied over at the end for whatever network issue.

That mirroring implementation would violate the principle that
last-modified should only be updated when the mirroring run was
completed successfully.
Is this documented anywhere? I don't see it in PEP381. I agree that
last-modified should only be updated when mirroring was completed
successfully but I don't see that specified as part of the mirroring protocol
which could lead to inconsistent definitions of what last-modified is. 

Regards,
Martin
_______________________________________________
Catalog-SIG mailing list
 

_______________________________________________
Catalog-SIG mailing list
Catalog-SIG <at> python.org
http://mail.python.org/mailman/listinfo/catalog-sig
Permalink | Reply |
headers
Martin v. Löwis | 17 Apr 2012 00:20
Picon
Gravatar

Re: PyPI mirrors are all up to date

>> That mirroring implementation would violate the principle that
>> last-modified should only be updated when the mirroring run was
>> completed successfully.
> Is this documented anywhere? I don't see it in PEP381.

It's certainly meant by "the last synchronisation date the mirror
maintains". Should I qualify that with "successful synchronization"
to make it more clear?

Regards,
Martin
Permalink | Reply |
headers
Donald Stufft | 17 Apr 2012 00:24
Picon
Gravatar

Re: PyPI mirrors are all up to date


On Monday, April 16, 2012 at 6:20 PM, "Martin v. Löwis" wrote:

That mirroring implementation would violate the principle that
last-modified should only be updated when the mirroring run was
completed successfully.
Is this documented anywhere? I don't see it in PEP381.

It's certainly meant by "the last synchronisation date the mirror
maintains". Should I qualify that with "successful synchronization"
to make it more clear?
That is more clear. Don't mean to be pedantic about it, just that when I
personally was implementing pep381 the fact that should be stored
_after_ the synchronization didn't occur to me (I ended up doing it that way
for other reasons anyways) and I thought that I might not be the only one. 

Regards,
Martin
 

_______________________________________________
Catalog-SIG mailing list
Catalog-SIG <at> python.org
http://mail.python.org/mailman/listinfo/catalog-sig
Permalink | Reply |
headers
zopyxfilter | 15 Apr 2012 17:31
Picon

Re: PyPI mirrors are all up to date

Hanno Schlichting wrote:
> On Sun, Apr 15, 2012 at 4:44 PM, Ken Cochrane <kencochrane <at> gmail.com> wrote:
>> Yeah I need to clean that up and make it more standard but this is how it
>> works now.
>>
>> Age < 5 min = excellent
>> Age > 5 min and age < 15 min = awesome
>> Age > 15 min and age < 1 hour = great.
>> Age > 1 hour and age <  6 hour = good
>> Age > 6 hours and age < 12 hour = OK
>> Age > 12 hours and age < 1 day = getting stale
>> Age > 1 day = out of date
>>
>> Feel free to make suggestions on how to improve.
> 
> My suggestion, keep it simple:
> 
> Age < 15 minutes - green (fresh)
> Age < 1 day - yellow (oldish)
> Age > 1 day - red (old)

+1

-aj
Permalink | Reply |
headers
Dylan Jay | 16 Apr 2012 09:10
Favicon
Gravatar

Re: PyPI mirrors are all up to date

Hi,

Some DNS load balancing services have api's. For example  
edgedirector.com, but there others (and perhaps cheaper).

Why not go to the next level and get the script on this page to update  
a DNS load balancer with all green mirrors?
Then switching mirrors would be automatic.

---
Dylan Jay
Technical Solutions Manager
PretaWeb: Multisite Performance Support
P: +612 80819071 | M: +61421477460 | twitter.com/djay75 | linkedin.com/ 
in/djay75

On 15/04/2012, at 11:07 PM, ken cochrane wrote:

> It looks like it took a little while, but all of the PyPI mirrors are
> now up to date. Thank you to everyone that helped get those back up to
> date. If you want to check out the current status I created this
> little website to make it easier to see where things stand mirror
> wise. I'm going to be adding a few more things, but it is fully useful
> right now. If you have any suggestions, please let me know.
>
> http://www.pypi-mirrors.org
>
>
> If you run an unofficial PyPI mirror and want it added to the list,
> just let me know.
>
> Thanks,
> Ken Cochrane
>  <at> KenCochrane
> _______________________________________________
> Catalog-SIG mailing list
> Catalog-SIG <at> python.org
> http://mail.python.org/mailman/listinfo/catalog-sig
Permalink | Reply |
headers
Noah Kantrowitz | 16 Apr 2012 09:42
Gravatar

Re: PyPI mirrors are all up to date


On Apr 16, 2012, at 12:10 AM, Dylan Jay wrote:

> Hi,
> 
> Some DNS load balancing services have api's. For example edgedirector.com, but there others (and
perhaps cheaper).
> 
> Why not go to the next level and get the script on this page to update a DNS load balancer with all green mirrors?
> Then switching mirrors would be automatic.

This wouldn't be a good idea since getting redirected to two different mirrors during one install attempt
would likely confuse most existing scripts if versions were slightly out of sync.

As for uptime scanning, I can add sensors for this to the PSF's existing Pingdom account and you could use
their API in the same way as http://ispypiup.com/. That would give some basic site-indpendence since
Pingdom has check servers all over the world (we currently require 6 sites to be in agreement before a
downtime alert is issued). If this interests you, let me know :-)

--Noah


_______________________________________________
Catalog-SIG mailing list
Catalog-SIG <at> python.org
http://mail.python.org/mailman/listinfo/catalog-sig
Permalink | Reply |
headers
Dylan Jay | 16 Apr 2012 10:05
Favicon
Gravatar

Re: PyPI mirrors are all up to date

On 16/04/2012, at 5:42 PM, Noah Kantrowitz wrote:

>
> On Apr 16, 2012, at 12:10 AM, Dylan Jay wrote:
>
>> Hi,
>>
>> Some DNS load balancing services have api's. For example  
>> edgedirector.com, but there others (and perhaps cheaper).
>>
>> Why not go to the next level and get the script on this page to  
>> update a DNS load balancer with all green mirrors?
>> Then switching mirrors would be automatic.
>
> This wouldn't be a good idea since getting redirected to two  
> different mirrors during one install attempt would likely confuse  
> most existing scripts if versions were slightly out of sync.

I don't think it would direct you to two different mirrors during one  
install attempt. Wouldn't a single python session cache DNS resolution?

>
> As for uptime scanning, I can add sensors for this to the PSF's  
> existing Pingdom account and you could use their API in the same way  
> as http://ispypiup.com/. That would give some basic site-indpendence  
> since Pingdom has check servers all over the world (we currently  
> require 6 sites to be in agreement before a downtime alert is  
> issued). If this interests you, let me know :-)
>
> --Noah
>
Permalink | Reply |
headers
Noah Kantrowitz | 16 Apr 2012 10:12
Gravatar

Re: PyPI mirrors are all up to date


On Apr 16, 2012, at 1:05 AM, Dylan Jay wrote:

> On 16/04/2012, at 5:42 PM, Noah Kantrowitz wrote:
> 
>> 
>> On Apr 16, 2012, at 12:10 AM, Dylan Jay wrote:
>> 
>>> Hi,
>>> 
>>> Some DNS load balancing services have api's. For example edgedirector.com, but there others (and
perhaps cheaper).
>>> 
>>> Why not go to the next level and get the script on this page to update a DNS load balancer with all green mirrors?
>>> Then switching mirrors would be automatic.
>> 
>> This wouldn't be a good idea since getting redirected to two different mirrors during one install
attempt would likely confuse most existing scripts if versions were slightly out of sync.
> 
> I don't think it would direct you to two different mirrors during one install attempt. Wouldn't a single
python session cache DNS resolution?

I don't see how you could assume that given that it would follow the system resolver's behavior in most
cases, and that isn't the kind of thing to count on. Regardless the uptime of PyPI will shortly become a
non-issue :-)

--Noah
_______________________________________________
Catalog-SIG mailing list
Catalog-SIG <at> python.org
http://mail.python.org/mailman/listinfo/catalog-sig
Permalink | Reply |
headers
ken cochrane | 16 Apr 2012 21:44
Picon
Gravatar

Re: PyPI mirrors are all up to date

> As for uptime scanning, I can add sensors for this to the PSF's existing Pingdom account and you could use
their API in the same way as http://ispypiup.com/. That would give some basic site-indpendence since
Pingdom has check servers all over the world (we currently require 6 sites to be in agreement before a
downtime alert is issued). If this interests you, let me know :-)
>

That might be interesting, I'll need to figure out how to fit it with
the current setup, but I'm open to suggestions.

Ken
Permalink | Reply |
headers
Martin v. Löwis | 16 Apr 2012 15:30
Picon
Gravatar

Re: PyPI mirrors are all up to date

> Why not go to the next level and get the script on this page to update a
> DNS load balancer with all green mirrors?

I still think that mirror selection should be on the client side. First,
if you have an infrastructure that updates the mirror list, it is again
a single point of failure. Plus, clients would want to select a mirror
based on network performance, which a DNS load balancer cannot achieve.

Regards,
Martin
Permalink | Reply |
headers
Andreas Jung | 16 Apr 2012 15:38

Re: PyPI mirrors are all up to date


Martin v. Löwis wrote:
>> Why not go to the next level and get the script on this page to
>> update a DNS load balancer with all green mirrors?
> 
> I still think that mirror selection should be on the client side.
> First, if you have an infrastructure that updates the mirror list, it
> is again a single point of failure. Plus, clients would want to
> select a mirror based on network performance, which a DNS load
> balancer cannot achieve.

+1

-aj
Attachment (lists.vcf): text/x-vcard, 310 bytes
_______________________________________________
Catalog-SIG mailing list
Catalog-SIG <at> python.org
http://mail.python.org/mailman/listinfo/catalog-sig
Permalink | Reply |
headers
Donald Stufft | 16 Apr 2012 15:44
Picon
Gravatar

Re: PyPI mirrors are all up to date

fwiw I go back and forth on how I feel about this. For sure the client wants to select the mirror
based on network performance to them. But I feel like if a mirror get's to be "stuck" and not
updating that PyPI should at some point remove it from the pool (and notify the owner that it has done so)
until it catches back up.

On Monday, April 16, 2012 at 9:30 AM, "Martin v. Löwis" wrote:

Why not go to the next level and get the script on this page to update a
DNS load balancer with all green mirrors?

I still think that mirror selection should be on the client side. First,
if you have an infrastructure that updates the mirror list, it is again
a single point of failure. Plus, clients would want to select a mirror
based on network performance, which a DNS load balancer cannot achieve.

Regards,
Martin
_______________________________________________
Catalog-SIG mailing list
 

_______________________________________________
Catalog-SIG mailing list
Catalog-SIG <at> python.org
http://mail.python.org/mailman/listinfo/catalog-sig
Permalink | Reply |
headers
Martin v. Löwis | 16 Apr 2012 16:25
Picon
Gravatar

Re: PyPI mirrors are all up to date

Am 16.04.2012 15:44, schrieb Donald Stufft:
> fwiw I go back and forth on how I feel about this. For sure the client
> wants to select the mirror
> based on network performance to them. But I feel like if a mirror get's
> to be "stuck" and not
> updating that PyPI should at some point remove it from the pool (and
> notify the owner that it has done so)
> until it catches back up.

This is actually how it works. There may be differences though in what
the threshold for removal is. I'd personally set it to
a) trying to reach the operator at least three times, *and*
b) the mirror being outdated by more than six month (sic).

>From my own experience, I find it very plausible to not be able to
work on a problem for several months, so an outage by that time
doesn't bother me - as long as clients can reliably detect the outage.
If the mirror was corrupt in some other way (e.g. claiming to be
up-to-date even though it is not) is a different story; that could
cause immediate removal.

Regards,
Martin
Permalink | Reply |
headers
Dylan Jay | 16 Apr 2012 19:04
Favicon
Gravatar

Re: PyPI mirrors are all up to date


On 16/04/2012, at 11:30 PM, Martin v. Löwis wrote:

>> Why not go to the next level and get the script on this page to  
>> update a
>> DNS load balancer with all green mirrors?
>
> I still think that mirror selection should be on the client side.  
> First,
> if you have an infrastructure that updates the mirror list, it is  
> again
> a single point of failure. Plus, clients would want to select a mirror
> based on network performance, which a DNS load balancer cannot  
> achieve.

One of the services offered by edgedirector.com etc is "geo-targetted  
global load balancing" which means they will serve up the IP closest  
to your location.

In any case offering a load balanced domain like "any.pypi.org"  
doesn't preclude individual mirror domains. The user could choose.

>
> Regards,
> Martin
Permalink | Reply |
headers
ken cochrane | 16 Apr 2012 21:49
Picon
Gravatar

Re: PyPI mirrors are all up to date

Dyn offers those services as well, and I believe they are the current
DNS provider for python.org, If they are going to add those sort of
features, (which I have suggested in the past), then it might be best
to stick the current provider.

http://dyn.com/dns/dynect-managed-dns/

Ken

On Mon, Apr 16, 2012 at 1:04 PM, Dylan Jay <djay <at> pretaweb.com> wrote:
>
> On 16/04/2012, at 11:30 PM, Martin v. Löwis wrote:
>
>>> Why not go to the next level and get the script on this page to update a
>>> DNS load balancer with all green mirrors?
>>
>>
>> I still think that mirror selection should be on the client side. First,
>> if you have an infrastructure that updates the mirror list, it is again
>> a single point of failure. Plus, clients would want to select a mirror
>> based on network performance, which a DNS load balancer cannot achieve.
>
>
> One of the services offered by edgedirector.com etc is "geo-targetted global
> load balancing" which means they will serve up the IP closest to your
> location.
>
> In any case offering a load balanced domain like "any.pypi.org" doesn't
> preclude individual mirror domains. The user could choose.
>
>
>>
>> Regards,
>> Martin
>
>
Permalink | Reply |
headers
Tarek Ziadé | 16 Apr 2012 09:23
Favicon
Gravatar

Re: PyPI mirrors are all up to date

On 4/15/12 3:07 PM, ken cochrane wrote:
> It looks like it took a little while, but all of the PyPI mirrors are
> now up to date. Thank you to everyone that helped get those back up to
> date. If you want to check out the current status I created this
> little website to make it easier to see where things stand mirror
> wise. I'm going to be adding a few more things, but it is fully useful
> right now. If you have any suggestions, please let me know.
>
> http://www.pypi-mirrors.org
>
>
> If you run an unofficial PyPI mirror and want it added to the list,
> just let me know.
extra !

a nice feature would be to send a mail automatically in this ML when a 
mirror becomes "old"

Cheers & thanks for this work
Tarek

>
> Thanks,
> Ken Cochrane
>  <at> KenCochrane
> _______________________________________________
> Catalog-SIG mailing list
> Catalog-SIG <at> python.org
> http://mail.python.org/mailman/listinfo/catalog-sig
Permalink | Reply |
headers
ken cochrane | 16 Apr 2012 21:49
Picon
Gravatar

Re: PyPI mirrors are all up to date

On the list, thanks..

On Mon, Apr 16, 2012 at 3:23 AM, Tarek Ziadé <tarek <at> ziade.org> wrote:
>
> a nice feature would be to send a mail automatically in this ML when a
> mirror becomes "old"
Permalink | Reply |

Gmane