Wouter Verhelst | 20 Aug 06:22
Favicon
Gravatar

Hardware compatibility test: draft proposal

Hi,

As I mentioned in my blog[1], I kindof like the suggestion that Bdale
came up with during Debconf that we write a hardware compatibility test
of sorts that hardware vendors could run on their own hardware to test
whether Debian works on their system. The rationale for such a test:
while most of us know how Debian works and how one should test whether
their hardware actually works with Debian, it is not reasonable to
expect the same of hardware vendors for /all/ GNU/Linux distributions
out there. According to Bdale, currently all other major operating
system vendors, including the commercial Linux distributions, already
provide such a test to the hardware vendors. For this purpose, it is
okay if such a test is interactive to some extent (after all, it is
something that you would run on a hardware prototype in a lab, not on
each and every production machine), although I'm thinking a hardware
compatibility test could be useful in more cases, where it might be
better if it wasn't interactive.

So, after more than twelve hours of boredom on an airplane and half a
night of not-being-able-to-sleep-due-to-jetlag, which is certainly
enough to think about this problem, I came up with the following things
such a system could need:

- It should be modular. People who maintain driver packages for
  particular hardware may want to write additional tests that a vendor
  may want to run; and if this particular package supports it, the
  driver package maintainer may want to provide pointers to a particular
  package so that an inexperienced user may be able to configure their
  hardware after running the test themselves.
- The different tests should each be able to communicate what type of
(Continue reading)

Frank Lichtenheld | 21 Aug 19:54
Favicon

Re: Hardware compatibility test: draft proposal

On Wed, Aug 20, 2008 at 06:24:07AM +0200, Wouter Verhelst wrote:
> So, that's probably it at this point. I should have a look at bdale's
> talk once it becomes available at meetings-archive.debian.net, but that
> doesn't yet seem to be the case. If someone who was at bdale's talk has
> anything to add, that'd be welcome. If someone could think of anything
> even without being at bdale's talk, that's welcome too, of course.

http://meetings-archive.debian.net/pub/debian-meetings/2008/debconf8/high/508_hp.comgodebian.ogg

Gruesse,
--

-- 
Frank Lichtenheld <djpig <at> debian.org>
www: http://www.djpig.de/

Petter Reinholdtsen | 21 Aug 20:27

Re: Hardware compatibility test: draft proposal


[Wouter Verhelst]
> As I mentioned in my blog[1], I kindof like the suggestion that
> Bdale came up with during Debconf that we write a hardware
> compatibility test of sorts that hardware vendors could run on their
> own hardware to test whether Debian works on their system.

I saw Bdales talk and I like it too.  I did not spend time thinking
about your proposal, but wanted to let you know how we test
compatibility in a Debian Edu installation.

It is done in a installation where PXE installing work, and the first
step in the compatibility test is to install Debian Edu automatically
with all questions preseeded.  If installation work, the hard drive
work.

If the KDE login manager show up (kdm), the graphics card work.

Next step is to log in, and if a small sound is played, the sound card
is working.

Start a web browser, and visit <URL:http://www.skolelinux.org/>.  If
it work, the network card is functioning.

Next is 3D graphics.  Select"Science & Math->Stellarium" from the K
meny, and see if the program is snappy.  If it is, accelerated OpenGL
is working and the 3D stuff should work fine.

Insert a USB memory stick, and see if a dialog box pop up after a few
seconds to ask what should be done.  If it does, USB is working.
(Continue reading)

Favicon

Re: Hardware compatibility test: draft proposal

On Thu, 21 Aug 2008, Petter Reinholdtsen wrote:
> It is done in a installation where PXE installing work, and the first
> step in the compatibility test is to install Debian Edu automatically
> with all questions preseeded.  If installation work, the hard drive
> work.

[...]

Your idea is good, and a simple way to partially test things, yes.

I'd add a pre-run of the Linux firmware kit, and also grep the kernel log
for any ACPI strings.  IMHO, one should reject outright any machines that
outputs any warnings from the firmware kit, unless the vendor promises to
fix it.  Even warnings.  These things come back to bite you later.

--

-- 
  "One disk to rule them all, One disk to find them. One disk to bring
  them all and in the darkness grind them. In the Land of Redmond
  where the shadows lie." -- The Silicon Valley Tarot
  Henrique Holschuh

Chris Lamb | 22 Aug 12:05
X-Face
Gravatar

Re: Hardware compatibility test: draft proposal

Wouter Verhelst wrote:

> As I mentioned in my blog[1], I kindof like the suggestion that Bdale
> came up with during Debconf that we write a hardware compatibility test
> of sorts that hardware vendors could run on their own hardware

I think this is a great idea, and I generally agree with your decisions and
assumptions.

>   - Scripts should not just test for availability of hardware. Instead,
>     they should test the actual functionality; e.g., tests for a network
>     interface should be done by trying to DHCP off that interface, X.org
>     drivers should try to start X and ask for input using a graphical
>     dialog, and tests for a hard disk should be done by trying to read
>     some data from the disk.

I think this the most important paragraph of all.

What I think is missing is some really concrete info on just how various
hardware items would be tested. For example, in the case of ethernet
adaptors, I feel that simply successfully DHCP-ing on an interface is really
not an acceptable test.

As an example, Etch's kernel contains various modules (such as sky2) that
"kinda" work on today's hardware - whilst the driver would probably pass a
DHCP test, the actual performance or reliability of the device is completely
inadequate. Wireless devices would pose an even more difficult problem, as
support for the various encryption modes that the device supports tend to be
developed at different times, etc.

(Continue reading)

Wouter Verhelst | 24 Aug 01:52
Favicon
Gravatar

Re: Hardware compatibility test: draft proposal

On Fri, Aug 22, 2008 at 11:05:49AM +0100, Chris Lamb wrote:
> Wouter Verhelst wrote:
> 
> > As I mentioned in my blog[1], I kindof like the suggestion that Bdale
> > came up with during Debconf that we write a hardware compatibility test
> > of sorts that hardware vendors could run on their own hardware
> 
> I think this is a great idea, and I generally agree with your decisions and
> assumptions.
> 
> >   - Scripts should not just test for availability of hardware. Instead,
> >     they should test the actual functionality; e.g., tests for a network
> >     interface should be done by trying to DHCP off that interface, X.org
> >     drivers should try to start X and ask for input using a graphical
> >     dialog, and tests for a hard disk should be done by trying to read
> >     some data from the disk.
> 
> I think this the most important paragraph of all.
> 
> What I think is missing is some really concrete info on just how various
> hardware items would be tested.

Well, that's the hard bit, isn't it? ;-)

I didn't go into much detail here yet, simply because I feel this is
something we'll have to figure out as we write everything.

> For example, in the case of ethernet adaptors, I feel that simply
> successfully DHCP-ing on an interface is really not an acceptable
> test.
(Continue reading)

Ben Hutchings | 30 Aug 14:20

Re: Hardware compatibility test: draft proposal

On Fri, 2008-08-22 at 11:05 +0100, Chris Lamb wrote: 
> Wouter Verhelst wrote:
> 
> > As I mentioned in my blog[1], I kindof like the suggestion that Bdale
> > came up with during Debconf that we write a hardware compatibility test
> > of sorts that hardware vendors could run on their own hardware
> 
> I think this is a great idea, and I generally agree with your decisions and
> assumptions.
> 
> >   - Scripts should not just test for availability of hardware. Instead,
> >     they should test the actual functionality; e.g., tests for a network
> >     interface should be done by trying to DHCP off that interface, X.org
> >     drivers should try to start X and ask for input using a graphical
> >     dialog, and tests for a hard disk should be done by trying to read
> >     some data from the disk.
> 
> I think this the most important paragraph of all.
> 
> What I think is missing is some really concrete info on just how various
> hardware items would be tested. For example, in the case of ethernet
> adaptors, I feel that simply successfully DHCP-ing on an interface is really
> not an acceptable test.
<snip> 

At a bare minimum the HCT should try to run the driver's offline
self-test (ethtool -t $dev offline).  For a more comprehensive (and
generic) test we would really want to use two machines, though an
external loopback cable may suffice.

(Continue reading)

Giacomo Catenazzi | 22 Aug 12:55
Favicon

Re: Hardware compatibility test: draft proposal

Wouter Verhelst wrote:
> Hi,
> 
> As I mentioned in my blog[1], I kindof like the suggestion that Bdale

Yes, I find the talk very interesting.

> So, after more than twelve hours of boredom on an airplane and half a
> night of not-being-able-to-sleep-due-to-jetlag, which is certainly
> enough to think about this problem, I came up with the following things
> such a system could need:

/me too

> - It should support a notion of what I'll call 'profiles'. A 'server'
>   profile should check for different things than a 'desktop' or 'laptop'
>   profile; e.g., it's usually okay if a server doesn't have graphical
>   support or wireless drivers, while the same isn't true for a laptop.

No. I don't think we should provide profiles.
The check should be:
"complete": test all hardware that it is installed on system.
If the system doesn't have the video-card or the wifi just it doesn't
test it, but it write a notice.
Listening the available hardware is pretty trivial (see i.e. my
AutoKernConf).

Additionally a component test should be furnished, for component
hardware vendor. (e.g. a network card vendor would test
only the network card, not the whole system).
(Continue reading)

Joe Smith | 22 Aug 23:14
Favicon

Re: Hardware compatibility test: draft proposal


"Giacomo Catenazzi" <cate <at> debian.org> wrote in message 
news:48AE9B26.4040704 <at> debian.org...
> Wouter Verhelst wrote:
>> Hi,
>>
>> As I mentioned in my blog[1], I kindof like the suggestion that Bdale
>
> Yes, I find the talk very interesting.
>
>> So, after more than twelve hours of boredom on an airplane and half a
>> night of not-being-able-to-sleep-due-to-jetlag, which is certainly
>> enough to think about this problem, I came up with the following things
>> such a system could need:
>
> /me too
>
>> - It should support a notion of what I'll call 'profiles'. A 'server'
>>   profile should check for different things than a 'desktop' or 'laptop'
>>   profile; e.g., it's usually okay if a server doesn't have graphical
>>   support or wireless drivers, while the same isn't true for a laptop.
>
> No. I don't think we should provide profiles.
> The check should be:
> "complete": test all hardware that it is installed on system.
> If the system doesn't have the video-card or the wifi just it doesn't
> test it, but it write a notice.
> Listening the available hardware is pretty trivial (see i.e. my
> AutoKernConf).

(Continue reading)

Giacomo A. Catenazzi | 25 Aug 18:51
Favicon

Re: Hardware compatibility test: draft proposal

Joe Smith wrote:
 >
 > "Giacomo Catenazzi" <cate <at> debian.org> wrote in message
 > news:48AE9B26.4040704 <at> debian.org...
 >> Wouter Verhelst wrote:
 >>> Hi,
 >>>
 >>> As I mentioned in my blog[1], I kindof like the suggestion that Bdale
 >>
 >> Yes, I find the talk very interesting.
 >>
 >>> So, after more than twelve hours of boredom on an airplane and half a
 >>> night of not-being-able-to-sleep-due-to-jetlag, which is certainly
 >>> enough to think about this problem, I came up with the following things
 >>> such a system could need:
 >>
 >> /me too
 >>
 >>> - It should support a notion of what I'll call 'profiles'. A 'server'
 >>>   profile should check for different things than a 'desktop' or 
'laptop'
 >>>   profile; e.g., it's usually okay if a server doesn't have graphical
 >>>   support or wireless drivers, while the same isn't true for a laptop.
 >>
 >> No. I don't think we should provide profiles.
 >> The check should be:
 >> "complete": test all hardware that it is installed on system.
 >> If the system doesn't have the video-card or the wifi just it doesn't
 >> test it, but it write a notice.
 >> Listening the available hardware is pretty trivial (see i.e. my
(Continue reading)

Wouter Verhelst | 22 Aug 20:10
Gravatar

Re: Hardware compatibility test: draft proposal

On Fri, Aug 22, 2008 at 12:55:34PM +0200, Giacomo Catenazzi wrote:
> Wouter Verhelst wrote:
> > Hi,
> > 
> > As I mentioned in my blog[1], I kindof like the suggestion that Bdale
> 
> Yes, I find the talk very interesting.
> 
> > So, after more than twelve hours of boredom on an airplane and half a
> > night of not-being-able-to-sleep-due-to-jetlag, which is certainly
> > enough to think about this problem, I came up with the following things
> > such a system could need:
> 
> /me too
> 
> > - It should support a notion of what I'll call 'profiles'. A 'server'
> >   profile should check for different things than a 'desktop' or 'laptop'
> >   profile; e.g., it's usually okay if a server doesn't have graphical
> >   support or wireless drivers, while the same isn't true for a laptop.
> 
> No. I don't think we should provide profiles.
> The check should be:
> "complete": test all hardware that it is installed on system.
> If the system doesn't have the video-card or the wifi just it doesn't
> test it, but it write a notice.

That will create ambiguity, which I think we should avoid at all cost.
If you tell a user of this test that "it seems to work, but you should
check this and this and this output to make sure we didn't miss
anything", then such a test is worthless; hardware vendors *will* miss
(Continue reading)

Giacomo A. Catenazzi | 25 Aug 19:11
Favicon

Re: Hardware compatibility test: draft proposal

Wouter Verhelst wrote:
> On Fri, Aug 22, 2008 at 12:55:34PM +0200, Giacomo Catenazzi wrote:
>> Wouter Verhelst wrote:
>>> Hi,
>>>
>>> As I mentioned in my blog[1], I kindof like the suggestion that Bdale
>> Yes, I find the talk very interesting.
>>
>>> So, after more than twelve hours of boredom on an airplane and half a
>>> night of not-being-able-to-sleep-due-to-jetlag, which is certainly
>>> enough to think about this problem, I came up with the following things
>>> such a system could need:
>> /me too
>>
>>> - It should support a notion of what I'll call 'profiles'. A 'server'
>>>   profile should check for different things than a 'desktop' or 'laptop'
>>>   profile; e.g., it's usually okay if a server doesn't have graphical
>>>   support or wireless drivers, while the same isn't true for a laptop.
>> No. I don't think we should provide profiles.
>> The check should be:
>> "complete": test all hardware that it is installed on system.
>> If the system doesn't have the video-card or the wifi just it doesn't
>> test it, but it write a notice.
> 
> That will create ambiguity, which I think we should avoid at all cost.
> If you tell a user of this test that "it seems to work, but you should
> check this and this and this output to make sure we didn't miss
> anything", then such a test is worthless; hardware vendors *will* miss
> this kind of thing.

(Continue reading)

Giacomo Catenazzi | 26 Aug 09:02
Favicon

Re: Hardware compatibility test: draft proposal

Giacomo A. Catenazzi wrote:
> Wouter Verhelst wrote:
>> On Fri, Aug 22, 2008 at 12:55:34PM +0200, Giacomo Catenazzi wrote:
>>> Wouter Verhelst wrote:

>>> I'm not so sure that the test should be packages.
>>> The system is too different on early boot from a Debian system.
>>
>> In what way? Note that a Debian Live system *is* a normal Debian system;
>> you build a Debian Live image by giving the debian-live script a bunch
>> of packages; it then installs them into a chroot, changes a few things
>> so that the system will actually boot from CD-ROM, and then create an
>> ISO image from it.
>>
>> How is that "too different" from a Debian system? Or are you talking
>> about something else?

ok. I was thinking about doing hardware test a lot earlier
(i.e. in initramfs).
But probably it is not really needed, and we probably could assume
that system is enough reasonable.
Anyway, we can try with debian-live, and ev. move things
earlier.

>>> The "modular" thing should be put mainly on the testing
>>> development side.
>>
>> Yeah, I'm not suggesting we provide vendors with a bunch of packages;
>> the packages thing is a way for us to make it easier to actually develop
>> this without having one maintainer who has to write/accept/maintain
(Continue reading)

Mikhail Gusarov | 23 Aug 17:22
Favicon
Gravatar

Re: Hardware compatibility test: draft proposal

Twas brillig at 06:24:07 20.08.2008 UTC+02 when wouter <at> debian.org did gyre and gimble:

[]

JFYI, there's great piece of software which can be reused: http://www.inquisitor.ru/about/

--

-- 

Gmane