Using Linux Raid Systems

Shaohua Li | 9 Aug 2012 10:58

[patch 2/2 v3]raid5: create multiple threads to handle stripes

This is a new tempt to make raid5 handle stripes in multiple threads, as
suggested by Neil to have maxium flexibility and better numa binding. It
basically is a combination of my first and second generation patches. By
default, no multiple thread is enabled (all stripes are handled by raid5d).

An example to enable multiple threads:
#echo 3 > /sys/block/md0/md/auxthread_number
This will create 3 auxiliary threads to handle stripes. The threads can run
on any cpus and handle stripes produced by any cpus.

#echo 1-3 > /sys/block/md0/md/auxth0/cpulist
This will bind auxiliary thread 0 to cpu 1-3, and this thread will only handle
stripes produced by cpu 1-3. User tool can further change the thread's
affinity, but the thread can only handle stripes produced by cpu 1-3 till the
sysfs entry is changed again.

If stripes produced by a CPU aren't handled by any auxiliary thread, such
stripes will be handled by raid5d. Otherwise, raid5d doesn't handle any
stripes.

Signed-off-by: Shaohua Li <shli <at> fusionio.com>
---
 drivers/md/md.c    |    8 -
 drivers/md/md.h    |    8 +
 drivers/md/raid5.c |  406 ++++++++++++++++++++++++++++++++++++++++++++++++++---
 drivers/md/raid5.h |   19 ++
 4 files changed, 418 insertions(+), 23 deletions(-)

Index: linux/drivers/md/raid5.c
===================================================================

(Continue reading)

headers

Jianpeng Ma | 11 Aug 2012 10:45

Re: [patch 2/2 v3]raid5: create multiple threads to handle stripes

On 2012-08-09 16:58 Shaohua Li <shli <at> kernel.org> Wrote:
>This is a new tempt to make raid5 handle stripes in multiple threads, as
>suggested by Neil to have maxium flexibility and better numa binding. It
>basically is a combination of my first and second generation patches. By
>default, no multiple thread is enabled (all stripes are handled by raid5d).
>
>An example to enable multiple threads:
>#echo 3 > /sys/block/md0/md/auxthread_number
>This will create 3 auxiliary threads to handle stripes. The threads can run
>on any cpus and handle stripes produced by any cpus.
>
>#echo 1-3 > /sys/block/md0/md/auxth0/cpulist
>This will bind auxiliary thread 0 to cpu 1-3, and this thread will only handle
>stripes produced by cpu 1-3. User tool can further change the thread's
>affinity, but the thread can only handle stripes produced by cpu 1-3 till the
>sysfs entry is changed again.
>
>If stripes produced by a CPU aren't handled by any auxiliary thread, such
>stripes will be handled by raid5d. Otherwise, raid5d doesn't handle any
>stripes.
>
I tested and found two problem(maybe not).

1:print cpulist of auxth, you maybe lost print the '\n'.
diff --git a/drivers/md/raid5.c b/drivers/md/raid5.c
index 7c8151a..3700cdc 100644
--- a/drivers/md/raid5.c
+++ b/drivers/md/raid5.c
 <at>  <at>  -4911,9 +4911,13  <at>  <at>  struct raid5_auxth_sysfs {
 static ssize_t raid5_show_thread_cpulist(struct mddev *mddev,

(Continue reading)

headers

Shaohua Li | 13 Aug 2012 02:21

Re: [patch 2/2 v3]raid5: create multiple threads to handle stripes

2012/8/11 Jianpeng Ma <majianpeng <at> gmail.com>:
> On 2012-08-09 16:58 Shaohua Li <shli <at> kernel.org> Wrote:
>>This is a new tempt to make raid5 handle stripes in multiple threads, as
>>suggested by Neil to have maxium flexibility and better numa binding. It
>>basically is a combination of my first and second generation patches. By
>>default, no multiple thread is enabled (all stripes are handled by raid5d).
>>
>>An example to enable multiple threads:
>>#echo 3 > /sys/block/md0/md/auxthread_number
>>This will create 3 auxiliary threads to handle stripes. The threads can run
>>on any cpus and handle stripes produced by any cpus.
>>
>>#echo 1-3 > /sys/block/md0/md/auxth0/cpulist
>>This will bind auxiliary thread 0 to cpu 1-3, and this thread will only handle
>>stripes produced by cpu 1-3. User tool can further change the thread's
>>affinity, but the thread can only handle stripes produced by cpu 1-3 till the
>>sysfs entry is changed again.
>>
>>If stripes produced by a CPU aren't handled by any auxiliary thread, such
>>stripes will be handled by raid5d. Otherwise, raid5d doesn't handle any
>>stripes.
>>
> I tested and found two problem(maybe not).
>
> 1:print cpulist of auxth, you maybe lost print the '\n'.
> diff --git a/drivers/md/raid5.c b/drivers/md/raid5.c
> index 7c8151a..3700cdc 100644
> --- a/drivers/md/raid5.c
> +++ b/drivers/md/raid5.c
>  <at>  <at>  -4911,9 +4911,13  <at>  <at>  struct raid5_auxth_sysfs {

(Continue reading)

headers

Jianpeng Ma | 13 Aug 2012 03:06

Re: Re: [patch 2/2 v3]raid5: create multiple threads to handle stripes

On 2012-08-13 08:21 Shaohua Li <shli <at> kernel.org> Wrote:
>2012/8/11 Jianpeng Ma <majianpeng <at> gmail.com>:
>> On 2012-08-09 16:58 Shaohua Li <shli <at> kernel.org> Wrote:
>>>This is a new tempt to make raid5 handle stripes in multiple threads, as
>>>suggested by Neil to have maxium flexibility and better numa binding. It
>>>basically is a combination of my first and second generation patches. By
>>>default, no multiple thread is enabled (all stripes are handled by raid5d).
>>>
>>>An example to enable multiple threads:
>>>#echo 3 > /sys/block/md0/md/auxthread_number
>>>This will create 3 auxiliary threads to handle stripes. The threads can run
>>>on any cpus and handle stripes produced by any cpus.
>>>
>>>#echo 1-3 > /sys/block/md0/md/auxth0/cpulist
>>>This will bind auxiliary thread 0 to cpu 1-3, and this thread will only handle
>>>stripes produced by cpu 1-3. User tool can further change the thread's
>>>affinity, but the thread can only handle stripes produced by cpu 1-3 till the
>>>sysfs entry is changed again.
>>>
>>>If stripes produced by a CPU aren't handled by any auxiliary thread, such
>>>stripes will be handled by raid5d. Otherwise, raid5d doesn't handle any
>>>stripes.
>>>
>> I tested and found two problem(maybe not).
>>
>> 1:print cpulist of auxth, you maybe lost print the '\n'.
>> diff --git a/drivers/md/raid5.c b/drivers/md/raid5.c
>> index 7c8151a..3700cdc 100644
>> --- a/drivers/md/raid5.c
>> +++ b/drivers/md/raid5.c

(Continue reading)

headers

Shaohua Li | 13 Aug 2012 04:13

Re: Re: [patch 2/2 v3]raid5: create multiple threads to handle stripes

On Mon, Aug 13, 2012 at 09:06:45AM +0800, Jianpeng Ma wrote:
> On 2012-08-13 08:21 Shaohua Li <shli <at> kernel.org> Wrote:
> >2012/8/11 Jianpeng Ma <majianpeng <at> gmail.com>:
> >> On 2012-08-09 16:58 Shaohua Li <shli <at> kernel.org> Wrote:
> >>>This is a new tempt to make raid5 handle stripes in multiple threads, as
> >>>suggested by Neil to have maxium flexibility and better numa binding. It
> >>>basically is a combination of my first and second generation patches. By
> >>>default, no multiple thread is enabled (all stripes are handled by raid5d).
> >>>
> >>>An example to enable multiple threads:
> >>>#echo 3 > /sys/block/md0/md/auxthread_number
> >>>This will create 3 auxiliary threads to handle stripes. The threads can run
> >>>on any cpus and handle stripes produced by any cpus.
> >>>
> >>>#echo 1-3 > /sys/block/md0/md/auxth0/cpulist
> >>>This will bind auxiliary thread 0 to cpu 1-3, and this thread will only handle
> >>>stripes produced by cpu 1-3. User tool can further change the thread's
> >>>affinity, but the thread can only handle stripes produced by cpu 1-3 till the
> >>>sysfs entry is changed again.
> >>>
> >>>If stripes produced by a CPU aren't handled by any auxiliary thread, such
> >>>stripes will be handled by raid5d. Otherwise, raid5d doesn't handle any
> >>>stripes.
> >>>
> >> I tested and found two problem(maybe not).
> >>
> >> 1:print cpulist of auxth, you maybe lost print the '\n'.
> >> diff --git a/drivers/md/raid5.c b/drivers/md/raid5.c
> >> index 7c8151a..3700cdc 100644
> >> --- a/drivers/md/raid5.c

(Continue reading)

headers

Shaohua Li | 13 Aug 2012 04:20

Re: Re: [patch 2/2 v3]raid5: create multiple threads to handle stripes

2012/8/13 Shaohua Li <shli <at> kernel.org>:
> On Mon, Aug 13, 2012 at 09:06:45AM +0800, Jianpeng Ma wrote:
>> On 2012-08-13 08:21 Shaohua Li <shli <at> kernel.org> Wrote:
>> >2012/8/11 Jianpeng Ma <majianpeng <at> gmail.com>:
>> >> On 2012-08-09 16:58 Shaohua Li <shli <at> kernel.org> Wrote:
>> >>>This is a new tempt to make raid5 handle stripes in multiple threads, as
>> >>>suggested by Neil to have maxium flexibility and better numa binding. It
>> >>>basically is a combination of my first and second generation patches. By
>> >>>default, no multiple thread is enabled (all stripes are handled by raid5d).
>> >>>
>> >>>An example to enable multiple threads:
>> >>>#echo 3 > /sys/block/md0/md/auxthread_number
>> >>>This will create 3 auxiliary threads to handle stripes. The threads can run
>> >>>on any cpus and handle stripes produced by any cpus.
>> >>>
>> >>>#echo 1-3 > /sys/block/md0/md/auxth0/cpulist
>> >>>This will bind auxiliary thread 0 to cpu 1-3, and this thread will only handle
>> >>>stripes produced by cpu 1-3. User tool can further change the thread's
>> >>>affinity, but the thread can only handle stripes produced by cpu 1-3 till the
>> >>>sysfs entry is changed again.
>> >>>
>> >>>If stripes produced by a CPU aren't handled by any auxiliary thread, such
>> >>>stripes will be handled by raid5d. Otherwise, raid5d doesn't handle any
>> >>>stripes.
>> >>>
>> >> I tested and found two problem(maybe not).
>> >>
>> >> 1:print cpulist of auxth, you maybe lost print the '\n'.
>> >> diff --git a/drivers/md/raid5.c b/drivers/md/raid5.c
>> >> index 7c8151a..3700cdc 100644

(Continue reading)

headers

Jianpeng Ma | 13 Aug 2012 04:25

Re: Re: [patch 2/2 v3]raid5: create multiple threads to handle stripes

On 2012-08-13 10:20 Shaohua Li <shli <at> kernel.org> Wrote:
>2012/8/13 Shaohua Li <shli <at> kernel.org>:
>> On Mon, Aug 13, 2012 at 09:06:45AM +0800, Jianpeng Ma wrote:
>>> On 2012-08-13 08:21 Shaohua Li <shli <at> kernel.org> Wrote:
>>> >2012/8/11 Jianpeng Ma <majianpeng <at> gmail.com>:
>>> >> On 2012-08-09 16:58 Shaohua Li <shli <at> kernel.org> Wrote:
>>> >>>This is a new tempt to make raid5 handle stripes in multiple threads, as
>>> >>>suggested by Neil to have maxium flexibility and better numa binding. It
>>> >>>basically is a combination of my first and second generation patches. By
>>> >>>default, no multiple thread is enabled (all stripes are handled by raid5d).
>>> >>>
>>> >>>An example to enable multiple threads:
>>> >>>#echo 3 > /sys/block/md0/md/auxthread_number
>>> >>>This will create 3 auxiliary threads to handle stripes. The threads can run
>>> >>>on any cpus and handle stripes produced by any cpus.
>>> >>>
>>> >>>#echo 1-3 > /sys/block/md0/md/auxth0/cpulist
>>> >>>This will bind auxiliary thread 0 to cpu 1-3, and this thread will only handle
>>> >>>stripes produced by cpu 1-3. User tool can further change the thread's
>>> >>>affinity, but the thread can only handle stripes produced by cpu 1-3 till the
>>> >>>sysfs entry is changed again.
>>> >>>
>>> >>>If stripes produced by a CPU aren't handled by any auxiliary thread, such
>>> >>>stripes will be handled by raid5d. Otherwise, raid5d doesn't handle any
>>> >>>stripes.
>>> >>>
>>> >> I tested and found two problem(maybe not).
>>> >>
>>> >> 1:print cpulist of auxth, you maybe lost print the '\n'.
>>> >> diff --git a/drivers/md/raid5.c b/drivers/md/raid5.c

(Continue reading)

headers

NeilBrown | 13 Aug 2012 06:21

Re: [patch 2/2 v3]raid5: create multiple threads to handle stripes

On Mon, 13 Aug 2012 10:20:01 +0800 Shaohua Li <shli <at> kernel.org> wrote:

> >> >> 1:print cpulist of auxth, you maybe lost print the '\n'.
> >> >> diff --git a/drivers/md/raid5.c b/drivers/md/raid5.c
> >> >> index 7c8151a..3700cdc 100644
> >> >> --- a/drivers/md/raid5.c
> >> >> +++ b/drivers/md/raid5.c
> >> >>  <at>  <at>  -4911,9 +4911,13  <at>  <at>  struct raid5_auxth_sysfs {
> >> >>  static ssize_t raid5_show_thread_cpulist(struct mddev *mddev,
> >> >>         struct raid5_auxth *thread, char *page)
> >> >>  {
> >> >> +       int n;
> >> >>         if (!mddev->private)
> >> >>                 return 0;
> >> >> -       return cpulist_scnprintf(page, PAGE_SIZE, &thread->work_mask);
> >> >> +       n = cpulist_scnprintf(page, PAGE_SIZE - 2, &thread->work_mask);
> >> >> +       page[n++] = '\n';
> >> >> +       page[n] = 0;
> >> >> +       return n;
> >> >>  }
> >> >>
> >> >>  static ssize_t
> >> >
> >> >some sysfs entries print out '\n', some not, I don't mind add it
> >> I search kernel code found places which like this print out '\n';
> >> Can you tell rule which use or not?
> >> Thanks!
> >
> > I'm not aware any rule about this

(Continue reading)

headers

Jianpeng Ma | 14 Aug 2012 12:39

Re: Re: [patch 2/2 v3]raid5: create multiple threads to handle stripes

On 2012-08-13 10:20 Shaohua Li <shli <at> kernel.org> Wrote:
>2012/8/13 Shaohua Li <shli <at> kernel.org>:
>> On Mon, Aug 13, 2012 at 09:06:45AM +0800, Jianpeng Ma wrote:
>>> On 2012-08-13 08:21 Shaohua Li <shli <at> kernel.org> Wrote:
>>> >2012/8/11 Jianpeng Ma <majianpeng <at> gmail.com>:
>>> >> On 2012-08-09 16:58 Shaohua Li <shli <at> kernel.org> Wrote:
>>> >>>This is a new tempt to make raid5 handle stripes in multiple threads, as
>>> >>>suggested by Neil to have maxium flexibility and better numa binding. It
>>> >>>basically is a combination of my first and second generation patches. By
>>> >>>default, no multiple thread is enabled (all stripes are handled by raid5d).
>>> >>>
>>> >>>An example to enable multiple threads:
>>> >>>#echo 3 > /sys/block/md0/md/auxthread_number
>>> >>>This will create 3 auxiliary threads to handle stripes. The threads can run
>>> >>>on any cpus and handle stripes produced by any cpus.
>>> >>>
>>> >>>#echo 1-3 > /sys/block/md0/md/auxth0/cpulist
>>> >>>This will bind auxiliary thread 0 to cpu 1-3, and this thread will only handle
>>> >>>stripes produced by cpu 1-3. User tool can further change the thread's
>>> >>>affinity, but the thread can only handle stripes produced by cpu 1-3 till the
>>> >>>sysfs entry is changed again.
>>> >>>
>>> >>>If stripes produced by a CPU aren't handled by any auxiliary thread, such
>>> >>>stripes will be handled by raid5d. Otherwise, raid5d doesn't handle any
>>> >>>stripes.
>>> >>>
>>> >> I tested and found two problem(maybe not).
>>> >>
>>> >> 1:print cpulist of auxth, you maybe lost print the '\n'.
>>> >> diff --git a/drivers/md/raid5.c b/drivers/md/raid5.c

(Continue reading)

headers

Shaohua Li | 15 Aug 2012 05:51

Re: Re: [patch 2/2 v3]raid5: create multiple threads to handle stripes

2012/8/14 Jianpeng Ma <majianpeng <at> gmail.com>:
> On 2012-08-13 10:20 Shaohua Li <shli <at> kernel.org> Wrote:
>>2012/8/13 Shaohua Li <shli <at> kernel.org>:
>>> On Mon, Aug 13, 2012 at 09:06:45AM +0800, Jianpeng Ma wrote:
>>>> On 2012-08-13 08:21 Shaohua Li <shli <at> kernel.org> Wrote:
>>>> >2012/8/11 Jianpeng Ma <majianpeng <at> gmail.com>:
>>>> >> On 2012-08-09 16:58 Shaohua Li <shli <at> kernel.org> Wrote:
>>>> >>>This is a new tempt to make raid5 handle stripes in multiple threads, as
>>>> >>>suggested by Neil to have maxium flexibility and better numa binding. It
>>>> >>>basically is a combination of my first and second generation patches. By
>>>> >>>default, no multiple thread is enabled (all stripes are handled by raid5d).
>>>> >>>
>>>> >>>An example to enable multiple threads:
>>>> >>>#echo 3 > /sys/block/md0/md/auxthread_number
>>>> >>>This will create 3 auxiliary threads to handle stripes. The threads can run
>>>> >>>on any cpus and handle stripes produced by any cpus.
>>>> >>>
>>>> >>>#echo 1-3 > /sys/block/md0/md/auxth0/cpulist
>>>> >>>This will bind auxiliary thread 0 to cpu 1-3, and this thread will only handle
>>>> >>>stripes produced by cpu 1-3. User tool can further change the thread's
>>>> >>>affinity, but the thread can only handle stripes produced by cpu 1-3 till the
>>>> >>>sysfs entry is changed again.
>>>> >>>
>>>> >>>If stripes produced by a CPU aren't handled by any auxiliary thread, such
>>>> >>>stripes will be handled by raid5d. Otherwise, raid5d doesn't handle any
>>>> >>>stripes.
>>>> >>>
>>>> >> I tested and found two problem(maybe not).
>>>> >>
>>>> >> 1:print cpulist of auxth, you maybe lost print the '\n'.

(Continue reading)

headers

Jianpeng Ma | 15 Aug 2012 08:21

Re: Re: [patch 2/2 v3]raid5: create multiple threads to handle stripes

On 2012-08-15 11:51 Shaohua Li <shli <at> kernel.org> Wrote:
>2012/8/14 Jianpeng Ma <majianpeng <at> gmail.com>:
>> On 2012-08-13 10:20 Shaohua Li <shli <at> kernel.org> Wrote:
>>>2012/8/13 Shaohua Li <shli <at> kernel.org>:
>>>> On Mon, Aug 13, 2012 at 09:06:45AM +0800, Jianpeng Ma wrote:
>>>>> On 2012-08-13 08:21 Shaohua Li <shli <at> kernel.org> Wrote:
>>>>> >2012/8/11 Jianpeng Ma <majianpeng <at> gmail.com>:
>>>>> >> On 2012-08-09 16:58 Shaohua Li <shli <at> kernel.org> Wrote:
>>>>> >>>This is a new tempt to make raid5 handle stripes in multiple threads, as
>>>>> >>>suggested by Neil to have maxium flexibility and better numa binding. It
>>>>> >>>basically is a combination of my first and second generation patches. By
>>>>> >>>default, no multiple thread is enabled (all stripes are handled by raid5d).
>>>>> >>>
>>>>> >>>An example to enable multiple threads:
>>>>> >>>#echo 3 > /sys/block/md0/md/auxthread_number
>>>>> >>>This will create 3 auxiliary threads to handle stripes. The threads can run
>>>>> >>>on any cpus and handle stripes produced by any cpus.
>>>>> >>>
>>>>> >>>#echo 1-3 > /sys/block/md0/md/auxth0/cpulist
>>>>> >>>This will bind auxiliary thread 0 to cpu 1-3, and this thread will only handle
>>>>> >>>stripes produced by cpu 1-3. User tool can further change the thread's
>>>>> >>>affinity, but the thread can only handle stripes produced by cpu 1-3 till the
>>>>> >>>sysfs entry is changed again.
>>>>> >>>
>>>>> >>>If stripes produced by a CPU aren't handled by any auxiliary thread, such
>>>>> >>>stripes will be handled by raid5d. Otherwise, raid5d doesn't handle any
>>>>> >>>stripes.
>>>>> >>>
>>>>> >> I tested and found two problem(maybe not).
>>>>> >>

(Continue reading)

headers

Shaohua Li | 15 Aug 2012 10:04

Re: Re: [patch 2/2 v3]raid5: create multiple threads to handle stripes

2012/8/15 Jianpeng Ma <majianpeng <at> gmail.com>:
> On 2012-08-15 11:51 Shaohua Li <shli <at> kernel.org> Wrote:
>>2012/8/14 Jianpeng Ma <majianpeng <at> gmail.com>:
>>> On 2012-08-13 10:20 Shaohua Li <shli <at> kernel.org> Wrote:
>>>>2012/8/13 Shaohua Li <shli <at> kernel.org>:
>>>>> On Mon, Aug 13, 2012 at 09:06:45AM +0800, Jianpeng Ma wrote:
>>>>>> On 2012-08-13 08:21 Shaohua Li <shli <at> kernel.org> Wrote:
>>>>>> >2012/8/11 Jianpeng Ma <majianpeng <at> gmail.com>:
>>>>>> >> On 2012-08-09 16:58 Shaohua Li <shli <at> kernel.org> Wrote:
>>>>>> >>>This is a new tempt to make raid5 handle stripes in multiple threads, as
>>>>>> >>>suggested by Neil to have maxium flexibility and better numa binding. It
>>>>>> >>>basically is a combination of my first and second generation patches. By
>>>>>> >>>default, no multiple thread is enabled (all stripes are handled by raid5d).
>>>>>> >>>
>>>>>> >>>An example to enable multiple threads:
>>>>>> >>>#echo 3 > /sys/block/md0/md/auxthread_number
>>>>>> >>>This will create 3 auxiliary threads to handle stripes. The threads can run
>>>>>> >>>on any cpus and handle stripes produced by any cpus.
>>>>>> >>>
>>>>>> >>>#echo 1-3 > /sys/block/md0/md/auxth0/cpulist
>>>>>> >>>This will bind auxiliary thread 0 to cpu 1-3, and this thread will only handle
>>>>>> >>>stripes produced by cpu 1-3. User tool can further change the thread's
>>>>>> >>>affinity, but the thread can only handle stripes produced by cpu 1-3 till the
>>>>>> >>>sysfs entry is changed again.
>>>>>> >>>
>>>>>> >>>If stripes produced by a CPU aren't handled by any auxiliary thread, such
>>>>>> >>>stripes will be handled by raid5d. Otherwise, raid5d doesn't handle any
>>>>>> >>>stripes.
>>>>>> >>>
>>>>>> >> I tested and found two problem(maybe not).

(Continue reading)

headers

Jianpeng Ma | 15 Aug 2012 10:19

Re: Re: [patch 2/2 v3]raid5: create multiple threads to handle stripes

On 2012-08-15 16:04 Shaohua Li <shli <at> kernel.org> Wrote:
>2012/8/15 Jianpeng Ma <majianpeng <at> gmail.com>:
>> On 2012-08-15 11:51 Shaohua Li <shli <at> kernel.org> Wrote:
>>>2012/8/14 Jianpeng Ma <majianpeng <at> gmail.com>:
>>>> On 2012-08-13 10:20 Shaohua Li <shli <at> kernel.org> Wrote:
>>>>>2012/8/13 Shaohua Li <shli <at> kernel.org>:
>>>>>> On Mon, Aug 13, 2012 at 09:06:45AM +0800, Jianpeng Ma wrote:
>>>>>>> On 2012-08-13 08:21 Shaohua Li <shli <at> kernel.org> Wrote:
>>>>>>> >2012/8/11 Jianpeng Ma <majianpeng <at> gmail.com>:
>>>>>>> >> On 2012-08-09 16:58 Shaohua Li <shli <at> kernel.org> Wrote:
>>>>>>> >>>This is a new tempt to make raid5 handle stripes in multiple threads, as
>>>>>>> >>>suggested by Neil to have maxium flexibility and better numa binding. It
>>>>>>> >>>basically is a combination of my first and second generation patches. By
>>>>>>> >>>default, no multiple thread is enabled (all stripes are handled by raid5d).
>>>>>>> >>>
>>>>>>> >>>An example to enable multiple threads:
>>>>>>> >>>#echo 3 > /sys/block/md0/md/auxthread_number
>>>>>>> >>>This will create 3 auxiliary threads to handle stripes. The threads can run
>>>>>>> >>>on any cpus and handle stripes produced by any cpus.
>>>>>>> >>>
>>>>>>> >>>#echo 1-3 > /sys/block/md0/md/auxth0/cpulist
>>>>>>> >>>This will bind auxiliary thread 0 to cpu 1-3, and this thread will only handle
>>>>>>> >>>stripes produced by cpu 1-3. User tool can further change the thread's
>>>>>>> >>>affinity, but the thread can only handle stripes produced by cpu 1-3 till the
>>>>>>> >>>sysfs entry is changed again.
>>>>>>> >>>
>>>>>>> >>>If stripes produced by a CPU aren't handled by any auxiliary thread, such
>>>>>>> >>>stripes will be handled by raid5d. Otherwise, raid5d doesn't handle any
>>>>>>> >>>stripes.
>>>>>>> >>>

(Continue reading)

headers

Jianpeng Ma | 24 Sep 2012 13:15

Re: Re: [patch 2/2 v3]raid5: create multiple threads to handle stripes

Sorry for using this title to send this.
This code maybe error,so it should correct(If multiple-thread dosen't enable,it maybe also error,but no obviously).
In func handle_parity_checks5:
>>conf->mddev->resync_mismatches += STRIPE_SECTORS;
If multi-thread handle stripe on different cpu, there will be raced on resync_mismatches.

So the patch is:

Subject: [PATCH] md:change resync_mismatches to atomic64_t to avoid races

Signed-off-by: Jianpeng Ma <majianpeng <at> gmail.com>
---
 drivers/md/md.c     |    7 ++++---
 drivers/md/md.h     |    2 +-
 drivers/md/raid1.c  |    2 +-
 drivers/md/raid10.c |    2 +-
 drivers/md/raid5.c  |    4 ++--
 5 files changed, 9 insertions(+), 8 deletions(-)

diff --git a/drivers/md/md.c b/drivers/md/md.c
index 308e87b..b0b9474 100644
--- a/drivers/md/md.c
+++ b/drivers/md/md.c
 <at>  <at>  -4278,7 +4278,8  <at>  <at>  static ssize_t
 mismatch_cnt_show(struct mddev *mddev, char *page)
 {
 	return sprintf(page, "%llu\n",
-		       (unsigned long long) mddev->resync_mismatches);
+		       (unsigned long long)
+		       atomic64_read(&mddev->resync_mismatches));

(Continue reading)

headers

NeilBrown | 26 Sep 2012 03:26

Re: [patch 2/2 v3]raid5: create multiple threads to handle stripes

On Mon, 24 Sep 2012 19:15:12 +0800 "Jianpeng Ma" <majianpeng <at> gmail.com> wrote:

> Sorry for using this title to send this.

It isn't hard to change the Subject line for the Email, is it?

> This code maybe error,so it should correct(If multiple-thread dosen't enable,it maybe also error,but
no obviously).
> In func handle_parity_checks5:
> >>conf->mddev->resync_mismatches += STRIPE_SECTORS;
> If multi-thread handle stripe on different cpu, there will be raced on resync_mismatches.
> 
> So the patch is:
> 
> Subject: [PATCH] md:change resync_mismatches to atomic64_t to avoid races

I doubt this is really necessary, as resync_mismatches doesn't need to be
precise.
But it doesn't really hurt either, so I've applied it.

Thanks,
NeilBrown

> 
> Signed-off-by: Jianpeng Ma <majianpeng <at> gmail.com>
> ---
>  drivers/md/md.c     |    7 ++++---
>  drivers/md/md.h     |    2 +-
>  drivers/md/raid1.c  |    2 +-
>  drivers/md/raid10.c |    2 +-

(Continue reading)

headers

Jianpeng Ma | 13 Aug 2012 11:11

Re: Re: [patch 2/2 v3]raid5: create multiple threads to handle stripes

On 2012-08-13 08:21 Shaohua Li <shli <at> kernel.org> Wrote:
>2012/8/11 Jianpeng Ma <majianpeng <at> gmail.com>:
>> On 2012-08-09 16:58 Shaohua Li <shli <at> kernel.org> Wrote:
>>>This is a new tempt to make raid5 handle stripes in multiple threads, as
>>>suggested by Neil to have maxium flexibility and better numa binding. It
[snip]
>> 2: Test 'dd if=/dev/zero of=/dev/md0 bs=2M ', the performance regress remarkable.
>> auxthread_number=0, 200MB/s;
>> auxthread_number=4, 95MB/s.
>
>So multiple threads handle stripes reduce request merge. In your
>workload, raid5d isn't a bottleneck at all. In practice, I thought only
>array which can drive high IOPS needs enable multi thread. And
>if you create multiple threads, better let the threads handle different
>cpus.
If dd write using buffer-mode, the make-rquest is exec by kernel-thread "flush-n:0".
So we only used multiple thread to write by odirect or sync to achieve the result which we wanted.
>
>Thanks,
>Shaohua

headers

NeilBrown | 13 Aug 2012 06:29

Re: [patch 2/2 v3]raid5: create multiple threads to handle stripes

On Thu, 9 Aug 2012 16:58:08 +0800 Shaohua Li <shli <at> kernel.org> wrote:

> This is a new tempt to make raid5 handle stripes in multiple threads, as
> suggested by Neil to have maxium flexibility and better numa binding. It
> basically is a combination of my first and second generation patches. By
> default, no multiple thread is enabled (all stripes are handled by raid5d).
> 
> An example to enable multiple threads:
> #echo 3 > /sys/block/md0/md/auxthread_number
> This will create 3 auxiliary threads to handle stripes. The threads can run
> on any cpus and handle stripes produced by any cpus.
> 
> #echo 1-3 > /sys/block/md0/md/auxth0/cpulist
> This will bind auxiliary thread 0 to cpu 1-3, and this thread will only handle
> stripes produced by cpu 1-3. User tool can further change the thread's
> affinity, but the thread can only handle stripes produced by cpu 1-3 till the
> sysfs entry is changed again.
> 
> If stripes produced by a CPU aren't handled by any auxiliary thread, such
> stripes will be handled by raid5d. Otherwise, raid5d doesn't handle any
> stripes.
> 
> Signed-off-by: Shaohua Li <shli <at> fusionio.com>
> ---
>  drivers/md/md.c    |    8 -
>  drivers/md/md.h    |    8 +
>  drivers/md/raid5.c |  406 ++++++++++++++++++++++++++++++++++++++++++++++++++---
>  drivers/md/raid5.h |   19 ++
>  4 files changed, 418 insertions(+), 23 deletions(-)
>

(Continue reading)

headers

Shaohua Li | 13 Aug 2012 08:22

Re: [patch 2/2 v3]raid5: create multiple threads to handle stripes

On Mon, Aug 13, 2012 at 02:29:47PM +1000, NeilBrown wrote:
> On Thu, 9 Aug 2012 16:58:08 +0800 Shaohua Li <shli <at> kernel.org> wrote:
> 
> > This is a new tempt to make raid5 handle stripes in multiple threads, as
> > suggested by Neil to have maxium flexibility and better numa binding. It
> > basically is a combination of my first and second generation patches. By
> > default, no multiple thread is enabled (all stripes are handled by raid5d).
> > 
> > An example to enable multiple threads:
> > #echo 3 > /sys/block/md0/md/auxthread_number
> > This will create 3 auxiliary threads to handle stripes. The threads can run
> > on any cpus and handle stripes produced by any cpus.
> > 
> > #echo 1-3 > /sys/block/md0/md/auxth0/cpulist
> > This will bind auxiliary thread 0 to cpu 1-3, and this thread will only handle
> > stripes produced by cpu 1-3. User tool can further change the thread's
> > affinity, but the thread can only handle stripes produced by cpu 1-3 till the
> > sysfs entry is changed again.
> > 
> > If stripes produced by a CPU aren't handled by any auxiliary thread, such
> > stripes will be handled by raid5d. Otherwise, raid5d doesn't handle any
> > stripes.
> > 
> > Signed-off-by: Shaohua Li <shli <at> fusionio.com>
> > ---
> >  drivers/md/md.c    |    8 -
> >  drivers/md/md.h    |    8 +
> >  drivers/md/raid5.c |  406 ++++++++++++++++++++++++++++++++++++++++++++++++++---
> >  drivers/md/raid5.h |   19 ++
> >  4 files changed, 418 insertions(+), 23 deletions(-)

(Continue reading)

headers

Shaohua Li | 7 Mar 2013 08:31

Re: [patch 2/2 v3]raid5: create multiple threads to handle stripes

Hi Neil,

I just noticed this one is missed. It used to be in your branch, but get
dropped some time. Is anything I missed? I can resend it if you like.

Thanks,
Shaohua

On Thu, Aug 09, 2012 at 04:58:08PM +0800, Shaohua Li wrote:
> This is a new tempt to make raid5 handle stripes in multiple threads, as
> suggested by Neil to have maxium flexibility and better numa binding. It
> basically is a combination of my first and second generation patches. By
> default, no multiple thread is enabled (all stripes are handled by raid5d).
> 
> An example to enable multiple threads:
> #echo 3 > /sys/block/md0/md/auxthread_number
> This will create 3 auxiliary threads to handle stripes. The threads can run
> on any cpus and handle stripes produced by any cpus.
> 
> #echo 1-3 > /sys/block/md0/md/auxth0/cpulist
> This will bind auxiliary thread 0 to cpu 1-3, and this thread will only handle
> stripes produced by cpu 1-3. User tool can further change the thread's
> affinity, but the thread can only handle stripes produced by cpu 1-3 till the
> sysfs entry is changed again.
> 
> If stripes produced by a CPU aren't handled by any auxiliary thread, such
> stripes will be handled by raid5d. Otherwise, raid5d doesn't handle any
> stripes.
> 
> Signed-off-by: Shaohua Li <shli <at> fusionio.com>

(Continue reading)

headers

NeilBrown | 12 Mar 2013 02:39

Re: [patch 2/2 v3]raid5: create multiple threads to handle stripes

On Thu, 7 Mar 2013 15:31:23 +0800 Shaohua Li <shli <at> kernel.org> wrote:

> Hi Neil,
> 
> I just noticed this one is missed. It used to be in your branch, but get
> dropped some time. Is anything I missed? I can resend it if you like.
> 
> Thanks,
> Shaohua
> 
> On Thu, Aug 09, 2012 at 04:58:08PM +0800, Shaohua Li wrote:
> > This is a new tempt to make raid5 handle stripes in multiple threads, as
> > suggested by Neil to have maxium flexibility and better numa binding. It
> > basically is a combination of my first and second generation patches. By
> > default, no multiple thread is enabled (all stripes are handled by raid5d).
> > 
> > An example to enable multiple threads:
> > #echo 3 > /sys/block/md0/md/auxthread_number
> > This will create 3 auxiliary threads to handle stripes. The threads can run
> > on any cpus and handle stripes produced by any cpus.
> > 
> > #echo 1-3 > /sys/block/md0/md/auxth0/cpulist
> > This will bind auxiliary thread 0 to cpu 1-3, and this thread will only handle
> > stripes produced by cpu 1-3. User tool can further change the thread's
> > affinity, but the thread can only handle stripes produced by cpu 1-3 till the
> > sysfs entry is changed again.
> > 
> > If stripes produced by a CPU aren't handled by any auxiliary thread, such
> > stripes will be handled by raid5d. Otherwise, raid5d doesn't handle any
> > stripes.

(Continue reading)

headers

Stan Hoeppner | 13 Mar 2013 01:44

Re: [patch 2/2 v3]raid5: create multiple threads to handle stripes

On 3/11/2013 8:39 PM, NeilBrown wrote:
> On Thu, 7 Mar 2013 15:31:23 +0800 Shaohua Li <shli <at> kernel.org> wrote:
...
>>> #echo 1-3 > /sys/block/md0/md/auxth0/cpulist
>>> This will bind auxiliary thread 0 to cpu 1-3, and this thread will only handle
>>> stripes produced by cpu 1-3. User tool can further change the thread's
>>> affinity, but the thread can only handle stripes produced by cpu 1-3 till the
>>> sysfs entry is changed again.

Would it not be better to use the existing cpusets infrastructure for
this, instead of manually binding threads to specific cores or sets of
cores?

Also, I understand the hot cache issue driving the desire to have a raid
thread only process stripes created by its CPU.  But what of the
scenario where an HPC user pins application threads to cores and needs
all the L1/L2 cache?  Say this user has a dual socket 24 core NUMA
system with 2 NUMA nodes per socket, 4 nodes total.  Each NUMA node has
6 cores and shared L3 cache.  The user pins 5 processes to 5 cores in
each node, and wants to pin a raid thread to the remaining core in each
node to handle the write IO generated by the 5 user threads on the node.

Does your patch series allow this?  Using the above example, if the user
creates 4 cpusets, can he assign a raid thread to that set and the
thread will execute on any core in the set, and only that set, on any
stripes created by any CPU in that set, and only that set?

The infrastructure for this already exists, has since 2004.  And it
seems is more flexible than what you've implemented here.  I suggest we
make use of it, as it is the kernel standard for doing such things.

(Continue reading)

headers

NeilBrown | 28 Mar 2013 07:47

Re: [patch 2/2 v3]raid5: create multiple threads to handle stripes

On Tue, 12 Mar 2013 19:44:19 -0500 Stan Hoeppner <stan <at> hardwarefreak.com>
wrote:

> On 3/11/2013 8:39 PM, NeilBrown wrote:
> > On Thu, 7 Mar 2013 15:31:23 +0800 Shaohua Li <shli <at> kernel.org> wrote:
> ...
> >>> #echo 1-3 > /sys/block/md0/md/auxth0/cpulist
> >>> This will bind auxiliary thread 0 to cpu 1-3, and this thread will only handle
> >>> stripes produced by cpu 1-3. User tool can further change the thread's
> >>> affinity, but the thread can only handle stripes produced by cpu 1-3 till the
> >>> sysfs entry is changed again.
> 
> Would it not be better to use the existing cpusets infrastructure for
> this, instead of manually binding threads to specific cores or sets of
> cores?
> 
> Also, I understand the hot cache issue driving the desire to have a raid
> thread only process stripes created by its CPU.  But what of the
> scenario where an HPC user pins application threads to cores and needs
> all the L1/L2 cache?  Say this user has a dual socket 24 core NUMA
> system with 2 NUMA nodes per socket, 4 nodes total.  Each NUMA node has
> 6 cores and shared L3 cache.  The user pins 5 processes to 5 cores in
> each node, and wants to pin a raid thread to the remaining core in each
> node to handle the write IO generated by the 5 user threads on the node.
> 
> Does your patch series allow this?  Using the above example, if the user
> creates 4 cpusets, can he assign a raid thread to that set and the
> thread will execute on any core in the set, and only that set, on any
> stripes created by any CPU in that set, and only that set?
>

(Continue reading)

headers

Stan Hoeppner | 28 Mar 2013 17:53

Re: [patch 2/2 v3]raid5: create multiple threads to handle stripes

I'm CC'ing Christoph and Dave hoping they may have some insights or
recommendations WRT this md RAID kernel write thread scheduling, and if
use cpusets might be handy or worth considering.

On 3/28/2013 1:47 AM, NeilBrown wrote:
> On Tue, 12 Mar 2013 19:44:19 -0500 Stan Hoeppner <stan <at> hardwarefreak.com>
> wrote:
> 
>> On 3/11/2013 8:39 PM, NeilBrown wrote:
>>> On Thu, 7 Mar 2013 15:31:23 +0800 Shaohua Li <shli <at> kernel.org> wrote:
>> ...
>>>>> #echo 1-3 > /sys/block/md0/md/auxth0/cpulist
>>>>> This will bind auxiliary thread 0 to cpu 1-3, and this thread will only handle
>>>>> stripes produced by cpu 1-3. User tool can further change the thread's
>>>>> affinity, but the thread can only handle stripes produced by cpu 1-3 till the
>>>>> sysfs entry is changed again.
>>
>> Would it not be better to use the existing cpusets infrastructure for
>> this, instead of manually binding threads to specific cores or sets of
>> cores?
>>
>> Also, I understand the hot cache issue driving the desire to have a raid
>> thread only process stripes created by its CPU.  But what of the
>> scenario where an HPC user pins application threads to cores and needs
>> all the L1/L2 cache?  Say this user has a dual socket 24 core NUMA
>> system with 2 NUMA nodes per socket, 4 nodes total.  Each NUMA node has
>> 6 cores and shared L3 cache.  The user pins 5 processes to 5 cores in
>> each node, and wants to pin a raid thread to the remaining core in each
>> node to handle the write IO generated by the 5 user threads on the node.
>>

(Continue reading)

headers

Shaohua Li | 29 Mar 2013 03:34

Re: [patch 2/2 v3]raid5: create multiple threads to handle stripes

On Thu, Mar 28, 2013 at 05:47:44PM +1100, NeilBrown wrote:
> On Tue, 12 Mar 2013 19:44:19 -0500 Stan Hoeppner <stan <at> hardwarefreak.com>
> wrote:
> 
> > On 3/11/2013 8:39 PM, NeilBrown wrote:
> > > On Thu, 7 Mar 2013 15:31:23 +0800 Shaohua Li <shli <at> kernel.org> wrote:
> > ...
> > >>> #echo 1-3 > /sys/block/md0/md/auxth0/cpulist
> > >>> This will bind auxiliary thread 0 to cpu 1-3, and this thread will only handle
> > >>> stripes produced by cpu 1-3. User tool can further change the thread's
> > >>> affinity, but the thread can only handle stripes produced by cpu 1-3 till the
> > >>> sysfs entry is changed again.
> > 
> > Would it not be better to use the existing cpusets infrastructure for
> > this, instead of manually binding threads to specific cores or sets of
> > cores?
> > 
> > Also, I understand the hot cache issue driving the desire to have a raid
> > thread only process stripes created by its CPU.  But what of the
> > scenario where an HPC user pins application threads to cores and needs
> > all the L1/L2 cache?  Say this user has a dual socket 24 core NUMA
> > system with 2 NUMA nodes per socket, 4 nodes total.  Each NUMA node has
> > 6 cores and shared L3 cache.  The user pins 5 processes to 5 cores in
> > each node, and wants to pin a raid thread to the remaining core in each
> > node to handle the write IO generated by the 5 user threads on the node.
> > 
> > Does your patch series allow this?  Using the above example, if the user
> > creates 4 cpusets, can he assign a raid thread to that set and the
> > thread will execute on any core in the set, and only that set, on any
> > stripes created by any CPU in that set, and only that set?

(Continue reading)

headers

Stan Hoeppner | 29 Mar 2013 10:36

Re: [patch 2/2 v3]raid5: create multiple threads to handle stripes

I'm CC'ing Joe Landman as he's already building systems of the caliber
that would benefit from this write threading and may need configurable
CPU scheduling.  Joe I've not seen a post from you on linux-raid in a
while so I don't know if you've been following this topic.  Shaohua has
created patch sets to eliminate, or dramatically mitigate, the horrible
single threaded write performance of md/RAID 1, 10, 5, 6 on SSD.
Throughput no longer hits a wall due to peaking one core, as with the
currently shipping kernel code.  Your thoughts?

On 3/28/2013 9:34 PM, Shaohua Li wrote:
...
> Frankly I don't like the cpuset way. It might just work, but it's just another
> API to control process affinity and has no essential difference against my
> approach (which directly sets process affinity). Generally we use cpuset
> instead of process affinity is because of something like inherit affinity.
> While the raid5 process doesn't involve those.

First I should again state I'm not a developer, but a sysadmin, and this
is the viewpoint from which I speak.

The essential difference I see is the user interface the sysadmin will
employ to tweak thread placement/behavior.  Hypothetically, say I have a
64 socket Altix UV machine w/8 core CPUs, 512 cores.  Each node board
has two sockets, two distinct NUMA nodes, 64 total, but these share a
NUMALink hub interface chip connection to the rest of the machine, and
share a PCIe mezzanine interface.

We obviously want to keep md/RAID housekeeping bandwidth (stripe cache,
RMW reads, etc) isolated to the node where it is attached so it doesn't
needlessly traverse NUMALink eating precious, limited, 'high' latency

(Continue reading)

headers

Shaohua Li | 1 Apr 2013 03:57

Re: [patch 2/2 v3]raid5: create multiple threads to handle stripes

On Fri, Mar 29, 2013 at 04:36:14AM -0500, Stan Hoeppner wrote:
> I'm CC'ing Joe Landman as he's already building systems of the caliber
> that would benefit from this write threading and may need configurable
> CPU scheduling.  Joe I've not seen a post from you on linux-raid in a
> while so I don't know if you've been following this topic.  Shaohua has
> created patch sets to eliminate, or dramatically mitigate, the horrible
> single threaded write performance of md/RAID 1, 10, 5, 6 on SSD.
> Throughput no longer hits a wall due to peaking one core, as with the
> currently shipping kernel code.  Your thoughts?
> 
> On 3/28/2013 9:34 PM, Shaohua Li wrote:
> ...
> > Frankly I don't like the cpuset way. It might just work, but it's just another
> > API to control process affinity and has no essential difference against my
> > approach (which directly sets process affinity). Generally we use cpuset
> > instead of process affinity is because of something like inherit affinity.
> > While the raid5 process doesn't involve those.
> 
> First I should again state I'm not a developer, but a sysadmin, and this
> is the viewpoint from which I speak.
> 
> The essential difference I see is the user interface the sysadmin will
> employ to tweak thread placement/behavior.  Hypothetically, say I have a
> 64 socket Altix UV machine w/8 core CPUs, 512 cores.  Each node board
> has two sockets, two distinct NUMA nodes, 64 total, but these share a
> NUMALink hub interface chip connection to the rest of the machine, and
> share a PCIe mezzanine interface.
> 
> We obviously want to keep md/RAID housekeeping bandwidth (stripe cache,
> RMW reads, etc) isolated to the node where it is attached so it doesn't

(Continue reading)

headers

Stan Hoeppner | 1 Apr 2013 21:31

Re: [patch 2/2 v3]raid5: create multiple threads to handle stripes

On 3/31/2013 8:57 PM, Shaohua Li wrote:
> On Fri, Mar 29, 2013 at 04:36:14AM -0500, Stan Hoeppner wrote:
>> I'm CC'ing Joe Landman as he's already building systems of the caliber
>> that would benefit from this write threading and may need configurable
>> CPU scheduling.  Joe I've not seen a post from you on linux-raid in a
>> while so I don't know if you've been following this topic.  Shaohua has
>> created patch sets to eliminate, or dramatically mitigate, the horrible
>> single threaded write performance of md/RAID 1, 10, 5, 6 on SSD.
>> Throughput no longer hits a wall due to peaking one core, as with the
>> currently shipping kernel code.  Your thoughts?
>>
>> On 3/28/2013 9:34 PM, Shaohua Li wrote:
>> ...
>>> Frankly I don't like the cpuset way. It might just work, but it's just another
>>> API to control process affinity and has no essential difference against my
>>> approach (which directly sets process affinity). Generally we use cpuset
>>> instead of process affinity is because of something like inherit affinity.
>>> While the raid5 process doesn't involve those.
>>
>> First I should again state I'm not a developer, but a sysadmin, and this
>> is the viewpoint from which I speak.
>>
>> The essential difference I see is the user interface the sysadmin will
>> employ to tweak thread placement/behavior.  Hypothetically, say I have a
>> 64 socket Altix UV machine w/8 core CPUs, 512 cores.  Each node board
>> has two sockets, two distinct NUMA nodes, 64 total, but these share a
>> NUMALink hub interface chip connection to the rest of the machine, and
>> share a PCIe mezzanine interface.
>>
>> We obviously want to keep md/RAID housekeeping bandwidth (stripe cache,

(Continue reading)

headers

Shaohua Li | 2 Apr 2013 02:39

Re: [patch 2/2 v3]raid5: create multiple threads to handle stripes

On Mon, Apr 01, 2013 at 02:31:22PM -0500, Stan Hoeppner wrote:
> On 3/31/2013 8:57 PM, Shaohua Li wrote:
> > On Fri, Mar 29, 2013 at 04:36:14AM -0500, Stan Hoeppner wrote:
> >> I'm CC'ing Joe Landman as he's already building systems of the caliber
> >> that would benefit from this write threading and may need configurable
> >> CPU scheduling.  Joe I've not seen a post from you on linux-raid in a
> >> while so I don't know if you've been following this topic.  Shaohua has
> >> created patch sets to eliminate, or dramatically mitigate, the horrible
> >> single threaded write performance of md/RAID 1, 10, 5, 6 on SSD.
> >> Throughput no longer hits a wall due to peaking one core, as with the
> >> currently shipping kernel code.  Your thoughts?
> >>
> >> On 3/28/2013 9:34 PM, Shaohua Li wrote:
> >> ...
> >>> Frankly I don't like the cpuset way. It might just work, but it's just another
> >>> API to control process affinity and has no essential difference against my
> >>> approach (which directly sets process affinity). Generally we use cpuset
> >>> instead of process affinity is because of something like inherit affinity.
> >>> While the raid5 process doesn't involve those.
> >>
> >> First I should again state I'm not a developer, but a sysadmin, and this
> >> is the viewpoint from which I speak.
> >>
> >> The essential difference I see is the user interface the sysadmin will
> >> employ to tweak thread placement/behavior.  Hypothetically, say I have a
> >> 64 socket Altix UV machine w/8 core CPUs, 512 cores.  Each node board
> >> has two sockets, two distinct NUMA nodes, 64 total, but these share a
> >> NUMALink hub interface chip connection to the rest of the machine, and
> >> share a PCIe mezzanine interface.
> >>

(Continue reading)

headers

Stan Hoeppner | 2 Apr 2013 05:12

Re: [patch 2/2 v3]raid5: create multiple threads to handle stripes

On 4/1/2013 7:39 PM, Shaohua Li wrote:
> On Mon, Apr 01, 2013 at 02:31:22PM -0500, Stan Hoeppner wrote:
>> On 3/31/2013 8:57 PM, Shaohua Li wrote:
>>> On Fri, Mar 29, 2013 at 04:36:14AM -0500, Stan Hoeppner wrote:
>>>> I'm CC'ing Joe Landman as he's already building systems of the caliber
>>>> that would benefit from this write threading and may need configurable
>>>> CPU scheduling.  Joe I've not seen a post from you on linux-raid in a
>>>> while so I don't know if you've been following this topic.  Shaohua has
>>>> created patch sets to eliminate, or dramatically mitigate, the horrible
>>>> single threaded write performance of md/RAID 1, 10, 5, 6 on SSD.
>>>> Throughput no longer hits a wall due to peaking one core, as with the
>>>> currently shipping kernel code.  Your thoughts?
>>>>
>>>> On 3/28/2013 9:34 PM, Shaohua Li wrote:
>>>> ...
>>>>> Frankly I don't like the cpuset way. It might just work, but it's just another
>>>>> API to control process affinity and has no essential difference against my
>>>>> approach (which directly sets process affinity). Generally we use cpuset
>>>>> instead of process affinity is because of something like inherit affinity.
>>>>> While the raid5 process doesn't involve those.
>>>>
>>>> First I should again state I'm not a developer, but a sysadmin, and this
>>>> is the viewpoint from which I speak.
>>>>
>>>> The essential difference I see is the user interface the sysadmin will
>>>> employ to tweak thread placement/behavior.  Hypothetically, say I have a
>>>> 64 socket Altix UV machine w/8 core CPUs, 512 cores.  Each node board
>>>> has two sockets, two distinct NUMA nodes, 64 total, but these share a
>>>> NUMALink hub interface chip connection to the rest of the machine, and
>>>> share a PCIe mezzanine interface.

(Continue reading)