Development around kexec ()

Atsushi Kumagai | 29 Jun 2012 04:13

[RFC PATCH v2 0/10] makedumpfile: cyclic processing to keep memory consumption.

Hello,

I improved prototype of cyclic processing as version 2.
If there is no objection to basic idea, I want to consider the things
related to performance as next step. (Concretely, buffer size and the patch set
HATAYAMA-san sent a short time ago.) 

Version 1:

  http://lists.infradead.org/pipermail/kexec/2012-May/006363.html

Introduction:

 - The purpose of cyclic processing is to fix memory consumption.
 - cyclic processing doesn't use temporary bitmap file, store partial bitmap data
   in memory only for each cycle instead.
 - The prototype was passed the regression test which is used by every release.

How to use:

  Specify '--cyclic' option, then makedumpfile works cyclically.

Example:

    $ makedumpfile --cyclic -cd31 vmcore testdump.cd31
    Copying data                       : [  5 %]
    Excluding free pages               : [100 %]
    ...
    Excluding free pages               : [100 %]
    Copying data                       : [100 %]

(Continue reading)

headers

Atsushi Kumagai | 29 Jun 2012 04:16

[RFC PATCH v2 1/10] Add flag to enable cyclic processing.

Introduce --cyclic option to enable cyclic processing. If --cyclic option is
specified, then makedumpfile works cyclically and the memory usage will be constant.

Usage:
    # makedumpfile --cyclic /proc/vmcore dumpfile

Signed-off-by: Atsushi Kumagai <kumagai-atsushi@...>
---
 makedumpfile.8 |   12 ++++++++++++
 makedumpfile.c |    4 ++++
 makedumpfile.h |    1 +
 print_info.c   |    9 +++++++++
 4 files changed, 26 insertions(+)

diff --git a/makedumpfile.8 b/makedumpfile.8
index 1fb9756..6ab2599 100644
--- a/makedumpfile.8
+++ b/makedumpfile.8
 <at>  <at>  -346,6 +346,18  <at>  <at>  on the following example.
 .br
 # makedumpfile \-\-reassemble dumpfile1 dumpfile2 dumpfile

+
+.TP
+\fB\-\-cyclic\fR
+Creating bitmaps and writing pages cyclically for constant target memory range.
+As a result, makedumpfile can works in constant memory space regardless of 
+system memory size.
+This feature is \fITESTING VERSION\fR now.
+.br

(Continue reading)

headers

Atsushi Kumagai | 29 Jun 2012 04:17

[RFC PATCH v2 2/10] Prepare partial bitmap for cyclic processing.

cyclic processing uses partial bitmap instead of temporary bitmap file.
partial bitmap is saved in memory only for each cycle.

This patch introduce partial bitmap and extend some accessor functions
to manage partial bitmap.

Signed-off-by: Atsushi Kumagai <kumagai-atsushi@...>
---
 makedumpfile.c |   99 ++++++++++++++++++++++++++++++++++++++++++++++++++------
 makedumpfile.h |   23 +++++++++++++
 2 files changed, 113 insertions(+), 9 deletions(-)

diff --git a/makedumpfile.c b/makedumpfile.c
index 87bd680..9e77913 100644
--- a/makedumpfile.c
+++ b/makedumpfile.c
 <at>  <at>  -165,6 +165,7  <at>  <at>  is_in_same_page(unsigned long vaddr1, unsigned long vaddr2)

 #define BITMAP_SECT_LEN 4096
 static inline int is_dumpable(struct dump_bitmap *, unsigned long long);
+static inline int is_dumpable_cyclic(char *bitmap, unsigned long long);
 unsigned long
 pfn_to_pos(unsigned long long pfn)
 {
 <at>  <at>  -2719,6 +2720,12  <at>  <at>  initialize_bitmap(struct dump_bitmap *bitmap)
 }

 void
+initialize_bitmap_cyclic(char *bitmap)
+{

(Continue reading)

headers

Atsushi Kumagai | 29 Jun 2012 04:17

[RFC PATCH v2 3/10] Change the function related to excluding unnecessary pages.

Extend the function related to excluding unnecessary pages.
The function creates partial bitmap corresponding to range of target region
when cyclic flag is on.

Signed-off-by: Atsushi Kumagai <kumagai-atsushi@...>
---
 makedumpfile.c |   86 ++++++++++++++++++++++++++++++++++++++++++++++++--------
 1 file changed, 75 insertions(+), 11 deletions(-)

diff --git a/makedumpfile.c b/makedumpfile.c
index 9e77913..e7c9f5e 100644
--- a/makedumpfile.c
+++ b/makedumpfile.c
 <at>  <at>  -3264,9 +3264,9  <at>  <at>  reset_bitmap_of_free_pages(unsigned long node_zones)
 				}
 				for (i = 0; i < (1<<order); i++) {
 					pfn = start_pfn + i;
-					clear_bit_on_2nd_bitmap_for_kernel(pfn);
+					if (clear_bit_on_2nd_bitmap_for_kernel(pfn))
+						found_free_pages++;
 				}
-				found_free_pages += i;

 				previous = curr;
 				if (!readmem(VADDR, curr+OFFSET(list_head.next),
 <at>  <at>  -3300,7 +3300,7  <at>  <at>  reset_bitmap_of_free_pages(unsigned long node_zones)
 		ERRMSG("Can't get free_pages.\n");
 		return FALSE;
 	}
-	if (free_pages != found_free_pages) {

(Continue reading)

headers

Atsushi Kumagai | 29 Jun 2012 04:18

[RFC PATCH v2 4/10] Add function to update target region.

update_cyclic_region() updates target region with recalculating partial bitmap
when traverse target region.
This function helps cyclic processing to update partial bitmap for target region.

Signed-off-by: Atsushi Kumagai <kumagai-atsushi@...>
---
 makedumpfile.c |   21 +++++++++++++++++++++
 makedumpfile.h |    9 +++++++++
 2 files changed, 30 insertions(+)

diff --git a/makedumpfile.c b/makedumpfile.c
index e7c9f5e..67b3499 100644
--- a/makedumpfile.c
+++ b/makedumpfile.c
 <at>  <at>  -3878,6 +3878,27  <at>  <at>  exclude_unnecessary_pages_cyclic(void)
 }

 int
+update_cyclic_region(unsigned long long pfn)
+{
+	if (is_cyclic_region(pfn))
+		return TRUE;
+
+	info->cyclic_start_pfn = round(pfn, PFN_CYCLIC);
+	info->cyclic_end_pfn = info->cyclic_start_pfn + PFN_CYCLIC;
+
+	if (info->cyclic_end_pfn > info->max_mapnr)
+		info->cyclic_end_pfn = info->max_mapnr;
+
+	if (!create_1st_bitmap_cyclic())

(Continue reading)

headers

Atsushi Kumagai | 29 Jun 2012 04:19

[RFC PATCH v2 5/10] Add function to get num_dumpable for cyclic processing.

get_num_dumpable_cyclic() gets the number of dumpable pages with cyclic processing.
This function is used to get info->num_dumpable and it is necessary to decide
the offset of kdump-compressed page data.

Signed-off-by: Atsushi Kumagai <kumagai-atsushi@...>
---
 makedumpfile.c |   17 +++++++++++++++++
 makedumpfile.h |    6 ++++++
 2 files changed, 23 insertions(+)

diff --git a/makedumpfile.c b/makedumpfile.c
index 67b3499..30857e3 100644
--- a/makedumpfile.c
+++ b/makedumpfile.c
 <at>  <at>  -4094,6 +4094,8  <at>  <at>  create_dump_bitmap(void)
 	if (info->flag_cyclic) {
 		if (!prepare_bitmap_buffer_cyclic())
 			goto out;
+
+		info->num_dumpable = get_num_dumpable_cyclic();
 	} else {
 		if (!prepare_bitmap_buffer())
 			goto out;
 <at>  <at>  -4555,6 +4557,21  <at>  <at>  get_num_dumpable(void)
 	return num_dumpable;
 }

+unsigned long long
+get_num_dumpable_cyclic(void)
+{

(Continue reading)

headers

Atsushi Kumagai | 29 Jun 2012 04:21

[RFC PATCH v2 6/10] Implement the main routine of cyclic processing for kdump-compressed format.

Implement the function which write out kdump-compressed dumpfile cyclically.
The function is similar to current write_kdump_XXX(), but not use temporary bitmap file.
Instead, use partial bitmap with updating it each cycle by cycle.

As result, makedumpfile can work with constant memory consumption even in
large memory system.

Signed-off-by: Atsushi Kumagai <kumagai-atsushi@...>
---
 makedumpfile.c |  250 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 250 insertions(+)

diff --git a/makedumpfile.c b/makedumpfile.c
index 30857e3..25d857a 100644
--- a/makedumpfile.c
+++ b/makedumpfile.c
 <at>  <at>  -5061,6 +5061,154  <at>  <at>  out:
 	return ret;
 }

+int
+write_kdump_pages_cyclic(struct cache_data *cd_header, struct cache_data *cd_page,
+			 struct page_desc *pd_zero, off_t *offset_data)
+{
+	unsigned long long pfn, per;
+	unsigned long long start_pfn, end_pfn;
+	unsigned long size_out;
+	struct page_desc pd;
+	unsigned char buf[info->page_size], *buf_out = NULL;
+	unsigned long len_buf_out;

(Continue reading)

headers

Atsushi Kumagai | 29 Jun 2012 04:22

[RFC PATCH v2 7/10] Add function to get number of PT_LOAD for cyclic processing.

get_loads_dumpfile_cyclic() gets the final number of PT_LOAD with cyclic processing.
This function is necessary to decide the offset of PT_LOAD for writing ELF format.

Signed-off-by: Atsushi Kumagai <kumagai-atsushi@...>
---
 makedumpfile.c |   84 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 makedumpfile.h |    2 ++
 2 files changed, 86 insertions(+)

diff --git a/makedumpfile.c b/makedumpfile.c
index 25d857a..420f103 100644
--- a/makedumpfile.c
+++ b/makedumpfile.c
 <at>  <at>  -4875,6 +4875,90  <at>  <at>  read_pfn(unsigned long long pfn, unsigned char *buf)
 }

 int
+get_loads_dumpfile_cyclic(void)
+{
+	int i, phnum, num_new_load = 0;
+	long page_size = info->page_size;
+	unsigned char buf[info->page_size];
+	unsigned long long pfn, pfn_start, pfn_end, num_excluded;
+	unsigned long frac_head, frac_tail;
+	Elf64_Phdr load;
+
+	/*
+	 * Initialize target reggion and bitmap.
+	 */
+	info->cyclic_start_pfn = 0;

(Continue reading)

headers

Atsushi Kumagai | 29 Jun 2012 04:23

[RFC PATCH v2 8/10] Implement the main routine of cyclic processing for ELF format.

Implement the function which write out ELF dumpfile cyclically.
The basic idea is same as the routine for kdump-compressed format.

Signed-off-by: Atsushi Kumagai <kumagai-atsushi@...>
---
 makedumpfile.c |  243 ++++++++++++++++++++++++++++++++++++++++++++++++++++++--
 1 file changed, 238 insertions(+), 5 deletions(-)

diff --git a/makedumpfile.c b/makedumpfile.c
index 420f103..5670bcf 100644
--- a/makedumpfile.c
+++ b/makedumpfile.c
 <at>  <at>  -4290,9 +4290,16  <at>  <at>  write_elf_header(struct cache_data *cd_header)
 	/*
 	 * Get the PT_LOAD number of the dumpfile.
 	 */
-	if (!(num_loads_dumpfile = get_loads_dumpfile())) {
-		ERRMSG("Can't get a number of PT_LOAD.\n");
-		goto out;
+	if (info->flag_cyclic) {
+		if (!(num_loads_dumpfile = get_loads_dumpfile_cyclic())) {
+			ERRMSG("Can't get a number of PT_LOAD.\n");
+			goto out;
+		}
+	} else {
+		if (!(num_loads_dumpfile = get_loads_dumpfile())) {
+			ERRMSG("Can't get a number of PT_LOAD.\n");
+			goto out;
+		}
 	}

(Continue reading)

headers

Atsushi Kumagai | 29 Jun 2012 04:24

[RFC PATCH v2 9/10] Enabling --split option with cyclic processing.

This patch enable cyclic processing to split the dump data to multiple dumpfiles.
At this time, the buffer of partial bitmap is prepared for each child process,
you need to consider the amount of memory consumption.

This patch split the dump data based on only max_mapnr and num_dumpfile, so the size
of each splitted dumpfiles will be different by excluding unnecessary pages.

Signed-off-by: Atsushi Kumagai <kumagai-atsushi@...>
---
 makedumpfile.c |   42 +++++++++++++++++++++++++++++-------------
 1 file changed, 29 insertions(+), 13 deletions(-)

diff --git a/makedumpfile.c b/makedumpfile.c
index 5670bcf..be0e15e 100644
--- a/makedumpfile.c
+++ b/makedumpfile.c
 <at>  <at>  -5432,6 +5432,13  <at>  <at>  write_kdump_pages_cyclic(struct cache_data *cd_header, struct cache_data *cd_pag
 	start_pfn = info->cyclic_start_pfn;
 	end_pfn   = info->cyclic_end_pfn;

+	if (info->flag_split) {
+		if (start_pfn < info->split_start_pfn)
+			start_pfn = info->split_start_pfn;
+		if (end_pfn > info->split_end_pfn)
+			end_pfn = info->split_end_pfn;
+	}
+
 	for (pfn = start_pfn; pfn < end_pfn; pfn++) {

 		if ((num_dumped % per) == 0)

(Continue reading)

headers

Atsushi Kumagai | 29 Jun 2012 04:25

[RFC PATCH v2 10/10] Change num_dumped value to global for debug messages.

num_dumped value is used to count the number of dumped pages.
The value is defined as local value and doesn't work correctly for cyclic
processing. (Because the value will be initialized by each cycle.)

This patch fix the above issue.

Signed-off-by: Atsushi Kumagai <kumagai-atsushi@...>
---
 makedumpfile.c |   11 ++++++-----
 1 file changed, 6 insertions(+), 5 deletions(-)

diff --git a/makedumpfile.c b/makedumpfile.c
index be0e15e..67e3727 100644
--- a/makedumpfile.c
+++ b/makedumpfile.c
 <at>  <at>  -44,6 +44,8  <at>  <at>  unsigned long long pfn_cache_private;
 unsigned long long pfn_user;
 unsigned long long pfn_free;

+unsigned long long num_dumped;
+
 int retcd = FAILED;	/* return code */

 #define INITIALIZE_LONG_TABLE(table, value) \
 <at>  <at>  -4627,7 +4629,7  <at>  <at>  write_elf_pages(struct cache_data *cd_header, struct cache_data *cd_page)
 	int i, phnum;
 	long page_size = info->page_size;
 	unsigned long long pfn, pfn_start, pfn_end, paddr, num_excluded;
-	unsigned long long num_dumpable, num_dumped = 0, per;
+	unsigned long long num_dumpable, per;

(Continue reading)

headers

Vivek Goyal | 2 Jul 2012 14:39

Re: [RFC PATCH v2 0/10] makedumpfile: cyclic processing to keep memory consumption.

On Fri, Jun 29, 2012 at 11:13:20AM +0900, Atsushi Kumagai wrote:
> Hello,
> 
> I improved prototype of cyclic processing as version 2.
> If there is no objection to basic idea, I want to consider the things
> related to performance as next step. (Concretely, buffer size and the patch set
> HATAYAMA-san sent a short time ago.) 
> 

Hi Atushi san,

Good to see this work making progress. I have few queries.

- Do you have some numbers for bigger machines like 1TB or higher memory. 
  I am curious to know how bad is the time penalty.

- Will this work with option -F (flattned format). Often people save
  filtered dump over ssh and we need to make sure it does work with -F
  option too.

> > Version 1:
>  
>   http://lists.infradead.org/pipermail/kexec/2012-May/006363.html

- I have few queries about the diagram in the link above.

- What is 1st cycle, 2nd cycle and 3rd cycle. Are we cycling thorough
  all the pages 3 times for everything?

- What is 1st bitmap and 2nd bitmap and page_header? And why 3 cycles for

(Continue reading)

headers

Atsushi Kumagai | 4 Jul 2012 07:54

Re: [RFC PATCH v2 0/10] makedumpfile: cyclic processing to keep memory consumption.

Hello Vivek,

On Mon, 2 Jul 2012 08:39:05 -0400
Vivek Goyal <vgoyal@...> wrote:

> On Fri, Jun 29, 2012 at 11:13:20AM +0900, Atsushi Kumagai wrote:
> > Hello,
> > 
> > I improved prototype of cyclic processing as version 2.
> > If there is no objection to basic idea, I want to consider the things
> > related to performance as next step. (Concretely, buffer size and the patch set
> > HATAYAMA-san sent a short time ago.) 
> > 
> 
> Hi Atushi san,
> 
> Good to see this work making progress. I have few queries.
> 
> - Do you have some numbers for bigger machines like 1TB or higher memory. 
>   I am curious to know how bad is the time penalty.

I'm afraid that I don't have such a large machine, so I need someone who can
measure execution time in large machine.

> - Will this work with option -F (flattned format). Often people save
>   filtered dump over ssh and we need to make sure it does work with -F
>   option too.

Yes, the cyclic processing supports flattened format too.

(Continue reading)

headers

HATAYAMA Daisuke | 4 Jul 2012 10:52

Re: [RFC PATCH v2 0/10] makedumpfile: cyclic processing to keep memory consumption.

From: Atsushi Kumagai <kumagai-atsushi@...>
Subject: Re: [RFC PATCH v2 0/10] makedumpfile: cyclic processing to keep memory consumption.
Date: Wed, 4 Jul 2012 14:54:06 +0900

> Hello Vivek,
> 
> On Mon, 2 Jul 2012 08:39:05 -0400
> Vivek Goyal <vgoyal@...> wrote:
> 
>> On Fri, Jun 29, 2012 at 11:13:20AM +0900, Atsushi Kumagai wrote:
>> > Hello,
>> > 
>> > I improved prototype of cyclic processing as version 2.
>> > If there is no objection to basic idea, I want to consider the things
>> > related to performance as next step. (Concretely, buffer size and the patch set
>> > HATAYAMA-san sent a short time ago.) 
>> > 
>> 
>> Hi Atushi san,
>> 
>> Good to see this work making progress. I have few queries.
>> 
>> - Do you have some numbers for bigger machines like 1TB or higher memory. 
>>   I am curious to know how bad is the time penalty.
> 
> I'm afraid that I don't have such a large machine, so I need someone who can
> measure execution time in large machine.
> 

I can prepare such machine but I cannot say I can use the machine on

(Continue reading)

headers

HATAYAMA Daisuke | 11 Jul 2012 07:23

Re: [RFC PATCH v2 0/10] makedumpfile: cyclic processing to keep memory consumption.

From: HATAYAMA Daisuke <d.hatayama@...>
Subject: Re: [RFC PATCH v2 0/10] makedumpfile: cyclic processing to keep memory consumption.
Date: Wed, 4 Jul 2012 17:52:11 +0900

> From: Atsushi Kumagai <kumagai-atsushi@...>
> Subject: Re: [RFC PATCH v2 0/10] makedumpfile: cyclic processing to keep memory consumption.
> Date: Wed, 4 Jul 2012 14:54:06 +0900
> 
>> Hello Vivek,
>> 
>> On Mon, 2 Jul 2012 08:39:05 -0400
>> Vivek Goyal <vgoyal@...> wrote:
>> 
>>> On Fri, Jun 29, 2012 at 11:13:20AM +0900, Atsushi Kumagai wrote:
>>> > Hello,
>>> > 
>>> > I improved prototype of cyclic processing as version 2.
>>> > If there is no objection to basic idea, I want to consider the things
>>> > related to performance as next step. (Concretely, buffer size and the patch set
>>> > HATAYAMA-san sent a short time ago.) 
>>> > 
>>> 
>>> Hi Atushi san,
>>> 
>>> Good to see this work making progress. I have few queries.
>>> 
>>> - Do you have some numbers for bigger machines like 1TB or higher memory. 
>>>   I am curious to know how bad is the time penalty.
>> 
>> I'm afraid that I don't have such a large machine, so I need someone who can

(Continue reading)

headers

Atsushi Kumagai | 13 Jul 2012 02:36

Re: [RFC PATCH v2 0/10] makedumpfile: cyclic processing to keep memory consumption.

Hello HATAYAMA-san,

On Wed, 11 Jul 2012 14:23:59 +0900 (JST)
HATAYAMA Daisuke <d.hatayama@...> wrote:

[...]
> >>> Hi Atushi san,
> >>> 
> >>> Good to see this work making progress. I have few queries.
> >>> 
> >>> - Do you have some numbers for bigger machines like 1TB or higher memory. 
> >>>   I am curious to know how bad is the time penalty.
> >> 
> >> I'm afraid that I don't have such a large machine, so I need someone who can
> >> measure execution time in large machine.
> >> 
> > 
> > I can prepare such machine but I cannot say I can use the machine on
> > this day precisely for example. But I think it must be until the end
> > of this month at most. So, I would like to fix content of benchmark
> > first.
> > 
> 
> Hello Kumagai-san,
> 
> I'm now trying to reserve the machine with big memory, but I'm not
> accurately sure when I can use the machine; expecting within this
> month? In advance, I want to make sure what I measure in this
> benchmark in more detail.

(Continue reading)

headers

HATAYAMA Daisuke | 13 Jul 2012 07:18

Re: [RFC PATCH v2 0/10] makedumpfile: cyclic processing to keep memory consumption.

From: Atsushi Kumagai <kumagai-atsushi@...>
Subject: Re: [RFC PATCH v2 0/10] makedumpfile: cyclic processing to keep memory consumption.
Date: Fri, 13 Jul 2012 09:36:11 +0900

> Hello HATAYAMA-san,
> 
> On Wed, 11 Jul 2012 14:23:59 +0900 (JST)
> HATAYAMA Daisuke <d.hatayama@...> wrote:
> 
> [...]
>> >>> Hi Atushi san,
>> >>> 
>> >>> Good to see this work making progress. I have few queries.
>> >>> 
>> >>> - Do you have some numbers for bigger machines like 1TB or higher memory. 
>> >>>   I am curious to know how bad is the time penalty.
>> >> 
>> >> I'm afraid that I don't have such a large machine, so I need someone who can
>> >> measure execution time in large machine.
>> >> 
>> > 
>> > I can prepare such machine but I cannot say I can use the machine on
>> > this day precisely for example. But I think it must be until the end
>> > of this month at most. So, I would like to fix content of benchmark
>> > first.
>> > 
>> 
>> Hello Kumagai-san,
>> 
>> I'm now trying to reserve the machine with big memory, but I'm not

(Continue reading)

headers

Atsushi Kumagai | 13 Jul 2012 10:10

Re: [RFC PATCH v2 0/10] makedumpfile: cyclic processing to keep memory consumption.

Hello HATAYAMA-san,

On Fri, 13 Jul 2012 14:18:01 +0900 (JST)
HATAYAMA Daisuke <d.hatayama@...> wrote:

> From: Atsushi Kumagai <kumagai-atsushi@...>
> Subject: Re: [RFC PATCH v2 0/10] makedumpfile: cyclic processing to keep memory consumption.
> Date: Fri, 13 Jul 2012 09:36:11 +0900
> 
> > Hello HATAYAMA-san,
> > 
> > On Wed, 11 Jul 2012 14:23:59 +0900 (JST)
> > HATAYAMA Daisuke <d.hatayama@...> wrote:
> > 
> > [...]
> >> >>> Hi Atushi san,
> >> >>> 
> >> >>> Good to see this work making progress. I have few queries.
> >> >>> 
> >> >>> - Do you have some numbers for bigger machines like 1TB or higher memory. 
> >> >>>   I am curious to know how bad is the time penalty.
> >> >> 
> >> >> I'm afraid that I don't have such a large machine, so I need someone who can
> >> >> measure execution time in large machine.
> >> >> 
> >> > 
> >> > I can prepare such machine but I cannot say I can use the machine on
> >> > this day precisely for example. But I think it must be until the end
> >> > of this month at most. So, I would like to fix content of benchmark
> >> > first.

(Continue reading)

headers

HATAYAMA Daisuke | 18 Jul 2012 02:57

Re: [RFC PATCH v2 0/10] makedumpfile: cyclic processing to keep memory consumption.

From: Atsushi Kumagai <kumagai-atsushi@...>
Subject: Re: [RFC PATCH v2 0/10] makedumpfile: cyclic processing to keep memory consumption.
Date: Fri, 13 Jul 2012 17:10:39 +0900

> Hello HATAYAMA-san,
> 
> On Fri, 13 Jul 2012 14:18:01 +0900 (JST)
> HATAYAMA Daisuke <d.hatayama@...> wrote:
> 
>> From: Atsushi Kumagai <kumagai-atsushi@...>
>> Subject: Re: [RFC PATCH v2 0/10] makedumpfile: cyclic processing to keep memory consumption.
>> Date: Fri, 13 Jul 2012 09:36:11 +0900
>> 
>> > Hello HATAYAMA-san,
>> > 
>> > On Wed, 11 Jul 2012 14:23:59 +0900 (JST)
>> > HATAYAMA Daisuke <d.hatayama@...> wrote:
>> > 
>> > [...]
>> >> >>> Hi Atushi san,
>> >> >>> 
>> >> >>> Good to see this work making progress. I have few queries.
>> >> >>> 
>> >> >>> - Do you have some numbers for bigger machines like 1TB or higher memory. 
>> >> >>>   I am curious to know how bad is the time penalty.
>> >> >> 
>> >> >> I'm afraid that I don't have such a large machine, so I need someone who can
>> >> >> measure execution time in large machine.
>> >> >> 
>> >> >

(Continue reading)

headers

Vivek Goyal | 6 Aug 2012 22:47

Re: [RFC PATCH v2 0/10] makedumpfile: cyclic processing to keep memory consumption.

On Fri, Jun 29, 2012 at 11:13:20AM +0900, Atsushi Kumagai wrote:
> Hello,
> 
> I improved prototype of cyclic processing as version 2.
> If there is no objection to basic idea, I want to consider the things
> related to performance as next step. (Concretely, buffer size and the patch set
> HATAYAMA-san sent a short time ago.) 

Hi Atsushi San,

Just checking that what's the state of these patches now. Are they ready
to be included in makedumpfile?

I would love to see new makedumpfile where memory usage does not grow
by physical memory present in the system. (Assuming computig overhead
of cycles is bearable).

Thanks
Vivek

> 
> 
> Version 1:
>  
>   http://lists.infradead.org/pipermail/kexec/2012-May/006363.html
> 
> Introduction:
> 
>  - The purpose of cyclic processing is to fix memory consumption.
>  - cyclic processing doesn't use temporary bitmap file, store partial bitmap data

(Continue reading)

headers

HATAYAMA Daisuke | 7 Aug 2012 09:31

Re: [RFC PATCH v2 0/10] makedumpfile: cyclic processing to keep memory consumption.

From: Vivek Goyal <vgoyal@...>
Subject: Re: [RFC PATCH v2 0/10] makedumpfile: cyclic processing to keep memory consumption.
Date: Mon, 6 Aug 2012 16:47:31 -0400

> On Fri, Jun 29, 2012 at 11:13:20AM +0900, Atsushi Kumagai wrote:
>> Hello,
>> 
>> I improved prototype of cyclic processing as version 2.
>> If there is no objection to basic idea, I want to consider the things
>> related to performance as next step. (Concretely, buffer size and the patch set
>> HATAYAMA-san sent a short time ago.) 
> 
> Hi Atsushi San,
> 
> Just checking that what's the state of these patches now. Are they ready
> to be included in makedumpfile?
> 
> I would love to see new makedumpfile where memory usage does not grow
> by physical memory present in the system. (Assuming computig overhead
> of cycles is bearable).
> 
> Thanks
> Vivek
> 

Hello Vivek,

I'm just now benchmarking cycle processing on our machine. Please wait
for a while.

(Continue reading)

headers

HATAYAMA Daisuke | 10 Aug 2012 10:39

Re: [RFC PATCH v2 0/10] makedumpfile: cyclic processing to keep memory consumption.

From: HATAYAMA Daisuke <d.hatayama@...>
Subject: Re: [RFC PATCH v2 0/10] makedumpfile: cyclic processing to keep memory consumption.
Date: Tue, 7 Aug 2012 16:31:20 +0900

> From: Vivek Goyal <vgoyal@...>
> Subject: Re: [RFC PATCH v2 0/10] makedumpfile: cyclic processing to keep memory consumption.
> Date: Mon, 6 Aug 2012 16:47:31 -0400
> 
>> On Fri, Jun 29, 2012 at 11:13:20AM +0900, Atsushi Kumagai wrote:
>>> Hello,
>>> 
>>> I improved prototype of cyclic processing as version 2.
>>> If there is no objection to basic idea, I want to consider the things
>>> related to performance as next step. (Concretely, buffer size and the patch set
>>> HATAYAMA-san sent a short time ago.) 
>> 
>> Hi Atsushi San,
>> 
>> Just checking that what's the state of these patches now. Are they ready
>> to be included in makedumpfile?
>> 
>> I would love to see new makedumpfile where memory usage does not grow
>> by physical memory present in the system. (Assuming computig overhead
>> of cycles is bearable).
>> 
>> Thanks
>> Vivek
>> 
> 
> Hello Vivek,

(Continue reading)

headers

Vivek Goyal | 10 Aug 2012 16:36

Re: [RFC PATCH v2 0/10] makedumpfile: cyclic processing to keep memory consumption.

On Fri, Aug 10, 2012 at 05:39:38PM +0900, HATAYAMA Daisuke wrote:

[..]
> 
> I finished benchmarking filtering time and demonstrate the result.
> But I failed to collect amount of memory consumption by my mistake. If
> they are necessary, I'll again try to collect them. But we have 9 days
> vacation starting tommorow, so I'll do that after the vacation.
> 

Thanks a lot for doing this benchmarking.

> The machine spec I used is as follows:
> 
>   Memory: 2TB
>   CPU: Intel(R) Xeon(R) CPU E7- 8870   <at>  2.40GHz
>        (8 sockets, 10 cores, 2 threads)
> 
> In the first step, I chosed buffer size 10KB and it took about 3h 45m
> 57s. So, next I changed the buffer size to 512KB and measured up to
> 8MB.

What is this buffer size? Is user supposed to specify it? Is it some fixed
size buffer which makedumpfile can use to read in memory and once we cross
the buffer size we need to let some data from buffer go?

> 
> The result is as follows:
> 
> | buffer size | time       |

(Continue reading)

headers

HATAYAMA Daisuke | 14 Aug 2012 13:55

Re: [RFC PATCH v2 0/10] makedumpfile: cyclic processing to keep memory consumption.

From: Vivek Goyal <vgoyal@...>
Subject: Re: [RFC PATCH v2 0/10] makedumpfile: cyclic processing to keep memory consumption.
Date: Fri, 10 Aug 2012 10:36:55 -0400

> On Fri, Aug 10, 2012 at 05:39:38PM +0900, HATAYAMA Daisuke wrote:
> 
> [..]
>> 
>> I finished benchmarking filtering time and demonstrate the result.
>> But I failed to collect amount of memory consumption by my mistake. If
>> they are necessary, I'll again try to collect them. But we have 9 days
>> vacation starting tommorow, so I'll do that after the vacation.
>> 
> 
> Thanks a lot for doing this benchmarking.
> 
>> The machine spec I used is as follows:
>> 
>>   Memory: 2TB
>>   CPU: Intel(R) Xeon(R) CPU E7- 8870   <at>  2.40GHz
>>        (8 sockets, 10 cores, 2 threads)
>> 
>> In the first step, I chosed buffer size 10KB and it took about 3h 45m
>> 57s. So, next I changed the buffer size to 512KB and measured up to
>> 8MB.
> 
> What is this buffer size? Is user supposed to specify it? Is it some fixed
> size buffer which makedumpfile can use to read in memory and once we cross
> the buffer size we need to let some data from buffer go?
>

(Continue reading)

headers

Atsushi Kumagai | 15 Aug 2012 08:27

Re: [RFC PATCH v2 0/10] makedumpfile: cyclic processing to keep memory consumption.

Hello HATAYAMA-san,

On Tue, 14 Aug 2012 20:55:32 +0900 (JST)
HATAYAMA Daisuke <d.hatayama@...> wrote:

> From: Vivek Goyal <vgoyal@...>
> Subject: Re: [RFC PATCH v2 0/10] makedumpfile: cyclic processing to keep memory consumption.
> Date: Fri, 10 Aug 2012 10:36:55 -0400
> 
> > On Fri, Aug 10, 2012 at 05:39:38PM +0900, HATAYAMA Daisuke wrote:
> > 
> > [..]
> >> 
> >> I finished benchmarking filtering time and demonstrate the result.
> >> But I failed to collect amount of memory consumption by my mistake. If
> >> they are necessary, I'll again try to collect them. But we have 9 days
> >> vacation starting tommorow, so I'll do that after the vacation.
> >> 

Thank you for your help.
Could you continue to measure amount of memory consumption ?

> > 
> > Thanks a lot for doing this benchmarking.
> > 
> >> The machine spec I used is as follows:
> >> 
> >>   Memory: 2TB
> >>   CPU: Intel(R) Xeon(R) CPU E7- 8870   <at>  2.40GHz
> >>        (8 sockets, 10 cores, 2 threads)

(Continue reading)

headers

Vivek Goyal | 15 Aug 2012 15:31

Re: [RFC PATCH v2 0/10] makedumpfile: cyclic processing to keep memory consumption.

On Wed, Aug 15, 2012 at 03:27:10PM +0900, Atsushi Kumagai wrote:

[..]
> > >> 
> > >> STEP [Excluding free pages       ] : 49.846321 seconds
> > >> STEP [Excluding unnecessary pages] : 6.339228 seconds
> > >> STEP [Excluding free pages       ] : 48.595884 seconds
> > >> STEP [Excluding unnecessary pages] : 6.530479 seconds
> > >> STEP [Excluding free pages       ] : 48.598879 seconds
> > >> STEP [Excluding unnecessary pages] : 6.527133 seconds
> > >> STEP [Excluding free pages       ] : 48.602401 seconds
> > >> STEP [Excluding unnecessary pages] : 6.502681 seconds
> > >> STEP [Excluding free pages       ] : 48.602010 seconds
> > >> STEP [Excluding unnecessary pages] : 6.469853 seconds
> > >> STEP [Excluding free pages       ] : 48.601637 seconds
> > >> STEP [Excluding unnecessary pages] : 6.431381 seconds
> > >> STEP [Excluding free pages       ] : 48.601195 seconds
> > >> STEP [Excluding unnecessary pages] : 6.416676 seconds
> > >> STEP [Excluding free pages       ] : 48.602221 seconds
> > >> STEP [Excluding unnecessary pages] : 6.387611 seconds
> > >> STEP [Excluding free pages       ] : 48.589972 seconds
> > >> STEP [Excluding unnecessary pages] : 0.816955 seconds
> > > 
> > > So what does above represent. Each step is taking 48 seconds or total
> > > time taken to filter vmcore is 48 seconds? What's the buffer size used
> > > here.
> > > 
> > 
> > The free_list logic always filteres a whole memory range even if the
> > range we need to filter is only a cerntain part, so it took about 48

(Continue reading)

headers

HATAYAMA Daisuke | 20 Aug 2012 02:12

Re: [RFC PATCH v2 0/10] makedumpfile: cyclic processing to keep memory consumption.

From: Atsushi Kumagai <kumagai-atsushi@...>
Subject: Re: [RFC PATCH v2 0/10] makedumpfile: cyclic processing to keep memory consumption.
Date: Wed, 15 Aug 2012 15:27:10 +0900

> Hello HATAYAMA-san,
> 
> On Tue, 14 Aug 2012 20:55:32 +0900 (JST)
> HATAYAMA Daisuke <d.hatayama@...> wrote:
> 
>> From: Vivek Goyal <vgoyal@...>
>> Subject: Re: [RFC PATCH v2 0/10] makedumpfile: cyclic processing to keep memory consumption.
>> Date: Fri, 10 Aug 2012 10:36:55 -0400
>> 
>> > On Fri, Aug 10, 2012 at 05:39:38PM +0900, HATAYAMA Daisuke wrote:
>> > 
>> > [..]
>> >> 
>> >> I finished benchmarking filtering time and demonstrate the result.
>> >> But I failed to collect amount of memory consumption by my mistake. If
>> >> they are necessary, I'll again try to collect them. But we have 9 days
>> >> vacation starting tommorow, so I'll do that after the vacation.
>> >> 
> 
> Thank you for your help.
> Could you continue to measure amount of memory consumption ?
> 

OK, Kumagai-san, please wait for a while.

Thanks.

(Continue reading)

headers

HATAYAMA Daisuke | 29 Aug 2012 04:50

Re: [RFC PATCH v2 0/10] makedumpfile: cyclic processing to keep memory consumption.

From: Atsushi Kumagai <kumagai-atsushi@...>
Subject: Re: [RFC PATCH v2 0/10] makedumpfile: cyclic processing to keep memory consumption.
Date: Wed, 15 Aug 2012 15:27:10 +0900

> Hello HATAYAMA-san,
> 
> On Tue, 14 Aug 2012 20:55:32 +0900 (JST)
> HATAYAMA Daisuke <d.hatayama@...> wrote:
> 
>> From: Vivek Goyal <vgoyal@...>
>> Subject: Re: [RFC PATCH v2 0/10] makedumpfile: cyclic processing to keep memory consumption.
>> Date: Fri, 10 Aug 2012 10:36:55 -0400
>> 
>> > On Fri, Aug 10, 2012 at 05:39:38PM +0900, HATAYAMA Daisuke wrote:
>> > 
>> > [..]
>> >> 
>> >> I finished benchmarking filtering time and demonstrate the result.
>> >> But I failed to collect amount of memory consumption by my mistake. If
>> >> they are necessary, I'll again try to collect them. But we have 9 days
>> >> vacation starting tommorow, so I'll do that after the vacation.
>> >> 
> 
> Thank you for your help.
> Could you continue to measure amount of memory consumption ?
> 

Hello Kumagai-san, here is the benchmark result for the amount of
actual memory consumption when there are multiple child processes
using --split option.

(Continue reading)

headers

Vivek Goyal | 29 Aug 2012 14:35

Re: [RFC PATCH v2 0/10] makedumpfile: cyclic processing to keep memory consumption.

On Wed, Aug 29, 2012 at 11:50:31AM +0900, HATAYAMA Daisuke wrote:
> From: Atsushi Kumagai <kumagai-atsushi@...>
> Subject: Re: [RFC PATCH v2 0/10] makedumpfile: cyclic processing to keep memory consumption.
> Date: Wed, 15 Aug 2012 15:27:10 +0900
> 
> > Hello HATAYAMA-san,
> > 
> > On Tue, 14 Aug 2012 20:55:32 +0900 (JST)
> > HATAYAMA Daisuke <d.hatayama@...> wrote:
> > 
> >> From: Vivek Goyal <vgoyal@...>
> >> Subject: Re: [RFC PATCH v2 0/10] makedumpfile: cyclic processing to keep memory consumption.
> >> Date: Fri, 10 Aug 2012 10:36:55 -0400
> >> 
> >> > On Fri, Aug 10, 2012 at 05:39:38PM +0900, HATAYAMA Daisuke wrote:
> >> > 
> >> > [..]
> >> >> 
> >> >> I finished benchmarking filtering time and demonstrate the result.
> >> >> But I failed to collect amount of memory consumption by my mistake. If
> >> >> they are necessary, I'll again try to collect them. But we have 9 days
> >> >> vacation starting tommorow, so I'll do that after the vacation.
> >> >> 
> > 
> > Thank you for your help.
> > Could you continue to measure amount of memory consumption ?
> > 
> 
> Hello Kumagai-san, here is the benchmark result for the amount of
> actual memory consumption when there are multiple child processes

(Continue reading)

headers

HATAYAMA Daisuke | 30 Aug 2012 02:55

Re: [RFC PATCH v2 0/10] makedumpfile: cyclic processing to keep memory consumption.

From: Vivek Goyal <vgoyal@...>
Subject: Re: [RFC PATCH v2 0/10] makedumpfile: cyclic processing to keep memory consumption.
Date: Wed, 29 Aug 2012 08:35:39 -0400

> On Wed, Aug 29, 2012 at 11:50:31AM +0900, HATAYAMA Daisuke wrote:
>> From: Atsushi Kumagai <kumagai-atsushi@...>
>> Subject: Re: [RFC PATCH v2 0/10] makedumpfile: cyclic processing to keep memory consumption.
>> Date: Wed, 15 Aug 2012 15:27:10 +0900
>> 
>> > Hello HATAYAMA-san,
>> > 
>> > On Tue, 14 Aug 2012 20:55:32 +0900 (JST)
>> > HATAYAMA Daisuke <d.hatayama@...> wrote:
>> > 
>> >> From: Vivek Goyal <vgoyal@...>
>> >> Subject: Re: [RFC PATCH v2 0/10] makedumpfile: cyclic processing to keep memory consumption.
>> >> Date: Fri, 10 Aug 2012 10:36:55 -0400
>> >> 
>> >> > On Fri, Aug 10, 2012 at 05:39:38PM +0900, HATAYAMA Daisuke wrote:
>> >> > 
>> >> > [..]
>> >> >> 
>> >> >> I finished benchmarking filtering time and demonstrate the result.
>> >> >> But I failed to collect amount of memory consumption by my mistake. If
>> >> >> they are necessary, I'll again try to collect them. But we have 9 days
>> >> >> vacation starting tommorow, so I'll do that after the vacation.
>> >> >> 
>> > 
>> > Thank you for your help.
>> > Could you continue to measure amount of memory consumption ?

(Continue reading)

headers

Atsushi Kumagai | 30 Aug 2012 08:29

Re: [RFC PATCH v2 0/10] makedumpfile: cyclic processing to keep memory consumption.

Hello HATAYAMA-san,

On Thu, 30 Aug 2012 09:55:06 +0900 (JST)
HATAYAMA Daisuke <d.hatayama@...> wrote:

> From: Vivek Goyal <vgoyal@...>
> Subject: Re: [RFC PATCH v2 0/10] makedumpfile: cyclic processing to keep memory consumption.
> Date: Wed, 29 Aug 2012 08:35:39 -0400
> 
> > On Wed, Aug 29, 2012 at 11:50:31AM +0900, HATAYAMA Daisuke wrote:
> >> From: Atsushi Kumagai <kumagai-atsushi@...>
> >> Subject: Re: [RFC PATCH v2 0/10] makedumpfile: cyclic processing to keep memory consumption.
> >> Date: Wed, 15 Aug 2012 15:27:10 +0900
> >> 
> >> > Hello HATAYAMA-san,
> >> > 
> >> > On Tue, 14 Aug 2012 20:55:32 +0900 (JST)
> >> > HATAYAMA Daisuke <d.hatayama@...> wrote:
> >> > 
> >> >> From: Vivek Goyal <vgoyal@...>
> >> >> Subject: Re: [RFC PATCH v2 0/10] makedumpfile: cyclic processing to keep memory consumption.
> >> >> Date: Fri, 10 Aug 2012 10:36:55 -0400
> >> >> 
> >> >> > On Fri, Aug 10, 2012 at 05:39:38PM +0900, HATAYAMA Daisuke wrote:
> >> >> > 
> >> >> > [..]
> >> >> >> 
> >> >> >> I finished benchmarking filtering time and demonstrate the result.
> >> >> >> But I failed to collect amount of memory consumption by my mistake. If
> >> >> >> they are necessary, I'll again try to collect them. But we have 9 days

(Continue reading)

headers

Atsushi Kumagai | 8 Aug 2012 07:14

Re: [RFC PATCH v2 0/10] makedumpfile: cyclic processing to keep memory consumption.

Hello Vivek,

On Mon, 6 Aug 2012 16:47:31 -0400
Vivek Goyal <vgoyal@...> wrote:

> On Fri, Jun 29, 2012 at 11:13:20AM +0900, Atsushi Kumagai wrote:
> > Hello,
> > 
> > I improved prototype of cyclic processing as version 2.
> > If there is no objection to basic idea, I want to consider the things
> > related to performance as next step. (Concretely, buffer size and the patch set
> > HATAYAMA-san sent a short time ago.) 
> 
> Hi Atsushi San,
> 
> Just checking that what's the state of these patches now. Are they ready
> to be included in makedumpfile?

Yes, v2 patches with some fixes are ready to be merged into mainline.
At this time, makedumpfile can work in constant memory space.
(To make sure it's correct, we need to see the result of HATAYAMA-san's benchmark.)

> I would love to see new makedumpfile where memory usage does not grow
> by physical memory present in the system. (Assuming computig overhead
> of cycles is bearable).

I planed to release the next version with the new method to exclude free pages
(based on HATAYAMA-san's RFC patches). This method looks up members of each page
instead of free_list, it's only for performance.

(Continue reading)

headers

Vivek Goyal | 8 Aug 2012 15:25

Re: [RFC PATCH v2 0/10] makedumpfile: cyclic processing to keep memory consumption.

On Wed, Aug 08, 2012 at 02:14:00PM +0900, Atsushi Kumagai wrote:
> Hello Vivek,
> 
> On Mon, 6 Aug 2012 16:47:31 -0400
> Vivek Goyal <vgoyal@...> wrote:
> 
> > On Fri, Jun 29, 2012 at 11:13:20AM +0900, Atsushi Kumagai wrote:
> > > Hello,
> > > 
> > > I improved prototype of cyclic processing as version 2.
> > > If there is no objection to basic idea, I want to consider the things
> > > related to performance as next step. (Concretely, buffer size and the patch set
> > > HATAYAMA-san sent a short time ago.) 
> > 
> > Hi Atsushi San,
> > 
> > Just checking that what's the state of these patches now. Are they ready
> > to be included in makedumpfile?
> 
> Yes, v2 patches with some fixes are ready to be merged into mainline.
> At this time, makedumpfile can work in constant memory space.
> (To make sure it's correct, we need to see the result of HATAYAMA-san's benchmark.)
> 
> > I would love to see new makedumpfile where memory usage does not grow
> > by physical memory present in the system. (Assuming computig overhead
> > of cycles is bearable).
> 
> I planed to release the next version with the new method to exclude free pages
> (based on HATAYAMA-san's RFC patches). This method looks up members of each page
> instead of free_list, it's only for performance.

(Continue reading)

headers

Atsushi Kumagai | 9 Aug 2012 08:44

Re: [RFC PATCH v2 0/10] makedumpfile: cyclic processing to keep memory consumption.

Hello Vivek,

On Wed, 8 Aug 2012 09:25:10 -0400
Vivek Goyal <vgoyal@...> wrote:

> On Wed, Aug 08, 2012 at 02:14:00PM +0900, Atsushi Kumagai wrote:
> > Hello Vivek,
> > 
> > On Mon, 6 Aug 2012 16:47:31 -0400
> > Vivek Goyal <vgoyal@...> wrote:
> > 
> > > On Fri, Jun 29, 2012 at 11:13:20AM +0900, Atsushi Kumagai wrote:
> > > > Hello,
> > > > 
> > > > I improved prototype of cyclic processing as version 2.
> > > > If there is no objection to basic idea, I want to consider the things
> > > > related to performance as next step. (Concretely, buffer size and the patch set
> > > > HATAYAMA-san sent a short time ago.) 
> > > 
> > > Hi Atsushi San,
> > > 
> > > Just checking that what's the state of these patches now. Are they ready
> > > to be included in makedumpfile?
> > 
> > Yes, v2 patches with some fixes are ready to be merged into mainline.
> > At this time, makedumpfile can work in constant memory space.
> > (To make sure it's correct, we need to see the result of HATAYAMA-san's benchmark.)
> > 
> > > I would love to see new makedumpfile where memory usage does not grow
> > > by physical memory present in the system. (Assuming computig overhead

(Continue reading)