Björn-Egil Dahlberg | 7 Feb 16:27 2013

Re: [eeps] New EEP: setrlimit(2) analogue for Erlang

I dug out what I wrote a year ago ..

eep-draft:
https://github.com/psyeugenic/eep/blob/egil/system_limits/eeps/eep-00xx.md

Reference implementation:
https://github.com/psyeugenic/otp/commits/egil/limits-system-gc/OTP-9856
Remember, this is a prototype and a reference implementation.

There is a couple of issues not addressed or at least open-ended.

* Should processes be able to set limits on other processes? I think not though my draft argues for it. It introduces unnecessary restraints on erts and hinders performance. 'save_calls' is such an option.

* ets - if your table increases beyond some limit. Who should we punish? The inserter? The owner? What would be the rationale? We cannot just punish the inserter, the ets table is still there taking a lot of memory and no other process could insert into the table. They would be killed as well. Remove the owner and hence the table (and potential heir)? What kind of problems would arise then? Limits should be tied into a supervision strategy and restart the whole thing.

* In my draft and reference implementation I use soft limits. Once a process reaches its limit it will be marked for termination by an exit signal. The trouble here is there is no real guarantee for how long this will take. A process can continue appending a binary for a short while and ending the beam with OOM still. (If I remember it correctly you have to schedule out to terminate a process in SMP thus you need to bump all reduction. But, not all things handle return values from the garbage collector, most notably within the append_binary instruction). There may be other issues as well.

* Message queues. In the current implementation of message queues we have two queues. An inner one which is locked by the receiver process while executing and an outer one which other processes will use and thus not compete for a message queue lock with the executing process. When the inner queue is depleted the receiver process will lock the outer queue and move the entire thing to the inner one. Rinse and repeat. The only guarantee we have to ensure with our implementation is: signal order between two processes. So, in the future we might have several queues to improve performance. If you introduce monitoring of the total number messages in the abstracted queue (all the queues) this will most probable kill any sort of scalability. For instance a sender would not be allowed to check the inner queue for this reason. Would a "fast" counter check in the inner queue be allowed? Perhaps if it is fast enough, but any sort of bookkeeping costs performance. If we introduce even more queues for scalability reasons this will cost even more.

* What about other memory users? Drivers? NIFs?

I do believe in increments in development as long it is path to the envisioned goal.
And to reiterate, i'm not convinced that limits on just processes is the way to go. I think a complete monitoring system should be envisioned, not just for processes.

// Björn-Egil

On 2013-02-06 23:03, Richard O'Keefe wrote:
Just today, I saw Matthew Evans' This pertains to a feature I would like to see in Erlang. The ability to set an optional "memory limit" when a process and ETS table is created (and maybe a global optional per-process limit when the VM is started). I've seen a few cases where, due to software bugs, a process size grows and grows; unfortunately as things stand today the result is your entire VM crashing - hopefully leaving you with a crash_dump. Having such a limit could cause the process to terminate (producing a OOM crash report in erlang.log) and the crashing process could be handled with supervisor rules. Even better you can envisage setting the limits artificially low during testing to catch these types of bugs early on. in my mailbox. I have seen too many such e-mail messages. Here's a specific proposal. It's time _something_ was done about this kind of problem. I don't expect that my EEP is the best way to deal with it, but at least there's going to be something for people to point to.

_______________________________________________ eeps mailing list eeps <at> erlang.org http://erlang.org/mailman/listinfo/eeps

<div>
    <div class="moz-cite-prefix">I dug out what I wrote a year ago ..<br><br>
      eep-draft:<br><a class="moz-txt-link-freetext" href="https://github.com/psyeugenic/eep/blob/egil/system_limits/eeps/eep-00xx.md">https://github.com/psyeugenic/eep/blob/egil/system_limits/eeps/eep-00xx.md</a><br><br>
      Reference implementation:<br><a class="moz-txt-link-freetext" href="https://github.com/psyeugenic/otp/commits/egil/limits-system-gc/OTP-9856">https://github.com/psyeugenic/otp/commits/egil/limits-system-gc/OTP-9856</a><br>
      Remember, this is a prototype and a reference implementation.<br><br>
      There is a couple of issues not addressed or at least open-ended.<br><br>
      * Should processes be able to set limits on other processes? I
      think not though my draft argues for it. It introduces unnecessary
      restraints on erts and hinders performance. 'save_calls' is such
      an option.<br><br>
      * ets - if your table increases beyond some limit. Who should we
      punish? The inserter? The owner? What would be the rationale? We
      cannot just punish the inserter, the ets table is still there
      taking a lot of memory and no other process could insert into the
      table. They would be killed as well. Remove the owner and hence
      the table (and potential heir)? What kind of problems would arise
      then? Limits should be tied into a supervision strategy and
      restart the whole thing.<br><br>
      * In my draft and reference implementation I use soft limits. Once
      a process reaches its limit it will be marked for termination by
      an exit signal. The trouble here is there is no real guarantee for
      how long this will take. A process can continue appending a binary
      for a short while and ending the beam with OOM still. (If I
      remember it correctly you have to schedule out to terminate a
      process in SMP thus you need to bump all reduction. But, not all
      things handle return values from the garbage collector, most
      notably within the append_binary instruction). There may be other
      issues as well.<br><br>
      * Message queues. In the current implementation of message queues
      we have two queues. An inner one which is locked by the receiver
      process while executing and an outer one which other processes
      will use and thus not compete for a message queue lock with the
      executing process. When the inner queue is depleted the receiver
      process will lock the outer queue and move the entire thing to the
      inner one. Rinse and repeat. The only guarantee we have to ensure
      with our implementation is: signal order between two processes.
      So, in the future we might have several queues to improve
      performance. If you introduce monitoring of the total number
      messages in the abstracted queue (all the queues) this will most
      probable kill any sort of scalability. For instance a sender would
      not be allowed to check the inner queue for this reason. Would a
      "fast" counter check in the inner queue be allowed? Perhaps if it
      is fast enough, but any sort of bookkeeping costs performance. If
      we introduce even more queues for scalability reasons this will
      cost even more.<br><br>
      * What about other memory users? Drivers? NIFs?<br><br>
      I do believe in increments in development as long it is path to
      the envisioned goal.<br>
      And to reiterate, i'm not convinced that limits on just processes
      is the way to go. I think a complete monitoring system should be
      envisioned, not just for processes.<br><br>
      // Bj&ouml;rn-Egil<br><br>
      On 2013-02-06 23:03, Richard O'Keefe wrote:<br>
</div>
    <blockquote cite="mid:85819AB4-AD04-401F-B814-88157B71BE07 <at> cs.otago.ac.nz" type="cite">
      Just today, I saw Matthew Evans'

	This pertains to a feature I would like to see
	in Erlang.  The ability to set an optional
	"memory limit" when a process and ETS table is
	created (and maybe a global optional per-process
	limit when the VM is started).  I've seen a few
	cases where, due to software bugs, a process size
	grows and grows; unfortunately as things stand
	today the result is your entire VM crashing -
	hopefully leaving you with a crash_dump. 

	Having such a limit could cause the process to
	terminate (producing a OOM crash report in
	erlang.log) and the crashing process could be
	handled with supervisor rules.  Even better you
	can envisage setting the limits artificially low
	during testing to catch these types of bugs early on.

in my mailbox.  I have seen too many such e-mail messages.
Here's a specific proposal.  It's time _something_ was done
about this kind of problem.  I don't expect that my EEP is
the best way to deal with it, but at least there's going to
be something for people to point to.

      <br><br>_______________________________________________
eeps mailing list
<a class="moz-txt-link-abbreviated" href="mailto:eeps <at> erlang.org">eeps <at> erlang.org</a>
<a class="moz-txt-link-freetext" href="http://erlang.org/mailman/listinfo/eeps">http://erlang.org/mailman/listinfo/eeps</a>

    </blockquote>
    <br>
</div>
Matthew Evans | 7 Feb 18:51 2013
Picon

Re: [eeps] New EEP: setrlimit(2) analogue for Erlang


________________________________
> Date: Thu, 7 Feb 2013 16:27:49 +0100 
> From: egil <at> erlang.org 
> To: eeps <at> erlang.org 
> CC: erlang-questions <at> erlang.org 
> Subject: Re: [erlang-questions] [eeps] New EEP: setrlimit(2) analogue 
> for Erlang 
> 
> I dug out what I wrote a year ago .. 
> 
> eep-draft: 
> https://github.com/psyeugenic/eep/blob/egil/system_limits/eeps/eep-00xx.md 
> 
> Reference implementation: 
> https://github.com/psyeugenic/otp/commits/egil/limits-system-gc/OTP-9856 
> Remember, this is a prototype and a reference implementation. 
> 
> There is a couple of issues not addressed or at least open-ended. 
> 
> * Should processes be able to set limits on other processes? I think 
> not though my draft argues for it. It introduces unnecessary restraints 
> on erts and hinders performance. 'save_calls' is such an option. 

I agree, other processes should not be able to do this. It kind of goes against Erlang's ideas of process isolation.

> * ets - if your table increases beyond some limit. Who should we 
> punish? The inserter? The owner? What would be the rationale? We cannot 
> just punish the inserter, the ets table is still there taking a lot of 
> memory and no other process could insert into the table. They would be 
> killed as well. Remove the owner and hence the table (and potential 
> heir)? What kind of problems would arise then? Limits should be tied 
> into a supervision strategy and restart the whole thing. 

I think just the owner is good enough. It would be nice if the inserter could do it, but I imagine that's a
non-trivial feature to implement. 

> 
> * Message queues. In the current implementation of message queues we 
> have two queues. An inner one which is locked by the receiver process 
> while executing and an outer one which other processes will use and 
> thus not compete for a message queue lock with the executing process. 
> When the inner queue is depleted the receiver process will lock the 
> outer queue and move the entire thing to the inner one. Rinse and 
> repeat. The only guarantee we have to ensure with our implementation 
> is: signal order between two processes. So, in the future we might have 
> several queues to improve performance. If you introduce monitoring of 
> the total number messages in the abstracted queue (all the queues) this 
> will most probable kill any sort of scalability. For instance a sender 
> would not be allowed to check the inner queue for this reason. Would a 
> "fast" counter check in the inner queue be allowed? Perhaps if it is 
> fast enough, but any sort of bookkeeping costs performance. If we 
> introduce even more queues for scalability reasons this will cost even 
> more. 

I wasn't aware that the workings of the message queues was that complex internally. You are correct, that
this monitoring must not be at the expense of performance, maybe the "check" could be made when the
scheduler decides to schedule and execute a process. Or maybe it can check every X reductions? This means a
process may not terminate at exactly the user-defined threshold, but I think most people can live with that.

> I do believe in increments in development as long it is path to the 
> envisioned goal. 
> And to reiterate, i'm not convinced that limits on just processes is 
> the way to go. I think a complete monitoring system should be 
> envisioned, not just for processes. 

Personally I think a complete monitoring system would be great, but I think process limits are also invaluable. 

I point you to Joe Armstrong's thesis "Making reliable distributed systems in the presence of software
errors". Certainly memory leaks are a perfectly valid software error, one that as things stand today will
crash your VM.

Cheers

Matt

 		 	   		  
Ville Tuulos | 7 Feb 20:12 2013
Picon

Re: [eeps] New EEP: setrlimit(2) analogue for Erlang

On Thu, Feb 7, 2013 at 7:27 AM, Björn-Egil Dahlberg <egil <at> erlang.org> wrote:
> I dug out what I wrote a year ago ..
>
> eep-draft:
> https://github.com/psyeugenic/eep/blob/egil/system_limits/eeps/eep-00xx.md
>
> Reference implementation:
> https://github.com/psyeugenic/otp/commits/egil/limits-system-gc/OTP-9856
> Remember, this is a prototype and a reference implementation.
>
> There is a couple of issues not addressed or at least open-ended.

Looks great! I truly hope this will get accepted rather sooner than
later, at least as an experimental feature.

Does the proposal cover refc binaries as well? Almost all interesting
use cases that I can imagine for limits involve binaries.

Here's one test case:

1. Create a web server in Erlang e.g. with Mochiweb, Cowboy, Inets.
2. Create a request handler that expects to receive Zip files, which
it extracts with zip:extract(Request, [memory]).
3. Create a zip bomb [1]: dd if=/dev/zero bs=1M count=8000 | zip req.zip -
4. POST the small req.zip to the web server.
5. See the VM go down in flames.

Obviously the per-process limits would elegantly solve this problem if
they covered binaries as well. A far less elegant solution would be to
handle the unsafe decompression with an external process and open_port
(inefficient) or implement a sophisticated alternative to the zip
module which handles the limits by itself (inefficient, annoying).

I understand that limiting the message queue / ets / NIFs can be
trickier. Just covering the basic max-heap with binaries would be a
good starting point.

Ville

[1] http://en.wikipedia.org/wiki/Zip_bomb

> * Should processes be able to set limits on other processes? I think not
> though my draft argues for it. It introduces unnecessary restraints on erts
> and hinders performance. 'save_calls' is such an option.
>
> * ets - if your table increases beyond some limit. Who should we punish? The
> inserter? The owner? What would be the rationale? We cannot just punish the
> inserter, the ets table is still there taking a lot of memory and no other
> process could insert into the table. They would be killed as well. Remove
> the owner and hence the table (and potential heir)? What kind of problems
> would arise then? Limits should be tied into a supervision strategy and
> restart the whole thing.
>
> * In my draft and reference implementation I use soft limits. Once a process
> reaches its limit it will be marked for termination by an exit signal. The
> trouble here is there is no real guarantee for how long this will take. A
> process can continue appending a binary for a short while and ending the
> beam with OOM still. (If I remember it correctly you have to schedule out to
> terminate a process in SMP thus you need to bump all reduction. But, not all
> things handle return values from the garbage collector, most notably within
> the append_binary instruction). There may be other issues as well.
>
> * Message queues. In the current implementation of message queues we have
> two queues. An inner one which is locked by the receiver process while
> executing and an outer one which other processes will use and thus not
> compete for a message queue lock with the executing process. When the inner
> queue is depleted the receiver process will lock the outer queue and move
> the entire thing to the inner one. Rinse and repeat. The only guarantee we
> have to ensure with our implementation is: signal order between two
> processes. So, in the future we might have several queues to improve
> performance. If you introduce monitoring of the total number messages in the
> abstracted queue (all the queues) this will most probable kill any sort of
> scalability. For instance a sender would not be allowed to check the inner
> queue for this reason. Would a "fast" counter check in the inner queue be
> allowed? Perhaps if it is fast enough, but any sort of bookkeeping costs
> performance. If we introduce even more queues for scalability reasons this
> will cost even more.
>
> * What about other memory users? Drivers? NIFs?
>
> I do believe in increments in development as long it is path to the
> envisioned goal.
> And to reiterate, i'm not convinced that limits on just processes is the way
> to go. I think a complete monitoring system should be envisioned, not just
> for processes.
>
> // Björn-Egil
>
> On 2013-02-06 23:03, Richard O'Keefe wrote:
>
> Just today, I saw Matthew Evans'
>
> 	This pertains to a feature I would like to see
> 	in Erlang.  The ability to set an optional
> 	"memory limit" when a process and ETS table is
> 	created (and maybe a global optional per-process
> 	limit when the VM is started).  I've seen a few
> 	cases where, due to software bugs, a process size
> 	grows and grows; unfortunately as things stand
> 	today the result is your entire VM crashing -
> 	hopefully leaving you with a crash_dump.
>
> 	Having such a limit could cause the process to
> 	terminate (producing a OOM crash report in
> 	erlang.log) and the crashing process could be
> 	handled with supervisor rules.  Even better you
> 	can envisage setting the limits artificially low
> 	during testing to catch these types of bugs early on.
>
> in my mailbox.  I have seen too many such e-mail messages.
> Here's a specific proposal.  It's time _something_ was done
> about this kind of problem.  I don't expect that my EEP is
> the best way to deal with it, but at least there's going to
> be something for people to point to.
>
>
>
> _______________________________________________
> eeps mailing list
> eeps <at> erlang.org
> http://erlang.org/mailman/listinfo/eeps
>
>
>
> _______________________________________________
> erlang-questions mailing list
> erlang-questions <at> erlang.org
> http://erlang.org/mailman/listinfo/erlang-questions
>
Björn-Egil Dahlberg | 7 Feb 23:24 2013
Picon

Re: [eeps] New EEP: setrlimit(2) analogue for Erlang

2013/2/7 Ville Tuulos <tuulos <at> gmail.com>

On Thu, Feb 7, 2013 at 7:27 AM, Björn-Egil Dahlberg <egil <at> erlang.org> wrote:
> I dug out what I wrote a year ago ..
>
> eep-draft:
> https://github.com/psyeugenic/eep/blob/egil/system_limits/eeps/eep-00xx.md
>
> Reference implementation:
> https://github.com/psyeugenic/otp/commits/egil/limits-system-gc/OTP-9856
> Remember, this is a prototype and a reference implementation.
>
> There is a couple of issues not addressed or at least open-ended.

Looks great! I truly hope this will get accepted rather sooner than
later, at least as an experimental feature.

Well as I said previously, I still have some issues with this approach.
Envision the goal and then see if this is a good first step.
Meaning: if we want to include other resource limits that is not process oriented how would we then design the interface then? A resource limits interface perhaps?
  

Does the proposal cover refc binaries as well? Almost all interesting
use cases that I can imagine for limits involve binaries.

My implementation doesn't cover it. It covers process memory blocks, i.e. everything except refc-binaries. (well their headers are covered: procbins)

Binaries are special beasts and needs special care. The easiest way to implement refc binary limits would be to use the virtual binary heap concept already present in the gc.

By using vheaps we would count all the memory referenced by procbins in the process heap. Now, here it gets interesting. Several procbins may refer to the same refc binary blob and that memory would be counted several times. You would have to take special accounting care if you want refc uniqueness which would make it kind of expensive. You can't mark the binary as counted since other processes might be gc:ed and they might also reference it so you would have to make some other accommodations.

Other than that you can just use the same idea as heap_size limits. The checks are similar and the api could be too: erlang:process_flag(limits, [{bin_vheap_size, Size}]). 
 
// Björn-Egil


Here's one test case:

1. Create a web server in Erlang e.g. with Mochiweb, Cowboy, Inets.
2. Create a request handler that expects to receive Zip files, which
it extracts with zip:extract(Request, [memory]).
3. Create a zip bomb [1]: dd if=/dev/zero bs=1M count=8000 | zip req.zip -
4. POST the small req.zip to the web server.
5. See the VM go down in flames.

Obviously the per-process limits would elegantly solve this problem if
they covered binaries as well. A far less elegant solution would be to
handle the unsafe decompression with an external process and open_port
(inefficient) or implement a sophisticated alternative to the zip
module which handles the limits by itself (inefficient, annoying).

I understand that limiting the message queue / ets / NIFs can be
trickier. Just covering the basic max-heap with binaries would be a
good starting point.

Ville

[1] http://en.wikipedia.org/wiki/Zip_bomb

> * Should processes be able to set limits on other processes? I think not
> though my draft argues for it. It introduces unnecessary restraints on erts
> and hinders performance. 'save_calls' is such an option.
>
> * ets - if your table increases beyond some limit. Who should we punish? The
> inserter? The owner? What would be the rationale? We cannot just punish the
> inserter, the ets table is still there taking a lot of memory and no other
> process could insert into the table. They would be killed as well. Remove
> the owner and hence the table (and potential heir)? What kind of problems
> would arise then? Limits should be tied into a supervision strategy and
> restart the whole thing.
>
> * In my draft and reference implementation I use soft limits. Once a process
> reaches its limit it will be marked for termination by an exit signal. The
> trouble here is there is no real guarantee for how long this will take. A
> process can continue appending a binary for a short while and ending the
> beam with OOM still. (If I remember it correctly you have to schedule out to
> terminate a process in SMP thus you need to bump all reduction. But, not all
> things handle return values from the garbage collector, most notably within
> the append_binary instruction). There may be other issues as well.
>
> * Message queues. In the current implementation of message queues we have
> two queues. An inner one which is locked by the receiver process while
> executing and an outer one which other processes will use and thus not
> compete for a message queue lock with the executing process. When the inner
> queue is depleted the receiver process will lock the outer queue and move
> the entire thing to the inner one. Rinse and repeat. The only guarantee we
> have to ensure with our implementation is: signal order between two
> processes. So, in the future we might have several queues to improve
> performance. If you introduce monitoring of the total number messages in the
> abstracted queue (all the queues) this will most probable kill any sort of
> scalability. For instance a sender would not be allowed to check the inner
> queue for this reason. Would a "fast" counter check in the inner queue be
> allowed? Perhaps if it is fast enough, but any sort of bookkeeping costs
> performance. If we introduce even more queues for scalability reasons this
> will cost even more.
>
> * What about other memory users? Drivers? NIFs?
>
> I do believe in increments in development as long it is path to the
> envisioned goal.
> And to reiterate, i'm not convinced that limits on just processes is the way
> to go. I think a complete monitoring system should be envisioned, not just
> for processes.
>
> // Björn-Egil
>
> On 2013-02-06 23:03, Richard O'Keefe wrote:
>
> Just today, I saw Matthew Evans'
>
>       This pertains to a feature I would like to see
>       in Erlang.  The ability to set an optional
>       "memory limit" when a process and ETS table is
>       created (and maybe a global optional per-process
>       limit when the VM is started).  I've seen a few
>       cases where, due to software bugs, a process size
>       grows and grows; unfortunately as things stand
>       today the result is your entire VM crashing -
>       hopefully leaving you with a crash_dump.
>
>       Having such a limit could cause the process to
>       terminate (producing a OOM crash report in
>       erlang.log) and the crashing process could be
>       handled with supervisor rules.  Even better you
>       can envisage setting the limits artificially low
>       during testing to catch these types of bugs early on.
>
> in my mailbox.  I have seen too many such e-mail messages.
> Here's a specific proposal.  It's time _something_ was done
> about this kind of problem.  I don't expect that my EEP is
> the best way to deal with it, but at least there's going to
> be something for people to point to.
>
>
>
> _______________________________________________
> eeps mailing list
> eeps <at> erlang.org
> http://erlang.org/mailman/listinfo/eeps
>
>
>
> _______________________________________________
> erlang-questions mailing list
> erlang-questions <at> erlang.org
> http://erlang.org/mailman/listinfo/erlang-questions
>
_______________________________________________
erlang-questions mailing list
erlang-questions <at> erlang.org
http://erlang.org/mailman/listinfo/erlang-questions

<div>
<p>2013/2/7 Ville Tuulos <span dir="ltr">&lt;<a href="mailto:tuulos <at> gmail.com" target="_blank">tuulos <at> gmail.com</a>&gt;</span><br></p>
<div class="gmail_quote">
<blockquote class="gmail_quote">
<div class="im">On Thu, Feb 7, 2013 at 7:27 AM, Bj&ouml;rn-Egil Dahlberg &lt;<a href="mailto:egil <at> erlang.org">egil <at> erlang.org</a>&gt; wrote:<br>
&gt; I dug out what I wrote a year ago ..<br>
&gt;<br>
&gt; eep-draft:<br>
&gt; <a href="https://github.com/psyeugenic/eep/blob/egil/system_limits/eeps/eep-00xx.md" target="_blank">https://github.com/psyeugenic/eep/blob/egil/system_limits/eeps/eep-00xx.md</a><br>
&gt;<br>
&gt; Reference implementation:<br>
&gt; <a href="https://github.com/psyeugenic/otp/commits/egil/limits-system-gc/OTP-9856" target="_blank">https://github.com/psyeugenic/otp/commits/egil/limits-system-gc/OTP-9856</a><br>
&gt; Remember, this is a prototype and a reference implementation.<br>
&gt;<br>
&gt; There is a couple of issues not addressed or at least open-ended.<br><br>
</div>Looks great! I truly hope this will get accepted rather sooner than<br>
later, at least as an experimental feature.<br>
</blockquote>
<div><br></div>
<div>Well as I said previously, I still have some issues with this approach.</div>
<div>Envision the goal and then see if this is a good first step.</div>
<div>Meaning: if we want to include other resource limits that is not process oriented how would we then design the interface then? A resource limits interface perhaps?</div>
<div>&nbsp;&nbsp;</div>
<blockquote class="gmail_quote">

<br>
Does the proposal cover refc binaries as well? Almost all interesting<br>
use cases that I can imagine for limits involve binaries.<br>
</blockquote>
<div><br></div>
<div>My implementation doesn't cover it. It covers process memory blocks, i.e. everything except refc-binaries. (well their headers are covered: procbins)</div>
<div><br></div>
<div>Binaries are special beasts and needs special care. The easiest way to implement refc binary limits would be to use the virtual binary heap concept already present in the gc.</div>
<div><br></div>
<div>By using vheaps we would count all the memory referenced by procbins in the process heap. Now, here it gets interesting. Several procbins may refer to the same refc binary blob and that memory would be counted several times. You would have to take special accounting care if you want refc uniqueness which would make it kind of expensive. You can't mark the binary as counted since other processes might be gc:ed and they might also reference it so you would have to make some other&nbsp;accommodations.</div>
<div><br></div>
<div>Other than that you can just use the same idea as heap_size limits. The checks are similar and the api could be too: erlang:process_flag(limits, [{bin_vheap_size, Size}]).&nbsp;</div>
<div>&nbsp;</div>
<div>// Bj&ouml;rn-Egil</div>
<div><br></div>
<blockquote class="gmail_quote">
<br>
Here's one test case:<br><br>
1. Create a web server in Erlang e.g. with Mochiweb, Cowboy, Inets.<br>
2. Create a request handler that expects to receive Zip files, which<br>
it extracts with zip:extract(Request, [memory]).<br>
3. Create a zip bomb [1]: dd if=/dev/zero bs=1M count=8000 | zip req.zip -<br>
4. POST the small req.zip to the web server.<br>
5. See the VM go down in flames.<br><br>
Obviously the per-process limits would elegantly solve this problem if<br>
they covered binaries as well. A far less elegant solution would be to<br>
handle the unsafe decompression with an external process and open_port<br>
(inefficient) or implement a sophisticated alternative to the zip<br>
module which handles the limits by itself (inefficient, annoying).<br><br>
I understand that limiting the message queue / ets / NIFs can be<br>
trickier. Just covering the basic max-heap with binaries would be a<br>
good starting point.<br><br>
Ville<br><br>
[1] <a href="http://en.wikipedia.org/wiki/Zip_bomb" target="_blank">http://en.wikipedia.org/wiki/Zip_bomb</a><br><div class="HOEnZb"><div class="h5">
<br>
&gt; * Should processes be able to set limits on other processes? I think not<br>
&gt; though my draft argues for it. It introduces unnecessary restraints on erts<br>
&gt; and hinders performance. 'save_calls' is such an option.<br>
&gt;<br>
&gt; * ets - if your table increases beyond some limit. Who should we punish? The<br>
&gt; inserter? The owner? What would be the rationale? We cannot just punish the<br>
&gt; inserter, the ets table is still there taking a lot of memory and no other<br>
&gt; process could insert into the table. They would be killed as well. Remove<br>
&gt; the owner and hence the table (and potential heir)? What kind of problems<br>
&gt; would arise then? Limits should be tied into a supervision strategy and<br>
&gt; restart the whole thing.<br>
&gt;<br>
&gt; * In my draft and reference implementation I use soft limits. Once a process<br>
&gt; reaches its limit it will be marked for termination by an exit signal. The<br>
&gt; trouble here is there is no real guarantee for how long this will take. A<br>
&gt; process can continue appending a binary for a short while and ending the<br>
&gt; beam with OOM still. (If I remember it correctly you have to schedule out to<br>
&gt; terminate a process in SMP thus you need to bump all reduction. But, not all<br>
&gt; things handle return values from the garbage collector, most notably within<br>
&gt; the append_binary instruction). There may be other issues as well.<br>
&gt;<br>
&gt; * Message queues. In the current implementation of message queues we have<br>
&gt; two queues. An inner one which is locked by the receiver process while<br>
&gt; executing and an outer one which other processes will use and thus not<br>
&gt; compete for a message queue lock with the executing process. When the inner<br>
&gt; queue is depleted the receiver process will lock the outer queue and move<br>
&gt; the entire thing to the inner one. Rinse and repeat. The only guarantee we<br>
&gt; have to ensure with our implementation is: signal order between two<br>
&gt; processes. So, in the future we might have several queues to improve<br>
&gt; performance. If you introduce monitoring of the total number messages in the<br>
&gt; abstracted queue (all the queues) this will most probable kill any sort of<br>
&gt; scalability. For instance a sender would not be allowed to check the inner<br>
&gt; queue for this reason. Would a "fast" counter check in the inner queue be<br>
&gt; allowed? Perhaps if it is fast enough, but any sort of bookkeeping costs<br>
&gt; performance. If we introduce even more queues for scalability reasons this<br>
&gt; will cost even more.<br>
&gt;<br>
&gt; * What about other memory users? Drivers? NIFs?<br>
&gt;<br>
&gt; I do believe in increments in development as long it is path to the<br>
&gt; envisioned goal.<br>
&gt; And to reiterate, i'm not convinced that limits on just processes is the way<br>
&gt; to go. I think a complete monitoring system should be envisioned, not just<br>
&gt; for processes.<br>
&gt;<br>
&gt; // Bj&ouml;rn-Egil<br>
&gt;<br>
&gt; On 2013-02-06 23:03, Richard O'Keefe wrote:<br>
&gt;<br>
&gt; Just today, I saw Matthew Evans'<br>
&gt;<br>
&gt; &nbsp; &nbsp; &nbsp; This pertains to a feature I would like to see<br>
&gt; &nbsp; &nbsp; &nbsp; in Erlang. &nbsp;The ability to set an optional<br>
&gt; &nbsp; &nbsp; &nbsp; "memory limit" when a process and ETS table is<br>
&gt; &nbsp; &nbsp; &nbsp; created (and maybe a global optional per-process<br>
&gt; &nbsp; &nbsp; &nbsp; limit when the VM is started). &nbsp;I've seen a few<br>
&gt; &nbsp; &nbsp; &nbsp; cases where, due to software bugs, a process size<br>
&gt; &nbsp; &nbsp; &nbsp; grows and grows; unfortunately as things stand<br>
&gt; &nbsp; &nbsp; &nbsp; today the result is your entire VM crashing -<br>
&gt; &nbsp; &nbsp; &nbsp; hopefully leaving you with a crash_dump.<br>
&gt;<br>
&gt; &nbsp; &nbsp; &nbsp; Having such a limit could cause the process to<br>
&gt; &nbsp; &nbsp; &nbsp; terminate (producing a OOM crash report in<br>
&gt; &nbsp; &nbsp; &nbsp; erlang.log) and the crashing process could be<br>
&gt; &nbsp; &nbsp; &nbsp; handled with supervisor rules. &nbsp;Even better you<br>
&gt; &nbsp; &nbsp; &nbsp; can envisage setting the limits artificially low<br>
&gt; &nbsp; &nbsp; &nbsp; during testing to catch these types of bugs early on.<br>
&gt;<br>
&gt; in my mailbox. &nbsp;I have seen too many such e-mail messages.<br>
&gt; Here's a specific proposal. &nbsp;It's time _something_ was done<br>
&gt; about this kind of problem. &nbsp;I don't expect that my EEP is<br>
&gt; the best way to deal with it, but at least there's going to<br>
&gt; be something for people to point to.<br>
&gt;<br>
&gt;<br>
&gt;<br>
&gt; _______________________________________________<br>
&gt; eeps mailing list<br>
&gt; <a href="mailto:eeps <at> erlang.org">eeps <at> erlang.org</a><br>
&gt; <a href="http://erlang.org/mailman/listinfo/eeps" target="_blank">http://erlang.org/mailman/listinfo/eeps</a><br>
&gt;<br>
&gt;<br>
&gt;<br>
</div></div>
<div class="HOEnZb"><div class="h5">&gt; _______________________________________________<br>
&gt; erlang-questions mailing list<br>
&gt; <a href="mailto:erlang-questions <at> erlang.org">erlang-questions <at> erlang.org</a><br>
&gt; <a href="http://erlang.org/mailman/listinfo/erlang-questions" target="_blank">http://erlang.org/mailman/listinfo/erlang-questions</a><br>
&gt;<br>
_______________________________________________<br>
erlang-questions mailing list<br><a href="mailto:erlang-questions <at> erlang.org">erlang-questions <at> erlang.org</a><br><a href="http://erlang.org/mailman/listinfo/erlang-questions" target="_blank">http://erlang.org/mailman/listinfo/erlang-questions</a><br>
</div></div>
</blockquote>
</div>
<br>
</div>
Tony Rogvall | 8 Feb 00:25 2013
Picon

Re: [eeps] New EEP: setrlimit(2) analogue for Erlang

For what it is worth.
I located my old implementation (three years old :-)


A very brief description (I presented some early stuff for OTP around then)
I do not think that max_message_queue_len is implemented,
not really defined. Too many options here.

/Tony

Resource Limits in Erlang
=========================

# Why?

- Ability to detect and kill runaway processes. 
- Detect and kill zombies.
- Basis of safe mode framework.
- Excellent to use in debugging.

# How?

- Limits are checked at context switch and garbage collection time. 
- Relatively light weight.
- Create limits by using spawn_opt.
- Limits are inherited by spawned processes.
- If a limit is reached a signal 'system_limit' is raised.

## max_memory
Limit the amount of memory a process may use. Account for all memory a 
process is using at any given time. This includes heap, stack, tables, 
links and messages. The memory is shared among all processes spawned by the 
process that where limited. spawn\_opt can be used to give 
away some of that memory to child processes.

## max\_time
Controls how many milliseconds a process may use in "wall clock" time.
Created (sub-)processes will only be able to run for the remaining time.

## max\_cpu
Controls how many milliseconds a process may run in cpu time. 
The cpu time is consumed by the process and all it’s spawned (sub-)processes.

## max\_reductions
A lighter version of max\_cpu. One reductions does approximately corresponds
to a function call.

## max\_processes
Limit number of running (sub-)processes that may be running at any given time.

## max\_ports
Limit number of open ports that can be open at any given time.

## max\_tables
Limit number of ets tables that may be open at any given time.

## max\_message\_queue\_len
Limit the size of the message queue. Who dies when the limit is reached?
Either sender or receiver? Maybe add a dangerous block option?

# process\_info

process\_info is used to read the current limits and the
"remaining" quota. 
process_info(Pid, max\_cpu) is used to read the number of
milliseconds set for execution while process_info(Pid, remaining\_cpu)
return how many cpu milliseconds that remain to execute.
The items include: max\_process, max\_ports, max\_tables, max\_memory, 
max\_reductions, max\_message\_queue\_len, max\_cpu, max\_time,
remaining\_process, remaining\_ports, remaining_tables, remaining\_memory, 
remaining\_reductions, remaining\_message\_queue\_len, 
remaining\_cpu, remaining\_time


On 7 feb 2013, at 16:27, Björn-Egil Dahlberg <egil <at> erlang.org> wrote:

I dug out what I wrote a year ago ..

eep-draft:
https://github.com/psyeugenic/eep/blob/egil/system_limits/eeps/eep-00xx.md

Reference implementation:
https://github.com/psyeugenic/otp/commits/egil/limits-system-gc/OTP-9856
Remember, this is a prototype and a reference implementation.

There is a couple of issues not addressed or at least open-ended.

* Should processes be able to set limits on other processes? I think not though my draft argues for it. It introduces unnecessary restraints on erts and hinders performance. 'save_calls' is such an option.

* ets - if your table increases beyond some limit. Who should we punish? The inserter? The owner? What would be the rationale? We cannot just punish the inserter, the ets table is still there taking a lot of memory and no other process could insert into the table. They would be killed as well. Remove the owner and hence the table (and potential heir)? What kind of problems would arise then? Limits should be tied into a supervision strategy and restart the whole thing.

* In my draft and reference implementation I use soft limits. Once a process reaches its limit it will be marked for termination by an exit signal. The trouble here is there is no real guarantee for how long this will take. A process can continue appending a binary for a short while and ending the beam with OOM still. (If I remember it correctly you have to schedule out to terminate a process in SMP thus you need to bump all reduction. But, not all things handle return values from the garbage collector, most notably within the append_binary instruction). There may be other issues as well.

* Message queues. In the current implementation of message queues we have two queues. An inner one which is locked by the receiver process while executing and an outer one which other processes will use and thus not compete for a message queue lock with the executing process. When the inner queue is depleted the receiver process will lock the outer queue and move the entire thing to the inner one. Rinse and repeat. The only guarantee we have to ensure with our implementation is: signal order between two processes. So, in the future we might have several queues to improve performance. If you introduce monitoring of the total number messages in the abstracted queue (all the queues) this will most probable kill any sort of scalability. For instance a sender would not be allowed to check the inner queue for this reason. Would a "fast" counter check in the inner queue be allowed? Perhaps if it is fast enough, but any sort of bookkeeping costs performance. If we introduce even more queues for scalability reasons this will cost even more.

* What about other memory users? Drivers? NIFs?

I do believe in increments in development as long it is path to the envisioned goal.
And to reiterate, i'm not convinced that limits on just processes is the way to go. I think a complete monitoring system should be envisioned, not just for processes.

// Björn-Egil

On 2013-02-06 23:03, Richard O'Keefe wrote:
Just today, I saw Matthew Evans' This pertains to a feature I would like to see in Erlang. The ability to set an optional "memory limit" when a process and ETS table is created (and maybe a global optional per-process limit when the VM is started). I've seen a few cases where, due to software bugs, a process size grows and grows; unfortunately as things stand today the result is your entire VM crashing - hopefully leaving you with a crash_dump. Having such a limit could cause the process to terminate (producing a OOM crash report in erlang.log) and the crashing process could be handled with supervisor rules. Even better you can envisage setting the limits artificially low during testing to catch these types of bugs early on. in my mailbox. I have seen too many such e-mail messages. Here's a specific proposal. It's time _something_ was done about this kind of problem. I don't expect that my EEP is the best way to deal with it, but at least there's going to be something for people to point to.

_______________________________________________ eeps mailing list eeps <at> erlang.org http://erlang.org/mailman/listinfo/eeps

_______________________________________________
erlang-questions mailing list
erlang-questions <at> erlang.org
http://erlang.org/mailman/listinfo/erlang-questions

"Installing applications can lead to corruption over time. Applications gradually write over each other's libraries, partial upgrades occur, user and system errors happen, and minute changes may be unnoticeable and difficult to fix"



<div>For what it is worth.<div>I located my old implementation (three years old :-)<div><br></div>
<div><a href="https://github.com/tonyrog/otp/tree/limits/">https://github.com/tonyrog/otp/tree/limits/</a></div>
<div>
<br><div>A very brief description (I presented some early stuff for OTP around then)</div>
<div>I do not think that max_message_queue_len is implemented,</div>
<div>not really defined. Too many options here.</div>
<div><br></div>
<div>/Tony</div>
<div><br></div>
<div>
<div>Resource Limits in Erlang</div>
<div>=========================</div>
<div><br></div>
<div># Why?</div>
<div><br></div>
<div>- Ability to detect and kill runaway processes.&nbsp;</div>
<div>- Detect and kill zombies.</div>
<div>- Basis of safe mode framework.</div>
<div>- Excellent to use in debugging.</div>
<div><br></div>
<div># How?</div>
<div><br></div>
<div>- Limits are checked at context switch and garbage collection time.&nbsp;</div>
<div>- Relatively light weight.</div>
<div>- Create limits by using spawn_opt.</div>
<div>- Limits are inherited by spawned processes.</div>
<div>- If a limit is reached a signal 'system_limit' is raised.</div>
<div><br></div>
<div>## max_memory</div>
<div>Limit the amount of memory a process may use. Account for all memory a&nbsp;</div>
<div>process is using at any given time. This includes heap, stack, tables,&nbsp;</div>
<div>links and messages. The memory is shared among all processes spawned by the&nbsp;</div>
<div>process that where limited. spawn\_opt can be used to give&nbsp;</div>
<div>away some of that memory to child processes.</div>
<div><br></div>
<div>## max\_time</div>
<div>Controls how many milliseconds a process may use in "wall clock" time.</div>
<div>Created (sub-)processes will only be able to run for the remaining time.</div>
<div><br></div>
<div>## max\_cpu</div>
<div>Controls how many milliseconds a process may run in cpu time.&nbsp;</div>
<div>The cpu time is consumed by the process and all it&rsquo;s spawned (sub-)processes.</div>
<div><br></div>
<div>## max\_reductions</div>
<div>A lighter version of max\_cpu. One reductions does approximately corresponds</div>
<div>to a function call.</div>
<div><br></div>
<div>## max\_processes</div>
<div>Limit number of running (sub-)processes that may be running at any given time.</div>
<div><br></div>
<div>## max\_ports</div>
<div>Limit number of open ports that can be open at any given time.</div>
<div><br></div>
<div>## max\_tables</div>
<div>Limit number of ets tables that may be open at any given time.</div>
<div><br></div>
<div>## max\_message\_queue\_len</div>
<div>Limit the size of the message queue. Who dies when the limit is reached?</div>
<div>Either sender or receiver? Maybe add a dangerous block option?</div>
<div><br></div>
<div># process\_info</div>
<div><br></div>
<div>process\_info is used to read the current limits and the</div>
<div>"remaining" quota.&nbsp;</div>
<div>process_info(Pid, max\_cpu) is used to read the number of</div>
<div>milliseconds set for execution while process_info(Pid, remaining\_cpu)</div>
<div>return how many cpu milliseconds that remain to execute.</div>
<div>The items include: max\_process, max\_ports, max\_tables, max\_memory,&nbsp;</div>
<div>max\_reductions, max\_message\_queue\_len, max\_cpu, max\_time,</div>
<div>remaining\_process, remaining\_ports, remaining_tables, remaining\_memory,&nbsp;</div>
<div>remaining\_reductions, remaining\_message\_queue\_len,&nbsp;</div>
<div>remaining\_cpu, remaining\_time</div>
</div>
<div><br></div>
<div>
<br><div>
<div>On 7 feb 2013, at 16:27, Bj&ouml;rn-Egil Dahlberg &lt;<a href="mailto:egil <at> erlang.org">egil <at> erlang.org</a>&gt; wrote:</div>
<br class="Apple-interchange-newline"><blockquote type="cite">

    <div text="#000000" bgcolor="#FFFFFF">
    <div class="moz-cite-prefix">I dug out what I wrote a year ago ..<br><br>
      eep-draft:<br><a class="moz-txt-link-freetext" href="https://github.com/psyeugenic/eep/blob/egil/system_limits/eeps/eep-00xx.md">https://github.com/psyeugenic/eep/blob/egil/system_limits/eeps/eep-00xx.md</a><br><br>
      Reference implementation:<br><a class="moz-txt-link-freetext" href="https://github.com/psyeugenic/otp/commits/egil/limits-system-gc/OTP-9856">https://github.com/psyeugenic/otp/commits/egil/limits-system-gc/OTP-9856</a><br>
      Remember, this is a prototype and a reference implementation.<br><br>
      There is a couple of issues not addressed or at least open-ended.<br><br>
      * Should processes be able to set limits on other processes? I
      think not though my draft argues for it. It introduces unnecessary
      restraints on erts and hinders performance. 'save_calls' is such
      an option.<br><br>
      * ets - if your table increases beyond some limit. Who should we
      punish? The inserter? The owner? What would be the rationale? We
      cannot just punish the inserter, the ets table is still there
      taking a lot of memory and no other process could insert into the
      table. They would be killed as well. Remove the owner and hence
      the table (and potential heir)? What kind of problems would arise
      then? Limits should be tied into a supervision strategy and
      restart the whole thing.<br><br>
      * In my draft and reference implementation I use soft limits. Once
      a process reaches its limit it will be marked for termination by
      an exit signal. The trouble here is there is no real guarantee for
      how long this will take. A process can continue appending a binary
      for a short while and ending the beam with OOM still. (If I
      remember it correctly you have to schedule out to terminate a
      process in SMP thus you need to bump all reduction. But, not all
      things handle return values from the garbage collector, most
      notably within the append_binary instruction). There may be other
      issues as well.<br><br>
      * Message queues. In the current implementation of message queues
      we have two queues. An inner one which is locked by the receiver
      process while executing and an outer one which other processes
      will use and thus not compete for a message queue lock with the
      executing process. When the inner queue is depleted the receiver
      process will lock the outer queue and move the entire thing to the
      inner one. Rinse and repeat. The only guarantee we have to ensure
      with our implementation is: signal order between two processes.
      So, in the future we might have several queues to improve
      performance. If you introduce monitoring of the total number
      messages in the abstracted queue (all the queues) this will most
      probable kill any sort of scalability. For instance a sender would
      not be allowed to check the inner queue for this reason. Would a
      "fast" counter check in the inner queue be allowed? Perhaps if it
      is fast enough, but any sort of bookkeeping costs performance. If
      we introduce even more queues for scalability reasons this will
      cost even more.<br><br>
      * What about other memory users? Drivers? NIFs?<br><br>
      I do believe in increments in development as long it is path to
      the envisioned goal.<br>
      And to reiterate, i'm not convinced that limits on just processes
      is the way to go. I think a complete monitoring system should be
      envisioned, not just for processes.<br><br>
      // Bj&ouml;rn-Egil<br><br>
      On 2013-02-06 23:03, Richard O'Keefe wrote:<br>
</div>
    <blockquote cite="mid:85819AB4-AD04-401F-B814-88157B71BE07 <at> cs.otago.ac.nz" type="cite">
      Just today, I saw Matthew Evans'

	This pertains to a feature I would like to see
	in Erlang.  The ability to set an optional
	"memory limit" when a process and ETS table is
	created (and maybe a global optional per-process
	limit when the VM is started).  I've seen a few
	cases where, due to software bugs, a process size
	grows and grows; unfortunately as things stand
	today the result is your entire VM crashing -
	hopefully leaving you with a crash_dump. 

	Having such a limit could cause the process to
	terminate (producing a OOM crash report in
	erlang.log) and the crashing process could be
	handled with supervisor rules.  Even better you
	can envisage setting the limits artificially low
	during testing to catch these types of bugs early on.

in my mailbox.  I have seen too many such e-mail messages.
Here's a specific proposal.  It's time _something_ was done
about this kind of problem.  I don't expect that my EEP is
the best way to deal with it, but at least there's going to
be something for people to point to.

      <br><br>_______________________________________________
eeps mailing list
<a class="moz-txt-link-abbreviated" href="mailto:eeps <at> erlang.org">eeps <at> erlang.org</a>
<a class="moz-txt-link-freetext" href="http://erlang.org/mailman/listinfo/eeps">http://erlang.org/mailman/listinfo/eeps</a>

    </blockquote>
    <br>
</div>

_______________________________________________<br>erlang-questions mailing list<br><a href="mailto:erlang-questions <at> erlang.org">erlang-questions <at> erlang.org</a><br>http://erlang.org/mailman/listinfo/erlang-questions<br>
</blockquote>
</div>
<br><div>
<span class="Apple-style-span"><div>
<span class="Apple-style-span">"Installing applications can lead to corruption over time.&nbsp;</span><span class="Apple-style-span">Applications gradually write over each other's libraries, partial upgrades occur, user and system errors happen, and minute changes may be unnoticeable and difficult to fix"</span>
</div>
<div><span class="Apple-style-span"><br></span></div></span><br class="Apple-interchange-newline">
</div>
<br>
</div>
</div>
</div>
</div>
Björn-Egil Dahlberg | 8 Feb 02:23 2013
Picon

Re: [eeps] New EEP: setrlimit(2) analogue for Erlang



2013/2/8 Tony Rogvall <tony <at> rogvall.se>
For what it is worth.
I located my old implementation (three years old :-)

^^
 


A very brief description (I presented some early stuff for OTP around then)
I do not think that max_message_queue_len is implemented,
not really defined. Too many options here.

I forgot about that one =) I wonder if I was present at your presentation. I recall something about safe-erlang with quotas, is that the same thing?

Give process groups allotment of time (reductions) and memory .. well any resource really. That is a bit better then "Don't OOM!"

Hmm .. inheritance .. neat, but I don't know.

Do you have any docs about your thoughts about it?

// Björn-Egil

/Tony

Resource Limits in Erlang
=========================

# Why?

- Ability to detect and kill runaway processes. 
- Detect and kill zombies.
- Basis of safe mode framework.
- Excellent to use in debugging.

# How?

- Limits are checked at context switch and garbage collection time. 
- Relatively light weight.
- Create limits by using spawn_opt.
- Limits are inherited by spawned processes.
- If a limit is reached a signal 'system_limit' is raised.

## max_memory
Limit the amount of memory a process may use. Account for all memory a 
process is using at any given time. This includes heap, stack, tables, 
links and messages. The memory is shared among all processes spawned by the 
process that where limited. spawn\_opt can be used to give 
away some of that memory to child processes.

## max\_time
Controls how many milliseconds a process may use in "wall clock" time.
Created (sub-)processes will only be able to run for the remaining time.

## max\_cpu
Controls how many milliseconds a process may run in cpu time. 
The cpu time is consumed by the process and all it’s spawned (sub-)processes.

## max\_reductions
A lighter version of max\_cpu. One reductions does approximately corresponds
to a function call.

## max\_processes
Limit number of running (sub-)processes that may be running at any given time.

## max\_ports
Limit number of open ports that can be open at any given time.

## max\_tables
Limit number of ets tables that may be open at any given time.

## max\_message\_queue\_len
Limit the size of the message queue. Who dies when the limit is reached?
Either sender or receiver? Maybe add a dangerous block option?

# process\_info

process\_info is used to read the current limits and the
"remaining" quota. 
process_info(Pid, max\_cpu) is used to read the number of
milliseconds set for execution while process_info(Pid, remaining\_cpu)
return how many cpu milliseconds that remain to execute.
The items include: max\_process, max\_ports, max\_tables, max\_memory, 
max\_reductions, max\_message\_queue\_len, max\_cpu, max\_time,
remaining\_process, remaining\_ports, remaining_tables, remaining\_memory, 
remaining\_reductions, remaining\_message\_queue\_len, 
remaining\_cpu, remaining\_time


On 7 feb 2013, at 16:27, Björn-Egil Dahlberg <egil <at> erlang.org> wrote:

I dug out what I wrote a year ago ..

eep-draft:
https://github.com/psyeugenic/eep/blob/egil/system_limits/eeps/eep-00xx.md

Reference implementation:
https://github.com/psyeugenic/otp/commits/egil/limits-system-gc/OTP-9856
Remember, this is a prototype and a reference implementation.

There is a couple of issues not addressed or at least open-ended.

* Should processes be able to set limits on other processes? I think not though my draft argues for it. It introduces unnecessary restraints on erts and hinders performance. 'save_calls' is such an option.

* ets - if your table increases beyond some limit. Who should we punish? The inserter? The owner? What would be the rationale? We cannot just punish the inserter, the ets table is still there taking a lot of memory and no other process could insert into the table. They would be killed as well. Remove the owner and hence the table (and potential heir)? What kind of problems would arise then? Limits should be tied into a supervision strategy and restart the whole thing.

* In my draft and reference implementation I use soft limits. Once a process reaches its limit it will be marked for termination by an exit signal. The trouble here is there is no real guarantee for how long this will take. A process can continue appending a binary for a short while and ending the beam with OOM still. (If I remember it correctly you have to schedule out to terminate a process in SMP thus you need to bump all reduction. But, not all things handle return values from the garbage collector, most notably within the append_binary instruction). There may be other issues as well.

* Message queues. In the current implementation of message queues we have two queues. An inner one which is locked by the receiver process while executing and an outer one which other processes will use and thus not compete for a message queue lock with the executing process. When the inner queue is depleted the receiver process will lock the outer queue and move the entire thing to the inner one. Rinse and repeat. The only guarantee we have to ensure with our implementation is: signal order between two processes. So, in the future we might have several queues to improve performance. If you introduce monitoring of the total number messages in the abstracted queue (all the queues) this will most probable kill any sort of scalability. For instance a sender would not be allowed to check the inner queue for this reason. Would a "fast" counter check in the inner queue be allowed? Perhaps if it is fast enough, but any sort of bookkeeping costs performance. If we introduce even more queues for scalability reasons this will cost even more.

* What about other memory users? Drivers? NIFs?

I do believe in increments in development as long it is path to the envisioned goal.
And to reiterate, i'm not convinced that limits on just processes is the way to go. I think a complete monitoring system should be envisioned, not just for processes.

// Björn-Egil

On 2013-02-06 23:03, Richard O'Keefe wrote:
Just today, I saw Matthew Evans' This pertains to a feature I would like to see in Erlang. The ability to set an optional "memory limit" when a process and ETS table is created (and maybe a global optional per-process limit when the VM is started). I've seen a few cases where, due to software bugs, a process size grows and grows; unfortunately as things stand today the result is your entire VM crashing - hopefully leaving you with a crash_dump. Having such a limit could cause the process to terminate (producing a OOM crash report in erlang.log) and the crashing process could be handled with supervisor rules. Even better you can envisage setting the limits artificially low during testing to catch these types of bugs early on. in my mailbox. I have seen too many such e-mail messages. Here's a specific proposal. It's time _something_ was done about this kind of problem. I don't expect that my EEP is the best way to deal with it, but at least there's going to be something for people to point to.

_______________________________________________ eeps mailing list eeps <at> erlang.org http://erlang.org/mailman/listinfo/eeps

_______________________________________________
erlang-questions mailing list
erlang-questions <at> erlang.org
http://erlang.org/mailman/listinfo/erlang-questions

"Installing applications can lead to corruption over time. Applications gradually write over each other's libraries, partial upgrades occur, user and system errors happen, and minute changes may be unnoticeable and difficult to fix"




_______________________________________________
erlang-questions mailing list
erlang-questions <at> erlang.org
http://erlang.org/mailman/listinfo/erlang-questions


<div>
<br><br><div class="gmail_quote">2013/2/8 Tony Rogvall <span dir="ltr">&lt;<a href="mailto:tony <at> rogvall.se" target="_blank">tony <at> rogvall.se</a>&gt;</span><br><blockquote class="gmail_quote">
<div>For what it is worth.<div>I located my old implementation (three years old :-)</div>
</div>
</blockquote>
<div><br></div>
<div>^^</div>
<div>&nbsp;</div>
<blockquote class="gmail_quote">
<div><div>
<div><br></div>
<div><a href="https://github.com/tonyrog/otp/tree/limits/" target="_blank">https://github.com/tonyrog/otp/tree/limits/</a></div>
<div>
<br><div>A very brief description (I presented some early stuff for OTP around then)</div>
<div>I do not think that max_message_queue_len is implemented,</div>
<div>not really defined. Too many options here.</div>
</div>
</div></div>
</blockquote>
<div><br></div>
<div>I forgot about that one =) I wonder if I was present at your presentation. I recall something about safe-erlang with quotas, is that the same thing?</div>
<div><br></div>
<div>Give process groups allotment of time (reductions) and memory .. well any resource really. That is a bit better then "Don't OOM!"</div>
<div><br></div>
<div>Hmm ..&nbsp;inheritance&nbsp;.. neat, but I don't know.</div>
<div><br></div>
<div>Do you have any docs about your thoughts about it?</div>
<div><br></div>
<div>// Bj&ouml;rn-Egil</div>
<blockquote class="gmail_quote">
<div>
<div><div>
<div><br></div>
<div>/Tony</div>
<div><br></div>
<div>
<div>Resource Limits in Erlang</div>
<div>=========================</div>
<div><br></div>
<div># Why?</div>
<div><br></div>
<div>- Ability to detect and kill runaway processes.&nbsp;</div>
<div>- Detect and kill zombies.</div>
<div>- Basis of safe mode framework.</div>
<div>- Excellent to use in debugging.</div>
<div><br></div>
<div># How?</div>
<div><br></div>
<div>- Limits are checked at context switch and garbage collection time.&nbsp;</div>
<div>- Relatively light weight.</div>
<div>- Create limits by using spawn_opt.</div>
<div>- Limits are inherited by spawned processes.</div>
<div>- If a limit is reached a signal 'system_limit' is raised.</div>
<div><br></div>
<div>## max_memory</div>
<div>Limit the amount of memory a process may use. Account for all memory a&nbsp;</div>
<div>process is using at any given time. This includes heap, stack, tables,&nbsp;</div>
<div>links and messages. The memory is shared among all processes spawned by the&nbsp;</div>
<div>process that where limited. spawn\_opt can be used to give&nbsp;</div>
<div>away some of that memory to child processes.</div>
<div><br></div>
<div>## max\_time</div>
<div>Controls how many milliseconds a process may use in "wall clock" time.</div>
<div>Created (sub-)processes will only be able to run for the remaining time.</div>
<div><br></div>
<div>## max\_cpu</div>
<div>Controls how many milliseconds a process may run in cpu time.&nbsp;</div>
<div>The cpu time is consumed by the process and all it&rsquo;s spawned (sub-)processes.</div>
<div><br></div>
<div>## max\_reductions</div>
<div>A lighter version of max\_cpu. One reductions does approximately corresponds</div>
<div>to a function call.</div>
<div><br></div>
<div>## max\_processes</div>
<div>Limit number of running (sub-)processes that may be running at any given time.</div>
<div><br></div>
<div>## max\_ports</div>
<div>Limit number of open ports that can be open at any given time.</div>
<div><br></div>
<div>## max\_tables</div>
<div>Limit number of ets tables that may be open at any given time.</div>
<div><br></div>
<div>## max\_message\_queue\_len</div>
<div>Limit the size of the message queue. Who dies when the limit is reached?</div>
<div>Either sender or receiver? Maybe add a dangerous block option?</div>
<div><br></div>
<div># process\_info</div>
<div><br></div>
<div>process\_info is used to read the current limits and the</div>
<div>"remaining" quota.&nbsp;</div>
<div>process_info(Pid, max\_cpu) is used to read the number of</div>
<div>
milliseconds set for execution while process_info(Pid, remaining\_cpu)</div>
<div>return how many cpu milliseconds that remain to execute.</div>
<div>The items include: max\_process, max\_ports, max\_tables, max\_memory,&nbsp;</div>
<div>max\_reductions, max\_message\_queue\_len, max\_cpu, max\_time,</div>
<div>remaining\_process, remaining\_ports, remaining_tables, remaining\_memory,&nbsp;</div>
<div>remaining\_reductions, remaining\_message\_queue\_len,&nbsp;</div>
<div>remaining\_cpu, remaining\_time</div>
</div>
<div><br></div>
<div>
<br><div>
<div><div class="h5">
<div>On 7 feb 2013, at 16:27, Bj&ouml;rn-Egil Dahlberg &lt;<a href="mailto:egil <at> erlang.org" target="_blank">egil <at> erlang.org</a>&gt; wrote:</div>
<br>
</div></div>
<blockquote type="cite">
<div><div class="h5">

    

  <div text="#000000" bgcolor="#FFFFFF">
    <div>I dug out what I wrote a year ago ..<br><br>
      eep-draft:<br><a href="https://github.com/psyeugenic/eep/blob/egil/system_limits/eeps/eep-00xx.md" target="_blank">https://github.com/psyeugenic/eep/blob/egil/system_limits/eeps/eep-00xx.md</a><br><br>
      Reference implementation:<br><a href="https://github.com/psyeugenic/otp/commits/egil/limits-system-gc/OTP-9856" target="_blank">https://github.com/psyeugenic/otp/commits/egil/limits-system-gc/OTP-9856</a><br>
      Remember, this is a prototype and a reference implementation.<br><br>
      There is a couple of issues not addressed or at least open-ended.<br><br>
      * Should processes be able to set limits on other processes? I
      think not though my draft argues for it. It introduces unnecessary
      restraints on erts and hinders performance. 'save_calls' is such
      an option.<br><br>
      * ets - if your table increases beyond some limit. Who should we
      punish? The inserter? The owner? What would be the rationale? We
      cannot just punish the inserter, the ets table is still there
      taking a lot of memory and no other process could insert into the
      table. They would be killed as well. Remove the owner and hence
      the table (and potential heir)? What kind of problems would arise
      then? Limits should be tied into a supervision strategy and
      restart the whole thing.<br><br>
      * In my draft and reference implementation I use soft limits. Once
      a process reaches its limit it will be marked for termination by
      an exit signal. The trouble here is there is no real guarantee for
      how long this will take. A process can continue appending a binary
      for a short while and ending the beam with OOM still. (If I
      remember it correctly you have to schedule out to terminate a
      process in SMP thus you need to bump all reduction. But, not all
      things handle return values from the garbage collector, most
      notably within the append_binary instruction). There may be other
      issues as well.<br><br>
      * Message queues. In the current implementation of message queues
      we have two queues. An inner one which is locked by the receiver
      process while executing and an outer one which other processes
      will use and thus not compete for a message queue lock with the
      executing process. When the inner queue is depleted the receiver
      process will lock the outer queue and move the entire thing to the
      inner one. Rinse and repeat. The only guarantee we have to ensure
      with our implementation is: signal order between two processes.
      So, in the future we might have several queues to improve
      performance. If you introduce monitoring of the total number
      messages in the abstracted queue (all the queues) this will most
      probable kill any sort of scalability. For instance a sender would
      not be allowed to check the inner queue for this reason. Would a
      "fast" counter check in the inner queue be allowed? Perhaps if it
      is fast enough, but any sort of bookkeeping costs performance. If
      we introduce even more queues for scalability reasons this will
      cost even more.<br><br>
      * What about other memory users? Drivers? NIFs?<br><br>
      I do believe in increments in development as long it is path to
      the envisioned goal.<br>
      And to reiterate, i'm not convinced that limits on just processes
      is the way to go. I think a complete monitoring system should be
      envisioned, not just for processes.<br><br>
      // Bj&ouml;rn-Egil<br><br>
      On 2013-02-06 23:03, Richard O'Keefe wrote:<br>
</div>
    <blockquote type="cite">
      Just today, I saw Matthew Evans'

	This pertains to a feature I would like to see
	in Erlang.  The ability to set an optional
	"memory limit" when a process and ETS table is
	created (and maybe a global optional per-process
	limit when the VM is started).  I've seen a few
	cases where, due to software bugs, a process size
	grows and grows; unfortunately as things stand
	today the result is your entire VM crashing -
	hopefully leaving you with a crash_dump. 

	Having such a limit could cause the process to
	terminate (producing a OOM crash report in
	erlang.log) and the crashing process could be
	handled with supervisor rules.  Even better you
	can envisage setting the limits artificially low
	during testing to catch these types of bugs early on.

in my mailbox.  I have seen too many such e-mail messages.
Here's a specific proposal.  It's time _something_ was done
about this kind of problem.  I don't expect that my EEP is
the best way to deal with it, but at least there's going to
be something for people to point to.

      <br><br>_______________________________________________
eeps mailing list
<a href="mailto:eeps <at> erlang.org" target="_blank">eeps <at> erlang.org</a>
<a href="http://erlang.org/mailman/listinfo/eeps" target="_blank">http://erlang.org/mailman/listinfo/eeps</a>

    </blockquote>
    <br>
</div>
</div></div>
<div class="im">

_______________________________________________<br>erlang-questions mailing list<br><a href="mailto:erlang-questions <at> erlang.org" target="_blank">erlang-questions <at> erlang.org</a><br><a href="http://erlang.org/mailman/listinfo/erlang-questions" target="_blank">http://erlang.org/mailman/listinfo/erlang-questions</a><br>
</div>
</blockquote>
</div>
<br><div>
<span><div>
<span>"Installing applications can lead to corruption over time.&nbsp;</span><span>Applications gradually write over each other's libraries, partial upgrades occur, user and system errors happen, and minute changes may be unnoticeable and difficult to fix"</span>
</div>
<div><span><br></span></div></span><br>
</div>
<br>
</div>
</div></div>
</div>
<br>_______________________________________________<br>
erlang-questions mailing list<br><a href="mailto:erlang-questions <at> erlang.org">erlang-questions <at> erlang.org</a><br><a href="http://erlang.org/mailman/listinfo/erlang-questions" target="_blank">http://erlang.org/mailman/listinfo/erlang-questions</a><br><br>
</blockquote>
</div>
<br>
</div>
ok | 8 Feb 03:39 2013
Picon

Re: [eeps] New EEP: setrlimit(2) analogue for Erlang

>
> Hmm .. inheritance .. neat, but I don't know.

It's not clear how much sense inheritance makes.

There have been concurrent systems in which a process is
born with some share of system resources and can portion
its share out amongst its children, so that a bound on
memory or CPU time or whatever really _is_ a bound on the
total resource consumption on and on behalf of that activity.
Then you run into issues like having a dead child return
unused resources to its parent's account.  This protects
against malice.

If inheritance is just inheritance, then a limited process
can consume unbounded amounts of memory by creating lots of
children.  This protects against accident, sort of, but has
no protection against malice.

Michael Truog | 8 Feb 03:01 2013
Picon

Re: [eeps] New EEP: setrlimit(2) analogue for Erlang

An idea which seems like it should be included in this EEP, is simply a function to determine how much real memory is consumed by an erlang variable.  I understand we have erlang:external_size which provides the size based on the external term format.  I also understand that the memory pools make erlang consume more than the size that would be returned.  There is some documentation on memory consumption of erlang data here: http://www.erlang.org/doc/efficiency_guide/advanced.html#id68921 .  I attempted to use an older version of this documentation to write my own function, but ran into problems (created output that seemed incorrect) and taking into account HiPE data is a separate part that would generate different results.

So, these erlang limits are great additions, but it seems like we would need a function for understanding where the limits are broken, for debugging and development (and possibly dynamic runtime changes).  It would be interesting if the function was allowed in guard tests, which might help for embedded development.

Thanks,
Michael

On 02/07/2013 07:27 AM, Björn-Egil Dahlberg wrote:
I dug out what I wrote a year ago ..

eep-draft:
https://github.com/psyeugenic/eep/blob/egil/system_limits/eeps/eep-00xx.md

Reference implementation:
https://github.com/psyeugenic/otp/commits/egil/limits-system-gc/OTP-9856
Remember, this is a prototype and a reference implementation.

There is a couple of issues not addressed or at least open-ended.

* Should processes be able to set limits on other processes? I think not though my draft argues for it. It introduces unnecessary restraints on erts and hinders performance. 'save_calls' is such an option.

* ets - if your table increases beyond some limit. Who should we punish? The inserter? The owner? What would be the rationale? We cannot just punish the inserter, the ets table is still there taking a lot of memory and no other process could insert into the table. They would be killed as well. Remove the owner and hence the table (and potential heir)? What kind of problems would arise then? Limits should be tied into a supervision strategy and restart the whole thing.

* In my draft and reference implementation I use soft limits. Once a process reaches its limit it will be marked for termination by an exit signal. The trouble here is there is no real guarantee for how long this will take. A process can continue appending a binary for a short while and ending the beam with OOM still. (If I remember it correctly you have to schedule out to terminate a process in SMP thus you need to bump all reduction. But, not all things handle return values from the garbage collector, most notably within the append_binary instruction). There may be other issues as well.

* Message queues. In the current implementation of message queues we have two queues. An inner one which is locked by the receiver process while executing and an outer one which other processes will use and thus not compete for a message queue lock with the executing process. When the inner queue is depleted the receiver process will lock the outer queue and move the entire thing to the inner one. Rinse and repeat. The only guarantee we have to ensure with our implementation is: signal order between two processes. So, in the future we might have several queues to improve performance. If you introduce monitoring of the total number messages in the abstracted queue (all the queues) this will most probable kill any sort of scalability. For instance a sender would not be allowed to check the inner queue for this reason. Would a "fast" counter check in the inner queue be allowed? Perhaps if it is fast enough, but any sort of bookkeeping costs performance. If we introduce even more queues for scalability reasons this will cost even more.

* What about other memory users? Drivers? NIFs?

I do believe in increments in development as long it is path to the envisioned goal.
And to reiterate, i'm not convinced that limits on just processes is the way to go. I think a complete monitoring system should be envisioned, not just for processes.

// Björn-Egil

On 2013-02-06 23:03, Richard O'Keefe wrote:
Just today, I saw Matthew Evans' This pertains to a feature I would like to see in Erlang. The ability to set an optional "memory limit" when a process and ETS table is created (and maybe a global optional per-process limit when the VM is started). I've seen a few cases where, due to software bugs, a process size grows and grows; unfortunately as things stand today the result is your entire VM crashing - hopefully leaving you with a crash_dump. Having such a limit could cause the process to terminate (producing a OOM crash report in erlang.log) and the crashing process could be handled with supervisor rules. Even better you can envisage setting the limits artificially low during testing to catch these types of bugs early on. in my mailbox. I have seen too many such e-mail messages. Here's a specific proposal. It's time _something_ was done about this kind of problem. I don't expect that my EEP is the best way to deal with it, but at least there's going to be something for people to point to.

_______________________________________________ eeps mailing list eeps <at> erlang.org http://erlang.org/mailman/listinfo/eeps



_______________________________________________ eeps mailing list eeps <at> erlang.org http://erlang.org/mailman/listinfo/eeps

<div>
    <div class="moz-cite-prefix">An idea which seems like it should be
      included in this EEP, is simply a function to determine how much
      real memory is consumed by an erlang variable.&nbsp; I understand we
      have <span class="bold_code">erlang:external_size which provides
        the size based on the external term format.&nbsp; I also understand
        that the memory pools make erlang consume more than the size
        that would be returned.&nbsp; There is some documentation on memory
        consumption of erlang data here: </span><a class="moz-txt-link-freetext" href="http://www.erlang.org/doc/efficiency_guide/advanced.html#id68921">http://www.erlang.org/doc/efficiency_guide/advanced.html#id68921</a>
      .&nbsp; I attempted to use an older version of this documentation to
      write my own function, but ran into problems (created output that
      seemed incorrect) and taking into account HiPE data is a separate
      part that would generate different results.<br><br>
      So, these erlang limits are great additions, but it seems like we
      would need a function for understanding where the limits are
      broken, for debugging and development (and possibly dynamic
      runtime changes).&nbsp; It would be interesting if the function was
      allowed in guard tests, which might help for embedded development.<br><br>
      Thanks,<br>
      Michael<br><br>
      On 02/07/2013 07:27 AM, Bj&ouml;rn-Egil Dahlberg wrote:<br>
</div>
    <blockquote cite="mid:5113C7F5.9050500 <at> erlang.org" type="cite">
      <div class="moz-cite-prefix">I dug out what I wrote a year ago ..<br><br>
        eep-draft:<br><a moz-do-not-send="true" class="moz-txt-link-freetext" href="https://github.com/psyeugenic/eep/blob/egil/system_limits/eeps/eep-00xx.md">https://github.com/psyeugenic/eep/blob/egil/system_limits/eeps/eep-00xx.md</a><br><br>
        Reference implementation:<br><a moz-do-not-send="true" class="moz-txt-link-freetext" href="https://github.com/psyeugenic/otp/commits/egil/limits-system-gc/OTP-9856">https://github.com/psyeugenic/otp/commits/egil/limits-system-gc/OTP-9856</a><br>
        Remember, this is a prototype and a reference implementation.<br><br>
        There is a couple of issues not addressed or at least
        open-ended.<br><br>
        * Should processes be able to set limits on other processes? I
        think not though my draft argues for it. It introduces
        unnecessary restraints on erts and hinders performance.
        'save_calls' is such an option.<br><br>
        * ets - if your table increases beyond some limit. Who should we
        punish? The inserter? The owner? What would be the rationale? We
        cannot just punish the inserter, the ets table is still there
        taking a lot of memory and no other process could insert into
        the table. They would be killed as well. Remove the owner and
        hence the table (and potential heir)? What kind of problems
        would arise then? Limits should be tied into a supervision
        strategy and restart the whole thing.<br><br>
        * In my draft and reference implementation I use soft limits.
        Once a process reaches its limit it will be marked for
        termination by an exit signal. The trouble here is there is no
        real guarantee for how long this will take. A process can
        continue appending a binary for a short while and ending the
        beam with OOM still. (If I remember it correctly you have to
        schedule out to terminate a process in SMP thus you need to bump
        all reduction. But, not all things handle return values from the
        garbage collector, most notably within the append_binary
        instruction). There may be other issues as well.<br><br>
        * Message queues. In the current implementation of message
        queues we have two queues. An inner one which is locked by the
        receiver process while executing and an outer one which other
        processes will use and thus not compete for a message queue lock
        with the executing process. When the inner queue is depleted the
        receiver process will lock the outer queue and move the entire
        thing to the inner one. Rinse and repeat. The only guarantee we
        have to ensure with our implementation is: signal order between
        two processes. So, in the future we might have several queues to
        improve performance. If you introduce monitoring of the total
        number messages in the abstracted queue (all the queues) this
        will most probable kill any sort of scalability. For instance a
        sender would not be allowed to check the inner queue for this
        reason. Would a "fast" counter check in the inner queue be
        allowed? Perhaps if it is fast enough, but any sort of
        bookkeeping costs performance. If we introduce even more queues
        for scalability reasons this will cost even more.<br><br>
        * What about other memory users? Drivers? NIFs?<br><br>
        I do believe in increments in development as long it is path to
        the envisioned goal.<br>
        And to reiterate, i'm not convinced that limits on just
        processes is the way to go. I think a complete monitoring system
        should be envisioned, not just for processes.<br><br>
        // Bj&ouml;rn-Egil<br><br>
        On 2013-02-06 23:03, Richard O'Keefe wrote:<br>
</div>
      <blockquote cite="mid:85819AB4-AD04-401F-B814-88157B71BE07 <at> cs.otago.ac.nz" type="cite">
        Just today, I saw Matthew Evans'

	This pertains to a feature I would like to see
	in Erlang.  The ability to set an optional
	"memory limit" when a process and ETS table is
	created (and maybe a global optional per-process
	limit when the VM is started).  I've seen a few
	cases where, due to software bugs, a process size
	grows and grows; unfortunately as things stand
	today the result is your entire VM crashing -
	hopefully leaving you with a crash_dump. 

	Having such a limit could cause the process to
	terminate (producing a OOM crash report in
	erlang.log) and the crashing process could be
	handled with supervisor rules.  Even better you
	can envisage setting the limits artificially low
	during testing to catch these types of bugs early on.

in my mailbox.  I have seen too many such e-mail messages.
Here's a specific proposal.  It's time _something_ was done
about this kind of problem.  I don't expect that my EEP is
the best way to deal with it, but at least there's going to
be something for people to point to.

        <br><br>_______________________________________________
eeps mailing list
<a moz-do-not-send="true" class="moz-txt-link-abbreviated" href="mailto:eeps <at> erlang.org">eeps <at> erlang.org</a>
<a moz-do-not-send="true" class="moz-txt-link-freetext" href="http://erlang.org/mailman/listinfo/eeps">http://erlang.org/mailman/listinfo/eeps</a>

      </blockquote>
      <br><br><br>_______________________________________________
eeps mailing list
<a class="moz-txt-link-abbreviated" href="mailto:eeps <at> erlang.org">eeps <at> erlang.org</a>
<a class="moz-txt-link-freetext" href="http://erlang.org/mailman/listinfo/eeps">http://erlang.org/mailman/listinfo/eeps</a>

    </blockquote>
    <br>
</div>
ok | 8 Feb 04:04 2013
Picon

Re: [eeps] New EEP: setrlimit(2) analogue for Erlang

> An idea which seems like it should be included in this EEP, is simply a
> function to determine how much real memory is consumed by an erlang
> variable.

Given a language with immutable data structures and free sharing,
I am having a hard time trying to think what this might mean.
Consider as one of the *easy* cases

    L0 = [],
    L1 = [L0|L0],
    ...
    L99 = [L98|L98]

The amount of memory it really uses is just 99 cons cells,
presumably 198 words.  The amount that will be calculated
by a recursive sizer is about 2^99 (approximately).

There are at least the following notions:
 - size in external representation
 - amount copied when sent to a local PID
 - amount found by a recursive walker
 - amount used taking internal cycles into account
 - fair share considering all references in the current PID
 - fair share including references to binaries from other PIDs.

It seems to me that there is one thing that _could_ be done
but would be a fair bit of work, and that's allocation
profiling.  Arrange to generate different BEAM code which
records each (amount of memory, source location where
allocation happens) so that you can run a test case and see
which expressions are responsible for allocating how much
memory.

Björn-Egil Dahlberg | 8 Feb 04:27 2013
Picon

Re: [eeps] New EEP: setrlimit(2) analogue for Erlang



2013/2/8 <ok <at> cs.otago.ac.nz>
> An idea which seems like it should be included in this EEP, is simply a
> function to determine how much real memory is consumed by an erlang
> variable.

Given a language with immutable data structures and free sharing,
I am having a hard time trying to think what this might mean.
Consider as one of the *easy* cases

    L0 = [],
    L1 = [L0|L0],
    ...
    L99 = [L98|L98]

The amount of memory it really uses is just 99 cons cells,
presumably 198 words.  The amount that will be calculated
by a recursive sizer is about 2^99 (approximately).

Btw, If its just the matter of memory then there exists erts_debug:size/1 and erts_debug:flat_size/1 (calculates number of words):

Eshell V5.9  (abort with ^G)
1> A = {1,2,3}.
{1,2,3}
2> B = [A|A].
[{1,2,3}|{1,2,3}]
3> erts_debug:size(B).
6
4> erts_debug:flat_size(B).
10

 

There are at least the following notions:
 - size in external representation
 - amount copied when sent to a local PID
 - amount found by a recursive walker
 - amount used taking internal cycles into account
 - fair share considering all references in the current PID
 - fair share including references to binaries from other PIDs.

It seems to me that there is one thing that _could_ be done
but would be a fair bit of work, and that's allocation
profiling.  Arrange to generate different BEAM code which
records each (amount of memory, source location where
allocation happens) so that you can run a test case and see
which expressions are responsible for allocating how much
memory.


_______________________________________________
erlang-questions mailing list
erlang-questions <at> erlang.org
http://erlang.org/mailman/listinfo/erlang-questions

<div>
<br><br><div class="gmail_quote">2013/2/8  <span dir="ltr">&lt;<a href="mailto:ok <at> cs.otago.ac.nz" target="_blank">ok <at> cs.otago.ac.nz</a>&gt;</span><br><blockquote class="gmail_quote">
<div class="im">&gt; An idea which seems like it should be included in this EEP, is simply a<br>
&gt; function to determine how much real memory is consumed by an erlang<br>
&gt; variable.<br><br>
</div>Given a language with immutable data structures and free sharing,<br>
I am having a hard time trying to think what this might mean.<br>
Consider as one of the *easy* cases<br><br>
&nbsp; &nbsp; L0 = [],<br>
&nbsp; &nbsp; L1 = [L0|L0],<br>
&nbsp; &nbsp; ...<br>
&nbsp; &nbsp; L99 = [L98|L98]<br><br>
The amount of memory it really uses is just 99 cons cells,<br>
presumably 198 words. &nbsp;The amount that will be calculated<br>
by a recursive sizer is about 2^99 (approximately).<br>
</blockquote>
<div><br></div>
<div>Btw, If its just the matter of memory then there exists erts_debug:size/1 and erts_debug:flat_size/1 (calculates number of words):</div>
<div><br></div>
<div>
<div>Eshell V5.9 &nbsp;(abort with ^G)</div>
<div>1&gt; A = {1,2,3}.</div>
<div>{1,2,3}</div>
<div>2&gt; B = [A|A].</div>
<div>[{1,2,3}|{1,2,3}]</div>
<div>3&gt; erts_debug:size(B).</div>
<div>
6</div>
<div>4&gt; erts_debug:flat_size(B).</div>
<div>10</div>
</div>
<div><br></div>
<div>&nbsp;</div>
<blockquote class="gmail_quote">
<br>
There are at least the following notions:<br>
&nbsp;- size in external representation<br>
&nbsp;- amount copied when sent to a local PID<br>
&nbsp;- amount found by a recursive walker<br>
&nbsp;- amount used taking internal cycles into account<br>
&nbsp;- fair share considering all references in the current PID<br>
&nbsp;- fair share including references to binaries from other PIDs.<br><br>
It seems to me that there is one thing that _could_ be done<br>
but would be a fair bit of work, and that's allocation<br>
profiling. &nbsp;Arrange to generate different BEAM code which<br>
records each (amount of memory, source location where<br>
allocation happens) so that you can run a test case and see<br>
which expressions are responsible for allocating how much<br>
memory.<br><div class="HOEnZb"><div class="h5">
<br><br>
_______________________________________________<br>
erlang-questions mailing list<br><a href="mailto:erlang-questions <at> erlang.org">erlang-questions <at> erlang.org</a><br><a href="http://erlang.org/mailman/listinfo/erlang-questions" target="_blank">http://erlang.org/mailman/listinfo/erlang-questions</a><br>
</div></div>
</blockquote>
</div>
<br>
</div>
Michael Truog | 8 Feb 04:36 2013
Picon

Re: [eeps] New EEP: setrlimit(2) analogue for Erlang

On 02/07/2013 07:27 PM, Björn-Egil Dahlberg wrote:


2013/2/8 <ok <at> cs.otago.ac.nz>
> An idea which seems like it should be included in this EEP, is simply a
> function to determine how much real memory is consumed by an erlang
> variable.

Given a language with immutable data structures and free sharing,
I am having a hard time trying to think what this might mean.
Consider as one of the *easy* cases

    L0 = [],
    L1 = [L0|L0],
    ...
    L99 = [L98|L98]

The amount of memory it really uses is just 99 cons cells,
presumably 198 words.  The amount that will be calculated
by a recursive sizer is about 2^99 (approximately).

Btw, If its just the matter of memory then there exists erts_debug:size/1 and erts_debug:flat_size/1 (calculates number of words):

Eshell V5.9  (abort with ^G)
1> A = {1,2,3}.
{1,2,3}
2> B = [A|A].
[{1,2,3}|{1,2,3}]
3> erts_debug:size(B).
6
4> erts_debug:flat_size(B).
10

That is what I was looking for.  Thanks!


<div>
    <div class="moz-cite-prefix">On 02/07/2013 07:27 PM, Bj&ouml;rn-Egil
      Dahlberg wrote:<br>
</div>
    <blockquote cite="mid:CAMjYFoOsvWB6mw9u6oYi6YgaCNs5_p1R3ZaG5MQwRqTGJxx8pg <at> mail.gmail.com" type="cite">
      <br><br><div class="gmail_quote">2013/2/8 <span dir="ltr">&lt;<a moz-do-not-send="true" href="mailto:ok <at> cs.otago.ac.nz" target="_blank">ok <at> cs.otago.ac.nz</a>&gt;</span><br><blockquote class="gmail_quote">
          <div class="im">&gt; An idea which seems like it should be
            included in this EEP, is simply a<br>
            &gt; function to determine how much real memory is consumed
            by an erlang<br>
            &gt; variable.<br><br>
</div>
          Given a language with immutable data structures and free
          sharing,<br>
          I am having a hard time trying to think what this might mean.<br>
          Consider as one of the *easy* cases<br><br>
          &nbsp; &nbsp; L0 = [],<br>
          &nbsp; &nbsp; L1 = [L0|L0],<br>
          &nbsp; &nbsp; ...<br>
          &nbsp; &nbsp; L99 = [L98|L98]<br><br>
          The amount of memory it really uses is just 99 cons cells,<br>
          presumably 198 words. &nbsp;The amount that will be calculated<br>
          by a recursive sizer is about 2^99 (approximately).<br>
</blockquote>
        <div><br></div>
        <div>Btw, If its just the matter of memory then there exists
          erts_debug:size/1 and erts_debug:flat_size/1 (calculates
          number of words):</div>
        <div><br></div>
        <div>
          <div>Eshell V5.9 &nbsp;(abort with ^G)</div>
          <div>1&gt; A = {1,2,3}.</div>
          <div>{1,2,3}</div>
          <div>2&gt; B = [A|A].</div>
          <div>[{1,2,3}|{1,2,3}]</div>
          <div>3&gt; erts_debug:size(B).</div>
          <div>
            6</div>
          <div>4&gt; erts_debug:flat_size(B).</div>
          <div>10</div>
        </div>
        <div><br></div>
      </div>
    </blockquote>
    That is what I was looking for.&nbsp; Thanks!<br><br><br>
</div>
Michael Truog | 8 Feb 04:47 2013
Picon

Re: [eeps] New EEP: setrlimit(2) analogue for Erlang

On 02/07/2013 07:04 PM, ok <at> cs.otago.ac.nz wrote:
>> An idea which seems like it should be included in this EEP, is simply a
>> function to determine how much real memory is consumed by an erlang
>> variable.
> Given a language with immutable data structures and free sharing,
> I am having a hard time trying to think what this might mean.
> Consider as one of the *easy* cases
>
>     L0 = [],
>     L1 = [L0|L0],
>     ...
>     L99 = [L98|L98]
>
> The amount of memory it really uses is just 99 cons cells,
> presumably 198 words.  The amount that will be calculated
> by a recursive sizer is about 2^99 (approximately).
>
> There are at least the following notions:
>  - size in external representation
>  - amount copied when sent to a local PID
>  - amount found by a recursive walker
>  - amount used taking internal cycles into account
>  - fair share considering all references in the current PID
>  - fair share including references to binaries from other PIDs.
>
> It seems to me that there is one thing that _could_ be done
> but would be a fair bit of work, and that's allocation
> profiling.  Arrange to generate different BEAM code which
> records each (amount of memory, source location where
> allocation happens) so that you can run a test case and see
> which expressions are responsible for allocating how much
> memory.
The erts_debug:size/1 that Björn-Egil mentioned is really what I want, but we would need it to be
documented to have this low-level view of memory consumption be official (it would also be cool if it could
be mentioned in the memory portion of the efficiency guide).

I know the view of the allocators with profiling can be done with instrument and erts_alloc_config, which
provides a high-level view of memory usage by processes and allocators.  So, that part seems fine, though
it is pretty complex.  Having a function like erts_debug:size/1 can help the developer avoid all the
allocation complexity, and just focus on potential memory problems when errors occur, which seems like
the next step after this EEP would be implemented.

Richard Carlsson | 8 Feb 14:18 2013
Picon

Re: [eeps] New EEP: setrlimit(2) analogue for Erlang

On 02/08/2013 04:47 AM, Michael Truog wrote:
> The erts_debug:size/1 that Björn-Egil mentioned is really what I want, but we would need it to be
documented to have this low-level view of memory consumption be official (it would also be cool if it could
be mentioned in the memory portion of the efficiency guide).

Also, it's currently very slow on large structures, being implemented in 
Erlang (using gb_trees and debugging BIFs like erts_debug:same/2). Do 
not use in production if the structures may be huge.

The erts_debug:flat_size/1 function is fast and implemented in C, but 
doesn't give you the true size since it ignores sharing. If Kostis 
Sagonas' work on sharing-preserving copying operations was included in 
OTP, we could have an efficient version of erts_debug:size/1.

    /Richard

Björn-Egil Dahlberg | 8 Feb 14:29 2013

Re: [eeps] New EEP: setrlimit(2) analogue for Erlang

On 2013-02-08 14:18, Richard Carlsson wrote:
> On 02/08/2013 04:47 AM, Michael Truog wrote:
>> The erts_debug:size/1 that Björn-Egil mentioned is really what I 
>> want, but we would need it to be documented to have this low-level 
>> view of memory consumption be official (it would also be cool if it 
>> could be mentioned in the memory portion of the efficiency guide).
>
> Also, it's currently very slow on large structures, being implemented 
> in Erlang (using gb_trees and debugging BIFs like erts_debug:same/2). 
> Do not use in production if the structures may be huge.
Good point! I would say don't use it in production *period*. As the name 
suggests it's used for debugging/development purposes.

>
> The erts_debug:flat_size/1 function is fast and implemented in C, but 
> doesn't give you the true size since it ignores sharing. If Kostis 
> Sagonas' work on sharing-preserving copying operations was included in 
> OTP, we could have an efficient version of erts_debug:size/1.

I haven't seen the final implementation yet. I wonder if it's complete? 
Preserved sharing in copy would be really nice as long as it is efficient.

// Björn-Egil
Kostis Sagonas | 11 Feb 22:47 2013
Picon

Re: [eeps] New EEP: setrlimit(2) analogue for Erlang

On 02/08/2013 02:29 PM, Björn-Egil Dahlberg wrote:
>
>>
>> The erts_debug:flat_size/1 function is fast and implemented in C, but
>> doesn't give you the true size since it ignores sharing. If Kostis
>> Sagonas' work on sharing-preserving copying operations was included in
>> OTP, we could have an efficient version of erts_debug:size/1.
>
> I haven't seen the final implementation yet. I wonder if it's complete?
> Preserved sharing in copy would be really nice as long as it is efficient.

First of all, I would like to correct Richard a bit. Although I am very 
heavily involved in the implementation of an Erlang VM that preserves 
term sharing, most of the work so far has been performed by my very good 
colleague Nikos Papaspyrou.  A paper we wrote about this work has been 
published in the 2012 ACM Erlang Workshop and can be accessed at:

   http://dx.doi.org/10.1145/2364489.2364493

or more conveniently at:

   http://user.it.uu.se/~kostis/Papers/erlang12_sharing.pdf

Although its contents are quite technical, I strongly recommend reading 
at least its introduction and Section 2 which explains all you wanted to 
know about term sharing in Erlang/OTP.

Now, regarding the actual implementation it is also publicly available 
and can be found at:

   https://github.com/nickie/otp

It's based on vanilla R16A but adds a --enable-sharing-preserving 
configure flag that can be optionally enabled to give a VM that 
preserves term sharing in intra-node process spawns and in message passing.

As far as we know it works at well as the default system and its 
overhead is extremely small in all things we have tried. (Of course, it 
also gives quite big/unbounded? speedups whenever a lot of shared terms 
are copied between processes.)

We have discussed with the OTP team the possibility of the system being 
included in R16B, so that the community has a chance to try and evaluate 
it, but no decision has been reached so far.

The only remaining issue we are aware of has to do with code unloading: 
since terms in the constant pool of a module are not copied, as the 
vanilla system does, the code loader has to either preserve these terms 
or copy them on demand when a module is purged/reloaded. The OTP team 
has promised to look into this at some point but this work has not been 
prioritized yet. But the whole system should be usable without problems 
for all applications where unloading of modules with constants does not 
take place.

Note that the implementation is restricted to intra-node term sharing: 
i.e. it does yet not support term sharing when terms are sent to 
different nodes. This is something we are currently working on and will 
most likely require the design of a new extern term format. But this is 
an orthogonal issue.

As a relatively minor point we have not tested this on Windows; though 
we suspect that there are only minor issues to possibly address there. 
Somebody with a Windows development environment should do that (hint, 
hint...).

Kostis
Richard Carlsson | 12 Feb 10:45 2013
Picon

Re: [eeps] New EEP: setrlimit(2) analogue for Erlang

On 2013-02-11 22:47, Kostis Sagonas wrote:
> The only remaining issue we are aware of has to do with code unloading:
> since terms in the constant pool of a module are not copied, as the
> vanilla system does, the code loader has to either preserve these terms
> or copy them on demand when a module is purged/reloaded. The OTP team
> has promised to look into this at some point but this work has not been
> prioritized yet. But the whole system should be usable without problems
> for all applications where unloading of modules with constants does not
> take place.

While trying to figure out the much harder problem of garbage collecting 
globally shared constant pool data, couldn't you modify your copying 
routine to just make an old-style non-sharing-preserving copy of the 
subterms that belong to the constant pool?

Since you already check whether a pointer points to the local heap (the 
common case), it should be cheap to add some extra check to the case of 
pointers outside the heap. Possibly, you don't even have to do any extra 
checking, just default to the old-style copying routines for all 
off-heap subterms. This doesn't preserve sharing within constant terms 
in modules when sent as messages, but at least would enable 
sharing-preserving copying for all dynamic data. Or am I missing something?

    /Richard


Gmane