shingo | 1 Dec 21:58 2011

SocketTimeoutException: Read timed out - while delete

I used mongo-java-driver to operate mongodb(version 2.7.2)  in my
project.

More than 100 threads connect to the mongodb server  simultaneously
and each thread has an insert and an a delete .
It is strange that all inserts succeed but a small quantity of deletes
failed.

I tried many times , got the same situation.
At last,  I add the  WriteConcern.SAFE on the delete method, then get
the exception belows.

Caused by: java.net.SocketTimeoutException: Read timed out
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.read(SocketInputStream.java:129)
        at java.io.BufferedInputStream.fill(BufferedInputStream.java:
218)
        at java.io.BufferedInputStream.read1(BufferedInputStream.java:
258)
        at java.io.BufferedInputStream.read(BufferedInputStream.java:
317)
        at org.bson.io.Bits.readFully(Bits.java:35)
        at org.bson.io.Bits.readFully(Bits.java:28)
        at com.mongodb.Response.<init>(Response.java:39)
        at com.mongodb.DBPort.go(DBPort.java:123)
        at com.mongodb.DBPort.go(DBPort.java:82)
        at com.mongodb.DBPort.findOne(DBPort.java:142)
        at com.mongodb.DBPort.runCommand(DBPort.java:151)
        at
com.mongodb.DBTCPConnector._checkWriteError(DBTCPConnector.java:121)
(Continue reading)

Antoine Girbal | 2 Dec 01:04 2011

Re: SocketTimeoutException: Read timed out - while delete

it looks like you set a timeout value on the socket, what is it set
to?
If you are doing a high volume of insert/delete, things can get slow
if the server has to use the disk, and queries will timeout as a
consequence.
Do you monitor the load on the server (cpu, load, iostat, etc)?
Also you can look at the mongod log for slow queries.

On Dec 1, 12:58 pm, shingo <6623662...@...> wrote:
> I used mongo-java-driver to operate mongodb(version 2.7.2)  in my
> project.
>
> More than 100 threads connect to the mongodb server  simultaneously
> and each thread has an insert and an a delete .
> It is strange that all inserts succeed but a small quantity of deletes
> failed.
>
> I tried many times , got the same situation.
> At last,  I add the  WriteConcern.SAFE on the delete method, then get
> the exception belows.
>
> Caused by: java.net.SocketTimeoutException: Read timed out
>         at java.net.SocketInputStream.socketRead0(Native Method)
>         at java.net.SocketInputStream.read(SocketInputStream.java:129)
>         at java.io.BufferedInputStream.fill(BufferedInputStream.java:
> 218)
>         at java.io.BufferedInputStream.read1(BufferedInputStream.java:
> 258)
>         at java.io.BufferedInputStream.read(BufferedInputStream.java:
> 317)
(Continue reading)

shingo | 2 Dec 04:11 2011

Re: SocketTimeoutException: Read timed out - while delete

Thank you for your reply.
You mean the reason of the deleting failed is the delete operation
must do a query at first, and that's why all inserts succeed but a
small quantity of deletes failed?

This is my config option (spring-mongodb config).
<mongo:options auto-connect-retry="true"
                        connections-per-host="20" connect-
timeout="1000" max-wait-time="1500"
			slave-ok="true" socket-keep-alive="true" socket-timeout="5000"
			write-fsync="false" write-number="0" write-timeout="0" />

On 12月2日, 上午8时04分, Antoine Girbal <anto... <at> 10gen.com> wrote:
> it looks like you set a timeout value on the socket, what is it set
> to?
> If you are doing a high volume of insert/delete, things can get slow
> if the server has to use the disk, and queries will timeout as a
> consequence.
> Do you monitor the load on the server (cpu, load, iostat, etc)?
> Also you can look at the mongod log for slow queries.
>
> On Dec 1, 12:58 pm, shingo <6623662...@...> wrote:
>
>
>
>
>
>
>
> > I used mongo-java-driver to operate mongodb(version 2.7.2)  in my
(Continue reading)

shingo | 2 Dec 04:33 2011

Re: SocketTimeoutException: Read timed out - while delete

An insert and a delete in each thread, it's my business logic.
Is there any way to make sure all the delete operations succeed ?

Antoine Girbal | 2 Dec 19:39 2011

Re: SocketTimeoutException: Read timed out - while delete

first of, make sure that you always use safe writes by setting
WriteConcern.SAFE.
Otherwise it is possible that your insert/delete would fail but your
application would not know about it.
Then you should remove the "socket-timeout" because if your thread
waits more than 5s then the client will abort.
If your database cannot keep up with the insert/delete, at least it
will make your client wait.
If you want to keep the timeout, then your application should be ready
to catch the timeout exception and retry or log the error somewhere.
Note that in general your application should always be ready to catch
exceptions, and handle the error appropriately (retry or send error
upstream).

Now you should also look at your delete statement, maybe it is not
designed properly and does a full scan.
What is the query you use for deleting? If you run an explain() on the
same query, does it properly use an index?

On Dec 1, 7:33 pm, shingo <6623662...@...> wrote:
> An insert and a delete in each thread, it's my business logic.
> Is there any way to make sure all the delete operations succeed ?

--

-- 
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To post to this group, send email to mongodb-user@...
To unsubscribe from this group, send email to mongodb-user+unsubscribe <at> googlegroups.com.
For more options, visit this group at http://groups.google.com/group/mongodb-user?hl=en.

(Continue reading)

shingo | 5 Dec 08:12 2011

Re: SocketTimeoutException: Read timed out - while delete

My delete statement is similar to "deleteById" ,   index is
automatically created for the _id field.

This is the code segment for deleting.

protected final static String ID_ATTRIBUTE = "_id";
getCollection(collectionName).remove(new BasicDBObject(ID_ATTRIBUTE,
id)),WriteConcern.SAFE);

The _id value set by my business logic.

Can "new BasicDBObject(ID_ATTRIBUTE, id)" hint the _id index
automatically ?

On 12月3日, 上午2时39分, Antoine Girbal <anto... <at> 10gen.com> wrote:
> first of, make sure that you always use safe writes by setting
> WriteConcern.SAFE.
> Otherwise it is possible that your insert/delete would fail but your
> application would not know about it.
> Then you should remove the "socket-timeout" because if your thread
> waits more than 5s then the client will abort.
> If your database cannot keep up with the insert/delete, at least it
> will make your client wait.
> If you want to keep the timeout, then your application should be ready
> to catch the timeout exception and retry or log the error somewhere.
> Note that in general your application should always be ready to catch
> exceptions, and handle the error appropriately (retry or send error
> upstream).
>
> Now you should also look at your delete statement, maybe it is not
(Continue reading)

Nat | 5 Dec 08:15 2011
Picon

Re: Re: SocketTimeoutException: Read timed out - while delete

Deleting by using _id should use the index automatically. You shouldn't need any hint.
-----Original Message-----
From: shingo <6623662005@...>
Sender: mongodb-user@...
Date: Sun, 4 Dec 2011 23:12:42 
To: mongodb-user<mongodb-user@...>
Reply-To: mongodb-user@...
Subject: [mongodb-user] Re: SocketTimeoutException: Read timed out - while delete

My delete statement is similar to "deleteById" ,   index is
automatically created for the _id field.

This is the code segment for deleting.

protected final static String ID_ATTRIBUTE = "_id";
getCollection(collectionName).remove(new BasicDBObject(ID_ATTRIBUTE,
id)),WriteConcern.SAFE);

The _id value set by my business logic.

Can "new BasicDBObject(ID_ATTRIBUTE, id)" hint the _id index
automatically ?

On 12月3日, 上午2时39分, Antoine Girbal <anto... <at> 10gen.com> wrote:
> first of, make sure that you always use safe writes by setting
> WriteConcern.SAFE.
> Otherwise it is possible that your insert/delete would fail but your
> application would not know about it.
> Then you should remove the "socket-timeout" because if your thread
> waits more than 5s then the client will abort.
(Continue reading)

shingo | 5 Dec 11:01 2011

Re: SocketTimeoutException: Read timed out - while delete

So, my delete statement should working well.
And now, the problem is that a small quantity of deletes failed, i
removed the "socket-timeout" yet.

On 12月5日, 下午3时15分, "Nat" <nat.lu...@...> wrote:
> Deleting by using _id should use the index automatically. You shouldn't need any hint.
>
>
>
>
>
>
>
> -----Original Message-----
> From: shingo <6623662...@...>
> Sender: mongodb-user@...
> Date: Sun, 4 Dec 2011 23:12:42
> To: mongodb-user<mongodb-user@...>
> Reply-To: mongodb-user@...
> Subject: [mongodb-user] Re: SocketTimeoutException: Read timed out - while delete
>
> My delete statement is similar to "deleteById" ,   index is
> automatically created for the _id field.
>
> This is the code segment for deleting.
>
> protected final static String ID_ATTRIBUTE = "_id";
> getCollection(collectionName).remove(new BasicDBObject(ID_ATTRIBUTE,
> id)),WriteConcern.SAFE);
>
(Continue reading)

Nat | 5 Dec 11:03 2011
Picon

Re: Re: SocketTimeoutException: Read timed out - while delete

Deleting by id should be very quick. Did you see any error in mongod log? Does the error disappear if you
increase socket timeout?
-----Original Message-----
From: shingo <6623662005@...>
Sender: mongodb-user@...
Date: Mon, 5 Dec 2011 02:01:44 
To: mongodb-user<mongodb-user@...>
Reply-To: mongodb-user@...
Subject: [mongodb-user] Re: SocketTimeoutException: Read timed out - while delete

So, my delete statement should working well.
And now, the problem is that a small quantity of deletes failed, i
removed the "socket-timeout" yet.

On 12月5日, 下午3时15分, "Nat" <nat.lu...@...> wrote:
> Deleting by using _id should use the index automatically. You shouldn't need any hint.
>
>
>
>
>
>
>
> -----Original Message-----
> From: shingo <6623662...@...>
> Sender: mongodb-user@...
> Date: Sun, 4 Dec 2011 23:12:42
> To: mongodb-user<mongodb-user@...>
> Reply-To: mongodb-user@...
> Subject: [mongodb-user] Re: SocketTimeoutException: Read timed out - while delete
(Continue reading)

shingo | 5 Dec 11:24 2011

Re: SocketTimeoutException: Read timed out - while delete

I tried again.
5000000 insert  and 5000000 delete

At last , 55069 rows remained. It should be zero if everything work
fine.

Is it relation to "journing"?
My mongodb version is 2.0.2rc1 , journing was  enbaled by default.

On 12月5日, 下午6时03分, "Nat" <nat.lu...@...> wrote:
> Deleting by id should be very quick. Did you see any error in mongod log? Does the error disappear if you
increase socket timeout?
>
>
>
>
>
>
>
> -----Original Message-----
> From: shingo <6623662...@...>
> Sender: mongodb-user@...
> Date: Mon, 5 Dec 2011 02:01:44
> To: mongodb-user<mongodb-user@...>
> Reply-To: mongodb-user@...
> Subject: [mongodb-user] Re: SocketTimeoutException: Read timed out - while delete
>
> So, my delete statement should working well.
> And now, the problem is that a small quantity of deletes failed, i
> removed the "socket-timeout" yet.
(Continue reading)

Nat | 5 Dec 11:27 2011
Picon

Re: Re: SocketTimeoutException: Read timed out - while delete

It could be that you inserted but didn't use writeconcern.safe before calling delete. Therefore, your
delete may actually be processed before insert. Can you try performing both ops with WriteConcern.SAFE?
-----Original Message-----
From: shingo <6623662005@...>
Sender: mongodb-user@...
Date: Mon, 5 Dec 2011 02:24:21 
To: mongodb-user<mongodb-user@...>
Reply-To: mongodb-user@...
Subject: [mongodb-user] Re: SocketTimeoutException: Read timed out - while delete

I tried again.
5000000 insert  and 5000000 delete

At last , 55069 rows remained. It should be zero if everything work
fine.

Is it relation to "journing"?
My mongodb version is 2.0.2rc1 , journing was  enbaled by default.

On 12月5日, 下午6时03分, "Nat" <nat.lu...@...> wrote:
> Deleting by id should be very quick. Did you see any error in mongod log? Does the error disappear if you
increase socket timeout?
>
>
>
>
>
>
>
> -----Original Message-----
(Continue reading)

shingo | 5 Dec 11:45 2011

Re: SocketTimeoutException: Read timed out - while delete

OK, i was planning to do this.
May be the insert WriteConcern.SAFE is my miss out .

On 12月5日, 下午6时27分, "Nat" <nat.lu...@...> wrote:
> It could be that you inserted but didn't use writeconcern.safe before calling delete. Therefore, your
delete may actually be processed before insert. Can you try performing both ops with WriteConcern.SAFE?
>
>
>
>
>
>
>
> -----Original Message-----
> From: shingo <6623662...@...>
> Sender: mongodb-user@...
> Date: Mon, 5 Dec 2011 02:24:21
> To: mongodb-user<mongodb-user@...>
> Reply-To: mongodb-user@...
> Subject: [mongodb-user] Re: SocketTimeoutException: Read timed out - while delete
>
> I tried again.
> 5000000 insert  and 5000000 delete
>
> At last , 55069 rows remained. It should be zero if everything work
> fine.
>
> Is it relation to "journing"?
> My mongodb version is 2.0.2rc1 , journing was  enbaled by default.
>
(Continue reading)

shingo | 5 Dec 12:04 2011

Re: SocketTimeoutException: Read timed out - while delete

It's Done!!
Thank you very much, Nat and Antoine.
This problem indicates the getLastError is very important in real-
time  environment.

On 12月5日, 下午6时45分, shingo <6623662...@...> wrote:
> OK, i was planning to do this.
> May be the insert WriteConcern.SAFE is my miss out .
>
> On 12月5日, 下午6时27分, "Nat" <nat.lu... <at> gmail.com> wrote:
>
>
>
>
>
>
>
> > It could be that you inserted but didn't use writeconcern.safe before calling delete. Therefore, your
delete may actually be processed before insert. Can you try performing both ops with WriteConcern.SAFE?
>
> > -----Original Message-----
> > From: shingo <6623662...@...>
> > Sender: mongodb-user@...
> > Date: Mon, 5 Dec 2011 02:24:21
> > To: mongodb-user<mongodb-user@...>
> > Reply-To: mongodb-user@...
> > Subject: [mongodb-user] Re: SocketTimeoutException: Read timed out - while delete
>
> > I tried again.
> > 5000000 insert  and 5000000 delete
(Continue reading)


Gmane