Mongo High Memory Usage
Mongo RAM Usage
Mongo : It is one of the best nosql key value database . It is one of the fastest growing database in its category. Followings are the mail advantages of using Mongo db (http://www.mongodb.org)
1- Auto sharding
2- Full Index Support.
3- Replication and Querying.
4- Fast Results.
In my test environment I am also using Mongo DB to store my daily collection and more than 50GB of data every day.
I am also using the full featured Index support for my user to give me full time search and fastest result as compare to any other db. I have also setup my hardware as such to accomodte my complete set of Indexes into my RAM.
If you are using Mongo, we should worry about the RAM availability in your machine on which your are running RAM. As Mongo have no limitation on the usage of RAM and it will consume your all RAM in order to accommodate complete Indexes/ Documents in thinking to give as faster results next time as it can.
Lets consider a scenario where we are using Mongo DB. We will create a collection per day and will insert around 50GB of data per day. Mongo db is also configured to indexing all the data as user have full authority to retrieve the data of 1 year ago.
Now if you look carefully about the condition mentioned above you can see that, we need to handle the Mongo db, Hardware , db management in such a way so that it will never my any bad effect of any of the critical thing which might results in system failure of bad user impact.
RAM : As I mentioned above mongo can consumes as much as RAM available on your machine. Hence be sure that you have RAM amount that much enough to accommodate you all Indexes and Documents into RAM. Other wise you will face MAJOR PAGE FAULTS of mongod services and will directly hit the user experience.
IN MY TEST ENVIRONMENT:
I have 20GB of RAM and same amount of data as explained above. As per my requirement I need to keep my all indexes into RAM which cost about 5GB. But strange... My total RAM consumption is 19GB.
GOD... i got shocked after checking that, I have 2 Mongod instances and both of them are consuming 9GB each (as per top command output).
On tracking multiple times and performing very debugiing logging I found that, Mongo is actually consuming about 4GB or RAM and rest of the RAM is cached by it.
free free command of linux is very useful in this case. Below is the output of free -m command:
#free -m
total used free buffer
Memory 21452 19233 300 231
+ - buffer cache 12341 7423
As per above stats, my total RAM consumption is 19GB out of 20GB. But thats not true. This amount comes from the total RAM consumption i.e., in use RAM + cached RAM. Now mongo cached the required RAM for future usage probability and leave on OS to handle the memory management.
Actuall RAM usage comes from the + - buffer cache. It will give the clear picture of what amount of RAM is consumed and how much is left.
Restart Mongod services
I also tried to restart the Mongod services to clear the indexes and so free the usage of RAM. Just after the restart of Mongod services, my RAM usage comes down from 19GB to 6GB because this restart clears all the indexes of Mongod services.
Strange is that, within 30 mins after restart of mongod services, i found that mongod again reached to 9GB usage and my system RAM usage touched to 19GB.
But there is no issue, that all RAM usage is calculated with actual usage and cached usage in top command output but free command make a great relief from my mind that my RAM is still available for other services.
Final comment is that,, don't panic about the usage of high RAM usage. Its the mongo behaviour that it will try to acquire all RAM but still you have RAM available for other services, just check that + - buffer cahce of free -m command.
JUST BELIEVE IN TECHNOLOGY.......
LIKE AND COMMENT ON POST
Mongo : It is one of the best nosql key value database . It is one of the fastest growing database in its category. Followings are the mail advantages of using Mongo db (http://www.mongodb.org)
1- Auto sharding
2- Full Index Support.
3- Replication and Querying.
4- Fast Results.
In my test environment I am also using Mongo DB to store my daily collection and more than 50GB of data every day.
I am also using the full featured Index support for my user to give me full time search and fastest result as compare to any other db. I have also setup my hardware as such to accomodte my complete set of Indexes into my RAM.
If you are using Mongo, we should worry about the RAM availability in your machine on which your are running RAM. As Mongo have no limitation on the usage of RAM and it will consume your all RAM in order to accommodate complete Indexes/ Documents in thinking to give as faster results next time as it can.
Lets consider a scenario where we are using Mongo DB. We will create a collection per day and will insert around 50GB of data per day. Mongo db is also configured to indexing all the data as user have full authority to retrieve the data of 1 year ago.
Now if you look carefully about the condition mentioned above you can see that, we need to handle the Mongo db, Hardware , db management in such a way so that it will never my any bad effect of any of the critical thing which might results in system failure of bad user impact.
RAM : As I mentioned above mongo can consumes as much as RAM available on your machine. Hence be sure that you have RAM amount that much enough to accommodate you all Indexes and Documents into RAM. Other wise you will face MAJOR PAGE FAULTS of mongod services and will directly hit the user experience.
IN MY TEST ENVIRONMENT:
I have 20GB of RAM and same amount of data as explained above. As per my requirement I need to keep my all indexes into RAM which cost about 5GB. But strange... My total RAM consumption is 19GB.
GOD... i got shocked after checking that, I have 2 Mongod instances and both of them are consuming 9GB each (as per top command output).
On tracking multiple times and performing very debugiing logging I found that, Mongo is actually consuming about 4GB or RAM and rest of the RAM is cached by it.
free free command of linux is very useful in this case. Below is the output of free -m command:
#free -m
total used free buffer
Memory 21452 19233 300 231
+ - buffer cache 12341 7423
As per above stats, my total RAM consumption is 19GB out of 20GB. But thats not true. This amount comes from the total RAM consumption i.e., in use RAM + cached RAM. Now mongo cached the required RAM for future usage probability and leave on OS to handle the memory management.
Actuall RAM usage comes from the + - buffer cache. It will give the clear picture of what amount of RAM is consumed and how much is left.
Restart Mongod services
I also tried to restart the Mongod services to clear the indexes and so free the usage of RAM. Just after the restart of Mongod services, my RAM usage comes down from 19GB to 6GB because this restart clears all the indexes of Mongod services.
Strange is that, within 30 mins after restart of mongod services, i found that mongod again reached to 9GB usage and my system RAM usage touched to 19GB.
But there is no issue, that all RAM usage is calculated with actual usage and cached usage in top command output but free command make a great relief from my mind that my RAM is still available for other services.
Final comment is that,, don't panic about the usage of high RAM usage. Its the mongo behaviour that it will try to acquire all RAM but still you have RAM available for other services, just check that + - buffer cahce of free -m command.
JUST BELIEVE IN TECHNOLOGY.......
LIKE AND COMMENT ON POST
Comments
Post a Comment