How to install Zabbix on linux/Centos
Friday, August 9, 2013
Thursday, January 17, 2013
How to install Nginx on linux and configure with apache.
1.Download tar form mention link http://nginx.org/download/nginx-1.3.11.tar.gz
2.untar with tar -zxvf nginx-1.3.11.tar.gz
3.compile ./configure --prefix=/usr/local/nginx --with-http_stub_status_module --with-
http_realip_module --with-http_ssl_module --user=nobody --group=nobody --with-
http_realip_module.
4.make && make install
5.Start apache with 8080 port instead of 80.
6. Now vi /usr/local/nginx/conf/nginx.conf
7. add following line
server {
listen 80;
server_name www.domain.com domain.com;
location / {
root /var/www/domain.com/html;
index index.html index.htm;
}
}
location {
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_pass http://localhost:8080;
proxy_buffer_size 64k;
proxy_buffers 32 64K;
proxy_busy_buffers_size 64k;
proxy_temp_file_write_size 64k;
client_max_body_size 5m; #Set max body size to 5Mb
}
8. Now this is time to start nginx service on server run following command.
/usr/local/nginx/sbin/nginx -c /usr/local/nginx/conf/nginx.conf
Wednesday, March 16, 2011
mysql performence tunning for myisam
MySQL Server Memory Usage
May 17, 2006 By Peter Zaitsev 15 Comments
Every so often people ask me the question how should they estimate memory consumption by MySQL Server in given configuration. What is the formula they could use.
The reasons to worry about memory usage are quite understandable. If you configure MySQL Server so it uses too small amount of memory it will likey perform suboptimally. If you however configure it so it consumes too much memory it may be crashing , failing to execute queries or make operation to swap seriously slowing down. On now legacy 32bit platforms you could also run out of address space so that had to be watched as well.
Having said so, I do not think looking for the secret fomula to compute your possible memory usage is the right approach to this problem. The reasons are – this formula is very complex nowadays and what is even more important “theoretically possible” maximum it provides have nothing to do with real memory consumptions. In fact typical server with 8GB of memory will often run with maximum theoretical memory usage of 100GB or more. Furthermore there is no easy “overcommit factor” you can use – it really depends on application and configuration. Some applications will drive server to 10% of theoretical memory consumptions others only to 1%.
So what could you do instead ? First take a look at global buffers which are allocated at start and always where – these are key_buffer_size, innodb_buffer_pool_size, innodb_additional_memory_pool_size, innodb_log_buffer_size, query_cache_size. If you’re using MyISAM seriously you can also add the size of Operation System cache you would like MySQL to use for your table. Take this number add to it number of memory Operation System and other applications need, add might be 32MB more for MySQL Server code and various small static buffers. This is memory which you can consider used when you just start MySQL Server. The rest of memory is available for connections. For exampe with 8GB server you might have everything listed adding up to 6GB, so you have 2GB left for your threads.
Each thread connecting to MySQL server will needs its own buffers. About 256K is allocated at once even if thread is idle – they are used by default thread stack, net buffer etc. If transaction is started some more space can add up. Running small queries might only barely increase memory consumption for given thread, however if table will perform complex operations such as full table scans, sorts, or need temporary tables as much as read_buffer_size, sort_buffer_size, read_rnd_buffer_size, tmp_table_size of memory might be allocated. But they are only allocated upon the need and freed once given stage of query is done. Some of them are allocated as single chunk at once others, for example tmp_table_size is rather maximum amount of memory MySQL will allocate for this operation. Note it is more complicated than once may think – multiple buffers of the same type might be allocated for exampe to handle subqueries. For some special queries memory usage might be even larger – bulk inserts may allocate bulk_insert_buffer_size bytes of memory if done to MyISAM tables. myisam_sort_buffer_size used for ALTER TABLE, OPTIMIZE TABLE, REPAIR TABLE commands.
For OLTP applications with simple queries memory consumption is often less than 1MB per thread with default buffers, and you really do not need to increase per thread buffers unless you have complex queries. Sorting 10 rows will be as fast with 1MB sort buffer as with 16MB (actually 16MB might be even slower but it is other story).
Another approach you may take is to come up with amount of memory you want MySQL Server to consume at peak. This can be easily computed by memory needed for OS, File Cache and other applications. For 32bit envinronment you also should keep 32bit limits into account and probably limit “mysqld” size to about 2.5GB (exact number depens on a lot of factors). Now you can use “ps aux” to see VSZ – Virtual Memory allocated by MySQL process. You can also look at “Resident Memory” but I find it less helpful as it may down because of swapping – not what you would like to see. Monitor how the value changes so you know memory requirements with current settings and increase/decrease values appropriately.
Some may say, Hey we want to have 100% guarantee our server will never run out of memory, no matter which queries or users will decide to run. Unfortunately this is as much close to impossible to be impractical. Here is why:
List of rarely considered MySQL Server Memory Requirements
Thread buffers can be allocated more than once for each thread. Consider for example subqueries – each layer may need its own read_buffer,sort_buffer, tmp_table_size etc
Many variabes can be set per connection. So you can’t relay on global values if developers may use their local values to run some queries.
There can be mutiple key caches. Multiple key caches can be created to accomodate query executions
Query Parsing and optimization needs memory. This is usually small to be ignored but certain queries can have very large memory requrement for this step, especially specially crafted ones.
Stored Procedures. Compex stored procedures may require a lot of memory
Prepared statements and Cursors. Single connection may have many prepared statements and cursors. Their number finally can be limited but each of them still can have very large memory consumption
Innodb Table Cache. Innodb has its own table cache in which meta data about each table accessed from the start is stored. It is never purged and may be large if you have a lot of tables. It also means user having CREATE TABLE privilege should be able to run MySQL server out of memory
MyISAM buffers. MyISAM may allocate buffer which is large enough to contain largest record in the given table which is held until table is closed.
Federated Storage Engine. This may have unbound memory requirements retriving result sets from remove queries.
Blobs may require 3x time of memory. This is important if you’re deaing with large Blobs (your max_allowed_packet is large) Processing of 256MB of blob may require 768MB of memory.
Storage Engines. In general storage engines may have their own per thread or global memory allocations which are not tuned as buffers. Watch for these especially now with many storage engines being released for MySQL by various parties.
I do not pretend this to be complete list. On the contrary I’m quite sure I’ve missed something (drop me a note if you have something to add). But the main point is – there are a lot of memory consumers out where and trying to find peak possible usage for each is impractical – so my advice would be measure what you get in practice and how memory consumption reacts to changing various variables. For example you may find out increasing sort_buffer_size from 1MB to 4MB and 1000 max_connections increases peak memory consumption just 30MB not 3000MB as you might have counted.
May 17, 2006 By Peter Zaitsev 15 Comments
Every so often people ask me the question how should they estimate memory consumption by MySQL Server in given configuration. What is the formula they could use.
The reasons to worry about memory usage are quite understandable. If you configure MySQL Server so it uses too small amount of memory it will likey perform suboptimally. If you however configure it so it consumes too much memory it may be crashing , failing to execute queries or make operation to swap seriously slowing down. On now legacy 32bit platforms you could also run out of address space so that had to be watched as well.
Having said so, I do not think looking for the secret fomula to compute your possible memory usage is the right approach to this problem. The reasons are – this formula is very complex nowadays and what is even more important “theoretically possible” maximum it provides have nothing to do with real memory consumptions. In fact typical server with 8GB of memory will often run with maximum theoretical memory usage of 100GB or more. Furthermore there is no easy “overcommit factor” you can use – it really depends on application and configuration. Some applications will drive server to 10% of theoretical memory consumptions others only to 1%.
So what could you do instead ? First take a look at global buffers which are allocated at start and always where – these are key_buffer_size, innodb_buffer_pool_size, innodb_additional_memory_pool_size, innodb_log_buffer_size, query_cache_size. If you’re using MyISAM seriously you can also add the size of Operation System cache you would like MySQL to use for your table. Take this number add to it number of memory Operation System and other applications need, add might be 32MB more for MySQL Server code and various small static buffers. This is memory which you can consider used when you just start MySQL Server. The rest of memory is available for connections. For exampe with 8GB server you might have everything listed adding up to 6GB, so you have 2GB left for your threads.
Each thread connecting to MySQL server will needs its own buffers. About 256K is allocated at once even if thread is idle – they are used by default thread stack, net buffer etc. If transaction is started some more space can add up. Running small queries might only barely increase memory consumption for given thread, however if table will perform complex operations such as full table scans, sorts, or need temporary tables as much as read_buffer_size, sort_buffer_size, read_rnd_buffer_size, tmp_table_size of memory might be allocated. But they are only allocated upon the need and freed once given stage of query is done. Some of them are allocated as single chunk at once others, for example tmp_table_size is rather maximum amount of memory MySQL will allocate for this operation. Note it is more complicated than once may think – multiple buffers of the same type might be allocated for exampe to handle subqueries. For some special queries memory usage might be even larger – bulk inserts may allocate bulk_insert_buffer_size bytes of memory if done to MyISAM tables. myisam_sort_buffer_size used for ALTER TABLE, OPTIMIZE TABLE, REPAIR TABLE commands.
For OLTP applications with simple queries memory consumption is often less than 1MB per thread with default buffers, and you really do not need to increase per thread buffers unless you have complex queries. Sorting 10 rows will be as fast with 1MB sort buffer as with 16MB (actually 16MB might be even slower but it is other story).
Another approach you may take is to come up with amount of memory you want MySQL Server to consume at peak. This can be easily computed by memory needed for OS, File Cache and other applications. For 32bit envinronment you also should keep 32bit limits into account and probably limit “mysqld” size to about 2.5GB (exact number depens on a lot of factors). Now you can use “ps aux” to see VSZ – Virtual Memory allocated by MySQL process. You can also look at “Resident Memory” but I find it less helpful as it may down because of swapping – not what you would like to see. Monitor how the value changes so you know memory requirements with current settings and increase/decrease values appropriately.
Some may say, Hey we want to have 100% guarantee our server will never run out of memory, no matter which queries or users will decide to run. Unfortunately this is as much close to impossible to be impractical. Here is why:
List of rarely considered MySQL Server Memory Requirements
Thread buffers can be allocated more than once for each thread. Consider for example subqueries – each layer may need its own read_buffer,sort_buffer, tmp_table_size etc
Many variabes can be set per connection. So you can’t relay on global values if developers may use their local values to run some queries.
There can be mutiple key caches. Multiple key caches can be created to accomodate query executions
Query Parsing and optimization needs memory. This is usually small to be ignored but certain queries can have very large memory requrement for this step, especially specially crafted ones.
Stored Procedures. Compex stored procedures may require a lot of memory
Prepared statements and Cursors. Single connection may have many prepared statements and cursors. Their number finally can be limited but each of them still can have very large memory consumption
Innodb Table Cache. Innodb has its own table cache in which meta data about each table accessed from the start is stored. It is never purged and may be large if you have a lot of tables. It also means user having CREATE TABLE privilege should be able to run MySQL server out of memory
MyISAM buffers. MyISAM may allocate buffer which is large enough to contain largest record in the given table which is held until table is closed.
Federated Storage Engine. This may have unbound memory requirements retriving result sets from remove queries.
Blobs may require 3x time of memory. This is important if you’re deaing with large Blobs (your max_allowed_packet is large) Processing of 256MB of blob may require 768MB of memory.
Storage Engines. In general storage engines may have their own per thread or global memory allocations which are not tuned as buffers. Watch for these especially now with many storage engines being released for MySQL by various parties.
I do not pretend this to be complete list. On the contrary I’m quite sure I’ve missed something (drop me a note if you have something to add). But the main point is – there are a lot of memory consumers out where and trying to find peak possible usage for each is impractical – so my advice would be measure what you get in practice and how memory consumption reacts to changing various variables. For example you may find out increasing sort_buffer_size from 1MB to 4MB and 1000 max_connections increases peak memory consumption just 30MB not 3000MB as you might have counted.
Monday, February 14, 2011
innodb variables which can fast your mysql performence
Hi,
2.#this has to be set to 50-80% of your physical memory (depends what else you run in this machine), but not higher than 2G for 32bit platform.
innodb_buffer_pool_size=2G
3.# this has to be set to about 25% of innodb_buffer_pool_size variable
4.# 0 here disables flush on every commit, instead of that mysql flushes logs every second. So if your box goes down you might loose 1 second of data which is not important for zabbix but might be important for other application. Consider this before disabling flush on commit. This improves write performance alot.
innodb_flush_log_at_trx_commit=0
5.# Following variables might improve or might not improve your performance. If your box has several CPUs and fast disk system set this to the number of CPUs you have. If you however have slow disk this might create thread tharshing and work towards performance degradation. If not sure set small value like 2 to have 2 threads running instead of 1.
innodb_thread_concurrency=2
thread_concurrency = 2
6.# this should be more than number of zabbix_server daemons, if you have queue backup and your box is not very busy most likely you need to increase number of zabbix_server daemons, dont forget to increase this number as well as because zabbix server completely crashes if any daemon is not able to connect to mysql.
max_connections = 200
7.O_DIRECT so that the data cached in the innodb buffer pool would not be duplicated in the filesystem buffer cache. So, in /etc/my.cnf:
innodb_buffer_pool_size=12000M
innodb_flush_method=O_DIRECT
8.other parameters
innodb_file_per_table
Use file_per_table to keep tablespaces more compact and use "optimize table" periodically. And when you set this value in my.cnf you don't get an actual file_per_table until you run an
optimize on all the tables. This'll take a long time on the large zabbix history* and trends* tables.
9.thread_cache_size=4
10.I forgot to ask the consultant about this setting but it seemed to have a large effect when I set it initially (EDIT: reading up online it seems to affect the hit rate of Threads_created per Connection in 'show global status' -- with it set to 4 i had 3228483 Connections and 5840 Threads_created, which is a 99.2% hit rate -- higher number of Threads_created is worse).
query_cache_limit=1M
Consultant advised keeping this to 1MB or less. I had tuned it higher.
11.query_cache_size=128M
12.Consultant advised that this was a good value and not raising this any higher
tmp_table_size=256M
max_heap_table_size=256M
13.I tuned these based on some other tuning docs, and the consultant concurred, but they shouldn't go any higher, and they should be set to the same value.
table_cache=256
join_buffer_size=256k
read_buffer_size=256k
read_rnd_buffer_size=256k
Friday, February 19, 2010
Important Linux Commands
To display the network counters from the 24th:
Manish Kumar Rai
#9899856382
Find Out The Top 10 Memory Consuming Process
# ps -auxf | sort -nr -k 4 | head -10
Find Out top 10 CPU Consuming Process
# ps -auxf | sort -nr -k 3 | head -10
sar - Collect and Report System Activity
# sar -n DEV | moreTo display the network counters from the 24th:
# sar -n DEV -f /var/log/sa/sa24 | moreManish Kumar Rai
#9899856382
Subscribe to:
Posts (Atom)