Aug 4 11:10:10 rhel6-node2 kernel: imklog 4.6.2, log source = /proc/kmsg started. Aug 4 11:10:10 rhel6-node2 rsyslogd: [origin software="rsyslogd" swVersion="4.6.2" x-pid="1069" x-info="http://www.rsyslog.com"] (re)start Aug 4 11:10:10 rhel6-node2 corosync[1454]: [TOTEM ] A processor failed, forming new configuration. Aug 4 11:10:16 rhel6-node2 qdiskd[1506]: qdisk cycle took more than 1 second to complete (6.260000) Aug 4 11:10:21 rhel6-node2 qdiskd[1506]: qdiskd: read (system call) has hung for 2 seconds Aug 4 11:10:21 rhel6-node2 qdiskd[1506]: In 2 more seconds, we will be evicted Aug 4 11:10:23 rhel6-node2 corosync[1454]: [TOTEM ] A processor joined or left the membership and a new membership was formed. Aug 4 11:10:23 rhel6-node2 corosync[1454]: [CPG ] downlist received left_list: 0 Aug 4 11:10:23 rhel6-node2 corosync[1454]: [CPG ] downlist received left_list: 0 Aug 4 11:10:23 rhel6-node2 corosync[1454]: [CPG ] downlist received left_list: 0 Aug 4 11:10:23 rhel6-node2 corosync[1454]: [CPG ] downlist received left_list: 0 Aug 4 11:10:23 rhel6-node2 corosync[1454]: [CPG ] downlist received left_list: 0 Aug 4 11:10:23 rhel6-node2 corosync[1454]: [CPG ] downlist received left_list: 0 Aug 4 11:10:23 rhel6-node2 corosync[1454]: [CPG ] downlist received left_list: 0 Aug 4 11:10:23 rhel6-node2 corosync[1454]: [CPG ] downlist received left_list: 0 Aug 4 11:10:23 rhel6-node2 corosync[1454]: [CPG ] chosen downlist from node r(0) ip(192.168.2.65) Aug 4 11:10:23 rhel6-node2 corosync[1454]: [MAIN ] Completed service synchronization, ready to provide service. Aug 4 11:10:25 rhel6-node2 qdiskd[1506]: qdiskd: write (system call) has hung for 2 seconds Aug 4 11:10:25 rhel6-node2 qdiskd[1506]: In 2 more seconds, we will be evicted Aug 4 11:10:25 rhel6-node2 qdiskd[1506]: qdisk cycle took more than 1 second to complete (7.940000) Aug 4 11:10:30 rhel6-node2 qdiskd[1506]: qdisk cycle took more than 1 second to complete (4.710000) Aug 4 11:10:35 rhel6-node2 qdiskd[1506]: qdiskd: write (system call) has hung for 2 seconds Aug 4 11:10:35 rhel6-node2 qdiskd[1506]: In 2 more seconds, we will be evicted Aug 4 11:10:37 rhel6-node2 qdiskd[1506]: qdisk cycle took more than 1 second to complete (5.560000) Aug 4 11:10:45 rhel6-node2 qdiskd[1506]: qdiskd: write (system call) has hung for 2 seconds Aug 4 11:10:45 rhel6-node2 qdiskd[1506]: In 2 more seconds, we will be evicted Aug 4 11:10:45 rhel6-node2 qdiskd[1506]: qdisk cycle took more than 1 second to complete (7.690000) Aug 4 11:10:56 rhel6-node2 qdiskd[1506]: qdiskd: write (system call) has hung for 2 seconds Aug 4 11:10:56 rhel6-node2 qdiskd[1506]: In 2 more seconds, we will be evicted Aug 4 11:11:02 rhel6-node2 qdiskd[1506]: qdisk cycle took more than 1 second to complete (15.320000) Aug 4 11:11:06 rhel6-node2 qdiskd[1506]: qdiskd: write (system call) has hung for 2 seconds Aug 4 11:11:06 rhel6-node2 qdiskd[1506]: In 2 more seconds, we will be evicted Aug 4 11:11:08 rhel6-node2 qdiskd[1506]: qdisk cycle took more than 1 second to complete (4.820000) Aug 4 11:11:19 rhel6-node2 qdiskd[1506]: qdiskd: write (system call) has hung for 2 seconds Aug 4 11:11:19 rhel6-node2 qdiskd[1506]: In 2 more seconds, we will be evicted Aug 4 11:11:20 rhel6-node2 qdiskd[1506]: qdisk cycle took more than 1 second to complete (10.910000) Aug 4 11:11:28 rhel6-node2 qdiskd[1506]: qdiskd: write (system call) has hung for 2 seconds Aug 4 11:11:28 rhel6-node2 qdiskd[1506]: In 2 more seconds, we will be evicted Aug 4 11:11:30 rhel6-node2 qdiskd[1506]: qdisk cycle took more than 1 second to complete (8.590000) Aug 4 11:11:39 rhel6-node2 qdiskd[1506]: qdiskd: write (system call) has hung for 2 seconds Aug 4 11:11:39 rhel6-node2 qdiskd[1506]: In 2 more seconds, we will be evicted Aug 4 11:11:43 rhel6-node2 qdiskd[1506]: qdisk cycle took more than 1 second to complete (11.220000) Aug 4 11:11:46 rhel6-node2 qdiskd[1506]: qdisk cycle took more than 1 second to complete (2.380000) Aug 4 11:11:56 rhel6-node2 kernel: __ratelimit: 306 callbacks suppressed Aug 4 11:11:56 rhel6-node2 kernel: kswapd0: page allocation failure. order:0, mode:0x20 Aug 4 11:11:56 rhel6-node2 kernel: Pid: 26, comm: kswapd0 Not tainted 2.6.32-54.el6.i686 #1 Aug 4 11:11:56 rhel6-node2 kernel: Call Trace: Aug 4 11:11:56 rhel6-node2 kernel: [] ? __alloc_pages_nodemask+0x584/0x760 Aug 4 11:11:56 rhel6-node2 kernel: [] ? smp_apic_timer_interrupt+0x54/0x90 Aug 4 11:11:56 rhel6-node2 kernel: [] ? get_a_page+0x3d/0x50 [virtio_net] Aug 4 11:11:56 rhel6-node2 kernel: [] ? try_fill_recv+0x113/0x1a0 [virtio_net] Aug 4 11:11:56 rhel6-node2 kernel: [] ? __alloc_skb+0x4f/0x140 Aug 4 11:11:56 rhel6-node2 kernel: [] ? virtnet_poll+0x3d4/0x730 [virtio_net] Aug 4 11:11:56 rhel6-node2 kernel: [] ? net_rx_action+0xde/0x1c0 Aug 4 11:11:56 rhel6-node2 kernel: [] ? net_tx_action+0xc7/0x120 Aug 4 11:11:56 rhel6-node2 kernel: [] ? __do_softirq+0x8f/0x1b0 Aug 4 11:11:56 rhel6-node2 kernel: [] ? move_native_irq+0x11/0x50 Aug 4 11:11:56 rhel6-node2 kernel: [] ? do_softirq+0x3d/0x50 Aug 4 11:11:56 rhel6-node2 kernel: [] ? irq_exit+0x65/0x70 Aug 4 11:11:56 rhel6-node2 kernel: [] ? do_IRQ+0x50/0xc0 Aug 4 11:11:56 rhel6-node2 kernel: [] ? common_interrupt+0x30/0x38 Aug 4 11:11:56 rhel6-node2 kernel: [] ? put_page+0x2/0x30 Aug 4 11:11:56 rhel6-node2 kernel: [] ? drop_buffers+0x73/0x90 Aug 4 11:11:56 rhel6-node2 kernel: [] ? try_to_free_buffers+0x36/0x90 Aug 4 11:11:56 rhel6-node2 kernel: [] ? try_to_release_page+0x23/0x40 Aug 4 11:11:56 rhel6-node2 kernel: [] ? shrink_page_list.clone.0+0x35a/0x460 Aug 4 11:11:56 rhel6-node2 kernel: [] ? shrink_inactive_list+0x2bd/0x630 Aug 4 11:11:56 rhel6-node2 kernel: [] ? irq_exit+0x35/0x70 Aug 4 11:11:56 rhel6-node2 kernel: [] ? do_IRQ+0x50/0xc0 Aug 4 11:11:56 rhel6-node2 kernel: [] ? common_interrupt+0x30/0x38 Aug 4 11:11:56 rhel6-node2 kernel: [] ? shrink_zone+0x30d/0x460 Aug 4 11:11:56 rhel6-node2 kernel: [] ? kswapd+0x669/0x840 Aug 4 11:11:56 rhel6-node2 kernel: [] ? isolate_pages_global+0x0/0x220 Aug 4 11:11:56 rhel6-node2 kernel: [] ? autoremove_wake_function+0x0/0x40 Aug 4 11:11:56 rhel6-node2 kernel: [] ? complete+0x40/0x60 Aug 4 11:11:56 rhel6-node2 kernel: [] ? kswapd+0x0/0x840 Aug 4 11:11:56 rhel6-node2 kernel: [] ? kthread+0x74/0x80 Aug 4 11:11:56 rhel6-node2 kernel: [] ? kthread+0x0/0x80 Aug 4 11:11:56 rhel6-node2 kernel: [] ? kernel_thread_helper+0x7/0x10 Aug 4 11:11:56 rhel6-node2 kernel: Mem-Info: Aug 4 11:11:56 rhel6-node2 kernel: DMA per-cpu: Aug 4 11:11:56 rhel6-node2 kernel: CPU 0: hi: 0, btch: 1 usd: 0 Aug 4 11:11:56 rhel6-node2 kernel: Normal per-cpu: Aug 4 11:11:56 rhel6-node2 kernel: CPU 0: hi: 186, btch: 31 usd: 126 Aug 4 11:11:56 rhel6-node2 kernel: active_anon:1187 inactive_anon:1376 isolated_anon:0 Aug 4 11:11:56 rhel6-node2 kernel: active_file:4587 inactive_file:77968 isolated_file:32 Aug 4 11:11:56 rhel6-node2 kernel: unevictable:32485 dirty:8245 writeback:40 unstable:0 Aug 4 11:11:56 rhel6-node2 kernel: free:735 slab_reclaimable:3027 slab_unreclaimable:3524 Aug 4 11:11:56 rhel6-node2 kernel: mapped:10107 shmem:7186 pagetables:448 bounce:0 Aug 4 11:11:56 rhel6-node2 kernel: DMA free:2004kB min:84kB low:104kB high:124kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:6348kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15844kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 4 11:11:56 rhel6-node2 kernel: lowmem_reserve[]: 0 492 492 492 Aug 4 11:11:56 rhel6-node2 kernel: Normal free:936kB min:2792kB low:3488kB high:4188kB active_anon:4748kB inactive_anon:5504kB active_file:18348kB inactive_file:305524kB unevictable:129940kB isolated(anon):0kB isolated(file):128kB present:503916kB mlocked:129940kB dirty:32980kB writeback:160kB mapped:40428kB shmem:28744kB slab_reclaimable:12108kB slab_unreclaimable:14096kB kernel_stack:1056kB pagetables:1792kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:128 all_unreclaimable? no Aug 4 11:11:56 rhel6-node2 kernel: lowmem_reserve[]: 0 0 0 0 Aug 4 11:11:56 rhel6-node2 kernel: DMA: 1*4kB 0*8kB 1*16kB 0*32kB 1*64kB 1*128kB 1*256kB 1*512kB 1*1024kB 0*2048kB 0*4096kB = 2004kB Aug 4 11:11:56 rhel6-node2 kernel: Normal: 0*4kB 1*8kB 0*16kB 1*32kB 0*64kB 1*128kB 1*256kB 1*512kB 0*1024kB 0*2048kB 0*4096kB = 936kB Aug 4 11:11:56 rhel6-node2 kernel: 91658 total pagecache pages Aug 4 11:11:56 rhel6-node2 kernel: 0 pages in swap cache Aug 4 11:11:56 rhel6-node2 kernel: Swap cache stats: add 0, delete 0, find 0/0 Aug 4 11:11:56 rhel6-node2 kernel: Free swap = 1048568kB Aug 4 11:11:56 rhel6-node2 kernel: Total swap = 1048568kB Aug 4 11:11:56 rhel6-node2 kernel: 131066 pages RAM Aug 4 11:11:56 rhel6-node2 kernel: 0 pages HighMem Aug 4 11:11:56 rhel6-node2 kernel: 3162 pages reserved Aug 4 11:11:56 rhel6-node2 kernel: 59440 pages shared Aug 4 11:11:56 rhel6-node2 kernel: 75719 pages non-shared Aug 4 11:11:56 rhel6-node2 qdiskd[1506]: qdiskd: write (system call) has hung for 2 seconds Aug 4 11:11:56 rhel6-node2 qdiskd[1506]: In 2 more seconds, we will be evicted Aug 4 11:11:57 rhel6-node2 qdiskd[1506]: qdisk cycle took more than 1 second to complete (8.550000) Aug 4 11:12:02 rhel6-node2 qdiskd[1506]: qdiskd: read (system call) has hung for 2 seconds Aug 4 11:12:02 rhel6-node2 qdiskd[1506]: In 2 more seconds, we will be evicted Aug 4 11:12:11 rhel6-node2 corosync[1454]: [TOTEM ] A processor failed, forming new configuration. Aug 4 11:12:11 rhel6-node2 qdiskd[1506]: qdiskd: write (system call) has hung for 2 seconds Aug 4 11:12:11 rhel6-node2 qdiskd[1506]: In 2 more seconds, we will be evicted Aug 4 11:12:13 rhel6-node2 qdiskd[1506]: qdisk cycle took more than 1 second to complete (16.100000) Aug 4 11:12:21 rhel6-node2 qdiskd[1506]: qdiskd: write (system call) has hung for 2 seconds Aug 4 11:12:21 rhel6-node2 qdiskd[1506]: In 2 more seconds, we will be evicted Aug 4 11:12:22 rhel6-node2 qdiskd[1506]: qdisk cycle took more than 1 second to complete (7.430000) Aug 4 11:12:30 rhel6-node2 qdiskd[1506]: qdisk cycle took more than 1 second to complete (7.190000) Aug 4 11:12:30 rhel6-node2 corosync[1454]: [TOTEM ] A processor joined or left the membership and a new membership was formed. Aug 4 11:12:32 rhel6-node2 qdiskd[1506]: qdisk cycle took more than 1 second to complete (2.000000) Aug 4 11:12:38 rhel6-node2 qdiskd[1506]: qdiskd: write (system call) has hung for 2 seconds Aug 4 11:12:38 rhel6-node2 qdiskd[1506]: In 2 more seconds, we will be evicted Aug 4 11:12:41 rhel6-node2 qdiskd[1506]: qdisk cycle took more than 1 second to complete (6.440000) Aug 4 11:12:48 rhel6-node2 qdiskd[1506]: qdiskd: write (system call) has hung for 2 seconds Aug 4 11:12:48 rhel6-node2 qdiskd[1506]: In 2 more seconds, we will be evicted Aug 4 11:12:50 rhel6-node2 qdiskd[1506]: qdisk cycle took more than 1 second to complete (8.270000) Aug 4 11:12:50 rhel6-node2 corosync[1454]: [CPG ] downlist received left_list: 0 Aug 4 11:12:50 rhel6-node2 corosync[1454]: [CPG ] downlist received left_list: 0 Aug 4 11:12:50 rhel6-node2 corosync[1454]: [CPG ] downlist received left_list: 0 Aug 4 11:12:51 rhel6-node2 corosync[1454]: [CPG ] downlist received left_list: 0 Aug 4 11:12:51 rhel6-node2 corosync[1454]: [CPG ] downlist received left_list: 0 Aug 4 11:12:51 rhel6-node2 corosync[1454]: [CPG ] downlist received left_list: 0 Aug 4 11:12:52 rhel6-node2 corosync[1454]: [CPG ] downlist received left_list: 0 Aug 4 11:12:52 rhel6-node2 corosync[1454]: [CPG ] downlist received left_list: 0 Aug 4 11:12:52 rhel6-node2 corosync[1454]: [CPG ] chosen downlist from node r(0) ip(192.168.2.65) Aug 4 11:12:55 rhel6-node2 corosync[1454]: [MAIN ] Completed service synchronization, ready to provide service. Aug 4 11:12:56 rhel6-node2 qdiskd[1506]: qdisk cycle took more than 1 second to complete (5.160000) Aug 4 11:13:03 rhel6-node2 qdiskd[1506]: qdisk cycle took more than 1 second to complete (6.160000) Aug 4 11:13:10 rhel6-node2 qdiskd[1506]: qdiskd: write (system call) has hung for 2 seconds Aug 4 11:13:10 rhel6-node2 qdiskd[1506]: In 2 more seconds, we will be evicted Aug 4 11:13:14 rhel6-node2 qdiskd[1506]: qdisk cycle took more than 1 second to complete (7.270000) Aug 4 11:13:20 rhel6-node2 corosync[1454]: [CMAN ] lost contact with quorum device Aug 4 11:13:20 rhel6-node2 corosync[1454]: [QUORUM] Members[8]: 1 2 3 4 5 6 7 8 Aug 4 11:13:20 rhel6-node2 fenced[1676]: node_history_cluster_remove no nodeid 0 Aug 4 11:13:22 rhel6-node2 corosync[1454]: [QUORUM] Members[8]: 1 2 3 4 5 6 7 8 Aug 4 11:13:25 rhel6-node2 qdiskd[1506]: qdiskd: write (system call) has hung for 2 seconds Aug 4 11:13:25 rhel6-node2 qdiskd[1506]: In 2 more seconds, we will be evicted Aug 4 11:13:26 rhel6-node2 qdiskd[1506]: qdisk cycle took more than 1 second to complete (10.800000) Aug 4 11:13:31 rhel6-node2 kernel: kswapd0: page allocation failure. order:0, mode:0x20 Aug 4 11:13:31 rhel6-node2 kernel: Pid: 26, comm: kswapd0 Not tainted 2.6.32-54.el6.i686 #1 Aug 4 11:13:31 rhel6-node2 kernel: Call Trace: Aug 4 11:13:31 rhel6-node2 kernel: [] ? __alloc_pages_nodemask+0x584/0x760 Aug 4 11:13:31 rhel6-node2 kernel: [] ? get_a_page+0x3d/0x50 [virtio_net] Aug 4 11:13:31 rhel6-node2 kernel: [] ? try_fill_recv+0x113/0x1a0 [virtio_net] Aug 4 11:13:31 rhel6-node2 kernel: [] ? __alloc_skb+0x4f/0x140 Aug 4 11:13:31 rhel6-node2 kernel: [] ? virtnet_poll+0x3d4/0x730 [virtio_net] Aug 4 11:13:31 rhel6-node2 kernel: [] ? handle_irq+0x32/0x60 Aug 4 11:13:31 rhel6-node2 kernel: [] ? net_rx_action+0xde/0x1c0 Aug 4 11:13:31 rhel6-node2 kernel: [] ? __do_softirq+0x8f/0x1b0 Aug 4 11:13:31 rhel6-node2 kernel: [] ? handle_irq+0x32/0x60 Aug 4 11:13:31 rhel6-node2 kernel: [] ? move_native_irq+0x11/0x50 Aug 4 11:13:31 rhel6-node2 kernel: [] ? do_softirq+0x3d/0x50 Aug 4 11:13:31 rhel6-node2 kernel: [] ? irq_exit+0x65/0x70 Aug 4 11:13:31 rhel6-node2 kernel: [] ? do_IRQ+0x50/0xc0 Aug 4 11:13:31 rhel6-node2 kernel: [] ? common_interrupt+0x30/0x38 Aug 4 11:13:31 rhel6-node2 kernel: [] ? ftrace_raw_event_mm_anon_unmap+0x60/0xa0 Aug 4 11:13:31 rhel6-node2 kernel: [] ? shrink_page_list.clone.0+0xaa/0x460 Aug 4 11:13:31 rhel6-node2 kernel: [] ? common_interrupt+0x30/0x38 Aug 4 11:13:31 rhel6-node2 kernel: [] ? shrink_inactive_list+0x2bd/0x630 Aug 4 11:13:31 rhel6-node2 kernel: [] ? __do_softirq+0xb5/0x1b0 Aug 4 11:13:31 rhel6-node2 kernel: [] ? move_native_irq+0x11/0x50 Aug 4 11:13:31 rhel6-node2 kernel: [] ? shrink_zone+0x30d/0x460 Aug 4 11:13:31 rhel6-node2 kernel: [] ? kswapd+0x669/0x840 Aug 4 11:13:31 rhel6-node2 kernel: [] ? isolate_pages_global+0x0/0x220 Aug 4 11:13:31 rhel6-node2 kernel: [] ? autoremove_wake_function+0x0/0x40 Aug 4 11:13:31 rhel6-node2 kernel: [] ? complete+0x40/0x60 Aug 4 11:13:31 rhel6-node2 kernel: [] ? kswapd+0x0/0x840 Aug 4 11:13:31 rhel6-node2 kernel: [] ? kthread+0x74/0x80 Aug 4 11:13:31 rhel6-node2 kernel: [] ? kthread+0x0/0x80 Aug 4 11:13:31 rhel6-node2 kernel: [] ? kernel_thread_helper+0x7/0x10 Aug 4 11:13:31 rhel6-node2 kernel: Mem-Info: Aug 4 11:13:31 rhel6-node2 kernel: DMA per-cpu: Aug 4 11:13:31 rhel6-node2 kernel: CPU 0: hi: 0, btch: 1 usd: 0 Aug 4 11:13:31 rhel6-node2 kernel: Normal per-cpu: Aug 4 11:13:31 rhel6-node2 kernel: CPU 0: hi: 186, btch: 31 usd: 93 Aug 4 11:13:31 rhel6-node2 kernel: active_anon:1196 inactive_anon:1376 isolated_anon:0 Aug 4 11:13:31 rhel6-node2 kernel: active_file:4784 inactive_file:74634 isolated_file:32 Aug 4 11:13:31 rhel6-node2 kernel: unevictable:32485 dirty:6578 writeback:2245 unstable:0 Aug 4 11:13:31 rhel6-node2 kernel: free:739 slab_reclaimable:3026 slab_unreclaimable:3534 Aug 4 11:13:31 rhel6-node2 kernel: mapped:10092 shmem:7186 pagetables:449 bounce:0 Aug 4 11:13:31 rhel6-node2 kernel: DMA free:2004kB min:84kB low:104kB high:124kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:6348kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15844kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:16kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 4 11:13:31 rhel6-node2 kernel: lowmem_reserve[]: 0 492 492 492 Aug 4 11:13:31 rhel6-node2 kernel: Normal free:952kB min:2792kB low:3488kB high:4188kB active_anon:4784kB inactive_anon:5504kB active_file:19136kB inactive_file:292188kB unevictable:129940kB isolated(anon):0kB isolated(file):128kB present:503916kB mlocked:129940kB dirty:26312kB writeback:8980kB mapped:40368kB shmem:28744kB slab_reclaimable:12104kB slab_unreclaimable:14136kB kernel_stack:1040kB pagetables:1796kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:64 all_unreclaimable? no Aug 4 11:13:31 rhel6-node2 kernel: lowmem_reserve[]: 0 0 0 0 Aug 4 11:13:31 rhel6-node2 kernel: DMA: 1*4kB 0*8kB 1*16kB 0*32kB 1*64kB 1*128kB 1*256kB 1*512kB 1*1024kB 0*2048kB 0*4096kB = 2004kB Aug 4 11:13:31 rhel6-node2 kernel: Normal: 238*4kB 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 952kB Aug 4 11:13:31 rhel6-node2 kernel: 88520 total pagecache pages Aug 4 11:13:31 rhel6-node2 kernel: 0 pages in swap cache Aug 4 11:13:31 rhel6-node2 kernel: Swap cache stats: add 0, delete 0, find 0/0 Aug 4 11:13:31 rhel6-node2 kernel: Free swap = 1048568kB Aug 4 11:13:31 rhel6-node2 kernel: Total swap = 1048568kB Aug 4 11:13:31 rhel6-node2 kernel: 131066 pages RAM Aug 4 11:13:31 rhel6-node2 kernel: 0 pages HighMem Aug 4 11:13:31 rhel6-node2 kernel: 3162 pages reserved Aug 4 11:13:31 rhel6-node2 kernel: 59989 pages shared Aug 4 11:13:31 rhel6-node2 kernel: 75227 pages non-shared Aug 4 11:13:31 rhel6-node2 kernel: kswapd0: page allocation failure. order:0, mode:0x20 Aug 4 11:13:31 rhel6-node2 kernel: Pid: 26, comm: kswapd0 Not tainted 2.6.32-54.el6.i686 #1 Aug 4 11:13:31 rhel6-node2 kernel: Call Trace: Aug 4 11:13:31 rhel6-node2 kernel: [] ? __alloc_pages_nodemask+0x584/0x760 Aug 4 11:13:31 rhel6-node2 kernel: [] ? cache_alloc_refill+0x2e1/0x540 Aug 4 11:13:31 rhel6-node2 kernel: [] ? __kmalloc+0x142/0x180 Aug 4 11:13:31 rhel6-node2 kernel: [] ? __alloc_skb+0x4f/0x140 Aug 4 11:13:31 rhel6-node2 kernel: [] ? __alloc_skb+0x4f/0x140 Aug 4 11:13:31 rhel6-node2 kernel: [] ? __netdev_alloc_skb+0x1b/0x40 Aug 4 11:13:31 rhel6-node2 kernel: [] ? virtnet_poll+0x20b/0x730 [virtio_net] Aug 4 11:13:31 rhel6-node2 kernel: [] ? handle_irq+0x32/0x60 Aug 4 11:13:31 rhel6-node2 kernel: [] ? net_rx_action+0xde/0x1c0 Aug 4 11:13:31 rhel6-node2 kernel: [] ? __do_softirq+0x8f/0x1b0 Aug 4 11:13:31 rhel6-node2 kernel: [] ? move_native_irq+0x11/0x50 Aug 4 11:13:31 rhel6-node2 kernel: [] ? do_softirq+0x3d/0x50 Aug 4 11:13:31 rhel6-node2 kernel: [] ? irq_exit+0x65/0x70 Aug 4 11:13:31 rhel6-node2 kernel: [] ? do_IRQ+0x50/0xc0 Aug 4 11:13:31 rhel6-node2 kernel: [] ? common_interrupt+0x30/0x38 Aug 4 11:13:31 rhel6-node2 kernel: [] ? gfs2_releasepage+0x0/0x200 [gfs2] Aug 4 11:13:31 rhel6-node2 kernel: [] ? log_write_header+0x450/0x520 [gfs2] Aug 4 11:13:31 rhel6-node2 kernel: [] ? try_to_free_buffers+0x23/0x90 Aug 4 11:13:31 rhel6-node2 kernel: [] ? try_to_release_page+0x23/0x40 Aug 4 11:13:31 rhel6-node2 kernel: [] ? shrink_page_list.clone.0+0x35a/0x460 Aug 4 11:13:31 rhel6-node2 kernel: [] ? common_interrupt+0x30/0x38 Aug 4 11:13:31 rhel6-node2 kernel: [] ? shrink_inactive_list+0x2bd/0x630 Aug 4 11:13:31 rhel6-node2 kernel: [] ? __do_softirq+0xb5/0x1b0 Aug 4 11:13:31 rhel6-node2 kernel: [] ? move_native_irq+0x11/0x50 Aug 4 11:13:31 rhel6-node2 kernel: [] ? shrink_zone+0x30d/0x460 Aug 4 11:13:31 rhel6-node2 kernel: [] ? kswapd+0x669/0x840 Aug 4 11:13:31 rhel6-node2 kernel: [] ? isolate_pages_global+0x0/0x220 Aug 4 11:13:31 rhel6-node2 kernel: [] ? autoremove_wake_function+0x0/0x40 Aug 4 11:13:31 rhel6-node2 kernel: [] ? complete+0x40/0x60 Aug 4 11:13:31 rhel6-node2 kernel: [] ? kswapd+0x0/0x840 Aug 4 11:13:31 rhel6-node2 kernel: [] ? kthread+0x74/0x80 Aug 4 11:13:31 rhel6-node2 kernel: [] ? kthread+0x0/0x80 Aug 4 11:13:31 rhel6-node2 kernel: [] ? kernel_thread_helper+0x7/0x10 Aug 4 11:13:31 rhel6-node2 kernel: Mem-Info: Aug 4 11:13:31 rhel6-node2 kernel: DMA per-cpu: Aug 4 11:13:31 rhel6-node2 kernel: CPU 0: hi: 0, btch: 1 usd: 0 Aug 4 11:13:31 rhel6-node2 kernel: Normal per-cpu: Aug 4 11:13:31 rhel6-node2 kernel: CPU 0: hi: 186, btch: 31 usd: 93 Aug 4 11:13:31 rhel6-node2 kernel: active_anon:1196 inactive_anon:1376 isolated_anon:0 Aug 4 11:13:31 rhel6-node2 kernel: active_file:4784 inactive_file:74634 isolated_file:32 Aug 4 11:13:31 rhel6-node2 kernel: unevictable:32485 dirty:6578 writeback:2245 unstable:0 Aug 4 11:13:31 rhel6-node2 kernel: free:739 slab_reclaimable:3026 slab_unreclaimable:3534 Aug 4 11:13:31 rhel6-node2 kernel: mapped:10092 shmem:7186 pagetables:449 bounce:0 Aug 4 11:13:31 rhel6-node2 kernel: DMA free:2004kB min:84kB low:104kB high:124kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:6348kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15844kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:16kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 4 11:13:31 rhel6-node2 kernel: lowmem_reserve[]: 0 492 492 492 Aug 4 11:13:31 rhel6-node2 kernel: Normal free:952kB min:2792kB low:3488kB high:4188kB active_anon:4784kB inactive_anon:5504kB active_file:19136kB inactive_file:292188kB unevictable:129940kB isolated(anon):0kB isolated(file):128kB present:503916kB mlocked:129940kB dirty:26312kB writeback:8980kB mapped:40368kB shmem:28744kB slab_reclaimable:12104kB slab_unreclaimable:14136kB kernel_stack:1040kB pagetables:1796kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:64 all_unreclaimable? no Aug 4 11:13:31 rhel6-node2 kernel: lowmem_reserve[]: 0 0 0 0 Aug 4 11:13:31 rhel6-node2 kernel: DMA: 1*4kB 0*8kB 1*16kB 0*32kB 1*64kB 1*128kB 1*256kB 1*512kB 1*1024kB 0*2048kB 0*4096kB = 2004kB Aug 4 11:13:31 rhel6-node2 kernel: Normal: 238*4kB 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 952kB Aug 4 11:13:31 rhel6-node2 kernel: 88520 total pagecache pages Aug 4 11:13:31 rhel6-node2 kernel: 0 pages in swap cache Aug 4 11:13:31 rhel6-node2 kernel: Swap cache stats: add 0, delete 0, find 0/0 Aug 4 11:13:31 rhel6-node2 kernel: Free swap = 1048568kB Aug 4 11:13:31 rhel6-node2 kernel: Total swap = 1048568kB Aug 4 11:13:31 rhel6-node2 kernel: 131066 pages RAM Aug 4 11:13:31 rhel6-node2 kernel: 0 pages HighMem Aug 4 11:13:31 rhel6-node2 kernel: 3162 pages reserved Aug 4 11:13:31 rhel6-node2 kernel: 59989 pages shared Aug 4 11:13:31 rhel6-node2 kernel: 75227 pages non-shared Aug 4 11:13:31 rhel6-node2 kernel: kswapd0: page allocation failure. order:0, mode:0x20 Aug 4 11:13:31 rhel6-node2 kernel: Pid: 26, comm: kswapd0 Not tainted 2.6.32-54.el6.i686 #1 Aug 4 11:13:31 rhel6-node2 kernel: Call Trace: Aug 4 11:13:31 rhel6-node2 kernel: [] ? __alloc_pages_nodemask+0x584/0x760 Aug 4 11:13:31 rhel6-node2 kernel: [] ? cache_alloc_refill+0x2e1/0x540 Aug 4 11:13:31 rhel6-node2 kernel: [] ? __kmalloc+0x142/0x180 Aug 4 11:13:31 rhel6-node2 kernel: [] ? __alloc_skb+0x4f/0x140 Aug 4 11:13:31 rhel6-node2 kernel: [] ? __alloc_skb+0x4f/0x140 Aug 4 11:13:31 rhel6-node2 kernel: [] ? __netdev_alloc_skb+0x1b/0x40 Aug 4 11:13:31 rhel6-node2 kernel: [] ? virtnet_poll+0x20b/0x730 [virtio_net] Aug 4 11:13:31 rhel6-node2 kernel: [] ? handle_irq+0x32/0x60 Aug 4 11:13:31 rhel6-node2 kernel: [] ? net_rx_action+0xde/0x1c0 Aug 4 11:13:31 rhel6-node2 kernel: [] ? __do_softirq+0x8f/0x1b0 Aug 4 11:13:31 rhel6-node2 kernel: [] ? move_native_irq+0x11/0x50 Aug 4 11:13:31 rhel6-node2 kernel: [] ? do_softirq+0x3d/0x50 Aug 4 11:13:31 rhel6-node2 kernel: [] ? irq_exit+0x65/0x70 Aug 4 11:13:31 rhel6-node2 kernel: [] ? do_IRQ+0x50/0xc0 Aug 4 11:13:31 rhel6-node2 kernel: [] ? common_interrupt+0x30/0x38 Aug 4 11:13:31 rhel6-node2 kernel: [] ? gfs2_releasepage+0x0/0x200 [gfs2] Aug 4 11:13:31 rhel6-node2 kernel: [] ? log_write_header+0x450/0x520 [gfs2] Aug 4 11:13:31 rhel6-node2 kernel: [] ? try_to_free_buffers+0x23/0x90 Aug 4 11:13:31 rhel6-node2 kernel: [] ? try_to_release_page+0x23/0x40 Aug 4 11:13:31 rhel6-node2 kernel: [] ? shrink_page_list.clone.0+0x35a/0x460 Aug 4 11:13:31 rhel6-node2 kernel: [] ? common_interrupt+0x30/0x38 Aug 4 11:13:31 rhel6-node2 kernel: [] ? shrink_inactive_list+0x2bd/0x630 Aug 4 11:13:31 rhel6-node2 kernel: [] ? __do_softirq+0xb5/0x1b0 Aug 4 11:13:31 rhel6-node2 kernel: [] ? move_native_irq+0x11/0x50 Aug 4 11:13:31 rhel6-node2 kernel: [] ? shrink_zone+0x30d/0x460 Aug 4 11:13:31 rhel6-node2 kernel: [] ? kswapd+0x669/0x840 Aug 4 11:13:31 rhel6-node2 kernel: [] ? isolate_pages_global+0x0/0x220 Aug 4 11:13:31 rhel6-node2 kernel: [] ? autoremove_wake_function+0x0/0x40 Aug 4 11:13:31 rhel6-node2 kernel: [] ? complete+0x40/0x60 Aug 4 11:13:31 rhel6-node2 kernel: [] ? kswapd+0x0/0x840 Aug 4 11:13:31 rhel6-node2 kernel: [] ? kthread+0x74/0x80 Aug 4 11:13:31 rhel6-node2 kernel: [] ? kthread+0x0/0x80 Aug 4 11:13:31 rhel6-node2 kernel: [] ? kernel_thread_helper+0x7/0x10 Aug 4 11:13:31 rhel6-node2 kernel: Mem-Info: Aug 4 11:13:31 rhel6-node2 kernel: DMA per-cpu: Aug 4 11:13:31 rhel6-node2 kernel: CPU 0: hi: 0, btch: 1 usd: 0 Aug 4 11:13:31 rhel6-node2 kernel: Normal per-cpu: Aug 4 11:13:31 rhel6-node2 kernel: CPU 0: hi: 186, btch: 31 usd: 93 Aug 4 11:13:31 rhel6-node2 kernel: active_anon:1196 inactive_anon:1376 isolated_anon:0 Aug 4 11:13:31 rhel6-node2 kernel: active_file:4784 inactive_file:74634 isolated_file:32 Aug 4 11:13:31 rhel6-node2 kernel: unevictable:32485 dirty:6578 writeback:2245 unstable:0 Aug 4 11:13:31 rhel6-node2 kernel: free:739 slab_reclaimable:3026 slab_unreclaimable:3534 Aug 4 11:13:31 rhel6-node2 kernel: mapped:10092 shmem:7186 pagetables:449 bounce:0 Aug 4 11:13:31 rhel6-node2 kernel: DMA free:2004kB min:84kB low:104kB high:124kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:6348kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15844kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:16kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 4 11:13:31 rhel6-node2 kernel: lowmem_reserve[]: 0 492 492 492 Aug 4 11:13:31 rhel6-node2 kernel: Normal free:952kB min:2792kB low:3488kB high:4188kB active_anon:4784kB inactive_anon:5504kB active_file:19136kB inactive_file:292188kB unevictable:129940kB isolated(anon):0kB isolated(file):128kB present:503916kB mlocked:129940kB dirty:26312kB writeback:8980kB mapped:40368kB shmem:28744kB slab_reclaimable:12104kB slab_unreclaimable:14136kB kernel_stack:1040kB pagetables:1796kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:64 all_unreclaimable? no Aug 4 11:13:31 rhel6-node2 kernel: lowmem_reserve[]: 0 0 0 0 Aug 4 11:13:31 rhel6-node2 kernel: DMA: 1*4kB 0*8kB 1*16kB 0*32kB 1*64kB 1*128kB 1*256kB 1*512kB 1*1024kB 0*2048kB 0*4096kB = 2004kB Aug 4 11:13:31 rhel6-node2 kernel: Normal: 238*4kB 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 952kB Aug 4 11:13:31 rhel6-node2 kernel: 88520 total pagecache pages Aug 4 11:13:31 rhel6-node2 kernel: 0 pages in swap cache Aug 4 11:13:31 rhel6-node2 kernel: Swap cache stats: add 0, delete 0, find 0/0 Aug 4 11:13:31 rhel6-node2 kernel: Free swap = 1048568kB Aug 4 11:13:31 rhel6-node2 kernel: Total swap = 1048568kB Aug 4 11:13:31 rhel6-node2 kernel: 131066 pages RAM Aug 4 11:13:31 rhel6-node2 kernel: 0 pages HighMem Aug 4 11:13:31 rhel6-node2 kernel: 3162 pages reserved Aug 4 11:13:31 rhel6-node2 kernel: 59989 pages shared Aug 4 11:13:31 rhel6-node2 kernel: 75227 pages non-shared Aug 4 11:13:31 rhel6-node2 kernel: kswapd0: page allocation failure. order:0, mode:0x20 Aug 4 11:13:31 rhel6-node2 kernel: Pid: 26, comm: kswapd0 Not tainted 2.6.32-54.el6.i686 #1 Aug 4 11:13:31 rhel6-node2 kernel: Call Trace: Aug 4 11:13:31 rhel6-node2 kernel: [] ? __alloc_pages_nodemask+0x584/0x760 Aug 4 11:13:31 rhel6-node2 kernel: [] ? cache_alloc_refill+0x2e1/0x540 Aug 4 11:13:31 rhel6-node2 kernel: [] ? __kmalloc+0x142/0x180 Aug 4 11:13:31 rhel6-node2 kernel: [] ? __alloc_skb+0x4f/0x140 Aug 4 11:13:31 rhel6-node2 kernel: [] ? __alloc_skb+0x4f/0x140 Aug 4 11:13:31 rhel6-node2 kernel: [] ? __netdev_alloc_skb+0x1b/0x40 Aug 4 11:13:31 rhel6-node2 kernel: [] ? virtnet_poll+0x20b/0x730 [virtio_net] Aug 4 11:13:31 rhel6-node2 kernel: [] ? handle_irq+0x32/0x60 Aug 4 11:13:31 rhel6-node2 kernel: [] ? net_rx_action+0xde/0x1c0 Aug 4 11:13:31 rhel6-node2 kernel: [] ? __do_softirq+0x8f/0x1b0 Aug 4 11:13:31 rhel6-node2 kernel: [] ? move_native_irq+0x11/0x50 Aug 4 11:13:31 rhel6-node2 kernel: [] ? do_softirq+0x3d/0x50 Aug 4 11:13:31 rhel6-node2 kernel: [] ? irq_exit+0x65/0x70 Aug 4 11:13:31 rhel6-node2 kernel: [] ? do_IRQ+0x50/0xc0 Aug 4 11:13:31 rhel6-node2 kernel: [] ? common_interrupt+0x30/0x38 Aug 4 11:13:31 rhel6-node2 kernel: [] ? gfs2_releasepage+0x0/0x200 [gfs2] Aug 4 11:13:31 rhel6-node2 kernel: [] ? log_write_header+0x450/0x520 [gfs2] Aug 4 11:13:31 rhel6-node2 kernel: [] ? try_to_free_buffers+0x23/0x90 Aug 4 11:13:31 rhel6-node2 kernel: [] ? try_to_release_page+0x23/0x40 Aug 4 11:13:31 rhel6-node2 kernel: [] ? shrink_page_list.clone.0+0x35a/0x460 Aug 4 11:13:31 rhel6-node2 kernel: [] ? common_interrupt+0x30/0x38 Aug 4 11:13:31 rhel6-node2 kernel: [] ? shrink_inactive_list+0x2bd/0x630 Aug 4 11:13:31 rhel6-node2 kernel: [] ? __do_softirq+0xb5/0x1b0 Aug 4 11:13:31 rhel6-node2 kernel: [] ? move_native_irq+0x11/0x50 Aug 4 11:13:31 rhel6-node2 kernel: [] ? shrink_zone+0x30d/0x460 Aug 4 11:13:31 rhel6-node2 kernel: [] ? kswapd+0x669/0x840 Aug 4 11:13:31 rhel6-node2 kernel: [] ? isolate_pages_global+0x0/0x220 Aug 4 11:13:31 rhel6-node2 kernel: [] ? autoremove_wake_function+0x0/0x40 Aug 4 11:13:31 rhel6-node2 kernel: [] ? complete+0x40/0x60 Aug 4 11:13:31 rhel6-node2 kernel: [] ? kswapd+0x0/0x840 Aug 4 11:13:31 rhel6-node2 kernel: [] ? kthread+0x74/0x80 Aug 4 11:13:31 rhel6-node2 kernel: [] ? kthread+0x0/0x80 Aug 4 11:13:31 rhel6-node2 kernel: [] ? kernel_thread_helper+0x7/0x10 Aug 4 11:13:31 rhel6-node2 kernel: Mem-Info: Aug 4 11:13:31 rhel6-node2 kernel: DMA per-cpu: Aug 4 11:13:31 rhel6-node2 kernel: CPU 0: hi: 0, btch: 1 usd: 0 Aug 4 11:13:31 rhel6-node2 kernel: Normal per-cpu: Aug 4 11:13:31 rhel6-node2 kernel: CPU 0: hi: 186, btch: 31 usd: 93 Aug 4 11:13:31 rhel6-node2 kernel: active_anon:1196 inactive_anon:1376 isolated_anon:0 Aug 4 11:13:31 rhel6-node2 kernel: active_file:4784 inactive_file:74634 isolated_file:32 Aug 4 11:13:31 rhel6-node2 kernel: unevictable:32485 dirty:6578 writeback:2245 unstable:0 Aug 4 11:13:31 rhel6-node2 kernel: free:739 slab_reclaimable:3026 slab_unreclaimable:3534 Aug 4 11:13:31 rhel6-node2 kernel: mapped:10092 shmem:7186 pagetables:449 bounce:0 Aug 4 11:13:31 rhel6-node2 kernel: DMA free:2004kB min:84kB low:104kB high:124kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:6348kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15844kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:16kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 4 11:13:31 rhel6-node2 kernel: lowmem_reserve[]: 0 492 492 492 Aug 4 11:13:31 rhel6-node2 kernel: Normal free:952kB min:2792kB low:3488kB high:4188kB active_anon:4784kB inactive_anon:5504kB active_file:19136kB inactive_file:292188kB unevictable:129940kB isolated(anon):0kB isolated(file):128kB present:503916kB mlocked:129940kB dirty:26312kB writeback:8980kB mapped:40368kB shmem:28744kB slab_reclaimable:12104kB slab_unreclaimable:14136kB kernel_stack:1040kB pagetables:1796kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:64 all_unreclaimable? no Aug 4 11:13:31 rhel6-node2 kernel: lowmem_reserve[]: 0 0 0 0 Aug 4 11:13:31 rhel6-node2 kernel: DMA: 1*4kB 0*8kB 1*16kB 0*32kB 1*64kB 1*128kB 1*256kB 1*512kB 1*1024kB 0*2048kB 0*4096kB = 2004kB Aug 4 11:13:31 rhel6-node2 kernel: Normal: 238*4kB 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 952kB Aug 4 11:13:31 rhel6-node2 kernel: 88520 total pagecache pages Aug 4 11:13:31 rhel6-node2 kernel: 0 pages in swap cache Aug 4 11:13:31 rhel6-node2 kernel: Swap cache stats: add 0, delete 0, find 0/0 Aug 4 11:13:31 rhel6-node2 kernel: Free swap = 1048568kB Aug 4 11:13:31 rhel6-node2 kernel: Total swap = 1048568kB Aug 4 11:13:31 rhel6-node2 kernel: 131066 pages RAM Aug 4 11:13:31 rhel6-node2 kernel: 0 pages HighMem Aug 4 11:13:31 rhel6-node2 kernel: 3162 pages reserved Aug 4 11:13:31 rhel6-node2 kernel: 59989 pages shared Aug 4 11:13:31 rhel6-node2 kernel: 75227 pages non-shared Aug 4 11:13:31 rhel6-node2 kernel: kswapd0: page allocation failure. order:0, mode:0x20 Aug 4 11:13:31 rhel6-node2 kernel: Pid: 26, comm: kswapd0 Not tainted 2.6.32-54.el6.i686 #1 Aug 4 11:13:31 rhel6-node2 kernel: Call Trace: Aug 4 11:13:31 rhel6-node2 kernel: [] ? __alloc_pages_nodemask+0x584/0x760 Aug 4 11:13:31 rhel6-node2 kernel: [] ? cache_alloc_refill+0x2e1/0x540 Aug 4 11:13:31 rhel6-node2 kernel: [] ? __kmalloc+0x142/0x180 Aug 4 11:13:31 rhel6-node2 kernel: [] ? __alloc_skb+0x4f/0x140 Aug 4 11:13:31 rhel6-node2 kernel: [] ? __alloc_skb+0x4f/0x140 Aug 4 11:13:31 rhel6-node2 kernel: [] ? __netdev_alloc_skb+0x1b/0x40 Aug 4 11:13:31 rhel6-node2 kernel: [] ? virtnet_poll+0x20b/0x730 [virtio_net] Aug 4 11:13:31 rhel6-node2 kernel: [] ? handle_irq+0x32/0x60 Aug 4 11:13:31 rhel6-node2 kernel: [] ? net_rx_action+0xde/0x1c0 Aug 4 11:13:31 rhel6-node2 kernel: [] ? __do_softirq+0x8f/0x1b0 Aug 4 11:13:31 rhel6-node2 kernel: [] ? move_native_irq+0x11/0x50 Aug 4 11:13:31 rhel6-node2 kernel: [] ? do_softirq+0x3d/0x50 Aug 4 11:13:31 rhel6-node2 kernel: [] ? irq_exit+0x65/0x70 Aug 4 11:13:31 rhel6-node2 kernel: [] ? do_IRQ+0x50/0xc0 Aug 4 11:13:31 rhel6-node2 kernel: [] ? common_interrupt+0x30/0x38 Aug 4 11:13:31 rhel6-node2 kernel: [] ? gfs2_releasepage+0x0/0x200 [gfs2] Aug 4 11:13:31 rhel6-node2 kernel: [] ? log_write_header+0x450/0x520 [gfs2] Aug 4 11:13:31 rhel6-node2 kernel: [] ? try_to_free_buffers+0x23/0x90 Aug 4 11:13:31 rhel6-node2 kernel: [] ? try_to_release_page+0x23/0x40 Aug 4 11:13:31 rhel6-node2 kernel: [] ? shrink_page_list.clone.0+0x35a/0x460 Aug 4 11:13:31 rhel6-node2 kernel: [] ? common_interrupt+0x30/0x38 Aug 4 11:13:31 rhel6-node2 kernel: [] ? shrink_inactive_list+0x2bd/0x630 Aug 4 11:13:31 rhel6-node2 kernel: [] ? __do_softirq+0xb5/0x1b0 Aug 4 11:13:31 rhel6-node2 kernel: [] ? move_native_irq+0x11/0x50 Aug 4 11:13:31 rhel6-node2 kernel: [] ? shrink_zone+0x30d/0x460 Aug 4 11:13:31 rhel6-node2 kernel: [] ? kswapd+0x669/0x840 Aug 4 11:13:31 rhel6-node2 kernel: [] ? isolate_pages_global+0x0/0x220 Aug 4 11:13:31 rhel6-node2 kernel: [] ? autoremove_wake_function+0x0/0x40 Aug 4 11:13:31 rhel6-node2 kernel: [] ? complete+0x40/0x60 Aug 4 11:13:31 rhel6-node2 kernel: [] ? kswapd+0x0/0x840 Aug 4 11:13:31 rhel6-node2 kernel: [] ? kthread+0x74/0x80 Aug 4 11:13:31 rhel6-node2 kernel: [] ? kthread+0x0/0x80 Aug 4 11:13:31 rhel6-node2 kernel: [] ? kernel_thread_helper+0x7/0x10 Aug 4 11:13:31 rhel6-node2 kernel: Mem-Info: Aug 4 11:13:31 rhel6-node2 kernel: DMA per-cpu: Aug 4 11:13:31 rhel6-node2 kernel: CPU 0: hi: 0, btch: 1 usd: 0 Aug 4 11:13:31 rhel6-node2 kernel: Normal per-cpu: Aug 4 11:13:31 rhel6-node2 kernel: CPU 0: hi: 186, btch: 31 usd: 93 Aug 4 11:13:31 rhel6-node2 kernel: active_anon:1196 inactive_anon:1376 isolated_anon:0 Aug 4 11:13:31 rhel6-node2 kernel: active_file:4784 inactive_file:74634 isolated_file:32 Aug 4 11:13:31 rhel6-node2 kernel: unevictable:32485 dirty:6578 writeback:2245 unstable:0 Aug 4 11:13:31 rhel6-node2 kernel: free:739 slab_reclaimable:3026 slab_unreclaimable:3534 Aug 4 11:13:31 rhel6-node2 kernel: mapped:10092 shmem:7186 pagetables:449 bounce:0 Aug 4 11:13:31 rhel6-node2 kernel: DMA free:2004kB min:84kB low:104kB high:124kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:6348kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15844kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:16kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 4 11:13:31 rhel6-node2 kernel: lowmem_reserve[]: 0 492 492 492 Aug 4 11:13:31 rhel6-node2 kernel: Normal free:952kB min:2792kB low:3488kB high:4188kB active_anon:4784kB inactive_anon:5504kB active_file:19136kB inactive_file:292188kB unevictable:129940kB isolated(anon):0kB isolated(file):128kB present:503916kB mlocked:129940kB dirty:26312kB writeback:8980kB mapped:40368kB shmem:28744kB slab_reclaimable:12104kB slab_unreclaimable:14136kB kernel_stack:1040kB pagetables:1796kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:64 all_unreclaimable? no Aug 4 11:13:31 rhel6-node2 kernel: lowmem_reserve[]: 0 0 0 0 Aug 4 11:13:31 rhel6-node2 kernel: DMA: 1*4kB 0*8kB 1*16kB 0*32kB 1*64kB 1*128kB 1*256kB 1*512kB 1*1024kB 0*2048kB 0*4096kB = 2004kB Aug 4 11:13:31 rhel6-node2 kernel: Normal: 238*4kB 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 952kB Aug 4 11:13:31 rhel6-node2 kernel: 88520 total pagecache pages Aug 4 11:13:31 rhel6-node2 kernel: 0 pages in swap cache Aug 4 11:13:31 rhel6-node2 kernel: Swap cache stats: add 0, delete 0, find 0/0 Aug 4 11:13:31 rhel6-node2 kernel: Free swap = 1048568kB Aug 4 11:13:31 rhel6-node2 kernel: Total swap = 1048568kB Aug 4 11:13:31 rhel6-node2 kernel: 131066 pages RAM Aug 4 11:13:31 rhel6-node2 kernel: 0 pages HighMem Aug 4 11:13:31 rhel6-node2 kernel: 3162 pages reserved Aug 4 11:13:31 rhel6-node2 kernel: 59989 pages shared Aug 4 11:13:31 rhel6-node2 kernel: 75227 pages non-shared Aug 4 11:13:31 rhel6-node2 kernel: kswapd0: page allocation failure. order:0, mode:0x20 Aug 4 11:13:31 rhel6-node2 kernel: Pid: 26, comm: kswapd0 Not tainted 2.6.32-54.el6.i686 #1 Aug 4 11:13:31 rhel6-node2 kernel: Call Trace: Aug 4 11:13:31 rhel6-node2 kernel: [] ? __alloc_pages_nodemask+0x584/0x760 Aug 4 11:13:31 rhel6-node2 kernel: [] ? cache_alloc_refill+0x2e1/0x540 Aug 4 11:13:31 rhel6-node2 kernel: [] ? __kmalloc+0x142/0x180 Aug 4 11:13:31 rhel6-node2 kernel: [] ? __alloc_skb+0x4f/0x140 Aug 4 11:13:31 rhel6-node2 kernel: [] ? __alloc_skb+0x4f/0x140 Aug 4 11:13:31 rhel6-node2 kernel: [] ? __netdev_alloc_skb+0x1b/0x40 Aug 4 11:13:31 rhel6-node2 kernel: [] ? virtnet_poll+0x20b/0x730 [virtio_net] Aug 4 11:13:31 rhel6-node2 kernel: [] ? handle_irq+0x32/0x60 Aug 4 11:13:31 rhel6-node2 kernel: [] ? net_rx_action+0xde/0x1c0 Aug 4 11:13:31 rhel6-node2 kernel: [] ? __do_softirq+0x8f/0x1b0 Aug 4 11:13:31 rhel6-node2 kernel: [] ? move_native_irq+0x11/0x50 Aug 4 11:13:31 rhel6-node2 kernel: [] ? do_softirq+0x3d/0x50 Aug 4 11:13:31 rhel6-node2 kernel: [] ? irq_exit+0x65/0x70 Aug 4 11:13:31 rhel6-node2 kernel: [] ? do_IRQ+0x50/0xc0 Aug 4 11:13:31 rhel6-node2 kernel: [] ? common_interrupt+0x30/0x38 Aug 4 11:13:31 rhel6-node2 kernel: [] ? gfs2_releasepage+0x0/0x200 [gfs2] Aug 4 11:13:31 rhel6-node2 kernel: [] ? log_write_header+0x450/0x520 [gfs2] Aug 4 11:13:31 rhel6-node2 kernel: [] ? try_to_free_buffers+0x23/0x90 Aug 4 11:13:31 rhel6-node2 kernel: [] ? try_to_release_page+0x23/0x40 Aug 4 11:13:31 rhel6-node2 kernel: [] ? shrink_page_list.clone.0+0x35a/0x460 Aug 4 11:13:31 rhel6-node2 kernel: [] ? common_interrupt+0x30/0x38 Aug 4 11:13:31 rhel6-node2 kernel: [] ? shrink_inactive_list+0x2bd/0x630 Aug 4 11:13:31 rhel6-node2 kernel: [] ? __do_softirq+0xb5/0x1b0 Aug 4 11:13:31 rhel6-node2 kernel: [] ? move_native_irq+0x11/0x50 Aug 4 11:13:31 rhel6-node2 kernel: [] ? shrink_zone+0x30d/0x460 Aug 4 11:13:31 rhel6-node2 kernel: [] ? kswapd+0x669/0x840 Aug 4 11:13:31 rhel6-node2 kernel: [] ? isolate_pages_global+0x0/0x220 Aug 4 11:13:31 rhel6-node2 kernel: [] ? autoremove_wake_function+0x0/0x40 Aug 4 11:13:31 rhel6-node2 kernel: [] ? complete+0x40/0x60 Aug 4 11:13:31 rhel6-node2 kernel: [] ? kswapd+0x0/0x840 Aug 4 11:13:31 rhel6-node2 kernel: [] ? kthread+0x74/0x80 Aug 4 11:13:31 rhel6-node2 kernel: [] ? kthread+0x0/0x80 Aug 4 11:13:31 rhel6-node2 kernel: [] ? kernel_thread_helper+0x7/0x10 Aug 4 11:13:31 rhel6-node2 kernel: Mem-Info: Aug 4 11:13:31 rhel6-node2 kernel: DMA per-cpu: Aug 4 11:13:31 rhel6-node2 kernel: CPU 0: hi: 0, btch: 1 usd: 0 Aug 4 11:13:31 rhel6-node2 kernel: Normal per-cpu: Aug 4 11:13:31 rhel6-node2 kernel: CPU 0: hi: 186, btch: 31 usd: 93 Aug 4 11:13:31 rhel6-node2 kernel: active_anon:1196 inactive_anon:1376 isolated_anon:0 Aug 4 11:13:31 rhel6-node2 kernel: active_file:4784 inactive_file:74634 isolated_file:32 Aug 4 11:13:31 rhel6-node2 kernel: unevictable:32485 dirty:6578 writeback:2245 unstable:0 Aug 4 11:13:31 rhel6-node2 kernel: free:739 slab_reclaimable:3026 slab_unreclaimable:3534 Aug 4 11:13:31 rhel6-node2 kernel: mapped:10092 shmem:7186 pagetables:449 bounce:0 Aug 4 11:13:31 rhel6-node2 kernel: DMA free:2004kB min:84kB low:104kB high:124kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:6348kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15844kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:16kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 4 11:13:31 rhel6-node2 kernel: lowmem_reserve[]: 0 492 492 492 Aug 4 11:13:31 rhel6-node2 kernel: Normal free:952kB min:2792kB low:3488kB high:4188kB active_anon:4784kB inactive_anon:5504kB active_file:19136kB inactive_file:292188kB unevictable:129940kB isolated(anon):0kB isolated(file):128kB present:503916kB mlocked:129940kB dirty:26312kB writeback:8980kB mapped:40368kB shmem:28744kB slab_reclaimable:12104kB slab_unreclaimable:14136kB kernel_stack:1040kB pagetables:1796kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:64 all_unreclaimable? no Aug 4 11:13:31 rhel6-node2 kernel: lowmem_reserve[]: 0 0 0 0 Aug 4 11:13:31 rhel6-node2 kernel: DMA: 1*4kB 0*8kB 1*16kB 0*32kB 1*64kB 1*128kB 1*256kB 1*512kB 1*1024kB 0*2048kB 0*4096kB = 2004kB Aug 4 11:13:31 rhel6-node2 kernel: Normal: 238*4kB 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 952kB Aug 4 11:13:31 rhel6-node2 kernel: 88520 total pagecache pages Aug 4 11:13:31 rhel6-node2 kernel: 0 pages in swap cache Aug 4 11:13:31 rhel6-node2 kernel: Swap cache stats: add 0, delete 0, find 0/0 Aug 4 11:13:31 rhel6-node2 kernel: Free swap = 1048568kB Aug 4 11:13:31 rhel6-node2 kernel: Total swap = 1048568kB Aug 4 11:13:31 rhel6-node2 kernel: 131066 pages RAM Aug 4 11:13:31 rhel6-node2 kernel: 0 pages HighMem Aug 4 11:13:31 rhel6-node2 kernel: 3162 pages reserved Aug 4 11:13:31 rhel6-node2 kernel: 59989 pages shared Aug 4 11:13:31 rhel6-node2 kernel: 75227 pages non-shared Aug 4 11:13:31 rhel6-node2 kernel: kswapd0: page allocation failure. order:0, mode:0x20 Aug 4 11:13:31 rhel6-node2 kernel: Pid: 26, comm: kswapd0 Not tainted 2.6.32-54.el6.i686 #1 Aug 4 11:13:31 rhel6-node2 kernel: Call Trace: Aug 4 11:13:31 rhel6-node2 kernel: [] ? __alloc_pages_nodemask+0x584/0x760 Aug 4 11:13:31 rhel6-node2 kernel: [] ? cache_alloc_refill+0x2e1/0x540 Aug 4 11:13:31 rhel6-node2 kernel: [] ? __kmalloc+0x142/0x180 Aug 4 11:13:31 rhel6-node2 kernel: [] ? __alloc_skb+0x4f/0x140 Aug 4 11:13:31 rhel6-node2 kernel: [] ? __alloc_skb+0x4f/0x140 Aug 4 11:13:31 rhel6-node2 kernel: [] ? __netdev_alloc_skb+0x1b/0x40 Aug 4 11:13:31 rhel6-node2 kernel: [] ? virtnet_poll+0x20b/0x730 [virtio_net] Aug 4 11:13:31 rhel6-node2 kernel: [] ? handle_irq+0x32/0x60 Aug 4 11:13:31 rhel6-node2 kernel: [] ? net_rx_action+0xde/0x1c0 Aug 4 11:13:31 rhel6-node2 kernel: [] ? __do_softirq+0x8f/0x1b0 Aug 4 11:13:31 rhel6-node2 kernel: [] ? move_native_irq+0x11/0x50 Aug 4 11:13:31 rhel6-node2 kernel: [] ? do_softirq+0x3d/0x50 Aug 4 11:13:31 rhel6-node2 kernel: [] ? irq_exit+0x65/0x70 Aug 4 11:13:31 rhel6-node2 kernel: [] ? do_IRQ+0x50/0xc0 Aug 4 11:13:31 rhel6-node2 kernel: [] ? common_interrupt+0x30/0x38 Aug 4 11:13:31 rhel6-node2 kernel: [] ? gfs2_releasepage+0x0/0x200 [gfs2] Aug 4 11:13:31 rhel6-node2 kernel: [] ? log_write_header+0x450/0x520 [gfs2] Aug 4 11:13:31 rhel6-node2 kernel: [] ? try_to_free_buffers+0x23/0x90 Aug 4 11:13:31 rhel6-node2 kernel: [] ? try_to_release_page+0x23/0x40 Aug 4 11:13:31 rhel6-node2 kernel: [] ? shrink_page_list.clone.0+0x35a/0x460 Aug 4 11:13:31 rhel6-node2 kernel: [] ? common_interrupt+0x30/0x38 Aug 4 11:13:31 rhel6-node2 kernel: [] ? shrink_inactive_list+0x2bd/0x630 Aug 4 11:13:31 rhel6-node2 kernel: [] ? __do_softirq+0xb5/0x1b0 Aug 4 11:13:31 rhel6-node2 kernel: [] ? move_native_irq+0x11/0x50 Aug 4 11:13:31 rhel6-node2 kernel: [] ? shrink_zone+0x30d/0x460 Aug 4 11:13:31 rhel6-node2 kernel: [] ? kswapd+0x669/0x840 Aug 4 11:13:31 rhel6-node2 kernel: [] ? isolate_pages_global+0x0/0x220 Aug 4 11:13:31 rhel6-node2 kernel: [] ? autoremove_wake_function+0x0/0x40 Aug 4 11:13:31 rhel6-node2 kernel: [] ? complete+0x40/0x60 Aug 4 11:13:31 rhel6-node2 kernel: [] ? kswapd+0x0/0x840 Aug 4 11:13:31 rhel6-node2 kernel: [] ? kthread+0x74/0x80 Aug 4 11:13:31 rhel6-node2 kernel: [] ? kthread+0x0/0x80 Aug 4 11:13:31 rhel6-node2 kernel: [] ? kernel_thread_helper+0x7/0x10 Aug 4 11:13:31 rhel6-node2 kernel: Mem-Info: Aug 4 11:13:31 rhel6-node2 kernel: DMA per-cpu: Aug 4 11:13:31 rhel6-node2 kernel: CPU 0: hi: 0, btch: 1 usd: 0 Aug 4 11:13:31 rhel6-node2 kernel: Normal per-cpu: Aug 4 11:13:31 rhel6-node2 kernel: CPU 0: hi: 186, btch: 31 usd: 93 Aug 4 11:13:31 rhel6-node2 kernel: active_anon:1196 inactive_anon:1376 isolated_anon:0 Aug 4 11:13:31 rhel6-node2 kernel: active_file:4784 inactive_file:74634 isolated_file:32 Aug 4 11:13:31 rhel6-node2 kernel: unevictable:32485 dirty:6578 writeback:2245 unstable:0 Aug 4 11:13:31 rhel6-node2 kernel: free:739 slab_reclaimable:3026 slab_unreclaimable:3534 Aug 4 11:13:31 rhel6-node2 kernel: mapped:10092 shmem:7186 pagetables:449 bounce:0 Aug 4 11:13:31 rhel6-node2 kernel: DMA free:2004kB min:84kB low:104kB high:124kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:6348kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15844kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:16kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 4 11:13:31 rhel6-node2 kernel: lowmem_reserve[]: 0 492 492 492 Aug 4 11:13:31 rhel6-node2 kernel: Normal free:952kB min:2792kB low:3488kB high:4188kB active_anon:4784kB inactive_anon:5504kB active_file:19136kB inactive_file:292188kB unevictable:129940kB isolated(anon):0kB isolated(file):128kB present:503916kB mlocked:129940kB dirty:26312kB writeback:8980kB mapped:40368kB shmem:28744kB slab_reclaimable:12104kB slab_unreclaimable:14136kB kernel_stack:1040kB pagetables:1796kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:64 all_unreclaimable? no Aug 4 11:13:31 rhel6-node2 kernel: lowmem_reserve[]: 0 0 0 0 Aug 4 11:13:31 rhel6-node2 kernel: DMA: 1*4kB 0*8kB 1*16kB 0*32kB 1*64kB 1*128kB 1*256kB 1*512kB 1*1024kB 0*2048kB 0*4096kB = 2004kB Aug 4 11:13:31 rhel6-node2 kernel: Normal: 238*4kB 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 952kB Aug 4 11:13:31 rhel6-node2 kernel: 88520 total pagecache pages Aug 4 11:13:31 rhel6-node2 kernel: 0 pages in swap cache Aug 4 11:13:31 rhel6-node2 kernel: Swap cache stats: add 0, delete 0, find 0/0 Aug 4 11:13:31 rhel6-node2 kernel: Free swap = 1048568kB Aug 4 11:13:31 rhel6-node2 kernel: Total swap = 1048568kB Aug 4 11:13:31 rhel6-node2 kernel: 131066 pages RAM Aug 4 11:13:31 rhel6-node2 kernel: 0 pages HighMem Aug 4 11:13:31 rhel6-node2 kernel: 3162 pages reserved Aug 4 11:13:31 rhel6-node2 kernel: 59989 pages shared Aug 4 11:13:31 rhel6-node2 kernel: 75227 pages non-shared Aug 4 11:13:31 rhel6-node2 kernel: kswapd0: page allocation failure. order:0, mode:0x20 Aug 4 11:13:31 rhel6-node2 kernel: Pid: 26, comm: kswapd0 Not tainted 2.6.32-54.el6.i686 #1 Aug 4 11:13:31 rhel6-node2 kernel: Call Trace: Aug 4 11:13:31 rhel6-node2 kernel: [] ? __alloc_pages_nodemask+0x584/0x760 Aug 4 11:13:31 rhel6-node2 kernel: [] ? cache_alloc_refill+0x2e1/0x540 Aug 4 11:13:31 rhel6-node2 kernel: [] ? __kmalloc+0x142/0x180 Aug 4 11:13:31 rhel6-node2 kernel: [] ? __alloc_skb+0x4f/0x140 Aug 4 11:13:31 rhel6-node2 kernel: [] ? __alloc_skb+0x4f/0x140 Aug 4 11:13:31 rhel6-node2 kernel: [] ? __netdev_alloc_skb+0x1b/0x40 Aug 4 11:13:31 rhel6-node2 kernel: [] ? virtnet_poll+0x20b/0x730 [virtio_net] Aug 4 11:13:31 rhel6-node2 kernel: [] ? handle_irq+0x32/0x60 Aug 4 11:13:31 rhel6-node2 kernel: [] ? net_rx_action+0xde/0x1c0 Aug 4 11:13:31 rhel6-node2 kernel: [] ? __do_softirq+0x8f/0x1b0 Aug 4 11:13:31 rhel6-node2 kernel: [] ? move_native_irq+0x11/0x50 Aug 4 11:13:31 rhel6-node2 kernel: [] ? do_softirq+0x3d/0x50 Aug 4 11:13:31 rhel6-node2 kernel: [] ? irq_exit+0x65/0x70 Aug 4 11:13:31 rhel6-node2 kernel: [] ? do_IRQ+0x50/0xc0 Aug 4 11:13:31 rhel6-node2 kernel: [] ? common_interrupt+0x30/0x38 Aug 4 11:13:31 rhel6-node2 kernel: [] ? gfs2_releasepage+0x0/0x200 [gfs2] Aug 4 11:13:31 rhel6-node2 kernel: [] ? log_write_header+0x450/0x520 [gfs2] Aug 4 11:13:31 rhel6-node2 kernel: [] ? try_to_free_buffers+0x23/0x90 Aug 4 11:13:31 rhel6-node2 kernel: [] ? try_to_release_page+0x23/0x40 Aug 4 11:13:31 rhel6-node2 kernel: [] ? shrink_page_list.clone.0+0x35a/0x460 Aug 4 11:13:31 rhel6-node2 kernel: [] ? common_interrupt+0x30/0x38 Aug 4 11:13:31 rhel6-node2 kernel: [] ? shrink_inactive_list+0x2bd/0x630 Aug 4 11:13:31 rhel6-node2 kernel: [] ? __do_softirq+0xb5/0x1b0 Aug 4 11:13:31 rhel6-node2 kernel: [] ? move_native_irq+0x11/0x50 Aug 4 11:13:31 rhel6-node2 kernel: [] ? shrink_zone+0x30d/0x460 Aug 4 11:13:31 rhel6-node2 kernel: [] ? kswapd+0x669/0x840 Aug 4 11:13:31 rhel6-node2 kernel: [] ? isolate_pages_global+0x0/0x220 Aug 4 11:13:31 rhel6-node2 kernel: [] ? autoremove_wake_function+0x0/0x40 Aug 4 11:13:31 rhel6-node2 kernel: [] ? complete+0x40/0x60 Aug 4 11:13:31 rhel6-node2 kernel: [] ? kswapd+0x0/0x840 Aug 4 11:13:31 rhel6-node2 kernel: [] ? kthread+0x74/0x80 Aug 4 11:13:31 rhel6-node2 kernel: [] ? kthread+0x0/0x80 Aug 4 11:13:31 rhel6-node2 kernel: [] ? kernel_thread_helper+0x7/0x10 Aug 4 11:13:31 rhel6-node2 kernel: Mem-Info: Aug 4 11:13:31 rhel6-node2 kernel: DMA per-cpu: Aug 4 11:13:31 rhel6-node2 kernel: CPU 0: hi: 0, btch: 1 usd: 0 Aug 4 11:13:31 rhel6-node2 kernel: Normal per-cpu: Aug 4 11:13:31 rhel6-node2 kernel: CPU 0: hi: 186, btch: 31 usd: 93 Aug 4 11:13:31 rhel6-node2 kernel: active_anon:1196 inactive_anon:1376 isolated_anon:0 Aug 4 11:13:31 rhel6-node2 kernel: active_file:4784 inactive_file:74634 isolated_file:32 Aug 4 11:13:31 rhel6-node2 kernel: unevictable:32485 dirty:6578 writeback:2245 unstable:0 Aug 4 11:13:31 rhel6-node2 kernel: free:739 slab_reclaimable:3026 slab_unreclaimable:3534 Aug 4 11:13:31 rhel6-node2 kernel: mapped:10092 shmem:7186 pagetables:449 bounce:0 Aug 4 11:13:31 rhel6-node2 kernel: DMA free:2004kB min:84kB low:104kB high:124kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:6348kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15844kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:16kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 4 11:13:31 rhel6-node2 kernel: lowmem_reserve[]: 0 492 492 492 Aug 4 11:13:31 rhel6-node2 kernel: Normal free:952kB min:2792kB low:3488kB high:4188kB active_anon:4784kB inactive_anon:5504kB active_file:19136kB inactive_file:292188kB unevictable:129940kB isolated(anon):0kB isolated(file):128kB present:503916kB mlocked:129940kB dirty:26312kB writeback:8980kB mapped:40368kB shmem:28744kB slab_reclaimable:12104kB slab_unreclaimable:14136kB kernel_stack:1040kB pagetables:1796kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:64 all_unreclaimable? no Aug 4 11:13:31 rhel6-node2 kernel: lowmem_reserve[]: 0 0 0 0 Aug 4 11:13:31 rhel6-node2 kernel: DMA: 1*4kB 0*8kB 1*16kB 0*32kB 1*64kB 1*128kB 1*256kB 1*512kB 1*1024kB 0*2048kB 0*4096kB = 2004kB Aug 4 11:13:31 rhel6-node2 kernel: Normal: 238*4kB 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 952kB Aug 4 11:13:31 rhel6-node2 kernel: 88520 total pagecache pages Aug 4 11:13:31 rhel6-node2 kernel: 0 pages in swap cache Aug 4 11:13:31 rhel6-node2 kernel: Swap cache stats: add 0, delete 0, find 0/0 Aug 4 11:13:31 rhel6-node2 kernel: Free swap = 1048568kB Aug 4 11:13:31 rhel6-node2 kernel: Total swap = 1048568kB Aug 4 11:13:31 rhel6-node2 kernel: 131066 pages RAM Aug 4 11:13:31 rhel6-node2 kernel: 0 pages HighMem Aug 4 11:13:31 rhel6-node2 kernel: 3162 pages reserved Aug 4 11:13:31 rhel6-node2 kernel: 59989 pages shared Aug 4 11:13:31 rhel6-node2 kernel: 75227 pages non-shared Aug 4 11:13:31 rhel6-node2 kernel: kswapd0: page allocation failure. order:0, mode:0x20 Aug 4 11:13:31 rhel6-node2 kernel: Pid: 26, comm: kswapd0 Not tainted 2.6.32-54.el6.i686 #1 Aug 4 11:13:31 rhel6-node2 kernel: Call Trace: Aug 4 11:13:31 rhel6-node2 kernel: [] ? __alloc_pages_nodemask+0x584/0x760 Aug 4 11:13:31 rhel6-node2 kernel: [] ? cache_alloc_refill+0x2e1/0x540 Aug 4 11:13:31 rhel6-node2 kernel: [] ? __kmalloc+0x142/0x180 Aug 4 11:13:31 rhel6-node2 kernel: [] ? __alloc_skb+0x4f/0x140 Aug 4 11:13:31 rhel6-node2 kernel: [] ? __alloc_skb+0x4f/0x140 Aug 4 11:13:31 rhel6-node2 kernel: [] ? __netdev_alloc_skb+0x1b/0x40 Aug 4 11:13:31 rhel6-node2 kernel: [] ? virtnet_poll+0x20b/0x730 [virtio_net] Aug 4 11:13:31 rhel6-node2 kernel: [] ? handle_irq+0x32/0x60 Aug 4 11:13:31 rhel6-node2 kernel: [] ? net_rx_action+0xde/0x1c0 Aug 4 11:13:31 rhel6-node2 kernel: [] ? __do_softirq+0x8f/0x1b0 Aug 4 11:13:31 rhel6-node2 kernel: [] ? move_native_irq+0x11/0x50 Aug 4 11:13:31 rhel6-node2 kernel: [] ? do_softirq+0x3d/0x50 Aug 4 11:13:31 rhel6-node2 kernel: [] ? irq_exit+0x65/0x70 Aug 4 11:13:31 rhel6-node2 kernel: [] ? do_IRQ+0x50/0xc0 Aug 4 11:13:31 rhel6-node2 kernel: [] ? common_interrupt+0x30/0x38 Aug 4 11:13:31 rhel6-node2 kernel: [] ? gfs2_releasepage+0x0/0x200 [gfs2] Aug 4 11:13:31 rhel6-node2 kernel: [] ? log_write_header+0x450/0x520 [gfs2] Aug 4 11:13:31 rhel6-node2 kernel: [] ? try_to_free_buffers+0x23/0x90 Aug 4 11:13:31 rhel6-node2 kernel: [] ? try_to_release_page+0x23/0x40 Aug 4 11:13:31 rhel6-node2 kernel: [] ? shrink_page_list.clone.0+0x35a/0x460 Aug 4 11:13:31 rhel6-node2 kernel: [] ? common_interrupt+0x30/0x38 Aug 4 11:13:31 rhel6-node2 kernel: [] ? shrink_inactive_list+0x2bd/0x630 Aug 4 11:13:31 rhel6-node2 kernel: [] ? __do_softirq+0xb5/0x1b0 Aug 4 11:13:31 rhel6-node2 kernel: [] ? move_native_irq+0x11/0x50 Aug 4 11:13:31 rhel6-node2 kernel: [] ? shrink_zone+0x30d/0x460 Aug 4 11:13:31 rhel6-node2 kernel: [] ? kswapd+0x669/0x840 Aug 4 11:13:31 rhel6-node2 kernel: [] ? isolate_pages_global+0x0/0x220 Aug 4 11:13:31 rhel6-node2 kernel: [] ? autoremove_wake_function+0x0/0x40 Aug 4 11:13:31 rhel6-node2 kernel: [] ? complete+0x40/0x60 Aug 4 11:13:31 rhel6-node2 kernel: [] ? kswapd+0x0/0x840 Aug 4 11:13:31 rhel6-node2 kernel: [] ? kthread+0x74/0x80 Aug 4 11:13:31 rhel6-node2 kernel: [] ? kthread+0x0/0x80 Aug 4 11:13:31 rhel6-node2 kernel: [] ? kernel_thread_helper+0x7/0x10 Aug 4 11:13:31 rhel6-node2 kernel: Mem-Info: Aug 4 11:13:31 rhel6-node2 kernel: DMA per-cpu: Aug 4 11:13:31 rhel6-node2 kernel: CPU 0: hi: 0, btch: 1 usd: 0 Aug 4 11:13:31 rhel6-node2 kernel: Normal per-cpu: Aug 4 11:13:31 rhel6-node2 kernel: CPU 0: hi: 186, btch: 31 usd: 93 Aug 4 11:13:31 rhel6-node2 kernel: active_anon:1196 inactive_anon:1376 isolated_anon:0 Aug 4 11:13:31 rhel6-node2 kernel: active_file:4784 inactive_file:74634 isolated_file:32 Aug 4 11:13:31 rhel6-node2 kernel: unevictable:32485 dirty:6578 writeback:2245 unstable:0 Aug 4 11:13:31 rhel6-node2 kernel: free:739 slab_reclaimable:3026 slab_unreclaimable:3534 Aug 4 11:13:31 rhel6-node2 kernel: mapped:10092 shmem:7186 pagetables:449 bounce:0 Aug 4 11:13:31 rhel6-node2 kernel: DMA free:2004kB min:84kB low:104kB high:124kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:6348kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15844kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:16kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Aug 4 11:13:31 rhel6-node2 kernel: lowmem_reserve[]: 0 492 492 492 Aug 4 11:13:31 rhel6-node2 kernel: Normal free:952kB min:2792kB low:3488kB high:4188kB active_anon:4784kB inactive_anon:5504kB active_file:19136kB inactive_file:292188kB unevictable:129940kB isolated(anon):0kB isolated(file):128kB present:503916kB mlocked:129940kB dirty:26312kB writeback:8980kB mapped:40368kB shmem:28744kB slab_reclaimable:12104kB slab_unreclaimable:14136kB kernel_stack:1040kB pagetables:1796kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:64 all_unreclaimable? no Aug 4 11:13:31 rhel6-node2 kernel: lowmem_reserve[]: 0 0 0 0 Aug 4 11:13:31 rhel6-node2 kernel: DMA: 1*4kB 0*8kB 1*16kB 0*32kB 1*64kB 1*128kB 1*256kB 1*512kB 1*1024kB 0*2048kB 0*4096kB = 2004kB Aug 4 11:13:31 rhel6-node2 kernel: Normal: 238*4kB 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 952kB Aug 4 11:13:31 rhel6-node2 kernel: 88520 total pagecache pages Aug 4 11:13:31 rhel6-node2 kernel: 0 pages in swap cache Aug 4 11:13:31 rhel6-node2 kernel: Swap cache stats: add 0, delete 0, find 0/0 Aug 4 11:13:31 rhel6-node2 kernel: Free swap = 1048568kB Aug 4 11:13:31 rhel6-node2 kernel: Total swap = 1048568kB Aug 4 11:13:31 rhel6-node2 kernel: 131066 pages RAM Aug 4 11:13:31 rhel6-node2 kernel: 0 pages HighMem Aug 4 11:13:31 rhel6-node2 kernel: 3162 pages reserved Aug 4 11:13:31 rhel6-node2 kernel: 59989 pages shared Aug 4 11:13:31 rhel6-node2 kernel: 75227 pages non-shared Aug 4 11:13:32 rhel6-node2 qdiskd[1506]: qdiskd: write (system call) has hung for 2 seconds Aug 4 11:13:32 rhel6-node2 qdiskd[1506]: In 2 more seconds, we will be evicted Aug 4 11:13:37 rhel6-node2 qdiskd[1506]: qdisk cycle took more than 1 second to complete (10.170000) Aug 4 11:13:43 rhel6-node2 corosync[1454]: [CMAN ] lost contact with quorum device Aug 4 11:13:43 rhel6-node2 corosync[1454]: [QUORUM] Members[8]: 1 2 3 4 5 6 7 8 Aug 4 11:13:43 rhel6-node2 qdiskd[1506]: qdiskd: read (system call) has hung for 2 seconds Aug 4 11:13:43 rhel6-node2 qdiskd[1506]: In 2 more seconds, we will be evicted Aug 4 11:13:47 rhel6-node2 corosync[1454]: [QUORUM] Members[8]: 1 2 3 4 5 6 7 8 Aug 4 11:13:47 rhel6-node2 corosync[1454]: [QUORUM] Members[7]: 1 2 3 4 6 7 8 Aug 4 11:13:47 rhel6-node2 corosync[1454]: [QUORUM] Members[6]: 1 2 3 4 7 8 Aug 4 11:13:47 rhel6-node2 corosync[1454]: [QUORUM] Members[5]: 1 2 3 4 8 Aug 4 11:13:47 rhel6-node2 corosync[1454]: [TOTEM ] A processor joined or left the membership and a new membership was formed. Aug 4 11:13:47 rhel6-node2 kernel: dlm: closing connection to node 5 Aug 4 11:13:47 rhel6-node2 kernel: dlm: closing connection to node 6 Aug 4 11:13:47 rhel6-node2 kernel: dlm: closing connection to node 7 Aug 4 11:13:47 rhel6-node2 corosync[1454]: [TOTEM ] A processor joined or left the membership and a new membership was formed. Aug 4 11:13:47 rhel6-node2 corosync[1454]: [QUORUM] Members[6]: 1 2 3 4 5 8 Aug 4 11:13:47 rhel6-node2 corosync[1454]: [QUORUM] Members[6]: 1 2 3 4 5 8 Aug 4 11:13:47 rhel6-node2 corosync[1454]: [QUORUM] Members[7]: 1 2 3 4 5 6 8 Aug 4 11:13:47 rhel6-node2 corosync[1454]: [QUORUM] Members[7]: 1 2 3 4 5 6 8 Aug 4 11:13:47 rhel6-node2 corosync[1454]: [QUORUM] Members[8]: 1 2 3 4 5 6 7 8 Aug 4 11:13:47 rhel6-node2 corosync[1454]: [QUORUM] Members[8]: 1 2 3 4 5 6 7 8 Aug 4 11:13:51 rhel6-node2 qdiskd[1506]: qdiskd: write (system call) has hung for 2 seconds Aug 4 11:13:51 rhel6-node2 qdiskd[1506]: In 2 more seconds, we will be evicted Aug 4 11:13:52 rhel6-node2 qdiskd[1506]: qdisk cycle took more than 1 second to complete (15.390000) Aug 4 11:13:59 rhel6-node2 qdiskd[1506]: qdiskd: write (system call) has hung for 2 seconds Aug 4 11:13:59 rhel6-node2 qdiskd[1506]: In 2 more seconds, we will be evicted Aug 4 11:14:00 rhel6-node2 qdiskd[1506]: qdisk cycle took more than 1 second to complete (7.810000) Aug 4 11:14:01 rhel6-node2 corosync[1454]: [CPG ] downlist received left_list: 3 Aug 4 11:14:01 rhel6-node2 corosync[1454]: [CPG ] downlist received left_list: 3 Aug 4 11:14:01 rhel6-node2 corosync[1454]: [CPG ] downlist received left_list: 5 Aug 4 11:14:01 rhel6-node2 corosync[1454]: [CPG ] downlist received left_list: 5 Aug 4 11:14:01 rhel6-node2 corosync[1454]: [CPG ] downlist received left_list: 5 Aug 4 11:14:01 rhel6-node2 corosync[1454]: [CPG ] downlist received left_list: 3 Aug 4 11:14:01 rhel6-node2 corosync[1454]: [CPG ] downlist received left_list: 3 Aug 4 11:14:01 rhel6-node2 corosync[1454]: [CPG ] downlist received left_list: 3 Aug 4 11:14:01 rhel6-node2 corosync[1454]: [CPG ] chosen downlist from node r(0) ip(192.168.2.65) Aug 4 11:14:01 rhel6-node2 fenced[1676]: fencing deferred to rhel6-node1 Aug 4 11:14:01 rhel6-node2 kernel: GFS2: fsid=fabbione:gfs2.1: jid=6: Trying to acquire journal lock... Aug 4 11:14:01 rhel6-node2 corosync[1454]: [MAIN ] Completed service synchronization, ready to provide service. Aug 4 11:14:06 rhel6-node2 qdiskd[1506]: qdisk cycle took more than 1 second to complete (3.500000) Aug 4 11:14:08 rhel6-node2 qdiskd[1506]: qdisk cycle took more than 1 second to complete (1.660000) Aug 4 11:14:17 rhel6-node2 qdiskd[1506]: Node 7 evicted Aug 4 11:14:20 rhel6-node2 qdiskd[1506]: Node 6 evicted Aug 4 11:14:22 rhel6-node2 corosync[1454]: [TOTEM ] A processor failed, forming new configuration. Aug 4 11:14:24 rhel6-node2 kernel: INFO: task gfs2_quotad:2303 blocked for more than 120 seconds. Aug 4 11:14:24 rhel6-node2 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Aug 4 11:14:24 rhel6-node2 kernel: gfs2_quotad D 00000000 0 2303 2 0x00000080 Aug 4 11:14:24 rhel6-node2 kernel: df877a90 00000046 dda841c4 00000000 de7509c4 dfb1c800 de13d000 00009d37 Aug 4 11:14:24 rhel6-node2 kernel: 00000000 ddaa7040 000000a5 c0ac7080 2f374c37 000000a5 c0ac7080 df877d38 Aug 4 11:14:24 rhel6-node2 kernel: c0ac7080 c0ac2b34 c0ac7080 df877d38 df2e6000 00000001 e128c1de 00000000 Aug 4 11:14:24 rhel6-node2 kernel: Call Trace: Aug 4 11:14:24 rhel6-node2 kernel: [] ? do_xmote+0x12e/0x200 [gfs2] Aug 4 11:14:24 rhel6-node2 kernel: [] ? gfs2_glock_holder_wait+0x5/0x10 [gfs2] Aug 4 11:14:24 rhel6-node2 kernel: [] ? __wait_on_bit+0x45/0x70 Aug 4 11:14:24 rhel6-node2 kernel: [] ? gfs2_glock_holder_wait+0x0/0x10 [gfs2] Aug 4 11:14:24 rhel6-node2 kernel: [] ? gfs2_glock_holder_wait+0x0/0x10 [gfs2] Aug 4 11:14:24 rhel6-node2 kernel: [] ? out_of_line_wait_on_bit+0x68/0x80 Aug 4 11:14:24 rhel6-node2 kernel: [] ? wake_bit_function+0x0/0x60 Aug 4 11:14:24 rhel6-node2 kernel: [] ? gfs2_glock_wait+0x2a/0x40 [gfs2] Aug 4 11:14:24 rhel6-node2 kernel: [] ? gfs2_statfs_sync+0x4c/0x1b0 [gfs2] Aug 4 11:14:24 rhel6-node2 kernel: [] ? gfs2_statfs_sync+0x45/0x1b0 [gfs2] Aug 4 11:14:24 rhel6-node2 kernel: [] ? process_timeout+0x0/0x10 Aug 4 11:14:24 rhel6-node2 kernel: [] ? quotad_check_timeo+0x3e/0xa0 [gfs2] Aug 4 11:14:24 rhel6-node2 kernel: [] ? finish_wait+0x31/0x80 Aug 4 11:14:24 rhel6-node2 kernel: [] ? gfs2_quotad+0x1fc/0x240 [gfs2] Aug 4 11:14:24 rhel6-node2 kernel: [] ? autoremove_wake_function+0x0/0x40 Aug 4 11:14:24 rhel6-node2 kernel: [] ? gfs2_quotad+0x0/0x240 [gfs2] Aug 4 11:14:24 rhel6-node2 kernel: [] ? kthread+0x74/0x80 Aug 4 11:14:24 rhel6-node2 kernel: [] ? kthread+0x0/0x80 Aug 4 11:14:24 rhel6-node2 kernel: [] ? kernel_thread_helper+0x7/0x10 Aug 4 11:14:25 rhel6-node2 qdiskd[1506]: Node 5 evicted Aug 4 11:14:32 rhel6-node2 kernel: dlm: closing connection to node 5 Aug 4 11:14:32 rhel6-node2 kernel: dlm: closing connection to node 6 Aug 4 11:14:32 rhel6-node2 kernel: dlm: closing connection to node 7 Aug 4 11:14:32 rhel6-node2 corosync[1454]: [QUORUM] Members[7]: 1 2 3 4 6 7 8 Aug 4 11:14:32 rhel6-node2 corosync[1454]: [QUORUM] Members[6]: 1 2 3 4 7 8 Aug 4 11:14:32 rhel6-node2 corosync[1454]: [QUORUM] Members[5]: 1 2 3 4 8 Aug 4 11:14:32 rhel6-node2 corosync[1454]: [TOTEM ] A processor joined or left the membership and a new membership was formed. Aug 4 11:14:32 rhel6-node2 corosync[1454]: [CPG ] downlist received left_list: 3 Aug 4 11:14:32 rhel6-node2 corosync[1454]: [CPG ] downlist received left_list: 3 Aug 4 11:14:32 rhel6-node2 corosync[1454]: [CPG ] downlist received left_list: 3 Aug 4 11:14:32 rhel6-node2 corosync[1454]: [CPG ] downlist received left_list: 3 Aug 4 11:14:32 rhel6-node2 corosync[1454]: [CPG ] downlist received left_list: 3 Aug 4 11:14:32 rhel6-node2 corosync[1454]: [CPG ] chosen downlist from node r(0) ip(192.168.2.65) Aug 4 11:14:32 rhel6-node2 corosync[1454]: [MAIN ] Completed service synchronization, ready to provide service. Aug 4 11:14:33 rhel6-node2 kernel: GFS2: fsid=fabbione:gfs2.1: jid=6: Busy Aug 4 11:14:33 rhel6-node2 kernel: GFS2: fsid=fabbione:gfs2.1: jid=5: Trying to acquire journal lock... Aug 4 11:14:33 rhel6-node2 kernel: GFS2: fsid=fabbione:gfs2.1: jid=5: Busy Aug 4 11:14:33 rhel6-node2 kernel: GFS2: fsid=fabbione:gfs2.1: jid=4: Trying to acquire journal lock... Aug 4 11:14:34 rhel6-node2 kernel: GFS2: fsid=fabbione:gfs2.1: jid=4: Busy Aug 4 11:14:36 rhel6-node2 qdiskd[1506]: qdisk cycle took more than 1 second to complete (1.590000) Aug 4 11:14:46 rhel6-node2 qdiskd[1506]: qdiskd: write (system call) has hung for 2 seconds Aug 4 11:14:46 rhel6-node2 qdiskd[1506]: In 2 more seconds, we will be evicted Aug 4 11:14:47 rhel6-node2 qdiskd[1506]: qdisk cycle took more than 1 second to complete (7.200000) Aug 4 11:14:53 rhel6-node2 qdiskd[1506]: qdiskd: read (system call) has hung for 2 seconds Aug 4 11:14:53 rhel6-node2 qdiskd[1506]: In 2 more seconds, we will be evicted Aug 4 11:14:57 rhel6-node2 qdiskd[1506]: qdiskd: write (system call) has hung for 2 seconds Aug 4 11:14:57 rhel6-node2 qdiskd[1506]: In 2 more seconds, we will be evicted Aug 4 11:14:58 rhel6-node2 qdiskd[1506]: qdisk cycle took more than 1 second to complete (9.110000) Aug 4 11:15:02 rhel6-node2 qdiskd[1506]: qdisk cycle took more than 1 second to complete (1.730000) Aug 4 11:15:06 rhel6-node2 qdiskd[1506]: qdisk cycle took more than 1 second to complete (2.870000) Aug 4 11:15:10 rhel6-node2 qdiskd[1506]: qdisk cycle took more than 1 second to complete (2.320000) Aug 4 11:15:12 rhel6-node2 qdiskd[1506]: qdisk cycle took more than 1 second to complete (1.140000) Aug 4 11:15:17 rhel6-node2 qdiskd[1506]: qdisk cycle took more than 1 second to complete (2.210000) Aug 4 11:15:27 rhel6-node2 qdiskd[1506]: qdisk cycle took more than 1 second to complete (3.670000) Aug 4 11:15:32 rhel6-node2 qdiskd[1506]: qdisk cycle took more than 1 second to complete (1.920000) Aug 4 11:15:36 rhel6-node2 qdiskd[1506]: qdisk cycle took more than 1 second to complete (2.410000) Aug 4 11:15:38 rhel6-node2 qdiskd[1506]: qdisk cycle took more than 1 second to complete (1.040000) Aug 4 11:15:45 rhel6-node2 qdiskd[1506]: qdisk cycle took more than 1 second to complete (4.910000) Aug 4 11:15:48 rhel6-node2 qdiskd[1506]: qdisk cycle took more than 1 second to complete (1.820000) Aug 4 11:15:52 rhel6-node2 qdiskd[1506]: qdisk cycle took more than 1 second to complete (3.880000) Aug 4 11:15:59 rhel6-node2 qdiskd[1506]: qdisk cycle took more than 1 second to complete (3.460000) Aug 4 11:16:01 rhel6-node2 qdiskd[1506]: qdisk cycle took more than 1 second to complete (1.250000) Aug 4 11:16:08 rhel6-node2 qdiskd[1506]: qdisk cycle took more than 1 second to complete (3.860000) Aug 4 11:16:11 rhel6-node2 qdiskd[1506]: qdiskd: write (system call) has hung for 2 seconds Aug 4 11:16:11 rhel6-node2 qdiskd[1506]: In 2 more seconds, we will be evicted Aug 4 11:16:15 rhel6-node2 qdiskd[1506]: qdisk cycle took more than 1 second to complete (6.300000) Aug 4 11:16:20 rhel6-node2 qdiskd[1506]: qdiskd: write (system call) has hung for 2 seconds Aug 4 11:16:20 rhel6-node2 qdiskd[1506]: In 2 more seconds, we will be evicted Aug 4 11:16:21 rhel6-node2 qdiskd[1506]: qdisk cycle took more than 1 second to complete (3.390000) Aug 4 11:16:25 rhel6-node2 qdiskd[1506]: qdisk cycle took more than 1 second to complete (1.670000) Aug 4 11:16:30 rhel6-node2 qdiskd[1506]: qdisk cycle took more than 1 second to complete (1.940000) Aug 4 11:16:34 rhel6-node2 qdiskd[1506]: qdisk cycle took more than 1 second to complete (1.490000) Aug 4 11:16:39 rhel6-node2 qdiskd[1506]: qdiskd: write (system call) has hung for 2 seconds Aug 4 11:16:39 rhel6-node2 qdiskd[1506]: In 2 more seconds, we will be evicted Aug 4 11:16:39 rhel6-node2 qdiskd[1506]: qdisk cycle took more than 1 second to complete (4.290000) Aug 4 11:16:42 rhel6-node2 qdiskd[1506]: qdiskd: write (system call) has hung for 2 seconds Aug 4 11:16:42 rhel6-node2 qdiskd[1506]: In 2 more seconds, we will be evicted Aug 4 11:16:42 rhel6-node2 qdiskd[1506]: qdisk cycle took more than 1 second to complete (2.230000) Aug 4 11:16:46 rhel6-node2 qdiskd[1506]: qdisk cycle took more than 1 second to complete (1.500000) Aug 4 11:16:48 rhel6-node2 qdiskd[1506]: qdisk cycle took more than 1 second to complete (1.380000) Aug 4 11:16:53 rhel6-node2 qdiskd[1506]: qdisk cycle took more than 1 second to complete (1.000000) Aug 4 11:16:58 rhel6-node2 qdiskd[1506]: qdisk cycle took more than 1 second to complete (3.270000) Aug 4 11:17:00 rhel6-node2 qdiskd[1506]: qdisk cycle took more than 1 second to complete (1.390000) Aug 4 11:17:05 rhel6-node2 qdiskd[1506]: qdisk cycle took more than 1 second to complete (3.340000) Aug 4 11:17:15 rhel6-node2 qdiskd[1506]: qdiskd: write (system call) has hung for 2 seconds Aug 4 11:17:15 rhel6-node2 qdiskd[1506]: In 2 more seconds, we will be evicted Aug 4 11:17:19 rhel6-node2 qdiskd[1506]: qdisk cycle took more than 1 second to complete (8.340000) Aug 4 11:17:22 rhel6-node2 qdiskd[1506]: qdisk cycle took more than 1 second to complete (1.870000) Aug 4 11:17:25 rhel6-node2 qdiskd[1506]: qdisk cycle took more than 1 second to complete (1.750000) Aug 4 11:17:27 rhel6-node2 corosync[1454]: [TOTEM ] A processor failed, forming new configuration. Aug 4 11:17:28 rhel6-node2 qdiskd[1506]: qdisk cycle took more than 1 second to complete (1.580000) Aug 4 11:17:39 rhel6-node2 qdiskd[1506]: qdiskd: read (system call) has hung for 2 seconds Aug 4 11:17:39 rhel6-node2 qdiskd[1506]: In 2 more seconds, we will be evicted Aug 4 11:17:40 rhel6-node2 qdiskd[1506]: qdisk cycle took more than 1 second to complete (8.730000) Aug 4 11:17:45 rhel6-node2 corosync[1454]: [TOTEM ] A processor joined or left the membership and a new membership was formed. Aug 4 11:17:45 rhel6-node2 corosync[1454]: [CPG ] downlist received left_list: 0 Aug 4 11:17:45 rhel6-node2 corosync[1454]: [CPG ] downlist received left_list: 0 Aug 4 11:17:45 rhel6-node2 corosync[1454]: [CPG ] downlist received left_list: 0 Aug 4 11:17:45 rhel6-node2 corosync[1454]: [CPG ] downlist received left_list: 0 Aug 4 11:17:45 rhel6-node2 corosync[1454]: [CPG ] downlist received left_list: 0 Aug 4 11:17:45 rhel6-node2 corosync[1454]: [CPG ] chosen downlist from node r(0) ip(192.168.2.65) Aug 4 11:17:45 rhel6-node2 corosync[1454]: [MAIN ] Completed service synchronization, ready to provide service. Aug 4 11:17:47 rhel6-node2 qdiskd[1506]: qdiskd: write (system call) has hung for 2 seconds Aug 4 11:17:47 rhel6-node2 qdiskd[1506]: In 2 more seconds, we will be evicted Aug 4 11:17:48 rhel6-node2 qdiskd[1506]: qdisk cycle took more than 1 second to complete (7.210000) Aug 4 11:17:56 rhel6-node2 qdiskd[1506]: qdisk cycle took more than 1 second to complete (1.870000) Aug 4 11:18:03 rhel6-node2 qdiskd[1506]: qdisk cycle took more than 1 second to complete (1.140000) Aug 4 11:18:05 rhel6-node2 qdiskd[1506]: qdisk cycle took more than 1 second to complete (1.290000) Aug 4 11:18:14 rhel6-node2 qdiskd[1506]: qdiskd: write (system call) has hung for 2 seconds Aug 4 11:18:14 rhel6-node2 qdiskd[1506]: In 2 more seconds, we will be evicted Aug 4 11:18:16 rhel6-node2 qdiskd[1506]: qdisk cycle took more than 1 second to complete (4.840000) Aug 4 11:18:18 rhel6-node2 qdiskd[1506]: qdiskd: write (system call) has hung for 2 seconds Aug 4 11:18:18 rhel6-node2 qdiskd[1506]: In 2 more seconds, we will be evicted Aug 4 11:18:19 rhel6-node2 qdiskd[1506]: qdisk cycle took more than 1 second to complete (2.550000) Aug 4 11:18:28 rhel6-node2 qdiskd[1506]: qdiskd: write (system call) has hung for 2 seconds Aug 4 11:18:28 rhel6-node2 qdiskd[1506]: In 2 more seconds, we will be evicted Aug 4 11:18:31 rhel6-node2 qdiskd[1506]: qdisk cycle took more than 1 second to complete (11.190000) Aug 4 11:18:36 rhel6-node2 corosync[1454]: [TOTEM ] A processor failed, forming new configuration. Aug 4 11:18:36 rhel6-node2 qdiskd[1506]: qdiskd: write (system call) has hung for 2 seconds Aug 4 11:18:36 rhel6-node2 qdiskd[1506]: In 2 more seconds, we will be evicted Aug 4 11:18:36 rhel6-node2 qdiskd[1506]: qdisk cycle took more than 1 second to complete (3.890000) Aug 4 11:18:41 rhel6-node2 qdiskd[1506]: qdisk cycle took more than 1 second to complete (1.800000) Aug 4 11:18:48 rhel6-node2 qdiskd[1506]: qdiskd: write (system call) has hung for 2 seconds Aug 4 11:18:48 rhel6-node2 qdiskd[1506]: In 2 more seconds, we will be evicted Aug 4 11:18:48 rhel6-node2 qdiskd[1506]: qdisk cycle took more than 1 second to complete (3.170000) Aug 4 11:18:49 rhel6-node2 kernel: dlm: closing connection to node 1 Aug 4 11:18:49 rhel6-node2 kernel: dlm: closing connection to node 3 Aug 4 11:18:49 rhel6-node2 kernel: dlm: closing connection to node 4 Aug 4 11:18:49 rhel6-node2 kernel: dlm: closing connection to node 8 Aug 4 11:18:49 rhel6-node2 corosync[1454]: [QUORUM] Members[4]: 2 3 4 8 Aug 4 11:18:49 rhel6-node2 corosync[1454]: [QUORUM] Members[3]: 2 4 8 Aug 4 11:18:49 rhel6-node2 corosync[1454]: [QUORUM] Members[2]: 2 8 Aug 4 11:18:49 rhel6-node2 corosync[1454]: [QUORUM] Members[1]: 2 Aug 4 11:18:49 rhel6-node2 corosync[1454]: [TOTEM ] A processor joined or left the membership and a new membership was formed. Aug 4 11:18:50 rhel6-node2 corosync[1454]: [TOTEM ] A processor joined or left the membership and a new membership was formed. Aug 4 11:18:50 rhel6-node2 corosync[1454]: [QUORUM] Members[2]: 2 3 Aug 4 11:18:50 rhel6-node2 corosync[1454]: [QUORUM] Members[2]: 2 3 Aug 4 11:18:50 rhel6-node2 corosync[1454]: [QUORUM] Members[3]: 2 3 4 Aug 4 11:18:50 rhel6-node2 corosync[1454]: [QUORUM] Members[3]: 2 3 4 Aug 4 11:18:50 rhel6-node2 corosync[1454]: [TOTEM ] A processor joined or left the membership and a new membership was formed. Aug 4 11:18:50 rhel6-node2 corosync[1454]: [QUORUM] Members[4]: 2 3 4 8 Aug 4 11:18:50 rhel6-node2 corosync[1454]: [QUORUM] Members[4]: 2 3 4 8 Aug 4 11:18:50 rhel6-node2 corosync[1454]: [QUORUM] Members[5]: 1 2 3 4 8 Aug 4 11:18:50 rhel6-node2 corosync[1454]: [QUORUM] Members[5]: 1 2 3 4 8 Aug 4 11:18:50 rhel6-node2 kernel: GFS2: fsid=fabbione:gfs2.1: jid=7: Trying to acquire journal lock... Aug 4 11:18:50 rhel6-node2 fenced[1676]: fencing node rhel6-node8 Aug 4 11:18:50 rhel6-node2 corosync[1454]: [CPG ] downlist received left_list: 0 Aug 4 11:18:50 rhel6-node2 corosync[1454]: [CPG ] downlist received left_list: 0 Aug 4 11:18:50 rhel6-node2 corosync[1454]: [CPG ] downlist received left_list: 2 Aug 4 11:18:50 rhel6-node2 corosync[1454]: [CPG ] downlist received left_list: 2 Aug 4 11:18:50 rhel6-node2 corosync[1454]: [CPG ] downlist received left_list: 2 Aug 4 11:18:50 rhel6-node2 corosync[1454]: [CPG ] chosen downlist from node r(0) ip(192.168.2.66) Aug 4 11:18:50 rhel6-node2 corosync[1454]: [MAIN ] Completed service synchronization, ready to provide service. Aug 4 11:18:51 rhel6-node2 fenced[1676]: fence rhel6-node8 dev 0.0 agent fence_virsh result: error from agent Aug 4 11:18:51 rhel6-node2 fenced[1676]: fence rhel6-node8 failed Aug 4 11:18:54 rhel6-node2 fenced[1676]: fencing node rhel6-node8 Aug 4 11:18:58 rhel6-node2 fenced[1676]: fence rhel6-node8 success Aug 4 11:18:58 rhel6-node2 fenced[1676]: fencing node rhel6-node1 Aug 4 11:19:03 rhel6-node2 fenced[1676]: fence rhel6-node1 success Aug 4 11:19:06 rhel6-node2 qdiskd[1506]: Assuming master role Aug 4 11:19:06 rhel6-node2 corosync[1454]: [TOTEM ] A processor failed, forming new configuration. Aug 4 11:19:07 rhel6-node2 qdiskd[1506]: Writing eviction notice for node 1 Aug 4 11:19:07 rhel6-node2 qdiskd[1506]: Writing eviction notice for node 8 Aug 4 11:19:08 rhel6-node2 qdiskd[1506]: Node 1 evicted Aug 4 11:19:08 rhel6-node2 qdiskd[1506]: Node 8 evicted Aug 4 11:19:17 rhel6-node2 qdiskd[1506]: qdiskd: write (system call) has hung for 2 seconds Aug 4 11:19:17 rhel6-node2 qdiskd[1506]: In 2 more seconds, we will be evicted Aug 4 11:19:17 rhel6-node2 qdiskd[1506]: qdisk cycle took more than 1 second to complete (8.420000) Aug 4 11:19:19 rhel6-node2 kernel: dlm: closing connection to node 1 Aug 4 11:19:19 rhel6-node2 kernel: dlm: closing connection to node 8 Aug 4 11:19:19 rhel6-node2 corosync[1454]: [QUORUM] Members[3]: 2 3 4 Aug 4 11:19:19 rhel6-node2 corosync[1454]: [QUORUM] Members[3]: 2 3 4 Aug 4 11:19:19 rhel6-node2 corosync[1454]: [TOTEM ] A processor joined or left the membership and a new membership was formed. Aug 4 11:19:21 rhel6-node2 corosync[1454]: [CPG ] downlist received left_list: 2 Aug 4 11:19:21 rhel6-node2 qdiskd[1506]: qdiskd: write (system call) has hung for 2 seconds Aug 4 11:19:21 rhel6-node2 qdiskd[1506]: In 2 more seconds, we will be evicted Aug 4 11:19:21 rhel6-node2 corosync[1454]: [CPG ] downlist received left_list: 2 Aug 4 11:19:21 rhel6-node2 dlm_controld[1699]: cpg_mcast_joined retry 100 start Aug 4 11:19:22 rhel6-node2 corosync[1454]: [CPG ] downlist received left_list: 2 Aug 4 11:19:22 rhel6-node2 corosync[1454]: [CPG ] chosen downlist from node r(0) ip(192.168.2.66) Aug 4 11:19:22 rhel6-node2 qdiskd[1506]: qdisk cycle took more than 1 second to complete (2.700000) Aug 4 11:19:22 rhel6-node2 corosync[1454]: [MAIN ] Completed service synchronization, ready to provide service. Aug 4 11:22:28 rhel6-node2 kernel: INFO: task kslowd001:1182 blocked for more than 120 seconds. Aug 4 11:22:28 rhel6-node2 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Aug 4 11:22:28 rhel6-node2 kernel: kslowd001 D 00000001 0 1182 2 0x00000080 Aug 4 11:22:28 rhel6-node2 kernel: dc626560 00000046 ffffffff 00000001 00000000 dc62605c 00000000 000461cb Aug 4 11:22:28 rhel6-node2 kernel: 00000000 df2c3ac0 00000102 c0ac7080 02c2fb70 00000102 c0ac7080 dc626808 Aug 4 11:22:28 rhel6-node2 kernel: c0ac7080 c0ac2b34 c0ac7080 dc626808 dca60000 00000001 00000000 c1609080 Aug 4 11:22:28 rhel6-node2 kernel: Call Trace: Aug 4 11:22:28 rhel6-node2 kernel: [] ? rwsem_down_failed_common+0x75/0x1a0 Aug 4 11:22:28 rhel6-node2 kernel: [] ? rwsem_down_read_failed+0x1a/0x24 Aug 4 11:22:28 rhel6-node2 kernel: [] ? call_rwsem_down_read_failed+0x7/0xc Aug 4 11:22:28 rhel6-node2 kernel: [] ? down_read+0x14/0x20 Aug 4 11:22:28 rhel6-node2 kernel: [] ? dlm_lock+0x43/0x180 [dlm] Aug 4 11:22:28 rhel6-node2 kernel: [] ? vsnprintf+0x2ea/0x3f0 Aug 4 11:22:28 rhel6-node2 kernel: [] ? gdlm_lock+0xda/0x120 [gfs2] Aug 4 11:22:28 rhel6-node2 kernel: [] ? gdlm_ast+0x0/0xf0 [gfs2] Aug 4 11:22:28 rhel6-node2 kernel: [] ? gdlm_bast+0x0/0x50 [gfs2] Aug 4 11:22:28 rhel6-node2 kernel: [] ? gdlm_lock+0x0/0x120 [gfs2] Aug 4 11:22:28 rhel6-node2 kernel: [] ? do_xmote+0x12e/0x200 [gfs2] Aug 4 11:22:28 rhel6-node2 kernel: [] ? gfs2_glock_nq+0x143/0x350 [gfs2] Aug 4 11:22:28 rhel6-node2 kernel: [] ? gfs2_glock_nq_num+0x53/0x80 [gfs2] Aug 4 11:22:28 rhel6-node2 kernel: [] ? gfs2_recover_work+0x81/0x720 [gfs2] Aug 4 11:22:28 rhel6-node2 kernel: [] ? dequeue_entity+0x1a9/0x200 Aug 4 11:22:28 rhel6-node2 kernel: [] ? __switch_to+0xd7/0x1a0 Aug 4 11:22:28 rhel6-node2 kernel: [] ? finish_task_switch+0x33/0xa0 Aug 4 11:22:28 rhel6-node2 kernel: [] ? gfs2_glock_nq_num+0x4c/0x80 [gfs2] Aug 4 11:22:28 rhel6-node2 kernel: [] ? slow_work_execute+0x1df/0x2a0 Aug 4 11:22:28 rhel6-node2 kernel: [] ? prepare_to_wait_exclusive+0x5b/0x60 Aug 4 11:22:28 rhel6-node2 kernel: [] ? slow_work_thread+0x123/0x340 Aug 4 11:22:28 rhel6-node2 kernel: [] ? autoremove_wake_function+0x0/0x40 Aug 4 11:22:28 rhel6-node2 kernel: [] ? slow_work_thread+0x0/0x340 Aug 4 11:22:28 rhel6-node2 kernel: [] ? kthread+0x74/0x80 Aug 4 11:22:28 rhel6-node2 kernel: [] ? kthread+0x0/0x80 Aug 4 11:22:28 rhel6-node2 kernel: [] ? kernel_thread_helper+0x7/0x10 Aug 4 11:22:28 rhel6-node2 kernel: INFO: task flush-252:16:2083 blocked for more than 120 seconds. Aug 4 11:22:28 rhel6-node2 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Aug 4 11:22:28 rhel6-node2 kernel: flush-252:16 D c57413c0 0 2083 2 0x00000080 Aug 4 11:22:28 rhel6-node2 kernel: ddb7c030 00000046 17eca722 c57413c0 d36fd438 00000001 00000001 df304eec Aug 4 11:22:28 rhel6-node2 kernel: c0470aa3 ddaa7580 0000010a c0ac7080 6bf5f923 0000010a c0ac7080 ddb7c2d8 Aug 4 11:22:28 rhel6-node2 kernel: c0ac7080 c0ac2b34 c0ac7080 ddb7c2d8 00000050 d4c82780 000ce5d8 d4618bec Aug 4 11:22:28 rhel6-node2 kernel: Call Trace: Aug 4 11:22:28 rhel6-node2 kernel: [] ? wake_up_bit+0x13/0x30 Aug 4 11:22:28 rhel6-node2 kernel: [] ? gfs2_glock_holder_wait+0x5/0x10 [gfs2] Aug 4 11:22:28 rhel6-node2 kernel: [] ? __wait_on_bit+0x45/0x70 Aug 4 11:22:28 rhel6-node2 kernel: [] ? gfs2_glock_holder_wait+0x0/0x10 [gfs2] Aug 4 11:22:28 rhel6-node2 kernel: [] ? gfs2_glock_holder_wait+0x0/0x10 [gfs2] Aug 4 11:22:28 rhel6-node2 kernel: [] ? out_of_line_wait_on_bit+0x68/0x80 Aug 4 11:22:28 rhel6-node2 kernel: [] ? wake_bit_function+0x0/0x60 Aug 4 11:22:28 rhel6-node2 kernel: [] ? gfs2_glock_wait+0x2a/0x40 [gfs2] Aug 4 11:22:28 rhel6-node2 kernel: [] ? gfs2_write_inode+0x72/0x180 [gfs2] Aug 4 11:22:28 rhel6-node2 kernel: [] ? gfs2_write_inode+0x69/0x180 [gfs2] Aug 4 11:22:28 rhel6-node2 kernel: [] ? writeback_single_inode+0x1c8/0x270 Aug 4 11:22:28 rhel6-node2 kernel: [] ? writeback_sb_inodes+0xc1/0x180 Aug 4 11:22:28 rhel6-node2 kernel: [] ? writeback_inodes_wb+0x83/0x150 Aug 4 11:22:28 rhel6-node2 kernel: [] ? wb_writeback+0x1e3/0x230 Aug 4 11:22:28 rhel6-node2 kernel: [] ? schedule_timeout+0x135/0x260 Aug 4 11:22:28 rhel6-node2 kernel: [] ? wb_do_writeback+0x139/0x190 Aug 4 11:22:28 rhel6-node2 kernel: [] ? bdi_writeback_task+0x2a/0xa0 Aug 4 11:22:28 rhel6-node2 kernel: [] ? bdi_start_fn+0x67/0xc0 Aug 4 11:22:28 rhel6-node2 kernel: [] ? bdi_start_fn+0x0/0xc0 Aug 4 11:22:28 rhel6-node2 kernel: [] ? kthread+0x74/0x80 Aug 4 11:22:28 rhel6-node2 kernel: [] ? kthread+0x0/0x80 Aug 4 11:22:28 rhel6-node2 kernel: [] ? kernel_thread_helper+0x7/0x10 Aug 4 11:22:28 rhel6-node2 kernel: INFO: task gfs2_quotad:2303 blocked for more than 120 seconds. Aug 4 11:22:28 rhel6-node2 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Aug 4 11:22:28 rhel6-node2 kernel: gfs2_quotad D 00000000 0 2303 2 0x00000080 Aug 4 11:22:28 rhel6-node2 kernel: df877a90 00000046 c043d5d2 00000000 00000000 00000000 df8cc5b4 1cc53621 Aug 4 11:22:28 rhel6-node2 kernel: 00000000 dc7cf900 00000107 c0ac7080 3e907cf7 00000107 c0ac7080 df877d38 Aug 4 11:22:28 rhel6-node2 kernel: c0ac7080 c0ac2b34 c0ac7080 df877d38 dc438000 00000292 c046cb48 ffd9da60 Aug 4 11:22:28 rhel6-node2 kernel: Call Trace: Aug 4 11:22:28 rhel6-node2 kernel: [] ? __wake_up+0x42/0x60 Aug 4 11:22:28 rhel6-node2 kernel: [] ? queue_work_on+0x38/0x60 Aug 4 11:22:28 rhel6-node2 kernel: [] ? update_curr+0x167/0x2c0 Aug 4 11:22:28 rhel6-node2 kernel: [] ? rwsem_down_failed_common+0x75/0x1a0 Aug 4 11:22:28 rhel6-node2 kernel: [] ? rwsem_down_read_failed+0x1a/0x24 Aug 4 11:22:28 rhel6-node2 kernel: [] ? call_rwsem_down_read_failed+0x7/0xc Aug 4 11:22:28 rhel6-node2 kernel: [] ? down_read+0x14/0x20 Aug 4 11:22:28 rhel6-node2 kernel: [] ? dlm_lock+0x43/0x180 [dlm] Aug 4 11:22:28 rhel6-node2 kernel: [] ? update_curr+0x167/0x2c0 Aug 4 11:22:28 rhel6-node2 kernel: [] ? check_preempt_wakeup+0x1fb/0x380 Aug 4 11:22:28 rhel6-node2 kernel: [] ? gdlm_lock+0xda/0x120 [gfs2] Aug 4 11:22:28 rhel6-node2 kernel: [] ? gdlm_ast+0x0/0xf0 [gfs2] Aug 4 11:22:28 rhel6-node2 kernel: [] ? gdlm_bast+0x0/0x50 [gfs2] Aug 4 11:22:28 rhel6-node2 kernel: [] ? gdlm_lock+0x0/0x120 [gfs2] Aug 4 11:22:28 rhel6-node2 kernel: [] ? do_xmote+0x12e/0x200 [gfs2] Aug 4 11:22:28 rhel6-node2 kernel: [] ? gfs2_glock_nq+0x143/0x350 [gfs2] Aug 4 11:22:28 rhel6-node2 kernel: [] ? lock_timer_base+0x27/0x50 Aug 4 11:22:28 rhel6-node2 kernel: [] ? gfs2_statfs_sync+0x4c/0x1b0 [gfs2] Aug 4 11:22:28 rhel6-node2 kernel: [] ? gfs2_statfs_sync+0x45/0x1b0 [gfs2] Aug 4 11:22:28 rhel6-node2 kernel: [] ? process_timeout+0x0/0x10 Aug 4 11:22:28 rhel6-node2 kernel: [] ? quotad_check_timeo+0x3e/0xa0 [gfs2] Aug 4 11:22:28 rhel6-node2 kernel: [] ? finish_wait+0x31/0x80 Aug 4 11:22:28 rhel6-node2 kernel: [] ? gfs2_quotad+0x1fc/0x240 [gfs2] Aug 4 11:22:28 rhel6-node2 kernel: [] ? autoremove_wake_function+0x0/0x40 Aug 4 11:22:28 rhel6-node2 kernel: [] ? gfs2_quotad+0x0/0x240 [gfs2] Aug 4 11:22:28 rhel6-node2 kernel: [] ? kthread+0x74/0x80 Aug 4 11:22:28 rhel6-node2 kernel: [] ? kthread+0x0/0x80 Aug 4 11:22:28 rhel6-node2 kernel: [] ? kernel_thread_helper+0x7/0x10 Aug 4 11:22:28 rhel6-node2 kernel: INFO: task git:2329 blocked for more than 120 seconds. Aug 4 11:22:28 rhel6-node2 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Aug 4 11:22:28 rhel6-node2 kernel: git D c16090dc 0 2329 1409 0x00000080 Aug 4 11:22:28 rhel6-node2 kernel: de668560 00000086 00000101 c16090dc de13558c c16090c0 de13558c de668560 Aug 4 11:22:28 rhel6-node2 kernel: de135560 dc728ac0 00000101 c0ac7080 bb9c371d 00000101 c0ac7080 de668808 Aug 4 11:22:28 rhel6-node2 kernel: c0ac7080 c0ac2b34 c0ac7080 de668808 c080d29f 00000001 000c5412 00000000 Aug 4 11:22:28 rhel6-node2 kernel: Call Trace: Aug 4 11:22:28 rhel6-node2 kernel: [] ? schedule+0x42f/0xb00 Aug 4 11:22:28 rhel6-node2 kernel: [] ? gfs2_glock_holder_wait+0x5/0x10 [gfs2] Aug 4 11:22:28 rhel6-node2 kernel: [] ? __wait_on_bit+0x45/0x70 Aug 4 11:22:28 rhel6-node2 kernel: [] ? gfs2_glock_holder_wait+0x0/0x10 [gfs2] Aug 4 11:22:28 rhel6-node2 kernel: [] ? gfs2_glock_holder_wait+0x0/0x10 [gfs2] Aug 4 11:22:28 rhel6-node2 kernel: [] ? out_of_line_wait_on_bit+0x68/0x80 Aug 4 11:22:28 rhel6-node2 kernel: [] ? wake_bit_function+0x0/0x60 Aug 4 11:22:28 rhel6-node2 kernel: [] ? gfs2_glock_wait+0x2a/0x40 [gfs2] Aug 4 11:22:28 rhel6-node2 kernel: [] ? gfs2_glock_nq_num+0x53/0x80 [gfs2] Aug 4 11:22:28 rhel6-node2 kernel: [] ? gfs2_createi+0x1f1/0x9f0 [gfs2] Aug 4 11:22:28 rhel6-node2 kernel: [] ? run_queue+0x8e/0x140 [gfs2] Aug 4 11:22:28 rhel6-node2 kernel: [] ? gfs2_glock_nq+0x143/0x350 [gfs2] Aug 4 11:22:28 rhel6-node2 kernel: [] ? gfs2_create+0x60/0x140 [gfs2] Aug 4 11:22:28 rhel6-node2 kernel: [] ? gfs2_createi+0x8e/0x9f0 [gfs2] Aug 4 11:22:28 rhel6-node2 kernel: [] ? gfs2_glock_nq_num+0x4c/0x80 [gfs2] Aug 4 11:22:28 rhel6-node2 kernel: [] ? vfs_create+0xad/0xd0 Aug 4 11:22:28 rhel6-node2 kernel: [] ? do_filp_open+0x928/0xb10 Aug 4 11:22:28 rhel6-node2 kernel: [] ? do_page_fault+0x0/0x480 Aug 4 11:22:28 rhel6-node2 kernel: [] ? do_sys_open+0x58/0x130 Aug 4 11:22:28 rhel6-node2 kernel: [] ? sys_open+0x2c/0x40 Aug 4 11:22:28 rhel6-node2 kernel: [] ? syscall_call+0x7/0xb Aug 4 11:25:27 rhel6-node2 kernel: imklog 4.6.2, log source = /proc/kmsg started. Aug 4 11:25:27 rhel6-node2 rsyslogd: [origin software="rsyslogd" swVersion="4.6.2" x-pid="1082" x-info="http://www.rsyslog.com"] (re)start Aug 4 11:25:27 rhel6-node2 kernel: Initializing cgroup subsys cpuset Aug 4 11:25:27 rhel6-node2 kernel: Initializing cgroup subsys cpu Aug 4 11:25:27 rhel6-node2 kernel: Linux version 2.6.32-54.el6.i686 (mockbuild@x86-002.build.bos.redhat.com) (gcc version 4.4.4 20100726 (Red Hat 4.4.4-13) (GCC) ) #1 SMP Tue Jul 27 23:35:55 EDT 2010 Aug 4 11:25:27 rhel6-node2 kernel: KERNEL supported cpus: Aug 4 11:25:27 rhel6-node2 kernel: Intel GenuineIntel Aug 4 11:25:27 rhel6-node2 kernel: AMD AuthenticAMD Aug 4 11:25:27 rhel6-node2 kernel: NSC Geode by NSC Aug 4 11:25:27 rhel6-node2 kernel: Cyrix CyrixInstead Aug 4 11:25:27 rhel6-node2 kernel: Centaur CentaurHauls Aug 4 11:25:27 rhel6-node2 kernel: Transmeta GenuineTMx86 Aug 4 11:25:27 rhel6-node2 kernel: Transmeta TransmetaCPU Aug 4 11:25:27 rhel6-node2 kernel: UMC UMC UMC UMC Aug 4 11:25:27 rhel6-node2 kernel: BIOS-provided physical RAM map: Aug 4 11:25:27 rhel6-node2 kernel: BIOS-e820: 0000000000000000 - 000000000009a800 (usable) Aug 4 11:25:27 rhel6-node2 kernel: BIOS-e820: 000000000009a800 - 00000000000a0000 (reserved) Aug 4 11:25:27 rhel6-node2 kernel: BIOS-e820: 00000000000f0000 - 0000000000100000 (reserved) Aug 4 11:25:27 rhel6-node2 kernel: BIOS-e820: 0000000000100000 - 000000001fffb000 (usable) Aug 4 11:25:27 rhel6-node2 kernel: BIOS-e820: 000000001fffb000 - 0000000020000000 (reserved) Aug 4 11:25:27 rhel6-node2 kernel: BIOS-e820: 00000000fffbc000 - 0000000100000000 (reserved) Aug 4 11:25:27 rhel6-node2 kernel: DMI 2.4 present. Aug 4 11:25:27 rhel6-node2 kernel: last_pfn = 0x1fffb max_arch_pfn = 0x400000 Aug 4 11:25:27 rhel6-node2 kernel: PAT not supported by CPU. Aug 4 11:25:27 rhel6-node2 kernel: init_memory_mapping: 0000000000000000-000000001fffb000 Aug 4 11:25:27 rhel6-node2 kernel: NX (Execute Disable) protection: active Aug 4 11:25:27 rhel6-node2 kernel: RAMDISK: 1f419000 - 1ffeaed9 Aug 4 11:25:27 rhel6-node2 kernel: ACPI: RSDP 000f8400 00014 (v00 BOCHS ) Aug 4 11:25:27 rhel6-node2 kernel: ACPI: RSDT 1fffde80 00030 (v01 BOCHS BXPCRSDT 00000001 BXPC 00000001) Aug 4 11:25:27 rhel6-node2 kernel: ACPI: FACP 1ffffe70 00074 (v01 BOCHS BXPCFACP 00000001 BXPC 00000001) Aug 4 11:25:27 rhel6-node2 kernel: ACPI: DSDT 1fffdfd0 01E22 (v01 BXPC BXDSDT 00000001 INTL 20090123) Aug 4 11:25:27 rhel6-node2 kernel: ACPI: FACS 1ffffe00 00040 Aug 4 11:25:27 rhel6-node2 kernel: ACPI: SSDT 1fffdf90 00037 (v01 BOCHS BXPCSSDT 00000001 BXPC 00000001) Aug 4 11:25:27 rhel6-node2 kernel: ACPI: APIC 1fffdeb0 00072 (v01 BOCHS BXPCAPIC 00000001 BXPC 00000001) Aug 4 11:25:27 rhel6-node2 kernel: 0MB HIGHMEM available. Aug 4 11:25:27 rhel6-node2 kernel: 511MB LOWMEM available. Aug 4 11:25:27 rhel6-node2 kernel: mapped low ram: 0 - 1fffb000 Aug 4 11:25:27 rhel6-node2 kernel: low ram: 0 - 1fffb000 Aug 4 11:25:27 rhel6-node2 kernel: node 0 low ram: 00000000 - 1fffb000 Aug 4 11:25:27 rhel6-node2 kernel: node 0 bootmap 00002000 - 00006000 Aug 4 11:25:27 rhel6-node2 kernel: (9 early reservations) ==> bootmem [0000000000 - 001fffb000] Aug 4 11:25:27 rhel6-node2 kernel: #0 [0000000000 - 0000001000] BIOS data page ==> [0000000000 - 0000001000] Aug 4 11:25:27 rhel6-node2 kernel: #1 [0000001000 - 0000002000] EX TRAMPOLINE ==> [0000001000 - 0000002000] Aug 4 11:25:27 rhel6-node2 kernel: #2 [0000006000 - 0000007000] TRAMPOLINE ==> [0000006000 - 0000007000] Aug 4 11:25:27 rhel6-node2 kernel: #3 [0000400000 - 0000bd8864] TEXT DATA BSS ==> [0000400000 - 0000bd8864] Aug 4 11:25:27 rhel6-node2 kernel: #4 [001f419000 - 001ffeaed9] RAMDISK ==> [001f419000 - 001ffeaed9] Aug 4 11:25:27 rhel6-node2 kernel: #5 [000009a800 - 0000100000] BIOS reserved ==> [000009a800 - 0000100000] Aug 4 11:25:27 rhel6-node2 kernel: #6 [0000bd9000 - 0000be1071] BRK ==> [0000bd9000 - 0000be1071] Aug 4 11:25:27 rhel6-node2 kernel: #7 [0000007000 - 000000b000] PGTABLE ==> [0000007000 - 000000b000] Aug 4 11:25:27 rhel6-node2 kernel: #8 [0000002000 - 0000006000] BOOTMAP ==> [0000002000 - 0000006000] Aug 4 11:25:27 rhel6-node2 kernel: found SMP MP-table at [c00f8450] f8450 Aug 4 11:25:27 rhel6-node2 kernel: kvm-clock: Using msrs 12 and 11 Aug 4 11:25:27 rhel6-node2 kernel: kvm-clock: cpu 0, msr 0:ac7041, boot clock Aug 4 11:25:27 rhel6-node2 kernel: Zone PFN ranges: Aug 4 11:25:27 rhel6-node2 kernel: DMA 0x00000001 -> 0x00001000 Aug 4 11:25:27 rhel6-node2 kernel: Normal 0x00001000 -> 0x0001fffb Aug 4 11:25:27 rhel6-node2 kernel: HighMem 0x0001fffb -> 0x0001fffb Aug 4 11:25:27 rhel6-node2 kernel: Movable zone start PFN for each node Aug 4 11:25:27 rhel6-node2 kernel: early_node_map[2] active PFN ranges Aug 4 11:25:27 rhel6-node2 kernel: 0: 0x00000001 -> 0x0000009a Aug 4 11:25:27 rhel6-node2 kernel: 0: 0x00000100 -> 0x0001fffb Aug 4 11:25:27 rhel6-node2 kernel: Using APIC driver default Aug 4 11:25:27 rhel6-node2 kernel: ACPI: PM-Timer IO Port: 0xb008 Aug 4 11:25:27 rhel6-node2 kernel: ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] enabled) Aug 4 11:25:27 rhel6-node2 kernel: ACPI: IOAPIC (id[0x01] address[0xfec00000] gsi_base[0]) Aug 4 11:25:27 rhel6-node2 kernel: IOAPIC[0]: apic_id 1, version 17, address 0xfec00000, GSI 0-23 Aug 4 11:25:27 rhel6-node2 kernel: ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) Aug 4 11:25:27 rhel6-node2 kernel: ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) Aug 4 11:25:27 rhel6-node2 kernel: ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) Aug 4 11:25:27 rhel6-node2 kernel: ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) Aug 4 11:25:27 rhel6-node2 kernel: ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) Aug 4 11:25:27 rhel6-node2 kernel: Using ACPI (MADT) for SMP configuration information Aug 4 11:25:27 rhel6-node2 kernel: SMP: Allowing 1 CPUs, 0 hotplug CPUs Aug 4 11:25:27 rhel6-node2 kernel: PM: Registered nosave memory: 000000000009a000 - 000000000009b000 Aug 4 11:25:27 rhel6-node2 kernel: PM: Registered nosave memory: 000000000009b000 - 00000000000a0000 Aug 4 11:25:27 rhel6-node2 kernel: PM: Registered nosave memory: 00000000000a0000 - 00000000000f0000 Aug 4 11:25:27 rhel6-node2 kernel: PM: Registered nosave memory: 00000000000f0000 - 0000000000100000 Aug 4 11:25:27 rhel6-node2 kernel: Allocating PCI resources starting at 20000000 (gap: 20000000:dffbc000) Aug 4 11:25:27 rhel6-node2 kernel: Booting paravirtualized kernel on KVM Aug 4 11:25:27 rhel6-node2 kernel: NR_CPUS:32 nr_cpumask_bits:32 nr_cpu_ids:1 nr_node_ids:1 Aug 4 11:25:27 rhel6-node2 kernel: PERCPU: Embedded 15 pages/cpu @c1600000 s38392 r0 d23048 u2097152 Aug 4 11:25:27 rhel6-node2 kernel: pcpu-alloc: s38392 r0 d23048 u2097152 alloc=1*2097152 Aug 4 11:25:27 rhel6-node2 kernel: pcpu-alloc: [0] 0 Aug 4 11:25:27 rhel6-node2 kernel: kvm-clock: cpu 0, msr 0:1609041, primary cpu clock Aug 4 11:25:27 rhel6-node2 kernel: Built 1 zonelists in Zone order, mobility grouping on. Total pages: 129940 Aug 4 11:25:27 rhel6-node2 kernel: Kernel command line: ro root=/dev/mapper/VolGroup-lv_root rd_LVM_LV=VolGroup/lv_root rd_LVM_LV=VolGroup/lv_swap rd_NO_LUKS rd_NO_MD rd_NO_DM LANG=en_US.UTF-8 SYSFONT=latarcyrheb-sun16 KEYBOARDTYPE=pc KEYTABLE=dk-latin1 rhgb quiet Aug 4 11:25:27 rhel6-node2 kernel: PID hash table entries: 2048 (order: 1, 8192 bytes) Aug 4 11:25:27 rhel6-node2 kernel: Dentry cache hash table entries: 65536 (order: 6, 262144 bytes) Aug 4 11:25:27 rhel6-node2 kernel: Inode-cache hash table entries: 32768 (order: 5, 131072 bytes) Aug 4 11:25:27 rhel6-node2 kernel: Enabling fast FPU save and restore... done. Aug 4 11:25:27 rhel6-node2 kernel: Enabling unmasked SIMD FPU exception support... done. Aug 4 11:25:27 rhel6-node2 kernel: Initializing CPU#0 Aug 4 11:25:27 rhel6-node2 kernel: Initializing HighMem for node 0 (00000000:00000000) Aug 4 11:25:27 rhel6-node2 kernel: Memory: 498740k/524268k available (4175k kernel code, 24884k reserved, 2240k data, 524k init, 0k highmem) Aug 4 11:25:27 rhel6-node2 kernel: virtual kernel memory layout: Aug 4 11:25:27 rhel6-node2 kernel: fixmap : 0xffad5000 - 0xfffff000 (5288 kB) Aug 4 11:25:27 rhel6-node2 kernel: pkmap : 0xff600000 - 0xff800000 (2048 kB) Aug 4 11:25:27 rhel6-node2 kernel: vmalloc : 0xe07fb000 - 0xff5fe000 ( 494 MB) Aug 4 11:25:27 rhel6-node2 kernel: lowmem : 0xc0000000 - 0xdfffb000 ( 511 MB) Aug 4 11:25:27 rhel6-node2 kernel: .init : 0xc0a45000 - 0xc0ac8000 ( 524 kB) Aug 4 11:25:27 rhel6-node2 kernel: .data : 0xc0813f16 - 0xc0a44248 (2240 kB) Aug 4 11:25:27 rhel6-node2 kernel: .text : 0xc0400000 - 0xc0813f16 (4175 kB) Aug 4 11:25:27 rhel6-node2 kernel: Checking if this processor honours the WP bit even in supervisor mode...Ok. Aug 4 11:25:27 rhel6-node2 kernel: Hierarchical RCU implementation. Aug 4 11:25:27 rhel6-node2 kernel: NR_IRQS:2304 nr_irqs:256 Aug 4 11:25:27 rhel6-node2 kernel: Console: colour VGA+ 80x25 Aug 4 11:25:27 rhel6-node2 kernel: console [tty0] enabled Aug 4 11:25:27 rhel6-node2 kernel: allocated 2621440 bytes of page_cgroup Aug 4 11:25:27 rhel6-node2 kernel: please try 'cgroup_disable=memory' option if you don't want memory cgroups Aug 4 11:25:27 rhel6-node2 kernel: Detected 2659.828 MHz processor. Aug 4 11:25:27 rhel6-node2 kernel: Calibrating delay loop (skipped) preset value.. 5319.65 BogoMIPS (lpj=2659828) Aug 4 11:25:27 rhel6-node2 kernel: pid_max: default: 32768 minimum: 301 Aug 4 11:25:27 rhel6-node2 kernel: Security Framework initialized Aug 4 11:25:27 rhel6-node2 kernel: SELinux: Initializing. Aug 4 11:25:27 rhel6-node2 kernel: Mount-cache hash table entries: 512 Aug 4 11:25:27 rhel6-node2 kernel: Initializing cgroup subsys ns Aug 4 11:25:27 rhel6-node2 kernel: Initializing cgroup subsys cpuacct Aug 4 11:25:27 rhel6-node2 kernel: Initializing cgroup subsys memory Aug 4 11:25:27 rhel6-node2 kernel: Initializing cgroup subsys devices Aug 4 11:25:27 rhel6-node2 kernel: Initializing cgroup subsys freezer Aug 4 11:25:27 rhel6-node2 kernel: Initializing cgroup subsys net_cls Aug 4 11:25:27 rhel6-node2 kernel: Initializing cgroup subsys blkio Aug 4 11:25:27 rhel6-node2 kernel: mce: CPU supports 10 MCE banks Aug 4 11:25:27 rhel6-node2 kernel: Performance Events: unsupported p6 CPU model 2 no PMU driver, software events only. Aug 4 11:25:27 rhel6-node2 kernel: alternatives: switching to unfair spinlock Aug 4 11:25:27 rhel6-node2 kernel: SMP alternatives: switching to UP code Aug 4 11:25:27 rhel6-node2 kernel: Freeing SMP alternatives: 15k freed Aug 4 11:25:27 rhel6-node2 kernel: ACPI: Core revision 20090903 Aug 4 11:25:27 rhel6-node2 kernel: Enabling APIC mode: Flat. Using 1 I/O APICs Aug 4 11:25:27 rhel6-node2 kernel: ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 Aug 4 11:25:27 rhel6-node2 kernel: CPU0: Intel QEMU Virtual CPU version 0.12.3 stepping 03 Aug 4 11:25:27 rhel6-node2 kernel: Brought up 1 CPUs Aug 4 11:25:27 rhel6-node2 kernel: Total of 1 processors activated (5319.65 BogoMIPS). Aug 4 11:25:27 rhel6-node2 kernel: devtmpfs: initialized Aug 4 11:25:27 rhel6-node2 kernel: regulator: core version 0.5 Aug 4 11:25:27 rhel6-node2 kernel: NET: Registered protocol family 16 Aug 4 11:25:27 rhel6-node2 kernel: ACPI: bus type pci registered Aug 4 11:25:27 rhel6-node2 kernel: PCI: PCI BIOS revision 2.10 entry at 0xffe77, last bus=0 Aug 4 11:25:27 rhel6-node2 kernel: PCI: Using configuration type 1 for base access Aug 4 11:25:27 rhel6-node2 kernel: bio: create slab at 0 Aug 4 11:25:27 rhel6-node2 kernel: ACPI: Interpreter enabled Aug 4 11:25:27 rhel6-node2 kernel: ACPI: (supports S0 S3 S4 S5) Aug 4 11:25:27 rhel6-node2 kernel: ACPI: Using IOAPIC for interrupt routing Aug 4 11:25:27 rhel6-node2 kernel: ACPI: No dock devices found. Aug 4 11:25:27 rhel6-node2 kernel: ACPI: PCI Root Bridge [PCI0] (0000:00) Aug 4 11:25:27 rhel6-node2 kernel: pci 0000:00:01.3: quirk: region b000-b03f claimed by PIIX4 ACPI Aug 4 11:25:27 rhel6-node2 kernel: pci 0000:00:01.3: quirk: region b100-b10f claimed by PIIX4 SMB Aug 4 11:25:27 rhel6-node2 kernel: Unable to assume PCIe control: Disabling ASPM Aug 4 11:25:27 rhel6-node2 kernel: ACPI: PCI Interrupt Link [LNKA] (IRQs 5 *10 11) Aug 4 11:25:27 rhel6-node2 kernel: ACPI: PCI Interrupt Link [LNKB] (IRQs 5 *10 11) Aug 4 11:25:27 rhel6-node2 kernel: ACPI: PCI Interrupt Link [LNKC] (IRQs 5 10 *11) Aug 4 11:25:27 rhel6-node2 kernel: ACPI: PCI Interrupt Link [LNKD] (IRQs 5 10 *11) Aug 4 11:25:27 rhel6-node2 kernel: vgaarb: device added: PCI:0000:00:02.0,decodes=io+mem,owns=io+mem,locks=none Aug 4 11:25:27 rhel6-node2 kernel: vgaarb: loaded Aug 4 11:25:27 rhel6-node2 kernel: SCSI subsystem initialized Aug 4 11:25:27 rhel6-node2 kernel: usbcore: registered new interface driver usbfs Aug 4 11:25:27 rhel6-node2 kernel: usbcore: registered new interface driver hub Aug 4 11:25:27 rhel6-node2 kernel: usbcore: registered new device driver usb Aug 4 11:25:27 rhel6-node2 kernel: PCI: Using ACPI for IRQ routing Aug 4 11:25:27 rhel6-node2 kernel: NetLabel: Initializing Aug 4 11:25:27 rhel6-node2 kernel: NetLabel: domain hash size = 128 Aug 4 11:25:27 rhel6-node2 kernel: NetLabel: protocols = UNLABELED CIPSOv4 Aug 4 11:25:27 rhel6-node2 kernel: NetLabel: unlabeled traffic allowed by default Aug 4 11:25:27 rhel6-node2 kernel: Switching to clocksource kvm-clock Aug 4 11:25:27 rhel6-node2 kernel: pnp: PnP ACPI init Aug 4 11:25:27 rhel6-node2 kernel: ACPI: bus type pnp registered Aug 4 11:25:27 rhel6-node2 kernel: pnp: PnP ACPI: found 6 devices Aug 4 11:25:27 rhel6-node2 kernel: ACPI: ACPI bus type pnp unregistered Aug 4 11:25:27 rhel6-node2 kernel: NET: Registered protocol family 2 Aug 4 11:25:27 rhel6-node2 kernel: IP route cache hash table entries: 4096 (order: 2, 16384 bytes) Aug 4 11:25:27 rhel6-node2 kernel: TCP established hash table entries: 16384 (order: 5, 131072 bytes) Aug 4 11:25:27 rhel6-node2 kernel: TCP bind hash table entries: 16384 (order: 5, 131072 bytes) Aug 4 11:25:27 rhel6-node2 kernel: TCP: Hash tables configured (established 16384 bind 16384) Aug 4 11:25:27 rhel6-node2 kernel: TCP reno registered Aug 4 11:25:27 rhel6-node2 kernel: NET: Registered protocol family 1 Aug 4 11:25:27 rhel6-node2 kernel: pci 0000:00:00.0: Limiting direct PCI/PCI transfers Aug 4 11:25:27 rhel6-node2 kernel: pci 0000:00:01.0: PIIX3: Enabling Passive Release Aug 4 11:25:27 rhel6-node2 kernel: pci 0000:00:01.0: Activating ISA DMA hang workarounds Aug 4 11:25:27 rhel6-node2 kernel: Trying to unpack rootfs image as initramfs... Aug 4 11:25:27 rhel6-node2 kernel: Freeing initrd memory: 12103k freed Aug 4 11:25:27 rhel6-node2 kernel: apm: BIOS not found. Aug 4 11:25:27 rhel6-node2 kernel: audit: initializing netlink socket (disabled) Aug 4 11:25:27 rhel6-node2 kernel: type=2000 audit(1280913922.508:1): initialized Aug 4 11:25:27 rhel6-node2 kernel: HugeTLB registered 2 MB page size, pre-allocated 0 pages Aug 4 11:25:27 rhel6-node2 kernel: VFS: Disk quotas dquot_6.5.2 Aug 4 11:25:27 rhel6-node2 kernel: Dquot-cache hash table entries: 1024 (order 0, 4096 bytes) Aug 4 11:25:27 rhel6-node2 kernel: msgmni has been set to 998 Aug 4 11:25:27 rhel6-node2 kernel: alg: No test for stdrng (krng) Aug 4 11:25:27 rhel6-node2 kernel: ksign: Installing public key data Aug 4 11:25:27 rhel6-node2 kernel: Loading keyring Aug 4 11:25:27 rhel6-node2 kernel: - Added public key ECEC034A9B73CBB4 Aug 4 11:25:27 rhel6-node2 kernel: - User ID: Red Hat, Inc. (Kernel Module GPG key) Aug 4 11:25:27 rhel6-node2 kernel: - Added public key D4A26C9CCD09BEDA Aug 4 11:25:27 rhel6-node2 kernel: - User ID: Red Hat Enterprise Linux Driver Update Program Aug 4 11:25:27 rhel6-node2 kernel: Block layer SCSI generic (bsg) driver version 0.4 loaded (major 252) Aug 4 11:25:27 rhel6-node2 kernel: io scheduler noop registered Aug 4 11:25:27 rhel6-node2 kernel: io scheduler anticipatory registered Aug 4 11:25:27 rhel6-node2 kernel: io scheduler deadline registered Aug 4 11:25:27 rhel6-node2 kernel: io scheduler cfq registered (default) Aug 4 11:25:27 rhel6-node2 kernel: pci_hotplug: PCI Hot Plug PCI Core version: 0.5 Aug 4 11:25:27 rhel6-node2 kernel: pciehp: PCI Express Hot Plug Controller Driver version: 0.4 Aug 4 11:25:27 rhel6-node2 kernel: acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 Aug 4 11:25:27 rhel6-node2 kernel: acpiphp: Slot [1] registered Aug 4 11:25:27 rhel6-node2 kernel: acpiphp: Slot [2] registered Aug 4 11:25:27 rhel6-node2 kernel: acpiphp: Slot [3] registered Aug 4 11:25:27 rhel6-node2 kernel: acpiphp: Slot [4] registered Aug 4 11:25:27 rhel6-node2 kernel: acpiphp: Slot [5] registered Aug 4 11:25:27 rhel6-node2 kernel: acpiphp: Slot [6] registered Aug 4 11:25:27 rhel6-node2 kernel: acpiphp: Slot [7] registered Aug 4 11:25:27 rhel6-node2 kernel: acpiphp: Slot [8] registered Aug 4 11:25:27 rhel6-node2 kernel: acpiphp: Slot [9] registered Aug 4 11:25:27 rhel6-node2 kernel: acpiphp: Slot [10] registered Aug 4 11:25:27 rhel6-node2 kernel: acpiphp: Slot [11] registered Aug 4 11:25:27 rhel6-node2 kernel: acpiphp: Slot [12] registered Aug 4 11:25:27 rhel6-node2 kernel: acpiphp: Slot [13] registered Aug 4 11:25:27 rhel6-node2 kernel: acpiphp: Slot [14] registered Aug 4 11:25:27 rhel6-node2 kernel: acpiphp: Slot [15] registered Aug 4 11:25:27 rhel6-node2 kernel: acpiphp: Slot [16] registered Aug 4 11:25:27 rhel6-node2 kernel: acpiphp: Slot [17] registered Aug 4 11:25:27 rhel6-node2 kernel: acpiphp: Slot [18] registered Aug 4 11:25:27 rhel6-node2 kernel: acpiphp: Slot [19] registered Aug 4 11:25:27 rhel6-node2 kernel: acpiphp: Slot [20] registered Aug 4 11:25:27 rhel6-node2 kernel: acpiphp: Slot [21] registered Aug 4 11:25:27 rhel6-node2 kernel: acpiphp: Slot [22] registered Aug 4 11:25:27 rhel6-node2 kernel: acpiphp: Slot [23] registered Aug 4 11:25:27 rhel6-node2 kernel: acpiphp: Slot [24] registered Aug 4 11:25:27 rhel6-node2 kernel: acpiphp: Slot [25] registered Aug 4 11:25:27 rhel6-node2 kernel: acpiphp: Slot [26] registered Aug 4 11:25:27 rhel6-node2 kernel: acpiphp: Slot [27] registered Aug 4 11:25:27 rhel6-node2 kernel: acpiphp: Slot [28] registered Aug 4 11:25:27 rhel6-node2 kernel: acpiphp: Slot [29] registered Aug 4 11:25:27 rhel6-node2 kernel: acpiphp: Slot [30] registered Aug 4 11:25:27 rhel6-node2 kernel: acpiphp: Slot [31] registered Aug 4 11:25:27 rhel6-node2 kernel: pci-stub: invalid id string "" Aug 4 11:25:27 rhel6-node2 kernel: input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 Aug 4 11:25:27 rhel6-node2 kernel: ACPI: Power Button [PWRF] Aug 4 11:25:27 rhel6-node2 kernel: ACPI: CPU Hot Add is currently disabled for x86 32-bit. Aug 4 11:25:27 rhel6-node2 kernel: processor: probe of LNXCPU:00 failed with error -22 Aug 4 11:25:27 rhel6-node2 kernel: isapnp: Scanning for PnP cards... Aug 4 11:25:27 rhel6-node2 kernel: isapnp: No Plug & Play device found Aug 4 11:25:27 rhel6-node2 kernel: hpet_acpi_add: no address or irqs in _CRS Aug 4 11:25:27 rhel6-node2 kernel: Non-volatile memory driver v1.3 Aug 4 11:25:27 rhel6-node2 kernel: Linux agpgart interface v0.103 Aug 4 11:25:27 rhel6-node2 kernel: crash memory driver: version 1.0 Aug 4 11:25:27 rhel6-node2 kernel: Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled Aug 4 11:25:27 rhel6-node2 kernel: serial8250: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A Aug 4 11:25:27 rhel6-node2 kernel: brd: module loaded Aug 4 11:25:27 rhel6-node2 kernel: loop: module loaded Aug 4 11:25:27 rhel6-node2 kernel: input: Macintosh mouse button emulation as /devices/virtual/input/input1 Aug 4 11:25:27 rhel6-node2 kernel: Fixed MDIO Bus: probed Aug 4 11:25:27 rhel6-node2 kernel: ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver Aug 4 11:25:27 rhel6-node2 kernel: ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver Aug 4 11:25:27 rhel6-node2 kernel: uhci_hcd: USB Universal Host Controller Interface driver Aug 4 11:25:27 rhel6-node2 kernel: ACPI: PCI Interrupt Link [LNKD] enabled at IRQ 11 Aug 4 11:25:27 rhel6-node2 kernel: uhci_hcd 0000:00:01.2: PCI INT D -> Link[LNKD] -> GSI 11 (level, high) -> IRQ 11 Aug 4 11:25:27 rhel6-node2 kernel: uhci_hcd 0000:00:01.2: UHCI Host Controller Aug 4 11:25:27 rhel6-node2 kernel: uhci_hcd 0000:00:01.2: new USB bus registered, assigned bus number 1 Aug 4 11:25:27 rhel6-node2 kernel: uhci_hcd 0000:00:01.2: irq 11, io base 0x0000c020 Aug 4 11:25:27 rhel6-node2 kernel: usb usb1: New USB device found, idVendor=1d6b, idProduct=0001 Aug 4 11:25:27 rhel6-node2 kernel: usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1 Aug 4 11:25:27 rhel6-node2 kernel: usb usb1: Product: UHCI Host Controller Aug 4 11:25:27 rhel6-node2 kernel: usb usb1: Manufacturer: Linux 2.6.32-54.el6.i686 uhci_hcd Aug 4 11:25:27 rhel6-node2 kernel: usb usb1: SerialNumber: 0000:00:01.2 Aug 4 11:25:27 rhel6-node2 kernel: usb usb1: configuration #1 chosen from 1 choice Aug 4 11:25:27 rhel6-node2 kernel: hub 1-0:1.0: USB hub found Aug 4 11:25:27 rhel6-node2 kernel: hub 1-0:1.0: 2 ports detected Aug 4 11:25:27 rhel6-node2 kernel: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 Aug 4 11:25:27 rhel6-node2 kernel: serio: i8042 KBD port at 0x60,0x64 irq 1 Aug 4 11:25:27 rhel6-node2 kernel: serio: i8042 AUX port at 0x60,0x64 irq 12 Aug 4 11:25:27 rhel6-node2 kernel: mice: PS/2 mouse device common for all mice Aug 4 11:25:27 rhel6-node2 kernel: input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input2 Aug 4 11:25:27 rhel6-node2 kernel: rtc_cmos 00:01: rtc core: registered rtc_cmos as rtc0 Aug 4 11:25:27 rhel6-node2 kernel: rtc0: alarms up to one day, 114 bytes nvram Aug 4 11:25:27 rhel6-node2 kernel: cpuidle: using governor ladder Aug 4 11:25:27 rhel6-node2 kernel: cpuidle: using governor menu Aug 4 11:25:27 rhel6-node2 kernel: usbcore: registered new interface driver hiddev Aug 4 11:25:27 rhel6-node2 kernel: usbcore: registered new interface driver usbhid Aug 4 11:25:27 rhel6-node2 kernel: usbhid: v2.6:USB HID core driver Aug 4 11:25:27 rhel6-node2 kernel: TCP cubic registered Aug 4 11:25:27 rhel6-node2 kernel: Initializing XFRM netlink socket Aug 4 11:25:27 rhel6-node2 kernel: NET: Registered protocol family 17 Aug 4 11:25:27 rhel6-node2 kernel: Using IPI No-Shortcut mode Aug 4 11:25:27 rhel6-node2 kernel: registered taskstats version 1 Aug 4 11:25:27 rhel6-node2 kernel: IMA: No TPM chip found, activating TPM-bypass! Aug 4 11:25:27 rhel6-node2 kernel: rtc_cmos 00:01: setting system clock to 2010-08-04 09:25:21 UTC (1280913921) Aug 4 11:25:27 rhel6-node2 kernel: Initalizing network drop monitor service Aug 4 11:25:27 rhel6-node2 kernel: Freeing unused kernel memory: 524k freed Aug 4 11:25:27 rhel6-node2 kernel: Write protecting the kernel text: 4176k Aug 4 11:25:27 rhel6-node2 kernel: Write protecting the kernel read-only data: 1812k Aug 4 11:25:27 rhel6-node2 kernel: dracut: dracut-004-29.el6 Aug 4 11:25:27 rhel6-node2 kernel: dracut: rd_NO_LUKS: removing cryptoluks activation Aug 4 11:25:27 rhel6-node2 kernel: device-mapper: uevent: version 1.0.3 Aug 4 11:25:27 rhel6-node2 kernel: device-mapper: ioctl: 4.17.0-ioctl (2010-03-05) initialised: dm-devel@redhat.com Aug 4 11:25:27 rhel6-node2 kernel: udev: starting version 147 Aug 4 11:25:27 rhel6-node2 kernel: dracut: Starting plymouth daemon Aug 4 11:25:27 rhel6-node2 kernel: dracut: rd_NO_DM: removing DM RAID activation Aug 4 11:25:27 rhel6-node2 kernel: dracut: rd_NO_MD: removing MD RAID activation Aug 4 11:25:27 rhel6-node2 kernel: usb 1-1: new full speed USB device using uhci_hcd and address 2 Aug 4 11:25:27 rhel6-node2 kernel: input: ImExPS/2 Generic Explorer Mouse as /devices/platform/i8042/serio1/input/input3 Aug 4 11:25:27 rhel6-node2 kernel: scsi0 : ata_piix Aug 4 11:25:27 rhel6-node2 kernel: scsi1 : ata_piix Aug 4 11:25:27 rhel6-node2 kernel: ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc000 irq 14 Aug 4 11:25:27 rhel6-node2 kernel: ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc008 irq 15 Aug 4 11:25:27 rhel6-node2 kernel: usb 1-1: New USB device found, idVendor=0627, idProduct=0001 Aug 4 11:25:27 rhel6-node2 kernel: usb 1-1: New USB device strings: Mfr=3, Product=2, SerialNumber=1 Aug 4 11:25:27 rhel6-node2 kernel: usb 1-1: Product: QEMU USB Tablet Aug 4 11:25:27 rhel6-node2 kernel: usb 1-1: Manufacturer: QEMU 0.12.3 Aug 4 11:25:27 rhel6-node2 kernel: usb 1-1: SerialNumber: 1 Aug 4 11:25:27 rhel6-node2 kernel: usb 1-1: configuration #1 chosen from 1 choice Aug 4 11:25:27 rhel6-node2 kernel: input: QEMU 0.12.3 QEMU USB Tablet as /devices/pci0000:00/0000:00:01.2/usb1/1-1/1-1:1.0/input/input4 Aug 4 11:25:27 rhel6-node2 kernel: generic-usb 0003:0627:0001.0001: input,hidraw0: USB HID v0.01 Pointer [QEMU 0.12.3 QEMU USB Tablet] on usb-0000:00:01.2-1/input0 Aug 4 11:25:27 rhel6-node2 kernel: ACPI: PCI Interrupt Link [LNKC] enabled at IRQ 10 Aug 4 11:25:27 rhel6-node2 kernel: virtio-pci 0000:00:03.0: PCI INT A -> Link[LNKC] -> GSI 10 (level, high) -> IRQ 10 Aug 4 11:25:27 rhel6-node2 kernel: virtio-pci 0000:00:04.0: PCI INT A -> Link[LNKD] -> GSI 11 (level, high) -> IRQ 11 Aug 4 11:25:27 rhel6-node2 kernel: ACPI: PCI Interrupt Link [LNKA] enabled at IRQ 10 Aug 4 11:25:27 rhel6-node2 kernel: virtio-pci 0000:00:05.0: PCI INT A -> Link[LNKA] -> GSI 10 (level, high) -> IRQ 10 Aug 4 11:25:27 rhel6-node2 kernel: ACPI: PCI Interrupt Link [LNKB] enabled at IRQ 11 Aug 4 11:25:27 rhel6-node2 kernel: virtio-pci 0000:00:06.0: PCI INT A -> Link[LNKB] -> GSI 11 (level, high) -> IRQ 11 Aug 4 11:25:27 rhel6-node2 kernel: vda: vda1 vda2 Aug 4 11:25:27 rhel6-node2 kernel: vdb: vdb1 vdb2 vdb3 Aug 4 11:25:27 rhel6-node2 kernel: dracut: Scanning devices vda2 vdb3 for LVM logical volumes VolGroup/lv_root VolGroup/lv_swap Aug 4 11:25:27 rhel6-node2 kernel: dracut: Skipping clustered volume group test Aug 4 11:25:27 rhel6-node2 kernel: dracut: Skipping volume group test Aug 4 11:25:27 rhel6-node2 kernel: dracut: inactive '/dev/VolGroup/lv_root' [10.51 GiB] inherit Aug 4 11:25:27 rhel6-node2 kernel: dracut: inactive '/dev/VolGroup/lv_swap' [1.00 GiB] inherit Aug 4 11:25:27 rhel6-node2 kernel: EXT4-fs (dm-0): INFO: recovery required on readonly filesystem Aug 4 11:25:27 rhel6-node2 kernel: EXT4-fs (dm-0): write access will be enabled during recovery Aug 4 11:25:27 rhel6-node2 kernel: EXT4-fs (dm-0): orphan cleanup on readonly fs Aug 4 11:25:27 rhel6-node2 kernel: EXT4-fs (dm-0): 1 orphan inode deleted Aug 4 11:25:27 rhel6-node2 kernel: EXT4-fs (dm-0): recovery complete Aug 4 11:25:27 rhel6-node2 kernel: EXT4-fs (dm-0): mounted filesystem with ordered data mode Aug 4 11:25:27 rhel6-node2 kernel: dracut: Mounted root filesystem /dev/mapper/VolGroup-lv_root Aug 4 11:25:27 rhel6-node2 kernel: dracut: Loading SELinux policy Aug 4 11:25:27 rhel6-node2 kernel: SELinux: Disabled at runtime. Aug 4 11:25:27 rhel6-node2 kernel: type=1404 audit(1280913923.188:2): selinux=0 auid=4294967295 ses=4294967295 Aug 4 11:25:27 rhel6-node2 kernel: dracut: /sbin/load_policy: Can't load policy: No such file or directory Aug 4 11:25:27 rhel6-node2 kernel: dracut: Switching root Aug 4 11:25:27 rhel6-node2 kernel: udev: starting version 147 Aug 4 11:25:27 rhel6-node2 kernel: piix4_smbus 0000:00:01.3: SMBus Host Controller at 0xb100, revision 0 Aug 4 11:25:27 rhel6-node2 kernel: EXT4-fs (vda1): mounted filesystem with ordered data mode Aug 4 11:25:27 rhel6-node2 kernel: Adding 1048568k swap on /dev/mapper/VolGroup-lv_swap. Priority:-1 extents:1 across:1048568k Aug 4 11:25:27 rhel6-node2 kernel: NET: Registered protocol family 10 Aug 4 11:25:27 rhel6-node2 kernel: lo: Disabled Privacy Extensions Aug 4 11:25:27 rhel6-node2 rpc.statd[1118]: Version 1.2.2 starting Aug 4 11:25:27 rhel6-node2 sm-notify[1119]: Version 1.2.2 starting Aug 4 11:25:28 rhel6-node2 kernel: RPC: Registered udp transport module. Aug 4 11:25:28 rhel6-node2 kernel: RPC: Registered tcp transport module. Aug 4 11:25:28 rhel6-node2 kernel: RPC: Registered tcp NFSv4.1 backchannel transport module. Aug 4 11:25:28 rhel6-node2 kernel: Slow work thread pool: Starting up Aug 4 11:25:28 rhel6-node2 kernel: Slow work thread pool: Ready Aug 4 11:25:28 rhel6-node2 kernel: FS-Cache: Loaded Aug 4 11:25:28 rhel6-node2 kernel: FS-Cache: Netfs 'nfs' registered for caching Aug 4 11:25:28 rhel6-node2 acpid: starting up Aug 4 11:25:28 rhel6-node2 acpid: 1 rule loaded Aug 4 11:25:28 rhel6-node2 acpid: waiting for events: event logging is off Aug 4 11:25:28 rhel6-node2 kernel: Installing knfsd (copyright (C) 1996 okir@monad.swb.de). Aug 4 11:25:28 rhel6-node2 kernel: NFSD: Using /var/lib/nfs/v4recovery as the NFSv4 state recovery directory Aug 4 11:25:28 rhel6-node2 kernel: NFSD: starting 90-second grace period Aug 4 11:28:10 rhel6-node2 kernel: DLM (built Jul 27 2010 23:36:51) installed Aug 4 11:28:10 rhel6-node2 corosync[1456]: [MAIN ] Corosync Cluster Engine ('1.2.3'): started and ready to provide service. Aug 4 11:28:10 rhel6-node2 corosync[1456]: [MAIN ] Corosync built-in features: nss rdma Aug 4 11:28:10 rhel6-node2 corosync[1456]: [MAIN ] Successfully read config from /etc/cluster/cluster.conf Aug 4 11:28:10 rhel6-node2 corosync[1456]: [MAIN ] Successfully parsed cman config Aug 4 11:28:10 rhel6-node2 corosync[1456]: [TOTEM ] Initializing transport (UDP/IP). Aug 4 11:28:10 rhel6-node2 corosync[1456]: [TOTEM ] Initializing transmit/receive security: libtomcrypt SOBER128/SHA1HMAC (mode 0). Aug 4 11:28:10 rhel6-node2 corosync[1456]: [TOTEM ] The network interface [192.168.2.66] is now up. Aug 4 11:28:10 rhel6-node2 corosync[1456]: [QUORUM] Using quorum provider quorum_cman Aug 4 11:28:10 rhel6-node2 corosync[1456]: [SERV ] Service engine loaded: corosync cluster quorum service v0.1 Aug 4 11:28:10 rhel6-node2 corosync[1456]: [CMAN ] CMAN 3.0.12 (built Jul 28 2010 04:37:33) started Aug 4 11:28:10 rhel6-node2 corosync[1456]: [SERV ] Service engine loaded: corosync CMAN membership service 2.90 Aug 4 11:28:10 rhel6-node2 corosync[1456]: [SERV ] Service engine loaded: openais checkpoint service B.01.01 Aug 4 11:28:10 rhel6-node2 corosync[1456]: [SERV ] Service engine loaded: corosync extended virtual synchrony service Aug 4 11:28:10 rhel6-node2 corosync[1456]: [SERV ] Service engine loaded: corosync configuration service Aug 4 11:28:10 rhel6-node2 corosync[1456]: [SERV ] Service engine loaded: corosync cluster closed process group service v1.01 Aug 4 11:28:10 rhel6-node2 corosync[1456]: [SERV ] Service engine loaded: corosync cluster config database access v1.01 Aug 4 11:28:10 rhel6-node2 corosync[1456]: [SERV ] Service engine loaded: corosync profile loading service Aug 4 11:28:10 rhel6-node2 corosync[1456]: [QUORUM] Using quorum provider quorum_cman Aug 4 11:28:10 rhel6-node2 corosync[1456]: [SERV ] Service engine loaded: corosync cluster quorum service v0.1 Aug 4 11:28:10 rhel6-node2 corosync[1456]: [MAIN ] Compatibility mode set to whitetank. Using V1 and V2 of the synchronization engine. Aug 4 11:28:10 rhel6-node2 corosync[1456]: [TOTEM ] A processor joined or left the membership and a new membership was formed. Aug 4 11:28:10 rhel6-node2 corosync[1456]: [QUORUM] Members[1]: 2 Aug 4 11:28:10 rhel6-node2 corosync[1456]: [QUORUM] Members[1]: 2 Aug 4 11:28:10 rhel6-node2 corosync[1456]: [CPG ] downlist received left_list: 0 Aug 4 11:28:10 rhel6-node2 corosync[1456]: [CPG ] chosen downlist from node r(0) ip(192.168.2.66) Aug 4 11:28:10 rhel6-node2 corosync[1456]: [MAIN ] Completed service synchronization, ready to provide service. Aug 4 11:28:10 rhel6-node2 corosync[1456]: [TOTEM ] A processor joined or left the membership and a new membership was formed. Aug 4 11:28:10 rhel6-node2 corosync[1456]: [QUORUM] Members[2]: 2 3 Aug 4 11:28:10 rhel6-node2 corosync[1456]: [QUORUM] Members[2]: 2 3 Aug 4 11:28:20 rhel6-node2 corosync[1456]: [TOTEM ] A processor joined or left the membership and a new membership was formed. Aug 4 11:28:20 rhel6-node2 corosync[1456]: [QUORUM] Members[3]: 2 3 4 Aug 4 11:28:20 rhel6-node2 corosync[1456]: [QUORUM] Members[3]: 2 3 4 Aug 4 11:28:20 rhel6-node2 corosync[1456]: [QUORUM] Members[4]: 2 3 4 5 Aug 4 11:28:20 rhel6-node2 corosync[1456]: [QUORUM] Members[4]: 2 3 4 5 Aug 4 11:28:20 rhel6-node2 corosync[1456]: [CMAN ] quorum regained, resuming activity Aug 4 11:28:20 rhel6-node2 corosync[1456]: [QUORUM] This node is within the primary component and will provide service. Aug 4 11:28:20 rhel6-node2 corosync[1456]: [QUORUM] Members[5]: 2 3 4 5 6 Aug 4 11:28:20 rhel6-node2 corosync[1456]: [QUORUM] Members[5]: 2 3 4 5 6 Aug 4 11:28:20 rhel6-node2 corosync[1456]: [QUORUM] Members[6]: 2 3 4 5 6 7 Aug 4 11:28:20 rhel6-node2 corosync[1456]: [QUORUM] Members[6]: 2 3 4 5 6 7 Aug 4 11:28:20 rhel6-node2 corosync[1456]: [QUORUM] Members[7]: 2 3 4 5 6 7 8 Aug 4 11:28:20 rhel6-node2 corosync[1456]: [QUORUM] Members[7]: 2 3 4 5 6 7 8 Aug 4 11:28:20 rhel6-node2 corosync[1456]: [QUORUM] Members[8]: 1 2 3 4 5 6 7 8 Aug 4 11:28:20 rhel6-node2 corosync[1456]: [QUORUM] Members[8]: 1 2 3 4 5 6 7 8 Aug 4 11:28:20 rhel6-node2 corosync[1456]: [CPG ] downlist received left_list: 0 Aug 4 11:28:20 rhel6-node2 corosync[1456]: [CPG ] downlist received left_list: 0 Aug 4 11:28:20 rhel6-node2 corosync[1456]: [CPG ] downlist received left_list: 0 Aug 4 11:28:20 rhel6-node2 corosync[1456]: [CPG ] downlist received left_list: 0 Aug 4 11:28:20 rhel6-node2 corosync[1456]: [CPG ] downlist received left_list: 0 Aug 4 11:28:20 rhel6-node2 corosync[1456]: [CPG ] downlist received left_list: 0 Aug 4 11:28:20 rhel6-node2 corosync[1456]: [CPG ] downlist received left_list: 0 Aug 4 11:28:20 rhel6-node2 corosync[1456]: [CPG ] downlist received left_list: 0 Aug 4 11:28:20 rhel6-node2 corosync[1456]: [CPG ] chosen downlist from node r(0) ip(192.168.2.66) Aug 4 11:28:20 rhel6-node2 corosync[1456]: [MAIN ] Completed service synchronization, ready to provide service. Aug 4 11:28:21 rhel6-node2 fenced[1512]: fenced 3.0.12 started Aug 4 11:28:22 rhel6-node2 dlm_controld[1532]: dlm_controld 3.0.12 started Aug 4 11:28:22 rhel6-node2 gfs_controld[1580]: gfs_controld 3.0.12 started Aug 4 11:28:29 rhel6-node2 kernel: dlm: Using TCP for communications Aug 4 11:28:29 rhel6-node2 kernel: dlm: got connection from 1 Aug 4 11:28:29 rhel6-node2 kernel: dlm: got connection from 6 Aug 4 11:28:29 rhel6-node2 kernel: dlm: got connection from 8 Aug 4 11:28:29 rhel6-node2 kernel: dlm: got connection from 3 Aug 4 11:28:29 rhel6-node2 kernel: dlm: got connection from 7 Aug 4 11:28:29 rhel6-node2 kernel: dlm: got connection from 5 Aug 4 11:28:29 rhel6-node2 kernel: dlm: connecting to 4 Aug 4 11:28:30 rhel6-node2 kernel: dlm: got connection from 4 Aug 4 11:28:30 rhel6-node2 clvmd: Cluster LVM daemon started - connected to CMAN Aug 4 11:28:40 rhel6-node2 kernel: GFS2 (built Jul 27 2010 23:37:30) installed Aug 4 11:28:40 rhel6-node2 kernel: GFS2: fsid=: Trying to join cluster "lock_dlm", "fabbione:gfs2" Aug 4 11:28:40 rhel6-node2 kernel: GFS2: fsid=fabbione:gfs2.1: Joined cluster. Now mounting FS... Aug 4 11:28:41 rhel6-node2 kernel: GFS2: fsid=fabbione:gfs2.1: jid=1, already locked for use Aug 4 11:28:41 rhel6-node2 kernel: GFS2: fsid=fabbione:gfs2.1: jid=1: Looking at journal... Aug 4 11:28:41 rhel6-node2 kernel: GFS2: fsid=fabbione:gfs2.1: jid=1: Done Aug 4 11:30:56 rhel6-node2 corosync[1456]: [TOTEM ] Process pause detected for 7104 ms, flushing membership messages. Aug 4 11:30:56 rhel6-node2 corosync[1456]: [TOTEM ] A processor failed, forming new configuration. Aug 4 11:31:08 rhel6-node2 corosync[1456]: [QUORUM] Members[7]: 2 3 4 5 6 7 8 Aug 4 11:31:08 rhel6-node2 corosync[1456]: [QUORUM] Members[6]: 2 3 4 6 7 8 Aug 4 11:31:08 rhel6-node2 corosync[1456]: [QUORUM] Members[5]: 2 3 4 7 8 Aug 4 11:31:08 rhel6-node2 corosync[1456]: [CMAN ] quorum lost, blocking activity Aug 4 11:31:08 rhel6-node2 corosync[1456]: [QUORUM] This node is within the non-primary component and will NOT provide any services. Aug 4 11:31:08 rhel6-node2 corosync[1456]: [QUORUM] Members[4]: 2 3 4 8 Aug 4 11:31:08 rhel6-node2 corosync[1456]: [TOTEM ] A processor joined or left the membership and a new membership was formed. Aug 4 11:31:08 rhel6-node2 kernel: dlm: closing connection to node 1 Aug 4 11:31:08 rhel6-node2 kernel: dlm: closing connection to node 5 Aug 4 11:31:08 rhel6-node2 kernel: dlm: closing connection to node 6 Aug 4 11:31:08 rhel6-node2 kernel: dlm: closing connection to node 7 Aug 4 11:31:21 rhel6-node2 corosync[1456]: [TOTEM ] A processor joined or left the membership and a new membership was formed. Aug 4 11:31:23 rhel6-node2 corosync[1456]: [CMAN ] quorum regained, resuming activity Aug 4 11:31:23 rhel6-node2 corosync[1456]: [QUORUM] This node is within the primary component and will provide service. Aug 4 11:31:23 rhel6-node2 corosync[1456]: [QUORUM] Members[5]: 2 3 4 5 8 Aug 4 11:31:23 rhel6-node2 corosync[1456]: [QUORUM] Members[5]: 2 3 4 5 8 Aug 4 11:31:23 rhel6-node2 corosync[1456]: [QUORUM] Members[6]: 2 3 4 5 6 8 Aug 4 11:31:23 rhel6-node2 corosync[1456]: [QUORUM] Members[6]: 2 3 4 5 6 8 Aug 4 11:31:23 rhel6-node2 corosync[1456]: [QUORUM] Members[7]: 2 3 4 5 6 7 8 Aug 4 11:31:23 rhel6-node2 corosync[1456]: [QUORUM] Members[7]: 2 3 4 5 6 7 8 Aug 4 11:31:23 rhel6-node2 corosync[1456]: [QUORUM] Members[8]: 1 2 3 4 5 6 7 8 Aug 4 11:31:23 rhel6-node2 corosync[1456]: [QUORUM] Members[8]: 1 2 3 4 5 6 7 8 Aug 4 11:31:32 rhel6-node2 corosync[1456]: [CPG ] downlist received left_list: 4 Aug 4 11:31:32 rhel6-node2 corosync[1456]: [CPG ] downlist received left_list: 4 Aug 4 11:31:32 rhel6-node2 corosync[1456]: [CPG ] downlist received left_list: 4 Aug 4 11:31:32 rhel6-node2 corosync[1456]: [CPG ] downlist received left_list: 4 Aug 4 11:31:32 rhel6-node2 corosync[1456]: [CPG ] downlist received left_list: 4 Aug 4 11:31:32 rhel6-node2 corosync[1456]: [CPG ] downlist received left_list: 4 Aug 4 11:31:32 rhel6-node2 corosync[1456]: [CPG ] downlist received left_list: 4 Aug 4 11:31:32 rhel6-node2 corosync[1456]: [CPG ] downlist received left_list: 4 Aug 4 11:31:32 rhel6-node2 corosync[1456]: [CPG ] chosen downlist from node r(0) ip(192.168.2.65) Aug 4 11:31:33 rhel6-node2 gfs_controld[1580]: cpg_mcast_joined retry 100 start Aug 4 11:31:33 rhel6-node2 corosync[1456]: [MAIN ] Completed service synchronization, ready to provide service. Aug 4 11:31:33 rhel6-node2 gfs_controld[1580]: cpg_mcast_joined error 12 handle 836c40e00000001 start Aug 4 11:31:33 rhel6-node2 fenced[1512]: cpg_mcast_joined error 12 handle 520eedd100000001 start Aug 4 11:35:21 rhel6-node2 kernel: INFO: task flush-252:16:882 blocked for more than 120 seconds. Aug 4 11:35:21 rhel6-node2 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Aug 4 11:35:21 rhel6-node2 kernel: flush-252:16 D dc524ac0 0 882 2 0x00000000 Aug 4 11:35:21 rhel6-node2 kernel: df8ffa90 00000046 95e2beb5 dc524ac0 d3399cf8 00000001 00000001 c0542788 Aug 4 11:35:21 rhel6-node2 kernel: d3399cf8 dfacfe40 00000056 c0ac7080 05b026cf 00000056 c0ac7080 df8ffd38 Aug 4 11:35:21 rhel6-node2 kernel: c0ac7080 c0ac2b34 c0ac7080 df8ffd38 00000001 e1142ed0 0001133e dd8f94b4 Aug 4 11:35:21 rhel6-node2 kernel: Call Trace: Aug 4 11:35:21 rhel6-node2 kernel: [] ? submit_bh+0xe8/0x120 Aug 4 11:35:21 rhel6-node2 kernel: [] ? gfs2_get_block_noalloc+0x0/0x40 [gfs2] Aug 4 11:35:21 rhel6-node2 kernel: [] ? gfs2_glock_holder_wait+0x5/0x10 [gfs2] Aug 4 11:35:21 rhel6-node2 kernel: [] ? __wait_on_bit+0x45/0x70 Aug 4 11:35:21 rhel6-node2 kernel: [] ? gfs2_glock_holder_wait+0x0/0x10 [gfs2] Aug 4 11:35:21 rhel6-node2 kernel: [] ? gfs2_glock_holder_wait+0x0/0x10 [gfs2] Aug 4 11:35:21 rhel6-node2 kernel: [] ? out_of_line_wait_on_bit+0x68/0x80 Aug 4 11:35:21 rhel6-node2 kernel: [] ? wake_bit_function+0x0/0x60 Aug 4 11:35:21 rhel6-node2 kernel: [] ? gfs2_glock_wait+0x2a/0x40 [gfs2] Aug 4 11:35:21 rhel6-node2 kernel: [] ? gfs2_write_inode+0x72/0x180 [gfs2] Aug 4 11:35:21 rhel6-node2 kernel: [] ? gfs2_write_inode+0x69/0x180 [gfs2] Aug 4 11:35:21 rhel6-node2 kernel: [] ? writeback_single_inode+0x1c8/0x270 Aug 4 11:35:21 rhel6-node2 kernel: [] ? writeback_sb_inodes+0xc1/0x180 Aug 4 11:35:21 rhel6-node2 kernel: [] ? writeback_inodes_wb+0x83/0x150 Aug 4 11:35:21 rhel6-node2 kernel: [] ? wb_writeback+0x1e3/0x230 Aug 4 11:35:21 rhel6-node2 kernel: [] ? wb_do_writeback+0x7e/0x190 Aug 4 11:35:21 rhel6-node2 kernel: [] ? bdi_writeback_task+0x2a/0xa0 Aug 4 11:35:21 rhel6-node2 kernel: [] ? bdi_start_fn+0x67/0xc0 Aug 4 11:35:21 rhel6-node2 kernel: [] ? bdi_start_fn+0x0/0xc0 Aug 4 11:35:21 rhel6-node2 kernel: [] ? kthread+0x74/0x80 Aug 4 11:35:21 rhel6-node2 kernel: [] ? kthread+0x0/0x80 Aug 4 11:35:21 rhel6-node2 kernel: [] ? kernel_thread_helper+0x7/0x10 Aug 4 11:35:21 rhel6-node2 kernel: INFO: task gfs2_quotad:1740 blocked for more than 120 seconds. Aug 4 11:35:21 rhel6-node2 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Aug 4 11:35:21 rhel6-node2 kernel: gfs2_quotad D 00000000 0 1740 2 0x00000080 Aug 4 11:35:21 rhel6-node2 kernel: ddb2fa90 00000046 dc70a1c4 00000000 de09add0 de661c00 dc50c000 00001586 Aug 4 11:35:21 rhel6-node2 kernel: 00000000 dc6d4580 00000055 c0ac7080 9b42a80a 00000055 c0ac7080 ddb2fd38 Aug 4 11:35:21 rhel6-node2 kernel: c0ac7080 c0ac2b34 c0ac7080 ddb2fd38 de2a4000 00000001 e113c1de 00000000 Aug 4 11:35:21 rhel6-node2 kernel: Call Trace: Aug 4 11:35:21 rhel6-node2 kernel: [] ? do_xmote+0x12e/0x200 [gfs2] Aug 4 11:35:21 rhel6-node2 kernel: [] ? gfs2_glock_holder_wait+0x5/0x10 [gfs2] Aug 4 11:35:21 rhel6-node2 kernel: [] ? __wait_on_bit+0x45/0x70 Aug 4 11:35:21 rhel6-node2 kernel: [] ? gfs2_glock_holder_wait+0x0/0x10 [gfs2] Aug 4 11:35:21 rhel6-node2 kernel: [] ? gfs2_glock_holder_wait+0x0/0x10 [gfs2] Aug 4 11:35:21 rhel6-node2 kernel: [] ? out_of_line_wait_on_bit+0x68/0x80 Aug 4 11:35:21 rhel6-node2 kernel: [] ? wake_bit_function+0x0/0x60 Aug 4 11:35:21 rhel6-node2 kernel: [] ? gfs2_glock_wait+0x2a/0x40 [gfs2] Aug 4 11:35:21 rhel6-node2 kernel: [] ? gfs2_statfs_sync+0x4c/0x1b0 [gfs2] Aug 4 11:35:21 rhel6-node2 kernel: [] ? gfs2_statfs_sync+0x45/0x1b0 [gfs2] Aug 4 11:35:21 rhel6-node2 kernel: [] ? process_timeout+0x0/0x10 Aug 4 11:35:21 rhel6-node2 kernel: [] ? quotad_check_timeo+0x3e/0xa0 [gfs2] Aug 4 11:35:21 rhel6-node2 kernel: [] ? finish_wait+0x31/0x80 Aug 4 11:35:21 rhel6-node2 kernel: [] ? gfs2_quotad+0x1fc/0x240 [gfs2] Aug 4 11:35:21 rhel6-node2 kernel: [] ? autoremove_wake_function+0x0/0x40 Aug 4 11:35:21 rhel6-node2 kernel: [] ? gfs2_quotad+0x0/0x240 [gfs2] Aug 4 11:35:21 rhel6-node2 kernel: [] ? kthread+0x74/0x80 Aug 4 11:35:21 rhel6-node2 kernel: [] ? kthread+0x0/0x80 Aug 4 11:35:21 rhel6-node2 kernel: [] ? kernel_thread_helper+0x7/0x10 Aug 4 11:35:21 rhel6-node2 kernel: INFO: task git:1746 blocked for more than 120 seconds. Aug 4 11:35:21 rhel6-node2 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Aug 4 11:35:21 rhel6-node2 kernel: git D c16090dc 0 1746 1409 0x00000080 Aug 4 11:35:21 rhel6-node2 kernel: de11b030 00000082 00000055 c16090dc ddd6058c c16090c0 ddd6058c 00010fd5 Aug 4 11:35:21 rhel6-node2 kernel: 00000000 dc6d4580 00000056 c0ac7080 055c66d9 00000056 c0ac7080 de11b2d8 Aug 4 11:35:21 rhel6-node2 kernel: c0ac7080 c0ac2b34 c0ac7080 de11b2d8 dd8c2000 00000001 e113c1de 00000001 Aug 4 11:35:21 rhel6-node2 kernel: Call Trace: Aug 4 11:35:21 rhel6-node2 kernel: [] ? do_xmote+0x12e/0x200 [gfs2] Aug 4 11:35:21 rhel6-node2 kernel: [] ? gfs2_glock_holder_wait+0x5/0x10 [gfs2] Aug 4 11:35:21 rhel6-node2 kernel: [] ? __wait_on_bit+0x45/0x70 Aug 4 11:35:21 rhel6-node2 kernel: [] ? gfs2_glock_holder_wait+0x0/0x10 [gfs2] Aug 4 11:35:21 rhel6-node2 kernel: [] ? gfs2_glock_holder_wait+0x0/0x10 [gfs2] Aug 4 11:35:21 rhel6-node2 kernel: [] ? out_of_line_wait_on_bit+0x68/0x80 Aug 4 11:35:21 rhel6-node2 kernel: [] ? wake_bit_function+0x0/0x60 Aug 4 11:35:21 rhel6-node2 kernel: [] ? gfs2_glock_wait+0x2a/0x40 [gfs2] Aug 4 11:35:21 rhel6-node2 kernel: [] ? gfs2_inplace_reserve_i+0x400/0x710 [gfs2] Aug 4 11:35:21 rhel6-node2 kernel: [] ? kmem_cache_alloc_notrace+0xa0/0xb0 Aug 4 11:35:21 rhel6-node2 kernel: [] ? gfs2_alloc_get+0x3a/0xb0 [gfs2] Aug 4 11:35:21 rhel6-node2 kernel: [] ? gfs2_write_begin+0x2f5/0x4b0 [gfs2] Aug 4 11:35:21 rhel6-node2 kernel: [] ? generic_file_buffered_write+0xef/0x270 Aug 4 11:35:21 rhel6-node2 kernel: [] ? __generic_file_aio_write+0x200/0x4e0 Aug 4 11:35:21 rhel6-node2 kernel: [] ? file_read_actor+0x7f/0xf0 Aug 4 11:35:21 rhel6-node2 kernel: [] ? generic_file_aio_write+0x5e/0xc0 Aug 4 11:35:21 rhel6-node2 kernel: [] ? gfs2_file_aio_write+0x6f/0xa0 [gfs2] Aug 4 11:35:21 rhel6-node2 kernel: [] ? do_sync_write+0xd5/0x120 Aug 4 11:35:21 rhel6-node2 kernel: [] ? autoremove_wake_function+0x0/0x40 Aug 4 11:35:21 rhel6-node2 kernel: [] ? schedule+0x42f/0xb00 Aug 4 11:35:21 rhel6-node2 kernel: [] ? security_file_permission+0xc/0x10 Aug 4 11:35:21 rhel6-node2 kernel: [] ? rw_verify_area+0x66/0xe0 Aug 4 11:35:21 rhel6-node2 kernel: [] ? do_sync_write+0x0/0x120 Aug 4 11:35:21 rhel6-node2 kernel: [] ? vfs_write+0xa0/0x190 Aug 4 11:35:21 rhel6-node2 kernel: [] ? sys_write+0x41/0x70 Aug 4 11:35:21 rhel6-node2 kernel: [] ? syscall_call+0x7/0xb Aug 4 11:37:21 rhel6-node2 kernel: INFO: task flush-252:16:882 blocked for more than 120 seconds. Aug 4 11:37:21 rhel6-node2 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Aug 4 11:37:21 rhel6-node2 kernel: flush-252:16 D dc524ac0 0 882 2 0x00000000 Aug 4 11:37:21 rhel6-node2 kernel: df8ffa90 00000046 95e2beb5 dc524ac0 d3399cf8 00000001 00000001 c0542788 Aug 4 11:37:21 rhel6-node2 kernel: d3399cf8 dfacfe40 00000056 c0ac7080 05b026cf 00000056 c0ac7080 df8ffd38 Aug 4 11:37:21 rhel6-node2 kernel: c0ac7080 c0ac2b34 c0ac7080 df8ffd38 00000001 e1142ed0 0001133e dd8f94b4 Aug 4 11:37:21 rhel6-node2 kernel: Call Trace: Aug 4 11:37:21 rhel6-node2 kernel: [] ? submit_bh+0xe8/0x120 Aug 4 11:37:21 rhel6-node2 kernel: [] ? gfs2_get_block_noalloc+0x0/0x40 [gfs2] Aug 4 11:37:21 rhel6-node2 kernel: [] ? gfs2_glock_holder_wait+0x5/0x10 [gfs2] Aug 4 11:37:21 rhel6-node2 kernel: [] ? __wait_on_bit+0x45/0x70 Aug 4 11:37:21 rhel6-node2 kernel: [] ? gfs2_glock_holder_wait+0x0/0x10 [gfs2] Aug 4 11:37:21 rhel6-node2 kernel: [] ? gfs2_glock_holder_wait+0x0/0x10 [gfs2] Aug 4 11:37:21 rhel6-node2 kernel: [] ? out_of_line_wait_on_bit+0x68/0x80 Aug 4 11:37:21 rhel6-node2 kernel: [] ? wake_bit_function+0x0/0x60 Aug 4 11:37:21 rhel6-node2 kernel: [] ? gfs2_glock_wait+0x2a/0x40 [gfs2] Aug 4 11:37:21 rhel6-node2 kernel: [] ? gfs2_write_inode+0x72/0x180 [gfs2] Aug 4 11:37:21 rhel6-node2 kernel: [] ? gfs2_write_inode+0x69/0x180 [gfs2] Aug 4 11:37:21 rhel6-node2 kernel: [] ? writeback_single_inode+0x1c8/0x270 Aug 4 11:37:21 rhel6-node2 kernel: [] ? writeback_sb_inodes+0xc1/0x180 Aug 4 11:37:21 rhel6-node2 kernel: [] ? writeback_inodes_wb+0x83/0x150 Aug 4 11:37:21 rhel6-node2 kernel: [] ? wb_writeback+0x1e3/0x230 Aug 4 11:37:21 rhel6-node2 kernel: [] ? wb_do_writeback+0x7e/0x190 Aug 4 11:37:21 rhel6-node2 kernel: [] ? bdi_writeback_task+0x2a/0xa0 Aug 4 11:37:21 rhel6-node2 kernel: [] ? bdi_start_fn+0x67/0xc0 Aug 4 11:37:21 rhel6-node2 kernel: [] ? bdi_start_fn+0x0/0xc0 Aug 4 11:37:21 rhel6-node2 kernel: [] ? kthread+0x74/0x80 Aug 4 11:37:21 rhel6-node2 kernel: [] ? kthread+0x0/0x80 Aug 4 11:37:21 rhel6-node2 kernel: [] ? kernel_thread_helper+0x7/0x10 Aug 4 11:37:21 rhel6-node2 kernel: INFO: task gfs2_quotad:1740 blocked for more than 120 seconds. Aug 4 11:37:21 rhel6-node2 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Aug 4 11:37:21 rhel6-node2 kernel: gfs2_quotad D 00000000 0 1740 2 0x00000080 Aug 4 11:37:21 rhel6-node2 kernel: ddb2fa90 00000046 dc70a1c4 00000000 de09add0 de661c00 dc50c000 00001586 Aug 4 11:37:21 rhel6-node2 kernel: 00000000 dc6d4580 00000055 c0ac7080 9b42a80a 00000055 c0ac7080 ddb2fd38 Aug 4 11:37:21 rhel6-node2 kernel: c0ac7080 c0ac2b34 c0ac7080 ddb2fd38 de2a4000 00000001 e113c1de 00000000 Aug 4 11:37:21 rhel6-node2 kernel: Call Trace: Aug 4 11:37:21 rhel6-node2 kernel: [] ? do_xmote+0x12e/0x200 [gfs2] Aug 4 11:37:21 rhel6-node2 kernel: [] ? gfs2_glock_holder_wait+0x5/0x10 [gfs2] Aug 4 11:37:21 rhel6-node2 kernel: [] ? __wait_on_bit+0x45/0x70 Aug 4 11:37:21 rhel6-node2 kernel: [] ? gfs2_glock_holder_wait+0x0/0x10 [gfs2] Aug 4 11:37:21 rhel6-node2 kernel: [] ? gfs2_glock_holder_wait+0x0/0x10 [gfs2] Aug 4 11:37:21 rhel6-node2 kernel: [] ? out_of_line_wait_on_bit+0x68/0x80 Aug 4 11:37:21 rhel6-node2 kernel: [] ? wake_bit_function+0x0/0x60 Aug 4 11:37:21 rhel6-node2 kernel: [] ? gfs2_glock_wait+0x2a/0x40 [gfs2] Aug 4 11:37:21 rhel6-node2 kernel: [] ? gfs2_statfs_sync+0x4c/0x1b0 [gfs2] Aug 4 11:37:21 rhel6-node2 kernel: [] ? gfs2_statfs_sync+0x45/0x1b0 [gfs2] Aug 4 11:37:21 rhel6-node2 kernel: [] ? process_timeout+0x0/0x10 Aug 4 11:37:21 rhel6-node2 kernel: [] ? quotad_check_timeo+0x3e/0xa0 [gfs2] Aug 4 11:37:21 rhel6-node2 kernel: [] ? finish_wait+0x31/0x80 Aug 4 11:37:21 rhel6-node2 kernel: [] ? gfs2_quotad+0x1fc/0x240 [gfs2] Aug 4 11:37:21 rhel6-node2 kernel: [] ? autoremove_wake_function+0x0/0x40 Aug 4 11:37:21 rhel6-node2 kernel: [] ? gfs2_quotad+0x0/0x240 [gfs2] Aug 4 11:37:21 rhel6-node2 kernel: [] ? kthread+0x74/0x80 Aug 4 11:37:21 rhel6-node2 kernel: [] ? kthread+0x0/0x80 Aug 4 11:37:21 rhel6-node2 kernel: [] ? kernel_thread_helper+0x7/0x10 Aug 4 11:37:21 rhel6-node2 kernel: INFO: task git:1746 blocked for more than 120 seconds. Aug 4 11:37:21 rhel6-node2 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Aug 4 11:37:21 rhel6-node2 kernel: git D c16090dc 0 1746 1409 0x00000080 Aug 4 11:37:21 rhel6-node2 kernel: de11b030 00000082 00000055 c16090dc ddd6058c c16090c0 ddd6058c 00010fd5 Aug 4 11:37:21 rhel6-node2 kernel: 00000000 dc6d4580 00000056 c0ac7080 055c66d9 00000056 c0ac7080 de11b2d8 Aug 4 11:37:21 rhel6-node2 kernel: c0ac7080 c0ac2b34 c0ac7080 de11b2d8 dd8c2000 00000001 e113c1de 00000001 Aug 4 11:37:21 rhel6-node2 kernel: Call Trace: Aug 4 11:37:21 rhel6-node2 kernel: [] ? do_xmote+0x12e/0x200 [gfs2] Aug 4 11:37:21 rhel6-node2 kernel: [] ? gfs2_glock_holder_wait+0x5/0x10 [gfs2] Aug 4 11:37:21 rhel6-node2 kernel: [] ? __wait_on_bit+0x45/0x70 Aug 4 11:37:21 rhel6-node2 kernel: [] ? gfs2_glock_holder_wait+0x0/0x10 [gfs2] Aug 4 11:37:21 rhel6-node2 kernel: [] ? gfs2_glock_holder_wait+0x0/0x10 [gfs2] Aug 4 11:37:21 rhel6-node2 kernel: [] ? out_of_line_wait_on_bit+0x68/0x80 Aug 4 11:37:21 rhel6-node2 kernel: [] ? wake_bit_function+0x0/0x60 Aug 4 11:37:21 rhel6-node2 kernel: [] ? gfs2_glock_wait+0x2a/0x40 [gfs2] Aug 4 11:37:21 rhel6-node2 kernel: [] ? gfs2_inplace_reserve_i+0x400/0x710 [gfs2] Aug 4 11:37:21 rhel6-node2 kernel: [] ? kmem_cache_alloc_notrace+0xa0/0xb0 Aug 4 11:37:21 rhel6-node2 kernel: [] ? gfs2_alloc_get+0x3a/0xb0 [gfs2] Aug 4 11:37:21 rhel6-node2 kernel: [] ? gfs2_write_begin+0x2f5/0x4b0 [gfs2] Aug 4 11:37:21 rhel6-node2 kernel: [] ? generic_file_buffered_write+0xef/0x270 Aug 4 11:37:21 rhel6-node2 kernel: [] ? __generic_file_aio_write+0x200/0x4e0 Aug 4 11:37:21 rhel6-node2 kernel: [] ? file_read_actor+0x7f/0xf0 Aug 4 11:37:21 rhel6-node2 kernel: [] ? generic_file_aio_write+0x5e/0xc0 Aug 4 11:37:21 rhel6-node2 kernel: [] ? gfs2_file_aio_write+0x6f/0xa0 [gfs2] Aug 4 11:37:21 rhel6-node2 kernel: [] ? do_sync_write+0xd5/0x120 Aug 4 11:37:21 rhel6-node2 kernel: [] ? autoremove_wake_function+0x0/0x40 Aug 4 11:37:21 rhel6-node2 kernel: [] ? schedule+0x42f/0xb00 Aug 4 11:37:21 rhel6-node2 kernel: [] ? security_file_permission+0xc/0x10 Aug 4 11:37:21 rhel6-node2 kernel: [] ? rw_verify_area+0x66/0xe0 Aug 4 11:37:21 rhel6-node2 kernel: [] ? do_sync_write+0x0/0x120 Aug 4 11:37:21 rhel6-node2 kernel: [] ? vfs_write+0xa0/0x190 Aug 4 11:37:21 rhel6-node2 kernel: [] ? sys_write+0x41/0x70 Aug 4 11:37:21 rhel6-node2 kernel: [] ? syscall_call+0x7/0xb Aug 4 11:39:21 rhel6-node2 kernel: INFO: task flush-252:16:882 blocked for more than 120 seconds. Aug 4 11:39:21 rhel6-node2 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Aug 4 11:39:21 rhel6-node2 kernel: flush-252:16 D dc524ac0 0 882 2 0x00000000 Aug 4 11:39:21 rhel6-node2 kernel: df8ffa90 00000046 95e2beb5 dc524ac0 d3399cf8 00000001 00000001 c0542788 Aug 4 11:39:21 rhel6-node2 kernel: d3399cf8 dfacfe40 00000056 c0ac7080 05b026cf 00000056 c0ac7080 df8ffd38 Aug 4 11:39:21 rhel6-node2 kernel: c0ac7080 c0ac2b34 c0ac7080 df8ffd38 00000001 e1142ed0 0001133e dd8f94b4 Aug 4 11:39:21 rhel6-node2 kernel: Call Trace: Aug 4 11:39:21 rhel6-node2 kernel: [] ? submit_bh+0xe8/0x120 Aug 4 11:39:21 rhel6-node2 kernel: [] ? gfs2_get_block_noalloc+0x0/0x40 [gfs2] Aug 4 11:39:21 rhel6-node2 kernel: [] ? gfs2_glock_holder_wait+0x5/0x10 [gfs2] Aug 4 11:39:21 rhel6-node2 kernel: [] ? __wait_on_bit+0x45/0x70 Aug 4 11:39:21 rhel6-node2 kernel: [] ? gfs2_glock_holder_wait+0x0/0x10 [gfs2] Aug 4 11:39:21 rhel6-node2 kernel: [] ? gfs2_glock_holder_wait+0x0/0x10 [gfs2] Aug 4 11:39:21 rhel6-node2 kernel: [] ? out_of_line_wait_on_bit+0x68/0x80 Aug 4 11:39:21 rhel6-node2 kernel: [] ? wake_bit_function+0x0/0x60 Aug 4 11:39:21 rhel6-node2 kernel: [] ? gfs2_glock_wait+0x2a/0x40 [gfs2] Aug 4 11:39:21 rhel6-node2 kernel: [] ? gfs2_write_inode+0x72/0x180 [gfs2] Aug 4 11:39:21 rhel6-node2 kernel: [] ? gfs2_write_inode+0x69/0x180 [gfs2] Aug 4 11:39:21 rhel6-node2 kernel: [] ? writeback_single_inode+0x1c8/0x270 Aug 4 11:39:21 rhel6-node2 kernel: [] ? writeback_sb_inodes+0xc1/0x180 Aug 4 11:39:21 rhel6-node2 kernel: [] ? writeback_inodes_wb+0x83/0x150 Aug 4 11:39:21 rhel6-node2 kernel: [] ? wb_writeback+0x1e3/0x230 Aug 4 11:39:21 rhel6-node2 kernel: [] ? wb_do_writeback+0x7e/0x190 Aug 4 11:39:21 rhel6-node2 kernel: [] ? bdi_writeback_task+0x2a/0xa0 Aug 4 11:39:21 rhel6-node2 kernel: [] ? bdi_start_fn+0x67/0xc0 Aug 4 11:39:21 rhel6-node2 kernel: [] ? bdi_start_fn+0x0/0xc0 Aug 4 11:39:21 rhel6-node2 kernel: [] ? kthread+0x74/0x80 Aug 4 11:39:21 rhel6-node2 kernel: [] ? kthread+0x0/0x80 Aug 4 11:39:21 rhel6-node2 kernel: [] ? kernel_thread_helper+0x7/0x10 Aug 4 11:39:21 rhel6-node2 kernel: INFO: task gfs2_quotad:1740 blocked for more than 120 seconds. Aug 4 11:39:21 rhel6-node2 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Aug 4 11:39:21 rhel6-node2 kernel: gfs2_quotad D 00000000 0 1740 2 0x00000080 Aug 4 11:39:21 rhel6-node2 kernel: ddb2fa90 00000046 dc70a1c4 00000000 de09add0 de661c00 dc50c000 00001586 Aug 4 11:39:21 rhel6-node2 kernel: 00000000 dc6d4580 00000055 c0ac7080 9b42a80a 00000055 c0ac7080 ddb2fd38 Aug 4 11:39:21 rhel6-node2 kernel: c0ac7080 c0ac2b34 c0ac7080 ddb2fd38 de2a4000 00000001 e113c1de 00000000 Aug 4 11:39:21 rhel6-node2 kernel: Call Trace: Aug 4 11:39:21 rhel6-node2 kernel: [] ? do_xmote+0x12e/0x200 [gfs2] Aug 4 11:39:21 rhel6-node2 kernel: [] ? gfs2_glock_holder_wait+0x5/0x10 [gfs2] Aug 4 11:39:21 rhel6-node2 kernel: [] ? __wait_on_bit+0x45/0x70 Aug 4 11:39:21 rhel6-node2 kernel: [] ? gfs2_glock_holder_wait+0x0/0x10 [gfs2] Aug 4 11:39:21 rhel6-node2 kernel: [] ? gfs2_glock_holder_wait+0x0/0x10 [gfs2] Aug 4 11:39:21 rhel6-node2 kernel: [] ? out_of_line_wait_on_bit+0x68/0x80 Aug 4 11:39:21 rhel6-node2 kernel: [] ? wake_bit_function+0x0/0x60 Aug 4 11:39:21 rhel6-node2 kernel: [] ? gfs2_glock_wait+0x2a/0x40 [gfs2] Aug 4 11:39:21 rhel6-node2 kernel: [] ? gfs2_statfs_sync+0x4c/0x1b0 [gfs2] Aug 4 11:39:21 rhel6-node2 kernel: [] ? gfs2_statfs_sync+0x45/0x1b0 [gfs2] Aug 4 11:39:21 rhel6-node2 kernel: [] ? process_timeout+0x0/0x10 Aug 4 11:39:21 rhel6-node2 kernel: [] ? quotad_check_timeo+0x3e/0xa0 [gfs2] Aug 4 11:39:21 rhel6-node2 kernel: [] ? finish_wait+0x31/0x80 Aug 4 11:39:21 rhel6-node2 kernel: [] ? gfs2_quotad+0x1fc/0x240 [gfs2] Aug 4 11:39:21 rhel6-node2 kernel: [] ? autoremove_wake_function+0x0/0x40 Aug 4 11:39:21 rhel6-node2 kernel: [] ? gfs2_quotad+0x0/0x240 [gfs2] Aug 4 11:39:21 rhel6-node2 kernel: [] ? kthread+0x74/0x80 Aug 4 11:39:21 rhel6-node2 kernel: [] ? kthread+0x0/0x80 Aug 4 11:39:21 rhel6-node2 kernel: [] ? kernel_thread_helper+0x7/0x10 Aug 4 11:39:21 rhel6-node2 kernel: INFO: task git:1746 blocked for more than 120 seconds. Aug 4 11:39:21 rhel6-node2 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Aug 4 11:39:21 rhel6-node2 kernel: git D c16090dc 0 1746 1409 0x00000080 Aug 4 11:39:21 rhel6-node2 kernel: de11b030 00000082 00000055 c16090dc ddd6058c c16090c0 ddd6058c 00010fd5 Aug 4 11:39:21 rhel6-node2 kernel: 00000000 dc6d4580 00000056 c0ac7080 055c66d9 00000056 c0ac7080 de11b2d8 Aug 4 11:39:21 rhel6-node2 kernel: c0ac7080 c0ac2b34 c0ac7080 de11b2d8 dd8c2000 00000001 e113c1de 00000001 Aug 4 11:39:21 rhel6-node2 kernel: Call Trace: Aug 4 11:39:21 rhel6-node2 kernel: [] ? do_xmote+0x12e/0x200 [gfs2] Aug 4 11:39:21 rhel6-node2 kernel: [] ? gfs2_glock_holder_wait+0x5/0x10 [gfs2] Aug 4 11:39:21 rhel6-node2 kernel: [] ? __wait_on_bit+0x45/0x70 Aug 4 11:39:21 rhel6-node2 kernel: [] ? gfs2_glock_holder_wait+0x0/0x10 [gfs2] Aug 4 11:39:21 rhel6-node2 kernel: [] ? gfs2_glock_holder_wait+0x0/0x10 [gfs2] Aug 4 11:39:21 rhel6-node2 kernel: [] ? out_of_line_wait_on_bit+0x68/0x80 Aug 4 11:39:21 rhel6-node2 kernel: [] ? wake_bit_function+0x0/0x60 Aug 4 11:39:21 rhel6-node2 kernel: [] ? gfs2_glock_wait+0x2a/0x40 [gfs2] Aug 4 11:39:21 rhel6-node2 kernel: [] ? gfs2_inplace_reserve_i+0x400/0x710 [gfs2] Aug 4 11:39:21 rhel6-node2 kernel: [] ? kmem_cache_alloc_notrace+0xa0/0xb0 Aug 4 11:39:21 rhel6-node2 kernel: [] ? gfs2_alloc_get+0x3a/0xb0 [gfs2] Aug 4 11:39:21 rhel6-node2 kernel: [] ? gfs2_write_begin+0x2f5/0x4b0 [gfs2] Aug 4 11:39:21 rhel6-node2 kernel: [] ? generic_file_buffered_write+0xef/0x270 Aug 4 11:39:21 rhel6-node2 kernel: [] ? __generic_file_aio_write+0x200/0x4e0 Aug 4 11:39:21 rhel6-node2 kernel: [] ? file_read_actor+0x7f/0xf0 Aug 4 11:39:21 rhel6-node2 kernel: [] ? generic_file_aio_write+0x5e/0xc0 Aug 4 11:39:21 rhel6-node2 kernel: [] ? gfs2_file_aio_write+0x6f/0xa0 [gfs2] Aug 4 11:39:21 rhel6-node2 kernel: [] ? do_sync_write+0xd5/0x120 Aug 4 11:39:21 rhel6-node2 kernel: [] ? autoremove_wake_function+0x0/0x40 Aug 4 11:39:21 rhel6-node2 kernel: [] ? schedule+0x42f/0xb00 Aug 4 11:39:21 rhel6-node2 kernel: [] ? security_file_permission+0xc/0x10 Aug 4 11:39:21 rhel6-node2 kernel: [] ? rw_verify_area+0x66/0xe0 Aug 4 11:39:21 rhel6-node2 kernel: [] ? do_sync_write+0x0/0x120 Aug 4 11:39:21 rhel6-node2 kernel: [] ? vfs_write+0xa0/0x190 Aug 4 11:39:21 rhel6-node2 kernel: [] ? sys_write+0x41/0x70 Aug 4 11:39:21 rhel6-node2 kernel: [] ? syscall_call+0x7/0xb Aug 4 11:41:21 rhel6-node2 kernel: INFO: task flush-252:16:882 blocked for more than 120 seconds. Aug 4 11:41:21 rhel6-node2 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Aug 4 11:41:21 rhel6-node2 kernel: flush-252:16 D dc524ac0 0 882 2 0x00000000 Aug 4 11:41:21 rhel6-node2 kernel: df8ffa90 00000046 95e2beb5 dc524ac0 d3399cf8 00000001 00000001 c0542788 Aug 4 11:41:21 rhel6-node2 kernel: d3399cf8 dfacfe40 00000056 c0ac7080 05b026cf 00000056 c0ac7080 df8ffd38 Aug 4 11:41:21 rhel6-node2 kernel: c0ac7080 c0ac2b34 c0ac7080 df8ffd38 00000001 e1142ed0 0001133e dd8f94b4 Aug 4 11:41:21 rhel6-node2 kernel: Call Trace: Aug 4 11:41:21 rhel6-node2 kernel: [] ? submit_bh+0xe8/0x120 Aug 4 11:41:21 rhel6-node2 kernel: [] ? gfs2_get_block_noalloc+0x0/0x40 [gfs2] Aug 4 11:41:21 rhel6-node2 kernel: [] ? gfs2_glock_holder_wait+0x5/0x10 [gfs2] Aug 4 11:41:21 rhel6-node2 kernel: [] ? __wait_on_bit+0x45/0x70 Aug 4 11:41:21 rhel6-node2 kernel: [] ? gfs2_glock_holder_wait+0x0/0x10 [gfs2] Aug 4 11:41:21 rhel6-node2 kernel: [] ? gfs2_glock_holder_wait+0x0/0x10 [gfs2] Aug 4 11:41:21 rhel6-node2 kernel: [] ? out_of_line_wait_on_bit+0x68/0x80 Aug 4 11:41:21 rhel6-node2 kernel: [] ? wake_bit_function+0x0/0x60 Aug 4 11:41:21 rhel6-node2 kernel: [] ? gfs2_glock_wait+0x2a/0x40 [gfs2] Aug 4 11:41:21 rhel6-node2 kernel: [] ? gfs2_write_inode+0x72/0x180 [gfs2] Aug 4 11:41:21 rhel6-node2 kernel: [] ? gfs2_write_inode+0x69/0x180 [gfs2] Aug 4 11:41:21 rhel6-node2 kernel: [] ? writeback_single_inode+0x1c8/0x270 Aug 4 11:41:21 rhel6-node2 kernel: [] ? writeback_sb_inodes+0xc1/0x180 Aug 4 11:41:21 rhel6-node2 kernel: [] ? writeback_inodes_wb+0x83/0x150 Aug 4 11:41:21 rhel6-node2 kernel: [] ? wb_writeback+0x1e3/0x230 Aug 4 11:41:21 rhel6-node2 kernel: [] ? wb_do_writeback+0x7e/0x190 Aug 4 11:41:21 rhel6-node2 kernel: [] ? bdi_writeback_task+0x2a/0xa0 Aug 4 11:41:21 rhel6-node2 kernel: [] ? bdi_start_fn+0x67/0xc0 Aug 4 11:41:21 rhel6-node2 kernel: [] ? bdi_start_fn+0x0/0xc0 Aug 4 11:41:21 rhel6-node2 kernel: [] ? kthread+0x74/0x80 Aug 4 11:41:21 rhel6-node2 kernel: [] ? kthread+0x0/0x80 Aug 4 11:41:21 rhel6-node2 kernel: [] ? kernel_thread_helper+0x7/0x10