OpenFlow rules missing for service/node after certain period of depoloyment

[cloudrarch]

Description

I have deployed a customized k8s cluster and on top of that installed Contiv as CNI networking.
All worked fine, but after 2 or 3 days noticed that OpenFlow rules for a specific service/node missing.

Expected Behavior

All the PODs should be accessible from any of the host systems. ie host ---> pod communication

Observed Behavior

Unable to connect to the POD and contivh1 interface IP from other hosts.Due to this unable to connect to the k8s services hosted on a specific host.

After some investigation, I found that the OpenFlow rules for that particular POD IP and contivh1 interface IP are missing.

The is sometime happening for POD IP and sometimes for contivh1 interface IP.

[root@]# kubectl get pod -n kube-system |grep contiv
contiv-netmaster-bj2nc 1/1 Running 0 6d
contiv-netmaster-kqkgq 1/1 Running 0 6d
contiv-netplugin-6rrgc 1/1 Running 0 6d
contiv-netplugin-fdbtd 1/1 Running 0 6d
contiv-netplugin-j7qc9 1/1 Running 0 6d
contiv-netplugin-k4p6q 1/1 Running 0 6d
contiv-netplugin-kpnq9 1/1 Running 0 6d
contiv-netplugin-p6k59 1/1 Running 0 6d
contiv-netplugin-t6zbh 1/1 Running 0 6d
contiv-netplugin-zsd9g 1/1 Running 0 6d
[root@]#

[root@]# kubectl exec contiv-netplugin-6rrgc -n kube-system -- ovs-ofctl -O Openflow13 dump-flows contivVxlanBridge | grep 132.1.1.8
[root@s ~]#

[root@~]# kubectl exec contiv-netplugin-fdbtd -n kube-system -- ovs-ofctl -O Openflow13 dump-flows contivVxlanBridge | grep 132.1.1.8
cookie=0x1dd37, duration=9532.039s, table=4, n_packets=0, n_bytes=0, priority=100,ip,metadata=0x100000000/0xff00000000,nw_dst=132.1.1.8 actions=write_metadata:0/0xfffe,goto_table:5
cookie=0x1dd36, duration=9532.039s, table=7, n_packets=0, n_bytes=0, priority=100,ip,metadata=0x100000000/0xff00000000,nw_dst=132.1.1.8 actions=set_field:0x1->tun_id,set_field:00:00:11:11:11:11->eth_dst,output:267
[root@ ~]#

[root@ ~]# kubectl exec contiv-netplugin-j7qc9 -n kube-system -- ovs-ofctl -O Openflow13 dump-flows contivVxlanBridge | grep 132.1.1.8
cookie=0x499, duration=340671.602s, table=4, n_packets=86, n_bytes=8428, priority=100,ip,metadata=0x100000000/0xff00000000,nw_dst=132.1.1.8 actions=write_metadata:0/0xfffe,goto_table:5
cookie=0x498, duration=340671.603s, table=7, n_packets=86, n_bytes=8428, priority=100,ip,metadata=0x100000000/0xff00000000,nw_dst=132.1.1.8 actions=set_field:0x1->tun_id,set_field:00:00:11:11:11:11->eth_dst,output:12
[root@ ~]#

[root@ ~]# kubectl exec contiv-netplugin-k4p6q -n kube-system -- ovs-ofctl -O Openflow13 dump-flows contivVxlanBridge | grep 132.1.1.8
[root@ ~]#

[root@ ~]# kubectl exec contiv-netplugin-kpnq9 -n kube-system -- ovs-ofctl -O Openflow13 dump-flows contivVxlanBridge | grep 132.1.1.8
cookie=0x1f, duration=604562.586s, table=4, n_packets=329709, n_bytes=31594881, priority=100,ip,metadata=0x100000000/0xff00000000,nw_dst=132.1.1.8 actions=write_metadata:0/0xfffe,goto_table:5
cookie=0x1e, duration=604562.586s, table=7, n_packets=391455, n_bytes=49623858, priority=100,ip,metadata=0x100000000/0xff00000000,nw_dst=132.1.1.8 actions=set_field:00:00:11:11:11:11->eth_src,set_field:02:02:84:01:01:08->eth_dst,output:10
[root ~]#

[root@s~]# kubectl exec contiv-netplugin-p6k59 -n kube-system -- ovs-ofctl -O Openflow13 dump-flows contivVxlanBridge | grep 132.1.1.8
[root@~]#

[root@ ~]# kubectl exec contiv-netplugin-t6zbh -n kube-system -- ovs-ofctl -O Openflow13 dump-flows contivVxlanBridge | grep 132.1.1.8
cookie=0x49f, duration=340798.618s, table=4, n_packets=3, n_bytes=294, priority=100,ip,metadata=0x100000000/0xff00000000,nw_dst=132.1.1.8 actions=write_metadata:0/0xfffe,goto_table:5
cookie=0x49e, duration=340798.618s, table=7, n_packets=3, n_bytes=294, priority=100,ip,metadata=0x100000000/0xff00000000,nw_dst=132.1.1.8 actions=set_field:0x1->tun_id,set_field:00:00:11:11:11:11->eth_dst,output:13
[root@ ~]#

[root@ ~]# kubectl exec contiv-netplugin-zsd9g -n kube-system -- ovs-ofctl -O Openflow13 dump-flows contivVxlanBridge | grep 132.1.1.8
[root@ ~]#

Steps to Reproduce (for bugs)

Your Environment

• netctl version

  Client Version:
  Version: 1.2.0
  GitCommit: f78851a
  BuildTime: 12-14-2017.07-32-25.UTC

  Server Version:
  Version: 1.2.0
  GitCommit: f78851a
  BuildTime: 12-14-2017.07-32-25.UTC

• Orchestrator version (e.g. kubernetes, mesos, swarm):
  Kubernetes -1.8.5

• Operating System and version
  CentOS-7.3