Database not yet running. Waiting...

[deepak71]

Is this a BUG REPORT or FEATURE REQUEST?

Choose one: BUG REPORT

When we create a simple MySQL cluster in a namespace other than default one its failing to create a new MySQL cluster after creating the mysql-agent ServiceAccount and RoleBinding in that new namespace. But it successfully creating the MySQL cluster in default namespace.

Versions

MySQL Docker Image 8.0.12-1.1.7

MySQL Operator Version:
mysql-agent version 0.3.0

Environment:

• Kubernetes version (use kubectl version):
  Client Version: version.Info{Major:"1", Minor:"14", GitVersion:"v1.14.2", GitCommit:"66049e3b21efe110454d67df4fa62b08ea79a19b", GitTreeState:"clean", BuildDate:"2019-05-16T18:55:03Z", GoVersion:"go1.12.5", Compiler:"gc", Platform:"darwin/amd64"}
  Server Version: version.Info{Major:"1", Minor:"14", GitVersion:"v1.14.1", GitCommit:"b7394102d6ef778017f2ca4046abbaa23b88c290", GitTreeState:"clean", BuildDate:"2019-04-08T17:02:58Z", GoVersion:"go1.12.1", Compiler:"gc", Platform:"linux/amd64"}

• Cloud provider or hardware configuration:
  vSphere

-What happened?
serviceaccount/mysql-agent created
rolebinding.rbac.authorization.k8s.io/mysql-agent created
cluster.mysql.oracle.com/my-app-db created

NAME READY STATUS RESTARTS AGE
pod/my-app-db-0 0/2 CrashLoopBackOff 2 40s

What you expected to happen?

It supposed to create MySQL cluster.

How to reproduce it (as minimally and precisely as possible)?

Create a mysql cluster in non-default namespace.

Anything else we need to know?

kubectl -n my-cluster logs my-app-db-0 --all-containers

• base=1000
  ++ cat /etc/hostname
  ++ grep -o '[^-]*$'
• index=0
  ++ expr 1000 + 0
• /entrypoint.sh --server_id=1000 --datadir=/var/lib/mysql --user=mysql --gtid_mode=ON --log-bin --binlog_checksum=NONE --enforce_gtid_consistency=ON --log-slave-updates=ON --binlog-format=ROW --master-info-repository=TABLE --relay-log-info-repository=TABLE --transaction-write-set-extraction=XXHASH64 --relay-log=my-app-db-0-relay-bin --report-host=my-app-db-0.my-app-db --log-error-verbosity=3
  [Entrypoint] MySQL Docker Image 8.0.12-1.1.7
  sed: couldn't open temporary file /etc/sed5IaEqq: Permission denied
  Starting mysql-agent version 0.3.0
  I0904 17:40:25.632334 1 main.go:48] FLAG: --address="0.0.0.0"
  I0904 17:40:25.632383 1 main.go:48] FLAG: --alsologtostderr="false"
  I0904 17:40:25.632390 1 main.go:48] FLAG: --cluster-name="my-app-db"
  I0904 17:40:25.632395 1 main.go:48] FLAG: --healthcheck-port="10512"
  I0904 17:40:25.632402 1 main.go:48] FLAG: --hostname="my-app-db-0"
  I0904 17:40:25.632407 1 main.go:48] FLAG: --log-backtrace-at=":0"
  I0904 17:40:25.632414 1 main.go:48] FLAG: --log-dir=""
  I0904 17:40:25.632419 1 main.go:48] FLAG: --log-flush-frequency="5s"
  I0904 17:40:25.632425 1 main.go:48] FLAG: --logtostderr="true"
  I0904 17:40:25.632430 1 main.go:48] FLAG: --min-resync-period="12h0m0s"
  I0904 17:40:25.632436 1 main.go:48] FLAG: --namespace="my-cluster"
  I0904 17:40:25.632441 1 main.go:48] FLAG: --stderrthreshold="2"
  I0904 17:40:25.632445 1 main.go:48] FLAG: --v="4"
  I0904 17:40:25.632451 1 main.go:48] FLAG: --vmodule=""
  I0904 17:40:25.641414 1 cluster_manager.go:116] Database not yet running. Waiting...
  I0904 17:40:35.647823 1 cluster_manager.go:116] Database not yet running. Waiting...
  I0904 17:40:45.655278 1 cluster_manager.go:116] Database not yet running. Waiting...
  I0904 17:40:55.661899 1 cluster_manager.go:116] Database not yet running. Waiting...
  I0904 17:41:05.668283 1 cluster_manager.go:116] Database not yet running. Waiting...
  I0904 17:41:15.675703 1 cluster_manager.go:116] Database not yet running. Waiting...
  I0904 17:41:25.682909 1 cluster_manager.go:116] Database not yet running. Waiting...
  I0904 17:41:35.689685 1 cluster_manager.go:116] Database not yet running. Waiting...
  I0904 17:41:45.696172 1 cluster_manager.go:116] Database not yet running. Waiting...
  I0904 17:41:55.703842 1 cluster_manager.go:116] Database not yet running. Waiting...
  I0904 17:42:05.710808 1 cluster_manager.go:116] Database not yet running. Waiting...
  I0904 17:42:15.717695 1 cluster_manager.go:116] Database not yet running. Waiting...
  I0904 17:42:25.724160 1 cluster_manager.go:116] Database not yet running. Waiting...
  I0904 17:42:35.730750 1 cluster_manager.go:116] Database not yet running. Waiting...
  I0904 17:42:45.737124 1 cluster_manager.go:116] Database not yet running. Waiting...
  I0904 17:42:55.743524 1 cluster_manager.go:116] Database not yet running. Waiting...
  I0904 17:43:05.750738 1 cluster_manager.go:116] Database not yet running. Waiting...
  I0904 17:43:15.758064 1 cluster_manager.go:116] Database not yet running. Waiting...

[SebaKalina]

Same error :/ Dunno what to do ...?

[jwklugh]

I wish I had a solution to this problem, but what I have at the moment is a workaround (which completely defeats the purpose of how to replicate the bug).

I ran into the same error installing on an openshift cluster in the default namespace and using the global flag in the helm values file. oracle/mysql-operator seems to have a lot of trouble when working in a namespace that isn't mysql-operator. If you are able to work within that namespace, that's what I would recommend. It is what worked for me.

I don't know if anyone still working with this has experience with the .go files, but cluster_manager.go:116 is a line in a file in the git that has
if !isDatabaseRunning(ctx) {
glog.V(2).Infof("Database not yet running. Waiting...")
return false
}
at the top of a Sync function. I've been unable to find isDatabaseRunning() to see if it has the source of the namespace issue, or if it's somewhere further in the helm chart (such as almost every if statement in 02-rbac.yaml). I'll keep trying to search for a solution to it, but I don't know for how long.

[bweston92]

mysql-operator/pkg/controllers/cluster/manager/innodb_cluster.go

Line 42 in 9aebcc3

func isDatabaseRunning(ctx context.Context) bool {