Cronjobber does not work with slim tzdata format

[scorsi]

Hello,

I have a two CronJob the one set to 0 1 * * * in Europe/Paris which is running at 3 CEST, the other one scheduled at 0 3 * * * in Europe/Paris ran at 5 CEST...

Do you have any idea why it's not running at the time it was scheduled ?

Thanks,

[u6f6o]

Hello,

Same issue here. We normally scale up our pods in the evening, yet they have been triggered in the morning today. I found the following text in the logs:

2020-10-23T08:43:58+0000 Local Time Zone database updated to version 2020c on /tmp/zoneinfo

Best,
u6f6o

[pscheid92]

Hello, @u6f6o and I have the exact same problem.

We have 4 different namespaces with 2 jobs each.
Of those jobs, 4 are scheduled for 19:50 PM and 4 are scheduled for 20:30 PM, each day.

Today, they all ran at exactly the same time (10:44 AM).
Maybe it is a coincidence, but the updatetz container logged the right before all jobs ran:

2020-10-23T08:43:58+0000 Local Time Zone database updated to version 2020c on /tmp/zoneinfo

Our local time zone is CEST (+0200), thus the database update was logged at 10:43:58 AM local time.

[scorsi]

May be related to #24. Actually don't have the workload to investigate deeper. Switched to default Kubernetes CronJob while the issue is not fixed/investigated.

[obitech]

Running into the same problem here. Now my TZCronJobs won't start at all, at least not during my specified time. I haven't been able to debug the tzdb inside the container, has there been an issue with the update?

[hiddeco]

Sorry about this unexpected behaviour, and I hope it did not result in too much chaos. Can you all please confirm you are using the latest cronjobber-updatetz release (v0.1.1)? This helps me narrow my search.

[pscheid92]

Yes, we (@u6f6o, @obitech and me) used the v0.1.1 release (with sidecar.enabled=true; installed per Helm chart).

[apicht]

Just noticed this is happening for us as well. Running cronjobber:0.2.0 and cronjobber-updatetz:0.1.0. Latest log message from updatetz is

2020-10-22T23:03:54+0000 Local Time Zone database updated to version 2020c on /tmp/zoneinfo

[maksimu]

Yeap, same for me. Hopefully it can be fixed soon

[maksimu]

When I look at the logs of the cronjobber I see this errors that keep repeating over and over again:

2020-10-25T07:09:06.312819787Z cronjobber E1025 07:09:06.312568       1 reflector.go:134] github.com/hiddeco/cronjobber/pkg/client/informers/externalversions/factory.go:117: Failed to list *v1alpha1.TZCronJob: tzcronjobs.cronjobber.hidde.co is forbidden: User "system:serviceaccount:bama-prod:cronjobber" cannot list resource "tzcronjobs" in API group "cronjobber.hidde.co" at the cluster scope
2020-10-25T07:09:07.182842095Z cronjobber E1025 07:09:07.182647       1 controller.go:122] can't list Jobs: jobs.batch is forbidden: User "system:serviceaccount:bama-prod:cronjobber" cannot list resource "jobs" in API group "batch" at the cluster scope
2020-10-25T07:09:07.316420688Z cronjobber E1025 07:09:07.316221       1 reflector.go:134] github.com/hiddeco/cronjobber/pkg/client/informers/externalversions/factory.go:117: Failed to list *v1alpha1.TZCronJob: tzcronjobs.cronjobber.hidde.co is forbidden: User "system:serviceaccount:bama-prod:cronjobber" cannot list resource "tzcronjobs" in API group "cronjobber.hidde.co" at the cluster scope

Maybe something is wrong w/ the service account? In fact my service account that was installed before was gone.

[obitech]

@maksimu Your errors should be unrelated. Your service account seems to be missing the correct ClusterRole permissions.

apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
  name: cronjobber
rules:
- apiGroups: ['cronjobber.hidde.co']
  resources: ['*']
  verbs: ['*']
- apiGroups: ['batch']
  resources: ['jobs']
  verbs: ['*']
- apiGroups: ['']
  resources: ['events']
  verbs: ['create', 'patch', 'update']

[hiddeco]

The problem is due to zic changing the default format from fat to slim since >=2020-b, this format is however not correctly understood by Go (golang/go#42138), resulting in an UTC fallback.

The best option here is likely to start using Go 1.15, and make use of the embedded timezone information there, which makes the timezone update utility obsolete.

[obitech]

@hiddeco do you find time to fix this? Otherwise I can draft a PR.

[hiddeco]

I have just crafted a release, once it has been pushed to the registry, using the https://raw.githubusercontent.com/hiddeco/cronjobber/master/deploy/deploy.yaml manifest with quay.io/hiddeco/cronjobber:0.3.0 should solve this issue with a workaround.

I will track the Go issue to release a new version once it is able to understand the slim format as well, so the updatetz utility can be used again. Keeping this issue open until then.

[obitech]

Works again with 0.3.0, thank you! 🙏

[hiddeco]

The bug fix for this problem has landed in the release branch for Go 1.15.4, with the release scheduled for next week. Expect a patch release as soon as this version is available in container / image land. 🌻

[rmak-cpi]

I am just wondering when cronjobber is going to be patched with the golang fix (starting from 1.15.4) for handling slim tzdata given that we are already past the 2021 DST transition. Thanks!

[hiddeco]

Hello all,

It is likely not what you hoped, but I do at present not have the bandwidth to further maintain this project. See #43 (comment) for more information.