Graceful termination

[ltagliamonte-dd]

Is there a way to gracefully shutdown cloudprober and propagate the signal to all external probes as well?
Briefly looking at the code i can see only enable/disable cpu and mem profiling.

[manugarg]

@ltagliamonte-dd No, currently we don't have a way to do graceful termination. We currently rely on the fact that cloudprober and child external probe processes will get killed together if cloudprober's process group is killed. Here is a trick that cloudprober_startup.sh uses to convert process's signals into process group signals:

cloudprober/tools/cloudprober_startup.sh

Line 51 in f668a7f

trap "trap - SIGTERM && kill -- -$$" SIGINT SIGTERM EXIT

There are certain mechanisms in place in cloudprober code, but the whole thing is not connected. For example, cloudprober probe's Start method takes a context, which can actually be used to clean things properly, but we don't trigger cancelation of that context through a process signal.

We can look at implementing this.

[ltagliamonte-dd]

I would love you pass along the signal to all running probes.
I wrote a kubernetes probe that creates and deleted k8s resources and when I deploy a new version I need to make sure that I cleanup before I exit.
Currently I made sure of that at container bootstrap but isn't the most elegant solution XD

[manugarg]

I have a change under review for this.

[manugarg]

@ltagliamonte-dd Code has been changed. I verified that killing cloudprober kills the external probe processes started by cloudprober. One thing to note though, only processes directly started by cloudprober get killed. It doesn't kill processes started by the external probe processes themselves (for example, shell script forking off other processes). There is really no good way to do that[1].

I am not sure if you want to try with the latest docker image or want to wait for the next release v0.10.8 (in about 2 weeks). I've filed a separate bug to release v0.10.8: #425. I am closing this bug now. Please feel free to re-open if you encounter an issue.

[1]: One thing that cloudprober's Go serverutils do is that they kill the external process when stdin gets closed.

[ltagliamonte-dd]

This is wonderful @manugarg will try the next stable release!
Thanks for tackling this in such short period