Lots of unexpected logging in Python library

[kamalaboulhosn]

Summarized from a customer issue:

Somehow, something with this google PubSub service is printing out Mutex.cc logs at an enormous scale of 3.5M / 5 minutes. The logs look as follows:

Error
2021-11-03 15:08:55.861 EDT[mutex.cc : 435] RAW: Lock blocking 0x7f0c66b9cb78 @
Error
2021-11-03 15:08:55.861 EDT[mutex.cc : 435] RAW: Lock returning 0x7f0c66b9cb78 @
Error
2021-11-03 15:08:55.861 EDT[mutex.cc : 435] RAW: Unlock 0x7f0c66b9cb78 @
Error
2021-11-03 15:08:55.861 EDT[mutex.cc : 435] RAW: Lock blocking 0x7f0c66b9cb78 @
Error
2021-11-03 15:08:55.861 EDT[mutex.cc : 435] RAW: Lock returning 0x7f0c66b9cb78 @
Error
2021-11-03 15:08:55.861 EDT[mutex.cc : 435] RAW: Unlock 0x7f0c66b9cb78 @
Error
2021-11-03 15:08:55.861 EDT[mutex.cc : 435] RAW: Lock blocking 0x7f0c66b9cb78 @
Error
2021-11-03 15:08:55.861 EDT[mutex.cc : 435] RAW: Lock returning 0x7f0c66b9cb78 @
Error
2021-11-03 15:08:55.861 EDT[mutex.cc : 435] RAW: Unlock 0x7f0c66b9cb78 @
Error
2021-11-03 15:08:55.861 EDT[mutex.cc : 435] RAW: Lock blocking 0x7f0c66b9cb78 @
Error
2021-11-03 15:08:55.861 EDT[mutex.cc : 435] RAW: Lock returning 0x7f0c66b9cb78 @
Error
2021-11-03 15:08:55.861 EDT[mutex.cc : 435] RAW: Unlock 0x7f0c66b9cb78 @
Error
2021-11-03 15:08:55.861 EDT[mutex.cc : 435] RAW: Lock blocking 0x7f0c66b9cb78 @
Error
2021-11-03 15:08:55.861 EDT[mutex.cc : 435] RAW: Lock returning 0x7f0c66b9cb78 @
Error
2021-11-03 15:08:55.861 EDT[mutex.cc : 435] RAW: Unlock 0x7f0c66b9cb78 @
Error
2021-11-03 15:08:55.861 EDT[mutex.cc : 435] RAW: Lock blocking 0x7f0c66b9cb78 @
Error
2021-11-03 15:08:55.861 EDT[mutex.cc : 435] RAW: Lock returning 0x7f0c66b9cb78 @
Error
2021-11-03 15:08:55.861 EDT[mutex.cc : 435] RAW: Unlock 0x7f0c66b9cb78 @
Error
2021-11-03 15:08:55.861 EDT[mutex.cc : 435] RAW: Lock blocking 0x7f0c66b9cb78 @
Error
2021-11-03 15:08:55.861 EDT[mutex.cc : 435] RAW: Lock returning 0x7f0c66b9cb78 @

The information provided about version:

1. We are using version 2.3.0 of google-cloud-pubsub
2. The spike in logs was not tied to an upgrade of pubsub or another library, we actually haven't touched the code in months

The customer suggested "I believe the core of the issue is that pubsub is sending messages without a delay and no exponential backoff, thus causing massive spikes in utilization and an over-use of the logging module."

[kamalaboulhosn]

@plamut would you be able to take a look into this while @pradn is out? Thanks!

[plamut]

@kamalaboulhosn This appears to be happening at the level below Python. Can the customer post some more details about their OS, the Python version, the output of $ pip freeze, and any other environment info that might be relevant? Is the issue consistently reproducible/observable?

The output doesn't seem like the logging we do in our Pub/Sub client code (at the Python level), and it probably doesn't come from the interpreter itself either, which is written in C. Seems like some C++ code, maybe `grpc?

For a start, I would at least try upgrading Pub/Sub to the latest version (2.9.0) and see if it solves the problem.

[kamalaboulhosn]

@plamut I will ask about the additional information. I agree that it doesn't look to be something at the Pub/Sub library level. My suspicion is that debug logging is turned on the gRPC level somehow.

[plamut]

@lidizheng A question for you guys, do these logs resemble grpc logs, do they look familiar? And if yes, is it a known thing with a quick fix?

Thanks!

[lidizheng]

@plamut gRPC today uses Abseil C++ for their mutex implementations. There is no mutex.cc inside gRPC code base. If gRPC trace log is turned on, it should be pretty obvious, the log will talk about polling strategy and plugins.

[plamut]

@lidizheng Thanks.

@kamalaboulhosn had a look into Abseil, found the mutex.cc file and the very log messages discussed here. The most probable theory seems to be that the customer somehow enabled the debug logging level, resulting in the flood that was reported.

[kamalaboulhosn]

They say they have no adjusted log levels in their environment.

They use these Python libraries:

• pymysql==0.8.1
• google-cloud-pubsub==2.3.0
• sqlalchemy==1.3.18
• pdf2image==1.14.0

And these Alpine Linux libraries:

• gcc
• libc-dev
• libffi-dev
• mariadb-dev
• libstdc++
• openssh,jpeg-dev
• zlib-dev
• poppler
• poppler-utils

[pradn]

Another customer is seeing the same issue.

The logs are coming from here.

I asked the ABSL team for info on how the raw log verbosity levels are set. abseil/abseil-cpp#1069

[pradn]

I pinged derekmauro who commented on abseil/abseil-cpp#1069 via internal chat. Hoping he can help us further with this.