2018-04-10
(this is a repost of a medium article of which I am the author)
Hey, have you read this gorgeous article from Filippo Valsorda about how to write a Python extension in Go? You definitely should. When I first read it, I was searching for a solution to a problem affecting all the MacBooks in our open space. You have this very same problem if you develop AppEngine applications on MacOS X. You should notice it when you launch dev_appserver.py and your CPU cooler starts going crazy. At this point, you can feel a warm breeze flowing out of your laptop and your SSD starts to age faster than it should.
But what is the problem, exactly? Well, it resides in the mtime_file_watcher.py part of the Google Cloud SDK. This is the default file watcher for MacOS X. The file watcher is the component that watches your project source code and restarts a module to reload the modified code. On Linux, the default file watcher is the inotify_file_watcher.py so you won’t experience that warm breeze unless you don’t specify--use_mtime_file_watcher=yes as a dev_appserver.py option. If you are desperate, you can also disable the watcher entirely --automatic_restart=no but soon you will be even more desperate. Another way to mitigate this phenomenon is to fine tune the watched paths with --watcher_ignore_re=".*/app/client/.*" but honestly, when you have more than one module to run this will not produce the effects you would expect.
When I first started wandering the net searching for a solution, I found some alternatives. The most impressive article was from zeekay. In its implementation, he was using another fsevents module in Python. I have tried to use it, but it segfaulted on my MacOS. It’s C code, and I didn’t feel like rebuilding and and debugging it. I have to admit that I am more interested in Go than ol’ school C (best TIOBE language of 2017 and my favorite in 2002–2004, though).
Enter Filippo. His article sufficiently sparked my curiosity and was complete enough to get my extension running. He covers the structure of a Python extension written in Go, with the minimal C infrastructure to expose it to the Python interpreter. He shows how to parse two integers and how to return them to the caller.
My aim was to parse a list of paths and call a Python callback when an event was raised on one of the given paths. Returning the modified file and the action flags (Created, Modified, Deleted and so on). I wanted to do that in a parallel way (ie. call the callback immediately, once the event is raised. goroutines are great for that.)
Once the POC was complete, there was one more obstacle to avoid: packaging. That means getting rid of “it works on my machine!” and making it a *product*. Enter the Python scripts to backup, replace (and eventually restore) the mtime file watcher that comes with AppEngine.
On my machine, the main Python process started by the dev_appserver.py was consuming 156% CPU, constantly. In early January, I was about to ask my manager for a new MacBook. Today, the code in the https://github.com/nilleb/fsevents-watcher repository allows Python to consume only 0.3% CPU. And, of course, the AppEngine application modules are being reloaded when needed. What’s next I would really like to write some unit tests, and understand a little more about what makes the Python interpreter delivered with the OS binary incompatible with the home brew one (even if they share the same major.minor version numbers). Once this is done, I will then perhaps build a pip package and publish it on pypi.org.
If you update the Google Cloud components, you have to re-replace mtime_file_watcher.py with the scripts provided with the module.
This project was a fruit of the first LumApps Polish Week, held from January 22 to January 26. Special thanks to Lou for igniting and supporting this event.
What’s a polish week? That’s a week during which a part of LumApps takes a break from delivery deadlines, in order to improve the collective health. Everyone is free to suggest a problem and volunteer to solve it.