Log Streaming / Pubnub compression


#1

Hi All,

We’re currently in the middle of looking at ways to get our logs streamed (via Logstash) into ELK. As far as I can tell from the resin-supervisor code, you’re using pubnub to stream logs 32kb at a time - I’ve managed to glean a lot of useful from this github issue:

A couple of questions:

  1. Are you currently using any compression on what’s being sent to pubnub? I’ve found some mention of being able to use gzipped streams: https://support.pubnub.com/support/discussions/topics/14000006317 - I’m not sure if this is available in their golang SDK, and I couldn’t see any sign in the supervisor source that the logs are being compressed.
  2. I’ve seen that there’s an easy way (through environment vars) to disable log streaming completely. Is there any way to use a different log streaming implementation? For example, I’d like to look at using filebeat (within a container on the device) to stream our logs into logstash. I know that this has the side-effect that we wouldn’t get logging in the resin dashboard, however given all the beautiful things we can do with the log data when it hits ELK, this isn’t such a big problem :wink:

I know that there are other ways to achieve this, for example, we could use the resin SDK to continuously stream logs from each device wherever we need them to go, I just feel like that adds an unnecessary layer of complexity (i.e. keeping track of which devices need to be streamed from, having a central instance which actually makes the api calls etc), which wouldn’t exist if we had some flexibility about where / when / how our logs are sent from the device itself.

Thanks for any thoughts on the topic!


#2

Hi @cmoss!
I didn’t know Pubnub had that compression support, we’ll take a look and see if we can add support for it. It looks like the JS SDK doesn’t support it (or it’s not documented), but I’ve seen it might not be that hard to just make an HTTP request to publish the logs. We’ll see.

As for using your own logging solution, I guess you can do it in your app like anything else, but you wouldn’t be seeing the supervisor messages (e.g. “Downloading application…”) - but I guess you could use the supervisor API from your app to get equivalent information. I’ll give it some more thought though, as we’re working on several changes in the supervisor and logging is one of them.


#3

Hi Pablo,

Exactly - this snippet here looks pretty easy to integrate:

https://support.pubnub.com/support/discussions/topics/14000006316

I can imagine that for some of the smaller devices the extra CPU hit on the compression wouldn’t be acceptable, but for those of us with CPU to burn, it would be a huge help (in our case, CPU is not an issue, but network traffic is expensive.)

Regarding rolling our own logging solution - I’d actually prefer not to, as I like the approach you’ve taken, if there was some way for us to get an aggregated log set across all devices for a given app I’d use that in a heart-beat, but as far as I know there’s not. By way of example, the data-API provides an interface to get device status for all devices attached to a given application:

https://api.resin.io/v1/application(ID)?$expand=device

If I could do something like the following I’d forget all about our own logging solution:

https://api.resin.io/v1/application(ID)?$expand=logs&$since=last_request

And of course, if I knew that the log stream is gzipped, I’d be even happier about it :wink:

Thanks again!

Corin


#4

(post withdrawn by author, will be automatically deleted in 24 hours unless flagged)


#5

Just in case anyone is looking for a way to push their docker logs to an external logstash instance, but not lose the normal resin log flow, this image is very helpful: