Adaptive bitrate doesn't work

Hi @pedroSG94
I'm testing BitrateAdapter and I've implemented it the same way as it is in the example.
When I switch from high-speed WiFI to 3G the bitrate is not adapted properly and it is very high.
The same situation when I start streaming from 3G. The tested upload speed of 3G is ~1.5Mb/s, however, adapted bitrate is ~5Mb/s (which is my max bitrate) resulting in lags on stream.
Maybe I'm doing something wrong, but I've checked the source code and the bitrate that you pass to onNewBitrateRtmp(bitrate) callback may not be correct. It's taken from RtmpConnection.publishVideoData(size) just after you write the data to the socket. I think that socket has an internal buffer and the data size you write to socket's outputstream doesn't have to be equal to the size of data that has been uploaded.
I've compared this bitrate value to the value from TrafficStats.getTotalTxBytes(), which shows the amount of data that has been uploaded and there are discrepancies between them.
Here is the code to quickly compare it
private long bitrateSum = 0;
  private long initialTotalUploadBytes = 0;
  private long previousCheckTimeMs = 0L;
  private long lastTotalUploadBytes = 0;

  private BitrateAdapter adapter = new BitrateAdapter(new BitrateAdapter.Listener() {
    @Override
    public void onBitrateAdapted(int bitrate) {
      Log.d("lol2", "bitrate adapted: " + bitrate);
      rtmpCamera2.setVideoBitrateOnFly(bitrate);
    }
  });

  @Override
  public void onConnectionSuccessRtmp() {
    adapter.setMaxBitrate(5 * 1024 * 1024);
    initialTotalUploadBytes = TrafficStats.getTotalTxBytes();
  }

  @Override
  public void onNewBitrateRtmp(long bitrate) {
    Log.d("lol", "onNewBitrate: " + bitrate);
    adapter.adaptBitrate((int) bitrate);

    bitrateSum += bitrate;
    Log.d("lol", "Bitrate sum so far: " + bitrateSum / 1024f / 1024f + " Mb");

    long uploadedBytesSoFar = TrafficStats.getTotalTxBytes() - initialTotalUploadBytes;
    Log.d("lol", "Real upload so far: " + uploadedBytesSoFar * 8f / 1024f / 1024f + " Mb");

    long bytesDiff = uploadedBytesSoFar - lastTotalUploadBytes;
    long nowMs = System.currentTimeMillis();
    int timeDiff = (int) ((nowMs - previousCheckTimeMs) / 1000f);
    float realUploadSpeed = bytesDiff * 8f / timeDiff / 1024f / 1024f;
    Log.d("lol", "Real upload speed: " + realUploadSpeed + " Mb/s");

    previousCheckTimeMs = nowMs;
    lastTotalUploadBytes = uploadedBytesSoFar;
  }

There is also one interesting thing that I don't fully understand, maybe you could explain.
Being on high-speed WiFI, setting rtmpCamera2.setVideoBitrateOnFly(20 * 1024 * 1024);
determines the video bitrate but also the upload speed. I mean, when I set it to 20Mb/s like above the upload speed will be 20Mb/s, when I set it to 2Mb/s the upload speed will be 2Mb/s. So is there any contract in the RTMP that video quality affects upload speed? Can't I upload 2Mb/s quality video with 5Mb/s speed? Or maybe it depends on the internal buffer size?

I need test this code. I'm not sure if socket has a buffer but this is the only way to do it(That I know).
Keep in mind that your code also count traffic with others apps.
About second question. I don't fully understand but when you set video bitrate your stream upload traffic speed is basically video + audio bitrate. Of course, if you have 20Mb/s of internet connection you can do 20Mb/s stream or less. Resolution and upload speed is not related you can do stream to 4k but with 1Mb/s bitrate (obviously you will have a pixelated stream)

I was testing bitrate callback and work fine. I did it:
With default parameters 1200 * 1024 video bitrate + 64 * 1024 audio bitrate = 1294336 bps
You need keep moving device to get max bitrate because H264 algorithm reduce bitrate if you have a static image to reduce traffic.
If you set a logcat in callback you get this:
@Override
  public void onNewBitrateRtmp(long bitrate) {
    Log.e("Pedro", "Lib: " + bitrate);
  }

2020-04-20 15:46:30.128 26416-26573/com.pedro.rtpstreamer E/Pedro: Lib: 88
2020-04-20 15:46:31.129 26416-26573/com.pedro.rtpstreamer E/Pedro: Lib: 1379104
2020-04-20 15:46:32.133 26416-26573/com.pedro.rtpstreamer E/Pedro: Lib: 1469184
2020-04-20 15:46:33.155 26416-26573/com.pedro.rtpstreamer E/Pedro: Lib: 1175216
2020-04-20 15:46:34.174 26416-26573/com.pedro.rtpstreamer E/Pedro: Lib: 1332528
2020-04-20 15:46:35.174 26416-26573/com.pedro.rtpstreamer E/Pedro: Lib: 1335928
2020-04-20 15:46:36.174 26416-26573/com.pedro.rtpstreamer E/Pedro: Lib: 1422536
2020-04-20 15:46:37.176 26416-26573/com.pedro.rtpstreamer E/Pedro: Lib: 1443672
2020-04-20 15:46:38.203 26416-26573/com.pedro.rtpstreamer E/Pedro: Lib: 1208616
2020-04-20 15:46:39.229 26416-26573/com.pedro.rtpstreamer E/Pedro: Lib: 1199072
2020-04-20 15:46:40.232 26416-26573/com.pedro.rtpstreamer E/Pedro: Lib: 1125568
2020-04-20 15:46:41.232 26416-26573/com.pedro.rtpstreamer E/Pedro: Lib: 1328304
2020-04-20 15:46:42.232 26416-26573/com.pedro.rtpstreamer E/Pedro: Lib: 1368376
2020-04-20 15:46:43.233 26416-26573/com.pedro.rtpstreamer E/Pedro: Lib: 1208864
2020-04-20 15:46:44.235 26416-26573/com.pedro.rtpstreamer E/Pedro: Lib: 1245304
2020-04-20 15:46:45.243 26416-26573/com.pedro.rtpstreamer E/Pedro: Lib: 1383432

With TrafficStats this is the result:
  private long trafficBits = 0;

  @Override
  public void onNewBitrateRtmp(long bitrate) {
    trafficBits = TrafficStats.getTotalTxBytes() - trafficBits;
    //convert to bits
    Log.e("Pedro", "Traffic: " + (trafficBits * 8));
  }

2020-04-20 15:50:44.455 26763-26903/com.pedro.rtpstreamer E/Pedro: Traffic: 1380680
2020-04-20 15:50:45.457 26763-26903/com.pedro.rtpstreamer E/Pedro: Traffic:<PHONE_NUMBER>
2020-04-20 15:50:46.477 26763-26903/com.pedro.rtpstreamer E/Pedro: Traffic: 1988040
2020-04-20 15:50:47.479 26763-26903/com.pedro.rtpstreamer E/Pedro: Traffic:<PHONE_NUMBER>
2020-04-20 15:50:48.495 26763-26903/com.pedro.rtpstreamer E/Pedro: Traffic: 3799392
2020-04-20 15:50:49.507 26763-26903/com.pedro.rtpstreamer E/Pedro: Traffic:<PHONE_NUMBER>
2020-04-20 15:50:50.518 26763-26903/com.pedro.rtpstreamer E/Pedro: Traffic: 5320176
2020-04-20 15:50:51.534 26763-26903/com.pedro.rtpstreamer E/Pedro: Traffic:<PHONE_NUMBER>
2020-04-20 15:50:52.556 26763-26903/com.pedro.rtpstreamer E/Pedro: Traffic: 6626216
2020-04-20 15:50:53.561 26763-26903/com.pedro.rtpstreamer E/Pedro: Traffic:<PHONE_NUMBER>
2020-04-20 15:50:54.575 26763-26903/com.pedro.rtpstreamer E/Pedro: Traffic: 7976584
2020-04-20 15:50:55.592 26763-26903/com.pedro.rtpstreamer E/Pedro: Traffic:<PHONE_NUMBER>
2020-04-20 15:50:56.594 26763-26903/com.pedro.rtpstreamer E/Pedro: Traffic: 9295552
2020-04-20 15:50:57.611 26763-26903/com.pedro.rtpstreamer E/Pedro: Traffic:<PHONE_NUMBER>
2020-04-20 15:50:58.634 26763-26903/com.pedro.rtpstreamer E/Pedro: Traffic: 10569024
2020-04-20 15:50:59.664 26763-26903/com.pedro.rtpstreamer E/Pedro: Traffic:<PHONE_NUMBER>
2020-04-20 15:51:00.673 26763-26903/com.pedro.rtpstreamer E/Pedro: Traffic: 11868016

@pedroSG94 Thanks for checking this. Did you try it on slow Internet with upload speed like 1Mb/s? You would also have to set initial bitrate in prepareVideo() to say 3-5Mb/s, otherwise, you may not see the issue.
How do you assume from these logs that it is working fine? I mean these values are very close to each other (~1Mb/s). I guess you'd need to switch from faster Internet to slower and back to the good one in order to test it well. So for WiFi the bitrate should raise, up to max bitrate (5Mb/s) and once you switch it should decrease to ~1Mb/s.
You can use my code to test as the logs there shows well the discrepancies in a nice way. If you'd like to use 3G instead of LTE, there is an option in Network settings like "Preferred network type" where you can set 3G as the default network.
Regarding the socket buffer I've mentioned, it is being set in RtmpConnection like socket.setSendBufferSize(acknowledgementWindowsize); with the value of  5 million in my case.

10Mbs using 3G (upload speed around 4Mbs).
2020-04-22 12:16:15.965 18740-18890/com.pedro.rtpstreamer E/lol: onNewBitrate: 8.392334E-5 Mb/s
2020-04-22 12:16:15.968 18740-18890/com.pedro.rtpstreamer E/lol: Real upload speed: 2.1545145E-6 Mb/s
2020-04-22 12:16:16.973 18740-18890/com.pedro.rtpstreamer E/lol: onNewBitrate: 17.849495 Mb/s
2020-04-22 12:16:16.974 18740-18890/com.pedro.rtpstreamer E/lol: Real upload speed: 0.37243652 Mb/s
2020-04-22 12:16:17.980 18740-18890/com.pedro.rtpstreamer E/lol: onNewBitrate: 8.128471 Mb/s
2020-04-22 12:16:17.981 18740-18890/com.pedro.rtpstreamer E/lol: Real upload speed: 0.51049805 Mb/s
2020-04-22 12:16:19.001 18740-18890/com.pedro.rtpstreamer E/lol: onNewBitrate: 11.777885 Mb/s
2020-04-22 12:16:19.003 18740-18890/com.pedro.rtpstreamer E/lol: Real upload speed: 1.1706848 Mb/s
2020-04-22 12:16:20.028 18740-18890/com.pedro.rtpstreamer E/lol: onNewBitrate: 1.742897 Mb/s
2020-04-22 12:16:20.030 18740-18890/com.pedro.rtpstreamer E/lol: Real upload speed: 1.8318481 Mb/s
2020-04-22 12:16:21.054 18740-18890/com.pedro.rtpstreamer E/lol: onNewBitrate: 15.497047 Mb/s
2020-04-22 12:16:21.055 18740-18890/com.pedro.rtpstreamer E/lol: Real upload speed: 3.0480347 Mb/s
2020-04-22 12:16:22.057 18740-18890/com.pedro.rtpstreamer E/lol: onNewBitrate: 15.295479 Mb/s
2020-04-22 12:16:22.058 18740-18890/com.pedro.rtpstreamer E/lol: Real upload speed: 4.0150604 Mb/s
2020-04-22 12:16:23.081 18740-18890/com.pedro.rtpstreamer E/lol: onNewBitrate: 9.659401 Mb/s
2020-04-22 12:16:23.083 18740-18890/com.pedro.rtpstreamer E/lol: Real upload speed: 3.8979187 Mb/s
2020-04-22 12:16:24.110 18740-18890/com.pedro.rtpstreamer E/lol: onNewBitrate: 1.9280396 Mb/s
2020-04-22 12:16:24.111 18740-18890/com.pedro.rtpstreamer E/lol: Real upload speed: 3.4475403 Mb/s

After delete line socket.setSendBufferSize(acknowledgementWindowsize);:
2020-04-22 12:17:51.831 19076-19214/com.pedro.rtpstreamer E/lol: onNewBitrate: 4.0180664 Mb/s
2020-04-22 12:17:51.832 19076-19214/com.pedro.rtpstreamer E/lol: Real upload speed: 4.3313904 Mb/s
2020-04-22 12:17:52.943 19076-19214/com.pedro.rtpstreamer E/lol: onNewBitrate: 3.9006958 Mb/s
2020-04-22 12:17:52.944 19076-19214/com.pedro.rtpstreamer E/lol: Real upload speed: 3.4375 Mb/s
2020-04-22 12:17:54.087 19076-19214/com.pedro.rtpstreamer E/lol: onNewBitrate: 3.7883148 Mb/s
2020-04-22 12:17:54.087 19076-19214/com.pedro.rtpstreamer E/lol: Real upload speed: 4.0672913 Mb/s
2020-04-22 12:17:55.262 19076-19214/com.pedro.rtpstreamer E/lol: onNewBitrate: 4.4927063 Mb/s
2020-04-22 12:17:55.263 19076-19214/com.pedro.rtpstreamer E/lol: Real upload speed: 4.1234436 Mb/s
2020-04-22 12:17:56.382 19076-19214/com.pedro.rtpstreamer E/lol: onNewBitrate: 3.8261108 Mb/s
2020-04-22 12:17:56.384 19076-19214/com.pedro.rtpstreamer E/lol: Real upload speed: 3.8874817 Mb/s
2020-04-22 12:17:57.465 19076-19214/com.pedro.rtpstreamer E/lol: onNewBitrate: 3.7574768 Mb/s
2020-04-22 12:17:57.466 19076-19214/com.pedro.rtpstreamer E/lol: Real upload speed: 4.792206 Mb/s
2020-04-22 12:17:58.507 19076-19214/com.pedro.rtpstreamer E/lol: onNewBitrate: 4.0800247 Mb/s
2020-04-22 12:17:58.508 19076-19214/com.pedro.rtpstreamer E/lol: Real upload speed: 3.2387543 Mb/s

You are right socket buffer was the problem.

Hi @pedroSG94
Is it safe to remove this buffer? What it was used for before?
I have also some concerns regarding the algorithm itself. Imagine this case:

We're on fast WiFi (10Mb/s) and the max bitrate is set to 5Mb/s. Adapted bitrate will be ~5Mb/s then.
Then, we turn off WiFI and switch to 3G where upload speed is 3.5Mb/s.
Unfortunately, the algorithm won't decrease the bitrate to 3.5Mb/s. It is because of this line
 if (averageBitrate < oldBitrate * 0.65) { // decrease by 10% }
a) so the averageBitrate is 5Mb/s, and the 0.65 of it = 3.25 (won't decrease)
b) after 5 seconds the average bitrate will be ~3.5Mb/s, oldBitrate is still 5Mb/s, so
3.5 < 3.25 will still return false and the bitrate won't decrease.

You can try it yourself. Anyways it would be very useful to have such an algorithm unit tested, otherwise it will be hard to test all cases.
Moreover, I can still notice discrepancies between real upload speed and bitrate from onNewBitrate method. I think the problem is the way the bitrate is being calculated.
In the BitrateManager.calculateBitrate() there is
 if (timeDiff >= 1000) {
      connectCheckerRtmp.onNewBitrateRtmp(bitrate);

The timeDiff may be 3 seconds so the real bitrate is bitrate / 3s. IMO it should be like this
 if (timeDiff >= 1000) {
      connectCheckerRtmp.onNewBitrateRtmp((long) (bitrate / (timeDiff / 1000f)));

Now it's better but there is another issue. You removed setting the buffer, but the buffer is still there, so I guess there is some default value.

Yes, buffer size has a default value (1048576) but I'm not sure if change that value to 1 could produce a problem (0 is not allowed according to java documentation) and now, both bitrate are similar, so I'm doing a fix to decrease bitrate in all case

I think this could work:
  private int getBitrateAdapted2(int bitrate) {
    if (bitrate >= maxBitrate) { //You have high speed and max bitrate. Keep max speed
      oldBitrate = maxBitrate;
      return oldBitrate;
    } else if (bitrate <= oldBitrate) { //You have low speed and bitrate too high. Reduce bitrate by 10%.
      oldBitrate = (int) (bitrate * 0.9);
      return oldBitrate;
    } else { //You have high speed and bitrate too low. Increase bitrate by 10%.
      oldBitrate = (int) (bitrate * 1.1);
      if (oldBitrate > maxBitrate) oldBitrate = maxBitrate;
      return oldBitrate;
    }
  }

Hello Pedro, are you planning a release soon fixing this kind of things? I'm launching an app next week and I'd like to compile the best version of your library, thanks!

Yes, I will do it soon. I think this monday

@pedroSG94 This algorithm will reduce the bitrate if needed, but I'm not sure if it will increase it.
Let's bring back again this scenario 5Mb/s to 3Mb/s.

I switch off WiFi (5Mb/s) and switch on 3G (3Mb/s).
Bitrate decreases to 3Mb/s. So far so good.
Now I switch back again WiFI and bitrate won't increase because the amount of data being uploaded will be the same as bitrate, right? So we have a bitrate of 3Mb/s and it will be also the upload speed I guess? That way the condition else if (bitrate <= oldBitrate) may be false most of the time.

Tell me if I'm wrong, but I guess the bitrate doesn't show the real upload speed. Anyways the case is worth testing.

Yes, I will do it soon. I think this monday

Excellent!

@marcin-adamczewski You are right, bitrate it not totally real speed but it is very close (at least in my case the different is around 0.2 or 0.3Mb/s less than 10% error) and sometimes (bitrate <= oldBitrate) could be true and decrease bitrate. For this reason, I wanted ask your about it. My other idea is use (bitrate <= oldBitrate * 0.9f) to decrease chance to be true but this could produce others problems like lose packets because you can't decrease if upload speed is 10% less than bitrate used in app.
The best solution is use TrafficStats methods but we can't discard upload speed used in background apps.

Yeap that's probably better. I'd need to test it though. Regarding some small packets lose it this case won't the buffer help here?
Regarding the buffer, maybe it's size should be configurable? Have a look at this article https://www.red5pro.com/blog/what-is-buffer-time-and-how-does-it-affect-your-streaming/
So as I understand the bigger buffer the fewer lags on the client side, but on the other hand the stream is less real-time. So depending on the case you may want to adjust it. Other cases where buffer may be useful are slow devices, where video encoding to h264 could be slow, which actually reduces real bandwidth if there is no buffer. Btw I'm completely not an expert here so take my opinion with a grain of salt.

This is the value that we set with setSendBufferSize:
https://docs.oracle.com/javase/8/docs/api/java/net/SocketOptions.html#SO_SNDBUF
In my experience(maybe I'm wrong), this isn't a buffer that you can use to discard frames because if you have this buffer full and you continue send packets, that is stored in memory waiting to enter in buffer but you haven't a limit(if you have a low speed the result is OutOfMemory error).
I think that buffer work like:
You send packets to buffer, socket wait fill that buffer and send it to server (this could explain upload speed difference).
About red5pro. I think they are referring to other buffer not that method. Maybe something like I have here(I use this to avoid OutOfMemory error):
https://github.com/pedroSG94/rtmp-rtsp-stream-client-java/blob/master/rtmp/src/main/java/net/ossrs/rtmp/SrsFlvMuxer.java#L124

I used this adaptive bitrate algorithm in the past and it worked really well, and should avoid the oldBitrate newBitrate comparison and problems @marcin-adamczewski mentioned, based on comparing actual bytes sent vs bitrate.

During streaming on timer check delta of TrafficStats.getTotalTxBytes()  - it's supposed to be per app, meaning it would be actual bytes sent by app per last second.
Compare the delta with current bitrate setting,
If it is below the bitrate for 2-3 consecutive seconds, then trigger insufficientBandwidth routine
If it more than or equals current bitrate for 2-3 seconds then trigger sufficientBandwidth
In insufficientBandwidth - reduce bitrate in half OR reduce by 512kbps up to minimum such as 100kbps
In sufficientBandwidth - (If fps is more than half of target) set bitrate = bitrate + 128kbps up to maximum of target bitrate

Please let me know if it would work

Hello @mkrn,
Actually I'm doing a similar way that you suggest but increase and decrease bitrate is different and I change it every second instead of every 2 or 3 seconds.
I don't think that the problem is the number of increase or decrease in bitrate I think that the problem is cache stored when you have internet problems. For example:
1 - Start stream with connection of 10Mbs and you set bitrate to 5Mbs. No problem here for now so all is working fine.
2 - After few minutes you have connection problems and your connection downgrade to 1Mbs that means that you can't send 5Mbs to this need to decrease.
3 - Before bitrate reduction is executed you are still generating frames with 5Mbs because it is not detected yet (you need wait 1s to detect it). For this reason, that frames are stored in the queue and this will need more time to send because you can't send it on real time.
This effect produce stream issues until that frames of 5Mbs are send and incomming frames of 1Mbs(bitrate reduced) is descarded because you can't send more frames to the queue. After all frames of 5Mbs are send, you can continue stream as expected with 1Mbs but you lost frames.
The question here is. What should we do with that 5Mbs frames stored in queue?
Clear the queue and lose that frames?
About TrafficStats.getTotalTxBytes() according with official documentation: Return number of bytes transmitted since device boot. Counts packets across all network interfaces. It seems to be of all apps, anyway, get internet connection using bytes send by socket is really close to the real measure so I don't think that I should change it. I think the problem is the reduction algorithm and how to handle with buffer queue.

Thank you for explaining the queue I think I will experiment with reducing the cache size, to mitigate it.
Is https://developer.android.com/reference/android/net/TrafficStats#getUidTxBytes(int)  getUidTxBytes the right function as it's per UID?
Also, have you considered using  BITRATE_MODE_CBR (Constant bitrate mode) to make encoder bitrate more constant and not spike for movement causing the issues?

I'm trying different algorithms but really can't think of a good one because:
Actual Bitrate is all over the place, you set one bitrate, encoder can produce different bitrate, and since we're counting length of frames sent, the queue is also varying the bitrate.
Do you think it's better count lost frames and if there are > certain amount per period of time, reduce bitrate, and if the framerate is good, then increase until maximum?

Thank you for explaining the queue I think I will experiment with reducing the cache size, to mitigate it.
Is https://developer.android.com/reference/android/net/TrafficStats#getUidTxBytes(int) getUidTxBytes the right function as it's per UID?
Also, have you considered using BITRATE_MODE_CBR (Constant bitrate mode) to make encoder bitrate more constant and not spike for movement causing the issues?

I need investigate about that method to calculate traffic but actual method is working fine so I think that it has not sense to implement it.
About CBR mode. You are right, I considered it before but not all device support it for this reason I let it by default by set it if supported is a good idea.

Do you think it's better count lost frames and if there are > certain amount per period of time, reduce bitrate, and if the framerate is good, then increase until maximum?

It is not a good idea. You need wait until queue is full to know if you have internet problem and normally this is slower than calculate it using bitrate. Also the problem of how to handle with buffers in the queue is not solved.
I think that discard all frames except keyframes until the queue is half full could be a good solution.

Basically, I didn't have luck with the current implementation so I had to adjust it to my needs. I'll describe it below as it may be helpful for others. There were 2 biggest problems I faced:

Measuring the upload speed the way it is now, gives very often false results because of the socket buffer. So on my device, it was 10Mb buffer (I believe it is a device-specific value). So when your encoder produces 3Mb/s and sends it through the socket it actually saves all 3Mb of data to the buffer first. So it may calculate your upload speed as 3Mb/s but in fact, it may be 1Mb/s.
To resolve the issue I had to set socket.sendSendBufferSize to some lower value like 1KB. Keep in mind you have to do this before connecting to the socket, otherwise, it won't have any effect.
Periodical increase of bitrate always caused some lags. There is some logic in the current implementation that increases the bitrate by 10% if everything is fine. It always caused that at some moment the bitrate was too high for the upload speed and there was a lag for 2 seconds until the bitrate has been reduced. Because of this, I had to change the algorithm, which looks more or less like this:

define the minimum and maximum bitrate (in my case 1Mb/s-3Mb/s)
start a global BitrateEstimator that constantly measures current upload speed
if the video frames buffer size is greater than 20% (the one from SrsFlvMuxer) there is probably congestion. Run a new BitrateEstimator for 6 seconds to measure current upload speed, but immediately set a new bitrate using the currently estimated bitrate from the global estimator for a fast recover. After 6 seconds you'll have a new estimated upload speed.
when either stream has connected or network changed (wifi <-> cellular), run new estimator for 6 seconds, starting with some initial bitrate (each network type has its own initial bitrate)

The drawback is that it can only increase the bitrate on network switch or reconnection. But in my case, it's good enough.

Thanks @pedroSG94 @marcin-adamczewski
Do you think the adaptive bitrate algorithm could be done based solely on the queue size of the SrsFlvMuxer?
What's the variance of actual bitrate produced by encoder in your experience? When I set bitrate using setBitrateOnFly` to a value like 5.5 Mbps the calculateBitrate returns values that range from 2.2 to 8.7 Mbps.

Hello,
I did 2 commit for it.

One to reduce socket buffer and get better bitrate calculation:
https://github.com/pedroSG94/rtmp-rtsp-stream-client-java/commit/bee8acb9f7af84eb54666596ac1af934f523dddb
Another one to calculate adaptBitrate using video queue state:
https://github.com/pedroSG94/rtmp-rtsp-stream-client-java/commit/938bcb1f9034c11dfb7d2456fd399219a44555c2
In this last you can use like this:

//Replace
if (bitrateAdapter != null) bitrateAdapter.adaptBitrate(bitrate);
//To
if (bitrateAdapter != null) bitrateAdapter.adaptBitrate(bitrate, rtmpCamera2.hasCongestion());

Thanks @pedroSG94
We've tested that and it seems like on some devices it works as intended, and the queue has 1-2 frames in it always while on some devices the queue is constantly full from the very beginning, as a result hasCongestion always true and the bitrate is extremely low (one is galaxy A20S)...
What could this be related to?

@mkrn

Do you think the adaptive bitrate algorithm could be done based solely on the queue size of the SrsFlvMuxer?

If you want to only reduce bitrate I believe it's enough to use this queue size. As far I remember the queue by default can store  30 video frames which is 1 second for 30fps video. If this queue starts filling up, say it's 20% filled then you most probably have congestion and can reduce the bitrate.
The problem is how to increase the bitrate. You could do that for example when the buffer has 3 or less frames but the amount you increase the bitrate may be too high, resulting in a lag.

What's the variance of actual bitrate produced by encoder in your experience? When I set bitrate using setBitrateOnFly` to a value like 5.5 Mbps the calculateBitrate returns values that range from 2.2 to 8.7 Mbps.

I guess there was no big variance after reducing the buffer size of the socket, I'd need to test it though.

@pedroSG94 I believe that you'd have to call the setSendBufferSize method before calling socket.connect(). I had to do this way, otherwise, it had no effect.

@mkrn I always have congestion at the beginning. I'm not sure 100% of the cause but it may be the TCP slow-start mechanism. Basically at the beginning of TCP connection, the speed is very slow and increases with time. I guess it's a matter of 1-2 seconds to get full speed (not sure). That's why the buffer may be full at the beginning.
I don't want to confuse you too much but if the current solution won't be enough for you can check out my implementation
https://github.com/marcin-adamczewski/rtmp-rtsp-stream-client-java/blob/master/app/src/main/java/com/pedro/rtpstreamer/openglexample/OpenGlRtmpActivity.java
The class name is BitrateAdjuster
And as I said it won't increase the bitrate dynamically but only on a new connection or when Internet type switches.
It also hasn't been released to users yet so it's not tested by life :D

@pedroSG94 I believe that you'd have to call the setSendBufferSize method before calling socket.connect(). I had to do this way, otherwise, it had no effect.

I was doing tests and it was working not matter when it is called.
Could this depend of device used?
I was testing with this method:
https://docs.oracle.com/javase/7/docs/api/java/net/Socket.html#getSendBufferSize()
If I call setSendBufferSize = 1 that method return 4096 no matter when it is called and If I never called setSendBufferSize then return about 10000
For this reason, since I can't call it before connection using SSL socket (because that method return a connected socket) I'm doing it after connect in both cases.

@pedroSG94 I've checked it and you're right. It doesn't matter if you call it before or after connecting. I probably confused it with the setReadBufferSize() method which requires setting it before the connect() in some cases.
BTW in my case, the getSendBufferSize method always returns twice a bigger size buffer than set by setSendBufferSize.
Maybe it's not important but it may be risky to set the buffer to 1 as there is actually no buffer. I guess it's there for a reason to increase writing speed, that's why I've set it to 10 * 1024 (10kB). It didn't impact the bitrate calculations in my case compared to the default value of the buffer which in my case is 1MB.

@pedroSG94  @marcin-adamczewski  we found setBufferSize(1) didn't work on some devices.
socket.setSendBufferSize(2*1024*1024); works reliably on variety of devices.

@mkrn 2*1024*1024 is 2MB which is 16Mb. I guess it's gonna definitely impact the bitrate calculations as the bitrate is usually ~1-3Mb/s. So this buffer can contain 5-16 seconds of a stream.

@pedroSG94 @marcin-adamczewski we found setBufferSize(1) didn't work on some devices.

It is doing a crash or what do you mean?
Tecnically, if you set it to 1, that method delegate to kernel and select the valid value closer to your value selected

@marcin-adamczewski Actually I end up not relying on upload speed calculation at all in adaptive bitrate.
What we do now is - if there is congestion (i.e. queue is 15% full) then we immediately drop the target bitrate by set amount, then re-evaluate every 2s, if there's no congestion raise by lower amount. Basically adding and subtracting to the same target, instead of manipulating the value received from upload speed calculation.

@pedroSG94 I've noticed that reducing the send buffer size impacts the streaming performance of some devices. I set the buffer size to 10kB and it reduced bandwidth by ~60% on my Nexus 5x. Our QA reported such an issue on Samsung S8 I guess. Setting the buffer size to 50kB resolved the issue but who knows how it's gonna behave on other devices.
Here's a very interesting answer on SO that explains why there may be congestion when the buffer is too small compared to bandwidth and latency https://stackoverflow.com/a/41747545/8065223
Based on that I made some calculations assuming a fast network (100Mb/s) and quite a big round trip time of 50ms.
bufferSize = 100Mb/s * 50ms = 100Mb/s * 0.05s = 5Mb/s = 5000kb/s = 625kB/s
We probably don't need to utilize a full bandwidth of 100Mb/s for 3Mb/s bitrate video, but if we'd like to fully utilize 10Mb/s speed we'd still need 62KB in the buffer.
Anyways as I don't want to risk congestion I decided to use TrafficStats class for now.

@mkrn and @marcin-adamczewski Thank you for all test and provide this info.
If this could produce problems in somes devices. I think that I will delete setBufferSize assuming that default value will provide a fair performance in all devices.

Hi,
I am facing the same objective that to offer my customers the best possible experience.
Bref.
By reading this thread, thanks for all to share code, reflexions et opinions ... if the problem, in the end, is to detect when the buffer is to heavy because the bandwidth comes slow but haven't yet downgrade the bitrate, alors it will be good to watch the evolution of the buffer.
I am not a great mathematician ... but in my opinion there is a little window there: use a algorythm, maybe with a little IA, that monitor the evolution of the size of the buffer to "anticipate" a bandwidth problem and downgrade the bitrate before it really need it.
I think, user that broadcast has 75% always the same bandwidth environnement so ia is to get custom action for each user.
My idea is that the buffer has a normal "saturation" when everything go normally and when it start to not act normally it's a sign that it may gonna have problem.
Voilà tout.

I took a deep dive into the adaptive bitrate in this library (latest version, 2.1.4), and I've been able to fix most of the adaptive bitrate problems.
Here's what I found:
Problems with default BitrateAdapter

Average was not properly calculated
Bandwidth in the last second (onNewBitrateRtmp) is calculated as the total data size send to the socket, with packet headers and audio packets. Yet bitrate is set for video encoder only (audio bitrate set separately).
If there is congestion, need to immediately reduce bitrate, to minimize drop frames and even worse congestion
Keep the cache size (queue length) to the minimum, so congestion can be quickly discovered (it's 20% of the cache size), and so the backlog of old frames is easier to push out. I suggest 30.
It is necessary to account for what was in the queue for bandwidth calculation before raising bitrate, because of the bitrate adjustments. Wait for a few cycles after lowering to raise again.

Notice: All of this makes sense with Constant Bitrate only!
Here's my version of BitrateAdapter that accounts for audio bitrate, packet overhead, and reacts faster to congestion.
package pro.eventlive;

import android.util.Log;

public class BitrateAdapter {

  public interface Listener {
    void onBitrateAdapted(int bitrate);
  }

  private int maxBitrate;
  private int minBitrate = 100 * 1024;
  private int audioBitrate = 96 * 1000;
  private int oldBitrate;
  private int averageBitrate;
  private int cont;
  private int cyclesSinceReduced = 0;
  private Listener listener;
  private float decreaseRange = 0.8f; //20%
  private float increaseRange = 1.2f; //20%

  private int decreaseBy = 256 * 1024;
  private int increaseBy = 128 * 1024;
  private int packetOverhead = 15 * 1024; // Magic number

  public BitrateAdapter(Listener listener) {
    this.listener = listener;
    reset();
  }

  public void setMaxBitrate(int bitrate) {
    this.maxBitrate = bitrate;
    this.oldBitrate = bitrate;
    reset();
  }

  public void adaptBitrate(long actualBitrate) {
    averageBitrate += actualBitrate;
    if (cont > 0) {
        averageBitrate /= 2;
    }
    cont++;
    if (cont >= 3) { // lowered the measurement interval from 5s to 3s
      if (listener != null && maxBitrate != 0) {
        listener.onBitrateAdapted(getBitrateAdapted(averageBitrate));
        reset();
      }
    }
  }

  /**
   * Adapt bitrate on fly based on queue state.
   */
  public void adaptBitrate(long actualBitrate, boolean hasCongestion) {

    averageBitrate += actualBitrate;
    if (cont > 0) {
        averageBitrate /= 2;
    }
    cont++;

    if (hasCongestion) {
        // Immediately react with adjustments
        listener.onBitrateAdapted(getBitrateAdapted(averageBitrate, hasCongestion));
        reset();
    }

    if (cont >= 3) { // lowered the measurement interval from 5s to 3s
      if (listener != null && maxBitrate != 0) {
        cyclesSinceReduced++;
        listener.onBitrateAdapted(getBitrateAdapted(averageBitrate, hasCongestion));
        reset();
      }
    }
  }

  // Version that doesn't take the queue size into account
  private int getBitrateAdapted(int averageBw) {
    if (averageBw >= maxBitrate) { //You have high speed and max bitrate. Keep max speed
      oldBitrate = maxBitrate;
    } else if (averageBw <= oldBitrate * 0.9f) { //You have low speed and bitrate too high. Reduce bitrate
      oldBitrate = Math.max(averageBw - audioBitrate - decreaseBy, minBitrate);
    } else if (averageBw >= oldBitrate) { //You have high speed and bitrate too low. Increase bitrate
      oldBitrate = Math.min(oldBitrate + increaseBy, maxBitrate);
    }
    // keep it otherwise
    return oldBitrate;
  }

  private int getBitrateAdapted(int averageBw, boolean hasCongestion) {
    // what we expect should have been sent over the network
    int expectedBandwidth = oldBitrate + audioBitrate + packetOverhead;
    // Explicitly has congestion in the queue or average bandwidth 90% less than expected
    if (hasCongestion || (averageBw <= expectedBandwidth * 0.9f)) {
        // Reduce! decreaseBy is added for when there's congestion but bitrate - audioBitrate is within current bitrate
        oldBitrate = Math.max(averageBw - audioBitrate - decreaseBy, minBitrate);
        cyclesSinceReduced = 0; // wait a few cycles to let the higher bitrate frames in the queue pass through
    } else if (averageBw >= expectedBandwidth&& cyclesSinceReduced >= 3) {
        // When fully recovered, attempt to increase by a bit
        oldBitrate = Math.min(oldBitrate + increaseBy, maxBitrate);
    }

    // keep it otherwise
    return oldBitrate;
  }

  public void reset() {
    averageBitrate = 0;
    cont = 0;
  }

  public float getDecreaseRange() {
    return decreaseRange;
  }

  /**
   * @param decreaseRange in percent. How many bitrate will be reduced based on oldBitrate.
   * valid values:
   * 0 to 100 not included
   */
  public void setDecreaseRange(float decreaseRange) {
    if (decreaseRange > 0f && decreaseRange < 100f) {
      this.decreaseRange = 1f - (decreaseRange / 100f);
    }
  }

  public float getIncreaseRange() {
    return increaseRange;
  }

  /**
   * @param increaseRange in percent. How many bitrate will be increment based on oldBitrate.
   * valid values:
   * 0 to 100
   */
  public void setIncreaseRange(float increaseRange) {
    if (increaseRange > 0f && increaseRange < 100f) {
      this.increaseRange = 1f + (increaseRange / 100f);
    }
  }
}

Suggestions to improve further:

If the queue is congested and not sending the frames, then calculateBitrate will not be invoked and onNewBitrateRtmp will not be called and bitrate would not be adjusted. Instead, I suggest using a timer or callback for congestion.
When the queue is full we should not discard audio packets because they're small compared to video. Ideally, we should not discard Keyframes because without them the picture falls apart. How to do it? Two separate queues? Or queue clean up?   It is better to miss some video frames than audio - because you will hear pauses in the audio but may not notice low frame rate.

@pedroSG94  I can make a PR with the BitrateAdapter improvements, just need your input on some things:

Should we pass Audio bitrate into it?
Can we make separate audio and video queues?
Can we discard regular video frames before keyframes?
Can onNewBitrateRtmp be called exactly every second, instead of whenever the time elapsed is over 1 second?

I meant passing audioBitrate to BitrateAdapter.
onNewBitrateRtmp reporting bandwidth for last second is good. That way you can display it in the app as your upload rate. In BitrateAdapter it's important to know the audio bitrate to calculate the video bitrate... i.e. videoBitrate = Bandwidth - audioBitrate - overhead

This is the branch for this:
https://github.com/pedroSG94/rtmp-rtsp-stream-client-java/tree/adaptativebitrate
For now I only did the 1. I will develop 2 and 3 tomorrow

I updated branch with number 3.
About number 2 this is producing a big performance problem and sync both queues to avoid unnecessary wait time it is not working fine.
You only need this because you want check remaing capacity in video queue, right? For now, test with point 1 and 3 and I will think other way to do it


Feature request throws

It would be cool to have some way to document exceptions. Maybe as follows.
throw new Error('foo') // => throws Error('foo')

Also, is there some way to deal with Promises?
Promise.resolve(123) // => resolves to 123
Promise.reject('problem') // => rejects with 'problem'

Good suggestions! I'm not actively working on this project but feel free to send a PR.


Update compute plan

PR #80 should be merged first

Tested against all the companion PRs, the full suite of test passes! Well done!

Found by @Kelvin-M: you can't update a compute plan with just testtuples.


fix: Remove argumentError doctest

First, the test itself should work because the character has a :name, so I can't get that test to "pass". When changing it so it does throw an error, the whole liveview process throws an error:

Options I attempted:

I tried the original with no name passed to the struct
I tried using assert_raise with the third param, including the message but isn't required and made the test long (and still failed)
I tried to use assert_raise as well but the argument error is still propagated to the livebook, and kills compilation.

  iex> assert_raise ArgumentError, fn -> %Character{} end
  :ok

There's a CI issue I should be able to fix fairly easily when I have time. Since it is an improvement, I'm going to merge this in y. Thank you, @jonathanstiansen, for the catch!


branch-3.0 Avoid interrupt daemon thread and use proper polling interval (#42210)

pick https://github.com/apache/doris/pull/42210

run buildall


Set env variable to indicate current thread

I tried cypress-parallel but I ran into an issue where the different threads are filling the same mysql db
It would be great if an env variable is set to indicate which thread is running the current test.
That way, I can possibly seed a different db for each thread
Thank you

process.env.PARALLEL_THREAD already contains the thread number

Seems like config.env.PARALLEL_THREAD was removed with the new cypress update (6.0.0)
I could sumbit a PR to add the env variable if you'd like
Cheers


Python requirements and versions matching

Hello!
We are trying to run this script on Mac (both M1 / M2 / Apply Silicon -- newest). We have just realized that:

Apple Silicon doesn't natively support Python 3.8
Conda does not support Python 3.6 but still supports Python 3.8, but the required package versions cannot be met. For instance, conda install -c tensorflow-gpu=2.2.0 or keras=2.3.1. would not work (with conda-forge)[1]. With python 3.8, import statements such as import tensorflow.compat.v2 as tf in /opt/anaconda3/envs/py38/lib/python3.8/site-packages/keras/src/engine/functional.py would not work as it requires lower-version tensorflow and keras, which again Conda doesn't support now.

Given this, what would be the best way to run the SynthSeg script, locally on the Macbook? Is there a plan to support this script in later Python versions (3.10, 3.11) that would be even more compatible with the newer hardware structures in the future?
[1] Log for conda create -n synthseg_38 python=3.8 tensorflow-gpu=2.2.0 keras=2.3.1 nibabel matplotlib -c anaconda -c conda-forge
Channels:

anaconda
conda-forge
defaults
Platform: osx-arm64
Collecting package metadata (repodata.json): done
Solving environment: failed

PackagesNotFoundError: The following packages are not available from current channels:

tensorflow-gpu=2.2.0*
keras=2.3.1*

Current channels:

https://conda.anaconda.org/anaconda
https://conda.anaconda.org/conda-forge
defaults

Hi, @jennykim1016 I noticed this issue while working on my own. While I haven't tested the full range of synthseg's functionality, my preliminary work suggests you should be able to use this code (barring a minor infraction here and there) with a newer version of TF/Keras and Python. The farthest I could go was TF=2.15 and Python=3.11 with generation (at least).  I am currently looking into prediction and will look into training very soon. If my tests turn out well on these three aspects (generation, training, and prediction), it will suffice to say SynthSeg can run on newer versions seamlessly.

@hvgazula It would be awesome if you share your requirements_python311.txt. Good luck with training!

@hvgazula It would be awesome if you share your requirements_python311.txt. Good luck with training!

Here they are:
ss_reks_centos.txt
ss_reks_osx.txt

Hi,
thanks all for the questions and answers!
I think I started looking into this a while ago and I was able to find a stable list of requirements for python 3.x (x>10), but I dropped it because I didn't like the new way of debugging keras models (it didn't allow anymore to access the values inside the call methods of custom keras layers).
I seem to remember that using this new list of requirements only needed minor changes in the code to make it run for training/testing. Unfortunately I don't think I saved this list of requirements... But using conda, it should be kind of straightforward to establish for any newer Python version.
Let me know if @hvgazula's list of requirements works, and then I can add it to the installation instructions :)
Thanks all!
Benjamin

Thank you so much everyone for these helpful discussions! We have actually decided to move forward with Windows + appropriate versions for now, but if we do decide to test with a new Python version, I will make sure to test it and update the thread. Thanks!

Im testing this now. As an aside, given that synthseg and mri_easy_reg are both part of freesurfer, is it likely that if I get the gpu to work in synthseg, that I can get it to work on the easy-reg software (after telling its torch code to use the gpu device).?

Well I'm sure there are some dependency differences between SynthSeg and EasyReg, so you might to use two different venv etc.


Mapping of user domain to 'users' table

Set up fails because 'user' is a reserved name in some RDBMS like Postgresql

Without any migration this change will make all previous users unusable.

Hey @Nighoo ,sorry and let me do that next

@Nighoo is very right, so I cannot accept this PR. However, we can think about a second rollout for Postgresql?


[LTR]About lambda pair generation.

I found that not all of the valid pairs were constructed, when making lambda pairs here.
For example, given rank label list [2, 2, 1, 0, 0, 0], the valid index pairs should be (0, 2), (0, 3), (0, 4), (0, 5), (1, 2), (1, 3), (1, 4), (1, 5), (2, 3), (2, 4), (2, 5). But the implementation in rank_obj.cc:76-93 seems to randomly drop some valid pairs.
Is there any reason of this implementation or paper that claims this kind of pair selection is helpful ?
Thanks!

closed


Update to use modular lodash and some linting fixes.

Fixes #20

I needed to install eslint-plugin-react because Atom was complaining that I needed it to do linting. Presumably because the .eslintrc has some react rules. Feel free to remove those rules and the dependency if you like.

Now in v0.3.10,  thanks for helping out!


[Bug]: ValidationError: 2 validation errors for StructuredRefineResponse answer

Bug Description
While using response synthesizer in below way -:
response_synthesizer = get_response_synthesizer(service_context=service_context, refine_template=DEFAULT_REFINE_PROMPT, structured_answer_filtering=True, use_async=True, verbose=True)
where I am passing structured_answer_filtering=True I am getting below error for few queries for others it works -:
ValidationError: 2 validation errors for StructuredRefineResponse answer field required (type=value_error.missing) query_satisfied field required (type=value_error.missing)
it gets answer from few nodes but in later chunks it is not getting required field like query satisfied, so we should have flag, when this error comes whatever response has been generated that should be given, because it is failing due to some reason in after few nodes traversing, ideally it should have found at least some context.
I am not much aware about the behind the scene functionality of structured_answer_filtering param, but keeping it True gives meaningful response for few queries for others the error is given.
And if I do not use this the answer which I get is not meaningful, so need to see if there is a fix I can do to it.
Version
0.8.14
Steps to Reproduce
Pass structured_answer_filtering param as True, and for few queries where data retrieved is large than this issue occurs since it stops in between, and some required params which pydantic is expecting like query_satisfied is not given.
Relevant Logs/Tracbacks
File "pydantic/main.py", line 549, in pydantic.main.BaseModel.parse_raw
  File "pydantic/main.py", line 526, in pydantic.main.BaseModel.parse_obj
  File "pydantic/main.py", line 341, in pydantic.main.BaseModel.__init__
pydantic.error_wrappers.ValidationError: 2 validation errors for StructuredRefineResponse
answer
  field required (type=value_error.missing)
query_satisfied
  field required (type=value_error.missing)

This info I have been through with docs, I know this feature requires function calling which should be supported by llm, but since it is working for other queries and for few queries it is breaking because of larger context retrieved for model and after few iteration it is failing if it would have failed on second call itself or would not be working for other queries could have considered it related to llm limitation of functional calling, but here it is not, so what I understand is, it will traverse to each chunk and will gather some info from llm, than it will query again with next chunk and make sure info only gets added if it is relevant. So after few iteration due to some reason it is failing. Than it should be able to give whatever response it has got after going through few chunks. Rather than throwing exception without any result. Let me know if there is something I am missing or additional context is required.

And just to add for the llama team, I think this can be checked inside _refine_response_single method of refine.py inside respose_synthesizer module, we can make some fix.

Proposing fix in PR -: https://github.com/jerryjliu/llama_index/pull/7523, please check and update accordingly

Added one more change in _give_response_single because sometime in first chunk till the response is None this issue occurs, meanwhile some other chunks could give some answer pushed it as well and updated async function too. All in all the main idea of the PR is to highlight the issue area if the reason why sometime below line is not getting executed -:
structured_response = cast(
                        StructuredRefineResponse, program(context_str=cur_text_chunk)
                    )

If that can be fixed this exception handling and respose returning could be removed, but for now working as hotfix for me so proposing it

Closing issue since fix has been merged with this PR -: https://github.com/jerryjliu/llama_index/pull/7523


change-to-tech-preview

Changing "Tech Preview" to "Technical Preview"

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.


Added support for regtest and Azure Storage Emulator

Sample configuration for regtest will have the following:
<add key="Bitcoin.Network" value="regtest"/>
Sample configuration for Azure Storage Emulator will have the following:
<add key="Azure.AccountName" value="devstoreaccount1"/>
<add key="Azure.Key" value="Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw=="/>
<add key="AzureStorageEmulatorUsed" value="true"/>

I merge it but by experience the dev storage does not EXACTLY mirror the real API, I don't know if you'll have problem.


Running x64 containers on apple arm

When trying to use lspcontainers on arm mac, I'm getting errors like

[ERROR][2024-07-09 17:14:18] .../vim/lsp/rpc.lua:770	"rpc"	"/usr/local/bin/docker"	"stderr"	"Unable to find image 'lspcontainers/typescript-language-server:latest' locally\n"
[ERROR][2024-07-09 17:14:20] .../vim/lsp/rpc.lua:770	"rpc"	"/usr/local/bin/docker"	"stderr"	"latest: Pulling from lspcontainers/typescript-language-server\n"
[ERROR][2024-07-09 17:14:20] .../vim/lsp/rpc.lua:770	"rpc"	"/usr/local/bin/docker"	"stderr"	"docker: no matching manifest for linux/arm64/v8 in the manifest list entries.\nSee 'docker run --help'.\n"

As far as I understand rosetta allows to run x64 containers on arm chips. But I'm a bit struggling to understand what should I change in setup to make lspcontainers to download x64 containers and run those instead of searching for containers build specifically for arm
Will be very glad for any hints :)

We could probably look at putting --platform linux/amd64 on the run commands... I've not spent much time in this codebase though so I wouldn't know the best way to implement it.


add nextcloud parser, bruteforce detection and collection (#345)

add nextcloud parser, bruteforce detection and collection by

@umglurf : I guess it was the mysteries of the CI :sweat_smile:
I merge, thank you very much !


Bug fix: is_sanitized() did not consistently take all sanitization functions into account for final result

The $sanitizingFunctions and $unslashingSanitizingFunctions are two distinct lists and custom sanitizing / unslashsanitizing functions are only merged with one or the other.
The $sanitizingFunctions list currently contained an (incomplete) overlap with the $unslashingSanitizingFunctions which would become even more incomplete if/when custom functions would be merged in.
The $unslashingSanitizingFunctions was only considered to determine whether or not unslashing was needed.
Functions contained in that list which were not also contained in the $sanitizingFunctions list were not taken into account for the final result of the is_sanitized() function.
This has now been fixed.

Removed overlap between the functions listed in the two properties
is_sanitized() now also takes the $unslashingSanitizingFunctions into account when doing the final determination of whether something is sanitized or not.

@JDGrimes It's fine - though unnecessary - for the lists to overlap/intersect as long as they are both checked in the is_sanitized() function. The problem was caused by the fact that only one of the lists was being checked and that functions which had been added to the lists since where not consistently added to both.
Feels more like the original properties should have had documentation that anything in unslashsanitize should also be in sanitize and that documentation could then have been removed with the current change.
I don't mind adding extra info to the docs, but this feels like it would create more confusion instead of reducing it.

@jrfnl I don't know how others think about the properties, but I am accustomed to thinking about these two properties as being independent, intersecting lists. The sanitizing functions list contained all of the functions, and the unslashing-sanitizing list contained just those that also implicitly unslashed the data. I just think that people will assume that a $sanitizingFunctions property will contain a list of all sanitizing functions. And if they notice the other property, they may think that it is a separate list. I just think that it would be better to be explicit about this in the documentation here, since now the $sanitizingFunctions property doesn't contain a list of all sanitizing functions on its own. I realize what you are saying about it no longer making a difference, in practice, whether a function is in both lists or not. But I still think that people could get tripped up in regard to the non-exhaustive nature of this list when they are just looking at the code.

@JDGrimes Point taken. I've added documentation to this effect to both lists.
Making them explicitly complementary instead of intersecting will hopefully prevent further mishaps of sanitizing not being correctly recognized - as it was now by the lists not being in sync (quite apart from the lists always being out of sync if there would be custom unslashsanitize functions).


Wrong jump on second layer with label s

Hi. I'm including a screencast to better illustrate the problem.
Here's what I'm doing step by step.

My cursor at the start of the file 0,0
In this example I'm targeting line 58 Plug('hrsh7th/cmp-copilot')
Press s followed by '
Press )
At this point we can see there is red s on the first line and blue s on line 58
Press <space> to enable the second part
Blue label turned to red
Press s
Lightspeed jump to ' on the first line without label (previously was s) instead of line 58
If I press other label f & n on as on the screencast. Lightspeed jump correctly to the line

Thank you

Good catch, thank you!


ignore more keys to ignore on the vim side

I could reproduce alt-shift and win-shift. Not sure if there are more keys, I'll open more PRs if I find some.
fixes #415

Awesome! Thanks @despairblue - you're on 🔥 🔥 🔥 with these PRs!


[REQUEST] Feature Flags

Is your feature request related to a problem? Please describe.
Not an issue, more an enhancement for conditional access requirements.
Describe the solution you'd like
A flag in the beginning of the script that will set status of certain functions to enable or disable.

Pushing this back until I have an idea of how to do this cleanly


Fix error when calling coroutine with await in _ready

The code paths for calling async functions seemed to be missing in some cases, causing a debug break and false positive error.
I don't know if this is the proper fix, I might be missing something. But it does fix issues I had with using await in _ready function and doesn't seem to break anything.
Fixes #44101

Already fixed as part of #43890, closing.

Reopening as per https://github.com/godotengine/godot/issues/44101#issuecomment-747536293.


Card Issues

I've been pondering cards and it seems to me that you have to build this one pretty loose for userland.  I think this module is going to be hard to understand sans documentation.
Our children first concept has worked pretty good so far, but I think with a couple components we are going to hit some snags..
It mainly goes down to the fact that most cards are going to be CSS themed(or should be).  It'd be sweet if we could rebuild them with flexbox.

Card Proposal Layout:
<Card>
  <CardTitle>
    <CardTitleText></CardTitleText>
  </CardTitle>
  <CardTitleMedia></CardTitleMedia>
  <CardTitleText></CardTitleText>
  <CardTitleActions></CardTitleActions>
  <CardMenu></CardMenu>
</Card>


Trigger UI breaks down on mobile

Pretty big oversight on mobile - hovering isn't a thing, so the whole editor-opening UI falls apart.
Immediate thought - if on mobile show persistent edit button if editable. Could even do that on desktop too, there's no reason it needs to be hidden when not hovering, it's going to be equally obstructive to actual elementby editing regardless.
I'd also like to get the edit button out of the way of the collection by default (ie: scrolls up past the too of the collection if room available), will take a but of hacking with position sticky though

Managed to get the button sit on top of the top of the collection pretty easily, but it actually looks kinda weird, especially now that it's persistent. Will leave with current behavior for now, and maybe make a new issue to look at it again later.


Module not found: Can't resolve 'keycode'

Failed to compile
<EMAIL_ADDRESS>Module not found: Can't resolve 'keycode' in '../ory-editor/packages/plugins/layout/background/node_modules/@material-ui/lab/Slider'

You can try on node version 8. It working for me.


dependencies install error

I'm using Ubuntu 14.04 LTS and when I try this line:
sudo apt-get install build-essential ccache libgc-dev graphviz libunwind-dev pkg-config
I receive this error message:
Reading package lists... Done
Building dependency tree
Reading state information... Done
E: Unable to locate package libunwind-dev
But with command:
sudo apt-get install build-essential ccache libgc-dev graphviz libunwind8-dev pkg-config
it worked fine, I've just renamed "libunwind-dev" to "libunwind8-dev"
Thank you.

Hi,
Yeah we've already experienced the problem, sorry for not fixing that :/
However I'm not sure the proposed fix is safe since the version of
libunwind depends on the OS of the user, and if it is available on the
platform or not
I'm using Debian based mint, and on my machine, libunwind 7 is available
only, on a raspbian, no libunwind is available...
I think maybe we should just remove it altogether and request the end user
to search for his own version depending on his configuration.
Thanks for the hint !
On 28 Feb 2015 18:10, "Ouajdi"<EMAIL_ADDRESS>wrote:

I'm using Ubuntu 14.04 LTS and when I try this line:
sudo apt-get install build-essential ccache libgc-dev graphviz
libunwind-dev pkg-config
I receive this error message:
Reading package lists... Done
Building dependency tree
Reading state information... Done
E: Unable to locate package libunwind-dev
But with command:
sudo apt-get install build-essential ccache libgc-dev graphviz
libunwind8-dev pkg-config
it worked fine, I've just renamed "libunwind-dev" to "libunwind8-dev"
Thank you.
—
Reply to this email directly or view it on GitHub
https://github.com/privat/nit/issues/1182.

Thank for the issue.
The list specific list of required packages depends on the distribution and its version, since we cannot list and maintain each distribution, i suggests that we just choose a popular one and suggest to find equivalent packages for the others.

The list of dependencys it's availebale for debian 7.x (Wheezy) .

I udated the website to precise some versions


Implement temperature sensor

Please note that I don't have an SPI module so I couldn't test if it works with that interface

Yes, I've tested it on an actual hardware but only with I2C since I don't have an SPI module.


Dockerfile  build failing

RUN cp "$(swift build --package-path /build -c release --show-bin-path)/Server" ./
Dockerfile expects "Server" as executable name
Name of compiled executable is "App"  as defined in Package.swift
targets: [
.executableTarget(
name: "App",
Solution : change name in Dockerfile or Package.swift
ps. change "vapor user" in "hummingbird user" in Dockerfile :-)

Oh I forgot to update the Dockerfiles for many of the examples when merging them into one target.

#73 should fix this. Those docker files were ancient. They referenced swift 5.3


Fix dist folder static site

With this change the dist folder is properly created for a static site, fixing the issue on the repo

#1 FIXED


Lambda Function Unable To Find Header Files

Context:
I'm quite new to AWS and am go illiterate as well (for now). My current aim is to reproduce the experiment from this blog so that I can see how llama works and then hand it off to an intern to do more research to understand how we can integrate llama into our current meson-and-ninja based build system to build our firmware applications. However, in my efforts to build LLVM, I hit the following type of errors:
[98/2572] Building CXX object lib/Support/CMakeFiles/LLVMSupport.dir/AMDGPUMetadata.cpp.o FAILED: lib/Support/CMakeFiles/LLVMSupport.dir/AMDGPUMetadata.cpp.o  /home/njpau/go/bin/llamac++  -DGTEST_HAS_RTTI=0 -D_GNU_SOURCE -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS -Ilib/Support -I/home/njpau/git/llvm-project/llvm/lib/Support -I/usr/include/libxml2 -Iinclude -I/home/njpau/git/llvm-project/llvm/include -fPIC -fvisibility-inlines-hidden -Werror=date-time -std=c++11 -Wall -Wextra -Wno-unused-parameter -Wwrite-strings -Wcast-qual -Wno-missing-field-initializers -pedantic -Wno-long-long -Wno-maybe-uninitialized -Wdelete-non-virtual-dtor -Wno-comment -fdiagnostics-color -ffunction-sections -fdata-sections -O0    -fno-exceptions -fno-rtti -MD -MT lib/Support/CMakeFiles/LLVMSupport.dir/AMDGPUMetadata.cpp.o -MF lib/Support/CMakeFiles/LLVMSupport.dir/AMDGPUMetadata.cpp.o.d -o lib/Support/CMakeFiles/LLVMSupport.dir/AMDGPUMetadata.cpp.o -c /home/njpau/git/llvm-project/llvm/lib/Support/AMDGPUMetadata.cpp _root/home/njpau/git/llvm-project/llvm/lib/Support/AMDGPUMetadata.cpp:16:28: fatal error: llvm/ADT/Twine.h: No such file or directory  #include "llvm/ADT/Twine.h"                             ^ compilation terminated. Running llamacc: invoke: exit 1
Everything I have done today is from scratch, including setting up my free tier AWS account. These are the steps I've gone through:

Installed go on my 4C/8T Debian x86_64 GNU/Linux machine and set up an AWS account for myself.
Installed llama from source using the instructions in the Install llama section under Getting Started in the README. Did the llamac++ symlinking as well.
Installed awscli2 and my creds have been appropriately configured and stored in ~/.aws
Ran llama bootstrap. After completion, I went to my aws console and verified that the cloudformation stack had been successful created and that I had an empty S3 bucket, and empty ECR repo and an IAM role for basic lambda execution.
Ran home/njpau/go/src/github.com/nelhage/llama/scripts/build-gcc-image --local-headers to build my debian image and lambda function. Verified that I see 'gcc' image and function in my aws console under ECR and Lambda.
Cloned the llvm-project from github and checked-out the release/8.x branch (because I'm running python3.5 on my system)
cd llvm-project && mkdir -p build && cd build
Ran cmake -GNinja \ -DCMAKE_BUILD_TYPE=Release \ -DLLVM_ENABLE_PROJECTS=clang \ -DLLVM_TARGETS_TO_BUILD=X86 \ -DLLVM_PARALLEL_LINK_JOBS=8 \ -DLLVM_BUILD_TOOLS=OFF \ -DLLVM_BUILD_UTILS=OFF  \ -DCMAKE_CXX_FLAGS_RELEASE="-O0" \ -DCLANG_ENABLE_STATIC_ANALYZER=OFF  \ -DCLANG_ENABLE_ARCMT=OFF \ ../llvm/ to test building locally first.
Ran ninja build. Build successfully completed in roughly 20 mins.
Deleted the build dir, created a new one and cd-ed into it.
 And that's when the build bombed out with 99 errors similar to the one above (i.e., reporting 'No such file or directory' for required headers).

What am I doing wrong here? Have I configured llama incorrectly? I have tried to follow the README + this blog post.

Huh, that looks correct offhand. I can build fine on the release/8.3 branch on my desktop, so that's not the issue.
Can you try
LLAMACC_VERBOSE=1 ninja -v lib/Support/CMakeFiles/LLVMSupport.dir/AMDGPUMetadata.cpp.o

and paste the full output?  Running llamacc: invoke: exit 1 ninja: build stopped: subcommand failed.

Interesting, it's not picking up any of the headers.

Running that command by hand doesn't throw any compilation errors.

The only difference I can see, from what you've described on your blog post, is that I'm not passing the '-DLLVM_USE_LINKER=lld' argument to cmake, because I don't have lld. But then again, llvm still builds fine locally without that argument.

That command should output a list of dependencies for AMDGPUMetadata.cpp. Is it doing so, or just returning with no error and no output? If there's no output, that suggests something might be wrong with your local compiler configuration.

Yep, it is just returning with no error and no output.
I haven't really changed anything WRT my local compiler configuration, I'm on a debian buster VM. Moreover, if there was something wrong with my local compiler config, wouldn't that have broken my local LLVM build too? I was able to build fine locally, without using llama.

What if you run echo '#include <iostream>' | g++ -M -MF - -xc++ /dev/stdin -- test out your compiler's -MF with a minimal test case. Does that generate any output?

Hmm, that's strange, running that command doesn't generate any output either.

How did you get your buster VM? Is it a stock image somewhere I could easily reproduce so I can investigate? This is surprising behavior to me and smells obviously buggy, although – as you point out – the fact it can build llvm locally suggests it's mostly fine.

The VM is built using a preseeded iso that we have here in the office, and unfortunately I can't point you to a reproduceable image. Is there anything in particular that I can check out in my VM and send you outputs for?
Worst case scenario - i'll probably just try and make this work on a vm run from a standard debian stretch image.

What is your C compiler? which g++ / g++ --version?
I've never seen this issue before and am a bit stumped offhand about what might be the issue.

g++ (Debian 6.3.0-18+deb9u1) 6.3.0 20170516
Copyright (C) 2016 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

Hmm, that's not the normal buster gcc, which should be 8.x: https://packages.debian.org/buster/gcc
That looks to be an oldstable version: https://packages.debian.org/stretch/gcc-6
I'm still surprised it doesn't appear to support -M properly, but are you able to upgrade gcc and try again?

Oh gosh, I just grabbed a docker image with that version of gcc, and it seems that -MF - on that version writes to the literal file -, instead of stdout. I can easily work around that.

Thanks for the fix @nelhage ! I am now able to build using llama without any difficulties :)


request: React custom Provider for discord options

Is your feature request related to a problem? Please describe.
Managing profiles would be easier and more intuitive with React Providers
Describe the solution you'd like
<DiscordOptionsContext.Provider value={discordOptions}>
  <DiscordMessages>
    <DiscordMessage>...</DiscordMessage>
  </DiscordMessages>
  <DiscordMessages>
    <DiscordMessage>...</DiscordMessage>
  </DiscordMessages>
</DiscordOptionsContext.Provider>

Reference : react-discord-message by Danktuary
Describe alternatives you've considered
Custom components, but that feels hacky.
Additional context

This is not possible because the React code is fully autogenerated by Stencil (unlike the referenced package) so we cannot add custom code, especially not without jumping through major hoops of fire that would make the build process extremely complex.

Closing this as per the answer above


Don't make me write a slug

Expected behavior:

Slug should autopopulate based on title entered in previous field
Slug field should remain editable in the event we want to rewrite it.
Slug should update if the title field is changed in pre-publish state
Slug field should not auto update if the title field is changed in post-publish state

I think we should push the theme field up above the slug, and make the slug autopopulate based on the title and theme... so sbnation-my-fancy-graphic


handlers.go No metrics for pod

heapster: 1.3.0-beta.0
kubernetes: 1.5.2
docker: 1.12.6
OS: ubuntu 14.04
I'm trying to use horizontal autoscaling feature for my logstash processing containers.
Deployment definition:
https://gist.github.com/r0bj/c787265c932ba8a1de1467cebd2a906d
So I run:
kubectl autoscale deployment logstash --cpu-percent=50 --min=10 --max=30
kube-controller-manager logs:
W0119 13:54:06.641342       1 horizontal.go:104] Failed to reconcile logstash: failed to compute desired number of replicas based on CPU utilization for Deployment/ops/logstash: failed to get CPU utilization: unable to get metrics for resource cpu: no metrics returned from heapster

heapster logs:
I0119 13:48:36.675011       1 handlers.go:265] No metrics for container logstash in pod ops/logstash-3396208917-6lzxt
I0119 13:48:36.675017       1 handlers.go:216] No metrics for pod ops/logstash-3396208917-6lzxt
I0119 13:48:36.675025       1 handlers.go:265] No metrics for container logstash in pod ops/logstash-3396208917-3j8sj
I0119 13:48:36.675029       1 handlers.go:216] No metrics for pod ops/logstash-3396208917-3j8sj
I0119 13:48:36.675037       1 handlers.go:265] No metrics for container logstash in pod ops/logstash-3396208917-92f96
I0119 13:48:36.675041       1 handlers.go:216] No metrics for pod ops/logstash-3396208917-92f96
I0119 13:49:06.636598       1 handlers.go:216] No metrics for pod ops/logstash-3396208917-jll9j
I0119 13:49:06.636606       1 handlers.go:216] No metrics for pod ops/logstash-3396208917-3j8sj
I0119 13:49:06.636613       1 handlers.go:216] No metrics for pod ops/logstash-3396208917-7zt2j

Hi @r0bj ,
did you find a solution?  I'm stuck with same.. heapster same version, but on k8s 1.5.1

@Dmitry1987 unfortunately no, I didn't find any solution for that.

What log level was Heapster running at?  Can you run at a higher one?  Did you see any messages about skipping rate calculations in your logs?  How about other errors?

Do you have thin_ls enabled?

I think it's because Rancher didn't installed cAdvisor on nodes... some post says they use "docker stats" directly for the dashboard, and 'removed cAdvisor' from installation since Rancher version 1.1 or something like that ...  will try tomorrow to setup cAdvisor sidecar on all nodes and see if it helps.

yeah, I'm not sure how rancher works, but basically, Heapster gets its metrics via the Kubelet stats summary API.  Kubelet derives these stats via a built-in copy of cAdvisor, so I'm not sure how much good a sidecar cAdvisor will do.  Does Rancher just not support the HPA at all?

@DirectXMan12  seems like they don't, because my current setup is the "out of the box" latest Rancher 1.3 with latest Kubernetes from it's catalog (Rancher creates K8s infra containers from a template), and Heapster+grafana+influxDB  stack from catalog doesn't show/collect any pod stats. Grafana shows "node" stats though. so i'm trying to understand now how to make heapster receive it's needed data.
Heapster log shows this non stop:
listers.go:68] can not retrieve list of objects using index : object has no meta: object does not implement the Object interfaces.
handlers.go:216] No metrics for pod xxxxxxxxxxx
handlers.go:265] No metrics for container   xxxxxxxxxx
handlers.go:216] No metrics for pod xxxxxxxxxxx
handlers.go:265] No metrics for container   xxxxxxxxxx

and funny thing is, the pods it says "no metric from" are the 2 pods I applied HPA on their deployment. it doesn't seem to even try to fetch info about other pods, which is weird. Will be reading about how all this works...
using  "kubernetes/heapster:canary" version.
will be glad for any advice or idea about what it might be  :)

@DirectXMan12 yes I can see some skipping rate calculations in logs with --vmodule=*=4:
I0209 12:54:35.152486       1 rate_calculator.go:51] Skipping rates for namespace:ops/pod:logstash-1576143268-3xjwl/container:logstash - different create time new:2017-02-08 18:58:30.485800088 +0000 UTC  old:0001-01-01 00:00:00 +0000 UTC

More logs with --vmodule=*=4:
https://gist.github.com/r0bj/5b89014cd7dd3f37cb70fa89f1ab6e8a

I'm not using Rancher, just bare metal kubernetes installation.

and funny thing is, the pods it says "no metric from" are the 2 pods I applied HPA on their deployment. it doesn't seem to even try to fetch info about other pods, which is weird. Will be reading about how all this works...

Yeah, that message is actually logged from the API handlers when they can't find the info in the in-memory cache.

. so i'm trying to understand now how to make heapster receive it's needed data.
Heapster log shows this non stop:

If the Kubelet doesn't serve the summary API, there's not much we can do.  If the Kubelet does serve the summary API, then we have something to work with.  Is the Rancher Kubelet serving the summary API?

@r0bj
W0209 12:54:37.625232 1 listers.go:68] can not retrieve list of objects using index : object has no meta: object does not implement the Object interfaces
If I had to guess, there's a version mismatch in your cluster, or something else is up -- the "skipping rate calculation" message you posted is generally caused by stub pod entries.

For me this was solved with Rancher 1.4.0  which brings new "stack" of K8s with it... after I updated the K8s components, suddenly Kubelets started collecting data and heapster filled influxDB with all metrics, now I have everything in Grafana :+1:  as expected.  Seems like Rancher devs put back the cAdvisor (in previous release it was without it, I've read on their forum).
but still sometimes I see the  "can not retrieve list of objects using index : object has no ....."  errors logged, have no idea what causes this...   btw, what are "stub pods" @DirectXMan12  ?  I'm doing manual cleanup of any exited containers on hosts, and manual cleanup of unused images, can this cause such issue?

btw, what are "stub pods" @DirectXMan12 ?

Stub pods are fake pod entries created when Heapster has some data from one source, but not from another.  They cause all sorts of issues, but you usually don't see them with the summary API

I'm doing manual cleanup of any exited containers on hosts, and manual cleanup of unused images, can this cause such issue?

That maybe could cause issues... not sure.  Is the built-in Kube cleanup not working sufficiently?  Kube should do that stuff automatically as necessary.

oh, i totally forgot I've read somewhere about kubelet cleaning the host (but only when gets full, to thresholds, right?).... i need to remove my manual cleaner cron job for sure. thanks :)

@DirectXMan12 In my case I don't do any manual cleanup. All k8s components are in version 1.5.2, docker is 1.12.6.

indeed,@Dmitry1987,i have been stuck this issue before and discover the 'broken metrics' matter about rancher-k8s(version1.1 or higher).The reason is just for the state collection method which via docker stat and container stat,but for heapster the more appropriate choice is cAdvisor(summary api for more efficient approach).after upgrade the rancher version deployed recently(v1.4 or v1.4.1),you can find the "lost pod* and container* metrics" in your back-end storage(e.g. influxdb) which will get the same response when invoke the API.
what is worth mentioning,if i just upgrade the k8s components,"no metrics" errors arised.But cleaning the etcd cluster will vanish this.So i make a guess at the metrics mechanism,etcd maybe not renovate if you just update component.
good luck ^_^

Thanks @aeoluswing ,
as I mentioned, in rancher 1.4 the data is being collected properly, and Heapster+Grafan+InfluxDB stack from catalog works well. Regarding those logs:
can not retrieve list of objects using index : object has no .....

I always see them.  Do you mean a redeploy/cleanup of etcd helps get rid of those?  cleanup sounds a bit frightening, as all K8s data stored in there :D  , what did you do when cleaning up etcd?

@Dmitry1987,etcd records the registry kv info for k8s,my step just trigger this procedure,i suggest a data validation for cadvisor collecting metrics configurations,as we all known,cadvisor can collect any structured information(including name of metrics and regexps).Make sure cAdvisor application-metric labels(prefix is io.cadvisor.metric) contain the pod and container fields(i think these might be more easier than cleanup for prod)

Interesting... though I have no idea where cAdvisor hiding in those modified Rancher K8s images (they recompile some of the components, and put them all into custom special docker images), but it's worth to research. Thanks for the info @aeoluswing  :+1:

My pleasure.Capturing different structured metrics is a very helpful for cluster research,especially on cluster performance and schedule.It's a  a piece of cake for discussion on any topic about k8s,docker and rancher.haha^_^

Thanks @DirectXMan12

I0209 12:54:35.152486       1 rate_calculator.go:51] Skipping rates for namespace:ops/pod:logstash-1576143268-3xjwl/container:logstash - different create time new:2017-02-08 18:58:30.485800088 +0000 UTC  old:0001-01-01 00:00:00 +0000 UTC

As you mentioned, the "skipping rate" message is caused by a version mismatch in the cluster, do you know witch version is mismated, the heapster vs. k8s or k8s vs. etcd?
And in this message the old create time is "0001-01-01 00:00:00 +0000 UTC", looks like not set.
According to my understanding, the metric data is retrieved from cadvisor, witch is embed in kubelet, are there any wrong with kubelet?

As you mentioned, the "skipping rate" message is caused by a version mismatch in the cluster, do you know witch version is mismated, the heapster vs. k8s or k8s vs. etcd?

I was referring to heapster vs k8s.

And in this message the old create time is "0001-01-01 00:00:00 +0000 UTC", looks like not set.
According to my understanding, the metric data is retrieved from cadvisor, witch is embed in kubelet, are there any wrong with kubelet?

That happens usually when heapster has to create a placeholder entry because it doesn't have some data when it needs it.

I have met this issue with k8s version 1.6.3, the kubectl get hpa shows like
#kubectl get hpa
NAME         REFERENCE               TARGETS    MINPODS   MAXPODS   REPLICAS   AGE
php-apache   Deployment/php-apache   <UNKNOWN> / 50%   1         10        1          19h

the logs of heapster shows like
I0526 05:52:59.974002       1 handlers.go:264] No metrics for container php-apache in pod default/php-apache-3933604115-ssq2r
I0526 05:52:59.974789       1 handlers.go:215] No metrics for pod default/php-apache-3933604115-ssq2r
I0526 05:53:29.980479       1 handlers.go:215] No metrics for pod default/php-apache-3933604115-ssq2r
I0526 05:53:59.987180       1 handlers.go:215] No metrics for pod default/php-apache-3933604115-ssq2r

the php-apache  is
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: php-apache
spec:
  replicas: 1
  template:
    metadata:
      labels:
        run: php-apache
    spec:
      containers:
      - name: php-apache
        image: gcr.io/google_containers/hpa-example
        imagePullPolicy: IfNotPresent
        ports:
          - containerPort: 80

the hpa logs like

I0526 14:17:28.287910   28262 horizontal.go:515] Successfully updated status for php-apache
E0526 14:17:28.288323   28262 horizontal.go:201] failed to compute desired number of replicas based on listed metrics for Deployment/default/php-apache: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from heapster
I0526 14:17:58.282302   28262 horizontal.go:515] Successfully updated status for php-apache
E0526 14:17:58.282334   28262 horizontal.go:201] failed to compute desired number of replicas based on listed metrics for Deployment/default/php-apache: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from heapster

then I add resource quta in php-apache by referring #6293, it works fine.
new php-apache looks like:
#cat php-apche.yml
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: php-apache
spec:
  replicas: 1
  template:
    metadata:
      labels:
        run: php-apache
    spec:
      containers:
      - name: php-apache
        image: gcr.io/google_containers/hpa-example
        imagePullPolicy: IfNotPresent
        ports:
          - containerPort: 80
        resources:
          limits:
            cpu: 200m
            memory: 400Mi
          requests:
            cpu: 100m
            memory: 200Mi

I guess since k8s version 1.6,  hpa support custom metrics more than cpu and memory,  k8s may not set default metric, so we must manually write resource quta in deployment

@jeremywu0127 ,  you should verify your Heapster instance can reach all nodes on needed ports and that other components can reach it back (500 and 9000 something, please check in heapster docs...)  verify the kubelet ports are open to access for Heapster. I had a very similar scenario, reason was Heapster not reaching all servers (as a test - open all traffic between server where Heapster hosted and few other nodes with pods, to see if you'll suddenly have HPA working for those pods).
hope it helps.

@Dmitry1987 sorry for late response. heapster and other components (kubelet and kube-proxy) run on same node, I manually run curl, it looks like
#curl http://<IP_ADDRESS>:8082/apis/metrics/v1alpha1/namespaces/default/pods?labelSelector=run%3Dphp-apache
{
  "metadata": {},
  "items": null
 }

seems heapster can reach each other component, any ideas about the problem?

mm.. no idea what else might cause this, especially on same host o_O really strange.. please share if you solve it :)

@jeremywu0127  can you try switching to different versions of heapster? maybe something with the schema changes... btw regarding RBAC, what if the policies are not configured properly and block 'something somewhere'?  (I'm getting lost in the rising complexity of k8s with each new version, still can't get my head around all aspects related to RBAC.. but worth to check this direction in troubleshooting... just a guess)

Hi, I have the same problem on GKE cluster. After updating to 1.6.2 version I can't see metrics.

@Dmitry1987 I add resources request it works. manifests like the following
before:
      containers:
        - name: netacuity
          image: myhub.com/netacuity:latest
          imagePullPolicy: Always

now:
      containers:
        - name: netacuity
          image: myhub.com/netacuity:latest
          imagePullPolicy: Always
          resources:
            requests:
              cpu: 100m
              memory: 50Mi

seems that it must specify the request resources of a container, then hpa and heapster  work

HPA specifically needs requests.  This is documented in the HPA documentation.  Simply accessing the metrics from Heapster should not require requests to be set.

Hey guys, thanks for reporting back, it's a useful info :+1:

heapster: 1.4.0-beta.0
kubernetes: 1.6.4
docker: 1.12.6
OS: ubuntu 14.04
I0627 14:14:55.203314       1 rate_calculator.go:53] Skipping rates for namespace:ops/pod:logstash-1163116248-c1gnc - different create time new:2017-06-07 09:13:42.843665465 +0000 UTC  old:0001-01-01 00:00:00 +0000 UTC
I0627 14:14:55.203350       1 rate_calculator.go:53] Skipping rates for namespace:prod/pod:helios-486069052-cpdfq/container:helios - different create time new:2017-06-20 13:40:54.279525385 +0000 UTC  old:0001-01-01 00:00:00 +0000 UTC
I0627 14:14:55.203367       1 rate_calculator.go:53] Skipping rates for namespace:logs-ingress/pod:fluentd-listener-3760655665-x7b0k/container:fluentd-listener - different create time new:2017-05-18 09:49:33.437557364 +0000 UTC  old:0001-01-01 00:00:00 +0000 UTC
I0627 14:14:55.203462       1 pod_based_enricher.go:129] Container namespace:ops/pod:celery-mediawiki-main-4064416888-zkwqz/container:mediawiki-main not found, creating a stub
I0627 14:14:55.203545       1 pod_based_enricher.go:100] Pod namespace:kube-system/pod:calico-node-xv682 not found, creating a stub
I0627 14:14:55.203579       1 pod_based_enricher.go:129] Container namespace:kube-system/pod:calico-node-xv682/container:install-cni not found, creating a stub
[...]
I0627 14:14:55.213923       1 node_aggregator.go:44] No metric for node node:k8s-worker-s11, cannot perform node level aggregation.
I0627 14:14:55.213941       1 node_aggregator.go:44] No metric for node node:k8s-controller-s3, cannot perform node level aggregation.
I0627 14:14:55.213953       1 node_aggregator.go:44] No metric for node node:k8s-worker-s12, cannot perform node level aggregation.

More logs:
https://gist.github.com/r0bj/29cb420a761f0dd71d29887e4994d200
Horizontal pod autoscaling fails with this issue.
Is there anything I can do to debug it deeper?

@r0bj which source are you using? --sink=kubernetes or --sink=kubernetes.summary_api?

Ok, so first try with --source=kubernetes.summary_api:https://kubernetes.default.svc.  If that doesn't work, check if you can you manually curl the summary API endpoint from the Heapster pod.  The summary API endpoint is at https://$NODE:10255/stats/summary.

@DirectXMan12 thanks for the hint, so now messages are little different:
I0629 09:06:25.582615       1 rate_calculator.go:49] Skipping rate calculations for namespace:kube-system/pod:kube-proxy-247jk - new batch (2017-06-29 09:06:14 +0000 UTC) was not scraped strictly after old batch (2017-06-29 09:06:14 +0000 UTC)
[...]
I0629 09:06:29.689475       1 handlers.go:215] No metrics for pod fandom/upstream-prod-1277166791-ghcjp

Full log:
https://gist.github.com/r0bj/07e4854728862e5cf1ef1c13b9102aa7
So instead of: different create time new:[...]  old:0001-01-01 00:00:00 +0000 UTC there is now: new batch ([...]) was not scraped strictly after old batch ([...])
kubelet summary API endpoint seems working fine:
# docker ps -qf "label=io.kubernetes.container.name=heapster" | xargs docker inspect --format '{{.State.Pid}}'
14691
# nsenter -t 14691 -n curl http://k8s-worker-s10:10255/stats/summary
{
  "node": {
   "nodeName": "k8s-worker-s10",
   "systemContainers": [
    {
     "name": "kubelet",
     "startTime": "2017-06-27T22:12:25Z",
     "cpu": {
      "time": "2017-06-29T09:22:24Z",
      "usageNanoCores":<PHONE_NUMBER>,
      "usageCoreNanoSeconds":<PHONE_NUMBER>4362565
     },
     "memory": {
      "time": "2017-06-29T09:22:24Z",
      "availableBytes":<PHONE_NUMBER>6,
      "usageBytes":<PHONE_NUMBER>0,
      "workingSetBytes":<PHONE_NUMBER>2,
      "rssBytes": 502865920,
      "pageFaults":<PHONE_NUMBER>8,
      "majorPageFaults": 37805
     },
     "userDefinedMetrics": null
    },
[...]

Full stats summary:
https://gist.github.com/r0bj/72049ac2ca2d2e02f62b2c44127b0143
Stats for other nodes are also available from heapster pod.

so, I'm guessing you're hitting a bug where your Heapster collection interval is not always larger than the cAdvisor collection interval.  What's your Heapster collection interval?  Default?  We had a bug a while back where thin_ls stats were causing issues, but that should have gotten merged a while ago...

hmm... you're using 1.5.2... that doesn't have the patch for disabling thin_ls stats...
try bind-mounting /dev/null over the thin_ls binary and see if that fixes your issue.

I've used hepaster with --metric-resolution=10s. Now I've switched to --metric-resolution=20s and it seems that there is no issue.
I've done some digging and it seems that heapster polling interval should be > 2*
polling duration of cadvisor:
https://github.com/kubernetes/kubernetes/issues/18044#issuecomment-161435365
And cadvisor polling interval is currently hardcoded to 10s:
https://github.com/kubernetes/kubernetes/pull/19374
So the minimal heapster polling interval can be 20s.
@DirectXMan12 Does it make any sense to you?

I really need to put this in a document somewhere:
With a set collection interval of 10s in cAdvisor, the actual cAdvisor collection interval can be up to 30s, due to jitter and backoff mechanisms.  Your Heapster collection interval should be no lower than 30s.

@r0bj out of curiosity why do you need lower metric-resolution?

@piosz I can live with 30s interval of heapster polling, I just use 10s resolution in my metrics infrastructure so it was natural idea.
Anyway, thanks for help!

closing since the issue seems to be resolved.

@r0bj Does adding --housekeeping-interval=5s flag in kubelet work? I tried set this flag and metric-resolution set 15s, but still got no metrics returned from heapster error.  If changed metric-resolution to 20s, the issue is gone. Seams the flag housekeeping-interval set in kubelet not work. I restart the kubelet and had seen the flag with systemctl status kubelet.


Net Core samples?

have any Net Core sample to show?
I try to run in Net Core 2.0 but is not working.

@albsilva ,  I've been doing some work with .Net Core 2.0.  I found that I needed to add the pre-release versions of the NuGet packages the documentation mentions.  Mine are:

Microsoft.TeamFoundationServer.Client (15.126.0-preview)
Microsoft.VisualStudio.Services.Client (15.126.0-preview)
Microsoft.VisualStudio.Services.InteractiveClient (15.126.0-preview)

When I switched to those, a lot of the using statements referenced by their libraries stopped throwing exceptions.  Good luck.

@MarkIannucci do you have any sample of a csproj with those nuget packages? I've been trying to get those working with a .net core 2.0 project but no luck on getting those references working.
Thanks

@tiagosalgado see below:
<Project Sdk="Microsoft.NET.Sdk.Web">

  <PropertyGroup>
    <TargetFramework>netcoreapp2.0</TargetFramework>
    <ApplicationInsightsResourceId>secret</ApplicationInsightsResourceId>
    <ApplicationInsightsAnnotationResourceId>another secret</ApplicationInsightsAnnotationResourceId>
  </PropertyGroup>

  <ItemGroup>
    <Folder Include="wwwroot\" />
  </ItemGroup>

  <ItemGroup>
    <PackageReference Include="Microsoft.ApplicationInsights.AspNetCore" Version="2.2.1" />
    <PackageReference Include="Microsoft.ApplicationInsights.SnapshotCollector" Version="1.2.1-pre-20180313.2" />
    <PackageReference Include="Microsoft.AspNetCore.All" Version="2.0.6" />
    <PackageReference Include="Microsoft.TeamFoundationServer.Client" Version="15.126.0-preview" />
    <PackageReference Include="Microsoft.VisualStudio.Services.Client" Version="15.126.0-preview" />
    <PackageReference Include="Microsoft.VisualStudio.Services.InteractiveClient" Version="15.126.0-preview" />
    <PackageReference Include="Microsoft.VisualStudio.Web.CodeGeneration.Design" Version="2.0.3" />
    <PackageReference Include="Pdb2Pdb" Version="1.1.0-beta1-62316-01" />
  </ItemGroup>

  <ItemGroup>
    <DotNetCliToolReference Include="Microsoft.VisualStudio.Web.CodeGeneration.Tools" Version="2.0.1" />
  </ItemGroup>

  <ItemGroup>
    <Content Update="appsettings.json">
      <CopyToOutputDirectory>Always</CopyToOutputDirectory>
    </Content>
  </ItemGroup>

  <ItemGroup>
    <WCFMetadata Include="Connected Services" />
  </ItemGroup>

  <Import Project="shared project" Label="Shared" />

  <Target Name="PreBuild" BeforeTargets="PreBuildEvent">
	<Exec Command="COPY /Y $(ProjectDir)..\appsettings.json $(ProjectDir) & IF EXIST $(ProjectDir)appsettings.*.json (COPY /Y $(ProjectDir)appsettings.*.json $(TargetDir))" />
  </Target>

</Project>

Do you get all those nuget packages referenced properly? No matter what I do, I get

I get the same even using the last preview versions (15.131.0-preview).

Mine also have the yellow triangles and I get a few NU1701 warnings saying that package may not be fully compatible with my project.
For example:
Warning	NU1701	Package 'Microsoft.AspNet.WebApi.Client 5.2.2' was restored using '.NETFramework,Version=v4.6.1' instead of the project target framework '.NETCoreApp,Version=v2.0'. This package may not be fully compatible with your project.
I think I've been able to handle the potential deployment issues because the VMs that are running this also have .Net 4.6.1 installed...

My problem is before even deploying anywhere.

Guess until the client libraries are fully supported by .net core, will face this issue.

I would agree it was a duplicate.  I would just love a status/state of where things are for a full .Net Core version?  Or at least a guestimate of when it will be released.  Thank you.

It looks like v16 supports .NET Standard 2.0:
16.143.1
Might be possible now!


added basic dashboard with most of the requirements mentioned in the issue

I have used static data inside the flask app instead of a database.

hey @i-am-waseem pls let me know whether it is the right path i am going.


⚠ Quirk Club Argentina (API endpoint) has degraded performance

In 40640af, Quirk Club Argentina (API endpoint) (https://us-central1-quirkclub-dev.cloudfunctions.net/api/check/api) experienced degraded performance:

HTTP code: 200
Response time: 9334 ms

Resolved: Quirk Club Argentina (API endpoint) performance has improved in a018855.


 At first glance - taking my hat of very naive end-user:

Where is my app repo in this flow? App repo is missing
Need for IBM Cloud and region box?
What are the COS bucket, SM, SCC icons for at the bottom - how do the relate to the pipelines?
Crossing arrows
What are the arrows representing? References? Flow of data?
What is the code risk analyzer icon for? Is this a service
What are evidences? Issue tracking? Repo artifacts?
How does devops insight fit in the flow here - what value does it provide?
Where is my cluster / code engine project in this diagram?

We are working with Design on an updated Diagram.


WIP: Try to lint paket

Mhm doesn't work on mono!?
/cc @duckmatt

I had the same problem linting FSharpLint itself, the solution was to add the FSharpLint directory as an assembly search path: #I @"packages/FSharpLint/"

thanks. Now I get https://travis-ci.org/fsprojects/Paket/builds/49967776#L1924
The file exists but it's a relative path and not found.

It seems like the paths in MSBuild are getting set relative to where FSharpLint is being run from (build.fsx) so <SolutionDir Condition="$(SolutionDir) == '' Or $(SolutionDir) == '*Undefined*'">..\..\</SolutionDir> is setting up $(SolutionDir) to /home/travis/build/fsprojects/Paket/../../
A quick fix would be to add a condition to the import so that it's not included if it can't be found:
<Import Project="$(SolutionDir)\.paket\paket.targets" Condition="Exists('$(SolutionDir)\.paket\paket.targets')" />

I'll add an issue to FSharpLint to have a look into getting the MSBuild paths to be relative to the .fsproj files

It should hopefully now work without that quick fix in 0.1.15

and we have a new strange error: https://travis-ci.org/fsprojects/Paket/builds/50064698#L1897

I see there's also problems on the windows build, I'll clone the repo and get it running correctly for 0.1.16. Thanks for letting me know about these issues, I think this is the first time anyone has tried to integrate it into a large project other than it linting itself, so this will be a good test

:shipit:

0.1.16 runs successfully on Paket now: https://github.com/duckmatt/Paket/tree/Lint
Might be worth turning off MaxCharactersOnLine as that rule is generating a lot of errors - it's one I've been considering turning off by default. Let me know if there's anything I can do to help and if there's any changes you'd like to see to the linter

ok I'll close this one. If you are interested please send your own PR with suggested fixes for Paket.


Logging

Logging to be added throughout, for every function writing to a different log file for each script. e.g. database.py would log to ../logs/database_log.txt.

I think I can help you out with that!
Just a question: do the log files should be rewritten every time the program starts?

I think the logging should come as part of a try, except on each function so that we can log anything that occurs each time a function is run.
e.g.
def do_something(something):
try:
do something()
logging.info('I did something')
except:
logging.critical('Do something broke')
That way as the program improves we can catch any errors, know specifically what they are and where they occur.
If you can think of any other use for logging in the program feel free to suggest it.

"image found at", "image url", "downloaded" in the for loop can be logged, users need to know those details. What do you think?

@Algogator I agree, I don't they do. The GUI overhall I'm doing has a progress bar instead which I think is a bit better, I will check and merge your PR when I get a chance 👍


Rename konnectd to idp

I rebased, because GitHub said "out of date"

I rebased, because GitHub said "out of date"