Compilation "warnings"

After a week on BLEClient testing constant reconnects by forcing disconnects every 7 seconds to simulate reconnects over time, BLEClient would always stop working after several reconnects, leaving the loop() locked.
As simple as NimBLE was to "adapt" over, I just changed the header, and my code still worked as is.
//#include "BLEDevice.h" #include "NimBLEDevice.h"
Now running the reconnect tests, surprisingly so far it is not stopping... Still gonna run that overtime to see if it behaves any different over more reconnects.
Anyways, during compilation with NimBle, I get these errors:
C:\Users\Tony\Documents\Arduino\ESP32_BTLE_MORIMOTO-2\BLE_DATA.ino: In function 'void auto_connect_ble()': C:\Users\Tony\Documents\Arduino\ESP32_BTLE_MORIMOTO-2\BLE_DATA.ino:41:15: warning: unused variable 'r' [-Wunused-variable]        uint8_t r = random(1, 255);                ^ C:\Users\Tony\Documents\Arduino\ESP32_BTLE_MORIMOTO-2\BLE_DATA.ino:42:15: warning: unused variable 'g' [-Wunused-variable]        uint8_t g = random(1, 255);                ^ C:\Users\Tony\Documents\Arduino\ESP32_BTLE_MORIMOTO-2\BLE_DATA.ino:43:15: warning: unused variable 'b' [-Wunused-variable]        uint8_t b = random(1, 255);                ^ C:\Users\Tony\Documents\Arduino\ESP32_BTLE_MORIMOTO-2\XBT_Controls.ino: In function 'void XBT_power(int, bool)': C:\Users\Tony\Documents\Arduino\ESP32_BTLE_MORIMOTO-2\XBT_Controls.ino:3:137: warning: ISO C++ says that these are ambiguous, even though the worst conversion for the first is better than the worst conversion for the second:      if ( XBT_client[controller]->isConnected() ) XBT_characteristic[controller]->writeValue(controller_switch, sizeof(controller_switch));                                                                                ^ In file included from C:\Users\Tony\Documents\Arduino\libraries\NimBLE-Arduino\src/NimBLERemoteService.h:25:0,                  from C:\Users\Tony\Documents\Arduino\libraries\NimBLE-Arduino\src/NimBLEClient.h:27,                  from C:\Users\Tony\Documents\Arduino\libraries\NimBLE-Arduino\src/NimBLEDevice.h:31,                  from C:\Users\Tony\Documents\Arduino\ESP32_BTLE_MORIMOTO-2\ESP32_BTLE_MORIMOTO-2.ino:13: C:\Users\Tony\Documents\Arduino\libraries\NimBLE-Arduino\src/NimBLERemoteCharacteristic.h:115:52: note: candidate 1: bool NimBLERemoteCharacteristic::writeValue(const uint8_t*, size_t, bool)      bool                                           writeValue(const uint8_t* data,                                                     ^ C:\Users\Tony\Documents\Arduino\libraries\NimBLE-Arduino\src/NimBLERemoteCharacteristic.h:126:10: note: candidate 2: bool NimBLERemoteCharacteristic::writeValue(const T&, bool) [with T = unsigned char [2]]      bool writeValue(const T &s, bool response = false) {           ^ C:\Users\Tony\Documents\Arduino\ESP32_BTLE_MORIMOTO-2\XBT_Controls.ino: In function 'void XBT_setColor(int, uint8_t, uint8_t, uint8_t, int, int, int)': C:\Users\Tony\Documents\Arduino\ESP32_BTLE_MORIMOTO-2\XBT_Controls.ino:43:147: warning: ISO C++ says that these are ambiguous, even though the worst conversion for the first is better than the worst conversion for the second:      if ( XBT_client[controller]->isConnected() ) XBT_characteristic[controller]->writeValue(XBT_colors[controller], sizeof(XBT_colors[controller]));                                                                                ^ In file included from C:\Users\Tony\Documents\Arduino\libraries\NimBLE-Arduino\src/NimBLERemoteService.h:25:0,                  from C:\Users\Tony\Documents\Arduino\libraries\NimBLE-Arduino\src/NimBLEClient.h:27,                  from C:\Users\Tony\Documents\Arduino\libraries\NimBLE-Arduino\src/NimBLEDevice.h:31,                  from C:\Users\Tony\Documents\Arduino\ESP32_BTLE_MORIMOTO-2\ESP32_BTLE_MORIMOTO-2.ino:13: C:\Users\Tony\Documents\Arduino\libraries\NimBLE-Arduino\src/NimBLERemoteCharacteristic.h:115:52: note: candidate 1: bool NimBLERemoteCharacteristic::writeValue(const uint8_t*, size_t, bool)      bool                                           writeValue(const uint8_t* data,                                                     ^ C:\Users\Tony\Documents\Arduino\libraries\NimBLE-Arduino\src/NimBLERemoteCharacteristic.h:126:10: note: candidate 2: bool NimBLERemoteCharacteristic::writeValue(const T&, bool) [with T = unsigned char [20]]      bool writeValue(const T &s, bool response = false) {           ^ C:\Users\Tony\Documents\Arduino\ESP32_BTLE_MORIMOTO-2\XBT_Controls.ino: In function 'void sync_XBT_controllers(int)': C:\Users\Tony\Documents\Arduino\ESP32_BTLE_MORIMOTO-2\XBT_Controls.ino:87:102: warning: ISO C++ says that these are ambiguous, even though the worst conversion for the first is better than the worst conversion for the second:      XBT_characteristic[controller]->writeValue(XBT_colors[controller], sizeof(XBT_colors[controller]));                                                                                ^ In file included from C:\Users\Tony\Documents\Arduino\libraries\NimBLE-Arduino\src/NimBLERemoteService.h:25:0,                  from C:\Users\Tony\Documents\Arduino\libraries\NimBLE-Arduino\src/NimBLEClient.h:27,                  from C:\Users\Tony\Documents\Arduino\libraries\NimBLE-Arduino\src/NimBLEDevice.h:31,                  from C:\Users\Tony\Documents\Arduino\ESP32_BTLE_MORIMOTO-2\ESP32_BTLE_MORIMOTO-2.ino:13: C:\Users\Tony\Documents\Arduino\libraries\NimBLE-Arduino\src/NimBLERemoteCharacteristic.h:115:52: note: candidate 1: bool NimBLERemoteCharacteristic::writeValue(const uint8_t*, size_t, bool)      bool                                           writeValue(const uint8_t* data,                                                     ^ C:\Users\Tony\Documents\Arduino\libraries\NimBLE-Arduino\src/NimBLERemoteCharacteristic.h:126:10: note: candidate 2: bool NimBLERemoteCharacteristic::writeValue(const T&, bool) [with T = unsigned char [20]]      bool writeValue(const T &s, bool response = false) {           ^ C:\Users\Tony\Documents\Arduino\ESP32_BTLE_MORIMOTO-2\XBT_Controls.ino: In function 'void smooth_color_switching(int, int, int, int, int, int)': C:\Users\Tony\Documents\Arduino\ESP32_BTLE_MORIMOTO-2\XBT_Controls.ino:135:151: warning: ISO C++ says that these are ambiguous, even though the worst conversion for the first is better than the worst conversion for the second:          if ( XBT_client[controller]->isConnected() ) XBT_characteristic[controller]->writeValue(XBT_colors[controller], sizeof(XBT_colors[controller]));                                                                                ^ In file included from C:\Users\Tony\Documents\Arduino\libraries\NimBLE-Arduino\src/NimBLERemoteService.h:25:0,                  from C:\Users\Tony\Documents\Arduino\libraries\NimBLE-Arduino\src/NimBLEClient.h:27,                  from C:\Users\Tony\Documents\Arduino\libraries\NimBLE-Arduino\src/NimBLEDevice.h:31,                  from C:\Users\Tony\Documents\Arduino\ESP32_BTLE_MORIMOTO-2\ESP32_BTLE_MORIMOTO-2.ino:13: C:\Users\Tony\Documents\Arduino\libraries\NimBLE-Arduino\src/NimBLERemoteCharacteristic.h:115:52: note: candidate 1: bool NimBLERemoteCharacteristic::writeValue(const uint8_t*, size_t, bool)      bool                                           writeValue(const uint8_t* data,                                                     ^ C:\Users\Tony\Documents\Arduino\libraries\NimBLE-Arduino\src/NimBLERemoteCharacteristic.h:126:10: note: candidate 2: bool NimBLERemoteCharacteristic::writeValue(const T&, bool) [with T = unsigned char [20]]      bool writeValue(const T &s, bool response = false) {           ^
Other than that, it does compile, and upload fine.
Also rocking at 238K free heap on NimBLE with 2 connections, while BLEClient was around 162K with same 2 clients connected :)
This is how i test multiple client reconnects:
 static uint32_t t = millis();   if ( millis() - t > 7000 ) {     for ( int i = 0; i < 4; i++ ) if ( XBT_client[i]->isConnected() ) XBT_client[i]->disconnect();
I also connect directly to the mac address thats programmed, I don't connect to the named device. Connections are done in loop while a volatile flag is set from the async scan callback
EDIT: Interesting, after a longer run with NimBLE, disconnect() stops working after awhile, I noticed that the clients don't cut out if i cut their power, so I added:
 for ( int i = 0; i < 4; i++ ) {     XBT_client[i]->setConnectionParams(12, 12, 0, 51);     XBT_client[i]->setConnectTimeout(3);   } to setup(); Now they drop out almost immediately if powered down, and async scan reconnects to them fine. It's weird tho that disconnect() stops working after awhile

Looks like the compiler is complaining about the data types in your arrays. You might need to manually cast these to uint8_t* to clear the warnings or just remove the sizeof(XBT_colors[controller] and the template will sort it out for you.

Interesting, after a longer run with NimBLE, disconnect() stops working after awhile, I noticed that the clients don't cut out if i cut their power

This is something that has come up recently and I have done a lot of testing. What happens is the response to disconnect from the peripheral device for one reason or another does not get received so the controller waits for the supervision timeout (+ a few ms) while waiting for the response. When that time expires it will send the disconnect event at that time. As you have seen, if you set the connection parameters with a lower supervision timeout this happens faster.

thanks, its working really well in terms of reconnects and BLE client disconnects, however I cant get getService to work with either 128 or 16 bit UUID, so ot never finalized the connection to write anything, been at it for awhile last night and put BLEClient back to make sure I didnt kill my BLE devices lol and they connect and write fine there


Icon in Text Field

How to show icon in text field as show in figure?

It seems impossible for now, but I can try to implement in the future releases.
@shukerullah, how can you achieve this with ReactNative's built-in TextInput?


Community members would make great triagers

Some repos have successfully shared triage rights with community members

dotnet/iot does this at a public triage meeting, and gave the Github triage role to one community member so far (@pgrawehr) , and reached out to another. Per @richlander this has been a good experience.

dotnet/roslyn has given a small number of community members the Github triage role (which allows add/remove labels and closing issues). These folks have helped identify duplicate issues, clarified issues, and helped route and prioritize. @jinujoseph felt this was super useful. They currently ask these triagers to not close issues unless they are duplicates.

The rustlang project uses 'triage bot' instead of assigning the Github role. We should take a look at that.
We should determine what we can learn from this.
todo -- @terrajobst will follow up with to find out whether there's general guidelines.

This is great to hear! Successful projects take input from throughout the community and the opportunity to receive triage/maintainer/etc. access is an excellent route.

@terrajobst will follow up with to find out whether there's general guidelines.

After talking to a few folks, we don't see any issues with giving externals triage permissions. In fact, we believe this is the way to build community. People who earn the trust of the project maintainers should be given more powers. In the projects we do this, we've found that this helps maintainers with the busy work and people are lurking around all the time feel their support is rewarded & makes a difference.

@333fred could you share Roslyn's experience, the approach you took and what you learned?

Sure. We've given a very small set of particularly active contributors triage access in a couple of repositories now: dotnet/csharplang, and dotnet/roslyn. The particular community members we chose (3 in csharplang, 5 in roslyn) had a long history of positive community interaction, and that we felt that they would be able to both well represent the broader .NET community and uphold the highest standards of our Code of Conduct. We also have a private teams chat for to discuss conduct and other issues that would need to be done in a private conversation, as needed.
The experience has been super positive, particularly in the csharplang repository, where I need to give special credit to @YairHalberstadt for helping us drive down the number of open issues from well over 1,000, many of which were duplicates, incomplete, or not really issues, to 420 open issues that are mostly high-enough quality for the LDT to look at. I can honestly say that the quality of the repository and the interactions with the rest of the community have markedly improved since we implemented this. In the roslyn repo, contributors have been especially helpful in routing bugs, closing duplicates, and flagging higher-priority items. This last bit, in particular, has been especially useful and is not easily done by bot systems. These contributors were often basically doing this job before we added them as triagers, and @jaredpar had expressed that often he'd just look to see what a community member had said about the bug then just do what they said to do (close as duplicate or use their feedback to decide what milestone the bug needed to go in).

@YairHalberstadt do you have any thoughts to share about the experience from the community point of view? If we were to do something similar in another large repo, would you recommend we do anything differently?

I'll also tag our other contributors @jnm2, @svick, @alrz, and @youssef1313 for their thoughts.

@danmoseley
I can't think of anything that required changing really. The most tricky part was hashing out which issues should be closed, which should be moved to discussion, and which should stay as is, but we worked that out on a case by case basis over the teams chat, and by the end I had a pretty good idea.
On a separate note the teams app did not work well for me at all, and often due to a bug in teams I couldn't access the community contributors group at all, but I understand there's not really much the dotnet team can do about that.
Finally at the moment I've recently moved jobs and now have much less free time (turns out my OSS contributions were directly proportional to how bored I was at work), and I feel embarrassed about how little I'm doing on either community. If the team wants to limit the total number of community contributors they may want to consider rotating them around as new promising candidates appear.

Of course, Teams calls with too many people won't work well or will be inefficient, but I guess that problem can be solved once it arises (maybe by separating the call into several for different topics). The dotnet/iot team is small, but even then it rarely happens that everybody can join. Most calls have only like 4 people in them (out of I think 8 that are invited).

@richlander @HongGit @zkat met, and also chatted with @eiriktsarpalis , and we had these thoughts for an experiment in dotnet/runtime:
(Richard, Kat, Hong, Dan)
What would be the value to maintainers?
1. Enabling community to feel more included/engaged and increase transparency in how we make decisions
2. Help triage be more efficient and responsive
3. (Lower priority) Building confidence and expertise in individuals eg to grow regular committers.

What would be the value to community members?
1. Making easier something they are already passionate about
2. Giving them more influence over the direction of the product and quality
3. Career building -- item on resume
4. Offering deeper access to the product team, and opportunity to build expertise.
5. Public recognition (similar to folks who offer PR's) including list on area owners page, and possibly on the release blog page under the code contributors

Triager responsibilities
1. Route issues (ie., fix area labels)
2. Ask for more info (eg., help clarify repro steps)
3. Identify related issues and duplicates (and close the obvious duplicates)
4. Ad hoc offer input into prioritization
5. Help communicate our backlog and prioritization to others in the community. (dotnet/iot repo achieve this with access to a triage group that meets Thursdays)

Triager non responsibilities
1. Make prioritization decisions (eg., set milestone or close as out of scope) although their input is welcome and encouraged and the experiment could evolve.

Maintainer reponsibilities
1. Be engaged and keep them informed and offer recognition.
2. Provide a point of contact among maintainers and reach out often (eg email, Teams, Discord etc)
3. If requested, configure the labeling bot to notify them of labels interesting to them if they wish.
4. If all parties wish, we could bring them into area owner triage calls. This is the dotnet/iot model. Note not all areas do such calls. For the responsibilities above, it shouldn't be necessary, but it may be interesting!

How to select folks
• Do we pick folks, or ask for expressions of interest. Decision: we try picking. this is what was done in dotnet/roslyn and dotnet/iot.
• How many are we looking for? We agreed to start small - say, 3 in the repo.
• We are looking for folks with (a) history of engagement in the repo one way or another (b) knowledge and interest in one or more of our areas (c) friendly! (d) willing and interested!
• Identify champ for each person - the one who proposed them, perhaps - and give some guidance about communication.

Next steps
• Draft plan, get internal consensus (M2's, Jamshed, …)
• Email internal repo contributors alias to collect suggested names (explaining responsibilities of mentor)
• Pick a small number and reach out until we have 3 agree to do it
• If requested, update notification bot.
• We don't need an announcement as this is pretty limited.

Future directions
• Would triagers specialize in an "area"? A: We assume this will happen naturally, based on interests.
• Better document what we do -- our process, our bar, how we prioritize. Although, we aren't necessarily consistent today, and we aren't asking triagers to prioritize at this point.

Risks/mitigations
• If folks lose interest/don't participate
• Creating an excluding "in group"

The rustlang project uses 'triage bot' instead of assigning the Github role. We should take a look at that.

FWIW here's an instance of a user asking for something like that to help us out.

@eiriktsarpalis perhaps worth creating an issue for a triage bot specifically? We can look around for other prior art, and consider whether to give it a trial and where, and what that would take.

3 community members have now accepted the role of triager in dotnet/runtime repo. They will each have a contact in the product team, and figuring out how things will work. Expect them to get rights within the next week and figure out parameters and announce broadly within the next couple weeks.
We'll record here what we learn and how it goes.

3 community members have accepted the triager in dotnet/wpf. Expect them to get right with next 1-2 weeks.
@gurpreet-wpf  is working on announcement and work with them to share the responsibility information.

FWIW I'm seeing considerably more activity now from dotnet/runtime community triagers compared to the initial months of the experiment. Folks tend to be cautious initially while learning the ropes so active encouragement and coaching is probably necessary even after permissions have been granted.


tracking unique redux-saga tasks

I am using sagas to track multiple async tasks, but there is one problem I haven't been able to completely solve:
function* performTask1() {
    // Some logic here to takeLatest for the relevant component

    // check if component id matches?

    // Only perform API call with the latest
    const { result } = yield takeLatest('doAsync2')
}

function* performTask2() {
    const { result } = yield call(api, args)
    // do something with results (not relevant)
}

function* watchAsyncTasks() {
    yield takeEvery('doAsync2', performTask2)
    yield takeEvery('doAsync1', performTask1)
}

componentA dispatches doAsync1

componentB dispatches doAsync1

component C dispatches doAsync2 (for good measure)

componentA dispatches doAsync1

componentB dispatches doAsync1

How can I use sagas to ensure that only sagas 3, 4, and 5 complete their API call?

hm, im not 100% sure if I understand your requirements, but from what I do it seems that

performTask2 should be performed for each doAsync2
performTask1 should be performed for latest doAsync1 but scoped per some component id which is included in the doAsync1 action

So for those requirements I've written a simple general use-case helper and rewritten your snippet using that
function* takeLatestPerProps(propsOrSelector, pattern, worker, ...args) {
	const channelsMap = {}

	while (true) {
		const action = take(pattern)
		const propsValue = typeof propsOrSelector === 'function' ? propsOrSelector(action) : action[propsOrSelector]
		if (!channelsMap[propsValue]) {
			channelsMap[propsValue] = channel()
			yield takeLatest(channelsMap[propsValue], worker, ...args)
		}
		yield put(channelsMap[propsValue], action)
	}
}

function* performTask1() {
	// already scoped per 'id'
    const { result } = yield call(api, args)
    // do something with results (not relevant)
}

function* performTask2() {
    const { result } = yield call(api, args)
    // do something with results (not relevant)
}

function* watchAsyncTasks() {
    yield takeEvery('doAsync2', performTask2)
    yield takeLatestPerProps('id', 'doAsync1', performTask1)
}

NOTE: also the form you have used in your snippet - const { result } = yield takeLatest('doAsync2') wont work as takeLatest returns a task descriptor (so it could be canceled at a later time)

you are correct in detailing the requirements. I did not think of using a helper to select by id, but this definitely works better than my solution.
I thought of using an action channel to process requests serially, but definitely prefer being able to use takeLatest, since I'd like to cancel an unfinished request if another for the same component fires.
Thank you! This was very instructive. From my understanding: this helper will create a buffer channel per id given, the yield passes the action to the proper channel, and thus concurrent doAsync1 actions can be dispatched/processed for unique components.
I will close this once I implement and finish. I have a question about how propsVal works, but I think I have enough to go on and answer my own question in the process.

Your understanding is correct - just creating new channels for not yet created ones per props (+ spawning takeLatest with the created channel passed in) and takeLatestPerProps serves as proxy for passing actions to appropriate channel and then takeLatest can do its thing.
propsValue is the unique identifier to be used for creating and storing channels in the map. Ive included obtaining it simply by accessing property of the action or by running a selector on the action with that line
const propsValue = typeof propsOrSelector === 'function' ? propsOrSelector(action) : action[propsOrSelector]

which can come handy in some situations. Its pretty similar to the concept of take(pattern) which you can use by simply passing a string in like so take(ACTION_TYPE) or take(ac => ac.type === ACTION_TYPE)

Mid-implementation, I have one question about channelsMap:
Upon completion of task at channelsMap['a'], does channelsMap['a'] clear or will I have to take care of that garbage collection?
Since this was a custom-helper function you created, I am going to assume yes (for now)
Does this mean I just have to add this line after the yielding put?
`
yield put(channelsMap[propsValue], action)
delete channelsMap[propsValue]
`

No. You dont need to do it. takeLatest will take any items in those channels and handle only the latest one with a worker generator. So those channels should have at most (for a brief moment) 1 item buffered in them, which are gonna be immediately taken from them, so there is no need for clearing them - they will clear themselves automatically (thanks to takeLatest)

Closed. I believe this works, but I am opening a separate issue for how your takeLatest unit test works. I am using it as a template for generating my own unit tests for takeLatestByProps

Arrived here some time later via Google. Had to fix a couple of things to make takeLatestPerProps work:
function takeLatestPerProps(propsOrSelector, pattern, worker, ...args) { // Not a generator
  return fork(function* () { // Fork a generator here to make it work like takeLatest
    const channelsMap = {}

    while (true) {
      const action = yield take(pattern) // yield necessary here
      const propsValue = (typeof propsOrSelector === 'function' ? propsOrSelector(action) : action[propsOrSelector]) + ''
      if (!channelsMap[propsValue]) {
        channelsMap[propsValue] = channel()
        yield takeLatest(channelsMap[propsValue], worker, ...args)
        console.log(`channelsMap: ${Object.keys(channelsMap).join(`, `)}`) // List buildup of channels
      }
      yield put(channelsMap[propsValue], action)
    }
  })
}

I have one major issue with this though. When used with a large number of different propsValues, it builds up an equal number of channels and forked tasks, and those are never removed. I added a log call to list them.
My use case is, I am loading items by ID and would like to cancel earlier requests to the same ID. Any suggestions?

What I've ended up with looks like this:
import { Action } from 'redux'
import { channel, Channel } from 'redux-saga'
import { fork, put, takeEvery, takeLatest } from 'redux-saga/effects'

interface IChannels<A extends Action> {
  [scope: string]: Channel<A>
}

/**
 * Implements `takeLatest` handling per unique value returned by `scopeSelector`.
 */
export function takeLatestPerScope<A extends Action>(
  scopeSelector: (action: A) => string, pattern: A['type'], worker: (action: A) => void,
) {
  return fork(function*() {
    const channels: IChannels<A> = {}

    yield takeEvery(pattern, function*(action: A) {
      const scope = scopeSelector(action)
      if (!channels[scope]) {
        channels[scope] = channel()
        yield takeLatest(channels[scope], worker)
      }
      yield put(channels[scope], action)
    })
  })
}

What is notably different from @andraaspar and @Andarist's examples is my use of takeEvery.
The while block + const action = yield take(pattern) approach in their examples misses the dispatching of an action matching pattern that occurs during the execution of the rest of the while block; the action is missed because we haven't gotten back to blocking on yield take(pattern).
In my case, I was using yield all([ put({ type: FOO, scope: 1 }), put({ type: FOO, scope: 2 }) ]) to dispatch two of the same action for parallel execution. The action with scope 2 was never being captured by take.
Using takeEvery solves this problem.

Regarding one major issue mentioned by @andraaspar here are my comments.


Support for TypeScript 1.0

Doesn't work under Typescript 1.0;  There are several compile-time issues; some are obvious to fix, however, there appears to be an issue with the DSTree where dequeue returns a Tand in one case a BSTreeNode is expected.

Works fine at the moment. :rose:


more api renaming

yeah I agree martin here... I also think domain makes more sense for some instinctive feeling

My only concern is that people think of domain in terms of DNS.

i get that, my problem with key is that everyone i described it too thought of it as a key-value thingie when i say key they expect a value.... lets thinks about this more

My problem is that key is not specific enough. With domain you can actually encompass a wider meaning. A key to me is something you use in a key-value store to retrieve (single) items, whereas a domain is something you can use to retrieve a whole domain of keys. Domain adds one more layer of granularity.

But yeah, I agree with seiflotfy and with you mbarkhau, let's think about this more thoroughly...

did you consider leveldb since its part of go now, which means no external dependency?

@seiflotfy will this be a problem when using leveldb?

Only a single process (possibly multi-threaded) can access a particular database at a time.

@mbarkhau, @seiflotfy: regarding the naming problem on whether we call it key or domain, why not just call it for what it is, a "data domain"? I think this terminology makes much sense. In this way we don't need to rename our variables, we just should explain it in our README.md. This is also the official terminology by Flajolet et al.

I agree but found the word data redundant since all we work with is data. Thus I thought domain makes sense :D
—
Sent from Mailbox
On Fri, Aug 14, 2015 at 10:56 AM, Martin Pinto-Bazurco
<EMAIL_ADDRESS>wrote:

@mbarkhau, @seiflotfy: regarding the naming problem on whether we call it key or domain, why not just call it for what it is, a "data domain"? I think this terminology makes much sense. In this way we don't need to rename our variables, we just should explain it in our README.md. This is also the official terminology by Flajolet et al.
Reply to this email directly or view it on GitHub:
https://github.com/seiflotfy/counts/pull/19#issuecomment-131030207

yeah of course, we only need to describe it in our README.md and that's all


Redundancy in Sort and OrderBY Operation

We currently have two subclasses of the Operation class which both handle the sorting of an input, called Sort and OrderBy.
One of them is used for an ordering that was specified by the user via an ORDER BY clause in the query, and the other is used for orderings that are automatically inserted for technical reasons (e.g. because some operations require a sorted input).
Since both classes provide the exact same functionality, there is a lot of code duplication here and we should get rid of one of them for better maintainability.

This is not up to date anymore as we will need two operations with different Semantics: "Sort" is sorting on internal IDs for joining whereas  "ORDER BY" is the sorting according to the actual values of the IDs such that end users get the desired order.


bone socket tutorial

Tutorial on how to use bones as sockets to calculate the position of something.

@ipzaur Thank you very much for this great addition! :)


[Radarr/Sonarr] No access to media folder

Radarr NAS/ Sonarr NAS

Addon name : Radarr NAS / Sonarr NAS
Addon version : Latest

Describe the bug
Hi I have No access to media folder, I see here that the config.json have media:rw set but when I trying to add folder I see empty media folder and even that I can’t set this folder for a download.
I have mounted ssd drive via this script https://gist.github.com/eklex/c5fac345de5be9d9bc420510617c86b5, Plex Media Server add-on from community repo have access to this media folder and folders on this drive too, when I check this https://github.com/hassio-addons/addon-plex/blob/main/plex/config.yaml we can see that media have :rw to and I have access.
I see only one difference in map section, media folder in your config is on the 3rd place, config and share folders works at my installation fine.
Do anyone here have same problem? Maybe you can change the place of folders in config or tell me how to check anything in my installation?
I have latest Hassio system on raspberry pi

Hi,
Indeed there is a single place to mount media folders so I don't see why it wouldn't work... Otherwise, why don't you mount your disk directly in Radarr & Sonarr using the localdisks option ? Then it will be mounted in /mnt. Perhaps not what you want but it could work ;-)
Btw do you have the same user ID & group ID specified in the addons and in the script used to mount ? The best is to keep 0 & 0 to be sure to be root, that should avoid some permissions issues

I've put a media file and a folder in my HA media folder, reinstalled radarr nas, and I can see both in /media... So the media folder is correctly mounted. If you can't see it, perhaps it is a permission issue... And is there no error messages in the supervisor log when running the addon? Errors from the addon log would probably not be relevant if it is a HA permission or apparmor issue...

Oh my gosh, you helped me, the problem was ID and Group ID, one the 0 everything is fine, thanks for your help and sorry for bothering you.

That's great, happy it works!


Prometheus' /query endpoint returning data After requested time

Bug Report
What did you do?
I sent a query at timestamp<PHONE_NUMBER>.3346083
What did you expect to see?
I expected to see a result with a timestamp of<PHONE_NUMBER>.3346083
What did you see instead? Under which circumstances?
I saw a result with a timestamp of<PHONE_NUMBER>.335 -- which is technically after my requested time. I am able to reproduce this on master as well-- the issue comes from here where the parseTime actually rounds the nanoseconds. This is specifically an issue in promxy as this results in data after the requested time which the promql engine will ignore (generally it handles data that is at or before the time, not after).
https://play.golang.org/p/VYCV1JFPIKN showcases the issue. Go playground doesn't do time properly but if you run it locally you'll get output like:
old
1567093768.0287952
2019-08-29 08:49:28.029 -0700 PDT
1567093768.029
new
1567093768.029
2019-08-29 08:49:28.029 -0700 PDT
1567093768.029

So I see 2 paths forward (1) remove the rounding from parseTime or (2) change the rounding to truncate. I'm happy to provide a patch for either, I just need to know which approach we prefer.

Prometheus only handles milliseconds, so providing more precision is undefined really.

true, the pain point is that someone can send a time (as described) and get an invalid response (data in the future). So for using the prom UI this is a non-issue, but of anyone is using the API and sends higher-precision it can return an invalid response. IMO prom should either (1) truncate or (2) error the way it is now is just confusing :/

I don't think I'd class the response as invalid, and the question is more what the expected behaviour is with Javascript which only supports floats.

I'd classify it as "invalid" only because the data returned is for a time strictly after the time requested (it should match -- not be before or after)
e.g.<PHONE_NUMBER>.0287952 shouldn't ever return<PHONE_NUMBER>.029
Which is why I think truncating or erroring would make more sense.

it should match -- not be before or after

That's not possible, if you provide more precision than milliseconds then it's always going to violate one of those. Truncating doesn't help, and erroring would run afoul of Javascript and be a bit user hostile.

True, but rounding is IMO worse than truncation because its either before or after instead of being consistent. Especially if you wire the API up to the promql engine it is very strict about not taking data after the requested time.

It's consistent, and makes no practical difference for graphing.
The query API is not a storage API so that's not going to work well for other reasons, and in any case the engine can't request anything other than milliseconds.

The engine can request more than Ms, internally lookups are just time.Time
which allows more precision (I had this linked issue using the promql
engine).
On Sun, Sep 1, 2019, 2:44 AM Brian Brazil<EMAIL_ADDRESS>wrote:

It's consistent, and makes no practical difference for graphing.
The query API is not a storage API so that's not going to work well for
other reasons, and in any case the engine can't request anything other than
milliseconds.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/prometheus/prometheus/issues/5972?email_source=notifications&email_token=AAMLPHMMBKAAZ6VCBMDPSA3QHOFGBA5CNFSM4ISD2UOKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD5T6RJA#issuecomment-526903460,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAMLPHNHYVCELT2OMNS2J5DQHOFGBANCNFSM4ISD2UOA
.

It can't, it's all millis: https://github.com/prometheus/prometheus/blob/cb5213e91438d0a0341631f8049c6c637d394a9b/storage/interface.go#L49

The engine is time.Time (
https://github.com/prometheus/prometheus/blob/master/promql/engine.go#L311)
which means somewhere it's getting translated, so maybe the same problem at
that level. I'm addition the go client is time.Time (
https://github.com/prometheus/client_golang/blob/master/api/prometheus/v1/api.go#L236)
I can submit a pr to truncate in there
On Sun, Sep 1, 2019, 8:23 AM Brian Brazil<EMAIL_ADDRESS>wrote:

It can't, it's all millis:
https://github.com/prometheus/prometheus/blob/cb5213e91438d0a0341631f8049c6c637d394a9b/storage/interface.go#L49
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/prometheus/prometheus/issues/5972?email_source=notifications&email_token=AAMLPHM4IUOCRESVQJ5CMT3QHPNABA5CNFSM4ISD2UOKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD5UEQJY#issuecomment-526927911,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAMLPHJT5INR5G2IH2L3FS3QHPNABANCNFSM4ISD2UOA
.

What I am saying is that PromQL can only ask for millis from storage, so I'm not seeing any actual problem here.

The issue is the conversion. If a time is sent that's too precise the web
rounds and sends millis to the storage, which as previously mentioned can
lead to results which are strictly after the requested time -- hence a
problem
On Sun, Sep 1, 2019, 9:20 AM Brian Brazil<EMAIL_ADDRESS>wrote:

What I am saying is that PromQL can only ask for millis from storage, so
I'm not seeing any actual problem here.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/prometheus/prometheus/issues/5972?email_source=notifications&email_token=AAMLPHMEZMGDQ4LE6KUQWRTQHPTWHA5CNFSM4ISD2UOKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD5UFVYQ#issuecomment-526932706,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAMLPHM7ZBZCPCWOFN2YTYDQHPTWHANCNFSM4ISD2UOA
.

I haven't been involved with the API usages  at all so my answer is just a guess, but it seems that error-ing is the right approach here. I mean if the request is not with the expected values the API should return an error (or a warning if that can be part of the API response.)

erroring would run afoul of Javascript and be a bit user hostile.

@brian-brazil can you give an example why you think this?

Javascript has no support for integers, so users can't sanely control if they end up with a value that's not an exact millisecond value.

How does this correlate with error-ing or warning when making a request with a higher than the supported precision?

Floating point inaccuracy would cause unexpected errors.

Than instead of an error can return the results along with just a warning.

I'm not sure this is an appropriate use of a warning, especially as some users will treat warnings as fatal errors.

But this is their choice right? The api returns the results and they can decide what to do with it.

@jacksontj would that work for your use case?

Javascript has no support for integers, so users can't sanely control if they end up with a value that's not an exact millisecond value.

If this is the case maybe we should make this behavior client dependant? e.g. have the client_golang client error, but the javascript side won't? That seems somehow worse though

@jacksontj would that work for your use case?

Unfortunately not, the warnings today are only storage level warnings and specifically are treated as non-fatal (IMO they seem very much like an afterthought and still have a lot of kinks to work out). In addition they are just strings so I wouldn't be able to reliably determine that one of the warnings was "time is too granular". Even if we solved those it wouldn't solve the real issue where I'd get data after my request -- I still have to truncate the time somewhere else. In my use-case I just wrapped the prom API to always truncate the time to milliseconds so I won't get this "data after request" issue.

Closing as discussed in https://github.com/prometheus/client_golang/pull/636 .


Export/Import a single action

I would like to export and import a single action, so that I can share already defined actions with my friends.

done.


Integrate modern optimized zlib

Related to #348 zlib-ng is a project attempting to modernize the zlib codebase:
https://github.com/zlib-ng/zlib-ng

Ping, still relevant

@gdevenyi the page you cite compares zlib-ng to Cloudflare. It is useful, but I think it is a little out of date. More recent patches to Cloudflare provide good MacOS support. When I looked into this (perhaps a year ago) the benefit of Cloudflare was that it is faster. However, Cloudflare has two limitations: it will fail on old CPUs (more than 10 years old) and it has different licensing.
Specifically, to get optimal performance from CloudFlare you must build with a a bit of [GPL code]. (https://github.com/cloudflare/zlib/blob/gcc.amd64/contrib/amd64/crc32-pclmul_asm.S) which makes it incompatible with some projects. If you must retain the pure zlib license, you should stick with zlib-ng, build CloudFlare without requiring crc32-pclmul_asm.S or make a clean-room replacement for that code. The final option would not be too hard, as Intel provided open source sample code available on Github when they introduced these instructions.

Indeed @neurolabusc zlib-ng intends to retain the zlib licence, and pulls from all available zlib forks where licencing allows (intel, cloudflare, chromium, others?)

zlib-ng has reached release candidate https://github.com/zlib-ng/zlib-ng/releases/tag/v2.0.0-RC1

zlib-ng has reached release candidate https://github.com/zlib-ng/zlib-ng/releases/tag/v2.0.0-RC1

@gdevenyi do you have time and will to include zlib-ng into ITK?
@thewtex if done in time, should we include it into 5.2 final?

@gdevenyi do you have time and will to include zlib-ng into ITK?
@thewtex if done in time, should we include it into 5.2 final?

What's the timeframe for 5.2?

What's the timeframe for 5.2?

Weeks. Currently scheduled for Jan 29th.

Weeks. Currently scheduled for Jan 29th.

I'm not sure I can make it, so others are welcome to try :)

I'm not sure I can make it, so others are welcome to try :)

@gdevenyi we are working on the first ITK 5.2 release candidate, but we will have multiple release candidates -- it is reasonable to integrate this for the 5.2 release.

@gdevenyi we are working on the first ITK 5.2 release candidate, but we will have multiple release candidates -- it is reasonable to integrate this for the 5.2 release.

I haven't started trying yet, but this will probably bug us out as well, https://github.com/zlib-ng/zlib-ng/issues/827

I haven't started trying yet, but this will probably bug us out as well, https://github.com/zlib-ng/zlib-ng/issues/827

I haven't started trying yet, but this will probably bug us out as well, zlib-ng/zlib-ng#827

Perhaps this is addressed by:

Include zlib.h when zlib compatibility is on (see -DZLIB_COMPAT=ON or --zlib-compat)

? per https://github.com/zlib-ng/zlib-ng/issues/736#issuecomment-768027183

Yes, should probably try to investigate this again since 2.0.4 has been released.

After digging into this today, the zlib module in ITK uses the git subtree feature to track changes in a such a way where it is beyond my ability to successfully switch to zlib-ng.
I always end up at:
error: Entry 'Modules/ThirdParty/ZLIB/src/itkzlib/.gitattributes' overlaps with 'Modules/ThirdParty/ZLIB/src/itkzlib/.gitattributes'.  Cannot bind.

I suspect this might be due to the current ITK zlib being a diverged fork of the official zlib while zlib-ng is a different diverged fork.

Perhaps we should create a new root commit for zlib-ng? @bradking is this the right approach? If so can you help Gabriel do it? If not, what do you suggest?

@gdevenyi please push your current changes to Modules/ThirdParty/ZLIB/UpdateFromUpstream.sh to a branch in your fork and let me know the name.

There's essentially nothing done, but here it is:
https://github.com/gdevenyi/ITK/tree/zlib-ng

That's a good start:

Add a preceding commit that does nothing but git rm -rf Modules/ThirdParty/ZLIB/src/itkzlib to remove the old files before running the new import.  That will avoid the "overlaps" problem.

Replace name="zlib" with name="zlib-ng"

After git_archive, add
  pushd "${extractdir}/${name}-reduced"
  echo "* -whitespace" > .gitattributes
  popd

Restore the paths='...' list to import only the needed subset of files.  I sometimes compute this by cloning the upstream manually, using git rm locally, and trying to build it.  Modify CMakeLists.txt if needed to minimize files.  After importing, add an extra commit that modifies CMakeLists.txt in the same way (if necessary).

That's a good start:

Add a preceding commit that does nothing but git rm -rf Modules/ThirdParty/ZLIB/src/itkzlib to remove the old files before running the new import.  That will avoid the "overlaps" problem.

I had tried that and it hadn't sorted the overlaps :)

Replace name="zlib" with name="zlib-ng"

Good catch :+1:

After git_archive, add  pushd "${extractdir}/${name}-reduced"
  echo "* -whitespace" > .gitattributes
  popd

This is probably what actually solves the overlaps problem :)

Restore the paths='...' list to import only the needed subset of files.  I sometimes compute this by cloning the upstream manually, using git rm locally, and trying to build it.  Modify CMakeLists.txt if needed to minimize files.  After importing, add an extra commit that modifies CMakeLists.txt in the same way (if necessary).

What exactly is the purpose of purging all these extra files? I know zlib-ng has now arch-specific files and such that I can't directly test the need for building, but may mean ITK can't build this properly on other arches.
Thanks very much for your help!

What exactly is the purpose of purging all these extra files?

We exclude those to minimize the size of the imported content.  We aren't developing zlib-ng or building it for external distribution, so none of the associated files is needed.  Tests and documentation are not needed.  Packaging files are not needed.  Etc.

An exclude option for the UpdateFromUpstream sounds pretty nice right now :)

With third party modules, do we typically build with tests?

We typically remove tests entirely when importing a third-party library (as Brad King suggested above). So we don't test third party libraries, we assume they are fine.


[INFRA] fix draft rendering css on mobile or when browser window is narrow

fixes #491
it's probably (I think) the nav__title element's background that needs to be changed --> BUT ONLY when the width of the browser is of a certain width or below, see screenshot from browser console (in Chrome):

I applied red color to nav__title but:

this results in unwanted coloring on wide screens, see screenshot:

this gets overridden when the screen is narrow (or the browser window is made narrow), see the "strike-through" CSS from the first screenshot in this thread, ... and then what we want to be red is blue again.

So we somehow need to edit/customize the logic that kicks in for narrow browser windows --> and to have that add the red color.
EDIT: tried that via https://stackoverflow.com/questions/13979443/if-screen-resolution-is-less-than-x-append-css/13979463 in https://github.com/bids-standard/bids-specification/pull/889/commits/fc1f272c0c39bf1fd5d703529aa85d3e6256ad70 ... will revert if it doesn't work.

An alternative might be to set the colors via :root {}, see: https://squidfunk.github.io/mkdocs-material/setup/changing-the-colors/#custom-colors

okay, that works! conditionally applying CSS when screens are narrow ... and preventing that to be overridden, using !important:

I think that's what we wanted originally and in #491 ... and it's better than the alternative solution from https://github.com/bids-standard/bids-specification/pull/889/commits/1b5d2bcdfee73ded8014e2ddbadd3e46ddffde4b (and corresponding comment https://github.com/bids-standard/bids-specification/pull/889#discussion_r720666008)

Looks good on my phone. Haven't looked from a laptop.


wasm: TCP stats share state with HTTP stats, and configurations race

TCP and HTTP stats reporting apply to the same root context, meaning the last one wins. This is mostly a concern for customizations and reporting durations.
FYI @douglas-reid @gargnupur @mandarjog

This only applies to v8 but not nullvm right?
Isn't this a configuration issue? This can be worked around by either share the same metric customization between TCP and HTTP, or use different root context id to distinguish.

It applies to nullvm from what I can tell. There's a still single nullvm root context for stats inbound.
I think it makes more sense to split the root context than trying to retrofit the same root context for two purposes. WDYT?

It applies to nullvm from what I can tell. There's a still single nullvm root context for stats inbound.

oh yes, that is right. Null vm instance is also thread local and shared by streams.

I think it makes more sense to split the root context than trying to retrofit the same root context for two purposes. WDYT?

Yeah that makes sense to me. I think we used to talk about adding TCP as separated root context, which is something like this: https://github.com/istio/proxy/compare/master...bianpengyuan:tcp-context?expand=1

Good catch, this should go on the wiki. I think it makes sense to split the root contexts.
There is some shared code between the tcp and http but that can be refactored as library / common code.
Would it make sense to just explicitly separate them? Right now in several places we have if (tcp) type code.

Given that I'm adding interval reporting that is very similar to TCP, I
think we should keep the combined implementation and just introduce a
couple more root contexts.
On Fri, Oct 16, 2020 at 5:26 PM mandarjog<EMAIL_ADDRESS>wrote:

Good catch, this should go on the wiki. I think it makes sense to split
the root contexts.
There is some shared code between the tcp and http but that can be
refactored as library / common code.
Would it make sense to just explicitly separate them? Right now in several
places we have if (tcp) type code.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/istio/istio/issues/28017#issuecomment-710708196, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/ACIYRRWW62EGT6T4RHCN56DSLDQDRANCNFSM4SST46DA
.

Change of plans: adding new root context leads to unexpected crashes, so something is broken in Wasm (again). Let's keep one root and share the config instead.

CC @PiotrSikora @douglas-reid
I think this is a serious flaw in the design for onConfigure that makes it hard to implement a usable telemetry API on top. There are two major problems:

Its effect is immediate, retroactive, and permanent.  Whenever a new xDS config is pushed, the existing report batches and on-going streams are affected. If the xDS configuration is ultimately rejected (because of Wasm or something else), it is still applied. Once an incorrect configuration is applied, it is not reverted. The same configuration is applied repeatedly from many listeners that carry it (which might be causing the CPU spikes).

It breaks isolation between listeners. TCP and HTTP listener configuration apply to the same root. Depending which one is applied last during xDS, it wins. The only sensible way it can work is if the entire xDS is swapped atomically. If one mixes stale LDS for HTTP with fresh LDS for TCP, onConfigure will flip flop between the two versions, with a CPU spike. If LDS is not 100% synced with ECDS, it will flip flop.

@kyessenov i think i understand issues with new config. does this also imply an issue with per-listener customizations (which i think was your second point)?

@douglas-reid The way I understand it, we cannot have per-listener customizations in principle with onConfigure. It's a singleton.

OK, I've looked at the code. The WasmVM is keyed on vm_id + vm_configuration + plugin bytecode, but not on the plugin configuration, so that's obviously going to be broken if we have different plugin configurations randomly reconfiguring the only instance of the extension running inside the same WasmVM.
There are a few solutions that come to mind:

Add plugin configuration to the key. This is easy to implement, but will results in many more WasmVMs being active.
For each config update, create a new instance of the root context inside the WasmVM. This will result in many different instances of the same plugin running at the same time, but they'll have different root_context_id, so as long as they don't store any global variables that should be fine.

Separately, we should be ignoring config updates if nothing changed. It sounds like that's not the case right now.

Should we consider this a blocking bug? I agree about the need to change our modules to diff incoming configs.

Should we consider this a blocking bug? I agree about the need to change our modules to diff incoming configs.

Probably not if the short term fix to have different contexts does not work (https://github.com/istio/istio/issues/28017#issuecomment-712315223). This is not a regression so I am reluctant to make this a release blocker.

And we should try to make the context separation work by 1.8 if not a very hard fix.

Context separation is constrained by v8. We'll double the number of VMs if we split TCP and HTTP root context. Even then, each root context suffers from the same problem (e.g. if you submit a bad EnvoyFilter, it will not undo the changes by removing the EnvoyFilter).

For plugin configurationto work, we would have to add something in config to say this is tcp and http. Currently the config is also same for both http and tcp

@douglas-reid The way I understand it, we cannot have per-listener customizations in principle with onConfigure. It's a singleton.

This seems like a big drawback, and could be hard to make visible properly to users (especially in a Telemetry API).

A few design ideas, although solving this probably deserves a real re-design:

Add a generation number to plugin config with the last one winning. Plugins must keep the old generation config if the new one is broken.
Debounce onConfigure, so that Envoy applies it every 1-10 seconds. In case of conflicts, sort them in some deterministic order.
Detach root config from plugin config, and make it a first class resource in bootstrap extensions. Implement ECDS for it.

Detach root config from plugin config, and make it a first class resource in bootstrap extensions. Implement ECDS for it.

I think this is the right way to go, although not sure what do you mean by bootstrap extensions. I think it would be cleaner if we have another discovery resource (Wasm Context discovery), and Wasm VM and context lifecycle will tie to this resource.

@PiotrSikora @kyessenov onConfigure is called on the baseVM before being called on the worker, correct?
In that case we should be assured that the config is valid.
If we can do something equivalent to the how it happens in native filters that should work.
I think that is what Piotr suggested as well.

@mandarjog
In our extensions, onConfigure always returns true, or the proxy crashes with LDS update.
Native filters have two general patterns:

Shared fate with the listener. This would get rid of root_id concept, and just create a root for every listener. It would be created once listener is accepted and deleted once listener is drained. This works fine for stats filter I think.
Key by configuration. Instantiate thread local roots for each configuration (again, not sure why we need root ID). Create when a new configuration is mentioned, delete when all listeners referencing it are drained.

Should we consider this a blocking bug? I agree about the need to change our modules to diff incoming configs.

I think so. Even though it's not a regression, it's terribly broken.

Detach root config from plugin config, and make it a first class resource in bootstrap extensions. Implement ECDS for it.

But root config is plugin config.

In our extensions, onConfigure always returns true, or the proxy crashes with LDS update.

Could you file an issue for that in Envoy repo? If you always return true, then there is no way to catch configuration issues, and you'll end up with events dispatched on misconfigured plugin.

https://github.com/proxy-wasm/proxy-wasm-cpp-host/pull/78 and https://github.com/envoyproxy/envoy/pull/13753 are ready for review and they should fix the issue if anyone wants to give it a try.

I think we should have separate configs for TCP and HTTP now, but needs end-to-end validation.

I think we should have separate configs for TCP and HTTP now, but needs end-to-end validation.

The configuration race has been fixed in 1.9 already, but I don't believe we updated the Istio plugins accordingly.
@kyessenov @bianpengyuan could one of you update them to use separate configs instead of relying on root_id for inbound/outbound?

@kyessenov @bianpengyuan ping.

Moving to 1.12 due to recent inactivity

@kyessenov @bianpengyuan Any updates? I don't want to just push this back without checking in.

Still an issue.
/reopen

@kyessenov i assume still an issue?


[FIX] Fix initialisation of hierarchical_binning.

When writing some tests to increase the code coverage I noticed an edge case bug.
The first row is incorrectly initialized because we start counting the sum from 0 but we have to start at the weight/cardinality of the first user bin. Otherwise the second cell in the first row (this cell indicated that UB1 and UB2 are merged into TB1) only contains the weight/cardinality of UB2.
This PR corrects this one line and changes all the tests.
The second commit adds a new test where I noticed the faulty behavior.
In the case of the hierarchical_binning_test in the small_example we even reach a better outcome.
    data.filenames = {"seq0", "seq1", "seq2", "seq3", "seq4", "seq5", "seq6",  "seq7"};
    data.kmer_counts = {500, 1000, 500, 500, 500, 500, 500, 500};

Before: 3 Merged bins, Expected bin sizes [500, 1000, 1000, 2000], Maximum 2000
#FILES  BIN_INDICES     NUMBER_OF_BINS
seq7    0       1
seq5    1;0     1;32
seq6    1;32    1;32
seq3    2;0     1;32
seq4    2;32    1;32
seq1    3;0     1;52
seq0    3;52    1;6
seq2    3;58    1;6

After: 2 Merged bins, Expected bin sizes [500, 1500, 1500, 1000], Maximum 1500
#FILES  BIN_INDICES     NUMBER_OF_BINS
seq7    0       1
seq4    1;0     1;22
seq5    1;22    1;21
seq6    1;43    1;21
seq0    2;0     1;22
seq2    2;22    1;21
seq3    2;43    1;21
seq1    3       1

Note that the coverage fail will be fixed by #32


Unit test failure on Linux (Ubuntu 16.04)

Latest version of object store master (ad161a5) as well as core 2.7.0 and sync 1.9.0.
tests is a Catch v1.7.0 host application.
Run with -? for options

-------------------------------------------------------------------------------
sync: encrypt local realm file
  open a session with realm file encryption and then open the same file
  directly
-------------------------------------------------------------------------------
/home/kristian/realm-object-store/tests/sync/session/session.cpp:510
...............................................................................

/home/kristian/realm-object-store/tests/sync/session/session.cpp:522: FAILED:
  {Unknown expression after the reported line}
due to a fatal error condition:
  SIGSEGV - Segmentation violation signal

===============================================================================
test cases:   84 |   83 passed | 1 failed
assertions: 6796 | 6795 passed | 1 failed

Segmentation fault (core dumped)

Threads:
* 1    Thread 0x7fbed648e700 (LWP 28908) 0x0000000000b51d5c in realm::Array::get_is_inner_bptree_node_from_header (header=0x208 <error: Cannot access memory at address 0x208>) at /home/kristian/realm-core/src/realm/array.hpp:1596
  2    Thread 0x7fbed6c8f700 (LWP 28907) 0x00007fbed7cace23 in epoll_wait () at ../sysdeps/unix/syscall-template.S:84
  3    Thread 0x7fbed7490700 (LWP 28843) 0x00007fbed7cace23 in epoll_wait () at ../sysdeps/unix/syscall-template.S:84
  4    Thread 0x7fbed940c780 (LWP 28842) 0x00007fbed873ab2d in fsync () at ../sysdeps/unix/syscall-template.S:84

Thread 1 stacktrace:
#0  0x0000000000b51d5c in realm::Array::get_is_inner_bptree_node_from_header (header=0x208 <error: Cannot access memory at address 0x208>) at /home/kristian/realm-core/src/realm/array.hpp:1596
#1  0x00000000010e2678 in realm::Array::init_from_mem (this=0x7fbed00067f0, mem=...) at array.cpp:215
#2  0x0000000000b51685 in realm::Array::init_from_ref (this=0x7fbed00067f0, ref=520) at /home/kristian/realm-core/src/realm/array.hpp:1317
#3  0x00000000010862b0 in (anonymous namespace)::HistoryImpl::update_from_ref (this=0x7fbed00034d0, ref=520, version=3) at sync/history.cpp:849
#4  0x000000000108596c in (anonymous namespace)::HistoryImpl::do_update_from_parent (this=0x7fbed00034d0, current_version=3) at sync/history.cpp:775
#5  0x000000000108553e in (anonymous namespace)::HistoryImpl::get_upload_download_bytes (this=0x7fbed00034d0, downloaded_bytes=@0x7fbed648dc08: 0, downloadable_bytes=@0x7fbed648dc10: 0, uploaded_bytes=@0x7fbed648dc18: 0, uploadable_bytes=@0x7fbed648dc20: 0, snapshot_version=@0x7fbed648dc28: 0)
    at sync/history.cpp:621
#6  0x00000000010998e7 in (anonymous namespace)::SessionImpl::call_progress_handler (this=0x2ce8db0) at sync/client.cpp:3426
#7  0x000000000108cbde in (anonymous namespace)::SessionImpl::<lambda()>::operator()(void) const (__closure=0x7fbed648dca0) at sync/client.cpp:718
#8  0x00000000010ae5bb in realm::util::network::Service::PostOper<(anonymous namespace)::SessionImpl::set_progress_handler(std::function<void(long unsigned int, long unsigned int, long unsigned int, long unsigned int, long unsigned int, long unsigned int)>)::<lambda()> >::recycle_and_execute(void) (this=0x2cea5f0)
    at ../realm/util/network.hpp:1875
Python Exception <class 'TypeError'> expected string or bytes-like object:
#9  0x0000000000ff3207 in realm::util::network::Service::execute<realm::util::network::Service::AsyncOper> (lenders_ptr=) at ../realm/util/network.hpp:2561
#10 0x0000000000ff1424 in realm::util::network::Service::Impl::run (this=0x2cf4cf0) at util/network.cpp:1181
#11 0x0000000000fed7a4 in realm::util::network::Service::run (this=0x2d0b128) at util/network.cpp:1373
#12 0x0000000001093983 in (anonymous namespace)::ClientImpl::run (this=0x2d0b120) at sync/client.cpp:2303
#13 0x0000000001099eb0 in realm::sync::Client::run (this=0x2cc2f60) at sync/client.cpp:3551
#14 0x0000000000fb92d8 in realm::_impl::SyncClient::SyncClient(std::unique_ptr<realm::util::Logger, std::default_delete<realm::util::Logger> >, realm::sync::Client::ReconnectMode)::{lambda()#1}::operator()() const (__closure=0x2ce4f58) at /home/kristian/realm-object-store/src/sync/impl/sync_client.hpp:57
#15 0x0000000000fc5068 in std::_Bind_simple<realm::_impl::SyncClient::SyncClient(std::unique_ptr<realm::util::Logger, std::default_delete<realm::util::Logger> >, realm::sync::Client::ReconnectMode)::{lambda()#1} ()>::_M_invoke<>(std::_Index_tuple<>) (this=0x2ce4f58) at /usr/include/c++/5/functional:1531
#16 0x0000000000fc4eee in std::_Bind_simple<realm::_impl::SyncClient::SyncClient(std::unique_ptr<realm::util::Logger, std::default_delete<realm::util::Logger> >, realm::sync::Client::ReconnectMode)::{lambda()#1} ()>::operator()() (this=0x2ce4f58) at /usr/include/c++/5/functional:1520
#17 0x0000000000fc4e3e in std::thread::_Impl<std::_Bind_simple<realm::_impl::SyncClient::SyncClient(std::unique_ptr<realm::util::Logger, std::default_delete<realm::util::Logger> >, realm::sync::Client::ReconnectMode)::{lambda()#1} ()> >::_M_run() (this=0x2ce4f40) at /usr/include/c++/5/thread:115
#18 0x00007fbed82424a0 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#19 0x00007fbed87316ba in start_thread (arg=0x7fbed648e700) at pthread_create.c:333
#20 0x00007fbed7cac82d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109

Thread 2 stacktrace:
#0  0x00007fbed7cace23 in epoll_wait () at ../sysdeps/unix/syscall-template.S:84
#1  0x0000000000fecb54 in realm::util::network::Service::IoReactor::wait_and_activate (this=0x2cadd28, timeout=..., now=...) at util/network.cpp:562
#2  0x0000000000fed237 in realm::util::network::Service::IoReactor::wait_and_advance (this=0x2cadd28, timeout=..., now=..., interrupted=@0x7fbed6c8ed7e: false, completed_ops=...) at util/network.cpp:805
#3  0x0000000000ff210b in realm::util::network::Service::Impl::wait_and_process_io (this=0x2cadd20, now=..., interrupted=@0x7fbed6c8ed7e: false) at util/network.cpp:1354
#4  0x0000000000ff14f6 in realm::util::network::Service::Impl::run (this=0x2cadd20) at util/network.cpp:1215
#5  0x0000000000fed7a4 in realm::util::network::Service::run (this=0x2cf3f20) at util/network.cpp:1373
#6  0x0000000000e81968 in (anonymous namespace)::ServerImpl::run (this=0x2cf3240) at sync/server.cpp:333
#7  0x0000000000e8e76a in realm::sync::Server::run (this=0x7ffcc4bd3cb0) at sync/server.cpp:2952
#8  0x0000000000dd2c37 in SyncServer::<lambda()>::operator()(void) const (__closure=0x2cd8dc8) at /home/kristian/realm-object-store/tests/util/test_file.cpp:161
#9  0x0000000000dd472e in std::_Bind_simple<SyncServer::start()::<lambda()>()>::_M_invoke<>(std::_Index_tuple<>) (this=0x2cd8dc8) at /usr/include/c++/5/functional:1531
#10 0x0000000000dd4684 in std::_Bind_simple<SyncServer::start()::<lambda()>()>::operator()(void) (this=0x2cd8dc8) at /usr/include/c++/5/functional:1520
#11 0x0000000000dd4614 in std::thread::_Impl<std::_Bind_simple<SyncServer::start()::<lambda()>()> >::_M_run(void) (this=0x2cd8db0) at /usr/include/c++/5/thread:115
#12 0x00007fbed82424a0 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#13 0x00007fbed87316ba in start_thread (arg=0x7fbed6c8f700) at pthread_create.c:333
#14 0x00007fbed7cac82d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109

Thread 3 stacktrace:
#0  0x00007fbed7cace23 in epoll_wait () at ../sysdeps/unix/syscall-template.S:84
#1  0x0000000000fb2227 in realm::_impl::ExternalCommitHelper::DaemonThread::listen (this=0x1b02f40 <realm::_impl::ExternalCommitHelper::DaemonThread::shared()::daemon_thread>) at /home/kristian/realm-object-store/src/impl/epoll/external_commit_helper.cpp:268
#2  0x0000000000fb1928 in realm::_impl::ExternalCommitHelper::DaemonThread::DaemonThread()::{lambda()#1}::operator()() const () at /home/kristian/realm-object-store/src/impl/epoll/external_commit_helper.cpp:200
#3  0x0000000000fb343c in std::_Bind_simple<realm::_impl::ExternalCommitHelper::DaemonThread::DaemonThread()::<lambda()>()>::_M_invoke<>(std::_Index_tuple<>) (this=0x2cbca38) at /usr/include/c++/5/functional:1531
#4  0x0000000000fb3392 in std::_Bind_simple<realm::_impl::ExternalCommitHelper::DaemonThread::DaemonThread()::<lambda()>()>::operator()(void) (this=0x2cbca38) at /usr/include/c++/5/functional:1520
#5  0x0000000000fb3322 in std::thread::_Impl<std::_Bind_simple<realm::_impl::ExternalCommitHelper::DaemonThread::DaemonThread()::<lambda()>()> >::_M_run(void) (this=0x2cbca20) at /usr/include/c++/5/thread:115
#6  0x00007fbed82424a0 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#7  0x00007fbed87316ba in start_thread (arg=0x7fbed7490700) at pthread_create.c:333
#8  0x00007fbed7cac82d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109

Thread 4 stacktrace:
#0  0x00007fbed873ab2d in fsync () at ../sysdeps/unix/syscall-template.S:84
#1  0x000000000154ccb2 in realm::util::EncryptedFileMapping::sync (this=0x2d20d50) at util/encrypted_file_mapping.cpp:468
#2  0x000000000154c219 in realm::util::EncryptedFileMapping::~EncryptedFileMapping (this=0x2d20d50, __in_chrg=<optimized out>) at util/encrypted_file_mapping.cpp:342
#3  0x00000000010cae9e in realm::util::SharedPtr<realm::util::EncryptedFileMapping>::decref (this=0x2d1df10) at ../realm/util/shared_ptr.hpp:114
#4  0x00000000010ca2fc in realm::util::SharedPtr<realm::util::EncryptedFileMapping>::~SharedPtr (this=0x2d1df10, __in_chrg=<optimized out>) at ../realm/util/shared_ptr.hpp:42
#5  0x00000000010c9c02 in realm::util::mapping_and_addr::~mapping_and_addr (this=0x2d1df10, __in_chrg=<optimized out>) at util/file_mapper.cpp:89
#6  0x00000000010cc3a6 in __gnu_cxx::new_allocator<realm::util::mapping_and_addr>::destroy<realm::util::mapping_and_addr> (this=0x2c707b0, __p=0x2d1df10) at /usr/include/c++/5/ext/new_allocator.h:124
#7  0x00000000010cbd35 in std::allocator_traits<std::allocator<realm::util::mapping_and_addr> >::destroy<realm::util::mapping_and_addr> (__a=..., __p=0x2d1df10) at /usr/include/c++/5/bits/alloc_traits.h:542
#8  0x00000000010cb691 in std::vector<realm::util::mapping_and_addr, std::allocator<realm::util::mapping_and_addr> >::_M_erase (this=0x2c707b0, __position={mapping = {m_ptr = 0x2d20d50, m_count = 0x2d1a860}, addr = 0x7fbed9433000, size = 4096}) at /usr/include/c++/5/bits/vector.tcc:147
#9  0x00000000010ca719 in std::vector<realm::util::mapping_and_addr, std::allocator<realm::util::mapping_and_addr> >::erase (this=0x2c707b0, __position={mapping = {m_ptr = 0x2d20d50, m_count = 0x2d1a860}, addr = 0x7fbed9433000, size = 4096}) at /usr/include/c++/5/bits/stl_vector.h:1147
#10 0x00000000010c82e1 in realm::util::remove_mapping (addr=0x7fbed9433000, size=4096) at util/file_mapper.cpp:185
#11 0x00000000010c8ef9 in realm::util::munmap (addr=0x7fbed9433000, size=4096) at util/file_mapper.cpp:277
#12 0x00000000010c5009 in realm::util::File::unmap (addr=0x7fbed9433000, size=4096) at util/file.cpp:927
#13 0x0000000000d521a8 in realm::util::File::MapBase::unmap (this=0x2cc4d28) at /home/kristian/realm-core/src/realm/util/file.hpp:1049
#14 0x0000000000d52172 in realm::util::File::MapBase::~MapBase (this=0x2cc4d28, __in_chrg=<optimized out>) at /home/kristian/realm-core/src/realm/util/file.hpp:1031
#15 0x00000000010c74c8 in realm::util::File::Map<char>::~Map (this=0x2cc4d28, __in_chrg=<optimized out>) at ../realm/util/file.hpp:1091
#16 0x00000000010d8566 in realm::SlabAlloc::MappedFile::~MappedFile (this=0x2cc4cf0, __in_chrg=<optimized out>) at alloc_slab.cpp:81
#17 0x00000000010e258a in __gnu_cxx::new_allocator<realm::SlabAlloc::MappedFile>::destroy<realm::SlabAlloc::MappedFile> (this=0x2cc4cf0, __p=0x2cc4cf0) at /usr/include/c++/5/ext/new_allocator.h:124
#18 0x00000000010e252f in std::allocator_traits<std::allocator<realm::SlabAlloc::MappedFile> >::destroy<realm::SlabAlloc::MappedFile> (__a=..., __p=0x2cc4cf0) at /usr/include/c++/5/bits/alloc_traits.h:542
#19 0x00000000010e23df in std::_Sp_counted_ptr_inplace<realm::SlabAlloc::MappedFile, std::allocator<realm::SlabAlloc::MappedFile>, (__gnu_cxx::_Lock_policy)2>::_M_dispose (this=0x2cc4ce0) at /usr/include/c++/5/bits/shared_ptr_base.h:531
#20 0x0000000000a66a10 in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release (this=0x2cc4ce0) at /usr/include/c++/5/bits/shared_ptr_base.h:150
#21 0x0000000000a6551f in std::__shared_count<(__gnu_cxx::_Lock_policy)2>::~__shared_count (this=0x7ffcc4bd3788, __in_chrg=<optimized out>) at /usr/include/c++/5/bits/shared_ptr_base.h:659
#22 0x00000000010d85f6 in std::__shared_ptr<realm::SlabAlloc::MappedFile, (__gnu_cxx::_Lock_policy)2>::~__shared_ptr (this=0x7ffcc4bd3780, __in_chrg=<optimized out>) at /usr/include/c++/5/bits/shared_ptr_base.h:925
#23 0x00000000010d8daa in std::__shared_ptr<realm::SlabAlloc::MappedFile, (__gnu_cxx::_Lock_policy)2>::reset (this=0x2d07f38) at /usr/include/c++/5/bits/shared_ptr_base.h:1022
#24 0x00000000010cfe7a in realm::SlabAlloc::detach (this=0x2d07f08) at alloc_slab.cpp:188
#25 0x000000000133a7e2 in realm::SharedGroup::close (this=0x2d07ef0) at group_shared.cpp:1328
#26 0x000000000133a574 in realm::SharedGroup::~SharedGroup (this=0x2d07ef0, __in_chrg=<optimized out>) at group_shared.cpp:1299
#27 0x0000000000c86448 in std::default_delete<realm::SharedGroup>::operator() (this=0x2cf0238, __ptr=0x2d07ef0) at /usr/include/c++/5/bits/unique_ptr.h:76
#28 0x0000000000c85139 in std::unique_ptr<realm::SharedGroup, std::default_delete<realm::SharedGroup> >::~unique_ptr (this=0x2cf0238, __in_chrg=<optimized out>) at /usr/include/c++/5/bits/unique_ptr.h:236
#29 0x0000000000f12554 in realm::Realm::~Realm (this=0x2cf0120, __in_chrg=<optimized out>) at /home/kristian/realm-object-store/src/shared_realm.cpp:190
#30 0x0000000000e72248 in realm::Realm::make_shared_realm(realm::Realm::Config, std::shared_ptr<realm::_impl::RealmCoordinator>)::make_shared_enabler::~make_shared_enabler() (this=0x2cf0120, __in_chrg=<optimized out>) at /home/kristian/realm-object-store/src/shared_realm.hpp:276
#31 0x0000000000e72268 in __gnu_cxx::new_allocator<realm::Realm::make_shared_realm(realm::Realm::Config, std::shared_ptr<realm::_impl::RealmCoordinator>)::make_shared_enabler>::destroy<realm::Realm::make_shared_realm(realm::Realm::Config, std::shared_ptr<realm::_impl::RealmCoordinator>)::make_shared_enabler>(realm::Realm::make_shared_realm(realm::Realm::Config, std::shared_ptr<realm::_impl::RealmCoordinator>)::make_shared_enabler*) (this=0x2cf0120, __p=0x2cf0120) at /usr/include/c++/5/ext/new_allocator.h:124
#32 0x0000000000e7221f in std::allocator_traits<std::allocator<realm::Realm::make_shared_realm(realm::Realm::Config, std::shared_ptr<realm::_impl::RealmCoordinator>)::make_shared_enabler> >::destroy<realm::Realm::make_shared_realm(realm::Realm::Config, std::shared_ptr<realm::_impl::RealmCoordinator>)::make_shared_enabler>(std::allocator<realm::Realm::make_shared_realm(realm::Realm::Config, std::shared_ptr<realm::_impl::RealmCoordinator>)::make_shared_enabler>&, realm::Realm::make_shared_realm(realm::Realm::Config, std::shared_ptr<realm::_impl::RealmCoordinator>)::make_shared_enabler*) (__a=..., __p=0x2cf0120)
    at /usr/include/c++/5/bits/alloc_traits.h:542
#33 0x0000000000e7210f in std::_Sp_counted_ptr_inplace<realm::Realm::make_shared_realm(realm::Realm::Config, std::shared_ptr<realm::_impl::RealmCoordinator>)::make_shared_enabler, std::allocator<realm::Realm::make_shared_realm(realm::Realm::Config, std::shared_ptr<realm::_impl::RealmCoordinator>)::make_shared_enabler>, (__gnu_cxx::_Lock_policy)2>::_M_dispose() (this=0x2cf0110) at /usr/include/c++/5/bits/shared_ptr_base.h:531
#34 0x0000000000a66a10 in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release (this=0x2cf0110) at /usr/include/c++/5/bits/shared_ptr_base.h:150
#35 0x0000000000a6551f in std::__shared_count<(__gnu_cxx::_Lock_policy)2>::~__shared_count (this=0x7ffcc4bd3ac8, __in_chrg=<optimized out>) at /usr/include/c++/5/bits/shared_ptr_base.h:659
#36 0x0000000000b58aee in std::__shared_ptr<realm::Realm, (__gnu_cxx::_Lock_policy)2>::~__shared_ptr (this=0x7ffcc4bd3ac0, __in_chrg=<optimized out>) at /usr/include/c++/5/bits/shared_ptr_base.h:925
#37 0x0000000000b58b0a in std::shared_ptr<realm::Realm>::~shared_ptr (this=0x7ffcc4bd3ac0, __in_chrg=<optimized out>) at /usr/include/c++/5/bits/shared_ptr.h:93
#38 0x0000000000e00601 in ____C_A_T_C_H____T_E_S_T____34 () at /home/kristian/realm-object-store/tests/sync/session/session.cpp:542
#39 0x0000000000b8c866 in Catch::FreeFunctionTestCase::invoke (this=0x2c6e3a0) at /home/kristian/realm-object-store/tests/../external/catch/single_include/catch.hpp:6971
#40 0x0000000000b76c9b in Catch::TestCase::invoke (this=0x2c9eaa0) at /home/kristian/realm-object-store/tests/../external/catch/single_include/catch.hpp:7937
#41 0x0000000000b8b7c8 in Catch::RunContext::invokeActiveTestCase (this=0x7ffcc4bd4f90) at /home/kristian/realm-object-store/tests/../external/catch/single_include/catch.hpp:6551
#42 0x0000000000b8b483 in Catch::RunContext::runCurrentTest (this=0x7ffcc4bd4f90, redirectedCout="", redirectedCerr="") at /home/kristian/realm-object-store/tests/../external/catch/single_include/catch.hpp:6522
#43 0x0000000000b89f7b in Catch::RunContext::runTest (this=0x7ffcc4bd4f90, testCase=...) at /home/kristian/realm-object-store/tests/../external/catch/single_include/catch.hpp:6345
#44 0x0000000000b7414a in Catch::runTests (config=...) at /home/kristian/realm-object-store/tests/../external/catch/single_include/catch.hpp:6690
#45 0x0000000000b8c118 in Catch::Session::run (this=0x7ffcc4bd52a0) at /home/kristian/realm-object-store/tests/../external/catch/single_include/catch.hpp:6798
#46 0x0000000000b8c00a in Catch::Session::run (this=0x7ffcc4bd52a0, argc=1, argv=0x7ffcc4bd54f8) at /home/kristian/realm-object-store/tests/../external/catch/single_include/catch.hpp:6777
#47 0x0000000000b7cd36 in main (argc=1, argv=0x7ffcc4bd54f8) at /home/kristian/realm-object-store/tests/../external/catch/single_include/catch.hpp:10827

Not sure whether this is a bug in core, sync, or object store.
@finnschiermer @austinzheng

Here is how I built and ran the test:
rm -fr _build && mkdir _build && (cd _build && cmake .. -DREALM_CORE_PREFIX=/home/kristian/realm-core -DREALM_SYNC_PREFIX=/home/kristian/realm-sync -DREALM_ENABLE_SYNC=1 && make)
ulimit -c unlimited
for ((;;)); do echo "------$i------" && _build/tests/tests || break; done
gdb -c core _build/tests/tests

This happens for me roughly every second time I run the test suite.

CMake config output:
-- The C compiler identification is GNU 5.4.1
-- The CXX compiler identification is GNU 5.4.1
-- Check for working C compiler: /usr/bin/cc
-- Check for working C compiler: /usr/bin/cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Check for working CXX compiler: /usr/bin/c++
-- Check for working CXX compiler: /usr/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Looking for epoll_create
-- Looking for epoll_create - found
-- Looking for pthread.h
-- Looking for pthread.h - found
-- Looking for pthread_create
-- Looking for pthread_create - not found
-- Looking for pthread_create in pthreads
-- Looking for pthread_create in pthreads - not found
-- Looking for pthread_create in pthread
-- Looking for pthread_create in pthread - found
-- Found Threads: TRUE
Dependencies: REALM_CORE_VERSION=2.7.0;REALM_SYNC_VERSION=1.9.0
-- Found OpenSSL: /usr/lib/x86_64-linux-gnu/libssl.so;/usr/lib/x86_64-linux-gnu/libcrypto.so (found version "1.0.2g")
-- Found PkgConfig: /usr/bin/pkg-config (found version "0.29.1")
-- Found ZLIB: /usr/lib/x86_64-linux-gnu/libz.so (found version "1.2.8")
-- Configuring done
-- Generating done
-- Build files have been written to: /home/kristian/realm-object-store/_build

The backtrace looks very similar to that in #461.

This still happens with core 2.8.1.

Here is a variation (core 2.8.1):
Threads:
* 1    Thread 0x7fdb9ffff700 (LWP 25876) 0x0000000000b54e5a in realm::Array::get_is_inner_bptree_node_from_header (header=0x2e8 <error: Cannot access memory at address 0x2e8>) at /home/kristian/realm-core/src/realm/array.hpp:1596
  2    Thread 0x7fdba70af780 (LWP 25809) 0x00007fdba667e2c2 in ?? () from /lib/x86_64-linux-gnu/libcrypto.so.1.0.0
  3    Thread 0x7fdba4938700 (LWP 25875) 0x00007fdba5955e23 in epoll_wait () at ../sysdeps/unix/syscall-template.S:84
  4    Thread 0x7fdba5139700 (LWP 25810) 0x00007fdba5955e23 in epoll_wait () at ../sysdeps/unix/syscall-template.S:84

Thread 1 stack trace:
#0  0x0000000000b54e5a in realm::Array::get_is_inner_bptree_node_from_header (header=0x2e8 <error: Cannot access memory at address 0x2e8>) at /home/kristian/realm-core/src/realm/array.hpp:1596
#1  0x00000000010e7fbc in realm::Array::init_from_mem (this=0x7fdba000a490, mem=...) at array.cpp:215
#2  0x0000000000b54783 in realm::Array::init_from_ref (this=0x7fdba000a490, ref=744) at /home/kristian/realm-core/src/realm/array.hpp:1317
#3  0x0000000001325d35 in realm::Group::attach (this=0x7fdba0008ba0, top_ref=744, create_group_when_missing=false) at group.cpp:303
#4  0x000000000132606a in realm::Group::attach_shared (this=0x7fdba0008ba0, new_top_ref=744, new_file_size=4096, writable=false) at group.cpp:363
#5  0x00000000013439b9 in realm::_impl::GroupFriend::attach_shared (group=..., new_top_ref=744, new_file_size=4096, writable=false) at ../realm/group.hpp:1156
#6  0x0000000001341b9f in realm::SharedGroup::do_begin_read (this=0x7fdba0008b90, version_id=..., writable=false) at group_shared.cpp:1836
#7  0x0000000001341357 in realm::SharedGroup::begin_read (this=0x7fdba0008b90, version_id=...) at group_shared.cpp:1676
#8  0x0000000000ed565b in realm::ReadTransaction::ReadTransaction (this=0x7fdb9fffeb30, sg=...) at /home/kristian/realm-core/src/realm/group_shared.hpp:677
#9  0x000000000108a9e3 in (anonymous namespace)::HistoryImpl::get_upload_download_bytes (this=0x7fdba0004a40, downloaded_bytes=@0x7fdb9fffec08: 0, downloadable_bytes=@0x7fdb9fffec10: 0, uploaded_bytes=@0x7fdb9fffec18: 0, uploadable_bytes=@0x7fdb9fffec20: 0, snapshot_version=@0x7fdb9fffec28: 0)
    at sync/history.cpp:619
#10 0x000000000109edc1 in (anonymous namespace)::SessionImpl::call_progress_handler (this=0x1f2a5f0) at sync/client.cpp:3426
#11 0x00000000010920a6 in (anonymous namespace)::SessionImpl::<lambda()>::operator()(void) const (__closure=0x7fdb9fffeca0) at sync/client.cpp:718
#12 0x00000000010b3a95 in realm::util::network::Service::PostOper<(anonymous namespace)::SessionImpl::set_progress_handler(std::function<void(long unsigned int, long unsigned int, long unsigned int, long unsigned int, long unsigned int, long unsigned int)>)::<lambda()> >::recycle_and_execute(void) (this=0x1f1b870)
    at ../realm/util/network.hpp:1875
#13 0x0000000000ff734d in realm::util::network::Service::execute<realm::util::network::Service::AsyncOper> (lenders_ptr=std::unique_ptr<realm::util::network::Service::AsyncOper> containing 0x0) at ../realm/util/network.hpp:2561
#14 0x0000000000ff556a in realm::util::network::Service::Impl::run (this=0x1f1ba10) at util/network.cpp:1181
#15 0x0000000000ff18ea in realm::util::network::Service::run (this=0x1f59e28) at util/network.cpp:1373
#16 0x0000000001098e5d in (anonymous namespace)::ClientImpl::run (this=0x1f59e20) at sync/client.cpp:2303
#17 0x000000000109f38a in realm::sync::Client::run (this=0x1f4a810) at sync/client.cpp:3551
#18 0x0000000000fbd41e in realm::_impl::SyncClient::SyncClient(std::unique_ptr<realm::util::Logger, std::default_delete<realm::util::Logger> >, realm::sync::Client::ReconnectMode)::{lambda()#1}::operator()() const (__closure=0x1f03278) at /home/kristian/realm-object-store/src/sync/impl/sync_client.hpp:57
#19 0x0000000000fc91ae in std::_Bind_simple<realm::_impl::SyncClient::SyncClient(std::unique_ptr<realm::util::Logger, std::default_delete<realm::util::Logger> >, realm::sync::Client::ReconnectMode)::{lambda()#1} ()>::_M_invoke<>(std::_Index_tuple<>) (this=0x1f03278) at /usr/include/c++/5/functional:1531
#20 0x0000000000fc9034 in std::_Bind_simple<realm::_impl::SyncClient::SyncClient(std::unique_ptr<realm::util::Logger, std::default_delete<realm::util::Logger> >, realm::sync::Client::ReconnectMode)::{lambda()#1} ()>::operator()() (this=0x1f03278) at /usr/include/c++/5/functional:1520
#21 0x0000000000fc8f84 in std::thread::_Impl<std::_Bind_simple<realm::_impl::SyncClient::SyncClient(std::unique_ptr<realm::util::Logger, std::default_delete<realm::util::Logger> >, realm::sync::Client::ReconnectMode)::{lambda()#1} ()> >::_M_run() (this=0x1f03260) at /usr/include/c++/5/thread:115
#22 0x00007fdba5ee6c80 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#23 0x00007fdba63d16ba in start_thread (arg=0x7fdb9ffff700) at pthread_create.c:333
#24 0x00007fdba595582d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109

Thread 2 stack trace:
#0  0x00007fdba667e2c2 in ?? () from /lib/x86_64-linux-gnu/libcrypto.so.1.0.0
#1  0x00007fdba667ee03 in AES_cbc_encrypt () from /lib/x86_64-linux-gnu/libcrypto.so.1.0.0
#2  0x0000000001f5223c in ?? ()
#3  0x0000000001f5231c in ?? ()
#4  0x00007fffaa010dd0 in ?? ()
#5  0x0000000001f77290 in ?? ()
#6  0x0000000001f782a0 in ?? ()
#7  0x0000000000000480 in ?? ()
#8  0x0000000001f31a70 in ?? ()
#9  0x00007fffaa010e40 in ?? ()
#10 0x929a40a8ed01e302 in ?? ()
#11 0x6d79e381b003ba1e in ?? ()
#12 0x00007fdba70d7020 in ?? ()
#13 0x0000000001f12610 in ?? ()
#14 0x0000000000000001 in ?? ()
#15 0x0000000000000000 in ?? ()

Thread 3 stack trace:
#0  0x00007fdba5955e23 in epoll_wait () at ../sysdeps/unix/syscall-template.S:84
#1  0x0000000000ff0c9a in realm::util::network::Service::IoReactor::wait_and_activate (this=0x1f259d8, timeout=..., now=...) at util/network.cpp:562
#2  0x0000000000ff137d in realm::util::network::Service::IoReactor::wait_and_advance (this=0x1f259d8, timeout=..., now=..., interrupted=@0x7fdba4937d7e: false, completed_ops=...) at util/network.cpp:805
#3  0x0000000000ff6251 in realm::util::network::Service::Impl::wait_and_process_io (this=0x1f259d0, now=..., interrupted=@0x7fdba4937d7e: false) at util/network.cpp:1354
#4  0x0000000000ff563c in realm::util::network::Service::Impl::run (this=0x1f259d0) at util/network.cpp:1215
#5  0x0000000000ff18ea in realm::util::network::Service::run (this=0x1f44d90) at util/network.cpp:1373
#6  0x0000000000e8537e in (anonymous namespace)::ServerImpl::run (this=0x1f440b0) at sync/server.cpp:333
#7  0x0000000000e92180 in realm::sync::Server::run (this=0x7fffaa011ab0) at sync/server.cpp:2952
#8  0x0000000000dd64b7 in SyncServer::<lambda()>::operator()(void) const (__closure=0x1ef7b08) at /home/kristian/realm-object-store/tests/util/test_file.cpp:161
#9  0x0000000000dd7fae in std::_Bind_simple<SyncServer::start()::<lambda()>()>::_M_invoke<>(std::_Index_tuple<>) (this=0x1ef7b08) at /usr/include/c++/5/functional:1531
#10 0x0000000000dd7f04 in std::_Bind_simple<SyncServer::start()::<lambda()>()>::operator()(void) (this=0x1ef7b08) at /usr/include/c++/5/functional:1520
#11 0x0000000000dd7e94 in std::thread::_Impl<std::_Bind_simple<SyncServer::start()::<lambda()>()> >::_M_run(void) (this=0x1ef7af0) at /usr/include/c++/5/thread:115
#12 0x00007fdba5ee6c80 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#13 0x00007fdba63d16ba in start_thread (arg=0x7fdba4938700) at pthread_create.c:333
#14 0x00007fdba595582d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109

Thread 4 stack trace:
#0  0x00007fdba5955e23 in epoll_wait () at ../sysdeps/unix/syscall-template.S:84
84	in ../sysdeps/unix/syscall-template.S

The backtrace looks very similar to that in #461.

@bdash I agree. They both seem to revolve around HistoryImpl::get_upload_download_bytes() at sync/history.cpp:619. I'll close #461 as duplicate.
cc @morten-krogh

@kspangsege @bdash
Do you agree with these statements?
The segfault happens in the event loop thread.
At some point set_progress_handler is called.
This could be right after run(). Is that what you see?
Probably set_progress_handler has been called from outside earlier and is just waiting in
the queue of post operations.
After setting the handler (not shown in the stack trace),
call_progress_handler() is called immediately to get the first callback.
This leads to a call of HistoryImpl::get_upload_download_bytes()
This function just does the usual
   ReadTransaction rt(*m_shared_group); // Throws
   version_type current_client_version = rt.get_version();
   do_update_from_parent(current_client_version); // Throws

do_update_from_parent gets into m_top->init_from_ref for instance.
The thread hits a memory corruption around here.
We could insert breakpoints to check further.
Doesn't this seem like the Realm is corrupted in some way and
get_upload_download_bytes() just happens to be the first function that uses the Realm
and hits an init_from_ref problem?
What are your thoughts?
Should I try to debug this further?

What change in Core or Sync led to these failures?

Yes, it is
#3  0x00000000010862b0 in (anonymous namespace)::HistoryImpl::update_from_ref (this=0x7fbed00034d0, ref=520, version=3) at sync/history.cpp:849
That is exactly
top->init_from_ref();
inside this code part
        using sgf = _impl::SharedGroupFriend;
        Group& group = sgf::get_group(*m_shared_group);
        using gf = _impl::GroupFriend;
        Allocator& alloc = gf::get_alloc(group);
        std::unique_ptr<Array> top(new Array(alloc)); // Throws
        top->init_from_ref(ref);

#461 apparently fails already at ReadTransaction.

@morten-krogh I agree with your assessments.

What change in Core or Sync led to these failures?

Don't know yet.
I have now analyzed the core to a level where I am confident that:

At most one thread is accessing the (SharedGroup, ClientHistory) object pair, since at most one thread is calling sync::Client::run(), and ClientImpl::m_file is not "leaked" to any other thread beyond the event loop thread in client.cpp.
sync::Client is not destroyed while sync::Client::run() is executing.

Remaining possibilities for the cause therefore seems to be:

Core level bug.
Someone / something tampering with the Realm file concurrently via means other than a SharedGroup.
General memory corruption / stray writes.

@morten-krogh I agree that it looks like either the file, or the accessor states have been already corrupted at the time HistoryImpl::get_upload_download_bytes() is called.

Yes. Does this only happen with encrypted Realms?

Don't know. Possibly. It always seem to happen in the context of open a session with realm file encryption and then open the same file directly.

Yes. Does this only happen with encrypted Realms?

Seems like a viable hypothesis, as when we run the sync level test suite, we don't test a lot with encrypted Realms, as far as I know.

By the way, do we even support sync + encryption? I think I remember someone ( @jpsim ) saying that we don't.
cc @bdash

Ok, we only have one sync-level unit test using Realm file encryption (Sync_EncryptClientRealmFiles).

Encryption for the sync client was introduced by @fealebenpae a few months ago.
Maybe, we should turn encryption on in many more Sync unit tests.

I'll try to turn it on in all tests, just to see what happens.

Encryption for the sync client was introduced by @fealebenpae a few months ago.

Ok, cool.

I just saw a failure in CI related to encrypted synchronized Realms, and having something to do with array.hpp. Not sure if this is another version of the same problem, or if I should file a separate ticket.
[linux_sync] /home/jenkins/workspace/m_realm-object-store_PR-463-VHM5B3YLXKQCJYNAQDPMLFKCRSUXV3SAJOINAJWBLV7LQFSAUIBA@2/coverage.build/CMakeFiles/realm-core/src/realm-core/src/realm/array.hpp:1935: [realm-core-2.7.0] Assertion failed: is_attached()
[linux_sync] IMPORTANT: if you see this error, please send this log to<EMAIL_ADDRESS>[linux_sync] Thread name: tests
[linux_sync]
[linux_sync] ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
[linux_sync] tests is a Catch v1.7.0 host application.
[linux_sync] Run with -? for options
[linux_sync]
[linux_sync] -------------------------------------------------------------------------------
[linux_sync] sync: encrypt local realm file
[linux_sync]   open a session with realm file encryption and then open the same file
[linux_sync]   directly
[linux_sync] -------------------------------------------------------------------------------
[linux_sync] /home/jenkins/workspace/m_realm-object-store_PR-463-VHM5B3YLXKQCJYNAQDPMLFKCRSUXV3SAJOINAJWBLV7LQFSAUIBA@2/tests/sync/session/session.cpp:510
[linux_sync] ...............................................................................
[linux_sync]
[linux_sync] /home/jenkins/workspace/m_realm-object-store_PR-463-VHM5B3YLXKQCJYNAQDPMLFKCRSUXV3SAJOINAJWBLV7LQFSAUIBA@2/tests/sync/session/session.cpp:522: FAILED:
[linux_sync]   {Unknown expression after the reported line}
[linux_sync] due to a fatal error condition:
[linux_sync]   SIGABRT - Abort (abnormal termination) signal
[linux_sync]
[linux_sync] ===============================================================================
[linux_sync] test cases:   84 |   83 passed | 1 failed
[linux_sync] assertions: 6796 | 6795 passed | 1 failed
[linux_sync]

I am no longer so sure that this issue is encryption related. I enabled encryption in all Sync_* tests in the sync-level test suite and ran the suite 500 times overnight. No problems were reported.
@morten-krogh

Almost certainly fixed by https://github.com/realm/realm-sync/pull/1391 (although some changes will be needed in object store due to API changes, see https://github.com/realm/realm-sync/blob/e289c51eba9cb138ee192828666b94ccabd3d703/CHANGELOG.md#breaking-changes).

Waiting for confirmation.

Object store master now depends on sync 1.10.0 as of #454, so you could now test this again to confirm that the issue has been resolved.

Fixed by realm/realm-sync#1391


feat: update helix-env (guards, session scope, referrer)

update to the helix-env
testing URLs...
https://helix-env-update--spark-website--adobe.hlx.page/express/pricing?helix-env=stage
https://helix-env-update--spark-website--adobe.hlx.page/express/pricing?helix-env=prod
https://helix-env-update--spark-website--adobe.hlx.page/express/pricing?helix-env=
see "buy now" and "login urls"

temporary circle ci seems to be expected


first commit

Pull Request Created

Hey @avezqureshi14, Please mention the issue you have worked on and also provide some screenshots or videos to showcase your work.

ok @Kanika637 dii , Issue 52 More components on footer #52

Any updates @avezqureshi14 ?

Can you help me to fix the branch conflicts, I mean in this case my resolve conflicts button is disabled

@avezqureshi14  you have to resolve it locally.
You can join this server and ping me, I can help you with this
https://discord.gg/xqsassk3

Kindly resolve the conflicts @avezqureshi14, as the footer is incomplete!


Use flex-grid for Topics. So it's a shorter list and responsive

Use flex-grid for Topics. So it's a shorter list and responsive

Hi @pmario this needs more work, it turns out that the CSS is also affecting other elements:

I do see a problem with Recent Updates, but I think it's OK with contributors. ... right?

I do see a problem with Recent Updates, but I think it's OK with contributors. ... right?

I think it's better for the contributor list to remain vertical unless/until we get to a few dozen contributors. It's a lot easier to visually scan an alphabetical vertical list. Anyhow, I've reverted the whole change for the moment.

I think it's better for the contributor list to remain vertical unless/until we get to a few dozen contributors. It's a lot easier to visually scan an alphabetical vertical list. Anyhow, I've reverted the whole change for the moment.

hmmm. I don't scan for names on the contributor list, because they are aliases. I look for the avatar image.
I personally prefer the flex list, because I don't need to scroll. ... but it's your page.

I also think, that it's unfair, that the list is alphabetical and not random or the order the contributors joined the list.

I would prefer a look like this:

Make the avatar 2em x 2em .. Name vertical aligned top and the number of links below

I would prefer a look like this:

Make the avatar 2em x 2em .. Name vertical aligned top and the number of links below

That does look better.

...That does look better.

I'll try to create a PR with a "card style" layout


Not receiving any response in Skype and Facebook channels - Web chat works

I have developed a bot using the C# Bot Builder SDK (Bot builder version 3.5.2) and deployed it on Azure. This runs successfully on the Emulator as well as the web chat channel.
I have enabled it in Skype and Facebook channels and in those channels the bot doesn't respond. I have tried unregistering and re-registering too in both the channels, however that didnt help either. In addition I have verified Facebook Page Id, App ID, App secret, page access token. All are correct.
Facebook - The bot does not respond at all
Skype - The bot says typing for sometime and then stops. No further response.
On verifying the errors on the Bot configuration page it displays the below error for both facebook and skype channels.
"There was an error sending this message to your bot: HTTP status code InternalServerError"
I enabled Application insights and monitored the errors using Live Metrics stream and found the below.
Facebook - Looks like the bot sends 500 response code for Facebook.
Skype - A 500 response code but after an outgoing command - not sure what that meant (screenshot of failure below)

My Bot ID - anithas_test_bot
Note - This bot was working fine in Facebook till yesterday. I made a few changes to the bot to store and retrieve User data in to the state and redeployed and doesn't seem to work ever since. Assumed it might be due to my changes and reverted them and even then the same problem.
Is there a way i can debug the FB/Skype channels from my code? Any other debugging tips/suggestions?
I seem to be kind of stuck and any help is much appreciated!!

Hi Anitha, I am facing the same problem. Have you found any solution yet ?

https://docs.botframework.com/en-us/technical-faq/#how-can-i-troubleshoot-my-bot has some tips, but in particular, it sounds like your bot's web service is throwing an unhandled exception which leads to a 500 error http status code.  You might consider attaching a debugger or using something like App Insights.