an integration testing framework for redux-saga: RFC (and ideas)

I'm working on this router: https://github.com/cellog/react-redux-saga-router and have been testing it happily along the way.  Recently, I refactored to use a number of separate sagas.  Each saga is fully tested, but I have run into many situations where I was accidentally passing in the wrong parameters to a connected saga, or the timing of different effects was out of whack causing endless loops or things not to be implemented properly.
So I need integration tests, testing the interaction of sagas.  I want to test only these things

the order of dispatches, takes and call/fork effects
each specific workflow in isolation

This way, when I break one aspect of the thing, it won't break unrelated tests as a side effect.  As an example, let's say I have a test that makes sure when a user changes state, the router notices it, and calls the saga that converts that state into a url, then pushes a new url.  If I were to accidentally break the code that generates the url, so that it pushes the wrong url, I still want the integration test to pass.
The idea
I would like to use a generator to express the complex timing order of my integration tests.  This way, much as redux-saga expresses complex asynchronous events as a simple declarative list of actions to undertake, we can write tests that do the same thing.
This means I need a way to spy on an actual running saga to see the order of effects, and I need a separate runner to read the test and look for the effects.
Fortunately, there are 2 resources built into redux-saga that can solve this issue.

SagaMonitor
runSaga

My primary thought is I can set up the sagas with the store and an initial mock state, pass a SagaMonitor with effectTriggered and then use runSaga to run the test.  The test will use take to listen for effects that are expected, and use the full range of power with the pattern for take to do things like listen for a range of effects or just actions or just call.  I'd also like to use put to be able to inject effects into the tested saga frames.
The code (thus far)

function sagaStore(state, reducers = { routing: reducer, week: fakeWeekReducer }) {
  const log = []
  const logger = store => next => action => { // eslint-disable-line
    log.push(action)
    return next(action)
  }
  const monitor = {
    log: [],
    cursor: 0,
    listener: false,
    effectTriggered(props) {
      console.log('triggered', props)
      log.push(props.effect)
      if (monitor.listener) {
        monitor.listener(props.effect)
      }
    },
    effectResolved(props, result) {
      console.log('resolved', props, result)
    }
  }
  const sagaMiddleware = createSagaMiddleware({ sagaMonitor: monitor })

  const store = createStore(combineReducers(reducers),
    state, applyMiddleware(logger, sagaMiddleware))
  return {
    log,
    store,
    sagaMiddleware,
    monitor
  }
}

export const testSaga = (state, reducers) => (sagas, iterator) => {
  const { sagaMiddleware, monitor } = sagaStore(state, reducers)
  const listeners = []
  monitor.listener = effect => listeners.map(listener => listener(effect))
  sagas.map(saga => sagaMiddleware.run(saga))
  runSaga(
    iterator(),
    {
      subscribe(callback) {
        const mine = (effect) => {
          console.log(effect)
          callback(effect)
        }
        listeners.push(mine)
        return () => listeners.splice(listeners.indexOf(mine), 1)
      },
      dispatch(output) {
        console.log(output)
        sagaMiddleware.run(function *() {
          yield output
        })
      },
      getState() {
        return monitor.log
      }
    }
  )
}

export const tests = Object.keys(utils.asEffect).reduce((effects, name) => ({
  ...effects,
  [name]: effect => utils.is.notUndef(utils.asEffect[name](effect))
}), {
  isPromise: effect => utils.is.promise(effect),
  isHelper: effect => utils.is.helper(effect),
  isIterator: effect => utils.is.iterator(effect),
})

Here's a sample test:
import * as effects from 'redux-saga/effects'
import { delay } from 'redux-saga'
import * as helpers from '../test_helper'

describe('react-redux-saga-router integration tests: url changes', () => {

  function another() {
    return 'hi'
  }

  function *foo() {
    while (true) {
      yield effects.take('hi')
      yield delay(1500)
      yield effects.call(another)
      yield effects.put({ type: 'there', payload: 'wheee' })
    }
  }

  const run = helpers.testSaga()
  it('proof of concept', (done) => {
    run([foo], function *() {
      yield effects.put({ type: 'hi', payload: 'there' })
      let effect = yield effects.take(helpers.tests.take)
      expect(first).eqls(effects.put({ type: 'hi', payload: 'there' }))
      effect = yield effects.take(helpers.test.take)
      expect(first).eqls(effects.put({ type: 'there', payload: 'wheee' }))
      yield effects.call(done)
    })
  })
})

What I would like to do is not quite the same as traditional take.  I'd like to use take to filter events.  So that we can skip irrelevant effects (like the root effect when a saga starts) and listen for the one we want.
But take will simply wait indefinitely until the effect shows up.  Perhaps I could use a helper saga that sets a filter for the listener callback and then it will log the effects but not return them.
In any case, this is where I'm at.  Any ideas?

one problem with this approach: the saga monitor does not return every effect as passed in.  If you yield call() a regular function, it just returns the result of calling the function, and not the effect itself.  This is inconsistent.  If you yield call() another saga, the call() effect is passed to the monitor.  This means trying to match something like yield call(delay, 1500) does not work.

got it working!
test helper code:
import createSagaMiddleware, { runSaga } from 'redux-saga'
import * as effects from 'redux-saga/effects'
import * as utils from 'redux-saga/utils'
import { createStore, combineReducers, applyMiddleware } from 'redux'

export const tests = Object.keys(utils.asEffect).reduce((effects, name) => ({
  ...effects,
  [name]: effect => utils.is.notUndef(utils.asEffect[name](effect))
}), {
  isPromise: effect => utils.is.promise(effect),
  isHelper: effect => utils.is.helper(effect),
  isIterator: effect => utils.is.iterator(effect),
  takeEvery: effect => utils.is.helper(effect),
  takeLatest: effect => utils.is.helper(effect),
  throttle: effect => utils.is.helper(effect),
  takem: effect => utils.is.notUndef(utils.asEffect.take(effect)) && effect.TAKE.maybe,
  apply: () => false,
  spawn: effect => utils.is.notUndef(utils.asEffect.fork(effect)) && effect.FORK.detached,
})

function sagaStore(state, reducers = { routing: reducer, week: fakeWeekReducer }) {
  const log = []
  const logger = store => next => action => { // eslint-disable-line
    log.push(action)
    return next(action)
  }
  const monitor = {
  }
  const sagaMiddleware = createSagaMiddleware({ sagaMonitor: monitor })

  const store = createStore(combineReducers(reducers),
    state, applyMiddleware(logger, sagaMiddleware))
  return {
    log, // this is used in the rendering tests, not relevant for testing sagas
    store,
    sagaMiddleware,
    monitor,
  }
}

export const effectNames =
  Object.keys(utils.asEffect).reduce((effects, name) => ({
    ...effects,
    [name]: name,
  }), {
    isPromise: 'promise',
    isHelper: 'helper',
    isIterator: 'iterator',
    takeEvery: 'takeEvery',
    takeLatest: 'takeLatest',
    throttle: 'throttle',
    takem: 'takem',
    apply: 'apply',
    spawn: 'spawn',
  })

export const effectName = effect =>
  effectNames[Object.keys(tests).map(test => tests[test](effect) && test).filter(t => t)[0]]

export const toEffectAction = effect => ({
  type: effectName(effect),
  effect
})

export const START = { type: '###@@@start' }

export function testSaga(state = undefined) {
  const { sagaMiddleware, store, monitor } = sagaStore(state)
  return (sagas, test) => {
    sagaMiddleware.run(function*() {
      yield effects.take('###@@@start')
      yield sagas.map(args => effects.fork(...args))
    })
    runSaga(test(), {
      subscribe: (callback) => {
        monitor.effectTriggered = ({ effect }) => {
          if (effect.length) {
            effect.forEach((e) => {
              callback(toEffectAction(e))
            })
          } else {
            callback(toEffectAction(effect))
          }
        }
        return () => null
      },
      dispatch: (output) => {
        store.dispatch(output)
      }
    })
  }
}

to use in a test:
import * as effects from 'redux-saga/effects'
import { delay } from 'redux-saga'
import * as helpers from '../test_helper'

describe('react-redux-saga-router integration tests: url changes', () => {

  function another() {
    return 'hi'
  }

  function *foo(d) {
    while (true) {
      yield effects.take('hi')
      yield delay(d)
      yield effects.call(another)
      yield effects.put({ type: 'there', payload: 'wheee' })
    }
  }

  const run = helpers.testSaga()
  it('proof of concept', (done) => {
    run([
      [foo, 1500]
    ], // external sagas we are testing, saga followed by arguments, if any
    function *() { // the actual test
      yield effects.put.resolve(helpers.START) // start the external sagas
      yield effects.put({ type: 'hi', payload: 'there' }) // inject actions into the saga we're testing
      let input = yield effects.take('take')

      expect(input.effect).eqls(effects.take('hi'))

      // yield effects.take('promise') // unnecessary, but can be in here if we want
      input = yield effects.take('call')
      expect(input.effect).eqls(effects.call(another))

      input = yield effects.take('put')
      expect(input.effect).eqls(effects.put({ type: 'there', payload: 'wheee' }))

      yield effects.call(done)
    })
  })
})

Basically the way it works is you tell the test helper which sagas to load, and pass any parameters in, which allows mocking things if necessary. The sagas are not run until you yield the START action. You can inject any redux action using put into the running sagas.  Then, to test the order of the effects, you use take with the name of the effect.  The yielded effect is packaged in a redux action with type equal to the effect name.  Since our test runs in its own context, there is zero chance of conflicting with the running sagas action types.
This took a LOT of work to figure out.  I erased and rewrote my code from scratch 3 times to get to here.
Any feedback on improving this is appreciated.

Thank you so much for your contributions.  Since this GH issue was created a number of years ago with little traction, I'm going to close it for now.
If you still feel like this GH issue is important, I'd be happy to continue to chat with you about it.
In general, the FE ecosystem has moved away from unit/functional tests and have most closer to leveraging msw to mock an API and run the sagas as-is alongside the UI.


Add the Once BITten! Podcast

https://anchor.fm/daniel-prince6

@6102bitcoin Your comment got deleted after I rebased, but the indentation should be fixed now.

@6102bitcoin Friendly ping 🙂

Closes #202


Parse PDF w/o images?

Is it possible to use the TEI and parse the full text but ignore the figures/images within the PDF? Thanks

Hello @mikey21211
Do you have a use case to understand what exactly you want to achieve or avoid?
I probably not understood your goal, but you can always select the content of interest in the resulting XML/TEI and ignore some elements during the XML parsing... When processing a PDF, it makes sense to consider the possibility of figures to distinguish those elements/blocks from the the rest of the content body.

That makes sense what you're saying, and I think that the figures might be
throwing off my result. Is there a way to ignore the figures? Or rather,
strip away the images within my pdf before the parse?
On Feb 20, 2018 12:40 AM, "Patrice Lopez"<EMAIL_ADDRESS>wrote:

Hello @mikey21211 https://github.com/mikey21211
Do you have a use case to understand what exactly you want to achieve or
avoid?
I probably not understood your goal, but you can always select the content
of interest in the resulting XML/TEI and ignore some elements during the
XML parsing... When processing a PDF, it makes sense to consider the
possibility of figures to distinguish those elements/blocks from the the
rest of the content body.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/kermitt2/grobid/issues/288#issuecomment-366881531,
or mute the thread
https://github.com/notifications/unsubscribe-auth/Ai8lr5aBz934n2DofUITrHX6AkerEevnks5tWmjRgaJpZM4SLALV
.

Be careful what you wish for. :) You could first of all use pdfimages to see what "all images" means for your PDF. It might be more than you think.


Add --interval-on-error flag for timed restart on failure

Could you add a flag --interval-on-error which pauses xx seconds before restarting the app (if it dies with a non-zero exit code).
Right now, it just keeps immediately restarting the app on error which can hammer the system.

Currently there is a one second wait when the last start attempt was done less than one second ago, but I understand that indeed it may not be enough if many processes are being (re)started at the same time.
A fixed wait time may also result in a similar issue if there are many instances of dmon which are monitoring processes that died simultaneously. While a bit less likely, I can see that happening e.g. if dmon is used to monitor processes as part of system boot.
I think it would make sense to use exponential backoff with some random jitter added to avoid unlucky coincidences, and instead make the option configure the maximum amount of time for the backoff. Once the wait time reaches that value it would stop increasing.
@hongkongkiwi How does the exponential backoff idea sound like?

Yes, I could see an exponential backoff also being useful, although I think being able to set the limits would also be also be helpful.
E.g. --exp-backoff --exp-backoff-max=360s --exp-backoff-min=10s


Don't log ParametersRequest & AlarmLimitsRequest initialization in firmware

This PR fixes #341 by:

Making the Parameters and AlarmLimits request/response services only generate log events for parameters and alarm limits changes (besides ventilation start/stop) after the first ParametersRequest and AlarmLimitsRequest messages have been received from the backend, respectively. This means that unplugging the MCU and plugging it back in while the firmware is running will generate log events on loss and reestablishment of the MCU-backend connection (provided by #344) and on stoppage and starting of ventilation, but it won't clutter the event log with redundant parameter & alarm limits change events.
Making the StateSegment store in the firmware track whether it has received an initial ParametersRequest and an initial AlarmLimitsRequest.
Renaming the Pufferfish::Application::States class to the Pufferfish::Application::Store class, to be consistent with naming in the backend and frontend.

Note: this PR does not provide the same behavior in the backend simulator. This should happen in a future PR, after/when functionality is added to save/load the event log from the filesystem. Only after that's done would we be able to test it.
This PR should be merged after #344 is merged into develop.

Everything looks good. tried to generate erroneous/extra log events multiple times on disconnecting/reconnecting MCU and it works as expected.

Great! Then I'll merge in this PR as prototype-grade code. For records-keeping:

This project is licensed under Apache License v2.0 for any software, and Solderpad Hardware License v2.1 for any hardware - do you agree that your contributions to this project will be under these licenses, too? Yes
Were any of these contributions also part of work you did for an employer or a client? No
Does this work include, or is it based on, any third-party work which you did not create? No


Interfaces, ok or not?

📖 Documentation
I am not sure whether using interfaces is OK or not in functional programming. From what I know, it should not be used. But not sure why. Are you using interfaces or not?

OK, related to https://github.com/gcanti/fp-ts/issues/953


[SPARK-44485][SQL] Optimize TreeNode.generateTreeString

What changes were proposed in this pull request?
Optimize several critical code path in TreeNode.generateTreeString
Why are the changes needed?
In TreeNode.generateTreeString, we observed inefficiency in scala collection operations and virtual function call in our internal workload.
This inefficiency become significant in large plan (we hit a example of more than 1000 nodes). So it’s worth optimizing the super hot code path. By rewriting into native Java code(not so sweet as scala syntax sugar though), we should be able to get rid of most of the overhead.

ArrayBuffer.append

Seq.last

SeqLike.$colon$plus

StringOps.$times

Does this PR introduce any user-facing change?
No
How was this patch tested?
Existing UTs

So how much does it improve in general? cc @MaxGekk FYI

So how much does it improve in general? cc @MaxGekk FYI

In our use case with >1000 nodes plan, the whole explain invocation is ~50% faster, considering this happen on each AQE cycle (where AQE have to update plan change to Spark UI), the improvement is significant.

@liuzqt Can you retry the failed GA task?
On the other hand, does this optimization have the same effect for Scala 2.13 as well?

@liuzqt Can you retry the failed GA task?
On the other hand, does this optimization have the same effect for Scala 2.13 as well?

According to the profiling flamegraphs, most overhead comes "itable stub", which refers to the internal method dispatch mechanism used by the Scala compiler for virtual method calls. After changing to Java implementation, those overhead has gone.
We haven't got environment to test agains scala 2.13, but I suppose the improvement should apply.

Changed LinkedList to ArrayList, in our benchmarks we observed more GC pressure with LinkedList, while ArrayList provided more robust performance boost.

Merged to master.


[Experiment] Convert parser namespace into a module

@typescript-bot user test tsserver
@typescript-bot test tsserver top100
@typescript-bot perf test this

Beating me to it! I would have sent this sooner but my stashed copy had too many conflicts...

The results make sense for the parser, but it does seem like the binder might have gotten slower, and overall time occasionally got slower too. I would chalk it up to variance (plus, bind-time is more sensitive since it's a smaller measurement), but just to be safe:
@typescript-bot perf test this


Question about updated translation rules and such.

As translations are now English-based, is the content in KC3Kai/src/data/translations/jp/README.md still true, and is it still true for other language?

what's the maximum length for the first parameter in battle.json's detection section?

Is it possible to test translations before they're merged and published onto chrome store?

Yes, the note still applies. Here's the order of events:

The very first intended effect was, using the Japanese source from the game's master data, we get a ship or item's translations from our built-in files, and if they don't exist, use the source, which is already in Japanese in game data. So for JP, leaving ships.json and items.json as empty files works.
@Diablohu did an enhancement to use English by default if it doesn't exist in the localized files. This meant JP will also use English by default so the intention above did not work, as pointed out by @kololz to me on wikia chat.
His update was really useful and opened the gates to better translation workflow. Like for translators to only need to include translatable words in settings and developers, not needing ids and values for localized files.
However, by that time, we still have problems syncing between languages, like additional developer entry still needed all languages to be updated, even though you only need to edit its developer description. Also, new translators must adapt to the huge settings object, which might get confusing.
Starting #355 I changed it back to "each language for itself, if you don't have it, we'll use the source". So the README.md udner JP again applies. The downside to this is that if a language do not have a translation for some words, it will either show a variable name like MenuPlayAPIDesc which is also an intended effect to point out to viewers that the translations are ongoing and they don't need to report each "hey I selected Filipino and this word is still in English"

Right now we're using short-hands for airbattle in all interfaces, such as "AS+" and not "Air Supremacy". So right now it doesn't matter how long the full name is, but is just good to have on hand for later use.

For engagement, it uses the long hand "Green T", and it has a considerably small space, but I cannot specify what's the max exactly since it will also depend on the font, where right now jp and scn uses a specialized font-family when used

Since you already have your fork, you can clone it to your desktop and load it onto you chrome via:

https://github.com/dragonjet/KC3Kai/wiki/Setup-development-build#step-2-install

so if I understand it correctly, ship/item name in traditional Chinese that share same name as they are in Japanese can still omit those entry, right?
What i were asking was the "detection" section.
i see.
the current terms.json do not ask translation to translate the word pixel, which i think it would be used in setting page?
And you have mentioned theme names are not there because name are not translated, but i believe names are usually getting translated in Chiense, Japanese, or Korean, even like for themes that have a name "V-Ten", it would be translated into something like ビー・テン in Japanese, 비텐 in Korean, or 威天 in Chinese according to its sound. Actually, I have just checked wikipedia and all 4 skins of it are translated in both Japanese and Korean, with 2 out of 4 skin name translated in Chinese language.

Yes if there are languages who also want to use the Japanese source, no need to have the entries. Just leave those two files as {}

Yes, for some technical terms, I think we do not need to translate them. I dont event recommend translating things like background-position on the settings description "See CSS background-position property" where I saw one language I was transferring, didn't have the background-position word. These are terms that we also use in code, if curious people were to search for it background-position keyword would still show the best results. About px being also used in code, would be not recommended to translate, in the mindset that the user is also being a dev himself inputting 20px in code.

I think this one is subjective really. Depends on our software if we want it to. Though in my perspective I prefer all theme names to be of the same names. That if we ever implement specialized themes like Echo's, it would include romanized "EchoHolic" in all languages in respect to the one who made it.

I believe the term used in setting panel is pixel/pixels not px.
Well, just that i think some people would not have enough English knowledge to understand what is the meaning of words like "dark" or "default".

@c933103
In the 4th point; I think your suggestion is also good for normal persons (everyone) who use this KC3改. Some people might have any no clues about what px mean is.
In the 5th point; How about "more guiding in the theme selection"? because these themes really have their formal names (aka. technical terms). And like in the @dragonjet 4th point, in the view of webdevs, the technical terms should be use in English for referencing BTW).
In the 6th point; For now this app has no font customizing mode yet. Might be good if we are implemented this for more flexibility (with no any problems with CSS-based coding).

@nicky008
for the point number 4, what i mean was the text being used in the interface now,
and for the point number 6, I just heard DJ said on another issue that it's hard to say about character limit in exact, due to difference in font usage or etc., (or was i misreading that part).

hm this is a tough call. Then in moving forward, we will add those pixels and theme names onto terms.json, though it is the choice of the translator to edit or not. We just have to note that it's not recommended but all boils down to, if they think its understandable enough in local language or not. I understand things get awkward in some languages, so at least I'd like to give them option to do so.

We apply custom fonts throughout the whole web page if using specific fonts founds in \src\library\modules\Translation.js under applyHTML function. If another language needs a special font, they may add it or create an issue for us to add it.

@c933103 Oh, I see... I tough you were suggesting on something other than the interface.

@dragonjet .. @kololz said (1) is not true for equipment now?

hm let me see, I will update the Localization article on the github wiki today

ok....?

up

still...is https://github.com/dragonjet/KC3Kai/commit/f7acc5b6705dd867d70b13c14b21d44e735e1fb4#diff-1d4d9b72605d0442271ca8623f986f2b needed?

right now, yes - even if the characters are the same with Japanese
the exception was made only for jp settings that will not extend romaji ships and items
unless you may want to add an exception for tcn as well?
are the characters the same?

50% equipment name and 80% ship name have same character in both tcn and jp. And to regular tcn user, untranslated name in jp are more understandable than en. Therefore I think it would be a good idea to add tcn into the exception. (scn's condition is similar too but the percentage of exactly same character would be lower than tcn)

now added to exceptions 712b0b9d3fd02cf37e8f0f3f805db8691237fbb1

because of age, will be closing this since it seems questions have been answered and the commit for the request has been made. if there are other translation concerns, we can create a new ticket.


example now selects providers that offer at least two qubits

Summary
The hello_quantum example was repeatedly failing because it usually selected the 1-qubit Armonk provider, but it needs two qubits to run.
Details and comments
I've added a filter to ensure that only usable providers are selected.
There are no automated tests for this example, but it now runs on my machine. (I know, I know...)

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.


"Add to homescreen" fail

After following the  instructions and having something that seams to work

manifest file served
default service worker activated and is running

The service worker source field identifies 2 issues

And trying to test the Add to homescreen function i always get the following error
Site cannot be installed: the page does not work offline

Any help? :)

The problem seems to be kind of solved if you update the service worker file.
First update serviceworker.js file path in settings.py as shown in the readme. Then add the following code to tell Django you actually want to get something if offline.
self.addEventListener("fetch", function(event){

});

Sources - SO post + this video


FEAT-#6522: Update to pandas 2.1.*

What do these changes do?

[x] first commit message and PR title follow format outlined here

NOTE:  If you edit the PR title to match this format, you need to add another commit (even if it's empty) or amend your last commit for the CI job that checks the PR title to pick up the new PR title.

[x] passes flake8 modin/ asv_bench/benchmarks scripts/doc_checker.py
[x] passes black --check modin/ asv_bench/benchmarks scripts/doc_checker.py
[x] signed commit with git commit -s
[x] Resolves #6522
[ ] tests added and passing
[x] module layout described at docs/development/architecture.rst is up-to-date

@vnlitvinov thanks for the review! Ready for another round.


Returning a fake response so test_no_domains will pass

jjsendor has a better fix he put in osxcollector. Cancelling this.


Fusion clause cause problem when used with loop-extract

The fusion transformation throw a null pointer exception when used as a clause in the loop-extract directive.

Fix with commit 22ae8e1a2d9063d5329c40d960b4ddc4dd6e06b4


adding batch methods for chain and cronjob

adding batch methods for the rest of the chains, and restablizing the job.


[WIP] Generate Python Users Manual from scripts

Another one to consider:

https://github.com/pystitch/stitch

Also:

https://github.com/jankatins/knitpy
https://github.com/mwouts/jupytext

NB: We ideally need something that goes from .pymd to .md, in order to integrate with rest of documentation system.

Given that we've decided on the .pymd → .md pathway (for the time being), which is supported by multiple tool chains, I'm going to stick with pweave for the moment given that it's working in our current configuration, and can be replaced at a later date.


update output method

Hey @pachay @kulapard, thanks for contributing.
@pachay is correct about set-ouput being deprecated, and @kulapard is 100% that proper multiline output must be used.
To make things quicker (as some housekeeping with changelog must also be done), I've merged changes from #23.
Included you both as contributors to CHANGELOG and Release notes. Hopefeully that's ok.
Thanks a ton


Implement a basic version of cross-repo depends-on

@stefannica It looks like I missed a part (commit amended). I've set that job to rebuild: https://ci.suse.de/job/cloud-ardana-testbuild-gerrit-test/15/

I think this will work, it would really help if the contained a high level overview of what is going on.

Any suggestions where I should put that? Or do you just mean better code comments?

What happens with circular depends-on. Simple case a review depends on it self.

It should be handled. As it processes each change it puts it into the changes dict with the gerrit id as the key. It also adds any dependencies to be processed to the change_ids, and if a dependency is encountered that is already in the changes dict it is skipped.

What happens with branching, what if the depends on change id does not have the branch the source review is against.

This is a good point. It should only be an issue when using a depends-on within the same repository. Ideally this shouldn't be done as you should push your commits in order to gerrit. However, if it does happen, the branch from the first change that is processed is used. The other changes are then git fetched and git merged onto that. So in theory it'll work, but it'd also be a mess and possibly incorrect.
Because we aren't doing strict gating, this may be acceptable. Otherwise we could try and catch this and fail the build.

Are cross branch depends-on supported? Are there limits to the context in which they are supported.

As above, it should work across different repos. Depends-on within the same repo will likely give odd results.

What happens if a change ID has more than one review associated with it it?

Good question. At the moment I think it'll fail. We should handle this appropriately.
This most commonly occurs with cherry-picks across branches. This means we'll need to track the branch state more closely and likely make some assumptions that if multiple changes with the same ID exist you likely want the one on the same branch.

What happens if all reviews associated with a change ID are already in the branch. (review state merged). If did not see where the review state was checked and I don't think I saw where the branch was checked to see if the change was already present.

So we could add some extra checks around this, but as far as I know it should be okay as is. Specifically it will perform a git merge on a sha that is already in the tree. This should be a safe operation and git will do nothing. (Please correct me if I'm wrong)

I don't think any of these should gate this change which I think will do what it intends, but it would be good to have any limitations noted in the file. Since that will help in debugging failures. Which can get very tricking with depends-on. I have seen depend-on chains exceeding twenty change ids. Which make finding issues very tricky.

Okay, I think this comes back to better comments.
Could you please refer me to a large depends-on change to help with my testing?
Thanks,
Josh

My suggestion would be to add some explanatory documentation to the code, e.g. a docstring at the top of the file explaining what the intend of the script is, and and a high level overview of how the script is intended to operate.

Also we probably should start to make a clear distinction between a Gerrit "change", hereafter referred to as a Gerrit patch#, which is a decimal number identifying a specific review patch set in Gerrit, vs a Gerrit "Change ID", which is a hexadecimal hash value preceeded by a capital "I".
A  gerrit "Change ID" represents a set of "equivalent" patches across various branches; equivalence here being that they achieve the same desired effect for the code stream they are found in. And since a Depends-On reference in a commit message, if we are emulating the Zuul mode of operation, is usually a Gerrit "Change ID", and not a patch#, when we lookup a Gerrit "Change ID" we should really be looking for the patch set associated with the branch we are working on.
This distinction is especially important for a number of reasons:

A patch set for the wrong branch may not apply cleanly and we would get a merge failure
A patch set for the wrong branch may be missing additional changes needed in the current branch to operate correctly
Or similarly the wrong patch set may contain additional changes that that cause a failure when used with the current branch
A patchset may not exist for the current branch, which may indicate an error/omission on someones part, e.g. cherry-picked a dependent patch without also cherry-picking patches it depends on, or removing unnecessary Depends-On references. For the Gozer/Zuul CI I'm not sure how this is handled... @brett060102 may have a better idea of how this causes grief, but it usually causes problems for gating if not check runs... Personally I think it might be better to have a clear failure if we detect a Depends-On reference that cannot be satisfied for the current branch, as that would force us to recognise that we need to either (a) cherry-pick in another change to this branch or (b) don't in fact need the references patches for this branch
Patch sets for different branches may have different Depends-On chain references, e.g. I cherry-pick a patch from master to stable/pike to pull in a fix for an issue that has been found... But the patch on master has an implicit dependency on some patch that has already been merged to master, but didn't need to be cherry-picked back to stable/pike for whatever reason, so now I need to cherry-pick both patches back, and add the appropriate Depends-On entry to my first patch to validate it in conjunction with the dependent patch...

I'm sure we can come up with more scenarios but that should cover most of the problem scenarios I have seen working with Gerrit/Zuul for HOS/Ardana and with upstream OpenStack over the past few years...
Also as @brett060102 pointed out we shouldn't be trying to apply changes that have already merged to the current branch, as that could either result in merge failures or really strange/subtle issues if subsequent patches on the current branch allowed it to merge again.
So in summary we need to keep in mind the following concepts:

a Gerrit "Change ID" identifies a set of one or more Gerrit "Patch Sets"
a Gerrit "Patch Set" identifies the set of specific changes, and possible Depends-On references for a specific branch
a Depends-On reference should only be resolved to a patch set in the context of the current branch, and should be skipped if we determine the referenced patch set has already been merged to the current branch.

Note that the proposed change is a move in the right direction, so I'm not saying we need to fix everything in it... Just that we have a ways to go yet. ;-)
But this patch is definitely a step in the right direction... Maybe just add some more detailed comments as to how it is intended to operate for now...

Nothing in this comment needs to happen in this PR.
Note that Zuul prevents the merge of a commit where any Depends-On targets are not yet merged. This implementation lacks that and thus allows merging of changes without their dependencies.

And since a Depends-On reference in a commit message, if we are emulating the Zuul mode of operation, is usually a Gerrit "Change ID", and not a patch#

While that was true in Zuul 2.5, newer versions prefer URLs to individual pull requests / patch sets / changes (which are all specific to one branch) and Change-Ids are deprecated (see https://zuul-ci.org/docs/zuul/user/gating.html#cross-project-dependencies ). Adopting this would allow supporting this feature also for Github PRs and OBS submit requests.
The implementation before and after this PR fails for Change-Ids that are on multiple branches:
$ curl 'https://review.openstack.org/changes/I32328c5206b9cd6fb8d8764c079f22b6ea8bfa9e/?o=CURRENT_REVISION&o=CURRENT_COMMIT'
Multiple changes found for I32328c5206b9cd6fb8d8764c079f22b6ea8bfa9e
$ curl 'https://review.openstack.org/changes/I32328c5206b9cd6fb8d8764c079f22b6ea8bfa9e/?o=CURRENT_REVISION'
Multiple changes found for I32328c5206b9cd6fb8d8764c079f22b6ea8bfa9e

If we need to support the old syntax then we can use a slightly different API (returns a list instead of one item):
https://review.openstack.org/changes/?q=I32328c5206b9cd6fb8d8764c079f22b6ea8bfa9e&o=CURRENT_REVISION&o=CURRENT_COMMIT

I'm still working on this. I've just pushed up where I'm up to with the refactor which is currently untested but I'm finishing for the day. I'm hoping I'll have time to test it later tonight to get it into a good intermediate state before I address all of the feedback (thanks for which btw :-))

This is ready for re-review now. Please note the updated PR message.

Yep, you can see one here: https://ci.suse.de/job/cloud-ardana-testbuild-gerrit/56/console


MUT saying header doesn't match

Literally modifying the the stock template, adding serial numbers (and that's it) in Excel, save file, import into MUT, says the header doesn't match. Here's a screenshot of the top of the file. It LOOKS fine to me... thoroughly confused.
Running version 6.

just tried with fresh template file, just adding serials in sublime. same error. =(

big fat user error. somehow didn't have latest templates.


Fix Fitness machine status characteristic id, verify example with Suito

https://github.com/zacharyedwardbull/pycycling/issues/33

Also fix one typo where 'f' was missing in f-string.

@tensorturtle thanks for this!
@coniesco please could you verify then I will merge it?

@zacharyedwardbull Looks good!
@tensorturtle Thank you!


bumped django test versions to final 3.2, and latest security releases

Description
Django test matrix bumps:

3.2 final
latest security releases

Types of Changes

[ ] Core
[ ] Bugfix
[ ] New feature
[x] Enhancement/optimization
[ ] Documentation

Issues Fixed or Closed by This PR

Checklist

[ ] My code follows the code style of this project.
[ ] My change requires a change to the documentation.
[ ] I have updated the documentation accordingly.
[ ] I have read the CONTRIBUTING document.
[ ] I have added tests to cover my changes.
[ ] I have tested the changes and verified that they work and don't break anything (as well as I can manage).

Hi, thanks for contributing to Strawberry 🍓!
We noticed that this PR is missing a RELEASE.md file. We use that to automatically do releases here on GitHub and, most importantly, to PyPI!
So as soon as this PR is merged, a release will be made 🚀.
Here's an example of RELEASE.md:
Release type: patch

Description of the changes, ideally with some examples, if adding a new feature.

Release type can be one of patch, minor or major. We use semver,so make sure to pick the appropriate type. If in doubt feel free to ask :)

Should I include a release file?

Should I include a release file?

No, it's fine to skip it for this PR 😊


Adding a parse_chat step to core agent loop

Description
Issue

In the spirit of our recent refactor push, we want the perceptual modules to be stateless. But the current perceive()  implementation in loco_mc_agent does not allow for a generic way for agents to pass inputs to their perceptual modules. For reference, it calls the perceptual modules as

for v in self.perception_modules.values():
            v.perceive(force=force)

Moreover, loco_mc_agent.perceive() also processes the incoming chat. This entanglement does not allow for a generic way to fetch inputs that self.perception_modules might need.

Fix

I pulled out the chat perceive into a separate function parse_chat. So the core event loop has an additional step now which is separate from the rest of perceive.
This allows downstream agents to implement their custom perceive functions to fetch states that perception might need, or use the default one if they don't need to. The locobot agent implements a custom perceive already to fetch states for its perceptual modules. See here.

Fixes # (issue)
This also fixes https://github.com/facebookresearch/droidlet/issues/577
Type of change
Please check the options that are relevant.

[x] Bug fix (non-breaking change that fixes an issue)
[ ] Proposes a change (non-breaking change that isn't necessarily a bug)
[x] Refactor
[ ] New feature (non-breaking change that adds a new functionality)
[ ] Breaking change (fix or feature that would break some existing functionality downstream)
[ ] This is a unit test
[ ] Documentation only change
[ ] Datasets Release
[ ] Models Release

Type of requested review

[ ] I want a thorough review of the implementation.
[ ] I want a high level review.
[x] I want a deep design review.

Testing
Tested that commands now work on the locobot agent

Checklist:

[x] I have performed manual end-to-end testing of the feature in my environment.
[ ] I have added Docstrings and comments to the code.
[ ] I have made changes to existing documentation where needed.
[ ] I have added tests that show that the PR is functional.
[ ] New and existing unit tests pass locally with my changes.
[ ] I have added relevant collaborators to review the PR before merge.

ok, at a high level, I strongly disagree with the approach in the PR- semantic parsing is supposed to go in the .perceive().  if the problem is that some state is needed, store whatever state is needed in memory.   to me this looks like just pulling the bits you don't like out of perceive and putting them in some other method, rather than fixing the issue...

look here:
https://github.com/facebookresearch/droidlet/blob/0ee6fa533318802538395203d53b295e2954382e/agents/loco_mc_agent.py#L310
in the craftassist agent were are doing this by calling super.perceive() here
https://github.com/facebookresearch/droidlet/blob/main/agents/craftassist/craftassist_agent.py#L234

In a few iterations, all of this will be properly handled by agent in the sense that all memory writes and reads from the parser will be mediated by the agent- for now, I think its best to keep this on locobot/robot how it is in craftassist agent

Fixing it in https://github.com/facebookresearch/droidlet/pull/614 instead.


Buyt 12 solving issues

pull request

Problems to be solved:

n+1
no tickets scraped out of INFOBUS service


Page crash iOS Safari in Private mode: infinite loop SessionStorage read warning

As of a few weeks ago, user reported issue with iOS Safari in Private mode. Normal mode works fine.
When loading the page with the AI script, the browser reports eventually in a message bar:
A problem occured with this webpage so it was reloaded.
and the page content has only the message:
A problem repeatedly occurred on "<URL>"..
Reproduced with Browserstack real browser on various devices and on my own iPhone.
The console log has repeatedly this message (up to 6400 times):
ai.0.js:1 AI (Internal): NONUSRACT_BrowserCannotWriteSessionStorage message:"Browser failed write to session storage. " props:"{exception:[object DOMException]{\code\:22,\name\:\QuotaExceededError\,\message\:\QuotaExceededError: DOM Exception 22\,\line\:1,\column\:16686,\sourceURL\:\https://az416426.vo.msecnd.net/scripts/a/ai.0.js\}}"
When Safari is in Private mode it doesn't allow to read session storage.
The AI code has a try/catch around that, but somehow the attempt is repeated.

I also have the issue for my web site.
For Safari in private browsing the browser hands out the window.localStorage objet but throws an exception when using the SetItem member.
The problem is in Util.ts. The _getStorageObject and _getSessionStorageObject functions return the storage object but when the write functions use the storage object it throws an exception causing the write functions to return false. This causes the caller loop.
The _getStorage*Object functions should use a pattern where they validate the storage object by performing a test write operation before handing out the object. This is a common pattern - e.g. see https://mathiasbynens.be/notes/localstorage-pattern.

Hope the AI team can review the PR which should fix this issue. I can't run the unit tests locally, the PS script opens an HTML page with empty unit tests. I can test the results via Browserstack.

@tsj0603 thanks for explaining that. While you were typing that out, I was making the PR, teamwork! :smile:

As confirmed in PR #146, the issue has been fixed and CDN has been updated. Again, thanks a lot for the quick fix, @AlexBulankou. :smile:


Load image tutorials

You didn't use keras_ds or feature_map_batch in model.fit

@fardindadashi Please provide details about what platform you are using (operating system, architecture). Also include your TensorFlow version. Also, did you compile from source or install a binary?
Make sure you also include the reproducible code snippet. If you are unclear what to include see the issue template displayed in the Github new issue template.
We ask for this in the issue submission template, because it is really difficult to help without that information. Thanks!

I'm just saying the tutorials in https://www.tensorflow.org/tutorials/load_data/images the  keras_ds or feature_map_batch have not been used in model.fit function. Is that OK?

@fardindadashi In the mentioned tutorial, the model.fit uses the dataset of image_paths and image_labels that is ds. Thanks!

Automatically closing due to lack of recent activity. Please update the issue when new information becomes available, and we will reopen the issue. Thanks!


Last snippet is error.

https://github.com/dotnet/samples/blob/master/csharp/roslyn-sdk/SemanticQuickStart/Program.cs must use Snippet13

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: f68fa439-55b3-ff14-e2d1-2373615d978b
Version Independent ID: 33c84a67-2ca2-fd1c-3739-60ad3df79f8e
Content: Get started with semantic analysis
Content Source: docs/csharp/roslyn-sdk/get-started/semantic-analysis.md
Product: dotnet-csharp
GitHub Login: @BillWagner
Microsoft Alias: wiwagn

You're right. Do you want to submit a Pull Request that makes that change? If you do and don't know how, you can follow this guide. If you have any questions, feel free to ask here.


feat: handle exited/exiting validators during top up

Handle deposit for exiting and exited validators:

If the validator is exiting, move the deposit to the end of state.pendingBalanceDeposits via depositsToPostpone
If the validator is already exited, directly increase its balance without consuming churn (state.depositBalanceToConsume)

Spec PR: https://github.com/ethereum/consensus-specs/pull/3776

:tada: This PR is included in v1.22.0 :tada:


⚠ Samsung has degraded performance

In 9ee4aab, Samsung (https://samsung.com) experienced degraded performance:

HTTP code: 200
Response time: 1871 ms

Resolved: Samsung performance has improved in 0e75d10.


twister: add --aggrissive-no-clean option

Useful option during development and when applying smaller change to the
code and verifying changes using tests. This works fine as long as no
dependencies or major changes are done, i.e. when editing few files and
in a subsystem. Use with caution and use on your own risk :-)
When building multiple tests, this provide significant boost, up to 300%
faster than when rebuilding everything from scratch or when re-running
cmake every single time.
Signed-off-by: Anas Nashif<EMAIL_ADDRESS>

nit. typo in commit


Goo Blobs not working in Safari

🐛 Bug Report
Goo Blobs are not working in Safari.
To Reproduce
Open https://www.react-spring.io/docs/hooks/use-trail in Safari. Goo Blobs appear to be buggy.

Expected behavior
Must be same like chrome.
Link
Checkout the blob example in Safari.
https://www.react-spring.io/docs/hooks/use-trail
Environment

One used in the documentation

Probably related to this:
https://bugs.webkit.org/show_bug.cgi?id=184601

The issue here is that WebKit doesn't support references to SVG filters when doing "accelerated" filters.


Fix feedback button style

Depending on the theme, the button could be hard to spot:

This changes the button to use an IconButton provided by Storybook, same as the other IconButtons in the toolbar, the  styles are taken from @etchteam/storybook-addon-css-variables-theme

:tada: This PR is included in version 4.0.0 :tada:
The release is available on:

npm package (@latest dist-tag)
GitHub release

Your semantic-release bot :package::rocket:


new email layout

refs #39

Codecov Report

Merging #50 into master will decrease coverage by 57.47%.
The diff coverage is 49.12%.

@@             Coverage Diff             @@
##           master      #50       +/-   ##
===========================================
- Coverage     100%   42.52%   -57.48%
===========================================
  Files           1        2        +1
  Lines           9       87       +78
===========================================
+ Hits            9       37       +28
- Misses          0       49       +49
- Partials        0        1        +1

Impacted Files
Coverage Δ

mail/mailer.go
35.89% <49.12%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update f3d41cc...c99dc7b. Read the comment docs.


Add fallback assistant suggestions

Description

This PR resolves https://github.com/dust-tt/dust/issues/3277.
This PR introduces the concept of fallback suggestions to the assistant suggestions in our input bar. The primary goal is to initially recommend assistants from the user's personal list. If the list yields less than 7 suggestions, it then includes accessible assistants not present on the user’s list. The two sets of suggestions will be sorted respectively. We may revisit this sorting approach in the future.
This enhancement does not impact the speed of suggestions. Initially, as we fetch all suggestions, only those from the user's list will be displayed.

In this GIF, dusttestflavien isn't in my list, but is suggested when we have less than 7 suggestions.
Risk

Deploy Plan

Code looks good, do we intend the loading to be fast with the call to "all"?
However I wonder about the user side
As a user, I don't like having a cluttered list of suggestions;
Also, if I removed an assistant from my list, I'll be annoyed to see it pop up when I type (even if it is not selected in the end); the list narrowing down is kind of a visual confirmation that I typed the I wanted right
I feel that this makes the distinction "in my list" / "not in my list" less useful. If we are sure we want to add these suggestions, then as a user I would prefer:
that they only appear if none of my agents match;
that the UI makes it clear (ie. a small title "assistants shared by other users" in the suggestions) so that I know that nothing matches from my list and that what I'm shown is a suggestion that's from outside it
ccing @Duncid to discuss

Code looks good, do we intend the loading to be fast with the call to "all"?
I wonder about the user side of things, thus the changes requested, see com below

As mentioned in the PR description, retrieving "all" the assistants doesn't significantly affect our suggestions because they serve as a fallback. Therefore, this won't worsen today's performances 🙃.

Ok understood, IMHO the visual separation would make a diff, and solve the pb from the strange effect mentionned by flavien (since it makes the user understand that new suggestions are coming).
But fine to see how it goes like this and change after


Instance image name retrieval might need caching

it needs to be checked if openstack4j caches image information; if not caching image name by id needs to be introduced

#9 #9


About Performance Issues Using Accessors

I found that it was slower when using accessors than when using arrays directly.For example,
z[i][j] += x[i][k] * y[k][j]; is faster than acc_out[Point<2>(j,i)]+=acc_x[Point<2>(k,i)]*acc_y[Point<2>(j,k)];.
This problem is particularly obvious when I try to write GEMM, by creating a two-dimensional array, reading matrices A and B into the two-dimensional array through the value of acc in the accessor, calculating it, and then writing it back to the output region. It is much faster than using the accessor directly for matrix multiplication calculations. Because if I use a two-dimensional matrix, acc_A, acc_B, and acc_C each only needs to be accessed N*N times, and when I use accessors directly for matrix budgeting, each accessor needs to be accessed N*N*N times.When I try a matrix multiplication of 256 * 256, it slows me down by more than ten times.
What's the reason for the slower accessors, and is it a good idea to read into an array and recalculate?
By the way, why is the performance of
const FieldAccessor<READ_ONLY,double,1,coord_t,Realm::AffineAccessor<double,1,coord_t>>acc_x(regions[0], FID_X);
far greater than that of
const FieldAccessor<READ_ONLY,double,1> acc_x(regions[0], FID_X);

Because if I use a two-dimensional matrix, acc_A, acc_B, and acc_C each only needs to be accessed NN times, and when I use accessors directly for matrix budgeting, each accessor needs to be accessed NN*N times.When I try a matrix multiplication of 256 * 256, it slows me down by more than ten times.

I don't understand this. Can you explain again? Using an accessor in the way that you've described should compile down to the same code as the direct array access.
One difference I see with your accessor statement is that you are accessing the arrays with different strides than in the non-accessor case, which greatly affects the locality that you are getting on your accesses (i.e. acc_out is being accessed (j, i), while z is being accessed (i, j), and others...)
AffineAccessor is much faster than a generic accessor as it is heavily templated to make sure that affine accessor accesses compile down into direct memory accesses.

@rohany
Here's the thing, sir.

It seems that you can get better performance by taking the data out of the Accessor, calculating it, and then saving it back, than by using more Accessors.

I want to know what overhead the accessor adds compared to direct array access.

Even though the AffineAccessor is heavily templated, I am curious wether the compiler can vectorize the access, while using the raw pointer, the compiler could easier apply the vectorization optimization.

May I see the entire code you’re comparing, in particular the instantiation of the accessors?
There could be a few things wrong — you might have bounds checks on for accessors, or you might be mapping an instance that isn’t laid out the way you expect (the default mapper allocates instances in column major).
the best thing to do here though is to just look at the assembly of your loops — if everything is working as expected, the assembly should look nearly the same between accessor and no accessor.

@eddy16112 I don't think the compiler is vectorizing the raw pointer, I'm using g++, and I'm using -O0 optimizations.

@rohany
`void gemm_task(const Task *task,
const std::vector &regions,
Context ctx, Runtime *runtime)
{
assert(regions.size() == 3);
assert(task->regions.size() == 3);
const int point = task->index_point.point_data[0];
const int row = ((const int)task->args);
const FieldAccessor<
LEGION_READ_ONLY,double,2,coord_t,
Realm::AffineAccessor<double,2,coord_t>>
acc_x(regions[0], FID_X);
const FieldAccessor<
LEGION_READ_ONLY,double,2,coord_t,
Realm::AffineAccessor<double,2,coord_t>>
acc_y(regions[1], FID_Y);
const FieldAccessor<
LEGION_WRITE_DISCARD,double,2,coord_t,
Realm::AffineAccessor<double,2,coord_t>>
acc_out(regions[2], FID_Z);
Rect<2> rect = runtime->get_index_space_domain(ctx,
task->regions[2].region.get_index_space());
for(int i=0;i<row;i++){
for(int j=0;j<row;j++){
for(int k=0;k<row;k++){
acc_out[Point<2>(j,i)]+=acc_x[Point<2>(k,i)]*acc_y[Point<2>(j,k)];
}
}
}
}     IndexLauncher gemm_launcher(GEMM_TASK_ID, color_is,
TaskArgument(&num_elements, sizeof(num_elements)), arg_map);
gemm_launcher.add_region_requirement(
RegionRequirement(input_lp, 0/projection ID/,
READ_ONLY, EXCLUSIVE, input_lr));
gemm_launcher.region_requirements[0].add_field(FID_X);
gemm_launcher.add_region_requirement(
RegionRequirement(input_lp_y, 0/projection ID/,
READ_ONLY, EXCLUSIVE, input_lr_y));
gemm_launcher.region_requirements[1].add_field(FID_Y);
gemm_launcher.add_region_requirement(
RegionRequirement(output_lp, 0/projection ID/,
WRITE_DISCARD, EXCLUSIVE, output_lr));
gemm_launcher.region_requirements[2].add_field(FID_Z);`
The code looks like this, by contrast, just fetching the data from the accessor in gemm_task and calculating it.

What about the performance with O2 or O3? Accessors only get optimizes away if you turn on optimizations

As I mentioned before, these two codes aren’t exactly comparable either — you’re making accesses of different strides / in different index orders between the accessor version and the non accessor version

I'm going to try O2 or O3, acc_out[Point<2>(j, i)] I think it's going to be out[i][j], because when I try the pointer pir++ of a two-dimensional array, From acc_out[Point<2>(j, i)] to acc_out[Point<2>(j+1, i)]

@rohany yes! When I use O2 optimization, I can already get a huge performance boost, very close to direct array computing, although still a little slower.

I'm going to try O2 or O3, acc_out[Point<2>(j, i)] I think it's going to be out[i][j], because when I try the pointer pir++ of a two-dimensional array, From acc_out[Point<2>(j, i)] to acc_out[Point<2>(j+1, i)]

As i mentioned above, the default mapper (which I assume you're using) allocates instances in fortran order,  or column major order. If you change line https://gitlab.com/StanfordLegion/legion/-/blob/master/runtime/mappers/default_mapper.cc?ref_type=heads#L2253 to be reversed, like https://github.com/rohany/taco/blob/DISTAL/DISTAL/runtime/taco_mapper.cpp#L391, then the instance will be allocated in row-major order, where doing acc_out[Point<2>(i, j)] will give you the memory location you expect.

@rohany Well, I know this thing, my benefactor, I wish you health and happiness every day.


bug hunt/fix reconfiguration

Postponed due to out of memory error: continuing with #298
Reconfiguration at least via http is not working for me atm. Going on a bug hunt. Will try to automate corresponding integration tests alongside. Will post update/notes here.

[ ] there's bitfiles mentioning ids (IDBitFiles) in the bitfiles scripts folder. Are those using skeleton version 01 or 02?
Check IDBitfiles with skelver 01 and env5.1 (env5 revision 1)

[ ] via http
[ ] via usb protocol

https://github.com/es-ude/elastic-ai.runtime.enV5/tree/refs/heads/302-bug-huntfix-reconfiguration
fixes a bug in the usb protocol implementation that broke usb bit stream flashing

postponed for now since #306 fixes the bug in usb protocol that was preventing bitstream updates via usb.


incorrect bcc

Hi,
I've noticed that at line 35 you concat the bcc addresses in the to instead of using the bcc_address option.
Using the "to" for bcc you lose the original recipient of the email and also it sends more emails and false your data stats on mandrill dashboard.

@devinus

Multiple BCC email addresses are appended to the to as type bcc (https://github.com/Rebelmail/nodemailer-mandrill-transport/blob/master/lib/mandrill-transport.js#L47) following the Mandrill API docs (https://mandrillapp.com/api/docs/messages.JSON.html#example-errorblock-send).
The bcc_address appears to only support one recipient. I'm not sure what kind of behavior you're expected.

@david1983 are you looking to use bcc_address or simply add several bcc recipients?

This behavior is not the behavior I expected from a bcc. If you send an email from Gmail with a bcc, the "to" recipient gets the email without being able to see the bcc. The "bcc" recipient gets the email and is also able to see the "to" recipient. What I get from this package is both the "bcc" and the "to" recipients get emails that have them as the "to" recipient even though one of them was "bcc"ed.


add abapGit-Repository-Notification to entry.txt

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.


SAC doesn't run with livy on zeppelin but works fine on pyspark-shell on master node

Hi Team,
I have installed and configured SAC on master node and it works fine with pyspark shell. However, when I try to run the same code using livy on zeppelin it gives me the below error (I have checked without configuring SAC and running via livy and works fine as well) -
20/10/28 02:57:27 INFO Client: Uploading resource file:/usr/lib/livy/rsc-jars/netty-all-4.1.17.Final.jar -> hdfs://ip-172-26-51-231.us-west-2.compute.internal:8020/user/ff08/.sparkStaging/application_1603849413644_0050/netty-all-4.1.17.Final.jar
20/10/28 02:57:27 INFO Client: Uploading resource file:/usr/lib/livy/rsc-jars/livy-rsc-0.6.0-incubating.jar -> hdfs://ip-172-26-51-231.us-west-2.compute.internal:8020/user/ff08/.sparkStaging/application_1603849413644_0050/livy-rsc-0.6.0-incubating.jar
20/10/28 02:57:27 INFO Client: Uploading resource file:/usr/lib/livy/repl_2.11-jars/commons-codec-1.9.jar -> hdfs://ip-172-26-51-231.us-west-2.compute.internal:8020/user/ff08/.sparkStaging/application_1603849413644_0050/commons-codec-1.9.jar
20/10/28 02:57:27 INFO Client: Uploading resource file:/usr/lib/livy/repl_2.11-jars/livy-core_2.11-0.6.0-incubating.jar -> hdfs://ip-172-26-51-231.us-west-2.compute.internal:8020/user/ff08/.sparkStaging/application_1603849413644_0050/livy-core_2.11-0.6.0-incubating.jar
20/10/28 02:57:27 INFO Client: Uploading resource file:/usr/lib/livy/repl_2.11-jars/livy-repl_2.11-0.6.0-incubating.jar -> hdfs://ip-172-26-51-231.us-west-2.compute.internal:8020/user/ff08/.sparkStaging/application_1603849413644_0050/livy-repl_2.11-0.6.0-incubating.jar
20/10/28 02:57:27 INFO Client: Uploading resource file:/usr/lib/spark/conf/atlas-application.properties -> hdfs://ip-172-26-51-231.us-west-2.compute.internal:8020/user/ff08/.sparkStaging/application_1603849413644_0050/atlas-application.properties
20/10/28 02:57:27 INFO Client: Uploading resource file:/usr/lib/spark/R/lib/sparkr.zip#sparkr -> hdfs://ip-172-26-51-231.us-west-2.compute.internal:8020/user/ff08/.sparkStaging/application_1603849413644_0050/sparkr.zip
20/10/28 02:57:27 INFO Client: Uploading resource file:/usr/lib/spark/python/lib/pyspark.zip -> hdfs://ip-172-26-51-231.us-west-2.compute.internal:8020/user/ff08/.sparkStaging/application_1603849413644_0050/pyspark.zip
20/10/28 02:57:27 INFO Client: Uploading resource file:/usr/lib/spark/python/lib/py4j-0.10.7-src.zip -> hdfs://ip-172-26-51-231.us-west-2.compute.internal:8020/user/ff08/.sparkStaging/application_1603849413644_0050/py4j-0.10.7-src.zip
20/10/28 02:57:27 WARN Client: Same name resource file:///usr/lib/spark/python/lib/pyspark.zip added multiple times to distributed cache
20/10/28 02:57:27 WARN Client: Same name resource file:///usr/lib/spark/python/lib/py4j-0.10.7-src.zip added multiple times to distributed cache
20/10/28 02:57:27 INFO Client: Uploading resource file:/mnt/tmp/spark-7b32ab91-1a6d-4470-9038-631a08b155a4/__spark_conf__3554264488014972258.zip -> hdfs://ip-172-26-51-231.us-west-2.compute.internal:8020/user/ff08/.sparkStaging/application_1603849413644_0050/spark_conf.zip
20/10/28 02:57:27 INFO SecurityManager: Changing view acls to: livy,ff08
20/10/28 02:57:27 INFO SecurityManager: Changing modify acls to: livy,ff08
20/10/28 02:57:27 INFO SecurityManager: Changing view acls groups to:
20/10/28 02:57:27 INFO SecurityManager: Changing modify acls groups to:
20/10/28 02:57:27 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(livy, ff08); groups with view permissions: Set(); users  with modify permissions: Set(livy, ff08); groups with modify permissions: Set()
20/10/28 02:57:28 INFO Client: Submitting application application_1603849413644_0050 to ResourceManager
20/10/28 02:57:28 INFO YarnClientImpl: Submitted application application_1603849413644_0050
20/10/28 02:57:28 INFO Client: Application report for application_1603849413644_0050 (state: ACCEPTED)
20/10/28 02:57:28 INFO Client:
client token: N/A
diagnostics: [Wed Oct 28 02:57:28 +0000 2020] Application is Activated, waiting for resources to be assigned for AM.  Details : AM Partition = CORE ; Partition Resource = <memory:491520, vCores:64> ; Queue's Absolute capacity = 100.0 % ; Queue's Absolute used capacity = 0.0 % ; Queue's Absolute max capacity = 100.0 % ;
ApplicationMaster host: N/A
ApplicationMaster RPC port: -1
queue: default
start time:<PHONE_NUMBER>513
final status: UNDEFINED
tracking URL: http://ip-172-26-51-231.us-west-2.compute.internal:20888/proxy/application_1603849413644_0050/
user: ff08
20/10/28 02:57:28 INFO ShutdownHookManager: Shutdown hook called
20/10/28 02:57:28 INFO ShutdownHookManager: Deleting directory /mnt/tmp/spark-7b32ab91-1a6d-4470-9038-631a08b155a4
20/10/28 02:57:28 INFO ShutdownHookManager: Deleting directory /mnt/tmp/spark-817fe12d-4410-4781-a29f-affd97ce07f1
stderr:
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/lib/spark/spark-atlas-connector-assembly-0.1.0-SNAPSHOT.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/lib/spark/jars/slf4j-log4j12-1.7.16.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
YARN Diagnostics:
Application application_1603849413644_0050 was killed by user livy at <IP_ADDRESS>
at org.apache.zeppelin.livy.BaseLivyInterpreter.createSession(BaseLivyInterpreter.java:360)
at org.apache.zeppelin.livy.BaseLivyInterpreter.initLivySession(BaseLivyInterpreter.java:210)
at org.apache.zeppelin.livy.LivySharedInterpreter.open(LivySharedInterpreter.java:59)
at org.apache.zeppelin.interpreter.LazyOpenInterpreter.open(LazyOpenInterpreter.java:69)
at org.apache.zeppelin.livy.BaseLivyInterpreter.getLivySharedInterpreter(BaseLivyInterpreter.java:191)
at org.apache.zeppelin.livy.BaseLivyInterpreter.open(BaseLivyInterpreter.java:164)
at org.apache.zeppelin.interpreter.LazyOpenInterpreter.open(LazyOpenInterpreter.java:69)
at org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer$InterpretJob.jobRun(RemoteInterpreterServer.java:616)
at org.apache.zeppelin.scheduler.Job.run(Job.java:188)
at org.apache.zeppelin.scheduler.FIFOScheduler$1.run(FIFOScheduler.java:140)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Is there anything that I am missing?

I was able to resolve it by copying the uber jar and atlas-application.properties to all the core nodes


LIcense

@chufengt  can you please add the license does it has MIT ?
Thank sin advance

I add the Apache License 2.0. Hope this helps.


Fix settings icon on iOS 12, replace with new icon

Changes the settings icon to a real svg (and not a SVG with inline image data), which allows it to be recolored and lets it work on iOS 12.0.x
Note: This uses a new icon from the latest Figma spec
Linked issues:
Fixes #528
The back arrow is fixed for iOS 12.0.x in #686
Screenshots:
Before:

After:

How to test
Verify the icon is visible on a simulator running iOS 12.0.x

Jared approved the new icon.


[SPARK-27280][SQL]infer more filter from join or condition

What changes were proposed in this pull request?
current spark can handle this query to infer fitler from join condition:
select * from t1 join t2 where ((t1.a = t2.d and t1.b = 1) or (t1.a = t2.d and t1.b = 5))

(t1.b =1 or t1.b =5) can be infered, and pushed to table t1.

but it can not handle following query:
select * from t1 join t2 where (t1.a = t2.d and t1.b = 1 and t2.e=3 ) or (t1.a = t2.d and t1.b = 5 and t2.e=7)

(t1.b=1 or t1.b=5) should be infered and push to t1
(t2.e=3 or t2.e=7) should be infered and push to t2

How was this patch tested?
unit test added

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/9357/
Test PASSed.

Test build #104008 has started for PR 24228 at commit 879c98e.

Is this SQL grammar supported by Hive or standard SQL?

Test build #104008 has finished for PR 24228 at commit 879c98e.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/104008/
Test PASSed.

#24973  I want to do the same thing ...

I also think we need not a band-aid fix, but a general solution for this issue as @cloud-fan suggested in #24973. Have you checked the previous activity for the CNF normalization? https://issues.apache.org/jira/browse/SPARK-6624

I also think we need not a band-aid fix, but a general solution for this issue as @cloud-fan suggested in #24973. Have you checked the previous activity for the CNF normalization? https://issues.apache.org/jira/browse/SPARK-6624

thanks for Issue link. I will work on this in partition pureing part.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/15707/
Test PASSed.


Service Discovery Tag Order in Azure

ISSUE TYPE

Bug Report

AS3 BUILD/ VERSION

f5-appsvcs-3.13.1-1

BIGIP VERSION

<IP_ADDRESS>

SUMMARY

Tag order in Azure determines if service discovery work:
I've noticed that the tag used for service discovery in azure must
be the 1st tag configured on the VM or NIC.  If not, service discovery
will not add the resource to the pool.

STEPS TO REPRODUCE

Create a resource with tags:
env: dev
name: demo-dev-app

The run the declaration:
{
     "class": "AS3",
     "action": "deploy",
     "persist": true,
     "declaration": {
         "class": "ADC",
         "schemaVersion": "3.0.0",
         "id": "example-declaration-01",
         "label": "Sample 1",
         "remark": "Simple HTTP application with round robin pool",
         "www": {
             "class": "Tenant",
             "Application_1": {
                 "class": "Application",
                 "template": "http",
             "serviceMain": {
                 "class": "Service_HTTP",
                 "virtualAddresses": [
                     "<IP_ADDRESS>"
                 ],
                 "pool": "web_pool"
                 },
                 "web_pool": {
                     "class": "Pool",
                     "monitors": [
                         "http"
                     ],
                     "members": [
                        {
                        "servicePort": 80,
                        "addressDiscovery": "azure",
                        "updateInterval": 10,
                        "tagKey": "name",
                        "tagValue": "demo-dev-app",
                        "addressRealm": "private",
                        "resourceGroup": "demo-dev-rg",
                        "subscriptionId": "****",
                        "directoryId": "****",
                        "applicationId": "****",
                        "apiAccessKey": "****",
                        "credentialUpdate": false

                         }
                     ]
                 }
             }
         }
     }
 }

You'll notice the pool members are empty.  Go into the Azure Portal or CLI and delete the env tag.  The pool members will show up.  Add the env tag back, which will make the name tag the 1st tag, and the pool members will stay in the member list.

EXPECTED RESULTS

AS3 Service Discovery should look at all tags not just the 1st.

ACTUAL RESULTS

I have to use the 1st tag on the resource as the service discovery tag.

Closing for inactivity. If you are still encountering issue, please reopen this ticket for a file a new one.