Modifications to req.request via middleware are restricted - Expected behavior?

As a part of my project's authentication I need to be able to add a __nonce property to the request body.
My initial approach looked something like this:
  const apolloNetworkInterface = createNetworkInterface('http://localhost:1234/graphql');
  apolloNetworkInterface.use([{
    applyMiddleware(req, next) {
      req.request.__nonce = uuid.v4();
      // ...
      next();
    }
  }]);

However, this modification to the req.request object isn't persisted in the network request:

It looks like the reason for this is because before the request is actually sent out, it is processed by the printRequest function here; the definition of which is here.
I'm not sure what the purpose of this function is except, possibly to produce the outcome that is actually the cause of my issue. So my main question, is: Is this expected behavior?
I'll be submitting a PR to "fix" this issue as it's the only way to currently work around my issue but I would like some feedback on what the purpose of printRequest is.

Further, it looks like printRequest was added at 2a36e65, before which the request was just sent as is. This leads me to believe that there was a reason but it's still not clear to me.

Update: I've since noticed the subtle print(query) and the corresponding type differences between the Response and PrintedResponse properties which may account for the existence of the printResponse method.

As noted by the reference comments, I have the fix for this issue but am still working on adding tests for it.

We have printRequest because the network interface actually gets the query AST, rather than the query string. It would be great to pass through any additional properties.
However, I think the best option is to use an HTTP header for this, instead of an extra field on the body.

@stubailo I agree in principle that it would be best to stick to headers for this specific use case; however, our existing authentication mechanism as well as other use cases either can't or won't.
Regardless, thanks for the support 👍


Does not "Pin to Start"

Hello
I downloaded the latest Windows 10  build (VSCodeUserSetup-x64-1.60.0.exe)  and installed Visual Studio Code. The desktop icon is created , however when I search for "Visual Studio Code" on the Windows 10 Start menu and select the "See more information" arrow alongside it i don't see the option to "Pin to Start".
After trying various ways to do this I have determined that there is no longer a way to Pin the Visual Studio Code to Start menu anymore.
Is this correct?

however when I search for "Visual Studio Code" on the Windows 10 Start menu and select the "See more information" arrow alongside it i don't see the option to "Pin to Start".

Can you show me a screenshot of this?

This is Visual Studio 2019 community. Here you can see the "Unpin from Start", because i have already pinned it to the Start.

If you choose Open file location, where does that point to?

It points to the shortcut on my desktop. I think this was created when i did the install.

I see. If you reinstall VS Code, make sure you pick the option to add a shortcut to the Start Menu. Once the installer does that, you should be able to easily pin it.

That did not work as there is no " If you reinstall VS Code, make sure you pick the option to add a shortcut to the Start Menu"
Can you show me what you mean?

Oh I see. We add it by default, without any option. I wonder why Windows is borking the shortcut on your machine, though. I recommend reaching out to Windows support and asking.


Using Iohook in Electron 12

I am trying to add iohook to my electron app, but can't get it to work.  I think the problem is that there is no prebuilt version for the most recent electron runtime.
I used electron-forge with webpack as the boilerplate for my app.
My Electron version ist: v12.0.1
My Node version ist: v14.16.0
Do you have any solution on how to (manually) add iohook to my app or do you have a timeline for support of the latest electron version? Excuse my question it it is nooby, I am a beginner and did not find any helpful instruction (for me) on how to use native node modules (e.g. I followed the instruction on https://www.electronjs.org/docs/tutorial/using-native-node-modules and ran .\node_modules.bin\electron-rebuild.cmd after installing iohook with npm, but that did not work).
Thank you very much for any help!

Got same problem with electron v 12.0.0.

Same issue with 12.0.2

Same issue with 12.0.2, does anyone know any alternative to iohook? (I'd like to capture key events out of the browser and terminal)

Guys. It all already documented here https://wilix-team.github.io/iohook/manual-build.html

@WilixLead
Thank you for the quick response!
Sorry, I'm having some trouble trying to understand the steps only by reading your documentation.
I'm on Windows, node version: 14.15.5, electron version: 12.0.1.
I'm kind of new on this so I don't usually manually build packages. Where can I find more help on manually building the package?

Please wait today. I doing update for Electron v12. Should help you

Please wait today. I doing update for Electron v12. Should help you

Thanks a lot sir! Can you notify me whenever you make the update please? I can give you feedback on it!

@WilixLead Good morning!
I saw you updated the package to electron version 12, but now I encountered a new problem. I get thrown an error saying:
Error: \?\D:\myprojects\full-projects\bindbox\node_modules\iohook\builds\electron-v87-win32-x64\build\Release\iohook.node is not a win32 valid aplication.
Any ideas?

Done. Please check it. Closing issue for now, if something wrong just open it again


Update device_allocations.md

Content update.
Context
Changes proposed in this pull request
Guidance to review

Hi @fofr - are you able to help us release this update please? Thanks

Hi @benilovj - are you or your team able to help release this update please? Thanks


Fixed month_shortname/1 Examples

Summary of changes
I'll review the commits, so I mostly want to understand the "why" rather than the "what"
Checklist

[ ] New functions have typespecs, changed functions were updated
[ ] Same for documentation, including moduledocs
[ ] Tests were added or updated to cover changes
[ ] Commits were squashed into a single coherent commit

Coverage increased (+0.08%) to 67.219% when pulling 63c258b1898ef49c5ff90c9d7c7f4f2b25263010 on lpedromachado:patch-1 into 0b905092ea064614283ddbb0454e58016142d290 on bitwalker:master.

Thanks! Good catch!


block specific url in puppeteer [REQUEST]

i can code javascript/c# to block specific urls but if you could add a block for it which doing it automaticly it would help me really much thanks!

I cannot understand what you mean, please provide an example or some code.

if (
// ...
req.url() === "https://cdn-prod.securiti.ai/consent/sdk-stub.js"
) {
req.abort();
}

for make Puppeteer faster, when i go to setting and click load only documents and javascripts site isnt working so the only way is to block a site custom like the example that i made

Okay I understood now, I can implement this.

Okay, thanks!


Error: StageName cannot be updated

Hello,
This morning we tried to deploy a version of our application and have the following cloudformation error.

Another issue is having the same error and posted about it in the discussion forums.  We use SAM to deploy everything so we are sort of at a stand still.  Please let me know if you need any further clarification!
https://forums.aws.amazon.com/thread.jspa?messageID=797010#797010

Hello, looks like there was a regression with a recent deployment. Can you check again if everything is fine? We have reverted our updates.

Looks good, thanks so much!

This should be fixed now.. can you confirm?

Hi, we're having the same issue. Yesterday everything was fine, but now it keeps trying to update the stage name and failing to update, then rolls back. The issue went away for a moment maybe an hour ago, but it's back again and we can't get past the deployment. Please help!!

it seems it's creating an additional stage "Stage", if I specify 'StageName: Stage' I get 'Stage already exists', if I change it to 'StageName: stage' then I see two stages 'stage' and 'Stage'. Didn't see this yesterday.

@mrybertt We have rolled back the change that caused original regression. Can you tell me the region this problem still exists in? When did you first create the stack?
@cheeham51 This additional Stage was a bug that we were fixing as part of the deployment that got rolled back. This will be fixed soon.

Currently we are having this bug now (again) on eu-west-1.

We're seeing this in eu-west-1 as well. Our StageName is v1 but previously when deploying our generated API Gateway would contain an auto-generated stage named Stage along with our defined v1 stage.

Maybe this is helpful... As we first experienced the error on Friday we deleted our stack and recreated it. I noticed that the stage "Stage" was not created anymore. With that recreated stack updates where possible.
The change that leads to the regression was rolled back then. Now with that version updates are failing again. I deleted the stack again and recreated it and now updates are working again.

We're trying this now on our development stack right now but we are also hoping to update our production stack today. This stack hasn't been updated in around a week so it hopefully hasn't been affected by this issue however I'm still nervous deploy because deleting and recreating will be a bigger deal in our prod environment.

We are having this issue on eu-central (Frakfurt)

We are having the same issue in us-east-1. Please fix/share a workaround asap. Thank you!

Same on eu-west-1

can confirm this is still ongoing on eu-west-1

Same issue on us-east-1 using serverless transform and Events.

Looking at this more closely now..

Looks like a known issue. Here is a workaround to unblock your stack until we fix it on our side.
You need to trigger a redeployment of API resources to unblock the stack. Here is how you do it.

If you use an external Swagger file, just add a comma, space or an extra character that will change the Swagger's SHA. Deploy your stack now, and it should work.

If you use an implicit API (the ones defined using Events property on AWS::Serverless::Function), you can add a new dummy API to your Events and redeploy. Example:

RestApiFunction:
     Type: AWS::Serverless::Function
     Properties:
       Handler: index.handler
       Runtime: nodejs4.3
       CodeUri: s3://bucket/code.zip
       Events:
         AddItem:
           Properties:
             Method: get
             Path: /add
           Type: Api
        DummyApiForWorkaround:
           Properties:
             Method: get
             Path: /some-dummy-api
           Type: Api

Once the deployment is successful, you can revert the workaround and future deployments should succeed.

@sanathkr thank you so much!!
modifying my swagger (making the deploy assume it changed) was all that was needed.
this was not obvious, thank you again!

@sanathkr can confirm the second workaround worked painlessly for me as well o>

I'm seeing this issue, and using inline swagger.
It looks like adding a ' ' to a comment of the inline data works around the issue.
Annoying that this hasn't been fixed 6 days after reported here

I believe I am seeing a similar issue, I added a comment with more detail to this issue #191 but also adding here for reference.

In us-east-1 region this problem still exist

any update ?


What about docs?

what should we do?

[ ] #7
[ ] #8
[ ] #9
[x] #10

Hey there, I wonder what we should do for documentation

what should we do?

[ ]  Create a  docs folder.
[ ]  Create a new index.md folder.
[ ]  Turn on GitHub Pages
[ ] Decide if we want to go with something complex

Thanks ❤

/issue


Implement the new Profile Design

Just need to revamp the UI of the Profile Section.
Need to implement this design.

Here is a link to the Figma File.
Figma File
Note : This will be done after the Project Admin card and Project Card UI is done.

Hey @Rajdip019 can I work on this issue ?

Yes once the cards are finished we can start working on this.

Hey, @Rajdip019  can I also work on this issue?

As this is on first come first serve basis @arun-kushwaha04 will get to work on this issue first. But we will be adding more issues by today night.
Also if you find any bug or want to enhance something on the website you can always open an issue for that. @Yash-Parwani

@Rajdip019 thanks for assign this to me. I will start working on this on.

@Rajdip019 Let me know if I can work on this issue after them.

Hey, @arun-kushwaha04 do you want to work on this now?

Yes

Go ahead now you can start working on this. @arun-kushwaha04

@arun-kushwaha04 Any Updates?

@Debajyoti14, Yes I have started working on the new profile section. Please Exculse me i got busy with some family work.

@Rajdip019 where can I find google client id and secret id for local setup ?

@Rajdip019 @Debajyoti14 I didn't received api credentials for local setup.

@arun-kushwaha04 mail us once again at<EMAIL_ADDRESS>with the PR link and your GitHub ID

@Debajyoti14 sent.

Hey @Rajdip019 can I work on this issue ?

Hey brother as I dont have any work to do for right now can i do this work. thank you

@arun-kushwaha04 are you still working on the issue?


Custom StyleSheet injection target

Is it possible to insert the generated stylesheet into an element of a shadow DOM rather than the parent document? I'd like to use this library but that doesn't seem to be possible.

Currently there is no way to inject the styles into an element of a shadow DOM. Would you mind telling us your use case so we could discuss this functionality a bit more?

I'm writing a Chrome extension that inserts a React app into third-party sites (in my case, code repositories like maven, pypi etc.). I'd like to encapsulate the app in a shadow DOM to avoid style conflicts from the parent document.
It would be convenient if I could call something like :
styled.styleRoot(el)
I really like styled's API, so currently I'm going to include all: unset on each component as needed rather than using a shadow DOM, but obviously this isn't an ideal or foolproof solution

I think that's an edge case we will not support in the near future.
My idea would be to use an iframe to avoid style conflicts and then talk trough the postmessage api to the website.

I actually like this idea. We're pretty close to settling on a }>
        {this.props.children}
      </FrameComponent>
    );
  }
}

export default FramedComponent;

// FooBar.js
import React from 'react';
import styled from 'styled-components';
import FramedComponent from './FramedComponent';

const Root = styled.div`
  text-align: center;
  color: red;
`;

const Heading = styled.h1`
  font-size: 24px;
  color: blue;
`;

const FooBar = () => {
  return (
    <FramedComponent>
      <Root>
        <Heading>Example of</Heading>

styled-components with react-frame-component

      </Root>
    </FramedComponent>
  );
};

export default FooBar;

This is exactly right, if you don't have any dynamic variables that are set at run time, or for example if you use the theme provider. This would kinda throw a wrench into the works as the style sheet is changing you have no way to hang on to that persistence.

I use redux, so I just pass redux store through context  to FramedComponent and on each store update replace frames stylesheets with StyleSheet.instance.toHTML(). It's a very dirty hack and can cause problems with perfomance. Also that solution will not help if you use stateful components.

Updated my example with how I'm updating the styles through state. @Fer0x Is this how your doing it?

@owcc @stephenway @Fer0x
I just updated styled-components and have updated the FramedComponent to not rely on the StyleSheet class.
class FramedComponent extends React.Component {
  constructor(props) {
    super(props)
    this.state = {
      styles: "",
    }
    setTimeout(this.updateStyles, 1000)
  }

  componentDidMount() {
    this.updateStyles()
  }

  componentDidUpdate() {
    this.updateStyles()
  }

  updateStyles = () => {
    // Make a new div
    const div = document.createElement("div")

    // Find all style tags
    const styleTags = document.head.getElementsByTagName("style")

    // Iterate over all the tags
    for (let i = 0; i < styleTags.length; i++) {
      let tag = styleTags[i]

      // If the style tag is from styled-components then copy it into the div.
      if (tag.hasAttribute("data-styled-components")) {
        div.appendChild(tag.cloneNode(true))
      }
    }

    // Get the innerHTML of the div as a string
    let styles = div.innerHTML

    if (this.state.styles !== styles) {
      this.setState({ styles })
    }
  }

  render() {
    return (
      <Frame
        head={[
]}
        className="bfFrame"
        scrolling="no"
        style={{ zIndex: "1" }}
        frameBorder="0"
      >
        {this.props.children}
      </Frame>
    )
  }
}```

@ncphillips i don't think that code will catch dynamic changes to styled components as it's not going to get props updates in relation to style changes (nothing is passing new props to this) or you're only updating when parent state is changing (which could have side effects). If you don't want to go to the StyleSheet (which is bad I know). Then you'll have to create your own internal watch that keeps track of style updates. I've done something of this by memoizing the styles, and comparing them. I'm also keeping a timer for a debounce/throttle. This way it waits for the threshold of 100 ms where no changes have occurred to the style sheet to make any changes, and then update state. I hope this helps.
let lastStyle = null;
let lastTime = null;
class ContentFrame extends React.Component<any> {
  state = {
    styles: null
  };
  styledComponentWatch = () => {
    // get memoized  data
    const newStyle = this.getStyles();
    if (lastStyle === null) {
      console.log('starting');
      lastStyle = newStyle;
    } else if (lastStyle !== newStyle) {
      console.log('change found');
      lastStyle = newStyle;
      lastTime = new Date();
    } else if (lastStyle === newStyle && lastTime !== null) {
      console.log('nothing is changing');
      let duration = new Date().getTime() - lastTime.getTime();
      if (duration > 100) {
        lastTime = null;
        console.log('setting state');
        this.setState({ styles: newStyle });
      }
    }
    setTimeout(this.styledComponentWatch, 25);
  };
  componentDidMount() {
    this.styledComponentWatch();
  }
  getStyles = () => {
    const styleTags = document.head.getElementsByTagName('style');
    let styles = [];
    for (let i = 0; i < styleTags.length; i++) {
      const style = styleTags[i];
      if (style.hasAttribute('data-styled-components')) {
        styles.push(style.innerHTML);
      }
    }
    return styles.join('/n');
  };
  render() {
    const { children, ...rest } = this.props;
    return (
      <Frame {...rest} scrolling="no" frameBorder="0" head={}>
        {children}
      </Frame>
    );
  }
}

Here's a solution that worked for us:
/* @flow */

import * as React from 'react'
import { StyleSheetManager } from 'styled-components'

import Portal from '../Portal'

type Props = {| title: string, children: React.Node |}

type State = {| iframe: ?HTMLIFrameElement |}

export default class Iframe extends React.Component<Props, State> {
  static defaultProps = {
    title: 'Frame',
  }

  state = { iframe: null }

  ref = (iframe: ?HTMLIFrameElement) => {
    this.setState({ iframe })
  }

  render() {
    const { ref, state: { iframe }, props: { title, children } } = this

    return (
      <React.Fragment>

        {iframe && (

        )}
      </React.Fragment>
    )
  }
}

Hi. Just wondering if this is still being looked upon. I am having a problem with using styled components with React Shadow DOM for a chrome extension. Thank you.

Yes! That's what I used. I had a hard time implementing it with the shadow DOM though because all sample codes used iFrame. Thank you for pointing it out!

I used @migueloller 's solution. It works great in Chrome and Safari. It doesn't seem to work in Firefox. Any idea why?

Here's a complete example showing how to inject styles into shadow-dom using StyleSheetManager:
import * as React from "react"
import * as ReactDOM from "react-dom"
import {StyleSheetManager} from "styled-components"
import {App} from "components/App"

const wrapper = document.createElement("div")
const shadowRoot = wrapper.attachShadow({mode: "open"})

const styleContainer = document.createElement("div")
const appContainer = document.createElement("div")
const hostStyle = document.createElement("style")
hostStyle.textContent = `
:host {
  display: block;
}`

shadowRoot.appendChild(styleContainer)
shadowRoot.appendChild(appContainer)
shadowRoot.appendChild(hostStyle)

document.body.appendChild(wrapper)

const Root: React.SFC = () => {
  return (

  )
}

ReactDOM.render(
  <Root />,
  appContainer,
)

Given that there a bunch of working solutions in this thread and this is more of a usage question than a bug or feature request, going to close.

@probablyup Can you point me to a few working solutions in this thread? I've been trying out a few here but none has worked for me.

Heres my current workaround but may not fit everyone's needs. Just using this for a component playground and needed Demo to be in Iframe to manipulate viewport.
Using react-frame-component & html2react, simply passing document head to Iframe head.
import Frame from 'react-frame-component'
import HTML2React from 'html2react'

  <Frame  head={HTML2React(document.head.innerHTML)} >
<SomeStyledComponent />
 </Frame>

@jacek213 thanks a lot. I can't use React 16+ (I use 15.6) by some reasons, so you solution works for me. I just want to note that attachShadow has low browser support, so I've made like:
    const wrapper = document.createElement('div');
    const styleContainer = document.createElement('div');
    const offerContainer = document.createElement('div');

    wrapper.appendChild(styleContainer);
    wrapper.appendChild(offerContainer);

    const Root = () => (

    );

    ReactDOM.render(<Root />, offerContainer);

Every solution I found here is a dirty hack, I'm surprised how easily you close a bug because you don't like it.

Hey, I am building a chrome extension which is injected as content script. I am using a library which has used styled-components and facing the same issue. The CSS style tag is getting injected into document head. I want to insert it into iframe head. Please help.

https://styled-components.com/docs/api#stylesheetmanager
See “target”
On Tue, Jan 11, 2022 at 5:41 AM Mehul Lakhanpal @.***>
wrote:

Hey, I am building a chrome extension which is injected as content script.
I am using a library which has used styled-components and facing the same
issue. The CSS style tag is getting injected into document head. I want to
insert it into iframe head. Please help.
—
Reply to this email directly, view it on GitHub
https://github.com/styled-components/styled-components/issues/659#issuecomment-1009834204,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAELFVXQW33QW5TCMGIN24LUVQCM3ANCNFSM4DGXWFFA
.
You are receiving this because you were mentioned.Message ID:
@.***>

https://styled-components.com/docs/api#stylesheetmanager See “target”
…
On Tue, Jan 11, 2022 at 5:41 AM Mehul Lakhanpal @.> wrote: Hey, I am building a chrome extension which is injected as content script. I am using a library which has used styled-components and facing the same issue. The CSS style tag is getting injected into document head. I want to insert it into iframe head. Please help. — Reply to this email directly, view it on GitHub <#659 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAELFVXQW33QW5TCMGIN24LUVQCM3ANCNFSM4DGXWFFA . You are receiving this because you were mentioned.Message ID: @.>
But the external UI library (created by me) has styled-components. Current app has it as a dependency. So what to change in the build process?

Facing the same issue, external lib is injecting its styles to head instead of shadow root.


Bump phoenix_html to 4.1.1

The inputs_for and hidden_inputs_for functions have been moved to phoenix_html_helpers
https://hexdocs.pm/phoenix_html/changelog.html#v4-0-0-2023-12-19

The app doesn't start:

app-1       | Because "your app" depends on "phoenix_html empty" which doesn't match any versions, version solving failed.
app-1       | ** (Mix) Hex dependency resolution failed
app-1 exited with code 1

This is because we also have phoenix_html as a dependency in our demo project, not only in Backpex itself (see demo/mix.exs).
I deleted the dependency from the demo project. This will start the app, but raise an error if I try to access http://localhost:4000/admin/users/new.

app-1       | ** (Protocol.UndefinedError) protocol Enumerable not implemented for {:safe, []} of type Tuple. This protocol is implemented for the following type(s): CircularBuffer, DBConnection.PrepareStream, DBConnection.Stream, Date.Range, Ecto.Adapters.SQL.Stream, File.Stream, Floki.HTMLTree, Function, GenEvent.Stream, HashDict, HashSet, IO.Stream, Jason.OrderedObject, List, Map, MapSet, Phoenix.LiveView.LiveStream, Postgrex.Stream, Range, Stream
app-1       |     (elixir 1.15.7) lib/enum.ex:1: Enumerable.impl_for!/1
app-1       |     (elixir 1.15.7) lib/enum.ex:166: Enumerable.reduce/3
app-1       |     (elixir 1.15.7) lib/enum.ex:4387: Enum.filter/2
app-1       |     (backpex 0.3.2) lib/backpex/fields/has_many_through.ex:180: Backpex.Fields.HasManyThrough.render_form/1

The reason might be that we do not use the inputs_for function in Backpex.Fields.HasManyThrough#L507 as intended. The HasManyThrough field is still in beta state and there needs to be some refactoring done, before we can safely update to phoenix_html v4, I think.

Ah sorry I didn't see the demo project. I can try to take a look at the previous state and what happens on the update to see if there's any obvious fixes :)

Ah sorry I didn't see the demo project. I can try to take a look at the previous state and what happens on the update to see if there's any obvious fixes :)

That's awesome, thanks! 🙏

I have come a bit further. We need to change this line:
-    inputs_for(form, association.pivot.field)
+    form.impl.to_form(form.source, form, association.pivot.field, [])

Because before we were using this variant of inputs_for/3 of which is now deprecated. But the change above replicates the old behaviour.
The field is still broken though...

The addresses field doesn't seem to work on the latest develop either – selecting any values has no effect.
You can also see that happening here actually: https://backpex.live/admin/users/new

The addresses field doesn't seem to work on the latest develop either – selecting any values has no effect. You can also see that happening here actually: https://backpex.live/admin/users/new

Thanks for your report. I provided a fix for this in #387

The addresses field doesn't seem to work on the latest develop either

Yep, thats what I meant with:

The field is still broken though...

Thanks @Flo0807 for the fix! :-)
Then all we need here is the update in the demo and the line change I suggested.
@guess Are you going to finish this or should I take over?

Yay! Pushed and tested that it works in the demo project 🎉 @Flo0807 @pehbehbeh


Slots are not filling in forms

Rasa Open Source version
3.1
Rasa SDK version
No response
Python version
3.8
What operating system are you using?
Windows
What happened?
Slots are not filled in forms
Command / Request
rasa run -m models --enable-api --cors "*" --debug

Relevant log output
2022-06-07 22:28:42 DEBUG    rasa.core.processor  - Policy prediction ended with events '[]'.
2022-06-07 22:28:42 DEBUG    rasa.core.processor  - Action 'action_session_start' ended with events '[<rasa.shared.core.events.SessionStarted object at 0x00000272E9176040>, ActionExecuted(action: action_listen, policy: None, confidence: None)]'.
2022-06-07 22:28:42 DEBUG    rasa.core.processor  - Current slot values:
        work_type: None
        sector: None
        requested_slot: None
        session_started_metadata: None
2022-06-07 22:28:42 DEBUG    rasa.engine.runner.dask  - Running graph with inputs: {'__message__': [<rasa.core.channels.channel.UserMessage object at 0x00000272E9176E80>]}, targets: ['run_RegexMessageHandler'] and ExecutionContext(model_id='9934a898634246ca8a0ae7d46f9d5e34', should_add_diagnostic_data=False, is_finetuning=False, node_name=None).
2022-06-07 22:28:42 DEBUG    rasa.engine.graph  - Node 'nlu_message_converter' running 'NLUMessageConverter.convert_user_message'.
2022-06-07 22:28:42 DEBUG    rasa.engine.graph  - Node 'run_WhitespaceTokenizer0' running 'WhitespaceTokenizer.process'.
2022-06-07 22:28:42 DEBUG    rasa.engine.graph  - Node 'run_RegexFeaturizer1' running 'RegexFeaturizer.process'.
2022-06-07 22:28:42 DEBUG    rasa.engine.graph  - Node 'run_LexicalSyntacticFeaturizer2' running 'LexicalSyntacticFeaturizer.process'.
2022-06-07 22:28:42 DEBUG    rasa.engine.graph  - Node 'run_CountVectorsFeaturizer3' running 'CountVectorsFeaturizer.process'.
2022-06-07 22:28:42 DEBUG    rasa.engine.graph  - Node 'run_CountVectorsFeaturizer4' running 'CountVectorsFeaturizer.process'.
2022-06-07 22:28:42 DEBUG    rasa.engine.graph  - Node 'run_ResponseSelector5' running 'ResponseSelector.process'.
2022-06-07 22:28:42 DEBUG    rasa.nlu.classifiers.diet_classifier  - There is no trained model for 'ResponseSelector': The component is either not trained or didn't
receive enough training data.
2022-06-07 22:28:42 DEBUG    rasa.nlu.selectors.response_selector  - Adding following selector key to message property: default
2022-06-07 22:28:42 DEBUG    rasa.engine.graph  - Node 'run_DIETClassifier6' running 'DIETClassifier.process'.
2022-06-07 22:28:43 DEBUG    rasa.engine.graph  - Node 'run_EntitySynonymMapper7' running 'EntitySynonymMapper.process'.
2022-06-07 22:28:43 DEBUG    rasa.engine.graph  - Node 'run_FallbackClassifier8' running 'FallbackClassifier.process'.
2022-06-07 22:28:43 DEBUG    rasa.engine.graph  - Node 'domain_provider' running 'DomainProvider.provide_inference'.
2022-06-07 22:28:43 DEBUG    rasa.engine.graph  - Node 'run_RegexMessageHandler' running 'RegexMessageHandler.process'.
2022-06-07 22:28:43 DEBUG    rasa.core.processor  - Received user message 'stage' with intent '{'name': 'ask_work', 'confidence': 0.9999005794525146}' and entities '[{'entity': 'work_type', 'start': 0, 'end': 5, 'confidence_entity': 0.9148279428482056, 'value': 'stage', 'extractor': 'DIETClassifier'}]'
2022-06-07 22:28:43 DEBUG    rasa.core.processor  - Current slot values:
        work_type: None
        sector: None
        requested_slot: None
        session_started_metadata: None
2022-06-07 22:28:43 DEBUG    rasa.core.processor  - Logged UserUtterance - tracker now has 4 events.
2022-06-07 22:28:43 DEBUG    rasa.core.actions.action  - Validating extracted slots: work_type
2022-06-07 22:28:43 DEBUG    rasa.core.processor  - Default action 'action_extract_slots' was executed, resulting in 1 events: SlotSet(key: work_type, value: stage)
2022-06-07 22:28:43 DEBUG    rasa.engine.runner.dask  - Running graph with inputs: {'__tracker__': <rasa.shared.core.trackers.DialogueStateTracker object at 0x00000272E91766A0>}, targets: ['select_prediction'] and ExecutionContext(model_id='9934a898634246ca8a0ae7d46f9d5e34', should_add_diagnostic_data=False, is_finetuning=False, node_name=None).
2022-06-07 22:28:43 DEBUG    rasa.engine.graph  - Node 'rule_only_data_provider' running 'RuleOnlyDataProvider.provide'.
2022-06-07 22:28:43 DEBUG    rasa.engine.graph  - Node 'domain_provider' running 'DomainProvider.provide_inference'.
2022-06-07 22:28:43 DEBUG    rasa.engine.graph  - Node 'run_MemoizationPolicy0' running 'MemoizationPolicy.predict_action_probabilities'.
2022-06-07 22:28:43 DEBUG    rasa.core.policies.memoization  - Current tracker state:
[state 1] user intent: ask_work | user entities: ('work_type',) | previous action name: action_listen | slots: {'work_type': (1.0,)}
2022-06-07 22:28:43 DEBUG    rasa.core.policies.memoization  - There is no memorised next action
2022-06-07 22:28:43 DEBUG    rasa.engine.graph  - Node 'run_RulePolicy1' running 'RulePolicy.predict_action_probabilities'.
2022-06-07 22:28:43 DEBUG    rasa.core.policies.rule_policy  - Current tracker state:
[state 1] user text: stage | previous action name: action_listen | slots: {'work_type': (1.0,)}
2022-06-07 22:28:43 DEBUG    rasa.core.policies.rule_policy  - There is no applicable rule.
2022-06-07 22:28:43 DEBUG    rasa.core.policies.rule_policy  - Current tracker state:
[state 1] user intent: ask_work | user entities: ('work_type',) | previous action name: action_listen | slots: {'work_type': (1.0,)}
2022-06-07 22:28:43 DEBUG    rasa.core.policies.rule_policy  - There is no applicable rule.
2022-06-07 22:28:43 DEBUG    rasa.engine.graph  - Node 'select_prediction' running 'DefaultPolicyPredictionEnsemble.combine_predictions_from_kwargs'.
2022-06-07 22:28:43 DEBUG    rasa.core.policies.ensemble  - Made prediction using user intent.
2022-06-07 22:28:43 DEBUG    rasa.core.policies.ensemble  - Added `DefinePrevUserUtteredFeaturization(False)` event.
2022-06-07 22:28:43 DEBUG    rasa.core.policies.ensemble  - Predicted next action using RulePolicy.
2022-06-07 22:28:43 DEBUG    rasa.core.processor  - Predicted next action 'action_default_fallback' with confidence 0.30.
2022-06-07 22:28:43 DEBUG    rasa.core.processor  - Policy prediction ended with events '[<rasa.shared.core.events.DefinePrevUserUtteredFeaturization object at 0x00000272EDC22850>]'.
2022-06-07 22:28:43 DEBUG    rasa.core.processor  - Action 'action_default_fallback' ended with events '[<rasa.shared.core.events.UserUtteranceReverted object at 0x00000272EB57E850>]'.
2022-06-07 22:28:43 DEBUG    rasa.core.processor  - Current slot values:
        work_type: None
        sector: None
        requested_slot: None
        session_started_metadata: None
2022-06-07 22:28:43 DEBUG    rasa.core.processor  - Predicted next action 'action_listen' with confidence 1.00.
2022-06-07 22:28:43 DEBUG    rasa.core.processor  - Policy prediction ended with events '[]'.
2022-06-07 22:28:43 DEBUG    rasa.core.processor  - Action 'action_listen' ended with events '[]'.
2022-06-07 22:28:43 DEBUG    rasa.core.lock_store  - Deleted lock for conversation '2e474164-1450-4ba3-a205-ddcbf0e0de6d

@bilalltf In order for us to triage this issue, we need a few more details:

your config, domain and training data files
which endpoint are you using from the HTTP API (I see that in the rasa run command you've enabled that) and the body of your request?

Without looking at the training data files, I can't assert why your assistant predicts action_default_fallback whose side effect is that it causes an event (UserUtteranceReverted) which resets any previous slots set back to None (for example, from the logs, I can see slot work_type got set to value stage and then reset later to None by this UserUtteranceReverted event).
There's also no form action being predicted, please confirm if you've added rules for form activation and submission?


Add extended logging to SwiftIntefaceTests.testSystemModuleInterface

This enables lower-level logging for a single test case, which will hopefully help us figure out why that test is sometimes failing in CI.
rdar://104871745 tracks that we should remove this extended logging again once it has fulfilled its purpose.

@swift-ci Please test


Update browserify-transform-tools

We had problem because of browserify-transform-tools and falafel.

thanks!

published<EMAIL_ADDRESS>


Failed to load jar for dependency APACHE_COMMONS_POOL

After a restart this issue came up:  https://gist.github.com/KasperFranz/99d92014b7af2f61f47a84e6c41321d7
which made so luckPerms didn't start, GriefPrevention etc. didn't work because of this etc. etc.
It got fixed by a restart.
If it is any help i activated Redis on it yesterday (it had restarted without any problems)

Sadly it didn't fix the problem - still getting this error on load
https://gist.github.com/KasperFranz/700202c5d9904cc08a524fd0d89605d3
It works if I disable redis in the config

That fixed it :) Thanks man


Minor addon improvements, fix tests

This PR fixes the Windows tests and introduces some minor code-reuse in the modify* and map* addons.
@mplattner: There shouldn't be much controvery here, feel free to take a look and merge if you are happy! 😃

Looks good to me. :+1:
Just the code cov. test is failing. Can/should I still merge?

Codecov we could ignore, individual coverage we can't. Will fix asap! :)


Options don't seem to work

I've tried several different iterations with the options and non are working.
lane :distribute do
      updated_version = 75
      increment_version_code(
        gradle_file_path: "./app/build.gradle",
        ext_constant_name: "versionCode",
        version_code: updated_version.to_i,
      )
      gradle(task: "clean")
      gradle(task: "assemble", build_type: "Release")
      firebase_app_distribution()
end

I've tried:

moving increment_version_code after the gradle tasks
wrapping the options for increment_version_code in {}
removing the versionCode entry from the build.gradle file
using .to_s instead of .to_i for the version_code option

Am I missing something obvious, or is this plugin broken?

Hi, what do you mean by 'non are working'?
What is the actual result?
I am using increment_version_code(gradle_file_path: "app/build.gradle") to simply increment the current value by 1.


python311Packages.demetriek: 0.4.0 -> 1.0.0

Diff: https://github.com/frenck/python-demetriek/compare/refs/tags/v0.4.0...v1.0.0
Changelog: https://github.com/frenck/python-demetriek/releases/tag/v1.0.0

Things done

Built on platform(s)

[ ] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)

[ ] sandbox = relaxed
[ ] sandbox = true

[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[ ] Tested basic functionality of all binary files (usually in ./result/bin/)
25.05 Release Notes (or backporting 24.11 and 25.05 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module

[ ] Fits CONTRIBUTING.md.

Add a :+1: reaction to pull requests you find important.

nixpkgs-review result
Generated using nixpkgs-review.
Command: nixpkgs-review pr 364165

x86_64-linux

:x: 1 package failed to build:

home-assistant-component-tests.lametric

:white_check_mark: 4 packages built:

python311Packages.demetriek
python311Packages.demetriek.dist
python312Packages.demetriek
python312Packages.demetriek.dist


IServiceCollection.ConfigureApiBehaviorOptions overload for IServiceProvider

Is your feature request related to a problem? Please describe.
I am trying to use IServiceCollection.ConfigureApiBehaviorOptions to set an InvalidModelStateResponseFactory, but my factory has a number of dependencies so I want to resolve this via dependency injection.
Currently I am having to build a service provider just for this:
IInvalidModelStateResponseFactory invalidModelStateResponseFactory =
	services.BuildServiceProvider().GetService<IInvalidModelStateResponseFactory>();

services.ConfigureApiBehaviorOptions(options =>
		options.InvalidModelStateResponseFactory = invalidModelStateResponseFactory.Create);

Describe the solution you'd like
An overload of the IServiceCollection.ConfigureApiBehaviorOptions(...) method that can provide an IServiceProvider for you to get a service from, this could be used like:
services.ConfigureApiBehaviorOptions(options, provider =>
	options.InvalidModelStateResponseFactory =
		provider.GetService<IInvalidModelStateResponseFactory>.Create);

Additional context
Lack of this functionality also means that you must call ConfigureApiBehaviorOptions after you have registered any dependencies required. Ideally the order in which dependencies are registered should not matter.
Apologies if there is already a way of solving this problem, I couldn't see one.

You can already achieve this without building an IServiceProvider, using AddOptions and Configure:
services.AddOptions<ApiBehaviorOptions>()
    .Configure<IInvalidModelStateResponseFactory>((o, invalidModelStateResponseFactory) =>
    {
        o.InvalidModelStateResponseFactory = invalidModelStateResponseFactory.Create;
    });

See Use DI services to configure options in the docs for more.

Fantastic, thanks! I didn't know about that :)


Description for "FilterUsersAndGroupsOnlyFromPrimaryDomain" property in identity.xml is misleading

FilterUsersAndGroupsOnlyFromPrimaryDomain is a new property introduced in IS 5.8.0 which will enable filtering users and groups in the primary user store if the a domain is not included in the query param or in the attribute value.
But the description [1] for the above property does not mention about the impact by the domain value in a request. This needs to be fixed. Till the issue is fixed, this needs to be documented.
[1] - If you set the property value to true, regardless the Users endpoint and Groups endpoint, it will filter users or groups only from 'PRIMARY' user store. if the property value is false, it will filter users or groups across all user stores.  Default value : false

This issue is being closed due to extended inactivity. Please feel free to reopen it if further attention is needed. Thank you for helping us keep the issue list relevant and focused!


Google drive security?

This is in the drive code:
rcloneClientSecret = "8p/yms3OlNXE9OTDl/HLypf9gdiJ5cT3"
Is this not a security concern?

This is what prompted me originally: http://security.stackexchange.com/questions/107370/should-i-trust-grive2/107372
However maybe this is common?
http://stackoverflow.com/questions/4419915/how-to-keep-the-oauth-consumer-secret-safe-and-how-to-react-when-its-compromis
http://stackoverflow.com/questions/1934187/oauth-secrets-in-mobile-apps
...

We considered all the alternatives, and all of them are worse. There are secrets for other platforms as well.
There is not good way of doing oauth from an open source app, unless it is binary only releases.
At least this way, we don't rely on some proxy setup, that is just as bad, and furthermore puts user login information at potential risk.
The only risk is that someone will claim to Google (and the user) that they are "rclone". It doesn't allow access to any information unless the user explicitly grants access. They could "piggyback" on the "rclone.conf" file, but any way of storing login information locally exposes that risk.
You could open an issue to add optional encryption of rclone.conf- that is way more relevant.

Supposing a user allowed (on his machine) rclone to access its GOogle Drive. Does it mean that someone else using rclone on another machine has access? It sounds it's only half of the piece but that sounds a lot. Please confirm.
As for improvement, couldn't it be generated once per machine/user? Even having it into a public binary is just making it a bit harder but that's not changing really the security.
If that's not possible, I'd offer during the setup to either use the default key, or let users provide their own (with a link to Google Developer Console to help set one up).

Supposing a user allowed (on his machine) rclone to access its Google Drive. Does it mean that someone else using rclone on another machine has access?

Absolutely not.
The only thing you can do with the rclone client secret is pretend to be the rclone app which doesn't gain you anything really.

Even having it into a public binary is just making it a bit harder but that's not changing really the security.

Note that the secret is obfuscated so you can't get it from looking at the source without a bit of work.  Again not really making it a lot harder to derive it.
I should probably rename the constant to rcloneEncryptedClientSecret or something like that which will give people who are viewing the source code a bit more confidence.

I'd suggest to offer during the setup to either use the default key (with an notice it may reduce security), or let users provide their own (with a link to Google Developer Console to help set one up).

You can already provide your client details in the setup process.
Doing so doesn't change the security of anything though.

I just read the linked article: http://security.stackexchange.com/questions/107370/should-i-trust-grive2/107372
I think the accepted answer is wrong

If you allow this application to be trusted by your Google account, then any permissions you give the app will be inherently granted to anyone who decides to maliciously make use of the client secret together with the client ID.

You may give permissions to the app which is identified by the client secret, but you need to go through the oauth2 workflow to get a bearer token to actually use those permissions.  This requires interaction with the user and the user putting in their password.  The bearer token is the real security.

I think the accepted answer is wrong

It is. Otherwise there is no reason for the user to authenticate at all.
That said, we could rather easily allow users to encrypt rclone.conf. They would of course have to enter the key every time they start rclone, but that would make it impossible for programs to use the credentials in rclone.conf without the user knowing.
It looks like I could get some time next week, I can give it a shot.

Encrypting rclone.conf sounds like something that should already be taken care of on most Linux (most allow and offer to encrypt home directory by default). So I wouldn't dwell on that too much. I'd just make sure it has user's and user's only access (like .ssh folder only requires chmod go-rwx).

Not everybody are so fortunate to have such OS features :)
You could argue that is the users problem, if they run rogue software. But if I somehow got a copy of your rclone.conf, I would be able to access your files until you revoked rclone access on your cloud provider.

Hmm, sounds like a common problem then.
@ncw:
I think you might be right about the answer being wrong.
As I understand it, the role the sk plays that an app con pretend to be rClone, and get a token using the google auth UI. But any app could just as well pretend to be rclone, with their own appid and secret anyway. There would be no token granted without a new user access approval (that right?), and afaik if the token was already obtained somehow, it can be used without the sk (is that right?).
I think the more pertinent info is:

Furthermore, the client secret leak breaks a layer of protection designed to ensure the legitimacy of the communicating server.
Which seems to be the case, and also, something in a flaw for use cases involving oath and client-side code :-/

Incidentally, I don't think binaries or 'Encrypted' secret ids are that great.
I took the SE commentors advice, and setup my own app & key. When I inserted it into the rClone code, I got an error about it not bein a valid base64 code, or something - it wasn't hard to figure it out; namely find:

fs.Reveal(rcloneClientSecret),

func Reveal(y string) string {
x, err := base64.StdEncoding.DecodeString(y)
if err != nil {
log.Fatalf("Failed to reveal %q: %v", y, err)
}

in the code, and base64 encode it. Incidentally, I think it would give the wrong idea to call this 'encrypted', when it's really just obfuscated. If I can do it, anyone capable of exploiting an Oauth SK could easily do it too.
As for binaries - If the app eventually sends the SK, it has to decode it, at which point a debugger trace / memory dump can reveal it. A bit harder, but not a vast challenge.
I think ultimately, any client-side code holding a SK needs either a per-client key, and if the protocol doesn't handle this case, obfuscation is maybe the best that can be done?

You can already provide your client details in the setup process.

I hadn't realised this actually - v. useful.
Here is a suggestion that would impact user ease a bit, but maybe the obfuscated app/sk could be put on the github page, and any user wanting to use them have to enter them.
The advantage would be they would naturally wonder why they have to do this, and the gh page can explain the situation in full, hence informing them of the issues. This, rather than "just working" out of the box, but with default credentials.
Just an idea :-)

I agree on having more a warning and easy way to setup your own during initial setup/config. I wouldn't force a user to follow a complex process just to use the lesser safe ones. That would make it harder to test the app. That's why I was suggesting a prompt in the shell that allows to input a client key while accepting empty input to use the default one (with a warning).

@wernight:
I believe that's what happens now, but maybe without an explicit warning.

@Chris2048 wrote

There would be no token granted without a new user access approval (that right?)

Correct

and afaik if the token was already obtained somehow, it can be used without the sk (is that right?).

Also correct. Though the token will expire eventually and the sk will be needed at renewal.

I took the SE commentors advice, and setup my own app & key. When I inserted it into the rClone code

That was a rather long way round!  Next time just set client_id and client_secret in the config file, or during the setup process
Google Application Client Id - leave blank normally.
client_id>
Google Application Client Secret - leave blank normally.
client_secret>

I think ultimately, any client-side code holding a SK needs either a per-client key, and if the protocol doesn't handle this case, obfuscation is maybe the best that can be done?

That was my view.  The Obfuscation of the secret key is enough to stop it appearing in the source directly, but won't survive an attacker.  Neither will any of the other methods though (witness the long line of cracked DRM technologies).

Here is a suggestion that would impact user ease a bit, but maybe the obfuscated app/sk could be put on the github page, and any user wanting to use them have to enter them.

Not wildly keen on that idea on ease of use grounds!  But I see where you are coming from.
I'm yet to be convinced that there are any security implications for the users data by using rclone's credentials.
The implications are more on rclone's side - namely using up rclone's drive quota, doing bad things and getting rclone throttled / banned on drive etc.
@wernight wrote

That's why I was suggesting a prompt in the shell that allows to input a client key while accepting empty input to use the default one (with a warning).

That is pretty much what it does aleady (see above).
As for a warning - what would it say?

@ncw The current wording sounds like "you should leave it empty unless you've special needs to change it for whatever reason". What I'm suggesting is something like:
Google Application Client ID (can be empty to use default):
NOTE: Using the default key is slightly less secure than getting your own key (see http://link-to-help-page-like-wiki-of-this-project-or-such).

NOTE: Using the default key is slightly less secure than getting your own key

Could you please elaborate how it is "slightly less secure", because AFAIK that isn't true.

@klauspost https://security.stackexchange.com/questions/107370/should-i-trust-grive2/107372 may be alarmist more than necessary.

@wernight - They obviously don't know how oauth works. To access anything they need your access/refresh token, which is in your rclone configuration file. The client "secret" is used for nothing more than identifying the rclone client at Google. Using another client secret does not in any way provide additional security, it only means that Google sees your client as another piece of software.

I agree with you that I don't see a security issue.

@klauspost @wernight
True, but one thing is right in the SE page

the client secret leak breaks a layer of protection designed to ensure the legitimacy of the communicating server
There is maybe a risk there, but it would have to be combined with other breaches. Most people probably blindly click 'Allow' anyway... :-/ :-B

Closed as-as-big-a-deal-as-first-thought :-B

@ncw @klauspost  I think you guys are missing the security issue from having the Secret publicly known and the stackexchange thread is actually correct.
An attacker can setup a phishing page that will try to get a fresh token for rclone app.
If the user is using rclone and is currently logged into his google account (which is very common) the attacker will get a valid token. Since the user previously allowed rclone app to access his data he will not be required to approve it again and this can probably be done without any user awareness.
The only thing which might protect rclone users from seeing this method being used in the wild is the low user base and popularity of this project. but we can assume that the more people using rclone the more we can expect this issue to be exploited.
I'm not sure if there is any good way to prevent this values from being publicly available, but if there isn't giving more information about this issue in the initial config as suggested by @wernight  would be a bit better.

@yonjah

I created a folder at the root of a crypt remote called the_test. Inside this one I have another one called test_within. I want to move the_test (including its subdirectory) inside a folder called fixed.

I'm not sure how that would work.  rclone opens known URLs in the browser directly or uses https to known URLs so I'm not sure how you could insert a phishing site in the process.

I'm not sure how that would work. rclone opens known URLs in the browser directly or uses https to known URLs so I'm not sure how you could insert a phishing site in the process.
You do not need to do it as part of rclone regular auth process.You just need to make sure rclone user is visiting your phishing site (like by posting a link in rclone related discussion).

I'll try and do a few tests end of next week and see if this is something exploitable or if the secret is even needed.
I think with google it may not be an issue but I'm not sure if this will be consistent with all providers

I've done some testing and these doesn't look exploitable.
Google drive will require user approval whenever a new auth is requested.
You still need to use the secret even the documentation might indicate otherwise.
I also tested with Onedrive, there the situation is a bit different. After the user gives permission to the app any future auth requests won't need any user interactions, but this is still not exploitable since the redirect URL is fixed. rclone redirects back to local machine so if an attacker can somehow listen to a local socket this might still be exploitable but I don't think this is possible from just visiting a web page and if an attacker convinces the user to install anything on his computer than there is not much rclone can do any way.
There might be other providers who will not lock the redirect URL but than it is probably more of an issue with the service provider than with rclone

There might be other providers who will not lock the redirect URL but than it is probably more of an issue with the service provider than with rclone

I'm pretty sure having a fixed redirect URL is part of the oauth spec - certainly all the providers I've used with rclone you have to specify the redirect URL as part of the signup for a new app.


Changing network in navbar always redirects to Landing page

Thank you for your contribution to the KodaDot - One Stop Shop for Polkadot NFTs.
👇 __ Let's make a quick check before the contribution.
PR Type

[x] Bugfix
[x] Feature
[ ] Refactoring

Context
Go to

/prefix/create
/blog/xxx
/blog
/_prefix/massmint
/_prefix/massmint/onboarding
/sales
/hot
/series-insight
/chain/explore/collectibles
/chain/explore/items

and check what if the page is redirected to / or stays

[x] Closes #6416
[ ] Requires deployment <snek/rubick/worker>

Did your issue had any of the "$" label on it?

[x] Fill up your DOT address: Payout

Community participation

[x] Are you at KodaDot Discord?

Screenshot 📸

Copilot Summary

🤖 Generated by Copilot at f71db16
This pull request improves the user experience of switching chains on the navbar by adding logic to the ChainSelectDropdown component and moving the usePrefix composable to the component file.

🤖 Generated by Copilot at f71db16

Sing, O Muse, of the skillful coder who devised
A clever composable, usePrefix, and moved it with care
To the ChainSelectDropdown, where it fit well and thrived
And helped the user navigate the chains with flair.

@prury pls check , both of them issues should work

@prury pls check , both issues should be fixed

Blog:
https://github.com/kodadot/nft-gallery/assets/36627808/eff5e419-3dc4-425c-8950-1bd099e9b9c9
/series-insight/
https://github.com/kodadot/nft-gallery/assets/36627808/902131b8-07da-49a8-ac1a-8e835cdeca34

@prury pls check , both issues should be fixed

@prury pls check , both issues should be fixed

thank you so much!
i know i said i'll open a issue for both /hot and /sales, but i failed at explaining that they were not showing the correct information on that page(unrelated to this PR), but i still need you to redirect those pages when there is a network change if you could.

@prury pls check , both issues should be fixed

thank you so much!
i know i said i'll open a issue for both /hot and /sales, but i failed at explaining that they were not showing the correct information on that page(unrelated to this PR), but i still need you to redirect those pages when there is a network change if you could. We can make a followup issue for this if the PR is getting to big, lmk.
also, on mobile, when you change the network, the menu sidebar should close when choosing an option

what do you mean by by but i still need you to redirect those pages when there is a network change if you could , you mean refresh the content?

what do you mean by by but i still need you to redirect those pages when there is a network change if you could , you mean refresh the content?

yup
https://github.com/kodadot/nft-gallery/assets/36627808/f764646e-5c41-4814-abc3-025ae6085158

@prury pls check , both issues should be fixed

thank you so much!
i know i said i'll open a issue for both /hot and /sales, but i failed at explaining that they were not showing the correct information on that page(unrelated to this PR), but i still need you to redirect those pages when there is a network change if you could. We can make a followup issue for this if the PR is getting to big, lmk.
also, on mobile, when you change the network, the menu sidebar should close when choosing an option

I think they should be separate issues since the feature/bugfix it's already done and it's changing the chain globally those pages are not listening to chain changes , same issue as series-insights had, i can still fix it here if you want
what do you mean by by but i still need you to redirect those pages when there is a network change if you could , you mean refresh the content?

what do you mean by by but i still need you to redirect those pages when there is a network change if you could , you mean refresh the content?

yup
content.mp4

ok will fix it here

sorry for the trouble and thanks!

Checking everything again:
/prefix/create - ✔
/blog/xxx : ❌
/blog ✔
/_prefix/massmint ✔
/_prefix/massmint/onboarding ✔
/sales ❌ (snek,basilisk and both assethubs redirects to homepage)
/hot ✔
/series-insight ✔ (Ahp and Ahk prob left out intentionally, since prob not yet implemented)
/chain/explore/collectibles ✔
/chain/explore/items ✔

@hassnian we are almost there! i appreciate that you took this issue, its not an easy one and is a huge enhancement, keep it going!
will make a followup issue for:
profile page
transfer page

both assethubs

what do you mean by both assethubs

@Jarsen136 I think we should be good now , @prury made some small changes , changed the regex a bit , I tested it on my end and everything worked as expected , pls check just to make sure that everything works fine
thanks :D

pay 70 usd

😍 Perfect, I’ve sent the payout
💵 $70 @ 5.03 USD/DOT ~ 13.917 $DOT
🧗 16faLfsywwNATaEfbH2ah75dn6ZmctQWpMS5G4KFhbmj5hnD
🔗 0xc9b53ca189ef0255d594513a24a69af96ee07c53742df826325cfa590a9a51b5
🪅 Let’s grab another issue and get rewarded!
🪄 github.com/kodadot/nft-gallery/issues


Ingestion - Allowing Multiple Databases

Allowing multiple database support from a connector instead of providing one.

Overriding get_inspectors
For instance, snowflake connector doesn't work if we don't pass database to it.
we will need to override get_inspectors, run the query to show databases and loop through it.

Reopening this issue - to be taken up in 0.9.2 release.

This should be handled already by the addition of the db schema entity


⚠ Bofrost.de has degraded performance

In ae0e1ea, Bofrost.de (https://www.bofrost.de) experienced degraded performance:

HTTP code: 200
Response time: 1473 ms

Resolved: Bofrost.de performance has improved in 4f6cf62.


Not an Issue but Thanking

Sorry that I made a issue but I just wanted to thank you guys for your hard work.
This is so awesome.
After 4 hours getting my audio output right on the raspberry pi and after rebooting
it finaly worked.
Like a charm!
My friends can connect to it with their spotify software on their laptop and play music.
So awesome
Thanks for this. keep updating it. awesome package

Agree. It is awesome! I connected from my phone once, and can listen to music all day from the raspberrypi. Huge thanks and props to the team :+1:

Thanks @e-3-s and @dimitrisTim! :)


Be able to revert a wrongly appointed goal

I'm working on it ;-)

While you're at it @elgervb can you fix this also?

I think the easiest fix is to delete the player "AnnieAnnieAnnieAnnieAnnie"

Max plager name is already limited to 26 chars due to my poor css skills 😜

This functionality does not work as advertised, it seems that the only player you can revert a goal from is the top-right most player (in the above screenshot 'Annie...')
Also, when I revert a goal from Annie when she has no goals left it will be extracted from the blue team.

Correct. I have a fix ready. Shall check iT in somewhere thuis weekend

That's good hoor :bowtie:

It does not seem to work on mobile. I tested it on an iPhone and the long-press did not change the buttons to wiggly-waggle and invert.

Ps. please refrain from todo's in parts of the code that do not involve this functionality

@elgervb it would be great if you can you take another look at this, there are two topics to look into:

when reverting a goal from the red team, the blue defender also gets one goal reverted
it does not work on mobile


Hide the warning message from PS Module Import

Per request, I removed PS warning message from Console log.
Test Result
== Befor ==
10/23/2018 06:43:17 : [INFO ] File kvp_client32 successfully downloaded in Tools folder: kvp_client32.
10/23/2018 06:43:17 : [INFO ] kvp_client64 file not found in Tools folder.
10/23/2018 06:43:17 : [INFO ] Downloading required files from blob Storage Location
10/23/2018 06:43:17 : [INFO ] File kvp_client64 successfully downloaded in Tools folder: kvp_client64.
10/23/2018 06:43:17 : [INFO ] nc.exe file not found in Tools folder.
10/23/2018 06:43:17 : [INFO ] Downloading required files from blob Storage Location
10/23/2018 06:43:17 : [INFO ] File nc.exe successfully downloaded in Tools folder: nc.exe.
10/23/2018 06:43:17 : [INFO ] Test parameters have been validated successfully. Continue running the test.
10/23/2018 06:43:17 : [INFO ] Updating .\XML\GlobalConfigurations.xml
WARNING: The names of some imported commands from the module 'CommonFunctions' include unapproved verbs that might make them less discoverable. To find the commands with unapproved verbs, run the Import-Module command again
with the Verbose parameter. For a list of approved verbs, type Get-Verb.
WARNING: The names of some imported commands from the module 'Framework' include unapproved verbs that might make them less discoverable. To find the commands with unapproved verbs, run the Import-Module command again with
the Verbose parameter. For a list of approved verbs, type Get-Verb.
WARNING: The names of some imported commands from the module 'Test-Framework' include unapproved verbs that might make them less discoverable. To find the commands with unapproved verbs, run the Import-Module command again
with the Verbose parameter. For a list of approved verbs, type Get-Verb.
10/23/2018 06:43:18 : [INFO ] Updated GlobalConfigurations.xml
10/23/2018 06:43:18 : [INFO ] Auto selecting storage account : eosgsouthcentralus as per your test region.
WARNING: The names of some imported commands from the module 'CommonFunctions' include unapproved verbs that might make them less discoverable. To find the commands with unapproved verbs, run the Import-Module command again
with the Verbose parameter. For a list of approved verbs, type Get-Verb.
WARNING: The names of some imported commands from the module 'Framework' include unapproved verbs that might make them less discoverable. To find the commands with unapproved verbs, run the Import-Module command again with
the Verbose parameter. For a list of approved verbs, type Get-Verb.
WARNING: The names of some imported commands from the module 'Test-Framework' include unapproved verbs that might make them less discoverable. To find the commands with unapproved verbs, run the Import-Module command again
with the Verbose parameter. For a list of approved verbs, type Get-Verb.
10/23/2018 06:43:18 : [INFO ] Using provided secrets file: AzureSecrets.xml
10/23/2018 06:43:18 : [INFO ] AzureSecrets.xml found.
10/23/2018 06:43:18 : [INFO ] ---------------------------------
10/23/2018 06:43:18 : [INFO ] Authenticating Azure PS session..
== After ==
10/23/2018 06:57:00 : [INFO ] kvp_client32 file not found in Tools folder.
10/23/2018 06:57:00 : [INFO ] Downloading required files from blob Storage Location
10/23/2018 06:57:00 : [INFO ] File kvp_client32 successfully downloaded in Tools folder: kvp_client32.
10/23/2018 06:57:00 : [INFO ] kvp_client64 file not found in Tools folder.
10/23/2018 06:57:00 : [INFO ] Downloading required files from blob Storage Location
10/23/2018 06:57:00 : [INFO ] File kvp_client64 successfully downloaded in Tools folder: kvp_client64.
10/23/2018 06:57:00 : [INFO ] nc.exe file not found in Tools folder.
10/23/2018 06:57:00 : [INFO ] Downloading required files from blob Storage Location
10/23/2018 06:57:00 : [INFO ] File nc.exe successfully downloaded in Tools folder: nc.exe.
10/23/2018 06:57:00 : [INFO ] Test parameters have been validated successfully. Continue running the test.
10/23/2018 06:57:00 : [INFO ] Updating .\XML\GlobalConfigurations.xml
10/23/2018 06:57:01 : [INFO ] Updated GlobalConfigurations.xml
10/23/2018 06:57:01 : [INFO ] Auto selecting storage account : eosgwestus2 as per your test region.
10/23/2018 06:57:02 : [INFO ] Using provided secrets file: AzureSecrets.xml
10/23/2018 06:57:02 : [INFO ] AzureSecrets.xml found.
10/23/2018 06:57:02 : [INFO ] ---------------------------------
10/23/2018 06:57:02 : [INFO ] Authenticating Azure PS session..

10/23/2018 07:09:03 : [INFO ] ~~~~~~~~~~~~~~~TEST END : VERIFY-DEPLOYMENT-PROVISION~~~~~~~~~~
10/23/2018 07:09:03 : [INFO ] CURRENT - PASS    - 1
10/23/2018 07:09:03 : [INFO ] CURRENT - FAIL    - 0
10/23/2018 07:09:03 : [INFO ] CURRENT - ABORTED - 0
10/23/2018 07:09:03 : [INFO ] Checking background cleanup jobs..
10/23/2018 07:09:03 : [INFO ] Removed 0 context files.
10/23/2018 07:09:03 : [INFO ] Cycle Finished.. TC-58922
10/23/2018 07:09:03 : [INFO ]
Test Results Summary
ICA test run on 10/23/2018 06:57:17
ARM Image under test SUSE : SLES : 15 : 2018.07.16
Total Executed TestCases        1 (       1 Pass,        0 Fail,        0 Abort)
Total Execution Time(dd:hh:mm) 0:0:11
XML file: TestConfiguration
#Sr. Test Name : Test Result : Test Duration (in minutes)

VERIFY-DEPLOYMENT-PROVISION : PASS : 11.71
FirstBoot : PASS
FirstBoot : Call Trace Verification : PASS
Reboot : PASS
Reboot : Call Trace Verification : PASS

Logs can be found at .\TestResults\2018-22-10-23-56-57-0798
10/23/2018 07:09:03 : [INFO ] Creating 'Azure-58922-buildlogs.zip' from '.\TestResults\2018-22-10-23-56-57-0798'
10/23/2018 07:09:03 : [INFO ] C:\LISAv2\YHXO58922\Azure-58922-buildlogs.zip created successfully.
10/23/2018 07:09:03 : [INFO ] Copied : .\report\report_TC-58922.xml --> .\report\report_TC-58922-junit.xml
10/23/2018 07:09:04 : [INFO ] Analysing results..
10/23/2018 07:09:04 : [INFO ] PASS  : 1
10/23/2018 07:09:04 : [INFO ] FAIL  : 0
10/23/2018 07:09:04 : [INFO ] ABORT : 0
Copying all files back to original working directory: C:\Users\juhlee\Source\Repos\LISAv2.

Just a note:
I'm not getting these warnings by default.
In my PC this is set to silent by default.
$VerbosePreference = "silentlycontinue"
I get these warning only after I set $VerbosePreference = "continue"

@santoshx You are correct. But not all people set that parameter. I still pursue this change.


Unable to connect to Kafka with SASL_SSL and mechanism SCRAM-SHA-512 with MSK

Code
const { Kafka, logLevel } = require('kafkajs')

       const fs = require('fs')
        const  kafka = new Kafka({
            clientId: 'local-producer',
            brokers: ['XXXXXX1:9096', 'XXXXXX2:9096'],
            connectionTimeout: 10000,
            authenticationTimeout: 10000,
            requestTimeout: 10000,
            logLevel: logLevel.DEBUG,
            ssl: {
        rejectUnauthorized: false,
      },
              sasl: {
              mechanism: 'scram-sha-512',
              username: 'msk_user',
              password: 'msk_pass'
            },
          })

        const producer = kafka.producer()
        console.log('about to hit producer code');

        var sendMessage = async () => {
           try {

            // Producing
            console.log('about to hit producer code inside try');
            await producer.connect()
            console.log('connected producer code');
            await producer.send({
              topic: 'AWSKafkaPocTopic',
              messages: [
                { value: "hi hello EC2 js"},
              ],
            })
            await producer.disconnect()
        }
        catch(err)
        {
            console.error("could not write message " + err)
        }
          }
        sendMessage();
        console.log('about to hit producer code inside END');`

Receiving Error
{"level":"ERROR","timestamp":"2021-06-28T06:17:28.078Z","logger":"kafkajs","message":"[Connection] Connection timeout","broker":"XXXXX1:9096","clientId":"XXX-local-producer"}
{"level":"ERROR","timestamp":"2021-06-28T06:17:28.079Z","logger":"kafkajs","message":"[BrokerPool] Failed to connect to seed broker, trying another broker from the list: Connection timeout","retryCount":4,"retryTime":4910}
{"level":"ERROR","timestamp":"2021-06-28T06:17:43.005Z","logger":"kafkajs","message":"[Connection] Connection timeout","broker":"XXX:9096","clientId":"XXX-local-producer"}
{"level":"ERROR","timestamp":"2021-06-28T06:17:43.005Z","logger":"kafkajs","message":"[BrokerPool] Failed to connect to seed broker, trying another broker from the list: Connection timeout","retryCount":5,"retryTime":9790}
could not write message KafkaJSNumberOfRetriesExceeded: Connection timeout
{"level":"ERROR","timestamp":"2021-06-28T06:18:42.699Z","logger":"kafkajs","message":"[Connection] Connection error: connect ETIMEDOUT <IP_ADDRESS>:9096","broker":"XXXXXX:9096","clientId":"XXXX-local-producer","stack":"Error: connect ETIMEDOUT <IP_ADDRESS>:9096\n    at TCPConnectWrap.afterConnect [as oncomplete] (node:net:1133:16)"}
{"level":"ERROR","timestamp":"2021-06-28T06:18:52.937Z","logger":"kafkajs","message":"[Connection] Connection error: connect ETIMEDOUT <IP_ADDRESS>:9096","broker":"XXXXXX:9096","clientId":"XXXX-local-producer","stack":"Error: connect ETIMEDOUT <IP_ADDRESS>:9096\n    at TCPConnectWrap.afterConnect [as oncomplete] (node:net:1133:16)"}
{"level":"ERROR","timestamp":"2021-06-28T06:19:05.225Z","logger":"kafkajs","message":"[Connection] Connection error: connect ETIMEDOUT <IP_ADDRESS>:9096","broker":"XXXXXX:9096","clientId":"XXX-local-producer","stack":"Error: connect ETIMEDOUT <IP_ADDRESS>:9096\n    at TCPConnectWrap.afterConnect [as oncomplete] (node:net:1133:16)"}
{"level":"ERROR","timestamp":"2021-06-28T06:19:15.465Z","logger":"kafkajs","message":"[Connection] Connection error: connect ETIMEDOUT <IP_ADDRESS>:9096","broker":"XXXXX:9096","clientId":"XXXX-local-producer","stack":"Error: connect ETIMEDOUT <IP_ADDRESS>:9096\n    at TCPConnectWrap.afterConnect [as oncomplete] (node:net:1133:16)"}
{"level":"ERROR","timestamp":"2021-06-28T06:19:27.753Z","logger":"kafkajs","message":"[Connection] Connection error: connect ETIMEDOUT <IP_ADDRESS>:9096","broker":"bXXXXXX:9096","clientId":"XXX-local-producer","stack":"Error: connect ETIMEDOUT <IP_ADDRESS>:9096\n    at TCPConnectWrap.afterConnect [as oncomplete] (node:net:1133:16)"}

As the error says, the connection is timing out. Make sure you verify the connectivity between where you're running this and the MSK cluster, perhaps the AWS Reachability Analyzer can be used, or if not, then perhaps just netcat from one of your instances to see that it can connect.
Since this is not an issue in KafkaJS, I'm going to close this issue.

@Nevon  I tried java client and consumers and python too from the same EC2 instance and they work with no issues


chore(ci): increase timeout for slow e2e tests

I hereby agree to the terms of the RisingWave Labs, Inc. Contributor License Agreement.
What's changed and what's your intention?
Seems like it can still timeout https://buildkite.com/risingwavelabs/main-cron/builds/2549

Checklist

[ ] I have written necessary rustdoc comments
[ ] I have added necessary unit tests and integration tests
[ ] I have added test labels as necessary. See details.
[ ] I have added fuzzing tests or opened an issue to track them. (Optional, recommended for new SQL features #7934).
[ ] My PR contains breaking changes. (If it deprecates some features, please create a tracking issue to remove them in the future).
[ ] All checks passed in ./risedev check (or alias, ./risedev c)
[ ] My PR changes performance-critical code. (Please run macro/micro-benchmarks and show the results.)

[ ] My PR contains critical fixes that are necessary to be merged into the latest release. (Please check out the details)

Documentation

[ ] My PR needs documentation updates. (Please use the Release note section below to summarize the impact on users)

Release note
If this PR includes changes that directly affect users or other significant modifications relevant to the community, kindly draft a release note to provide a concise summary of these changes. Please prioritize highlighting the impact these changes will have on users.

Will wait for main-cron to pass before triggering merge.


automatic inversion of x axis with pandas dataframe read from a csv

Description
The x axis of histogram is automatically inverted when plotted with a pandas dataframe read from a csv file. It's fine when the data is not from a csv file as far as I observed.
Steps to reproduce
import numpy as np
import pandas as pd
import proplot as plot

rng = np.random.default_rng()
x = rng.standard_normal(500)
x = pd.DataFrame(x, columns=['col'])

f = 'test.csv'
x.to_csv(f)
x1 = pd.read_csv(f, index_col=0)
fig, ax = plot.subplots()
ax.hist(x1.col.values)
plot.show()

Expected behavior: [What you expected to happen]
Actual behavior: [What actually happened]

Equivalent steps in matplotlib
import matplotlib.pyplot as plt
fig, ax = plt.subplots()
ax.hist(x1.col.values)

Proplot version
0.6.4

Thanks for the report, this was a very simple fix: 1290536

This seems to happen when I specify the number of bins?
But doesn't without that...


Allow value filtering on SHOW TAG VALUES

This pull request allows users to filter on the value field in the SHOW TAG VALUES command:
SHOW TAG VALUES WITH KEY = "mytag" WHERE "value" = 'myvalue'

Previously this command would return all values.
Fixes https://github.com/influxdata/influxdb/issues/9555

@e-dard Added tests in e28073c.

During a run of megacheck the following issues were discovered:

@e-dard Is this ok to merge?

@benbjohnson go fmt issue?

During a run of megacheck the following issues were discovered:


increase frequency of ci-locks gha cron

🚀 Pull Request
Description

This pull-request increases the frequency of the ci-locks GHA cron to run every 2nd day.
This is in response to the recent rasterio behaviour regression (see #1131) which caused disruption of the pyvista pull-request workflow due to failing geovista integration tests.
I'd rather that geovista tests and updates its dependencies little and often. We can monitor and review the timeline frequency, but I'm keen to limit disruptions on pyvista as much as possible.
@banesullivan @tkoyama010 :+1:

A provocative idea! I can see the logic. Are you planning to merge every 2 days, or is the important thing to get a CI run every 2 days?

Having a clean test run seems good enough to me.
We could also consider auto merging only for lock updates with a bot if we weren't adverse to this?
Not 100% sure about that though, as I do scan through the changes to make sure I'm happy with them.
Anyways, at the moment I'd rather that we take on a minor maintenance burden, as opposed to pyvista  considering that geovista is unstable

Having a clean test run seems good enough to me.
We could also consider auto merging only for lock updates with a bot if we weren't adverse to this?
Not 100% sure about that though, as I do scan through the changes to make sure I'm happy with them.
Alternatively, we could schedule more frequent runs to simply resolve the environment to the latest dependencies, build that environment, then simply run the tests with no pull-request. It's purpose is simply to make sure nothing is broken, and revert back to weekly pull request updates. Seems more appetising to me ...
Anyways, at the moment I'd rather that we take on a minor maintenance burden, as opposed to pyvista  considering that geovista is unstable


Consider adding extension method to the RouteCollection to register a delegate callback for a route

It would make writing inline handlers easier.
routes.MapRoute("hello/{name}", async ctx =>
{
    await ctx.HttpContext.Response.WriteAsync("Hello " + ctx.Values["name"]);
});

Somebody could even extend this and write a routing microframework:
routes.Get("hello/{name}", async ctx =>
{
    await ctx.HttpContext.Response.WriteAsync("Hello " + ctx.Values["name"]);
});

@davidfowl would this make routing better outside of MVC?

@davidfowl does this issue still active? right now proposed feature already presented in rc2

Ah, yes, we actually did end up doing this in 1.0.0-rc2. See the APIs here: https://github.com/aspnet/Routing/blob/dev/src/Microsoft.AspNetCore.Routing/RequestDelegateRouteBuilderExtensions.cs

Nice!!


improve bigquery perf

Querying anything from BQ is significantly slower than other data sources in my anecdotal experience. This is my attempt to put together a demonstration.
Note, I did see #716 which I don't think is the cause here, since in both cases there is no LIMIT, but all rows are returned.
BQ Console
SELECT * FROM <project_identifier>.austin_bikeshare.bikeshare_trips`;

CLI (Hybrid)
> \timing
Timing is on
> select * from bq.austin_bikeshare.bikeshare_trips;
┌──────────┬─────────────────┬────────┬──────────────────────┬───┬────────────────────┬────────────────┬──────────────────┬──────────────────┐
│  trip_id │ subscriber_type │ bikeid │ start_time           │ … │ start_station_name │ end_station_id │ end_station_name │ duration_minutes │
│       ── │ ──              │ ──     │ ──                   │   │ ──                 │ ──             │ ──               │               ── │
│    Int64 │ Utf8            │ Utf8   │ Timestamp<ns, UTC>   │   │ Utf8               │ Utf8           │ Utf8             │            Int64 │
╞══════════╪═════════════════╪════════╪══════════════════════╪═══╪════════════════════╪════════════════╪══════════════════╪══════════════════╡
│ 26848024 │ Explorer        │ 1782   │ 2022-06-06T07:35:32Z │ … │ East 11th/Victory… │ 3791           │ Lake Austin/Enf… │               53 │
│ 26848731 │ Explorer        │ 19552  │ 2022-06-06T09:50:06Z │ … │ Red River/Cesar C… │ 4060           │ Red River/Cesar… │               58 │
│ 26849269 │ 3-Day Weekender │ 19645  │ 2022-06-06T11:06:54Z │ … │ South Congress/Ma… │ 4048           │ South Congress … │                7 │
│ 26852707 │ Local365        │ 19688  │ 2022-06-06T18:45:20Z │ … │ Red River/Cesar C… │ 7189           │ 28th/Rio         │               52 │
│ 26856577 │ Local365        │ 21859  │ 2022-06-07T12:33:00Z │ … │ 5th/Campbell       │ 4050           │ 5th/Campbell     │               23 │
│ 26858549 │ Pay-as-you-ride │ 19576  │ 2022-06-07T16:33:12Z │ … │ South Congress @ … │ 7187           │ South Congress/… │                7 │
│ 26861048 │ Local365        │ 19688  │ 2022-06-07T20:41:38Z │ … │ 16th/San Antonio   │ 7189           │ 28th/Rio         │               53 │
│ 26865937 │ Explorer        │ 21789  │ 2022-06-08T17:10:45Z │ … │ 6th/Lavaca         │ 3790           │ Lake Austin Blv… │               60 │
│ 26871124 │ 3-Day Weekender │ 21490  │ 2022-06-09T12:24:10Z │ … │ Lake Austin/Enfie… │ 3790           │ Lake Austin Blv… │               11 │
│ 26875306 │ Student Member… │ 19479  │ 2022-06-09T19:35:33Z │ … │ Rio Grande/12th    │ 7189           │ 28th/Rio         │               24 │
│        … │ …               │ …      │ …                    │ … │ …                  │ …              │ …                │                … │
│ 25614103 │ Student Member… │ 19734  │ 2021-11-08T18:17:42Z │ … │ 21st/University    │ 7125           │ 23rd/San Gabriel │                5 │
│ 25615374 │ Student Member… │ 1543   │ 2021-11-09T00:52:07Z │ … │ 21st/University    │ 7125           │ 23rd/San Gabriel │                5 │
│ 25696052 │ Student Member… │ 19432  │ 2021-11-21T18:04:43Z │ … │ 21st/University    │ 7125           │ 23rd/San Gabriel │                5 │
│ 25730112 │ Student Member… │ 21652  │ 2021-11-29T16:17:17Z │ … │ 21st/University    │ 7125           │ 23rd/San Gabriel │                4 │
│ 25731901 │ Student Member… │ 21472  │ 2021-11-29T21:08:38Z │ … │ 21st/University    │ 7125           │ 23rd/San Gabriel │                5 │
│ 25621445 │ Student Member… │ 19748  │ 2021-11-09T19:17:46Z │ … │ 21st/University    │ 7125           │ 23rd/San Gabriel │                6 │
│ 25658903 │ U.T. Student M… │ 18253  │ 2021-11-15T12:48:16Z │ … │ 21st/University    │ 7125           │ 23rd/San Gabriel │                4 │
│ 25563775 │ U.T. Student M… │ 16935  │ 2021-11-01T14:56:18Z │ … │ 21st/University    │ 7125           │ 23rd/San Gabriel │                4 │
│ 25696581 │ U.T. Student M… │ 19432  │ 2021-11-21T20:13:38Z │ … │ 21st/University    │ 7125           │ 23rd/San Gabriel │                5 │
│ 25696667 │ Student Member… │ 272G   │ 2021-11-21T20:40:44Z │ … │ 21st/University    │ 7125           │ 23rd/San Gabriel │                6 │
└──────────┴─────────────────┴────────┴──────────────────────┴───┴────────────────────┴────────────────┴──────────────────┴──────────────────┘
 1847746 rows (20 shown)
Time: 26.269s

I looked through the implementation and there's nothing here that seems like it should obviously cause things to be slow. Because this is a (sort of public) but still internally filed ticket, I'm going to follow up in a "here's how to debug cases like this," sort of way.

How much slower is this than querying the same data via the source file (CSV/Parquet Locally; CSV/Parquet in object store)? Having a baseline will help us understand how much of the performance loss is due to the indirection through BQ.
Datasources that contact external providers (like BQ and snowflake) are going to be slower (in different ways) than ones that read files locally. There's also going to be some slowness for queries that interact with the object store: that's sort of inevitable. We definitely have to have reasonable expectations and understand the relative performance of these different classes of datasources.
Data format matters, and things like CSV are always going to be slow, and experiment with different encodings (particularly parquet or whatever "native" format they have).
Do you observe the same timing effect when running the client inside of GCP?
The experience of performance is made up of three main components: time-to-first-result (initial-latency), processing-time (often a proxy for throughput), and total-time (from submit to the EOF of the response. also latency, but different) I suspect that BQ is slower than other data sources mostly for initial-latency, because of auth. It might also have some spin-up/cold start time.

Particularly for smaller queries you might see the initial latency piece more than you will for larger ones, because startup costs might be fixed and shorter queries will amortize that cost less effectively.
I would want to run some comparison queries and try to watch memory use during the query (if this is easy) you might see that the BQ datasource reads more data into local memory before returning results, say, which might be good to know, and explain some of this.

select * are an interesting case (and I don't know the specifics of the data flow) but depending on input format, we could end up bouncing back and forth between row-format and column formats for the data, and batching might end up slowing things down (where if we were aggregating, we'd get some benefit there.)

I'm trying to prepare the data in an easy way, but hit this: https://github.com/GlareDB/glaredb/issues/2512 :(


name/mac derivation not unique on some hosted and physical machines

This is no real issue - just a hint.
Online.net has cheap dedicated hosts (Dedibox) which seem to share all the same product UUID:
12345678-1234-5678-90AB-CDDEEFAABBCC.
This causes identical peer names, which results in a non-working weave installation.
To run weave on these hosts, the peer names has to be specified manually with e.g. --name ::1.
I suggest an entry in the FAQ section of the documentation regarding this problem. Maybe there are other hosters that do the same thing with their firmware...

Interesting. Is there is some other unique ID we could use here?

As it can be seen https://botbot.me/freenode/weavenetwork/2016-07-06/?msg=69184491&page=1 , the user got "cannot connect to ourself, retry: never". Perhaps, we could improve error reporting for this case.

I don't think there is any better unique (hardware-dependant) value. I can send you the full dmidecode output if you want.
As @brb suggested already, better error reporting would be sufficient. The confusing part for me was:
# weave status connections -> $PEERIP:6783   failed      cannot connect to ourself, retry: never
while
# weave status connections -> $PEERIP:6783 (Peer ca:dd:16:be:df:42)  failed      cannot connect to ourself, retry: never
or
# weave status connections -> $PEERIP:6783   failed      cannot connect to ourself (Me: ca:dd:16:be:df:42, Peer: ca:dd:16:be:df:42), retry: never
would be better in this case.

We could actually detect this situation very precisely by transmitting a per-run id in the handshake. Then when detect the same peer id but different run id we know we must have a peer id clash.

(and note that we already have such a run id - the peer uid, which is distinct from the name)

So there is really two sides to this issue:

improve the detection / error message
come up with a better way of deriving a UUID

Re 2... the questions whether there is anything sufficiently durable (i.e. survives reboots) and unique that we could derive a UUID from.

@opthomas-prime what specific Dedibox flavour are you running?

It's the "DEDIBOX® XC 2016 SSD" (https://www.online.net/en/dedicated-server/dedibox-xc).
These are custom-built, blade-style servers with their own custom  firmware:
Handle 0x0000, DMI type 0, 24 bytes
BIOS Information
	Vendor: Online Labs
	Version: <IP_ADDRESS>1
	Release Date: 03/29/2016
...
Handle 0x0001, DMI type 1, 27 bytes
System Information
	Manufacturer: Online Labs
	Product Name: SR
	Version: (^_^)
	Serial Number: 42
	UUID: 12345678-1234-5678-90AB-CDDEEFAABBCC
	Wake-up Type: Power Switch
	SKU Number: (^_^)
	Family: Avoton
...

If you need further information or want me to test something, feel free to ask.

On unique IDs: http://0pointer.de/blog/projects/ids.html

On unique IDs: http://0pointer.de/blog/projects/ids.html

Yes. We have gone through much deliberation and research here. Poke around github, the comments and the commit messages.

What about using the mac address of the first ethernet interface as prefix for the product UUID instead of a static string (https://github.com/weaveworks/weave/blob/master/weave#L531)?
Something like:
NIC=`find /sys/class/net/ -maxdepth 1 -type l -name "en*" -or -name "eth*" | sort | head -n 1`
if [ -e ${NIC} ]; then
  ADDR=`cat ${NIC}/address`
  UUID="$ADDR-$UUID"
fi

AFAIK interface order is not stable on modern linuxes. And MAC addresses may not be stable either.

Is the CPUID/EAX=3 (As Lennart P. describes in http://0pointer.de/blog/projects/ids.html) stable in virtualized environments?
Anyway, i don't want to interfer your disscussion. :)
But thx for caring about this problem!

We also have the same issue where the product_uuid is the same on multiple hosts.  For example, we have 12 x Dell C6145 servers (each with 2 mainboards), out of those 24 mainboards there are only 6 unique product_uuid.  This has caused us many headaches.
How about prefixing product_uuid with the contents of /var/lib/dbus/machine-id (or /etc/machine-id) if present?
Another possible solution would to add an option and allow the file containing a unique ID to be specified.

How about prefixing product_uuid with the contents of /var/lib/dbus/machine-id (or /etc/machine-id) if present?

Is /etc/machine-id a) distinct on all your machines, and b) constant across reboots?

To run weave on these hosts, the peer names has to be specified manually with e.g. --name ::1.

That this isn't the only issue. The UUID is used to generate the MAC for the weave bridge, which in turn is used as the peer name. If the UUIDs aren't unique then both the weave bridge MACs and peer names aren't unique. Setting the peer names with --name fixes the latter but not the former. Having multiple peers with the same weave bridge MAC will cause MAC collisions, which in turn prevents weave expose from working properly.

The UUID being used for the weave bridge MAC is exactly we we are having our issues.  In my example of 24 servers 18 of the 24 had a non-unique MAC on the weave bridge because of this.
As far as your question:
Is /etc/machine-id a) distinct on all your machines, and b) constant across reboots?

It usually is, but not 100%, I have seen cases where it was specifically set up to regenerate on every boot.  This is why I suggested a command line option so that a file containing a UUID could be specified when launching weave.  This would allow the user to control what file is being used and if needed create their own UUID and store in their own file that is persisted across reboots.

@opthomas-prime same question to you, i.e. is /etc/machine-id a) distinct on all your machines, and b) constant across reboots?

@rade both, a) and b) are true on the problematic online.net hosts!
This is not surprising, since this ID is random-generated during the OS installation (https://www.freedesktop.org/software/systemd/man/machine-id.html).
BUT: Only on systemd boxes!

@brb we should also fix https://github.com/weaveworks/go-checkpoint/blob/master/checkpoint.go#L342

Note we are getting more reports of duplicate IDs from users using Kubernetes.
https://github.com/weaveworks/weave-kube/issues/19 relates

we should also fix https://github.com/weaveworks/go-checkpoint/blob/master/checkpoint.go#L342

We should factor this out (along with its copy in the weave script) to a single place so it can't get out of sync.

I am hitting same issue in scaleway servers on weave-1.7.2 (deployed using kontena)

How about adding an option to provide uuid via env?

@jakolehm you can set the peer-name with --name; is it much easier for you to use an environment variable?

@bboreham peer-name does not affect mac? (we are using expose)

@jakolehm ok, I guess that was mentioned at https://github.com/weaveworks/weave/issues/2427#issuecomment-232256346
Have you tried it?  I am not entirely convinced that the mac clash will cause a problem in practice.

@rade points out that packets entering the Weave network with a source NAT is one example where it is likely to cause a problem.

What about using a combinaiton of sources?
Find N things permanent in the system (productID, machine-ID, FooEnvVar, etc), use X of N to generate the needed UUID (eg: 3 of 5) thrown error is there is not enough sources.

@jalberto we already have two sources; #2460 added two more but then we realised it needs to be backwards-compatible otherwise an upgrade will break an existing cluster.
Then the decision that must be taken is, keep it back-compatible but a blocker for new deploys in baremetal/datacenters, or broke in some point with older versions but be able to use weave in this areas.
a classic startegy will be to use a deprecaiton flag that be enable/disable for few versions.
whatever decision is taken it must be clear in the documents, so people don't spend time implementing something that will not work for some uses-cases and that will affect projects based on weave in a way or another.

What's wrong with using the bridge mac / persisted peer id if they exist, and generate new IDs according to the expanded formula?
It would mean that a machine which is restarted after an upgrade to the new version, and lost its persisted data, will get a new ID. That strikes me as a worthwhile sacrifice to make.

@jalberto Have you managed to solve this on Scaleway?
How?
/sys/class/dmi/id/product_uuid cannot be overwritten. Therefore, the only possible solution is to use a custom image of weave that uses /etc/machine-id. This is sub-optimal to say the least.
As for a backwards compatible solution, you could check for existence of a custom file (e.g. /var/weave-machine-uuid) and if it exists, use its content as a unique identifier instead of /sys/class/dmi/id/product_uuid.
Then all thats left to do is to document the existing issue with /sys/class/dmi/id/product_uuid in the troubleshooting section of the documentation and suggest a solution: populate /var/weave-machine-uuid with a random value.

I needed a temporary workaround to make this work on Scaleway.
Here is how to use it:
# On every node run
mkdir /etc/weave
uuidgen > /etc/weave/machine-uuid

And to setup weave-daemonset:
kubectl create -f https://gist.githubusercontent.com/gajus/ce8fd5ffae4583e4943c199245d3e317/raw/46b666e97b515b4f6126f433774764bb7a85bc04/weave-daemonset.yaml

Note:
Don't use it in production. The image is not going to be maintained. This is a quick workaround that enables setting up development environment on Scaleway (and other providers that do not ensure unique /sys/class/dmi/id/product_uuid UUID value).

What's wrong with using the bridge mac / persisted peer id if they exist

I believe the original objection to this was that a user creating cloned VM images after running Weave Net would get duplicated IDs.  But that's probably not a likely scenario.
It does need some revamping in the code, to introduce something which can read the persisted ID at the point it is needed.  Also to cope with the different locations of persisted data for weave and weave-kube.


DataInterpolations not precompiling

I recently ran a script mixing DataInterpolations and Zygote but I'm not able to precompile DataInterpolations anymore. Here is the error message
Warning: Package DataInterpolations does not have Zygote in its dependencies:
│ - If you have DataInterpolations checked out for development and have
│   added Zygote as a dependency but haven't updated your primary
│   environment's manifest file, try `Pkg.resolve()`.
│ - Otherwise you may need to report an issue with DataInterpolations
└ Loading Zygote into DataInterpolations from project dependency, future warnings for DataInterpolations are suppressed.

 Also, looks like that's using the released version. The new version isn't tagged yet, but will be in like 5 seconds

Try it now

(update)

resolved. thanks


Only subscribe on connect/reconnect, remove refresh interval

Previously, epic-wallet was unsubscribing, and re-subscribing every time a challenge was received from epicbox over WS
This was unnecessary, and subscription only needs to happen once per connection (provided connection remains alive)
This commit moves initial subscription from a challenge response, to central Subscribe function

These changes have been extensively tested by EpicBox Server chat testers, and plays nicely with https://github.com/fastepic/epicboxnodejs.

Will likely be adding to these changes to remove most of the refresh_interval code, as it's not needed in most places any longer... But, wanted to get these in so that epicbox initiatives can move forward.

Just added a commit in https://github.com/EpicCash/epic-wallet/pull/69/commits/81731712554cbd260f5b6b6317a2092939d45736 that removes the unnecessary refresh interval.
Something that is going on with challenges indicates we may be missing some needed signaling.  Something in the sleep, first_run, new_challenge code is fixing a real problem, but there is probably a better way to go about it.
When removing everything, epic-wallet hangs if listening to epicbox, and a send command is issued.  The result is that the generated tx never gets sent to epicbox.
Code submitted here does not have this issue. But I'll be working on finding a more clear and straightforward way to accomplish what this code is doing.