allow to pass a parameterized funciton as a string to page.evaluate

await page.evaluate(`console.log(arguments);`, 'FOO');

produces ReferenceError: arguments is not defined error and as far as I can tell, there is no way to pass a parameterised function as a string to page.evaluate.
I have tried await page.evaluate((a) => { console.log('a', a); }, 'FOO'); as well, with no luck.
Meanwhile, stringifying methods at runtime produces unexpected results when used in combination with traspilers such as Babel, e.g.
import fetch from 'fetch';

await page.evaluate(() => {
  fetch();
});

In this case, the code will be transpiled to () => {_xfetch2()}. xfetch2 does not exist in the browser scope.
One workaround is to prefix all clashing variables with window., though this approach is error prone.

How about simply exposing the fetch function?

@gajus the following works just fine for me:
const puppeteer = require('puppeteer');

(async() => {
  const browser = await puppeteer.launch();
  const page = await browser.newPage();
  page.on('console', msg => console.log(msg.text));
  await page.evaluate((a) => { console.log('a', a); }, 'FOO');
  await page.close();
  await browser.close();
})();

@aslushnikov, Didn't he want to expose Fetch from the fetch library? :/

@entrptaher not sure. It also can be that the issue is in interop with babel.
@gajus: you can use string interpolation to inject "arguments" to string literals passed as functions:
const arg = 'FOO';
await page.evaluate(`console.log(${arg});`);

@entrptaher not sure. It also can be that the issue is in interop with babel.

It says right there in the original issue:

Meanwhile, stringifying methods at runtime produces unexpected results when used in combination with traspilers such as Babel, e.g.

The intention of the code is to access fetch API.
However, because a transpiler sees import of fetch in the scope of the function, it tries to assign a local reference to the code.
If it is still not clear, here is an example of the transpilation.
https://babeljs.io/repl/#?babili=false&browsers=&build=&builtIns=false&code_lz=JYWwDg9gTgLgBAMwKYwMYAtFQiOByZNdPAbgCgywBDAcyQDokA3KgGwFcqYkAKHgSjgBeAHxwA3mTiIUGAeQC-_EkA&debug=false&circleciRepo=&evaluate=false&lineWrap=true&presets=es2015%2Creact%2Cstage-2&targets=&version=6.26.0

I keep the injected and main codes separate for such reason. I was having similar problem with scopes. But passing a promise/function with all scope related information seems like an overkill to me. The node scope and the browser scope both has different api for same things.
Example code to keep things tidy.
const puppeteer = require('puppeteer');
const pageFunctions = require("pageFunctions");

(async () => {
    const browser = await puppeteer.launch();
    const page = await browser.newPage();
    await page.goto('https://example.com');
    await page.evaluate(pageFunctions['fetchThingsForMe']);
    await browser.close();
})();

btw, what's wrong with naming things this way to simplify the problem?
import _fetch from 'fetch';
page.evaluate(() => {
  fetch();
});

or this one?
import puppeteer from 'puppeteer';
import _got from 'got';

puppeteer.launch({headless: false}).then(async(browser) => {
    const page = await browser.newPage();
    page.on('console', msg => console.log(msg));

    await page.goto("http://example.com");
    await page.exposeFunction('got', async(...args) =>{
        let data = await _got(...args);
        return data.body;
    });
    await page.evaluate(() => {
        got("http://echo.jsontest.com/key/value/one/two").then(response => console.log(response));
    });
});

I get the result,

The question is more related to how to ignore a function in babel.
If I remember correctly, some other library had this function where one could access both scope from each other. ie, casperJS.

You can just do:
import fetch from 'fetch';

page.evaluate(() => {
  window.fetch();
});

Thats not the point. This approach is error prone.
I would must rather be able to do this:
import fetch from 'fetch';

await page.evaluate(`() => {
  fetch();
}`);

and avoid all the uncertainty from the start.
Implicit behaviour is never good.

So you want this behaviour,
var links = [];
var casper = require('casper').create();

function getLinks() {
    var links = document.querySelectorAll('h3.r a');
    return Array.prototype.map.call(links, function(e) {
        return e.getAttribute('href');
    });
}

casper.start('http://google.fr/', function() {
   // Wait for the page to be loaded
   this.waitForSelector('form[action="/search"]');
});

casper.then(function() {
   // search for 'casperjs' from google form
   this.fill('form[action="/search"]', { q: 'casperjs' }, true);
});

casper.then(function() {
    // aggregate results for the 'casperjs' search
    links = this.evaluate(getLinks);
    // now search for 'phantomjs' by filling the form again
    this.fill('form[action="/search"]', { q: 'phantomjs' }, true);
});

casper.then(function() {
    // aggregate results for the 'phantomjs' search
    links = links.concat(this.evaluate(getLinks));
});

casper.run(function() {
    // echo results in some pretty fashion
    this.echo(links.length + ' links found:');
    this.echo(' - ' + links.join('\n - ')).exit();
});

Great. Let's make the solution then. :)

Wrong issue thread?

All I want is to be able to pass function as a string to the page.evaluate method.

This is really interesting, I wanted to do the same, but ended up splitting the files and code,
if you look at the example I showed of casperjs, you can see there is this line where you are basically executing a function on the browser scope. We can create a simple wrapper or use browserify to overcome this problem for now.

We could expose with browserify/webpack.
import puppeteer from 'puppeteer';
import cryptoRandomString from 'crypto-random-string';

(async() => {
  const browser = await puppeteer.launch({headless: false});
  const page = await browser.newPage();
  page.on('console', msg => console.log(msg));

  await page.goto("http://example.com");
  await page.exposeFunction('cryptoRandomString', cryptoRandomString);
  await page.evaluate(`
		cryptoRandomString(10)
		.then((data)=>{
			console.log(data)
		})
		`);
})();

I don't feel like helping anymore when I go such length and get dislikes. :) I'm out of here. :D

The https://github.com/GoogleChrome/puppeteer/issues/982#issuecomment-335706623 provides a reasonable solution for the issue.

This solution does not address the inherent design flaw in Puppeteer – converting a function to a string at runtime is not safe.
There is no guarantee that function myAwesomeFunction () { return arguments; } is not compiled into an incompatible expression at the compile time.

This is a very simple issue, try this in your babelrc, and the code I mentioned or any code that you want. But expose the function, that's how you can keep things seperate.
.babelrc
{
  "presets": ["es2015", "react"],
  "plugins": [
    "transform-es2015-function-name", ["transform-es2015-modules-commonjs-simple", {
      "noMangle": true
    }]
  ]
}

Don't rename, don't do anything, just use the proper babel plugin and expose whatever you want to expose to that page.

Peace.

If you have the solution feel free to share :)

I do. Allow to pass a parameterised function as a string, e.g.
page.evaluate(`(foo) => {
  console.log(foo);
}`, 'FOO');

You are evaluating string provided to page.evaluate anyway.
Before returning the result of evaluating page.evaluate, check if the result is a Function. If it is – call that function with the provided parameters and return the result.
It is that simple.

There is no point in asking someone to provide a solution if there is no follow up feedback.

It could be because people had a hard time understand the concept. Or they forgot to reply.
I forgot to reply :D

And I simply lack the knowledge to implement the solution you provided. It went above my head. :D

ping @aslushnikov

This solution does not address the inherent design flaw in Puppeteer – converting a function to a string at runtime is not safe.

@gajus if you don't like passing in functions, you can always pass in a string. There's no need to add parameters to the string since you can use template strings to embed them.

I have tried your suggestion.
const arg = 'FOO';
await page.evaluate(`console.log(${arg});`);

Gives:
Error: Evaluation failed: ReferenceError: FOO is not defined
    at <anonymous>:1:13
    at ExecutionContext.evaluateHandle (/Users/gajus/[..]/node_modules/puppeteer/lib/ExecutionContext.js:54:15)
    at <anonymous>
    at process._tickDomainCallback (internal/process/next_tick.js:228:7)

@gajus I forgot quotes around console.log argument.
The following works for me:
const pptr = require('puppeteer');

(async() => {
  const browser = await pptr.launch();
  const page = await browser.newPage();
  await page.on('console', msg => console.log(msg.text));
  const arg = 'FOO';
  await page.evaluate(`console.log('${arg}');`);
  await browser.close();
})();

This method is limited to strings. It is not passing anything. It is interpolating the value.

Anyway, if anyone else stumbles upon this, a reasonable workaround is to use new Function, e.g.
await page.evaluate(new Function('foo', 'console.log(foo);'), {foo: 'bar'});

I too am running into issues with Babel processing code I want to run in the browser.
Is it possible to pass a string to page.evaluate which can reference ElementHandles passed as args? e.g.
return page.evaluate((el, index) => {
      const reader = new FileReader();
      const promise = new Promise((resolve) => { reader.onload = resolve; });
      reader.readAsText(el.files[index]);
      return promise.then(() => reader.result);
    }, input, index);

Currently, I get the error Evaluation failed: ReferenceError: _promise2 is not defined, but I want to be able to reference the element handle

@maxbates, this easy tweak does not help?
https://github.com/GoogleChrome/puppeteer/issues/982#issuecomment-335756760

I am having the same issue as the OP.
I would like to "[use] the browsers fetch api in the NODE context".
I am writing a browser extension and want to test it in the context of the browser rather than mock chrome/other APIs. So for example, I have code that uses the MutationObserver and want to evaluate that function.
The code I have that does not work is:
const browser = await puppeteer.launch()
const page = await browser.newPage()
await page.setContent(html) // just an html string
await page.exposeFunction('myFunction', myFunction) // my function sets up a MutationObserver to watch for dom elements
await page.evaluate(async () => {
  const selector = '.example'
  const win = window as Window & WithReady
  win.myFunction(selector, () => {
    console.log('FOUND!')
  })
})

Am I thinking about this package all wrong? Is it meant more for integration tests with already published code?

@jdolle, maybe you can use .evaluateOnNewDocument for that purpose. And, Does the code run properly without puppeteer? I mean, can you manually execute the code first on your browser and see what happens?
For example, what is this line? What does it do if you execute normally on a page? Does it have any relation to myFunction?
const win = window as Window & WithReady?

@entrptaher Sorry, I should have specified -- my project is in TypeScript. myFunction is actually just called ready in my code, so WithReady is the type I made that contains ready specifically to add it to the Window type. As stated, I'm compiling everything with Babel (7 specifically, which has TS support), and that compiled code runs perfectly in my browser.
I'll give evaluateOnNewDocument a shot and report back. Thanks for the suggestion.

Sadly, I'm not able to get evaluateOnNewDocument to even run it seems. My test is as simple as possible:
it('fires when the element already exists', async (done) => {
  const page = await browser.newPage()
  await page.evaluateOnNewDocument(done)
  await page.goto('http://example.com/')
})

it('fires when the element already exists', async () => {
  const page = await browser.newPage();
  await page.goto('http://example.com/');
  const elementExists = !!(await page.$('selector'));
  expect(elementExists).toBe.true // or whatever condition it is.
})

That's very different -- hence I don't know if I'm thinking about this tool properly.
What I'm trying to do is call a function that has been imported by my test, within the context of the page. So say I have a function called selectElement that just calls document.querySelector for an element and returns that value so that I can check it in an assertion. Is that possible?

Whatever your function is, if you can return the value, you can use it and check it in assertion.
Just make sure the function you are running actually returns something when you run it.
const thisValueReturned = await page.evaluate(codeString);
expect(thisValueReturned).to.be.(string)

Hm. I think transforming the code causes issues here -- I am running it through babel and instrumenting w/nyc. Also, it seems impossible to determine coverage if you're eval-ing strings. Unfortunately I think I think mocking is my only option. Thank you for your help.

Thats not the point. This approach is error prone.

For the record in case someone else face this issue. I agree with @gajus's statement.
I have been facing an issue with @babel/plugin-transform-runtime. This plugin is hoisting some function leading to errors like:
Evaluation failed: ReferenceError: _from is not defined

not recommended for production but works best for dev
const puppeteer = require('puppeteer')
const fnsArr = [
    `hello = () => {
        return "Hello"
    }`,
    `returnX = (x) => {
        return x
    }`

]
const run = async() =>{
    const browser = await puppeteer.launch({headless : false});
    const page = await browser.newPage();
    await page.goto('https://www.google.com/');
    const ans = await page.evaluate((arr)=>{
        const fn1 = eval(arr[1]);
        let x = "You are Awesome"
        return fn1(x);
    },fnsArr)
    console.log(ans);
}
run();

The best practise for this is documented here:
https://github.com/GoogleChrome/puppeteer/blob/master/docs/troubleshooting.md#code-transpilation-issues

const arg = 'FOO';
await page.evaluate(`console.log(${arg});`);

Error: Evaluation failed: ReferenceError: FOO is not defined

const arg = 'FOO';
await page.evaluate(`console.log(${arg});`);

Error: Evaluation failed: ReferenceError: FOO is not defined

Try the following
const arg = 'FOO';
await page.evaluate((arg) => {
 console.log(arg);
}, arg);

I've used eval with success
        await page.evaluateOnNewDocument((mockProperty) => {
          mockProperty = eval(mockProperty)
          mockProperty('window.navigator.platform', 'Win32')
        }, mockProperty.toString())

Could you add this restriction to the documentation of page.evaluate()?
If given pageFunction is of type string, the args are NOT passed to the pageFunction?

Just giving this a wee bump...
Tried the most reasonable looking solutions:
https://github.com/puppeteer/puppeteer/issues/982#issuecomment-503452846
https://github.com/puppeteer/puppeteer/issues/982#issuecomment-338746321
...to no avail. Anything from the devs? I'm trying to get something along the lines of:
{
const evaluatedElement = await element.evaluate(x => x[DOMElementPropertyArg]);
}
...on the go, rather than having to have separate functions to get hrefs, outerHTML, etc.
No biggie if not, just unfortunate.

Oh, in case it helps any weary travellers, this is the solution I settled on. Did what I wanted it do to so eh, victory, I guess.
const evalFunctions = {
outerHTML: x => {
return x.outerHTML;
},
innerHTML: x => {
return x.innerHTML;
}
};
const getDOMProp = async (page, CSSSelector, evalProp) => {
const puppetArray = await page.$$(CSSSelector);
const promises = await puppetArray.map(async element => {
const evaluatedElement = await element.evaluate(evalFunctions[evalProp]);
return evaluatedElement;
});
const resolvedPromises = await Promise.all(promises);
return resolvedPromises;
};
const list = await getDOMProp(
page, // puppeteer page,
"#faq-left-nav > ul > li",
"outerHTML"
);

Oh, in case it helps any weary travellers, this is the solution I settled on. Did what I wanted it do to in that I can reuse my main function, even if I have to write a bunch of mini ones, so meh, victory, I guess. Woo.
const evalFunctions = {
outerHTML: x => {
return x.outerHTML;
},
innerHTML: x => {
return x.innerHTML;
}
};
const getDOMProp = async (page, CSSSelector, evalProp) => {
const puppetArray = await page.$$(CSSSelector);
const promises = await puppetArray.map(async element => {
const evaluatedElement = await element.evaluate(evalFunctions[evalProp]);
return evaluatedElement;
});
const resolvedPromises = await Promise.all(promises);
return resolvedPromises;
};
const list = await getDOMProp(
page, // puppeteer page,
"#left-nav > ul > li",
"outerHTML"
);

How could I potentially use moment.js in Puppeteer's evaluate? exposing the function is not working for me.

@dpw1

How could I potentially use moment.js in Puppeteer's evaluate? exposing the function is not working for me.

await page.evaluate((momentJsfn) => {
 console.log(momentJsfn);
}, momentJsfn);

Not sure if this helps or not - but this is what I did.
The best practice gave me this idea:
FROM:
await page.evaluate(`(async() => {
   console.log('1');
})()`);

TO:

const createBrowserCode = (x, y) => `
{
    const x = ${x};
    const y = ${y};
    return x * y;
}`;

const x = 10, y = 20;
await page.evaluate(`(async() => {
  ${createBrowserCode(x, y)};
})()`);

Allowing for args.
Possibly createBrowserCode() could read from a JavaScript file ...


Upgrade to k8s 1.6

Currently, we test etcd operator with k8s 1.5 . k8s 1.6 brings benefits: GC problematic logs reduced, kubectl working with TPR. We should upgrade our jenkins infra to 1.6 and fix problems for it.

We also need to change README: minimum version, replace curl hack with kubectl apply.

we need to move this to 0.2.5

done


Run a handler for all messages, then another only for commands

For a Telegram bot that manages chats, I need to be able to:

handle any "common" messages to the chat to perform some moderation;
if the message also happens to be a command, then run the corresponding command handler.

Not running chat_handler for command invocations is not an option because it enables malicious actors to hide bad messages under command prefixes.
Is it currently possible to achieve this with dptree? I thought of using filter_async, which would have an added benefit of not running command handlers at all if the message was deleted, but it appears that I would need something like try_filter_async (https://github.com/teloxide/dptree/issues/26) for proper error handling (Telegram API, database). Also, it this existed, would Teloxide be able to pass errors from the fallible filter to  Dispatcher's error_handler?

For context, I previously used python-telegram-bot's handler groups for this.


how to sendmail with non-anonymous user

hi, we want to notify message with alertmanager by send email.
however,the smtp server response 'Client was not authenticated', the detail tcp produce as below:
220 GD2-CAS01.corp.aaa.com Microsoft ESMTP MAIL Service ready at Fri, 20 May 2016 15:47:13 +0800
EHLO localhost
250-GD2-CAS01.corp.aaa.com Hello [10.xx.xx.xx]
250-SIZE 36700160
250-PIPELINING
250-DSN
250-ENHANCEDSTATUSCODES
250-STARTTLS
250-AUTH LOGIN
250-8BITMIME
250-BINARYMIME
250 CHUNKING
MAIL<EMAIL_ADDRESS>BODY=8BITMIME
530 5.7.1 Client was not authenticated
QUIT

I think that the go connect smtp without carry the SMTP_AUTH_USERNAME and SMTP_AUTH_PASSWORD so take me as anonymous user
Here is the part of alertmanager.yml configure:
global:
  # The smarthost and SMTP sender used for mail notifications.
  smtp_smarthost: '192.xxx.xxx.xxx:25'
  smtp_from<EMAIL_ADDRESS>  smtp_auth_username: '****'
  smtp_auth_identity: '****'
  smtp_auth_password: '****'

and I have also set 2 environment variables:
SMTP_AUTH_USERNAME=xxx
SMTP_AUTH_PASSWORD=xxx
please help me ,thanks

http://www.robustperception.io/sending-email-with-the-alertmanager-via-gmail/ should put you down the right path.

@brian-brazil Thanks you very much.
I have change the another alertmanager edition by your suggestion, and so that it seems to pass auth success.
However, now caught another exception with:
WARN[1200] Notify attempt 2 failed: starttls failed: x509: cannot validate certificate for 192.168.xx.xx because it doesn't contain any IP SANs  source=notify.go:193
Should I close the cls configure?  How can I deal with it, it seems I can't change the alertmanager code if released.
Thanks you very muchc again!

Hi jaminlai, this looks to me that you created the certificate by yourself, or you have a certificate for a fqdn but access it via an IP address. Maybe also have a look at this thread, seems to me pretty similar to your issue: https://github.com/elastic/logstash-forwarder/issues/221. Hope that helps you further.


Make error channel a buffered channel to avoid goroutine getting stuck

If the context Done channel is closed, connectionErrorChannel won't be read and goroutine will be stuck.

When Context reaches timeout the whole application is going down


feat(clerk-js,nextjs,clerk-react,types): Introduce <Waitlist /> component

Description
Show Join waitlist promt from <SignIn /> component when mode is waitlist
Change the text in RestrictedAccess component when mode is waitlist
Introduce <Waitlist /> component to allow users to join in the waitlist via email
Introduce joinWaitlist method in clerk class
Introduce redirectToWaitlist function in clerk class to allow user to redirect to waitlist page

Checklist

[ ] npm test runs as expected.
[ ] npm run build runs as expected.
[ ] (If applicable) JSDoc comments have been added or updated for any package exports
[ ] (If applicable) Documentation has been updated

Type of change

[ ] 🐛 Bug fix
[x] 🌟 New feature
[ ] 🔨 Breaking change
[ ] 📖 Refactoring / dependency upgrade / documentation
[ ] other:

<SignIn /> Component with sign_up.mode = waitlist

<SignUp /> Component with sign_up.mode = waitlist

<Waitlist /> Component

<Waitlist /> Component success screen

<Waitlist /> Component success screen with redirectUrl presented

!snapshot

Hey @nikospapcom - the snapshot version command generated the following package versions:

Package
Version

@clerk/astro
1.4.3-snapshot.ve498412ac8617eb535cbbeb78d27ded11ad8f45c

@clerk/backend
1.15.3-snapshot.ve498412ac8617eb535cbbeb78d27ded11ad8f45c

@clerk/chrome-extension
1.3.25-snapshot.ve498412ac8617eb535cbbeb78d27ded11ad8f45c

@clerk/clerk-js
5.31.0-snapshot.ve498412ac8617eb535cbbeb78d27ded11ad8f45c

@clerk/elements
0.18.1-snapshot.ve498412ac8617eb535cbbeb78d27ded11ad8f45c

@clerk/clerk-expo
2.2.31-snapshot.ve498412ac8617eb535cbbeb78d27ded11ad8f45c

@clerk/expo-passkeys
1.0.0-snapshot.ve498412ac8617eb535cbbeb78d27ded11ad8f45c

@clerk/express
1.3.5-snapshot.ve498412ac8617eb535cbbeb78d27ded11ad8f45c

@clerk/fastify
2.0.7-snapshot.ve498412ac8617eb535cbbeb78d27ded11ad8f45c

@clerk/localizations
3.5.0-snapshot.ve498412ac8617eb535cbbeb78d27ded11ad8f45c

@clerk/nextjs
6.2.0-snapshot.ve498412ac8617eb535cbbeb78d27ded11ad8f45c

@clerk/clerk-react
5.15.0-snapshot.ve498412ac8617eb535cbbeb78d27ded11ad8f45c

@clerk/remix
4.2.43-snapshot.ve498412ac8617eb535cbbeb78d27ded11ad8f45c

@clerk/clerk-sdk-node
5.0.56-snapshot.ve498412ac8617eb535cbbeb78d27ded11ad8f45c

@clerk/shared
2.11.1-snapshot.ve498412ac8617eb535cbbeb78d27ded11ad8f45c

@clerk/tanstack-start
0.4.19-snapshot.ve498412ac8617eb535cbbeb78d27ded11ad8f45c

@clerk/testing
1.3.17-snapshot.ve498412ac8617eb535cbbeb78d27ded11ad8f45c

@clerk/themes
2.1.41-snapshot.ve498412ac8617eb535cbbeb78d27ded11ad8f45c

@clerk/types
4.30.0-snapshot.ve498412ac8617eb535cbbeb78d27ded11ad8f45c

@clerk/ui
0.1.12-snapshot.ve498412ac8617eb535cbbeb78d27ded11ad8f45c

Tip: Use the snippet copy button below to quickly install the required packages.
@clerk/astro
npm i<EMAIL_ADDRESS>--save-exact

@clerk/backend
npm i<EMAIL_ADDRESS>--save-exact

@clerk/chrome-extension
npm i<EMAIL_ADDRESS>--save-exact

@clerk/clerk-js
npm i<EMAIL_ADDRESS>--save-exact

@clerk/elements
npm i<EMAIL_ADDRESS>--save-exact

@clerk/clerk-expo
npm i<EMAIL_ADDRESS>--save-exact

@clerk/expo-passkeys
npm i<EMAIL_ADDRESS>--save-exact

@clerk/express
npm i<EMAIL_ADDRESS>--save-exact

@clerk/fastify
npm i<EMAIL_ADDRESS>--save-exact

@clerk/localizations
npm i<EMAIL_ADDRESS>--save-exact

@clerk/nextjs
npm i<EMAIL_ADDRESS>--save-exact

@clerk/clerk-react
npm i<EMAIL_ADDRESS>--save-exact

@clerk/remix
npm i<EMAIL_ADDRESS>--save-exact

@clerk/clerk-sdk-node
npm i<EMAIL_ADDRESS>--save-exact

@clerk/shared
npm i<EMAIL_ADDRESS>--save-exact

@clerk/tanstack-start
npm i<EMAIL_ADDRESS>--save-exact

@clerk/testing
npm i<EMAIL_ADDRESS>--save-exact

@clerk/themes
npm i<EMAIL_ADDRESS>--save-exact

@clerk/types
npm i<EMAIL_ADDRESS>--save-exact

@clerk/ui
npm i<EMAIL_ADDRESS>--save-exact

One thing I just noticed about Astro: there's now a waitlistUrl in the Astro integration config and the SignUp and SignIn components. However, you can't really use this new feature when you've disabled the account portal because the Waitlist component is missing. I guess Astro support will be shipped later?

One thing I just noticed about Astro: there's now a waitlistUrl in the Astro integration config and the SignUp and SignIn components. However, you can't really use this new feature when you've disabled the account portal because the Waitlist component is missing. I guess Astro support will be shipped later?

Hey @mlafeldt
We're working on it and I'll have an update this week

One thing I just noticed about Astro: there's now a waitlistUrl in the Astro integration config and the SignUp and SignIn components. However, you can't really use this new feature when you've disabled the account portal because the Waitlist component is missing. I guess Astro support will be shipped later?

Hey @mlafeldt
We released the support of <Waitlist /> component for astro. You should install version<EMAIL_ADDRESS>:)
Feel free to reach out us if you running any issue
Thanks

@nikospapcom Added a waitlist to my Astro app without a hitch. 💥 Left some feedback here.


next-gen CI for bloom, migrating to poetry

This takes a next-gen approach to CI builds for python clients, using Circle. I don't have permissions to modify the repository - so let's discuss the approach, before more events are triggered.
The stock requirements.txt is used to install the dependencies for the 'build', tox and poetry. For the purpose of the branch - those builds have been configured to remove all logic from CircleCI, meaning anything can call these in the future. Circle is only responsible for installing them, and calling tox to do its work.
As below - we now build (in parallel) across all of the supported python versions. To add python versions to CI, all we have to do is edit the .circleci config and add a new python docker.
If we like this approach my proposal is to complete it by:

[x] Updating the README
[x] Add support for auto build, and publish to PyPI on tag, removing that from GitHub
[x] Changing the nightly to use the same code
[ ] (Optionally) Replace tox with pytest integrate since it can run codecov
[x] Integrating the codecov upload (currently broken because I don't have permissions)
[ ] Changing GitHub requirements to pass on the tests, rather than a specific build
[x] Lifting the early return bits I've introduced to other modules here.
[x] Applying to our other python projects.

@filipecosta90 or @guykorlandredis Do you have the ability to change the GitHub repo to depend on ci/circleci:build-latest instead of ci/circleci:build?


[Tests] Move tests to run on iPhone 6 and 64-bit architecture

The tests currently run against the old 32-bit armv7 devices and 4" screens (iPhone 5 specifically) -- any reason not to target the most recent devices? especially with the new iPhone coming next week?

@sahrens is there a reason for the tests to target old devices, or does this task make sense? I can help with this in a couple of weeks if someone else from the community or fb doesn't get around to it.

Really we should run on both, but that will require some rejiggering and recording new reference images for all the snapshot tests.  It also makes it a little more burdensome to change tests that require updating the snapshots, but probably worth it.
While we're only running on one device/config, I think it's better to be on older devices because they get much less organic usage (especially from developers) and bugs are more likely to slip past local testing.

Hi there! This issue is being closed because it has been inactive for a while.
But don't worry, it will live on with ProductPains! Check out its new home: https://productpains.com/post/react-native/tests-move-tests-to-run-on-iphone-6-and-64-bit-architecture
ProductPains helps the community prioritize the most important issues thanks to its voting feature.
It is easy to use - just login with GitHub.
Also, if this issue is a bug, please consider sending a PR with a fix. We rely on the community to provide
bugfixes as the core team is heavily focused working on performance.


Problem loading on Mac M3

I'm trying to run the ap on a Mac M3 system, and it's failing to load rgcam. Note I'm using R4.4. Here is what I'm getting when I try to install rgcam.
Downloading GitHub repo JGCRI/rgcam@HEAD
── R CMD build ──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
✔  checking for file ‘/private/var/folders/sp/_bjsbzwd6nq6zn9jmj01c5900000gn/T/RtmpKAk19a/remotes15d8716af875e/JGCRI-rgcam-ea47f08/DESCRIPTION’ (467ms)
─  preparing ‘rgcam’:
✔  checking DESCRIPTION meta-information
─  checking for LF line-endings in source and make files and shell scripts
─  checking for empty or unneeded directories
   Removed empty directory ‘rgcam/inst/ModelInterface/modelinterface’
   Omitted ‘LazyData’ from DESCRIPTION
─  building ‘rgcam_1.2.0.tar.gz’
   Warning: invalid gid value replaced by that for user 'nobody'

# Bootstrapping renv ${VERSION} ----------------------------------------------
Error in bootstrap(version, libpath) : failed to download renv ${VERSION}
Calls: source ... eval.parent -> eval -> eval -> eval -> eval -> bootstrap
Execution halted
Warning message:
In i.p(...) :
  installation of package ‘/var/folders/sp/_bjsbzwd6nq6zn9jmj01c5900000gn/T//RtmpKAk19a/file15d875de037f4/rgcam_1.2.0.tar.gz’ had non-zero exit status```

The issue seems to be with the renv package. Users had success deleting the renv folder in the project directory and then re-initializing the renv environment. Rgcam could then be installed using the standard process (see readme for details).


calculate external urls with username earlier

username was empty on external partner links generated from secondary accounts because it got filtered out before the links were calculated. i moved the calculation to before the filtering.
there's a warning on the page that clicking on one of these links will share your username with the third party, so i think this is fine.

@keybase/react-hackers @keybase/picnicsquad


Thumbnail with icc-profile

From https://github.com/kyamagu/psd-tools/issues/99

This is likely due to the difference between Pillow's handling of ICC profiles for PSD files. Current psd-tools approach is to convert to sRGB space for RGB documents, and do nothing for other colors.

Thank you for answer.
Is there any chance to update as_PIL() function to be able to create image thumbnails from psd-tools as from Pillow? Right now I see that this conversion happens in function _apply_icc(), what are the negative sides to exclude this conversion to sRGB?

@andrey-hider Sure, it is reasonable to control the behavior perhaps with argument options. Will consider in the next release.
The current behavior is to reflect the display color of the document in Photoshop UI. Without sRGB conversion, color looks different from Photoshop UI.

@andrey-hider ver 1.8.12 supports the following:
icc_profile = psd.image_resources.get_data('ICC_PROFILE')
image = psd.compose(apply_icc=False)
image.save('output.png', icc_profile=icc_profile)

@kyamagu Thank you for update.
I'm trying to execute new version and got next error:
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/chalice/app.py", line 670, in _get_view_function_response
    response = view_function(**function_args)
  File "<path>/app.py", line 156, in test_psd_icc_profile
    from psd_tools.api.psd_image import PSDImage
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/psd_tools/__init__.py", line 2, in <module>
    from .api.psd_image import PSDImage
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/psd_tools/api/psd_image.py", line 10, in <module>
    from psd_tools.psd import PSD, FileHeader, ImageData, ImageResources
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/psd_tools/psd/__init__.py", line 10, in <module>
    from .base import BaseElement
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/psd_tools/psd/base.py", line 207, in <module>
    class NumericElement(ValueElement):
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/psd_tools/psd/base.py", line 211, in NumericElement
    value = attr.ib(default=0.0, type=float, converter=float)
TypeError: attr() got an unexpected keyword argument 'type'

Code that I used:
def test_psd_icc_profile():
    from psd_tools.api.psd_image import PSDImage

    psd = PSDImage.open('<path_to_file>/TEST_merged.psd')
    no_icc = psd.compose(apply_icc=False)
    with_icc = psd.compose(apply_icc=True)

    print(no_icc.getextrema())
    print(with_icc.getextrema())

Do you have ideas why I have this error?
Versions of required extensions are:
Python 3.6.4
psd-tools == 1.8.12
numpy == 1.16.1
scipy == 1.2.1
Thank you!

Can you try this?
pip install -U attrs

@kyamagu Thank you, it works.


Expose sorting variable argument in the command line interface

As discussed with @cheyenneland, time_variable is an argument to stitchee() and is used for determining the file order before concatenation, but time_variable is not yet exposed in the CLI defined in run_stitchee.py.   Being able to set time_variable when running stitchee from the command line will allow setting it to the appropriate value for each mission's data.

@cheyenneland, this has been merged into the develop branch, with a new name of sorting_variable and a default to not sort the input datasets.  So I think this is ready to test with CERES data.


Improve message when a component cannot be loaded

Original discussion can be found in #167.
Proposal
If the module is not aliased, we should add a hint:
warning: cannot render <Form> (module Form could not be loaded)

Hint: Make sure module `Form` can be successfully compiled.

If the module is namespaced, you can use its full name. For instance:

  <MyProject.Components.Form>

or add a proper alias so you can use just `<Form>`:

  alias MyProject.Components.Form

If the module is already aliased, we don't need the hint but we have to show both, the alias and full module's name:
warning: cannot render <Form> (module MyProject.Components.Form could not be loaded)

This is totally what I was looking for when I got the error about module Link is not loaded and could not be found
Maybe adding the text here will help someone 👍
Thank you so much, @msaraiva !


View issues assigned to nobody

is there currently a way to filter issues assigned to nobody?

Nope...

cheers, this would be quite useful


docs: add an example function to README.md

Think this is sufficient?

Coverage remained the same at 99.356% when pulling 5bb842e5675e0f19009bad6cc6d43f1f4ee30f3b on pbredenberg:docs-examples into 7a9f8c3498585679fe4038b5cf0ebf9d04e5db6e on silvermine:master.


[ ] UI Enhancement - Filters/Slicers Pane:

          - [ ] UI Enhancement - Filters/Slicers Pane:
- Task: Update the styling of the filters/slicers pane by adding a distinct background color or some form of highlighting to visually separate it from the dashboard charts.
- Objective: Improve visual separation and enhance user focus on different sections of the dashboard.

Originally posted by @arturoboquin in https://github.com/UBC-MDS/DSCI-532_2024_12_bigmac/issues/70#issuecomment-2059954811

This issue has been addressed during the update on milestone 3. Check this release (0.3.0).


Plugin Beta

Plugin Method:

parseNovelAndChapters parseNovel(novelPath: string):

return: novel and its firstPage chapters: ChapterItem[] (or all chapters if it doesnt have pages).

parsePage(novelPath: string, page: string):

return chapter list ChapterItem[] and latestChapter (for DESC only novels).

Plugin properties:

state: YourDefinedStateType = {}

get a state value: state.someState;
set a state value: state.someState = someValue;

Plugin related types:   - NovelItem: url -> path

SourceNovel: url -> path, totalPages: number
ChapterItem: url -> path, page?: string

If page is not provided, default value is page parameter passed in fetchChapters or 1 if parseNovel
Which case we should provide a page for ChapterItem?: If novel has volumes (eg: volume 1, artbook, short story), app will treat them as pages.

1 done in https://github.com/LNReader/lnreader-sources/pull/769
3 done in https://github.com/LNReader/lnreader-sources/pull/757


Constructing ODataError from WebResponseException

OData has a standardised error reporting mechanism via ODataError, but WebResponseException just exposes a Response string. Additional MS don't seem to provide any easy way of de-serializing this structure back into an ODataError.
As this is useful on the client, provide extension methods to convert/expose the ODataError to standardize error handling.
I can probably provide a pull request for this if your happy

Being dumb and/or tired - JsonConvert.Deserialize


python3Packages.spyder_3: remove broken application

Motivation for this change
python3Packages.spyder_3 was failing due to incorrectly defined desktop item:
https://hydra.nixos.org/build/117762361
Things done
python3Packages.spyder_3 should only be used as a library (python3Packages.spyder gets made into an application), so I removed all the application stuff from it. This reduced closure size and fixed the build.

[x] Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
Built on platform(s)

[x] NixOS
[ ] macOS
[ ] other Linux distributions

[ ] Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
[ ] Tested compilation of all pkgs that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
[ ] Tested execution of all binary files (usually in ./result/bin/)
[ ] Determined the impact on package closure size (by running nix path-info -S before and after)
[ ] Ensured that relevant documentation is up to date
[ ] Fits CONTRIBUTING.md.

Also tested cq-editor, which is the only attribute that uses spyder_3, and it works fine.
I should probably backport this to 20.03 as well.

Why shouldn't this be possible to use as application as well?

@FRidh I didn't think anyone would use this v3.3.6 over the v4.1.2 we currently have in nixpkgs.
But if anyone would like to use it I'm happy to add it in. Do you want it added?

OK, I see that 4 works now as well (https://github.com/NixOS/nixpkgs/pull/83038). Why should it remain as library then?

Yeah, just cq-editor. I actually tried to modify the cq-editor code to work with spyder v4, but that task was too difficult for me.


Unable to sign in

Android beginner trying to learn FIrestore here.
When trying to run this app with all the dependencies updated to the latest available version, I am unable to to sign in. I enter my email and click on next but the home screen is only shown for a second and it immediately switches back to the Sign in screen asking me to enter my email. The tutorial says I should be able to sign in and see the home screen without changing any code. Please forgive me if this issue is because I did something wrong. Thanks!
P.S. I have tried running the app as is, with all the included outdated dependencies and I still face the same issue.

@ganesh3s3 are there any errors in the logs?  Do you have email authenticaiton enabled in the Firebase console?

Ty for your reply. I managed to fix the problem myself. The issue was I had other apps in the project and I used my personal email to sign-in in those. Since I gave this app the same email it was unable to sign me in since that email address was already in the list of users.
I was really confused since neither the app nor the logcat gave me enough information about what went wrong. Now everything is clear.


BUG: order of elements in some sections

Is there an existing issue for this?

[X] I have searched the existing issues

What happened?
The map should come after form in "Contact " section
The form should also have better UI
Record

[X] I agree to follow this project's Code of Conduct
[X] I'm a GSSOC'24 contributor
[X] I want to work on this issue

Is there an existing issue for this?

[x]  I have searched the existing issues

What happened?
The map should come after form in "Contact " section The form should also have better UI
Record

[x]  I agree to follow this project's Code of Conduct
[x]  I'm a GSSOC'24 contributor
[x]  I want to work on this issue

Assigned!
Make sure to attach before & after screenshots in your PR and include the issue number.
Also, please star the repo before working on it. Happy open-source contributing!


UI Inconsistency: Forms (new/edit) buttons labels

🐛 Bug Report
There is an inconsistency in the form save button labels when we create a new entity (patient, appointment, lab, medication, imaging request, incident)
To Reproduce
Create a new patient OR new appointment OR new medication OR new lab OR new incident OR new imaging request
Notice the footer buttons in some instances you have "Save" in other instances you have "Request"; we need to have uniformity.
Expected behavior
New entry creation

For creating a new Patient the save button label should say "Create Patient"
For scheduling an appointment the save button label should say "Schedule Appointment"
For requesting a lab the save button label should say "Request Lab"
For requesting a medication the save button label should say "Request Medication"
For requesting imaging the save button should say "Request Imaging"
For reporting an incident the save button should say "Report Incident"

Editing existing entries

Make sure all edit forms say "Update" + the name of the entity being updated e.g. "Update Patient", "Update Lab", "Update Medication", "Update Incident" , "Update Imaging", "Update Appointment"

Your Environment

node version: 6,8,10
fastify version: >=1.0.0
os: Mac, Windows, Linux
any other relevant information

Hi, @fox1t @tehKapa @jackcmeyer -- would love to take this on!

Hi, @blestab @fox1t @tehKapa @jackcmeyer -- would love to take this on!

Thanks @cpondoc , all yours.


Introduce Maven BOM for AssertJ artifacts

Summary
To ease consumption of AssertJ artifacts, it would be great to introduce a Maven Bill Of Materials POM (search for BOM here) for all AssertJ artifacts that are released as part of an AssertJ release (assertj-core, assertj-swing, assertj-guava, …).
A BOM simplifies consumption of artifacts that wish to use one or more AssertJ dependencies. It bundles compatible artifact versions. BOM usage no longer requires dependency descriptors to declare versions for individual artifacts, only a single version identifier is required to import the BOM.
See the netty-bom project from Netty for inspiration.

I have created a new project for the assertj bom and moved this issue there https://github.com/assertj/bom/issues/1.
Might be a bit overkill for just one pom.xml ... but I haven't thought of a better option at the moment.

Reopening as we'll add a BOM as a consequence of #2424.


Update azure-pipelines-poi.yml for Azure Pipelines

Abandoning request, it's pulling to the wrong branch


Bot doesnt follow back on Bsky

The bot does not like me! What did I do to them :(

can you send a link to your profile?

I sent you an email

There are a few requirements: https://fed.brid.gy/docs#troubleshooting Make sure you have a profile picture and that your account isn't hidden from guests (since that property can't be translated into ActivityPub).

Thank you! The missing profile picture was the issue.


revise discrete dp

Clean up per https://github.com/QuantEcon/lecture-source-jl/issues/439.

OK @jlperla see now.


feat: add generic Box helper

Description
Add generic Box helper class that allows sharing values.
How Has This Been Tested?
Please describe the tests that you ran to verify your changes.
Minimal checklist:

[x] I have performed a self-review of my own code
[ ] I have added DocC code-level documentation for any public interfaces exported by the package
[x] I have added unit and/or integration tests that prove my fix is effective or that my feature works

I would add comment that it's only intended for value types.


Cloud Api keys if already exist,Plan does not create new keys.

Can we implement Cloud Api keys if already exist,Plan does not create new keys.

I created Kafka Api keys for app-consumer,when next time I run pipeline it should not created other api keys.
Is it any way I can restrict that.

Could you share your TF config? It's not clear why pipeline created other API Keys if you have created Kafka API Key already. @Davinder2609

@linouk23
terraform {
required_providers {
confluent = {
source  = "confluentinc/confluent"
version = "1.34.0"
}
}
}
provider "confluent" {
cloud_api_key    = var.confluent_kafka_cloud_api_key
cloud_api_secret = var.confluent_kafka_cloud_api_secret
}
resource "confluent_service_account" "ServiceAccount" {
display_name = var.confluent_kafka_service_account_name
description  = var.confluent_kafka_service_account_description
lifecycle {
prevent_destroy = true
}
}

I see, does terraform plan for this pipeline results in creation of any API Keys at all?
BTW cloud_api_key = var.confluent_kafka_cloud_api_key make sure that confluent_kafka_cloud_api_key here is Cloud API Key and not Kafka API Key.

cc @Davinder2609

@linouk23  Yes this plan create Cluster Api key and SchemaRegistry keys.but issue is when we re run this plan again.It created Keys again

It sounds a little bit unexpected to me @Davinder2609 since your .tf config doesn't have any API Keys defined at all. Do you want to share your terraform plan output (make sure to delete all the sensitive information) that shows creation of Cluster / SR API Keys?

@linouk23
Initialize terraform
terraform init
echo "Creating terraform plan....."
Creation of terraform plan
terraform plan -var "confluent_kafka_service_account_id=$CONFLUENT_KAFKA_SERVICE_ACCOUNT_ID"
-var "confluent_kafka_cloud_api_key=$CONFLUENT_KAFKA_CLOUD_API_KEY"
-var "confluent_kafka_cloud_api_secret=$CONFLUENT_KAFKA_CLOUD_API_secret"
-var "confluent_kafka_env_id=$CONFLUENT_KAFKA_ENV_ID"
-var "confluent_kafka_cluster_id=$CONFLUENT_KAFKA_CLUSTER_ID"
-var "confluent_schema_cluster_id=$CONFLUENT_SCHEMA_CLUSTER_ID"
-var "confluent_kafka_cluster_rest_end_point=$CONFLUENT_KAFKA_CLUSTER_REST_END_POINT"
-var "confluent_kafka_service_account_name=$CONFLUENT_KAFKA_SERVICE_ACCOUNT_NAME"
var"confluent_kafka_service_account_description=$CONFLUENT_KAFKA_SERVICE_ACCOUNT_DESCRIPTION"
-out output.terraform
echo "Execution of terraform main plan module to create service account and schema registry keys is completed."
terraform apply "output.terraform"
#creating json files to add keys.
echo "Creating terraform deployment output json file."
terraform output -json > $TERRAFORM_MAIN_DIR/keys.json
If We re run this plan keys will created again

Gotcha, could you share the output of
terraform plan -var "confluent_kafka_service_account_id=$CONFLUENT_KAFKA_SERVICE_ACCOUNT_ID"
-var "confluent_kafka_cloud_api_key=$CONFLUENT_KAFKA_CLOUD_API_KEY"
-var "confluent_kafka_cloud_api_secret=$CONFLUENT_KAFKA_CLOUD_API_secret"
-var "confluent_kafka_env_id=$CONFLUENT_KAFKA_ENV_ID"
-var "confluent_kafka_cluster_id=$CONFLUENT_KAFKA_CLUSTER_ID"
-var "confluent_schema_cluster_id=$CONFLUENT_SCHEMA_CLUSTER_ID"
-var "confluent_kafka_cluster_rest_end_point=$CONFLUENT_KAFKA_CLUSTER_REST_END_POINT"
-var "confluent_kafka_service_account_name=$CONFLUENT_KAFKA_SERVICE_ACCOUNT_NAME"
var"confluent_kafka_service_account_description=$CONFLUENT_KAFKA_SERVICE_ACCOUNT_DESCRIPTION"

@Davinder2609
Based on
terraform {
required_providers {
confluent = {
source = "confluentinc/confluent"
version = "1.34.0"
}
}
}

provider "confluent" {
cloud_api_key = var.confluent_kafka_cloud_api_key
cloud_api_secret = var.confluent_kafka_cloud_api_secret
}

resource "confluent_service_account" "ServiceAccount" {
display_name = var.confluent_kafka_service_account_name
description = var.confluent_kafka_service_account_description
lifecycle {
prevent_destroy = true
}
}

it looks like your TF should only create 1 service account.

Are you sure there's just one .tf file in your folder / module / $TERRAFORM_MAIN_DIR? @Davinder2609

@linouk23  we have main.tf, output.tf, variables.tf files

Sounds good @Davinder2609!
Gotcha, could you share the output of
terraform plan -var "confluent_kafka_service_account_id=$CONFLUENT_KAFKA_SERVICE_ACCOUNT_ID"
-var "confluent_kafka_cloud_api_key=$CONFLUENT_KAFKA_CLOUD_API_KEY"
-var "confluent_kafka_cloud_api_secret=$CONFLUENT_KAFKA_CLOUD_API_secret"
-var "confluent_kafka_env_id=$CONFLUENT_KAFKA_ENV_ID"
-var "confluent_kafka_cluster_id=$CONFLUENT_KAFKA_CLUSTER_ID"
-var "confluent_schema_cluster_id=$CONFLUENT_SCHEMA_CLUSTER_ID"
-var "confluent_kafka_cluster_rest_end_point=$CONFLUENT_KAFKA_CLUSTER_REST_END_POINT"
-var "confluent_kafka_service_account_name=$CONFLUENT_KAFKA_SERVICE_ACCOUNT_NAME"
var"confluent_kafka_service_account_description=$CONFLUENT_KAFKA_SERVICE_ACCOUNT_DESCRIPTION"

It's also kinda weird that you use that many variables given that you're not using them in

provider "confluent" {
cloud_api_key = var.confluent_kafka_cloud_api_key
cloud_api_secret = var.confluent_kafka_cloud_api_secret
}

resource "confluent_service_account" "ServiceAccount" {
display_name = var.confluent_kafka_service_account_name
description = var.confluent_kafka_service_account_description
lifecycle {
prevent_destroy = true
}
}

Based on the overall description, it appears to be a misconfiguration of Terraform rather than an issue with the TF Provider for Confluent. I am inclined to resolve this issue since I am unable to reproduce it. However, if you could provide a complete output of terraform plan that highlights the creation of all these new API Keys (confluent_api_key), it would be helpful.

@linouk23
Changing to terraform main directory
Initializing the backend...
Initializing provider plugins...

Finding confluentinc/confluent versions matching "1.22.0"...
Installing confluentinc/confluent v1.22.0...
Installed confluentinc/confluent v1.22.0 (signed by a HashiCorp partner, key ID D4)

Partner and community providers are signed by their developers.
If you'd like to know more about provider signing, you can read about it here:
https://www.terraform.io/docs/cli/plugins/signing.html
Terraform has created a lock file .terraform.lock.hcl to record the provider
selections it made above. Include this file in your version control repository
so that Terraform can guarantee to make the same selections by default when
you run "terraform init" in the future.
Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.
If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.
confluent_service_account.ServiceAccount: Importing from ID "sa-03r***"...
confluent_service_account.ServiceAccount: Import prepared!
  Prepared confluent_service_account for import
confluent_service_account.ServiceAccount: Refreshing state... [id=sa-03r****]

Import successful!
The resources that were imported are shown above. These resources are now in
your Terraform state and will henceforth be managed by Terraform.

Creating terraform plan.....
confluent_service_account.ServiceAccount: Refreshing state... [id=sa-03r**]
Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
+ create
~ update in-place

Terraform will perform the following actions:
  # confluent_api_key.kafka-cluster-api-key will be created
  + resource "confluent_api_key" "kafka-cluster-api-key" {
+ description            = (sensitive value)
+ disable_wait_for_ready = false
+ display_name           = (sensitive value)
+ id                     = (known after apply)
+ secret                 = (sensitive value)
  + managed_resource {
      + api_version = "cmk/v2"
      + id          = (sensitive value)
      + kind        = "Cluster"

      + environment {
          + id = (sensitive value)
        }
    }

  + owner {
      + api_version = "iam/v2"
      + id          = "sa-03r****"
      + kind        = "User"
    }
}

  # confluent_api_key.schema-registry-api-key will be created
  + resource "confluent_api_key" "schema-registry-api-key" {
+ description            = (sensitive value)
+ disable_wait_for_ready = false
+ display_name           = (sensitive value)
+ id                     = (known after apply)
+ secret                 = (sensitive value)
  + managed_resource {
      + api_version = "srcm/v2"
      + id          = (sensitive value)
      + kind        = "Cluster"

      + environment {
          + id = (sensitive value)
        }
    }

  + owner {
      + api_version = "iam/v2"
      + id          = "sa-03r***"
      + kind        = "User"
    }
}

  # confluent_role_binding.schema-manager-schema-registry-admin will be created
  + resource "confluent_role_binding" "schema-manager-schema-registry-admin" {
+ crn_pattern = (sensitive value)
+ id          = (known after apply)
+ principal   = "User:sa-03r***"
+ role_name   = "DeveloperRead"
}
  # confluent_service_account.ServiceAccount will be updated in-place
  ~ resource "confluent_service_account" "ServiceAccount" {
# Warning: this attribute value will be marked as sensitive and will not
      # display in UI output after applying this change. The value is unchanged.
~ description  = (sensitive value)
# Warning: this attribute value will be marked as sensitive and will not
      # display in UI output after applying this change. The value is unchanged.
~ display_name = (sensitive value)
id           = "sa-03r****"
# (2 unchanged attributes hidden)
}
Plan: 3 to add, 1 to change, 0 to destroy.

Changes to Outputs:
+ kafka_cluster_api_key      = (sensitive value)
+ kafka_cluster_api_secret   = (sensitive value)
+ schema_registry_api_key    = (sensitive value)
+ schema_registry_api_secret = (sensitive value)
Saved the plan to: output.terraform
To perform exactly these actions, run the following command to apply:
terraform apply "output.terraform"
Execution of terraform main plan module to create service account and schema registry keys is completed.
 Resources: 3 added, 1 changed, 0 destroyed.

Outputs:
kafka_cluster_api_key =
kafka_cluster_api_secret =
schema_registry_api_key =
schema_registry_api_secret =
service_account_id =
Creating terraform deployment output json file.
Output json file is created.

OK so we can see that your TF config contains other resources listed outside of main.tf:
�[0m�[1mconfluent_role_binding.schema-manager-schema-registry-admin: Still creating... [1m10s elapsed]�[0m�[0m
�[0m�[1mconfluent_api_key.kafka-cluster-api-key: Still creating... [1m10s elapsed]�[0m�[0m

Moreover, it seems like you terraform apply went through:
Apply complete! Resources: 3 added, 1 changed, 0 destroyed.

Gotcha, well the question is why

Resource confluent_api_key.app-consumer-schema-registry-api-key has lifecycle.prevent_destroy set, but the plan calls for this resource to be destroyed.

Could you remove lifecycle.prevent_destroy and rerun terraform plan to see a more detailed error message? @Davinder2609

without lifecycle.prevent_destroy -
Terraform will perform the following actions:
confluent_api_key.app-consumer-schema-registry-api-key must be replaced
-/+ resource "confluent_api_key" "app-consumer-schema-registry-api-key" {
~ id                     = "L******" -> (known after apply)
~ secret                 = (sensitive value)
# (3 unchanged attributes hidden)
  ~ owner {
        id          = "sa-***"
      ~ kind        = "ServiceAccount" -> "User" # forces replacement
        # (1 unchanged attribute hidden)
    }

    # (1 unchanged block hidden)
}

Plan: 1 to add, 0 to change, 1 to destroy.
───────────────────────────────────

@linouk23
lifecycle {
prevent_destroy = true
}
is not working for schemaRegistry keys

Right and could you share IDs as I suggested in the message above? I wanted to double check it's not a backend bug even though I can't reproduce it locally now.

cc @Davinder2609

@Davinder2609 could you confirm that drift
  ~ owner {
        id          = "sa-***"
      ~ kind        = "ServiceAccount" -> "User" # forces replacement
        # (1 unchanged attribute hidden)
    }

    # (1 unchanged block hidden)
}
Plan: 1 to add, 0 to change, 1 to destroy.

went away and new output for terraform plan command is
No changes. Your infrastructure matches the configuration.

Terraform has compared your real infrastructure against your configuration and found no differences, so no changes are needed.

or something now so we could close this issue?

@linouk23  thanks ! You can close the issue.

@linouk23  Is it any way we can restrict  cloud api key.schema key creation if already exist through terraform

I don't think so -- a user could use CLI / CC UI to create a new Schema Registry API Key if that's what your question is about. Feel free to create a new issue if anything.


Making lbry.tv GDPR compliant

Our advertising partner requires that we become gdpr compliant on lbry.tv by Sept 30.
While there are several facets of compliance, a main and obvious one is the 'cookie warning banner' you see on many sites today.
https://www.cookieyes.com/gdpr-cookie-consent-banner-examples/
An 'informational' type is a good start for us.  It should run along the bottom of the screen and disappear when agreed-to.

lbry.tv uses cookies and the GDPR is dumb [Agreed]

Do we just need to change the text on the banner that is already being used? Or are larger changes required?

I think this banner is a perfect place.  We just need to use the version of
the text that says they are agreeing to the cookies, not the option of
rejecting them.
On Thu, Aug 27, 2020 at 2:56 PM Sean Yesmunt<EMAIL_ADDRESS>wrote:

Do we just need to change the text on the banner that is already being
used? Or are larger changes required?
[image: Screen Shot 2020-08-27 at 2 54 57 PM]
https://user-images.githubusercontent.com/16882830/91483204-50a48b80-e875-11ea-8dbf-9fd000cb66d3.png
—
You are receiving this because you were assigned.
Reply to this email directly, view it on GitHub
https://github.com/lbryio/lbry-desktop/issues/4713#issuecomment-682132062,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AD5BQDAO2KX4DFLDNI4X6KDSC2T5DANCNFSM4QNC4R6A
.

Does this still need to be done? What is this holding back, if anything?

I suggest creating something like https://www.youtube-nocookie.com/embed/ but for embedded odysee videos. As a business owner, I wouldn't embed odysee videos unless there is a way to embed them without cookies.
As for the cookie banner at the bottom of the page you can buy something like OneTrust CookiePro or Cookiebot. It is not enough to just display a banner and click something to dismiss the banner. You need to be able to have your users control which cookies they are willing to accept/decline. Parts of the website will not work unless cookies are accepted.
See this as an example of what you should create:
https://stripe.com/cookie-settings
Something needs to be done about cookies otherwise you will have some big legal fines to pay.


proposed way to fix . not being included in @INC

This same thing will need to be done to all of the .pl files in the EDS/opac that include eds-methods.pl

Got bitten by this recently as well.
Here's some official words on the subject: https://metacpan.org/pod/release/XSAWYERX/perl-5.26.0/pod/perldelta.pod#Removal-of-the-current-directory-(".")-from-@INC


NSM architecture review.

Hi guys,
After some local discussions, and working on some parts of project like local NSM <-> remote NSM connectivity,
I think we need to re-think of how architecture is designed right now.
Since we are on very early stage we could have this more property designed and discussed.
Right now we are forcing an issue with too many "failure points" and requirement to have a transaction log on NSC -> NSE connection establishment and doing a proper rollback of all operations during Close, failures.
In case of NSC and NSE are on different nodes/etc we need to monitor troubles on two Dataplanes and in two NSMs and do a complicated distributed cleanup or think abour smart connection re-establish if it will be possible, (Should be possible in case of VxLan, not sure about all cases).
So main question here, do we really need a multiple NSM on every node?
We already have a distributed NSMD-k8s NSE registry centralization point using a decentralized shared Key Value storage.
Right now I've see following trouble points for us to handle connection:

And same for handling Close()

So I've have following idea to discuss:

Have Decentralized registry will hold following information

NSE/Dataplanes registrations.
NSM liveness state.
ClientConnection's

Every operation we perform on establish, close, recovery of connection.
A current responsible NSM.

Make NSMD a fully statelesss and have 1-M per cluster, we do not need it for be on every node.

Will check registry for other NSM die to continue their work.
Handle Client requests to put information into de-centralized storage.

Dataplanes will be same on every done.

Will be connected to any of NSM for live monitoring.
In case of NSM die, will connect to any live one.
Will report CrossConnects, connection failures to any of NSM and start Close()/Recovery operations.

Clients will connect to any suitable NSM for Requesting connection to NSE, and we respond async for execution of connection establish.

NSM will respond immidiatelly if there is no NSE.
NSM will respond async after all points will be configured.

And illustration of how it will work:

So I see a lot of complex scenarios will be easier to do and it would be more scallable and cluster oriented.
Also with this architecture we could have one NSM(fully centralyzed) variant with all information stored to ACID storage(file system) on local volume, so we are mostly independent of cluster implementation. Or we could use some of de-centralyzed data bases to implement this logic.
In this approach we do not need to have Dataplanes to hold any of cross connects, we could have monitoring API to be done on every NSM for all network.
Please share your thoughts.

Handling Failures during Setup of Remote Cross Connect

NSC sends local.Request(local.Connection) to nsmd1
nsmd1 sends FindNetworkService(FindNetworkServiceRequest) to API Server (admittedly, indirectly via nsmd-k8s).
nsmd1 sends remote.Request(remote.Connection) to nsmd2
nsmd2 sends local.Request(local.Connection) to the NSE Pod
NSE Pod Replies with a local.Connection
nsmd2 sends crossconnect.Request(CrossConnect) to the Dataplane on Node2.
The Dataplane on Node2 Replies to nsm2 with a CrossConnect
nsmd2 Replies to nsmd1 with a remote.Connection
nsmd1 sends crossconnect.Request(CrossConnect) to the Dataplane on Node1.
The Dataplane on Node1 Replies with a CrossConnect to nsmd1
nsmd1 Replies to NSC with a local.Connection

Any breakdown along this path can be fairly easily recovered in a distributed way, without need of a central transaction log or connection repository.
Recovering for something that goes wrong in the course of this flow is fairly simple:

If something goes wrong when processing a 'Request', call 'Close()' on any downstream state you have created with your own Requests, and then return an error to the party that sent you a Request.

This actually works for more complex situations that are shown here (say for example the NSE Pod itself 'Requests' a Network Service in response to nsmd2's Request ... if it follows this simple rule, we get proper cleanup.
More in subsequent entries on how to reconcile more complex failure modes.

Handling NSC Dying

NSC Dies
Dataplane Detects NSC is gone by detecting its NetNs is gone.  Marks local.Connection for NSC as state 'SRCDOWN' (TODO: File bug for local.Connection.State) in CrossConnect
MonitorCrossConnect returns from Dataplane on Node1 to nsmd1 a CrossConnectEvent of type Update with the CrossConnect with the updated Source Connection State.
nsmd1 extracts the CrossConnect.GetRemoteDestination() from the updated CrossConnect, and sends remote.Close(remote.Connection) to nsmd2
nsmd2 looks up in its memory the CrossConnect for which the Connection closed by nsmd2 is the Source.  It gets CrossConnect.LocalDestination() to get the Connection for the local NSE.  nsmd2 sends Close(Connection) to the NSE
NSE Pod Replies
nsmd2 sends Close(CrossConnect) to the Dataplane for Node2.
Dataplane on Node2 sends Reply
nsmd2  Replies to nsmd1
nsmd1 sends Close(CrossConnect) to the Dataplane on Node1
Dataplane on Node1 replies.

Handling NSE Dying (Autohealing)

NSE Dies
Dataplane2 detects the NetNS of NSE is gone
Dataplane2 returns MonitorCrossConnectEvent to nsmd2.  Type == Update, Con3.State == DST_DOWN.
nsmd2 sends RemoveNSE to API Server (indirectly)
nsmd2 sends Close(XC2) to Dataplane2
Dataplane2 Replies
nsmd2 returns MonitorConnectionEvent to nsmd1.  Type == DELETE
nsmd1 return MonitorConnectionEvent to NSC, Type == UPDATE, Con1.State == DST_DOWN.  Note: This is optional, as most NSC won't have called MonitorConnection.

Which leaves us with this:

nsmd1 sends FindNetworkService to the API Service (indirectly) to find a new NSE that provides the Network Service
nsmd1 sends Request(Con4) to nsmd3
nsmd3 sends Request(Con5) to the NSE
NSE sends Reply to NSE
nsmd3 sends Request(XC3) to Dataplane3
Dataplane3 Replies to nsmd3
nsmd3 returns Reply(Con4) to nsmd1
nsmd1 replaces Con2 in XC1.RemoteDestination with Con4 and sends Request(XC1) to Dataplane1
Dataplane1 recognizes XC1 (by id) is an existing CrossConnect, and that XC1.Destination has been replaced.  Dataplane1 creates its end of Con4, and updates the XCon from Con1<->Con2 to Con1<->Con4 and then Dataplane1 Replies to nsmd1.

Note: #9-#15 is just the normal Connection Setup flow.

nsmd1 returns to NSC a MonitorConnectionEvent of Type == UPDATE and Con1.State == UP.

One other note... it should be true within the Dataplane that translations from CrossConnects to things done in vppagent are purely deterministic.  This means that if the Dataplane needs to delete or recreate something from the CrossConnect, it can.

Note: All images for this issue are here: https://docs.google.com/presentation/d/1TGNqhV41dl4RlCR3eeLeMIUnUeuU8m3LsrmdlPgX12c/edit#slide=id.g4a27cce692_0_0
In case you want to copy/modify.

Thanks @edwarnicke, I've not supposed you will describe all cases with code links, I've not talk this is imposible, as we discussed we are working on re-think of how it should be, and variant we choose right now looks a bit more complicated when I expected.
Assume situation with remote connection is going on, NSM1 pass to NSM2, all is configured on NSM2 connection is returned to NSM1 and we start program Dataplane, and NSM1 node dies, in this case NSM2 will have partially configured connection and nobody knows it had, client will have error and will try again. So we somehow need to store into ACID of NSM1 about actions we perform.
Same if NSM2 node dies, but in this case we need to monitor from NSM1 about NSM2 cross connect changes, and we need to do so actually to be able to auto-heal.
But this suddenly could happen for every NSM do monitor for every NSM in cluster, and this not looks good to me, will think more about this questions.

@haiodo I'm halfway through writing those sorts of cases up... but net net... you don't have to monitor every nsmd, only every nsmd with which you are a peer (ie, have a connection).  You can learn that from the CrossConnects from the Dataplane if the nsmd restarts.

Yes, but it could happen :-)

@haiodo It can only happen if you have a connection from the nsmd to every other nsmd in the cluster.
Fundamentally... central DBs lag reality.  Peer connections reflect reality.

I've not talk about centralyzed DB, I've think about we could use ETCD directly and for example NSMD will know NSMD he has connections to and monitor a ETCD values for liveness for this particular NSMD.

@haiodo Fundamentally, NSM if able of cross-domain behavior... no central DB will ever be able to hold all relevant information, because not all of it is going to live within the view of a single Network Service Registry domain.  Similiar to how we can't store TCP connection state in a central DB for the world... because much of it is not in its view.

@haiodo I could see using bolt to store some information locally.  Tends to be super light.  But the world still moves on between when the nsmd goes down and when it comes back... so you still have to synchronize with your peers to reflect reality.

@haiodo As to nsmd liveliness... there's a reason we have a "LastSeenField" in the NSM CR... we don't use it well yet... but we do have it :)

Ok, Thanks, will think more about this today.
Also I'm living in theory if something could happen, when users will make this happen for sure :-)

@haiodo Not a bad theory :)  I also live in the theory that the state in the DB is not truth, it is at best hints of what was true in the past.  There's always a time lag between when you were last writing the the DB, and the now... so you still have to figure out what's real.

nsmd restarts
When an nsmd starts (or restarts) it currently waits for a Dataplane to Register.
Once the Dataplane has registered:

nsmd calls MonitorCrossConnects on Dataplane, and learns the set of all CrossConnects the Dataplane has.
From the CrossConnects, nsmd can learn all of the CrossConnect.RemoteSource() and CrossConnect.RemoteDestinations().  Each of those are remote.Connections, and container the remote nsmd name, network service name, etc.
From that the nsmd can establish a new remote.MonitorConnection to each peer nsmd, learning what connections that peer believes itself to have with the nsmd.
from the CrossConnects, nsmd can learn all of the CrossConnect.LocalDestination() and thus the workspace.
From the Workspace the nsmd can attempt to call local.MonitorConnections() on each workspace to get local NSE's opinions about their Connections.

Reconciling remotePeers
So for each remotePeers, the nsmd now has two views of reality: the remotePeer's, andthe Dataplane's.
These need to be reconciled.  The basic priniciples of reconciliation are:

If the Dataplane has a CrossConnect with a RemoteDestination() that is unknown to the remotePeers, first attempt to Request() the Connection learned from the Dataplane.  If that fails, carry out the autohealing procedure.
If the Datplane has a CrossConnect with a RemoteSource() that is unknown to the remotePeers, Close the Connection with the local NSE and Close the CrossConnect with the Dataplane
If the remotePeers has a Connection with the nsmd for which the nsmd is listed as the source nsmd the Connection is unknown to the Dataplane, send Close(Connection) to the remotePeers for those Connections

Reconciling localPeers
For each localPeer, the nsmd now has two views of reality: the localPeer's, and the Dataplane's.
These need to be reconciled.  The basic principles of reconciliation are:

If the Dataplane has a CrossConnect with a LocalDestination() that is unknown to the local NSE, Request() that local.Connection to the local NSE.
If the localPeer has a Connection that is unknown to the Dataplane, Close() the Connection with the local NSE.

@haiodo I have given some thoughts to having the vppagent-dataplane process squirrel away some data in bolt, in case the vppagent-dataplane process dies for some reason (and is restarted by supervisord) and the vppagent and vpp processes keep on going.

Some notes on keeping the state in process.
Have a look at what @lobkovilya did in #605 in terms of having the MonitorXXXServer be both the in memory store of state and also the thing that notifies any possible in process listeners for state changes.

Yes, it was based on our internal discussions.

Questions is just for entire node dead handling ( NSMD + Dataplane) die together.

@haiodo If the entire Node goes down... its all gone, there's nothing to recover :)  All NSC and NSEs on the Node are also gone.

@haiodo The remotePeers of the nsmd on the Node that goes down will either:

Not be able to re-establish their MonitorConnection calls after a reasonable period of time
Re-establish them to an nsmd that believes it has no Connections with that remotePeer, the remotePeer can then take appropriate action:
2.1 If the remotePeer has a Connection with the nsmd as a Destination, it can first try to reconnect to the NSE, and if that fails, autoheal for its local NSC
2.2 Ifthe remotePeer has a Connection with the nsmd as a Source, it can Close() the connections with its local NSEs.

@edwarnicke After continuous discussion we discovered some cases that won't be covered by monitors between nsmd instances.
In case of Node goes down and it had NSE. Who will notify registry about NSE dies? Also when Node just disconnected from cluster and then comes back. Should it update registry about NSE that it has?
So maybe if we have distributed system we also should have distributed registry? Every nsmd will know only about NSEs that it has on the Node. It will share information about NSEs with nsmd on other nodes. It can be implemented as some routing protocol. I think it helps us to avoid situation when registry has NSE that already dies.
What do think about it?

@lobkovilya Good job on thinking through the corner cases!
Let's walk through your cases
Case 1. The Node goes down, and there are NSEs in the Network Service Registry (in the K8s case, the K8s API Server as NSE Custom Resources), and the Node does not come back up.
Case 2. The Node is partitioned from the Network Service Registry (in the K8s case, the K8s master), and has had NSEs register while partitioned from the master.
In Case1, the thought was to use NetworkServiceManager.last_seen.  Note: While the field is present now, we do not properly update it.  The basic algorithm would be to select an update interval, update last_seen every update interval, and consider the Network Service Manager gone if it has been more than 3*${update interval} since its last_seen.  This algorithm could be applied by an nsmd on lookup, it could check for timedout NSMs before selecting NSEs (each of which has an NSM).  One could also imagine having a Deployment independently monitoring and cleaning up NSMs (and their attendant NSEs) if they are timed out.
Now I know @platoff has enough experience (and I suspect @haiodo does to) to object:  "What about clock skew?!" and "What if different nsmds have different opinions about what the update interval is?!".  To which I would respond: do you guys have ideas to make the last_seen mechanism more robust to those issues? :)
In Case2 lets look at other effects of Node disconnect from master.  First, new Pods cannot be scheduled to a Node that is disconnected from master.  So that eliminates one class of situations that might result in NSEs not being properly registered.  So we are left with the case that a Pod runs, then the Node is partitioned from the master, and then the Pod seeks to register as an NSE.  In that case, the Pod calls RegisterNSE on the nsmd, the nsmd calls RegisterNSE on the nsmd-k8s, the nsmd-k8s seeks to register the NSE CR in the API Server.   nsmd-k8s will (eventually) fail producing an error.  That error should be passed back to the nsmd in response to its RegisterNSE call.  The nsmd should then pass that error back to the NSE in response to its RegisterNSE call.  So we are left with the case that no one, not the Pod, not the nsmd, and not the Network Service Registry believes that NSE has been registered.  In other words: Consistency.  So from the CAP Theorem, we choose C and P (Consistency and Partition) over Availability (you can't register an NSE if you are Partitioned from your Network Service Registry).  I'm open to discussing whether a different CAP choice is better, but this is at least a valid one :)
As to a fully distributed Registry.  Its an interesting thought.  Right now, we have a partial mesh between peers actively interacting with each other.  If there are 5000 nsmds in the Cluster (for 5k Nodes, which is the current K8s scale at the top) and a particular nsmd has connections with 10 of them, it only has 10 peers.  If we go full mesh, each nsmd would have need 5k peers.  If we start bringing in things from other clusters, and other eNSM cases it gets even bigger.  There are some real aspects of what we are doing in NSM that bring in routing kinds of wisdom, but it is probably not true that we want to go full mesh.

@lobkovilya @haiodo @platoff You guys are doing a good job thinking through the robustness architecture, asking good questions, bringing good ideas.  It sounds like you are close to having through the corner cases.
When you get there, it would be good to break this down into fairly granular issues and start working through them.  There are lots of things here that can be done as small independent pieces.   Since we are attracting more contributors, it's super helpful to have granular issues that folks can pick up if they get to them before you do.

Note on Connection.State.
When discussing Connect.State, I talked about states UP and DST_DOWN.  I think we can get away with just states UP and DOWN, with DOWN being the default enum value.  Note: This means a recipient of a "Request" will need to insure the Connections are marked "UP" before returning.

Hi folks, nice to see this discussion, and I personally still think we should have basically two types of components - NSM per cluster(s) and NSM Dataplanes (per node), for HA reasons there could be N NSMs per cluster, but conceptually it's a single nsm per cluster(s) (or single nsm per cluster and nsms from different clusters connected together).
Why? The decision to go distributed (nsm per node) opened a can of worms, and I believe this would reveal many problems in the future to catch on. Distributed systems are hard (even simple distributed system is hard). As a big fan of distributed systems I'd be happy to work on one, but for NSM case I do not see any benefit of distributed architecture - centralized solution would be easier to implement, more performant (scales better), and has more power in terms of future capabilities because of having centralized view/control of cluster from the single place.
If there is any benefit of the distributed solution versus a centralized one, I'll definitely change my view on this. As for performance scalability, we can go with some tests, but at this time, simple counting number of calls in distributed scenario, including calls to single-point-of-failure-and-performance-bottleneck k8s CRD database, we can see that centralized solution will be a way faster.
Potentially, fully distributed scenario (without k8s CRD database) can scale better, but only in case of bottleneck (centralized NSM) will not be able to perform its operations fast, which probably starts at millions of requests per second in a cluster, which is never.
The current design is not actually a distributed system, but a set of clients (nodes) communicating via shared database, and performance of the whole system is limited by the performance of that database. Switching to centralized NSM would have similar benefits as old desktop systems communicating through shared file database switch to client-server (application server) architecture (which is centralized nsm).
The current design very much resembles the hardware-based network, where NSM is chassis deployed to each node, and dataplanes installed into NSM slots. But we're clearly doing Software Defined Networking. Quick googling for what SDN is shows me this article (I'm pretty sure there are much better articles), but I'd like to quote this one to support the idea we should disaggregate dataplane and nsm. https://www.sdxcentral.com/sdn/definitions/inside-sdn-architecture/

In the SDN architecture, the splitting of the control and data forwarding functions is referred to as “disaggregation,” because these pieces can be sourced separately, rather than deployed as one integrated system. This architecture gives the applications more information about the state of the entire network from the controller, as opposed to traditional networks where the network is application aware.

The SDN architecture APIs are often referred to as northbound and southbound interfaces, defining the communication between the applications, controllers, and networking systems. A Northbound interface is defined as the connection between the controller and applications, whereas the Southbound interface is the connection between the controller and the physical networking hardware. Because SDN is a virtualized architecture, these elements do not have to be physically located in the same place.

Hi, great discussions here.
My 2cts: having a centralized NSMd probably makes sense as long as you stay in a NSM domain. But it seems we will still need some NSMd to NSMd interactions once we'll introduce eNSM use case, or Multi domain connectivity.
For instance, if we take the secure-intranet-connectivity use case, for now we only manage pods inside a single cluster. But we can easily think that the other side of the vpn (in the intranet) will be managed by a pod that terminates the vpn connection, and that this pod will also be managed by the NSMd located in the cluster of the intranet. The creation of the two pods (one in the cloud, one in the intranet) managing the vpn connectivity dynamically could happen through a negotiation between the two NSMd (one in the cloud domain, one in the intranet domain), as it is done currently between NSMd.
Yet, I'm not sure this use case is in the scope of NSM :)

@platoff I totally get your thinking here, and why a centralized NSMd sounds like a good idea.  It's the same idea that led the entire networking industry off the cliff in the first gen of SDN.
I was personally very deeply involved in that run at SDN, being a major contributor to the initial OpenFlow plugin on the OpenDaylight controller, doing a bunch of the work on the core of the current implementation of the ovsdb plugin in OpenDaylight, and contributing to many applications there.  I've got a lot of code under my fingernails in SDN.
I also followed from there the evolution of most other SDN solutions.
Experience is a harsh teacher.  What pretty much everyone in SDN learned over time is that:

Central controllers don't scale
Central controllers don't perform
HA for controllers is super super super hard to do right

What's evolved from that experience is that most SDN solutions have evolved into distributed systems that looks a lot like what we are doing here:

Central storage of critical, slow changing, intent information and information, and advertisement of globally relevant operational data
Moving decision making and canonical operational data as close to the dataplane that touches the packets as possible.

If you look at what we are doing here with NSM, it super analogous to TCP + DNS, which scales well, performs well, and is stable.  Moving the NSMd centrally is sort of analogous to moving the connection setup and teardown parts of the TCP centrally.
The really interesting new element of the system is that unlike the TCP stack, where if the kernel crashes you loose the whole Node, or to upgrade the TCP stack you have to take down the whole Node, in NSM we have the opportunity to survive the restart of the local nsmd or dataplane.  This introduces what looks like additional complexity, but its not "The system is more complex than TCP + DNS", its "The system has more opportunities than TCP + DNS, because we are not welded to the kernel".

@matrohon NSM is definitely interested in the use case of collaborating with things outside the cluster.  See the external NSM (eNSM) stuff in the K8s presentations :)

The main issue about centralized controllers that are not scaling is mainly an issue about maintaining states in databases. When you begin to have a lot of datas in the DB, updates and concurrents lookup can become costly. Is this the kind of scale issue you were having at Opendayligth @edwarnicke ?
Here we have three main components :

the registry (k8s CRD)
the controller (nsmd)
the dataplane-agent

If we go along with your TCP/DNS analogous, the registry is the DNS, while the TCP stack is divided into:

the controller for the connection establishment (SYN/SYNACK/ACK)
the dataplane-agent for packet forwarding

It seems all of those components are stateful, am I wrong? As shown during the discussion, it is painful to maintain.
If I'm correct, the NSMd should be stateless, as it is only there to connect NSC to NSE via a lookup in the registry.
If we talk about a new design for NSM, let's talk about the totally decentralized way (except for the registry) : don't you think it's possible to remove the NSMd, and move the server part inside endpoint pods, and the client part inside NSM client init-containers? This way, we would mimic the TCP/DNS model @edwarnicke  mentioned

I think current architecture can’t beat centralized nsmd (cnsmd) architecture in terms of scalability.
Let’s write down calls required to connect NSC to NSE in current architecture:
1a. NSC calls NSMD1
2a. NSMD1 calls local nsmd-k8s
3a. Local nsmd-k8s performs 1..N calls to k8s master
4a. NSMD1 calls NSMD2
5a. NSMD2 calls NSE
6a. NSMD2 calls Dataplane2
7a. NSMD1 calls Dataplane1
Centralized NSMD calls would be the following:
1b. NSC calls CNSMD
2b. CNSMD calls NSE
3b. CNSMD calls Dataplane2
4b. CNSMD calls Dataplane1
In current architecture we assume calls are distributed among different NSMD/Dataplane instances, so it’s hard to impossible to understand limitations of the whole system without real simulation, but call (3a) is a clear bottleneck in the current architecture:
First, this is a call to shared resource (k8s API server) and performance of the whole system limited to k8s API server performance.
Second, it’s highly inefficient. Inefficient in the same way like finding a record in the database by reading a whole database file and iterating through to find a record.
Current @haiodo tests show that finding NSM via registry takes 1-1.5 second when there are 6000 NSEs registered.
Any distribution of workload in the system would not let the system to work faster than nsmd-k8s -> k8s API server interaction. The obvious solution to make this interaction faster is client-server (application server), which is CNSMD.
So call 3a in current architecture which takes seconds would be replaced with call 1a, which is essentially single hash table lookup on CNSMD side.
I’d like to say that registry is a bottleneck here, and replacing registry with any other database (file) would not improve the situation much. If we would replace it with own registry server (which is a better registry with fast lookups) — a kind of half-cooked CNSMD, it would render current NSMDs obsolete, since they will be simple proxying calls back and forth and it’s better to move all NSMD logic to CNSMD.
I believe current NSM performance with k8s registry is OK for all possible deployment scenarios. My point here is not about milliseconds, but I’d like to argue that nothing can beat CNSMD in terms of scalability and reliability as well as simplicity. So the question is why we’re making the system complex and semi-distributed without scalability benefits?

Also I’d like to add, that service mesh approach is very good in terms of proxing of traffic as Envoy or Istio do. But our NSC only creates single Request() to NSMD, gets interfaces injected and uses them to send traffic. So we have no advantages of this nsmd-per-node architecture.

But since we are going to implement nodes interaction in different clusters it would suit our purposes better to use nsmd-ped-cluster approach.

@platoff Good breakdown.  I agree, 3a (Local nsmd-k8s performs 1..N calls to k8s master) is a bottleneck.
I'd suggest we look at dealing with this in two steps:

Caching - K8s API Machinery already has easy to use caching.  And the ability to have event handlers called on updates.  A preliminary look suggest we could use an IndexInformer to respond to updates from the K8s API Server to compute (and store for retrieval) the FindNetworkServiceResponse for all network_service_names.  This makes the lookup in nsmd-k8s basically as fast as an in memory map lookup.
Make nsmd's ${hostname}:${port} configurable for the nsmd-k8s.  As you've no doubt noticed, eventually as the number of Nodes grows, the notification of nsmd-k8s per Node can become a bottleneck.  For small to medium sized clusters, collocating with the nsmd in the same Pod still probably makes sense.  For large clusters, you want to be able to deploy some number of replicas less than one per Node for nsmd-k8s and put them behind a K8s Service.  For this to work, the nsmd's ${hostname}:${port} for the nsmd-k8s has to be configurable.

#1 should fix the scalability issue you pointed out in 3a.  #2 should allow us to handle the new bottleneck of notifications to nsmd-k8s (when it becomes an issue).


Error while doing import for a redhat base image

I created a base image tar file from one of our redhat machines.
I than took that tar to another redhat machine on which i have docker installation and i ran import but it fails.
I took the import command from the docker docs:
cat ./rhel64.tar | sudo docker import - exampleimagelocal:new
and the error i'm getting:
FATA[0051] Error: Driver devicemapper failed to create image rootfs 9d40582502573f5f4e34475fe6d0ae39cb4442f6ad493d7130161698dbc5bcf8: Error running DeviceCreate (createSnapDevice) dm_task_run failed
Docker info:
Containers: 0
Images: 0
Storage Driver: devicemapper
Pool Name: docker-8:2-920680-pool
Pool Blocksize: 65.54 kB
Data file: /var/lib/docker/devicemapper/devicemapper/data
Metadata file: /var/lib/docker/devicemapper/devicemapper/metadata
Data Space Used: 0 B
Data Space Total: 107.4 GB
Metadata Space Used: 577.5 kB
Metadata Space Total: 2.147 GB
Library Version: 1.02.89-RHEL6 (2014-09-01)
Execution Driver: native-0.2
Kernel Version: 2.6.32-504.el6.x86_64
Operating System:
CPUs: 2
Total Memory: 3.743 GiB

sorry, seems I had an issue with my installation of docker and it got corrupted.
I did uninstall, removed the entire /var/lib/docker and install again and it worked.

No problem, thanks for reporting back and closing; much appreciated!


QDK fixes for QIR targets

This change implements a few fixes that allow submission to QIR targets.

/azp run


Migrate project maintainer data over to cncf/people from maintainers.csv

Overview
Lift and shift maintainer data so that it can be used by cncf/sheriff to control access to repos in the CNCF GitHub Org.
Benefits: Maintainer Data will have a single source of truth for use in CNCF Process Automation.
Data
Source
https://github.com/cncf/foundation/blob/main/project-maintainers.csv
Destination
https://github.com/cncf/people/blob/main/people.json
and
https://github.com/cncf/people/blob/main/config.yaml
Approaches
Multiple ways we could do this. Here are two suggestions. Open to hearing others.
Big bang
Write a script to process the full project-maintainer.csv file so it updates the destination files in one go (we would have to merge the data so that there were now repeats) this would generate a single PR for cncf/people
Might be hard to review (will have to gauge the overlap between both datasets)
Create a PR for each Project in maintainers.csv
Write a script to create a per-Project/Team PR. This would allow Projects and the CNCF to review the changes on a per project basis offering the opportunity to review team membership as we work through those PRs.

With 147 projects (and 3 in voting) - can we do both?

If we have already tooling in place to manage tooling access, it needs to be in one-go. Otherwise it'll diff against the existing entries in places like ServiceDesk and y'know. Delete them all.
Prefer we coordinate doing one big PR and then scrub references to old maintainers CSV(s).
The tooling work should happen in parallel to (or ideally before) this.

We at Hyperledger are super interested in adopting this workflow

Met with @jeefy yesterday (March 9th) to design out how to integrate CSV maintainer data into cncf/people.json
We went over three possible iterations and settled on the following for a initial pass at this.
Add the following maintainer object to the person record in cncf/people.json
Maintainer with voting rights looks like this
     "maintainer" : {
       "Kube-virt" : "true",
    }

-OR-
Maintainer with no voting rights looks like this
     "maintainer" : {
       "Kubernetes" : {
         "voting-rights": "false",
       },
      }

Notes:

Maintainers have voting rights unless explicitly excluded in the current project-maintainers.csv (Embedded in one of the field values) captures this as above for
Maintainer email addresses are stored in a private file that should be used a data source to produce the list of voters to facilitate voting procedures.
In addition to point 2. maintainer-based reports should be considered for @amye (chime in during review of this work!)
The existing Pull Request review process of the cncf/people.json that allows for human oversight of changes to the file is a key process requirement here and  MUST remain in place as-is.
In cncf/people the Category field is like a Role
We talked about teams and the current absence explicit  Tool/Resource modelling in cncf/people which we will revisit on future tasks.

Development notes

I went with Big Bang approach,
There were maintainers in the CSV file that were not present in cncf/people so I add them and log that as a creation event
I added a source field to indicate that the record came from the migration script,
I will report on new and updated voters and non-voters.


Feature Request: Pixel locking between different image displays in the same notebook

A companion issue for #62 but for a simpler case of pixel locking without worrying about WCS. This would assume that Image A and Image B have the same pointing and rotation. This would be useful to compare, say, a science scene and its residual image, or a science scene and its uncertainties, etc.
I thought I got it figured out but turns out I was wrong. @mwcraig suggested using ipywidgets.link on zoom and pan traitlets tied to ImageWidget but I couldn't get that idea to work, given my inexperience with widgets and traitlets.
xref spacetelescope/dat_pyinthesky#7

p.s. In the future, we also want to link 3 different displays in the same notebook, so the solution needs to be scalable. Actual use case that triggered this was a PSF photometry workflow where user wanted to link the original science scene, its residual after PSF subtraction, and the associated uncertainties.


Enable create incidents by webform without "Resources Affected"

Is your feature request related to a problem? Please describe.
Can't create incidents without Monitor associated using webform
Describe the solution you'd like
Remove required field notifications for "monitor affected" in the incident creation using webform, because by API its possible create in this way.
Additional context

This is fixed and will be released by EOD today.

Confirmed in our side that now its working as expected. Thanks so much for the quick implementation


Fix Codacy coverage report

Linux builds are failing due to some Codacy coverage reporting issues.

Seems like it's working on master. But then why did it fail for https://github.com/ftsrg/theta/pull/137? Is it because it was a PR from a separate fork?

This is due to a separate fork I think. It seems logical, because 1) anyone can fork a public repo 2) Actions (and secrets) could be using external (paid) resources.
It is the same with @LendvaiDenes 's PR. (I was talking to him, hence the related PR).
Main issue, I think, is that forks do not get the codacy secrets of the main project.
(I'll reopen the issue, I do not fully understand what it means that it is closed. I think this issue is missing a conclusion. Is this a wontfix, or it is solved as the main project (i.e. not a fork) does not have the issue? )

Yeah, I thought that this is a wontfix, but I agree that it would be nicer to fix so that people have meaningful build results on forks. I know that we can skip certain steps based on branch names, is it also possible to do based on forks? https://github.community/t/how-to-detect-a-pull-request-from-a-fork/18363 seems to give some suggestions