Validation for Group ownership is broken

There are recently added undocumented validation rules for Group ownership in Microsoft Graph which are broken. Discussions with the service team would indicate that this validation is intended to affect Microsoft 365 (unified) groups with the goal of preventing group creation without a group having an owner that is a user principal. However at the same time we are seeing errors around group ownership for security groups too and the behavior is inconsistent, sometimes preventing creation of a group no matter what combination of owners are specified.
At group creation time, this seems to manifest in a 400 response with the following error description:
HTTP/1.1 400 Bad Request

Date: Thu, 29 Jul 2021 10:20:37 GMT
Content-Type: application/json
Cache-Control: no-cache
Transfer-Encoding: chunked
Content-Encoding: gzip
Vary: Accept-Encoding
Strict-Transport-Security: max-age=31536000
request-id: fe3ae0a7-46d6-4a29-9c23-9ed2c4ad0b0a
client-request-id: fe3ae0a7-46d6-4a29-9c23-9ed2c4ad0b0a
x-ms-ags-diagnostic: {"ServerInfo":{"DataCenter":"West Europe","Slice":"E","Ring":"5","ScaleUnit":"003","RoleInstance":"AM1PEPF000051D7"}}
Link: <https://developer.microsoft-tst.com/en-us/graph/changes?$filterby=beta,PrivatePreview:Restricted_AU_Properties&from=2021-04-01&to=2021-05-01>;rel="deprecation";type="text/html"
Deprecation:
Sunset:
x-ms-resource-unit: 1

{
    "error": {
        "code": "Request_BadRequest",
        "innerError": {
            "client-request-id": "fe3ae0a7-46d6-4a29-9c23-9ed2c4ad0b0a",
            "date": "2021-07-29T10:20:38",
            "request-id": "fe3ae0a7-46d6-4a29-9c23-9ed2c4ad0b0a"
        },
        "message": "Request contains a property with duplicate values."
    }
}

This does not correlate with the request payload which might for example look like the following:
POST https://graph.microsoft.com/beta/659c0000-0000-0000-0000-00000000a82e/groups
HTTP/1.1

Host: graph.microsoft.com
User-Agent: Hamilton (Go-http-client/1.1) HashiCorp Terraform/1.0.0 (+https://www.terraform.io) Terraform Plugin SDK/2.6.1 terraform-provider-azuread/1.6.0 pid-222c6c49-1b0a-5959-a213-6608f9eb8820
Content-Length: 233
Accept: application/json
Authorization: Bearer ...
Content-Type: application/json; charset=utf-8
Accept-Encoding: gzip

{
    "displayName": "foobar2",
    "mailEnabled": false,
    "mailNickname": "6fb69a33-5961-f0a4-6041-7af7712c0d06",
<EMAIL_ADDRESS>[
        "https://graph.microsoft.com/beta/directoryObjects/06ee5fa0-7a61-4313-bae7-4d1bf50cbb03"
    ],
    "securityEnabled": true
}

This error is returned despite:

The group being created not being a unified group but is a "traditional" AAD security group
The group has a valid owner specified
There are no duplicate fields or values, for owners or any other property

In addition, when removing owners from a group, where there are two owners and one of these is a service principal and the other is a user principal, and the group is a security group and not a unified group, a 400 response can sometimes be received with the following error:
The group must have at least one owner, hence this owner cannot be removed

These behaviors have been documented by ourselves and several community members, in the following related issues:
https://github.com/hashicorp/terraform-provider-azuread/issues/464
https://github.com/hashicorp/terraform-provider-azuread/issues/473
https://github.com/hashicorp/terraform-provider-azuread/issues/478
https://github.com/hashicorp/terraform-provider-azuread/issues/567

Hi,
I was experiencing pretty much the same issue too.
As per the documentation here :  https://registry.terraform.io/providers/hashicorp/azuread/latest/docs/resources/group

API Permissions
The following API permissions are required in order to use this resource.
When authenticated with a service principal, this resource requires one of the following application roles: Group.ReadWrite.All or Directory.ReadWrite.All
When authenticated with a user principal, this resource requires one of the following directory roles: Groups Administrator, User Administrator or Global Administrator

It worked for me after having granted Group Administrator role.
Hope this helps.

Thanks for the tip @otabut. However, in the enterprise I am working at, it is impossible for ordinary mortals to acquire such elevated rights.
My workaround for daily use in Terraform is to create the group via Azure CLI and import it into the state for further management via Terraform, e.g. like this:
AD_APP_DISPLAY_NAME='foo.bar'
AD_APP_TF_STATE_ADDR='module.foo.azuread_group.bar'

az ad group create --display-name "${AD_APP_DISPLAY_NAME}" --mail-nickname "$(cat /proc/sys/kernel/random/uuid)"

terraform import "${AD_APP_TF_STATE_ADDR}" "$(az ad group show --group "${AD_APP_DISPLAY_NAME}" --query 'objectId' --output 'tsv')"

Any update/progress on resolving this issue? As getting elevated permission is not an option for us and would be forced to do @ju-la-berger 's solution above which kind of defeats the point of using terraform...

Hi Team, any update its known issue for quite long time :)

Agreed - I can confirm this is still an issue when using Azure AD API to remove user as owner when a service principal is the rightful owner of the group - but is not being recognized as such - therefore preventing the removal of a user as owner.  Please advise on the timelines for a fix on this issue.  Thank you.

Any updates here? Been over a year.

please fix this!

I agree, I hope this ticket can get some prioritie.
This issue cause delay and automation challenges in our delivery of Azure Landingzones to Microsoft Customers. Expensive delay for everyone.

I would also like to point out that the current API is not idempotent.
Without using the above workaround (i.e. no using separate service principle creating the resource, but instead using my identity to create the resource):
Calling the API and setting the "owner" only as "Identity2" (not including my identity).  The Group resource has both my identity and "Identity2" in the "Owners" field.
Applying the same change using the same parameters, results in my identity in the "owners" field being removed (breaking idempotency).
I can't create a service principle for this (too many hoops to jump through).  Has the workaround been tested for idempotency?

Any update on this issue or #653 @darrelmiller @baywet ?

Hi everyone,
Apologies for the late reply here: we (the SDKs & tooling team) are not watching/triaging this repo. Only using it to capture metadata defects to improve SDKs generation.
The service team is not watching this repository either. The best way to get their attention for this kind of issues is to open a support ticket on a production tenant (see the links at the bottom of the page).
Don't assume you do not need to open a ticket because somebody else on this thread might already have opened one.
I'm going to go ahead and close this issue to make it clear no resolution will come from here directly.

Hi everyone, Apologies for the late reply here: we (the SDKs & tooling team) are not watching/triaging this repo. Only using it to capture metadata defects to improve SDKs generation. The service team is not watching this repository either. The best way to get their attention for this kind of issues is to open a support ticket on a production tenant (see the links at the bottom of the page). Don't assume you do not need to open a ticket because somebody else on this thread might already have opened one. I'm going to go ahead and close this issue to make it clear no resolution will come from here directly.

Thanks for the clarification!


LFS-655: Handle re-login globally

This PR is in draft mode as the standardized re-login procedure has not been added to all fetch() calls requiring authentication. Nonetheless, any feedback on the changed functionalities thus far would be greatly appreciated. These changed functionalities are:

User dashboard

Subjects page

Forms page

Any individual subject page (eg. a patient chart)

Any individual form (eg. a Demographics Data form)

The Delete button (eg. delete a form), has some minor issues handling incorrect logins when deleting a form when the form is open.

To test, login to LFS in two different browser tabs. On one browser tab logout. On the other tab, try the various above listed functionalities (eg. visit the User dashboard, visit the Subjects page, etc...)

Not sure if this is a problem. I loged out in one tab, in the other one I tried to add a filter

Also I noticed in console the following error happening quite often:

I had LFS opened in 2 tabs. On one tab I had Dashboard , on the other one I was editing Radiation form. On the tab with Dashboard I logged out, after which I was able to continue with Radiotherapy form edits. Once I clicked on SAVE button, the form seems to get frozen.

Then I logged in the first tab (Dashboard), when I refreshed the other tab with Radio form I was prompted to logi in again. Is this expected?

no complaints from me either


Explain how to use panels

Heya! Thanks for the PR. Using Panels in NovaSettings is no different from using Panels in any Resource. Do you think it should be documented anyway? :) I thought it'd be obvious and I'd like to keep the documentation as short as possible.

Hey there @Tarpsvo I think it may be obvious to those who have used them in Nova core before but not maybe to those who are new/newer to Nova. I spent probably 2 hours trying to figure out how you did it. I dove your source and everything and could not figure it out. I finally found a StackOverflow on how to do it and figured it would be nice to add to the docs.

Alright, thanks! I added it just in case. :) Thank you!

Awesome! Thank you!


Incompatible with vite-plugin-pages

Without the plugin in main.js, vite will throw error can not import virtual:generated-pages, though when I do add the plugin, every request will be redirected to /virtual:xxx. which wouldn't work since it doesn't exist.

I have a similar issue. My package.json contains:
  "dependencies": {
    "vue": "^3.1.4",
    "vue-router": "^4.0.10"
  },
  "devDependencies": {
    "@storybook/addon-actions": "^6.3.2",
    "@storybook/addon-essentials": "^6.3.2",
    "@storybook/addon-links": "^6.3.2",
    "@storybook/storybook-deployer": "^2.8.10",
    "@storybook/vue3": "^6.3.2",
    "@vitejs/plugin-vue": "^1.2.5",
    "@vue/compiler-sfc": "^3.1.4",
    "autoprefixer": "^10.3.1",
    "markdown-it-prism": "^2.1.8",
    "postcss": "^8.3.5",
    "sass": "^1.35.1",
    "storybook-builder-vite": "^0.0.12",
    "storybook-dark-mode": "^1.0.8",
    "tailwindcss": "^2.2.4",
    "typescript": "^4.3.5",
    "vite": "^2.4.2",
    "vite-plugin-md": "^0.6.7",
    "vite-plugin-pages": "^0.15.0"
  },

The plugin vite-plugin-pages requires import routes from "virtual:generated-pages".
But yarn run start-storybook has the following error:
yarn run v1.21.0
$ /home/a/j3/tealweb/node_modules/.bin/start-storybook
info @storybook/vue3 v6.3.4
info
info => Loading presets

info => Using cached manager
ERR! Error: The following dependencies are imported but could not be resolved:
ERR!
ERR!   virtual:generated-pages (imported by /home/a/j3/tealweb/src/router.ts)
ERR!
ERR! Are they installed?
ERR!     at optimizeDeps (/home/a/j3/tealweb/node_modules/vite/dist/node/chunks/dep-11db14da.js:73026:15)
ERR!     at processTicksAndRejections (node:internal/process/task_queues:96:5)
ERR!     at async runOptimize (/home/a/j3/tealweb/node_modules/vite/dist/node/chunks/dep-11db14da.js:73684:48)
ERR!     at async createServer (/home/a/j3/tealweb/node_modules/vite/dist/node/chunks/dep-11db14da.js:73713:9)
ERR!     at async Object.start (/home/a/j3/tealweb/node_modules/storybook-builder-vite/index.js:33:20)
ERR!     at async Promise.all (index 0)
ERR!     at async storybookDevServer (/home/a/j3/tealweb/node_modules/@storybook/core-server/dist/cjs/dev-server.js:123:28)
ERR!     at async buildDevStandalone (/home/a/j3/tealweb/node_modules/@storybook/core-server/dist/cjs/build-dev.js:112:31)
ERR!     at async buildDev (/home/a/j3/tealweb/node_modules/@storybook/core-server/dist/cjs/build-dev.js:154:5)
ERR!  Error: The following dependencies are imported but could not be resolved:
ERR!
ERR!   virtual:generated-pages (imported by /home/a/j3/tealweb/src/router.ts)
ERR!
ERR! Are they installed?
ERR!     at optimizeDeps (/home/a/j3/tealweb/node_modules/vite/dist/node/chunks/dep-11db14da.js:73026:15)
ERR!     at processTicksAndRejections (node:internal/process/task_queues:96:5)
ERR!     at async runOptimize (/home/a/j3/tealweb/node_modules/vite/dist/node/chunks/dep-11db14da.js:73684:48)
ERR!     at async createServer (/home/a/j3/tealweb/node_modules/vite/dist/node/chunks/dep-11db14da.js:73713:9)
ERR!     at async Object.start (/home/a/j3/tealweb/node_modules/storybook-builder-vite/index.js:33:20)
ERR!     at async Promise.all (index 0)
ERR!     at async storybookDevServer (/home/a/j3/tealweb/node_modules/@storybook/core-server/dist/cjs/dev-server.js:123:28)
ERR!     at async buildDevStandalone (/home/a/j3/tealweb/node_modules/@storybook/core-server/dist/cjs/build-dev.js:112:31)
ERR!     at async buildDev (/home/a/j3/tealweb/node_modules/@storybook/core-server/dist/cjs/build-dev.js:154:5)

WARN Broken build, fix the error above.
WARN You may need to refresh the browser.

error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
Command failed: node /home/a/.yvm/versions/v1.22.10/bin/yarn.js run start-storybook

I have a similar issue. My package.json contains:
  "dependencies": {
    "vue": "^3.1.4",
    "vue-router": "^4.0.10"
  },
  "devDependencies": {
    "@storybook/addon-actions": "^6.3.2",
    "@storybook/addon-essentials": "^6.3.2",
    "@storybook/addon-links": "^6.3.2",
    "@storybook/storybook-deployer": "^2.8.10",
    "@storybook/vue3": "^6.3.2",
    "@vitejs/plugin-vue": "^1.2.5",
    "@vue/compiler-sfc": "^3.1.4",
    "autoprefixer": "^10.3.1",
    "markdown-it-prism": "^2.1.8",
    "postcss": "^8.3.5",
    "sass": "^1.35.1",
    "storybook-builder-vite": "^0.0.12",
    "storybook-dark-mode": "^1.0.8",
    "tailwindcss": "^2.2.4",
    "typescript": "^4.3.5",
    "vite": "^2.4.2",
    "vite-plugin-md": "^0.6.7",
    "vite-plugin-pages": "^0.15.0"
  },

The plugin vite-plugin-pages requires import routes from "virtual:generated-pages".
But yarn run start-storybook has the following error:
yarn run v1.21.0
$ /home/a/j3/tealweb/node_modules/.bin/start-storybook
info @storybook/vue3 v6.3.4
info
info => Loading presets

info => Using cached manager
ERR! Error: The following dependencies are imported but could not be resolved:
ERR!
ERR!   virtual:generated-pages (imported by /home/a/j3/tealweb/src/router.ts)
ERR!
ERR! Are they installed?
ERR!     at optimizeDeps (/home/a/j3/tealweb/node_modules/vite/dist/node/chunks/dep-11db14da.js:73026:15)
ERR!     at processTicksAndRejections (node:internal/process/task_queues:96:5)
ERR!     at async runOptimize (/home/a/j3/tealweb/node_modules/vite/dist/node/chunks/dep-11db14da.js:73684:48)
ERR!     at async createServer (/home/a/j3/tealweb/node_modules/vite/dist/node/chunks/dep-11db14da.js:73713:9)
ERR!     at async Object.start (/home/a/j3/tealweb/node_modules/storybook-builder-vite/index.js:33:20)
ERR!     at async Promise.all (index 0)
ERR!     at async storybookDevServer (/home/a/j3/tealweb/node_modules/@storybook/core-server/dist/cjs/dev-server.js:123:28)
ERR!     at async buildDevStandalone (/home/a/j3/tealweb/node_modules/@storybook/core-server/dist/cjs/build-dev.js:112:31)
ERR!     at async buildDev (/home/a/j3/tealweb/node_modules/@storybook/core-server/dist/cjs/build-dev.js:154:5)
ERR!  Error: The following dependencies are imported but could not be resolved:
ERR!
ERR!   virtual:generated-pages (imported by /home/a/j3/tealweb/src/router.ts)
ERR!
ERR! Are they installed?
ERR!     at optimizeDeps (/home/a/j3/tealweb/node_modules/vite/dist/node/chunks/dep-11db14da.js:73026:15)
ERR!     at processTicksAndRejections (node:internal/process/task_queues:96:5)
ERR!     at async runOptimize (/home/a/j3/tealweb/node_modules/vite/dist/node/chunks/dep-11db14da.js:73684:48)
ERR!     at async createServer (/home/a/j3/tealweb/node_modules/vite/dist/node/chunks/dep-11db14da.js:73713:9)
ERR!     at async Object.start (/home/a/j3/tealweb/node_modules/storybook-builder-vite/index.js:33:20)
ERR!     at async Promise.all (index 0)
ERR!     at async storybookDevServer (/home/a/j3/tealweb/node_modules/@storybook/core-server/dist/cjs/dev-server.js:123:28)
ERR!     at async buildDevStandalone (/home/a/j3/tealweb/node_modules/@storybook/core-server/dist/cjs/build-dev.js:112:31)
ERR!     at async buildDev (/home/a/j3/tealweb/node_modules/@storybook/core-server/dist/cjs/build-dev.js:154:5)

WARN Broken build, fix the error above.
WARN You may need to refresh the browser.

error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
Command failed: node /home/a/.yvm/versions/v1.22.10/bin/yarn.js run start-storybook

Yeah, so the thing is storybook uses a dedicated vite config instead of your project vite config, so you might have to copy-paste relevant code.

Thank you @viztor for your response.
I have carefully read the documentation https://storybook.js.org/docs/vue/configure
but my attempts have failed…
My file .storybook/main.js:
module.exports = {
  "stories": [
    "../src/**/*.stories.mdx",
    "../src/**/*.stories.@(js|jsx|ts|tsx)"
  ],
  "addons": [
    "@storybook/addon-links",
    "@storybook/addon-essentials",
    'storybook-dark-mode'
  ],
  "core": {
    "builder": "storybook-builder-vite",
  },
  async viteFinal(config, { configType }) {
    config.resolve.alias = [
      { find: '@/', replacement: '/src/' },
      { find: 'vue', replacement: 'vue/dist/vue.esm-bundler.js' }
    ]
    return config;
  },
}

This previous main.js is fine for the command yarn run build-storybook.
But yarn run start-storybook fails.
When I add const custom = require('../vite.config.js') in the .storybook/main.js,
I got this error:
ERR! /home/a/j3/tealweb/vite.config.js:1
ERR! import path from "path"
ERR! ^^^^^^
ERR!
ERR! SyntaxError: Cannot use import statement outside a module
[...]

Do you know how to copy the relevant part of my Vite config to the Storybook config ?
My vite.config.js:
import path from "path"
import { defineConfig } from "vite"
import markdown from "vite-plugin-md"
import pages from "vite-plugin-pages"
import vue from "@vitejs/plugin-vue"
import prism from "markdown-it-prism"

export default defineConfig({
  plugins: [
    markdown({
      markdownItOptions: {
        html: true,
        linkify: true,
        typographer: true,
      },
      markdownItSetup(md) {
        md.use(prism)
      },
      wrapperClasses: "prose prose-sm m-auto text-left",
    }),
    pages({
      extensions: ["vue", "md"],
    }),
    vue({
      include: [/\.vue$/, /\.md$/],
    }),
  ],
  resolve: {
    alias: { "@/": `${path.resolve(__dirname, "src")}/` },
  },
})

@micheartin
just do
config.plugins.push({
     ...your plugins (exclude the vue plugin since it's already included)
})

Thank you @viztor
Simplified version of my .storybook/main.js:
const pagesPlugin = require("vite-plugin-pages")         // 🠜 add this

module.exports = {
  // typescript: ...
  // stories: ...
  // addons: ...
  // core: ...
  async viteFinal(config, { configType }) {
    config.resolve.alias = [
      { find: '@/', replacement: '/src/' },
      { find: 'vue', replacement: 'vue/dist/vue.esm-bundler.js' }
    ]
    config.plugins.push(                                 // 🠜
      pagesPlugin.default({ extensions: ["vue", "md"] }) // 🠜 add that
    )                                                    // 🠜
  },
}

But yarn run start-storybook produces the following Vite errors: 😕
10:19:32 PM [vite] Internal server error: Cannot read property 'get' of undefined
  Plugin: vite:css-post
  File: /home/a/j3/tealweb/src/assets/index.css
      at TransformContext.transform (/home/a/j3/tealweb/node_modules/vite/dist/node/chunks/dep-11db14da.js:28172:56)
      at Object.transform (/home/a/j3/tealweb/node_modules/vite/dist/node/chunks/dep-11db14da.js:50885:53)
      at async transformRequest (/home/a/j3/tealweb/node_modules/vite/dist/node/chunks/dep-11db14da.js:66636:29)
      at async viteTransformMiddleware (/home/a/j3/tealweb/node_modules/vite/dist/node/chunks/dep-11db14da.js:66774:32)
10:19:35 PM [vite] Internal server error: Cannot read property 'get' of undefined
  Plugin: vite:css-post
  File: /home/a/j3/tealweb/src/App.vue?vue&type=style&index=0&lang.sass
      at TransformContext.transform (/home/a/j3/tealweb/node_modules/vite/dist/node/chunks/dep-11db14da.js:28172:56)
      at Object.transform (/home/a/j3/tealweb/node_modules/vite/dist/node/chunks/dep-11db14da.js:50885:53)
      at async transformRequest (/home/a/j3/tealweb/node_modules/vite/dist/node/chunks/dep-11db14da.js:66636:29)
      at async viteTransformMiddleware (/home/a/j3/tealweb/node_modules/vite/dist/node/chunks/dep-11db14da.js:66774:32)

My temporary workaround: remove vite-plugin-pages and set all routes by hand. 🤷♂
I will try again in a couple of months… :wink:


Glyph Integration

I see Glyph is here (https://github.com/gogcom/galaxy-integrations-python-api/blob/master/PLATFORM_IDs.md) but I can't find any plugin. Is there one?

You may report integration request on community repo https://github.com/Mixaill/awesome-gog-galaxy


Rufused To Connect

After successful deploy to Azure, the app is refusing to connect whenever I searched the web using the given name in the dns. What could be the possible problem?

What do you mean by refusing to connect? what do you see in your browser.

This is what I see in the browser… Sent from Mail for Windows From: Nicolas DorierSent: Tuesday, May 17, 2022 9:58 AMTo: btcpayserver/btcpayserver-azureCc: estar-app; AuthorSubject: Re: [btcpayserver/btcpayserver-azure] Rufused To Connect (Issue #5) What do you mean by refusing to connect? what do you see in your browser.—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you authored the thread.Message ID<EMAIL_ADDRESS>
For anyone else seeing this, the site took a good 30 minutes to become available after the initial deploy. I was seeing the same "Refused to connect" error before.

same problem, I followed the one click install for azure. every resource seems healthy but when using ip address or dns address or dnsadress/server/maintenance I see the same in the browser:
ERR_CONNECTION_REFUSED

Has anyone resolved this? I'm facing the same issue after the one click azure deployment. Browser error, "the site can't be reached"

@timgriffeth2024 I have a reply earlier in this thread but I think you just need to wait a bit longer. For me it was 30 minutes.

@timgriffeth2024 I have a reply earlier in this thread but I think you just need to wait a bit longer. For me it was 30 minutes.

Thanks for replying. Waited overnight, and still the same error.

Ah okay, yeah must be something else. I just saw someone else said their issue wasn't due to time either, hopefully they will respond. Sorry!

Ah okay, yeah must be something else. I just saw someone else said their issue wasn't due to time either, hopefully they will respond. Sorry!

Are you still running it? If so, I'm curious on how it's currently set up.

Not sure what the deal was with the VM that was created from the template. I created my own VM, installed BTCPay Server manually and my server is running successfully.


Issue with react useCallback and other functions which are typed to return the function given.

Hey there! It seems like this plugin is getting confused by react's useCallback function, and other memoization funcs.
if the function you're passing accepts no arguments, the name displayed appears like this:

When the function accepts an argument, the name takes on whatever name that argument is.

In both of these cases, the name given to this parameter should be callback

Duplicate of #10

Ah totally missed that one. I'll close this one then.


Use CurrentAttributes to store and populate log context

What problem does this pull request solve?
Trello card: https://trello.com/c/tjnJrmjR/7-improve-default-information-we-add-to-logs
We already use Lograge to add attributes about the request context, such as the request_id, form_id etc to the request logs for incoming requests to the app.
For other application logs, we use a class called EventLogger to log events as JSON, and pass around a logging_context object that we store on the ApplicationController and add to in various places throughout a request to add useful JSON fields to the log lines.
Simplify this approach by using an implementation of the ActiveSupport::CurrentAttributes abstract super class, which provides a thread-isolated attributes singleton, which resets automatically before and after each request. This implementation - CurrentLoggingAttributes stores the attributes we expect to be present on most requests.
Introduces a custom logger class - ApplicationLogger to use for application logs. This makes use of the CurrentAttributes implementation to extract common logging fields and appends them to the JSON object for a log event.
This way, we do not have to pass around a logging_context between different methods, which can be a bit confusing - and if we forget to do this or make a mistake with how we do it, it will cause us not to log the request context attributes.
This also means that Rails default logs, such as for outgoing requests to other apps will have all the requests context attributes.
We can still pass in log-line specific additional attributes as a hash when we log and these will be added to the JSON object that we log.
Notes
This PR attempts to keep as much of the logging context the same as possible. In most cases, it just replaces adding to the logging_context hash we pass around by adding things to the CurrentLoggingAttributes, so attributes will appear on the same lines as before.
Things to consider when reviewing

Ensure that you consider the wider context.
Does it work when run on your machine?
Is it clear what the code is doing?
Do the commit messages explain why the changes were made?
Are there all the unit tests needed?
Has all relevant documentation been updated?

This PR attempts to keep as much of the logging context the same as possible. In most cases, it just replaces adding to the logging_context hash we pass around by adding things to the CurrentLoggingAttributes, so attributes will appear on the same lines as before.

To help with the review please can you call out any changes to this, otherwise I feel subtle changes might be easily missed from reviewing the code alone. As mentioned Anne uses the events in her reporting so please confirm whether anything relating to that changes.

This PR attempts to keep as much of the logging context the same as possible. In most cases, it just replaces adding to the logging_context hash we pass around by adding things to the CurrentLoggingAttributes, so attributes will appear on the same lines as before.

To help with the review please can you call out any changes to this, otherwise I feel subtle changes might be easily missed from reviewing the code alone. As mentioned Anne uses the events in her reporting so please confirm whether anything relating to that changes.

Good point! The only thing that's intentionally changed here is that the event logs now include a message field which is just message: "Form event" for consistency.
Other than that, I did make sure that everything stays the same. I've done my best to test this both with automated tests and pushing up to dev and testing it there, so I have pretty good confidence in this.


fix(sec): upgrade org.apache.hive:hive-exec to 3.1.3

What happened？
There are 1 security vulnerabilities found in org.apache.hive:hive-exec 2.3.5

CVE-2021-34538

What did I do？
Upgrade org.apache.hive:hive-exec from 2.3.5 to 3.1.3 for vulnerability fix
What did you expect to happen？
Ideally, no insecure libs should be used.
The specification of the pull request
PR Specification from OSCS
<EMAIL_ADDRESS>

fixed in #16046


Feature request: Autorun set of command when atom opens

When I open atom, it also open my last opened project. Before I start developing, I always run 3 commands (i.e. jekyll serve, guard, gulp) in seperate terminal instance. I would be nice if the terminal just executed those commands automatically simultaneously with Atom.

You could create your own file that will run jekyll/guard/gulp and then launch atom, then set a sym link to that file.
This way you could just run atom or atom .   and the system would run those commands as well.


FlowRouter.subscriptions not working if there are 7 subscriptions inside

FlowRouter.subscriptions = ->
  @register 'myUserData', allSubs.subscribe('userData')
  @register 'fActiveGroup', allSubs.subscribe('activeGroup')
  @register 'myuserPhotos', allSubs.subscribe('userPhotos')
  @register 'fUserManagerList', allSubs.subscribe('userManagerList')
  @register 'fuserPhotoManager', allSubs.subscribe('userPhotoManager')
  @register 'fGroupNameList', allSubs.subscribe('groupNameList')
  @register 'fGroupManager', allSubs.subscribe('groupManager')
  return

Don't know why but It will remove the first subscription (myUserData)

I tried to remove any one subscription, then it works fine with 6 remaining subscriptions

@arunoda

May be one of your subscription is getting failed. Could you work on a sample repo for this.

This may be since one of your subscription is not getting completed. So, I'm closing this issue since I don't have any way to check this.


[SPARK-15245][SQL] Stream API throws an exception for non-directory path with incorrect message.

What changes were proposed in this pull request?
https://issues.apache.org/jira/browse/SPARK-15245
When stream() takes a non-direcotry path, this throws an exception during execution with a message Option 'basePath' must be a directory.
In DataSource, this can be checked before. This PR prevents to give a non-directory path for stream().
How was this patch tested?
Unittest in FileStreamSourceSuite and ./build/sbt scalastyle

Test build #58209 has started for PR 13021 at commit bd8ff3e.

I am closing this after talking with @srowen in the JIRA ticket.

Test build #58209 has finished for PR 13021 at commit bd8ff3e.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/58209/
Test PASSed.


Onboarding to running experiments

Most options in the sidebar are about viewing experiments, which doesn't make much sense for a new project without any experiments. It would be good to focus on running experiments as the starting point.
@shcheklein Made the comparison to launch.json, and I think we can borrow a lot from VS Code's debugging setup. When you first try to run an experiment, we can try to funnel users through a workflow like:

Pick the file to run the experiment. If it's not Python, what do we do?
Once the file is picked, we can show the instructions to add DVCLive, and maybe we can even validate that DVCLive has been added to the code before allowing users to continue.
Once DVCLive is added, users can choose whether to configure a pipeline (similar to the option to create launch.json).
If they choose not to generate a pipeline, any actions to run experiments should execute the file (can it instead run the active cell in a notebook when appropriate?).
If they choose to generate a pipeline, we know the source code file. If we can determine the DVCLive output directory, we could pre-fill dvc.yaml with something like:

stages:
  train:
    cmd: python source_file.py
    deps:
      - source_file.py
    outs:
      - dvclive:
          cache: false

Summary of the discussion during the product sync with @dberenbaum :

Next step for the onboarding is to focus on "launch configurations" (helping creating a dvc.yaml e.g. when user clicks on a run button)
We can show a + Pipeline config on the experiments side panel
On the DVCLive snippets screen we don't show anything extra yet. Eventually we can show alternatives for other languages.

[x] Add a quick pick to prompt the user for their training script to add to the dvc.yaml file being created
[x] Remove beaker button to show experiments and replace with the play button to run experiment (like it is currently is when showing the table)
[x] If there is a dvc.yaml file, also check for a pipeline in the file (check for stages key or run dvc stage list)
[x] Add a mention in the last onboarding step ("Your project contains no data") to tell the user to run their training script after adding DVCLive. (Should be common sense to work with the extension, but not so much if expecting the setup pages to show something in between those steps - "I've added DVCLive, now what?")
[ ] Add a button below the table if there is no dvc.yaml file ("Add configuration")
[x] Either replace the "Show Experiments" button with "Run Experiment" when the table is open or replace the "Show Experiments and Plots" button with it at all times
[x] Accept other training scripts (not just python)

@dberenbaum I think we pretty much do the same now. Except we don't ask for a file to run (what if there is no file yet?). If project is not initialized, DVC is not installed we guide now you thought the steps on a separate screen.
Pipelines is the next stage for us. It can be on the same Setup screen that we have, but also Run button should be generating it, etc - that where we'll indeed start asking about files potentially (what is they are nor Python though?)
If we forget about pipelines for now, do you feel we should move DVCLive snippet from a separate screen (that now also helps with with DVC install, Git install, project init, etc) into something like Quick Pick + code snippets? Or it's fine that we keep it copy-pastable on a screen?

Pipelines is the next stage for us. It can be on the same Setup screen that we have, but also Run button should be generating it, etc - that where we'll indeed start asking about files potentially (what is they are nor Python though?)

💯 Putting this first to clarify that it's related/about this. We should guide users to run experiments by either adding a dvc.yaml pipeline or executing a single file, similar to how VS Code encourages adding launch.json but also allows debugging with auto config:

Maybe the proposal was not that clear. We can have further discussion or a call if it's still unclear. I think it's similar to your ideas for pipelines already, with the addition of being able to run experiments without a pipeline (which could be added after the pipelines setup is implemented).

If project is not initialized, DVC is not installed we guide now you thought the steps on a separate screen.

👍  The workflow to initialize a DVC repo there looks good now IMO thanks to the recent onboarding improvements. This proposal would be what you see after you initialize.

I think we pretty much do the same now.

Once I initialize DVC in a project, here's what I see:

If I click on one of the blue buttons, I see this:

Okay, I added that to my file, but I still see this screen. Now what? Right now, the initial sidebar has no visible option to run an experiment, which I might assume is a prerequisite to showing experiments. IMO we should have a prominent action in the sidebar at all times to run experiments.
If you somehow find the option to run experiments, it will try kick off dvc exp run despite the pipeline not being set up. Instead, can we detect there is no pipeline? This is where I would introduce the workflow above.

Except we don't ask for a file to run (what if there is no file yet?).

Then we need to tell them to create one, right? I don't think we can run experiments without at least some code file.

what is they are nor Python though?

Right, I had the same question above. IMHO let's see how big the need is before worrying about it - there's no reason we can't run non-Python files, but since DVCLive already pushes towards Python, it makes sense to start with Python.

If we forget about pipelines for now, do you feel we should move DVCLive snippet from a separate screen (that now also helps with with DVC install, Git install, project init, etc) into something like Quick Pick + code snippets? Or it's fine that we keep it copy-pastable on a screen?

I don't have a strong opinion. IMHO the quick pick/code snippets seem useful once I know what I'm doing and want to make it easier to add DVCLive to my code. I think we need the copy/paste screen to tell people to add it in the first place.

@shcheklein If there is somewhere better for these type of product discussions, let me know.

Summary of the discussion during the product sync with @dberenbaum :

Next step for the onboarding is to focus on "launch configurations" (helping creating a dvc.yaml e.g. when user clicks on a run button)
We can show a + Pipeline config on the experiments side panel
On the DVCLive snippets screen we don't show anything extra yet. Eventually we can show alternatives for other languages.

On the DVCLive snippets screen we don't show anything extra yet. Eventually we can show alternatives for other languages.

We could add some basic language/link to the docs to cover it without too much effort for now. Maybe it's less about other languages and more about providing ways for interested users to get a deeper understanding of what the code snippets are doing and how dvc metrics/plots/params work.

Related to https://github.com/iterative/vscode-dvc/issues/3021

I’ve been testing the onboarding and thinking about this. We will never see the the experiments table if we haven’t first ran anything. So before even making the distinction between running experiments with or without a configuration file, we need to find a way to show the run button so that it can be run a first time. Like @dberenbaum  was saying, there is no way to run experiments from the extension  after successfully setting up a project. Until then, we’re stuck with the “Your project has no data” screen. My suggestion for this is adding a big blue “Run Experiment” button in the sidebar (probably leave it there at all times). The other solution would be to add the button “Run Experiment” inside of the setup (”Your project has no data” screen). The one con of this second solution is that the user won’t necessarily understand easily how to run other experiments if they depend on this temporary button.
I'll add the button in the sidebar for now (it's practically no effort) so that I can move forward. iIf anything, we can change strategy later.

That sounds good, @sroy3. The next step/question is what happens if you don't have a dvc.yaml pipeline and click that run experiment button? Should we suggest they run the script where they added DVCLive themselves? Do we open a pre-filled dvc.yaml pipeline template for them?

That sounds good, @sroy3. The next step/question is what happens if you don't have a dvc.yaml pipeline and click that run experiment button? Should we suggest they run the script where they added DVCLive themselves? Do we open a pre-filled dvc.yaml pipeline template for them?

Currently on a spike / POC to create a dvc.yaml file

Adding an empty dvc.yaml file (as a first step) to the project https://github.com/iterative/dvc-get-started-cv/tree/1-notebook-dvclive makes it able to run experiment and add a new experiment to the experiments table. It works for this project because it is already set up correctly, but it seems that a new project would need more than that to show a table. Adding a params.yaml file creates the appropriate columns.
Currently, the last onboarding step is the screen that says "Your project contains no data". From there, we suggest that the user add DVCLive to their project. Should we ask them to create a params.yaml file as well? Are there other missing files that are required?
I'm not certain how a project should look before adding DVCLive to it.

Summarizing yesterday's meeting
Tasks

[ ] Add a quick pick to prompt the user for their training script to add to the dvc.yaml file being created
[ ] Remove beaker button to show experiments and replace with the play button to run experiment (like it is currently is when showing the table)
[ ] If there is a dvc.yaml file, also check for a pipeline in the file (check for stages key or run dvc stage list)
[ ] Add a mention in the last onboarding step ("Your project contains no data") to tell the user to run their training script after adding DVCLive. (Should be common sense to work with the extension, but not so much if expecting the setup pages to show something in between those steps - "I've added DVCLive, now what?")

Other possible tasks

[ ] Add a button below the table if there is no dvc.yaml file ("Add configuration")
[ ] Either replace the "Show Experiments" button with "Run Experiment" when the table is open or replace the "Show Experiments and Plots" button with it at all times

Specifications and clarifications
These might have been obvious for most people, but here are things I learned from the meeting that is helpful to me:

Running the training script should populate the table on its own (no need to use the run experiment button)
No params.yaml file should be necessary
dvc stage list will show pipelines if any

If there is a dvc.yaml file, also check for a pipeline in the file (check for stages key or run dvc stage list)

I would suggest dvc stage list -R .. This should be the most robust way to check for stages throughout the project.

@sroy3 perfect summary, thanks for making it (I really really appreciate it, it's super important for us to be aligned). Please, feel free also to pull the list of items (tasks) to the description and maintain it there. If needed we can split even into smaller tickets.

Remove beaker button to show experiments and replace with the play button to run experiment (like it is currently is when showing the table)

Whilst doing this can we remove all of the view/title entries for dvc.showExperiments & dvc.showPlots? IMO it doesn't make sense to duplicate them with the massive buttons available above.

If there is a dvc.yaml file, also check for a pipeline in the file (check for stages key or run dvc stage list)

I would suggest dvc stage list -R .. This should be the most robust way to check for stages throughout the project.

Maybe it's something that's not being done on our side, but pipelines in nested dvc.yaml files aren't being run in the extension.

To follow-up #3197:
To run a notebook instead of a python script, I found that you can convert and execute a notebook with the following command jupyter nbconvert --to ANY_NAME --inplace --execute SCRIPT_PATH.ipynb. This works pretty well. The only downside is that jupyter needs to be explicitly installed.
We might also want to customize the first stage a little more. I'd add a first step to name the stage of the pipeline with a default value of "train". Then instead of the "Enter the path to your training script or select it" step, we could simply ask "Enter the full command to run, or select  a python script or a Jupyter notebook to run". This way, I think we would cover most basic cases. Should we also ask for other files (dependencies, outputs...) or simply invite them to edit the dvc.yaml file to suit their needs?

We might also want to customize the first stage a little more. I'd add a first step to name the stage of the pipeline with a default value of "train". Then instead of the "Enter the path to your training script or select it" step, we could simply ask "Enter the full command to run, or select a python script or a Jupyter notebook to run". This way, I think we would cover most basic cases. Should we also ask for other files (dependencies, outputs...) or simply invite them to edit the dvc.yaml file to suit their needs?

Concerning this, and mainly running a non-Python command. Are we talking about scripts in other languages or more complex bash commands? For other languages, we could probably have some sort of mapping script extension to command that would make this easier for us. If we are talking about more complex commands (python script1.py && python scripts2.py), how  often are we expected to encounter this? Should we simply allow one script and automatically open the dvc.yaml file with the newly created pipeline with a comment to pipeline documentation so that they add other stages and customize them to their liking?

Should we also ask for other files (dependencies, outputs...) or simply invite them to edit the dvc.yaml file to suit their needs?

We can add a dep on the script file (e.g. train.py).
We can add additional fields commented out to show more option + a link to the docs.

Are we talking about scripts in other languages or more complex bash commands?

I would not bother too much with this, notebook and Python are enough. Everything else - I would put them into cmd as-is.

We might also want to customize the first stage a little more. I'd add a first step to name the stage of the pipeline with a default value of "train"

Yep, sounds good! All the suggestions are reasonable.

For the add configuration button. Here are two placement possibilities:
A (bottom right corner)
https://user-images.githubusercontent.com/3683420/218545777-1a5d29bd-4fc7-4772-b526-5dde6e9f35f7.mov
B (bottom of table)
https://user-images.githubusercontent.com/3683420/218546180-1ca5f23f-b0a9-4e95-ab75-0332c3219c39.mov
Let me know which would be preferred.

@sroy3 let's make bottom of the table. I also think it's fine for the button to be hidden if table is big enough (no need for it to float on top of the table).

We should disable this if there is already a dvc.yaml with stages
Can we make it a link not a button?
We can then include some text (in dark grey, similar to the onboarding screen or to the VS Code empty screen if close alls tabs) - "Add a project file for reproducibility, for queueing and running experiments in parallel. "

wdyt?

We should disable this if there is already a dvc.yaml with stages

Already implemented (still only in local version)

Can we make it a link not a button?

A button seems like a better choice for what it is supposed to do. It's an action. Links are for linking. We expect to be redirected when clicking. Clicking "Add a configuration" will simply pop the same implemented steps to create a basic pipeline. Unless we want to create a different process (create a dvc.yaml file with a template where the user fills everything on their own?). If we really want to go with a link, I'll do it, but IMO a button suits the purpose better.

We can then include some text (in dark grey, similar to the onboarding screen or to the VS Code empty screen if close alls tabs) - "Add a project file for reproducibility, for queueing and running experiments in parallel. "

That's possible. I'll do that one in a second step to keep the PR context light.

Looks good @sroy3!
@shcheklein As a follow up, what do you think of making it so that all the run/queue actions (like the play button at the top of the table) redirect to add configuration if there's no existing stages? This will also help users understand the benefits of adding it, along with hints in the file itself.

As a follow up, what do you think of making it so that all the run/queue actions (like the play button at the top of the table) redirect to add configuration if there's no existing stages? This will also help users understand the benefits of adding it, along with hints in the file itself.

Yes, definitely. They all should behave the same way.


[FR] Transform Chaining, Lazy Mapping

Feature request
Currently using a map call processes and duplicates the whole dataset, which takes both time and disk space.
The solution is to allow lazy mapping, which is essentially a saved chain of transforms that are applied on the fly whenever a slice of the dataset is requested.
The API should look like map, as set_transform changes the current dataset while map returns another dataset.
Motivation
Lazy processing allows lower disk usage and faster experimentation.
Your contribution
_

You can use with_transform to get a new dataset object.
Support for lazy map has already been discussed here a little bit. Personally, I'm not a fan, as this would make map even more complex.

I read about IterableDataset, and it seems to have lazy mapping. But I can't figure out how to convert an IterableDataset into a normal one when needed.
with_transform still does not chain AFAIU.

I read about IterableDataset, and it seems to have lazy mapping. But I can't figure out how to convert an IterableDataset into a normal one when needed.

You must cache an IterableDataset to disk to load it as a Dataset. One way to do this is with Dataset.from_generator:
from functools import partial
from datasets import Dataset

def gen_from_iterable_dataset(iterable_ds)
    yield from iterable_ds

ds = Dataset.from_generator(partial(gen_from_iterable_dataset, iterable_ds), features=iterable_ds.features})

with_transform still does not chain AFAIU.

Yes, not supported yet - the solution is to combine the transforms into a single one.

I wonder if it would be beneficial to have a dedicated method to do that ? Maybe a .save_to_disk() so that the user can reload the resulting dataset later ?

from functools import partial
from datasets import Dataset

def gen_from_iterable_dataset(iterable_ds)
    yield from iterable_ds

ds = Dataset.from_generator(partial(gen_from_iterable_dataset, iterable_ds), features=iterable_ds.features})

@mariosasko With these complex mapping functions, what hash will be used to cache this dataset?

The params passed to Dataset.from_generator will be used to compute the hash (partial encapsulates the iterable_ds value, so changing it will also change the hash)

Nesting doesn't seem to work:
from datasets import Dataset
d1 = Dataset.from_dict({"a": [0, 1, 2]})

def transform1(example):
    example["col1"] = [x * 2 for x in example["a"]]
    return example

d2 = d1.with_transform(transform1)
print(d2[0])

def transform2(example):
    example["col2"] = [x * 3 for x in example["a"]]
    return example
d3 = d2.with_transform(transform2)

print(d1[0])

print(d2[0])

print(d3[0])

>>> print(d1[0])
{'a': 0}
>>>
>>> print(d2[0])
{'a': 0, 'col1': 0}
>>>
>>> print(d3[0])
{'a': 0, 'col2': 0}
>>>

Hi, I think this feature would be very useful. I want to concatenate large datasets with heterogeneous columns. I dislike map since I don't want multiple copy of that datasets locally. I tried to use "set_transform" on each dataset to convert it to a standard features format, but datasets.concatenate_datasets ignores the updated format of the datasets.  A work around is to use torch.utils.data.ConcatDataset. Is there a neat way to do it using HF datasets?


Refresh

Note to self: Change auto to un-comment
    #- conda inspect linkages -p $PREFIC hdfeos2  # [not win]
    #- conda inspect objects -p $PREFIX hdfeos2  # [osx]

when fetching these recipes.

Hi! This is the friendly automated conda-forge-linting service.
I just wanted to let you know that I linted all conda-recipes in your PR (recipe) and found it was in an excellent condition.


when using onEditQuery, the explorer scroll resets to the beginning

Is there an existing issue for this?

[X] I have searched the existing issues

Current Behavior
In the context of onEditQuery handler, every time it fires, i had to set a state which then resets the scroll to the beginning.
Expected Behavior
The scroll state should persist based on the active query or remain as is without resetting to the beginning.
Steps To Reproduce
const test = ()=>{
const [query, setQuery] = useState("");
const plugins = [explorer];
const onEditQuery = (query) => {
setQuery(query);
};
return ( )
}
Environment

GraphiQL Version:  3.0.10
OS: Windows/Linux
Browser: Chrome
Bundler:
react 18.1.0
graphql Version: 16.4.0

Anything else?
No response

You must to wrap onEditQuery with useCallback


how to keep metadata from wav? (question)

Hi, this is not an issue, but I don't know about a forum or a place where I could ask this question. How can I keep the metadata from a wav file when using wavpack. I am using this command: wavpack -hh -x4 -m --threads input.wav
When I use ffmpeg the metadata is still present in the wv file with: -c:a wavpack -compression_level 3 output.wv
Thanks for any help :)
edit: I have noticed that after decoding wv in wav with wvunpack the metadata reappears, is there a way to see it when the file is wv?

There is a dedicated WavPack forum here, but posting an issue right here works too (although I'm probably the only one who will see it).
I'm a little confused because normally I would think that the WavPack command-line programs would be better than FFmpeg for retaining metadata in WAV files. And, as you discovered, when you restore a WAV file using the command-line wvunpack program the original WAV file is always exactly restored (so obviously all metadata is back).
First of all, what metadata are we talking about? Is it like artist and title and stuff like that? Or is it loop and cue and technical metadata like that?
And so you claim that when you create WavPack files using FFmpeg something is retained that you can access afterward, but when you use the wavpack command-line program it doesn't show up. How is it that you try to view this metadata?
Another thing to try is to use the command-line wvunpack -ss file.wv to examine the two files. That will tell you exactly what metadata is in there and also the size of the stored WAV headers (which also can contain metadata). Comparing the output from each file should tell us exactly what's going on.
Hope this helps!

Thank you for your answer.

First of all, what metadata are we talking about? Is it like artist and title and stuff like that? Or is it loop and cue and technical metadata like that?

I am not talking about any fajny metadata, actually the basic stuff like Artist, Album, Title.

And so you claim that when you create WavPack files using FFmpeg something is retained that you can access afterward, but when you use the wavpack command-line program it doesn't show up. How is it that you try to view this metadata?

I use mediainfo
og wav

ffmpeg wavpack, the metadata is slightly changed, kinda different arrangement but you can see it even in wv form

native wavpack, here its missing

Another thing to try is to use the command-line wvunpack -ss file.wv to examine the two files. That will tell you exactly what metadata is in there and also the size of the stored WAV headers (which also can contain metadata). Comparing the output from each file should tell us exactly what's going on.

This gave me these results. Yes they look pretty different.
ffmpeg wavpack

native wavpack

Okay, I see what's happening. The metadata is stored in the RIFF header in the WAV file. That's kind of a non-standard way of storing them, but apparently FFmpeg knows about it. Where did these WAV files come from?
Anyway, if the metadata is in ID3 tags (possible) then you could add the --import-id3 to the wavpack encoder command-line and they would get picked up then. Otherwise the only way would be to use some third-party tagging utility like Mp3tag (which comes from your country I think).
If it's not too much trouble I would like to have one of these WAV files to check (maybe this is something I should support, even though I've never seen it before). You could compress a short track with WavPack using -b2 and it would get really small, and that would be fine (I just want the header). Thanks!

From music streaming providers, not sure which one, but it is originally a flac, I made the wav with ffmpeg because xiph flac CLI removes all metadata. Sure here it is.
input.zip
I already tried this command --import-id3 and it didn't work.
Awesome a fix for that in the future would be appreciated :) Already the new --thread option is a game changer. It should be the default option.

Cool, thanks for the file. I'll take a look.
And thanks for the comments about --threads. I have considered making it the default (and adding a --no-threads to disable), especially because it doesn't seem to slow down even single-core machines.

You welcome. Yes exactly and probably not everyone knows about it, and is missing out. It is also a strong argument against going with ffmpeg when you are into wavPack. I can understand leaving --optimize-int32 because of back compat, but it will also have be the default in the future.

Here is a native flac, maybe this helps somehow
Flac.zip

Yes, thanks, the flac can help too. I also will probably just have to look into FFmpeg and see what it's doing with these tags to make sure I get them all.

As Im planing to switch all flac files to wavpack, how should I handle the metadata that is stored in the flac files? As you have to convert flac back to wav and then to WavPack. It would be nice to see the metadata in the wv files.

I think the best strategy would be to avoid converting through WAV altogether. WAV files don't really have a standard method of holding metadata, and some of the ad-hock methods that are used have severe limitations (like field length or character set availability).
I generally use Foobar2000 for trancoding because it handles WavPack great and is very good and preserving metadata (even cover art) and it uses the command-line program to encode. FFmpeg also does a good job (although at one point cover art was sometimes lost). If you don't like using the FFmpeg WavPack encoder, one thing that I often do is use the WavPack command-line program to reëncode the resulting files with the mode and features you want. This makes it a fresh file with no leftover FFmpeg-isms (except their encoder name which they add to the tags, but this can be deleted).
If you insist on going through WAV files, then using a tag editor like Mp3tag (mentioned above) to copy the tags afterward would work.

I'm fine without converting to wav, but does wavPack accept flac as input? I tried it and the cmd window closes. ffmpeg is very slow as it doesn't have the latest --threads option, and generally I wanted to use the native encoder. I never tried foobar2000

No, WavPack does not accept FLAC as input. You have to use something that decodes FLAC, like FFmpeg or Foobar2000.
FFmpeg uses their own implementation of WavPack for both encoding and decoding. It does not generate the latest version of the format (version 4 instead of 5 as you discovered above) but that's not really a big deal. They're lossless and the audio is correct, but you're right that they don't support multithreading for encode. But that's why I suggested using the default mode in FFmpeg for a quick-and-dirty conversion and then reëncoding in place using the command-line program. You can use --threads and whatever mode you'd like and the results are fully modern version 5.
Foobar2000 uses the WavPack command-line program for encoding and has libwavpack internal for decode, so no such issues there. And you can specify any options you want in the conversion configuration dialogs.

I also will probably just have to look into FFmpeg and see what it's doing with these tags to make sure I get them all.

Okay I just hope that in future the metadata can be transfered from the source wav (like ffmpeg), that would be a bit more straightforward ^^'


Promotion oppertunities

Action from meeting 15th September 2022
Action 6 (for all): All members to report where there has been uses or references to the JCDP to help with tracking usage of the system
Action 7 (for all):  Promote the JCDP within your own networks of data collectors and providers, and any suggestions of other opportunities (conferences and meeting)
Action 8: DONE LC Investigate whether a shared calendar can be set up in GitHub or SharePoint for members to add promotion & engagement opportunities, such as conferences and key meetings.

Action 8 Update
LC has added a shared calendar to SharePoint - https://community.ices.dk/ExpertGroups/WGJCDP/_layouts/15/start.aspx#/Lists/WGJCDP calendar/calendar.aspx
Follow on action: please add any upcoming meeting, events and other opportunities, including details of whether you are planning to represent the JCDP in some way.


Easybuilder MSB1008 on first try / full source

I get this (MSBUILD : error MSB1008: Only one project can be specified. ) when i try to build mangos with EasyBuild ServerTwo
About to run....
C:\Program Files\CMake\bin\cmake.exe  "D:\MangOS\MangOS Two\serverTwo" -DBUILD_TOOLS:BOOL=1 -DBUILD_MANGOSD:BOOL=1 -DBUILD_REALMD:BOOL=1 -DSOAP:BOOL=0 -DSCRIPT_LIB_ELUNA:BOOL=1 -DSCRIPT_LIB_SD3:BOOL=1 -DPLAYERBOTS:BOOL=0 -DCMAKE_INSTALL_PREFIX="D:\MangOS\MangOS Two\serverTwo_install" --build  "D:\MangOS\MangOS Two\serverTwo" -G "Visual Studio 15 Win64"
-- Selecting Windows SDK version 10.0.17763.0 to target Windows 10.0.17134.
-- The C compiler identification is MSVC 19.16.27027.1
-- The CXX compiler identification is MSVC 19.16.27027.1
-- Check for working C compiler: C:/Program Files (x86)/Microsoft Visual Studio/2017/Community/VC/Tools/MSVC/14.16.27023/bin/Hostx86/x64/cl.exe
-- Check for working C compiler: C:/Program Files (x86)/Microsoft Visual Studio/2017/Community/VC/Tools/MSVC/14.16.27023/bin/Hostx86/x64/cl.exe -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Check for working CXX compiler: C:/Program Files (x86)/Microsoft Visual Studio/2017/Community/VC/Tools/MSVC/14.16.27023/bin/Hostx86/x64/cl.exe
-- Check for working CXX compiler: C:/Program Files (x86)/Microsoft Visual Studio/2017/Community/VC/Tools/MSVC/14.16.27023/bin/Hostx86/x64/cl.exe -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Detected 64-bit platform
-- Found Git: C:/Program Files/Git/cmd/git.exe
-- OpenSSL library: optimized;C:/Program Files/OpenSSL-Win64/lib/VC/ssleay32MD.lib;C:/Program Files/OpenSSL-Win64/lib/VC/libeay32MD.lib;debug;C:/Program Files/OpenSSL-Win64/lib/VC/ssleay32MDd.lib;C:/Program Files/OpenSSL-Win64/lib/VC/libeay32MDd.lib
-- OpenSSL headers: C:/Program Files/OpenSSL-Win64/include
-- OpenSSL version: 1.0.2k
-- Found MySQL library: C:/Program Files/MySQL/MySQL Server 5.6/lib/libmysql.lib
-- Found MySQL headers: C:/Program Files/MySQL/MySQL Server 5.6/include
-- Install server to     : D:/MangOS/MangOS Two/serverTwo_install
-- Install configs to    : D:/MangOS/MangOS Two/serverTwo_install/
-- Search configs from   : binary directory (default)
-- Support for SOAP      : No (default)
-- Script engine Eluna   : Yes (default)
-- Script engine SD3     : Yes (default)
-- Enable Player Bots    : No
-- Build main server     : Yes (default)
-- Build login server    : Yes (default)
-- Build tools           : Yes (default)
-- Build in debug-mode   : No  (default)
-- Configuring done
-- Generating done
-- Build files have been written to: D:/MangOS/MangOS Two/serverTwo_build
VS2017: msBuildPath 64Bit:C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\MSBuild\15.0\BinC:\Program Files (x86)\Microsoft Visual Studio\2017\Community\MSBuild\15.0\Bin
About to run:C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\MSBuild\15.0\Bin\msbuild.exe D:\MangOS\MangOS Two\serverTwo_build/INSTALL.vcxproj /p:Platform=x64 /p:VisualStudioVersion=15.0 /tv:15.0 /p:TargetFramework=v15.0 /detailedsummary /v:m /t:Build /maxcpucount /p:Configuration=Release /p:Platform=x64
Build Started: 2/24/2019 2:57:26 AM
Microsoft (R) Build Engine version 15.9.21+g9802d43bc3 for .NET Framework
Copyright (C) Microsoft Corporation. All rights reserved.
MSBUILD : error MSB1008: Only one project can be specified.
Switch: Two\serverTwo_build/INSTALL.vcxproj
For switch syntax, type "MSBuild /help"
=========================================================
Build Started: 2/24/2019 2:57:26 AM
Build Complete: 2/24/2019 2:57:26 AM

@wizzymore My apologies for the late reply.
There is a known bug in easybuild which means the path must not have a space in it
So instead of having a path: "D:\MangOS\MangOS Two\serverTwo"
Change this to the following: "D:\MangOS\MangOS_Two\serverTwo" and it should build correctly.

Regards

This is now checked for and the build prevented.


BLD: Add support for python 3.10 and drop 3.6

Codecov Report

Merging #106 (1a519f1) into main (75dda57) will not change coverage.
The diff coverage is 100.00%.

@@            Coverage Diff            @@
##              main      #106   +/-   ##
=========================================
  Coverage   100.00%   100.00%
=========================================
  Files            2         2
  Lines          135       135
=========================================
  Hits           135       135

Impacted Files
Coverage Δ

polyagamma/__init__.py
100.00% <100.00%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 75dda57...1a519f1. Read the comment docs.


[Task]: Introduce new Flink source implementations based on FLIP-27 Source interfaces.

What needs to happen?
This task is a part of Flink batch runner migration. Because we are going to use DataStream for both streaming and batch job execution, we need to change the existing source implementations to be based on FLIP-27 Source interfaces. This will allow DataStream to run agains bounded input in batch execution mode.
Issue Priority
Priority: 2 (default / most normal work should be filed as P2)
Issue Components

[ ] Component: Python SDK
[ ] Component: Java SDK
[ ] Component: Go SDK
[ ] Component: Typescript SDK
[ ] Component: IO connector
[ ] Component: Beam examples
[ ] Component: Beam playground
[ ] Component: Beam katas
[ ] Component: Website
[ ] Component: Spark Runner
[X] Component: Flink Runner
[ ] Component: Samza Runner
[ ] Component: Twister2 Runner
[ ] Component: Hazelcast Jet Runner
[ ] Component: Google Cloud Dataflow Runner

.take-issue


support {param} style and and array query params

Would solve https://github.com/rricard/koa-swagger/issues/2

@theomega could you please the lint errors

@philip1986 done

Seems good to me: thanks!

@philip1986 oh yea, sorry!

0.3.0 published!

@rricard tanks a lot. My npm user name is also philip1986.


pytorch support

This is a nice package for HPO,  but it seems that it is  built only on TF.
Just wondering is there any version that supports pytorch?
I think that will make the package more popular for HPO reseachers!

Thanks @ifangcheng !
In fact, there is a person that's working on it; but I don't know when it will be ready.
I presume that at least some part of this package (the reverse mode hypergradient) should be easier to implement in Pythorch since - as far as I know - Pythorch should support by default dynamic computational graphs. Unfortunately I don't have much experience with Pythorch myself.... but, of course,  if you want to help and get involved, you'd be very welcomed! :-)
Best,
Luca

Hi,
Do you have any news on the work for the PyTorch implementation ? I'm really interested because i'd like to use specifically the reverse mode HG in PyTorch ! :D
Thanks for your great work, it's still very helpful even only on TF !
Best,
Antoine

Hi @AntoineHX
We are shortly releasing a PyTorch implementation of an algorithm related to RTHO (real time hyperparamter optimization) for optimizing learning rate schedules online.
Unfortunately at the moment we have no specific short term plans of releasing a general implementation/wrappers for reverse mode HG in PyTorch, but this may change in the future.
I'll keep you updated!
Cheers,
Luca

Hi all,
just wanted to post the link of a new package for gradient-based hyperparameter tuning in pytorch.
https://github.com/awslabs/adatune
Right now it only contains the code for running an enhanced version of RTHO, but it may grow in the future months.
Cheers,
Luca


Is there a way to set the owner and owning group of an azure storage file and directory

Is your feature request related to a problem? Please describe.
The sdk documentation is not complete, and I can't find a python based function to set the owner and owning group of a file/directory in an azure storage account.
Describe the solution you'd like
Documentation of an api call to set the owner and owning group of an azure storage directory and file using python. This would include how to reference $superuser in an such an api call.
Describe alternatives you've considered
I am able to do this in powershell, but there is enormous convenience in doing it using python. I was wondering if this could be passed as part of the acl to set_access_control, but I cannot find documentation that indicates all of the options of this method.
Additional context
Here is how I do this in powershell:
Update-AzDataLakeGen2Item -Context $ctx -FileSystem $filesystem -path $file.Name -Owner "`$superuser"

@Mark-Knutson Apologies for the late reply. Thanks for reaching out to us and reporting this issue. Could you please check if the below sample helps ?

from azure.storage.file import DataLakeServiceClient
from azure.storage.file import ContentSettings

# Retrieve the account connection string from the environment variable
# AZURE_STORAGE_CONNECTION_STRING.
conn_str = os.getenv('AZURE_STORAGE_CONNECTION_STRING')

# Create the service client with the connection string
service = DataLakeServiceClient.from_connection_string(conn_str=conn_str)

# Create a file system client
file_system_client = service.get_file_system_client(file_system="filesystem1")

# Create a directory client
dir_client = file_system_client.get_directory_client(directory_path="dir1")

# Create a file client
file_client = dir_client.get_file_client(file_path="file1")

# Update the file with the new content settings
file_client.set_content_settings(
    content_settings=ContentSettings(
        content_type="image/jpeg",
        content_encoding="UDF8",
        content_language="EN-US",
        cache_control="READ",
        content_disposition="True",
        content_md5="i727sP7HigloQDsqadNLHw=="
    )
)

# Update the file with the new permissions
file_client.set_permissions(
    permissions="rw-rw-rwx",
    owner="$superuser",
    group="$superuser"
)

# Update the file with the new metadata
file_client.set_metadata(
    metadata={
        "tag1": "value1",
        "tag2": "value2"
    }
)

# Get the file
file = file_client.get_file_properties()

# Print the file properties
print("File properties:")
print(file)

# Get the file acl
acl = file_client.get_access_control()

# Print the file acl
print("File ACL:")
print(acl)

# Get the file permissions
permissions = file_client.get_permissions()

# Print the file permissions
print("File permissions:")
print(permissions)

# Get the file metadata
metadata = file_client.get_metadata()

# Print the file metadata
print("File metadata:")
print(metadata)

The script did not work for me. I note the following things I had to change to get it to run:
Imports changed to this:
from azure.storage.filedatalake import DataLakeServiceClient
from azure.storage.filedatalake._models import ContentSettings
got error messages on named parameters which were cured with positional parameters:
Create a directory client
#dir_client = file_system_client.get_directory_client(directory_path="/")
dir_client = file_system_client.get_directory_client("/")
Create a file client
#file_client = dir_client.get_file_client(file_path="folder_1")
file_client = dir_client.get_file_client("folder_1")
For the operative code,  I got this error:
file_client.set_permissions(
owner="$superuser",
group="$superuser"
)
AttributeError: 'DataLakeFileClient' object has no attribute 'set_permissions'
I am running this in vscode. In this environment, I am able to set acl for files and directories. I noted that the group and owner showed up in the acl from get_access_control. The setting of properties and metadata did not appear to be relevant.
I also tried to include superuser in the acl string, and got a different error:
'user:$superuser:rwx,group:$superuser:r-x,other::---,mask::rwx'
Message: The named user or named group in the access control list is not valid.
This option was just a guess on my part, not in the example sent.
Thanks.
From: navba-MSFT @.>
Sent: Wednesday, April 19, 2023 1:12 AM
To: Azure/azure-sdk-for-python @.>
Cc: Mark C Knutson @.>; Mention @.>
Subject: [External] Re: [Azure/azure-sdk-for-python] Is there a way to set the owner and owning group of an azure storage file and directory (Issue #29861)
CAUTION: This email was sent from outside of Hennepin County. Unless you recognize the sender and know the content, do not click links or open attachments.
@Mark-Knutsonhttps://github.com/Mark-Knutson Apologies for the late reply. Thanks for reaching out to us and reporting this issue. Could you please check if the below sample helps ?
from azure.storage.file import DataLakeServiceClient
from azure.storage.file import ContentSettings
Retrieve the account connection string from the environment variable
AZURE_STORAGE_CONNECTION_STRING.
conn_str = os.getenv('AZURE_STORAGE_CONNECTION_STRING')
Create the service client with the connection string
service = DataLakeServiceClient.from_connection_string(conn_str=conn_str)
Create a file system client
file_system_client = service.get_file_system_client(file_system="filesystem1")
Create a directory client
dir_client = file_system_client.get_directory_client(directory_path="dir1")
Create a file client
file_client = dir_client.get_file_client(file_path="file1")
Update the file with the new content settings
file_client.set_content_settings(
content_settings=ContentSettings(

    content_type="image/jpeg",

    content_encoding="UDF8",

    content_language="EN-US",

    cache_control="READ",

    content_disposition="True",

    content_md5="i727sP7HigloQDsqadNLHw=="

)

)
Update the file with the new permissions
file_client.set_permissions(
permissions="rw-rw-rwx",

owner="$superuser",

group="$superuser"

)
Update the file with the new metadata
file_client.set_metadata(
metadata={

    "tag1": "value1",

    "tag2": "value2"

}

)
Get the file
file = file_client.get_file_properties()
Print the file properties
print("File properties:")
print(file)
Get the file acl
acl = file_client.get_access_control()
Print the file acl
print("File ACL:")
print(acl)
Get the file permissions
permissions = file_client.get_permissions()
Print the file permissions
print("File permissions:")
print(permissions)
Get the file metadata
metadata = file_client.get_metadata()
Print the file metadata
print("File metadata:")
print(metadata)

Reply to this email directly, view it on GitHubhttps://github.com/Azure/azure-sdk-for-python/issues/29861#issuecomment-1514183993, or unsubscribehttps://github.com/notifications/unsubscribe-auth/A2UYQMBHTIOKUYXLVASYOJ3XB564LANCNFSM6AAAAAAW4CZ2BY.
You are receiving this because you were mentioned.Message ID<EMAIL_ADDRESS>Disclaimer: If you are not the intended recipient of this message, please immediately notify the sender of the transmission error and then promptly permanently delete this message from your computer system.

@Mark-Knutson Thanks for your reply. Apologies for the confusion. Here is the same code to set the ACL on directory and file in Azure datalake storage account.
import os, uuid, sys
from azure.storage.filedatalake import DataLakeServiceClient
from azure.core._match_conditions import MatchConditions
from azure.storage.filedatalake._models import ContentSettings

# Set the Storage account connection string from the environment variable
conn_str = "XXXXXXXXXXXXX"

# Create the service client with the connection string
service_client = DataLakeServiceClient.from_connection_string(conn_str=conn_str)

# Create a file system client
file_system_client = service_client.get_file_system_client(file_system="temp")

#try:
# Create a directory client
dir_client = file_system_client.create_directory("test1")

# Create a file client
file_client = dir_client.get_file_client("data.xlsx")

# set the permissions of the parent directory
new_dir_permissions = 'rwx------'
dir_client.set_access_control(permissions=new_dir_permissions)

# Set and display the Owner and Owner Group
dir_client.set_access_control(owner="OwnerOID", group="MyGroupName")
acl_props = dir_client.get_access_control()
print("New permissions of directory '{}' are {}.".format("test1", acl_props['permissions']))

file_client.set_access_control(permissions=new_dir_permissions)

print("Set the permissions of file '{}' to {}.".format("data.xlsx", new_dir_permissions))

If you want to add a complex acl setting, you can see the below syntax and pass this as a parameter to set_access_control():
acl = 'user::rwx,group::r-x,other::r--,user:xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx:r--'

set_access_control(acl=acl)

More Info here.
You need to tweak the above code using the set_access_control to add the owner and group to your directory / file:

**set_access_control(owner=None, group=None, permissions=None, acl=None, kwargs)

owner (str) – Optional. The owner of the file or directory.
group (str) – Optional. The owning group of the file or directory.
permissions (str) – Optional and only valid if Hierarchical Namespace is enabled for the account. Sets POSIX access permissions for the file owner, the file owning group, and others. Each class may be granted read, write, or execute permission. The sticky bit is also supported. Both symbolic (rwxrw-rw-) and 4-digit octal notation (e.g. 0766) are supported. permissions and acl are mutually exclusive.
acl (str) – Sets POSIX access control rights on files and directories. The value is a comma-separated list of access control entries. Each access control entry (ACE) consists of a scope, a type, a user or group identifier, and permissions in the format “[scope:][type]:[id]:[permissions]”. permissions and acl are mutually exclusive.
More Info here.

@Mark-Knutson Thanks for the update. Regarding your below pending asks:

Set recursive ACL via Python SDK: Refer the below sample code which calls set_access_control_recursive() function and update the right ACL permissions:

def set_permission_recursively(is_default_scope):

    try:
        file_system_client = service_client.get_file_system_client(file_system="my-container")

        directory_client = file_system_client.get_directory_client("my-parent-directory")

        acl = 'user::rwx,group::r-x,other::r--,user:xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx:r--'

        if is_default_scope:
           acl = 'default:user::rwx,default:group::r-x,default:other::r--,default:user:xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx:r--'

        directory_client.set_access_control_recursive(acl=acl)

        acl_props = directory_client.get_access_control()

        print(acl_props['permissions'])

    except Exception as e:
     print(e)

More Info here.

Regarding the documentation improvement, I have created a workitem and shared the feedback internally with the Product owners. They will look into the docs enhancement request and track it separately.


Develop

These code style promote to use always if over unless; that's ok! but evaluating a conjunction condition, i.e, using a && operator, with the unless statement is more efficient than transforming in a if statement with disjoint condition evaluation, i.e, using the || operator, since the && operator does not evaluates the second expression if the first fails. More over, not always is convenient to interchange the operators because evaluating the second expression may be undesired and it may affects the future flow of the execution control.


actions failing

@charlesneimog eu removi o folder 'libpd' e agora o Linux não acerta a compilation, como arrumar isso?

need help here?

yup, and this looks simple, just remove the deleted folder... I'd actually like to learn where to edit and deal with this.

it's in the actions script .github/workflows/c-cpp.yml (it starts with a dot, so it's hidden folder)
i just removed this line here, cause libpd doesn't exist anymore
https://github.com/porres/pd-else/commit/b633a35ab9a088cf7525edaae7411bac5ad52a7a#diff-a6b2b0a5a900a2b301f746f7bafdce303c6c647e00ade3ee28e8f8ebea11c7a5L89
and changed this cause you moved extra_source to inside of Compiled
https://github.com/porres/pd-else/commit/b633a35ab9a088cf7525edaae7411bac5ad52a7a#diff-a6b2b0a5a900a2b301f746f7bafdce303c6c647e00ade3ee28e8f8ebea11c7a5R91
and then just replaced signal with audio

thanks


toggl integration - stop and start timer (also in task view)

shift+c starts toggl timetracking but it doesn't seem to work in task view (i.e. after clicking 'i' on a given task in task list view). Can we make it work also in that view?

shift+c doesn't seem to be stopping the timer. Could we change it so that shift+c actually flips the state of the button (click if it's stopped, stop it when it's running)?

Hey, thanks for bringing my attention to this!
For me shift+c does stop the timer, but only if the current task selection has the timer in it.  I suppose it could click an active timer button if visible.  However, that seems inconsistent to me for the case where you view a project that doesn't have the active item in it.  In that case, shift+c would instead activate a timer rather than toggling the timer state.
I believe this is probably confounded by the toggl button(s) usually being invisible - #119
I think I've fixed it working in the task view in the work on #114

I've just released version 77, has the task view fix


Consider removing difference between app and lib until publishing

Today a zOS user needs to decide up front whether he'll be using zOS to create a shared lib or an app, and that decision cannot be changed later.
We can leverage lightweight mode to allow for postponing this decision: initially, all development is in lightweight mode, but we can give our users the chance to publish their work as an app or lib (or both) later down the road, based on whether they want their work to be reusable by other teams.

This came up from an idea by @maraoz. After discussing it with @facuspagnuolo, we arrived to the conclusion that an app is currently a strict superset of a lib (except that the app arbitrarily blocks the freeze command).
We could very easily implement this by just dropping the --lib flag to the init command. This would imply:

Users do not need to decide whether their project will be a lib or an app when initializing the project.
Every project will start ligthweight, and switch into full mode upon publishing (we could even drop the need for push during lightweight later down the road)
Partners working on a lib would be able to create proxies to test their own code

On the other hand, we see the following downsides:

Any proxy created by a lib developer for testing will be included in the zos.network.json, and broadcasted to all consumers of the lib; the CLI will ignore this info though

sounds great!! We can deal with hiding or removing the "unwanted proxies" later in development :)


Is there way to programatically create browse categories from a csv in DPUL, versus manually tagging individual items in DPUL?

As a site builder, I would like to create browse categories for each building in the forthcoming grounds and buildings collection in DPUL. I have a csv that has component ids that correspond to each building, along with the arks for each image.  Is there a way to automate this process?

@kelea99 There is not currently a way to do this but it could be implemented.

@hackmastera , @kelynch and I were having a conversation about collections creation in figgy and I think question indirectly relates. It might be a good idea for a few of us, and @tpendragon as lead for figgy to talk about potential best practices / uses for each before something like this is considered? Id hate to waste your time.


Build a prod environment: FAILED! => {"msg": "Timeout (12s) waiting for privilege escalation prompt: "}

Issue Type

Bug Report / Support Request

Your Environment
Vagrant 2.0.4
VirtualBox 5.2.10r122088
ansible 2.5.2
  config file = /home/jazzper/WebDev/Sites/ksro/ansible.cfg
  configured module search path = [u'/home/jazzper/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib/python2.7/dist-packages/ansible
  executable location = /usr/bin/ansible
  python version = 2.7.12 (default, Dec  4 2017, 14:50:18) [GCC 5.4.0 20160609]

Your OS

Linux (Ubuntu 16.04)

Full console output
Sorry I don't have the full console output. I thought the problem was a failing Internet connection (I am in a hostel at the moment). But after running the playbook over and over again I started searching for alternative explanations and stumbled upon:
https://github.com/ansible/ansible/issues/14426
I have (almost) exactly the same. Namely the following error, at random times:
FAILED! => {"msg": "Timeout (12s) waiting for privilege escalation prompt: "}
TASK [geerlingguy.nginx : Ensure nginx is installed.] **************************
fatal: [<IP_ADDRESS>]: FAILED! => {"msg": "Timeout (12s) waiting for privilege escalation prompt: "}

Summary
I followed https://www.jeffgeerling.com/blog/2017/soup-nuts-using-drupal-vm-build-local-and-prod to build a production environment but the playbook fails at random times. Running the playbook with "-c paramiko" solves the problem.
config.yml
---
# Composer project settings.
drupal_build_composer_project: false
drupal_build_composer: false
drupal_composer_dependencies: []

# Drupal install settings.
drupal_site_name: "Khaosanroad Online"
drupal_core_path: "/var/www/ksro/web"
drupal_install_site: true
drupal_major_version: 8

drupalvm_cron_jobs:
  - name: "Drupal Cron"
    minute: "*/15"
    job: "{{ drush_path }} -r {{ drupal_core_path }} core-cron --quiet"

# The web server software to use. Can be either 'apache' or 'nginx'.
drupalvm_webserver: nginx

# Other overrides.
php_version: "7.2" # ideally PHP 7.2, which is supported as of Drupal 8.5.0
php_sendmail_path: "/usr/sbin/sendmail -t -i"
installed_extras:
  - drush

# Other secure defaults.
dashboard_install_dir: ''

# Enable a more hardened security configuration.
extra_security_enabled: true
security_autoupdate_blacklist: "linux-image"
security_autoupdate_mail_to<EMAIL_ADDRESS>
# Restrict the firewall to only ports that are required for external services.
firewall_allowed_tcp_ports:
  - "22"
  - "80"
  - "443" #https
firewall_log_dropped_packets: true

prod.config.yml
---
# Deploy from the project's Git repository.
drupal_deploy: true
drupal_deploy_repo: "git@bitbucket.org:jazzper23/ksro.git"
drupal_deploy_dir: /var/www/ksro

# Set the domain for this site appropriately.
drupal_domain: "khaosanroadonline.com"
vagrant_hostname: "{{ drupal_domain }}"

nginx_hosts:
  - server_name: "{{ drupal_domain }} www.{{ drupal_domain }}"
    root: "{{ drupal_core_path }}"
    is_php: true

Does it work if you add ansible_ssh_pipelining: true to config.yml?

Strange! Please let us know if it works after setting that to false.

Please see: https://github.com/geerlingguy/drupal-vm/issues/1883 — Drupal VM's use as a production environment management tool is now explicitly experimental, with no support being offered for that use case.


chore: allow dag resyncing after node is out of pbft sync

If a node gets out of sync (this is known to happen while deleting old period_data blocks in a light node which blocks us from pbft progressing) and pbft syncing is restarted, a malicious peer exception can happen once pbft blocks are synced but dag blocks are not. Allow dag re-syncing once pbft syncing completes.

prnet cleaned-up successfully


Add compatibility with Develocity Maven extension 1.21

Thanks! I'm surprise dependabot hasn't picked this one up yet.

The coordinates have changed

The coordinates have changed

Good point. This is different than what happened for the DV Gradle plugin though


Collision check with some objects, but not others?

Hello, thanks for making this!

I was wondering if there is a way to somehow filter collisions for some objects?
Like "players should not collide with other players, but with enemies"?
I think I could store some information in the data field and could check for that onCollision, but that would be after the collision check has been made. Is there a way to omit collision checks on certain conditions?

Not a contributor to this project in any way, but just wanted to fill in my solution for this;
You could set flags on your colliders and each tick you could separate the colliders based on flags and only insert one flag at a time, do the checks, clear rbush (crash), and move on to the next flag. Does that make sense? :)

if you want a body to collide with anything you need to add it to rbush
after the collision is detected, then based on any flags/props on body you can skil movement on it
at least in detect-collisions, using system.checkAll(handleCollision), instead of system.separate() https://prozi.github.io/detect-collisions/
from looking at crash code I can see you should use crash.search(collider).forEach(handleCollision)
and then
function handleCollision(response) {
  if (!response.a.isSpaceShip) {
    // push back
  }
}


Relationship table for nested basins (Isobasins tool)

It would be useful if the tool Isobasins could produce a relationship table for the delimited watersheds. In other words, I would like to know, for each watershed, in which other one they flow. For example, if I have four watersheds and that #1 flow into #2 than itself flow into #3 but #4 flow into none I would like a simple TXT/CSV file with a table like:

Upstream
Downstream

1
2

2
3

3
0 or NA

4
0 or NA

With that in hand, I can use other tools to know that #3 is also downstream of #1 and that if something changes in #1, it could have an effect on both #2 and #3.
I'm currently writing something in Python to do that using the flow pointer, flow accumation and isobasins rasters but I'm pretty sure it will be quite inefficient compared to a built-in solution.

Would the UnnestBasins tool be useful for this?

Thanks for the suggestion, I justed tried it and I think it could do it but it will not be straightforward, at least for my purpose:

I need a script to generate the pour points first (I have to do this anyway... but it would be nice if Basins and Isobasins could optionally output pour points).
It generates one raster by nested level (could be a problem with several levels with a very large raster).
I will need another script to loop through the rasters and add their value of the pour points under.

I'll try something and see...

I just committed an updated version of the Isobasin tool that can optionally output a CSV file containing an upstream/downstream connection table. If you wouldn't mind testing the tool and reporting any issues that you run into with it. I have done a brief test with the sample Vermont DEM and it seemed to work well.

Works great, no issue to report. Thanks!


Upgrade to relay modern

Hi @denvned,
do you have any plan to upgrade the version of react-relay to 1.0.0?

@hung-phan I have a PR to support classic mode in<EMAIL_ADDRESS>https://github.com/denvned/isomorphic-relay/pull/66
For relay modern, I have an example of how to do isomorphic rendering without isomorphic-relay: https://github.com/robrichard/relay-modern-isomorphic-example


Make compatible with Laravel 5.2

According to the upgrade guide

The Illuminate\Foundation\Support\Composer class has been moved to Illuminate\Support\Composer. This is unlikely to affect your application if you were not manually using this class.

+1
I already upgraded to 5.2 and this broke. I can't wait so i will fork and fix this for now :/

+1

Pull request here: #215 by @EspadaV8

+1

+1

+1

+1

:+1:

:thumbsup:

+1

+1

If your looking for a package to generate slugs in Laravel 5.2 take a look one at this one I coded up last week: https://github.com/spatie/laravel-sluggable
Colin's package is more feature rich, mine does only the basics. It also has PHP 7 as a minimum requirement.

+1

+1

+1

+1

Try the 3.1.4 release now, please.

Thanks!

So far id does work! :D :+1:

Can you all also try the 4.0.0-beta release, which should work with L5.2.


cluster variable modification == unexpected status 401

I have deployed two times flynn with the same result:
$ flynn -c staging.cluster -a controller env get AUTH_KEY
b11f78a7755e4ad98ce04af263102b15

$ NEW_KEY=$(openssl rand -hex 16)
$ flynn -c staging.cluster -a controller env set -t web AUTH_KEY=$NEW_KEY,$(flynn -a controller env get AUTH_KEY)
unexpected close of deployment event stream: GET https://controller.cluster/events?app_id=1c74501c-8c6f-49c5-b169-f95e5cd9ad37&object_id=3468f732-fcbe-4142-a5ab-9e4e319697d4&object_types=deployment&past=true: httpclient: raw req: unexpected status 401

$ flynn -c staging.cluster -a controller env get AUTH_KEY
GET https://controller.cluster/apps/controller/release: httpclient: raw req: unexpected status 401

Neither the systemctl restart flynn-host.service  nor the restart of the instance solve the problem: the get AUTH_KEY always return 401
I would like to add one key for each developer that will work on the project, I have understood from documentation that this is how to do it, if not, please confirm me.

Your commands look correct, so I'm not sure why it isn't working. I just tried myself:
$ flynn -a controller env get AUTH_KEY
1234

$ NEW_KEY=$(openssl rand -hex 16)

$ flynn -a controller env set -t web AUTH_KEY=$NEW_KEY,$(flynn -a controller env get AUTH_KEY)
Created release a65f7b13-9d11-4b12-ba45-954ad70af864.

$ flynn -a controller env get AUTH_KEY
1234

$ flynn -a controller env get -t web AUTH_KEY
472b756f09d45f00a3119e61594e4e12,1234

Do you have a different default cluster configured? $(flynn -a controller env get AUTH_KEY) doesn't specify a cluster explicitly.


fix artifacthub annotations

Pull Request Description
Describe what this PR does / why we need it
Does this pull request fix one issue?

Describe how you did it
Describe how to verify it
Special notes for reviews

Codecov Report

Merging #247 (21f8d1c) into main (044aee8) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##             main     #247   +/-   ##
=======================================
  Coverage   27.41%   27.41%
=======================================
  Files          23       23
  Lines        1674     1674
=======================================
  Hits          459      459
  Misses       1163     1163
  Partials       52       52

Flag
Coverage Δ

unittests
27.41% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 044aee8...21f8d1c. Read the comment docs.


Bug Report: has no field SKIP_STEP_LIMIT and Missing type declaration

i think these both are just problems with kade engine in general

It is happened to me last time, If you are working with mac, try doing macos instead of windows on rebuilding extension-webm


question

answer made bit more logical and simple.

Dear reply+ANQ7EA3QWU4D7SYBFUALVF6DZD7KXEVBNHHH34KSAM,

Thank you so much for contacting qweq.
We have received your e-mail message and we have already create a support ticket for this issue.
An agent will follow up with you as soon as possible and he/she will assist you for this issue.

Please note: This e-mail was sent from a notification-only address that cannot accept incoming e-mail.
Please do not reply to this message.
=================================================
Your message to us:
answer made bit more logical and simple.

You can view, comment on, or merge this pull request online at:
  https://github.com/sudheerj/reactjs-interview-questions/pull/297
Commit Summary
2a29840  Question 1 answer changes...answer is made simple , short and to the point.
File Changes  (1 file)
M
README.md
(4)

Patch Links:
https://github.com/sudheerj/reactjs-interview-questions/pull/297.patch
https://github.com/sudheerj/reactjs-interview-questions/pull/297.diff
—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you are subscribed to this thread.Message ID: @.>
[
{
@.": "http://schema.org",
@.": "EmailMessage",
"potentialAction": {
@.": "ViewAction",
"target": "https://github.com/sudheerj/reactjs-interview-questions/pull/297",
"url": "https://github.com/sudheerj/reactjs-interview-questions/pull/297",
"name": "View Pull Request"
},
"description": "View this Pull Request on GitHub",
"publisher": {
@.***": "Organization",
"name": "GitHub",
"url": "https://github.com"
}
}
]

The current answer is quite good and the proposed answer should be improved grammatically.