issue with v6.5.0 (There is already an extension with this id error) on inputFactories

After upgrading to the latest version, it builds successfully, but throws error on the run time. The console says  (There is already an extension with this id error) on inputFactories.
it looks like the inputFactories got imported twice ?
appreciate any insights.
@kaisalmen
@CGNonofr

@OliverwengFiltered and @cdietrich thanks for reporting. We should add that this to our troubleshooting section.

Done: https://github.com/TypeFox/monaco-languageclient#vite-dev-server-troubleshooting

Had the same problem with vite cdietrich/my-monaco-editor-react-example@7f597fb

This problem is also happening in Nextjs, I'm not sure if Next has a dedupe feature yet
any Idea on how to tackle it there ?

I'm not sure it's required. it's required in the monaco-vscode-api demo because there is a local dependency
I think this is just a duplicated dependency issue here

@kaisalmen , the troubleshooting link only mention a recommendation for Vite. Anything recommendation for Webpack ?

I think this is just a duplicated dependency issue here

@CGNonofr is probably right. When I remove the dedupe instruction here, it just works, because dependencies are clear (=no mismatching versions)
@OliverwengFiltered the webpack verification example works also fine:
https://github.com/TypeFox/monaco-languageclient/tree/main/packages/verify/webpack
I will clarify the README.

I will clarify the README.

Done ✅

wonder where the mismatching versions came from when i just had a dependency to monaco-editor-react

<EMAIL_ADDRESS>peer
node_modules/vscode
  vscode@"npm<EMAIL_ADDRESS>from<EMAIL_ADDRESS>  node_modules/@codingame/monaco-vscode-configuration-service-override
<EMAIL_ADDRESS>from<EMAIL_ADDRESS>    node_modules/monaco-editor-wrapper
      peer<EMAIL_ADDRESS>from<EMAIL_ADDRESS>      node_modules/@typefox/monaco-editor-react
<EMAIL_ADDRESS>from the root project
  vscode@"npm<EMAIL_ADDRESS>from<EMAIL_ADDRESS>  node_modules/@codingame/monaco-vscode-languages-service-override
<EMAIL_ADDRESS>from<EMAIL_ADDRESS>    node_modules/monaco-languageclient
<EMAIL_ADDRESS>from<EMAIL_ADDRESS>      node_modules/monaco-editor-wrapper
        peer<EMAIL_ADDRESS>from<EMAIL_ADDRESS>        node_modules/@typefox/monaco-editor-react
<EMAIL_ADDRESS>from the root project
  vscode@"npm<EMAIL_ADDRESS>from<EMAIL_ADDRESS>  node_modules/@codingame/monaco-vscode-model-service-override
<EMAIL_ADDRESS>from<EMAIL_ADDRESS>    node_modules/monaco-languageclient
<EMAIL_ADDRESS>from<EMAIL_ADDRESS>      node_modules/monaco-editor-wrapper
        peer<EMAIL_ADDRESS>from<EMAIL_ADDRESS>        node_modules/@typefox/monaco-editor-react
<EMAIL_ADDRESS>from the root project

<EMAIL_ADDRESS>peer
node_modules/monaco-languageclient/node_modules/vscode
  vscode@"npm<EMAIL_ADDRESS>from<EMAIL_ADDRESS>  node_modules/monaco-languageclient
<EMAIL_ADDRESS>from<EMAIL_ADDRESS>    node_modules/monaco-editor-wrapper
      peer<EMAIL_ADDRESS>from<EMAIL_ADDRESS>      node_modules/@typefox/monaco-editor-react
<EMAIL_ADDRESS>from the root project

I can confirm the vscode version mismatch in<EMAIL_ADDRESS>

@OliverwengFiltered @cdietrich @rajialtooro and @CGNonofr it because of the missing ~ for vscode here:
"dependencies": {
    "monaco-editor": "~0.43.0",
    "vscode": "npm<EMAIL_ADDRESS>    "@codingame/monaco-vscode-languages-service-override": "~1.82.2",
    "@codingame/monaco-vscode-model-service-override": "~1.82.2",
    "vscode-jsonrpc": "~8.1.0",
    "vscode-languageclient": "~8.1.0"
  },

As far as I know the ~ is not excepted in the notation and it was not a problem before, because there was only one package. Does anyone know a valid npm: notation allowing the ~

@kaisalmen
according to chatGPT 😭
"vscode": "npm<EMAIL_ADDRESS><1.83.0"

I added it to my package.json

@rajialtooro that seems to work indeed:

@kaisalmen  so assuming it will need a release on the monaco-languageclient side to fix the version mismatch here?

Yes, exactly

New release is available:
https://www.npmjs.com/package/monaco-languageclient/v/6.5.1
Thank you all for getting to the bottom of this fairly quickly!

@OliverwengFiltered just checked the examples again incl. webpack verification and also with monaco-editor-wrapper. It works fine.

@OliverwengFiltered I'm facing the same issue with Nextjs, when I import anything from monaco-languageclient the editor component DOM stays blank,
when I tried to load it asynchronously like this
// in editor.tsx
import('./lsp').then(t =>
  t.performInit().catch(err => {
    console.log(err);
  })
);
...

// lsp.ts
import { initServices } from 'monaco-languageclient';

export const performInit = async () => {
  await initServices({ userServices: {}, debugLogging: true });
};

I get this weird error that not even the .catch is catching

@rajialtooro when i saw something like this in the past the problem was a bad polyfill for buffer. can you double check which version your use?

@cdietrich these are the related packages I have, keep in mind that with the same config version 5.0.1 was working
    "monaco-editor": "^0.43.0",
    "monaco-editor-webpack-plugin": "^7.1.0",
    "monaco-languageclient": "^6.5.1",
    "next": "^13.2.4",

@rajialtooro the question is what else may pull "buffer" and if yes in which version

@cdietrich I ran npm list buffer, though I don't see any package from monaco-languageclient that uses it

i had to resolve buffer to a newer version in package.json
"resolutions": {
   ...
    "buffer": "^6.0.3",
    ...
  }

@cdietrich Worked like a charm, thank you 🥳

@kaisalmen might this be something for the troubleshooting section? this is including mine the 3rd occurrance of this problem?

@cdietrich good suggestion, see here

thx for the quick fixes @kaisalmen

I have this error (Assertion failed (There is already an extension with this id)) also in 7.3.0 (react.js, webpack) whenI tempororay commented out MonacoWebpackPlugin
because with it I have error on webpacking -  Error: Cannot find module 'vs/editor/standalone/browser/quickAccess/standaloneGotoLineQuickAccess'
is there any workaround to fix this errors?

You can still use MonacoWebpackPlugin, you just need to disable the features. I don't recommend using it though
@kaisalmen maybe we should go back to monaco-editor-core as we don't use any of the features added by monaco-editor on top of the core

@CGNonofr thanks for quick answer. But I don't know, what feature I need to disable?
this is my imports
import React, { createRef, useEffect, useRef, useState } from 'react';
import { setDiagnosticsOptions } from 'monaco-yaml';
import "monaco-editor/esm/vs/basic-languages/yaml/yaml";
import {useEventListener} from "usehooks-ts";

import { useDarkMode } from 'usehooks-ts';
import { v4 as uuidv4 } from 'uuid';

import "./monacoEditor.scss";
import { createUrl, createWebSocket, removeOrAddComment } from './MonacoHelper';
import { initServices } from 'monaco-languageclient';
import { createConfiguredEditor, createModelReference } from 'vscode/monaco';
import getServiceOverride from '@codingame/monaco-vscode-base-service-override';
import { editor, languages, Uri, KeyMod, KeyCode } from 'monaco-editor/esm/vs/editor/editor.api.js';
import '@codingame/monaco-vscode-theme-defaults-default-extension';
//import 'vscode/default-extensions/lua'; //dont know yet what should replace it
//import 'vscode/default-extensions/css';
//import 'vscode/default-extensions/yaml';
import getConfigurationServiceOverride from '@codingame/monaco-vscode-configuration-service-override';
import getKeybindingsServiceOverride from '@codingame/monaco-vscode-keybindings-service-override';
import getThemeServiceOverride from '@codingame/monaco-vscode-theme-service-override';
import getTextmateServiceOverride from '@codingame/monaco-vscode-textmate-service-override';
import { updateUserConfiguration } from '@codingame/monaco-vscode-configuration-service-override';

see https://github.com/CodinGame/monaco-vscode-api/issues/219

@CGNonofr thanks it helps to webpack problem but I still have
Uncaught Error: Assertion failed (There is already an extension with this id)
when I try start monaco editor

It seems that this is minimal repo ...
import React, { createRef, useEffect, useRef, useState } from 'react';
import { initServices } from 'monaco-languageclient';

export default function MonacoEditor(props) {

	return (
			<React.Fragment>

			</React.Fragment>
			);
}
;

then I got it:

without import { initServices } from 'monaco-languageclient'; there is no error

@kaisalmen maybe we should go back to monaco-editor-core as we don't use any of the features added by monaco-editor on top of the core

Isn't this default what @codingame/monaco-editor-treemended loads if you just import it? Or do we need to change it back to only import it like this  import * as monaco from 'monaco-editor/esm/vs/editor/edcore.main.js'; (editor-core)? Honestly, you confused me a bit by that question.

@CGNonofr Ok, so whatever we have it is not compatible with regular monaco-editor tooling in the end or needs proper customization (see https://github.com/CodinGame/monaco-vscode-api/issues/219).
Wouldn't it then make sense to fully rely/rename (also for all your packages) on a self-standing npm package e.g. called monaco-vscode-api-editor that can be used in combination with overrides / resolutions to override monaco-editor. It is basically what is done here already.
This way people realize the have something else and could force it in the dependency chain instead of monaco-editor. Also, then people no longer need such tooling (MonacoWebpackPlugin), right? Does it make sense? WDYT?

let's discuss it on https://github.com/CodinGame/monaco-vscode-api/discussions/325

What bundler do you use? - I use webpack
with what configuration? - this is webpack.config.js :

var path = require('path');
const MonacoWebpackPlugin = require('monaco-editor-webpack-plugin');
module.exports = {
	entry: './src/index.js',
	mode: 'production',
	output: {
		path: path.resolve(__dirname, 'build'),
		filename: 'index.js',
		libraryTarget: 'commonjs2'
	},

	module: {
		rules: [
			{
				test: /\.js$/,
				include: path.resolve(__dirname, 'src'),
				exclude: /(node_modules|bower_components|build)/,
				use: {
					loader: 'babel-loader',
					options: {
						presets: ["@babel/preset-env", "@babel/preset-react"]
					}
				}
			},
			{test: /\.css$/, use: ['style-loader', 'css-loader']},
			{test: /\.scss$/, use: ['style-loader', 'css-loader', 'sass-loader']},
			{
				test: /\.(woff|woff2|eot|ttf|otf)$/i,
				type: 'asset/resource',
			}
		]
	},
	plugins: [new MonacoWebpackPlugin({
		features: ['!gotoLine', '!inspectTokens', '!quickCommand', '!quickHelp', '!quickOutline', '!referenceSearch', '!toggleHighContrast'],
		publicPath: "resources/react/dist/",
		languages: ['yaml'],
		customLanguages: [
			{
				label: 'yaml',
				entry: 'monaco-yaml',
				worker: {
					id: 'monaco-yaml/yamlWorker',
					entry: 'monaco-yaml/yaml.worker'
				}
			}
		]
	})],
	externals: {
		'react': 'commonjs react',
		"axios": "axios",
		"js-cookie": "js-cookie",
		"react-singleton-hook": "react-singleton-hook",
		"@chiragrupani/fullscreen-react": "@chiragrupani/fullscreen-react",
	}
};

what versions of the libs do you use?
"@codingame/monaco-vscode-configuration-service-override": "1.85.0",
		"@codingame/monaco-vscode-keybindings-service-override": "1.85.0",
		"@codingame/monaco-vscode-textmate-service-override": "1.85.0",
		"@codingame/monaco-vscode-theme-defaults-default-extension": "1.85.0",
		"@codingame/monaco-vscode-theme-service-override": "1.85.0",
"monaco-editor": "0.45.0",
		"monaco-editor-webpack-plugin": "^7.1.0",
		"monaco-languageclient": "7.3.0",
		"monaco-yaml": "^4.0.4",
		"vscode": "npm<EMAIL_ADDRESS>		```
- did you try what was discussed previously about buffer? - **No because I've thought its about another issue.
Now I've just added**

"resolutions": {
"buffer": "^6.0.3"
},
```
in package.json but it don't help me. The same error.

are you sure there is no duplicated versions of some libs?
I'm not sure. I've run npm list vscode and I get

so I've change 1.85.0 to 1.85.5 and now import { initServices } from 'monaco-languageclient'; doesn't generating error!
Thank you very much!
After restoring all the code, I get the error Error: Services are already initialized

but maybe its some usage thing or I will create new issue. Thanks again.


OS server not able to tell client IPs

All IPs look like <IP_ADDRESS> to the server.

You need this?

Yes, we do. For analytics and administration (specifically, potentially required IP bans which jbayfield already has had to do for gwiddle services)

Potential solution: Use X-Forwarded-For which I think nginx sets.

This isn't really an issue anymore.


Shrink the README file, and create a project Wiki

A lengthy README is far less informative than a project Wiki, where each functionality's description can have their internals, example use, and arguments specified.

Also, add dependency installation steps for other platforms (Debian-Ubuntu-Mint/Fedora-CentOS-RedHat/Void).


Fix 3192

@sbrunner Please let me know when you add a moment to ensure if something was missing from https://github.com/mapfish/mapfish-print/pull/3241/commits/7aa53e4a5fd770f376cd6c4678c3be3106a05ca7


Feature Request: Provide mesh config programmatically via an API

Hello wonderful team. I have the following idea for a feature which I believe would increase the usability of graphql-mesh.
Description
In the following I would refer to using graphql-mesh not as a standalone server but as part of an existing server / existing codebase. The idea is that graphql-mesh exposes an API or an option inside of findAndParseConfig() (not 100% sure about it) function to override or simply use configuration which is a javascript object, in which all individual path references are gone but instead replaced with real functions that are programmatically imported in whatever codebase is using graphql-mesh.
In other words:

Following the inversion of control principle, I would like to supply to graphql-mesh resolver functions from my project's ecosystem, rather than letting graphql-mesh load / import those on its own.

Example use case
My team and I would really like to use graphql-mesh inside of our existing server as anything else is not an option for us at the moment. However, our existing server is using specific webpack development and production configurations. At the end, we deploy a simple bundle.js which is a ready-made executable for our server. If we were to use graphql-mesh we would need to maintain a separate configuration system and then somehow inject the additional-resolvers files, for example, next to our bundle.js since graphql-mesh is loading those internally. It would be nice to have the option to provide graphql-mesh directly all resolvers instead of letting it import them on it's own.
Further, use case is using development tools such as TS/ FLow/ Jest, Babel, etc.
Proposed solution
Looking at your code I found out that inside findAndParseConfig() you are building couple of key-value-maps (objects) which are a result of importing / loading modules. So I propose that these maps are simply built by an options parameter which is passed somewhere in findAndParseConfig(). I am not sure about the exact details. I could be wrong but I guess you already get what I am trying to achieve. I would also try looking at the source for the exact place where something like this can be implemented.
Example for how it could look like without breaking API
const resolversObj = {
  // ...
}

const queryComposerResolverFn = (next) => (...args) => // ...

const meshConfig = {
  additionalResolvers: resolversObj,
  transforms: {
    resolversComposition: {
      resolver: 'Query.*',
      composer: queryComposerResolverFn
    }
  }
}

findAndParseConfig({
  customConfig: meshConfig
})

I would be happy to hear your feedback about such a feature

This is a awesome feature and i need it.

You can use getMesh like this. It's not documented yet but if you're using TypeScript, it will be easy for you to figure it out.

That's what I am thinking of. We should refactor all handlers and transforms to move external module loading logic outside of them. I am not sure about supporting functions as part of initial config because the schema of configuration is generated from GraphQL. But I'll figure it out soon and keep you posted.


Readme typo

At least on windows, command from the guide doesn't work without extension

Thank you my friend,
fixed :)


Resolving doc issues

Resolved  https://github.com/pyansys/pyfluent/issues/1484

----------------------------------------------------------------------------

Resolved  https://github.com/pyansys/pyfluent/issues/1465

----------------------------------------------------------------------------

Resolved  https://github.com/pyansys/pyfluent/issues/1457

----------------------------------------------------------------------------

Resolved  https://github.com/pyansys/pyfluent/issues/1453

----------------------------------------------------------------------------

Resolved  https://github.com/pyansys/pyfluent/issues/1456

----------------------------------------------------------------------------

Resolved  https://github.com/pyansys/pyfluent/issues/1460

----------------------------------------------------------------------------

Resolved  https://github.com/pyansys/pyfluent/issues/1459

We have resolved regression related to navigation issue.

Corrected -

@seanpearsonuk
We have added and restructured .rst files to resolve https://github.com/pyansys/pyfluent/issues/1459.

@hpohekar I think you will have conflicts. case.py is now case_file.py. Case is CaseFile.

@hpohekar I think you will have conflicts. case.py is now case_file.py. Case is CaseFile.

@seanpearsonuk
We have resolved it.

@hpohekar I think you will have conflicts. case.py is now case_file.py. Case is CaseFile.

@seanpearsonuk
We have resolved it.

Wherever we mention case reader, we could mention instead CaseFile class. @hpohekar

@hpohekar Some follow-up comments (sorry, I was still going through the updated docs)

Wherever we mention case reader, we could mention instead CaseFile class.
It would be useful if we can click on FluentConnection from <Session_class>.__init__(fluent_connection : FluentConnection)
This line "PRESENTLY SAME AS MESHING WITHOUT THE SWITCH TO SOLVER*" in session_pure_meshing is not needed (at least not in this format)

I will create a new issue to cover this work.

No issues at all. We will try to resolve this as soon as possible. Thank you.


Change in latest nightly breaks type inference (phantom type?)

src/sdl2/audio.rs:320:25: 320:62 error: unable to infer enough type information to locate the impl of the trait `audio::AudioFormatNum<T>` for the type `_`; type annotations required
src/sdl2/audio.rs:320                 format: AudioFormatNum::<T>::get_audio_format(),
                                              ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
src/sdl2/audio.rs:320:25: 320:62 note: required by `audio::AudioFormatNum::get_audio_format`
src/sdl2/audio.rs:320                 format: AudioFormatNum::<T>::get_audio_format(),
                                              ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

I would have fixed this myself but it looks like it might need a change to the API to get the proper type information so I figured it merited some discussion first.

How can this be fixed? Does Rust allow you to call the trait for Self yet?

How can this be fixed? Does Rust allow you to call the trait for Self yet?

I think this can be fixed by adding an empty struct with generic parameter AudioFormatFor<T> and then pass this as parameter to the get_audio_format. A method can be impled on AudioFormatFor to make this nicer.

I think this can be fixed by adding an empty struct with generic parameter AudioFormatFor<T> and then pass this as parameter to the get_audio_format. A method can be impled on AudioFormatFor to make this nicer.

I have an idea that doesn't require a breaking change. Let me hack on it for a bit.

I have an idea that doesn't require a breaking change. Let me hack on it for a bit.

@bvssvni @AngryLawyer Opened #252.

@bvssvni @AngryLawyer Opened #252.


C# windows form

Type of issue
Typo
Description
[Enter feedback here]
Page URL
https://learn.microsoft.com/en-us/dotnet/csharp/language-reference/language-specification/introduction?redirectedfrom=MSDN
Content source URL
https://github.com/dotnet/csharpstandard/blob/draft-v8/standard/introduction.md

At the moment, there's no clear information in this issue. Please add more details, or I'll just close this issue.


New data model

new data model fitted for when messages can come from several receivers
payload stored as Vec<u8>
also ~ is now accepted in path for configuration file and output file
--expire becomes --history-expire
messages are decoded in their async task to benefit from potential multi-thread/core
new option --stats true to explore statistics

@emctoo would you have a quick look?
(it actually reveals some performance issues, but that's for later)

According to this graph, the decoding time if an order of magnitude faster than the throughput time. Yet there seems to be some drift in the reception of messages (i.e. system_timestamp - gnss_timestamp) when we decode many receivers.
It is a bit hard to investigate right now (maybe to be transformed into an issue)


Leak? on MSI for mobo lights

There is a leak i guess on msi website. There you can download Mystic Light Api sdk:
http://download.msi.com/uti_exe/Mystic_Light_2.zip
Maybe they can replace that link, then i can upload to GDrive. I didn't do that for now bec it's not of course safe to trust 3rd party guy for that (me) but i can only upload header and dll files with documentation pdf for be safe. That support would be awesome.

I looked through it and it doesn’t really have anything useful there, that’s not available by other means.
Thanks anyway!


Fix the index out of range bug.

Referenced here:
https://github.com/eResearchSA/reporting/issues/47

Fixed.


Testing with docker swarm

Please test with docker swarm. Take into consideration this issue (i.e. recreation of the containers).

The docker has been tested inside docker-swarm of NCSR-D's bde-cluster and works fine.


Change Css Priority File Order + Add PageCss Priority #3578

https://github.com/dnnsoftware/Dnn.Platform/issues/3578
Summary

2 files changed
FileOrder.cs
modifies the DefaultPriority for all other CSS to 90
Sets PortalCss priority to 95
Creates and Sets PageCss priority to 100
Default.aspx.cs
Moves CustomStylesheet (Page.css) to load after all other stylesheets both programmatically in page source and using the new stylesheet file load order PageCss  priority.

@thabaum

The PR includes a lot of unrelated changes, can you please rebase this
I would really like to have a step by step of what css styling is not possible to do now so that we can review if we as a community do want to change this, it has a potential to be a huge breaking change. We try to avoid breaking changes as much as possible unless they are really required. Load order is only one thing considered for css rules, usually the best way to style something without causing a breaking change is to just be a little more specific thatn the rule to override. Also there is a default.css cleanup coming up for 10 which maybe can solve the issue at hand without it being a breaking change...

I was trying to push this to version 10.  What am I suppose to base this on to do so?  I was not going to put in 9 as it would be breaking possibly.
try to change logo size using Page.css without

create  a page
upload a pagename.css to the root site directory
include the following in the pagename.css file

.navbar .brand {max-width:50px;}

Add pagename.css as Page Stylesheet in PersonaBar Page Settings editor
Refresh the page notice nothing happens
put the same code in Portal.css and it works however other CSS stylesheets load after Portal.css taking priority

Pretty much only way to style things is to use !important a lot.
But then you find a problem when !important is used as there are times I cannot edit styles of modules
There is NO end CSS file a DNN administrator can use that makes sense to ensure the CSS is loaded last and takes priority over other css.  I think only people that have used Community Edition a lot for projects notice this issue but it is surely an issue.
Let me know what to base this PR on I will redo it as I don't know how to rebase quickly...
As discussed in my version 9 PR this is a version 10 solution so I rebased to 10.   Now what?

Curious, is EVOQ default.aspx file different with a change similar to what I have made or does it act the same? It would show the neglect by corporate to the community IMO if it did as this should of been fixed for everyone.  And if not they should toss me a bone ;)

The PR includes a lot of unrelated changes, can you please rebase this
I would really like to have a step by step of what css styling is not possible to do now so that we can review if we as a community do want to change this, it has a potential to be a huge breaking change. We try to avoid breaking changes as much as possible unless they are really required. Load order is only one thing considered for css rules, usually the best way to style something without causing a breaking change is to just be a little more specific thatn the rule to override. Also there is a default.css cleanup coming up for 10 which maybe can solve the issue at hand without it being a breaking change...

Agreed 100%

Ok, for the unrelated changes, you had it right but the PR was submitted targetting the develop branch. There was a branch reorganization and develop is the branch to use for most things that will make it into the very next release like 9.5.1 or 9.6.0 depending on what comes in. For the potential breaking changes or major new stuff, the branch is future/10 for version 10. After changing that target we now have only 2 changed files which is good.
Now for the actual css, !important is really a last resort thing and changing file load order is just the last condition css uses to define what wins, just being a tiny little bit more specific will make your rule win over a competing one. For instance, using your example, you could just do:
header .navbar .brand {max-width:50px;}

And this rule would win. Load order only affects what wins if the competing rules have exactly the same specificity. I am not saying by any means that the current load order is right or wrong, just that it is not a major problem and changing how it is currently would cause a breaking change on any unkown cases without a big need for it. So certainly worth discussing to see if we want to get into that. It is now clearer with the PR fix to see the changes involved...

OK, but I have been submitting issues for 10 years... should i dig them up?  I think some are on Jira

I thought I had based this on future/10
This what was recommended my last PR which I targeted develop 9.x... so I did a new one as Mitch and Brian recommended, AND added the fix for putting Page.css to load last as it should as requested by Sebastian.
So not sure what to do here I am getting bounced around like a ping pong ball as far as where to base it.
This load order needs fixed IMO badly.  I will fight for it and others notice the issue as well.
Why load Page.css before default.css?  It currently loads first when it should load last as my directions if you follow point out.  Is this correct?

One last note before i got to go.  This issue i would be creating many issues for.... over and over and wait another 10 years.  OR it was much faster although it took some days of studying the code source to resolve this, much less time than banging my head trying to get some simple stupid CSS style to take effect.

from my experience, there is an issue with loading sequence of CSS files, which doesn't follow documented order. There are options as well for skins and modules to override the documented (i.e. intended) order of css files (e.g. using includes), which interfere with combined css files, i.e. we never use this option at our clients (while the option of including files is helpful in complex scenarios with hierarical skin files supporting multiple variations, splitting up the skin into multiple hierarchical include files for headers, logos, login popups, link menues, footer lists etc..

the original ideal of combining files in DNN CDF system was reduction of files to be transferred, due to HTTP 1.x limitations. IMO we need to investigate, whether this still makes sense in HTTP 2.x, which supports a larger number of parallel downloads. This might improve performance using a larger number of smaller files. especially, if some of the files are alread transferred by previos page. We need to investigate, whether combining CSS (and JS files) might have become obsolete from a performance perspective.

I feel what you are talking about Sebastian.  I would love any solution as well if it increases performance and things are functioning as intended.  I can rebase this to 9 for a fix and maybe look for something new for 10?  Just let me know I am happy to help as I can.  There are a couple of ways of handling things here.  This just put things in logical order for my mind and what is currently being used in DNN 9.  I had even thought of removing PortalCss priority and use DefaultPriority for both Portal and page Custom stylesheets and set them up at the end of the page in order after any other CSS is called upon.   I am just trying to tune things in a bit here to make things work as expected, nothing for performance as DNN 11 should have many things changed by then.
If the current way stays and something like this PR works ok I might even suggest adding a Module Custom Stylesheet that can get set similar to page.  Upload a css file and set it in the module settings to use custom CSS and be the very last to load after Page.css.
I liked that idea Sebastian also mentioned. And I also like the idea of being able to setup priorities in the database for all CSS files so you can set it to the old ways, use the new way OR create your own way for CSS or JS files to load.  A bit beyond the scope of what I was address as that is adding new features when I am trying to "Fix" what I consider a bug.
Maybe this is obsolete that I am working  on?  As long as I know something is in the works to make this stuff load in order I expect and everyone talks it should load in I am good to go.
Currently it is the wild west as far as knowing what will load last.

What kind of change would anyone be in support of if any?  Skin.css and Portal.css loads after Page.css, can we at least put Page.css set to default priority or have it load after all other CSS files load in default priority?
Portal.css I still believe should load after default priority.  Portal.css does currently and still will load after Skin css container css and module css files.  This order DOES NOT change in my PR nor am I proposing the fundamental order change.  So please let me know you understand this.  Thank you.

Portal after Default = would mean that you cannot override portal, which would be the reverse issue of your current issue.
If anything, can you attend a future TAG meeting to discuss this?

So you wish to override portal.css?  Something that is written by an administrator to make a portal have styles?
I can try to attend a meeting and schedule it in if it can help here.
Page.css should still load after portal.css.  DefaultPriority overrides Portal.css by design I take it.  So the fact is Portal.css does not load last as everyone is under the impression it does.  And page.css loads before skin.css.
Those are my two findings which came up from an added discussion with Sebastien when he brought up the page.css load order and I looked all over thinking it was default priority, just to find it with no priority loading above other css includes.

Page.css should still load after portal.css

I don't disagree and would be totally fine with this if we where just building something new, but we need to consider upgrades and 3rd parties that are used to this order. Also overriding a rule should not be done only by the file load order, and it should not be done with !important, it should be done by providing a more specific rule, that will win no matter the file load order.

And if anything here we can at least clean up the documentation as needed so we can all understand this load order.

@thabaum Can you provide a link to which documentation is wrong, I think this would be where to fix this other than just creating a breaking change. If documentation about load order is wrong, we need to fix that documentation. If people just don't understand CSS, unfortunately documenting that is out of our scope... Load order is really the last thing to consider for overriding styles, see https://www.w3.org/TR/CSS2/cascade.html

Like @david-poindexter I am against this change. It would not fix all cases (only rules with same specificity), it would be a breaking change (swapping all existing 3rd party rules with same specificity), and it would encourage CSS bad practices (people will incorrectly fix the breaking change with !important and be under the wrong path thinking load order is more important than specificity). The correct way to not get into troubles is to know how css specificity works and understanding the Dnn load order that never changed as far as I know).
The correct best practice is to scope themes with a wrapper id or class, scope all theme rules to it. Same for modules, wrap the module with a class (not an id because there could be more than one on the page) and scope the module specific rules to that class. No conflicts, you know what wins and no !important needed anywhere. Like @mitchelsellers said on the issue, the admin accessible css UI options are a last resort and I guess if all else fails, this would be the only place where !important could be used as a very last resort.

This what was recommended my last PR which I targeted develop 9.x... so I did a new one as Mitch and Brian recommended, AND added the fix for putting Page.css to load last as it should as requested by Sebastian. So not sure what to do here I am getting bounced around like a ping pong ball as far as where to base it.

Sorry for that and I understand the frustration, some of us only manage to get time to reply a few days later than issues or PRs are submitted. Also many of us where super busy due to Dnn Summit. I for one am just catching up to put my comments now :) And well different people will have different opinions about changes, I for one am against breaking changes where they are not absolutely needed. You will get different comments and opinions for sure and when there are diverging opinions we need to take it into a TAG meeting to take a final decision. For sure at a minimum if the consensus ends up being to take it in, the move to Dnn10 is still valid at the least.

As per todays TAG approvers meeting, we looked at a couple of module vendors and according to the usages found, this would be a breaking change for many 3rd parties, for that reason and because there are better ways to do this properly in CSS, we are closing this PR. We think this is an unnecessary breaking change that would not even solve the issue in all cases.

Could this be added to DNN Platform Version 11 possibly?
If not should we document to use !important in css for page and portal css to take priority over all other css files that are loaded.
A module or skin loads top priority overriding Portal.css and Page.css which everyone expected before to be the last files to load.  Also if a developer loads a css file without declaring a load priority it takes priority over all others.  Instead they actually fight with all other modules and skins that have css files loading with default priority so you never get a true css file load order, each page generally varies depending on which modules or skins are loaded and how the developer declared the css files.  That module or skin then overrides Page.css and Portal.css which should load after default priority.   So for a developer you get best results for what you are trying to do however if someone tries to modify after getting a vendor module you generally MUST use !important in order for any CSS changes to take affect.

Could this be added to DNN Platform Version 11 possibly?
My vote would be no, we do not want to intruduce unnecessary breaking changes no matter the version, if there is wrong documentation about how it works, we should correct the documentation, not introduce a breaking change that is not required.

If not should we document to use !important in css for page and portal css to take priority over all other css files that are loaded.
It cannot be on our responsability to document html/css/javascript, there are core web technologies that are not Dnn specific. Also !important is to avoid as much as possible, instead rules should be as specific as they need to be in order to do their job. Themes and Modules should use an enclosing scope so their css does not leak out of their intent and are still able to override more general rules. It is really rare that !important is required and many css frameworks make extensive use of it, which I hate.

To prepare for the big change everyone can simply put !important on everything in the Portal.css and Page.css files to prepare for the switch.
Hmm, no, big no, this would be such an anti-pattern and would not replicate just a load order change!!! !important breaks css cascading, it is to avoid at all costs unless there is absolutely no other way.


feat(etl): add GPHL sequencing PDF ETL script

This adds the GPHL sequencing ETL script from cape-ph/etl-gphl-sequencing-alert into the infrastructure.
To Test

Grab the 240717-CPO-seq-report-MOCK.pdf file from the test data folder on Box (under gphl-sequencing subfolder)
Upload it to the HAI raw bucket with the prefix of gphl-sequencing/
Check the Glue Jobs to see that it run and was successful
Check the clean bucket to see the generated output
Can also check the glue crawler after it's next scheduled crawl to see that it was crawled and indexed into the data catalog

I have reviewed the parts @mehalter wrote (and tested deployment). he will need to comment on the changes i made

@thecaffiend looks good to me! Feel free to approve it!


Spotify Module Stopped working

I've had the companion module setup and running for a few days now and it has worked flawlessly.  It stopped working this afternoon and I can't get it working again.  I have reinstalled companion, re-auth Spotify, and deleted and reinstalled the module. Any ides or suggestions on how to get it working again

?

Is the active device ID set correctly?
I am still using the module with no issues.

Ahh, yes, I reset active device ID and it is working again. Thanks!


Git url is not shown in npmjs.com

Hi,
I noticed that npmjs is not providing a link to the github page: https://www.npmjs.com/package/serverless-pseudo-parameters
I think the definition you have in package.json is wrong and npmjs isn't picking it up. Maybe try the following definition:
"repository": {
    "type": "git",
    "url": "https://github.com/svdgraaf/serverless-pseudo-parameters"
  },

Thank you, this is now fixed :)


LED switching in MIDI example

@jacoburge it would be awesome if the MIDI example could also show how to trigger the LED's
Where can I find the normal firmware? I only see the examples here - there I could also look how the LED's are triggered

LEDs are handled automatically with button presses so this is already taken care of.  Currently custom control of LEDs is not supported.


Fix cloned user display name

Rationale
New ClonedUser constructor was confused about which parameter was which, causing
org.labkey.inventory.query.InventoryBoxTable$TriggerTest to fail and likely other problems
Related Pull Requests

https://github.com/LabKey/platform/pull/5203

Always good to lead with friendly

Is it time to add getPreferredPronouns()?


Add a timeout to DNS seeder requests to avoid hangs

Is your feature request related to a problem? Please describe.
When zebrad launches, it asks some DNS seeders for a list of initial peers. But if DNS or the seeder is slow or down, zebrad can hang without any output.
We've observed these hangs in our CI, but we don't know if it's just one seeder that's having issues. If it is, this fix could make our CI more reliable.
Describe the solution you'd like

[ ] in parse_peers, execute each to_socket_addrs call in a separate spawn_blocking tokio task, wrap each task in a timeout, and wait on them completing
[ ] log a warning if any of the tasks time out
[ ] add a timeout argument to parse_peers and initial_peers
[ ] in add_initial_peers, log a warning if there are no initial peers
[ ] check if we still need ZEBRAD_SKIP_NETWORK_TESTS on Windows and Linux CI - this fix might help with their unreliable network behaviour

We don't panic on any of these errors, because they are recoverable - zebrad can discover peers from inbound connections, if its address is distributed by a DNS seeder (or manually).
Describe alternatives you've considered
Do nothing - zebrad will hang if there are DNS issues
Panic - these errors are recoverable, and "no initial peers" is a valid config
Additional context
We discovered this issue while troubleshooting bind hangs in #1535

Tentatively adding this task to Sprint 2, because:

we see startup hangs in CI on some platforms - this issue will probably fix them (and definitively give us better diagnostics)
the fix might allow us to run our full CI on more platforms, which is important, because we've discovered some platform-specific hangs recently
a hang at startup with no output is a very negative user experience

Tentatively adding this task to Sprint 2, because:

we see startup hangs in CI on some platforms - this issue will probably fix them (and definitively give us better diagnostics)
the fix might allow us to run our full CI on more platforms, which is important, because we've discovered some platform-specific hangs recently
a hang at startup with no output is a very negative user experience

I increased the estimate here, because we've added some possible acceptance tests changes.

I increased the estimate here, because we've added some possible acceptance tests changes.


Feature Request: Markdown export

The option to export conversations to markdown format would be great

Another vote for markdown format export.
It should be built in, but for a model of how the UI and export format might look, check out pionxzh/chatgpt-exporter: Export and Share your ChatGPT conversation history.  It's a Tampermonkey userscript that installs an export button on the OpenAI ChatGTP web interface.

I've submitted PR #684, which adds an "Export Markdown" button that exports conversations in Markdown in a zip file.

@mckaywrigley I just did a pull of this commit https://github.com/mckaywrigley/chatbot-ui/commit/369087057e0d90a38188ef06a6af0d3b685a81ee and started the app, not seeing the option to export to markdown? Any ideas? Thanks!


Add CLI options to filter usable extractors

Before submitting a pull request make sure you have:

[x] At least skimmed through adding new extractor tutorial and youtube-dl coding conventions sections
[x] Searched the bugtracker for similar pull requests

What is the purpose of your pull request?

[ ] Bug fix
[ ] New extractor
[x] New feature

Description of your pull request and other information
Add --enable-extractors and --disable-extractors options, which make
it possible to restrict the set of extractors to be considered when
downloading. This is useful to handle URLs that match multiple
extractors (although this should be rare), or only using particular
modes of some extractors (for example, only live videos for Twitch,
enabling only twitch:stream).
Both options can be specified multiple times, and each argument is
interpreted as a comma-separated list of fnmatch patterns, to allow the
use of wildcards. Comparisons to extractor names are case-insensitive.
The order of the arguments is not relevant - matching always proceeds as
follows:

Initialize the set of considered extractors to all available
If --enable-extractors is specified, remove all extractors that
don't match those patterns from consideration
If --disable-extractors is specified, remove all extractors that do
match those patterns from consideration
If --age-limit is specified, remove all extractors that are not
suitable from consideration

Therefore, disables and the age limit take precedende over enables.

This won't work in general since many extractors cross delegate extraction to another extractors.
For example, I only want to be able to download from AlJazeera thus enable it:
> py -2.7 .\youtube_dl\__main__.py http://www.aljazeera.com/programmes/the-slum/2014/08/deliverance-201482883754237240.html --enable-extractors aljazeera
[AlJazeera] deliverance-201482883754237240: Downloading webpage
[brightcove:new]<PHONE_NUMBER>001: Downloading webpage
[brightcove:new]<PHONE_NUMBER>001: Downloading JSON metadata
[brightcove:new]<PHONE_NUMBER>001: Downloading m3u8 information
[download] Destination: The Slum - Episode 1 - Deliverance-3792260579001.mp4
[download]   0.1% of 1.71GiB at  1.53MiB/s ETA 19:07
ERROR: Interrupted by user

It does work even though it delegates to brightcove.
Then I want to download from Discovery and enable it:
> py -2.7 .\youtube_dl\__main__.py http://www.discovery.com/tv-shows/mythbusters/videos/mission-impossible-outtakes/ -F --enable-extractors discovery
[Discovery] mission-impossible-outtakes: Downloading JSON metadata
[download] Downloading playlist: Mission Impossible Outtakes
[Discovery] playlist Mission Impossible Outtakes: Collected 1 video ids (downloading 1 of them)
[download] Downloading video 1 of 1
ERROR: no suitable InfoExtractor for URL http://players.brightcove.net/103207/default_default/index.html?videoId=ref:585f8ee5a885fcaa59a868f970721423d68dbe2f

But it does not work despite it uses the same brightcove as embed but via playlist indirection.
In this case I'm forced to enable all the extractors that Discovery may use in similar way. This is misleading and requires me as end user to know the implementation details of the particular extractor in some cases but does not in others.

That will never be fixable without some explicit metadata for dependencies,
and adding it would be a much larger undertaking. As-is this already
fulfills relevant use cases for many extractors without such "hidden"
requirements.
Em dom, 18 de set de 2016 08:28, Sergey M<EMAIL_ADDRESS>escreveu:

This won't work in general since many extractors cross delegate extraction
to another extractors.
For example, I only want to be able to download from AlJazeera thus enable
it:

py -2.7 .\youtube_dl_main_.py http://www.aljazeera.com/programmes/the-slum/2014/08/deliverance-201482883754237240.html --enable-extractors aljazeera
[AlJazeera] deliverance-201482883754237240: Downloading webpage
[brightcove:new]<PHONE_NUMBER>001: Downloading webpage
[brightcove:new]<PHONE_NUMBER>001: Downloading JSON metadata
[brightcove:new]<PHONE_NUMBER>001: Downloading m3u8 information
[download] Destination: The Slum - Episode 1 - Deliverance-3792260579001.mp4
[download]   0.1% of 1.71GiB at  1.53MiB/s ETA 19:07
ERROR: Interrupted by user

It does work even though it delegates to brightcove.
Then I want to download from Discovery and enable it:

py -2.7 .\youtube_dl_main_.py http://www.discovery.com/tv-shows/mythbusters/videos/mission-impossible-outtakes/ -F --enable-extractors discovery
[Discovery] mission-impossible-outtakes: Downloading JSON metadata
[download] Downloading playlist: Mission Impossible Outtakes
[Discovery] playlist Mission Impossible Outtakes: Collected 1 video ids (downloading 1 of them)
[download] Downloading video 1 of 1
ERROR: no suitable InfoExtractor for URL http://players.brightcove.net/103207/default_default/index.html?videoId=ref:585f8ee5a885fcaa59a868f970721423d68dbe2f

But it does not work despite it uses the same brightcove as embed but via
playlist indirection.
In this case I'm forced to enable all the extractors that Discovery may
use in similar way. This is misleading and requires me as end user to know
the implementation details of the particular extractor in some cases but
does not in others.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/rg3/youtube-dl/pull/10691#issuecomment-247842328, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAtnTowvFDgpn17-anCgnHAcPoQR-Tj6ks5qrSB1gaJpZM4J_y21
.

That will never be fixable without some explicit metadata for dependencies

That's why disabling or enabling extractors does not make sense.

Of course it does. My use case is only wanting to download live Twitch
streams. There are others, such as restricting the set of desirable
extractors for redirects, avoiding slow extractors that try to fetch large
amounts of metadata (of which Twitch is another example), or disabling
extractors that are broken for reasons external to yt-dl (such as a
corporate proxy).
Em dom, 18 de set de 2016 08:46, Sergey M<EMAIL_ADDRESS>escreveu:

That will never be fixable without some explicit metadata for dependencies
That's why disabling or enabling extractors does not make sense.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/rg3/youtube-dl/pull/10691#issuecomment-247843082, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAtnThFm7YOMQwfn8ZJkBw4jfkwpGMIEks5qrSSDgaJpZM4J_y21
.

Not getting it. It's possible to feed youtube-dl the URL of a Twitch live stream and it starts downloading. What's the problem?
Anyway features known to be broken shouldn't be landed.

It does not until it works in all cases in uniform way. At least I'm not going to accept semi-broken options.
In case of twitch all that should be done is removing fallback to profile extraction in stream extractor that AFAIR has been added because previously profile videos has been displayed on the same page as stream (now they are not).

Not getting it. It's possible to feed youtube-dl the URL of a Twitch live stream and it starts downloading. What's the problem?

If the stream isn't live the Twitch extractor will download VODs, which I don't want. And it expends a long time parsing dozens of videos I don't care about to even do that.

Anyway features known to be broken shouldn't be landed.

I made an explicit effort to easily allow multiple extractors to be chosen and wildcards to be used to make it possible to manually handle cases such as the dependency one.

It does not until it works in all cases in uniform way.

Do you plan on adding dependency metadata to every extractor?

At least I'm not going to accept semi-broken options.

The option is not broken. It does exactly what it promises,

In case of twitch all that should be done is removing fallback to profile extraction in stream extractor that AFAIR has been added because previously profile videos has been displayed on the same page as stream (now they are not).

That doesn't solve any of the other use cases I mentioned.
You cannot project your issues with the wildness of the different extractors into this particular feature.

Do you plan on adding dependency metadata to every extractor?

Dependency list is not a solution. For example, TumblrIE delegates GenericIE if it can't find a Tumblr video. In this case lots of extractors are added. I guess it's not what --enable-extractors tries to achieve.

@yan12125 That seems like a perfect example of the usefulness of this feature. What if I want the tumblr extractor but not the generic fallback that includes dozens of other video platforms?

I want the tumblr extractor but not the generic fallback that includes dozens of other video platforms?

Is there a real need to achieve this?

Just looked at twitch.py. I guess you don't want this fallback?
        # Fallback on profile extraction if stream is offline
        if not stream:
            return self.url_result(
                'http://www.twitch.tv/%s/profile' % channel_id,
                'TwitchProfile', channel_id)

@yan12125 Is there a real reason why I should need to use all extractors or none whatsoever? Why is only wanting to download from some particular sources that problematic?

Just looked at twitch.py. I guess you don't want this fallback?

Yes, but being able to choose extractors can be useful for other different cases. Specially when there are hundreds of extractors mixing with each other without much care.

@dstftw Also, why be so quick to close instead of providing me feedback and giving me the chance to improve/fix any issues?

Because there is no easy way to fix this.

About the problem of Twitch: this fallback was added in 240b9b7a5cd5f19f50ae28fb3ef991fede216ef6. My idea is removing the fallback. Personally I expect youtube-dl tell me there's currently no live stream (and better if youtube-dl tells me when the live stream will start) rather than downloading previous videos of the publisher. If a user want all previous recordings, they should use the profile page.

I've already explained why this fallback was added previously and it's already removed.

@dstftw  Bullshit. Your original example only fails because one case delegates directly to an extractor and the other to an URL. There is no reason why the filters could not be kept in the YoutubeDL object and used when dealing with dependent extractors. I had to go and look that up myself, since you chose to reject the feature from the start instead of giving me information so that I could improve the PR.

Then go and fix it. Note that we are not going to maintain additional explicit dependencies per extractor just in order to add an option that won't be used by 99% of users.

@dstftw Sorry I didn't notice there's a new commit.
@danielkza If you're going to continue this work, here are a few suggestions:

--enable-extractors should never include any other extractors implicitly. In the example of AlJazeera, the download should fail.
Check related issues/PRs: #9487, #6501, #10174, #6497 (maybe more)
Add tests. From what I can see this will be a fragile feature.


BIGTOP-3165. Bump Spark to 2.4.4.

Bump Spark to 2.4.4.
./docker-hadoop.sh -c 1 -s passed on my local with the following configuration.
docker:
        memory_limit: "16g"
        image:  "bigtop/puppet:trunk-ubuntu-16.04"

repo: "http://repos.bigtop.apache.org/releases/1.4.0/ubuntu/16.04/$(ARCH)"
distro: debian
components: [hdfs, yarn, spark]
enable_local_repo: true
smoke_test_components: [spark]

Since Spark src tarball doesn't inculde license files for binary as of 2.4, I added the the same binary license files to be included in bigtop binary.

Basically the PR looks good to me but I'd like to upgrade Spark to 2.4.5, because 2.4.4 doesn't work with the recent k8s versions. Could you address it?
https://issues.apache.org/jira/browse/SPARK-28921

@sekikn Thank you for your comment. I will update this PR to 2.4.5 after BOM is updated on BIGTOP-3123.

I've no much knowledge here so CMIIW.
Why we need to add these LICENSE files? I though it should be able to obtain from spark's code release.

@evans-ye Thank you for your review! I agree with you, license files should be able to obtain from spark's source but they aren't.
Spark community devided license files between source licenses and binary licenses from Spark 2.4. Spark github repository contains both license files, but as I wrote in bigtop-packages/src/common/spark/install_spark.sh src tarball no longer contains the license files for binary distribution at lease at Spark 2.4. Src tarball only have the license files only for source distribution.
See https://issues.apache.org/jira/browse/SPARK-24654
In Bigtop, we build Spark from src tarball so we have several choices here. For example, 1) Just remove every license files from Bigtop Spark binary. 2) Get license files from Spark github repository and include them into Bigtop Spark binary. I thought 1 is not acceptable as an Apache project so I PRed 2, but please let me know if you have a better way.

Thanks for the detailed answer. And I'm glad that the change is captured instead of sliped through.
So what I was worry about is that the LICENSE files can be outdated if we maintain our own copy.
How about we add some safety to make sure that in any future upgrade, the licence file can be updated?
Possible solutions:

Auto fetch licence file when building. The downside is this require network dependency, which may not be possible in company intranet.
Add a comment "remember to fetch new licence-binary" in common/spark alongside with the spark version defined in bigtop.bom. So when someone change it, the comment should be read.
Just get it in and hope that we can remember it in future upgrades ;)

I slightly prefer 2..., any input from your end?

Thanks a lot @evans-ye . The concern you wrote make sense and 2 seems to be a reasonable solution for us. So I will fix this PR by adding some safety message and changing the target version to 2.4.5 instead of 2.4.4.

Changed the version to Spark 2.4.5 and added safety comments in both bigtop.bom and bigtop-packages/src/common/spark/install_spark.sh . Could you check it again, please? @evans-ye @sekikn

+1, I confirmed that it was successfully built on CentOS 8, Debian 10, and Ubuntu 18.04. Also, its smoke test using the built packages passed in an Ubuntu 16.04 container.

@tsudukim I merged your PR manually rather than using GitHub's merge button so that I edited the commit summary (2.4.4 -> 2.4.5). So I'm going to close this PR. Thank you for the contribution!


iOS 15 Safari issue

There seems to be an issue on iOS 15 Safari with it's new navigation bar and simpleParallax. The parallax images are "jumping" while scrolling through. Maybe because the body is being affected by the navigation bar on scroll? The issue also persists on the demo page: https://simpleparallax.com/#example-max-transition
It's only on Safari. When creating a standalone app of the website on iOS 15 the issue is gone.
Already tried to set the delay to "0". But if you scroll quickly it's still jumping. Other than that it seemed to be slightly jittering.

This is fixed with latest version.


Prevents N+1 SQL queries in miq_request_workflow.rb

This has 3 different fixes for 3 different N+1's:

Handles .writable_storages with MiqPreloader properly
Collects all host records at once in .get_selected_hosts
Does a HACK to MiqPreloader to handle polymorphic relationships, and allows the .get_ems_metadata_tree to properly preload hosts.

The hack should be isolated to just this methods that need it, and should work just like the normal MiqPreloader.preload when there isn't a hash passed in, but it is admittedly pretty gross...
Benchmarks
The benchmark script used here basically runs the following:
ManageIQ::Providers::Vmware::InfraManager::ProvisionWorkflow.new.init_from_dialog

And is targeting a fairly large EMS, with about 600 hosts.

ms
queries
rows

before
15023
1961
48395

after
11722
59
48395

Before                                                     After
------                                                     -----

Total allocated:<PHONE_NUMBER> bytes (23226536 objects)   |   Total allocated:<PHONE_NUMBER> bytes (21900482 objects)
                                                       |
allocated objects by gem                               |   allocated objects by gem
-----------------------------------                    |   -----------------------------------
   9665227  pending                                    |      9665227  pending
   5561668  manageiq/app  <<<<<<<<<<                   |      5557939  manageiq/app  <<<<<<<<<<
   3513258  manageiq/lib  <<<<<<<<<<                   |      3508974  manageiq/lib  <<<<<<<<<<
   2512007  activerecord-5.0.7  <<<<<<<<<<             |      1576834  activerecord-5.0.7  <<<<<<<<<<
    861270  activesupport-5.0.7  <<<<<<<<<<            |       808731  activesupport-5.0.7  <<<<<<<<<<
    418737  ancestry-2.2.2                             |       418737  ancestry-2.2.2
    278793  activemodel-5.0.7  <<<<<<<<<<              |       274449  activemodel-5.0.7  <<<<<<<<<<
    178419  ruby-2.3.3/lib  <<<<<<<<<<                 |        52875  manageiq-providers-vmware-0be2f13a0dc9
    165577  arel-7.1.4                                 |        14424  fast_gettext-1.2.0
     52875  manageiq-providers-vmware-0be2f13a0dc9     |        13366  arel-7.1.4  <<<<<<<<<<
     14424  fast_gettext-1.2.0                         |         8453  ruby-2.3.3/lib  <<<<<<<<<<
      4115  other                                      |          309  other
        75  default_value_for-3.0.5                    |           75  default_value_for-3.0.5
        68  concurrent-ruby-1.0.5                      |           66  concurrent-ruby-1.0.5
        16  memoist-0.15.0                             |           16  memoist-0.15.0
         4  rubygems                                   |            4  rubygems
         3  manageiq-providers-lenovo-470f5d68672b     |            3  manageiq-providers-lenovo-470f5d68672b

Note: The benchmarks for this change do NOT include the changes from other PRs in the links below.  Benchmarks of all changes can be found here.
Links

Partially addresses https://bugzilla.redhat.com/show_bug.cgi?id=1569626
Full collection of proposed changes:  git compare
Document keeping track of overall improvements for this effort:  gist
Related PRs:

https://github.com/ManageIQ/manageiq/pull/17355
https://github.com/ManageIQ/manageiq/pull/17357
https://github.com/ManageIQ/manageiq/pull/17358
https://github.com/ManageIQ/manageiq/pull/17359
https://github.com/ManageIQ/manageiq/pull/17360
https://github.com/ManageIQ/manageiq/pull/17381
https://github.com/ManageIQ/manageiq/pull/17402
https://github.com/ManageIQ/manageiq/pull/17403

@miq-bot assign @gmcculloug
@miq-bot add_label performance

Thanks for the review, I appreciate it!

I'm not sure why the changes to relationship_mixin.rb, relationship.rb, and miq_preloader.rb are part of this PR. I think the changes in those 3 files should be viewed by a larger audience.

The main reason is the fulltree_opts portion of the diff requires the three of those changes as part of the optimization (see the last commit for more details).
That said, I agree with your statement that those files feel pretty out of place compared to the rest of the changes.  I just figured it made sense to include all of the changes in a single digestible commit instead of splitting things up into two more additional PRs:

One to introduce the MiqPreloader changes
One to integrate those changes into the Relationship code
One to integrate the above into this PR

Though, it would pad my PR stats a bit more... 🤔

Not opposed to moving that last commit out of this PR and try to integrate that later.  That said, I think the last commit has the biggest impact over all in terms of N+1s in this PR.  Thoughts?

For posterity, these are the new metrics I was working up after removing the last commit in this PR:

ms
queries
query (ms)
rows

before
15023
1961
873.4
48395

after
12149
709
566.1
48395

Before                                                     After
------                                                     -----

Total allocated:<PHONE_NUMBER> bytes (23226536 objects)   |   Total allocated:<PHONE_NUMBER> bytes (22396387 objects)
                                                       |
allocated objects by gem                               |   allocated objects by gem
-----------------------------------                    |   -----------------------------------
   9665227  pending                                    |      9665227  pending
   5561668  manageiq/app                               |      5560427  manageiq/app
   3513258  manageiq/lib                               |      3513288  manageiq/lib
   2512007  activerecord-5.0.7                         |      1909097  activerecord-5.0.7
    861270  activesupport-5.0.7                        |       830581  activesupport-5.0.7
    418737  ancestry-2.2.2                             |       418737  ancestry-2.2.2
    278793  activemodel-5.0.7                          |       252515  activemodel-5.0.7
    178419  ruby-2.3.3/lib                             |       121139  arel-7.1.4
    165577  arel-7.1.4                                 |       108860  ruby-2.3.3/lib
     52875  manageiq-providers-vmware-0be2f13a0dc9     |        14424  fast_gettext-1.2.0
     14424  fast_gettext-1.2.0                         |         1609  other
      4115  other                                      |          318  manageiq-providers-vmware/app
       ...                                             |          ...

Of note, this doesn't include the changes that I was working on for load_ar_obj, but that will be addressed in another PR now.

@miq-bot add_label fine/yes, gaprindashvili/yes

Fine backport details:
$ git log -1
commit 3f5401bf13b776226e243f4eba3b336c9f61866a
Author: Keenan Brock<EMAIL_ADDRESS>Date:   Fri May 18 12:08:20 2018 -0400

    Merge pull request #17354 from NickLaMuro/sql_improvements_for_miq_request_workflow

    Prevents N+1 SQL queries in miq_request_workflow.rb
    (cherry picked from commit 65631b83ab169071c9691b130da84112de477ef6)

    https://bugzilla.redhat.com/show_bug.cgi?id=1593798

Gaprindashvili backport details:
$ git log -1
commit 258b588a68c8b9dac0da056f2faba6efb3deed5a
Author: Keenan Brock<EMAIL_ADDRESS>Date:   Fri May 18 12:08:20 2018 -0400

    Merge pull request #17354 from NickLaMuro/sql_improvements_for_miq_request_workflow

    Prevents N+1 SQL queries in miq_request_workflow.rb
    (cherry picked from commit 65631b83ab169071c9691b130da84112de477ef6)

    https://bugzilla.redhat.com/show_bug.cgi?id=1593797


245 Add 404 Not Found error page

Also changes according to comments about ErrorPage from #248

Client side task. Test on the server side is failing.


add lite push-upll update model, simple unit tests, and do computeds to forget their listening when reupdate

pode aceitar, eu dxo


Check for opcache.enable_cli and apc.enable_cli

Q
A

Bug fix?
No

New feature?
Yes

BC breaks?
No

Deprecations?
No

Fixed tickets
#1385

I resolved #1385 .
1. Output message to STDERR and exit(1)
If an error occurs in dep, output a message to STDERR, set the error code to 1 .
example : if user use php 5.x.
$ ./bin/dep
Upgrade to php7
Deployer 5.x supports only php7 and above.
If you want to use older php version use Deployer 4.x
$ echo $?
1

I think this is good way to end script by error.
commit : https://github.com/deployphp/deployer/commit/3f44148c4672ac7ae94bdc1dafdfb218b8a86be9
2. Check for opcache.enable_cli and apc.enable_cli
if user set opcache.enable_cli or apc.enable_cli, Inform user to disable these config.
Question
Now in this case, dep just print message.
commit : https://github.com/deployphp/deployer/commit/978ca0af92b80f1c95ba2f1ad326e7f953839436
$ ./bin/dep -V
Disable opcache.enable_cli

... runnning script ...

As with using php 5.x, should I end the script ?

Please split PR.


setComputed does not set computed values

Seems to be a bug, I've not tested using Jest.
Using vue-cli webpack tempate.
<EMAIL_ADDRESS>Vue 2.5
Karma 1.4.1
Mocha 3.2.0
import {shallow} from 'vue-test-utils'
const cmp = {
  data () {
    return {
      a: 1
    }
  },
  computed: {
    b () {
      return this.a * 2
    }
  }
}

describe('testing vue-test-utils', () => {
  it('has default properties', () => {
    const wrapper = shallow(cmp)
    expect(wrapper.vm.a).to.equal(1)
    expect(wrapper.vm.b).to.equal(2)
  })

  it('sets data', () => {
    const wrapper = shallow(cmp)
    wrapper.setData({a: 2})
    expect(wrapper.vm.a).to.equal(2)
    expect(wrapper.vm.b).to.equal(4)
  })

  it('sets computed', () => {
    const wrapper = shallow(cmp)
    wrapper.setComputed({b: 3})
    expect(wrapper.vm.b).to.equal(3)
  })
})

  testing vue-test-utils
    ✓ has default properties
    ✓ sets data
    ✗ sets computed
	expected 2 to equal 3

I put some console.logs inside of the setComputed function in vue test utils, and the value does appear to set --  I guess it's a bug elsewhere. Will keep playing around. I wonder if this is the same problem as #176 ...

Should setComputed make it so the computed function always returns the value that was set until setComputed is called again. Or do you think the computed function in your example return a different value after a changes?
I have a fix if we think calling setComputed should permanently change the computed value returned.

I think if you are setting it for the test, then it should then not be mutable by the original computer getter

So that’s a yes to the first question in plain English :)

If I do setComputed, I expect it to be the value I set it to. However I think it should reset each test - eg, every it statement, I would expect the component back in it's initial state.

Yes, agreed.


make cache duration configurable via ENV

Currently tts files are stored in cache for 24 hours.  We should have an env like JAMBONES_TTS_CACHE_DURATION_MINS that allows this to be overridden.

closed as PR is merged


timedatectl reports system clock as not synced

In 2.47.1+rev1:

root@localhost:~# timedatectl
Local time: Sun 2020-03-15 21:00:44 UTC
Universal time: Sun 2020-03-15 21:00:44 UTC
RTC time: Sun 2020-03-15 21:00:44
Time zone: n/a (UTC, +0000)
System clock synchronized: no
NTP service: n/a
RTC in local TZ: no

Connects-to: https://github.com/balena-os/meta-balena/issues/1852

This is fixed in 2.83.18+rev1


Problem in mnist.tflite file

I had downloaded corect file from https://colab.research.google.com/github/tensorflow/examples/blob/master/lite/codelabs/digit_classifier/ml/step2_train_ml_model.ipynb#scrollTo=Q_Z5yLxrwbpI. I Put this file to final verions (end ) of android application . But a still got this error
" E  FATAL EXCEPTION: pool-2-thread-1 (Ask Gemini)
Process: org.tensorflow.lite.codelabs.digitclassifier, PID: 8694
java.lang.IllegalArgumentException: Internal error: Cannot create interpreter: Didn't find op for builtin opcode 'FULLY_CONNECTED' version '12'. An older version of this builtin might be supported. Are you using an old TFLite binary with a newer model?
                                                                                Registration failed.
                                                                                "

Hi, @DanielVacha-dv
Thank you for bringing this issue to our attention, I believe you're following the exact steps mentioned in this codelab tutorial  in that it's using the older version of TensorFlow so could you please try with latest version by adding below line, if you want to use GPU then please add this line implementation 'org.tensorflow:tensorflow-lite-gpu:+' or else ignore it
dependencies {
    implementation 'org.tensorflow:tensorflow-lite:+'
    implementation 'org.tensorflow:tensorflow-lite-gpu:+'      // Import the GPU delegate plugin Library for GPU inference
}

We do have TensorFlow Lite Digit Classification Demo Application if you want to give it try
Please let us know after trying with latest version is it resolving your issue or not ? if issue still persists please let us know with error log and steps which you followed before encountering the error message that will help us to investigate your issue further.
Thank you for your cooperation and patience.

Hi, @DanielVacha-dv
Thank you for trying things and detailed analysis about issues so I'll try from my end and see is it working as expected or not and will update you.
Thank you for your cooperation and patience.


Unit bitcoind.service could not be found.

Finally finished downloading the blockchain from a trusted laptop node. Raspiblitz initially said that bitcoind service was running, but after an hour wait, bitcoind service is NOT running.
Should I try to restart everything from scratch again or is there a way I can salvage/debug this?
Link to error logs:
https://termbin.com/bkpg

at this point best to reflash the Sdcard. If you have the chain on the disk already it will be picked up during the new new bitcoin node setup.

Thank you. Reflashing did the trick


Mouse Without Borders

Would love for the PowerToys team to consider adding Mouse Without Borders as an additional feature. I believe it would be a great utility and through PowerToys may gain visibility. Hopefully, the Windows team implements it into Windows 11.
https://www.microsoft.com/en-us/garage/wall-of-fame/mouse-without-borders/

For Windows 11 related issue, I suggest you to file this in the Feedback Hub. But this can be implements to PT as new feature as well.

Great suggestion @LD617, we'll log this in our Suggested Ideas milestone for review following our general stability efforts. Thanks!

Great suggestion @LD617, we'll log this in our Suggested Ideas milestone for review following our general stability efforts. Thanks!

Thanks for the response! I'm not a developer but I appreciate what you and the team contribute.

Please add your ideas in #77

Duplicate of #77


Set thermo state iteration params

This PR changes our thermodynamic state iteration parameters to use

a maximum of 1 iteration
a tolerance of 3 K

are we ready? :)

LGTM if it passes tests. (And if it passes, can we make the tolerance even smaller, e.g., 2 K?)

It's not passing / working with some experiments. The following break:

held suarez (ρe_tot) equilmoist topography
baroclinic wave (ρe_tot) equilmoist topography
Compresible single column EDMF DYCOMS_RF01

There are some behavioral changes in other examples, but those are non-blocking

LGTM if it passes tests. (And if it passes, can we make the tolerance even smaller, e.g., 2 K?)

It's not passing / working with some experiments. The following break:

held suarez (ρe_tot) equilmoist topography
baroclinic wave (ρe_tot) equilmoist topography
Compresible single column EDMF DYCOMS_RF01

There are some behavioral changes in other examples, but those are non-blocking

What does "break" mean here? (There may be small changes in those, of course. We do not have exact solutions for them.)

What does "break" mean here? (There may be small changes in those, of course. We do not have exact solutions for them.)

We error in saturation adjustment, so the solution at the end of the iteration is larger than 3 K. Here's one example:
  | maxiter reached in saturation_adjustment:
  | Method=NewtonsMethod, e_int=28262.293520894877, ρ=1.134188857091223, q_tot=0.009, T=283.80521815319395, maxiter=1, tol=2152.518070462876
  | ERROR: LoadError: TaskFailedException
...

Our options seem to be:

Try further increasing the temperature tolerance
Try increasing maxiter
Thread these parameters throughout the model and expose them to the command line args and make them job dependent (we'd probably want to make these non-optional arguments so that we don't accidentally have inconsistent parameters)

That said, I suspect fixing #795 will give us a bigger performance gain than reducing the number of iterations alone. Are we sure that now is the best time to make this performance/accuracy trade-off? My recommendation would be to make these changes after things are a bit more stable--especially if twiddling this parameter impacts stability (which, IIRC, it did back when experimenting with this in TC, cc @yairchn)

Try further increasing the temperature tolerance
Try increasing maxiter
Thread these parameters throughout the model and expose them to the command line args and make them job dependent (we'd probably want to make these non-optional arguments so that we don't accidentally have inconsistent parameters)

Try to see if maxiter = 2 resolves this. If not, increase temperature tolerance.

That said, I suspect fixing #795 will give us a bigger performance gain than reducing the number of iterations alone. Are we sure that now is the best time to make this performance/accuracy trade-off?

The best time to make this change was a year ago; second best time is now. At high resolution, all will be waiting for the point with the slowest convergence...

A slightly bigger issue, which may affect accuracy, is that there's a term missing in the derivative of the saturation specific humidity. I'll add it sometime soon.

And one more point: The way this is usually done (and what we really should be doing) is to use the temperature at the previous time step as the initial condition for the iterations. I mentioned this a few times, and it got into the purity of code design. But is there any serious reason why we cannot pass in an initial guess for the iterations as an input argument, and use the previous time step temperature as that initial guess.
I guarantee with that we'll be fine with 1 iteration.

And one more point: The way this is usually done (and what we really should be doing) is to use the temperature at the previous time step as the initial condition for the iterations. I mentioned this a few times, and it got into the purity of code design. But is there any serious reason why we cannot pass in an initial guess for the iterations as an input argument, and use the previous time step temperature as that initial guess.
I guarantee with that we'll be fine with 1 iteration.

We have an issue for it, I think it's just not been prioritized

Would be good to do. Given how much time we've spent discussing convergence issues, it'll probably pay quickly to get it done.

This has gone a little stale. Do we still want those changes? @szy21 @charleskawczynski

I think we want it but it just didn't work :(

We want maxiter = 1 in regular GCM runs. Are we still not doing this? If so, we should get to the bottom of it and finish it (with the previous stage/time step as initial guess). Again, this works in all other GCMs I know and should work for us.

We want maxiter = 1 in regular GCM runs. Are we still not doing this? If so, we should get to the bottom of it and finish it (with the previous stage/time step as initial guess). Again, this works in all other GCMs I know and should work for us.

No, we are not doing this. I can look into it when I have some more time, unless someone else wants to.

I can take this on. Since I have unearthed the Thermodynamics PRs

I can take this on. Since I have unearthed the Thermodynamics PRs

haha, thanks!

Superseded by #1648


Unable to view & enable notifications for private repos

Just logged in to the app on MI Note 5 Pro.
Granted access to organisation during auth.
Update org. settings to remove all restrictions.
Still can't see the org. or non-org. private repo. in Watched Repo. section.

Only 2 repos are appearing in this section. They are 2 public repos I already chose to watch from the browser.

Any tips to resolve this ??

https://docs.github.com/en/free-pro-team@latest/github/setting-up-and-managing-organizations-and-teams/about-oauth-app-access-restrictions

Some users hit this restriction, so I documented it at https://github.com/Daskiworks/ghwatch#notifications-from-github-private-repositories

@velias
I already checked out the said page & enabled all access for the app from my account & for the organization. Still facing the issue. I'll share a few pics in a bit to show the settings for better clarity.

@velias
See the below images to confirm the needed account settings.
1.)

2.)

If you need anything else to assess the situation, let me know. Thanks for your time !!

@velias
See the below images to confirm the needed account settings.
1.)

2.)

If you need anything else to assess the situation, let me know. Thanks for your time !!

Hmm, everything looks good to me, I compared it with my profile. But I'm not big expert in these Github settings as I do not manage any private repo and do not use them too much at all :-(
I read again your initial info, and I'm not sure if I understood it correctly. "Watched Repositories" view shows only repos you started to watch from the browser, this is how it works (you can't add new watched repo through this app). You mentioned there that you started to watch two public repos. Do you watch this private repo also you are asking about (I expect yes, but only to be sure)?

Hmm, everything looks good to me, I compared it with my profile. But I'm not big expert in these Github settings as I do not manage any private repo and do not use them too much at all :-(
I read again your initial info, and I'm not sure if I understood it correctly. "Watched Repositories" view shows only repos you started to watch from the browser, this is how it works (you can't add new watched repo through this app). You mentioned there that you started to watch two public repos. Do you watch this private repo also you are asking about (I expect yes, but only to be sure)?

Before I started using the ghwatch app, I had previously watched 2 public repos & 2 private repos from the GitHub web app. But in the app, only the public repos appear in the Watched Repositories list. I hope that clears your queries.

Before I started using the ghwatch app, I had previously watched 2 public repos & 2 private repos from the GitHub web app. But in the app, only the public repos appear in the Watched Repositories list. I hope that clears your queries.

Thanks for the clarification. I must say I do not see anything wrong in settings you showed in screenshots.
In your description, it looks like step "Update org. settings to remove all restrictions." is after you authorized the app. Github docs says that apps automatically get access after you remove these restrictions, but maybe you should try to remove access for GH::watch and try to login/authorize it again now when the restrictions are removed from the organization/repo.

Thanks for the clarification. I must say I do not see anything wrong in settings you showed in screenshots.
In your description, it looks like step "Update org. settings to remove all restrictions." is after you authorized the app. Github docs says that apps automatically get access after you remove these restrictions, but maybe you should try to remove access for GH::watch and try to login/authorize it again now when the restrictions are removed from the organization/repo.

@velias
I tried that already when the issue happened. Although, I recently created some new private repos & began receiving notifications for them. Also, after this happened, I was able to receive notifications about the private repos created before as well.
I can't figure out why it started working all of a sudden days after I changed the settings & reconfigured the app.
Anyways, I think the issue can be closed now unless you wish to investigate this further.

@velias
I tried that already when the issue happened. Although, I recently created some new private repos & began receiving notifications for them. Also, after this happened, I was able to receive notifications about the private repos created before as well.
I can't figure out why it started working all of a sudden days after I changed the settings & reconfigured the app.
Anyways, I think the issue can be closed now unless you wish to investigate this further.

Thanks for the info. It seems to me like some internal problem in Github itself, I can hardly do anything with this from the app side. So closing this issue. Thanks

Thanks for the info. It seems to me like some internal problem in Github itself, I can hardly do anything with this from the app side. So closing this issue. Thanks


scipy.ndimage.convolve1d and correlate1d don't behave properly with mode='mirror' on length-1 dimensions

scipy.ndimage.convolve1d and correlate1d don't behave properly with mode='mirror' on length-1 axes. It seems to be picking up random data from outside the array.
Reproducing code example:
import numpy as np
import scipy.ndimage as ndi
x = np.arange(24).reshape((1, 2, 3, 4)).astype(np.int32)
w = np.ones(3, np.int32)
out = ndi.convolve1d(x, w, mode='mirror', axis=0)

which has out equal to:
array([[[[          0, -2147483648, -2147483648, -2147483648],
         [-2147483648,           5,           6,           7],
         [-2147483648,           9, -2147483648,          11]],

        [[-2147483648, -2147483648,          14,          15],
         [         16,          17,          18,          19],
         [         20,          21,          22,          23]]]],
      dtype=int32)

but I expect a result more like:
array([[[[ 0,  3,  6,  9],
         [12, 15, 18, 21],
         [24, 27, 30, 33]],

        [[36, 39, 42, 45],
         [48, 51, 54, 57],
         [60, 63, 66, 69]]]], dtype=int32)

Clearly that output isn't right at all (has negative values). Also, it should have tripled every value, so numbers that weren't made negative didn't change (essentially assuming constant-0 mode, not mirror).
Smaller arrays exhibit this issue as well, just less obvious since the negatives don't show up. However, look for the tripling of values (that never happens). Floating-point arrays work as well, just easier to see with ints.
The requirements are that the mode is 'mirror' and the dimension that is calculated along has a length of 1.
Scipy/Numpy/Python version information:
1.4.1 1.16.4 sys.version_info(major=3, minor=7, micro=5, releaselevel='final', serial=0)

Thanks for reporting this @coderforlife.
It seems that the nd variant does not suffer from the same problem. (extending w to size 1 along all other axes and using convolve instead of convolve1d gives the expected result:
import numpy as np
import scipy.ndimage as ndi
x = np.arange(24).reshape((1, 2, 3, 4)).astype(np.int32)
w = np.ones((3, 1, 1, 1), np.int32)
out = ndi.convolve(x, w, mode='mirror')


Add support to application/json

resolved in https://github.com/brazil-data-cube/sample-db-utils/pull/29


Show "Pumpen"-Status, link to OSM

It's great to see the OSM data for Berliner Pumpen used for this app.
Based on https://www.openstreetmap.org/note/2204622, I think some improvements would be helpful for the users:
Idea 1: Show status in the app
The pump-status ist tracked with pump:status=broken pump:status=ok and others, see https://wiki.openstreetmap.org/wiki/DE:Berliner_Straßenbrunnen
This status could be shown in the app.
I suggest to also show the check_date=2020-06-06 to indicate, when the status has last been checked in OSM.
Idea 2: Link to OSM
Since the position on the map is quite rough, it would be nice to link to the OSM object, so that users might see that they can comment more directly and find it and maybe update it.
See https://www.openstreetmap.org/note/2204622 for example, where the comment note was place a bit away from the pump-node (https://www.openstreetmap.org/node/760002892).
It might also be helpful to link to https://wiki.openstreetmap.org/wiki/DE:Berliner_Straßenbrunnen which explains how the Pumpen are mapped in OSM-Berlin.

Hi @tordans Thanks for the suggestion. The pumps data set is pretty basic currently. See the geojson.
I guess idea 1 having them and keeping them up to date would need the following things.
Automated process of

aggregation (from where?)
and (maybe) cleaning
storing on a public location where we can access it (AWS s3?)

I think this should not be done in this repo. This would be a something the harvester should do on a daily basis.
Idea 2 is more a thing for the FAQ and some additional text. Is there a way we can link directly with an ID to a pump?
@fdnklg @sebastian-meier how was the pumps dataset created (also related to #132 )?

I guess idea 1 having them and keeping them up to date would need the following things.
Automated process of
* aggregation (from where?)

@fabianmoronzirfas  You could use the Overpass API [1]. They offer a web frontend to get familiar with the API [2]. Beside seeing the hits on a map, you can switch to the xml-data of the result set.
This query should provide the data you are looking for.
area
  [boundary=administrative]
  [admin_level=4]
  ["name"="Berlin"]->.searchArea;
node
  [man_made=water_well](area.searchArea);
out;

[1] https://wiki.openstreetmap.org/wiki/Overpass_API
[2a] https://wiki.openstreetmap.org/wiki/Overpass_turbo
[2b] https://overpass-turbo.eu/s/Vtd (incl. query mentioned above)

@fe60 Thanks. Very nice. So the following call gives we a xml that I can convert to geojson and map it? Could be part of the build or could be run on a daily basis.
curl "https://overpass-api.de/api/interpreter?data=area%0A%20%20%5Bboundary%3Dadministrative%5D%0A%20%20%5Badmin_level%3D4%5D%0A%20%20%5B%22name%22%3D%22Berlin%22%5D-%3E.searchArea%3B%0Anode%0A%20%20%5Bman_made%3Dwater_well%5D%28area.searchArea%29%3B%0Aout%3B%0A"

@fabianmoronzirfas The API offers the option to ask for different output formats [1]. Unfortunately it does not provide the data as GeoJSON. But there is a command line tool doing the required conversion [2].
[1] https://wiki.openstreetmap.org/wiki/Overpass_API/Overpass_QL#Output_Format_.28out:.29
[2a] https://wiki.openstreetmap.org/wiki/Overpass_turbo/GeoJSON
[2b] https://github.com/tyrasd/osmtogeojson

Idea 1: Show status in the app
The pump-status ist tracked with pump:status=broken pump:status=ok and others, see https://wiki.openstreetmap.org/wiki/DE:Berliner_Straßenbrunnen
This status could be shown in the app.
I suggest to also show the check_date=2020-06-06 to indicate, when the status has last been checked in OSM.

@tordans survey:date seems to be the better choice compared to check_date.
https://wiki.openstreetmap.org/wiki/Key:survey:date

@tordans survey:date seems to be the better choice compared to check_date.
https://wiki.openstreetmap.org/wiki/Key:survey:date

There is no consensus on either and the local guideline uses check_date (https://wiki.openstreetmap.org/wiki/DE:Berliner_Straßenbrunnen) so I recommend sticking with it - for this tagging schema.

subscribe
I recommend to use the "description=Berliner Straßenbrunnen" tag to query the public Berlin water pumps from the OSM data base (http://overpass-turbo.eu/s/PQ3). Some month ago I cleaned up the OSM data and this query works reliably. Otherwise you get many pumps on private property like garden pumps or historic water wells. I regularly check the database to maintain consistency.
"survey:date" might be the better choice following the OSM wiki - I'll update the OSM Berlin pump guidelines accordingly. In any case, different variants should be considered for queries.

Siehe auch https://overpass-turbo.eu/s/VLo

Hi! I am sad to see that you switched away from using OSM in #188.
Is there a place that explains the reason for this change; or could you explain it here?
https://github.com/technologiestiftung/giessdenkiez-de/commit/9902ae908a76cb58fdfc7d0a08587af551237935 suggest that the data from the city is more complete and accurate. I would love to see the data comparison for this, so we can improve OSM as well. In my experience, the city data is at best as incomplete as OSM. But it cannot be updated by the community.
FYI @SupaplexOSM

@tordans we will have a meeting next week where we will review that decision. I think the reason was that the dataset coming from the WFS service was more complete than the current data we had. As I remember it @Lisa-Stubert told me that the official data was used to update the OSM data. So we might switch back.

@fabianmoronzirfas The API offers the option to ask for different output formats [1]. Unfortunately it does not provide the data as GeoJSON. But there is a command line tool doing the required conversion [2].

I just saw that in https://github.com/jbelien/osm-compare/blame/master/routes/compare.js#L31 they use https://www.npmjs.com/package/osmtogeojson to create a geojson from osm-overpass-data.

Thanks @fabianmoronzirfas

I think the reason was that the dataset coming from the WFS service was more complete than the current data we had. As I remember it @Lisa-Stubert told me that the official data was used to update the OSM data. So we might switch back.

Yes, I heard from @SupaplexOSM that he is already talking to Lisa-Stubert.
I looked into the data with https://github.com/tordans/osm_conflate/commit/cebb428121b7c40ac656bf6732230381261b4362. From a first glance it looks like there is more data in the Randbezirke (green=missing in OSM), but also a bit of missing data (red=missing in Citydata).

I send the results to @SupaplexOSM to have a look. We might need to create a https://maproulette.org/ challenge to add the data to OSM. (https://pic4review.pavie.info/#/ is no option unfortunately for this data and use case.)

@tordans  thanks for the analysis of the different data situation. Exactly, we switched to the official data at that time, because these contain considerably more pumps and because we actually try to use as much open data of the public administration as possible. We are already in discussion with the administration and are trying to find out whether the quality of the data can be improved. Unfortunately this will not happen so quickly. Since there is so much interest in the functionality of the pumps of the Gieß den Kiez users, it makes sense to switch back to OSM. But it would be nice if the OSM community could enter the missing pumps from time to time. I'm already talking to @SupaplexOSM about this, but he has noted that due to the strict regulations of OSM, adding pumps is not so fast. I have no experience with maproulette, but it might be a good idea.

@all in this thread. Just a update. We are working on a script that aggregates the data from OSM to create our geojson.
We will run this on a weekly schedule. As soon as it is ready we will open source it and link it here.

Here is the harvester https://github.com/technologiestiftung/giessdenkiez-de-osm-pumpen-harvester/

Hey we added the status of the pumps on hover and we'd love to get some feedback. You can see a deploy preview here https://deploy-preview-218--internetoftrees.netlify.app/

Nice! Some ideas for small adjustments - however, they go into detail and only affect very few locations:

There is the status "out_of_order" (currently 3x in Berlin) which could be displayed as a bright point (category "not functional") - e.g. https://www.openstreetmap.org/node/6249473226.
In rare cases the pumps are not open accessable to the public (OSM-Tag: access=private). Then they could also be shown in the light category with a corresponding note "No public access"/"Nicht öffentlich zugänglich" (see e.g. https://www.openstreetmap.org/node/5538995537)
There may be wells that do not belong to the network of Berlin street / emergency water wells, but are still accessible to the public (OSM-query "water_well=pump" + "access=yes" or "access=permissive"). These could possibly also appear on the map. However, I can currently only find one suitable case: https://www.openstreetmap.org/node/7851278239 (a freely accessible pump built on private initiative)

Now the only thing missing is the possibility for users to be able to play back a status report :) Maybe you have the capacity for a narrow solution someday like a online table or a mini-feedback-form. I'm happy to add messages/reports to the OSM database.

@SupaplexOSM thanks for the feedback. 1. 2. 3. sound like edge cases. I added the out_of_order pumps and the missing_beam to the defekt category so these will be covered.
Getting feedback back in would be awesome. We get a lot of feedback about missing trees and trees on private property. So there is a demand on sending information upstream again. This could also contain information about the pumps states, but we will have to add this to the backlog. Since we are on feature freeze now.

Perhaps we could set up a channel in our Gieß-den-Kiez-Slack as a quick but helpful alternative to a feedback form. Then we would also add a section "How can I report broken/missing pumps or missing information about pumps?" to the FAQ's and refer to the Slack-channel. Unfortunately we don't have time to maintain the channel/process the reports, but for the OSM community it would be a possibility to get information?

Would be a good solution for now. I could transfer the data if someone reports something.

The pumps are online. Thanks 2 everyone for the effort.