Microsoft Language Server Protocol

Hi!
This project is awesome! I'm really excited about it.
I have been working on this project:  https://github.com/tjdevries/nvim-langserver-shim
It is a working shim for at least some of the features for communicating to language servers that run the MS Language server protocol. Even if you don't use my shim, I think you should really consider using the protocol. It will be very helpful when more and more language servers adopt the protocol.
I'm wondering if you'd like to use it to extend some of the features that are there (or I'd do that as well obviously :smile: ).
Anyways, let me know what you think.
Again, really cool project!

Also, just in case you haven't seen it: https://github.com/neovim/neovim/pull/5686 This is pretty cool.

Hi TJ! Thanks for the kind words and checking this project out :)
Great point, I agree 110% - the protocol makes a lot of sense to implement. Actually the language extensibility features for VSCode were a major inspiration and driving factor - the interfaces I have so far are pretty similiar because of that, and I think mapping to the language service makes a lot of sense. Will be way easier to add the language features by these already-written services.
I actually tried really, really hard to like VSCode and use it (plus the vim plugin here: https://github.com/VSCodeVim/Vim) but there are just enough gaps that it still is problematic, and there are also features I would like to see in the editor that unfortunately their API surface is not quite extensible enough to implement, but this project takes a lot of inspiration from VSCode (and Atom and LightTable). It's a great editor though and there is a ton of momentum around the plugins and supported languages - so it'd be great to leverage that.
I took a look through your repo and it's really helpful. I didn't realize how well the language service protocol is documented, and you've done extensive work in hooking up to it via the async capabilities. Really clean and easy to follow implementation, too. Awesome to see work being done on that - that potentially benefits everyone using neovim, including users who don't want a UI version.
In the context of this project, I'm trying to writing as minimal VimL as possible (JS / TS with the language features in the editor, debugger, etc is just more fun to work on and maintain for me :) ) - but I think that it would be possible to hook up to the language service protocol directly when we load plugins in browser/src/Plugins/Plugin.ts & PluginManager.ts - would just need some sort of shim/adapter layer, as the protocol being used is pretty close. Or maybe even just pivot to use their protocol.. If you're interested in collaborating on that, let me know

And that UI looks awesome! It's really cool to see the work being done with neovim and the scenarios it is enabling. I opened #9 to track integrating with the external popupmenu as a starting point/proof of concept

So one thing I'm working on that might help you a lot is making an api to handle responses for the language server protocol. The idea would be something like this:

User initiates some LSP (language server protocol) request
You do all the communication to some language server to get the information
You now have the response from the LSP. You have to decide what to do with it. This is where I think the nvim api would be useful. You would...
call langserver#api#textDocument#definition(response_dictionary, options)
The language server shim would handle the response and do all the neovim required items (managing a buffer, go to line, etc.)

This way you don't have to worry about keeping track of what happens in response to certain items and people don't have to duplicate work (because we would have to do the same thing, in essense.)
Does that make sense?
Another thing I want to add is the ability to make a request, and then have a custom callback attached to it. So you might want to initiate a request for a hover (https://github.com/Microsoft/language-server-protocol/blob/master/protocol.md#textDocument_hover). Well I already have a function for doing that (https://github.com/tjdevries/nvim-langserver-shim/blob/master/autoload/langserver/hover.vim#L19, although I would add this to the API with easily discoverable names... just haven't yet :smile: ). So you could call that function and add a callback for you to handle the response (since you'd do hover differently than regular neovim would, since you have fancy and pretty gui features).
Let me know if you have any questions about this. Most of these thoughts are still bouncing around in my head. The main goal would be that lots of GUIs / plugins would like to make use of the LSP within Neovim, so it'd be good if we could centralize all the knowledge and solve the difficult problems in just one place, instead of everyone having to do it over and over.
THanks for your time. :smile:

Thanks for taking the time to drill into details :) I really like the work you've done and you're thinking around centralizing the knowledge / not re-inventing the wheel.
My hesitation in terms of doing a deeper integration between this project and a VimL language server API layer stems from one of the goals I have in mind for the project:

Rich plugin development - using JavaScript, instead of VimL, allowing deep language integration.

And to be fair, this is partly for selfish reasons - I personally hate debugging and writing VimL :) But I also believe  there is a ton of value in JavaScript - even though it's not a perfect language by an stretch, there's a huge ecosystem and continual investment in tooling - and many of the modern editors like Atom, VSCode, went this route, and have been hugely successful in creating a vast ecosystem of high-value plugins. So I'm hoping that Oni could tap into that as well, and combined with the power of Neovim, could be a truly productive, modern (even next-gen), modal editor.
For those reasons, I would prefer not to call into a VimL layer to implement the language service abstraction, and instead implement it in the JS layer to align with the vision of the project.
However, there is a ton of value in your project and in general to have the language server layer accessible in Neovim, and not just Oni - it allows terminal users to get the benefit of the language services, which is huge!  Oni is essentially ignoring that need because it doesn't run in the terminal, so it is a departure from a large part of the vim/neovim audience - but your plugin addresses that gap.

Thanks for the reply.
I think I understand your concerns. The main reason I suggest using a VimL layer to people is to manage many of the built-in features that neovim already has. For example, in a goto situation, you would want to manage the jump-list (or tag-list, depending on how you want to view it) well so that you can do either a <c-i> or ` to move back to the original place before the movement. I think VimL is good at managing these items and having a place where it's managed centrally would reduce a lot of work.
But I definitely understand the distaste for VimL, it certainly has some interesting quirks haha :smile:.
My thinking would be to just use the VimL for places where the VimL is suited and do everything else in Javascript. In fact, I would recommend definitely using the vscode's javascript language server protocol client / server (https://github.com/Microsoft/vscode-languageserver-node) because it will most likely be faster and easier to use. It seems you are already interfacing a lot with VimL (you have several "built-in" vim plugin), so I think considering using the langserver-shim as another vim plugin to handle some of the more internal Neovim actions, rather than the GUI side items.
So in case I wasn't clear, steps 1 through 4 would be in javascript.  The only thing that would happen in Viml is the handling of the response received from the server.
Anyways, I'm obviously just being selfish as well, cause I want people to use my project ;) Thanks for the convo, it's giving me some new ideas to implement.

@tjdevries: you seem to have a decent amount of knowledge on the subject. After we hijack VSCode's implementation and use it for our own nefarious purposes, would we just have the gui act as a sort of middleman between langserver and your shim?

Initial LSP support is now in Oni - a reference plugin leveraging the language client is available in the oni\vim\core\oni-plugin-csharp.
As mentioned in the PR, have several follow up issues being tracked (additional functionality, additional out-of-the-box language support, etc):
#382 #381 #380 #379 #378 #377 #376 #375

Closing as the core implementation is complete - tracking remaining work in other issues.


Add support for running in AOT

Currently when I depend on the Extism SDK from my C# AOT command line program, I get this error:
System.InvalidOperationException: Reflection-based serialization has been disabled for this application. Either use the source generator APIs or explicitly configure the 'JsonSerializerOptions.TypeInfoResolver' property.
   at System.Text.Json.ThrowHelper.ThrowInvalidOperationException_JsonSerializerIsReflectionDisabled()
   at System.Text.Json.JsonSerializerOptions.ConfigureForJsonSerializer()
   at System.Text.Json.JsonSerializer.GetTypeInfo(JsonSerializerOptions options, Type inputType)
   at System.Text.Json.JsonSerializer.GetTypeInfo[T](JsonSerializerOptions options)
   at System.Text.Json.JsonSerializer.Serialize[TValue](TValue value, JsonSerializerOptions options)
   at Extism.Sdk.Plugin..ctor(Manifest manifest, HostFunction[] functions, Boolean withWasi)

This is because Extism is trying to serialize the manifest as JSON using reflection on this line right here:
https://github.com/extism/dotnet-sdk/blob/234610e4e7f45f4c7c43fcbf3b7443b2821a553a/src/Extism.Sdk/Plugin.cs#L49
There's a lot more information here: https://devblogs.microsoft.com/dotnet/system-text-json-in-dotnet-8/
To reproduce this error, depend on the Extism SDK and add these lines to your csproj:
    <PropertyGroup>
        <PublishAot>true</PublishAot>
    </PropertyGroup>

Can we add support for running this library from .NET AOT? I don't mind contributing.

Hi @nloum, great catch!

Can we add support for running this library from .NET AOT? I don't mind contributing.

Absolutely, please do! Let me know if you need anything to get started, I'd be happy to help

We also have a dotnet-sdk channel in our discord server:
https://discord.gg/myuWqjBq
Feel free to ask questions either here, or on Discord


fix typo in build_environment doc

Hi.
I looked at the documentation for gradle proxy use the gradle.properties.
i saw that some text correction was needed, so I create pull request.
https://docs.gradle.org/current/userguide/build_environment.html#sec:accessing_the_web_via_a_proxy

@bot-gradle test and merge

OK, I've already triggered a build for you.

Hi @pasudo123,
I've reverted this change in https://github.com/gradle/gradle/pull/19579. The documentation is actually correct.
Java expects the system property to be http.nonProxyHosts and not https.nonProxyHosts. https://docs.oracle.com/javase/8/docs/api/java/net/doc-files/net-properties.html
Sorry for the confusion, but thanks for the contribution!

@big-guy
This is something I found out again. Thank you for answer.


[SQLLab] Cosmetic Layout issue

SQLLab cosmetic layout issue
Expected results
Correct SQLLab UI
Actual results

Environment
The latest master branch build from docker-compose up

@betodealmeida  Thank you for the quick fix! closed this issue.


Fix #1760: do not console.error handled exceptions

When an exception is handled in an errorHandler function, do not re-raise it.
fix #1760

What kind of change does this PR introduce? (check at least one)

[x] Bugfix
[ ] Feature
[ ] Code style update
[ ] Refactor
[ ] Build-related changes
[ ] Other, please describe:

Does this PR introduce a breaking change? (check one)

[ ] Yes
[x] No

If yes, please describe the impact and migration path for existing applications:
The PR fulfills these requirements:

[x] It's submitted to the dev branch.
[x] When resolving a specific issue, it's referenced in the PR's title (e.g. fix #xxx[,#xxx], where "xxx" is the issue number)
[ ] All tests are passing: https://github.com/vuejs/vue-test-utils/blob/dev/.github/CONTRIBUTING.md#development-setup
[x] New/updated tests are included

If adding a new feature, the PR's description includes:

[ ] A convincing reason for adding this feature (to avoid wasting your time, it's best to open a suggestion issue first and wait for approval before working on it)

Other information:

I thought the whole idea of using createLocalVue with an error handler was that we DO want to console.error? It looks like it was added here intentionally: https://github.com/marcgibbons/vue-test-utils/commit/e91effeda17e68817aa7bafaaf24a7ffa8185776#diff-dd59fc30dc63197a9d0939e3119aef809349799e5aa021fa73c20301d8c7bd96R8
Am I missing something?

I thought the whole idea of using createLocalVue with an error handler was that we DO want to console.error? It looks like it was added here intentionally: https://github.com/marcgibbons/vue-test-utils/commit/e91effeda17e68817aa7bafaaf24a7ffa8185776#diff-dd59fc30dc63197a9d0939e3119aef809349799e5aa021fa73c20301d8c7bd96R8
Am I missing something?


⚠ Fetchers has degraded performance

In b1582b7, Fetchers ($STATUS_URL) experienced degraded performance:

HTTP code: 200
Response time: 524 ms

Resolved: Fetchers performance has improved in f74ccc3.


Doesn't show error

Problem: Doesn't show any error when I mistakenly write an error code. (sass)

Please fix these issues. This thing is most important in sass. Hope to take it seriously and fix it quickly.

Hey @Shams-Raju,
It should definitely show the output on error, that is unless you have set the showOutputOn setting set to "None" - which will only show critical errors or output relative to a command (i.e. liveSass.command.debugInclusion).
See this example project that shows an Error in the status bar as well as having an output error: Issue 76 - Sample Project.zip - also see below screenshot

Could you please confirm the above works the same for yourself

It is becoming clear that I need to change the default for showOutputOn
When I changed the showOutput setting to showOuputOn it became more of a logger - rather than a show everything or nothing boolean.
The creation of this issue and #70 - and some other issues that have mentioned no output - shows that the default behaviour of this new setting needs to match Ritwick's original extension. I suppose when people migrate they expect it to work the same way as the original.
I will schedule a release where the default for showOutputOn is Information and no longer Warning - this will give the same output as the original extension.

It is becoming clear that I need to change the default for showOutputOn
When I changed the showOutput setting to showOuputOn it became more of a logger - rather than a show everything or nothing boolean.
The creation of this issue and #70 - and some other issues that have mentioned no output - shows that the default behaviour of this new setting needs to match Ritwick's original extension. I suppose when people migrate they expect it to work the same way as the original.
I will schedule a release where the default for showOutputOn is Information and no longer Warning - this will give the same output as the original extension.

Yes, I saw the difference. When I set ("showOutputWindow": none) then I can't see the sass error in output windows.
But when I set ("showOutputWindow": warning) then this is perfectly working. But there is a problem.
When I set "warning" then I write a half code like:  color: ------ then instant output windows show me the disgusting warning.
No need for this warning in output windows.😑

Yes, I saw the difference. When I set ("showOutputWindow": none) then I can't see the sass error in output windows.
But when I set ("showOutputWindow": warning) then this is perfectly working.

Glad that cleared up the original issue

But there is a problem.
When I set "warning" then I write a half code like: color: ------ then instant output windows show me the disgusting warning.
No need for this warning in output windows.😑

I'm afraid that this is out of scope. This extension is a compiler. This means, to state the obvious, it converts SASS into CSS. So, when you save, it will always reports errors in your code at that point - that's just how it works.
What you need is something that does SASS IntelliSence. This reports errors live, which is what shows in the bottom left - as you have highlighted in your screenshot

Yes, I saw the difference. When I set ("showOutputWindow": none) then I can't see the sass error in output windows.
But when I set ("showOutputWindow": warning) then this is perfectly working.

Glad that cleared up the original issue

But there is a problem.
When I set "warning" then I write a half code like: color: ------ then instant output windows show me the disgusting warning.
No need for this warning in output windows.😑

I'm afraid that this is out of scope. This extension is a compiler. This means, to state the obvious, it converts SASS into CSS. So, when you save, it will always reports errors in your code at that point - that's just how it works.
What you need is something that does SASS IntelliSence. This reports errors live, which is what shows in the bottom left - as you have highlighted in your screenshot

What you need is something that does SASS IntelliSense. This reports errors live, which is what shows in the bottom left - as you have highlighted in your screenshot

I understood. Thanks for your reply <3

You're welcome.
In going to re-close this issue as I assume you reopened it by mistake


Scala rewrite

Fixes #44 #41 #37 #30. Subsumes #40 #43.
TODO

[x] Update buildbot scripts
[x] Write new Dockerfile
[x] Explicit typed let
[ ] ~views~
[x] reductions
[ ] capabilities
[ ] iterator stack checking
[x] Fix #47
[ ] subtyping review

Re: Examples. They have been moved to src/tear/resources and are run with the test suite.
The examples directory is used by Sachille for something, but he can starting migrating things to the same place.


Better error when the LHS of an && is always true

Motivation

Fixes #6138
Test plan

See included automated tests.

I can't decide whether I admire or despair that we use magic variable names like this.

The magic variable names are partly the problem—if we ever decide that we want to handle the x && y case where x is a variable, we'd need to change this to something more explicit like <Magic>.<and-and-else>(x) or something.


CLI throws AttributeError when using verify and generate commands

Hi, thanks for providing this awesome dataset and devkit.
I have downloaded the dataset using the CLI tool. I want to verify it and generate an export to MS-COCO format using the CLI as well. When I do, however, I get an AttributeError: 'function' object has no attribute 'base_type' error. I've tried this with fresh python3.9 and python3.10 installations for both the verify as well as the generate command.
zod verify --dataset-root=full --subset=frames --version=full

/root/.pyenv/versions/3.9.20/lib/python3.9/site-packages/zod/cli/verify.py:98 in verify
> zod = subset.dataset_cls(str(dataset_root.expanduser()), version.value)
/root/.pyenv/versions/3.9.20/lib/python3.9/site-packages/zod/_zod_dataset.py:31 in __init__
> self._infos, self._train_ids, self._val_ids = self._load_infos()
 /root/.pyenv/versions/3.9.20/lib/python3.9/site-packages/zod/_zod_dataset.py:97 in _load_infos
> infos = process_map(func, chain.from_iterable(all_ids.values()), ...)
.......
/root/.pyenv/versions/3.9.20/lib/python3.9/concurrent/futures/_base.py:391 in __get_result
>>> AttributeError: 'function' object has no attribute 'base_type'

Limiting the dataclass-wizard version to >=0.22.2,<=0.31.0 fixes this issue for me.


DSD-891: Adding a11y docs for page layout components

Fixes JIRA ticket DSD-891
This PR does the following:

Adds table of contents and accessibility docs for HorizontalRule, SimpleGrid, StructuredContent, Table, and Template.

How has this been tested?
Storybook.
Accessibility concerns or updates

More docs.

Checklist:

[ ] I have updated the Storybook documentation accordingly.
[ ] I have added relevant accessibility documentation for this pull request.
[ ] All new and existing tests passed.

Front End Review:

[ ] View the example in Storybook

Tugboat has finished building the preview for this pull request!
Link:

https://pr954-xlskj5aapcjqmeffwyyjr1zxdlidl4hh.tugboat.qa

Dashboard:

https://dashboard.tugboat.qa/62586d68bda4981539f77f49

Tugboat has finished building the preview for this pull request!
Link:

https://pr954-xlskj5aapcjqmeffwyyjr1zxdlidl4hh.tugboat.qa

Dashboard:

https://dashboard.tugboat.qa/62586d68bda4981539f77f49

Tugboat has finished building the preview for this pull request!
Link:

https://pr954-xlskj5aapcjqmeffwyyjr1zxdlidl4hh.tugboat.qa

Dashboard:

https://dashboard.tugboat.qa/62586d68bda4981539f77f49

Updated

Tugboat has finished building the preview for this pull request!
Link:

https://pr954-xlskj5aapcjqmeffwyyjr1zxdlidl4hh.tugboat.qa

Dashboard:

https://dashboard.tugboat.qa/62586d68bda4981539f77f49


Approval Needed for Next Step

Please comment "approve" to proceed with the third workflow.
Minimum Approvals: 1
Approval Words: approve, approved
Rejection Words: deny, denied, reject, rejected

Timed out after waiting for 60 minutes for approval.


Fix/add shortcut editor minor issues

Fix #2475
Fix some pending issues from https://github.com/spyder-ide/spyder/pull/2367

"Fuzzy" string matching

Todo

[x] Fix docstrings
[x] Change toggle behavior in shortcut dialog
[x] Add fuzzy sorting
[x] Change information display for conflicting shortcuts
[x] Handle sequences bigger than 4 keys
[x] Relocate HTMLDelegate to widgets/helperwidgets.py
[ ] Fix dead strokes input

For another PR (when I have access to a Mac...)

Fix text color on selected text (in Mac)

@SylvainCorlay can you give me an example of what happens with the dead stroke issues?
What type of keyboard layout you have in your pc... OS.. etc..
Cheers

@goanpeca: One example that @SylvainCorlay showed me in person: pressing Ctrl + <tilde> + e where <tilde> is the key to get é on the screen. That's a dead key :-)

@goanpeca if you use a US-international keyboard layout, the following key sequences do
` then e   -> è
^ then e   -> ê
' then e   -> é
meaning that the first character is a dead key and does not do anything by itself. If you just want a quote, you need to press the space bar.
If you use a Spanish layout, I think that ^ is already a dead key.
I don't think that we really need to support dead keys for keyboard shortcuts. Although in the French layout, things like é are a single key stroke, and should not create any problem.

I cannot reproduce that behavior.
On ubuntu with US-International, if I press Control+' and (while holding control) press "e", I get
Ctrl+'; Ctrl+e

@goanpeca, do you have a Latin American keyboard to test? This is what I'm getting with Ctrl+<tilde>+e:
Ctrl+ោ

Not right now...

Ok, no prob. We can check it out week when we meet in Colombia :-)

Any comments on the search order screenshots... ?

@goanpeca, let's leave the dead keystroke task for another PR.
@blink1073, would you mind to take a look at the fuzzy matching code introduced by @goanpeca in utils/stringmatching.py?

@goanpeca, please address @blink1073 and my comments so that we can merge this for beta1. I think we need to merge this before releasing it because it has important improvements.

@ccordoba12 I made the corrections, can you test on your side?
Cheers

@goanpeca, I have a patch (against your latest commit in this branch) that solves all issues I have right now with the ShortcutEditor dialog:
@@ -139,7 +139,8 @@ class ShortcutEditor(QDialog):
         self.text_new_sequence = CustomLineEdit(self)
         self.text_new_sequence.setPlaceholderText(sequence)
         self.helper_button = HelperToolButton()
-        self.label_warning = QLabel('
')
+        self.label_warning = QLabel()
+        self.label_warning.hide()

         bbox = QDialogButtonBox(QDialogButtonBox.Ok | QDialogButtonBox.Cancel)
         self.button_ok = bbox.button(QDialogButtonBox.Ok)
@@ -164,7 +165,7 @@ class ShortcutEditor(QDialog):
         self.label_warning.setFocusPolicy(Qt.NoFocus)

         # Layout
-        spacing = 24
+        spacing = 5
         layout_sequence = QGridLayout()
         layout_sequence.addWidget(self.label_info, 0, 0, 1, 3)
         layout_sequence.addItem(QSpacerItem(spacing, spacing), 1, 0, 1, 2)
@@ -267,8 +268,8 @@ class ShortcutEditor(QDialog):

             if len(self.keys) == 1 and key == Qt.Key_Escape:
                 self.set_sequence('')
-                self.toggle_state()
-                return
+                self.label_warning.setText(_("Please introduce a different "
+                                             "shortcut"))

             if len(self.keys) == 1 and key in [Qt.Key_Return, Qt.Key_Enter]:
                 self.toggle_state()
@@ -278,7 +279,7 @@ class ShortcutEditor(QDialog):
                 self.nonedit_keyrelease(e)
             else:
                 debug_print('keys: {}'.format(self.keys))
-                if self.keys:
+                if self.keys and key != Qt.Key_Escape:
                     self.validate_sequence()
                 self.keys = set()
                 self.key_modifiers = set()
@@ -304,13 +305,14 @@ class ShortcutEditor(QDialog):

         if warning_type == NO_WARNING:
             warn = False
-            tip = '

'
+            tip = ''
         elif warning_type == SEQUENCE_CONFLICT:
             template = '{0}{1}'
             tip_title = _('The new shorcut conflicts with:') + '
'
             tip_body = ''
             for s in conflicts:
                 tip_body += ' - {0}: {1}
'.format(s.context, s.name)
+            tip_body = tip_body[:-4]  # Removing last

             tip = template.format(tip_title, tip_body)
             warn = True
         elif warning_type == SEQUENCE_LENGTH:
@@ -326,6 +328,7 @@ class ShortcutEditor(QDialog):
             warn = True

         if warn:
+            widget_message.show()
             widget.setIcon(get_std_icon('MessageBoxWarning'))
         else:
             widget.setIcon(get_std_icon('DialogApplyButton'))

This patch:

Fixes the spacing issues in the dialog (i.e. a huge empty space between the shortcut line edit and the buttons).
Uses Esc to reset a conflicting shortcut, and let the user introduce a new one at the same time.

I'm fine with hidding label_warning at first and then showing it when a conflict appears.

Forget about my last patch, I have other improvements so I'll do a PR against yours :-)

Ok, please go ahead!
Should I make a new PR for the plugins (the pending @Nodd pr?)

Yes, please! ;-) We need to start moving on that front ASAP, but remember that that's work for beta2.

So what is missing for beta1 at this point?

Take a look at our shared Google doc. There is the list of missing things.

Got it...

Ok, feel free to merge this one then :-)


[Enhancement] Add additional configuration options to chickens stats (capping stats at a lower level and rate limiting cross bred chickens)

Had a couple of configuration tweak ideas:
First
Have a Max stat limitation for each chicken...e.g. you can specify that diamond chickens cap out at say 2/2/2, or ender pearl chickens at 4/4/10.
Second
Have some sort of cross breeding limiter (similar mechanic to agricraft cross breeding) for each stat (Growth/Gain/Strength), make this a decimal value between 0-1.
This limiter gets applied to the scores of the new egg produced by cross breeding:
Configuration value  (missing)  == keep the original calculation
Configuration value= 0 ==reset the given stat back to 1
Configuration value= 1 ==keep the originally calculated stat
Configuration value between 0 & 1  ==  multiply the value to the calculated score and round it down
For instance if you had a limiter of say
Growth = .5
Gain = 0
Strength = 1
When you breed two 10/10/10 chickens, instead you would end up with an egg that has the stats of 5/1/10.

I agree. An option in the config to handle breeding/mutation stats  would be amazing. It would allow pack creators more flexibility.

Perhaps and option to require minimum stats for cross-breading? Getting purple from red and blue chickens might only be 2/2/2 or 3/3/3, but breeding the diamond requires 10/10/10. Add the loss multiplier and you'll need to breed two 10/10/10 chickens to cross-breed a 1/1/1 diamond chicken. Now, that would be cruel, but still better than getting the base chickens to 10/10/10 and not sweating the rest.


inherit dark mode from system theme?

Is your feature request related to a problem? Please describe.
Not really. Keeping theme consistent reduces eyestrain. But that's not really a "problem" problem.
Describe the solution you'd like
I would like the UI to recognize that I am using dark mode on my system to switch to some dark mode, ideally the system one, but one you design would work too. It's just about the contrast.
Describe alternatives you've considered
No response
Additional context
I am on
OS: Ubuntu 22.04 jammy
DE: Xfce
WM: Xfwm4
WM Theme: Numix
GTK Theme: Adwaita-dark [GTK2]

I don't really expect a fix, it's more of a todo, that I didn't find in your issues yet. Maybe something as a good first issue for new contributors.
Software works beautifully, thanks for making it!

In Settings, you can change "Color" to "System". It will then match your system theme.

This is true and I found the setting. It was already defaulting to "system". In that case something else is going on, because it "doesn't work on my machine".

You also need to set the second option, "Color".

I got the .deb file (from the link on the github readme) and installed it via software center.
And I can confirm that setting "color" to "system" also doesn't work (as in inheriting it).
If I set it to OLED it works, but that's because then the theme is also set to dark (which is also a bit weird). This even happens when the theme is set to "light".

Ok, so it's no sandbox issue.
My guess as to what's happening is that LocalSend uses the standardized portal for determining whether to use a light or dark appearance. But maybe XFCE isn't updating the portal value. As I understand it, XFCE doesn't have a distinct light/dark mode, it just uses a GTK theme.
Maybe try running gsettings set org.gnome.desktop.interface color-scheme prefer-dark

But I have a system theme.
What does it matter to the application whether that happens to be light or dark.
gsettings get org.gnome.desktop.interface color-scheme
'default'

So I guess that's the how. But still. It's not supposed to use a default, I have a color scheme defined.
I get that that's not your fault and I'm actually curious where and how these things work. I would really like to help and push for clarification / write code to solve this issue.

In the past (and into the present), many apps would try to determine whether to be light or dark based on your GTK theme. That mostly works, but it's a bit inelegant because not all desktops use GTK and so may not set a theme.
The portal method is meant to standardize things. Rather than analyzing the GTK theme, an app just asks the portal "should I use a light or dark theme" and the portal responds with a yes or no.
But XFCE is a stable but slow moving project, so  I guess they just haven't gotten around to implementing good support for the portal. Ideally XFCE should be detecting if your GTK theme is light or dark and update the portal value accordingly.
The "default" value for Gnome is a light theme, so the portal is telling LocalSend to use a light theme. Setting it to "prefer-dark" should tell LocalSend to use a dark theme instead.


Reveal in File Explorer should reuse existing windows on Windows

On Windows, if there's already a File Explorer open, doing Reveal in File Explorer should open a new tab in the existing window.
@black-pwq if there are multiple windows, I don't care about which one is chosen (though some heuristics are arguably better than others, of course). The point is exactly that I want to have at most one File Explorer window open. Either I open File Explorer explicitly, or I open it via VS Code. And if VS Code were to implement this, I'd have the desired behavior

That's not a general idea. If there are multiple opening explorer windows, which one to choose?

This feature request has not yet received the 20 community upvotes it takes to make to our backlog. 10 days to go. To learn more about how we handle feature requests, please see our documentation.
Happy Coding!

:slightly_frowning_face: In the last 60 days, this feature request has received less than 20 community upvotes and we closed it. Still a big Thank You to you for taking the time to create this issue! To learn more about how we handle feature requests, please see our documentation.
Happy Coding!


[Bug] Own 2B model is crashing with errors on Snapdragon 8 gen 2 Android phones

🐛 Bug
When I deploy my own 2B model using MLC on Android, the model interface initializes successfully and displays the "Ready to chat" prompt after opening. However, the app crashes after sending a message, and the phone's developer mode reports the following error.
org.apache.tvm.Base$TVMError: TVMError: OpenCL build error for device=0x7d49dc1aa8
Error: CL_OUT_OF_HOST_MEMORY
Pass
	at org.apache.tvm.Base.checkCall(Base.java:173)
	at org.apache.tvm.Function.invoke(Function.java:130)
	at ai.mlc.mlcllm.JSONFFIEngine.runBackgroundLoop(JSONFFIEngine.java:64)
	at ai.mlc.mlcllm.MLCEngine$backgroundWorker$1.invoke(MLCEngine.kt:42)
	at ai.mlc.mlcllm.MLCEngine$backgroundWorker$1.invoke(MLCEngine.kt:40)
	at ai.mlc.mlcllm.BackgroundWorker$start$1.invoke(MLCEngine.kt:19)
	at ai.mlc.mlcllm.BackgroundWorker$start$1.invoke(MLCEngine.kt:18)
	at kotlin.concurrent.ThreadsKt$thread$thread$1.run(Thread.kt:30)

I tested with three Android phones: Redmi K70 (Snapdragon 8 Gen 2, 16GB RAM) , vivo iQOO Neo8 (Snapdragon 8+, 12GB RAM) and Mate40 Pro (Kirin 9000, 8GB RAM). Surprisingly, the error occurs on both Snapdragon phones, preventing smooth conversations, but it works fine on the Mate40 Pro.
I converted the model using q4f16_1, and the gen_config is based on the qwen2 template. After converting the model weights, I tested using mlc_llm chat without any issues and was able to chat smoothly. However, after building the Android project, I encountered the problem mentioned above. Since testing passed on a phone with 8GB of RAM, I believe the issue might not be due to insufficient memory as the error suggests. I checked previous issues and found that someone reported the same error back in March, but it has been fixed in the latest version.
My custom 2B model configuration is as follows. It is a model obtained by modifying the layer structures of LlamaForCausalLM and then pre-training.
{
  "_name_or_path": "/home/jovyan/wangn/iter900000",
  "architectures": [
    "LlamaForCausalLM"
  ],
  "attention_bias": false,
  "attention_dropout": 0.0,
  "bos_token_id": 0,
  "eos_token_id": 9,
  "hidden_act": "silu",
  "hidden_size": 1536,
  "initializer_range": 0.02,
  "intermediate_size": 10240,
  "max_position_embeddings": 32768,
  "mlp_bias": false,
  "model_type": "llama",
  "num_attention_heads": 16,
  "num_hidden_layers": 32,
  "num_key_value_heads": 4,
  "pretraining_tp": 1,
  "rms_norm_eps": 1e-05,
  "rope_scaling": null,
  "rope_theta": 5000000.0,
  "tie_word_embeddings": false,
  "torch_dtype": "bfloat16",
  "transformers_version": "4.38.2",
  "use_cache": false,
  "vocab_size": 153600
}

This issue has been troubling me for several days. Has anyone else encountered the same problem?
To Reproduce
Steps to reproduce the behavior:

I followed the instructions at https://llm.mlc.ai/docs/install/mlc_llm.html#mlcchat-build-from-source to set up the Python environment and dependencies. I am using Python 3.10 on a Linux system with a CPU.
I followed the instructions at https://llm.mlc.ai/docs/deploy/android.html to package the Android project and imported my own model weights into the phone.

Environment

Platform (e.g. WebGPU/Vulkan/IOS/Android/CUDA): Android
Operating system (e.g. Ubuntu/Windows/MacOS/...): Ubuntu
Device (e.g. iPhone 12 Pro, PC+RTX 3090, ...):
 Does the error also exist if you run some prebuilt existing model? For example like phi3.5, llama3 2B, etc.


log the location where the error occured

Codecov Report

Merging #967 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master     #967   +/-   ##
=======================================
  Coverage   77.26%   77.26%
=======================================
  Files          52       52
  Lines        5572     5572
=======================================
  Hits         4305     4305
  Misses       1267     1267

Impacted Files
Coverage Δ

vent/api/plugins.py
80.36% <ø> (ø)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update f11216e...a16a243. Read the comment docs.


How to disable mod pans notification to IOS using mod_apns.erl file

Hello @mrDoctorWho @zinid
How to disable notification in IOS using mod_apns.erl file and I am using mysql database.
We are using below mod_apns.erl file.
mod_apns.erl - https://github.com/joanlopez/ejabberd-mod-apns/blob/master/src/mod_apns.erl
there is always add notification_enabled = 1

How can I prevent notification if I will change notification_enabled = 0?
I am not receiving group chat notification using mod_apns.erl even notification group flag enabled.

notification_group_enabled = 1
Please help me for 1 and 2 questions.

Closing as duplicate of #27.


Multiple Subscriptions not being loaded in the Database

The recommendations table currently only shows the Subscription name of the  Subscriptions where the AOE solution resides, as shown below

We have more than 10 Subscriptions, so how can this data get populated?
I am not sure if this is related to an earlier issue that I have reported #80

Please refer to the article in the documentation dedicated to your issue.

This was already done., but still shows the same in the DB. Please advise where I need to look further to find the root cause.

@arunjvai , can you please run the following query in the AOE Log Analytics workspace? It should return all your subscriptions and their respective names, provided you assigned the Reader role to the AOE MU in those subscriptions or in a management group containing them.
AzureOptimizationResourceContainersV1_CL
| where TimeGenerated > ago(2d)
| where ContainerType_s =~ 'microsoft.resources/subscriptions'
| summarize arg_max(TimeGenerated, *) by SubscriptionGuid_g
| project SubscriptionGuid_g, SubscriptionName = ContainerName_s

Yes, the query that you provided returns the Subscription GUID and name successfully.
I have already given the AA system level Managed Identity Reader at the Root Management Group, this has been applied also to the VM's Managed Identity Role. Can you please expand on the  Abbreviation MU that you were referring to?

The MU abbreviation was a typo. I meant MI (Managed Identity). It seems you have all the conditions for the subscription name to populate in the database. Bear in mind that some recommendations do not have an associated subscription. This is the case of Azure AD service principal credentials recommendations, for example. But, for the vast majority of recommendations, the subscription name should be there. Can you do the following additional checks?

In the recommendationsexports container of the AOE storage account, open one of the advisor-asis-*.json blobs and check whether all entries have a SubscriptionName property correctly filled.
In the AOE Log Analytics workspace, run the query below. It should return all the subscription names that have been generating recommendations.

AzureOptimizationRecommendationsV1_CL
| where TimeGenerated > ago(8d) and isnotempty(SubscriptionName_s)
| distinct SubscriptionName_s

Open the Recommendations Workbook available in the AOE Log Analytics workspace. You should find there all the subscriptions in the Subscriptions filter.

If you're still seeing a single subscription name in all the 3 tests, than it means there's only one subscription generating recommendations. All the others have no conditions for recommendations at the moment.

So, I can see the Subscription names in the  advisor-asis-*.json blobs and I can also see the subscription names while running the Kusto query in the mentioned LAW. The Recommendations Workbook also lists the Subscriptions in the Subscriptions Filter.
Just checked the Recommendations Table in the DB again and I still see the NULL values under the Subscription Column.

I am waiting for the schedule to run tomorrow for the job Ingest-RecommendationsToSQLServer, and see the result as it has been failing till now due to various issues. But as you had mentioned, I was able to see only two entries by running the query you had mentioned.

Subscription name where the AOE solution has been installed.
Null value

If the NULL value signifies the tenant level recommendations then what is the process to enable this at each subscription level ? Or do I need to wait for this job to run and check this again ?

The tenant-level recommendations are generated by the Recommend-AADExpiringCredentialsToBlobStorage runbook. All the other Recommend-* generate subscription-level recommendations,. The Ingest-RecommendationsToSQLServer runbook ingests all those recommendations into the SQL Database. Let's wait for this job to run successfully and, hopefully, you'll see all the subscriptions in the Power BI report.

@arunjvai, did the recommendations database eventually got populated with additional subscriptions? Can we close this issue?

Yes, it has been reflected.


Kubelogin is not able to convert the kubeconfig to azurecli format for 1.24.0 AKS cluster

Step Performed:

az login
az aks get-credentials -g "abc" -n  "def"
kubelogin convert-kubeconfig -l azurecli

Actual config in the .kube/config file
- name: clusterUser_rc-a-aks-1
  user:
    exec:
      apiVersion: client.authentication.k8s.io/v1beta1
      args:
      - get-token
      - --environment
      - AzurePublicCloud
      - --server-id
      - xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
      - --client-id
      - xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
      - --tenant-id
      - xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
      - --login
      - devicecode
      command: kubelogin
      env: null
      provideClusterInfo: false

Expected Config:
- name: clusterUser_rc-zaks-1
  user:
    exec:
      apiVersion: client.authentication.k8s.io/v1beta1
      args:
      - get-token
      - --server-id
      - xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
      - --login
      - azurecli
      command: kubelogin
      env: null
      provideClusterInfo: false

Environment (please complete the following information):

Kubectl Client CLI Version - 1.24.1
Kubernetes version [e.g. 1.24.0]
Kubelogin -
git hash: v0.0.13/52e83a071f39e9e039e95aa9a6fbea04855eae13
Go version: go1.17.9
Build time: 2022-04-23T00:49:14Z

+1, discovered same issue today after upgrading to 1.24.0

since 1.24.0, AKS default returns exec format kubeconfig so that it doesn't actually need kubelogin conversion. currently, kubelogin's convert-kubeconfig sub command doesn't do anything unless the kubeconfig is pre 1.24 format using azure auth. one can argue that convert-kubeconfig should probably convert the exec format as well to support different login method.

@gyuvaraj10, the kubeconfig you got from 1.24 cluster should be as good unless you require to use azurecli login?

@weinong I like to connect to cluster and run some commands (example: 'kubectl get nodes') with in a CI pipeline. So, I want to connect to the cluster in non-interactive mode.  Can you suggest the approach for the non-interactive communication to the cluster if kubelogin doesn't support.

@gyuvaraj10 how would azurecli mode help? do you use a service principal to login?

Azurecli can be use in a non-interactive mode, as per https://github.com/Azure/kubelogin#azure-cli-token-login-non-interactive

@weinong if we are performing az cli authentication using SPN and get the config using SPN then expectation would be that Kubectl commands execution happens non-interactively which is not the case. its asking us to sign in (SPM is meant for M2M)
Even though the  config is downloaded in the context of authenticated SPN, the EXEC has property --login with value "devicecode" which is leading to interactive mode

Hi,
Please try with v0.0.14 release and let us know if this can be closed.

Just recently updated kubelogin. On a 1.22 AKS cluster, convert-kubeconfig for azurecli is no longer working for me. The command runs fine, but any kubectl command I get the deprecation warning message as if I never ran the convert command.

You are getting the same result as on my end.
❯ kubectl version --short
Flag --short has been deprecated, and will be removed in the future. The --short output will become the default.
W0708 15:12:34.922692  125926 azure.go:92] WARNING: the azure auth plugin is deprecated in v1.22+, unavailable in v1.25+; use https://github.com/Azure/kubelogin instead.
To learn more, consult https://kubernetes.io/docs/reference/access-authn-authz/authentication/#client-go-credential-plugins
Client Version: v1.24.2
Kustomize Version: v4.5.4
Server Version: v1.22.6
WARNING: version difference between client (1.24) and server (1.22) exceeds the supported minor version skew of +/-1

The behavior prior to updating was after running kubelogin convert-kubeconfig -l azurecli I no longer needed to authenticate through the browser and the warning message went away. I'm guessing this is due to the change in the kubeconfig.

@davidbgonz i'm not following. are you saying you are still being prompted to login after converting to azurecli using 0.0.14? Would you mind sharing the converted kubeconfig? you can omit the fqdn and CA to make it generic

@weinong That was the case, but actually I just got it working again. There was some funny business going on with my config.
The steps I ran:

deleted my config
added back one my clusters using the az aks get-credentials command
ran kubelogin convert-kubeconfig -l azurecli again

I am not sure why the config file got in such a state. I also had to remove a leftover config.lock that would not clear up. This was something else I noticed after the update but not sure if it's related since I was messing around with the az and kubelogin commands.

@davidbgonz no problem. If you encounter the issue consistently, please open a new issue with the kubeconfig before and after conversion.


Assertion error when using snippetx

I'm getting the following error when I hit alt+x to apply a snippet on a single line of csv data:
Traceback (most recent call last):
File "/Applications/Sublime Text.app/Contents/MacOS/sublime_plugin.py", line 798, in run_
return self.run(edit)
File "snippetx in /Users/natet/Library/Application Support/Sublime Text 3/Installed Packages/SnippetX.sublime-package", line 142, in run
File "snippetx in /Users/natet/Library/Application Support/Sublime Text 3/Installed Packages/SnippetX.sublime-package", line 88, in getData
AssertionError
If I copy the regular expression used in snippetx and attempt to use it in a find dialog, I get an error "Ran out of stack space trying to match the regular expression.
I'm using sublimetext 3 build 3118 on a Mac

hey, thanks for the info and apologies for the late reply.  I'm going to fix this asap.

<EMAIL_ADDRESS>The snippet is a custom snippet:
<snippet>     <content><![CDATA[ update g_job_final_resolved set FIRST_NAME=$2, LAST_NAME=$3, MIDDLE_NAME=$4, PHONE=$5, EMAIL_ADDR=$6 where id_user='$1'; ]]></content>     <!-- Optional: Set a tabTrigger to define how to trigger the snippet -->     <tabTrigger>mscfdata</tabTrigger>     <!-- Optional: Set a scope to limit where the snippet will trigger -->     <!-- <scope>source.python</scope> -->     <scope>source.sql</scope> </snippet>

I think the issue has something to do with the block of text above where I'm attempting to do the snippet replacement. In my file, I have 70 lines of sql statements in a solid block. If I put blank lines at various points in the file, then the replacement works. With the above sample data, if I take the line generated by the snippet and clone it 21 times then attempt to run the snippetx block, I get the error. I'm guessing it has something to do with the number of characters in the block being matched by the regex.

Sorry, I'm natet. That's my other account.


Page rendering is over the perf budget

Our page rendering speed is over budget on Speedtracker - let's investigate how to get it back down.
http://ghp.wellcomecollection.org/speedtracker/

Breaking this down a bit:

[ ] #891
[ ] #892
[ ] #893
[ ] #894

You could use a project board?
https://github.com/wellcometrust/wellcomecollection.org/projects/6

Closing as has been broken down into smaller issues


Clusternator searches for git HEAD needlessly

...and if there are no commits, it fails
$ clusternator deploy -d master

/Users/rafy/.nvm/versions/node/v4.2.1/lib/node_modules/clusternator/node_modules/q/q.js:155
                throw e;
                ^

Error: git rev-parse HEAD terminated with exit code: 128 fatal: ambiguous argument 'HEAD': unknown revision or path not in the working tree.
Use '--' to separate paths from revisions, like this:
'git <command> [<revision>...] -- [<file>...]'

    at failError (/Users/rafy/.nvm/versions/node/v4.2.1/lib/node_modules/clusternator/lib/cli-wrappers/child-process.js:70:11)
    at ChildProcess.<anonymous> (/Users/rafy/.nvm/versions/node/v4.2.1/lib/node_modules/clusternator/lib/cli-wrappers/child-process.js:107:16)
    at emitTwo (events.js:87:13)
    at ChildProcess.emit (events.js:172:7)
    at maybeClose (internal/child_process.js:818:16)
    at Socket.<anonymous> (internal/child_process.js:319:11)
    at emitOne (events.js:77:13)
    at Socket.emit (events.js:169:7)
    at Pipe._onclose (net.js:469:12)

This has been fixed, but it will need to be verified


The import package in the project's go file has the problem of irregular sorting.

@yanyhui Thank you for raising issue.
Would you can add some golang official links about package import definitions?


Enhancement: OTA

Hi @stas-demydiuk ,
The Zigbee2MQTT allows OverTheAir (OTA) updates for certain devices to update their firmware. It would be interesting to expose this functionality from Domoticz.
Proposal:

record in the device structure the need to update the firmware
add switch on the plugin front end to enable the updates (remove MQTT Client ID parameter)
loop over all devices that need an update and trigger this update when needed

The Zigbee2MQTT project indicates that the updates consume quite some bandwidth so should really be a user triggered action.
If you agree I can try some prototyping.

I will support!
And also add to the plugin the construction of a zigbee network.
The prototype is ready:
https://github.com/kovainfo/Domoticz-zigbee2mqtt-network-map
Moreover, you can develop a whole page on managing Zigbee devices through a separate tab (for example, through a custom template)

@kovainfo that looks like a nice addition as well. Please make a separate issue or PR for it so Stas can check it and merge if accepted.

@kovainfo wow, nice Like Hass will Zigbee2mqtt directly support in Domoticz :)

@kovainfo Awesome I was looking for this for a while. About to get it installed just pulling the latest Domoticz dev branch down

may be @stas-demydiuk

@kovainfo Awesome I was looking for this for a while. About to get it installed just pulling the latest Domoticz dev branch down

I hope that @stas-demydiuk  will pick up this project)) and we will get not only a beautiful network map, but also a full interface for zigbee2mqtt
I just don’t know JS very well, and this solution also uses external utilities. Instead of these utilities, everything could be done on the python plugin. For example, creating a service device "coordinator" that would work out various commands such as building a network, device joining, decoupling, updating OTA firmware, etc. And for the user - a separate page for configuring and displaying the entire zigbee2mqtt infrastructure.

I would love to add such custom page to control zigbee2mqtt, but unfortunately currently I don't know any way how to setup communication channel between such page and python plugin.

I would love to add such custom page to control zigbee2mqtt, but unfortunately currently I don't know any way how to setup communication channel between such page and python plugin.

I propose the following communication scheme:
The plugin creates an additional service device of type Text, where the sValue field will be a command field.
Operations to interact with the plugin and MQTT broker can be performed through the domoticz API / JSON url's:
https://www.domoticz.com/wiki/Domoticz_API/JSON_URL's
eg:
https: //  /json.htm?type=command&param=udevice&idx=437&svalue=Delete%200x12312312312321
The plugin receives the change in the sValue field and processes it in accordance with the internal logic.

for example:

Implemented in scope of #291


tokio.rs example doesn't work correctly

The example in examples/tokio.rs doesn't seem to function correctly. Using the latest version of Lapin and RabbitMQ (2.3.1 and 3.12.2 respectively) the example exits as soon as the queue is empty.
As I'm fairly new to Rust, I'm unsure what the issue is exactly, but the intended functionality of the example is presumably - the same as in the other examples - for the process to wait for more messages to enter the queue and consume those as well. Since there is no documentation for set_delegate() I'm unsure whether this is expected behavior and you have to add something to the example to get it to wait for more messages, or if it's supposed to function how I'm describing and this is just a bug.

All am also experiencing the same, any solutions @eliaseinarsson

Adding a future::pending().await in the end so that it runs indefinitely should do the tricks. They're more like code snippets, they're not exactly meant to be ran as is


Covering all the error cases

Hi:
I'm playing around with sh. It's very interesting compared to subprocess. However, there are a few error cases I was trying to explore to make sure I know how to write a Swiss Army knife wrapper around it.
For flexibility reasons, I'm always going to need to use the Command() constructor. It looks like exit codes are well handled but the following are either not documented entirely or I am unsure how to catch them:

sh.CommandNotFound: Currently printing this exception just prints the command. I didn't see this in the Exception section of the docs.
Permission denied: What if the program is not executable or we don't have the privileges to run it? This appears to be caught by 1) above.

Thanks,
-Clint

https://amoffat.github.io/sh/sections/command_class.html#commandnotfound


rework free-regions to be handled more specially

This may not be the most elegant way to do it, but it was the quickest way I saw to get to a viable prototype. Free regions are still considered points in the CFG, but they are not connected to the normal CFG. Instead, during region inference, when we have an outlives relationship A: B @ P, if we find that the end of the function is reachable from P (within B), then we copy all skolemized-end points found in B into A. We also initialize the "inference variable" for free regions to contain the entire CFG + the appropriate skolemized end-point.
There is now error checking (I stole some of @Nashenas88's code for that). It works by designating the region variables that correspond to name regions as "capped", meaning that if the inference constraints would require them to grow, that results in an error. I also overhauled the error-reporting mechanism to be a bit cleaner.
More tests are desperately needed.

Gonna land this.


Special Behavior for Event Picker in Market creation and Outcomes for Multiple Choice Markets

For the multiple choice market type (for all sports not just Football, just using it as an example)The teams being used in the outcome picker associated to the Event can only be used once....so in the example
Which NFL team will win the [Year] [Event]? and Event is AFC Championship, the outcomes should only allw AFC teams to be selected,
****But once a team is selected it needs to be grayed out and cant be added to the list again
we cant allow them to pick the same team more than once....So for who will win the AFC Championship:

Field
Chiefs
Raiders
Chargers
Chiefs
Broncos
Invalid
So in this example outcome #2 and #5 are the same...can't allow that

market creation form won't allow user to proceed if outcomes has duplicates. closing


chore: difficulty fix

Previous fix did not include case when we are syncing. The difficulty correction is now done on restart and when we reach block 155600

prnet cleaned-up successfully


aws-iot-client: add secure tunnel back in

Issue #, if available:
Related to issue https://github.com/awslabs/aws-iot-device-client/issues/411
Description of changes:
Based on comments in https://github.com/awslabs/aws-iot-device-client/issues/411#issuecomment-1867735644 issues with secure tunneling in https://github.com/awslabs/aws-iot-device-client/issues/411 were resolved in 1.19, however the "st" package config wasn't added back in with https://github.com/aws4embeddedlinux/meta-aws/commit/80c74bfdeafef4fb8a32604b72c347a592469e46.
I've verified this configuration works with secure tunneling using the kas config at https://github.com/Trellis-Logic/meta-aws-iot-demo/blob/main/conf/kas/rpi-rauc-aws-master.yml as well as the kirkstone branch at https://github.com/Trellis-Logic/meta-aws-iot-demo/blob/main/conf/kas/rpi-rauc-aws.yml and I'd ultimately like this backported to kirkstone, let me know if you'd like a PR for kirkstone-next, assuming this is the correct change to make.
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

/run


Several Updates to the User Guide

update screenshots to find, findp, findm, help
small restructuring of sentences
update to the sample data upon loading up application for the first time

Codecov Report
Base: 70.44% // Head: 70.44% // No change to project coverage :thumbsup:

Coverage data is based on head (a1a2dce) compared to base (5d9100e).
Patch coverage: 71.42% of modified lines in pull request are covered.

Additional details and impacted files
@@            Coverage Diff            @@
##             master     #251   +/-   ##
=========================================
  Coverage     70.44%   70.44%
  Complexity      610      610
=========================================
  Files            97       97
  Lines          2003     2003
  Branches        236      236
=========================================
  Hits           1411     1411
  Misses          496      496
  Partials         96       96

Impacted Files
Coverage Δ

...seedu/address/logic/commands/AddModuleCommand.java
100.00% <ø> (ø)

.../java/seedu/address/model/util/SampleDataUtil.java
16.66% <ø> (ø)

...eedu/address/logic/commands/EditModuleCommand.java
94.93% <71.42%> (ø)

...java/seedu/address/logic/commands/FindCommand.java
25.00% <0.00%> (ø)

...eedu/address/logic/commands/FindModuleCommand.java
100.00% <0.00%> (ø)

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

:umbrella: View full report at Codecov.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.


Slow component rendering

Having simple component:
var React = require('react');
var Container = React.createClass({

    render: function () {
        return (
            <div>
                Simple container
            </div>
        );
    }
});

module.exports = Container;

render_component takes about 1 second. Is there any way to improve speed of rendering? Setting django_webpack.settings.DEBUG to True doesn`t seem to help.

You can get a small improvement by adding DJANGO_REACT['DEBUG'] = False to your settings, this sets NODE_ENV=production.
I've generally wrapped the rendering in a DEBUG check...
if django_react.settings.DEBUG:
    markup = component.render_container()
else:
    markup = component.render_to_string()

Performance is fast for dev, and on prod - where the rendering typically matters most - you can usually cache the generated markup.
In any case, the rendering cost is high and definitely needs to be improved. I suspect - but haven't profiled - that a lot of the cost is booting up node and loading everything in.
I took a poke around the React gem for Rails and it looks like they spin up a number of JS processes so that they have a pool to utilise on demand (https://github.com/reactjs/react-rails#server-rendering-1). My major road block on pursuing this is a strong desire to avoid the complexity which comes from managing that pool. If there is a lib, or third party solution (supervisor..?) perhaps that would be of use.

I'll leave this open as it's a large problem.
The priority is some sort of profiling suite which can indicate the bottlenecks.

Thanks again, Mark. Will keep my eye on this issue.

Yeah, the overhead on a cold boot of the renderer was the big bottleneck. I got a ~50 fold increase in performance when I switched over to a prototype of a render server which sits alongside the python process and responds over HTTP.
The rendering performance problem is down to the issue that every call to render_component requires a cold boot of node before anything gets rendered. Switching to a persistent, standalone node process would alleviate that overhead.
The only issues I can see with a persistent process sitting alongside the python process are:

Ensuring that the process is closed when the python process is shutdown
Easiest solution appears to be https://docs.python.org/2/library/atexit.html
Bugs as a by-product of JS's mutability and global scope
There isn't really a solution to this, other than folks following best practices.
Port allocation
Probably easiest to simply expose the selected port as a setting and allow users to optionally configure it for their environment.

standalone render server performance as a percentage of django_react.render_component
--------------------------------------------------------------------------------
max: 8.61138243181%
min: 1.86210436925%
avg: 2.3553003031%
median: 2.05074496877%

django_react.render_component
--------------------------------------------------------------------------------
times: [0.21702003479003906, 0.21471405029296875, 0.2179570198059082, 0.21650290489196777, 0.21557211875915527, 0.21570301055908203, 0.21838903427124023, 0.21029996871948242, 0.21512484550476074, 0.21892595291137695, 0.21367788314819336, 0.21283388137817383, 0.21916508674621582, 0.21521496772766113, 0.21786785125732422, 0.21573996543884277, 0.2140181064605713, 0.22343897819519043, 0.2166588306427002, 0.21626591682434082, 0.2243211269378662, 0.214310884475708, 0.2212519645690918, 0.22001218795776367, 0.21599102020263672]
max: 0.224321126938
min: 0.210299968719
mean: 0.216839103699
median: 0.216265916824

render via standalone node instance
--------------------------------------------------------------------------------
times: [0.019317150115966797, 0.004524946212768555, 0.0044269561767578125, 0.005631208419799805, 0.004354953765869141, 0.004359006881713867, 0.004597902297973633, 0.004567861557006836, 0.005520820617675781, 0.00423884391784668, 0.004435062408447266, 0.004353046417236328, 0.004564046859741211, 0.004564046859741211, 0.0044438838958740234, 0.00471806526184082, 0.004436969757080078, 0.004353046417236328, 0.004261970520019531, 0.0042459964752197266, 0.004274129867553711, 0.004177093505859375, 0.004342079162597656, 0.004431009292602539, 0.004540205001831055]
max: 0.019317150116
min: 0.00417709350586
mean: 0.00510721206665
median: 0.00443506240845

I'll update this when I've pushed a functional replacement for the current render process.

@WhackoJacko
The latest codebase and PyPI version spins up a dedicated render server which offers some massive performance boosts.
When rendering a simple 'Hello, World' component, my machine is now averaging 3% of the time formerly required.
By default, there is an initial overhead when the first component is rendered, which is down to the render server being spun up, but every successive render call will be massively faster.
If you want to avoid the first render's overhead and start with a warm render server, define DJANGO_REACT['START_RENDER_SERVER_ON_INIT'] = True, this will cause the render server to boot during your python server's startup.
Here's some perf starts comparing the new render against the older one.
(django-react)~/Projects/django-react $ python django_react/tests/performance/run_tests.py

Running perf test with a warm render server...
DJANGO_REACT['RENDERER'] = 'django_react.render_server.ReactRenderServer' perf test
--------------------------------------------------------------------------------
Total time taken to render a component 25 times: 0.154497146606
Times: [0.06791496276855469, 0.00417780876159668, 0.003203868865966797, 0.0030939579010009766, 0.003473043441772461, 0.004724025726318359, 0.003371000289916992, 0.003133058547973633, 0.005461931228637695, 0.0033960342407226562, 0.0039010047912597656, 0.0035049915313720703, 0.004064083099365234, 0.0034401416778564453, 0.003448963165283203, 0.003170013427734375, 0.003443002700805664, 0.003534078598022461, 0.0033609867095947266, 0.003365039825439453, 0.003216981887817383, 0.0036149024963378906, 0.003699064254760742, 0.00333404541015625, 0.003450155258178711]
Max: 0.0679149627686
Min: 0.003093957901
Mean: 0.00617988586426
Median: 0.00344896316528

Running perf test with a render server booted on demand...
DJANGO_REACT['RENDERER'] = 'django_react.render_server.ReactRenderServer' perf test
--------------------------------------------------------------------------------
Total time taken to render a component 25 times: 0.473686933517
Times: [0.388322114944458, 0.004202127456665039, 0.0031290054321289062, 0.0032699108123779297, 0.0031850337982177734, 0.0045430660247802734, 0.0034651756286621094, 0.0034558773040771484, 0.005578041076660156, 0.003610849380493164, 0.003265857696533203, 0.0032219886779785156, 0.0037429332733154297, 0.0033538341522216797, 0.0035910606384277344, 0.003610849380493164, 0.0033957958221435547, 0.0032958984375, 0.0032989978790283203, 0.003326892852783203, 0.0033888816833496094, 0.0035238265991210938, 0.0032958984375, 0.003248929977416992, 0.003364086151123047]
Max: 0.388322114944
Min: 0.00312900543213
Mean: 0.0189474773407
Median: 0.00338888168335

Running perf test with a renderer which boots node on every render request...
DJANGO_REACT['RENDERER'] = 'django_react.renderer.ReactRenderer' perf test
--------------------------------------------------------------------------------
Total time taken to render a component 25 times: 5.52515673637
Times: [0.2518901824951172, 0.21484017372131348, 0.2172248363494873, 0.21650409698486328, 0.22143292427062988, 0.22011089324951172, 0.21518206596374512, 0.22161293029785156, 0.21700501441955566, 0.21958208084106445, 0.22993087768554688, 0.22188711166381836, 0.22236108779907227, 0.21548700332641602, 0.22033190727233887, 0.2214500904083252, 0.22034907341003418, 0.22076106071472168, 0.21924901008605957, 0.21874308586120605, 0.2151811122894287, 0.21937799453735352, 0.22022104263305664, 0.23113703727722168, 0.2133040428161621]
Max: 0.251890182495
Min: 0.213304042816
Mean: 0.221006269455
Median: 0.22011089325


activity leaks

Hello.
using LeakCanary, I think I have identified more instances of activity leaks when GoogleApiClient is left connected...
IDPSignInContainerActivity can be leaked when mIDPProvider is instanceof GoogleProvider,
because it has not been built with auto manage enabled nor is disconnect being called.
suggest add:
@Override
protected void onStop() {
    super.onStop();
    if ( GoogleProvider.class.isInstance( mIDPProvider ) ) {
        ( ( GoogleProvider)mIDPProvider).disconnect();
    }
}

SaveCredentialsActivity is being leaked because mCredentialsApiClient has not been built with auto manage enabled nor is disconnect being called.
suggest add:
@Override
protected void onStop() {
    super.onStop();
    if ( mCredentialsApiClient != null ) {
        mCredentialsApiClient.disconnect();
    }
}

ChooseAccountActivity is being leaked.
ex. (in onActivityResult, line 203)
mCredentialsApi.handleCredential(credential);

@gubo thanks for calling this out, we should likely move to using automanage internally.

Looks like the "9.2.1" versions became available yesterday. I am now using
// Android Studio v2.2 Preview 4 now works with Firebase and Google Play Services (works in emulator too)
compile 'com.firebaseui:firebase-ui-database:0.4.0'
compile 'com.google.firebase:firebase-core:9.2.1'
compile 'com.google.firebase:firebase-auth:9.2.1'
compile 'com.google.firebase:firebase-database:9.2.1'
compile 'com.google.android.gms:play-services-appindexing:9.2.1'

@gubo this has been fixed in master and the fix will be rolled into the next release.

0.4.4 has been released!


error on google

logger=ngalert.multiorg.alertmanager t=2023-01-15T21:34:33.72+0000 lvl=info msg="starting MultiOrg Alertmanager"
logger=provisioning t=2023-01-15T21:34:33.72+0000 lvl=eror msg="Failed to provision dashboard" error="Failed to create provisioner: Failed to read dashboards config: could not parse provisioning config file: customdashboards.yml error: read /etc/grafana/provisioning/dashboards/customdashboards.yml: is a directory"
logger=server t=2023-01-15T21:34:33.72+0000 lvl=eror msg="Stopped background service" service=*provisioning.ProvisioningServiceImpl reason="Failed to create provisioner: Failed to read dashboards config: could not parse provisioning config file: customdashboards.yml error: read /etc/grafana/provisioning/dashboards/customdashboards.yml: is a directory"
logger=ngalert t=2023-01-15T21:34:33.72+0000 lvl=eror msg="unable to fetch orgIds" msg="context canceled"
logger=http.server t=2023-01-15T21:34:33.72+0000 lvl=info msg="HTTP Server Listen" address=[::]:3000 protocol=http subUrl=/grafana socket=
*provisioning.ProvisioningServiceImpl run error: Failed to create provisioner: Failed to read dashboards config: could not parse provisioning config file: customdashboards.yml error: read /etc/grafana/provisioning/dashboards/customdashboards.yml: is a directory
logger=server t=2023-01-15T21:34:33.72+0000 lvl=eror msg="Server shutdown" error="*provisioning.ProvisioningServiceImpl run error: Failed to create provisioner: Failed to read dashboards config: could not parse provisioning config file: customdashboards.yml error: read /etc/grafana/provisioning/dashboards/customdashboards.yml: is a directory"

fix with entery path full on google gcc


are there any quick wins for reducing the install size of nyc?

When installed, this package and its dependencies come out to more than 25mb, which is really quite large. Just wanted to open up a conversation to see if perhaps there's any way to start slimming it down!

@jescalan I certainly wouldn't complain about having a second set of eyes. Having said this, we've done quite a bit of analysis:

a big chunk of the size is related to istanbul and babel, which we can't easily do away with since it's how everything works.
we bundle all our dependencies for nyc; even though the final size on disk is large, nyc comes down as a tarball in a single HTTP request, so the install time is quite fast.
25MB isn't that big for a development tool; would certainly stink if there was a compelling reason to ship nyc as a front-end package -- but I don't think this will ever be a use-case?

@jescalan were you interested in digging into this a bit?

Yeah so I poked around a little, and it seems like babel is a large part of it (with its dependency on core-js). I guess there is no way to remove this if you want to provide coverage for es6 code, right?

@jescalan correct, and I definitely feel like switching to using babel as the instrumenter has been beneficial to the project... we could dig into https://github.com/istanbuljs/istanbul-lib-instrument, and see if any of the babel dependencies can be trimmed down, we could also dive into the babel project itself and see if there are any tricks we could use to reduce the code size.

I was doing a quick check of my node modules folder and saw that nyc was one of the largest dependencies and saw that this issue had already been opened.
I was wondering if there had been any progress on this - I did a few quick checks to count the number of files and size of each directory and found that nyc downloaded 4000 files with a total size of 25mb.
> find ./node_modules/nyc/ -type f | wc -l
4000

Listing the top 20 node_modules gave the following result:
> du -sk ./node_modules/nyc/node_modules/* | sort -rn | head -n 20
6752	./node_modules/nyc/node_modules/core-js
4964	./node_modules/nyc/node_modules/lodash
3048	./node_modules/nyc/node_modules/handlebars
964	./node_modules/nyc/node_modules/babel-runtime
784	./node_modules/nyc/node_modules/uglify-js
784	./node_modules/nyc/node_modules/source-map
528	./node_modules/nyc/node_modules/yargs
292	./node_modules/nyc/node_modules/babylon
216	./node_modules/nyc/node_modules/babel-traverse
172	./node_modules/nyc/node_modules/babel-types
172	./node_modules/nyc/node_modules/async
156	./node_modules/nyc/node_modules/istanbul-reports
144	./node_modules/nyc/node_modules/optimist
132	./node_modules/nyc/node_modules/babel-generator
104	./node_modules/nyc/node_modules/cliui
100	./node_modules/nyc/node_modules/debug
92	./node_modules/nyc/node_modules/mkdirp
80	./node_modules/nyc/node_modules/istanbul-lib-report
76	./node_modules/nyc/node_modules/istanbul-lib-instrument
76	./node_modules/nyc/node_modules/convert-source-map

Where the numbers are in kb. Doing some checks using npm ls showed this for the top 3:
> cd node_modules/nyc
> npm ls core-js
└─┬<EMAIL_ADDRESS>  └─┬<EMAIL_ADDRESS>    └─┬<EMAIL_ADDRESS>      └──<EMAIL_ADDRESS>> npm ls lodash
├─┬<EMAIL_ADDRESS>│ ├─┬<EMAIL_ADDRESS>│ │ └──<EMAIL_ADDRESS>deduped
│ ├─┬<EMAIL_ADDRESS>│ │ └──<EMAIL_ADDRESS>deduped
│ ├─┬<EMAIL_ADDRESS>│ │ └──<EMAIL_ADDRESS>deduped
│ └─┬<EMAIL_ADDRESS>│   └──<EMAIL_ADDRESS>deduped
└──<EMAIL_ADDRESS>> npm ls handlebars
└─┬<EMAIL_ADDRESS>  └──<EMAIL_ADDRESS>
Looking at istanbul-lib-instrument's package.json (https://github.com/istanbuljs/istanbuljs/blob/master/packages/istanbul-lib-instrument/package.json) it seems like its using older versions of babel which may make a difference in size, but I'm not entirely sure if there are easy wins here. I just wanted to check if someone anyone was still looking at this or looking to optimize the package. Thanks.

Is 61160771 still needed?  I just checked node_modules/ and node_modules/nyc/node_modules/.  69 modules were installed in both locations.  Some may be required due to different versions, but those 69 modules are 7.4 megs in the nyc bundled copy.

install it globally, or better yet, put it in user $HOME
#!/usr/bin/env bash
(
  mkdir $HOME/.my-nyc
  cd $HOME/.my-nyc
  npm init -f
  npm install -S nyc
)

then when you do your thing include that location in your $PATH
#!/usr/bin/env bash

export PATH="$HOME/.my-nyc/node_modules/.bin:${PATH}"
# now run your tests
nyc ...   # nyc will be the one installed in `$HOME/.my-nyc/node_modules/.bin`

@ORESoftware nyc gets listed in devDependencies because it is used by npm test.  Installing it globally would not do anything, the package would still install it locally as a dev dependency and use the local copy instead of the global one.

@coreyfarrell who/what is putting nyc in devDependencies though?

I just looked into the same thing. To be honest, I'm not a fan of bundled dependencies, especially with super common ones as it's the case here. The original reason to do so was to ship a patched version of spawn-wrap. Since that's resolved, maybe it's time to consider that approach?

One thing I've noticed is that nyc seems to require node.js 4.x+.  I created a fork, enabled travis-ci on it with the following changes:

I cleared the devDependencies.
Set the script for npm test to ./bin/nyc.js ./bin/nyc.js --help.

This command fails on nodejs 0.10 and 0.12.  The issue is due to use of ES2015 features both within nyc itself and in dependencies (find-up at minimum).
Am I missing something or is it acceptable to explicitly require node.js >= 4.0 so we can update remaining dependencies?  If so what about https://github.com/istanbuljs/istanbuljs/?  0.10 and 0.12 are EOL long ago, the oldest supported version of node.js is 6.x.  Updating dependencies will reduce duplication, in my own private install I've reduced the npm install --prod size by 16% (from 30MB to 25MB).  I've verified npm test still passes with the updated dependencies.
I still believe that dependencies should not be bundled without a technical reason.  Even if such a reason exists for specific modules most/all others should be unbundled?

@coreyfarrell  why are you stuck on 0.10 or 0.12 ? Those are very old versions now.

@ORESoftware I'm not, I run 8.x.  My hope is that nyc and istanbuljs can drop support for those versions so dependencies can be brought up to date.

Oh I see, that is a good point


add a classification model for places365

create a module for image classification and add an image classification for places365

#55


Add the ability to use a custom reporter

This PR allows the user of koa-logger to tie it into their existing logging infrastructure.
Before:

After:

IMO if it's hard to implement new loggers as regular middleware we're not making it easy enough, some things like the node events for finish etc are really lame, but otherwise I'd say it's best to just implement new middleware for different implementations

Makes sense. Based on your feedback, I instead created custom middleware for my use case: https://github.com/justmoon/koa-mag


Sending data even with Django DEBUG=True

Setting Django DEBUG to True does not stop elastic agent from sending data to APM.
django.VERSION = (2, 2, 10, 'final', 0)
elastic-apm==5.4.3
This all runs in a docker image python:3.8.2

APM server is hosted at Elastic Cloud (they pointed me here), I don't see where to check version, but since it was deployed 2 weeks ago, it should be the latest version (at least there is no option to update right now).
My tests:
lr@lr-local:~/app/kernelratings$ ./manage.py elasticapm check
[2020-03-26 14:58:51,341|INFO|customers.apps] Loading all registries...
[2020-03-26 14:58:55,294|INFO|customers.apps] Loaded all registries in 3.95 seconds!
Service name is set, good job!

SERVER_URL https://************************.apm.eu-west-1.aws.cloud.es.io looks fine

Looks like you're running in DEBUG mode. ElasticAPM will NOT gather any data while DEBUG is set to True.

If you want to test ElasticAPM while DEBUG is set to True, you can force ElasticAPM to gather data by setting ELASTIC_APM["DEBUG"] to True, like this

   ELASTIC_APM = {
       "DEBUG": True,
       # your other ELASTIC_APM settings
   }

Tracing middleware is configured! Awesome!

Please fix the above errors.

Then, I go to Kibana --> APM and I can see there is data sent from that container. I check the container-id reported in APM and it is the id of the container that reported the message pasted above (checked with "docker ps")
If I totally remove the ELASTIC_APM setting (whiwle keeping Django DEBUG=True) and re-run Django, it shows errors (elastic-apm is then trying to connect to localhost..) all the time like these:
[2020-03-24 11:03:51,771|WARNING|urllib3.connectionpool] Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f3afa00fa30>: Failed to establish a new connection: [Errno 111] Connection refused')': /intake/v2/events

[2020-03-24 11:03:39,133|WARNING|urllib3.connectionpool] Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f3afa110940>: Failed to establish a new connection: [Errno 111] Connection refused')': /config/v1/agents

requirements.txt:

numpy==1.18.1
scipy==1.4.1
scikit-learn==0.22.1
pandas==1.0.1
sklearn-pandas==1.8.0
xgboost==0.81
amqp==2.5.2
argparse==1.4.0
billiard==<IP_ADDRESS>
boto==2.49.0
boto3==1.12.7
celery==4.4.1
certifi==2019.11.28
cffi==1.14.0
cryptography==2.8
cython==0.29.15
elastic-apm==5.4.3
Django==2.2.10
django-ace-overlay==0.8.0
django-appconf==1.0.3
django-bootstrap-toolkit==2.15.0
django-celery-beat==2.0.0
django-celery-results==1.2.1
django-compressor==2.4
django-extensions==2.2.8
django-health-check==3.12.1
django-mysql==3.3.0
django-redis==4.11.0
django-statsd-mozilla==0.4.0
django-storages==1.9.1
django-timezone-field==4.0
django-tracking2==0.5.0
docopt==0.6.2
gender-guesser==0.4.0
google-api-python-client==1.7.8
google-measurement-protocol==1.0.0
gunicorn==19.9.0
ipython==7.13.0
jinja2==2.10.1
joblib==0.14.1
jsonschema==3.0.2
kombu==4.6.8
libsaas==0.4
maxminddb==1.4.1
meld3==2.0.0
mysqlclient==1.4.6
ndg-httpsclient==0.5.1
nltk==3.4.4
openpyxl==3.0.3
Pillow==6.1.0
psutil==5.6.3
pycparser==2.19
pycurl==<IP_ADDRESS>
Pygments==2.4.2
pymailcheck==1.0.0
pyOpenSSL==19.0.0
python-dateutil==2.8.1
pytz==2019.3
pyzmq==18.1.0
redis==3.4.1
requests==2.23.0
rollbar==0.14.7
setproctitle==1.1.10
simple-salesforce==0.74.2
six==1.12.0
South==1.0.2
statsd==3.3.0
tornado==6.0.3
ua-parser==0.7.1
Unidecode==1.1.1
urllib3==1.25.6
user_agents==1.1.0
xlrd==1.2.0

Hi @xavier-lr
what kind of data do you see in Kibana for that container? Just metrics or also errors and transactions?

Right now it looks like it's just metrics.

This is expected behavior. Because metrics don't collect app-specific data we continue to send them regardless of the Django DEBUG setting.
We're going to be adding a RECORDING config very soon for pausing the agent entirely. Stay tuned: #575

(For now if you use ACTIVE=False you can also turn off the agent that way. It's just a hard off where the agent won't even get remote config from Kibana. RECORDING will be a soft off.)

(For now if you use ACTIVE=False you can also turn off the agent that way. It's just a hard off where the agent won't even get remote config from Kibana. RECORDING=False will be a soft off.)

On my Django settings for dev envs, I add ACTIVE=False:
ELASTIC_APM = {
    'SERVICE_NAME': docker_service,
    'SECRET_TOKEN': '**************',
    # 'SERVER_URL':
    #     '*******.apm.eu-west-1.aws.cloud.es.io', # noqa
    'ENVIRONMENT': None,
    'ACTIVE': False,
}

But it does not seem to have any effect, it's still trying to connect all the time.
WARNING|urllib3.connectionpool] Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fb09edf6370>: Failed to establish a new connection: [Errno 111] Connection refused')': /intake/v2/events

Hi @xavier-lr
The recording flag that @basepi mentioned is almost finished and available in PR #790. Would you mind giving that a try to see if it fulfills your use case? You can install it like this:
pip install "https://github.com/beniwohli/apm-agent-python/archive/enabled-recording.zip#egg=elastic-apm==100"

Then you can set enabled to false to completely disable the agent. We will deprecate active in one of our next releases.

I finally found that these settings disable all the attempts to connect to APM:
    'DISABLE_SEND': True,
    'CENTRAL_CONFIG': False,


Drush Error: unknown command "drush" for "ddev" on second ddev drush st

Is there an existing issue for this?

[X] I have searched the existing issues

Output of ddev debug test
Expand `ddev debug test` diagnostic information
[COPY-PASTE HERE THE OUTPUT OF `ddev debug test`]
➜  bo_drupal git:(main) ✗ ddev debug test
Running bash [-c /tmp/test_ddev.sh]
======= Existing project config =========
These config files were loaded for project bmp-bo<EMAIL_ADDRESS><EMAIL_ADDRESS>name: bmp-bo
type: drupal10
docroot: web
php_version: 8.1
webserver_type: nginx-fpm
webimage: ddev/ddev-webserver:v1.22.2
database: {mariadb 10.4}
hooks: map[post-start:[map[exec:platform self:update -qy --no-major || true] map[exec:mkdir -p ${PLATFORM_CACHE_DIR} || true] map[exec:[ ! -z "${PLATFORMSH_CLI_TOKEN:-}" ] && (platform ssh-cert:load  -y || true)] map[composer:install] map[exec:set -e
] map[exec:set -e
php ./drush/platformsh_generate_drush_yml.php
# if drupal is installed, will call the following drush commands:
#   - `cache-rebuild`
#   - `updatedb`
#   - and if config files are present, `config-import`
cd web
bash $PLATFORM_APP_DIR/drush/platformsh_deploy_drupal.sh
] map[exec:phpenmod blackfire] map[exec:platform self:update -qy --no-major || true] map[exec:mkdir -p ${PLATFORM_CACHE_DIR} || true] map[exec:[ ! -z "${PLATFORMSH_CLI_TOKEN:-}" ] && (platform ssh-cert:load  -y || true)] map[composer:install] map[exec:set -e
] map[exec:set -e
php ./drush/platformsh_generate_drush_yml.php
# if drupal is installed, will call the following drush commands:
#   - `cache-rebuild`
#   - `updatedb`
#   - and if config files are present, `config-import`
cd web
bash $PLATFORM_APP_DIR/drush/platformsh_deploy_drupal.sh
] map[exec:phpenmod blackfire]]]
webimage_extra_packages: [figlet php8.1-redis php8.1-apcu php-sodium]
project_tld: ddev.site
use_dns_when_possible: true
composer_version: 2.1
disable_settings_management: true
nodejs_version: 18
default_container_timeout: 120
======= Creating dummy project named  tryddevproject-23547 in ../tryddevproject-23547 =========
OS Information: Linux PMLSCAL35 5.15.0-88-generic #98-Ubuntu SMP Mon Oct 2 15:18:56 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
User information<EMAIL_ADDRESS>gid=448400513(utilisa. du<EMAIL_ADDRESS>groupes=448400513(utilisa. du domaine@telegramme.fr),4(adm),24(cdrom),27(sudo),30(dip),46(plugdev),109(kvm),115(lpadmin),136(sambashare),998(docker),448411284(gg photos@telegramme.fr),448411285(gld photos@telegramme.fr),448413935(gg tariectse@telegramme.fr),448413936(gld tariectse@telegramme.fr),448414845(gg production@telegramme.fr),448414846(gld production@telegramme.fr),448415373(gg ftp saisiepaextcom@telegramme.fr),448415374(gld ftp saisiepaextcom@telegramme.fr),448415939(gg etudes@telegramme.fr),448415940(gld etudes@telegramme.fr),448416084(gg traficfab trafic@telegramme.fr),448416085(gld traficfab trafic@telegramme.fr),448416506(gg imprimante impsecure@telegramme.fr),448416507(gld imprimante impsecure@telegramme.fr),448422969(reportinggroup {110d66ed-65dc-4640-9b06-4af654ec25cb}@telegramme.fr),448423148(gld nouveaux contenus@telegramme.fr),448423150(gg nouveaux contenus@telegramme.fr),448424621(gg pso svc info@telegramme.fr),448424622(gld pso svc info@telegramme.fr),448427783(gld trevors tse@telegramme.fr),448427784(gg trevors tse@telegramme.fr),448428148(gld verdon tse@telegramme.fr),448428149(gg verdon tse@telegramme.fr),448428885(gg utilisateurs vra@telegramme.fr),448428886(gld utilisateurs vra@telegramme.fr),448431641(gg wiki dsi@telegramme.fr),448431642(gld wiki dsi@telegramme.fr),448444496(gg gsuite enterprise<EMAIL_ADDRESS>DDEV version:  ITEM             VALUE
 DDEV version     v1.22.2
 architecture     amd64
 db               ddev/ddev-dbserver-mariadb-10.4:v1.22.2
 ddev-ssh-agent   ddev/ddev-ssh-agent:v1.22.2
 docker           24.0.5
 docker-compose   v2.21.0
 docker-platform  Linux Mint 21.2
 mutagen          0.17.2
 os               linux
 router           ddev/ddev-traefik-router:v1.22.2
 web              ddev/ddev-webserver:v1.22.2
PROXY settings: HTTP_PROXY='' HTTPS_PROXY='' http_proxy='' NO_PROXY=''
======= DDEV global info =========
Global configuration:
instrumentation-opt-in=true
omit-containers=[]
performance-mode=none
router-bind-all-interfaces=false
internet-detection-timeout-ms=3000
disable-http2=false
use-letsencrypt=false
letsencrypt-email=
table-style=default
simple-formatting=false
auto-restart-containers=false
use-hardened-images=false
fail-on-hook-fail=false
required-docker-compose-version=v2.21.0
use-docker-compose-from-path=false
project-tld=
xdebug-ide-location=
no-bind-mounts=false
router=traefik
wsl2-no-windows-hosts-mgt=false
router-http-port=80
router-https-port=443
mailpit-http-port=8025
mailpit-https-port=8026
traefik-monitor-port=10999

======= DOCKER info =========
docker location: -rwxr-xr-x 1 root root 26011832 août  21 21:50 /usr/bin/docker
docker version:
Client:
 Version:           24.0.5
 API version:       1.43
 Go version:        go1.20.3
 Git commit:        24.0.5-0ubuntu1~22.04.1
 Built:             Mon Aug 21 19:50:14 2023
 OS/Arch:           linux/amd64
 Context:           default

Server:
 Engine:
  Version:          24.0.5
  API version:      1.43 (minimum version 1.12)
  Go version:       go1.20.3
  Git commit:       24.0.5-0ubuntu1~22.04.1
  Built:            Mon Aug 21 19:50:14 2023
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.7.2
  GitCommit:
 runc:
  Version:          1.1.7-0ubuntu1~22.04.1
  GitCommit:
 docker-init:
  Version:          0.19.0
  GitCommit:
DOCKER_DEFAULT_PLATFORM=notset
======= Mutagen Info =========
======= Docker Info =========
Docker platform: Linux Mint 21.2
Using Docker context: default (unix:///var/run/docker.sock)
docker-compose: v2.21.0
Using DOCKER_HOST=unix:///var/run/docker.sock
Docker version: 24.0.5
Able to run simple container that mounts a volume.
Able to use internet inside container.
Docker disk space:
Filesystem                Size      Used Available Use% Mounted on
overlay                 233.2G    191.1G     30.1G  86% /

 Container ddev-bmp-bo-redis  Stopped
 Container ddev-bmp-bo-db  Stopped
 Container ddev-bmp-bo-web  Stopped
 Container ddev-bmp-bo-db  Stopped
 Container ddev-bmp-bo-web  Stopped
 Container ddev-bmp-bo-redis  Stopped
 Container ddev-bmp-bo-db  Removed
 Container ddev-bmp-bo-redis  Removed
 Container ddev-bmp-bo-web  Removed
 Network ddev-bmp-bo_default  Removed
Project bmp-bo has been stopped.
Removing container: ddev-bodrupal-db
Removing container: ddev-bodrupal-web
Project bodrupal has been stopped.
 Container ddev-quoi29-db  Stopped
 Container ddev-quoi29-web  Stopped
 Container ddev-quoi29-web  Stopped
 Container ddev-quoi29-db  Stopped
 Container ddev-quoi29-db  Removed
 Container ddev-quoi29-web  Removed
 Network ddev-quoi29_default  Removed
Project quoi29 has been stopped.
The ddev-ssh-agent container has been removed. When you start it again you will have to use 'ddev auth ssh' to provide key authentication again.
Existing docker containers:
CONTAINER ID   IMAGE     COMMAND   CREATED   STATUS    PORTS     NAMES
Network ddev_default created

 TIP OF THE DAY
 Sign up for the DDEV newsletter! https://ddev.com/newsletter/

Starting tryddevproject-23547...
 Container ddev-ssh-agent  Created
 Container ddev-ssh-agent  Started
ssh-agent container is running: If you want to add authentication to the ssh-agent container, run 'ddev auth ssh' to enable your keys.
 Network ddev-tryddevproject-23547_default  Created
 Container ddev-tryddevproject-23547-web  Created
 Container ddev-tryddevproject-23547-db  Created
 Container ddev-tryddevproject-23547-web  Started
 Container ddev-tryddevproject-23547-db  Started
 Container ddev-router  Created
 Container ddev-router  Started
Successfully started tryddevproject-23547
Project can be reached at https://tryddevproject-23547.ddev.site https://<IP_ADDRESS>:32792
======== Curl of site from inside container:
HTTP/1.1 200 OK
Server: nginx
Date: Thu, 09 Nov 2023 15:24:26 GMT
Content-Type: text/html; charset=UTF-8
Connection: keep-alive
Vary: Accept-Encoding

======== curl -I of http://tryddevproject-23547.ddev.site from outside:
HTTP/1.1 200 OK
Content-Type: text/html; charset=UTF-8
Date: Thu, 09 Nov 2023 15:24:26 GMT
Server: nginx
Vary: Accept-Encoding

======== full curl of http://tryddevproject-23547.ddev.site from outside:
Success accessing database... db via TCP/IP
ddev is working. You will want to delete this project with 'ddev delete -Oy tryddevproject-23547'
======== Project ownership on host:
drwxr-xr-x 4<EMAIL_ADDRESS>utilisa. du<EMAIL_ADDRESS>4096 nov.   9 16:23 ../tryddevproject-23547
======== Project ownership in container:
drwxr-xr-x 4<EMAIL_ADDRESS><EMAIL_ADDRESS>4096 Nov  9 15:23 /var/www/html
======== In-container filesystem:
Filesystem     Type 1K-blocks      Used Available Use% Mounted on
/dev/nvme0n1p2 ext4 244506940 200533600  31480284  87% /var/www/html
======== curl again of tryddevproject-23547 from host:
Success accessing database... db via TCP/IP
ddev is working. You will want to delete this project with 'ddev delete -Oy tryddevproject-23547'
Thanks for running the diagnostic. It was successful.
Please provide the output of this script in a new gist at gist.github.com
Running ddev launch in 5 seconds

Please run cleanup after debugging with 'ddev debug testcleanup'

Expected Behavior
that command ddev drush xxx works
Actual Behavior
➜  bo_drupal git:(main) ✗ ddev drush generate module

 Welcome to module generator!
––––––––––––––––––––––––––––––

 Module name:
 ➤ Failed to run drush generate module: exit status 137
➜  bo_drupal git:(main) ✗ ddev drush generate module
Error: unknown command "drush" for "ddev"

Did you mean this?
        push

Run 'ddev --help' for usage.

Steps To Reproduce
No response
Anything else?
No response

Hi @stloc - I think you executed the command:

outside a project directory or
in a project that is not of type drupal*

Hi @rfay
sudo apt update && sudo apt upgrade -y ddev
ddev is already the most recent version (1.22.2).

You have more than one ddev installed I imagine. The current version of DDEV is v1.22.4.
which ddev will tell you the one you're using. If it's /usr/local/bin/ddev, then delete it and use the one you've installed properly via apt.
sudo rm /usr/local/bin/ddev
hash -r
which ddev
ddev --version

I hope this is working for you but hope you've upgraded to current, now DDEV v1.22.5, and would love to hear more about this. Happy to continue the conversation here or in Discord, https://discord.gg/hCZFfAMc5k
Closing for now.

@rfay I ran into same issue via a config_switch script to mimic ci/cd of gitlab.
did an upgrade:
Upgrading ddev/ddev/ddev
1.22.7 -> 1.23.1
Still, when I run ddev config/switch, it will composer install but than breaks on drush with same error:
Error: unknown command "drush" for "ddev"
Did you mean this?
push
Run 'ddev --help' for usage.
Failed to run config_switch ; error=exit status 255
If I run config/switch again, it picks up where it left off but this time finds drush.
the config_switch executes:
`#!/usr/bin/env bash
Switch sites and import configuration.

Usage: config/switch
Abort if anything fails
set -e
echo -e "${green_bg} Step 1 ${NC} ${green}Which environment do you want to switch to? ${NC}"
PS3='Select numeric option from above: '
options=("local" "dev" "prod")
select opt in "${options[@]}"
do
case $opt in
"local")
env='local'
break
;;
"dev")
env='dev'
break
;;
"prod")
env='prod'
break
;;
esac
done
ddev config --web-environment=CURRENT_SITE=${env}
ddev restart
ddev composer install
ddev drush cr
ddev drush cim -y
ddev drush updb -y
ddev drush cr
ddev drush cache-clear plugin
ddev exec git restore .ddev/config.yaml
`
Since it always seems to work with next run, I tried adding an additional ddev restart after composer install but still same error
So one last ditch effort by also adding ddev exec git restore .ddev/config.yaml before the 2nd ddev restart, and whala, that works...

I am having the same issue on a fresh install of Drupal 10. I am using the CMS Quickstart guide for Drupal 10 and have gotten through "ddev composer require drush/drush" just fine. But when I run the next step, I get the "unknown command 'drush' for 'ddev'" error, and if I add "exec" to the command, then I get "[preflight] Package "drupal/core" is not installed."

Well, you apparently haven't done a ddev composer install, or you left out the ddev composer create drupal/recommended-project. It's telling you that drush can't run because you don't have Drupal in there. I imagine that because you didn't get Drupal installed, your project is probably of type php (look in the .ddev/config.yaml, or ddev describe. As discussed above, a project of type php doesn't offer drush.

Ack, you're right; I missed a step! Thanks for getting me back on track. I'm all set now!


Provide json schemas for webhook payloads

Each webhook payload should have a jsonschema available in the codebase to provide documentation, validation, and guidance to developers writing consumers for the webhooks.

Reviewing the code, it seems that there's only 1 format for the webhook payload JSON.
Subscriptions
In the Catalog Initialization there's a scheduled task for handling notifications from the queue that both populates messages in the queues table, and then sends them out via notifications.
So the actual webhook that gets sent out appears to be something like this, which is a combination of the columns in QueueItem and what is generated from notifications.make_notification
I did also notice that there are other places where a notification can get queued, but not seeing where a consumer of the queue would actually send out a webhook:
Events
Pretty much whenever an event is generated, but the format should be pretty consistent: add_event_json
Images
Whenever an image tag gets updated?  In the image watcher
Are there any other areas where a possibly different schema could get generated for a webhook? And do the Events/Images notifications get sent via webhooks ever?

Hmm. After getting an analysis_update event to fire for a new image, got a payload like this delivered:
{
   "record_state_key":"active",
   "created_at":1597085212,
   "userId":"admin",
   "tries":0,
   "last_updated":1597085212,
   "dataId":"a12dc2f2-84be-4aef-9b6a-434299c6ae74",
   "queueId":"analysis_update",
   "max_tries":1597085242,
   "data":{
      "notification_user":"admin",
      "notification_user_email":"admin@myanchore",
      "notification_type":"analysis_update",
      "notification_payload":{
         "notificationId":"a12dc2f2-84be-4aef-9b6a-434299c6ae74",
         "userId":"admin",
         "subscription_key":"docker.io/postgres:9",
         "curr_eval":{
            "analysis_status":"analyzed",
            "annotations":{

            },
            "imageDigest":"sha256:570431d9c2f68af2f1490e6de30723711a42550a6f063144b324b993d3529eef"
         },
         "last_eval":{
            "analysis_status":"analyzing",
            "annotations":{

            },
            "imageDigest":"sha256:570431d9c2f68af2f1490e6de30723711a42550a6f063144b324b993d3529eef"
         },
         "subscription_type":"analysis_update"
      }
   }
}

So, it seems like the notification_payload can vary a bit beyond the standard fields.