What versions of .NET are supported?

What versions of .NET are supported? We have a problem with Zoom control in .NET 4.0 application.
Thanks.

Hi, GraphX supports .NET Framework version 4.0.3 and higher. What kind of issue you have with ZoomControl?

Seems that we have problems because we build all projects in .NET 4.0, not in 4.0.3 for UI projects.  For 4.0.3 everything works fine. Thank you.
Error message : The tag 'ZoomControl' does not exist in XML namespace 'clr-namespace:GraphX.Controls;assembly=GraphX.WPF.Controls'

I'm glad you've managed to solve this issue. If you'll have any questions please feel free to ask.


Splitting of stack creation and alignment

Overview

Changes in codes (primarily main and align) to split IMOD processes of stack creation and alignment
Added new entry point in setup (o2r.align.stacks) to call function

New usage caveats

Still need to create input YAML using o2r.align.new
o2r.align.stacks only creates .st files --- No metadata updates .
If user wants to carry on with IMOD alignment, use o2r.align.run_ext as if stacks are from external sources. Metadata updated here for downstream O2R processes
Old command o2r.align.run still works, but it is now a 2-in-1 command --- o2r.align.stacks + o2r.align.run_ext

Codecov Report

Merging #25 (179fa45) into main (66c97e4) will increase coverage by 0.01%.
The diff coverage is 28.57%.

:exclamation: Current head 179fa45 differs from pull request most recent head 374b5bf. Consider uploading reports for the commit 374b5bf to get more accurate results

@@            Coverage Diff             @@
##             main      #25      +/-   ##
==========================================
+ Coverage   16.32%   16.34%   +0.01%
==========================================
  Files          12       12
  Lines        1231     1236       +5
==========================================
+ Hits          201      202       +1
- Misses       1030     1034       +4

Impacted Files
Coverage Δ

src/Ot2Rec/align.py
16.29% <0.00%> (-0.25%)
:arrow_down:

src/Ot2Rec/main.py
9.42% <40.00%> (+0.13%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 66c97e4...374b5bf. Read the comment docs.


[DDW-641] Include wallet names in dialog windows

This PR includes wallet names in dialog windows.
Todos

[x] Add label to dialogs component
[x] Extact wallet name and parse it to Dialog component

Screenshots

Test Scenario
Scenario 1 - Check wallet name is present on the dialog
Given that I am on the |Dialog|
When I check the header
Then I can see that the name of the corresponding wallet is visible

|Dialog|
Share QR
Confirm Transaction
Change Password
Verify recovery phrase
View public key
Delete wallet
Undelegate wallet

Testing Checklist

Slack QA thread

Review Checklist
Basics

[ ] PR has been assigned and has appropriate labels (feature/bug/chore, release-x.x.x)
[ ] PR is updated to the most recent version of the target branch (and there are no conflicts)
[ ] PR has a good description that summarizes all changes
[ ] PR has default-sized Daedalus window screenshots or animated GIFs of important UI changes:

[ ] In English
[ ] In Japanese

[ ] CHANGELOG entry has been added to the top of the appropriate section (Features, Fixes, Chores) and is linked to the correct PR on GitHub
[ ] Automated tests: All acceptance and unit tests are passing (yarn test)
[ ] Manual tests (minimum tests should cover newly added feature/fix): App works correctly in development build (yarn dev)
[ ] Manual tests (minimum tests should cover newly added feature/fix): App works correctly in production build (yarn package / CI builds)
[ ] There are no flow errors or warnings (yarn flow:test)
[ ] There are no lint errors or warnings (yarn lint)
[ ] There are no prettier errors or warnings (yarn prettier:check)
[ ] There are no missing translations (running yarn manage:translations produces no changes)
[ ] Text changes are proofread and approved (Jane Wild / Amy Reeve)
[ ] Japanese text changes are proofread and approved (Junko Oda)
[ ] UI changes look good in all themes (Alexander Rukin)
[ ] Storybook works and no stories are broken (yarn storybook)
[ ] In case of dependency changes yarn.lock file is updated

Code Quality

[ ] Important parts of the code are properly commented and documented
[ ] Code is properly typed with flow
[ ] React components are split-up enough to avoid unnecessary re-renderings
[ ] Any code that only works in main process is neatly separated from components

Testing

[ ] New feature/change is covered by acceptance tests
[ ] New feature/change is manually tested and approved by QA team
[ ] All existing acceptance tests are still up-to-date
[ ] New feature/change is covered by Daedalus Testing scenario
[ ] All existing Daedalus Testing scenarios are still up-to-date

After Review

[ ] Merge the PR
[ ] Delete the source branch
[ ] Move the ticket to done column on the YouTrack board
[ ] Update Slack QA thread by marking it with a green checkmark

@alexander-rukin can you please review this one again 🙏

@aleksandardjordjeviciohk please check these screens:

Fixed @daniloprates

@aleksandardjordjeviciohk I have found a lot of issues - please see to fix those.
@input-output-hk/daedalus-qa please make sure to re-test all of these to make sure the fixes are good. Thanks!
Wallet recovery phrase verification, View public key, and Delete wallet dialogs seem to have the content too close to the wallet name subtitle:

Restore a wallet dialog and Delegate wallet dialog have an overlap of stepper subtitle with the title

@input-output-hk/qa @nikolaglumac fixed

Not okay - okay
Not okay - okay
Not okay - okay
Starting position for scroll bars are also Not okay - okay when scrollbar start is aligned with content start
Not okay - okay

Latest Build Testnet 17944.

Restoring Byron Legacy wallet :
Pic1
a. Space is too big comparing with design.
Restoring Shelley wallet :
Pic2
a \ b. Text is missing.

Issues would be reported one by one as it requires step by steps validations for multiple dialogs.

Restoring Byron Legacy wallet :
Pic1
a. Space is too big comparing with design.
Restoring Shelley wallet :
Pic2
a \ b. Text is missing.

Issues would be reported one by one as it requires step by step validations for multiple dialogs.

@dmitrii-gaico can you recheck now, gap on the success dialog. As for the message, that case is ok, as that text is only present if you restore Shelley based wallet.

@dmitrii-gaico @alexander-rukin is this one OK now?

@dmitrii-gaico @alexander-rukin is this one OK now?

QA Team is working on this + we want to run some small sanity/regression set. I don't think we can Approve/Reject it today. Optimistically we can have a decision tomorrow. Thank you.

@aleksandardjordjeviciohk Here are a few issues for the latest build (17962):

On the headers for delegation, restore, and voting wizards there is too much space between the title and the steps. For reference, here's how it looks in develop.
On "Native token settings" the data for the native token is displayed in uppercase on the header and tooltip. Check this screenshot.

@aleksandardjordjeviciohk The issues on this comment are fixed. There is a new issue with some dialogs for build 17969.

For the "success delegation" there is a lot of space between the title and the image. Check this screenshot. For reference, here's how it looks for a similar dialog.
On the "confirm transaction" dialog, there is a lot of space between the title and the name of the wallet. Check this screenshot.

@aleksandardjordjeviciohk
For tracking purposes.
Native token settings dialog - not ok
Design - ok

@aleksandardjordjeviciohk The fixes for the "confirm transaction" tab look good on 17972. A few issues on this build:

The space between the title and the image for the dialog after a successful delegation is smaller now, but it looks different from similar dialogs. Check this screeshot. For reference, here's how it looks on develop.
On the "Native token settings" the details on the header are still not centered. Check this screenshot.
However, this issue also reproduces on develop (screenshot) as the layout does not looks as on Zeplin. Maybe this should be addressed on another task?

@aleksandardjordjeviciohk The issues on this comment are fixed. There is a new issue with some dialogs for build 17969.

For the "success delegation" there is a lot of space between the title and the image. Check this screenshot. For reference, here's how it looks for a similar dialog.
On the "confirm transaction" dialog, there is a lot of space between the title and the name of the wallet. Check this screenshot.

Fixed

@aleksandardjordjeviciohk The fixes for the "confirm transaction" tab look good on 17972. A few issues on this build:

The space between the title and the image for the dialog after a successful delegation is smaller now, but it looks different from similar dialogs. Check this screenshot. For reference, here's how it looks on develop.
On the "Native token settings" the details on the header are still not centered. Check this screenshot.
However, this issue also reproduces on develop (screenshot) as the layout does not looks as on Zeplin. Maybe this should be addressed on another task?

Fixed

Fixed

Hi @aleksandardjordjeviciohk , On build 17975
There's no gap between the paragraph and the element above it. Please see: screenshot1, screenshot2, screenshot3

Hi @aleksandardjordjeviciohk , On build 17975
On Delegation dialog, there's no gap between the paragraph and the element above it. Please see: screenshot1, screenshot2, screenshot3

Added 20px margin

@aleksandardjordjeviciohk, these screenshots are unusable for the release notes since they are from Storybook and there are no Japanese screensots.


Pull events from LinkedIn?

Feature
Why is this feature being added?

What should your feature do?

I see this as a very low-priority stretch goal, @rickr.

Actually, LinkedIn doesn't have events.


[BUG]:

How are you running AnythingLLM?
AnythingLLM desktop app
What happened?
Using the @ symbol anywhere in the prompt seems to invoke the agent.
This makes prompts with code problematic
The following does not work as expected
class TestClass:
@staticmethod def some_method():
      pass

And outputs
Swapping over to agent chat. Type /exit to exit agent execution loop early.

However removing the @ works
class TestClass:
staticmethod def some_method():
      pass

Outputs
It seems like you are asking about a class named “TestClass” and its method “some_method”. Is there anything specific you would like to know about this class or method?

Are there known steps to reproduce?
Using @ anywhere in prompt triggers agent chat

Image of what happens when you have @ anywhere in the input message

Note 1: This image is just a supporting image for this Issue as the Original Poster didn't attached any images
Note 2: Kindly ignore the error message saying "Incorrect API Key" (I just entered some random words as API Key)


Fix spelling of navel.

Both naval and navel are valid words.

I agree with @LeonardMH. Maybe this should just be closed?

Heh, all this consideration of navels going on here. Very meta!
Still, as "navel" refers the body part, and "naval" is in reference to the navy I find myself falling in favor of the spelling "navel". I'm going to go ahead and accept the change :)


Put idea folder under gitignore

This belongs to your local global gitignore as it depends on your setup and not the project.
https://sebastiandedeyne.com/setting-up-a-global-gitignore-file/


Implement overcommit + styleguide

Hey,
This PR adds overcommit + our standard style guide configuration.
Please check it out.
Thanks!
✅ I will abide by the code of conduct.

@buys-fran Great, thanks! 👍


[FEATURE-2648] Enabling Natural Language Graph Queries in GraphScope with chatGPT

What do these changes do?
[TL;DR] We add LLM support to enhance the basic graph queries, allowing users to interact with graphs more casually.

Specifically, the detailed steps can be shown as:

Using the Langchain to send natural language sentences to LLM (chatGPT, chatGLM, and so on) with manually designed prompts.
Adding the schema of graphs to form more expected results from LLM.
Extract Cypher statements from the results.
Running Cypher on the Graphs through the Cypher interactive backend.
Return results from the backend.

Our contribution can be summarized as：

1. Well-designed Langchain pipeline as well as prompt templates  (langchain_cypher.py).
2. Exposed the query interface (query.py::query_to_cypher).
Related issue number
We are related to the issue #2648
Fixes

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.


feat: Custom Delegation roles command line

Description
Add a feature to a user as admin to use Custom Delegations

[x] Add support to choose to use Custom Delegations on Bootstrap
[x] Add support to be able to sign a custom delegation
[x] Add support to create new Custom Delegations when RSTUF is running
[x] Add support to delete Custom Delegation

TODO:

[ ] Add support to update Custom Delegation
[ ] Add support to send new, update, and delete delegations
[ ] ~unit tests~ (We will do it in another iteration)

Types of changes

[ ] Bug fix (non-breaking change which fixes an issue)
[x] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality not to work as expected)

Additional requirements

[ ] Tests have been added for the bug fix or new feature
[ ] Docs have been added for the bug fix or new feature

Code of Conduct
By submitting this PR, you agree to follow our Code of Conduct.

[x] I agree to follow this project's Code of Conduct

I realized you either decide to use the online key in your custom delegation or you are required to add 2 PEM/Sisgstore keys.
There is no middle ground to allow 1 online key and one PEM/Sigstore key.

You can if you explicitly declare the online key information while Add Key.
Building it in a nice flow in CLI is also possible. I don't see is a must to have.


(putty.install)x86 platform installer is gone and replaced with arm64 installer

Expected Behavior
on x86 platform, successfully install/update

Current Behavior
putty.install package files upgrade completed. Performing other installation steps.
Installing 32-bit putty.install...
WARNING: Installation of this MSI is not supported on this platform. Contact package maintainer(s) if you feel this is i
n error or if you need an architecture that is not available with the current packaging.
ERROR: Running ["C:\Windows\System32\msiexec.exe" /i "C:\ProgramData\chocolatey\lib\putty.install\tools\putty-arm64-0.75
-installer.msi" /qn /norestart /l*v "C:\Users\(snip)\AppData\Local\Temp\chocolatey\putty.install\0.75\install.log" ] was
not successful. Exit code was '1633'. Exit code indicates the following: Installation of this MSI is not supported on th
is platform. Contact package maintainer(s) if you feel this is in error or if you need an architecture that is not avail
able with the current packaging..
The upgrade of putty.install was NOT successful.
Error while running 'C:\ProgramData\chocolatey\lib\putty.install\tools\chocolateyInstall.ps1'.
 See log for details.

Possible Solution

Steps to Reproduce (for bugs)

Context

Your Environment

Package Version used: PuTTY (Install) 0.75
Operating System and version:
Chocolatey version: Chocolatey v0.10.15
Install/uninstall gist:

Please update the package with the necessary binary.

Can update automatic/putty/update.ps1 it with something like this:
--- a/automatic/putty/update.ps1        2021-09-05 13:09:52.913491973 +0300
+++ b/automatic/putty/update.ps1        2021-09-05 13:05:30.333505423 +0300
@@ -14,7 +14,7 @@
   $download_page = Invoke-WebRequest -Uri $releases -UseBasicParsing

   #https://the.earth.li/~sgtatham/putty/latest/x86/putty-0.67-installer.msi
-  $url32Installer = $download_page.links | ? href -match '\.msi$' | ? href -notmatch '64bit' | select -First 1 -expand href
+  $url32Installer = $download_page.links | ? href -match '\.msi$' | ? href -notmatch '64bit|arm' | select -First 1 -expand href
   $url64Installer = $download_page.links | ? href -match '64bit.*\.msi$' | select -First 1 -expand href

   $version = $url32Installer -split '\-' | select -last 1 -skip 1

I don't know which version of regular expressions is required, but something like this should solve issue.


Open Question: feasibility of datastore using dynamodb

I'm looking for your thoughts on implementating a database backend for authzed in dynamodb. Zenzibar paper did specifically say that it is using spanner, but it reads like it is really zookie that plays the important role of guaranteeing causal acl. I don't have any concrete code right now, but my feeling is that it may be possible to design the db with mostly eventual consistency, with small amount of strong consistency parts. The caveat would be that eval node answers may be different from each other, abit only for a short period after write. This should be alright in real life use cases. Most IAM from big cloud providers are not strongly consistent.
Is there any technical difficulty I have overlooked?

@imacks The biggest open question is handling of zedtokens and "full consistency": if a zedtoken is provided, or full consistency is required, SpiceDB is required to honor that request by the client. Thus, there must either be a way in DynamoDB to request the fully consistent updates that have been applied or SpiceDB will have to wait until it can guarantee that the updates have been applied.
So long as this is doable (or it is acceptable to wait), I would imagine implementation should be fairly straightforward

Where strong consistency is required, we can use conditional writes and strongly consistent reads. Having said that, the biggest advantage I hope from DynamoDB backend is better latency as things scale up, which suggests an eventual consistency model.

Yeah, strong consistency will only be required when requested by the client. If the ZedToken's requirements are already met by the eventual consistency at the time of the call, or if minimize_latency is requested (which is the default), then eventual consistency is fine

Tks for the tip. I will try my hands on making a POC and then circle back. Cheers!

@imacks any progress? Quite curious :)

@imacks Ping!

@imacks Sounds great! Let us know if we can help in any way

@imacks Would you be willing to share your progress? I'm quite curious :)

I just wanted to follow up on this thread as we've seen requests for DynamoDB usage in multi-region SpiceDB deployments.
DynamoDB can be used to implement a single-region deployment, but in a multi-region DynamoDB configuration, writes across regions are replicated in an eventually consistent, asynchronous fashion. This means requests with a full_consistency ZedToken in this configuration would produce incorrect results.

@imacks Checking in again to see if you have anything you'd mind sharing :)

I have quite a bit of experience with Dynamodb and Go. Also we have written a go lib around making working with Dynamo much easier. We were thinking to make it open source. @imacks happy to help with anything you need.

Pinging here to see if there have been any changes since earlier this year?  I have thought about taking a crack at this myself but dont want to pull the trigger if someone else has it mostly done.

@benny-yamagata As far as we know, no updates. Feel free to give it a shot and we'd be happy to work with you on it

@benny-yamagata As far as we know, no updates. Feel free to give it a shot and we'd be happy to work with you on it

Highly considering it 😉


[Feature Request] Transform that stacks data for agents with identical specs

Motivation
Some multi-agent environments, like VmasEnv, stack all of the tensors for observations, rewards, etc. for different agents that have identical specs. For instance, in one of these stacked environments, if there are 2 agents that each have 8 observations, the observation spec might look like this:
Composite(
    group_0: Composite(
        observation: UnboundedContinuous(
            shape=torch.Size([2, 8]),
            ...),
        shape=torch.Size([2])),
    shape=torch.Size([]))

In contrast, other environments, like UnityMLAgentsEnv, have separate keys for each agent, even if the agents' specs are identical. For instance, with 2 agents that each have 8 observations, the observation spec might look like this:
Composite(
    group_0: Composite(
        agent_0: Composite(
            observation: UnboundedContinuous(
                shape=torch.Size([8]),
                ...),
            shape=torch.Size([])),
        agent_1: Composite(
            observation: UnboundedContinuous(
                shape=torch.Size([8]),
                ...),
            shape=torch.Size([])),
        shape=torch.Size([])),
    shape=torch.Size([]))

It is not easy to apply the same training script to two environments that use these two different formats. For instance, applying the multi-agent PPO tutorial to a Unity env is not straightforward.
Solution
If we had an environment transform that could stack all the data from different keys, we could convert an environment that uses the unstacked format into an environment that uses the stacked format. Then it should be straightforward to use the same (or almost the same) training script on the two different environments.
Alternatives
Additional context
Checklist

[x] I have checked that there is no similar issue in the repo (required)

Have you taken a look at the group_map argument? When set to MarlGroupMapType.ALL_IN_ONE_GROUP the environment should return all agents in a single group (when possible, otherwise in more than one group.
If you are using this, then imo this is more of an issue with the implementation of the grouping of agents in UnityMLAgentsEnv.

@thomasbbrunner, the group_map=ALL_IN_ONE_GROUP argument does combine all the agents into one group key, but the specific issue is that the data for each of the agents do not get stacked all into the same set of tensors--each agent still has a separate key containing separate tensors, like the example I showed in the issue description.
One of the reasons why UnityMLAgentsEnv does not stack the tensors for all the agents is that each of the agents in a Unity env are free to have different specs--for instance, the strikers and goalies have different specs in the "StrikersVsGoalies" example environment. In cases like that, it's not possible to stack all of the agents' data together. So when I was writing the wrapper, it seemed natural to keep the agents under separate agent keys, rather than try to automatically figure out which agents' specs match each other and can be stacked together. Furthermore, it seemed best to organize the data in a way that is consistent with how the ml_agents library itself does it.
But I'm happy to change the behavior if there is good reason to do so.

Ah, I see! Thanks for the explanation. It seems that the UnityMLAgentsEnv has a different behavior than the other multi-agent environments (my experience is mostly based on PettingZoo here).
PettingZooEnv has three possible grouping types:

ONE_GROUP_PER_AGENT (self explanatory)
None (default): will group as many agents as possible, while keeping the groups homogeneous (i.e., all agents in the group have the same obs and action spec).
ALL_IN_ONE_GROUP: (self explanatory) + if the agents cannot be grouped together (due to mismatched specs), it will return some weird spec and any environment operation will lead to an error.

Personally, I don't see the benefit of grouping heterogeneous agents. Maybe that makes sense in your use-case, but I'd argue that there is no difference between having one group for each agent and a group containing sub-groups of heterogenous agents.
I quite like the default behavior of PettingZooEnv (grouping where possible) and I don't understand why there isn't a GROUP_HOMOGENEOUS (or similar) in MarlGroupMapType.
Would be interested in hearing more about your use-case!

It would be nice if the behavior of all TorchRL environments would be consistent with each other. It sometimes feels like multi-agent is not part of the "core" TorchRL capabilities and I'm hoping we can change that!

I agree that we should make environments consistent with each other when possible. At the time, I didn't think that putting the Unity env agents under separate keys would be a significant inconsistency with other TorchRL envs  (at least compared to VMAS, PettingZoo, and OpenSpiel)--it just seemed like the right choice because it's more consistent with the underlying ml_agents library. But I see that it probably would be better to stack the data of agents that share the same group, by default.
@vmoens, do you agree with this?

I don't understand why there isn't a GROUP_HOMOGENEOUS

That sounds like a good idea to me. Feel free to submit an issue

Personally, I don't see the benefit of grouping heterogeneous agents.

I don't either, but then again, I'm fairly new here. Maybe I took ALL_IN_ONE_GROUP too literally

It would be nice if the behavior of all TorchRL environments would be consistent with each other. It sometimes feels like multi-agent is not part of the "core" TorchRL capabilities and I'm hoping we can change that!

I'm sorry this is the impression you have, @matteobettini gave a great deal of effort in unifying MARL APIs, but if there are inconsistencies we should address them!

One of the reasons why UnityMLAgentsEnv does not stack the tensors for all the agents is that each of the agents in a Unity env are free to have different specs--for instance, the strikers and goalies have different specs in the "StrikersVsGoalies" example environment. In cases like that, it's not possible to stack all of the agents' data together. So when I was writing the wrapper, it seemed natural to keep the agents under separate agent keys, rather than try to automatically figure out which agents' specs match each other and can be stacked together. Furthermore, it seemed best to organize the data in a way that is consistent with how the ml_agents library itself does it.

I 100% agree on this.
My take when writing wrappers is usually that I want the wrapper to have the minimal interference with the underlying API – the only thing we want is to present the data in a tensordict-friendly manner: ie, convert to tensors, make sure that the data is on the desired device, minimize the cost of operations etc. Once that is done, we leave it to the user to deal with the data.
The reason we want to do this that any change we make in the way the data is presented to the user will create edge cases that we may not know of.
Now RE the "consistentcy" problem highlighted by @thomasbbrunner, as I said earlier we should make sure that things can be made uniform at a low cost.
My vision here would be for TorchRL to deal itself with the various grouping strategies with its own API, possibly inspired by pettingzoo if that is a convention in the community.
The API would consist in appending a transform that re-arranges the tensordict given some flag, for instance you could create your Unity env in its wrapper and pass a transform
env = UnityMLAgentsEnv(...)
transform = GroupMARLAgents(MARLGroups.ONE_GROUP_PER_AGENT)
env = env.append_transform(transform)

and then, internally, we make sure that the transform does its job for every MARL library we support.
Would that make sense?
nb: Using metaclasses, we could even pass directly a group argument in each wrapper that automatically append the transform if required.

Maybe i am missing what the group_map input to the MLAgents constructor does. Cause that is what I would use to change the data grouping and address this issue

I think I understand what you're saying, @matteobettini. I think two main points to summarize this discussion are:

By default, UnityMLAgentsEnv should not stack agent data, and the structure of the specs should reflect the group IDs assigned by MLAgents. (At least Vincent and I seem to agree on this)
All agents that are in the same MARL group should always be stacked together.

An important distinction is that the MLAgents group IDs are not the same thing as MARL group keys.
At the moment, UnityMLAgentsEnv honors point 1, but not point 2. So how do we honor both points?
Consider an example unity env where there are two unity group IDs, which each have 3 agents.
Right now, the default group map that UnityMLAgentsEnv gives would be this:
{
    'group_0': ['agent_0', 'agent_1', 'agent_2'],
    'group_1': ['agent_3', 'agent_4', 'agent_5'],
}

In order to honor both points 1 and 2, we would need to change UnityMLAgentsEnv's default group map to:
{
    ('group_0', 'agent_0'): ['agent_0'],
    ('group_0', 'agent_1'): ['agent_1'],
    ('group_0', 'agent_2'): ['agent_2'],
    ('group_1', 'agent_3'): ['agent_3'],
    ('group_1', 'agent_4'): ['agent_4'],
    ('group_1', 'agent_5'): ['agent_5'],
}

And also, to allow the user to specify a group map where some agents are in the same MARL group, UnityMLAgentsEnv needs to know how to stack/unstack the data coming out of and going into MLAgents.
Does this make sense to everyone?
If so, then the next question is whether the Stack transform would still be useful. I suppose it could--maybe a user wants to have fine-grain control over which data fields get stacked and which are kept unstacked

Isn't only axiom 2 violated? The specs and tensordicts currently do match for the MLAgents wrapper. That is, unless you've seen a bugged case where that's not true. As far as I understand, the only thing that MLAgents violates is the part of axiom 2 that says the specs and tensordicts in a group must be stacked.

Nice! I think I had understood wrongly

Is my solution https://github.com/pytorch/rl/issues/2566#issuecomment-2499296045 insufficient in some way?

I think it makes sense but I am not sure I understood it fully. I think it is because I have not clear what the group_map arg currently does.
Two points about that solution that I am still wondering about:

isn't group_map == mlagents_group_map the best default value? Why are we having one group per agent as default?
The names of the groups cannot be tuples (this is just a nit)

It is totally fine if you want to remove completely the group_map arg from MLagents, but I personally think it is a nice feature to have and it aligns with the many envs already present.
There is also a point about the transform adding an overhead which we can talk about today in our meeting

A further point is that if we used a transform for all MARL groupings, rather than requiring the env wrappers to implement the MARL grouping logic, then the implementation of env wrappers would become a bit simpler and less prone to potentially inefficient or bugged implementations of the MARL grouping logic. It seems like it would be nice to have the MARL grouping logic implemented in one common place.

I agree with this but I think it could be done in a utility function (which many MARL envs can use) rather than a transform.
Since all MARL envs already provide this functionality and it is now part of the MARL API, I don't see why removing it. Instead we could consider making a utli function to help in providing it.

Let me take a concrete example.
This is how VMAS stacks the data coming from the step output into the groups dictated by the group_map
https://github.com/pytorch/rl/blob/d537dcb6347e2370fcdaed553bf3474d653cb5a6/torchrl/envs/libs/vmas.py#L577-L608
The original vmas interface just outputs lists  (e.g., obs[i] is the obs of agent i).
Can you see how doing the flexible stacking at this stage is better than creating a tensordict for each individual agent, outputting it, and doing the stacking later with another for loop over the agents?
The amout of code you would have if you do the "one td per agent" solution would be the exact same, but it would just be more inefficient


"Research" documentation

Add a main summary page explaining how everything around the Research department:

how the Research department works
future plans/roadmap/edition plan
definition of basic qualitative metrics
the research team and the tools used (repositories)
how to get involved

Docs page: https://docs.web3privacy.info/research/

@Msiusko Thanks for adding the list of research projects to the documentation, but that's not exactly the goal of Research page (in fact it's an unnecessary duplication and complication). There is a Github project and a website for listing research projects - this should be mainly the documentation of the research department itself. In the same style as the Events page.


Remove all shebangs from Python modules and checker

As discussed, Python modules which aren't intended to be invoked
as scripts should not include a shebang line.

/assign

This file here https://github.com/kubernetes-client/python-base/blob/master/hack/boilerplate/boilerplate.py.txt

The file should still be there. But I forgot to remove the shebang line from some modules, so now the test is failing.
Along the way, we now guarantee that no one add a shebang where it's not necessary, and if it is necessary - as in the script hack/boilerplate/boilerplate.py one can add it to the list of SKIP_FILES in this script.

@micw523 I pushed new changes documenting how to handle scripts that should include a shebang.
This is addresses your concern about this.
I also squashed all the commits so now this PR needs a new approval.

/lgtm
/approve


Migrate Feature Switches into API

I'd like to make feature switches more controllable, so we'll move them out of where they are and set up a simple config API also.

Resolved in build 92


Behavior of DialogService becomes not consistent with DialogOptions.CloseDialogOnEsc value

Describe the bug
Behavior of DialogService becomes not consistent with DialogOptions.CloseDialogOnEsc value if user firstly opens dialog with DialogOptions.CloseDialogOnEsc == false and then - with DialogOptions.CloseDialogOnEsc == true
To Reproduce
Steps to reproduce the behavior:

User opens a dialog, that must not be closed on pressing Esc, according to DialogOptions.
User presses Esc - nothing happends, dialog remains opened, as expected.
User closes dialog.
User opens another dialog, that meant to be closed on Esc.
User presses Esc - dialog closes, as expected.
User again opens first dialog, that must not be closed on Esc.
User presses Esc - dialog closes!

Sample project:
https://github.com/obratim/RadzenDialogServiceIssue
Expected behavior
I was expecting that dialog's behavior will follow DialogOptions parameters, no matter if previously was opened other dialogs with other parameters.
Desktop (please complete the following information):

OS: Windows 10
Browser: ms edge, chrome
Version: Radzen.Blazor 4.6.5

Additional context
Type of application: Blazor WebAssembly with webservice.

Behavior of clicking around dialog is consistent with value DialogOptions.CloseDialogOnOverlayClick.
Only DialogOptions.CloseDialogOnEsc has issues.


[Guide] Using Netlify CMS with NextJS

Summary
Another guide for #2092.
I went with a slightly different approach to make this guide as simple as possible to "code along" with. Hopefully, my list of "internet cats" won't offend anybody 😂
I didn't want to depend on any boilerplate code, nor use complex configuration (like remarkable for parsing). I do however understand if you want to do things differently.
This is my first PR ever, so I'm sorry in advance if I messed up somewhere!
I used some of the other guides as inspiration and copied the "Enable Identity and Git Gateway" part directly from the Gatsby documentation.

So pumped to see this @steffenz!
@verythorough any chance you'd have time to review this?

I'm pretty swamped right now, but I'll try to take a look later in the week.

@steffenz is this still in progress? If it's ready to go, drop the WIP in the title and we'll get this moved forward.

@steffenz is this still in progress? If it's ready to go, drop the WIP in the title and we'll get this moved forward.

I think this is pretty much done. I removed the [WIP] tag now. I guess there's always room for improvements, but as a first version I think it's ready to go 😄

Agreed, if anyone would still like to review, please open a new issue with change suggestions (or a PR with improvements).

Awesome - glad I could help! Thanks for being so helpful and friendly, everybody. Hopefully, I'll find something else I can contribute to further down the road! 😁 Really love the work you do, rock on!


In Android studio this plug-in is not working

Hi,
In Android studio this plugin is available but when we enter the token it shows Invalid token? Can you please update the same.

I also got the same "invalid token" window on PhpStorm. Probably a problem with the plugin for all JetBrains IDEs.

@dkage I'm working on it.

me too

@dkage @intbjw have been fixed, you can try it out and see if there are any problems

Working great now, thanks mate. I guess you can close the issue @si9ma.


TagBot trigger issue

This issue is used to trigger TagBot; feel free to unsubscribe.
If you haven't already, you should update your TagBot.yml to include issue comment triggers.
Please see this post on Discourse for instructions and more details.

Triggering TagBot for merged registry pull request: https://github.com/JuliaRegistries/General/pull/24543

Triggering TagBot for merged registry pull request: https://github.com/JuliaRegistries/General/pull/25671

Triggering TagBot for merged registry pull request: https://github.com/JuliaRegistries/General/pull/27869

Triggering TagBot for merged registry pull request: https://github.com/JuliaRegistries/General/pull/27869


MISRA: fix rule 14.4

MISRA: fix rule 14.4
Description
MISRA: fix rule 14.4
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

/bot run check

/bot run checks


gapi.load resulting in 404

I have been using the gapi JS client successfully for a while now.
However, today, I suddenly get a 404 without any error reporting when using gapi.load.
This is my code:
return gapi.load('client:auth2', {
  callback: function () {
    // Handle gapi.client initialization.
    resolve(_gapiInit());
  },
  onerror: function (err) {
    reject(new Error('gapi.client failed to load - ' + (err.stack || JSON.stringify(err))));
  },
  timeout: 5000, // 5 seconds.
  ontimeout: function () {
    // Handle timeout.
    reject(new Error('gapi.client timeout'));
  }

This is what my console tells me:

gapi.js:218 GET https://apis.google.com//scs/apps-static//js/k=oz.gapi.en_US.ud0X1ANlltY.O/m=auth2,client/rt=j/sv=1/d=1/ed=1/am=AQE/rs=AGLTcCOcKpCWy1ppnaz1rAfxsno0yxWUNA/cb=gapi.loaded_0 404 ()
debugUtil.js:27 gapi init failed - Error: gapi.client timeout

Note: I get the 404 right away. It does not even get reported as an error; instead the timeout hits five seconds later.

I found the issue: I had a downloaded/cached version of api.js and it got outdated without a warning. Getting the latest version fixes it.
Is there any plans to support versioning for this library?
Maybe an NPM module even?

Hi @Domiii, please upvote #432 for the npm support issue where that issue can be discussed.


Data List: The create new item link has been omitted

If ''Can add inline' is set to Yes, the button/link that is supposed to show to allow users to add a new record does not display
Steps to reproduce the behavior:

Go to 'https://function-adminportal-test.shesha.dev/shesha/forms-designer?id=b9d308f1-dbd4-45c8-8f5e-56a17cf94d14'
Click on 'dataList'
Scroll down to 'Can Add inline' and check the configuration
Preview the form
Actual Results: The 'Add New Item' link has been omitted out

Expected behavior
A user should be able to add a new record

The implementation has changed


[Schema 1] Definitions for exportable types

@rjnagal @vishh This is part 1 of #342
The presented types are intended for use from both within the module and from external REST API consumers. They represent a hierarchical in-memory structure of the cluster's historical state.
Open Issues:

Limitations on the historical data length of []*MetricTimeseries
Processing of Kubernetes' event stream for real-time information regarding container/pod/node deletions

Can one of the admins verify this patch?

ok to test

Just a few comments. Overall LGTM

@vishh addressed comments in the latest commit

LGTM. Will merge once CI passes.

FYI: You can an IDE hook for running gofmt on save.


feat: add voice option in useSpeech

:tada: This PR is included in version 9.8.0 :tada:
The release is available on:

npm package (@latest dist-tag)
GitHub release

Your semantic-release bot :package::rocket:


Implement TimeMixer as an imputation model

What does this PR do?

fixing #397;

Before submitting

[x] Was this discussed/approved via a GitHub issue? Please add a link to it if that's the case.
[x] I have commented my code, particularly in hard-to-understand areas.
[x] I have written necessary tests and already run them locally.

Pull Request Test Coverage Report for Build<PHONE_NUMBER>1
Details

266 of 416 (63.94%)  changed or added relevant lines in 8 files are covered.
3 unchanged lines in 2 files lost coverage.
Overall coverage decreased (-0.7%) to 83.254%

Changes Missing Coverage
Covered Lines
Changed/Added Lines
%

pypots/imputation/timemixer/core.py
21
23
91.3%

pypots/imputation/timemixer/model.py
71
74
95.95%

pypots/nn/modules/timemixer/layers.py
83
97
85.57%

pypots/nn/modules/timemixer/backbone.py
77
208
37.02%

Files with Coverage Reduction
New Missed Lines
%

pypots/clustering/vader/model.py
1
85.43%

pypots/nn/modules/fedformer/layers.py
2
45.99%

Totals

Change from base Build<PHONE_NUMBER>9:
-0.7%

Covered Lines:
10863

Relevant Lines:
13048

💛  - Coveralls


Changes to tasks

This does a few things:

It fetches the minimum number of results

When a filter is created, applies the "is:open" filter to grab everything
From then on it adds the "updated:>XYZ" to only grab tasks that have been updated
This allows us to remove pagination from the app.

Only comment and state changes will mark a task as updated. An issue won't be unarchived automatically when the body or a label is changed.

There are some other minor things that I'll leave comments about below.
cc @cheshire137 @summasmiff

to only grab tasks that have been updated

Updated since when?

Updated since when?

Since the last time that filter query was fetched by the app.


Make methods into getters wherever possible on D2Reader

This is to use the TS getter functionality so that d2Reader.toc() becomes d2Reader.toc for example.

should be covered as much as possible in this PR https://github.com/d-i-t-a/R2D2BC/pull/306


animated grid view

I can add animated grid view for UI. kindly assign me this suggestion.

Please add 2-3 screens with different style of grid views

ok sure

completed kindly check


Default index=main used for container name lookup

The default index to create the "docker_containername" lookup is currently set to "main".
If  you are using a different index, you'll need to change the index name under "docker_logs" in the app-docker>default>macros.conf.

Can be added into the README. Closing Issue.


incorporating fast zlib changes

Is there any plan of incorporating the changes made by fast zlib. The speed differences are significant.
https://github.com/gildor2/fast_zlib
https://github.com/gildor2/fast_zlib/wiki

It would need pretty exhaustive cleanup because a lot of macros and defines are either different or removed in zlib-ng.

I just abandoned the patch that ports the changes from fast-zlib.
Can we close this issue as well? The change is too intrusive to get merged in zlib-ng. It also does not provide enough performance speedup or better compression on several data sets.

@sebpop so in your opinion, is it still worth the time/work? or zlib-ng should outperform fast-zlib in most cases?

I suggest integrating fastzlib into the latest benchmarking and see if it's worth investigating

From the description of the algorithm changes on fastzlib website:

Original implementaition was very straightforward. It takes first 3 bytes of the string, computes their hash, and looks over history using hash table to find a match. If it will find a longer string, it remembers it, and continues search using the same hash for first 3 bytes. The penalty of this approach is when code looks for a string which first 3 bytes (or bytes with the same hash value) are used very often in a file.

Optimized version does an attempt to look over history with smaller number of checks. The idea is very simple: when algorithm finds a match (intermediate one), it computes hashes for all matching bytes, and selects a different hash chain - that one which points farther from current position. This is obvious - if we have 2 hash chains, one points at distance 10, and another one at distance 1000, it is not worth checking distance 10 because other bytes will not match - the closest possible match will occur at position 1000 or farther.

It sounds like the algorithm will skip ahead and make some assumptions that farther matches will always yield better results. Perhaps this is what @sebpop encountered when he said that it does not provide better compression on several data sets.

Comparing hashes of first 3 bytes is almost always inefficient if best match is longer than 3 bytes... That is why some levels hash 4 bytes instead... It could be future optimization to switch to hashing 4 bytes after the initial comparing on levels that still hash just first 3 bytes. Hashing more than 4 bytes might require more bits or there would be too many collisions.

The penalty of this approach is when code looks for a string which first 3 bytes (or bytes with the same hash value) are used very often in a file.

I think they have to be frequent in the window, not necessarily the file.

It sounds like the algorithm will skip ahead and make some assumptions that farther matches will always yield better results.

No assumptions, algorithm just allows to perform longer jumps when seeking for match, without losing anything. These longer jumps. So, if there's a limit say for 100 jumps, performance will be the same as for original zlib, however my algorithm will process larger set of data, what makes compression better when compression level is smaller than 9. At level 9, longest_match function is almost unrestricted, so both versions of algorithm looks over whole sliding window - but with my algorithm, it does less checks due to longer jumps, so it works faster.
Again, there's no "assumptions" done, the algorithm is not losing anything compared to original one.

Perhaps this is what @sebpop encountered when he said that it does not provide better compression on several data sets.

It don't have to provide better compression on all data sets. In the worst case, compression will be the same, with the same speed. But in average, it's better.

MADLER
 Tool: minigzip-madler.exe
 Runs: 32
 Levels: 1-9
 Trimworst: 10

 Level   Comp   Comptime min/avg/max  Decomptime min/avg/max  Compressed size
 1      36.452%    2.715/2.773/2.853       0.966/1.001/1.044  77259041
 2      35.387%    3.032/3.086/3.150       0.969/0.992/1.022  75002289
 3      34.427%    3.705/3.762/3.837       0.948/0.974/0.999  72967052
 4      33.500%    4.001/4.074/4.164       0.940/0.964/0.989  71002829
 5      32.632%    5.483/5.562/5.691       0.926/0.957/0.995  69161697
 6      32.191%    7.508/7.608/7.829       0.929/0.946/0.977  68228443
 7      32.055%    8.967/9.070/9.223       0.920/0.949/0.980  67939659
 8      31.939% 13.656/13.834/14.211       0.923/0.939/0.960  67693439
 9      31.916% 17.365/17.589/18.047       0.925/0.943/0.965  67644560

 avg1   33.389%                7.484                   0.963
 tot                        1481.883                 190.664

FAST-ZLIB
 Tool: minigzip-fast-zlib.exe
 Runs: 32
 Levels: 1-9
 Trimworst: 10

 Level   Comp   Comptime min/avg/max  Decomptime min/avg/max  Compressed size
 1      36.452%    2.678/2.753/2.856       0.977/1.000/1.031  77259041
 2      35.387%    2.863/2.995/3.121       0.950/0.989/1.040  75002289
 3      34.427%    3.424/3.568/3.735       0.936/0.984/1.023  72967052
 4      33.482%    3.758/3.944/4.098       0.918/0.966/1.012  70963457
 5      32.489%    4.898/5.081/5.254       0.911/0.950/0.991  68859143
 6      32.065%    5.879/6.090/6.283       0.903/0.946/0.987  67961891
 7      31.983%    6.216/6.415/6.613       0.907/0.947/0.989  67786277
 8      31.913%    6.714/6.953/7.137       0.901/0.944/0.982  67639830
 9      31.910%    7.017/7.222/7.516       0.902/0.939/0.980  67633229

 avg1   33.345%                5.002                   0.963
 tot                         990.481                 190.642

ZLIB-NG
Tool: minigzip-head.exe
Runs: 32
Levels: 1-9
Trimworst: 10

Level   Comp   Comptime min/avg/max  Decomptime min/avg/max  Compressed size
1      47.684%    1.022/1.051/1.085       0.687/0.699/0.720  101065293
2      35.540%    1.803/1.881/1.965       0.710/0.734/0.759  75325919
3      34.226%    2.153/2.216/2.302       0.692/0.717/0.744  72541436
4      32.953%    2.591/2.680/2.775       0.668/0.686/0.715  69843427
5      32.665%    2.819/2.889/2.994       0.670/0.692/0.725  69233255
6      32.515%    3.307/3.427/3.584       0.670/0.690/0.738  68914325
7      32.257%    4.498/4.645/4.834       0.668/0.690/0.729  68367431
8      32.172%    6.996/7.175/7.433       0.667/0.686/0.717  68186831
9      32.156%   9.657/9.915/10.146       0.668/0.688/0.714  68154021

avg1   34.685%                3.987                   0.698
tot                         789.356                 138.226

It seems that the fast-zlib method is not currently compatible with zlib-ng because zlib-ng hash table consists of 4 byte hashes, where as original zlib uses rolling hash or hash of consecutive bytes. Fast-zlib takes advantage of the zlib hashing method to optimize longest_match lookup.
From @mtl1979:

It's faster to hash 4 bytes starting from some location than to constantly accumulate rolling hash. There was also a case when rolling hash didn't give correct result, because the next byte to hash was not one byte after the previously hashed byte, sometimes it hashed things backwards and that caused performance penalty when using rolling hash.

I'm not sure if this can be overcome.

Here is another longer run to make sure my benchmarks weren't messed up:
MADLER
 Tool: minigzip-madler.exe
 Runs: 120
 Levels: 1-9
 Trimworst: 90

 Level   Comp   Comptime min/avg/max  Decomptime min/avg/max  Compressed size
 1      36.452%    2.708/2.724/2.733       0.977/0.986/0.992  77259041
 2      35.387%    3.010/3.023/3.032       0.967/0.978/0.986  75002289
 3      34.427%    3.653/3.678/3.691       0.957/0.967/0.973  72967052
 4      33.500%    3.954/3.971/3.983       0.943/0.956/0.961  71002829
 5      32.632%    5.371/5.396/5.405       0.936/0.947/0.951  69161697
 6      32.191%    7.371/7.388/7.397       0.932/0.941/0.944  68228443
 7      32.055%    8.782/8.814/8.828       0.924/0.936/0.941  67939659
 8      31.939% 13.378/13.412/13.427       0.924/0.936/0.940  67693439
 9      31.916% 17.014/17.039/17.056       0.926/0.934/0.938  67644560

 avg1   33.389%                7.272                   0.953
 tot                        1963.357                 257.435

FAST-ZLIB
 Tool: minigzip-fast-zlib.exe
 Runs: 120
 Levels: 1-9
 Trimworst: 90

 Level   Comp   Comptime min/avg/max  Decomptime min/avg/max  Compressed size
 1      36.452%    2.623/2.644/2.654       0.984/0.992/0.999  77259041
 2      35.387%    2.828/2.842/2.853       0.964/0.977/0.984  75002289
 3      34.427%    3.379/3.403/3.412       0.953/0.970/0.974  72967052
 4      33.482%    3.699/3.721/3.734       0.944/0.956/0.961  70963457
 5      32.489%    4.807/4.832/4.842       0.930/0.947/0.952  68859143
 6      32.065%    5.760/5.774/5.782       0.932/0.940/0.944  67961891
 7      31.983%    6.075/6.098/6.108       0.926/0.936/0.940  67786277
 8      31.913%    6.592/6.612/6.618       0.925/0.934/0.939  67639830
 9      31.910%    6.857/6.874/6.884       0.926/0.933/0.939  67633229

 avg1   33.345%                4.755                   0.954
 tot                        1283.984                 257.525

ZLIB-NG
 Tool: minigzip-head.exe
 Runs: 120
 Levels: 1-9
 Trimworst: 90

 Level   Comp   Comptime min/avg/max  Decomptime min/avg/max  Compressed size
 1      47.684%    1.046/1.056/1.060       0.697/0.703/0.705  101065293
 2      35.540%    1.819/1.830/1.835       0.716/0.724/0.727  75325919
 3      34.226%    2.155/2.169/2.177       0.698/0.708/0.712  72541436
 4      32.953%    2.606/2.620/2.625       0.686/0.691/0.693  69843427
 5      32.665%    2.813/2.825/2.834       0.681/0.686/0.689  69233255
 6      32.515%    3.305/3.320/3.327       0.678/0.685/0.688  68914325
 7      32.257%    4.480/4.492/4.500       0.678/0.685/0.688  68367431
 8      32.172%    6.933/6.951/6.961       0.677/0.682/0.685  68186831
 9      32.156%    9.589/9.611/9.621       0.679/0.683/0.687  68154021

 avg1   34.685%                3.875                   0.694
 tot                        1046.243                 187.395

Fast-zlib changes have been merged into develop.


could I change the locales dynamicly on the page?

I want to implement the language changing function on my blog site. But the table list and other in the *.yml cannot change the text.

My solution is only to localize text into 1 language. If you want to provide multiple versions of translated content you'll need to build a more robust solution.
There are also Jekyll plugins out there to help with this, but they don't work with GitHub Pages if you're hosting there.

jekyll-localization
jekyll-multiple-languages-plugin

Thank you. I get it.. I prefer the Multilingual Jekyll. The solution is simliar with our application, using ref.


AttributeError: torch._dynamo.config.vocab_size does not exist

I can load the model using below code:
from transformers import AutoModelForCausalLM, AutoTokenizer
model_id = "/root/private_data/models/Meta-Llama-3.1-70B-Instruct"

model = AutoModelForCausalLM.from_pretrained(model_id, device_map='auto',load_in_4bit=True,attn_implementation="flash_attention_2")
tokenizer = AutoTokenizer.from_pretrained(model_id)

However, When I try to load the model using unsloth, it shows below error. Could you please tell me where was wrong?
from unsloth import FastLanguageModel
from transformers import TextStreamer
import re
from tqdm import tqdm

max_seq_length = 16384
dtype = None
load_in_4bit = True

model, tokenizer = FastLanguageModel.from_pretrained(
    model_name = "/root/private_data/models/Meta-Llama-3.1-70B-Instruct",
    max_seq_length = max_seq_length,
    dtype = dtype,
    load_in_4bit = load_in_4bit,
)

FastLanguageModel.for_inference(model)
text_streamer = TextStreamer(tokenizer)

🦥 Unsloth: Will patch your computer to enable 2x faster free finetuning.
==((====))==  Unsloth 2024.8: Fast Llama patching. Transformers = 4.44.0.
   \\   /|    GPU: NVIDIA A800 80GB PCIe LC. Max memory: 79.138 GB. Platform = Linux.
O^O/ \_/ \    Pytorch: 2.3.0+cu121. CUDA = 8.0. CUDA Toolkit = 12.1.
\        /    Bfloat16 = TRUE. FA [Xformers = 0.0.26.post1. FA2 = True]
 "-____-"     Free Apache license: http://github.com/unslothai/unsloth
Loading checkpoint shards: 100%
30/30 [08:33<00:00, 14.69s/it]
---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
File /opt/conda/lib/python3.10/site-packages/torch/utils/_config_module.py:142, in ConfigModule.__getattr__(self, name)
    141 try:
--> 142     return self._config[name]
    143 except KeyError as e:
    144     # make hasattr() work properly

KeyError: 'vocab_size'

The above exception was the direct cause of the following exception:

AttributeError                            Traceback (most recent call last)
Cell In[1], line 10
      7 dtype = None
      8 load_in_4bit = True
---> 10 model, tokenizer = FastLanguageModel.from_pretrained(
     11     model_name = "/root/private_data/models/loras/writer_70b_v1_lora",
     12     max_seq_length = max_seq_length,
     13     dtype = dtype,
     14     load_in_4bit = load_in_4bit,
     15 )
     17 FastLanguageModel.for_inference(model)
     18 text_streamer = TextStreamer(tokenizer)

File /opt/conda/lib/python3.10/site-packages/unsloth/models/loader.py:272, in FastLanguageModel.from_pretrained(model_name, max_seq_length, dtype, load_in_4bit, token, device_map, rope_scaling, fix_tokenizer, trust_remote_code, use_gradient_checkpointing, resize_model_vocab, revision, *args, **kwargs)
    269     tokenizer_name = None
    270 pass
--> 272 model, tokenizer = dispatch_model.from_pretrained(
    273     model_name        = model_name,
    274     max_seq_length    = max_seq_length,
    275     dtype             = dtype,
    276     load_in_4bit      = load_in_4bit,
    277     token             = token,
    278     device_map        = device_map,
    279     rope_scaling      = rope_scaling,
    280     fix_tokenizer     = fix_tokenizer,
    281     model_patcher     = dispatch_model,
    282     tokenizer_name    = tokenizer_name,
    283     trust_remote_code = trust_remote_code,
    284     revision          = revision if not is_peft else None,
    285     *args, **kwargs,
    286 )
    288 if resize_model_vocab is not None:
    289     model.resize_token_embeddings(resize_model_vocab)

File /opt/conda/lib/python3.10/site-packages/unsloth/models/llama.py:1403, in FastLlamaModel.from_pretrained(model_name, max_seq_length, dtype, load_in_4bit, token, device_map, rope_scaling, fix_tokenizer, model_patcher, tokenizer_name, trust_remote_code, **kwargs)
   1393 tokenizer_name = model_name if tokenizer_name is None else tokenizer_name
   1394 tokenizer = load_correct_tokenizer(
   1395     tokenizer_name    = tokenizer_name,
   1396     model_max_length  = max_position_embeddings,
   (...)
   1400     fix_tokenizer     = fix_tokenizer,
   1401 )
-> 1403 model, tokenizer = patch_tokenizer(model, tokenizer)
   1404 model = model_patcher.post_patch(model)
   1406 # Patch up QKV / O and MLP

File /opt/conda/lib/python3.10/site-packages/unsloth/models/_utils.py:470, in patch_tokenizer(model, tokenizer)
    468     if len(check_pad_token) != 1:
    469         possible_pad_token = None
--> 470     if check_pad_token[0] >= config.vocab_size:
    471         possible_pad_token = None
    472 pass

File /opt/conda/lib/python3.10/site-packages/torch/utils/_config_module.py:145, in ConfigModule.__getattr__(self, name)
    142     return self._config[name]
    143 except KeyError as e:
    144     # make hasattr() work properly
--> 145     raise AttributeError(f"{self.__name__}.{name} does not exist") from e

AttributeError: torch._dynamo.config.vocab_size does not exist

I'm getting this error as well! following the colab instructions, with the meta-llama/Meta-Llama-Llama3.1-8B-Instruct model. when doing it with the model from unsloth, it works fine

I think it related to commit 8001d30. line 470 should have model.config instead of config only.

@danielhanchen sorry for tagging, but I think it is breaking bugs bcs it prevent loading the model.

this could be temporary fix if you use colab
!pip install "unsloth[colab-new] @ git+https://github.com/unslothai/unsloth.git@bfe38e6ea8d3d7cf8ce9e37962de03c71c90cbe2"
!pip install --no-deps "xformers<0.0.27" "trl<0.9.0" peft accelerate bitsandbytes
I installed specific commit before 8001d30

I believe that this may not be a problem directly in unsloth but a problem with a dependency (another argument for pinned dependencies...). For me, even checking out https://github.com/unslothai/unsloth/tree/July-Llama-2024 fails with the above error.

Whoops - my bad that's a bug! I accidentally forgot to put model.config instead of just config! Updated the main branch! For local installations, please update Unsloth via pip uninstall unsloth -y pip install --upgrade --no-cache-dir "unsloth[colab-new] @ git+https://github.com/unslothai/unsloth.git" (Colab and Kaggle just Disconnect and Delete Runtime)

Just wanted to confirm that the bug reported in this issue has been fixed. I pulled the latest changes from the main branch and tested the scenario. Everything is working perfectly now!
Thanks for the quick fix and the great work!

issue resolved! Thanks for the fix ::)


[Bug]:  Update js collections calls are being sent before onPageLoad calls

Is there an existing issue for this?

[X] I have searched the existing issues

Description
If there is a discrepancy in the backend and the client with respect to js collection, the client makes an update call to correct the values.
This should take place after all evaluations are done as part of post eval process but in some case encountered this to be happening before on-page load actions are finished delaying entire loading time
Steps To Reproduce

Create an workspace or go to existing one
import an app from github with a large JS object or multiple
after importing, go to a page which has js object and few on page load actions
check network tab, first PUT call for js object happened and then on page load actions got executed

Public Sample App
No response
Issue video log
No response
Version
Cloud

Moving this card to Product backlog as we've encountered some blockers. We need some product discussion on this.
Explained the issues here https://www.loom.com/share/ee6786b057824e0890214b0a8bc37800


KYLIN-4752 Refine server mode checking

Proposed changes
Describe the big picture of your changes here to communicate to the maintainers why we should accept this pull request. If it fixes a bug or resolves a feature request, be sure to link to that issue.
Types of changes
What types of changes does your code introduce to Kylin?
Put an x in the boxes that apply

[ ] Bugfix (non-breaking change which fixes an issue)
[x] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)
[ ] Documentation Update (if none of the other choices apply)

Checklist
Put an x in the boxes that apply. You can also fill these out after creating the PR. If you're unsure about any of them, don't hesitate to ask. We're here to help! This is simply a reminder of what we are going to look for before merging your code.

[x] I have create an issue on Kylin's jira, and have described the bug/feature there in detail
[ ] Commit messages in my PR start with the related jira ID, like "KYLIN-0000 Make Kylin project open-source"
[ ] Compiling and unit tests pass locally with my changes
[ ] I have added tests that prove my fix is effective or that my feature works
[ ] If this change need a document change, I will prepare another pr against the document branch
[ ] Any dependent changes have been merged

Further comments
If this is a relatively large or complex change, kick off the discussion at user@kylin or dev@kylin by explaining why you chose the solution you did and what alternatives you considered, etc...

Pull Request Test Coverage Report for Build 6432

1 of 12 (8.33%)  changed or added relevant lines in 6 files are covered.
4 unchanged lines in 2 files lost coverage.
Overall coverage increased (+0.002%) to 28.053%

Changes Missing Coverage
Covered Lines
Changed/Added Lines
%

core-common/src/main/java/org/apache/kylin/common/util/ServerMode.java
0
1
0.0%

core-job/src/main/java/org/apache/kylin/job/impl/curator/CuratorScheduler.java
0
1
0.0%

server-base/src/main/java/org/apache/kylin/rest/service/AclTableMigrationTool.java
0
1
0.0%

core-job/src/main/java/org/apache/kylin/job/impl/threadpool/DefaultScheduler.java
1
3
33.33%

core-job/src/main/java/org/apache/kylin/job/impl/threadpool/DistributedScheduler.java
0
3
0.0%

server-base/src/main/java/org/apache/kylin/rest/service/QueryService.java
0
3
0.0%

Files with Coverage Reduction
New Missed Lines
%

core-job/src/main/java/org/apache/kylin/job/impl/threadpool/DistributedScheduler.java
1
0%

core-job/src/main/java/org/apache/kylin/job/impl/threadpool/DefaultScheduler.java
3
79.07%

Totals

Change from base Build 6416:
0.002%

Covered Lines:
26253

Relevant Lines:
93584

💛  - Coveralls


Thread safe or Not ?  [ com.cloudant.client.api.Database ]

The javadoc does not explicitly mention if the Database objects are thread safe, so would request you to confirm.
So far we have seen that most of the methods on "Database" object are thread safe, except for findByIndex(). As part of a stress test against our application, we have seen that findByIndex() returns multiple results even though cloudant database has only one single document matching the cloudant selector query. This occurs very rarely , under 0.5% over a one hour test which includes more than 50 reads per second.
The version used was 2.4.2 when the test was run.

Yes, we consider the Database object thread safe. [1]
Regarding the multiple results from findByIndex() I would be very surprised if that was related to threading on the client side. Are you seeing identical document ID and revision for all the results? What type of index and selector are you using?

In this specific use case, currently the application does not expect multiple entries to be returned and simply throws an exception if it finds more than one results. We will try to make a code change and re-run the test to exactly log the the document and rev ids. The occurrence is very rare, can be reproduced only in a stress test.
We are using a "Text" type of index based on Apache Lucene [ Not Json index ] . The total number of records is apprx. 40 million and the selector json is very simple -
{"selector":{"empUserId":"abcdefghijklmno"}}

An update -
We added a quick fix to log _id and _rev and found that intermittently when we had multiple results, they were exactly the same record ... Same _id as well as _rev.
Few excerpts from the log after removing some details -
W readProfileByIndex Multiple profiles (2) found
W 0:id:47f92fd717f911db8dd4d14dcc71f2db
W 0:rev:1-21e44eccfb45a5915fe3ec2e068175c1
W 1:id:47f92fd717f911db8dd4d14dcc71f2db
W 1:rev:1-21e44eccfb45a5915fe3ec2e068175c1
The stress test we are running can almost certainly recreate this issue.
We added a work around to just retry the search , i.e. exact same call again [ findByIndex ] and it works fine with retry , we get exactly one result back.
So we have a work around in place now.

That seems very unusual - I suggest you send that information to<EMAIL_ADDRESS>(or open a support request from your Cloudant dashboard). Make sure to include your Cloudant account name as that will help them start investigating immediately.


[ISSUES #1503] fix some typo in docs/cn/design.md

What is the purpose of the change
fix typo in #1503
Brief changelog
fix some typo
Verifying this change
done
Follow this checklist to help us incorporate your contribution quickly and easily. Notice, it would be helpful if you could finish the following 5 checklist(the last one is not necessary)before request the community to review your PR.

[x] Make sure there is a Github issue filed for the change (usually before you start working on it). Trivial changes like typos do not require a Github issue. Your pull request should address just this issue, without pulling in other changes - one PR resolves one issue.
[x] Format the pull request title like [ISSUE #123] Fix UnknownException when host config not exist. Each commit in the pull request should have a meaningful subject line and body.
[x] Write a pull request description that is detailed enough to understand what the pull request does, how, and why.
[x] Write necessary unit-test(over 80% coverage) to verify your logic correction, more mock a little better when cross module dependency exist. If the new feature or significant change is committed, please remember to add integration-test in test module.
[x] Run mvn -B clean apache-rat:check findbugs:findbugs checkstyle:checkstyle to make sure basic checks pass. Run mvn clean install -DskipITs to make sure unit-test pass. Run mvn clean test-compile failsafe:integration-test  to make sure integration-test pass.
[ ] If this contribution is large, please file an Apache Individual Contributor License Agreement.

Coverage increased (+0.02%) to 49.589% when pulling 006efae3aecf31ae5ea61300b2fb8d9615d49a43 on Switch-vov:develop into d99862a88a6a42babc68d2a417287b19fcddf298 on apache:develop.


make sure domain strings are only sent once

What does this PR do?
consolidates all domain strings into a single list, so that duplicates are not sent
Motivation
more efficient packaging of submission data
Checklist

[x] A release note has been added or the changelog/no-changelog label has been applied.

Superceded by #8808 (rebase made a mess of this pr)


updated heading and help text for http codes

@manishmsfte : Thanks for your contribution! The author(s) have been notified to review your proposed change.

@manishmsfte : Thanks for your contribution! The author(s) have been notified to review your proposed change.

Hi @manishmsfte - This pull request was opened in the public repo. PMs should work in the private repo, per the Microsoft Docs contributor guide. Would you make this content update in the private repo instead? Thank you! We'll close this PR.

Hi @manishmsfte - This pull request was opened in the public repo. PMs should work in the private repo, per the Microsoft Docs contributor guide. Would you make this content update in the private repo instead? Thank you! We'll close this PR.


Multiline label slot in SelectMenu

Description
SelectMenu is based on Input.
It's possible to render as "multiline" Input or Textarea, when #label slot has many selected options?
Additional context
No response
### Tasks

You should be able to do whatever you want with the #label slot:
<USelectMenu class="w-20">
  <template #label>
    Select Menu
  </template>
</USelectMenu>


Problem injecting Osiris

Whenever I inject Osiris (using Sazzinjector) with LoadLibrary, nothing happens in-game; The menu never shows up. And when I use Manual Map, the game just crashes. Any ideas on what could be causing this? I have used Osiris in the past with non of these issues, using this same computer, which makes me wonder.

I believe that LoadLibrary is detected, you can try Memject or shtreeba

download shtreeba

Ty


Reimplement timeseries charts using a Charting component that can properly normalize data over time.

https://github.com/morrisjs/morris.js
https://github.com/square/cubism
https://github.com/danvk/dygraphs
https://github.com/wa0x6e/cal-heatmap
https://github.com/marmelab/EventDrops

Closed by 0.2.1 release


[DOC]: Add information on how to generate new sample data for the ABP nvsmi example

How would you describe the priority of this documentation request
Medium
Describe the future/missing documentation
Without running NetQ, there is no way to generate new sample data for the ABP nvsmi detection example. However, something like the following should allow users to generate new data from a local GPU.
import time

import pandas as pd
from pynvml.smi import NVSMI_QUERY_GPU
from pynvml.smi import nvidia_smi

# Output name
output_file = "nvsmi.json"

# Interval
interval_ms = 1000

query_opts = NVSMI_QUERY_GPU.copy()

# Remove the timestamp and supported clocks from the query
del query_opts["timestamp"]
del query_opts["supported-clocks"]

nvsmi = nvidia_smi.getInstance()

with open(output_file, "w", encoding="UTF-8") as f:

    while (True):

        dq = nvsmi.DeviceQuery(list(query_opts.values()))

        output_dicts = []

        # Flatten the GPUs to allow for a new row per GPU
        for gpu in dq["gpu"]:
            single_gpu = dq.copy()

            # overwrite the gpu list with a single gpu
            single_gpu["gpu"] = gpu

            output_dicts.append(single_gpu)

        df = pd.json_normalize(output_dicts, record_prefix="nvidia_smi_log")

        # Rename the id column to match the XML converted output from NetQ
        df.rename(columns={"gpu.id": "gpu.@id", "count": "attached_gpus"}, inplace=True)

        df.rename(columns=lambda x: "nvidia_smi_log" + "." + x, inplace=True)

        # Add the current timestamp
        df.insert(0, "timestamp", time.time())

        df.to_json(f, orient="records", lines=True)

        f.flush()

        time.sleep(interval_ms / 1000.0)

This should be included in the documentation to make it clearer for the user how to run their own data through the example.
Where have you looked?
No response
Code of Conduct

[X] I agree to follow this project's Code of Conduct
[X] I have searched the open documentation issues and have found no duplicates for this bug report

@mdemoret-nv Thank you for your response. We have run the script you provided and it is working fine. However, most of the values are coming back as N/A. When we give the generated data from the script to the "abp_nvsmi_detection" model, it produces an error: "Inference Rate[Complete]: 0 inf [00:00, ? inf/s]". The error message indicates that the model is missing the "nvidia_smi_log.gpu.pci.tx_util" column. Since NetQ is not open source, we are unable to generate new sample data for the ABP nvsmi detection example. Is there any other alternative way to do this? Thank you for your help in advance.

@ABHIPATEL98 could you try using this notebook to retrain the model with the intersection of columns between /datasets/training-data/abp-sample-nvsmi-training-data.json and your generated dataset? If the accuracy is acceptable, you can deploy the new model to Triton and run the inference pipeline against that.
Before running the pipeline, you'll have to update the columns file. Also, model_fea_len in your pipeline run command. For example, if your new column count is 18 (default is 29) you would need to update your command to use this:
pipeline-fil --model_fea_length=18 --columns_file=${MORPHEUS_ROOT}/morpheus/data/columns_fil.txt \


DNS_ZONE_NAME not being picked up from config.json

When I run deploy_pcf.sh it is failing saying that DNS_ZONE_NAME required. This value should be set based on the default / config.json. I was able to get the deployment to work by setting DNS_ZONE_NAME manually (export DNS_ZONE_NAME=pcf-zone)
running step: acceptEulas
accepted EULAs for 6 products
enabling Google Cloud APIs
waiting for 25 service enable operation(s) to complete
waiting for 3 service enable operation(s) to complete
waiting for 1 service enable operation(s) to complete
enusred 25 API(s) are enabled
validating Google Cloud Compute Engine quotas
Compute Engine quota is adequate, satisfied 21 rules
DNS_ZONE_NAME required

@dfoleypivotal
I believe this is an issue because I updated the Go variable names to uppercase acronyms (a style improvement). This trickled down into the reading of the config.json config file. Now, reading an old config file is incompatible with master branch because certain fields have different keys.
Try updating the keys of your config to match this:
cat env/pcf/config.json
{
  "DNSZoneName": "pcf-zone",
  "ProjectID": "my-project",
  "BaseImageURL": "https://storage.cloud.google.com/ops-manager-us/pcf-gcp-2.3-build.194.tar.gz",
  "EnvName": "pcf",
  "Region": "us-east1",
  "PivnetAPIToken": "mytoken",
  "Zone1": "us-east1-b",
  "Zone2": "us-east1-c",
  "Zone3": "us-east1-d",
  "SmallFootprint": true,
  "IncludeHealthwatch": false
}

Notice the DNSZoneName. I bet your config has DnsZoneName.

@dfoleypivotal there's more to it than that. There is a bug in the source-config command which is how the environment variables get set. I'm working on a fix now


modified example and added description about

I have modified example for Error state and delegates. When I use SkyFloatingLabelTextField, I think the example isn't proper at README.md
Because when user typing, func textField(_ textField: UITextField, shouldChangeCharactersIn range: NSRange, replacementString string: String) -> Bool method is called but This method's textfield.text bring value that previous value currently entered.
if a user is typing "A", textfield.text is "", if a user is typing "AB", textfield.text is "A". and then if a user is removing "B" of "AB", textfield.text is "AB". so I think that this situation can cause to be confused.

Thanks @devmjun 👍


Error launching server with lms in Linux

lms server start gives this output:
Starting server...
Waking up LM Studio service...
node:events:496
throw er; // Unhandled 'error' event
^
Error: spawn /tmp/.mount_LM_StumCkjVD/lm-studio ENOENT
at ChildProcess._handle.onexit (node:internal/child_process:286:19)
at onErrorNT (node:internal/child_process:484:16)
at process.processTicksAndRejections (node:internal/process/task_queues:82:21)
Emitted 'error' event on ChildProcess instance at:
at ChildProcess._handle.onexit (node:internal/child_process:292:12)
at onErrorNT (node:internal/child_process:484:16)
at process.processTicksAndRejections (node:internal/process/task_queues:82:21) {
errno: -2,
code: 'ENOENT',
syscall: 'spawn /tmp/.mount_LM_StumCkjVD/lm-studio',
path: '/tmp/.mount_LM_StumCkjVD/lm-studio',
spawnargs: [ '--minimized', '--run-as-service' ]
}
Node.js v20.12.2

same error, waiting for some guidance

Same error
OS : Linux ubuntu 22.04
`Starting server...
Waking up LM Studio service...
node:events:496
throw er; // Unhandled 'error' event
^
Error: spawn /tmp/.mount_LM_StuSaBA2B/lm-studio ENOENT
at ChildProcess._handle.onexit (node:internal/child_process:286:19)
at onErrorNT (node:internal/child_process:484:16)
at process.processTicksAndRejections (node:internal/process/task_queues:82:21)
Emitted 'error' event on ChildProcess instance at:
at ChildProcess._handle.onexit (node:internal/child_process:292:12)
at onErrorNT (node:internal/child_process:484:16)
at process.processTicksAndRejections (node:internal/process/task_queues:82:21) {
errno: -2,
code: 'ENOENT',
syscall: 'spawn /tmp/.mount_LM_StuSaBA2B/lm-studio',
path: '/tmp/.mount_LM_StuSaBA2B/lm-studio',
spawnargs: [ '--minimized', '--run-as-service' ]
}
Node.js v20.12.2
`

So I've done some testing and this appears to work if lmstudio gui is running. It utilizes the preexisting .models directory in the tmp folder and doesn't have any issues. however when the GUI isn't running, that directory is removed and lms cli obviously can't access it. But this unfortunately can not work unless you intend to pointlessly run 2 separate servers since the GUI server is already running. lms cli will try to bind itself to the same port, etc. (Obviously expected behavior)
Needless to say I'm having the same issue with lms cli not creating the temporary directory as needed. I verified that not only is /tmp/ is set to 777 but also stickied at 1777 so only the owner can remove it's files in /tmp. If /tmp is wide open, I'm not sure what else needs to be done for the system to create the .models directory in /tmp. Unfortunately the CLI is completely unusable for me at the moment.
This appears to be a bug in the application itself, or an issue/mismatch between lms cli versions and appimage files.
Arch Linux:
lms server start
Starting server...
Waking up LM Studio service...
node:events:496
throw er; // Unhandled 'error' event
^
Error: spawn /tmp/.mount_LM_StukDNA7n/lm-studio ENOENT
at ChildProcess._handle.onexit (node:internal/child_process:286:19)
at onErrorNT (node:internal/child_process:484:16)
at process.processTicksAndRejections (node:internal/process/task_queues:82:21)
Emitted 'error' event on ChildProcess instance at:
at ChildProcess._handle.onexit (node:internal/child_process:292:12)
at onErrorNT (node:internal/child_process:484:16)
at process.processTicksAndRejections (node:internal/process/task_queues:82:21) {
errno: -2,
code: 'ENOENT',
syscall: 'spawn /tmp/.mount_LM_StukDNA7n/lm-studio',
path: '/tmp/.mount_LM_StukDNA7n/lm-studio',
spawnargs: [ '--minimized', '--run-as-service' ]
}
Node.js v20.12.2


Adding Safari support

Hey folks, in response to your toot https://infosec.exchange/@lartsch/109540360663111201 I'd love to try adding support for Safari!
Perhaps I could open an issue in the repo to keep track of the progress?
My first step will be to convert the extension using Xcode and check for compatibility issues that may need to be addressed.
Regarding distribution of the extension, I guess this can be done manually at first?
App Store distribution requires a developer program membership. One option is to publish it using mine, I'd be happy to do it, but that may not be the right solution. There are options to waiver the membership fee, but it's not without some paperwork 🙈 if a non-profit organization doesn't exist yet, https://developer.apple.com/support/membership-fee-waiver/.

Hi Konstantin, thanks again for your involvement.
It would be no issue for me if it was published using your subscription. I think most Apple users like things to be simple and installing via store would be the most user-friendly option. As I do not want to setup a non-profit, the fee waiver is not an option for me right now (but it's good to know that this option exists).
If you prefer access to the repository to create your own branch over creating a fork (in case adjustments need to made for compatibility), please let me know.
When functionality is proven, we can think about a workflow for releases that works for you.

@telsin that's awesome! I'm slowly resuming after the holidays and plan to pick this up in the next couple of days.

I created a repository for the Xcode project and carried out the conversion 🎊 .
The extension converted without warnings and was ready for use “as is” on macOS! The pop-up also works (although some CSS changes may be needed).
Testing the extension with an actual server was not successful (yet) due to a CORS error https://github.com/kkostov/FediActSafari/issues/1 - it seems Safari is validating CORS headers for requests from service workers.
Next, I will see if it's a Safari issue or intentional and if there are configuration options available to make it ignore CORS headers for the extension (otherwise we'd need to whitelist mastodon hosts in the manifest).

The CORS error was addressed by using the manifest for Firefox (v2).
However, fetchBearerToken() does not seem to fetch the access token. The value of text always comes back with `"access_token": null":

@Lartsch do you have any ideas as to what could be the cause of this?

Would love to have a Safari version of this!


Distributed table can't write data to remote shard (v18.14.9)

We're experiencing problems with writing data to tables with Distributed engine after upgrading to version 18.14.9. After inserting data to distributed table, it can be successfully inserted to local shard, but can't be inserted to shard on remote server. I can see following in the log:
On server with data in table with Distributed engine:
2018.10.18 12:10:46.472687 [ 39 ] {} <Trace> distributed_events.Distributed.DirectoryMonitor: Started processing `/var/lib/clickhouse/data/<cluster>/<distributed_table>/<user_name>:<password>@<server_name>:9500/1.bin`
2018.10.18 12:10:46.472780 [ 39 ] {} <Trace> Connection (<remote_server>:9500, <remote_server_ip>): DB::NetException: Unexpected packet from server <remote_server>:9500, <remote_server_ip> (expected Pong, got Data)
2018.10.18 12:10:46.472798 [ 39 ] {} <Trace> Connection (<remote_server>:9500, <remote_server_ip>): Connection was closed, will reconnect.
2018.10.18 12:10:46.472831 [ 39 ] {} <Trace> Connection (<remote_server>:9500, <remote_server_ip>): Connecting. Database: (not specified). User: <user_name>
2018.10.18 12:10:46.473386 [ 39 ] {} <Trace> Connection (<remote_server>:9500, <remote_server_ip>): Connected to ClickHouse server version 18.14.9.
2018.10.18 12:10:46.475212 [ 39 ] {} <Error> <distributed_table>.Distributed.DirectoryMonitor: Code: 49, e.displayText() = DB::Exception: Logical error: empty block received as table structure, e.what() = DB::Exception, Stack trace:

on remote shard
2018.10.18 12:26:16.486439 [ 44 ] {42319601-feec-4ea8-ab11-5f75a6ae50fc} <Error> executeQuery: Code: 210, e.displayText() = DB::NetException: I/O error: Broken pipe:  while reading from socket (<remote_server_ip:54094), e.what() = DB::NetException (from <remote_server_ip>:54094) (in query: INSERT INTO <table> FORMAT Native), Stack trace:

0. /usr/bin/clickhouse-server(StackTrace::StackTrace()+0x16) [0x99e9626]
1. /usr/bin/clickhouse-server(DB::WriteBufferFromPocoSocket::nextImpl()+0x740) [0x9a1f600]
2. /usr/bin/clickhouse-server(DB::WriteBuffer::next()+0x26) [0x3087466]
3. /usr/bin/clickhouse-server(DB::TCPHandler::runImpl()+0x791) [0x3094c11]
4. /usr/bin/clickhouse-server(DB::TCPHandler::run()+0x2b) [0x3095b1b]
5. /usr/bin/clickhouse-server(Poco::Net::TCPServerConnection::start()+0xf) [0x9b67d7f]
6. /usr/bin/clickhouse-server(Poco::Net::TCPServerDispatcher::run()+0x16a) [0x9b6815a]
7. /usr/bin/clickhouse-server(Poco::PooledThread::run()+0x77) [0x9c84f37]
8. /usr/bin/clickhouse-server(Poco::ThreadImpl::runnableEntry(void*)+0x38) [0x9c80aa8]
9. /usr/bin/clickhouse-server() [0xa3da90f]
10. /lib/x86_64-linux-gnu/libpthread.so.0(+0x80a4) [0x7feca00b80a4]
11. /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7fec9f6e087d]

2018.10.18 12:26:16.486613 [ 44 ] {42319601-feec-4ea8-ab11-5f75a6ae50fc} <Warning> TCPHandler: Client has gone away.
2018.10.18 12:26:16.486703 [ 44 ] {} <Information> TCPHandler: Processed in 30.001 sec.
2018.10.18 12:26:16.486724 [ 44 ] {} <Information> TCPHandler: Done processing connection.

We are having the same issue.
ch 18.14.9 / ubuntu 18.04.

We are having the same issue.
ClickHouse server version 18.14.10 revision 54409
ubuntu 18.04.

Somehow I can't reproduce the issue, but judging from the number of upvotes it should be easy. Can anyone affected share their cluster configuration (<remote_servers>, what kind of ENGINE local tables use)?

@ztlpn
engine = ReplicatedMergeTree
cluster:
<CLUSTER_NAME>
      <shard>
        <internal_replication>true</internal_replication>
        <weight>0</weight>
        <replica>
          <host>hdd1</host>
          <port>9000</port>
        </replica>
        <replica>
          <host>hdd2</host>
          <port>9000</port>
        </replica>
      </shard>
      <shard>
        <internal_replication>true</internal_replication>
        <weight>1</weight>
        <replica>
          <host>ssd1</host>
          <port>9000</port>
        </replica>
        <replica>
          <host>ssd2/host>
          <port>9000</port>
        </replica>
      </shard>
      <shard>
        <internal_replication>true</internal_replication>
        <weight>1</weight>
        <replica>
          <host>ssd3</host>
          <port>9000</port>
        </replica>
        <replica>
          <host>ssd4</host>
          <port>9000</port>
        </replica>
      </shard>
    </CLUSTER_NAME>
  </remote_servers>

@denisov-vlad Thanks, but still no luck.
Some additional questions:

Are the local tables replicated (I assume they are, because internal_replication is true)
Are there any materialized views attached to local/distributed tables?
Is the set of columns the same for local and distributed tables? Do the tables contain something other than ordinary columns (aliases, etc.)? Were the tables recently altered?
Were all servers in the cluster the same version (18.14.9) at the moment of error?
What additional settings did you enable?
Does the error appear right away or did some data get successfully inserted? What is the time interval between the last success and the appearance of error?
Any additional details that you can think of.

Yes.
No
Same; no; they were recently created.
Yes.
I can show you config-preprocessed file via telegram.
No.
I'll send you a message through telegram..

The bug appeared when data was inserted into a Distributed table in the Native format. It is fixed in master and in the upcoming v18.14.11 fix-release.


Move DynamicFileCatalog back to core

Which issue does this PR close?
No specific issue but mentioned here https://github.com/apache/datafusion/issues/4850#issuecomment-2142190951

Rationale for this change
Discussed with @alamb here. https://github.com/apache/datafusion/pull/10619#issuecomment-2132163536

What changes are included in this PR?
Move DynamicFileCatalog and object storage-related features from CLI to the DataFusion core.

Are these changes tested?
No, just move original tests to core.

Are there any user-facing changes?
No

There are many changes in datafusion-cli/Cargo.lock. However, I just ran cargo update in datafusion-cli. I'm not sure if it's correct.

Hi @alamb, I encountered a dependency issue with wasm-build.
https://github.com/apache/datafusion/actions/runs/9327817555/job/25678224652?pr=10745
After some research, I found that some people encountered a similar error message, and someone mentioned it could be an issue with Tokio's limited support for WASM. I found that the failed crate, mio, is used by tokio and aws-config. datafusion-cli enables more of Tokio's features than core. I think I might have moved too many objects to core. I might need to keep aws, gcp, or other object-store-related things in cli. I'll keep trying.
However, this also means that this feature will have limited support for WASM.
Do you have any ideas about this issue? Thanks.

To solve the WASM building issue, I disabled the object_store related feature for WASM. There is one remaining issue with macOS that I don't know how to fix.
GitHub Actions Log:
libunwind: stepWithCompactEncoding - invalid compact unwind encoding
error: test failed, to rerun pass `-p datafusion --lib`

Caused by:
  process didn't exit successfully: `/Users/runner/work/datafusion/datafusion/target/debug/deps/datafusion-e14cd1eb6c13d6be` (signal: 6, SIGABRT: process abort signal)

After some research, it seems to be a known issue with Xcode 15. Refer to:

JetBrains YouTrack Issue
Rust Issue #113783

I'm not sure if it's related to the macos-latest GitHub runner.
I have tried the following:

Upgrading Rust to version 1.78.
Upgrading AWS-related crates to the latest version.
Rolling back AWS-related crates to version 0.55.3.

None of these solutions have worked.

I'm not sure if it's related to the macos-latest GitHub runner. I can't reproduce it in my local.

Here is the revised version with fixed grammar:
I have checked that the macos-latest GitHub runner has been updated to Xcode 15.4 after https://github.com/actions/runner-images/pull/9842. So, it may not be an Xcode issue.

Thanks @alamb
I agree with this approach. The dependency of the core should be simpler. Therefore, I think I can wait for the discussion in #10782 to plan the next steps for this issue.

Converting to draft as we work on separating out the catalog a bit more

Hi @alamb, is there any update on this topic? Or maybe I can help by splitting out some traits into the new crate datafusion-catalog first?

Hi @alamb, is there any update on this topic? Or maybe I can help by splitting out some traits into the new crate datafusion-catalog first?

Hi @goldmedal  -- sorry for the lack of update. From my perspective, it is not a good idea to add any more dependencies (even optional) to the core datafusion crate.
After thinking about this some more, I think there is a difference between:

The logic that interprets table names as potential object store locations
The logic that handles urls like s3://... by automatically registering object store factories

Putting the first in the datafusion core makes sense. I think the second comes with many dependencies and thus doesn't make sense.
Thus, what do you think about somehow splitting the two pieces of functionalty apart? Maybe via a trait like
struct DynamicTableProvider {
...
  /// A callback function that is
  url_lookup: Arc<dyn UrlLookup>
}

/// Trait for looking up the correct object store instance based on url
pub trait UrlLookup{
  fn lookup(&self, url: &Url) -> Result<Arc<dyn ObjectStore>>;
}

Thank you for working on this PR / IDE

I think we should close this PR to keep the discussions. Then, I'll create another PR for DynamicTableProvider implementation. WDYT @alamb?

. I'm not sure if there is any similar feature (installing an extension or table function, then query dynamically) in DataFusion, but I can implement DynamicTableProvider first.

I think this is what https://docs.rs/datafusion/latest/datafusion/execution/context/struct.SessionContext.html#method.register_object_store is used for
And access to the remote store is hanlded via and ObjectStore

I think we should close this PR to keep the discussions. Then, I'll create another PR for DynamicTableProvider implementation. WDYT @alamb?

I think this is a good idea
Given how this project has grown, I think it might also be valuable to file a ticket describing the high level idea too -- are you able to do so? I can do so too, but likely won't have a chance until tomorrow

Given how this project has grown, I think it might also be valuable to file a ticket describing the high level idea too -- are you able to do so? I can do so too, but likely won't have a chance until tomorrow

Sure, I can file a ticket for it.

I have filed #10986. Let's move the follow up discussion to it.


ARROW-15961: [C++] Handle some non-nullable fields in Parquet

@kaoutherab I think this should get your PR unstuck. However it's not a complete fix for the underlying issue, we should file a separate JIRA to fix all that up.


[feature request] influx -import options to specify destination db/rp

Add destination database and retention policy arguments to influx -import to override the DML options.
Use case: A large database is exported from TSM storage in compressed format. The resulting compressed file includes DDL and DML statements to create and set the target database and retention policy for imported points.
influx_inspect export -database=mydb -datadir=/var/lib/influxdb/data -waldir=/var/lib/influxdb/wal -compress -out=here.gz

# DML
# CONTEXT-DATABASE:mydb
# CONTEXT-RETENTION-POLICY:myrp

If users wants to import the data in a different database or retention policy, they need to unzip and modify the file - which can be infeasible due to the size of the resulting uncompressed file (e.g. 100GB can result in more than 1TB uncompressed).
Proposed feature request:
Include command line arguments like destdb,destrp to override the DML defintions specified in the export file.
influx -import -compressed -path=dumpfile -pps=1000 -destdb=newdb -destrp=newrp

+1

@sachinrase in this case the destination database and retention policy can be manually created by the user before the import is started - just so we can keep this request simple. Otherwise a number of parameters would be needed like retention policy duration, shard duration, replication factor.

I just had a talk with @kostasb about this and should have a PR submitted on Monday afternoon, fingers crossed ;)

is there any  candidate release rpm i can try to test ?

This would indeed be a useful enhancement.
It took me by surprise actually since the influx -import statement allows for a -database option but it's just not taken into account.
Especially with the crippled backup/restore support in InfluxCloud I'd expect better import/export options to play around with the data...

+1


[Bug]: Cannot see result of postgres query

Is there an existing issue for this?

[X] I have searched the existing issues

Description
response tab indicates that 10 rows were returned, but is blank:

Steps To Reproduce

Create a datasource for the internal DB. Get credentials from here

3. Create a simple query: select * FROM github_issues limit 10;
4. Run it, and you'll see that the response is empty
Public Sample App
No response
Version
Cloud

In a query edit page, appsmith doesn't know how to render data types that are array. For example this kind of column in postgres
redirect_uris character varying(255)[] NOT NULL DEFAULT ARRAY[]::character varying[],

This just shows a blank response. The rest works

@sh-TU thanks for the information! You are right and we are working towards fixing this ASAP

A fix has been created for this on the client side. @rohan-arthur @Nikhil-Nandagopal @sh-TU


[BUG]: User menu should show Command, not Control, on mac

Describe the bug
On mac, it should show the command symbol for the User Settings shortcut.
Steps to Reproduce

yarn tron:start
Click user menu

Expected Behavior
This is what the current production app looks like.

Screenshots and Recordings
This is what I see now.

Desktop OS
macOS 14.6.1
Browser
No response
Version
main, post electron aac758b396
Additional Context
No response

this is so weird since it was fine on tauri


foxit-pdf-reader: Add version <IP_ADDRESS>938

Deprecation note added in #6584

/verify

/verify what about now?

No problem from Germany...

How about cdn02 for you?

Is download speed still an issue for you?

No problem from Germany...

No problem from Taiwan either (3MB/s)

Quite slow for me (120Mbps fibre)

Quite slow for me (120Mbps fibre)

Does cdn07 or cdn09 work better?
Both are fine (>2MB/s) in Taiwan.

What about cdn06? It redirects to cdn06.foxitsoftware.com.cdn.cloudflare.net and may has better speed.

cdn07 is unreachable
cdn09 is slightly better (230Kbps)
cdn06 is 6Mbps, works best


Live transcription silence is being treated as an error

What is the current behavior?
Once I establish a connection, if I'm not sending audio in the next 20-30 seconds, I receive this message:

Deepgram did not provide a response message within the timeout window. See https://dpgr.am/net0000

Steps to reproduce
Connect but don't send bytes and wait (with the keep alive working)
Expected behavior
I would expect the connection to stay open as long as I call keep alive, so I can start sending bytes when the user speaks without having to connect, OR for the reconnection and keep-alive logic to be handled internally by the library.
Please tell us about your environment

Windows 10
C#

Other information
I wan't sure if this is a bug or a feature, but since the documentation for net0000 doesn't say much about this situation, and there's already a keepalive to ensure continued connections, this behavior seemed surprising to me.

I guess the follow up question would be, if this is by design, what should be the best practice? Parse the error for "net0000" and reconnect automatically? Or should I do audio processing to ensure there is relevant data beforehand?

Related to #131
This will be fixed by #132


Broken links in Readme

The links to the Crypto primitives and Plonk Prover are broken.

I can send quick fix

Sure