Building Multi architecture Kolibri on travis

Finally got this to work :smiley:
Supported architectures amd64,arm,aarch64 (arm64)
Please check treehouses/kolibri on Raspberry Pie

Could you investigate more on this @sahilph

=========BUILD MESSAGE=========
Pushing Multiarch Manifests to cloud
===============================
time="2018-05-22T14:23:54Z" level=fatal msg="Couldn't mount blobs for cross-repository push: Blob mount failed to url https://registry-1.docker.io/v2/treehouses/kolibri/blobs/uploads/?from=treehouses%2Fkolibri-tags&mount=sha256%3A9dbc1730be1e0b3e0c6c1e322807366bd74bd16e95b5b049c1bf80b577b23f1f: HTTP status 401"

Could you investigate more on this @sahilph

=========BUILD MESSAGE=========
Pushing Multiarch Manifests to cloud
===============================
time="2018-05-22T14:23:54Z" level=fatal msg="Couldn't mount blobs for cross-repository push: Blob mount failed to url https://registry-1.docker.io/v2/treehouses/kolibri/blobs/uploads/?from=treehouses%2Fkolibri-tags&mount=sha256%3A9dbc1730be1e0b3e0c6c1e322807366bd74bd16e95b5b049c1bf80b577b23f1f: HTTP status 401"

@mappuji This is because of the Travis PR build. It sets tags differently.  Our script is not designed to handle that. That why I disabled it. Also Push builds does the job so no need for PR builds.

Looks like it is working.

Last can we test manifest push with current kolibri-travis-build3 branch and verify together with the result? Then after that, we can set manifest push to master branch again.

@mappuji Sure. I have temporarily change the branch name. Check this build

Works properly. Tested on the rock64 board :+1:


fix(vectorstores.py): fetch embedding for the near_vector method

Solves this issue: #132

The near_vector method accepts a parameter called "near_vector", for which the method will perform vector search to get the nearest vector(s). We should get the vector using the embedding object, just as we did in the hybrid search method, 4 lines above it.

@hkhairy thanks for your contribution!
Unfortunately, the team decided to remove calls to near_vector (see #141) in favor of using hybrid search to do the searches. This is a more flexible option since users can configure how much vector vs keyword search they want by configuring the alpha param.
So, I will close this PR.

@hsm207
Okay, thanks for the headsup :raised_hands:
I have to say, your solution is more robust, since it consolidates all the calls to embedding.embed_query in one place.
Looking forward to collaborating with you as well :ok_hand:


Fix AUD JWT default function

The default aud function for Access Token in JWT format is incorrect, it has client id like an ID Token instead of the resource supposed it is for


vSphere VM Template not found

Versions:

Version of Chef-Provisioning: 2.7.0
Version of Chef-Provisioning-vSphere: 2.3.1

Platform Details

Version of vSphere/vCenter:  <IP_ADDRESS>00
Version of ESXi: VMware ESXi, 6.5.0, 8294253

Scenario:
I want to create a VM from a template, bootstrap it to Chef server and run a cookbook on it.
Steps to Reproduce:
Below is the excerpt from my .kitchen.yml. I tried different "resource_pool" entries.
driver:
  name: vsphere
  driver_options:
    host: <IP_ADDRESS>
    user: terraform
    password: ***
    insecure: true
  machine_options:
    start_timeout: 600
    create_timeout: 600
    ready_timeout: 90
    bootstrap_options:
      use_linked_clone: true
      datacenter: 'Datacenter'
      template_name: 'CM_TMPL_TF_W10Pro1709x64_Symantec'
      vm_folder: 'Terraform'
      num_cpus: 2,
      network_name:
        - '415-SDE-Virtual'
      memory_mb: 4096
      resource_pool: 'vSAN_Tier2'
      ssh:
        user: admin
        password: ***
        elevated: true
        port: 5985
...
platforms:
 - name: Win2012x64
suites:
  - name: default
    run_list:
      - recipe[m4::default]

Expected Result:
VM is created from a template on vCenter as a first step.
Actual Result:
-----> Starting Kitchen (v1.20.0)
-----> Cleaning up any prior instances of <default-Win2012x64>
-----> Destroying <default-Win2012x64>...
       Finished destroying <default-Win2012x64> (0m0.00s).
-----> Testing <default-Win2012x64>
-----> Creating <default-Win2012x64>...
[2018-10-31T16:11:15+01:00] WARN: No cookbooks directory found at or above current directory.  Assuming /root/gitlab-runner/builds/6567bb15/0/gitlab/Cookbooks/m4.
creating machine default-Win2012x64-922ba976 on vsphere://<IP_ADDRESS>/sdk?use_ssl=true&insecure=true
  datacenter: "Datacenter"
  template_name: "CM_TMPL_TF_W10Pro1709x64_Symantec"
  vm_folder: "Terraform"
  num_cpus: 2
  network_name: ["415-SDE-Virtual"]
  memory_mb: 4096
  ssh: {:user=>"terraform", :password=>"*********", :elevated=>true, :port=>5985}
establishing connection to <IP_ADDRESS>
create node default-Win2012x64-922ba976 at chefzero://localhost:1
  add normal.chef_provisioning = {}
  add normal.tags = nil
>>>>>> ------Exception-------
>>>>>> Class: Kitchen::ActionFailed
>>>>>> Message: 1 actions failed.
>>>>>>     Failed to complete #create action: [vSphere VM Template not found [/CM_TMPL_TF_W10Pro1709x64_Symantec]] on default-Win2012x64
>>>>>> ----------------------
>>>>>> Please see .kitchen/logs/kitchen.log for more details
>>>>>> Also try running `kitchen diagnose --all` for configuration

I would appreciate any help in the subject.
Best regards,
Marek

Hey @marekmrowca
You might need to specify the template_folder under bootstrap_options

Thank you @josh-barker
It helped!


rspec upgrade

upgrade to rspec 3.0
fix rspec warning
fix stalled rspec run

Coverage decreased (-1.86%) to 84.68% when pulling 119c35e554a4419c5c9025986d37e4e7fb07cd89 on niamster:master into df9f343ef9bbc063a2b15f53bd967a839fc57253 on celluloid:master.

Closing in favour of #45


Documentation on how users can write their own functions.

Why are we implementing it? (sales eng)
What are the typical use cases?
Communication goals (e.g. detailed howto vs orientation)
Good locations for content in docs structure
How does this work? (devs)
Example sql
Corner cases, gotchas
Are there relevant blog posts or outside documentation about the concept/feature?
Link to relevant commits and regression tests if applicable

Can you elaborate on this? Like a section about plpgsql basics?


Improve SchemaObject interface by allowing ReferenceObject in some properties

Improved SchemaObject interface. Many of its properties can be of type ReferenceObject per OpenAPI specification.
Issue
Currently OpenAPI document that make use of $ref in SchemaObjects do not pass TypeScript type checking because SchemaObject does not allow ReferenceObjects although the OpenAPI specification allows.
Currently the following valid use cases of ReferenceObject do not pass type checking.
import {
  SchemaObject,
} from 'express-openapi-validate/dist/OpenApiDocument'

const refInOneOf: SchemaObject = {
  oneOf: [
    {
      $ref: '#/components/schemas/FooBar'
    }
  ]
}

const refInNot: SchemaObject = {
  not: {
    $ref: '#/components/schemas/FooBar'
  }
}

const refInItems: SchemaObject = {
  type: 'array',
  items: {
    $ref: '#/components/schemas/FooBar'
  }
}

const refInAdditionalProperties: SchemaObject = {
  additionalProperties: {
    $ref: '#/components/schemas/FooBar'
  }
}

const refInProperty: SchemaObject = {
  type: 'object',
  properties: {
    fooBar: {
      $ref: '#/components/schemas/FooBar'
    }
  }
}

Argument for the changes in this PR
The use cases demonstrated above should be valid according to OpenAPI specification at https://github.com/OAI/OpenAPI-Specification/blob/main/versions/3.0.3.md#schema-object and should thus pass type checking.

allOf - Inline or referenced schema MUST be of a Schema Object and not a standard JSON Schema.
oneOf - Inline or referenced schema MUST be of a Schema Object and not a standard JSON Schema.
anyOf - Inline or referenced schema MUST be of a Schema Object and not a standard JSON Schema.
not - Inline or referenced schema MUST be of a Schema Object and not a standard JSON Schema.
items - Value MUST be an object and not an array. Inline or referenced schema MUST be of a Schema Object and not a standard JSON Schema. items MUST be present if the type is array.
properties - Property definitions MUST be a Schema Object and not a standard JSON Schema (inline or referenced).
additionalProperties - Value can be boolean or object. Inline or referenced schema MUST be of a Schema Object and not a standard JSON Schema. Consistent with JSON Schema, additionalProperties defaults to true.

Explanation on the part of properties leaves open the question whether individual properties can be ReferenceObjects. But I think they can because there is an example of such usage in the same specification document https://github.com/OAI/OpenAPI-Specification/blob/main/versions/3.0.3.md#simple-model.
{
  "type": "object",
  "required": [
    "name"
  ],
  "properties": {
    "name": {
      "type": "string"
    },
    "address": {
      "$ref": "#/components/schemas/Address"
    },
    "age": {
      "type": "integer",
      "format": "int32",
      "minimum": 0
    }
  }
}

Hei @Hilzu! Could you take a look at this pull request? I would like to have this issue fixed here in the upstream.


Feature/skip links

Adds the skip-to links feature.
Minor (visual) change when tabbed to, from our current version: upped the font-size from nominally 16px to 21px after a quick chat with @AndrewArch, and used our relative spacing mixin for the top/left offset instead of a static 10px.
This should cater for #22.
Current feature documentation: http://guides.service.gov.au/design-guide/patterns/navigation/index.html#skip-links
This PR also contains the a merge of develop to avoid conflicts. :)

@dominikwilkowski —
For some reason the two partials in lib/ are still showing in the file listing but they are defs. deleted (and inaccessible in the online quick-edit mode).
Should be it?
I also found two other instances of /dist/ — edited those.


Added governance example to release-1.1

Update governance example Cargo.toml + examples README for release-1.1

Once Bonsai is deployed with 1.1, we'll rerun CI here to ensure it passes and then merge.


Any thoughts on porting / testing this on Theia?

Theia being a cloud version of VS Code, of sorts, they now support VS Code plugins. Others I have tried seemed to work, at least in part. Thanks in advance.
https://github.com/theia-ide/theia/wiki/Testing-VS-Code-extensions

An our purpose to support Marp in VS Code is not change. However, I think Theia is a interesting project.
Currently Theia's Markdown preview seems not to be able to extend by VS Code extension, but the library used to render Markdown internally is very similar to VS Code and Marp. It might be good to request adding support of VS Code's Markdown extension to Theia.
Or an extension for Theia based on Marp Core can create, just like a package for Atom.

Note: The support of Gitpod (Theia based IDE) was added to Marp CLI by contributor (marp-team/marp-cli#113). It does not a support of Theia as an extension but might useful for someone.

The gitpod team will implement features that support VScode extensions.
I will talk to Gitpod team as part of the activation of Gitpod in Japan.


Renaming batches of results

First, thanks for open-source this great project.
I would like to suggest a useful feature that will enable to edit batch names (really useful when you have many batches and you want to rename them to be with a meaningful concise names).

Hi David,
It is a very good idea, likely useful to many users.
Since it's a small feature, I expect to have something to show you by next week.
I'll add a change label menu item before the redo:

Hi David,
I added the feature, and will push/deploy it very soon 👍
Arthur

This is what is looks like:

Looks great, many thanks!


Private Chain

Epic for tracking Private Chain related issues/tickets

Epic for tracking Private Chain related issues/tickets

Where are those?


Demo Site: Add redis cache-handler

Description
This adds redis to docker-compose and a cache-handler implementation based on "a big project of vivid-planet".
I wanted to have the cache-handler written in typescript, but was not able to use ts-node for that. next.config.js needs the path (usually using require.resolve) to the cache-handler file, not the import. But when running using ts-node the js file doesn't exist, and ts is not supported. I didn't find a way to get this running with ts-node, so I switched that to a build-step using tsc also in dev.
Alternative: write cache-handler not in typescript (all examples I found do that)
Open TODOs/questions

[x] Merge parent PR #2595
[ ] decide if the compile step (in dev) is Ok, or find a better solution
[ ] add to starter

decide if the compile step (in dev) is Ok, or find a better solution

I think it's okay, why wouldn't it be?
Alternative: @ts-check comment, as we do for next.config.js: https://github.com/vivid-planet/comet-starter/blob/main/site/next.config.js#L3

why wouldn't it be?

because it is not using ts-node anymore and requires the compile step also locally

@ts-check

I would like to write "real" typescript

because it is not using ts-node anymore and requires the compile step also locally

IMO this doesn't make much difference.


Is it possible to specify update strategy (e.g. x days since update)?

Hello and thanks for the great work.
We want to define our update policy, something simple like:

Package must be updated if there were more than Y patches since local version
Package must be updated only if X days passed since the author updated the repo (to avoid updating too fast and catch early bugs)

Is this possible?

Seems like a use case that would be more appropriate for dependency management bots like @dependabot and such.

@i0natan True if you don't want to automate that.


Using Multiple GPUs getting multiprocessing error (2 GPUs Each 24 GB)

Error:
AttributeError: Can't pickle local object 'add_hook_to_module..new_forward'
[W CudaIPCTypes.cpp:16] Producer process has been terminated before all shared CUDA tensors released. See Note [Sharing CUDA tensors]

I'm getting the same error. @NITESHCHACHRIYA  did you solve it?

What GPU memory you have? I was unable to use multiple GPUs was getting same error.  So on single GPU I put model.requires_grad=False and it worked for me for batch size=32. I hope this will help. I had 23 GB GPU memory.

On 25-Jun-2024, at 3:50 AM, gerald4 @.***> wrote:
I'm getting the same error. @NITESHCHACHRIYA https://github.com/NITESHCHACHRIYA did you solve it?
—
Reply to this email directly, view it on GitHub https://github.com/llm-attacks/llm-attacks/issues/97#issuecomment-2187508364, or unsubscribe https://github.com/notifications/unsubscribe-auth/AQHFBOCURS5P7TF2TIE3EJ3ZJCLTHAVCNFSM6AAAAABIF2IEDKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCOBXGUYDQMZWGQ.
You are receiving this because you were mentioned.

No, Authors have mentioned somewhere in Github issues that multiple GPUs are not supported. Use single gpu reduce batch size to 32 and use mode.requires_grad=False. I was able to run individual behaviours and demo after these changes.

On 25-Jun-2024, at 3:50 AM, gerald4 @.***> wrote:
I'm getting the same error. @NITESHCHACHRIYA https://github.com/NITESHCHACHRIYA did you solve it?
—
Reply to this email directly, view it on GitHub https://github.com/llm-attacks/llm-attacks/issues/97#issuecomment-2187508364, or unsubscribe https://github.com/notifications/unsubscribe-auth/AQHFBOCURS5P7TF2TIE3EJ3ZJCLTHAVCNFSM6AAAAABIF2IEDKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCOBXGUYDQMZWGQ.
You are receiving this because you were mentioned.

Why do you need to write mode.requires_grad = False?

What GPU memory you have? I was unable to use multiple GPUs was getting same error. So on single GPU I put model.requires_grad=False and it worked for me for batch size=32. I hope this will help. I had 23 GB GPU memory.
…
On 25-Jun-2024, at 3:50 AM, gerald4 @.***> wrote: I'm getting the same error. @NITESHCHACHRIYA https://github.com/NITESHCHACHRIYA did you solve it? — Reply to this email directly, view it on GitHub <#97 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AQHFBOCURS5P7TF2TIE3EJ3ZJCLTHAVCNFSM6AAAAABIF2IEDKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCOBXGUYDQMZWGQ. You are receiving this because you were mentioned.

I also have 23GB for single gpu, and i reduce the batch size to 32, and set model.requires_grad=False right after loading the model. yet still i get cuda out of memory error, how do you solve this?

The error shouldn’t come for individual_behaviours. I had the same setup I was able to run it.

On 18-Jul-2024, at 1:34 PM, SophieZheng998 @.***> wrote:
What GPU memory you have? I was unable to use multiple GPUs was getting same error. So on single GPU I put model.requires_grad=False and it worked for me for batch size=32. I hope this will help. I had 23 GB GPU memory.
… x-msg://1/#
On 25-Jun-2024, at 3:50 AM, gerald4 @.***> wrote: I'm getting the same error. @NITESHCHACHRIYA https://github.com/NITESHCHACHRIYA https://github.com/NITESHCHACHRIYA did you solve it? — Reply to this email directly, view it on GitHub <#97 (comment) https://github.com/llm-attacks/llm-attacks/issues/97#issuecomment-2187508364>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AQHFBOCURS5P7TF2TIE3EJ3ZJCLTHAVCNFSM6AAAAABIF2IEDKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCOBXGUYDQMZWGQ. You are receiving this because you were mentioned.
I also have 23GB for single gpu, and i reduce the batch size to 32, and set model.requires_grad=False right after loading the model. yet still i get cuda out of memory error, how do you solve this?
—
Reply to this email directly, view it on GitHub https://github.com/llm-attacks/llm-attacks/issues/97#issuecomment-2235881795, or unsubscribe https://github.com/notifications/unsubscribe-auth/AQHFBOHTREZ4UCDPVV4BAH3ZM5ZJNAVCNFSM6AAAAABIF2IEDKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEMZVHA4DCNZZGU.
You are receiving this because you were mentioned.


Also suggested by @gkleiman

and I also think it would be good if the UI would show non-running tasks with a reservation as “suspended” or something like that instead of as “waiting"

Flagging as bug

I believe we should also expose a way to send POST /v2/tasks/delete?wipe=true to "wipe" the volumes.

After talking with @gkleiman and @aquamatthias we agreed that the correct solution here is to add a "Kill & Wipe" button for stateful tasks.


Forced dark theme doesn't work on HTML elements.

Describe the issue
Hi
While using ready to use pico.css or sass generated pico.css I tried to set a dark theme for a nav element:
<nav data-theme="dark">
...
</nav>

but the light theme remains active.
If change header to the dark mode - just a color changes.
<header data-theme="dark">

Current Behavior
According to the docs Themes can be forced on document level or on any HTML element, but data-theme="dark" not working on nav (the same is with any ul or li within nav, but works with html).
Expected Behavior
Dark theme should work on a specific HTML element.
Reproduction URL
Sandbox
Environment
Example: OS, versions, browser details.
Ubuntu 22.04
Pico v.1.5.11
Tested with Chromium, Opera, Firefox.

The same is with picocss v2

Yes, "...or on any HTML element" is too broad a statement.
It's true that the CSS variables impact all HTML elements, but not all HTML elements specifically define the color/background CSS properties.
And I would prefer to keep it like this. To limit the amount of CSS definitions and because most of the time, we want the backgrounds to stay transparent (if you play with backgrounds in the parent for example).
I will update the docs.

@igolkotek, I improved the documentation on color schemes.
https://v2.picocss.com/docs/color-schemes
Is it better? Any suggestions?

Thank you, view it's ok. At least we know what to do with it )


[GHSA-fwr7-v2mv-hh25] Prototype Pollution in async #199

Successor of #199 and possibly #200.

Oh my, I really do not understand that branching :thinking: there seem to be two branches opened in this repo for every PR and a fork of this repo for my user too? I suppose I can only edit my fork... but that's not where my PR originates from.
Well I can't switch back to the original PR now that its branches have been (automatically?) removed. So this one will stay open, eventhough the improvement check fails.

Thank you for providing information about the backported fix.


move browserify as peerDependency?

The new 7.0 was released some days ago, but grunt-browserify is using 6.x, maybe list it as a peerDependency?

Why not move it as a peer again with a condition like this: >= 6.0.0 < 9
(Note sure about the lower version)

So it looks like I pulled the peer dep (see here back when it appeared that peer deps were going to be deprecated in NPM. Which NPM is still considering: https://github.com/npm/npm/issues/5080

In this case I think we could put it back... This will prevent a project from getting multiple browserify versions (see http://blog.nodejs.org/2013/02/07/peer-dependencies/)

Yeah it can always be pulled again if NPM deprecates it. I'd say that the minimum version in the range should match the most recent major version release though. (So in this case 7.0.0)


Dockerise application

There has been quite a bit of confusion surrounding node versions and other dependants. Locking it 14 LTS and dockerising the app should help with this

May I claim this issue ?? Seems simple enough. The only thing we need to do is create a docker image with Node release version 14 ?

I've seen that you are already working on this issue. Nevermind


fix: quickstart filename capitalisation not clear

Fixes Issue
Fixes #6911

Changes proposed

Changed the wording of point number 7 of quickstart documentation for better clarity.
Before

After

Check List (Check all the applicable boxes)

[x] My code follows the code style of this project.
[x] My change requires changes to the documentation.
[x] I have updated the documentation accordingly.
[x] All new and existing tests passed.
[x] This PR does not contain plagiarized content.
[x] The title of my pull request is a short description of the requested changes.

Note to reviewers

Thanks fort the contribution @zaidahmad25 👍
Great collaboration!


[cpr] update to 1.6.2

Library name: cpr
New version number: 1.6.2
Current vcpkg support is for version 1.5.2 and latest available version is 1.6.2. https://github.com/whoshuu/cpr

Hello,
Any update on this request?

Hello,
Any update on this request?

This pr has encountered some obstacles, I will deal with it

@MFMokbel
cpr has been updated to 1.6.2, please update to the latest vcpkg to rebuild this port.

Thank you, @NancyLi1013. All's good.


chore(deps): upgrade dependencies

Upgrades project dependencies. See details in workflow run.

Automatically created by projen via the "upgrade-2.x" workflow

⚪ Backport skipped
The pull request was not backported as there were no branches to backport to. If this is a mistake, please apply the desired version labels or run the backport tool manually.
Manual backport
To create the backport manually run:
backport --pr 499

Questions ?
Please refer to the Backport tool documentation


chore(release): v1.1.0 🎉

:robot: I have created a release beep boop
1.1.0 (2024-06-14)
Features

alias: support dobj alias defs used as input var names (e0ddf3e)

This PR was generated with Release Please. See documentation.

:robot: Release is at https://github.com/ehmpathy/dynamodb-dao-generator/releases/tag/v1.1.0 :sunflower:


perf(runtime-dom): Array.from and concat is more efficient than ...

https://codesandbox.io/s/xenodochial-fire-gdrkn?file=/src/index.js

hello, in you test case and the result I get .... :

@buns-li  I'm surprised it's not stable.  that's interesting

it's stable in my<EMAIL_ADDRESS>It may be version-related.

it's stable in my<EMAIL_ADDRESS>It may be version-related.

my chrome version is :

it's stable in my<EMAIL_ADDRESS>It may be version-related.

my chrome<EMAIL_ADDRESS>when  I try again the case and the result.....

@buns-li  wtf.... Only God knows why.
test in local index.html
<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <title>Document</title>
</head>
<body>

<script>

let set = new Set()
set.add(1)
set.add(2)
set.add(3)
set.add(4)
let val = 0

console.log([val,...set])
console.log([val].concat(Array.from(set)))
console.time('...')
for (let i = 0; i < 10000; i++) {
    let val1 = [val,...set]
}
console.timeEnd('...')

console.time('concat')
for (let i = 0; i < 10000; i++) {
    let val2 = [val].concat(Array.from(set))
}
console.timeEnd('concat')
</script>

</body>
</html>

forget it . It's not important logic.

I created a performance benchmark for this:

It seems that [... transitionClasses, value] is the fastest in most cases (even much faster than [value, ...transitionClasses] in Chrome when the set is relatively large).

@Justineo good job, thanks


cannot add gateway

i try install from ansible & docker images
it's a same problem cannot select network server and gateway-profile

my mistake i forget add service profile


feat: add session keys (alias) for offchain spaces

Summary

Closes: #282
This is the last PR required to complete all the features needed in #282
This PR introduces alias (or session key), allowing a logged in user to send and save request on the sequencer, without being prompt for message signing.
This is done by signing the message via a proxy wallet (see #282 for description on how it works, or the original code from snapshot:
https://github.com/snapshot-labs/snapshot/blob/bcf4331b2496871eeb9ae4deca56562dba9287ad/src/composables/useAliasAction.ts#L1-L7
Alias signing are currently enabled on follow and unfollow spaces requests.
Will be added to profile edition later
How to test

Connect to an ETH wallet
Go to an offchain spaces
Follow/Unfollow some spaces
On the first time, you will be prompt to sign a transaction to create an alias
After signing, the follow/unfollow request should succeed
Subsequent follow/unfollow request should not ask for message signing
Clear the localstorage for ui.alias
It should ask to sign an alias creation transaction again, when following/unfollowing an offchain spaces.

NOTE: You should not be prompted for signing a follow/unfollow tx anymore, only alias when not existing
To-Do

[ ] Improve code
[ ] Waiting for #293 to be merged, and change this master base to master

When I try to follow or unfollow, it throw an error:

We shouldn't have tooltip in the Follow / Following button
When I login with a different account I still see my previous logged account followed spaces
Button "Unfollow" should have red borders
Left sidebar with followed space should be scrollable (similar than proposal page right sidebar)

Let's review things in order, and review this one first https://github.com/snapshot-labs/sx-monorepo/pull/288

Can you fix conflicts here?

Done
Just noticed that alias does not work with Starknet wallet, fix in progress

mhhh, maybe we should wait after #359 to support starknet wallets

@wa0x6e can you resolve conflicts and merge?


when using huggingface pretrained model with multi-gpu, model parameters were duplicate for every gpu in ram

Bug description
when using huggingface pretrained model with multi-gpu, model parameters were duplicate for every gpu in ram
How to reproduce the bug
trainer = Trainer(
        max_epochs=1,
        devices=args.num_devices,
        precision=16,
        strategy="deepspeed_stage_3",
        accelerator='gpu',
        num_nodes=args.num_nodes,

    )
from transformers import (
    AdamW,
    GPTNeoForCausalLM,
    GPT2Tokenizer,
    AutoTokenizer,
    AutoModelForCausalLM,
    get_linear_schedule_with_warmup,
)
class AlpsModule(LightningModule):
    def __init__(
        self,
        model_name_or_path: str = "EleutherAI/gpt-j-6B",
        cache_dir: str ="/mntnlp/yumu/gpt-neo-x/" ,
        num_labels: int = 2,
        learning_rate: float = 5e-6,
        adam_epsilon: float = 3e-8,
        warmup_steps: int = 30,
        weight_decay: float = 0.01,
        **kwargs,
    ):
        super().__init__()
        self.save_hyperparameters()

        self.model = AutoModelForCausalLM.from_pretrained(model_name_or_path
                                                        ,pad_token_id=self.tokenizer.pad_token_id
                                                        ,bos_token_id=self.tokenizer.bos_token_id
                                                        ,eos_token_id=self.tokenizer.eos_token_id
                                                        , cache_dir=cache_dir
                                                         # ,low_cpu_mem_usage=True
                                                        ).half()

Error messages and logs
# Error messages and logs here please

Environment

Current environment
#- Lightning Component (e.g. Trainer, LightningModule, LightningApp, LightningWork, LightningFlow):
#- PyTorch Lightning Version (e.g., 1.5.0):
#- Lightning App Version (e.g., 0.5.2):
#- PyTorch Version (e.g., 2.0):
#- Python version (e.g., 3.9):
#- OS (e.g., Linux):
#- CUDA/cuDNN version:
#- GPU models and configuration:
#- How you installed Lightning(`conda`, `pip`, source):
#- Running environment of LightningApp (e.g. local, cloud):

More info
No response
cc @awaelchli

if you got 8 gpu, the cost cpu memory = 2 * model_size* gpu_numbers

Hey @linyubupa
This is expected in the way you are initializing the model. I can see from the code snippet that you create the model in __init__. This isn't wrong, but for large models like yours it is inefficient. I recommend moving the initialization into this special Lightning hook:
def configure_sharded_model(self):
      self.model = AutoModelForCausalLM.from_pretrained(...)

Here is the documentation for working with deepspeed models (and also documentation for configure_sharded_model): https://pytorch-lightning.readthedocs.io/en/stable/advanced/model_parallel.html#shard-model-instantly-to-reduce-initialization-time-memory

Please let me know if that helps :)

Please let me know if that helps :)

I had the same problem, but this method didn't solve it

Hey @linyubupa
This is expected in the way you are initializing the model. I can see from the code snippet that you create the model in __init__. This isn't wrong, but for large models like yours it is inefficient. I recommend moving the initialization into this special Lightning hook:
def configure_sharded_model(self):
      self.model = AutoModelForCausalLM.from_pretrained(...)

Here is the documentation for working with deepspeed models (and also documentation for configure_sharded_model): https://pytorch-lightning.readthedocs.io/en/stable/advanced/model_parallel.html#shard-model-instantly-to-reduce-initialization-time-memory

I had the same problem, but this method didn't solve it

Hey @linyubupa
This is expected in the way you are initializing the model. I can see from the code snippet that you create the model in __init__. This isn't wrong, but for large models like yours it is inefficient. I recommend moving the initialization into this special Lightning hook:
def configure_sharded_model(self):
      self.model = AutoModelForCausalLM.from_pretrained(...)

Here is the documentation for working with deepspeed models (and also documentation for configure_sharded_model): https://pytorch-lightning.readthedocs.io/en/stable/advanced/model_parallel.html#shard-model-instantly-to-reduce-initialization-time-memory

Yeah had same issue, and above does not solve it

def configure_sharded_model(self):

sorry for late reply，I build up model in configure_sharded_model , but the cpu memory still cost amountly

Same issue. After I put the model initialization into the configure_sharded_model, I return a new error that shows the loaded parameters are trying to assign to empty tensors.

Same issue. After I put the model initialization into the configure_sharded_model, I return a new error that shows the loaded parameters are trying to assign to empty tensors.

It seems the model initialization should be here, but loading pre-trained weights should not be put here.

Same issue. After I put the model initialization into the configure_sharded_model, I return a new error that shows the loaded parameters are trying to assign to empty tensors.

It seems the model initialization should be here, but loading pre-trained weights should not be put here.

Did you find a solution for this?

any solution about this? i really need help.

I am facing a similar issue

I think one possible solution is to convert the pretrained model weights to the deepspeed zero3 shared model formate, but I haven't tried it yet.

Is there code to try it out?

any solution about this? i really need help.

@saketsathe  @KzZheng
i tried like this.
`def configure_sharded_model(self):
print("start configure sharded model ")
# 모델 랜덤 초기화
llamaconfig = LlamaConfig.from_pretrained("decapoda-research/llama-7b-hf")
self.model = LlamaForCausalLM(llamaconfig)
    self.model.set_adapter(self.adapter_config)
    freeze_except_adapter(self.model, self.adapter_config)

    # 단일 weight의 list
    params_to_gather = [self.model.model.layers[0].self_attn.q_proj.weight]

    # 각 프로세스마다 실행됨.
    # checkpoint shard 에 있는 namedparameter 찾아서, 내 모델에서 동일한 named parameter 있으면 변경.
    # 일단 한번 출력. 이 weight가 값이 몇인지. 나중에 같은 코드로 0으로 변경된거 보기.

    # 변경 전 값 확인
    # check value before change
    with deepspeed.zero.GatheredParameters(params_to_gather, modifier_rank=0):
        print("\n random 초기화 된 weight \n", self.model.model.layers[0].self_attn.q_proj.weight[0, : 5])
        # 없음.

    time.sleep(3)

    # 1. 파일 하나씩 부르기
    # 2. 모델 파라미터랑 파일이랑 key 매칭.
    # 3. 값 넣기
    # GPU 한개 에서만 돌림.
    if torch.distributed.get_rank() == 0:
        with deepspeed.zero.GatheredParameters(params_to_gather, modifier_rank=0):
            self.model.model.layers[0].self_attn.q_proj.weight[0, : 5] = 0
        # 없음.
        #  # 체크포인트 파일 확인
        # SHARDED_FILE_PATH = "/home2/leeg/.cache/huggingface/hub/models--decapoda-research--llama-7b-hf/snapshots/5f98eefcc80e437ef68d457ad7bf167c2c6a1348"

        # # 내 모델 올린거에서 state dict
        # #model_named_params = self.model.model.named_parameters()

        # # self.model.get_parameter()

        # # 체크포인트 파일 불러오기
        # PATH_LIST = [os.path.join(SHARDED_FILE_PATH, f"pytorch_model-000{i:02}-of-00033.bin") for i in range(1, 34)]
        # for PATH in tqdm(PATH_LIST):

        #     # single checkpoint shard file  's   state dict load
        #     file_state_dict = torch.load(PATH)

        #     # 내 모델에 있는 named_parameters 에서
        #     named_parameters = dict(self.model.model.named_parameters())

        #     # 불러온 checkpoint file에 있는 key를 가지고 비교.  key가 내 모델에도 있으면, 파일에서 value 가져옴
        #     params_to_gather = [named_parameters[k] for k in file_state_dict.keys() if k in named_parameters]

        #     # for cp_k, cp_v in file_state_dict.items():
        #     #     if "inv_freq" in cp_k:
        #     #         continue

        #         # model_p = self.model.model.get_parameter(cp_k)
        #         # sharded_model_ps_dict[cp_k] = self.model.model.get_parameter(cp_k)
        #     with deepspeed.zero.GatheredParameters(params=params_to_gather, modifier_rank=0):
        #         self.model.model.load_state_dict(file_state_dict, strict=False)

    dist.barrier()

    # 각 프로세스마다 실행됨.
    # 변경 후 값 확인
    # check value after change
    with deepspeed.zero.GatheredParameters(params_to_gather, modifier_rank=0):
        print("\n barrier weight ", self.model.model.layers[0].self_attn.q_proj.weight[0, : 5])
        # 없음.`

The problem is, when i use from_pretrained("~~") in LightningModule's configure_sharded_model, Lightning Strategy Deepspeed 3  disturbs from_pretrained's assign work.
so, i tried using manual assignment rather than using from_pretrained, from sharded checkpoint file's parameter tensor to my model variable.
i didn't firmly figured out all of this, but i experimented below things.

can call my random initialized parameters from GPUs or CPUs... where ever it is sent by action of deepspeed 3 strategy.
can change 1 (parameter)'s value in context manager     deepspeed.zero.GatheredParameters( ~~ ).

so, i think calling pretrained parameter file manually, and change my random initialized model parameters in  deepspeed.zero.GatheredParameters  is suitable approach.

I solve this by using deepspeed init with transformers trainer : https://huggingface.co/docs/transformers/main_classes/deepspeed
、、、
deepspeed --num_gpus 8 --num_nodes 2 --hostfile hostfile --master_addr hostname1 --master_port=9901
your_program.py  --deepspeed ds_config.json
、、、

Is there any update on this issue?

update?

same...

any updates?


feat: OptimizationModel controller/service/repository

This PR integrates initial optimization model management (creation/retire/get by id) according to SCRUM-21
I also implemented full integration tests for each feature (which require Docker to be installed on the machine to be running since I'm using Testcontainers to run Postgres inside the test).

Test is not passing prob due to GitHub configuration, try to run them inside IntelliJ IDEA

I will check pipeline later today, its running surefire:test so it should pass if all tests are configure/annotated properly.
for double check you can run surefire:test in plain terminal to check if its working
error message: Errors:
Error:    OptimizationModelIntegrationTests.givenCreatedModel_customerShouldGetForbidden » ParameterResolution Failed to resolve parameter [it.polimi.productionoptimiserapi.repositories.OptimizationModelRepository repository] in constructor [public…..

You need to approve the PR to merge


strip before parsing?

Issue description
it looks like a leading space makes the parser crash. works: "45 5 * * sun" breaks: " 45 5 * * sun". could fugit be changed to simply strip the string before parsing?
How to reproduce
c = Fugit.parse(" 45 5 * * sun")

Error and error backtrace (if any)
Message: invalid cron string " 45 5 * * sun"

Expected behaviour
cron string to parse without error
Context
ruby-clock (0.8.0.rc3)
rufus-scheduler (3.8.0)
fugit (1.5.2)
et-orbi (1.2.5)
raabro (1.4.0)

Could fugit be changed to simply strip the string before parsing?

OK, I will do it then.

Hello @jjb please tell me if those changes are OK and/or if you have further suggestions/feedback. Once it's all good, I'll release fugit 1.6.0.
Thanks for reporting this inconsistency!

Wow, nice! Sorry for not responding earlier, I need to adjust my github notifications apparently.
Thanks!


This page is missing VB 16 and soon 16.9

[Enter feedback here]
It needs to track latest version

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 72e781e2-125c-978e-6b24-72dfd6427218
Version Independent ID: 0d6ac342-0d18-443f-bb4b-b6a2f02e93bb
Content: Select the Visual Basic language version - Visual Basic
Content Source: docs/visual-basic/language-reference/configure-language-version.md
Product: dotnet-visualbasic
GitHub Login: @KathleenDollard
Microsoft Alias: kdollard

Thanks @paul1956
It looks like @Youssef1313 has already opened a PR to address this.


proposal: compress/zstd: implement support for compressing and decompressing

Zstandard (RFC 8878) is well-positioned to replaced GZIP (RFC 1952) as the de-facto compression format.
Zstandard:

is faster to compress/decompress than GZIP,
has much better compression ratios,
is well-specified,
is backed by a prominent company, and
is seeing increased adoption in use with archives and as a browser content encoding.

Given the likely ubiquitous place Zstandard is going have in the future, I propose first-class adoption of Zstandard in the stdlib.
Existing Go implementations:

@klauspost has an excellent Zstandard implementation at github.com/klauspost/compress/zstd, but the API is a somewhat complex and it makes use of native assembly.
@ianlancetaylor implemented a Zstandard decompressor in internal/zstd without any use of unsafe or assembly.

Some goals for this package:

The API be simple and similar to existing packages like compress/gzip.
It does not use unsafe or assembly similar to the existing compress package.

Performance is always nice, but Go community surveys consistently shows security as more important than performance.
Users who just need a decently fast enough Zstandard implementation can use compress/zstd, while those want the best performance and/or advanced features of Zstandard can use @klauspost's package. Any stdlib packages (e.g., net/http that make use of compress/zstd should make it possible to swap over to a different zstd implementation).
Note that the same is already true of compress/gzip, which is generally good enough (although there is still room for performance optimizations), but power users can use github.com/klauspost/compress/gzip, which is much faster (as it makes use of assembly).

We could:

adopt @klauspost's pure-Go (i.e., no assembly) implementation with a cleaned-up API
extend @ianlancetaylor's work to support compressing
do a combination of both

Related issues:

#62492
#55107
#62446

Some comments on other popular compression formats and why Zstandard makes the cut, but not others:

Brotli (RFC 7932) has many of the benefits of Zstandard, but has several detriments:

It uses a 120KiB static dictionary, which must be linked into every Go binary that uses brotli. This doesn't suite Go well which always aims for static binaries and avoids shared libraries.
The overall design seems geared towards web data, while Zstandard is more general purpose.

XZ (or LZMA) has great compression ratios, but is really slow for compression and decompression. There is no specification.
bzip2 has fairly good compression ratios, but is slow for compression and decompression. There is no specification. While there exists a decompression implementation in the stdlib (i.e., compress/bzip2), I don't believe bzip2 should have made the cut for inclusion into the stdlib.
snappy is heavily geared towards high speed compression and decompression, but sacrifices compression ratio. Somewhat of a niche use case.
lz4 aims for similar goals to snappy.

IMO, the lack of a formal specification should bar inclusion in stdlib as it makes "correctness" somewhat undefined.

IMO, the lack of a formal specification should bar inclusion in stdlib as it makes "correctness" somewhat undefined.

Makes sense but I have to say that I'm not sure that anybody could write a correct zstd decompressor based solely on reading RFC 8878.

I'm not sure that anybody could write a correct zstd decompressor based solely on reading RFC 8878.

Heh... you're the expert there, having done it. I had written a Brotli decompressor solely from the draft RFC without looking at the implementation. This effort led to a number of updates to the draft so that it would be unambiguous when finalized.

To add to the list of good points zstd has an extremely really wide range of levels available, depending on the compressor options you can have lz4 or gzip like ratios which have very low resource usage.
Or by turning up a few knobs and or changing a few algorithms in the compressor you can get ratios slightly bellow xz while being transparently compatible with the decompressor (and by using less resources).

@dsnet Just to clarify some mistakes in your initial post.
zstd compression (nor deflate for that matter) uses any assembly or "unsafe" for compression. The assembly is used for decompression and can be fully disabled with a "noasm" tag. So obviously that is trivial to exclude.

the API is a somewhat complex

Could you elaborate what you find particularly complex?
I expect the stdlib would have the multithreading pulled out and only implement a fully synchronous mode, as well as possibly omitting dictionary features. Without that the API isn't really more complex than flate, gzip and similar. Without concurrency you can have the options on the Encoder (or Writer, whatever it is called), since you don't have any sharing to care about.

security as more important

I could interpret this as an underhanded way of implying my library isn't safe to use. Probably not the intention, but just to be clear there is continuous fuzzing running and there is third party security review being done.
Considering it doesn't use assembly, nor unsafe, I think it is fair to say that a battletested code is more secure than a freshly written code. If you want to write it, I will of course not take that fun from you - but to wrap it in an argument of "security" seems disingenuous or the very least not at all researched to me.
I do think that features like concurrent (de)compression and possibly dictionaries should be taken out from the stdlib. I think having the EncodeAll([]byte) []byte on an encoder is a good may to control allocations - and without concurrency it will be easier to control anyway.

@klauspost, thanks for your thoughts.

I expect the stdlib would have the multithreading pulled out and only implement a fully synchronous mode, as well as possibly omitting dictionary features

Correct. There should be no usages of goroutines in the stdlib implementation. The example in klauspost/compress#479 should be perfectly thread safe (as it is with the other stdlib packages).
One "dictionary" feature we probably want to keep is the ability to specify the maximum window size.

I could interpret this as an underhanded way of implying my library isn't safe to use. Probably not the intention,

Not at all the intention; I use your package quite extensively myself.
That said, the stdlib avoids the use of assembly except for a few select places. Assembly can be a challenge to review and maintain. Continuous fuzzing is great, but takes a long time before confidence in the implementation is obtained. Fuzzing doesn't help when trying to review if a change to the assembly implementation is correct.
I should also mention the asynchronous nature zstd.NewWriter has actually lended itself to race conditions that could then lead to security problems. To be clear, it's not a race in zstd, but the API design made it easy for users to hold it wrong. In general, library code should be synchronous (from the perspective of the caller) unless the naming or the functionality makes it obvious that it operates in a asynchronous manner.

Considering it doesn't use assembly, nor unsafe, I think it is fair to say that a battletested code is more secure than a freshly written code.

I'm confused. Just above, you mentioned that assembly is used in your package for decompression, but here you saying it doesn't use assembly? Did you mean if you built it under noasm?

If you want to write it, I will of course not take that fun from you - but to wrap it in an argument of "security" seems disingenuous or the very least not at all researched to me.

I'm not itching to write a zstd implementation from scratch. My end goal is to see support for it in stdlib. In my proposal, I deliberately didn't make a statement about how this would come to be fulfilled. Using the pure Go (i.e., no assembly) implementation of your package would be a reasonable possibility.

I think having the EncodeAll([]byte) []byte on an encoder is a good may to control allocations - and without concurrency it will be easier to control anyway.

I personally find it rather strange that the Encoder is either a streaming encoder or essentially a variable to hold encoder settings and shared resources to use with EncodeAll.
Given the acceptance of #54078, I would rather see something like:
func AppendCompress(dst, src []byte, level int) ([]byte, error)
func AppendDecompress(dst, src []byte, maxSize int) ([]byte, error)

One "dictionary" feature we probably want to keep is the ability to specify the maximum window size.

Yes. That is an essential security DOS mitigation. Though when I refer to dictionaries I only refer to predefined dictionaries that seeds the first blocks with backreferences and predefined entropy coders.

I'm confused. Just above, you mentioned that assembly is used in your package for decompression, but here you saying it doesn't use assembly? Did you mean if you built it under noasm?

I assumed you were only looking for compression given there already is an internal decompressor, which I unfortunately haven't had the time to test. But since everything is marked by tags, taking it out is extremely easy.

I should also mention the asynchronous nature zstd.NewWriter has actually lended itself to race conditions that could then lead to security problems.

I am not sure how you come to that. If you use it for streaming, you can only use it for one stream - that isn't different than a sync Writer. If you want writes to only happen when you are calling Write, yes, you need to disable concurrency - but that is an assumption you are making that doesn't hold. The documentation should be pretty clear on what you can expect. Either way, it is a non-issue since the concurrency should go.
Both the compressor and the decompressor have an "async" and "sync" code path. Ripping out the async one is a minor task. Of course some cleanup can probably be done, but that is cosmetics.

func AppendCompress(dst, src []byte, level int) ([]byte, error)
func AppendDecompress(dst, src []byte, maxSize int) ([]byte, error

So you want each of them to allocate, or try to hide the allocs with internal sync.Pools?
EncodeAll/DecodeAll are basically this (they also append), but it keeps the allocs within the Reader/Writer, and buffers, etc can be allocated with settings for the compression level needed.
Being able to use the same Reader/Writer for concurrent Encode/Decode operations would have to go, but it would at least give control over how each is reused. Sidenote, AppendCompress shouldn't need to return an error.

So you want each of them to allocate, or try to hide the allocs with internal sync.Pools?

Stack allocate if possible, reusing the destination buffer if possible (e.g., we don't need to maintain the LZ77 window for decompression since it can be obtained from dst itself), and otherwise sync.Pool.

To comment more on the EncodeAll hung off as methods where the pooling occurs within an Encoder: in a sufficiently large program, I'm observing multiple instances of global Encoder variables in various packages that each use more or less the same settings. It is somewhat silly that across packages, the resources is not shared since resource pooling occurs within an Encoder.

While zstd is a noble goal, I woul like to highlight that it still has zero support in browsers as of today. So for anyone looking to use a more efficient Content-Encoding for HTTP, brotli is still the go-to and will likely remain for the next few years at least:

https://caniuse.com/zstd
https://caniuse.com/brotli

@silverwind I think this is more relevant for https://github.com/golang/go/issues/62492. ZSTD has found many uses outside Content-Encoding already.

To offer a bit of commentary on how it would be used: at my company Zstandard is becoming our default general-purpose compression algorithm because it's strictly better than gzip on all the dimensions we care about. There are cases where we would decide to use gzip (for compatibility) or snappy (for very high throughput) but in the absence of specific requirements, Zstandard is the go-to.

@silverwind I think this is more relevant for #62492. ZSTD has found many uses outside Content-Encoding already.

Yes, just wanted to mention it. I'm in favor of adding any of the "better then gzip" mechanisms to the standard library for both http transfer, but also go:embed compression, which as of today does not even support gzip.

So you want each of them to allocate, or try to hide the allocs with internal sync.Pools?

Stack allocate if possible, reusing the destination buffer if possible (e.g., we don't need to maintain the LZ77 window for decompression since it can be obtained from dst itself), and otherwise sync.Pool.

Correct (and it already uses dst instead of a dedicated history). I can see the single-use decoder working stateless with hidden, internal buffers (for things like literal decompression, entropy decoders for example).
For encoding you have rather big hash tables to deal with, which will put quite some pressure on the stack. Level 1 has 256K, Level 2 has 1.2MB, Level 3 is 4MB. So these would also need some internal pooling, since having those on stack and forcing zeroing on each call, wouldn't be very optimal. Again entropy/literal coders would need to have internal pools, since they are rather expensive to set up.

Streams will of course be stateful, so ignoring those for now.
That said, I don't really see any problem in separating single-shot state from the Reader/Writer state and having an internal pool for those.

To comment more on the EncodeAll hung off as methods where the pooling occurs within an Encoder: in a sufficiently large program, I'm observing multiple instances of global Encoder variables in various packages that each use more or less the same settings. It is somewhat silly that across packages, the resources is not shared since resource pooling occurs within an Encoder.

Sure. I can see that. I try to avoid global package state. That is the main reason I haven't added it, since it would be rather annoying if one bad actor could cause problems for others.
A global "pool" is nice for resource control. It is pointless to run more than GOMAXPROCS EncodeAll/DecodeAll concurrently, since each will consume exactly one thread fully until it is done. That has a "play nice" feel for preemption, and limiting this below GOMAXPROCS will of course make sure that compression/decompression will never oversaturate a system.
Without concurrency there isn't the main "annoyance" with the current API - that you have to Close to release resources. It is not that different from many other things, but it is too easy to forget and cause leaks.
So yeah, I can see that as a way of simplifying things. Let me know if I understood your ideas clearly.

@dsnet I tested the internal/zstd - there are a bunch of problems in there and some differences to accepted blocks compared to zstd. This is mostly the snappy testset. input files, zstd compressed.
BenchmarkDecoder_DecoderSmall/kppkn.gtb.zst/unbuffered-32<PHONE_NUMBER> ns/op         328.38 MB/s         174 B/op          1 allocs/op
(Failed - zstd decompression error at 41415: offset past window)

BenchmarkDecoder_DecoderSmall/geo.protodata.zst/unbuffered-32               1107           1064937 ns/op         890.85 MB/s          59 B/op          1 allocs/op
BenchmarkDecoder_Internal/geo.protodata.zst-32     	     669	   1801050 ns/op	 526.75 MB/s	  554226 B/op	      31 allocs/op

BenchmarkDecoder_DecoderSmall/plrabn12.txt.zst/unbuffered-32                  81          14196832 ns/op         271.53 MB/s         217 B/op          1 allocs/op
(Failed - zstd decompression error at 106901: offset past window)

BenchmarkDecoder_DecoderSmall/lcet10.txt.zst/unbuffered-32<PHONE_NUMBER>8 ns/op         324.96 MB/s         173 B/op          1 allocs/op
(Failed - zstd decompression error at 89343: offset past window)

BenchmarkDecoder_DecoderSmall/asyoulik.txt.zst/unbuffered-32<PHONE_NUMBER> ns/op         274.69 MB/s         150 B/op          1 allocs/op
BenchmarkDecoder_Internal/asyoulik.txt.zst-32      	     188	   6349734 ns/op	 157.71 MB/s	  615777 B/op	      31 allocs/op

BenchmarkDecoder_DecoderSmall/alice29.txt.zst/unbuffered-32<PHONE_NUMBER> ns/op         257.90 MB/s          97 B/op          1 allocs/op
(Failed - zstd decompression error at 57404: offset past window)

BenchmarkDecoder_DecoderSmall/html_x_4.zst/unbuffered-32<PHONE_NUMBER> ns/op        1517.70 MB/s         109 B/op          1 allocs/op
(Failed - zstd decompression error at 14867: offset past window)

BenchmarkDecoder_DecoderSmall/paper-100k.pdf.zst/unbuffered-32              5551            211883 ns/op        3866.29 MB/s          48 B/op          1 allocs/op
BenchmarkDecoder_Internal/paper-100k.pdf.zst-32    	    2862	    436787 ns/op	1875.51 MB/s	  672010 B/op	      30 allocs/op

BenchmarkDecoder_DecoderSmall/fireworks.jpeg.zst/unbuffered-32             10000            101969 ns/op        9657.29 MB/s          48 B/op          1 allocs/op
BenchmarkDecoder_Internal/fireworks.jpeg.zst-32    	    9979	    149482 ns/op	6587.71 MB/s	  131808 B/op	       3 allocs/op

BenchmarkDecoder_DecoderSmall/urls.10K.zst/unbuffered-32                      94          12117244 ns/op         463.53 MB/s         405 B/op          1 allocs/op
(Failed - zstd decompression error at 74547: offset past window)

BenchmarkDecoder_DecoderSmall/html.zst/unbuffered-32                        1040           1153069 ns/op         710.45 MB/s          80 B/op          1 allocs/op
BenchmarkDecoder_Internal/html.zst-32              	     592	   2033895 ns/op	 402.77 MB/s	  557693 B/op	      33 allocs/op

BenchmarkDecoder_DecoderSmall/comp-data.bin.zst/unbuffered-32              14134             84502 ns/op         385.89 MB/s          48 B/op          1 allocs/op
BenchmarkDecoder_Internal/comp-data.bin.zst-32     	    7734	    158308 ns/op	 205.98 MB/s	   32780 B/op	      19 allocs/op

Assembly disabled. Multithreading disabled. Bytes() []byte interface disabled. I could not use the Reset(...) function, since it doesn't provide correct output when used (see below).
So "not great, not terrible" applies it seems. With assembly the difference grows significantly. Also, I did find some bugs while testing casually.

Bugs found
I ran my fuzz set through the decoder and found some differences/bugs. I don't really know where to report them (should I open an issue?)
Here are a bunch of samples: zstd-internal-problems.zip

test-diverge-N.zst is the input file.
test-diverge-N.txt contains the error message returned by the library.
test-diverge-N.got is the output from zstd commandline.

offset past window & invalid offset
There is a logic problem somewhere and this check fails on valid data (as can be seen above).
I suspect in some cases it ends up with wrong offsets (maybe repeats? - I can provide decoded sequence output if you want)
unexpected EOF
It seems there are a bunch of unexpected EOFs, that shouldn't really be unexpected. Not sure if it can happen on actual output of zstd.
invalid checksum: got A want B
Not sure if this is because of the offset problem - seems likely.
dictionaries are not supported
Dictionaries with ID 0 is the same as "no dictionary". The decoder should ignore this.
no sequences and no literals
This is allowed - even if it is a bit silly. Check against latest versions - this is something I had clarified while testing.
(*Reader).Reset is not working
I was unable to make (*Reader).Reset work as intended and it kept just spitting out errors. I suspect it doesn't reset all state.

All failures for https://github.com/klauspost/compress/blob/master/zstd/testdata/benchdecoder.zip are fixed when window size is configured according to RFC instead of zero.

When Single_Segment_Flag is set, Window_Descriptor is not present. In this case, Window_Size is Frame_Content_Size

I've created #63224 to address this.

@klauspost Thanks for the bug reports, @AlexanderYastrebov Thanks for the quick fix.
@klauspost Please do feel free to open bug reports against the code on this issue tracker.  Or I'll take a look either way.
Thanks.

So I've tested combined changes #63224 #63248 #63251 #63252 against both benchdecoder.zip and zstd-internal-problems.zip from https://github.com/golang/go/issues/62513#issuecomment-1733944811 and the only failures are due to unsupported dictionaries.

=== RUN   TestFileSamples/01a779ca.test-diverge-5.zst
    zstd_test.go:272: zstd decompression error at 4: dictionaries are not supported
--
=== RUN   TestFileSamples/1bd2d273.test-diverge-24.zst
    zstd_test.go:272: zstd decompression error at 113: dictionaries are not supported
--
=== RUN   TestFileSamples/47db12fc.test-diverge-16.zst
    zstd_test.go:272: zstd decompression error at 4: dictionaries are not supported
=== RUN   TestFileSamples/494ab243.test-diverge-13.zst
    zstd_test.go:272: zstd decompression error at 4: dictionaries are not supported
--
=== RUN   TestFileSamples/89e8d8c9.test-diverge-31.zst
    zstd_test.go:272: zstd decompression error at 139: dictionaries are not supported
=== RUN   TestFileSamples/89e8d8c9.test-diverge-32.zst
    zstd_test.go:272: zstd decompression error at 139: dictionaries are not supported
--
=== RUN   TestFileSamples/e3b0c442.test-diverge-23.zst
    zstd_test.go:272: zstd decompression error at 4: dictionaries are not supported
=== RUN   TestFileSamples/e3b0c442.test-diverge-33.zst
    zstd_test.go:272: zstd decompression error at 70: dictionaries are not supported
--
=== RUN   TestFileSamples/e3b0c442.test-diverge-8.zst
    zstd_test.go:272: zstd decompression error at 4: dictionaries are not supported
--
=== RUN   TestFileSamples/e4325775.test-diverge-3.zst
    zstd_test.go:272: zstd decompression error at 14: dictionaries are not supported

Dictionaries with ID 0 is the same as "no dictionary". The decoder should ignore this.

This is what it does but in the failing samples Dictionary_ID_Flag is not 0.

@AlexanderYastrebov Thanks very much.

@AlexanderYastrebov Great stuff!
Dictionary set, but ID==0 is the same as "no dictionary" when using the official zstd, so you can safely support that.

I support the inclusion of the ZSTD in the standard lib. While the specification  gets updates, the format itself is stable. I did a private experimental pure-go implementation  a few years ago and had only minor issues and reported them to the author of the format resulting in updates to the spec.
I definitely suggest to include multi-threading support at least for the compressor.  The speed increase for files larger than a megabyte is significant.
@dsnet You wrote that there is no specification for XZ. This is in so far correct that there is no single document describing the whole format. There is a specification of the XZ file format. I created a description of the LZMA2  format. There is also a specification of the LZMA file format including the LZMA encoding.

As far as API is concerned, I took a look at the zstd RFC as well as our compress/gzip and compress/flate APIs, and I think a reasonable first cut at a compress/zstd API would be:
package zstd

// These match flate and gzip constants, intentionally,
// but the package does not import flate.
const (
	NoCompression = 0
	BestSpeed = 1
	BestCompression = 9
	DefaultCompression = -1
)

var (
	ErrChecksum = errors.New("zstd: invalid checksum")
	ErrHeader = errors.New("zstd: invalid header")
	ErrCorrupt = errors.New("zstd: corrupt data")
)

type Reader struct {
	... unexported fields ...
}

func NewReader(r io.Reader) (*Reader, error)

func (z *Reader) Reset(r io.Reader) error

func (z *Reader) AddDict([]byte)
func (z *Reader) SetRawDict([]byte)

func (z *Reader) Read(p []byte) (int, error)
func (z *Reader) WriteTo(io.Writer) (int64, error)

func (z *Reader) Close() error

type Writer struct {
	... unexported fields ...
}

func NewWriter(w io.Writer) *Writer

func (z *Writer) Reset(w io.Writer)

func (z *Writer) SetLevel(int)
func (z *Writer) AddDict([]byte)
func (z *Writer) SetRawDict([]byte)

func (z *Writer) Write([]byte) (int, error)
func (z *Writer) ReadFrom(io.Reader) (int64, error)

func (z *Writer) Flush() error
func (z *Writer) Close() error

Feedback welcome.

For a little context, I'd love to be able to use this to parse the zstd:chunked archives the redhat folks have been using e.g. https://github.com/containers/image/pull/1084
This requires getting access to data inside of skippable frames.
I'm not exactly sure what we'd want that API to look like, but perhaps we could add something analogous to gzip.Reader.Multistream()? We'd also need to expose the frame header (similar to gzip.Reader.Header) and probably a way to ask for the skippable frames instead of skipping them.
Would it make sense to expose an iterator that returns each frame?
I can see this stuff being added on later, but wanted to bring this up in case it might influence the initial API in some way.

I wonder if that would be more appropriate for one of the zstd implementations outside of the standard library.

Are there more examples of what skippable frames get used for and how programs would like to process them? There are lots of possible APIs (channel, callback, Read stopping at a skippable frame, ...).

func (z *Reader) AddDict([]byte)
func (z *Reader) SetRawDict([]byte)

func (z *Writer) AddDict([]byte)
func (z *Writer) SetRawDict([]byte)

This API suggest you can call AddDict and SetRawDict during compression or decompression, I'm not sure if that allowed or intended. I guess .AddDict could be used to interleave new dictionaries for different part of the compressed streams but I doubt .SetRawDict could do anything.
Klauspost's zstd package uses functional option pattern: https://pkg.go.dev/github.com/klauspost/compress/zstd#EOption (not saying we should do that since it's not a pattern widely used in the std, but maybe something like NewReaderWithRawDict(r io.Reader, rawDict []byte) and .ResetWithRawDict(r io.Reader, rawDict []byte))
Also I think theses methods should be func([]byte) error to gently warn the user if they do mistakes like registering the same dictionary ID multiple times or pass malformed dictionaries (only for AddDict).

The dictionary methods could simply return an error, or perhaps panic, if called after compression/decompression has already started.  You could only call them immediately after NewReader or NewWriter, or a call to the Reset method.

Errors would catch the bug at runtime, so I think
func (z *Reader) AddDict([]byte) error
func (z *Reader) SetRawDict([]byte) error

func (z *Writer) AddDict([]byte) error
func (z *Writer) SetRawDict([]byte) error

makes even more sense than the error-less version.

I wonder if that would be more appropriate for one of the zstd implementations outside of the standard library.

Also totally reasonable! I mostly wanted to bring up the parallel between gzip's Multistream API and zstandard framing in case we wanted to match what compress/gzip is doing, but I'm also fine with keeping a very conservative API given that the average stdlib user just wants a reader/writer.

Are there more examples of what skippable frames get used for and how programs would like to process them?

Two other examples I've found:

Storing a "seek table" for seekable zstandard streams (looks very similar to what red hat is doing):
https://github.com/facebook/zstd/blob/dev/contrib/seekable_format/zstd_seekable_compression_format.md

A proposal to use them for extensions in zstd-compresed WARC files: https://iipc.github.io/warc-specifications/specifications/warc-zstd/

The second one doesn't feel quite "real", but the first seems like something that might be worth supporting eventually. I think as a first pass, we should probably just ignore skippable frames, but if we wanted to support seekable zstandard format, we could expose an io.ReaderAt version of this, similar to what I'm doing with (very experimental) gsip.

Overall the API looks fine, but I wonder if we should keep it minimal for the initial release:
package zstd

const (
	NoCompression = 0
	BestSpeed = 1
	BestCompression = 9
	DefaultCompression = -1
)

type Reader struct { /* ... unexported fields ... */ }
func NewReader(r io.Reader) (*Reader, error)
func (z *Reader) Reset(r io.Reader) error
func (z *Reader) Read(p []byte) (int, error)
func (z *Reader) Close() error

type Writer struct { /* ... unexported fields ... */ }
func NewWriter(w io.Writer) *Writer
func (z *Writer) Reset(w io.Writer)
func (z *Writer) SetLevel(int)
func (z *Writer) Write([]byte) (int, error)
func (z *Writer) Flush() error
func (z *Writer) Close() error

In particular, I removed:

Methods that implement io.ReaderFrom and io.WriterTo. These sound fine, but it seems that we should add them altogether to the other "compress/..." packages rather than just "zstd" (perhaps as a separate proposal?)
Any sentinel errors. It's unclear to me at the moment what the right API is for proper error classification. For some prior precedence with problems with "gzip"'s own sentinel errors, see #61797.
Any API for dictionaries. This is a nice feature of "zstd", but I suspect it can be added later. The latest discussion also shows that we're not sure what the right API is. Given that dictionaries have tables and structure to them, perhaps they should be a separate Dictionary type that you can set on a reader and writer. The construction of Dictionary (e.g., via BuildDictionary) would validate the dictionary (and return an error if relevant) and also do any processing work up front (and can therefore be cached).

IMO, the biggest early decision that needs to be settled is whether we go with:

Variadic options similar to @klauspost's own "zstd" package
The setter pattern being proposed by @rsc, or
An options struct

I have a slight preference for variadic options over the setter pattern, but don't feel strongly. I find options structs more cumbersome to work with especially when zero options is a common situation being dealt with.

@dsnet I think you nailed it 💯 - I have no relevant opinions on your questions.

@dsnet Maybe it was mentioned before, but why the level is not a special defined type?

I replicated @rsc's proposal on that regard and I believe he's just copying what the "flate" and "gzip" package does. IMO, the main utility of it being a named type depends on the options pattern we go with. If we go with variadic options, there is a benefit to having it be a named type that implements the options type, so that you can pass it in directly without going through some options constructor.

Re API, I think we want compress/zstd to be a mostly drop-in replacement for compress/flate except for the bytes it generates. So I think having dictionaries is important. A "raw dictionary" is meant to handle what https://datatracker.ietf.org/doc/html/rfc8878#name-dictionary-format refers to as a "raw content" dictionary and most closely matches compress/flate. But then we also need the zstd-format dictionaries.
As for variadic vs setters, setters seem to work fine in the other compress packages. The variadic options can easily get overwhelming, and I think they're better avoided here.
https://github.com/golang/go/issues/62513#issuecomment-2250794563's example for skippable frames makes clear that for starters we can have the decompressor just skip over them. We could also add an API that asks "am I now at a skippable frame" and could be called after every read, or we could have a Reader mode that stops at a skippable frame with ErrAtSkippable or something like that. (Plus we'd need a way to get its data.) But neither of those would actually help the Facebook seekable streams, so for now we should probably just leave them out for now, until more examples arise.
I don't mind leaving out ReadFrom/WriteTo. For some reason I thought compress/flate had them, but it doesn't.
So that would make the API:
package zstd

// These match flate and gzip constants, intentionally,
// but the package does not import flate.
const (
	NoCompression = 0
	BestSpeed = 1
	BestCompression = 9
	DefaultCompression = -1
)

// Unexported for now; speak up if you think they should be exported.
var (
	errChecksum = errors.New("zstd: invalid checksum")
	errHeader = errors.New("zstd: invalid header")
	errCorrupt = errors.New("zstd: corrupt data")
)

type Reader struct {
	... unexported fields ...
}

func NewReader(r io.Reader) (*Reader, error)

func (z *Reader) Reset(r io.Reader) error

func (z *Reader) AddDict([]byte) error
func (z *Reader) SetRawDict([]byte)

func (z *Reader) Read(p []byte) (int, error)

func (z *Reader) Close() error

type Writer struct {
	... unexported fields ...
}

func NewWriter(w io.Writer) *Writer

func (z *Writer) Reset(w io.Writer)

func (z *Writer) SetLevel(int)
func (z *Writer) AddDict([]byte) error
func (z *Writer) SetRawDict([]byte)

func (z *Writer) Write([]byte) (int, error)

func (z *Writer) Flush() error
func (z *Writer) Close() error

Feedback still welcome.

@rsc FWIW, I have never seen "raw dictionaries" used, nor had them requested. Only the "structured dictionaries" - and that at a low rate. So I am not sure if that needs to be part of the initial API.
func (z *Writer) AddDict([]byte) error
I assume this adds a structured dictionary. This will mainly be useful for small encodes - in the 0 -> 64KB range. As I briefly went into above, this data range will pose a significant overhead for processing the dictionary, if added to every encode.
If this dictionary would be persisted across Reset calls, it would be feasible, but a bit clumsy to use. (New -> AddDict -> use -> sync.Pool.Put -> sync.Pool.Get -> Reset).
For similar case, I have gone for the type Dict struct {... unexported....} approach, where the dictionary gets parsed once and can be used for encodes.
func ParseDict([]byte) (*Dict, error) and for the raw dictionaries it would be func MakeRawDict([]byte) (*Dict, error). The dictionary would then know which internal states should be modified on the Writer when used.
With these functions a func (z *Writer) SetDict(*Dict) would be a pointer copy, and the dictionary can keep internal data structures for pre-indexed dictionaries for different compression levels, with one instance needed for dictionary. The nil/zero value would mean "no dictionary". The Reader would have an equivalent interface.
FWIW, I used a similar approach with S2, which has worked out fine. The dictionary also offers one-shot []byte -> []byte encode/decode, mostly for convenience.

I think we need to keep RawDict because it makes it easier to transition from compress/flate (or zlib) to compress/zstd.
I don't mind ParseDict+AddDict(*Dict). As I understand it, you are allowed to have multiple dicts for a reader/writer to choose from, so AddDict not SetDict.

I have never seen "raw dictionaries" used, nor had them requested

I use them using your lib. I didn't requested it because they are already there and work so :tada:.
I use them as one of the primitive to do delta updates between two versions of the same file.

It sounds like the revised API would be as follows. Do I have this right?
package zstd

// These match flate and gzip constants, intentionally,
// but the package does not import flate.
const (
	NoCompression = 0
	BestSpeed = 1
	BestCompression = 9
	DefaultCompression = -1
)

// Unexported for now; speak up if you think they should be exported.
var (
	errChecksum = errors.New("zstd: invalid checksum")
	errHeader = errors.New("zstd: invalid header")
	errCorrupt = errors.New("zstd: corrupt data")
)

type Dict struct { ... }
func ParseDict(enc []byte) (*Dict, error)

type Reader struct {
	... unexported fields ...
}

func NewReader(r io.Reader) (*Reader, error)

func (z *Reader) Reset(r io.Reader) error

func (z *Reader) AddDict(*Dict)
func (z *Reader) SetRawDict([]byte)

func (z *Reader) Read(p []byte) (int, error)

func (z *Reader) Close() error

type Writer struct {
	... unexported fields ...
}

func NewWriter(w io.Writer) *Writer

func (z *Writer) Reset(w io.Writer)

func (z *Writer) SetLevel(int)
func (z *Writer) AddDict(*Dict)
func (z *Writer) SetRawDict([]byte)

func (z *Writer) Write([]byte) (int, error)

func (z *Writer) Flush() error
func (z *Writer) Close() error

Should Dict implement encoding.BinaryMarshaler and encoding.BinaryUnmarshaler?

@dsnet That seems very reasonable and convenient.

It sounds like the revised API would be as follows. Do I have this right?

AFAICT the stdlib compressors that accept options do it with separate constructors and not a settter like the SetLevel(int) suggested?
<EMAIL_ADDRESS>

The only complication that comes to mind is the dicts without tables. I don't like the if magic != 0xec30a437 { // assume raw }, since it potentially could collide.

Perhaps this is a reason not to have ParseRawDict and ParseDict both return a *Dict.
Another reason is that you can only have one raw dict but you can have many dicts.
So let's keep SetRawDict([]byte) and AddDict(*Dict) and then *Dict is only about the non-raw Dicts.

It's true that we changed to setters instead of NewWriterLevel, NewWriterLevelDict, but I think the setters are quite a bit clearer and avoid NewWriterLevelRawDict, NewWriterLevelRawDictWithOtherDicts, etc. The old approach did not scale.
I should also add an error result from SetLevel, although we should make it clear that if you are setting a constant level like BestCompression it will never fail.

Based on the discussion above, this proposal seems like a likely accept.
— rsc for the proposal review group
The proposal details are in https://github.com/golang/go/issues/62513#issuecomment-2289360138.

Another datapoint on a potential dictionary API, ideally it should be done in a way that the user can manage the lifecycle of their own dictionaries. klausspost/compress doesn't handle this well today: https://github.com/klauspost/compress/discussions/953
In multi-tenanted systems — or any dealing with multiple data categories — you want to be able to hold many dictionaries in memory and switch between them with minimal duplication. We rely on cgo for this today, leveraging the native library functions for ZSTD_createDDict_byReference and ZSTD_createCDict_byReference.
Should dictionary support be added to the stdlib, I'd love this use-case to be considered.

@coxley I don't understand what you mean by managing the life cycle. It seems like the ParseDict approach gives you a read-only Dict that can be shared by many encoders/decoders. What more is needed?
Is the point for ParseDict not to copy the []byte it has been passed?

No change in consensus, so accepted. 🎉
This issue now tracks the work of implementing the proposal.
— rsc for the proposal review group
The proposal details are in https://github.com/golang/go/issues/62513#issuecomment-2289360138.


Add multi-threading to the sync process

As a user, I want the sync process to be run in parallel, so the sync tasks finish faster.

https://docs.python.org/3/library/threading.html


Save user ui theme preference

Make GuiSettings save user's UI theme preference.
Update storage tests.
Fix unchecked type warning in ThemeWindow for ListView.

Also delete UI old classes that are no longer used.

Codecov Report

Merging #120 into master will increase coverage by 8.07%.
The diff coverage is 34.48%.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 1337157...5e0dd7c. Read the comment docs.


Bug in creating fields that cannot be displayed in short profile

Repro:

Click "Add a new profile field".
Check "Display in profile summary".
Switch to a field type that can't be displayed in the profile summary (e.g., Long text). The checkbox correctly disappears.
Fill out the info and click "Add".

Expected: Field should get created.
Actual:

@yogesh-sirsat @sahil839  FYI

@alya  can I pick this bug?


[Bug] 热力图不能同时根据x,y轴的值排序吗？heatmap can not be sorted based on the x-axis and y-axis

Version
5.2.2 及之前
Link to Minimal Reproduction
No response
Steps to Reproduce
这是我用来制图的参数(this is the configuration I used to produce the heatmap)
option={
"dataset": [
    {
        "id": "raw",
        "dimensions": [
            "Pclass",
            "Age",
            "TotalFare"
        ],
        "source": [
            [
                3,
                74,
                7.775
            ],
            [
                3,
                70.5,
                7.75
            ],
            [
                2,
                1,
                76.0042
            ],
            [
                2,
                0.83,
                47.75
            ],
            [
                1,
                55,
                30.5
            ],
            [
                1,
                54,
<PHONE_NUMBER>9999997
            ]

        ]
    },
    {
        "fromDatasetIndex": 0,
        "id": "sorting",
        "transform": {
            "type": "sort",
            "config": [
                {
                    "dimension": "Age",
                    "order": "asc"
                },
                {
                    "dimension": "Pclass",
                    "order": "asc"
                }
            ]
        }
    }
],
"xAxis": {
    "type": "category",
    "name": "Age",
},
"yAxis": {
    "type": "category",
    "name": "Pclass",
},
"series": [
    {
        "type": "heatmap",
        "datasetIndex": 1,
        "encode": {
            "x": "Age",
            "y": "Pclass",
        }
    }
],
"visualMap": [
    {
        "dimension": "TotalFare",
        "min": 0,
        "max": 1491.7042,
        "calculable": true,
        "orient": "vertical",
        "bottom": "50%",
        "right": "1%"
    }
]}

Current Behavior
目前的结果是这样(this is the current output)

Expected Behavior
这是我所期望的结果, 我希望其也能根据y轴来排序， 比如我希望（0.83，2）和（1，2）这两块能在y轴中间位置，例如下图（下图中x轴依然存在乱序的问题）
(this is the excepted output)

Environment
- OS:
- Browser:
- Framework:

Any additional comments?
No response

You are asking echarts to make a dataset for you which is first sorted by "Age" dimension, and if the "Age" dimension is equal only then sort it by "Pclass" which then goes like that:
| Position | Age | Pclass |
|-------------------------|
|       1      | 0.83 |    2     |  <---- Lowest Age
|       2      |    1   |    2     |  <---- Age higher than previous
|       3      |   54  |    1     |  <---- Age higher than previous
|       4      |   55  |    1     |  <---- Age higher than previous
|       5      | 70.5 |    3     |  <---- Age higher than previous
|       6      |  74   |    3     |  <---- Age higher than previous
So, essentially you are sorting only by Age. In order to achieve your expected result you want your transform to look like this:
"transform": {
            "type": "sort",
            "config": [
              {
                    "dimension": "Pclass",
                    "order": "asc"
                },
                {
                    "dimension": "Age",
                    "order": "desc"
                },
            ]
        }


Facing issue in building my project on build server.

I am facing issue in building my project on build server.
My .net core configurations are
   D:\>dotnet --info
   .NET Command Line Tools (1.0.0-preview2-1-003177)

   Product Information:
    Version:1.0.0-preview2-1-003177
    Commit SHA-1 hash:  a2df9c2576

   Runtime Environment:
    OS Name: Windows
    OS Version:  6.3.9600
    OS Platform: Windows
    RID: win81-x64

My Solution structure is
 abc.sln
      a.csproj
      b.xproj

where b has multitargeting class library which supports net461 and netstandard1.6
while a is in net461 and has reference of b.
I want to build this project on my build server that is jenkins (installed on windows) which has dotnet core installed.
Please suggest how to build this project via command line.
FYI : It builds on my Local via VS15.

I think the best approach would be to migrate your xproj to a csproj and get onto the 1.0 RTM tooling. After that, you should just be able to run dotnet build.
@livarcocc

Correct. Migrate your xproj to csproj using the dotnet migrate command.

Your scenario isn't well supported. We didn't have good support for xproj / csproj interop in VS 2015. And we don't support xproj in VS 2017 where the interop between various .NET project types is much better because everything is based on msbuild.
Closing because this experience isn't going to get better. Best plan is to move to VS 2017 to get the experience I believe you are looking for.


color is wrong

old version is right, latest is wrong, should set default color = curentColor

Some Icons are now black instead of white...

yup, worked for me. Had to set the fill: currentcolor prop on each solid-icon.

Without workaround of adding fill property

With workaround applied

I have a custom Icon component which takes IconType value and renders a Dynamic component which is an imported icon. Colors could be set by either passing the color prop or just by inheriting from currentColor set by a TW color class. Now both don't work anymore. This is extremely annoying as it forces me to micro-manage colors on a per-icon basis.
My code (shortened):
type IconProps = {
  type: IconTypes
  size?: IconSize
  color?: IconColor
  class?: string
}

export const IconType = {
  home: ImHome,
  dashboard: BiSolidDashboard,
  domain: FiGlobe,
} as const

export type IconType = ValuesOf<typeof IconType>

export const IconColor = {
  white: '#ffffff',
  black: '#000000',
} as const

export type IconColor = ValuesOf<typeof IconColor>

export const IconSize = {
  sm: 16,
  md: 20, // default
  lg: 24,
} as const

export type IconSize = ValuesOf<typeof IconSize>

export const Icon: Component<IconProps> = (props) => {
  return (
    <Dynamic
      component={props.type}
      size={props.size ?? IconSize.md}
      color={props.color}
      class={props.class}
    />
  )
}

With newest changes I'd have to add a style property of fill which maps either color prop or currentColor and also add conditionals on a per-icon basis depending on if they show as intended or not (after visual confirmation). Any update on this in sight? I wanted to sit it out first, thinking it's a bug. But seeing that that it's been like this for a few weeks already, I'd like to ask on current/future status.

Hello, I've been looking at this lately but I couldn't reproduce it, can you send a repro of this on github or codesandbox please?

@x64Bits
It seems to be a specific issue with certain icon sets.
https://stackblitz.com/edit/solid-ssr-vite-tj6b2f?file=src%2Froutes%2Findex.tsx
In my application I am using a wide variety of different source sets and the problem seems to be pretty wide-spread.

I personally have the problem with versions > 1.0.4.
which is why I think there are so many downloads of this version.
https://www.npmjs.com/package/solid-icons?activeTab=versions

@Wantod
Yes, we also rolled back to 1.0.4

Hello everyone, one PR #51 was merge that address this issue, can anyone take a look it's the 1.0.12 versions display the icons correctly?

@x64Bits
I've updated the example stackblitz from above with the new version and it does indeed seem to be working fine again.

Hi!, thank you very much for all the help, since version 1.1.0 even more problems with the color of the icons have been solved, I will close this issue, greetings!