Docs describing onboarding do not have valid links to gitbooks pages

Impacted Docs
Docs describing onboarding do not have valid links to gitbooks pages

https://github.com/communitybridge/easycla/tree/master/getting-started#contributor
https://github.com/communitybridge/easycla/tree/master/getting-started#corporate-cla-manager
https://github.com/communitybridge/easycla/tree/master/getting-started#corporate-cla-signatory

The broken links have been fixed. Will be available in PROD and Staging soon.


Fix SVD image input

Found a bug of SVD after the SV3D update:
If the input is a RGB image for SVD, The variable 'input_image' will not be assigned and raises:
UnboundLocalError: local variable 'input_image' referenced before assignment

Using copy() rather than direct assignment, or the following error will raise since the file is closed by the 'with' context manager:
TypeError: int() argument must be a string, a bytes-like object or a real number, not 'PngImageFile'

Another suggestion is to use torchivision.io.write_video() rather than imageio.mimwrite() to save the video. The current output cannot take fps into account and is not compatible with many video players. For example:
from torchvision.io import write_video
...
write_video(video_path, vid, fps=fps_id + 1)


Guide on how to add Citizens?

Is there supposed to be a panel or something because Im not seeing it.

Adding citizens is only a developer thing (requires coding), its not a thing users can do.

Adding citizens is only a developer thing (requires coding), its not a thing users can do.

Ahh sorry. I misunderstood. Thanks!


Fix ldap.yml path:

The current path is incorrect, it should be  path: /instance_groups/name=web/jobs/name=web/properties/ldap_auth? . The current path must be the old path from previous versions. I am currently using v5 and above.

thanks!


Added 16 files

Got some code for us? Awesome 🎊!
Please include a description of your change & check your PR against this list, thanks!

[ ] There's a clear use-case for this code change, explained below
[ ] Commit message has a short title & references relevant issues
[ ] The build will pass (run yarn test:all and yarn lint)

We appreciate your contribution!
Also, if you'd be interested in writing code like this for us more regularly, we're hiring:
https://careers.ghost.org

Looks good.

Sorry @ErisDS was testing something ... ignore that PR!


Log System [ ElasticSearch + LogStash + Kibana ]

https://www.digitalocean.com/community/tutorials/how-to-use-logstash-and-kibana-to-centralize-and-visualize-logs-on-ubuntu-14-04

https://www.ddreier.com/setting-up-elasticsearch-kibana-and-logstash/

http://thepracticalsysadmin.com/introduction-to-logstashelasticsearchkibana/

It's done by system administrators.


failed to connect to IDE run session notification endpoint, retrying...   {"error": "tls: failed to verify certificate: x509: certificate has expired or is not yet valid: current time XYZ is after XYZ "}

Is there an existing issue for this?

[x] I have searched the existing issues

Describe the bug
Starting an aspire project (that worked just two days ago) now fails with:
failed to connect to IDE run session notification endpoint, retrying...   {"error": "tls: failed to verify certificate: x509: certificate has expired or is not yet valid: current time 2024-10-09T14:21:17+02:00 is after 2024-10-08T09:30:21Z"}

No indication what cert might be expire and/or untrusted.
PS C:\Users\petrhouska> dotnet dev-certs https
A valid HTTPS certificate is already present.
PS C:\Users\petrhouska> dotnet dev-certs https --trust
Trusting the HTTPS development certificate was requested. A confirmation prompt will be displayed if the certificate was not previously trusted. Click yes on the prompt to trust the certificate.
Successfully trusted the existing HTTPS certificate.

Doesn't solve the problem.
Works for my co-workers so I assume it must be some local problem, but since I don't know what cert the culprit is it's hard to solve it.
Expected Behavior
Aspire project starts.
Steps To Reproduce
No response
Exceptions (if any)
No response
.NET Version info
dotnet --info
.NET SDK:
Version:           9.0.100-rc.1.24452.12
Commit:            81a714c6d3
Workload version:  9.0.100-manifests.174af1cf
MSBuild version:   17.12.0-preview-24422-09+d17ec720d
Runtime Environment:
OS Name:     Windows
OS Version:  10.0.22631
OS Platform: Windows
RID:         win-x64
Base Path:   C:\Program Files\dotnet\sdk\9.0.100-rc.1.24452.12\
.NET workloads installed:
Configured to use loose manifests when installing new manifests.
[aspire]
Installation Source: VS 17.11.35303.130
Manifest Version:    8.2.0/8.0.100
Manifest Path:       C:\Program Files\dotnet\sdk-manifests\8.0.100\microsoft.net.sdk.aspire\8.2.0\WorkloadManifest.json
Install Type:        FileBased
Host:
Version:      9.0.0-rc.1.24431.7
Architecture: x64
Commit:       static
.NET SDKs installed:
6.0.423 [C:\Program Files\dotnet\sdk]
7.0.410 [C:\Program Files\dotnet\sdk]
8.0.400 [C:\Program Files\dotnet\sdk]
8.0.403 [C:\Program Files\dotnet\sdk]
9.0.100-rc.1.24452.12 [C:\Program Files\dotnet\sdk]
.NET runtimes installed:
Microsoft.AspNetCore.App 6.0.33 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 7.0.20 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 8.0.8 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 8.0.10 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 9.0.0-rc.1.24452.1 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.NETCore.App 6.0.33 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 7.0.20 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 8.0.8 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 8.0.10 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 9.0.0-rc.1.24431.7 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.WindowsDesktop.App 6.0.33 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
Microsoft.WindowsDesktop.App 7.0.20 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
Microsoft.WindowsDesktop.App 8.0.8 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
Microsoft.WindowsDesktop.App 8.0.10 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
Microsoft.WindowsDesktop.App 9.0.0-rc.1.24452.1 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
Other architectures found:
x86   [C:\Program Files (x86)\dotnet]
registered at [HKLM\SOFTWARE\dotnet\Setup\InstalledVersions\x86\InstallLocation]
Environment variables:
Not set
global.json file:
Not found
Learn more:
https://aka.ms/dotnet/info
Download .NET:
https://aka.ms/dotnet/download
Anything else?
Aspire version: 8.2.0+75fdcff28495bdd643f6323133a7d411df71ab70

Updating VS to:
PS C:\Users\petrhouska> dotnet --info
.NET SDK:
 Version:           9.0.100-rc.1.24452.12
 Commit:            81a714c6d3
 Workload version:  9.0.100-manifests.174af1cf
 MSBuild version:   17.12.0-preview-24422-09+d17ec720d

Runtime Environment:
 OS Name:     Windows
 OS Version:  10.0.22631
 OS Platform: Windows
 RID:         win-x64
 Base Path:   C:\Program Files\dotnet\sdk\9.0.100-rc.1.24452.12\

.NET workloads installed:
Configured to use loose manifests when installing new manifests.
 [aspire]
   Installation Source: VS 17.11.35327.3
   Manifest Version:    8.2.0/8.0.100
   Manifest Path:       C:\Program Files\dotnet\sdk-manifests\8.0.100\microsoft.net.sdk.aspire\8.2.0\WorkloadManifest.json
   Install Type:        FileBased

Host:
  Version:      9.0.0-rc.1.24431.7
  Architecture: x64
  Commit:       static

.NET SDKs installed:
  6.0.423 [C:\Program Files\dotnet\sdk]
  7.0.410 [C:\Program Files\dotnet\sdk]
  8.0.403 [C:\Program Files\dotnet\sdk]
  9.0.100-rc.1.24452.12 [C:\Program Files\dotnet\sdk]

.NET runtimes installed:
  Microsoft.AspNetCore.App 6.0.35 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 7.0.20 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 8.0.10 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 9.0.0-rc.1.24452.1 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.NETCore.App 6.0.35 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 7.0.20 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 8.0.10 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 9.0.0-rc.1.24431.7 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.WindowsDesktop.App 6.0.35 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
  Microsoft.WindowsDesktop.App 7.0.20 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
  Microsoft.WindowsDesktop.App 8.0.10 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
  Microsoft.WindowsDesktop.App 9.0.0-rc.1.24452.1 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]

Other architectures found:
  x86   [C:\Program Files (x86)\dotnet]
    registered at [HKLM\SOFTWARE\dotnet\Setup\InstalledVersions\x86\InstallLocation]

Environment variables:
  Not set

global.json file:
  Not found

Learn more:
  https://aka.ms/dotnet/info

Download .NET:
  https://aka.ms/dotnet/download

fixed the problem
But I'd argue better error message would be beneficial. The VS version wasn't even old enough to be disallowed by MSFT policies.

Updating VS worked for me 2 weeks ago, but today it's back :-(
fail: Aspire.Hosting.Dcp.dcpctrl.IdeExecutableRunner[0]
failed to connect to IDE run session notification endpoint, retrying...   {"error": "tls: failed to verify certificate: x509: certificate has expired or is not yet valid: current time 2024-10-25T15:39:31+02:00 is after 2024-10-25T13:18:25Z"}
If you like frustration, try Aspire... my experience so far, sadly.

@kwaazaar if you restart VS, does the problem disappear?

We neeed to extend whatever timeout we have. People leave open their tools for days (especially with devbox).

@kwaazaar if you restart VS, does the problem disappear?

Yes it did @karolz-ms .Thanks!
My VS had been open for weeks, as I'm working on a big new feature. I don't care so much if I have to restart, but the error doesn't tell me to, so it didnt cross my mind.

@kwaazaar thank you.
@tlmii we should periodically refresh the cert that is given to DCP for securing the connection to the IDE. https://github.com/dotnet/aspire/blob/main/docs/specs/IDE-execution.md#enabling-ide-execution  As David said above, the IDE may be left open for a while (days or even weeks).


Made the new file name above the member list hide when no script or doc is opened

The behaviour before the file name introduced in #19388 was that the sort button hid when no script or doc was opened. Just adapting the new item to follow that.

Please, don't merge this, I already fixes on this PR #19334 and this will conflict with my changes again.

@guilhermefelipecgs Thanks for the heads-up, closing it.


feat: add zenbones theme

Pull Request Template
Discord username (optional)
ubmit#1502
Name of theme

Zenbones Light
Zenbones Dark

How Has This Been Tested?

Thanks @ubmit Sorry that I'm getting to this late; will update the discord with the contributor role


[go-machine-service] Adding driver for Vultr.com

Just noticed that there is now a generic UI for docker-machine. Awesome :tada:
I'd like to see support for the VULTR driver in rancher/go-machine-service and would be more than happy to contribute the necessary code.
https://github.com/janeczku/docker-machine-vultr
The driver provisions instances with RancherOS by default which makes it a great match for the Rancher platform.
I guess the first step would be to push the driver to rancher/machine?
Please advise!
/cc @ibuildthecloud

@janeczku we are in the process of switching to Machine 0.5.0 so I see no reason why we couldn't include this driver.  After we switch to 0.5.0 we will start working on making it possible to register driver plugins through the UI.
@wlan0

@ibuildthecloud

After we switch to 0.5.0 we will start working on making it possible to register driver plugins through the UI.

Would this comprise uploading the driver binary through the UI?

That's the idea, still working out how but on the admin screen you can upload and select which drivers are valid for your deployment

With the ability for users add in your own machine drivers and customize your own UI for machine drivers, I will close this issue.


The ukvm GDB stub is not sending the appropriate terminate message when exiting

When ukvm runs in --gdb mode and exits, the client complains with this:
A problem internal to GDB has been detected,
further debugging may prove unreliable.
Quit this debugging session? (y or n) Quit

The fix is to catch SIGTERM and other signals and send the appropriate message to the gdb client.

Fixed in #198.


export PATH from shell instead of makefile

the makefile PATH use makefile environment variable now,
if the PATH have some special character , the shell execution will fail.
/bin/bash: -c: line 0: syntax error near unexpected token ('`
In general, we should use the shell's environment variables，or use quotation marks 。

Hi @maplebeats. Thanks for your PR.
I'm waiting for a istio member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.
Once the patch is verified, the new status will be reflected by the ok-to-test label.
I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

/ok-to-test

#19774 fixed, close


Feature/jpark50/#422 implement modis snow cover fraction assimilation

@jpark50 if you can review your PR and make sure you are on right branch, you are currently merging to main. And once you move to develop, is your branch up to date with develop that would be first step. I will add label do not approve until this is cleaned up.

Sorry, @biljanaorescanin, this was not meant to be reviewed at this point.  There isn't even a commit on the new branch yet.
The PR is meant for Jongmin, Lauren and myself to collaborate on the implementation.
I converted the PR to "draft" status.
I tried changing the "base" to "develop" (via the "Edit" button in the top right) but this didn't work for some reason.
@jpark50, please review the guidance on how to create a pull request on the Wiki.  No need to do a Github "Issue" here because there's no question about the need for this work.  But the other Steps in the Wiki still apply.
cc: @weiyuan-jiang @lcandre2


Prominently display that Open SaaS is built on Wasp

We should communicate that Open SaaS uses Wasp earlier on.
Possible places could be the hero section...

...or under features on the homepage.

@vincanger Shall I work on this?
I will go through the contributing guidelines and come up with a way to enhance this soon!


MSI MEG Creation X399 support?

Unlikely to happen as the motherboard functions in a completely different way and I don't have one to test MSIRGB with.


Can someone take on audfprint-gui for audfprint or create a new gui for it?

This is a wonderful code that has been created. In combination with a program called audfprint-gui, myself and others have been able to ID tracks that were all considered previously impossible. I can't put into words how much impact this has  and it's potential to help even more.
The problem is the developer for audfprint-gui has abandoned the program; it has not had an update since Jul 27, 2022. Unfortunately, the Windows version has a problem where it's not functioning on a newly installed machine and essentially DOA. There are also other small bugs and important things not currently implemented like the 'allow multiple matches from database' option.
I openly admit I don't understand programming/coding; I desperately wish I did and have tried, but my brain can't wrap around the logic, it's like oil and water. I rely on the GUI of it in order to be able to use the program.
Is it possible someone could please take up development of the audfprint-gui or create a new GUI themselves? The current one is fully built and essentially working, it just needs minor fixes and such.

I can help you, but I don't have so much time. Do you have a Discord?

Not for this program specifically. You can PM me on discord though if you want, my name is GodzFire, ID<PHONE_NUMBER>82871552.


Block the login dialog

The logging dialog is blocked, so users can't skip it without adding a name or logging in with a third-party auth method. The problem is that in some browsers, when clicking outside the dialog, it closes. Another issue, we don't check if the user introduced an empty name.

Anyway, the logging dialog probably appears "too late" currently. We should not even be able to see the workspace if we are not logged in, right?

Yes. I think would be best to change to a login page before jupyterLab loads


[Feature] Delete review saying/session

Add a button allowing to delete a saying or a full session on the review tab, in case of someone who said something inappropriate.

I'm not sure about this unless you are thinking on sensible data, docs shouldn't be deleted.

Yes i was thinking about that.
At some point, some users will mistakenly type their card number on the chat, and you will need to delete that.

Makes sense to me. Though review is directly tied to analytics so deleting one would impact the other.

Agree with you on that, so maybe a way to edit it could help, by removing only the sensitive part.
imo, it could be more a user choice to either remove a saying/session and get the stats affected, or edit the sayings to keep the stats.

I think we should just add delete buttons to the session and review page. We'll prompt with a "Deleting this _____ will impact analytics and can not be reversed." type message.

Closed by 6fb1d140baee3dd430adbdcac56ec5cb0d397391


Adds methods to find a session by it's session_id

This PR addresses issue #169
Simply adds a method of find_by_session_id as well as a get_archive_by_session_id respectively.
I followed the OpenTok REST API reference to implement this feature.

Ok - this functionality is actually supported by this library - although not very well documented. Closing this PR, will re-open documentation suggestions. Sorry for the spam.
Thanks!


Black Lives Matter

Your FAQ directed website issues to GitHub, and I feel its appropriate for your organization to publicly respond to your public advocacy.
Black Lives Matter is a political organization attempting to advance a communist / marxist / socialist agenda.  I presume Microsoft is already aware of truths regarding behind the organizations which they support and have either chosen to accept them or be willingly ignorant toward them.  All Lives Matters goes without saying as much as Black Lives Matter goes without saying, as well as various other racial or ethnic groups, I merely question the politicization and weaponizing of "PowerShell Gallery" against the founding principles of Microsoft's own founding country.
Is Microsoft's public stance now to openly oppose our founding principles and to advocate for political parties -- or is it to develop software and tools?
Please provide clarification on the political advocacy on "PowerShell Gallery"

I'm closing this issue as off-topic to PowerShellGet.

This is not off-topic.  Per your contact page, website issues are to be posted to your GitHub:  https://www.powershellgallery.com/policies/Contact


Auto imported components are highlighted in red instead of green.

Hi! I have disabled vetur to not have any conflicts with volar and the green highlighting on the auto imported components is gone.
The code keeps working correctly, but the auto imported components appear in red. I am using the unplugin-vue-components from @antfu in a vite setup. If I activate vetur back, the highlighting becomes green again, but I do not want to keep it active in order to avoid conflicts.

Can you provide me a minimal reproduction?

@xiaoxiangmoe

I created a component in the components folder.
I have added the component to the App.vue root page of the app.
Due to having auto importing components plugin for vite, I do not have to explicitly import the component in order to work.
The component shows up in the browser and everything works fine, except the component in the template is highlighted in red. When I write the component, volar gives me the option to import that component. The component stays red unles I explicitly import that component in a page with: import Component from './components/Component'

Hey there! I'm getting the same issue. I'm using latest Vite/Vue/Typescript, and importing components using the following code:
const componentsPlugin: Plugin = {
  install: (app) => {
    const components = import.meta.globEager('./*.vue');

    for (const key in components) {
      const component = components[key].default
      app.component(component.name, component);
    }
  },
};

See https://github.com/antfu/unplugin-vue-components#typescript
Duplicate of #522

@johnsoncodehk The solution from issue #522  doesnt work for me. "editor.semanticHighlighting.enabled": true doesn't change aything for me. Also, the typescript section of the unplugin vue components doesnt apply to me because I am not using typescript.

@StefanGhiban
Try setting your config to generate the d.ts file: https://github.com/antfu/unplugin-vue-components#typescript. Even though you're using js you need to do this step.

This doesn't work for me
tsconfig.json
{
  "include": [
    "./components.d.ts"
  ],
}


Fix typo in thread_ts alias

Description
The Action input is defined as thread_ts (inline with the input expected for the Slack API), however the code is pulling inputs from tread_ts.

@ryanmab thanks!

@ryanmab thanks! can you run pnpm bundle to regen dist?

@vaporif No problem at all. dist should be regenerated now!


Missing Ingest Action in Table

I believe that the summary of supported actions at the top of this page is missing the INGEST Action

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 72676865-17c0-16aa-6ff7-3ef2e3b961dc
Version Independent ID: 80c163bb-3e28-9b67-855c-4ed7eaa272dc
Content: RESTful web API overview - Azure Data Explorer
Content Source: data-explorer/kusto/api/rest/index.md
Service: data-explorer
GitHub Login: @shsagir
Microsoft Alias: shsagir

@pnarsi  Thanks for the feedback! I have assigned the issue to the content author to evaluate and update as appropriate.

@pnarsi Thanks for contacting us with feedback about our product documentation. Depending on the complexity, it could take from a few business days to a few weeks to look further into your question, issue, or suggestion. We'll contact you if we need further information, and you'll receive an email message each time we "@mention" you in a comment.
Thanks again for submitting your feedback. Your suggestions help improve our documentation.

This missing information has been addressed in the above tagged PR. It will go live over the next 1-2 days. @pnarsi thanks for your contribution!
#please-close


useBrowserHistory

Hey! I just created a new hook which makes use of the browser history state, hope you find this one interesting.
It's main goal is for inside projects that have modals / popups / dialogs, they can easily be hooked into the browser's native history, so they can be closed (and optionally reopened) when the user presses back (or forward) in their browser.
It's a common UX bug I see in many, many websites so I hope this hook can help developers easily fix this issue.
It's especially bad on Android phones when modals don't use the browser history to create a new entry for itself, I find myself losing the page I was on all the time!
Anyway, hope you like this one! Thanks.
https://github.com/zcallan/use-browser-history
import React, { useState } from 'react'

import useBrowserHistory from 'use-browser-history'

function Modal({ isOpen, onClose, onOpen }) {
  const [handleBack] = useBrowserHistory( 'my-modal', isOpen, onClose, onOpen );

  if ( !isOpen )
    return null;

  return (

Hello!

        <button onClick={handleBack}>Close modal</button>

  )
}

function App() {
  const [isOpen, setIsOpen] = useState( false );

  return (

      <button onClick={() => setIsOpen( true )}>Open modal</button>

      <Modal
        isOpen={isOpen}
        onClose={() => setIsOpen( false )}
        onOpen={() => setIsOpen( true )}
      />

  );
}

Nice! For these cases I tend to push a new url to history and have url params as props that control the modal (using next.js). That way you kind of get this functionality for free. That said, not everyone may be using a framework that makes that easy to do and a simple hook that adds that is pretty nifty.


Add a function interface for saving a new dataset to replace  metadata.json

Add a function interface for saving a new dataset to replace  metadata.json
Description
New interface for users to both save the graph and generate metadata.json
Motivation and Context
Help users to generate correct metadata.json
How Has This Been Tested?
Use cora dataset for test.
from gli.io import save_graph, Attributes
# save_data("cora", **data)
save_graph(
    "cora",
    edge,
    node_list,
    edge_list=edge_list,
    node_attrs=[Attributes("NodeFeature",
                           description="Node features of Cora dataset, 1/0-valued vectors.",
                           data=node_feats,
                           data_type="int",
                           ),
                Attributes("NodeLabel",
                           description="Node labels of Cora dataset, int ranged from 1 to 7.",
                           data=node_class,
                           data_type="int",
                           ),
                ],
    edge_attrs=[],
    graph_attrs=[],
    citation="@inproceedings{yang2016revisiting,\ntitle={Revisiting semi-supervised learning with graph embeddings},\nauthor={Yang, Zhilin and Cohen, William and Salakhudinov, Ruslan},\nbooktitle={International conference on machine learning},\npages={40--48},\nyear={2016},\norganization={PMLR}\n}",
    is_heterogeneous = False
)

The corresponding test result is
{
    "description": "cora dataset",
    "Node": {
        "NodeFeature": {
            "description": "Node features of Cora dataset, 1/0-valued vectors.",
            "type": "int",
            "format": "SparseTensor",
            "file": {
                "file": "cora__graph__Node_NodeFeature__7032c9c380d1889061dcbbcd76b8c427.sparse.npz"
            },
            "key": "Node_NodeFeature"
        },
        "NodeLabel": {
            "description": "Node labels of Cora dataset, int ranged from 1 to 7.",
            "type": "int",
            "format": "Tensor",
            "file": {
                "file": "cora__graph__499e577d2533c2e69dd170791ffb6522.npz",
                "key": "Node_NodeLabel"
            },
            "key": "Node_NodeLabel"
        }
    },
    "Edge": {
        "_Edge": {
            "file": "cora__graph__499e577d2533c2e69dd170791ffb6522.npz",
            "key": "edge"
        }
    },
    "Graph": {
        "_NodeList": {
            "file": "cora__graph__499e577d2533c2e69dd170791ffb6522.npz",
            "key": "node_list"
        },
        "_EdgeList": {
            "file": "cora__graph__499e577d2533c2e69dd170791ffb6522.npz",
            "key": "edge_list"
        }
    },
    "citation": "@inproceedings{yang2016revisiting,\ntitle={Revisiting semi-supervised learning with graph embeddings},\nauthor={Yang, Zhilin and Cohen, William and Salakhudinov, Ruslan},\nbooktitle={International conference on machine learning},\npages={40--48},\nyear={2016},\norganization={PMLR}\n}",
    "is_heterogeneous": false
}

Closed in favor of #431


Adjust flight id for segmentation

No segments were being read from ORCESTRA data because the flight ID from the l3_ds was 20240926 (for example), and the flight ID in the segmentation file was "HALO-20240926a". This is not a super general fix for all types of segmentation files, though, so maybe there should be an entry in the config file?

I think this has been fixed in the recent version of the pipeline. have you reprocessed to level3 (deleting the interim_l3 files first)?

I did and I still get the same discrepancy.

I did and I still get the same discrepancy.

The folders in Level 0 should have the names of the flights. I adjusted this for IPFS, it might be that it's not working for you locally. I think we should assume that people name their folders correctly, as we cannot foresee any way that could be done different from the actual naming scheme

aha okay. So then I will change the names of the folders, makes sense. Thanks!


Update {admiral} presentation list

I will be updating the Presentation Archive in early December 2024.
@pharmaverse/admiral @pharmaverse/admiral_comm please could you edit this comment and add any presentations missed:
Past
useR 2024

@manciniedoardo @StefanThoma {admiral} - the {dplyr} of the pharmaceutical industry (video may be published here - @StefanThoma can we somehow provide slides?

Upcoming
Phuse EU Connect 2024

@manciniedoardo - Two hats, one noggin: Perspectives on working as a developer and as a user of the admiral R package for creating ADaMs  (slides, paper)
Yirong Cao. Yi Yan: Insights of Using {admiralonco} at BMS: Facilitating Oncology Efficacy ADaM Datasets (slides, paper)
@federicobaratin  - admiralvaccine Project JouRney (slides, paper)

@zdz2101 just did a presentation at CDISC - Zelos can you link here?
@Fanny-Gautier, you, me, @sadchla-codes did a workshop at R/Pharma.  Presentation here: https://pharmaverse.github.io/rpharma-2024-ADaM-workshop/#/title-slide.  Repo here: https://github.com/pharmaverse/rpharma-2024-ADaM-workshop

Hey @manciniedoardo I just published our slides.
https://stefanthoma.github.io/useR_2024_admiral
Let me contact the UseR2024 people about the video, I was under the impression that they do publish it, but I cannot find it still.

@zdz2101 just did a presentation at CDISC - Zelos can you link here?

@zdz2101 reminder to please provide link

Now merged - as @ddsjoberg noted, we'll do a release of admiraldiscovery in Jan.


New Version: RenewedVision.ProPresenter.7.8.2_117965313

[x] Have you signed the Contributor License Agreement?
[x] Have you checked that there aren't other open pull requests for the same manifest update/change?

[x] Does your manifest conform to the 1.0 schema?

Note: <path> is the name of the directory containing the manifest you're submitting.

Microsoft Reviewers: Open in CodeFlow

You would also need to update the version folder as well:
RenewedVision/ProPresenter/7.8.2+117965313/ -> RenewedVision/ProPresenter/7.8.2_117965313/

Just wondering why an underscore is recommended here. Semantic versioning expects build metadata after a plus sign. Is there any documentation you can point me to? I searched but couldn't find anything relevant.


emacs: add support for js2-mode

So currently https://github.com/jlongster/prettier/blob/master/editors/emacs/prettier-js.el#L67 makes it so that defun-before-save only works in js-mode, leaving anyone who uses other modes for editing JS (like js2-mode) out in the cold (or, you know, define their own hook function 😄 ). It would be nice to support js2-mode to avoid all of that. The change would be trivial:
(defun prettier-before-save ()
  "Add this to .emacs to run refmt on the current buffer when saving:
 (add-hook 'before-save-hook 'prettier-before-save)."
  (interactive)
  (when (member major-mode '(js-mode js2-mode)) (prettier)))

The list of supported modes/the mode which should be used could even be extracted into a customizable variable.

This can probably be closed since #276 has been merged.


TypeScript & ES2015 Examples

First, this library is awesome!
Second, I want to show off my TypeScript and ES2015 examples using react-async-component. I also have react-hot-loader working with it. It took me a look time to get everything figured out so I hope it helps others.
Features are TypeScript or ES2015, Universal (SSR), React Hot Loader 3, React Router 4, Redux, Redux Saga, Redux Form, Async Component Code Splitting, Hapi, Webpack 3.

hapi-react-hot-loader-example
typescript-hapi-react-hot-loader-example

Thanks dude!
I have published v1.0.0 on npm, which now bundles the typescript typings.

@ctrlplusb I just saw this new release today, but the typings file doesn't seem to be included in the package that's pulled down from npm?

Argh, hmmm, I'll have to add it to the files array I think.  Shall post a new version up shortly.

Yes, that's what it must be after poking around in the repo a little bit. :+1:

Done


[bug] weird behavior with any multiple rules with parenthesized_expression

⏯ Playground Link
Playground link with relevant code
💻 Code

Code:
void 0
void 99
void(0)

Pattern:
id: void number
language: typescript
rule:
  any:
  - pattern: void $NUMBER
  - pattern: void($NUMBER)
constraints:
  NUMBER:
    kind: "number"

🙁 Actual behavior
It only matched the first two lines.

🙂 Expected behavior
void(0) should be matched as well
There is an interesting workaround. It will work normally by moving void($NUMBER) to the first rule...

This is expected.  void is an operator, not a functional. https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Operators/void
void $NUMBER

is parsed as unary_expression.

So void $NUMBER can match both void 0 where $NUMBER matches 0 and void(0) where NUMBER matched (0), a parenthesis expression.
I would suggest using the utility rule to extract number or parenthesis expressions.

https://ast-grep.github.io/playground.html#eyJtb2RlIjoiQ29uZmlnIiwibGFuZyI6InR5cGVzY3JpcHQiLCJxdWVyeSI6InZvaWQgJE5VTUJFUiIsInJld3JpdGUiOiJ1bmRlZmluZWQiLCJjb25maWciOiJpZDogdm9pZCBudW1iZXJcbmxhbmd1YWdlOiB0eXBlc2NyaXB0XG51dGlsczpcbiAgaXMtbnVtYmVyOlxuICAgIGFueTpcbiAgICAgIC0ga2luZDogbnVtYmVyIFxuICAgICAgLSBraW5kOiBwYXJlbnRoZXNpemVkX2V4cHJlc3Npb25cbiAgICAgICAgaGFzOlxuICAgICAgICAgIG1hdGNoZXM6IGlzLW51bWJlclxucnVsZTpcbiAgcGF0dGVybjogdm9pZCAkTlVNQkVSIFxuICBoYXM6XG4gICAgbWF0Y2hlczogaXMtbnVtYmVyIiwic291cmNlIjoidm9pZCAwXG52b2lkIDk5XG52b2lkKDApXG52b2lkIGNhbGwoKTtcbnZvaWQgJ251bWVyJ1xudm9pZCAxMjNcbnZvaWQoKCgxMjMpKSkifQ==

I see. Thanks!


change mutex pointer to value

Pull Request Description
Describe what this PR does / why we need it
Does this pull request fix one issue?

Describe how you did it
Describe how to verify it
Special notes for reviews

Codecov Report

Merging #257 (88d2bca) into main (b9f818f) will not change coverage.
The diff coverage is 100.00%.

@@           Coverage Diff           @@
##             main     #257   +/-   ##
=======================================
  Coverage   25.55%   25.55%
=======================================
  Files          24       24
  Lines        1796     1796
=======================================
  Hits          459      459
  Misses       1285     1285
  Partials       52       52

Flag
Coverage Δ

unittests
25.55% <100.00%> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files
Coverage Δ

pkg/ipam/allocator/allocator.go
25.58% <100.00%> (ø)

pkg/ipam/allocator/dualstack_allocator.go
21.22% <100.00%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update b9f818f...88d2bca. Read the comment docs.


Bumped ccx-rules-ocp version

Codecov Report

Merging #196 (c14279a) into master (a27550f) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master     #196   +/-   ##
=======================================
  Coverage   70.23%   70.23%
=======================================
  Files           9        9
  Lines         336      336
=======================================
  Hits          236      236
  Misses         81       81
  Partials       19       19

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update a27550f...c14279a. Read the comment docs.


Adding PrintViewState for easier camera keyframe creation

Before merge:

[ ] Move component up to DisplayPanel component
[ ] Prop to disable it (default off)

Pull Request Test Coverage Report for Build 916444325

0 of 0   changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage remained the same at 100.0%

Totals

Change from base Build 849161526:
0.0%

Covered Lines:
2

Relevant Lines:
2

💛  - Coveralls


Feedback: 5.0 deprecations

See also #51000 / #51424
In 5.0, we've marked the following configurations as deprecated:

target: ES3
noImplicitUseStrict
keyofStringsOnly
suppressExcessPropertyErrors
suppressImplicitAnyIndexErrors
noStrictGenericChecks
charset
out
prepend in project references
implicitly OS-specific newLine

These configurations will continue to "work" until TypeScript 5.5, at which point they will be removed entirely. In TypeScript 5.0, 5.1, 5.2, 5.3, and 5.4, you can specify ignoreDeprecations: "5.0" to silence this warning.
What should I use instead?
target: ES3
ES3 runtimes are nearly 100% extinct, so we're removing support for this as an emit target.
Fix: Use target: ES5 as the next-closest thing along with a linter to detect any potentially-problematic ES3 constructs, or use target: ESNext and use a different downleveling emitter.
noImplicitUseStrict
This flag disables emitting a "use strict"; prologue at the top of module bodies when the target is not implicitly a module. This can lead to unexpected behavior since there are subtle behavioral differences between strict and nonstrict mode.
Fix: Remove any reliance on nonstrict behavior.
keyofStringsOnly
This flag removes symbol-based keys from the keyof T type operator. This can lead to unexpected type errors when code relying on this flag interacts with code that doesn't.
Fix: Use string & keyof T to filter keys to those with string types.
suppressExcessPropertyErrors
This flag disables excess property checking. This can lead to unexpected program behavior when an optional property name is misspelled, since no error is issued.
Fix: use a type assertion to silence this in cases where you really do want an optional property, or fix the target type.
suppressImplicitAnyIndexErrors
This flag disables the error produced when indexing an object which doesn't support indexing. This was always intended to be a temporary flag, so is going away.
Fix: Add an index signature to the relevant type (or use a type assertion at the indexing location).
noStrictGenericChecks
This flag disables certain checks relating the constraints of generic types, which can lead to undesirable unsound behavior.
Fix: Correct the types, or use a type assertion.
charset
This flag hasn't done anything for a very long time.
Fix: Remove it from your config file
out
This flag for emitting a single file from a set of nonmodule inputs incorrectly relied on the current working directory to compute the output filename, leading to unexpected problems when tsc was invoked programmatically or from a different directory.
Fix: use outFile instead.
prepend in project references
This setting allowed for upstream project reference outputs to be prepended to the current project's output files. Had we not needed this ourselves back in 3.0, we likely wouldn't have added it in the first place, and it hasn't seen wide adoption.
Fix: Use a single tsconfig to create output bundles, or use an external bundler on the outputs.
Implicitly OS-specific newLine
The newLine setting controls the line endings used in emitted files. If not specified, in TypeScript 4.9, this would default to the operating system's idiomatic newline (CR LF for Windows, LF for everyone else). Since all modern dev tools support LF endings without issue, we're removing this behavior to increase build predictability, and the new default is LF.
Fix: Pick one or the other, or use a tool after-the-fact to change outputs to the current OS's if this is really desired
Feedback?
We believe we've picked a fairly conservative set of options to deprecate in this release. If these deprecations are overly burdensome for your codebase, we'd like to understand why - please let us know!
Search terms: Flag is deprecated and will stop functioning in TypeScript 5.5. Specify ignoreDeprecations: 5.0 to silence this error

If noImplicitUseStrict is to be deprecated, shouldn’t the out-of-box behavior be to not emit the directive if the target is not a module?  Plain JS isn’t automatically strict in non-module code so it doesn’t feel like TS should be imposing it under the current design principles, as that changes the behavior of the code as written compared to JS. (no matter how bad an idea I think it is to ever write sloppy-mode code)

suppressExcessPropertyErrors

TIL this flag exists

If noImplicitUseStrict is to be deprecated, shouldn’t the out-of-box behavior be to not emit the directive if the target is not a module?

So the issue is that if you have the input code
export function fn() {
  if (this === undefined) {
    console.log('called free');
  } else {
    console.log('called bound');
  }
}
fn();
({ fn }).fn();

Per spec, this must print "called free, called bound", because all module code is in strict mode, and this must be a module file (it has a top-level export declaration), but without "use strict"; this prints "called bound, called bound". Removing this flag is basically saying you can't opt into this particular out-of-spec behavior.

To clarify, this doesn't change any behavior when the input file is not a module.

thanks, this was the part I was missing.

Will any-indexing still be allowed when "noImplicitAny": false is set?

Will any-indexing still be allowed when "noImplicitAny": false is set?

Yes

Personally I find the default behaviour of newLine weird. TypeScript compile output now depends on the build system operating system. I would love to see that option deprecated, default to "lf" (even crlf is fine, as long as it’s consistent), and ideally removed in a future version.
Similarly I think forceConsistentCasingInFileNames should be true by default, deprecated, and removed later. I see no reason to not support case sensitive file systems by default.

https://github.com/microsoft/TypeScript/pull/52298

The list is missing importsNotUsedAsValues and preserveValueImports.
As a general feedback, I'm having a headache currently how to make the deprecations go away in a backwards-compatible way. For context, in SvelteKit we're creating a generated tsconfig.json which could be used for versions prior or after 5.0. We could check ts.version, but that's only the TypeScript version the user has installed, not necessarily the VS Code TS version (as can be seen by this issue https://github.com/sveltejs/kit/issues/8650).
Ideally it would be possible to set ignoreDeprecations as a flag prior to 5.0 (e.g. backporting it to 4.9/4.8) - though that ship probably has probably sailed already since it's no use for the VS Code problem: there won't be releases for older TS versions and users on old VS Code versions won't update. But it's probably good for the future to allow to set deprecation values like 6.0 in versions before 6.0 (maybe starting with 5.5).

That's true, it never was legal before, but it also never occured before that a flag became deprecated and resulted in an error, so you could use your unchanged tsconfig across (major) versions.

Yep, good point. We've merged #52419 so will ship a patch release of 4.9 shortly that will allow specifying ignoreDeprecations.

Will it be possible to have something like "ignoreDeprecations": "all"?
I work on a library which runs code through the compiler API programatically on different versions of TS. Similar to dtslint type testing it reports type errors. The library reads tsconfig.json provided by a user. I think in this context it might make sense to simply ignore all deprecation related errors.

What specific changes need to be made to a tsconfig.json file to include the output of a project reference in an outputted file?
My .tsconfig for example currently looks like this:
{     "compileOnSave": true,     "compilerOptions": {         "target": "ES2021",         "module": "system",         "outFile": "teranova.js", // etc     },     "references": [         {             "path": "../../Javascript/Canvas",             "prepend": true         }, // etc     ] }

What specific changes need to be made to a tsconfig.json file to include the output of a project reference in an outputted file?

The functionality is going away entirely. You look into something like rollup, parcel, or just straight concatenation with a script.
Side observation: to be honest, using prepend with system+outFile was never even considered as a scenario when that feature was implemented, so the fact that worked in the first place is kind of a happy accident at best 😲

Well that's quite a badge of honour to be told I've been doing it wrong successfully for about seven years! 😲 I've got a custom development process where I clone various repositories (I.E. my libraries) into a sort of workspace and combine them into a single .js file for local testing, and deployment onto the web, often as SPAs which are essentially games oriented around a canvas HTML object. Just out of interest, what is the typical way a developer should be going about this? As in, to clone multiple repositories (where one of them is the app itself) and build them all and combine them all together for local testing and deployment? I understand that there are various options but obviously after your comment I'm keen to finally understand what a normal person would do!

Just to add, in case it makes it more obvious what I'm talking about, I had been approacing software development with TypeScript having come from a background of working in C# and Visual Studio, so my development process is as if I am creating a solution consisting of multiple projects which have been included in the solution and then when I build it produces a single executable.

Maybe it would also be good to switch the default module compiler option to node16. This is fairly impactful to users, but it is the most correct value in most cases. I believe many users are still not even aware of this option, causing packages to be stuck on not supporting it at all.

📰: TypeScript 4.9.5 is now out, which allows specifying "ignoreDeprecations": "5.0"

Still not online at
https://www.nuget.org/packages/Microsoft.TypeScript.MSBuild :-(
#48456
@RyanCavanaugh @DanielRosenwasser

Weird - the deploy timed out for just the NuGet build.
It should be out now!

@frost-cy Hello! Will there be a "google feedback" ticket for 5.0? Sorry for a ping.

@frost-cy Hello! Will there be a "google feedback" ticket for 5.0? Sorry for a ping.

yes, @mhausner is working on it.

What to use instead of suppressImplicitAnyIndexErrors could use an example for (or use a type assertion at the indexing location)
I can't change the underlying type of process, so ok I should use a type assertion, but how to write that isn't so clear.
old code w/ suppressImplicitAnyIndexErrors:
function replaceStdoutStderrStringWithStream(o: any): void {
    if (o.stream === 'stdout' || o.stream === 'stderr') {
        o.stream = process[o.stream];
    }
}

1st attempt at type assertion:
function replaceStdoutStderrStringWithStream(o: any): void {
    if (o.stream === 'stdout' || o.stream === 'stderr') {
        o.stream = process as {[p: string]: any}[o.stream];
    }
}

that almost works except
error TS2503: Cannot find namespace 'o'.
35         o.stream = process as {[p: string]: any}[o.stream];

Is there a simple fix for this? the o param is just an object with a stream property (and others which don't matter here)

o.stream = (process as any)[o.stream]

Closing since 5.0 is out! Feedback still welcomed though

I've got one more (if this is the wrong place, I can delete this and move it).
After updating to typescript 5.0.2. I just tried to set the "verbatimModuleSyntax": true option in my tsconfig.json file and tsc 5 throws an exception (it doesn't throw if I set it to false):
> tsc

/Projects/mysql-query/node_modules/typescript/lib/tsc.js:113959
      throw e;
      ^

Error: Debug Failure. False expression.
    at isReferencedAliasDeclaration (/Projects/mysql-query/node_modules/typescript/lib/tsc.js:80348:11)
    at canConvertImportDeclarationToTypeOnly (/Projects/mysql-query/node_modules/typescript/lib/tsc.js:78887:175)
    at checkImportsForTypeOnlyConversion (Projects/mysql-query/node_modules/typescript/lib/tsc.js:78902:11)
    at checkSourceFileWorker (/Projects/mysql-query/node_modules/typescript/lib/tsc.js:79465:9)
    at checkSourceFile (//Projects/mysql-query/node_modules/typescript/lib/tsc.js:79413:5)
    at checkSourceFileWithEagerDiagnostics (/Projects/mysql-query/node_modules/typescript/lib/tsc.js:79507:5)
    at getDiagnosticsWorker (/Projects/mysql-query/node_modules/typescript/lib/tsc.js:79515:7)
    at Object.getDiagnostics (/ /Projects/mysql-query/node_modules/typescript/lib/tsc.js:79492:14)
    at / /Projects/mysql-query/node_modules/typescript/lib/tsc.js:113984:76
    at runWithCancellationToken (/Projects/mysql-query/node_modules/typescript/lib/tsc.js:113954:14)

Node.js v18.15.0`

I can edit this with the rest of my tsconfig.json if that would be helpful, I didn't want the post to be too large with unhelpful info.

That's just a bug; I would definitely file that as a real issue.

o.stream = (process as any)[o.stream]

I find that quite dangerous, since process could be null, and we are muting possible null-access error with the "as any"...
I wandered here, as we have a lot of lines that should ideally look like:
x?.[Symbol.toStringTag] === "someConstant"
however x is actually unknown, so we need to cast it to something... and if you choose any as that something (which our linter will hate) , it is very easy to miss the possible null-access error.
So some of the safer but quite verbose alternatives are:

(x as Record<keyof never, unknown> | undefined)?.[Symbol.toStringTag] and
(obj as { [Symbol.toStringTag]: unknown})?.[Symbol.toStringTag]

I was wondering if suppressImplicitAnyIndexErrors would help us, but it is being deprecated -> could you recommend some not-so-verbose but reasonably safe alternative to (x as any)?.[Symbol.toStringTag]

ES3 happens to be the version under which you will be scripting  Adobe After Effects
You Mean to tell me that ES3 runtimes are Extinct with all these people scripting
the Adobe Creative Suite ... stuff Like After Effects ...Premiere Pro ... Photoshop ... and Illustrator.
You obviously are not looking at these people or ... may I say it directly ... at the right spot ... it is a pretty big statement you made ... if you ask me

ExtendScript by Adobe is an Extension of ES3

As of Version 16, which came out 6 years ago, the After Effects scripting engine has moved to ES2018. The ExtendScript version is considered legacy.

@dummdidumm I've added those to the list for anyone else reading.

@remcohaszing @DanielRosenwasser @mlippert @HolgerJeromin


archive/zip: add File.OpenRaw, Writer.CreateRaw, Writer.Copy

If you have a compressed file, know its uncompressed size and crc32 checksum, it'd be nice to be able to add it to an archive as is.
I have three current use-cases for this:

Repackaging a zip file, removing or adding files, without incurring the associated compression overheads for files that already exist (somewhat achieving https://github.com/golang/go/issues/15626)
Compress first and include later based on whether the compressed size was smaller than the original, without having to perform the compression twice.
Support concurrent compression: compress many files concurrently and then copy the already compressed files to the archive (similar to Apache Commons Compress' ParallelScatterZipCreator)

I see three different ways we could achieve this:

Create a zip.CreateHeaderRaw(fh *FileHeader) (io.Writer, error) function, that uses the FileHeader's CRC32 and UncompressedSize64 fields set by the user.
Use the existing CreateHeader(fh *FileHeader) (io.Writer, error) function, but have a new FileHeader field that indicates we're going to write already compressed data (and then use CRC32 and UncompressedSize64 fields set by the user)
Use the existing CreateHeader(fh *FileHeader) (io.Writer, error) function, but if CompressedSize64 has already been set, assume that data written is already compressed.

I'm going to assume that option 3 would be a no-go, because existing code might suddenly break if a user has already set the CompressedSize for whatever reason, but hopefully the other options are viable.

/cc @dsnet

Added change that uses option 1 (CreateHeaderRaw) because it feels weird to use a field in FileHeader solely for archiving as that structure is also used when reading an archive.

This is going to be new API, so turning into a proposal.
@dsnet Please add any thoughts you may have about this functionality.  Thanks.

The project I'm working on would appreciate this functionality for performance reasons. We need to replace a single file within a zip. The DEFLATE tax is high and wasteful since we decode/encode without needing or using the intermediate data.
I like @rsc's Copy implementation in https://github.com/rsc/zipmerge. It might not be quite as flexible as the other proposals, but I think it fits well within the existing API and supports efficiently creating a new zip from an existing zip while allowing the user to replace/delete/update specific files.
// Copy copies the file f (obtained from a Reader) into w.
// It copies the compressed form directly.
func (w *Writer) Copy(f *File) error {
	dataOffset, err := f.DataOffset()
	if err != nil {
		return err
	}
	if err := w.closeLastWriter(); err != nil {
		return err
	}

	fh := f.FileHeader
	h := &header{
		FileHeader: &fh,
		offset:     uint64(w.cw.count),
	}
	fh.Flags |= 0x8 // we will write a data descriptor
	w.dir = append(w.dir, h)

	if err := writeHeader(w.cw, &fh); err != nil {
		return err
	}

	r := io.NewSectionReader(f.zipr, dataOffset, int64(f.CompressedSize64))
	if _, err := io.Copy(w.cw, r); err != nil {
		return err
	}

	return writeDesc(w.cw, &fh)
}

Here are the latest compress/flate benchmarks from master at level 5 which archive/zip uses by default (Intel Xeon W-2135 CPU @ 3.70GHz × 6):
goos: linux
goarch: amd64
pkg: compress/flate
BenchmarkDecode/Digits/Level5/1e4-12         	    6530	    164347 ns/op	  60.85 MB/s	   40598 B/op	       7 allocs/op
BenchmarkDecode/Digits/Level5/1e5-12         	     960	   1131532 ns/op	  88.38 MB/s	   40842 B/op	      13 allocs/op
BenchmarkDecode/Digits/Level5/1e6-12         	     117	   9673619 ns/op	 103.37 MB/s	   44800 B/op	      79 allocs/op
BenchmarkDecode/Newton/Level5/1e4-12         	    8076	    164681 ns/op	  60.72 MB/s	   41114 B/op	      17 allocs/op
BenchmarkDecode/Newton/Level5/1e5-12         	    1653	    765625 ns/op	 130.61 MB/s	   44283 B/op	      31 allocs/op
BenchmarkDecode/Newton/Level5/1e6-12         	     180	   6408771 ns/op	 156.04 MB/s	   61144 B/op	     156 allocs/op
BenchmarkEncode/Digits/Level5/1e4-12         	    5710	    215617 ns/op	  46.38 MB/s
BenchmarkEncode/Digits/Level5/1e5-12         	     370	   3232511 ns/op	  30.94 MB/s
BenchmarkEncode/Digits/Level5/1e6-12         	      32	  34852520 ns/op	  28.69 MB/s
BenchmarkEncode/Newton/Level5/1e4-12         	    4774	    238334 ns/op	  41.96 MB/s
BenchmarkEncode/Newton/Level5/1e5-12         	     357	   2974790 ns/op	  33.62 MB/s
BenchmarkEncode/Newton/Level5/1e6-12         	      39	  30595764 ns/op	  32.68 MB/s
PASS

@escholtz I agree that having a (*Writer) Copy(f *File)  would be nice, but as you said, it isn't quite as flexible.
For comparison, copying a file with the patch I've submitted would look something like this:
func Copy(zw *zip.Writer, zf io.ReaderAt, f *zip.File) error {
	offset, err := f.DataOffset()
	if err != nil {
		return err
	}

	w, err := zw.CreateHeaderRaw(&f.FileHeader)
	if err != nil {
		return err
	}

	_, err = io.Copy(w, io.NewSectionReader(zf, offset, int64(f.CompressedSize64)))

	return err
}

Having Copy() would be great, but if for some reason only one solution could be added, I'd prefer it to be something along the lines of CreateHeaderRaw.
For the other use cases originally mentioned, I've written https://github.com/saracen/fastzip that takes advantage of CreateHeaderRaw also (but I've had to vendor archive/zip and include my modification for it to work).

Adding a whole new Create method just to set this extra bit seems unfortunate - what if there is a second bit we want control over later? Add four new methods? And so on.
Probably better to add a new bool in the CreateHeader. Perhaps WriteRaw bool?

Probably better to add a new bool in the CreateHeader. Perhaps WriteRaw bool?

@rsc I originally preferred this too. The reason I didn't for my implementation is because FileHeader is also used when reading an archive too. It felt odd having a WriteRaw bool as part of that structure in the context of reads. Is this not an issue?

@saracen it seems like less of an issue than an exponential blowup of create methods. The NonUTF8 field is probably most analogous, although it does get set in the reader.
I suppose the most interesting analogy is whether there's a need to get the raw bytes from the reader as well. There at least we could add an OpenRaw method to zip.File. It's much harder to do that in what CreateHeader returns, because it's an io.Writer.
I suppose another way to store compressed files would be to register a no-op compressor.
Another way would be to have a method on zip.Writer: WriteRaw(bool) to set whether the compressors should be avoided. You'd z.WriteRaw(true) before calling CreateHeader.
Does any of these sound like a winner to anyone?

suppose the most interesting analogy is whether there's a need to get the raw bytes from the reader as well.

@rsc At the moment, I believe there's two ways to access the raw bytes:

Open with a ReaderAt, and then use file.Offset and file.CompressedSize64 to access and read a specific section of the archive.
Register a decompressor and do whatever custom logic to the raw bytes.

I suppose another way to store compressed files would be to register a no-op compressor.

This sounds quite good, especially since it's similar to how you can handle raw reads by registering a decompressor.
Using a zip.NoOpCompressor would need to mean that the CompressedSize64 and CRC32 fields of the header are left unmodified when the header is closed:
zw := zip.NewWriter(archive)
zw.RegisterCompressor(zip.Deflate, zip.NoOpCompressor)

hdr := &zip.FileHeader{
  Name: "file.txt",

  // the following fields would be left untouched because
  // the writer detects that zip.NoOpCompressor is being used
  CRC32: 0xffffffff,
  CompressedSize64: 1024,
}

w, _ := zw.CreateHeader(hdr) (io.Writer, error)

// bytes are copied to the zip archive unmodified
io.Copy(w, buf)

The problem with the no-op compressor is that the uncompressed size would be wrong. That would have to be left alone. It's probably a little too subtle.
Given that you can already read the compressed data directly, as @saracen points out, maybe it's fine to focus on writing.
What if pre-filling CompressedSize64 (> 0) in the FileHeader means that the writes are expected to be pre-compressed? In that case CRC32 has to be filled in too, although zero is a valid CRC32 so we can't double-check that. But I believe 0 is not a valid CompressedSize64, and in any event it's not an interesting one.
In essence, CompressedSize64 = 0 means "I don't know, you compress it for me".

@saracen, @escholtz, @klauspost, any thoughts about the most recent suggestion (previous comment)?

According RFC1951, the compression stream requires at least one block (with the BFINAL bit set). Therefore, it is impossible to have a compressed DEFLATE stream that is zero-bytes in length.
However, the ZIP file format allows for a number of different compression methods to be used (DEFLATE happens to be the most common by far). I think we should think about 1) whether we want to support other compression methods, and 2) whether we ever want to support other compression format that may theoretically have a zero-length output for an empty input.

I've been hesitant to comment because I didn't want to hijack @saracen's original proposal.
For my use case, the func (w *Writer) Copy(f *File) error implementation would be sufficient. I think it fits cleanly with the existing package exports and still maintains simplicity.
I agree that having both CreateHeader and CreateHeaderRaw does not feel like a clean standard library package.
Is there precedent in the standard library for using the CompressedSize64 approach?

For my use case, func (w *Writer) Copy(f *File) error doesn't help, but I can see it's usefulness and how it relates this this problem.
@rsc In my original proposal, I'd written:

Use the existing CreateHeader(fh *FileHeader) (io.Writer, error) function, but if CompressedSize64 has already been set, assume that data written is already compressed.

I'm going to assume that option 3 would be a no-go, because existing code might suddenly break if a user has already set the CompressedSize for whatever reason, but hopefully the other options are viable.

Breaking existing code was my only reluctance to this approach. If we don't see that as an issue, it sounds like an easy way to support this.

@saracen I took another look at the three examples you originally provided and I think they could all be achieved with a Copy method?

Repackaging a zip file, removing or adding files, without incurring the associated compression overheads for files that already exist (somewhat achieving #15626)

Using existing zip file(s), don't Copy files you want to remove. Copy existing files you want to keep. Use existing methods to add new files.

Compress first and include later based on whether the compressed size was smaller than the original, without having to perform the compression twice.

Write a new zip file, read the compressed size, only Copy if it's smaller.

Support concurrent compression: compress many files concurrently and then copy the already compressed files to the archive (similar to Apache Commons Compress' ParallelScatterZipCreator)

Write multiple zip files concurrently and then Copy files from each into a single zip file.

Perhaps this approach isn't as elegant and has slightly more garbage/performance overhead but it avoids the tax of repeatedly compressing the same file which I would expect is the bottleneck for most cases.

Breaking existing code was my only reluctance to this approach. If we don't see that as an issue, it sounds like an easy way to support this.

For all Go code inside Google, I only found one instance of someone implicitly setting CompressedSize. The usage pattern looked like the following:
f, ... := rf.Open()
wc, ... := w.CreateHeader(&rf.FileHeader)
... io.Copy(wc, rc)

Essentially the code is copying one Zip file to another.
If we go with the semantics that a positive CompressedSize means to interpret the input as already compressed, then the pattern above will break since io.Copy will read uncompressed data from rc and write it to wc when the latter is expected compressed data.
We could "fix" this by adding a ReadFrom or WriteTo method that specially handles copying of compressed data from one zip.File to another. However, that approach seems a bit too magical and I can see other edge cases where it fails.

@dsnet, I don't think we have to fix this, but we could at least detect the problem: if the alg is deflate and we expected compressed data, we can at least check that the first few bytes are a proper deflate header. That would make the io.Copy return an error. But it is a little concerning.

proper deflate header.

Unfortunately DEFLATE doesn't really have any distinguishable magic numbers, unlike GZIP, which typically has a 10-byte sequence that is readily identifiable. It seems that the best we could do it decompress some number of bytes and assume the input is valid if it passes.

It sounds like I should look in the Go corpus I have for other code using this pattern. I'll try to do that for next week. Maybe overloading the CompressedSize here is too subtle.

Will try for next time.

It feels like we're loosing interest in this proposal and should draw a conclusion.
I don't think overloading CompressedSize is a good approach.
I still like the Copy approach that @rsc chose to use in the zipmerge package. I think all 3 use cases in the original proposal can be solved with a Copy method.

@escholtz, I haven't yet done the scan, but it doesn't seem like it's necessary - it's a plausible example that we'd break.
Re https://github.com/golang/go/issues/34974#issuecomment-659518287, the one thing that would still be good to support would be if you have the header bits in hand along with the compressed form, but you don't have them in an already-existing zip file. (Maybe you want more control over the compression, or you have DEFLATE or an alternate compressed form already from a different archive file, or something like that.)
After rereading this thread now with a bit more distance, I think the fundamental operation is really just access to raw data in both reading and writing. I suggest we add those directly, along with the Copy helper:
// like CreateHeader but the data written to the io.Writer should be precompressed
func (*Writer) CreateRaw(file *FileHeader) (io.Writer, error)

// like Open but the data read back is the compressed form
func (*File) OpenRaw() (io.Reader, error)

// helper - does f.OpenRaw, w.CreateRaw, io.Copy
func (w *Writer) Copy(f *File) error

That sounds reasonable. Looking at the package, I assumed this type of access was intentionally not exported for simplicity and reliability. If we're ok with making the package a little less simple, this would enable the advanced use cases we're trying to achieve.

Hi everybody!  :smile:
I was thinking about that for couple of months. After long thinking I created my solution.
I was dreaming to put my ideas in live when I found out this issue  :laughing:
So far I had the same 2 problems in mind

How to copy files without Decompression-Compression.
Precompress multiply files. Or compress parallelly

I had different iterations in my mind. First there was something like @saracen mentioned with the similar looking ReadCompres method. It was dirty and monstrous. So far after long thinking I created an ABSOLUTE copy of Copy method from here  :laughing: :laughing: :laughing: I even did the same optimization with writeDesc func. But  I had my genious moment when I discover that both tasks can be done only with Copy method.
But I am happy that I still can add my penny here and probably resolve this discussion about blowing up methods.
We don't need CreateRaw(file *FileHeader) method. And probably OpenRaw() (io.Reader, error) also.
It's easy resolved only with Copy
Here is the idea

func main() {
       // final writer
	zw := zip.NewWriter(bytes.NewBuffer([]byte{}))
       // get precompressed zip.File
	prFile := Precompress(data)
       // copy precompressed zip.File
	zw.Copy(prFile)

}

func Precompress(b []byte) zip.File {
	// create tmp zip writer
	buf := bytes.NewBuffer([]byte{})
	zw := zip.NewWriter(buf)
       // write/compress data in
	zwf, _ := zw.Create("test")
	zwf.Write(b)
	// extract file from tmp zip
	zr,_  := zip.NewReader(readerAt(buf.Bytes()), int64(buf.Len()))
	return zr.File[0]
}

In this way all compression, descriptor validation logic is still encapsulated(!). And there are no overheads (I think) . We compress data to buffer then we do io.Copy to the destination reader. It will be the same in case of compressing it directly.
In case if user want to copy couple of files from big zip and keep them in memory. It's again can be done with Copy. User copy needed files to the tmp zip and then they needed it copy it to destination zip.
I didn't think about the OpenRaw() method before. There are no much use cases so far. If user need to keep only 1 file in compressed state then them can use tmp zips. Use case can be when an user want to extract raw file and send it directly over http to the client :thinking:  . In my first iterations I achieved it through Compressor interface. I copied data to the buffer and ignored error from CRC2 check. It worked pretty well for long time. I will post code here when I find it.
Conclusion

The Copy method is critical
Other methods can be implemented through wrappers

I suggest we add those directly, along with the Copy helper:

@rsc

I'm not sure we need OpenRaw, as mentioned in https://github.com/golang/go/issues/34974#issuecomment-648982928 there's already a few ways to access the raw data. Although, OpenRaw is probably clearer.
Personally, I'd prefer CreateHeaderRaw over CreateRaw, I think it's clearer that it's a variant of CreateHeader.
I think Copy should also be added 👍

@klauspost's compress library added both CreateHeaderRaw and Copy. Obviously, this is an external library, but it would be nice if the API continued to match.
@im7mortal Using just Copy does work and @escholtz suggested this too. It's a good solution for many problems, but it relies on either your precompressed data being in a zip file or you first copying it to one. Copying precompressed data from a different archive format would require a temporary copy to a new zip file, only to be read back again.

Personally, I'd prefer CreateHeaderRaw over CreateRaw, I think it's clearer that it's a variant of CreateHeader.

It seems unnecessarily long. There's no way to have CreateRaw that doesn't also take a header.

If we plan on adding a CreateHeaderRaw, I suspect that we'll need to pay close attention to #22520. We'll want to make sure we get the semantics right from the start, otherwise they will be very hard to change later on.

Thanks for the pointer. I don't see why they are necessarily related.
CreateRaw should do the same as CreateHeader, just preparing for raw writes.
It shouldn't mix in other different semantics.
For that issue, I commented there but I think it would be fine to add a new field
that CreateHeader (and then by extension CreateRaw) would use.
I think that can be an independent decision though.

@saracen

... but it relies on either your precompressed data being in a zip file or you first copying it to one. Copying precompressed data from a different archive format would require a temporary copy to a new zip file, only to be read back again.

It's not avoidable in both cases. It's just the same but CreateHeaderRaw discards all incapsulated logic. Instead an user must implement it themselves.
Precompression with CreateHeaderRaw

Create crc32 checksum wrapper.
Create byte buffer
Create compressor writer
Write data to the compressor writer through crc32 wrapper
Create raw header zip.CreateHeaderRaw. Ensure crc32 and precompressed size is correctly set :exclamation:
Do copy from buffer(2) to writer(5)

func preompress(b []byte) (check uint32, c []byte, err error) {
	wsumm := crc32.NewIEEE() // an user must ensure correct check sum
	_, err = wsumm.Write(b)
	if err != nil {
		return
	}
	buff := bytes.NewBuffer([]byte{})
	wcmp, err := flate.NewWriter(buff, flate.DefaultCompression) // MUST MATCH with actual compression method
	if err != nil {
		return
	}
	_, err = wcmp.Write(b)
	if err != nil {
		return
	}
	return wsumm.Sum32(), buff.Bytes(), err
}

...
	w := zip.NewWriter(writer)
	crc32_, compressed, err := preompress(data)
	if err != nil {
		return
	}
	wf, err := zip.CreateHeaderRaw(zip.FileHeader{
		Name:               "tmp",
		CRC32:              crc32_,  // an user must ensure correct check sum
        // an user must ensure correct uncompressed size
        // it requires additional logic in case of the pipe writer. CRC32 also
		UncompressedSize64: uint64(len(data)),
		Method:             zip.Deflate,  // MUST MATCH with actual compression method
	})
	if err != nil {
		return
	}
	_, err = wf.Write(compressed)
	if err != nil {
		return
	}
...

:x: Need wrapper
:x: Additional Write-Read-Write
:x: Unencapsulated. User must implement crc32 sum and propagate it with length correctly
:x: Unencapsulated. of the zip.Decompressor is unused. User can use different method from which set in the header.
:x: Complex wrapper in case of the pipe writer
:x: Allow have corrupted file in the archive :warning: Can cause crash of receiver zip libraries :exclamation:
Precompress through tmp zip

Create tmp zip writer
Write data to the tmp zip. crc32 and byte counters are encapsulated
Extract zip.File
Perform zipWriter.Copy. Which do io.Copy

In both cases there is need to write wrappers.
func PrecompressInZip(b []byte, name string, method uint16) (zf zip.File, err error) {
	// create tmp zip writer
	buf := bytes.NewBuffer([]byte{})
	zw := zip.NewWriter(buf)
	// write/compress data in
	zwf, err := zw.CreateHeader(&zip.FileHeader{
		Name: name,
		Method: method,
	})
	if err != nil {
		return
	}
	_, err = zwf.Write(b)
	if err != nil {
		return
	}
	// extract file from tmp zip
	zr, err := zip.NewReader(readerAt(buf.Bytes()), int64(buf.Len()))
	return zr.File[0], err
}
...
	// final writer
	zw := zip.NewWriter(bytes.NewBuffer([]byte{}))
	// get precompressed zip.File
	prFile, err := Precompress(data, zip.Deflate)
	if err != nil {
		return
	}
	// copy precompressed zip.File
	zw.Copy(prFile)
...

:x: Need wrapper
:x: Additional Write-Read-Write
:warning: add directoryHeaderLen=46 bytes of metadata (it vary but very small)
:green_book: CRC32 summ and length are encapsulated and ensured by very good logic
:green_book: Unencapsulation of the zip.Decompressor is used fully. The writer use only corresponding compressor. No way to mix up!
:green_book: Pipe wrappers do not require additional logic and can be handled through std libs
:green_book: Don't allow have corrupted file in the archive.
Conclusion
@rsc The CreateHeaderRaw is very error prone. All encapsulated logic from zip library must be reimplemented by user.

it's not guaranteed that compression method matches with compression implementation
CRC2 hash and uncompressed size must be managed by user. Can be complicated in case of the pipe writers.
there are no advantages against zip.Copy from temporary zip. It require the same Write-Read-Write sequence.
it allow to write silently corrupted files to the archive what can cause crash on reader side of other libs

I agree with @im7mortal on this.
I think most gophers using the archive/zip package are looking for simplicity. They don't care about the specifics of the zip format, they just want to be able to read and write files.
A much smaller number of gophers are looking for a zip package that exports all of the guts that archive/zip currently hides. Building blocks to make it easier to build a custom / optimized zip package.
I think the standard library should lean more towards the simple case and not the building blocks case. I worry if we try to do both, we end up with a muddled package that is more complicated and confusing for the typical consumer.
I still don't know if Copy should be added (even though I personally want it so I don't have to maintain a fork). On the one hand, it maintains safety and doesn't add too much complexity. On the other hand, it's only necessary to support less common use cases. And it still doesn't support advanced use cases (copying from other archive formats, although I would suspect this is even more rare than copying between zips).

@im7mortal

It's not avoidable in both cases. It's just the same but CreateHeaderRaw discards all incapsulated logic.

There are cases where this is avoidable. You can have compressed data, know the uncompressed size and either already have the crc32 checksum of the uncompressed data or calculating it is still much faster than decompressing and recompressing.
The compressed data doesn't even need to be DEFLATE. This package already supports vendor compression methods, and there's certainly use-cases where I might want to move pre-existing compressed data (say zstd) into the zip structure, over another archive format.
The whole point of CreateRaw is to give more control over this and to go around the safety offered by the existing functionality. The zip package already allows for this to a certain degree, and it's already possible to create corrupt/invalid archives accidentally.
@escholtz Regarding simplicity, this is difficult to not agree with.
Supporting vendor compression methods and arbitrary extra data are also features that are not going to be used by everybody, but they've been added. I personally don't think that CreateRaw is going to confuse users or take away from the simplicitiy of the more common functions provided, but I understand your point.

@saracen

There are cases where this is avoidable. You can have compressed data, know the uncompressed size and either already have the crc32 checksum of the uncompressed data or calculating it is still much faster than decompressing and recompressing.

... into the zip structure, over another archive format.

I am agree that there are cases. I think we could consult with #15626 (Delete, Append, Replace methods).
The most common use cases parallel compress, precompress, copy between zips can use Copy with negligible overhead.

@im7mortal, you are right that much can be done with Copy. That's why I included it in the suggestion in https://github.com/golang/go/issues/34974#issuecomment-701705168 to add CreateRaw, OpenRaw, and Copy (all three), even though technically Copy can be implemented in terms of the other two.
In contrast, dropping the other two makes the use cases that need that low-level access fairly awkward and non-obvious.
It doesn't seem harmful to have these two important building blocks alongside the higher-level helper.

Does anyone object to adding CreateRaw, OpenRaw, and Copy?

Based on the discussion above, this seems like a likely accept.
This is subtle, though, so it would make sense to put off until Go 1.17 at this point.

No change in consensus, so accepted.

Maybe I missed this, but why not just open the file directly instead of using zip.Open if you want to write/read bytes directly?

@hherman1 This proposal is about working with compressed files that are contained within zip archives.  It's not about working with the zip archive itself.

When implementing this in https://golang.org/cl/312310 one challenge I discovered was figuring out how we should handle the trailing data descriptor (https://en.wikipedia.org/wiki/ZIP_(file_format)#Data_descriptor) for the OpenRaw and CreateRaw methods. @ianlancetaylor also raised concern in the review for good reason.
I chose to return the data descriptor in the OpenRaw reader and give the CreateRaw writer responsibility for calculating and appending the data descriptor when necessary. This seemed optimal in terms of giving the caller full control, which is the intention of this change, while also conforming to the interface we agreed to. And it also ensures the data descriptor isn't lost by the Copy method.
Perhaps I'm overlooking a different/better approach or perhaps we want to change the interface to allow the same data descriptor reading/writing functionality but in a safer way.
What are your thoughts?

For me, OpenRaw/CreateRaw was mostly about providing raw access to the file data and not modifying the fields that I provide in the header on my behalf. I still expected it to serialize the header data in the same way.
I don't know whether it's needed, but I understand the reason for adding it.

The data descriptor is part of the spec. We need to support it for the raw methods. The CRC32 and uncompressed size fields are not things we can compute within the raw methods. So we need to allow the caller to specify them.
The 3 data descriptor fields are also in the FileHeader but I don't think it's a good idea to overload them by copying the data descriptor fields into the FileHeader. Since it's a separate segment it should be treated as such.
The only other option I can think of is adding a DataDescriptor struct within the archive/zip package that handles marshalling and unmarshalling of the byte segment. But since the Raw methods are intended for advanced/rare use cases, I'm fine with allowing the caller to handle the marshalling and unmarshalling.
@rsc Are you ok with this approach (returning the data descriptor bytes from OpenRaw and requiring the caller to write the data descriptor bytes to the CreateRaw Writer)?

The 3 data descriptor fields are also in the FileHeader but I don't think it's a good idea to overload them by copying the data descriptor fields into the FileHeader. Since it's a separate segment it should be treated as such.

@escholtz We serialize the known crc32, uncompressed and compressed size fields from FileHeader when we write the local file header. What benefit is there to not doing the same in the data descriptor? Would they ever be different?

For the CreateRaw case, if the FileHeader Data Descriptor 0x8 Flag is not set, but a non-nil dataDescriptor argument is passed, would we treat that as an error?
And if the CRC32, CompressedSize, and UncompressedSize fields are non-zero in the FileHeader and a non-nil dataDescriptor argument is passed, would we treat that as an error?

@saracen They could be different when reading a zip that doesn't strictly adhere to the spec. Probably a rare use case, but I can imagine some caller might want to know if and when there is a discrepancy and choose how to handle it in their own code.

For the CreateRaw case, if the FileHeader Data Descriptor 0x8 Flag is not set, but a non-nil dataDescriptor argument is passed, would we treat that as an error?

Sure.

And if the CRC32, CompressedSize, and UncompressedSize fields are non-zero in the FileHeader and a non-nil dataDescriptor argument is passed, would we treat that as an error?

I don't see why.  But I am definitely not an expert.
Should we just take those values from the FileHeader and not worry about the data descriptor for CreateRaw?

They could be different when reading a zip that doesn't strictly adhere to the spec.

I think there's already a way this can be detected with the existing API. By using DataOffset and reading at the end of the compressed data size. It's such a rare case, I'm not sure we need the convience to detect it with the new API.
On the write side, I'm not sure there's a use-case where we would want to intentionally have a different crc32/size than we have in the local header. I could be wrong though.

I tend to agree with @saracen that we don't need API changes here.
OpenRaw and CreateRaw are about "raw compressed bytes" not "all possible raw access to the zip data structures".
On the read side, we can fill in the missing FileHeader fields using the data descriptor during Reader.init. That will benefit all users, not just users of OpenRaw.
On the write side, the fields are in FileHeader and can be copied out of file header into a new data descriptor.
Again there is no goal to permit different values in the file header versus the data descriptor. That's not a nice zip file at all, whether it is technically well-formed or not.

After taking a closer look at the spec, the data descriptor values should always match the corresponding values in the central directory.
The central directory is already the source of truth when a zip is opened, so I don't think we even need to check the data descriptor unless we want to detect mismatches with the central directory and return an error?
Since the local file header is mostly ignored in the read path, I'm fine with ignoring the data descriptor as well. Otherwise, we should probably only use it to detect mismatches.
Sorry for all the back and forth on the issue. First time going this deep on the zip spec and want to make sure I get it right.

4.4.7 CRC-32: (4 bytes)

   The CRC-32 algorithm was generously contributed by
   David Schwaderer and can be found in his excellent
   book "C Programmers Guide to NetBIOS" published by
   Howard W. Sams & Co. Inc.  The 'magic number' for
   the CRC is 0xdebb20e3.  The proper CRC pre and post
   conditioning is used, meaning that the CRC register
   is pre-conditioned with all ones (a starting value
   of 0xffffffff) and the value is post-conditioned by
   taking the one's complement of the CRC residual.
   If bit 3 of the general purpose flag is set, this
   field is set to zero in the local header and the correct
   value is put in the data descriptor and in the central
   directory. When encrypting the central directory, if the
   local header is not in ZIP64 format and general purpose
   bit flag 13 is set indicating masking, the value stored
   in the Local Header will be zero.

4.4.8 compressed size: (4 bytes)
4.4.9 uncompressed size: (4 bytes)

   The size of the file compressed (4.4.8) and uncompressed,
   (4.4.9) respectively.  When a decryption header is present it
   will be placed in front of the file data and the value of the
   compressed file size will include the bytes of the decryption
   header.  If bit 3 of the general purpose bit flag is set,
   these fields are set to zero in the local header and the
   correct values are put in the data descriptor and
   in the central directory.  If an archive is in ZIP64 format
   and the value in this field is 0xFFFFFFFF, the size will be
   in the corresponding 8 byte ZIP64 extended information
   extra field.  When encrypting the central directory, if the
   local header is not in ZIP64 format and general purpose bit
   flag 13 is set indicating masking, the value stored for the
   uncompressed size in the Local Header will be zero.


Sink analyser: Fix index out of bound bug

Fix empty callpath index out of bound bug.
Signed-off-by: Arthur Chan<EMAIL_ADDRESS>

Issue #670


PBC with GDF density fitting: difference in energy between a cell with k-points and a corresponding supercell in PySCF v2.3.0

Hi.
I'm trying to compute total DFT energy for AlAs using GDF density fitting, and I get a significant different between a calculation using k-points and that of a corresponding supercell.
Additionally, there are two ways to ask for GDF:
Option 1: kmf = dft.KRKS(cell, kpts).density_fit()
Option 2: kmf.with_df = df.df.DF(cell, kpts)
Option 3: kmf.with_df = df.rsdf.RSGDF(cell, kpts)
Option 1 especially results in a rather dramatic difference in energies.
It seem to make little difference if I use range-separated or default GDF, i.e. Option 2 or 3.
Here is my minimal example, running through all three options:
import sys
import numpy as np
from pyscf.pbc import df, dft, gto
from pyscf.pbc.tools import super_cell

XC = 'pbe,pbe'
DF_TYPE = 'GDF'
PP = 'gth-pbe'
BASIS = 'gth-szv-molopt-sr'

# cell
cell = gto.Cell()
cell.atom = '''
 Al  0.000000  0.000000  0.000000
 As  6.081570  3.511190  2.482790
'''
cell.basis = BASIS
cell.pseudo = PP
cell.a = np.eye(3) * 4.05438
cell.a[1, 0], cell.a[2, 0] = 2.02719, 2.02719
cell.a[1, 1] = 3.51119
cell.a[2, 1] = 1.17040
cell.a[2, 2] = 3.31039
cell.verbose = 4
cell.build()

# kmesh
nk = [2,2,2]
nkpts = np.prod(nk)
kpts = cell.make_kpts(nk)

# build supercell
nk1 = np.array([1,1,1])
supcell = super_cell(cell, nk)
gamma_kpt = supcell.make_kpts(nk1)

for i in [1, 2, 3]:
    DF_OPT = i
    # run SCF calculation
    if DF_OPT == 1:
        kmf = dft.KRKS(cell, kpts).density_fit(auxbasis = "weigend")
    elif DF_OPT == 2:
        kmf = dft.KRKS(cell, kpts)
        kmf.with_df = df.df.DF(cell, kpts)
        kmf.with_df.auxbasis = "weigend"
    elif DF_OPT == 3:
        kmf = dft.KRKS(cell, kpts)
        kmf.with_df = df.rsdf.RSGDF(cell, kpts)
        kmf.with_df.auxbasis = "weigend"
    else:
        kmf = dft.KRKS(cell, kpts)
    kmf.xc = XC
    kmf.verbose = 4
    e_kmf = kmf.kernel()

    # run supercell SCF calculation
    if DF_OPT == 1:
        print('1 rks DFT_OP = ', DF_OPT)
        mf = dft.RKS(supcell, gamma_kpt).density_fit(auxbasis = "weigend")
    elif DF_OPT == 2:
        print('2 rks DFT_OP = ', DF_OPT)
        mf = dft.RKS(supcell, gamma_kpt)
        mf.with_df = df.df.DF(supcell, gamma_kpt)
        mf.with_df.auxbasis = "weigend"
    elif DF_OPT == 3:
        print('3 rks DFT_OP = ', DF_OPT)
        mf = dft.RKS(supcell, gamma_kpt)
        mf.with_df = df.rsdf.RSGDF(supcell, gamma_kpt)
        mf.with_df.auxbasis = "weigend"
    else:
        print('4 rks DFT_OP = ', DF_OPT)
        mf = dft.RKS(supcell, gamma_kpt)
    mf.xc = XC
    mf.verbose = 4
    e_mf = mf.kernel()

    print('DF_OPT = ', DF_OPT)
    print('E_kmf = ', e_kmf * nkpts)
    print('E_mf = ', e_mf)
    print('E_kmf - E_mf = ', e_kmf * nkpts - e_mf)

And here are the energies I get when running the minimal example script with PySCF v2.3.0:
(I named the energy computed with k-points E_kmf and that of a supercell E_mf)
AlAs_pyscf230_opt_1.out:E_kmf =  -66.53058616857375
AlAs_pyscf230_opt_1.out-E_mf =  -66.68625150929245
AlAs_pyscf230_opt_1.out:E_kmf - E_mf =  0.15566534071869853

AlAs_pyscf230_opt_2.out:E_kmf =  -66.53060046615727
AlAs_pyscf230_opt_2.out-E_mf =  -66.52820303055472
AlAs_pyscf230_opt_2.out:E_kmf - E_mf =  -0.0023974356025462384

AlAs_pyscf230_opt_3.out:E_kmf =  -66.53060046481129
AlAs_pyscf230_opt_3.out-E_mf =  -66.52818288663258
AlAs_pyscf230_opt_3.out:E_kmf - E_mf =  -0.002417578178707913

Using the previous version, PySCF v2.2.1, does not result in such difference, as long as Option 2 or 3 is chosen:
AlAs_pyscf221_opt_1.out:E_kmf =  -66.53057746274987
AlAs_pyscf221_opt_1.out-E_mf =  -66.68861566834636
AlAs_pyscf221_opt_1.out:E_kmf - E_mf =  0.1580382055964833

AlAs_pyscf221_opt_2.out:E_kmf =  -66.53059173124741
AlAs_pyscf221_opt_2.out-E_mf =  -66.5305917994923
AlAs_pyscf221_opt_2.out:E_kmf - E_mf =  6.824488707479759e-08

AlAs_pyscf221_opt_3.out:E_kmf =  -66.53059952672413
AlAs_pyscf221_opt_3.out-E_mf =  -66.53059960688681
AlAs_pyscf221_opt_3.out:E_kmf - E_mf =  8.01626782731546e-08

My suspicion is that it might have to do with the electron-nuclei term in the energy and pseudopotentials, as I don't get the same difference in energy when computing it for HF, a system with only s- and p-shells.
Input cell in this case is:
cell = gto.Cell()
cell.atom = '''
 H   0.81252   1.47613   2.81966
 H   1.18600   1.19690   0.22918
 F   0.11649   1.99653   3.20061
 F   1.88203   0.67651   0.61013
'''
cell.basis = BASIS
cell.pseudo = PP
cell.a = np.eye(3) * 2.78686
cell.a[1, 1] = 2.67303
cell.a[1, 0] = -0.78834
cell.a[2, 2] = 5.18096
cell.build()

What is the difference when asking for density fitting in the two different ways as in Option 1 and 2?
And why is there a difference in energies no matter how ones asks for GDF in the newest version?
I would really appreciate any insights/help.

By default, XC functionals of DFT in PBC are computed with uniform grids. In kmf.density_fit(), it changes the dft grids to Becke grids. This causes the difference between option 1 and 2. The inconsistency between 2 and 3 are caused by a lattice sum truncation bug for non-orthogonal cell in recent optimization. For a workaround, you can tighten the cell.precision value to e.g. 1e-10.

Thank you, the workaround seem to help.


Bounty display is off 1 $

Small discrepancy https://www.bountysource.com/issues/1712071-hotkey-enable-double-pixeling-for-all-mods says I get $5 but the GitHub integration plugin announced $6 and my account now has  $5.50 =)

---
Want to back this issue? **[Post a bounty on it!](https://www.bountysource.com/issues/1760270-bounty-display-is-off-1?utm_campaign=plugin&utm_content=tracker%2F47&utm_medium=issues&utm_source=github)** We accept bounties via [Bountysource](https://www.bountysource.com/?utm_campaign=plugin&utm_content=tracker%2F47&utm_medium=issues&utm_source=github).

Specifically, the rounding is wrong on the front end. e.g. The $5.50 bounty shows as $5 on the badge and the GitHub plugin, but incorrectly as $6 here: https://www.bountysource.com/issues/8755961-add-a-view-issue-on-bountysource-link-to-issues


Please add a tokens-used/remaining to your computer-use-demo UI

I've just hit up against:

RateLimitError: Error code: 429 - {'type': 'error', 'error': {'type': 'rate_limit_error', 'message': 'Number of request tokens has exceeded your daily rate limit (https://docs.anthropic.com/en/api/rate-limits); see the response headers for current usage. Please reduce the prompt length or the maximum tokens requested, or try again later. You may also contact sales at https://www.anthropic.com/contact-sales to discuss your options for a rate limit increase.'}}

This does not feel good. Suddenly out of the blue to get cut off for a whole day, like this.
It creates a feeling in the user that they have done something wrong and are being penalized. It's a terrible UX.
And computer-use-demo is very token-hungry. Many users will hit this limit.
If you could simply show the user how many tokens they have used, and how many remain, this would solve the problem, as far as this project is concerned. Then they can see if they are close to the limit, and they will not experience this negative feeling.
Please consider wiring this in; it's important for Anthropic in terms of customer-sentiment, and I would like to see this company succeed!
PS I think the backend team should consider some kind of exponential backoff as a user approaches their limit: "Approaching rate limit, you may try again after 1/2/4/8 minutes".
To suddenly cut a user off for a whole day without any warning, this is terrible for anyone that has formed any dependency of their workflow upon your services.

I agree there is not much empathy for humans ;)


terror zone

everything else works great but when i add ,terror_zone to the list of runs it is ignored completely

hi, is it "IGNORED" or does it close the entire bot? if i only want my char to run terror_zone the bot simply crashes

Mine does not crash I have put the terror_zone argument in the beginning
middle and end of the run list but it just skips it as if it's not there
On Fri., Mar. 29, 2024, 2:53 a.m. Sunnyweeb, @.***>
wrote:

hi, is it "IGNORED" or does it close the entire bot? if i only want my
char to run terror_zone the bot simply crashes
—
Reply to this email directly, view it on GitHub
https://github.com/hectorgimenez/koolo/issues/116#issuecomment-2026969196,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/BG7OB7MTBWUHA73S73TEO5LY2UTZXAVCNFSM6AAAAABFN4MITKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAMRWHE3DSMJZGY
.
You are receiving this because you authored the thread.Message ID:
@.***>

I'm not sure this feature has ever worked. Anyone had this one working?

no i downloaded the first version that added terror_zones and it crashed (it crashed if you only put terror_zone in run:), i think they fix that after other things, funny because isn't terror zone of the best things int he game due to exp and always 85 area?

@childofwar1980  are you still experiencing this? Feel free to close if not. Try the latest beta release if you haven’t already. It’s fixed working there.


Title: Shell game

Is your feature request related to a problem? Please describe.
A ball is hidden under one of three bowls, which are then shuffled. The player must find the bowl containing the ball and guess its position after the shuffle. It's a fun and challenging game of observation and memory.
Describe the solution you'd like.
The user needs to find the ball where it was kept in the three balls.
Describe alternatives you've considered.
NA
Add any other context or screenshots about the feature request here.
No response

@Tamilkumaran17 All the best 👍🏻

I'm excited about taking the lead on the 30DaysOfJavaScript project. With my proficiency in HTML, CSS, JavaScript, and ReactJS, I'm confident in my ability to drive significant progress on this initiative. Please assign me to this project via GSSOC.


[DO NOT MERGE] Saves 9640 bytes of FLASH

This will save 9640 bytes of FLASH. It is dependant on this breaking (*manageable) change. https://github.com/mavlink/c_library_v2/pull/2/commits/9b3d6b2b62bf9b4c727d8d84ef507a8dcd0995f1
Commit 797c52e50287a8734540db5b67350338b3d89bf5 should be dropped if the PR is taken in upstream mavlink.
*The mavlink_inplement. file MUST be included in one and only one source file (.c or .cpp)

@davids5 bloaty shows no difference in binary size:

http://ci.px4.io:8080/blue/organizations/jenkins/Firmware/detail/master_v2_lhf/1/pipeline/40/
Is it correct?

Rebased on current master

Oh that was the px4io output, jenkins first runs bloaty (+ variations) on the px4io first.
Here's the px4fmu-v2 output.

Ah right, thanks!
Nice work @davids5!

Thank you @bkueng
@dagar This just reinforces the Truth....I should not do any work before coffee :)

Was this all done manually on the generated output or is there an actual https://github.com/mavlink/mavlink change?

@dagar the changes are here https://github.com/mavlink/c_library_v2/pull/2  but I did not realize it was generated as @LorenzMeier let me know after the fact. So the changes need to be applied the source but it is a breaking change in that the mavlink_inplement.h file MUST be included in one and only one source file (.c or .cpp) in the application.

@davids5 Daniel brought this in.