rhel10: move os package sets to yaml files

This is my RFC PR for moving package sets outside Go into yaml.

The code is of a PoC quality. The purpose is to start the conversation. However, I believe that after minor changes (converting all RHEL versions, or adding code to limit the change to RHEL 10), it could be actually deployed to production (this would be a yolo thing, though).
All terms are not final. I used spec because it's not used anywhere else in the Image Builder stack so there should be no confusion.

What I did
Image spec format
I introduced a new yaml-based format called image spec. Spec is a simple format for defining image properties. It's yaml-based, supports inheritance, and deduplication across multiple distro versions.
A quick snippet showing rhel 10's qcow2:
includes:
  - ../common.yaml
  - ../../centos-10/generic/qcow2.yaml
spec:
  packages:
    - subscription-manager-cockpit

I hope that this snippet is self-explaining, RHEL 10 qcow2 is composed of a common RHEL 10 spec, a Centos 10 qcow2 spec, and an extra package.

The whole specification is in specs/README.md.
Conversion of Centos Stream 10 and RHEL 10 and to image spec
Os package sets of both CS10 and RHEL 10 are now fully converted  to image specs. There should not be any functional changes. The image specs are embedded using go:embed, so everything feels the same.
I might need help from @achilleas-k to verify that nothing really changed, the generated manifests have different UUIDs for reasons that I don't understand.
Overriding embedded image specs
Embedded image specs can be fully overriden. Just pass the spec directory via OSBUILD_IMAGES_IMAGE_SPECS_DIR, and the library won't use the internal bits.
Next steps
We can start moving more and more stuff into yaml. I think that the next piece could be base partition tables:
spec:
  packages:
    - "@core"
  exclude_packages:
    - wifi-drivers
  base_partition_table:
    type: gpt
    partitions:
      - size: "10 GiB"
      - mountpoint: /
      # and more

The ultimate goal is that the Go code doesn't know anything about what's e.g. an ami. The yaml files drive the available image types. There are some gaps in the current spec definition for this to happen, but we can iterate on them.

I might need help from @achilleas-k to verify that nothing really changed, the generated manifests have different UUIDs for reasons that I don't understand.

Have you tried setting OSBUILD_TESTING_RNG_SEED=1?

OSBUILD_TESTING_RNG_SEED=1

Thanks! That did the trick for centos, it turns out that the x86_64 image installer was missing grub2-pc, so I added a arch-specific file for it. :)
For RHEL, it's a bit of a mess, because the package set merging works slightly differently that in images, so the arrays are not sorted in the same way. Didn't we invent something for this for otk?

Very cool and very necessary. As for naming, specifications or definitions both sounds fine.
You've only taken the RHEL packages here but in the Fedora package sets we have a "lot" of conditionals such as only enabling certain packages for certain versions. How would we handle this with the YAML files here?

I think I'm leaning towards the term "image definitions". We always refered to the go files as definitions, so I think we can continue using that term, just switch the "driver" to yaml. So I think I would rename the specs directory to definitions, and the spec package to defparser.

You've only taken the RHEL packages here but in the Fedora package sets we have a "lot" of conditionals such as only enabling certain packages for certain versions. How would we handle this with the YAML files here?

I think that Fedora would benefit from less deduplication than RHEL. So, one directory == one version, no sharing between them. I think that our long-term dream was/is to extract these yamls into a separate repository, and use the traditional branching model.
So for now, I suggest we just have three dirs: fedora-40, fedora-41 and fedora-rawhide (or 42, dunno).
Later, when they are in a separate repo (and maybe shipped in a separate RPM?), we can either teach images how to load them from system, or keep it simple and just embed them using submodules:
[submodule "fedora-40"]
        path = definitions/fedora-40
        url = https://pagure.io/fedora-kickstarts.git
        branch = f40
[submodule "fedora-41"]
        path = definitions/fedora-41
        url = https://pagure.io/fedora-kickstarts.git
        branch = f41
[submodule "fedora-rawhide"]
        path = definitions/fedora-rawhide
        url = https://pagure.io/fedora-kickstarts.git
        branch = main

Note that for CentOS/RHEL, I think I would keep the deduplication (as it exists in the go code) as much as possible. We know that it works for us, and for RHEL we are basically the maintainers. We can always reconsider this later.
I don't like that I'm basically proposing different models for Fedora and RHEL, but I think it sorta makes sense given the differences between these two distros.

I'm a bit surprised there's no apparent comparisons or references to other extant image format definitions already used in the Fedora-derivative ecosystem? There are many:

https://github.com/minimization/content-resolver-input/ which many of us are forced to maintain (I think it has implicit includes?)
kickstart (also has file includes, package excludes, etc.)
rpm-ostree's tree file (also quite similar)
comps is also related
Finally of course also of high importance is spec files which many people are forced to understand and maintain anyways

The thing that drove me towards spec files (a conclusion which honestly I didn't want to get to) is documented here https://gitlab.com/fedora/bootc/base-images-experimental/-/blob/main/README-build-rationale.md?ref_type=heads#key-aspect-reusing-spec-files
That whole file has also a lot of related discussion.
Certainly getting packages out of code into a declarative file format is something that makes obvious sense to me, but introducing a new package-list-in-YAML/toml/XML/specfile that isn't one of the above seems like it needs some rationale. Every single time someone has introduced one it has just been a new file format only understood by one tool.
Picking just one is impossible, but my personal inclination is to try to have more tools accept multiple formats - that was the idea behind https://github.com/coreos/rpm-ostree/pull/5119 (and I also have no problem having it accept blueprints too, but honestly they're pretty verbose for this use case).

Certainly getting packages out of code into a declarative file format is something that makes obvious sense to me, but introducing a new package-list-in-YAML/toml/XML/specfile that isn't one of the above seems like it needs some rationale.

The rationale is that this is the first step to externalising our image definitions.  These are currently defined as a combination of:

package sets (e.g. https://github.com/osbuild/images/blob/7b36025a743b8408430bf19fbdb2644dd5e47a9f/pkg/distro/fedora/package_sets.go)
default partition tables (e.g. https://github.com/osbuild/images/blob/main/pkg/distro/fedora/partition_tables.go)
an ImageConfig (https://github.com/osbuild/images/blob/7b36025a743b8408430bf19fbdb2644dd5e47a9f/pkg/distro/image_config.go)
OSCustomizations (https://github.com/osbuild/images/blob/7b36025a743b8408430bf19fbdb2644dd5e47a9f/pkg/manifest/os.go)

Now there's clearly a lot of history (and technical debt) here.  Some of the things in these configuration structures can be simplified, squashed together, etc, but the fact is that they're our current state and our starting point for this process.
Regardless, what we expect to end up with is some transformation (simplification) of all of these into an internal structure that's useful for us, for everything we need.
otk was our fist big attempt at this (see for example everything under otk.define here https://github.com/osbuild/otk/blob/bf22601c1557cf2ff19638c6f2a6e328210dad18/example/centos/centos-9-aarch64-ami.yaml).
I don't think something like kickstart (or any of the other examples in your list, at least the ones I'm familiar with) is going to cut it.  Or if it can, then we'd need to start parsing kickstarts to make partition tables, or image configs, or os customizations, and that's not only a lot of work, it's also kinda pointless.  We're not going to be working with an internal representation of a kickstart.
What we're doing here is taking code structures and pulling them out (and simplifying, either during the extraction of after the fact).  We're not creating an end-user-facing configuration format, just making our lives easier in a way that will hopefully make it easier for contributors (and hopefully distro maintainers) to modify image configurations without needing to navigate (or write) code and without needing to recompile go binaries.
Oh and XML is out of the question :smile:

Of course, I think this is an improvement and obviously none of my comments should be construed as trying to block anything. But at the same time, we can and should think about how the evolution of this fits in with at least some of the other ones. In the course of needing to succeed at doing my job I have had to interact with (or at least think about) relatively recently with every single one of the above things, and this would just add a new one to the mix.

In the course of needing to succeed at doing my job I have had to interact with (or at least think about) relatively recently with every single one of the above things, and this would just add a new one to the mix.

I understand where you're coming from.  There's a long-running struggle to figure out the target audience for each configuration format and level.  The configuration formats you mentioned are, in my opinion, serving similar purposes to our blueprints and blueprints are definitely the weakest of all of those.
I think a conceptual problem here is that every one of the configuration formats you mentioned span the whole range of use cases from lightly customizing a base image configuration to distro/spin image configuration from scratch, and we always kept those two ends separate.  A big part of the value of image builder is that (for example) selecting an ami as your image type includes a lot of decisions for how RHEL should be run on aws.  That's the stuff we have in code and that's what we want to extract in these image definitions.  What you get with blueprints is enough knobs and switches to modify that base ami to fit your needs, but not so many that you can turn it into an Azure image, or worse, make it unusable in AWS.
Now, you're right, the other configs are serving a similar (or sometimes the same) purpose as what we're doing here: they let you define an image configuration pretty much from scratch, with some useful abstractions.  But a lot of what we're defining here is coupled with how osbuild works (to varying degrees).  Pulling in a configuration format from another project doesn't really serve any purpose.  We could write a parser that reads a kickstart's partitioning directives, or a Kiwi <partitions></partitions> block, to instantiate a disk.PartitionTable (which in turn gets converted to an osbuild pipeline that creates a disk image), but who would that be for?  What's the value in that when we can write some yaml that matches disk.PartitionTable for our internal image definitions (e.g. "rhel-9/ami-pt.yaml is the default partition table for for RHEL 9 AMIs"), which lets us and certain types of users modify those base image types at runtime, and provide blueprints for everyone else?  And what happens when we need to encode configurations that aren't covered by another config format?  I'm not saying we have unique image building requirements that no other build tool has come across, but we're talking about our internally-defined image configurations here.  It needs to cover everything we can do to differentiate one image from another.  More importantly, these image definitions will be required for populating the internal image configurations of image builder.  In other words, without a rhel-9/ami-pt.yaml there can be no RHEL 9 AMI; there is no autopart or similar without them.


refactor: rename to TS template

Edit README to change the name to TS
change default wasm location to dist/plugin.wasm

See https://github.com/extism/cli/pull/85


Exception with Windows 10 Anniversary 14393

An exception occurs when NavigationService navigates to a page.
The following exception is thrown:
There are no more endpoints available from the endpoint mapper.
It comes from pageState?.Clear(); line 167 of the NavigationService file (inside NavigateToAsync method).
It may not be related to Template10, but as far as I know... I don't know.
I used Template10 1.1.12-preview-160712

The error is There are no more endpoints available from the endpoint mapper? Yikes, a new error in AU. Will take a little bit to research. Thanks for reporting it.

I can confirm this exception is coming everywhere in many .NET dependencies... Totally not related to Template10.

Thank you @bbougot


WIP: User Settings should be implemented for web, with a few exceptions noted

on devops.
Notably, notification settings will not have the default values -- but
this is because the Synapse Server is responsible for updating matrix
clients with default notification values.

This PR is outdated and has changes that are no longer applicable, as they have been superseded by PR #4


Button with "\n" in text can't be found

From<EMAIL_ADDRESS>on October 27, 2013 10:37:20
What steps will reproduce the problem? 1. Have a app with a button with for example "test\ntest" as content text
2. try to find it using Solo with exact that text What is the expected output? What do you see instead? It should find the button, instead it times out and can't find it. What version of the product are you using? On what operating system? 4.3 Please provide any additional information below. Very frustrating, since I rely on selecting buttons by the string text resource.
Original issue: http://code.google.com/p/robotium/issues/detail?id=543

From<EMAIL_ADDRESS>on October 28, 2013 00:39:45
Thanks for reporting this. It will be fixed in the next release.
Status: Accepted
Owner<EMAIL_ADDRESS>

From<EMAIL_ADDRESS>on November 04, 2013 08:33:20
Thanks again for reporting this. It has been fixed in 4.3.1.
Status: Verified


Question

Where do I setup the bot token?

Replace the pastebin link "https://pastebin.com/raw/FVg1kwqF" with your pastebin link or just remove the whole thing, I used the pastebin link to read a random bot token everytime on startup just in case one is getting removed.


Question about caption evaluation

Hi,
Where could I find the pretrained weights of 4.1 or 4.2, I have tested the weights provided on huggingface by 4.3. However, they all returned errors as the image shown.

The error in the image indicates the weights are not properly loaded. This might be caused by different implementation of GPT2Attention in different versions of huggingface transformers. In my case (transformers==4.30.2), the weights for attention bias are always ignored. If you still encounter certain error after installing transformers==4.30.2, please try editing this line into:
model.load_state_dict(checkpoint["model"], strict=False)

to skip the loading of attention bias.
All the weights for Vote2Cap-DETR could be found under this folder.

Thank you. It works! But for weights of Vote2Cap-DETR++, it still has an error of
"RuntimeError: Error(s) in loading state_dict for CaptionNet:
size mismatch for captioner.transformer.transformer.wpe.weight: copying a param with shape torch.Size([128, 256]) from checkpoint, the shape in current model is torch.Size([64, 256])."
Moreover, for the caption case, when I look into the evaluation results of Vote2Cap-DETR and visualize it, it seems have much fewer boundingbox with caption than the ground truth. Is the result correct?

Sorry that I have not currently released these codes for the Vote2Cap-DETR++ model, thus the weights currently cannot be properly loaded.
As for the visualization, this is not usual. Please consider using the following commands:
python demo.py --use_color --use_normal --dataset scene_scanrefer --vocabulary scanrefer --use_beam_search --detector detector_Vote2Cap_DETR --captioner captioner_dcc --test_ckpt path_to_your_weight.pth

This will generate some .json files that can be used to visualize the predictions. You can use the tools in this repo to help visualize the box estimations.

Thank you so much!


New documentation on App Service debugging, and some docs organizational changes

Purpose
This pull request includes changes to the README.md and docs/ files to improve the documentation of the azure-search-openai-demo project. The most important changes include:

adding a brand new doc on debugging App Service deployments
reorganizing the sections in the README.md
moving the images to a new docs/images/ directory
turning the FAQs into docs
adding an index page for the additional docs

I'll add inline comments on relevant changes to explain motivation. Feedback welcome!
Does this introduce a breaking change?

[X] Yes - @mattgotteiner will need to update any links that point to the login doc since I moved it into docs for consistency, like in the blog post.
[ ] No

Pull Request Type
What kind of change does this Pull Request introduce?

[ ] Bugfix
[ ] Feature
[ ] Code style update (formatting, local variables)
[ ] Refactoring (no functional changes, no api changes)
[X] Documentation content changes
[ ] Other... Please describe:

How to Test

Click new or changed links

cc @charris-msft Some of this was inspired by user studies. Particularly my comment about potential future size reductions of the README.

@dfberry Curious if you have any feedback as well. And wanted to make sure this didnt break any links from your article (not sure if you link to any subheadings).

I'm not dependent on the docs in the repo for Learn content - I appreciate you letting me know though. The root Readme, from what I've seen in UX and metrics, should be pretty slim covering just how to use the thing. You can see that in some of the Azure SDK work for Identity.

@dfberry Thanks for confirming! That's what I suspected, our README is not what one could call pretty slim. Am curious what @charris-msft and @mattgotteiner think about my suggestion of moving the 'azd env set' stuff to separate docs.

@dfberry Thanks for confirming! That's what I suspected, our README is not what one could call pretty slim. Am curious what @charris-msft and @mattgotteiner think about my suggestion of moving the 'azd env set' stuff to separate docs.

After hearing some other feedback this week about the pudgy README, I'm 100% in favor of identifying portions of the content that can logically be moved into their own docs.

Merged recent changes from @mattgotteiner to login doc from main.


python310Packages.zha-quirks: 0.0.88 -> 0.0.89

Diff: https://github.com/zigpy/zha-device-handlers/compare/refs/tags/0.0.88...0.0.89
Changelog: https://github.com/zigpy/zha-device-handlers/releases/tag/0.0.89
Description of changes

Things done

Built on platform(s)

[ ] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

[ ] For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[ ] Tested basic functionality of all binary files (usually in ./result/bin/)
23.05 Release Notes (or backporting 22.11 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module
[ ] (Release notes changes) Ran nixos/doc/manual/md-to-db.sh to update generated release notes

[ ] Fits CONTRIBUTING.md.

Result of nixpkgs-review pr 205920 run on x86_64-linux 1

1 package failed to build:

home-assistant-component-tests.homeassistant_hardware

6 packages built:

home-assistant-component-tests.hassio
home-assistant-component-tests.homeassistant_sky_connect
home-assistant-component-tests.homeassistant_yellow
home-assistant-component-tests.zha
python310Packages.zha-quirks
python39Packages.zha-quirks

Failure home-assistant-component-tests.homeassistant_hardware, see https://github.com/NixOS/nixpkgs/pull/205692#issuecomment-1345707357


Set code push install mode to immediate

Yes, no native changes here. Let's see how it works, ferran mentioned it could be a blink while it installs. So let's see. If it does not look good, we can always remove it again


Add reaction messages to PIGS server

Refresh Roles:
Same effect as this message: https://discord.com/channels/447157938390433792/705179062347366513/705179916915834891

Also add reaction to add these roles:
Warthogs
572838338470346757
Edgy
498885132468486175
PIGS Channel ID
976981250516594718

This exists already
https://discord.com/channels/487285826544205845/705249536477495337/705249747950108752
You just never made it public


HIVE-24155. Upgrade Arrow version to 1.0.1

Hi @medb ,
I already attempted to upgrade Arrow version to 0.11.0, but faced the same test failure as you: https://issues.apache.org/jira/browse/HIVE-23987
Seems like somehow after the upgrade, the number of rows inserted is 26624, instead of the expected 16384. I don't know if we don't use Arrow properly, or there is a bug in Arrow itself from version 0.11.

@bmaidics thanks for heads up! I plan to take a close look into failing tests, but Arrow 0.11.0 has this fix: https://github.com/apache/arrow/pull/2594
It's possible that because of this fix tests writes all data now and that's why fails. But will need to check failing tests to understand what the right result should be.

@bmaidics I have run test queries on Hive 3.1.2 and they produce 16384 records as expected by test, so there must be something wrong with how Hive uses Arrow after upgrade.


Am I worthy?

idk, check my profile...
Maybe I'll join sometime.
(T_T)

idk, check my profile...
Maybe I'll join sometime.
(T_T)

OMG Really? I'd be honored! Sent you the invite! (Sorry for the late reply, I was very busy.)


Swapchain layer inserting invalid std::unordered_map entries

The swapchain layer calls std::unordered_map::operator[] to retrieve records associated with handles without first checking to see if an entry for the handle exists in the map. When no entry is present for the handle, operator[] inserts a new entry with a value-initialized record. This can lead to an invalid pointer dereference with Windows Debug builds, where the memory is initialized with the 0xcdcdcdcd debug pattern. The operator[] calls need to replaced with find() to avoid adding unwanted entries to the map, and the value returned by find() needs to be checked against end() to determine if it was present before it is used.

Can you give an example of how I can reproduce this, please?

You can reproduce it by providing any unrecognized or invalid handle, such as VK_NULL_HANDLE, to the swapchain layer's vkDestroySrufaceKHR function.  The VkLayerTest.EnableWsiBeforeUse test will currently reproduce this error on Windows.
To see what is happening you can call vkDestroySurfaceKHR(instance, 0 , 0) without calling any other WSI functions and set a break point on line 861 of the current code.  At break, you will see that my_data->surfaceMap has a size of 0.  After stepping over this line, you will see the size increase to 1.
Basically, you don't want to use operator[] to search for an item in a map, as it has the side effect of inserting new entries for values it does not find.

I just created pull request #482 (see: https://github.com/KhronosGroup/Vulkan-LoaderAndValidationLayers/pull/482).  It addresses this and other issues.

This issue has been fixed AFAIK, closing


Authenticate hook requests

Both Transifex and Github permit signing requests to prevent processing unauthorized payloads. This PR adds two new configuration keys, TX_WEBHOOK_SECRET and GITHUB_WEBHOOK_SECRET. If these keys are present, txgh will reject requests that are incorrectly signed. If they are not set, txgh won't perform any authentication, even if the incoming requests are signed.
@jdoconnor @st33b @zvkemp

@camertron I resolved the conflicts and pushed into devel here: https://github.com/transifex/txgh/commit/76f7a415d83048673196b792e17e20a1261a4b36
If you want to take a quick looksee before I merge to master and close this PR.

Ah, sorry about the conflicts. Looks good to me! :ship:


[Popover] Fix paper position flash on open

[x] I have followed (at least) the PR section of the contributing guide.

Issues

#33308
#33743 (probably)

Changes
Previously popovers could have been rendered before they were positioned, resulting in them briefly appearing in the top-left corner particularly on slower devices and with low transitionDuration. This PR hides the popover paper until positioning is complete.
Drawbacks
On slower devices, the opening animation may be completely skipped since the animation would complete before positioning would finish. This can be demonstrated with the Chrome dev tools' CPU throttling feature. Due to the current way that popover is designed, there is a sort of catch-22 in which the the paper only mounts once the transition begins, but positioning can only occur after the paper mounts. Changing other parts of the code to get around that issue is certainly possible, but I have not attempted to do that in this PR.
Resolves #33308

Thanks for your work. Your fix seems to work. I'm going to test it more thoroughly.
As for the drawback you mentioned, I think we can accept it.
BTW, the affected components, such as Select will be implemented using MUI Base in the future, which doesn't suffer from this issue AFAIK. This way the implementation will be simpler.


Yuqiang/ai league product admin

UI:

if addons

@mrfinch yeah update on save is much better than re-open modal. :joy: i can do that. maybe also update old prepaid logic.
good catch! i update the variable names but forget in pug file 😂
and for naming things, nick suggest such 🎃 so i just letft them to see if global team can understand it 😂  also i would create a doc to describe them detailly. yeah, confusing.
the addon.teams and addon.tournaments are total numbers, instead of extra numbers.


Convert video MP3 to audio MP4 frontend

Accepting only mp4

It's done on PR >> https://github.com/joaothallis/live_media/pull/8


Fix multi gpu map example

use orch.cuda.set_device instead of CUDA_VISIBLE_DEVICES
add if __name__ == "__main__"

fix https://github.com/huggingface/datasets/issues/6186

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint.

Merging this one, but lmk if you have more comments for subsequent improvements @NielsRogge

This is a little hard to follow — where is the documentation currently? I am trying to follow from here:
>>>
>>> for i in range(torch.cuda.device_count()):  # send model to every GPU
...     model.to(f"cuda:{i}")
>>>
>>> def gpu_computation(example, rank):
...     torch.cuda.set_device(f"cuda:{rank}")  # use one GPU
...     inputs = tokenizer(texts, truncation=True, return_tensors="pt").to(f"cuda:{rank}")
...     outputs = model(**inputs)
...      ....

but I'm getting device errors (data is on device 3, but it thinks model is on device 0, despite setting torch.cuda.set_device
Is this correct? What version of Torch are you using for this?

Anyway, this didn't work for me:
torch.cuda.set_device(f"cuda:{rank}")  # use one GPU
but this did:
model.to(f"cuda:{rank}")

Yeah for me this issue isn't resolved yet, we need a better code example

Hi @alex2awesome, could you open a PR with your suggestion to improve this code snippet ?

i'm happy to when i get it fully working, but i feel like there are some fundamentals that I'm not fully understanding...
I've set it up twice now, for 2 GPU-processing pipelines.
In one pipelines, my memory usage is fine, it delivers me a huge speedup, and everything is great. In the second pipeline, I keep getting OOM errors when num_proc > 1 that I don't get when num_proc=1.
There is a discussion here: https://github.com/pytorch/pytorch/issues/44156 about CUDA memory leaks in multiprocessing setups, and I haven't had the time to fully read the source code to datasets.map to understand whether the situations are parallel.
I haven't fully tested out enough to see what the issue is but if I feel comfortable over the next several days to generate a slimmed-down example that I know will generalize to the real-world cases I'm working with now, then I will contribute it.

@lhoestq do you know how datasets does multiprocessing? Do we use:
https://pytorch.org/docs/stable/multiprocessing.html#module-torch.multiprocessing?
If so, there are lots of points around memory usage, here:
https://pytorch.org/docs/stable/notes/multiprocessing.html

After some more research and playing around, I can't pinpoint the source of my CUDA memory leak and I'm not really an expert on multiprocessing in general, but my gut is that the current set-up isn't ideal for multiprocessing.
Are either of you more experienced in this?

Not sure about your GPU's OOM :/
Still, I opened a PR with your suggestion here: https://github.com/huggingface/datasets/pull/6550

I still get only 0 rank...
Here is my code: https://pastebin.com/c6du8jaM
from this ^ i just improt one function:
from test import map_train
from multiprocess import set_start_method

set_start_method("spawn")
map_train()

And here is the traceback:
https://pastebin.com/YijspwQK

Also this code from your docs is not valid (source: https://huggingface.co/docs/datasets/main/en/process#multiprocessing):
for i in range(torch.cuda.device_count()):
    model.to(f"cuda:{i}")

This for me sends the model only to the second GPU
vae = AutoencoderKL.from_pretrained(
    pretrained_model_name_or_path, subfolder="vae"
)
vae.to("cuda:0")
vae.to("cuda:1")

Could you please provide a working example of multi-GPU mapping?
Not just an example in docs, but a real working example starting from all imports and datasets.

@alex2awesome the same issue with CUDA OOM. It should not be happening, since it should 2 different GPUs be handling different loads. But in fact something wrong is happening.

I haven't experimented much with the multi-GPU code documentation (see discussion at #6550)
Though if you have some fixes/improvements ideas feel free to open a PR !

@lhoestq the mapping does not start at all in this case:

@lhoestq with this code: https://pastebin.com/muDm78kp
i now getting this error:
Map (num_proc=2):   1%|       | 26288/3043663 [06:11<11:51:08, 70.72 examples/s]
Traceback (most recent call last):
  File "/workspace/compute.py", line 229, in <module>
    map_train()
  File "/workspace/compute.py", line 224, in map_train
    return train_dataset.map(compute_embeddings_fn, batched=True, batch_size=16, with_rank=True, num_proc=2, keep_in_memory=True)
  File "/usr/local/lib/python3.10/dist-packages/datasets/arrow_dataset.py", line 593, in wrapper
    out: Union["Dataset", "DatasetDict"] = func(self, *args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/datasets/arrow_dataset.py", line 558, in wrapper
    out: Union["Dataset", "DatasetDict"] = func(self, *args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/datasets/arrow_dataset.py", line 3193, in map
    for rank, done, content in iflatmap_unordered(
  File "/usr/local/lib/python3.10/dist-packages/datasets/utils/py_utils.py", line 658, in iflatmap_unordered
    raise RuntimeError(
RuntimeError: One of the subprocesses has abruptly died during map operation.To debug the error, disable multiprocessing.

Also when trying to download my dataset there were no issues from one machine, but from another:
SSLError: (MaxRetryError("HTTPSConnectionPool(host='huggingface.co', port=443): Max retries exceeded with url: /api/datasets/kopyl/3M_icons_monochrome_only_no_captioning/revision/753dca4be462dad7022f34cc273555ab6deb5832 (Caused by SSLError(SSLEOFError(8, '[SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol (_ssl.c:1007)')))"), '(Request ID: 7d0881f3-1b93-4d73-bcb6-52e816d84529)')

Can't download my dataset at all...

Hmm this is not good, do you know a way to make it work ?
Basically map creates two subprocesses and runs the function in the subprocesses. Since each function has a parameter rank it should be possible to choose which GPU to use

I can confirm that PR #6550 works. All GPUs are at full throttle. You have to manually move the model to all GPUs.

I haven't experimented much with the multi-GPU code documentation.
Can you try using the code example at #6550 instead ? That would be super helpful if you could confirm that it works on your side
Though if you have some fixes/improvements ideas feel free to open a PR !

I wrote a blog post with a complete example by compiling information from several PRs and issues here. Hope it can help. Let me know how it works.

Could you please provide a working example of multi-GPU mapping?
Not just an example in docs, but a real working example starting from all imports loading datasets and models.


Keeping PageView index

Hi
I have a PageView to show 4 pages and a BottomNavigationBar to switch them. I'm having troubles with the PageController.
In any page, I navigate to a new Screen, but when get back, the PageView return to the initial page, no to the last page.
Here is how I using the PageView
https://gist.github.com/pblinux/e158151f4d8216acdbd482b2ba3ec8c4
In the DashboardScreen for example, I navigate to a new screen like:
Navigator.of(context)             .push(MaterialPageRoute(builder: (context) => CodeScreen()))
This behaviour is normal? I need to use something like Redux or Bloc Pattern to achieve/fix this?
Thanks.

Please consider asking support questions in one of the other channels listed at http://flutter.io/support .


Support for multiple armor headers in ArmoredOutputStream

Currently ArmoredInputStream is able to handle multiple headers under the same name but ArmoredOutputStream is not able to create them. For example any additional "Comment" header overwrites the current one.
They are explicitly allowed by RFC 4880 in its section 6.2. Forming ASCII Armor.

Thanks for the patch. Merged!


🐞 [BUG] - Review PR

Description
https://github.com/amupedia2021/Project-Amupedia/pull/52
Current behaviour
Some issue! You have to find out what causing the issue
Expected behaviour
successfully build
Reproduction steps
npm run build

Screenshots
No response
Logs
No response
Browsers
Chrome
OS
Mac
Are you contributing for a program ?
GSSoC2023
Code of Conduct

[X] I have read and agree to the Contributing Guidelines of this project.

level 3


Let execute.if.predicate accept the parameters of Predicate

Like how execute.run accepts the parameters of MCFunction. It would make checks with predicates that are only ever used once much less verbose.
I am iffy on this suggestion though, because there are a lot of commands that could receive the same treatment but don't (for example, the item command could allow the creation of ItemModifiers via its parameters too), and I'm not confident this is a good design pattern for anything beyond functions. More opinions (or deeper consideration) would be needed.

All places where a Resource or /variables utility can be turned into a simple value it should be supported, because Sandstone is supposed to be a rolling abstraction precompiler where you can choose the amount of features you want to use. This will be done alongside other missing support in a future release.


Dependabot/cargo/bech32 0.7.2

Motivation
Trying to get this dep updated.   Want the compilation error visible to folks who know more than I about rust.
Have you read the [Contributing Guidelines on pull requests]
yes
Test Plan
Successful compilation and test.
Related PRs
none.

@metajack I decided to have conversion functions on the type for to/from bech32 Strings.

@bors-libra delegate=rexhoffman
This looks fine. Did you find out from the relevant module owners whether this code is needed?

:v: @rexhoffman can now approve this pull request

@metajack looking in to that now.   Wanted to be able to drive this to completion (lints/checks/type changes and all first).
@davidiw, @msmouse and @huitseeker have touched a bit of the account_number and may know the answer.   Any compelling reasons to not remove Bech32 support from account_number?  It's bitcoin like encoding we added in over 8 months ago that doesn't appear to be used outside of out tests.    The changes I made in this pr are the only usages.   The crate in question (Bech32) is here: https://docs.rs/crate/bech32/0.7.2

Also in the client we use hex for account number representation.

Kill it with fire. I see no point in having this unnecessary representation and library. I had wanted to explore it myself but in my unrelated journeys through the code have found no purpose to it.

Closing in favor of Bech32's removal.   https://github.com/libra/libra/pull/2619
Still glad I worked through this.


Dec'22 WP3.7 (UKD-UKD-S6) Quarterly update/plan

IPC form available at  https://ls.st/ikc-qu
copy and paste their corresponding DESC report into the IPC form if necessary
include under achievements that deliverables have been approved by Recipient Groups.
review LSST:UK Risk Register
ideally by 09/DEC/22

05/JAN/23

sent reminder

11/JAN/23

sugata emailed with a copy of the submission

30/MAY/23

At 19/MAY/23 Team meeting, Aprajita said if no feedback then assume  all is well with the report, hence ticket is closed.


[Feature Request] Plase add option to make requests only if in card screen.

Hello, Package make requests even when it's in another screen. I think it's better if only make requests in screen the cards are in use.
In my case, i only use it in main dashboard, but when i go to any resource page, requests still happens.

I have the same problem

I have the same problem

I think this is fine... but... It's make unnecessary requests and this can be improved!

I was able to reproduce this issue. I'll fix this soon

Hello. Any updates on that?

I'm currently quite busy and I haven't found the time to fix this issue yet. A PR is welcome

https://github.com/GijsGoudzwaard/nova-system-resources/pull/25
Hope this helps.

@wemersonrv
for now you can use my current master, till it's merged;
"require": {
        ...
        "gijsg/system-resources": "dev-master",
        ...
    },

    "repositories": {
        ...
        "gijsg/system-resources": {
            "type": "vcs",
            "url": "https://github.com/authanram/nova-system-resources.git"
        }
    }

Good luck.

@wemersonrv
For now you can use my current master, till it's merged;
    "require": {
        ...
        "gijsg/system-resources": "dev-master",
        ...
    },

    "repositories": {
        ...
        "gijsg/system-resources": {
            "type": "vcs",
            "url": "https://github.com/authanram/nova-system-resources.git"
        }
    }

Good luck.

Gread job!

This issue is fixed in #25. It can be used with version 1.0.4.


Update options to babylon

@hzoo Is this right?
I also updated my fork of babylon just in case. Let me know when there's a new published version and I'll switch to it.

Thanks!

Is this patch part of a current release? I'm currently getting errors while trying to use import, which I see this patch would fix via dynamicImport, I believe.

Yes we've had several releases since then. If you are on 0.11.0, please open an issue!


Implement onRequest versions of createContract, placeBet functions

Function status: these replacement functions are currently deployed on dev for testing, and not yet on production.
This enables these functions to serve future public API requests. Future public APIs will be converted in a similar fashion when desired.
The design of this change is to be backwards-compatible with the existing onCall machinery, so deploying these new versions of the functions will require no client changes. We will break compatibility to make necessary improvements (e.g. real HTTP status codes) in future steps. The most important thing about the new versions is that they now accept API keys (see #182) as an alternative authentication mechanism to the Firebase JWTs. No intentional changes were made to the function behavior except related to authentication.

If you've tested this sufficiently and everything works, feel free to merge it in.


columns property should throw console.warn in case of invalid syntax

The current behavior is silently fall back to default value without any warnings.
(This is the DX tests finding)

@platosha this was done in https://github.com/vaadin/vaadin-form-layout/pull/38 right?

@limonte correct.


add warning when non wgs84 geographic crs and optional wgs84 default

closes #165
This PR does:

add _geographic_crs private attribute (again)
add 2 env variable control:

MORECANTILE_IGNORE_NON_WGS84_GEOGRAPHIC: silent warning when geographic CRS is not WGS84
MORECANTILE_WGS84_GEOGRAPHIC: make WGS84 the default geographic CRS

@AndrewAnnex any opinions?

@vincentsarago for this PR since the TMS spec doesn't require WGS84, I feel like warnings related to WGS84 not being used should only be isolated to areas that actually may require it like GeoJSON in #165 (although I will comment there also, I don't like how RFC7946 is used in practice.. but more on that later in #165 comments). Looking at all the new warnings you need to configure to ignore in the pyproject.toml seems to confirm that to me.
Also this is just a bit of "explicit is better than implicit", like if someone uses NZTM2000Quad they know what they are asking for.
That said warnings are 10e10 more preferable to throwing exceptions. I just think it might be better to have the environment variables be a "opt-in" rather than an "opt-out".
For 99% of users, not issuing warnings when they wouldn't occur anyways is fine, for the 1% that know they aren't using WGS84 in their requests a warning about it doesn't help them. But if someone wants to be warned then I think they can enable to config.


I am having trouble getting the custom field value for "feature.jiraEpicIdFieldName". I ran the rest api against a jira issue that has super issue as EPIC. Please help here.

I am having trouble getting the custom field value for "feature.jiraEpicIdFieldName". I ran the rest api against a jira issue that has super issue as EPIC. Please help here.

@amp5208 - can you please take a look?

@roykaushik07 , Thank you for reaching out regarding the jiraEpicIdFieldName.  To get this field name, you have to perform the following API actions manually against your Jira instance, using a sample Jira issue that has a super issue as Epic, as you have indicated:
# In Jira, your instance will have its own custom field created for "super story" or "epic" back-end ID,
# which includes a list of information.  This field allows you to specify that data field for your instance
# of Jira.  Note:  You can retrieve your instance's epic ID field name via the following URI where your
# queried user story issue has a super issue (e.g., epic) tied to it; your custom field name describes the
# epic value you expect to see, and is the only field that does this for a given issue:
# https://[your-jira-domain-name]/rest/api/2/issue/[some-issue-name]
feature.jiraEpicIdFieldName=customfield_10400

Essentially, you have to use the Jira API to look up an existing issue that has a super issue tied to it, then Ctrl-F on the output for that super issue reference; the resulting field's name (usually starting with "customfield_") is the value that needs to be placed on this property.  This field ID is different for every installation of Jira, and different on most regions, given your internal Jira deployment plan.
The documentation above can be found as comments in the embedded properties file sample.  Let me know if you are still having this issue after trying to acquire the epic field name using the above instructions.  Note:  This question was also raised and answered in the following comment within an open GitHub issue; linking for clarity:  https://github.com/capitalone/Hygieia/issues/276#issuecomment-172060522

@roykaushik07  Better copy the result of the above mentioned URL into some json editor & navigate/search. You can easily find the value.

@roykaushik07 , @smsuresh  is correct:  That is the best option for retrieving the value at this time:  Save the output from the Jira response to a JSON file, use a JSON linter to format or "beautify" the JSON, and look for the field name and value pair.

Closed. Reopen if there are any problems. Use latest codebase and follow cleanup instructions provided in https://github.com/capitalone/Hygieia/pull/1137.


Update minikube version to 1.21

This pull request changes minikube version to 1.21 to address earlier issues with deploying kapp-controller in CI: https://github.com/vmware-tanzu/carvel-kapp-controller/runs/2956127108?check_suite_focus=true

Closing in favor of #258
Because there are issues with Test_PackageRepoDelete, this should be added to pr with those changes.


--output=json does not print json format

Hi. When I try to get the project id in json or yaml format, the output is the interactive table, not json
$ litmusctl get projects --output=json
$ litmusctl get projects --output json
$ litmusctl get projects -o=json
$ litmusctl get projects -o json

PROJECT ID                                      PROJECT NAME            CREATED AT
1234567-89ab-cdef-12345-6789abcdefab            admin-project           123-45-67 89:01:23 +0000 UTC
✔ Press Enter to show the next page (or type 'q' to quit): █

How can I print json format? I tried with litmusctl 1.1.0 and 1.2.0.
What I'm trying to do is to automate the creation of a chaos-environment with litmusctl, but it asks for a project-id first.
$ litmusctl create chaos-environment --name=my-environment --type=cluster --project-id=XXXXX

@braiamdev I think the code currently does not process the output flag but rather try to prompt the user for the output. This should be a quick fix.
outputFormat, _ := cmd.Flags().GetString("output")
@SarthakJain26 @neelanjan00


Update: WPSnapshots local copy

Description of the Change
Due to the new flag introduced in https://github.com/10up/wpsnapshots/pull/71, when running WPAcceptance locally against a cached environment, the process will be hung with the prompt asking if we want to overwrite local snapshot or not.
This PR fixes that bug by adding the flag to the command called by wpacceptance.

Verification Process

Checklist:

[x] I have read the CONTRIBUTING document.
[x] My code follows the code style of this project.
[ ] My change requires a change to the documentation.
[ ] I have updated the documentation accordingly.
[ ] I have added tests to cover my change.
[ ] All new and existing tests passed.

@dinhtungdu while testing things for ElasticPress I had a similar problem but when we don't have the snapshot already downloaded. It will wait for the user to answer if it should download files and/or database but as nothing is inputted it just times out.
I've tested adding --include_files --include_db to that command and that seemed to do the trick.
Thoughts? Do you think we can include these flags in this PR too? Thanks!

@dinhtungdu while testing things for ElasticPress I had a similar problem but when we don't have the snapshot already downloaded. It will wait for the user to answer if it should download files and/or database but as nothing is inputted it just times out.
I've tested adding --include_files --include_db to that command and that seemed to do the trick.
Thoughts? Do you think we can include these flags in this PR too? Thanks!


Non-pixel font sizes aren't converted correctly

Relates to https://github.com/jpuri/html-to-draftjs/issues/46
Per the CSS spec, font-size can take absolute and relative values, as well as global values, e.g:
font-size: medium;
font-size: larger;
font-size: inherit;
This library currently assumes all font sizes are in pixels, and therefore just adds 'px' to the end of the stored font-size.
Steps to reproduce:

Go to https://jpuri.github.io/react-draft-wysiwyg/#/demo
On the first demo (which shows you the outputted html), copy the text which is there, then paste.

Result:

The outputted html has font-size: medipx.
I am working on a fix for this and hope to get a PR open shortly.

Thanks for the PR @mjgreen145


xen: 4.15 -> 4.18

Description of changes
Required by #324911.

Updates Xen to 4.18.2 on nixpkgs/master.
Marks a couple of packages as broken, as they are outdated, and their older versions do not seem to support Xen 4.18.

Things done

Built on platform(s)

[x] x86_64-linux

Built with sandboxing enabled.

[x] sandbox = true

[x] Tested, as applicable:

No automated tests for pkgs.xen yet.

[x] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD".
[x] Tested basic functionality of all binary files (usually in ./result/bin/)
[x] Fits CONTRIBUTING.md.

Closes #320335, closes #26899.
Add a :+1: reaction to pull requests you find important.

Draft for now, as I still need to fix the fixupPhase to also strip/patchelf the binaries under $out/lib/xen/bin/*
Not doing this means that the systemd service that calls those binaries will fail.

Fixed the unpatched binaries.
Keep in mind XKCD 1513 as you review.

Result of nixpkgs-review pr 324693 run on x86_64-linux 1

2 packages marked as broken and skipped:

libvmi
qubes-core-vchan-xen

1 package blacklisted:

nixos-install-tools

17 packages built:

ocamlPackages.cohttp-mirage
ocamlPackages.conduit-mirage
ocamlPackages.irmin-mirage-git
ocamlPackages.mirage-bootvar-xen
ocamlPackages.mirage-net-xen
ocamlPackages.mirage-xen
ocamlPackages.netchannel
ocamlPackages.vchan
ocamlPackages.xenstore
ocamlPackages.xenstore-tool
ocamlPackages.xenstore_transport
qemu_xen (qemu_xen_4_18)
qemu_xen.debug (qemu_xen_4_18.debug)
qemu_xen.ga (qemu_xen_4_18.ga)
xen
xen-guest-agent
xen-slim

Would you be willing to cherry-pick the maintainers and ocamlPackages.xenstore commits to a separate PR and add me as reviewer?
Then we can try to merge bits and pieces of this so that only the gnarly stuff that I'm not comfortable with is left :smile:

Would you be willing to cherry-pick the maintainers and ocamlPackages.xenstore commits to a separate PR and add me as reviewer?
Then we can try to merge bits and pieces of this so that only the gnarly stuff that I'm not comfortable with is left 😄

Hi @ulrikstrid! The OCaml stuff is now on #324913. Can I remove reviewers from this PR?

Removed my maintainer commit as it was already added in #324913.

@SigmaSquadron You seem to be re-using a 3 year old patches, are you sure that they are still needed?

Yes. According to the documentation, ATA-DMA is still not a default, and iPXE will never use our git pre-fetched source, as this is a necessary upstream deviation. If you'll look at the full diff, you'll see that many patches were removed or rewritten into subsituteInPlace calls.Arguably the SeaBIOS patch could be rewritten into a substituteInPlace call that the build fails if it can't replace. Would that be preferable?On 9 Jul 2024, at 8:54 pm, KREYREN @.***> wrote:
@SigmaSquadron You seem to be re-using a 3 year old patches, are you sure that they are still needed?
—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you were mentioned.Message ID: @.***>

Also, uh, OfBorg has been trying to build Xen on aarch64-darwin for over a day now. I assumed it wouldn't do that, as Xen does not compile on anything but x86_64-linux. Should I do something to skip the darwin builds so this PR doesn't needlessly waste CI resources?

Also, uh, OfBorg has been trying to build Xen on aarch64-darwin for over a day now. I assumed it wouldn't do that, as Xen does not compile on anything but x86_64-linux. Should I do something to skip the darwin builds so this PR doesn't needlessly waste CI resources?

Is the meta.platforms set?

Patched for XSA #458: Double unlock in x86 guest IRQ handling. (https://xenbits.xenproject.org/xsa/advisory-458.html)
XSA #459 affects XAPI, not Xen.

Patched for XSA #458: Double unlock in x86 guest IRQ handling. (xenbits.xenproject.org/xsa/advisory-458.html) XSA #459 affects XAPI, not Xen. -- @SigmaSquadron (https://github.com/NixOS/nixpkgs/pull/324693#issuecomment-2230918601)

These # references seems to link to an unrelated issues on GitHub btw

About these references to security issues and that xen is used as a critical part of a system i would prefer someone from Xen to review this merge request to add Reviewed-by in case we missed something.

These # references seems to link to an unrelated issues on GitHub btw

I'm currently in the middle of nowhere, and I don't have a computer that has my GitHub credentials, so I can only use e-mail and Git. I forgot that the hash referenced in-repo issues.
If you can, would you mark this comment as off-topic as to not clutter the thread for code reviews?

About these references to security issues and that xen is used as a critical part of a system i would prefer someone from Xen to review this merge request to add Reviewed-by in case we missed something.
I don't know if the Xen people are knowledgeable enough with Nix to review the "gnarly" parts of this PR. Still, it's important to note that the Xen developers assisted me through many choices in this update, including using efi-binutils for the entire build, selecting XSA patches, and enabling non-built-in iPXE.

I could ask on their Matrix or the Xen mailing lists if anyone would like to review this, but I'd rather not bother upstream with a downstream packaging of Xen unless absolutely necessary. Although old, the Xen Wiki has much of the information needed to package the hypervisor, and I feel most of the issues with these PRs lie in my limited knowledge of the Nix language.
Also, it should also be noted that Nixpkgs is in no position to be security-critical right now. After a lengthy discussion in the Security room, I was informed that there is simply no way to quickly patch software in Nixpkgs, as none of the work here can be secret, and all Hydra builds are public and updates must be uploaded to the channels automatically. Maybe when the governance issues end, and the Xen module is a bit more modernised, we can work with the Security team to get NixOS on the Xen pre-disclosure list and get security patches merged and pre-built in a special "secret" Hydra.
Anyways, do you feel that it is really necessary to ask for upstream input here?

Also, it should also be noted that Nixpkgs is in no position to be security-critical right now. After a lengthy discussion in the Security room, I was informed that there is simply no way to quickly patch software in Nixpkgs, as none of the work here can be secret, and all Hydra builds are public and updates must be uploaded to the channels automatically. Maybe when the governance issues end, and the Xen module is a bit more modernised, we can work with the Security team to get NixOS on the Xen pre-disclosure list and get security patches merged and pre-built in a special "secret" Hydra. -- @SigmaSquadron (https://github.com/NixOS/nixpkgs/pull/324693#issuecomment-2234274527)

I feel that is a missunderstanding of how nixpkgs work.. Nixpkgs seems to in general have a very fast response to security issues (often faster than other distros) which is why [Nixium]{https://github.com/NiXium-org/NiXIum) is currently using that in tasks that are considered mission-critical on high-security environment.
It's true that it might take X time for packages to be updated, but security is prioritized and afaik worst case you if the cache can't be built fast enough you will have to compile the package yourself.
Refer to real life example of management major vulnerabilty: https://discourse.nixos.org/t/security-advisory-openssh-cve-2024-6387-regresshion-update-your-servers-asap/48220 including the timeline of the response https://discourse.nixos.org/t/security-advisory-openssh-cve-2024-6387-regresshion-update-your-servers-asap/48220/19

special "secret" Hydra. -- @SigmaSquadron (https://github.com/NixOS/nixpkgs/pull/324693#issuecomment-2234274527)

I would consider that a malware for my infra, if you want to have faster updates then you can implement the package and work on your overlay like ocalica does with rust in https://github.com/oxalica/rust-overlay, but i treat it as less secure due to less strict QA and only use that in devshells when i really need specific rust version faster and don't think that it's a good idea for Xen.
--
So please don't treat nixpkgs are inherently insecure and treat the Xen Security more seriously.. Reading the Xen source code and security recommendations from upstream there seems to be a potential for us to miss something critical so i would prefer Xen upstream involved.

I don't know if the Xen people are knowledgeable enough with Nix to review the "gnarly" parts of this PR. Still, it's important to note that the Xen developers assisted me through many choices in this update, including using efi-binutils for the entire build, selecting XSA patches, and enabling non-built-in iPXE. -- @SigmaSquadron (https://github.com/NixOS/nixpkgs/pull/324693#issuecomment-2234274527)

In terms of Nix i don't think it's difficult language to read and understand (never seemed to have been an issue for other devs for my things as it's basically just shell scripting with JSON around it that is unimportant), but it is a difficult language to write and master.. So i don't think that the language is a barrier if needs be it can be explained to them what we are doing and why.

I feel that is a missunderstanding of how nixpkgs work.. Nixpkgs seems to in general have a very fast response to security issues (often faster than other distros) which is why [Nixium]{https://github.com/NiXium-org/NiXIum) is currently using that in tasks that are considered mission-critical on high-security environment.
It's true that it might take X time for packages to be updated, but security is prioritized and afaik worst case you if the cache can't be built fast enough you will have to compile the package yourself.
So please don't treat nixpkgs are inherently insecure [...]

Nixpkgs is not inherently insecure, but as you stated, we have no way to upload security fixes without waiting for a channel rebuild. Security fixes are indeed merged remarkably quickly, but the issue is in the time between PR merged and channel updated. As there is no way to respond to security issues without waiting for the embargo to end and produce a PR with a security fix, Nixpkgs lags behind other distros who have access to pre-disclosure lists and can provide pre-built security patches when the embargo ends.

[...] and treat the Xen Security more seriously.. Reading the Xen source code and security recommendations from upstream there seems to be a potential for us to miss something critical so i would prefer Xen upstream involved.

While security is paramount for Xen, I'm not sure this is feasible. As I stated, improved security workflows would require major changes to the current Nixpkgs packaging process, and the best we can do is patch this derivation as soon as the embargo ends.
I'm still not convinced how the upstream developers can help us here, though. I feel like we'd be wasting their time.

In terms of Nix i don't think it's difficult language to read and understand (never seemed to have been an issue for other devs for my things as it's basically just shell scripting with JSON around it that is unimportant), but it is a difficult language to write and master.. So i don't think that the language is a barrier if needs be it can be explained to them what we are doing and why.

But why exactly would they want to be taught Nix? Packaging is historically a downstream concern, and this Xen does follow all of the security recommendations that were documented, as far as I'm aware. Most of the Xen developers are employees of big companies paid to work on the hypervisor, and I doubt they'd be willing to review PRs for us; at least until we become a major distro for Xen dom0s.

As there is no way to respond to security issues without waiting for the embargo to end and produce a PR with a security fix, -- @SigmaSquadron (https://github.com/NixOS/nixpkgs/pull/324693#issuecomment-2234371351)

There is, trusted community members have access to security briefings before the embargo on these is lifted which is handled on invidual bases through the package upstreams e.g. pre-disclosure lists as you mentioned.

would require major changes to the current Nixpkgs packaging process,  -- @SigmaSquadron (https://github.com/NixOS/nixpkgs/pull/324693#issuecomment-2234371351)

NixOS in general seems very flexible for major changes to be implemented, infrastructure is code and open-source and RFC is usually the way for distribution-wide changes, but i would suggest discussing this on the forums, though i currently don't see any issue from those described to be actionable and feel that you are missunderstanding the nixpkgs's management.

I feel like we'd be wasting their time. -- @SigmaSquadron (https://github.com/NixOS/nixpkgs/pull/324693#issuecomment-2234371351)

I would prefer the wasted time over missing a major/critical vulnerability and deploying that to potentially thousands of users.

But why exactly would they want to be taught Nix? -- @SigmaSquadron (https://github.com/NixOS/nixpkgs/pull/324693#issuecomment-2234371351)

I don't expect them to learn Nix i expect them to look at the changes to see what is it doing, they don't really need to know Nix to be able to tell what the merge request is doing, i don't expect them to be an expert on Nix to be able to understand potential problems with the implementation of Nix what i care about is if we are missing some critical patches and configuration and why is it important that we do them.

at least until we become a major distro for Xen dom0s. -- @SigmaSquadron (https://github.com/NixOS/nixpkgs/pull/324693#issuecomment-2234371351)

Securing downstream implementation especially one that potentially affects significant amount of Xen users as NixOS is a major distribution is almost always part of the job description for devs who get paid by major companies to work on software.. at least from my experience working on mission critical software, some are assholes who will refuse to help but Open-Source developers like to cooperate together as it quickly becames very disfunctional otherwise.

There is, trusted community members have access to security briefings before the embargo on these is lifted which is handled on invidual bases through the package upstreams e.g. pre-disclosure lists as you mentioned.

Source? I was told that NixOS has been denied access to pre-disclosure lists, such as OpenWall's distros list, as we have no way to build security patches in secret.

NixOS in general seems very flexible for major changes to be implemented, infrastructure is code and open-source and RFC is usually the way for distribution-wide changes, but i would suggest discussing this on the forums, though i currently don't see any issue from those described to be actionable and feel that you are missunderstanding the nixpkgs's management.

I was told that the current governance issues make it effectively impossible to make any sort of "official" major change. To do something as complex as set up a private build server and rewrite the channel update system to accept these pre-built security patches when the embargo ends would be impossible without an "official" undertaking from NixOS, and the way I understood it, nothing of the sort can be done until the constitutional assembly finishes their work and the security team prepares the infrastructure required to get on the distros list.

I would prefer the wasted time over missing a major/critical vulnerability and deploying that to potentially thousands of users.
I don't expect them to learn Nix i expect them to look at the changes to see what is it doing, they don't really need to know Nix to be able to tell what the merge request is doing, i don't expect them to be an expert on Nix to be able to understand potential problems with the implementation of Nix what i care about is if we are missing some critical patches and configuration and why is it important that we do them.

I feel like you're misunderstanding the Xen packaging process. It's hard to build an insecure Xen; just use the latest release and check the dates for any XSA patches that were announced between the last git tag and $CURRENT_DATE. I truly doubt there is a critical vulnerability in a simple package build derivation.
I still feel no need to bother them, but if you would like, you can send a message in their mailing lists or matrix rooms, asking for help. I personally believe that this derivation follows all documented upstream recommendations, however.

Securing downstream implementation especially one that potentially affects significant amount of Xen users as NixOS is a major distribution is almost always part of the job description for devs who get paid by major companies to work on software.. at least from my experience working on mission critical software, some are assholes who will refuse to help but Open-Source developers like to cooperate together as it quickly becames very disfunctional otherwise.

Uh, no, it most certainly is not part of their job description. Any work they do towards NixOS/nixpkgs comes from their own good will; they will not be paid for their volunteer help here, as it isn't part of their jobs.
Also, I'd like to ask you to avoid using profanities; the Xen people dislike it. Let us keep this a professional environment.

I hope our discussion hasn't derailed into incivility. If you feel I have been disrespectful, I'd like to assure you it was not my intention.

Source? I was told that NixOS has been denied access to pre-disclosure lists, such as OpenWall's distros list, as we have no way to build security patches in secret. -- @SigmaSquadron (https://github.com/NixOS/nixpkgs/pull/324693#issuecomment-2234496411)

Afaik this is not something given distribution-wide it's given to few trusted members who can make these patches in secret and deploy them appropriately e.g. my briefings for NiXium are often along the lines of "update/reconfigure package to version X, because it contains undisclosed critical vulnerability"
e.g. in example that doesn't reflect reality: Instead of saying "Fix for undisclosed vulnerability that enables root elevation in OpenSSH" is terrible for obvious reasons, but we can quite easily just bump the version or disguise the patch as something else, afaik this already happens for many packages
The issue that you may have wrongly understood is that everyone has full access to the derivations as everyone can easily reproduce them, rather the issue is often with package upstream as their management doesn't allow for a transparent way to handle this scenario in which case it's their bug as which only exposes attack vector for vulnerability like xz backdoor was.

I feel like you're misunderstanding the Xen packaging process. It's hard to build an insecure Xen; just use the latest release and check the dates for any XSA patches that were announced between the last git tag and $CURRENT_DATE. I truly doubt there is a critical vulnerability in a simple package build derivation. -- @SigmaSquadron (https://github.com/NixOS/nixpkgs/pull/324693#issuecomment-2234496411)

I am aware of that, i am tracking xen packaging in https://github.com/NiXium-org/NiXium/issues/27 which is why i am so involved in this merge request as when i was trying to package it, the correct configuration seemed like a major problem to get right.
As e.g. for packaging SNMP (protocol used to do event-driven triggers to monitor infrastructure) solution has a (imho) realistic risk of configuring the protocol in a way that makes the system root elevatable by everyone with snmpwalk installed as explained in https://youtu.be/-2yzXSIuC8o?t=446
Such scenario is to my knowledge not considered a bug as it's basically just a pebcak where from my talks with Qubes devs they seem to do a lot of management on top of Xen to make sure that Xen is deployed in a sane way that i wasn't yet able to fully research and understand which is what concerns me here.

Uh, no, it most certainly is not part of their job description. Any work they do towards NixOS/nixpkgs comes from their own good will; they will not be paid for their volunteer help here, as it isn't part of their jobs. -- @SigmaSquadron (https://github.com/NixOS/nixpkgs/pull/324693#issuecomment-2234496411)

From my professional experience and talks in our tech park with devs who are in similar or same situation (paid by these big corporations to work on open-source) it's almost always part of the job and clearly written in the paperwork and i often get paid for similar work.
This is my experience in central EU however, it's possible that this doesn't reflect the rest of the world as e.g. i am aware of this being less common in the US.

I hope our discussion hasn't derailed into incivility. If you feel I have been disrespectful, I'd like to assure you it was not my intention. -- @SigmaSquadron (https://github.com/NixOS/nixpkgs/pull/324693#issuecomment-2234496411)

I don't feel like it has, you rather felt to me as a guy either coming from proprietary development or freshman judging by your concern to not bother other devs (and your new GitHub Profile), for context i do lot of mentoring for open-source development, programming and engineering and such people are often this concerned about what i see as legitimate cooperation with other community members) to mention it with the intention to reaffirm that it's okay to ask people for help e.g. this exchange might be a good example to better explain to work environment https://github.com/NiXium-org/NiXium/pull/78 where the flake-parts developer was very helpful with me finding a better implementation for the functionality.

For Xen-based configuration Xen is a critical component which wrong configuration and management could expose the user to a critical vulnerability, me and merge request author highlighted few potential problems in how to properly package this that should be discussed.
CC NixOS Security Team for instructions how to manage the security of this package: @risicle @LeSuisse @mweinelt

[...]
for instructions how to manage the security of this package

I'm not sure if there is something really specific to it requiring to manage in a different way than usual.
Given the inherent complexity of something like xen it would however be wise if a team of people would take care of its maintenance in nixpkgs in a collaborative way. Having a single maintainer will likely lead to same result than today: something unmaintained, not secure and broken.
The discussion about getting getting information/patches before the public disclosure of an advisory is wider than "only" Xen and I'm not sure to see the point being made with this sentence:

Also, it should also be noted that Nixpkgs is in no position to be security-critical right now. After a lengthy discussion in the Security room, I was informed that there is simply no way to quickly patch software in Nixpkgs [...]

This may or may not be true depending on how you handle your systems and your specific threat model.
In the specific case of Xen being on their disclosure ML is unlikely to yield a significant advantage: the number of rebuilds involved in a Xen upgrade is not that big so it does not need to go through staging. The timing between the moment the PR is merged and the content is available in the binary caches should be quite reasonable. It also means it is also possible for an individual to apply the patch directly without needing a large amount of compute power to rebuild it.
Outside of the specific case of Xen:

no, NixOS does not have a representative on the linux-distros ML. Note that the representives are listed on this page https://oss-security.openwall.org/wiki/mailing-lists/distros (and membership requests are posted on the OSS Security ML)
yes, it is possible that sometimes individuals in the community get a heads up before the disclosure of embargoed information (because of their work, personal contacts...)

On the PR itself, Xen is not really in knowledge area so I'm not going to be really useful. However:

It would be nice if in the fetchpatch used in xsaPatch to set the name argument to something including the CVE ID. It helps vulnerability scanners, SBOM tool generators and similar to check if a publicly known vulnerability has been patched or not.
This upgrade should probably be mentioned in the 24.11 release notes.
Getting feedback from upstream when/if needed does not hurt if they are willing to do it.

On the PR itself, Xen is not really in knowledge area so I'm not going to be really useful. However:
It would be nice if in the fetchpatch used in xsaPatch to set the name argument to something including the CVE ID. It helps vulnerability scanners, SBOM tool generators and similar to check if a publicly known vulnerability has been patched or not.
The name attribute is already used in the URL that contains the patch, and not all Xen Security Advisories have CVEs assigned to them. Also, one XSA may contain multiple patches. See https://xenbits.xen.org/xsa/advisory-206.html, which contains numerous different .patch files, and no CVE assigned to any of them. On the other hand, there is https://xenbits.xen.org/xsa/advisory-218.html, which contains 3-4 patches per release, but only two assigned CVEs.
The ideal solution here would be if these vulnerabilities scanners were patched to identify XSA numbers as well. Failing that, how could the CVE information be exposed for XSAs that do include them?
This upgrade should probably be mentioned in the 24.11 release notes.
#324911 updates the NixOS module, and adds a release note.
Getting feedback from upstream when/if needed does not hurt if they are willing to do it.
My question is if upstream feedback is even needed here. I don't see how they can help at this point in time, and they have already assisted me with many parts of this PR that would have required their attention.

I have removed your assignment to this PR because you are not reviewing your own PR.
I'm not a good reviewer for this PR, because Xen is not in my knowledge area.
Still, I can provide basic/general feedback, to speed things up for when the right reviewer shows up. To advance possible objections. Like, I'd have used nixfmt and done by-name migration. (In a separate commit.)
I did have a look into proposed code. And I don't see a problem in it itself. Again, I don't know Xen specifics. I mean, package-wise. For specific Xen review, I suggest you to add other reviewers, for that, check git history and try people that have touched the files. In case you can't find reviewers, I think, we should go forward still.
Package quality is a consequence of maintainers. Ecosystems start small and grow as it gets polishment, then new maintainers show up to carry on the good deeds.
On security, having active maintainers (again) is the critical part. Building, having patches merged, is the easy part.
Individuals can cherry-pick fixes to a fork/overlay if urgently necessary. But there is no workaround to building, building will happen anyway. Individuals or a specific group of maintainers interested in a topic can receive heads up on matters.
I think, the feel you have is that there aren't many active Xen maintainers atm. And you are certainly one, but then, where are your peers to review your Xen proposals? So your reviews lag and you don't get specific feedback. (That's a negative feedback loop.)
I have a good experience maintaining K3s here. At first, it was only a single maintainer (euank), and now there are 6 active maintainers. As the package/project got investment, there is a positive feedback loop. I hope you are able to reverse the Xen trend here. You may be rekindling a trend here.

I guess the only appropriate reviewers for Xen PRs would be the former Xen maintainers. Would it be appropriate to add them as reviewers?

I guess the only appropriate reviewers for Xen PRs would be the former Xen maintainers. Would it be appropriate to add them as reviewers?

In practice, anyone can review this PR. Xen maintainers have to listed for review. People that touched files is a good guess when maintainers don't answer. Yes, it is appropriate to add them. In case, they don't want to review they can ignore or remove themselves from being a maintainer. A maintainer is to be notified of changes.

In this case, there's @radhus and @oxij. I think Johansson is no longer a part of NixOS/nixpkgs, however.

In this case, there's @radhus and @oxij. I think Johansson is no longer a part of NixOS/nixpkgs, however.

I can try to help out with reviewing, I just don’t have time to maintain Xen :)

Xen 4.19 will be releasing on the 29th. Once it releases, I'll be updating this PR to 4.19 instead, leaving 4.18 available as pkgs.xen_4_18.

Xen 4.19 will be releasing on the 29th. Once it releases, I'll be updating this PR to 4.19 instead, leaving 4.18 available as pkgs.xen_4_18.  -- @SigmaSquadron (https://github.com/NixOS/nixpkgs/pull/324693#issuecomment-2241347083)

Please wait at least 2 weeks before pushing 4.19 in stable release.

Please wait at least 2 weeks before pushing 4.19 in stable release in case major problems are found after release that need management.
For context i am trying to manage the xen deployment in NiXium's mission critical environment to quickly switch via disko to non-xen setup in case there are issues and this would be a welcomed management, ideally longer period between updates so that stable lags behind.

Use pkgs.xenPackages.xen_4_18 instead of pkgs.xen. I will continue to maintain 4.18 until it reaches EOL. Once your systems are ready, upgrade by changing your package to pkgs.xenPackages.xen_4_19. When in doubt, check the meta.branch attribute; only the main pkgs.xen will change branches without warning.

Out of respect to other users who need a bleeding-edge Xen, I will not be postponing major updates to the main package, as these pinned versions are available. -- @SigmaSquadron (https://github.com/NixOS/nixpkgs/pull/324693#issuecomment-2241781530)

Those are on the unstable branch, stable is meant to lag behind

Out of respect to other users who need a bleeding-edge Xen, I will not be postponing major updates to the main package, as these pinned versions are available. -- @SigmaSquadron (#324693 (comment))

Those are on the unstable branch, stable is meant to lag behind

I assume 24.11 will get 4.19. Once 4.20 releases, 4.18 and 4.19 will continue to receive security updates through the backports system, but 24.11 (stable) will not get 4.20.
If you're on stable, then you can leave pkgs.xen in your config, as it will only update to a breaking version at the next NixOS release. (and I will include release notes when that happens)

Out of respect to other users who need a bleeding-edge Xen, I will not be postponing major updates to the main package, as these pinned versions are available. -- @SigmaSquadron (#324693 (comment))

Those are on the unstable branch, stable is meant to lag behind

I assume 24.11 will get 4.19. Once 4.20 releases, 4.18 and 4.19 will continue to receive security updates through the backports system, but 24.11 (stable) will not get 4.20. If you're on stable, then you can leave pkgs.xen in your config, as it will only update to a breaking version at the next NixOS release. (and I will include release notes when that happens) -- @SigmaSquadron (https://github.com/NixOS/nixpkgs/pull/324693#issuecomment-2241785238)

1 version behind lag feels optimal, thanks

Update:

Merges Qubes patches.
Builds with systemdMinimal instead of the full package.
Re-enables hardening features.
Rewrites flags in bash scripts to use their long variants.
Disables Flask.
Renames 4.18/4.18.nix to 4.18/default.nix per package naming guidelines.
Improves the XSA fetchPatch wrapper to include meta attributes.

Update:

Adds XSA patching information to longDescription.

The full description now looks something like this:

Standard version of Xen. Uses forks of QEMU, SeaBIOS, OVMF and iPXE provided
by the Xen Project. This provides the vanilla Xen experince, but wastes space
and build time. A typical NixOS setup that runs lots of VMs will usually need
to build two different versions of QEMU when using this Xen derivation (one
fork and upstream).
Includes:

ipxe: Xen's pinned version of iPXE.
ovmf: Xen's mirror of OVMF.
seabios: Xen's mirror of SeaBIOS.
qemu-xen: Xen's fork of upstream QEMU.

This Xen has been patched against the following known security vulnerabilities:

XSA #458: Double unlock in x86 guest IRQ handling.

An optional feature of PCI MSI called "Multiple Message" allows a device
to use multiple consecutive interrupt vectors.  Unlike for MSI-X, the
setting up of these consecutive vectors needs to happen all in one go.
In this handling an error path could be taken in different situations,
with or without a particular lock held. This error path wrongly releases
the lock even when it is not currently held.

Fixes:

CVE-2024-31143

Disables Flask. -- @SigmaSquadron (https://github.com/NixOS/nixpkgs/pull/324693#issuecomment-2241840991)

Please elaborate in the code why is this disabled so that it doesn't get turn on by mistake or in an unsane way.

@SigmaSquadron is there a functional way to deploy this on a NixOS host to test this or do we need to wait for Qubes tools?

@SigmaSquadron is there a functional way to deploy this on a NixOS host to test this or do we need to wait for Qubes tools?

Wait for https://github.com/NixOS/nixpkgs/pull/324911.

Disables Flask. -- @SigmaSquadron (#324693 (comment))

Please elaborate in the code why is this disabled so that it doesn't get turn on by mistake or in an unsane way.

It's not there. The flag simply is not in the code. Why would I elaborate on something that isn't there to begin with?

@SigmaSquadron is there a functional way to deploy this on a NixOS host to test this or do we need to wait for Qubes tools?

Wait for #324911. -- @SigmaSquadron (https://github.com/NixOS/nixpkgs/pull/324693#issuecomment-2242495116)

In that case could you implement that as a part of this merge request so that we are not merging blind code that just builds in nixpkgs?

@SigmaSquadron is there a functional way to deploy this on a NixOS host to test this or do we need to wait for Qubes tools?

Wait for #324911. -- @SigmaSquadron (#324693 (comment))

In that case could you implement that as a part of this merge request so that we are not merging blind code that just builds in nixpkgs?

No. These commits were purposefully cherry-picked to allow for an easier time for reviewers of the package and reviewers of the module.

@SigmaSquadron is there a functional way to deploy this on a NixOS host to test this or do we need to wait for Qubes tools?

Wait for #324911. -- @SigmaSquadron (#324693 (comment))

In that case could you implement that as a part of this merge request so that we are not merging blind code that just builds in nixpkgs?

No. These commits were purposefully cherry-picked to allow for an easier time for reviewers of the package and reviewers of the module. -- @SigmaSquadron (https://github.com/NixOS/nixpkgs/pull/324693#issuecomment-2242499326)

How is that easier if i can't test that on a real system?

Factored PRs that can be reviewed separately are a good thing. Please don’t bog down @SigmaSquadron’s excellent work here with counterproductive nitpicks.

How is that easier if i can't test that on a real system?

Branch-off Nixpkgs master.
Copy the commits from this PR.
Copy the commits from the module PR.
Optionally, avoid copying the libvmi commit from this PR, and copy the commits from #328873
Profit.

Factored PRs that can be reviewed separately are a good thing. Please don’t bog down @SigmaSquadron’s excellent work here with counterproductive nitpicks. -- @emilazy (https://github.com/NixOS/nixpkgs/pull/324693#issuecomment-2242501805)

I don't want to bog down their work, but Xen's considered critical component of NiXium for me to want to test it in a controlled environment and ideally address any found issues.

How is that easier if i can't test that on a real system?

1. Branch-off Nixpkgs master.

2. Copy the commits from this PR.

3. Copy the commits from the module PR.

4. Optionally, avoid copying the libvmi commit from this PR, and copy the commits from [WIP: libvmi: 0.12.0 -> 0.14.0-unstable-2024-02-05, libkvmi: init at 1.1.0-unstable-2023-12-13 #328873](https://github.com/NixOS/nixpkgs/pull/328873)

5. Profit. -- @SigmaSquadron (https://github.com/NixOS/nixpkgs/pull/324693#issuecomment-2242506025)

That works for me :+1:

This is the NixOS tracker. NiXium is your responsibility.

This is the NixOS tracker. NiXium is your responsibility. -- @emilazy (https://github.com/NixOS/nixpkgs/pull/324693#issuecomment-2242513956)

Common interest in getting functional and sane xen applies.

Further generalised generic.nix. Updating is trivial now.
A side-effect of this change is that two rec calls were removed.

bumping this pr

Fails to build with:

fatal: clone of 'openssl/openssl' into submodule path '/nix/store/r7w8kfvxml6igdzaf01ksibzrjm15d61-ovmf/CryptoPkg/Library/OpensslLib/openssl' failed

Refer to NiXium-org/NiXium#121 (comment) for details
EDIT: it doesn't seem to be reproducible, no idea what it's caused by as my network (including multi-layer backup) didn't report any packet lost, unstable upstream? Build still fails.

I haven't gotten that, but I did catch a few more build failures with OVMF. I was already working on an update yesterday to build not only 4.18 reliably, but also 4.16 and 4.17 with minimal long-term error on my part.

Added a README, the other two supported Xen versions, and an update script. You'll notice that the only differences between the three versions is their git revisions.

Paging:
@matdibu for your acknowledgement of the changes to xen-guest-agent.
@radhus and @oxij as the former Xen maintainers.
@alyssais, as your insight is valuable here as a kernel maintainer and someone familiar with Xen through Qubes.

The changes to xen-guest-agent look good 👍🏻

Result of nixpkgs-review pr 324693 run on x86_64-linux 1

2 packages marked as broken and skipped:

libvmi
qubes-core-vchan-xen

34 packages built:

qemu_xen (qemu_xen_4_18)
qemu_xen.debug (qemu_xen_4_18.debug)
qemu_xen.ga (qemu_xen_4_18.ga)
qemu_xen_4_16
qemu_xen_4_16.debug
qemu_xen_4_16.ga
qemu_xen_4_17
qemu_xen_4_17.debug
qemu_xen_4_17.ga
xen
xen-guest-agent
xen-slim
xen-slim.boot
xen-slim.dev
xen-slim.man
xen.boot
xen.dev
xen.man
xenPackages.xen_4_16
xenPackages.xen_4_16-slim
xenPackages.xen_4_16-slim.boot
xenPackages.xen_4_16-slim.dev
xenPackages.xen_4_16-slim.man
xenPackages.xen_4_16.boot
xenPackages.xen_4_16.dev
xenPackages.xen_4_16.man
xenPackages.xen_4_17
xenPackages.xen_4_17-slim
xenPackages.xen_4_17-slim.boot
xenPackages.xen_4_17-slim.dev
xenPackages.xen_4_17-slim.man
xenPackages.xen_4_17.boot
xenPackages.xen_4_17.dev
xenPackages.xen_4_17.man

@alyssais, as your insight is valuable here as a kernel maintainer and someone familiar with Xen through Qubes.

I'm afraid this vastly overestimates my familiarity with Xen.  I think the only thing I can really say is, do we really need all these different versions?

@alyssais, as your insight is valuable here as a kernel maintainer and someone familiar with Xen through Qubes.

I'm afraid this vastly overestimates my familiarity with Xen. I think the only thing I can really say is, do we really need all these different versions?

Well, there's no real difference between having them all or just the latest. They're useful for future work regarding downstreams, such as building Qubes' patched Xen, which is pinned to 4.17.4 instead of >4.18. Additionally, some people prefer to use an earlier version of the package for stability reasons.

I can maintain them all with no issues, thanks to the generic.nix builder and a surplus of machines. -- @SigmaSquadron (https://github.com/NixOS/nixpkgs/pull/324693#issuecomment-2254472710)

Btw. They said that they can't implement the AArch64 version as they lack such system to test the runtime on. I am not sure if NixOS can provide such thing, but i will be deploying AArch64 to NiXium and plan to provide that to them or implement the support myself.

Yes, though I have also said I don't have a personal stake in AArch64 Xen, so I have no real interest in supporting it. If others have interest in Xen for ARM machines, they can maintain the ARM infrastructure themselves.

This will currently fail to build due to #331107.

Is there anything else that blocks review for merge?

So far, nobody who reviewed feels comfortable enough with Xen to approve this PR, so the fact this has no approvals is a blocker.

So far, nobody who reviewed feels comfortable enough with Xen to approve this PR, so the fact this has no approvals is a possible blocker. -- @SigmaSquadron (https://github.com/NixOS/nixpkgs/pull/324693#issuecomment-2258813678)

I meant implementation-wise like is there anything you still feel like needs to be handled or once the referenced issue is solved do you feel like it's ready for merge?

Yes, I do believe this is ready to merge, but we have an issue where there is an abundance of reviewers who are knowledgeable about Nix, and an abundance of reviewers that are knowledgeable about Xen, but basically no active reviewers who are knowledgeable about both. -- @SigmaSquadron (https://github.com/NixOS/nixpkgs/pull/324693#issuecomment-2258968521)

I don't think that is an issue, rather the blocking issue seems to be security considering the nature of Xen, but that (to me at least) seems like is sufficiently managed for NixOS standards.
That said i would prefer ~3 days of delay so that people who are concerned with Xen security can test it more, but my main concern now is just the 4,19 release that will stay in unstable anyway so that seems like good management to me.

@andyhhp Considering your reply in https://github.com/NixOS/nixpkgs/pull/324693#discussion_r1697463308 are you planing to co-maintain this package to address thomas's concern to make the package more reliable.
Optionally i am willing to help with maintenance if none's against it as NiXium's projected to use Xen on majority of it's systems.

As there aren't any automated dom0 tests, if the code looks fine from both a Xen and Nixpkgs perspective, and you're able to manually test dom0 with some domU's, shouldn't we just consider merging and let any issues be fixed in upcoming PRs -- @radhus (https://github.com/NixOS/nixpkgs/pull/324693#pullrequestreview-2208542471)

Yea, the lack of automated tests is kinda a problem too for a production environment, but in terms of management it's probably better to merge in the current state and add them later as this is already getting kinda complex to review and i am not sure what should we be checking in the nixos context.

Yea, the lack of automated tests is kinda a problem too for a production environment, but in terms of management it's probably better to merge in the current state and add them later

In my opinion, it's next to impossible to add more tests, at least without rewriting the Nixpkgs test infrastructure. Xen requires a bare-metal host to be a dom0 in, which isn't the case with NixOS tests, much less package tests

@andyhhp Considering your reply in #324693 (comment) are you planing to co-maintain this package to address thomas's concern to make the package more reliable?

Sorry, I'm an upstream Xen developer, and can advise on that side of things, but I don't have time (or indeed the expertise) to advise on the NixOS side of things.

Yea, the lack of automated tests is kinda a problem too for a production environment, but in terms of management it's probably better to merge in the current state and add them later

In my opinion, it's next to impossible to add more tests, at least without rewriting the Nixpkgs test infrastructure. Xen requires a bare-metal host to be a dom0 in, which isn't the case with NixOS tests, much less package tests -- @SigmaSquadron (https://github.com/NixOS/nixpkgs/pull/324693#issuecomment-2259120816)

That sounds like an over-reaction, the infra should have the tools needed, but it needs to b researched to figure out how to implement them else RFC

Sorry, I'm an upstream Xen developer, and can advise on that side of things, but I don't have time (or indeed the expertise) to advise on the NixOS side of things. -- @andyhhp (https://github.com/NixOS/nixpkgs/pull/324693#issuecomment-2259124047)

My apologies i confused you with nix developer and realized my mistake shortly after.

That sounds like an over-reaction, the infra should have the tools needed, but it needs to b researched to figure out how to implement them else RFC

Xen requires a bare-metal host to run. It's not a question of tooling, it's a question of not running tests inside containers/VMs, which isn't supported by Nixpkgs, and would require a tremendous amount of resources to effectively implement.

Dropped the dependency that wasn't compiling. I doubt anyone would actually use it.
Dropped unused dependencies per an upstream review.
Now passes nixpkgs-hammering xen -e missing-patch-comment -e explicit-phases.

Hopefully the final nixpkgs-review.
Result of nixpkgs-review pr 324693 run on x86_64-linux 1

2 packages marked as broken and skipped:

libvmi
qubes-core-vchan-xen

45 packages built:

qemu_xen (qemu_xen_4_19)
qemu_xen.debug (qemu_xen_4_19.debug)
qemu_xen.ga (qemu_xen_4_19.ga)
qemu_xen_4_16
qemu_xen_4_16.debug
qemu_xen_4_16.ga
qemu_xen_4_17
qemu_xen_4_17.debug
qemu_xen_4_17.ga
qemu_xen_4_18
qemu_xen_4_18.debug
qemu_xen_4_18.ga
xen
xen-guest-agent
xen-slim
xen-slim.boot
xen-slim.dev
xen-slim.man
xen.boot
xen.dev
xen.man
xenPackages.xen_4_16
xenPackages.xen_4_16-slim
xenPackages.xen_4_16-slim.boot
xenPackages.xen_4_16-slim.dev
xenPackages.xen_4_16-slim.man
xenPackages.xen_4_16.boot
xenPackages.xen_4_16.dev
xenPackages.xen_4_16.man
xenPackages.xen_4_17
xenPackages.xen_4_17-slim
xenPackages.xen_4_17-slim.boot
xenPackages.xen_4_17-slim.dev
xenPackages.xen_4_17-slim.man
xenPackages.xen_4_17.boot
xenPackages.xen_4_17.dev
xenPackages.xen_4_17.man
xenPackages.xen_4_18
xenPackages.xen_4_18-slim
xenPackages.xen_4_18-slim.boot
xenPackages.xen_4_18-slim.dev
xenPackages.xen_4_18-slim.man
xenPackages.xen_4_18.boot
xenPackages.xen_4_18.dev
xenPackages.xen_4_18.man

Congrats on this huge achievement!


Issue with loading sound- Uncaught DOMException: Failed to read the 'responseText' ...

Loading sound file with Hexi seems like did not work for me!
Trying to load a sound file in WAV format like this:
let thingsToLoad = ["sounds/music.wav"];
And caught up this error message in console:
"Uncaught DOMException: Failed to read the 'responseText' property from 'XMLHttpRequest': The value is only accessible if the object's 'responseType' is '' or 'text' (was 'arraybuffer'). at Resource._xhrOnLoad (file:///C:/Users/User/Desktop/Hexi%20-%20Copy/library/hexi.js:6041:65)"
Same with MP3/OGG format. Example 20_sounds does not work..caught with same error.. Is glad to know anyone can help me out with this..?

Did you try using an MP3 instead of a WAV file?
I had a similar problem when using a  .wav file, also make sure that the sounds directory is accessible and normally you access a directory using
_let thingsToLoad = ["./sounds/music.wav"]_

Is the error happening when your loading your assets? Does it work properly without loading any sound files?

Try this code inside your script and check your console to see if it's showing the files trying to load :)
let log = console.log;
let error = console.error;

let music

//An array of files you want to load
let assets = [
               "./sounds/music.wav"
             ];

let canvasWidth = 900
let canvasHeight = 900

//Initialize and start Hexi
let g = hexi(canvasWidth, canvasHeight, setup, assets, load);
g.fps = 30;
g.scaleToWindow();
g.start();
g.backgroundColor = "black"

function load(){
    // load assets
   log(`Loading ${g.loadingFile}`);
   log(`Progress ${g.loadingProgress}`);
   g.loadingBar();
}

function setup(){
   music = g.sound("./sounds/music.wav")

}

function play(){
  music.play()
}
let log = console.log;
let error = console.error;

let music

//An array of files you want to load
let assets = [
               "./sounds/music.wav"
             ];

let canvasWidth = 900
let canvasHeight = 900

//Initialize and start Hexi
let g = hexi(canvasWidth, canvasHeight, setup, assets, load);
g.fps = 30;
g.scaleToWindow();
g.start();
g.backgroundColor = "black"

function load(){
    // load assets
}

function setup(){
   music = g.sound("./sounds/music.wav")

}

function play(){
  music.play()
}

Same with zero progress.. Btw my code works fine without sound..

I'm assuming it does the same with any sound file, not just this one. So on that assumption, and looking at the error code it looks like that the file is not getting recognized as a string or array when it xhrOnLoad's. As long as you have full permissions on your system(admin, root) it should work.
If you want you can send me a link to the file your using and I can test it on my end.

@jrfoxw Thanks so much for your help with this!


Delayed Logins

Hello, I started using Travertine today and I noticed it now takes around 7-10 seconds to log in compared to 0-2 when using hexacord.
I'd like to continue to use Travertine to increase performance but I can't if player logins are delayed by around 10 seconds each time.
Thanks for reading, have a great rest of your day.

Issue was resolved on discord and was down to the async netty dns resolver


Compare certificate validity period correctly when the machine is using a time zone other than UTC

When comparing two DateTimes, the Kind attribute is ignored, so comparing an UTC DateTime and a local DateTime do not give the expected result.
While the validity period is encoded as UTC in the certificate itself, it is converted to local time before it is returned in the X509Certificate2.NotBefore / X509Certificate.NotAfter attributes. This means that if we want to check whether the certificate is currently valid, we should compare it with the current local time.
Fixes: #185

Really sneaky bug, good catch 🐛


lagging or slow

How can i make it smooth like native. it is very slow or lagging. how can disable animation ?

Same problem. any solution?

Same problem. any solution?

It was because debugger. Np.