Heterogeneous tiles implementation

Signed-off-by: Alessandro Comodi<EMAIL_ADDRESS>This PR is an initial implementation of the heterogeneous tiles introduced with https://github.com/verilog-to-routing/vtr-verilog-to-routing/pull/1208.
TODO:

[ ] find a better way to use the start_tile implementation so that it gets used more generally by the various tile import scripts.
[x] produce a new VtR master+wip with the heterogeneous tiles implementation

Update: I am testing this PR alongside with the VtR heterogenous implementation integrated in master+wip and the litex-linux base test seems to be working, meaning that it passes Memtest.
I need to setup locally a tftp server to load linux binaries in DDR

Update: I am testing this PR alongside with the VtR heterogenous implementation integrated in master+wip and the litex-linux base test seems to be working, meaning that it passes Memtest.
I need to setup locally a tftp server to load linux binaries in DDR

Awesome!

Edit: Linux boots correctly. I used UART to load binaries as I have some troubles with the tftp server (on the local host, so unrelated to the bitstream).

For some reason the vendor tool test failed. I think though, by looking at the log, that all the tests acutally passed.

For some reason the vendor tool buildfailed. I think though, by looking at the log, that all the tests acutally passed.

There was a failure, but I don't think it is related.

@litghost Would this be ok to merge given that all vendor tool tests actually passed?

I have found the vendor tools error:
[828/1771] cd /tmpfs/src/github/symbiflow-arch-defs-presubmit-xc7-vendor/build/xc/xc7/tests/bram_test && diff -u /tmpfs/src/github/symbiflow-arch-defs-presubmit-xc7-vendor/build/xc/xc7/tests/bram_test/bram_test_36/artix7-xc7a50t-basys3-roi-virt-xc7a50t-basys3-test/top.bit.fasm /tmpfs/src/github/symbiflow-arch-defs-presubmit-xc7-vendor/build/xc/xc7/tests/bram_test/bram_test_36/artix7-xc7a50t-basys3-roi-virt-xc7a50t-basys3-test/design_bram_test_36_vivado.bit.fasm
FAILED: cd /tmpfs/src/github/symbiflow-arch-defs-presubmit-xc7-vendor/build/xc/xc7/tests/bram_test && diff -u /tmpfs/src/github/symbiflow-arch-defs-presubmit-xc7-vendor/build/xc/xc7/tests/bram_test/bram_test_36/artix7-xc7a50t-basys3-roi-virt-xc7a50t-basys3-test/top.bit.fasm /tmpfs/src/github/symbiflow-arch-defs-presubmit-xc7-vendor/build/xc/xc7/tests/bram_test/bram_test_36/artix7-xc7a50t-basys3-roi-virt-xc7a50t-basys3-test/design_bram_test_36_vivado.bit.fasm
--- /tmpfs/src/github/symbiflow-arch-defs-presubmit-xc7-vendor/build/xc/xc7/tests/bram_test/bram_test_36/artix7-xc7a50t-basys3-roi-virt-xc7a50t-basys3-test/top.bit.fasm	2020-04-05 12:30:47.957112578 -0700
+++ /tmpfs/src/github/symbiflow-arch-defs-presubmit-xc7-vendor/build/xc/xc7/tests/bram_test/bram_test_36/artix7-xc7a50t-basys3-roi-virt-xc7a50t-basys3-test/design_bram_test_36_vivado.bit.fasm	2020-04-05 12:37:09.500094783 -0700
@@ -4088,7 +4088,6 @@
 INT_L_X6Y110.EE4BEG2.LOGIC_OUTS_L10
 INT_L_X6Y110.EL1BEG0.NR1END1
 INT_L_X6Y110.ER1BEG2.LOGIC_OUTS_L5
-INT_L_X6Y110.FAN_ALT5.FAN_BOUNCE1
 INT_L_X6Y110.GFAN0.GND_WIRE
 INT_L_X6Y110.GFAN1.GND_WIRE
 INT_L_X6Y110.IMUX_L25.GFAN0

This is an ROI-related test, I will investigate what is happening here.
This did not happen for the test PR, therefore I suppose a change introduced with this PR is generating this issue.

I think I do understand what is the reason, less sure about the reasons though.
By inspecting two route files (one with master and one with this PR for bram_test_36) and the respective fasm files, there is a difference in how the BRAM_REGCLKB is being routed to $true.
In the failing case, the $true VCC wire is not directly connected to the FAN_ALT5 (which then is routed to the BRAM_REGCLKB pin), but there is a jump through the FAN_BOUNCE1.
I am unsure why the router chooses this path, instead of connecting FAN_ALT5 directly to the VCC wire, and, more importantly, it's hard to say whether this is an issue of the heterogeneous tiles feature or other changes happened in VtR master.

I have run the bram_test_36 on test on symbiflow master against a local master+wip version with the updated master and explicit_ports and it seems to correctly route the constant net.
This indicates that there might be something not correctly working with:

The heterogeneous tiles VtR implementation: I do not think this is the case, because there is nothing in the heterogeneous tiles that changes routing behaviour. One thing to check though is the correctness of the post-placement synchronization of the net terminals, which should be fine though as, if they weren't, there would most probably be a VTR assertion failure. In addition, the BRAM tile has only one equivalent site as well as only one sub tile;
Changes introduced with this PR

@litghost I realized one other thing that is different w.r.t. symbiflow master is that we are now using the bucket router_heap. I have tried to remove it and use the default one and, at least locally, the failing bram_36_diff_fasm test passes correctly.
I think that might be the reason for the vendor tools to fail. Anyhow, I am unsure whether this is a strong failure, meaning that it could potentially be solved in fasm2bels, as probably, for this weird route through two FAN pips, the tcl won't include the fixed route to the REGCLKB site pin of the BRAM 36.

we are now using the bucket router_heap

We were using the bucket heap before, the difference is the default router heap was set to the bucket heap, and now it matches upstream, and the default router heap is binary again.  See https://github.com/SymbiFlow/vtr-verilog-to-routing/pull/462/files#diff-f021ce5c0df49c7bf6020d0a4d94db66L1720

@litghost Indeed, what I meant was that by leaving the default heap (which is the binary with the new master+wip) the vendor tools pass.
What remains to understand is why using the bucketed one (still to verify with CI though) the bram_test_36 fails.

@litghost Indeed, what I meant was that by leaving the default heap (which is the binary with the new master+wip) the vendor tools pass.

I expect randomness?  I doubt the default heap always passes.  It is worth making sure the bucket heap is completing faster than the binary heap, and by a significant amount.  If the binary heap is 1% slower, but always has a green CI, that is not the same situation as a 10-15% slower, right?

@litghost I would say we let this CI run finish and extract run-time data. As I suppose, the bucket heap should give us greater performance, in that case we may consider solving the weird jump on the FAN_BOUNCE in fasm2bels?
What I have experienced is that, for several runs, the bucket heap always failed that test, instead the binary one did succeed everytime.

in that case we may consider solving the weird jump on the FAN_BOUNCE in fasm2bels?

I've considered it.  But I wasn't off hand able to come up with a good closed form algorithm for doing so.

@litghost All right, then is it worth restoring the bucket heap (if it results to be required to maintain the great improvement w.r.t. the binary one), removing the diff fasm test from the bram_test_36 and opening an issue to keep track of this?

@litghost All right, then is it worth restoring the bucket heap (if it results to be required to maintain the great improvement w.r.t. the binary one), removing the diff fasm test from the bram_test_36 and opening an issue to keep track of this?

To be clear, I don't believe switching from the binary to bucket or vise-versa actually fixes the vendor tools test.  However if the vendor tools test is consistently green on binary, then let's do that.

If CI is green, merge

xc7 vendor tools went green even though GH does report it as still running. Merging.


[Bug] Bubble ImagePullErr and ImagePullBackoff to the Ray CRD

Search before asking

[X] I searched the issues and found no similar issues.

KubeRay Component
ray-operator
What happened + What you expected to happen
I deployed a rayjob with a bad image reference (image does not exist)
The RayJob stayed in "Initializing" phase and didn't get updated/bubble up the error from starting the Driver Pod.
Reproduction script
TBD
Anything else
No response
Are you willing to submit a PR?

[ ] Yes I am willing to submit a PR!

We are working on improving RayCluster observability with new conditions APIs which should hopefully surface these types of failures.
@rueian do you know if the existing implementation would surface ImagePullErr and ImagePullBackoff errors?
See https://docs.google.com/document/d/1bRL0cZa87eCX6SI7gqthN68CgmHaB6l3-vJuIse-BrY/edit?usp=sharing for more details.

Happy to see catching ImagePullBackOff in this doc for Ray CRD. Are there updates on the implementation on this one?

@fiedlerNr9 can you try with Kuberay v1.2? You need to enable the feature gate for new conditions API. See https://docs.ray.io/en/latest/cluster/kubernetes/user-guides/observability.html#raycluster-status-conditions

I followed these docs but still see the same behaviour.
k get pods -n jan-playground-development | grep a27tckdrxzgx2s4kc4gb
a27tckdrxzgx2s4kc4gb-n0-0-raycluster-dcs6k-head-phwb8             0/1     ImagePullBackOff        0          2m35s
a27tckdrxzgx2s4kc4gb-n0-0-raycluster-dcs6k-ray-gro-worker-8gxcl   0/1     Init:ImagePullBackOff   0          2m35s

k get rayjobs -n jan-playground-development
NAME                        JOB STATUS   DEPLOYMENT STATUS   START TIME             END TIME   AGE
a27tckdrxzgx2s4kc4gb-n0-0                Initializing        2024-10-04T17:58:32Z              2m19s

Just to be on the same page, I would expect the status of the ray job to change to the status of the underlying pods.

At the moment we only update the RayCluster status, can you check the status there?
We should support mirroring the new conditions in the RayJob status though

I'm sorry for not getting back to you sooner.

We are working on improving RayCluster observability with new conditions APIs which should hopefully surface these types of failures.
@rueian do you know if the existing implementation would surface ImagePullErr and ImagePullBackoff errors?
See https://docs.google.com/document/d/1bRL0cZa87eCX6SI7gqthN68CgmHaB6l3-vJuIse-BrY/edit?usp=sharing for more details.

The existing StatusCondition implementation only reflects errors when calling the Kube API. We also haven't changed the old Status behavior even when the RayClusterStatusConditions feature gate is enabled. Therefore, errors like ImagePullErr and ImagePullBackoff are not reflected and not bubbled yet.
Right now, we have the following status conditions for the case of ImagePullErr if the feature gate is enabled:
Status:
  Conditions:
    Last Transition Time:   2024-10-04T19:33:59Z
    Message:                containers with unready status: [ray-head]
    Reason:                 ContainersNotReady
    Status:                 False
    Type:                   HeadPodReady
    Last Transition Time:   2024-10-04T19:33:59Z
    Message:                RayCluster Pods are being provisioned for first time
    Reason:                 RayClusterPodsProvisioning
    Status:                 False
    Type:                   RayClusterProvisioned

I think we could improve this by carrying the ImagePullErr/ImagePullBackoff messages into the HeadPodReady and RayClusterProvisioned conditions and then finding a way to bubble this into RayJob.

The RayCluster CRD can be considered a set of Kubernetes ReplicaSets (with each head or worker group similar to a ReplicaSet). Therefore, we aimed to make the observability consistent with ReplicaSets. However, Kubernetes ReplicaSets do not provide information on ImagePullBackOff errors.
For example, I created a ReplicaSet with image nginx:1.210 which doesn't exist.

Although this is not supported by Kubernetes ReplicaSets, we have received these requests multiple times. We will take it into consideration. If we decide to support this, we should clearly define which Pod-level errors should be surfaced by the KubeRay CR.


[W6.1f][W15-B1] Jackie Tan Jin Wei

change execute in Command to abstract.

@freesoup
Nothing much to review here. Hence closing with minimal review


Issue with running the v3 version of the API..getting an error

Thanks for stopping by to let us know something could be better!
PLEASE READ: If you have a support contract with Google, please create an issue in the support console instead of filing on GitHub. This will ensure a timely response.
Please run down the following list and make sure you've tried the usual "quick fixes":

Search the issues already opened: https://github.com/googleapis/python-documentai/issues
Search StackOverflow: https://stackoverflow.com/questions/tagged/google-cloud-platform+python

If you are still having issues, please be sure to include as much information as possible:
Environment details

OS type and version: AI Notebook, Python 3.7
Python version: python --version : Python 3.7
pip version: pip --version: pip 20.1.1 from /opt/conda/lib/python3.7/site-packages/pip (python 3.7)
google-cloud-documentai version: pip show google-cloud-documentai: jupyter@python-20200630-170900:~$ pip show google-cloud-documentai
Name: google-cloud-documentai
Version: 0.3.0
Summary: UNKNOWN
Home-page: https://github.com/googleapis/python-documentai
Author: Google LLC
Author-email<EMAIL_ADDRESS>License: Apache 2.0
Location: /opt/conda/lib/python3.7/site-packages
Requires: google-api-core, proto-plus
Required-by:

Steps to reproduce

I am trying to run the OOTB code shared for batch processing of the invoices. Here is the sample call that invokes the code: batch_process_documents(project_id='', location='us', processor_id='', gcs_input_uri='gs:///Invoice.pdf', gcs_output_uri='gs://*', gcs_output_uri_prefix='inv')
Run the boiler plate code.

Code example
# example

Stack trace
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-30-42be8c6d0dd8> in <module>
----> 1 batch_process_documents(project_id='***', location='us', processor_id='***', gcs_input_uri='gs://***/Invoice.pdf', gcs_output_uri='gs://***', gcs_output_uri_prefix='inv')

<ipython-input-28-f0fe586dfadb> in batch_process_documents(project_id, location, processor_id, gcs_input_uri, gcs_output_uri, gcs_output_uri_prefix)
     66     for i, blob in enumerate(blob_list):
     67         # Download the contents of this blob as a bytes object.
---> 68         blob_as_bytes = blob.download_as_bytes()
     69         document = documentai.types.Document.from_json(blob_as_bytes)
     70

AttributeError: 'Blob' object has no attribute 'download_as_bytes'

Making sure to follow these steps will guarantee the quickest resolution possible.
Thanks!

@rajatgupta143 I think your gcs_output_uri and gcs_output_uri_prefix are formatted wrong. Looks like it is very clear either from the customer perspective.
It might be also related to "bucket related (either that bucket doesnt exist or  bucket with prefix cannot be read correctly)".
Could you share your inputs for gcs_output_uri and gcs_output_uri_prefix format by replacing the sensitive info with test or some_bucket_name and share them ?
Could you also make sure that output bucket exists?
I will update the sample and add some example format in repo.


announce accepted completion to screen reader users

cc @jooyoungseo and @rperez030
@gdomingu got some feedback that it could be useful to alert when a completion is accepted.
I can imagine why this would be helpful, but also, the user has taken the action and knows what the completion is, so wonder if this would just add noise.

IMO, this is not necessary b/c users  take the accept gesture; thus, the event result is very obvious.

Agree, it's also clear because the user's focus is in the editor right where the suggestion was inserted.

Will wait for more feedback, but seems likely we'll close this.

I agree, however that announcement is currently present if I accept a suggestion from the accessible view. I wonder if it should be removed there for consistency. There is a change of context that happens because the accessible view closes and focus goes to the editor, which I believe serves as a confirmation.

It seems like I should verify that a screen reader alert does not happen when accepting a completion in the "accessible view". Which accessible view is this?

Correct, the inline completions accessible view, so trigger an inline completion, then use alt+f2


Support user provided MAC addresses

Fixes https://github.com/projectcalico/libnetwork-plugin/issues/97

Could you add a test for this change?

Seems I missed modifying Join() network_driver.go#L273

Pls squash

OK, looking good. Please squash then I can merge.

Squashed


Ability to debug flutter code when using "add-to-app" functionality by including "aar" file in the native android app

Right now , when we build the flutter aar file and include it in the native app. there are times when what used to work separately in flutter doesn't work when included in the native app.
At some point it becomes highly essential to have the ability to debug flutter code when using the add-to-app functionality and running the native app.

@xster Do we have any docs for this?

Not too clear what the question is. Is https://flutter.dev/docs/development/add-to-app/debugging what you're looking for?

Yes this is exactly what I was referring. Thank you

Can you let us know where you looked for documentation initially? It could help make sure we put documentation at the most intuitive places.

I think it would be better to change the title of the doc page here. On the first glance it just looks like a normal flutter debugging guide unless u read it furthur inside.

I may have come across this page but would have left thinking they're just talking about how to do debugging.

Did you mean via SEO or did you browse through the navigation bar on the left?

Via SEO

How about "Debugging your add-to-app module"?

The title's kind of unclear if u stumble upon that page from Google


Program steps very vague.

Specifically, when running it and getting to the four steps after I select my headset speakers and microphone.

Unmute your main mic and set it's appropriate sound level (pavucontrol/pulsemixer)
Move any programs that you want audio streaming ON to FunnelSink
and change the volume of OutputMixer according to your friend's liking.

"set it's appropriate sound level (pavucontrol/pulsemixer)" What does this mean, how do I do that?
"Move any programs that you want audio streaming ON to FunnelSink" Again, how do I do that?

I would also like to note after attempting to use this, even when closing it my main microphone no longer works.


BayesClassifier trainedWithDocument event

When using the BayesClassifier classifier, according to the docs an event of type trainedWithDocument is sent when calling classifier.train()
classifier.events.on('trainedWithDocument', function (obj) {
                self.logger.info("Trained [%s of %d]",obj.index,obj.total);
                // {
                //  total: 23 // There are 23 total documents being trained against
                //   index: 12 // The index/number of the document that's just been trained against
                //   doc: {...} // The document that has just been indexed
                //}
            });

at every train of the current document. So, given, this, how to wait until classifier ends in order to save the work when it is done with
classifier.save(self._options.classifierFilename, function(err, classifier) {
                // the classifier is saved to the classifier.json file!
                if(!err) self.logger.info("Classifier successfully saved.");
                else self.logger.error("An error occurred saving the classifier: %s", err.toString());

                return done(null, res);

            });

Thanks

[UPDATE]
I did further checks, and the classifier.train() is a synchronized call, so basically the events can be intercepted in the trainedWithDocument callback, and this could be make async like
classifier.events.on('trainedWithDocument', function (obj) {
  if( obj. total >= MAXDOCS ) { // we are ending here.
   self.end();
 }
});

setTimeout(() => {
  classifier.train();
},150)


Minor text fix for "ApiServer"

It looks a little weird to apply camel case style for the noun "apiserver", i didn't see somewhere else spelling it in that way.

This change is

Coverage decreased (-0.04%) to 45.342% when pulling 0b6f4d27703cea7f42a39a2140f7a3ac6fcf62d2 on xialonglee:patch-1 into 75124bc9f124ec3b95ea8b0ca43a0f905bac36e6 on kubernetes:master.

cc @aledbf

/lgtm

@xialonglee thanks!


Do a Zenodo data release that corresponds to 0.4.0

Use the databeta.sh script to generate a Docker-ized data release that provides access to fully processed datasets and a software environment that can be used to work with them, via the image we specify for the pudl-examples repository.

[ ] Update environment.yml file in the pudl-examples repo to use newly released PUDL.
[ ] Let the CI build a new docker image
[ ] Pull that Docker image locally and update databeta script to refer to it.
[ ] Generate a new databeta tarball to archive
[ ] Extract that tarball locally and try to run it following the included instructions
[ ] Run all of the example notebooks to verify that the packaged up data + software work together.
[ ] Upload the new archive to Zenodo and write a brief README file for the archive.

The main blocking issue here is that something has changed either about how Dask works, or the nature of our Parquet dataset, maybe because of changes to the PyArrow / Apache Arrow libraries. The pudl-dexamples notebook that deals with EPA CEMS no longer works. The following code used to efficiently serialize reading all the EPA CEMS data in and summarizing annual emissions, by reading one state/year combo at a time. But now it attempts to read in all of the data, and runs out of memory:
import dask.dataframe as dd

epacems_cols = [
    "year",
    "plant_id_eia",
    "co2_mass_tons",
    "so2_mass_lbs",
    "nox_mass_lbs",
]

epacems_path = pathlib.Path(pudl_settings["parquet_dir"]) / "epacems"

epacems_ddf = (
    dd.read_parquet(
        epacems_path,
        columns=epacems_cols,
        engine="pyarrow",
    )
    .astype({"year": int})
    .groupby(by=["year", "plant_id_eia"])
    .sum()
)
epacems_annual_emissions = epacems_ddf.compute()

We need to figure out what changed and make it work again.  Looking at the Parquet dataset, prior to aggregation it has npartitions==1223 so the dataset is partitioned initially, which I think means that Dask should be smart enough to just do one small chunk of work at a time.

Another weird problem I'm having is that "Run All Cells" sometimes is not running all cells. It'll skip a block in the middle of a notebook and then run more cells down toward the end.


build/ruff and versioning

build: update formatting config and generate version dynamically
style(pyflocker): apply ruff
bump: version 0.4.1 → 0.4.2

Summary by CodeRabbit

Bug Fixes

Fixed issues with the cryptography library, enhancing memoryview support and adding equality checks for ECC.

Refactor

Improved code readability and performance in AES operations.
Streamlined import statements across several modules.

Style

Adjusted code formatting for better readability in various files.
Added empty lines for consistency in module docstrings.

Documentation

Updated the changelog for the latest version with detailed descriptions of changes.

Chores

Updated .gitignore and project configuration files to align with current dependencies and tools.

Pull Request Test Coverage Report for Build<PHONE_NUMBER>
Details

5 of 5 (100.0%)  changed or added relevant lines in 4 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage increased (+0.001%) to 96.195%

Totals

Change from base Build<PHONE_NUMBER>:
0.001%

Covered Lines:
2557

Relevant Lines:
2657

💛  - Coveralls


npm is not installing

Operating system and version:
nvm debug output:

nvm ls output:

How did you install nvm?

What steps did you perform?
What happened?
What did you expect to happen?
Is there anything in any of your profile files that modifies the PATH?

If you are having installation issues, or getting "N/A", what does curl -I --compressed -v https://nodejs.org/dist/ print out?

npm comes with node, so unless you’re trying to install node older than 0.4, it shouldn’t be possible not to get npm.
Either way, please fill out the issue template (it’s there for a reason) and I’ll be happy to reopen.

I have been running new version of the node but I was searching on google and on that I was guided that I should use nvm and when I run the commands written in the file they don't even work. On curl link, it gives an error and on the 2nd link it denies it


Parser changes

Hello there!
Gleam's parser has been changed to only permit let and use directly within functions and block. Code such as this will now be rejected:
let x = let y = 1

This is a subset of the previously accepted syntax so the current tree sitter will still parse all valid programs, but perhaps perhaps it wants to be restricted also?

I think tree-sitter parsers shouldn't be so restrictive that they are basically linters but on the other hand it's a nice experience when your syntax highlighting hints that your program will not compile. So I think we should stay close to the upstream parser here and restrict this syntax 👍

Agreed


CSS

CSS
User Story
As a boot camp student
I want the pre-work notes to be structured on a webpage
So that I can easily find and read the information.
Acceptance Criteria
GIVEN a Prework Study Guide website
WHEN I view the study guide
THEN I see a dark blue header and footer, and four boxes with a shadow.

Completed CSS section of the prework modules


rpc get block info error

{"code":500,"message":"Internal Service Error","error":{"code":2,"name":"timeout_exception","what":"deadline 2020-02-20T04:02:21.564 exceeded by 21us ","details":[{"message":"deadline 2020-02-20T04:02:21.564 exceeded by 21us ","file":"json.cpp","line_number":499,"method":"escape_string"}]}}

The timeout_exception  with that error message is an indication that you exceeded the configured http-max-response-time-ms of nodeos during the conversion to json of your rpc call.


[docker] update base image

Motivation
lump all docker base image updates (#9284~#9292) in one.
Test Plan
CI + canary

/canary

/canary

:sunny: Canary successful

Ah sorry I'd missed this before bringing it up in slack.   Was digging through all the updates.


Minor bug fixes and improvements

Purpose

Improves error messages for update failures:

From:"contract ReceiveName {receiveName = "counter.receive"} at "ContractAddress" failed with code -2.
To: 'receive' in 'counter' at {"index":4,"subindex":0} failed with code -2.

Fixes incorrect parsing of ReceiveName when the 'contract' field contained a dot:

Example:
{"contract": "contrName.withDot", "func": "funcName"}

would be parsed as
{"contract": "contrName", "func": "withDot.funcName"}

Now, the validity of the contract name is checked on its own, and the example produces an error.
Changes
For 1: Update error messages.
For 2:

Add a check for the validity of the 'contract' field on its own
Add regression test.

Checklist

[x] My code follows the style of this project.
[x] The code compiles without warnings.
[x] I have performed a self-review of the changes.
[x] I have documented my code, in particular the intent of the
hard-to-understand areas.

Could you please record these changes in the changelog?


Added overrides for keys to 1.2-wip.

Fucked something up, gotta recommit.


Consider a web component interface to Filament

This weekend I prototyped out a web component wrapper for filament https://github.com/richardanaya/pbr-scene maybe it's inspiration for something you all might want to make officially?  I would love for Filament to compete with A-Frame.

Interesting. However I think I see Filament as being lower-level than this, and I'm worried about the maintenance burden.  Filament is more parallel to ThreeJS than it is to A-Frame.


Make LegacyReportingTests.test_old_report_redirect test solid.

This test randomly failed when redirect url was
http://testserver/reports/rep/UserProfile1079/?month=January&year=2011
instead of
http://testserver/reports/rep/UserProfile1079/?year=2011&month=January

@akatsoulas r?


Bug: appimage-run fails to run any appimage

appimage-run cannot access my filesystem and fails to run any appimage because of this.
I suspect this is due to me using impermanence and using bind mounts or something.
here is the log when trying to run an appimage:
[N] ~ $ appimage-run -d balenaEtcher-1.18.11-x64.AppImage                                                                                appimage-run-tests
+ getopts x:w:dh option
+ shift 1
+ '[' -n '' ']'
++ realpath balenaEtcher-1.18.11-x64.AppImage
+ APPIMAGE=/nix/persistent/home/user/balenaEtcher-1.18.11-x64.AppImage
+ shift
+ '[' -n '' ']'
+ '[' -n true ']'
+ '[' -f /nix/persistent/home/user/balenaEtcher-1.18.11-x64.AppImage ']'
+ apprun
++ sha256sum /nix/persistent/home/user/balenaEtcher-1.18.11-x64.AppImage
++ awk '{print $1}'
+ SHA256=f87bbbd1439c98b5f874f39810103669d05d5ff0bc472a7fd7b6ef8be8d47d50
+ export APPDIR=/nix/persistent/home/user/.local/cache/appimage-run/f87bbbd1439c98b5f874f39810103669d05d5ff0bc472a7fd7b6ef8be8d47d50
+ APPDIR=/nix/persistent/home/user/.local/cache/appimage-run/f87bbbd1439c98b5f874f39810103669d05d5ff0bc472a7fd7b6ef8be8d47d50
+ '[' -x /nix/persistent/home/user/.local/cache/appimage-run/f87bbbd1439c98b5f874f39810103669d05d5ff0bc472a7fd7b6ef8be8d47d50/squashfs-root ']'
+ '[' '!' -x /nix/persistent/home/user/.local/cache/appimage-run/f87bbbd1439c98b5f874f39810103669d05d5ff0bc472a7fd7b6ef8be8d47d50 ']'
++ dirname /nix/persistent/home/user/.local/cache/appimage-run/f87bbbd1439c98b5f874f39810103669d05d5ff0bc472a7fd7b6ef8be8d47d50
+ mkdir -p /nix/persistent/home/user/.local/cache/appimage-run
mkdir: cannot create directory ‘/nix/persistent/home/user/.local/cache/appimage-run’: Read-only file system
+ unpack /nix/persistent/home/user/balenaEtcher-1.18.11-x64.AppImage /nix/persistent/home/user/.local/cache/appimage-run/f87bbbd1439c98b5f874f39810103669d05d5ff0bc472a7fd7b6ef8be8d47d50
+ local src=/nix/persistent/home/user/balenaEtcher-1.18.11-x64.AppImage
+ local out=/nix/persistent/home/user/.local/cache/appimage-run/f87bbbd1439c98b5f874f39810103669d05d5ff0bc472a7fd7b6ef8be8d47d50
+ local appimageSignature
++ LC_ALL=C
++ readelf -h /nix/persistent/home/user/balenaEtcher-1.18.11-x64.AppImage
++ awk 'NR==2{print $10$11;}'
+ appimageSignature=4149
+ local appimageType
++ LC_ALL=C
++ readelf -h /nix/persistent/home/user/balenaEtcher-1.18.11-x64.AppImage
++ awk 'NR==2{print $12;}'
+ appimageType=02
+ '[' 4149 '!=' 4149 ']'
+ case "$appimageType" in
++ LC_ALL=C
++ readelf -h /nix/persistent/home/user/balenaEtcher-1.18.11-x64.AppImage
++ awk 'NR==13{e_shoff=$5} NR==18{e_shentsize=$5} NR==19{e_shnum=$5} END{print e_shoff+e_shentsize*e_shnum}'
+ offset=188392
++ basename /nix/persistent/home/user/balenaEtcher-1.18.11-x64.AppImage
+ echo 'Uncompress balenaEtcher-1.18.11-x64.AppImage of type 02 @ offset 188392'
Uncompress balenaEtcher-1.18.11-x64.AppImage of type 02 @ offset 188392
+ unsquashfs -q -d /nix/persistent/home/user/.local/cache/appimage-run/f87bbbd1439c98b5f874f39810103669d05d5ff0bc472a7fd7b6ef8be8d47d50 -o 188392 /nix/persistent/home/user/balenaEtcher-1.18.11-x64.AppImage

FATAL ERROR: dir_scan: failed to make directory /nix/persistent/home/user/.local/cache/appimage-run/f87bbbd1439c98b5f874f39810103669d05d5ff0bc472a7fd7b6ef8be8d47d50, because No such file or directory
+ chmod go-w /nix/persistent/home/user/.local/cache/appimage-run/f87bbbd1439c98b5f874f39810103669d05d5ff0bc472a7fd7b6ef8be8d47d50
chmod: cannot access '/nix/persistent/home/user/.local/cache/appimage-run/f87bbbd1439c98b5f874f39810103669d05d5ff0bc472a7fd7b6ef8be8d47d50': No such file or directory
++ basename /nix/persistent/home/user/balenaEtcher-1.18.11-x64.AppImage
+ echo 'balenaEtcher-1.18.11-x64.AppImage is now installed in /nix/persistent/home/user/.local/cache/appimage-run/f87bbbd1439c98b5f874f39810103669d05d5ff0bc472a7fd7b6ef8be8d47d50'
balenaEtcher-1.18.11-x64.AppImage is now installed in /nix/persistent/home/user/.local/cache/appimage-run/f87bbbd1439c98b5f874f39810103669d05d5ff0bc472a7fd7b6ef8be8d47d50
+ export PATH=/nix/store/q1c2flcykgr4wwg5a6h450hxbk4ch589-bash-5.2-p15/bin:/nix/store/1fn92b0783crypjcxvdv6ycmvi27by0j-binutils-2.40/bin:/nix/store/bblyj5b3ii8n6v4ra0nb37cmi3lf8rz9-coreutils-9.3/bin:/nix/store/7wfya2k95zib8jl0jk5hnbn856sqcgfk-gawk-5.2.2/bin:/nix/store/k93s5p63dsyyxih5lvl0mc6f0g7svbp7-libarchive-3.7.2/bin:/nix/store/sr6v1d6zyq1afwbbfryj2cyzh48p7g2v-pv-1.8.5/bin:/nix/store/325b9q4z86is6wzxqz59x3ik2ssfxa78-squashfs-4.6.1/bin:/run/wrappers/bin:/usr/bin:/usr/sbin:/nix/persistent/home/user/.local/share/opam/default/bin:/nix/store/kbcrs84s1x8yd5bp1nq6q6ihda8nd2lp-bash-interactive-5.2-p15/bin:/nix/store/ywwjpdyhar4f3vcqf4qk77vrbr3vj5wl-patchelf-0.15.0/bin:/nix/store/7wkshj58fcsl1f3zyi67qsxgl1p8nki1-gcc-wrapper-12.2.0/bin:/nix/store/dcd1zhv56rk0d2z7akzfjgzr076c4jl9-gcc-12.2.0/bin:/nix/store/lyvhsvwp2pzy74fkcn7qbs5vcgy5d7vl-glibc-2.37-8-bin/bin:/nix/store/ahkfdxq8mcpsb5kvdvgqr1wv8zjngbh4-coreutils-9.1/bin:/nix/store/rbwalnmk5a6nj9mbldxzmj11gvacbrsi-binutils-wrapper-2.40/bin:/nix/store/22p5nv7fbxhm06mfkwwnibv1nsz06x4b-binutils-2.40/bin:/nix/store/ahkfdxq8mcpsb5kvdvgqr1wv8zjngbh4-coreutils-9.1/bin:/nix/store/4qrbfa42xyicny7vlgbbqxz07kcsjbx2-findutils-4.9.0/bin:/nix/store/6bcqajyagcc4s3azx0zqnb3rqxd7gdnv-diffutils-3.9/bin:/nix/store/gy82r5cf12hgkmhzifsyx3fp6cnf6j2l-gnused-4.9/bin:/nix/store/zrls4w5lxynqakh1jlrp03kg4bxzp9yi-gnugrep-3.7/bin:/nix/store/s3yl3l8na93v53k2cfsyz0jrb50bnk8n-gawk-5.2.1/bin:/nix/store/czy796d2241296bdy45w87cmaiig50zr-gnutar-1.34/bin:/nix/store/hdbb5ayyrz1b5hicbsb4am8vin0xxxg6-gzip-1.12/bin:/nix/store/hd573q7097cxj5q29pnc3z0rvakz7ga3-bzip2-1.0.8-bin/bin:/nix/store/nnis9ph3wj866bbs0lalbf22i5bq8yry-gnumake-4.4.1/bin:/nix/store/rhvbjmcfnkg8i2dxpzr114cp1ws7f667-bash-5.2-p15/bin:/nix/store/x4ckkjsxfl4ihl82v0bzsyxs4ay2d5b5-patch-2.7.6/bin:/nix/store/8xjzmhw6hgqmynb5f1v7aw2lvafij405-xz-5.4.3-bin/bin:/nix/store/lcjsmsxqwd4vkq2wj5mmbyjzny3i6k49-file-5.44/bin:/nix/persistent/home/user/.local/share/opam/default/bin:/run/wrappers/bin:/nix/persistent/home/user/.local/bin:/nix/persistent/home/user/.nix-profile/bin:/nix/persistent/home/user/.local/state/nix/profile/bin:/nix/persistent/home/user/.local/state/nix/profile/bin:/etc/profiles/per-user/user/bin:/nix/var/nix/profiles/default/bin:/run/current-system/sw/bin:/nix/store/1alqjnr40dsk7cl15l5sn5y2zdxidc1v-binutils-wrapper-2.40/bin:/nix/store/lvx8bvn5lc6aqjq2jalnq7xc9lhqbpzg-pciutils-3.10.0/bin:/nix/persistent/home/user/usr/bin
+ PATH=/nix/store/q1c2flcykgr4wwg5a6h450hxbk4ch589-bash-5.2-p15/bin:/nix/store/1fn92b0783crypjcxvdv6ycmvi27by0j-binutils-2.40/bin:/nix/store/bblyj5b3ii8n6v4ra0nb37cmi3lf8rz9-coreutils-9.3/bin:/nix/store/7wfya2k95zib8jl0jk5hnbn856sqcgfk-gawk-5.2.2/bin:/nix/store/k93s5p63dsyyxih5lvl0mc6f0g7svbp7-libarchive-3.7.2/bin:/nix/store/sr6v1d6zyq1afwbbfryj2cyzh48p7g2v-pv-1.8.5/bin:/nix/store/325b9q4z86is6wzxqz59x3ik2ssfxa78-squashfs-4.6.1/bin:/run/wrappers/bin:/usr/bin:/usr/sbin:/nix/persistent/home/user/.local/share/opam/default/bin:/nix/store/kbcrs84s1x8yd5bp1nq6q6ihda8nd2lp-bash-interactive-5.2-p15/bin:/nix/store/ywwjpdyhar4f3vcqf4qk77vrbr3vj5wl-patchelf-0.15.0/bin:/nix/store/7wkshj58fcsl1f3zyi67qsxgl1p8nki1-gcc-wrapper-12.2.0/bin:/nix/store/dcd1zhv56rk0d2z7akzfjgzr076c4jl9-gcc-12.2.0/bin:/nix/store/lyvhsvwp2pzy74fkcn7qbs5vcgy5d7vl-glibc-2.37-8-bin/bin:/nix/store/ahkfdxq8mcpsb5kvdvgqr1wv8zjngbh4-coreutils-9.1/bin:/nix/store/rbwalnmk5a6nj9mbldxzmj11gvacbrsi-binutils-wrapper-2.40/bin:/nix/store/22p5nv7fbxhm06mfkwwnibv1nsz06x4b-binutils-2.40/bin:/nix/store/ahkfdxq8mcpsb5kvdvgqr1wv8zjngbh4-coreutils-9.1/bin:/nix/store/4qrbfa42xyicny7vlgbbqxz07kcsjbx2-findutils-4.9.0/bin:/nix/store/6bcqajyagcc4s3azx0zqnb3rqxd7gdnv-diffutils-3.9/bin:/nix/store/gy82r5cf12hgkmhzifsyx3fp6cnf6j2l-gnused-4.9/bin:/nix/store/zrls4w5lxynqakh1jlrp03kg4bxzp9yi-gnugrep-3.7/bin:/nix/store/s3yl3l8na93v53k2cfsyz0jrb50bnk8n-gawk-5.2.1/bin:/nix/store/czy796d2241296bdy45w87cmaiig50zr-gnutar-1.34/bin:/nix/store/hdbb5ayyrz1b5hicbsb4am8vin0xxxg6-gzip-1.12/bin:/nix/store/hd573q7097cxj5q29pnc3z0rvakz7ga3-bzip2-1.0.8-bin/bin:/nix/store/nnis9ph3wj866bbs0lalbf22i5bq8yry-gnumake-4.4.1/bin:/nix/store/rhvbjmcfnkg8i2dxpzr114cp1ws7f667-bash-5.2-p15/bin:/nix/store/x4ckkjsxfl4ihl82v0bzsyxs4ay2d5b5-patch-2.7.6/bin:/nix/store/8xjzmhw6hgqmynb5f1v7aw2lvafij405-xz-5.4.3-bin/bin:/nix/store/lcjsmsxqwd4vkq2wj5mmbyjzny3i6k49-file-5.44/bin:/nix/persistent/home/user/.local/share/opam/default/bin:/run/wrappers/bin:/nix/persistent/home/user/.local/bin:/nix/persistent/home/user/.nix-profile/bin:/nix/persistent/home/user/.local/state/nix/profile/bin:/nix/persistent/home/user/.local/state/nix/profile/bin:/etc/profiles/per-user/user/bin:/nix/var/nix/profiles/default/bin:/run/current-system/sw/bin:/nix/store/1alqjnr40dsk7cl15l5sn5y2zdxidc1v-binutils-wrapper-2.40/bin:/nix/store/lvx8bvn5lc6aqjq2jalnq7xc9lhqbpzg-pciutils-3.10.0/bin:/nix/persistent/home/user/usr/bin
+ wrap
+ export APPIMAGE_SILENT_INSTALL=1
+ APPIMAGE_SILENT_INSTALL=1
+ '[' -n '' ']'
+ exec /nix/persistent/home/user/.local/cache/appimage-run/f87bbbd1439c98b5f874f39810103669d05d5ff0bc472a7fd7b6ef8be8d47d50/AppRun
/usr/bin/appimage-exec.sh: line 89: /nix/persistent/home/user/.local/cache/appimage-run/f87bbbd1439c98b5f874f39810103669d05d5ff0bc472a7fd7b6ef8be8d47d50/AppRun: No such file or directory

https://github.com/NixOS/nixpkgs/blob/nixos-23.05/pkgs/build-support/trivial-builders/default.nix#L87
This is the code for appimage-run
@ncfavier @happysalada @hsjobeki @roberth
Metadata
my flake
[I] ~ [127]$ nix-shell -p nix-info --run "nix-info -m"
this path will be fetched (0.00 MiB download, 0.00 MiB unpacked):
  /nix/store/nn818zlhq4hnx794c57adijwikdw6cqz-nix-info
copying path '/nix/store/nn818zlhq4hnx794c57adijwikdw6cqz-nix-info' from 'https://cache.nixos.org'...
 - system: `"x86_64-linux"`
 - host os: `Linux 6.1.63, NixOS, 24.05 (Uakari), 24.05.20231124.fc23c76`
 - multi-user?: `yes`
 - sandbox: `yes`
 - version: `nix-env (Nix) 2.18.1`
 - nixpkgs: `/nix/var/nix/profiles/per-user/root/channels/nixos`

questions are probably best asked on discourse.nixos.org, for future reference

this is not a question it's a bug. I haven't been able to run appimages for like 6 months

as you said, it's likely related to misconfiguring impermanence, which is not a bug.

The first error seems to be this one:
mkdir: cannot create directory ‘/nix/persistent/home/user/.local/cache/appimage-run’: Read-only file system

The remainder are just results of that directory not existing further down the line.
What's confusing to me is that this points to ~/.local/cache instead of ~/.cache, which would be caused by a changed XDG_CACHE_HOME.
This does indeed look like a misconfiguration of sorts.
If the following command works as intended, it is likely that XDG_CACHE_HOME points at a wrong directory, however this pretty much rules out an issue with appimage-run.
env "XDG_CACHE_HOME=$(mktemp -dp /tmp)" appimage-run -d balenaEtcher-1.18.11-x64.AppImage

(note that running this will create a directory in /tmp, which should be possible even if your home directory is inaccessible in some locations, however it also means repeatedly running this command will create additional directories which you may want to clean up, if you're running on tmpfs it also requires some amount of memory to unpack the AppImage, you can point the XDG_CACHE_HOME to any directory that is actually writeable for your user)
For any further assistance the Discourse would be more appropriate.
(checking for what's mounted in your home directory using findmnt --target ~/.local/cache will probably point you in the right direction)

By default if I try to run an appimage .../cache/appimage-run won't even be created.
I have also tried changing the permissions of the directory and the appimage directory but it didn't fix anything.

Correct, the creation of the directory fails with "Read-only file system", which means this is unrelated to permissions, and instead is a filesystem mounted read-only in that space, permissions won't change the outcome here.

Anyways using the command you suggested fixes that error but I still don't understand why it's happening in the first place.
As you can see the cache directory doesn't pose any problems to any other programs except for app-image run.

The link to the appimage-run source code you sent in the opening post is only half of the magic used by the tool.
The more relevant part is the chrootenv script which is used to isolate the appimage process into a different mount namespace.
The entire appimage-run code you linked earlier does therefore run in a different mount namespace with a separate directory structure and some bind mounts.
For instance, the following list of directories is not mounted at all in the namespace:
https://github.com/NixOS/nixpkgs/blob/1a5412e14030d2a00281021d5f8ef05b31027912/pkgs/build-support/build-fhsenv-chroot/chrootenv/src/chrootenv.c#L21
The difficult part here is to determine what causes the filesystem to be read-only in the created mount namespace.
If you check the content of the appimage-run wrapper (cat $(command -v appimage-run)) you will find the path to the chrootenv binary.
With this binary you can reproduce the environment in which the error occurs, for instance by running this command (replace the path to your chrootenv according to the content of your appimage-run wrapper):
/nix/store/nqkfvyxi0hhaf7k251lxvz50xx41934r-chrootenv/bin/chrootenv $SHELL

This will allow you to enter an interactive shell in that environment to determine the root cause.
Despite some genuine effort I have not been able to reproduce any issues, so what that may help pin down what exactly the problem is would be to compare the output of the following command within the chrootenv shell and outside of it:
findmnt --output TARGET,FSTYPE,OPT-FIELDS,OPTIONS -N $$

Any difference in mounted filesystems, especially missing filesystems or diverging options would no doubt be relevant.
Furthermore double check whether any of the involved paths have any symlinks in them as these may not necessarily resolve correctly (e.g. if a symlink were to point to /opt or /bin which is not present in the chrootenv environment this would fail), using realpath for instance.

As a side-note:

Yes I have set XDG_CACHE_HOME to a path in my users home directory. […] I choose to set it to the default value suggested in the freedesktop standard.

FYI: The XDG Base Directory Specification refers to $HOME/.config, not $HOME/.local/config.

@benaryorg I've checked to contents of appimage-run and could not find any instances of chrootenv. Probably because of some system settings I have enabled.
Here's the output.
[I] ~ $ cat $(command -v appimage-run)

#!/nix/store/q1c2flcykgr4wwg5a6h450hxbk4ch589-bash-5.2-p15/bin/bash
ignored=(/nix /dev /proc /etc)
ro_mounts=()
symlinks=()
etc_ignored=()
for i in /nix/store/bcyir8pj2bddq74aijmqq45xgbsr28q0-appimage-run-fhs/*; do
  path="/${i##*/}"
  if [[ $path == '/etc' ]]; then
    :
  elif [[ -L $i ]]; then
    symlinks+=(--symlink "$(/nix/store/bblyj5b3ii8n6v4ra0nb37cmi3lf8rz9-coreutils-9.3/bin/readlink "$i")" "$path")
    ignored+=("$path")
  else
    ro_mounts+=(--ro-bind "$i" "$path")
    ignored+=("$path")
  fi
done

if [[ -d /nix/store/bcyir8pj2bddq74aijmqq45xgbsr28q0-appimage-run-fhs/etc ]]; then
  for i in /nix/store/bcyir8pj2bddq74aijmqq45xgbsr28q0-appimage-run-fhs/etc/*; do
    path="/${i##*/}"
    # NOTE: we're binding /etc/fonts and /etc/ssl/certs from the host so we
    # don't want to override it with a path from the FHS environment.
    if [[ $path == '/fonts' || $path == '/ssl' ]]; then
      continue
    fi
    ro_mounts+=(--ro-bind "$i" "/etc$path")
    etc_ignored+=("/etc$path")
  done
fi

for i in '/etc/static' '/etc/nix' '/etc/shells' '/etc/bashrc' '/etc/zshenv' '/etc/zshrc' '/etc/zinputrc' '/etc/zprofile' '/etc/passwd' '/etc/group' '/etc/shadow' '/etc/hosts' '/etc/resolv.conf' '/etc/nsswitch.conf' '/etc/profiles' '/etc/login.defs' '/etc/sudoers' '/etc/sudoers.d' '/etc/localtime' '/etc/zoneinfo' '/etc/machine-id' '/etc/os-release' '/etc/pam.d' '/etc/fonts' '/etc/alsa' '/etc/asound.conf' '/etc/ssl/certs' '/etc/ca-certificates' '/etc/pki'; do
  if [[ "${etc_ignored[@]}" =~ "$i" ]]; then
    continue
  fi
  if [[ -L $i ]]; then
    symlinks+=(--symlink "$(/nix/store/bblyj5b3ii8n6v4ra0nb37cmi3lf8rz9-coreutils-9.3/bin/readlink "$i")" "$i")
  else
    ro_mounts+=(--ro-bind-try "$i" "$i")
  fi
done

declare -a auto_mounts
# loop through all directories in the root
for dir in /*; do
  # if it is a directory and it is not ignored
  if [[ -d "$dir" ]] && [[ ! "${ignored[@]}" =~ "$dir" ]]; then
    # add it to the mount list
    auto_mounts+=(--bind "$dir" "$dir")
  fi
done

declare -a x11_args
# Always mount a tmpfs on /tmp/.X11-unix
# Rationale: https://github.com/flatpak/flatpak/blob/be2de97e862e5ca223da40a895e54e7bf24dbfb9/common/flatpak-run.c#L277
x11_args+=(--tmpfs /tmp/.X11-unix)

# Try to guess X socket path. This doesn't cover _everything_, but it covers some things.
if [[ "$DISPLAY" == :* ]]; then
  display_nr=${DISPLAY#?}
  local_socket=/tmp/.X11-unix/X$display_nr
  x11_args+=(--ro-bind-try "$local_socket" "$local_socket")
fi

cmd=(
  /nix/store/xkl370pyh843b6ymd81622cs38js2y1y-bubblewrap-0.8.0/bin/bwrap
  --dev-bind /dev /dev
  --proc /proc
  --chdir "$(pwd)"

  --die-with-parent
  --ro-bind /nix /nix
  # Our glibc will look for the cache in its own path in `/nix/store`.
  # As such, we need a cache to exist there, because pressure-vessel
  # depends on the existence of an ld cache. However, adding one
  # globally proved to be a bad idea (see #100655), the solution we
  # settled on being mounting one via bwrap.
  # Also, the cache needs to go to both 32 and 64 bit glibcs, for games
  # of both architectures to work.
  --tmpfs /nix/store/qn3ggz5sf3hkjs2c797xf7nan3amdxmp-glibc-2.38-27/etc \
  --symlink /etc/ld.so.conf /nix/store/qn3ggz5sf3hkjs2c797xf7nan3amdxmp-glibc-2.38-27/etc/ld.so.conf \
  --symlink /etc/ld.so.cache /nix/store/qn3ggz5sf3hkjs2c797xf7nan3amdxmp-glibc-2.38-27/etc/ld.so.cache \
  --ro-bind /nix/store/qn3ggz5sf3hkjs2c797xf7nan3amdxmp-glibc-2.38-27/etc/rpc /nix/store/qn3ggz5sf3hkjs2c797xf7nan3amdxmp-glibc-2.38-27/etc/rpc \
  --remount-ro /nix/store/qn3ggz5sf3hkjs2c797xf7nan3amdxmp-glibc-2.38-27/etc \
  --tmpfs /nix/store/nmimzmfi0ccr9cha4vgl0mgsbdzzibhj-glibc-2.38-27/etc \
  --symlink /etc/ld.so.conf /nix/store/nmimzmfi0ccr9cha4vgl0mgsbdzzibhj-glibc-2.38-27/etc/ld.so.conf \
  --symlink /etc/ld.so.cache /nix/store/nmimzmfi0ccr9cha4vgl0mgsbdzzibhj-glibc-2.38-27/etc/ld.so.cache \
  --ro-bind /nix/store/nmimzmfi0ccr9cha4vgl0mgsbdzzibhj-glibc-2.38-27/etc/rpc /nix/store/nmimzmfi0ccr9cha4vgl0mgsbdzzibhj-glibc-2.38-27/etc/rpc \
  --remount-ro /nix/store/nmimzmfi0ccr9cha4vgl0mgsbdzzibhj-glibc-2.38-27/etc \
  "${ro_mounts[@]}"
  "${symlinks[@]}"
  "${auto_mounts[@]}"
  "${x11_args[@]}"

  /nix/store/i94wwm46jw9xhwbjmvnq0zd9n18a2if6-appimage-run-init "$@"
)
exec "${cmd[@]}"

I'm 99% sure this line is the culprit near the end of the program --ro-bind /nix /nix.
My system uses impermanence so I have it setup to forget everything except for /nix on every poweroff.
[N] ~ $ tree -L 1 /nix
/nix
├── persistent
├── store
└── var

4 directories, 0 files

Naturally then I save the files I want to persist between reboots in /nix/persistent and let impermanence bind mount my state in /nix/persistent to the right places as you can see in my flake here.
I think to fix this issue I would have to replace --ro-bind /nix /nix by something like --ro-bind /nix/store /nix/store; --ro-bind /nix/var /nix/var.
I'm not sure how to test these changes on my system though so If you could point me to ressources explaining that or tell me what to do that would be greatly appreciated.

Oh my goodness, I humbly apologize, the one machine I used to test this has been stuck on using nix-env installed packages vs. the flakes everywhere else, which left appimage-run in a pre- 1baab4e14a6c9d5fd0f3e6af144aeeb61e911110.
Okay, now that I've caught up with the times, and bubblewrap seems to be the way to go, I agree.
Bind-mounting the entire /nix area as read-only seems problematic, and at least one other place where the nix-store is bind mounted read-only, the nixos container infrastructure, does indeed make more specific bind-mounts:
https://github.com/NixOS/nixpkgs/blob/9b9778125203413c8c30023be36ff7b8d3db9474/nixos/modules/virtualisation/nixos-containers.nix#L158-L162)
I think changing the originating code to use a similarly specific bind-mount is reasonable.

@benaryorg Do you know how I could change this code on my machine to test if this change would fix the issue?

So for one, if you really just wanna get things working for now, you could revert back to the previous chrootenv based approach by specifying the appimage-run package (e.g. in your environment.sysemPackages or similar) like this:
with pkgs; appimage-run.override { buildFHSEnv = buildFHSEnvChroot; }

This would use the script that I erroneously referred to previously which shouldn't suffer from that problem.
When it comes to actually making the bubblewrap version do that I can't seem to find an easy way that doesn't result in copy&pasting half the code though, since the relevant parts seem to be part of a let binding, which can't be easily overriden to my knowledge (i.e. changing the code itself, not its parameters).
However on second glance the more restrictive mounting wouldn't even solve your problem, as bubblewrap in this case would just not mount the directory at all then, neither read-write nor read-only, leaving you with a technically writable filesystem in its place but not one that actually uses your persistent directories either.
The path of /nix is [explicitly]https://github.com/NixOS/nixpkgs/blob/9b9778125203413c8c30023be36ff7b8d3db9474/pkgs/build-support/build-fhsenv-bubblewrap/default.nix#L119) excluded from the auto-mounting which leads me to believe that the /nix directory is subject to assumptions which stretch throughout nixpkgs; as far as I see it you shouldn't put your own data in /nix, as the structure of /nix is part of how NixOS handles things internally. The impermanence flake which you are using does not use a subdirectory of /nix in its documentation either, and using a path like /persistence or anything else that is outside the ignored directories (or special directories like proc, sys, etc.) would work just fine.
If you are really set on using that specific directory your best bet would likely be to somehow overwrite the extraBwrapArgs using an overlay like this:
# use via `pkgs.extend` or add to `overlays`
(final: prev: { appimage-run = prev.appimage-run.override { buildFHSEnv = (old: prev.buildFHSEnv (old // { extraBwrapArgs = [ "--bind /nix/persistent /nix/persistent" ]; })); }; })

TL;DR: I think the entire nixpkgs structure and the way that the /nix directory is laid out and used assume that you do not store any custom data in there and the easiest solution would be to store your persistent data quite literally anywhere else.

I solved this by not having my persistent data under /nix


A lot of warnings like 'warning: the variable "var_name" is unsafe as it has been set inside a case/cond/receive/if/&&/||.'

After upgrading to elixir 1.3 I have faced a lot of warnings! Here are the ones coming from ex_admin:

warning: the variable "controller_route" is unsafe as it has been set inside a case/cond/receive/if/&&/||. Please explicitly return the variable value instead. For example:
case int do
  1 -> atom = :one
  2 -> atom = :two
end

should be written as
atom =
  case int do
    1 -> :one
    2 -> :two
  end

Unsafe variable found at:
lib/ex_admin/register.ex:184
warning: the variable "resources" is unsafe as it has been set inside a case/cond/receive/if/&&/||. Please explicitly return the variable value instead. For example:
case int do
  1 -> atom = :one
  2 -> atom = :two
end

should be written as
atom =
  case int do
    1 -> :one
    2 -> :two
  end

Unsafe variable found at:
lib/ex_admin/form.ex
warning: the variable "controller_route" is unsafe as it has been set inside a case/cond/receive/if/&&/||. Please explicitly return the variable value instead. For example:
case int do
  1 -> atom = :one
  2 -> atom = :two
end

should be written as
atom =
  case int do
    1 -> :one
    2 -> :two
  end

Unsafe variable found at:
lib/ex_admin/register.ex:184
warning: the variable "controller_route" is unsafe as it has been set inside a case/cond/receive/if/&&/||. Please explicitly return the variable value instead. For example:
case int do
  1 -> atom = :one
  2 -> atom = :two
end

should be written as
atom =
  case int do
    1 -> :one
    2 -> :two
  end

Unsafe variable found at:
lib/ex_admin/register.ex:184
warning: the variable "controller_route" is unsafe as it has been set inside a case/cond/receive/if/&&/||. Please explicitly return the variable value instead. For example:
case int do
  1 -> atom = :one
  2 -> atom = :two
end

should be written as
atom =
  case int do
    1 -> :one
    2 -> :two
  end

Unsafe variable found at:
lib/ex_admin/register.ex:184
warning: the variable "controller_route" is unsafe as it has been set inside a case/cond/receive/if/&&/||. Please explicitly return the variable value instead. For example:
case int do
  1 -> atom = :one
  2 -> atom = :two
end

should be written as
atom =
  case int do
    1 -> :one
    2 -> :two
  end

Unsafe variable found at:
lib/ex_admin/register.ex:184
warning: variable speaker is unused
lib/ex_admin/show.ex:15
warning: the variable "controller_route" is unsafe as it has been set inside a case/cond/receive/if/&&/||. Please explicitly return the variable value instead. For example:
case int do
  1 -> atom = :one
  2 -> atom = :two
end

should be written as
atom =
  case int do
    1 -> :one
    2 -> :two
  end

Unsafe variable found at:
lib/ex_admin/register.ex:184

Here's my mix.lock information:
"ex_admin": {:hex, :ex_admin, "0.7.6", "8ea0b7d21a073bd901edf37a10808a0c504d9bd952e41b19cb78a896ab253c00", [:mix], [{:xain, "~> 0.5.3", [hex: :xain, optional: false]}, {:scrivener, "~> 1.0", [hex: :scrivener, optional: false]}, {:phoenix_html, "~> 2.5", [hex: :phoenix_html, optional: false]}, {:phoenix_ecto, "~> 2.0", [hex: :phoenix_ecto, optional: false]}, {:phoenix, "~> 1.1", [hex: :phoenix, optional: false]}, {:inflex, "~> 1.5", [hex: :inflex, optional: false]}, {:factory_girl_elixir, "~> 0.1.1", [hex: :factory_girl_elixir, optional: false]}, {:exactor, "~> 2.2.0", [hex: :exactor, optional: false]}, {:ex_queb, "~> 0.1", [hex: :ex_queb, optional: false]}, {:ecto, "~> 1.1", [hex: :ecto, optional: false]}, {:decimal, "~> 1.0", [hex: :decimal, optional: false]}, {:csvlixir, "~> 1.0.0", [hex: :csvlixir, optional: false]}, {:cowboy, "~> 1.0", [hex: :cowboy, optional: false]}]},

Most of this stuff has been fixed on my github master branch. I have been waiting to push a new hex release until I get more of the new ecto2 stuff finished.

There's still one left at
Unsafe variable found at:
  lib/ex_admin/form.ex:32

I'm running on master and the docs here: https://github.com/smpallen99/ex_admin/blob/880e511cf3e67081cd727b809eb115eb31a57966/lib/ex_admin/form.ex#L145 show and example that produces compile warnings like this. Is there another way to do conditional rendering? Or, alternatively, add a class to an inputs call?


Add field to CNP Status to indicate whether policy is in datapath

The policy status currently sets the OK field to true when the policy was loaded into the agent. We need a field which indicates true when the policy has been implemented in the datapath.
// CiliumNetworkPolicyNodeStatus is the status of a Cilium policy rule for a
// specific node
type CiliumNetworkPolicyNodeStatus struct {
        // OK is true when the policy has been installed successfully
        OK bool `json:"ok,omitempty"`

        // Error describes the error condition if OK is false
        Error string `json:"error,omitempty"`

        // LastUpdated contains the last time this status was updated
        LastUpdated Timestamp `json:"lastUpdated,omitempty"`
}

How can the logic be verified? The CiliumNetworkPolicyNodeStatus is per node, the policy is per endpoint. Does this means we should present the policy is implemented in the datapath in case there's one endpoint that is enforcing this given policy?

Revision - policy revision which was return when the policy was added to the repository internall
Enforcing - true when all endpoints are on >= Revision

After they are on that revision number I'll need to check which endpoints have the policy installed. And for that I need this https://github.com/cilium/cilium/issues/1556

After they are on that revision number I'll need to check which endpoints have the policy installed. And for that I need this #1556

I don't follow. When we receive a CRD, we import it into the policy repository and we get back a policy revision. We can update the status with the revision right away. We should then run a controller which waits for all endpoints on the node to run this revision. Alternatively, we can encode this logic into bumpPolicyRevision() if we don't want to poll. Once all endpoints have been bumped, the Enforcing it can be set.

ah that is easier then. I thought you wanted to mark the policy as Enforced only if there were endpoints that were being covered by the given policy.


need improve document or clarification on type interops between windows-sys and windows crates for mixed use cases

Suggestion
when writing libraries, I use types from windows-sys for public APIs, since I usually just make simple wrappers around the native API so no need for the convenient features of the windows crate, and as bonus it compiles faster, and is a lot easier to migrate from the winapi crate
however, when writing a complicated application using native Windows API (or wrapper crates), I would prefer the windows crate for the added features. so I often ended up have both windows-sys and windows at the same time, this raises the situation with many duplication of data types and I need to do a lot of conversions.
here's some examples:

for pointers, just cast the pointer using ptr::cast() or as _, but it's unchecked whether the pointee is actually the same type, for example, some structs have both A and W variants, it's prone to accidental typos when casting a pointer.
for structs, I can use mem::transmute(), but for very simple structs like POINT and RECT, I usually just destructure the fields of the struct (bind to individual variables such as x, y, width, height), but some API require LPRECT, in which case I have to reconstruct a new struct.
for handle types, windows uses newtype wrapper, windows-sys uses type aliases, the conversion is done by manually wrap/unwrap-ing the primitive types.
for enums and constants, it's similar to handle types.

this is both tedious and very error prone, and sometimes frustrating. I can't seem to find any documentation about what is the "proper" (or recommended) way to mix windows-sys and windows? I found #1330,  #1393, #1643, etc, but they are out of concern of the interoperability between windows-sys and windows.
I see there's :core::Type::abi() and core::Type::from_abi(), can we offer something similar, like Interop::into_sys() and Interop::from_sys()? it can be provided in a separate crate (e.g. named windows_helper, windows_util, windows_interop etc) on top of both windows-sys and windows, or maybe (for better or worse) we can add such functionality directly to windows, which means windows now depends on windows-sys. this approach would make the pointer type and structs conversion safer, although the conversion of handle types are still unchecked because they use type aliases , at least code feels more documented with the intentions.
currently, both windows-sys and windows are used by hundreds of crates on crates.io, I'm afraid it would eventually hinder the adoption of rust native windows programming if the ecosystem is segregated without good support for interoperability in the long run.
opnions?

Just getting over a nasty cold. I'll read this more carefully but briefly, all of the types in windows and windows-sys are size and layout compatible with each other as well as their C counterparts. That means you can always transmute and end up with the same memory layout.

Hi @nerditation, is this an issue that's popping up because you're re-exporting windows/windows-sys types? No judgement, just trying to build up context.

Just getting over a nasty cold. I'll read this more carefully but briefly, all of the types in windows and windows-sys are size and layout compatible with each other as well as their C counterparts

sorry to hear that, hope you're well now.
yes, I understand that the types are all binary compatible to the C counterparts, but they are different as for the rust type checker. in other words we have two different set of rust types with parallel name spaces for the same undelrying ffi types.
for example, this is what the dependency graph could be like for the ffi library openssl:

+-----------+     +---------+          rust  --->  ffi boundary  --->  C
|           | --> | openssl |                           |
|           |     +---------+\
|           |          ^      \                         |
|           |          |       \
|           |          |        \                       |
|           |     +---------+    \     +-------------+     +-------------+
|application| --> |   foo   |     +--> | openssl-sys | --> | openssl-dev |
|           |     +---------+    /     +-------------+     +-------------+
|           |          |        /                       |
|           |          |       /
|           |          V      /                         |
|           |     +---------+/
|           | --> |   bar   |                           |
+-----------+     +---------+

and here's ffi for the Windows SDK:

+-----------+                            rust  --->  ffi boundary  --->  C
|           | -----------------+                         |
|           |                   \
|           |                    \     +-------------+   |
|           |                     +--> |   windows   | ----+
|           |     +---------+    /     +-------------+   |  \
|application| --> |   foo   | --+             |              \
|           |     +---------+                 |          |    \    +-------------+
|           |          |                     ???               +-> | Windows SDK |
|           |          |                      |          |    /    +-------------+
|           |          V                      |              /
|           |     +---------+          +-------------+   |  /
|           | --> |   bar   | -------> | windows-sys | ----+
+-----------+     +---------+          +-------------+   |

my point is, currently, when interfacing the Windows API, you have to make a choice between windows and windows-sys (and don't forget winapi too), and your choice impacts the users of your crate. to my understanding, windows-sys exists because the compile time of windows is way too long to be accepted as THE crate for the Windows API, so a "slim" version with reduced feature set is provided as a middle ground. that's all well and good, if it's structured like a "full-featured" addon crate layered atop the "slim" core crate, and users have a smooth transition path between the two. but the reality is, the two crates are more like alternatives (as oppose to supplementaries) to each other.
of course you can always cast (transmute) between the two, they are just transparent ffi wrappers, but it just doesn't feel right to me.
anyway,  maybe it's just me being extra frustrated by the Windows API and somehow imagining rust would magically fix all the mess. sorry about the rant.

is this an issue that's popping up because you're re-exporting windows/windows-sys types?

yes, kind of. I'm not re-exporting types from windows-sys directly, but some type "leaked" into function signatures.
here's an example, I have a library (implemented using windows-sys) to manage embedded resources in the EXE file, and the API requires an argument of type HMODULE, I didn't call internally GetModuleHandleA(NULL) (which return a HMODULE to the running EXE) because occasionally I want to load resources from DLLs too. another example is an API that returns a HWND to a managed window.
since these are all opaque handles, I can use any opaque types with same C representation to "hide" the dependency on windows-sys, but what's that good for? if the caller is getting the handles from different crates (say windows or winapi), you still need a cast anyway. I know it's an API design issue and I'm going to change it anyway. for now, I'm probably gonna use the raw-window-handle crate for the public API.
but still, I just want to make a point, because it doesn't feel quite right to me.
btw, I also checked RawHandle (and HandleOrNull etc) from std, then I found out raw-window-handle just uses opaque pointer *mut c_void for both HWND and HINSTANCE:
https://github.com/rust-windowing/raw-window-handle/blob/4611ee8644d72c7b36fa47960d1b18c0c9648e08/src/windows.rs#L32-L39

Thanks for the feedback!
I don't recommend exporting types from windows or windows-sys mainly because those crates are not yet 1.0 as I work on stabilizing the underlying metadata for the rather vast Windows API surface. So, prefer to export something like *mut c_void or isize or some wrapper of your own if you must export something.
Regarding windows depending on windows-sys, the trouble there is that this would make use of the windows crate even more "expensive". I experimented with this, but found that it ended up causing more code to be effectively compiled. The windows crate can define a very streamlined ABI for its internal use that is even more concise than that of the windows-sys crate. This is something that can be revisited if an effective solution is discovered in future.


CompareCoverage in QEMU mode?

We support laf-intel in LLVM mode, why not support an equivalent instrumentation in QEMU mode?
I'm experimenting a bit with the x86 target and I will create a branch for this.
The idea is to not touch the fuzzer but work only in QEMU.
The implementation will be different from honggfuzz that uses a different map for CompareCoverage, my idea is to put all in the AFL map.
This can create problems with large targets and the user may need to enlarge MAP_SIZE but this is also true for laf-intel.
Obviously this mode is off by default and can be enabled setting an env variable (I called it AFL_QEMU_COMPCOV in my experiment of this evening).
Let me know your opinion.

If I'm correct this will make afl++ the first public fuzzer to support such instrumentation in binary-only mode.

@andreafioraldi that sounds very cool ... didnt think that would actually be possible :)

I uploaded the code in the qemu-compcov branch. The main changes are in https://github.com/vanhauser-thc/AFLplusplus/blob/qemu-compcov/qemu_mode/patches/i386-translate.diff and https://github.com/vanhauser-thc/AFLplusplus/blob/qemu-compcov/qemu_mode/patches/afl-qemu-cpu-translate-inl.h.
I don't know if this is really effective at the moment.

For the coverage of mem/strcmp and related functions I was thinking about creating an external library that have to be loaded with LD_PRELOAD, it's only and idea I will implement it when I will have time.

Simple tests are promising.
Fuzzing LAVA-M who for 50 minutes afl++ can discover ~30 unique crashes with compcov qemu vs. 2 with normal qemu. The majority of the crashes were discovered in the last 10 minutes cause the progress of the cmp is horizontal and so I think that the crashes will double when fuzzing for an hour (my train arrived and so I had to shutdown the laptop).

Note that the queue grows a lot (~3k vs 120) but  LAVA-M is a patological case for cmp bypassing, test on real targets is needed.

@andreafioraldi that sounds very good already!
I am busy today with testing fairfuzz and trying an aggressive trimming option in afl-fuzz.
what would a good target be? as its string compares I guess something like libxml2 I guess?

String compare are not supported atm but I will implement the library asap. This evening I will try to fuzz djpeg, it has a lot of cmp/sub instructions.

Results of ~7 hour and half of fuzzing libpng.

that looks good!
I am running this now on libtiff (as libxml2 crashes qemu - see issue #24) for 2 hours.
paths discovered do not tell you anything - see the discussion in issue #18. what you want to compare is coverage and this can be done with afl-cov. e.g. MOpt has fantastic numbers of discovered paths compared against anything, but when checked against the coverage its bad. (its a good mode but has to be used differently)
I will post my results later, so far the library seems to be effective

The first plot are basic block/time, the second the number of items in queue (what in labeled "paths" in the status screen). To measure performance of course bbs/time are needed but when I want monitor also if there is "queue explosion". The second graph is not for performance measurement.

what happens if you do to many things in parallel ... I tested libtiff but actually didnt check if there is actually something in there that would benefit from your implementation. because after 2h of fuzzing - same coverage.
so I compile with llvm_mode and laf-intel .... and it reported nothing for laf-intel to do ;)
testing unrar now, there are lots of memcmp etc in there

Yes it is normal, as described in the laf-intel blogpost (https://lafintel.wordpress.com/) there are targets that does not benefits from the technique and some other that have less performance. In addition, 2h may not be enough cause in some targets the advantages are in the long term (see their evaluation).
In general, when you have a closed binary with many comparison that depends directly on the input (the chunk names "gAMA" "iCCP" etc. in PNG) without a dictionary using laf-intel/compcov is a good idea. Of course, by default, this will be disabled and in fact I disagree with your choice (in TODO for 2.53d) to enable laf-intel by default in LLVM mode.

Regards parallel fuzzing a good idea is to enable CompareCoverage only in the master.

I added an example to test into the lib... directory. plus I added a debug routine (which kinda works but also makes it super slow, so -t 1000000 is kinda needed ;) with AFL_QEMU_COMPCOV_DEBUG=1

Maybe it's time to merge?

Your decision :)

Done after an improvement to libcompcov


Add partition_by_index?

Would work like .enumerate().partition(), except that the two partition collections get only the original items, not the (index, item) tuple.
Use case: Put elements of an iterator (which may have even or odd # of elements) into two alternating Vecs.
If you think this is a good addition, I'll make a PR.

I think using itertools' .enumerate().partition_map() covers this. It should be simple to make with that version of partition.

It is kind of clunky though:
.enumerate().partition_map(|(i, v)| if condition { Either::Left(v) } else { Either::Right(v) })
But I know you can't put every two-line function into itertools :)
Another version that might be interesting though:
distribute() -> B where B: Distribute<Self::Item>
and impls for Distribute for tuples of (FromIterator, ...)?
Or maybe distribute(x) where x is a tuple of &mut FromIterator...

I don't quite manage to fill in the blanks there. We use Extend to be able to build a collection piece by piece, but how does distribute work?

Sorry. Just like Extend, but distributing the items round-robin over several collections. In effect building 2 ... n at the same time.

I see, a general round robin distributer sounds more fun than partition_by_index.
I'm not sure what our editorial standards should be in itertools, we're getting more serious and less experimental by now, I'm not so fond of anything that requires helper traits right now. tuples/tuple_windows got a pass just because they were that awesome..

That's true, tuples and friends are great...
I think I'll make a pass at implementation anyway, even if in the end it's just for fun :)

Hey, this is not bad. Maybe a trait to handle the "add" part (and maybe it can do it by ref instead of passing collections by value.
playground
fn add<T, U, A, B>((mut t, mut u): (T, U), elt: (A, B)) -> (T, U)
    where T: Extend<A>, U: Extend<B>,
{
    t.extend(Some(elt.0));
    u.extend(Some(elt.1));
    (t, u)
}

fn main() {
    println!("{:?}", (0..12).tuples().fold((Vec::new(), Vec::new()), add));
    // ([0, 2, 4, 6, 8, 10], [1, 3, 5, 7, 9, 11])
}

I don't know, is this a better way to express it?
extern crate itertools;

use itertools::Itertools;

#[derive(Debug)]
pub struct ExtendTuple<T>(T);

impl<'a, A, B, T, U> Extend<(A, B)> for ExtendTuple<&'a mut (T, U)>
    where T: Extend<A>, U: Extend<B>,
{
    fn extend<I: IntoIterator<Item=(A, B)>>(&mut self, iter: I) {
        for (x, y) in iter {
            (self.0).0.extend(Some(x));
            (self.0).1.extend(Some(y));
        }
    }
}

fn main() {
    let mut v = (Vec::new(), Vec::new());
    ExtendTuple(&mut v).extend((0..12).tuples());
    println!("{:?}", v);
}

Interesting discussion, but I think we decided the partition by index thing is sufficiently covered by partition_map, and what's more we can make interesting Extend impls for various tuple wrappers, but that's not necessarily in scope for itertools.


Rust YJIT Support in ruby-build

First of all thank you for creating the incredibly useful tool that is ruby-build 🙏
As you may know, we have been working on porting YJIT to Rust, and this has now been upstreamed: https://github.com/ruby/ruby/pull/5826
Rust YJIT requires rustc 1.60 to be installed to build in release mode, and cargo to build in dev/debug mode. As such, by default, CRuby will build without YJIT support. In order to build YJIT, --enable-yjit has to be specified when running ./configure. More info on how to build YJIT can be found in the YJIT readme.
Given that ruby-build is the primary way that many people deploy Ruby and a central part of the Ruby ecosystem, we would really like to have the tool support building Rust YJIT. This would really help lower the bar for people to get YJIT running and could truly help YJIT see more use in the wild.
My colleague @byroot has told me that the ruby-build maintainers don't feel comfortable automatically installing the Rust toolchain. I understand your concerns with respect to security (and maybe the size of the download required). I was wondering if maybe we could explore other options.
Some ideas that come to mind:

Possibly ruby-build could check if rustc >= 1.60 is already installed, and set --enable-yjit if present
Maybe ruby-build could report to the user whether ruby is being built with or without YJIT support. If built without YJIT support, ruby-build could print a message explaining that rustc >= 1.60 is required to build YJIT.

I think we're likely going to freeze the rustc dependency at version 1.60 for some time so this requirement is going to be pretty stable. Happy to discuss pros and cons of various options and how we can help the ruby-build maintainers make this happen if you are open to it.
Tagging my colleagues @byroot @xrxr @noahgibbs.

JFYI, I have Rust YJIT installed by the following workaround command.
% rustc --version
rustc 1.60.0 (7737e0b5c 2022-04-04)
% RUBY_CONFIGURE_OPTS=--enable-yjit rbenv install 3.2.0-dev
(snip)

% ruby --yjit -ve 'p RubyVM::YJIT.enabled?'
ruby 3.2.0dev (2022-05-06T09:13:22Z master 67950a4c0a) +YJIT [x86_64-darwin19]
true

This is a current method as a user. It would be useful if an installation (or information) would be provided upstream without requiring the RUBY_CONFIGURE_OPTS option. ruby-build user may not notice that YJIT is not built by default in Ruby 3.2 because unlike Ruby 3.1.

Possibly ruby-build could check if rustc >= 1.60 is already installed, and set --enable-yjit if present

Like @koic already pointed out, wouldn't it be better that the user is in control over when they set --enable-yjit, especially since it has the Rust dependency? RUBY_CONFIGURE_OPTS was made for situations like these.

The key difference is that YJIT was enabled by default (on all supported platforms) with Ruby 3.1. As of now, with Ruby head and Ruby 3.2, it will only be opt-in and only available if the Rust toolchain is installed. Unfortunately, I'm sure many users won't be aware of this.
I was hoping that we could help make the users aware of the difference and maybe point them to some quick instructions on how to build Ruby with YJIT. Hold their hand a little bit.

Thanks for explaining!

Possibly ruby-build could check if rustc >= 1.60 is already installed, and set --enable-yjit if present

I like this approach. In usage instructions for ruby-build, we could suggest that users install rustc when compiling Ruby 3.2+. However, I wouldn't make Rust a requirement, and I definitely wouldn't automatically install rustc on their behalf if it's not already present on the system.
As someone who is not familiar with YJIT, what are the user-facing drawbacks if someone compiles Ruby without support for it?

Maybe ruby-build could report to the user whether ruby is being build with or without YJIT support.

That's another viable option. For example, we do print a warning to the user if their Ruby version was built without openssl support, since we assume that most usages of Ruby will sooner or later need to make HTTPS requests.

what are the user-facing drawbacks if someone compiles Ruby without support for it?

Not much if you use the interpreter, however if you try to pass the --jit flag to Ruby, it will error out (or print a warning?)

How important is it to the average Ruby user to know whether YJIT is enabled or not?

I'd say what is important is that they get a clear information on how to get it if they wish to.

I'm thinking we could update https://github.com/rbenv/ruby-build/wiki to include rustc, maybe as a separate section if we don't want to suggest it as default since it's not "a required dependency" and not needed for older Ruby versions.
I thought CRuby automatically detects if a recent enough rustc is around, does it not do that anymore?
About every builtin C ext does such automatic detection so it seems the mechanics people are used to nowadays.

We only look for rustc when the user configures --enable-yjit now.

Not much if you use the interpreter, however if you try to pass the --jit flag to Ruby, it will error out (or print a warning?)

Actually it will just use MJIT instead in that case it seems, --yjit OTOH warns:
$ ruby --jit -v -e itself
ruby 3.2.0dev (2022-06-26T06:36:14Z master c2e37c8ff7) +MJIT [x86_64-linux]

$ ruby --yjit -e0
ruby: warning: Ruby was built without YJIT support

I think the second idea is fine (and should be very easy to implement), PR welcome.
The first idea seems more risky because e.g. it could potentially cause extra compilation problems, etc for something the user did not opt-in explicitly.

I'm thinking we could update https://github.com/rbenv/ruby-build/wiki to include rustc, maybe as a separate section if we don't want to suggest it as default since it's not "a required dependency" and not needed for older Ruby versions.

Seems like a good idea to update the wiki and mention how to install the rust toolchain in there, and that it's necessary to build YJIT 👍

The first idea seems more risky because e.g. it could potentially cause extra compilation problems, etc for something the user did not opt-in explicitly.

Just to be sure, you mean you'd accept a PR for the following:

ruby-build could report to the user whether ruby is being built with or without YJIT support. If built without YJIT support, ruby-build could print a message explaining that rustc >= 1.60 is required to build YJIT.

?

Yes, exactly.
It could be another function like this: https://github.com/rbenv/ruby-build/blob/c0d6e22177b8fceb0eb0cd5b2007f9e4fc4f9f8d/bin/ruby-build#L1208-L1242

Opened a PR for this: https://github.com/rbenv/ruby-build/pull/2005

Since https://github.com/ruby/ruby/pull/6662 was merged, maybe the logic in ruby-build can be simplified now, WDYT?

Definitely.

I think the YJIT team would still like to warn users if Ruby is compiled without YJIT, so it would be nice to keep that part (if the ruby-build maintainers aren't opposed) but other than that, #2005 can pretty much be reverted.

From what I understand, enabling YJIT using rustc found on the system is now part of the standard build process in Ruby main branch and we can remove the enable_yjit step and associated implementation from ruby-build?

Right, we'd keep the warning: https://github.com/rbenv/ruby-build/pull/2005/files#diff-9161f3dbce993baa919f6535fbe4c32d2d2cd1ba9e5c93aa899188063c9022acR1277.
But for that we basically need the rest of the logic, and the only part which changes is then removing https://github.com/rbenv/ruby-build/pull/2005/files#diff-9161f3dbce993baa919f6535fbe4c32d2d2cd1ba9e5c93aa899188063c9022acR1274-R1275 because that's now automatic in ruby/ruby.

Keeping the warning means:

We have to forever duplicate the logic of discovering rustc on the system the same way that the Ruby build process does. If we fail to do so, our warning might become a false positive.
We have to keep enable_yjit step in the build-definition on every Ruby version >= 3.2 until the end of time.

I do not strongly feel that this is a responsibility that ruby-build should keep. If the Ruby build process wants to warn the user about missing rustc in its own build log, it can do that and we can do better to surface that warning on stderr even when writing the build log to a file, but I don't feel we should be doing anything too special with Ruby 3.2+ build definitions other than ./configure && make && make install.


Incompatibility with earthhack 1.8.5

Hi!
I'm getting a crash when using this mod alongside earthhack 1.8.5 (https://github.com/3arthqu4ke/3arthh4ck/releases/tag/1.8.5)
I attach the crash log: https://pastebin.com/ivpLN934
Thank you for your help and for this awesome mod :)

Mod load order issue due to Mixin 0.8.5 requirement.
Quick fix: rename xaeroplus-XX.jar to 0xaeroplus-XX.jar in your mods folder.
MC mods are loaded in alphabetical order so this makes XaeroPlus load first.


ALZ bicep modules multi-region support

Overview/Summary
[WIP - Draft PR for early review]
This PR is adding examples and guidance on how to deploy ALZ platform resources into multiple regions.
This PR fixes/adds/changes/removes

Replace me
Replace me
Replace me

Breaking Changes

Replace me
Replace me

Testing Evidence
Replace this with any testing evidence to show that your Pull Request works/fixes as described and planned (include screenshots, if appropriate).
As part of this Pull Request I have

[ ] Read the Contribution Guide and ensured this PR is compliant with the guide
[ ] Ensured the resource API versions in .bicep file/s I am adding/editing are using the latest API version possible
[ ] Checked for duplicate Pull Requests
[ ] Associated it with relevant GitHub Issues
[ ] (ALZ Bicep Core Team Only) Associated it with relevant ADO Items
[ ] Ensured my code/branch is up-to-date with the latest changes in the main branch
[ ] Performed testing and provided evidence.
[ ] Updated one or more of the following tests (if required)

Unit
Linting
E2E (End-To-End)
ValidateAzCloud (Base validation in Azure Cloud)
ValidateMcCloud (Base validation in Azure China Cloud)

[ ] Updated relevant and associated documentation (e.g. Contribution Guide, Module READMEs, Wiki Docs etc.)
[ ] If relevant, created or updated Code Tours here

@oZakari @jtracey93 This is a still a WIP but wanted to share an early draft of the hubNetworking multi-region guidance. Please let me know if you have comments/feedback so I can start on the vWan module next.

This PR is dependent on https://github.com/Azure/ALZ-Bicep/pull/805

Hey @sebassem, I think the docs are concise and clear. I have made some updates to the hub networking module readme, particularly with switching to East US and West US since they are paired regions. Also, demonstrates that they will need to look out for the regions that do/dont support availability zones to determine which parameter file to use.
I also removed the example parameter files to avoid bloat as I think it should be pretty easy to follow that they just need to change the regions. Let me know if you object though and we can discuss.
Could you please update the East US 2 example output to West US? After that I think it is good to merge as long as good on your end.

Thanks Zach, screenshot has been updated

/azp validate azcloud


This is an update for migrations commands to use data pulled from the vi...

...sual studio startup project
for DataDirectory and the Configuration file.
fix for issue #823

@bricelam

I think the logging can be improved. Here is what I would want to see (all verbose, no warnings).
Using start-up project 'WebApplication1'.
    -or- More than one start-up project found. Using 'ClassLibrary1' instead.
    -or- No start-up project found. Using 'ClassLibrary1' instead.
Using application base 'C:\Projects\Solution1\ClassLibrary1\bin\Debug'.
Using application configuration 'C:\Projects\Solution1\WebApplication1\Web.config'.
    -or- No application configuration file found.
Using data directory 'C:\Projects\Solution1\WebApplication1\App_Data'.
Using current directory 'C:\Projects\Solution1\WebApplication1\bin'.

I wonder if we'll need the -StartUpProject parameters...

This is ready for review, I've responded to all of the feedback except the -StartUpProject parameter.  If we really feel that this is necessary I can add the behavior.

:shipit: (After addressing feedback)


Bug : Script execution error 1_Movie_Analysis.py

When I tried to open the link of the webpage https://scrape-ml.streamlit.app/Movie_Analysis, I got the following error :
File "/mount/src/scrape-ml/Web_app/pages/1_Movie_Analysis.py", line 93
              try:
             ^
IndentationError: unexpected indent

Please assign me this issue to correct the error.

@sanjay-kv
Even after fixing the indentation error, I found these errors:
One of them is :
Using return statement outside the function. Temporarily, I commented out these lines as shown(lines 93 to 97 in the mentioned file) :
try:
        # return pd.read_csv(uploaded_file, encoding='latin1')
except UnicodeDecodeError:
        st.error("File encoding not supported. Please upload a CSV file with UTF-8 or Latin1 encoding.")
        # return None

After running the code again, I got this error :
ImportError: cannot import name 'analyze_reviews' from 'utils'
File "C:\Users\DELL\AppData\Local\Programs\Python\Python310\lib\site-packages\streamlit\runtime\scriptrunner\script_runner.py", line 589, in _run_script
    exec(code, module.__dict__)
File "E:\Git\recode-hive-scrape-ml\Scrape-ML\Web_app\pages\1_Movie_Analysis.py", line 3, in <module>
    from utils import analyze_reviews, recommend_movies

It shows that analyze_reviews.py and recommend_movies.py files are not present in utils directory.
Please tell me what should I do now.
Will you accept my pull request after resolving indentation error as I could not write code for review.py and recommend_movies.py and also we should choose only a small topic to create an issue?


Azcopy fails on copy operation  from onprem server to azure blob storage for some files

Which version of the AzCopy was used?
10.26.0
Note: The version is visible when running AzCopy without any argument
Which platform are you using? (ex: Windows, Mac, Linux)
Windows
What command did you run?
azcopy copy \.pdf https://.blob.core.windows.net///.pdf.processed? --cap-mbps 30 --log-level INFO --output-type text c 3 --recursive=true
Note: Please remove the SAS to avoid exposing your credentials. If you cannot remember the exact command, please retrieve it from the beginning of the log file.
What problem was encountered?
azcopy returns failed result.
How can we reproduce the problem in the simplest way?
We can produce error only in our production environment.
Have you found a mitigation/solution?
Some files completed successfully by increasing --block-size-mb parameter to 100 MB. However some files fail again.
Azcopy Log:
PS C:\06f9e930-0864-694c-6af8-1e8a1631ee9e.log
2024/08/08 06:28:43 AzcopyVersion  10.26.0
2024/08/08 06:28:43 OS-Environment  windows
2024/08/08 06:28:43 OS-Architecture  amd64
2024/08/08 06:28:43 Log times are in UTC. Local time is 8 Aug 2024 01:28:43
2024/08/08 06:28:43 ISO 8601 START TIME: to copy files that changed before or after this job started, use the parameter --include-before=2024-08-08T06:28:38Z or --include-after=2024-08-08T06:28:38Z
2024/08/08 06:28:43 Any empty folders will not be processed, because source and/or destination doesn't have full folder support
2024/08/08 06:28:43 Job-Command copy \.pdf https://.blob.core.windows.net///.pdf.processed? --cap-mbps 30 --log-level INFO --output-type text --block-size-mb 3 --recursive=true
2024/08/08 06:28:43 Number of CPUs: 4
2024/08/08 06:28:43 Max file buffer RAM 2.000 GB
2024/08/08 06:28:43 Max concurrent network operations: 32 (Based on number of CPUs. Set AZCOPY_CONCURRENCY_VALUE environment variable to override)
2024/08/08 06:28:43 Check CPU usage when dynamically tuning concurrency: true (Based on hard-coded default. Set AZCOPY_TUNE_TO_CPU environment variable to true or false override)
2024/08/08 06:28:43 Max concurrent transfer initiation routines: 64 (Based on hard-coded default. Set AZCOPY_CONCURRENT_FILES environment variable to override)
2024/08/08 06:28:43 Max enumeration routines: 16 (Based on hard-coded default. Set AZCOPY_CONCURRENT_SCAN environment variable to override)
2024/08/08 06:28:43 Parallelize getting file properties (file.Stat): false (Based on AZCOPY_PARALLEL_STAT_FILES environment variable)
2024/08/08 06:28:43 Max open files when downloading:<PHONE_NUMBER> (auto-computed)
2024/08/08 06:28:43 Final job part has been created
2024/08/08 06:28:43 Final job part has been scheduled
2024/08/08 06:28:43 INFO: [P#0-T#0] Starting transfer: Source "\\?\\\\\.pdf" Destination "https://***.blob.core.windows.net//.pdf". Specified chunk size 3145728
2024/08/08 06:28:45 PERF: primary performance constraint is Unknown. States: X:  0, O:  0, M:  0, L:  0, R:  0, D:  0, W:  0, F:  0, B:  5, E:  0, T:  5, GRs: 32
2024/08/08 06:28:45 0.0 %, 0 Done, 0 Failed, 1 Pending, 0 Skipped, 1 Total, 2-sec Throughput (Mb/s): 31.474
2024/08/08 06:28:45 ==> REQUEST/RESPONSE (Try=1/75.1497ms, OpTime=1.7910386s) -- RESPONSE SUCCESSFULLY RECEIVED
PUT https://.blob.core.windows.net//.pdf?blockid=********
X-Ms-Request-Id: [987984b8-e01e-00b1-3f5c-e9773b000000]
2024/08/08 06:28:45 ERR: [P#0-T#0] UPLOADFAILED: \?******.pdf : 409 : 409 Blob operation is not supported.. When Staging block. X-Ms-Request-Id: 987984b8-e01e-00b1-3f5c-e9773b000000
Dst: https://.blob.core.windows.net//*.pdf
2024/08/08 06:28:45 JobID=06f9e930-0864-694c-6af8-1e8a1631ee9e, Part#=0, TransfersDone=1 of 1
2024/08/08 06:28:45 all parts of entire Job 06f9e930-0864-694c-6af8-1e8a1631ee9e successfully completed, cancelled or paused
2024/08/08 06:28:45 is part of Job which 1 total number of parts done
2024/08/08 06:28:47 PERF: primary performance constraint is Unknown. States: X:  0, O:  0, M:  0, L:  0, R:  0, D:  0, W:  0, F:  0, B:  0, E:  0, T:  0, GRs: 32
2024/08/08 06:28:47 0.0 %, 0 Done, 1 Failed, 0 Pending, 0 Skipped, 1 Total, 2-sec Throughput (Mb/s): 1.4728
2024/08/08 06:28:47
Diagnostic stats:
IOPS: 1
End-to-end ms per request: 1856
Network Errors: 0.00%
Server Busy: 0.00%
Job 06f9e930-0864-694c-6af8-1e8a1631ee9e summary
Elapsed Time (Minutes): 0.0669
Number of File Transfers: 1
Number of Folder Property Transfers: 0
Number of Symlink Transfers: 0
Total Number of Transfers: 1
Number of File Transfers Completed: 0
Number of Folder Transfers Completed: 0
Number of File Transfers Failed: 1
Number of Folder Transfers Failed: 0
Number of File Transfers Skipped: 0
Number of Folder Transfers Skipped: 0
Total Number of Bytes Transferred: 0
Final Job Status: Failed
2024/08/08 06:28:47 Closing Log

Hi @mustafakemalsigirci, please attach complete azcopy logs to debug this. Also, could you try with a different block size?

Closing because of inactivity, please reopen/open a new issue if this persists.


security/acme-client Retrieve Cert/Key via REST API

Important notices
Before you add a new report, we ask you kindly to acknowledge the following:
[-] I have read the contributing guide lines at https://github.com/opnsense/core/blob/master/CONTRIBUTING.md
[-] I have searched the existing issues and I'm convinced that mine is new.
Is your feature request related to a problem? Please describe.
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]
Describe the solution you'd like
A clear and concise description of what you want to happen.
(e.g. I would like an input field in the /ui/firewall/alias which would add .... to ....)
Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.
Additional context
Add any other context or screenshots about the feature request here.

empty template, if the intend is to open a ticket for acme client, please do so in plugins containing all relevant information.

Not really sure what happened to all the stuff I entered but I'll get that remedied and submitted in the right section. Sorry about that!

no problem, things happen :)


Correct version of kube

Regarding Installation and "Select a version of kube along with the generated k8s-openapi types corresponding for your cluster version:"
How do I find out the correct versions in my cluster?

kubectl version --short should show versions of your client and server. Something like
Client Vesion: v1.19.4
Server Version: v1.20.2

Use the version matching your server version.


Unauthenticate health check status

For unauthenticated users, what should be in the payload? Spring boot has this policy on HTTP health endpoint format and access restrictions. It sounds reasonable to me. Do we need to follow the similar approach or doing something different? We need to spec this.

I think you are referring to https://docs.spring.io/spring-boot/docs/current/reference/html/production-ready-monitoring.html#production-ready-health-access-restrictions
This sounds reasonable. I'd propose that the spec states that:

The HTTP health endpoint SHOULD be accessible for unauthenticated users
The producer SHOULD discard any data from the payload (leaving only the global outcome) for unauthenticated users
The producer SHOULD discard any data from the payload for authenticated users if they have not enough privilege/permissions to read such data

Maybe we should carry over the protocol spec document so we can PR it here?

@jmesnil yes. @heiko-braun can you bring over the protocol spec to this repo?

@Emily-Jiang The protocol spec resides in spec/protocol-wireformat.adoc
@jmesnil Feel free to submit a PR including the proposed changes above

I am going to add to @jmesnil lists:

Unauthenticated requests should have no payload, but leave the status mapping intact

And going to drop:

The producer SHOULD discard any data from the payload for authenticated users if they have not enough privilege/permissions to read such data

Introducing authz priviledges is too complex IMO.

Carried over from #47:

For reference, Spring supports authentication for the health check endpoints generated by their actuators. As noted, it acts as a filter on the returned payload (it has no bearing on which checks are executed). Unauthenticated requests get the sparest of answers {"status":"UP"} or {"status":"DOWN"}. You only get the details if you're logged in. To me, that is the difference between use by automation (up/down) and use by a human (what exactly is going on with this server...).

This feature belong to a broader scope. Idealy it should addressed by an MP "umbrella" spec

Why does this one closed without any resolution?


show latest tweet on room list page

---
Want to back this issue? **[Post a bounty on it!](https://www.bountysource.com/issues/22504668-show-latest-tweet-on-room-list-page?utm_campaign=plugin&utm_content=tracker%2F9578624&utm_medium=issues&utm_source=github)** We accept bounties via [Bountysource](https://www.bountysource.com/?utm_campaign=plugin&utm_content=tracker%2F9578624&utm_medium=issues&utm_source=github).

widget example http://codepen.io/anon/pen/mJBXEb


Compilation error after update Arduino 1.8.16 and TensyDuino 1.55

Hi Fernando
Today I have updated to Arduino 1.8.16 and TensyDuino 1.55, which include last version of your library, but I encountered  a compilation error below when I try to compile your blink example:

Arduino : 1.8.16 (Windows 7), TD: 1.55, Carte : "Teensy 3.6, Serial, 180 MHz, Faster, US English"
C:\Program Files (x86)\Arduino\hardware\teensy\avr\libraries\TeensyThreads\TeensyThreads.cpp:29:25: fatal error: utils/debug.h: No such file or directory
compilation terminated.
Erreur de compilation pour la carte Teensy 3.6

Please remove the offending line from TeensyThreads.cpp:
#include "utils/debug.h"
I don't know how it got there. It seems to have be put in by mistake during
a pull request.
On Thu, Sep 16, 2021 at 12:04 PM BOBILLIER @.***> wrote:

Hi Fernando
Today I have updated to Arduino 1.8.16 and TensyDuino 1.55, which include
last version of your library, but I encountered a compilation error below
when I try to compile your blink example:
Arduino : 1.8.16 (Windows 7), TD: 1.55, Carte : "Teensy 3.6, Serial, 180
MHz, Faster, US English"
C:\Program Files
(x86)\Arduino\hardware\teensy\avr\libraries\TeensyThreads\TeensyThreads.cpp:29:25:
fatal error: utils/debug.h: No such file or directory
compilation terminated.
Erreur de compilation pour la carte Teensy 3.6
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/ftrias/TeensyThreads/issues/29, or unsubscribe
https://github.com/notifications/unsubscribe-auth/ABRWXIXKH3436NJQXFP33GLUCIIR5ANCNFSM5EFDGYHA
.
Triage notifications on the go with GitHub Mobile for iOS
https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675
or Android
https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

TeensyDuino 1.56 in Beta - fixed code for Paul could be included for next release