ignoring / concatenating HDF5 chunks

Some netcdf4 / HDF5 files are created with really tiny chunks (<100kB) which with kerchunk often result in a task graph that is way too big for the scheduler (in my case: ~13000 files result in a task graph with >1000000 tasks). Additionally, the variables per file are usually smaller than the standard chunk size (<100MB, in my case ~2MB), but being able to concatenate / ignore those chunks so the "virtual" zarr has one chunk per variable per file would help already.
Is there a way to do this with kerchunk (current or future), or does the file layout force me to rewrite the data?
example encoding
{'zlib': True,
 'shuffle': False,
 'complevel': 6,
 'fletcher32': False,
 'contiguous': False,
 'chunksizes': (1, 100, 100),
 'source': '...',
 'original_shape': (1, 1062, 680),
 'dtype': dtype('int16'),
 '_FillValue': -32768,
 'scale_factor': 0.01,
 'add_offset': 273.15}

Firstly, if the dask graph size is the ultimate problem, as opposed to creating and reading the very large JSON reference file (where I recommend Zstd compression, by the way), then you can simply pass chunks={...} to xarray, meaning that each dask task can have a partition size much bigger than the physical storage chunk size. Since fsspec loads many chunks concurrently (for http, s3, gcs, abfs), you may well get good data throughput this way.
The "earthbigdata" example shows how you might have one chunk per input file, in that case for TIFF data (so only one variable per file). You require a codec that says how to go from the file to desired output, and this is all totally doable for any file file type that we can read into python in any way, including HDF5.
The performance of reading these files will depend on the access pattern. Since we can do nothing about the internal chunking, there will probably be fail cases where reading is very slow; or that performance depends heavily on various levels of caching. None of this might be important in the case that the standard approach simply doesn't work, but whole-file chunks does.
Separately. @d70-t has been thinking about concatenating chunks, but the context is different: that case is about concatenating the bytes of uncompressed pieces, possibly of different arrays shapes, to make larger chunks. You would still need to store all the individual offsets.

Please report back on your use case when you have something nice working!


Don't assume numpy/pandas is available in guess_param_types

Implementing the suggestion in https://github.com/holoviz/param/pull/629#discussion_r876410658, needs https://github.com/holoviz/param/pull/629 to be merged.
guess_param_types is actually not used at all across HoloViz and not documented. The PR that added this utility - https://github.com/holoviz/param/pull/317 - says that it "has cropped up several times and has resulted in some level of code duplication" and points to HoloViews Streams.define classmethod which indeed could make use of this utility, since it's basically inlining a partial implementation of it.
https://github.com/holoviz/param/pull/317 provides more details about the usefulness of this utility, and on the conversions it doesn't yet support.

Codecov Report

Merging #630 (fa88e59) into master (d960230) will increase coverage by 1.42%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##           master     #630      +/-   ##
==========================================
+ Coverage   82.91%   84.34%   +1.42%
==========================================
  Files           5        5
  Lines        3073     3085      +12
==========================================
+ Hits         2548     2602      +54
+ Misses        525      483      -42

Impacted Files
Coverage Δ

param/parameterized.py
83.84% <100.00%> (+0.51%)
:arrow_up:

numbergen/__init__.py
84.21% <0.00%> (+3.50%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update d960230...fa88e59. Read the comment docs.


bpf: daemon: Replace run_probes.sh with Golang code

Implement all BPF probes as a part of Cilium code instead of
using Bash script.
Related-to: #1600
Signed-off-by: Michal Rostecki<EMAIL_ADDRESS>

This change is

Coverage decreased (-0.07%) to 43.411% when pulling<PHONE_NUMBER>0dc24287aff44fdd63ae2f573ed2f4 on mrostecki:run-probes-golang into 6bdf6d13ea8aea7586cc64fb02777a10d27065df on cilium:master.

test-me-please

test-me-please

@aanm @borkmann PTAL

test-me-please

test-me-please

test-me-please

@joestringer @nebril Thanks for review. Your comments are addressed.

18:46:32 START: probes_test.go:57: ProbesSuite.TestProbeKernelConfig
18:46:32 probes_test.go:61:
18:46:32     c.Assert(err, IsNil)
18:46:32 ... value *errors.errorString = &errors.errorString{s:"missing kernel configuration"} ("missing kernel configuration")
18:46:32
18:46:32 FAIL: probes_test.go:57: ProbesSuite.TestProbeKernelConfig
18:46:32
18:46:32 START: probes_test.go:76: ProbesSuite.TestProbeRunLl
18:46:32 PASS: probes_test.go:76: ProbesSuite.TestProbeRunLl	0.531s
18:46:32
18:46:32 START: probes_test.go:40: ProbesSuite.TestReadKernelConfiguration
18:46:32 probes_test.go:43:
18:46:32     c.Assert(err, IsNil)
18:46:32 ... value *errors.errorString = &errors.errorString{s:"missing kernel configuration"} ("missing kernel configuration")
18:46:32
18:46:32 FAIL: probes_test.go:40: ProbesSuite.TestReadKernelConfiguration

That's really weird. It means that the test VM doesn't have kernel configuration anywhere...

@mrostecki I suspect that's more common than you might think. The existing code handles such situations gracefully, by just warning about it and not calling Fatal. I suggest we retain that behaviour.

@joestringer Yes, you are right.
I modified the unit test. I pushed an example config which is going to be used when no config is available. That should fix the issue on Travis and let us test the functionality anyway.
Does this PR look good for you in the current shape? If yes, I will squash commits.
I think I will follow up later with more tests (which will check what was actually written in buffers) and I will try to simplify the logic how warnings and infos are handled later by daemon. Maybe I will be able to get rid of info and warning files.

test-me-please

test-me-please

test-me-please

test-me-please

test-me-please

test-me-please

test-me-please

test-me-please

Sorry for squashing this PR, despite the fact that it's not approved, but there were already too many commits and it was problematic to rebase this branch.
I don't think it will make review harder. All comments should be addressed.

test-me-please

@mrostecki I'm going to move this into wip state since it hasn't been worked on for a while.

test-me-please

test-me-please

test-me-please

Be aware that #6921 reworks some of this code (again).

I'm fine with waiting until #6921 gets merged


continuous webjobs logs are not persisted to application logs as per site settings

Hello,
According to the documentation here:
https://github.com/projectkudu/kudu/wiki/Web-jobs#logging
"For continuous WebJobs - Console.Out and Console.Error are routed to the "application logs", they will show up as file, blob or table storage depends on your configuration of the application logs (similar to your Website)."
If I use the command
$ azure site log tail sitename
I am able to see the logs as they are currently running. However these logs are not written to D:\home\LogFiles\Application
The only logs I am able to retrieve for a continuous webjob is the first 100 lines per execution at
D:\home\data\jobs\continuous\jobname\job_log.txt
Because this file stops logging after 100 lines this is the only log I am able to retrieve for my continuous webjob:
[10/13/2015 19:49:08 > 13a748: WARN] Reached maximum allowed output lines for this run, to see all of the job's logs you can enable website application diagnostics
[10/13/2015 20:09:53 > 13a748: SYS INFO] WebJob is still running
[10/14/2015 07:33:10 > 13a748: SYS ERR ] Job failed due to exit code 1
right before 07:33:10 the application would have emitted a stack trace to the console so that I could see the reason for the crash. However because this log is truncated I am unable to tell why it crashed from this log.
The main problem I am having is that the logs for this webjob should be available in D:\home\LogFiles\Application as per the documentation. Note that I do have diagnostic logs turned on in my webapp and they are

Steps to reproduce:

Create a webapp and turn on diagnostic logs as per above image

Add a continuous webjob to D:\home\site\wwwroot\App_Data\jobs\continuous\testjob here is a test script I used:

app.js:
var startDate = new Date();
var workerId = startDate.getTime();
var index = 0;
var pid = process.pid;

console.log('started workerid=', workerId, 'pid=', pid);

function run() {
    index++;
    console.log('log workerid =', workerId, 'pid=', pid, 'index=', index, 'start=', startDate.toISOString(), 'now=', (new Date()).toISOString());
    setTimeout(function(){
        run();
    }, 60000);
}

run();

use the azure cli to see that the output of the continuous job can be streamed:

$ azure site log tail sitename
...
2015-10-15T20:14:32  PID[2868] Information log workerid =<PHONE_NUMBER>697 pid= 3752 index= 1 start= 2015-10-15T20:14:32.697Z now= 2015-10-15T20:14:32.712Z
2015-10-15T20:15:32  PID[2868] Information log workerid =<PHONE_NUMBER>697 pid= 3752 index= 2 start= 2015-10-15T20:14:32.697Z now= 2015-10-15T20:15:32.724Z
2015-10-15T20:16:32  PID[2868] Information log workerid =<PHONE_NUMBER>697 pid= 3752 index= 3 start= 2015-10-15T20:14:32.697Z now= 2015-10-15T20:16:32.724Z
Note that the above output tells me that the logs are being output properly, however they are not being persisted in the application logs D:\home\LogFiles\Application

look in the application logs to try to find the logs by looking at the contents of D:\home\LogFiles\Application>

Expected results:
As per the documentation at https://github.com/projectkudu/kudu/wiki/Web-jobs#logging I would expect to see the non-truncated logs in the application logs.
Actual results:
Only truncated logs are available at D:\home\data\jobs\continuous\jobname\job_log.txt which prevents us from diagnosing long running continuous job crashes because the logs do not extend past 100 lines.
Other notes:
azure-cli seems to be able to tail off the log files from but I am unsure where these logs are available on the filesystem if at all.
I believe this is a bug however it could be that I just am unsure of the location of where these log files are stored.
Also note:
to test to see if this is just a problem with logging node.js specific jobs I created a bash script which also does not seem to log to the application logs. Here is the bash script I tested:
run.sh
START=`date +"%Y-%m-%dT%H:%M:%SZ"`
WORKER=`date +%s`
COUNT=0
while true
do
  COUNT=$((COUNT+1))
  echo worker id=$WORKER index=$COUNT start=$START now=`date +"%Y-%m-%dT%H:%M:%SZ"`
  sleep 1m
done

Any help is appreciated as I am having a difficult time solving crashes that occur late at night because I am unable to log exceptions.
Originally I discussed this issue here:
https://github.com/projectkudu/kudu/issues/1744
also here:
https://github.com/projectkudu/kudu/issues/1733#issuecomment-146992336
Tagging @mathewc  as per https://github.com/projectkudu/kudu/issues/1744#issuecomment-148558231

+1: Just wanted to say that we need this issue fixed badly as well. It's surprisingly difficult just to get the console output of our jobs.
There is logging to Blob/Table Storage, but they're comparatively hard/slow to read given that the blobs/rows are split by instance and time. I'd really gotten used to the nice format the job_logs are in.

What if we added an App Setting to set the max number of lines, with 0 meaning no max?

@davidebbo sorry I missed this post here.. I would say that yes this would be an acceptable solution.
One could simply manually recycle a process to truncate the log manually.  However I feel like if the settings.job file had similar settings to iis node:
https://github.com/tjanczuk/iisnode/blob/master/src/samples/configuration/iisnode.yml#L118-L126
where one could limit the amount of logging automatically that might be better.  I think the complication in this would be that maybe the jobs don't support having multiple log files per pid.

Has this issue not been solved yet? I get the "Reached maximum allowed output lines for this run, to see all of the job's logs you can enable website application diagnostics" in my webjobs as well and have not been able to solve it...

This issue is really disappointing for us.  Being able to look at the console to see what is going on currently with a continuous (long running) job was going to be a very nice feature for us, but because of this limit, after the 1st hour or so we hit the "Reached maximum allowed output lines for this run, to see all of the job's logs you can enable website application diagnostics".  We don't want to store this type of data in a hard-to-access blob or table storage.  Just being able to glance the console and see the last X number of lines was what we wanted and what we thought we were getting.  Even stopping and restarting the continuous job doesn't refresh the log.

This really needs to be fixed, working with web jobs is quite frustrating atm. It should at least be possible to clear the log, allowing for new messages to show up.

same problem.

Actually, this works today, but when configuring logs to go to File System, they go to a different LogFiles folder: D:\home\LogFiles\SiteExtensions\Kudu. If you need to stream them, you can run (log in with your deployment credentials):
curl https://{app}.scm.azurewebsites.net/api/logstream/SiteExtensions/Kudu -u YourDeploymentUser

Or if using azure site log tail (which wraps the same API), pass -p SiteExtensions/Kudu to tell it to stream from that path.

Would really like to see this issue fixed; this makes blob storage of logs a useless feature as I am hitting the
"WARN Reached maximum allowed output lines for this run, to see all of the job's logs you can enable website application diagnostics"
very quickly; this only working with file storage is a poor work around.

@rholsman you need to enable application logs per the message you're getting. Unless you're saying that you are seeing this message in the app logs that you send to blobs? That should not happen.

Was there no resolution to this ? Is it possible to get some kind of option to auto-truncate the webjob logs for continuous web jobs. Seeing the first 100 lines of a log for a continuous webjob is of little use.
In our case we have full diagnostic logs in our log server - we don't need a long running history from the webjob logs. In the case where we are not seeing logs in the log server we need to webjob logs to see the current state of the job.

@graemechristie the recommended course of action seems to be to configure the WebApp log to Blob storage instead.
It seems from @davidebbo's comment that the reason for showing the first 100 lines rather than the latest 100 is done to avoid continuously writing to the filesystem, which affects overall performance? I think that makes sense. We had an issue a while back where the filesystem was being written to a lot by webjobs, and it very severely degraded the web app performance.
I ended up using Serilog with the Elasticsearch and Diagnostics.Trace sinks (with the Trace sink output going to Blob storage). I also use a rolling file sink for webjobs (which are sort of separate from the app now) - if the log file size goes over N MB, it's recreated.


Erroneous parsing of decorators in JSDoc code blocks

When using deno doc --json, example code blocks that contain decorators prove to be problematic for the parser. I'm not sure if this problem persists in --html or in the pretty-printed format in the CLI, but I would assume so since they all use the same parser under the hood AFAIK.
It stems from the @ prefix, which has also been a long running problem with the built-in JSDoc parser used by VSCode's intellisense, and a lot of other projects that incorporate JSDoc comment parsing in their logic. The parser gets confused and thinks the decorator is actually a malformed JSDoc tag, leading to unexpected (and usually terribly formatted) results.
Here's an example from the module doc comment in a JSR package of mine. Note the "unsupported" tags, particularly the last one. You can see it's parsing tags from inside an example code block.
{
  "version": 1,
  "nodes": [
    // ...
    {
      "name": "",
      "location": {
        "filename": "file:///workspaces/decorators/packages/types/mod.ts",
        "line": 1,
        "col": 0,
        "byteIndex": 0
      },
      "declarationKind": "export",
      "jsDoc": {
        "tags": [
          {
            "kind": "module"
          },
          {
            "kind": "unsupported",
            "value": "@log(\"class\") class Example {\n// ...\n"
          },
          {
            "kind": "unsupported",
            "value": "@log(\"method\") method() { return \"foo\" }\n"
          },
          {
            "kind": "unsupported",
            "value": "@log(\"getter\") get field() { return 42 }\n}\n```"
            // ---- the tell-tale sign its in an example code block  ^^^
          }
        ]
      },
      "kind": "moduleDoc"
    }
  ]
}

Just checked the output of deno doc --html and it is indeed affected by this issue. However, since the HTML doesn't render unsupported tags, the problem manifests itself as broken example code blocks. Notice in the screenshot below that the example cuts off before the class body for Config - that's because the very first line in the example class has a decorator on it.


Change sysctl from template to sysctl module

Overall Review of Changes:
Changed from configuring sysctl with template to using sysctl module. This also changes it to write to /etc/sysctl.conf instead of overwriting the rhel 8 default symlink at /etc/sysctl.d/99-sysctl.conf
Issue Fixes:
#230
Enhancements:
Allows idempotent sysctl tasks
How has this been tested?:
Tested against RHEL 8 server

I know I've talked with you guys before on the Discord about changing this and auditd to not use these templates, mostly for idempotence reasons, but the current way sysctl is setup is causing issues in my environment. This is because we have configurations in /etc/sysctl.conf, which we we're not really able to change, and overwriting the symlink causes /etc/sysctl.conf to not be loaded in. If you guys don't want to use this, I'll just maintain it separately, but I wanted to at least try to get this merged

hi @JacobBuskirk
Thank you for raising the issue and PR. While we understand your requirement, i wonder if just making the file template output a variable to would improve your stance. Meaning it can be put into any file you need without hitting your current config.
We moved to the template and notify method on all the repositories we maintain into improve the speed on large environments which is significant on clients which, we have running 000's of end points.
Let us see what we can do to accommodate as many options as possible.
many thanks
uk-bolly

It doesn't really work for me, since changing the file name would mean that it no longer has precedence over everything else. I get the desire to increase speed, but for sysctl this creates issues. Overwriting the default symlink at /etc/sysctl.d/99-sysctl.conf stops anything in /etc/sysctl.conf from applying, so the right way to do this would be writing to /etc/sysctl.conf. Overwriting the entire /etc/sysctl.conf file also has issues if you have other configurations in it, hence the need to change to the sysctl module.
Like I said I'm also fine with maintaining this separately.
One other thing I wasn't sure how to handle in this, the /etc/sysctl.d/99-sysctl.conf symlink needs recreated. What I have would work for any server where this role would be running for the first time, but if this has been run with the current code then the symlink would have already been overwritten with a file, so it needs added back. Regardless of whether you accept the change from the template to the sysctl module, the role probably shouldn't be overwriting the default /etc/sysctl.d/99-sysctl.conf symlink.

hi @JacobBuskirk
Thank you for the feedback it is always good to get feedback and see how we can adapt. We build and test these systems on completely clean builds where the sysctl.conf is blank.
I have just been testing this and you can rename if you wish rather than amending and still get the precedence.
e.g.
* Applying /etc/sysctl.d/99-sysctl.conf ...
* Applying /etc/sysctl.d/99a-sysctl.conf ...

This allowed the symlink and a later file to be added ensuring that we keep the system intact.
I hope this helps some way but we will discuss it further here.
thanks again
uk-bolly

What output do you get if you run sysctl --system? When I have it setup this way it indicates that the symlink takes precedence over /etc/sysctl.d/99a-sysctl.conf
ll /etc/sysctl.d/
total 4
-rw-------. 1 root root 2009 Sep 28 16:32 99a-sysctl.conf
lrwxrwxrwx. 1 root root   16 Sep 28 16:32 99-sysctl.conf -> /etc/sysctl.conf

sysctl --system

* Applying /etc/sysctl.d/99a-sysctl.conf ...
kernel.kexec_load_disabled = 1
fs.protected_symlinks = 1
fs.protected_hardlinks = 1
kernel.dmesg_restrict = 1
kernel.perf_event_paranoid = 2
kernel.randomize_va_space = 2
kernel.core_pattern = |/bin/false
net.ipv4.conf.default.accept_redirects = 0
net.ipv6.conf.default.accept_redirects = 0
net.ipv4.conf.all.send_redirects = 0
net.ipv4.icmp_echo_ignore_broadcasts = 1
net.ipv4.conf.all.accept_source_route = 0
net.ipv6.conf.all.accept_source_route = 0
net.ipv4.conf.default.accept_source_route = 0
net.ipv6.conf.default.accept_source_route = 0
net.ipv4.conf.all.forwarding = 0
net.ipv6.conf.all.forwarding = 0
net.ipv6.conf.all.accept_ra = 0
net.ipv6.conf.default.accept_ra = 0
net.ipv4.conf.default.send_redirects = 0
net.ipv4.conf.all.accept_redirects = 0
net.ipv6.conf.all.accept_redirects = 0
kernel.unprivileged_bpf_disabled = 1
kernel.yama.ptrace_scope = 1
kernel.kptr_restrict = 1
user.max_user_namespaces = 0
net.ipv4.conf.all.rp_filter = 1
net.core.bpf_jit_harden = 2
* Applying /etc/sysctl.conf ...
kernel.randomize_va_space = 2
<removed rest of output>

hi @JacobBuskirk
I've delved into this further and you are spot on, sysctl.conf is the final file to be loaded in every instance. I can understand why the solution does not work for you.
Let me discuss this further internally, this is indeed interesting way angle. I will see if we can get feedback on from other too on how they manage and maintain these settings.
Thanks again for working with us.
uk-bolly

@uk-bolly Appreciate you taking a look at it. Just for some more information, we had an issue with our Ansible Automation Platform service, which we ended up putting in a ticket with RedHat for. They found that some settings in /etc/sysctl.conf weren't being applied, and it was RedHat who told us that the /etc/sysctl.d/99-sysctl.conf file on a default install is a symlink to /etc/sysctl.conf. They said we needed to restore the symlink to pull in settings /etc/sysctl.conf.

@uk-bolly Appreciate you taking a look at it. Just for some more information, we had an issue with our Ansible Automation Platform service, which we ended up putting in a ticket with RedHat for. They found that some settings in /etc/sysctl.conf weren't being applied, and it was RedHat who told us that the /etc/sysctl.d/99-sysctl.conf file on a default install is a symlink to /etc/sysctl.conf. They said we needed to restore the symlink to pull in settings /etc/sysctl.conf.

Hi @JacobBuskirk
I have raised a new branch that has an alternate approach. Using the module as you are but also allowing users to set their own sysctl end point. This is still checked by the audit for both config and running config.
branch: systcl_and_collections (collections change due to galaxy changes and older ansible versions)
Hope this works for you.
Many thanks
uk-bolly

@uk-bolly Looks like that'll work. Thanks!


AD-165751: Retrieve HTML Content from STAX using GraphQL

2 Integration tests currently not working but all features working as intended.

Replaced by AD-165751-v2


add landing page, display page. integrate pages together, configure grafana

Great amount of functionality added here sorry for all the comments but I think the issues are minor overall. Please address and notify when you want a re-review

Can you please push up your changes to revert the springboot app / jar to previous commit?


lang: Anchor IDL should be uploaded to program binary instead of separate PDA

To date, only 25% of Anchor programs have uploaded their IDL 🤯  (1,324 programs out of 5,335 anchor programs currently on mainnet).
If anchor IDLs were uploaded with program binaries, then we would have 100% IDL adoption, leading to increased transparency on Solana.
Instead of requiring developers to write the compressed IDL bytes to the IDL address, anchor build should write the zipped bytes to an ELF section called .anchor.idl. Example: https://github.com/nifty-oss/asset/blob/main/include-idl/src/lib.rs
Developers should still be able to opt-out of this default behavior and deploy the IDLs the old way.

Overall, this would be great.

If anchor IDLs were uploaded with program binaries, then we would have 100% IDL adoption, leading to increased transparency on Solana.

It would increase the percentage, but I don't think we'd ever see 100% IDL adoption, especially if we give people the chance to opt out of this behavior.

Developers should still be able to opt-out of this default behavior and deploy the IDLs the old way.

Is this for backwards compatibility? We have to get rid of the IDL instructions when the IDL exists in the binary, otherwise the binary size will grow significantly.

It would increase the percentage, but I don't think we'd ever see 100% IDL adoption, especially if we give people the chance to opt out of this behavior.

Yes this is true. I think aiming for 50% adoption or higher over next year would be attainable, with this change alone.
On backwards compatibility, I am agnostic.
On removing IDL instructions from the binary, this is super exciting because then deploying IDLs with your binary is essentially "free". In other words - this shouldn't change deployment costs 😄 ! That should make adopting this easier🤞

Core does not recommend putting IDLs in program binary. They are trying to shave all bits of program binaries down as much as possible.
Accordingly, working on a different approach. Foundation is working on getting a program to upload program metadata on-chain like IDL, solana-verify args, and more. It will handle buffer history versioning too. It will be immutable ASAP.

Here is roughly the program spec we are thinking about: https://solanafoundation.notion.site/Program-Metadata-128d36dad52d801e974fcc3a30736506

Okay, after further discussion with Anza VM devs here, it seems that this was a miscommunication. Core seems to be fine with having arbitrary metadata in the ELF section headers.
Given how much friction is caused by uploading IDLs via IdlInstructions, I think uploading IDLs in section headers is the better way to go.
@febo and I have published solana-include-idl and solana-include-idl-cli https://github.com/nifty-oss/asset/blob/main/include-idl/README.md under solanafoundation are testing it out with OreV2 & some other programs. We will be working to support this new place for IDLs in the Explorer over the next 2-3 weeks.
An additional major benefit of uploading IDLs into binaries is that we can avoid the social consensus issue of getting program developers to use an IDL program.

If the goal is adoption, why not include the IDL transactions to the anchor deploy and solana program deploy commands to make it opt-out rather than opt-in?
Requiring to fetch the whole program data account just to read a small portion of it is likely to increase the load for sites like explorers quite a bit, especially when compared to just fetching the PDA account, which pretty much only contains the required data. Maybe we need a new RPC method to fetch IDLs from program data accounts?
I'm still fine with it though, just noting some downsides/concerns.

Adding IDL transactions to anchor deploy makes sense, I thought it used to be included FWIW
Adding IDL transactions to solana program deploy is theoretically fine, but requires one of 2 difficult options:

Either every program must have IdlInstructions (injected by Anchor, or manually added to native programs)
Or we have some universal IDL program & we have to work on migration and education

I don't think either of those options are particularly great solutions compared to shoving ELF metadata into program headers for now.

Requiring to fetch the whole program data account just to read a small portion of it is likely to increase the load for sites like explorers quite a bit

Yup this is true. I'm working on more efficient parsers. Worst case, we host a public API to serve IDLs to the Explorer.

Maybe we need a new RPC method to fetch IDLs from program data accounts?

Core said "no way Jose" to new RPC methods until RPC v2. Probably best solution to host an API.
Thanks for the feedback Acheron!


[QUESTION] Yarn Dev Setup

Hi,
When I try to run it in dev mode doing (yarn dev) i am having this kind of error in the frontend
Module not found: Error: You attempted to import /Users/x/Desktop/flowise/Flowise/node_modules/react-refresh/runtime.js which falls outside of the project src/ directory. Relative imports outside of src/ are not supported.
You can either move it inside src/, or add a symlink to it from project's node_modules/.
ERROR in ./src/App.js 1:40-126
Module not found: Error: You attempted to import /Users/x/Desktop/flowise/Flowise/node_modules/react-refresh/runtime.js which falls outside of the project src/ directory. Relative imports outside of src/ are not supported.
You can either move it inside src/, or add a symlink to it from project's node_modules/.
ERROR in ./src/api/apikey.js 1:40-126
Module not found: Error: You attempted to import /Users/x/Desktop/flowise/Flowise/node_modules/react-refresh/runtime.js which falls outside of the project src/ directory. Relative imports outside of src/ are not supported.
You can either move it inside src/, or add a symlink to it from project's node_modules/.
ERROR in ./src/api/assistants.js 1:40-126
Module not found: Error: You attempted to import /Users/x/Desktop/flowise/Flowise/node_modules/react-refresh/runtime.js which falls outside of the project src/ directory. Relative imports outside of src/ are not supported.
You can either move it inside src/, or add a symlink to it from project's node_modules/.
ERROR in ./src/api/chatflows.js 1:40-126
Module not found: Error: You attempted to import /Users/x/Desktop/flowise/Flowise/node_modules/react-refresh/runtime.js which falls outside of the project src/ directory. Relative imports outside of src/ are not supported.
You can either move it inside src/, or add a symlink to it from project's node_modules/.
ERROR in ./src/api/chatmessage.js 1:40-126
Module not found: Error: You attempted to import /Users/x/Desktop/flowise/Flowise/node_modules/react-refresh/runtime.js which falls outside of the project src/ directory. Relative imports outside of src/ are not supported.
You can either move it inside src/, or add a symlink to it from project's node_modules/.
ERROR in ./src/api/client.js 1:40-126
Module not found: Error: You attempted to import /Users/x/Desktop/flowise/Flowise/node_modules/react-refresh/runtime.js which falls outside of the project src/ directory. Relative imports outside of src/ are not supported.
You can either move it inside src/, or add a symlink to it from project's node_modules/.
ERROR in ./src/api/config.js 1:40-126
Module not found: Error: You attempted to import /Users/x/Desktop/flowise/Flowise/node_modules/react-refresh/runtime.js which falls outside of the project src/ directory. Relative imports outside of src/ are not supported.
You can either move it inside src/, or add a symlink to it from project's node_modules/.
/Users/x/Desktop/flowise/Flowise/node_modules/react-scripts/config/placeholder.js
Referenced from: /Users/x/Desktop/flowise/Flowise/.eslintrc.js
Any help ???

actually never seen these errors. but suggest you running on higher node version, do a fresh install and rebuild

still facing this error
to reproduce :

git clone https://github.com/FlowiseAI/Flowise.git
cd Flowise
create .env file for ui and server port 8080 and 3000
yarn install
yarn dev

node version : v20.2.0 ; yarn version : 1.22.11 running on mac m1
any idea ?

Did you run yarn install inside of the package directories by chance? I've seen similar happen in that situation.
Yarn install must only be run at the root dir, and only once.

Hi @automaton82  ,
I run it at the root dir just after cloning the project, and still facing these issues. yarn install and yarn dev

It's the one in the repo in .\Dockerfile


missing quote in unit-whitelist

I happened to notice the missing quote. Fixed

ty @cedmax, you have eagle eyes 🦅


Fix expected error text in assertion

It seems that a recent upgrade to librure would have changed the error text when grouping parentheses aren't balanced.
Changing the test below has made the tests pass on my machine with the latest librure

Ah, yes indeed, thank you!


ci/docs: Use commit sha for all non-master artifact uploads

Signed-off-by: Ryan Northey<EMAIL_ADDRESS>Commit Message: ci/docs: Use commit sha for all non-master artifact uploads
Additional Description:
Follow up from #13826
Risk Level:
Testing:
Docs Changes:
Release Notes:
Platform Specific Features:
[Optional Runtime guard:]
[Optional Fixes #Issue]
[Optional Deprecated:]

@lizan does this resolve the issues you raised on #13826 ?


[Compat Request] Ceramics Clay Bucket

So currently the Ceramics Clay Bucket disobeys hardcore buckets.
https://minecraft.curseforge.com/projects/ceramics

Probably like most custom buckets. This will be a problem if compatibility needs to be implemented for everything


Finish paper

Barring some bug fixing and if anyone wants to run any more benchmarks (using this repo), we should have all we need to write the paper.

Paper finished :) woo!


monitored params for WAIC fail for state-space model

Reproducible example below.
Problem has to do with what WAIC thinks it does (or doesn't) need monitored for WAIC, which it gets wrong in this case, I think because of the deterministic declarations for x[1,1] and y[1,1].
set.seed(0)
k <- 10
n <- 75
phi <- .9
p <- 0.5
y <- array(NA, c(n,k))

for(i in 1:n) {
    y[i,1] <- 1
    x <- 1
    for(t in 2:k) {
        x <- rbinom(1, prob=x*phi, size=1)
        y[i,t] <- rbinom(1, prob=p*x, size=1)
    }
}

library(nimble)

(N <- dim(y)[1])
(T <- dim(y)[2])

code <- nimbleCode({
    p   ~ dbeta(1,1)
    phi ~ dbeta(1,1)
    for(i in 1:N) {
        x[i,1] <- 1
        y[i,1] <- 1
        for(t in 2:T) {
            x[i,t] ~ dbern(phi * x[i,t-1])
            y[i,t] ~ dbern(p * x[i,t])
        }
    }
})

constants <- list(N = N, T = T)
data <- list(y = y)
inits <- list(x = (1^y), p = 0.5, phi = 0.5)
Rmodel <- nimbleModel(code, constants, data, inits)
Rmodel$calculate()    ## -935.7487

conf <- configureMCMC(Rmodel)
Rmcmc <- buildMCMC(conf, enableWAIC = TRUE)
compiledList <- compileNimble(list(model=Rmodel, mcmc=Rmcmc))
Cmodel <- compiledList$model; Cmcmc <- compiledList$mcmc

Cmodel$calculate()   ## -935.7487

set.seed(0)
out <- runMCMC(Cmcmc, 10000, WAIC = TRUE)

out$WAIC
## [1] NaN

Cmodel$calculate()
## -Inf

I should also have added, that if x is monitored also, then everything works fine.
e.g., WAIC's determination of "what needs to be monitored for WAIC to work" should also include x here (I believe)

Thanks a bunch for finding this and providing such a useful example Daniel!  You're correct that the issue here arises from calculating the WAIC after not monitoring x during the MCMC.  However, I'm not sure that the deterministic nature of x is the cause of the miscalculation.
Within the WAIC algorithm, we simulate values for un-monitored stochastic nodes that are downstream of the monitored parameters (here, monitored parameters are phi and p), and "upstream" of data nodes.  When x is not monitored, this means that the algorithm is simulating values for x conditioned on the sampled values for phi and p.  This model has the interesting feature that if an x[i,j] value is simulated to be 0, when y[i,j] is 1, then log( p(y[i,j] | x[i,j]) ) (as determined by model$calculate(y[i,j])) will be -Inf. However, this causes a problem, as the WAIC algorithm (as given in equation 12 of this paper) requires calculation of the posterior variance of the log predictive density, which involves simulating an x[i,j] value from each posterior value of phi and p, and taking the variance of log( p(y[i,j] | x[i,j]) ) over all such simulated x[i,j].  When any such log( p(y[i,j] | x[i,j]) ) = -Inf, this variance becomes NaN.
It seems as though the WAIC calculation is not designed to handle cases when e.g.  log( p(y[i,j] | x[i,j]) ) = -Inf.  These cases seem like they can can arise in models when intermediate stochastic nodes are not monitored, and simulated values for these nodes disagree entirely with the observed data.
It's not clear to me that there's a good fix for this, as in some sense the NaN value is the WIAC algo performing according to its definition.  One option would be to emit a warning if the WAIC returns an NaN value to alert the user that:
a) Some sets of monitored nodes, even if technically valid for calculating a WAIC value, can lead to NaN WAICs such as above
b) Even if all stochastic nodes are monitored, if inserting any set of posterior values for these nodes into the model ever leads to a model$calculate() value of -Inf (unclear when or if this would be the case but may be possible?), then again we would expect an NaN WAIC value
I'm open to any other suggestions for how to address this behavior, and please let me know if the explanation above was unclear as well.  Thanks!

@NLMichaud Thanks for the full report.  It sounds like there are some questions here about how we want WAIC to behave, although it does sound like the algorithm is operating as intended.
Just one more thing to point out, which is still disturbing to me.  Even if WAIC shouldn't work in this case, e.g. should return NaN for the reasons you explained.  What's troubling to me is that after calling Cmcmc$calculateWAIC(), it leaves the Cmodel object in a different state than originally.  Specifically, the "calculation" of WAIC overwrites the values of x in the model, presumably with the newly simulated values, which are no longer valid, and thus Cmodel$calculate() now returns -Inf.
One easy & quick fix for this, would be instead of the WAIC algorithm just saving part of the model state as currentVals <- values(model, sampledNodes), then restoring these values at the end with:  values(model, sampledNodes) <<- currentVals, changing these two lines to:
currentVals <- values(model)
## WAIC calculation here
values(model) <<- currentVals
model$calculate()

in a very defensive manner, so that regardless of what happens in the WAIC calculation (correct, NaN, or whatever!), the model state is guaranteed to be returned to the same, original, presumably valid state, that is was in before the WAIC "calculation".
What do you think @NLMichaud?  I'm happy to make this change, if you agree.

Great, will do, Nick.  Thanks for getting back.

@perrydv @NLMichaud @paciorek I've resolved the issue of the model changing, using Chris's suggestion of "saving" the model values and also the logProb values, rather than using a model$calculate().   This was fixed in PR #808.
I also added a warning message, when WAIC is calculated as NaN, indicating that latent states may also need to be monitored.


Q: filter a list of either monads

I have a list of Option<A,B> and I want to filter to have a list of A and a list of B
List<Option<A,B>> source = GetList();

var a =
    source.Select(x => x.Match(a => a, b => null).Where(x => x != null);

var b =
    source.Select(x => x.Match(a => null, b => b).Where(x => x != null);

Is there a better way of doing this?

Hi,
It depends. In general, the preferred way to do this would be:
var a = source.SelectMany(x => x.ToEnumerable());

However, this only works for the Some case, and for symmetry reasons I would stick an approach that works for both cases, when both cases are needed.
If this is a recurring pattern in your code base, you could consider defining specific helper methods for such purposes, e.g.:

// Allows you to write: source.SelectMany(option => option.WithoutValue().ToEnumerable());

public static Option<TException> WithoutValue<TValue, TException>(this Option<TValue, TException> option)
{
    return option.Match(_ => Option.None<TException>(), exception => exception.Some());
}

// Optionally, you could even write specific LINQ extensions:

public static IEnumerable<TValue> Values<TValue, TException>(this IEnumerable<Option<TValue, TException>> source)
{
    if (source == null) throw new ArgumentNullException(nameof(source));
    return source.SelectMany(option => option.ToEnumerable());
}

public static IEnumerable<TException> Exceptions<TValue, TException>(this IEnumerable<Option<TValue, TException>> source)
{
    if (source == null) throw new ArgumentNullException(nameof(source));
    return source.SelectMany(option => option.WithoutValue().ToEnumerable());
}

In general, I don't think you should be afraid to introduce a bit of custom functionality, to ease the integration with the rest of your code base - I typically do this all the time (and only the most general use cases are integrated back into Optional itself).
However, I do have some plans for the Either-version of Option, but am still considering how to make these as painless to consumers of the library as possible.

Ah I see, thanks! This is a better way of doing this.
Great to hear you have plans for the Either monad. I've been playing around with railway oriented programming so I'm definitely all ears.


Change feed processor retry mechanism on failure not working as expected

We are in the process implementing a change feed processor application using the java sql api. We are looking into creating a new container from the feed/ monitored container to optimize our read requirements. The primary container was partitioned to maximize our write requirements and the new container helps us with our read requirements(data loss is not tolerated).
We are using the azure-cosmos java sdk(version 4.38.1) and ours is a spring boot application. We have questions related to the error handling mechanism mentioned in this documentation.
Following was our initial implementation,
  @Bean
  public ChangeFeedProcessor changeFeedProcessor(CosmosAsyncContainer cosmosMonitoredContainer,
      CosmosAsyncContainer cosmosLeaseContainer, CosmosAsyncContainer materializedViewContainer)
      throws UnknownHostException {
    ChangeFeedProcessorOptions changeFeedProcessorOptions = new ChangeFeedProcessorOptions();
    changeFeedProcessorOptions.setStartFromBeginning(true);
    changeFeedProcessorOptions.setLeasePrefix("txn-materialized-view");

    return new ChangeFeedProcessorBuilder()
        .options(changeFeedProcessorOptions)
        .feedContainer(cosmosMonitoredContainer)
        .leaseContainer(cosmosLeaseContainer)
        .hostName(Inet4Address.getLocalHost().getHostName())
        .handleChanges((List<JsonNode> docs) -> {
          for (JsonNode document : docs) {
            materializedViewContainer.upsertItem(document).subscribe();
          }
        })
        .buildChangeFeedProcessor();
  }

Our starter class,
import com.azure.cosmos.ChangeFeedProcessor;
import javax.annotation.PostConstruct;
import javax.annotation.PreDestroy;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Component;

@Component
public class ChangeFeedProcessorRunner {
  @Autowired
  private ChangeFeedProcessor changeFeedProcessor;

  @PostConstruct
  public void start() {
    changeFeedProcessor.start()
        .doOnSuccess(a -> System.out.println("Change feed processor started successfully!!!"))
        .subscribe();
  }

  @PreDestroy
  public void stop() {
    changeFeedProcessor.stop().block();
  }
}

With this implementation the scheduler thread(worker) just submits the upsert operation to create/ update a record on the materialized view container on separate child threads(parallel). It does not wait for the creation or update activity to be completed.
If we were to encounter any runtime exceptions while persisting the data on the materialized view container, the retry guarantee mentioned in the documentation would not be met, since the scheduler thread is reactive or asynchronous in nature. It would exit the handle changes block as soon as the upsert tasks are all submitted. Also, updating lease container and moving on to the next batch.
The change feed processor is resilient to user code errors. That means that if your delegate implementation has an unhandled exception (step #4), the thread processing that particular batch of changes will be stopped, and a new thread will be created. The new thread will check which was the latest point in time the lease store has for that range of partition key values, and restart from there, effectively sending the same batch of changes to the delegate. This behavior will continue until your delegate processes the changes correctly and it's the reason the change feed processor has an "at least once" guarantee.
In order to solve this problem, we tried using the following two approaches,
Approach 1 - Using sequential or serial execution with block() instead of subscribe()
 @Bean
 public ChangeFeedProcessor changeFeedProcessor(CosmosAsyncContainer cosmosMonitoredContainer,
     CosmosAsyncContainer cosmosLeaseContainer, CosmosAsyncContainer materializedViewContainer)
     throws UnknownHostException {
   ChangeFeedProcessorOptions changeFeedProcessorOptions = new ChangeFeedProcessorOptions();
   changeFeedProcessorOptions.setStartFromBeginning(true);
   changeFeedProcessorOptions.setLeasePrefix("txn-materialized-view");

   return new ChangeFeedProcessorBuilder()
       .options(changeFeedProcessorOptions)
       .feedContainer(cosmosMonitoredContainer)
       .leaseContainer(cosmosLeaseContainer)
       .hostName(Inet4Address.getLocalHost().getHostName())
       .handleChanges((List<JsonNode> docs) -> {
         for (JsonNode document : docs) {
           materializedViewContainer.upsertItem(document).block();
         }
       })
       .buildChangeFeedProcessor();
 }

With this implementation the scheduler thread(worker) is blocked until the upsert operation is completed before proceeding to the next items in the list. Hence, this would adhere to the retry guarantee mentioned in the documentation.
But the downside of this approach, it is sequential or serial in terms of the execution. The application's ability to process data in parallel would be hampered leading to a degraded throughput.
Approach 2 - Using subscribe() but holding the scheduler thread after all the upsert tasks are all submitted
 @Bean
 public ChangeFeedProcessor changeFeedProcessor(CosmosAsyncContainer cosmosMonitoredContainer,
     CosmosAsyncContainer cosmosLeaseContainer, CosmosAsyncContainer materializedViewContainer)
     throws UnknownHostException {
   ChangeFeedProcessorOptions changeFeedProcessorOptions = new ChangeFeedProcessorOptions();
   changeFeedProcessorOptions.setStartFromBeginning(true);
   changeFeedProcessorOptions.setLeasePrefix("txn-materialized-view");

   return new ChangeFeedProcessorBuilder()
       .options(changeFeedProcessorOptions)
       .feedContainer(cosmosMonitoredContainer)
       .leaseContainer(cosmosLeaseContainer)
       .hostName(Inet4Address.getLocalHost().getHostName())
       .handleChanges((List<JsonNode> docs) -> {
         AtomicInteger successfulUpsert = new AtomicInteger(); // counter to keep track of successful upserts
         AtomicBoolean hasExceptionOccurred = new AtomicBoolean(false); // flag to indicate any runtime exception
         for (JsonNode document : docs) {
           materializedViewContainer.upsertItem(document)
               .doOnNext(a -> successfulUpsert.incrementAndGet())
               .doOnError(throwable -> hasExceptionOccurred.set(true))
               .subscribe();
         } // till this the scheduler runs in an asynchronous fashion

        // holding the scheduler thread until we get to know there were no exceptions and all of them succeeded
         while (docs.size() != successfulUpsert.get()) {
           if (hasExceptionOccurred.get()) {
             // throwing an exception to make the stop the scheduler and retry the same batch on a different thread
             throw new RuntimeException(
                 "Exception occurred while persisting data in materialized view container");
           }
         }
       })
       .buildChangeFeedProcessor();

With this approach, we are trying to hold the scheduler(worker) thread forcefully until all the child parallel threads are completed successfully to proceed to the next batch. If we were to encounter any exceptions, we throw a runtime exception which leads to retrying the same batch on a different scheduler thread(satisfying retry guarantee).
Also, the activity to submit the upsert operation happens in parallel. Once all the upsert operations are submitted, we are making the scheduler thread wait till the status is known.
Questions

We want to get an idea on the approach to be used to achieve the proper error handling and retry mechanism mentioned in the documentation(Approach 1 VS Approach 2).
We also want to get an idea on alternate mechanisms available in the library to signal the retry mechanism upon encountering a failure in the reactive or asynchronous implementation.
We want to get your opinion on approach 2 on our way of handling the retry guarantee and ensure we have covered all the cases(data loss cannot be tolerated in our case. We are dependent on the at least once guarantee delivery committed on the documentation).

We also came across this comment in the change feed processor samples wherein a time based mechanism was employed. In our case we cannot use that as it cannot be deterministic.

Hello @joshfree, do you have any update on this for us?

Sorry for the late reply. Here are some comments in response to your questions:

It is entirely up to you, both approaches appear to satisfy the requirement for per-batch built-in retries in response to change feed processor user code exceptions. Approach 1 is the more typical approach, as latency from serial processing is typically tolerable, and there is already parallelism by virtue of being able to deploy multiple delegates (workers) which will process in parallel across partition ranges. But approach 2 looks fine as well.

There are no other effective built-in mechanism for retrying a batch of changes in change feed, but perhaps you want to consider per-request retries for certain types of failure that have been introduced in SDK recently. For example, retriable writes, end-to-end timeout policy.

As above, it is not the typical approach, but if you really need to minimize the replication latency to materialised view, it looks fine.

Lastly, you may want to take a look at the Materialized views, now a built in feature in Cosmos DB, to see if it meets your need.


New release

Hello,
Thanks for your work!
Should we be expecting a new release so that we can make use of any recent changes? We're especially interested on the updates of the helper-pod configuration on the helm chart.

v0.0.23 is released. Thank you and any feedback is appreciated.


[CON-5276] make plugin compatible with 5.5

Coverage increased (+0.06%) to 42.308% when pulling a468289ef6ba6e406f3ffc87db3ffc9ba34c209e on CON-5276 into 59386cf18b43a5966987fc84cc916497831050f5 on master.


Plugin Replacement for Dynamic Font Mapping in react-native-unistyles v3?

Description
In version 2 of react-native-unistyles, I used a custom plugin to automatically map the fontWeight property to the corresponding fontFamily. This plugin was essential for dynamically and efficiently managing fonts in my project. However, after migrating to version 3, where plugin support has been removed, I can no longer implement this functionality in the same way.
I would like to know if there is an alternative way to achieve this in version 3, considering the removal of plugin support.

Steps to Reproduce

In version 2, create a plugin to map fontWeight to fontFamily
Apply the plugin in the project and observe that the mapping works automatically.

Snack or Repository Link (Optional)
No response
Unistyles Version
3.0.0-beta.1
React Native Version
0.76.3
Platforms
Android, iOS, Web
Expo
Yes

Hey it was asked multiple times already:
https://github.com/jpudysz/react-native-unistyles/discussions?discussions_q=plugin
Here is more details: https://github.com/jpudysz/react-native-unistyles/discussions/361
Long story short, right now there is no replacement. Could be in the future (maybe closer to stable release).
I will keep this issue open for transparency.


Tunnel Timeout?

Is there some kind of timeout built into this? I ask because it seems random, but while running some tests, on occasion, usually after a pause for a GET command I will get the "check your firewall settings" which again, has nothing to do with the firewall since it just worked a few minutes ago.
The only thing I can think of is some sort of timeout on the tunnel. Do I need to set up a keepalive to stop this from happening. Or is the tunnel itself just unstable?

You can use a service like PM2 or Forever to keep the tunnel running forever. I use forever all the time and my site stays up for weeks (even months)

The thing is that this is running as part of an application that is both the webserver and the tunnel. When the tunnel fails it takes down the entire application. This is due to an unhandled exception inside the tunnel code itself. I would suggest some sort of tunnel.crash callback or something similar since I could use that to restart the tunnel if that happens without it taking down the entire application.

Another part of the issue may just be a connection hickup, there's a PR open with a retry policy: #311.
Maybe you can upvote it to bring it to the maintainers attention?


Make a tyk authorization plugin/middleware for Casbin

tyk is a new open-source API gateway like OpenResty, APISIX, Kong, etc, but developed in Go: https://github.com/TykTechnologies/tyk

Authn plugin: https://tyk.io/docs/plugins/auth-plugins/
Plugin table: https://tyk.io/docs/plugins/supported-languages/
Custom plugins: https://github.com/TykTechnologies/custom-plugins#custom-gateway-plugins

Especially this one would be similar to what we need to do:

You can also refer to the plugins we develop for other API gateways: https://casbin.org/docs/en/middlewares

@JalinWang @imp2002
/cc @tangyang9464


Add saving_listeners arg to TrainSpec

System information

TensorFlow version (you are using): 1.12.0 but this feature does not exist on 1.13.x either nor 2.0.
Are you willing to contribute it (Yes/No): Yes

Describe the feature and the current behavior/state.
(Estimator|TPUEstimator).train() accepts a saving_listeners kwarg.  TrainSpec does not. It's strange because TrainSpec can be configured with every other argument to .train.
Will this change the current api? How?
Yes, it will add a saving_listeners arg to TrainSpec that can be passed down to .train.
Who will benefit with this feature?
Anyone expecting a consistent interface and a single place to configure training.

I've just noticed that these are then also missing from EstimatorSpec|TPUEstimatorSpec. I guess it would have to "trickle down" into them as well.

@jschneier,
Sorry for the delayed response. In the Tensorflow Version 2.x, since we use TF Keras predominantly and don't use Estimators much, can you please let us know if this Feature is still relevant? Thanks!


Duplicate Libraries on Load

Most recent build is creating multiple tabs for the same library for some reason. Checked the database and this appears to be a client-side issue.

Seems to be corrupted data on on the server-side. Only the dev organization seems to be affected.

Manually deleted the dev organization from the database and everything appears to be working now.


Update Readme File

Refactor the Readme file and make it more readable.

Let's try putting some Animations and/or symbols on the topics, for e.g: for SQL, we can use:

I feel like you'd want to use the above image for NoSQL databases, i.e. non-relational databases, which there aren't any resources for right now. It'd be confusing to have the SQL section have an image that says NO SQL!
In any case, I alphabetized the body topics and sub-topics, so it's easier to navigate. I've submitted a PR on my branch more-language-resources.

for me @djc2702 's suggestion makes more sense.   @SubhashKotra Do you want contribute submit a PR with those update.

I wanna write read me for this , as I am collecting resources for python and comp programming for this repo

@djc2702 and @Thuva4 I meant mentioning them with some logos, NOSQL logo was just an example for any NOSQL resource definitely not for a SQL resource. :-)

I want to contribute to the web development of the readme

I added a tutorial to inspire girls' programmers to start with django, here it is: #37


Addition to MPR package manager

Not precisely a package manager rather package helper like utility for debian/ubuntu derivatives.
Have added the git version to MPR, kindly review and comment.
MPR
MPR-Package

@Aeres-u99 thank you Kai for the contribution and your interest in HSTR!

*Kei :P and truly pleasure was mine!


Undetermined shared module version when importing a file from the module directly

Bug report

What is the current behavior?
When importing a file directly from a module (i.e. import "react-dom/test-utils") when it's configured to be shared via ModuleFedrationPlugin a warning with a nonsensical instruction may be emitted when running a build:
WARNING in shared module react-dom
No required version specified and unable to automatically determine one. Unable to find required version for "react-dom" in description file (/home/user/test/node_modules/react-dom/package.json). It need to be in dependencies, devDependencies or peerDependencies.

If the current behavior is a bug, please provide the steps to reproduce.
Using the following webpack config
"use strict";
const path = require("path");
const webpack = require("webpack");

module.exports = (env = {}) => ({
    entry: "./index.js",
    output: {
        publicPath: "auto",
        filename: "bundle.js",
        path: path.resolve(__dirname, "dist"),
    },
    mode: "production",
    plugins: [
        new webpack.container.ModuleFederationPlugin({
            name: "test",
            shared: {
                "react-dom": { singleton: true },
            },
        }),
    ],
});

And an index.js file with the content
import "react-dom/test-utils"

Along with a package.json file with the dependencies described in reproduces the behaviour. See the complete example in this repository.

What is the expected behavior?

I expect either the version of react-dom to be correctly deduced from package.json or a sensible warning/error message.
Other relevant information:
webpack version:  5.37.1
Node.js version: 16.0.0
Operating System: Arch Linux
Additional tools:

Expected, you want share react-dom, but request react-dom/test-utils

Maybe, but in my case it was a dependency of a dependency that imported react-dom/test-utils and I can't deduce that from the error message.

I have the same issue happening with @angular/common .
shared module @angular/common - Warning: No required version specified and unable to automatically determine one. Unable to find required version for "@angular/common" in description file (.../node_modules/@angular/common/package.json). It need to be in dependencies, devDependencies or peerDependencies.
In the package.json file there is no entry for @angular/common under those properties.

@sokra does webpack version 5.41.1 fix this issue?

No, you can send a fix

same problem.

Getting similar issue, any fixes?
WARNING in shared module react
No required version specified and unable to automatically determine one. Unable to find required version for "react" in description file (/Users/deepaknegi/Desktop/projects/email-template-ui/node_modules/@emotion/react/jsx-runtime/package.json). It need to be in dependencies, devDependencies or peerDependencies.

Possibly related https://github.com/mui-org/material-ui/issues/24013

@arrgson It is because of secondary point.
According to manfred amazing book.
The reason for this is the secondary entry point @angular/common/http which is a bit like an npm
package within an npm package. Technically, it’s just another file exposed by the npm package
@angular/common .
Unsurprisingly, @angular/common/http uses @angular/common and webpack recognizes this. For this
reason, webpack wants to find out which version of @angular/common is used. For this, it looks into
the npm package’s package.json ( @angular/common/package.json ) and browses the dependencies
there. However, @angular/common itself is not a depdency of @angular/common and hence,the version
cannot be found.
You will have the same challenge with other packages using secondary entry points,e.g. @angular/material .
To avoid this situation, you can assign versions to all shared libraries manually:
 shared: {
  "@angular/core": {
   singleton: true,
   strictVersion: true,
   requiredVersion: '12.0.0'
 },
   "@angular/common": {
   singleton: true,
   strictVersion: true,
   requiredVersion: '12.0.0'
 },
   "@angular/common/http": {
   singleton: true,
   strictVersion: true,
   requiredVersion: '12.0.0'
 },
   "@angular/router": {
   singleton: true,
   strictVersion: true,
   requiredVersion: '12.0.0'
 },

 },

To avoid this situation, you can assign versions to all shared libraries manually

The issue with this seems to be that you lose the valuable warnings thrown from non-matching shared singletons.
For example, package X which you depend on requires v1 of dependency A but your build says requiredVersion for dependecy A is v2, then in this case you would not get that warning that versions mismatch because the required version is not determined anymore by package X but by your static "v2" in the module federation's config.
Seems to me like this is not a satisfactory solution. It wasn't in my case at least.

There seems to be another variation of this issue for certain dependencies, such as yaml.  See this PR for further discussion:
https://github.com/eemeli/yaml/pull/396
And a reproducer project here:
https://github.com/EricWittmann/module-federation-with-yaml-issue
Would be interested to know if this is something webpack should be handling - e.g. by searching harder for a parent package.json file under certain circumstances?

e.g. by searching harder for a parent package.json file under certain circumstances?

from https://docs.npmjs.com/creating-a-package-json-file
package.json fields:
Required name and version fields

So basically this is always required fields.

I fail to see how this issue has been resolved. Using the reproduction repository linked in the first post with webpack v5.73.0 and webpack-cli v4.9.2 I still get the following nonsensical warning message
WARNING in shared module react-dom
No required version specified and unable to automatically determine one. Unable to find required version for "react-dom" in description file (~/webpack-bug/node_modules/react-dom/package.json). It need to be in dependencies, devDependencies or peerDependencies.

It's telling me it's unable to find the version of react-dom in react-dom's package.json file. It does not make any sense for react-dom to depend on itself, the warning is therefore nonsensical regardless of whether the behaviour of not being able to find the required version is expected or not.

ok, webpack should implement for shared modules package.json more smart resolving..

@vankop Should this issue be reopened, or are you filing a separate issue for the planned change?

Спасибо 🙏

following

well Im getting byproducts of this same issue from other stuff not being articulated properly under the hood.
any workaround?
main.ts:22 Error: Unsatisfied version 0 from my-host of shared singleton module @apollo/client/core (required ^3.7.1)
    at getStrictSingletonVersion (consumes:49:1)
    at consumes:130:1

attempted this one...
const coreLibraries = new Set([
  '@angular/animations: { singleton: true, strictVersion: true, eager: true, requiredVersion: "^14.2.7" }',
  '@angular/cdk: { singleton: true, strictVersion: true, eager: true, requiredVersion: "^14.2.7" }',
  '@angular/common: { singleton: true, strictVersion: true, eager: true, requiredVersion: "^14.2.7" }',
  '@angular/compiler: { singleton: true, strictVersion: true, eager: true, requiredVersion: "^14.2.7" }',
  '@angular/core: { singleton: true, strictVersion: true, eager: true, requiredVersion: "^14.2.7" }',
  '@angular/forms: { singleton: true, strictVersion: true, eager: true, requiredVersion: "^14.2.7" }',
  '@angular/router: { singleton: true, strictVersion: true, eager: true, requiredVersion: "^14.2.7" }',
  '@apollo/client: { singleton: true, strictVersion: true, eager: true, requiredVersion: "^3.7.1" }',

  '@ngneat/transloco: { singleton: true, strictVersion: true, eager: true, requiredVersion: "^4.1.1" }',
  '@ngrx/component: { singleton: true, strictVersion: true, eager: true, requiredVersion: "^14.3.2" }',
  '@ngrx/component-store: { singleton: true, strictVersion: true, eager: true, requiredVersion: "^14.3.2" }',
  '@ngrx/effects: { singleton: true, strictVersion: true, eager: true, requiredVersion: "^14.3.2" }',
  '@ngrx/entity: { singleton: true, strictVersion: true, eager: true, requiredVersion: "^14.3.2" }',
  '@ngrx/router-store: { singleton: true, strictVersion: true, eager: true, requiredVersion: "^14.3.2" }',
  '@ngrx/store: { singleton: true, strictVersion: true, eager: true, requiredVersion: "^14.3.2" }',
  '@ngrx/store-devtools: { singleton: true, strictVersion: true, eager: true, requiredVersion: "^14.3.2" }',

  'apollo-angular: { singleton: true, strictVersion: true, eager: true, requiredVersion: "^4.1.0" }',
  'graphql: { singleton: true, strictVersion: true, eager: true, requiredVersion: "^16.6.0" }',
]);

module.exports = {
  name: 'host-yeti',
  remotes: ['ui-auth'],
  additionalShared: [
    '@angular/animations'
  ],
  shared: (libraryName, defaultConfig) => {
    if (coreLibraries.has(libraryName)) {
      return defaultConfig;
    }

    // Returning false means the library is not shared.
    return false;
  },
};

terminal warning
shared module @apollo/client/core -><EMAIL_ADDRESS>- Warning: No version specified and unable to automatically determine one. No version in description file (usually package.json). Add version to description file<EMAIL_ADDRESS>or manually specify version in shared config.
back to same thing
browser console
Error: Unsatisfied version 0 from host-yeti of shared singleton module @apollo/client/core (required ^3.7.1)
    at getStrictSingletonVersion (consumes:49:1)

following

bump

You must add version into shared package!
before: shared: { 'react': { singleton: true }, 'react-dom': { singleton: true } }
after: shared: {<EMAIL_ADDRESS>{ singleton: true }<EMAIL_ADDRESS>{ singleton: true } }

You must add version into shared package! before: shared: { 'react': { singleton: true }, 'react-dom': { singleton: true } } after: shared: {<EMAIL_ADDRESS>{ singleton: true }<EMAIL_ADDRESS>{ singleton: true } }

I think that has been discussed above.

Hi @eemeli, I've been reading your comments in regards yaml and I understand that this is something that we would need to tackle from webpack side but I just wanted to ask, from your point of view, is there a workaround that we could do in the consumer's modules in the meantime?
If that's not the case, no worries, thanks in advance :v:

Manually adding name and version property to yaml/browser/package.json seems to work, but according to @eemeli this is not the way to go
{
  "name": "yaml", //<--- added
  "version": "2.2.2", //<--- added
  "type": "module"
}

@lordrip I'm not aware of a workaround; this bug still needs to get fixed in webpack. I've suggested one possible solution above.
I'm not myself using Webpack at the moment, so would recommend that someone who's directly affected submit a PR fixing this.

Manually adding name and version property to yaml/browser/package.json seems to work, but according to @eemeli this is not the way to go
{
  "name": "yaml", //<--- added
  "version": "2.2.2", //<--- added
  "type": "module"
}

I tried this but it is not wirking, warnings about shared dependencies still in webpack serve log

i am also getting this warning on upgrading @sentry/react version from v7.x to v8.x.. Application working fine but this warning is disturbing.. Why this occuring first of all.. i have installed from a source and migrated following their documentation, despite it showing failed to see a version.. Gosh!!!!

@MegaDhakse2 Please provide a reproducible repository using github, thank you

Can we reopen this issue or open new? The problem with dual ESM+CJS packages is still here

@krutoo Can you create a reproducible test repo, need investigate

I had the same issue with “deep-object-diff” with this. Noticed the required version and version.
new ModuleFederationPlugin({ … shared: { …packageJSONDeps, “deep-object-diff”: { singleton: true, version: packageJSONDeps[“deep-object-diff”], requiredVersion: packageJSONDeps[“deep-object-diff”] }}}

Hi all,
I found a lead.. Someone asks for a demo repo or reproductive steps. For that i have integrated in a standalone project to reproduce that error. But it woks fine without warning there.
Then only i found that, this is only occurs in Module Fedaration apps. For standalone alone apps it works fine..
So can anyone HELP me to get out from this..

@krutoo Can you create a reproducible test repo, need investigate

@alexander-akait here is the example of problem:
https://github.com/krutoo/module-federation-dual-packages

@krutoo Yeah, because there is a bug in @sentry/browser, look at<EMAIL_ADDRESS>{"type":"module","sideEffects":false}

No name and version.
From docs:

If you plan to publish your package, the most important things in your package.json are the name and version fields as they will be required.

If you plan to publish your package, the most important things in your package.json are the name and version fields as they will be required.

https://docs.npmjs.com/cli/v10/configuring-npm/package-json
The package is published, and it is necessary to specify these fields, even if it is used for switching the type.
Thear already have:
"exports": {
   "./package.json": "./package.json",
   ".": {
     "import": {
       "types": "./build/npm/types/index.d.ts",
       "default": "./build/npm/esm/index.js"
     },
     "require": {
       "types": "./build/npm/types/index.d.ts",
       "default": "./build/npm/cjs/index.js"
     }
   }
 },

Therefore, the presence of an additional files is not even required, but can only break the logic of determining versions and names.
I don't think we are making a mistake here, rather the opposite, we are allowing this problem to be caught. The only solution is to specify the version manually, you can read the package.json of @sentry/browser so you don't have to write it manually.
I would advise you to open an issue in this repository and refer to this issue, such problems will most likely appear in any bundlers (which support module federation), and even most likely in other tools.

@alexander-akait but name and version is defined in root package.json
according to this logic, all information from the root package.json should be duplicated
I did this in my packages and specified the version, but then the federation module complained about the lack of dependency versions

@krutoo

according to this logic, all information from the root package.json should be duplicated

No, package.json do not inherit information from the previous one
Any package.json should have name and version if it is published, even for nested, this is the rule

@alexander-akait @sentry/node has exactly same structure and same nested package.json, when you import it in Node.js .mjs file it works fine
I think the rule you are talking about applies only to the main package.json

The npm docs on this are misleading, and unfortunately my attempt to clarify them was recently closed by @wraithgar: npm/documentation#134.

@eemeli - Yeah I see, because changing the resolver logic can be dangerous without spec/docs and because developers close this issue I would interpret this as mandatory fields always for any published projects


Can not auth

Question

Can not auth

Run GoogleCloudPlatform/github-actions/setup-gcloud@master
Successfully installed gcloud Cloud SDK
/opt/hostedtoolcache/gcloud/270.0.0/x64/bin/gcloud --quiet auth activate-service-account *** --key-file /tmp/tmp-27025Sjs2nWuhTxI
WARNING: .p12 service account keys are not recomended unless it is necessary for backwards compatability. Please switch to a newer .json service account key for this account.
ERROR: (gcloud.auth.activate-service-account) PyOpenSSL is not available. If you have already installed PyOpenSSL, you will need to enable site packages by setting the environment variable CLOUDSDK_PYTHON_SITEPACKAGES to 1. If that does not work, see https://developers.google.com/cloud/sdk/crypto for details or consider using .json private key instead.
##[error]The process '/opt/hostedtoolcache/gcloud/270.0.0/x64/bin/gcloud' failed with exit code 1

What I do is

make service account
gen service account key
set the github secret  SA_EMAIL, GOOGLE_APPLICATION_CREDENTIALS, GCE_PROJECT

Hi @MaximSungmo
As the issue template instructs, please share your complete action.yml file.

@sethvargo
I did not base64 encode the .json file.
used GOOGLE_APPLICATION_CREDENTIALS raw data of private_key as value..

Hello everyone!
It doesn't work for me. I have the same error:
WARNING: .p12 service account keys are not recomended unless it is necessary for backwards compatability. Please switch to a newer .json service account key for this account.
ERROR: (gcloud.auth.activate-service-account) PyOpenSSL is not available. If you have already installed PyOpenSSL, you will need to enable site packages by setting the environment variable CLOUDSDK_PYTHON_SITEPACKAGES to 1. If that does not work, see https://developers.google.com/cloud/sdk/crypto for details or consider using .json private key instead.
##[error]The process '/opt/hostedtoolcache/gcloud/270.0.0/x64/bin/gcloud' failed with exit code 1

My workflow is:
jobs:
  setup-build-publish:
    if: github.event.pull_request.merged == true
    name: name
    runs-on: ubuntu-latest
    steps:

    - name: Checkout
      uses: actions/checkout@v1

    # Setup gcloud CLI
    - uses: GoogleCloudPlatform/github-actions/setup-gcloud@master
      with:
        version: '270.0.0'
        service_account_email: ${{ secrets.GCP_EMAIL }}
        service_account_key: ${{ secrets.GCP_KEY }}

I created a service account with its JSON key.
I created the secrets in github repository.
In GCP_EMAIL I have the "client_email" field of the JSON key file.
In GCP_KEY I have the "private_key" field of the JSON key file.
For example, the private_key is like:
"-----BEGIN PRIVATE KEY-----\nMYPRIVATEKEY\n-----END PRIVATE KEY-----"

Does anyone have a solution for me?
Thanks


Update package.json

So this components shows up on http://react.parts/web

Of course, you may not want that but I had to track this down through a GitHub search and your original slider.


Support for injecting Senzing stream-loader directives

Pull request:

https://github.com/Senzing/stream-producer/pull/111


update changelog: range support for the convenience wrapper redBlackTree

follow-up to #4041 - maybe it's a good idea to document the addition?
ping @schveiguy
Btw I don't know why there are two "Library changes" sections - is this by design or a bug?

:white_check_mark:

Btw I don't know why there are two "Library changes" sections - is this by design or a bug?

This is for further explanations that can't be done in simple 1-line list item. Note that the list items above link to them.
Hard to see since this ddoc file doesn't get built by auto tester.


flutter 中没有类似 enablesReturnKeyAutomatically实现吗? text为空   retrun禁止

Steps to reproduce
flutter 中没有类似 enablesReturnKeyAutomatically实现吗? text为空 retrun 禁止
Actual results
flutter 中没有类似 enablesReturnKeyAutomatically实现吗? text为空 retrun 禁止
Logs

Logs

<!-- Paste your logs here -->

Flutter Doctor output

Doctor output

<!-- Paste your output here -->

flutter 中没有类似 enablesReturnKeyAutomatically实现吗? text为空 retrun 禁止

"Is there no implementation similar to enablesReturnKeyAutomatically in flutter? text is empty retrun is prohibited" - Using Google translate
This sounds like similar to https://github.com/flutter/flutter/issues/112198. You may follow https://github.com/flutter/flutter/issues/93915 as well. Closing as a duplicate.


The is no export functionality

No export functionality there, atleast excel export functionality should be there

There is an option for export. You can find the demo at https://gijgo.com/grid/methods/downloadCSV


[ISSUE #8927] Fix wrong queue offset of Lmq

Which Issue(s) This PR Fixes

Fixes #8927
Brief Description

How Did You Test This Change?

Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 47.55%. Comparing base (60e68da) to head (fef319f).

Additional details and impacted files
@@              Coverage Diff              @@
##             develop    #8928      +/-   ##
=============================================
- Coverage      47.62%   47.55%   -0.07%
+ Complexity     11743    11731      -12
=============================================
  Files           1304     1304
  Lines          91058    91058
  Branches       11675    11675
=============================================
- Hits           43362    43305      -57
- Misses         42352    42399      +47
- Partials        5344     5354      +10

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

@DongyuanPan Could yo review this pr pls, cause I"m not sure if I missed the update for your comment
6 QUEUEOFFSET, need update later
:)

This pr is not finished yet.


Add mechanism to resolve updates

Fixes #3.

@ctrueden do you mind testing this on a non-Windows system? I just made a new file in the /update structure and verified it was copied over to the equivalent location, and that an empty file in /update would lead to the corresponding file deletion.
We're currently missing special logic to handle updating launcher files 😬 any thoughts there?

I rebased over the latest main branch, and changed move to mv. I may look into the stat versus length thing later this weekend. If not, we can check it on Monday.

@hinerm I updated the File class to have a length field now instead of a stat() function. I also tidied up the Windows code such that length and lines() share common code as appropriate. I tested on Linux and all is working. Could you please test on Windows? If it still works on Windows, we can merge it!

I did find one limitation in my tests: if there is a file in the update folder inside a directory that does not exist in the main directory structure, the logic does not create the matching directory first, so the mv operation fails. We can remedy this by doing a mkdir (recursively as needed) before moving the file.

All seems good on windows

@ctrueden ok, making empty intermediate directories should work now. Anything else before merging?

 Task :compileKotlinMacosX64 FAILED
e: file:///Users/runner/work/jaunch/jaunch/src/posixMain/kotlin/file.kt:89:29 Type mismatch: inferred type is UInt but mode_t /* = UShort */ was expected

Nuh uh.... ☹


Line endings CRLF/LF

https://github.com/kneasle/wheatley/pull/159#issuecomment-808427078
https://github.com/kneasle/wheatley/pull/159#issuecomment-808455694
On Windows a clean checkout will result in lots of linter warnings due to automatic CRLF conversion.
I think we should suppress the warning rather than try and change default git and editor behaviour. We could then look at turning it on just as a pull request check.

Hmm this is interesting.  So does git convert all the line endings to CRLF?  I don't really mind what happens, so long as we don't end up with files having mixed line endings since this seems pretty bad and my text editor can't deal with it.
I don't do anything development on Windows, so I'm happy for you to test this out and make a fix for it...


fix typos

Please check if the PR fulfills these requirements:
(Add x between brackets to check.)

[x] I’ve read contributing guidelines
[ ] I’ve signed and sent the Readium ICLA
[ ] I’m making this pull request against the develop branch
[ ] I’ve updated from the develop branch before proposing this pull request
[ ] I’ve tested the changes for bug fixes and/or features
[ ] I’ve documented new code

What kind of change does this PR introduce? (Bug fix, feature, docs update, other)
DOC UPDATE
What is the current behavior? (You can also link to an open issue here)
What is the new behavior?
Does this PR introduce a breaking change? (What changes might users need to make in their application due to this PR?)
Other information:

@llemeurfr Quick question: for such PRs (e.g. typos, docs, etc.), do we really need the Readium ICLA signed and sent? Or is it just for code?

I would consider the CSS authoring should get IP consideration as well as code. SO, yes, proposing modifications to a CSS should be covered by an ICLA IMO.

Ok, since it’s a very minor change to the github template (i.e. administrative), unless someone disagrees, I think we can make an exception and merge the PR into the master branch – besides the corrected template would not be updated if it were merged into develop and would require another one from develop to master.
I’m also going to commit the updated template to the contributing guidelines in the master branch directly (and reflect handling of the ICLA for typos). Then sync the develop branch.

Thanks for catching those typos, @innovimax


Update NOTES.txt

Remove references to port-forwarding. Point users at Kubecost Cloud

Closes #9

LGTM, but having an eng review would likely be useful!

@biancaburtoiu I just installed a the agent on a new cluster for one of my prod teams and I am still getting the old installation message with port-forwarding instructions. Can you please help me investigate this?


Automate RequestDigest

Automate the process to get X-RequestDigest within the library. Could be done in a rest call and maintain the RequestDigest until it expires.
Making new RequestDigest calls conservatively would be a plus.

https://github.com/omkarkhair/sp-scotch/commit/98a27e875db9307d4014add6dc66b52747952215


Prohibit MRB_INT64 on 32-bit architecture

I think mrb_int size should be less than or equal to pointer size. In the
case of MRB_INT64 on 32-bit architecture, struct RString etc sizes
exceed 6 words at least for now.

I will oppose.

In 32-bit mode kernel space, 32-bit integers are insufficient (in personal terms, I'm glad if I can handle unsigned integer with more than 64 bits...).
If you define MRB_WITHOUT_FLOAT, the restriction is more strong.

gembox "full-core" succeeds even if mrb_int type included in struct RArray or struct RString is replaced with intptr_t (currently mruby-complex and mruby-rational requires modification).
I do not think that there is a problem because it can not be longer than 32 bits length in 32-bit mode (it would be a bug if there were any).

Thank you for your comment.
Then, as you say, mrb_int size will be allowed to exceed pointer size and type for sizes such as string length it may be better to limit to pointer size (introduce a new type, e.g. mrb_size, which is an alias for intptr_t or uintptr_t?).
Or, it may be possible to limit mrb_int size to pointer size and introduce a type for Fixnum that be allowed to exceed pointer size (e.g. mrb_fixnum).

There is no objection to the introduction of mrb_size and a change of type.
I think that it is better to leave the definition of mrb_int as it is from the viewpoint of the compatibility so far.

I checked what kind of transition the type of the len field of struct RString has followed:

Initially it was size_t:
https://github.com/mruby/mruby/blame/2fc1b8af2db70e2e008cecb9fd2d8600034f95f9/include/mruby/string.h#L29
It became long:
https://github.com/mruby/mruby/commit/2970b6aadbb0d5afef683854e0fe36b05024fc36#diff-d1ee00f2602251e1778ccf96047654c7L29
It became int:
https://github.com/mruby/mruby/commit/f4cf8ea423d8f9df1d341adba63d1416779aae8a#diff-d1ee00f2602251e1778ccf96047654c7L29
It has now become the current mrb_int:
https://github.com/mruby/mruby/commit/7e910bd4c2623775c00eb0fa052efc9e62d7b115#diff-d1ee00f2602251e1778ccf96047654c7L32
Release mruby-1.0.0.

Other object structures that exceed 6 words in 32-bit mode are struct RBreak defined in include/mruby/error.h.
https://github.com/mruby/mruby/blob/366617dbc1cbdc96292295f9d31b519579e6968f/include/mruby/error.h#L35-L39
If we define MRB_USE_FLOAT, it will be 6 words length.

print object size:
% cc -m32 -Iinclude -o sizeof sizeof.c && ./sizeof
size of struct RBreak: 28
% cc -m32 -Iinclude -DMRB_USE_FLOAT -o sizeof sizeof.c && ./sizeof
size of struct RBreak: 24

cat sizeof.c
#include <mruby.h>
#include <mruby/error.h>
#include <stdio.h>

int
main(int argc, char *argv[])
{
  printf("size of struct RBreak: %zd\n", sizeof(struct RBreak));
  return 0;
}

Thank you for your investigation.
I think another fix is necessary for RBreak, so it might be better to create another ticket.

I close this PR because it has been resolved via #4662 and #4835.


Unparsing into string array

In the project I am working there is a method which need command line arguments as a string array which I can't change.  So how to unparse into string array.
I can get a string by unparsing using var arguments = Parser.Default.FormatCommandLine(Program.cliOption).
But my argument will be something like this.
app install --path "c:/My folder/app" --validate = false
So if I use above workaround I can't use string.Split(' ') since it possible to contains spaces  in file paths. So is there any workaround to get string array from the object?

The develop branch contain a new method FormatCommandLineArgs that generate array of strings.
You can download the package LineParser.2.8.0-ci-176.nupkg from CI server.
This feature will be available in v2.9
var options = new Options { FilePath = "c:/My folder/app", Validate = false };
string[] arguments = Parser.Default.FormatCommandLineArgs(options);

Output array:
--path
c:/My folder/app

@moh-hassan answer is clear. Thank you so much and closing the Issue.


Fixed: used filepath.join instead of concataning string...

Codecov Report

Merging #380 into master will increase coverage by 0.13%.
The diff coverage is 36.36%.

@@            Coverage Diff             @@
##           master     #380      +/-   ##
==========================================
+ Coverage   56.06%   56.19%   +0.13%
==========================================
  Files          51       51
  Lines        6161     6152       -9
==========================================
+ Hits         3454     3457       +3
+ Misses       2399     2387      -12
  Partials      308      308

Impacted Files
Coverage Δ

monitor/linuxmonitor/cgnetcls/cgnetcls.go
18.91% <0%> (+1.84%)
:arrow_up:

processmon/processmon.go
72.72% <75%> (ø)
:arrow_up:

monitor/contextstore/contextstore.go
63.79% <75%> (+1.97%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update e75cc34...4d5a640. Read the comment docs.


Adaptive lighting is not changing bulb brightness or colour since upgrading to Tuya V2

Version information:
Adaptive_lighting: 1.0.14
OS: Home Assistant OS 6.5 on Raspberry Pi 4
Supervisor: supervisor-2021.10.0
Core: core-2021.10.4
Description:
Adaptive lighting is not changing bulb brightness or colour since upgrading to Tuya V2. Was fine on core-2021.9 and previous release of Tuya.
 Smart Bulb', 'supported_features': 0}' with context.id='574299c8d4c6ff21359ae322db68f41b'
2021-10-13 16:32:01 DEBUG (MainThread) [custom_components.adaptive_lighting.switch] office lights: Detected an 'off' → 'on' event for 'light.hey_smart_bulb' with context.id='574299c8d4c6ff21359ae322db68f41b'
2021-10-13 16:32:01 DEBUG (MainThread) [custom_components.adaptive_lighting.switch] office lights: '_update_attrs_and_maybe_adapt_lights' called with context.id='adapt_lgt_a0d5_light_event_1'
2021-10-13 16:32:01 DEBUG (MainThread) [custom_components.adaptive_lighting.switch] Adaptive Lighting: office lights: '_adapt_lights(['light.hey_smart_bulb'], 1.0, force=True, context.id=adapt_lgt_a0d5_light_event_1)' called
2021-10-13 16:32:01 DEBUG (MainThread) [custom_components.adaptive_lighting.switch] office lights: Scheduling 'light.turn_on' with the following 'service_data': {'entity_id': 'light.hey_smart_bulb', 'brightness': 255, 'color_temp': 189} with context.id='adapt_lgt_a0d5_light_event_1'
2021-10-13 16:32:01 DEBUG (MainThread) [custom_components.adaptive_lighting.switch] Detected an 'light.turn_on('['light.hey_smart_bulb']')' event with context.id='adapt_lgt_a0d5_light_event_1'
2021-10-13 16:32:58 DEBUG (MainThread) [custom_components.adaptive_lighting.switch] office lights: '_update_attrs_and_maybe_adapt_lights' called with context.id='adapt_lgt_a0d5_interval_2'
2021-10-13 16:32:58 DEBUG (MainThread) [custom_components.adaptive_lighting.switch] Adaptive Lighting: office lights: '_adapt_lights(['light.hey_smart_bulb_39', 'light.tradfri_driver', 'light.hey_smart_bulb'], None, force=False, context.id=adapt_lgt_a0d5_interval_2)' called
2021-10-13 16:32:58 DEBUG (MainThread) [custom_components.adaptive_lighting.switch] office lights: Scheduling 'light.turn_on' with the following 'service_data': {'entity_id': 'light.hey_smart_bulb_39', 'brightness': 255, 'color_temp': 189} with context.id='adapt_lgt_a0d5_interval_2'
2021-10-13 16:32:58 DEBUG (MainThread) [custom_components.adaptive_lighting.switch] office lights: Scheduling 'light.turn_on' with the following 'service_data': {'entity_id': 'light.hey_smart_bulb', 'brightness': 255, 'color_temp': 189} with context.id='adapt_lgt_a0d5_interval_2'
2021-10-13 16:32:58 DEBUG (MainThread) [custom_components.adaptive_lighting.switch] Detected an 'light.turn_on('['light.hey_smart_bulb_39']')' event with context.id='adapt_lgt_a0d5_interval_2'
2021-10-13 16:32:58 DEBUG (MainThread) [custom_components.adaptive_lighting.switch] Detected an 'light.turn_on('['light.hey_smart_bulb']')' event with context.id='adapt_lgt_a0d5_interval_2'
Any advice would be appreciated. Thanks.

The Tuya V2 has a bug where you can't set the brightness and color at the same time. I fixed it on my local copy, but the bug is still open on github:
https://github.com/tuya/tuya-home-assistant/pull/413

This was fixed in 2021.11.1

This was fixed in 2021.11.1


Chests won't open on server

Hi I was playing on my server with someone and everything was fine until we noticed lootr chests couldn't be opened. The person I was playing with managed to kill a dragon from the Ice and Fire mod which was cool af because they're hard to kill but it was sad because they went over to where the dragon usually sits on all the gold and stuff where the chests are and the chests couldn't be opened at all.
Latest.log files:
latest.log
latest2.log
Dragon area where chests couldn't be opened:

There was also a lootr chest in a village nearby that couldn't be opened either. I also noticed when loot was dropped out of the inventory sometimes it would completely just disappear and not be an item on the ground to be picked up again at all. Also the modded minecraft version is 1.12.2.

There's a considerable amount of errors in the first log file that you provided:
[05:00:17] [Server thread/ERROR] [Lootr]: replacement TE net.minecraft.tileentity.TileEntityChest@81e7929 is not an ILootTile 0 at BlockPos{x=174, y=23, z=18}
[05:00:17] [Server thread/ERROR] [Lootr]: replacement TE net.minecraft.tileentity.TileEntityChest@4fc1bac5 is not an ILootTile 0 at BlockPos{x=187, y=25, z=10}
[05:00:17] [Server thread/WARN] [net.minecraft.server.MinecraftServer]: Can't keep up! Did the system time change, or is the server overloaded? Running 40013ms behind, skipping 800 tick(s)
[05:00:17] [Server thread/ERROR] [Lootr]: replacement TE net.minecraft.tileentity.TileEntityChest@21ed8d6a is not an ILootTile 0 at BlockPos{x=222, y=19, z=22}
[05:00:18] [Server thread/WARN] [net.minecraft.world.storage.loot.LootTable]: Tried to over-fill a container
[05:00:18] [Server thread/ERROR] [Lootr]: replacement TE net.minecraft.tileentity.TileEntityChest@53b75531 is not an ILootTile 0 at BlockPos{x=309, y=48, z=5}
[05:00:18] [Server thread/ERROR] [Lootr]: replacement TE net.minecraft.tileentity.TileEntityChest@6437060c is not an ILootTile 0 at BlockPos{x=340, y=31, z=18}
[05:00:18] [Server thread/WARN] [FML]: Battle Towers loaded a new chunk [20, -1] in dimension 0 (overworld) while populating chunk [21, 0], causing cascading worldgen lag.
[05:00:18] [Server thread/WARN] [FML]: Please report this to the mod's issue tracker. This log can be disabled in the Forge config.
[05:00:18] [Server thread/INFO] [STDOUT]: [atomicstryker.battletowers.common.AS_WorldGenTower:generate:538]: Dimension 0 Battle Tower type SmoothStone spawned at [ 336 | 0 ], underground: false
[05:00:18] [Server thread/ERROR] [Lootr]: replacement TE net.minecraft.tileentity.TileEntityChest@2185a4b9 is not an ILootTile 0 at BlockPos{x=346, y=20, z=10}
[05:00:18] [Server thread/ERROR] [Lootr]: replacement TE net.minecraft.tileentity.TileEntityChest@4bc12cea is not an ILootTile 0 at BlockPos{x=386, y=29, z=10}

These are probably the most significant, and, based on what happened a few seconds earlier in the mod, most likely are a result of Roguelike Dungeons causing cascading chunk loading which is overriding Lootr converted containers:
[05:00:12] [Server thread/WARN] [FML]: Roguelike Dungeons loaded a new chunk [-9, -4] in dimension 0 (overworld) while populating chunk [0, 0], causing cascading worldgen lag.
[05:00:12] [Server thread/WARN] [FML]: Please report this to the mod's issue tracker. This log can be disabled in the Forge config.
[05:00:14] [Server thread/INFO] [roguelike]: Trying to spawn dungeon with id builtin:grassland at {"x": 74, "y": 0, "z": 2}...
[05:00:14] [Server thread/WARN] [FML]: Roguelike Dungeons loaded a new chunk [3, -1] in dimension 0 (overworld) while populating chunk [0, 0], causing cascading worldgen lag.
[05:00:14] [Server thread/WARN] [FML]: Please report this to the mod's issue tracker. This log can be disabled in the Forge config.
[05:00:14] [Server thread/WARN] [FML]: Roguelike Dungeons loaded a new chunk [4, -1] in dimension 0 (overworld) while populating chunk [0, 0], causing cascading worldgen lag.
[05:00:14] [Server thread/WARN] [FML]: Please report this to the mod's issue tracker. This log can be disabled in the Forge config.
[05:00:14] [Server thread/WARN] [FML]: Roguelike Dungeons loaded a new chunk [5, -1] in dimension 0 (overworld) while populating chunk [0, 0], causing cascading worldgen lag.
[05:00:14] [Server thread/WARN] [FML]: Please report this to the mod's issue tracker. This log can be disabled in the Forge config.

Roguelike Dungeons seems to be mostly responsible for these issues. It's also possible that Lootr is performing conversions and then Roguelike Dungeons is "undoing" them, although it does seem exceedingly widespread.
In addition, the last half of the first file consists almost entirely of errors involving the network, most of which are being triggered by Ice & Fire.
The second log file doesn't contain any errors; I imagine most of your issues come from errors with world generation. Unfortunately, I can't provide any support beyond this for 1.12.2 as I do not develop for the version and, given the sheer quantity of mods, wouldn't even know where to begin with debugging.

I ended up doing a little bit of googling after posting this and saw this post https://github.com/LootrMinecraft/Lootr/issues/16 and noticed the Ice and Fire mod was on that list. Is it incompatible with this mod in terms of the chests?
Also sorry I found out the issue I had with items disappearing was actually just JEI's cheat mode thing that I left on by accident which I would've been doing testing with before playing and I'm guessing it probably transferred as enabled onto the server through config files aswell but who knows. I didn't know it was a feature and I was having so much fun with the person that I was playing with that I didn't even notice the text thingy too lolol but anyway just in case anyone has this problem and comes across this post just know JEI's cheat mode has a feature where it can delete items that you drag out of your inventory like when your dropping items to people and stuff. Just thought I'd say that in case it helps anyone.

That's specifically with the 1.16 and later versions of Ice & Fire, not the 1.12.2 version.
And yes, that's definitely part of cheat mode, although generally it's only if you drag items onto the actual JEI interface. Dragging them above or below the container shouldn't have that effect.

That's specifically with the 1.16 and later versions of Ice & Fire, not the 1.12.2 version.
And yes, that's definitely part of cheat mode, although generally it's only if you drag items onto the actual JEI interface. Dragging them above or below the container shouldn't have that effect.

Ohh ok also I confirmed it was the JEI cheat mode doing it I was dragging stuff off to the left aswell away from the interface and it was happening it actually shows text saying that it's about to delete stuff too I just was doing it pretty fast and was having a lot of fun with the person I was playing with that I didn't actually realize it but anyway it was 100% the JEI cheat mode causing items to disappear when dropping them.
Also I'll be doing some testing with just Lootr and Ice and Fire and then introduce roguelike dungeons into the mix and see what happens because the stuff you said about roguelike dungeons is interesting and is probably causing the problems. I'm going to be testing it with and without pregenerating the world too. I'll get back to you here soon.
Anyway merry christmas man.