buildkite: pass through environment variables for coveralls-python

The coverage-uploader coveralls-python understands buildkite
natively, as long as it has access to the required environment
variables from outside the docker container.

Coverage remained the same at 84.282% when pulling fa02cbee55b4823d25c310c4ae1e3b0d103f877c on bugfix/coveralls-understands-buildkite into 8123b482be9799d8210b3be72fe464c37b91a2f6 on develop.

This change means that:

coveralls has more metadata, like a connection to the pull request (see "Build Type > PR 363" on  https://coveralls.io/jobs/46962445)
coveralls can do a comment like the above (but it can be turned off under https://coveralls.io/github/stellargraph/stellargraph/settings "Leave Comments?")
(slight downside) the build identifiers on coveralls are UUIDs referring back to the specific Buildkite step and thus are longer

@huonw thanks for that! As I'm not qualified to review this PR, I've assigned @TheDen as a reviewer :)


0.49 breaks compatibility with wasm32-unknown-unknown target

Regression in 0.49, as 0.48 works fine. Panic due to unimplemented SystemTime.
panicked at 'time not implemented on this platform', library/std/src/sys/wasm/../unsupported/time.rs:31:9

Does wasm32-unknown-unknown not support std?


Node: add JSON.ARRINDEX comand

Issue link
https://github.com/valkey-io/valkey-glide/issues/2529
Checklist
Before submitting the PR make sure the following are checked:

[x] This Pull Request is related to one issue.
[x] Commit message has a detailed description of what changed and why.
[x] Tests are added or updated.
[x] CHANGELOG.md and documentation files are updated.
[x] Destination branch is correct - main or release
[x] Commits will be squashed upon merging.


Pawns Ignoring Job Assignments

IMPORTANT:
Please first search existing bugs to ensure you are not creating a duplicate bug report!
Make sure you have the latest version of Rimworld!

[ ] Reproduce the issue in 2.5.16 or later with the same mods
[ ] try to Reproduce the issue in 2.5.16 vanilla RT
[ ] Fix / Patch

Describe the bug
Pawns will no longer do jobs assigned to them beyond their top level job.  For instance, one pawn that is set to cook, butcher, and brew first will do those tasks anytime they are available, but will not do any other jobs assigned to her.  She will just go idle instead and wander.  I can force these pawns to do things, but as soon as i undo that or they complete the task, they return to idle if there is not a task to perform under that one 'main' job.  Changing priorities does not change that 'main' job that they will do.  Drafting them doesn't effect that either.  Disabling RimThreaded is the only thing that removes this issue so far.  It works every time i disable the mod, load my save, save it again, and then load that save.
Steps to reproduce the behavior (VERY IMPORTANT)

Load Save
Watch pawns until you see them start to go idle  (usually happens within the first couple minutes)

HugsLib log
https://gist.github.com/83ea93c0052cdf56139836dbd3c0048f
Save file
ModTesting2 (RT Enabled;FB Disabled).zip
Mod list (Preferably a RimPy compatible list.)
ModTest (RT Enabled).txt

I have  the same problem.
My log.
https://gist.github.com/HugsLibRecordKeeper/ebd9875e4c84b3f23e5f5eac7e432549
My pawn don't cutting rock. I setting priority to 1.   Disable rimthreaded problem gone
My save file.
bug.zip

As of right now, I can confirm that some aspects of this issue persist in modded saves, but I do not experience issues in vanilla. Initially, I was having trouble with pawns completely ignoring job assignments unless manually tasked, but now the problem seems to be that they will complete a job (crafting task, mining a cell of stone, etc.) and then spend several hours (while scheduled to work) wandering aimlessly. This is also not creating any log errors.
I tried using a vanilla save with any mods I could think of that I was using that created new job types or modified pawn behavior. Hopefully this helps; the following mods do not appear to be resulting in issues based on my tests:
Stabilize
Do Your F****** Research
Auto-Cut Blight
Misc. Training
Map Reroll
Perfect Pathing
Incident Person Stat
Graze Up
Color Coded Mood Bar
Misc. Robots
Real Ruins
Dubs Break Mod
EdB Prepare Carefully
Wall Light
Allow Tool
Quarry
DoorMat
Drag Select
Beta Traps
Dubs Mint Minimap
RT Fuse
Fuse Plus
Defenses Expanded
Reinforced Walls
Defensive Machine Gun Turret Pack
Quest Rewards Rebalance
Tilled Soil
This was tested by using dev tools to generate a test colony repeatedly and observing how colonists behaved, however, neither DLC was used. None of these were allowed to run for more than a few minutes, however, so I may update this if it turns out to be an issue that takes a little while to arise.

Following complaints of this affecting Human Resources, I investigated the matter. The only thing I can say for sure is your patch for JobGiver_Work for some reason isn't calling the WorkGiver like it's supposed to. In HR's case, that's WorkGiver_Knowledge, derived from the vanilla WorkGiver_DoBill. It overrides the vanilla HasJobOnThing method, and it would be the first step in a pawn determining there's work to do. With Rimthreaded, that's never called. But that's as far as I could dig.
I figured this could be potentially useful information. If not, please ignore.


Add module 'fd' that exposes file handle operations

I need to interact with basic file handles such as stdin and stdout and I didn't find a way of doing that with the existing 0.5.x API.  This PR adds support for the basic file handles and also an entry point for wrapping arbitrary file descriptors.

The tests are failing on nightly for reasons unrelated to this PR, I hope that's okay! :-)

Overlaps with EventedFd somewhat.

@dimbleby maybe we should rather expose EventedFd and constructors for the standard file descriptors? I had the impression  that EventedFd is not really part of the cross platform since it's exported in an ad hoc manner and Unix specific; it would be nice to have something that's eventually supported on all platforms.

FileStream also looks pretty platform-specific: per its use of std::os::unix::io::RawFd and the failing AppVeyor build.
But realIy I just wanted to point out the overlap.  I guess I do think that if mio ends up with both EventedFd and the proposed FileStream then that would look odd, but I don't have any particular view on how best to rationalise this.

I will of course try to fix the builds on other platforms, but before it would be good to know what the preferred action is: keeping this interface and maybe also supporting it on Windows, or extending EventedFd so that it also supports Read and Write?

I have abandoned this approach and gone back to using Io in mio version 0.4.x, but I will leave the PR open for 0.5.x.

Thanks for the PR!
The implementation (using Io) would also imply that when the handle goes out of scope, it will get closed.
It is pretty *nix specific which would require some thought. I think that I will stick with the EventedFd bit for now.
If you think there is a different strategy or still think that this is worth while, I would open an issue for discussion.


Make mesos master run as non-root

TODO:

[x] Move logrotate to /var/lib/dcos/mesos/logs/
[x] Move master work dir to /var/lib/dcos/mesos/master

Do we need to add another logrotate, which is root, for mesos slaves?

After talking with Harpreet, do need the special case for agents. Added that, although haven't updated the 3dt test, (so integration should end up failing).

Did some minor sanity check on the build looks good. :+1:
Just a note to be mindful of the fact that users upgrading to a build with this change will have older log files in /var/log/mesos and /var/log/mesos/archive


Run the only ROM inside a folder without entering it

could you include folders as being considered like a filename
example:
/usr/games/nintendo/Top-Gun-US.nes/Top-Gun-US.nes
the first Top-Gun-US.nes be listed as the game, or even the second, doesnt matter.
This paves the way for a cleaner system structure for roms (a la Higan) like
/usr/games/Top-Gun.nes/program.rom
/usr/games/Top-Gun.nes/cheats.bml
/usr/games/Top-Gun.nes/screenshot-0001.png
/usr/games/Top-Gun.nes/state-1.bst
/usr/games/Top-Gun.nes/save.ram
If we begin supporting the structure then it makes for a much better project that is becoming its own distribution entirely.

Agreed. This would be nice to have. Especially for Wii and Gamecube game rips.


update benchmarks to latest master

Update code to work with latest master.

The CI pipeline was cancelled due to failure one of the required jobs.
Job name: test-linux-stable
Logs: https://gitlab.parity.io/parity/mirrors/substrate/-/jobs/2184320

The CI pipeline was cancelled due to failure one of the required jobs.
Job name: test-linux-stable
Logs: https://gitlab.parity.io/parity/mirrors/substrate/-/jobs/2184321


update Thai

For localization you should use Transifex:
https://www.transifex.com/traccar/traccar/

Sorry! I’ll join

Thank you for the contribution.


fix timeout ginkgo

In the old ginkgo, the timeout by default is 24h
In the new ginkgo (v2) the timeout by default is 1h
1 hour is not enough to pass the integration tests
I suggest that you explicitly pass the timeout parameter equal to 24h, so that there is enough time to pass the integration tests

@Mergifyio backport release-2.5


[C++] Upgrade zstd to v1.5.4

Describe the enhancement requested
Zstd has release it's latest version, v1.5.4, which containing a lots of performance improvement under x86 and aarch64. The release note can be seen here: https://github.com/facebook/zstd/releases/tag/v1.5.4
Notice that arrow is using zstd v1.5.2, can we upgrade it to v1.5.4?
Component(s)
C++

@kou @pitrou What do you think of this? If it's ok, I can help upgrade it

Yes, please!


Wallet recovery from multisig seeds

Add a wallet recovery dialog.
Recovering funds from the two multisig seeds (online, offline) involves generating all addresses until wereAddressesSpentFrom returns false and then checking the balance.
The funds should be transferred to a newly created wallet.
Ideally, this should also happen offline, the problem with this is that checking balances offline is not possible, meaning that the software needs to generate in advance, for example, 200 addresses and then it is checked online. But in case the address count was exceeded, the wallet would require to generate another batch of addresses, which is cumbersome but the only way.
An easier way would be to just to it online, but this exposes the seeds for short time to the internet.
I'm leaning towards the offline solution, since it's more secure and users usually generate less than 200 addresses...

Update: iotacooler-smidgen support implemented, now GUI part in progress...


Combo box dropdown size Linux/Mac

First item in combo box dropdown fills window. Could not fix with changing size policy, setting minimum height, or changing .view().setMaximumHeight. Appears to be a bug with QDarkStyle on Max and Linux: https://github.com/ColinDuquesnoy/QDarkStyleSheet/issues/200
Attempt to fix and check if problem persists on Windows.

@ljjh20 you can pass styleSheet="combobox-popup: 0;" as keyword argument to your QComboBox() call.  That will do.


Reconsider starting Ghost while url generation is still in progress

Issue Summary
With 1.24.0, it's possible that you will receive a 503 when you serve your blog too fast.
That means, Ghost is still generating the resource urls.
This concept was a change we have introduced with dynamic routing beta.
It's possible to wait till all url's are generated till the http server connects to the port.
We would like to reconsider and test how difficult it is to change this concept again.
That would mean you would no longer receive a 503 when you are hitting your blog too fast.
The disadvantage is, that your blog starts a little slower than before because it has to wait till url generation completed.

Closing. We would like to keep the functionality for now.
Discussed in slack.

503 is the correct handling inside of Ghost not being ready instantly


We need automation...

We should add automation to the tests rather than doing it manually

WYDM? automation in test ?

Contributor

I meant we can do automated testing using jenkins or puppeteer or cypress etc. to decrease our manual code testing time and use it on problem solving procedures()


added javadocs to project

Codecov Report

Merging #1190 into enhancement/v4 will not change coverage.
The diff coverage is n/a.

@@               Coverage Diff               @@
##           enhancement/v4    #1190   +/-   ##
===============================================
  Coverage           27.12%   27.12%
===============================================
  Files                 358      358
  Lines               14719    14719
  Branches             1312     1312
===============================================
  Hits                 3993     3993
  Misses              10253    10253
  Partials              473      473

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 6680369...b8f4552. Read the comment docs.


The Azure Data Lake Storage Gen2 connector is fully supported on Databricks Runtime 5.2 and above

[Enter feedback here]
The bellow page says that "The Azure Data Lake Storage Gen2 connector is fully supported on Databricks Runtime 5.2 and above".
https://docs.azuredatabricks.net/spark/latest/data-sources/azure/azure-datalake-gen2.html#azure-data-lake-storage-gen2

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 4f41dfad-9591-8a8d-6119-259d90da18c7
Version Independent ID: f4bc3316-a230-a486-d1f7-69962ec8030e
Content: Upgrade your big data analytics solutions from Azure Data Lake Storage Gen1 to Azure Data Lake Storage Gen2
Content Source: articles/storage/blobs/data-lake-storage-upgrade.md
Service: storage
GitHub Login: @normesta
Microsoft Alias: normesta

Thank you for logging an issue. Can you help me understand what the issue is? Thank you.

@yualan We will now proceed to close this thread. If there are further questions regarding this matter, please tag me in your reply. We will gladly continue the discussion and we will reopen the issue.

Issue is doc inconsistency: According to the Databricks doc Runtime version has to be 5.2 and higher, in this article 5.1 and higher.


Compilation Error in macro default__information_schema (macros\cross_db_utils.sql)

Versions
dbt: 0.19.1
snowflake: 0.19.1
dbt_profiler: 0.1.3
Steps To Reproduce
 dbt run-operation print_profile_docs --args '{"relation_name": "shipment__fct_shipment_stop"}'
Or this
 dbt run-operation print_profile_docs --args '{"relation_name": "shipment__fct_shipment_stop", "schema": "shipment"}'
I also tried several other combinations of the above arguments and with each profile macro.
Results
Encountered an error while running operation: Compilation Error in macro default__information_schema (macros\cross_db_utils.sql)
'None' has no attribute 'information_schema'

in macro information_schema (macros\cross_db_utils.sql)
called by macro get_profile (macros\get_profile.sql)
called by macro print_profile_docs (macros\print_profile_docs.sql)
called by macro default__information_schema (macros\cross_db_utils.sql)

Thanks for reporting this. It looks like the relation variable is None which means that it doesn't exist or you don't have the permissions to read it. I've changed the behavior PR#25 so that the get_profile now throws an exception if the relation does not exist or the user is not authorized.

Did you run my_third_model after copy pasting the configs? If not, the profiling fails because there is no such relation in the database. You can also try specifying the schema and database arguments for the print_profile macro.

Yes, I ran the model and checked to see if it was created in the database.  I also passed in the schema variable to print_profile, and that did not help either.  I even turned my computer off and back on again. :)
It seems like it should work. Before writing this comment, I reverted the code changes, redid my repro steps, and got the same results.
This might be a good opportunity for me to learn how to debug dbt macros.

Thanks! That worked!  I am still curious to know why it worked on some models without passing in the database and schema names.  I let you know if I find the answer.


Task/g628 metadata receipt

Applied pagination changes for Metadata and Receipt endpoints

ReceiptRepository should provide 2 or 3 independent searches so the user knows exactly what are the accepted parameters and the returned types.
https://github.com/nemtech/symbol-sdk-typescript-javascript/blob/task/g628_metadata_receipt/src/infrastructure/ReceiptRepository.ts#L27
Java as an example:
https://github.com/nemtech/symbol-sdk-java/blob/metadata-receipt/sdk-core/src/main/java/io/nem/symbol/sdk/api/ReceiptRepository.java#L32


Update cats-effect to 3.3.5

Updates org.typelevel:cats-effect from 3.3.4 to 3.3.5.
GitHub Release Notes - Version Diff
I'll automatically update this PR to resolve conflicts as long as you don't change it yourself.
If you'd like to skip this version, you can just close this PR. If you have any feedback, just mention me in the comments below.
Configure Scala Steward for your repository with a .scala-steward.conf file.
Have a fantastic day writing Scala!

Ignore future updates
Add this to your .scala-steward.conf file to ignore future updates of this dependency:
updates.ignore = [ { groupId = "org.typelevel", artifactId = "cats-effect" } ]

labels: library-update, early-semver-patch, semver-spec-patch, commit-count:1

Codecov Report

Merging #610 (78e5cc1) into main (5d08b44) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##             main     #610   +/-   ##
=======================================
  Coverage   85.57%   85.57%
=======================================
  Files         124      124
  Lines        1622     1622
  Branches       40       40
=======================================
  Hits         1388     1388
  Misses        234      234

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 5d08b44...78e5cc1. Read the comment docs.


PAPPL on Windows support

The current PAPPL implementation is almost ready for use on Windows. Changes that are needed include:

[ ] Implement wrapper types/functions for POSIX threading APIs
[ ] Determine replacements for signal handling

A parallel issue regarding the pthread function calls was brought up by @elfring in #113 - basically the pthread_mutex and pthread_rwlock APIs return a status code when locking, but the current code does not check the return values. When implementing the wrapper functions, the POSIX locking APIs should also be wrapped to support retries and potentially an abort if the unthinkable occurs.

Implement wrapper types/functions for POSIX threading APIs

How do you think about to use the current support for multi-threading by the C programming language?

@elfring This C11 API is not widely implemented and there is no support for a native reader-writer lock, having likely been limited to what Microsoft could easily support...

:crystal_ball: I assume that your relationships can reduce undesirable software limitations another bit, can't they?

@elfring Huh?

[master 0233ca4] Initial implementation of POSIX threading functions needed for PAPPL on Windows (Issue #116)


Generate boilerplate from the existing func file

This PR to allow developers generate the boilerplate from existing func files.
How it works:
Developer has a func file:
$ ls -la
total 8
drwxr-xr-x   3 denismakogon  staff   96 Aug 10 12:15 .
drwxr-xr-x  20 denismakogon  staff  640 Aug  8 13:23 ..
-rw-r--r--   1 denismakogon  staff  139 Aug 10 12:12 func.yml

Generating boilerplate:
$ fn init --from-func-file
Runtime: go
Function boilerplate generated.
func.yml  created.

Boilerplate generated:
ls -la
total 32
drwxr-xr-x   6 denismakogon  staff  192 Aug 10 12:16 .
drwxr-xr-x  20 denismakogon  staff  640 Aug  8 13:23 ..
-rw-r--r--   1 denismakogon  staff  127 Aug 10 12:16 Gopkg.toml
-rw-r--r--   1 denismakogon  staff  469 Aug 10 12:16 func.go
-rw-r--r--   1 denismakogon  staff  139 Aug 10 12:16 func.yml
-rw-r--r--   1 denismakogon  staff  505 Aug 10 12:16 test.json

No impact on current UX:
fn init --runtime go
Runtime: go
Function boilerplate generated.
func.yaml  created.

$ fn init

Fn: Function file already exists, aborting

See 'fn <command> --help' for more information. Client version: 0.4.135

Closes: #371

Hi. I put some questions on the issue for this: https://github.com/fnproject/cli/issues/371#issuecomment-412436595

If you're proposing to supply an external func.yaml, do you see the CLI ensuring it conforms to the correct current schema version and updating accordingly?  How does this interplay with specifying --trigger to fn init where the CLI would add a trigger node and subnodes to the YAML?


Infinite loop when using reduce() inside tf.data.Dataset.map()

System information

Have I written custom code: yes
OS Platform and Distribution: Linux Ubuntu 20.04
TensorFlow installed from: Binary
TensorFlow version: v2.4.0-49-g85c8b2a817f 2.4.1
Python version: 3.8.5
CUDA/cuDNN version: None
GPU model and memory: None

Describe the current behavior
I'm trying to create a 3D image dataset using my previous dataset of sorted 2D images. To do so, I use map() method of tf.data.Dataset along with a reduce() function that takes some 2D images from the same dataset (or a copy, from what I tried it doesn't matter) and adds them together. Nevertheless, when I use functools.reduce() or tf.foldl() functions inside of a tf.data.Dataset.map(), even in a simpler case, it enters an infinite loop.
For the example, I've simplified the case to a dummy dataset of 2 elements with shape (2, 3). And I'm using reduce() instead of tf.fold() to avoid using a generator in the input of the function.
The result is also an infinite loop.
Describe the expected behavior
In my understanding, I should get a new dataset with one element of shape (1, 2, 3). But if this transformation is not possible, I think it should raise an error.
Standalone code to reproduce the issue
import tensorflow as tf
from functools import reduce
ds = tf.data.Dataset.from_tensor_slices([[[1,2,3],[4,5,6]],[[7,8,9], [10,11,12]]])
new_ds = ds.map(lambda _: reduce(lambda a, b: tf.concat([tf.expand_dims(a[0], axis=0),tf.expand_dims(b[0], axis=0)], 0), ds))

I am able to replicate this on tf 2.3,2.4 and nightly, please find the gist here

Was able to replicate the issue in TF 2.6.0-dev20210603,please find the gist here..Thanks !

Hi,
Thank you for opening this issue. Since this issue has been open for a long time, the code/debug information for this issue may not be relevant with the current state of the code base.
The Tensorflow team is constantly improving the framework by fixing bugs and adding new features. We suggest you try the latest TensorFlow version with the latest compatible hardware configuration which could potentially resolve the issue. If you are still facing the issue, please create a new GitHub issue with your latest findings, with all the debugging information which could help us investigate.
Please follow the release notes to stay up to date with the latest developments which are happening in the Tensorflow space.


npm package list support

This extends #99 to support npm, as part of the ongoing effort towards #83

The behaviour of hard failing when hitting an unpublished package on the critical list seems sensible to me, however it should probably be aligned with how this is handled for the pypi implementation.

The behaviour of hard failing when hitting an unpublished(all existing versions removed, at the given point in time) package on the critical list seems sensible to me, however it should probably be aligned with how this is handled for the pypi implementation for a similar scenario. Any insight on this @Qinusty?

This could be quite problematic for a deployment with an NPM feed configured with many packages for critical polling. A single package being unpublished has the potential to prevent polling of all NPM packages specified.

The behaviour of hard failing when hitting an unpublished(all existing versions removed, at the given point in time) package on the critical list seems sensible to me, however it should probably be aligned with how this is handled for the pypi implementation for a similar scenario. Any insight on this @Qinusty?

This could be quite problematic for a deployment with an NPM feed configured with many packages for critical polling. A single package being unpublished has the potential to prevent polling of all NPM packages specified.

Yes this is definitely a concern, it should be looked at as part of #107 and addressed along with other brittle areas


[Request] Siren: Blood Curse (True HD Mod)

Here's the original release thread.
Author: Silent_Pavel of PSX Scene
Version: um... 1?
Config: All
Files:  https://drive.google.com/open?id=1kKt4WJv6ZrOhXufSEJluQRoWxVJjRvb7
^This is my own pre-edited file, so no hex edited needed - it's already been done.
I... Don't really know how github works, I'm sorry if this is a mess.

Thanks for this! Few questions about as I haven't modded this game before. Where does the common.dat file install to? It's for all regions too?

@wh1ter0se-x sorry for the delay in response time, my work schedule is really weird and I'm usually up all night.
So everything about this file is inside the original thread.
The original thread details about how you going to the games files and how to hex edit a certain config for some display settings. It goes into more elaborate detail in that thread which is still online thank God. So are the original files included.
I remember I did it years ago and I saved the file I had so I didn't have to try hex editing ever again.
I'm not in front of my PS3 at the moment but I can look up and see exactly what folder this one goes into, but it pretty much just over rights and already existing file and ups the graphical quality.
I don't know about Region honestly - this patch SHOULD work on all Regions for the game, but I can't confirm that. I CAN confirm that it works on the NTSC PSN release.

Yes I had noticed you've done the most part, that's amazing thank you. I've found the readme which does help a lot and unfortunately don't have the game installed neither, think I may need to link into the one a little further as I believe it's a psn game, right? So I may need to make an application update to support this and allow for installing the folders of that kind of region, but should be fine. Please do submit all mods that you can find, greatly appreciated of your help.

Could I confirm that these are the correct installation paths, starting at root:
"dev_hdd0/game/{REGION}/PS3_GAME/USRDIR/sirenx/data/commonen.dat",
"dev_hdd0/game/{REGION}/PS3_GAME/USRDIR/sirenx/data/commonuk.dat"

Thanks.

I think the NTSC PKG, it's just "common", so it may be "commonen" as that's what I remember, but when I look in the games folder, for me, there was no "commonen.dat", BUT there was a "common.dat".
I spent last night finding some MK9 mods - luckily those are easier to install. I'll start a separate thing on those later.
I promise I'll look into it on my PS3.
But yeah. Just drag and drop into the folder.
Get Outlook for Androidhttps://aka.ms/ghei36

From: Ashley<EMAIL_ADDRESS>Sent: Sunday, September 1, 2019 3:31:44 PM
To: wh1ter0se-x/ModioX<EMAIL_ADDRESS>Cc: HeadstrongWolf<EMAIL_ADDRESS>Mention<EMAIL_ADDRESS>Subject: Re: [wh1ter0se-x/ModioX] [Request] Siren: Blood Curse (True HD Mod) (#1)
Could I confirm that these are the correct installation paths, starting at root:
"dev_hdd0/game/{REGION}/PS3_GAME/USRDIR/sirenx/data/commonen.dat",
"dev_hdd0/game/{REGION}/PS3_GAME/USRDIR/sirenx/data/commonuk.dat"
Thanks.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://github.com/wh1ter0se-x/ModioX/issues/1?email_source=notifications&email_token=AKJ2JFJKNZBYRBMB7N5WJOTQHQRDBA5CNFSM4ISSZ5P2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD5UKCKI#issuecomment-526950697, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AKJ2JFPOBJFPPXJEEAUWHOLQHQRDBANCNFSM4ISSZ5PQ.

Thanks for you contributions! I'm going to add a section in the readme for these type of contributions.
At this stage, ModioX is unable install the appropriate common.dat file and depend upon on the users region, so unfortunately all versions of the file will need to be installed to ensure at least the correct one is there. You can always go into the folders yourself and remove the files that aren't necessary afterwards, but I doubt it'll affect anything.
Please be sure to request any other mods you can find.
Regards.

@HeadstrongWolf also, I've found and added a mk9 mod but needs to be tested, if ever yiu want to try it out, before I get to soon. Many more other games and mods added too.

I'd love to. I may already have.
I'm at work but I'll check it out and upload what I got :)
Get Outlook for Androidhttps://aka.ms/ghei36

From: Ashley<EMAIL_ADDRESS>Sent: Tuesday, September 3, 2019 5:35:14 PM
To: wh1ter0se-x/ModioX<EMAIL_ADDRESS>Cc: HeadstrongWolf<EMAIL_ADDRESS>Mention<EMAIL_ADDRESS>Subject: Re: [wh1ter0se-x/ModioX] [Request] Siren: Blood Curse (True HD Mod) (#1)
@HeadstrongWolfhttps://github.com/HeadstrongWolf also, I've found and added a mk9 mod but needs to be tested, if ever yiu want to try it out, before I get to soon. Many more other games and mods added too.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://github.com/wh1ter0se-x/ModioX/issues/1?email_source=notifications&email_token=AKJ2JFL5FUN2P3WQG5NS2YTQH3RCFA5CNFSM4ISSZ5P2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD5ZY5BA#issuecomment-527666820, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AKJ2JFJGBMW6YYQWWSES4NTQH3RCFANCNFSM4ISSZ5PQ.


Add the ability to list all clusters

This adds an option to list all clusters associated with a Red Hat account that are spread across different AWS accounts.
This is the default view when using the Hybrid Console.  This change brings the same functionality to the ROSA CLI.
Related: https://issues.redhat.com/browse/OCM-1021

/lgtm

@jjulien Thanks for your contribution:)

@jjulien can you please fix the linter error?

/retest

/retest-required

@oriAdler I fixed the lint error, but looks like I can't trigger a re-test.

/retest

@oriAdler Sorry, missed the lint error about the line too long.  That is fixed now too.

/retest

/hold

/retest

/lgtm


null pointer dereference in exp

Issue type
Bug
Have you reproduced the bug with TensorFlow Nightly?
Yes
Source
source
TensorFlow version
tf 2.14.0
Custom code
Yes
OS platform and distribution
Ubuntu 18.04.6
Mobile device
No response
Python version
Python 3.8.3
Bazel version
bazel 5.3.0
GCC/compiler version
gcc 7.5.0
CUDA/cuDNN version
No response
GPU model and memory
No response
Current behavior?
A maliciously constructed exp operator model leads to op_context.input being empty, causing a null pointer dereference in the Prepare function.
// exp.cc
TfLiteStatus Prepare(TfLiteContext* context, TfLiteNode* node) {
  OpData* data = static_cast<OpData*>(node->user_data);
  TF_LITE_ENSURE_EQ(context, NumInputs(node), 1);
  TF_LITE_ENSURE_EQ(context, NumOutputs(node), 1);

  ExpContext op_context(context, node);
  const TfLiteTensor* input = op_context.input;
  TfLiteTensor* output = op_context.output;

  TfLiteIntArray* output_dims = TfLiteIntArrayCopy(input->dims);
  output->type = input->type;   // input is nullptr

exp.zip
Standalone code to reproduce the issue
I use the benchmark tool built according to [this official guide](https://www.tensorflow.org/lite/guide/build_cmake#step_1_install_cmake_tool), as follows:
1. git clone https://github.com/tensorflow/tensorflow.git tensorflow_src
2. mkdir tflite_build && cd tflite_build
3. cmake ../tensorflow_src/tensorflow/lite
4. cmake --build . -j
5. cmake --build . -j -t benchmark_model

The benchmark is in the tools directory

When I use the benchmark tool for PoC validation, it causes the TensorFlow Lite inference process to be subjected to a DOS（coredump).
❯ ./benchmark_model --graph=../poc/exp.tflite
INFO: STARTING!
INFO: Log parameter values verbosely: [0]
INFO: Graph: [../poc/exp.tflite]
INFO: Loaded model ../poc/exp.tflite
ERROR: Invalid tensor index 1025 in inputs. The subgraph has 3 tensors

INFO: Created TensorFlow Lite XNNPACK delegate for CPU.
[1]    6687 segmentation fault (core dumped)  ./benchmark_model --graph=../poc/exp.tflite

Relevant log output
No response

I was able to reproduce this issue on r2.14.

@pkgoogle Could you please look into this issue?
Thanks.

I was able to reproduce on nightly:
./benchmark_model --graph=exp.tflite
INFO: STARTING!
INFO: Log parameter values verbosely: [0]
INFO: Graph: [exp.tflite]
INFO: Loaded model exp.tflite
ERROR: Invalid tensor index 1025 in inputs. The subgraph has 3 tensors

INFO: Created TensorFlow Lite XNNPACK delegate for CPU.
Segmentation fault

@alankelly, can you please take a look. Thanks.

Hi, @SiriusHsh
Thanks for raising this issue. Are you aware of the migration to LiteRT? This transition is aimed at enhancing our project's capabilities and providing improved support and focus for our users. As we believe this issue is still relevant to LiteRT we are moving your issue there. Please follow progress here: https://github.com/google-ai-edge/LiteRT/issues/114
Let us know if you have any questions. Thanks.


python3Packages.sentry-sdk: 1.3.0 -> 1.3.1

Motivation for this change
Update to latest upstream release 1.3.1
Change log: https://github.com/getsentry/sentry-python/blob/master/CHANGELOG.md#131
Things done

[x] Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
Built on platform(s)

[x] NixOS
[ ] macOS
[ ] other Linux distributions

[ ] Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
[ ] Tested compilation of all pkgs that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
[ ] Tested execution of all binary files (usually in ./result/bin/)
21.11 Release Notes (or backporting 21.05 Relase notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module

[x] Fits CONTRIBUTING.md.

Result of nixpkgs-review pr 132244 run on x86_64-linux 1

2 packages failed to build:

home-assistant
octoprint

7 packages built:

alerta-server
cura
gns3-gui
gns3-server
moodle-dl
python38Packages.sentry-sdk
python39Packages.sentry-sdk


Update cloudbuild workflow for launcher

Create a overall cloudbuild file can trigger image building and testing cloudbuild workflow.
Signed-off-by: Jiankun Lu<EMAIL_ADDRESS>

/gcbrun

/gcbrun

/gcbrun


Added arm64 arch

Signed-off-by: Fernando Ike<EMAIL_ADDRESS>
- What I did
This is an implementation (#668) to build (and perhaps release) artifacts to support Linux AMR64 and Mac ARM64 architecture.
- How to verify it
Artifacts for Linux and Mac are with a suffix -arm64 and x86 or amd64. I can´t test the whole of scenarios because I don't have the credentials to do that. Feel free to commit here if you (maintainers) think this PR can be useful.
- Description for the changelog

Added ARM64 architecture support.

We maybe check the goreleaser new feature before closing this PR


Hackathon module

First commit, Created HackathonTheme

Let's start testing / fixing this PR so it can be merged soon.
Things todo:

[ ] Code review
[ ] Fix the forms (see the VueForms refactor #118)
[ ] Add some e2e tests


Memory Issues

Anyone else seeing memory issues with recent version of Chrome, Version 41.0.2272.104 (64-bit)?
See this stack overflow post Memory seems to grow indefinitely... Even when setting a debugger the memory use just keeps climbing...

Also, here is a video of the problem at work: https://youtu.be/FKv5CP3gxeA

I use karma-runner/jasmine/squire/900 tests and experienced phantomjs crash due to memory climbing up to 800Mb.
Same config with Chrome, Version 41.0.2272.118 takes 270Mb memory.

I create new Squire instance before each test and load/mock dependencies thru this instance. Some my dependencies depend on jquery. With Chrome profiler I discovered that jquery code takes a lot of memory. Looks like loading dependencies who depends on jquery via Squire causes additional jquery instantiation.
I fixed my memory issue by providing same jquery to each Squire:
define([
    "jquery",
    "squire"
], function ($, Squire) {
    "use strict";

    describe("Spec", function () {
        var utils;

        beforeEach(function (done) {
            injector = new Squire();
            injector
                .mock("jquery", $)
                .require(["utils"], function (_utils) {
                    utils = _utils;
                    done();
                });
        });

        afterEach(function () {
            injector.remove();
        });

        //tests for utils
    });
});

Same issue here but it doesn't seem related to the Chrome version.
We have an app that use angular 1.2.26 and we'd like to upgrade to 1.3.15.
When switching to 1.3.15, our test suite (3000 tests) slows down a lot and crashes on Chrome.
The suite slows down when the memory consumption reaches 1.5GB.
With the 1.2 version, it never goes over 1.0GB.
Another thing : The test suite work well until angular 1.3.0-beta18. All newer versions have the problem.
I'll report this also on the angular issue tracker as I'm not really sure it's related directly to Karma.


[question] Can I include multiple log levels instead of just one?

class API < Grape::API
  # ...
  logger.formatter = GrapeLogging::Formatters::Default.new
  log_file = File.open('./log/logfile.log', 'a')
  log_file.sync = true
  logger Logger.new GrapeLogging::MultiIO.new(STDOUT, log_file)
  insert_after Rack::Head, GrapeLogging::Middleware::RequestLogger, logger: logger, log_level: :info, include: [
    GrapeLogging::Loggers::ClientEnv.new,
    GrapeLogging::Loggers::RequestHeaders.new
  ]

  # ...
end

If I am understanding the readme file correctly, I am only able to specify one log log level, is there a way to specify multiple? And if there is, can it be added to the readme , that would be amazing :)

you cannot specify multiple log levels at the same time; though if specify for example DEBUG it will log DEBUG and higher up (so INFO, ERROR etc)


Add device42 provider

Problem Statement
Add device42 provider as a backend secret store
See - https://api.device42.com/#resource_Passwords
Related Issue
Fixes #...
Proposed Changes
How do you like to solve the issue and why?
Checklist

[X] I have read the contribution guidelines
[X] All commits are signed with git commit --signoff
[X] My changes have reasonable test coverage
[X] All tests pass with make test
[X] I ensured my PR is ready for review with make reviewable

Run make check-diff to see what's missing. :)

Please also update docs/introduction/stability-support.md.

check out this diff for an example on how: https://github.com/external-secrets/external-secrets/pull/3477/files#diff-3907c517f75c1bec56715c6226a0b6b7061092ea3a4506f90f9641bb1947d199

@Skarlso - thanks again.  I updated stability-support.md and also implemented network validation based on that link.


Implement a robots.txt file

Is your feature request related to a problem? Please describe.
Most people likely don't want their Portainer instance to be visible to Google. A robots.txt file disallowing crawling would be useful in this case.
Describe the solution you'd like
Add a robots.txt file that simply disallows all crawlers, or create a new configuration option for additional customizability
Describe alternatives you've considered
Meta tags could also be added to each page either preventing indexing or specifying a canonical URL.
Additional context
This is how a Portainer instance appears in Google search console. You can see that it is indexable.

Hi, thanks for the suggestion. We would be open to any contributions to prevent crawling from happening.

I'm new to Angular and can't be of much help... If someone could implement a robots.txt file with the content below that would be great!
User-agent: *
Disallow: /


Allow for more fonts

There is a need for the user to change the font when they implement receiver. The current request is to allow the ability to use Inter.

font-family is an inherited property, so what we should do for this is to remove all of the font-family properties from our CSS. That way the site's font will be applied to Receiver.
We should add a class to the Window component with an easy-to-remember name, like "receiver-window" and then add to the "custom styling" section of the README something that has a tip + example:
"To apply custom fonts to Receiver, you can use the "receiver-window" classname".receiver-window {
  font-family: 'Your Favorite Font';
}

Are we 100% sure we don't want to have a preselected font in place to start instead of inheriting whatever it's going to inherit from the site? Inheritance is how it looked awful in that one pic Sean posted, and it didn't even seem to match the site much

Did profiles support #79 include this?
This is part of Sybil PR path I think.


JSON schema validation always returns true

Trying to do JSON schema validation and it always returns true.
response contains the actual JSON response.
expectedSchema contains the expected schema structure
Below is the assertion code:
expect(response.body).to.have.schema(expectedSchema);
Executed the above code by providing the improper schema structure but still the assertion gets passed.

Thanks for the issue. Could you provide the schema and the JSON you are validating?

hi,
below is the sample code
var chakram = require('chakram'),
expect = chakram.expect;
// Schema for a source
var expectScheme = {
    Host : {type: 'string'},
    origin : {type: 'string'},
    link : {type: 'string'}

};
// describe defines the test suite
describe("Chakram", function() {
it("should check that the returned JSON object satisifies a JSON schema", function () {
return chakram.get("http://httpbin.org/get").then(function(response){

    // Displaying the response body of Get call
    console.log(response.body);
   expect(response).to.have.schema(expectScheme);
})

});
});
Response:

If you see the response schema and the expected schema are not the same.
Please let me know if I am missing anything

Thanks for the additional information. I am work at the moment, but will try it out later.
One thing to try:
var expectScheme = {
    type: "object",
    properties: {
       Host : {type: 'string'},
       origin : {type: 'string'},
       link : {type: 'string'},
   },
   required: [
      'Host',
      'link',
      'origin'
   ]
};

Thanks for the quick response. Looks like validation fails, if the required set property is not matching the schema..
Please let me know why we need to use required set... it is required for schema assertion?

Yes validation will fail. We are saying Host, link and origin are required in the response. Which they are not. I was trying to demonstrate that schema validation always passes as the issue suggests.

Your examples you sent pass as you have not got that required field. The schema is valid for the response received.

so you mean required set should be defined for the schema validation?

tnx  dareid..

how can we validate the data type is returned properly or not...
// Schema for a source
var expectSchema = {
  created_at: {type: 'string'},
    created_by: {type: 'number'},
    description: {type: 'string'},

required: [
'description',
'created_by',
]
};
in my response, created_by value is string  but still the assertion gets passed.

Could you provide the JSON which is incorrectly passing your defined schema above?

{ created_at: '2016-05-11T15:29:58Z',
created_by: 'abc',
description: 'Created......',
properties: null }

So your schema is incorrect. It should be something like this:
var expectScheme = {
    type: "object",
    properties: {
       created_at: {type: 'string'},
       created_by: {type: 'number'},
       description: {type: 'string'}
   },
   required: [
      'description',
      'created_by'
   ]
};

Maybe have a read through the documentation http://json-schema.org/

Got it.. Tnx for the resolution and timely support

hello @dareid , I need help for schema validation, could you please guide me on this. I tried the example above still always get the test passed. I want to change the expectedSchema in such a way that it should fail the test.
Here is the expectedSchema and actual result.
Actual Result (an array) :
[
  {
    "giftCardConfigurationId": 2,
    "applicableFeeName": "Admission Fee",
    "promotionPercent": 100,
    "generationLimit": 2,
    "active": true
  },
  {
    "giftCardConfigurationId": 4,
    "applicableFeeName": "Admission Fee",
    "promotionPercent": 100,
    "generationLimit": 1,
    "active": true
  }
]

Expected Schema ::

var expectedSchema = {
    "type": "array",
    "properties":
    {
      "giftCardConfigurationId": { "type": "number" },
      "applicableFeeName": { "type": "number" },
      "promotionPercent": { "type": "number" },
      "generationLimit": { "type": "number" },
      "active": { "type": "number" }
    },
    "required": [
      "giftCardConfigurationId",
      "applicableFeeName",
      "promotionPercent",
      "generationLimit",
      "active"
    ]
  };

Got the solution - Posting solution in case anyone else needs it.
Here is the link where you can convert the data to JSON schema with required fields.
http://jsonschema.net/


[BUG] Can't change tuning after configuring camera

Describe the bug
I want to be able to run the camera, do some analysis, then run it again with modified tuning parameters. I'm fairly sure in a previous version (I'm afraid I didn't note the version) this was possible - but at least using what's current on pypi and in Bookworm's python3-picamera2 package, it currently won't work.
The symptom is that the tuning is simply ignored for subsequent initialisations. I'm guessing the tuning gets somehow "baked in" but I am afraid I can't figure out how, after some amount of poring over the source code. It's possible something has changed in libcamera rather than picamera2 but I'm not really sure how I'd figure that out.
To Reproduce
I run the following (as test_tuning.py) with pytest, but hopefully it's easy enough to follow what's happening.

import os
from picamera2 import Picamera2
import time

import pytest

def load_default_tuning():
    with Picamera2() as cam:
        cp = cam.camera_properties
        fname = f"{cp['Model']}.json"
        return cam.load_tuning_file(fname)

def generate_bad_tuning():
    default_tuning = load_default_tuning()
    bad_tuning = default_tuning.copy()
    bad_tuning["version"] = 999
    return bad_tuning

def _test_bad_tuning_after_good_tuning(configure):
    bad_tuning = generate_bad_tuning()
    default_tuning = load_default_tuning()
    print("opening camera with explicitly specified tuning")
    with Picamera2(tuning=default_tuning) as cam:
        if configure:
            cam.configure(cam.create_preview_configuration())
    del cam
    with pytest.raises(IndexError):
        # The bad version should cause a problem
        with Picamera2(tuning=bad_tuning):
            print("Success (not expected)!")

def test_bad_tuning_after_good_tuning_noconfigure():
    _test_bad_tuning_after_good_tuning(False)

def test_bad_tuning_after_good_tuning_configure():
    _test_bad_tuning_after_good_tuning(True)

Expected behaviour
All tests should pass: each time we load a known-bad tuning, it should notice and fail. What actually happens is that test_bad_tuning_after_good_tuning_configure fails, because no exception is raised when we initialise the camera with the bad tuning.
Hardware :
Pi 4 2Gb, running Bookworm, and Raspberry Pi Camera Module v2.1

I wonder if there's been a change inside libcamera where the camera manager was getting (effectively) garbage collected when the Python handles were dropped...?

Presumably when we open the camera there might be the opportunity to load a different tuning file?

I think only CameraManager::stop() followed by CameraManager::start() will enable us to reload the tuning file.

As far as I can tell, there's a gCameraManager weak global pointer in the Python bindings layer that will hold onto the camera manager, unless everyone "drops" it. When the first Picamera2 instance is closed, however, this weak pointer doesn't get cleared, suggesting to me that something is still holding on to the camera manger - and therefore it doesn't get recreated for the second "open".
I assume that at one time, presumably, it was getting dropped "correctly", so the camera manager was being shut down and restarted. At the moment I can't see what it is that might be holding on to it.
Of course, this is still not good behaviour, because you couldn't change the tuning for one camera if a second camera was open. The real solution needs to be passing some kind of "blob" in when a camera is opened/acquired.

Thanks for this - I've located a previously-used SD card in the lab, with older dependencies. Using libcamera v0.0.4+22-923f5d70 and picamera2==0.3.12 (the latter installed with pip), I can confirm that both tests pass, albeit I had to modify them slightly as the exceptions raised were different. Modified version below for completeness.
import os
from picamera2 import Picamera2
import time

import pytest

def load_default_tuning():
    with Picamera2() as cam:
        cp = cam.camera_properties
        fname = f"{cp['Model']}.json"
    try:
        return cam.load_tuning_file(fname)
    except RuntimeError:
        dir="/usr/share/libcamera/ipa/raspberrypi"  # from picamera2 v0.3.9
        # The directory above has been removed from the search path, which I
        # find odd - as that's where the files currently are on a default
        # Raspbian image. This may need updating if the files have moved
        # in future updates to the system libcamera package
        return cam.load_tuning_file(fname, dir=dir)

def generate_bad_tuning():
    default_tuning = load_default_tuning()
    bad_tuning = default_tuning.copy()
    bad_tuning["version"] = 999
    return bad_tuning

def _test_bad_tuning_after_good_tuning(configure):
    bad_tuning = generate_bad_tuning()
    default_tuning = load_default_tuning()
    print("opening camera with explicitly specified tuning")
    with Picamera2(tuning=default_tuning) as cam:
        if configure:
            cam.configure(cam.create_preview_configuration())
    del cam
    with pytest.raises(Exception):
        # The bad version should cause a problem
        with Picamera2(tuning=bad_tuning):
            print("Success (not expected)!")

def test_bad_tuning_after_good_tuning_noconfigure():
    _test_bad_tuning_after_good_tuning(False)

def test_bad_tuning_after_good_tuning_configure():
    _test_bad_tuning_after_good_tuning(True)

I had to run the tests separately, as otherwise the camera was left in a half-initialised state and wouldn't work for the second test, but I can confirm that pytest test_tuning.py -k _noconfigure and pytest test_tuning.py -k _configure both pass.

I assume that at one time, presumably, it was getting dropped "correctly", so the camera manager was being shut down and restarted. At the moment I can't see what it is that might be holding on to it.

This makes perfect sense, thank you for explaining. I agree it would be lovely to pass in a blob explicitly rather than relying on tempfiles and environment variables, but I also realise that requires quite a lot of moving parts to fix what is probably a bit of an edge case.
For now, I will look for a way to ensure we call CameraManager::stop() and CameraManager::start() in between the two calls. If I can come up with a pure-Python work-around, that would be brilliant. Any pointers from either of you would be gratefully received!

I've done some poking around (see attached) and as far as the output of the gc module is concerned, the number of references to the cameramanager singleton does drop to zero when the camera is closed, even after I configure it (or get the sensor modes. I can see some debug output from libcamera when I create the singleton instance manually, and if I open and close a camera (without configuring it) this debug output repeats if I delete the instance and recreate it.
However, if I configure the camera, while the number of referrers still drops to zero, I don't see the debug output a second time - that suggests that it's not being garbage collected, as you suspected. I guess I will need to delve a level down to figure out what's keeping it alive - garbage collection in C python extensions is new to me, I'm afraid...
My script and output are below, for the record.
python investigate_cameramanager.py (investigate_cameramanager.py - renamed with .patch because I can't attach .py) yields:
Making singleton (should see debug output below)
[45:16:09.653402009] [23326]  INFO Camera camera_manager.cpp:284 libcamera v0.1.0+118-563cd78e
[45:16:09.699163713] [23332]  WARN RPiSdn sdn.cpp:39 Using legacy SDN tuning - please consider moving SDN inside rpi.denoise
[45:16:09.701348250] [23332]  WARN RPI vc4.cpp:390 Mismatch between Unicam and CamHelper for embedded data usage!
[45:16:09.702212787] [23332]  INFO RPI vc4.cpp:444 Registered camera /base/soc/i2c0mux/i2c@1/imx219@10 to Unicam device /dev/media4 and ISP device /dev/media0
[45:16:09.702272102] [23332]  INFO RPI pipeline_base.cpp:1142 Using configuration file '/usr/share/libcamera/pipeline/rpi/vc4/rpi_apps.yaml'
Referrers: __main__,
Making Picamera2
Referrers: <picamera2.picamera2.CameraManager object at 0xebb4acf0>, __main__,
cam.close()
Referrers: __main__,
deleted cam, manager -- should start afresh from here.
Making singleton (should see debug output below)
[45:16:09.922438028] [23326]  INFO Camera camera_manager.cpp:284 libcamera v0.1.0+118-563cd78e
[45:16:09.966077454] [23345]  WARN RPiSdn sdn.cpp:39 Using legacy SDN tuning - please consider moving SDN inside rpi.denoise
[45:16:09.968047120] [23345]  WARN RPI vc4.cpp:390 Mismatch between Unicam and CamHelper for embedded data usage!
[45:16:09.968900102] [23345]  INFO RPI vc4.cpp:444 Registered camera /base/soc/i2c0mux/i2c@1/imx219@10 to Unicam device /dev/media4 and ISP device /dev/media0
[45:16:09.968957639] [23345]  INFO RPI pipeline_base.cpp:1142 Using configuration file '/usr/share/libcamera/pipeline/rpi/vc4/rpi_apps.yaml'
Referrers: __main__,
Making Picamera2
Referrers: <picamera2.picamera2.CameraManager object at 0xebb4acf0>, __main__,
cam.configure()
[45:16:09.983732361] [23326]  INFO Camera camera.cpp:1183 configuring streams: (0) 640x480-XBGR8888 (1) 640x480-SBGGR10_CSI2P
[45:16:09.984295843] [23345]  INFO RPI vc4.cpp:608 Sensor: /base/soc/i2c0mux/i2c@1/imx219@10 - Selected sensor format: 640x480-SBGGR10_1X10 - Selected unicam format: 640x480-pBAA
Referrers: <picamera2.picamera2.CameraManager object at 0xebb4acf0>, __main__,
cam.close()
Referrers: __main__,
deleted cam, manager -- should start afresh from here.
Making singleton (should see debug output below)
Referrers: __main__,

The lack of debug output between the last two lines indicates that the cameramanager object did not get destroyed. This was using apt-installed libcamera and picamera2:
Package: python3-libcamera
Version: 0.1.0+rpt20231122-1
Package: python3-picamera2
Version: 0.3.16-1

I think I might have a hideous workaround. After deleting the Picamera2 object (cam in your script), add:
from picamera2.picamera2 import CameraManager
import gc
del Picamera2._cm
Picamera2._cm = None
gc.collect()
Picamera2._cm = CameraManager()

before opening the next instance. At least, that seemed to do the trick for me.
I wonder if the moral of the story is that this has never been guaranteed to work because it is all totally at the mercy of when and how Python garbage collection happens (which is a mystery to me too, frankly). And it seems that our luck may finally have run out. But the better plan is obviously to work towards a proper API for passing in this information when we open a camera.

If my understanding of the comments above is correct, I think that would solve my issue - I would be very happy to manually call close then open on the CameraManager, having first ensured there are no open cameras. I don't know the internals of libcamera well enough to know if that's a safe thing to do, but I would be delighted to try it out.
I think I have a proof-of-principle working. As I said, I am slightly concerned I'm doing something dangerous, but it does seem to run OK. Steps to reproduce my current setup:
sudo apt install libcamera-dev
python3 -m venv .venv  # Note: system-site-packages is not enabled
source .venv/bin/activate
git clone https://github.com/rwb27/pylibcamera.git
cd pylibcamera
pip install .  # Note: I had to increase swap to 500Mb on my 2Gb Pi 4.
pip install picamera2
cp -R /usr/lib/python3/dist-packages/pykms/ .venv/lib/python3.11/site-packages/ # can't install via pip

I can then run the attached python script, which calls libcamera.CameraManager.singleton().restart() after closing the Picamera2 object. This seems to reload everything.
investigate_gc.py
I've modified my test script above, and with the addition of the reset() in between, both tests now pass - i.e. it appears to correctly reload the tuning.
import os
from picamera2 import Picamera2
import libcamera
import time

import pytest

def load_default_tuning():
    with Picamera2() as cam:
        cp = cam.camera_properties
        fname = f"{cp['Model']}.json"
    try:
        return cam.load_tuning_file(fname)
    except RuntimeError:
        dir="/usr/share/libcamera/ipa/raspberrypi"  # from picamera2 v0.3.9
        # The directory above has been removed from the search path, which I
        # find odd - as that's where the files currently are on a default
        # Raspbian image. This may need updating if the files have moved
        # in future updates to the system libcamera package
        return cam.load_tuning_file(fname, dir=dir)

def generate_bad_tuning():
    default_tuning = load_default_tuning()
    bad_tuning = default_tuning.copy()
    bad_tuning["version"] = 999
    return bad_tuning

def _test_bad_tuning_after_good_tuning(configure):
    bad_tuning = generate_bad_tuning()
    default_tuning = load_default_tuning()
    print("opening camera with explicitly specified tuning")
    with Picamera2(tuning=default_tuning) as cam:
        if configure:
            cam.configure(cam.create_preview_configuration())
    del cam
    libcamera.CameraManager.singleton().restart()
    with pytest.raises(Exception):
        # The bad version should cause a problem
        with Picamera2(tuning=bad_tuning):
            print("Success (not expected)!")

def test_bad_tuning_after_good_tuning_noconfigure():
    _test_bad_tuning_after_good_tuning(False)

def test_bad_tuning_after_good_tuning_configure():
    _test_bad_tuning_after_good_tuning(True)

Thanks so much for your quick response!

I would be delighted to submit my reset() function to libcamera, but I'm not 100% sure the most appropriate way to do it - I am sure that my fork of pylibcamera is not the right way, but I think the actual libcamera repository is at linuxtv.org so a PR to the Raspberry Pi fork might not be helpful?

Just for the record, since closing this I've actually implemented both work-arounds in the OpenFlexure Microscope software (specifically https://github.com/rwb27/labthings-picamera2). I have learned:

@davidplowman's work-around is definitely nicer when it works (no errors printed to STDERR), but it does not work for me all the time. Clearly, there's something making references that don't get cleaned up. I have not yet found the time to chase these down, but my hunch is that they are at least partially in the C extension libcamera which makes them harder to find.
My (probably even nastier) work-around does work, but it prints errors to the console about closing devices that are in use. Clearly that's something I would like to fix. For now, it doesn't seem to be causing a problem - but I will come back to this when time permits. If I find a nicer solution, I'll do my best to remember to share it here.


port.write sends incomplete data

Summary of Problem

What are you trying to do?
I am trying to send a string to a serialport that will print it on a thermal printer

What happens?
I am sending the string to the port, but only the 10 first characters are being received.

What should have happened?
All characters in the string should have been sent.
When I sent it through a terminal using command :

 screen -L /dev/tty.usbserial-DN03U84D 115200

and then typing it and pressing Enter, all characters are received
Example :
String being sent : 41iabcdefghijklmnopqrst\r
Printer shows : 41iabcdefg
Sending the same string through the terminal, printer shows : 41iabcdefghijklmnopqrst
Additional data :
As indicated in the documentation I've turned on the debug, and here are the results :
  serialport/stream _write 24 bytes of data +3s
  serialport/binding-abstract write 24 bytes +3s
  serialport/stream drain +1ms
  serialport/binding-abstract drain +0ms
  serialport/bindings/unixWrite Starting write 24 bytes offset 0 bytesToWrite 24 +3s
  serialport/bindings/unixWrite write returned null 24 +0ms
  serialport/bindings/unixWrite wrote 24 bytes +0ms
  serialport/bindings/unixWrite Finished writing 24 bytes +0ms
  serialport/stream binding.write write finished +0ms
  serialport/stream binding.drain finished +0ms
  serialport/bindings/poller received "readable" +4s
  serialport/binding-abstract read +1s
  serialport/bindings/unixRead Starting read +4s
  serialport/bindings/unixRead Finished read 3 bytes +0ms
  serialport/stream binding.read finished +1s

Code to Reproduce the Issue
const SerialPort = require("serialport");

const baudRate = 115200;
const comPort = "/dev/tty.usbserial-DN03U84D";
const lock = true;

const device = new SerialPort(comPort, { baudRate, lock });

device.write('41iabcdefghijklmnopqrst\r');

Versions, Operating System and Hardware

<EMAIL_ADDRESS>Node.js v10.0.0
OSX Mojave

@reconbot In this example code yes the code is quitting automatically, but the general code stays up. I have an array of commands to be executed, the code execute the first one every second, and stays up as long as we don't quit the execution.
setInterval(() => {
  if (queue.length) {
    const instruction = queue.splice(0, 1);
    console.log("Exécution : " + instruction);
    device.write(instruction[0] + '\r', 'utf8', function (err) {
      if (err) {
        return console.log('Error on write: ', err.message)
      }
    });
    device.drain();
  }
}, 1000);

Same problem, both on MacOS and Windows some symbols randomly missing

@wasitum I'm currently experiencing a similar issue. Did you find a solution?


Issue 2283: Improve error message if authorization is not enabled

Motivation
Fixes #2283.
If authorizationEnabled is not enabled, authorization service is not set. All the accesses to this authorization service will throw NPE and clients will receive Internal Server Error. The message is meaningless. It is better to return a more meaningful message.
Changes
Return Not implemented if authorization is not enabled.

run java8 tests


protoc-3.7.0-freebsd-x86_64

Hi @kjopek, protobuf 3.7.0 is out (#59)
Interested in making protoc-3.6.0-freebsd-x86_64 available?

Sure. Today!

#64 is ready

Thanks! Will release soon, hopefully with #62

Released to maven central


facebook: can't distinguish some photos as public vs private

I added a photo post on my site (Known, self hosted) this morning which syndicated to Facebook, Twitter and Flickr (and manually to Google+). I noticed there are no comments appearing from Facebook so I decided to dive into it.
The post: https://tinokremer.nl/2016/hiep-hiep-hoera
The FB syndicated post: https://facebook.com/504157529757046
Both are fully public and the photo post does have the "Original" link in there which works correctly.
I'm wondering what is causing it to not appear. I can think of these reasons:

The post has not been discovered for some reason, I notice my photo appears in my wife's timeline as "Tino Kremer has posted 2 photos), could that be the reason?

There are 2 fails on the FB Bridgy page:

That failure is probably another bug which could be Known and not Bridgy. The link is https://tinokremer.nl/10102571892817971/conversatie-tussen-mijn-zoon-en-dochter-4zoon-ik-vind-het which does not appear to be working. Perhaps it did work before as I did receive quite a few comments on that post as well, original here: https://tinokremer.nl/2016/conversatie-tussen-mijn-zoon-en-dochter-4zoon-ik-vind-het

Something else?

I don't think I can see access what Bridgy sees on my FB profile, but I'm wondering what is causing it and if I can do something to fix it.

https://tinokremer.nl/10102571892817971/conversatie-tussen-mijn-zoon-en-dochter-4zoon-ik-vind-het

Ah darn, this is actually an unintended consequence of my changes to Known's permalink routing. It is now expecting a 4 digit year, whereas it used to accept any number. So when facebook changes 2016 to<PHONE_NUMBER>2817971, it doesn't work anymore.
Blech I didn't realize Facebook was rewriting links in the body of the post now. That suuucks.

Blech I didn't realize Facebook was rewriting links in the body of the post now. That suuucks.

Update: ok fortunately it isn't that. The post that's failing has a "See Original" link, rather than a link in the body, and that is what Facebook is rewriting. (Which is well known)

Is that the case? In this post it looks like it's just text "Original: link": "Original: https://tinokremer.nl/2016/hiep-hiep-hoera". I'm talking about the not discovered post https://facebook.com/504157529757046

You're right, the photo post seems like it should have been discovered. I wouldn't guess that has anything to do with the webmentions to https://tinokremer.nl/10102571892817971/conversatie-tussen-mijn-zoon-en-dochter-4zoon-ik-vind-het that are failing... Webmentions in the queue are more or less independent.

oh boy, fun!
for more background, the facebook bug that rewrites years in URLs to longer ids is #368.
@kylewm is right about the rest; we can look into it. looks like your bridgy user page is https://brid.gy/facebook/403320719840728 .

looks like this line in the log is the problem:
2016-01-24 16:04:28.001640 I Skipping non-public activity tag:facebook.com,2013:504157529757046

evidently we don't think https://facebook.com/504157529757046 is public, even though it is. not sure why; i'll investigate.

whee. so figuring out an FB post or photo's privacy setting is surprisingly hard, and can take multiple conditional API calls. background in https://github.com/snarfed/bridgy/issues/559#issuecomment-159642227 .
in this case, it looks like we followed the bread crumb trail correctly, from photo (object) id<PHONE_NUMBER>57046 back to consolidated "uploaded 2 photos" post id<PHONE_NUMBER>40728_504157529757046, which has privacy CUSTOM with no extra details:
"privacy": {
  "value": "CUSTOM",
  "description": "",
  "friends": "",
  "allow": "",
  "deny": ""
},

privacy field docs here...but they don't say anything about what CUSTOM with no details means. currently we're conservative and only interpret EVERYONE as public.
@tinokremer, does that photo or the album have anything more than just "Public" selected in privacy settings? maybe under "More Options"?
(ah facebook. never change!)

You're correct @snarfed. The 'album' (which is virtual, I haven't created a single album on Facebook myself) has it's setting to "Custom", meaning every photo will have its own privacy setting:

This is quite normal. I have several automatically created albums (I never create one myself) and all have that setting. I don't think I can change it either. This is what the 'album' page on Facebook says:

I'm actually annoyed Facebook does create albums and decides to show several completely unrelated photos as a group. That's what we have to deal with, it's a common issue with Silo's, they tend to do things a certain way and you, as a user, have no way to alter that.
I'm not an expert on the Facebook API (in fact, I know it exists, but haven't actually looked at it closely). It seems like "Custom" means the privacy setting of the individual photo's should be used instead.
I learned something as well, I should not post pictures when something like this is popping up within the next 24 hours. This is a 'card' letting my friends and family know it's both my kids birthday today. One of the most interesting ones to save comments from.

thanks for the details @tinokremer! sadly the photo object has no privacy field, and the post connected to it (by object_id) has the blank CUSTOM privacy field i posted above, se we fall back to the album's.
we've seen this before, e.g. https://github.com/snarfed/bridgy/issues/559#issuecomment-159642227. we originally defaulted to interpreting these cases as public, and we ended up backfeeding non-public responses.
that's definitely not acceptable, so we had to stop that. sadly it's looking like the tradeoff may be that we lose some public posts like this one, since there's no way to distinguish that they're public. i'd love to find a way though!

and you're definitely righ that facebook consolidating photo posts is annoying! we've had to do a lot of work to handle that so far, for both backfeed (#562) and publish (#571).

Can we get Bridgy to process the comments manually in this particular case? Or should I just leave it like it is for now or write something to import it manually myself? :)

@tinokremer sure! you can send webmentions for each comment and like manually, using https://brid.gy/about#source-urls to generate the bridgy source URLs.
that is a lot of tedious work though. #579 is probably what you're looking for, triggering backfeed for an entire post, but that's not supported yet, especially not for private posts. that check is very strict right now.
sorry. feel free to subscribe to #579 so you'll know if we do eventually do it!

Done and massive thanks for the support @kylewm and @snarfed !

FWIW, I believe "Custom" can also mean some complicated union of "These groups and not these groups". For a long time I had my default setting to share with "Friends minus people from high school I don't really like".

You could be very right @kylewm. I think more people use it like that. Unfortunately we have to deal with the woes of using those Silo's. There is a reason we're posting on our own sites after all :)

Just a heads-up for people that may encounter the same issue. I removed a few posts from FB yesterday, including the second 'photo' that was causing this virtual album and a while later all comments and likes were imported. Bridgy probably was now able to retreive everything as the virtual album was removed and it's a single photo again.
Do we want to keep this issue open @snarfed?

glad to hear it!
let's keep this issue open, since the problem still exists in general. that way, if it happens to someone else, and they come here, they'll be able to find this issue.

Facebook support is dead (#817, #826), so I'm closing this as obsolete. feel free to reopen if you have any concerns!


LND: Update anomaly in versions on screen and lncli

This is more of an anomaly than an issue:
I just updated my LND from v0.5 to v0.5.1. LND is running and i can normally operate lncli. I don't have any channels open so can't comment on that. The anomaly is:
The screen shows LND v0.5-beta while output of lncli --version is lncli version 0.5.1-beta commit=
I think while updating lnd was running. Can this be the cause? Should i clean install lnd? If so, how do i remove lnd from my pi?
Thank you!

I did not try LND 0.5.1 as the changes are minimal and not really relevant for this Beginner's Guide. What "screen" do you mean?

I mean, i have the 3.5" lcd screen attached to my Pi. Also, to be clear i'm using @rootzoll's Raspiblitz SD image for my Pi and since it's heavily based on your work and refers to your instructions, i posted the question here as even i follow your guides. Apologies if i shouldn't have posted the question here..

Also, a lot of people have been facing issues with LND 0.5.1 update with lost balances and closed channels, i wanted to update first before i seriously start opening channels and routing.

I've already created an issue with the lnd repository here: https://github.com/lightningnetwork/lnd/issues/2389


[CI] Pin Cython version to fix Cython compilation

Cython 3.0.0 was recently released, but it is incompatible with the current .pxi definitions in python/tvm/_ffi/_cpython.
Cython has been pinned for TVM in PR: 15353 until a working fix is created. This patch replicates the changes for tlcpack installation, as we are seeing the same Cython errors when trying to use tlcpack Ubuntu  Docker images.
cc @leandron @areusch @konturn @Mousius @lhutton1 @tqchen

Closing this PR as another has been raised in its place that doesn't exceed the scope of the patch and try to fix other CI issues too.


fix: property pane in preview mode

Description
Fixes an issue where property pane appeared in preview mode under certain conditions.
Fixes #37278
Automation
/ok-to-test<EMAIL_ADDRESS>:mag: Cypress test results

[!CAUTION]
If you modify the content in this section, you are likely to disrupt the CI result for your PR.

Communication
Should the DevRel and Marketing teams inform users about this change?

[ ] Yes
[ ] No

Summary by CodeRabbit

New Features

Introduced a new StaticLayout component for improved layout management.

Bug Fixes

Enhanced readability and consistency in defining column widths for various editor states.

Chores

Updated imports and exports to replace the deprecated UnanimatedLayout with StaticLayout.

/build-deploy-preview skip-tests=true


Fix memory leak from Cancellable helper

@aviau @mlefebvre
One of the memory leak at least. This pretty much leak every message...

So each message is kept in ram?

yeah, the task was kept alive forever


Building v1.7.0 is broken with vs2019 16.6.2

Describe the bug
Tinyexr has C++17 features that are illegal in C++20, yet needs to have the lang version specified to C++20 for build. More about this feature causing the issue can be found here. MS used to erroneously warn about it, but now correctly errors and thus we can't build. If we try to switch just those projects to C++17 where unnamed class typedefs with static members or member functions are legal, then the rest of the build fails as requiring at least C++20.
To Reproduce
Steps to reproduce the behavior:

Check out v1.7.0 and try to build with VS2019 v16.6.2.

Expected behavior
Should compile.
Screenshots
If applicable, add screenshots to help explain your problem.
Desktop (please complete the following information):

OS: Win 10 1909
GPU: RTX 2060
Backend: Vulkan

Additional context
Add any other context about the problem here.

I believe newer commits solved the issue, maybe time for a new release?

I think we've updated tinyexr since the 1.7.0 release: https://github.com/google/filament/pull/2644. Can you try building on the latest main?

I'll try a fresh build from head, just didn't know what was up with skybox gen. If I switch just the tinyexr to c++17 I can build, I had erroneously changed a second project depending on it to c++17 as well when I filed this. Will update after test.

Skygen was recently removed.

@romainguy yeah I saw the comments about a real time system replacement, just didn't see if that was already in or what the roadmap was etc.

It's not there yet.

Issue is solved as of 1af254d. Built from there with no issues. Thanks for your time everyone! Thanks for the library. :)


dynatrace_web_application: Tag with UniqueId 1 was deleted before

Describe the bug
We are attempting to add User Tagging to our Web Applications, as shown in the examples from this project.
We are obtaining the following error from applications that have already had User Tag rules defined and removed in the past:
Tag with UniqueId 1 was deleted before
This seems to be due to the fact that Dynatrace auto-increments user_tags.tag.id, and an existing ID can never be reused once a rule has been deleted.
This feature is unusable if we have to manually manage tag IDs for each application, which are an implementation detail.
To Reproduce
Steps to reproduce the behavior:

Manually define an initial User Tag rule on an application (it will have id = 1)
Delete the User Tag rule
Attempt to apply a User Tag rule to the application with Terraform:

user_tags {
  tag {
	cleanup_rule = "cleanup rule here"
	id           = 1
	ignore_case  = false
	metadata_id  = 1
  }
 }
}

Expected behavior
The user tag is applied with the lowest ID number allowed by Dynatrace.
Additional context

Terraform 1.0.8
Dynatrace provider 1.28.0
Adding the ID to ignore_changes has no effect on the problem

Hello Matt,
What you've stumbled across is a limitation that exists for all clients of the Dynatrace REST API. The details are a bit complicated, but in order to maintain a history of Visits, it is not allowed to re-use these unique IDs. Also our team was not aware of that fact until you filed the issue.
With the upcoming release, in order to deal with these restrictions properly, the property id won't be taken into consideration anymore. The attribute is still available within the Schema, in order to stay backwards compatible. But the Provider won't consider its value anymore. The provider will maintain a computed attribute unique_id - and it will determine it's value autonomously.
best regards,
Reinhard

That sounds perfect. Can you tell me when the release with this fix will be published?

We usually push out updates once per week. So, mid next week either v1.30.2 (if the release just contains bug fixes) or v1.31.0 (if it contains new functionality) will contain the fix.

Thank you.

I can confirm that this issue is resolved.


feat: Compress web UI with gzip

From 60kb → 20kb.

This won't work because I'm using a global find/replace of text to apply parameters. So it can't find/replace in the gzip encoded string.


Break on line 90 - main.go

Break on line 90 in main.go breaks the loop
Also -discord flag and -remote flag causes problems
V3.2

Yep sorry about that, using test mode only iterates a single time and I'm not testing the main loop sigh. Removed 3.3, I've been testing sms and that's been working still trying to look into discord (I had thought it was fixed but I guess I was wrong)

Yep sorry about that, using test mode only iterates a single time and I'm not testing the main loop sigh. Removed 3.3, I've been testing sms and that's been working still trying to look into discord (I had thought it was fixed but I guess I was wrong)

sms is sending a bad link to an empty shopping cart

@fokeiro2020 Can you run it a few times, apparently we've been seeing inconsistent results with the sms (I've had it work on LAN/LTE and after clearing my browser cache multiple times). Other folks haven't.

Yep sorry about that, using test mode only iterates a single time and I'm not testing the main loop sigh. Removed 3.3, I've been testing sms and that's been working still trying to look into discord (I had thought it was fixed but I guess I was wrong)

sms is sending a bad link to an empty shopping cart

I'm curious, what mobile OS are you using (andriod or IOS)?
also, are you using a web interface to access your messages (like messages.google.com)?
@ianmarmour Do you think it's because these apps like discord are accessing the link first? I really don't know, just guessing.

I'm using messenger built into IOS (and yes, that's one theory I have)

Yep sorry about that, using test mode only iterates a single time and I'm not testing the main loop sigh. Removed 3.3, I've been testing sms and that's been working still trying to look into discord (I had thought it was fixed but I guess I was wrong)

sms is sending a bad link to an empty shopping cart

I'm curious, what mobile OS are you using (andriod or IOS)?
also, are you using a web interface to access your messages (like messages.google.com)?
@ianmarmour Do you think it's because these apps like discord are accessing the link first? I really don't know, just guessing.

Android s10+ , google sms app and link open to chrome

@rjprevost So just verified manually sending a link through gmail to someone else.... that worked fine, did the same thing via discord and it breaks...
Pretty sure discord is monitoring people's posted links and visiting them.... real creepy.

Yeah, that's what I'm thinking.
I think Google sms is doing the same thing, though I've only tested it via their webapp (messages.google.com)

Yeah I'm 99.99% sure this is why :/ not really sure the best way to get around this (but hey it's nice to know Apple isn't harvesting url metadata on cellphones (maybe they are concerned about privacy)

revert for now this is an easy fix but I'm going to bundle it with the notifications changes.

So I think I may have cracked it... pretty sure discord attempts auto embeds unless you wrap a link in <>

That makes sense. I'll test it real quick.

nvidia-clerk-windows.zip
Here's a build with the change

I'm running a mac, but I think it worked for me. Just added the <> and it seemed to work fine.

Awesome also for your sms, I think you can just disable the link preview feature on android.

Not going to test sms right now, but Discord is working consistently. That fixed it.

Disabling link preview fixed the sms link for me! Good catch.

https://github.com/ianmarmour/nvidia-clerk/releases/tag/v3.5


add uploadLLCSegment endpoint in TableResource

Description
To complete deep store bypass for realtime segment completion, we need to create an endpoint to upload LLC segment to segment store.
Will create another pr for controller periodic task: RealtimeSegmentValidationManager to call this endpoint for LLC segment without segment store copy.
Upgrade Notes
Does this PR prevent a zero down-time upgrade? No.
Does this PR fix a zero-downtime upgrade introduced earlier? No.
Does this PR otherwise need attention when creating release notes? No.
Documentation
This change is proposed by: By-passing deep-store requirement for Realtime segment completion

Codecov Report

Merging #6653 (01f78a2) into master (1beaab5) will decrease coverage by 0.61%.
The diff coverage is 62.64%.

@@            Coverage Diff             @@
##           master    #6653      +/-   ##
==========================================
- Coverage   66.44%   65.83%   -0.62%
==========================================
  Files        1075     1360     +285
  Lines       54773    66562   +11789
  Branches     8168     9705    +1537
==========================================
+ Hits        36396    43818    +7422
- Misses      15700    19619    +3919
- Partials     2677     3125     +448

Flag
Coverage Δ

unittests
65.83% <62.64%> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files
Coverage Δ

...e/pinot/broker/api/resources/PinotBrokerDebug.java
0.00% <0.00%> (-79.32%)
:arrow_down:

...pinot/broker/api/resources/PinotClientRequest.java
0.00% <0.00%> (-27.28%)
:arrow_down:

...ot/broker/broker/AllowAllAccessControlFactory.java
71.42% <ø> (-28.58%)
:arrow_down:

.../helix/BrokerUserDefinedMessageHandlerFactory.java
33.96% <0.00%> (-32.71%)
:arrow_down:

...ker/routing/instanceselector/InstanceSelector.java
100.00% <ø> (ø)

...ava/org/apache/pinot/client/AbstractResultSet.java
66.66% <ø> (+9.52%)
:arrow_up:

...n/java/org/apache/pinot/client/BrokerResponse.java
100.00% <ø> (ø)

.../main/java/org/apache/pinot/client/Connection.java
35.55% <ø> (-13.29%)
:arrow_down:

...org/apache/pinot/client/DynamicBrokerSelector.java
82.85% <ø> (+10.12%)
:arrow_up:

...n/java/org/apache/pinot/client/ExecutionStats.java
68.88% <ø> (ø)

... and 1243 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 5137025...01f78a2. Read the comment docs.

Hi @liuchang0520, similar work is currently worked from https://github.com/apache/incubator-pinot/pull/6567.
Are you aware of this?

Hi @liuchang0520, this feature is already being worked on at #6567.
Are you aware of this?

Hi @liuchang0520, this feature is already being worked on at #6567.
Are you aware of this?

Hi @snleee , my understanding is this change covers a different segment upload path: server uploads LLC segment to segment store, though I am not sure. @chenboat Is there an overlap between this pr and #6567 ?

Yes. This PR is about deep store by pass and not related to #6567. The added endpoint is for the Pinot servers to upload segments to deep store.


Drawer menu is hidden immediately in iOS 11.3

env: iOS 11.3 beta 6

Open the drawer menu in Safari
Scroll the drawer menu (address bar will be resized)
The drawer menu will be hidden immediately.

workarounds:
$drawer.drawer({
	iscroll: {
		preventDefault: true,
		click: true,
	},
});

You can disable the resize event of the drawer.
$(window).off('resize.drawer');

But it seems necessary to update this plugin as well.

Neither of those fix the issue for me on iOS 11.3.

@koogawa @blivesta It appears while that fix worked on iOS it has now broken on Android. Back to the drawing board.

The solution suggested by @koogawa solved it for me too. But don't forget to add mouseWheel: true to the iscroll properties, just to make sure it scrolls on desktop.


Postmessage Error

porthole.js:314 Uncaught TypeError: this.getTargetWindow(...).postMessage is not a functionPorthole.WindowProxyHTML5.Porthole.WindowProxyBase.extend.dispatchMessage @ porthole.js:314Porthole.WindowProxyBase.PortholeClass.extend.post @ porthole.js:204window.onload @ (index):62

Note: This was in chrome.

I've got the same issue than you and I it was a mistake from me...
Your iframe need to have a name attribute and an id to work perfectly.


[BUG] glob-style pattern with range ([0-9]) not working

setuptools version
68.0.0
Python version
Python 3.10.12
OS
Ubuntu 22.04 under WSL2
Additional environment information
Environment
$ python --version
Python 3.10.12

$ pip freeze
build==0.10.0
packaging==23.1
pyproject_hooks==1.0.0
# Editable install with no version control (setuptools-tests==1.0)
-e <somedir>/setuptools-test
tomli==2.0.1

Setup to reproduce the problem
setuptools-globstyle-bug.zip
Description
When MANIFEST.in contains global-exclude *202*, any file with this string is being excluded such as file3_20230814T111336.py (WORKS AS INTENDED).
However, when  MANIFEST.in contains global-exclude *202[0-9]*, any file with a string that follows that glob-style pattern gets included, such as one named file3_20230814T111336.py.  That is the BUG.
Note that I tested the glob-style pattern an it is valid:
$ find . -name '*202[0-9]*'
./src/file3_20230814T111336.py

Last note, when  MANIFEST.in contains global-exclude *202[23]*, that does work and file3_20230814T111336.py gets excluded.  The [] syntax seams to work partially, but its "range" functionality does not.
Setuptools is not following what its documentation is referring to at Controlling files in the distribution which points to PyPA’s Packaging User Guide.  In this last page, we do find:

matches any one of the characters between the square brackets (which may contain character ranges, e.g., [a-z] or [a-fA-F0-9]).

Expected behavior
global-exclude *202[0-9]*, any file with a string that follows that glob-style pattern should get excluded.
How to Reproduce
setuptools-globstyle-bug.zip provides an environment where one can test this.  Simply edit MANIFEST.in to test the glob-style pattern you plan to test and call build to see what gets included in the wheel.
Output
When MANIFEST.in contains global-exclude *202[0-9]*, we notice below that the file file3_20230814T111336.py gets included.
unzip -l dist/*.whl

  Length      Date    Time    Name
---------  ---------- -----   ----
       45  2023-08-14 13:09   file1.py
       45  2023-08-14 13:09   file2.py
       45  2023-08-14 13:09   file3_20230814T111336.py
       45  2023-08-14 13:09   file3_print.py
      265  2023-08-14 16:58   setuptools_tests-1.0.dist-info/METADATA
       92  2023-08-14 16:58   setuptools_tests-1.0.dist-info/WHEEL
       46  2023-08-14 16:58   setuptools_tests-1.0.dist-info/top_level.txt
      599  2023-08-14 16:58   setuptools_tests-1.0.dist-info/RECORD
---------                     -------
     1182                     8 files

Hi @deragon, thank you very much for opening this issue.
I am afraid, there is a bit more to this problem that it looks on the surface :P
Regarding the PyPA’s Packaging User Guide, please note the following items:

The following files are included in a source distribution by default:

all Python source files implied by the py_modules and packages setup() arguments
...

Since you are not providing the py_modules/packages argument to setup() (or any of the equivalents in pyproject.toml or setup.cfg), you ended up triggering the auto-discovery which will scan the project directory to auto-fill these configurations (before MANIFEST.in is evaluated).
As a result file3_20230814T111336 is found as an item in py_modules and is added in the sdist/wheel because it matches the first item in the PyPA docs (the lack of configuration for packages/py_modules implies that src/file3_20230814T111336.py should be included in the distribution).
I think the main problem comes with the line in PyPA docs:

Default files can even be removed from the sdist with the appropriate MANIFEST.in command.

This has proven to be imprecise, as in this case py_modules are not excluded.
I don't know exactly where is statement in the PyPA docs is coming from, but I had a look on the code and it was quite hard to find the combination that would result on that. (Specifically setuptools uses a mixture of sdist, egg_info, build_py, egg_info.manifest_maker to find files for the sdist so it is not a trivial task).
If you or any member of the community would like to investigate this further, any efforts are welcome. Otherwise, I wonder if it would be better if we just amend PyPA docs to reflect what actually happens in the current state of the implementation.
Regardless your specific problem, I think the best workaround would be to not rely on auto-discovery, but instead specify the configurations for packages, package_dir and py_modules explicitly.
Please find an example below.
> docker run --rm -it python:3.10-bullseye /bin/bash

cd /tmp
wget https://github.com/pypa/setuptools/files/12338343/setuptools-globstyle-bug.zip
unzip setuptools-globstyle-bug.zip
cd setuptools-globstyle-bug

cat <<EOF >> pyproject.toml
[tool.setuptools]
package-dir = {"" = "src"}
py-modules = ["file1", "file2", "file3_print"]
EOF

python -m venv .venv
.venv/bin/python -m pip install build
.venv/bin/python -m build

tar tf dist/*.tar.gz
# setuptools-tests-1.0/
# setuptools-tests-1.0/MANIFEST.in
# setuptools-tests-1.0/PKG-INFO
# setuptools-tests-1.0/README.md
# setuptools-tests-1.0/pyproject.toml
# setuptools-tests-1.0/setup.cfg
# setuptools-tests-1.0/src/
# setuptools-tests-1.0/src/file1.py
# setuptools-tests-1.0/src/file2.py
# setuptools-tests-1.0/src/file3_print.py
# setuptools-tests-1.0/src/setuptools_tests.egg-info/
# setuptools-tests-1.0/src/setuptools_tests.egg-info/PKG-INFO
# setuptools-tests-1.0/src/setuptools_tests.egg-info/SOURCES.txt
# setuptools-tests-1.0/src/setuptools_tests.egg-info/dependency_links.txt
# setuptools-tests-1.0/src/setuptools_tests.egg-info/requires.txt
# setuptools-tests-1.0/src/setuptools_tests.egg-info/top_level.txt

unzip -Z1 dist/*.whl
# file1.py
# file2.py
# file3_print.py
# setuptools_tests-1.0.dist-info/METADATA
# setuptools_tests-1.0.dist-info/WHEEL
# setuptools_tests-1.0.dist-info/top_level.txt
# setuptools_tests-1.0.dist-info/RECORD


perlPackages.FutureAsyncAwait: 0.58 -> 0.66

Description of changes
https://metacpan.org/release/PEVANS/Future-AsyncAwait-0.66/source/Changes#L3-74
Needs https://github.com/NixOS/nixpkgs/pull/262100 first!

Things done

Built on platform(s)

[ ] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[x] aarch64-darwin

[ ] For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[x] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[x] Tested basic functionality of all binary files (usually in ./result/bin/)
23.11 Release Notes (or backporting 23.05 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module

[x] Fits CONTRIBUTING.md.

@GrahamcOfBorg build perlPackages.FutureAsyncAwait


Dataflow runner jobs timeout due to long pip resolution times

Summary
The installation of the apache-beam[gcp] package takes too long due to BEAM-22218, causing the Dataflow runner jobs to timeout and fail after an hour without any progress.
Reproducing Steps
# set up .cdsapirc
git clone https://github.com/google/weather-tools.git
python3 -m venv env
. env/bin/activate
cd weather-tools
pip install .
weather-dl configs/era5_example_config_local_run.cfg --runner DataflowRunner --project <PROJECT> --temp_location <SOME_GCS_PATH> --job_name morbius --region us-central1

Expectation
Dataflow pipeline runs successfully
Outcome
Dataflow worker is created but job never starts, fails after an hour with the error "The Dataflow job seems to be stuck".
Symptoms

Worker VM CPU usage at 100%
Fetch job step never starts running, other steps never produce an output
Infinite loop of INFO logs like:

INFO: pip is looking at multiple versions of google-cloud-bigquery to determine which version is compatible with other requirements. This could take a while.

Final error:

The Dataflow job appears to be stuck because no worker activity has been seen in the last 1h. Please check the worker logs in Stackdriver Logging. You can also get help with Cloud Dataflow at https://cloud.google.com/dataflow/support.

Diagnosis
Build fails after 2h when using --prebuild_sdk_container_engine=cloud_build
Pip tries over and over again to find the right version of google-cloud-* packages
According to BEAM-22218, the issue seems to be resolved in newer pip versions but it's hard to apply their workaround without either using a custom image or downgrading Beam version.
Workaround
Fixing GCP package versions and moving beam to the end of the list:
base_requirements = [
    "cdsapi",
    "ecmwf-api-client",
    "numpy>=1.19.1",
    "pandas",
    "xarray",
    "requests>=2.24.0",
    "firebase-admin>=5.0.0",
    "google-cloud-firestore",
    "urllib3==1.26.5",
    "google-cloud-bigquery==2.34.4",
    "google-cloud-bigquery-storage==2.14.1",
    "google-cloud-bigtable==1.7.2",
    "google-cloud-core==1.7.3",
    "google-cloud-datastore==1.15.5",
    "google-cloud-dlp==3.8.0",
    "google-cloud-language==1.3.2",
    "google-cloud-pubsub==2.13.4",
    "google-cloud-pubsublite==1.4.2",
    "google-cloud-recommendations-ai==0.2.0",
    "google-cloud-spanner==1.19.3",
    "google-cloud-videointelligence==1.16.3",
    "google-cloud-vision==1.0.2",
    "apache-beam[gcp]==2.40.0",
]

Env resolution takes about 5m after this.

Moving apache-beam to the end of the list as a solution is extremely clever!


Proposal: Syntax for specifying binding rules

It might be nice to have a syntax for specifying lexical binding rules for languages. Currently, the best way to do name resolution is inside an attribute/operation, but this makes it more difficult to reuse the results. A declarative syntax could be useful for (say) an IDE that wants support renaming variables, without having to rely on language- and implementation-specific attributes/operations.
One idea I had is a syntax to embed the binding rules in the grammar itself: https://gist.github.com/pdubroy/984ad6e3ecf181b1c15c#file-gistfile1-txt. However, this would complicate the grammars somewhat, and would likely only be able to handle simple cases.
Another option would be some sort of supplemental syntax -- maybe separate rules in the grammar that define how names are resolved. E.g.:
ML {
  Expr = ...
  LetExpr = ...

  @LetExpr = /* binding syntax goes here */
}

Some relevant papers:

A Theory of Name Resolution with Extended Coverage and Proofs
Romeo: a System For More Flexible
Binding-Safe Programming

Also: Toward a General Theory of Names, Binding and Scope

If we implement this, it would require a lot of thought and careful design, and we should start by implementing it as a library feature first.

Eelco Visser and co. seem to have a nice way to specify this kind of thing. We should study their approach carefully. Here are some links:

http://researchr.org/publication/NeronTVW15
http://researchr.org/publication/VanAntwerpenNTVWPEPM2016
http://conf.researchr.org/event/POPL-2016/popl-2016-tutorials-t2-part-2-declare-your-language-name-binding-with-scope-graphs