duckdb gives an internal error when trying to select from multiple GRAPH_TABLE's

What happens?
When attempting to run a query that selects from multiple property graph tables, the following error is given: "Error: INTERNAL Error: Attempted to dereference unique_ptr that is NULL!"
According to the property graph sql standard described in ISO/IEC 9075-16, selecting from multiple GRAPH_TABLE's (implicit cross join) should be possible.
To Reproduce
CREATE TABLE cities (
name VARCHAR,
lat DECIMAL,
lon DECIMAL
);

CREATE TABLE cities_are_adjacent (
city1name VARCHAR,
city2name VARCHAR
);

-CREATE PROPERTY GRAPH citymap
VERTEX TABLES (
cities PROPERTIES (name,lat,lon) LABEL city
)
EDGE TABLES (
cities_are_adjacent SOURCE KEY ( city1name ) REFERENCES cities ( name )
DESTINATION KEY ( city2name ) REFERENCES cities ( name )
LABEL adjacent
);

D -select * from GRAPH_TABLE (citymap MATCH (s:city)-[r:adjacent]->(t:city)) g1;

┌─────────┬───────────────┬───────────────┬───┬───────────────┬───────────────┐
│ name │ lat │ lon │ … │ lat_1 │ lon_1 │
│ varchar │ decimal(18,3) │ decimal(18,3) │ │ decimal(18,3) │ decimal(18,3) │
├─────────────────────────────────────────────────────────────────────────────┤
│ 0 rows │
└─────────────────────────────────────────────────────────────────────────────┘
D -select * from GRAPH_TABLE (citymap MATCH (s:city)-[r:adjacent]->(t:city)) g1, GRAPH_TABLE (citymap MATCH (s:city)-[r:adjacent]->(t:city)) g2;
Error: INTERNAL Error: Attempted to dereference unique_ptr that is NULL!

OS:
Linux x86_64
DuckDB Version:
v0.10.1-dev17 bb9b820
DuckDB Client:
c++
Full Name:
Jeff Cavano
Affiliation:
eBay
What is the latest build you tested with? If possible, we recommend testing with the latest nightly build.
I have tested with a stable release
Did you include all relevant data sets for reproducing the issue?
Yes
Did you include all code required to reproduce the issue?

[X] Yes, I have

Did you include all relevant configuration (e.g., CPU architecture, Python version, Linux distribution) to reproduce the issue?

[X] Yes, I have

Got it, thanks for the update.

Fixed in https://github.com/cwida/duckpgq-extension/pull/122 and should now be available when doing force install 'duckpgq' in DuckDB v1.0.0


Containerservice 2022 09 02 preview fleet

ARM API Information (Control Plane)
MSFT employees can try out our new experience at OpenAPI Hub  - one location for using our validation tools and finding your workflow.

Azure 1st Party Service can try out the Shift Left experience to initiate API design review from ADO code repo.  If you are interested, may request engineering support by filling in with the form https://aka.ms/ShiftLeftSupportForm.
Changelog
Add a changelog entry for this PR by answering the following questions:

What's the purpose of the update?

[ ] new service onboarding
[ ] new API version
[ ] update existing version for new feature
[ ] update existing version to fix swagger quality issue in s360
[ ] Other, please clarify

When are you targeting to deploy the new service/feature to public regions? Please provide the date or, if the date is not yet available, the month.
When do you expect to publish the swagger? Please provide date or, the the date is not yet available, the month.
By default, Azure SDKs of all languages (.NET/Python/Java/JavaScript for both management-plane SDK and data-plane SDK, Go for management-plane SDK only ) MUST be refreshed with/after swagger of new version is published. If you prefer NOT to refresh any specific SDK language upon swagger updates in the current PR, please leave details with justification here.

Contribution checklist (MS Employees Only):

[x] I commit to follow the Breaking Change Policy of "no breaking changes"
[x] I have reviewed the documentation for the workflow.
[x] Validation tools were run on swagger spec(s) and errors have all been fixed in this PR. How to fix?

If any further question about AME onboarding or validation tools, please view the FAQ.
ARM API Review Checklist

Applicability: :warning:
If your changes encompass only the following scenarios, you should SKIP this section, as these scenarios do not require ARM review.

Change to data plane APIs
Adding new properties
All removals

Otherwise your PR may be subject to ARM review requirements. Complete the following:

[x] Check this box if any of the following apply to the PR so that the label "ARMReview" and "WaitForARMFeedback" will be added by bot to kick off ARM API Review.  Missing to check this box in the following scenario may result in delays to the ARM manifest review and deployment.

Adding a new service
Adding new API(s)
Adding a new API version
[] To review changes efficiently, ensure you copy the existing version into the new directory structure for first commit and then push new changes, including version updates, in separate commits. You can use OpenAPIHub to initialize the PR for adding a new version. For more details refer to the wiki.

[x] Ensure you've reviewed following guidelines including ARM resource provider contract and REST guidelines. Estimated time (4 hours). This is required before you can request review from ARM API Review board.

[ ] If you are blocked on ARM review and want to get the PR merged with urgency, please get the ARM oncall for reviews (RP Manifest Approvers team under Azure Resource Manager service) from IcM and reach out to them.

Breaking Change Review Checklist
If you have any breaking changes as defined in the Breaking Change Policy, request approval from the Breaking Change Review Board.
Action: to initiate an evaluation of the breaking change, create a new intake using the template for breaking changes. Additional details on the process and office hours are on the Breaking Change Wiki.
NOTE: To update API(s) in public preview for over 1 year (refer to Retirement of Previews)
Please follow the link to find more details on PR review process.

The contribution checklist for this draft PR is complete, but the PR is marked as 'DRAFT'. If this is no longer a draft, and you are ready for ARM review, please send email to the current ARM API Review oncall.

Just noticing that the target branch is just another private branch, so this can postpone ARM approval if required, as long as the PR for merging the final branch is reviewed.
Azure:dev-containerservice-Microsoft.ContainerService-2022-09-02-preview

All looking pretty good to me so far!


control non-javascript menu

When I visit a TiddlyWebWiki without javascript I get the message
"This page requires JavaScript to function properly.
If you do not use JavaScript you may still browse the content of this wiki."
I would like to change this message and link in a similar way to MarkupPreHead / MarkupPostHead so that I can explore in TiddlySpace improving the experience of non-js users in my space. It would also be useful to authoring SplashScreenPlugins.

doesn't seem to relevant anymore, wontfix


List required Rails version in Requirements

Bootsy uses the optional option of Rails' belongs_to, which is only available in Rails 5. It would be helpful if the precise Rails version requirement was listed in the requirements section of your README.

Rails 4 and older are no longer supported by the Rails Core Team. Our README assumes that by now everybody already upgraded to Rails 5.

I disagree with this assumption. The Rails Core team still supports 4.2.x:

Not only that, but we have many clients with legacy code bases that we support, from 2.x all the way to 4.x.
@mlaco isn't asking for 4.x support. Just a note in the readme so we know not to waste time trying to get Bootsy to work.

I disagree with this assumption. The Rails Core team still supports 4.2.x

The link you referred to is outdated. See https://github.com/rails/rails/blob/master/guides/source/maintenance_policy.md

we have many clients with legacy code bases that we support, from 2.x all the way to 4.x.

I'd recommend you to encourage your clients to update those applications asap, then.

@mlaco isn't asking for 4.x support. Just a note in the readme so we know not to waste time trying to get Bootsy to work.

Alright, then. I listed "Rails 5" in the Requirements section: https://github.com/volmer/bootsy#requirements

Thank you, appreciate it.


Fixed fail on empty from field

Hi,
message.from can be nil (https://core.telegram.org/bots/api#message).
In that case the gem throws an exception.
I've found basic specs only. Should I write a spec for this case?

Thanks, merged!


the lines of the logs

Hello, I would have liked to increase the lines of the logs. I tried with this command line, "sudo -n tail -n 500 -f /var/log/homebridge.log" but it did not work. Can you help me?

Hi @djrams10,
I'm assuming your using docker based on your previous issue.
It's currently not possible to change the log settings for docker, the app forces the certain settings when it detects it's running inside oznu/docker-homebridge. Unfortunately this means it's also not possible to increase the tail line beyond 200.
I have been meaning to increase this to 500 by default; as that's what systemd logging's default is.

thank you for your explanations, and for your work.

Change will be in 4.6.0


Understanding Amazon EC2

Proposed title of article
Understanding Amazon EC2
Proposed article introduction
AWS is effective in enhancing your cloud security. It provides control as well as confidence that simultaneously allow flexibility within your cloud environment. It is highly beneficial in ensuring that identities, devices, applications as well as information are secure.
The Amazon EC2 offers computing capabilities that are scalable and flexible. This help in managing the cost for the hardware as servers are run in the virtual environment. AWS is effective in enhancing your cloud security. It provides control as well as confidence that simultaneously allow flexibility within your cloud environment. It is highly beneficial in ensuring that identities, devices, applications as well as information are secure. However, various steps are followed to ensure security is highly achieved.
Key takeaways

Basics of AWS EC2
Types of Firewall related to AWS
Features of AWS (IAM)
Effective functions of Amazon EC2

Article quality
I will write the article based on my own understanding to ensure that it is original and of quality.
References
N/A
Conclusion
N/A

Unfortunately, most of your key takeaways are similar to https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/concepts.html Feel free to suggest another topic. Thanks


Moved to HashSet in ScatterHighlight

New Contributors:
please review CONTRIBUTING.md
Purpose:
Uses HashSet<int> in ScatterHighlight instead of bool[] which saves some memory. https://github.com/ScottPlot/ScottPlot/issues/709#issuecomment-767032539
New Functionality:
N/A

Hey @bclehmann,
This PR suffers from some unfortunate timing - I just deprecated this plot type 😅
Point distance logic now lives in the scatter plot (as defined by IHasPoints) and fancy things like managing lists/maps/arrays of multiple highlighted points is no longer handled by the plottable (that's the a user to implement according to their needs).
I updated the WinForms, WPF, and Avalonia "highlight point under cursor" demos to use the latest strategy. I think the new system makes more sense, but I certainly welcome feedback!

Hey @bclehmann,
This PR suffers from some unfortunate timing - I just deprecated this plot type 😅
Point distance logic now lives in the scatter plot (as defined by IHasPoints) and fancy things like managing lists/maps/arrays of multiple highlighted points is no longer handled by the plottable (that's the a user to implement according to their needs).
I updated the WinForms, WPF, and Avalonia "highlight point under cursor" demos to use the latest strategy. I think the new system makes more sense, but I certainly welcome feedback!


Create a blog post in speech to text conversion

Create a blog post on speech to text conversion using Vosk and Kaldi.

not required


Strict CFLAGS break the build

This is probably really a ring bug. (Or, WontFix, if I am too weird.) Let me know if you think I should file it there, too.
I use weird CFLAGS because I am a weirdo:
~/src/rust/cargo-supply-chain % echo $CFLAGS
-Weverything -Werror -O0 -std=c11

I'm on macOS with clang:
~/src/rust/cargo-supply-chain % clang --version
Apple clang version 12.0.0 (clang-1<IP_ADDRESS>)
Target: x86_64-apple-darwin20.3.0
Thread model: posix
InstalledDir: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin

I leave them in my standard shell environment just to enjoy the explosions. Here's how ring explodes:
error: failed to run custom build command for `ring v0.16.20`

Caused by:
  process didn't exit successfully: `/Users/chris/src/rust/cargo-supply-chain/target/debug/build/ring-f05a2e2ad93a365a/build-script-build` (exit code: 101)
  --- stdout
  OPT_LEVEL = Some("0")
  TARGET = Some("x86_64-apple-darwin")
  HOST = Some("x86_64-apple-darwin")
  CC_x86_64-apple-darwin = None
  CC_x86_64_apple_darwin = None
  HOST_CC = None
  CC = Some("clang")
  CFLAGS_x86_64-apple-darwin = None
  CFLAGS_x86_64_apple_darwin = None
  HOST_CFLAGS = None
  CFLAGS = Some("-Weverything -Werror -O0 -std=c11")
  CRATE_CC_NO_DEFAULTS = None
  DEBUG = Some("true")

  --- stderr
  running "clang" "-O0" "-ffunction-sections" "-fdata-sections" "-fPIC" "-g" "-fno-omit-frame-pointer" "--target=x86_64-apple-darwin" "-Weverything" "-Werror" "-O0" "-std=c11" "-I" "include" "-pedantic" "-pedantic-errors" "-Wall" "-Wextra" "-Wcast-align" "-Wcast-qual" "-Wconversion" "-Wenum-compare" "-Wfloat-equal" "-Wformat=2" "-Winline" "-Winvalid-pch" "-Wmissing-field-initializers" "-Wmissing-include-dirs" "-Wredundant-decls" "-Wshadow" "-Wsign-compare" "-Wsign-conversion" "-Wundef" "-Wuninitialized" "-Wwrite-strings" "-fno-strict-aliasing" "-fvisibility=hidden" "-fstack-protector" "-gfull" "-DNDEBUG" "-c" "-o/Users/chris/src/rust/cargo-supply-chain/target/debug/build/ring-7381a064665e3f2d/out/aesni-x86_64-macosx.o" "/Users/chris/.cargo/registry/src/github.com-1ecc6299db9ec823/ring-0.16.20/pregenerated/aesni-x86_64-macosx.S"
  error: include location '/usr/local/include' is unsafe for cross-compilation [-Werror,-Wpoison-system-directories]
  <built-in>:93:9: error: macro is not used [-Werror,-Wunused-macros]
  #define __SIG_ATOMIC_MAX__<PHONE_NUMBER>
          ^
  <built-in>:343:9: error: macro is not used [-Werror,-Wunused-macros]
  #define __SSE2_MATH__ 1
          ^
  <built-in>:341:9: error: macro is not used [-Werror,-Wunused-macros]
  #define __SSE3__ 1
          ^
  <built-in>:305:9: error: macro is not used [-Werror,-Wunused-macros]
  #define __GCC_ATOMIC_CHAR16_T_LOCK_FREE 2
          ^
  <built-in>:342:9: error: macro is not used [-Werror,-Wunused-macros]
  #define __SSE2__ 1
          ^
  <built-in>:94:9: error: macro is not used [-Werror,-Wunused-macros]
  #define __CHAR16_TYPE__ unsigned short
          ^
  <built-in>:288:9: error: macro is not used [-Werror,-Wunused-macros]
  #define __UINT_FAST64_FMTX__ "llX"
          ^
  <built-in>:95:9: error: macro is not used [-Werror,-Wunused-macros]
  #define __CHAR32_TYPE__ unsigned int
          ^
  <built-in>:345:9: error: macro is not used [-Werror,-Wunused-macros]
  #define __SSE_MATH__ 1
          ^
  <built-in>:96:9: error: macro is not used [-Werror,-Wunused-macros]
  #define __UINTMAX_WIDTH__ 64
          ^
  <built-in>:347:9: error: macro is not used [-Werror,-Wunused-macros]
  #define __GCC_HAVE_SYNC_COMPARE_AND_SWAP_1 1
          ^
  <built-in>:97:9: error: macro is not used [-Werror,-Wunused-macros]
  #define __UINTPTR_TYPE__ long unsigned int
          ^
  <built-in>:98:9: error: macro is not used [-Werror,-Wunused-macros]
  #define __UINTPTR_FMTo__ "lo"
          ^
  <built-in>:99:9: error: macro is not used [-Werror,-Wunused-macros]
  #define __UINTPTR_FMTu__ "lu"
          ^
  <built-in>:346:9: error: macro is not used [-Werror,-Wunused-macros]
  #define __MMX__ 1
          ^
  <built-in>:344:9: error: macro is not used [-Werror,-Wunused-macros]
  #define __SSE__ 1
          ^
  <built-in>:83:9: error: macro is not used [-Werror,-Wunused-macros]
  #define __SIZE_FMTo__ "lo"
          ^
  <built-in>:348:9: error: macro is not used [-Werror,-Wunused-macros]
  #define __GCC_HAVE_SYNC_COMPARE_AND_SWAP_2 1
          ^
  fatal error: too many errors emitted, stopping now [-ferror-limit=]
  20 errors generated.
  thread 'main' panicked at 'execution failed', /Users/chris/.cargo/registry/src/github.com-1ecc6299db9ec823/ring-0.16.20/build.rs:656:9
  note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
warning: build failed, waiting for other jobs to finish...
error: build failed

I should add that, yes, unset CFLAGS causes the build to work just fine.

It's probably not even a ring issue, really.

Thanks for the report! This is not directly actionable for cargo supply-chain, but ring might be interested in the report.
I see that it includes no AVX, so I suspect this is due to the way ring selects an optimized implementation. Then again, maybe there is a bit of dead code after all!


Error when booting Into usb

I get error when booting from usb on hp probook 6470b. I booted from usb and clicked on the option to boot from macOS Big Sur
I changed settings in bios
Bios version is f.74

This is a Impossible
This is A SPAME issue will be delete


No  "duration" detected in some WMV files

Here is the sample:
data1.wmv.txt
I guess that if the duration is not indicated in the Shorten, Atrac, ADTS or MLP formats, it's because there are raw streams not fully parsed, but in this WMV case that's look weird to me.
So perhaps this file is only malformed or corrupt ? In that case this issue is invalid and should be closed.

The "play duration" is set to 0, so another method should be used for computing play duration (e.g. check the last packet duration?).
Note that e.g. VLC shows no duration too.
As WMV is nearly dead, I don't see any interest in working on a workaround, I close the issue but anyone does not hesitate to open again the issue if there is any interested in it.

@Sami32 please use https://github.com/MediaArea/MediaInfoLib/issues about file detection, this project ("MediaInfo") is about the interfaces (GUI, CLI, DLL...)

@JeromeMartinez OK. Thank you for taking time to explain :+1:
Sorry for the inconvenience 8/

@Sami32 no problem, knowing the limits and having sample files is good, keep doing it even if I close some (note: I close because I don't see myself interest, but as it is reference anyone could say he is interested in removing this limitation).


Fixing AD login if there is a infinity loop in memberOf groups.

This code changes the way the code will interact with AD. It should fix teh issue #1011

This is affecting me also, can we please merge this ?

I sent a new PR to fix the conflict. #1118


Fix template search

Fixes #520

@blueorangutan package

@rhtyd @ravening Please reveiw

Packaging result: :heavy_check_mark:centos :heavy_check_mark:debian :heavy_check_mark:archive.
QA: http://primate-qa.cloudstack.cloud:8080/client/pr/543 (JID-2349)

@blueorangutan package

@davidjumani a Jenkins job has been kicked to build primate packages. I'll keep you posted as I make progress.

Packaging result: :heavy_check_mark:centos :heavy_check_mark:debian :heavy_check_mark:archive.
QA: http://primate-qa.cloudstack.cloud:8080/client/pr/543 (JID-2383)

@rhtyd @ravening Please review. Moved the search to DeployVM itself to simplify things and for consistancy

@davidjumani I can no longer see "Centos template" while deploying vm while in old ui, I can still see it
"isdynamicallyscalable": false,
      "isextractable": true,
      "isfeatured": true,
      "ispublic": true,
      "isready": true,
      "name": "CentOS 5.5(64-bit) no GUI (KVM)",
      "ostypeid": "90004bff-c29e-11ea-a58c-06c95c00338b",
      "ostypename": "CentOS 5.5 (64-bit)",

Old ui

New ui

@ravening That's proabably because it's not available in the selected zone. Could you check it via cmk by passing the zoneid ?

@davidjumani yes it belongs to the same zone
(local) mgt01 > list templates templatefilter=featured filter=name zoneid=576f-d7d-44d-8eeb-be71712
{
  "count": 3,
  "template": [
    {
      "name": "Ubuntu 16.04 x64 5G 20200617T1316"
    },
    {
      "name": "ubuntu1"
    },
    {
      "name": "CentOS 5.5(64-bit) no GUI (KVM)"
    }
  ]
}

@ravening Could you check the api call being made?
This is what I'm getting
page=1
pageSize=10
zoneid=04ccc336-d730-42fe-8ff6-5ae36e141e81
templatefilter=featured
command=listTemplates
response=json

And able to see the default CentOS template

@ravening Could you check the api call being made?
This is what I'm getting
page=1
pageSize=10
zoneid=04ccc336-d730-42fe-8ff6-5ae36e141e81
templatefilter=featured
command=listTemplates
response=json

And able to see the default CentOS template

Yes i can see those parameters but cant see the centos template in ui

@blueorangutan package

@davidjumani a Jenkins job has been kicked to build primate packages. I'll keep you posted as I make progress.

Packaging result: :heavy_check_mark:centos :heavy_check_mark:debian :heavy_check_mark:archive.
QA: http://primate-qa.cloudstack.cloud:8080/client/pr/543 (JID-2529)

@blueorangutan package

Yes i can see those parameters but cant see the centos template in ui

@ravening Is the api query returning the right response? Since no keyword is being passed to it. Or perhaps you're within a project

Yes i can see those parameters but cant see the centos template in ui

@ravening Is the api query returning the right response? Since no keyword is being passed to it. Or perhaps you're within a project

I was logged in as root admin but if you are seeing centos then its fine


Enable verbosity from the server tags

For now, we can enable initrd debug by setting up a tag with INITRD_DEBUG=1


Add support for Clearlinux OS

Clearlinux OS provides a fast working linux distribution (https://www.phoronix.com/scan.php?page=search&q=Clear Linux). Please help to add Clearlinux to be supported to build ROCm.

Hi @namm2
Thanks for reaching out.
We are not officially supporting Clearlinux OS.
Thank you.


Replace a text-based expectation on phylogeny with an reference-based expectation as per Model 2.0

Currently, we use a term called expected_phyloreference_named to store the label of the phyloreference that a particular node is expected to match. However, we should instead create an expected_phyloreference property that allows us to link to the phyloreference by identifier rather than by label.

This should be straightforward to implement once https://github.com/phyloref/phyx.js/issues/37 has been implemented.

As of PR #230, Klados allows users to change the expected resolution of a phyloreference with a drop-down menu, using the expectedResolution property. It references the phylogeny with a local reference (e.g. #phylogeny0) rather than a text-based expectation.


Signing fails via GPG

I started ST3 with the GitSavvy (v2.13.0) plug-in via cmder to run a signed git commit. Signing fails with the following error in the GitSavvy output panel:
`%CMDER_ROOT%\vendor\git-for-windows\cmd\git.EXE commit -q -F -` failed with following output:

gpg: cannot open tty `no tty': No such file or directory
error: gpg failed to sign the data
fatal: failed to write commit object

Using cmder directly works as intended.
I wasn't prompted for a password by GitSavvy while attempting to sign.

You may need to cache your github password, check https://help.github.com/articles/caching-your-github-password-in-git/#platform-windows
I will let the more experienced Windows users answer this issue.

Can make a log with GitSavvy
https://github.com/divmain/GitSavvy/blob/master/docs/debug.md

I think this isn't related to github, issue #36 discusses signing commits, and how to enable them via a global config.
As far as I know, there isn't a clean way to sign with a password protected key.

@asfaltboy as I implied above by mention the lack of a prompt for the password, the key is protected.

@randy3k @asfaltboy I'm not intending to cache my password.

I'm sorry @jrappen , but password prompts are not currently supported.

Would it be possible to prompt via an input panel within Sublime similar to when renaming files and then passing the input through?

@asfaltboy My ~/.gitconfig has core.editor = subl, user.signingkey = ... and commit.gpgsign = true. Usually I would do a git commit from cmder, which opens ST3 to write the message. Closing the window there, makes cmder continue to prompt me for the password to gpg sign my git commit and if successful do my signed commit.

We can ask for the passphrase, but the issue is communicating that input to GPG securely. As a workaround, I suggest leaving sublime out of the passphrase step, and use pinentry.exe (installed with gpg4win) for this step.
Start by making pinentry dialog for passphrase work from command line commit (e.g git commit -m "test", without sublime text) with following this guide:  https://jamesmckay.net/2016/02/signing-git-commits-with-gpg-on-windows/ . Once this works well, I imagine you should be able to enter your commit message in ST and get the pinentry dialog for the passphrase.

As @asfaltboy mentioned, it is prohibitively difficult to support dynamic interactions with Git from within Sublime.  This is largely due to the way we interact with subprocesses in Python and the constraints of Sublime itself.  Since a work-around has been provided, I'm going to mark this as closed.  Sorry if this isn't the answer you were looking for!


[Bug]: Hubspot is broken wrt POST commands

Is there an existing issue for this?

[X] I have searched the existing issues

Description
When the user tries access Hubspot command after saving the datasource it is observed that the other fields wrt to the command selected does not appear
Steps To Reproduce

Add Hubspot Datasource
Click on Add new Datasource
Select "Create Table"
and observe

Public Sample App
No response
Issue video log
No response
Version
Release

hi @prapullac tested the steps above in release, this is working for me. Would you please check again?
https://images.zenhubusercontent.com/275993885/a1efca23-ab03-409f-9ddf-2f32e27d6576/screen_recording_2022_12_05_at_17_25_09.mov


Issue with forwarding HTTP body after reading in proxy request filter method

I want to proxy an HTTP API and add simple authentication for client. The authentication requires computing an MD5 hash of the POST request's body and URL parameters, then placing this hash in the auth field of the request header. In the request_filter method of the ProxyHttp, I plan to verify the MD5 hash, and only if it matches will the request be forwarded to the upstream server.
However, I found that after using the session.read_request_body method to obtain the body content in the request_filter method, the code stops forwarding the body after outputting the log "pingora_proxy::proxy_h1: Sending header to upstream RequestHeader xxx". How can I get a copy of the body in the request_filter method without affecting the forwarding to the upstream server?

I think the requestBody needs to be read in async fn request_body_filter(  method.

I think the requestBody needs to be read in async fn request_body_filter( method. https://github.com/cloudflare/pingora/blob/main/pingora-proxy/src/proxy_trait.rs#L94

If the authentication is handled in request_body_filter, it won't be possible to block the request to the downstream.

I think the requestBody needs to be read in async fn request_body_filter( method. https://github.com/cloudflare/pingora/blob/main/pingora-proxy/src/proxy_trait.rs#L94

If the validation logic is handled in request_body_filter, the client's request headers will be forwarded to the upstream first, and then the connection will be aborted. This doesn't align with the logic of authenticating first.

Hmm, I retried to buffer the request into the CTX, and then set back in the request_body_filter, but it does not work anymore.

Do you have an example for us to look at? I'm guessing this not working has to do with read_request_body consuming the body.

#[derive(Debug)]
pub struct MyGate;

pub struct RequestCtx {
    meter: Meter,
    timers: HashMap<&'static str, SystemTime>,
    body: Option<Bytes>,
}

#[async_trait]
impl ProxyHttp for MyGate {
    type CTX = RequestCtx;
    fn new_ctx(&self) -> Self::CTX {
        RequestCtx {
            meter: global::meter("test_rust_application"),
            timers: HashMap::new(),
            body: None,
        }
    }

    #[tracing::instrument(
        skip(session,ctx),
        fields(
            req_uri=%session.req_header().uri.path_and_query().unwrap()
        ))]
    async fn request_filter(&self, session: &mut Session, ctx: &mut Self::CTX) -> Result<bool> {
        ctx.timers.insert("request_time_ms", SystemTime::now());
        let body = session.read_request_body().await.unwrap();
        session.enable_retry_buffering();
        if session.req_header().uri.path() == "/graphql" {
            error!("before in the header");
            if let Some(x) = body {
                match session
                    .req_header()
                    .headers
                    .get("content-type")
                    .map(|v| v.to_str())
                {
                    Some(Ok("application/json")) => {
                        error!("before in the json header");
                        let body_json: RequestBody = serde_json::from_slice(&x).unwrap();
                        if let Some(b) = body_json.argument {
                            let body_bytes = Bytes::from(b);
                            ctx.body.replace(body_bytes.clone());
                        }
                    }
                    _ => {}
                }
            }
        }
        if session.req_header().uri.path().starts_with("/login")
            && !check_login(session.req_header())
        {
            let resp = error_resp::gen_error_response(403).clone();

            session
                .write_response_header(Box::new(resp), false)
                .await
                .unwrap_or_else(|e| {
                    error!("failed to send error response to downstream: {e}");
                });
            // true: early return as the response is already written
            // self.rejected_metric.inc();
            return Ok(true);
        }
        Ok(false)
    }

    async fn request_body_filter(
        &self,
        _session: &mut Session,
        body: &mut Option<Bytes>,
        _end_of_stream: bool,
        ctx: &mut Self::CTX,
    ) -> Result<()>
    where
        Self::CTX: Send + Sync,
    {
        error!("before in the body");
        if let Some(x) = ctx.body.clone() {
            error!("test in the body");
            body.replace(x);
        }
        Ok(())
    }

    async fn upstream_request_filter(
        &self,
        session: &mut Session,
        upstream_request: &mut RequestHeader,
        ctx: &mut Self::CTX,
    ) -> Result<()> {
        upstream_request.remove_header("content-length");
        upstream_request.insert_header("content-length", "262");
        upstream_request.remove_header("up-stream");
        upstream_request.remove_header("up-stream-scheme");
        upstream_request.remove_header("up-stream-port");
        let path = session.req_header().uri.path();
        if path.starts_with("/view") {
            let upstream_path = if path == "/view" {
                path.strip_prefix("/view").unwrap_or("").to_string()
            } else {
                String::from("/") + path.strip_prefix("/view").unwrap_or("")
            };
            upstream_request.set_uri(
                Uri::builder()
                    .authority(
                        upstream_request
                            .uri
                            .authority()
                            .unwrap_or(&Authority::from_static("tokio.rs"))
                            .clone(),
                    )
                    .scheme(session.req_header().uri.scheme_str().unwrap_or("http"))
                    .path_and_query(
                        upstream_path + (session.req_header().uri.query().unwrap_or("")),
                    )
                    .build()
                    .unwrap(),
            );
        }
        error!("{:?}", ctx.body);
        Ok(())
    }

    async fn upstream_peer(
        &self,
        session: &mut Session,
        _ctx: &mut Self::CTX,
    ) -> Result<Box<HttpPeer>> {
        let mut router_addr = match session.req_header().headers.get("up-stream") {
            None => "<IP_ADDRESS>".to_string(),
            Some(header_value) => header_value.to_str().unwrap_or_default().to_string(),
        };

        if session.req_header().uri.path() == "/view" {
            router_addr = "localhost".to_string();
        }

        let up_stream_scheme = match session.req_header().headers.get("up-stream-scheme") {
            None => false,
            Some(header_value) => {
                "Https".eq_ignore_ascii_case(header_value.to_str().unwrap_or_default())
            }
        };

        let mut router_port = match session.req_header().headers.get("up-stream-port") {
            None => {
                if up_stream_scheme {
                    443
                } else {
                    80
                }
            }
            Some(header_value) => header_value.to_str().unwrap().parse().unwrap(),
        };

        if session.req_header().uri.path() == "/view" {
            router_port = 3000;
        }

        debug!("connecting to {router_addr} {router_port}");

        let peer = Box::new(HttpPeer::new(
            (router_addr, router_port),
            up_stream_scheme,
            "one.one.one.one".to_string(),
        ));
        Ok(peer)
    }

    async fn response_filter(
        &self,
        _session: &mut Session,
        upstream_response: &mut ResponseHeader,
        _ctx: &mut Self::CTX,
    ) -> Result<()>
    where
        Self::CTX: Send + Sync,
    {
        // replace existing header if any
        upstream_response
            .insert_header("Server", "PingoraGateway")
            .unwrap();
        // because we don't support h3
        upstream_response.remove_header("alt-svc");
        Ok(())
    }

    async fn logging(
        &self,
        session: &mut Session,
        _e: Option<&pingora_core::Error>,
        ctx: &mut Self::CTX,
    ) {
        let tracing = global::tracer("request_summary");
        let request_time_ms = ctx
            .timers
            .remove("request_time_ms")
            .unwrap()
            .elapsed()
            .unwrap()
            .as_millis();
        tracing.in_span("request_summary", |cx| {
            tracing::info!(
                monotonic_counter.request_summary = 1_u64,
                key_1 = "request_time",
                key_2 = 10,
                "{} response time: {request_time_ms}",
                self.request_summary(session, ctx)
            );
        });
    }
}

the request will not enter request_body_filter.

I have a simpler like this,
execute curl -X POST -H "auth: 37dec66febcdb7e0e80a32565d7a0ac6662d079f4db21a4609e3631176cbd680" -H "uid: uid1" "http://<IP_ADDRESS>:8080/" -d "abcd" -v, the upstream can receive the request headers but not the body.  And the curl downstream is stuck.
use async_trait::async_trait;
use bytes::Bytes;
use pingora::{
    prelude::HttpPeer, proxy::{http_proxy_service, ProxyHttp, Session}, server::Server, Error, ErrorType, Result
};
use sha2::{Digest, Sha256};
use tracing::info;
use tracing_subscriber::{fmt, EnvFilter};
use tracing_subscriber::prelude::*;

struct ApiProxy;

#[async_trait]
impl ProxyHttp for ApiProxy {
    type CTX = ();

    fn new_ctx(&self) -> Self::CTX {}

    async fn request_filter(&self, session: &mut Session, _ctx: &mut Self::CTX) -> Result<bool> {
        let req_header = session.req_header();

        if let Some(auth) = req_header.headers.get("auth") {
            let auth = auth.to_str().unwrap().to_owned();
            let uid = req_header
                .headers
                .get("uid")
                .map_or("", |v| v.to_str().unwrap())
                .to_string();
            let body = session.read_request_body().await?;
            let sum_payload = match body {
                Some(body) => {
                    let mut hash = Sha256::new();
                    hash.update(uid.as_bytes());
                    hash.update("USER_SECRET".as_bytes());
                    hash.update(body.slice(..).as_ref());
                    &format!("{:X}", hash.finalize())
                }
                None => "",
            };
            if sum_payload.to_lowercase() == auth {
                return Ok(true);
            } else {
                return Err(pingora::Error::new(ErrorType::Custom("auth err")));
            }
        }
        Err(pingora::Error::new(ErrorType::Custom("auth err")))
    }

    async fn upstream_peer(
        &self,
        _session: &mut Session,
        _ctx: &mut Self::CTX,
    ) -> Result<Box<HttpPeer>> {
        let peer = HttpPeer::new(("<IP_ADDRESS>", 80), false, "".to_owned());

        Ok(Box::new(peer))
    }

    async fn request_body_filter(
        &self,
        _session: &mut Session,
        _body: &mut Option<Bytes>,
        _end_of_stream: bool,
        _ctx: &mut Self::CTX,
    ) -> Result<()>
    where
        Self::CTX: Send + Sync,
    {
        info!("not working here");
        Ok(())
    }
}

fn main() {
    let fmt_layer = fmt::layer().with_target(false);
    let filter_layer = tracing_subscriber::EnvFilter::try_from_default_env()
        .or_else(|_| EnvFilter::try_new("debug"))
        .unwrap();

    tracing_subscriber::registry()
        .with(filter_layer)
        .with(fmt_layer)
        .init();

    let mut srv = Server::new(None).unwrap();
    srv.bootstrap();
    let mut pxy = http_proxy_service(&srv.configuration, ApiProxy {});
    pxy.add_tcp("<IP_ADDRESS>:8080");
    srv.add_service(pxy);

    srv.run_forever();
}

I see. Fundamentally it seems like you need to tell pingora-proxy to run the request body filter, even if the downstream session's body is no longer present from pingora-proxy's perspective. This does seem hard to do with what we have today, and is worth its own feature.

I face the same problem. Btw, it's worth noting that read_request_body doesn't read the entire body but instead only a chunk of it and the docs "Read the request body. Ok(None) if no (more) body to read" don't clarify this
async fn upstream_request_filter(
    &self,
    session: &mut Session,
    upstream_request: &mut RequestHeader,
    ctx: &mut Self::CTX,
) -> Result<()>
where
    Self::CTX: Send + Sync,
{
    while let Some(x) = session.read_request_body().await? {
        tracing::error!("got body {}", x.len());
        ctx.body_buf.put_slice(&x);
    }
    Ok(())
}

gives:
2024-08-30T07:45:39.025242Z ERROR g_hole::gateway: got body 2690
2024-08-30T07:45:39.025286Z ERROR g_hole::gateway: got body 61440
2024-08-30T07:45:39.025324Z ERROR g_hole::gateway: got body 65536
2024-08-30T07:45:39.025365Z ERROR g_hole::gateway: got body 65536
2024-08-30T07:45:39.025407Z ERROR g_hole::gateway: got body 65536
2024-08-30T07:45:39.025467Z ERROR g_hole::gateway: got body 65536
2024-08-30T07:45:39.025492Z ERROR g_hole::gateway: got body 65536
2024-08-30T07:45:39.025516Z ERROR g_hole::gateway: got body 65536
2024-08-30T07:45:39.025538Z ERROR g_hole::gateway: got body 65536
2024-08-30T07:45:39.025640Z ERROR g_hole::gateway: got body 65536
2024-08-30T07:45:39.025678Z ERROR g_hole::gateway: got body 27092

@drcaramelsyrup: is there any workaround now to achieve reading body before sending headers to the upstream?
Now, reading body in upstream_request_filter results in:
2024-08-30T07:45:39.026536Z ERROR pingora_proxy: Fail to proxy: Upstream PrematureBodyEnd context: Peer: addr: <IP_ADDRESS>:8080, scheme: HTTP,sni: <IP_ADDRESS>, cause:  context: Content-length: 615510 bytes written: 0, status: 502, tries: 1, retry: false, POST /test/200, Host: localhost:6191
even before it reaches request_body_filter

I was facing the same situation, but found that the following code can successfully process the request while getting the request body.
If the problem has been resolved by an update of pingora, please disregard it.
// ver. 3.0.0
async fn request_filter(&self, session: &mut Session, ctx: &mut Self::CTX) -> Result<bool>
    where
        Self::CTX: Send + Sync
    {
       // any process...

        session.enable_retry_buffering();
        session.read_body_or_idle(false).await.unwrap().unwrap();
        let request_body = session.get_retry_buffer();

        // any process...
     }

@maclo2012 Yeah, this looks like a solution, but the crucial thing is to not keep the whole body in RAM at once, but instead to be able to write it to a temporary file (say, if the requests are quite large)

I was facing the same situation, but found that the following code can successfully process the request while getting the request body.
If the problem has been resolved by an update of pingora, please disregard it.
// ver. 0.3.0
async fn request_filter(&self, session: &mut Session, ctx: &mut Self::CTX) -> Result<bool>
    where
        Self::CTX: Send + Sync
    {
       // any process...

        session.enable_retry_buffering();
        session.read_body_or_idle(false).await.unwrap().unwrap();
        let request_body = session.get_retry_buffer();

        // any process...
     }

That actually fixes my issue too.

@maclo2012  If the request body content is small, there's no problem, but what should be done if the request body content is large?  This seems to only read a small portion of the request body. It needs to read in a loop, but after looping, it cannot call session.get_retry_buffer to return the content.

This does seem to be an issue, but would it be feasible to increase the retry buffer to a sufficiently large value?
In my case, the body is relatively small than 64K.

We added a feature flag to use a spoolfile as a retry buffer here: https://github.com/lindenbaum/pingora/tree/feature/retry_buffer_tempfile
We didn't open a PR, because

Its probably a niche use case
We are handling AI requests to LLMs, which will take long anyway, so we didn't mind totally destroying pingoras performance by waiting for the writes to disk to happen before the proxy event loop continues and waiting for the disk load to happen before entering the proxy event loop. One could mitigate this by using an actor and a task queue for the writes and altering the event loop not to send the whole retry buffer at the top, but reading it piece by piece and sending the individual pieces before reading the next one, giving upstream the possibility to send a piece of the response before we event sent the whole retry_buffer (a bit like it is done with caching, but way less complex).

So if you want to try it out, keep that in mind.


Failed to enable IPFS CLI from IPFS Desktop on Debian/Ubuntu based OS

I was trying to enable the ipfs CLI binary after installing the IPFS Desktop app.
The app was installed via snap, on a Debian/Ubuntu based machine.
My machine specs follow:
elementary OS 5.1.3 Hera
Built on Ubuntu 18.04.3 LTS
Linux 4.15.0-96-generic
GTK 3.22.30
The log  states the error ocurred when trying to find pkexec and/or kdesudo. I checked my local system and pkexec is present at /usr/bin (which is in my PATH).
I suspect this may be a permission issue since I am running the snap package of IPFS Desktop and snap's are sandboxed. I'm not too familiar with snap development however, so I could be wrong.
Specifications

OS: linux (elementaryOS
IPFS Desktop Version: 0.10.4
Electron Version: 7.1.14
Chrome Version: 78.0.3904.130

Error
Error: Unable to find pkexec or kdesudo.
    at test (/snap/ipfs-desktop/9/resources/app.asar/node_modules/sudo-prompt/index.js:205:18)
    at /snap/ipfs-desktop/9/resources/app.asar/node_modules/sudo-prompt/index.js:212:47
    at FSReqCallback.oncomplete (fs.js:165:21)

I was prompted to create this issue from the Desktop app. I've just checked other open issues and found that this seems to be a duplicate of #1386 .

Closing as duplicate of #1311.
And you're right, it probably is related to Snap and it's sandboxing. It's also weird that you're not getting a prompt. I'll check on this.


[Automation] Generate Fluent Lite from extendedlocation#

[Automation] Generate Fluent Lite from extendedlocation#

/azp run prepare-pipelines

/azp run java - extendedlocation - ci


Restoring a protocol snapshot without TTL

I've added an option to the ProtocolCallback which skips setting the TTL on a key, even if it had an expiry set in the original .rdb file.
The reason for this is to copy a snapshot of a production redis instance where many keys are set to expire into a test redis instance where the keys can be safely examined.
Example usage, copying from a dump of a production database to a private redis running in Docker:
 ~/Projects/redis-rdb-tools $ rdb -c snapshot prod2.rdb | redis-cli -h <IP_ADDRESS> --pipe

@Sekenre thanks for that patch, seems useful.
The only thing i'm not sure about is the interface, i think i would rather add an optional argument such as --omit-expire rather than a new command.
another idea that can be useful is to offset all expire times, so something like --amend-expire=30000 (in seconds). would you like to try have a go at that?

I will write a proper test case shortly.

@Sekenre thanks a lot. all looks good.
one last petty request, if you don't mind squashing this into one commit?
Or alternatively i can squash when meting (and write a new commit comment)

@Sekenre thanks a lot for the contribution and adding all the adjustments i asked for.

@Sekenre FYI, just released 0.1.15 with your changes.
thanks again.


Add Github-Link Everywhere.

At least in the Play Store (and I think in the app itself), there's no link to 8Vim's GitHub-page. (https://github.com/flide/8VIM)
I suggest fixing that, so that more people know where to post their ideas and/or bugs.

Also mention that it's an open-source application. As far as I can see, in the Play Store, it's never mentioned.
(On F-Droid it's obvious that it's open-source, so no need to add this there)

Done! The changes on the google play-store should reflect it in a short while.


Make use of RBAC v1

This PR changes the RBAC API version to the current v1 (which is GA since I think 1.8: https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG-1.8.md#auth).

Hi @johscheuer. Thanks for your PR.
I'm waiting for a istio member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.
Once the patch is verified, the new status will be reflected by the ok-to-test label.
I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

/ok-to-test
sorry the ci got broken by a dependency, will merge this in when it's fixed , someone is working on it

/retest

/cherrypick release-1.3

@rlenglet: new pull request created: #155

In response to this:

/cherrypick release-1.3

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.


Server not stopping

When tpaplusplus is installed to my server and I send the stop command it won't get past [Server thread/INFO]: JDA shut-down gracefully when I disable the mod it stops just fine, no issue.
Server version: Fabric 1.20.1
If needed, I can give more info.

This issue has been fixed as of commit #d18084f; A release with this patch implemented is on it's way to Modrinth / Curseforge, thank you for reporting this issue!


Added the NotePlan extension

Please add this new PopClip extension for the great application NotePlan. Thanks.

Closing as won't merge. Apologies for long delay.


Endless loop using the move command

Hello,
I'm trying to code a webmail using your library but I encounter an issue when I try to implement the "move" (it seems that I've the same issue with marking a message read/unread).

At the beginning the message status is 2 (READY)

I'm using the command suggested in the readme:
JMAP.mail.move(messages, JMAP.mail.systemMailboxIds.get('trash'), null);

The message deletion is well sent.

The query looks like:
0: ["setMessages",…]   ## The expected command is there
1: ["getMailboxes",…]
2: ["getThreads", {ids: ["bcb97880f9f80d9f90c38e976a5045fb4c2830da"], fetchMessages: true,…}, "2"]

and the response (from the jmap-perl proxy) looks like:
0: ["messagesSet",…]
1: ["mailboxes", {accountId: "c3147952-4d8c-11e5-9396-0242ac110067",…}, "1"]
2: ["threads", {accountId: "c3147952-4d8c-11e5-9396-0242ac110067",…}, "2"]
3: ["messages", {accountId: "c3147952-4d8c-11e5-9396-0242ac110067",…}, "2"]

As there is a "messages" section into, the callback "JMAP.mail.handle( Message, ...#messages)" defined in "jmap-js/source/mail/Message.js" is applied. Then it tells to the store that the message has been updated with:
this.get( 'store' ).sourceDidFetchPartialRecords( Message, updates );

in this method, the lib has a status of 50 for this message so it wants to fetch its data again:
     // If OBSOLETE, the record may have changed since the fetch was
     // initiated. Since we don't want to overwrite any preemptive
     // changes, ignore this data and fetch it again.
     // Similarly if the record is committing, we don't know for sure
     // what state the update was applied on top of, so fetch again
     // to be sure.
     if ( status & (COMMITTING|OBSOLETE) ) {
         this.setStatus( storeKey, status & ~LOADING );
         this.fetchData( storeKey );
         continue;
     }

So, a new "getMessages" is sent to the JMAP server. The new query looks like:
[["getMessages", {ids: ["88dcb8982a31ac3a0ae33513c6a15e4b36934a28"],…}, "0"]]

and the response:
["messages", {accountId: "c3147952-4d8c-11e5-9396-0242ac110067",…}, "0"]

So the code goes back to the "step 4" when the response is received. It makes an endless loop ..

Do you understand what I'm doing wrong ? Any clue ?
Thanks!

Status 50 means it's still got the COMMITTING bit (32) set, so it sounds like something went wrong with the messagesSet response. Since the COMMITTING bit never gets cleared, it gets stuck in the loop as you described. As you omitted the content of the messagesSet response, I'm not quite sure what was going on (and it would certainly be easier to debug with a live copy of the code anyway). It could be a bogus response from the JMAP-Perl library.
Have you tried doing the move on the hosted version of the demo webmail? This uses the same library underneath – have a look at the data exchanges there and hopefully the difference will become apparent.

Hello @neilj, thanks for your reply!
I've compared exchanges between my app and your demo app, there are some differences but I can't find out what's the problematic thing.
Divergencies that I've noted:

I've a "getThreads" section beside "setMessages" in the performed request, there is not in the demo

--> so there are "threads" and "messages" properties in the response given by the jmap-proxy

After getting the "setMessages" response, in both apps the jmap-js lib is performing a second request.

For the demo app, there is a "getMessageListUpdates" section which is not send with my app
For my app, there is a "getMessages" section which is not send with your demo app

--> I'm getting a "messages" section in my response, a "messageListUpdates" in the demo response.
I guess that this difference comes because I'm observing this email for changes with the code below. Can it be the issue ?
  var record = JMAP.store.getRecord(JMAP.Message, emailId);
  record.addObserverForKey('*', request, 'callback');

The jmap-js starts its endless loop at this point for my app by doing the "getMessages" request.
In the demo, I guess that the COMMIT is well done so it doesn't do anymore request.

As I'm not fluent as you about JMAP talking, I'm joining requests and responses that I've captured.
May be that you'll find out the annoying thing that makes this endless loop?

with my custom app: https://gist.github.com/thomas-hilaire/0f5143db28f10df843e4
with the demo app: https://gist.github.com/thomas-hilaire/dcd0306e6860364fe149

Thanks again, regards,
Thomas

Thanks for the logs; the problem is clearly here:
[
  "messagesSet",
  {
     "accountId" : "97d3e0d6-517c-11e5-86a6-0242ac11006f",
     "created" : {},
     "deleted" : "",
     "newState" : "29",
     "notCreated" : {},
     "notDeleted" : null,
     "notUpdated" : null,
     "oldState" : null,
     "updated" : ""
  },
  "0"
]

This response from the server is bogus: updated and destroyed (deleted is even the wrong name!) should be objects, not strings, and the id of the message you tried to modify MUST be in either updated or notUpdated. So since it never gets a response from the server, the library presumes it's still committing, so gets into this loop where it's waiting for the commit to finish.
Are you using the hosted version of the proxy, at https://proxy.jmap.io?  If the latter, make sure you have the latest version of the proxy code.
The good news is it doesn't look like there's anything wrong with your code! The bad news is this is probably a bug in the JMAP Proxy – please file a bug there and hopefully @brong will take a look and fix it.


Test the new Software Page behind Dirsaptor

This might be a good issue so that @BastianGrahm and @Dumpling97 can collaborate to ensure a smooth handover for the discourse things and related stuff.
Test all the workflows of the new software page and write down things that do not work and things that can be improved.
For instance things as (I only found them during fast clicking around, there might be other such issues):

At this page I noticed that the upload submissions are empty after I have clicked on the Docker images icon.
Running a Docker software does not work anymore (The error in the network tools is "Blocked: NS_BINDING_ABORTED". So it looks like the request is not send to the server)
Running Jobs are not displayed.
Write also down usability issues: Are there situations where you do not know what to do? What would help you? For instance, the upload docker images window is currently missing the "personalized documentation" that I wrote and I guess without this it is not userfriendly.

I just clicked around a little and after heading to the "Virtual Machine Submission" none of the button clicks do anything anymore. So I can't change to the other tabs, also adding a software or starting the VM doesn't trigger any view changes. Anyway, when I reload the page the actions that I started with my clicks worked. Softwares got created and the machine is starting.

When uploading a file in the "Upload Submission" tab and it works fine the response is

Maybe a little misleading.

I just clicked around a little and after heading to the "Virtual Machine Submission" none of the button clicks do anything anymore. So I can't change to the other tabs, also adding a software or starting the VM doesn't trigger any view changes. Anyway, when I reload the page the actions that I started with my clicks worked. Softwares got created and the machine is starting.
EDIT: Actually the "Connection Info" button does work in this scenario.

So this is due to an error thrown here https://github.com/tira-io/tira/blob/c29be3480418fdac7cc833bb1ad12bb03f960d6d/application/src/tira/frontend/src/components/vmsubmissionpanel.vue#L350-L362
The for loop loops over the indices of the softwares but then ends on a variable called _super. It obviously tries to access this.software[_super].software but this doesn't exist so this breaks everything.
This is probably connected to this: https://github.com/emberjs/ember.js/issues/19289

Most of this was fixed by https://github.com/tira-io/tira/commit/9fa2bda5e0160ca48fd286a99e7a171726133ff9.
There still are 2 bugs we could find immediately

When adding a new docker container the page has to be reloaded for it to be shown
when running a docker container the "running processes panel" is not directly shown but only after reloading

when adding more processes to it they are covered by the poll that happens every 10s so they are then loaded dynamically

Adding a new docker container now instantly updates the list of existing containers. Same for container deletion. For reference see 9fa2bda5e0160ca48fd286a99e7a171726133ff9.

Adding a software now works as intended via dfc2f6b842e8178e828f8d0c2d47208f7d88b7de

Everything here should be solved with the newest version. Closing this for now.


Add see also section

Closes #22078
@SogoGolf Can you look whether this fixes your issue?

I don't know that this answers any question, nor if that question was appropriate for this particular article 😄 If anything the see also should link over to https://docs.microsoft.com/dotnet/core/versions/selection

I'm going to close this PR as I don't think it's appropriate for the article. Thanks though 😺


How can I change truncation tail text?

How can I change truncation tail text like "See more"? and click that as a URL?

Hi, @veila ,
You can truncation tail text by yourself, and the remains string uses CuteAttribute to handle the tap action.


Load local tokenizer

Due to some network issues, I need to first download and load the tokenizer from local path. But the current tokenizer only supports identifier-based loading from hf. Is it possible to add a local load from path function like AutoTokenizer in transformers lib?

if you replace Tokenizer.from_pretrained with Tokenizer.from_file in the source does it work or is the tokenizer not in the right format? If it works I can add a check to see if the tokenizer name is a valid path and if so load using from_file in that case

Yeah, I'm currently using from_file and it works fine

I will add a check then. Are you passing the path to the folder or to the json file directly?

json file

Also maybe we should add option for BPE tokenizer in MinhashDedup?

Also maybe we should add option for BPE tokenizer in MinhashDedup?

you mean instead of word_tokenize?

Also maybe we should add option for BPE tokenizer in MinhashDedup?

you mean instead of word_tokenize?

Yeah, maybe we could support both by changing the function a bit?

Could we use AutoTokenizer from transformers for this? It would be much more flexible than using the raw tokenizers class.

I would really like to avoid the dependency on transformers just for this, which is a big lib. I'll make a PR with the Tokenizer.from_file change

Take a look at the linked PR


Support control flow in DAGCircuit.collect_runs

What should we add?
Transitively part of #9417.
Transpiler optimisation level 3 adds in extra passes to resynthesise all runs of 2q blocks.  Doing this involves finding the runs of blocks, which is the pass Collect2qRuns, which mostly just delegates to DAGCircuit.collect_runs (via DAGCircuit.collect_2q_runs).  This whole chain needs upgrading to be able to handle control-flow.
In particular: DAGCircuit.collect_runs needs an upgrade to its output format to allow the specification runs that are nested within control flow.  Control-flow operations that act on two qubits should not themselves be considered part of a run, although their contained block may be a run of itself.
For the new data format, there is likely some overlap here with #9423.  The new data format should be backwards compatible with the old one. That is, a circuit without control flow should produce the same output in the old and new forms of this function, since it is a user-facing method.
Fixing this should automatically upgrade the analysis transpiler passes Collect2qBlocks and Collect1qRuns (and this should be part of the test suite). This alone will not enable the transformation pass ConsolidateBlocks to read this information - for that, see #9426.

I'm finding the details of the relevant structures are slightly different than they appear in the comment above. On today's main branch, as well as that on Jan 23, 2023. The main ideas are the same. To try to get the details straight:

I find do find the pass called  Collect2qBlocks, but find no Collect2qRuns in ./qiskit or ./test. Is Collect2qRuns somewhere else?
Collect2qBlocks calls the method DAGCircuit.collect_2q_runs.
DAGCircuit.collect_2q_runs calls rustworkx.collect_bicolor_runs. The latter function implements the algorithm directly without calling to another high-level function.
Each of DAGCircuit.collect_runs and DAGCircuit.collect_1q_runs directly calls rustworkx.collect_runs, which again does all the work without calling another high-level function. The two Python methods differ mostly in the filter function that the pass to rustworkx.

A few questions:
It seems clear that we need to recognize control flow nodes in rustworkx.collect_bicolor_runs. I don't immediately see a way to do this without changing the API of this function. This is not a problem for qiskit because collect_bicolor_runs is called in only one place and it is not exposed. But it would change the API of rustworkx. And it would do it in a way that begins to look tailored to the needs of the problem at hand. An alternative is to rename a copy of  collect_bicolor_runs and modify it.
Do we need to do something similar for  rustworkx.collect_runs which is callled by DAGCircuit.collect_runs ? The code for these is currently independent of that in the previous paragraph. (The opening comment seems to imply a call chain collect_2q_runs $\rightarrow$ collect_runs. But this is not the case in the code today.)
We could simply include the control-flow nodes in the output of the rustworkx routines and then postprocess it in Python. But I guess this is unacceptably inefficient. It is better to do this work on the rust side. Even if it's possible to analyze the PyObjects on the rust side, I don't see this done in the code at present. Also, it would be qiskit-specific and would not belong in rustworkx. Instead this is always done by passing python functions to the rust side. Two options are: 1) Modify our new copy of rustworkx.collect_bicolor_runs to take three functions rather than two as arguments. Pass a boolean function that returns true on a control flow node. 2) Use only two Python callbacks, but instead of a boolean function, use a function that returns an integer. This would indicate whether the node is a gate or a control flow node or something else.
How do we indicate that a run is in a block in a control flow node? We could  wrap the run in a class. This could be created by yet another Python callback on the rust side. It would be nice to find a better way.
How will these blocks be used? That would inform the design of the data structure. For example suppose we are resynthesizing 2q gates. Is the idea to descend into blocks in control-flow and do the same. In this case, I just a wrapper as described above is sufficient. But we might want to use this data structure for other purposes as well.

If I read the OP correctly there shouldn't be any rustworkx api changes needed. I see it more of a division of work that's needed here. First we need to update the filter function argument to the rustworkx calls used in the dagcircuit api to set it to not include control flow ops. Then we need to update the pass to recurse into control flow blocks and run the dagcircuit collection methods again inside each of those and store that nested run/block info in the property set. Finally we'll then need to update the consolidate blocks pass to read whatever data format is put on the property set and consolidate the internal runs/blocks into unitaries identified inside the control flow blocks.

About collect_2q_runs delegating to collect_runs: I probably misread the functions, or I just looked at them at thought "those look very similar" then screwed up when writing the post.  The point is that the all upgrade in the same way, at least.
About Collect2qRuns: I probably typoed and meant Collect2qBlocks, or whichever way round it's meant to be.  There's like 4 passes that are Collect<X><Y>, where Y appears to be pseudorandomly selected from "blocks" and "runs".
For the rest, what Matthew said is what I meant.

thought "those look very similar"

I think the "moral" of story in the OP was clear enough.  Good that you cleared this up. I was not 100 sure of the details because occasionally strings I don't find with grep are in another repo.
Ok. So the new logic is all on the Python side.

It's likely that we finish at least a few of these issues for supporting control flow in optimization before releasing. Maybe we can do a release note for all of them (or a group of them) rather than a release note per PR.
In particular, it's time to make a release note for #10260. Or, we can make separate release notes, and consolidate them a single note looks better.
Thoughts?

@kevinhartman I think my comment is mistaken. Considering control flow ops on the DAG, the blocks of ops inside are QuantumCircuits not DAGCircuits. In particular, they are not indexed in the top-level DAG.
So in order to work with ConsolidateBlocks I have to add more structure. Furthermore, it would be nice to avoid converting repeatedly from DAG to Circuit (more than we already do)

If we merge

#10355

then this issue (#9425) is obsolete, unless I'm mistaken. It might become relevant again if we redesign in the future.

This issue is obsolete. We don't have any near term need to complete this task.


clock_en_i no longer connected

Hello,
It seems that the top level signal 'clock_en_i' no longer drives anything. Is this intentional?

Hello,
yes in the master branch, zero-riscy has been adapted to work with the more recent platforms PULP and PULPissimo which have a different event unit than PULPino, and the core now handles its clock gate by itself without relying on external units. In the PULPino branch the clock enable is still driving logic.


How to make backup?

Cannot make a backup of my configuration , so when I have to take away the machine , I cannot just load the snapshot backup from a file but I will have to do again from the beginning

Not sure what you mean with "take away the machine"? Why would you "take it away"?
I need to know more details on your OS to help you with how you can backup. Your confirmation is typically stored in /var/snap/home-assistant-snap/current
If you're just re-installing the snap (but why would you?), then you can simply make a copy of this directory, e.g with rsync.
If you're using Ubuntu Core, then you should use snap snapshots. But they're probably "gone away" if you're taking your machine away. 🤪

Not sure what you mean with "take away the machine"? Why would you "take it away"?
I need to know more details on your OS to help you with how you can backup. Your configuration is typically stored in /var/snap/home-assistant-snap/current
If you're just re-installing the snap (but why would you?), then you can simply make a copy of this directory, e.g with rsync.
If you're using Ubuntu Core, then you should use snap snapshots. But they're probably "gone away" if you're taking your machine away. 🤪

My server sometimes dies suddendly and i dont know why. I think some components are faulty. So i have to move the homeassistant installation to another machine.
Not sure how to do this. i think i will install home assistant on a raspberry pi 3. Then how will i do to restore all the snapshots?

What OS are you on now? And does you current machine still function? If so, just take a backup to your computer with scp
scp -R <user>@<my-ha-instance:IP>:/var/snap/home-assistant-snap/current ~/ha-backup

(You can use this method from anywhere having scp available to pull a backup, e.g with a crontab).
then push it to the new instance with
scp -R ~/ha-backup/* <user>@<my-new-ha-instance:IP>:/var/snap/home-assistant-snap/current

Make sure that you have installed the snap on the new instance first, and I recommend that you stop it and clear the current-directory and wait to start it until the old backup is in place. :)
Hope it's helpful and explained good enough. But it's hard to know how your system works without any details.

What OS are you on now? And does you current machine still function? If so, just take a backup to your computer with scp
scp -R <user>@<my-ha-instance:IP>:/var/snap/home-assistant-snap/current ~/ha-backup

(You can use this method from anywhere having scp available to pull a backup, e.g with a crontab).
then push it to the new instance with
scp -R ~/ha-backup/* <user>@<my-new-ha-instance:IP>:/var/snap/home-assistant-snap/current

Make sure that you have installed the snap on the new instance first, and I recommend that you stop it and clear the current-directory and wait to start it until the old backup is in place. :)
Hope it's helpful and explained good enough. But it's hard to know how your system works without any details.

Im on ubuntu 18,04. Yes i can do the backup turning on the machine.
Thanks you for the solution!

Happy to help. Closing for now, feel free to re-open. :) Happy New Year!

Happy to help. Closing for now, feel free to re-open. :) Happy New Year!


[5.9] Fix sendResetFailedResponse to return the error message with the right key name

Resubmitting PR #27601, previously submitted to branch 5.7 and closed because of potential breaking changes.
Please let me know if you need more info or a different solution.
Currently, if $this->broker()->reset() returns and error message, this message will always be sent back to the previous location under the key 'email'. This way the error messages will be shown in reset.blade.php as an 'email' error, even if the error is a 'passwords.password' or a 'passwords.token'. This simple fix will check wich is the type of the error message and change the key name accordingly.

I intentionally built it this way to just lump all authentication errors under one key.


Tracking: Unflagging exports in node.js

Land Flagged Implementation
WIP PR: https://github.com/nodejs/ecmascript-modules/pull/72

[x] Respond and implement Jordans comments re handling "exports": null cases etc.
[x] Remove dot main support.
[x] Make Core PR.

Follow-up Work

[ ] Consider key / target validations (if any?).
[ ] CommonJS implementation.
[ ] Make call on support for differential exports.
[ ] Make call on support for package-internal entries.
[ ] Plug the hole on windows where x\\foo.js can get to foo.js even if it isn't exported.
[ ] Normalize exports mappings in CJS so that URL paths turn into valid file paths.

Replaced with project: https://github.com/orgs/nodejs/projects/3?add_cards_query=is%3Aopen


refactor(json): remove branks function

Changes
This PR removes the regex check for non-whitespace data with trim().length.

I don't think this is a desirable change. chunk.trim() allocates another string, which is unnecessary and something we should avoid

I don't think this is a desirable change. chunk.trim() allocates another string, which is unnecessary and something we should avoid

I see. @std has a few places where trim() is used for a comparison.
https://github.com/denoland/std/blob/f3a22c05535f549d68282496e3241ba8f6ffd97d/http/etag.ts#L205
https://github.com/denoland/std/blob/f3a22c05535f549d68282496e3241ba8f6ffd97d/internal/diff_str.ts#L188
https://github.com/denoland/std/blob/f3a22c05535f549d68282496e3241ba8f6ffd97d/cli/parse_args.ts#L374
Should we change the check there so @std is consistent?

I think triming only for checking should be generally avoided.


Warn user to not type full path of disk

This is not so important, but as I ran into this problem myself I figured I should share this.
In the apply script (app/*/apply), it would be nice to warn the user not to type the full path of the disk (eg /dev/sda) otherwise it will mess up the sed command when replacing values.
- ... "${YELLOW}Please choose your boot disk (e.g., nvme0n1, sda): ${NC}" ...
+ ... "${YELLOW}Please choose your boot disk (e.g., nvme0n1, sda) (do not type the full path "/dev/"): ${NC}" ...

Thanks, this is a good suggestion. I have improved the prompt in https://github.com/dustinlyons/nixos-config/commit/7d06bf749002418589ff97598d6b8fdb3404e37a.
For later reference, here is the command I ran on my Macbook at the root of the repository:
find . -type f -exec sh -c 'LC_ALL=C LANG=C sed -i "" -e "s/Please choose your boot disk (e.g., nvme0n1, sda):/Please enter the name of your boot disk (e.g., sda, nvme0n1). Do not include the full path (\"\\/dev\\/\\\"):/g" "$1"' _ {} \;


Marketplace broken when proxy blocks the experiment service endpoint

I debugged this issue with someone at Ignite. When he searched the marketplace in vscode, he got an error, and I found that this was from failing to download the experiments. But when we disabled experiments, searching worked fine.
He didn't have a proxy configured in VS Code, and he said that he wasn't using one, but he actually was using some company proxy or VPN. We found that some domains like microsoft.com or visualstudio.com were whitelisted and could be accessed, but the experiments domain msecnd.net was not. Also when VS Code accessed it, it got a "successful" response with a non-JSON response (the login page for the VPN). Normally if the request fails we handle that gracefully. But here, all requests to the marketplace first await the experiment service, and that failed, so no searches went through.
So I guess two issues

Handle an invalid response for the experiments data - I technically own this and will fix it
Should we provide a list of domains that we access for this sort of scenario? Or try to use recognizable domains like microsoft and visualstudio? @sandy081 do you know anything about this?

But when we disabled experiments, searching worked fine.

I am wondering how experiment services is related to marketplace search. Marketplace search should work irrespective of that. Are you able to repro this?

Should we provide a list of domains that we access for this sort of scenario? Or try to use recognizable domains like microsoft and visualstudio? @sandy081 do you know anything about this?

These services are listed in our product.json. We can go through them and whitelist.

Experiments can affect marketplace search results, so we have to load them first.

These services are listed in our product.json. We can go through them and whitelist.

This isn't exposed to users, I wouldn't expect them to look through it for things to whitelist. Also I think the whitelisting is something the IT department did, probably not for VS Code in particular.

Experiments can affect marketplace search results, so we have to load them first.

Still I do not think marketplace request shall fail when loading experiments fails. In such a case why don't marketplace requests can be made as if experiments are disabled?

This isn't exposed to users, I wouldn't expect them to look through it for things to whitelist. Also I think the whitelisting is something the IT department did, probably not for VS Code in particular.

Sorry then I have not understood your initial question regarding this.

Still I do not think marketplace request shall fail when loading experiments fails. In such a case why don't marketplace requests can be made as if experiments are disabled?

It shouldn't, but it does, I will fix that. And if they are disabled, then we skip that step entirely so it doesn't matter.
I guess I'm wondering whether we've thought about this before and how someone would know that VS Code needs to access the msecnd.net domain so they can whitelist it.


Program no longer opens - JS error

OS:
Windows 10 Home 64-bit
Issue Descriptions:
When I try to open the player I get the below error dialog box. The process continues to run but the screen never appears.
"A Javascript error occurred in the main process"
Uncaught Exception:
Syntax Error: Unexpected Token
Steps to Reproduce:
Happens every time I try to open the program. Have uninstalled/reinstalled with the latest version.
The last time I used the program succesfully I linked my last.fm account.

Delete the settings file found at
%APPDATA%/Google Play Music Desktop Player/json_store


refactor, chore(egen): Tensorflow version fix, grammar corrections, other corrections from template

This PR consists of the following updates:

Includes tensorflow installation step in installation section.
Refactors the notebook according to the notebook template (removes boilerplate elements, renames REGION with LOCATION and re-structures the sections).
Adds link for Colab enterprise.
Removes the usage of future tense(will) according to the guidelines.
Changes sentence style to second person.

REQUIRED: Fill out the below checklists or remove if irrelevant

If you are opening a PR for Official Notebooks under the notebooks/official folder, follow this mandatory checklist:

[x] Use the notebook template as a starting point.
[x] Follow the style and grammar rules outlined in the above notebook template.
[x] Verify the notebook runs successfully in Colab since the automated tests cannot guarantee this even when it passes.
[x] Passes all the required automated checks. You can locally test for formatting and linting with these instructions.
[ ] You have consulted with a tech writer to see if tech writer review is necessary. If so, the notebook has been reviewed by a tech writer, and they have approved it.
[ ] This notebook has been added to the CODEOWNERS file under the Official Notebooks section, pointing to the author or the author's team.
[x] The Jupyter notebook cleans up any artifacts it has created (datasets, ML models, endpoints, etc) so as not to eat up unnecessary resources.

/gcbrun


Why folder named as "GMLake"? - https://github.com/intelligent-machine-learning/glake/tree/main/GMLake

A typo?

Currently there'll be a few sub-dir each with specific functions: it could user to contribute and pick up to verify
We'll provide a guide at top README, in the future we may refactor the codes


Non-transpose implementation?

I'm afraid my C++ isn't quite up to par, so I haven't been able to get this working correctly. What would need to change in order to compute A * B instead of A^T * B? I'm afraid Apple's code has outsmarted me. 😕
Thanks in advance for the help.

Hi Collin. I'm sorry for not seeing your question sooner. My email is a little backed up at the moment.
It's tricky to switch from A^T * B to A * B. I think Apple used A^T * B for their performance test because it's a fast memory layout for measuring gflops. The individual values from A that get multiplied and accumulated against in B are both laid out sequentially in memory. Maximum cache hits. Etc.
The way the kernel works is that it chooses 8 rows of A and 8 columns of B and uses that to create an 8x8 sector of C. With A transposed, the problem translates to 8 columns of A^T and 8 columns of B to accumulate into an 8x8 sector of C. The kernel traverses 8 sequential column positions in each input matrix and then advances to the next row. strideA and strideB are used to advance the matrix pointers to the next row of 8 values.
If A is no longer transposed, the thing that changes is that we're feeding 8 columns of matrix A into the kernel when the matrix multiplication wants 8 rows from A. I think it could be possible for this  approach in the kernel to still work somehow.
The outer products that are accumulated from accumulateOuterProduct() into s00, s01, s10, s11 would probably have to be transposed but the bigger issue is that inputA and inputB would no longer share the same number of rows. Instead inputA would have to increment 8 elements for each loop instead of incrementing 8 rows. (The next sector of A would be "to the right" instead of "down".)
Even with that worked out, there's a chance that the accumulated-transposed 4x4 outer products would have to be placed into the output matrix in a different order.
Sorry for not having a better answer. I hope that helps.

@collinhundley, I'm hoping you checked out WWDC this year. Tons of stuff showing up in macOS 10.13. Including Metal Performance Shader stuff that makes the compute kernel in this repo obsolete.

Hey @otto-schnurr, sorry I never responded to your last message but I really appreciate your help.
Very excited for MPS on High Sierra, they definitely make life a lot easier!


Add linux aarch64 wheel support

Added linux aarch64 wheel support.
Related to #82. @maxbachmann, could you please review this PR.

Some more notes on the build system:

I am planning to drop support for Python2.7 in the next release (cibuildwheel will drop it in the next months anyways)
-> it is fine to remove it from the CI Build
It would probably make sense to test all wheels after building using CIBW_TEST_COMMAND instead of testing them beforehand. This would also remove the dependency of running all tests and then all wheel builds which would be faster.

I dropped Python2 support and added the testing of the wheels on the main branch (already identified a bug in the PyPy implementation, which I never tested)


DOF can effect transparent objects now,

Re-copy depth_texture when transparent objects write to depth, so DOF can effect transparent objects.
fix #28240

Thanks!


Failing React imports for validation UI

Please list the package(s) involved in the issue, and include the version you are using

Current Shopify CLI version: 3.56.
"react": "^18.0.0",
"@shopify/ui-extensions": "unstable",
"@shopify/ui-extensions-react": "unstable"
Also tested with:
"@shopify/ui-extensions": "2024.1.x",
"@shopify/ui-extensions-react": "2024.1.x"

Describe the bug
Impossible to run the Shopify extension due to issue with Shopify node modules
Steps to reproduce the behavior:

Follow the official Shopify tutorial https://shopify.dev/docs/apps/checkout/validation/create-complex-validation-rules/build-ui
Run npm run dev

✘ [ERROR] No matching export in<EMAIL_ADDRESS>for import "FunctionSettings"
11:18:19 │ extensions │ ✘ [ERROR] No matching export in<EMAIL_ADDRESS>for import "FunctionSettings"
11:18:19 │ extensions │
11:18:19 │ extensions │     extensions/my-validation-settings/src/ValidationSettings.jsx:6:2:
11:18:19 │ extensions │       6 │   FunctionSettings,
11:18:19 │ extensions │         ╵   ~~~~~~~~~~~~~~~~
11:18:19 │ extensions │
11:18:19 │ extensions │ ✘ [ERROR] No matching export in<EMAIL_ADDRESS>for import "Section"
11:18:19 │ extensions │
11:18:19 │ extensions │     extensions/my-validation-settings/src/ValidationSettings.jsx:7:2:
11:18:19 │ extensions │       7 │   Section,
11:18:19 │ extensions │         ╵   ~~~~~~~

Expected behavior
The @shopify/ui-extensions-react should export correctly

Hi @bernier! This looks like an issue with your lockfile. Can you try removing your package-lock.json or yarn.lock file and re-installing your node modules?
Because we publish FunctionSettings and Section to the unstable tag, npm can try to use one of the stable versions instead, which is the cause of the error.


Master Key

Hi all, thx
I need someone to explain to me how I generate the master key and the others, I don't quite understand that part, I'm sorry.
thx

Hello @EloFran!
You set the master key when running the MeiliSearch instance:

Here I set the master key to random-key-with-salt.
But, in production, you should set a more robust master key, like a big random and generated string.
Here more docs about authentication: https://docs.meilisearch.com/guides/advanced_guides/authentication.html#master-key


Need to Update Permalink Property in Config

The permalink property in the _config.yml needs to be updated to be unique.  Currently, it sets the post link to just the title name; however, if there are two posts with the same title, it only defaults to the latest.
Updated the two "Happenings" post to be unique; however, those will need to be updated when the permalink property is fixed.

//
Is likely a format that can be adopted.  Unfortunately, this would mean a break in metrics

commit 44e0a77eeb88a4e8e6366641ed7cbf25fde00c78


The upgrade process not works for me

Hello,
I used the following code to initialize a database.
var db = new Dexie("Users");

db.version(1).stores({
    friends: "firstName,lastName"
});

db.friends.put({firstName: "George", lastName: "Tzinos"});

Then i tried to use the upgrade process to add a new field with name "comment"
var db = new Dexie("Users");

db.version(1).stores({
    friends: "firstName,lastName"
});

db.version(2).stores({
    friends: "firstName,lastName,comment"
}).upgrade (trans => {
    return trans.friends.toCollection().modify (friend => {
        friend.comment = "Test commend";
    });
});

But it seems like i executed no command in the database.

The upgrade to version 2 will only happen once. Maybe you had already ran version 2 without the upgrade code and added it later? What happens if you add a version(3) with the upgrader attached? Does it run this time?
var db = new Dexie("Users");

db.version(1).stores({
    friends: "firstName,lastName"
});

db.version(2).stores({
    friends: "firstName,lastName,comment"
}).upgrade (trans => {
    return trans.friends.toCollection().modify (friend => {
        friend.comment = "Test commend";
    });
});

db.version(3).upgrade (trans => {
    return trans.friends.toCollection().modify (friend => {
        friend.comment = "Test commend";
    });
});

I upgraded only once. But it doesn't work.
Chrome version: Version 70.0.3538.77 (Official Build) unknown (64-bit)
OS: Ubuntu 16.04

Haven't heard this issue before. Are you using dexie@next or the stable one?

I put this test based on the master branch (version 3.0.0-alpha) but it worked as expected:
promisedTest ("issue-765", async ()=>{
    // Version 1
    let db = new Dexie('issue765');
    db.version(1).stores({
        friends: "firstName,lastName"
    });
    await db.friends.put({firstName: "George", lastName: "Tzinos"});
    db.close();

    // Version 2
    db = new Dexie('issue765');
    db.version(1).stores({
        friends: "firstName,lastName"
    });
    db.version(2).stores({
        friends: "firstName,lastName,comment"
    }).upgrade (trans => {
        return trans.friends.toCollection().modify (friend => {
            friend.comment = "Test commend";
        });
    });

    // Verifying
    const friends = await db.friends.toArray();
    deepEqual(friends, [{
        firstName: "George",
        lastName: "Tzinos",
        comment: "Test commend"
    }], "Upgrader should have run");
});

Now it works great, probably something makes the difference in your code.
After some tests, I found that it doesn't work without the following line:
   // Verifying
  const friends = await db.friends.toArray();

Ok I see. So you were still on version 1 all the time.
The reason that the line await db.friends.toArray() must happen, is because db.open() must be called for an upgrade to happen. I normally leave out the explicit call to db.open(), as it is implicitely called whenever an operation is done towards the db. In this case toArray().

Cool, thank you very much!


Integration and acceptance testsuite execution in Newman environment

Note : Created this issue to document the problem , analysis , workaround  and progress regarding the execution of acceptance and integration test suite collection in Newman environment.

Problem Description :
While running the integration and acceptance testsuite in the Newman environment , the testsuite execution stopped because of the following error message ,
FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
 1: 0x5582463a0e74 node::Abort() [node]
 2: 0x5582463a0ec2  [node]
 3: 0x558246598aaa v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [node]
 4: 0x558246598d25 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [node]
 5: 0x55824694a0a6  [node]
 6: 0x55824694a1e4  [node]
 7: 0x55824695c70b v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::GCCallbackFlags) [node]
 8: 0x55824695cec9 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [node]
 9: 0x55824695e2a6 v8::internal::Heap::CollectAllAvailableGarbage(v8::internal::GarbageCollectionReason) [node]
10: 0x55824695f878 v8::internal::Heap::AllocateRawWithRetryOrFail(int, v8::internal::AllocationSpace, v8::internal::AllocationAlignment) [node]
11: 0x55824692bfeb v8::internal::Factory::AllocateRawArray(int, v8::internal::PretenureFlag) [node]
12: 0x55824692c488 v8::internal::Factory::NewFixedArrayWithFiller(v8::internal::Heap::RootListIndex, int, v8::internal::Object*, v8::internal::PretenureFlag) [node]
13: 0x5582468f0fa5  [node]
14: 0x558246b62aba v8::internal::Runtime_GrowArrayElements(int, v8::internal::Object**, v8::internal::Isolate*) [node]
15: 0x26889115bf1d

Analysis :
The following factors are checked in the vm where Newman environment is available ,

Monitored the memory and CPU consumption using the top command. And in the top command , we could able to see a higher utilization of the CPU. While monitored this in detail , noticed that the CPU utilized by the Jenkin user is only less than 10% of CPU. So , the utilization of CPU in regards to the Newman process is reasonable.
We noticed that some other unwanted process were also running in the same vm which is utilizing the CPU. So killed those unwanted process in this VM.

top - 12:06:17 up 468 days,  7:06,  1 user,  load average: 1.67, 1.33, 1.30
Tasks: 383 total,   2 running, 381 sleeping,   0 stopped,   0 zombie
%Cpu(s): 13.0 us,  0.8 sy,  0.0 ni, 86.0 id,  0.0 wa,  0.2 hi,  0.0 si,  0.0 st
MiB Mem :  15872.9 total,   1756.3 free,  12117.7 used,   1998.9 buff/cache
MiB Swap:   7940.0 total,   7209.5 free,    730.5 used.   3366.8 avail Mem

   PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
3820173 git       20   0 1023452 622544  22252 R  99.0   3.8   0:21.86 bundle
1921517 git       20   0 1966144 830672  22276 S   2.7   5.1  16688:00 bundle
4160889 root      20   0   23.3g   4.0g  21592 S   2.3  25.8  40732:46 java
596641 gitlab-+  20   0  719724  16356   5888 S   1.0   0.1 633:17.26 node_exporter
596668 gitlab-+  20   0 3817408 224036  39488 S   1.0   1.4   2986:33 prometheus

Further , we noticed that the node version in the server is v10 , and noticed in the post there is a memory leak or difference between how Node.js <= 10 and Node.js > 10 manage memory. So , upgraded the node version to latest (ie) V16. And allocated 16 GB memory for the Newman process 'export NODE_OPTIONS=--max_old_space_size=16384'

By doing the points 2 and 3 , in the GCP , we are able to resolve the issue related to memory and we can able to generate reports.
Note : The average integration test suites execution time is 2 hrs.
Whereas in the testbed , the acceptance test suites execution time is comparatively higher than the integration test suites as the number of requests are more , still we the memory issue for few applications.
For example :
ApplicationLayerTopology takes more than 10 hrs to complete the execution
And after applying the workaround (point 2 and 3) also , the suits are stopped because of the "JavaScript heap out of memory" issue.
Continuing investigation with the acceptance test suite.

As discussed prepared a dummy suite to reproduce the issue and tested, the issue is reproduced on multiple environments from Jenkins and local servers.
Run the dummy suites with below multiple options from postman and newman.
Connectionlive+proxy takes long time for dummy suite
Connectionlive+noproxy takes less time for dummy suite
Connectionclose+proxy takes less time for dummy suite
Connectionclose+noproxy takes less time for dummy suite
Had discussion with team and analyzed the reports, based on it's proxy settings taking long time.
Proxy settings changed as per environment and restarted the servers .
Tested with dummy suite and other applications suites which is working fine taking less time for each request.


Add awscli2-boshrelease

Thanks. What do I do now to get previous final releases to be published on https://bosh.io/releases/github.com/starkandwayne/awscli2-boshrelease?all=1 (currently it is blank but there are 2 releases at https://github.com/starkandwayne/awscli2-boshrelease/tree/master/releases/awscli2)


Layers with symlinks differ depending on OS

I've noticed that when I run patched skopeo on Linux, it succeeds, but when I run it with the same JSON on Darwin, I get
FATA[0001] writing blob: writing to temporary on-disk layer: happened during read: Digest did not match, expected sha256:..., got sha256:...

And the layer in question is a .so library that has symlinks in it.
It looks like nix2containers produces different hashes on different OSs if the layer contains symlinks. To check this theory, I've added a symlink to test data and ran in on two different systems, I got on Darwin:
$ ln -s file1 data/tar-directory/symlink; go test ./nix -run TestTar
--- FAIL: TestTar (0.00s)
    tar_test.go:19: Digest is sha256:9a9402c601020cb24b71d4287b143d7bda875869970a1ed35792caa1702fdb69 while it should be sha256:efccbbe35209d59cfeebd8e73785258d3679fa258f72a7dfbc2eec65695fd5c8
FAIL
FAIL    github.com/nlewo/nix2container/nix      0.244s
FAIL

And on Linux:
$ ln -s file1 data/tar-directory/symlink; go test ./nix -run TestTar
--- FAIL: TestTar (0.00s)
    tar_test.go:19: Digest is sha256:077af73ad0fb226436e92a272318b777b6976b85c3a05d86183274818dd634f8 while it should be sha256:efccbbe35209d59cfeebd8e73785258d3679fa258f72a7dfbc2eec65695fd5c8
FAIL
FAIL    github.com/nlewo/nix2container/nix      0.013s
FAIL

Note that the digests are different in both cases, even though they should match. Without this symlink, test passes, so the digests are the same.
Note that this prevents me from using nix2container on macOS with Linux builder.

@YorikSar Thx for this nice bug report!
I never used Darwin and i don't know what could differ regarding symlinks...
If you set reproducible=false in a nix2container image or layer, the produced store path contains the JSON file and the tar file. This could allow us to diff the two produced tar files in order to know what differs (with tar -tvf for instance).
It would be really useful if you could provide such diff on a simple image (containing a symlink) built on Linux and Darwin!

Digging a bit deeper, I found that the NAR format is only described in Eelco's thesis and it includes following wording (on page 90 of the thesis, 98 of the PDF):

Fortunately, for our purposes—which is storing components in the Nix store—most of these features are either not relevant (drive letters, device nodes, etc.), can be ignored (hard links), or are never used in practice (streams). Some others, such as permissions, we will ignore to make life simpler.

He then proceeds with describing NAR format that indeed ignores all permissions except the executable flag on regular files. Since NAR format is used to transfer store paths between machines, we can expect to have such discrepancies in the OS-dependant things like symlinks (and they are, very much, OS-dependant, as it appears).
I guess since it's impossible to change symlink permissions on Linux, and Docker always (*) runs on Linux, and Nix doesn't care about permissions anyway, we should be forcing symlink permissions to 777 in layers.
(*) Docker can natively run containers on Linux and Windows, and I know of effort to bring it to FreeBSD. FreeBSD is officially not supported by Docker, and efforts to run Docker on it without emulation didn't get anywhere (yet?). Windows is not supported by Nix (yet?), and doesn't really have Unix permissions. I don't know of other container runtimes that can run outside Linux.

Created a PR with a fix for this.


Add custom-elements.json

Apparently it's a thing? https://github.com/webcomponents/custom-elements-manifest

Fixed in https://github.com/steren/stereo-img/commit/c0b4a5210b4e6c9a021e7c2bf6c0590f010f9326


Implementation of @fabric-ds/core

The Slider component now uses shared logic from @fabric-ds/core as well as classes exported from @fabric-ds/css/component-classes instead of @fabric-ds/component-classes, which is one of our recent changes to the css repo.
The Breadcrumbs component also uses shared logic from the core module.
Updates Webpack to a newer version for ESM support

See @fabric-ds/core here.

react-use-gesture is still in deps and can be nuked now