Bump Jetty to version 9.4.38.v20210224

jetty-9.4.38.v20210224 - 24 February 2021

4275 Path Normalization/Traversal - Context Matching
5963 Improve QuotedQualityCSV for CVE-2020-27223
5977 Cache-Control header set by a filter is override by the value from DefaultServlet configuration
5994 QueuedThreadPool "free" threads
5999 HttpURI ArrayIndexOutOfBounds
6001 Ambiguous URI legacy compliance mode

jetty-9.4.37.v20210219 - 19 February 2021

4275 Path Normalization/Traversal - Context Matching
5492 Add ability to manage start modules by java feature
5605 Blocked IO Thread not woken
5787 Make ManagedSelector report better JMX data
5851 org.eclipse.jetty.websocket.servlet.WebSocketServlet cleanup
5859 Classloader leaks from ShutdownThread and QueuedThreadPool
5909 Cannot disable HTTP OPTIONS Method
5937 Unnecessary blocking in ResourceService
5950 Deadlock due to logging inside classloaders
5973 Proxy client TLS authentication example
5977 Cache-Control header set by a filter is override by the value from DefaultServlet configuration
5979 Configurable gzip Etag extension

jetty-9.4.36.v20210114 - 14 January 2021

5310 Jetty Http2 client discards the response fames when there is GOAWAY and sends RST_STREAM
5499 Improve temporary buffer usage for WebSocket PerMessageDeflate
5633 Allow to configure HttpClient request authority
5689 Jetty ssl keystorePath doesn't work with absolute path
5755 Cannot configure maxDynamicTableSize on HTTP2Client
5783 Fix ConnectionStatistics.*Rate() methods
5785 Reduce log level for WebSocket connections closed by clients
5794 ServerConnector leaks closed sockets which can lead to file descriptor exhaustion
5824 Build up of ConstraintMappings when stopping and starting WebAppContext
5830 Jetty-util contains wrong Import-Package
5844 download flag to jetty-start causes NullPointerException
5845 Use UTF-8 encoding for client basic auth if requested
5855 HttpClient may not send queued requests
5870 jetty-maven-plugin fails to run ServletContainerInitializer on Windows due to URI case comparison bug

Can you add exerts from VERSION.txt for 36, 37 and 38 to the PR description.

Done.

@joakime I think we should close this and go straight to a more recent release


[v11] Clean up deprecated app delegate methods

#no-changelog

It would, but I would be surprised if the old APIs are still being used since they've been deprecated for awhile.


Thank you!

Feel free to delete this, I only wanted to say thank you for Closeread and Sverto and your great presentations at posit::conf!
I've started playing with Closeread and is super fun, I'll be following its development!

Thanks so much, Georgios! I got to watch @andrewpbray's talk online, and he knocked it outta the park imho 😉 We're both keen to keep making Closeread better!

Looking forward to it! Here's the repo of my first story if you want to have a look 😃
https://github.com/gkaramanis/tidytuesday/tree/master/2024/2024-week_33

That's awesome - thanks for reaching out and sharing your work! It's so gratifying to see what others can do with these tools.


chore: Set gorilla oidc secret

Where did we land with merging this change? Was there any concern with adding this in?


Adding prefix/suffix for data

I want to add % symbol to my data, but then the value will be a String, so sorting is not OK.

Expected Behavior

If I have a column with this data: [10.9%, 30.5%, 6.4%, 3%, 9.8% ]
I want to see on Sort: [30.5%, 10.9%, 9.8%, 6.4%, 3%]
Current Behavior

If I have a column with this data: [10.9%, 30.5%, 6.4%, 3%, 9.8% ]
I got on Sort: [9.8%, 6.4%, 30.5%, 3%, 10.9%]
Steps to Reproduce (for bugs)

Add these data for a sortable column
Click on the column header, to sort
It sorts as a String, but I have to sort as a number

Your Environment

Tech
Version

Material-UI
4.11.0

MUI-datatables
3.7.1

React
17.0.1

browser
Chrome

etc

Uups, sorry!
It is resolvable with "sortCompare" property


time seems invalid in influxdb

Hi,
First of all, thanks for the module, good job :+1: .
But I have an issue I'm not sure why.
It actually writes to the influxdb but the time in influxdb indicates 1970, like it was close to timestamp 0.
Have you seen that behaviour before?

That's a great catch, thanks!
What's happening is we're letting InfluxDB create the timestamp instead of using the one from Logrus.
Resolved: https://github.com/Abramovic/logrus_influxdb/issues/2

Please let me know if this fixes things for you :)

I actually still see 1970-01-01T00:00:01.446548954Z, I will dig into it and continue contributing to the repo.

I actually removed Precision and it now works.
My influxdb version is v<IP_ADDRESS>, I compiled it from scratch with their dockerfiles and shell scripts.
I hope it doesn't create any issue on your side and that we can merge the PR.

bump

Merged :)

:+1:


Header fileld problem when i use this crypt plugin

Hello, when I install this plugin I got this error in some http request
Request header field sessionIdsTT is not allowed by Access-Control-Allow-Headers.
I use Ionic and AngularJS.
Any idea to fix that? Only Happen in Android, in iOS works well.
Thanks

You need to have cross origin header on the server side, otherwise it won't work. At least for XHR requests I think.


I can't select an option

I do it all, and It works fine, but when I show all the options, I can't select nothing from the dropdown. I don't know if it's an option of the FormType or what, but it's impossible for me to select nothing.

Same problem for me


Repeatable segfault while editing HTML file

From<EMAIL_ADDRESS>on 2013-06-26T14:36:03Z
Spyder Version:  2.3.0dev3 (2872: rfe6f59363a2b )
Python Version:  2.7.4
Qt Version:      4.8.4, PyQt4 (API v1) 4.10 on Linux (Ubuntu 13.04)
What steps will reproduce the problem?

The attache file is just

Try to edit it to become

 What is the expected output? What do you see instead? Spyder fails with segfault once a first ' -' is entered.

Please provide any additional information below
.
The bug did not repeat after I restarted Spyder, and hit ? -> Report bug.. to open this page.
I attach debug logs, and I am afraid I don't know what to do next.
Attachment: crash.html gdb.txt
Original issue: http://code.google.com/p/spyderlib/issues/detail?id=1462

From<EMAIL_ADDRESS>on 2013-12-12T10:25:36Z
remark  : if you do
2.

t

3.

t

4.

It doesn't crash.

From contrebasse on 2013-12-12T12:55:20Z
I can reproduce this bug, but it segfaults when I enter the second '-', I have no problem with the first one.
It doesn't crash if the file has a .py extension rather than html.
I managed to have an error in the internal console (and no crash) by doing :
1.

3.
<!- table -->

Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/spyderlib/widgets/sourcecode/syntaxhighlighters.py", line 840, in highlightBlock
self.formats[key])
KeyError: 'multiline_comment_end'
As the crash is only with .html file, I suppose that the crash is also due to the syntax highlighting when the file is invalid.
I have:
Spyder Version:  2.3.0dev (today's last mater)
Python Version:  2.7.6
Qt Version    :  4.8.5, PyQt4 (API v2) 4.10.3 on Linux
pyflakes >=0.5.0:  0.7.3 (OK)
pep8 >=0.6      :  1.4.6 (OK)
IPython >=0.13  :  1.1.0 (OK)
rope >=0.9.2    :  0.9.4 (OK)
sphinx >=0.6.6  :  1.1.3 (OK)
matplotlib >=1.0:  1.3.1 (OK)
sympy >=0.7.0   :  0.7.4 (OK)
pylint >=0.25   :  1.0.0 (OK)

From ccordoba12 on 2015-01-02T19:04:22Z
Labels: MS-v2.3.3

From ccordoba12 on 2015-01-09T18:50:55Z
Labels: -MS-v2.3.3 MS-v2.3.4

I can't reproduce this anymore. Maybe it was fixed when pygments was integrated in spyder ? @techtonik can you check if it's fixed for you ?

I had a look last night and I could reproduce it. I'll try to work on a fix later today.
On 09:16, 24 Jun 2016, at 09:16, Joseph Martinot-Lagarde<EMAIL_ADDRESS>wrote:

I can't reproduce this anymore. Maybe it was fixed when pygments was
integrated in spyder ? @techtonik can you check if it's fixed for you ?

You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/spyder-ide/spyder/issues/1462#issuecomment-228284964

:(
Here is my config, for reference:
Version and main components

Spyder Version: 3.0.0.dev0
Python Version: 3.5.1
Qt Versions: 5.6.0, PyQt5 5.6 on Linux

Dependencies
pyflakes >=0.6.0:  1.2.3 (OK)
pep8 >=0.6      :  1.7.0 (OK)
pygments >=2.0  :  2.1.3 (OK)
qtconsole >=4.0 :  4.2.1 (OK)
nbconvert >=4.0 :  4.2.0 (OK)
pandas >=0.13.1 :  0.18.1 (OK)
numpy >=1.7     :  1.11.0 (OK)
sphinx >=0.6.6  :  1.4.1 (OK)
rope >=0.9.4    :  0.9.4-1 (OK)
jedi >=0.8.1    :  0.9.0 (OK)
psutil >=0.3    :  4.1.0 (OK)
matplotlib >=1.0:  1.5.1 (OK)
sympy >=0.7.3   :  None (NOK)

I wonder if Spyder is exploitable..

@techtonik Could you be more constructive ?

I think he means because of malformed source code.

I found an issue in the HTML syntax highlighter. If an occurrence of  --> (indicating the end of an HTML comment) is not matched with a  then the  exception is printed in the console and Spyder does not crash.
I have prepared a pull request to fix the issue.
Here is my configuration, for reference. The main difference with Nodd seems to be that I have Qt4 while he has Qt5.

Spyder Version: 3.0.0.dev0 dbbca91
Python Version: 3.5.1
Qt Versions: 4.8.7, PyQt4 (API v2) 4.11.4 on Linux

Dependencies
pyflakes >=0.6.0:  1.2.3 (OK)
pep8 >=0.6      :  1.7.0 (OK)
pygments >=2.0  :  2.1.3 (OK)
qtconsole >=4.0 :  4.2.1 (OK)
nbconvert >=4.0 :  4.2.0 (OK)
pandas >=0.13.1 :  0.18.1 (OK)
numpy >=1.7     :  1.11.0 (OK)
sphinx >=0.6.6  :  1.4.4 (OK)
rope >=0.9.4    :  0.9.4-1 (OK)
jedi >=0.8.1    :  0.9.0 (OK)
psutil >=0.3    :  4.3.0 (OK)
matplotlib >=1.0:  1.5.1 (OK)
sympy >=0.7.3   :  1.0 (OK)
pylint >=0.25   :  1.5.6 (OK)

@Nodd if program ends with a crash. it may be exploited to do the things the attacker wants by forcing user to open a malicious HTML document.

@ccordoba12 why you closed thise? Where is the fix?


Missing Swank lisp file?

Hi, you write:

The following Lisp file, loaded by app_main, contains a couple of snippets of code I copied from the Emacs Slime client that launches the Swank server.

However I don't see the code listed anywhere.
Could you please add this file to the repo or provide a link to sample, thanks.
Mick.

I solved my issue eventually, for those new to swank (like me) the file would have been handy, cheers.

@MickDuprez Thanks for creating the issue! I'm reopening it as a reminder to update the document, something I'm hoping to do.

No problem, the article is a great resource 👍

I think it will be also useful to link this here as well https://github.com/slime/slime/blob/master/start-swank.lisp


Resteasy async client

Still not finished yet, put it here to record the progress and for reference.

SpecialResourceTest.test583() failed because of httpcomponents minor version update (It uses stricter mime type checking, so the test need to be rewritten. Currently it is ignored)

AsyncTimeoutExceptionsTest.futureTimeoutAndMoreRequestsTest() won't timeout when async client is used.

AsyncTimeoutExceptionsTest .invocationCallbackTimeoutAndMoreRequestsTest() won't timeout when async client is enabled.

Current progress:

Migrated Markus PR
Made async-client fully optional (When async-client is in classpath, it will be used, or else it will fallback to the existing asyncExecutor)
legacy-tests test with out async-client
non legacy integration tests are tested with async-client now

This is pretty cool stuff. I've written a test that uses the NIO client, and it seems to be kinda long:
   @Test
   public void testNIO() throws Exception {
      HttpAsyncClientBuilder asyncClientBuilder = HttpAsyncClientBuilder.create();
      CloseableHttpAsyncClient httpAsyncClient = asyncClientBuilder.build();
      ClientHttpEngine engine = new ApacheHttpAsyncClient4Engine(httpAsyncClient, true);
      ResteasyClientBuilder clientBuilder = new ResteasyClientBuilder();
      clientBuilder.httpEngine(engine);
      Client client = clientBuilder.build();
      WebTarget target = client.target("http://localhost:8081/test");
      Builder builder = target.request();
      AsyncInvoker asyncInvoker = builder.async();
      Future<Response> future = asyncInvoker.get();
      Response response = future.get();
      String report = response.readEntity(String.class);
      System.out.println("response: " + report);
      client.close();
   }

Obviously, I'm not taking advantage of the fluent API, which was just for my own understanding, but building the Client takes a lot of work. Is there an easier way?
-Ron

How about something like ResteasyClientBuilder.buildAsync() that creates an ApacheHttpAsyncClient4Engine?

I'm working on this at https://github.com/resteasy/Resteasy/pull/1249 ...


Automated Weave version update 2.6.5-20230417

Automated changes by the cron-weave-update GitHub action
Adds [Weave add-on](https://kurl.sh/docs/add-ons/weave) version 2.6.5-20230417.

It has just a flake so it shows fine get merged:


Warning for deprecated aad-pod-identity

According to the official documentation of AAD Pod Identity, (https://github.com/Azure/aad-pod-identity/tree/master?tab=readme-ov-file) it has been deprecated. The recommendation is to transition to Azure Workload Identity.

@MGoedtel
Can you review the proposed changes?
When the changes are ready for publication, add a #sign-off comment to signal that the PR is ready for the review team to merge.
#label:"aq-pr-triaged"
@MicrosoftDocs/public-repo-pr-review-team

#please-close

Hello @MGoedtel thank you for your comment. If you could provide an example of the note that you are using across the other related articles, it would be immensely helpful. This way, I can make the necessary updates to the PR in line with the suggested changes. I would like to make some contributions to the document.


[gui error report] Error: invalid ip address

👉 Please describe what you were doing when this error happened.
Specifications

OS: win32 10.0.22000
IPFS Desktop Version: 0.24.0
Electron Version: 19.0.9
Chrome Version: 102.0.5005.167

Error
Error: invalid ip address
    at ip2bytes (C:\Program Files\IPFS Desktop\resources\app.asar\node_modules\multiaddr\src\convert.js:106:11)
    at Function.convertToBytes [as toBytes] (C:\Program Files\IPFS Desktop\resources\app.asar\node_modules\multiaddr\src\convert.js:72:14)
    at C:\Program Files\IPFS Desktop\resources\app.asar\node_modules\multiaddr\src\codec.js:109:35
    at Array.map (<anonymous>)
    at stringTuplesToTuples (C:\Program Files\IPFS Desktop\resources\app.asar\node_modules\multiaddr\src\codec.js:103:17)
    at stringToBytes (C:\Program Files\IPFS Desktop\resources\app.asar\node_modules\multiaddr\src\codec.js:221:13)
    at Object.fromString (C:\Program Files\IPFS Desktop\resources\app.asar\node_modules\multiaddr\src\codec.js:231:10)
    at new Multiaddr (C:\Program Files\IPFS Desktop\resources\app.asar\node_modules\multiaddr\src\index.js:56:26)
    at multiaddr (C:\Program Files\IPFS Desktop\resources\app.asar\node_modules\multiaddr\src\index.js:621:10)
    at parseMultiaddr (C:\Program Files\IPFS Desktop\resources\app.asar\src\daemon\config.js:118:7)

Thanks for submitting the issue @rachidlamari. Looks like this error is originating from: https://github.com/ipfs/ipfs-desktop/blob/main/src/daemon/config.js#L115 which implies an invalid IP address was saved as config. Can you please confirm what's the IP address in your configuration? Also can you share more logs from your setup? you can find the config and log files here.


Failed to compile project when importing redux-pack

Node version: v7.2.0
Project created via create-react-app with react-scripts v0.7.0
I've tried to install with both npm install -S redux-pack and yarn add redux-pack, but when I'm importing redux-pack into project:
import { middleware as reduxPackMiddleware } from 'redux-pack';
I'm receiving this error:
Failed to compile.

Error in ./~/redux-pack/lib/handle.js
Module not found: 'deline' in [path to project]/node_modules/redux-pack/lib

 @ ./~/redux-pack/lib/handle.js 1:1303-1320

I've already tried manually install 'deline' module and it appears in 'node_modules' folder, but haven't succeed.
Can someone please share any thoughts on possible solutions and I will try to fix it?

https://github.com/airbnb/deline/issues/1
It seem "Deline" package have an incorrect main entrypoint in package.json file.
You can change it in "node_modules/deline/package.json".
I think Airbnb will resolve the problem asap.
If you use WebPack, just use "safe-deline" package : https://github.com/kMeillet/safe-deline and add module aliasing :) .

Hey guys,
appears I have a possibly related similar issue - after installing & importing redux-pack I get:
TransformError: <project>/node_modules/deline/build/deline.js: Couldn't find preset "airbnb" relative to directory "<project>/node_modules/deline"

Installing babel-preset-airbnb works around this for me, but I thought I let you know anyway.


Sort Does Not Work on Org Repo List

About: https://github.com/Mottie/GitHub-userscripts/wiki/GitHub-sort-content
Steps to Reproduce:

Go to https://github.com/jquery (as in the example gif in the GitHub-sort-content wiki page)
Notice the lack of the sort indicator

Notes:
It still works in the People tab...

Hi @AdrianTP!
Thanks, I'll work on this soon.

Please review the change log for this update.

@Mottie Lovely, thanks so much for the quick turnaround!


Installation/running issue.

Thank you for this software!  I receive the error message "Unable to access jarfile procyon-decompiler-0.5.36" when I enter "java -jar procyon-decompiler-0.5.36 -jar json-target-dynamic.jar -o destdir" at the command prompt  (without quotation marks).
I'm running Windows 10, JDK 1_8_231, latest version of Procyon (0.5.36).    Re-started the CMD prompt, but still can't run file.  I converted to zip, extracted, re-zipped, renamed to jar, still can't run.  I tried the 0.5.30 file, same issue.
I added the directory where I placed the Procyon file to my environment variables: Path, Classpath and even JAVA_HOME.  Still same error.
Sorry in advance if I'm making a noob mistake in any way.
Any ideas on how to fix this and get it running would be greatly appreciated.
Thank you

Evidently I needed to include the full run path for the Procyon file and for the decompiled file.  Shortening the decompiler name to Procyon.jar helped as well.
Problem resolved.


[Storage] Using AzureSasCredential.

This PR introduces AzureSasCredential to Storage clients and is a continuation from https://github.com/Azure/azure-sdk-for-python/pull/15946 .
Please notice that it diverges from point 2 in https://github.com/Azure/azure-sdk/issues/1954#issuecomment-718190611 .
The reason is that it already has behavior that precedes this design here https://github.com/Azure/azure-sdk-for-python/blob/80c3919700ada1d63998bbe446ea80a0f5eda026/sdk/storage/azure-storage-blob/azure/storage/blob/_blob_client.py#L90-L93 and is capable to accept SAS as string already.
This PR is pending Core release.

/azp run python - storage - ci

/azp run python - storage - ci

/azp run python - storage - tests

/azp run python - storage - tests

/azp run python - storage - tests

/azp run python - storage - tests


feat: modal elements and crypto icons

Can we add this checkmark icon with a circle around it for success modal? And a similar one for error state, i.e. Close icon with a red circle around it?

Link to figma: https://www.figma.com/file/aXMwUrkMr2xT3pXhqaW8VG/Lido?node-id=7044%3A44495


New ContainerTasker which will use either DockerTasker

Signed-off-by: Robert Cerven<EMAIL_ADDRESS>Maintainers will complete the following section:

[x] Commit messages are descriptive enough
[x] "Signed-off-by:" line is present in each commit
[x] Code coverage from testing does not decrease and new code is covered
[x] JSON/YAML configuration changes are updated in the relevant schema
[x] Changes to metadata also update the documentation for the metadata
[x] Pull request includes link to an osbs-docs PR for user documentation updates
[x] New feature can be disabled from a configuration file

This pull request introduces 1 alert when merging 0c4a4522daffdb88483e48a326cd5a065a3200f4 into 4d8d213aa4cb56c8150e11496f089b78afc4eee6 - view on LGTM.com
new alerts:

1 for Unused import

This project has automated code review enabled, but doesn't use the LGTM GitHub App. Migrate over by installing the app. Read about the benefits of migrating to GitHub Apps in the blog.

Comment posted by LGTM.com

Should be occurrences of DockerTasker replaced by ContainerTasker? Namely:
atomic_reactor/buildimage.py:39:        self.tasker = DockerTasker()
atomic_reactor/core.py:81:        self.tasker = DockerTasker()
atomic_reactor/outer.py:44:        self.dt = DockerTasker()
atomic_reactor/outer.py:84:            dt = DockerTasker()
atomic_reactor/plugins/build_orchestrate_build.py:269:        :param tasker: DockerTasker instance  # many occurrences in docstrings

@MartinBasti  those are used for standalone building with atomic_reactor
which don't use regular workflow
but I can change comments in plugins about tasker param to be ContainerTasker

changed comments about tasker param in plugins

rn updated: - New ContainerTasker which will use either DockerTasker or PodmanTasker when that is implemented


cmd/go: option to initialize go mod from git submodules

Hi,
I have historically used git submodules as the most reliable way to manage library dependencies in my Go projects. I would very much like to migrate to modern Go modules. However, the go mod init system does not appear to support generating go.mod configurations from git submodules.
When I try to do this, I either get NO modules in the resulting go.mod file, or else I get the latest versions of everything, which of course breaks my application. Please include automation for migrating vendor/ directories from git submodules to go.mod.
Furthermore, when I try to explicitly copy the exact same git refs for the dependency versions, then go mod operations corrupt these. go mod checks out a completely different version of the dependency tree than I have in the original vendor/ with git submodules.
If someone can help me to reliably migrate the exact same dependency tree to go mod, then I would be grateful.
Example:
https://github.com/mcandre/octane/tree/61e9374d9695f2b3f1489741396f4f3082f3fd74

CC @bcmills @matloob @jayconrod

Could you say more about how you're using submodules? Concepts in Go modules and Git submodules don't map directly to each other, so it's not clear to me how this would work or whether it would be feasible.
It sounds like you have submodules checked out into the vendor directory, presumably at locations that correspond to repository root paths. If that's the case, then git submodule status would show something like this for each repo:
012345abcdef vendor/example.com/repo

It's probably feasible for go mod init to list directories inside vendor containing .go files, find what submodule they belong to, then build a go.mod based on that. Does that sound about right?
We're probably not going to be able to get to this soon; I don't think many people are in this situation. In the mean time, you can simulate the result by using go get to pull in each dependency.
go get -d example.com/repo@012345abcdef

Could you say more about how you're using submodules? Concepts in Go modules and Git submodules don't map directly to each other, so it's not clear to me how this would work or whether it would be feasible.

This never got a response, and it seems like noone else needs this or cares to implement it, so I'm going to close this feature request for now.


#3635 fix(containers) sort volumes in container creation view

Closes #3635
Sorts volumes in the container creation view drop-down into alphabetical order as shown in the image below.

Gonna ask @chiptus to have a look at the code

Thanks @DarkAEther :+1:


Scientific publishing amendments

Sorry, looks like I broke a snapshot test in master by forking my last PR off a slightly old commit from master.
As this branch needs a manual merge with master, could you please also update the webapp snapshots after merging?


Information about Unique session Ids missing

Hi, would you be able to provide more info on 'uniqueSessionIDs' ? How do I find them if I am building an independent web-application?

See https://docs.api.ai/docs/query#query-parameters-and-json-fields

A string token up to 36 symbols long, used to identify the client and to manage session parameters (including contexts) per client.

Hi @rishigb
See https://docs.api.ai/docs/query#query-parameters-and-json-fields

A string token up to 36 symbols long, used to identify the client and to manage session parameters (including contexts) per client.

Thank you. I have read this before. My question is how do I generate session Ids if I am building an independent web application. Is there a link you could guide me to?

Sorry, but now we doesn't have any detailed guid for sessionId. But, I'll try to explain...
Some data should be stored on server-side (contexts, user entities, etc...), and we should identify current user for store data for each unique user.
For example, you can see to information about slot filling https://docs.api.ai/docs/guidelines-slot-filling
This is concept store some partial data for create dialog.
On your site you can user some session id (like php) or some hashes of user id or email. If your user does not authorized you can generate UUID and store it into cookies and when user has send request, you send his unique id.

@rishigb I used this lib to generate session IDs
https://www.npmjs.com/package/node-uuid
and I simply stored sessions in a map (map was enough for my app, of course it could not be the ideal solution)

How to map the Web Session ID with DialogFlow Session ID?

How to map the Web Session ID with DialogFlow Session ID?

A way to get this, is to use the local storage on JavaScript
`var myStorage = localStorage;
if (!myStorage.getItem("session")) {
myStorage.setItem("session", createUUID());
}
function createUUID() {
// http://www.ietf.org/rfc/rfc4122.txt
var s = [];
var hexDigits = "0123456789abcdef";
for (var i = 0; i < 36; i++) {
s[i] = hexDigits.substr(Math.floor(Math.random() * 0x10), 1);
}
s[14] = "4"; // bits 12-15 of the time_hi_and_version field to 0010
s[19] = hexDigits.substr((s[19] & 0x3) | 0x8, 1); // bits 6-7 of the clock_seq_hi_and_reserved to 01
s[8] = s[13] = s[18] = s[23] = "-";
var uuid = s.join("");
return uuid;
}
`


Document how to debug slowness of the proof

The documentation says:

Model checking allows you to prove non-trivial properties about programs, and check those proofs in roughly the same amount of time as a traditional test suite would take to run. The downside is many types of properties can quickly become "too large" to practically model-check, and so writing "proof harnesses" (very similar to property tests and fuzzer harnesses) requires some skill to understand why the solver is not terminating and fix the structure of the problem you're giving it so that it does. This process basically boils down to "debugging" the proof.

However I was not able to find any documentation on how to do such debugging. I've encountered some slow tests and would love to learn how to make them faster.

Thanks! I will look into it. I suspect loops are what made my private experiment slow. There's another public code we have that implements u256 which obviously has to contain multiplication/modulo/division. I guess we'll just accept the slowness for now.

Remove sources of nondeterminism from the program

Is there any way to do this that doesn't decrease the power of the proof? The linked issue suggests that it's better to just exhaustively test some values (bools?).

Thanks! I will look into it. I suspect loops are what made my private experiment slow. There's another public code we have that implements u256 which obviously has to contain multiplication/modulo/division. I guess we'll just accept the slowness for now.

Thanks for sharing your code! Please feel free to participate in issues related to slow operations or open new ones if they don't match your experience.

Is there any way to do this that doesn't decrease the power of the proof?

I can't think of any, assuming that "power" refers to the "generality" of the proof. But it's OK to decompose one proof into multiple harnesses (where one input gets fixed to a different value each time) to make the individual harnesses more efficient. Of course, the more general proof would be preferred in most cases, but that might not be feasible in some of them.


Fix/console cleanup

Description
Able to resolve most console warnings and errors, not all of them.
Checklist
Remove any items which are not applicable.

[x] I have performed a self-review of my own code
[ ] I have commented my code, particularly in hard-to-understand areas
[ ] I have made corresponding changes to the documentation
[ ] I have added tests that prove my fix is effective or that my feature works AND the tests pass

❗ ❗ Just adding this again, please test heavily during review ❗ ❗


use Feign, Hystrix, and Consul always fails in the first time, I know the reason, but feign client configure about time-out does not work

Excuse me, I hava see the problem in this issue, and I meet the same problem to him. I know that in the first time, some class lazy initiallize that will need more time. So I try to configure the time of time-out. But it does not work. I configure 12s in time out, but it always enter hystrix in 2s. I dont know the reason, and I have block in the problem a long time. The amazing thing is that if I configure little time, it works. Such as if I configure 10ms, the request will always fails. If I configure 2s, it fails in the first time, and sussceed then. If I configure 12s, it still fails in the first time, it can not reach 12s, it will fails quickly within 2s. I really dont know the reason.Thanks very much if someone knows the reason~
This is my feign client:
@FeignClient(value = "spring-boot-demo", fallback = HelloClientFallback.class, configuration = FeignConfigure.class)
public interface TestClient {
    @RequestMapping(method = RequestMethod.POST, value = "/test/hello")
    public String getAnswerResult(@RequestParam(value = "param1") String param1,
            @RequestParam(value = "param2") String param2,
            @RequestParam(required = false, value = "param3") String param3);
}

This is my feign configure:
@Configuration
public class FeignConfigure {
     public static int connectTimeOutMillis = 12000;
     public static int readTimeOutMillis = 12000;
     @Bean
     public Request.Options options() {
            return new Request.Options(connectTimeOutMillis, readTimeOutMillis);
     }
}

I write a small demo specifically to reproduce the problem, this is the git repository: https://github.com/jiangchao123/feign-test

Can you provide the stack trace and tell us what version of Spring Cloud you are using?

I am very happy for recieving your reply. I use version Camden.SR6. When I use feign client to request in the first time, it will time out. I have known the reason when I use virsual VM, it is because of the class of DynamicServerListLoadBalancer. The stack trace below:
2017-05-09 00:06:34.484  INFO 1915 --- [ing-boot-demo-1] s.c.a.AnnotationConfigApplicationContext : Refreshing org.springframework.context.annotation.AnnotationConfigApplicationContext@4057b31: startup date [Tue May 09 00:06:34 CST 2017]; parent: org.springframework.boot.context.embedded.AnnotationConfigEmbeddedWebApplicationContext@61f05988
2017-05-09 00:06:34.533  INFO 1915 --- [ing-boot-demo-1] f.a.AutowiredAnnotationBeanPostProcessor : JSR-330 'javax.inject.Inject' annotation found and supported for autowiring
2017-05-09 00:06:34.656  INFO 1915 --- [ing-boot-demo-1] c.netflix.config.ChainedDynamicProperty  : Flipping property: spring-boot-demo.ribbon.ActiveConnectionsLimit to use NEXT property: niws.loadbalancer.availabilityFilteringRule.activeConnectionsLimit =<PHONE_NUMBER>
2017-05-09 00:06:34.683  INFO 1915 --- [ing-boot-demo-1] c.n.u.concurrent.ShutdownEnabledTimer    : Shutdown hook installed for: NFLoadBalancer-PingTimer-spring-boot-demo
2017-05-09 00:06:34.769  INFO 1915 --- [ing-boot-demo-1] c.netflix.loadbalancer.BaseLoadBalancer  : Client:spring-boot-demo instantiated a LoadBalancer:DynamicServerListLoadBalancer:{NFLoadBalancer:name=spring-boot-demo,current list of Servers=[],Load balancer stats=Zone stats: {},Server stats: []}ServerList:null
2017-05-09 00:06:34.776  INFO 1915 --- [ing-boot-demo-1] c.n.l.DynamicServerListLoadBalancer      : Using serverListUpdater PollingServerListUpdater
2017-05-09 00:06:34.815  INFO 1915 --- [ing-boot-demo-1] c.netflix.config.ChainedDynamicProperty  : Flipping property: spring-boot-demo.ribbon.ActiveConnectionsLimit to use NEXT property: niws.loadbalancer.availabilityFilteringRule.activeConnectionsLimit =<PHONE_NUMBER>
2017-05-09 00:06:34.818  INFO 1915 --- [ing-boot-demo-1] c.n.l.DynamicServerListLoadBalancer      : DynamicServerListLoadBalancer for client spring-boot-demo initialized: DynamicServerListLoadBalancer:{NFLoadBalancer:name=spring-boot-demo,current list of Servers=[<IP_ADDRESS>:7001],Load balancer stats=Zone stats: {unknown=[Zone:unknown;	Instance count:1;	Active connections count: 0;	Circuit breaker tripped count: 0;	Active connections per server: 0.0;]
},Server stats: [[Server:<IP_ADDRESS>:7001;	Zone:UNKNOWN;	Total Requests:0;	Successive connection failure:0;	Total blackout seconds:0;	Last connection made:Thu Jan 01 08:00:00 CST 1970;	First connection made: Thu Jan 01 08:00:00 CST 1970;	Active Connections:0;	total failure count in last (1000) msecs:0;	average resp time:0.0;	90 percentile resp time:0.0;	95 percentile resp time:0.0;	min resp time:0.0;	max resp time:0.0;	stddev resp time:0.0]
]}ServerList:ConsulServerList{serviceId='spring-boot-demo', tag=null}
2017-05-09 00:06:35.046  INFO 1915 --- [nio-7001-exec-9] c.j.feign.controller.HelloController     : response: hello world hello world hello world hello world hello world hello world hello world
2017-05-09 00:06:35.459 ERROR 1915 --- [nio-7001-exec-8] o.a.c.c.C.[.[.[/].[dispatcherServlet]    : Servlet.service() for servlet [dispatcherServlet] in context with path [] threw exception [Request processing failed; nested exception is com.netflix.hystrix.exception.HystrixRuntimeException: HelloClient#getAnswerResult() timed-out and no fallback available.] with root cause

java.util.concurrent.TimeoutException: null
	at com.netflix.hystrix.AbstractCommand.handleTimeoutViaFallback(AbstractCommand.java:980) ~[hystrix-core-1.5.6.jar:1.5.6]
	at com.netflix.hystrix.AbstractCommand.access$500(AbstractCommand.java:59) ~[hystrix-core-1.5.6.jar:1.5.6]
	at com.netflix.hystrix.AbstractCommand$12.call(AbstractCommand.java:595) ~[hystrix-core-1.5.6.jar:1.5.6]
	at com.netflix.hystrix.AbstractCommand$12.call(AbstractCommand.java:587) ~[hystrix-core-1.5.6.jar:1.5.6]
	at rx.internal.operators.OperatorOnErrorResumeNextViaFunction$4.onError(OperatorOnErrorResumeNextViaFunction.java:140) ~[rxjava-1.1.10.jar:1.1.10]
	at rx.internal.operators.OnSubscribeDoOnEach$DoOnEachSubscriber.onError(OnSubscribeDoOnEach.java:87) ~[rxjava-1.1.10.jar:1.1.10]
	at rx.internal.operators.OnSubscribeDoOnEach$DoOnEachSubscriber.onError(OnSubscribeDoOnEach.java:87) ~[rxjava-1.1.10.jar:1.1.10]
	at com.netflix.hystrix.AbstractCommand$HystrixObservableTimeoutOperator$1.run(AbstractCommand.java:1121) ~[hystrix-core-1.5.6.jar:1.5.6]
	at com.netflix.hystrix.strategy.concurrency.HystrixContextRunnable$1.call(HystrixContextRunnable.java:41) ~[hystrix-core-1.5.6.jar:1.5.6]
	at com.netflix.hystrix.strategy.concurrency.HystrixContextRunnable$1.call(HystrixContextRunnable.java:37) ~[hystrix-core-1.5.6.jar:1.5.6]
	at com.netflix.hystrix.strategy.concurrency.HystrixContextRunnable.run(HystrixContextRunnable.java:57) ~[hystrix-core-1.5.6.jar:1.5.6]
	at com.netflix.hystrix.AbstractCommand$HystrixObservableTimeoutOperator$2.tick(AbstractCommand.java:1138) ~[hystrix-core-1.5.6.jar:1.5.6]
	at com.netflix.hystrix.util.HystrixTimer$1.run(HystrixTimer.java:99) ~[hystrix-core-1.5.6.jar:1.5.6]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_111]
	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) ~[na:1.8.0_111]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) ~[na:1.8.0_111]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) ~[na:1.8.0_111]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[na:1.8.0_111]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[na:1.8.0_111]
	at java.lang.Thread.run(Thread.java:745) [na:1.8.0_111]

2017-05-09 00:06:35.790  INFO 1915 --- [erListUpdater-0] c.netflix.config.ChainedDynamicProperty  : Flipping property: spring-boot-demo.ribbon.ActiveConnectionsLimit to use NEXT property: niws.loadbalancer.availabilityFilteringRule.activeConnectionsLimit =<PHONE_NUMBER>

Now, My primary problem is that. When I use the configure of time out of feign client. It does not work. My configure is below:
@Configuration
public class FeignConfigure {
     public static int connectTimeOutMillis = 12000;
     public static int readTimeOutMillis = 12000;
     @Bean
     public Request.Options options() {
            return new Request.Options(connectTimeOutMillis, readTimeOutMillis);
     }
}

The controller does not have any things except sleeping 500ms, the code is below:
@RestController
@RequestMapping(value = "/test")
public class HelloController {

    private static final Logger logger = LoggerFactory.getLogger(HelloController.class);

    @RequestMapping(value = "/hello", method = RequestMethod.POST)
    public String unit() {

        String response = "hello world hello world hello world hello world hello world hello world hello world ";
        logger.info("response: " + response);
        try {
            Thread.sleep(500);
        } catch (InterruptedException e) {
            e.printStackTrace();
        }
        return response;
    }
}

So, the very amazing thing is that when I configure the time out which is very small such as 10ms, it works. Because it will always time out, so it proves that my configure works. The code is below:
@Configuration
public class FeignConfigure {

    public static final int connectTimeOutMillis = 10;
    public static final int readTimeOutMillis = 10;

//    public static final int connectTimeOutMillis = 12000;
//    public static final int readTimeOutMillis = 12000;

    @Bean
    public Request.Options options() {
        return new Request.Options(connectTimeOutMillis, readTimeOutMillis);
    }
}

But when I configure long time of time out such as 12s, I expect that it will not appear time out. the configure is below:
@Configuration
public class FeignConfigure {

//    public static final int connectTimeOutMillis = 10;
//    public static final int readTimeOutMillis = 10;

    public static final int connectTimeOutMillis = 12000;
    public static final int readTimeOutMillis = 12000;

    @Bean
    public Request.Options options() {
        return new Request.Options(connectTimeOutMillis, readTimeOutMillis);
    }
}

The result is that, it still fails in the first time, and then OK. I have configured 12s!!! It means the configure  of time out does not work. So I am very confused. Is my configure is wrong?

Have your tried configuring the timeout via hystrix.command.default.execution.isolation.thread.timeoutInMilliseconds?

No, I dont try it. Because I am not sure that configure can configure more than one.  I have many API, So different API will configure different hystrix time of time-out. They are in the same application. If I configure via
hystrix.command.default.execution.isolation.thread.timeoutInMilliseconds?

in the yaml configure file, I think it will be global effective.

I try the configure in the yaml file via :
hystrix:
  metrics:
    enabled: true
  command:
    default:
      execution:
        isolation:
          thread:
            timeoutInMilliseconds: 12000

It works, but as I say above there are some APIs, diffrent API may want to have different time of time-out. So I think that the
hystrix.command.default.execution.isolation.thread.timeoutInMilliseconds

will be global effective. And I am confused too that what is difference of Request.Options and hystrix.command.default.execution.isolation.thread.timeoutInMilliseconds?

try   hystrix.command.<hystrixcommandkey>.execution.isolation.thread.timeoutInMilliseconds

Thanks, But I use the mode of feignclient and the hystrix is default enable, I dont use the mode of command. So, what can I get the hystrixcommandkey? Must I use mode of command?

The hystrixcommandkey takes the format of FeignClassName#methodSignature.  See #1864.

I am very thanks for your reply, now I have some other urgent things to do . I will try it on the weekend, and then feedback.

I've stumbled upon a similar issue, using Ribbon directly.
The first request always fails, no matter how high the time-out value is.
An issue has been logged, but I'm not sure it will get any response. Maybe it helps you figure out where to look.
https://github.com/Netflix/ribbon/issues/339

@TomVanWemmel this repo is for issues related to Spring Cloud Netflix.  Any issue related to Ribbon directly needs to be handled in the Ribbon repo.

I am going to close this issue due to lack of activity.  Please reopen if there is anything else we can do.

thanks, I have try it, it works.

dear all:
@Configuration
public class FeignRetry2 {
@Bean
Retryer feignRetryer() {
return new Default(1000, 5000, 5);
}
}

Hystrix command key takes >>FeignClassName#methodSignature, # is considered the beginning of a comment in application.properties, Then how it is is going to be loaded as Hystrix property?
Please explain...

Use quotes (single or double) around the key.


Fehlermeldung

[18:52:21] [Server thread/CRITICAL]: ErrorException: "file_put_contents(/home/bedwars/stats.json): failed to open stream: No such file or directory" (EXCEPTION) in "src/pocketmine/utils/Config" at line 241

For now, just create the directory /home/bedwars, later I will add an automatic folder creation

Reopen it


Baseclass.Contrib... 2.2.0-xbuild02	not found with VS2017

Hi everyone,
Is it expected that Windows build should use Baseclass.Contrib... 2.2.0-xbuild02?  A couple of projects in the solution reference it, however NuGet can't download it.  I've upgraded the projects to the more generic 2.3.0 version of the package, but would like to see why we are using the older one?

Yes we require it due to the views cross platform GUI web browser library we're using in linux (Gecko). It has a dependency on that older Baseclass library.


How do I store multiple aws secrets?

I have this issue but looks like this was supposed to be fixed? https://github.com/spulec/moto/issues/1893
conn = boto3.client("secretsmanager", region_name="us-west-1")
secrets = ["one", "two", "three"]
for secret in secrets:
   conn.create_secret(Name=secret, SecretString="sdfsdfsd")
mymodule.method_that_gets_value_of_bunch_of_secrets("one", "two", "three")

The method_that_gets_value_of_bunch_of_secrets() gets the value of several secrets and does stuff
create_secret() can only store one secret it looks like- or maybe Im using it wrong?
I want to create 3 secrets and have them available to my method
Im not using the list secrets method. I just want to store multiple secrets and call a method that gets their value

What version are you using, @red8888? The following code retrieves all three secrets in 1.3.16:
import boto3
from moto import mock_secretsmanager

with mock_secretsmanager():
	conn = boto3.client("secretsmanager", region_name="us-west-1")
	secrets = ["one", "two", "three"]
	for secret in secrets:
		conn.create_secret(Name=secret, SecretString="sdfsdfsd")

	secs = conn.list_secrets()
	print(secs["SecretList"])
	for secret in secrets:
		print(conn.get_secret_value(SecretId=secret))

What version are you using, @red8888? The following code retrieves all three secrets in 1.3.16:
import boto3
from moto import mock_secretsmanager

with mock_secretsmanager():
	conn = boto3.client("secretsmanager", region_name="us-west-1")
	secrets = ["one", "two", "three"]
	for secret in secrets:
		conn.create_secret(Name=secret, SecretString="sdfsdfsd")

	secs = conn.list_secrets()
	print(secs["SecretList"])
	for secret in secrets:
		print(conn.get_secret_value(SecretId=secret))

Closing this due to a lack of response - feel free to reopen if there's still an issue with this


Use the Custom AWS Logs integration with a non-AWS S3-compatible service (Linode Object Storage)

Logged into https://cloud.linode.com/object-storage/buckets with my existing account and created a new Object Storage bucked named mbranca-esf-logs:

Docs about how to set up an S3 client at https://www.linode.com/docs/guides/migrate-to-linode-object-storage/

I used the Frankfurt, DE (eu-central-1) region, so the bucket URL is https://mbranca-esf-logs.eu-central-1.linodeobjects.com/
The endpoint config setting to set up the custom AWS Logs integration is https://eu-central-1.linodeobjects.com (bucket URL minus the bucket name).

Another essential setting is the Access Key:

Time to run some tests.
Here's the aws-s3 input settings from the agent policy:
inputs:
  - id: aws-s3-aws_logs-afe56f1c-6312-411f-a8c8-b369327c943f
    name: aws_logs-1
    revision: 9
    type: aws-s3
    use_output: default
    meta:
      package:
        name: aws_logs
        version: 0.5.1
    data_stream:
      namespace: default
    package_policy_id: afe56f1c-6312-411f-a8c8-b369327c943f
    streams:
      - id: aws-s3-aws_logs.generic-afe56f1c-6312-411f-a8c8-b369327c943f
        data_stream:
          dataset: aws_logs.generic
        access_key_id: <REDACTED>
        secret_access_key: <REDACTED>
        parsers: null
        sqs.max_receive_count: 5
        max_bytes: 10MiB
        non_aws_bucket_name: mbranca-esf-logs
        max_number_of_messages: 5
        tags:
          - preserve_original_event
          - forwarded
        publisher_pipeline.disable_host: true
        file_selectors: null
        endpoint: 'https://eu-central-1.linodeobjects.com'
        bucket_list_prefix: 2023-02-14-13-41-08-79BF7A8FA7821B47_D
        number_of_workers: 5
        sqs.wait_time: 20s
        bucket_list_interval: 120s

I uploaded a couple of file into the bucket:

And here is the end result in Elasticsearch:

The test is successful!


binary ninja: optimize the order of computing LLIL to partially address #2402

ref #2402
This PR does two things for the Binary Ninja backend:

build the call graph up front, which is more cache friendly, and
fetch instruction LLIL once and provide it via the instruction handler context.

With these two changes, against 321338196a46b600ea330fc5d98d0699.exe_ capa analysis time drops from 232s to 191s, a savings of around 18%.
@xusheng6 please review.
Checklist

[ ] No CHANGELOG update needed

[x] No new tests needed

[x] No documentation update needed

there's also a pending opportunity to reduce the number of mlil objects fetched, which may save another 10s or so. we read mlil to compute basic blocks during feature extraction, and then again to compute the file layout.

there might be something else we can do around enumerating instructions, which takes around 14s (7%), but otherwise there's not many obvious hotspots:

Thx for the PR! I will have a look at it ASAP

@williballenthin Below are some of my recent thoughts, basically I checked the places where I need to access the IL of the functions:

In is_stub_function, the code checks whether a function is a stub function to another function or imported API by examining the LLIL. The case can be seen in al-khaser_x64.exe_ at 0x14004b532.

This should really be just implemented in binja's core analysis. In other words, binja should detect this is a stub function and treat it as such, so that the feature extractor does not need to bother with that.
Also, from your perspective, how often do you see a debug build like this when you analyze malware? If this is often enough, maybe we should actually prioritize this internally.
Even before we get to a proper fix, we could first do a pass on all of the functions and calculate and cache the result of is_stub_function

In extract_function_calls_to, I had to check the LLIL instruction that actually makes the xref to exclude some false positives. In fact, the caller_sites property of a Function object almost returns the things it the exactor wants, but I noticed a false positive during development. For example, if another function has some code like push func1, then the caller_sites of func1 will also include the push func1 instruction, although it is not technically a call to func1.

This is actually a known issue in https://github.com/Vector35/binaryninja-api/issues/3722. However, adding the extra check in the implementation of the property does not really help much, since it just moved the calculation from in capa to binja's API.
A possible workaround is to relax the restriction here, and just allow things like non-calls to be included in it. The reason is, even if another funciton does not call this function explicitly, having a xref to the start of the function still means they have very close relationship, and even potentially using it as a callback or sth. If we want this, we will need to change the unit test for binja a bit to make it pass again

In get_basic_blocks, I am matching the disassembly basic block with the MLIL basic block. The MLIL basic block is later used for stack string detection. I know this is probably where the worst part of the binja extractor xD. In fact, I can directly know the existence of the stack string at the function level by checking things like builtin_strcpy, etc. It is in fact a detour to match the MLIL bbl with the disassembly bbl, only because capa wants that info at the basic block level.

I am curious if we can change that a bit to be more flexible. I know other backends probably cannot tell this at the function level, but for binja, it is very straightforward

Also, from your perspective, how often do you see a debug build like this when you analyze malware?

It's not uncommon that we see this for samples we analyze manually - I'd guess 10-20%. However, looking at larger scale analysis (e.g. from last week) it's more around 1% (using the capa debug build match results).

Also, from your perspective, how often do you see a debug build like this when you analyze malware?

It's not uncommon that we see this for samples we analyze manually - I'd guess 10-20%. However, looking at larger scale analysis (e.g. from last week) it's more around 1% (using the capa debug build match results).

Thx for the valuable info! Also, are there any cases where the stub/thunk function take another form besides the normal one where it is just a jmp to the real function?
Also, I just realized that I can actually directly get the LLIL of a particular instruction at a given address without first retrieving the IL function it belongs to. This means I no longer need to retrieve the IL of any other functions during the analysis of one function. This could bring drastic performance!
Also, the stack string detection does not have to be done using MLIL. Previously, I already had an implementation of it without using the IL (just like what other backend does). This could even make the binja extractor work without accessing the IL of any function, which could make things even faster. For this, I think we can gate it behind a setting.

i will rebase and see if there's any reason to continue these optimizations. without data, i'm not yet sure.

did the other changes supersede this PR?

I do not think https://github.com/mandiant/capa/pull/2511 alone can do it, but the combination of these three can probably do:

https://github.com/mandiant/capa/pull/2511
https://github.com/mandiant/capa/issues/2520
https://github.com/mandiant/capa/issues/2516

Even with all these done, it is still meaningful to see if this PR can push the performance further. However, I recommend @williballenthin to only start working on it after I have finished all changes from my side

Hi @williballenthin , finally got some to look at your changes! It seems to me that binja: provide llil to instruction handlers via ctx implements the idea described in https://github.com/mandiant/capa/issues/2520 and it should also hopefully fix https://github.com/mandiant/capa/issues/2517. Could you please rebase that commit on top of the latest master? You can keep https://github.com/mandiant/capa/pull/2509/commits/999f91baa1c5a4aa842465fda786ab8cc41f288a and https://github.com/mandiant/capa/pull/2509/commits/a909d022a6f5e0a9160df6ba4a8c1e17a5d18b6c if you would like to since I feel like they will be quite useful. Besides, you may wish to create a new branch because despite I think the other 3 commits are no longer useful on top of the latest master, we would better still keep it in case we want it later


Rulesgen for v3 packs

Peaked at Slashies Splatoon pack and updated rulesgen to match v3 rules.
Run in the same folder as log.txt, (cemu root), default is set to x1 scaling to match new rule style.
rules.txt has been updated, gitrules.txt hasn't.. waiting for the final build rules.
updated, typo fix template
rulesGen.zip

Hmm something is funky with the scaling.. this WIP3 with old graphics pack.
Similar results if i try to scale with new rules, wonder if need to account for full GX2SetScissor ?
Also noted that rulesGen incorrectly scales 1920x1080.. I just used the 1280 as a template in the header instead of actually fetching values :/   Will be fixed if I can get things to scale correctly..

Yup, just rewrote the 360 -> 368 rule.. and now it scales correctly..   Why would old packs use the new GX2SetScissor logic?
Then again, most of the GX2SetScissor calls probably round to the same number, so a search replace for all "new" resolutions will probably fix the issue.

aah, now I get it, there's one res for the 8 rounding and one for the viewport, and both need to be defined.
[TextureRedefine]
width = 448
height = 240
formats =  0x820
overwriteWidth = ($width / $gameWidth) * 428
overwriteHeight  = ($height/$gameHeight) * 240
[TextureRedefine]
width = 426
height = 240
formats =  0x820
overwriteWidth = ($width / $gameWidth) * 426
overwriteHeight  = ($height/$gameHeight) * 240
[TextureRedefine]
width = 320
height = 192
#formats =  0x816,0x01a
overwriteWidth = ($width / $gameWidth) * 320
overwriteHeight  = ($height/$gameHeight) * 192

[TextureRedefine]
width = 320
height = 180
#formats =  0x816,0x01a
overwriteWidth = ($width / $gameWidth) * 320
overwriteHeight  = ($height/$gameHeight) * 180

width = 448
height = 240
formats =  0x820
overwriteWidth = ($width / $gameWidth) * 448
overwriteHeight  = ($height/$gameHeight) * 240```

Yeah. I just want to share it in here since you might not know about what exactly happened yet.

So, basically, it's a pretty major update and this new repository should be used as a rework since we won't be able to just ship old packs with some variables changed.
Also, just wanna mention that Nsight also now shows texture information from Cemu so you can easily track which viewport or texture you're looking at. This is very handy for tracking the exact texture you're looking at in Nsight in the file dumps or in the new Texture Cache window in Cemu.
Some additional information about how I use these features:
First of all, you find a normal viewport in Nsight. I look what physical address this viewport has and search it in the Texture Cache window.

Since they share the same physical address you can assume that the viewport size got accessed with a different resolution.
Just wanna mention that textures that show as mipmap's in Nsight are unrelated to this, that should just use old behavior (scaling mip00 should scale all of them just fine).
Also, feel free to edit/correct me or something, it's probably pretty unwelcoming stuff for anyone that would consider making a graphic pack themselves so I think we probably should make a wiki page like we had previously.

It's more work,  but the ability to see previous lookup is sooo much better than what we had before.
I'd say it's even easier for someone new to understand.

For most packs overwrite all calls not related to  0x31,0x32,0x33,0x34,0x35,0x431,0x432,0x433,0x434,0x435
Skip stuff that's 1:1 256x256
Skip anything below 160x90

And you should have a good start... the rest is just, open texture cache, check what formats are left, test, repeat.  We should have been nagging Exzap for better logging earlier, would have saved us a bunch of time on the BOTW stuff.
Unfortunately there's no way to get the rounding part form logs right now, so rulesgen only gets you the basics.

What about vs/ps stuff? I don't see any way that they would extend the variables from rules.txt
What exactly do you mean?

I assume we want shaders to scale to the res the user selects.
R1f.y = R0f.y + intBitsToFloat(uf_remappedPS[0].y) / resXScale;
We ca do sizeOf lookups.. but getting resXScale from rules.txt would be nicer..
Like
[Preset]
name = Scale
$resXScale  = width  / gameWidth

Ah, yeah, that's indeed not supported. But, arithmetic stuff shouldn't be so difficult in OpenGL (unless we're dealing with preprocessor declarations like #define and such)
Also some noteworthiness information that might be handy in this situation is that by default it outputs doubles in the shaders and int's in the rules.txt file's. You can force ints in shaders by adding a :int like this in the preset declaration:
[Preset]
name = Scale
$resXScale:int = 300

It's all very simple "typing".

I can't exactly find this "sizeOf" you're talking about, can you elaborate.

I haven't test it but maybe  textureSize(textureUnitPS0, 0) and/or uf_fragCoordScale would give us the actual res, not emulated.

Both would result into their viewport resolution, pretty sure.

This doesn't work as expected, sticking it at the top rules.txt just disables all viewport rules
Anyone who knows if overwriteRelativeLodBias syntax has changed?
[TextureRedefine] #lod override, shimmer and possible sharper textures
formats  = 0x01a,0x31,0x32,0x33,0x34,0x35,0x431,0x432,0x433,0x434,0x435
overwriteRelativeLodBias = -8 #

remove 0x1a

Breaks either way
[TextureRedefine] #lod override, shimmer and possible sharper textures
formats  = 0x433
overwriteLodBias = -8 #

[TextureRedefine] #lod override, shimmer and possible sharper textures
formats  = 0x433
overwriteRelativeLodBias = -8 #

Looks like it's unsigned int only :(
This just  throws

[18:31:11] graphicPack: can't parse preset "LOD -4": not enough parameters for equation

As soon as you asign anything but positive values
[Definition]
titleIds =<PHONE_NUMBER>1ED700,00050000101ED800,0005000010131D00
name = Negative texture LOD bias
path = "Tokyo Mirage Sessions FE/Graphics/Negative Texture LOD"
description = Texture LOD override, possible shimmer but sharper textures
version = 3

[Preset]
name = LOD -4
$LOD = (-4)

[Preset]
name = LOD -1
$LOD = (-4)

[Preset]
name = LOD -2
$LOD = (-4)

[Preset]
name = LOD -4
$LOD = (-4)

[Preset]
name = LOD -8
$LOD = (-4)

[Preset]
name = LOD -8
$LOD = (-16)

[TextureRedefine]
formats = 0x01a,0x31,0x32,0x33,0x34,0x35,0x431,0x432,0x433,0x434,0x435
overwriteRelativeLodBias = $LOD

copy paste from pull request. Maybe you guys already have settled this?
Since we’re going towards a PC config style with resolutions..why not continue along that line? Now that we can expand the menu settings per game, it won’t be as crowded. I figured anything that changes graphics sorts under graphics, PC game style.
When you expand graphics it easy to see what's active. This would include clarity, no-Gui, shadows, wide screen etc
I would separate work arounds and gameplay settings as they potentially break stuff when active. Nvida/AMD fixes, 60 fps, cheats
So I end up with game/graphics/... and game/modification or gameplay (or some better name)

Hm, the less drop downs the better. I think we should probably stick with what we've got right now.

Thinking about it I'd prefer a flat structure..

No need to expand a bunch of menus, easy to debug whats active. With the exception of BOTW, it should be simpler (keep that as a separate workaround folder).   If you need further info about the option, the description box should cover that.

Looks like Ninja Gaiden is testing the classic comment injection on cemu!  ;)

I do hope Exzap has salted his keys

I'd say that's a promising start...

Are you guys running into general instability ?
Running a very basic rules.txt only scaling 1920x1080 I find that simply launching a game with resolution pack hangs like 1/5 of the time.. Or if I let the intro demo of twilight princess run for about 10 minutes, it usually hangs when looping?

Hm, I've noticed not that much myself.


[SEC-F10] Buyer could make the seller lose money by requesting refunds

Observation
verify_invoice stores the balance that was paid in the amountPaid field in the invoice:

balance is stored in the invoice
but what was actually transferred is balance - 10000 , where 10’000 is the transfer fee.

In refund_invoice, it is possible to refund amount == amountPaid .
This results in a situation where the seller refunds 10’000 more than what was actually received from the buyer.
The buyer will have no cost here, as they receive everything back, including the fee they paid.
Risk Description
If a buyer can repeatedly do this (e.g. in the likely case that refunding is automated), they can drain money from the seller (10’000 per iteration) at no cost for the buyer.
This would require 10’000 iterations to drain 1 ICP from the seller.
Recommendations

Make sure the buyer pays the fee for the transfer.
Alternately, remove the refund feature

Resolved by #30


RabbitMQ password

It seems like the ability to set a fixed password for RabbitMQ disappeared sometime between Preview 3 and later nightly builds. The (generated) password on initial startup (no volumes/new volume) is correctly exposed in connection string, but any subsequent runs results in clients not being able to connect, as the password seems to change across reboots.
So, two questions:

Why was the ability to set a fixed password removed?
Is the behavior above expected, and if so, does any remedy exist when connecting from clients?

Its back in preview 5. Here is an example:
var rabbitMqPassword = builder.AddParameter("rabbitmq-password", secret: true);
var messaging = builder.AddRabbitMQ("messaging", password: rabbitMqPassword)
                       .WithDataVolume()
                       .WithManagementPlugin()
                       .PublishAsContainer();

Try it out and let us know if it works for you.

Its back in preview 5. Here is an example:
var rabbitMqPassword = builder.AddParameter("rabbitmq-password", secret: true);
var messaging = builder.AddRabbitMQ("messaging", password: rabbitMqPassword)
                       .WithDataVolume()
                       .WithManagementPlugin()
                       .PublishAsContainer();

Try it out and let us know if it works for you.

Great!
(Preview 5 isn't out yet, right? - has the "release build" been cut on nightlies? If so - which one :D)

Preview 5 will be released VERY soon :)


Fail to start PowerShell session with PowerShell 6.2.x installed

OS: Mac
Build Version: 20200706.1
Repro Steps:

Install PowerShell 6.2.6.
Create a PowerShell project with HttpTrigger function.
Open the .ps file of this project.
Check the PowerShell session.

Expect:
PowerShell session shows successfully with 6.2 version.
Actual:
Keep "Starting PowerShell..." state, an error occurs when clicking it.

More Info:

Can't debug the PowerShell project due to this issue.
This issue does not reproduce with 7.0.2 PowerShell installed, but PowerShell 7 isn't supported yet.

That error is coming from the PowerShell extension. We don't control the PowerShell session


Adding Default QR Code

At the start, the QR Code Section is empty, So we could add some default QR Code, that may play some animation of the brand name (Shuffle Pay) or anything else

👋 Thanks for your feedback! @vedprakashnautiyal. Would you like us to assign this task to you, or would you prefer for our team to handle it? 👍

Let's wait some more discussion on this issue.....!!


Default Qargs "expandSubcollections=true" and "all-options" should be included in many HTTP transactions

Including "expandSubcollections=true" will change how we implement get_collections which will in turn change the implementation of the RESTWalker

We've never successfully verified that this should be the case.   We have implemented an appropriate mechanism ("requests_params") to submit qargs if we decide to.  Since we aren't sure about making these universal I'm closing this as "wontfix".