LMDB vs LevelDB

LMDB is an alternative to LevelDB which stands out to me specifically due to their claimed performance gains in read/writes of large values. Since SSB frequently works with JSON objects in the 100b-1kb range, we may find the performance difference to be important. LMDB also supports multi-thread concurrency with a global write-lock, which may be able to provide throughput gains if used correctly. The LMDB node binding seems to be in decent shape (notable bugs, https://github.com/Venemo/node-lmdb/issues/25 and https://github.com/Venemo/node-lmdb/issues/31).
Comparatively, LevelDB has more active use in the node community, and seems therefore like a more well-tested option. I recommend building a fork of SSB which uses LMDB so that we can compare their respective performance and investigate stability & portability.

@pfraze since we have leveldown shaped bindings for lmdb https://www.npmjs.com/package/lmdb
we actually just drop it into the current ssb... the "fork" would only be a few lines different... what we'd need though, is a benchmark suite.

ah, handy


Already running Containers not responding during scale out.

While updating the replica count, during the deployment the already running container become unresponsive.

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 6ac12ec3-526c-071d-184e-c7988a104db4
Version Independent ID: 342a7347-6e3a-4a3c-b6d7-9996f61a291a
Content: Tutorial- Scale an app running in Azure Service Fabric Mesh
Content Source: articles/service-fabric-mesh/service-fabric-mesh-tutorial-template-scale-services.md
Service: service-fabric-mesh
GitHub Login: @rwike77
Microsoft Alias: ryanwi

Thanks for the feedback! I will try to reproduce your issue and share the update.

@vikramdadwal Apologize for the delay on this. Are you still seeing this issue?
Can you please elaborate on the issue you are seeing and provide error messages if any. Thanks :)

We will now close this issue. If there are further questions regarding this matter, please tag me in a comment. I will reopen it and we will gladly continue the discussion.


Refunded payment

I did all the procedures with the transparent checkout, and charged correctly, cashed on the card, everything was ok.
However 7 days later the payment was extorted by the credit card operator, I contacted to see, they said that the purchase was missing. that it was approved but was not captured.
On other platforms there is the method of sending and then the capture, I found nothing here in the library, and in the pagseguro documentation there is only one URL for capture that this discontinued does not work the call, can someone help me is URGENT!

abri a mesma issue em portugues


Inference Graph supports Raw Deployment

/kind feature
isvc supports both Raw Deployment and Serverless, but IG only supports Serverless (Knative service), it is better to add supporting for Raw Deployment.

Please assign this issue to me.

/assign bmopuri

/assign @rachitchauhan43


Expose hipchat notification text as a template config

I just started experimenting with hipchat notifications, and it'd be great if we could customize the message sent to hipchat (e.g. include a link to the commit in drone, convert emails to hipchat @nicknames, etc.). I think most of this would be solved if we could provide a template in the config.

most notification templates, including hipchat, now support custom message templates in handlebars format, for example:
notify:
  hipchat:
    template: |
      {{ build.number }} finished with status {{ build.status }}


IBM Watson Connection Details Not Working

I am trying to connect to an IBM Watson Assistant of my own and I am having difficulties.
Since you have created the constants defined in the.env file, the IBM UI has probably changed as I can find no credentials with any of the following names:
CONVERSATION_USERNAME
CONVERSATION_PASSWORD
WORKSPACE_ID
I attach a redacted screenshot showing the IBM Watson API credentials I have used for each of these constants (I am assuming their naming conventions have just changed slightly), but I am still having trouble with Watson authentication.

Below is what I am returned in the watsonError property of each message:
  watsonError:
   { Error: Unauthorized: Access is denied due to invalid credentials.
       at Request._callback (/Users/joeharvey/Development/watson-bot/node_modules/watson-developer-cloud/lib/requestwrapper.js:102:21)
       at Request.self.callback (/Users/joeharvey/Development/watson-bot/node_modules/watson-developer-cloud/node_modules/request/request.js:185:22)
       at Request.emit (events.js:182:13)
       at Request.<anonymous> (/Users/joeharvey/Development/watson-bot/node_modules/watson-developer-cloud/node_modules/request/request.js:1157:10)
       at Request.emit (events.js:182:13)
       at Gunzip.<anonymous> (/Users/joeharvey/Development/watson-bot/node_modules/watson-developer-cloud/node_modules/request/request.js:1079:12)
       at Object.onceWrapper (events.js:273:13)
       at Gunzip.emit (events.js:187:15)
       at endReadableNT (_stream_readable.js:1098:12)
       at process.internalTickCallback (internal/process/next_tick.js:72:19)
     cause:
      { Error: Unauthorized: Access is denied due to invalid credentials.
          at Request._callback (/Users/joeharvey/Development/watson-bot/node_modules/watson-developer-cloud/lib/requestwrapper.js:102:21)
          at Request.self.callback (/Users/joeharvey/Development/watson-bot/node_modules/watson-developer-cloud/node_modules/request/request.js:185:22)
          at Request.emit (events.js:182:13)
          at Request.<anonymous> (/Users/joeharvey/Development/watson-bot/node_modules/watson-developer-cloud/node_modules/request/request.js:1157:10)
          at Request.emit (events.js:182:13)
          at Gunzip.<anonymous> (/Users/joeharvey/Development/watson-bot/node_modules/watson-developer-cloud/node_modules/request/request.js:1079:12)
          at Object.onceWrapper (events.js:273:13)
          at Gunzip.emit (events.js:187:15)
          at endReadableNT (_stream_readable.js:1098:12)
          at process.internalTickCallback (internal/process/next_tick.js:72:19)
        code: 401,
        error: 'Unauthorized',
        body: 'Unauthorized',
        'x-global-transaction-id': '7ecac92c5c091449e4751229' },
     isOperational: true,
     code: 401,
     error: 'Unauthorized',
     body: 'Unauthorized',
     'x-global-transaction-id': '7ecac92c5c091449e4751229' } }

Can anyone provide an insight into what the correct mapping of the credentials is and what I am doing wrong?

I am told via the Watson Development Community Slack channel:

We've moved from basic auth using username/pass, to iam using apikey.

Which probably explains why my assistants (created within the last two weeks) are unable to authenticate using this codebase.
Would that be accurate?

Hi,
Did you find the answer?


Kostal Pico in Master/Slave configuration

Hello,
is it also possible to integrate a Master/Slave configuration where both devices have a seperate IP-Adresse?

Hey,
mhm I have no information about how that works in your Inverter.
You could run the plugin twice an get the values of both inverters, if that is what you want?

Hi,
I tried to run it twice (i added the other inverter with IP-Adress). Unfortunately it is not possible as I can't assign a name to each instance.

Thanks a lot - I had also to replace the Sensor name from "Kostal Piko xxx" to "Kostal Piko Slave xxx" in the const.py file. To Identify the Integration I changed in the manifest.json file the name of teh integration to "Kostal PIKO Solar Inverter Slave"
{
"domain": "kostal_piko_slave",
"name": "Kostal PIKO Solar Inverter Slave",
"documentation": "https://github.com/sla89/hassio-kostal-piko",
"issue_tracker": "https://github.com/sla89/hassio-kostal-piko",
"requirements": [],
"codeowners": ["sla89"],
"version": "1.10",
"iot_class": "local_polling"
}

So it is working for you? :)

Yes - perfect!


Update documentation : fix spnego properties names

 Thank you for your submission, we really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.


bug: 500 error when bench limit-count plugin

limit count conf:
policy: redis
count: 100
time_window: 60
key: remote_addr
os: CentOS 7.6
benchmark tool: apache benchmark
I got runtime error which result in 500 response when i bench 1million request, 1k concurrency by ab.It seems red:ttl(key) return nil in limit-count-redis.lua. The follow snippet is runtime error detail:
lua entry thread aborted: runtime error: .../apisix/apisix/plugins/limit-count/limit-count-redis.lua:76: attempt to compare nil with number
stack traceback:
coroutine 0:
        .../apisix/apisix/plugins/limit-count/limit-count-redis.lua: in function 'incoming'
        /usr/local/apisix/apisix/plugins/limit-count.lua:121: in function 'phase_fun'
        /usr/local/apisix/apisix/init.lua:147: in function 'run_plugin'
        /usr/local/apisix/apisix/init.lua:385: in function 'http_access_phase'
        access_by_lua(nginx.conf:161):2: in main chunk

Is it a bug? @membphis @moonming

yes, that is a bug. we need to check the return value first.
welcome PR ^_^

Another issue: in my bench scenario, the redis connections is much more than pool size, there are many timewait , close wait connections, Is pool connection is not reusable long connection? BTW, my pool size is 100 and keepalive timeout is 10 minutes.

one issue focus on one thing. if you have another question, please create a new issue.

I will deal with this problem.


error: comparison between two arrays is deprecated in C++20 [-Werror=array-compare] (IDFGH-11161)

Answers checklist.

[x] I have read the documentation ESP-IDF Programming Guide and the issue is not addressed there.
[x] I have updated my IDF branch (master or release) to the latest version and checked that the issue is present there.
[x] I have searched the issue tracker for a similar issue and not found a similar issue.

IDF version.
esp-idf-v5.1.1
Espressif SoC revision.
ESP32
Operating System used.
Linux
How did you build your project?
Command line with idf.py
If you are using Windows, please specify command line type.
None
Development Kit.
Custom Board
Power Supply used.
USB
What is the expected behavior?
i want to write a Test-Case to compare 2 Strings
TEST_CASE("string", "[string]")
{
    char q[] = "GeeksforGeeks";
    setString("CF-A", "string", q);
    char r[] = "Hello";
    TEST_ASSERT_NOT_EQUAL(q, r);
    ESP_LOGI(TAG, "before: %s %s", q, r);
    getString("CF-A", "string", "unknown", r, 33U);
    ESP_LOGI(TAG, "after: %s %s", q, r);
    TEST_ASSERT_EQUAL_STRING(q, r);
}

What is the actual behavior?
Compile Error see below.
Debug Logs.
In file included from /home/franz/esp-idf-v5.1.1/components/unity/unity/src/unity.h:16,
                 from /home/franz/git/zas/IHMKratosCAN/Application/components/Settings/test/test_settings.cpp:13:
/home/franz/git/zas/IHMKratosCAN/Application/components/Settings/test/test_settings.cpp: In function 'void test_func_48()':
/home/franz/esp-idf-v5.1.1/components/unity/unity/src/unity.h:134:130: error: comparison between two arrays is deprecated in C++20 [-Werror=array-compare]
  134 | #define TEST_ASSERT_NOT_EQUAL(expected, actual)                                                    UNITY_TEST_ASSERT(((expected) !=  (actual)), __LINE__, " Expected Not-Equal")
      |                                                                                ~~~~~~~~~~~^~~~~~~~~~~~
/home/franz/esp-idf-v5.1.1/components/unity/unity/src/unity_internals.h:710:102: note: in definition of macro 'UNITY_TEST_ASSERT'
  710 | #define UNITY_TEST_ASSERT(condition, line, message)                                              if (condition) {} else {UNITY_TEST_FAIL((UNITY_LINE_TYPE)(line), (message));}
      |                                                                                ^~~~~~~~~
/home/franz/git/zas/IHMKratosCAN/Application/components/Settings/test/test_settings.cpp:53:5: note: in expansion of macro 'TEST_ASSERT_NOT_EQUAL'
   53 |     TEST_ASSERT_NOT_EQUAL(q, r);
      |     ^~~~~~~~~~~~~~~~~~~~~
/home/franz/esp-idf-v5.1.1/components/unity/unity/src/unity.h:134:130: note: use unary '+' which decays operands to pointers or '&'view_convert_expr' not supported by dump_decl<declaration error>[0] != &'view_convert_expr' not supported by dump_decl<declaration error>[0]' to compare the addresses
  134 | #define TEST_ASSERT_NOT_EQUAL(expected, actual)                                                    UNITY_TEST_ASSERT(((expected) !=  (actual)), __LINE__, " Expected Not-Equal")
      |                                                                                ~~~~~~~~~~~^~~~~~~~~~~~
/home/franz/esp-idf-v5.1.1/components/unity/unity/src/unity_internals.h:710:102: note: in definition of macro 'UNITY_TEST_ASSERT'
  710 | #define UNITY_TEST_ASSERT(condition, line, message)                                              if (condition) {} else {UNITY_TEST_FAIL((UNITY_LINE_TYPE)(line), (message));}
      |                                                                                ^~~~~~~~~
/home/franz/git/zas/IHMKratosCAN/Application/components/Settings/test/test_settings.cpp:53:5: note: in expansion of macro 'TEST_ASSERT_NOT_EQUAL'
   53 |     TEST_ASSERT_NOT_EQUAL(q, r);
      |     ^~~~~~~~~~~~~~~~~~~~~
cc1plus: some warnings being treated as errors
ninja: build stopped: subcommand failed.

More Information.
No response

@franz-ms-muc
Macro
TEST_ASSERT_NOT_EQUAL(q, r);

expands to generic != comparison:
UNITY_TEST_ASSERT(((expected) !=  (actual)), __LINE__, " Expected Not-Equal")

And this is why the compiler is comiling.
You can override this and compare two string pointers by casting char[] to char*:
TEST_CASE("string", "[string]")
{
    char q[] = "GeeksforGeeks";
    setString("CF-A", "string", q);
    char r[] = "Hello";
    TEST_ASSERT_NOT_EQUAL((char*)q, (char*)r);
    ESP_LOGI(TAG, "before: %s %s", q, r);
    getString("CF-A", "string", "unknown", r, 33U);
    ESP_LOGI(TAG, "after: %s %s", q, r);
    TEST_ASSERT_EQUAL_STRING(q, r);
}

If you have a look at the description of TEST_ASSERT_NOT_EQUAL() here you can see that it compares integers only. You can not give arrays into it, this is why you got this build error.
Using TEST_ASSERT_EQUAL_STRING is correct here.
Closing it for now, feel free to reopen it if there are any questions. Thanks.


Mag/baro rates in SITL unrealistic

SITL gazebo sensor rates are very different from real hardware. Baro, mag etc are all published at 250Hz, the same as the gyro/accel, instead of ~50Hz as real sensors would do.
Why is this an issue?

It makes eg. profiling difficult.
Vehicles behave differently

That indeed seems like a pretty significant thing to change to improve SITL. @TSC21 publishing every 5th measurement seems like a trivial change and it would be great to see this being executed swiftly.

Tomorrow is done.


Users/ajya/azurestacktasksupport

@Ajay-MS , I could see many package-lock.json in Tests folder of Task.
Is it actually required ?


Dynamic log label API

This PR introduces dynamic log labelling and a Rails-like API for configuration them for Rack applications.
Label values can take the form of either static strings or a
user-defined function (any object that responds to #call) which is
passed the current Rack request environment at the moment of generating
the log entry.
The label key remains static and the return value of the function
becomes the label value in the log entry.
The tests show an example of writing labelling log entries with a value
from an HTTP header.
We use this in production to tie together all traces from a user session.

@bestie Thanks for submitting the PR. The changes look great, just left couple comments on the change.

My pleasure, thanks for the fast feedback.
All comments seem like good ideas to me, will make changes ASAP.


1.20.1 Midnight Controls - 1.8.2 GUI

Describe the bug
Cursor's are trash and still behind the GUI, if used with immersive controls the controls don't work as well. The inventory button GUI With REI mod isn't able to be selected.
To Reproduce
Steps to reproduce the behavior:

Turn on virtual mouse
nothing works after that invisible inventory and so forth.

Expected behavior
a usable stable mod but is inoperable on version 1.20.1 which has 90% of the most current mods on it.
Desktop (please complete the following information):

OS: Windows
Minecraft [e.g. 1.20.1]
Fabric [e.g. fabric 0.14.2]
Version [e.g. 1.20.1]
Branch [e.g. Release 1.8.2]

Additional context
Please fix this i need this mod for my controller players!, you can't use the most up to date version because its on 1.20.2!

Fixed :)


Subchannel Sharing

@dgquintas - heads up

I am actively reviewing this pull request and will update once I am done with the review. I wish github had a feature where I can save my comments as drafts and publish them all at once..

I am done with my review. Other than a few questions, I do not have any comments.

This has merge conflicts now (sorry, I took too long to merge). Can you resolve the conflicts and resend the PR ?

Updated. A conflict in avl.h of all places!


Finish backend with Django and SQLite

This is an optional backend if MongoDB and Express do not work.

To install dependencies. Do the following:
cd backend
python3 -m venv .env
pip install -r requirements.txt

@jayphan14  @Justin-Jingcheng-Peng, I have finished all functionalities for the app. There's some stuff to do left like design and a few CSS-related stuff. But all the functionalities needed should work now.


How to use Google Access Token to call Google Drive API

I’m using Firebase Authentication mechanism and the scope I have provided https://www.googleapis.com/auth/drive
While running application, it asked for user login along with user's consent for Google drive access. After successful authentication it returns a Firebase Id Token.
In the next step application requires to access user's Google Drive. As I know Google Drive API requires OAuth Access token while application has Firebase Token.
https://www.googleapis.com/drive/v3/files
My questions is: How to generate OAuth Access token which is capable to access user's Google drive? OR Is there any alternate solution for this?

@Doppelklick I'm using Firebase Plugin in NativeScript. Plugin returns access token which I'm using to call Google Drive APIs.


allow more geometry types for search

It seems that right now only Polygon (POST) and bbox (GET) are supported
https://github.com/arturo-ai/arturo-stac-api/blob/7d4c9572981e935de2521441878f3ffb78f6b9b7/stac_api/clients/postgres/core.py#L321-L327
https://github.com/arturo-ai/arturo-stac-api/blob/master/stac_api/models/schemas.py#L228-L235
the STAC API specs says:

Searches items by performing intersection between their geometry and provided GeoJSON geometry. All GeoJSON geometry types must be supported.
ref:
https://github.com/radiantearth/stac-api-spec/blob/f64a08235cb0ae04dfdb37bd8d6940c3814d057c/item-search/README.md#query-parameter-table

cc @kylebarron @geospatial-jeff

@vincentsarago I don't think is an issue. I added some test cases with https://github.com/stac-utils/stac-fastapi/pull/264 to show that other geojson geometries are working. Maybe this was fixed at some point? I think that naming this poly: poly = shape(search_request.intersects) is confusing.

Yes I think this was fixed


Better mesh representation in NGE

Please make sure to label your PR with "bug", "new feature" or "breaking change" label(s).
To prevent this PR from going to the changelog marked it with the "skip changelog" label.

Snapshot stored with reference name:
refs/pull/14188/merge
Test environment:
https://babylonsnapshots.z22.web.core.windows.net/refs/pull/14188/merge/index.html
To test a playground add it to the URL, for example:
https://babylonsnapshots.z22.web.core.windows.net/refs/pull/14188/merge/index.html#WGZLGJ#4600
Links to test babylon tools with this snapshot:
https://playground.babylonjs.com/?snapshot=refs/pull/14188/merge
https://sandbox.babylonjs.com/?snapshot=refs/pull/14188/merge
https://gui.babylonjs.com/?snapshot=refs/pull/14188/merge
https://nme.babylonjs.com/?snapshot=refs/pull/14188/merge
To test the snapshot in the playground with a playground ID add it after the snapshot query string:
https://playground.babylonjs.com/?snapshot=refs/pull/14188/merge#BCU1XR#0


build: add false-positive for CVE-2024-24828

PR summary
Add false-positive for CVE-2024-24828, and bump our version of babel to address a potential vulnerability.
PR Checklist
General checklist
Please make sure that your PR fulfills the following requirements:

[ ] The commit message follows the Angular Commit Message Guidelines.
[ ] Tests for the changes have been added (for bug fixes / features)
[ ] Docs have been added / updated (for bug fixes / features)
[ ] Dependencies have been updated as needed
[ ] .secrets.baseline updated as needed?

Checklist for adding a new validation rule:

[ ] Added new validation rule definition (packages/ruleset/src/rules/*.js, index.js)
[ ] If necessary, added new validation rule implementation (packages/ruleset/src/functions/*.js, updated index.js)
[ ] Added new rule to default configuration (packages/ruleset/src/ibm-oas.js)
[ ] Added tests for new rule (packages/ruleset/test/*.test.js)
[ ] Added docs for new rule (docs/ibm-cloud-rules.md)

:tada: This PR is included in version 1.15.4 :tada:
The release is available on npm package (@latest dist-tag)
Your semantic-release bot :package::rocket:

:tada: This PR is included in version 1.16.4 :tada:
The release is available on npm package (@latest dist-tag)
Your semantic-release bot :package::rocket:


Add subscription heartbeats

While we do take many precautions we are still occasionally leaking connections - an example redeploy of Socketshark showed 10% of DMS subscriptions are stale (e.g. do not have a corresponding subscription in Socketshark), and that's on top of us deleting subscriptions after 7 days.
This PR introduces an optional heartbeat mechanism so that the service can keep track of heartbeats and do its own cleanup if it wants to.

Will add some docs and release a new version once it's tested in production.


#839- Fixed the Recover Password mobile page

Fixed the layout of the Recover Password page. Users can see the button when they use the keyboard.
Changed:
-Login.js
-Heading.js

Hi @mel-chiu @idjevm
Build URL is not available: 503 Service Temporarily Unavailable

@galeza Yea, that's weird. When I checked yesterday, it still worked.

Hi @mel-chiu @idjevm
Build URL is not available: 503 Service Temporarily Unavailable

Fixed.

Hi @mel-chiu , @idjevm
Tested on:
simulators:  iPhone 11 Pro Max,  iPhone 11,  iPhone XS
emulator: Android 9.0 - Samsung Galaxy Note 10 Plus

Real device:
Xiaomi Redmi - Android 10

@rabbi08 Hi, Do you mean the logo is not fixed when the user scrolls down the screen?

@mel-chiu
While testing on iPhone 11 Pro Max-
The Fightpandemics logo hide when keyboard open. And when user scroll down the link to 'Back to login screen' hides. Please see below screenshots:

@rabbi08 Hi, Do you mean the logo is not fixed when the user scrolls down the screen?

@mel-chiu

While testing on iPhone 11 Pro Max-

The Fightpandemics logo hide when keyboard open. And when user scroll down the link to 'Back to login screen' hides. Please see below screenshots:

Yes the logo hides when the keyboard pops up.
Expected: the logo should also be displayed while keyboard is open

@rabbi08
If possible could you check it using staging URL?

@rabbi08 I understand your expectation. However, if we are going to change it then other similar views should follow the same rule as well.  How do you think?

@rabbi08
If possible could you check it using staging URL?
On simulator I see:

@galeza
I compared it with STG, the logo remains intact while keyboard is open.
Actually I am using ph so can't send the video on GitHub from ph app, let me see if I can share the video from STG and this new feature for comparison

@mel-chiu @galeza
I have tested the feature on iphone 11 pro max and below are the screen recordings from feature build and STG. The FP logo does NOT stay intact on the screen when keyboard pops up.
Feature screen recording:
https://share.getcloudapp.com/X6uNKNm4
STG password screen recording:
https://share.getcloudapp.com/04uq6qp1
Let me know if I am missing something here. Thanks.

@rabbi08  I see what you mean. I will take a look when I have free time later. However, as I said above, The logo on similar pages also moves out of the screen when the user scrolls down the screen.
From a user experience perspective, the users care more about finishing the task instead of looking at our logo. Therefore, I think giving them a clear view to finish their tasks is more important. Unless we decide that marketing our brand is a priority at this moment then we can remind them of our brand all the time.

@rabbi08 I see what you mean. I will take a look when I have free time later. However, as I said above, The logo on similar pages also moves out of the screen when the user scrolls down the screen.
From a user experience perspective, the users care more about finishing the task instead of looking at our logo. Therefore, I think giving them a clear view to finish their tasks is more important. Unless we decide that marketing our brand is a priority at this moment then we can remind them of our brand all the time.

As per my understanding, Its not as marketing, actually it was like all screen components fitting in the screen. Looking at other screens, currently in STG, i.e. the lower links and FP logo hiding when keyboard is open in the phone.
So I would say, if we can fit the logo in this screen without other text overlapping then its fine, this implementation is pretty smooth while user's keyboard pops up.
@galeza @jessica-lau @sheeba1  please add your thoughts, if any.

Using iPhone 7 Plus, I can see the button on the recover password screen when the keyboard is opened.

@rabbi08 @mel-chiu - Regarding the FP logo not being fixed, I only see the logo move out of the screen when the user scrolls down the screen on the login, sign up, and recover password pages. The logo is fixed at the top on other pages of the app (help board, about us, clicking into posts, etc). I checked this both on the STG url and Build url.
Maybe a separate PR can be created to correct this if it's something that should be changed as it seems like the focus of this PR is to fix the recover password button.

@jessica-lau @rabbi08  cc @mel-chiu @galeza
The non-fixed logo - is not related to this issue or the resolution of it. Basically these pages use the LogoLayout (not sure why it's named that way) instead of the more common NavigationLayout (which does indeed have a fixed header). It could be a new issue but super low priority (probably not even worth opening a new ticket at this current moment...).
The goal here is to fix the overlapping links or "scrunching" (without breaking anything else):

I'll give me own review soon (I've redeployed review branch).

This is outdated/fixed in the current staging.
Closing this for now.
Thanks everyone.


embiggen disk

disk is 17G/25G right now, we should get more disk space. I suspect disk space is cheap. if not we can think about other ways to get more disk (remove most rf replays from staging, for example) but this is the simplest way I can think of

I did a disk cleanup before. I used du to find the things that used the most disk space and if we didn't need it I cleaned it out. One thing that was using a lot of space were the systemd logs

oh interesting. we should probably set up a ttl on the systemd logs somehow too then

Disk is 55GB now


Bug: Home Page Not Displaying Books from Database and Loading Spinner Issue.

The current behavior of the home page is not in line with the expected functionality. Instead of rendering books fetched from the database, it continuously displays the loading spinner without presenting any books on the UI.
The reason for this current behavior is, at line 613 of the Library.jsx component, books fetched from the db are displayed if the db is not empty (there must be at least one book). Else, the loading spinner is displayed. There is currently no book in the db, which is why the loading spinner is continuously running.
To fix this issue, the following changes are recommended:
- While waiting for a response from the database, the loading spinner should be displayed.
- if there are books in db, display it.
- In cases of database errors or when the database is unavailable, populate the UI  with books from data.json.
-  else, display an "Add Book " button, along with a message politely informing the user that no books are currently stored in the database.

@gbowne1 This issue should be deleted from the issues tab. It has been closed.

Ok @BlackBond06.
Closing this as completed.


Allow addressing of devices just with the instance name

currently, I have to use instance.service to address a device. But service is always "_io4edge_core._tcp" and therefore redundant.

Done with commit a794f61160d11f7cd39e39856d520a9303885752


Executing modifyCloudToDeviceConfig within Cloud Function

In my cloud function, I want to send configurations programatically towards the Device. However, calling modifyCloudToDeviceConfig fails because it still wants to ask for authentication.
{ Error: Request is missing required authentication credential. Expected OAuth 2 access token, login cookie or other valid authentication credential.  }

Is there a way to allow API call within the same account hosting the Cloud Functions without the need to authenticate (because Im also sending the configurations towards my device) ?
Or maybe an alternative?
Code snippets from my Cloud Function:
const google = require('googleapis');
...
const getClient = () => {
  return new Promise((resolve, reject) => {
    const discoveryUrl = `${DISCOVERY_API}?version=${API_VERSION}`;
    google.discoverAPI(discoveryUrl, {}, (err, client) => {
      if (err) {
        console.log('Error during API discovery', err);
        return undefined;
      }
      resolve(client);
    });
  });
}
...

exports.SendToMyMachine = function (request, response) {
  console.log(`SYDNEY>>>> ${request}`);
  console.log(`request.resource.project_id ${request.resource.project_id}`)
  console.log(`request.resource.region ${request.resource.region}`)
  getClient().then(client => {
    console.log(client);
    const projectId = 'XXXX';
    const cloudRegion = 'XXX';
    const registryId = 'XXX';
    const deviceId = 'XXX'
    const parentName = `projects/${projectId}/locations/${cloudRegion}`;
    const registryName = `${parentName}/registries/${registryId}`;
    const deviceName = `${registryName}/deviceid/${deviceId}/`;
    const body = { versionToUpdate: 23, binaryData: "YmF0bWFuMTIzNA==" };

    const pushRequest = {
      name: `${registryName}/devices/${deviceId}`,
      resource: body
    };

    client.projects.locations.registries.devices.modifyCloudToDeviceConfig(pushRequest, (err, data) => {
      if (err) {
        console.log('Error configuring :', deviceId);
        console.log('Error >>>>> :', err);
        response.status(200).send('Error');
      } else {
        console.log('Configured device:', deviceId);
        console.log(data);
        response.status(200).send(JSON.stringify(data));
      }
    });

  })

Thanks!

I had the same issue. This works for me: https://github.com/nqbit/mongoose_esp32_gcp/blob/master/gcp_function/index.js

Thanks @nqbit . After i tried, seems like google.cloudiot('v1') function is not in the googleapis.
This is the failure when deploying it
Deployment failure:
Function load error: Code in file index.js can't be loaded.
Is there a syntax error in your code?
Detailed stack trace: TypeError: google.cloudiot is not a function
    at Object.<anonymous> (/user_code/index.js:6:23)

Closing this.
After reading @nqbit 's code, i applied the same approach using the google.auth.getApplicationDefault with the APIs returned from Discovery URL and not the google.cloudit('v1') and it worked as expected.
For those interested, heres what i did https://github.com/xtrycatchx/google-cloud-experiments/blob/master/cloud-function-experiment/index.js


Add db npm scripts and update readme.

Pull Request Test Coverage Report for Build 57

0 of 0 (NaN%) changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage remained the same at 74.242%

Totals

Change from base Build 34:
0.0%

Covered Lines:
145

Relevant Lines:
183

💛  - Coveralls

Switching travis to use ./node_modules/.bin/command instead of npm run. For some reason, npm blows up when standard doesn't return a success exit code.


Markdown section headings are not rendered with the correct font size

The section titles seem off when rendered on the website.
For example, note the size of the title Configuring mesh-wide Egress:
https://github.com/openservicemesh/osm-docs/blob/main/content/docs/tasks_usage/traffic_management/egress.md
On Github (correct):

On the website (wrong):

Duplicate of https://github.com/openservicemesh/osm-docs/issues/155


Use semaphore job to update helm index

Description
Updating the helm index file has proved difficult for releasers in recent releases, so lets move it to a semaphore job and kick it as soon as the release is published in github.
Related issues/PRs

Todos

[ ] Tests
[ ] Documentation
[ ] Release note

Release Note

None required

Looks like we may need to set GOMOD_VENDOR as per https://github.com/projectcalico/go-build/blob/v0.62/Makefile.common#L123-L128 in order to get CI working. I'm not too sure, but since our vendoring appears to be using glide, we might be better off removing it since setting GOMOD_VENDOR might not even work in this case.

Not sure if this PR is the cause, but it seems there are some serious issues with the helm repo at the moment: https://github.com/projectcalico/calico/issues/5143


Wisconsin broke its data

Wisconsin's system has made the "current year" file for 2024, but disappeared the 2023 data set. We could try to work the scraper around the problem but really Wisconsin needs to fix this.
https://dwd.wisconsin.gov/dislocatedworker/warn/

Do we need to do a workaround in the meantime?
From: Mike Stucka @.>
Date: Tuesday, January 2, 2024 at 8:47 AM
To: biglocalnews/warn-scraper @.>
Cc: Subscribed @.***>
Subject: [biglocalnews/warn-scraper] Wisconsin broke its data (Issue #594)
Wisconsin's system has made the "current year" file for 2024, but disappeared the 2023 data set. We could try to work the scraper around the problem but really Wisconsin needs to fix this.
https://dwd.wisconsin.gov/dislocatedworker/warn/
—
Reply to this email directly, view it on GitHubhttps://github.com/biglocalnews/warn-scraper/issues/594, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AAEFU3XQURXS2FZRXFSQSRLYMQT2JAVCNFSM6AAAAABBKGS4DKVHI2DSMVQWIX3LMV43ASLTON2WKOZSGA3DENRQGI2TCOA.
You are receiving this because you are subscribed to this thread.Message ID: @.***>

@cephillips I think in this case the right thing to do is to let Wisconsin know they've actually deleted some data, and hopefully they can get it restored. They're all set up for 2024 so when they get a 2024 entry the scraper should pick right up and get back to work. Big Local News is not missing any data so the scraper error isn't a problem for us, but the missing data is a problem for the public. I'll message the state momentarily.
CT and DC have a problem where the scrapers are set up to look for filenames with the year in them (e.g., "warn2024.html") and files for the new year have not been created. We're not missing any data as 2023 files are complete there. I can build workarounds for those but it'd likely involve scraping and parsing the web site, which may be more prone to break than the current systems ("get files from the current year back to 2015"). I'll message you separately on Slack ...

I stand corrected; the Wisconsin data's there. I'll work up a fix probably this afternoon.


Direct Line Speech: Find out discrepancies between Direct Line and Direct Line Speech

Feature Request
Direct Line Speech channel is a new protocol and it works very differently than Direct Line channel.
For example, how Adaptive Cards and OAuth works in existing Direct Line channel, may change drastically compare to Direct Line Speech channel.
We need to write up a doc talking about the difference of expectations and user stories both channel fit in. And suggests the user how to select between two protocols.

[ ] What benefits will Direct Line Speech bring?

Why it is not a flip of switch?

[ ] What features will be missing from Direct Line Speech?

On the user story side, why these features are not important for Direct Line Speech clients?
How to tweak the user story so Direct Line Speech will continue to work?

[ ] How to enable Direct Line Speech?

[x] Reference to the Direct Line Speech documentation for setting up the bot
The bot should send speak property with SSML tags inside, otherwise, it will not be synthesized

Draft
Why and how
Direct Line Speech is designed for Voice Assistant scenario (smart display, automotive dashboard, navigation system, etc with low-latency requirement) on single-page application and progressive web apps (PWA). It is not focusing on transcript-based UI in traditional websites. Thus, some features required by Web Chat's traditional transcript-based UI will be missing (conversation history, reconnection)
Our 13.b.smart-display sample show how to use Direct Line Speech in its designed scenario.
Known issues

User ID behaviors

If user ID is not specified

conversationUpdate/membersAdded will be sent without any member ID

Please file a bug to us if this is an issue for you

message activity will be sent with a GUID-based random user ID

If user ID is specified

conversationUpdate/membersAdded will be sent with the specified user ID
message activity will be sent with the specified user ID

conversationUpdate/membersAdded will be sent on every connect and reconnect

Please file a bug to us if this is an issue for you

Bot is now responsible for handling the output of speech recognition

For example, bot should handle different text normalization mode (ITN, masked ITN, etc)
Text normalization is used for recognizing numbers, e.g. "two four pieces of chicken nuggets", instead of "24 pieces of chicken nuggets".

We do not support reconnection after page refresh

Different conversation ID on every page refresh
Please file a bug to us if this is an issue for you

We do not keep/resend conversation history

Please file a bug to us if this is an issue for you

For speech input, we do not support piggybacking additional data

Mechanism outlined in this sample will not work for speech input
It will continue to work with keyboard input

Speech recognition language cannot be switched on-the-fly

Please file a bug to us if this is an issue for you

Proactive message is not supported

Please file a bug to us if this is an issue for you

Emulator do not support Direct Line Speech protocol

Please file a bug to us if this is an issue for you

Aborting recognition is not supported

Please file a bug to us if this is an issue for you

A lot of features in Cognitive Services is not supported in Direct Line Speech, refer to the matrix in DIRECT_LINE_SPEECH.md for details

Please file a bug to us if this is an issue for you

[Enhancement]

Fixed in #2621.


window_settings example never becomes visible

Bevy version
bug started appearing after https://github.com/bevyengine/bevy/pull/11227
Relevant system information
INFO bevy_render::renderer: AdapterInfo { name: "NVIDIA GeForce RTX 4080", vendor: 4318, device: 9988, device_type: DiscreteGpu, driver: "NVIDIA", driver_info: "546.33", backend: Vulkan }
INFO bevy_diagnostic::system_information_diagnostics_plugin::internal: SystemInfo { os: "Windows 10 Pro", kernel: "19045", cpu: "", core_count: "8", memory: "63.9 GiB" }

What you did
cargo run --example window_settings
What went wrong
The window never becomes visible
Additional information
For some reason, Update systems never get called when the window is invisible.

It's working for me on linux. Probably a platform specific issue
2024-01-09T19:30:19.951951Z  INFO bevy_render::renderer: AdapterInfo { name: "AMD Radeon RX 7900 XTX (RADV NAVI31)", vendor: 4098, device: 29772, device_type: DiscreteGpu, driver: "radv", driver_info: "Mesa 23.3.2-arch1.2", backend: Vulkan }
2024-01-09T19:30:20.446509Z  INFO bevy_diagnostic::system_information_diagnostics_plugin::internal: SystemInfo { os: "Linux rolling EndeavourOS", kernel: "6.6.10-arch1-1", cpu: "", core_count: "8", memory: "31.2 GiB" }

Works on macOS too, and failure reproduced in CI on Windows


[core] add raise_for_status to HttpResponse

fixes #16336
Calling response.raise_for_status() will raise and HttpResponseError if status code >= 400, do nothing if not.
Adding this code now for the low level client, where we want to make a network request without automatically raising the error ourself, and instead allow users to raise the error themselves
Purposely does not include error model deserialization

Please update core version to 1.11.0.

Please update core version to 1.11.0.

/azp run python - storage - ci

/azp run python - storage - ci


Release Google.Cloud.Security.PrivateCA.V1 version 3.7.0

Changes in this release:
New features

Add encoding format to .google.cloud.security.privateca.v1.CaPool Resource (commit 260057c)

Documentation improvements

A comment for field maximum_lifetime in message .google.cloud.security.privateca.v1.CaPool is changed (commit 260057c)
A comment for field maximum_lifetime in message .google.cloud.security.privateca.v1.CertificateTemplate is changed (commit 260057c)
A comment for field subject_key_id in message .google.cloud.security.privateca.v1.CertificateConfig is changed (commit 260057c)
A comment for method FetchCaCerts in service CertificateAuthorityService is changed (commit 260057c)
A comment for field ignore_dependent_resources in message .google.cloud.security.privateca.v1.DisableCertificateAuthorityRequest is changed (commit 260057c)
A comment for field ignore_dependent_resources in message .google.cloud.security.privateca.v1.DeleteCertificateAuthorityRequest is changed (commit 260057c)
A comment for field ignore_dependent_resources in message .google.cloud.security.privateca.v1.DeleteCaPoolRequest is changed (commit 260057c)
A comment for field ca_certs in message .google.cloud.security.privateca.v1.FetchCaCertsResponse is changed (commit 260057c)

Created release for Google.Cloud.Security.PrivateCA.V1-3.7.0


cleanup duplicate code for list records function

As discussed in the issue https://github.com/dominikbraun/timetrace/issues/163, changing ListRecords to call the function loadAllRecords as the implementation is the same.

Issue: https://github.com/dominikbraun/timetrace/issues/163

You're Welcome!! Thank you for letting me pick this up and contribute. What does the merge process look like? Will you do it?

Yes, I'm going to merge this probably later today. I've got quite some issues and PRs open and want to make sure only specific ones go into the following 0.12.0 release. This one will definitely included in 0.12.0.

sounds good. Thank you.


Dropdown value is [object Object] when selecting an item that is an object using DropdownItemProps

Bug Report
If you provide an object for the items prop on Dropdown (fitting the DropdownItemProps interferace) and a defaultValue the Dropdown seems to break and not show content.
Steps
Create Dropdown items that might look like this:
const inputItems = [
  {
    key: "BruceW",
    content: "Bruce Wayne",
    active: false
  },
  {
    key: "Natasha",
    content: "Natasha Romanoff"
  },
  {
    key: "Steven",
    content: "Steven Strange"
  }
];

const DropdownExample = () => (
  <Dropdown
    items={inputItems}
    defaultValue={inputItems[1]}
    getA11ySelectionMessage={{
      onAdd: item => `${item} has been selected.`
    }}
  />
);

Here I need to create objects for these items, because I need to pass in the active property for some of my items.
I passed in a defaultValue which is the second item in my items array. But when I do this, I get this result:

So instead I changed the Dropdown to this:
const DropdownExample = () => (
  <Dropdown
    items={inputItems}
    defaultValue={inputItems[1].content}
    getA11ySelectionMessage={{
      onAdd: item => `${item} has been selected.`
    }}
  />
);

Which does give me a correct initial render:

But after selecting another item in this dropdown, I get back to a broken "[object Object]" state:

Expected Result
The expected result would be that defaultValue should render the content on the DropdownItemProps, as well as when selecting other values Dropdown should render the content property as well.
Actual Result
The defaultValue renders "[object Object]" and even if you explicitly pass the content prop, it breaks when selecting another value.
Version
0.44.0
Testcase
https://codesandbox.io/s/fluent-ui-example-qv4e8

Have same issue. Would love to know if it's a bug or user error on my part.

I managed to find a workaround (for my use case anyway) by supplying itemToString. My code:
export const FrameSelector = () => {
    const items = [

Item 1
,
        "Item 2"
    ];

    function itemToString(value) {
        const placeholder = value && value.props && value.props.placeholder;
        return placeholder || "PLACEHOLDER NOT PROVIDED";
    }

    return <Dropdown itemToString={itemToString} items={items}/>
};

My goal is simply to indent some items to make the dropdown look tree-like. I don't know if the above is best practice but it serves my purpose.

@jugglingcats Thanks for the tip on itemToString, that got me closer (sort of..) and I ended up with this:
https://codesandbox.io/s/fluent-ui-example-wo34b
The issue I'm running into now is that the active property doesn't seem to be respected, and I'm able to select items even if it's set to active: false.


SSH config support

/kind enhacement
Using this provider, one can use SSH to connect to libvirt daemon on remote host. It can be done via connection uri with parameters like user, hostname, keyfile, etc.
It would be very helpful to be able to provider Host, as defined in ~/.ssh/config which already has these parameters (user, hostname, keyfile) defined. AFAIK, native Go's ssh client library cannot do this, however, there's ssh_config which can parse OpenSSH config files.
Example:
Given that functionality is implemented, following uri:
uri = "qemu-ssh://root@host.example.org/system?keyfile=/home/myuser/.ssh/id_rsa"
is equivalent to
uri = "qemu-ssh://machine"
given that in ~/.ssh/config there's following content:
Host machine
        HostName host.example.org
        User root
        IdentityFile /home/myuser/.ssh/id_rsa

Motivation:
in AWS provider, one can specify multiple ways of connecting to AWS API. One of them is profile. If there's profile named foo configured on machine of engineer John with his credentials and profile with different credentials for engineer Daphne on her machine also named foo, they can share the same Terraform configuration with ease because profile name is the same. I wanted something similar for this provider.
I do know it might be very specific usage and you may not be keen on including feature. However, if it is okay with you, I can implement this.

Not sure if this covers your use case, but a similar feature request was just merged to main . Apologies in advance if you are looking for something else.

Hi @inflatador
I did look at mentioned PR but AFAIK, this does not implement what I want. It does implement support for SSH agent but configuration file is parsed by client (ssh command in case of openssh) - at least from what I was able to get from investigation.

@inflatador :: The latest 0.6.11 presented a regression in my case.
This is the connection string I was using with 0.6.10:
qemu+ssh://rgomes@host.example.com/system?keyfile=$HOME/.ssh/id_ed25519_laptop.example.com

Now it is failing with 0.6.11.
In more than 25 years using more than 12 flavors of Unixes, I've never had to use ssh-agent and I do not feel comfortable with the idea of being forced to be use it at this point. I'm reverting to 0.6.10.
The standard approach for configuring a ssh client is populating ~/.ssh/config.

@dmacvicar WDYT?

@inflatador :: The latest 0.6.11 presented a regression in my case.
This is the connection string I was using with 0.6.10:
qemu+ssh://rgomes@host.example.com/system?keyfile=$HOME/.ssh/id_ed25519_laptop.example.com

Now it is failing with 0.6.11.
In more than 25 years using more than 12 flavors of Unixes, I've never had to use ssh-agent and I do not feel comfortable with the idea of being forced to use it at this point. I'm reverting to 0.6.10.
Probably the most powerful approach and arguably most popular approach for configuring a ssh client is populating ~/.ssh/config.

@frgomes can you please read https://github.com/dmacvicar/terraform-provider-libvirt/issues/886#issuecomment-933228205 check if that solves it for you, and contribute to data in #886.
It is clear that I will not add a feature only because another functionality is not working as expected.

@dmacvicar WDYT?

In general I like the use-case you describe, about sharing a terraform configuration and keeping the user-specific settings local to the user.
I am only worried about this changing the ssh settings behinds user backs and then generating bug reports. Any idea how to avoid that?

@dmacvicar @frgomes is #933 a potential fix for this issue?

Related to #1058


[GSB] Infer requirements from parents of types.

A type Foo<...>.Bar may only exist conditionally (i.e. constraints on the
generic parameters of Foo), in which case those conditional requirements
should be implied when Foo<...>.Bar is mentioned.
Fixes SR-6850.

@swift-ci please smoke test

@huonw , will you be pulling this into the 4.1 branch?

@swift-ci please smoke test

@swift-ci please smoke test and merge
Woohoo, fixed 2 crashers.

@swift-ci please smoke test and merge

@swift-ci please smoke test and merge

@swift-ci please smoke test and merge

@swift-ci please smoke test and merge

@swift-ci please smoke test and merge
Maybe someday I'll remember how to do this properly.

@swift-ci please smoke test and merge

@swift-ci please smoke test and merge

@swift-ci please smoke test.

@swift-ci please smoke test macOS

@swift-ci please smoke test macOS


Clean up environment.yml and requirements.txt files

This has been discussed before but the introduction of a security risk with one of the dependencies that can't be resolved without creating conflicts has elevated this issue.
https://github.com/NCAR/METplus/network/alert/requirements.txt/Pillow/open
Dependabot cannot create a pull request as one or more other dependencies require a version that is incompatible with this update.
Our current environment files include many additional packages that are not required to run METplus. They are used to run tests, build documentation, run plotting scripts, etc. The only actual requirement of the METplus wrappers that does not come standard with python3 is dateutil.
Also, the current environment files are not cross-platform compatible, so it may not work on machines that differ from the environment they were created. There is a conda command that will generate a file that is cross-platform compatible (see
https://docs.conda.io/projects/conda/en/latest/user-guide/tasks/manage-environments.html#exporting-an-environment-file-across-platforms)
Describe the Enhancement
We should replace the existing env files with simple versions and create another environment that can be used for development. This environment can be expanded as needed as more packages are needed, but the files in the top level of the repo should remain as simple as possible.
Time Estimate
1 - 3 days
Sub-Issues
Consider breaking the enhancement down into sub-issues.

[x] Create a baseline python 3.6.3 environment and install dateutil
[x] Regenerate environment.yml (conda env export --from-history > environment.yml)
[x] Regenerate requirements.txt (pip freeze > requirements.txt)
[x] Replace existing files with the stripped-down versions
[ ] Recreate an environment that has the packages needed for development, testing, building docs, plot wrappers (? at least for testing these tools until they move out of the METplus repo), etc.
[ ] Check that the new environment resolves the Pillow security issue. A branch could be used to set the dev environment files in the top-level to see if the security issue persists.
[ ] Regenerate environment.yml and requirements.txt files and check them in (under internal_tests?) so they are readily available to developers

Relevant Deadlines
List relevant project deadlines here or state NONE.
Funding Source
Define the source of funding and account keys here or state NONE.
Define the Metadata
Assignee

[X] Select engineer(s) or no engineer required
[X] Select scientist(s) or no scientist required

Labels

[X] Select component(s)
[X] Select priority
[X] Select requestor(s)

Projects and Milestone

[X] Review projects and select relevant Repository and Organization ones
[X] Select milestone

Define Related Issue(s)
Consider the impact to the other METplus components.

[ ] Consider doing the same for other repositories that use python to simplify the dependencies listed to the users
METdb, METviewer, METexpress, METcalcpy, METplotpy

[ ] Consider enhancing METplus wrappers time interval logic to only require dateutil if using time intervals of months or years. These intervals require using dateutil's relativedelta to determine the length of these dynamic intervals. Most use cases in METplus don't use these intervals, in which case datetime's timedelta (comes with baseline python) can be used. This would reduce the dependencies to run METplus wrappers to just python 3.6.3 unless user needs the additional functionality.

Enhancement Checklist
See the METplus Workflow for details.

[X] Complete the issue definition above.
[x] Fork this repository or create a branch of develop.
Branch name:  feature_<Issue Number>_<Description>
[x] Complete the development and test your changes.
[x] Add/update unit tests.
[x] Add/update documentation.
[x] Push local changes to GitHub.
[x] Submit a pull request to merge into develop.
Pull request: feature <Issue Number> <Description>
[ ] Iterate until the reviewer(s) accept and merge your changes.
[ ] Delete your fork or branch.
[ ] Close this issue.

Created pull requests to update environment.yml and requirements.txt for develop and master_v3.0 (delete requirements.txt in master) with changes we discussed in the METplus meeting today.
Still need to create a conda environment.yml file to use for development and verify that security alert goes away.


Repo of the new advanced version in beta: https://infinite.benjamin.sh/advanced

Hello, this is not an issue but a request. Do you have the repo of the new advanced version in beta: https://infinite.benjamin.sh/advanced?
BTW excellent work!

Thank you! The code for the advanced mode is in this repository: Advanced


azure network vpn-gateway create giving an unclear error

Tried to add a GW using the following command
aanandr@AANANDR-WORK:/mnt/c/Windows/System32$ az network vpn-gateway create --name VNet2GW --public-ip-address VNet2GWPubIP --resource-group VV2 --subnet-id GatewaySubnet
Getting the following error. Its difficult to understand the cause of the problem. It seems to be about the format used to specify the Gateway Subnet but the msg. doesnt give any useful guidance
At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-debug for usage details. {
"error": {
"code": "InvalidRequestFormat",
"message": "Cannot parse the request.",
"details": [
{
"code": "InvalidJsonReferenceFormat",
"message": "Reference Id GatewaySubnet is not formatted correctly. The Id is expected to reference resources of type virtualNetworks/subnets. Path properties.ipConfigurations[0].properties.subnet."
}
]
}
}  Correlation ID: 40dc667e-5ccc-4deb-a51a-8850d9a4f282

This is the error from the server, but it is correct.
Reference Id GatewaySubnet is not formatted correctly
--subnet-id requires a fully qualified ID. You cannot use a name.


Migrate Configs to pyproject.toml (PEP518)

I got rid of the setup.cfg and setup.py files, switching from setuptools to flint.
Added a mypy.ini (no thanks to Guido for not merging
a PR implementing support for PEP518).
Also added some hypothetical environments in tox.ini (docs, tests).
Lint env already runs (yay) with errors (oh).

Disclaimer: I have no clue on how mypy's innards work or how it's configured (I'm a pyre guy) - if my config's shit, let me know!

lgtm.
One suggestion below.

Did as suggested, updated tox.ini and config accordingly!

Force merging this, as I mereley updated meta data as requested.


$request->replace Method not working in the Testing Environment

Laravel Version: 5.6.24
PHP Version: any
Database Driver & Version:

Description:
Using $request->replace() is not working as expected when calling an endpoint using json('method','uri') in the Unit/Integration tests.
When i use  $this->replace(['name' => 'test']) in the form request and try to access the value in the controller using $request->get('name') it returns null, and it seems only happen when call an endpoint using the json() method while it works fine with other methods like get(),post(),put().
I put the following in the Form Request
    public function rules()
    {
        $this->replace([
            "name" => "Test",
        ]);

        return [
            //
        ];
    }

In the controller i did the following
    public function sanitize(SanitizeRequest $request)
    {
        return [
            'name' => $request->get('name')
        ];
    }

The $request->get('name') return null.
Steps To Reproduce:
I made a git repository with Test  to replicate this issue
https://github.com/skmail/laravel-request-sanitize
Files to review:

app/Http/Requests/SanitizeRequest.php
app/Http/Controllers/RequestSanitizedController.php

Tests
tests/Feature/RequestSanitizeTest.php
PS: $request->all(), returns the replaced items without any issues
Thanks

You probably don't want to be calling Request::get almost ever btw, use Request::input ... Request::all uses Request::input ... Request::input is Laravel's method for retrieving inputs (uses the priorities that Laravel prefers, get is from symfony request)

Using the correct method to pull from the correct input source, $request->input(), you shouldn't have an issue.
Though this does illustrates a difference between the FormRequest and a regular Request. When the regular Request is created the request property is set to the input source, $request->request = $request->getInputSource();. When sending a json request this will get set to the json parameter bag from $request->json(). $request->request === $request->json(). When calling $request->replace() it gets the input source, json, and calls replace on that parameter bag ... which happens to be the same bag as $request->request.
For the FormRequest this does not happen. It gets filled with arrays that end up in new parameter bags. When you call $request->replace() it is calling replace on the correct input source, json. If you were to get that parameter bag it will have the replacements. $request->request is not the same object as $request->json() so $request->request is still holding the original values or lack there of.
$request->get() (the symfony method) is checking $this->attributes, $this->query and $this->request to find the key.   Illuminate\Http\Request::input is pulling the input source, json in this case, and the querystring looking for the key. It is the preferred way to get inputs from the request.

Thanks for the clarification

No problem.
Now I am interested to figure out if that difference between FormRequest and Request in this respect was intentional or not.  :-)

@themsaid Would you happen to know why Illuminate\Http\Request@createFromBase has a $request->request = $request->getInputSource() but @createFrom does not do this tying of $request->request to the current input source?


FIX: getObject works with ACL header: public-read

When setting the 'public-read' ACL permission of the object, any account can get an object (objectGet) / get its metadata (objectHead) even if the account has no right to the bucket

LGTM :+1:

👍 but let's do it against rel/1.1

Also please don't forget to mention the issue in the commit body.


Remove assembly and output debug output to console when running UTs for java client

After we publish java client to repository.a.o, we do not need to create an assembly tarball for users any more.
Let's remove it.

Fixed by #1266


[Bug] Tab (SwiftUI) Doesn't scale correctly when dynamic sizes change

This has not been merged to main yet and is on branch 717-but-tab-swiftui-doesnt-scale-correctly-when-dynamic-sizes-change

There is a visual bug after scaling (it disappears when I rotate the screen) :
https://github.com/adevinta/spark-ios/assets/142396647/ab5bd187-07de-4e11-bdb2-bb6538b36c8f

I've opened another ticket for this:
https://github.com/adevinta/spark-ios/issues/746


Possibly wrong pattern match in VirtualizingStackPanel.GetControl

Describe the bug
In C# the pattern x is not A or B is interpreted as x is (not A) or B and not x is not (A or B).
I was grep'ing my local git repositories for code that matched and stumbled upon
https://github.com/AvaloniaUI/Avalonia/blob/82e3613939af5451597d3c19f3a48e0a8ba888f3/src/Avalonia.Controls/VirtualizingStackPanel.cs#L284
So this code is not checking whether direction is "neither First nor Last".
As Last is covered by not First Roslyn even lowers this to != First.
As such this looks like a bug to me

SharpLab
To Reproduce
I found this by analyzing the code and didn't write any code I don't know and haven't looked into if this potential bug can be hit.
Expected behavior
No response
Avalonia version
master
OS
No response
Additional context
No response

It is wrong, you are correct.
Whole condition should be changed from
if (count == 0 ||
-                fromControl is null && direction is not NavigationDirection.First or NavigationDirection.Last)
+               (fromControl is null && direction != NavigationDirection.First && direction != NavigationDirection.Last)

@jnyrup maybe you can file a PR with the fix? 🙏 thx

@jnyrup maybe you can file a PR with the fix? 🙏 thx

Unfortunately I don't have the time to do this.
I'll pass it on to other contributors.


Increase MaxConnections and MaxStartup in sshd config

When more than 10 data movement requests come in for a particular
rabbit, the default sshd configuration starts dropping 30% of
connections and drops all after 100 connections (the default value is
set to 10:30:100). This causes data movement requests to fail since any
concurrency over 10 causes ssh to close connections (from mpirun).
This change increases that value to be able handle the max theoretical
load for a particular rabbit. This image runs on 1 pod per rabbit node
(i.e. nnf-dm-worker-*) and each rabbit node supports 16 compute nodes of
192 cores. Each core on a compute node could be creating a data movement
request.
16 * 192 = 3072
Bump it up to an power of 2 for good measure -> 4096.
This is a start. I believe this number will need to be much higher if there is a
lustre job that spans every node and every node tries to start DM requests via SCR.
But 4096 is much better than 10.
Signed-off-by: Blake Devcich<EMAIL_ADDRESS>

This addresses https://github.com/NearNodeFlash/NearNodeFlash.github.io/issues/196


Fixed broken link for WeWork Plasma

They don't seem to have a landing page for their set of design systems, but this site may "work" for now.

Thanks for fixing!


Create a receiver transform to map the patient's age to an OBX segment

User Story
As ReportStream, I want the ability to create OBX segments with a patient's age, so that RADx MARS receivers can get data in the right format.
Description/Use Case
The RADx MARS HL7 v2 format expects an OBX segment with the patient's age. We currently only get a patient's DOB and not their age. The RADx MARS format expects OBX segment to appear in this sequence: The COVID-19 result should be the first OBX segment, followed by any other result OBX segments, and then the Age, Sympotmatic, and Date of Onset OBX segments.
Risks/Impacts/Considerations
RADx MARs messages will not contain the age OBX segment.
Dev Notes
Details on how this OBX segment should be mapped can be found on this spreadsheet https://docs.google.com/spreadsheets/d/15dKu5zSY23-VVL0oJphILQyHjyDoj_WmTmXUxGXinzQ/edit#gid=0
Depending on how tickets #9531 and #9532 are implemented it will change how this gets done.
Acceptance Criteria

[ ] ReportStream sends the patient's age as an OBX segment in the correct sequence.

Hey team! Please add your planning poker estimate with Zenhub @jimmyfagan @arnejduranovic @jack-h-wang @JessicaWNava @victor-chaparro @mkalish @thetaurean @TurkeyFried

Missed that this is still blocked. Unassigning...


Syntax Highlighting -- Inline Comments After DEBUG() Statements

Very small issue (put at end of to-do list): End-of-line comments on a DEBUG line look different than on standard code lines.

Thanks.  I was wondering when someone would find that one. ;-)
Because DEBUG (with the apostrophe) breaks from the syntax of the rest of Spin/PASM, this became an issue I couldn't solve right away.  I need to update the syntax highlighter code to count parentheses (matching them up) and only when the DEBUG statement is finished (ie: after the trailing parenthesis), then it can treat the apostrophe as a single-line comment again.
Thanks for noting it in issues.


Fix specific scenarios where Drawables can get drawn before they're updated

Fixes the second part of ppy/osu#2578 where the borders flash for one frame when being added to the catcher.
We don't want the drawables to come alive and then proceed to potentially not be updated for the rest of the frame, and then being drawn.

In principle I am not against this change, but I'd be extremely cautious. We added every line of code there with a specific purpose, so I wonder what the purpose of this particular checkChildLife code was (I do not remember and can not dig deeper right now).
If you're confident nothing will be broken, feel free to merge.

Going to go ahead with merging this then. We'll likely find any regressions quickly if it does happen to cause issues.

@Tom94 To explain a bit, the first occurrence of this was either in https://github.com/ppy/osu-framework/pull/884 which was to account for transforms or in https://github.com/ppy/osu-framework/pull/838 which was to allow for depth to change dynamically.
In thinking about this a little bit more, this will probably break if something else changes our depth in their Update after our own Update has run. I believe this is a very edge case and borderline undefined behaviour right now, though.


Neutrinos and IGRB data missing

Seems like in the last (2024) version of the plot, $\nu + \bar \nu$ and $\gamma$ IGRB data are missing, but the captions are still there.

Thanks for noticing. In fact as I'm rewriting the code, I took profit to add only well referenced datasets. And I realized that I can't remember where I found the gamma-ray data. I'm working on it.


Make lockfiles package public

As we discussed, I think it would be great if the lockfiles package was made into a public package other people could depend on and contribute to.

To me lockfiles makes the most sense to me, since it's a collection of different lockfile parsers.
That said, lockfiles probably isn't too accurate either, since this package includes parsing non-lockfile files (e.g. requirements.txt). How about manifests?
Note that these don't have to go into separate repos either. You can have multiple go modules in the same repo (https://go.dev/doc/modules/managing-source#multiple-module-source) or more simply just chuck them into pkg/ per https://github.com/golang-standards/project-layout#pkg.

more simply just chuck them into pkg/

oh awesome - that didn't come up in my searches but it's exactly what I was hoping for as a first step since I can just chuck them in there for now.
How does it work with internal dependencies? I've got a few types sitting at the top of internal to avoid circular dependencies between the packages - I guess I'd have to put them into pkg as well? (or otherwise do some casting 🤔)

How does it work with internal dependencies? I've got a few types sitting at the top of internal to avoid circular dependencies between the packages - I guess I'd have to put them into pkg as well? (or otherwise do some casting 🤔)

I believe you can keep things in internal and import them from pkg from the same project.

Done in #102 - I decided to stick with lockfile for now.

Awesome!! thank you @G-Rath :)

No problem - I've also landed the API support, which I'm planning to release sometime this weekend (just seeing if I can add support for extracting commits to a few more parsers first).
Let me know what you think should be tackled next :)


Authenticate and authorize against Cognito-protected Elasticsearch cluster

I'm trying to dump an Elastic cluster index stored in AWS that is protected by Cognito and accessible via a VPC endpoint.
What is the correct way to allow elasticdump to authenticate and authorize in that scenario? I've tried the available aws options and AWS_PROFILE, to no avail. Error messages are either:
{"Message":"User: anonymous is not authorized to perform: es:ESHttpGet"}
{"Message":"User: arn:aws:sts::12345678:assumed-role/GroupName/MyEmail is not authorized to perform: es:ESHttpGet"}
{"message":"The security token included in the request is expired"}
Elasticdump version: 6.18.0
Elasticsearch: 7.1 (AWS managed service)
Node: v13.6.0

Useful for me, thanks @dirkgomez and @ferronrsmith


RETS 1.5 unable to get property metadata

I am not to RETS and am running on version 1.5. I am having trouble getting the examples working. Login is working correctly, but I am getting hungup when trying to get metadata or perform searches. I'm getting a response, but there is no data...
request options var:
{"uri":"http://connectmls-rets.mredllc.com/rets/server/getmetadata","jar":true,"qs":{"Type":"METADATA-TABLE","Id":"Property:RESI","Format":"COMPACT"}}
response:
"/RETS ReplyCode="0" ReplyText="Operation Successful" ><RETS-STATUS ReplyCode="20503" ReplyText="Operation Successful" /      /RETS"
Does this have to do with my version? If I was to do a search query to get all active properties, what is the request URL and parameters? At the end of the day, I want to sync the properties into my system.

Hi Tyler,
It may have something to do with the RETS version, not sure though.
One thing I notice is that the field "id":"Property:RESI" is normally just
"RESI" on the RETS server I've tested with.
Are you able to use a tool like postman in chrome or rest client in
firefox? It might be worth trying to make the URL calls from there,
starting with login, to see what the RETS server returns?
-Steve
On Mon, Feb 23, 2015 at 7:02 AM, Tyler Harrington<EMAIL_ADDRESS>wrote:

I am not to RETS and am running on version 1.5. I am having trouble
getting the examples working. Login is working correctly, but I am getting
hungup when trying to get metadata or perform searches. I'm getting a
response, but there is no data...
request options var:
{"uri":"http://connectmls-rets.mredllc.com/rets/server/getmetadata","jar":true,"qs":{"Type":"METADATA-TABLE","Id":"Property:RESI","Format":"COMPACT"}
http://connectmls-rets.mredllc.com/rets/server/getmetadata","jar":true,"qs":{"Type":"METADATA-TABLE","Id":"Property:RESI","Format":"COMPACT"}}
response:
"\r\n\r\n\r\n"
Does this have to do with my version? If I was to do a search query to get
all active properties, what is the request URL and parameters? At the end
of the day, I want to sync the properties into my system.
—
Reply to this email directly or view it on GitHub
https://github.com/sbruno81/rets-client/issues/3.

I was able to see results in the browser. Might just use your auth and then do a standard http request from there. Do you think that'll do it?
Also, there is a field called Rental Price (note the space) which shows up in my meta data:
GetMetadata?Type=METADATA-TABLE&ID=Property:RentalHome&Select=StandardName&Format=COMPACT-DECODED
I am trying to do a search on this field but am getting an invalid select error:
search?SearchType=Property&Class=RentalHome&QueryType=DMQL2&Format=COMPACT-DECODED&StandardNames=1&Select=Rental%20Price&Query=(City=Chicago),(MlsStatus=ACTV)&Count=1&Limit=10
Is there a way to determine which fields are selectable?

Great.
I don't know of a way to find out which fields are selectable. Do other
fields work?
On Mon, Feb 23, 2015 at 10:34 AM, Tyler Harrington<EMAIL_ADDRESS>
wrote:

I was able to see results in the browser. Might just use your auth and
then do a standard http request from there. Do you think that'll do it?
Also, there is a field called Rental Price (note the space) which shows up
in my meta data:
GetMetadata?Type=METADATA-TABLE&ID=Property:RentalHome&Select=StandardName&Format=COMPACT-DECODED
I am trying to do a search on this field but am getting an invalid select
error:
search?SearchType=Property&Class=RentalHome&QueryType=DMQL2&Format=COMPACT-DECODED&StandardNames=1&Select=Rental%20Price&Query=(City=Chicago),(MlsStatus=ACTV)&Count=1&Limit=10
Is there a way to determine which fields are selectable?
—
Reply to this email directly or view it on GitHub
https://github.com/sbruno81/rets-client/issues/3#issuecomment-75626023.

Hi guys, I had a similar issue with rental price, but I found a clue in your conversation.
search?SearchType=Property&Class=RentalHome&QueryType=DMQL2&Format=COMPACT&StandardNames=0&Query=(CIT=chicago),(ST=ACTV)&Count=1&Limit=10&Select=RP
We just need to set StandardNames to 0 and then use SystemNames. Thank you for the ideas.


objective bound fix #43

Update objbound after everything finished.
Use best bound of nlp solver if no integral variables or integral in root node.

Codecov Report

Merging #44 into master will increase coverage by 0.1%.
The diff coverage is 95.65%.

@@            Coverage Diff            @@
##           master      #44     +/-   ##
=========================================
+ Coverage   95.51%   95.62%   +0.1%
=========================================
  Files          21       21
  Lines        1784     1806     +22
=========================================
+ Hits         1704     1727     +23
+ Misses         80       79      -1

Impacted Files
Coverage Δ

src/model.jl
89.62% <0%> (-0.67%)
:arrow_down:

test/basic.jl
100% <100%> (ø)
:arrow_up:

src/BnBTree.jl
91% <100%> (+0.97%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update c4d2153...f3e3dd2. Read the comment docs.


chore(next): re-export

#537  👈
main

This stack of pull requests is managed by Graphite. Learn more about stacking.

Join @hi-ogawa and the rest of your teammates on  Graphite


Invalid actions break CRS 3.1 on rule 912160

The following rule will be treated as invalid by LibModSecurity current master + ModSecurity-nginx v1.0.0.
nginx: [emerg] "modsecurity_rules_file" directive Rules error. File: /etc/nginx/modsecurity.d/include.conf. Line: 14. Column: 29. Expecting an action, got:  ,\ in /etc/nginx/nginx.conf:39

The following is the rule that causes the issues.
SecRule IP:DOS_COUNTER "@ge %{tx.dos_counter_threshold}" \
    "id:912160,\
    phase:5,\
    pass,\
    t:none,\
    nolog,\
    tag:'application-multi',\
    tag:'language-multi',\
    tag:'platform-multi',\
    tag:'attack-dos',\
    chain"
    SecRule &IP:DOS_BURST_COUNTER "@eq 0" \
        "setvar:'ip.dos_burst_counter=1',\
        setvar:'!ip.dos_counter',\
        expirevar:'ip.dos_burst_counter=%{tx.dos_burst_time_slice}'"

The lines in question that cause the issue are the last two
        setvar:'!ip.dos_counter',\
        expirevar:'ip.dos_burst_counter=%{tx.dos_burst_time_slice}'"

I hypothosis this is because setvar in libmodsecurity doesn't support setvar with a '!'.
Additionally i'm unsure if expirevar or variables expansion is supported here.

Fixed as of https://github.com/SpiderLabs/ModSecurity/commit/764a2e43ff03370fafa8272f9270160bcad32023


Design Page: User-Created Dashboards

Features defined in User Interface/User Dashboard Page

test test test


⚠ Bofrost.be has degraded performance

In b6ac23b, Bofrost.be (https://www.bofrost.be) experienced degraded performance:

HTTP code: 200
Response time: 1609 ms

Resolved: Bofrost.be performance has improved in 179df7c.


FEAT: Ingest Glides Events

Glides events are passed through a Kinesis Stream where we can read them and archive them in parquet files.

Create KinesisReader class for connecting and reading from a stream.
Create classes for converting records into parquet records and appending them to files hosted on s3.
Move explode_table_column and flatten_schema methods to ingestion utilities so they can be used more broadly.
rename ingestion/ingest.py to ingestion/ingest_delta.py since we have multiple ingestion flows now.
Add 404 detection to s3.download_file function to avoid logging failures that are acceptable.

Asana Task: 🪂 Establish process to consume and store Glides data

Looks good, so are theses all just getting dumped into singular files? Do we have any idea what file sizes/record counts were expecting per day/month?

i am putting them into a single file. the editor changes and op sign ins are tiny. like 15KB / day.  the trip updates are a bit bigger, around 150KB per day. probably something to monitor, but we can partition down the road if we start seeing performance issues?


Custom date component to match designs

Styling for the date picker modal.
To do

[x] Custom day component to match styles
[ ] Header & CTA styling
[ ] Frosted glass overlay on iOS
[ ] Proper accessibility tagging

Pre-flight check-list
Before raising a pull request

[ ] Documentation
[ ] Unit tests

Pre-merge check-list

[ ] Tester approved

#118 may be worth taking a look at while working on this


Create Posts “5afefa”

Automatically generated by Netlify CMS

👷 Deploy Preview for benevolent-pika-f5fbe0 processing.

Name
Link

🔨 Latest commit
6f4bfc6961adcce6fe171ce8e3b211e4ef861182

🔍 Latest deploy log
https://app.netlify.com/sites/benevolent-pika-f5fbe0/deploys/63f2add5d683d00008a8b073


Bump jenkins library to 1.0.4

Description
This new version uses the same version for library within library that avoids lib conflicts.
See https://github.com/opensearch-project/opensearch-build-libraries/pull/19 https://github.com/opensearch-project/opensearch-build-libraries/pull/20 for details.
Issues Resolved
resolves #2755
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Is removal of vars/README.md expected here @gaiksaya ?

Yes, its not relevant to this repository anymore. Will move to build-lib repo.


Fiddle: Builder Preview not receiving component

In https://mitosis.builder.io/, the bottom half of the screen (the Builder Preview) is no longer receiving the generated component

closed by #638


merge from template-coveralls

package.json

chore(deps): add ^9.0.0 remove ^8.3.21 (devDependencies.npm-pkgbuild)

:tada: This PR is included in version 5.0.105 :tada:
The release is available on:

npm package (@latest dist-tag)
v5.0.105
GitHub release

Your semantic-release bot :package::rocket:


Update org

Repackage sources from com.twilio -> dev.guardrail to match new GitHub org.
I think before this gets merged it would be good to add a scalafix rewrite rule to scala-steward.
The rules:

import com.twilio.guardrail -> import _root_.dev.guardrail, to avoid possible conflicts considering the new root (dev is somewhat of an awkward root package, so _root_ for safety)
com.twilio.guardrail.FooBar -> _root_.dev.guardrail.FooBar for the same reason, but to handle Term.Select's

Opening this PR to announce intent

i do find the soft-need to prefix with _root_ to be kinda icky, though i do agree that it's a good idea.

i do find the soft-need to prefix with _root_ to be kinda icky, though i do agree that it's a good idea.

Missing where that happened in this PR -- the only reason the io package is prefixed is because it conflicts with some libraries, but I'm not seeing it in the diff here

no i mean the need for customers to prefix with _root_ if they use guardrail as a library.

no i mean the need for customers to prefix with _root_ if they use guardrail as a library.

Ah. Yes. Such is life with these new TLDs though. Is dev common enough as a package that this will be an issue?

it's probably not that big a deal, i guess.


Performance issues with batch decompilation

Is your feature request related to a problem? Please describe.
Some of my scripts and plugins rely on decompiling large numbers of functions. I noticed that in particular, decompiling multiple functions at once does not scale past roughly 10 concurrent threads, even when not limited by the processor/memory. To test this, I repurposed the FindPotentialDecompilerProblems.java script to just decompile all functions in a medium sized binary (25k functions), and adjusted the core.override. Decompilation speed basically hits a peak at about 10-12 threads and is noticeably worse with core.override=16 and beyond (although the machine I was testing on has a lot more than 16 cores available)
My best guess is that the decompiler I/O is being the bottleneck here, but any help diagnosing/increasing performance would be most welcome.
Additional context
Ghidra 9.2.2

Assuming you are using the decompilation to markup a program, then the program database lock will become the bottleneck.  If you are using the decompilation without reading/writing the program, then you are likely bumping into the natural limitations of the JVM running on your particular machine.   If you have access, you could try running your scripts on a server-grade machine to separate hardware from Ghidra software limitations.

My machine isn't quite server grade, but it is close.
My scripts do not write to the program while analysis is ongoing (decompiling all functions accounts for > 95% of runtime), so this script has very similar performance. Do you think the program database lock is the bottleneck here, or simply the JVM?
import java.util.*;

import ghidra.app.decompiler.*;
import ghidra.app.decompiler.parallel.*;
import ghidra.app.script.GhidraScript;
import ghidra.app.tablechooser.*;
import ghidra.program.model.address.Address;
import ghidra.program.model.listing.*;
import ghidra.program.model.pcode.*;
import ghidra.program.model.symbol.Reference;
import ghidra.program.model.symbol.ReferenceIterator;
import ghidra.util.task.TaskMonitor;
import util.CollectionUtils;

public class ModifiedFindPotentialDecompilerProblems extends GhidraScript {

	@Override
	public void run() throws Exception {

		if (isRunningHeadless()) {
			printf("This script cannot be run in headless mode.\n");
			return;
		}

		DecompilerCallback<Void> callback =
			new DecompilerCallback<Void>(currentProgram, new BasicConfigurer(currentProgram)) {
				@Override
				public Void process(DecompileResults results, TaskMonitor tMonitor)
						throws Exception {
					return null;
				}
			};

		Set<Function> funcsToDecompile = new HashSet<>();
		FunctionIterator fIter = currentProgram.getFunctionManager().getFunctionsNoStubs(true);
		fIter.forEach(e -> funcsToDecompile.add(e));

		ParallelDecompiler.decompileFunctions(callback, funcsToDecompile, monitor);
		monitor.checkCanceled();
	}

	static class BasicConfigurer implements DecompileConfigurer {
		private Program p;

		public BasicConfigurer(Program prog) {
			p = prog;
		}

		@Override
		public void configure(DecompInterface decompiler) {
			decompiler.toggleCCode(true);
			decompiler.toggleSyntaxTree(true);
			decompiler.setSimplificationStyle("decompile");
			DecompileOptions opts = new DecompileOptions();
			opts.grabFromProgram(p);
			decompiler.setOptions(opts);
		}
	}
}

Based on this code, I'd say you are not running into program lock limitations.    You would have to profile the app to find the bottleneck, as it could be any part of your machine.

I have barely touched the decompiler internals, but my first guess would be ArchitectureGhidra, where the Ghidra client gets overloaded. Looks like I'll have to compile it after all ...

(See my updated comment above.)
We have a hard-coded limit of 10 for the option you are using).   There is no way to override this without changing the source code, whether the SystemUtilities class that parses the property you are updating or the ParallelDecopmiler, which is what your script ends up calling into.

The simple
println("CpuCoreOverride + "System.getProperty("cpu.core.override"));

returns whatever I set VMARGS=-Dcpu.core.override= to.
https://github.com/NationalSecurityAgency/ghidra/blob/45927bb9c33330d8dad3597eb459e2e9a35c3323/Ghidra/Framework/Utility/src/main/java/ghidra/util/SystemUtilities.java#L433
and
https://github.com/NationalSecurityAgency/ghidra/blob/45927bb9c33330d8dad3597eb459e2e9a35c3323/Ghidra/Framework/Utility/src/main/java/ghidra/util/SystemUtilities.java#L470
seem to imply my option will be respected

Ghidra does not change the value of the system property.   Rather, it calls the method getCPUOverride() and then will cap the value returned to a limit of 10.   This is the code that I liked above.

https://github.com/NationalSecurityAgency/ghidra/blob/45927bb9c33330d8dad3597eb459e2e9a35c3323/Ghidra/Framework/Utility/src/main/java/ghidra/util/SystemUtilities.java#L433-L436
Doesn't it get returned right away without being capped here?

Haha.   I knew one of us was mistaken.   Clearly I am still suffering from brain fog today.
Yes, you are correct.

Yeah, that's what I was dreading - hopefully my solution won't involve some nonsense involving multiple clients/programs

Please keep us updated with what you find.  We would like to fix any clear algorithmic mistakes.  It will obviously be more difficult to make architecture-level changes.

I replaced the decompilation with an expensive math calculation with a roughly similar distribution of time per task. The resulting runtime scales almost linearly with number of threads in the pool up to 64.
My suspicion is that some (instance?) component of the Ghidra (Java) client is getting bottlenecked, either:
1.) Reading the database
2.) Sending data to the decompilers
3.) Receiving and parsing data from the decompilers

Based on a dump when running 64 parallel decompilers, I am pretty sure the bottlenecks are in program.database; a snapshot shows 35 of them waiting at various locks on the database records @dragonmacher
      2 	at ghidra.program.database.code.CodeManager.getDefinedDataAt(CodeManager.java:1319)
      4 	at ghidra.program.database.code.CodeManager.getInstructionAt(CodeManager.java:1283)
      9 	at ghidra.program.database.code.CodeUnitDB.getBytes(CodeUnitDB.java:648)
      5 	at ghidra.program.database.code.InstructionDB.getDefaultFallThrough(InstructionDB.java:718)
     10 	at ghidra.program.database.function.FunctionDB.getEntryPoint(FunctionDB.java:314)
      2 	at ghidra.program.database.function.FunctionDB.isFunctionFlagSet(FunctionDB.java:2396)
      1 	at ghidra.program.database.references.ReferenceDBManager.getReferencesFrom(ReferenceDBManager.java:796)
      1 	at ghidra.program.database.symbol.NamespaceManager.getAddressSet(NamespaceManager.java:265)
      1 	at ghidra.program.database.symbol.SymbolDB.getSymbolData1(SymbolDB.java:804)

A list of locks that I observed:
      8 	at ghidra.program.database.code.CodeManager.getCodeUnitContaining(CodeManager.java:1006)
     23 	at ghidra.program.database.code.CodeManager.getDefinedDataAt(CodeManager.java:1319)
     74 	at ghidra.program.database.code.CodeManager.getInstructionAt(CodeManager.java:1283)
     82 	at ghidra.program.database.code.CodeUnitDB.getBytes(CodeUnitDB.java:648)
      1 	at ghidra.program.database.code.DataDB.getNumComponents(DataDB.java:740)
      5 	at ghidra.program.database.code.DataDB.hasMutability(DataDB.java:359)
     52 	at ghidra.program.database.code.InstructionDB.getDefaultFallThrough(InstructionDB.java:718)
      9 	at ghidra.program.database.code.InstructionDB.getFallThrough(InstructionDB.java:193)
      7 	at ghidra.program.database.data.ArrayDB.getDataType(ArrayDB.java:124)
      1 	at ghidra.program.database.data.DataTypeManagerDB.getArrayDataType(DataTypeManagerDB.java:2194)
      1 	at ghidra.program.database.data.DataTypeManagerDB.getBuiltInDataType(DataTypeManagerDB.java:2053)
      2 	at ghidra.program.database.data.DataTypeManagerDB.getCompositeDataType(DataTypeManagerDB.java:2141)
      1 	at ghidra.program.database.data.DataTypeManagerDB.getTypedefDataType(DataTypeManagerDB.java:2171)
      1 	at ghidra.program.database.data.FunctionDefinitionDB.getGenericCallingConvention(FunctionDefinitionDB.java:567)
      6 	at ghidra.program.database.data.PointerDB.getLength(PointerDB.java:188)
      5 	at ghidra.program.database.data.StructureDB.getLength(StructureDB.java:742)
      8 	at ghidra.program.database.function.FunctionDB.getCallFixup(FunctionDB.java:2668)
     12 	at ghidra.program.database.function.FunctionDB.getCallingConventionName(FunctionDB.java:2528)
     86 	at ghidra.program.database.function.FunctionDB.getEntryPoint(FunctionDB.java:314)
      1 	at ghidra.program.database.function.FunctionDB.getLocalVariables(FunctionDB.java:1168)
     11 	at ghidra.program.database.function.FunctionDB.getName(FunctionDB.java:220)
      2 	at ghidra.program.database.function.FunctionDB.getParameters(FunctionDB.java:1130)
      1 	at ghidra.program.database.function.FunctionDB.getReturn(FunctionDB.java:350)
      3 	at ghidra.program.database.function.FunctionDB.getSignatureSource(FunctionDB.java:2442)
     19 	at ghidra.program.database.function.FunctionDB.isFunctionFlagSet(FunctionDB.java:2396)
      2 	at ghidra.program.database.function.FunctionDB.isThunk(FunctionDB.java:110)
      6 	at ghidra.program.database.function.FunctionManagerDB.getFunctionAt(FunctionManagerDB.java:621)
      1 	at ghidra.program.database.function.FunctionManagerDB.getFunctionContaining(FunctionManagerDB.java:647)
     22 	at ghidra.program.database.references.ReferenceDBManager.getReferencesFrom(ReferenceDBManager.java:796)
      2 	at ghidra.program.database.symbol.NamespaceManager.getAddressSet(NamespaceManager.java:265)
      1 	at ghidra.program.database.symbol.SymbolDB.getAddress(SymbolDB.java:116)
      1 	at ghidra.program.database.symbol.SymbolDB.getName(SymbolDB.java:169)
      9 	at ghidra.program.database.symbol.SymbolDB.getParentSymbol(SymbolDB.java:716)
     18 	at ghidra.program.database.symbol.SymbolDB.getSymbolData1(SymbolDB.java:804)
      2 	at ghidra.program.database.symbol.SymbolManager.getPrimarySymbol(SymbolManager.java:1103)
      2 	at ghidra.program.database.symbol.SymbolManager.getSymbols(SymbolManager.java:987)
      2 	at ghidra.program.database.symbol.SymbolManager.getSymbol(SymbolManager.java:752)
     44 	at ghidra.program.database.util.AddressRangeMapDB.getValue(AddressRangeMapDB.java:167)

This makes sense.    We have discussed the idea of moving away from using a single db locking mechanism to use  a read/write locking scheme that allows the reads to be non-blocking.   It is hard to say without testing, but that should be quite beneficial as the number of threads on the system grows.

As an experiment, I refactored ghidra.util.Lock to java.util.concurrent.locks.ReentrantReadWriteLock; for now, all lock.acquire() calls have been refactored to lock.writeLock().lock()
Interestingly, this leads to a roughly 30% speedup when performing batch decompilation. I will slowly change some uses of the writeLock to reads and see how much speed I get from it.

Introducing read locks can get complicated due to the need to synchronize the cache and lazy initialization.  This will force additional synchronized protections.  It is not as simple as just using a different lock on the read methods if they can change internal state (e.g., DB objects).

Interestingly, this leads to a roughly 30% speedup when performing batch decompilation (150s -> 115s on a 25k function binary) with just the refactor. I will slowly change some uses of the writeLock to reads and see how much speed I get from it.

I am interested in seeing what you find.

Introducing read locks can get complicated due to the need to synchronize the cache and lazy initialization. This will force additional synchronized protections. It is not as simple as just using a different lock on the read methods if they can change internal state (e.g., DB objects).

That's definitely true, I quickly found that naively replacing writelock with readlock on 'getters' led to deadlocks of all sorts. Not being able to upgrade the lock makes it pretty tough; I might have to consider ReentrantReadWriteUpdateLock or StampedLock.

I had a few years back had a failed attempt to implement a concurrent read/write lock scheme and had refactored all of the the internal synchronization which was quite extensive and invasive.  At the time I was unable to get a read lock in the same thread that held a write lock (deadlock).  This seemed to contradict the documentation.  This is a necessary capability of the read/write locking since write-lock methods need to invoke public read-lock methods.

It has been slow going. I tried refactoring  ghidra.program.database.code.CodeUnitDB.getBytes(CodeUnitDB.java:648) to use a ReadLock, but quickly found out that CodeManager uses a DbObjectCache which is lazily created, meaning even innocuous sounding functions in it like getCodeUnitAt must be treated carefully.
I'm pretty much compiling, running, and fixing deadlocks as I go. Is there a way to rebuild ghidra without having to recompile the slaspec and javadocs? A typical gradle buildGhidra takes 3 minutes or so, mostly spent on these two tasks.

Is there a way to rebuild ghidra without having to recompile the slaspec and javadocs?

gradle buildGhidra -x zipJavadocs -x sleighCompile

As I indicated above, I have been through this exercise once before and a significant amount of refactoring with synchronization was required.
You should not need to build Ghidra in order to test and you should be fine testing from Eclipse.

The refactors needed to make ReadWrite locks work go all the way down and I've basically given up, sadly.

This is certainly one of the most difficult areas to navigate and change.  Thanks for the effort and the info you have provided.

I couldn't quite leave this alone and made one last moonshot:
1.) Replace ghidra.util.Lock to use a ReentrantReadWriteLock, with acquire and release
2.) Add static boolean  noLock to the Lock. When noLock is true, calls to acquire will just pass through. From the GhidraScript, set noLock while batch decompiling and unset after.
3.) Disable writes to DBObjectCache when noLock is active
As a side note, LinkedList<T> hardCache looks weird here - there is an explicit constructor call which sets this size to 1000, which sounds extremely non-performant.
I then noticed I was deadlocked on DBHandle in Table.java, usually calling getRecord* functions
4.) Added a ReentrantReadWriteLock to DBHandle and replaced all synchronized blocks with read or write locks, appropriately
5.) NodeMgr and BufferMgr throw up frequent exceptions,  due to concurrently calling NodeMgr.releaseNodes.

In case it's helpful to anyone, I worked around this issue by caching the decompiler output on disk: https://github.com/rcslab/AutoCPA/blob/master/scripts/bcpi/BcpiDecompiler.java
It works for me because I frequently re-run my analysis.  Doesn't help on the first run when it has to decompile everything, of course.


Incompatible with pip 18.1

The newest pip release 18.1 finally moved their API around and pipdeptree no longer works.
Traceback (most recent call last):
  File "/home/moritz/.virtualenvs/test-cobra/lib/python3.6/site-packages/pipdeptree.py", line 17, in <module>
    from pip._internal import get_installed_distributions
ImportError: cannot import name 'get_installed_distributions'

Somehow I was under the impression that pipdeptree was not using the internal pip API but that's obviously wrong. Importing from a module marked as _internal seems... less than ideal :grimacing:

I hereby confirm this incompatibility :joy:


[BugFix] Fixed incorrect schema id in CreateReplicaTask

CreateReplicaTask always uses the index id as the schema id, but for tables that have performed fast schema evolution, the schema id is not equal to the index id.
What type of PR is this:

[x] BugFix
[ ] Feature
[ ] Enhancement
[ ] Refactor
[ ] UT
[ ] Doc
[ ] Tool

Does this PR entail a change in behavior?

[ ] Yes, this PR will result in a change in behavior.
[x] No, this PR will not result in a change in behavior.

If yes, please specify the type of change:

[ ] Interface/UI changes: syntax, type conversion, expression evaluation, display information
[ ] Parameter changes: default values, similar parameters but with different default values
[ ] Policy changes: use new policy to replace old one, functionality automatically enabled
[ ] Feature removed
[ ] Miscellaneous: upgrade & downgrade compatibility, etc.

Checklist:

[ ] I have added test cases for my bug fix or my new feature
[ ] This pr needs user documentation (for new or modified features or behaviors)

[ ] I have added documentation for my new feature or new function

[ ] This is a backport pr

Bugfix cherry-pick branch check:

[x] I have checked the version labels which the pr will be auto-backported to the target branch

[x] 3.2
[ ] 3.1
[ ] 3.0
[ ] 2.5

@sevev @wyb PTAL


What about docs?

Let's work on the documentation with jojos suggestions:

[ ] #6
[ ] #7
[ ] #8
[ ] decide if we want to go with something more complex

Thanks!
❤


PDFViewerApplication.findController.executeCommand deprecated?

I've been using this function in my viewer, I updated my pdf.js build to the latest last night and this function is no longer available. I'm a little confused - can someone advise what the replacement function looks like?
In my code I'm using it in this context:
  iframe.contentWindow.PDFViewerApplication.findController.executeCommand('find', {query: term});

It's still there: https://github.com/mozilla/pdf.js/blob/master/web/pdf_find_controller.js#L286. Closing as answered since no more information is provided.