az role assignment: ERROR: A hash conflict was encountered for the role Assignment ID. Please use a new Guid.

This is autogenerated. Please review and update as needed.
Describe the bug
when running az role assignment create , a user may hit hash conflict which he cannot control , only re-run the command.
this is a source of frustration and issue that exists for many years so far.
since there is no API to allow users to pass a GUID or any other mean of uniqueness I believe this is a bug , not feature request.
Also in the Error description , users are asked to "use a new Guid , where in fact there is no option to do so.
Command Name
az role assignment create
Errors:
ERROR: A hash conflict was encountered for the role Assignment ID. Please use a new Guid.

To Reproduce:
Steps to reproduce the behavior. Note that argument values have been redacted, as they may contain sensitive information.

run az role assignment many times and you will hit the issue
issue is already reported since 2020 and has not been addressed since
https://github.com/Azure/azure-cli/issues/13368

Expected Behavior

either provide API so users can pass their own Guid
fix the issue on Az CLI server side and perform retry mechanism if needed
fix the error description as its misleading

Environment Summary
Linux-<IP_ADDRESS>-microsoft-standard-WSL2-x86_64-with-glibc2.31, Ubuntu 20.04.4 LTS
Python 3.10.4
Installer: DEB

azure-cli 2.37.0 *

Extensions:
aks-preview 0.5.91
azure-devops 0.25.0
containerapp 0.3.7

Dependencies:
msal 1.18.0b1
azure-mgmt-resource 21.1.0b1

Additional Context

@jiasli for awareness

There is no fancy logic in Azure CLI to create role assignment name - simply a GUID created with Python's built-in uuid.uuid4() function:
https://github.com/Azure/azure-cli/blob/98f7865402569ec9a6ae05ccdd0d27d5ea20927c/src/azure-cli/azure/cli/command_modules/role/custom.py#L1563-L1564
I tried to hardcode assignment name to a GUID ccfed017-7179-4c62-b6b5-236377f1e9b3 when creating role assignments:
https://github.com/Azure/azure-cli/blob/98f7865402569ec9a6ae05ccdd0d27d5ea20927c/src/azure-cli/azure/cli/command_modules/role/custom.py#L201-L203
Then if I run az role assignment create multiple times, it doesn't give any error:
> az role assignment create --assignee-object-id 9b25c27f-0eb3-43a3-83e4-a513fc3ef7ce --assignee-principal-type ServicePrincipal --role reader --scope /subscriptions/0b1f6471-1bf0-4dda-aec3-cb9272f09590/resourceGroups/azure-cli-test-rg --debug
...
msrest.http_logger: Request URL: 'https://management.azure.com/subscriptions/0b1f6471-1bf0-4dda-aec3-cb9272f09590/resourceGroups/azure-cli-test-rg/providers/Microsoft.Authorization/roleAssignments/ccfed017-7179-4c62-b6b5-236377f1e9b3?api-version=2020-04-01-preview'
msrest.http_logger: Request method: 'PUT'
msrest.http_logger: Request headers:
msrest.http_logger:     'Accept': 'application/json'
msrest.http_logger:     'Content-Type': 'application/json; charset=utf-8'
msrest.http_logger:     'accept-language': 'en-US'
msrest.http_logger:     'Content-Length': '270'
msrest.http_logger:     'User-Agent': 'python/3.10.7 (Windows-10-10.0.22621-SP0) msrest/0.7.1 msrest_azure/0.6.4 azure-mgmt-authorization/0.61.0 Azure-SDK-For-Python AZURECLI/2.41.0 (PIP)'
msrest.http_logger: Request body:
msrest.http_logger: {"properties": {"roleDefinitionId": "/subscriptions/0b1f6471-1bf0-4dda-aec3-cb9272f09590/providers/Microsoft.Authorization/roleDefinitions/acdd72a7-3385-48ef-bd42-f606fba81ae7", "principalId": "9b25c27f-0eb3-43a3-83e4-a513fc3ef7ce", "principalType": "ServicePrincipal"}}
...
msrest.http_logger: Response status: 201
msrest.http_logger: Response headers:
msrest.http_logger:     'Cache-Control': 'no-cache'
msrest.http_logger:     'Pragma': 'no-cache'
msrest.http_logger:     'Content-Length': '889'
msrest.http_logger:     'Content-Type': 'application/json; charset=utf-8'
msrest.http_logger:     'Expires': '-1'
msrest.http_logger:     'x-ms-request-id': 'a34b2413-44b9-49b8-820d-fd5a48212424'
msrest.http_logger:     'X-Content-Type-Options': 'nosniff'
msrest.http_logger:     'Strict-Transport-Security': 'max-age=31536000; includeSubDomains'
msrest.http_logger:     'Set-Cookie': 'x-ms-gateway-slice=Production; path=/; secure; samesite=none; httponly'
msrest.http_logger:     'x-ms-ratelimit-remaining-subscription-writes': '1199'
msrest.http_logger:     'x-ms-correlation-request-id': 'd1aabcbb-cf66-4591-87cf-b12486bcc5c8'
msrest.http_logger:     'x-ms-routing-request-id': 'SOUTHEASTASIA:20221013T112447Z:d1aabcbb-cf66-4591-87cf-b12486bcc5c8'
msrest.http_logger:     'Date': 'Thu, 13 Oct 2022 11:24:47 GMT'
msrest.http_logger: Response content:
msrest.http_logger: {"properties":{"roleDefinitionId":"/subscriptions/0b1f6471-1bf0-4dda-aec3-cb9272f09590/providers/Microsoft.Authorization/roleDefinitions/acdd72a7-3385-48ef-bd42-f606fba81ae7","principalId":"9b25c27f-0eb3-43a3-83e4-a513fc3ef7ce","principalType":"ServicePrincipal","scope":"/subscriptions/0b1f6471-1bf0-4dda-aec3-cb9272f09590/resourceGroups/azure-cli-test-rg","condition":null,"conditionVersion":null,"createdOn":"2022-10-13T11:24:44.0399539Z","updatedOn":"2022-10-13T11:24:44.3993881Z","createdBy":null,"updatedBy":"0d504196-1423-4569-9a6e-15149656f0ee","delegatedManagedIdentityResourceId":null,"description":null},"id":"/subscriptions/0b1f6471-1bf0-4dda-aec3-cb9272f09590/resourceGroups/azure-cli-test-rg/providers/Microsoft.Authorization/roleAssignments/ccfed017-7179-4c62-b6b5-236377f1e9b3","type":"Microsoft.Authorization/roleAssignments","name":"ccfed017-7179-4c62-b6b5-236377f1e9b3"}

This means the Role Assignments - Create API is idempotent.
It seems to be ARM RBAC service issue. Let's discuss internally.

You will not be able to reproduce this by running it many times as GUID conflict are hard to reproduce.
The bug I've opened is around the fact that on one hand the error exists and happens quite often to teams which are running many workload, while the error message provided is wrong.
The error message claims "you should call the command with new GUID" while in fact we don't have ANY API to pass the GUID -> and that is the real bug here.
If for example we could have passed a GUID , we would always pass the same GUID for the same role assignment , hence avoiding this conflict
This is actually how ARM is handling this - they let users to pick either a name or a constant GUID/String to always use the same one for the same role assignment. -> that is the ask here .

https://github.com/Azure/azure-cli/pull/24324 worked around this issue by introducing --name in az role assignment create. You can now provide your own GUID as --name to have a predictable role assignment created and bypass this service issue.

@jiasli  great ! thanks a lot !


backend: use peer address as grpc addr for tiflash store

What problem does this PR solve?
close #391
What is changed and how it works?
Use store.GetPeerAddress as the address to open gRPC connect and add a integration test for tiflash
Check List
Tests

Unit test
Integration test
Manual test (add detailed scripts or steps below)
No code

Side effects
Related changes

/run-all-tests


Rate app componant for Android

I see there is one for iOS.

Audit app is what I'm lookng for I guess


add ignore_tags and non_splitting_tags options

Hey @wikiti 👋

DeepL Pro API:

https://www.deepl.com/api.html

add ignore_tags option
add non_splitting_tags option

I'm currently trying to get deepl-rb working with i18n-tasks to have feature parity with the google translate setup the ignore_tags option is important.
What do you think about this addition? I'm missing why my vcr response is ASCII-8BIT instead of UTF-8 (and readable).

Hey @neumayr!
I've merged this PR after performing minor tweaks to your contributions. I have:

Allowed conversions from array to strings for ignore_tags and non_splitting_tags.
Updated the rspec tests to run a sample using ignore_tags (this query parameter was not being used).
Updated the README.md file.

Thank you for your contribution!


LNBits installation displays empty url screen and no failure if PostgreSQL assertion was not met

Tried to install btcpay & lnbits in one go.
the BTCPay installation failed because of the failed assertion below. It displayed a screen that the installation failed.
Afterwards LNBits was being installed and the same assertion failed (see below).
But then a screen appeared that LNBits is available under http:// and not that the installation failed like with BTCPay.
FAIL - Was not able to start PostgreSQL service
○<EMAIL_ADDRESS>- PostgreSQL Cluster 15-main
     Loaded: loaded<EMAIL_ADDRESS>enabled; preset: enabled)
     Active: inactive (dead)
     Assert: start assertion failed at Mon 2024-09-30 13:00:56 BST; 30s ago
             AssertPathExists=/etc/postgresql/15/main/postgresql.conf was not met

Sep 30 12:58:16 raspberrypi systemd[1]: Starting<EMAIL_ADDRESS>- PostgreSQL Cluster 15-main...
Sep 30 12:58:18 raspberrypi systemd[1]: Started<EMAIL_ADDRESS>- PostgreSQL Cluster 15-main.
Sep 30 12:58:25 raspberrypi systemd[1]: Stopping<EMAIL_ADDRESS>- PostgreSQL Cluster 15-main...
Sep 30 12:58:25 raspberrypi systemd[1]<EMAIL_ADDRESS>Deactivated successfully.
Sep 30 12:58:25 raspberrypi systemd[1]: Stopped<EMAIL_ADDRESS>- PostgreSQL Cluster 15-main.
Sep 30 12:58:30 raspberrypi systemd[1]<EMAIL_ADDRESS>Starting requested but asserts failed.
Sep 30 12:58:30 raspberrypi systemd[1]: Assertion failed for<EMAIL_ADDRESS>- PostgreSQL Cluster >
Sep 30 13:00:56 raspberrypi systemd[1]<EMAIL_ADDRESS>Starting requested but asserts failed.
Sep 30 13:00:56 raspberrypi systemd[1]: Assertion failed for<EMAIL_ADDRESS>- PostgreSQL Cluster >
Failed to start lnbits.service: Unit lnbits.service not found.

In PR #4768 I added a check if Postgres is already installed - preventing it to try to install again. This cleared the Assertion error on my lnbits install. @cstenglein please check if this also fixes your scenario

This cleared the Assertion error on my lnbits install

My issue was primary with the "success" screen on a unsuccessful installation:

Good that you found another issue there as well :+1:

TODOs for @rootzoll:

do the same fix for BTCPayServer
add error screen if LNBits install fails

Fix didn't work unfortunately
psql: error: connection to server on socket "/var/run/postgresql/.s.PGSQL.5432" failed: No such file or directory
        Is the server running locally and accepting connections on that socket?
# postgresConfig failed -> SELECT datname FROM pg_database;
Failed to start lnbits.service: Unit lnbits.service not found.

I tested in 4568-lnbits branch on my x86 VM: in services SSH menu selected both BTCPay & LNBits to install .. and they both ran thru to install.
But it also was a fresh install .. is your scenario with old data in PostgresDB, maybe?

One issue is at least, that the off command in the postgres script only uninstalls postgresql https://github.com/raspiblitz/raspiblitz/blob/dev/home.admin/config.scripts/bonus.postgresql.sh#L227C1-L227C32
The on command installs postgresql-VERSION, thus postgresql-15 https://github.com/raspiblitz/raspiblitz/blob/dev/home.admin/config.scripts/bonus.postgresql.sh#L35

Tried to do the install with the new Scripts
Still didn't work but most likely the issue is on my side.
The screen for lnbits didn't display after the failed install tough, so that works now. 👍


new feature request：runSpeed(int speed); like AccelStepper

I was wondering if it would be possible to add a method that runs based on a given speed

runSpeed() from AccelStepper is not using acceleration. So not sure, if this is requested.
A possible implementation (without error checking and not compiler tested):
func runSpeed(int32_t speed_in_Hz, FastAccelStepper *stepper) {
    if (speed == 0) {
         stepper->stopMove();
    }
    else {
         if (stepper > 0) {
              stepper->setSpeedInHz(speed_in_Hz);
              stepper->runForward();
         }
         else {
              stepper->setSpeedInHz(-speed_in_Hz);
              stepper->runBackward();
         }
    }
}

Now this implementation for every setSpeedInXxx() variant (without stepper parameter) and maintaining another variable for the user callled setSpeedInXxx() for the absolute maximum speed.

Thank you for your reply. I have just tried the idea you mentioned. It can basically achieve the function, but compared to accelstepper's runspeed(), the motor movement is delayed.

which is the delay ?
Any change in speed takes approx. 10ms for being applied (forward planning in 4ms cyclic task). From one speed value to the next value, acceleration/deceleration is performed, which can take even longer.

It may be more than 10ms. I visually estimated that the corresponding delay is between 20~50ms. Although I adjusted the acceleration very large, the delay was still not reduced. I am using the as5600 magnetic encoding chip to obtain the angular velocity. The delay under runSpeed() is very small, and it can almost be considered a real-time response. Part of your code is as follows:
if ((int)FinalMotorSpeed == 0) {
stepper->stopMove();
} else {
if ((int)FinalMotorSpeed > 0) {
stepper->setSpeedInHz((int)FinalMotorSpeed);
stepper->runForward();
} else {
stepper->setSpeedInHz(-(int)FinalMotorSpeed);
stepper->runBackward();
}
}

I had the forward planning time incorrectly in mind. It is actually 20ms.
There could be now two options:

Forward planning time could be a configurable option. Perhaps down to 5ms. This could cause issues on slower AVR parts or on esp32 due to task blocking.
There is kind of an equivalent to runSpeed() by using low level API calls on queue level. But this is more difficult to be controlled by application and requires, that the application ensures, that the queue does not run out of commands. Acceleration and deceleration from ramp generator would not be available anymore. If queue filling is ensured, then the reaction time can come down to two times the command length.

First point 1 is tracked here: #253
For point 2, there has not been any update.
=> Close this issue


CNF-14648: Fix errors during TemplateID update

This PR fixes the following:

race condition after updating the templateId field, the version is not updated in time for the next update to succeed. updating the templateId field first if needed and requeue to continue with validation. Wait a bit for the cache to updated with the new object version. Thanks @Missxiaoguo for the solution
uuid passed by the user was not properly updated and always regenerated.

/lgtm

/approve


UMAP segmentation fault

umap.UMAP(n_neighbors=11, metric='jaccard',
                      min_dist=0.5).fit(X)

Error - segmentation fault
umap-learn 0.5.1 - Bad
umap-learn 0.5.0 - Bad
umap-learn 0.4.6 - Good
But if you increase it n_neighbors to 150, then everything works on new versions.

That's not good :-(
Can you share data or provide a small reproducer for this?

That's not good :-(
Can you share data or provide a small reproducer for this?

Yes, I can share data. Where can I send them to you?

You can reach me at leland DOT mcinnes AT gmail DOT com.

leland DOT mcinnes AT gmail DOT com

Im send

Thanks -- the error is reproducible, so that's good. I'm trying to track down what the actual problem is, and that might take a little while.

So this was actually quite complicated because it is a subtle issue. In the end it really comes down to the nature of your data and the construction of random projection trees to initialize a nearest neighbour search combining with floating point error accumulation that ends up causing bad things to happen (tree splits not occurring properly). A long term fix for that is a little harder, but there is a very simple short term fix:
Since you are using Jaccard distance, which only cares about whether entries are non-zero or not, you can simply pass in a binarized version of your data and everything should work.
umap.UMAP(n_neighbors=11, metric='jaccard',
                      min_dist=0.5).fit(X != 0)

And errors on other metrics.
New versions don't work when n_neighbors <500
And the old ones don't work when n_neighbors> 20
Very strange .....

Any angular metric will have similar issues. Your data is very skewed. Try using StandardScaler or similar on your data first to at least get the different columns in the same value ranges.


Cannot find the MMM-voice.dic

ERROR: "dict.c", line 275: Failed to open dictionary file 'modules/MMM-voice/MMM-voice.dic' for reading: No such file or directory
xps:~/MagicMirror$ more modules/MMM-voice/debug.log
INFO: pocketsphinx.c(153): Parsed model-specific feature parameters from /usr/local/share/pocketsphinx/model/en-us/en-us/feat.params
Current configuration:
[NAME]                  [DEFLT]         [VALUE]
-agc                    none            none
-agcthresh              2.0             2.000000e+00
-allphone
-allphone_ci            yes             yes
-alpha                  0.97            9.700000e-01
-ascale                 20.0            2.000000e+01
-aw                     1               1
-backtrace              no              no
-beam                   1e-48           1.000000e-48
-bestpath               yes             yes
-bestpathlw             9.5             9.500000e+00
-ceplen                 13              13
-cmn                    live            batch
-cmninit                40,3,-1         41.00,-5.29,-0.12,5.09,2.48,-4.07,-1.37,-1.78,-5.08,-2.05,-6.45,-1.42,1.17
-compallsen             no              no
-dict                                   modules/MMM-voice/MMM-voice.dic
-dictcase               no              no
-dither                 no              no
-doublebw               no              no
-ds                     1               1
-fdict
-feat                   1s_c_d_dd       1s_c_d_dd
-featparams
-fillprob               1e-8            1.000000e-08
-frate                  100             100
-fsg
-fsgusealtpron          yes             yes
-fsgusefiller           yes             yes
-fwdflat                yes             yes
-fwdflatbeam            1e-64           1.000000e-64
-fwdflatefwid           4               4
-fwdflatlw              8.5             8.500000e+00
-fwdflatsfwin           25              25
-fwdflatwbeam           7e-29           7.000000e-29
-fwdtree                yes             yes
-hmm                                    /usr/local/share/pocketsphinx/model/en-us/en-us
-input_endian           little          little
-jsgf
-keyphrase
-kws
-kws_delay              10              10
-kws_plp                1e-1            1.000000e-01
-kws_threshold          1e-30           1.000000e-30
-latsize                5000            5000
-lda
-ldadim                 0               0
-lifter                 0               22
-lm                                     modules/MMM-voice/MMM-voice.lm
-lmctl
-lmname
-logbase                1.0001          1.000100e+00
-logfn
-logspec                no              no
-lowerf                 133.33334       1.300000e+02
-lpbeam                 1e-40           1.000000e-40
-lponlybeam             7e-29           7.000000e-29
-lw                     6.5             6.500000e+00
-maxhmmpf               30000           30000
-maxwpf                 -1              -1
-mdef
-mean
-mfclogdir
-min_endfr              0               0
-mixw
-mixwfloor              0.0000001       1.000000e-07
-mllr
-mmap                   yes             yes
-ncep                   13              13
-nfft                   512             512
-nfilt                  40              25
-nwpen                  1.0             1.000000e+00
-pbeam                  1e-48           1.000000e-48
-pip                    1.0             1.000000e+00
-pl_beam                1e-10           1.000000e-10
-pl_pbeam               1e-10           1.000000e-10
-pl_pip                 1.0             1.000000e+00
-pl_weight              3.0             3.000000e+00
-pl_window              5               5
-rawlogdir
-remove_dc              no              no
-remove_noise           yes             yes
-remove_silence         yes             yes
-round_filters          yes             yes
-samprate               16000           1.600000e+04
-seed                   -1              -1
-sendump
-senlogdir
-senmgau
-silprob                0.005           5.000000e-03
-smoothspec             no              no
-svspec                                 0-12/13-25/26-38
-tmat
-tmatfloor              0.0001          1.000000e-04
-topn                   4               4
-topn_beam              0               0
-toprule
-transform              legacy          dct
-unit_area              yes             yes
-upperf                 6855.4976       6.800000e+03
-uw                     1.0             1.000000e+00
-vad_postspeech         50              50
-vad_prespeech          20              20
-vad_startspeech        10              10
-vad_threshold          3.0             3.000000e+00
-var
-varfloor               0.0001          1.000000e-04
-varnorm                no              no
-verbose                no              no
-warp_params
-warp_type              inverse_linear  inverse_linear
-wbeam                  7e-29           7.000000e-29
-wip                    0.65            6.500000e-01
-wlen                   0.025625        2.562500e-02
INFO: feat.c(715): Initializing feature stream to type: '1s_c_d_dd', ceplen=13, CMN='batch', VARNORM='no', AGC='none'
INFO: acmod.c(162): Using subvector specification 0-12/13-25/26-38
INFO: mdef.c(518): Reading model definition: /usr/local/share/pocketsphinx/model/en-us/en-us/mdef
INFO: mdef.c(531): Found byte-order mark BMDF, assuming this is a binary mdef file
INFO: bin_mdef.c(336): Reading binary model definition: /usr/local/share/pocketsphinx/model/en-us/en-us/mdef
INFO: bin_mdef.c(516): 42 CI-phone, 137053 CD-phone, 3 emitstate/phone, 126 CI-sen, 5126 Sen, 29324 Sen-Seq
INFO: tmat.c(149): Reading HMM transition probability matrices: /usr/local/share/pocketsphinx/model/en-us/en-us/transition_matrices
INFO: acmod.c(113): Attempting to use PTM computation module
INFO: ms_gauden.c(127): Reading mixture gaussian parameter: /usr/local/share/pocketsphinx/model/en-us/en-us/means
INFO: ms_gauden.c(242): 42 codebook, 3 feature, size:
INFO: ms_gauden.c(244):  128x13
INFO: ms_gauden.c(244):  128x13
INFO: ms_gauden.c(244):  128x13
INFO: ms_gauden.c(127): Reading mixture gaussian parameter: /usr/local/share/pocketsphinx/model/en-us/en-us/variances
INFO: ms_gauden.c(242): 42 codebook, 3 feature, size:
INFO: ms_gauden.c(244):  128x13
INFO: ms_gauden.c(244):  128x13
INFO: ms_gauden.c(244):  128x13
INFO: ms_gauden.c(304): 222 variance values floored
INFO: ptm_mgau.c(475): Loading senones from dump file /usr/local/share/pocketsphinx/model/en-us/en-us/sendump
INFO: ptm_mgau.c(499): BEGIN FILE FORMAT DESCRIPTION
INFO: ptm_mgau.c(562): Rows: 128, Columns: 5126
INFO: ptm_mgau.c(594): Using memory-mapped I/O for senones
INFO: ptm_mgau.c(837): Maximum top-N: 4
INFO: phone_loop_search.c(114): State beam -225 Phone exit beam -225 Insertion penalty 0
ERROR: "dict.c", line 275: Failed to open dictionary file 'modules/MMM-voice/MMM-voice.dic' for reading: No such file or directory
Originally posted by @nosee818 in https://github.com/fewieden/MMM-voice/issues/32#issuecomment-439326570

Can you type ls -la in your terminal inside the MMM-voice directory and post the content.

Some issue, first-time keeps "Initialising" status long time, even after a weekend.
then I stop the mirror, add debug true, the status show error / undefined.
The debug file show can't find **MMM-voice.dic file.

Could you please delete words.json and restart the mirror. It should then try to regenreate MMM-voice.dic

Thanks for your reply.
Actually, I read the closed issue about the DIC file.
I have deleted the words.json / error.log/  debug.log several times and restarted mirror.  Still same error.

After you deleted the files, could you see some of the following messages in the terminal where you start the MagicMirror?
Starting module helper: MMM-voice
MMM-voice: Checking files.
MMM-voice: Generating dictionairy and language model.
MMM-voice: Couldn't save words.json!
MMM-voice: Saved words.json successfully.

You could also add a console.log('Failed to generate dic file', err); in between line 199 and 200

There is no any message in the terminal when I deleted the words.json file.
First time hang on Initialising long time.
Then I shop and re-run, shows error and no generate a file named console.log.

Did you add the code line I posted in the previous comment?
Don't forget to always delete words.json before you start as long as we're debugging it.

It looks like a problem with the dependency lmtool, it was not updated in over 3 years and it might conflict now with some of the other dependencies. There is a work around though.
Create a file words.txt with all words from words.json in the following format:
HELLO
WORLD
MAGIC
MIRROR

Then uplodád the file here http://www.speech.cs.cmu.edu/tools/lmtool-new.html
After that extract the file you can download there. Rename the .dic and .lm file to MMM-voice.dic and MMM-voice.lm and place them both in the module directory. Restart your mirror without deleting words.json.

Good news! Looks like it's working. I so appreciate your patience.
Starting module: MMM-Clash-Royale
Connecting socket for: MMM-voice
Starting module helper: MMM-voice
Sockets connected & modules started ...
Launching application.
Create new calendar fetcher for url: https://www.officeholidays.com/ics/ics_country_code.php?iso=CN - Interval: 300000
Create new news fetcher for url: https://investors.delltechnologies.com/rss/news-releases.xml - Interval: 300000
Get RANDOM_DECK data for url http://www.clashapi.xyz/api/random-deck
MMM-voice: Checking files.
MMM-voice: Starting pocketsphinx.
MMM-voice has recognized: TO IS
MMM-voice has recognized: GO
MMM-voice has recognized: GO
MMM-voice has recognized: GO
MMM-voice has recognized: THE
MMM-voice has recognized: WHO
MMM-voice has recognized: WHO WHO
MMM-voice has recognized: THE
MMM-voice has recognized: UP
MMM-voice has recognized: MAGIC MIRROR
MMM-voice has recognized: TO VOICE
MMM-voice has recognized: TO
MMM-voice has recognized: TO
MMM-voice has recognized: TO MIRROR
MMM-voice has recognized: MIRROR
MMM-voice has recognized: MIRROR THE WHO MIRROR

Is there have a way can update imtool or fixed the issue locally?

Having the exact same issue verbatim (on Ub16.04)...interestingly, I installed just 2 days ago on 18.04 without issue but reloaded due to not being able to get snowboy to install.
the lmools keeps throwing an error for the corpus file, regardless of encoding. Can you please just post the generated files? Nice easter egg, btw.

It doesn't make sense to upload those files, as they need to look different based on the modules you have installed. Did you try to download them from the link above? https://github.com/fewieden/MMM-voice/issues/34#issuecomment-442219951

It doesn't make sense to upload those files, as they need to look different based on the modules you have installed. Did you try to download them from the link above? #34 (comment)

I had figured that due to the way you recommended to manually generate it, that it would be a static file. I guess it could be, until you add other modules.
I tired the method you outlined in the link above, that's whar I was talking about with the corpus error. It kept failing no matter what text file I fed it.
I got the files from another user, not I get further (says "Ready" and "Undefined") so I am troubleshooting that bit.


Support windows

It would be helpful to support windows if possible. It looks like the upstream repo has a conda recipe for a windows build:
https://github.com/choderalab/openmmtools/tree/master/devtools/conda-recipe
and there is a windows build on the omnia channel:
https://anaconda.org/omnia/openmmtools

@synapticarbors : Looking over meta.yaml, it looks like the blocker might be ambertools, which does not have win builds.
However, I can't find any instances of using ambertools in the openmmtools source. Do you want to try submitting a PR to remove ambertools and the skip for win from the recipe?

Also tagging in @mikemhenry who can hopefully help here!

However, I can't find any instances of using ambertools in the openmmtools source.

I am not sure either, but take into account that ambertools provides many packages and executables that are not covered by that search query.

@synapticarbors can you check if the latest packages work for your purposes on Windows?


Publish 8.7-rc-4

Publish 8.7-rc-4

@bot-gradle test and merge

The merge queue build has started. Click here to see all failures if any.


Payment to specific countries options (like iDeal)

Would it be an idea to "show" a payment method if either the billing OR Shipping address contain the country used for validation? (I think it is billing address now only) ...
95% orders from and for the same country
5% order whilst abroad or on holiday and has it delivered to their country (living abroad, grey nomads)

Hi Sean,
It is only possible to use one country to retrieve the payment methods. We have chosen to use the country in the billing address. Magento does the same with the setting Payment to Specific countries.
Regards,
Rik
Adyen

We have made a change so ideal will be shown if it is in the directory lookup. see https://github.com/Adyen/magento/commit/c66574bb13ca31e54edf2fbea3e915a2dc709223


Update README.md

https://cmake.org/cmake/help/latest/variable/CMAKE_MSVC_RUNTIME_LIBRARY.html
MultiThreadedRelease is not exist

When using a Windows environment, there were macros with too many arguments in the macros.hpp file, so /Zc:preprocessor had to be added. I think it would be a good idea to update this as well.


[adc_ctrl] Ensure minimum power-up time is adhered

This supersedes #18178.
It basically makes sure we are on the safe side by checking that the power down signal stays low long enough, and that the channel select signal remains stable throughout the entire period. The SVA is updated so that it only samples input signals in order to make the check explicit.
The RTL is adapted accordingly to fulfill this requirement.
CC @meisnere

The ROM CI failures are unrelated to this change, hence I am merging this.
@zi-v let me know if further alignment on the AST SVA is needed.


GPIO Page not displaying data in some columns

As per below screenshot - GPIO page - Type, command and args columns are not displaying information despite being defined in the EEPROM cape-inputs.json

And how they appear within FPP

EEPROM file attached.
HE123D-SN-de97de6c9064-v6.0.eeprom.zip

fixed in 42b721ac917bde67099a410d94f6ddf7828095db
https://github.com/computergeek1507/CapeEEPROMViewer/suites/8689977484/artifacts/391965047

Looking good at my end - thanks for the fix


Organize the tests in a proper way

[kind/Enhancement]

Which functionality do you think we should update/improve?
Tests
Why is this needed?
The current test is not properly organized and issues are encountered while improving the test scenarios.
Inspired from - https://testing.googleblog.com/2015/04/just-say-no-to-more-end-to-end-tests.html

Current test tree structure
$ tree -L 2
.
├── OWNERS
└── e2e
    ├── README.md
    ├── cmp_sub_test.go
    ├── cmp_test.go
    ├── component.go
    ├── e2e.go
    ├── e2e_test.go
    ├── examples
    ├── helper
    ├── java_test.go
    ├── json_test.go
    ├── link_test.go
    ├── login_test.go
    ├── service_test.go
    ├── source_test.go
    ├── template_cleantest_test.go
    ├── utils.go
    └── watch_test.go

3 directories, 16 files

Proposal
$ tree -L 2
.
├── OWNERS
├── e2e-scenarios
│   └── e2e_test.go
├── helper
│   ├── helper_filesystem.go
│   ├── helper_generic.go
│   ├── helper_oc.go
│   └── helper_run.go
├── integration
│   ├── cmp_sub_test.go
│   ├── cmp_test.go
│   ├── component.go
│   ├── e2e_test.go
│   ├── integration_test.go
│   ├── java_test.go
│   ├── json_test.go
│   ├── link_test.go
│   ├── login_test.go
│   ├── service_test.go
│   ├── source_test.go
│   └── watch_test.go
├── test-src
│   ├── binary
│   └── source
└── test-template
    └── template_cleantest_test.go

7 directories, 19 files

Integration test - It covers only the feature test. For example in json format feature only that feature will be tested like odo url create -o json, odo storage create -o json, odo project list -o json ...
Purpose: Just to make sure the feature works fine and doesn't break any existing feature.
End to End test - It covers the mostly the real time user specific scenario. For example as a user i can do
1. login
2. Create component from source like local or git
3. Push component with watch
4. link component
5. mount storage
6. create service
7. Delete project
8. logout

Purpose : Just to make sure as a single run all the feature works fine
So don't mix integration test with end to end test.

├── test-src

what is test-src? Are those the sample application on which we run tests?

/kind testing
/assign @amitkrout

├── test-src

what is test-src? Are those the sample application on which we run tests?

Yes, Will put those binary and source (Which are currently in examples directory) in this directory.

├── test-src

what is test-src? Are those the sample application on which we run tests?

Yes, Will put those binary and source (Which are currently in examples directory) in this directory.

not sure if test-src is a good name for that

@girishramnani @kadel Doc [1] will be the reference for writing the test from scratch.Tomorrow i will categorize the integration test and update here with more details.
[1] https://docs.google.com/document/d/1RKva9lvIR6dLmDm2tlqmp8V4nFjlrXdulOQ4FSmLkr8/edit?ts=5c715ffd#heading=h.eqonbgz6nece

Need some more time to update the new test workflow in a separate google doc then will share with the team for feedback.

Need some more time to update the new test workflow in a separate google doc then will share with the team for feedback.

lets figure out the structure first

Need some more time to update the new test workflow in a separate google doc then will share with the team for feedback.

Let's figure out the structure first.
We need to keep writing tests, and we need to know where to put what test

I ma looking structure like this
.
├── OWNERS
├── e2e-scenarios
├── helper
├── integration
├── examples
└── test-template

Which folder contains what
e2e-scenarios - End to end scenario
helper - Only contains library
integration - Component lavel testing
examples - Binary and source for test scenario
test-template - Only a demo test of how to write test
So total no of test suite will be used  - 3
E2E test
Integration Test
Test template
Why need 3 testsuite - To solve the propose what they are created for.
Where they run -
Integration test - Runs only on prs
E2E test + Integration test - Runs only in nightly
Test template - This will only be available for manual run just to see how to write clean test
@girishramnani @mohammedzee1000 @kadel does it make sense ?

E2E test + Integration test - Runs only in nightly

I think that we will be able to run e2e on PRs. Because of the nature of our project e2e tests are going to be faster than integration tests. If we run them in a separate job it won't add any time to the total duration of the test.
In general, this looks good. Now the main question. How do we move to this new format?

I think that we will be able to run e2e on PRs. Because of the nature of our project e2e tests are going to be faster than integration tests. If we run them in a separate job it won't add any time to the total duration of the test.

No doubt of E2E duration will be less than the whole integration test duration but once integration test start growing bigger then E2E test duration matters a lot because e2e also keeps updated time to time.  That's why i kept it in a separate job to avoid any future rework of split, moreover the nightly job will be like a release bits of master.

In general, this looks good. Now the main question. How do we move to this new format?

This should go in a single pr, no followup pr unless there is a hard requirement. moreover i am in the favor of single mindset and one time effort so that it completely replace the old structure. I will draft a pr.


Error with private key | Not OpenSSH private key format

I am getting error while running the main.py file on local bench. I have updated the config.yml to point to new relic and SDFC.
Getting error "Not OpenSSH private key format" in load_ssh_private_key method
Description
Getting "Not OpenSSH private key format" error in "load_ssh_private_key" method of "ssh.py"
I have created the private key and certificate using this link  also I tried other methods too but this one look official.
Still the application is throwing this exception.
I have added the log statements to confirm that the file content was read.
Steps to Reproduce

Clone the solution.
Update the config.yml to with new relic keys.
Create a new self signed certificate using this documentation and upload it on the connected app.
Update the config.yml to use the private key created in above step.
Run the solution

Expected Behavior
The solution shouldn't throw exception.
Relevant Logs / Console output
2024-05-23 13:44:10,065 (28128/MainThread) newrelic.core.agent INFO - New Relic Python Agent (9.7.0)
2024-05-23 13:44:15,475 (28128/NR-Activate-Session/New Relic Salesforce Exporter) newrelic.core.agent_protocol INFO - Reporting to: https://rpm.newrelic.com/accounts//applications/
{"message": "Integration start. Using program arguments []", "timestamp":<PHONE_NUMBER>475, "level": "info"}
{"message": "Cache disabled", "timestamp":<PHONE_NUMBER>491, "level": "info"}
{"message": "Running instance "sfdc-logs"", "timestamp":<PHONE_NUMBER>491, "level": "info"}
{"message": "unknown exception occurred: 'Authenticator' object has no attribute 'instance_name'", "timestamp":<PHONE_NUMBER>491, "level": "error"}
Traceback (most recent call last):
File "\newrelic-salesforce-exporter\src\newrelic_logging\auth.py", line 114, in authenticate_with_jwt
key = serialization.load_ssh_private_key(private_key.encode(), password=b'')
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "\newrelic-salesforce-exporter\sfdcexport\Lib\site-packages\cryptography\hazmat\primitives\serialization\ssh.py", line 614, in load_ssh_private_key
raise ValueError("Not OpenSSH private key format")
ValueError: Not OpenSSH private key format
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\Application_Development\Personal\newrelic-salesforce-exporter\src_main_.py", line 278, in
main()
File "C:\Application_Development\Personal\newrelic-salesforce-exporter\sfdcexport\Lib\site-packages\newrelic\packages\wrapt\wrappers.py", line 598, in call
return self.self_wrapper(self.wrapped, self.self_instance,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Application_Development\Personal\newrelic-salesforce-exporter\sfdcexport\Lib\site-packages\newrelic\api\background_task.py", line 111, in wrapper
return wrapped(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^
File "\newrelic-salesforce-exporter\src_main.py", line 272, in main
run(config, event_type_fields_mapping, numeric_fields_list)
File "\newrelic-salesforce-exporter\src_main.py", line 213, in run
run_once(
File "\newrelic-salesforce-exporter\src_main_.py", line 165, in run_once
).run()
^^^^^
File "\newrelic-salesforce-exporter\src\newrelic_logging\integration.py", line 53, in run
raise e
File "\newrelic-salesforce-exporter\src\newrelic_logging\integration.py", line 37, in run
instance.harvest(session)
File "\newrelic-salesforce-exporter\src\newrelic_logging\instance.py", line 24, in harvest
self.api.authenticate(session)
File "\newrelic-salesforce-exporter\src\newrelic_logging\api.py", line 81, in authenticate
self.authenticator.authenticate(session)
File "\newrelic-salesforce-exporter\src\newrelic_logging\auth.py", line 197, in authenticate
self.authenticate_with_jwt(session)
File "\newrelic-salesforce-exporter\src\newrelic_logging\auth.py", line 116, in authenticate_with_jwt
raise LoginException(f'authentication failed for {self.instance_name}. error message: {str(e)}')
^^^^^^^^^^^^^^^^^^
AttributeError: 'Authenticator' object has no attribute 'instance_name'. Did you mean: 'instance_url'?
2024-05-23 13:44:15,516 (28128/MainThread) newrelic.core.agent INFO - New Relic Python Agent Shutdown
Your Environment
Local bench

ex: Browser name and version:
ex: Operating System and version:

Additional context

@Ashutosh2547 , the library we use to load the private key expects the key to be in the "OpenSSH Private Key Format".  Here are two resources I found when investigating this for another customer that talk about this format:

https://coolaj86.com/articles/the-openssh-private-key-format/
https://www.thedigitalcatonline.com/blog/2021/06/03/public-key-cryptography-openssh-private-keys/

My understanding is that ssh-keygen on MacOS defaults to this format. You will know it's an OpenSSH format private key if you see this in the encoded key.
-----BEGIN OPENSSH PRIVATE KEY-----
Could you try creating a key in this format?  I believe you will need to update the connected app settings with the new key and certificate as well but after doing that, I think the exporter will be able to load the key.

Generating the secret using below steps resolved the issue for me. Below are the steps (recommended by ChatGPT 💯 )
Generate an OpenSSH Private Key
ssh-keygen -t rsa -b 2048 -m PEM -f server.key
Convert the OpenSSH Private Key to PEM Format
openssl rsa -in server.key -outform PEM -out server.pem
Generate a Self-Signed Certificate
openssl req -new -x509 -key server.pem -out server.crt -days 365
After creating the secret use the path of server.pem in the config.yml in corresponding to private_key.

Another alternative that works without following the above steps is to comment/remove the below line from [auth.py](https://github.com/newrelic/newrelic-salesforce-exporter/blob/main/src/newrelic_logging/auth.py) file
 key = serialization.load_ssh_private_key(private_key.encode(), password=b'')
Even if we create the secret using the Salesforce recommended method , commenting/removing the above line would eliminate the issue.

Generate an OpenSSH Private K

For me just converting to openSSH and point to it in confir.yml works
ssh-keygen -p -f <filename>

@Ashutosh2547 that line is historical and I'm not sure why that particular serialization method was chosen.  What type of key is generated with the recommended method you pointed to?  It seems like we should align with whatever is in the Salesforce docs.
Also, are you saying that if you just use the raw result of this line that everything works?

@Ashutosh2547 any updates on my questions above?  Thanks.

Hi @Ashutosh2547 Any update?  I will go ahead and close this issue next week if there aren't any updates at this time.  Feel free to reopen if necessary.

Hey @sdewitt-newrelic ,
You may close this issue. I will reopen this if needed.
Thank you

Thanks @Ashutosh2547. Closing per above.


Bump to convex 1.3.1 and add format param to api calls

Got go test to pass!
Uses the unreleased record type, but 🤷

looks good; the build on main is broken right now because an upstream dep was yanked so I'm going to fix that, rebase your PR, then merge

merging proved difficult; I'm just cherry picking the commit to fix the build


Scripts

Only about 1/4 of texts use @rendition to specify scripts. I suspect it is
because scripts are represented with unreadable identifiers from OpenTheso, as
in rendition="class:83225 maturity:83213". The relevant page on OpenTheso is
https://opentheso.huma-num.fr/opentheso/?idt=th347 (links in the EGD are no
longer valid).
People apparently do not know what they should choose for maturity, and there
are scripts for which the available values of maturity make no sense (e.g.
"Chinese script"), so we could make it optional.
For script classes, we currently have a hierarchy e.g. /Brāhmī and derivatives/southern class Brāhmī/Tamil script/Tamil with Grantha. This
hierarchy is not apparent in the encoding used in inscriptions, and people can
choose either a class that has children (e.g. "Tamil script") or a class that
does not (e.g. "Tamil with Grantha"). I do not think they see the difference,
if only because the contextual help generated from our schemas does not make it
explicit. Things would probably be clearer if the hierarchy was flattened a bit.
To simplify the encoding, I suggest we allow script names (or ISO 15924 script codes,
when available) in place of integers. For instance, we could have
rendition="class:Sundanese" or rendition="class:Sund" or even
rendition="Sund", which would then be mapped to the internal id class:57470 that is
used in OpenTheso.

Thanks for noticing that the links in the EGD are no longer valid. I've noted this for the next version.
Script maturity is for use only with the class "Brahmi and derivatives" (and its subcategories); for any other script classes it is not optional but "forbidden". For Brahmi, it is mandatory.
For the hierarchy of script classes (and for all other hierarchies in our controlled vocabularies), the idea is precisely that people ought to use tokens from varying levels. In some cases, it is possible to be specific (e.g. Nāgarī), while in others one must be satisfied with a more generic category (e.g. northern class Brāhmī). But there remain many things to improve in these hierarchies, and guidance on their use is sorely lacking. The only guidance is the draft Memo on controlled vocabularies which I sketched up more than three years ago, but some of it is now dated, and it has received only very cursory input from PIs and none at all from metadata experts.
The class "Tamil with Grantha" is in my opinion misleading and should be eliminated, because our metadata sheets now allow recording multiple scripts and grantha characters in a Tamil inscription are, as far as I know, encoded specifically as such. I have pointed this out before (#197), but there hasn't been a response in almost a year.
I also find the need to encode these weird code numbers in our files; fortunately for me, all the inscriptions I work with are in the same script class and almost all of them in the same maturity category, so I just reuse the numbers from texts I encoded earlier. Others have a more difficult time, so if it is feasible to implement a canonical list (or whatever) that would be mapped to our OpenTheso definitions, then this may be helpful. But I don't think doing so is high-priority; I'm sure there are more urgent things to do. I'm also worried about adding yet another layer of complexity to the encoding, and about the difficulty of adding new script classes in the future, which would then have to be done both in OpenTheso and in that canonical list.
ISO script codes have been rejected early on in favour of a vocabulary of our own, primarily because the ISO system, as far as I know, is unaware that history exists, so the code for e.g. Tamil script means the Tamil script used right now for writing modern Tamil, while the code for Brahmi means basically Ashokan Brahmi. So even where an ISO code seems to be available for a script in our texts, the definition does not correspond to what we have.
One question that arises as I think about your suggestion of using a canonical list: if we have one, what then is the advantage of also having a vocabulary in OpenTheso? Is it more interoperable with potential future applications of our texts? Or more easily accessible? Or could we simply get rid of it and use just that list, which would then include definitions and hierarchical relations instead of references to the OpenTheso numbers? It was never really explained to me why we need OpenTheso in the first place.

Noted for the use of maturity, I will amend the schema accordingly.
The class "Tamil with Grantha" is not used anywhere, so I just deleted it.
Concerning OpenTheso, I would rather not use it at all and have instead a few XML files that describe the same data. OpenTheso has some nice features that make it appealing to librarians in particular (you can borrow a taxonomy from another library's catalog and adapt it instead of creating a new one from scratch, for instance), but this adds a lot of complexity, implementation-wise. In fact, we are not interfacing with it at all for now; language and script names are manually replicated in our schema, and from there into our database.
That said, Adeline seems familiar with it and is in charge of the metadata vocabulary, so it is really up to her. I can write a filter to export the data from OpenTheso into a few files that would then become the "real" authority and that would be tracked by git in project-documentation. In any case, the whole thesaurus will need to be imported into our own database at some point, to enable filtering by text type, material, etc.

Opentheso is a software created to manage the concepts and terms that we use to describe a specific domain (in our case, epigraphy). It isn’t used only by librarians, but also by the research community (mainly the French community of course, but very active) in the fields of antiquity and medieval studies to have specific and homogeneous vocubularies to publish digital editions and databases of scientific data.
Using a thesaurus, instead of a simple list of terms, can be appropriate to create standardized reference sources (with URIs, interoperability, etc.) that can be reused by a wider community of researchers, not only to work on our corpora.
In order to create such thing, it would be interesting to gather a large work group to discuss these questions. As Opentheso is a collaborative software, it would have been interesting to use it in this way. So far, after several presentations on thesauri and Opentheso, I haven’t been able to provoke some interests.
If you think that there is no point to use it, I don’t mind to stop giving up this solution. But if we choose to continue as it is, I suggest for the xml edition to change the use of @rendition, according to Michaël’s proposal (e.g. class:sundanese), and maybe using a script to replace the explicit terms by the URIs.
The actual thesaurus can be exported in SKOS or CSV.

Thanks, @alevivier , for the added information. Deciding whether we stick with OpenTheso should not be up to me but mainly to you in discussion with the PIs and probably Michaël. My amateurish impression is that if we conceive of the life cycle of a vocabulary in the following stages, then the usefulness of OpenTheso varies from stage to stage:

Brainstorming, initial planning: OK, but less convenient for collaboration than freeform tools such as Google sheets and docs
Finalisation, rigorous design: probably useful
Use, allowing for ongoing expansion and revision: potentially useful, but at the cost of greatly increased complexity of processing and integration
Dissemination and adaptation to other projects: useful

I am in favor of sticking with OpenTheso. But of course we do need finally to produce some documentation on our vocabularies and on how to use OpenTheso. I don't feel that replacing number codes for concepts by actual words is really necessary, but I am not against it either.
It occurs to me that the batch import of Campā inscription may again be to blame for a percentage of the offending files: they have never been updated to reflect changes in encoding rules after being imported according to an early DHARMA template. Besides that, since we never really finished the discussions about controlled vocabularies, I think we may have forgotten to disseminate to all encoders what decisions we made on encoding things like script class and maturity.

Can we try to come to a conclusion on this matter?
@michaelnmmeyer:

I am not sure you grasped, through Dan's and my responses, that the low rate of actual suage of @rendition in our xml files is due to the fact (if I err not) that the EGD has never made it to its v2, while consequently the template that people are using for encoding new items has never moved to it v4, so that the template simpkly does not contain @rendition in the relevant place and it has not been possible to instruct encoders to revise their files that have been encoded as per v1.
I think is it urgent to make available v4 of the template and this probably requires lifting the break on Dan finalizing v2 of EGD.
I notice that the system does not record any error at https://dharmalekha.info/errors for files that lack @rendition, e.g. a file I have just committed that still only shows
. Wouldn't it be helpful if such lacunae were flagged as (non-fatal) errors?

Depending on Michaël's responses to the above, a team will need to be formed to push our documentation on script maturity and classification to released state, and Dan will (with my help) need to reserve time for finalizing the release of EGD v2.

@arlogriffiths
For now, nothing is done to verify that @xml:lang and @rendition are used properly in
. I will add checks for this, but this will only work on the website, because the logic is impossible to express in XML schemas.

I think most people use the google doc version of the EGD for daily reference, though I may be wrong. At any rate, I too have been wondering recently whether I should return to working on the next EGD release. I think starting soon would make it possible to finish it by summer. I'm planning to take a longer leave from mid/late July and would not mind being done by then. (I'll also be on leave from tomorrow till the end of next week, so if this discussion continues, I will not be a very active participant.)
@michaelnmmeyer : I think Arlo's suggestion was for the schema simply to check whether @rendition is present on the edition div, not whether it has an approved (combination of) value(s).


DMARC bounces?

Hey @derekrushforth! Do we want DMARC bounces in the field manual? If so, I've got a ton I can add.

Hey @fm!
Definitely! Feel free to add them here or submit a pull request.

Awesome! I'll open an PR


DL3-113 Redirect on LTI 1.3 controller that pulls token

Description

This code allows for browsers that block third party cookies to use a cookieless route to complete the OIDC flow. Instead of cookies, this route verifies the state through the stages of the OIDC launch by using postMessages passed between the middleware and the LMS.
More specifically, this code bypasses some cookie-based nonce checks in various controllers (thanks Leanne!) and also adds a state and nonce check to the frontend before allowing the user to either access content in their LMS or add content through deeplinking (instructors).
Motivation and Context

Safari and iOS users are unable to use Lumen One because of the LTI1.3 flow that uses cookies. This PR implements the cookie-less LTI1.3 flow.
Fixes
DL3-134
How Has This Been Tested?

-Locally across browsers and on iOS using Canvas and local middleware tool
-Currently testing other LMS.
-If the state and nonce check does not pass, currently this is failing with no user feedback (just a blank screen presents that does not advance foward) DL3-118 will add in more elegant user feedback to errors.
Screenshots
Screenshots and screen recordings are not super helpful, but here is a launch on my iPhone using Safari. If you'd like to watch the process but with access to dev tools and some logs to verify the process, happy to host a meeting to show you! Just put time on my calendar or DM me.
https://user-images.githubusercontent.com/63607748/224076666-69933ac0-63f4-4e8c-b7e6-0840b5b2dce6.mov

Database changes

none
⚠ Deployment instructions ⚠

For the Front End React Component

cd ./src/main/frontend/
then run npm install to install styled components dependency

New ENV variables

N/A

Checklist

[x] I have reviewed the AC for this feature and my changes meet all requirements
[ ] (For UI changes) I have considered the accessibility of this feature (see https://www.a11yproject.com/checklist/ for a high-level checklist)
[x] I have performed a self-review of my own code
[ ] I have commented my code, particularly in hard-to-understand areas
[x] My changes generate no new warnings
[ ] I have added tests that prove my fix is effective or that my feature works
[ ] New and existing unit tests pass locally with my changes
[ ] Any dependent changes have been merged and published in downstream modules
[ ] My change requires a change to the documentation.
[ ] I have updated the documentation accordingly.
[x] I have requested a review from at least one other dev

@smith750 Do we want to confirm which rc branch this work should go in?

Good call, @LeftShotDev - this should go into rc/unuk-middleware


feat: Add traefik example YAML for local development with Docker Compose

The intent here is to offer an alternative example that allows users to rapidly demo this playground to external parties, eliminating the need for an ngrok API key. However, if this Pull Request doesn't align with the project's direction, feel free to close it. I'm perfectly okay with that decision.


Fixing error notification on the Leadership PDF Calendar

Fixing error notification on the Leadership PDF Calendar
This PR contains commits from https://github.com/cfpb/cfgov-refresh/pull/2250
Changes

Updated cfgov/v1/util/util.py to make the error message dynamic through string interpolation.
Updated cfgov/jinja2/v1/about-us/the-bureau/leadership-calendar/print/index.html to fix issue with logic and formatting.
Updated cfgov/cal/views.py to add validation for the CalendarPDFForm1.
Updated cfgov/jinja2/v1/_includes/macros/filters.html to fix issue rendering the notification.

Testing

Disable Javascript in the browser.
Visit the http://localhost:8000/about-us/the-bureau/leadership-calendar/ page.
Click the Download PDF button. You should see an error notification once redirected.
Enter in 123456 in the From : field. Click the Download PDF button. You should see an error notification once redirected.

Review

@richaagarwal
@chosak
@kurtrwall

Screenshots

Notes

Most of the code was contained in https://github.com/cfpb/cfgov-refresh/blob/master/cfgov/v1/forms.py. It probably makes sense to delete that code at this time, since much of it has been migrated to views.py.

I am removing all query-string parameters when redirecting. It might make sense to add back the valid parameters.

There are two forms on the leadership-calendar page sharing the same query-string parameters. We might want to consider an alternative approach.

Todos

Write Tests.

Checklist

[ ] Changes are limited to a single goal (no scope creep)
[ ] Code can be automatically merged (no conflicts)
[ ] Code follows the standards laid out in the front end playbook
[ ] Passes all existing automated tests
[ ] New functions include new tests
[ ] New functions are documented (with a description, list of inputs, and expected output)
[ ] Placeholder code is flagged
[ ] Visually tested in supported browsers and devices
[ ] Project documentation has been updated (including the "Unreleased" section of the CHANGELOG)

Would it be possible to have the error redirect go to the download options section (via a hash on the URL) instead of to the top of the page? Currently if there's an error it's not immediately clear what happened and you have to scroll down to see the message.

Thanks for fixes! 👍

👍

👍  after merge conflicts

@cfpb/cfgov-backends, I'm not sure how to resolve the testing error. Any help would be appreciated.

@sebworks I cleaned up those broken tests a bit and got them passing. I'm able to validate the error message behavior and also generate print layout calendars locally. I don't have PDFreactor locally so can't test regular PDF generation. Is this something that you can validate?

@sebworks I cleaned up those broken tests a bit and got them passing. I'm able to validate the error message behavior and also generate print layout calendars locally. I don't have PDFreactor locally so can't test regular PDF generation. Is this something that you can validate?

@chosak, I don't know of a way to test locally. Maybe we could use a virtual box?

@chosak, thanks for helping me on this PR.

@sebworks I refactored the forms and form elements into a separate module and added a whole bunch of tests around the expected behavior. I tested this locally using calendar data and filtering and download seem to work as expected. Would you mind taking another look?

💯 , this is really clean. Only the one comment.


Python Grpc client service method does not return when called from httpd on Windows

What version of gRPC and what language are you using?
Python gRPC client executed from wsgi module of Apache httpd
I tested grpcio packages from 1.1.0, to 1.4.4 inclusively
What operating system (Linux, Windows, …) and version?
Windows server 2012
What runtime / compiler are you using (e.g. python version or version of gcc)
Tested grpcio packages from 1.1.0, to 1.4.4 inclusively on python 2.7
What did you do?
When trying to connect to a Grpc Server from a simple Python Grpc client launched from httpd with mod_wsgi, the process freezes during initialization.  It really looks like a race condition because enabling the all logs seems to slow or lock the process long enough to make it work sometimes.  The same code works fine when launched alone.
What did you expect to see?
I would expect it to carry on.
What did you see instead?
It does not return from call.
Make sure you include information that can help us debug (full error message, exception listing, stack trace, logs).
NotWorking.txt
Working.txt
Anything else we should know about your project / environment?

I'm not familiar with Apache mod_wsgi. Does it fork the process by any chance? If so, interplay of gRPC and fork is a tricky domain and there are known issues.

The mod_wsgi module runs in httpd process on Windows.  The "daemon" mode is not available.   So in our case, whenever the call is made from the Grpc Client, the server is blocked.
http://modwsgi.readthedocs.io/en/develop/user-guides/processes-and-threading.html

Can you please give us a minimal test case that reproduces the problem?

I can't paste the company source code here so I rewrote it.  It is almost the simplest case possible based on the documentation sample.  I believe the fastidious part is to setup and configure Apache, mod_wsgi and make it call the client code.  When the server requests the grpc client (list_devices in this case), it never returns.  The server code doesn't matter because it is not invoked.
import grpc

import DeviceService_pb2

def get_config_stub(settings):
    channel = grpc.insecure_channel(settings.GRPC_CHANNEL)
    stub = DeviceService_pb2.ConfigurationStub(channel)
    return stub

def list_devices(settings, model):
    request_filter = DeviceService_pb2.Filter()
    request_filter.model = model
    stub = get_config_stub(settings)
    devices = stub.ListDevices(request_filter)
    return devices

syntax = "proto3";

package DeviceService;

message Filter
{
    string model = 1;
}

message Device
{
    string id = 1;
    string model = 2;
}

message Devices
{
    repeated Device items = 1;
}

service Configuration
{
    rpc ListDevices(Filter) returns (Devices);
}

Thanks. This is great. I'll try setting up a Windows machine to try and reproduce. Can you point me to an Apache httpd/mod_wgsi setup tutorial on Windows you suggest I follow to get to a close configuration to what you have?

In fact, I wasn't involved on the web server setup, but I'll try to get what I could.  I worked on the Grpc service which works pretty well when the client reaches it.

Is there a way for you to set GPRC_TRACE=all GPRC_VERBOSITY=debug environment variables for the process hosting gRPC to get some trace information?

Nevermind... I did some fancy grepping to extract the info :)

Our setup is using DJango with mod_wsgi on Apache httpd.  It is based on the django documentation:
https://docs.djangoproject.com/en/1.11/howto/deployment/wsgi/modwsgi
In case it helps, I detailed the version we're using
Python 2.7.13
Apache Httpd 2.4.27
mod_wsgi 4.5.17
django 1.11.2

When looking at the NotWorking log, it looks like the service is freezing when using the CompletionQueue, which is part of the Windows I/O Completion Ports API.  Both Grpc and Httpd implementation create a CompletionQueue from that API, using all threads available.  It is possible that the issue happens because the wsgi_mod is in the httpd process, so httpd and our Grpc client are both creating a CompletionQueue in that same process, and both are jammed.  If I am right, It would be possible to narrow down the problem to that case.
https://msdn.microsoft.com/en-us/library/windows/desktop/aa365198(v=vs.85).aspx
Threads and Concurrency
The most important property of an I/O completion port to consider carefully is the concurrency value. The concurrency value of a completion port is specified when it is created with CreateIoCompletionPort via the NumberOfConcurrentThreads parameter. This value limits the number of runnable threads associated with the completion port. When the total number of runnable threads associated with the completion port reaches the concurrency value, the system blocks the execution of any subsequent threads associated with that completion port until the number of runnable threads drops below the concurrency value.

So, the way I think it's supposed to work is that each Completion Port has its own threadpool, and the value we specify on the creation is the one that is attached specifically for that Completion Port. These threads are created by the kernel, for the purpose of this Completion Port, and will only be used by the kernel to, well, execute the operations you're asking. Then it's possible to max out these threads by filling the completion queue with operations, which is in fact kind of the ideal scenario. If your system is busy enough that all of the threads in the threadpool are doing things, without growing the queue itself, then it's the best optimized scenario.
In all cases, two scenarios can occur then:

each completion port has its own threadpool attached, in which case, one shouldn't block the other.
all completion ports share the same threadpool, which then can get a high contention if there's not enough threads, but then the worst case scenario is "things get slow", as the threadpool is struggling to keep up with the demand.

I think the most likely scenario however is that you see you are blocked into the Completion Port because there's simply nothing happening.
In fact, I would question the threading model you are using, because by definition, if you have two event loops that are going to block on polling, they can't work with each other, since they are going to prevent each other from working. The "race condition" at that point wouldn't be thread-related, but rather event-related, where you have a different set of event sent to two different completion queues, and sometimes it'll work, sometime they'll be stuck waiting forever. The smoking gun here would be the "slowness" to start, which would be caused by completion port 1 reaching a timeout state waiting for events, going outside, reaching into completion port 2, which then manages to trigger some events.

In light of what @nicolasnoble documented above, I'd like to point out that having two "event managers" in the same process (e.g. gRPC + gevent) is not yet supported by gRPC Python and this situation categorically falls under the same sort of issues. We are actively thinking about at least some of these scenarios, like gevent that I mentioned, but have not yet decided on a path forward.

Thank for your support.
@nicolasnoble I agree with you about our problematic threading model.  When I first made the the C++ gRPC Service, I didn't expect the gRPC client (the Python request handlers) would be in the Apache httpd process.  Even then, I didn't expect gRPC client request could interfere with Apache server.  If mod_wsgi could be run in daemon mode, this would not be an issue, but Windows implementation doesn't support it.  I think however that the limitation should be mentioned in the documentation.
At this point, I rewrote the request handlers to spawn a Python "gRPC client proxy" that communicates json request/response through standard streams, but this is an ugly solution.
Again, thanks for your help.


vscode: Show notification when upgrade is available

Will add a ss in the AM.

How can I disable this update check?

How can I disable this update check?

Is there a reason you need to? code-server will never actually upgrade itself. It's only a notification. Perhaps we should make it a weekly/monthly notification instead?

Basically 2 reasons:

It is repacked in another Docker container as a Home Assistant add-on. All those releases are tested before releasing it to our userbase, your update cycle doesn't match ours. Neither does the versioning. This will be confusing to them.

As for my personal system; I'm not interested in having it phone home every couple of hours for an update check.

It is repacked in another Docker container as a Home Assistant add-on. All those releases are tested before releasing it to our userbase, your update cycle doesn't match ours. Neither does the versioning. This will be confusing to them.
I simply don't want a notification. I rather control the things I'm notified of. It is distracting while working actually.

Fair enough, I've opened https://github.com/cdr/code-server/issues/2361 for the next release.

As for my personal system; I'm not interested in having it phone home every couple of hours for an update check.

It doesn't hit any server hosted by us. It just checks against the latest release on GitHub.

Thank you!

How to disable the update notification? I see the PR don't have any options/command-args to disable that!


✨ Raise a more clear error when a type is not valid

Currently using Dict[...], List[...] or Union[...] will break with an error stating that issubclass() cannot be used for non types. This is something users won't understand and I think SQLModel should handle this in a better way.
PR will fix the issue and contains a test to validate the code was broken before and is now fixed. The normal ValueError(f"The field {field.name} has no matching SQLAlchemy type") will then be raised.

Thanks! I updated the tests to use models, as that would be a realistic use case, then I fixed a small bug in the implementation detected by that.
This will be available in the next release. 🤓


v4 ARM64 image broken

Quits with:
fatal: chdir(/usr/libexec/postfix): No such file or directory

That was fast. It's not even out of the oven yet. I'll check it out.

I cannot reproduce this issue.
docker run --platform linux/arm64 --pull always -e ALLOW_EMPTY_SENDER_DOMAINS=yes --rm -it boky/postfix:v4.0.0
v4.0.0: Pulling from boky/postfix
Digest: sha256:2a6d595d17c60f62571b7d25bd3b1e82f609eaa0eb6b625a80c53520487122eb
Status: Downloaded newer image for boky/postfix:v4.0.0
★★★★★ POSTFIX STARTING UP ★★★★★
‣ INFO  Not setting any timezone for the container
‣ INFO  Using plain log format for rsyslog.
‣ NOTE  Emails in the logs will not be anonymized. Set ANONYMIZE_EMAILS to enable this feature.
‣ DEBUG Reowning root: /var/spool/postfix/
‣ DEBUG Reowning root: /var/spool/postfix/pid/
‣ DEBUG Reowning postfix:postdrop /var/spool/postfix/private/
‣ DEBUG Reowing postfix:postdrop /var/spool/postfix/public/
‣ INFO  Preparing files for Postfix chroot:
        '/var/spool/postfix/usr/lib/zoneinfo/localtime' -> '/etc/localtime'
        '/etc/localtime' -> '/var/spool/postfix/etc/localtime'
        '/etc/nsswitch.conf' -> '/var/spool/postfix/etc/nsswitch.conf'
        '/etc/resolv.conf' -> '/var/spool/postfix/etc/resolv.conf'
        '/etc/services' -> '/var/spool/postfix/etc/services'
        '/etc/host.conf' -> '/var/spool/postfix/etc/host.conf'
        '/etc/hosts' -> '/var/spool/postfix/etc/hosts'
        '/etc/passwd' -> '/var/spool/postfix/etc/passwd'
‣ WARN  Detected old hash: and btree: references in the config file, which are not supported anymore. Upgrading to lmdb:
‣ DEBUG Creating new postalias for lmdb:/etc/aliases.
‣ DEBUG Creating new postalias for lmdb:/etc/aliases.
‣ INFO  Using unlimited message size.
‣ INFO  Setting smtp_tls_security_level: may
‣ NOTE  Will try to deliver emails directly to the final server. Make sure your DNS is setup properly!
/scripts/common-run.sh: line 341: pluginviewer: command not found
‣ INFO  Using default private network list for trusted networks.
‣ INFO  Debugging is disabled.
‣ DEBUG DKIM_AUTOGENERATE not set -- you will need to provide your own keys.
‣ INFO  No DKIM keys found, will not use DKIM.
‣ INFO  Applying custom postfix setting: message_size_limit=0
‣ INFO  Applying custom postfix setting: myhostname=8a3a412be30b
‣ INFO  Applying custom postfix setting: mynetworks=<IP_ADDRESS>/8,<IP_ADDRESS>/8,<IP_ADDRESS>/12,<IP_ADDRESS>/16
‣ INFO  Applying custom postfix setting: smtp_tls_security_level=may

Works on my machine using the following command:
docker run --platform linux/arm64 --pull always -e ALLOW_EMPTY_SENDER_DOMAINS=yes --rm -it boky/postfix:latest
latest: Pulling from boky/postfix
Digest: sha256:2a6d595d17c60f62571b7d25bd3b1e82f609eaa0eb6b625a80c53520487122eb
Status: Downloaded newer image for boky/postfix:latest
....

How are you running it, what's your uname -a?

I'll try running this directly on another ARM machine. I'm running this natively on ARM mac and it works.

That shouldn't be an issue. It works for me, though, if I try to run it directly from the command line with your variables:
docker run --platform linux/arm64 --pull always -e HOSTNAME=domain.tld -e TZ=Europe/Amsterdam -e 'MYNETWORKS=<IP_ADDRESS>/0,[::]/0' -e ALLOW_EMPTY_SENDER_DOMAINS=yes -e RELAYHOST=xxx:587 -e RELAYHOST_USERNAME=xxxx -e RELAYHOST_PASSWORD=xxxx --rm -it boky/postfix:latest
latest: Pulling from boky/postfix
Digest: sha256:2a6d595d17c60f62571b7d25bd3b1e82f609eaa0eb6b625a80c53520487122eb
Status: Image is up to date for boky/postfix:latest
★★★★★ POSTFIX STARTING UP ★★★★★
‣ NOTE  Setting container timezone to: Europe/Amsterdam
‣ INFO  Using plain log format for rsyslog.
‣ NOTE  Emails in the logs will not be anonymized. Set ANONYMIZE_EMAILS to enable this feature.
‣ DEBUG Reowning root: /var/spool/postfix/
‣ DEBUG Reowning root: /var/spool/postfix/pid/
‣ DEBUG Reowning postfix:postdrop /var/spool/postfix/private/
‣ DEBUG Reowing postfix:postdrop /var/spool/postfix/public/
‣ INFO  Preparing files for Postfix chroot:
        '/var/spool/postfix/usr/lib/zoneinfo/localtime' -> '/etc/localtime'
        '/etc/localtime' -> '/var/spool/postfix/etc/localtime'
        '/etc/nsswitch.conf' -> '/var/spool/postfix/etc/nsswitch.conf'
        '/etc/resolv.conf' -> '/var/spool/postfix/etc/resolv.conf'
        '/etc/services' -> '/var/spool/postfix/etc/services'
        '/etc/host.conf' -> '/var/spool/postfix/etc/host.conf'
        '/etc/hosts' -> '/var/spool/postfix/etc/hosts'
        '/etc/passwd' -> '/var/spool/postfix/etc/passwd'
‣ WARN  Detected old hash: and btree: references in the config file, which are not supported anymore. Upgrading to lmdb:
‣ DEBUG Creating new postalias for lmdb:/etc/aliases.
‣ DEBUG Creating new postalias for lmdb:/etc/aliases.
‣ INFO  Using unlimited message size.
‣ INFO  Setting smtp_tls_security_level: may
‣ NOTE  Forwarding all emails to xxx:587 using username xxxx and password (redacted).
/scripts/common-run.sh: line 341: pluginviewer: command not found
‣ DEPRECATED! MYNETWORKS variable is deprecated. Please use POSTFIX_mynetworks instead.
‣ NOTE  Using custom allowed networks: <IP_ADDRESS>/0,[::]/0
‣ INFO  Debugging is disabled.
‣ DEBUG DKIM_AUTOGENERATE not set -- you will need to provide your own keys.
‣ INFO  No DKIM keys found, will not use DKIM.
‣ INFO  Applying custom postfix setting: message_size_limit=0
‣ INFO  Applying custom postfix setting: myhostname=domain.tld
‣ INFO  Applying custom postfix setting: mynetworks=<IP_ADDRESS>/0,[::]/0
‣ INFO  Applying custom postfix setting: smtp_tls_security_level=may
‣ NOTE  Starting: rsyslog, postfix
2023-10-27 18:39:24,656 INFO Set uid to user 0 succeeded
2023-10-27 18:39:24,657 INFO supervisord started with pid 7
2023-10-27 18:39:25,661 INFO spawned: 'cron' with pid 475
2023-10-27 18:39:25,663 INFO spawned: 'opendkim' with pid 476
2023-10-27 18:39:25,664 INFO spawned: 'postfix' with pid 477
2023-10-27 18:39:25,665 INFO spawned: 'rsyslog' with pid 478
rsyslogd: could not load module 'omstdout.so', errors: trying to load module /usr/lib/aarch64-linux-gnu/rsyslog/omstdout.so: /usr/lib/aarch64-linux-gnu/rsyslog/omstdout.so: cannot open shared object file: No such file or directory [v8.2302.0 try https://www.rsyslog.com/e/2066 ]
2023-10-27 18:39:25,668 INFO success: cron entered RUNNING state, process has stayed up for > than 0 seconds (startsecs)
2023-10-27 18:39:25,668 INFO success: postfix entered RUNNING state, process has stayed up for > than 0 seconds (startsecs)
2023-10-27T18:39:25.669193+02:00 ERR     rsyslogd: could not load module 'omstdout.so', errors: trying to load module /usr/lib/aarch64-linux-gnu/rsyslog/omstdout.so: /usr/lib/aarch64-linux-gnu/rsyslog/omstdout.so: cannot open shared object file: No such file or directory [v8.2302.0 try https://www.rsyslog.com/e/2066 ]
2023-10-27T18:39:25.669205+02:00 INFO    rsyslogd: [origin software="rsyslogd" swVersion="8.2302.0" x-pid="478" x-info="https://www.rsyslog.com"] start
2023-10-27T18:39:26.733362+02:00 INFO    postfix/postfix-script[566]: starting the Postfix mail system
2023-10-27T18:39:26.737175+02:00 INFO    postfix/master[567]: daemon started -- version 3.7.6, configuration /etc/postfix
2023-10-27 18:39:27,739 INFO success: rsyslog entered RUNNING state, process has stayed up for > than 2 seconds (startsecs)
2023-10-27 18:39:30,747 INFO success: opendkim entered RUNNING state, process has stayed up for > than 5 seconds (startsecs)

Same problem with amd64 build and latest tag image: boky/postfix:latest. Rollback to image: boky/postfix:v3.6.1 makes the stack back to healthy state. I got the same trace than those above with latest version :
2023-10-27 23:44:05,997 INFO Set uid to user 0 succeeded
2023-10-27 23:44:05,999 INFO supervisord started with pid 7
2023-10-27 23:44:07,002 INFO spawned: 'cron' with pid 126
2023-10-27 23:44:07,003 INFO spawned: 'opendkim' with pid 127
2023-10-27 23:44:07,005 INFO spawned: 'postfix' with pid 128
2023-10-27 23:44:07,007 INFO spawned: 'rsyslog' with pid 129
2023-10-27T23:44:07.017594+02:00 INFO    rsyslogd: [origin software="rsyslogd" swVersion="8.2302.0" x-pid="129" x-info="https://www.rsyslog.com"] start
2023-10-27T23:44:07.017876+02:00 CRIT    postfix[128]: fatal: chdir(/usr/libexec/postfix): No such file or directory
2023-10-27 23:44:07,018 INFO success: cron entered RUNNING state, process has stayed up for > than 0 seconds (startsecs)
2023-10-27 23:44:07,018 INFO success: postfix entered RUNNING state, process has stayed up for > than 0 seconds (startsecs)
2023-10-27 23:44:08,018 WARN exited: postfix (exit status 1; not expected)
2023-10-27 23:44:09,020 INFO success: rsyslog entered RUNNING state, process has stayed up for > than 2 seconds (startsecs)
2023-10-27 23:44:12,024 INFO success: opendkim entered RUNNING state, process has stayed up for > than 5 seconds (startsecs)

Could this be because of the use of debian-slim image rather than debian mentioned in #149 ?

#149 was about changing debian to debian-slim, so no, it does not have anything to do with it.
But I think it does with switch from alpine to debian.
/usr/libexec/postfix does sound like something Alpine would have.
I don't think that it's an issue with the image, but with existing (Alpine-based) configuration.
I see above that @kroese has /etc/postfix bound to a folder on disk (although this is not recommended), I'm guessing you have the same.
Can you @undefinedid and @kroese check or maybe even share your configs, so we find in which setting folder appears?

Can you @undefinedid and @kroese check or maybe even share your configs, so we find in which setting folder appears?

I can confirm that switching to image: boky/postfix:latest-alpine (build amd64) solved the problem! I don't have arm64 hardware easily accessible actually to test that build.
As @kroese said, /etc/postfix/main.cf  include the path for the daemon: /usr/libexec/postfix.
It seems that this is the default Postfix config file, created at first launch of the container, in which custom settings are written at the very end of the file via the container environment variables.
Do you need a specific config or Compose file to help fix this issue?

Can you check the latest edge, plese? It includes a fix for this issue, but I haven't been able to test it properly just yet.
If it does work, I'll release v4.0.1.

This should work now with v4.0.1. Closing the issue. If it reappears, please reopen.

I think there may have been a regression here. I'm seeing this happening again starting with v4.0.4. But if I downgrade to 4.0.3 the problem disappears.

That sounds really odd. These were literary the only changes between v4.0.3 and v4.0.4:
diff --git a/Dockerfile b/Dockerfile
index bf92c5a..ae4c04f 100644
--- a/Dockerfile
+++ b/Dockerfile
@@ -1,4 +1,4 @@
-# syntax=docker/dockerfile:1.2
+# syntax=docker/dockerfile:1.6

 ARG BASE_IMAGE=debian:bookworm-slim
 # ARG BASE_IMAGE=ubuntu:jammy
@@ -61,7 +61,7 @@ VOLUME     [ "/var/spool/postfix", "/etc/postfix", "/etc/opendkim/keys" ]
 USER       root
 WORKDIR    /tmp

-HEALTHCHECK --interval=30s --timeout=5s --start-period=10s --retries=3 CMD printf "EHLO healthcheck\nquit\n" | { while read l ; do sleep 1; echo $l; done } | nc <IP_ADDRESS> 587 -w 2 | grep -qE "^220.*ESMTP Postfix"
+HEALTHCHECK --interval=30s --timeout=5s --start-period=10s --start-interval=2s --retries=3 CMD printf "EHLO healthcheck\nquit\n" | { while read l ; do sleep 1; echo $l; done } | nc -w 2 <IP_ADDRESS> 587 | grep -qE "^220.*ESMTP Postfix"

 EXPOSE     587
 CMD        [ "/bin/sh", "-c", "/scripts/run.sh" ]
diff --git a/build.sh b/build.sh
index 280104e..17814fc 100755
--- a/build.sh
+++ b/build.sh
@@ -41,5 +41,6 @@ fi
 #    arg_list="$arg_list --platform linux/amd64,linux/arm64,linux/arm/v7"
 #fi

-docker buildx build ${arg_list} $* .
+set -x
+exec docker buildx build ${arg_list} $* .

diff --git a/helm/mail/values.yaml b/helm/mail/values.yaml
index 2efa6a9..32b9ba9 100644
--- a/helm/mail/values.yaml
+++ b/helm/mail/values.yaml
@@ -211,7 +211,7 @@ readinessProbe:
       - -c
       - >-
         [ ! -f /tmp/container_is_terminating ] &&
-        printf "EHLO healthcheck\n" | nc <IP_ADDRESS> 587 | grep -qE "^220.*ESMTP Postfix"
+        printf "EHLO healthcheck\nquit\n" | { while read l ; do sleep 1; echo $l; done } | nc -w 2 <IP_ADDRESS> 587 | grep -qE "^220.*ESMTP Postfix"
 livenessProbe:
   initialDelaySeconds: 5
   periodSeconds: 5

@joekrill How exactly are you starting the image?

Sorry, I should clarify - the error I'm seeing is slightly different, so this may be a different issue. I'm seeing the same message as the one in this comment above - which is why I initially thought it was the same issue:
CRON[2779]: pam_env(cron:session): Unable to open env file: /etc/default/locale: No such file or directory

But I'm not seeing the fatal: chdir(/usr/libexec/postfix): No such file or directory message. But it does seem to be preventing the service from becoming healthy in my cluster. Let me know if I should create a separate issue for this.
The full output:
★★★★★ POSTFIX STARTING UP (debian) ★★★★★
‣ INFO  Not setting any timezone for the container
‣ INFO  Using plain log format for rsyslog.
‣ NOTE  Emails in the logs will not be anonymized. Set ANONYMIZE_EMAILS to enable this feature.
‣ DEBUG Reowning root: /var/spool/postfix/
‣ DEBUG Reowning root: /var/spool/postfix/pid/
‣ DEBUG Reowning postfix:postdrop /var/spool/postfix/private/
‣ DEBUG Reowning postfix:postdrop /var/spool/postfix/public/
‣ INFO  Preparing files for Postfix chroot:
ln: failed to create symbolic link '/var/spool/postfix/usr/lib/zoneinfo/': No such file or directory
        '/etc/localtime' -> '/var/spool/postfix/etc'
        '/etc/nsswitch.conf' -> '/var/spool/postfix/etc'
        '/etc/resolv.conf' -> '/var/spool/postfix/etc'
        '/etc/services' -> '/var/spool/postfix/etc'
        '/etc/host.conf' -> '/var/spool/postfix/etc'
        '/etc/hosts' -> '/var/spool/postfix/etc'
        '/etc/passwd' -> '/var/spool/postfix/etc'
‣ NOTE  Switching default_database_type to lmdb to ensure cross-distro compatibility.
‣ WARN  Detected old hash: and btree: references in the config file, which are not supported anymore. Upgrading to lmdb:
‣ DEBUG Creating new postalias for lmdb:/etc/aliases.
‣ DEBUG Creating new postalias for lmdb:/etc/aliases.
‣ INFO  Using unlimited message size.
‣ INFO  Setting smtp_tls_security_level: may
‣ NOTE  Forwarding all emails to email-smtp.us-east-1.amazonaws.com:587 using username (removed) and password (redacted).
‣ INFO  Using default private network list for trusted networks.
‣ INFO  Debugging is disabled.
‣ INFO  Setting up allowed SENDER domains: (removed)
‣ DEBUG DKIM_AUTOGENERATE not set -- you will need to provide your own keys.
‣ INFO  No DKIM keys found, will not use DKIM.
‣ INFO  Applying custom postfix setting: message_size_limit=0
‣ INFO  Applying custom postfix setting: myhostname=smtp-0
‣ INFO  Applying custom postfix setting: mynetworks=<IP_ADDRESS>/8,<IP_ADDRESS>/8,<IP_ADDRESS>/12,<IP_ADDRESS>/16
‣ INFO  Applying custom postfix setting: smtp_tls_security_level=may
‣ NOTE  Starting: rsyslog, crond, postfix
2024-01-08 22:03:02,092 INFO Set uid to user 0 succeeded
2024-01-08 22:03:02,099 INFO supervisord started with pid 7
2024-01-08 22:03:03,104 INFO spawned: 'cron' with pid 354
2024-01-08 22:03:03,108 INFO spawned: 'opendkim' with pid 355
2024-01-08 22:03:03,112 INFO spawned: 'postfix' with pid 356
2024-01-08 22:03:03,117 INFO spawned: 'rsyslog' with pid 357
2024-01-08T22:03:03.143135+00:00 INFO    rsyslogd: [origin software="rsyslogd" swVersion="8.2302.0" x-pid="357" x-info="https://www.rsyslog.com"] start
2024-01-08 22:03:03,144 INFO success: cron entered RUNNING state, process has stayed up for > than 0 seconds (startsecs)
2024-01-08 22:03:03,144 INFO success: postfix entered RUNNING state, process has stayed up for > than 0 seconds (startsecs)
2024-01-08T22:03:03.596738+00:00 INFO    postfix/postfix-script[429]: starting the Postfix mail system
2024-01-08T22:03:03.608740+00:00 INFO    postfix/master[430]: daemon started -- version 3.7.9, configuration /etc/postfix
2024-01-08 22:03:05,611 INFO success: rsyslog entered RUNNING state, process has stayed up for > than 2 seconds (startsecs)
2024-01-08 22:03:08,615 INFO success: opendkim entered RUNNING state, process has stayed up for > than 5 seconds (startsecs)
2024-01-08T22:17:01.130027+00:00 ERR     CRON[2779]: pam_env(cron:session): Unable to open env file: /etc/default/locale: No such file or directory
2024-01-08T22:17:01.131518+00:00 INFO    CRON[2779]: pam_unix(cron:session): session opened for user root(uid=0) by (uid=0)
2024-01-08T22:17:01.134201+00:00 INFO    CRON[2780]: (root) CMD (cd / && run-parts --report /etc/cron.hourly)
2024-01-08T22:17:01.138975+00:00 INFO    CRON[2779]: pam_unix(cron:session): session closed for user root
2024-01-08T22:23:03.242681+00:00 INFO    rsyslogd: -- MARK --

I'm using the Helm chart to start this in a Kubernetes cluster. These are the values I'm using:
fullnameOverride: smtp
metrics:
  enabled: true
  serviceMonitor:
    enabled: false
    labels:
      release: prometheus-stack
existingSecret: smtp-settings
config:
  general:
    ALLOWED_SENDER_DOMAINS: mydomain.com
service:
  port: 25

And the smtp-settings secrets contains values for RELAYHOST (specifically: email-smtp.us-east-1.amazonaws.com:587), RELAYHOST_USERNAME, and RELAYHOST_PASSWORD
If I downgrade to 4.0.3 the problem goes away. So perhaps it instead has something to do with the loop in the healthcheck causing the startup to hang?


feat(cli): use Config for Etherscan and RPCs

Motivation
Closes #4484
Closes #4474
Closes #4057
Closes #3824

Solution
Refactor EthereumOpts into EtherscanOpts and RpcOpts, which implement figment traits, for usage in Cast and Forge subcommands since rpc_url and the rest were always separate and had inconsistent flags/envs etc...

@DaniPopes don't see it mentioned above, but checking the code this might also resolve #4401 right?

Sorry one other thing, as i saw there's now a breaking change with remappings :sweat_smile:
I think there's one other inconsistency as current remappings in foundry.toml accepts an array where --remappings only supports a single string. Not sure if there is an issue for that.

I don't think so, the only thing I modified here is the short flag from 'r' to 'R'; to pass multiple remappings you have to use the flag multiple times like so: forge build -R a=b -R c=d, and I think it was always this way


How to display saved pictures？

@mzl1988 this project is discontinued.
Anyway: success callback gives you the file path and how to display it in your application is up to you (and the specific platform).


How to repeat element after one element?

Below is example as I want  TEXT  element to repeat again after the FEED element
<print>
       <text width="2" height="2"/>
        <text reverse="false" ul="false" em="true" color="color_1"/>
        <text>DELIVERY TICKET
</text>
        <feed unit="12"/>
        <text>
</text>
        <text align="left"/>
</print>

I don't think that is currently possible with this module.

Hi @wankdanker any ways to achieve this functionality. Other then that this LIB is almost suitable for my need just only this thing blocks to go further with this.


resize-sensor wider than the view port

Hi, I've encountered a situation where the .resize-sensor element is wider than the  view port thus enforcing a horizontal scroll-bar.
My fix was to just add the rule:
.resize-sensor{
  max-width:  100%;
}

Maybe this should be added to the source code.

@marcj done #238


JS: Add array callbacks js/use-of-returnless-function query

Using a callback function that does not return a value in e.g. Array.prototype.filter is generally an error.
Example:
const changePackages = packages.filter(f => {
 danger.git.modified_files.includes(f)
})
if (changePackages.length) {...}

It started out as a separate query, but after copy-pasting enough from js/use-of-returnless-function I integrated it into that query.
Just looking for methods named filter, map etc. is too imprecise and gives rise to way too many false positives. The query therefore uses type tracking to see if the receiver is an array.
It does not find many new errors, but the ones it does find are good: https://lgtm.com/query/1772403078964278178/
(The benchmarks for the above were found using the Chris API on a version of the query that did not include type tracking)
While the query on its own is slower now, it seems that performance is still good when combined with another dataflow query.
Performance evaluation is on the way.
Lots of changed lines from moving most of ExprHasNoEffect.ql into a .qll file.

@asger-semmle, anything else you'd like to see addressed?

@erik-krogh can you post a link to the evaluation? (based on an earlier commit is fine)
Otherwise LGTM

@erik-krogh can you post a link to the evaluation? (based on an earlier commit is fine)

Here is an old one.
https://git.semmle.com/erik/dist-compare-reports/tree/domitian.ti.semmle.com_1570435565541

Hm, yeah that used to happen to me a lot. Setting the branch name is crucial.
Anyways, I'm fine with landing it if you remember the evaluation as being good. @max-schaefer?

Yeah, should be fine.


feat: remember requested security classes, support inactive provisioning entries

This PR adds two additional (optional) fields to the PlannedProvisioningEntry (and by extension IncludedProvisioningEntry) interfaces.
The status field can assume the following values:
ProvisioningEntryStatus {
	Active,
	Inactive,
}

where an absent status will be assumed to be Active for backwards compatibility.
This can be used to disable provisioning entries. Inactive entries will be kept, but won't cause an inclusion when the corresponding node announces itself.
An entry can be set to inactive manually, or automatically during exclusion using the new possible value for the unprovision argument of beginExclusion:
await driver.controller.beginExclusion("inactive");
// this will exclude a node and set the corresponding provisioning entry to inactive afterwards

fixes: https://github.com/zwave-js/node-zwave-js/issues/3682

The optional requestedSecurityClasses (readonly) field keeps track of the security classes that have been requested by the device via QR code. This can be used to disable the checkboxes for security classes that haven't been requested, in order to avoid confusion caused by granting non-requested classes.
This is also added to the QRProvisioningInformation interface, so applications which copy the properties rather than pass a QRProvisioningInformation as a PlannedProvisioningEntry should take care of this new property.
fixes: https://github.com/zwave-js/node-zwave-js/issues/4250

cc @kpine @robertsLando


Must rename "australia" to "australia_nsw"

There might be a new Australia source soon.
Make sure there are comments here on how to migrate existing data, and whoever merges to master should put the changes on the server and migrate straight away.

Make sure there are comments here on how to migrate existing data, and whoever merges to master should put the changes on the server and migrate straight away.

We didn't do this - thanks to rename to Kingfisher there was no existing data on server, so that worked out well.


Implement more OData filter options

OData filter is currently not completely implemented.
See methods ODataGetBase._applyFilter and ODataGetBase._convertAstToLoopbackFilter for details.

Hi, @htammen , what kind of $filter are supported so far? When I try to use $filter=substringof, it says: Error: invalid $filter parameter

sorry, this is my fault, I forgot to provide the property to filter for


The daily jenkins OS installer jobs are currently failing

this jobs run every 24 hrs and run the tests in https://github.com/RackHD/RackHD/blob/master/test/tests/api/v1_1/os_install_tests.py

There was some modification to the payloads of certain OSes due to the introduction of tasks schema(required properties etc..) and other factors. these changes were updated in rackHD documentation at https://github.com/RackHD/RackHD/tree/master/example/samples

@keedya can you put in a link to the internal jenkins that hosts running these tests? Right now I can't personally reproduce them locally, so I couldn't even begin to vet them myself - but I recognize we want to keep them operational - just need a way to make sure they're good.
FYI @yyscamper I suspect there's some fallout from the updates you were making related to task schema that didn't get propagated to integration tests

here is the internal link to jenkins OS install jobs:
http://rackhdci.lss.emc.com/job/Continuous-Test/job/Continuous-OS-Install/

@keedya @heckj : I'll look at it.

PT story has been created to track this issue at https://www.pivotaltracker.com/story/show/130614329

I took a look at test result and console log provided by Andre (http://rackhdci.lss.emc.com/job/Continuous-Test/job/Continuous-OS-Install/).  All the OS installation test (ESXi, Windows 2K2, CentOS, OpenSUES, ...) were failed from 9/4 till now with same failure as below -- can't connect to vagrant VM. So we should get this issue fixed at first. @jlongever @panpan0000 @changev

23:02:05 --2016-09-04 23:02:05--  http://localhost:9090/api/1.1/nodes
23:02:05 Resolving localhost (localhost)... ::1, <IP_ADDRESS>
23:02:05 Connecting to localhost (localhost)|::1|:9090... failed: Connection refused.
23:02:05 Connecting to localhost (localhost)|<IP_ADDRESS>|:9090... connected.
23:02:05 HTTP request sent, awaiting response... Read error (Connection timed out) in headers.
23:02:20 Giving up.

FYI @leoyjchang , the vagrant issue should be resolved with this PR: https://github.com/RackHD/RackHD/pull/415

Continuous OS install job in Jenkins is now passing  and tests are stable now:
http://rackhdci.lss.emc.com/job/Continuous-Test/job/Continuous-OS-Install/


Small documentation improvements

I picked up #61 and #133 as small additions to the documentation. To make the links in the speccy lint command work I needed to adjust the base URL of the docs. I'm not sure if that's correct.
Also I don't have experience with Ruby and jekyll but wanted to try to help with docs there. I seems to work when serving the docs locally so I hope it's done correctly.

Coverage remained the same at 87.863% when pulling 1a865b2c1f44ec1f86c638e3869e3bbe692956fe on AnianZ:master into a2107e05c7a9cb9165d1da4bfc1cebff6b2d909e on wework:master.

Coverage remained the same at 87.863% when pulling 1a865b2c1f44ec1f86c638e3869e3bbe692956fe on AnianZ:master into a2107e05c7a9cb9165d1da4bfc1cebff6b2d909e on wework:master.


AttributeError: module 'types' has no attribute 'UnionType'. Did you mean: 'FunctionType'?

First Check

[X] I added a very descriptive title to this issue.
[X] I used the GitHub search to find a similar issue and didn't find it.
[X] I searched the FastAPI documentation, with the integrated search.
[X] I already searched in Google "How to X in FastAPI" and didn't find any information.
[X] I already read and followed all the tutorial in the docs and didn't find an answer.
[X] I already checked if it is not related to FastAPI but to Pydantic.
[X] I already checked if it is not related to FastAPI but to Swagger UI.
[X] I already checked if it is not related to FastAPI but to ReDoc.

Commit to Help

[x] I commit to help with one of those options 👆

Example Code
This is the app where I have the error 'https://github.com/lolio-dev/Shoppingify-server/tree/main/app' but I also have the error with the most basic app:

from fastapi import FastAPI

app = FastAPI()

@app.get('/')
def root():
    return {'hello': 'word'}

### Description

With a very simple and basic application I have the error `AttributeError: module 'types' has no attribute 'UnionType'. Did you mean: 'FunctionType'?`. The error happens when I import fastapi
![ksnip_20220625-201701](https://user-images.githubusercontent.com/58036293/175785782-70e59ea9-ba3e-49d5-a88d-576ab7c0e997.png)
I have only install `pip install fastapi[all]`

### Operating System

Linux

### Operating System Details

_No response_

### FastAPI Version

0.78.0

### Python Version

3.10

### Additional Context

_No response_

I am not sure if it would help but I would recommend to update your Python installation to a stable version. You are currently on beta 4, so might be because of that ?

Oh it was the solution, it was a risky installation of python3.10 and I did not seen that it was in beta version. Now with the last version all is ok. Thanks you so much !!