Rate limited on the OpenAI embeddings API, sleeping before retrying...

Please provide us with the following information:

This issue is for a: (mark with an x)
- [x ] bug report -> please search issues before submitting
- [ ] feature request
- [ ] documentation issue or request
- [ ] regression (a behavior that used to work and stopped in a new release)

Minimal steps to reproduce

prepdocs.py '..\data*' --storageaccount myaccount --container mycontainer --searchservice mysearch --index myindex -v

Any log messages given by the failure

Ensuring search index gptkbindex exists
Search index gptkbindex already exists
Processing files...
Processing '../pdf/xxx.pdf'
Uploading blob for page 0 -> xxx-0.pdf
Extracting text from '../pdfs/xxx.pdf' using Azure Form Recognizer
Indexing sections from 'xxx.pdf' into search index 'gptkbindex'
Splitting '../pdfs/xxx.pdf' into sections
Rate limited on the OpenAI embeddings API, sleeping before retrying...
Rate limited on the OpenAI embeddings API, sleeping before retrying...
Rate limited on the OpenAI embeddings API, sleeping before retrying...
Rate limited on the OpenAI embeddings API, sleeping before retrying...
Rate limited on the OpenAI embeddings API, sleeping before retrying...
Rate limited on the OpenAI embeddings API, sleeping before retrying...
Rate limited on the OpenAI embeddings API, sleeping before retrying...
Rate limited on the OpenAI embeddings API, sleeping before retrying...
Rate limited on the OpenAI embeddings API, sleeping before retrying...
Rate limited on the OpenAI embeddings API, sleeping before retrying...
Rate limited on the OpenAI embeddings API, sleeping before retrying...
Rate limited on the OpenAI embeddings API, sleeping before retrying...
Rate limited on the OpenAI embeddings API, sleeping before retrying...
Rate limited on the OpenAI embeddings API, sleeping before retrying...
Traceback (most recent call last):
File "/path/.local/lib/python3.10/site-packages/tenacity/init.py", line 382, in call
result = fn(*args, **kwargs)
File "/path/scripts/prepdocs.py", line 216, in compute_embedding
return openai.Embedding.create(engine=args.openaideployment, input=text)["data"][0]["embedding"]
File "/path/.local/lib/python3.10/site-packages/openai/api_resources/embedding.py", line 33, in create
response = super().create(*args, **kwargs)
File "/path/.local/lib/python3.10/site-packages/openai/api_resources/abstract/engine_api_resource.py", line 149, in create
) = cls.__prepare_create_request(
File "/path/.local/lib/python3.10/site-packages/openai/api_resources/abstract/engine_api_resource.py", line 83, in __prepare_create_request
raise error.InvalidRequestError(
openai.error.InvalidRequestError: Must provide an 'engine' or 'deployment_id' parameter to create a <class 'openai.api_resources.embedding.Embedding'>

The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/path/scripts/prepdocs.py", line 367, in
index_sections(os.path.basename(filename), sections)
File "/path/scripts/prepdocs.py", line 262, in index_sections
for s in sections:
File "/path/scripts/prepdocs.py", line 208, in create_sections
section["embedding"] = compute_embedding(content)
File "/path/.local/lib/python3.10/site-packages/tenacity/init.py", line 289, in wrapped_f
return self(f, *args, **kw)
File "/path/.local/lib/python3.10/site-packages/tenacity/init.py", line 379, in call
do = self.iter(retry_state=retry_state)
File "/path/.local/lib/python3.10/site-packages/tenacity/init.py", line 326, in iter
raise retry_exc from fut.exception()
tenacity.RetryError: RetryError[<Future at 0x7f4635e70ac0 state=finished raised InvalidRequestError>]
Expected/desired behavior

To create vector values so I can use the hybrid drop-down to chat to my docs

OS and Version?

Linux (Ubuntu 22.04)

azd version?

run azd version and copy paste here.

Versions

azd version 1.1.0 (commit ea9cb12575734ee6a5f99c4d415c1a51d6f32d3e)

Mention any other details that might be useful

Thanks! We'll be in touch soon.

I met the same problem.

@shulkx  can you give an example I didn't understand you properly? should I add them to my azd up command ?

The same problem is occurring.
Where can I get--openaideployment?
Which files need to be changed?

I didn't use the azd up because it contains everything inside with less customizable.
I run the prepdocs.py manually by passing in parameters to specific services (e.g., blob storage, form recognizer key, etc.,). Therefore, I can choose to pass in the above 3 parameters I mentioned.
After the indexing process is done, directly run azd deploy to deploy the webapp.

The recent change to vector search also changed the prepdocs scripts, like prepdocs.sh:
https://github.com/Azure-Samples/azure-search-openai-demo/blob/main/scripts/prepdocs.sh
The two added arguments were:
--openaiservice "$AZURE_OPENAI_SERVICE" --openaideployment "$AZURE_OPENAI_EMB_DEPLOYMENT"

If you run prepdocs.sh or prepdocs.ps1, then they will get those values automatically from your current azd environment.
If you are running prepdocs.py manually, then you'll need to find those values in .azure/CURRENT-ENV-NAME/.env and pass them in.

@pamelafox
I'm running the latest version of Azure-search-openai-demo (initialized from template yesterday), so I have those changes, but I still get the rate limit problem that results in an exception being raised as described by bitcoincode.
I have 147 documents containing only ~20k word or just under 120k characters, so I shouldn't even be hitting the rate limit.

Thanks for the details. We're looking into various approaches to work around the rate limiting. For now I'd suggest breaking the work up (like into multiple folders and running it on each folder). Sorry about the errors!

Are there any updates regarding this issue?
ERROR: failed running post hooks: 'postprovision' hook failed with exit code: '1', Path: './scripts/prepdocs.sh'. : exit code: 1
"statusCode": 401, "message": "Unauthorized. Access token is missing, invalid, audience is incorrect

Not sure if this is related to this issue thread:
Rate limited on the OpenAI embeddings API, sleeping before retrying...
Rate limited on the OpenAI embeddings API, sleeping before retrying...
Rate limited on the OpenAI embeddings API, sleeping before retrying...
Rate limited on the OpenAI embeddings API, sleeping before retrying...
Rate limited on the OpenAI embeddings API, sleeping before retrying...
Rate limited on the OpenAI embeddings API, sleeping before retrying...
Rate limited on the OpenAI embeddings API, sleeping before retrying...
Rate limited on the OpenAI embeddings API, sleeping before retrying...
Rate limited on the OpenAI embeddings API, sleeping before retrying...
Rate limited on the OpenAI embeddings API, sleeping before retrying...
Traceback (most recent call last):
File "C:\Users\tyler\AZUREOPENI\azure-search-openai-demo\scripts.venv\Lib\site-packages\openai\api_requestor.py", line 403, in handle_error_response
error_data = resp["error"]
~~~~^^^^^^^^^
KeyError: 'error'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\Users\tyler\AZUREOPENI\azure-search-openai-demo\scripts.venv\Lib\site-packages\tenacity_init_.py", line 382, in call
result = fn(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^
File "C:\Users\tyler\AZUREOPENI\azure-search-openai-demo\scripts\prepdocs.py", line 229, in compute_embedding
return openai.Embedding.create(engine=args.openaideployment, input=text)["data"][0]["embedding"]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\tyler\AZUREOPENI\azure-search-openai-demo\scripts.venv\Lib\site-packages\openai\api_resources\embedding.py", line 33, in create
response = super().create(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\tyler\AZUREOPENI\azure-search-openai-demo\scripts.venv\Lib\site-packages\openai\api_resources\abstract\engine_api_resource.py", line 153, in create
response, _, api_key = requestor.request(
^^^^^^^^^^^^^^^^^^
File "C:\Users\tyler\AZUREOPENI\azure-search-openai-demo\scripts.venv\Lib\site-packages\openai\api_requestor.py", line 298, in request
resp, got_stream = self._interpret_response(result, stream)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\tyler\AZUREOPENI\azure-search-openai-demo\scripts.venv\Lib\site-packages\openai\api_requestor.py", line 700, in _interpret_response
self._interpret_response_line(
File "C:\Users\tyler\AZUREOPENI\azure-search-openai-demo\scripts.venv\Lib\site-packages\openai\api_requestor.py", line 763, in _interpret_response_line
raise self.handle_error_response(
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\tyler\AZUREOPENI\azure-search-openai-demo\scripts.venv\Lib\site-packages\openai\api_requestor.py", line 405, in handle_error_response
raise error.APIError(
openai.error.APIError: Invalid response object from API: '{ "statusCode": 500, "message": "Internal server error", "activityId": "699ff4be-563c-4f43-821a-5fa50ac1996d" }' (HTTP response code was 500)
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "C:\Users\tyler\AZUREOPENI\azure-search-openai-demo\scripts\prepdocs.py", line 380, in
index_sections(os.path.basename(filename), sections)
File "C:\Users\tyler\AZUREOPENI\azure-search-openai-demo\scripts\prepdocs.py", line 275, in index_sections
for s in sections:
File "C:\Users\tyler\AZUREOPENI\azure-search-openai-demo\scripts\prepdocs.py", line 221, in create_sections
^^^^^^^^^^^^^^^^^^^^^^^^^^
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\tyler\AZUREOPENI\azure-search-openai-demo\scripts.venv\Lib\site-packages\tenacity_init_.py", line 289, in wrapped_f
return self(f, *args, **kw)
^^^^^^^^^^^^^^^^^^^^
File "C:\Users\tyler\AZUREOPENI\azure-search-openai-demo\scripts.venv\Lib\site-packages\tenacity_init_.py", line 379, in call
do = self.iter(retry_state=retry_state)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\tyler\AZUREOPENI\azure-search-openai-demo\scripts.venv\Lib\site-packages\tenacity_init_.py", line 326, in iter
raise retry_exc from fut.exception()
tenacity.RetryError: RetryError[<Future at 0x1f4c47d20d0 state=finished raised APIError>]

Not sure if this is related to this issue thread: Rate limited on the OpenAI embeddings API, sleeping before retrying... Rate limited on the OpenAI embeddings API, sleeping before retrying... Rate limited on the OpenAI embeddings API, sleeping before retrying... Rate limited on the OpenAI embeddings API, sleeping before retrying... Rate limited on the OpenAI embeddings API, sleeping before retrying... Rate limited on the OpenAI embeddings API, sleeping before retrying... Rate limited on the OpenAI embeddings API, sleeping before retrying... Rate limited on the OpenAI embeddings API, sleeping before retrying... Rate limited on the OpenAI embeddings API, sleeping before retrying... Rate limited on the OpenAI embeddings API, sleeping before retrying... Traceback (most recent call last): File "C:\Users\tyler\AZUREOPENI\azure-search-openai-demo\scripts.venv\Lib\site-packages\openai\api_requestor.py", line 403, in handle_error_response error_data = resp["error"] ~~~~^^^^^^^^^ KeyError: 'error'
During handling of the above exception, another exception occurred:
Traceback (most recent call last): File "C:\Users\tyler\AZUREOPENI\azure-search-openai-demo\scripts.venv\Lib\site-packages\tenacity__init__.py", line 382, in call result = fn(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^ File "C:\Users\tyler\AZUREOPENI\azure-search-openai-demo\scripts\prepdocs.py", line 229, in compute_embedding return openai.Embedding.create(engine=args.openaideployment, input=text)["data"][0]["embedding"] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\tyler\AZUREOPENI\azure-search-openai-demo\scripts.venv\Lib\site-packages\openai\api_resources\embedding.py", line 33, in create response = super().create(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\tyler\AZUREOPENI\azure-search-openai-demo\scripts.venv\Lib\site-packages\openai\api_resources\abstract\engine_api_resource.py", line 153, in create response, _, api_key = requestor.request( ^^^^^^^^^^^^^^^^^^ File "C:\Users\tyler\AZUREOPENI\azure-search-openai-demo\scripts.venv\Lib\site-packages\openai\api_requestor.py", line 298, in request resp, got_stream = self._interpret_response(result, stream) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\tyler\AZUREOPENI\azure-search-openai-demo\scripts.venv\Lib\site-packages\openai\api_requestor.py", line 700, in _interpret_response self._interpret_response_line( File "C:\Users\tyler\AZUREOPENI\azure-search-openai-demo\scripts.venv\Lib\site-packages\openai\api_requestor.py", line 763, in _interpret_response_line raise self.handle_error_response( ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\tyler\AZUREOPENI\azure-search-openai-demo\scripts.venv\Lib\site-packages\openai\api_requestor.py", line 405, in handle_error_response raise error.APIError( openai.error.APIError: Invalid response object from API: '{ "statusCode": 500, "message": "Internal server error", "activityId": "699ff4be-563c-4f43-821a-5fa50ac1996d" }' (HTTP response code was 500)
The above exception was the direct cause of the following exception:
Traceback (most recent call last): File "C:\Users\tyler\AZUREOPENI\azure-search-openai-demo\scripts\prepdocs.py", line 380, in index_sections(os.path.basename(filename), sections) File "C:\Users\tyler\AZUREOPENI\azure-search-openai-demo\scripts\prepdocs.py", line 275, in index_sections for s in sections: File "C:\Users\tyler\AZUREOPENI\azure-search-openai-demo\scripts\prepdocs.py", line 221, in create_sections ^^^^^^^^^^^^^^^^^^^^^^^^^^ ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\tyler\AZUREOPENI\azure-search-openai-demo\scripts.venv\Lib\site-packages\tenacity__init__.py", line 289, in wrapped_f return self(f, *args, **kw) ^^^^^^^^^^^^^^^^^^^^ File "C:\Users\tyler\AZUREOPENI\azure-search-openai-demo\scripts.venv\Lib\site-packages\tenacity__init__.py", line 379, in call do = self.iter(retry_state=retry_state) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\tyler\AZUREOPENI\azure-search-openai-demo\scripts.venv\Lib\site-packages\tenacity__init__.py", line 326, in iter raise retry_exc from fut.exception() tenacity.RetryError: RetryError[<Future at 0x1f4c47d20d0 state=finished raised APIError>]

And in the GUI I receive the following:

@GitTxb2
Azure OAI Outage in EAST-US effecting emedding models
FYI
Your support case @####$#%$^#&%&$&&$  is related to an ongoing outage in your region.
STATUS: In-Progress 8/8/2023 4:43:29 AM UTC
SUMMARY OF IMPACT: You may experience sporadic issues accessing your resources. This could cause you to experience a 5xx error when calling Davinci model or Ada embedding model in EastUS region.
NEXT STEPS: Azure OpenAI feature team are urgently investigating the root cause and coming up with a plan to resolve as soon as possible. We'll send an update once we know more.

@vrajroutu Thanks so much for letting me know.  Thanks again

@vrajroutu
I am getting the same error as well.
Where can I find that information?
It is not listed at
https://azure.status.microsoft/ja-jp/status

@ikka86 My CSP reports the issue has been fixed. I'll do some further testing today.

Closing this issue as it's been reported fixed in multiple venues. Sorry about that!

Sorry, re-opening since this issue was originally about the prepdocs.py issue, which folks may still run into.

Hi Any update to this issue ? or any workaround at the moment ?
I get the error even when adding a single document now .

There's a workaround in this PR: https://github.com/Azure-Samples/azure-search-openai-demo/pull/529

1- Clear the data from the data folder when doing azd up
2- Put the date on separate folder on your system
3- Once the deployment done put the files back to your data folder and use prepdoc.ps1 not prepdoc.py as .sh file will have the right parameters.
for more info look at @pamelafox  comment in here https://github.com/Azure-Samples/azure-search-openai-demo/issues/431#issuecomment-1643661657

I also experienced this error this week, but just when I use it with the option --openaikey.
My workaround is to comment or delete "refresh_openai_token()" on line 234 in prepdocs.py
This refresh_openai_token() throws an error, which then leads to the false errormessage about the rate limited.

@smeury Ah that's actually a bit of a bug, I've proposed a fix in #573


feature: delete key in terminal

Is your feature request related to a problem? Please describe.
Sometimes I naturally reach for the Delete key to delete my input from the terminal but am forced to use backspace instead.
Describe the solution you'd like
Add the intuitive Delete key behavior: to delete the character to the right of the cursor.
Describe alternatives you've considered

Just using backspace

Additional context
None needed, simple quality of life improvement for devs

fixed with #378


Add workflow to add post-pr linting suggestions

As this config is equal to actinia-core repository and has no impact on the tests, I will merge directly.


Remove active option of tasks?

There is an active option of task. The usage is generally as demontrated in the example
!echo "hello" > a.txt
task: active=not path('a.txt').exists()
sh:
    echo "Task executed"

or
task: active=-1
    echo "Only the last substep"

However, both usages can be replaced by the stop_if option
!echo "hello" > a.txt

stop_if(path('a.txt').exists())
task:
sh:
    echo "Task executed"

and the second usage is of little practical use with
input: active=-1
task:
    echo "Only the last substep"

so I am wondering if we should remove option active.

The real problem with this option is the handling of _output. For example,
input: for_each={'i': range(5)}
output: f'test_{i}.txt'

task: active=1
sh: expand=True
 touch {_output}

sos will fail because the step expects output test_0.txt, test_1.txt etc but only test_1.txt will be generated by the task.
input: for_each={'i': range(5)}, active=1
output: f'test_{i}.txt'

task:
sh: expand=True
 touch {_output}

does not have this problem.
Arguably this is the same situation with stop_if and the problem can be solved by not specifying output:,
input: for_each={'i': range(5)}, active=1

task:
sh: expand=True
 touch test_{i}.txt

but it can be confusing if user runs the workflow, removes the output, and runs again. The step would be ignored because no output in the signature.

Good move! But I did not realize we can do it for input? I have tried your example:
[1]
input: for_each={'i': range(5)}, active=1
output: f'test_{i}.txt'
sh: expand=True
 touch {_output}

Got this using sos-0.16.11, though
ERROR: [default_1]: Failed to process input statement for_each={'i': range(5)}, active=1
: Unrecognized input option active

Sorry, I mixed what I planned and what is actually implemented. I will update the ticket.

active was introduced to allow the writing of
if _index == 1:
  sh(script)

as
sh: active=1
  script

because the former does not allow the use of script format. It was later extended to task to allow controlling entire tasks.
My major concern is
input: for_each={'i': range(5)}
output: f'test_{i}.txt'

task: active=1
sh: expand=True
 touch {_output}

because it causes a problem that is not easy to fix, and I am not particularly happy with either of the solutions.

Ahh sure, I actually use this a lot:
if _index == 1:
  sh(script)

(yes, without script it is not very convenient. That's why I often wrap it with sos_run).
So is this usercase the only reason active now exists? If so, we might instead try to think of how to make this case easier, then we get rid of active

My point was that task is always the last part of the substep, and there is no syntax to wrap it in if or for. Therefore,
task: active =1

can be achieved by
stop_if(_index!=1)
task:

and the latter is safer because it removes _output of _index==1 from step_output.
Basically, task is usually the main part of the substep that produces useful _output, Cancelling it with active= without removing _output from step_output makes this option more harmful than useful.

Okay, but I cannot do:
stop_if(_index!=1)
R: ...

right? I'd have no hesitation to remove active if i can do this. But I guess we want to remove it anyways because as you pointed out it is more harmful than useful, given all things considered.

Nothing stops you from doing
stop_if(_index!=1)
R: ...

but in this case you might not want the side effect of removing _output from step_output.


socksEcho tests failing on Mac

I'm seeing 2 failed tests in socksEcho:

These tests (e.g. https://github.com/uProxy/uproxy/blob/master/src/lib/integration-tests/socks-echo/base-spec.core-env.ts#L276) are also a little weird because they are written in a way where the test can pass regardless of whether testModule.connect fulfills or rejects.  Maybe we want to remove the code with "This code should not run" comments?

I'll take a look at this before next week's release cycle starts up.

So, on my system, If I disable these two tests then all is well:

connect to many different servers in parallel
attempt to connect to a nonexistent DNS name with reproxy

Neither seems extremely important so I'll send a PR and perhaps @bemasc can have a go at investigating.


Wrong Assignment in State-Estimation

This line should be a 0. Otherwise, the publish-frequency is larger than 10Hz. Note that later in the same loop, count is alsoincremented. So in the loop-cycle of publishing, count is 2 in the end (first set to 1 and then incremented by 1).

Thank you for your feedback. I'll check.


Support new CLI OAuth flow

An upcoming CLI release will support OAuth (PR) and this will break user expectations as the OAuth flow requires user interaction. To continue to support automated environments like CI the --token flag is still supported and effectively skips the OAuth flow.
NOTE: This PR should not be merged until the associated CLI release is published.

Closing this PR until we have an understanding of the PKCE flow being proposed.


Choiceset cannot be closed after Card-Update in MS-Teams

Hello, we have developed an MS-Teams chat bot.
This issue seems at first insignificant, but is extremely annoying to our clients.
The problem:

The user opens up a Choiceset.
We update his card.
The old Choiceset is still open and cannot be closed.

Example

The client can select his Tags and apply changes with the Check. Now we have to update his card, since all values (all checked tags) disappear after a submit. After updating the card the Choiceset cannot be closed and hides the content beneath. We experience this issue with 3 other cards.
Why updating the card?
When the user clicks submit all inputs are cleared. We have another card where the user can create a question. When his inputs are invalid (e.g. too few words) we need to update his card with the sent values. Otherwise the user has to rewrite everything again.
We desperately need a solution for this issue, since we serve two big organizations.

I have experienced this issue a month ago. I could not find a workaround.
A bug-fix would be nice...

@siduppal could you help take look to identify if this is a teams issue at all? @dclaux / @matthidinger / @paulcam206 FYI

@shalinijoshi19 , Thanks for bringing up this issue, We can reproduce this as Bug. We are looking into this.
We will let you know once this got resolved.

@Trinetra-MSFT, thank you. If you need any support just let me know.

@shalinijoshi19 @janandreschweiger , The Bug is Resolved now with Pull Request 265446. Please check and let us know.

@shalinijoshi19 , Can you help to close this, This is resolved now

Thank you 😍! I recognized the change yesterday. Really awesome work of you guys!
In addition, I experienced another major change. When users submit their cards, the value (e.g. TextField input) persists. This is great, because we no longer need to update the cards (e.g. Otherwise the inputs are lost, if the action fails).
Do you know if these two changes are permanent? Is there any way to learn about a major change in advance?


HADOOP-17363. ABFS: Fix Oauth 2.0 username-password authentication issues

https://hadoop.apache.org/docs/current/hadoop-azure/abfs.html
I have tried OAuth 2.0 authentication with the username and password written above.
However, it failed with the following exception.
Exception in thread "main" HTTP Error 400; url='https://login.microsoftonline.com/3070a5de-410e-4885-XXXX-XXXXXXXXXXXX/oauth2/token' AADToken: HTTP connection to https://login.microsoftonline.com/3070a5de-410e-4885-XXXX-XXXXXXXXXXXX/oauth2/token failed for getting token from AzureAD.; requestId='187c97a4-82a0-4b36-b764-XXXXXXXXXXXX'; contentType='application/json; charset=utf-8'; response '{"error":"unauthorized_client","error_description":"AADSTS700016: Application with identifier 'jiro' was not found in the directory '3070a5de-410e-4885-XXXX-XXXXXXXXXXXX'. This can happen if the application has not been installed by the administrator of the tenant or consented to by any user in the tenant. You may have sent your authentication request to the wrong tenant.\r\nTrace ID: 187c97a4-82a0-4b36-b764-a3b8b1c45201\r\nCorrelation ID: 4eb4a71e-2eef-4788-9c8c-24f4c84f6981\r\nTimestamp: 2020-11-07 11:49:21Z","error_codes":[700016],"timestamp":"2020-11-07 11:49:21Z","trace_id":"187c97a4-82a0-4b36-b764-a3b8b1c45201","correlation_id":"4eb4a71e-2eef-4788-9c8c-24f4c84f6981","error_uri":"https://login.microsoftonline.com/error?code=700016"}'org.apache.hadoop.fs.azurebfs.oauth2.AzureADAuthenticator$HttpException: HTTP Error 400; url='https://login.microsoftonline.com/3070a5de-410e-4885-b6cd-95fe759ced2b/oauth2/token' AADToken: HTTP connection to https://login.microsoftonline.com/3070a5de-410e-4885-XXXX-XXXXXXXXXXXX/oauth2/token failed for getting token from AzureAD.; requestId='187c97a4-82a0-4b36-b764-XXXXXXXXXXXX'; contentType='application/json; charset=utf-8'; response '{"error":"unauthorized_client","error_description":"AADSTS700016: Application with identifier 'jiro' was not found in the directory '3070a5de-410e-4885-XXXX-XXXXXXXXXXXX'. This can happen if the application has not been installed by the administrator of the tenant or consented to by any user in the tenant. You may have sent your authentication request to the wrong tenant.\r\nTrace ID: 187c97a4-82a0-4b36-b764-a3b8b1c45201\r\nCorrelation ID: 4eb4a71e-2eef-4788-9c8c-24f4c84f6981\r\nTimestamp: 2020-11-07 11:49:21Z","error_codes":[700016],"timestamp":"2020-11-07 11:49:21Z","trace_id":"187c97a4-82a0-4b36-b764-a3b8b1c45201","correlation_id":"4eb4a71e-2eef-4788-9c8c-24f4c84f6981","error_uri":"https://login.microsoftonline.com/error?code=700016"}'
	at org.apache.hadoop.fs.azurebfs.services.AbfsRestOperation.executeHttpOperation(AbfsRestOperation.java:215)
	at org.apache.hadoop.fs.azurebfs.services.AbfsRestOperation.execute(AbfsRestOperation.java:134)
	at org.apache.hadoop.fs.azurebfs.services.AbfsClient.createPath(AbfsClient.java:293)
	at org.apache.hadoop.fs.azurebfs.AzureBlobFileSystemStore.createDirectory(AzureBlobFileSystemStore.java:445)
	at org.apache.hadoop.fs.azurebfs.AzureBlobFileSystem.mkdirs(AzureBlobFileSystem.java:409)
	at org.apache.hadoop.fs.FileSystem.mkdirs(FileSystem.java:2355)
	at com.sample.HelloWorld.main(HelloWorld.java:116)
Caused by: org.apache.hadoop.fs.azurebfs.oauth2.AzureADAuthenticator$HttpException: HTTP Error 400; url='https://login.microsoftonline.com/3070a5de-410e-XXXX-XXXXXXXXXXXX/oauth2/token' AADToken: HTTP connection to https://login.microsoftonline.com/3070a5de-410e-4885-XXXX-XXXXXXXXXXXX/oauth2/token failed for getting token from AzureAD.; requestId='187c97a4-82a0-4b36-b764-a3b8b1c45201'; contentType='application/json; charset=utf-8'; response '{"error":"unauthorized_client","error_description":"AADSTS700016: Application with identifier 'jiro' was not found in the directory '3070a5de-410e-4885-XXXX-XXXXXXXXXXXX'. This can happen if the application has not been installed by the administrator of the tenant or consented to by any user in the tenant. You may have sent your authentication request to the wrong tenant.\r\nTrace ID: 187c97a4-82a0-4b36-b764-a3b8b1c45201\r\nCorrelation ID: 4eb4a71e-2eef-4788-9c8c-24f4c84f6981\r\nTimestamp: 2020-11-07 11:49:21Z","error_codes":[700016],"timestamp":"2020-11-07 11:49:21Z","trace_id":"187c97a4-82a0-4b36-b764-a3b8b1c45201","correlation_id":"4eb4a71e-2eef-4788-9c8c-24f4c84f6981","error_uri":"https://login.microsoftonline.com/error?code=700016"}'
	at org.apache.hadoop.fs.azurebfs.oauth2.AzureADAuthenticator.getTokenSingleCall(AzureADAuthenticator.java:394)
	at org.apache.hadoop.fs.azurebfs.oauth2.AzureADAuthenticator.getTokenCall(AzureADAuthenticator.java:291)
	at org.apache.hadoop.fs.azurebfs.oauth2.AzureADAuthenticator.getTokenCall(AzureADAuthenticator.java:273)
	at org.apache.hadoop.fs.azurebfs.oauth2.AzureADAuthenticator.getTokenUsingClientCreds(AzureADAuthenticator.java:96)
	at org.apache.hadoop.fs.azurebfs.oauth2.UserPasswordTokenProvider.refreshToken(UserPasswordTokenProvider.java:54)
	at org.apache.hadoop.fs.azurebfs.oauth2.AccessTokenProvider.getToken(AccessTokenProvider.java:50)
	at org.apache.hadoop.fs.azurebfs.services.AbfsClient.getAccessToken(AbfsClient.java:670)
	at org.apache.hadoop.fs.azurebfs.services.AbfsRestOperation.executeHttpOperation(AbfsRestOperation.java:168)
	... 6 more

The cause of the error seems to be that UserPasswordTokenProvider is calling getTokenUsingClientCreds() for the service principal.
https://docs.microsoft.com/en-us/azure/active-directory/develop/v2-oauth-ropc
I checked the API specifications of Azure and fixed the cause of this error.

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Logfile
Comment

+0 :ok:
reexec
26m 48s

Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  0s

No case conflicting files found.

+0 :ok:
markdownlint
0m  0s

markdownlint was not available.

+1 :green_heart:
@author
0m  1s

The patch does not contain any @author tags.

-1 :x:
test4tests
0m  0s

The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.

_ trunk Compile Tests _

+1 :green_heart:
mvninstall
33m 47s

trunk passed

+1 :green_heart:
compile
0m 37s

trunk passed with JDK Ubuntu-11.0.9+11-Ubuntu-0ubuntu<IP_ADDRESS>

+1 :green_heart:
compile
0m 35s

trunk passed with JDK Private Build-1.8.0_272-8u272-b10-0ubuntu1~18.04-b10

+1 :green_heart:
checkstyle
0m 27s

trunk passed

+1 :green_heart:
mvnsite
0m 38s

trunk passed

+1 :green_heart:
shadedclient
16m  4s

branch has no errors when building and testing our client artifacts.

+1 :green_heart:
javadoc
0m 31s

trunk passed with JDK Ubuntu-11.0.9+11-Ubuntu-0ubuntu<IP_ADDRESS>

+1 :green_heart:
javadoc
0m 29s

trunk passed with JDK Private Build-1.8.0_272-8u272-b10-0ubuntu1~18.04-b10

+0 :ok:
spotbugs
0m 59s

Used deprecated FindBugs config; considering switching to SpotBugs.

+1 :green_heart:
findbugs
0m 56s

trunk passed

_ Patch Compile Tests _

+1 :green_heart:
mvninstall
0m 30s

the patch passed

+1 :green_heart:
compile
0m 30s

the patch passed with JDK Ubuntu-11.0.9+11-Ubuntu-0ubuntu<IP_ADDRESS>

+1 :green_heart:
javac
0m 30s

the patch passed

+1 :green_heart:
compile
0m 26s

the patch passed with JDK Private Build-1.8.0_272-8u272-b10-0ubuntu1~18.04-b10

+1 :green_heart:
javac
0m 26s

the patch passed

-0 :warning:
checkstyle
0m 18s
/diff-checkstyle-hadoop-tools_hadoop-azure.txt
hadoop-tools/hadoop-azure: The patch generated 2 new + 2 unchanged - 0 fixed = 4 total (was 2)

+1 :green_heart:
mvnsite
0m 29s

the patch passed

+1 :green_heart:
whitespace
0m  0s

The patch has no whitespace issues.

+1 :green_heart:
shadedclient
14m 27s

patch has no errors when building and testing our client artifacts.

+1 :green_heart:
javadoc
0m 26s

the patch passed with JDK Ubuntu-11.0.9+11-Ubuntu-0ubuntu<IP_ADDRESS>

+1 :green_heart:
javadoc
0m 24s

the patch passed with JDK Private Build-1.8.0_272-8u272-b10-0ubuntu1~18.04-b10

+1 :green_heart:
findbugs
0m 57s

the patch passed

_ Other Tests _

-1 :x:
unit
1m 28s
/patch-unit-hadoop-tools_hadoop-azure.txt
hadoop-azure in the patch passed.

+1 :green_heart:
asflicense
0m 33s

The patch does not generate ASF License warnings.

103m 34s

Reason
Tests

Failed junit tests
hadoop.fs.azure.TestBlobMetadata

hadoop.fs.azure.TestNativeAzureFileSystemConcurrency

hadoop.fs.azure.TestNativeAzureFileSystemFileNameCheck

hadoop.fs.azure.TestWasbFsck

hadoop.fs.azure.TestNativeAzureFileSystemOperationsMocked

hadoop.fs.azure.TestOutOfBandAzureBlobOperations

hadoop.fs.azure.TestNativeAzureFileSystemContractMocked

hadoop.fs.azure.TestNativeAzureFileSystemMocked

Subsystem
Report/Notes

Docker
ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2445/1/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/2445

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle markdownlint

uname
Linux c8880252bfb0 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/bin/hadoop.sh

git revision
trunk / ae7b00a9988

Default Java
Private Build-1.8.0_272-8u272-b10-0ubuntu1~18.04-b10

Multi-JDK versions
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.9+11-Ubuntu-0ubuntu<IP_ADDRESS> /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_272-8u272-b10-0ubuntu1~18.04-b10

Test Results
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2445/1/testReport/

Max. process+thread count
420 (vs. ulimit of 5500)

modules
C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure

Console output
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2445/1/console

versions
git=2.17.1 maven=3.6.0 findbugs=4.1.3

Powered by
Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

Those Test* failure are unrelated, but the testing abfs policy is always "PR supplier must declare which Azure endpoint they ran all the mvn verify integration tests against
Nominating @snvijaya and @bilaharith for review


Improve styling

Improve styling
Description
This is to improve the styling, mainly of the download list which often gets misshapen by long file names. The filename has a title on it so now when you hover over the card you can see the full name.
Happy to make more changes if you aren't happy with it.
Type of change

[x] Style

How Has This Been Tested?
Visually.

Checklist:

[x] My code follows the style guidelines of this project
[x] I have performed a self-review of my own code
[x] I have commented my code, particularly in hard-to-understand areas
[ ] I have made corresponding changes to the documentation None needed?
[ ] I have added tests that prove my fix is effective or that my feature works
[x] New and existing unit tests pass with my changes
[x] I have checked my code and corrected any misspellings

Thanks @JJeeff248!
This looks nice and I suck at building web interfaces.
I'll QA later today!


chore: Remove panic on arm64 linux builds for bb binary at compile time

Description
With the addition of #3437 -- we can compile without a backend. If one is on an arm64 linux build, they will get a compile time panic. This PR changes it to a warning and just doesn't set the environment variables for the bb binary.
Problem*
Resolves
Summary*
Additional Context
Documentation*
Check one:

[ ] No documentation needed.
[ ] Documentation included in this PR.
[ ] [Exceptional Case] Documentation to be submitted in a separate PR.

PR Checklist*

[ ] I have tested the changes locally.
[ ] I have formatted the changes with Prettier and/or cargo fmt on default settings.

just doesn't set the environment variables for the bb binary.

This is currently still set but for the x86 binary

just doesn't set the environment variables for the bb binary.

This is currently still set but for the x86 binary

Yep, we just don't want it for arm64 -- Additionally, we can follow up with another PR that allows users to specifically set that they want no backend like what was done here CC @guipublic to take that on as a followup work to your "compile no-backend" PR


Updated - Resolve sync only after heads have been added

Builds off of #38, resolving issue with event listeners.
Allows https://github.com/orbitdb/orbit-db/pull/606 to be merged.

@chmanie For you to review as well :)

This has been a long time coming :)
LGTM, really! Happy we're getting this in after all

What say you, @haadcode and @shamb0t ?

@msterle Thanks for your patience in getting this merged. I'm having some issues running the automated tests against this, but I don't think it's your fault at the moment. It looks like an js-ipfs version thing. Would you be willing to help me unravel this? We can meet up on Gitter and walk through it. I just made a commit to orbit-db-test-utils that unblocks us partly.

@msterle LGTM! :+1:


Terminal/Output panel print '[Object]' instead of value from level-3 on within json disturbing debugging

VSCode Version: 1.14.0
OS Version: win7/64bit

Steps to Reproduce:

js:

console.log({a: {b: {c:  {d:1} } }})

print in Output Panel( same as Termial Panel):

{ a: { b: { c: [Object] } } }

Reproduces without extensions: No

What version of node are you using?

{ a: { b: { c: [Object] } } } is the expected output.
Just run node from the command line and you will get the same result.
So this behaviour is independent from VS Code.


phased variant problem

Hello
I am having trouble getting freebayes to call phase variants.  To illustrate my problem (and avoid confidentiality issues with my company) I’ve created a simulated dataset (as shown below) which closely resembles my actual data except the two linked deletions in the simulation are homozygous instead of a mix of reads with 2 deletions and reads with none.  In brief my actual reads are stitched (merged) 250BP PE Illumina reads derived from a 374 bp amplicon.  Therefore the stitched reads cover the entire amplicon.
Creating simulated dataset:
wgsim -N 1000 -d 342 -s 0 -1 250 -2 250 -h  ampR.2dels.fa  read1.2dels.fq read2.2dels.fq
flash  -m 25 -M 300 -o test -t 1 read1.2dels.fq read2.2dels.fq |tee test.flash.log
bowtie2-build ampR.fa ampR.fa
bowtie2 --sensitive-local -p 4 -x ampR.fa -U test.extendedFrags.fastq  -S test.extendedFrags.fastq-vs-ampR.fa.sam 2>bowtie2.log
samtools view -Sb -o  test.extendedFrags.fastq-vs-ampR.fa.bam test.extendedFrags.fastq-vs-ampR.fa.sam
samtools sort test.extendedFrags.fastq-vs-ampR.fa.bam test.extendedFrags.fastq-vs-ampR.fa.sort
samtools index test.extendedFrags.fastq-vs-ampR.fa.sort.bam
I then ran freebayes
freebayes -f ampR.fa -F 0.1 -C 2 -K --haplotype-length 100 test.extendedFrags.fastq-vs-ampR.fa.sort.bam >test.extendedFrags.fastq-vs-ampR.fa.vcf
which produces:
CHROM  POS     ID      REF     ALT     QUAL    FILTER  INFO    FORMAT  unknown
ampR    65      .       AACGGATGTAATGA  AA      5382.63 .       AB=0;ABP=0;AC=2;AF=1;AN=2;AO=859;CIGAR=1M12D1M;DP=873;DPB=124.714;DPRA=0;EPP=7.25971;EPPR=0;GTI=0;LEN=12;MEANALT=2;MQM=12.2154;MQMR=0;NS=1;NUMALT=1;ODDS=1192.79;PAIRED=0;PAIREDR=0;PAO=0;PQA=0;PQR=0;PRO=0;QA=14603;QR=0;RO=0;RPL=0;RPP=1868.31;RPPR=0;RPR=859;RUN=1;SAF=409;SAP=7.25971;SAR=450;SRF=0;SRP=0;SRR=0;TYPE=del        GT:DP:DPR:RO:QR:AO:QA:GL      1/1:873:873,859:0:0:859:14603:-800.827,-258.585,0
ampR    81      .       ACCCGCAGCTATCTCCCGCTTA  AA      5580.28 .       AB=0;ABP=0;AC=2;AF=1;AN=2;AO=896;CIGAR=1M20D1M;DP=896;DPB=81.4545;DPRA=0;EPP=7.28539;EPPR=0;GTI=0;LEN=20;MEANALT=1;MQM=12.269;MQMR=0;NS=1;NUMALT=1;ODDS=1244.09;PAIRED=0;PAIREDR=0;PAO=0;PQA=0;PQR=0;PRO=0;QA=15136;QR=0;RO=0;RPL=0;RPP=1948.65;RPPR=0;RPR=896;RUN=1;SAF=427;SAP=7.28539;SAR=469;SRF=0;SRP=0;SRR=0;TYPE=del GT:DP:DPR:RO:QR:AO:QA:GL      1/1:896:896,896:0:0:896:15136:-831.754,-269.723,0
Herein I see no indication that these two deletions are linked though 839 of the 1000 read pairs contain this partial cigar string: 12D4M20D.
Help would be much appreiciated.
Thanks
Mark
simulated_data.zip

Hi Mark
This post on the freebayes mailing list might help clarify the phasing in freebayes: https://groups.google.com/forum/#!topic/freebayes/fyhho8_H7J0

Thanks for the suggestion but it appears that there is a problem with freebayes in handling the situation I described.  I sent a test set to Erik that reproduces the problem which he confirmed.  He said he would work on it but had no explanation for its failure.  He suggested another app (whatshap) but it didn't work either so I've had to roll my own solution to this particular problem

Hi Robert
Thanks for the suggestion but it appears that there is a problem with freebayes in handling the situation I described. I sent a test set to Erik that reproduces the problem which he confirmed. He said he would work on it but had no explanation for its failure. He suggested another app (whatshap) but it didn't work either so I've had to roll my own solution to this particular problem
Mark
From: Robert Syme<EMAIL_ADDRESS>Sent: Wednesday, December 21, 2016 3:16 AM
To: ekg/freebayes<EMAIL_ADDRESS>Cc: Rose Mark USRE<EMAIL_ADDRESS>Author<EMAIL_ADDRESS>Subject: Re: [ekg/freebayes] phased variant problem (#317)
Hi Mark
This post on the freebayes mailing list might help clarify the phasing in freebayes: https://groups.google.com/forum/#!topic/freebayes/fyhho8_H7J0
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHubhttps://github.com/ekg/freebayes/issues/317#issuecomment-268462466, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AAsxJn1iCE4hXMbBiJngRsSy_r8bQlE3ks5rKOAkgaJpZM4J6IIS.

This message may contain confidential information. If you are not the designated recipient, please notify the sender immediately, and delete the original and any copies. Any use of the message by you is prohibited.


jfes.c issues part 1

You need to verify the pointers against NULL before dereferencing them in jfes_memcmp() and jfes_memcpy().

static const void *jfes_memcpy(const void *dst, const void *src, jfes_size_t count)
dst should not be const;
return void should not be const as well.

jfes_allocate_string(): what if malloc() fails? still returning success?

I guess, pointers to jfes_malloc, jfes_free, jfes_memcpy functions should be verified against NULL before being called.

jfes_is_null(const char *data, jfes_size_t length), jfes_is_boolean(const char *data, jfes_size_t length)
Why do you need length parameter here? Better calculate data length within the function.

Accepted.
Accepted.
Accepted.
C99 Standard says about memcpy:

If an argument to a function has an invalid value (such as a value outside the domain of the function, or a pointer outside the address space of the program, or a null pointer, or a pointer to non-modifiable storage when the corresponding parameter is not const-qualified) or a type (after promotion) not expected by a function with variable number of arguments, the behavior is undefined.

So, I don't sure I need to check this.
5. Accepted. But I'll make this arguments optional, as in many jfes_... functions. You can pass zero and then length will be calculated by jfes_strlen().


Including RulePolicy leads to infinite max_history being used to deduplicate training trackers

Rasa version: 2.4.1
Issue:
When rules/RulePolicy are included in dialogue management, in order to check for conflicts with stories, effectively infinite max_history is used. This greatly increases the time it takes to load and process dialogue data.
Command or request that led to error:
rasa data validate stories

Definition of Done

[ ] set max_history during rasa data validate stories based on the training data

one can read rules first, calculate the length of the longest rule, then use it as rulepolicy’s effective max_history which can be compared with max_history of other policies to calculate unique_last_num_states limit for deduplication of generated trackers

@melindaloubser1 I tried to reproduce this on Sara but the validation is super quick. Note that the max_history currently has to be set manually (I tried with it set and with the default / none set) and we don't distinguish for story vs. rule validation.
I also don't see how max history should be a big deal in my opinion as rules are somewhat enforced to be very short anyway. cc @JEM-Mosig

And from what I can see from the code the included policies shouldn't matter for rasa data validate stories at all

No you're right; I'm thinking I miswrote this and it applies only to loading during rasa train - would that make more sense?

That makes more sense I think 🤔 I think what you're then suggesting to set max_history here in an automated fashion then? What do you think of that @samsucik ?

@wochinge I explored the relevant code parts and I agree with automatically setting max_history to a value as low as possible. However, we can't compute this value before training the policy, right? Because a rule can lead to a long sequence of states even if it has just a single user intent (the intent can be followed by many actions), and so we need to see the actual rule data to determine the minimal max_history.

Yeah, fair, that's what would need to happen... Like this, right?

initialise all policies
load the data and determine max_history
update rule policy's max_history

What about batches, though? I think we'd like to use the same number of turns for tracker deduplication across batches. And perhaps we can say that rule data won't come in batches, but story data will... Maybe we'd need an "initial" pass through the data to determine things like this unique_last_num_states value -- kind of similar to an initial pass in the NLU pipeline for the purposes of constructing the tokeniser's vocab? 🤔 I guess I'm less worried about it for now, but more in the context of Rasa 3.0 🙂

load the data and determine max_history

We can even do this during train within the RulePolicy, no? In my opinion we shouldn't leak such a RulePolicy-specific thing to code outside of the RulePolicy.
I don't understand the part about the batches. The RulePolicy has no concept of batches or whatsoever. Do you mean training in chunks?

We can even do this during train within the RulePolicy, no?

I think this doesn't work because we use the determined max_history value in loading the training stories for other policies, or not? 🤔

I don't understand the part about the batches. The RulePolicy has no concept of batches or whatsoever. Do you mean training in chunks?

Fair, we call these "chunks", not "batches". I know that RulePolicy currently doesn't do anything with chunks, but I was worried about this in terms of future versions of Rasa where, if we have some general way of loading data from disk and providing it to different policies, I don't see a good reason for saying "any policy can consume data in chunks, but RulePolicy can't". But maybe I'm misunderstanding the extent to which Rasa 3.0 will be flexible regarding this. If there's still the guarantee that all rule data will be loaded at once (and before any story data), then I'm not worried at all.

I think this doesn't work because we use the determined max_history value in loading the training stories for other policies, or not? 🤔

We do but that they can just use whatever max_history they want to use, right? The policies are not dependent on each other.

I know that RulePolicy currently doesn't do anything with chunks, but I was worried about this in terms of future versions of Rasa where

I think you made an excellent point here.

The policies are not dependent on each other.

Hm, fair point, I didn't think about it this way. I worried about it because right now we load & de-duplicate all the Core data at once, for all policies, and so all policies receive trackers that have been de-duplicated using the same unique_last_num_states value. If each policy receives data de-duplicated based on that concrete policy's max_history, then what you're saying sounds perfect and my worry vanishes 🙂

If each policy receives data de-duplicated based on that concrete policy's max_history, then what you're saying sounds perfect and my worry vanishes 🙂

I'm not sure that I understand this. Are you saying that all policies must have the same max_history because things would go wrong otherwise? I was under the impression that every policy gets fully trackers (deduplicated and augmented but nothing else) and than the featurizer in the policy handles featurization of the tracker based on the max_history param which we pass into it.

I'm getting lost in my own thoughts and am making mistakes along the way, sorry. I'll try to backtrack and clarify:

I'm not against the current approach where we take the maximum max_history, use it for de-duplication of all trackers, and pass these trackers to all policies.
The same way, I'm not against each policy using its own max_history (it can even do its own de-duplication of trackers, whatever)

I'll go back to:

The policies are not dependent on each other.

I agree with this. I guess what I had in mind is the fact that we load and de-duplicate all trackers (rules, stories) together. If we can process rules vs stories a bit separately, then I can totally imagine a scenario where we:

let the non-rule policies do their things as they do today (determine the maximum max_history value > load all stories > de-duplicate using that value)
do a separate thing for RulePolicy where we load the rules > determine max_history from them > de-duplicate the rule trackers

I hope I made things a bit clearer now, sorry for the convoluted thoughts in my earlier replies...

Yes, you did! Rule trackers are already deduplicated separately.
So your 2nd approach would be completely feasible.
I'll do some debugging later to find out where we spend so much time exactly (during deduplication or featurization) and then maybe a PR so we have something more tangible to discuss.

Oh, no, I completely overlooked the separation in the code. I learned something new today...

Investigation results on rasa-demo:

the contradiction checks takes the huge chunk of time (1.2 seconds for the actual training, 13 seconds for the contradiction check)
Setting a mix history for the RulePolicy doesn't change this
Configuring a max_history for the rule generation leads to contradictions 🤔
I also profiled the RulePolicy training (see results here - viewable with snakeviz). the results stress that the contradiction check takes the longest amount of time. In my opinion this clearly shows that it's not related to max_history but rather to the amount of predictions. The biggest part seems to be caused by the json.loads calls. If we actually wanna do something then we need to revisit the mechanics of the policy itself (e.g. how data is serialized) and how we can speed up predictions. This should in my opinion be done in a separate issue.

A quick fix for this issue would to use lru_cache on _rule_key_to_state. By doing this I lowered the contradiction check time from 13 seconds to ~4 seconds. The issue is that it rather addresses symptoms than causes.

@wochinge wow, these are very interesting findings! Btw have you uploaded the right profiling results? In these ones, I don't see json.loads or any other expected bits anywhere...
I agree with creating a separate issue for the root cause.
As for the quick fix using caching, do you think it's safe with regards to Rasa's usage of RAM? If the usage stays below the kind of magnitudes we require for actually training the policies, then it sounds good to me and it would definitely make people's lives (mine included) easier -- way before we get to address the actual cause.
Aaand I'm especially curious in the details around this:

Configuring a max_history for the rule generation leads to contradictions 🤔

Sorry, it was the wrong link 🙈 Updated it with the correct one 👍🏻

As for the quick fix using caching, do you think it's safe with regards to Rasa's usage of RAM?

We can set a max_size for the cache 👍🏻 It's just a couple jsons and their dict representations so I'd expect it be tiny.
So next steps:

create PR with cache (I include details about memory usage)
separate issue for overall performance?

didn't really dive deeper into it tbh. I used the max_history which was specific in the config of the AugmentedMemoizationPolicy for Sara. I guess 5 might be too short for some rules?

I used the max_history which was specific in the config of the AugmentedMemoizationPolicy for Sara. I guess 5 might be too short for some rules?

Ah, okay, makes sense. I'm also getting such errors when I use only RulePolicy and hard-code its max_history to anything smaller than 4 🙂

Posting the details for the memory usage here as I haven't created the PR yet and need to put the results somewhere 😄

With Sara I've a 248 cached items with 1 424 684 hits after the training.
The cache stores a mapping of call params and output. 248 items have a total size of 862 920 (88 992 + 773 928) bytes. With a max_size of 1000 items for the cache we have max memory usage of 3 479 516 bytes (3.32 MiB) which seems okay to me 👍🏻


[feat] Allow Tauri configurations to be set in Tauri.toml and/or Cargo.toml

Describe the problem
As a Rust developer I have a certain love for the TOML format. There's nothing directly wrong with the JSON configuration file, it's just not as easy to manage as a TOML file. For example, you can't add comments in a JSON file, something that would be very useful for the Tauri config seeing as it has gotten pretty long, there's a lot of stuff to keep in memory regarding why you do things in your configs and it would be very nice to be able to put comments in there.
Describe the solution you'd like
Add the ability to write configurations in a Tauri.toml file, and/or directly in the Cargo.toml file. Due to best practises with Cargo.toml however it's probably best to use a Tauri.toml file.
# Tauri.toml
allowlist = { all = true }

[build]
beforeBuildCommand = "yarn build"
beforeDevCommand = "yarn dev"

# Cargo.toml
[package.metadata.tauri]
allowlist = { all = true }

[package.metadata.tauri.build]
beforeBuildCommand = "yarn build"
beforeDevCommand = "yarn dev"

Implementing this feature shouldn't be much more work than to find the configuration from either Cargo.toml or Tauri.toml, converting TOML into JSON, then passing it forward into existing functionality. As can be seen above with how I put allowlist at the root of the Tauri.toml file, it may be preferable to do some restructuring of the configurations if you add them to TOML since a Tauri.toml file containing a [tauri] section doesn't make much sense, but since that can make it a bit difficult to maintain both the JSON and TOML variants restructuring isn't really necessary.
Alternatives considered
Alternatively a less user friendly variant but makes it easier to maintain both the JSON and TOML variants:
[package.metadata.tauri]
config = {
    # Pretty much just directly the contents of tauri.conf.json
}

This variant might make it a bit easier to maintain both the JSON and TOML variants,  since the other suggested solution may involve some slight restructuring of the config file.
Additional context
Here's a little utility function I wrote for Cargo Commander for converting JSON to Toml. Can be used as inspiration.
https://github.com/adaptive-simon/cargo-commander/blob/main/src/utils.rs#L240

you can't add comments in a JSON file

We do have JSON5 support though. And we could easily add toml support.

I saw in your commit that you're adding tauri.conf.toml, which makes sense, but is there any chance Tauri.toml could also be added? 100% just a matter of preference, I already have Cargo.toml and Commands.toml, having Tauri.toml would look nicer in my file tree than tauri.conf.toml 😅

I would prefer Tauri.toml too but it would mean some new additions that doesn't align well with the current API (that only leverages different file extensions).

Ahh I just remembered the config module is documented as unstable so I can do some changes :) (otherwise adding any new config value would be impossible since everything is also public for the codegen step).

@adaptive-simon Your wish is my command :)

@adaptive-simon Your wish is my command :)

Omg awesome! You're the best!! 😄

Now we just need to update the documentation :)


Sending front end logs with "fields" attribute results in a 400 (on Vercel)

I am seeing repeated 400 errors with axiom logs for front-end logs in production.
For instance, this call (where connectors is an empty array):
log.debug("getConnectors", { connectors });
Generates a request with this body
'[{"level":"debug","message":"getConnectors","_time":"2023-05-24T10:02:53.039Z","fields":{"connectors":[]},"vercel":{"environment":"production","source":"frontend"}}]'
This throws a 400 and returns {}.
If I manually trigger this same request using curl using this body:
'[{"level":"debug","message":"getConnectors","_time":"2023-05-24T10:02:53.039Z","vercel":{"environment":"production","source":"frontend"}}]'
I get a 200 back.
If I send a message that generates this body, everything is also fine:
[{"level":"debug","message":"Page loaded /my/path","_time":"2023-05-24T10:02:46.300Z","fields":{},"vercel":{"environment":"preview","source":"frontend"}}]
Is it not possible to log a Javascript object/array to Axiom or is this a bug?

hmmm, that should be fine, the payload looks correct to me. can you provide more info on

which next and next-axiom versions are you using?
and is this on vercel platform with axiom integration enabled?
does the 400 error has a description saying why it fails?
have you tried other log lines, are they are responding with 400?

@schehata Thanks for taking a look.
In our Package.json:
"next-axiom": "^0.14.0"```

And in PNPM lock:

* next-axiom<EMAIL_ADDRESS>* next 13.2.4(@babel/core@7.21.0)(react-dom<EMAIL_ADDRESS>
This is using the Vercel Axiom integration. We created the integration a few months ago, so perhaps I need to do it again?

The 400 does not give a description of the error, just an empty {} in the body.

Other log lines are working, so the credentials supplied by the Axiom integration appear to be correct. Anything that does not contain this second argument seems to be accepted fine:

I thought this could potentially be Cloudflare blocking the /_axiom route because we have Cloudflare in front of the web app. But I've also tested on a Vercel preview and have the same issue.

Sometimes the logs send multiple log lines as an array. If any of those lines has a value for the "fields" key then the entire request will fail.

Hi folks, we are still experiencing this and are unable to rely on Axiom for logs. Can anyone help here? I'm happy to give you access to an account on our web app to see what's going on.

hi @stefl , I am sorry that you are facing such problems that is affecting your experience using Axiom. I have been trying to reproduce this but I couldn't. I have tried just now to send some logs with the second argument and it worked fine.

I took this back to the team, and I will hopefully get back with answers. The only guess I have at the moment is the number of columns you are sending, there is a limit of number of columns that could be saved, but otherwise the json you showed looks fine to me.

@stefl hey, does this problem persist? can you check the response of /_axiom/logs? it should print an error now.

Having the same problem.
I have following env variable:
NEXT_PUBLIC_AXIOM_INGEST_ENDPOINT=https://axiom-vitals.com/api/v1/send?configurationId=ic...0&projectId=ef...b0&type=web-vitals
The response of _axiom/logs returns "bad request".
When removing the this variable (on localhost) and everything works fine.
What I'm trying to do is to log browser errors, next to having the Vercel integration logs
So I Added an event listener on error. window.addEventListener("error", errorHandler); And in the errorHandler I try to send a log to axiom (which fails when deployed on vercel)

I am wondering if this is happening because you reach Axiom's fields limit of your accounts?


Selection Highlight works improperly on version 1.87.0

Does this issue occur when all extensions are disabled?: Yes

Version: 1.87.0 (user setup)
Commit: 019f4d1419fbc8219a181fab7892ebccf7ee29a2
Date: 2024-02-27T23:41:44.469Z
Electron: 27.3.2
ElectronBuildId: 26836302
Chromium: 118.0.5993.159
Node.js: 18.17.1
V8: <IP_ADDRESS>-electron.0
OS: Windows_NT x64 10.0.19045
Steps to Reproduce:

Match Whole Word is off (Alt+W)
Select any word with double click
Switch tab and return to previous tab
All matched words remain selected and highlighted
Modify file to remove selection

Another case:
I just press s, then Esc, switch tab and come back, s is highlighted. Why?

I have the same

Same here, and I think someone found the issue in the 1.8.7 insiders build 3 weeks ago but it wasn't addressed: https://github.com/microsoft/vscode/issues/204656

Jumping in here to say that this is likely the same issue as #206486, and should be fixed in the latest insiders build.
To confirm that this is the latest issue, did you have the setting "editor.occurrencesHighlight" set to "off"? This was the cause of stuck highlighting upon editors gaining focus as demonstrated in the gifs and images above.

Closing in favor of the issue tracked in the above comment. The fix is out on insiders and is coming in stable recovery 1.87.1
/duplicate of https://github.com/microsoft/vscode/issues/206486

Thanks for creating this issue! We figured it's covering the same as another one we already have. Thus, we closed this one as a duplicate. You can search for similar existing issues. See also our issue reporting guidelines.
Happy Coding!


Fix bookmark menu toggle

The current action only adds the bookmark and doesn't toggle it off.

This is fixed and will be closed once published in the next alpha release.


[iOS] - Background music not playing on iOS

From forum topic https://forums.xamarin.com/discussion/comment/141708#Comment_141708
Issue 1 (workaround in place):
Within a CCLayer's AddedToScene I am setting up some background music using an mp3 file with
CCSimpleAudioEngine.SharedEngine.PlayBackgroundMusic ("sounds/Music/GameTrack1", true);
This worked out of the box when running on Android. On iOS, the background music wasn't playing at all but there were no errors about missing the file.
Then on a whim I decided to just add right after the above line:
CCSimpleAudioEngine.SharedEngine.BackgroundMusicVolume = 1.0f;
And now this plays fine on iOS too. These are the first calls to any use of CCSimpleAudioEngine in the app. Is this a known issue that you have to explicitly tickle the volume setting on iOS but not Android?

Closed via commit : https://github.com/mono/CocosSharp/commit/e888b105e07e87956fb60034bc59063da278b7ff


time: time.String() defines format only in comment, with no matching Parse string

The documentation for time.String() shows the format used to convert time into a string: http://golang.org/pkg/time/#Time.String.
This string appears to be different to the const formats already defined in the time package, so if I wanted to naively (not using time.Format) convert a time -> string -> time I'd need to copy the format from the docs into my code and use it in a call to time.Parse.
In addition this format string is used a few times in http://golang.org/src/time/format_test.go as a literal string.
Isn't this format string a good candidate to be placed in the consts?

+1
I've considered raising this an an issue before.
Too late now, but I wish time.String() produced RFC3339,
the same format used by default in the marshaling methods.

https://github.com/golang/go/blob/master/src/time/format.go#L400
We cant change it currently but we could use a const here and add it to the format consts that are exported. That would not change the output and give others access to the format string. If later a incompatible change is possible we can define that new const as RFC3339.
What would be a good name for it? StdDate?

I suggested TimeString as a possibility. I did consider Std, but it's not really "standard".

Since it is already in the time package. time.FormatString sounds better to me.

What about "time.Go" since we have already "time.Ruby" and this would resonate.

We have "time.RubyDate" so then it would rather be "time.GoDate" however this sounds more like its special to me and not so much the default.

This is a duplicate of issue #6587

So it is, and that issue was closed as working as intended.  So, closing.


Ask detailed questions about the permutation strategy in the RAR

Hello author, I am very grateful for your excellent work and generous open source. After reading your source code, I have a small question, why the permute strategy does not shuffle the initial condition? Anyway, you will return to the original permutation after annealing.  Here are some of the source codes that I am confused about:
 # cls_token, condition, the permute does not impact these prefix tokens.
prefix = 2
pos_embed_prefix = pos_embed[:, :prefix]
pos_embed_postfix = self.shuffle(pos_embed[:, prefix:prefix+self.image_seq_len], orders)

Is it because of the contribution of target-aware positional embedding, it doesn't matter if it is not shuffled?

I think it is because that the prefix does not belongs to the fine-grained image content.

Thanks, as @pansanity666 suggested, prefix refers to class token and condition token, while we only permute the image tokens (referred as postfix in the code).

Thanks for your great work! @cornettoyu  I’m wondering about the difference between cls_token and condition_token. Shouldn't the condition_token just be some class_id tokens with a randomly augmented none_class token? Is there any particular reason to use a separate cls_token?
Thanks!

Thanks for your great work! @cornettoyu I’m wondering about the difference between cls_token and condition_token. Shouldn't the condition_token just be some class_id tokens with a randomly augmented none_class token? Is there any particular reason to use a separate cls_token?
Thanks!

Hi,
I see the terms could be abused and misleading. To clarify, here condition_token indicates the external condition input for guiding generation (e.g., ImageNet class), while cls_token refers to the learnable placehold token, as is used by the original ViT paper. Since we try to minimize the architecture change, we keep the cls_token as in the original ViT, but it should be fine removing it.


Trying to remove this program and folder continues to reappear

https://user-images.githubusercontent.com/43358890/168947253-4bf1be6f-6d20-4050-8663-151c5ae9b386.mov

I believe that the reappearing folder contains a cache.json file, meaning that the script is still running in background.
Follow the instructions here to disable and remove the launch agent first.


:sparkles: Add Redis streaming guide

Adds documentation for streaming Postgres data to Redis and updates SQS backfill description
This PR adds a new guide explaining how to stream Postgres database changes to Redis Streams in real-time. The guide covers:

Prerequisites and setup requirements
Step-by-step configuration of Redis sinks
Connection parameter details
Verification and debugging steps
Links to related documentation

It also improves the clarity of the backfill description in the SQS streaming guide.

[!WARNING]
This pull request is not mergeable via GitHub because a downstack PR is open. Once all requirements are satisfied, merge this PR as a stack on Graphite.
Learn more

#577
#576
#575
#574  👈 (View in Graphite)
#573
#570
main

This stack of pull requests is managed by Graphite. Learn more about stacking.


Language redirect

Hey,
didn't know if you got a notification, because I've posted it on the closed issue.
I've just tried it with example "basic-tap-i18n" and updated versions - the auto language redirect is still not working. I've made the following changes (also added a "de.i18n.json"):
init.js
I18NConf.configure({
languages: ["it", "en", "es", "de"],
defaultLanguage: 'en',
autoConfLanguage: true
});

Router.configure({
layoutTemplate: 'layout',
   i18n: {

    langCodeForDefaultLanguage: false
}
})

If I visit "localhost:3000", I get the english version now, but will not be redirected to the german one (browser language) automatically. If I set langCodeForDefaultLanguage: true I'll get redirected automatically to the german one - but I don't want an extra route ("/en") for default language.
So autoConfLanguage:true isn't working when we have langCodeForDefaultLanguage: false

Hi,
this is not a bug but an intended feature, as explained in #71 :
if you configure langCodeForDefaultLanguage: false you are saying in your case that "en" routes should not have a prefix so when you call / (home route without any prefix) it's like if you are calling /en therefore you are SETTING language to en (i.e. "asking the document in en" in HTTP terms) no matter what your browser says.
I could add a configuration option to override this behavior, that is to always give precedence to navigator language but it wouldn't make much sense not being able to explicitly ask for a page/document in a specific language (would it?).

Hi,
yeah I think it would, because if the users language is available, he should read the page in his language. For example, www.spotify.com redirects me automatically to www.spotify.com/de - a US user stays on spotify.com, because it is the main version.
Think it also looks better for SEO, when you have a main version in english on the root path. If I would activate langCodeForDefaultLanguage: true, there is only "/en" available and no version on the root path.

Ok I see there are cases where this seems to makes sense, I might add a special option for this (but leave the behavior disabled by default).
My point is that by always letting the navigator language override the language in this case there will be non bookmarkable resources. In your cases a user whose browser language is "de" will not be able to reach the english version of a page, moreover GET will not be idempotent: the same url will mean two different documents for two different users having different browser languages. Not how the web is supposed to work but I understand it's something some people want :-)
As for the last thing please open separate issues for separate problems.

All in all what you are asking for is allow the navigator language to always override the url language, this could be already simply done on a Meteor.startup:
Meteor.startup(function() {
   var navLang = I18NConf.getNavigatorLanguage();
   var bestLang = I18NConf.getBestMatchingLanguage(navLang);
   I18NConf.setLanguage(bestLang);
});

This would always override the url language no matter what i18n url the user ask for.


Add link to "The 'F' Word" video series

See https://fortran-lang.discourse.group/t/2679
cc @fluidnumerics-joe

@awvwgk , Thanks for making this suggestion!

@fluidnumerics-joe is there any alternative name that we could use? How about "The 'Fortran' Word"?
I personally feel this title is unfortunate as it coincides with a word that we should not be using here. If others think this title is ok, then I am fine to be outvoted on this.

I personally feel this title is unfortunate as it coincides with a word that we should not be using here. If others think this title is ok, then I am fine to be outvoted on this.

I think this coincidence is wished. In the context I think it is quite clear what is meant. However, we can certainly adjust the description to make it clearer. Suggestions are welcome, I just copied a bit from Joe's post on discourse.

I had the same thought for a few seconds but I think it's fine. I find it cheeky and smart.

I bet every reader of this will have the same thought. I provided a suggestion above. I just wanted to check on this, but it looks like it's fine with others, so I am ok overall.

@certik, much agreed. Thanks for the suggestion to keep this a bit more palatable.

@fluidnumerics-joe thanks for all your videos! Fortran needs that.
Once LFortran gets to beta (from alpha), this year, I am hoping to collaborate with you and others who have Fortran codes. There is all kinds of cool work that we could do, from GPU support to optimizations. We are adding some excellent optimizations, we just merged this one: https://gitlab.com/lfortran/lfortran/-/merge_requests/1649, which optimizes if (modulo(int(Nd), 2) == 1) x = -x into floating point "bit hacks" in LLVM/assembly using shift and xor. GFortran wasn't able to do that. Once we have enough of such optimizations, I think people will be very excited with the performance.

I also found the title smart and provocative in the given context.

#build_preview

Thanks everybody, I'm going ahead and merge.
#delete_preview

Once LFortran gets to beta (from alpha), this year, I am hoping to collaborate with you and others who have Fortran codes. There is all kinds of cool work that we could do, from GPU support to optimizations. We are adding some excellent optimizations, we just merged this one: https://gitlab.com/lfortran/lfortran/-/merge_requests/1649, which optimizes if (modulo(int(Nd), 2) == 1) x = -x into floating point "bit hacks" in LLVM/assembly using shift and xor, which we needed for a very fast sin(x) function implementation. GFortran wasn't able to do that. Once we have enough of such optimizations, I think people will be very excited with the performance.

@certik I'd love to dive into LFortran with you all. I recall some time ago, we had briefly discussed Focal (@LKedward ) and the possibility of supporting device subroutines and other Fortran native approaches for GPU acceleration. That being said, I'm still game for the Fortran-C interoperability approach.


Update Set-WinHomeLocation.md

added space to create a clickable link to http://go.microsoft.com/fwlink/?LinkID=242308

Docs Build status updates of commit c268860:
:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

docset/winserver2012-ps/internationalcmdlets/Set-WinHomeLocation.md
:white_check_mark:Succeeded

For more details, please refer to the build report.
Note: If you changed an existing file name or deleted a file, broken links in other files to the deleted or renamed file are listed only in the full build report.

@JohanFreelancer9 Minor style checks needed for this PR. Thanks.

@mrbodean Would you be able to follow-up with revision suggestions? We would like to proceed with merge. Thank you.

@mrbodean Closing as there has been no follow-up with the reviewers' revision suggestions. Please re-open the PR to proceed with the review. Thank you for your patience with the process and your contributions.


Issue in rainfall plot

Hi,
I am running maftools for creating rainfall plots with detectChangePoints set to TRUE. However, i am not able to get Start_Position, End_Position, nMuts, Avg_intermutation_dist, Size, number of C>A, C>T, T>C, C>G, T>A, T>G changes in my result. I am only able to get the chromosome and position in my result. Is there any way that i can get those details in my result? Thanks
Regards
Raman

Hi Raman,
This feature is only available on Github version of maftools for now. Maybe you can install it from github and try.
library("devtools")
install_github(repo = "PoisonAlien/maftools")

Let me know if you find any issues.

I have installed using the github version of maftools. But i am getting the output mentioning the chromosome and position only
Thanks

Thanks a lot

Thank you, it worked.
May i know how to change the colour of the rainfall plot. I am having issues with it.
Thanks

Hi,
you need to define a color scheme (a names vector) for each Transition and
Transversion classification and pass it to col argument of plotTiTv .
col = c('coral4', 'lightcyan4', 'cornflowerblue', 'lightsalmon1', '
forestgreen', 'deeppink3') names(col) = c('C>T', 'C>G', 'C>A', 'T>A', 'T>C',
'T>G')
On Tue, Sep 26, 2017 at 8:43 PM, raman91<EMAIL_ADDRESS>wrote:

Thank you, it worked.
May i know how to change the colour of the rainfall plot. I am having
issues with it.
Thanks
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/PoisonAlien/maftools/issues/65#issuecomment-332185865,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AHyS3vEnLB8lTiZVSzR4QKs0wMqZ2m90ks5smPFlgaJpZM4PjqNO
.

Thank you :)

For some of my samples, i am getting an error:
Error in multiple.mean.norm(data, mul.method = method, penalty, pen.value,  :
Minimum segment legnth is too large to include a change in this data
What does it mean?
Thanks
Regards
Raman

I am guessing you're working on Kataegis? Anyways, original method describing identification of such genomic loci used PCF method

For each sample, all mutations were ordered by chromosomal position and the intermutation distance, defined as the number of base pairs from each mutation to the next one, was calculated. Intermutation distances were then segmented using the piecewise constant fitting (PCF) method63 to find regions of constant intermutation distance. Parameters used for PCF were γ = 25 and kmin = 2 and were trained on the set of kataegis foci that had been manually identified, curated and validated using orthogonal sequencing platforms6. Putative regions of kataegis were identified as those segments containing six or more consecutive mutations with an average intermutation distance of less than or equal to 1,000 bp.

But the method being used in maftools uses changepoint detection method. I have used this method on  published results to to see if it matches and it seems to work well. However, it is bit sensitive i.e, it does include few false positives and larger segments with high inter-mutation distance. The error that you're seeing is due to this issue, it is identifying larger segments; I am still working on it. Maybe over the next few weeks it should be fixed.
Till then be-careful with the regions identified, you may have to further filter them based on inter mutation distance.

Ok, thanks a lot :)


Build error when having multiple attritbutes with colon in name in defineProps

Vue version
3.3.4
Link to minimal reproduction
https://stackblitz.com/edit/quasarframework-webpack-lgobhe?file=src%2Fcomponents%2FDTable.vue
Steps to reproduce

Go to reproduction Link
Error comes during build

What is expected?
When using an interface with multiple colons as attributes, it compiles without problems and can be used in the component.
Example:
<template>

Hello

</template>

What is actually happening?
During build an error is thrown. If there is only one prop with a colon in the attribute it doesn't throw.
Error in error handler:
Error: offset is longer than source length! offset 206 > length 184
at numberToPos (/home/projects/quasarframework-webpack-lgobhe/node_modules/vite/dist/node/chunks/dep-6e2fe41e.js:18524:15)
at formatError (/home/projects/quasarframework-webpack-lgobhe/node_modules/vite/dist/node/chunks/dep-6e2fe41e.js:39102:35)
at error (/home/projects/quasarframework-webpack-lgobhe/node_modules/vite/dist/node/chunks/dep-6e2fe41e.js:39076:19)
at transform (/home/projects/quasarframework-webpack-lgobhe/node_modules/vite/dist/node/chunks/dep-6e2fe41e.js:39320:25)
Unexpected token, expected "," (7:14)
Error in error handler:
Error: offset is longer than source length! offset 206 > length 184
at numberToPos (/home/projects/quasarframework-webpack-lgobhe/node_modules/vite/dist/node/chunks/dep-6e2fe41e.js:18524:15)
at formatError (/home/projects/quasarframework-webpack-lgobhe/node_modules/vite/dist/node/chunks/dep-6e2fe41e.js:39102:35)
at error (/home/projects/quasarframework-webpack-lgobhe/node_modules/vite/dist/node/chunks/dep-6e2fe41e.js:39076:19)
at transform (/home/projects/quasarframework-webpack-lgobhe/node_modules/vite/dist/node/chunks/dep-6e2fe41e.js:39320:25)
11:55:03 [vite] Internal server error: Unexpected token, expected "," (7:14)
Plugin: vite:vue
File: /home/projects/quasarframework-webpack-lgobhe/src/components/DTable.vue
Error in error handler:
Error: offset is longer than source length! offset 206 > length 184
at numberToPos (/home/projects/quasarframework-webpack-lgobhe/node_modules/vite/dist/node/chunks/dep-6e2fe41e.js:18524:15)
at formatError (/home/projects/quasarframework-webpack-lgobhe/node_modules/vite/dist/node/chunks/dep-6e2fe41e.js:39102:35)
at error (/home/projects/quasarframework-webpack-lgobhe/node_modules/vite/dist/node/chunks/dep-6e2fe41e.js:39076:19)
at transform (/home/projects/quasarframework-webpack-lgobhe/node_modules/vite/dist/node/chunks/dep-6e2fe41e.js:39320:25)
11:55:09 [vite] Internal server error: Unexpected token, expected "," (7:14)
Plugin: vite:vue
File: /home/projects/quasarframework-webpack-lgobhe/src/components/DTable.vue
Error in error handler:
Error: offset is longer than source length! offset 206 > length 184
at numberToPos (/home/projects/quasarframework-webpack-lgobhe/node_modules/vite/dist/node/chunks/dep-6e2fe41e.js:18524:15)
at formatError (/home/projects/quasarframework-webpack-lgobhe/node_modules/vite/dist/node/chunks/dep-6e2fe41e.js:39102:35)
at error (/home/projects/quasarframework-webpack-lgobhe/node_modules/vite/dist/node/chunks/dep-6e2fe41e.js:39076:19)
at transform (/home/projects/quasarframework-webpack-lgobhe/node_modules/vite/dist/node/chunks/dep-6e2fe41e.js:39320:25)
11:55:49 [vite] Internal server error: Unexpected token, expected "," (7:14)
Plugin: vite:vue
File: /home/projects/quasarframework-webpack-lgobhe/src/components/DTable.vue
System Info
System:
    OS: Windows 10 10.0.19045
    CPU: (8) x64 Intel(R) Core(TM) i7-6820HQ CPU @ 2.70GHz
    Memory: 12.65 GB / 31.88 GB
  Binaries:
    Node: 18.17.1 - C:\Program Files\nodejs\node.EXE
    npm: 9.3.1 - ~\AppData\Roaming\npm\npm.CMD
  Browsers:
    Edge: Spartan (44.19041.1266.0), Chromium (117.0.2045.31)
    Internet Explorer: 11.0.19041.1566
  npmPackages:
    vue: ^3.3.4 => 3.3.4

Any additional comments?
No response

Duplicate of https://github.com/vuejs/core/issues/8910


Icon library request: Circum Icons

Hi there,
I'm the maintainer of Circum Icons a library of 285 icons. You can see them at circumicons.com and you can find our repo here. Our icons are available for React, Vue and Svelte. It would be nice to have an alternative for Angular.
Let me know if I can do something.
Thank you!

Thank you for your suggestion! I'm always looking for high quality iconsets to add to the project and I hadn't come across this one before. Looks great, certainly one I will use myself in future!
I have added your iconset which should now be available to view here and the package should be available on NPM:
https://ng-icons.github.io/ng-icons/#/circum-icons
Let me know if you encounter any issues! Thanks!

Wow that was fast! Thank you for the kind words @ashley-hunter, I'm glad that you like it. Just one thing, I feel bad about asking it, but could you add our website url in your README.md like the others?

Thank you again and have a nice day!

Apologies, I forgot to do that, will update it now 😊.

Done 😊

No problem, thank you! 😃


Exclude java/lang/String/StringRepeat.java on windows32 with jdk17

Test java/lang/String/StringRepeat.java failed on window32 with jdk17, which is a  known bug https://bugs.openjdk.java.net/browse/JDK-8221400. Fixes in. However probably old 32-bit system was forgot. https://github.com/adoptium/aqa-tests/issues/1272#issuecomment-522137671
In this case we'd like to exclude this test.
https://ci.adoptopenjdk.net/job/Test_openjdk17_hs_sanity.openjdk_x86-32_windows/93/testReport/
To exclude: adding one exclusion statements in https://github.com/adoptium/aqa-tests/blob/master/openjdk/excludes/ProblemList_openjdk17.txt (it should belong to jdk_lang section, like just after Line 81 ).
java/lang/String/StringRepeat.java#id1 https://bugs.openjdk.java.net/browse/JDK-8221400 windows-x86

Hi @sophia-guo  @AdamBrousseau  Do you mind providing an exclusion statement here!


Add telemetry attribute + New Encryption mode for post-account manager broker.

related: https://github.com/AzureAD/ad-accounts-for-android/pull/2238

Will put the attributes in broker repo.


How to pass environment variables to pkg??

I have tried setting OS environment variables like below.
export NODE_PATH=/path/from/root/shared/config:/path/from/root/shared/lib:/path/from/root/config:/path/from/root/lib:/path/from/root/relay
OR
export PATH=${PATH}:/path/from/root/shared/config:/path/from/root/shared/lib:/path/from/root/config:/path/from/root/lib:/path/from/root/relay

environment variables defined in package.json (using better-npm-run):
  "env": {
	"NODE_ENV": "development",
	"NODE_PATH": "../shared/config:../shared/lib:./config:./lib:./relay",
	..
  }

I have tried below, it works if the paths inside require are relative.
  "pkg":{
    "scripts": ["config/**/*.js", "lib/**/*.js", "relay/server/**/*.js", "../shared/config/**/*.js", "../shared/lib/**/*.js"]
  }

But my application does not have relative paths in require. Because it assumes environment variables are set.

Same here...


[Request] Server Lagger / Crasher

Playin Comp and meet with 3 guy that had 5ms ping, and the server start lagging and then crashing.

It's already patched dumbo.

Patched close this

Patched

It wasn't, got patched and someone revive the feature already, not much than 2 weeks before.

and it gat patched AGAIN


Update test version

To make pipeline test the spring-boot version 2.5.10 and 2.6.4.
To make sure we support the latest version.

API changes have been detected in com.azure.spring:spring-messaging-azure. You can review API changes here

API changes have been detected in com.azure.spring:spring-messaging-azure-eventhubs. You can review API changes here

API changes have been detected in com.azure.spring:spring-messaging-azure-servicebus. You can review API changes here

API changes have been detected in com.azure.spring:spring-cloud-azure-autoconfigure. You can review API changes here

API changes have been detected in com.azure.spring:spring-cloud-azure-stream-binder-servicebus. You can review API changes here

API changes have been detected in com.azure.spring:spring-cloud-azure-stream-binder-eventhubs. You can review API changes here


author and organisation hal ID are not stored in a consistent way

In the AUTHOR_IDENTIFIER the ID is stored as the number while in the ORGANISATION_IDENTIFIER as struct-$number, is this normal?
Example:
author:
SELECT * FROM PERSON_IDENTIFIER WHERE personID = 2;

2	2	2096	halAuthorId
65962	2	52503	halAuthorId
3	2	philippe-chassaing	idHal

while organisation is:
SELECT * FROM ORGANISATION_IDENTIFIER WHERE organisationID = 6;

6	6	struct-300009	halId

This is correct, as in HAL the ids have different structure for different entity type (author, organisation, etc..)


Failed to connect to packagist.org port 80

This exception is thrown sometimes. (not always)
But the problem is immediately gone after remove prestissimo.  I couldn't figure out the reason or condition.

Same problem here on Linux Mint 17 (64 bits) and php 5.6.
But on my Macbook it works fine

Please let me know again when reproduced.


Modal component and service

Here is the PR focused on the modal component and service. I believe that it is ready for final review. It might continue to evolve but I think it's at a state that is robust and useable.
Once merged I would like to use it to finish up the work on the Demo PR.

Regarding Modal VS Dialog comparison. I can point to two things that are different (and might stay different).

Dialog has some variants in how we can call the open() method.

Depending on how it is called it will return and handle the returned promise a little differently. I am not a big user of Aurelia Dialog so I don't know how important these variants are. In my opinion I prefer when a service like this has one consistent API. Therefore I would rather stick with a single open() method that works in the same manner all the time.

The Dialog attaches the whenClosed() method to the returned Promise<> from the open() api. This comes handy as it requires one less then() or await but I prefer the way the modalService return a standard Promise pointing to the UxModal ViewModel. When consumed, the open() api requires one more then() or await but then the returned value can do more than just whenClosed() => it gives full access to the ViewModel.

Both of these could lead to breaking changes when switching from Aurelia Dialog to Aurelia Ux Modal but they should not remove any important functionality that I am aware of.

@ben-girardet looks good to me. I have no further comments for now. Let's go first and continue enhancing this.


Where i can find the private key?

Hello,
where i can find the private key of Bitcoin address or any other coin ?
thanks

There is no built in way to do this in HyperDEX, but the seed is fully compatible with other Komodo tools like marketmaker, iguana, Agama ETC.
You can import your HyperDEX seed into any of those existing programs to derive your private keys.


Fix test_benchmark build

It was failing on Debian 12 due to missing  include:
kirr@deca:~/src/tools/go/pygolang-master/3rdparty/ratas/src/test$ g++ test_benchmark.cc
test_benchmark.cc: In function ‘int main(int, char**)’:
test_benchmark.cc:223:21: error: variable ‘std::string value’ has initializer but incomplete type
  223 |         std::string value = s;
      |                     ^~~~~
test_benchmark.cc:234:21: error: variable ‘std::string value’ has initializer but incomplete type
  234 |         std::string value = s;
      |                     ^~~~~

Thanks!

Thanks for merging.


DOC/Demo App -  webcredentials not needed

The README states that webcredentials entitelment must be added in order to use UniversalLinks - I don't think this is the case? Its part of the demo apps too, but this is a distinct feature of domain association to my understanding

The webcredentials entitlement ensures, that things like logins/password can be shared amongst apps/web-sites via the use of the iCloud-keychain. So, technically universal links will work without this feature but would force the user to reenter his credentials.
Do you want us to remove this?

The webcredentials entitlement ensures, that things like logins/password can be shared amongst apps/web-sites via the use of the iCloud-keychain. So, technically universal links will work without this feature but would force the user to reenter his credentials.

As far as I understand webcredentials its about username/password shared between an app and the website of the same developer, I don't see how that has any relation to the Demo Application or generally the functionality of the SDK (The Doc currently states that the entitlement is necessary I think).
The user never enters a username/password on the redirect domain (e.g. eunid.github.io in our demo case, for which the entitlement would be valid) nor in the Demo App - the SDK is all about not having to do that as an app that logs in a users using one of the login flows for example. The Demo App also does not try to use this API (iOS side)
If I'm missing something here let me know.

You must add your domain for webcredentials as well as for applinks to make it work both in app2web and app2app scenarios.

In relation to doc - I don't think anything would change if the webcredentials entitelment is removed in the Demo App

You are right, it is not necessary, we updated the project settings and documentation accordingly.


property not explained in pipeline-templates documentation

Page: /guides/user/pipeline/pipeline-templates/instantiate/
Feedback:
In the Add new stages section, what is the config field? It's not mentioned anywhere.  The JSON config for a Wait stage looks like this:
    {
      "name": "Wait",
      "refId": "3",
      "requisiteStageRefIds": [
        "1"
      ],
      "restrictExecutionDuringTimeWindow": false,
      "sendNotifications": true,
      "stageEnabled": {
        "type": "expression"
      },
      "type": "wait",
      "waitTime": 30
    }

Check out the V2StageDefinition used by pipeline templates. The config houses the configuration that is specific to the wait stage.

Thanks for pointing out, but as a suggestion, I would say that I don't think every user will drill down the code looking for answers, specially those who are not familiar with Java (like me).


MAINT: LinearVectorFunction.J is ndarray closes #11886

@mdhaber.
Makes sure LinearVectorFunction.J is an np.ndarray. Full explanation is in #11886.

Thanks. Could we add a test to https://github.com/scipy/scipy/blob/master/scipy/optimize/tests/test_minimize_constrained.py:
def test_bug_11886:
    def opt(x):
        return x[0]**2+x[1]**2
    A = np.matrix(np.diag([1,1]))
    lin_cons = scipy.optimize.LinearConstraint(A, -1, np.inf)
    res = scipy.optimize.minimize(opt, 2*[1], constraints = lin_cons)
    assert_array_equal(res.x, [0, 0])

I tried adding a test, but then a PendingDeprecationWarning is emitted (which turns to an error with the test suite) when creating the matrix. I added your test though.

When CI finishes, I'll accept my suggestion (just a comment explaining why the test doesn't appear to test anything) and merge.

test_heequb failure is not from this PR. Thanks @andyfaff.


Make setting up the app possible without a file explorer installed

Describe your suggested feature
I was trying to set this up in my Jailbroken Tolino Reader but since Tachi 0.15. it apparently requires you to set a folder for downloads before being able to use the app. This is impossible in the heavily modified Android of these devices.
Please make it possible to get the app usable without this step. Make a default folder an option or something in that direction
Other details
No response
Acknowledgements

[X] I have searched the existing issues and this is a new ticket, NOT a duplicate or related to another open or closed issue.
[X] I have written a short but informative title.
[X] I have updated the app to version 0.16.3.
[X] I will fill out all of the requested information in this form.

Not sure if my issue is the same as this.
I just tried to install it on my Realme Stick (running Google TV), and I couldn't get past the setup.
Tried installing other file explorers, but none were detected to work for the setup. Looked it up and saw lots of people on XDA having the same problem.
https://github.com/mihonapp/mihon/assets/159751719/83b66fd2-afd3-446c-abd6-f592ccb516fd

To support newer versions of Android we now require the User to give us permission to write to a folder instead of having access to every single file and folder. So selecting a folder is crucial.
Also file picker is a core Android component. If it's missing from your build then you should complain to the devs of that build.


Use "type" of MessageEvent for SSE Events instead of "lastEventId"

As discussed in Twitter DMs:
blitz_api sends the messages with an id which is then used in a MessageEvent as lastEventId(MDN documentation here.
In raspiblitz-web I'm differentiating between the type of the MessageEventthough, which not lets the web UI receive the messages, but it can't process them.
Changing the  lines here works, but needs to be done for the "regular" events as well.

Fixed with #38


Downgrade default scope for better security

User complains Drone CI asks for read and write permission which is too risky.

As a compare, Travis CI asks for lower permission:

diff --git a/remote/github/github.go b/remote/github/github.go
index d7c793b..99e1fdd 100644
--- a/remote/github/github.go
+++ b/remote/github/github.go
@@ -21,7 +21,7 @@ import (
 const (
        DefaultURL   = "https://github.com"
        DefaultAPI   = "https://api.github.com"
-       DefaultScope = "repo,repo:status,user:email"
+       DefaultScope = "read:org,user:email,repo_deployment,repo:status,write:repo_hook"
 )

The above proof of concept show that lower permission should be enough to trigger a Drone CI build, like what Travis CI does.
However, when activate a repo using such a permission, Drone CI complains Unable to activate this repository. You must have administrative access to this repository. Also, please check the third-party application access policy to ensure Drone is approved.
Besides the complaining, the build itself works fine. (But the user frone page fail to list activated repo)

this has been addressed in other issues if you search the issue tracker https://github.com/drone/drone/issues/309#issuecomment-176271104

@bradrydzewski Thank you for the comment, after read #309 I compare Drone CI with travis-ci.com as well as travis-ci.org, and I found travis-ci.com requires same permission like drone.io, since travis-ci.com needs access to private github repos.
For me it seems https://github.com/drone/drone/issues/91 is the best solution which fits everyone's need.

When I tried to sign up using my Github account, drone.io not only wanted full access to my repositories, it wanted full access to my personal information.
The thought that pops into my head is, what responsible developer would allow a third party to write their personal information, and to the code repositories of their employer?
And since responsible developers are obviously the target audience, I must be missing something. What am I missing?

@sheldonh this is for open source drone and is not for drone.io which are two different codebases. The drone.io codebase is legacy and is no longer being maintained, with all effort being focused on this open source codebase.

When I tried to sign up using my Github account, drone.io not only wanted full access to my repositories, it wanted full access to my personal information.

Drone originally needed write access to the account to upload ssh keys to clone private repositories. There are also repository-level deploy keys that we could have used, but these keys are limited to only access a single repository, which doesn't work well for most projects that need to fetch private dependencies during the build (via npm install, go get, git submodules, etc)
The open source version of Drone changed the approach to use the oauth token to clone repositories, which was a new feature added by GitHub a while back. This removed the need to upload an ssh key. But as I said, the drone.io website is a legacy codebase and this change is not being backported.

what responsible developer would allow a third party to write their personal information, and to the code repositories of their employer?

GitHub does not provide a read only scope for private repositories. Re-read that sentence and let it sink in. The only scope available is repo which provides full write access to private repositories. If GitHub provides a repo:read scope we would gladly use it, but it isn't an option. Perhaps you should direct your concern to GitHub.

And since responsible developers are obviously the target audience, I must be missing something. What am I missing?

I think you are missing a clear understanding of GitHub scopes and what 3rd party solutions have available to work with. Travis CI and Circle CI both have to request write access to your public and private repositories.  Do you think we haven't considered these things? We work with what GitHub gives us.