Vault Search

Search the entire vault Titles / Contents.
Related to issue #120 and #93
Using Minisearch.js
Only the index creation is ready. Will create a .json from all the html files stored in the export directory (potential flaw/feature) and store it in the export directory.
I will focus on fixing the backend for now.
Frontend support should be as easy as adding  filed where you want it and style it how you want it and I would rather leave it to you since you seemed to have an idea on how to style it. I will be working on the search.js for the frontend too.
This pull is extremely dirty. The search.ts has dirty module importing and I forgot to add the dependencies in the requirements, I would appreciate if you could look into it

Awesome, if in the code you could "brand" this as a general indexing rather than creating an index only for search, I think this data could be used for other purposes as well in the future, and creating to be extensible is favorable.
Also keep in mind that obsidian already has a very versatile index of all the files and metadata that you can likely use instead of a 3rd party library.
Thank you for taking the time to tackle this issue!

Awesome, if in the code you could "brand" this as a general indexing rather than creating an index only for search, I think this data could be used for other purposes as well in the future, and creating to be extensible is favorable.
Also keep in mind that obsidian already has a very versatile index of all the files and metadata that you can likely use instead of a 3rd party library.
Thank you for taking the time to tackle this

Awesome, if in the code you could "brand" this as a general indexing rather than creating an index only for search, I think this data could be used for other purposes as well in the future, and creating to be extensible is favorable.
Also keep in mind that obsidian already has a very versatile index of all the files and metadata that you can likely use instead of a 3rd party library.
Thank you for taking the time to tackle this issue!

Sure, I will edit the branding.
AFAIK the new index generation is required because of people choosing only certain documents to export and not the whole vault, but if you know a faster and reliable way of indexing and generating a json with Title and content feel free to step in and I will focus only on the frontend searching implementation!

This should be the last commit for the index generator.
Also committed the updated package and package lock.
The index now has relative path, and doesn't encounter word merging bugs with li and ul tags in lists (need experimentation because there might be more tags that cause this issue).
The index generation is now extremely faster and generates a json that is 4 times smaller than in the first version.
Next step is implementing a search frontend script.

Obsidian's index can be filtered im pretty sure to only include certain files. But id have to look to see if it can do what we would want.

Obsidian's index can be filtered im pretty sure to only include certain files. But id have to look to see if it can do what we would want.

I think that after using the obsidian index we would still need to go and parse it to make it supported by the search frontend script so going this way might be faster

What format will this json file be, is the goal just to have a dictionary that associates the file name with the text of the document?

What format will this json file be, is the goal just to have a dictionary that associates the file name with the text of the document?
searchIndex.json

This is my searchindex generated from approximately 350 documents

It should be possible to create the index during generation rather than trying to create an index retroactively. Because during generation we have a list of all the files being exported as well as their paths, content, names, etc

Ahhhh okay the file you sent me makes a lot more sense

It should be possible to create the index during generation rather than trying to create an index retroactively. Because during generation we have a list of all the files being exported as well as their paths, content, names, etc

Yeah but it would make it even more resource hungry then. I'm on a 32gb ram machine so I wouldn't feel it but users with lower amounts of ram could be impacted by this elaboration. Just try locally and see how fast it is just to generate the index after the html is done. Of course you can start generating the index from the md files but you would just have to start it as a parallel process as I don't think you can use data you already have during generation unless you do some HEAVY integration between the index generation and the html generation :')

This all makes sense and I am learning a lot about indexing. One last thing is why use the cheerio library instead of just using the html parser, strip out unnecessary parts and grabbing the textContent property. Then we avoid a dependency. Maybe there is something about cheerio though that is a better solution?

This all makes sense and I am learning a lot about indexing. One last thing is why use the cheerio library instead of just using the html parser, strip out unnecessary parts and grabbing the textContent property. Then we avoid a dependency. Maybe there is something about cheerio though that is a better solution?

Do you mean htmlparser2? That would be a dependency anyways

No just built in to JavaScript

Like create a DOM out of the html

No just built in to JavaScript / Typescript

Oh you mean DOMParser, well I fear that it doesn't have the { xmlMode: true } option that magically saved me while I was trying to handle the ul and li items not being spaced ahahha

But why does the html spacing matter if we are just grabbing text content?

But why does the html spacing matter if we are just grabbing text content?

Because somehow an html like WNT

Slow Growth etc etc was getting parsed as WNTSlow Growth etc etc and that obviously created a wrong word (WNTSlow) in the index

Ohh, yeah that's not good, I guess the dependency is fine to fix that then.

Ohh, yeah that's not good, I guess the dependency is fine to fix that then.

With this last commit I improved performance even more (There was an oversight while parsing ul and li)
Added support for tags. Unfortunately they can't be searched with #tagname but will have to be searched by tag~tagname.
Simply edit the href of the tags to point to '?q= tag~tagname' to implement tag search functionality

Awesome, I really appreciate the work on this

The client search script is now ready, too.
All that is left to do now is just add something like  in the html wherever you want it and style it how you want it.
The results will appear under the searchbar as soon as any word is written (you will also need to style the container)
I commented additional styling so you can decide if you wanna leave it in the js or move it to the css.
Let me know if you need help/troubleshooting with this pr, I will now be moving to the sticker as a logo feature.
If I can tackle the sticker as a logo feature fast enough I will also try to style and put the searchbar to my personal taste :')

Thank you so much, you're amazing :)
I will do a full look over this today or tomorrow

I saw you added the searchbar setting. That's a nice addition, but then you should also disable the index making. Makes no sense to create an index if it's not gonna be displayed anyways hahah

Ah true. I also need to implement search for offline use. Right now the index is being saved in the vault directory, but if it's being used in a self contained file it needs to be embeded into the file itself. This can be done using the new asset system in version 1.7.4 which isn't in the master branch yet, so I will hold off on that until it's all merged.
Secondly, all of the links actually reload the page instead of loading the document. The events that intercept the normal link behavior need to be set up on the new links. There is a function for this.
Finally, it would probably be good if the links actually directed to the location in the document that that search was found. I don't think this is stored in the index at the moment though.

Ah true. I also need to implement search for offline use. Right now the index is being saved in the vault directory, but if it's being used in a self contained file it needs to be embeded into the file itself. This can be done using the new asset system in version 1.7.4 which isn't in the master branch yet, so I will hold off on that until it's all merged.
Secondly, all of the links actually reload the page instead of loading the document. The events that intercept the normal link behavior need to be set up on the new links. There is a function for this.
Finally, it would probably be good if the links actually directed to the location in the document that that search was found. I don't think this is stored in the index at the moment though.

I think it's pretty simple to redirect people to the first appearance of the search term and highlight all the appearances of such word. If that's what you mean, I might try to implement it

Ah yes, right. I am going to try and merge everything in 1.7.4 currently into master, and then work can continue with all features together

@7Mik Master has been merged into the 1.7.4 branch, if you'd like to continue work do so by forking the 1.7.4 branch

@7Mik Master has been merged into the 1.7.4 branch, if you'd like to continue work do so by forking the 1.7.4 branch

Sure thing! Would it be possible to prepare a project table or a list of things to do? I would love to help!

These are the main bug fixes I am looking at right now:

For some reason after the merge it thinks that I can't construct a Path object which cases the plugin to not load.

I can look at the index disabling and the tag redirecting since it is pretty much adding to a code I already wrote. Will also look into highlighting and redirecting as you proposed.
For all the other issues, maybe is better if you open a new dedicated issue thread for all of them so we can coordinate better: this thread is becoming quite a mess hahah

Before we can continue we have to fix the Path object problem because it makes it so the plugin doesn't even load to start with. I will create new issues for things as I come to them, and you can do the same.

Before we can continue we have to fix the Path object problem because it makes it so the plugin doesn't even load to start with. I will create new issues for things as I come to them, and you can do the same.

Did you already create the issue thread for this path object problem? I will now build from the new 1.7.4 source and see what I can do

@7Mik Just did: #313


Type Mismatch For using OpenAI Interface

System Info
I use OpenAIEmbeddings from langchain.embedding,  and using openAI function.
However I find it's a problem that when we call OpenAI Interface, the input is a 2D list, not a 1D list.
If the input is a 1D list, it could work for embedding.create func
like this:

and I found out that openai api does not support 2D list as input.

how could I solve this problem

the error is shown below:
Traceback (most recent call last):
  File "..\main.py", line 47, in <module>
    langchain_openai()
  File "..\main.py", line 42, in langchain_openai
    query_result = embeddings.embed_query(text)
  File "C:\ProgramData\Anaconda3\envs\python3.8\lib\site-packages\langchain\embeddings\openai.py", line 501, in embed_query
    return self.embed_documents([text])[0]
  File "C:\ProgramData\Anaconda3\envs\python3.8\lib\site-packages\langchain\embeddings\openai.py", line 473, in embed_documents
    return self._get_len_safe_embeddings(texts, engine=self.deployment)
  File "C:\ProgramData\Anaconda3\envs\python3.8\lib\site-packages\langchain\embeddings\openai.py", line 359, in _get_len_safe_embeddings
    response = embed_with_retry(
  File "C:\ProgramData\Anaconda3\envs\python3.8\lib\site-packages\langchain\embeddings\openai.py", line 108, in embed_with_retry
    return _embed_with_retry(**kwargs)
  File "C:\ProgramData\Anaconda3\envs\python3.8\lib\site-packages\tenacity\__init__.py", line 289, in wrapped_f
    return self(f, *args, **kw)
  File "C:\ProgramData\Anaconda3\envs\python3.8\lib\site-packages\tenacity\__init__.py", line 379, in __call__
    do = self.iter(retry_state=retry_state)
  File "C:\ProgramData\Anaconda3\envs\python3.8\lib\site-packages\tenacity\__init__.py", line 314, in iter
    return fut.result()
  File "C:\ProgramData\Anaconda3\envs\python3.8\lib\concurrent\futures\_base.py", line 437, in result
    return self.__get_result()
  File "C:\ProgramData\Anaconda3\envs\python3.8\lib\concurrent\futures\_base.py", line 389, in __get_result
    raise self._exception
  File "C:\ProgramData\Anaconda3\envs\python3.8\lib\site-packages\tenacity\__init__.py", line 382, in __call__
    result = fn(*args, **kwargs)
  File "C:\ProgramData\Anaconda3\envs\python3.8\lib\site-packages\langchain\embeddings\openai.py", line 105, in _embed_with_retry
    response = embeddings.client.create(**kwargs)
  File "C:\ProgramData\Anaconda3\envs\python3.8\lib\site-packages\openai\api_resources\embedding.py", line 33, in create
    response = super().create(*args, **kwargs)
  File "C:\ProgramData\Anaconda3\envs\python3.8\lib\site-packages\openai\api_resources\abstract\engine_api_resource.py", line 153, in create
    response, _, api_key = requestor.request(
  File "C:\ProgramData\Anaconda3\envs\python3.8\lib\site-packages\openai\api_requestor.py", line 298, in request
    resp, got_stream = self._interpret_response(result, stream)
  File "C:\ProgramData\Anaconda3\envs\python3.8\lib\site-packages\openai\api_requestor.py", line 700, in _interpret_response
    self._interpret_response_line(
  File "C:\ProgramData\Anaconda3\envs\python3.8\lib\site-packages\openai\api_requestor.py", line 763, in _interpret_response_line
    raise self.handle_error_response(
openai.error.InvalidRequestError: parse parameter error: type mismatch

Who can help?
@hwchase17 @agola11
Information

[x] The official example notebooks/scripts
[ ] My own modified scripts

Related Components

[ ] LLMs/Chat Models
[X] Embedding Models
[ ] Prompts / Prompt Templates / Prompt Selectors
[ ] Output Parsers
[ ] Document Loaders
[ ] Vector Stores / Retrievers
[ ] Memory
[ ] Agents / Agent Executors
[ ] Tools / Toolkits
[ ] Chains
[ ] Callbacks/Tracing
[ ] Async

Reproduction

def langchain_openai():
    from langchain.embeddings import OpenAIEmbeddings

    embeddings = OpenAIEmbeddings(
        model="text-embedding-ada-002",
        openai_api_base="",
        openai_api_key=""
    )
    text = "This is a test query."
    query_result = embeddings.embed_query(text)
    print(query_result[:5])

Expected behavior
I think it's a bug or something to fix

I try your way, however it's still type mismatch.

If I use offical openai interface.if input is a array, it could work.

if input is a 2D array, it will type mismatch.

However, your input to _embed_with_retry is 2D like pic bleow

Answer generated by a 🤖
Answer
Hello,
Thank you for reaching out and providing detailed information about the issue you're facing.
From your description, it seems like you're trying to use the embed_query method with a single string input, which should work as expected. The embed_query method is designed to handle a single string and internally wraps it into a list before passing it to the embed_documents method. The embed_documents method is designed to handle a list of strings.
Here's the correct way to use these methods:
from langchain.embeddings import OpenAIEmbeddings

# Initialize the OpenAIEmbeddings object
embeddings = OpenAIEmbeddings(
    model="text-embedding-ada-002",
    openai_api_base="",
    openai_api_key=""
)

# For a single string input
text = "This is a test query."
query_result = embeddings.embed_query(text)
print(query_result[:5])

If you're still encountering the same error, it might be due to an issue with the OpenAI API or the specific model you're using. Please ensure that your OpenAI API key is correct and that the model "text-embedding-ada-002" is available and supports the operations you're trying to perform.
If the issue persists, could you please provide more information about the version of LangChain you're using and the exact steps to reproduce the error? This will help us better understand the issue and provide a more accurate solution.
Best regards, Dosu
Sources

libs/langchain/langchain/embeddings/openai.py

This response is meant to be useful, save you time, and share context. It is not meant to be a precise solution, but rather a starting point for your own research.
Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

I try your way, however it's still type mismatch.

If I use offical openai interface.if input is a array, it could work.

if input is a 2D array, it will type mismatch.

However, your input to _embed_with_retry is 2D like pic bleow

Answer generated by a 🤖
Answer
Hello,
Thank you for reaching out and providing a detailed description of the issue you're facing. Based on your description, it seems like the embed_query method from the OpenAIEmbeddings class in the LangChain Python framework is expecting a string as input, but the _embed_with_retry method might be receiving a 2D array instead, which is causing the type mismatch error.
The embed_query method in the OpenAIEmbeddings class has the following signature:
def embed_query(self, text: str) -> List[float]:

This method expects a single argument text which should be of type str. This argument represents the text to be embedded. The method returns a list of floating point numbers (List[float]), which represents the embedding for the input text.
However, the _embed_with_retry method, which is used internally by the embed_query method, expects a dictionary as input. This dictionary should contain the arguments to be passed to the embeddings.client.create method.
If you're calling the embed_query method with a single string as input and still encountering the error, it's possible that there's an issue with how the LangChain framework is handling the input internally. Specifically, it might be converting the input into a 2D array before passing it to the _embed_with_retry method, which is causing the type mismatch error.
Unfortunately, I can't see the screenshots you've provided, but based on your description, it seems like the OpenAI interface works fine when the input is a 1D array but throws a type mismatch error when the input is a 2D array. This further suggests that the issue might be with how the LangChain framework is handling the input internally.
To resolve this issue, I would recommend checking the internal implementation of the embed_query method and the _embed_with_retry method in the LangChain framework to ensure that they're handling the input correctly. If you find that the input is being incorrectly converted into a 2D array, you might need to adjust the code to ensure that a dictionary is passed to the _embed_with_retry method instead.
If you're comfortable with it, you could also consider submitting a pull request to the LangChain repository with the necessary changes. This would not only resolve the issue for you, but also for other users who might be facing the same problem.
I hope this helps! If you have any further questions or need more clarification, please don't hesitate to ask.
Best, Dosu
Sources

libs/langchain/langchain/embeddings/openai.py

This response is meant to be useful, save you time, and share context. It is not meant to be a precise solution, but rather a starting point for your own research.
Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

Could you ask some real person?


Proposal: Igniter.CLI for expanded interactivity support

Original discussion on Discord
Prerequisite: #126
This proposal is for expanding Igniter's CLI support to offer better integration between command line arguments and interactive user prompts.
As an example, consider a version of mix phx.new written using Igniter. It supports many options to configure a new project, including things like --no-ecto, --database postgres, etc. When implemented, this proposal would allow for an interactive series of prompts that would guide users through those options, using command line arguments to skip prompts when appropriate.
$ mix igniter.phx.new my_app

[Database] Set up an Ecto Repo?

[y/N] (default y)> y

[Database] Which database will you use?

1. Postgres
2. MySQL
3. MSSQL
4. SQLite

(default 1): 1

...

With the --no-ecto flag, specified, those prompts would be skipped:
$ mix igniter.phx.new --no-ecto my_app

[Database] Set up an Ecto Repo? NO (--no-ecto)

...

And to skip all prompts, the Igniter-global --yes could be used:
$ mix igniter.phx.new --yes

[Database] Set up an Ecto Repo? YES (--yes)
[Database] Which database will you use? Postgres (--yes)
...

To support this, a new Igniter.CLI module will be created. The specific API is a work in progress, but it may look something like this:
@impl Igniter.Mix.Task
def info(_argv, _composing_task) do
  %Igniter.Mix.Task.Info{
    schema: [
      no_ecto: :boolean,
      database: :string
    ],
    ...
  }
end

@impl Igniter.Mix.Task
def igniter(igniter) do
  igniter
  |> Igniter.CLI.yes_no_prompt(:no_ecto, "[Database] Set up an Ecto Repo?",
    # specify that the prompt is asking the inverse
    # i.e. "y" means no_ecto: false and "n" means no_ecto: true
    inverse: true
  )
  |> Igniter.CLI.select_prompt(:database, "[Database] Which database will you use?",
    # only issue this prompt if the function returns truthy
    if: &(not &1.args.options.no_ecto),
    options: [
      {"postgres", "Postgres"},
      {"mysql", "MySQL"},
      {"mssql", "MSSQL"},
      {"sqlite", "SQLite"}
    ]
  )
  |> generate_ecto()
end

defp generate_ecto(igniter) do
  if igniter.args.options.no_ecto do
    igniter
  else
    database = igniter.args.options.database
    ...
  end
end

Looks good! One tiny note: the way you do no- options is by doing ecto: :boolean and OptionParser figures out that --no-ecto means --ecto false IIRC.

Oh! Awesome. I'll update the example to reflect that.


[Question]: The number of packets after feature extraction is is less than the number of original packets

What's up?
Hi,
I installed nprint in my MacOS and used it to extract some pcap files including 10000 packets. But the number of packets is less than 10000. It is 9995 or 9990. I don't know which packets were dropped.
Due diligence

[X] I have read the docs

There are a large number of reasons this could have happened, and the information above gives no insight into how to fix the problem.


Setup validations for forms with Pydantic

We should use pydantic to create models for our form structure.
This issue should purely implement the models, there is no reason to add any routes for this yet.
Where possible we should perform as much validation as possible to ease stray data entering the database.
We should sanitise all fields that are not in the SCHEMA.md file in the root of the repository to avoid having MongoDB control characters entering the fray.

Resolved by #11 . Closing.


Questions about the templates

Hello!
I have some questions regarding the example provided here:

Why do we need a PVC, and how can we size it regarding our usage, what is stored in it ?
Can the agent and the proxy run with multiple replicas?
You specify, Recreate as strategy in the Deployment, what the limitation about not using strategy?

Thank you!

Hey,

Why do we need a PVC, and how can we size it regarding our usage, what is stored in it ?

It is used to store a local cache of the rules from our Saas, which will be used for fast / fresh start (before the first http request) and in case of failure from our platform so you always have the last ruleset available even if we are down.
But it's not mandatory, agent can work fine without it, it's just recommended for better availability

Can the agent and the proxy run with multiple replicas?

Yes they can, proxy request to the agent are stateless, so you can have a tcp loadbalancer strategy between them, there is no limit to the number of agent or proxy

You specify, Recreate as strategy in the Deployment, what the limitation about not using strategy?

None, this is an error, we should keep the default here (rolling update) (all strategies should work)

Thank you for the fast answer!

Hey,
What would you use for the livenessProbe and readinessProbe ?
Plus, it seems that when I deploy a new pod of the agent, I need to rollout restart the proxy, is that a normal behaviour regarding the Nginx?
Thank you

Hello,

What would you use for the livenessProbe and readinessProbe ?

We do not provide any endpoint for those probe ATM, but agent should be ready once it's port is available, and you can check that the port is available for liveness also, all request are stateless, so even if one failed it should not impact other request

Plus, it seems that when I deploy a new pod of the agent, I need to rollout restart the proxy, is that a normal behaviour regarding the Nginx?

Nginx will use persistent connections between him and the agent with a pool of connections in order to avoid a tcp connect workflow for each request, it remove them / invalidate them if there was a connection error (or if there is more connections than authorized given the pool configuration)
So you have a direct link between nginx and the agent there may be bad connection keeped when deploying a new pod (since old connections are keep)
I'm not sure how does the kubernetes service handle this problem, if it keep the link or does a hot update it, can you try to use a StatefulSet instead of a Deployment for the agent maybe it will be better ?
To be honest we are definitely not expert on kubernetes but we are slowly gaining knowledge on it  as we want to update some of our internal service to k8s; There will be update to the agent in the next month to have a better integration (readiness / liveness / prometheus metrics / etc ....) but we have no offical ETA for those features.


Incorrect typing of ConfigureConnectorParams

🐛 Current behavior
This might be a duplicate of https://github.com/algolia/instantsearch/issues/6297 or https://github.com/algolia/instantsearch/issues/5989
I have recently updated to the latest version:
"vue-instantsearch": "^4.19.2",
"algoliasearch": "^4.24.0",
And I am now getting a TS error for the ConfigureConnectorParams:
Object literal may only specify known properties, and 'filters' does not exist in type 'PlainSearchParameters'.
(Same issue for 'insideBoundingBox' and 'page')
I am using the connectStateResult connector and accessing the configure param this way:
instantsearchState.value.instantSearchInstance.renderState[indexName]?.configure?.refine({
 filters: ""
})

I have tracked down the type of configure here: https://github.com/algolia/instantsearch/blob/62b6e4bbd27743cd0008548d7d59bd7cdb680b82/packages/instantsearch.js/src/connectors/configure/connectConfigure.ts#L22
It appears to be set to PlainSearchParameters whereas to me it should be SearchParameters no ? The code works and follows what is in the documentation, but the type does not reflect it.
🔍 Steps to reproduce
https://github.com/algolia/instantsearch/blob/62b6e4bbd27743cd0008548d7d59bd7cdb680b82/packages/instantsearch.js/src/connectors/configure/connectConfigure.ts#L22
Live reproduction
//
💭 Expected behavior
Type should be SearchParameters instead of PlainSearchParameters ?
Package version
"vue-instantsearch": "^4.19.2"
Operating system
No response
Browser
No response
Code of Conduct

[X] I agree to follow this project's Code of Conduct

Related to this I am also getting an issue with algoliasearch-helper typing when using setQueryParameter()
Argument of type '"maxValuesPerFacet"' is not assignable to parameter of type 'keyof PlainSearchParameters'.
Yet according the documentation I should be able to use 'maxValuesPerFacet' and the code works.
The type definition is here https://github.com/algolia/instantsearch/blob/62b6e4bbd27743cd0008548d7d59bd7cdb680b82/packages/algoliasearch-helper/index.d.ts#L654

SearchParameters is the class, PlainSearchParameters is the interface, so the type is correct, but it seems somewhere (similar to the issues you mention) PlainSearchParameters (probably SearchOptions) becomes any/unknown. Do you have a reproduction somewhere on codesandbox or a repo that we can see and debug? Also ensure you're definitely on the latest version of specifically algoliasearch-helper, which is the relevant package here.

this is fixed in the latest version, thanks for noticing


Add a Fourth vulnscan Instance to the Production Environment

🗣 Description
This PR adds a fourth vulnscan instance to the CyHy environment.

💭 Motivation and Context
@dav3r mentioned wanting to add another Nessus scanner (vulnscan instance) to the environment. This will allow us to handle increased load, and look at lightening the per-instance load. We would need to update https://github.com/cisagov/cyhy_amis/blob/1dd200561bf7cdf7e083ed483cfa86a420557860/ansible/roles/cyhy_commander/files/commander.conf#L19 to adjust the load per instance if we wanted to go down that route. We could also look at the cyhy-commander configuration for IPs per VULNSCAN.

🧪 Testing
No testing at this point, but since this is just an instance addition it should be a clean deployment using our existing tooling.

✅ Checklist

[x] This PR has an informative and human-readable title.
[x] Changes are limited to a single goal - eschew scope creep!
[x] All relevant type-of-change labels have been added.
[x] I have read the CONTRIBUTING document.
[x] These code changes follow cisagov code standards.
[x] All new and existing tests pass.

This will be merged next week before deploying the new instance. I have added a blocked label until that time.

This will be merged next week before deploying the new instance. I have added a blocked label until that time.


Unset PYTHONNOUSERSITE in environment before calling batch script

Checklist
GitHub

[X] I've given this PR a concise, self-descriptive, and meaningful title
[X] I've linked relevant issues in the PR body
[X] I've applied the relevant labels to this PR
[X] I've applied a release milestone (major, minor, patch) in line with Semantic Versioning guidelines
[X] I've assigned a reviewer

PR contents

[ ] I've consulted SCT's internal developer documentation to ensure my contribution is in line with any relevant design decisions
[ ] I've added relevant tests for my contribution
[ ] I've updated the relevant documentation for my changes, including argparse descriptions, docstrings, and ReadTheDocs tutorial pages

Description

This PR adds step 3:

sct_run_batch is called.
PYTHONNOUSERSITE is set to True in compat/launcher.py for the context of sct_run_batch.
PYTHONNOUSERSITE is unset for the context of the pipeline shell script.
(Optional) A pipeline script calls sct_<function>.
(Optional) PYTHONNOUSERSITE is set to True in compat/launcher.py, but only in the context of sct_<function>.

Step 3 is needed because PYTHONNOUSERSITE=True would leak into sct_run_batch pipeline shell scripts and affect non-SCT commands. Specifically, the command gradunwarp couldn't access its dependencies if they were installed in the user's site packages directory.
Linked issues

Fixes #3200.

My understanding:

We added PYTHONNOUSERSITE=True in https://github.com/neuropoly/spinalcordtoolbox/issues/3067, because conda's venv was not properly  isolated from the user's site-packages (which still baffles me. I tried printing sys.path on a fresh native python3 venv and it did not include the user site-packages directory whereas a fresh conda venv does :confused: )
This causes an issue with https://github.com/neuropoly/spinalcordtoolbox/issues/3200, because we want to use an external dependency with sct_run_batch, so we temporarily unset the variable just inside this script.

If gradient_unwarp is meant to be used with sct then why is it not installed within sct's virtual environment? Installing it outside and then modifying the environment to look in the user site-packages feels a bit hacky, but I am also worried that it may cause another dependency conflict down the line considering that we don't know what kind of scripts users may feed sct_run_batch.
@joshuacwnewton what do you think? Doesn't it make more sense to install dependencies within the virtual environment instead of doing this?

If gradient_unwarp is meant to be used with sct then why is it not installed within sct's virtual environment?

Right now, I don't think this is possible, because gradunwarp doesn't use setuptools. :slightly_frowning_face:
More details here: https://github.com/sct-pipeline/ukbiobank-spinalcord-csa/issues/31

Right now, I don't think this is possible, because gradunwarp doesn't use setuptools.

Perhaps, but that's their problem (unfortunately).
I guess if we really care we can patch it for them? I just want to make sure that we solve the right problem.

Perhaps, but that's their problem (unfortunately).

That's fair!

I guess if we really care we can patch it for them? I just want to make sure that we solve the right problem.

I appreciate that. Let me try answering from a different angle.

If gradient_unwarp is meant to be used with sct then why is it not installed within sct's virtual environment?

I'm not sure gradient_unwarp is an SCT dependency. I see it more as an external program. It can be used alongside SCT, but it's not necessary for SCT, so I think it should kept separate from SCT's installation environment.

and then modifying the environment to look in the user site-packages feels a bit hacky,

Well, to be fair, setting PYTHONNOUSERSITE=True is already kind of a a hacky change. :sweat_smile:
@kousu was the person who first convinced me PYTHONNOUSERSITE=True was not a good solution in https://github.com/sct-pipeline/ukbiobank-spinalcord-csa/issues/31#issuecomment-771647698:

On our side, I think setting PYTHONUSERSITE=True and other global variables like it is not a good solution. Most apps, reasonably, assume that they are running on a standard-ish system. Breaking those assumptions makes SCT an incompatible thing like Matlab that people have to containerize to keep running. If we're going to force people to containerize we should at least provide the docker image ourselves.
For another hacky solution: We could try to isolate PYTHONUSERSITE to just SCT itself, maybe? Like, make sure every: subprocess.run([...], env={k: os.environ[k] for k in set(os.environ.keys()) - {'PYTHONUSERSITE'}}). That would be a hacky workaround but at least it wouldn't leak out beyond SCT anymore.

So, we would only modify the environment here because we want to restore how the environment was before SCT's modification, to prevent external programs from being affected. :sweat_smile:

:grimacing: aoyoe.
I think if we solve https://github.com/neuropoly/spinalcordtoolbox/issues/1526 this will go away. But for now, sure.

I knew I was looking for something else...
so is it too late to throw this one out there?
import site; site.ENABLE_USER_SITE = False

I think this is the same as PYTHONNOUSERSITE, but it should only exist in the current process. That should probably replace both #3067 and this.

I think this is the same as PYTHONNOUSERSITE, but it should only exist in the current process. That should probably replace both #3067 and this.

Hrm. I dug a little deeper and I'm not sure they are the same. It looks like site.ENABLE_USER_SITE is just a status indicator, rather than having any control over what gets added to sys.path:

Flag showing the status of the user site-packages directory. True means that it is enabled and was added to sys.path. False means that it was disabled by user request (with -s or PYTHONNOUSERSITE).

(Probably should have been called site.USER_SITE_ENABLED, eh?)

As well, the process for adding the user site-packages dir to sys.path seems to be done on initialization. From the site docs:

This module is automatically imported during initialization. The automatic import can be suppressed using the interpreter’s -S option.
Importing this module will append site-specific paths to the module search path and add a few builtins, unless -S was used.  In that case, this module can be safely imported with no automatic modifications to the module search path or additions to the builtins.

So, running import site again (a la import site; site.ENABLE_USER_SITE = False) I think has no effect.

Huhm. But then how come this workaround for this pip bug works? https://github.com/pypa/pip/issues/7953#issuecomment-645133255
With it, I can install pip install -e; without it I can't.
I guess maybe pip could be paying attention to it directly instead.

I guess maybe pip could be paying attention to it directly instead.

Indeed!


Does this project work?

Looking for a solution of a self hosted localtunnel. Since localtunnel itself doesn't work on Node v18+ I'd like to give this project a try.

@creazy231 it will help you run localtunnel server in your machine:
https://github.com/localtunnel/server/issues/203#issuecomment-1693807702
Let me know if it works.

Can you share error logs ?

Follow this instructions:
https://github.com/localtunnel/server/issues/203#issuecomment-1728643319


Provide syntax highlighting

A unified syntax highlighting for Tokay's .tok-files should be made available for major editors.
There had already been efforts for a simple syntax highlighting for hljs and sublime.
IMHO, there may be a simpler, generic way to provide a syntax highlighting for different highlighters and editors, but I'm not on this topic currently. This issue is a general collector for any related stuff.

After some research, I've found that TextMate seems to be a generic way to describe syntax highlighters.
Related links:

https://macromates.com/manual/en/language_grammars
https://code.visualstudio.com/api/language-extensions/syntax-highlight-guide


Module 2.3 Assignment

👋 Hi @MeesterDaaf the screenshot you have here is the Module 2.2 assignment. Please upload the 2.3 assignment.


Extension does nothing if file isn't *.md, should be mentioned in readme. Also, option to enable for .txt files too

I understand why you wouldn't want the keyboard shortcuts to be changing for files that aren't Markdown, but regardless, README.md and the Overview page should mention this requirement. I would also like a option, perhaps disabled by default, to allow this extension to work on .txt files as well.
Thanks for this great extension!

The activation events is before we can register command and configs. In .txt file, the extension is not activate at all.
A solution is to select language manually using the right bottom button in vscode

(using markdown mode in txt file)


Cursor position in Source Control

Issue Type: Bug
This might only apply when submodules are in use.
The cursor location in Source Control will jump from the current position inside of the commit message field to another submodule's commit field after staging a file.
To trigger this:

Have a Source Control section with multiple submodules.
Edit one or more files in the first submodule so that they are modified and able to stage for a commit.
Hit the Stage / Stage All button for the file.
While the staging operation is running, start typing in the commit message field until the staging operation is completed.

What will happen - while typing your commit message, the cursor will jump to a different submodule commit message field when the staging process completes, and all text typed afterward is directed to that field instead of the intended commit field.
VS Code version: Code 1.57.1 (507ce72a4466fbb27b715c3722558bb15afa9f48, 2021-06-17T13:28:07.755Z)
OS version: Windows_NT x64 10.0.18363
Restricted Mode: No

System Info

Item
Value

CPUs
Intel(R) Core(TM) i5-9400H CPU @ 2.50GHz (8 x 2496)

GPU Status
2d_canvas: enabledgpu_compositing: enabledmultiple_raster_threads: enabled_onoop_rasterization: enabledopengl: enabled_onrasterization: enabledskia_renderer: enabled_onvideo_decode: enabledvulkan: disabled_offwebgl: enabledwebgl2: enabled

Load (avg)
undefined

Memory (System)
31.83GB (21.22GB free)

Process Argv
--crash-reporter-id 06e9bb31-b3df-4039-a838-4f6b5f5bb3c9

Screen Reader
no

VM
0%

Extensions (7)

Extension
Author (truncated)
Version

gitlens
eam
11.5.1

python
ms-
2021.6.944021595

vscode-pylance
ms-
2021.7.1

jupyter
ms-
2021.6.999662501

cpptools
ms-
1.4.1

hexeditor
ms-
1.7.1

markdown-preview-enhanced
shd
0.5.22

A/B Experiments
vsliv368cf:30146710
vsreu685:30147344
python383:30185418
vspyt653:30270858
pythonvspyt602:30300191
vspor879:30202332
vspor708:30202333
vspor363:30204092
pythonvspyt639:30300192
pythontb:30283811
pythonvspyt551cf:30311713
vspre833:30321513
pythonptprofiler:30281270
vshan820:30294714
vscoreces:30322571
pythondataviewer:30285071
vscus158:30321503
pythonvsuse255cf:30323309
vscorehov:30309549
vscod805cf:30301675
pythonvspyt200cf:30331938
binariesv615:30325510
aatestflight:30331711

#135386

This also happens with a workspace with multiple folders, each with a git repo. It seems to happen when vscode runs git to see the changes, every 8 seconds or so - it jumps between the commit message boxes. Don't need to have any changes in the folders to see it.


Build failure: findimagedupes

Steps To Reproduce
Steps to reproduce the behavior:

install findimagedupes:$ nix profile install nixpkgs#findimagedupes

run  it with help option:$ findimagedupes --help
[...]

For some reasons it tries build some stuff after the main package has been built and installed into the profile, I don't understand this behaviour.

Build log
https://gist.github.com/olebedev/bed511344d2fa8b0d1f2f8b0677aaf2e
Notify maintainers
@Stunkymonkey
Metadata
[user@system:~]$ nix-shell -p nix-info --run "nix-info -m"
 - system: `"aarch64-darwin"`
 - host os: `Darwin 22.3.0, macOS 13.2.1`
 - multi-user?: `yes`
 - sandbox: `no`
 - version: `nix-env (Nix) 2.12.0`
 - channels(root): `"nixpkgs"`
 - nixpkgs: `/nix/var/nix/profiles/per-user/root/channels/nixpkgs`

maybe we should switch to https://gitlab.com/opennota/findimagedupes
Not sure about migration for a package.

i tried fixing it. But the perl binding are out of date. Therefore I removed it.
I will migrate to https://github.com/NixOS/nixpkgs/pull/242522

@Stunkymonkey I found a workaround (I don't know if it is appropriate).

Change
https://github.com/NixOS/nixpkgs/blob/9ef5b7d793a0ac1f1742696b5ec668169ff9c80d/pkgs/tools/graphics/findimagedupes/default.nix#L1
to

{ lib, stdenv, fetchurl, makeWrapper, perl, perlPackages, installShellFiles, libxcrypt }:

Change
https://github.com/NixOS/nixpkgs/blob/9ef5b7d793a0ac1f1742696b5ec668169ff9c80d/pkgs/tools/graphics/findimagedupes/default.nix#L33
to

      --replace "DIRECTORY => '/usr/local/lib/findimagedupes';" "DIRECTORY => '/tmp'; use Inline C => config => ccflagsex => \"-I${libxcrypt}/include\";" \

Reference: https://metacpan.org/dist/Inline-C/view/lib/Inline/C.pod#C-CONFIGURATION-OPTIONS
findimagedupes is still better for my use case because it is a command line utility but the python variant is not.

@amalgame21 findimagedupes is already removed in #242697. If you want it to add it again. Please go ahead an submit an PR.


Add operator type inspections

Operators (in particular arithmetic operators) should ensure their operands are covariant of number.
The following certainly shouldn't be allowed:
---@type boolean
local aBoolean = true + true -- Illegal

However, non-arithmetic operators should also be checked for compatibility e.g.
---@type number
local a = 1

---@type number
local length = #a -- Illegal

Be careful when dealing overloaded operators, though :)


Cannot compile standard libraries?

I don't know why I cannot compile standard libraries in my code. I want to use rand() function but it shows undefined reference to rand() I thought this is because of the -nostdlibs flag so I commented that flag but now it says cannot find -lc,lgloss,crt0.o?? Is it because the toolchain does not comes with these standard libraries?? I have tried in both windows(WSL) and ubuntu and the output is same how do I solve this error I could not find anything related to this on the internet please help if you guys have any idea on this...

I'm going to start by advising not to do this, particularly if what you are after is rand, there are better rand implementations than the one in the standard library. If you just need rand from the standard library, I recommend looking into pcg random or an xorshift variant. Either are easy to incorporate into a project without additional dependencies and provide better statistical properties than the standard library implementation.
In order to get standard libraries to work, you will need a standard library that supports microcontrollers. However, it's not quite that simple (you can't just drop it in and expect it to work) since the library doesn't know how to interact with the hardware. Typically, people use either newlib or picolibc (personally, I've only used newlib sometime in the past). You will need to implement a certain subset of "stub" functions that allow the library to interact with the underlying hardware in the way you want (these basically implement functions that are usually provided by the linux kernel). Hackaday article, newlib documentation.
Hope this is helpful.
Soli Deo Gloria

If it's just rand you need, there are a ton of ways to get a pseudo-random number that's usable for many cases where you'd normally use stdlib (e.g. NOT secure crypto).
Something like this is easy to copy/paste into an application:
static uint16_t random(void){
  static unsigned long lfsr = 0xfeedfaceUL; // or seed this from a floating ADC, or some other hard-to-predict value
  if(lfsr & 1) lfsr = (lfsr >>1) ^ LFSR_MASK ; else lfsr >>= 1;
  return(lfsr);
}

The reason things like rand() are not included in ch32v003fun is because they are so context-dependent, i.e.
You could use a reference implementation like this one:
https://github.com/dolthub/musl/blob/8aec69db0851e6e197e69b8dca8da3344190f6dd/src/prng/rand.c#L11
It's not very good though, and, because the ch32v003 doesn't have a multiply, it's going to be slow.

Also for more background, these sorts of features go back and forth a lot, see this : https://github.com/dolthub/musl/commit/c79cd27e9e81eb5e223728094f1233ee2fc12dda

Thank you everyone for your guidance, it is very helpful.


Toward V1.0 : _def_functions, _plot_preset, CES...

SET DPARAM IS STILL BROKEN AND IS BROKEN IN OTHER BRANCHES
Summary :

Readme now have an explanation on how to contribute

_models now have a a _def_functions file, and an exploration functions pgm.get_available_functions linked to it

New plots have been created and are accessible in presets

CES explicit functions has been written in _def_func

SET DPARAM IS NOT WORKING ANYMORE see #147

New functions !
Motivation : A lot of functions are shared from models to models, and some models are simply one field to be different. It is easier to write files, compare them and modify them if they have the same bricks.
How is the file structured: The file is an ensemble of class, each having a docstring and functions inside. Each class is a general category (population, productivity...) and contains logic elements. For example :

The formalism is the same as in a model file !
Each function is implicitly associated with a specific field in mind ( it is not explicit for the moment).
Sometimes there can be two fields in a same class, as they are linked in their definitions.

Some more elaborate case exist. For example, a production function is often associated with optimisation of profit : the expression of L is deduced from the expression of Y which in term of equations might be difficult.
I created in consequence subclasses, each fields should be loaded together in consequence :

It can be read through pgm.get_available_functions()

The files now read as :

instead of

Is it better ? I'd love to discuss it with you all.
Improvements :

itself should not be used at that type of place
The name of fields it should be applied on should be more explicit ?
There should be a field in pgm.get_available_functions() giving which models use it (same as in get_available_fields)

They are lambda function because it allows better readability ! functions are possible of course.

Plots
New plots
You can plot all the variables sharing the same units : ( plot name : plotbyunits ). Not much control on this.

You can plot 3D with colors using plot3D with the following controls _plot3D(hub, x, y, z, cinf, cmap='jet', index=0, title=''), x,y,z,cinf are name of fields.

Same idea with phasespace  x='omega', y='lambda', color='time',

You can plot as many y axis on the same figures if you want to be concise in term of figure density :

'x': 'time',
'y': [['lambda', 'omega'],
['d'],
['kappa', 'pi'],
],
_plotnyaxis(hub, x='time', y=[[]], idx=0, title='', lw=1)
Calling these plots in preset
A new formalism has been introduced to associate figures and preset :
         'plots': {
            'timetrace': [],
            'plotnyaxis': [{'x': 'time',
                           'y': [['lambda', 'omega'],
                                 ['d'],
                                 ['kappa', 'pi'],
                                 ],
                            'idx':0,
                            'title':'',
                            'lw':1}],
            'phasespace': [{'x': 'lambda',
                            'y': 'omega',
                            'color': 'd',
                            'idx': 0}],
            'plot3D': [{'x': 'lambda',
                        'y': 'omega',
                        'z': 'd',
                        'cinf': 'pi',
                        'cmap': 'jet',
                        'index': 0,
                        'title': ''}],
            'plotbyunits': [{'title': '',
                             'lw': 1,       # optional
                             'idx': 0,      # optional
                             'color': 'k'},  # optional
                            ],

The structure :
it is a dict, each key is a plot in _def_plot, and the value is a list.
each ingredient of the list is a dictionary, that contains all the arguments to pass to the plot function.
To use it
If you have your preset loaded : hub.plot_preset() after a run
if you want to use the preset plots without the preset (it might not work if you need multiple indexes in the preset ) hub.plot_preset(preset='default')
CES
The extensive file GK-CES is ready.
There are many ways to write the production function, I tried to show the different characteristics that I found interesting under the spotlight !
pgm._core._plotnyaxis(hub,x='time',y=[['l'],['omega','lambda'],['d','nu']])

Next step

[ ] Finishing the file reduced equations
[ ] Verify that they give the same dynamics
[ ] Reproduce all graphs from the article\

Hello @DaluS! Thanks for opening this PR.

In the file pygemmes/_models/_def_functions.py:

Line 10:1: E302 expected 2 blank lines, found 1
Line 10:15: E203 whitespace before ':'
Line 26:1: E302 expected 2 blank lines, found 1
Line 26:12: E203 whitespace before ':'

In the file pygemmes/_models/_model_GoodwinKeen-CES.py:

Line 7:101: E501 line too long (120 > 100 characters)
Line 89:28: E231 missing whitespace after ','
Line 89:32: E231 missing whitespace after ','
Line 89:36: E231 missing whitespace after ','
Line 89:40: E231 missing whitespace after ','
Line 89:67: E231 missing whitespace after ','
Line 90:36: E231 missing whitespace after ','
Line 91:42: E231 missing whitespace after ','
Line 91:46: E231 missing whitespace after ','
Line 91:50: E231 missing whitespace after ','
Line 91:54: E231 missing whitespace after ','
Line 93:101: E501 line too long (103 > 100 characters)
Line 95:38: E203 whitespace before ':'
Line 97:1: E302 expected 2 blank lines, found 1
Line 101:46: E203 whitespace before ':'
Line 104:1: E302 expected 2 blank lines, found 1
Line 180:66: E203 whitespace before ':'
Line 180:101: E501 line too long (116 > 100 characters)
Line 181:18: E203 whitespace before ':'
Line 185:18: E203 whitespace before ':'
Line 189:18: E203 whitespace before ':'
Line 195:18: E203 whitespace before ':'
Line 199:18: E203 whitespace before ':'
Line 203:18: E203 whitespace before ':'
Line 204:13: E123 closing bracket does not match indentation of opening bracket's line
Line 212:34: E231 missing whitespace after ','
Line 243:9: E266 too many leading '#' for block comment
Line 245:20: E203 whitespace before ':'
Line 249:20: E203 whitespace before ':'
Line 253:9: E266 too many leading '#' for block comment
Line 255:20: E203 whitespace before ':'
Line 258:18: E231 missing whitespace after ':'
Line 259:20: E203 whitespace before ':'
Line 262:18: E231 missing whitespace after ':'
Line 263:20: E203 whitespace before ':'

Do see the Hitchhiker's guide to code style

@guittcam @Didou09 if you can check this new version, I'd love to have your feedback !
I'm working on my models in the meantime

The new function formalism is probably better because

it makes the file very clean
it reduces the risk of typo causing errors

Can you test that :

Your code is working with it (especially set_dparam which has a weird behavior)
You can use all the new functionalities !

Added doc\BasinExploration.py, a few elements to explore basin of attractions in a more automated way

All these new functionalities are very cool !
Congratulations and sorry for the long silence, these are busy times,
Agree with @guittcam , the new function formalism is cleaner and less prone to typos.
I'll try to come back gradually in the coming week and do some tests and clean-up in the unit tests (and come back to work on the distribution topic).


Implemented prompt for survey using A/B test framework

For #6957

[x] Pull request represents a single change (i.e. not fixing disparate/unrelated things in a single PR)
[x] Title summarizes what is changing
[x] Has a news entry file (remember to thank yourself!)
[x] Appropriate comments and documentation strings in the code
[x] Has sufficient logging.
[x] Has telemetry for enhancements.
[x] Unit tests & system/integration tests are added/updated
~[ ] Test plan is updated as appropriate~
~[ ] package-lock.json has been regenerated by running npm install (if dependencies have changed)~
~[ ] The wiki is updated with any design decisions/details.~

Codecov Report

Merging #7090 into master will increase coverage by 0.28%.
The diff coverage is 100%.

@@            Coverage Diff             @@
##           master    #7090      +/-   ##
==========================================
+ Coverage   57.67%   57.96%   +0.28%
==========================================
  Files         483      483
  Lines       21025    21379     +354
  Branches     3446     3447       +1
==========================================
+ Hits        12127    12392     +265
- Misses       8124     8209      +85
- Partials      774      778       +4

Impacted Files
Coverage Δ

src/client/activation/extensionSurvey.ts
98.11% <100%> (+0.19%)
:arrow_up:

src/client/common/experimentGroups.ts
100% <100%> (ø)
:arrow_up:

src/client/testing/serviceRegistry.ts
48.64% <0%> (-44.21%)
:arrow_down:

...t/datascience/jupyter/jupyterDataRateLimitError.ts
66.66% <0%> (-33.34%)
:arrow_down:

src/client/formatters/serviceRegistry.ts
75% <0%> (-25%)
:arrow_down:

src/client/common/utils/icons.ts
83.33% <0%> (-16.67%)
:arrow_down:

src/client/testing/codeLenses/main.ts
44.44% <0%> (-15.56%)
:arrow_down:

src/client/common/platform/platformService.ts
32.5% <0%> (-10%)
:arrow_down:

src/client/common/platform/pathUtils.ts
66.66% <0%> (-5.56%)
:arrow_down:

src/client/linters/pydocstyle.ts
84.61% <0%> (-5.13%)
:arrow_down:

... and 34 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 1b7ab99...89a270f. Read the comment docs.


Change Mouse Cursor/Pointer on Android

Issue details
Android 7.0 and newer lets developers customize the mouse cursor with different system pointers, custom bitmaps, or hiding the pointer. It would be nice if libGDX supported this.
Official Documentation:
https://developer.android.com/about/versions/nougat/android-7.0#custom_pointer_api
https://developer.android.com/reference/android/view/PointerIcon.html
I wrote a Stack Overflow answer explaining how to use it:
https://stackoverflow.com/questions/15636855/custom-mouse-pointer-on-android/55482761#55482761
Version of LibGDX and/or relevant dependencies
libGDX v1.9.9
Please select the affected platforms

[x] Android
[ ] iOS (robovm)
[ ] iOS (MOE)
[ ] HTML/GWT
[ ] Windows
[ ] Linux
[ ] MacOS

The core interface already has methods for it, so no one will stop you if you submit a PR for the Android backend. :)

I don't have experience working with Git or Maven projects. I've been tinkering with it, so maybe I'll be able to do a PR when I know more about it.

You can join the discord if you wanna some help

@SimonIT I got help on Discord, and now I'm working on adding this.

How could someone test this?
Can I test this without an extra mouse?

@intrigus You need to connect a mouse to test as far a I know.
I already got this working on my fork. I never got around to making a pull request. It's different than libGDX's API for other platforms.
https://github.com/GrantOjanen/libgdx/tree/android-cursor

Closing. The code on the linked branch is nowhere near being merge-ready, the pointers are a handful premade ones.
Given the fact that using a mouse on Android is rarely used, and games changing the cursor are also rare, there's not much need in that feature. If anyone needs this and opens a PR though, we are open to test and add it.


Enhancement to parametric heatsink function

Even if the number of columns is 1, we can leave this as a parameter so that it can be changed down the line

Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 76.44%. Comparing base (5660dbb) to head (b0c4513).

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #4498      +/-   ##
==========================================
- Coverage   81.42%   76.44%   -4.98%
==========================================
  Files         110      110
  Lines       53510    53508       -2
==========================================
- Hits        43570    40906    -2664
- Misses       9940    12602    +2662


Small documentation fixes

Hi,
I've noticed a couple of typos while digging into Nitrogen's documentation, which this PR fixes.
P.S. Amazing project, so glad I found it!

Thank you very much! And welcome to Nitrogen!


Fix activation quantization

It looks like the turbo_util wrapping and grouped gemm pasting broke it. This PR fixes it.

Oops - thanks for the fix! If you fix the turbo installation issue then I can test the quantization features more easily :)

Do you want me to cut a new release with these fixes?

Do you want me to cut a new release with these fixes?

@tgale96 can we cut one? that would be amazing :)

Done!
https://github.com/stanford-futuredata/megablocks/releases/tag/v0.3.1


Using FluentUi with Binding in GetX

i'm using FluentApp and it didn't work with GetX binding, any alternative ?

You may find some instructions in #72


Why are rooms and equipment listed under users endpoint?

What is the reason that meeting rooms or equipment can be listed via /users endpoint?
When calling the /v1.0/users endpoint it returns also "users" associated with a meeting room, or a piece of equipment (theirs mailBoxSettings.UserPurpose is either a room or a piece of equipment).
I cannot use POST /users to create a room or a piece of equipment, so it's a little bit confusing.
It's also hard to filter only real users.

Document Details
⚠ Do not edit this section. It is required for learn.microsoft.com ➟ GitHub issue linking.

ID: 71215894-d1c8-8d36-9dbb-adb35cfbc5f9
Version Independent ID: 34e2e6e4-578a-9309-ebcf-94b640dc4e4c
Content: user resource type - Microsoft Graph v1.0
Content Source: api-reference/v1.0/resources/user.md
Product: users
Technology: microsoft-graph
GitHub Login: @yyuank
Microsoft Alias: MSGraphDocsVteam

I agree with you. This is probably some legacy design where some services used User entity to represent non-user resources. Today we have review processes set up and we make sure any new services can't use the User entity this way.


cargo build error :  error: could not compile sqlx-core @v0.5.3

Did a cargo update with the following changes to my cargo.toml file:

sqlx = { version = "0.5.3", features = [ "runtime-actix-rustls", "postgres","chrono","json"] }
then did cargo update
then cargo run

But now get error : "error: could not compile sqlx-core"
Platform  => vscode, osx, rust analyzer
Tried reverting my cargo.toml file to :  sqlx = { version = "0.5.1", features = [ "runtime-actix-rustls", "postgres","chrono","json"] }
then cargo update
But still get the same error.
How does one revert back to a working version.

Fixed in 0.5.4, thanks @ETCaton

i'm using sqlx = { version = "0.5", features = [ "runtime-actix-rustls", "postgres","chrono","json"] }
But now my cargo build just hangs....
platform : osx, vscode, rust analyzer

Should be fixed in 0.5.5


Watir::Browser.new = nil:NilClass

Meta -
Watir Version: 6.15.0
Selenium Version: 3.11.0
OS Version: 10.14
Expected Behavior -
browser = Watir::Browser.new
=> #<Watir::Browser:..
Actual Behavior -
browser = Watir::Browser.new
=> NoMethodError: undefined method `[]' for nil:NilClass
It was running perfectly fine 10 minutes beforehand, and then this happened. Anyone have any ideas?

Are you running this as part of a test runner or just in irb? Can you provide a Stack Trace for the error?

Hi @titusfortner,
It was part of a rake task and then I tried it in irb too and got the same response.
This is my Stack Trace in irb:
irb(main):005:0> require 'webdrivers'
=> false
irb(main):006:0> require 'watir'
=> false
irb(main):007:0> browser = Watir::Browser.new
Traceback (most recent call last):
2: from (irb):7
1: from (irb):7:in new' NoMethodError (undefined method []' for nil:NilClass)

Please set the following and rerun
Webdrivers.logger.level = :debug
Selenium::WebDriver.logger.level = :debug

I can not reproduce this and expect it is an issue with configuration on the system. If you still need help, please reach out to us in one of the ways specified here: http://watir.com/help/


Searchguard support in Report for 5.4.1

Is there any possibilities to support following configuration (highlighted) in 5.4.1 or can we have some manual steps to enable such support?
report:
active: true
tmp_path: /tmp/
search_guard: true
simple_authentication: false

I believe this might require backporting a few bits from later versions. Perhaps @sergibondarenko knows

@sergibondarenko Since this was bit important for us. I went ahead and made following changes in my test environment to enable this in 5.4.1. It appears to be working fine so far. However, I would like to review my changes with you before moving them to next environment. Could you please review them once and let me know if this breaks something else or it's fine to move ahead?
Changes:
replaced /usr/share/kibana/plugins/sentinl/server/lib/action.js with /usr/share/kibana/plugins/sentinl/server/lib/actions
replaced /usr/share/kibana/plugins/sentinl/server/lib/helpers.js
replaced /usr/share/kibana/plugins/sentinl/node_modules
replaced /usr/share/kibana/plugins/sentinl/server/lib/scheduler.js
replaced /usr/share/kibana/plugins/sentinl/init.js
updated /usr/share/kibana/plugins/sentinl/index.js
report: Joi.object({
search_guard: Joi.boolean().default(false),
simple_authentication: Joi.boolean().default(false),
upadated /usr/share/kibana/plugins/sentinl/server/lib/classes/watcher.js
import actionFactory from '../actions/actions';
Appreciate your quick response on this.

@lmangani: Could you please assign this to @sergibondarenko

@varun0002 we can't review your changes unless a PR is submitted


Remove cap on CategoricalArrays.jl and DecisionTree.jl

#64

Code will fail until:

[ ] https://github.com/alan-turing-institute/MLJBase.jl/pull/54 is merged
[ ] MLJBase 0.7.0 is released.


Add Git Cloning?

Is it possible to add git cloning into git-plus?

Gitplus work with existing project so git clone no work directly and this process run only once
git init
git remote add origin $url_of_clone_source
git fetch origin
git checkout -b master --track origin/master # origin/master is clone's default

http://stackoverflow.com/questions/5377960/whats-the-best-practice-to-git-clone-into-an-existing-folder

You can use terminal into ide instance
https://github.com/platformio/platformio-atom-ide-terminal

See #278


Added CM ASP.NET extensions and CM metapackage

API change check
APIView has identified API level changes in this PR and created following API reviews.
Azure.CloudMachine.All


adding entity framework sample

@jeffhollan sorry about moving things around as you were writing this. Now that things are released and will be stable, want to update this so we can get the sample in?

@fabiocav @mhoeger ok - made all suggested changes. I think this is ready to go now. LMK


Why so many uncommented PRs?

I'm a bit concerned about the number of PRs that have been sitting out there with no comments. Is this module still being actively maintained/developed?

Hi @justinlevi ... this repo has fallen a bit behind, as you can see, but we're in the process of getting it up-to-date. The module on DO was being community maintained and fell out of sync with this repo. We have some security updates being tested now, then we'll work with the main maintainer on DO to get that repo synced with this one.
I'll answer your other two issues as soon as I can.


Custom Android toasts not rendering correctly

Steps to reproduce
No response
✔ Expected Behavior
No response
❌ Actual Behavior
No response
Other Software
No response
Have you specified the version of Windows Subsystem for Android under Milestone?

[X] I have set this issue's milestone

https://github.com/microsoft/WSA/issues/44#issue-1398782471


Stop raising AttributeError when accessing missing calibration

Summary
In #9597 we updated the InstructionProperties.calibration to internally store CalibrationEntry subclass. However, there still be the case None is stored instead when user/system doesn't provide any calibration, e.g. reset instruction or any control flow operation. Accessing calibration of such entry raises AttributeError, which must be suppressed and it must return None instead.
Details and comments

Pull Request Test Coverage Report for Build<PHONE_NUMBER>

2 of 2 (100.0%)  changed or added relevant lines in 1 file are covered.
No unchanged relevant lines lost coverage.
Overall coverage increased (+0.002%) to 85.343%

Totals

Change from base Build<PHONE_NUMBER>:
0.002%

Covered Lines:
67528

Relevant Lines:
79125

💛  - Coveralls


feat(performance): The removal of extra LRU promotes.

There is definitely a performance win here.  When the extra promotes
are removed both flamegraphs and benchmarksjs show a difference.

nice


Fix corrupt scene when export var has setter

Closes #45357
Where after saving a scene can generate an empty property name in .tscn, making it corrupted, for instance:
[node name="BallLight" type="Node3D"]
script = ExtResource( 2 )
 = null

The issue is caused when an @export variable has a setter, such as:
@export var isOn: bool = false:
	set(val):
		isOn = val

@akien-mga Thanks!  Just in case, the explanation of this change is a little more precise here: https://github.com/godotengine/godot/issues/45357#issuecomment-805997489

Thanks!


Some individuals dont have types

Description of the issue
Some individuals dont have types which is an error. This has to be corrected before #167
Ideas of solution
individuals with ideas where to put them:

Appliance: delete because we have a class with the same name
Aviation: delete because we have a class with the same name
BatteryElectric: delete because we have a class with the same name
CH4: GreenhouseGas
Database: delete because we have a class with the same name
Hydrogen: delete because we have a class with the same name
InternalCombustion
InternalCombustionVehicle
Month: delete because it should be a class
N2O: GreenhouseGas
NF3: GreenhouseGas
NuclearPower: delete because we have a class with the same name
Oil: delete because it should be a class
Passive
Plot: delete because we have a class with the same name
PortionOfCoal: delete because we have a class with the same name
PortionOfHardCoal: delete because we have a class with the same name
PortionOfLignite: delete because we have a class with the same name
PortionOfWater: delete because we have a class with the same name
Rail: delete because we have a class with the same name
SF6: GreenhouseGas
V2Grid: delete because we have a class with the same name
Wind: delete because we have a class with the same name
Year: delete because it should be a class

Workflow checklist

[ ] I discussed the issue with someone else than me before working on a solution
[ ] I already read the latest version of the workflow for this repository
[ ] The goal of this ontology is clear to me

I am aware that

[ ] every entry in the ontology should have an annotation
[ ] classes should arise from concepts rather than from words
[ ] class or property names should follow the UpperCamelCase

They are the following:
Appliance
Aviation
BatteryElectric
Database
Hydrogen
InternalCombustion
InternalcombustionVehicle
Month
NuclearPower
Oil
Passive
Plot
PortionOfCoal
PortionOfHardCoal
PortionOfLignite
PortionOfWater
Rail
V2Grid
Wind
Year
I think many of these should perhaps be classes rather than individuals?

yes, most of them are classes already, I made a list above

Sorted list:
delete because we have a class with the same name:

Appliance
Aviation
BatteryElectric
Database
Hydrogen
NuclearPower
Plot
PortionOfCoal
PortionOfHardCoal
PortionOfLignite
PortionOfWater
Rail
V2Grid
Wind

equivalent class with assigned type:

CH4: GreenhouseGas
N2O: GreenhouseGas
NF3: GreenhouseGas
SF6: GreenhouseGas

should-be-classes:

Month: one-dimensional temporal region
Oil: or rather PortionOfOil: PortionOfMatter
Year: one-dimensional temporal region

others:

InternalCombustion: Combustion? Make class?
InternalCombustionVehicle: InternalCombustion or Combustion?
Passive: Modus?

please have a look at especially the last two categories as they are the only ones with "creative input" from me

I agree with most of these. All the "delete because they are a class already" - yes. Month and year are definitely temporal regions and should be classes rather than individuals. Oil should be portion of oil, classified beneath portion of matter. Do you think internal combustion should be classified as a type of energy generation technology? And an internal combusion vehicle is a vehicle that is powered by an internal combustion engine. In any case, these two should also be classes.
The only one that I would not be clear about is that I do not know what passive is. If it is a modifier, what is it a modifier of?

The only one that I would not be clear about is that I do not know what passive is. If it is a modifier, what is it a modifier of?

I put it unter modus because active and inactive are modis

Do you think internal combustion should be classified as a type of energy generation technology? And an internal combusion vehicle is a vehicle that is powered by an internal combustion engine. In any case, these two should also be classes.

Yes. energy generation technology would fit better than combustion. I'm not sure if we even need a "InternalCombustionVehicle" Class/ Instance

I put it unter modus because active and inactive are modis

Okay, sounds good. Passive is either a synonym of inactive, or very closely related.

@han-f, @l-emele, @stap-m what do you think about InternalCombustion and InternalCombustionVehicle?

I'd say InternalCombustion is a subclass somewhere in the Technology tree and InternalCombustionVehicle is a subclass of ArtificialObject.

ok, then I'll put InternalCombustion under EnergyGeneratorTechnology and the vehicle under EnergyGenerator.
I want to get this issue closed fast so we can do the modularisation so can we for the moment dont add definitions? I'll implement this now


Close disconnected session

Description
This PR leverages the KeepAlive building block to keep track of client opened sessions (at Crossdata Driver).

Centralices session management.
Closes sessions as jobs are cancelled after an inactive session has expired.
Terminates jobs belonging to explicitly closed sessions.
Removes few vestigial code snippets.

Testing

[x] Unit, integration tests

Nice! 👍

:+1:


Would it be possible to use flyway to compare two database schemas and upgrade one's schema to match the other?

I'm using a pretty niche ORM, Kotlin based Exposed library.
https://github.com/JetBrains/Exposed
To my knowledge, there exists no official way to do database migrations using this library, or the Kotlin backend stack.
Clearly, to put this stack into production (ktor + exposed), we need to figure out a way to do migrations.
So back to raw SQL. Here's an idea I have:

There's the prod server with old schema
There's the dev server with new schema

I want to say

Hey prod server, get ready to update. I want prod DB schema to look exactly like dev DB schema. Do some auto-magic migration script generation and make it so.

Would it be possible to extend flyway to become something like this?
Or can you recommend any other directions that you think I should explore to accomplish this?

We've had another issue asking about JetBrains' Exposed. We don't support it, and aren't aware of any third party integrations either.
As for your idea, Flyway would certainly enable the workflow you described. You can run the baseline command on your production DB to get it ready for Flyway migrations. However the main limitation is that Flyway doesn't automagically generate migrations.
Migration generation is something you'd have to solve yourself. There are numerous comparison engines available for different databases (SQL Compare, PostgresCompare, etc). These can generate a SQL script representing the diff of dev to production. You can use that diff script as a Flyway migration.
So the deployment workflow would be:

Compare dev and production using the comparison engine
Save the diff script as a Flyway migration
Run flyway migrate on production

This can probably be achieved with a script (PowerShell, Bash, etc) that calls into the various APIs. If you want something built in, Flyway is open source, so you can fork the repo and build this yourself.

SQL generation is beyond the scope of Flyway.

@Aditya94A This might sound crazy, but I'm pretty certain IntelliJ's JPA Explorer (formerly JPA Buddy) uses liquibase under the covers to do exactly this. So while the tool allows you to generate a Flyway migration based on the diff between two databases, the actual SQL generation for the migration is done via liquibase tooling under the covers.

I am not sure, but according this article, and according features described in paid version (enterprise and also teams) of flyway, comparison and migration generation is possible.
Through, free version of flyway does not know how o generate migration.
PS: There is bytebase project which claims to do migration for you. There are also other projects which does that (visual studio for example). Beware of difference between various products. Sometimes renaming column (or changing its type) can be done in various approaches and some of them is not feasible on tables with many data. Some generators are so clever, that they are able to do "magic" migrations using tricks for safety/performance.


flagext -> flagx, runtimeext -> rtx

Rename two libraries.  Followed by two followup PRs fixing all usage I could find.

This change is

Pull Request Test Coverage Report for Build 166

2 of 2 (100.0%)  changed or added relevant lines in 1 file are covered.
No unchanged relevant lines lost coverage.
Overall coverage remained the same at 62.844%

Totals

Change from base Build 159:
0.0%

Covered Lines:
137

Relevant Lines:
218

💛  - Coveralls


FIx group chat confirmation list is displayed with delay

Details

The delay is because we set the opacity to 0 on initial render.
Fixed Issues

$ https://github.com/Expensify/App/issues/39245
PROPOSAL: https://github.com/Expensify/App/issues/39245#issuecomment-2027433410
Tests

Same as QA Steps

[x] Verify that no errors appear in the JS console

Offline tests

Same as QA Steps
QA Steps

Press FAB and select Start chat
Add multiple people to group selection
Press Next
Verify the list is displayed without delay on the confirmation page

[x] Verify that no errors appear in the JS console

PR Author Checklist

[x] I linked the correct issue in the ### Fixed Issues section above
[x] I wrote clear testing steps that cover the changes made in this PR

[x] I added steps for local testing in the Tests section
[x] I added steps for the expected offline behavior in the Offline steps section
[x] I added steps for Staging and/or Production testing in the QA steps section
[x] I added steps to cover failure scenarios (i.e. verify an input displays the correct error message if the entered data is not correct)
[x] I turned off my network connection and tested it while offline to ensure it matches the expected behavior (i.e. verify the default avatar icon is displayed if app is offline)
[x] I tested this PR with a High Traffic account against the staging or production API to ensure there are no regressions (e.g. long loading states that impact usability).

[x] I included screenshots or videos for tests on all platforms
[x] I ran the tests on all platforms & verified they passed on:

[x] Android: Native
[x] Android: mWeb Chrome
[x] iOS: Native
[x] iOS: mWeb Safari
[x] MacOS: Chrome / Safari
[x] MacOS: Desktop

[x] I verified there are no console errors (if there's a console error not related to the PR, report it or open an issue for it to be fixed)
[x] I followed proper code patterns (see Reviewing the code)

[x] I verified that any callback methods that were added or modified are named for what the method does and never what callback they handle (i.e. toggleReport and not onIconClick)
[x] I verified that the left part of a conditional rendering a React component is a boolean and NOT a string, e.g. myBool && <MyComponent />.
[x] I verified that comments were added to code that is not self explanatory
[x] I verified that any new or modified comments were clear, correct English, and explained "why" the code was doing something instead of only explaining "what" the code was doing.
[x] I verified any copy / text shown in the product is localized by adding it to src/languages/* files and using the translation method

[x] If any non-english text was added/modified, I verified the translation was requested/reviewed in #expensify-open-source and it was approved by an internal Expensify engineer. Link to Slack message:

[x] I verified all numbers, amounts, dates and phone numbers shown in the product are using the localization methods
[x] I verified any copy / text that was added to the app is grammatically correct in English. It adheres to proper capitalization guidelines (note: only the first word of header/labels should be capitalized), and is either coming verbatim from figma or has been approved by marketing (in order to get marketing approval, ask the Bug Zero team member to add the Waiting for copy label to the issue)
[x] I verified proper file naming conventions were followed for any new files or renamed files. All non-platform specific files are named after what they export and are not named "index.js". All platform-specific files are named for the platform the code supports as outlined in the README.
[x] I verified the JSDocs style guidelines (in STYLE.md) were followed

[x] If a new code pattern is added I verified it was agreed to be used by multiple Expensify engineers
[x] I followed the guidelines as stated in the Review Guidelines
[x] I tested other components that can be impacted by my changes (i.e. if the PR modifies a shared library or component like Avatar, I verified the components using Avatar are working as expected)
[x] I verified all code is DRY (the PR doesn't include any logic written more than once, with the exception of tests)
[x] I verified any variables that can be defined as constants (ie. in CONST.js or at the top of the file that uses the constant) are defined as such
[x] I verified that if a function's arguments changed that all usages have also been updated correctly
[x] If any new file was added I verified that:

[x] The file has a description of what it does and/or why is needed at the top of the file if the code is not self explanatory

[x] If a new CSS style is added I verified that:

[x] A similar style doesn't already exist
[x] The style can't be created with an existing StyleUtils function (i.e. StyleUtils.getBackgroundAndBorderStyle(theme.componentBG))

[x] If the PR modifies code that runs when editing or sending messages, I tested and verified there is no unexpected behavior for all supported markdown - URLs, single line code, code blocks, quotes, headings, bold, strikethrough, and italic.
[x] If the PR modifies a generic component, I tested and verified that those changes do not break usages of that component in the rest of the App (i.e. if a shared library or component like Avatar is modified, I verified that Avatar is working as expected in all cases)
[x] If the PR modifies a component related to any of the existing Storybook stories, I tested and verified all stories for that component are still working as expected.
[x] If the PR modifies a component or page that can be accessed by a direct deeplink, I verified that the code functions as expected when the deeplink is used - from a logged in and logged out account.
[x] If the PR modifies the UI (e.g. new buttons, new UI components, changing the padding/spacing/sizing, moving components, etc) or modifies the form input styles:

[x] I verified that all the inputs inside a form are aligned with each other.
[x] I added Design label and/or tagged @Expensify/design so the design team can review the changes.

[x] If a new page is added, I verified it's using the ScrollView component to make it scrollable when more elements are added to the page.
[x] If the main branch was merged into this PR after a review, I tested again and verified the outcome was still expected according to the Test steps.

Screenshots/Videos

Android: Native

https://github.com/Expensify/App/assets/50919443/bb2cef2a-1e3f-4aad-ae93-f47e068c8a37

Android: mWeb Chrome

https://github.com/Expensify/App/assets/50919443/6c1d0033-341c-44fe-ac31-d0f69c59a234

iOS: Native

https://github.com/Expensify/App/assets/50919443/a4533c5c-c865-4570-9ed7-a2350d3ae1b0

iOS: mWeb Safari

https://github.com/Expensify/App/assets/50919443/84208a8d-01e3-4cb8-8448-63e59130da42

MacOS: Chrome / Safari

https://github.com/Expensify/App/assets/50919443/3ac15245-1255-4a27-a0ac-7d2026a6e768

MacOS: Desktop

https://github.com/Expensify/App/assets/50919443/1df399d5-278b-40bc-b6d9-1dfe00c7ff3f

Out of scope, I think the bottom spacing is too big on iOS.

That's really weird.
Btw, I can see the "blink" now in this PR very briefly, the delay you and I see is most likely because the list scrolls to the target location and it's blank because not all items are rendered on the list.
https://github.com/Expensify/App/assets/50919443/78cdc5a7-db38-41ec-9a1a-b29ec8dc8a63
Weirdly though, the set state here somehow makes it faster.
https://github.com/Expensify/App/blob/488664d0091602b7f8f8f77e011219d832224b6c/src/components/SelectionList/BaseSelectionList.tsx#L355
I can put any set state and it makes it faster. Pretty sure that is an unintentional positive(?) effect.
I think the safest way is to have a new prop (maybe call it shouldHideListOnInitialRender defaults to true and set to false for group confirmation page) to conditionally apply isInitialSectionListRender to set the opacity instead of fully removing it, wdyt?

@Ollyws updated

🚀 Deployed to production by https://github.com/Julesssss in version: 1.4.60-13 🚀

platform
result

🤖 android 🤖
success ✅

🖥 desktop 🖥
success ✅

🍎 iOS 🍎
success ✅

🕸 web 🕸
success ✅


Update help.js

Made it ignore categories if undefined

Instead of ignoring can we possibly put it in a "Default" category? @TECHNOPAWS

@NightSling That should do the job.


D45 docs viz api bugfix, Tableau Exchange change, doc updates

Updates to the documentation to fix:

VizAPI custom title doc bug
Tableau Exchange (rebranding)
Various doc fixes found by @bcantoni

Note: this branch has already been merged with dev
https://github.com/tableau/extensions-api/commit/bee097b580d6706ba8b9f8de9487278aa68078ae

@seanmakesgames  - This PR has the same commit baggage as the PR for dev => main. That's probably because I created my branch off of dev and not main.  And then I also merged dev into my branch to resolve some branch conflicts. Should we try a different approach? Maybe cherry pick commits?  Wait until another day?  Create a gh-pages branch and start with that?

We'll resolve it together quick

Closing this one and coordinating with the team.


Update docs and snforge init to use dev-dependencies

https://github.com/software-mansion/scarb/issues/717
Scarb supports dev dependencies now (in next version)

Will be unblocked when new Scarb is released

This will make impossible to use call trace and other cheatcodes in contracts

@piotmag769 will this work without plugins?

It will

Closed in #1871


git-init: do not assume master as default branch

As of today, when the user doesn't specific a revision (branch, commit, …), we are defaulting to master. There is some problems with that: master might not be present on the remote. The recent wave of change of default branches on several project, including https://github.com/spring-projects/spring-petclinic are putting this problem into light : see https://github.com/tektoncd/catalog/pull/365.
Instead of making the assumption that we should use master when the revision is empty, we should inspect the remote. git can give us this information.
$ git remote show origin
* remote origin
  Fetch URL: https://github.com/spring-projects/spring-petclinic
  Push  URL: https://github.com/spring-projects/spring-petclinic
  HEAD branch: main
  Remote branches:
    gh-pages  new (next fetch will store in remotes/origin)
    main      new (next fetch will store in remotes/origin)
    wavefront new (next fetch will store in remotes/origin)

After doing the git init and adding the remote git remote add origin …, we should get the default branch (remote HEAD) if the revision isn't specified. The way to do it should be as simple as git symbolic-ref refs/remotes/origin/HEAD :angel:
/kind bug

/assign


Error on load MTConnector.dll in MTApi 1.15 - MT4

Hello,
i built successfully from source code the entire solution in visual studio 2019.
Then i run the MtApiInstaller.msi and, also, the MtApi_Setup.exe in a VPS where i have an instance of MT4.
I followed all the instructions and i put the .ex4 in the right folder and also the other files like "json.mqh", for example, in the include folder.
If i try to insert the Expert in a chart i get the attached error.
If i look at the chart window i can see the smile face for a while and then it disappers and, also, the 8222 port is not open.
Since today i had the 1.14 release installed and everything worked fine (same VPS).
The OS is a Windows 7 Professional edition 64 bit.
Do you have any suggestions, please?
Thank you very much and best regards

May I ask to try install and test pre-built release https://github.com/vdemydiuk/mtapi/releases/tag/MT4-v1.0.43 ?
Will it have the same error for your?

Hi, i had the same problem but with the exe of your link now i can attach mtapi to chart.
The problem now is that i can't connect. The function apiClient_ConnectionStateChanged is never call and the state is disconnected. Call to connect don't throw any exception.
Port of mtapi is setted to 8222.
Where is the problem?
Thanks.
This is the code:
        _apiClient = new MtApiClient();
        _apiClient.ConnectionStateChanged += apiClient_ConnectionStateChanged;

        try
        {
            _apiClient.BeginConnect("<IP_ADDRESS>", 8222);
        }
        catch (Exception ex)
        {
            throw ex;
        }

...................
...................
private void apiClient_ConnectionStateChanged(object sender, MtConnectionEventArgs args)
{
string connectionStaus = args.Status.ToString();
}


Update README.md

missing golang as requirement

It is not a requirement. It is like saying that linux requires gcc. The binaries are built and uploaded with TravisCI into releases and you can download them. golang is required only for development or building from sources.


Menu Entry Swapper: Quick-leave instead of Leave tomb at Pyramid Plunder

Adds a menu entry swapper option for Quick-Leaving instead of Leave tomb at Pyramid Plunder.
Closes #9912

This seems like a bad default, I've accidentally clicked the exits before thinking they were doors.

This seems like a bad default, I've accidentally clicked the exits before thinking they were doors.

You're correct! I will change it so it's not on per default.

added in 5344cd6ddbd0e435bad7a088dbc266738319a5fc


Added test on the velocity operator, and computation of the group velocities from the velocity operator

Thank you for all the work @atztogo . I have implemented the computation of the group velocities from the velocity operator.
RTA solution works perfectly, both at the Wigner and Peierls-Boltzmann levels. I have tested the code on CsPbBr3 (with read-gamma option) and it works perfectly.
Both test_kappa_LBTE.py and test_kappa_LBTE_Wigner.py are failing on my Mac after the manual update (they were failing before this pull request), but they work on GitHub and on a linux system.
I spent some time to look for the problem, but the code has changed considerably compared to the previous version and I haven't found how to solve. Are test_kappa_LBTE.py and test_kappa_LBTE_Wigner.py working on your Mac?

Thanks @MSimoncelli for checking PR #83.

Are test_kappa_LBTE.py and test_kappa_LBTE_Wigner.py working on your Mac?

Yes both on Intel mac and m1 mac.
You should merge develop first before modifying your develop.
https://github.com/MSimoncelli/phono3py/commit/f10605aa52e41d14ed6e3443d0ae51428a610365
Smaller PR is better. So before your struggling, could you contact me more often?
Here, please do not modify .gitignore and do not add unnecessary files in repository,
https://github.com/MSimoncelli/phono3py/commit/b3bdf644656e1c09270ec099fe83f90ac092c4dc
https://github.com/MSimoncelli/phono3py/commit/eb597a41c817ca523ae4785dc241528586ef1b47
https://github.com/MSimoncelli/phono3py/commit/5be50f17abfc69f437514614b6c57aea792e26c3
Perhaps do you use the command git commit  -a (or git add .)? This is not recommended. First find the modified files git status, and specify necessary files to be committed by like git commit file-name1 file-name2. You can revert the changes of files by git commit -- file-name.
I think the test of the velocity operation exists in the develop branch if I understand correctly. Why did you need reintroduction? Maybe I am wrong.
https://github.com/MSimoncelli/phono3py/commit/b775211ec83bbc30415ee7d75dab32198d251ed3
Please do not touch the code before fixing your issue running tests of the phono3py/develop.
https://github.com/MSimoncelli/phono3py/commit/7292ec0b34d5fd2f3fec019d6baafad9a5e35dda
Please run pre-commit hooks before git commit. A short instruction is written README.md here
https://github.com/phonopy/phono3py

Thanks  @atztogo for going thorough all the comments, and for the guidelines about git.
I didn't expect the gitignore and related unnecessary files to sync from my fork to your repo, apologises for that.
Anyway, aside from these git-related issues, I modified and tested the code on a linux system.
The test on the velocity operator was not updated to the last version (committed in phonopy few days ago), and it was skipped in the manual merge. I have updated such test; it is successful on a Linux system, on github, and on my Mac.
I have updated the computation of the group velocity from the velocity operator in the case of degeneracies.
This is needed only when the diagonal elements of the velocity operator are used as group velocities in the direct solution of the population (particle-like) part of the Wigner equation.
This is expected to produce no changes in the RTA conductivity. I tested CsPbBr3 after this modification and indeed found perfectly compatible results (less than 0.001 W/mK in the thermal conductivity). Then ran also the other tests on the conductivity test_kappa_RTA_si_nosym fails with tolerance 0.5. Increasing the test tolerance to 0.8 results in a pass. I have not understood what we are testing here, since I would expect the conductivity tensor of silicon to be diagonal and isotropic, but the reference results for this test in the standard phono3py version (before the Wigner update) are non-diagonal and non isotropic. I notices that in the reference tensor value for this test in the standard phono3py versuion, the difference between the diagonal components of the expected-to-be-diagonal tensor are of about 1 W/mK, thus using 0.8 W/mK as tolerance seems reasonable.
Do not hesitate to let me know if you need any further action from my side!
Thanks, Michele

I understand what you wrote. Just for convenience, could you abandon this PR and recreate it cleanly? I think this is rather easy for you to do so than for me to struggle on merging it. The points are as I wrote above, e.g., first merging phono3py/develop to the point where you started to work on this PR or clone phono3py repository and start working on it. If possible, the later is better. You tend to write comments (lines starting #) inside code, but often the code is updated but not the comment, then the code and comment start to disagree. So it is recommended to write minimum comments. If we need to explain something, it is good to think either if it is written as comment line or in docstring. Please run pre-commit before the commit as explained above. In addition, you probably work on your develop branch of your phono3py repository. But it is good to make another branch by
% git checkout -b topic_branch_name

In this way, it becomes easier to go back to the point where you started working on the parent code and to compare with other topic branches. Git is designed to use branches to communicate between developers in collaboration.
One question. I see your update
gv = np.einsum("iij->ij", gv_operator).real

Do you assume the first two dimensions have the same size? And is it truer?

Got it, thanks for the explanations. I'll discard this PR and do a single PR.
Concerning your question, the answer is yes.
At fixed q-point and cartesian direction, the velocity operator is a square matrix of size (3N_atoms_in_primitive_cell)x(3N_atoms_in_primitive_cell).
In absence of degeneracies, the group velocities at fixed q-point and at fixed direction are simply the diagonal elements of the aforementioned square matrix. If degeneracies are present, the group velocities are obtained diagonalising the velocity operator in the degenerate subspace. Hope this fully answers your questions, do not hesitate to get in touch if needed.

I'll discard this PR and do a single-commit PR.

Thanks.
About gv, I understand you are correct theoretically. But your gv_operator is allocated as
https://github.com/phonopy/phono3py/blob/993f10781dfc06433db081e2228d9a04f4986172/phono3py/conductivity/rta.py#L809-L811
Here, num_band0 can be different from nat3. num_band0 != nat3 when --bi option is specified.

Ok I see what you mean. I'll take care of that


Build error 2

I get this error when trying to build. Iv tried changing Openssl version and Cpprestsdk version in dependencies CMakeLists. Do you have a solution?
make[2]: *** No rule to make target ../libs/cpprestsdk/build.release/Binaries/libcpprest.a', needed by micro-service'.  Stop. make[1]: *** [CMakeFiles/micro-service.dir/all] Error 2 make: *** [all] Error 2

Hi Iskers,
It looks like you have not build the C++ REST SDK, please leave the build_dependencies as is, and please verify the path:
../libs/cpprestsdk/build.release/Binaries/

it should contain the file libcpprest.a, e.i.:
$ ls ../libs/cpprestsdk/build.release/Binaries

you should see this output:
$ ls ../libs/cpprestsdk/build.release/Binaries
libcpprest.a

if that file is not there then that's your problem, the C++ REST SDK is not build.
Thanks,
Ivan.

The file is definetly there in my case, I am still getting the same error. It fails on step 5, and says
"""
make[2]: *** No rule to make target /usr/local/Cellar/openssl/1.0.2p/lib/libssl.1.0.0.dylib', needed by micro-service'.  Stop.
make[1]: *** [CMakeFiles/micro-service.dir/all] Error 2
make: *** [all] Error 2
"""
Any ideas? Couldn't find anything in the comments above. Thanks!

Same error, here, after running 'make -j 8"
running macOS 10.14.3

you have to install OpenSSL, If you ar on macOS type:
$ brew install cmake git openssl boost zlib

That workaround fixed it for me. Thank you so much

Thanks, I'm glad it helped.

Hello Ivan,  good morning.
Machine: Ubuntu 18.04 X86 AMD
libcpprest.a is created by using ./build_dependencies.sh
cmake is done in micro-service/build$
during make -j 8  the following error msg coming
undefined reference to `deflate'
Could you kindly advice
Thanks a lot in advance
jk


common: argon_provider - add flag for key's security level

Description
Add keySecLevel flag to allow for greater security levels to be configured (2 is default, 1-3 is available)
Type of change

Enhancement (a non-breaking change which adds functionality)

Checklist:

[ ] My code follows the style guidelines for this project
[ ] I have performed a self-review of my own code
[ ] I have commented my code, particularly in hard-to-understand areas
[ ] I have made corresponding changes to the documentation
[ ] New and existing unit tests pass locally with my changes

#202 already did it