Add model interface support for remote and local custom models

Description
Adds support for the interface field of model registration. Currently only adding support for the register_remote_model and register_local_custom_model since pretrained/ local sparse encoding models do not currently work with the model interface.
Registering and deploying remote model with model interface :
curl -i -XPOST "localhost:9200/_plugins/_flow_framework/workflow?provision=true" -H "Content-Type:application/json" --data '{"name":"createconnector-registerremotemodel-deploymodel","description":"test case","use_case":"TEST_CASE","version":{"template":"1.0.0","compatibility":["2.12.0","3.0.0"]},"workflows":{"provision":{"nodes":[{"id":"workflow_step_1","type":"create_connector","user_inputs":{"name":"OpenAI Chat Connector","description":"The connector to public OpenAI model service for GPT 3.5","version":"1","protocol":"http","parameters":{"endpoint":"api.openai.com","model":"gpt-3.5-turbo"},"credential":{"openAI_key":"12345"},"actions":[{"action_type":"predict","method":"POST","url":"https://${parameters.endpoint}/v1/chat/completions"}]}},{"id":"workflow_step_2","type":"register_remote_model","previous_node_inputs":{"workflow_step_1":"connector_id"},"user_inputs":{"name":"openAI-gpt-3.5-turbo","description":"test model","interface":{"input":{"properties":{"parameters":{"properties":{"messages":{"type":"string","description":"This is a test description field"}}}}},"output":{"properties":{"inference_results":{"type":"array","items":{"type":"object","properties":{"output":{"type":"array","items":{"properties":{"name":{"type":"string","description":"This is a test description field"},"dataAsMap":{"type":"object","description":"This is a test description field"}}},"description":"This is a test description field"},"status_code":{"type":"integer","description":"This is a test description field"}}},"description":"This is a test description field"}}}},"deploy":true,"node_timeout":"15s"}}]}}}'
HTTP/1.1 201 Created
X-OpenSearch-Version: OpenSearch/3.0.0-SNAPSHOT (opensearch)
content-type: application/json; charset=UTF-8
content-length: 38

{"workflow_id":"gNg7O48BaP6uKfWC1aKh"}

curl -i -XGET "http://localhost:9200/_plugins/_ml/models/hNg7O48BaP6uKfWC2qJv?pretty"
HTTP/1.1 200 OK
X-OpenSearch-Version: OpenSearch/3.0.0-SNAPSHOT (opensearch)
content-type: application/json; charset=UTF-8
content-length: 1318

{
  "name" : "openAI-gpt-3.5-turbo",
  "model_group_id" : "gtg7O48BaP6uKfWC2aJg",
  "algorithm" : "REMOTE",
  "model_version" : "1",
  "description" : "test model",
  "model_state" : "DEPLOYED",
  "created_time" :<PHONE_NUMBER>273,
  "last_updated_time" :<PHONE_NUMBER>735,
  "last_deployed_time" :<PHONE_NUMBER>734,
  "auto_redeploy_retry_times" : 0,
  "planning_worker_node_count" : 1,
  "current_worker_node_count" : 1,
  "planning_worker_nodes" : [
    "TpCVBoolSOCJOR4dkq66ng"
  ],
  "deploy_to_all_nodes" : true,
  "is_hidden" : false,
  "connector_id" : "gdg7O48BaP6uKfWC2KIR",
  "interface" : {
    "output" : "{\"properties\":{\"inference_results\":{\"description\":\"This is a test description field\",\"type\":\"array\",\"items\":{\"type\":\"object\",\"properties\":{\"output\":{\"description\":\"This is a test description field\",\"type\":\"array\",\"items\":{\"properties\":{\"name\":{\"description\":\"This is a test description field\",\"type\":\"string\"},\"dataAsMap\":{\"description\":\"This is a test description field\",\"type\":\"object\"}}}},\"status_code\":{\"description\":\"This is a test description field\",\"type\":\"integer\"}}}}}}",
    "input" : "{\"properties\":{\"parameters\":{\"properties\":{\"messages\":{\"description\":\"This is a test description field\",\"type\":\"string\"}}}}}"
  }
}

For registering and deploying local custom model :
curl -i -XPOST "localhost:9200/_plugins/_flow_framework/workflow?provision=true" -H "Content-Type:application/json" --data '{"name":"registermodelgroup-registerlocalmodel-deploymodel","description":"test case","use_case":"TEST_CASE","version":{"template":"1.0.0","compatibility":["2.12.0","3.0.0"]},"workflows":{"provision":{"nodes":[{"id":"workflow_step_1","type":"register_local_custom_model","user_inputs":{"node_timeout":"60s","name":"all-MiniLM-L6-v2","version":"1.0.0","description":"test model","model_format":"TORCH_SCRIPT","function_name":"SPARSE_TOKENIZE","model_content_hash_value":"c15f0d2e62d872be5b5bc6c84d2e0f4921541e29fefbef51d59cc10a8ae30e0f","model_type":"bert","embedding_dimension":"384","framework_type":"sentence_transformers","all_config":"{\"_name_or_path\":\"nreimers/MiniLM-L6-H384-uncased\",\"architectures\":[\"BertModel\"],\"attention_probs_dropout_prob\":0.1,\"gradient_checkpointing\":false,\"hidden_act\":\"gelu\",\"hidden_dropout_prob\":0.1,\"hidden_size\":384,\"initializer_range\":0.02,\"intermediate_size\":1536,\"layer_norm_eps\":1e-12,\"max_position_embeddings\":512,\"model_type\":\"bert\",\"num_attention_heads\":12,\"num_hidden_layers\":6,\"pad_token_id\":0,\"position_embedding_type\":\"absolute\",\"transformers_version\":\"4.8.2\",\"type_vocab_size\":2,\"use_cache\":true,\"vocab_size\":30522}","url":"https://artifacts.opensearch.org/models/ml-models/huggingface/sentence-transformers/all-MiniLM-L6-v2/1.0.1/torch_script/sentence-transformers_all-MiniLM-L6-v2-1.0.1-torch_script.zip","interface":{"input":{"properties":{"text_docs":{"type":"array"},"return_number":{"type":"boolean"},"target_response":{"type":"array"}}}}}},{"id":"workflow_step_2","type":"deploy_model","previous_node_inputs":{"workflow_step_1":"model_id"}}],"edges":[{"source":"workflow_step_1","dest":"workflow_step_2"}]}}}'
HTTP/1.1 201 Created
X-OpenSearch-Version: OpenSearch/3.0.0-SNAPSHOT (opensearch)
content-type: application/json; charset=UTF-8
content-length: 38

{"workflow_id":"lch-O48BUXkJrbtALpC3"}

curl -i -XGET "http://localhost:9200/_plugins/_ml/models/mMh-O48BUXkJrbtAMZCk?pretty"
HTTP/1.1 200 OK
X-OpenSearch-Version: OpenSearch/3.0.0-SNAPSHOT (opensearch)
content-type: application/json; charset=UTF-8
content-length: 1687

{
  "name" : "all-MiniLM-L6-v2",
  "model_group_id" : "lsh-O48BUXkJrbtAMJCA",
  "algorithm" : "SPARSE_TOKENIZE",
  "model_version" : "1",
  "description" : "test model",
  "model_format" : "TORCH_SCRIPT",
  "model_state" : "DEPLOYED",
  "model_content_size_in_bytes" : 91790008,
  "model_content_hash_value" : "c15f0d2e62d872be5b5bc6c84d2e0f4921541e29fefbef51d59cc10a8ae30e0f",
  "model_config" : {
    "model_type" : "bert",
    "embedding_dimension" : 384,
    "framework_type" : "SENTENCE_TRANSFORMERS",
    "all_config" : "{\"_name_or_path\":\"nreimers/MiniLM-L6-H384-uncased\",\"architectures\":[\"BertModel\"],\"attention_probs_dropout_prob\":0.1,\"gradient_checkpointing\":false,\"hidden_act\":\"gelu\",\"hidden_dropout_prob\":0.1,\"hidden_size\":384,\"initializer_range\":0.02,\"intermediate_size\":1536,\"layer_norm_eps\":1e-12,\"max_position_embeddings\":512,\"model_type\":\"bert\",\"num_attention_heads\":12,\"num_hidden_layers\":6,\"pad_token_id\":0,\"position_embedding_type\":\"absolute\",\"transformers_version\":\"4.8.2\",\"type_vocab_size\":2,\"use_cache\":true,\"vocab_size\":30522}"
  },
  "created_time" :<PHONE_NUMBER>965,
  "last_updated_time" :<PHONE_NUMBER>895,
  "last_registered_time" :<PHONE_NUMBER>144,
  "last_deployed_time" :<PHONE_NUMBER>894,
  "auto_redeploy_retry_times" : 0,
  "total_chunks" : 10,
  "planning_worker_node_count" : 1,
  "current_worker_node_count" : 1,
  "planning_worker_nodes" : [
    "zTtalEMSTRGrYfx29Gh4sQ"
  ],
  "deploy_to_all_nodes" : true,
  "is_hidden" : false,
  "interface" : {
    "input" : "{\"properties\":{\"target_response\":{\"type\":\"array\"},\"return_number\":{\"type\":\"boolean\"},\"text_docs\":{\"type\":\"array\"}}}"
  }
}

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Instead of relaying on customer to escape strings and having to figure out how to give that user input we can use ParseUtils.parseArbitraryStringToObjectMapToString(configurationsMap); like we do for create index and pipelines so we do it for them.

Instead of relaying on customer to escape strings and having to figure out how to give that user input we can use ParseUtils.parseArbitraryStringToObjectMapToString(configurationsMap); like we do for create index and pipelines so we do it for them.

ignore this comment :)


Allow using collection accessors on strings

@djbe updated

In the docs, I'd change (https://github.com/stencilproject/Stencil/blob/master/docs/templates.rst#variables):

Array lookup (first, last, count, index)

To

Collection lookup (first, last, count, index) for Array and String


Network: Design, test and automate Netmask tests

Design and automate Netmask tests

Done:
https://gerrit.openbmc-project.xyz/#/c/openbmc/openbmc-test-automation/+/22136/


How to use Libertea by Nginx

I have two sites running on Nginx, each on a different subdomain. However, when installing Libertea, Nginx prevents it from running, showing the following error:
ERROR: Your panel domain [example.domain.com](http://example.domain.com/) is not accessible.
       Please ensure that your domain's DNS is pointing to the server's IP (***.***.***.***).

Turning off Nginx resolves the issue, but I'm unable to do so. What should I do in this situation ?

Run your nginx on another port (like 8080), and set Libertea's camouflage domain to <IP_ADDRESS>:8080.
More info


Out of memory issue during training and testing

My env setting is Pytorch 0.4.0 + CUDA 9.0, I disabled the CUDNN batch norm. However, it seems that there is a GPU memory leak. Does anyone encounter this problem?

Problem solved after switching to another server ...


[Map] DM-Coagulate

Hash: cc163f0680caee20fd93e5f93679d8d07ed94e60
Current name: DM-Coagulate
The author of this version isn't Epic Games.  The readme indicates that this map uses a PS2 version by Ken "DrunKenobi" Lynch as a base - no author is given for the revisions to produce this current version.

Updated to "Ken "DrunKenobi" Lynch"


Option to make Non-Transferable Tokens (NTT)

Non-transferability of tokens can be useful when the token represents achievements, certifications, events, etc. that the owner is related to. If the token can be sold/bought, it becomes meaningless.
The current token program doesn't seem to provide this type of functionality.
I wonder what you all think about adding a "non-transferable" flag to the token initializer function.
Related issue on Ethereum: https://github.com/ethereum/EIPs/issues/1238
Related blog post: https://vitalik.ca/general/2022/01/26/soulbound.html

This sounds like a great way to spam a user's wallet with advertisements/scams that they can't get rid of.  This is already happening regularly but at least the victims can currently dispose of the token.

I actually think this has utility to create soulbound tokens which have some interesting uses (decentralized identity comes to mind).
Could you make it so the token can be burned but not transfered? This would help with the spam issue.

This sounds like a great way to spam a user's wallet with advertisements/scams that they can't get rid of. This is already happening regularly but at least the victims can currently dispose of the token.

Valid concern. But in this case, the non-transferable tokens could still be burnt.

Wallets need to support burning tokens first.

Draft PR: https://github.com/solana-labs/solana-program-library/pull/2912

FYI @joxoby we have been exploring a similar thing using a program we call the "token_manager" https://github.com/cardinal-labs/cardinal-token-manager
If you are the mint creator, can you not just utilize freeze authority to freeze everytime you mint someone a token? Also I feel like non-transferability could be modeled more flexibly with a transfer_authority: COption<Pubkey>, we have done this in our program by requiring a transfer_authority to create a "transfer_receipt" that someone can use to transfer the token to the approved party. Would be a cool feature in token-program and also solve non-transferability / soul tokens by just setting transfer_authority to None

Added with #3178

Hello
Can I solve the non-transferability without any contract?

I was gonna develop the passport mint website and I used metaplex.
Now I need to do non-transferability of nft.
Do I have to develop the smart contract?

This is off-topic, but to quickly answer -- the feature is available on token-2022, but has not been released to mainnet yet.  Also, Metaplex is not using token-2022 at the moment.  If you need this soon, then you will have to develop your own on-chain program.


Dependency update: prompt-toolkit

What current issue(s) does this address, or what feature is it adding?
Updated the code to work with the new version of prompt-toolkit, which in v2.x had a few breaking changes, in particular around the Style and the prompt function signature: https://github.com/jonathanslenders/python-prompt-toolkit/blob/master/CHANGELOG.
See also:

https://github.com/jonathanslenders/python-prompt-toolkit/blob/deb7488b7b24814ae12518639a2fbe0b67c43ec3/examples/print-text/print-formatted-text.py
https://github.com/jonathanslenders/python-prompt-toolkit/blob/deb7488b7b24814ae12518639a2fbe0b67c43ec3/examples/prompts/bottom-toolbar.py

How did you solve this problem?
Updating the code based on the prompt-toolkit examples.
The only thing I could not migrate was the bottom toolbar styling. It seems the new version doesn't allow foreground colors in the bottom toolbar anymore: https://github.com/CityOfZion/neo-python/blob/deps-prompt/neo/bin/prompt.py#L171
It looks like this now:

How did you make sure your solution works?
Manual tests
Are there any special changes in the code that we should be aware of?
No, except the changed bottom toolbar styling
Please check the following, if applicable:

[ ] Did you add any tests?
[X] Did you run make lint?
[X] Did you run make test?
[X] Are you making a PR to a feature branch or development rather than master?
[ ] Did you add an entry to CHANGELOG.rst? (if not, please do)

Looks good! I'm ok with the different bottom toolbar styling-

Actually, after playing around with this for a bit I'm noticing some odd things such as:  <prompt_toolkit.styles.style.Style object at 0x10fc36f60> being printed after any command using FormattedText.  Will revert for now


Fix register.lua: moved functions to table minetest

Some functions were defined globally instead of inside minetest.
Also no, I don't have rights to push to organization's repo, so doing this way.

@herrscher-of-sleeping As I said, as a member you should have direct access to this repository. Please feel free to clone it and make your edits directly. Even if you make serious mistakes in some definitions, it will be easier for me to fix your mistakes than to write everything from scratch.  Thanks :3!

@AFCMS Bonjour! Can you please give the necessary rights for @herrscher-of-sleeping?

@herrscher-of-sleeping I made you the owner of this organization, so now you should have all possible rights. I'm not sure if this is the right way to transfer rights, but since @AFCMS is not responding, I have no better solutions.

Sorry, didn't see your message until today.
You did it the right way.


[ETL-309] add ci service acct for recover

Purpose:
The AWS accounts recover-dev and recover-prod needs service account users for CI. This adds it.
PR Checklist:
[X] Describe and explain your intentions for this change
[X] Setup pre-commit and run the validators (info in README.md)

This looks good to me.  I defer to @brucehoff / @zaro0508 for final review.

@rxu17 I don't think a service account is needed to allow GitHub to deploy to the Recover AWS account.  Rather what's needed is OIDC integration with an IAM role that a certain GitHub repo' can assume.  @zaro0508 has worked out how to do this for IT deployments and can give more guidance.

@brucehoff / @zaro0508, Since there is a January 27th deadline to set up the pilot ETL, I would like this or the OIDC integration to be worked out ASAP as this is currently semi-blocking our work. (I say semi, because we can technically deploy using using our admin jumpcloud privileges, but ideally we just use CI/CD)
Does it make most sense to set up an hour meeting to work through the steps of OIDC integration?  Also, would OIDC integration work on multiple accounts?  We need to be able to deploy in both the dev and prod account?

please try setting up with github OIDC access.  example can be found at https://github.com/Sage-Bionetworks-IT/organizations-infra/blob/master/org-formation/650-identity-providers/_tasks.yaml

Sounds good. Added new PR #674 with the updates


Compact View!

Added Compact View -- toggle only affects the Thumbnails view (at least for now) 😎

note that preview.component.html is 99% just whitespace change ✅

Misc CSS & code cleanup as well 👍
Updates colors to be lighter than before for the gallery and sidebar ☀
Addresses a lot of #102 tasks 👍

I'll be doing more cleanup, but would rather create a new PR to keep PRs relatively small 👍

Really looking forward to trying this out! 🎉


rename ratio fields

Coverage decreased (-0.08%) to 97.412% when pulling 58c32778f49601b05eeb50bd28640b64e230c0eb on feature/rename-ratio-fields into 24f4bbe18074ee6188a6ef8668e20b2d340bfbd8 on develop.

@tetianakh could you take a look?


Sample project for .Net with SQL on app service needs a nugget dependency

The sample project on this page:
https://docs.microsoft.com/en-us/azure/app-service/app-service-web-tutorial-dotnet-sqldatabase
Only works if roslyn is installed.
I solved it with
Menu -> Tools -> NuGet Package Manager -> Package Manager Console
Shell: Install-Package Microsoft.CodeAnalysis

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 72d855c6-51fd-a1da-5c44-8b44658c0211
Version Independent ID: 15c87b07-e89e-2b0c-6337-3940a5259101
Content: Build ASP.NET app with SQL Database - Azure App Service
Content Source: articles/app-service/app-service-web-tutorial-dotnet-sqldatabase.md
Service: app-service-web
GitHub Login: @cephalin
Microsoft Alias: cephalin

@meller Thanks for bringing this to our attention! I've assigned this issue to the content author to evaluate and update as appropriate.

@Meller: Thanks for the share

@meller Unfortunately we have not been able to reproduce this behavior even though this is not the first time we heard about this issue. It may have to do with the type of development suites you have in the VS installer. If you happen to have a screenshot of the development suites for your VS installation, please show us what's installed. Thank you.
#please-close


Initial draft of cutome route table

Hi Anton,
I have started the development of a custom route table for the show and hide the columns. Please review and suggest changes.

Hi Anton,
Done some changes. Please review.

Hi Anton,
Done some changes.
How should I define the minimum width for the column so that it can completely visible on the screen?

Hi Anton,
Incorporated your suggestions. Please review.

Hi Anton,
Code formatting is done.
flex: 1 -> Could not find any way to set it to 1 by default.
https://material-ui.com/components/data-grid/columns/#fluid-width

Hi Anton,
Changes are done. Please review.

Hi Anton,
I have tried one more refactoring. Please review.

Hi Anton,
Please check now.

Created separate methods for coordinate formatting. please check and review.

Done. Please check.

Merged. Thanks for fixing all the comments.

Thank you.
Should I replace tables in all the reports with DataGrid?

Makes sense to me.

ok. Will open a separate PR next week.


[W5][W10-4]Kerryn Eer

Add a new sort command to sort names by alphabetical order
Ready for review

Hi Kerryn, everything looks good for this PR. Very thorough updates for both I/O and JUnit testing

Hi kerryn, very nice step by step commits that document your journey in implementing this new command. Keep it up !


Alarm API

The use cases document mentions this.
There is http://www.w3.org/TR/2013/WD-web-alarms-20130205/ (latest version link is broken) written by @cdumez.
The web obviously needs something like this so we can have https://clock.org/ and https://calendar.org/ and https://reminders.org/ all be applications that have the same capabilities as native.
The question is however whether background synchronization is the right place to address this. Or whether that should be a similar API but with a distinct opt in.

They would certainly have a very similar API, but I don't think that's sufficient reason to make them part of the same platform feature.
Is it better for the web if the developers's model for background sync is always "These events are coalesced, and you can't guarantee that they're even going to happen"?
(The question is important -- the exact alarms use cases are certainly on the list and part of the reason for that is to consider whether they're even in scope for this API)

Seems like we're saying the same thing. For what it's worth, given that alarms require precision and require to be activated at the precise moment I'm inclined to think they are different from synchronization.
Main thing I'm struggling with is UX.

I've always though of them as being a separate api too.
If we tackle one-off sync first, then vague sync, then precise, we can
tackle the UX starting easiest-first
On Wed, 11 Feb 2015 19:07 Anne van Kesteren<EMAIL_ADDRESS>wrote:

Seems like we're saying the same thing. For what it's worth, given that
alarms require precision and require to be activated at the precise moment
I'm inclined to think they are different from synchronization.
Main thing I'm struggling with is UX.
—
Reply to this email directly or view it on GitHub
https://github.com/slightlyoff/BackgroundSync/issues/38#issuecomment-73942779
.

Agree, alarms are for another API. Closing the issue.

Shouldn't the use cases document be amended first in this case? This issue was filed for a reason...


Add angle bracket whitespace rule

Fixes #580
Adds a lint for extra whitespace around type parameters which resolves the previously added failing tests.

@Tapchicoma want to take a look at this one? Feel free to merge it in if it looks good to you.

Just popped into my mind, what if the argument list is huge and we want to split the types per line? E.g., this would work with the current implementation:
ClassWithGenerics<String,
    Int,
    Boolean,
    ...,
    Boo>

But this would already fail:
ClassWithGenerics<
    String,
    Int,
    Boolean,
    ...,
    Boo
>

So if you want to allow the latter case, then we need to add another check for a newline.

@rom4ek added valid point about covering writing type parameters per line. I've checked kotlin and android style guides and don't found anything there. IDEA allows following formatting:
class AngleTest<
    S,
    G,
    Z,
    B
    > {

}

Also would be nice to add tests for nested angle brakets cases, for example:
class AngleTest<B : Map<Int, List<String>>>

Sounds good on all counts. Will update.

@christinalee hi, any updates on this PR?

Closing this in favour of #769


JWTResponseListener / Request is null

Hello everyone,
I just updated bundle, and my listener which was previously working doesn't anymore.
I ran onto issue #128, which gave me the info that Request system has been changed in 1.4.
Here's a snipped of my listener :
public function onAuthenticationSuccessResponse(AuthenticationSuccessEvent $event)
{
       $data    = $event->getData();
        $user    = $event->getUser();
       $request = $event->getRequest();

$request used to be filled, suddenly gone to NULL after update.
So, tracked down to this point :
Security\Http\Authentication\AuthenticationSuccessHandler.php : 44
 public function onAuthenticationSuccess(Request $request, TokenInterface $token)
{
         $user = $token->getUser();
         $jwt  = $this->jwtManager->create($user);

         $response = new JsonResponse();
         $event    = new AuthenticationSuccessEvent(['token' => $jwt], $user, null, $response);

Changed to :
public function onAuthenticationSuccess(Request $request, TokenInterface $token)
{
         $user = $token->getUser();
         $jwt  = $this->jwtManager->create($user);

         $response = new JsonResponse();
         $event    = new AuthenticationSuccessEvent(['token' => $jwt], $user, $request, $response);

solved the problem. However, I feel (very) uncomfortable about this fix : as stated everywhere in your code, this is deprecated and soon removed; and I really don't want a custom version of a bundle (eek.)
So, I was wondering if :
A°) "It's not a bug, it's a feature" ? (or is it, really ?)
B°) If it's a feature, I didn't see anywhere in docs a way to get back my request data in my response listener ?
I didn't found either any closed issue on this topic, am I the only one ? (really ?)
Juste one last thing : it's my first issue opened on Github... please excuse me if something is wrong with it (any advice is appreciated !)
Anyway : thanks for your bundle !
( Edit : had some troubles with code snipped markup... switched to "quote" one. )

@Wonsy First, thank you very much for opening this issue, even more if it is your first opened issue on github.
Thanks to this issue, I realized we introduced a BC break in #200.
It should be fixed now. Could you please try to update the bundle (the fix being tagged as `v1.7.1') and confirm that all work as before from your side?

So, I was wondering if :
A°) "It's not a bug, it's a feature" ? (or is it, really ?)
B°) If it's a feature, I didn't see anywhere in docs a way to get back my request data in my response listener

Well, it's indeed a feature at first, which removed a bad practice from our code:
You should consider to avoid retrieving the Request object from the event itself.
Explanation:
Since symfony 2.4, it's deprecated to inject the @request service and a bad practice to inject some Request instances in your classes.
The reason is that a RequestStack class and its @request_stack service have been introduced in the same time, representing the right way to access the current request from anywhere in your code.
In the 1.x releases  of this bundle, pretty much all Event classes take a Request instance as constructor argument, allowing users to use this request in their listeners, as you are doing.
In 2.0 we grabbed the opportunity to avoid this practice by removing those Requestinjections from ourEventandEventListener` classes (and some other).
Here is what you need to do for avoiding this practice (and btw have a smooth upgrade to the 2.0 version of this bundle which should be released this week, with a bunch of nice new features).
Before
services:
   app.jwt_authentication_success_listener:
       class: AppBundle\EventListener\JWTAuthenticationSuccessListener
       tags:
           - { name: kernel.event_listener, event: lexik_jwt_authentication.on_authentication_success, method: onAuthenticationSuccessResponse }

use Lexik\Bundle\JWTAuthenticationBundle\Event\AuthenticationSuccessEvent;

class JWTAuthenticationSuccessListener
{
    public function onAuthenticationSuccessResponse(AuthenticationSuccessEvent $event)
    {
        $request = $event->getRequest();
        // ...
    }
}

After
services:
    app.jwt_authentication_success_listener:
        class: AppBundle\EventListener\JWTAuthenticationSuccessListener
        arguments: [ '@request_stack' ]
        tags:
            - { name: kernel.event_listener, event: lexik_jwt_authentication.on_jwt_created, method: onJWTCreated }

use Lexik\Bundle\JWTAuthenticationBundle\Event\AuthenticationSuccessEvent;
use Symfony\Component\HttpFoundation\RequestStack;

class JWTAuthenticationSuccessListener
{
    private $requestStack;

    public function __construct(RequestStack $requestStack)
    {
        $this->requestStack = $requestStack;
    }

    public function onAuthenticationSuccessResponse(AuthenticationSuccessEvent $event)
    {
        $request = $this->requestStack->getCurrentRequest();
    }
}

This way you can get the same Request instance from many event listeners, just need to inject the @request_stack service and call RequestStack::getCurrentRequest().
You can see all the upgrade path for 2.0 in the UPGRADE.md file of the 2.0 branch of this bundle, which will become master when released.
I hope you can easily make the change in your code, in order to have a smooth upgrade to 2.0 as soon as this one will be available.
Please do not hesitate to give feedbacks, it will be really appreciated. And if this solve your problem, please close the issue :).
Again, thank you.

Closing as fixed


MetaBalancer: add mechanism to inform user of AtSync call that resulted in actual load balancing

Original author: Nikhil Jain
Original issue: https://charm.cs.illinois.edu/redmine/issues/262

Per discussion with Prof Kale and Jonathan, we need to add a mechanism to make user aware of whether load balancing was done by RTS in the recently invoked AtSync. One solution that seemed feasible was to overload ResumeAtSync with a function parameter for the same. This overloaded ResumeAtSync in turn can invoke ResumeFromSync (without parameter) so that the existing code base works.

Original date: 2017-04-11 20:54:30

The AMPI use case can be covered using ckJustMigrated: ~~https://charm.cs.illinois.edu/gerrit/#/c/2200/~~ https://github.com/UIUC-PPL/charm/commit/24e8f59b5afc40bd4f51ae8190eb0ed1325b2a28


Migrate project to TypeScript

What is this feature about? (1 sentence)
Migrating project to TypeScript
Why is it needed? What is the value? For whom do we build it?
Currently we use flow.js as static type checker. As project grew, it became a mess:

it's not widely used, so we have a lot of missing type definitions
error messages are not helpful
a lot of noise is added to the code editor
it's not doing the thing it meant to do: helping us scale the project and preventing errors

With TypeScript we can get:

it's widely used and the popularity is growing, almost every package offers type definitions now
better code editor support
we can use latest javascript features
more joy and happiness as developers

High-level overview of the feature
IMO it can be done two ways: fast or slow.
Fast:
add typescript, enable loose compiler mode, replace missing types with any
Slow:
add typescript, enable strict compiler mode, fix errors until the project compiles, we can also use some existing tools (flow-to-ts)
Helpful links

https://davidgomes.com/porting-30k-lines-of-code-from-flow-to-typescript/
https://github.com/Khan/flow-to-ts
https://medium.com/inato/migrating-from-flow-to-typescript-why-how-worth-it-5b7703d12089

@lukasschor, @mikheevm, @pablofullana
What you think about implementing this issue in the next sprint?
We've discussed it with @nicosampler and @Agupane and we estimate it won't take more than one day, and conflicts will be around typing, nothing else.

@fernandomg i would like to include it in the next sprint but I’m not sure about the priorities :)

I think it's fine if we include this in the next sprint. Considering the estimate of 5 is still valid. :)

Implementation could be an estimation of 2 o 3. Then we will need to test everything. So a 5, including testing seems reasonable to me.

Implementation could be an estimation of 2 o 3. Then we will need to test everything. So a 5, including testing seems reasonable to me.

This depends on what we mean by "Migrate project":

Make it compile with TypeScript? This is definitely 3
Make it work? This is more like 5
All the above + updating the types? This is 5 + 3 atleast 😄

I already did a POC. It would be great if you can take a look. https://github.com/gnosis/safe-react/pull/737


Support renaming functions

We do this by adding a func_id field that is used to determine the
backend name and that is shared between all functions sharing a name.

I don't like the func_id name. I'd suggest backend_name instead.


[java-client] ConnectionHandler: Log stack trace instead of printing

https://github.com/apache/incubator-pulsar/blob/2bc48129bfb4433182f1b05a9008faef76e387ac/pulsar-client/src/main/java/org/apache/pulsar/client/impl/ConnectionHandler.java#L74
The pulsar java client is currently very noisy in case of connection loss and it cannot be muted, as the stack trace is printed with .printStackTrace(). Moving this to the logging system will allow developers to configure this behavior.

@thrathschlag thanks for reporting. This printout was indeed committed by mistake when fixing a previous issue.


Fix error message for min version

With min_version = 0.50 and Hugo version < 0.50, the error received is in the form:
ERROR 2018/11/03 17:56:00 ACADEMIC theme does not support Hugo version 0.49.2. Minimum version required is 0.5
The 0.5 in the error message should read 0.50. Obviously there's a big difference between minor version 5 and 50.
I have checked and this little bug still exists in the latest version (Hugo v.50) - for example, to test, set min_version = 0.60 in a theme.toml.

The 0.5 in the error message should read 0.50. Obviously there's a big difference between minor version 5 and 50.

That is true, but I think most people get the real message.

I have looked at the code. And that value is read from theme.toml, so if you say:
min_version=0.50

Which I suspect is the case here, that is a float == 0.5.
What you should do is:
min_version="0.50"

I don't think we can guess here.

@bep Thanks, I assumed that Hugo would force cast min version to string. I'll update the theme code :)

This is perhaps more a docs issue then. All the docs I have seen in the past seem to imply that a float should be used. For example, https://gohugo.io/getting-started/code-toggle/#another-config-toggler

This is perhaps more a docs issue then. The docs imply that a float should be used: https://gohugo.io/contribute/themes/#create-a-theme-toml-file

I assumed that Hugo would force cast min version to string.

We do, but how do you convert 1.5 to "1.50" without making assumptions?

An added note: Travis (the build server we use) had a similar issue when Go version rounded "1.10". They accept a float, but "Go 1.1" was not a good choice.

FYI the README in the hugoThemes repo also states that min_version should be provided as a float rather than string: https://github.com/gohugoio/hugoThemes


fMRIprep not starting (hanging with no errors after run command) on HCP using Singularity

What version of fMRIPrep are you using?
20.2.3
What kind of installation are you using? Containers (Singularity, Docker), or "bare-metal"?
Singularity 3.6.0. Used singularity build fmriprep_latest.simg docker://nipreps/fmriprep:20.2.3
What is the exact command-line you used?
singularity run --cleanenv -B /dcl01/smart/data/hpc900:/dcl01/smart/data/hpc900 fmriprep_latest.simg /dcl01/smart/data/hpc900/disk1_bids /dcl01/smart/data/hpc900/bwinston/derivatives participant --fs-license-file /dcl01/smart/data/hpc900/bwinston/license.txt --output-spaces anat --participant-label 103818

Have you checked that your inputs are BIDS valid?
Yes. Actually, I used the same command on a local linux machine, and everything ran fine.
Are you reusing previously computed results (e.g., FreeSurfer, Anatomical derivatives, work directory of previous run)?
No.
Issue
Hi :) trying to run fMRIprep on a cluster. I requested 200G of memory, so that shouldn't be the issue. After I run the above singularity run command, there is no terminal output. fMRIprep creates a work directory; inside is bids_db (inside that is layout_index.sqlite). But that's it. Otherwise hanging. I searched the issues but wasn't able to find anyone with this problem. I'm quite perplexed: any ideas on what could be going on?
Big thanks.

Thanks for the detailed report!
My first guess would be that your working directory is on a network filesystem, which can dramatically slow down the indexing process. If you set --bids-database-dir to a directory that is local to the compute node, that should recover the speed. It doesn't need to be a large filesystem, though if you have 200GB you could probably do it on a tmpfs and get peak performance.

Thanks! I will reach out the sysadmin to see about storing files on a compute node, but this might be hard since I'm hoping to run a bunch of subjects in parallel on different nodes, and I will not always have 200GB of memory. Do you have a ballpark idea of how long indexing would take if I left things as-is? Anything on the order of hours would be fine since I just have to run this once on all the subjects, but I don't want to be waiting for weeks!
Thanks again.

Thanks! You were absolutely right. Hate to be that guy, but I'm now experiencing a perplexing BIDS issue that is totally unrelated. Pls let me know if I should open a separate issue. I'm getting a quick validation error as seen below:
[compute-066 /users/bwinston]$ singularity run --cleanenv -B /dcl01/smart/data/hpc900:/dcl01/smart/data/hpc900 fmriprep_latest.simg /dcl01/smart/data/hpc900/disk1_bids /dcl01/smart/data/hpc900/bwinston/derivatives participant --fs-license-file /dcl01/smart/data/hpc900/bwinston/license.txt --output-spaces anat --participant-label 103818
<EMAIL_ADDRESS>
	1: [ERR] Quick validation failed - the general folder structure does not resemble a BIDS dataset. Have you chosen the right folder (with "sub-*/" subfolders)? Check for structural/naming issues and presence of at least one subject. (code: 61 - QUICK_VALIDATION_FAILED)
		.disk1_bids

Strangely, I tried using the cli BIDS validator on the same directory, and I got some warnings, but it was valid.
[compute-066 /users/bwinston]$ singularity run -B /dcl01/smart/data/hpc900/:/dcl01/smart/data/hpc900/ bids_validator.sif /dcl01/smart/data/hpc900/disk1_bids
<EMAIL_ADDRESS>
	1: [WARN] Not all subjects contain the same files. Each subject should contain the same number of files with the same naming unless some files are known to be missing. (code: 38 - INCONSISTENT_SUBJECTS)
		./sub-100206/fmap/sub-100206_run-03_magnitude.nii.gz
			Evidence: Subject: sub-100206; Missing file: sub-100206_run-03_magnitude.nii.gz
		./sub-100206/fmap/sub-100206_run-03_phasediff.json
			Evidence: Subject: sub-100206; Missing file: sub-100206_run-03_phasediff.json
		./sub-100206/fmap/sub-100206_run-03_phasediff.nii.gz
			Evidence: Subject: sub-100206; Missing file: sub-100206_run-03_phasediff.nii.gz
		./sub-100206/fmap/sub-100206_run-04_magnitude.nii.gz
			Evidence: Subject: sub-100206; Missing file: sub-100206_run-04_magnitude.nii.gz
		./sub-100206/fmap/sub-100206_run-04_phasediff.json
			Evidence: Subject: sub-100206; Missing file: sub-100206_run-04_phasediff.json
		./sub-100206/fmap/sub-100206_run-04_phasediff.nii.gz
			Evidence: Subject: sub-100206; Missing file: sub-100206_run-04_phasediff.nii.gz
		./sub-100307/fmap/sub-100307_run-03_magnitude.nii.gz
			Evidence: Subject: sub-100307; Missing file: sub-100307_run-03_magnitude.nii.gz
		./sub-100307/fmap/sub-100307_run-03_phasediff.json
			Evidence: Subject: sub-100307; Missing file: sub-100307_run-03_phasediff.json
		./sub-100307/fmap/sub-100307_run-03_phasediff.nii.gz
			Evidence: Subject: sub-100307; Missing file: sub-100307_run-03_phasediff.nii.gz
		./sub-100307/fmap/sub-100307_run-04_magnitude.nii.gz
			Evidence: Subject: sub-100307; Missing file: sub-100307_run-04_magnitude.nii.gz
		... and 675 more files having this issue (Use --verbose to see them all).

	Please visit https://neurostars.org/search?q=INCONSISTENT_SUBJECTS for existing conversations about this issue.

	2: [WARN] Each _phasediff.nii[.gz] file should be associated with a _magnitude1.nii[.gz] file. (code: 92 - MISSING_MAGNITUDE1_FILE)
		./sub-100206/fmap/sub-100206_run-01_phasediff.nii.gz
		./sub-100206/fmap/sub-100206_run-02_phasediff.nii.gz
		./sub-100307/fmap/sub-100307_run-01_phasediff.nii.gz
		./sub-100307/fmap/sub-100307_run-02_phasediff.nii.gz
		./sub-100408/fmap/sub-100408_run-01_phasediff.nii.gz
		./sub-100408/fmap/sub-100408_run-02_phasediff.nii.gz
		./sub-100610/fmap/sub-100610_run-01_phasediff.nii.gz
		./sub-100610/fmap/sub-100610_run-02_phasediff.nii.gz
		./sub-100610/fmap/sub-100610_run-03_phasediff.nii.gz
		./sub-101006/fmap/sub-101006_run-01_phasediff.nii.gz
		... and 323 more files having this issue (Use --verbose to see them all).

	Please visit https://neurostars.org/search?q=MISSING_MAGNITUDE1_FILE for existing conversations about this issue.

	3: [WARN] The recommended file /README is missing. See Section 03 (Modality agnostic files) of the BIDS specification. (code: 101 - README_FILE_MISSING)

	Please visit https://neurostars.org/search?q=README_FILE_MISSING for existing conversations about this issue.

        Summary:                   Available Tasks:        Available Modalities:
        8840 Files, 16.44GB        Gambling                MRI
        113 - Subjects
        1 - Session

	If you have any questions, please post on https://neurostars.org/tags/bids.

When I run fmriprep skipping bids validation, I get fmriprep: error: One or more participant labels were not found in the BIDS directory: 103818.
A bit confused on how to proceed. Thank you so much.

Thanks! You were absolutely right. It turns out the files in this directory were also dangling symlinks, which caused fMRIprep not to work.


[SPARK-32932][SQL] Do not use local shuffle reader on RepartitionByExpression when coalescing disabled

What changes were proposed in this pull request?
Do not use local shuffle reader on RepartitionByExpression by disallowing changing number of partitions when spark.sql.adaptive.coalescePartitions.enabled=false.
Why are the changes needed?
Users usually repartition with partition column on dynamic partition overwrite. AQE could break it by removing shuffle with local shuffle reader. That could lead to a large number of output files, even exceeding the file system limit.
Does this PR introduce any user-facing change?
Yes.
How was this patch tested?
Add test.

Test build #128851 has started for PR 29797 at commit fc831f6.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/33474/
Test PASSed.

cc @maryannxue @cloud-fan @JkSelf @viirya

Test build #128858 has started for PR 29797 at commit 93775e8.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/33481/
Test PASSed.

Test build #128851 has finished for PR 29797 at commit fc831f6.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

Test build #128858 has finished for PR 29797 at commit 93775e8.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/128858/
Test FAILed.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/128851/
Test FAILed.

AQE could break it by removing shuffle with local shuffle reader.

Yea this is an issue, thanks for reporting! BTW why is it OK when coalesce partition is enabled?

@cloud-fan
It's not ok but I haven't thought of a simple way to disable local shuffle reader on dynamic partition overwrite (bucket table overwrite as well). Any ideas ?

seems like we should be stricter to apply local shuffle reader, to make sure we can satisfy the expected parallelism.

I think the root cause is, we give local shuffle reader expected parallelism, but it's not always satisfied. For example, too many mappers can lead to over-parallelism and many small files.
We should add a check and only apply local shuffle reader if expected parallelism can be satisfied.

@cloud-fan

we give local shuffle reader expected parallelism

Could you please elaborate on this ? How will it solve the small files issue due to removed repartition when writing to dynamic partition or bucket table ?

See OptimizeLocalShuffleReader.getPartitionSpecs:
val expectedParallelism = advisoryParallelism.getOrElse(numReducers)

If we skip local shuffle reader, we fix the small files issue.

the small files issue due to removed repartition

Are you talking about a different issue? IIUC the repartition (shuffle) is still there and the only problem is local shuffle reader.

I mean the physical shuffle doesn't happen so that each shuffle task will generate at most numReducers files. The overall number will be numMappers * numReducers.
If we add a check, I'm not sure whether the local shuffle reader will ever be applied in practice.  In our use cases, the target bucket tables usually have more than 1000 buckets.

OK I get your point. The file write node works better with certain partitioning and the local shuffle reader breaks it.
Then how about we update InsertAdaptiveSparkPlan, and have a way to disable local shuffle reader if the root node is data writing node?

Then how about we update InsertAdaptiveSparkPlan, and have a way to disable local shuffle reader if the root node is data writing node?

That will also disable local shuffle reader for broadcast join. How about adding a variable isDataWritingCommand in the context and bypassing local shuffle reader for shuffle exchange if that's true ?

That will also disable local shuffle reader for broadcast join

We can just disable local shuffle reader in the final stage, if the root node is data writing node. What do you think?

@cloud-fan Redo the PR as you suggested. Please help review.

Test build #129110 has started for PR 29797 at commit 7e0d766.

Test build #129110 has finished for PR 29797 at commit 7e0d766.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/129110/
Test FAILed.

Test build #129111 has started for PR 29797 at commit 84134b0.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/33731/

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/33732/

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/33731/

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/33731/
Test PASSed.

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/33732/

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/33732/
Test PASSed.

Test build #129111 has finished for PR 29797 at commit 84134b0.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/129111/
Test PASSed.

Test build #129580 has started for PR 29797 at commit b112133.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34185/

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34185/

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/34185/
Test FAILed.

Test build #129580 has finished for PR 29797 at commit b112133.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/129580/
Test PASSed.

Test build #129691 has started for PR 29797 at commit 4fdee62.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34297/

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34297/

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/34297/
Test PASSed.

Test build #129691 has finished for PR 29797 at commit 4fdee62.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/129691/
Test PASSed.

Test build #129767 has started for PR 29797 at commit d391269.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34373/

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34373/

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/34373/
Test PASSed.

Test build #129767 has finished for PR 29797 at commit d391269.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/129767/
Test PASSed.

thanks, merging to master!

@manuzhang This change seems incomplete. The problem here is not about the writer, but rather about the repartition Exchange. We should avoid letting LSR rule work on such shuffles, not just for the writer case.
All you need to do is: 1) move LSR rule into postStageCreationRules; and 2) make the LSR rule match an Exchange first:
    plan match {
      case e: Exchange if canUseLocalShuffleReader(e.child) =>
        e.withNewChildren(Seq(createLocalReader(e.child)))
      case s: SparkPlan =>
        createProbeSideLocalReader(s)
    }

So that it'll only modify the child node of an Exchange, which is the purpose of this rule in the first place.

@maryannxue This could miss a LSR optimization on build side of leaf BHJ in multiple joins as follows.
*(6) BroadcastHashJoin [b#24], [a#33], Inner, BuildLeft, false
:- BroadcastQueryStage 7
:  +- BroadcastExchange HashedRelationBroadcastMode(List(cast(input[3, int, false] as bigint)),false), [id=#390]
:     +- CustomShuffleReader local
:        +- ShuffleQueryStage 6
:           +- Exchange hashpartitioning(b#24, 5), true, [id=#364]
:              +- *(5) BroadcastHashJoin [key#13], [a#23], Inner, BuildLeft, false
:                 :- BroadcastQueryStage 4
:                 :  +- BroadcastExchange HashedRelationBroadcastMode(List(cast(input[0, int, false] as bigint)),false), [id=#264]
:                 :     +- CustomShuffleReader local
:                 :        +- ShuffleQueryStage 0
:                 :           +- Exchange hashpartitioning(key#13, 5), true, [id=#172]
:                 :              +- *(1) Filter (isnotnull(value#14) AND (cast(value#14 as int) = 1))
:                 :                 +- *(1) SerializeFromObject [knownnotnull(assertnotnull(input[0, org.apache.spark.sql.test.SQLTestData$TestData, true])).key AS key#13, staticinvoke(class org.apache.spark.unsafe.types.UTF8String, StringType, fromString, knownnotnull(assertnotnull(input[0, org.apache.spark.sql.test.SQLTestData$TestData, true])).value, true, false) AS value#14]
:                 :                    +- Scan[obj#12]
:                 +- ShuffleQueryStage 1
:                    +- Exchange hashpartitioning(a#23, 5), true, [id=#179]
:                       +- *(2) Filter (b#24 = 1)
:                          +- *(2) SerializeFromObject [knownnotnull(assertnotnull(input[0, org.apache.spark.sql.test.SQLTestData$TestData2, true])).a AS a#23, knownnotnull(assertnotnull(input[0, org.apache.spark.sql.test.SQLTestData$TestData2, true])).b AS b#24]
:                             +- Scan[obj#22]
+- *(6) BroadcastHashJoin [n#93], [a#33], Inner, BuildRight, false
   :- CustomShuffleReader local
   :  +- ShuffleQueryStage 2
   :     +- Exchange hashpartitioning(n#93, 5), true, [id=#196]
   :        +- *(3) Filter (n#93 = 1)
   :           +- *(3) SerializeFromObject [knownnotnull(assertnotnull(input[0, org.apache.spark.sql.test.SQLTestData$LowerCaseData, true])).n AS n#93, staticinvoke(class org.apache.spark.unsafe.types.UTF8String, StringType, fromString, knownnotnull(assertnotnull(input[0, org.apache.spark.sql.test.SQLTestData$LowerCaseData, true])).l, true, false) AS l#94]
   :              +- Scan[obj#92]
   +- BroadcastQueryStage 5
      +- BroadcastExchange HashedRelationBroadcastMode(List(cast(input[0, int, false] as bigint)),false), [id=#310]
         +- CustomShuffleReader local
            +- ShuffleQueryStage 3
               +- Exchange hashpartitioning(a#33, 5), true, [id=#210]
                  +- *(4) Filter (a#33 = 1)
                     +- *(4) SerializeFromObject [knownnotnull(assertnotnull(input[0, org.apache.spark.sql.test.SQLTestData$TestData3, true])).a AS a#33, unwrapoption(IntegerType, knownnotnull(assertnotnull(input[0, org.apache.spark.sql.test.SQLTestData$TestData3, true])).b) AS b#34]
                        +- Scan[obj#32]

Can you highlight the problem? I do see CustomShuffleReader local in all the build sides.

@cloud-fan ShuffleQueryStage 1 which currently has a CustomShuffleReader local parent

I assume in this case, the repartition node has been optimized out. Then still this fix won't cover just the repartition case without a writer, right?

We have some fundamental problems hanging around in our optimizations. In general, when we optimize a Shuffle over another Shuffle that is semantically equivalent (almost equivalent in this case), we should get rid of the child shuffle instead of the parent one, yet it's a little hard to implement right now.
But again, this is a "repartition" issue rather than the write command issue. Introducing this kind of hack in AQE plan does not make sense.

@maryannxue thanks for pointing out the fundamental problems while I'm not familiar with the design behind. When starting out, I was trying to solve the most urgent issue in our use cases, too many output files caused by LSR, which broke the downstream jobs on the pipeline. Meanwhile, disabling LSR entirely downgrades the performance of BHJ switched from SMJ. Hence, this PR with help of @cloud-fan.

But again, this is a "repartition" issue rather than the write command issue.  Introducing this kind of hack in AQE plan does not make sense.

Is that ok or even good if repartition is optimized away in cases other than on write command ?
Moving forward, what's your suggestion ? Do you want me revert this PR ?

I'm reading the classdoc of OptimizeLocalShuffleReader, and I do feel the design is a bit hacky. We add local shuffle reader if

the shuffle is the root node of a query stage.
the shuffle is BHJ build side.

The reason for condition 1 is it will never introduce shuffle. This is true, but this may change the final output partitioning which may be bad for cases like write command.
I like the idea from @maryannxue which is more general: 1) move LSR rule into postStageCreationRules; and 2) make the LSR rule match an Exchange first (so condition 1 becomes: the shuffle is a direct child of an exchange). By doing this we can skip LSR rule in the last stage, as the last stage's root node is not exchange.
I'm not very sure why the current approach can add LSR to BHJ probe side. This seems like an accident to me as it's not mentioned in the classdoc.

In LSR, we have a sanity check to make sure that the output partitioning requirement is not broken after this rule. For example, if there's a parent join sitting above the current BHJ for which we are trying to optimize the probe side to LSR, and that parent join has to take advantage of the output partitioning of the current BHJ. When applying the LSR rule, the check would fail, and we would back out of it.
It's a similar situation here with repartition, only that repartition has been optimized away. So if we can introduce an idea like "required partitioning" (and in the future maybe even required sort order) of the query, then when applying the LSR rule we would know it could break the required partitioning. Hope it makes sense, @manuzhang

@maryannxue thanks for the thorough explanation. Let me rework this.

Thank you, @manuzhang! I have a local fix already, and I'll submit a PR shortly. I assume that the coalescing rule has a similar issue where a repartition shuffle with a specific number is optimized out because of another shuffle introduced by join or agg, and later that shuffle gets coalesced, which means the specified repartition number is ignored.

@maryannxue Great to know. I've been testing a draft fix which doesn't look good without optimizing the probe side of BHJ. I can help test your PR. Coalescing won't be applied to repartition if user specifies a repartition number.

@manuzhang

Coalescing won't be applied to repartition if user specifies a repartition number.
It does apply when the specified number happen to be the default partition number, right?


possible bug in German?

i got this feedback from a user:
"If you check out the German internationalization, the names of the drains come up as {thing_name}"
i don't know exactly what this means or how to check it, but maybe someone can take a look?

This should be fixed by https://github.com/sfbrigade/adopt-a-drain/commit/78639d9499e838774f0fe76333dc92b761862e76.
It means that German users would have seen {thing_name} rather than the adopted name of the drain in the drain details. Unfortunately, I'm sure there are some other missing pieces to make sure the app is fully internationalized -- I might recommend just restricting to English unless have anyone that can manage the translations and other internationalization concerns.

thanks, @jszwedko. how do we restrict to English? works for me!

We can do that in the code -- I'll make that adjustment now.

Should be done by 5a32412


Can not find model of 'vit_base_patch16_clip_224.openai_ft_in1k' in my TRANSFORMERS_CACHE

I can not find cached model when I created a model, that is 'vit_base_patch16_clip_224.openai_ft_in1k'. I have set TRANSFORMERS_CACHE for huggingface models. Where timm caches these models?
Thansk!

@LuoXin-s from cmd line huggingface-cli scan-cache  .. you can also clone the model from the HF hub (it's just a git repo) to any location you'd like


API property page issues

input boxes should be longer
"is" case sensitive
reserved word validation doesn't work
Buttons need to be of the standard size (this one is too small)
Space is too much. Need to be the standard value.
The redundant button should be removed.

Fixed all except item 4 and 2.
2 - The documentation says the same as the text in the UI. https://docs.wso2.com/display/AM260/Adding+Custom+Properties+to+APIs.
4 - The button size is the default button size of MUI.

@shaniR what do you think of 2?

Yes we should change the documentation as well. It should be "is not". In addition to it, I think the helper text is too long. Should we add those as validation text? and as a tool tip?
Property name :
- should be unique.
- should not contain spaces.
- should not be case-sensitive.
- should not be any of the following reserved keywords: provider, version, context, status, description, subcontext, doc, lcState, name, tags.
Please consider the following as well:

The mandatory asterisk should be on the field name.
The space validation also does not happen.
The table listing the properties should have table headings

reopening to fix 2.

Property name :

should not be case-sensitive.

@shaniR it should be
" is case-sensitive "
and -1 for tooltip this information needs to be always visible

I still can observe issue (3).

Issue [3] has been fixed with https://github.com/wso2/carbon-apimgt/pull/7873


AnalyzeVacuumUtility - Unused arg in analyze-vacuum-schema.py

Hi,
I noticed that the main(argv) function in analyze-vacuum-schema.py accepting argv as an argument (line 116) but it doesn't use it anywhere the main(argv) function. I'm confused on what the purpose of passing this argv. Please advise.
Thank you,

The command line arguments are parsed earlier in the script. You are right, the argv variable is not used in the main function.
The command arguments can be used from either the sys.argv or from the earlier global variable 'parse_args'. This doesnt affect in any way the script, so leaving it as is.


Updated logos

Added f-droid
Shortend url for google play
added newline

thanks for the PR @mueller-ma


Use multiplier for motor PWM

As a user being able to swap out equipment is helpful for reducing cost. The current values used by the PWM signals are specific to the motor driver being used.
Rather than using a hard value in the code, use a multiplier instead with the actual value coming from the config file on the SD card.

Related to #90

Most likely the config file will need to store a percentage between 0 and 100 to indicate the motor strength. This value would be used in defining the percentage of the motors for the flywheels and the pusher.
The motor controller min and max defines the actual values specific to the driver/ESC being used that gets multiplied by the percentage to determine the actual value fed into the PWM generator.


[CT-3005] Support dictionary of additional configs in dbt_project.yml

Housekeeping

[X] I am a maintainer of dbt-core

Short description
Allow setting a dictionary of arbitrary configs in dbt_project.yml.
We don't need to validate the keys in this dictionary, nor store them in the project config class.
I don't have a strong opinion about what this is named, but this is the big idea:
# dbt_project.yml
cloud:
  project-id: 1234
  ...

Acceptance criteria
It is possible to set a dictionary of arbitrary configs in dbt_project.yml, as metadata for another tool.
Impact to Adapters
zero impact
Context
internal slack thread
internal jira ticket

Open question:

how will they access these keys? are they intending on parsing these themselves?


Interpreting 2d coordinate output

I am able to get 2d coordinate by following code:
poses_3d, poses_2d = parse_poses(inference_result, input_scale, stride, fx, is_video)
and the output looks like this:
[ 9.87000000e+02  8.40000000e+01  7.44423568e-01 -8.00000000e+00
  -8.00000000e+00 -1.00000000e+00 -1.00000000e+00 -1.00000000e+00
  -1.00000000e+00  9.95000000e+02  8.40000000e+01  7.20652282e-01
   1.02900000e+03  1.43000000e+02  4.71996784e-01  1.02900000e+03
   1.85000000e+02  4.42449659e-01  9.95000000e+02  2.10000000e+02
   6.77076042e-01  9.61000000e+02  2.86000000e+02  6.98154628e-01
   9.53000000e+02  3.54000000e+02  5.87647200e-01  9.53000000e+02
   8.40000000e+01  6.99335754e-01  9.28000000e+02  1.43000000e+02
   3.37824076e-01  9.28000000e+02  1.51000000e+02  1.61912560e-01
   9.61000000e+02  2.10000000e+02  6.83217466e-01  9.87000000e+02
   2.86000000e+02  6.84145570e-01  9.95000000e+02  3.45000000e+02
   5.94412923e-01 -8.00000000e+00 -8.00000000e+00 -1.00000000e+00
  -8.00000000e+00 -8.00000000e+00 -1.00000000e+00 -8.00000000e+00
  -8.00000000e+00 -1.00000000e+00 -8.00000000e+00 -8.00000000e+00
  -1.00000000e+00  1.42583218e+01]

How do I know which body part it is referring to?
Any advice would be appreciated.

Hi! You can check draw_poses, which parses and draws 2D poses coordinates.

Hi! You can check draw_poses, which parses and draws 2D poses coordinates.

Thanks for your helpful hint. I have read the code but got stuck on one thing.
We only have 18 body parts:

But we have 19 kpt_id:
for kpt_id in range(pose.shape[1])
What is the missing one?
Any hint would be appreciated

Nothing is missed. 2D network detects 18 keypoints you have found above. The order of 2D keypoints is aligned to match the 3D network output. 2 is reserved value for pelvis in 3D network. Now it is not predicted, it was left for the future.

Thank you very much!

@Daniil-Osokin sorry so there is no predicted pelvis in this model?

Right.

@Daniil-Osokin but parse poses emits a pelvis in index 2? Is that not an accurate calculation or something?

It always has "-1" as coordinates values. It is left for the legacy reasons.


Ignore boot and main in package

Added

*/boot.py and */main.py can be ignored during the check with --ignore-boot-main, see #8

Codecov Report

Merging #9 (d577539) into main (43d7a3e) will increase coverage by 0.11%.
The diff coverage is 100.00%.

:exclamation: Your organization is not using the GitHub App Integration. As a result you may experience degraded service beginning May 15th. Please install the Github App Integration for your organization. Read more.
@@            Coverage Diff             @@
##             main       #9      +/-   ##
==========================================
+ Coverage   96.27%   96.38%   +0.11%
==========================================
  Files           2        2
  Lines         161      166       +5
  Branches       22       24       +2
==========================================
+ Hits          155      160       +5
  Misses          3        3
  Partials        3        3

Flag
Coverage Δ

unittests
96.38% <100.00%> (+0.11%)
:arrow_up:

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files
Coverage Δ

src/setup2upypackage/setup2upypackage.py
96.36% <100.00%> (+0.11%)
:arrow_up:

:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more


wrap the functions in order to be IDE friendly

@bashirmindee same here, could you provide a short description of the desired behaviour ?


fix(build): fix build on windows

Description
Fixes a bug where running various npm scripts on Windows (including the three build: scripts) would fail due to Linux / Unix style commands. This fixes that by using cross platform node modules for file copying, making a directory and setting environment variables.
Checklist

[x] All tests passing
[ ] Created tests which fail without the change (if possible)
[ ] Extended the README / documentation, if necessary
[x] Commits and PR follow conventions

This should be done now. Let me know if there's anything else needed.


Using the same dependency twice but with different features.

It would be useful to be able to depend on the same package twice, but using different features, that is, being able to write something like:
[dependencies.some_library]
git = "fixme"
features = [ "feature1" ]

[dependencies.some_library]
git = "fixme"
features = [ "feature2" ]

However, this will cause some naming problems as the two libraries will have the same name. To prevent this, we could add a name entry to the depencency to rename it:
[dependencies.some_library]
git = "fixme"
name = "some_library_1"
features = [ "feature1" ]

[dependencies.some_library]
git = "fixme"
name = "some_library_2"
features = [ "feature2" ]

Or perhaps we could let the user specify the features when he does an extern crate:
extern crate "some_library" with feature1 as some_library_1
extern crate "some_library" with feature2 as some_library_2

and the dependency on cargo would not specify any feature as they will be inferred by the extern crate:
[dependencies.some_library]
git = "fixme"

My use-case for this is to allow the user to use the 3df32 feature of ncollide (for 3d collision detection) together with the 2df32 feature (for 2d collision detection) in the same application.

I don't think this issue was ever solved -- if you want to re-use the same crate with a different set of features this doesn't seem to be possible. The resolver automatically takes the union of features, and if you try to separate these into distinct dependencies using a different name for each you get 'depends on crate multiple times with different names'. Why shouldn't/couldn't these different names be treated as distinct dependencies?

I don't see any technical reasons not to let users depend on the same crate twice, if that's what they need. A warning when doing that is logical, but an error is too much.
I know that in theory all crate features should be additive and you shouldn't ever need to depend on the same crate twice with different features, but in the real world that's not always the case, and cargo blocking the only solution to it for no reason doesn't seem logical to me.

@alexcrichton Could this issue be reopened? It seems that it was closed by accident because of the linked PR, which did not in fact solve this issue.

@alexcrichton Bumping this; this would be very useful for differential fuzzing in cases where different features are used (e.g. no-std vs. std environment).

This is an 8 year old issue, please open a new one with more up-to-date information.

I have similar question. Current solution people use in my case is macro. But having crate imported 2 times with different features would work. Basically sometimes I want TypeScript in Rust. This could solve it for certain use cases.

parking_lot is a good example of exclusive features. Here is an error when importing dependencies which both use different parking_lot features.
[the `send_guard` and `deadlock_detection` features cannot be used together](error: the `send_guard` and `deadlock_detection` features cannot be used together)


Need a note in install/get started guides about using QDK on Ubuntu

I was trying to do some demos on stream today where I wanted to start from a base Ubuntu distro and spin up to a working Q# console app. I started with these docs and ran into an annoying bug. I had the following docker file:
ARG VARIANT="focal"
FROM mcr.microsoft.com/vscode/devcontainers/base:0-${VARIANT}
USER root
RUN wget https://packages.microsoft.com/config/ubuntu/20.10/packages-microsoft-prod.deb -O packages-microsoft-prod.deb
RUN sudo dpkg -i packages-microsoft-prod.deb
RUN apt-get update && export DEBIAN_FRONTEND=noninteractive \
   && apt-get -y install --no-install-recommends dotnet-sdk-3.1

from there I installed the project templates as instructed here: https://docs.microsoft.com/en-us/azure/quantum/install-command-line-qdk?tabs=tabid-vscode#to-configure-for-another-environment
and tried to run just the template generated console app, and got the following error:
Unhandled exception: System.DllNotFoundException: Unable to load shared library 'Microsoft.Quantum.Simulator.Runtime.dll' or one of its dependencies. In order to help diagnose loading problems, consider setting the LD_DEBUG environment variable: libMicrosoft.Quantum.Simulator.Runtime.dll: cannot open shared object file: No such file or directory
   at Microsoft.Quantum.Simulation.Simulators.QuantumSimulator.Init()
   at Microsoft.Quantum.Simulation.Simulators.QuantumSimulator..ctor(Boolean throwOnReleasingQubitsNotInZeroState, Nullable`1 randomNumberGeneratorSeed, Boolean disableBorrowing)
   at Microsoft.Quantum.EntryPointDriver.Simulation`3.<>c.<Simulate>b__0_0()
   at Microsoft.Quantum.EntryPointDriver.Simulation`3.RunSimulator(IEntryPoint`2 entryPoint, ParseResult parseResult, Func`1 createSimulator)
   at Microsoft.Quantum.EntryPointDriver.Simulation`3.Simulate(DriverSettings settings, IEntryPoint`2 entryPoint, ParseResult parseResult, String simulator)
   at Microsoft.Quantum.EntryPointDriver.Driver`3.Simulate(ParseResult parseResult, String simulator)
   at System.CommandLine.Invocation.CommandHandler.GetResultCodeAsync(Object value, InvocationContext context)
   at System.CommandLine.Invocation.ModelBindingCommandHandler.InvokeAsync(InvocationContext context)
   at System.CommandLine.Invocation.InvocationPipeline.<>c__DisplayClass4_0.<<BuildInvocationChain>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c.<<UseParseErrorReporting>b__20_0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c__DisplayClass15_0.<<UseHelp>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c__DisplayClass24_0.<<UseVersionOption>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c__DisplayClass22_0.<<UseTypoCorrections>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c.<<UseSuggestDirective>b__21_0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c.<<UseParseDirective>b__19_0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c.<<UseDebugDirective>b__11_0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c.<<RegisterWithDotnetSuggest>b__10_0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c__DisplayClass13_0.<<UseExceptionHandler>b__0>d.MoveNext()

I couldn't find any solutions online, except for something that sounded similar on this SE question: https://stackoverflow.com/questions/52428334/unable-to-load-dll-microsoft-quantum-simulator-runtime-dll-centos-7
Once I installed libgomp it fixed everything but was really hard to sort that our when even the LD_DEBUG wasn't helpful.
Most science/research/data science users will be using Linux with the QDK so I strongly feel this needs to be noted in the docs or find a way to propagate a more helpful error message...

Possible duplicate of #2

@geduardo you linked to this issue, which one you mean?

Possible duplicate of #2

@geduardo you linked to this issue, which one did you mean?

Ups, I meant #20

Hi @crazy4pi314, thank you for bringing this up. I can confirm this issue on my side, at least when using docker images, as regular Ubuntu images already come with this dependency (libgomp1).
I'll look into resolving this issue.


New feature: Test mode?

Is it possible to have a test mode of some sort?  I ask because I have one CentOS 8 machine left to migrate to Rocky. (the rest were standard VMs that I could snapshot)  Its a hardware machine, and used as a file server or sorts to store several TB of data backups; so I can't really back it up, for a possible failure.  It doesn't really have anything special installed, except for one thing:
A driver for the LSI/3Ware/Avaya/Broadcom HBA card.
Serial Attached SCSI controller: Broadcom / LSI SAS2008 PCI-Express Fusion-MPT SAS-2 [Falcon] (rev 03)
if I recall, this was installed with information from
https://elrepoproject.blogspot.com/2019/08/rhel-80-and-support-for-removed-adapters.html
and the ELRepo repository.
It appears to be booting from a software RAID 1, connected to the motherboard,  and then 2 HBA cards connect 16 drives in a software RAID 60
Any comments would be welcome. thanks.

The unfortunate reality is that there is no way to accurately test the full scope of a migration before performing the migration.  migrate2rocky uses a number dnf and rpm commands, among other commands that must be performed in sequence in order to complete the migration.  Even a single dnf command can only partially be tested with the --assumeno option, and since the next command in line relies on the completion of the previous command it's impossible to know if a subsequent command will fail until the previous command is completed.
migrate2rocky does attempt to detect situations that will cause failure, though, via the use of a number of pre-migration checks and if any of these fail it will refuse to continue before it gets to the point where it will leave the system in an unstable state.  This is no guarantee but it does catch a number of common issues, and if you discover anything which it does not catch then I am always happy to hear about it so we can add a new test for it if possible.
In the event that a migration does fail it is usually possible to complete the migration by fixing the cause of the failure and then completing the remaining steps manually, often times just a dnf distro-sync is all that is needed at that point.  Always save the migrate2rocky.log file from a failed migration and share that with us so we can help you to complete the migration and get your system back up and running.  Also it pays to have a separate emergency boot image handy (disk, usb stick, etc).
Good luck with your migration!


Term merge dATP/ATP triphosphate pyrophosphatase

Hello,
Working on the annotation ticket https://github.com/geneontology/go-annotation/issues/3581 NUDT1 (P36639)
I think we need to edit the ontology as follows:

GO:0044714  2-hydroxy-(deoxy)adenosine-triphosphate pyrophosphatase activity (no annotations, mapped to Reactome R-HSA-2395818 "NUDT1 hydrolyses 2-OH-dATP to 2-OH-dAMP ") should merge into  "GO:0008828
dATP pyrophosphohydrolase activity" : RHEA:31583; EC:<IP_ADDRESS> (label is 2-hydroxy-dATP diphosphatase  in EC)

GO:0044713 2-hydroxy-adenosine triphosphate pyrophosphatase activity (no annotations, mapped to Reactome R-HSA-2395872 NUDT1 hydrolyses 2-OH-ATP to 2-OH-AMP ) should merge into GO:0008828 dATP pyrophosphohydrolase activity (RHEA:31583; EC:<IP_ADDRESS>)

GO:0035539 8-oxo-7,8-dihydrodeoxyguanosine triphosphate pyrophosphatase activity: Add EC:<IP_ADDRESS>

@ukemi
@deustp01
Do you agree ?
Thanks, Pascale

@ukemi
@deustp01
@kaxelsen
If there are no objections I will go ahead and implement this.
Thanks, Pascale

@ukemi
@deustp01
@kaxelsen
If there are no objections I will go ahead and implement this.
Thanks, Pascale

I have no objections

I have no objections

no objections here either

no objections here either


Add Super 67 Little Star Command Pod from Kerbal Stuff

This pull request was automatically generated by Kerbal Stuff on behalf of RaptorHunter, to add Super 67 Little Star Command Pod to CKAN.
Please direct questions about this pull request to RaptorHunter.

Has an extras folder, should probably be handled in a similar way as #1588 so we get a consistent handling of them.


pybigtools bigbed write appears to store chrom as "rest"

import pybigtools as pbt

f = pbt.open("/tmp/foo.bb", "w")
f.write(
    {"chr1": 1000, "chr2": 1000},
    [
        ("chr1", 0, 100, "foo\tbar\t3"),
        ("chr2", 100, 200, "xxx\tyyy\t1.0"),
        ("chr2", 100, 200, "zzz\twww\t1.0")
    ]
)
f.close()

f = pbt.open("/tmp/foo.bb", "r")
list(f.records("chr2"))

[(100, 200, 'chr2'), (200, 300, 'chr2')]

print(f.sql())

table bed3
"Simple bed"
(
    string chrom;        "Reference sequence chromosome or scaffold"
    uint   chromStart;   "Start position in chromosome"
    uint   chromEnd;     "End position in chromosome"
)

Uh, that's weird.

This is fixed in pybigtools v0.2.2.

It works! Thank you!
However, the autoSql still claims it's a bed3. Does autoSql have a way to declare at least something like "bed3+"?
Beyond that, it would be nice to able to store a custom autoSql definition, which we can use in bioframe.
Since in bioframe we can already infer common bed schemas from a dataframe and map numpy/pandas dtypes to autoSql types, we should be able to generate valid autoSql schemas from an input dataframe. Being able to store those upon writing a bigBed would keep that information in-band (and we could use it to automatically parse fields on read).


make orange loader 260x260

What does this PR do and why?
Provide a description of what this PR does. Link to any relevant GitHub issues, Notion tasks or Discord discussions.
Screenshots or screen recordings
If applicable, provide screenshots or screen recordings to demonstrate the changes.
Acceptance checklist

[ ] I have evaluated the Approval Guidelines for this PR.
[ ] (if relevant) I have tested this PR in all supported browsers.
[ ] (if relevant) I have tested this PR in dark mode and light mode (if applicable).

Current dependencies on/for this PR:

main

PR #3939   👈

This comment was auto-generated by Graphite.


Display line number where pending was called in backtrace

De-scoped from #22.
example:
     Scenario:
           Given pending
             TODO: Step not yet implemented (Cucumber::Core::Test::Result::Pending)
             features/step_definitions/steps.tcl:2
             features/test.feature:3:in `Given pending'

       1 scenario (1 pending)
       1 step (1 pending)

source locations in tcl are tricky. Here's my attempt. The frame level there will probably need some logic, as it's fine-tuned to the project set-up

@jowers can you take a look?

@jowers the updated version: in cucumber-tcl-wire that is more robust against frame level differences

Hi @d-led, thanks for the contribution on this.
I'll try to put together a test that we can try out - as you said, source locations can be tricky, and I'm finding that testing in a standlone file is returning locations one level out at the moment. Probably best to see how it behaves within the cucumber code before I go investiigating that though.

@jowers you're welcome! Here's the initial investigation that lead to the code http://ideone.com/mdBT4s

the shortcomings can be seen here: http://ideone.com/K5zZva


Running native projects on 0.10.8 is flaky

I've been having issues where launching tests or running main with the new 0.10.8 just doesn't work half the time. If I rerun a command like test, it'll start working, but from a cold start it just halts right before the run.

This is a bit too vague to be actionable. Please at least share some verbose output. Better share the project (when open source) or the build.sc or some reproduction setup.

And this never happens when using Mill 0.10.7?

This couldn't be run before on mill 0.10.7. Scala 3.2 wasn't supported yet.

Just to isolate the issue, is it possible to use Scala 3.1.x for a repro? When yes, does it fail too and with both Mill versions?

I've enabled all the compileTime/linkTime debug I had working on SBT, and this consistently happens
=================================================================
==30458==ERROR: AddressSanitizer: stack-buffer-overflow on address 0x7ffe5a27e6f8 at pc 0x55c79f233fd3 bp 0x7ffe5a27e570 sp 0x7ffe5a27e568
READ of size 8 at 0x7ffe5a27e6f8 thread T0
    #0 0x55c79f233fd2  (/home/mhammons/Projects/Dramaturge/out/dramaturge/test/nativeLink.dest/out+0x454fd2) (BuildId: 184adb681c5bbe43ec0662e1e30327fd7fdaaf0f)
    #1 0x55c79f23439c  (/home/mhammons/Projects/Dramaturge/out/dramaturge/test/nativeLink.dest/out+0x45539c) (BuildId: 184adb681c5bbe43ec0662e1e30327fd7fdaaf0f)
    #2 0x55c79f23a487  (/home/mhammons/Projects/Dramaturge/out/dramaturge/test/nativeLink.dest/out+0x45b487) (BuildId: 184adb681c5bbe43ec0662e1e30327fd7fdaaf0f)
    #3 0x55c79f23b0b3  (/home/mhammons/Projects/Dramaturge/out/dramaturge/test/nativeLink.dest/out+0x45c0b3) (BuildId: 184adb681c5bbe43ec0662e1e30327fd7fdaaf0f)
[warn] Force close java.lang.RuntimeException: Process /home/mhammons/Projects/Dramaturge/out/dramaturge/test/nativeLink.dest/o    #4 0x55c79f23b510  (/home/mhammons/Projects/Dramaturge/out/dramaturge/test/nativeLink.dest/out+0x45c510) (BuildId: 184adb681c5bbe43ec0662e1e30327fd7fdaaf0f)
...
 Turning on
  val opts = Seq("-fno-omit-frame-pointer", "-fsanitize=undefined,address")
  def nativeCompileOptions = super.nativeCompileOptions() ++ opts
  def nativeLinkingOptions = super.nativeLinkingOptions() ++ opts

makes the test crash constantly, while there's an only an error about a memory leak (probably the GC's heap).

Just to isolate the issue, is it possible to use Scala 3.1.x for a repro? When yes, does it fail too and with both Mill versions?

It happens with 3.1.x and 10.7/10.8 with scala native 0.4.5 too.

Regarding the lost results:
    val res = TestModule.handleResults(doneMsg, results, Some(T.ctx()))
    // Hack to try and let the Scala Native subprocess finish streaming it's stdout
    // to the JVM. Without this, the stdout can still be streaming when `close()`
    // is called, and some of the output is dropped onto the floor.
    Thread.sleep(100)

This might be the cause.

Thank you for all the details. It's probably specific to our Scala Native tool chain, which is not my field of expertise. Hopefully, Lorenzo aka @lolgab has some ideas or more questions.


Pinning the webui should pin the geoip data

Describe the bug
ipfs pin add $(ipfs name resolve webui.ipfs.io)

doesn't pin the geoip data.
Expected behavior
Pinning the webui should in an offline first manner, download the application fully.
To do so the CIDs published at webui.ipfs.io should embed a link to the geoip data.
Additional context
Loading the webui from IPFS can be slow, I expect that pinning the webui would cache it locally and make that fast.
It mostly works, however it doesn't fully, the geoip data is not pinned and just as slow.
You actually have find out by some dark magic that https://github.com/ipfs-shipyard/ipfs-geoip/blob/5b85b6d94d3ae40d20392ba40e156760d7d61886/src/lookup.js#L11 is the other thing you need to pin.
Pinning this CID substantially increase the peer tab speed. (like from ~>30s to resolve everything to ~<3s).

ipfs-webui is 27MiB, ipfs-geoip data is ~388 MiB:
ipfs dag stat --progress=true Qmbt1YbZAhMoqo7r1t6Y5EJrYGVRgcaisNAZhLeJY6ehfg
Size: 388840361, NumBlocks: 220203

Asking people who want to pin webui to also fetch 400MiB does not sound like a good default.
What we could do is to:

publish ipfs-geoup CID in https://github.com/ipfs/ipfs-webui/releases page so if someone wants it, they know what to pin  in addition to webui
If someone has time, switching ipfs-geoip to binary dag-cbor would decrease the size of geip DAG and improve load performance: https://github.com/ipfs-shipyard/ipfs-geoip/issues/85

I think we could propose two options:
We could give a webui CID and webui + geoip CID.
Obviously, the webui + geoip CID would be a folder linking to the webui and geoip CIDs directly. (so people pinning this webui + geoip folder would be able to share just the webui to thoses who want it).


Question about torch.distributed.elastic.multiprocessing.errors

I have the same problem

It could happen due to the mismatch of your cuda version and the version of cuda that your installed pytorch supports. Please run nvcc --version and python -c "import torch; print(torch.__version__); print(torch.version.cuda)" to check if they match. If they already match, you may update the LD_LIBRARY_PATH to include cuda libs by export LD_LIBRARY_PATH=/usr/local/cuda/lib64:$LD_LIBRARY_PATH. Otherwise, please reinstall pytorch via conda install pytorch torchvision torchaudio pytorch-cuda=YOUR_CUDA_VERSION_HERE -c pytorch -c nvidia.


An alternate implementation for the Acreuus spellbook timers.

Not something I want to try implement in RuneLite within the Timers plugin but it's a small feature me, a few clan mates and hopefully others will use.

What's the difference to the core timers plugin here? Slightly larger icons?

Core timers plugin only shows the sprite for a 'Reurrect Superior Skeleton' not matter what thrall you summon, my edit gives a custom icon I made that will show either a skeleton, ghost or zombie, no matter what tier of thrall is cast.
Also the time formatting does not have the 0:00 formatting and just shows the seconds total and the cooldown icons are colorful unlike the use of the sprite which comes from the icon in the spellbook when you don't have the runes to cast a spell.
Death charge is a little different, easier to see the changes in the picture on the README . But basically the cooldown icon is easier to see and the death charge active infobox does not have the unnecessary countdown on it.


feat: add card component

:tada: This PR is included in version 0.15.0 :tada:
The release is available on:

GitHub release
npm package (@latest dist-tag)

Your semantic-release bot :package::rocket:


Exploration mode -> Validation mode

Hi there,
I was doing a tutorial of ASReview and came across the use of 'Exploration' mode, which I think is now renamed to 'Validation' mode. I wanted to edit the use of this term throughout the asreview-academy repository and push it on a separate branch, for you guys to review/merge it, but unfortunately (and understandably!) people don't have access to just randomly create branches and commit changes.
I forked the project instead, it contains the updated branch (as well as a screenshot that should replace one of the images). Maybe this is useful for you guys? I hope it helps to keep the docs up-to-date! :)

Hello @Metais, That is wonderful. Could you make a pull request so that we can merge it? That would be a great contribution! If you don't know how to do that, let me know and I'll help you out.


Fixes # 637 bugfix.datagroup_tests added functional tests for datagroups

Also:

cleaned up/expanded existing tests.
added modify exception to sys/file/ datagroup resources

@caphrim007 or @wojtek0806 this is ready for review.

looks great! even got 100% code coverage!

@jasonrahm looks good to me we can merge it i guess :) :+1:


Mongo views

In this PR I created views for all the major resources we would need to list. I included a README on how to import them using studio 3T (see here: https://github.com/clowder-framework/clowder2/blob/e6cc94251d05cb7d1df29018817296168c97cdbc/scripts/mongoviews/README.md)
For most of the resources I just join the "dataset_id" with  the dataset field in "authorization collection".
Metadata and listeners are trickier because they could be on "file" or on "dataset". So I end up doing a "facet" (if statement) first to see if resource type is "file" or "dataset.
 Counts the number of characters, words and lines in the text file that was uploaded.",
            "creator" : null,
            "created" : ISODate("2023-02-20T17:21:08.857+0000"),
            "modified" : ISODate("2023-02-20T17:21:08.857+0000"),
            "properties" : {
                "author" : "Rob Kooper<EMAIL_ADDRESS>                "process" : {
                    "file" : [
                        "text/*",
                        "application/json"
                    ]
                },
                "maturity" : "Development",
                "name" : "ncsa.wordcount",
                "contributors" : [

                ],
                "contexts" : [
                    {
                        "lines" : "http://clowder.ncsa.illinois.edu/metadata/ncsa.wordcount#lines",
                        "words" : "http://clowder.ncsa.illinois.edu/metadata/ncsa.wordcount#words",
                        "characters" : "http://clowder.ncsa.illinois.edu/metadata/ncsa.wordcount#characters"
                    }
                ],
                "repository" : [
                    {
                        "id" : ObjectId("63f3ac04ea5ae920b0babafe"),
                        "repository_type" : "git",
                        "repository_url" : ""
                    }
                ],
                "external_services" : [

                ],
                "libraries" : [

                ],
                "bibtex" : [

                ],
                "default_labels" : [

                ],
                "categories" : [

                ],
                "parameters" : null,
                "version" : "2.0"
            }
        }
    },
    "created" : ISODate("2023-02-21T17:47:26.195+0000"),
    "file_details" : [
        {
            "_id" : ObjectId("63efe87d6d2f826653fa65fe"),
            "name" : "ChangeLog.txt",
            "creator" : {
                "id" : ObjectId("63a32586cdf079dcd6077a1a"),
                "email"<EMAIL_ADDRESS>                "first_name" : "Chen",
                "last_name" : "Wang"
            },
            "created" : ISODate("2023-02-17T20:50:05.371+0000"),
            "version_id" : "75f9f740-283b-4d29-9335-3cff8ae987e5",
            "version_num" : NumberInt(1),
            "dataset_id" : ObjectId("63e50a0fb6a10a46379335f2"),
            "folder_id" : null,
            "views" : NumberInt(0),
            "downloads" : NumberInt(2),
            "bytes" : NumberInt(14451),
            "content_type" : {
                "content_type" : "text/plain",
                "main_type" : "text"
            }
        }
    ],
    "auth" : [
        {
            "_id" : ObjectId("63cab262be029fb68ed002c4"),
            "creator"<EMAIL_ADDRESS>            "created" : ISODate("2023-01-20T15:25:22.470+0000"),
            "modified" : ISODate("2023-01-20T15:25:22.470+0000"),
            "dataset_id" : ObjectId("63e50a0fb6a10a46379335f2"),
            "user_id"<EMAIL_ADDRESS>            "role" : "owner"
        }
    ]
}

listener_jobs
{
    "_id" : ObjectId("63d146bc79a3d39c71d0e0b1"),
    "listener_id" : "ncsa.wordcount",
    "resource_ref" : {
        "collection" : "file",
        "resource_id" : ObjectId("63d140925cda10061fcd6768"),
        "version" : 1.0
    },
    "creator" : {
        "id" : ObjectId("63a32586cdf079dcd6077a1a"),
        "email"<EMAIL_ADDRESS>        "first_name" : "Chen",
        "last_name" : "Wang"
    },
    "parameters" : {

    },
    "created" : ISODate("2023-01-25T15:11:56.615+0000"),
    "started" : null,
    "updated" : ISODate("2023-01-25T14:11:56.000+0000"),
    "finished" : null,
    "duration" : 500.0,
    "latest_message" : "StatusMessage.done: Done processing.",
    "status" : "StatusMessage.done: Done processing.",
    "file_details" : [

    ],
    "auth" : [

    ]
}

listener_job_update
{
    "_id" : ObjectId("63d146bdeac2ff40a3c8c31e"),
    "job_id" : ObjectId("63f6f73302e05c3c0823098e"),
    "timestamp" : ISODate("2023-01-25T14:11:56.000+0000"),
    "status" : "StatusMessage.processing: Downloading file.",
    "listener_job_details" : [
        {
            "_id" : ObjectId("63f6f73302e05c3c0823098e"),
            "listener_id" : "ncsa.wordcount",
            "resource_ref" : {
                "collection" : "dataset",
                "resource_id" : ObjectId("63e50a0fb6a10a46379335f2"),
                "version" : null
            },
            "creator" : {
                "id" : ObjectId("63a32586cdf079dcd6077a1a"),
                "email"<EMAIL_ADDRESS>                "first_name" : "Chen",
                "last_name" : "Wang"
            },
            "parameters" : {

            },
            "created" : ISODate("2023-02-23T05:18:43.047+0000"),
            "started" : null,
            "updated" : null,
            "finished" : null,
            "duration" : null,
            "latest_message" : null,
            "status" : "CREATED"
        }
    ],
    "auth" : [
        {
            "_id" : ObjectId("63cab262be029fb68ed002c4"),
            "creator"<EMAIL_ADDRESS>            "created" : ISODate("2023-01-20T15:25:22.470+0000"),
            "modified" : ISODate("2023-01-20T15:25:22.470+0000"),
            "dataset_id" : ObjectId("63e50a0fb6a10a46379335f2"),
            "user_id"<EMAIL_ADDRESS>            "role" : "owner"
        }
    ]
}

Should we have a script that runs then the database is first created? Are these operations idempotent so that we can run the script every time we startup the services? or do we have to worry about doing it only the first time?

I think we need to import these when database is first created. I need to look into a more programmatic way to import those script. But once it's there we don't need to worry about it when we start up the services

I am going to mark this one approved. I was able to generate the views in Studio 3T and these also work with list access for the other pull request.

for reference, here is the mongo equivalent command without Studio 3T:
db.createView("datasets_view", "datasets", [{$lookup: {"from" : "authorization", "localField" : "_id", "foreignField" : "dataset_id", "as" : "auth"}}]);


Uncaught NotFoundError

Uncaught NotFoundError: Failed to execute 'remove' on 'Element': The node to be removed is no longer a child of this node. Perhaps it was moved in a 'blur' event handler?
C:\Users\BF100400\AppData\Local\atom\app-1.18.0\resources\app\node_modules\text-buffer\lib\text-buffer.js:854
Show Stack Trace
The error was thrown from the vim-mode package. This issue has already been reported.

Thanks for the report!  Closing since this package has been deprecated in favor of vim-mode-plus:
https://github.com/atom/vim-mode#current-status---deprecated-in-favor-of-vim-mode-plus
You may want to give that package a try.


Tor configuration invalid  unable to use the control socket

Select your issue type: (check at least one)

[ ] Bug
[x] Question
[ ] Suggestion
[ ] Other (please describe):

Describe your issue:
I run this command  :
instagram-py -u uname -pl pass.lst
then this problem arises :
[+] Started @ 2019-03-20 21:18:27.213440
fatal error:: Tor configuration invalid or server down :: unable to use the control socket
Ways to Reproduce the issue (optional):

hey man, did you get over this?

sudo killall tor
sudo service tor start

Hey man, I am trying to do this on windows, got everything set up apart
from the tor server, any help?
On Tue, 3 Mar 2020 at 15:33, analjesus<EMAIL_ADDRESS>wrote:

sudo killall tor
sudo service tor start
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/deathsec/instagram-py/issues/17?email_source=notifications&email_token=AOVWKKHYVN4EVVH774S3WXLRFUPNHA5CNFSM4G75WFO2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOENT6POQ#issuecomment-594012090,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AOVWKKFXEAXLISTRLA3RCJ3RFUPNHANCNFSM4G75WFOQ
.

Solution

Anybody found a solution to this? help ;-;

Really late but I updated the stem package (on windows I did pip3 uninstall stem and then pip3 install stem) and got a different error: Connection to host failed, check your connection and tor configuration.

Same issue here. im using termux

fatal error:: Tor configuration invalid or server down :: [Errno 111] Connection refused