Add spatial decomposition task

Task motivation
Spatial decomposition (also often referred to as Spatial deconvolution) is applicable to spatial transcriptomics data where the transcription profile of each capture location (spot, voxel, bead, etc.) do not share a bijective relationship with the cells in the tissue, i.e., multiple cells may contribute to the same capture location. The task of spatial decomposition then refers to estimating the composition of cell types/states that are present at each capture location.
Task description
No response
Proposed ground-truth in datasets

[ ] DestVI3: scRNA-seq is generated based on learn NB parameters from the destVI manuscripts leveraging sparsePCA. Number of cells and cell types present in each spatial spot is computed via combination of kernel-based parametrization of a categorical distribution and the NB model.

[ ] Pancreas (alpha=0.5)11: Human pancreas cells aggregated from single-cell (Dirichlet alpha=0.5).

[ ] Pancreas (alpha=1)11: Human pancreas cells aggregated from single-cell (Dirichlet alpha=1).

[ ] Pancreas (alpha=5)11: Human pancreas cells aggregated from single-cell (Dirichlet alpha=5).

[ ] Tabula muris senis (alpha=0.5)12: Mouse lung cells aggregated from single-cell (Dirichlet alpha=0.5).

[ ] Tabula muris senis (alpha=1)12: Mouse lung cells aggregated from single-cell (Dirichlet alpha=1).

[ ] Tabula muris senis (alpha=5)12: Mouse lung cells aggregated from single-cell (Dirichlet alpha=5).

Initial set of methods to implement

[ ] Cell2location (alpha=20, amortised, hard-coded)2: Cell2location is a decomposition method based on Negative Binomial regression that is able to account for batch effects in estimating the single-cell gene expression signature used for the spatial decomposition step. Note that since batch information is unavailable in this task, here we use either a hard-coded reference, or a negative-binomial learned reference without batch labels. The parameter alpha refers to the detection efficiency prior. Links: Docs.

[ ] Cell2location (alpha=1, reference hard-coded)2: Cell2location is a decomposition method based on Negative Binomial regression that is able to account for batch effects in estimating the single-cell gene expression signature used for the spatial decomposition step. Note that since batch information is unavailable in this task, here we use either a hard-coded reference, or a negative-binomial learned reference without batch labels. The parameter alpha refers to the detection efficiency prior. Links: Docs.

[ ] Cell2location (alpha=20, reference hard-coded)2: Cell2location is a decomposition method based on Negative Binomial regression that is able to account for batch effects in estimating the single-cell gene expression signature used for the spatial decomposition step. Note that since batch information is unavailable in this task, here we use either a hard-coded reference, or a negative-binomial learned reference without batch labels. The parameter alpha refers to the detection efficiency prior. Links: Docs.

[ ] Cell2location (alpha=200, reference hard-coded)2: Cell2location is a decomposition method based on Negative Binomial regression that is able to account for batch effects in estimating the single-cell gene expression signature used for the spatial decomposition step. Note that since batch information is unavailable in this task, here we use either a hard-coded reference, or a negative-binomial learned reference without batch labels. The parameter alpha refers to the detection efficiency prior. Links: Docs.

[ ] Cell2location (alpha=20, NB reference)2: Cell2location is a decomposition method based on Negative Binomial regression that is able to account for batch effects in estimating the single-cell gene expression signature used for the spatial decomposition step. Note that since batch information is unavailable in this task, here we use either a hard-coded reference, or a negative-binomial learned reference without batch labels. The parameter alpha refers to the detection efficiency prior. Links: Docs.

[ ] DestVI3: destVI is a decomposition method that leverages a conditional generative model of spatial transcriptomics down to the sub-cell-type variation level, which is then used to decompose the cell-type proportions determining the spatial organization of a tissue. Links: Docs.

[ ] Non-Negative Matrix Factorization (NMF)7: NMF is a decomposition method based on Non-negative Matrix Factorization (NMF) that reconstructs expression of each spatial location as a weighted combination of cell-type signatures defined by scRNA-seq. It is a simpler baseline than NMFreg as it only performs the NMF step based on mean expression signatures of cell types, returning the weights loading of the NMF as (normalized) cell type proportions, without the regression step. Links: Docs.

[ ] NMF-reg8: NMFreg is a decomposition method based on Non-negative Matrix Factorization Regression (NMFreg) that reconstructs expression of each spatial location as a weighted combination of cell-type signatures defined by scRNA-seq. It was originally developed for Slide-seq data. Links: Docs.

[ ] Non-Negative Least Squares4: NNLS13 is a decomposition method based on Non-Negative Least Square Regression (NNLS). It was originally introduced by the method AutoGenes. Links: Docs.

[ ] Random Proportions13: Random assignment of predicted celltype proportions from a Dirichlet distribution. Links: Docs.

[ ] RCTD5: RCTD (Robust Cell Type Decomposition) is a decomposition method that uses signatures learnt from single-cell data to decompose spatial expression of tissues. It is able to platform effect normalization step, which normalizes the scRNA-seq cell type profiles to match the platform effects of the spatial transcriptomics dataset. Links: Docs.

[ ] SeuratV39: SeuratV3 is a decomposition method that is based on Canonical Correlation Analysis (CCA). Links: Docs.

[ ] Stereoscope6: Stereoscope is a decomposition method based on Negative Binomial regression. It is similar in scope and implementation to cell2location but less flexible to incorporate additional covariates such as batch effects and other type of experimental design annotations. Links: Docs.

[ ] Tangram10: Tangram is a method to map gene expression signatures from scRNA-seq data to spatial data. It performs the cell type mapping by learning a similarity matrix between single-cell and spatial locations based on gene expression profiles. Links: Docs.

[ ] True Proportions13: Perfect assignment of predicted celltype proportions from the ground truth. Links: Docs.

Proposed control methods

[ ] Random Proportions: Random assignment of predicted celltype proportions from a Dirichlet distribution.
[ ] True Proportions: Perfect assignment of predicted celltype proportions from the ground truth.

Proposed Metrics

[ ] r2: R2, or the “coefficient of determination”, reports the fraction of the true proportion values’ variance that can be explained by the predicted proportion values. The best score, and upper bound, is 1.0. There is no fixed lower bound for the metric. The uniform/non-weighted average across all cell types/states is used to summarize performance.

After looking at the task, it seems to me that it's a bit more complicated workflow (i.e. more than just dataset prep -> method -> metric), since you technically have 2 different inputs that need to be processed separately

I added some suggestions on a workflow for spatial decomposition here: https://github.com/openproblems-bio/openproblems-v2/issues/292


Vue SSR not re-rendering parent component

Version
2.6.10
Reproduction link
https://codesandbox.io/s/vue-template-bbyl6?fontsize=14
Steps to reproduce
Run server rendering for the provided application. For some reason the parent component (App.vue) does not change the value of test in  its template.
What is expected?
Browser html to be the same with server html:

  ***initial!!!!!!!!!***

  ***initial!!!!!!!!!***

What is actually happening?
Server HTML doesn't contain change in parent component:

  ***initial***

  ***initial!!!!!!!!!***

Works correct in browser, but something happens in server env. I'm using node v10.16.3

@posva I'm using serverPrefetch, see PostList:20

@posva what event/promise should I use on the parent? I was trying to resolve child's serverPrefetch promise a second later (using additional setTimeout), but the issue still exists:
        setTimeout(() => {
          this.updateTest();
          resolve();
        }, 1500);

you need to wait for the same promise on both serverPrefetch, you can unify the action call so it only happens once with something like https://github.com/posva/p-singleton, but please use the forums to ask questions.


Added support for new bedrock models: Sonnet 3.5 v2.0 and Haiku 3.5

Added support for new (us-based, we don't need other regions for now) models using inference profiles.
New models:

Haiku 3.5  and
Sonnet 3.5 v2.0

Side note: this is my first pull request, so I'm not sure if I've done things right (e.g. I've renamed the file). At any rate, I hope this is useful to you, it's working well for us.
Important note: to test this you need to request model access in each of the regions for a given inference profile.

Hi @mcauthorn , thanks for your contribution!
Support for Claude 3.5 Sonnet v2 has already been added, see commit.
To add support for the new Claude 3.5 Haiku, adding the following lines should be sufficient:
# llm_bedrock_anthropic.py
    register(
        BedrockClaude("us.anthropic.claude-3-5-haiku-20241022-v1:0"),
        aliases=(
            "bedrock-claude-v3.5-haiku",
            "bedrock-haiku-v3.5",
            "bh-v3.5",
        ),
    )

No need to change the filename or anything in the pyproject.toml.


Fix nightly builds

This means we can make sure we are compatible with future php versions

To support php 8.0 we need to upgrade phpunit, or at least allow later versions to be installed.

@Nyholm What do you think about this?
It should allow to keep testing this for php 8.0, without breaking BC due to dependencies

Yeah. I like it. Good job!


Add URL image uploads

Using a simple fetch request, it would be possible to get image data from a URL, rather than uploading files. The user should be able to choose either one.

Solved by 83577bf0174ba0285af7e92e50b6d48935e7d52c.


Aws key rotation log typo

Description
There was a typo in a call to log definition (missing a parameter)
Issues Resolved
None
Contribution Check List

[X ] All tests pass.
[X ] New functionality includes testing.
[ ] New functionality has been documented in the README if applicable
[X ] New functionality has been documented in CHANGELOG.MD

          1 Warning

:warning:
Should this include readme changes

  Generated by :no_entry_sign: Danger


Tempus 310

This PR contains Logo feature, also resolved conflict of datatypes in hsql and postgres by creating separate folders under sql folder for creating schema.

Couple of small Changes,

Remove the print statements(if you need them, add them as loggers)
2.Folder upgrade can be removed which is at the same level at Postgres/hsql is as they both have their upgrades within them
Could you also check at your end if we need a modification in documentation for the change you have done(I could think of any, but others need to be informed to make this change in their respective branches)


Fix getTypeToUse within parent resolvers type

Related https://github.com/dotansimha/graphql-code-generator/issues/2556.

Thanks @blakeembrey . You are right, it should use ResolversParentTypes.
I rebased your PR and fixed the failing tests :)


Make simulator report proper instance state

A recent change to the ping logic to delete container that are not on host has exposed a flaw in our agent simulator. It was completely "forgetting" stopped instances and not reporting them, instead of reporting them as stopped.

This could cause random (but pretty often) test failures that were checking to see if a simulated instance stayed stopped after being stopped because the ping logic was removing it.

LGTM

Dont merge until we have a green build on it. I wanted to see what else failed :-)

LGTM

LGTM


unslick method not implemented yet

won't be implemented, same results can be achieved with unslick prop

@laveesingh can you provide one code example of this? Let's say I want to unslick when I click in a button, how can I do that through props?


List

Screenshot

Simple list showing people's names but it could be a list of anything this is probably an element

@gavinwye Component?

this is something that comes from GOV.UK


Failing to authenticate to FreeNAS

I have a new FreeNAS 11.3-U3.2 setup.  Have the provisioner 2.6 installed.  Haven't been able to get any NFS volumes provisioned.
I went into the provisioner and verified that my freenas.local box does resolve properly, so I dug into the tcpdump.  Curl works fine, but the provisioner does not.  Here's the curl I used from inside the provisioner bash shell:
curl --user root:mypassword http://freenas.local/api/v1.0/storage/dataset/metalgods/k8s/

tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
17:30:52.752752 IP freenas-nfs-provisioner-cffbb8f44-dpvp7.57348 > freenas.local.80: Flags [P.], seq 487283753:487283908, ack<PHONE_NUMBER>, win 219, options [nop,nop,TS val<PHONE_NUMBER> ecr<PHONE_NUMBER>], length 155: HTTP: GET /api/v1.0/storage/dataset/metalgods/k8s/ HTTP/1.1
E....~<EMAIL_ADDRESS>bp.
......P..\)..k......u.....
t.0..Pz>GET /api/v1.0/storage/dataset/metalgods/k8s/ HTTP/1.1
Host: freenas.local
Authorization: Basic cm9vdDpteXBhc3N3b3JkCg==
User-Agent: curl/7.52.1
Accept: */*

17:30:52.809679 IP freenas.local.80 > freenas-nfs-provisioner-cffbb8f44-dpvp7.57348: Flags [P.], seq 1:809, ack 155, win 1028, options [nop,nop,TS val<PHONE_NUMBER> ecr<PHONE_NUMBER>], length 808: HTTP: HTTP/1.1 200 OK
E..\..@.?...
...
bp..P....k...\............
.Pzvt.0.HTTP/1.1 200 OK
Server: nginx
Date: Tue, 23 Jun 2020 17:30:52 GMT
Content-Type: application/json
Transfer-Encoding: chunked
Connection: keep-alive
Vary: Accept, Accept-Language, Cookie
Cache-Control: no-cache
Content-Language: en
Strict-Transport-Security: max-age=0
X-Content-Type-Options: nosniff
X-XSS-Protection: 1

1cf
{"atime": "on", "avail":<PHONE_NUMBER>136, "comments": "Used by the cluster as remote PV storage", "compression": "lz4", "dedup": "off", "exec": "on", "inherit_props": ["compression", "aclinherit", "org.freebsd.ioc:active"], "mountpoint": "/mnt/metalgods/k8s", "name": "metalgods/k8s", "pool": "metalgods", "quota": 0, "readonly": "off", "recordsize": 131072, "refer": 253704, "refquota": 0, "refreservation": 0, "reservation": 0, "sync": "standard", "used": 253704}
0

Unfortunately, this is what I see when snooping the provisioner:
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
17:19:33.667899 IP freenas-nfs-provisioner-cffbb8f44-dpvp7.56334 > freenas.local.80: Flags [P.], seq<PHONE_NUMBER>:2892096082, ack<PHONE_NUMBER>, win 219, options [nop,nop,TS val<PHONE_NUMBER> ecr<PHONE_NUMBER>], length 233: HTTP: GET /api/v1.0/storage/dataset/metalgods/k8s/ HTTP/1.1
E....C<EMAIL_ADDRESS>bp.
......P.a.i.=.e...........
t..{=B;.GET /api/v1.0/storage/dataset/metalgods/k8s/ HTTP/1.1
Host: freenas.local:80
User-Agent: Go-http-client/1.1
Accept: application/json
Authorization: Basic cm9vdDpteXBhc3N3b3JkCg==
Content-Type: application/json
Accept-Encoding: gzip

17:19:33.704049 IP freenas.local.80 > freenas-nfs-provisioner-cffbb8f44-dpvp7.56334: Flags [P.], seq 1:281, ack 233, win 1028, options [nop,nop,TS val<PHONE_NUMBER> ecr<PHONE_NUMBER>], length 280: HTTP: HTTP/1.1 401 Unauthorized
E..L..@.?...
...
bp..P...=.e.a.R...........
=B;.t..{HTTP/1.1 401 Unauthorized
Server: nginx
Date: Tue, 23 Jun 2020 17:19:33 GMT
Content-Type: text/html; charset=utf-8
Transfer-Encoding: chunked
Connection: keep-alive
WWW-Authenticate: Basic Realm="django-tastypie"
Vary: Accept-Language, Cookie
Content-Language: en

0

17:19:33.819787 IP freenas-nfs-provisioner-cffbb8f44-dpvp7.56340 > freenas.local.80: Flags [P.], seq 442110070:442110303, ack<PHONE_NUMBER>, win 219, options [nop,nop,TS val<PHONE_NUMBER> ecr 65777863], length 233: HTTP: GET /api/v1.0/storage/dataset/metalgods/k8s/ HTTP/1.1
E....=<EMAIL_ADDRESS>bp.
......P.Z.v.grd...........
t.......GET /api/v1.0/storage/dataset/metalgods/k8s/ HTTP/1.1
Host: freenas.local:80
User-Agent: Go-http-client/1.1
Accept: application/json
Authorization: Basic cm9vdDpteXBhc3N3b3JkCg==
Content-Type: application/json
Accept-Encoding: gzip

17:19:33.848877 IP freenas.local.80 > freenas-nfs-provisioner-cffbb8f44-dpvp7.56340: Flags [P.], seq 1:281, ack 233, win 1028, options [nop,nop,TS val 65777892 ecr<PHONE_NUMBER>], length 280: HTTP: HTTP/1.1 401 Unauthorized
E..L..@.?...
...
bp..P...grd.Z._...........
....t...HTTP/1.1 401 Unauthorized
Server: nginx
Date: Tue, 23 Jun 2020 17:19:33 GMT
Content-Type: text/html; charset=utf-8
Transfer-Encoding: chunked
Connection: keep-alive
WWW-Authenticate: Basic Realm="django-tastypie"
Vary: Accept-Language, Cookie
Content-Language: en

0

It's unclear to me how I have set it up wrong,   I'm guessing the more restrictive Accept and Accept-Encoding are causing problems, but I'm far from an authentication expert.  Any ideas how to fix this?
JH

Stranger, still, that I added the exact same headers to the curl line as was present, in exactly the same order, and it worked fine.  I have no clue why the server would reject the one from the provisioner but work fine for curl.


[ZEPPELIN-2502] RemoteInterpreterServer hang forever during shutdown

What is this PR for?
There is the chance to have a RemoteServerInterpreter hang forever during shutdown
What type of PR is it?
[Bug Fix]
What is the Jira issue?
[ZEPPELIN-2502]
How should this be tested?
Unit test provided for the fix.
Questions:

Is there breaking changes for older versions?
Does this needs documentation?

@felixcheung good catch, after a second look I can see that while it is waiting on pollEvent it will also delay the sendEvent since it is synchronized too.
I guess all these problems are already solved by Java standard lib, if I have positive feedback I will go substituting eventQueue with a LinkedBlockingQueue and removing all synchronized statements since they become unuseful.

I think you are getting a 👍

Looking at the code it looks quite harmful also the pattern used here, here and here
if anything goes wrong the Thread can stay in wait status potentially forever and operations performed there are all but not transactional.
Could I ask if somebody can at least provide some kind of deadline also for those?
Better will be a little refactoring using something like ConcurrentHashMap for operations.
I can even do the refactoring if I have positive feedbacks, anyhow I believe this PR is ok as is and I open a new issue and new PR in case.

I'm all for refactoring the use of synchronized into using more proper data types instead. It's to easy to get stuck in a [live|dead]lock when doing the synchronized yourself

I have had to fix some escaping problems to make travis happy, and it also required some retries ....
anyhow travis is now happy: https://travis-ci.org/nokia/zeppelin/builds/230331180
I hope this PR is addressed and we can move on.
I still wait hint from @felixcheung and @Leemoonsoo on how to proceed on the rest.
thanks in advance

LGTM

Looking at the code it looks quite harmful also the pattern used here, here and here

Further refactoring on codes pointed out sounds good as well.

am I supposed to do anything else?

@andreaTP Changes on file ZeppelinIT.java are required? And It'll be great if this branch is rebased (or merge) to master and see CI becomes green.

@Leemoonsoo I can reproduce the error locally running on Chrome (i.e. the current encoding of the test doesn't work for me out of the box with chrome), should I separate these modifications in a separate PR?

@andreaTP Yes. separate PR would be more helpful.

@Leemoonsoo sounds good :-) I'll do that tomorrow!
and I will follow up with this PR. Thanks!

Thanks @andreaTP !

I reverted back modifications to concurrency management since it looks like they result in Ui response instability, I'm not sure why this happens, I believe that we can conclude with a clean PR like this.
In case I manage to refactor and use proper java.util.concurrent into remote I will send another PR.
WDYT? @Leemoonsoo @felixcheung

ping

Merge to master and branch-0.7 if no further comment


remove .npmrc

There is no need for this, and it breaks pnpm publish.

@bobheadxi No other repositories have it. Maybe you need it globally on your machine?


useLocalStorage shouldn't include "| undefined" in its type if an initial value is provided

What is the current behavior?
type Value = { hello: boolean };
const [value] = useLocalStorage<Value>('someKey', { hello: false });

value is now type Value | undefined, even though we know there's a correct initialValue provided.
Steps to reproduce it and if possible a minimal demo of the problem. Your bug will get fixed much faster if we can run your code and it doesn't have extra dependencies other than react-use. Paste the link to your JSFiddle or CodeSandbox example below: https://codesandbox.io/s/competent-mcnulty-o7nde?file=/src/App.tsx
What is the expected behavior?
value should be type Value.
A little about versions:

OS: N/A
Browser (vendor and version): N/A
React: N/A
react-use: 15.3.3
Did this worked in the previous package version? Yes; regression introduced in https://github.com/streamich/react-use/pull/835 (+1 to the feature add in that PR though! 😄 )

Hi: I am here as part of HacktoberFest. Can I fix this issue

value is now type Value | undefined, even though we know there's a correct initialValue provided.

It's initialValue - not defaultValue.
I guess it's like this because there's remove function, which implies the possibility of undefined.

I think there should be no remove function - set(undefined) should act like remove currently does.
Of course this like would have to be removed: https://github.com/streamich/react-use/blob/db07ab6/src/useLocalStorage.ts#L55
And then we could replace all T | undefined with T, and it'd solve the problem.

@JoshuaKGoldberg I fixed this issue in August, but still waiting for my PR #1438 to be accepted. 😕
@kamilmielnik I also considered letting set(undefined) do the same as the current remove function, but when I tried to do it, I think it was the the types that became kind of an issue. I feel having a separate remove function makes the intent clearer anyways though. That's how the underlaying storage works as well; you have separate setItem and removeItem.

Hi all! @react-hookz/web, the new library by one of react-use's former maintainers (background here and here) has this fixed. Check out the docs for useLocalStorageValue.
For those interested, there's an official migration guide for migrating from react-use to @react-hookz/web.
Hope this helps!

Closing out my old issues I no longer have context on. If anybody else is still seeing this, I'd encourage you to file a new issue with more information. Cheers!


TdsParserStateObject.bTmp should be releasable

The bTmp variable is used as a temporary buffer for various internal operations, if it is null or too small a larger byte[] is allocated and assgined to it. There is nothing that sets it to null. Over time if a process has operations that deal with large strings or byte arrays the bTmp of all pooled state objects will grow to be a sometimes significant amount of memory, i've seen 2/3 of a process allocated memory in bTmp.
We should look at providing a way to trim these buffers to be good citizens to the rest of the process. We should see if this can be done without adverse effects on overall throughput. Approaches I can think of:

null the buffer on return to the pool
detect gc pressure using Gen2GCCallback or something similar and null
rely on pool culling on timer and null if the parent has remained pooled for a certain minimum time.

1 sounds like the simplest solution. Would it make sense to null if it's greater than a certain size? To prevent any change in performance for very frequent, small queries.

That would work but would regress perf for frequent large queries by the time taken to allocate new arrays each time. I don't know how big that hit would be which is why i mentioned the other two options so we keep them around unless they're clearly not being used.


Acceleration of the balance bootstrapping process

Hi! is there a way (e.g. some rosetta-cli config option) to speed up the balance boostrapping process? Unfortunately, to bootstrap 85,000 balances it takes more than 10 hours (on a quite performant server).
To Reproduce

generate a big bootsrap_balances.json file with 85,000 accounts for example
start the rosetta-cli data check

Expected behavior
I assume that the bootstrap process is faster. IOTA nodes for example need to be boostrapped with the same data and it takes (when optiomized) less than a minute.
Additional context
IOTA full-nodes are able to start up from a more recent block instead of having to synchronize from genesis. This can be achieved by booting the IOTA full-node with a so called snapshot. A snapshot represents the ledger state at a certain block.
Starting from that loaded state, the IOTA node then syncs with it's peers.
To test our implementation we need to boot rosetta-cli with the same snapshot data (therefore we do have such a big bootstrap_balances.json file).

Hi @rufsam,
Is this happening even though you have a memory limit disabled [setting] (https://github.com/coinbase/rosetta-cli/blob/master/configuration/types.go#L407) turned off?

Hi @shrimalmadhur, this are the used configuration values:
{
  "network": {
    "blockchain": "iota",
    "network": "chrysalis-mainnet"
  },
  "online_url": "http://localhost:3030",
  "data_directory": "rosetta-cli-db",
  "http_timeout": 10,
  "max_retries": 5,
  "max_sync_concurrency": 5,
  "tip_delay": 15,
  "memory_limit_disabled": true,
  "data": {
    "historical_balance_enabled": false,
    "reconciliation_disabled": false,
    "active_reconciliation_concurrency": 5,
    "inactive_reconciliation_concurrency": 5,
    "pruning_disabled": false,
    "bootstrap_balances": "bootstrap_balances.json",
    "end_conditions": {
      "reconciliation_coverage": {
        "coverage": 0.95,
        "from_tip": true
      }
    }
  }
}

I see. thanks for the info. Is your rosetta code public? If yes, I would like to repro this with your bootstrap file. It would also help me to check where we can it more efficient.

This would be really great. Many thanks @shrimalmadhur!
I have just produced a release with all the latest changes, you can find the repository here: https://github.com/iotaledger/rosetta-iota
If there is anything I can help you with, just tell me!


Don't depend on ZooKeeper for coordination

Expected behavior
Pulsar should be able to function without ZooKeeper or provide other backends for Etcd and Consul
Actual behavior
ZooKeeper is required for Pulsar's operation.
One of the major hurdles of using Kafka in production is ZooKeeper. The component is a source for many operational problems and thus not suitable for use in small startups/shops.
If this project wants to succeed it should provide us with a way to replace this dependency with another one or implement all coordination on it's own (as Mesos is working on now)

Are you in an environment that is virtualized? When ZK gets confused are you having migrations? I've seen a case where Solr 5.5 clusters which also use ZK got confused in those scenarios especially when logging was configured off host. The solution required 5 ZK nodes to 3 Solr nodes. along with not using mounted disks for Jetty logs in front of the Solr nodes.
Also I think that the ZK dependency in Pulsar is due to the use of BookKeeper...

Bookkeeper seems to be Etcd ready: https://github.com/apache/bookkeeper/issues/1639
Kafka also already supports etcd.
Is it time for Pulsar to be reassessed for adding etcd support?

Pulsar is an amazing technology, and zookeeper is not.
Please remove the zookeeper dependency.

I'd like this, too. We use Etcd.
How solid is zetcd? It's not been updated since January, the last pre-release was 16 months ago, and the last release was 2 years ago. Readme doesn't have any information on whether it's production-ready. Not too promising.

When I stumbled upon Plusar I died twice. First I read the description and, coming from a Kafka / Kubernetes background, I thought man these guys really tick everything I could wish for. After that, I clicked the Docs page and when I discovered those nasty ZK boxes in the diagram the dream died as fast as it was emerging.
I can understand the argument of not repeating yourself and it may look like a good idea to have that complicated quorum stuff sorted out with a separate microservice (ZK). However, If you produce the situation that the state of multiple services are interconnected and can desynchronize, its an operational nightmare. At least in a containerised and managed environment like Kubernetes.
Kafka has (finally) accepted this reality and is moving away from an external solution to embed quorum (KIP-500). By having the Quorum State and Broker Data state in the same location, desynchronisation are no longer possible and operations are very much simplified.
So my two cents would be to not make ZK replaceable by something other but remove the need for an external solution altogether.

My understanding zookeeper has an issue in terms of "quorum" and "split Brain"
my finding correct me if I am wrong lets take we have 5 servers. The voting to elect the leader is based on the majority of the servers. the number of servers must accept specific server to become the leader, the other becomes the followers
So there is this bad thing that may happen called "split brain". A split brain is simply this, as far as I understand: The cluster of 5 servers splits into two parts, or let's call it "server teams", with maybe one part of 2 and the other of 3 servers. This is really a bad situation as if both "server teams" must execute a specific order how would you decide which team should be preferred? So it is really important to know what "server team" is still relevant and which one can/should be ignored.
When one server in {3,4,5} realizes the leader is gone, it triggers an election. Once a new leader is elected, it starts serving client requests normally. The other partitioned cluster {1,2} won't serve any client request and will be stuck in the election until the partition is resolved. So it waits till the issue resolved and the cluster runs normal again.
Currently, pulsar uses  Apache zookeeper to store its metadata. Data such as the location of partitions and the configuration of topics are stored outside of Kafka itself, in a separate Zookeeper cluster. so is there a plan to break this dependency and bring metadata management into Pulsar itself.

@kjaggann split brain is a situation where the minority "server team" accepts write/read command and leads to a situation where the two clusters becomes inconsistent (in terms of data). It does not occur trivially from just 2 servers being cut off (say AWS AZ connection outage) from the rest of the cluster. That's a clear cut case of minority / not holding the quorum. One potential scenario is when servers have partial connections to rest of the cluster. Say servers of ABCDE. C can talk to the whole cluster, but AB can't talk to DE. AB and DE then both elect a leader amongst themselves and function independently thinking they both have majority with C.
Even the above scenario does not cause split brain with ZK. ZK is a very robust and well tested software. Those nuances are already well tested and handled.
Have a read of this https://aphyr.com/posts/291-jepsen-zookeeper and other related posts to understand more about consistency / quorum. You'll also see why managing all that stuff inside Pulsar ..... might not be the best idea.

Also bear in mind you still have to maintain 3 separate machines with a special role that function just for raft, so it’s not that different. Kafka specifically adapted Raft to match their needs. IMO it just makes more difficult to fix issues.

I hope Pulsar and BookKeeper will ditch Zookeeper and come out with their internal built-in cluster coordination like Raft, what new Kafka 2.8 has developed KIP-500 and also how Docker Swarm cluster coordinate with each other without using any external system like Etcd, Consul or Zookeeper.

hope Pulsar devs don't make the same mistake Kafka did

I thought thats what Pulsar is :D
100% in agreement with @SleeperSmith ectd/consul/zk as plugable options seems consistent with the Pulsar's architecture. This would not only avoid the concern shared about kafka, but let operators choice the the tech they are most competent in .... isn't that the whole concern about zk, that some (including me) lack the ability to operate and fix it where as other options have bigger communities, more engineers that are familiar, and various commercial support options?  The way zk is used in Pulsar isnt necessarily a technical problem right?

FYI there's https://github.com/apache/pulsar/wiki/PIP-45%3A-Pluggable-metadata-interface which will later give the choice of choosing between multiple options.

@lhotari  is great to hear that Pulsar team is working towards a solution.  That is a good start though.  Having a choice to select is always better than none.  Kudos to you guys.  Really hope that the team from Apache BookKeeper will have the same plan going forward.

@lhotari is great to hear that Pulsar team is working towards a solution. That is a good start though. Having a choice to select is always better than none. Kudos to you guys.

Yes, Bookkeeper already supports a pluggable metadata store.

@lhotari is great to hear that Pulsar team is working towards a solution. That is a good start though. Having a choice to select is always better than none. Kudos to you guys.

Yes, Bookkeeper already supports a pluggable metadata store. https://bookkeeper.apache.org/bps/BP-29-metadata-store-api-module/

That is wonderful! Can we use that in Pulsar?
Do we just need the documentation for it?

kafka got rid of ZK and doesnt depend on etcd or consul either(way to go).

We love etcd too due to it simplification in deployment and easier to maintain than Zookeeper. We are using both and etcd is definitely take us less time to control and maintain it.

I believe this issue can be resolved by PIP-45 delivered and we implemented etcd as the metadata store backend.


[GLUTEN-3547][VL] [Minor] Remove the space in taskid name

What changes were proposed in this pull request?
The Velox TableWrite node generates the write file name based on the task ID name. However, in this case, Gluten passed the task ID name as Gluten [Stage: 3 TID: 3], which contains a space in the file name. Spark cannot read files with spaces in their names. This pull request resolves the issue by removing the space in the task ID. And the taskid name will changed from Gluten [Stage: 3 TID: 3] to Gluten_StageId_3_TaskId_3.
How was this patch tested?
no need

@JkSelf  Curious, can you show the final parquet file name ? thank you

@JkSelf Curious, can you show the final parquet file name ? thank you

@ulysses-you The final file name is Gluten_StageId_3_TaskId_3_0_1_cebc96e0-792a-4b68-9adc-240167474406. And the code is here.

please use "TID", it helps to filter log from executor.

@ulysses-you @Yohahaha Do you have any further comment?

===== Performance report for TPCH SF2000 with Velox backend, for reference only ====

query
log/native_3797_time.csv
log/native_master_11_21_2023_b40a5f094_time.csv
difference
percentage

q1
33.72
33.71
-0.006
99.98%

q2
24.45
24.92
0.476
101.95%

q3
37.12
37.63
0.514
101.39%

q4
36.99
35.99
-0.996
97.31%

q5
68.75
70.48
1.723
102.51%

q6
6.80
7.11
0.304
104.47%

q7
84.33
84.08
-0.255
99.70%

q8
87.18
87.02
-0.156
99.82%

q9
123.97
124.24
0.270
100.22%

q10
46.35
46.06
-0.284
99.39%

q11
19.92
19.24
-0.681
96.58%

q12
24.99
25.38
0.392
101.57%

q13
46.98
45.84
-1.140
97.57%

q14
15.84
18.33
2.488
115.71%

q15
29.50
28.80
-0.701
97.62%

q16
15.39
15.37
-0.019
99.88%

q17
99.11
100.95
1.836
101.85%

q18
148.13
146.37
-1.765
98.81%

q19
12.92
12.89
-0.032
99.76%

q20
27.04
28.27
1.234
104.56%

q21
225.76
222.09
-3.666
98.38%

q22
12.81
12.99
0.182
101.42%

total
1228.04
1227.75
-0.283
99.98%


Develop

Hi Tonya,  the changes are in and I think a new proven-cluster is ready to go.

This change is

Updates not included


create anonymous slug

Fixes https://github.com/opencollective/opencollective/issues/739

Coverage remained the same at 75.732% when pulling 518bddae21a73191c94f0d42cf29d4d07840e426 on fix739 into 87f60542185a6c6212a6639591414e4b419fb2a9 on master.


Auth rework

Please resolve all conflicts :)

You should probably test it first as well

why do you do this to me


Added path prefixes, optional help, response headers, regex lists, method lists and reflected origins

This software is great but to make it useful to me I need a few extra features. if you would please consider merging this pull request that would be great!

Coverage decreased (-6.5%) to 93.496% when pulling 0c2fbd16ea0c72592731d3d297f0dc1b2dfc25ca on PropellerAero:PR-Changes into efc13eec1fedaf01d1bb44a43d995228ce86fd61 on Rob--W:master.

Coverage decreased (-2.8%) to 97.154% when pulling 6310d3c9cb40bca2c75e0962011a67e0b61a9300 on PropellerAero:PR-Changes into efc13eec1fedaf01d1bb44a43d995228ce86fd61 on Rob--W:master.

Coverage remained the same at 100.0% when pulling 9f95b03c7f47317eea59455a131642c89e243557 on PropellerAero:PR-Changes into efc13eec1fedaf01d1bb44a43d995228ce86fd61 on Rob--W:master.

Thanks for this PR and your efforts to maintain 100% test coverage!
Some of the functionality seems useful, others not as much. I'll start with the things that I'd rather not see:

pathPrefixes - This "feature" is too obscure and can easily be implemented using req.url = req.url.replace(/^\/pathprefix/, '/'); (see https://github.com/Rob--W/cors-anywhere/issues/51#issuecomment-247844133).
originRegexWhitelist and originRegexBlacklist - These are not needed, the same functionality can be achieved with the checkRateLimit option.

And some questions about the other options before I do a code review:

showHelp - Drop this option, and modify showUsage to reply 404 if the help path is unset / an empty string.
wildcardOrigin - Why do you want to use explicit origins instead of a wildcard? In the past it was an explicit origin, but I changed it to a wildcard to avoid issues with caching (see #6 and also #8).
setResponseHeaders - What is the use case?
allowedMethods - What is the use case?

Thanks for taking the time to look at my PR! To address your questions:

pathPrefixes - My goal here is to accept and correctly proxy http://proxied.domain.com from http://cors.anywhere.com/http://proxied.domain.com, http://cors.anywhere.com/my/proxy/path/http://proxied.domain.com and
http://cors.anywhere.com/other/proxy/path/http://proxied.domain.com. Requests to http://cors.anywhere.com/bad/path/http://proxied.domain.com should just 404. Are you suggesting that I start up another http server and forward requests onto cors-anywhere or that I re-implement the feature using regex?

originRegexWhitelist and originRegexBlacklist - checkRateLimit seems like a fairly non-obvious place to put such a setting. Would it not be better to be more explicit about this? Alternatively I can alter originWhitelist and originBlacklist to accept both string and regex?

showHelp - can do.

The last three options come from a security standpoint. Exposing an open proxy has obvious security risks and so being able to lock down its functionality is important.

wildcardOrigin - The caching issue is annoying and I've come up against it before but it can also mean that if an allowed origin accesses a resource and the * is cached then an origin that is not allowed can access that resource via the cache. Setting explicit origins prevents that. To get around the caching issue we need the setResponseHeaders option.

setResponseHeaders - Setting the Vary header to Origin, means that new origins will go back to the server to get resource rather than checking the cache. This results in more hits to the server but in my case this is ok.

allowedMethods - Open proxies allow anyone to mask their activities with your name and so being able to limit the functionality of the proxy to only that which is required is important. In  my case I only need to support GET requests.

Let me know your thoughts and I'll work on making the required changes.

pathPrefixes - My goal here is to accept and correctly proxy http://proxied.domain.com from http://cors.anywhere.com/http://proxied.domain.com, http://cors.anywhere.com/my/proxy/path/http://proxied.domain.com and
http://cors.anywhere.com/other/proxy/path/http://proxied.domain.com. Requests to http://cors.anywhere.com/bad/path/http://proxied.domain.com should just 404. Are you suggesting that I start up another http server and forward requests onto cors-anywhere or that I re-implement the feature using regex?

What are those /other/proxy/path components doing in the URL? And yes, I'm probably suggesting that you put a server in front of CORS anywhere, like I showed in #43.

originRegexWhitelist and originRegexBlacklist - checkRateLimit seems like a fairly non-obvious place to put such a setting. Would it not be better to be more explicit about this? Alternatively I can alter originWhitelist and originBlacklist to accept both string and regex?

Blacklisting is equivalent to ratelimiting to 0 requests per period. The lists of origin strings is for simple use cases, any other complex logic can be interim checkRateLimit (that's why I added that option).

showHelp - can do.

Thanks!

The last three options come from a security standpoint. Exposing an open proxy has obvious security risks and so being able to lock down its functionality is important.

Agreed. If you're concerned about security, do not blindly follow redirects internally with the maxRedirects option.

wildcardOrigin - The caching issue is annoying and I've come up against it before but it can also mean that if an allowed origin accesses a resource and the * is cached then an origin that is not allowed can access that resource via the cache. Setting explicit origins prevents that. To get around the caching issue we need the setResponseHeaders option.
setResponseHeaders - Setting the Vary header to Origin, means that new origins will go back to the server to get resource rather than checking the cache. This results in more hits to the server but in my case this is ok.

If this is the only use case, add an option to reply with origin+vary (not all response headers).

allowedMethods - Open proxies allow anyone to mask their activities with your name and so being able to limit the functionality of the proxy to only that which is required is important. In my case I only need to support GET requests.

Sounds good.

Coverage decreased (-0.4%) to 99.552% when pulling 7a270288f52dedf3e1bb7ad1093e0079a2d836c5 on PropellerAero:PR-Changes into efc13eec1fedaf01d1bb44a43d995228ce86fd61 on Rob--W:master.

Coverage remained the same at 100.0% when pulling b262820bcddb8f1a2dbdb201eebfc6264836215a on PropellerAero:PR-Changes into efc13eec1fedaf01d1bb44a43d995228ce86fd61 on Rob--W:master.

Coverage remained the same at 100.0% when pulling f587e5803ccbdfb3b2c4264c547d556c671c03f4 on PropellerAero:PR-Changes into efc13eec1fedaf01d1bb44a43d995228ce86fd61 on Rob--W:master.

Coverage remained the same at 100.0% when pulling f587e5803ccbdfb3b2c4264c547d556c671c03f4 on PropellerAero:PR-Changes into efc13eec1fedaf01d1bb44a43d995228ce86fd61 on Rob--W:master.

Coverage remained the same at 100.0% when pulling 27f33376266de4de3684dbf467b5e4693154a2e2 on PropellerAero:PR-Changes into efc13eec1fedaf01d1bb44a43d995228ce86fd61 on Rob--W:master.

Hey @Rob--W, this is ready for you to take a look at now. :)

Coverage remained the same at 100.0% when pulling a23503ae54e35ff7aeedce64c14a7ab3ed4659ee on PropellerAero:PR-Changes into efc13eec1fedaf01d1bb44a43d995228ce86fd61 on Rob--W:master.

@Rob--W. Done. Let me know what you think.

Hi @keattang, do you want to address the above issues? Thanks!

@keattang Friendly reminder in case you have forgotten about this PR. Just a few minor changes and it is good to go.

@Rob--W , When/how should we merge this feature?
When I try to use the proxy, in Safari, I'll get this error
"Cannot use wildcard in Access-Control-Allow-Origin when credentials flag is true." for "https://cors-anywhere.herokuapp.com/https://aqueduct.microstrategy.com/MicroStrategyLibrary/api/config/authModes"
But in Chrome, I'll get the CORS error because it's OPTIONS request before Get.
Hope this code can work.
By the way, I have a question that how can I start the server.js in https? I tried to add the HttpsOptions in js, or start from http-server, but both of them do not work as expected.
Thanks

This PR is in an unmergeable state, so I'm closing it.
I'm not allowing cookies for various reasons, see https://github.com/Rob--W/cors-anywhere/pull/148#issuecomment-452465262


Use react-native 0.44.0 and react-native-vector-icons 4.1.0 init a new project, cause a link error

Use react-native 0.44.0 and react-native-vector-icons 4.1.0 init a new project, cause a link error:
ld: library not found for -lRNVectorIcons-tvOS
clang: error: linker command failed with exit code 1 (use -v to see invocation)
I manually remove RNVectorIcons-tvOS libarary in Build Phases, it becomes ok.
I think it's a small bug in the lastest version about react-native-vector-icons 4.1.0.
So I give a feedback here.
Additionally, I try 4.0.1 version at now, it's ok.

Same here. Removing  RNVectorIcons-tvOS on Build Phases as @windy said, works as expected.

Also was able to avoid this issue by removing the tvOS reference in Build Phases.

Duplicate of https://github.com/oblador/react-native-vector-icons/issues/461. Waiting on fix in react-native core

Since this seems to affect a big number of users and the tvOS users are quite small, I've reverted the commit adding support for tvOS for now so we don't have to wait on this being fixed in react native core. Upgrading to 4.1.1 should fix this!

@oblador got it, thanks for your work.

This version matters.Thank you.


Update fabric-gateway-java to 2.2.2

Updates org.hyperledger.fabric:fabric-gateway-java from 2.2.1 to 2.2.2.
GitHub Release Notes - Version Diff
I'll automatically update this PR to resolve conflicts as long as you don't change it yourself.
If you'd like to skip this version, you can just close this PR. If you have any feedback, just mention me in the comments below.
Configure Scala Steward for your repository with a .scala-steward.conf file.
Have a fantastic day writing Scala!

Ignore future updates
Add this to your .scala-steward.conf file to ignore future updates of this dependency:
updates.ignore = [ { groupId = "org.hyperledger.fabric", artifactId = "fabric-gateway-java" } ]

labels: library-update, semver-patch

Superseded by #224.


How to open country picker dialog without adding in xml

how can I show direct dialog without adding in XML. i want to show country picker dialog on click of particular view.
binding.btnSelectCountry.setOnClickListener {
            hideKeybord()
            //open dialog
}

how can i show runtime programmatically

okay got solution
 countryCodePicker.showCountryCodePickerDialog()
please add details in the document below
how to check isDialog open
how to close dialog runtime
how to open runtime


DockerBuildError

I followed the README and completed the installation/configuration, but faced this error when uploading instructor notebook:
2018-08-09 14:17:25,862 ERROR 31272 [xblock_jupyter_graded.xblock_jupyter_graded] xblock_jupyter_graded.py:264 -
Traceback (most recent call last):
  File "/opt/openedx/venv/local/lib/python2.7/site-packages/xblock_jupyter_graded/xblock_jupyter_graded.py", line 244, in handle_instructor_nb_upload
    str(self.course_id), str(self.location), f)
  File "/opt/openedx/venv/local/lib/python2.7/site-packages/xblock_jupyter_graded/nbgrader_utils.py", line 54, in generate_student_nb
    max_score = _run_assign_container(f.filename, course, unit)
  File "/opt/openedx/venv/local/lib/python2.7/site-packages/xblock_jupyter_graded/nbgrader_utils.py", line 79, in _run_assign_container
    build_container_if_not_exists(course_id)
  File "/opt/openedx/venv/local/lib/python2.7/site-packages/xblock_jupyter_graded/nbgrader_utils.py", line 213, in build_container_if_not_exists
    manager.build_container()
  File "/opt/openedx/venv/local/lib/python2.7/site-packages/xblock_jupyter_graded/container_manager.py", line 134, in build_container
    raise DockerBuildError(out.decode('utf-8'))
DockerBuildError

OS: Ubuntu 16.04 x64
OpenEdx version: Ginkgo

Your docker image didn't build. Did you upload a requirements.txt file?

Closing for now, @KaiserW , but feel free to reopen!


Request for dependencies used for Transfer Hook and Metadata Interface programs

Requesting the following dependencies so that examples can be run using Solana Playground
Rust Crates:
spl-token-metadata-interface = "0.2.0"
spl-type-length-value = "0.3.0"
spl-pod = "0.1.0"
spl-transfer-hook-interface = "0.4.1"
spl-tlv-account-resolution = "0.4.0"

TS library

@solana/spl-token-metadata

Example usage:

Anchor Transfer Hook program
Anchor Metadata program

Thanks, will add after https://github.com/solana-playground/solana-playground/issues/165 — sorry, this is taking a while.

Added all of them in https://github.com/solana-playground/solana-playground/commit/22d0028294b2e7ac68e126113d334369912616a4 but not some of them have different versions(compared to the ones you've given) due to spl-token-2022 dependency not yet being on 1.0.0. Can see the versions used in supported-crates.json.
Again, sorry for the delay!


k8sutil moved to internal package

Bug Report

What did you do?
Upgrade operator-sdk from 0.18.2 -> 1.0.0
What did you expect to see?

Hi, we were in the process of upgrading our golang operator from 0.18.x to 1.0.0, and noticed  k8sutil has been moved to an internal package, we removed its dependency from our codebase but, some of our other dependency like operator-marketplace still imports it causing our upgrade to halt. So I just wanted to ask is there any specific reason for why it is moved to an internal package now? I think it was used by many other projects as well
What did you see instead? Under which circumstances?

Environment
Operator type:

@Kaitou786 k8sutil is obsolete for use in operator projects as of SDK 1.0 because projects scaffolded by Operator SDK 1.0 no longer use it. In fact, there are no longer any public APIs in the Operator SDK repo.
The helper libraries that did exist and are still relevant have been moved to github.com/operator-framework/operator-lib.
Your project should no longer depend directly on operator-sdk, so you should be able to continue importing operator-marketplace and allow it to continue importing an older version of Operator SDK to resolve its import for k8sutil.
Can you provide more details about the issue you're running into?

What does your go.mod file contain?
What command are you running that is resulting in an error?
What is the output of that command?

Ah okay,
content of go.mod contents:
go 1.14

require (
	github.com/RHsyseng/operator-utils v0.0.0-20200108204558-82090ef57586
	github.com/coreos/prometheus-operator v0.41.1
	github.com/cucumber/gherkin-go/v11 v11.0.0
	github.com/cucumber/godog v0.10.0
	github.com/cucumber/messages-go/v10 v10.0.3
	github.com/go-logr/logr v0.1.0
	github.com/gobuffalo/packr/v2 v2.7.1
	github.com/google/uuid v1.1.1
	github.com/imdario/mergo v0.3.9
	github.com/infinispan/infinispan-operator v0.0.0-20200820122451-4ffef65c9410
	github.com/keycloak/keycloak-operator v0.0.0-20200826070637-fa5e7988b819
	github.com/machinebox/graphql v0.2.2
	github.com/matryer/is v1.4.0 // indirect
	github.com/onsi/ginkgo v1.12.1
	github.com/onsi/gomega v1.10.1
	github.com/openshift/api v3.9.1-0.20190924102528-32369d4db2ad+incompatible
	github.com/openshift/client-go v3.9.0+incompatible
	github.com/operator-framework/operator-lifecycle-manager v0.0.0-20200321030439-57b580e57e88
	github.com/operator-framework/operator-marketplace v0.0.0-20200826070533-bb7db1173eb2
	github.com/operator-framework/operator-sdk v1.0.0
	github.com/prometheus/common v0.10.0
	github.com/spf13/cobra v1.0.0
	github.com/stretchr/testify v1.6.1
	go.uber.org/zap v1.14.1
	gopkg.in/src-d/go-git.v4 v4.13.1
	gopkg.in/yaml.v2 v2.3.0
	k8s.io/api v0.18.6
	k8s.io/apiextensions-apiserver v0.18.6
	k8s.io/apimachinery v0.18.6
	k8s.io/client-go v12.0.0+incompatible
	k8s.io/kube-openapi v0.0.0-20200410145947-61e04a5be9a6
	sigs.k8s.io/controller-runtime v0.6.2
)

replace (
	k8s.io/api => k8s.io/api v0.18.6
	k8s.io/apiextensions-apiserver => k8s.io/apiextensions-apiserver v0.18.6
	k8s.io/apimachinery => k8s.io/apimachinery v0.18.6
	k8s.io/apiserver => k8s.io/apiserver v0.18.6
	k8s.io/cli-runtime => k8s.io/cli-runtime v0.18.6
	k8s.io/client-go => k8s.io/client-go v0.18.6
	k8s.io/cloud-provider => k8s.io/cloud-provider v0.18.6
	k8s.io/cluster-bootstrap => k8s.io/cluster-bootstrap v0.18.6
	k8s.io/code-generator => k8s.io/code-generator v0.18.6
	k8s.io/component-base => k8s.io/component-base v0.18.6
	k8s.io/cri-api => k8s.io/cri-api v0.18.6
	k8s.io/csi-translation-lib => k8s.io/csi-translation-lib v0.18.6
	k8s.io/kube-aggregator => k8s.io/kube-aggregator v0.18.6
	k8s.io/kube-controller-manager => k8s.io/kube-controller-manager v0.18.6
	k8s.io/kube-proxy => k8s.io/kube-proxy v0.18.6
	k8s.io/kube-scheduler => k8s.io/kube-scheduler v0.18.6
	k8s.io/kubectl => k8s.io/kubectl v0.18.6
	k8s.io/kubelet => k8s.io/kubelet v0.18.6
	k8s.io/legacy-cloud-providers => k8s.io/legacy-cloud-providers v0.18.6
	k8s.io/metrics => k8s.io/metrics v0.18.6
	k8s.io/sample-apiserver => k8s.io/sample-apiserver v0.18.6
)

// Required by Helm
replace github.com/docker/docker => github.com/moby/moby v0.7.3-0.20190826074503-38ab9da00309

// fix Azure bogus https://github.com/kubernetes/client-go/issues/628
replace github.com/Azure/go-autorest => github.com/Azure/go-autorest v13.3.2+incompatible // Required by OLM

2: Running go mod tidy
3: output of the command:
        github.com/operator-framework/operator-marketplace/pkg/apis/operators/shared imports
        github.com/operator-framework/operator-sdk/pkg/k8sutil: module github.com/operator-framework/operator-sdk@latest found (v1.0.0), but does not contain package github.com/operator-framework/operator-sdk/pkg/k8sutil```

Can you try removing github.com/operator-framework/operator-sdk v1.0.0 from your go.mod and retry go mod tidy?

Thanks, it replaced it with an earlier version of operator-sdk, github.com/operator-framework/operator-sdk v0.18.2

Also, have you read the 1.0 migration guide? Among some other great benefits to using the new project layout, you may be able to drastically reduce the content of your go.mod file by re-scaffolding your project with operator-sdk init.

@joelanford, actually I was following that guide only, and re-scaffolded by operator-sdk init, but ended up writing a lot in go.mod due to the pinning of k8s deps to a particular version, before that I was getting v0.0.0 unknown revision error.

Ok cool! Yeah, that must mean that one of your project's dependencies still pulls in an older version of k8s.io/kubernetes, which causes that (really annoying :slightly_smiling_face:) error.
Sounds like you're good to go now though, so I'll close this one. If you encounter anything else while migrating, definitely open more issues!

@joelanford just out of curiosity does the olm_cataloge dir removed from this new structure?

It is, things are restructured a bit. Generated packagemanifests and bundles will end up in the project root directory.
There are some base files that are used to generate packagemanifests and bundles in config/manifests.


⚠ Bofrost.es has degraded performance

In 80e26fd, Bofrost.es (https://www.bofrost.es) experienced degraded performance:

HTTP code: 200
Response time: 1495 ms

Resolved: Bofrost.es performance has improved in 95476e9 after 1 hour, 55 minutes.


Once training has started, print to cloudshell CLI the url for viewing video stream and log analysis

Also link to s3 bucket where logs will begin to upload for use in guru tool


Do we actually need to execute the notebooks twice?

I think execution and conversion to HTML can happen in one jupyter-nbconvert call.

Probably not. I copied from the scipy tutorial script.


[NeuralChat] Fix Qwen chat model output undesirable on HPU issue caused by input padding

Type of Change
bug fix
API not changed
Description
Fix Qwen chat model output undesirable on HPU issue caused by input padding.
JIRA ticket: xxx
Expected Behavior & Potential Risk
pass test
How has this PR been tested?
UT
Dependency Change?
no library dependency introduced or removed

could you elaborate more on the issue? this change will cause recompile graph for hpu if input prompt length change.

could you elaborate more on the issue? this change will cause recompile graph for hpu if input prompt length change.

For Qwen/Qwen-7B-Chat as an example, when gave model the prompt "你好", if the input token is padded, it outputs nothing, if input token is not padded, it outputs "你好！有什么我能帮助你的吗？".
 Qwen specific?

it is weird that Qwen does not support left padding in generation. if you batch 2 different lengths of prompt together and text generation with bs=2, what will happen with the smaller length of prompt.

it is weird that Qwen does not support left padding in generation. if you batch 2 different lengths of prompt together and text generation with bs=2, what will happen with the smaller length of prompt.

I've checked Qwen's repo and found that for batched generation with padding, we need to set pad token id to 151646 manually, after this, padded input also works. I will change model.generation_config.pad_token_id instead of padding= "max_length".

so, the generation_config.json in Qwen is incorrect.

so, the generation_config.json in Qwen is incorrect.

Yes, I think so, also I find that output of Qwen/Qwen-14B-Chat is undesirable even with 151646 as pad_token_id.
model output of padded input:

model output of unpadded input:


[Doc] The logo file in the lower left corner of the page is 404

Search before asking

[X] I had searched in the issues and found no similar issues.

Documentation Related
I found that the logo file in the lower left corner of the page is not loaded correctly：

After opening the browser console, I found that the file is 404：

Was this file renamed or moved to another directory? I thought I might be able to try to fix this. :)
Are you willing to submit PR?

[X] Yes I am willing to submit a PR!

Code of Conduct

[X] I agree to follow this project's Code of Conduct *

@Createsequence I've transferred this issue to eventmesh-site repo. You may submit a PR and link that PR to this issue.


Pipeline fixes

2 issues I ran into when running the pipeline:

The gcs_path always ends with the folder name of the web export. This means that if will end with latest, or whatever you named your export. As far as I understand this means you cannot upload to any valid channel url. A valid channel url would be gs://static-covid/static/v4/main etc. The original code will always generate an url such as gs://static-covid/static/v4/main/something.
single_result docstring is misleading, the parameter should point to a directory containing results.h5, and not the actual h5 file. I changed this target to point directly to the simulations directory. This makes is easier to configure with a local luigi config, since it will not change with every simulation run.

rebased on master


fix: disabled prop didn't make action grey

Please read and mark the following check list before creating a pull request:

[x] I read and followed the CONTRIBUTING.md guide.
[x] I read and followed the New Feature Checklist guide.

Short description of what this resolves:
Im not sure why eva.styles not reacting on disabled prop in topNavigation action. Maybe we need to update eva logic from inside mapping. Let me know what you think
Closes #1277

Basically, making PR to Eva repository is the only thing we need to close #1277. The action component will receive disabled stylings once it is defined.

Hey, @artyorsh. PR was made. Take a look.


[AIRFLOW-1448] Revert "[AIRFLOW-1399] Fix cli reading logfile in memory"

This reverts commit 2de4b7cfb12f5a36eeaf5e78d3ee0fb12d67f3b2 which was breaking CI due to a logical merge conflict.
@bolkedebruin @NielsZeilemaker @edgarRd

Revert lgtm - CI failure seems a perf/timing issue, I'm not sure if they are related or just a fluke on the CI.

What's a logical merge conflict? :-) (I'm really curious)

lgtm

Looks like it's not an official term but that's what I call them, a merge conflict where there are no conflicts on the same line (e.g. one commit changes x = 3 -> x = 4, and another changes x = 3 -> x = 5), but rather two commits that change the logic in different places and when combined cause a breakage (e.g. one commit x = 3 -> x = 4, and another adds a check to make sure that x is still 3 later on).


VASP reference

Provide support for VASP 6 as a reference level of theory.

Hello,
I was wondering if there was an estimated time of arrival for the VASP interface? Had been looking for a sequential learning algorithm that worked with MACE, and I'm trying to build a MLIAP for a phase transforming material, so this looked very well suited to my project. However all of my existing data is with VASP, so can't really restart with CP2K.

We no longer plan to support it, unfortunately. The reason is that psiflow is now being packaged in a Singularity container, along with PyTorch, PLUMED, and CP2K. VASP is licensed, and in addition not well-documented. I have not come across any molecular or material system in which CP2K (which also supports pure plane wave calculations using SIRIUS -- similar to QE) would not do the job.
For your case in particular -- what if you do the active learning with psiflow, using cp2k at more or less the same level of theory, and afterwards re-evaluate the obtained data with VASP? Unless we're talking about highly reactive systems, datasets do not exceed a few thousand states, so the computational cost of re-evaluating the VASP energy and forces is relatively limited ...

I had considered an approach like that; using CP2K to generate the structures through Psiflow, and then re-evaluating them using VASP, however I am a bit worried if CP2K and VASP predict a different set of ground state lattice constants, as the sampling for CP2K could be misaligned from that of VASP.
As for why I'm using VASP, thanks to a prior project I have a good idea of the DFT+U values necessary for my system, as well as KPOINT density, psuedopotential choice, etc, and learning enough CP2K to optimize inputs for my system could take a while.
I might try messing around with some of the code written for the CP2K interface; if I reuse as much as I can from the existing ase-vasp interface I suspect I might be able to get Psiflow working with VASP. If I'm able to make progress on that end I think I'll open up a thread in the discussions section.
Thanks for the quick response.

I had considered an approach like that; using CP2K to generate the structures through Psiflow, and then re-evaluating them using VASP, however I am a bit worried if CP2K and VASP predict a different set of ground state lattice constants, as the sampling for CP2K could be misaligned from that of VASP.

There are many ways to resolve this type of issue. The easiest would just be to apply a metadynamics bias on the individual walkers during active learning. This will ensure that out-of-distribution unit cell configurations (w.r.t the CP2K level of theory) are still sufficiently represented in the training data such that they also cover the distribution as would be obtained from VASP. You can also vary temperature and/or pressure of each of the walkers to achieve something similar.

I might try messing around with some of the code written for the CP2K interface; if I reuse as much as I can from the existing ase-vasp interface I suspect I might be able to get Psiflow working with VASP. If I'm able to make progress on that end I think I'll open up a thread in the discussions section.

You're certainly welcome to do so, but again, I don't think that's really necessary!


NIFI-13632: Updating content access.

Removing existing content viewer application.
Introduced new front end based content viewer applicaiton.
Introduced new bundled content viewers for hex and images.
Deleted previous image content viewer.
Migrated existing standard content viewer (which handles json, xml, yaml, csv, and text) to utilize new content access interface.
Moved standard content viewer into its own NAR.
Moved and renamed custom ui utils and content viewer utils into nifi-common.
Added mime type to FlowFileSummary response payload to help drive the initially opened content viewer.

Reviewing from a front-end perspective...

Thanks for the substantial work @mcgilman! I will review the server-side implementation soon.
I did notice a build failure related to a missing reference in nifi-code-coverage, so that should be removed for successful CI builds.

Testing this out a bit.  Making great progress.
I tried a scenario with GenerateFlowFile to a python script.  The python script writes out flowfile content that is just "1.25.0".  The flowfile has only filename, path, and uuid attributes set.  I then route to updateAttribute that is stopped.  I then list queue and view content for one of these.  It shows up by default in a hex editor.  That looks good.  Then when I select view-text it shows a blank/white screen.  If I go back to hex editor it looks good.  Then back to text and it looks good.
So something funky there but perhaps related to flowfiles with minimal attributes.
Also notable in that same panel if I click around to csv it will sometimes give the white screen and sometimes render.

additional testing after a hard refresh the white screen i described still happens but it is very rare. in a local docker deployment it seems a bit easier to reproduce.  in local mode without docker it is very hard to reproduce.  it happened once in maybe 50 attempts.
So in short - i dont think it is worth bothering with at this point.  might be a local issue/hard refresh certainly seems to have made the issue less likely.
thanks


Word-level-timestamps

Hi Xenova, great work with this repo. Do you know if it's possible to get word-level-timestamps with this? I know it's possible if I'm running whisper in the terminal, but I'm not sure if that functionality extends to this browser/huggingface version, and I don't know how to find out. If you're not sure, feel free to let me know and close the issue.

It's definitely possible; I just haven't gotten around to updating the user interface 😅
Here's example code for it:
let url = 'https://huggingface.co/datasets/Xenova/transformers.js-docs/resolve/main/jfk.wav';
let transcriber = await pipeline('automatic-speech-recognition', 'Xenova/whisper-tiny.en', {
    revision: 'output_attentions',
});
let output = await transcriber(url, { return_timestamps: 'word' });
// {
//   "text": " And so my fellow Americans ask not what your country can do for you ask what you can do for your country.",
//   "chunks": [
//     { "text": " And", "timestamp": [0, 0.78] },
//     { "text": " so", "timestamp": [0.78, 1.06] },
//     { "text": " my", "timestamp": [1.06, 1.46] },
//     ...
//     { "text": " for", "timestamp": [9.72, 9.92] },
//     { "text": " your", "timestamp": [9.92, 10.22] },
//     { "text": " country.", "timestamp": [10.22, 13.5] }
//   ]
// }

But if anyone else wants to update the UI to add this option, then go for it! :)

You're awesome, thank you!


Allow per network value for CoinbaseMaturity

Typically bitcoin has had a static value for coinbase maturity which all networks use.
This PR allows this value to be defined per network.

2022-06-02T11:47:40Z ERROR: ConnectBlock: Consensus::CheckTxInputs: ef96c241c412c2920940854bc1896919e7538f011e3520a983dd12d88d9b650e, bad-txns-premature-spend-of-coinbase/coinstake, tried to spend coinbase at depth 33
2022-06-02T11:47:40Z InvalidChainFound: invalid block=0ee77bcadc00632648e9a9bbfb2afcdc6070a9c069a76b1e858b97d8d9052538  height=44890  log2_work=51.985516  date=2014-03-06T11:29:38Z
2022-06-02T11:47:40Z InvalidChainFound:  current best=9a72fb9e9f4279bfc31ba695db217f192ec0235bec88ddf998ad2c57b50b00d7  height=44889  log2_work=51.985509  date=2014-03-06T11:28:26Z
2022-06-02T11:47:40Z ERROR: ConnectTip: ConnectBlock 0ee77bcadc00632648e9a9bbfb2afcdc6070a9c069a76b1e858b97d8d9052538 failed, bad-txns-premature-spend-of-coinbase/coinstake, tried to spend coinbase at depth 33
2022-06-02T11:47:40Z InvalidChainFound: invalid block=0ee77bcadc00632648e9a9bbfb2afcdc6070a9c069a76b1e858b97d8d9052538  height=44890  log2_work=51.985516  date=2014-03-06T11:29:38Z
2022-06-02T11:47:40Z InvalidChainFound:  current best=9a72fb9e9f4279bfc31ba695db217f192ec0235bec88ddf998ad2c57b50b00d7  height=44889  log2_work=51.985509  date=2014-03-06T11:28:26Z
2022-06-02T11:47:40Z ERROR: AcceptBlockHeader: block f9de2d8f1dc957bf9821c98f3d9cc0e5851f65cac32659b709897601a762901a is marked invalid

Is due to a premature spend. Merging now.


Broken tenant token generation

Description
Tenant token generation is broken on SDK v0.24.1 (tested with Meili v1.2.0).
Expected behavior
Tenant tokens can be generated and used.
Current behavior
Meilisearch returns InvalidApiKey error when searching with generated tenant tokens.
Screenshots or Logs
Described the issue in the discord channel here: https://discord.com/channels/1006923006964154428/1028701743917301892/threads/1123666232068018296.
Environment (please complete the following information):

OS: Ubuntu 22.04
Meilisearch version: v1.2.0
meilisearch-rust version: v0.24.1

Thanks a lot for raising this bug. It should be definitely fixed.

🙏 Good to know


Move aboutus.ejs to its own subdirectory like the other pages.

A small change should be made to keep the consistency of our directory structure... may also want to see if any other views added between making this issue and resolving it are doing the same thing.

This was intentional because I don't think it warrants needing its own directory. I envisioned the about us page to host a body of text and not really having a need for any special CSS or scripting since it's gonna be pretty static.
Let me know if you think otherwise though

I agree that it probably won't need CSS or JS, but I just think it can get confusing if we start having multiple EJS files per folder... each folder I think should logically represent one page of the app and everything required for it. Makes it easier for adding content later in case we decide that we do need to add custom CSS or JS.

Gotcha, then I can tackle this along with another issue I have in mind


any feature available in adaptive card where while user enters the text in input field the other element text should change

I have requirement where I need to find the text length as user enters the text and show how many remaining characters he has maximum to enter in input field.I dont see any out of the box function on adaptive card is there.any suggestions on this???

This is an adaptive card question, not specific to Composer.
As this is not a bug, I would suggest asking a question (or perhaps searching for existing question) on StackOverflow with the appropriate adaptive cards tag/label. If you find that this is a bug, you can open an issue in the Adaptive Cards repository.


added targetarch to dockerfile

Signed-off-by: Aditya Chatterjee<EMAIL_ADDRESS>Description
This PR fixes #
Notes for Reviewers
Signed commits

[ ] Yes, I signed my commits.

Codecov Report

Merging #5289 (dc9c581) into master (14c4ee4) will decrease coverage by 4.30%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master    #5289      +/-   ##
==========================================
- Coverage   17.39%   13.09%   -4.31%
==========================================
  Files         107      107
  Lines        9824     9824
==========================================
- Hits         1709     1286     -423
- Misses       7694     8262     +568
+ Partials      421      276     -145

Flag
Coverage Δ

gointegrationtests
?

unittests
13.09% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files
Coverage Δ

mesheryctl/internal/cli/root/system/update.go
1.05% <0.00%> (-58.95%)
:arrow_down:

mesheryctl/internal/cli/root/system/status.go
1.14% <0.00%> (-55.18%)
:arrow_down:

mesheryctl/internal/cli/root/system/start.go
1.91% <0.00%> (-54.55%)
:arrow_down:

mesheryctl/internal/cli/root/system/restart.go
1.92% <0.00%> (-48.08%)
:arrow_down:

mesheryctl/internal/cli/root/system/stop.go
2.15% <0.00%> (-44.61%)
:arrow_down:

mesheryctl/internal/cli/root/system/logs.go
3.26% <0.00%> (-36.96%)
:arrow_down:

mesheryctl/internal/cli/root/system/check.go
1.03% <0.00%> (-23.03%)
:arrow_down:

mesheryctl/internal/cli/root/system/dashboard.go
1.70% <0.00%> (-9.41%)
:arrow_down:

mesheryctl/pkg/utils/healthcheck.go
8.82% <0.00%> (-4.42%)
:arrow_down:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 5b06c8e...dc9c581. Read the comment docs.


numpy not found during python_api generation

System information

Have I written custom code (as opposed to using a stock example script provided in TensorFlow): No
OS Platform and Distribution (e.g., Linux Ubuntu 16.04): CentOS-7.4
Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy) if the issue happens on mobile device: None
TensorFlow installed from (source or binary): Source
TensorFlow version (use command below): 1.10.1, earlier versions
Python version: 2.7, 3.6 (separate rebuilds same issue)
Bazel version (if compiling from source): 0.17.1
GCC/Compiler version (if compiling from source): 4.8.5, 6.3.1 (separate rebuilds same issue)
CUDA/cuDNN version: CPU only build
GPU model and memory: Quadro K4000
Exact command to reproduce: Not reproducible with single command

Describe the problem
Problem manifests as "No module named numpy", during python api generation as detailed below in log section.
If placing a simple print(sys.path) or print(os.environ['PYTHONPATH']) in the listed __init__.py (in below log), none of the resulting paths listed include the path that numpy is included on via the PYTHONPATH, which is being passed along to the build environ included via bazels CLI opts, --action_env and --test_env arguments to the build call.
I believe this issue will be reproducible only under a state where someone has installed numpy to a nonstandard location, and is including it via PYTHONPATH and --action_env such as is listed. However, all python modules in our environment are in separate paths, as managed by an environment manager, so I cannot install them to a common system path for testing if the issue resolves itself. What is strange is that earlier steps that require numpy have built just fine and not errored from what I can tell.
Apologies if my details included are omitting some crucial detail. The specific set of config values seems unnecessary for detail here, but let me know if they are needed. (Every optional component driven by TF_NEED_{var} is turned off except for TF_NEED_JEMALLOC which does not appear to be part of the problem)
Source code / logs
build command, while prefaced by many environment variable changes and the ./configure step, is of the form:
bazel --batch build --action_env=PYTHONPATH --action_env=LD_LIBRARY_PATH --test_env=PYTHONPATH --test_env=LD_LIBRARY_PATH -c opt --copt=-mfpmath=both --copt=-msse4.2 //tensorflow/tools/pip_package:build_pip_package
ERROR: ${build_root}/tensorflow/BUILD:581:1: Executing genrule //tensorflow:tensorflow_python_api_gen failed (Exit 1) Traceback (most recent call last):   File "/home/${user}/.cache/bazel/_bazel_${user}/ff38b9a62494579437724e08cde1b695/execroot/org_tensorflow/bazel-out/host/bin/tensorflow/create_tensorflow.python_api.runfiles/org_tensorflow/tensorflow/python/tools/api/generator/create_python_api.py", line 27, in <module>     from tensorflow.python.tools.api.generator import doc_srcs   File "/home/${user}/.cache/bazel/_bazel_${user}/ff38b9a62494579437724e08cde1b695/execroot/org_tensorflow/bazel-out/host/bin/tensorflow/create_tensorflow.python_api.runfiles/org_tensorflow/tensorflow/python/__init__.py", line 49, in <module>     import numpy as np ImportError: No module named numpy

@maxnbk TensorFlow has a dependency on NumPy module. Please install NumPy and build again.

numpy is installed. Please read the details more thoroughly. It is simply not in a typical system-installed path, but is available on the PYTHONPATH.

Not sure how bazel resolves/uses PYTHONPATH I am guessing that may be a part of the problem.
Did you use pip install command to install numpy?

Wanted to report that I've run into the same issue when upgrading from tensorflow 1.6 to 1.10.1 [building from source, we also have a not-very-typical installation of python modules (including numpy). I haven't found a way to force my PYTHONPATH to be used (with --action_env or otherwise)

pip install was used to install numpy, but you must be aware that people can relocate pip packages with --prefix or --target commands (depending on the install procedure), and as such, PYTHONPATH should be used to find a python module, not relying that all systems or users install it in the same location on every type of host or user environment.
bazel declares in it's docs that --action_env is how one can ensure an environment variable makes it from the shell-executed environment to the build environment, and --test_env is similar but obviously for the test running environments. However it appears it is also required for a bazel build to declare something else I am unfamiliar with in order to propagate the environment to the task which runs the build components. That is the portion that I am declaring there is an issue with.
Glad to hear I am not alone with David Lange, as our last built tensorflow was 1.4.1 . I could attempt to nail down the exact version it stopped working in.

CC @annarev

I was able to produce a working 1.9.0 build, but unable to produce a working 1.10.0 build with the same exact build settings, with completely accurate details as above, e.g. did not do GPU build etc (nothing else was changed other than the release version that I selected.)

I'm running into a very similar problem as reported here, where import keras_applications fails during the build of TensorFlow 1.11.0 (cfr. #21518), even though Keras-Applications is installed in a location listed in $PYTHONPATH (installed with pip install --prefix, so not in a standard location).
Testing with python -c "import keras_applications" confirms that my environment (outside of the Bazel sandbox) is correctly set up.
Like @maxnbk I have tried instructing Bazel (v0.16.0 in my case) to pass down $PYTHONPATH using --action_env=PYTHONPATH, but that doesn't fix the issue...
Any updates on this?

I ran a build with the latest 1.12.0-rc0 and experienced the same issue, in case testing it against the most recent release helps locate the problem.

Can you try removing PYTHON_LIB_PATH setting from .tf_configure.bazelrc file in the root tensorflow directory? (this file is generated when running ./configure)
Why I think that might work:
We look for pip packages in location specified when running ./configure. This location is stored as value of PYTHON_LIB_PATH in .tf_configure.bazelrc file. If PYTHON_LIB_PATH is not set, then we seem to fallback to PYTHONPATH
(https://github.com/tensorflow/tensorflow/blob/master/third_party/py/python_configure.bzl#L177).

Actually I don't think my suggestion above would work. Seems like even if we pick up PYTHONPATH, we only take 1st path in PYTHONPATH. As I understand, this bug is for configurations that have Python modules installed in multiple locations.
@meteorcloudy do you know the right way to pass PYTHONPATH to bazel?

@annarev I think you can use --action_env=PYTHONPATH

Thank you @meteorcloudy!
hm passing --action_env PYTHONPATH= works for me. I tried the following:

Setup virtualenv "venv" and install pip dependencies except keras_applications
Setup virtualenv "venv1" and install only keras_applications
Then I activate "venv" and run:

bazel test tensorflow/tools/api/tests:api_compatibility_test --nocache_test_results

It fails because keras_applications is not found, as expected.

Then, I run the same but pass PYTHONPATH that points to venv1 site-packages:

bazel test --action_env PYTHONPATH="<path to venv1>/venv1/lib/python2.7/site-packages" tensorflow/tools/api/tests:api_compatibility_test --nocache_test_results

This command passes.
In case relevant, this is my setup:
virtualenv venv1
source venv1/bin/activate
pip install keras_applications==1.0.4
deactivate

virtualenv venv
source venv/bin/activate
pip install --upgrade setuptools==39.1.0
pip install keras_preprocessing==1.0.2
pip install --upgrade six mock portpicker scipy sklearn grpcio
pip install numpy==1.14.5
pip uninstall keras_applications

git clone https://github.com/tensorflow/tensorflow.git
cd tensorflow/
export TF_NEED_CUDA=0
yes "" | python configure.py

That's still just one path inside pythonpath, though? Creating a virtualenv will not be a solve if someone has, for example, some python packages installed at system root, and some at user root, and must build like that.
If you made two venvs and passed a PYTHONPATH var which contains two paths via --action_env would it work?

@maxnbk I believe so, PYTHONPATH is like PATH, it can contain a list of path instead of just one.
See more from https://docs.python.org/3/using/cmdline.html#envvar-PYTHONPATH

@meteorcloudy You slightly missed the point here, .. Only one specific path as set, out of many, on the PYTHONPATH is being consumed by the numpy build, or something to that effect, functionally.

I see, @annarev do you know why only one specific path is consumed by the numpy build?

My instructions above have a flaw. Really sorry about that! I actually tested by first running a successful build with keras-applications installed, then uninstalled keras-applications and and reran the test. It probably didn't rebuild api-generation genrule for the second run. That's why I missed the error.
I experimented a bit more. Looks like PYTHONPATH gets picked up at runtime but not at genrule execution time. I then tried passing --distinct_host_configuration=false to bazel and PYTHONPATH got picked up during genrule execution as well.
So, you can try --distinct_host_configuration=false as a workaround if you are building for the same platform where you are running the build.
@meteorcloudy does --action_env apply to host and/or target configuration?

That workaround (using --distinct_host_configuration=false as part of the bazel build flags) is working for me in non-GPU build tests.
Of course, I don't like workarounds, but it will get me through until a conclusive solve is worked out.
Thanks @annarev . I am happy to run any other tests needed / provide data to help debug further.

Ah, weird. Perhaps premature. The resulting wheel doesn't want to install on my platform, despite attempting to install for the same platform as the build was run for.

@annarev --action_env should apply to both host and target configuration

Apologies for waffling back and forth, I've gotten the --distinct_host_configuration=false wheel to install, I had a name wrong due to testing on the latest RC and the error is a little bit misleading.
That functions correctly for me. Is there a downside to this if my current platform will always be my target platform?

Also able to produce valid GPU build with above details still relevant.

Its great that it works. Thank you for trying it out @maxnbk. --distinct_host_configuration=false flag is fine so long as your target is same as current platform.
Thank you @meteorcloudy for investigating and opening a bazel bug!
It would be nice if it worked without that flag. I want to look at skipping this genrule during build (instead submitting some data that it generates). So, I will keep this issue open.

Nagging Assignee @annarev: It has been 14 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

@annarev, for a similar defect https://github.com/tensorflow/tensorflow/issues/23695, I created this patch: https://github.com/wdirons/tensorflow/commit/01c28a94e63edc8aa301ee377ca026107762858b
Please review and let me know if you think it should be submitted as a PR?

Nagging Assignee @annarev: It has been 14 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

Hi @maxnbk!
It seems you are using older versions(1.x versions) of Tensorflow which is not supported any more. Could you confirm whether issue is replicating after migrating to 2.6 or not ? Thanks!

My issue was worked around with the --distinct_host_configuration=false flag and my production environment has switched to using wheels after v2.0 I believe, so I don't believe I need this solved personally. I'd still recommend that someone independently verify the issue, as single-pythonpath-entry-consumption could be an issue in any number of other environments, but not mine anymore. Do what you will with the issue according to whichever principles this org operates by. Cheers.

Ok @maxnbk ! Thanks for confirmation. Closing this issue as it seems to be resolved from above comment.

Are you satisfied with the resolution of your issue?
Yes
No


provide API to get vulnerability details

Coverage decreased (-0.4%) to 54.705% when pulling a1a08ebed0407a2188e73575772efe3bd68e0b81 on reasonerjt:clair-integration into 269fa0a70a6ce4bc41ced37ac005ac3cc4cd0fc7 on vmware:master.


PHP Fatal error

I have installed diffbot using composer and it created the autoload files as normal.
My code is simply:
require_once "../../vendor/autoload.php";
$diffbot = new Diffbot('KEY');
BUT I get the error:
PHP Fatal error:  Class 'Diffbot' not found
I am running PHP Version 5.6.12

Use the fully qualified class name:
$diffbot = new Swader\Diffbot\Diffbot('KEY');

or import it at the top and then use the short form:
use Swader\Diffbot\Diffbot;

$diffbot = new Diffbot('KEY');

Thank you!


Replace the existing datatime microformat with the HTML5 time element

The datetime microformat has been superceded by the HTML5 time element. They offer better automated parsing (time elements can be easily extracted), better semantics (no misuse of the abbr element) and better accessibility (users with title reading turned on in their screenreader hear effectively a random string of numbers and letters with datetime microformats).

Oooh, awesome. This has been on my todo list for a while. :heart:
Changes look good to me :+1:. @edds might want to have a look too.

I’ve also snuck in one more fix for an issue I noticed in Firefox while testing this. I’ve applied the same fix to static in a separate PR.

I can't comment on the changes but the intention and description are great, awesome work.

:+1:


Add ws to deploy blacklists

ws doesn't compile for our openwrt build, but isn't necessary for socket.io to function correctly. Before https://github.com/tessel/t2-cli/commit/32f95fe163986ba482a96aae04d4eb1393ea564a, the missing .node was just ignored and socket.io worked just fine when deployed to a tessel. After https://github.com/tessel/t2-cli/commit/32f95fe163986ba482a96aae04d4eb1393ea564a, there are warnings about it being missing (which is the correct thing to do), but in this case, we know that it's not necessary.

@rwaldron are you getting the same TypeError: Cannot read property 'targets' of undefined error when you try to compile or something different?

No, I'm seeing
WARN Pre-compiled module is missing: ws.
This might be caused by any of the following:

1. The binary is platform specific and cannot be compiled for OpenWRT.
2. A pre-compiled binary has not yet been generated for this module.
3. The binary didn't compile correctly for the platform that you're developing on.
    It's possible that the binary is Linux-only or even OpenWRT specific,
    try npm installing with "--force" and rerun your deployment command.

Please file an issue at https://github.com/tessel/t2-cli/issues/new

Can you file a separate issue for that error—thanks!

@rwaldron to be clear, I got that error from t2-compiler - I wasn't using the CLI yet. I'll open an issue on t2-compiler.

No, it was:
node-pre-gyp info it worked if it ends with ok
node-pre-gyp info using<EMAIL_ADDRESS>node-pre-gyp info using<EMAIL_ADDRESS>| linux | x64
node-pre-gyp ERR! UNCAUGHT EXCEPTION
node-pre-gyp ERR! stack Error: ws package.json is not node-pre-gyp ready:
node-pre-gyp ERR! stack package.json must declare these properties:
node-pre-gyp ERR! stack main

Here is the issue I filed: https://github.com/tessel/t2-compiler/issues/17.

It looks like the missing main poperty has been resolved: https://github.com/websockets/ws/commit/523b61f7d7aa92fdc15ac406c5c6b03b71b4259a
Also, I believe the issues here were caused by my using a much older version of ws

Okay great - so once they make a new release, we can throw it up on packages.tessel.io and this shouldn't be an issue anymore.

I've just tried my example socket chat app with latest socket.io (which installs latest ws)
https://www.youtube.com/watch?v=q06xu8_E2IA

That has got to be the cutest socket.io example I've ever seen.

hahaha


Linking to some external doc fails

The following seem to fail (as if the symbols did not exist):

scala.Null
scala.Nothing
scala.util.Left
scala.util.Right
scala.Some (scala.None works, note that it's not a type like all the others)
java.lang.Runtime.exit
java.lang.String.copyValueOf (not needed, just for testing)

Currently they are not represented as links, but just as code fragments.

Should we close this as won't fix? I don't see this being ever addressed. If Scaladoc is not able to resolve those links, there's nothing we can do about it.


fix(datastore): query limit more than 300

Fixes: https://github.com/googleapis/google-cloud-php/issues/5039
Changes
Datastore loops through all pages via EntityPageIterator which uses PageIteratorTrait.
The trait expects resultLimit to be set while doing the subsequent calls, which was missing so any limit value of more than 300 was leading to all results getting fetched. Following bugs were fixed:

Under Grpc mode, when a Gql query string goes through runQuery, a parsed query object is returned with correct limit array. It doesn't need to be modified in subsequent fetches. Added a check to skip that.
We need to pass resultLimit in Operation->runQuery(...) via the EntityPageIterator instance, so that the iterator can loop through all pages.
Under EntityPageIterator, when we fetch the current page only then we need to save the limit value because in case of GqlQuery we do not have the limit parsed in client library.
System Tests to cover for these scenarios.

Testing

System tests run:

google-cloud-php/Datastore % XDEBUG_MODE=debug vendor/bin/phpunit -c phpunit-system.xml.dist --filter="QueryPaginationWithLimit"
PHPUnit 8.5.31 by Sebastian Bergmann and contributors.

....                                                                4 / 4 (100%)

Time: 7.63 seconds, Memory: 12.00 MB

OK (4 tests, 6 assertions)
google-cloud-php/Datastore %

Notes

PR in Ruby: https://github.com/googleapis/google-cloud-ruby/pull/19456

While researching this I came across a similar issue: https://github.com/googleapis/google-cloud-python/issues/1763
It looks like the solution they landed on is to update the limit on subsequent calls in a paged set and utilize offsets. This approach could provide some benefit over using resultLimit, as it looks like it should actually limit how much data is sent over the wire. With resultLimit we could have a response come back with 300 items in it, but only actually send 150 to the user as it is only a client side limiter (more intended to prevent further paging).
WDYT about using that approach here, too? I did some quick testing and it seems like we should be able to support it.

@dwsupplee our doc suggests that we should rather move away from offsets to save costs, so we should continue with cursors (current approach is similar to Ruby, etc).

To prevent over-fetching of records in last page, I switched to managing limits via a new remainingLimit variable passed by reference across callable of determineNextResultToken and runQuery.
I am keeping resultLimit for safety. Are you in favor of removing it?

@vishwarajanand Just a heads up this is on my radar! I'll be reviewing shortly.

@dwsupplee  Sure, thanks for the headsup. I have merged the latest main to pull in all changes from multiple-db feature.

@vishwarajanand and I discussed the PR offline a bit, looking good for the most part just a few minor updates to be expected.


Placeholder FOUC - Flash Of Unstyled Content

Hi all, using ui-select I noticed that sometimes when I fast-reload a page with ui-select in it, may happen that the string "{{$select.placeholder}}" appears for a couple of seconds in place of the final placeholder.
There is some way to solve this? (maybe with ng-bind?) ...I've tried with ngGloak but it doesn't change! :-/
I know I should reproduce it but it's very diffcult for me... because I think it's something related with the page weight and with a simple demo it wouldn't happens.

have you added the ng-cloak CSS to your desired CSS file?  Ng-cloak should surely do the trick.
You can also set the object via ng-hide and set flag in controller to enable

Oh god... the problem exists only when I leave the Firebug tools (on Firefox) open...
I'm so sorry, I didn't noticed it before... too much work for today.
I close the issue.


List containers should be displayed in sorted order

Thanks for contribution! Please go through following checklist before sending PR.
Links to issue https://github.com/Azure/Azurite/issues/275
PR Branch Destination

For Azurite V3, please send PR to dev branch.
For legacy Azurite V2, please send PR to legacy-dev branch.

Always Add Test Cases
Make sure test cases are added to cover the code change.
Development Guideline
Please go to CONTRIBUTION.md for steps about setting up development environment and recommended Visual Studio Code extensions.

 Thank you for your submission, we really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.:x: sangeetha5491 sign nowYou have signed the CLA already but the status is still pending? Let us recheck it.

build

How do I find out why the 4 tests were canceled?

@XiaoningLiu I can't see your comments.

@sangeetha5491

Please add the change log for the change in: https://github.com/Azure/Azurite/blob/master/ChangeLog.md, you can add it under "Upcoming Release" tag
Consider the change for marker  in adding test case.

@XiaoningLiu and @blueww I have addressed the review comments. Please review it again.


Distribution resize?

Hi, I've developed a dashboard for our team with grommet, and I was wondering if there was a way to resize the Distribution, as the text doesn't seem to conform or affect the size of the distribution. I've already set the large flag to true as well. The problem can be seen at the bottom of this page: http://sdnreporter.rose.hp.com/flaredashboard.html.
Thanks so much!

I think there are some things you can change to your page to get the Distribution to take the full width. It looks like the Distribution is inside a Tile, which is constraining its width. Can you try it without putting it in a Tile?


Chairbender Chat

About the PR
Revives https://github.com/space-wizards/space-station-14/pull/3125
Currently a draft because having the lobby enabled causes a client crash. Why? Because adding this arg public ChatBox(bool lobbyMode) to the constructor breaks the lobby xaml, causing an exception later on. And I don't know how to fix it.
There's some other changes I'd like to make but that's the main merge blocker AFAIK.
Screenshots

Soon:tm:
Changelog

WIP
:cl: chairbender and ike709

add: Added fun!
remove: Removed fun!

@PaulRitter please help with the xaml issue

Update your changelog.

Update your changelog.

Update your changelog.

Hi Ike STOP. in response to this message on Discord STOP. I say that this all looks pretty STOP. good to me and you STOP. should ship it. STOP.

Swept

Ready for review.

@metalgearsloth Addressed as much of your review as I know how to.

The viewport improvements broke this PR and I have no desire to fix it.

Nevermind I fixed it.