Delete training settings

Previous iterations of the tool included additional training customizations (selecting between algorithms, setting the K value, etc) that were shown on a pre-training screen. In the latest version of the tool, that screen only shows dropdowns to confirm data type for selected columns, which is redundant to functionality in the column inspector side panels. Thus I am removing the training screen, related components and residual code. Additionally, we no long use the option to select test data from the beginning of the dataset, so deleting that code here too.

@dancodedotorg with this change the  hideSpecifyColumns and hideChooseReserve parameters will no longer be needed.  If the user has selected an in-house dataset, we'll use the column datatypes from the JSON metadata; if it's a user-uploaded dataset, we default to categorical and show a selection dropdown in the column inspector side panel. Test data reservation percentage is now always set to 10%. Thanks!


Dialpad is not closing nor adding the outbound task on flex

When making an outbound call, I can dial the number and the phone can be answered on the other end but the dial pad stays on the screen without closing. Hold music is heard on the other end.

Hi @shageevan
It sounds like something is misconfigured. Is this happening on every call?

Hi @jhunter-twilio, I configured everything as mentioned. It is happening on all outbound calls. I will share a video later today.

It definitely sounds misconfigured
I'd recommend taking a clean checkout, ensuring all the SIDs are setup in the environment file correctly, in particular the sync service sid (it sounds like that specifically is missing), deploying the functions and deploying the asset as described in the instructions.
If it's still happening can you watch the console logs and the callStatusUpdateHandler function and look for errors

Sure. I will check on this and revert later today.

Done. Works now.


Productionize private key provider

Right now, the log private keys are provided in a private .pem file. Let's migrate this to a more robust key providing system. For GCP, let's use KMS.
We spoke about using Tink, but it's unclear what benefit it would bring in the immediate future.

As indicated in #36, GCP KMS signing costs too much for the CT ecosystem. We will explore the in-memory signing implementation.


Searching across Drummer (or any other "tools for thought)

@scripting wrote a blog post earlier this week about where to find different things he's written about via Drummer, although the question may be larger than that: How can we search across various "tools for thought" to find related thoughts, thought patterns, specific topics, etc...
I took a high-level stab at one possible way and outlined it in a repo. I also tested a very basic POC. Would love to hear feedback on what I've dubbed the "BrainBase" for now. 😉
Essentially, I want to suck in things I've written in outliners, on Twitter, etc.... into a database via standard formats, and create a web-based search tool across all of that data.

@kevinctofel -- first thanks for picking up the ball here. as far as i know you are the first.
okay -- next -- a bit of background. there are products in the "tools for thought" space that have both an editor and a database. those are the databases i want to get content into.
i want to use my editor and benefit from their database.
by doing so i hope to learn what their databases actually do (I've heard a lot of claims that sound kind of unbelievable to me, but let's give them the benefit of the doubt), and also break this assumption that to be in a database you have to use their editor. this is a huge problem in tech kevin. for example to have a substack newsletter i have to use their editor. to post to facebook i have to use their editor. medium has an api, but it's limited in a very important way, so effictvely you have to use their editor. see the pattern? i do not want that to happen here.
so it's not just any database i want to get my ideas into, it's those databases.
now some of the companies are receptive to this idea, esp my friends at LogSeq. but this is the kind of thing it's best if the users do. independence matters here.
i'm sure you get it, but to really nail it down, the other day we were having a discussion here about connecting Drummer with GitHub. Lots of power there, imho. But it wouldn't be the same if it were connected to a new different kind of repository manager. Getting into GitHub itself is the goal.
Now one other thing -- you can have OPML in JSON. We have a toolkit that reads OPML files and returns JavaScript structures. From there, do whatever you want. Save it as JSON. It's up to you. I wouldn't have left this stone un-turned-over, so before assuming something like that doesn't exist, ask.
Maybe you don't program in JavaScript, okay, let's convert that toolkit to the language you work in.

Ah, I see the distinction. I was approaching it from a different angle mainly because I've evaluated several of the tools for thought on the market and most of them were closed source when it comes to their database (if not everything). I know some are open-source but most are not.
So my mindset was how to get copies of my data out of their databases into something open and useful (to me). Appreciate the explanation and for pointing out the Node package for OPML in JSON. I do use JS for most of what I do, so this is super helpful. :)
I may continue down my path for a little while when time between classwork allows. There isn't much of that time to be had, unfortunately. But, it would be a good learning experience, for one. And while I'd hope these tools go in the direction you're thinking, until they do, my digital thoughts are all over the web.

@kevinctofel -- you got it right. i don't care about the source as much as they allow the users' data to flow where it wants to go. my approach is, as long as they leave their world closed off to my data, we'll just have to build everything they built behind their wall. but the logseq guys are the real thing, or so it seems. so as long as we have one partner, that's enough to get the ball rolling.
a similar situation happened with scriptable apps on the mac in the early 90s. we got a few apps to go first, and once we did all their competitors had to follow suit. if pagemaker was open to our scripting, then so would quark.
we had no such problems in rss, because of wired, red herring, motley fool and salon, and then we got the ny times and it was off to the races.
part of me hopes that roam and the other stay locked up, because i'd really like to use them as an example of how that approach can backfire. on the other hand we win if they open up, even bigger. ;-)


Quickstart documentation correction

Installation of packages via command
pip install "autogen-agentchat==0.4.0.dev13" "autogen-ext[openai]==0.4.0.dev13
results in error as under.
While running the code
from autogen_ext.models.openai import OpenAIChatCompletionClient
following error is generated.
ModuleNotFoundError: No module named 'azure'

I got the error while executing example Quickstart code given on the same documentation page.

Will address the dependency that probably should not be rquired in #4870


[SPARK-37120][BUILD] Add Daily GitHub Action jobs for Java11/17

What changes were proposed in this pull request?
This PR aims to add Daily GitHub Action jobs for Java 11/17.
Why are the changes needed?
To add a test coverage on Java 11/17.
Does this PR introduce any user-facing change?
No.
How was this patch tested?
N/A.

cc @HyukjinKwon

Jobs are running properly. Let me get this in and see how it gose - seems like PR builders don't get affected anyway.

Merged to master.

Thank you so much, @HyukjinKwon !

Thanks for ping me @dongjoon-hyun. LGTM +1


HPA improvements

Description
Helm charts need fixes for HPA to work (with less changes) out of the box:

Increase probe timings to avoid scaled up pods never getting to Ready state
Fix current (completely wrong) resource limit/request settings for inferencing components
Reduce default TGI scaling values to avoid issues when TGI values.yaml is not properly configured
Change ChatQnA Helm chart to work with Helm installed version of Prometheus
Related Helm README updates & fixes

Additionally PR also adds:

Support for enabling  HPA-scaling in ChatQnA chart only for relevant subcharts

(Best reviewed by checking individual commits.)
Issues
n/a
Type of change

[x] Bug fix (non-breaking change which fixes an issue)

Dependencies
n/a
Tests
Manual testing for the changes (before splitting them to separate commits & documenting them).

Force pushed pre-commit CI fixes (as there were no review comments yet).

If README.md HPA section is too long, it could be moved to its own HPA.md file.

If README.md HPA section is too long, it could be moved to its own HPA.md file.

Yes, that would be a good idea.

As to the generated manifests: https://github.com/opea-project/GenAIInfra/tree/main/microservices-connector/config/HPA/
I think those should be thrown away rather than updated.
Using HPA with GMC when inferencing is done on CPU does not make much sense.  GMC allows e.g. changing model at run-time, but when scaling the CPU services, it means that deployment resource requests etc should be updated accordingly (based on used model, data type used with it, TGI/TEI versions and underlying node resources), and I don't think GMC has any support for that.
(Using HPA with GMC for Gaudi accelerated services would be a bit different matter, but current HPA rules are just for CPU charts.)

@eero-t  To make HPA support in GMC is the target and @ichbinblau is working on this already.  GMC allows to change the model dynamically but it will also support static pipiline.

Not having used Helm before, I had not realized that gaudi-values.yaml overrides just specified parts from values.yaml file (in top-level and sub-charts), instead of replacing the values.yaml content completely.  I.e. enabling HPA in values.yaml means it being enabled also when gaudi-values.yaml (or nv-values.yaml) is used.
This got me thinking...
If settings needed for scaling (resource requests corresponding to specified model, probe timings etc) were in separate files, user could enable HPA without needing to modify values.yaml files in top-level + sub-charts.
Enabling HPA would then look something like this, for CPU:
helm install chatqna ... -f chatqna/hpa-values.yaml -f common/tgi/cpu-values.yaml -f common/tei/cpu-values.yaml

And for Gaudi:
helm install chatqna ... -f chatqna/hpa-values.yaml -f chatqna/gaudi-values.yaml

Thoughts?

Note: above does not fix the issue for (CPU) scaling, of scaled pods' resource requests needing to match their resource usage with the LLM model specified for them => overriding the model with a global Helm value, or by GMC, is not really a good idea as then there's a mismatch between those.
When Gaudi or GPU is used, that's less of a issue, as long as single device per Pod is OK.  If larger models would require multiple devices, or smaller models would require (for better utilization / cost reasons) sharing GPUs between pods, then changing models would also be bad idea, if resource requests are not changed accordingly.

Not having used Helm before, I had not realized that gaudi-values.yaml overrides just specified parts from values.yaml file (in top-level and sub-charts), instead of replacing the values.yaml content completely. I.e. enabling HPA in values.yaml means it being enabled also when gaudi-values.yaml (or nv-values.yaml) is used.
This got me thinking...
If settings needed for scaling (resource requests corresponding to specified model, probe timings etc) were in separate files, user could enable HPA without needing to modify values.yaml files in top-level + sub-charts.
Enabling HPA would then look something like this, for CPU:
helm install chatqna ... -f chatqna/hpa-values.yaml -f common/tgi/cpu-values.yaml -f common/tei/cpu-values.yaml

And for Gaudi:
helm install chatqna ... -f chatqna/hpa-values.yaml -f chatqna/gaudi-values.yaml

Thoughts?

In helm chart, values.yaml is alread used. Any addtional value files specified by -f <xxx.yaml> is just to add patch to the default values.yaml, not to replace it.
Since in component helm charts(those charts under common directory, HPA is disabled by default, i.e. in their values.yamlfiles HPA is set to false), there is no need to put HPA related settings in e2e helm chart's(i.e. chatqna) values.yaml file. What we need here is a separate hpa.yaml file in e2e helm chart, which enable the corresponding HPA for components and set default optimal configurations for the out-of-box e2e workload.
So helm install chatqna ... -f chatqna/hpa-values.yaml -f chatqna/gaudi-values.yaml seems a more propriate way to go, compared to helm install chatqna ... -f chatqna/hpa-values.yaml -f common/tgi/cpu-values.yaml -f common/tei/cpu-values.yaml. I don't think we should put separate hpa-values.yaml files in component helm chart, because it could be dependent on model and workload. So I would suggest put all the adjustable HPA related settings in hpa-values.yaml in top level e2e helm chart, in which we should also tell users to adjust its settings for their own model case if needed.

Since in component helm charts(those charts under common directory, HPA is disabled by default, i.e. in their values.yamlfiles HPA is set to false), there is no need to put HPA related settings in e2e helm chart's(i.e. chatqna) values.yaml file.

If user does NOT use -f hpa-values.yaml, (disabled) HPA setting is still needed in values.yaml as Helm errors on custom metrics template.
Btw. Theresa just mentioned that Helm refuses to install PrometheusAdapter custom metrics config (for ChatQnA) over configMap that has been installed by another Helm chart, the Prometheus one...  (I hadn't noticed this because I use Helm to generate manifest, and then apply that after reviewing the results, I don't install things directly with Helm. Also my Prometheus install comes from kube-prometheus manifests, not from Helm chart for that.)
=> I'm not sure how to best handle that for 1.0 milestone.  I could test doing all installs directly with Helm and find some way to workaround any issues, but that will take at least few days.  Another option is to document what I've actually tested ie. use Helm just to produce manifests, and installing those...

I would suggest put all the adjustable HPA related settings in hpa-values.yaml in top level e2e helm chart, in which we should also tell users to adjust its settings for their own model case if needed.

Because suitable subcomponent resource requests & probe timings differ between Gaudi and CPU (besides depending on model), and are needed for scaling to work in general (not just for HPA), I think they should be in separate file from generic HPA setup.  I.e. top-level chart would have cpu-values.yaml that contains values that are relevant when subcharts use CPU, regardless of whether HPA is used:

ChatQnA TGI/TEI on CPU: -f cpu-values.yaml

With HPA scaling: -f cpu-values.yaml -f hpa-values.yaml

ChatQnA TGI/TEI on Gaudi: -f gaudi-values.yaml

With HPA scaling: -f gaudi-values.yaml -f hpa-values.yaml

Model should also be set in top-level chart device values file, not values.yaml because model (and data type used with it) dictates the resource usage, how much RAM it needs, does it fit into single device, or need to be sharded over multiple ones etc.

FYI: I filed ticket against Prometheus-operator, so that it would support do config updates also for Prometheus-adapter: https://github.com/prometheus-operator/prometheus-operator/issues/6912
Only that would allow reasonable automation for custom metrics, so I'd appreciate thumb-ups in the ticket. :-)

Interestingly CI automatically started running tests for the new values files:

chatqna, xeon, cpu-values
chatqna, xeon, hpa-values

However, hpa-values requires compatible Prometheus install CRDs to be present.  And on Xeon, while cpu-values works fine alone, hpa-values is supposed to be used only with cpu-values.
There was also unrelated (Gaudi guardrails) test failure due to:
FailedScheduling  4m40s (x2 over 10m)  default-scheduler  0/2 nodes are available: 1 Insufficient habana.ai/gaudi

Since in component helm charts(those charts under common directory, HPA is disabled by default, i.e. in their values.yamlfiles HPA is set to false), there is no need to put HPA related settings in e2e helm chart's(i.e. chatqna) values.yaml file.

If user does NOT use -f hpa-values.yaml, (disabled) HPA setting is still needed in values.yaml as Helm errors on custom metrics template.
Btw. Theresa just mentioned that Helm refuses to install PrometheusAdapter custom metrics config (for ChatQnA) over configMap that has been installed by another Helm chart, the Prometheus one... (I hadn't noticed this because I use Helm to generate manifest, and then apply that after reviewing the results, I don't install things directly with Helm. Also my Prometheus install comes from kube-prometheus manifests, not from Helm chart for that.)
=> I'm not sure how to best handle that for 1.0 milestone. I could test doing all installs directly with Helm and find some way to workaround any issues, but that will take at least few days. Another option is to document what I've actually tested ie. use Helm just to produce manifests, and installing those...

I would suggest put all the adjustable HPA related settings in hpa-values.yaml in top level e2e helm chart, in which we should also tell users to adjust its settings for their own model case if needed.

Because suitable subcomponent resource requests & probe timings differ between Gaudi and CPU (besides depending on model), and are needed for scaling to work in general (not just for HPA), I think they should be in separate file from generic HPA setup. I.e. top-level chart would have cpu-values.yaml that contains values that are relevant when subcharts use CPU, regardless of whether HPA is used:

ChatQnA TGI/TEI on CPU: -f cpu-values.yaml

With HPA scaling: -f cpu-values.yaml -f hpa-values.yaml

ChatQnA TGI/TEI on Gaudi: -f gaudi-values.yaml

With HPA scaling: -f gaudi-values.yaml -f hpa-values.yaml

Model should also be set in top-level chart device values file, not values.yaml because model (and data type used with it) dictates the resource usage, how much RAM it needs, does it fit into single device, or need to be sharded over multiple ones etc.

The top-leve values.yaml is meant for cpu cases without HPA. We want users to be able to install helm chart without applying addtional value files on any standard xeon servers if they want to have a try.
How about we do the followings:
# generic CPU cases without HPA:(this is equal to applying by -f values.yaml)
helm install chatqna chatqna --set global.HFTOKEN ...

# generic CPU cases with HPA:
helm install chatqna chatqna -f hpa-for-cpu-value.yaml --set global.HFTOKEN ...

# gaudi case without HPA
helm install chatqna chatqna -f gaudi-values.yaml --set global.HFTOKEN ...

# gaudi case with HPA
helm install chatqna chatqna -f guadi-values.yaml -f hpa-for-gaudi-values.yaml --set global.HFTOKEN...

As for the PrometheusAdapter custom metrics config, maybe we should have it documented here before https://github.com/prometheus-operator/prometheus-operator/issues/6912 is resolved

Interestingly CI automatically started running tests for the new values files:

chatqna, xeon, cpu-values
chatqna, xeon, hpa-values

However, hpa-values requires Prometheus install first so that required API types are recognized. And on Xeon, while cpu-values works fine alone, hpa-values is supposed to be used only with cpu-values.
There was also unrelated (Gaudi guardrails) test failure due to: FailedScheduling 4m40s (x2 over 10m) default-scheduler 0/2 nodes are available: 1 Insufficient habana.ai/gaudi

Please contact @daisy-ycguo about the CI to skip some value files in CI. We might need to come up a unified way of how to skip some cases, now we have nv-values.yaml to be skipped, and I expect more to come.

How about we do the followings:
# generic CPU cases without HPA:(this is equal to applying by -f values.yaml)
helm install chatqna chatqna --set global.HFTOKEN ...

# generic CPU cases with HPA:
helm install chatqna chatqna -f hpa-for-cpu-value.yaml --set global.HFTOKEN ...

# gaudi case without HPA
helm install chatqna chatqna -f gaudi-values.yaml --set global.HFTOKEN ...

# gaudi case with HPA
helm install chatqna chatqna -f guadi-values.yaml -f hpa-for-gaudi-values.yaml --set global.HFTOKEN...

As for the PrometheusAdapter custom metrics config, maybe we should have it documented here before prometheus-operator/prometheus-operator#6912 is resolved

This looks sane to me.

As CPU values being in separate top-level chart was fine, I dropped the resource & probe timing updates to common/*/values.yaml files, the commit reverting them,  and rebased again to main.  I.e. there were no code or doc changes in the rebase.  Requested changes come in next commit(s).

The top-level values.yaml is meant for cpu cases without HPA.

I thought it was intended for Gaudi because it worked so badly on Xeon...
Without suitable probe timings and resources, out-of-the-box Xeon experience is rather awful, if user happens to run multiple different services, or does kubectl scale --replicas on the backends, regardless of whether HPA is used or not.
However, according to comments in TGI values.yaml file, OPEA does not want to specify resources by default, because  somebody may want to try OPEA in a setup (minikube) that does not really have resources for it (I guess user could then switch to a smaller model, which would mean default resource values mismatching it).
Therefore I put CPU timings + resource values to a separate cpu-values.yaml file that.
Different CPU resource requests are needed for different models and different data types.   I.e. if things should work out of the box, there need to be file per set of models, and data types specified for TGI & TEI, which in future it could look something like this:

cpu-mistral7b-bgebase.yaml
cpu-some-other-models.yaml
etc

Note that because it's not possible to share Gaudi, case for that is simpler.  One needs different resource spec only for models that do not fit into single Gaudi, but need to be sharded over several of them.  If model needs to be split, only then CPU side resource usage can be relevant (due to CPU side DeepSpeed overhead).

We want users to be able to install helm chart without applying additional value files on any standard xeon servers if they want to have a try.

Extra file is needed for Gaudi and Nvidia, so it would be consistent to have one also for CPU.

# gaudi case with HPA
helm install chatqna chatqna -f guadi-values.yaml -f hpa-for-gaudi-values.yaml --set global.HFTOKEN...

Whether one uses Gaudis or CPUs is not a relevant distinction for HPA enabling and replica limits. If cluster has enough of Gaudis or CPUs, the same values can be used.

Issues raised by Theresa when testing this PR, which must be fixed before merging:

Helm-installed Prometheus does not find ServiceMonitors

=> Add Chart values for Prometheus Helm install release name & namespace

Custom metrics config overwriting fails with Helm-installed Prometheus-adapter

Did not find a way to handle that reasonable with OPEA Helm charts, and Prometheus upstream does not seem keen on doing anything with the adapter (except replace it)
=> decided it's best if Helm creates configMap which name does NOT match with Prometheus one, and README has instructions on how to copy that over Prometheus-adapter configMap (before user restarts adapter).  That way it works same regardless of how Prometheus is installed

Other feedback from internal reviews:

Already merged HPA PR does not conform to Helm best practices (#327)

https://helm.sh/docs/chart_best_practices/templates/
=> rename custom metrics and HPA rule template files (+ check for empty values in them?)

Gaudi fine-tuning for HPA

=> Pre-existing resource requests in gaudi-values.yaml and probe timings in values.yaml files are probably fine for Gaudi, but would be good to verify

Support multiple separate HPA-scaled ChatQnA instances in same cluster

=> prefix custom metric names with ChatQnA instance name

Note: support for ChatQnA instances being in separate namespaces will definitely come only in some future PR.  Fiddling with Prometheus RBAC rules is clearly out of scope for this one.

Thank you for your PR @eero-t Will you rebase this pr and run test again? I've already disabled hpa-values test since it needs additional conditions. I've created an issue to track it: #413.  We can enable it later.

@eero-t  please update manifest as well under https://github.com/opea-project/GenAIInfra/tree/main/microservices-connector/config/HPA

Please update manifests under https://github.com/opea-project/GenAIInfra/tree/main/microservices-connector/config/HPA as well

@irisdingbj I think those should be thrown away rather than updated.  There are too many interdependent config options in charts for that to make sense, especially when GMC allows changing some of them without fixing the others to match.
Theresa is working on GMC support for HPA, so it could come after 1.0, but not as part of this PR.

Please update manifests under https://github.com/opea-project/GenAIInfra/tree/main/microservices-connector/config/HPA as well

@irisdingbj I think those should be thrown away rather than updated. There are too many interdependent config options in charts for that to make sense, especially when GMC allows changing some of them without fixing the others to match.
Theresa is working on GMC support for HPA, so it could come after 1.0, but not as part of this PR.

Thank you @eero-t    @ichbinblau   would you confirm we need HPA manifest or not fot GMC to support HPA?

I believe the 10 mins wait time for deployment ready(ROLLOUT_TIMEOUT_SECONDS) in helm CI test is too short for cpu-values.yaml configuration

Helm-installed Prometheus does not find ServiceMonitors
Custom metrics config overwriting fails with Helm-installed Prometheus-adapter
Already merged HPA PR does not conform to Helm best practices
Support multiple separate HPA-scaled ChatQnA instances in same cluster

Pushed fixes for these issues, meaning:

release label for monitors
Custom metric install has now multiple steps, not just adapter restart
Renamed template files to best practices
Custom metrics have not chart specific prefix

Theresa is working on GMC support for HPA, so it could come after 1.0, but not as part of this PR.

Thank you @eero-t @ichbinblau would you confirm we need HPA manifest or not fot GMC to support HPA?

Discussed with Theresa, and I will be doing minimal manifest update (later today).  I.e. fix non-HPA TGI/TEI deployments, and reduce HPA/ dir content to fixed HPA + custom metric rules and serviceMonitors.   Deployment resource limits will come later for GMC.
Handling manifest generation properly for deployments required changing replica and terminationTimeout settings to depend on something else than HPA.  Termination timeout really depends on whether pod is accelerated, not whether HPA is used, so there's no new accelDevice value for that (empty meaning CPU i.e. non-accelerated / slow).

Merged pre-commit.ci changes to earlier commits, applied several fixes to the documentation and updated GMC manifests.
GMC manifest update was done manually, because it's not really supported by Helm chart generated content.  Somebody would need to change all deployment templates to apply proper timings based on whether CPU or accelerator (e.g. Gaudi) is used, similarly to how I did with termination timeouts.
However, that's not really HPA problem, but a generic one in the charts, so it belongs to some other PR, not into v1.0 milestone one.
@irisdingbj is the PR now OK from your side?


Support rehashing in the HashIndexBuilder

Still needs to be integrated with the CSV reader.
The performance has taken a bit of a hit, HashIndexBuilder::append is roughly 60% slower than before, mostly I think due to some unnecessary copying when iterating over slots (it was worse before I fixed InMemDiskArrayBuilder to directly access the header in getNumElements instead of going through the unnecessary locking inherited from BaseDiskArray). I'm going to try and restructure things to remove the copies and speed things up.

Performance is fixed, with a slight improvement from before.
Testing on a csv containing 1 to 60 million in a random order, copying this from csv as a primary key takes on average on my 12-core machine (over six runs):

3053ms when read as an int64 column
21023ms when read as a string column

Compared to master:

3075ms when read as an int64 column
26002ms when read as a string column

The change on the int column is mostly within the range of noise. On the other hand, when I'd benchmarked HashIndexBuilder::append by itself, 120 million inserts (with bulk reserve) was taking ~15s, compared to ~20s on master, so other parts of the copy pipeline may be dominating the runtime (strings took ~37s vs 40s; the main difference in parallel is likely due to avoiding re-acquiring the mutex for the shared overflow file).
I think string performance could be further improved by using an rwlock instead of a mutex, since access to the overflow file is usually a read-only check to see if a key already exists in the index and could be done concurrently.

I did a quick test, and with an std::deque that only locks when appending, the end-to-end string benchmark I'd done in the previous comment improved to 12s, and an std::list with an append lock took only 9.8s. Presumably lists are sufficiently simpler to perform better, and we already have one level of indirection.
That should be better than an rwlock, as concurrent reads can be safely done even when writing and appending new pages (since we never delete anything and the only thing changing is the pointer at the end). I'll have to try some larger tests though to see how well it scales, since finding a particular index would be slower, so the fastest in this test may end up being slower with datasets which are much larger. Deques may scale better sometimes, but the MSVC std::deque implementation uses a block size so small that it's effectively an std::list, so I'm not keen on having to do something custom to get similar performance on all platforms, but it would at least be helpful to know if there is much of a performance gap with larger datasets.


Update NRFX to release 2.8.0

This fix regression with not working PWM on nRF5x.

Style check summary
No suggestions at this time!


Starter build missing from main

Hello dear genomekit team. Thanks for releasing the project it's been tremendously useful - I've been using it to help folks start using a model we made https://github.com/bowang-lab/Orthrus?tab=readme-ov-file#create-and-set-up-the-environment
It looks like the starter build link which is specified in the docs  now points to an empty page:
https://raw.githubusercontent.com/deepgenomics/GenomeKit/main/starter/build.sh
However, https://raw.githubusercontent.com/deepgenomics/GenomeKit/v6.0.3/starter/build.sh is still available.
Is it possible to bring that page back online?
Thanks

It was removed in #14 (which didn't create a release, and that's why the docs haven't been published).
The scripts are replaced with a docker image and the relevant docs have been updated on the docs site. I can force a release to publish the docs.
Can you have a look at the new docs https://github.com/deepgenomics/GenomeKit/blob/main/docs-src/quickstart.rst#generating-data-with-docker ? Let me know what you think.

Hey Phil, Avishai was able to get AWS to serve the OSS data, so you can use https://github.com/deepgenomics/GenomeKit/releases/tag/v6.2.0 and just have it download like back at DG.

Sorry I haven't gotten around to replying earlier - this is awesome, thanks a lot. It will simplify instructions to new users a lot


File not found

Hello!
From Electron app i get error "index.html:12 Command failed:<PHONE_NUMBER>" Error "500"
In console i see:
File not found
    at ...node_modules/nuts-serve/lib/nuts.js:278:27

If i check by browser link:
http://example.com:1337/update/win32/0.0.0
i get json with url http://example.com:1337/download/version/0.0.1/windows_32?filetype=zip" and can download file. Not worked only in electron app

Not sure if this is related, but I get
Command failed:<PHONE_NUMBER>
System.AggregateException: One or more errors occurred. ---> System.Net.WebException: The remote server returned an error: (404) Not Found.
   at System.Net.HttpWebRequest.EndGetResponse(IAsyncResult asyncResult)
   at System.Net.WebClient.GetWebResponse(WebRequest request, IAsyncResult result)
   at System.Net.WebClient.DownloadBitsResponseCallback(IAsyncResult result)
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Squirrel.Utility.<LogIfThrows>d__34.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Squirrel.FileDownloader.<DownloadFile>d__2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Squirrel.UpdateManager.DownloadReleasesImpl.<>c__DisplayClass2_0.<<DownloadReleases>b__1>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Squirrel.Utility.<>c__DisplayClass13_0`1.<<ForEachAsync>b__1>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Squirrel.UpdateManager.DownloadReleasesImpl.<DownloadReleases>d__2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Squirrel.UpdateManager.<DownloadReleases>d__8.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Squirrel.Update.Program.<Download>d__7.MoveNext()
   --- End of inner exception stack trace ---
   at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions)
   at System.Threading.Tasks.Task`1.GetResultCore(Boolean waitCompletionNotification)
   at System.Threading.Tasks.Task`1.get_Result()
   at Squirrel.Update.Program.executeCommandLine(String[] args)
   at Squirrel.Update.Program.main(String[] args)
   at Squirrel.Update.Program.Main(String[] args)
---> (Inner Exception #0) System.Net.WebException: The remote server returned an error: (404) Not Found.
   at System.Net.HttpWebRequest.EndGetResponse(IAsyncResult asyncResult)
   at System.Net.WebClient.GetWebResponse(WebRequest request, IAsyncResult result)
   at System.Net.WebClient.DownloadBitsResponseCallback(IAsyncResult result)
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Squirrel.Utility.<LogIfThrows>d__34.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Squirrel.FileDownloader.<DownloadFile>d__2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Squirrel.UpdateManager.DownloadReleasesImpl.<>c__DisplayClass2_0.<<DownloadReleases>b__1>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Squirrel.Utility.<>c__DisplayClass13_0`1.<<ForEachAsync>b__1>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Squirrel.UpdateManager.DownloadReleasesImpl.<DownloadReleases>d__2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Squirrel.UpdateManager.<DownloadReleases>d__8.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Squirrel.Update.Program.<Download>d__7.MoveNext()<---

Nevermind, it turns out it was a configuration problem with my web server.


Correctly apply mask when autofilling number

This PR fixes the autofilling of phone numbers (like iOS suggested fill at the top of the keyboard) by trimming the extra characters from the start of the input if it meets certain conditions.
This was a major issue before and essentially made autofilling impossible with this library.
The conditons are

Input starts with a country code
Input starts with a 0

Making this change means that instead of trimming the end of the number (current behaviour) and then the user needing to delete the autofilled data and manually enter their number leaving off the country code or 0 it will be automatically trimmed and therefore viable.
By checking for these conditions we preserve the excellent functionality of the library where the user cannot enter more characters than required for their countries mask.

Hey @GMoon777 , thanks for your PR...

Hey @GMoon777, thanks for your PR...and your issue is a great point...nice!
Im reviewing your PR and I think this PR is not the best way to solve your trouble...

With the logic of this PR, when I add a autofill with calling code (as +44), the country calling code still continues on input (withou the plus character, only)...and in my opnion...this doesn't look good on the input.
Why reason to keeps the digit 0 on start of a phone number? this make sense? (is an question, only)

I will study a better solution for your case soon...but for now, this PR is not a good solution...
Please, if Im wrong..let me know =)
🚀AstrOOnauta 🚀

@GMoon777 Autofilling fixed with next version. So, update your lib for the latest version v.0.7.5


How to set the duration of the animation?

Hello!
Awesome package, but is there anyway to modify the duration of the animation? It's changing between items too fast

Hi, thanks for the feedback!
Are you talking about the dismiss animation?

Yeah it seems too fast and it would be great to set the duration to make it look smoother

Hi, can this work?
I will add an external parameter for duration.

Available on v1.0.5.


Error adding musicxml

I tried to add musicxml support by adding the following bottom three values to schemas:
schemas = []string{
		"www.w3.org/2001/xml.xsd",
...
		"khronos.org/files/collada_schema_1_5",
		"www.musicxml.org/xsd/xml.xsd",
		"www.musicxml.org/xsd/xlink.xsd",
		"www.musicxml.org/xsd/musicxml.xsd",
	}

After this, I can rebuild and run xsd-makepkg and it properly generates musicxml.xsd.go, xlink.xsd.go, and xml.xsd.go. However, when creating a boilerplate program and simply including this new package:
package main
import fmt "fmt"
import "github.com/metaleap/go-xsd-pkg/www.musicxml.com/xsd/musicxml.xsd_go"

I get the following error at the command line when building or running this test program:
# github.com/metaleap/go-xsd-pkg/www.musicxml.com/xsd/musicxml.xsd_go
../../github.com/metaleap/go-xsd-pkg/www.musicxml.com/xsd/musicxml.xsd_go/musicxml.xsd.go:2234: undefined: XsdtNonNegativeInteger

I get the same error in a test.go file I created based on your other tests in go-xsd
Am I doing something wrong?

Not sure if this is useful for original poster anymore, but might help someone else running into a similar issue.
This looks similar to issue 12: https://github.com/metaleap/go-xsd/issues/12
It seems that the generator is writing XsdtType instead of xsdt.Type.
I believe the solution for your case is to change all instances of XsdtNonNegativeInteger to xsdt.NonNegativeInteger.


Specify app route in concourse job params

We're trying to blue-green deploy an app to multiple environments with different routes and environment variables in each environment.  The only thing preventing us from using the cf-resource is the lack of support for specifying routes in the concourse job.  As is, we can use different manifests for each environment, but it seems like overkill since there's only one difference between the environments (the route).
Looking at the code, it seems like this would involve adding an AddRoute method to https://github.com/concourse/cf-resource/blob/master/out/manifest.go .  I'd be happy to submit a pull request if you think this idea makes sense.

We have the same problem (for both routes and stack)

This would be great, we are also trying to push an app to multiple spaces and having route conflicts

I would look at a PR for this, but in general I don't think asking people to change their manifest via tasks is too big of an ask.
This could obviously end up in a combinatorial explosion where "my manifests are the same except for X parameter", and we end up in the same place as the old version of this resource that specified every parameter independently.


[lit/context] Declarative context registry

Should this be an RFC?

[X] This is not a substantial change

Which package is this a feature request for?
Context (@lit/context)
Description
Currently, the context API uses createContext to create a branded type using a cast. This is okay, but there is redundancy. If @lit/context exposed their own ContextRegistry interface for users to extend with their own symbol, the redundancy disappears. We also have stronger type guarantees and we can just pass around unique symbols as defined in TS.
For example, consider the following example:
const RouterContextKey = Symbol("RouterContextKey");
export const routerContext = createContext<
  RouterContext,
  typeof RouterContextKey
>(RouterContextKey)

where RouterContext is some value we are passing around.
Note there is a lot of redundancy in createContext because partial specialization (i.e. createContext<RouterContext>(RouterContextKey)) is not possible. With the proposed feature, this would change into
const RouterContextKey = Symbol("RouterContextKey");

declare module "@lit/context" {
  interface ContextMap {
    [RouterContextKey]: RouterContext;
  }
}

Note that this can all be implemented without breaking changes using overloads.
Alternatives and Workarounds
Just use the current API.

Oohhh excellent point! Thanks for raising this. I didn't realize createContext's typing had this problem.
Related TS issues:
https://github.com/microsoft/TypeScript/issues/10571
https://github.com/microsoft/TypeScript/issues/26242
I've naively just been using createContext<ContextValue>(key) without providing the second type arg which loses the type of the context key as it, instead of getting inferred, gets defaulted to unknown. Though I'm not sure how important that is in practice. I mostly cared that the context value type is present for the provider/consumer, and haven't come across a situation where the type of the context key mattered.
Nevertheless, it appears in the code that the intention was to allow stricter context type that incorporates the type of the key.
I don't know if the proposed ContextMap interface/registry is a more ergonomic solution though.
In the wild, creating a currying function which takes the specified type and returns a function that infers the rest seem to be the workaround people have adopted in the wild e.g. zustand, or similarly a static method that returns the type narrowed implementation e.g. react-redux.
Using zustand's approach would look something like this:
type CreateContext = {
  <ValueType>(): <K = unknown>(key: K) => Context<K, ValueType>;
  <ValueType, K = unknown>(key: K): Context<K, ValueType>;
};

function createContextImpl<ValueType, K = unknown>(key: K) {
  return key as Context<K, ValueType>;
}

export const createContext = (<K>(key?: K) =>
  key !== undefined
    ? createContextImpl(key)
    : createContextImpl) as CreateContext;

which allows
type Foo = {foo: string};
const key = Symbol('foo');
const ctx = createContext<Foo>()(key); // notice the extra `()`
//    ^? Context<typeof key, Foo>

I'm aware that currying is possible, but using a type registry would be better since we can just pass around the symbol instead of some branded object. It's more hassle to understand and as a bonus Lit users would be familiar with the pattern since we do

declare global {
    interface HTMLElementTagNameMap {
       ...
    }
}

That's a fair point. We have a clear relation of context key to context value so a registry makes sense.
The examples above are for state managers that take a custom user state type arg and infer the rest, without a clear mapping relation, and therefore resorting to the currying solution.

@jun-sheaf I am curious though, have you come across a situation where this kind of strict association of key type to the context object was needed? Maybe the problem is that we didn't need the key type to be part of the branded context to begin with. As long as the branded context includes the value type, what's the benefit of passing around a bare symbol versus a branded one?

I don't think createContext actually requires that boilerplate. It's an identity function that just adds a brand property to carry the value type. The key type is inferred from the argument, so you can write the key inline, like so:
export const routerContext = createContext<RouterContext>(Symbol('RouterContextKey'));

It could very well be the case that we don't really need to know the key since all we care about is it carrying the correct value type. I liken a registry to a list of dependencies. We use a custom version that has all the symbols in one file and declared in the registry. It's a good place to go for documentation, but it could also just be done using branded types.
Perhaps we can just remove the key type from the context.

The key type helps correctly type the identify function aspect of createContext() so I don't think we can remove it.
But removing it wouldn't lessen any boilerplate either, you would still write:
export const routerContext = createContext<RouterContext>(Symbol('RouterContextKey'));

^ the key type is completely inferred there and doesn't need to be written.

I don't think createContext actually requires that boilerplate. It's an identity function that just adds a brand property to carry the value type. The key type is inferred from the argument, so you can write the key inline, like so:

export const routerContext = createContext<RouterContext>(Symbol('RouterContextKey'));

The key type isn't inferred from the argument. However I think @augustjk and I agree that the key type is completely unnecessary.

The key type helps correctly type the identify function aspect of createContext() so I don't think we can remove it.

Could you elaborate? Are you talking about for comparison of to contexts? The question there is what reason do you have to compare two contexts? If you could come up with a reason, then this would strengthen the registry case as since it would imply users want a strong link between keys and values.


controller won't start due to secrets issue

ENV:
Kubernetes:
Client Version: version.Info{Major:"1", Minor:"2", GitVersion:"v1.2.0+5cb86ee", GitCommit:"5cb86ee022267586db386f62781338b0483733b3", GitTreeState:"not a git tree"}
Server Version: version.Info{Major:"1", Minor:"2", GitVersion:"v1.2.0", GitCommit:"5cb86ee022267586db386f62781338b0483733b3", GitTreeState:"clean"}`

Running in AWS (separate k8s masters, etcd cluster, nodes), CoreOS 899.11.0, workflow-dev
Pods:
ENG000657:scripts jasondumars$ kubectl get pods --namespace=deis
NAME                          READY     STATUS             RESTARTS   AGE
deis-builder-wm8uo            1/1       Running            1          16m
deis-controller-azbqs         0/1       CrashLoopBackOff   8          13m
deis-database-7qvq2           1/1       Running            0          16m
deis-logger-e1lxe             1/1       Running            0          16m
deis-logger-fluentd-ehrb0     1/1       Running            0          16m
deis-logger-fluentd-r4hx3     1/1       Running            0          16m
deis-minio-mswrt              1/1       Running            0          16m
deis-monitor-grafana-x3yct    1/1       Running            0          16m
deis-monitor-influxdb-mcogd   1/1       Running            0          16m
deis-monitor-stdout-8glu3     1/1       Running            0          16m
deis-monitor-telegraf-133mg   1/1       Running            0          16m
deis-monitor-telegraf-uudiz   1/1       Running            0          16m
deis-registry-qail0           1/1       Running            3          16m
deis-router-1cioi             1/1       Running            0          16m
deis-workflow-manager-iee7k   1/1       Running            0          16m

Controller will not start, and errors out with this:
$ kubectl --namespace=deis logs deis-controller-azbqs -p
system information:
Django Version: 1.9.6
Python 3.5.1
Django checks:
Traceback (most recent call last):
  File "/usr/lib/python3.5/site-packages/django/core/management/base.py", line 348, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/usr/lib/python3.5/site-packages/django/core/management/base.py", line 391, in execute
    saved_locale = translation.get_language()
  File "/usr/lib/python3.5/site-packages/django/utils/translation/__init__.py", line 183, in get_language
    return _trans.get_language()
  File "/usr/lib/python3.5/site-packages/django/utils/translation/__init__.py", line 57, in __getattr__
    if settings.USE_I18N:
  File "/usr/lib/python3.5/site-packages/django/conf/__init__.py", line 55, in __getattr__
    self._setup(name)
  File "/usr/lib/python3.5/site-packages/django/conf/__init__.py", line 43, in _setup
    self._wrapped = Settings(settings_module)
  File "/usr/lib/python3.5/site-packages/django/conf/__init__.py", line 120, in __init__
    raise ImproperlyConfigured("The SECRET_KEY setting must not be empty.")
django.core.exceptions.ImproperlyConfigured: The SECRET_KEY setting must not be empty.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/app/manage.py", line 12, in <module>
    execute_from_command_line(sys.argv)
  File "/usr/lib/python3.5/site-packages/django/core/management/__init__.py", line 353, in execute_from_command_line
    utility.execute()
  File "/usr/lib/python3.5/site-packages/django/core/management/__init__.py", line 345, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/usr/lib/python3.5/site-packages/django/core/management/base.py", line 360, in run_from_argv
    connections.close_all()
  File "/usr/lib/python3.5/site-packages/django/db/utils.py", line 230, in close_all
    for alias in self:
  File "/usr/lib/python3.5/site-packages/django/db/utils.py", line 224, in __iter__
    return iter(self.databases)
  File "/usr/lib/python3.5/site-packages/django/utils/functional.py", line 33, in __get__
    res = instance.__dict__[self.name] = self.func(instance)
  File "/usr/lib/python3.5/site-packages/django/db/utils.py", line 157, in databases
    self._databases = settings.DATABASES
  File "/usr/lib/python3.5/site-packages/django/conf/__init__.py", line 55, in __getattr__
    self._setup(name)
  File "/usr/lib/python3.5/site-packages/django/conf/__init__.py", line 43, in _setup
    self._wrapped = Settings(settings_module)
  File "/usr/lib/python3.5/site-packages/django/conf/__init__.py", line 120, in __init__
    raise ImproperlyConfigured("The SECRET_KEY setting must not be empty.")
django.core.exceptions.ImproperlyConfigured: The SECRET_KEY setting must not be empty.

Secret appears ok:
apiVersion: v1
data:
  secret-key: XxXxayBgSltIcFMzP14oakpFSVFxZVs/cyV8I3RMRXpFPF1yRkpDNHczPHhefT9tUHJGbkpiQEE2ZXZxS0dVbQ==
kind: Secret
metadata:
  annotations:
    chart.helm.sh/description: For testing only!
    chart.helm.sh/file: /Users/jasondumars/.helmc/workspace/charts/workflow-dev/manifests/deis-controller-secret-django-secret-key.yaml
    chart.helm.sh/name: workflow-dev
    chart.helm.sh/version: v2-beta
  creationTimestamp: 2016-05-11T17:20:35Z
  labels:
    heritage: deis
  name: django-secret-key
  namespace: deis
  resourceVersion: "16640"
  selfLink: /api/v1/namespaces/deis/secrets/django-secret-key
  uid: ac14043e-179c-11e6-af05-120592f139e9
type: Opaque```

Does this consistently happen? Can you cat /app/deis/settings.py from that box?

NAMESPACE   NAME                          READY     STATUS             RESTARTS   AGE
deis        deis-controller-wxsi8         0/1       CrashLoopBackOff   11         13m
I can't seem to get anything out of the pod.  It just says:
error: error executing remote command: Error executing command in container: container not found ("deis-controller")
Other pods exec fine, for example:
kubectl exec deis-workflow-manager-5imes --namespace=deis cat /etc/hosts --namespace=deis
Kubernetes-managed hosts file.
<IP_ADDRESS>	localhost

Have you observed this again?

I spoke with @jdumars and he can't reproduce this anymore. Going to close this until we we can reliably reproduce it


[WIP] React: Initial Setup & Main Page

[x] Init project
[x] Setup proper linting configuration
[x] Create Navbar Component
[ ] Create Footer Component
[ ] Create Main component for index.js page
[ ] Add Dark Mode and Theme Switching

This will be my todo-list for doing the initial main page set up for the React version. I'll be creating a draft PR and will be working on it. I'll add updates slowly after researching on making the project as modular as possible. If you have any suggestions or need me to do something in a certain way, do comment.

Latest commit will be deployed here: https://girlscript-chennai-website-deploy.now.sh/


API: expose isReadonlySymbol

Search Terms
readonly symbol property
Suggestion
Expose isReadonlySymbol as a method on TypeChecker.
Currently there's no way to tell for sure if a property is readonly.
While cumbersome userland implementations can check for readonly modifier in Symbol.declarations, if it's a GetAccessor without corresponding SetAccessor or if it originated in Object.definePropery(this, 'propName', {value, writable: false}).
However, there's one more thing: Readonly mapped type only sets Symbol.checkFlags which is internal.
So at the very least I need an API to tell if a property's symbol comes from a mapped type with a readonly modifier.
/cc @CSchulz you probably need the same API for the lint rule you want to develop.

This would be useful for us as well.


README: Kill keep-client quickstart reference

We have a lingering reference to a quickstart doc that no longer exists.  Updating the section to reflect current documentation for getting up and running.

added 2943 packages from 1542 contributors and audited 2413 packages in 171.65s

found 60 vulnerabilities (49 low, 11 high)

  run `npm audit fix` to fix them, or `npm audit` for details

Error: Cannot find module 'eth-json-rpc-filters'

🤔 not sure why this is failing in CI.

https://github.com/keep-network/keep-core/pull/1843 looks to  be failing elsewhere.  Possible a package is no  longer available, or moved, etc?

Error: Cannot find module 'eth-json-rpc-filters' should be fixed here
However, I'm not sure what is breaking Error: Cannot find module 'scrypt-shim'. Tried playing with web3 older versions, but didn't help.


Bug: curated syllabi cards layout

Steps to reproduce the problem

In the Curated Syllabi feature branch, create 3 syllabi
Notice that on large screens, the syllabi cards do not wrap to the next row

What is the current behavior?
Notice that on large screens, the syllabi cards do not wrap to the next row...
What is the expected behavior?
According to the design, the Syllabi card container should allow for two columns on larger breakpoints, and a single column at smaller breakpoints...
Screenshots
https://user-images.githubusercontent.com/6276074/193634753-ef037f7a-88ad-41c8-b97c-bca748295e6f.mov
Additional information
The implementation should follow this design layout:

This was closed via https://github.com/Qiskit/platypus/pull/1608


Chisel and Bits Compatability

This is in regard to a question that came up in the discord chat.
When using ContentTweaker, B.A.S.E, and Chisel and bits, you experience some graphical issues.
When Chiseled, Blocks of metal go to the purple and black unknown texture, and ores go to their base material, both with atint overlay of the ore they had on them.
For example, here is a screen shot of Block of Copper and Copper Ore. The bottom blocks have had one bit taken out of them.

After regenerating the models, normal blocks work. The only thing still not functioning properly (at least related to the two examples) is the ore.


Agent connection fail

Agent Version and Platform
Agent version 2.136.1 win-x64
Platform Windows 2016
VSTS Type and Version
On-Prem TFS 2018 Update 3,1
What's not working?
System wide agent failure after restart of application tier.  We are running 10 build machines and approximately 30 agents which all failured with the following
[2019-01-02 07:43:44Z INFO RSAEncryptedFileKeyManager] Loading RSA key parameters from file D:\Builds\Agent1.credentials_rsaparams
[2019-01-02 07:43:44Z INFO RSAEncryptedFileKeyManager] Loading RSA key parameters from file D:\Builds\Agent1.credentials_rsaparams
[2019-01-02 07:45:24Z WARN VisualStudioServices] GET request to http://tfsapp02:8080/tfs/_apis/connectionData?connectOptions=1&lastChangeId=32&lastChangeId64=32 timed out after 100 seconds.
[2019-01-02 07:45:24Z INFO VisualStudioServices] Finished operation Location.GetConnectionData
[2019-01-02 07:45:24Z INFO AgentServer] Catch exception during connect. 4 attemp left.
This problem first started after upgrading from TFS 2018 update 2 to TFS update 3.1, where we saw this problem after immediately updating.  Non of our agents could connect to the TFS instance.  After many hours of debugging the problem was resolved by removing the https binding.   The problem reoccured after a routine restart of application tieir a week ago.  After many restarts of our builder servers and the application tieir the problem was resolved by adding and removing a IIS binding on the application tieir.
Agent_20190102-074341-utc.log
Agent and Worker's Diagnostic Logs
I have atatched ahent log file.

Aditional information.
I have examined the ISS log on application tieir and it is the POST of the token that is failing,
POST /tfs/_apis/oauth2/token - 80 - <IP_ADDRESS> VstsAgentCore-win-x64/2.136.1+(Microsoft+Windows+10.0.14393)+VSServices/16.136.27801.0+(NetStandard;+Microsoft+Windows+10.0.14393) - 400 0 0 15
Is there any logging made for the oauth2/token?

@TingluoHuang thanks for that but the eventd for that period have been overwritten.  I have increase the size of the log.  Are there any other places that could help me debuf this issue?  Does that log output mean anything to you?

POST /tfs/_apis/oauth2/token - 80 - <IP_ADDRESS> VstsAgentCore-win-x64/2.136.1+(Microsoft+Windows+10.0.14393)+VSServices/16.136.27801.0+(NetStandard;+Microsoft+Windows+10.0.14393) - 400 0 0 15

the server returns 400  which is weird, you might want to capture Http trace in the agent side, and see what's in the payload of that 400 response.
https://docs.microsoft.com/en-us/azure/devops/pipelines/troubleshooting?view=vsts#windows


Feat(contracts)/upgradability

This PR follows the #1136 PR which introduced a UUPS Proxy.
A few changes to the upgradability mechanism were made. The SortitionModule and the KlerosCore have been made upgradeable. This is the first step before having the whole Kleros-v2 stack upgradeable. It can be considered WIP as the KlerosCore might need some design changes.
Upgrade Mechanism
Instead of relying on a boolean initialized which only allows initializing the proxy storage once, the upgradeable contracts inherit from Initializable.sol, copied from OpenZeppelin.
It introduces two modifiers: initializer and reinitializer(uint8 version), and the function _disableInitializers(). initializer acts the same as reinitializer(1).
This contract allows re-initializing the proxy storage, if needed, by using the reinitializer(uint8 version) with a higher version than the previous one. It is quite useful when adding new modules requiring initialization.
The version increment can be superior to 1, which allows initializing parent contracts in between if needed. OpenZeppelin has some docs about it here. The version cap is 255, a contract whose version is 255 cannot be re-initialized.
This is what _disableInitializers() do, bumping the initialize version to 255. Calling it in the constructor of the implementation contract will disable the initialization, preventing potential attackers from taking over the implementation contract (see self-destruct attacks..)
KlerosCore Upgradability
The KlerosCore contract is heavy: the deployed bytecode weighs 22.220 kiB in the dev branch (sizes are taken from hardhat-contract-sizer, running yarn hardhat size-contracts). Making it upgradeable (transferring the constructor to an initialize() function and inheriting from two abstract contracts UUPSProxiable and Initializable adds 2.183 kiB to the deployed bytecode. Without reducing the contract size, the upgradeable version of KlerosCore is above 24 kiB, making it impossible to deploy on mainnet.
This is why an optimization of the KlerosCore size is needed.
Optimization
With different optimizations (errors instead of require statements, private constants over unused public constants, caching calls to storage in a variable if there are more than 3 calls to it (empirical value based on tests, I haven't compared the ops but it should add up), removing cache variables if called less than 3 times (low impact, can even be considered negative as it reduces the readability) the size of KlerosCore have been reduced by almost 1 kiB.
Here is a recap table of the measured sizes of KlerosCore:

KlerosCore version
Size of KlerosCore (kiB) - hardhat-contract-sizer
Changes (+/- kiB)

Original (dev branch)
22.220
0

Upgradeable KlerosCore
24.403
+ 2.183

Optimized Upgradeable KlerosCore
23.422
- 0.980

Fig.1.1: KlerosCore size comparison
Here is a table giving measured changes of specific changes made to KlerosCore when optimizing it:

Change to KlerosCore
Measured impact of change - hardhat-contract-sizer (kiB)

Replace modifier onlyByGovernor by a private view function
- 0.366

Cache courts[_courtID] in storage variable court in changeCourtParameters()
- 0.326

Make public constants private
- 0.257

Remove cached variable disputeKit in DisputeKitJump case from appeal()
- 0.015

Remove cached variable court in getTimesPerPeriod()
-0.008

Remove cached variable rate in changeCurrencyRates()
- 0.005

Fig.1.2: Details of KlerosCore optimization
As the details show, removing cached variables when they are called less than 3 times has a minimal impact. In the case of appeal() the variable caches an extremely long path which is of great help for readability purposes. However, for the other two cases, caching a variable has small changes to readability. Even if pretty small, I'd take those changes into account.
Replacing a modifier by a private function is kind of desperate but, as the modifier is called many times, it has the biggest impact. It allows having 0.518 kiB free rather than 0.212 kiB. But even with 0.5 kiB, adding new functionality is rather limited.
This is why splitting the contracts in (at least) two should be the most considered option.
Splitting KlerosCore
Due to its size and the difficulty of reducing it through optimization, it is better to split up KlerosCore.
The split design choices must be well thought out and semantically correct. I haven't had time to think much about it. The functions specified in IArbitratorV2 could be split into one contract, as well as the governance-related functions.
Upgrade Scripts
Proxy Options
To deploy the upgradeable contracts through a custom UUPS proxy, changes to the deployment scripts had to be made, defining the proxy options and how hardhat-deploy should behave when upgrading the proxy. The previous PR #1136's part on the deployment script is outdated as the upgrading mechanism doesn't work due to missing flags.
Here is an updated version:
const sortitionModule = await deploy("SortitionModule", {
    from: deployer,
    proxy: {
      proxyContract: "UUPSProxy",
      proxyArgs: ["{implementation}", "{data}"],
      checkProxyAdmin: false,
      checkABIConflict: false,
      execute: {
        init: {
          methodName: "initialize",
          args: [deployer, KlerosCoreAddress, 1800, 1800, rng.address, RNG_LOOKAHEAD], // minStakingTime, maxFreezingTime
        },
        onUpgrade: {
          methodName: "governor",
          args: [],
        },
      },
    },
    log: true,
  });

Setting the checkProxyAdmin and checkABIConflict flags to false allows not checking if there is a deployed Admin contract or a selector conflict between the proxy functions and the implementation functions, which is only necessary for Transparent proxies.
The onUpgrade function is defined. This specifies the post-upgrade function, the function to be called once the upgrade is done. There usually is no need to call a post-upgrade function unless the proxy storage needs to be initialized again. However, if no updateMethod is defined (i.e. onUpgrade field is not defined), hardhat-deploy will try calling upgradeTo() which is not defined in UUPSProxiable.sol (as in the new versions of OpenZeppelin's proxy).
Calling a public variable getter is a workaround to bypass this limitation.
I have tweaked the hardhat-deploy package to allow specifying a custom upgrade function (which function with which arguments to be called when an upgrade needs to be done). For more details, you can check this issue and PR that I've proposed to hardhat-deploy, but no feedback has been given yet... and therefore not merged.
For testing purposes, a package has been published. It allows specifying the upgrade function to be used when upgrading through the new ProxyOptions field upgradeFunction:
upgradeFunction: {
    methodName: "upgradeToAndCall",
    upgradeArgs: ['{implementation}', '{data}']
},

Upgrading Multiple contracts
Selective Upgrades
The current deployment scripts deploy multiple contracts at once. When we want to upgrade a specific contract, using these scripts would re-deploy non-upgradeable contracts, which are not to be used by the upgraded contracts (unless the proxy storage is also updated). Therefore a script for upgrades only must be written.
However, we should be able to select which contract to upgrade rather than having hardhat-deploy trying to upgrade every upgradeable contract in one script.
To do so, every upgradeable contract should have its XX-upgrade-contract-name.ts script with at least two tags: ["Upgrade", "ContractName"]
Prior to v0.11.37, hardhat-deploy doesn't allow executing deployment scripts that intersect the given tags:
yarn hardhat deploy --network localhost --tags Upgrade,KlerosCore will try running all scripts with the tag "Upgrade" or "KlerosCore".
The version v0.11.37 introduces the flag --tags-require-all which will look for the intersection of the given tags:
yarn hardhat deploy --network localhost --tags Upgrade,KlerosCore --tags-require-all will try running all scripts with the tags "Upgrade" and "KlerosCore".
It allows a more selective choice of deployments with proper tags (not having a unique tag for the sole purpose of running it alone). However, it doesn't support combining AND & OR operations with tags.
But what if we want to upgrade multiple contracts in a specific order?
Upgrading contracts in chains
To upgrade multiple contracts, the scripts should have a common tag/group of tags (e.g. "Upgrade" for every upgrade script, ["Upgrade", "Arbitration"] to upgrade all arbitration contracts...).
Then, the scripts should be ordered with two digits at the beginning of the filename. For example, upgrading the KlerosCore before the SortitionModule would be: 00-upgrade-kleros-core.ts and 01-upgrade-sortition-module.ts.
NOTE on upgradeable contracts importing another upgradeable contract
The SortitionModule imports KlerosCore to have its interface and interact with it. As per the Solidity docs, the deployed bytecode includes an extra 33 bytes which are metadata. These metadata are the ipfs hash of the compiler version, the sources used, the ABI and NatSpec documentation.
More details can be found in this short article.
This metadata includes the sources used, so it is based on the whole source code of the contract, which includes the imported contract source code.
When modifying an imported contract (e.g. KlerosCore), it will modify the deployed bytecode of the contract importing the modified contract (e.g. SortitionModule). As a result, if we perform an upgrade of the KlerosCore, running all upgrade scripts thinking that only the KlerosCore will be upgraded (only changes made), hardhat-deploy will also upgrade the SortitionModule due to this metadata changes. If overlooked, it results in a loss of funds (and block size).
The metadata can be excluded from the bytecode with the flag --no-cbor-metadata when compiling. Metadata are used when verifying the source code, they are crucial for transparency towards users.
Importing an interface instead of the whole contract to make external calls to it should be considered.
Tests
Tests covering the UUPS proxy have been written, but they don't include the Initializable contract in it. It should be done though.
The arbitration/index.ts#DisputeKitClassic has been updated, replacing the SortitionModule & KlerosCore factories with a new deployment, specific to the test.
No tests including the upgrade scripts were written, it has been tested empirically.

Deploying a first version with yarn start-local
Redeploy the whole thing with yarn deploy-local, performing an upgrade while doing so (but ineffective as it re-deploy many contracts which aren't used)
OR
Upgrade by running chosen upgrade scripts, such as:

yarn hardhat deploy --network localhost --tags Upgrade
yarn hardhat deploy --network localhost --tags Upgrade,SortitionModule --tags-require-all
yarn hardhat deploy --network localhost --tags Upgrade,KlerosCore --tags-require-all

Some yarn scripts should be added to the package.json to easily launch upgrades.
Hardhat tests for the upgrading through hardhat-deploy should also be written. This can be done by making use of deployments.fixture. Broadly, more tests should be written, to have a better coverage of v2 contracts.

Iterations on @zmalatrax's work

DRY refactor of the deployment scripts to reuse the deployment logic more easily.
Made other contracts upgradable: the dispute kits, the gateways, the randomiser RNG, the policy and dispute template registries, in addition to KlerosCore and SortitionModule already migrated by @zmalatrax.
Some state variables were initialized during definition, this is now done in the initializer function (example 1, example 2).
Some state variables were immutable, the keyword was removed as these variables cannot be set by the initializer function.
Fixed some nonce arithmetics (since it takes 2 txs to deploy) and we usually want to compute the proxy address which is deployed after the implementation.
Inlined and removed the AbstractedDisputeKit, which made initialization harder and was going against our coding principles anyway.
Cherry-picked some changes from #1225 to reduce the KlerosCore bytecode size.
Added a script calling the Etherscan API to mark the proxy as such (so the buttons Read/Write as Proxy do show up).
The markdown generation script now renders the proxy and implementation as one line


Touching with lock_version

Hi,
Using the counter_culture :category, :touch => true works fine for updating datetime stamps but it doesn't work with optimistic lock_version columns (it doesn't bump the version number on recalculation).  Is there a reason for this?  And/or some way around this. I'm using it for a shopping cart type solution with quantity updates.  I would like these to bump the lock_version somehow when quantities change.
Thanks,

counter_culture doesn't call touch on the model, it updates the timestamp directly. See here. I'd be open to adding support for bumping the lock_version there if you wanted to open a PR.


[WIP] Use latest k3s version

Looks like using K8s 1.25 will have to wait for postgres-operator 1.9. While I'm not seeing the same postgres startup issue as reported when running the 1.8.2 operator on K8s v1.25, our db-init job keeps crash looping and reporting authentication failed for galaxydbuser in the log. Meanwhile, the chart continues to deploy fine with the operator on 1.24 as well as without the operator. Other than that, I don't have other leads as to why it won't run on 1.25.

Postgres Operator v1.9 has been released (https://github.com/zalando/postgres-operator/releases/tag/v1.9.0), so hopefully our tests pass now.
Meanwhile, K8s 1.26 has been released.


No failure logs attachement available

Describe the bug
Trying to contribute an update of libsmb2 port, builds are failing for android but I cannot access error logs in attachement.
See https://dev.azure.com/vcpkg/public/_build/results?buildId=110599&view=ms.vss-test-web.build-test-results-tab&runId=285585&resultId=100000&paneView=attachments

I need to know if it is an issue in the port or in the android build of the library.
Environment

OS: android

To Reproduce
Did not try to reproduce
Expected behavior
Be able to access failure logs.
Failure logs
NONE
Additional context
N/A

The logs are it the CI run's artifacts.
https://dev.azure.com/vcpkg/public/_build/results?buildId=110599&view=artifacts&pathAsName=false&type=publishedArtifacts

Thank you, I had found the page but didn't notice the dots on the right hand side.

Yeah, everybody earns extra :star: :star: :star: :star: :star: for spotting that detail.


cmd/cgo: use void keyword in generated headers on extern'd function declarations with no params

Proposal Details
I propose that the generated header files from --buildmode=c-shared have void in the parenthesis when the extern'd function takes no parameters.
In C, function declarations with an empty list of parameters means that the parameters are unknown. The way to declare a function that takes no parameters is to put the void keyword in the parenthesis. If you omit the void keyword then callers are allowed to specify any number of parameters to the function, although compilers can issue warnings when this happens.
In my case I'm using jextract to generate java bindings for a header file generated by a go project. It's is kind of a new tool. When it sees the empty parameter list it treats the function as variadic which looks slightly different on the java side. This doesn't really matter much though. I'm just including this here for context.
I'm unsure what backwards compatibility concerns something like this would have. In the case of jextract it changes everything about the function call on the java side. I don't know if there are other tools that would be greatly affected by this change.

I don't think this has to be a proposal, I think we can just make this change.  Want to send a patch?  Thanks.

Is this still open and needs a patch?

Yes.  Any patch should appear in this issue, and I don't see one.


[wip] travis databases

Main focus is to remove sbt docker image and use the one provided by travis as well as with databases
@getquill/maintainers

@mentegy does that mean that the build wouldn't use docker anymore?

@fwbrasil it will, we need to test sqlserver and orientdb. However travis build won't use bst docker image and other databases which is provided by it. I expect speed boost from it.
However images they will be kept for local dev experience.

@getquill/maintainers @fwbrasil @deusaquilus it's ready for review.

the build config for release is broken on master. Merging this to see if it's fixed

@fwbrasil I think it is relates to this change https://github.com/getquill/quill/commit/fb28553235e15af6ed9b5d0996f0e8a006392204#diff-bcda6e8e16aada096dc8c129dfe51284 ? I remember the same issue Please enter PGP passphrase (or ENTER to abort):  in my pet project when i set wrong credentials.sbt


Task List: Repos to fork and populate

We need to populate the MCB curriculum with all the coursework and tasks we are assigning.
Steps

Fork the matching CYF repo
Turn on Issues in settings
Add the new repo to the planner
Run npx batch-issue-cloner --source CodeYourFuture/Module-NAME --destination Migracode-Barcelona/Module-NAME

If you get a rate limit warning "submitted too quickly..." then run the command again after a bit. It will backfill any missing issues.
Key repo list

[x] https://github.com/CodeYourFuture/Course-Fundamentals
[x] https://github.com/CodeYourFuture/Module-HTML-CSS
[x] https://github.com/CodeYourFuture/Module-JS1
[x] https://github.com/CodeYourFuture/Module-JS2
[x] https://github.com/CodeYourFuture/Module-JS3
[x] https://github.com/CodeYourFuture/JS3-Module-Project/
[x] https://github.com/CodeYourFuture/Module-Frameworks
[x] https://github.com/CodeYourFuture/Module-Servers
[x] https://github.com/CodeYourFuture/Module-Databases
[x] https://github.com/CodeYourFuture/Full-Stack-Project-Assessment
[x] https://github.com/CodeYourFuture/Module-The-Launch

All repos for the modules have been completed successfully 👍🏼

Amazing!
@sairaheta1  please approve my pending github bearer token so I can set up https://github.com/Migracode-Barcelona/curriculum

@SallyMcGrath Approved!


ui; fix admin dashboard vue warnings

Description
Fixes warnings on admin dashboard in console.
Types of changes

[ ] Breaking change (fix or feature that would cause existing functionality to change)
[ ] New feature (non-breaking change which adds functionality)
[ ] Bug fix (non-breaking change which fixes an issue)
[ ] Enhancement (improves an existing feature and functionality)
[x] Cleanup (Code refactoring and cleanup, that may add test cases)
[ ] build/CI

Feature/Enhancement Scale or Bug Severity
Feature/Enhancement Scale

[ ] Major
[ ] Minor

Bug Severity

[ ] BLOCKER
[ ] Critical
[ ] Major
[ ] Minor
[ ] Trivial

Screenshots (if appropriate):
Without change

How Has This Been Tested?

How did you try to break this feature and the system with this change?

@blueorangutan ui

@shwstppr a Jenkins job has been kicked to build UI QA env. I'll keep you posted as I make progress.

UI build failed: :heavy_multiplication_x:
(SL-JID-360)

@blueorangutan ui

@shwstppr a Jenkins job has been kicked to build UI QA env. I'll keep you posted as I make progress.

UI build: :heavy_check_mark:
Live QA URL: https://qa.cloudstack.cloud/simulator/pr/9140 (QA-JID-361)

LGTM, but without toFixed(2) how can we ensure upto two max. decimal places?

LGTM, but without toFixed(2) how can we ensure upto two max. decimal places?

It must be handled by :format


parser: support check constraint info [DNM]

Signed-off-by: Arenatlx<EMAIL_ADDRESS>
What problem does this PR solve?
support check constraint info.
What is changed and how it works?
1：add constraint info in model.go
2：add a item to store the check constraint info in tableInfo.
Check List
Tests

Integration test

Code changes

Has changed the model

Related changes

Need to update the documentation

duplicated with #871


recompiling removes ts file content randomly.

Versions
Angular CLI: 6.0.3
Node: 9.1.0
OS: win32 x64
Angular: 6.0.2
... animations, common, compiler, compiler-cli, core, forms
... http, language-service, platform-browser
... platform-browser-dynamic, router

Package                           Version
-----------------------------------------------------------
@angular-devkit/architect         0.6.3
@angular-devkit/build-angular     0.6.3
@angular-devkit/build-optimizer   0.6.3
@angular-devkit/core              0.6.3
@angular-devkit/schematics        0.6.3
@angular/cdk                      5.0.4
@angular/cli                      6.0.3
@ngtools/webpack                  6.0.3
@schematics/angular               0.6.3
@schematics/update                0.6.3
rxjs                              6.1.0
typescript                        2.7.2
webpack                           4.8.3

Repro steps
not identified happens randomly.
Observed behavior
from time to time, when i have file open for editing or adding new file but important thing here is that angular is running in development mode and save reloads and rebuilds app. sometimes it complains complains about file not being found. whenever I open ts file after this file content is completely empty and something removes ts file content completely.
`
Desired behavior
file content is not removed .
Mention any other details that might be useful (optional)
this is normal angular 6 application the only thing is i have entries in tsconfig like those.

checking output information gives no error or exception.

This appears to be a bug but we will need to look at a reproduction to find and fix the problem. Can you setup a minimal reproduction please?
You can read here why this is needed. A good way to make a minimal reproduction is to create a new app via ng new repro-app and adding the minimum possible code to show the problem. Then you can push this repository to github and link it here.

I would love to do this but as i explained there is nothing particular to do except live experience that i have there is nothing that is some logic based because it might happen randomly and i can not really hunt for this but mostly it happens i think whenever i have app running in development mode and i add new file or entry in ts and save file but again, this is very general statement the only thing i know is the reality i see every day.

any update on this?

It's not something we can reproduce. I don't know of anything we do that can cause that... Have you tried in another machine? It might be something on your current setup.

tried on another machine and can not reproduce there as well but fact is primary workstation does this almost every day
reinstalling Windows is not an option :) maybe there is something i can collect logs from and send it to you?


Remove TODO in test/acvp_test.c related to setting AES-GCM iv.

Fixes #14330

Checklist

[ ] documentation is added or updated
[ ] tests are added or updated

24 hours has passed since 'approval: done' was set, but this PR has failing CI tests.  Once the tests pass it will get moved to 'approval: ready to merge' automatically, alternatively please review and set the label manually.

This pull request is ready to merge

Merged to master.


TypeError: Offload_LlamaModel.forward() got an unexpected keyword argument 'position_ids' after today's commits

Crash on first inference when running llama with gptq latest 128 groupsize models, was working till yesterday
Pulled latest code today on both webui and gptq repo and it's crashing

Pull latest webui commit
Pull latest commit in cuda branch in llama-gptq repo
Use this python server.py --wbits 4 --groupsize 128 --pre_layer 2 --model llama-65b-4bit-128g --cai-chat --listen --listen-port=8888 to load llama 65b 4bit 128 group size
The model loads normally and the UI starts
Send any message and click generate then error

Error:
Traceback (most recent call last):
   File "/app/modules/callbacks.py", line 64, in gentask
     ret = self.mfunc(callback=_callback, **self.kwargs)
   File "/app/modules/text_generation.py", line 222, in generate_with_callback
     shared.model.generate(**kwargs)
   File "/usr/local/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
     return func(*args, **kwargs)
   File "/usr/local/lib/python3.10/site-packages/transformers/generation/utils.py", line 1462, in generate
     return self.sample(
   File "/usr/local/lib/python3.10/site-packages/transformers/generation/utils.py", line 2478, in sample
     outputs = self(
   File "/usr/local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1194, in _call_impl
     return forward_call(*input, **kwargs)
   File "/usr/local/lib/python3.10/site-packages/transformers/models/llama/modeling_llama.py", line 710, in forward
     outputs = self.model(
   File "/usr/local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1194, in _call_impl
     return forward_call(*input, **kwargs)
 TypeError: Offload_LlamaModel.forward() got an unexpected keyword argument 'position_ids'

also opened an issue in webui repo https://github.com/oobabooga/text-generation-webui/issues/623

pip uninstall transformers
pip install git+https://github.com/huggingface/transformers

Didn't fix the problem, also this problem is only present when offloading to cpu (pre_layer present) anything other than that works

https://github.com/qwopqwop200/GPTQ-for-LLaMa/commit/a7143402398ba688fb2f8f6429ee1e6b476b6720 fixed it thanks a lot
Closing


python310Packages.enaml: 0.15.1 -> 0.15.2

Automatic update generated by nixpkgs-update tools. This update was made based on information from passthru.updateScript.
meta.description for python310Packages.enaml is: "Declarative User Interfaces for Python"
meta.homepage for python310Packages.enaml is: "https://github.com/nucleic/enaml"
meta.changelog for python310Packages.enaml is: ""
Updates performed

Ran passthru.UpdateScript

To inspect upstream changes
Impact

Checks done (click to expand)

built on NixOS
The tests defined in passthru.tests, if any, passed
found 0.15.2 with grep in /nix/store/whk23jm24fym5x7jzbfaq794vkhar96l-python3.10-enaml-0.15.2
found 0.15.2 in filename of file in /nix/store/whk23jm24fym5x7jzbfaq794vkhar96l-python3.10-enaml-0.15.2

Rebuild report (if merged into master) (click to expand)

6 total rebuild path(s)

6 package rebuild(s)

First fifty rebuilds by attrpath
inkcut
inkscape-extensions.inkcut
python310Packages.enaml
python310Packages.enamlx
python39Packages.enaml
python39Packages.enamlx

Instructions to test this update (click to expand)

Either download from Cachix:
nix-store -r /nix/store/whk23jm24fym5x7jzbfaq794vkhar96l-python3.10-enaml-0.15.2 \
  --option binary-caches 'https://cache.nixos.org/ https://nix-community.cachix.org/' \
  --option trusted-public-keys '
  nix-community.cachix.org-1:mB9FSh9qf2dCimDSUo8Zy7bkq5CX+/rkCWyvRCYg3Fs=
  cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY=
  '

(The Cachix cache is only trusted for this store-path realization.)
For the Cachix download to work, your user must be in the trusted-users list or you can use sudo since root is effectively trusted.
Or, build yourself:
nix-build -A python310Packages.enaml https://github.com/r-ryantm/nixpkgs/archive/9b0252fd50b19cfaa6c0c52102437b1500f30ccb.tar.gz

After you've downloaded or built it, look at the files and if there are any, run the binaries:
ls -la /nix/store/whk23jm24fym5x7jzbfaq794vkhar96l-python3.10-enaml-0.15.2
ls -la /nix/store/whk23jm24fym5x7jzbfaq794vkhar96l-python3.10-enaml-0.15.2/bin

Pre-merge build results
We have automatically built all packages that will get rebuilt due to
this change.
This gives evidence on whether the upgrade will break dependent packages.
Note sometimes packages show up as failed to build independent of the
change, simply because they are already broken on the target branch.
Result of nixpkgs-review run on x86_64-linux 1

6 packages built:

inkcut
inkscape-extensions.inkcut
python310Packages.enaml
python310Packages.enamlx
python39Packages.enaml
python39Packages.enamlx

Maintainer pings
cc @raboof for testing.

Result of nixpkgs-review pr 187523 run on x86_64-linux 1

6 packages built:

inkcut
inkscape-extensions.inkcut
python310Packages.enaml
python310Packages.enamlx
python39Packages.enaml
python39Packages.enamlx


Add modelci cli start func and add contribute guide

Closes #153 #152

What has been done to verify that this works as intended?
Add a start function based on this  StackOverflow question:  https://stackoverflow.com/questions/37041126/unexpected-keyword-argument-in-python-click, so we can start modelci with GPU enable use following script:
import modelci.cli
# with GPU avaliable
modelci.cli.__start__(gpu=True)

Why is this the best possible solution? Were any other approaches considered?
It's provide a way to start modelci with GPU support via python script
start with GPU support via command line
source scripts/setup_env.sh
python modelci/cli/__init__.py start --gpu

How does this change affect users? Describe intentional changes to behavior and behavior that could have accidentally been affected by code changes. In other words, what are the regression risks?
Does this change require updates to documentation?
Yes
Before submitting this PR, please make sure you have:

[ x ] run python -m pytest tests/ and confirmed all checks still pass.
[ x ] verified that any code or assets from external sources are properly credited.

@univerone can close the pr and propose a new pr for contributing.md


Code doesn't translate

import urllib.request
import json
import datetime
import random
import string
import time
import os
import sys
os.system("title WARP-PLUS-CLOUDFLARE By ALIILAPRO")
os.system('cls' if os.name == 'nt' else 'clear')
print('      _______ _      __________________       _______ _______ _______ _______\n'
'     (  ___  | \     \__   __|__   __( \     (  ___  |  ____ |  ____ |  ___  )\n'
'     | (   ) | (        ) (     ) (  | (     | (   ) | (    )| (    )| (   ) |\n'
'     | (___) | |        | |     | |  | |     | (___) | (____)| (____)| |   | |\n'
'     |  ___  | |        | |     | |  | |     |  ___  |  _____)     __) |   | |\n'
'     | (   ) | |        | |     | |  | |     | (   ) | (     | (\ (  | |   | |\n'
'     | )   ( | (____/\__) (_____) (__| (____/\ )   ( | )     | ) \ \_| (___) |\n'
'     |/     \(_______|_______|_______(_______//     \|/      |/   \__(_______)\n')
print ("[+] ABOUT SCRIPT:")
print ("[-] With this script, you can getting unlimited GB on Warp+.")
print ("[-] Version: 4.0.0")
print ("--------")
print ("[+] THIS SCRIPT CODDED BY ALIILAPRO")
print ("[-] SITE: aliilapro.github.io")
print ("[-] TELEGRAM: aliilapro")
print ("--------")
referrer = input("[#] Enter the WARP+ ID:")
def genString(stringLength):
	try:
		letters = string.ascii_letters + string.digits
		return ''.join(random.choice(letters) for i in range(stringLength))
	except Exception as error:
		print(error)
def digitString(stringLength):
	try:
		digit = string.digits
		return ''.join((random.choice(digit) for i in range(stringLength)))
	except Exception as error:
		print(error)
url = f'https://api.cloudflareclient.com/v0a{digitString(3)}/reg'
def run():
	try:
		install_id = genString(22)
		body = {"key": "{}=".format(genString(43)),
				"install_id": install_id,
				"fcm_token": "{}:APA91b{}".format(install_id, genString(134)),
				"referrer": referrer,
				"warp_enabled": False,
				"tos": datetime.datetime.now().isoformat()[:-3] + "+02:00",
				"type": "Android",
				"locale": "es_ES"}
		data = json.dumps(body).encode('utf8')
		headers = {'Content-Type': 'application/json; charset=UTF-8',
					'Host': 'api.cloudflareclient.com',
					'Connection': 'Keep-Alive',
					'Accept-Encoding': 'gzip',
					'User-Agent': 'okhttp/3.12.1'
					}
		req         = urllib.request.Request(url, data, headers)
		response    = urllib.request.urlopen(req)
		status_code = response.getcode()
		return status_code
	except Exception as error:
		print(error)

g = 0
b = 0
while True:
	result = run()
	if result == 200:
		g = g + 1
		os.system('cls' if os.name == 'nt' else 'clear')
		print("")
		print("                  WARP-PLUS-CLOUDFLARE (script)" + " By ALIILAPRO")
		print("")
		animation = ["[■□□□□□□□□□] 10%","[■■□□□□□□□□] 20%", "[■■■□□□□□□□] 30%", "[■■■■□□□□□□] 40%", "[■■■■■□□□□□] 50%", "[■■■■■■□□□□] 60%", "[■■■■■■■□□□] 70%", "[■■■■■■■■□□] 80%", "[■■■■■■■■■□] 90%", "[■■■■■■■■■■] 100%"]
		for i in range(len(animation)):
			time.sleep(0.5)
			sys.stdout.write("\r[+] Preparing... " + animation[i % len(animation)])
			sys.stdout.flush()
		print(f"\n[-] WORK ON ID: {referrer}")
		print(f"[:)] {g} GB has been successfully added to your account.")
		print(f"[#] Total: {g} Good {b} Bad")
		print("[*] After 18 seconds, a new request will be sent.")
		time.sleep(18)
	else:
		b = b + 1
		os.system('cls' if os.name == 'nt' else 'clear')
		print("")
		print("                  WARP-PLUS-CLOUDFLARE (script)" + " By ALIILAPRO")
		print("")
		print("[:(] Error when connecting to server.")
		print(f"[#] Total: {g} Good {b} Bad")

Not a python dev, not sure what the issue is, need some help.

Hi, thanks for the issue. Pytago tends to experience two types of bugs.

Things that cause Pytago to fail critically. These are high priority to fix and you've provided an example of one here. The reason this code doesn't even produce some kind of result is this line:

sys.stdout.write("\r[+] Preparing... " + animation[i % len(animation)])

There's some code to identify writing to a file that got triggered inappropriately here and then failed.  I'll try to get a fix out for that soon, but for now you can probably replace that with a print and get a partially correct translation.

The second kind of issue is translation errors. These can range from simple to incredibly complex so to fix those we need to create an issue that simplifies the feature being requested as much as possible. There's a lot of library usage in this script that there won't be good support for yet including...

strings.digits
sys.stdout.flush
urllib.request
os.system

Just to name a few. Each of these items will need individual issues to address them with well defined examples of the desired usage to flesh them out. If you make these (see the issue template) I will be happy to look into them.

Yes please, also thank you.


GPL-692 BUG - Lighthouse - Exceptions from Cog Uk Id creation

Describe the bug
Exceptions being thrown during Baracoda COG UK id creation.
RT Ticket Number
na
To Reproduce
Seems inconsistent. Several noticed in slack output over last few weeks.
Expected behaviour
What was expected to happen.
Screenshots
If applicable, add screenshots to help explain your problem.
Additional context
Example of Exception:
2020-10-12 13:07:04,232 lighthouse.blueprints.plates:47 ERROR Unable to create COG barcodes
Traceback (most recent call last):
File "/code/lighthouse/blueprints/plates.py", line 45, in create_plate_from_barcode
centre_prefix = add_cog_barcodes(samples)
File "/code/lighthouse/helpers/plates.py", line 58, in add_cog_barcodes
raise Exception("Unable to create COG barcodes")
Exception: Unable to create COG barcodes

Another of these today:
2020-10-15 10:49:28,531 lighthouse.blueprints.plates:47 ERROR Unable to create COG barcodes
Traceback (most recent call last):
File "/code/lighthouse/blueprints/plates.py", line 45, in create_plate_from_barcode
centre_prefix = add_cog_barcodes(samples)
File "/code/lighthouse/helpers/plates.py", line 58, in add_cog_barcodes
raise Exception("Unable to create COG barcodes")
Exception: Unable to create COG barcodes


TensorFlow version problem

New installations of deepbgc fail because of the installed new versions of TensorFlow:
root@a492456632b1:~# pip install deepbgc
Collecting deepbgc
  Using cached deepbgc-0.1.18-py3-none-any.whl (63 kB)
Collecting matplotlib<3.1,>=2.2.3
  Using cached matplotlib-3.0.3.tar.gz (36.6 MB)
Collecting numpy<1.17,>=1.16.1
  Using cached numpy-1.16.6.zip (5.1 MB)
ERROR: Could not find a version that satisfies the requirement tensorflow<2.0.0,>=1.12.0 (from deepbgc) (from versions: 2.2.0rc1, 2.2.0rc2, 2.2.0rc3, 2.2.0rc4, 2.2.0, 2.3.0rc0, 2.3.0rc1, 2.3.0rc2, 2.3.0)
ERROR: No matching distribution found for tensorflow<2.0.0,>=1.12.0 (from deepbgc)
root@a492456632b1:~#

Here in the info for the installed TensorFlow:
root@a492456632b1:~# pip show tensorflow
Name: tensorflow
Version: 2.3.0
Summary: TensorFlow is an open source machine learning framework for everyone.
Home-page: https://www.tensorflow.org/
Author: Google Inc.
Author-email<EMAIL_ADDRESS>License: Apache 2.0
Location: /usr/local/miniconda3/lib/python3.8/site-packages
Requires: wrapt, absl-py, keras-preprocessing, google-pasta, gast, grpcio, wheel, tensorboard, h5py, astunparse, opt-einsum, tensorflow-estimator, six, scipy, numpy, termcolor, protobuf
Required-by:
root@a492456632b1:~#

Hi @ahmedmoustafa, thanks for the heads up! Are you able to install with Python 3.7? I recommend installing using conda, which can also download the supported Python version for you.
Looks like on Python 3.8, only Tensorflow >= 2.2 is supported, and porting our code would currently be a big effort. I updated the README to reflect that.

Hi @prihoda I used to be able to install fine with Python 3.7 but no longer with 3.8. Thank you so much for the recommendation.


Add Mac M1 build

Title is clear :)

Can you test the binary if I provide one?

Of course I can

Please try out the binary at https://github.com/ahmetb/kubectx/releases/tag/v0.9.3.
For the time being, both Homebrew and Krew distributions use the old bash-based implementation (as that’s intended). In the future, we’ll be packaging and distributing the binaries instead.

Works like a charm :) Sagol!